Oct 28, 2024

Public workspaceAdherence to the PRISMA 2020 guidelines and the use of automation tools in the random sample of 1000 systematic reviews: a meta-epidemiological study

This protocol is a draft, published without a DOI.
  • Paul Posadzki1,
  • Jos Kleijnen <jos@systematic-reviews.com>2,
  • Ram Bajpai3,
  • Wojciech Kusa3,
  • Ernst, Edzard3
  • 1AWF Krakow;
  • 2KSR;
  • 3Keele University
Paul Posadzki
Icon indicating open access to content
QR code linking to this content
Protocol Citation: Paul Posadzki, Jos Kleijnen <jos@systematic-reviews.com>, Ram Bajpai, Wojciech Kusa, Ernst, Edzard 2024. Adherence to the PRISMA 2020 guidelines and the use of automation tools in the random sample of 1000 systematic reviews: a meta-epidemiological study. Protocol exchange https://protocols.io/view/adherence-to-the-prisma-2020-guidelines-and-the-us-dqmg5u3w
License: This is an open access protocol distributed under the terms of the Creative Commons Attribution License,  which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Protocol status: Working
We use this protocol and it's working
Created: October 27, 2024
Last Modified: October 28, 2024
Protocol Integer ID: 110984
Keywords: Meta-epidemiological study; automation tools; systematic reviews; PRISMA guidelines
Abstract
  • There is an ongoing debate about the use of artificial intelligence (AI) driven automation tools in systematic reviews (SRs).
  • Poor reporting and non-adherence to the most recent reporting guidelines continues to be an issue in SRs in Health & Medical Sciences.
Guidelines
n/a
Materials
n/a
Safety warnings
n/a
Ethics statement
n/a
Before start
n/a
Aims and objectives:
Aims and objectives:
To calculate adherence to the PRISMA 2020 guidelines in the random sample of 1000 systematic reviews published in the  KSR Evidence Database in 2022 (and in 2023)
To compare trends in adherence to the PRISMA 2020 guidelines between SRs published in 2022 versus 2023 using two different KSR Evidence samples/datasets.
To evaluate the prevalence of AI use for the screening purposes (at the title and abstract stage) in these same random samples 2022 and 2023.
To compare trends in AI use for the screening purposes between SRs published in 2022 versus 2023 using two different KSR Evidence samples/datasets.
To compare the accuracy of different prompts in extracting the data.
To compare the performance of the different GPT models i.e., Turbo vs Enhanced Memory vs ADA.
Methods:
Methods:
We will ‘ask’ ChatGPT (write five different prompts for each of the two below concepts) to ‘extract’ the data from the sample of 1000 SRs published in the KSR Evidence Database in 2022.  
These prompts will focus around two concepts: 
  • Prevalence of adherence to PRISMA 2020 guidelines.
  • Prevalence of SRs using automation tools for screening titles and abstracts
Then, we will ask ChatGPT to create a table in excel (C,S,V) for all 1000 SRs. Subsequently, a random sample (www.randomiser.org) of 100 systematic reviews will be validated by a human reviewer; and intra-correlation coefficient (Kappa) will be calculated.
Inclusion criteria:
Systematic reviews and meta-analyses in this domain only -> health sciences; field -> clinical medicine (and 32 subfields as classified by Ioannidis et al. 2023); and focussing on therapies, medicines (including adverse effects of those), drugs, devices, interventions will all be eligible. Studies published in any language will be included. Only ‘new systematic reviews’, published in 2022 and 2023 and those that  had searched in databases, registers and other sources (for the available primary studies) will be eligible.
Exclusion Criteria:
We will exclude SRs of diagnostic tests/tools, predictive, prognostic or prognostic modelling studies, qualitative studies; narrative, umbrella, scoping, aetiological, prevalence/incidence, realists, mixed-methods, rapid, scoping reviews, and any reviews not structured as SRs or meta-analyses. Will also exclude updated (or update) systematic reviews; and those published outside of the above-mentioned timeframe as well as SRs published as abstracts only. SRs of allied health professions e.g., physiotherapy, nursing will be excluded as well as those in social sciences, e.g., psychology (as per the above criteria).
Protocol references
Acknowledgements
none