Jul 21, 2022
Version 3
Standards-Compliant General Protocol for Systematic Reviews V.3
- 1Lancaster University, Evidence-Based Toxicology Collaboration;
- 2WAP Academic Consultancy
Protocol Citation: Paul Whaley, Stephen M Wattam 2022. Standards-Compliant General Protocol for Systematic Reviews. protocols.io https://dx.doi.org/10.17504/protocols.io.n92ldydzxl5b/v3Version created by Paul Whaley
License: This is an open access protocol distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Protocol status: In development
We are still developing and optimizing this protocol.
Created: July 21, 2022
Last Modified: July 21, 2022
Protocol Integer ID: 67278
Keywords: systematic review, environmental health, toxicology, protocol,
Funders Acknowledgement:
Lancaster University
Grant ID: Impact Acceleration Award
Abstract
This general protocol is designed to support the planning of SRs that are concerned with the effects of environmental exposures on human health outcomes, or the prevention or mitigation thereof.
It has been designed specifically to help researchers follow the COSTER recommendations for the conduct of systematic reviews, in a way that complies with the ROSES and PRISMA-P checklists for reporting SR protocols, PRISMA-S for search strategies, and provides data for PROSPERO records.
Note that due to some inconsistencies between the standards and checklists, not every element of every checklist is covered. A cross-walk of the standards and checklists will be provided with forthcoming protocol documentation. Code is being written to help generate automated draft protocol reports.
If you use the protocol in a systematic review, please cite not only this instance of the protocol template, but also the parent COSTER manuscript, DOI 10.1016/j.envint.2020.105926.
Training in use of the protocol is available, and comments are welcome. Please contact the author, Paul Whaley, for more information.
Image Attribution
Image by Paul Whaley.
Guidelines
Protocols.io has not yet been optimised as a means for reporting what was done in response to complex instructions such as those found in this protocol. Feedback on use of the protocol, and how to develop it to facilitate reporting of planned methods, would be very much appreciated.
Before start
Note on standards compliance: Simply following this template does not guarantee the same degree of compliance with all steps of the SR protocol development process. This is because some steps not only require something to be done, but for it to be done well. Compare, for example, selecting team members to fill competencies (step 2) with defining "appropriate" eligibility criteria (step 9). Author teams are recommended to implement some form of external auditing in their protocol development process, to help ensure that the planned methods are robust (see step 27).
A. Securing capacity, competencies, and tools
A. Securing capacity, competencies, and tools
4h
4h
[osf_record] Create an Open Science Framework (OSF) record for the project.
PRISMA-P 1a, 1b / PROSPERO 1
- Create an OSF record for the project (this template may help https://osf.io/zjevp/ - click the fork button, then click "duplicate template", and rename the template with your project name).
- Name the protocol "XYZ: a systematic review protocol" or "XYZ: a protocol for an update of a systematic review", as appropriate
- Paste the OSF record URL into the box below.
Mark the step as complete when the OSF record URL has been inserted below.
Required
[rec_osfurl] OSF Record URL[rec_title] Title of protocol
.
Note
Note that in the following protocol, file naming conventions are specified. Take care to ensure these are followed, as they are important for automatic extraction of data from OSF records.
Template documents have been created to assist with reporting information relevant to fulfilling the recommendations of COSTER and the PRISMA-P and ROSES reporting guidelines. These are linked from relevant steps below. The templates are only suggestive and can be changed to suit the protocol being developed.
[competencies] Assign team members to core systematic review competencies.
COSTER 1.1.1 / PRISMA-P 3b
- Assign a team member to each of the competencies listed below.
Mark the step as complete when at least one team member has been assigned to each competency.
[comp_infosci] Information science
[comp_eviapp] Evidence appraisal methods
[comp_stats] Statistical methods
[comp_topic] Topic expertise
[comp_srmeth] Systematic review methods
[comp_guarant] Guarantor of review
4h
[info_manage] Identify information management practices and tools for each stage of the review.
COSTER 1.1.2. / PRISMA-P 11a / PROSPERO 26
- State the software and relevant packages to be used in the review, using the form below.
Mark the step as complete when all the software to be used in the review has been catalogued.
[info_refman] Reference manager
[info_knowman] Knowledge management tool
[info_srsoft] Systematic review software
[info_statspack] Statistics software and packages
[info_aitools] AI support tools
[funding] State the funding sources of the systematic review.
PRISMA-P 5a, 5b, 5c / PROSPERO 12
Mark the step as complete when information about funding sources has been provided.
[fund_sources] Sources of financial and other support
[fund_funders] Names of review funders and/or sponsors
[fund_grantnum] Grant numbers (if relevant)
[fund_fundrole] Role of funders or sponsors in developing the protocol
[interests] List the potential conflicts of interest of the authors, and manage any identified apparent conflicts of interest.
COSTER 1.1.3 and 1.1.4. / ROSES for SR Protocols item 40 / PROSPERO 13
- Create a .docx file declaring the financial and non-financial interests of each of the contributors to the SR (a template that may be useful is here: https://osf.io/ejcyw/)
- Name each file "DisclosureOfInterest_LastnameFirstname"
- Evaluate the contributor interests for apparent conflicts
- Exclude from decision-making processes authors who have apparent conflicts of interest
- Summarise the apparent conflicts in a file named "ApparentConflicts" (even if it is to declare none were identified), including steps taken to remove authors with apparent conflicts from relevant decision-making roles
Mark the step as complete after a full set of author-completed Conflict of Interest Disclosure forms have been uploaded to the OSF record (#2), and the summary of apparent conflicts document has been uploaded (#5).
Note
Declarations of interest should include any financial and non-financial interests which readers need to be aware of in order to understand the motivations of the authors of the review.
Listing of interests as potential confirms that they are not apparent conflicts of interest, i.e. they cannot reasonably be expected to compromise the integrity of the systematic review. People with apparent conflicts of interest should be excluded from decision-making roles in the review.
Users of this protocol may wish to use the template below, that has been designed specifically around the COI recommendations of COSTER. Other DOI forms exist, e.g. ICMJE.
Declaration of Interests Form.docx B. Setting the research question ("problem formulation")
B. Setting the research question ("problem formulation")
[need] Demonstrate the need for a new review.
COSTER 1.2.1 / PRISMA-P item 6 / ROSES for SR Protocols items 6 and 7
- Create a .docx format document called "DemonstrationOfNeed" (a template that may be useful is here: https://osf.io/vp7ak/)
- Describe the scientific value of the question(s), i.e. why it is important that it be investigated.
- Describe the importance to stakeholders and/or affected parties of the question(s) being asked
- Summarise existing primary research and reviews that justify conducting a new systematic review
- Upload the document to the OSF record
Mark the step as complete when a document detailing the above items has been uploaded to the OSF record.
Note
- Description of the condition
- Description of the exposure or intervention
- Why it is important to do this review
MECIR also recommends explanation of how the intervention or exposure might work. This is covered under the "scientific rationale" step below.
[rationale] Articulate the scientific rationale for each question via development of a theoretical framework.
COSTER 1.2.2. / PRISMA-P 6 / ROSES for SR Protocols item 6
- Create a .docx format document called "TheoreticalFramework"
- Present the theoretical framework for each question in the document.
- Upload the document to the OSF record
Mark the step as complete when the theoretical framework document has been uploaded to the OSF record.
Note
The theoretical framework should describe how the exposure might have its effect or the proposed intervention might work. The theoretical framework should include discussion of the biological plausibility of the relationship being investigated.
[question] State the research question or questions to be answered by the systematic review. For each research question, define a statement of the research objective in terms of Population, Exposure or Intervention, Comparator, and Outcome.
COSTER 1.2.3. / PRISMA-P 7 / ROSES for SR Protocols items 7 and 8
- Define the PECO statement or equivalent via completing each substep below.
Mark this step as complete when all relevant substeps are complete.
CITATION
[q_pop] Define the target Population of interest. These are the entities which are being subjected to the exposures or interventions of interest.
Mark this step as complete when relevant population information has been provided.
[q_pop_species] Species
[q_pop_sex] Sex
[q_pop_age] Age
[q_pop_health] Health status
[q_pop_add] Additional characteristics
[q_exp] Define the target Exposure or Intervention of interest. This concerns the administered or observed change in conditions of the objects of investigation. It should include timing, duration and dose.
- Exposure: What is the exposure or intervention?
- Timing: When does the exposure or intervention happen?
- Duration: For how long does the exposure or intervention last?
- Dose: What is the dose regimen (amount, frequency)?
- Additional: Are there any other relevant characteristics of the exposure or intervention?
Mark this step as complete when relevant exposure / intervention information has been entered.
[q_exp_exp] Exposure or Intervention
[q_exp_timing] Timing
[q_exp_dur] Duration
[q_exp_dose] Dose
[q_exp_add] Additional characteristics
[q_comp] Define the target Comparator of interest. This concerns the characteristics of the exposure or intervention being used as the comparator to which the target exposure or intervention is being compared.
Mark this step as complete when relevant comparator information has been entered.
[q_comp_exp] Comparator
[q_comp_timing] Timing
[q_comp_dur] Duration
[q_comp_dose] Dose
[q_comp_add] Additional characteristics
[q_outcome] Define the target Outcome(s) of interest. This concerns the change being measured in the exposure or intervention group. These should be the primary outcomes of interest to the systematic review which form the hypothesis or hypotheses being tested. Secondary outcomes can also be listed.
Mark this step as complete when the primary and secondary outcomes have been entered.
[q_out_primary] Primary outcomes
[q_out_second] Secondary outcomes
C. Defining the eligibility criteria and designing the process for screening evidence for inclusion
C. Defining the eligibility criteria and designing the process for screening evidence for inclusion
[eligibility] Define and justify unambiguous and appropriate eligibility criteria for each component of the objective statement.
COSTER section 1.3 / PRISMA-P 8 / ROSES item 20
- Create a .docx format document named "EligibilityCriteria"
- Provide a complete list of design and reporting criteria criteria for deciding whether to include or exclude a study (the EligibilityCriteria template on the template OSF archive may be useful)
- Upload the document to the OSF record
Mark the step as complete when the EligibilityCriteria document has been uploaded to the OSF record.
Note
Note the following COSTER recommendations of relevance to this step of the protocol development process:
- 1.3.6. Include all relevant, publicly-available evidence, except for research for which there is insufficient methodological information to allow appraisal of internal validity.
- 1.3.7. Include evidence which is relevant to review objectives irrespective of whether its results are in a usable form.
- 1.3.8. Include relevant evidence irrespective of language.
- 1.3.9. Exclude evidence which is not publicly available.
Including all languages and all grey literature may be beyond the capacity of a review team. If some or all grey literature is to be excluded, the protocol should explain why and anticipate its implications as a limitation of review methods.
[screening] Define the method for screening each piece of evidence for inclusion. COSTER 3.1 recommends this be conducted by at least two people working independently, with an appropriate process (e.g. third-party arbitration) for identifying and settling disputes.
COSTER 3.1 / PRISMA-P 11b / ROSES 18, 19, 20
Mark this step as complete when the screening methodology has been described below.
[screen_team] Team members conducting screening
[screen_dupe] Duplicate screening or equivalent approach (describe)
[screen_conflict] Conflict resolution method
[multi_pubs] Define the strategy for handling multiple publications of the same study (e.g. multiple publications, conference abstracts etc.).
COSTER 3.4
Mark this step as complete when the multiple-publication strategy has been described.
[multi_pubs] Strategy for handling multiple publications of same study
.
Note
Note that COSTER item 3.4 recommends to not exclude multiple reports of the same research, but instead to collate the methodological information from each of the reports as part of the data extraction process for each unit of evidence.
D. Defining the strategy for searching for evidence relevant to the review objectives
D. Defining the strategy for searching for evidence relevant to the review objectives
[search] Design sufficiently sensitive search strategies, so that studies which meet the eligibility criteria of the review are not inadvertently excluded. Document the search methods in sufficient detail to render them transparent and reproducible.
COSTER 1.4.1, 2.6 / PROSPERO 17 / PRISMA-P 9, 10
- Create a .docx format document named "LiteratureSearch" (or use the template provided at https://osf.io/2ysk3)
- Complete each of the sub-steps below
- Upload the complete document to the OSF record
Mark this step as complete when all relevant sub-steps are complete and the LiteratureSearch document has been uploaded to the OSF Record.
CITATION
[s_databases] Search all the key scientific databases for the topic, including national, regional and subject-specific databases.
COSTER 2.1 / PRISMA-P 9 / ROSES 9
- List the databases, platforms, and study registries being searched in the LiteratureSearch document
Mark this step as complete when this information is added to the LiteratureSearch document.
[s_strings] Define search strings for each database, electronic and other source, using appropriate controlled vocabulary, free-text terms and logical operators in a manner which prioritises sensitivity.
COSTER 2.3, 2.6 / PRISMA-P 10 / ROSES 10
- Provide a complete search string for each database, platform, and registry in the LiteratureSearch document
Mark this step as complete when this information is added to the LiteratureSearch document.
CITATION
2d
[s_grey] Define reproducible strategies for identifying and searching sources of grey literature (databases, websites etc.). Document the search methods and results in sufficient detail to render them transparent and reproducible.
COSTER 2.2 and 2.6 / ROSES 9, 15 / PRISMA-P 10
- Provide a complete search method for each grey literature source in the LiteratureSearch document
Mark this step as complete when this information is added to the LiteratureSearch document.
2d
[s_hand] Search within the reference lists of included studies and other reviews relevant to the topic (“hand-searching”) and consider searching in the reference lists of documents which have cited included studies.
COSTER 2.4 / PRISMA-P 10 / ROSES 9
- Provide a citation searching method in the LiteratureSearch document
Mark this step as complete when this information is added to the LiteratureSearch document.
[s_contact] Search by contacting relevant individuals and organisations.
COSTER 2.5 / PRISMA-P 10 / ROSES 9
- Provide information about how individuals and organisations will be contacted for relevant evidence in the LiteratureSearch document
Mark this step as complete when this information is added to the LiteratureSearch document.
[rerun_search] Plan for re-running all searches and screen the results for potentially eligible studies within 12 months prior to publication of the review (screening at least at the level of title plus abstract).
COSTER 2.7 / ROSES 17
Mark this step as complete when the search re-run plan has been described in the boxes below.
[rerun_when] When will searches be updated?
[rerun_sources] Which sources will be searched again?
[rerun_level] What level of screening will be conducted?
[rerun_update] How will findings be updated?
E. Methods for Appraising Individual Studies
E. Methods for Appraising Individual Studies
[rob] Define the risk of bias assessment methods to be used for evaluating the internal validity of the included research. If observational studies are included, this should cover identification of plausible confounders.
COSTER 1.4.3 / PRISMA-P 14 / ROSES 22 / PROSPERO 27
- Create a .docx format document named "BiasAssessment" (or use the template provided at https://osf.io/h3egq/)
- Complete each of the sub-steps below
- Upload the complete document to the OSF record
Mark this step as complete when all relevant sub-steps are completed and the BiasAssessment document has been uploaded to the OSF Record.
Note
Review teams may find the FEAT (Focus-Extent-Application-Transparency) mnemonic to be useful in defining their risk of bias assessment methods. See Frampton et al. 2022 for a full guide.
- Focus: The focus of an appraisal tool should be the internal validity of a study. If other quality constructs are of interest, each should be assessed in a separate process.
- Extent: All the important threats to internal validity should be covered by the tool. If observational studies are being appraised, the threats should include all important confounders.
- Application: The appraisal process should produce consistent, accurate descriptions of the extent to which a study is vulnerable to each identified threat to internal validity. The judgements should be in a form which can be logically incorporated into the evidence synthesis.
- Transparency: The reason for each judgement should be documented, quoting as justification relevant text from the study documentation.
[rob_roles] Conduct risk of bias assessment with at least two people working independently, with an appropriate process (e.g. third-party arbitration) for identifying and settling disputes.
COSTER 5.3 / PROSPERO 27
[rob_roles_team] Team members conducting RoB assessment
[rob_roles_dupe] Duplicate assessment or equivalent method (describe)
[rob_roles_conflict] Conflict resolution method
[rob_tool] Select and modify as necessary a risk of bias assessment instrument.
COSTER 1.4.3 / PRISMA-P 14 / ROSES 22 / PROSPERO 27
- Add tool selection and modification information to the BiasAssessment document
Mark this step as complete when this information is added to the BiasAssessment document.
CITATION
[rob_summary] Define how the results of the risk of bias assessment will be summarised for each included study.
COSTER section 5 / PRISMA-P 14 / ROSES 23
- Add this information to the BiasAssessment document
Mark this step as complete when this information is added to the BiasAssessment document.
[rob_use] Define how the results of the risk of bias assessment will be used in developing the results and findings of the systematic review.
COSTER section 5 / PRISMA-P 14 / ROSES 23 / PROSPERO 27
- Add this information to the BiasAssessment document
Mark this step as complete when this information is added to the BiasAssessment document.
[other_appr] Define any other methods for critically appraising the included studies, any instruments for so doing, how the results of the appraisal of each study will be summarised, and how the results will be used in developing the results and findings of the systematic review.
COSTER 7.8 / ROSES 22
- Create a .docx format document named "OtherAppraisal" (or use the template provided at https://osf.io/2z6bf)
- Complete each section of the template (it follows the pattern of the BiasAssessment document)
- Upload the complete document to the OSF record
Mark this step as complete when the OtherAppraisal document is uploaded to the OSF record (or skip this step if not relevant to the SR).
F. Evidence Synthesis Methods
F. Evidence Synthesis Methods
[char_incl] Design the "characteristics of included studies" table.
COSTER 1.4.2
- Create a .docx format document named "CharacteristicsOfStudies" (or use the template provided at https://osf.io/h9sd8)
- Provide information about the data to be presented in the characteristics of included studies table
- Upload the complete document to the OSF record
Mark this step as complete when the CharacteristicsOfStudies document is uploaded to the OSF record.
Note
The characteristics of included studies table should summarise the key information a reader will need in order to understand the basic study information being synthesised by the SR into its overall findings.
[extract_forms] Design the data extraction forms.
COSTER 1.4.7 / PRISMA-P 12
- Design the data extraction forms
- Create a .docx format document named "ExtractionForms"
- Paste screenshots of the forms into the document
- Upload the complete document to the OSF record
Mark this step as complete when the ExtractionForms document is uploaded to the OSF record.
[synth] Design the methods for synthesising the included studies.
COSTER 1.4.4, section 6 / ROSES 30-35 / PRIMSA-P 15-16
This should cover:
- qualitative and quantitative methods (with full consideration given to synthesis methods to be used when meta-analysis is not possible);
- assessment of heterogeneity;
- choice of effect measure (e.g. RR, OR etc.);
- methods for meta-analysis and other quantitative synthesis;
- pre-defined, appropriate effect modifiers for sub-group analyses.
- Create a .docx format document named "SynthesisMethods" (or use the template provided at https://osf.io/8a43q)
- Describe in the document the synthesis methods to be used (these are complex, so use of the template is recommended)
- Upload the complete document to the OSF record.
Mark this step as complete when the SynthesisMethods document is uploaded to the OSF record.
Note
Refer to section 6 of COSTER for detailed recommendations for how evidence should be synthesised in systematic reviews. Popay et al. (2006) describes several useful approaches to structuring non-quantitative evidence syntheses.
[certainty] Define the methods for determining how, given strengths and limitations of the overall body of evidence, confidence in the results of the synthesis of the evidence for each outcome is to be captured and expressed. (For reviews which include multiple streams of evidence, this may need to be defined for each stream.)
COSTER 1.4.5 / ROSES 22 / PRISMA-P 17
- Create a .docx format document named "CertaintyAssessment" (or use the template provided at https://osf.io/vh29q/)
- Describe in the document the certainty assessment methods to be used
- Upload the complete document to the OSF record.
Mark this step as complete when the CertaintyAssessment document is uploaded to the OSF record.
[integration] For reviews which include multiple streams of evidence (e.g. animal and human studies), define the methods for integrating the individual streams into an overall result.
COSTER 1.4.6
This should include a description of the relative relevance of populations (e.g. species, age, comorbidities etc.), exposures (e.g. timing, dose), and outcomes (direct or surrogate, acute or chronic model of disease, etc.), as appropriate, per which inferences about predicted effects in target populations can be made from observed effects in study populations.
- Create a .docx format document named "EvidenceIntegration" (or use the template provided at https://osf.io/gc8ep/)
- Describe in the document the evidence integration methods to be used
- Upload the complete document to the OSF record.
Mark this step as complete when the EvidenceIntegration document is uploaded to the OSF record.
G. Pilot Testing (COSTER 1.4.7)
G. Pilot Testing (COSTER 1.4.7)
[pilot_screening] Pilot test the screening process. Make any necessary changes to the documentation of the screening process, described from , as needed (Section C).
COSTER 1.4.7
A general protocol for piloting the screening stage of a systematic review is available here:
Mark this step as complete when the screening process has been piloted and changes to methods finalised.
CITATION
[pilot_extraction] Pilot test data extraction forms. Make changes to the documentation of the data extraction process, described in , as needed.
COSTER 1.4.7 / ROSES 25, 26 / PRISMA-P 11c
Mark this step as complete when the screening process has been piloted and changes to methods finalised.
Note
Any piloting step should be applied to a representative range of studies eligible for inclusion in the systematic review. How will the review team be trained in use of the tool? How will the piloting process be judged as successful, and the planned methods determined to be performing acceptably?
[pilot_rob] Pilot test the risk of bias assessment. Make changes to the documentation of the risk of bias assessment process, described in , as needed.
COSTER 1.4.7
Mark this step as complete when the risk of bias assessment methodology has been piloted and changes to methods finalised.
[pilot_certainty] Pilot the process for the assessment of confidence in the results of the synthesis of the evidence. Make changes to the documentation of the certainty assessment process, described in , as needed
COSTER 1.4.7
Mark this step as complete when the certainty assessment methodology has been piloted and changes to methods finalised.
H. Registering the protocol
H. Registering the protocol
[write] Create a protocol document from the steps and documents created above.
Note
We are working on a script to parse the OSF record and the protocol run (it will create a draft document up to and including the previous step; will need editing for flow)
[registration] Create a permanent public record of intent to conduct the review (e.g. by registering the protocol in an appropriate registry). Do this prior to conducting the literature search.
COSTER 1.5.1 / PRISMA-P 2
Mark this step as complete when a unique identifier and link to the registration record has been added below.
[reg_registry] Registry used
[reg_ident] DOI or other identifier
[reg_url] URL
I. Publishing the protocol
I. Publishing the protocol
[peer_review] As appropriate for review planning and question formulation, secure peer-review and public feedback on the draft version of the protocol, incorporating comments into the final version of the protocol.
COSTER 1.5.2
Mark this step as complete when peer-review comments have been incorporated into the protocol.
[publication] Publish the final version of the protocol in a public archive. Do this prior to screening studies for inclusion in the review. This should be a read-only, date-stamped record on a third-party controlled website.
COSTER 1.5.3
Mark this step as complete when a unique identifier and registration record has been added below.
[pub_location] Location of protocol
[pub_ident] DOI or other identifier
[pub_url] URL
Well done! You have completed your SR protocol and can start screening the literature and following your meticulously-planned methods!
Citations
Step 12
Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, Koffel JB, PRISMA-S Group.. PRISMA-S: an extension to the PRISMA statement for reporting literature searches in systematic reviews.
https://doi.org/10.5195/jmla.2021.962Step 12.2
Gusenbauer M, Haddaway NR. What every researcher should know about searching - clarified concepts, search advice, and an agenda to improve finding in academia.
https://doi.org/10.1002/jrsm.1457Step 14.2
Frampton, G., Whaley, P., Bennett, M. et al.. Principles and framework for assessing the risk of bias for studies included in comparative quantitative environmental systematic reviews.
10.1186/s13750-022-00264-0Step 21
Paul Whaley, Ruth Garside, Jacqualyn F Eales. A General Protocol for Pilot-Testing the Screening Stage of a Systematic Review (Manual)
dx.doi.org/10.17504/protocols.io.bkc9ksz6Step 8
Morgan RL, Whaley P, Thayer KA, Schünemann HJ. Identifying the PECO: A framework for formulating good questions to explore the association of environmental and other exposures with health outcomes.
https://doi.org/10.1016/j.envint.2018.07.015