Aug 02, 2022
Open Science and COVID-19 RCTs
- 1Stanford University
Protocol Citation: John Borghi 2022. Open Science and COVID-19 RCTs. protocols.io https://dx.doi.org/10.17504/protocols.io.x54v9jx7zg3e/v1
License: This is an open access protocol distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Protocol status: Working
We use this protocol and it's working
Created: September 14, 2021
Last Modified: August 02, 2022
Protocol Integer ID: 53222
Keywords: Open Science, Data Sharing, Open Access, Preprints, Reproducibility
Abstract
Open science is an umbrella term that covers a wide variety of efforts focused on making scientific research more transparent and accessible. Practices related to open science exist along a continuum of openness and an individual research effort can be more or less open along a number of dimensions- including the accessibility of related publications, availability of data and code, and application of techniques such as pre-registration that enhance transparency related to project aims and methods. Among the many motivations for adopting open science practices is the link between openness and reproducibility - the more transparent the research effort, the more easily its conclusions can be examined, evaluated, and built upon by others.
This protocol outlines our methods for assessing the openness of randomized controlled trials (RCTs) related to the prevention and treatment of COVID-19. RCTs are commonly described as the "gold standard" of evidence in clinical research, which underlines the importance of their results are transparent.
Literature Searching
Literature Searching
Search Strategies
The following are the copy and paste-able versions of our literature searches.
PUBMED
(("randomized controlled trial" [pt] OR "controlled clinical trial" [pt] OR randomized [tiab] OR randomly [tiab] OR randomised [tiab] OR randomization [tiab] OR randomisation [tiab] OR placebo* [tiab] OR placebo* [ot] OR random* [ot] OR pragmatic [tiab] OR pragmatic [ot] OR blind* [ot] OR "Pragmatic Clinical Trial" [Publication Type] OR blind* [tiab]) AND ("covid 19" [tw] OR covid [tw] OR covid19 [tw] OR "ncov 2019" [tw] OR "novel coronavirus" [tw] OR "sars cov 2" [tw] OR "sars cov-2" [tw] OR "ncov 2019" [tw] OR sarscov2 [tw] OR (wuhan [tw] AND coronavirus* [tw]) OR "corona virus*" [tw] OR "coronavirus disease 2019" [tw] OR "coronavirus disease 19" [tw] OR "2019 ncov" [tw] OR 2019nCoV [tw] OR "coronavirus 2" [tw] OR "Coronavirus"[Mesh:NoExp] OR "SARS-CoV-2"[Mesh] OR "COVID-19 Testing"[Mesh] OR "COVID-19" [mesh] OR "COVID-19 Vaccines"[Mesh] OR "Receptors, Coronavirus"[Mesh] OR "Spike Glycoprotein, Coronavirus"[Mesh] OR "SARS-CoV-2 variants" [Supplementary Concept]) AND english [lang] NOT (review [pt] OR letter [pt] OR editorial [pt] OR "Systematic Review" [Publication Type] OR "Meta-Analysis" [Publication Type] OR "mental disorders" [mesh] OR depression [ti] OR psychology [sh] OR case reports [pt])
EMBASE
((randomized*:ti,ab,kw OR randomly:ti,ab,kw OR randomised:ti,ab,kw OR randomization:ti,ab,kw OR randomisation:ti,ab,kw OR placebo*:ti,ab,kw OR blind*:ti,ab,kw OR pragmatic:ti,ab,kw OR 'randomized controlled trial'/exp OR 'adaptive clinical trial'/exp OR 'controlled clinical trial'/exp) AND ('coronavirus disease 2019'/exp OR 'SARS-Cov-2 variant 501Y.V1'/exp OR 'coronavirus spike glycoprotein'/exp OR 'novel coronavirus':ti,kw,ab OR 'ncov 2019':ti,kw,ab OR 2019nCoV:ti,kw,ab OR 'corono virus':ti,ab,kw OR 'corona virus disease 19':ti,ab,kw OR 'coronavirus 2':ti,ab,kw OR 'covid 19':ti,kw,ab OR 'COVID-19 testing'/exp OR 'SARS-CoV-2 vaccine'/exp OR 'coronavirus receptor'/exp OR ((covid NEAR/2 19):ti,kw,ab) OR ((sars NEAR/2 'cov 2'):ti,kw,ab) OR '2019 ncov':ti,kw,ab OR ((wuhan:ti,kw,ab OR hubei:ti,kw,ab) AND coronavirus*:ti,kw,ab) OR ([2019-2020]/py AND (new:ti,kw,ab OR novel:ti,kw,ab OR pandemic:ti,kw,ab OR epidemic:ti,kw,ab) AND ('coronavirus infection'/exp OR coronavirus*:ti,kw,ab OR coronavirinae/de OR 'corona virus*':ti,kw,ab))) AND 'human'/de AND [embase]/lim NOT ([embase]/lim AND [medline]/lim) AND [english]/lim) NOT ('mental disease'/exp OR depression:ti OR 'systematic review'/exp OR 'review'/exp OR 'editorial'/exp OR 'meta analysis'/exp OR 'case report'/exp OR 'letter'/exp)
COCHRANE
(randomized OR randomly OR randomised OR randomization OR randomisation OR placebo* OR pragmatic OR blind*) AND ("covid 19" OR covid OR covid19 OR "ncov 2019" OR "novel coronavirus" OR "sars cov 2" OR "sars cov-2" OR "ncov 2019" OR sarscov2 OR (wuhan AND coronavirus*) OR "corona virus*" OR "coronavirus disease 2019" OR "coronavirus disease 19" OR "2019 ncov" OR 2019nCoV OR "coronavirus 2") NOT (depression)
Filter to ("Journal" or Academic Journal"):PT
Inclusion and Exclusion Criteria
Inclusion
- The trial must be described in a published manuscript (i.e. not a preprint - We'll get to these)
- The study must be examining a pharmaceutical intervention designed to prevent or treat COVID-19.
- The study design must be a randomized controlled trial (RCT) - Meaning participants must be prospectively and randomly assigned to different groups.
Exclusion
- The report is not a published manuscript (i.e. a preprint)
- The study examines an intervention that is not designed to prevent or treat COVID-19.
- The study examines treatments/procedures that are impacted by COVID but don't directly bear on COVID (e.g. studies of "treating XYZ in the time of COVID").
- Published manuscripts that cover studies that are not RCTs (observational studies, retrospective studies, etc)
- Meta-analyses/reviews that synthesize the results of previously published trials.
- Protocols and other works published in advance of data collection.
Screening
All papers will be screened by two reviewers to ensure they meet inclusion/exclusion criteria. In cases where there is a disagreement between reviewers, a third will adjudicate.
Data Extraction/Data Availability
Data Extraction/Data Availability
Locating Data Availability Statements
Data availability statements are (typically) short statements that describe how the data and other materials that underlie a given work can be accessed. The ICMJE requires such statements for any published clinical trial.
To find a data availability statement for a given paper:
- Check PubMed Central - PMC has a discrete section for data availability statements. The PMCID for each paper will be found using the PMCID - PMID - Manuscript ID - DOI Converter and will be available in your reporting sheet.
- Check the supplementary materials. Typically if it's here it'll be labeled "data availability". Papers published in NEJM, JAMA, and other journals will have statements in supplementary materials, so please make sure you check.
- For some papers, the statement won't be separated on in PubMed Central or the paper won't be in PubMed Central at all. If that's the case, search for "Data Sharing", "Data Availablity" and variants in the full text (if available).
An example of a data availability statement in PubMed Central. According to our coding scheme, this would be marked as data is available and through a repository (Dryad).
An example of a structured data availability statement that is included as supplementary materials. Though the authors responded "No", there does appear to be a pathway for accessing the data. According to our coding scheme, this statement would be marked as data is available and through a repository (Vivli).
For some journals, the statement will be included as a supplementary file.
Coding Data Availability Statements - Availability
If you can locate a data availability statement, enter "Yes" in the "data_avail_statement" column of the reporting sheet. Otherwise, enter "No".
Once a data availability statement is located, it should be copied to the "data_avail_statement_text" column. If you can not find a data availability statement, leave this column blank.
We will delete this column before sharing the data for copyright (??) reasons.
Please indicate the availability of data in the "data_availability" column using the following criteria:
- If the data is available (by any mechanism) enter "yes"
- If the data is explicitly not available enter "no"
Not that sometimes authors will state that data is not available but will then indicate that data is available through a certain mechanism (e.g. "Upon request"). In these cases, mark "yes".
In general, take a very (very) liberal view here. If the statement does not indicate that the data is NOT UNAVAILABLE, we're going to count that as available.
If the statement indicates that the data will be made available later:
- If the information is given- record when it will be made available (in months post-publication) in the "data_later_months" column.
- If the data will be made available after "final publication", put a 1 in the "data_later_publication" column
- If the data will be made available after a product approval, put a 1 in the "data_later_product" column.
Coding Data Availability Statements - Mechanism
If the data availability statement indicates that the data is available:
- If the data is available via request to an author, put a "yes" in "data_avail_request_author"
- If the data is available via request to a study sponsor, IRB, or other entity, put a "yes" in "data_avail_request_other"
- If the data is included in the manuscript and/or supplementary materials, put a "yes" in "data_avail_paper"
- If the data is available via a data repository, put a "yes" in "data_avail_repo"
- If the data availability statement points to a trial registry, put a "yes" in "data_avail_registry"
- If the data availability statement states that the data is available through some other means, put a "yes" in "data_avail_other"
- If it is unclear how the data is available, put a "yes" in "data_avail_unclear"
Data may be made available through multiple mechanisms, please mark every method that is described in the data availability statement.
- If the data availability statement indicates that data is available in a repository, enter the name of the repository in "data_avail_repo_name"
Finding Preprints
Finding Preprints
Finding Preprints
PubMed has begun to index preprints as part of the NIH Preprints Pilot. Please use the following workflow to try to identify if a paper has a corresponding preprint.
If a preprint is found:
- Enter "Found" in the "preprint_found" column, otherwise enter "Not Found".
- Copy the DOI of the preprint in the "preprint_doi" column
- Enter the name of the preprint server in the "preprint_server" column.
Check to see if the paper has an associated preprint in Europe PMC. The PMCID for the corresponding paper will be in the reporting sheet.
Europe PMC makes linked preprints very obvious, click through the link to identify the preprint server. Here is an example of where this works.
Europe PMC indexes preprints from a number of preprint servers, when a paper can be associated with a previously available preprint, you'll see the above message on that paper's page.
If no preprint is found using the above method, it may be that it was posted with a slightly different title. Here is an example of a paper and a preprint with different names.
To check, either click through the first author's name to see their publications or create a search string that contains the author's first and last name as well as (SRC:PPR) in the manner shown below. This will show the list of all the author's indexed preprints.
Take a quick scan through the results to see if there any titles that are like the paper's. If there are, put a Yes in the "preprint_possible" column.
We'll check on these later.
Finding Study Info
Finding Study Info
Registration Information
After clicking through the full text of the article, search for the term "registration" and variants.
- If you are able to find a registration, enter Yes in the "register" column in the reporting sheet. Otherwise, enter No.
- If you are able to find a registration, enter the ID in the "register_id" column.
Study Information
Though we're not doing the critical appraisal, some information about the study design will be helpful for our later analyses. In the "intervention" column, record the intervention being studied. We will normalize names (e.g. BNT162b2 vs tozinameran) later