Nov 21, 2022
Completeness in reporting of surrogate primary endpoints in Randomised Controlled Trials: A targeted review protocol
- Anthony Muchai Manyara1,
- Sathish Venkatasamy Dhayalan2,
- Philippa Davies3,
- Amber Young3,
- Imad Adamestam4,
- Onyeka Obuaya4,
- Valerie Wells1,
- Christopher J Weir4,
- Jane Blazeby5,
- Rod S Taylor1,6,
- Oriana Ciani7
- 1MRC/CSO Social and Public Health Sciences Unit, Institute of Health and Wellbeing, University of Glasgow, UK;
- 2James P Grant School of Public Health, BRAC University, Dhaka, Bangladesh;
- 3Population Health Sciences, Bristol Medical School, University of Bristol, UK;
- 4Edinburgh Clinical Trials Unit, Usher Institute, University of Edinburgh, UK;
- 5Bristol NIHR Biomedical Research Centre, University Hospitals Bristol and Weston NHS Foundation Trust, and University of Bristol;
- 6Robertson Centre for Biostatistics, Institute of Health and Well Being, University of Glasgow, UK;
- 7SDA Bocconi School of Management, Milan, Italy
Protocol Citation: Anthony Muchai Manyara, Sathish Venkatasamy Dhayalan, Philippa Davies, Amber Young, Imad Adamestam, Onyeka Obuaya, Valerie Wells, Christopher J Weir, Jane Blazeby, Rod S Taylor, Oriana Ciani 2022. Completeness in reporting of surrogate primary endpoints in Randomised Controlled Trials: A targeted review protocol. protocols.io https://dx.doi.org/10.17504/protocols.io.81wgbyx1nvpk/v1
License: This is an open access protocol distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Protocol status: Working
We use this protocol and it's working
Created: November 11, 2022
Last Modified: November 21, 2022
Protocol Integer ID: 72609
Keywords: Surrogate endpoints, transparency, randomised controlled trials, Reporting guidelines
Funders Acknowledgement:
The review is part of the development of SPIRIT and CONSORT extensions funded by the Medical Research Council
Grant ID: MR/V038400/1
Onyeka Obuaya is funded by the Medical Research Council, University of Edinburgh and University of Glasgow, as part of the Precision Medicine Doctoral Training Program
Grant ID: MR/W006804/1
Abstract
Introduction
Using a surrogate endpoint as a substitute for a patient-relevant final outcome enables randomised controlled trials (RCTs) to be conducted more efficiently. However, there is currently no consensus-driven reporting guideline for RCTs using a surrogate endpoint as a primary outcome. Therefore, we are developing SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) and CONSORT (Consolidated Standards of Reporting Trials) extensions to improve the design and reporting of these trials. As an initial step, a targeted review will identify participants to contribute to a Delphi consensus process and document current reporting of surrogate endpoints in trial protocols and reports.
Methods and analysis
We will search for RCT reports and protocols published from 2017 to mid-2022 in six high impact general medical journals (Annals of Internal Medicine, BMJ, Journal of the American Medical Association, New England Journal of Medicine, Lancet, and PLoS Medicine) and two journals that commonly publish protocols: BMJ Open and Trials. Analysis will be done using frequencies and simple thematic analysis.
Ethics and dissemination
Ethical approval is not required. The review will support the development of SPIRIT and CONSORT extensions for reporting surrogate primary endpoints (surrogate endpoint as the primary outcome) and future evaluation of the impact of the extensions. The findings will be published in an open-access publication.
Attachments
578-1213.docx
168KB
Guidelines
Introduction
Well designed, conducted and reported randomised controlled trials (RCTs) provide rigorous evidence for evaluating health interventions. To generate such rigorous evidence that meets scientific, statistical, legal, and ethical considerations, trials have increasingly become resource and time intensive [1]. Consequently, use of surrogate endpoints instead of patient relevant final outcomes can improve trial efficiency, i.e. shorter follow-up, smaller sample size hence lower cost [1]. Furthermore, surrogates that truly inform subsequent patients longer term endpoints are particularly valuable in pragmatic trials where long-term follow up is difficult - e.g., trials of interventions to reduce long term complications of diabetes. Despite this efficiency offered by surrogate endpoints, they are concerns about their limitations. Compared to patient relevant final outcomes, surrogate endpoints in trials have been found to overestimate health benefits [2]. Furthermore, some approvals of interventions based on surrogate endpoints has led to roll out of interventions with no benefit or that were harmful due to the surrogate not being in the causal pathway of the disease or unintended effects of the intervention [3, 4]. Given these limitations, trials that use primary outcomes which are surrogate endpoints should be clear on having used a surrogate, its validity in predicting intervention effect and the uncertainty and risks associated with its use [5]. However, most trials that use surrogate primary endpoints are not transparent in this regard: an analysis of 626 trials published in 2006 and 2007 found that of the 109 using a surrogate primary endpoint, 62 (57%) reported that the primary outcome was a surrogate endpoint and only 38 (35%) discussed the validity of the surrogate [6]. Therefore, we aim to update this review and analyse the current practice in the use and reporting of surrogate endpoints in RCTs. This review is part of our project that aims to develop consensus-driven SPIRIT and CONSORT extensions for reporting surrogate primary endpoints: SPIRIT-SURROGATE and CONSORT-SURROGATE – see [7] for full project protocol. The project has four phases: literature reviews; e-Delphi survey; consensus meeting; and knowledge translation. This targeted review is part of the literature reviews phase [8] and will serve two purposes:
- To identify researchers that have used surrogate endpoints in RCTs of any design in the last five years, and invite them to participate in an e-Delphi survey
- To document current reporting of surrogate endpoints in trial protocols and reports as a baseline for future evaluation of SPIRIT-SURROGATE and CONSORT-SURROGATE
Ethics and dissemination
Ethics approval is not required for this review. Findings of the review will be published in an open access journal.
Amendments
This protocol can be amended in the course of the review. When amended, a date and rationale of amendment will be provided.
Funding
The review is part of the development of SPIRIT and CONSORT extensions funded by the Medical Research Council (grant number MR/V038400/1). Onyeka Obuaya is funded by the Medical Research Council, University of Edinburgh and University of Glasgow, as part of the Precision Medicine Doctoral Training Program (grant number MR/W006804/1).
Acknowledgments
We are indebted to Prof. Amber Young (University of Bristol) for her contribution the to the planning and conduct of the CONSORT/SPIRIT-SURROGATE project and who sadly passed away in September 2022.
Timeline of the review
We anticipate the review will be completed and submitted for publication in 9 months – see Table 1
| Activity | Period | |
| Literature search | July 2022 | |
| Screening | July-October 2022 | |
| Data extraction and synthesis | November-January 2023 | |
| Write-up and submission | February-March 2023 |
References
- Umscheid, C.A., D.J. Margolis, and C.E. Grossman, Key concepts of clinical trials: a narrative review. Postgraduate medicine, 2011. 123(5): p. 194-204.
- Ciani, O., et al., Comparison of treatment effect sizes associated with surrogate and final patient relevant outcomes in randomised controlled trials: meta-epidemiological study. BMJ : British Medical Journal, 2013. 346: p. f457.
- Grimes, D.A. and K.F. Schulz, Surrogate end points in clinical research: hazardous to your health. Obstet Gynecol, 2005. 105(5 Pt 1): p. 1114-8.
- Thomas R. Fleming and David L. DeMets, Surrogate End Points in Clinical Trials: Are We Being Misled? Annals of Internal Medicine, 1996. 125(7): p. 605-613.
- Ciani, O., A. Manyara, and R.S. Taylor, Need for better reporting of trials with surrogate endpoints: SPIRIT|CONSORT-SURROGATE extensions. Journal of Epidemiology and Community Health, 2022. 76(9): p. 769-770.
- la Cour, J.L., J. Brok, and P.C. Gøtzsche, Inconsistent reporting of surrogate outcomes in randomised clinical trials: cohort study. BMJ, 2010. 341: p. c3653.
- Manyara, A.M., et al., Protocol for the development of SPIRIT and CONSORT extensions for randomised controlled trials with surrogate primary endpoints: SPIRIT-SURROGATE and CONSORT-SURROGATE. BMJ Open, 2022. 12(10): p. e064304.
- Manyara, A.M., et al., Scoping and targeted reviews to support development of SPIRIT and CONSORT extensions for randomised controlled trials with surrogate primary endpoints: protocol. BMJ Open, 2022. 12(10): p. e062798.
- Covidence, Covidence systematic review software, Veritas Health Innovation, Melbourne, Australia.
- Page, M.J., et al., The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ, 2021. 372: p. n71.
Appendix 1: Checklists
Checklist for trial protocols
| A | B | C | |
| Item | Checklist | Completeness check – Adequately, Inadequately or not reported | |
| 1 | SPIRIT 1 [M]: State that primary outcome(s) is considered a surrogate endpoint | ||
| 2 | SPIRIT 2 [M]: State the participant/patient relevant final outcome(s) that the surrogate endpoint is substituting and predicting for | ||
| 3 | SPIRIT 3 [M]: State the practical reason(s) for using a surrogate endpoint as a primary outcome | ||
| 4 | SPIRIT 4 [M]: Justification for selected surrogate: Evidence of validation | ||
| 5 | SPIRIT 5 [M]: Justification for selected surrogate: Evidence of being specific to setting used e.g., intervention, disease, population | ||
| 6 | SPIRIT 6 [M]: Clarify if the sample size calculation is explicitly informed by statistical metrics of surrogate validity (such as the surrogate threshold effect (STE) or its equivalent) | ||
| 7 | SPIRIT 7 [N]: State if trial participants will be informed before enrolment that trial was powered to evaluate an intervention’s effect using a surrogate endpoint (rather than a patient relevant final outcomes) | ||
| 8 | SPIRIT 8 [M]: Comment on whether the trial sample size and follow up period is sufficient to adequately capture potential harms of the intervention being tested | ||
| 9 | SPIRIT 9 [N]: State if there explicit to plans to extend follow up or conduct subsequent analyses/studies to verify benefit of current findings on the patient relevant final outcome |
Checklist for trial reports
| A | B | C | |
| Item | Checklist Items | Completeness check – Adequately, Inadequately or not reported | |
| 1 | State that primary outcome(s) is considered a surrogate endpoint | ||
| 2 | State the participant/patient relevant final outcome(s) that the surrogate endpoint is substituting and predicting for | ||
| 3 | State the practical reason(s) for using a surrogate endpoint as a primary outcome | ||
| 4 | Justification for selected surrogate: Evidence of validation | ||
| 5 | Justification for selected surrogate: Evidence of being specific to setting used e.g., intervention, disease, population | ||
| 6 | Clarify if the sample size calculation was explicitly informed by statistical metrics of surrogate validity (such as the surrogate threshold effect (STE) or its equivalent) | ||
| 7 | State if trial participants were informed before enrolment that trial was powered to evaluate an intervention’s effect using a surrogate endpoint (rather than a patient relevant final outcomes) | ||
| 8 | If the primary outcome is a composite outcome that includes a surrogate endpoint, report the intervention effect on all components | ||
| 9 | Comment on whether the trial sample size and follow up period is sufficient to adequately capture potential harms of the intervention being tested | ||
| 10 | State if there are explicit to plans to extend follow up or conduct subsequent analyses/studies to verify benefit of current findings on the patient relevant final outcome | ||
| 11 | Provide an estimate of the predicted effect based on the observed effect on the surrogate endpoint, and if not possible then a qualitative assessment | ||
| 12 | Interpretation of findings of the trial in the context of using a surrogate primary endpoint including its known validity and the potential benefit-risk ratio of the tested intervention for participants | ||
| 13 | If surrogate endpoint and patient relevant final outcome data were collected in the trial, state the open access arrangements for the data for future secondary research including the statistical evaluation of the surrogate |
Working definition
Working definition
Our working definition of a surrogate endpoint is ‘A biomarker or intermediate outcome used to substitute for a patient or participant relevant final outcome (i.e., severe morbidity; health related quality of life or mortality) and reliably predicts benefit or harm based on epidemiologic; therapeutic; pathophysiologic; or other scientific evidence’. As per this definition, we will consider intermediate outcomes including symptoms and functional outcomes to be surrogate endpoints – see Figure 1 for health outcome framework below.
Figure 1: Health Outcomes Framework showing what we considered a surrogate and final outcome in this review.
Search strategy and study selection
Search strategy and study selection
We will include RCT reports, and protocols of pragmatic and explanatory trials that use a primary outcome which is a surrogate endpoint (including if it’s part of a composite outcome) published in the last five years (2017 to mid-2022) in six high impact general medical journals (Annals of Internal Medicine, BMJ, Journal of the American Medical Association, New England Journal of Medicine, Lancet, and PLoS Medicine) and two journals that commonly publish protocols: BMJ Open and Trials.
Journal names will be searched through PubMed and searches limited to “randomised controlled trial” and publication years (2017 to June 2022).
Identified records will be exported to Covidence [9] where title, abstract and full text screening will be done by two reviewers.
Study selection will be performed independently by two reviewers with a third reviewer providing a decision where there is lack of agreement.
Inclusion criteria
Inclusion criteria
Full RCT protocols or reports including those describing follow up of completed trials.
Primary outcome is a surrogate endpoint (self-reported or considered so by reviewers) including clinical scales that measure both surrogate endpoints (e.g., symptoms and functional outcomes) and final outcomes (i.e., severe morbidity or death). Where the primary outcome is not explicitly stated, but all the outcomes are considered surrogates, such articles will also be included. Furthermore, trial protocols or reports whose primary outcome is a composite that includes a surrogate endpoint will be included. Finally, records with multiple final outcomes that include a surrogate endpoint will be included.
Exclusion criteria
Exclusion criteria
We will exclude the following RCT protocols and reports:
with two primary (co-primary) outcomes that include both a surrogate and a final patient-relevant outcome, e.g., progression free survival (surrogate) and survival (final outcome).
with a primary outcome that is a clinical scale that measures both surrogate endpoints (symptoms and functional outcomes) and final outcomes (severe morbidity or death) is dichotomised to compare severe morbidity versus moderate/no morbidity than we will consider it a final outcome and exclude.
that are feasibility or pilot trials with solely feasibility objectives.
with a primary outcome that is solely safety.
that solely report cost or process evaluation analyses.
reporting a secondary analysis of a trial.
reporting non-health outcomes e.g., intelligence quotient.
Sample size and data extraction
Sample size and data extraction
Based on the total number of included and relevant RCTs (protocols and reports), we will calculate the proportion of these studies that are based on surrogate primary outcome.
From the included RCTs, we will select a random sample of 100 protocols and 100 reports that used a surrogate endpoint as a primary outcome for data extraction and synthesis.
Data from sample of included trials and protocols will be extracted independently by a single reviewer and checked for accuracy by a second reviewer.
We will extract study characteristics such as author, journal, year of publication, RCT hypothesis (superiority/non-inferiority/equivalence), RCT design (parallel, cross over, cluster, and number of arms), study setting (e.g., primary care, community), study country, clinical/research area, type of surrogate (i.e., biomarker or intermediate), intervention type (e.g., drug, screening), comparator (e.g., placebo, usual care), sample size, length of follow-up and funder – see https://osf.io/jr53b.
We also extract data on completeness of reporting (see below).
Assessment for completeness in reporting
Assessment for completeness in reporting
We will use a list of reporting items synthesized from our recent scoping review to analyse for completeness in reporting of trials and protocols.
Items include statement of using a surrogate endpoint, rationale, and justification for using the selected surrogate, use of surrogate treatment effect or equivalent in sample size calculation, interpretation of findings in the context of limitations of surrogate endpoints among others – see full list of items in Appendix 1.
Initially we will do a pilot on the data extraction tool using six random records. Where possible and applicable, text will be extracted from papers to support coding decision.
After the pilot, synthesis for completeness will be done for all records by one reviewer and validated by a second reviewer.
Data synthesis and presentation
Data synthesis and presentation
Given the nature of this review, synthesis of results will primarily be descriptive. Presentation of results will include PRISMA flow-chart [10] of the whole process, counts and percentages summarising study characteristics and completeness of reporting per item, and if need be, tabulation of text to support assessment of completeness judgements.