Apr 24, 2023
Rigour of Development of European Society of Cardiology, American College of Cardiology and American Heart Association guidelines over a 10-year period (2013-2022): A systematic review of guidelines.
- 1University College London
Protocol Citation: r.providencia 2023. Rigour of Development of European Society of Cardiology, American College of Cardiology and American Heart Association guidelines over a 10-year period (2013-2022): A systematic review of guidelines.. protocols.io https://dx.doi.org/10.17504/protocols.io.j8nlkwq6xl5r/v1
License: This is an open access protocol distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Protocol status: Working
We use this protocol and it's working
Created: April 24, 2023
Last Modified: April 24, 2023
Protocol Integer ID: 80973
Abstract
This protocol describes in the detail the processes for a systematic review of cardiovascular guidelines (ACC/AHA/ESC) published over a 10 year time period (2013 to 2022)
Systematic Review ID and Affiliation
Systematic Review ID and Affiliation
Review title
Rigour of Development of European Society of Cardiology, American College of Cardiology and American Heart Association guidelines over a 10-year period (2013-2022): A systematic review of guidelines.
Original language title
Not applicable
Anticipated or actual start date
19 April 2023
Anticipated completion date
30 April 2023
Stage of review at the time of this submission
The committee is reviewing the protocol.
Named contact
Prof Rui Providencia
Named contact email
Named contact address
Institute of Health Informatics Research
University College London
222 Euston Road
London NW1 2DA
United Kingdom
Named contact phone number
+44 020 7679 0000
Organisational affiliation of the review
University College London
Organisation web address:https://www.ucl.ac.uk/health-informatics/ucl-institute-health-informatics
Review team members and their organisational affiliations
- Miss Sanjali Ahuja, Barts and The London School of Medicine and Dentistry, Turner St, London E1 2AD;sanjaliacahuja@gmail.com
- Miss Yi Ting YU, Barts and The London School of Medicine and Dentistry, Turner St, London E1 2AD; nikki.yu04@gmail.com
- Mr Robert English, Barts and The London School of Medicine and Dentistry, Turner St, London E1 2AD; robbie.english@hotmail.co.uk
- Prof Pedro Adragão, Hospital de Santa Cruz, Centro Hospitalar Lisboa Ocidental, Portugal;padragao@gmail.com
- Jonathan Bray, Institute of Life Sciences-2, Swansea Bay University Health Board and Swansea University Medical School, Swansea University, Swansea, UK;jonathanjhbray@gmail.com
- Dr Ferrucio Pelone, Institute of Health Informatics, University College London, London, UK; fpelone@gmail.com
- Dr Mahmood Ahmad, Royal Free London NHS Foundation Trust, United Kingdom; mahmood.ahmad2@nhs.net
- Dr Farhad Shokraneh, Systematic Review Consultants, 9 Sandfield Road, NG7 1QR, Nottingham, United Kingdom;farhad@systematicreview.info
- Dr Mohammed Khanji, Barts Health NHS Trust, London, UK;m.khanji@qmul.ac.uk
- Professor Rui Providencia, Institute of Health Informatics, University College London, London, UK; r.providencia@ucl.ac.uk
Funding sources/sponsors
None
Conflicts of interest
None.
Collaborators
Not applicable.
Protocol Details
Protocol Details
Review question
Are Cardiovascular guidelines developed by the main scientific cardiovascular societies in Europe and America Developed with High Rigour? What is the level of evidence underlying the different recommendations and what is the overall recommendations’ strength?
Searches
We will search for documents classified as guidelines by the European Society of Cardiology (ESC), American Heart Association (AHA) and American College of Cardiology (ACC)
From January 2013 to December 2022
The following official libraries will be used:
Library of Guidelines and Clinical Documents, American College of Cardiology, available athttps://www.acc.org/guidelines
Professional Heart Daily, Guidelines and Statements, American Heart Association, available athttps://professional.heart.org/en/guidelines-and-statements
Clinical Practice Guidelines, European Society of Cardiology, available athttps://www.escardio.org/Guidelines/Clinical-Practice-Guidelines
We performed manual searches using the terms “guidelines”, “ACC”, “AHA” and/or “ESC” to look for potential titles not on these libraries. We checked publications in the field for additional titles. Also, we contacted the ESC, AHA and ACC to confirm whether all currently active guidelines were available on the abovementioned websites.
Records will be manually searched.
Duplicate entries (same guideline published by different cardiovascular societies) will be removed. Similarly, only the most up-to-date version for each title will be included.
The searches will be conducted by three authors (SA, YTY & RE) and will be validated by the Senior author (RP).
A PRISMA chart will be created to describe the search process step by step.
We will search each title on PubMed to determine how many versions exist, and how many journals each guideline was published in. Searches will be conducted on google scholar and ISI Web of Science for determining the number of citations for each guideline.
URL to the search strategy
Not applicable.
Condition or domain being studied
We will be studying all guidelines focusing on cardiovascular conditions developed and published by the ESC, ACC and AHA over a 10-year time interval.
The ESC, ACC and AHA guidelines are considered to be amongst the most important cardiovascular disease guidelines in medicine (Tantawy 2023, Tricoci 2009, Faranoff 2019). These have been published since the 80s (Frye 1984) and 90s (Pyörälä 1994), and have become so popular that they are used by physicians around the world (Tantawy 2023, Tricoci 2009, Faranoff 2019).
Guidelines should have a clearly defined methodology, and by definition should be informed by systematic reviews of evidence with assessment of the benefits and harm of the available treatment and management options (6). This seems to be practice for governmental or intergovernmental organizations like the National Institute for Health and Care Excellence (WHO 2014) in the UK, or the World Health Organization (NICE 2022) with publicly available and published methodology (WHO 2014, NICE 2022).
Guidelines are composed of recommendations. For each recommendation there is a Class of Recommendation assigned by the Guideline writing committee, and a Level of Evidence (the existing clinical data, if any, supporting the recommendation). The classification used by the ESC, ACC and AHA overlaps, with four classes of recommendation and three different levels of evidence (Ponikowski 2016):Class I: conditions for which there is evidence and/or general agreement that a given procedure or treatment is useful and effective.Class II: conditions for which there is conflicting evidence and/or a divergence of opinion about the usefulness/efficacy of a procedure or treatment.Class IIa: weight of evidence/opinion is in favor of usefulness/efficacy.Class IIb: usefulness/efficacy is less well established by evidence/opinion.Class III: conditions for which there is evidence and/or general agreement that the procedure/treatment is not useful/effective and in some cases may be harmful.Level A:recommendation based on evidence from multiple randomized trials or meta-analyses.Level B:recommendation based on evidence from a single randomized trial or non-randomized studies.Level C:recommendation based on expert opinion, case studies, or standards of care.
Recent ACC/AHA guidelines (Isselbacher 2022) added extra details and sub-divisions to Class or recommendation (Class I – Strong; Class IIa – Moderate; Class IIb – Weak; Class III - No Benefit/Moderate; Class III – Harm/Strong) and Level of Evidence (Level B – Randomized; Level B – Nonrandomized; Level C-LD- limited data; Level C-EO – expert opinion).
References
1. Tantawy M, Marwan M, Hussien S, Tamara A, Mosaad S. The scale of scientific evidence behind the current ESC clinical guidelines. IJC Heart & Vasculature. 2023;45:101175.
2. Tricoci P, Allen JM, Kramer JM. Scientific Evidence Underlying the ACC/AHA Clinical Practice Guidelines. JAMA. 2009;301:831-841.
3. Fanaroff AC, Califf RM, Windecker S, Smith SC, Lopes RD. Levels of evidence supporting American College of Cardiology/American Heart Association and European Society of Cardiology Guidelines, 2008–2018. JAMA. 2019;321:1069–1080.
4. Frye RL, Collins JJ, DeSanctis RW, Dodge HT, Dreifus LS, Fisch C, Geths LS, Gillette PC, Parsonnet V, Reeves J, et al. Guidelines for permanent cardiac pacemaker implantation, May 1984: a report of the Joint American College of Cardiology/American Heart Association Task Force on Assessment of Cardiovascular Procedures (Subcommittee on Pacemaker Implantation). Circulation. 1984;70:331A-339A.
5. Pyörälä K, De Backer G, Graham I, Poole-Wilson P, Wood D. Prevention of coronary heart disease in clinical practice. Recommendations of the Task Force of the European Society of Cardiology, European Atherosclerosis Society and European Society of Hypertension. Eur Heart J. 1994;15:1300-31.
6. Graham R, Mancher M, Miller Wolman D, Greenfield S, Steinberg E. Clinical practice guidelines we can trust. Washington: National Academies Press; 2011. Available at:https://www.uspreventiveservicestaskforce.org/uspstf/sites/default/files/inline-files/Clinical%20Practice%20Guidelines%202011%20Insert_0.pdf, last accessed 16thApril 2023
7. World Health Organization. WHO handbook for guideline development. 2014; 2nd ed. World Health Organization. https://apps.who.int/iris/handle/10665/145714, last accessed 15thApril 2023
8. NICE. Process and methods [PMG20]. Developing NICE guidelines: the manual. Published: 31 October 2014 Last updated: 18 January 2022available at:https://www.nice.org.uk/process/pmg20/resources/developing-nice-guidelines-the-manual-pdf-72286708700869last accessed, 15thApril 2023
9. Ponikowski P, Voors AA, Anker SD, Bueno H, Cleland JG, Coats AJ, Falk V, González-Juanatey JR, Harjola VP, Jankowska EA, Jessup M, Linde C, Nihoyannopoulos P, Parissis JT, Pieske B, Riley JP, Rosano GMC, Ruilope LM, Ruschitzka F, Rutten FH, van der Meer P; ESC Scientific Document Group. 2016 ESC Guidelines for the diagnosis and treatment of acute and chronic heart failure: The Task Force for the diagnosis and treatment of acute and chronic heart failure of the European Society of Cardiology (ESC)Developed with the special contribution of the Heart Failure Association (HFA) of the ESC. Eur Heart J. 2016;37:2129–200.
10. Isselbacher EM, Preventza O, Hamilton Black 3rdJ, Augoustides JG, Beck AW, Bolen MA, Braverman AC, BrayBE , Brown-Zimmerman MM, Chen EP, Collins TJ, DeAnda Jr A, Fanola CL, Girardi LN, Hicks CW, Hui DS, Schuyler Jones W, Kalahasti V, Kim KM, Milewicz DM, Oderich GS, Ogbechie L, Promes SB, Gyang Ross E, SchermerhornML, Singleton Times S, Tseng EE, Wang GJ, Woo UJ. 2022 ACC/AHA Guideline for the Diagnosis and Management of Aortic Disease: A Report of the American Heart Association/American College of Cardiology Joint Committee on Clinical Practice Guidelines. Circulation. 2022;146:e334-e482.
Participants/population
Patients with or at risk of cardiovascular conditions for whom management and treatment guidelines have been developed by the ESC, AHA and ACC.
Intervention(s), exposure(s)
All interventions and or recommendations assessed in the guidelines.
Comparator(s)/control
Not applicable
Types of study to be included
Guideline Documents developed by the ESC, ACC and AHA. Publication date was specified as between 2013 and 2022. Any documents including information and recommendations regarding diagnosis, work-up, management, treatment or follow-up of cardiovascular conditions were considered potentially eligible. Furthermore, documents had to be endorsed by ESC, ACC and AHA as guidelines, and accessible in the public domain.
Only comprehensive guideline documents that included recommendations organized by class of recommendation and level of evidence, separated and highlighted from the rest of the text, were included for this systematic review. Expert consensus documents, performance measures, and appropriateness criteria were not included because as they do not report level of evidence. Due to lack of representativeness of the evidence base for an entire topic focused updates were not included.
When the same guideline developer published different formats (summary, short or full versions), or journals, we included the most recent full version published in the scientific society highest impact factor journal. Previous versions of each guideline, that have been replaced more recent publications, will be disregarded.
Context
A well-defined, transparent and reproducible search strategy, along with grey literature searches and selection criteria to include evidence should constitute the basis to develop evidence-based clinical guidelines (Brouwers 2016). From a methodological perspective, and as mentioned above, a clinical practice guideline needs to provide clear proof that a systematic search strategy and other systematic review steps are performed. When those criterion are not met, the document should be referred to as a “guidance document”, rather than a guideline (NGC and NQMC Inclusion Criteria 2018).
The AGREE Collaboration (Appraisal of Guidelines, Research and Evaluation) is an international group of researchers and guideline developers who have jointly created an instrument for assessing guideline development and quality. The AGREE II instrument comprises 23 items divided by 6 domains (Brouwers et al): I - Scope and purpose, II – Stakeholder involvement, III – Rigour of development, IV – Clarity of presentation, V – Applicability, and VI – Editorial independence (Brouwers 2010).
The Rigour of development domain is composed of 8 items, numbered from 7 to 14: 7. Systematic methods were used to search for evidence; 8. The criteria for selecting the evidence are clearly described; 9. The strengths and limitations of the body of evidence are clearly described; 10. The methods for formulating the recommendations are clearly described;11. The health benefits, side effects, and risks have been considered in formulating the recommendations; 12. There is an explicit link between the recommendations and the supporting evidence; 13. The guideline has been externally reviewed by experts prior to its publication; 14. A procedure for updating the guideline is provided. These are scored from 1 - Strongly disagree to 7 – Strongly agree, based on specific rules provided in the manual (Appendix - Brouwers 2010).
In areas like oncology, only a small portion of guidelines have been informed by scientific evidence identified through adequate systematic search methods (Trevisiol 2019). Over 20 years ago, an overview showed that nearly 90% of guidelines did not report any information on the systematic search, with a minor increase observed over the 90s (from 2% to 18%) (Grilli 2000). Quality of practice guidelines is therefore a matter of concern, as lack of rigorous criteria can undermine guideline credibility and lead to harm to the patient if the wrong recommendations are put into practice (Grilli 2000).
Whether or not cardiovascular clinical practice guidelines developed by the ESC, ACC and AHA meet high standards of rigour when assessed through formal methodology criteria remains to be determined.
References
1. Brouwers MC, Kerkvliet K, Spithoff K; AGREE Next Steps Consortium. The AGREE Reporting Checklist: a tool to improve reporting of clinical practice guidelines. BMJ. 2016;352:i1152
2. NGC and NQMC Inclusion Criteria. Content last reviewed October 2018. Agency for Healthcare Research and Quality, Rockville, MD. Available at:https://www.ahrq.gov/gam/summaries/inclusion-criteria/index.html#ref1, last accessed 15thApril 2023
3. Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, Fervers B, Graham ID, Grimshaw J, Hanna SE, Littlejohns P, Makarski J, Zitzelsberger L; AGREE Next Steps Consortium
et al. AGREE II: advancing guideline development, reporting and evaluation in health care. CMAJ. 2010;18:182.
4. Trevisiol C, Cinquini M, Fabricio ASC, Gion M, Rutjes AWS. Insufficient uptake of systematic search methods in oncological clinical practice guideline: a systematic review. BMC Medical Research Methodology. 2019;19:180.
5. Grilli R, Magrini N, Penna A, Mura G, Liberati A. Practice guidelines developed by specialty societies: the need for a critical appraisal. Lancet. 2000;355:103-6.
Main outcome(s)
Primary outcomes:
•Number and proportion of recommendations with class of recommendation I or III
•Number and proportion of recommendations with level of evidence A
•Presence of systematic review methodology
Secondary outcomes:
•Number of recommendations with class of recommendation IIa or IIb
•Number of recommendations with level of evidence B or C
•Presence of other type of simplified methodology
•Popularity measured by the number of citations on ISI Web of Science and Google Scholar (total, adjusted by number of years since publication)
Additional outcome(s)
If additional sub-divisions of levels of evidence or classes of recommendations are identified, further sub-analyses will be conducted.
Data extraction (selection and coding)
The search results will be imported into EndNote 20 and the duplicates will be removed. SA and RE will identify eligible papers. Full-agreement for inclusion is required. When agreement is not reached, a third examiner (RP) will assess the paper blind to the previous assessments and will make the final decision resolving the conflict.
The followingdata will be extracted from all studies and double-checked by an independent reviewer.
·Guideline characteristics: authors, year of publication, title / cardiovascular condition of interest, developing scientific cardiovascular societies.
·Guideline Methodology: systematic methods for evidence search, selecting evidence, assessment on strengths and limitations of the body of evidence, methods for formulating the recommendations, consideration of health benefits, side effects, and risks when formulating the recommendations, existence of a link between the recommendations and the supporting evidence, information on externally review by experts prior to publication, and information on procedures for updating the guideline.
- Outcomes: Class of recommendation I, IIa, IIb, and III. Level of evidence A, B and C.Presence of systematic review methodology (yes or no). Presence of any other reported simplified methodology.
For assessing degree of traceability of published evidence in the produced recommendations, we will follow the model previously suggested (Treviosol 2019).
The following three criteria are considered essential for a systematic strategy to be considered sufficient:
a)Explicit reporting of the use of at least one biomedical database (e.g. MEDLINE);
b)Use of at least one additional source to retrieve citations, such as another biomedical database, screening of reference lists of included reports, personal communication with investigators or organizations;
c)Explicit description of the search terms used in the bibliographic search(es)
Assessment will be conducted following the fluxogram below:
-Q1-Did the guideline authors explicitly mention that the preparation of the document was based on the assessment of the literature?
-A) No. Classified as: without explicit description of the use of literature analysis (noLA)
-B) Yes – Move to question 2
-Q2- Did the authors declare that the literature analysis was performed through a systematic search?
-A) No. Classified as: with literature analysis but without reference to the use of systematic search (noSR)
-B) Yes – Move to question 3
-Q3- Did the authors report details on systematic search methods?
-A) No. Classified as: described to be based on one or more systematic reviews, not providing details to demonstrate it (SRnoDT)
-B) Yes – Move to question 4
-Q4- Did the methods meet the predefined quality criteria?
-A) Search methods insufficient. Classified as: based on one or more systematic reviews, with a search strategy not meeting our quality criteria (SRinsDT).
-B) Systematic search performed. Classified as: based on one or more systematic reviews, with a search strategy meeting our quality criteria (SR)
Literature analyses will therefore be classified as noLA, noSR, SRnoDT, SRinsDT and SR.
Number of citations on ISI Web of Science and Google Scholar (total, adjusted by number of years since publication); for documents with multiple versions (summary and full/extended version) and publication in multiple journals, we will sum all available results. Only the most up-to-date version will be used.
We will use Microsoft Excel for data management.
References
1. Trevisiol C, Cinquini M, Fabricio ASC, Gion M, Rutjes AWS. Insufficient uptake of systematic search methods in oncological clinical practice guideline: a systematic review. BMC Medical Research Methodology. 2019;19:180.
Risk of bias (quality) assessment
Rigour of guideline development will be assessed using the 8-items from the 3rdDomain of AGREE-II (Brouwers 2010):
7. Systematic methods were used to search for evidence.
8. The criteria for selecting the evidence are clearly described.
9. The strengths and limitations of the body of evidence are clearlydescribed.
10. The methods for formulating the recommendations are clearly described.
11. The health benefits, side effects, and risks have been considered in formulating the recommendations.
12. There is an explicit link between the recommendations and the supporting evidence.
13. The guideline has been externally reviewed by experts prior to its publication.
14. A procedure for updating the guideline is provided.
These will be assessed independently and blindly by two reviewers (RP and MA). Agreement for final judgement is required. When this is not reached, a third reviewer (MK) will assess the paper blind to the previous assessments and will assess the manuscript. If the third reviewer classification matches one of the other two reviewers (RP and MA), that will be considered the final score. However, if no agreement is observed between the 3 reviewers, the average of the 3 classifications will be used.
In some of the published guidelines, the development process may be in the supplemental material/ societal website, rather than the main manuscript. The authors will look for that additional information whenever it cannot be found on the guideline.
Whenever possible, the Agree Plus platform will be used.
Prior to starting work on this section, MK, RP and MA will do a practice run using 5 guidelines not meeting inclusion criteria for this review. MK has experience in this type of assessment and will lead on discussion regarding discrepancies and how to apply the AGREE II. This is an approach for reducing inter-observer variation due to different interpretation of the same domain sections.
References
1. Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, Fervers B, Graham ID, Grimshaw J, Hanna SE, Littlejohns P, Makarski J, Zitzelsberger L; AGREE Next Steps Consortium
et al. AGREE II: advancing guideline development, reporting and evaluation in health care. CMAJ. 2010;18:182.
2. Agree. Agree Tools: My Agree Plus. Available at:https://www.agreetrust.org/resource-centre/agree-plus/last accessed 20thApril 2023
Strategy for data synthesis
a)Rate of Class I, IIa, IIb and III recommendations, and Level of evidence A, B and C for ESC and ACC/AHA guidelines combined.
b)AGREE-II – domain 3 scores (average of 8 items, and 8 items individually):
-b.1) For ESC, ACC/AHA combined
-b.2) ACC/ACC vs. ESC
c)Comparison of:
- c.1) total rates for ESC vs ACC/AHA
- c.2) ESC vs. ACC/AHA guidelines for cardiovascular disease with different guidelines produced by the medical societies.
d)Time trends across the 10 years for d-1) the 3 medical societies combined and d-2) ESC vs. ACC/AHA (correlation and graphic representation).
Chi-square will be used for comparing nominal variables and Student’s t-test for comparison of continuous variables, where appropriate; the Levene’s test was used in order to check the homogeneity of variance; equivalent non-parametric tests were used when Kolmogorov-Smirnov suggests absence of normal distribution. Results with p < 0.05 will be regarded as significant.
Regression analysis will be utilized for assessing the association between year of publication and rates of Class I, IIa, IIb and III, Level of evidence A, B & C, & items in Domain III of AGREE II. Correlation coefficient will be determined (between -1 and 1).
Graphics displays will be used for illustrating time-trends and group comparisons when required.
Analysis of subgroups or subsets
Depending on the availability of the data, and Guideline documents, we will consider running analyses 28-a) and 28-b.1) for the following subgroup analyses:
- Cardiovascular subspecialities (Fanaroff 2019)
oArrhythmias and Electrophysiology
oCoronary artery disease
oHeart failure or myocardial disease
oHeart valve disease or congenital heart disease
oPrevention
oGeneral cardiology
oVascular Medicine
- By date
oPublished in the first 5 years (2013, 2014, 2015, 2016 & 2017)
oPublished in the final 5 years (2018, 2019, 2020, 2021 & 2022)
References:
1. Fanaroff AC, Califf RM, Windecker S, Smith SC, Lopes RD. Levels of evidence supporting American College of Cardiology/American Heart Association and European Society of Cardiology Guidelines, 2008–2018. JAMA. 2019;321:1069–1080.
Type and method of review
Select the type of review, review method and health area from the lists below.
Type of review
•Diagnostic
•Meta-analysis
•Systematic review
Health area of the review
•Cardiovascular
Language
English
Country
United Kingdom, Italy and Portugal
Reference and/or URL for published protocol
Not applicable.
Dissemination plans
Yes, the review will be published in a peer-reviewed and MEDLINE-indexed journal.
Keywords
Guidelines; Methodology; cardiovascular; medicine; European Society of Cardiology; American College of Cardiology; American Heart Association; Quality; Rigour; Systematic; Search.
Details of any existing review of the same topic by the same authors
Not applicable.
Current review status
Ongoing
Any additional information
Not applicable.