Sep 29, 2023
Holistic Review in Graduate Medical Education: Scoping Review Protocol
- 1Georgetown University School of Medicine;
- 2Dahlgren Memorial Library;
- 3Georgetown University Medical Center;
- 4MedStar Georgetown University Hospital
Protocol Citation: Hassan Bazzi, Charles S. Dorris MLIS, Colin E Stewart 2023. Holistic Review in Graduate Medical Education: Scoping Review Protocol. protocols.io https://dx.doi.org/10.17504/protocols.io.e6nvwdbe7lmk/v1
License: This is an open access protocol distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Protocol status: Working
We are using this protocol and the review progress is on-going
Created: September 14, 2023
Last Modified: September 29, 2023
Protocol Integer ID: 87786
Keywords: Holistic review, graduate medical education, recruitment
Funders Acknowledgement:
Abstract
Introduction:
Holistic review (HR) is a systematic method of assessing trainees beyond the usual metrics of grades and exam scores that facilitates increased diversity, better mission alignment, structured consideration of the applicant in context, and is based on sound educational and selection policy and practice. Graduate Medical Education (GME) programs face rising application inflation, the change of the USMLE Step 1 Exam to Pass/Fail, and fewer schools offering tiered grades along with increasing demands for a more diverse, inclusive, and equitable learning environment all in the context of few resources. However, very little data exists about the use of holistic principles in GME recruitment, in general, and how to use them successfully in the very different GME recruitment process to reach mission-based institutional goals.
Review Question:
What is the existing state of the published literature on the use of the holistic review in graduate medical education recruitment, including existing best practices and challenges, and what gaps exist in the literature that could be filled by future research?
Methods:
Following PRISMA's reporting guidelines and other scoping review best practices, we plan to search MEDLINE, Embase, APA PsycInfo, and Web of Science for English language, peer-reviewed studies. Eligibility screening (title/abstract and full-text) and data extraction will be completed by at least two independent reviewers and conflicts will be resolved as a team. The draft data extraction tool will be modified and revised as necessary during the process of extracting data from each included article.
Introduction
Introduction
Holistic review (HR) is a systematic method of assessing trainees beyond the usual metrics of grades and exam scores that facilitates increased diversity, better mission alignment, structured consideration of the applicant in context, and is based on sound educational and selection policy and practice. Graduate Medical Education (GME) programs face rising application inflation, the change of the USMLE Step 1 Exam to Pass/Fail, and fewer schools offering tiered grades along with increasing demands for a more diverse, inclusive, and equitable learning environment all in the context of few resources. Utilization of holistic principles in medical school admissions has demonstrated the ability to increase the percentage of students from groups underrepresented in medicine (URM) in the applicant pool and among matriculating students (Grabowski 2018) (Grbic et al. 2019). However, very little data exists about the use of holistic principles in GME recruitment, in general, and how to use them successfully in the very different GME recruitment process to reach mission-based institutional goals. This scoping literature review is part of a larger project that seeks to close those gaps.
The intended outcomes of performing a scoping literature review on HR in graduate medical education are to assess the current state of the published literature on the use of holistic principles in GME recruitment, identify and disseminate existing best practices, identify gaps in the existing literature, and outline a research agenda for the field based on those gaps.
Previous studies (e.g. Barcelo 2021) and reviews (e.g. Mabeza 2023) have explored how utilizing holistic review has impacted diversity within specific GME programs and how a variety of different recruitment strategies have impacted diversity but there has not been a review that specifically examines the overall literature on HR in GME, identifies best practices in the utilization of HR across graduate medical education, and proposes a research agenda for the field to fill existing holes in the literature.
A preliminary search of Ovid MEDLINE, the Cochrane Database of Systematic Reviews, the JBI EBP Database, and Open Science Framework was conducted and no current or underway systematic reviews or scoping reviews on the topic were identified.
Review Question
Review Question
What is the existing state of the published literature on the use of the holistic review in graduate medical education recruitment, including existing best practices and challenges, and what gaps exist in the literature that could be filled by future research?
Eligibility Criteria
Eligibility Criteria
Participants
All graduate medical education trainees, including interns, residents, and fellows.
Concept
Holistic review (HR) is a beyond-the-numbers method of assessing trainees that has been demonstrated to help medical schools recruit more mission-aligned students and to increase the percentage of URM students in Undergraduate Medical Education (UME).
GME programs face a very different recruitment landscape than medical schools, including unique challenges and opportunities. GME programs have tried a wide variety of strategies for overcoming those challenges and taking advantage of those opportunities, including holistic review, which is widely used at the UME level. However, there has yet to be a comprehensive review of the literature examining the use of holistic review as an organizing framework in GME recruitment.
This review aims to assess the state of the literature on the use of holistic principles in GME recruitment, identify and disseminate existing best practices, identify gaps in the existing literature, and outline a research agenda for the field based on those gaps.
Context
The use of holistic review in graduate medical education around the world will be studied without limitations on country or level within GME. The review will examine holistic review irrespective of its intended outcome. The review will not examine the published literature on holistic review in other professions, such as law, nursing, dentistry, etc.
Types of Sources
This review will consider all study designs which include quantitative, qualitative, and mixed methods. Observational studies (prospective and retrospective cohort studies), case-control studies and analytical cross-sectional studies will be considered for inclusion. Conference abstracts will be included if they meet eligibility criteria and we will attempt to contact authors for additional information as needed. The references within any reviews that are retrieved from the search will be mined and screened for inclusion.
Methods
Methods
The proposed scoping review will be conducted in accordance with the framework outlined by Arksey and O'Malley (Arksey et al., 2005) and further refined by Levac et al. (Levac et al., 2010) and we plan to follow the reporting guidelines set forth in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) (Tricco et al., 2018).
Search strategy
The search strategy will aim to locate both published and unpublished studies. An initial limited search of MEDLINE was undertaken to identify exemplar articles on the topic. The text words contained in the titles, abstracts, and author-supplied keywords of relevant articles, and the Medical Subject Headings (MeSH) used to describe the articles were used to develop a full search strategy for MEDLINE via Ovid (see Appendix #1). The search strategy, including all identified keywords and Subject Headings, will be adapted for each included database and/or information source. The reference list of all included sources of evidence will be screened for additional studies.
Studies published in any language and within all year ranges will be included.
The databases to be searched include MEDLINE, Embase, APA PsycInfo, ERIC (all via Ovid), and Web of Science Core Collection. Sources of grey literature to be searched include MedEdPORTAL, Association of American Medical Colleges (AAMC), and other medical education organizations and associations.
Study/Source of Evidence Selection
Following the searches, all identified citations will be imported into EndNote 20 (Clarivate Analytics, PA, USA) and duplicates removed. The resulting citations will be uploaded to Rayyan for screening (Ouzzani, 2016). Following a pilot test, titles and abstracts will then be screened by two independent reviewers for assessment against the eligibility criteria for the review. The full text of selected citations will be assessed in detail against the eligibility criteria by two or more independent reviewers. Reasons for exclusion of sources of evidence at full text that do not meet the inclusion criteria will be recorded and reported in the PRISMA Flow Diagram. Any disagreements that arise between the reviewers at each stage of the selection process will be resolved through discussion. The results of the search and the study inclusion process will be reported in full in the final scoping review and presented in the PRISMA Flow Diagram.
Data Extraction
Data will be extracted from papers included in the scoping review by two or more independent reviewers using a data extraction tool developed by the reviewers. The data extracted will include bibliographic details about each article (author, year of publication, publication type, study design etc.) in addition to specific information regarding the participants (interns, residents, fellows, etc.), concept (how holistic review is incorporated into the recruitment process, challenges, outcomes, etc.), context (specialty information, geographic location, etc.), and key findings relevant to the review question.
The draft data extraction tool will be modified and revised as necessary during the process of extracting data from each included article. Any modifications will be detailed in the scoping review. Any disagreements that arise between the reviewers will be resolved through discussion as a team. If appropriate, authors of papers will be contacted to request missing or additional data, where required.
Appendix
Appendix
Ovid MEDLINE Search Strategy:
(holistic* OR whole applicant OR whole person OR whole individual).ti,ab,kf. OR ((mission? OR value?) ADJ2 (align* OR match* OR based OR center* OR harmoni*)).ti,ab,kf. OR (bias* ADJ1 mitigat*).ti,ab,kf.
AND
(review* OR recruit* OR select* OR apply OR applies OR application* OR applicant* OR assess* OR interview*).ti,ab,kf. OR Personnel Selection/ OR Interviews as Topic/
AND
(graduate medical education OR residency OR resident OR residents OR intern OR interns OR fellow*).ti,ab,kf. OR "Internship and Residency"/ OR Education, Medical, Graduate/
AND NOT (Long Term Care/ OR Nursing Homes/ OR exp Housing/ OR Residential Facilities/)
Protocol references
Arksey, H., & O’Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19–32.
Barceló, N. E., Shadravan, S., Wells, C. R., Goodsmith, N., Tarrant, B., Shaddox, T., ... & DeBonis, K. (2021). Reimagining merit and representation: promoting equity and reducing bias in GME through holistic review. Academic Psychiatry, 45, 34-42.
Grabowski C. J. (2018). Impact of holistic review on student interview pool diversity. Advances in health sciences education: theory and practice, 23(3), 487–498.
Grbic, D., Morrison, E., Sondheimer, H. M., Conrad, S. S., & Milem, J. F. (2019). The Association Between a Holistic Review in Admissions Workshop and the Diversity of Accepted Applicants and Students Matriculating to Medical School. Academic medicine, 94(3), 396–403.
Levac, D., Colquhoun, H., & O’Brien, K. K. (2010). Scoping studies: Advancing the methodology. Implementation Science, 5(1), 69.
Mabeza, R. M., Christophers, B., Ederaine, S. A., Glenn, E. J., Benton-Slocum, Z. P., & Marcelin, J. R. (2023). Interventions Associated With Racial and Ethnic Diversity in US Graduate Medical Education: A Scoping Review. JAMA network open, 6(1), e2249335.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016). Rayyan-a web and mobile app for systematic reviews. Systematic reviews, 5(1), 210.
Tricco, A. C., Lillie, E., Zarin, W., O'Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D. J., Horsley, T., Weeks, L., Hempel, S., Akl, E. A., Chang, C., McGowan, J., Stewart, L., Hartling, L., Aldcroft, A., Wilson, M. G., Garritty, C., Lewin, S., … Straus, S. E. (2018). PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Annals of internal medicine, 169(7), 467–473.