Guidance for Grading the Evidence in Quantitative Umbrella Reviews

Abstract

Background: Umbrella reviews (URs) synthesize evidence across multiple systematic reviews and meta-analyses to inform decision-making. However, evaluating and integrating evidence certainty across overlapping and sometimes conflicting meta-analyses remains a major methodological challenge of URs, limiting the reliability of conclusions.

Objective: To compare evidence evaluation frameworks for URs, assess their suitability for different study types, and provide guidance for managing overlapping evidence, conflicting certainty ratings, and forming overall conclusions.

Methods: We mapped published UR methods to identify current practices and gaps. Frameworks were compared across three dimensions: applicability to study types (interventional, causal observational, and descriptive observational), scalability and reproducibility, and capacity to handle methodological challenges specific to URs. Recommendations were developed through expert consensus and validated via case studies across diverse domains.

Results: Grading of Recommendations Assessment, Development and Evaluation (GRADE) is widely accepted for interventional evidence but shows limited applicability for observational URs due to subjective judgments and reproducibility issues. Credibility frameworks, based on predefined statistical thresholds, offer greater scalability and reproducibility but require adaptation for different study types. Key challenges include: (1) managing overlapping primary studies, (2) resolving conflicting certainty ratings between high-quality reviews, and (3) ensuring transparent evidence integration.

Conclusions: Framework selection should be tailored to study type and context, with credibility frameworks better suited for observational evidence and GRADE optimal for interventional studies. Systematic approaches to overlapping evidence and conflicting ratings are essential for UR validity. We provide practical recommendations for framework selection, strategies for common challenges, and enhanced reporting standards to improve transparency and reproducibility.

Keywords

credibility assessment / GRADE / umbrella reviews

References

[1] C. Stern, J. Li, J. Stone, et al., “Data Analysis and Presentation Methods in Umbrella Reviews/Overviews of Reviews in Health Care: A Cross-Sectional Study,” Research Synthesis Methods 17, no. 1 (2026): 210–224.

[2] R. Fernandez, A. M. Sharifnia, and H. Khalil, “Umbrella Reviews: A Methodological Guide,” European Journal of Cardiovascular Nursing 24, no. 6 (2025): 996–1002.

[3] S. Sadoyu, K. A. Tanni, N. Punrum, et al., “Methodological Approaches for Assessing Certainty of the Evidence in Umbrella Reviews: A Scoping Review,” PLoS ONE 17, no. 6 (2022): e0269009.

[4] C. J. Gosling, A. Solanes, P. Fusar-Poli, and J. Radua, “Metaumbrella: The First Comprehensive Suite to Perform Data Analysis in Umbrella Reviews With Stratification of the Evidence,” BMJ Mental Health 26, no. 1 (2023): e300534.

[5] J. P. Ioannidis, “Integration of Evidence From Multiple Meta-Analyses: A Primer on Umbrella Reviews, Treatment Networks and Multiple Treatments Meta-Analyses,” Cmaj 181, no. 8 (2009): 488–493.

[6] G. Castellini, M. Bruschettini, S. Gianola, C. Gluud, and L. Moja, “Assessing Imprecision in Cochrane Systematic Reviews: A Comparison of GRADE and Trial Sequential Analysis,” Systematic Review 7, no. 1 (2018): 110.

[7] G. Guyatt, A. D. Oxman, E. A. Akl, et al., “GRADE Guidelines: 1. Introduction—GRADE Evidence Profiles and Summary of Findings Tables,” Journal of Clinical Epidemiology 64, no. 4 (2011): 383–394.

[8] M. Hultcrantz, D. Rind, E. A. Akl, et al., “The GRADE Working Group Clarifies the Construct of Certainty of Evidence,” Journal of Clinical Epidemiology 87 (2017): 4–13.

[9] M. Mueller, M. D'Addario, M. Egger, et al., “Methods to Systematically Review and Meta-Analyse Observational Studies: A Systematic Scoping Review of Recommendations,” BMC Medical Research Methodology 18, no. 1 (2018): 44.

[10] B. J. George and I. B. Aban, “An Application of Meta-Analysis Based on DerSimonian and Laird Method,” Journal of Nuclear Cardiology 23, no. 4 (2016): 690–692.

[11] T. S. Avşar, H. McLeod, and L. Jackson, “Health Outcomes of Smoking During Pregnancy and the Postpartum Period: An Umbrella Review,” BMC Pregnancy Childbirth 21, no. 1 (2021): 254.

[12] A. B. Hill, “The Environment and Disease: Association or Causation?,” Proceedings of the Royal Society of Medicine 58, no. 5 (1965): 295–300.

[13] K. S. Blume, K. Dietermann, U. Kirchner-Heklau, et al., “Staffing Levels and Nursing-Sensitive Patient Outcomes: Umbrella Review and Qualitative Study,” Health Services Research 56, no. 5 (2021): 885–907.

[14] M. Purgato, C. Cadorin, E. Prina, et al., “Umbrella Systematic Review and Meta-Analysis: Physical Activity as an Effective Therapeutic Strategy for Improving Psychosocial Outcomes in Children and Adolescents,” Journal of the American Academy of Child and Adolescent Psychiatry 63, no. 2 (2024): 172–183.

[15] C. Rodríguez-Medina, S. Amaya Sánchez, A. Contreras, and J. E. Botero, “Grading the Strength and Certainty of the Scientific Evidence of the Bidirectional Association Between Periodontitis and Noncommunicable Diseases: An Umbrella Review,” Evidence-Based Dentistry 26, no. 3 (2025): 147.

[16] J. Radua, M. De Prisco, V. Oliva, G. Fico, E. Vieta, and P. Fusar-Poli, “Impact of Air Pollution and Climate Change on Mental Health Outcomes: An Umbrella Review of Global Evidence,” World Psychiatry 23, no. 2 (2024): 244–256.

[17] N. Fabiano, S. Wong, A. Gupta, et al., “Safety of Psychotropic Medications in Pregnancy: An Umbrella Review,” Molecular Psychiatry 30, no. 1 (2025): 327–335.

[18] J. Kang, H. Lee, J. Park, et al., “Comorbid Physical Health Outcomes in Patients With Bipolar Disorder: An Umbrella Review of Systematic Reviews and Meta-Analyses,” Asian Journal of Psychiatry 99 (2024): 104138.

[19] R. DerSimonian and N. Laird, “Meta-Analysis in Clinical Trials Revisited,” Contemporary Clinical Trials 45, no. Pt A (2015): 139–145.

[20] J. IntHout, J. P. Ioannidis, and G. F. Borm, “The Hartung-Knapp-Sidik-Jonkman Method for Random Effects Meta-Analysis Is Straightforward and Considerably Outperforms the Standard DerSimonian-Laird Method,” BMC Medical Research Methodology 14 (2014): 25.

[21] D. Pieper, S.-L. Antoine, T. Mathes, E. A. Neugebauer, and M. Eikermann, “Systematic Review Finds Overlapping Reviews Were Not Mentioned in Every Other Overview,” Journal of Clinical Epidemiology 67, no. 4 (2014): 368–375.

[22] K. I. Bougioukas, E. Bouras, F. Apostolidou-Kiouti, S. Kokkali, M. Arvanitidou, and A.-B. Haidich, “Reporting Guidelines on How to Write a Complete and Transparent Abstract for Overviews of Systematic Reviews of Health Care Interventions,” Journal of Clinical Epidemiology 106 (2019): 70–79.

[23] A. R. Jadad, D. J. Cook, and G. P. Browman, “A Guide to Interpreting Discordant Systematic Reviews,” Cmaj 156, no. 10 (1997): 1411–1416.

[24] M. H. Murad, V. M. Montori, J. P. Ioannidis, et al., “How to Read a Systematic Review and Meta-Analysis and Apply the Results to Patient Care: Users' Guides to the Medical Literature,” Jama 312, no. 2 (2014): 171–179.

[25] E. Aromataris, R. Fernandez, C. M. Godfrey, C. Holly, H. Khalil, and P. Tungpunkom, “Summarizing Systematic Reviews: Methodological Development, Conduct and Reporting of an Umbrella Review Approach,” International Journal of Evidence-Based Healthcare 13, no. 3 (2015): 132–140.

[26] G. J. Choi and H. Kang, “Introduction to Umbrella Reviews as a Useful Evidence-Based Practice,” Journal of Lipid and Atherosclerosis 12, no. 1 (2023): 3–11.

Rights & permissions

2026 The Author(s). Journal of Evidence-Based Medicine published by Chinese Cochrane Center, West China Hospital of Sichuan University and John Wiley & Sons Australia, Ltd.