Examining and Interpreting Doi Plot Asymmetry in Meta-Analyses of Randomized Controlled Trials

Abstract

Systematic reviews and meta-analyses are considered the highest level of evidence, but their reliability can be undermined by publication bias. Traditional methods for assessing publication bias, such as funnel plots and p-value-based tests (e.g., Egger test), have notable limitations, including reliance on subjective interpretation and dependence on the number of studies included in a meta-analysis (k). The Doi plot and LFK index offer promising alternatives, providing improved visualization and quantification of plot asymmetry. This study revisits the application of the Doi plot and LFK index for detecting publication bias, addresses recent criticisms, and evaluates their performance compared to p-value-based methods using simulation study. Simulations included scenarios with varying study numbers (k = 5, 10, 20, 50), study sample sizes (small, large), and simulated bias level (ρ = 0, –0.3, –0.5, –0.9) generated using the Copas selection model. Diagnostic performance metrics (i.e., sensitivity and specificity) were estimated and compared for the LFK index and Egger test. The LFK index exhibited consistent higher sensitivity across varying k and simulated bias levels. In contrast, the Egger test was highly dependent on k, with sensitivity declining sharply in small meta-analyses (k < 20). Specificity of the LFK index adjusted with random error, while Egger test specificity remained fixed at ∼90%. The Doi plot and LFK index effectively address the limitations of traditional methods, offering robust k-independent performance and more reliable detection of publication bias. These findings support a transition to the Doi plot and LFK index for publication bias assessment in meta-analyses.

Keywords

Doi plot / Eggers p / funnel plot / LFK index / meta-analysis / publication bias

References

[1] P. B. Burns, R. J. Rohrich, and K. C. Chung, “The Levels of Evidence and Their Role in Evidence-Based Medicine,” Plastic and Reconstructive Surgery 128, no. 1 (2011): 305-310.

[2] A. J. Sutton, “Publication bias,” The Handbook of Research Synthesis and Meta-Analysis. 2nd ed. (New York, NY, US: Russell Sage Foundation, 2009): 435-452.

[3] L. Lin and H. Chu, “Quantifying Publication Bias in Meta-Analysis,” Biometrics 74, no. 3 (2018): 785-794.

[4] J. Lau, J. P. Ioannidis, N. Terrin, C. H. Schmid, and I. Olkin, “The Case of the Misleading Funnel Plot,” BMJ (Clinical Research Ed) 333, no. 7568 (2006): 560-597.

[5] M. Simmonds, “Quantifying the Risk of Error When Interpreting Funnel Plots,” Systematic Reviews 4 (2015): 24.

[6] J. P. Hunter, A. Saratzis, A. J. Sutton, R. H. Boucher, R. D. Sayers, and M. J. Bown, “In Meta-Analyses of Proportion Studies, Funnel Plots Were Found to Be an Inaccurate Method of Assessing Publication Bias,” Journal of Clinical Epidemiology 67, no. 8 (2014): 897-903.

[7] H. A. Cheema, A. Shahid, M. Ehsan, and M. Ayyan, “The Misuse of Funnel Plots in Meta-Analyses of Proportions: Are They Really Useful?,” Clinical Kidney Journal 15, no. 6 (2022): 1209-1210.

[8] P. P. Zwetsloot, M. Van Der Naald, E. S. Sena, et al., “Standardized Mean Differences Cause Funnel Plot Distortion in Publication Bias Assessments,” eLife 6 (2017): e24260.

[9] J. L. Tang and J. L. Liu, “Misleading Funnel Plot for Detection of Bias in Meta-Analysis,” Journal of Clinical Epidemiology 53, no. 5 (2000): 477-484.

[10] S. W. Choi and D. M. Lam, “Funnels for Publication Bias—Have We Lost the Plot?,” Anaesthesia 71, no. 3 (2016): 338-341.

[11] M. A. Shamim, “Real-Life Implications of Prevalence Meta-Analyses? Doi Plots and Prediction Intervals Are the Answer,” The Lancet Microbe 4, no. 7 (2023): e490.

[12] L. Furuya-Kanamori, J. J. Barendregt, and S. A. R. Doi, “A New Improved Graphical and Quantitative Method for Detecting Bias in Meta-Analysis,” International Journal of Evidence-based Healthcare 16, no. 4 (2018): 195-203.

[13] M. Egger, G. Davey Smith, M. Schneider, and C. Minder, “Bias in Meta-Analysis Detected by a Simple, Graphical Test,” BMJ (Clinical Research Ed) 315, no. 7109 (1997): 629-634.

[14] C. B. Begg and M. Mazumdar, “Operating Characteristics of a Rank Correlation Test for Publication Bias,” Biometrics 50, no. 4 (1994): 1088-1101.

[15] J. L. Peters, A. J. Sutton, D. R. Jones, K. R. Abrams, and L. Rushton, “Comparison of Two Methods to Detect Publication Bias in Meta-Analysis,” JAMA 295, no. 6 (2006): 676-680.

[16] S. G. Thompson and S. J. Sharp, “Explaining Heterogeneity in Meta-Analysis: A Comparison of Methods,” Statistics in Medicine 18, no. 20 (1999): 2693-2708.

[17] L. Furuya-Kanamori, C. Xu, L. Lin, et al., “P Value-Driven Methods Were Underpowered to Detect Publication Bias: Analysis of Cochrane Review Meta-Analyses,” Journal of Clinical Epidemiology 118 (2020): 86-92.

[18] G. Schwarzer, G. Rücker, and C. Semaca, “LFK Index Does Not Reliably Detect Small-Study Effects in Meta-Analysis: A Simulation Study,” Research Synthesis Methods 15, no. 4 (2024): 603-615.

[19] [@affineincontrol] VT. “If You Start With the Wrong Framework Then the Ability to Do Complex Analyses May Seem Like It Is Giving Insight, but What You Are Mostly Doing Is Studying How Wrong Your Framework Is 2021.” (accessed Jan 2025), https://x.com/affineincontrol/status/1363485258520346629.

[20] J. A. Sterne, D. Gavaghan, and M. Egger, “Publication and Related Bias in Meta-Analysis: Power of Statistical Tests and Prevalence in the Literature,” Journal of Clinical Epidemiology 53, no. 11 (2000): 1119-1129.

[21] J. P. Higgins and S. G. Thompson, “Controlling the Risk of Spurious Findings From Meta-Regression,” Statistics in Medicine 23, no. 11 (2004): 1663-1682.

[22] A. Fingerhut, F. Lacaine, and A. Cuschieri, “Medical SPIN: Misinformation by Another Name,” Surgical Endoscopy 29, no. 6 (2015): 1257-1258.

[23] N. Terrin, C. H. Schmid, and J. Lau, “In an Empirical Evaluation of the Funnel Plot, Researchers Could Not Visually Identify Publication Bias,” Journal of Clinical Epidemiology 58, no. 9 (2005): 894-901.

[24] J. L. Peters, A. J. Sutton, D. R. Jones, K. R. Abrams, and L. Rushton, “Contour-Enhanced Meta-Analysis Funnel Plots Help Distinguish Publication Bias From Other Causes of Asymmetry,” Journal of Clinical Epidemiology 61, no. 10 (2008): 991-996.

[25] L. Lin, “Graphical Augmentations to Sample-Size-Based Funnel Plot in Meta-Analysis,” Research Synthesis Methods 10, no. 3 (2019): 376-388.

[26] A. Ghasemi and S. Zahediasl, “Normality Tests for Statistical Analysis: A Guide for Non-Statisticians,” International Journal of Endocrinology and Metabolism 10, no. 2 (2012): 486-489.

[27] J. Stone, U. Gurunathan, K. Glass, Z. Munn, P. Tugwell, and S. A. R. Doi, “Stratification by Quality Induced Selection Bias in a Meta-Analysis of Clinical Trials,” Journal of Clinical Epidemiology 107 (2019): 51-59.

[28] L. Furuya-Kanamori, C. Xu, S. S. Hasan, and S. A. Doi, “Quality Versus Risk-of-Bias Assessment in Clinical Research,” Journal of Clinical Epidemiology 129 (2021): 172-175.

[29] J. C. Stone, K. Glass, Z. Munn, P. Tugwell, and S. A. R. Doi, “Comparison of Bias Adjustment Methods in Meta-Analysis Suggests That Quality Effects Modeling May Have Less Limitations Than Other Approaches,” Journal of Clinical Epidemiology 117 (2020): 36-45.

[30] J. J. Deeks, P. Macaskill, and L. Irwig, “The Performance of Tests of Publication Bias and Other Sample Size Effects in Systematic Reviews of Diagnostic Test Accuracy Was Assessed,” Journal of Clinical Epidemiology 58, no. 9 (2005): 882-893.

[31] J. A. C. Sterne, A. J. Sutton, J. P. A. Ioannidis, et al., “Recommendations for Examining and Interpreting Funnel Plot Asymmetry in Meta-Analyses of Randomised Controlled Trials,” BMJ (Clinical Research Ed) 343 (2011): d4002.

[32] J. P. Ioannidis and T. A. Trikalinos, “The Appropriateness of Asymmetry Tests for Publication Bias in Meta-Analyses: A Large Survey,” CMAJ 176, no. 8 (2007): 1091-1096.

RIGHTS & PERMISSIONS

2025 The Author(s). Journal of Evidence-Based Medicine published by Chinese Cochrane Center, West China Hospital of Sichuan University and John Wiley & Sons Australia, Ltd.