Comparing Three Approaches to Handling Dependency: A Case Study of a Meta-Analysis of Writing Self-Efficacy and Writing Proficiency

Abstract

This study compared three approaches (i.e., averaging within-study effect sizes, three-level meta-analysis, and robust variance estimation) to handle dependent correlational effect sizes in conducting a meta-analysis. Data were from a meta-analytic study examining the relationship between writing self-efficacy and writing proficiency. To examine the differences in the performance of the three approaches, seven conditions were created by the number of studies and the number of effect sizes per study. While all three approaches produced similar results in the average effect size and standard error, the averaging approach had much smaller variance estimates. The patterns were basically consistent across different conditions. This study informs meta-analysts of appropriate procedures in handling the dependent effect sizes.

Keywords

dependency / effect size / averaging / three-level meta-analysis / robust variance estimation

Author Contributions

T.S.: conceptualization, methodology, software, data curation, writing-original draft preparation, visualization, investigation, writing-reviewing and editing; C.W.: conceptualization, methodology, writing-reviewing and editing, supervision, validation; R.G.L.: conceptualization, methodology, writing-reviewing and editing, supervision, validation. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

The data supporting the findings of this study are available from the corresponding author and can be shared upon reasonable request.

Conflicts of Interest

The authors declare no conflict of interest.

Use of AI and AI-Assisted Technologies

No AI tools were utilized for this paper.

References

[1] Borenstein M.; Hedges L.V.; Higgins J.P.; et al. Introduction to Meta-Analysis; John Wiley & Sons: Hoboken, NJ, USA, 2009.

[2] Moeyaert M.; Ugille M.; Beretvas S.N.; et al. Methods for dealing with multiple outcomes in meta-analysis: A comparison between averaging effect sizes, robust variance estimation and multilevel meta-analysis. Int. J. Soc. Res. Methodol. 2017, 20, 559-572. https://doi.org/10.1080/13645579.2016.1252189.

[3] Ahn S.; Ames A.J.; Myers N.D. A review of meta-analyses in education: Methodological strengths and weaknesses. Rev. Educ. Res. 2012, 82, 436-476. http://doi.org/10.3102/0034654312458162.

[4] Scammacca N.; Roberts G.; Stuebing K.K. Meta-analysis with complex research designs: Dealing with dependence from multiple measures and multiple group comparisons. Rev. Educ. Res. 2014, 84, 328-364. http://doi.org/10.3102/0034654313500826.

[5] Cheung S.F.; Chan D. Dependent effect sizes in meta-analysis: Incorporating the degree of interdependence. J. Appl. Psychol. 2004, 89, 780-791. https://doi.org/10.1037/0021-9010.89.5.780.

[6] Tanner-Smith E.; Tipton E. Robust variance estimation with dependent effect sizes: Practical considerations and a software tutorial in Stata and SPSS. Res. Synth. Methods 2014, 5, 13-30. https://doi.org/10.1002/jrsm.1091.

[7] Lambert R.G. The ROBUSTNESS of the Standard Errors of Summarized, Corrected Validity Coefficients to Non-Independence and Non-Normality of Primary Data. Doctoral Dissertation, Georgia State University, Atlanta, GA, USA, 1995.

[8] Hashemnejad F.; Zoghi M.; Amini D. The relationship between self-efficacy and writing performance across genders. Theory Pract. Lang. Stud. 2014, 4, 1045-1052. https://doi.org/10.4304/tpls.4.5.1045-1052.

[9] Perin D.; Lauterbach M.; Raufman J.; et al. Text-based writing of low-skilled postsecondary students: Relation to comprehension, self-efficacy and teacher judgments. Read. Writ. 2017, 30, 887-915. http://doi.org/10.1007/s11145-016-9706-0.

[10] Shintani N.; Aubrey S. The effectiveness of synchronous and asynchronous written corrective feedback on grammatical accuracy in a computer-mediated environment. Mod. Lang. J. 2016, 100, 296-319. http://doi.org/10.1111/modl.12317.

[11] Van den Noortgate, W.; López-López J.A.; Marín-Martínez F.; et al. Three-level meta-analysis of dependent effect sizes. Behav. Res. Methods 2013, 45, 576-594. http://doi.org/10.3758/s13428-012-0261-6.

[12] Kalaian H.A.; Raudenbush S.W. A multivariate mixed linear model for meta-analysis. Psychol. Methods 1996, 1, 227-235. http://doi.org/10.1037/1082-989X.1.3.227.

[13] Cooper H.M. Synthesizing Research: A Guide for Literature Reviews, 3rd ed.; Sage: Thousand Oaks, CA, USA, 1998.

[14] Cooper H.M. Research Synthesis and Meta-Analysis: A Step-by-Step Approach, 4th ed.; Sage: Thousand Oaks, CA, USA, 2010.

[15] Chambers E.A. An introduction to meta-analysis with articles from the Journal of Educational Research (1992-2002). J. Educ. Res. 2004, 98, 35-45. https://doi.org/10.3200/JOER.98.1.35-45.

[16] Cheung M.W.L. A guide to conducting a meta-analysis with non-independent effect sizes. Neuropsychol. Rev. 2019, 29, 387-396. http://doi.org/10.1007/s11065-019-09415-6.

[17] Cheung M.W.L. Modeling dependent effect sizes with three-level meta-analyses: A structural equation modeling approach. Psychol. Methods 2014, 19, 211-229. http://doi.org/10.1037/a0032968.

[18] Hedges L.V.; Tipton E.; Johnson M.C. Robust variance estimation in meta-regression with dependent effect size estimates. Res. Synth. Methods 2010, 1, 39-65. http://doi.org/10.1002/jrsm.5.

[19] Marín-Martínez F.; Sánchez-Meca J. Averaging dependent effect sizes in meta-analysis: A cautionary note about procedures. Span. J. Psychol. 1999, 2, 32-38. http://doi.org/10.1017/S1138741600005436.

[20] Raudenbush S.W.; Bryk A.S. Hierarchical Linear Models: Applications and Data Analysis Methods; Sage: Thousand Oaks, CA, USA, 2002.

[21] Konstantopoulos S. Fixed effects and variance components estimation in three-level meta-analysis. Res. Synth. Methods 2011, 2, 61-76. http://doi.org/10.1002/jrsm.35.

[22] Fernández-Castilla B.; Jamshidi Declercq L.; Beretvas S.N.; et al. The application of meta-analytic (multi-level) models with multiple random effects: A systematic review. Behav. Res. Methods 2020, 52, 2031-2052. https://doi.org/10.3758/s13428-020-01373-9.

[23] Geeraert L.; VandenNoortgate W.; Grietens H.; et al. The effects of early prevention programs for families with young children at risk for physical child abuse and neglect: A meta-analysis. Child Maltreatment 2004, 9, 277-291. https://doi.org/10.1177/1077559504264265.

[24] Tipton E. Robust variance estimation in meta-regression with binary dependent effects. Res. Synth. Methods 2013, 4, 169-187. https://doi.org/10.1002/jrsm.1070.

[25] Tanner-Smith E.; Tipton E.; Polanin J.R. Handling complex meta- analytic data structures using robust variance estimates: A tutorial in R. J. Dev. Life-Course Criminol. 2016, 2, 85-112. http://doi.org/10.1007/s40865-016-0026-5.

[26] Sun T.; Wang C.; Lambert R.G.; Liu L. Relationship between second language English writing self-efficacy and achievement: A meta-regression analysis. J. Second. Lang. Writ. 2021, 53, 100817.

[27] Peterson R.A.; Brown S.P. On the use of beta coefficients in meta-analysis. J. Appl. Psychol. 2005, 90, 175-181.

[28] Viechtbauer W. Conducting meta-analyses in R with the metaphor package. J. Stat. Softw. 2010, 36, 1-48. https://doi.org/10.18637/jss.v036.i03.

[29] Fisher Z.; Tipton E. Robumeta: Robust Variance Meta-Regression. 2014. Available online: http://cran.rproject.org/web/packages/robumeta/index.html. (accessed on1 September 2020).

[30] Higgins J.P.; Thompson S.G.; Deeks J.J.; et al. Measuring inconsistency in meta-analyses. BMJ 2003, 327, 557-560. http://doi.org/10.1136/bmj.327.7414.557.

[31] Rodgers M.A.; Pustejovsky J.E. Evaluating meta-analytic methods to detect selective reporting in the presence of dependent effect sizes. Psychol. Methods 2021, 26, 141.

[32] Thompson B. Guidelines for authors. Educ. Psychol. Meas. 1994, 54, 837-847.

[33] American Psychological Association. Publication Manual of the American Psychological Association, 4th ed.; American Psychological Association: Washington, DC, USA, 1994.

[34] American Psychological Association. Publication Manual of the American Psychological Association, 5th ed.; American Psychological Association: Washington, DC, USA, 2001.

[35] American Psychological Association. Publication Manual of the American Psychological Association, 6th ed.; American Psychological Association: Washington, DC, USA, 2010.

[36] American Psychological Association. Publication Manual of the American Psychological Association, 7th ed.; American Psychological Association: Washington, DC, USA, 2019.