Cut-off Points for RMSEA and SRMR in Structural Equation Modeling Using ULS and RULS

Abstract

The use of Likert scales in the field of social research is becoming increasingly common; therefore, it is necessary to investigate which is the most appropriate methodology to carry out the analysis of the ordinal data obtained. If they are ordinal, they should be treated as such, however, they are frequently analyzed considering them as continuous variables. One of the most widely used techniques to obtain construct validity evidence based on the internal structure within nomological measurement models is Confirmatory Factor Analysis (CFA). Using simulation studies in which four factors were manipulated (number of factors, number of items response categories, skewness and sample size) our objective is twofold: (1) to examine, under ordinal measurement, the Type I error rate and statistical power associated with common global fit indices, specifically the Root Mean Square Error of Approximation (RMSEA) and the Standardized Root Mean Square Residual (SRMR) when computed under ULS and robust ULS (RULS) estimation; and (2) to evaluate RMSEA and SRMR cut-off values using Receiver Operating Characteristic (ROC) analysis. It is found that, depending on the estimation method chosen, the Type I error and power vary, as well as the values reported by RMSEA and SRMR. RULS seems to obtain better results regardless of experimental factors manipulated. Finally, it is found that it would be convenient to review the cut-off points for these global fit indices recommended by the literature.

Keywords

ULS / RULS / RMSEA / SRMR / Type I error rate / statistical power / ROC

Author Contributions

Conceptualization: F.P.H.-T. and J.C.S.-F.; Data curation: F.P.H.-T., J.S.-G., J.M.R. and J.C.S.-F.; Formal analysis: F.P.H.-T., J.S.-G., J.M.R. and J.C.S.-F.; Validation: F.P.H.-T. and J.C.S.-F.; Methodology: F.P.H.-T., J.S.-G., J.M.R. and J.C.S.-F.; Software: F.P.H.-T.; Visualisation: F.P.H.-T., J.S.-G., J.M.R. and J.C.S.-F.; Supervision: F.P.H.-T. and J.C.S.-F.; Writing—original draft: F.P.H.-T., J.S.-G., J.M.R. and J.C.S.-F.; Writing—revision and editing: F.P.H.-T., J.S.-G., J.M.R. and J.C.S.-F. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

The data supporting the findings of this study are available from the corresponding author and can be shared upon reasonable request.

Acknowledgments

The authors also gratefully acknowledge the careful work of the reviewers and their contributions, which have significantly improved the paper.

Conflicts of Interest

The authors declare no conflict of interest.

Use of AI and AI-Assisted Technologies

The authors acknowledge the use of ChatGPT (OpenAI, GPT-5, 2025) as a writing assistant. The tool was used exclusively to improve the readability and linguistic accuracy of the manuscript in English, as the authors are non-native speakers. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the content of the published article. All conceptualization, analyses, and interpretations were entirely the responsibility of the authors.

References

[1] DeVellis R.F.; Thorpe C.T. Scale Development: Theory and Applications, 5th ed.; Sage: Los Angeles, CA, USA, 2021.

[2] Stevens S.S. On the Theory of Scales of Measurement. Science 1946, 103, 677-680. https://doi.org/10.1126/science.103.2684.677.

[3] Allen M.; Yen W. Introduction to Measurement Theory; Waveland Press: Long Grove, IL USA, 1979.

[4] Clark L.A.; Watson D. Constructing Validity: New Developments in Creating Objective Measuring Instruments. Psychol. Assess. 2019, 31, 1412-1427. https://doi.org/10.1037/pas0000626.

[5] American Educational Research Association; American Psychological Association; National Council on Measurement in Education. Standards for Educational and Psychological Testing; American Educational Research Association: Washington, DC, USA, 2014.

[6] Cronbach L.J.; Meehl P.E. Construct Validity in Psychological Tests. Psychol. Bull. 1955, 52, 281-302. https://doi.org/10.1037/h0040957.

[7] Van Bork R. Interpreting Psychometrics Models; Riet Van Bork: Utrecht, The Netherlands, 2019. https://doi.org/10.31237/osf.io/x6a7s.

[8] Brown T.A. Confirmatory Factor Analysis for Applied Research; Guilford Press: New York, NY USA, 2015.

[9] Catena A.; Ramos Álvarez M.M.; Trujillo H.M. Análisis Multivariado. Un Manual Para Investigadores; Biblioteca Nueva: Madrid, Spain, 2003.

[10] Jackson D.L.; Gillaspy J.A.; Purc-Stephenson R. Reporting Practices in Confirmatory Factor Analysis. Psychol. Methods 2009, 14, 6-23. https://doi.org/10.1037/a0014694.

[11] Pavlov G.; Maydeu-Olivares A.; Shi D. Using the Standardized Root Mean Squared Residual (SRMR) to Assess Exact Fit in Structural Equation Models. Educ. Psychol. Meas. 2021, 81, 110-130. https://doi.org/10.1177/0013164420926231.

[12] Xia Y.; Yang Y. RMSEA CFI, and TLI in Structural Equation Modeling with Ordered Categorical Data: The Story They Tell Depends on the Estimation Methods. Behav. Res. Methods 2019, 51, 409-428. https://doi.org/10.3758/s13428-018-1055-2.

[13] Wu H.; Leung S.O. Can Likert Scales Be Treated as Interval Scales? A Simulation Study. J. Soc. Serv. Res. 2017, 43, 527-532. https://doi.org/10.1080/01488376.2017.1329775.

[14] Rhemtulla M.; Brosseau-Liard P.É.; Savalei V. When Can Categorical Variables Be Treated as Continuous? A Comparison of Robust Continuous and Categorical SEM Estimation Methods under Suboptimal Conditions. Psychol. Methods 2012, 17, 354-373. https://doi.org/10.1037/a0029315.

[15] Shi D.; Maydeu-Olivares A.; Rosseel Y. Assessing Fit in Ordinal Factor Analysis Models: SRMR vs. RMSEA. Struct. Equ. Modeling 2020, 27, 1-15. https://doi.org/10.1080/10705511.2019.1611434.

[16] Kenny D.A.; Kashy D.A. Analysis of the Multitrait-Multimethod Matrix Using Confirmatory Factor Analysis. Psychol. Bull. 1992, 112, 165-172. https://doi.org/10.1037/0033-2909.112.1.165.

[17] Millsap R.E.; Everson H. Confirmatory Measurement Model Comparisons Using Latent Means. Multivar. Behav. Res. 1991, 26, 479-497. https://doi.org/10.1207/s15327906mbr2603_6.

[18] Byrne B.M. Structural Equation Modeling with LISREL, PRELIS, and SIMPLIS: Basic Concepts, Applications, and Programming; Lawrence Erlbaum Associates: Mahwah, NJ, USA, 1998.

[19] Huh I.; Gim J. Exploration of Likert Scale in Terms of Continuous Variable with Parametric Statistical Methods. BMC Med. Res. Methodol. 2025, 25, 218. https://doi.org/10.1186/s12874-025-02668-1.

[20] Jöreskog K.G.; Moustaki I. Factor Analysis of Ordinal Variables: A Comparison of Three Approaches. Multivar. Behav. Res. 2001, 36, 347-387. https://doi.org/10.1207/S15327906347-387.

[21] Holgado-Tello F.P.; Chacón-Moscoso S.; Barbero-García I.; et al. Polychoric versus Pearson Correlations in Exploratory and Confirmatory Factor Analysis of Ordinal Variables. Qual. Quant. 2010, 44, 153-166. https://doi.org/10.1007/s11135-008-9190-y.

[22] Coenders G.; Satorra A.; Saris W.E. Alternative Approaches to Structural Modeling of Ordinal Data: A Monte Carlo Study. Struct. Equ. Modeling 1997, 4, 261-282. https://doi.org/10.1080/10705519709540077.

[23] Freiberg A.; Stover J.B.; De la Iglesia G.; et al. Correlaciones Policóricas y Tetracóricas en Estudios Factoriales Exploratorios y Confirmatorios. Cienc. Psicol. 2013, 7, 151-164.

[24] Erceg-Hurn D.M.; Mirosevich V.M. Modern Robust Statistical Methods: An Easy Way to Maximize the Accuracy and Power of Your Research. Am. Psychol. 2008, 63, 591-601. https://doi.org/10.1037/0003-066X.63.7.591.

[25] Liu Q.; Okui R.; Yoshimura A. Generalized Least Squares Model Averaging. Econom. Rev. 2016, 35, 1692-1752. https://doi.org/10.1080/07474938.2015.1092817.

[26] Jöreskog K.G.; Sörbom D. LISREL 8 User’s Reference Guide; Scientific Software: Chicago, IL, USA, 1996.

[27] Savalei V. Improving Fit Indices in Structural Equation Modeling with Categorical Data. Multivar. Behav. Res. 2021, 56, 390-407. https://doi.org/10.1080/00273171.2020.1717922.

[28] Li C.H. Confirmatory Factor Analysis with Ordinal Data: Comparing Robust Maximum Likelihood and Diagonally Weighted Least Squares. Behav. Res. Methods 2016, 48, 936-949. https://doi.org/10.3758/s13428-015-0619-7.

[29] MacCallum R.C. Factor Analysis. In The SAGE Handbook of Quantitative Methods in Psychology; Millsap R.E., Maydeu-Olivares A.,Eds.; Sage: Los Angeles, CA, USA, 2009; pp. 123-147.

[30] Saris W.E.; Stronkhorst H. Causal Modelling in Non-Experimental Research: An Introduction to the LISREL Approach; Sociometric Research Foundation: Amsterdam, The Netherlands, 1984.

[31] Morata-Ramírez M.; Holgado-Tello F.P.; Barbero-García I.; et al. Análisis Factorial Confirmatorio: Recomendaciones Sobre Mínimos Cuadrados No Ponderados en Función del Error Tipo I de Ji-Cuadrado y RMSEA. Acción Psicol. 2015, 12, 79-90. https://doi.org/10.5944/ap.12.1.14362.

[32] Yang-Wallentin F.; Jöreskog K.G.; Luo H. Confirmatory Factor Analysis of Ordinal Variables with Misspecified Models. Struct. Equ. Modeling 2010, 17, 392-423. https://doi.org/10.1080/10705511.2010.489003.

[33] Batista J.M.; Coenders G. Modelos de Ecuaciones Estructurales: Modelos Para el Análisis de Relaciones Causales; Hespérides: Madrid, Spain, 2000.

[34] Schumacker R.E.; Lomax R.G. A Beginner’s Guide to Structural Equation Modeling; Lawrence Erlbaum Associates, Inc.: NJ USA, 1996.

[35] Sass D.A.; Schmitt T.A.; Marsh H.W. Evaluating Model Fit with Ordered Categorical Data within a Measurement Invariance Framework: A Comparison of Estimators. Struct. Equ. Modeling 2014, 21, 167-180. https://doi.org/10.1080/10705511.2014.882658.

[36] Jordan F.M. Valor de Corte de los Índices de Ajuste en el Análisis Factorial Confirmatorio. Psocial 2021, 7, 66-71.

[37] Powell D.A.; Shafer W.D. The Robustness of the Likelihood Ratio Chi-Square Test for Structural Equation Models: A Meta-Analysis. J. Educ. Behav. Stat. 2001, 26, 105-132. https://doi.org/10.3102/10769986026001105.

[38] Hayduk L.A. Shame for Disrespecting Evidence: The Personal Consequences of Insufficient Respect for Structural Equation Model Testing. BMC Med. Res. Methodol. 2014, 14, 124. https://doi.org/10.1186/1471-2288-14-124.

[39] McIntosh C.N. Improving the evaluation of model fit in confirmatory factor analysis: A commentary on Gundy, C.M., Fayers, P.M., Groenvold, M., Petersen, M. Aa., Scott, N.W., Sprangers, M.A.J., Velikov, G., Aaronson, N.K. ( 2011). Comparing higher-order models for the EORTC QLQ-C30. Qual. Life Res. 2012, 21, 1619-1621. https://doi.org/10.1007/s11136-011-0084-4.

[40] Shi D.; Lee T.; Maydeu-Olivares A. Understanding the Model Size Effect on SEM Fit Indices. Educ. Psychol. Meas. 2018, 79, 310-334. https://doi.org/10.1177/0013164418783530.

[41] Holgado-Tello F.P.; Suárez-Falcón J.C.; Martínez-Huertas M.A.; et al. Modelos de Ecuaciones Estructurales con LISREL y Lavaan; Sanz y Torres: Madrid, Spain, 2024.

[42] Shi D.; Maydeu-Olivares A. The Effect of Estimation Methods on SEM Fit Indices. Educ. Psychol. Meas. 2020, 80, 421-445. https://doi.org/10.1177/0013164419885164.

[43] Yin Y.; Fairchild A.J.; Shi D.; et al. Estimating the Root Mean Square Error of Approximation (RMSEA) with Multiply Imputed Data Under Non-Normality. Struct. Equ. Modeling 2025, 32, 832-857. https://doi.org/10.1080/10705511.2025.2515229.

[44] Lai K. Correct Point Estimator and Confidence Interval for RMSEA Given Categorical Data. Struct. Equ. Modeling 2020, 27, 678-695. https://doi.org/10.1080/10705511.2019.1687302.

[45] Bollen K.A. Structural Equations with Latent Variables; John Wiley & Sons: Hoboken, NJ, USA, 1989. https://doi.org/10.1002/9781118619179.

[46] Cho G.; Hwang H.; Sarstedt M.; et al. Cutoff Criteria for Overall Model Fit Indexes in Generalized Structured Component Analysis. J. Mark. Anal. 2020, 8, 189-202. https://doi.org/10.1057/s41270-020-00089-1.

[47] Hu L.; Bentler P.M. Cutoff Criteria for Fit Indexes in Covariance Structure Analysis: Conventional Criteria versus New Alternatives. Struct. Equ. Modeling 1999, 6, 1-55. https://doi.org/10.1080/10705519909540118.

[48] Kenny D.A.; Kaniskan B.; McCoach D.B. The Performance of RMSEA in Models with Small Degrees of Freedom. Sociol. Methods Res. 2015, 44, 486-507. https://doi.org/10.1177/0049124114543236.

[49] Maydeu-Olivares A.; Fairchild A.J.; Hall A.G. Goodness of Fit in Item Factor Analysis: Effect of the Number of Response Alternatives. Struct. Equ. Modeling 2017, 24, 495-505. https://doi.org/10.1080/10705511.2017.1289816.

[50] MacCallum R.C.; Browne M.W.; Sugawara H.M. Power Analysis and Determination of Sample Size for Covariance Structure Modeling. Psychol. Methods 1996, 1, 130-149. https://doi.org/10.1037/1082-989X.1.2.130.

[51] Kline R.B. Principles and Practice of Structural Equation Modeling, 3rd ed.; Guilford Press: New York, NY USA, 2011.

[52] Cole D.A. Utility of Confirmatory Factor Analysis in Test Validation Research. J. Consult. Clin. Psychol. 1987, 55, 584-594. https://doi.org/10.1037/0022-006X.55.4.584.

[53] Satorra A. Robustness Issues in Structural Equation Modeling: A Review of Recent Developments. Qual. Quant. 1990, 24, 367-386. https://doi.org/10.1007/BF00152011.

[54] Hu L.; Bentler P.M. Fit Indices in Covariance Structure Modeling: Sensitivity to Underparameterized Model Misspecification. Psychol. Methods 1998, 3, 424-453. https://doi.org/10.1037/1082-989X.3.4.424.

[55] Boomsma A.; Hoogland J.J. The Robustness of LISREL Modeling Revisited. In Structural Equation Modeling: Present and Future; Cudeck R., du Toit S., Sörbom D.,Eds.; Scientific Software International: Lincolnwood, IL, USA, 2001; pp. 139-168.

[56] Kane M. The Argument-Based Approach to Validation. Sch. Psychol. Rev. 2013, 42, 448-457. https://doi.org/10.1080/02796015.2013.12087465.

[57] Muthén B.; Kaplan D. A Comparison of Some Methodologies for the Factor Analysis of Non-Normal Likert Variables. Br. J. Math. Stat. Psychol. 1985, 38, 171-189. https://doi.org/10.1111/j.2044-8317.1985.tb00832.x.

[58] Newsom J.T. Longitudinal Structural Equation Modeling: A Comprehensive Introduction; Routledge: New York, NY USA, 2015.

[59] Akobeng A.K. Understanding Type I and Type II Errors, Statistical Power and Sample Size. Acta Paediatr. 2016, 105, 605-609. https://doi.org/10.1111/apa.13384.

[60] Fawcett T. An Introduction to ROC Analysis. Pattern Recognit. Lett. 2006, 27, 861-874. https://doi.org/10.1016/j.patrec.2005.10.010.

[61] Rota M.; Antolini L.; Valsecchi M.G. Optimal Cut-Point Definition in Biomarkers: The Case of Censored Failure Time Outcome. BMC Med. Res. Methodol. 2015, 15, 24. https://doi.org/10.1186/s12874-015-0009-y.

[62] Krzanowski W.J.; Hand D.J. ROC Curves for Continuous Data; CRC Press: Boca Raton, FL USA, 2009.

[63] Padgett R.N.; Morgan G.B. Multilevel CFA with Ordered Categorical Data: A Simulation Study Comparing Fit Indices across Robust Estimation Methods. Struct. Equ. Modeling 2021, 28, 51-68. https://doi.org/10.1080/10705511.2020.1759426.

[64] Sharma S.; Mukherjee S.; Kumar A.; et al. A Simulation Study to Investigate the Use of Cutoff Values for Assessing Model Fit in Covariance Structure Models. J. Bus. Res. 2005, 58, 935-943. https://doi.org/10.1016/j.jbusres.2003.10.007.

[65] Marsh H.W.; Hau K.T.; Wen Z. In Search of Golden Rules: Comment on Hypothesis-Testing Approaches to Setting Cutoff Values for Fit Indexes and Dangers in Overgeneralizing Hu and Bentler’s (1999) Findings. Struct. Equ. Modeling 2004, 11, 320-341. https://doi.org/10.1207/s15328007sem1103_2.

[66] Browne M.W.; MacCallum R.C.; Kim C.T.; et al. When Fit Indices and Residuals are Incompatible. Psychol. Methods 2002, 7, 403-421. https://doi.org/10.1037/1082-989X.7.4.403.

[67] Heene M.; Hilbert S.; Draxler C.; et al. Masking Misfit in Confirmatory Factor Analysis by Increasing Unique Variances: A Cautionary Note on the Usefulness of Cutoff Values of Fit Indices. Psychol. Methods 2011, 16, 319-336. https://doi.org/10.1037/a0024917.

[68] Savalei V.; Rhemtulla M.. The Performance of Robust Test Statistics with Categorical Data. Br. J. Math. Stat. Psychol. 2013, 66, 201-223. https://doi.org/10.1111/j.2044-8317.2012.02049.x.

[69] Flora D.B.; Curran P.J.. An Empirical Evaluation of Alternative Methods of Estimation for Confirmatory Factor Analysis with Ordinal Data. Psychol. Methods. 2004, 9, 466-491. https://doi.org/10.1037/1082-989X.9.4.466.

[70] Ximénez C.; Maydeu-Olivares A.; Shi D.; et al. Assessing Cutoff Values of SEM Fit Indices: Advantages of the Unbiased SRMR Index and Its Cutoff Criterion Based on Communality. Struct. Equ. Modeling. 2022, 29, 368-380. https://doi.org/10.1080/10705511.2021.1992596.