Design and development of scales in primary care: Practical steps and statistical methods^{☆, ☆☆}

Abstract

This article outlines the statistical methods and practical steps involved in designing and developing valid and reliable questionnaires in primary care. Based on literature review on questionnaire development and scale design, we proposed a standardized protocol for scale development in primary care. This process encompasses key practical steps and statistical methods, illustrated by cases from prior research.

The recommended seven-step approach includes: (1) defining the construction of measurement; (2) generating the pool of items; (3) selecting the scoring system and response format; (4) pre-testing (assessing content validity and face validity, etc.); (5)eliminating items by item analysis; (6)evaluating the scale initially, including evaluating the reliability and validity of the scale, and factor analysis or Rasch analysis; (7)re-evaluating the scale to re-examine the nature of the scale, including retesting reliability and constructing validity. In conclusion, scale development studies should adhere to standardized procedures, and the integrated use of Rasch model and factor analysis can make the measurements more objective.

Keywords

Primary health care / Scale development / Research design / Factor analysis / Rasch model

Declarations

Not applicable.

Ethical approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Availability of data and materials

Not applicable.

Funding

Not applicable.

Declaration of competing interest

All authors declare that there are no competing interests

CRediT authorship contribution statement

Wang Fei: Conceptualization, Writing - original draft. Tang Jingqi: Methodology, Data curation, Formal analysis, Funding acquisition, Project administration, Resources, Supervision, Validation. Sun Xiaonan: Writing - review & editing. Sun Xinying: Writing - review & editing. Li Jun: Writing - review & editing. Meng Xingxing: Methodology, Data curation, Formal analysis, Funding acquisition, Project administration, Resources, Supervision, Validation. Wu Yibo: Methodology, Data curation, Formal analysis, Funding acquisition, Project administration, Resources, Supervision, Validation.

Acknowledgements

We are deeply grateful to Associate Professor Gao Zhiqiang of the School of Philosophy, Anhui University, for his valuable guidance in the field of psychometrics. His course on psychometrics provided the authors with an early and foundational understanding of this discipline. We also acknowledge the contributions of all investigators involved in the Psychology and Behavior Investigation of Chinese Residents(PBICR) in 2021. Their dedicated participation enabled the collection of extensive data that underpins the figures and analyses presented in this study.

References

[1] Wang R, Wang S. Research on the role of general practitioners in China’s medical and health service system. Chin Gen Pract. 2020; 23(4):388-394 402. Chinese. doi: 10.12114/j.issn.1007-9572.2020.00.084.

[2] Trakman GL, Forsyth A, Hoye R, et al. Developing and validating a nutrition knowledge questionnaire: key methods and considerations. Public Health Nutr. 2017; 20(15):2670-2679. doi: 10.1017/S1368980017001471.

[3] Kouvelioti R, Vagenas G. Methodological and statistical quality in research evaluating nutritional attitudes in sports. Int J Sport Nutr Exerc Metab. 2015; 25(6):624-635. doi: 10.1123/ijsnem.2014-0010.

[4] Jin Y, Chen S, Bao Y, et al. Research progress on assessment tools for children with autism spectrum disorder. Nurs Pract Res. 2021; 18(9):1325-1329 Chinese. doi: 10.3969/j.issn.1672-9676.2021.09.016.

[5] Bond TG, Yan Z, Heene M. Applying the Rasch Model:Fundamental Measurement in the Human Sciences. New York: Routledge; 2020:1-348.

[6] Wang F, Wu YC, Sun XN, et al. Reliability and validity of the Chinese version of a short form of the family health scale. BMC Prim Care. 2022; 23(1):108. doi: 10.1186/s12875-022-01702-1.

[7] Gao Z, Zhang T. Development and application of the success fear questionnaire. Chin J Clin Psychol. 2011; 19(5):602-605 686. Chinese. doi: 10.16128/j.cnki.1005-3611.2011.05.009.

[8] Alsaffar AA. Validation of a general nutrition knowledge questionnaire in a Turkish student sample. Public Health Nutr. 2012; 15(11):2074-2085. doi: 10.1017/S1368980011003594.

[9] Folasire OF, Akomolafe AA, Sanusi RA. Does nutrition knowledge and practice of athletes translate to enhanced athletic performance? Cross-sectional study amongst Nigerian undergraduate athletes. Glob J Health Sci. 2015; 7(5):215-225. doi: 10.5539/gjhs.v7n5p215.

[10] Hu H, Zhang H, Wang J, et al. Development and preliminary evaluation of the psychological resilience scale for middle school students. Chin J Sch Health. 2009; 30(12):1097-1099 Chinese.

[11] Podsakoff PM, MacKenzie SB, Lee JY, et al. Common method biases in behavioral research: a critical review of the literature and recommended remedies. J Appl Psychol. 2003; 88(5):879.

[12] Zhao F, He Z, Yuan S, et al. Rasch model analysis of the Olweus bullying scale. J Southwest Univ (Soc Sci Ed). 2020; 46(5):115-121 Chinese. doi: 10.13718/j.cnki.xdsk.2020.05.012.

[13] Wang YJ, Kaierdebieke A, SY F, et al. Study protocol: a cross-sectional study on psychology and behavior investigation of Chinese residents, PBICR. Psychosom Med Res. 2022; 4(3):19.

[14] Linacre JM. Optimizing rating scale category effectiveness. J Appl Meas. 2002; 3(1):85-106.

[15] Perneger TV, Courvoisier DS, Hudelson PM, et al. Sample size for pre-tests of questionnaires. Qual Life Res. 2015; 24(1):147-151. doi: 10.1007/s11136-014-0752-2.

[16] Cheng Y, Wang Y, Li Y, et al. Development and reliability and validity test of the home care behavior scale for caregivers of disabled elderly. Chin J Gerontol. 2018; 38(21):5314-5316 Chinese.

[17] Gao Y, Pan B. Reliability and validity test of the stamps nurse job satisfaction scale. Chin Nurs Res. 2010; 24(7):645-646 Chinese.

[18] Rabin LA, Miles RT, Kamata A, et al. Development, item analysis, and initial reliability and validity of three forms of a multiple-choice mental health literacy assessment for college students (MHLA-c). Psychiatry Res. 2021; 300:113897. doi: 10.1016/j.psychres.2021.113897.

[19] Hui J, Pei J, Wang Y, et al. Rasch analysis of the stroke specific quality of life scale for acupuncture intervention. Chin Acupunct Moxibustion. 2013; 33(4):363-366 Chinese. doi: 10.13703/j.0255-2930.2013.04.029.

[20] Hua J, Zhang L, Gu G, et al. Preliminary development of the home environment scale for motor development of urban preschool children. Chin J Sch Health. 2011; 32(2):161-163 Chinese. doi: 10.16835/j.cnki.1000-9817.2011.02.015.

[21] Yang Z, Zhang H. Cross-cultural adaptation and reliability and validity test of the elderly health promotion scale. J Nurs Sci. 2021; 36(19):91-94 Chinese. doi: 10.3870/j.issn.1001-4152.2021.19.091.

[22] Lai JS, Cella D, Chang CH, et al. Item banking to improve, shorten and computerize self-reported fatigue: an illustration of steps to create a core item bank from the FACIT Fatigue Scale. Qual Life Res. 2003; 12(5):485-501. doi: 10.1023/a:1025014509626.

[23] Zwick R, Thayer DT, Lewis C. An empirical Bayes approach to Mantel-Haenszel DIF analysis. J Educ Meas. 1999; 36(1):1-28. doi: 10.1111/j.1745-3984.1999.tb00543.x.

[24] Du H, Li F. Analysis of the influence of sample size on the Mantel-Haenszel method for testing DIF effects. Exam Res. 2016; 12(5):55-62 Chinese.

[25] Gao S, Zhang X. Applying the Rasch model to analyze the Rosenberg self-esteem scale. Psychol Explor. 2018; 38(5):445-450 Chinese.

[26] Vaganian L, Bussmann S, Boecker M, et al. An item analysis according to the Rasch model of the German 12-item WHO disability assessment schedule (WHODAS 2.0). Qual Life Res. 2021; 30(10):2929-2938. doi: 10.1007/s11136-021-02872-8.

[27] Yang Z., House; HX.DP.S:SLP 2016. Chinese.

[28] Comrey AL. A First Course in Factor Analysis. New York: Academic Press; 1973.

[29] Cao C, Qi S, Jin T. Development of the impression management self-efficacy scale for college students. Mod Prev Med. 2021; 48(17):3199-3201 3225. Chinese..

[30] Chen G, Cai T, Hu F, et al. Revision of the emotional eating scale in Chinese adolescents. Chin J Clin Psychol. 2013; 21(4):572-575 588. Chinese. doi: 10.16128/j.cnki.1005-3611.2013.04.026.

[31] Wang M. Latent Variable Modeling and Mplus Application:Basic Edition. Chongqing: Chongqing University Press; 2014 Chinese.

[32] Guo J, Wang Y, Song Y, et al. Preliminary revision of the Chinese version of the psychological vulnerability questionnaire and reliability and validity analysis in community residents. Chin J Public Health. 2019; 35(2):129-133 Chinese. doi: 10.11847/zgggws1119908.

[33] Floyd FJ, Widaman KF. Factor analysis in the development and refinement of clinical assessment instruments. Psychol Assess. 1995; 7(3):286-299. doi: 10.1037/1040-3590.7.3.286.

[34] Liu L, Liu H, Guo H, et al. Reliability and validity test and applicability analysis of the Chinese version of the psychological need satisfaction in exercise scale for the elderly. Chin Gen Pract. 2021; 24(5):619-624 Chinese. doi: 10.12114/j.issn.1007-9572.2020.00.459.

[35] Liu A, Liu Y. Development and reliability and validity test of the health knowledge, attitude and practice questionnaire for patients with sudden deafness. J Audiol Speech Pathol. 2012; 20(5):444-448 Chinese. doi: 10.3969/j.issn.1006-7299.2012.05.010.

[36] Jian X, Dai B. SPSS 23.0 Statistical Analysis: Applications in Psychology and Education. Beijing: Beijing Normal University Publishing Group; 2017 Chinese.

[37] Cui C, Yue M, Li Y, et al. Reliability and validity study of the Chinese version of the Health Behavior Scale. J Nurs Sci. 2017; 32(12):62-65 Chinese. doi: 10.3870/j.issn.1001-4152.2017.12.062.

[38] Yang L, Zhai R, Qi Z, et al. Mental health quality assessment system: development of the cognitive style questionnaire for Chinese Adults. Psychol Behav Res. 2012; 10(5):332-339 Chinese.

[39] Wu YB, Dong SJ, Li XY, et al. The transcultural adaptation and validation of the Chinese version of the Duke anticoagulation satisfaction scale. Front Pharmacol. 2022; 13:790293. doi: 10.3389/fphar.2022.790293.

[40] You Y, Yu S, Liang B. Trial and evaluation of the beck depression inventory among teachers in disaster areas. J Sichuan Norm Univ (Nat Sci Ed). 2011; 34(3):439-442 Chinese. doi: 10.3969/j.issn.1001-8395.2011.03.033.

[41] Yan Z. Objective measurement in the field of psychological science: characteristics and development trends of the Rasch model. Adv Psychol Sci. 2010; 18(8):1298-1305 Chinese.

[42] Chen Y, Yang C, Wang D, et al. Sinicization of the nutritional literacy assessment tool and reliability and validity study in patients with diabetes: analysis based on CTT and Rasch models. Chin Gen Pract. 2020; 23(26):3342-3347 Chinese. doi: 10.12114/j.issn.1007-9572.2020.00.009.

[43] Zhao S, He F, Liu Y. Application of the Rasch model in the quality analysis of academic achievement tests. Educ Res Exp. 2013(1):87-91.

[44] Bagozzi RP, Yi Y. On the evaluation of structural equation models. J Acad Mark Sci. 1988; 16(1):74-94. doi: 10.1007/BF02723327.

[45] Zhang L, Gu Y, Huang M, et al. Confirmatory factor analysis of the evidence-based practice readiness assessment scale. Nurs J Chin PLA. 2019; 36(2):6-10 25. Chinese. doi: 10.3969/j.issn.1008-9993.2019.02.002.

[46] Sun X. Principles and Practice of Health Questionnaire Design. Beijing: Peking University Medical Press; 2020 Chinese.