Report on methodological quality assessment of primary care and general practice research in China in 2021: Quantitative Research, Systematic Review and Guidelines/Consensus Section^{☆, ☆☆}

Abstract

Background: As China advances healthcare reforms, research output in general practice and primary care has rapidly expanded in recent years. However, the methodological quality of this literature remains unclear.

Objective: This study evaluates the methodological quality of representative quantitative studies, systematic reviews, and guidelines published by Chinese researchers in the field of general practice and primary care in 2021, with the aim of providing an overview of methodological standards in this field.

Methods: From a pool of 3,122 papers collected in the Primary Care and General Practice Research Paper Productivity Report in China in 2021, 449 representative studies were sampled. A team of 22 researchers specializing in public health and general practice, from various institutions, assessed the methodological quality of 320 papers (71.3 %) using six design-specific tools (for cross-sectional studies, cohort studies, pre- and post-intervention studies, randomized controlled trials, systematic reviews, and clinical guidelines). Researchers worked in pairs under the supervision of an expert in evidence-based methodology, and descriptive statistics were used to present quality assessment results.

Results: In 114 cross-sectional studies, common methodological issues were identified in “whether the source population was representative of the study's target population” (41.2 %), “whether the reliability and validity of the survey instrument could be conclusively demonstrated” (32.5 %), and “whether the survey is clinically meaningful” (26.3 %). Among 25 cohort studies, quality issues were more concentrated in the areas of “whether the cohort was adequately followed up” (44.0 %) and “whether the co-intervention was similar among groups” (56.0 %). Of the 34 pre and post-intervention studies, quality issues were mostly found in the areas of “whether the target outcome was measured multiple times before and after the intervention” (97.1 %), “whether the sample size was large enough to generate confidence in the study results” (82.4 %), and “whether the study participants were representative of the eligible population” (61.8 %). Of the 122 randomized controlled trials, quality concerns were mostly in the areas of “blinding of different stakeholders” (25.4 %-61.5 %), “adequate concealment of random allocation” (41.8 %), and “other risks of bias” (72.1 %). Among the 19 systematic reviews, quality issues were mostly found in the areas “is the source of funding for the included studies reported” (100.0 %), “were the methods of the review developed before the start of the review” (94.7 %), “was heterogeneity reasonably discussed and explained” (84.2 %), and “was the risk of bias of individual studies considered” (84.2 %). Finally, the quality of all six clinical guidelines/consensus was rated low.

Conclusion: Overall, methodological quality remains limited in recent Chinese research in general practice and primary care, especially in cross-sectional studies, pre- and post-intervention studies, randomized controlled trials, and clinical guidelines. This highlights an urgent need for comprehensive research training, a stronger emphasis on evidence-based reporting standards, and the development of pragmatic guidelines in this field.

Keywords

General practice / Quantitative research / Systematic review / Methodological quality / Research design / Guidebooks

Declarations

Not applicable.

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Availability of data and materials

Not applicable.

Funding

Not applicable.

Authors' other information

Not applicable.

Declaration of competing interests

L.W., Y.H., W.Y., Z.Y. and L.C. are editorial members of Chinese General Practice Journal, C.X. is editor of Chinese General Practice Journal, they are not envolved in the editorial review or the decision to publish this article. All authors declare that there are no competing interests.

Acknowledgements

Not applicable.

References

[1] World Health Organization. Operational framework for primary health care: transforming vision into action. 2023 [cited 2023 Apr 10]. Available from: https://www.who.int/publications/i/item/9789240017832.

[2] State Administration for Market Regulation of the People’s Republic of China. Basic Healthcare and Health Promotion Law of the People’s Republic of China. (Chinese). 2019 Dec 28 [cited 2023 Apr 10]. Available from: https://gkml.samr.gov.cn/nsjg/bgt/202106/t20210610_330512.html.

[3] Cao X, Wang Y, Xu Z, et al. Research productivity on primary care and general practice in China in 2021. (Chinese). Chinese General Practice. 2022; 25(34):4232-4240 4258. doi: 10.12114/j.issn.1007-9572.2022.0701.

[4] Yu D. Research trends in general practice and community health: the future is here. (Chinese). Chinese General Practice. 2022; 25(34):4227-4231. doi: 10.12114/j.issn.1007-9572.2022.0702.

[5] Fu Q, Jin H, Yu D. Development and strategy of research capacity in general practice and community health in China from 2001 to 2020. (Chinese). Chinese General Practice. 2022; 25(34):4252-4258. doi: 10.12114/j.issn.1007-9572.2022.0705.

[6] Whiting P, Wolff R, Mallett S, et al. A proposed framework for developing quality assessment tools. (Chinese). Syst Rev. 2017; 6(1):204. doi: 10.1186/s13643-017-0604-6.

[7] Hummers-Pradier E, Beyer M, Chevallier P, et al. Scope, research needs, and applicable methods in general practice research: a translated summary of the “European General Practice /Family Medicine and Primary Healthcare Research Agenda ”. Chinese General Practice. 2022; 25(9):1027-1039. doi: 10.12114/j.issn.1007-9572.2022.02.006.

[8] Risk of Bias Instrument for Cross-Sectional Surveys of Attitudes and Practices. 2023 Oct 1 [cited 2023 Oct 1]. Available from: https://www.distillersr.com/resources/methodological-resources/risk-of-bias-instrument-for-cross-sectional-surveys-of-attitudes-and-practices-distillersr.

[9] Tool to assess risk of bias in cohort studies. 2023 Oct 1 [cited 2023 Oct 1]. Available from: https://www.distillersr.com/resources/methodological-resources/tool-to-assess-risk-of-bias-in-cohort-studies-distillersr.

[10] National Heart, Lung, and Blood Institute. Study quality assessment tools. 2020 [cited 2023 Oct 1]. Available from: https://www.nhlbi.nih.gov/health-topics/study-quality-assessment-tools.

[11] Study quality assessment tools - Quality assessment of controlled intervention studies. 2023 Oct 1 [cited 2023 Oct 1]. Available from: https://www.distillersr.com/resources/methodological-resources/tool-to-assess-risk-of-bias-in-randomized-controlled-trials-distillersr.

[12] Jue JJ, Cunningham S, Lohr K, et al. Developing and testing the agency for healthcare research and quality’s national guideline clearinghouse extent of adherence to trustworthy standards (NEATS) instrument. Ann Intern Med. 2019; 170(7):480-487. doi: 10.7326/M18-2950.

[13] Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017; 358. doi: 10.1136/bmj.j4008.

[14] Wang XF, Cheng ZS. Cross-sectional studies: strengths, weaknesses, and recommendations. Chest. 2020; 158(1S). doi: 10.1016/j.chest.2020.03.012.

[15] Martínez-Mesa J, DA González-Chica, Duquia RP, et al. Sampling: how to select participants in my research study? An Bras Dermatol. 2016; 91(3):326-330. doi: 10.1590/abd1806-4841.20165254.

[16] Criqui MH. Potential errors by non-response bias. Am J Public Health. 1980; 70(12):1301-1302. doi: 10.2105/ajph.70.12.1301-c.

[17] Shih CC, Shih YL, Chen JY. The association between homocysteine levels and cardiovascular disease risk among middle-aged and elderly adults in Taiwan. BMC Cardiovasc Disord. 2021; 21(1):191. doi: 10.1186/s12872-021-02000-x.

[18] Beaton DE, Bombardier C, Guillemin F, et al. Guidelines for the process of cross-cultural adaptation of self-report measures. Spine. 2000; 25(24):3186-3191. doi: 10.1097/00007632-200012150-00014.

[19] Boateng GO, Neilands TB, Frongillo EA, et al. Best practices for developing and validating scales for health, social, and behavioral research: a primer. Front Public Health. 2018; 6:149. doi: 10.3389/fpubh.2018.00149.

[20] Shi L, Zhu T, Shen M, et al. Analysis of cognition and barriers of community pharmacy care by family doctor teams in Chongming District, Shanghai. (Chinese). J Pract Pharm. 2021; 39(6):577-580. doi: 10.12206/j.issn.1006-0111.202104086.

[21] Zhan SY. Clinical Epidemiology. (Chinese). 2nd ed. Beijing: People’s Medical Publishing House; 2015.

[22] Shen HB, Qi XY. Epidemiology. (Chinese) . 8th ed. Beijing: People’s Medical Publishing House; 2013.

[23] Yan H, Xu YY. Medical Statistics. (Chinese). 3rd ed. Beijing: People’s Medical Publishing House; 2015.

[24] von Elm E, Altman DG, Egger M, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. PLoS Med. 2007; 4(10). doi: 10.1371/journal.pmed.0040296.

[25] UK Government. Before and after study: comparative studies. 2023 Feb 15 [cited 2023 Feb 15]. Available from: https://www.gov.uk/guidance/before-and-after-study-comparative-studies.

[26] Aggarwal R, Ranganathan P. Study designs: part 4 - interventional studies. Perspect Clin Res. 2019; 10(3):137-139. doi: 10.4103/picr.PICR_91_19.

[27] Hill AB. The environment and disease: association or causation? 1965. J R Soc Med. 2015; 108(1):32-37. doi: 10.1177/0141076814562718.

[28] Stratton SJ. Quasi-experimental design (pre-test and post-test studies) in prehospital and disaster research. Prehosp Disaster Med. 2019; 34(6):573-574. doi: 10.1017/S1049023X19005053.

[29] Zou C, Ou JM, Zeng X, et al. Conducting general practice education intervention research: from design to publication. (Chinese). Chinese General Practice. doi:10.12114/j.issn.1007-9572.2022.0597.

[30] Higgins JP, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Cochrane Book Series; 2008 [cited 2023]. Available from: https://training.cochrane.org/handbook.

[31] Wu XK, Stener-Victorin E, Kuang HY, et al. Effect of acupuncture and clomiphene in Chinese women with polycystic ovary syndrome: a randomized clinical trial. JAMA. 2017; 317(24):2502-2514. doi: 10.1001/jama.2017.7217.

[32] Zhishun L, Yan L, Huanfang X, et al. Effect of electroacupuncture on urinary leakage among women with stress urinary incontinence - a randomized clinical trial. Dtsch Z Für Akupunktur. 2017; 60(4):33-34. doi: 10.1016/S0415-6412(17)30124-8.

[33] Moher D, Hopewell S, Schulz KF, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. Int J Surg. 2012; 10(1):28-55. doi: 10.1016/j.ijsu.2011.10.001.

[34] Queen’s University Library. Systematic Reviews & Other Syntheses; 2023 Oct 1 [cited 2023 Oct 1]. Available from: https://guides.library.queensu.ca/knowledge-syntheses.

[35] JBI Manual for Evidence Synthesis. 2023 Oct 1 [cited 2023 Oct 1]. Available from: https://jbi-global-wiki.refined.site/space/MANUAL.

[36] Centre for Reviews and Dissemination. Health & Environmental Research Online. York: Centre for Reviews and Dissemination, University of York; 2013.

[37] Shamseer L, Moher D, Clarke M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ. 2015; 350. doi: 10.1136/bmj.g7647.

[38] Zhang JX, Wang JC, Han L, et al. Epidemiology, quality, and reporting characteristics of systematic reviews and meta-analyses of nursing interventions published in Chinese journals. Nurs Outlook. 2015; 63(4):446-455 e4. doi: 10.1016/j.outlook.2014.11.020.

[39] Zhang Y, Yu DD, Cui DH, et al. Report quality evaluation of systematic review or Meta-analysis published in China Journal of Chinese Materia Medica. China Journal of Chinese Materia Medica. 2018; 43(6):1254-1260. doi: 10.19540/j.cnki.cjcmm.20171220.001.

[40] Cao LJ, Yao L, Hui X, et al. Clinical Epidemiology in China series. Paper 3: The methodological and reporting quality of systematic reviews and meta-analyses published by China researchers in English-language is higher than those published in Chinese-language. J Clin Epidemiol. 2021; 140:178-188. doi: 10.1016/j.jclinepi.2021.08.014.

[41] Institute of Medicine Committee on Standards for Developing Trustworthy Clinical Practice Guidelines. Clinical Practice Guidelines We Can Trust. WashingtonDC: National Academies Press; 2011.

[42] Kwong J, Chen H, Sun X. Development of evidence-based recommendations: implications for preparing expert consensus statements. Chin Med J. 2016; 129:2998-3000. doi: 10.4103/0366-6999.195475.

[43] 43.U.S. Preventive Services Task Force. USPSTF Home. [cited 2023 Oct 1]. Available from: https://www.uspreventiveservicestaskforce.org/uspstf/.

[44] The Royal Australian College of General Practitioners. Guidelines For Preventive Activities in General Practice. East Melbourne: Vic; 2018.

[45] Andrews JC, Schünemann HJ, Oxman AD, et al. GRADE guidelines: 15. Going from evidence to recommendation —determinants of a recommendation’s direction and strength. J Clin Epidemiol. 2013; 66(7):726-735. doi: 10.1016/j.jclinepi.2013.02.003.

[46] JAMA Evidence. Decision making and the patient. [2023 Oct 1]. Available from: https://jamaevidence.mhmedical.com/content.aspx?bookid=847&sectionId=69031507.

[47] Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008; 336(7650):924-926. doi: 10.1136/bmj.39489.470347.AD.

[48] Guidelines Review Committee. Quality Assurance of Norms and Standards. 2nd Edition. WHO handbook for guideline development; 2014. Available from: https://www.who.int/publications-detail-redirect/9789241548960

[49] GIN-McMaster Guideline Development Checklist. [2023 Oct 1]. Available from: https://macgrade.mcmaster.ca/resources/ginmcmaster-guideline-development-checklist/.

[50] Ebell MH, Siwek J, Weiss BD, et al. Strength of recommendation taxonomy (SORT): a patient-centered approach to grading evidence in the medical literature. J Am Board Fam Pract. 2004; 17(1):59-67. doi: 10.3122/jabfm.17.1.59.

[51] Wright J. Introducing STEPS. Am Fam Physician. 2003; 68(8):1467.

[52] Yang N, Zhao W, Pan Y, et al. Development of a rating tool for the scientificity, transparency, and applicability of clinical practice guidelines. (Chinese). Chin Med J. 2022; 102(30):2329-2337. doi: 10.3760/cma.j.cn112137-20220219-00340.