Development and empirical validation of a competency-based evaluation index system for standardized residency training

Chang You , Yan Shen

Intelligent Pharmacy ›› 2026, Vol. 4 ›› Issue (3) : 245 -250.

PDF (3635KB)
Intelligent Pharmacy ›› 2026, Vol. 4 ›› Issue (3) :245 -250. DOI: 10.1016/j.ipha.2026.04.004
Full length article
Development and empirical validation of a competency-based evaluation index system for standardized residency training
Author information +
History +
PDF (3635KB)

Abstract

Background: Standardized residency training (SRT) is a core component of postgraduate medical education, and its quality is closely linked to residents' clinical performance. Currently, research on residents' post competency in China is limited, with a lack of comprehensive evaluation standards.

Objectives: To establish a competency-based evaluation index system for SRT and verify its effectiveness in predicting training outcomes.

Methods: This study integrated literature analysis, expert survey, and analytic hierarchy process (AHP) to construct the evaluation index system. AHP was used for weight assignment, with MATLAB software calculating indicator weights. Questionnaire surveys were conducted among residents from four departments for empirical validation, and t-test (SPSS 23.0) was used for statistical analysis.

Results: The index system included 5 first-level and 24 second-level indicators. The consistency ratio of AHP was 0.0201 (<0.1), confirming reasonable weight assignment, with "Competency improvement" having the highest weight (0.3852). T-test results showed no statistically significant differences between predicted and actual scores(all P > 0.05), and expert satisfaction was above 9.0.

Conclusions: The constructed index system is scientific and practical, effectively predicting SRT outcomes, and can provide a basis for optimizing residency training models.

Graphical abstract

Keywords

Resident / Standardized residency training Post competency / Evaluation index system / Analytic hierarchy process

Cite this article

Download citation ▾
Chang You, Yan Shen. Development and empirical validation of a competency-based evaluation index system for standardized residency training. Intelligent Pharmacy, 2026, 4 (3) : 245-250 DOI:10.1016/j.ipha.2026.04.004

登录浏览全文

4963

注册一个新账户 忘记密码

1 Introduction

Standardized Residency Training (SRT) is a key component of postgraduate medical education.1 During SRT, residents are not only required to perform a series of medical and health operations assigned by attending physicians but also responsible for the daily services of healthcare institutions and the routine treatment of patients.2 Conse-quently, establishing an accurate, appropriate, and efficient evaluation index system to predict the effectiveness of residents' SRT is crucial. Current medical talent training models struggle to meet the public's demand for high-quality medical services, and the structure of medical talent training in China urgently needs optimization3,4. With the increasing national attention to medical education, the quality of SRT has become closely linked to residents' subsequent clinical diagnostic performance. At present, research on residents' post competency in China is relatively limited, and there is a lack of comprehensive and innovative evaluation standards for residents' post competency. Therefore, this study intends to conduct a detailed analysis of the resident profession, integrate a series of relevant methods, and establish a competency-based SRT model and its evaluation index system. By exploring this training model and constructing the evaluation index system, we can analyze existing training models, gain in-depth insights into the core competencies required for residents' positions, and provide a basis for the improvement and optimization of future training models.

2 Materials and methods

2.1 Construction of the competency-based evaluation index system for residents in SRT

With the continuous development of the medical industry and the growing social demand for medical resources, increasing attention has been paid to the training and career development of residents5. As an important part of hospitals, residents play a vital role in the treatment and care of patients. To improve residents' professional capabilities and promote their career development, it is essential to develop effective training models and evaluation index systems6. Post competency, as a comprehensive evaluation, aims to measure an individual's knowledge, skills, attitudes, and other relevant traits in their specific job role7. For residents, constructing a complete comprehensive evaluation system is particularly important, as it not only relates to the personal professional growth of physicians but also directly affects the quality of medical services and patient safety8. SRT refers to training conducted in accordance with certain standards and norms, with the goal of enabling residents to achieve a certain professional level and work ability upon completion. Such training often involves a series of complex educational and skill training processes, including theoretical learning, practical operation, and clinical practice9,10. Considering that traditional post competency evaluation index systems may fail to comprehensively, reasonably, truly, and effectively assess the effectiveness of SRT for residents, this study introduced the six core competencies for residents proposed by the Accreditation Council for Graduate Medical Education(ACGME) and the consensus on the core competency framework for residents from the Elite Teaching Hospital Alliance for Resident Training in China, and combined them with the actual management of SRT to establish a new evaluation index system for residents in SRT. Secondly, experts from relevant fields such as medical education specialists, clinicians, and administrators were invited to collect their opinions and suggestions on residents' competency evaluation indicators through a questionnaire survey. Finally, integrating expert suggestions with the core competency content of residents, the evaluation index system for residents' post competency as shown in Table 1 was designed.

Table 1 presents the evaluation index system for residents' post competency constructed based on actual conditions. The system includes five first-level indicators: Completion of training manual, Training workload, Training quality, Daily management, and Competency improvement. Each first-level indicator is accompanied by detailed second-level indicators and their corresponding scores.

To construct a more accurate and comprehensive evaluation tool for residents' post competency, in addition to establishing the corresponding post competency evaluation index system, it is necessary to assign weights to each specific indicator in the system. First, the Analytic Hierarchy Process (AHP) was adopted to calculate the weights of each indicator in the system. This method decomposes factors related to decision-making into levels such as goals, criteria, and alternatives, and conducts qualitative and quantitative analysis on this basis. Although this method mainly relies on experts' personal judgments, it remains a mature method for determining indicator weights in the current research stage. AHP was used to assign values to each indicator in the aforementioned Delphi process and analyze their importance, thereby obtaining the mean importance of each indicator. Assuming the importance of any two indicators is respectively, the Saaty scale was calculated using the difference between the two indicators.

(1)aix−aiy>1.75  Saaty=91.25<aix−aiy≤1.50  Saaty=70.75<aix−aiy≤1.00  Saaty=50.25<aix−aiy≤0.50  Saaty=3

Equation (1) presents the specific values of the Saaty scale. When the scale is 9, the difference between the two indicators is greater than 1.75. When the scale is 7, the difference between the two indicators ranges from 1.25 to 1.50. When the scale is 5, the difference ranges from 0.75 to 1.00. When the scale is 3, the difference ranges from 0.25 to 0.50. For values falling in other ranges, the remaining four even numbers (2 to 8) were assigned in descending order to construct the corresponding judgment matrix.

(2)O=a11a12…a1na21a22…a2n…………an1an2…ann

Equation (2) represents the judgment matrix. The expert consultation results obtained above were filled into the judgment matrix and normalized, resulting in the expression shown in Equation (3) after normalization.

(3)bix=aix∑i=1naix

Equation (3) is the new expression obtained after normalization, where represents a constant. Each row in the matrix was calculated as shown in Equation (4).

(4)ω¯i=∑x=1nbix

Equation (4) is the formula for summing each row of the matrix. The sum of each row was normalized as shown in Equation (5).

(5)ω¯=[ω¯1,ω¯2,⋅⋅⋅ω¯n]r

Equation (5) is the formula for the normalized sum, where represents the normalized sum. On this basis, the expression of the corresponding weight vector can be obtained as shown in Equation (6).

(6)ω¯i=ω¯1∑i=1nω¯1

Equation (6) is the vector expression of weights. The maximum eigenvalue of the judgment matrix was calculated according to Equation(6).

(7)λ max=∑i=1nOωinωi

In Equation (7), is the maximum eigenvalue, and represents the -th component of the vector matrix. In addition, consistency testing of each indicator in the matrix was required to ensure that each element in the matrix meets the requirements.

(8)C.I.=λ max−nn−1

Equation (8) is the consistency test formula, where is the consistency index. When, it indicates complete consistency; when, it indicates basic consistency. A consistency ratio was introduced to simplify the consistency judgment process.

(9)C.R.=C.I.R.I.

In Equation (9), is the consistency ratio, and is the random consistency index, whose value varies with the order of the matrix.

2.2 Statistical data processing methods

Data related to residents' post competency, including but not limited to clinical skills, medical knowledge, and communication skills, were collected through questionnaire surveys, interviews, observations, and records. SPSS 23.0 software was used for the analysis of all data in the study. A t-test was employed to analyze the predicted scores and actual scores of indicators at all levels to verify the rationality of the weight setting of system. A p-value less than 0.05 indicated a statistically significant difference.

3 Results

3.1 Analysis of weight values of first-level indicators

Using the formulas in Section 2.1 and the final residents' competency evaluation index system presented in Section 2.1, MATLAB software was used to calculate the maximum eigenvalue of the judgment matrix as 7.1595, the consistency index as 0.0265, the random consistency index as 1.31, and the consistency ratio as 0.0201. Since is less than 0.1, it indicates that AHP achieved a good consistency result. Thus, the weight assignment is reasonable, and the weight ratios of each first-level and second-level indicator were calculated.

Table 2 shows the judgment matrix and weight values of the five first-level indicators. It can be seen that among the five first-level indicators(O1: Completion of training manual; O2: Training workload; O3: Training quality; O4: Daily management; O5: Competency improvement), Competency improvement has the highest weight ratio of 0.3852. The weight ratios of Completion of training manual, Training workload, Training quality, and Daily management are 0.1563, 0.1478,0.2153, and 0.0954, respectively.

3.2 Analysis of weight values of second-level indicators

Following the calculation method for the weights of first-level indicators, the weight values of the second-level indicators corresponding to each first-level indicator were obtained in sequence. Fig. 1 shows the distribution of weight values of each second-level indicator under the five first-level indicators.

Fig. 1 presents the distribution of weight values of each second-level indicator under the five first-level indicators. As shown in Fig. 1, O1 includes two second-level indicators: O1-1 (Completion of required disease types) and O1-2 (Completion of skill operations), with weight scores of 0.42 and 0.58, respectively. O2 consists of four second-level indicators: O2-1 (Number of teaching ward rounds), O2-2 (Number of training lectures), O2-3 (Number of case discussion meetings), and O2-4 (Number of skill training sessions), with weight scores of 0.31, 0.24, 0.16, and 0.29, respectively. O3 also includes four second-level indicators: O3-1 (Out-of-department examination), O3-2 (Regular evaluation), O3-3 (Annual assessment), and O3-4 (Graduation assessment), with weight scores of 0.21, 0.36, 0.19, and 0.24, respectively. O4 comprises seven second-level indicators: O4-1 (Attendance record), O4-2 (Timeliness of department admission reporting), O4-3 (Failure to rotate as planned), O4-4 (Delayed completion of APP filling), O4-5 (Completion of teaching evaluation), O4-6 (Award records), and O4-7 (Penalty records), with weight scores of 0.21, 0.08, 0.12, 0.06, 0.32, 0.06, and 0.15, respectively. O5 includes seven second-level indicators: O5-1 (Clinical research projects), O5-2 (Clinical papers), O5-3 (Clinical research achievements), O5-4 (Organizing/hosting teaching activities), O5-5 (Completion of teaching assignments), O5-6 (Participation in educational administration activities), and O5-7 (Participation in various competitions), with weight scores of 0.05, 0.13, 0.21, 0.26, 0.08, 0.10, and 0.17, respectively. In summary, the top three indicators with the highest weight values are Completion of skill operations, Number of skill training sessions, and Completion of teaching evaluation. Next, the reliability of the proposed indicator evaluation system was analyzed.

3.3 Comparison between predicted scores and actual scores of indicators at all levels

To verify that the constructed evaluation index system can effectively assess residents' abilities, a questionnaire titled "Residents' Post Competency Evaluation Indicators" was designed based on the evaluation index system. The questionnaire was distributed to residents in four departments of our hospital, including Internal Medicine, Surgery, Pediatrics, and Obstetrics & Gynecology, and the survey results were collected. The recovered questionnaires were statistically analyzed to examine the gap between the actual weight scores and predicted weight scores of each indicator by residents, thereby obtaining the comparison between predicted scores and actual scores of each indicator in practical application.

Fig. 2 shows the comparison of predicted weights, predicted scores, and actual scores of each first-level indicator. As indicated in Fig. 2, the predicted scores under the corresponding weights of the five first-level indicators are 3, 2.8, 6.3, 1.8, and 3.8, respectively, while the actual scores obtained from the recovered questionnaires are 2.8, 2.5, 6.5, 1.9, and 4, respectively.

Fig. 3 presents the score distribution of the four second-level indicators under first-level indicator O2. As shown in Fig. 3, the predicted scores of the four second-level indicators under O2 are 0.868, 0.672,0.448, and 0.812, respectively, and the actual scores are 0.859, 0.652,0.461, and 0.801, respectively.

Fig. 4 shows the score distribution of the four second-level indicators under first-level indicator O3. As indicated in Fig. 4, the predicted scores of the four second-level indicators under O3 are 1.32, 2.26, 1.21, and 1.51, respectively, and the actual scores are 1.30, 2.29, 1.19, and 1.48, respectively.

Fig. 5 presents the score distribution of the seven second-level indicators under first-level indicator O4. As shown in Fig. 5, the predicted scores of the seven second-level indicators under O4 are 0.378, 0.144,0.216, 0.108, 0.576, 0.108, and 0.27, respectively, and the actual scores are 0.375, 0.149, 0.210, 0.100, 0.571, 0.098, and 0.240, respectively.

Fig. 6 shows the score distribution of the seven second-level indicators under first-level indicator O5. As indicated in Fig. 6, the predicted scores of the seven second-level indicators under O5 are 0.190, 0.494, 0.798, 0.988, 0.304, 0.380, and 0.646, respectively, and the actual scores are 0.21, 0.490, 0.791, 1.000, 0.310, 0.350, and 0.650, respectively.

3.4 Statistical analysis of score data of indicators at all levels

A t-test was used to analyze whether the predicted scores and actual scores of indicators at all levels meet the requirements of statistical significance, thereby verifying the rationality of the weight setting of the system. Table 3 shows the t-test results of each first-level indicator.

Table 3 presents the t-test results of the five first-level indicators. The results indicate that there are no statistically significant differences between the predicted values and actual values of the five first-level indicators. This demonstrates that the designed indicator system can accurately predict the scores of each indicator for residents and has good practical application effects.

3.5 Expert satisfaction and prediction accuracy of the evaluation index system

The designed indicator system was used to comprehensively evaluate the competency of residents in four departments of our hospital (Internal Medicine, Surgery, Pediatrics, and Obstetrics & Gynecology). The results of expert satisfaction and prediction accuracy of the evaluation index system are shown in Fig. 7.

Fig. 7 presents the expert satisfaction and prediction accuracy of the competency evaluation index system applied in Internal Medicine, Surgery, Pediatrics, and Obstetrics & Gynecology. Assuming a full score of 10 for satisfaction, Fig. 7(a) shows that the expert satisfaction with the application of the competency evaluation index system in Internal Medicine, Surgery, Pediatrics, and Obstetrics & Gynecology is 9.26,9.53, 9.64, and 9.58, respectively. Fig. 7(b) indicates that the prediction accuracy of the designed competency evaluation index system for residents' competency in Internal Medicine, Surgery, Pediatrics, and Obstetrics & Gynecology is 9.51, 9.34, 9.35, and 9.62, respectively. In summary, the competency evaluation index system designed in this study has high expert satisfaction and practical prediction effects.

4 Discussion

The competency-based SRT model aims to cultivate residents' abilities and skills in specific fields to ensure they can be competent in relevant specialized medical work. In this training model, the construction of an evaluation index system plays a key role, as it can objectively assess residents' comprehensive abilities and training out-comes. Regarding the construction of the evaluation index system for residents' SRT, this study integrated the six core competencies for residents proposed by the Accreditation Council for Graduate Medical Education(ACGME) and the consensus on the core competency framework for residents from the Elite Teaching Hospital Alliance for Resident Training in China, forming a comprehensive evaluation index system. The system covers five first-level indicators, including completion of training manual, training workload, training quality, daily management, and competency improvement, and designs corresponding second-level indicators and scoring standards for each first-level indicator. The designed comprehensive evaluation system can comprehensively assess various aspects of residents' SRT and provide guidance for the refined management of training. The results analysis shows that through weight calculation, the relative importance of each first-level and second-level indicator was determined. For example, Competency improvement has the highest weight among all first-level indicators, indicating its significant impact on the evaluation of residents' competency. In addition, the comparison between predicted scores and actual scores, supported by statistical analysis, shows no significant differences, indicating that the designed evaluation index system can effectively predict residents' actual ability performance. Furthermore, experts have a high satisfaction with the evaluation index system, which further verifies its effectiveness and practicality. In summary, the competency-based SRT model and evaluation index system have been constructed and validated in this study. This model and index system can comprehensively assess various aspects of residents' SRT and provide a scientific basis for the improvement and optimization of training programs. However, this study still has the following limitations: First, only four departments(Internal Medicine, Surgery, Pediatrics, and Obstetrics & Gynecology) were selected as the research objects, resulting in a relatively small sample size. Future research can expand the sample scope to include more departments and medical institutions to enhance the representativeness and generalizability of the research results. Second, although a competency-based evaluation index system has been constructed, it still needs further optimization and improvement. Future research can improve the accuracy and comprehensiveness of the evaluation index system by adding relevant indicators or adjusting weight assignments. Finally, this study mainly focuses on residents' SRT and career development but does not involve long-term follow-up research on their career development. Future research can conduct long-term follow-up of residents' performance and development in their careers to evaluate the long-term effects of the training model and evaluation index system.

5 Conclusions

This study successfully constructs a competency-based evaluation index system for SRT by integrating literature analysis, expert survey, and AHP. The system is scientifically designed with reasonable weight assignment and good consistency. Empirical validation confirms that the system can effectively predict training outcomes, with no statistically significant differences between predicted and actual scores, and high expert satisfaction. The findings provide a reliable tool for optimizing SRT management and improving the quality of medical talent training. Future research should expand the sample scope and conduct long-term follow-up to enhance the system's generalizability and long-term effectiveness.

References

[1]

Xie WF, Zhang QL, Zhai HW, et al. Exploration on the construction of a competency model for surgical teaching faculty in standardized residency training. Chin J Postgraduate Medical Educat. 2019;3(6):486–491 (In Chinese).

[2]

Zeng QQ, Wang JL, Liu J, et al. Analysis of self-assessment results of residents' post competency. Chin J Postgraduate Medical Educat. 2021;41(8):744–748 (In Chinese).

[3]

Meng K, Zhou Y, Chen S, et al. Construction of a competency model for clinical nursing specialists in critical care medicine. Chinese J Modern Nursing. 2019;25(19):2400–2404 (In Chinese).

[4]

Han JF, Chen SQ, Wang SX, et al. Application of an emergency post competency model in the allocation of nursing human resources for COVID-19 prevention and control. Chinese Nursing Manag. 2021;21(1):46–48 (In Chinese).

[5]

Yang L, Zhang JH, Xu BB, et al. Construction of a competency model for primary elderly caregivers from the perspective of medical-nursing integration. J Cent South Univ. 2018;43(6):679–684 (In Chinese).

[6]

Liu M, Zhang YZ, Cui XE, et al. Analysis and discussion on teaching practice activities of standardized residency training in radiology. Chin J Medical Education Res. 2022;21(1):74–77 (In Chinese).

[7]

Xu GW, Xu SB, Xu MQ, et al. Application of formative assessment in standardized residency training in thoracic surgery. Chin J General Practice. 2020;18(2):307–310(In Chinese).

[8]

Ding M, Luo Q, Sun XL, et al. Quality control system for standardized residency training. Hospital Administration J Chin People's Liberation Army. 2021;28(7):686–689(In Chinese).

[9]

Minichiello V, Hayer S, Gillespie B, et al. Developing a mindfulness skills-based training program for resident physicians. Fam Med. 2020;52(1):48–52.

[10]

Witt CM, Helmer SM, Schofield P, et al. Training oncology physicians to advise their patients on complementary and integrative medicine: an implementation study for a manual-guided consultation. Cancer. 2020;126(13):3031–3041.

Rights & permissions

2026 The Authors. Publishing services by Elsevier B.V. on behalf of Higher Education Press and KeAi Communications Co. Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

PDF (3635KB)

0

Accesses

0

Citation

Detail

Sections
Recommended

/

〈 〉