1 Introduction
Standardized Residency Training (SRT) is a key component of postgraduate medical education.
1 During SRT, residents are not only required to perform a series of medical and health operations assigned by attending physicians but also responsible for the daily services of healthcare institutions and the routine treatment of patients.
2 Conse-quently, establishing an accurate, appropriate, and efficient evaluation index system to predict the effectiveness of residents' SRT is crucial. Current medical talent training models struggle to meet the public's demand for high-quality medical services, and the structure of medical talent training in China urgently needs optimization
3,4. With the increasing national attention to medical education, the quality of SRT has become closely linked to residents' subsequent clinical diagnostic performance. At present, research on residents' post competency in China is relatively limited, and there is a lack of comprehensive and innovative evaluation standards for residents' post competency. Therefore, this study intends to conduct a detailed analysis of the resident profession, integrate a series of relevant methods, and establish a competency-based SRT model and its evaluation index system. By exploring this training model and constructing the evaluation index system, we can analyze existing training models, gain in-depth insights into the core competencies required for residents' positions, and provide a basis for the improvement and optimization of future training models.
2 Materials and methods
2.1 Construction of the competency-based evaluation index system for residents in SRT
With the continuous development of the medical industry and the growing social demand for medical resources, increasing attention has been paid to the training and career development of residents
5. As an important part of hospitals, residents play a vital role in the treatment and care of patients. To improve residents' professional capabilities and promote their career development, it is essential to develop effective training models and evaluation index systems
6. Post competency, as a comprehensive evaluation, aims to measure an individual's knowledge, skills, attitudes, and other relevant traits in their specific job role
7. For residents, constructing a complete comprehensive evaluation system is particularly important, as it not only relates to the personal professional growth of physicians but also directly affects the quality of medical services and patient safety
8. SRT refers to training conducted in accordance with certain standards and norms, with the goal of enabling residents to achieve a certain professional level and work ability upon completion. Such training often involves a series of complex educational and skill training processes, including theoretical learning, practical operation, and clinical practice
9,10. Considering that traditional post competency evaluation index systems may fail to comprehensively, reasonably, truly, and effectively assess the effectiveness of SRT for residents, this study introduced the six core competencies for residents proposed by the Accreditation Council for Graduate Medical Education(ACGME) and the consensus on the core competency framework for residents from the Elite Teaching Hospital Alliance for Resident Training in China, and combined them with the actual management of SRT to establish a new evaluation index system for residents in SRT. Secondly, experts from relevant fields such as medical education specialists, clinicians, and administrators were invited to collect their opinions and suggestions on residents' competency evaluation indicators through a questionnaire survey. Finally, integrating expert suggestions with the core competency content of residents, the evaluation index system for residents' post competency as shown in Table 1 was designed.
Table 1 presents the evaluation index system for residents' post competency constructed based on actual conditions. The system includes five first-level indicators: Completion of training manual, Training workload, Training quality, Daily management, and Competency improvement. Each first-level indicator is accompanied by detailed second-level indicators and their corresponding scores.
To construct a more accurate and comprehensive evaluation tool for residents' post competency, in addition to establishing the corresponding post competency evaluation index system, it is necessary to assign weights to each specific indicator in the system. First, the Analytic Hierarchy Process (AHP) was adopted to calculate the weights of each indicator in the system. This method decomposes factors related to decision-making into levels such as goals, criteria, and alternatives, and conducts qualitative and quantitative analysis on this basis. Although this method mainly relies on experts' personal judgments, it remains a mature method for determining indicator weights in the current research stage. AHP was used to assign values to each indicator in the aforementioned Delphi process and analyze their importance, thereby obtaining the mean importance of each indicator. Assuming the importance of any two indicators is respectively, the Saaty scale was calculated using the difference between the two indicators.
Equation (1) presents the specific values of the Saaty scale. When the scale is 9, the difference between the two indicators is greater than 1.75. When the scale is 7, the difference between the two indicators ranges from 1.25 to 1.50. When the scale is 5, the difference ranges from 0.75 to 1.00. When the scale is 3, the difference ranges from 0.25 to 0.50. For values falling in other ranges, the remaining four even numbers (2 to 8) were assigned in descending order to construct the corresponding judgment matrix.
Equation (2) represents the judgment matrix. The expert consultation results obtained above were filled into the judgment matrix and normalized, resulting in the expression shown in Equation (3) after normalization.
Equation (3) is the new expression obtained after normalization, where represents a constant. Each row in the matrix was calculated as shown in Equation (4).
Equation (4) is the formula for summing each row of the matrix. The sum of each row was normalized as shown in Equation (5).
Equation (5) is the formula for the normalized sum, where represents the normalized sum. On this basis, the expression of the corresponding weight vector can be obtained as shown in Equation (6).
Equation (6) is the vector expression of weights. The maximum eigenvalue of the judgment matrix was calculated according to Equation(6).
In Equation (7), is the maximum eigenvalue, and represents the -th component of the vector matrix. In addition, consistency testing of each indicator in the matrix was required to ensure that each element in the matrix meets the requirements.
Equation (8) is the consistency test formula, where is the consistency index. When, it indicates complete consistency; when, it indicates basic consistency. A consistency ratio was introduced to simplify the consistency judgment process.
In Equation (9), is the consistency ratio, and is the random consistency index, whose value varies with the order of the matrix.
2.2 Statistical data processing methods
Data related to residents' post competency, including but not limited to clinical skills, medical knowledge, and communication skills, were collected through questionnaire surveys, interviews, observations, and records. SPSS 23.0 software was used for the analysis of all data in the study. A t-test was employed to analyze the predicted scores and actual scores of indicators at all levels to verify the rationality of the weight setting of system. A p-value less than 0.05 indicated a statistically significant difference.
3 Results
3.1 Analysis of weight values of first-level indicators
Using the formulas in Section 2.1 and the final residents' competency evaluation index system presented in Section 2.1, MATLAB software was used to calculate the maximum eigenvalue of the judgment matrix as 7.1595, the consistency index as 0.0265, the random consistency index as 1.31, and the consistency ratio as 0.0201. Since is less than 0.1, it indicates that AHP achieved a good consistency result. Thus, the weight assignment is reasonable, and the weight ratios of each first-level and second-level indicator were calculated.
Table 2 shows the judgment matrix and weight values of the five first-level indicators. It can be seen that among the five first-level indicators(O1: Completion of training manual; O2: Training workload; O3: Training quality; O4: Daily management; O5: Competency improvement), Competency improvement has the highest weight ratio of 0.3852. The weight ratios of Completion of training manual, Training workload, Training quality, and Daily management are 0.1563, 0.1478,0.2153, and 0.0954, respectively.
3.2 Analysis of weight values of second-level indicators
Following the calculation method for the weights of first-level indicators, the weight values of the second-level indicators corresponding to each first-level indicator were obtained in sequence. Fig. 1 shows the distribution of weight values of each second-level indicator under the five first-level indicators.
Fig. 1 presents the distribution of weight values of each second-level indicator under the five first-level indicators. As shown in Fig. 1, O1 includes two second-level indicators: O1-1 (Completion of required disease types) and O1-2 (Completion of skill operations), with weight scores of 0.42 and 0.58, respectively. O2 consists of four second-level indicators: O2-1 (Number of teaching ward rounds), O2-2 (Number of training lectures), O2-3 (Number of case discussion meetings), and O2-4 (Number of skill training sessions), with weight scores of 0.31, 0.24, 0.16, and 0.29, respectively. O3 also includes four second-level indicators: O3-1 (Out-of-department examination), O3-2 (Regular evaluation), O3-3 (Annual assessment), and O3-4 (Graduation assessment), with weight scores of 0.21, 0.36, 0.19, and 0.24, respectively. O4 comprises seven second-level indicators: O4-1 (Attendance record), O4-2 (Timeliness of department admission reporting), O4-3 (Failure to rotate as planned), O4-4 (Delayed completion of APP filling), O4-5 (Completion of teaching evaluation), O4-6 (Award records), and O4-7 (Penalty records), with weight scores of 0.21, 0.08, 0.12, 0.06, 0.32, 0.06, and 0.15, respectively. O5 includes seven second-level indicators: O5-1 (Clinical research projects), O5-2 (Clinical papers), O5-3 (Clinical research achievements), O5-4 (Organizing/hosting teaching activities), O5-5 (Completion of teaching assignments), O5-6 (Participation in educational administration activities), and O5-7 (Participation in various competitions), with weight scores of 0.05, 0.13, 0.21, 0.26, 0.08, 0.10, and 0.17, respectively. In summary, the top three indicators with the highest weight values are Completion of skill operations, Number of skill training sessions, and Completion of teaching evaluation. Next, the reliability of the proposed indicator evaluation system was analyzed.
3.3 Comparison between predicted scores and actual scores of indicators at all levels
To verify that the constructed evaluation index system can effectively assess residents' abilities, a questionnaire titled "Residents' Post Competency Evaluation Indicators" was designed based on the evaluation index system. The questionnaire was distributed to residents in four departments of our hospital, including Internal Medicine, Surgery, Pediatrics, and Obstetrics & Gynecology, and the survey results were collected. The recovered questionnaires were statistically analyzed to examine the gap between the actual weight scores and predicted weight scores of each indicator by residents, thereby obtaining the comparison between predicted scores and actual scores of each indicator in practical application.
Fig. 2 shows the comparison of predicted weights, predicted scores, and actual scores of each first-level indicator. As indicated in Fig. 2, the predicted scores under the corresponding weights of the five first-level indicators are 3, 2.8, 6.3, 1.8, and 3.8, respectively, while the actual scores obtained from the recovered questionnaires are 2.8, 2.5, 6.5, 1.9, and 4, respectively.
Fig. 3 presents the score distribution of the four second-level indicators under first-level indicator O2. As shown in Fig. 3, the predicted scores of the four second-level indicators under O2 are 0.868, 0.672,0.448, and 0.812, respectively, and the actual scores are 0.859, 0.652,0.461, and 0.801, respectively.
Fig. 4 shows the score distribution of the four second-level indicators under first-level indicator O3. As indicated in Fig. 4, the predicted scores of the four second-level indicators under O3 are 1.32, 2.26, 1.21, and 1.51, respectively, and the actual scores are 1.30, 2.29, 1.19, and 1.48, respectively.
Fig. 5 presents the score distribution of the seven second-level indicators under first-level indicator O4. As shown in Fig. 5, the predicted scores of the seven second-level indicators under O4 are 0.378, 0.144,0.216, 0.108, 0.576, 0.108, and 0.27, respectively, and the actual scores are 0.375, 0.149, 0.210, 0.100, 0.571, 0.098, and 0.240, respectively.
Fig. 6 shows the score distribution of the seven second-level indicators under first-level indicator O5. As indicated in Fig. 6, the predicted scores of the seven second-level indicators under O5 are 0.190, 0.494, 0.798, 0.988, 0.304, 0.380, and 0.646, respectively, and the actual scores are 0.21, 0.490, 0.791, 1.000, 0.310, 0.350, and 0.650, respectively.
3.4 Statistical analysis of score data of indicators at all levels
A t-test was used to analyze whether the predicted scores and actual scores of indicators at all levels meet the requirements of statistical significance, thereby verifying the rationality of the weight setting of the system. Table 3 shows the t-test results of each first-level indicator.
Table 3 presents the t-test results of the five first-level indicators. The results indicate that there are no statistically significant differences between the predicted values and actual values of the five first-level indicators. This demonstrates that the designed indicator system can accurately predict the scores of each indicator for residents and has good practical application effects.
3.5 Expert satisfaction and prediction accuracy of the evaluation index system
The designed indicator system was used to comprehensively evaluate the competency of residents in four departments of our hospital (Internal Medicine, Surgery, Pediatrics, and Obstetrics & Gynecology). The results of expert satisfaction and prediction accuracy of the evaluation index system are shown in Fig. 7.
Fig. 7 presents the expert satisfaction and prediction accuracy of the competency evaluation index system applied in Internal Medicine, Surgery, Pediatrics, and Obstetrics & Gynecology. Assuming a full score of 10 for satisfaction, Fig. 7(a) shows that the expert satisfaction with the application of the competency evaluation index system in Internal Medicine, Surgery, Pediatrics, and Obstetrics & Gynecology is 9.26,9.53, 9.64, and 9.58, respectively. Fig. 7(b) indicates that the prediction accuracy of the designed competency evaluation index system for residents' competency in Internal Medicine, Surgery, Pediatrics, and Obstetrics & Gynecology is 9.51, 9.34, 9.35, and 9.62, respectively. In summary, the competency evaluation index system designed in this study has high expert satisfaction and practical prediction effects.
4 Discussion
The competency-based SRT model aims to cultivate residents' abilities and skills in specific fields to ensure they can be competent in relevant specialized medical work. In this training model, the construction of an evaluation index system plays a key role, as it can objectively assess residents' comprehensive abilities and training out-comes. Regarding the construction of the evaluation index system for residents' SRT, this study integrated the six core competencies for residents proposed by the Accreditation Council for Graduate Medical Education(ACGME) and the consensus on the core competency framework for residents from the Elite Teaching Hospital Alliance for Resident Training in China, forming a comprehensive evaluation index system. The system covers five first-level indicators, including completion of training manual, training workload, training quality, daily management, and competency improvement, and designs corresponding second-level indicators and scoring standards for each first-level indicator. The designed comprehensive evaluation system can comprehensively assess various aspects of residents' SRT and provide guidance for the refined management of training. The results analysis shows that through weight calculation, the relative importance of each first-level and second-level indicator was determined. For example, Competency improvement has the highest weight among all first-level indicators, indicating its significant impact on the evaluation of residents' competency. In addition, the comparison between predicted scores and actual scores, supported by statistical analysis, shows no significant differences, indicating that the designed evaluation index system can effectively predict residents' actual ability performance. Furthermore, experts have a high satisfaction with the evaluation index system, which further verifies its effectiveness and practicality. In summary, the competency-based SRT model and evaluation index system have been constructed and validated in this study. This model and index system can comprehensively assess various aspects of residents' SRT and provide a scientific basis for the improvement and optimization of training programs. However, this study still has the following limitations: First, only four departments(Internal Medicine, Surgery, Pediatrics, and Obstetrics & Gynecology) were selected as the research objects, resulting in a relatively small sample size. Future research can expand the sample scope to include more departments and medical institutions to enhance the representativeness and generalizability of the research results. Second, although a competency-based evaluation index system has been constructed, it still needs further optimization and improvement. Future research can improve the accuracy and comprehensiveness of the evaluation index system by adding relevant indicators or adjusting weight assignments. Finally, this study mainly focuses on residents' SRT and career development but does not involve long-term follow-up research on their career development. Future research can conduct long-term follow-up of residents' performance and development in their careers to evaluate the long-term effects of the training model and evaluation index system.
5 Conclusions
This study successfully constructs a competency-based evaluation index system for SRT by integrating literature analysis, expert survey, and AHP. The system is scientifically designed with reasonable weight assignment and good consistency. Empirical validation confirms that the system can effectively predict training outcomes, with no statistically significant differences between predicted and actual scores, and high expert satisfaction. The findings provide a reliable tool for optimizing SRT management and improving the quality of medical talent training. Future research should expand the sample scope and conduct long-term follow-up to enhance the system's generalizability and long-term effectiveness.
2026 The Authors. Publishing services by Elsevier B.V. on behalf of Higher Education Press and KeAi Communications Co. Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).