1 Introduction
Critical care medicine has developed rapidly and become an indispensable medical specialty, especially in the COVID-19 pandemic, in which critical care medicine has played a pivotal role. Compared with other disciplines, the incidence of adverse events is higher in critical care medicine. This difference is related to the complex conditions of critically ill patients, who usually have impaired immune function or other complications. In addition, adverse events are influenced by limited treatment times, heavy medical staff workloads, and multiple invasive operations [
1]. To prevent mistakes and improve the treatment of patients, many countries have established a quality control system for critical care medicine to improve the quality of medical care in intensive care units (ICUs) [
2–
5]. Most of the quality control indicators of ICUs have been proposed from a three-dimensional perspective (i.e., focusing on structure-process-result indicators) [
2–
5]. There are currently 60–70 ICU quality control (ICUQC) indicators in different countries [
6]. In 2015, the China National Critical Care Quality Control Center (China-NCCQC) proposed 15 specific national clinical quality control indicators. Our team conducted the China ICU Quality Improvement National Action [
7] and showed that these 15 quality control indicators were an effective tool to improve medical care quality. However, the ICU is a high-investment, high-risk department and thus has limited resources for improving medical care quality, prompting questions about the internal relationship among indicators. For example, are all 15 indicators equally important in determining ICU quality? What are the key factors for improving ICU quality? How can the existing ICUQC indicator system be optimized? Exploratory factor analysis (EFA) is a method to determine the essential structure underlying multiple observable variables and reduce the dimensionality of data. EFA can condense variables with complex relationships into a few core factors [
8]. In this study, we applied EFA to explore the core elements underlying the 15 national quality control indicators of critical care medicine and screened for key quality control factors that affected the prognosis of ICU patients in Chinese mainland. Finally, we sought to establish a quality scoring system for critical care medicine based on these factors. We present the following article in accordance with the Standards for Quality Improvement Reporting Excellence (SQUIRE) 2.0 guidelines.
2 Materials and methods
2.1 Process of establishing an ICUQC system
At the beginning of 2010, various provinces, municipalities, and autonomous regions in China successively established provincial-level quality control centers for critical care medicine. The China-NCCQC was established on October 18, 2012. After several rounds of expert meetings, the Delphi method was used to establish a quality control index for critical care medicine in China. At the end of 2015, the China-NCCQC established an initial national-provincial quality control network for critical care medicine (except in Hong Kong, Macao and Taiwan). To popularize and promote these criteria (published in the “National Clinical Quality Control Indicators for Critical Care Medicine (2015 Edition)”), sampling surveys of medical care quality data were carried out; critical medical care quality surveys began in 2015.
2.2 Definitions of quality control indicators
The Chinese Society of Critical Care Medicine (CSCCM) was established in 2005 and issued guidelines on the construction and management of ICUs in China in 2006. These guidelines cover the basic requirements, scale, staffing, professional requirements, medical management principles, ward construction standards, and equipment for ICUs. ICUs are usually established according to these guidelines in Chinese mainland. In 2015, the China-NCCQC released 15 specific national clinical quality control indicators for critical care medicine. These indicators can be classified into three categories: structural indicators (the percentage of ICU patients among all inpatients, the percentage of ICU bed occupancy of the total inpatient bed occupancy, and the percentage of patients with Acute Physiology and Chronic Health Evaluation II (APACHE II) scores ≥ 15 among all ICU patients), procedural indicators (three-hour Surviving Sepsis Campaign (SSC) bundle compliance, six-hour SSC bundle compliance [
9,
10]), the rate of microbe detection before administration of antibiotics, the percentage of ICU patients receiving prophylaxis for deep-vein thrombosis (DVT) [
11,
12], the percentage of unplanned endotracheal extubations, the percentage of extubated patients reintubated within 48 h, the percentage of patients with unplanned ICU transfers, the 48-h ICU readmission rate, the incidence of ventilator-associated pneumonia (VAP)[
13], the incidence of catheter-related bloodstream infection (CRSI) [
14], and the incidence of catheter-associated urinary tract infections (CAUTIs) [
14,
15]), and outcome indicators (ICU mortality).
2.3 Administration of the survey
The quality control data for critical care medicine are recorded by the government system. A given ICU that has received a China-NCCQC notification can log into the system through state administrative means. The form collects specific data on the denominator and numerator for each indicator according to its definition. This collection process does not include case-by-case data but rather the aggregate data submitted by a hospital. Then, the information is uploaded to the China-NCCQC after being reviewed and approved by the Medical Administration Department of the hospital. After collecting the data, the China-NCCQC conducts data cleaning and analysis and compiles a white paper on the annual quality control and management of critical care medicine. The Institutional Review Board of Peking Union Medical Cooeleg Hospital approved the present survey (No. SK-1828).
This study had a retrospective design and analyzed all data from a quality control survey of critical care medicine collected from 2015 to 2020. Data from hospitals with less than 50% integrity were removed, and outliers were identified and excluded using the interquartile range. In China, hospitals are graded according to a three-tier system that incorporates the hospital’s ability to provide medical care and medical education and to conduct medical research. Therefore, hospitals can be classified as primary, secondary or tertiary institutions. Secondary hospitals are often located in medium-sized cities, counties or regions and can accommodate 100−500 inpatients. Secondary hospitals are responsible for providing comprehensive health services in a given county. Tertiary hospitals are general hospitals at the city, provincial or national level that can accommodate more than 500 inpatients. They are responsible for providing higher-level medical services and play a greater role in medical education and scientific research. In addition, tertiary hospitals act as medical hubs, providing care to multiple areas. We included secondary hospitals and tertiary hospitals in this study.
2.4 Statistical analysis
We needed to use classifications of measurable ICU quality indicators to assess otherwise unmeasurable indicators and calculate the total F score; by doing so, we were able to simplify the quality control system and use it to horizontally compare the results of this research with the reality of quality control in ICUs in clinical practice. Therefore, EFA was performed to simplify and evaluate the stability of China’s ICUQC system [
16]. An extension of the Wilcoxon rank-sum test and generalized estimating equations (GEEs) were applied to assess trends in quality control indicators from 2015 to 2020; a significant (
P < 0.05) increasing trend was observed [
17]. EFA was performed using varimax rotations on the 15 quality control indicators from each ICU [
8,
18]. The Kaiser–Meyer–Olkin (KMO) test for sampling adequacy was employed to assess partial correlations between variables. The Bartlett test of sphericity was used to assess whether the quality control indicators were suitable for factor analysis. We determined the number of factors based on factor eigenvalues greater than 1. We selected variables with factor loadings greater than 0.4 and used the resulting factors to estimate internal consistency [
19]. Missing values were handled with the listwise deletion method. The level for statistical significance was set at
P < 0.05. We used STATA 15.1 (StataCorp) for all statistical analyses.
3 Results
3.1 Trend analysis of single indicators: six-year trend and improvements
The data for this study were obtained from 3425 hospitals (including tertiary and secondary hospitals throughout Chinese mainland) and covered 2 110 685 ICU patients, for a total of 27 607 376 ICU hospitalization days. We conducted a rank trend test on the 15 ICU quanlity control indicators from 2015 to 2020 and found that the median value on most indicators increased significantly (P < 0.05). In contrast, the median values on 3 indicators (the admission rate of ICU patients (P = 0.224), the rate of patients reintubated within 48 h (P = 0.061), and the rate of ICU readmission within 48 h (P = 0.1) showed no significant trend of improvement. The detalied of six-year improvement trend analysis of the 15 indicators of ICU quanlity control are provided in Tab.1. The fitting results of a Generalized Estimating Equation (GEE) are presented in Tab.2. Over the six years of data analyzed, the quality control system has gradually improved and expanded. Among them, the most obvious improvement indicators are as follows: three-hour SSC bundle compliance (%), rate of microbe detection before antibiotics (%), percentage of DVT prophylaxis (%), 48-h ICU readmission rate (%), CRBSI (per 1000 Catheter Days) and CAUTI (per 1000 Catheter Days). Indeed, the number and reporting rate of ICUs increased annually and gradually plateaued, from 665 ICUs in 2014 to 3000+ ICUs after 2018; this expansion demonstrates the feasibility of the scoring system (Fig.1).
3.2 EFA
In light of the continuous improvements in data from 2015 to 2020, we used the quality control data from 2018 to ensure the stability and authenticity of subsequent analyses. The 2018 data set met the requirements for EFA based on the KMO parameter (KMO = 0.622) and Bartlett’s test for sphericity (χ2 = 4434.094, degrees of freedom = 120, P < 0.00001). An initial EFA was conducted on the 15 ICUQC indicators; subsequently, we eliminated the “rate of microbe detection before administering antimicrobial treatment”. The commonality of the data (the sum of the squares of the factor loadings on all factors of one indicator, representing the amount of variation in that indicator explained by all factors) was lower than 0.4. Next, we performed a second EFA on the 14 ICUQC indicators. A 6-factor solution with a factor eigenvalues > 1.0 fit the data best and explained 64.3% of the variance. A scree plot is shown in Fig.2; this plot illustrates that 6 factors (core quality control elements) were sufficient to reduce the dimensionality of the 14 quality control indicators. Tab.3 shows the variance explained and weights of each element. The varimax-rotated factors and factor loadings are provided in Tab.4, as well as the stability of the EFA model. Based on the indicator loadings on each factor in Tab.4, we deduced 6 core quality control elements. VAP, CRBSI, and CAUTI incidence constitute Element 1, representing nosocomial infection management; three-hour SSC bundle compliance and six-hour SSC bundle compliance constitute element 2, representing compliance with the guidelines for the SSC campaign; the percentage of ICU patients out of all inpatients and the percentage of ICU bed occupancy out of total inpatient bed occupancy constitute element 3, representing ICU resources; the percentage of unplanned endotracheal extubations and the percentage of patients reintubated within 48 h constitute element 4, representing airway management; the percentage of drug DVT prophylaxis and the percentage of mechanical DVT prophylaxis constitute element 5, representing DVT prevention; and the percentage of all ICU inpatients with an APACHE II score ≥ 15, unplanned transfers to the ICU, and the rate of ICU readmission within 48 h constitute element 6, representing the severity of patient condition (Fig.3).
3.3 Internal consistency
Cronbach’s alpha was computed for the combined 15 ICUQC indicators and for each individual element. Cronbach’s alpha coefficient was 0.6217 for the combined 15 ICUQC indicators. Cronbach’s alpha coefficients for the elements were as follows: nosocomial infection management = 0.869, compliance with SSC guidelines = 0.7653, ICU resources = 0.7166, airway management = 0.8084, DVT prevention = 0.3987, and severity of patient condition = 0.3236.
3.4 Comparison of factor scores among hospital levels
A preliminary model to calculate the total score F of the ICUQC indicators was constructed. First, we standardized the dimension of each variable, determined the percentage values, converted those percentages into a score, and then assigned the score to the corresponding variable. Second, the weights of the elements were calculated according to the matrix of scoring coefficients (Tab.5). For example, the calculation of the weight of nosocomial infection management is as follows: 0.047×0 – 0.107×1 – 0.045×2 + 0.010×3 + 0.009×4 + 0.005×6 + 0.051×7 – 0.024×8 – 0.041×9 – 0.046×10 – 0.033×11 + 0.099×12 + 0.451×13 + 0.435×14 + 0.340×15. Third, we calculated the total score F for each hospital according to the weight of the six elements (the weight column in Tab.2): F = 21.35%×nosocomial infection management + 17.97%×compliance with SSC guidelines + 17.46%×ICU resources + 15.53%×airway management + 14.07%×DVT prevention + 13.61%×severity of patient condition. Finally, we compared these scores with those from the 2019 data set and found that the comprehensive score F for tertiary hospitals was significantly higher than that for secondary hospitals (Fig.4, Tab.6). Overall, we found significant differences between tertiary hospitals and secondary hospitals, with tertiary and secondary comprehensive hospitals scoring higher than tertiary and secondary specialized hospitals (P < 0.05). The total score F was calculated to match the observed data.
4 Discussion
Our study showed that China’s quality control system for critical care medicine has been operating well over a six-year period and that 15 quality control indicators exhibited improvement to varying degrees. EFA showed these indicators could be condensed into six core elements. Calculating these six core elements allowed each ICU scoring separately. Under the current Chinese hospital grading system, this scoring is feasible and accurately reflects the operation of medical institutions and the quality of critical care medicine.
This study examined the stability and feasibility of the quality control system of critical care medicine in China. In the six years since its inception in 2015, we observed dynamic changes, followed by countermeasures for improvement of China’s ICUs and corresponding increases in related quality control indicators. Indeed, because of the standardized development and continuous improvement in ICUs, China’s critical care medical system was able to respond well to sudden public health problems during the COVID-19 pandemic. We further distilled six important elements with EFA, providing focus and direction for future ICU management and adjustment. A scoring system was created and verified according to the rating standards of China’s health system, providing initial confirmation of the stability and reliability of this scoring system. Our research showed that the national clinical quality control indicators for critical care medicine (2015 edition) have been implemented by most hospitals in Chinese mainland and are routinely applied in ICU quality management. Of the 15 indicators, 12 displayed significant improvement over the six-year period, with the exceptions being the percentage of ICU patients out of the total inpatients, the rate of patients reintubated within 48 h, and the rate of ICU readmission within 48 h, which showed no significant changes (Tab.1). This indicator system had good stability over six years of application. In addition, this quality control indicator system objectively reflected the quality of ICUs in China and guided improvements in ICU quality. Our team carried out the first phase of China’s ICU Quality Improvement National Action in 2016−2018. A total of 586 hospitals and 1 587 724 patients were included in this phase, which focused on these 15 national quality indicators through repeated training, continuous experience in clinical applications, regular data sharing, summary analysis, joint identification of quality weaknesses and proposed improvements. Although no impact on ICU mortality was observed at the end of the first phase, indicators such as the VAP incidence rate, DVT prophylaxis rate, and rate of microbe detection before administering antibiotics showed significant improvement [
7].
The first phase of China’s ICU Quality Improvement National Action and the application of national clinical quality control indicators for critical care medicine (2015 edition) over the six years study indicate that although these quality control indicators have good stability and can guide and improve the medical care quality of ICUs, not all indicators are equally important. We found different strengths of correlation between indicators, showing a trend of consistent changes in clinical practice. This trend is a common problem for quality control in critical care medicine worldwide. To date, there is no unified and recognized solution [
2,
3,
6,
20,
21].
Hence, we conducted this study to address these concerns. This is the first study based on clinical applications to identify core elements and weights from a large number of real-world clinical quality indicators. To determine the most critical elements for care quality from the 15 national quality control indicators, we conducted EFA and identified 6 core elements. These 6 core elements had different weights in the quality evaluation. Their contribution to quality (and corresponding weights) were as follows (ranked from high to low): (1) nosocomial infection management, (2) compliance with SSC guidelines, (3) ICU resources, (4) airway management, (5) DVT prevention, and (6) severity of patient condition. These findings have important implications for quality management. First, they provide the necessary basis for further optimization of the quality control system. The EFA results revealed correlations among some of the 15 indicators. Moreover, indicators reflecting the same element can be merged or deleted if necessary; to add quality control items, options other than these 6 core elements can be evaluated and screened. Second, the determination of element weights helps to identify the most important elements in quality management and to allocate resources to achieve the optimal input–output ratio in the process of quality improvement. Third, core elements and indicators contribute different amounts to quality, enabling integration of these 15 indicators according to their weights to scientifically evaluate the quality of an ICU as a whole. This is the first study worldwide to provide a scientific method for the objective and accurate evaluation of the overall quality of ICUs and compare the overall quality among ICUs. Fourth, the weights of the six core elements derived from EFA in this study are highly consistent with clinical practice and quality control in critical care medicine. The classic quality control theory outlined by Donabedian accurately describes the influence of structural indicators and process indicators on result indicators in detail [
22–
24]. This theory remains one of the basic principles followed by countries worldwide during formulation of quality control systems [
25–
27]. In the clinical practice of critical care medicine in China, as in many other countries, it is difficult for doctors to objectively evaluate patient treatment when a patient first enters the ICU. In this study, the massive amount of real-world data from clinical practice in China over six years showed that the severity of patient condition has the lowest weight among the six core elements. Thus, the nosocomial infection management, compliance with SSC guidelines, airway management, and DVT prevention of an ICU have greater weight in determining the level of quality.
In addition to demonstrating the importance of hospital management in ICUQC indicators, we initially verified (through EFA) that the use of these score in hospitals of various levels in China has a good evaluation value and that this scoring system has potential for future use in the quality control of China’s ICUs. We found that the scores of tertiary comprehensive hospitals were higher than those of secondary comprehensive hospitals. This scoring system can therefore be applied in the future for quality control evaluation and for assessing the practical value of the quality control system.
Our study has some limitations. First, we were only able to establish the stability and feasibility of the quality control system by combining the collected data with current data. We will confirm these findings in future quality control research. Second, different regions worldwide utilize different indicators; although EFA may be a good way to condense and compare these indicators, different countries may need to establish their own quality control systems according to other methods to evaluate the quality of their ICU operations. Third, regarding internal consistency, the alpha values of DVT prevention and severity of patient condition were both lower than 0.4. We believe this finding is because these two elements include the indicator of APACHE II score ≥ 15, which is the ratio of ICU patients with a high risk of death. This purely objective index is very different from many other management indicators, but it is related to these management indicators. Therefore, this indicator loads on DVT prevention and severity of patient condition (loadings of 0.51 and 0.48, respectively), but the loadings are not very high, which may explain why the consistency of these two factors is relatively low. After discussion by clinical experts, APACHE II scores are still regarded as important. We decided to retain the APACHE II score without adjustment for the time being and carry out further verification and evaluation later, when the indicator is used for quality management.
This is the study to explore the relationship among quality control indicators using large-scale real-world clinical quality data. This study also serves as an initial analysis of such data to identify core elements that affect ICU quality and their weights from large-scale real-world clinical quality data. This research used the scientific method to propose a new method of identifying the core elements of ICU quality management and optimizing the quality control system. In the future, this approach should be used to assess the management of critical care medicine, establishing a foundation for the gradual improvement of critical care medicine in China.
PDF
(1221KB)
