Methods and applications of DEA cross-efficiency: Review and future perspectives

Abstract

The field of engineering management usually involves evaluation issues, such as program selection, team performance evaluation, technology selection, and supplier evaluation. The traditional self-evaluation data envelopment analysis (DEA) method usually exaggerates the effects of several inputs or outputs of the evaluated decision-making unit (DMU), resulting in unrealistic results. To address this problem, scholars have proposed the cross-efficiency evaluation (CREE) method. Compared with the DEA method, CREE can rank DMUs more completely by using reasonable weights. With the extensive application of this technique, several problems, such as non-unique weights and non-Pareto optimal results, have arisen in CREE methods. Therefore, the improvement of CREE has attracted the attention of many scholars. This paper reviews the theory and applications of CREE, including the non-uniqueness problem, the aggregation of cross-efficiency data, and applications in engineering management. It also discusses the directions for future research on CREE.

Key words： cross-efficiency evaluation; efficiency; secondary goal model; aggregation; review

Introduction

Engineering management is a discipline that applies managerial principles to teams, projects, programs, and technologies to add value to the current operations of technology-based organizations (^{Chang, 2008}). With the development of disciplines, engineering management continues to be adjusted, and new concepts are being defined. Therefore, a completely accurate definition of engineering management is impossible to provide. Moreover, different areas of the world have different definitions of “engineering management” (^{Lannes, 2001}). In the field of engineering management, homogeneous decision-making units (DMUs), such as team performance (^{Brannick et al., 1997}), technology (^{Talluri and Paul Yoon, 2000}), project management (^{Li and Lei, 2007}), organizational operational efficiency (^{Andersen and Petersen, 1993}; ^{Sun et al., 2013}), and environmental performance (^{Xu and Li, 2012}), are often necessary to evaluate and rank. For the evaluation of DMUs, various methods have been proposed (^{Zopounidis and Doumpos, 2002}; ^{Sun et al., 2017b}). Among these methods, data envelopment analysis (DEA) has been attracting increasing attention from scholars (^{Cook and Seiford, 2009}; ^{Lim et al., 2014}). DEA provides evaluation results through a linear programming method but does not use subjective factors in weight setting and hence, produces acceptable evaluation results (^{Charnes et al., 1978}; ^{Sun et al., 2017c}). In a set of comparable DMUs, DEA can identify the best-performing DMUs and form an effective frontier. Furthermore, DEA can measure the efficiency levels of nonfrontier units and determine benchmarks against which such inefficient units can be compared (^{Cook and Seiford, 2009}).

Although widely used in performance evaluation, DEA has some deficiencies (^{Wu et al., 2011b}). For example, DEA cannot distinguish efficient DMUs sufficiently because all efficient DMUs receive the same perfect score (^{Sun et al., 2013}). In addition, the weights used to calculate efficiency scores are usually beneficial to the evaluated DMU, i.e., the weights may be extremely unrealistic, resulting in a false impression of DMU efficiency (^{Dyson and Thanassoulis, 1988}; ^{Wong and Beasley, 1990}). To address these problems, scholars have proposed various extended DEA models, of which the cross-efficiency evaluation (CREE) method is the most popular. Based on traditional DEA, CREE was first proposed by ^{Sexton et al. (1986)}. Unlike the traditional self-evaluation mechanism in DEA, CREE combines self- and peer-evaluation mechanisms (^{Anderson et al., 2002}). CREE can not only distinguish DMUs sufficiently (^{Boussofiane et al., 1991}) but also create a fair evaluation atmosphere. In CREE evaluation, if all DMUs jointly evaluate a particular DMU as the most efficient, then it truly is the best-performing DMU. With these advantages, CREE has been applied to performance evaluation and resource allocation in the fields of environment (^{Chen et al., 2017}), manufacturing (^{Shang and Sueyoshi, 1995}), supply chain (^{Yu et al., 2010}), Olympics (^{Wu et al., 2009b}), and transportation and logistics (^{Cui and Li, 2015}).

As the variety of CREE applications widens, several problems have emerged. For instance, the weights of CREE are obtained using traditional DEA, but the latter may have multiple weight solutions; this condition leads to cross-efficiency scores (CESs) being non-unique in general (^{Doyle and Green, 1994}; ^{Wu et al., 2012a}), a problem labeled as the non-uniqueness of weights. Moreover, the traditional CREE method uses an arithmetic average method to aggregate all CESs, meaning that the final results are not Pareto optimal and they may not be accepted by all decision-makers (^{Despotis, 2002}). To solve the two main problems, scholars have made various improvements to CREE and proposed extended models. Figure 1 shows the number of papers related to CREE published annually, revealing that it has been attracting increasing attention from scholars. Table 1 presents the top 10 authors, journals, and institutions, respectively, which have published most papers on CREE. The table indicates that most of the papers about CREE were published in journals for the fields of production operations, operations research, and industrial engineering.

Fig.1 Numbers of papers about CREE published annually (data source: Web of Science Core Collection; search topic: cross-efficiency; categories: management, operations research, and management science).

Full size|PPT slide

Tab.1 Top 10 authors, journals, and institutions with most published papers about CREE

Authors	Journals	Institutions
Wu, Jie	European Journal of Operational Research	Chinese Academy of Sciences, China
Liang, Liang	Computers & Industrial Engineering	University of Science and Technology of China, China
Wang, Ying ming	Journal of the Operational Research Society	Islamic Azad University, Iran
Ruiz, José L.	Expert Systems with Applications	Fuzhou University, China
Sun, Jiasen	Annals of Operations Research	Universidad Miguel Hernández, Spain
Sirvent, Inmaculada	International Journal of Production Research	City University of Hong Kong, China
Chu, Jun fei	Journal of Cleaner Production	Worcester Polytechnic Institute, USA
Yang, Feng	Omega	Soochow University, China
Chin, Kwai-Sang	RAIRO-Operations Research	Hefei University of Technology, China
Zhu, Qingyuan	Applied Mathematical Modeling	Sultan Qaboos University, Oman

Notes: Source: Web of Science Core Collection; Search topic: Cross-efficiency.

The main objective of this study is to review CREE in theory and practice. On the theoretical side, this paper reviews the non-uniqueness of CREE weights and the aggregation of CESs. On the practical side, this paper reviews the application of CREE in the field of engineering management and covers such areas as environment, transportation, manufacturing, and supply chains.

The remainder of this paper is organized as follows. Section 2 introduces the traditional CREE and its shortcomings. Section 3 reviews the CREE secondary goal models, which mainly solve the problem of the non-uniqueness of weights. Section 4 reviews the aggregation methods of CESs. Section 5 covers the applications of CREE. Section 6 presents the conclusions and directions for future research.

Traditional CREE

Assuming n DMUs, each DMU_j (j = 1, 2,..., n) consumes m different resources to produce s different outputs. The input and output vectors are denoted, respectively, as: X_j = (x_1j, x_2j,..., x_mj) and Y_j = (y_1j, y_2j,..., y_sj). For the evaluated DMU_d (d = 1, 2,..., n), the efficiency score can be obtained by linear programming with Model (1).

\max E_{dd}={\displaystyle \sum _{r=\mathrm{1}}^s{U}_{rd}}{y}_{rd}

s.t.\mathrm{\hspace{0.17em}}\mathrm{\hspace{0.17em}}\mathrm{\hspace{0.17em}}\mathrm{\hspace{0.17em}}\mathrm{\hspace{0.17em}}{\displaystyle \sum _{i=\mathrm{1}}^m{W}_{id}}{x}_{ij}-{\displaystyle \sum _{r=\mathrm{1}}^s{U}_{rd}}{y}_{rj}\ge \mathrm{0},\mathrm{}j=\mathrm{1},\mathrm{}\mathrm{2},\mathrm{...},\mathrm{}n,

{\displaystyle \sum _{i=\mathrm{1}}^m{W}_{id}}{x}_{id}=\mathrm{1},

,

W_{id}\ge \mathrm{0},\mathrm{}i=\mathrm{1},\mathrm{}\mathrm{2},\mathrm{...},\mathrm{}m,

(1)U_{rd}\ge \mathrm{0},\mathrm{}r=\mathrm{1},\mathrm{}\mathrm{2},\mathrm{...},\mathrm{}s.

In Model (1), W_id (i = 1, 2,..., m) and U_rd (r = 1, 2,..., s) represent the weights of x_ij and y_rj, respectively. The weight selection of Model (1) must follow the principle of maximizing efficiency E_dd, i.e., the weights are the most favorable for DMU_d to obtain its maximum efficiency score. Model (1) is also known as the self-evaluation DEA model (^{Charnes et al., 1978}). The main disadvantage of Model (1) is the selection of the most advantageous weights of each DMU for itself, thereby causing the efficiency scores of all DMUs to lack comparability (^{Wu et al., 2012a}; ^2016a). Aiming at this problem, ^{Sexton et al. (1986)} proposed a CREE technique based on the multiple use of Model (1). The main idea of CREE is to use a peer-evaluation mechanism to eliminate the drawbacks of the self-evaluation DEA model (^{Wu et al., 2012b}). A set of optimal weights \left(W_{\mathrm{1}d}^{*},\mathrm{...},\mathrm{}{W}_{md}^{*},\mathrm{}{U}_{\mathrm{1}d}^{*},\mathrm{...},\mathrm{}{U}_{sd}^{*}\right) of DMU_d is obtained by solving Model (1). Given the optimal weights of DMU_d, the CES of DMU_j can be obtained using the following equation:

(2)E_{dj}={\displaystyle \frac{{\displaystyle \sum _{r=\mathrm{1}}^s{U}_{rd}^{*}{y}_{rj}}}{{\displaystyle \sum _{i=\mathrm{1}}^m{W}_{id}^{*}{x}_{ij}}}},\mathrm{}d,\mathrm{}j=\mathrm{1},\mathrm{}\mathrm{2},\mathrm{ }\mathrm{ }\mathrm{ }\mathrm{...}\mathrm{ }\mathrm{ },\mathrm{}n.

The CESs of all DMUs can be obtained using Model (1) and Eq. (2). For DMU_j, the efficiency of using CREE is calculated using the following equation:

(3){\bar{E}}_j={\displaystyle \frac{\mathrm{1}}{n}}{\displaystyle \sum _{d=\mathrm{1}}^n{E}_{dj}},\mathrm{}j=\mathrm{1},\mathrm{}\mathrm{2},\mathrm{...},\mathrm{}n.

Three major defects remain in CREE. First, the optimal weights of Model (1) are unnecessarily unique; consequently, the CES obtained using Eq. (2) may be randomly generated (^{Doyle and Green, 1994}). Second, the efficiency obtained using the arithmetic averaging method in Eq. (3) loses the correlation between the weights and CESs; hence, decision-makers cannot obtain valuable information to improve the performance of DMUs (^{Wu et al., 2012b}). Third, the efficiency scores of DMUs obtained using the arithmetic averaging method are not Pareto optimal, making it difficult for several DMUs to accept such evaluation results (^{Wu et al., 2016a}). To solve these problems, scholars have proposed many improved models. The theoretical development of CREE is shown in Fig. 2. The progress of the research on CREE is reviewed in detail in the following sections.

Fig.2 Theoretical development of CREE.

Full size|PPT slide

Secondary goal models of CREE

Aiming at the problem of the non-uniqueness of weights, many secondary goal models have been proposed. These models can be divided into benevolent, aggressive, neutral, weight relaxation, and other strategic models.

Benevolent and aggressive secondary goal models

^{Doyle and Green (1994)} first proposed solving the problem of the non-uniqueness of weights by using secondary goal models, which include the benevolent and aggressive strategic models. In the benevolent strategic model, the optimal weights must maintain the self-evaluation efficiency of the evaluated DMU_d and maximize the average efficiency of the other DMUs simultaneously. In the aggressive strategic model, the optimal weights minimize the average efficiencies of the other DMUs while maintaining the self-evaluation efficiency of the evaluated DMU_d. To solve the nonlinear models of ^{Doyle and Green (1994)}, ^{Liang et al. (2008a)} used slack variables to extend the benevolent and aggressive secondary goal models, of which Model (4) is one.

\min {\displaystyle \sum _{j=\mathrm{1}}^n{\delta }_j^{\text{'}}}

s.t.\mathrm{\hspace{0.17em}}\mathrm{\hspace{0.17em}}\mathrm{\hspace{0.17em}}\mathrm{\hspace{0.17em}}{\displaystyle \sum _{r=\mathrm{1}}^s{U}_{rd}}{y}_{rj}-{\displaystyle \sum _{i=\mathrm{1}}^m{W}_{id}}{x}_{ij}+{\delta }_j^{\text{'}}=\mathrm{0},\mathrm{}j=\mathrm{1},\mathrm{}\mathrm{2},\mathrm{...},\mathrm{}n,

,

{\displaystyle \sum _{i=\mathrm{1}}^m{W}_{id}}{x}_{id}=\mathrm{1},

,

{\displaystyle \sum _{r=\mathrm{1}}^s{U}_{rd}}{y}_{rd}=\mathrm{1}-{\delta }_d^{*},

,

(4)W_{id},\mathrm{}{U}_{rd},\mathrm{}{\delta }_j^{\text{'}}\ge \mathrm{0},\mathrm{for\; all}i,r,j.

In Model (4), {\delta }_d^{*} is the inefficiency of DMU_d, and the objective function attempts to maximize the efficiencies of the other DMUs. ^{Liang et al. (2008a)} also proposed other strategic models and discussed their application scenarios. ^{Wang and Chin (2010a)} argued that the desirable target CES used by ^{Liang et al. (2008a)} was unrealistic because not every DMU could achieve perfect efficiency; hence, they changed the desired target efficiency from perfect efficiency to CCR (named after its developer Chames, Cooper and Rhodes) efficiency. ^{Wu et al. (2016b)} considered CCR efficiency as infeasible for several DMUs. To obtain feasible target efficiency for each DMU, ^{Wu et al. (2016b)} proposed a target-setting model to calculate desirable and undesirable CESs that were achievable. The two target scores were then simultaneously introduced into the secondary goal model. The target-setting model of ^{Wu et al. (2016b)} is expressed as:

{\phi }_{dj}^{\max /\min }=\max /\min {\displaystyle \sum _{r=\mathrm{1}}^s{U}_{rd}}{y}_{rj}

s.t.\mathrm{\hspace{0.17em}}\mathrm{\hspace{0.17em}}\mathrm{\hspace{0.17em}}{\displaystyle \sum _{i=\mathrm{1}}^m{W}_{id}}{x}_{ij}-{\displaystyle \sum _{r=\mathrm{1}}^s{U}_{rd}}{y}_{rj}\ge \mathrm{0},\mathrm{}j=\mathrm{1},\mathrm{}\mathrm{2},\mathrm{...},\mathrm{}n,

{\displaystyle \sum _{i=\mathrm{1}}^m{W}_{id}}{x}_{ij}=\mathrm{1},

\mathrm{ }\mathrm{ }\mathrm{ }{E}_{dd}^{*}\mathrm{ }{\displaystyle \sum _{i=\mathrm{1}}^m{W}_{id}}{x}_{id}-{\displaystyle \sum _{r=\mathrm{1}}^s{U}_{rd}}{y}_{rd}=\mathrm{0},

(5)W_{id},\mathrm{}{U}_{rd}\ge \mathrm{0},\mathrm{for\; all}i,r.

In Model (5), {\phi }_{dj}^{\min } and {\phi }_{dj}^{\max } are the undesirable and desirable target CES for DMU_j, respectively. A similar technique was presented by ^{Chen and Wang (2020)}, who proposed target-setting methods based on a cross-efficient and possible reference set to improve the CES of DMUs.

Replacing the traditional maximizations and minimizations of the average efficiency of other DMUs, ^{Lim (2012b)} used the minimization/maximization function as the objective function and proposed a bisection algorithm for solving a CREE model.

Neutral secondary goal models

^{Wang and Chin (2010b)} argued that the evaluated DMU_d should optimize its own weighted input or output without considering the effects on the efficiencies of the other DMUs, in contrast to the benevolent and aggressive models. Hence, they proposed a neutral secondary goal model to maximize each weighted output of the evaluated DMU_d. Subsequently, ^{Wang et al. (2011a)} extended this model, aiming to maximize each weighted input and output simultaneously. ^{Ramón et al. (2010b)} extended the slack elimination DEA model of ^{Ramón et al. (2010a)} into CREE and proposed a neutral secondary goal model that prevent inefficient DMUs from using unrealistic weights. ^{Ramón et al. (2011)} further proposed a peer-restricted CREE model to narrow the differences among weights and eliminate zero weights. ^{Wang et al. (2011b)} introduced a virtual DMU into the CREE model and proposed a series of secondary goal models by expanding or reducing the gaps between the weighted inputs or outputs of the evaluated DMU_d and those of the virtual DMU. A similar study (^{Shi et al., 2019}) proposed a neutral CREE by introducing virtual DMUs. To reduce the differences among weights and eliminate zero weights, ^{Wu et al. (2012a)} proposed the following weight balance model:

\min {\displaystyle \sum _{r=\mathrm{1}}^s\left|{\lambda }_r^d\right|}+{\displaystyle \sum _{i=\mathrm{1}}^m\left|{\gamma }_i^d\right|}

s.t.\mathrm{\hspace{0.17em}}\mathrm{\hspace{0.17em}}{\displaystyle \sum _{i=\mathrm{1}}^m{W}_{id}{x}_{ij}-{\displaystyle \sum _{r=\mathrm{1}}^s{U}_{rd}{y}_{rj}}}\ge \mathrm{0},\mathrm{}j=\mathrm{1},\mathrm{}\mathrm{2},\mathrm{...},\mathrm{}n,

{\displaystyle \sum _{i=\mathrm{1}}^m{W}_{id}}{x}_{id}=\mathrm{1},

{\displaystyle \sum _{r=\mathrm{1}}^s{U}_{rd}{y}_{rd}}=E_{dd},

\mathrm{ }\mathrm{ }{U}_{rd}{y}_{rd}+{\lambda }_r^d=E_{dd}/s,\mathrm{}r=\mathrm{1},\mathrm{}\mathrm{2},\mathrm{...},\mathrm{}s,

W_{id}{x}_{id}+{\gamma }_i^d=\mathrm{1}/m,\mathrm{}i=\mathrm{1},\mathrm{}\mathrm{2},\mathrm{...},\mathrm{}m,

W_{id}\ge \mathrm{0},\mathrm{}i=\mathrm{1},\mathrm{}\mathrm{2},\mathrm{...},\mathrm{}m,

(6)U_{rd}\ge \mathrm{0},\mathrm{}r=\mathrm{1},\mathrm{}\mathrm{2},\mathrm{...},\mathrm{}s.

In Model (6), {\gamma }_i^d ({\lambda }_r^d) is used to reduce the large differences among the weighted inputs (outputs) of DMU_d. Specifically, Model (6) is designed to equate each weighted input to 1/m and each weighted output to E_dd/s as much as possible. To reduce the number of zero weights, ^{Wu et al. (2012a)} also proposed a weight restriction model, which can also be found in ^{Wang et al. (2012)}.

The abovementioned models use weighted inputs and outputs. Several scholars have also proposed efficiency-neutral secondary target models. For example, ^{Liang et al. (2008a)} proposed a slack-based secondary goal model that made the CESs of other DMUs close to their averages. ^{Wu et al. (2016b)} adopted the idea that the secondary goal model should make the CES of each DMU close to a specific value (e.g., the average of the desirable and undesirable CES targets).

Weight relaxation secondary goal models

The benevolent, aggressive, and neutral models all follow a rule that the evaluated DMU_d uses only a set of weights to obtain its self-evaluation efficiency and the CESs of other DMUs. However, several scholars have relaxed this rule and proposed a series of extended CREE models. For example, ^{Ramón et al. (2014)} proposed two models for obtaining all the possible weights of DMUs to avoid the choice between benevolent and aggressive strategies. A similar idea is presented in ^{Yang et al. (2012)}. ^{Liang et al. (2008b)} introduced game theory into CREE and proposed a game CREE model, in which DMUs are regarded as players in a non-cooperative game and each DMU can evaluate the other DMUs by using a different set of weights. The game CREE model of ^{Liang et al. (2008b)} is as follows:

\max {\displaystyle \sum _{r=\mathrm{1}}^s{U}_{rj}^d}{y}_{rj}

s.t.\mathrm{\hspace{0.17em}}\mathrm{\hspace{0.17em}}\mathrm{\hspace{0.17em}}{\displaystyle \sum _{i=\mathrm{1}}^m{W}_{ij}^d}{x}_{il}-{\displaystyle \sum _{r=\mathrm{1}}^s{U}_{rj}^d}{y}_{rl}\ge \mathrm{0},\mathrm{}l=\mathrm{1},\mathrm{}\mathrm{2},\mathrm{...},\mathrm{}n,

{\displaystyle \sum _{i=\mathrm{1}}^m{W}_{ij}^d}{x}_{ij}=\mathrm{1},

{\rho }_d{\displaystyle \sum _{i=\mathrm{1}}^m{W}_{ij}^d}{x}_{id}-{\displaystyle \sum _{r=\mathrm{1}}^s{U}_{rj}^d}{y}_{rd}\le \mathrm{0},

(7)W_{ij}^d,\mathrm{}{U}_{rj}^d\ge \mathrm{0},\mathrm{for\; all}i,r.

In Model (7), the efficiency of DMU_d cannot be lower than r_d, and the other DMUs strive to maximize their CESs. Model (7) must be solved repeatedly, and a new efficiency r_d is obtained each time. To solve Model (7), ^{Liang et al. (2008b)} designed an algorithm, proved the convergence of the algorithm, and verified that the optimal solution obtained using the algorithm would always be a Nash equilibrium. This model effectively overcomes the non-uniqueness problem of weights in the traditional CREE method, strongly promoting the theory of the CREE method and extending its application in game scenarios. Following the work of ^{Liang et al. (2008b)}, ^{Wu et al. (2009a)} proposed a modified game CREE model to obtain nonnegative CESs and applied the model to evaluate the efficiency of countries participating in the Olympics. ^{Wu et al. (2009e)} applied the model of ^{Liang et al. (2008b)} to analyze preference voting, while ^{Wu and Liang (2012)} extended it to multiple-criteria decision-making (MCDM) and proposed a cross-evaluation MCDM method for ranking alternatives. ^{Liu et al. (2017a)} combined the ideas of the aggressive CREE and the game CREE of ^{Liang et al. (2008b)}, then proposed an aggressive game CREE model to distinguish among all DMUs. ^{Li et al. (2018)} proposed a balanced CREE model and applied the iterative procedure of ^{Liang et al. (2008b)} to obtain optimal balanced CESs for DMUs.

To obtain Pareto optimal CESs, ^{Wu et al. (2009d)} integrated the Nash bargaining game into the CREE method. Efficiency was obtained by bargaining between CCR efficiency and CES, and the authors concluded that the efficiency obtained using the Nash bargaining game model was Pareto optimal. ^{Wu et al. (2016a)} further proposed a Pareto optimal test model to determine if a CES was Pareto optimal and a CES Pareto improvement model to improve CESs that were not Pareto optimal.

Other strategic models of CREE

In addition to the benevolent, aggressive, neutral, and weight relaxation models, scholars have also proposed secondary goal models that use other strategies. For example, ^{Wu et al. (2009f)} proposed an innovative CREE model to optimize the order of each DMU:

\min \mathrm{ }\mathrm{ }\mathrm{ }{I}_d={\displaystyle \sum _{j=\mathrm{1}}^n{z}_j^d}

s.t.\mathrm{\hspace{0.17em}}\mathrm{\hspace{0.17em}}\mathrm{\hspace{0.17em}}\mathrm{\hspace{0.17em}}{\displaystyle \sum _{r=\mathrm{1}}^s{U}_{rd}}{y}_{rj}-{\displaystyle \sum _{i=\mathrm{1}}^m{W}_{id}}{x}_{ij}\le \mathrm{0},\mathrm{}j=\mathrm{1},\mathrm{}\mathrm{2},\mathrm{...},\mathrm{}n,

{\displaystyle \sum _{i=\mathrm{1}}^m{W}_{id}}{x}_{id}=\mathrm{1},

{\displaystyle \sum _{r=\mathrm{1}}^s{U}_{rd}}{y}_{rd}=E_{dd},

{\displaystyle \frac{{\displaystyle \sum _{r=\mathrm{1}}^s{U}_{rd}}{y}_{rj}}{{\displaystyle \sum _{i=\mathrm{1}}^m{W}_{id}}{x}_{ij}}}+h_j^d={\displaystyle \frac{{\displaystyle \sum _{r=\mathrm{1}}^s{U}_{rd}}{y}_{rd}}{{\displaystyle \sum _{i=\mathrm{1}}^m{W}_{id}}{x}_{id}}}=E_{dd},\mathrm{}j=\mathrm{1},\mathrm{}\mathrm{2},\mathrm{...},\mathrm{}n,

\mathrm{ }{z}_j^d\in \{\mathrm{0},\mathrm{}\mathrm{1}\}\mathrm{and}{h}_j^d\mathrm{ }\mathrm{ }\mathrm{ }\mathrm{is\; free},\mathrm{}j=\mathrm{1},\mathrm{}\mathrm{2},\mathrm{...},\mathrm{}n,

(8)W_{id},\mathrm{}{U}_{rd}\ge \mathrm{0},\mathrm{for\; all}i,r.

In Model (8), the first, second, and third constraints are to ensure the self-evaluation efficiency of DMU_d. In the fourth constraint, the efficiency of DMU_d is compared with the efficiencies of the other DMUs, and h_j^d represents the efficiency deviation between DMU_d and DMU_j. If (h_j^d>\mathrm{0}), then the efficiency of DMU_d is less (greater) than that of DMU_j. In the fifth and sixth constraints, z_j^d is a 0–1 variable, which acts as a counter in Model (8), M is a large positive number, and e is a non-Archimedean element. If h_j^d\ge \mathrm{0}, then we have z_j^d=\mathrm{0}; if h_j^d\le \mathrm{0}, then we have z_j^d=\mathrm{1}. In the objective function, I_d represents the total number of times that z_j^d=\mathrm{1} (j = 1, 2,…, n). In accordance with the objective function and the sixth constraint, the purpose of Model (8) is to find a set of optimal weights to let z_j^{d*}=\mathrm{0} as much as possible, i.e., to choose a set of weights to let as few times as possible.

In addition to the model of ^{Wu et al. (2009f)}, a multiple-criteria CREE model that considered multiple objective functions was proposed by ^{Örkcü and Bal (2011)} to solve the problem of the non-uniqueness of weights. ^{Lam (2010)} combined discriminant analysis, the super-efficiency DEA model, and mixed-integer linear programming with the CREE method to obtain a method that preserved the classification results of the traditional DEA. ^{Rödder and Reucher (2011)} proposed using the CREE method for input allocation. ^{Alcaraz et al. (2013)} discussed the problem of the non-uniqueness of weights in CREEs and proposed a sorting-based CREE program that generated a sorting range for each DMU and then ranked the DMUs by comprehensively analyzing the sorting ranges of all the DMUs. ^{Lin et al. (2016)} proposed an iterative CREE method for determining a unique weight set and reducing the number of zero weights.

To resolve the problem of negative CESs in the model of variable returns to scale (VRS), ^{Lim and Zhu (2015)} proposed a CREE method based on a geometric view of the relationship between the models of VRS and constant returns to scale. ^{Du et al. (2014)} proposed a resource allocation method based on an iterative CREE method and proved that this method was always feasible. ^{Lim (2012a)} incorporated CREE into a context-dependent DEA to overcome the drawbacks of the original context-related DEA and used an illustrative example to demonstrate the applicability and usefulness of this context-dependent CREE model. ^{Ruiz (2013)} introduced the direction distance function into CREE to evaluate the efficiency of a DMU on the basis of input and output perspectives. To evaluate the efficiency of a DMU with interval data, ^{Wu et al. (2013)} proposed the interval CREE model and used the multiattribute method to rank interval CESs. ^{Wu et al. (2016c)} introduced a concept of satisfaction into CREE models, designed an algorithm to solve them, and demonstrated that this algorithm could obtain a unique set of optimal weights for each DMU. ^{Sun et al. (2018c)} proposed an altruism CREE model that no longer guaranteed the self-evaluation efficiency of DMUs and allowed the self-evaluation efficiency of each DMU to change adaptively, thereby increasing the flexibility of the cross-evaluation process. To consider the attitude toward risks of the decision-maker, ^{Liu et al. (2019)} introduced prospect theory into CREE and proposed a prospect CREE model. ^{Kao and Liu (2019)} introduced CREE into network system analysis and proposed network CREE models, which were applied to practical cases with series and parallel structures.

Aggregation of CESs

The traditional CREE method uses the arithmetic average method to aggregate all CESs. However, the efficiency results obtained using the arithmetic average method lose the correlation between weights and CESs and are not Pareto optimal. To overcome the shortcomings, scholars have recently proposed a series of CES aggregation methods to replace the arithmetic average method. For instance, ^{Wu et al. (2009c)} regarded each DMU as a player in a cooperative game, defined the characteristic function values of the alliance and various sub-allies, and combined cooperative game theory with the CREE method. Then, the aggregation weights of each DMU were calculated using the Shapley values of the players in the cooperative game. Using the concept of information entropy, ^{Wu et al. (2011a}; ^2012b) transformed the CES matrix into an entropy matrix and proposed an entropy decision model, which used the obtained weights to aggregate all CESs. ^{Song et al. (2017)} used the benevolent and aggressive models to calculate two sets of CES matrices for all DMUs, then the entropy decision model of ^{Wu et al. (2011a)} was applied to aggregate all CESs. ^{Song and Liu (2018)} extended this model with a variance coefficient method. ^{Zerafat Angiz et al. (2013)} proposed a two-step sorting-based aggregation method. The CES matrix was first transformed into a ranking order matrix, and a first-order model was then used to obtain aggregation weights. ^{Wang and Wang (2013)} proposed a least-square deviation approach to measure the importance index of each CES and calculate the aggregation weights of all CESs in accordance with their importance indices. To rank efficient DMUs, ^{Hong and Jeong (2017)} proposed two heuristic approaches based on the CREE method. Unlike other CREE methods, these approaches do not use any linear programming model but can completely rank all DMUs.

The aforementioned methods objectively calculate the aggregation weights of CESs. Considering the subjective preferences of decision-makers in the aggregation, ^{Yang et al. (2013)} applied the evidence reasoning method to aggregate CESs. ^{Yang et al. (2012)} combined stochastic multi-criteria acceptance analysis with CREE and proposed an acceptability index to complete the aggregation process. ^{Wang and Chin (2011)} applied the ordered weighted averaging (OWA) method to aggregate CESs. In accordance with the subjective judgments of the decision-makers, the OWA method can assign different weights to the self- and peer-efficiencies of each DMU. Subsequently, ^{Oukil (2019)} proposed an improved OWA CREE aggregation method.

In reality, the input and output data of DMUs may be uncertain. On the basis of this situation, scholars have proposed various fuzzy CREE methods. Aiming at the aggregation problem of interval CES, ^{Wu et al. (2011b}; ²⁰¹³) proposed an improved technique for order of preference by similarity to ideal solution (TOPSIS). This proposed TOPSIS method can obtain aggregation weights through multiattribute decision-making techniques and calculate the final efficiency of each DMU using the aggregated weights. Considering the lack of sufficient discrimination capability of fuzzy DEA, ^{Chen and Wang (2016)} proposed a fuzzy cross-efficiency model and applied the minimax regret-based method of ^{Wang et al. (2005)} to rank interval efficiency. In the case of interval output–input data, ^{Jahanshahloo et al. (2011)} proposed a super-efficiency CREE method and applied the TOPSIS method to identify all alternatives. ^{Liu (2018)} used the aggressive and benevolent models to obtain the interval CESs of each DMU and applied a signal-to-noise ratio index to rank all DMUs.

Applications

The cross-efficiency method has been applied in engineering management, which includes the fields of environmental analysis, transportation and logistics, manufacturing industry, and supply chains. This section reviews the applications of CREE in the four fields.

Application of CREE in environmental analysis

With the deterioration of the environment, increasing attention is being paid by scholars to environmental issues (^{Sun et al., 2018a}; ^2018b; ^2018c; ²⁰¹⁹; ^{Wu et al., 2019a}; ^2019b; ²⁰²⁰), leading to the frequent use of CREE to evaluate environmental performance (^{Sun et al., 2017a}). ^{Sarkis and Weinrach (2001)} evaluated government-supported, environment-friendly waste treatment technologies by using DEA and CREE methods. The results showed that previous DEA methods could be used to evaluate these technologies, but CREE was more appropriate if managers wanted to differentiate among technologies. ^{Lu and Lo (2007)} regarded smoke, dust, and sulfur dioxide emissions as undesirable outputs and used a CREE model to evaluate the economic and environmental efficiency scores of 31 provinces in China. They found that the coastal areas of China were superior to the inland areas in terms of economy and environment. ^{Lee and Farzipoor Saen (2012)} proposed an advanced DEA model based on CREE to measure the sustainable performance of enterprises. The results showed that the proposed model could effectively identify the levels of sustainable development of enterprises. Using the concept of CREE, ^{Guo and Wu (2013)} proposed a maximal balance index to evaluate the environmental efficiency of 32 paper mills. Their empirical study showed that the proposed index yielded a stable and unique ranking for all the paper mills. ^{Mahdiloo et al. (2015)} proposed a multi-objective model by incorporating CREE to evaluate the technical, environmental, and ecological efficiency of suppliers at Hyundai Steel Company. They compared the proposed model with a traditional eco-efficiency measurement model and concluded that the former could effectively reduce the burden of calculation. ^{Lo Storto (2016)} used a DEA model and a CREE model to evaluate the eco-efficiency of 116 Italian provincial capital cities, then used the Shannon entropy index to aggregate all the efficiencies.

Considering undesirable outputs, ^{Liu et al. (2017b)} used a CREE model to evaluate the ecological efficiency of thermal power plants in China. They regarded ranking priority as the secondary goal of CREE to solve the problem of the non-uniqueness of weights. ^{Geng et al. (2017)} used a traditional DEA model and a CREE model to evaluate the environmental efficiency of ethylene production in China and found that the CREE model could fully rank all ethylene plants effectively. ^{Hatami-Marbini et al. (2017)} proposed a fuzzy CREE model to evaluate the eco-efficiency of a supplier and demonstrated the practicability of the proposed model by analyzing a case in the semiconductor industry. ^{Liu et al. (2018)} applied the CREE model to analyze the carbon emission efficiency of urban agglomerations in China from 2008 to 2015. The results showed that the carbon emission efficiency of urban agglomerations in China had not improved considerably. ^{Zoroofchi et al. (2018)} applied three methods, including CREE, to evaluate the sustainable performance of 15 suppliers of an Iranian soft drink company. The authors concluded that their models could improve the ability to identify desirable suppliers.

Application of CREE in transportation and logistics

Considering the extreme weighting of traditional DEA, ^{Sarkis (2000)} used a CREE approach to evaluate the operational efficiency of airports in the United States. However, the author indicated that the average weights of the cross-efficiency approach could not measure the importance of each input or output. ^{Sarkis and Talluri (2004)} applied the aggressive CREE method to assess the operational efficiency of 44 major airports in the United States. ^{Barros (2006)} applied a CREE method and other DEA models to assess the performances of Italian seaports and obtained empirical results showing that Italian seaports exhibited relatively high efficiency. ^{Lin and Tseng (2007)} used CREE and other DEA methods to analyze the operational efficiency of the main container ports in the Asia-Pacific region and identify the trends in port efficiency. They also discussed the effects of various inputs and outputs on port efficiency. ^{Wu et al. (2009g)} proposed an improved CREE method to analyze the efficiency of 28 container ports in 12 countries in Asia and concluded that the overall economy of a country had a remarkable effect on port efficiency performance. ^{Wu et al. (2010)} used the CREE method to evaluate the performance of 77 container ports worldwide and demonstrated that the CREE method could provide a unique ordering for all the container ports. ^{Wu and Goh (2010)} compared the operational efficiencies of ports in emerging and advanced markets by using the DEA and CREE approaches. Their empirical studies showed that no port in an advanced market would be an ideal role model in this field. ^{Rezaee et al. (2016)} proposed a game CREE approach for assessing the operational efficiency of transportation systems and demonstrated that their approach could distinguish among transportation systems effectively.

Application of CREE in the manufacturing industry

With its complete ranking capability, the CREE technique has been extensively used for performance evaluation in the manufacturing industry. This application of CREE includes the evaluation of flexible manufacturing systems (^{Shang and Sueyoshi, 1995}), the selection of industrial robots (^{Baker and Talluri, 1997}; ^{Braglia and Petroni, 1999}; ^{Sun et al., 2017b}), the justification of advanced manufacturing technologies (^{Talluri and Paul Yoon, 2000}), labor allocation in a cellular manufacturing system (^{Ertay and Ruan, 2005}), the performance evaluation of fabless enterprises (^{Chu et al., 2008}), and the efficiency evaluation of industrial systems (^{Wang and Chin, 2011}).

Scholars have also combined CREE with other methods to analyze manufacturing performance. ^{Sun (2002)} employed the aggressive CREE method and “false positive index” proposed by ^{Baker and Talluri (1997)} to evaluate and rank computer numerical control machines in accordance with their system specifications and costs. ^{Tan et al. (2017)} combined the CREE model and the balanced scorecard method to evaluate the service performance of the automobile industry.

Application of CREE in supply chains

^{Yu et al. (2010)} used the CREE method to estimate the efficiency of a supply chain in an information-sharing scenario. To classify inventory from multiple attributes, ^{Chen (2011)} proposed applying CREE to multi-criteria inventory classification and concluded that the CREE method could aggregate each item from the most and least advantageous attributes. ^{Park et al. (2014)} suggested a CREE-based weighted linear optimization method for the fine classification of inventory items and also compared other methods with the proposed method through simulation.

CREE has also been used for supplier selection. Considering the competition among suppliers, ^{Ma et al. (2014)} used the game CREE method of ^{Liang et al. (2008b)} to evaluate supplier performance. Their case studies showed that the method of ^{Liang et al. (2008b)} could obtain unique and Pareto optimal efficiency for suppliers. ^{Noorizadeh et al. (2013)} applied the CREE method to supplier selection while handling nondiscretionary inputs. To consider the undesirable output of a supplier, ^{Zoroufchi et al. (2012)} proposed a comprehensive method based on CREE and slack-based measure models. The results showed that this method could effectively identify the best suppliers. ^{Dotoli et al. (2016)} combined CREE and the Monte Carlo method to formulate a supplier selection method under uncertain conditions, which could help decision-makers select suitable partners when the input and output data are uncertain. Hatami-Marbini et al. (2017) proposed a CREE method and an algorithm for product-based sustainable supplier selection and used a case in the semiconductor industry to demonstrate the applicability of this method and the effectiveness of the algorithm.

Conclusions and research prospects

DEA is an excellent tool for evaluation in various fields, especially in engineering management. However, the classical DEA model cannot distinguish among DMUs that are efficient, making it generally impossible to be used directly in obtaining a full ranking of DMUs. The CREE method can effectively avoid the shortcomings of the classical DEA method by integrating two mechanisms: Self- and peer-evaluation. The final efficiency of each DMU is determined by the optimal weights chosen not only by itself but also by the other DMUs. Thus, the unrealistic weight problem in the classical DEA method can be solved, and all DMUs can be ranked sufficiently.

In terms of theory, much progress has been made in CREE techniques. In terms of applications, the CREE method has also been widely used, particularly in the field of engineering management. Nevertheless, some challenges and key problems should be solved by future CREE research. First, scholars have proposed many strategic secondary goal models to solve the problem of the non-uniqueness of weights. However, engineering management practitioners often encounter various evaluation standards, such as project scheme selection, risk identification, and human resource management. No accepted standard for determining which strategy model should be used for a particular scenario is available, which often confuses decision-makers who must select a model. In future research, scholars should classify the various evaluation problems of engineering management and list the application scenarios of the models in accordance with the characteristics of the CREE methods.

Second, in reality, competition and cooperation often exist simultaneously among DMUs. For example, competitors may establish research and development (R&D) partnerships for the acquisition and integration of external knowledge to maximize their benefits (^{Enberg, 2012}). Most of the existing game CREE methods assume that all DMUs are either cooperative or competitive. The simple competition or cooperative CREE methods obviously cannot be used directly in situations in which cooperation and competition exist simultaneously. In future research, a statistical analysis method, such as the cluster analysis method, can be used first to identify cooperative or competitive relationships among DMUs. Then, the coopetition CREE is used to evaluate the performance of these DMUs. The multi-objective planning method can also be introduced into CREE to characterize the cooperative or competitive relationship between the evaluated DMU_d and other DMUs. Therefore, studying how to expand the game CREE model by integrating a statistical analysis method or the multi-objective planning method to solve complex game problems is valuable.

Third, in the field of manufacturing or engineering, decision-makers often need to focus on productivity changes in different periods from a dynamic perspective. For example, the adjustment of a certain input (e.g., transmission line) of an electric power enterprise would affect the efficiency of not only the current period but also the next period (^{von Geymueller, 2009}). The existing CREE methods are based on static perspectives, and minimal attention has been paid to research on dynamic CREE. In a dynamic environment, the traditional static CREE methods have difficulty providing decision-makers with high-quality decision-making information. To measure dynamic performance, scholars have proposed the Malmquist productivity index (MPI) (^{Färe et al., 1994}). Nonetheless, the existing self-assessment MPI may overestimate the efficiency, resulting in the miscalculation of productivity changes (^{Ding et al., 2019}). Thus, studying how to combine the MPI with CREE to deal with real dynamic production is a worthy focus in the future.

Fourth, in the fields of engineering and manufacturing, several large enterprises, such as petroleum enterprises, often have subsystems or subenterprises (^{Song et al., 2015}). The production processes of such enterprises are considerably complex. Multiple processes that are difficult to measure using traditional methods should be evaluated to improve efficiency. Accordingly, many network DEA methods have been proposed, but they encounter the problem of having neither unique nor realistic weights. In future research, the network DEA model could be combined with CREE techniques. This network CREE model could not only overcome the shortcomings of the traditional network DEA model, but also provide scientific and reasonable results for the performance evaluation and improvement of network systems.

Fifth, resource allocation issues are often involved in production or manufacturing activities. For example, a manufacturing factory needs to plan carefully how to allocate resources to each plant to achieve optimal production (^{Yu and Hu, 2014}). For resource allocation, special attention should be paid to possible conflicts that involve related production activities that compete for limited resources. Although game CREE effectively combines game and DEA methods, most of these models can be used only for efficiency evaluation. Hence, a necessary and interesting direction for future research is the use of the game CREE method for resource allocation in complex situations involving such conflicts.

References

1 Alcaraz J, Ramón N, Ruiz J L, Sirvent I (2013). Ranking ranges in cross-efficiency evaluations. European Journal of Operational Research, 226(3): 516–521

2 Andersen P, Petersen N C (1993). A procedure for ranking efficient units in data envelopment analysis. Management Science, 39(10): 1261–1264

3 Anderson T R, Hollingsworth K, Inman L (2002). The fixed weighting nature of a cross-evaluation model. Journal of Productivity Analysis, 17(3): 249–255

4 Baker P C, Talluri S (1997). A closer look at the use of data envelopment analysis for technology selection. Computers & Industrial Engineering, 32(1): 101–108

5 Barros C P (2006). A benchmark analysis of Italian seaports using data envelopment analysis. Maritime Economics & Logistics, 8(4): 347–365

6 Boussofiane A, Dyson R G, Thanassoulis E (1991). Applied data envelopment analysis. European Journal of Operational Research, 52(1): 1–15

7 Braglia M, Petroni A (1999). Evaluating and selecting investments in industrial robots. International Journal of Production Research, 37(18): 4157–4178

8 Brannick M T, Salas E, Prince C W (1997). Team Performance Assessment and Measurement: Theory, Methods, and Applications. Hove, East Sussex: Psychology Press

9 Chang C M (2008). Engineering management in developing economies: The EMIDE approaches to meet the new challenges. International Journal of Innovation & Technology Management, 5(4): 381–400

10 Charnes A, Cooper W W, Rhodes E (1978). Measuring the efficiency of decision making units. European Journal of Operational Research, 2(6): 429–444

11 Chen J X (2011). Peer-estimation for multiple criteria ABC inventory classification. Computers & Operations Research, 38(12): 1784–1791

12 Chen L, Wang Y M (2016). Data envelopment analysis cross-efficiency model in fuzzy environments. Journal of Intelligent & Fuzzy Systems, 30(5): 2601–2609

13 Chen L, Wang Y M (2020). DEA target setting approach within the cross efficiency framework. Omega, 96: 102072

14 Chen W, Zhou K, Yang S (2017). Evaluation of China’s electric energy efficiency under environmental constraints: A DEA cross efficiency model based on game relationship. Journal of Cleaner Production, 164: 38–44

15 Chu M T, Shyu J Z, Khosla R (2008). Measuring the relative performance for leading fabless firms by using data envelopment analysis. Journal of Intelligent Manufacturing, 19(3): 257–272

16 Cook W D, Seiford L M (2009). Data envelopment analysis (DEA) — Thirty years on. European Journal of Operational Research, 192(1): 1–17

17 Cui Q, Li Y (2015). Evaluating energy efficiency for airlines: An application of VFB-DEA. Journal of Air Transport Management, 44– 45: 34–41

18 Despotis D K (2002). Improving the discriminating power of DEA: Focus on globally efficient units. Journal of the Operational Research Society, 53(3): 314–323

19 Ding L, Yang Y, Wang W, Calin A C (2019). Regional carbon emission efficiency and its dynamic evolution in China: A novel cross efficiency-malmquist productivity index. Journal of Cleaner Production, 241: 118260

20 Dotoli M, Epicoco N, Falagario M, Sciancalepore F (2016). A stochastic cross-efficiency data envelopment analysis approach for supplier selection under uncertainty. International Transactions in Operational Research, 23(4): 725–748

21 Doyle J, Green R (1994). Efficiency and cross-efficiency in DEA: Derivations, meanings and uses. Journal of the Operational Research Society, 45(5): 567–578

22 Du J, Cook W D, Liang L, Zhu J (2014). Fixed cost and resource allocation based on DEA cross-efficiency. European Journal of Operational Research, 235(1): 206–214

23 Dyson R G, Thanassoulis E (1988). Reducing weight flexibility in data envelopment analysis. Journal of the Operational Research Society, 39(6): 563–576

24 Enberg C (2012). Enabling knowledge integration in coopetitive R&D projects — The management of conflicting logics. International Journal of Project Management, 30(7): 771–780

25 Ertay T, Ruan D (2005). Data envelopment analysis based decision model for optimal operator allocation in CMS. European Journal of Operational Research, 164(3): 800–810

26 Färe R, Grosskopf S, Norris M, Zhang Z (1994). Productivity growth, technical progress, and efficiency change in industrialized countries. The American Economic Review, 84(1): 66–83

27 Geng Z, Dong J, Han Y, Zhu Q (2017). Energy and environment efficiency analysis based on an improved environment DEA cross-model: Case study of complex chemical processes. Applied Energy, 205: 465–476

28 Guo D, Wu J (2013). A complete ranking of DMUs with undesirable outputs using restrictions in DEA models. Mathematical and Computer Modelling, 58(5–6): 1102–1109

29 Hatami-Marbini A, Agrell P J, Tavana M, Khoshnevis P (2017). A flexible cross-efficiency fuzzy data envelopment analysis model for sustainable sourcing. Journal of Cleaner Production, 142: 2761–2779

30 Hong J D, Jeong K Y (2017). Cross efficiency based heuristics to rank decision making units in data envelopment analysis. Computers & Industrial Engineering, 111: 320–330

31 Jahanshahloo G R, Khodabakhshi M, Hosseinzadeh Lotfi F, Moazami Goudarzi M R (2011). A cross-efficiency model based on super-efficiency for ranking units through the TOPSIS approach and its extension to the interval case. Mathematical and Computer Modelling, 53(9–10): 1946–1955

32 Kao C, Liu S T (2019). Cross efficiency measurement and decomposition in two basic network systems. Omega, 83: 70–79

33 Lam K F (2010). In the determination of weight sets to compute cross-efficiency ratios in DEA. Journal of the Operational Research Society, 61(1): 134–143

34 Lannes W J (2001). What is engineering management? IEEE Transactions on Engineering Management, 48(1): 107–115

35 Lee K H, Farzipoor Saen R (2012). Measuring corporate sustainability management: A data envelopment analysis approach. International Journal of Production Economics, 140(1): 219–226

36 Li F, Zhu Q, Chen Z, Xue H (2018). A balanced data envelopment analysis cross-efficiency evaluation approach. Expert Systems with Applications, 106: 154–168

37 Li H B, Lei L (2007). DEA-based project knowledge management performance evaluation. In: International Conference on Management Science & Engineering. Harbin, 1561–1566

38 Liang L, Wu J, Cook W D, Zhu J (2008a). Alternative secondary goals in DEA cross-efficiency evaluation. International Journal of Production Economics, 113(2): 1025–1030

39 Liang L, Wu J, Cook W D, Zhu J (2008b). The DEA game cross-efficiency model and its Nash equilibrium. Operations Research, 56(5): 1278–1288

40 Lim S (2012a). Context-dependent data envelopment analysis with cross-efficiency evaluation. Journal of the Operational Research Society, 63(1): 38–46

41 Lim S (2012b). Minimax and maximin formulations of cross-efficiency in DEA. Computers & Industrial Engineering, 62(3): 726–731

42 Lim S, Oh K W, Zhu J (2014). Use of DEA cross-efficiency evaluation in portfolio selection: An application to Korean stock market. European Journal of Operational Research, 236(1): 361–368

43 Lim S, Zhu J (2015). DEA cross-efficiency evaluation under variable returns to scale. Journal of the Operational Research Society, 66(3): 476–487

44 Lin L C, Tseng C C (2007). Operational performance evaluation of major container ports in the Asia-Pacific region. Maritime Policy & Management, 34(6): 535–551

45 Lin R, Chen Z, Xiong W (2016). An iterative method for determining weights in cross efficiency evaluation. Computers & Industrial Engineering, 101: 91–102

46 Liu B, Tian C, Li Y, Song H, Ma Z (2018). Research on the effects of urbanization on carbon emissions efficiency of urban agglomerations in China. Journal of Cleaner Production, 197: 1374–1381

47 Liu H H, Song Y Y, Yang G L (2019). Cross-efficiency evaluation in data envelopment analysis based on prospect theory. European Journal of Operational Research, 273(1): 364–375

48 Liu S T (2018). A DEA ranking method based on cross-efficiency intervals and signal-to-noise ratio. Annals of Operations Research, 261(1–2): 207–232

49 Liu W, Wang Y M, Lv S (2017a). An aggressive game cross-efficiency evaluation in data envelopment analysis. Annals of Operations Research, 259(1–2): 241–258

50 Liu X, Chu J, Yin P, Sun J (2017b). DEA cross-efficiency evaluation considering undesirable output and ranking priority: A case study of eco-efficiency analysis of coal-fired power plants. Journal of Cleaner Production, 142: 877–885

51 Lo Storto C (2016). Ecological efficiency based ranking of cities: A combined DEA cross-efficiency and Shannon’s entropy method. Sustainability, 8(2): 124

52 Lu W M, Lo S F (2007). A closer look at the economic-environmental disparities for regional development in China. European Journal of Operational Research, 183(2): 882–894

53 Ma R, Yao L, Jin M, Ren P (2014). The DEA game cross-efficiency model for supplier selection problem under competition. Applied Mathematics & Information Sciences, 8(2): 811–818

54 Mahdiloo M, Farzipoor Saen R, Lee K H (2015). Technical, environmental and eco-efficiency measurement for supplier selection: An extension and application of data envelopment analysis. International Journal of Production Economics, 168: 279–289

55 Noorizadeh A, Mahdiloo M, Farzipoor Saen R (2013). Using DEA cross-efficiency evaluation for suppliers ranking in the presence of non-discretionary inputs. International Journal of Shipping and Transport Logistics, 5(1): 95–111

56 Örkcü H H, Bal H (2011). Goal programming approaches for data envelopment analysis cross efficiency evaluation. Applied Mathematics and Computation, 218(2): 346–356

57 Oukil A (2019). Embedding OWA under preference ranking for DEA cross-efficiency aggregation: Issues and procedures. International Journal of Intelligent Systems, 34(5): 947–965

58 Park J, Bae H, Bae J (2014). Cross-evaluation-based weighted linear optimization for multi-criteria ABC inventory classification. Computers & Industrial Engineering, 76: 40–48

59 Ramón N, Ruiz J L, Sirvent I (2010a). A multiplier bound approach to assess relative efficiency in DEA without slacks. European Journal of Operational Research, 203(1): 261–269

60 Ramón N, Ruiz J L, Sirvent I (2010b). On the choice of weights profiles in cross-efficiency evaluations. European Journal of Operational Research, 207(3): 1564–1572

61 Ramón N, Ruiz J L, Sirvent I (2011). Reducing differences between profiles of weights: A “peer-restricted” cross-efficiency evaluation. Omega, 39(6): 634–641

62 Ramón N, Ruiz J L, Sirvent I (2014). Dominance relations and ranking of units by using interval number ordering with cross-efficiency intervals. Journal of the Operational Research Society, 65(9): 1336–1343

63 Rezaee M J, Izadbakhsh H, Yousefi S (2016). An improvement approach based on DEA-game theory for comparison of operational and spatial efficiencies in urban transportation systems. KSCE Journal of Civil Engineering, 20(4): 1526–1531

64 Rödder W, Reucher E (2011). A consensual peer-based DEA-model with optimized cross-efficiencies — Input allocation instead of radial reduction. European Journal of Operational Research, 212(1): 148–154

65 Ruiz J L (2013). Cross-efficiency evaluation with directional distance functions. European Journal of Operational Research, 228(1): 181–189

66 Sarkis J (2000). An analysis of the operational efficiency of major airports in the United States. Journal of Operations Management, 18(3): 335–351

67 Sarkis J, Talluri S (2004). Performance based clustering for benchmarking of US airports. Transportation Research Part A, Policy and Practice, 38(5): 329–346

68 Sarkis J, Weinrach J (2001). Using data envelopment analysis to evaluate environmentally conscious waste treatment technology. Journal of Cleaner Production, 9(5): 417–427

69 Sexton T R, Silkman R H, Hogan A J (1986). Data envelopment analysis: Critique and extensions. New Directions for Program Evaluation, (32): 73–105

70 Shang J, Sueyoshi T (1995). A unified framework for the selection of a flexible manufacturing system. European Journal of Operational Research, 85(2): 297–315

71 Shi H, Wang Y, Chen L (2019). Neutral cross-efficiency evaluation regarding an ideal frontier and anti-ideal frontier as evaluation criteria. Computers & Industrial Engineering, 132: 385–394

72 Song L, Liu F (2018). An improvement in DEA cross-efficiency aggregation based on the Shannon entropy. International Transactions in Operational Research, 25(2): 705–714

73 Song M, Zhang J, Wang S (2015). Review of the network environmental efficiencies of listed petroleum enterprises in China. Renewable & Sustainable Energy Reviews, 43: 65–71

74 Song M, Zhu Q, Peng J, Santibanez Gonzalez E D R (2017). Improving the evaluation of cross efficiencies: A method based on Shannon entropy weight. Computers & Industrial Engineering, 112: 99–106

75 Sun J, Fu Y, Ji X, Zhong R Y (2017a). Allocation of emission permits using DEA-game-theoretic model. Operations Research, 17(3): 867–884

76 Sun J, Li G, Wang Z (2018a). Optimizing China’s energy consumption structure under energy and carbon constraints. Structural Change and Economic Dynamics, 47: 57–72

77 Sun J, Li G, Wang Z (2019). Technology heterogeneity and efficiency of China’s circular economic systems: A game meta-frontier DEA approach. Resources, Conservation and Recycling, 146: 337–347

78 Sun J, Wang Z, Li G (2018b). Measuring emission-reduction and energy-conservation efficiency of Chinese cities considering management and technology heterogeneity. Journal of Cleaner Production, 175: 561–571

79 Sun J, Wu J, Guo D (2013). Performance ranking of units considering ideal and anti-ideal DMU with common weights. Applied Mathematical Modelling, 37(9): 6301–6310

80 Sun J, Wu J, Wang Y, Li L, Wang Y (2018c). Cross-efficiency evaluation method based on the conservative point of view. Expert Systems, e12336

81 Sun J, Yang R, Ji X, Wu J (2017b). Evaluation of decision-making units based on the weight-optimized DEA model. Kybernetika, 53(2): 244–262

82 Sun J, Yuan Y, Yang R, Ji X, Wu J (2017c). Performance evaluation of Chinese port enterprises under significant environmental concerns: An extended DEA-based analysis. Transport Policy, 60: 75–86

83 Sun S (2002). Assessing computer numerical control machines using data envelopment analysis. International Journal of Production Research, 40(9): 2011–2039

84 Talluri S, Paul Yoon K (2000). A cone-ratio DEA approach for AMT justification. International Journal of Production Economics, 66(2): 119–129

85 Tan Y, Zhang Y, Khodaverdi R (2017). Service performance evaluation using data envelopment analysis and balance scorecard approach: An application to automotive industry. Annals of Operations Research, 248(1–2): 449–470

86 von Geymueller P (2009). Static versus dynamic DEA in electricity regulation: The case of US transmission system operators. Central European Journal of Operations Research, 17(4): 397–413

87 Wang Y M, Chin K S (2010a). Some alternative models for DEA cross-efficiency evaluation. International Journal of Production Economics, 128(1): 332–338

88 Wang Y M, Chin K S (2010b). A neutral DEA model for cross-efficiency evaluation and its extension. Expert Systems with Applications, 37(5): 3666–3675

89 Wang Y M, Chin K S (2011). The use of OWA operator weights for cross-efficiency aggregation. Omega, 39(5): 493–503

90 Wang Y M, Chin K S, Jiang P (2011a). Weight determination in the cross-efficiency evaluation. Computers & Industrial Engineering, 61(3): 497–502

91 Wang Y M, Chin K S, Luo Y (2011b). Cross-efficiency evaluation based on ideal and anti-ideal decision making units. Expert Systems with Applications, 38(8): 10312–10319

92 Wang Y M, Chin K S, Wang S (2012). DEA models for minimizing weight disparity in cross-efficiency evaluation. Journal of the Operational Research Society, 63(8): 1079–1088

93 Wang Y M, Greatbanks R, Yang J B (2005). Interval efficiency assessment using data envelopment analysis. Fuzzy Sets and Systems, 153(3): 347–370

94 Wang Y M, Wang S (2013). Approaches to determining the relative importance weights for cross-efficiency aggregation in data envelopment analysis. Journal of the Operational Research Society, 64(1): 60–69

95 Wong Y H, Beasley J E (1990). Restricting weight flexibility in data envelopment analysis. Journal of the Operational Research Society, 41(9): 829–835

96 Wu J, Chu J, Sun J, Zhu Q (2016a). DEA cross-efficiency evaluation based on Pareto improvement. European Journal of Operational Research, 248(2): 571–579

97 Wu J, Chu J, Sun J, Zhu Q, Liang L (2016b). Extended secondary goal models for weights selection in DEA cross-efficiency evaluation. Computers & Industrial Engineering, 93: 143–151

98 Wu J, Chu J, Zhu Q, Yin P, Liang L (2016c). DEA cross-efficiency evaluation based on satisfaction degree: An application to technology selection. International Journal of Production Research, 54(20): 5990–6007

99 Wu J, Huang D, Zhou Z, Zhu Q (2020). The regional green growth and sustainable development of China in the presence of sustainable resources recovered from pollutions. Annals of Operations Research, 290: 27–45

100 Wu J, Li M, Zhu Q, Zhou Z, Liang L (2019a). Energy and environmental efficiency measurement of China’s industrial sectors: A DEA model with non-homogeneous inputs and outputs. Energy Economics, 78: 468–480

101 Wu J, Liang L (2012). A multiple criteria ranking method based on game cross-evaluation approach. Annals of Operations Research, 197(1): 191–200

102 Wu J, Liang L, Chen Y (2009a). DEA game cross-efficiency approach to Olympic rankings. Omega, 37(4): 909–918

103 Wu J, Liang L, Song M (2010). Performance based clustering for benchmarking of container ports: An application of DEA and cluster analysis technique. International Journal of Computational Intelligence Systems, 3(6): 709–722

104 Wu J, Liang L, Yang F (2009b). Achievement and benchmarking of countries at the Summer Olympics using cross efficiency evaluation method. European Journal of Operational Research, 197(2): 722–730

105 Wu J, Liang L, Yang F (2009c). Determination of the weights for the ultimate cross efficiency using Shapley value in cooperative game. Expert Systems with Applications, 36(1): 872–876

106 Wu J, Liang L, Yang F, Yan H (2009d). Bargaining game model in the evaluation of decision making units. Expert Systems with Applications, 36(3): 4357–4362

107 Wu J, Liang L, Zha Y (2009e). Preference voting and ranking using DEA game cross efficiency model. Journal of the Operations Research Society of Japan, 52(2): 105–111

108 Wu J, Liang L, Zha Y, Yang F (2009f). Determination of cross-efficiency under the principle of rank priority in cross-evaluation. Expert Systems with Applications, 36(3): 4826–4829

109 Wu J, Sun J, Liang L (2012a). Cross efficiency evaluation method based on weight-balanced data envelopment analysis model. Computers & Industrial Engineering, 63(2): 513–519

110 Wu J, Sun J, Liang L (2012b). DEA cross-efficiency aggregation method based upon Shannon entropy. International Journal of Production Research, 50(23): 6726–6736

111 Wu J, Sun J, Liang L, Zha Y (2011a). Determination of weights for ultimate cross efficiency using Shannon entropy. Expert Systems with Applications, 38(5): 5162–5165

112 Wu J, Sun J, Song M, Liang L (2013). A ranking method for DMUs with interval data based on DEA cross-efficiency evaluation and TOPSIS. Journal of Systems Science and Systems Engineering, 22(2): 191–201

113 Wu J, Sun J, Zha Y, Liang L (2011b). Ranking approach of cross-efficiency based on improved TOPSIS technique. Journal of Systems Engineering and Electronics, 22(4): 604–608

114 Wu J, Xia P, Zhu Q, Chu J (2019b). Measuring environmental efficiency of thermoelectric power plants: A common equilibrium efficient frontier DEA approach with fixed-sum undesirable output. Annals of Operations Research, 275(2): 731–749

115 Wu J, Yan H, Liu J (2009g). Groups in DEA based cross-evaluation: An application to Asian container ports. Maritime Policy & Management, 36(6): 545–558

116 Wu Y C J, Goh M (2010). Container port efficiency in emerging and more advanced markets. Transportation Research Part E, Logistics and Transportation Review, 46(6): 1030–1042

117 Xu J, Li Z (2012). A review on ecological engineering based engineering management. Omega, 40(3): 368–378

118 Yang F, Ang S, Xia Q, Yang C (2012). Ranking DMUs by using interval DEA cross efficiency matrix with acceptability analysis. European Journal of Operational Research, 223(2): 483–488

119 Yang G L, Yang J B, Liu W B, Li X X (2013). Cross-efficiency aggregation in DEA models using the evidential-reasoning approach. European Journal of Operational Research, 231(2): 393–404

120 Yu M M, Ting S C, Chen M C (2010). Evaluating the cross-efficiency of information sharing in supply chains. Expert Systems with Applications, 37(4): 2891–2897

121 Yu V F, Hu K J (2014). An integrated approach for resource allocation in manufacturing plants. Applied Mathematics and Computation, 245: 416–426

122 Zerafat Angiz M, Mustafa A, Kamali M J (2013). Cross-ranking of decision making units in data envelopment analysis. Applied Mathematical Modelling, 37(1–2): 398–405

123 Zopounidis C, Doumpos M (2002). Multicriteria classification and sorting methods: A literature review. European Journal of Operational Research, 138(2): 229–246

124 Zoroofchi K H, Farzipoor Saen R, Lari P B, Azadi M (2018). A combination of range-adjusted measure, cross-efficiency and assurance region for assessing sustainability of suppliers in the presence of undesirable factors. International Journal of Industrial and Systems Engineering, 29(2): 163–176

125 Zoroufchi K H, Azadi M, Farzipoor Saen R (2012). Developing a new cross-efficiency model with undesirable outputs for supplier selection. International Journal of Industrial and Systems Engineering, 12(4): 470–484