Specification of a performance indicator using the evidential-reasoning approach

Knowledge-Based Systems 92 (2016) 138–150

Contents lists available at ScienceDirect

Knowledge-Based Systems journal homepage: www.elsevier.com/locate/knosys

Specification of a performance indicator using the evidential-reasoning approach Jian Cao a,b, Guanghua Chen c, Mohammad Khoveyni d, Robabeh Eslami e, Guoliang Yang f,∗ a

Institute of Psychology, Chinese Academy of Sciences, Beijing 100101, China College of Humanities & Social Sciences, University of Chinese Academy of Sciences, Beijing 100039, China c Donlinks School of Economics and Management, University of Science and Technology Beijing, Beijing 100083, China d Department of Applied Mathematics, College of Basic Sciences, Yadegar-e-Imam Khomeini (RAH) Shahr-e-Rey Branch, Islamic Azad University, Tehran, Iran e Department of Mathematics, Faculty of Technology and Engineering, South Tehran Branch, Islamic Azad University, Tehran, Iran f Institute of Policy and Management, Chinese Academy of Sciences, Beijing 100190, China b

a r t i c l e

i n f o

Article history: Received 10 June 2015 Revised 17 October 2015 Accepted 20 October 2015 Available online 28 October 2015 Keyword: Data envelopment analysis Efficient frontier Anti-efficient frontier The evidential-reasoning approach

a b s t r a c t There are three primary encountered problems in classic data envelopment analysis (DEA), which they decrease the effectiveness and reliability of decision making based on the obtained information from the classic DEA. These three problems include the following issues: (1) DEA efficiency scores overestimate efficiency and they are biased; (2) In certain cases, the standard DEA models are not as useful as expected in the sense of discriminating the decision making units (DMUs); (3) Specification of the evaluated DMUs as efficient by using DEA are peculiar rather than superiority. Tackling these mentioned problems is the motivation for creating this current study. To overcome these three problems in DEA together and enhance the effectiveness and reliability of the decision-making process, this paper uses the evidential-reasoning (ER) approach to construct a performance indicator for combining the efficiency and anti-efficiency obtained by DEA and inverted DEA models, which they are used to identify the efficient and anti-efficient frontiers, respectively. Numerical simulation tests indicate that our new performance indicator is more suitable for the cases where there are relatively few DMUs to be evaluated with respect to the number of input and output indicators. Furthermore, empirical studies demonstrate that this indicator has considerably more discrimination power than that of the standard DEA models, and also it reduces overestimation and addresses peculiar DMUs, properly. © 2015 Elsevier B.V. All rights reserved.

1. Introduction Data envelopment analysis (DEA), which first introduced by Charnes et al. [13], has been widely used in productivity or performance evaluation and also the efficiency analysis of many businesses and non-profit organizations. The core idea of the classic DEA is first to identify the production frontier and then, the decision making units (DMUs) on the frontier will be regarded as efficient. Those DMUs that are not on the frontier will be compared with their peers or projections on the frontier to measure their relative efficiencies. All of the DMUs on the frontier are considered to represent the best practices and they have the same level of performance. However, DEA efficiency scores usually overestimate the efficiency and they are biased [9]. Smith [38] argued that the classic DEA always overestimates the true efficiencies, and the main reason for the overestimation is that many inefficient units have been incorrectly classified as efficient by the classic DEA. The extent of the ∗

Corresponding author. Tel.: +86 10 59358816. E-mail address: [email protected] (G. Yang).

http://dx.doi.org/10.1016/j.knosys.2015.10.023 0950-7051/© 2015 Elsevier B.V. All rights reserved.

overestimation is dependent on the sample size and the complexity of the production process (as indicated by the number of inputs and outputs). This problem is denoted as overestimation in this paper. Second, as we know, one of the main advantages of DEA is to allow the DMUs to have full flexibility to select their most favorable weights for their assessments to achieve the maximum efficiency scores. This full flexibility of selecting weights is important for identifying inefficient DMUs. However, this full flexibility may reduce the discrimination capacity of DEA in the sense that there are often many DMUs on the frontier, which they cannot be ranked further in the classic DEA models. Entani et al. [21] noted that the number of evaluating DMUs as efficient will increase combinationally as the dimensions of inputs and outputs increase. When there are many input and output variables and only a few DMUs, decision makers may find that most DMUs are efficient. Adler et al. [1] argued: “Often decision-makers are interested in a complete ranking, beyond the dichotomized classification, to refine the evaluation of the units.” This problem is denoted as discrimination in this context. Third, Entani et al. [21] argued that some of the evaluated DMUs as efficient by using DEA are peculiar rather than superiority. In their

J. Cao et al. / Knowledge-Based Systems 92 (2016) 138–150

example with crisp data, they showed that peculiar DMUs, which are the intersection of efficient and anti-efficient frontiers, should not be evaluated as efficient, because the widths of their efficiency intervals are very large, and other superior DMUs (identified as efficient in their example) can dominate them. This problem is denoted as dealing with peculiar DMUs. These three main encountered problems in the classic DEA models, which are based on the distances to efficient frontiers, will decrease the reliability of decision making. There are already numerous published studies related to these three problems. However, most of these studies cannot solve the three problems together in a single framework. See Section 2 for details. This paper aims to provide a new idea to overcome some extent these three problems together including overestimation, discrimination and the handling of peculiar DMUs to rank DMUs using the evidential-reasoning approach (ER approach) (see, e.g., [47,48]) to combine the efficient and anti-efficient scores. The main contribution of this paper is to consider the efficient and anti-efficient scores as two pieces of evidence and then use the ER approach based on the evidence theory to combine two pieces of evidence for each DMU to: (1) Lower the overestimation; (2) Increase the discrimination power; (3) Deal with peculiar DMUs properly within a single framework. The remainder of this paper is organized as follows. Section 2 summarizes the related literatures for addressing the above three problems. We discuss about the efficient and anti-efficient frontiers in DEA models in Section 3. In Section 4, we first give a basic mathematical recall regarding the ER approach, and then we transform the efficiency scores or anti-efficiency scores to two pieces of evidence for combining the obtained information from both the best and worst viewpoints. Then, in this section, we conduct a numerical simulation process to test the performance of the proposed approach in this paper. In Section 5, we first provide an empirical example to illustrate the features of the ER approach, and then we perform a case study to examine the performance of the generated results from the ER approach. Finally, some conclusions are presented in Section 6. 2. Literature review 2.1. Overestimation Banker [9] recognized in theoretical work that DEA efficiency scores overestimate efficiency and they are biased for a finite example. He argued that the efficient frontier is biased below the true efficient frontier for a finite sample size. When the sample size is small, the efficiency scores of DMUs are considerably higher than their true efficiency scores. Smith [38] reported that “In the deterministic setting assumed here, using a convex production function, a well-specified DEA model will always overestimate efficiency. However, the extent of the overestimate is highly dependent on sample size. In effect, a larger sample size increases the possibility of encountering DMUs close to the production frontier, and therefore the DEA frontier approaches the true frontier asymptotically as sample size increases. For example, using the two input model, the average overestimate reduces from an average of 31% with samples of size 10 to just 8% as sample size increases to 80.” Alirezaee et al. [5] showed that the high average efficiency is the result of assuming that the units in the efficient set are 100% efficient. Galagedera and Silvapulle [23] contributed to this issue by investigating the sensitivity of DEA efficiency estimates to include inappropriate and/or by omitting several important variables in a large-sample DEA model. They found that DEA tends to overestimate the efficiency in nearly all production units for constant and decreasing returns to scale (RTS) processes with irrelevant inputs. The statistical properties of the nonparametric estimators were determined by the models developed by Kneip et al. [25] and Simar and Wilson [35], and they demonstrated that the speed of convergence of DEA estimators relies on (1) the smoothness of the unknown frontier and (2) the number of inputs and outputs

139

relative to the number of observations. If the number of variables is relatively large, the estimators show have very low rates of convergence, and a rather large quantity of data will be needed to avoid substantial variance and very wide confidence interval estimates. To avoid the dimensionality problem, Simar and Wilson [35] suggested that the number of observations should increase exponentially with the addition of variables. According to the Simar and Wilson bootstrap results, even the simple case with a single input and single output requires at least 25 observations and preferably more than 100 for the confidence intervals of the efficiency estimator to be nearly exact. Moreover, Banker [10], Simar and Wilson [36] and Pastor et al. [31] suggested statistical tests to measure the relevance of inputs or outputs and tests for considering potentially aggregating inputs or outputs. Simar and Wilson [37] also noted when the number of inputs and outputs is large, the imprecision of the results will be reflected in large bias, large variances and wide confidence intervals. However, large samples are not generally available in practice, and researchers try to handle small multivariate data sets. Hence, it is required to find alternative methods to avoid efficiency overestimation in DEA models to some extent in the case where there are limited observations. 2.2. Discrimination Cooper et al. [16] proposed a rule of thumb, for the number of required DMUs in DEA models, as

n ≥ max{m × s, 3(m + s)}, where n is the number of DMUs, and m and s are the number of inputs and outputs, respectively. However, the rule above is sometimes violated in reality in the case of a small sample of DMUs with many input and output variables. In this case, many DMUs will often be categorized as efficient DMUs in the standard DEA models, which they are not as useful as expected in the sense of discriminating the DMUs. Therefore, many researchers have attempted to improve the discrimination power of the standard DEA models. Podinovski and Thanassoulis [32] pointed out that there are some approaches which they can be used to improve the discrimination of DEA. It includes some simple methods (e.g., the aggregation of inputs or outputs, the use of longitudinal data) and more advanced methods (e.g., the use of weight restriction, production trade-offs, the use of selective proportionality between the inputs and outputs). Adler and Golany [2,3] suggested using principal components to improve discrimination in DEA with minimal loss of information. Despotis [18] introduced the global efficiency approach as a means to improve DEA discrimination power. Jenkins and Anderson [24] used the partial covariance analysis to choose a subset of variables for increasing the discrimination of DEA. Adler and Yazhemsky [4] applied Monte Carlo simulation to generalize and compare two discrimination improving methods (Principal component analysis applied to DEA and variable reduction based on partial covariance). In our view, there are three main approaches to improve discrimination in current DEA literatures. The first approach requires preferential or prior information of decision makers to increase the discrimination ability of DEA models, e.g., some scholars have developed the weights restriction [6,15,20,39] or preference change methods [28,29,51] to incorporate the decision makers’ value judgments into DEA models. Although this approach can increase the discrimination power of DEA significantly, it needs more prior information on preference of decision makers. Furthermore, this approach cannot solve the problems of overestimation well. The second popular approach is the super-efficiency method, which it obtains the score of the DMU being evaluated by excluding itself from the reference set [8,40]. The superefficiency method can also increase the discrimination power of DEA models, but it fails to address the overestimation and peculiar DMUs. Additionally, it is clear that this model uses different reference sets to evaluate the efficient DMUs and inefficient DMUs. Furthermore,

140

J. Cao et al. / Knowledge-Based Systems 92 (2016) 138–150

Banker and Chang [11] reported that the procedure using the superefficiency scores for ranking efficient observations proposed by Andersen and Petersen’s [8] has poor performance. The third approach is the cross-efficiency method in which DMUs are evaluated by both itself and its peers [19,26,27,33,42–44,46]. Unfortunately, in our view, the cross-efficiency scores have moved far away from the basic principle of DEA. For example, all of the DMUs in fact use the same weights to compute their cross-efficiency scores in the case of one standard input variable. However, one unique feature of DEA is that it allows the assessed DMUs to assign their most favorable weights to maximize their scores in the assessments. Furthermore, cross-efficiency cannot solve the problem of overestimation. 2.3. Dealing with peculiar DMUs (interval DEA) To increase the reliability of decision making by DEA, Entani et al. [21] developed a new DEA in which peculiar DMUs are addressed properly with an interval efficiency, which consists of efficiencies obtained from the optimistic and pessimistic viewpoints. Although Entani et al. [21] proposed the inverted DEA model from the pessimistic viewpoint, there are no relations between the efficiency of DEA and the anti-efficiency of inverted DEA model because they are essentially obtained by different models. The earliest work on this idea of DEA model from the anti-efficient viewpoint can be traced to “inverted” DEA model proposed by Yamada et al. [45]. In comparison with the standard DEA models which evaluate DMUs from the perspective of efficient frontier, “inverted” DEA model is to evaluate the performance of DMUs from the perspective of anti-efficient frontier. To this end, Entani et al. [21] proposed an interval DEA in which the efficiency of DMUs can be described as an efficiency interval, where the upper and lower limits are obtained from the maximizing and minimizing problems, respectively. They argued that two end points from the maximizing and minimizing problems can construct an efficiency interval. To evaluate DMU0 by an efficiency interval denoted as [θ0∗ , θ0∗ ], they used the preference relations for interval values. They defined a partial-order relation of intervals based on the relations of interval values. Unfortunately, their method decreases the discrimination of DEA obviously. Amirteimoori [7] argued that Entani et al. [21] were not explicitly interested to obtain the measures of efficiency and super-efficiency by considering both efficient and anti-efficient frontiers. Therefore, Amirteimoori [7] defined an alternative efficiency measure using the information from both efficient and anti-efficient frontiers. Then, he used slacks-based DEA and inverted DEA models to measure the weighted L1–distances from DMU0 to both an efficient and antiefficient frontier. 1 His idea is very brilliant as it provides the possibility to use more information from the existing dataset than traditional DEA. However, his work has several drawbacks: (1) The efficiency scores of these DMUs on the efficient frontier and anti-efficient frontier are 1 and −1, respectively, and this combined efficiency measure cannot overcome overestimation or improve the discrimination power of DEA models because those on the frontier will still have the same unit score; (2) The new efficiency measure cannot address peculiar DMUs properly, and they will be categorized as efficient DMUs; (3) His efficiency measure does not fully consider the decision maker’s preference structure, and it needs to be justified carefully before we could combine the information from efficient and anti-efficient frontiers. These three main aforementioned problems will decrease the reliability of decision making. To these ends, this paper aims to overcome these three problems together including overestimation, discrimination and the handling of peculiar DMUs to rank DMUs using a new idea for combining the obtained information from efficient

1

Note: There are some errors on inequalities in model (8) in Amirteimoori [7].

and anti-efficient scores. Yang et al. [46] argued that when a decision maker’s preference structure does not meet the “additive independence” condition [22], a new nonlinear aggregation method, called the ER approach, should be used to aggregate the cross-efficiency after transforming them into pieces of evidence. Thus, in the case of decision maker’s preference structure failing to meet the “additive independence”, in this paper, we will use the ER approach to combine the efficiency and anti-efficiency from both classic DEA and inverted DEA models if we transform them into pieces of evidence. The ER approach describes each attribute as an alternative by a distributed assessment via a belief structure and provides a new procedure for aggregating multiple attributes based on the distributed assessment framework and the evidence combination rule of Dempster–Shafer (D–S) evidence theory. This approach can overcome the aforementioned problems to some extent in the classic DEA and enhance the reliability of decision making. The ER approach has been used in many real multi-attribute decision analysis (MADA) problems and it has the advantage of aggregating the mutual independent pieces of evidence (see, e.g., [14,41,47,48]). 3. Efficient and anti-efficient frontiers As mentioned in Section 2, Amirteimoori [7] has proposed the idea to combine the obtained information from both efficient and anti-efficient frontiers through an alternative efficiency measure. Here, we briefly restate the description of efficient and anti-efficient frontiers and illustrate them intuitively. We consider the pairs of positive input and output vectors (x j , y j )( j = 1, 2, . . . , n) of n DMUs such that xj ≥ 0, xj = 0 and yj ≥ 0, yj = 0 for j = 1, 2, . . . , n. We call a pair of such non-negative input x ∈ Rm and output y ∈ Rs an activity and represent them by the notation (x, y). The set of feasible activities is called the production possibility set (PPS) and defined as PPS = {(x, y)|y can be produced from x}. The boundary of PPS is referred to as production frontier. DMUs, which are technically efficient, operate on the frontier, while those technically inefficient DMUs operate at points in the interior of PPS. Thus, it is rational to rank DMUs according to their distances to the production frontier. We postulate the properties of PPS as below: (P1) The observed activities (x j , y j )( j = 1, 2, . . . , n) belong to PPS. (P2) If (x, y) ∈ PPS, then (tx, ty) ∈ PPS for any positive scalar t. This property is called the constant returns to scale (CRS) assumption. ¯ y¯ ) ∈ PPS for x¯ ≥ x and y¯ ≤ y. This property (P3) If (x, y) ∈ PPS, then (x, is called the free disposability assumption. (P4) Any semi-positive2 linear combination of activities in PPS belongs to PPS. Then, the PPS is specified by using (P1), (P2), (P3), and (P4) as follows:



(x, y)|

PPS =

n 

λ j x j ≤ x,

j=1

n 



λ j y j ≥ y, λ j ≥ 0, j = 1, 2, . . . , n .

j=1

Let us denote the efficiency interval for DMU0 as [θ0∗ , θ0∗ ], where θ 0∗ and θ0∗ are obtained from the same optimization problems where

one is the maximizing problem and the other is the minimizing problem. The feasible set with a similar role to the production possibility set under the CRS assumption of DEA is defined as PH by PH = PPS ∩PI , where PI is denoted as follows:



PI =

(x, y)|

n  j=1

λ j x j ≥ x,

n 



λ j y j ≤ y, λ j ≥ 0, j = 1, 2, . . . , n .

j=1

In order to more explanation about the three sets PPS, PI , and PH , we use Fig. 1 which shows PPS, PI , and PH for a single input and a 2

A vector x is called semi-positive if x ≥ 0 and x = 0.

J. Cao et al. / Knowledge-Based Systems 92 (2016) 138–150

141

Fig. 2. Efficient and anti-efficient frontiers.

If we consider the slacks of inputs and outputs, we can transform model (2) into the following model:

Fig. 1. A visual description of PPS, PI , and PH in two dimensions.



single output case (i.e., m = s = 1). As depicted in Fig. 1, PH is the intersection of two sets PPS and PI . Based on these observations (xj , yj ), DEA models can construct a piecewise linear production frontier, which is a non-parametric estimation of the unobservable true frontier. Then, DEA models measure the efficiency of a DMU based on its distance to the estimated frontier. The DEA model (CCR type) is as follows:

h∗classic = min

θ

= max

θ −ε

m  i=1

s− i

+

s 



s+ r

r=1

⎧n − j=1 xi j λ j − si = θ xi0 , i = 1, . . . , m, ⎪ ⎪ n ⎪ ⎪ ⎨ j=1 yr j λ j + s+r = yr0 , r = 1, . . . , s s.t. λ j ≥ 0, j = 1, . . . , n ⎪ ⎪ − ⎪ ⎪ ⎩si ≥ 0, i = 1, . . . , m

(3)

s+ r ≥ 0, r = 1, . . . , s

⎧n ⎨ j=1 xi j λ j ≤ θ xi0 , i = 1, . . . , m, n yr j λ j ≥ yr0 , r = 1, . . . , s s.t. ⎩ j=1 λ j ≥ 0, j = 1, . . . , n

(1)

where h∗classic measures the degree of efficiency by radial measurement. Banker [9] provided a formal statistical foundation for DEA. He argued that while the efficient frontier is biased and below the true efficient frontier in the case of a finite sample size, the bias approaches zero for large samples. However, the estimated frontier could be far away from the true one when the sample size is small, so that the efficiency scores of DMUs are considerably higher than their true efficiency scores. For example, although some DMUs are very far from the true frontier, they can lie on the estimated frontier and cannot be discriminated in DEA models. To overcome this problem, we utilize information from the anti-efficient frontier. The anti-efficient frontier is the worst practice frontier generated from the worst practice DMUs. We can simply treat the inputs and outputs of DMUs as undesirable variables and use some DEA models with undesirable inputs and outputs to determine the anti-efficient frontier. The idea is simple: if the inputs and outputs of DMUs are all undesirable, one should maximize the inputs and minimize the outputs to find the efficient frontier. Therefore, using radial measurement and input orientation, we can have the following inverted DEA (CCR type) model:

h∗inverted = max

h∗inverted

θ

⎧n ⎨ j=1 xi j λ j ≥ θ xi0 , i = 1, . . . , m, n yr j λ j ≤ yr0 , r = 1, . . . , s s.t. ⎩ j=1 λ j ≥ 0, j = 1, . . . , n

(2)

Here, h∗inverted measures the degree of inefficiency or antiefficiency by radial measurement.

where ɛ is non-Archimedean construct. Based on the above inverted DEA models, we can have the antiproduction possibility set (APPS), which is the same as PI in Section 2. Definition 1. We define the anti-efficient frontier as the boundary of PI . Definition 2. DMU0 is strongly anti-efficient if and only if h∗inverted = = s+∗ 1 and s−∗ r = 0 in model (3). i Definition 3. DMU0 is weakly anti-efficient if and only if h∗inverted = 1 in model (2). Obviously, if h∗inverted = 1 in model (2), then the DMU is on the anti-efficient frontier. It should be noted that (h∗inverted )−1 is called the anti-efficiency score in the following sections. We assume there are only two inputs and a single output to illustrate the geometric meanings of the efficient and anti-efficient frontiers intuitively. As Fig. 2 shows, contrary to the DEA models using the best practice DMUs A, B, C, and F to produce the efficient frontier, inverted DEA model (2) uses the worst practice DMUs A, D, E, and F to form the anti-efficient frontier. Therefore, we can evaluate the DMU and compare it with virtual or real DMUs located on both efficient and anti-efficient frontiers. For instance, we can identify the peers P and P of DMU P, which are separately located on efficient and antiefficient frontiers, respectively. Thus, we can measure the distances from P to P (or P ) by using radial measurement OP /OP (or OP /OP). The larger OP /OP, the DMU P under evaluation, is closer to the efficient frontier and performs better. Meanwhile, the larger OP /OP, DMU P, is farther from the anti-efficient frontier and performs better. As for these efficient DMUs A, B, C, and F, DMUs A and F are peculiar DMUs and can be regarded as worse than the other efficient DMUs on the efficient frontier because DMUs A and F are also on the antiefficient frontier. Therefore, we can obtain more information regarding the performance of DMUs by utilizing the anti-efficient frontier generated by the worst practice DMUs. Thus, the discrimination power for DEA

142

J. Cao et al. / Knowledge-Based Systems 92 (2016) 138–150

analysis can be enhanced. Furthermore, we can see that DMUs A and F are the intersections of efficient and anti-efficient frontiers. They are evaluated as both efficient and anti-efficient. As argued by Entani et al. [21], these two DMUs are peculiar rather than superior and they should be addressed properly. From Fig. 2, we can see that the performance of DMUs A and F should be lower than other efficient DMUs if we consider the information from the anti-efficient frontier. In Section 3, we have illustrated the efficient and anti-efficient frontiers in DEA models. It is possible to combine the information from efficient and anti-efficient frontiers to increase the discrimination power of DEA models. Yang et al. [46] proposed the idea of transforming the efficiency in DEA into pieces of evidence such that the decision maker’s preference does not satisfy the "additive independence" condition by using the ER approach to combine DEA efficiencies. The use of the ER approach is justified because this approach assumes only that the efficiency and anti-efficiency scores for a DMU are generated in a way independent of whether those other DMUs are known and they are conjunctively combined to obtain a joint score for the DMU. In Section 4, we will give a brief introduction to the D– S theory and the transformation and combination of efficiency and anti-efficiency scores using the ER approach.

Suppose there are L pieces of evidence ei (i = 1, . . . , L). The process of the ER approach can be listed as follows. Step 1: Definition of the decision-making problem. (1-1) Define a set of L pieces of evidence ei as follows:

E = {e 1 , e 2 , . . . , e L }

(4)

We suppose that the above evidence set E includes all of the factors related to the assessment of the DMU to be evaluated. (1-2) Set the weights for these L pieces of evidence W = {ω1 , ω2 , . . . , ωL }, which satisfies L 

0 ≤ ωi ≤ 1 and

ωi = 1.

(5)

i=1

(1-3) We define G distinctive assessment grades Hg (g = 1, . . . , G) as a complete set of standards for assessing each DMU.

 = {H1 , H2 , . . . , Hg , . . . , HG }

(6)

(1-4) We can represent the decision-making problem by using the following distributions for a DMU0 on the piece of evidence ei (i = 1, . . . , L):



S(ei ) = Hg , βg,i , g = 1, . . . , G , i = 1, ldots, L

4. Combining efficiency and anti-efficiency

where β g, i represents a degree of belief and satisfies

4.1. D–S theory of evidence and the ER approach

βg,i ≥ 0 and

In the 1960s, Dempster [17] first conducted research on the evidence theory. Subsequently, Shafer [34] formalized the evidence theory, which is also called the Dempster–Shafer (D–S) theory of evidence. He argued that the D–S theory addresses weights of evidence with numerical degrees of support based on evidence and focuses on the fundamental operation of combination of evidence. The main characteristic of the D–S theory is that it has the power to describe uncertainty and ignorance implied in mass function and reserved in the process of aggregating evidence. The heart of the D–S evidence theory is Dempster’s rule for effecting this combination. Using this rule, pieces of evidence from different sources can be aggregated together to determine the final overall belief distributions. The prerequisite of this rule is that information sources are independent. The Dempster’s combination rule of aggregation is proved by Shafer [34] to satisfy commutativity and associativity. Thus, we can see that the combination of evidence can be conducted in a pairwise way. Buchanan and Shortliffe [12] showed that the direct use of the aggregation rule would lead to an exponential increase in computational complexity. Zadeh [50] argued that Dempster’s rule might lead to counter-intuitive results in the case of aggregating conflicting pieces of evidence. Murphy [30] noted that the crude application of the D–S theory and the combination rule can lead to irrational conclusions in the process of aggregating multiple pieces of evidence in conflict. To overcome this problem, Yang and Singh [48] proposed operational algorithms based on the evaluation analysis model for evidence aggregation, which can reduce the computational complexity to linear time using the characteristics of the evidence combination process. Subsequently, Yang and Sen [47] and Yang and Xu [49] proposed the overall frame of the ER approach and improved it further. The main difference between the D–S evidence theory and the ER approach is that the latter can be used to aggregate highly or completely conflicting evidence without suffering from the well-known counterintuitive problem of the former D–S evidence theory [48]. Yang and Xu [49] investigated the nonlinear features of the ER approach by examining typical reasoning patterns in aggregating harmonic, quasiharmonic, and contradictory decision information. So far, there are many theoretical studies on the ER approach and dozens of innovative applications. The details of the D–S theory and the ER approach can be easily found in existing literature (see, e.g., [34,49]). Here, we give a brief mathematical recall of the ER approach.

G 

βg,i ≤ 1

(7)

(8)

g=1

Step 2: The probability assignments for each piece of evidence for DMU0 . Let mg, i be a basic probability mass representing the degree to which the ith piece of evidence ei supports a hypothesis that this evidence at DMU0 is assessed to the gth grade Hg . We also let m, i be a remaining probability mass unassigned to any individual grade after the evidence ei has been assessed. Variables mg, i and m, i can be calculated as follows:

mg,i = ωi βg,i , g = 1, . . . , G m,i = 1 −

G 

mg,i = 1 −

g=1

G 

(9)

ωi βg,i , i = 1, . . . , L

(10)

g=1

Variable m, i can be decomposed into the following two factors by using the following formula:

¯ ,i + m ˜ ,i m,i = m

(11)

where

¯ ,i = 1 − ωi m



˜ ,i = ωi 1 − m

G 

βg,i

(12) (13)

g=1

¯ ,i and m ˜ ,i represent the remaining In formula (11), variables m probability mass unassigned to individual grades caused by the evidence’s weight and caused by the incompleteness or ignorance of the assessment S(ei ), respectively. The set EI(i) = {e1 , e2 , . . . , ei } is defined as the set of the first i pieces of evidence. Here, assume that mg, I(i) is the collective belief in Hg from EI(i) , and also m, i is unassigned belief. Step 3: Combination of probability assignments for a piece of evidence at DMU0 . ¯ ,I(1) = m ¯ ,1 , m ˜ ,I(1) = We let mg,I(1) = mg,1 (g = 1, . . . , G), m ˜ ,1 and m,I(1) = m,1 . The combined probability assignments m ¯ ,I(L) , m ˜ ,I(L) , and m, I(L) can be generated by mg,I(L) (g = 1, . . . , G), m the following evidential-reasoning rule:

  {Hg } : mg,I(i+1) = KI(i+1) mg,I(i) mg,i+1 +mg,I(i) m,i+1 + m,I(i) mg,i+1 , g = 1, . . . , G

(14)

J. Cao et al. / Knowledge-Based Systems 92 (2016) 138–150

{} : m˜ ,I(i+1) = KI(i+1) (m˜ ,I(i) m˜ ,i+1

¯ ,I(i) m ˜ ,I(i) m ˜ ,i+1 + m ¯ ,i+1 ) +m

  {} : m¯ ,I(i+1) = KI(i+1) m˜ ,I(i) m¯ ,i+1  KI(i+1) = 1

1−

G G  

(15) (16)

mt,I(i) m j,i+1 , i = {1, 2, . . . , L − 1}

t=1 j=1, j=t

(17) The overall belief distribution on the frame of discernment  after aggregating L pieces of evidence is given as follows:

  {Hg } : βg = mg,I(L) / 1 − m¯ ,I(L) , g = 1, . . . , G

(18)

  {} : β = m˜ ,I(L) / 1 − m¯ ,I(L)

(19)

So, the overall assessment of DMU0 based on the set of evidence E for the frame of discernment  is as follows:

S(E ) = {(Hg , βg ), g = 1, . . . , G}

(20)

where E = {e1 , e2 , . . . , eL }. The details of the ER approach can refer to Yang and Sen [47], Yang and Singh [48] and Yang and Xu [49]. 4.2. Transformation and combination of efficiency and anti-efficiency In DEA models, a DMU is DEA efficient if it has the best possible relative efficiency of unity; otherwise, it is DEA inefficient. On the contrary, in the inverted DEA model, if a DMU has the worst possible relative efficiency of unity, then it is DEA Anti-efficient; otherwise, it is DEA Anti-inefficient. Anti-inefficiency denotes a status that the DMU to be evaluated is not the worst possible practice on Antiefficient frontier. Yang et al. [46] proposed the idea of transforming the efficiency in DEA into pieces of evidence. The core of the idea is to define two mutually exclusive statuses named “Efficient” and “Not efficient”, and then the DEA efficiency is considered as the belief degree to these two statuses. Please note that in this paper, “Not efficient” is a different concept from “inefficient”. “Not efficient” denotes the case where the efficiency score approaches zero, which is the worst possible relative efficiency of the DMU to be evaluated. Similarly, we can transform the Anti-efficiency obtained from the inverted DEA model into another piece of evidence. Namely, we let “Not efficient” also denote the case where the Anti-efficiency score is unity, which is also the worst possible relative efficiency of DMU from a pessimistic viewpoint. We can see that these two evaluation grades are mutually exclusive and collectively exhaustive for the assessment. The efficiency or anti-efficiency scores of DMUs can be viewed as the belief distributions on both “Efficient” and “Not efficient”. For instance, if the efficiency score of one certain DMU in a classic DEA model is 0.7, then the belief distribution is {(Efficient, 0.7), (Not efficient, 0.3)}. This distribution means that the DMU evaluated is efficient with 70% possibilities and is not efficient with 30% possibilities. Similarly, we can address the anti-efficiency from the inverted DEA model: e.g., if the anti-efficiency score of one certain DMU is 0.6, then the distribution is {(Efficient, 0.4), (Not efficient, 0.6)}. In other words, it is efficient with 40% possibilities and is not efficient with 60% possibilities. Thus, we have two pieces of evidence generated in an independent way for each DMUj from two viewpoints from efficient and anti-efficient frontiers, respectively. Remark 1. Please note that an assumption is implied in the process of transforming efficiency (or anti-efficiency) into a piece of evidence. That is, any efficiency score (or anti-efficiency score) can be expressed by a linear combination of two statuses “Efficient” and “Not efficient”.

143

We can justify this assumption easily. Taking the efficiency score as an example, its range is the following interval (0, 1], which is exactly the range of the linear combination of the two statuses “Efficient” and “Not efficient”. Similarly, we can also transform the anti-efficiency score into the belief distribution of these two statuses. Next, we will use the ER approach to combine the obtained efficiency and anti-efficiency scores from the DEA model and inverted DEA model. It is appropriate to employ the ER approach because it only assumes that the efficiency and anti-efficiency for a DMU are generated in a way independent of whether efficiencies or antiefficiencies for other DMUs are known. See the following steps. Step 1: We let E j = {e j1 , e j2 } denote two pieces of evidence for DMU j ( j = 1, . . . , n), where ej1 and ej2 represent the evidence from efficiency score h∗classic, j (Model 1) and anti-efficiency score

(h∗inverted, j )−1 (Model 3) of DMU j ( j = 1, . . . , n), respectively.

Step 2: We suppose the weights of two pieces of evidence are given by W = (ω1 , ω2 )T , where ω1 , ω2 are the relative weight of the two pieces of evidence with 0 ≤ ω1 , ω2 ≤ 1 and ω1 + ω2 = 1. Weights can be determined by using several existing methods [52]. Step 3: We suppose that there are two distinctive evaluation grades H1 and H2 , that provide a complete set of grades for evaluation  = {H1 , H2 }, where H1 and H2 represent the grades of “Efficient” and “Not efficient”, respectively. Thus, we can represent the given assessment for ej1 and ej2 of DMU j ( j = 1, . . . , n) as the following distribution mathematically:













S e j1 = S e j2 =



 



H1 , β j,1,2 , H2 , β j,2,2

β j,1,1 =

where

 

H1 , β j,1,1 , H2 , β j,2,1

1 − (h∗inverted, j )−1 ,

β j,2,1 = β j,2,2 = (h∗inverted, j

h∗classic, j ,

(21) (22) 1 − h∗classic, j , )−1 . Namely,

and

β j,1,2 =

we

consider

that the efficiency score h∗classic, j of DMU j ( j = 1, . . . , n) represents the belief of “Efficient”, and 1 − h∗classic, j represents the belief of

“Not Efficient”. Similarly, the anti-efficiency score (h∗inverted, j )−1

of DMU j ( j = 1, . . . , n) represents the belief of “Not Efficient”, and 1 − (h∗inverted, j )−1 represents the belief of “Efficient”. Here, we know β j, 1, 1 ≥ 0, β j, 2, 1 ≥ 0, β j,1,1 + β j,2,1 = 1 and β j, 1, 2 ≥ 0, β j, 2, 2 ≥ 0, β j,1,2 + β j,2,2 = 1. Step 4: From the ER approach mentioned above, we know the mass function satisfies the following formulas:

  ω1 β j,1,1 ; m j,2,1 = ω1 β j,2,1 ; m j,,1 = 1 − m j,1,1 + m j,2,1 ;   = ω2 β j,1,2 ; m j,2,2 = ω2 β j,2,2 ; m j,,2 = 1 − m j,1,2 + m j,2,2 .

m j,1,1 = m j,1,2

where j = 1, . . . , n. Variables mj, 1, i and mj, 2, i (i = 1 or 2) are called the basic belief degree of evidence eji on H1 and H2 of DMU j ( j = 1, . . . , n) in the frame of discernment , and mj, , i (i = ¯ j,,i + m ˜ j,,i , 1 or 2) is called the unassigned belief. Let m j,,i = m ¯ j,,i = 1 − ωi is the unassigned belief where i = 1, 2. Variable m caused by weights, and variable



˜ j,,i = ωi 1 − β j,1,i − β j,2,i m



(23)

reflects the unassigned belief caused by ignorance. Step 5: We can combine the efficiency and anti-efficiency scores through the following aggregation rule (see, e.g., [47–49]): We define mj, 1, I(2) and mj, 2, I(2) as the collective belief degree in H1 and H2 from two pieces of evidence. Thus, we have

  {H1 } : m j,1,I(2) = K m j,1,1 m j,1,2 + m j,1,2 m j,,1 + m j,,2 m j,1,1

(24)

  {H2 } : m j,2,I(2) = K m j,2,1 m j,2,2 + m j,2,2 m j,,1 + m j,,2 m j,2,1

(25)

  {} : m˜ j,,I(2) = K m˜ j,,1 m˜ j,,2 + m¯ j,,1 m˜ j,,2 + m˜ j,,1 m¯ j,,2 (26)

144

J. Cao et al. / Knowledge-Based Systems 92 (2016) 138–150 Table 1 Numerical experiments in case of 10 DMUs. DMUs

DMU1 DMU2 DMU3 DMU4 DMU5 DMU6 DMU7 DMU8 DMU9 DMU10

True efficiency

0.7753 0.3677 0.4671 0.6901 0.4720 0.9489 0.7225 0.5712 0.0648 0.6160

CCR efficiency

Anti-CCR efficiency

H1

H2

H1

H2

1.0000 0.7629 0.7512 1.0000 0.7977 1.0000 0.9874 0.9158 0.0797 1.0000

0.0000 0.2371 0.2488 0.0000 0.2023 0.0000 0.0126 0.0842 0.9203 0.0000

0.8066 0.1125 0.8313 0.6339 0.7931 0.9198 0.8212 0.6920 0.0000 0.7492

0.1934 0.8875 0.1687 0.3661 0.2069 0.0802 0.1788 0.3080 1.0000 0.2508

Performance indicator PI

P1

| P1|

P2

P3

| P3|

0.9311 0.4186 0.8277 0.8610 0.8313 0.9725 0.9312 0.8428 0.0273 0.9088

20.09% 13.85% 77.19% 24.76% 76.14% 2.49% 28.89% 47.54% −57.90% 47.52%

20.09% 13.85% 77.19% 24.76% 76.14% 2.49% 28.89% 47.54% 57.90% 47.52%

28.98% 107.47% 60.81% 44.90% 69.02% 5.38% 36.67% 60.32% 22.95% 62.33%

4.04% −69.40% 77.97% −8.14% 68.03% −3.07% 13.67% 21.15% −100% 21.62%

4.04% 69.40% 77.97% 8.14% 68.03% 3.07% 13.67% 21.15% 100% 21.62%

28.06%

39.64%

49.88%

2.59%

38.71%

Average Note: P1 = (Performance scores − True efficiency)/True efficiency. |P1| denotes the absolute value of Pe1. P2 = (CCR efficiency − True efficiency)/True efficiency. P3 = ((1-Anti-CCR efficiency) − True efficiency)/True efficiency. |P3| denotes the absolute value of P3.

  {} : m¯ j,,I(2) = K m¯ j,,1 m¯ j,,2

(27)

where j = 1, . . . , n. Variable K is the normalization factor and is determined by the following formula:



K = 1/ 1 − m j,1,1 m j,2,2 − m j,2,1 m j,1,2



(28)

Thus, after combining two pieces of evidence, the overall belief distribution for the DMU j ( j = 1, . . . , n) on the frame of discernment  is as follows:

  {H1 } : β j,1 = m j,1,I(2) / 1 − m¯ j,,I(2)   {H2 } : β j,2 = m j,2,I(2) / 1 − m¯ j,,I(2)   {} : β j, = m˜ j,,I(2) / 1 − m¯ j,,I(2)

(29) (30) (31)

Therefore, for DMU j ( j = 1, . . . , n), the overall support function after aggregating these two pieces of evidence on the frame of discernment  is as follows:

S j (E ) =



 

H1 , β j,1 , H2 , β j,2



(32)

which reads that DMU j ( j = 1, . . . , n) is assessed to the grade H1 and grade H2 with the degree of belief of β j, 1 and β j, 2 , respectively. Based on formula (32), we can use the belief β j, 1 of H1 as the performance indicator (PI) of DMU j ( j = 1, . . . , n) and rank these DMUs to be evaluated. 4.3. Numerical simulation experiments Smith [38] used a Monte Carlo simulation to generate a large number of observations and tested the robustness of the DEA efficiency estimator. Similar to his work and for simplicity, in this subsection, we utilize a well-known Cobb–Douglas production process in which three inputs, represented by x1j , x2j , and x3j are used to produce a single output yj of DMUj . Thus, we have

yj = β

3 

xai ji η j ,

(33)

i=1

 where 3i=1 ai = 1 and ηj ∈ [0, 1] is the efficiency of DMUj and β and ai are known parameters. Formula (33) is a simple Cobb–Douglas function with constant returns to scale and incorporates a technical inefficiency factor ηj . An equivalent form of Eq. (33) is as follows:

 

log y j = log(β) +

m 

 

ai log xi j + E j ,

i=1

where E j ∈ (−∞, 0) is the natural logarithm of ηj .

(34)

It is necessary to specify the statistical distribution for variables log(yj ), log(xij ) and Ej . For log(xij ), we choose to assign normal distributions with unit variances. Variable Ej is drawn from a half-normal distribution with unit variance. Further, we assume that the three inputs have equal importance, and ai = 1/3. Variable β is set equal to one, and log(β) = 0. As proposed by Cooper et al. [16] rule of thumb, the number of DMUs required in DEA models should be n ≥ max{m × s, 3(m + s)}, where m and s represent the numbers of inputs and outputs, respectively. For three inputs and one single output, in our numerical experiments, we first assume that there are 10 DMUs and report the results as shown in Table 1. The true efficiency of each DMU is listed in the second column. Additionally, we use a classic input-based CCR model (1) and input-based inverted CCR model (3) to obtain the CCR efficiency and anti-CCR efficiency, respectively, and transform them into belief distributions as shown in Columns 3 and 4 and Columns 5 and 6. We set the CCR efficiency and anti-CCR efficiency to equal weights, and the PI scores are listed in Column 7. To see the gap between true efficiency and CCR efficiency and PI score and Anti-CCR efficiency, we list five ratios named P1, |P1|, P2, P3, and |P3|. See Table 1 for details. From Table 1, we can see that CCR efficiency overestimates the true efficiency in this example at an average rate of 49.88%. However, the average rate of scores for the performance indicator PI proposed in this paper overestimates the true efficiency at an average rate of 28.06%. If we consider the absolute value of P1, we can see that the bias is at an average rate of 39.64%. We can easily see that either P1 or |P1| features lower bias than P2. Moreover, we can see that the PI discriminates the four CCR-efficient DMUs (DMU1 , DMU4 , DMU6 , DMU10 ), whose CCR efficiencies are all one. Besides, as seen in Table 1, the absolute value of P3, i.e., |P3|, overestimates the true efficiency in this example at an average rate of 38.71%, hence P1 also features lower bias than |P3|. To test the robustness of this argument, we repeat the experiments above 10 times, and the results are listed in Table 2. From Table 2, we can see that in this case, the average bias of P1,|P1|,P2, P3, and|P3|of 10 numerical experiments are 18.98%, 35.08%, 62.52%, -22.58%, and 49.01%, respectively. This fact shows that our performance indicator features lower bias than CCR efficiency and Anti-CCR efficiency. We further conduct the experiments above 1000 times and also we calculate the average bias of P1, |P1|, P2, P3, and |P3| for some categories of DMUs in Column 2 through Column 10 in Table 3. For instance, as shown Column 3 in Table 3, the average rates of P1, |P1|, P2, P3, and |P3| are respectively 17.81%, 33.35%, 73.84%, −34.74%, and 54.10% for the case of 10 DMUs. Thus, we can conclude that, in this

J. Cao et al. / Knowledge-Based Systems 92 (2016) 138–150

145

Table 2 Average of 10 numerical experiments. No. of experiments

Efficiency changes P1

| P1|

P2

P3

| P3|

1 2 3 4 5 6 7 8 9 10

28.06% 70.29% 14.79% 25.12% −2.08% 8.54% 32.14% 4.27% 3.95% 4.71%

39.64% 70.29% 25.80% 35.30% 20.95% 33.63% 47.42% 23.34% 31.13% 23.28%

49.88% 128.50% 59.07% 66.02% 48.54% 54.58% 53.65% 61.43% 66.42% 37.08%

2.59% 13.62% −22.76% −17.47% −52.25% −25.51% 11.52% −49.53% −63.68% −22.37%

38.71% 56.99% 42.87% 40.18% 53.49% 43.51% 56.39% 51.31% 64.65% 42.03%

Average

18.98%

35.08%

62.52%

−22.58%

49.01%

Table 3 Results of 1000 numerical experiments. Numbers of DMUs

5

10

15

20

25

30

35

40

45

Average P1 Average |P1| Average P2 Average P3 Average |P3|

14.15% 33.76% 98.59% −65.32% 71.62%

17.81% 33.35% 73.84% −34.74% 54.10%

21.57% 35.10% 62.01% −16.50% 49.08%

24.14% 36.05% 54.53% −4.71% 47.09%

26.47% 37.11% 49.13% 4.82% 47.54%

28.78% 38.22% 45.12% 12.92% 48.92%

31.29% 39.89% 41.62% 21.08% 51.82%

32.10% 40.07% 38.97% 25.10% 52.83%

33.05% 40.52% 36.66% 29.18% 53.93%

example, our performance indicator PI is better than classic CCR efficiency. Furthermore, we expand the number of DMUs to see the performance of our indicator. We repeat the numerical experiments above 1000 times in case of different numbers of DMUs and see the average bias of P1, |P1|, P2, P3, and |P3|. See details in Table 3. Fig. 3 illustrates the changes of the average bias of P1, |P1|, P2, P3, and |P3|. With the increase of the numbers of DMUs, we can find that (1) the bias of traditional CCR efficiency will decrease, (2) the bias of our performance indicator will increase, and (3) the average bias of P1, |P1|, P2, P3, and |P3| are quite similar when the number of DMUs equals to or exceeds 35. The first finding is consistent with previous studies on this issue (see, e.g., [9,38]). The second and third findings show that our performance indicator is more suitable for the case where there are relatively few DMUs to be evaluated. Next, we test the extent to which our performance indicator can discriminate the DMUs and compare our performance indicator with the classic CCR model. Definition 4. We define a discrimination power (DP) index to measure the discrimination power as follows:

DP =

P−1 , n−1

(35)

where P and n denotes the number of DMUs with different ranking orders and the number of all DMUs, respectively. For example, if there are five DMUs whose ranking list is (1,1,3,4,5), then we know P = 4 and n = 5. Thus the discrimination index is DP = 0.75. We conduct the experiments above 1000 times on classic CCR model and our performance indicator PI, respectively, and the average discrimination indexes are listed in Table 4. We can easily see that the discrimination power of our performance indicator is much better than that of the traditional CCR model. The results of the above numerical experiments can demonstrate that this indicator has considerably more discrimination power than that of standard DEA models, that it reduces overestimation and that it addresses peculiar DMUs properly. The reasons can be stated qualitatively as follows: (1) Although DEA tends to overestimate true efficiency, inverted DEA often underestimates it. In this paper, we combine the information from both classic DEA and inverted DEA, so we could reduce the overestimation to some extent. (2) The discrimination power can also be strengthened when we take more

information from pessimistic perspectives into consideration. (3) The peculiar DMUs are evaluated as medium which is a reasonable way to address them. 5. Illustrative examples In this section, two illustrative examples are proposed regarding combining efficiency and anti-efficiency using the ER approach. First, we compare the efficiency score in the classic DEA and the indicator produced by efficiency and anti-efficiency. Second, we will use the ER approach to combine the efficiency and anti-efficiency in a case study on the performance evaluation of C9 universities in China in 2011. 5.1. Comparison of classic DEA and the indicator In this section, we compare the efficiency score in the classic DEA and the indicator produced by efficiency and anti-efficiency. The data set of the first example comes from Zhu [53], which is shown in Table 5. At first, we employ the input-oriented CCR model (1) and antiefficient CCR model (3) to calculate the efficiency and anti-efficiency scores. As noted by Entani et al. [21], we can see that Wal-Mart and Nippon Life are evaluated as efficient by DEA, and they are peculiar rather than superiority. Then, we use the ER approach to combine the efficiency and anti-efficiency to determine the performance indicator. With the intention to find the characteristics and differences of efficiency scores and the performance indicator scores, we ranked the DMUs according to the scores from the different approaches. As shown in Table 6, the Ranks R1, R2, and R3 are generated by the CCR efficiency score, inverted CCR Antiefficiency score, and performance indicator PI, respectively. It is clear that both the CCR and inverted CCR models have a weaker power of discrimination with respect to the proposed PI in this empirical study. Among 14 companies, seven companies are efficient in the CCR model, and six companies are anti-efficient in the inverted CCR model. We first transform the efficiency (Column 2) and anti-efficiency (Column 6) into the assessment distributions, thus, we have the assessment distributions for all 14 companies which they are presented

146

J. Cao et al. / Knowledge-Based Systems 92 (2016) 138–150

Average bias

120.00%

Average P 1

Average |P1|

100.00%

Average P 2

Average P3

80.00%

Average |P3|

60.00% 40.00% 20.00% 0.00% -20.00%

5

10

15

20

25

30

35

40

45

Number of DMUs

-40.00% -60.00% -80.00% Fig. 3. The changes of the average bias of P1, |P1|, P2 and P3, and |P3|. Table 4 Average of DP index in case of 1000 numerical experiments. Models

DP-CCR DP-PI

Numbers of DMUs 5

10

15

20

25

30

35

40

45

0.5300 0.9275

0.5627 0.9930

0.6098 0.9981

0.6429 0.9993

0.6764 0.9996

0.6976 0.9996

0.7138 0.9999

0.7273 0.9999

0.7402 1

Note: DP-CCR and DP-PI denote the average discrimination power index DP when applied to CCR model and our performance indicator PI. Table 5 Global Fortune 500 companies. Company

Mitsui Itochu General Motors Sumitomo Marubeni Ford Motor Toyota Motor Exxon Royal Dutch/Shell Group Wal-Mart Hitachi Nippon Life Insurance Nippon Telegraph & Telephone AT&T

Inputs

Outputs

Assets

Equity

Employees

Revenue

Profit

68,770.9 65,708.9 217,123.4 50,268.9 71,439.3 243,283 106,004.2 91,296 118,011.6 37,871 91,620.9 36,4762.5 127,077.3 88,884

5553.9 4271.1 23,345.5 6681 5239.1 24,547 49,691.6 40,436 58,986.4 14,762 29,907.2 2241.9 42,240.1 17,274

80,000 7182 709,000 6193 6702 346,990 146,855 82,000 104,000 675,000 331,852 89,690 231,400 299,300

181,518.7 169,164.6 168,828.6 167,530.7 161,057.4 137,137 111,052 110,009 109,833.7 93,627 84,167.1 83,206.7 81,937.2 79,609

314.8 121.2 6880.7 210.5 156.6 4139 2662.4 6470 6904.6 2740 1468.8 2426.6 2209.1 139

in Columns 4 and 5 and Columns 8 and 9, respectively. Therefore, we can obtain the scores of the performance indicators (Column 12) and the corresponding ranking of these companies (Column 13) such that the efficiency (Column 2) and Anti-efficiency (Column 6) have the same weight. See Table 6. Likewise, the last row of Table 6 shows the obtained DP index from CCR, Anti-CCR, and PI. As seen in Table 6, the DP index corresponding PI, i.e., 0.363, is greater than that of CCR and Anti-CCR models, (i.e., 0.182 and 0.545, respectively). Therefore, this fact indicates that the discrimination power of our performance indicator is better than that of CCR and Anti-CCR models. Furthermore, we analyze the cases where we set different weights for efficiency (Column 2) and Anti-efficiency (Column 6) to determine more information regarding the features of the performance indicator PI. Note that, the weights of efficiency and anti-efficiency reflect the relative importance of these two pieces of evidence. If the

decision maker considers that the evidence from the perspective of anti-efficient frontier is more important, then the corresponding weight of efficiency is higher. On the contrary, if the decision maker prefers the evidence from the pessimistic perspective, then more weight should be assigned to anti-efficiency. See Table 7 for details. We also illustrate the scores of the performance indicator under different weights using Fig. 4. We can see that the performance indicator is very discriminative. Furthermore, we can see that the efficiency and anti-efficiency of Marubeni are 0.97 and 0.29, respectively. Compared with Nippon Life and Wal-Mart (efficiency/anti-efficiency = 1/1), Marubeni performs better than these two peculiar DMUs such that the weight of anti-efficiency is larger than 0.3. This means that if we consider the information from an anti-efficient frontier and a CCR inefficient DMU, Marubeni performs better than the two peculiar DMUs on the intersection of efficient and anti-efficient frontiers.

J. Cao et al. / Knowledge-Based Systems 92 (2016) 138–150

147

Table 6 Results and ranks of companies using performance indicators. Company

CCR h∗classic

Mitsui Sumitomo Exxon General Motors Itochu Wal-Mart Nippon Life Marubeni Royal Group Ford Motor Toyota Motor Hitachi Nippon T&T AT&T DP index

1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.97 0.84 0.74 0.52 0.39 0.35 0.27

Inverted CCR R1

1 1 1 1 1 1 1 8 9 10 11 12 13 14 0.182

Performance indicator PI 1/h∗inverted

Assessment distribution H1

H2

1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.97 0.84 0.74 0.52 0.39 0.35 0.27

0 0 0 0 0 0 0 0.03 0.16 0.26 0.48 0.61 0.65 0.73

0.34 0.37 0.69 0.82 0.85 1.00 1.00 0.71 1.00 0.76 0.93 1.00 1.00 1.00

R2

1 2 3 6 7 9 9 4 9 5 8 9 9 9 0.545

Table 7 Results and ranks of companies using performance indicators with different weights. Company CCR Inverted CCR ω1 = 0.1 ω1 = 0.2 ω2 = 0.9 ω2 = 0.8

Mitsui Sumitomo Exxon General Motors Itochu Wal-Mart Nippon Life Marubeni Royal Group Ford Motor Toyota Motor Hitachi Nippon T&T AT&T

1.2 1

Assessment distribution H1 H2

Assessment distribution H1 H2

PI

PI

1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.97 0.84 0.74 0.52 0.39 0.35 0.27

0.66 0.63 0.31 0.18 0.15 0.00 0.00 0.29 0.00 0.24 0.07 0.00 0.00 0.00

0.6868 0.6581 0.3408 0.2057 0.1740 0.0122 0.0122 0.3186 0.0101 0.2551 0.0755 0.0045 0.0040 0.0030

0.7230 0.6967 0.3947 0.2596 0.2273 0.0588 0.0588 0.3699 0.0476 0.2860 0.0943 0.0201 0.0179 0.0136

0 0 0 0 0 0 0 0.03 0.16 0.26 0.48 0.61 0.65 0.73

Mitsui Wal -Mart Toyota Motor

0.34 0.37 0.69 0.82 0.85 1.00 1.00 0.71 1.00 0.76 0.93 1.00 1.00 1.00

Sumitomo Nippon Life Hitachi

Assessment distribution

Assessment distribution

H1

H2

H1

H2

0.66 0.63 0.31 0.18 0.15 0.00 0.00 0.29 0.00 0.24 0.07 0.00 0.00 0.00

0.34 0.37 0.69 0.82 0.85 1.00 1.00 0.71 1.00 0.76 0.93 1.00 1.00 1.00

0.87 0.86 0.70 0.62 0.60 0.50 0.50 0.67 0.39 0.49 0.25 0.15 0.13 0.10

0.13 0.14 0.30 0.38 0.40 0.50 0.50 0.33 0.61 0.51 0.75 0.85 0.87 0.90

PI

R3

0.87 0.86 0.70 0.62 0.60 0.50 0.50 0.67 0.39 0.49 0.25 0.15 0.13 0.10

1 2 3 5 6 7 7 4 10 9 11 12 13 14 0.636

ω1 = 0.3 ω2 = 0.7

ω1 = 0.4 ω2 = 0.6

ω1 = 0.5 ω2 = 0.5

ω1 = 0.6 ω2 = 0.4

ω1 = 0.7 ω2 = 0.3

ω1 = 0.8 ω2 = 0.2

ω1 = 0.9 ω2 = 0.1

0.7682 0.7455 0.4759 0.3496 0.3189 0.1552 0.1552 0.4483 0.1232 0.3362 0.1302 0.0496 0.0440 0.0331

0.8196 0.8015 0.5821 0.4759 0.4496 0.3077 0.3077 0.5520 0.2407 0.4056 0.1842 0.0936 0.0828 0.0621

0.8722 0.8593 0.7013 0.6239 0.6047 0.5000 0.5000 0.6694 0.3889 0.4874 0.2523 0.1494 0.1321 0.0989

0.9198 0.9118 0.8143 0.7670 0.7554 0.6923 0.6923 0.7815 0.5415 0.5695 0.3258 0.2107 0.1864 0.1398

0.9574 0.9533 0.9037 0.8805 0.8749 0.8448 0.8448 0.8709 0.6708 0.6398 0.3944 0.2699 0.2394 0.1804

0.9827 0.9810 0.9622 0.9537 0.9517 0.9412 0.9412 0.9300 0.7619 0.6913 0.4509 0.3210 0.2857 0.2169

0.9961 0.9958 0.9919 0.9902 0.9898 0.9878 0.9878 0.9607 0.8154 0.7236 0.4924 0.3611 0.3227 0.2469

PI

Exxon Marubeni Nippon T&T

PI

PI

General Motors Royal Group AT&T

PI

PI

PI

PI

Itochu Ford Motor

0.8 0.6 0.4 0.2 0

Fig. 4. The scores of the performance indicator under different weights.

Furthermore, we can see that in comparison with efficiency in the classic CCR, the more weight is assigned to anti-efficiency, and lower the performance scores of these DMUs. Thus, we know that the efficiency is pulled low when considering the obtained information from the anti-efficient frontier. Additionally, we can see that the peculiar DMUs are addressed properly.

5.2. A case study of C9 universities in China The C9 League is an alliance of nine universities in mainland China, which includes Tsinghua University, Peking University, Harbin Institute of Technology, University of Science and Technology of China, Fudan University, Zhejiang University, Nanjing University,

148

J. Cao et al. / Knowledge-Based Systems 92 (2016) 138–150 Table 8 Data of C9 universities in China in 2010. Universities

DMU1 DMU2 DMU3 DMU4 DMU5 DMU6 DMU7 DMU8 DMU9

Inputs

Outputs

FUND (RMB in million)

Paper (number)

TT income (RMB in million)

Award (number)

2932.6650 3634.6700 1808.0820 1747.4980 2722.3040 897.7940 3290.6070 941.1930 1050.9270

12012 11019 9520 8615 19899 5012 16263 3299 4454

1007.2920 397.3450 1.5800 43.5750 166.9950 3.3760 47.0550 13.1100 39.0780

34 90 47 31 79 17 135 15 41

Table 9 Performance indicators of C9 universities. Universities

DMU1 DMU2 DMU3 DMU4 DMU5 DMU6 DMU7 DMU8 DMU9

CCR h∗classic

Inverted CCR 1/h∗inverted

Efficiency

Ranking

Anti-efficiency

Ranking

1.0000 0.8159 0.7973 0.6744 1.0000 0.7637 1.0000 0.5101 1.0000

1 5 6 8 1 7 1 9 1

1.0000 1.0000 1.0000 0.8922 0.5405 1.0000 0.7835 1.0000 0.8082

5 5 5 4 1 5 2 5 3

Shanghai Jiao Tong University and Xi’an Jiao Tong University. Together, they account for 3% of the country’s researchers, but receive 10% of national research expenditures. They produce 20% of the academic publications and 30% of the total citations. Comprised mostly with the most renowned and oldest local universities, C9 is often analogous to the Russell Group in UK, the Go8 in Australia and the Ivy League in the USA.3 In this paper, we denote these universities as C9 universities in China. We use the indicators in “Science and Technology (S&T) statistics compilation in 2011”, which is published by the Ministry of Education (MOE) in China, to rank these C9 universities. These indicators include: (1) input indicator: S&T funds (FUND); (2) output indicators: papers, technology transfer income (TT INCOME) and awards. As an input indicator, FUND denotes the total income of the universities, including research funding and block grants in the statistical year. As output indicators, papers denote publications in important international and domestic journals in the statistical year. TT INCOME denotes the total income from the process of technology transfer in the university in the statistical year. Award refers to something given to a person or a group of people in the university to recognize their excellence in a certain field in the statistical year. The detailed data are shown in Table 8. First, we use a classic input-based CCR model (Model 1) and an input-based inverted CCR model (Model 3) to determine the efficiency scores and anti-efficiency scores (Columns 2 and 4, respectively) as listed in Table 9. We can see that DMU1 , DMU5 , DMU7 , and DMU9 are efficient and their efficiency scores are all one. On the contrary, DMU1 , DMU2 , DMU3 , DMU6 , and DMU8 are anti-efficient. DMU1 is evaluated as both efficient and anti-efficient by the CCR model and inverted CCR model so it is a peculiar DMU rather than a superior DMU. Because approximately 44% of the DMUs are efficient and 56% of the DMUs are antiefficient, this result reveals that the discrimination capabilities of the CCR and inverted CCR models are both relatively low in this example.

3

http://en.wikipedia.org/wiki/C9_League.

Performance indicator PI

Ranking

0.5000 0.3736 0.3620 0.3618 0.7802 0.3415 0.6465 0.2049 0.6313

4 5 6 7 1 8 2 9 3

We can verify the preference structure through a mutual preferential independence test, which is called “additive independence” [22]. The test result showed that the decision makers’4 preference structure failed to satisfy the “additive independence” condition mainly because the efficiency scores and anti-efficiency scores are generated from the same data set and linked to each other. The prerequisite of the ER approach is that the pieces of evidence are generated in an independent way. This condition is considerably milder than “additive independence”. In this case, we can see that the efficiency scores and anti-efficiency scores are generated from independent DEA models and there is no interdependence. Therefore, in this case, it is appropriate to use the aforementioned ER approach in Section 3 to combine the efficiency and anti-efficiency scores for obtaining the performance scores of these universities (The decision makers set the same weights (0.5, 0.5) to efficiency scores and anti-efficiency scores which are equally important for this case). See Column 6 in Table 9 for details. From the performance indicator, we can see that the discriminative power of PI is considerably higher than the efficiency and antiefficiency scores from CCR and inverted CCR models, which can give the full ranking of these nine universities. The peculiar DMU1 is intersection of the efficient frontier and anti-efficient frontier and is evaluated as medium (performance score: 0.5). Furthermore, we can compare the CCR efficiency and performance scores using the ER approach, which are listed in Fig. 5. As seen in this figure, we can lower the efficiency scores of DMUs by combining the efficiency and anti-efficiency scores from CCR and inverted CCR models, respectively, to reduce the overestimation of true efficiency in the classic DEA models. In this case, we also compare the obtained results from the performance indicators, the interval DEA method in Entani et al. [21], and the efficiency measure proposed by Amirteimoori [7]. See Table 10 for details.

4

The decision makers are the expert panel from the Chinese science community.

J. Cao et al. / Knowledge-Based Systems 92 (2016) 138–150

149

1.2 CCR-efficiency

1

Performance indicator

0.8 0.6 0.4 0.2 0 DMU 1

DMU 2

DMU 3

DMU 4

DMU 5

DMU 6

DMU 7

DMU 8

DMU 9

Fig. 5. Comparison of CCR efficiency and performance scores using the ER approach. Table 10 Comparison of different measures. Universities

DMU1 DMU2 DMU3 DMU4 DMU5 DMU6 DMU7 DMU8 DMU9

Performance indicator PI

Interval efficiency [21]

Efficiency measure [7]

Scores

Ranking

Intervals

Ranking

Scores

Ranking

0.5000 0.3736 0.3620 0.3618 0.7802 0.3415 0.6465 0.2049 0.6313

4 5 6 7 1 8 2 9 3

[0.2826, 1.0000] [0.2470, 0.8159] [0.0025, 0.7973] [0.0726, 0.6744] [0.1786, 1.0000] [0.0109, 0.7637] [0.0416, 1.0000] [0.0406, 0.5101] [0.1083, 1.0000]

1 2 6 6 2 6 5 6 4

1 −1 −1 −0.4587 1 −1 1 −1 1

1 6 6 5 1 6 1 6 1

As for the efficiency measures proposed by Amirteimoori [7], 5 we can see that: (1) The discrimination power of the efficiency measures proposed by Amirteimoori [7] is poor; and (2) The peculiar DMU1 is evaluated as an efficient DMU. As for the interval efficiency [21], we can see that: (1) The discrimination power is low; and (2) The peculiar DMU1 is evaluated as the only non-dominated DMU. Thus, we can conclude that our performance indicator is more reasonable than these two existing measures.

Acknowledgments

6. Conclusions and further study

References

There are three primary problems in the classic DEA models: (1) DEA models often overestimate the efficiency and they are biased; (2) Many researchers try to improve the discrimination capability of the standard DEA models; (3) How to address the evaluated DMUs as efficient by DEA, which are peculiar, rather than superior. In this paper, we use the ER approach to combine the generated efficiency scores and anti-efficiency scores from the classic CCR and inverted CCR models, respectively. The ER approach describes each attribute as an alternative by a distributed assessment via a belief structure, and also it provides a new procedure for aggregating multiple attributes based on the distributed assessment framework and the evidence combination rule of the D–S evidence theory. In order to highlight the performance of the proposed approach, we present some illustrative examples in Section 5 of this research. The obtained results from these examples reveal that our new performance indicator in capable of: (1) Reducing the efficiency scores; (2) Increasing the discrimination power; and (3) Specifying peculiar DMUs properly. These facts indicate that the proposed performance indicator features lower bias than CCR efficiency and anti-CCR efficiency. Consequently, this indicator is more suitable for the case that there are relatively few DMUs to be evaluated with respect to the number of input and output factors.

5

Note: There are some errors on inequalities in the model (8) in Amirteimoori [7].

We acknowledge the supports from the National Natural Science Foundation of China (grant nos. 71201158, 71371067) and National Soft Science Research Program (No. 2014GXQ4B168). We thank the anonymous referees for their valuable comments which helped us to improve the manuscript significantly.

[1] N. Adler, L. Friedman, Z. Sinuany-Stern, Review of ranking methods in the data envelopment analysis context, Eur. J. Oper. Res. 140 (2) (2002) 249–265. [2] N. Adler, B. Golany, Evaluation of deregulated airline networks using data envelopment analysis combined with principal component analysis with an application to Western Europe, Eur. J. Oper. Res. 132 (2) (2001) 18–31. [3] N. Adler, B. Golany, Including principal component weights to improve discrimination in data envelopment analysis, J. Oper. Res. Soc. 53 (2002) 985–991. [4] N. Adler, E. Yazhemsky, Improving discrimination in data envelopment analysis: PCA-DEA or variable reduction, Eur. J. Oper. Res. 202 (2010) 273–284. [5] M.R. Alirezaee, M. Howland, C. Van de Panne, Sampling size and efficiency bias in data envelopment analysis, J. Appl. Math. Decis. Sci. 2 (1) (1998) 51–64. [6] R. Allen, A. Athanassopoulos, R.G. Dyson, E. Thanassoulis, Weights restrictions and value judgments in data envelopment analysis: evolution, development and future directions, Ann. Oper. Res. 73 (1997) 13–34. [7] A. Amirteimoori, DEA efficiency analysis: efficient and anti-efficient frontier, Appl. Math. Comput. 186 (1) (2007) 10–16. [8] P. Andersen, N. Petersen, A procedure for ranking efficient units in data envelopment analysis, Manage. Sci. 39 (10) (1993) 1261–1264. [9] R.D. Banker, Maximum likelihood, consistency and data envelopment analysis: a statistical foundation, Manage. Sci. 39 (10) (1993) 1265–1273. [10] R.D. Banker, Hypothesis tests using data envelopment analysis, J. Prod. Anal. 7 (1996) 139–159. [11] R.D. Banker, H. Chang, The super-efficiency procedure for outlier identification, not for ranking efficient units, Eur. J. Oper. Res. 175 (2) (2006) 1311–1320. [12] B.G. Buchanan, E.H. Shortliffe, Rule-Based Expert System, Addison-Wesley, Reading, MA, 1984. [13] A. Charnes, W.W. Cooper, E. Rhodes, Measuring the efficiency of decision making units, Eur. J. Oper. Res. 2 (6) (1978) 429–444. [14] K.S. Chin, Y.M. Wang, J.B. Yang, K.K.G. Poon, An evidential reasoning based approach for quality function deployment under uncertainty, Expert Syst. Appl. 36 (3) (2009) 5684–5694. [15] W.D. Cook, J. Zhu, CAR-DEA: context dependent assurance regions in DEA, Oper. Res. 56 (1) (2008) 69–78.

150

J. Cao et al. / Knowledge-Based Systems 92 (2016) 138–150

[16] W.W. Cooper, L.M. Seiford, K. Tone, Data Envelopment Analysis: A Comprehensive Text with Models – Applications, References and DEA-Solver Software, Kluwer Academic Publishers, Boston, 2000. [17] A.P. Dempster, Upper and lower probabilities induced by a multi-valued mapping, Ann. Math. Stat. 38 (1967) 325–339. [18] D.K. Despotis, Improving the discrimination power of DEA: focus on globally efficient units, J. Oper. Res. Soc. 53 (2002) 314–323. [19] J.R. Doyle, R. Green, Efficiency and cross-efficiency in data envelopment analysis: derivatives, meanings and uses, J. Oper. Res. Soc. 45 (5) (1994) 567–578. [20] R.G. Dyson, E. Thanassoulis, Reducing weight flexibility in data envelopment analysis, J. Oper. Res. Soc. 39 (1988) 567–578. [21] T. Entani, Y. Maeda, H. Tanaka, Dual models of interval DEA and its extension to interval data, Eur. J. Oper. Res. 136 (1) (2002) 32–45. [22] P.C. Fishburn, Utility Theory for Decision Making, Wiley, New York, 1970. [23] D.U.A. Galagedera, P. Silvapulle, Experimental evidence on robustness of data envelopment analysis, J. Oper. Res. Soc. 54 (6) (2003) 654–660. [24] L. Jenkins, M. Anderson, Multivariate statistical approach to reducing the number of variables in data envelopment analysis, Eur. J. Oper. Res. 147 (2003) 51–61. [25] A. Kneip, L. Simar, P.W. Wilson, Asymptotics and consistent bootstraps for DEA estimators in non-parametric frontier models, Econ. Theory 24 (2008) 1663–1697. [26] L. Liang, J. Wu, W.D. Cook, J. Zhu, The DEA game cross-efficiency model and its Nash equilibrium, Oper. Res. 56 (5) (2008) 1278–1288. [27] S. Lim, K.W. Oh, J. Zhu, Use of DEA cross-efficiency evaluation in portfolio selection: an application to Korea stock market, Eur. J. Oper. Res. 236 (1) (2014) 361– 368. [28] W.B. Liu, W. Meng, D.Q. Zhang, Incorporating value judgments in DEA, in: N.K. Avkiran (Ed.), Productivity Analysis in the Service Sector using Data Envelopment Analysis, third ed., 2006, pp. 217–242. (Chapter 21). [29] W. Meng, D.Q. Zhang, L. Qi, W.B. Liu, Two-level DEA approaches in research evaluation, Omega 36 (6) (2008) 950–957. [30] C.K. Murphy, Combining belief functions when evidence conflicts, Decis. Supp. Syst. 29 (2000) 1–9. [31] J.T. Pastor, J.L. Ruiz, I. Sirvent, A statistical test for nested radial DEA models, Oper. Res. 50 (2002) 728–735. [32] V.V. Podinovski, E. Thanassoulis, Improving discrimination in data envelopment analysis: some practical suggestions, J. Prod. Anal. 28 (2007) 117–126. [33] T.R. Sexton, R.H. Silkman, A.J. Hogan, Data envelopment analysis: critique and extensions, in: R.H. Silkman (Ed.), Measuring Efficiency: An Assessment of Data Envelopment Analysis, Jossey-Bass, San Francisco, CA, 1986. [34] G. Shafer, A Mathematical Theory of Evidence, Princeton University Press, Princeton, New Jersey, 1976. [35] L. Simar, P.W. Wilson, Statistical inference in nonparametric frontier models: the state of the art, J. Prod. Anal. 13 (2000) 49–78.

[36] L. Simar, P.W. Wilson, Testing restrictions in nonparametric efficiency models, Commun. Stat. 30 (2001) 159–184. [37] L. Simar, P. Wilson, Statistical inference in nonparametric frontier models: recent developments and perspectives, in: H. Fried, C.K. Lovell, S. Schmidt (Eds.), The Measurement of Productive Efficiency, Oxford University Press, Oxford, 2007. [38] P. Smith, Model misspecification in data envelopment analysis, Ann. Oper. Res. 73 (1997) 233–252. [39] E. Thanassoulis, M.C. Portela, R. Allen, Incorporating value judgment in DEA, in: W.W. Cooper, L.M. Seiford, J. Zhu (Eds.), Handbook on Data Envelopment Analysis, Kluwer Academic Publishers, Netherlands, 2004. [40] K. Tone, A slacks-based measure of super-efficiency in data envelopment analysis[J], Eur. J. Oper. Res. 143 (1) (2002) 32–41. [41] Y.M. Wang, J.B. Yang, D.L. Xu, Environmental impact assessment using the evidential reasoning approach, Eur. J. Oper. Res. 174 (3) (2006) 1885–1913. [42] Y.M. Wang, K.S. Chin, Some alternative models for DEA cross-efficiency evaluation, Int. J. Prod. Econ. 128 (1) (2010) 332–338. [43] Y.M. Wang, K.S. Chin, The use of OWA operator weights for cross-efficiency aggregation, Omega – Int. J. Manage. Sci. 39 (5) (2011) 493–503. [44] J. Wu, L. Liang, Y. Chen, DEA game cross-efficiency approach to Olympic rankings, Omega – Int. J. Manage. Sci. 37 (4) (2009) 909–918. [45] Y. Yamada, T. Matsui, M. Sugiyama, An inefficiency measurement method for management systems, J. Oper. Res. Soc. Jpn. 37 (1994) 158–167. [46] G.L. Yang, J.B. Yang, W.B. Liu, X.X. Li, Cross-efficiency aggregation in DEA models using the evidential-reasoning approach, Eur. J. Oper. Res. 231 (2) (2013) 393–404. [47] J.B. Yang, P. Sen, A general multi-level evaluation process for hybrid MADM with uncertainty, IEEE Trans. Syst. Man Cybernet. 24 (10) (1994) 1458–1473. [48] J.B. Yang, M.G. Singh, An evidential reasoning approach for multiple attribute decision making with uncertainty, IEEE Trans. Syst. Man Cybernet. 24 (1) (1994) 1–18. [49] J.B. Yang, D.L. Xu, On the evidential reasoning algorithm for multiattribute decision analysis under uncertainty, IEEE Trans. Syst. Man Cybernet. Part A Syst. Humans 32 (3) (2002) 289–304. [50] L.A. Zadeh, Review of books: a mathematical theory of evidence, AI Mag. 5 (3) (1984) 81–83. [51] D.Q. Zhang, X.X. Li, W. Meng, W.B. Liu, Measure the performance of nations at Olympic Games using DEA models with different preferences, J. Oper. Res. Soc. 60 (7) (2009) 983–990. [52] M. Zeleny, Multiple Criteria Decision Making, McGraw-Hill, New York, 1982. [53] J. Zhu, Quantitative Models for Performance Evaluation and Benchmarking: DEA with Spreadsheets and DEA Excel Solver, Kluwer Academic Publishers, Boston, 2003.