Data envelopment analysis with estimated output data: Confidence intervals efficiency

Journal Pre-proofs Data Envelopment Analysis with Estimated Output Data: Confidence Intervals Efficiency Jesús A. Tapia, Bonifacio Salvador, Jesús M. Rodríguez PII: DOI: Reference:

S0263-2241(19)31228-X https://doi.org/10.1016/j.measurement.2019.107364 MEASUR 107364

To appear in:

Measurement

Received Date: Accepted Date:

27 June 2016 3 December 2019

Please cite this article as: J.A. Tapia, B. Salvador, J.M. Rodríguez, Data Envelopment Analysis with Estimated Output Data: Confidence Intervals Efficiency, Measurement (2019), doi: https://doi.org/10.1016/j.measurement. 2019.107364

This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

© 2019 Published by Elsevier Ltd.

5HYLVHGPDQXVFULSWZLWKFKDQJHVPDUNHG &OLFNKHUHWRYLHZOLQNHG5HIHUHQFHV

&OLFNKHUHWRDFFHVVGRZQORDG5HYLVHGPDQXVFULSWZLWKFKDQJHV PDUNHG0($6&,(65('UHYGHISGI

Data Envelopment Analysis with Estimated Output Data: Confidence Intervals Efficiency Jesús A. Tapiaa , Bonifacio Salvadora , Jesús M. Rodríguezb a Department

of Statistic and Operative Research, University of Valladolid, Campus Miguel Delibes, Paseo Belén nº7,47011 Valladolid, Spain b Statistical & Budgetary General Management of the Regional Government of Castile & Leon, Regional Tax Office, José Cantalapiedra nº2, 47014 Valladolid, Spain

Abstract Data Envelopment Analysis (DEA) is very useful for measuring the relative efficiency of such service producing units as hospitals or state services, etc. In recent years, some efficiency analysis approaches have used customer opinion as data as they consider it essential for measuring the quality of the service received. This paper measures the DEA efficiency of services by taking known inputs that explain satisfaction-opinion indexes (outputs) estimated with a customer sample. In this situation the DEA has to be stochastic and the obtained efficiency will be an estimation of the population efficiency scores that would have been obtained if we had had the population output data. Specifically, a bootstrap is performed that derives the confidence intervals needed to estimate the population DEA efficiency. A simulation is carried out which allows the confidence intervals to be validated and compared and an empirical application to measure the efficiency in the Spanish health environment is also described. Keywords: Stochastic data envelopment analysis, Confidence interval Efficiency, Bootstrap, Survey sampling. 2000 MSC: 62D05, 90C99 1. Introduction Measuring the relative efficiency of service-producing units has been an important purpose of Data Envelopment Analysis (DEA) for many years ([32], [28]). When the available public services information is deterministic, the efficiency is evaluated with the original DEA models proposed in Charnes et al. [8] or Banker et al. [2]. A vast academic literature sets out the applications of the classic DEA models, considering public services as decision-making units (DMU); for instance, health care ([30]), universities and research institutes ([21]), government ([15]), public libraries ([18], [19]), schools ([29]), or public transport services ([12]). In recent years, some approaches have considered the opinion of the customer to be crucial to appropriately defining outputs for the measurement of public service efficiency ([3], [14]). A consumer satisfaction-opinion survey allows the quality of the service to be measured and can be used to define opinion indexes as output variables of the DEA ( [23], [5], [25, 26], [29] and [17]). We propose to carry out satisfaction-opinion surveys of consumers, synthesizing their answers in satisfactionopinion indexes. We then use these indexes as data output in an efficiency DEA of public services. For example, Email addresses: [email protected] (Jesús A. Tapia), [email protected] ( Bonifacio Salvador), [email protected] (Jesús M. Rodríguez)

Preprint submitted to Measurement

November 29, 2019

in libraries, the resources (books, personnel,...) can explain the monthly mean time that an individual uses the library or the mean loans per user, information that can be estimated with a user survey sampling in the face of the impossibility of carrying out a user census. Data estimated with a sample incorporate random variations into the DEA analysis and different solutions to this situation have been proposed in the last few decades. One of them is chance-constrained programming, linear programming (LP) problems subject to constraints defined in terms of probability, where an "expected value" (E-model) or a "most probable value" (P-model) of the DEA efficiency is obtained ([6, 7], [22], [27], [9, 10], [20], [36], [34], [5]). Chance-constrained programming gives a point estimation solution to the efficiency measure. The methodology is different here as we use linear programming DEA models subject to deterministic constraints to obtain confidence intervals. In this paper, we focus on the measuring efficiency problem as a statistical problem. First, in each serviceproducing (our DMU), we estimate the unknown outputs using the opinion of a sample of customers. Second, we estimate the population DEA efficiency score with confidence intervals. By efficiency population, we understand that which would be obtained if, in each DMU, the output data were obtained with the opinion of the census of customers instead of the opinion of a customer sample. We propose two confidence interval methodologies that, to our knowledge, are novel. One is inspired in the optimistic/pessimistic point of view of DEA models and the other in the use of bootstrap replications from the sample of customers in each DMU. Finally, we also propose a solution to the sample size problem of customers needed to guarantee the confidence of our intervals. Ceyhan and Benneyan [4] discuss several approaches (Monte Carlo (MC), bootstrapping as in Simar et al. [31], and optimistic/pessimistic as in Wang et al. [35]) to obtain intervals for the efficiency with each methodology in the DEA problem conducted on values that include such estimated proportions as defect, satisfaction, mortality or adverse event rates estimated from samples of individuals. By means of a simulation procedure, Ceyhan and Benneyan show that the confidence of the MC is very low. The two confidence interval efficiency methodologies proposed in our paper are different from used by Ceyhan and Benneyan as we use the information of each customer sample, not just the estimated output. In a simulation similar to that of Ceyhan and Benneyan, we evaluate the confidence of our DEA interval efficiency methodology proposals by checking the good behavior of the bootstrapping confidence interval efficiency methodology. Being able to measure the confidence makes our efficiency measurement methodology completely different from the proposed efficiency intervals in Despotis and Smirlis [13], Wang et al. [35] or Azizi and Wang [1]. From the statistical point of view of our approach, the efficiency is a parameter and the quality of the measurement of that efficiency depends on the sample size of customers used to estimate the data output. Ceyhan and Benneyan [4] investigated the impact of the sample size, but they did not propose a solution to the necessary sample size to control the error in the efficiency measurements. With the same assumptions as in this paper, Tapia et al. [33] determined the customer sample size necessary in each public service-producing unit so that the efficiency estimation error in the service-producing units will be smaller than a previously fixed value; while in this paper, the customer sample size necessary in each public service-producing unit is determined so as to guarantee the confidence of our intervals. The rest of the paper is organized as follows. In the second section, the proposal of the problem is set out. Section three discusses methodologies to obtain the confidence interval efficiency in the DEA; while section four describes a simulation study that compares and evaluates these methodologies. Section five illustrates the application of the proposed intervals in a real example, which allows the efficiency to be analyzed in the healthcare environment of the autonomous communities of Spain. A summary of the main conclusions is given in the last section.

2

2. Approaching the problem We formally consider n fixed DMUs, each one using m inputs to produce s outputs. We denote Xj =
X1 j , .., Xm j as the value of the m known inputs n in the jth DMU o ( j = 1, . . . , n) . The s outputs of the jth
DMU are represented by a finite population Uj = U1j , . . . , UNj j of size N j , where Uk j = Uk1 j , . . . , Uks j are s different measures that represent the kth individual of the jth DMU, for example, s indices that measure their opinion. The unknown population outputs are:  

Yj = f Uj j=1,...,n = Y1 j , . . . ,Ys j j=1,...,n (1)
where f is a function that, applied to the population Uj , determines the s unknown outputs Y1 j , . . . ,Ys j . If, in the jth population, we consider V1j , . . . , Vnj j to be a random sample u j of size n j from the finite population Uj , where Vkj is a random vector of dimension s, the estimated outputs are:

b j = f u j = Yˆ1 j , . . . , Yˆs j ; j = 1, . . . , n Y

(2)

Having fixed a sample in each DMU, it is very convenient to distinguish between the estimators defined in (2) and their estimated values represented by:
b y j = yb1 j , . . . , ybs j ; j = 1, . . . , n

(3)

For example, in this paper, we use u j as a simple random sampling without replacement (WOR) of Uj to estimate the vector of population means, then: ( !) Nj Nj Nj
∑k=1 Ukj ∑k=1 Uks j ∑k=1 Uk1 j = ,..., (4) Yj = f Uj = Nj Nj Nj j=1,...,n

(

bj = f u j Y




nj

∑ Vkj = = k=1 nj

n

n

j j ∑k=1 Vks j ∑k=1 Vk1 j ,..., nj nj

!)

(5) j=1,...,n

h i and a confidence interval of level 1 − α for the rth output (population mean) of the jth DMU is YˆrLj , YˆrUj .
Having observed a sample vk1 j , . . . , vk1 j k = 1, . . . , n j with WOR design, in each DMU, then the estimation of j = 1, . . . , n

the output




b y j = yb1 j , . . . , ybs j =

n

n

j j ∑ vks j ∑k=1 vk1 j , . . . , k=1 nj nj

!

; j = 1, . . . , n

(6)

i h and the observed interval is ybLrj , ybU rj . Table 1 shows the variable returns-to-scale DEA model (BCC-O) that we work with. The choice of the output orientation is justified by the interest of observing the units which, while maintaining the same resources (inputs), can improve the opinion; we only consider the BCC model because, in the presence of output data estimated with an opinion survey sampling, we discard constant returns-to-scale. In this situation, we denote: 3


• Population efficiency index ϕ j j=1,...,n BCC-O as the maximization of the Table 1 model, taking the

data Xj , Y j j=1,...,n (see Figure 1). As Y j are unknown, ϕ j j=1,...,n are unknown parameters that we aim to estimate.  
bj of the efficiency indexes ϕ j j=1,...,n • Abusing the notation, we represent the estimators Φ j=1,...,n   bj BCC-O as the maximization of the Table 1 model, taking the data Xj , Y , in the understanding j=1,...,n
yj j=1,...,n to obtain the estimated value of the estimator that the models are maximized with the data Xj , b b j that we denote by ϕb j (see Figure 1). Φ Population Sample

Estimated output data ෡ ܻ

Population output data ܻ

Known Inputs ܺ DEA model BCC-O

Estimated Efficiency ෡ Ȱ

Population Efficiency ߮

Figure 1 The definition process of the population efficiency and how it is estimated.

Table 1 Output-oriented BCC model.


− s + max ϕ + ε ∑m s + s ∑ i=1 i r=1 r s.t. ∑nj=1 λ j yr j − s+ r = ϕ yro , r = 1, . . . , s − n ∑ j=1 λ j xi j + si = xio , i = 1, . . . , m + λ j ≥ 0, s− i ≥ 0, sr ≥ 0; j = 1, . . . , n; i = 1, . . . , m; r = 1, . . . , s n ∑ j=1 λ j = 1

4

i h b U in each DMU b L, Φ The main objective of this paper is to determine a confidence interval efficiency Φ j j that will guarantee, with a confidence 1 − α , that the interval contains the population efficiency scores ϕ j , j = 1, ..., n; that is, having fixed α ∈ (0, 1): i h  b Uj b Lj , Φ ≥ 1 − α ; j = 1, . . . , n p ϕj ∈ Φ

3. Confidence Interval Efficiency

In this section, we propose confidence intervals for the population DEA efficiency scores (CIEs) when the data are known inputs and unknown outputs such as those described in Section 2. Two types of interval are defined, one adapting the optimistic/pessimistic point of view to our problem and another using bootstrap. Let us consider n DMUs, m known inputs and s outputs estimated with confidence intervals. 3.1. Methodology Optimistic/Pessimistic Despotis and Smirlis [13] and Wang el al. [35] obtain interval efficiency using data known to lie within a bounded interval and the optimistic/pessimistic point of view. In other words, they give the efficiency of each DMU by means of interval data. These intervals are not confidence intervals because, for these authors, there is neither a statistical nor a stochastic problem, so they have no parameters to estimate. Our problem is different because we consider the input data to be known, while the unknown outputs are estimated with customer samples and, with this information, we estimate the population efficiency parameter represented in Figure 1. In each DMU, we propose a confidence interval to the population efficiency score based on the optimistic/pessimistic point of view. bL bU Models (7) and (8) derive h the observed i lower, ϕoOP , and upper, ϕoOP , bounds of our proposal of the conbL ,Φ bU fidence interval efficiency Φ for the DMUo , based on the optimistic/pessimistic point of view oOP

oOP

(OPCIE). The constraints of this model are different from the constraints used by Despotis and Smirlis and Wang el al. in the context of hinterval idata. h i In the notation of Section 2, YbrLj , YbrUj is the confidence interval output data and ybLrj , ybU r j is the observed con-

L , fidence interval. To obtain the lower bound of the observed confidence interval efficiency of the DMUo , ϕboOP we take the lower extreme of its observed confidence interval for the output data, ybLro as the output data of this DMU, and the upper bounds of all the observed confidence intervals n ofothe outputs of all the DMUs as the . To obtain the upper bound output data of the production frontier, including the one evaluated, ybU rj j=1,...,n

U , we take the upper bound of its observed of the observed confidence interval efficiency for the DMUo , ϕboOP confidence interval of the output data, ybU ro as the output data of this DMU, and the lower bounds of all observed confidence intervals of the outputs of all the DMUs as the output data of the production frontier, including the n o one evaluated, ybLrj . j=1,...,n

To obtain the lower bound of the OPCIE, model (7) uses the same production frontier in all the DMUs and the same happens with model (8) to obtain the upper bound. Therefore, the OPCIE uses two production frontiers, one for each interval bound, which allows us to obtain comparable interval bounds.

5

Table 2 Bound multiplicative models of the optimistic/pessimistic confidence interval for the population efficiency scores. Lower bound

Upper bound

L maxur ,vi ϕboOP = ∑sr=1 ur ybLro + ko s.t. ∑m i=1 vi Xio = 1 m ∑sr=1 ur ybU r j − ∑i=1 vi X i j + ko ≤ 0; j = 1, . . . , n ur , vi ≥ ε ∀r, i, ko free in sign

U = s maxur ,vi ϕboOP ∑r=1 ur ybU ro + ko s.t. ∑m i=1 vi Xio = 1 ∑sr=1 ur ybLrj − ∑m i=1 vi X i j + ko ≤ 0; j = 1, . . . , n ur , vi ≥ ε ∀r, i, ko free in sign

(7)

(8)

From model (8), it is clear that the optimal weights cannot guarantee upper bound efficiencies less than or equal to unity. No matter what weights the models use, the efficiencies of the DMUs are all limited to less than or equal to one. Theorem 1. The lower bound of the confidence interval efficiency obtained with the optimistic/pessimistic model (7) is strictly smaller than 1 for all the DMUs:

ϕbLjOP < 1; j = 1, ..., n

(9)

Proof. Reasoning by reductio ad absurdum, let us consider that there exists one unit DMUo for which the optimum solutions of model (7), {u∗r }r=1,...,s and {v∗i }i=1,...,m , verify L ϕboOP =

therefore

s

∑ u∗r ybLro + ko = 1

(10)

r=1

m

∑ v∗i Xio = 1

(11)

i=1

As ybLro < ybU ro ; r = 1, ..., s then

s

m

r=1

i=1

∑ u∗r ybUro − ∑ v∗i X io + ko ≤ 0

s

∑ u∗r ybUro + ko >

r=1

and

s

∑ r=1

which contradicts (12).

(12)

s

∑ u∗r ybLro + ko = 1

(13)

r=1

u∗r ybU ro + ko −

6

m

∑ v∗i X io > 0 i=1

(14)

3.2. Bootstrap Methodology Bootstrap is a method to approach the distribution of statistics when this is difficult or impossible to determine analytically. The general idea is to use resampling to estimate an empirical distribution of the estimator of the parameter of interest (Efron [16]). In essence, in bootstrap, computational power is used as a substitute for calculation.
For the problem suggested in Section 2 with n DMUs and m known inputs X = X , . . . , X , a
sample m j 1 j j
u j of size n j to estimate the s outputs Yj = Y1 j , . . . ,Ys j and the output estimations b yj = yb1 j , . . . , ybs j , as in (3), are considered, j = 1, . . . , n, the steps of the observed bootstrapping confidence interval for the population efficiency scores procedure are: i. In the jth unit, let us consider abootstrap sample n o of size n j with which we obtain the bootstrap version of

the output estimations, b y∗j = yb∗1 j , . . . , yb∗s j . j=1,...,n   ii. With the data Xj ,b y∗j , using the Table 1 model, we obtain the bootstrap version of the estimated   j=1,...,n . efficiency scores ϕb∗j j=1,...,n

iii. Steps i. and ii. should be repeated B times n andothe B bootstrap version of the estimated efficiency scores ∗(b) for the jth DMU, j= 1, . . . , n, should be ϕb j . b=1,..., B

1−α 0 ,

the observed percentile bootstrap confidence interval for iv. Having fixed a coverage intention of level the population efficiency scores is h i ∗(α 0 /2) ∗(1−α 0 /2) b b ; j = 1, . . . , n (15) ϕj ,ϕj ∗(α)

where ϕb j

∗(b)

is the α -percentile of the B values ϕb j

.

The interval (15) observed interval of the bootstrapping confidence interval for the population efficiency i h is the 0 /2) 0 /2) ∗(α ∗(1−α b b ; j = 1, . . . , n. ,Φ score (BCIE) Φ j j

4. Evaluation of the methods

Simulation is the only method that allows the confidence to be verified and the length of the optimistic / pessimistic and bootstrapping confidence intervals efficiency to be compared, as the answers of all the customers (census) are available and, therefore, the true value of outputs and population efficiencies. Ceyhan and Benneyan [4] investigate the impact of the sample size on the measures of DEA efficiency if outputs and/or inputs are estimated with the answers from a survey sampling; though they do not propose a solution to the necessary sample size to control the error in the efficiency measures. We take a sample size in each DMU that guarantees an estimation error, e, of all their outputs with a probability, 1 − α , of making that error. Simulation study Table 3 corresponds to the data used in Cooper et al. [11] concerning the number of doctors and nurses (inputs) and the number of outpatients and inpatients (outputs) in 12 health centers. We use these data to generate the simulated population model as follows: in the jth DMU, j = 1, . . . , 12, a finite population 7

o n Pj = U1j , . . . , UNj j , of size N j is generated, where N j are independent random variables with uniform distribution in [10000, 50000], and    2  0 z1 j /4 z1 j Ukj ,→ N2 , z2 j 0 z22 j /4


and z1 j , z2 j are the original value outputs of the jth DMU, columns 4 and 5 of Table 3. Although there are other possibilities to generate the population data, we choose normal distribution because it is related to the use of the means to estimate the outputs. Sampling from the simulated population data reduces the effect that the distribution used in the simulation, in this case the normal distribution, has on the results. Table 3 Number of doctors, nurses, outpatients and inpatients in 12 health centers.

Doctor Nurse Outpatient X1 X2 Z1 1 2.0 15.1 10 2 1.9 13.1 15 3 2.5 16 16 4 2.7 16.8 18 5 2.2 15.8 9.4 6 5.5 25.5 23 7 3.3 23.5 22 8 3.1 20.6 15.2 9 3 24.4 19 10 5 26.8 25 11 5.3 30.6 26 12 3.8 28.4 25 Source: Table 1.5 Cooper et al. [11] DMU

Inpatient BCC-O Z2 efficiency score 9 1 5 1 5.5 0.925 7.2 1 6.6 0.767 9 0.955 8.8 1 8 0.826 10 0.990 10 1 14.7 1 12 1

Table 4 shows the simulated population model. In the second column, we have the population size of individuals of each DMU, in columns 3 and 4 the known inputs, the same as in Table 3. Columns 5 and 6 show the simulated values of the outputs, Y1 and Y2 , which correspond to the population mean: ! Nj Nj
∑k=1 Uk1 j ∑k=1 Uk2 j Y1 j ,Y2 j = ; j = 1, . . . , 12 , Nj Nj The last column shows the BCC-O population efficiency scores.

8

Table 4 Simulated population model.

DMU 1 2 3 4 5 6 7 8 9 10 11 12

Population size 43,341 24,438 45,606 12,578 19,314 21,782 19,024 36,271 30,691 28,385 28,005 49,077

Doctor Nurse Y1 Y2 X1 X2 2.0 15.1 9.98 9.01 1.9 13.1 14.98 5.01 2.5 16 15.99 5.50 2.7 16.8 17.96 7.18 2.2 15.8 9.40 6.58 5.5 25.5 22.96 8.97 3.3 23.5 21.99 8.77 3.1 20.6 15.17 8.01 3 24.4 19.02 10.04 5 26.8 24.89 10.01 5.3 30.6 26.11 14.69 3.8 28.4 25.09 11.98

BCC-O population efficiency score 1 1 0.926 1 0.766 0.957 0.998 0.826 0.991 1 1 1

The approximation to the confidence of the two CIE methodologies, as well as the analysis of their amplitudes, follows these steps: i. Fix the estimation error e = 0.9 and the confidence 1 − α = 0.9 in the jth DMU, then calculate, considering the WOR design, the sample size nr j , r = 1, 2; j = 1, ..., 12 as nr j ≥ 

where no = and

2 τ1−α/2

(er )2 φ −1 is

in order to verify

no no Nj

 +1

(16)

σr j 2 , τ1−α/2 = φ −1 (1 − α /2 ) the standard normal inverse cumulative distribution function
p Yˆr j −Yr j ≤ e ≥ 1 − α

Then, the sample size in the jth DMU is

 n j = max n1 j , n2 j , j = 1, . . . , 12

(17)

ii. Take the sample of size n j in the
jth DMU and estimate yˆr j ; r = 1, 2; j = 1, . . . , 12, as in (6). iii. With the data x1 j , x2 j , yˆ1 j , yˆ2 j j=1,...,12 , the optimistic/pessimistic and bootstrapping confidence intervals  efficiency, I j j=1,...,12 , are obtained. iv. Steps ii.-iii. are repeated 1000 times, and we obtain 1000 CIEs for each method o n (k) (k)L (k)U ] j = 1, . . . , 12 I j = [I j , I j k = 1, . . . , 1000

v. The confidence of the CIE methodology, for the jth DMU, is approximated with C j=

1 1000 ∑ I(ϕ j ∈I j (k)) 1000 k=1 9

vi. The expected value of the bounds of the CIEs, for the jth DMU, are approximated with E Ij

L




(k)L (k)U
∑1000 ∑1000 U k=1 I j k=1 I j and E I j = = 1000 1000

Simulation results: Table 5 shows the sample sizes obtained for each DMU, as in (16) and (17), for the value of output estimation error e = 0.9 and confidence 1 − α = 0.9. Table 5 Sample size obtained for each DMU considering the WOR design, a fixed output estimation error e=0.9 and a confidence 1−α=0.9.

DMU 1 2 3 4 5 6 7 8 9 10 11 12

nj 47 112 122 146 44 256 207 111 152 301 307 275

Table 6 shows the approximate confidence of the CIEs for the different methodologies and output oriented BCC model, e = 0.9 and α = 0.1 being fixed. We select two values, (1 − α 0 ) = 0.98 and (1 − α 0 ) = 0.9, for the confidence of the BCIEs. The simulated confidence of the optimistic/pessimistic and bootstrapping confidence intervals, (1 − α 0 ) = 0.98, are quite similar. Table 6 Simulated confidence of the optimistic/pessimistic and bootstrapping confidence intervals for the population efficiency scores. BCC-O model.

DMU

OPCIE

1 2 3 4 5 6 7 8 9 10 11 12

100 100 99.3 100 99.8 99.9 99.7 100 100 100 100 100

BCIE 0 1-α =0.98 100 100 98 99.9 96.9 98.7 98.8 98.2 98.9 99.9 100 100

BCIE 0 1-α =0.9 100 100 90.3 99.9 89.7 90.6 94.2 88.9 90.3 98.8 100 100

Figure 2 represents, with box-plots in each DMU , the length of the 1000 intervals for the two methodologies of CIEs, taking α = 0.1 and (1 − α 0 ) = 0.98, in the BCC-O model. In all the cases, the length of the bootstrap intervals is smaller than the length of the OPCIE intervals. 10

0.35

0.3

0.3

0.3

0.25

0.25

0.25

0.2 0.15

Interval length

0.35

Interval length

Interval length

0.35

0.2 0.15 0.1

0.1

0.05

0.05

0.05

[OPCIE]

0

[Bootst]

[OPCIE]

0.35

0.3

0.3

0.3

0.25

0.25

0.25

0.2 0.15

Interval length

0.35

0.2 0.15

0.15

0.1

0.1

0.05

0.05

0.05

[OPCIE]

0

[Bootst]

[OPCIE]

DMU4

0

[Bootst]

0.3

0.3

0.3

0.25

0.25

0.25 Interval length

0.35

Interval length

0.35

0.15

0.2 0.15

0.2 0.15

0.1

0.1

0.1

0.05

0.05

0.05

[OPCIE]

0

[Bootst]

[OPCIE]

DMU7

0

[Bootst]

0.3

0.3

0.3

0.25

0.25

0.25 Interval length

0.35

Interval length

0.35

0.15

0.2 0.15

0.2 0.15

0.1

0.1

0.1

0.05

0.05

0.05

[OPCIE]

[Bootst]

0

[OPCIE]

DMU10

[Bootst] DMU11

[Bootst] DMU9

0.35

0

[OPCIE]

DMU8

0.2

[Bootst] DMU6

0.35

0

[OPCIE]

DMU5

0.2

[Bootst]

0.2

0.1

0

[OPCIE] DMU3

0.35

Interval length

Interval length

0

[Bootst] DMU2

DMU1

Interval length

0.15

0.1

0

Interval length

0.2

0

[OPCIE]

[Bootst] DMU12

Figure 2 In each DMU, box-plot of the lengths of the optimistic/pessimistic and bootstrapping confidence intervals, α=0.1 and 0 1-α =0.98. BCC-O model.

11

Table 7 synthesizes the data from Figure 2 and shows the approximation of the expected values of the interval bounds. The expected values of the BCIEs are “better”, a smaller or identical upper bound and a bigger or identical lower bound being observed in all cases. If we look at the efficient population units {1, 2, 11, 12}, a null length in BCIEs can be observed, so this method detects the efficient units in any case, according to the classification that can be seen in (19). Table 7 Approximation of the expected values of the bounds of the optimistic/pessimistic and bootstrapping confidence intervals for 0 the population efficiency scores, α=0.1 and 1-α =0.98. BCC-O model.

OPCIE

DMU E Ij L E Ij U 1 0.837 1 2 0.891 1 3 0.843 0.998 4 0.903 1 5 0.647 0.917 6 0.881 0.998 7 0.911 1 8 0.743 0.929 9 0.887 1 10 0.927 1 11 0.933 1 12 0.932 1

BCIE

E Ij L E Ij U 1 1 1 1 0.850 0.995 0.964 1 0.685 0.882 0.887 0.994 0.927 1 0.774 0.888 0.919 1 0.949 1 1 1 1 1

In conclusion, the bootstrapping confidence interval method has three advantages: first, the a priori control of α 0 can lead to the achievement of the confidence of the interval efficiency required by the experimenter; second, it allows the efficient units to be detected in any case; and third, it has the smallest length and the "best" expected value of the bounds. 5. Illustrative example In this section, we apply the proposed confidence interval efficiency methodologies to a set of real data that fit the DEA model envisioned in this paper. We show how these intervals allow the DMUs to be classified in three defined efficiency classes, as in Despotis and Smirlis [13]. These data are public and the sample size of the health system users has been decided by the person responsible for the "Health Barometer" statistic survey. Spain’s Health Ministry has, for some time, compiled two statistics: “The Statistics of Health Establishments with Internship Regime (SHEWIR) and the "Health Barometer" (HB). In the first, a wide range of information is gathered from the health system, for example, the personnel’s indicators linked to the Spanish Health System and expense indicators from the public health system. In the second, starting from a sample design, a group of individuals is selected in each of Spain’s Autonomous Communities (CCAA) and a questionnaire is carried out to test the health system. In order to have some real data to apply the theory set out in this paper, we take as inputs the indicators of the personnel connected to the public health system (Doctors per 1000 inhabitants, Nurses per 1000 inhabitants, Nursing assistants per 1000 inhabitants) and the expense indicators of the public health system (Expenses for beds, outpatients, stays) obtained from the SHEWIR of 2009. To synthesize the information and reduce the number of inputs, a principal components analysis (PCA) is carried out in each of the groups of variables. The PCA1 and PCA2 are thus obtained, interpreted as size 12

of personnel and expenses, respectively. The mean answer of the sample of individuals, in each CCAA, to question 4 of the HB (2009) is used as output: “And, are you satisfied or dissatisfied with the manner in which the public health system operates in your Autonomous Community? Answer from 1 to 10, where 1 means ‘very dissatisfied’ and 10 means ‘very satisfied’”. Table 8 shows the sample sizes in each CCAA, (n j ), and the mean satisfaction of the user sample in each CCAA (Y1). The sample sizes have supposedly been determined with a certain estimation error, using the WOR design. The confidence interval for the mean satisfaction of the population of users in the jth CCAA is obtained as # " h i α r   α r   c Ybj , Ybj + φ −1 c Ybj ; j = 1, . . . 18; (18) YbjL , YbjU = Ybj − φ −1 Var Var 2 2   d Ybj is the variance estimator with WOR and φ −1 is the standard normal where Ybj is the sample mean, Var inverse cumulative distribution function, where α = 0.1. Table 8 Size of personnel and expenses of the Spanish Health System in 2009 (inputs), sample size to estimate the users’ mean satisfaction in Spain’s health system in 2009 (output), and estimated efficiency scores BCC-O.

CCAA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

Inputs Personnel Expense PCA1 PCA2 8.58 6.36 7.26 10 7.07 8.75 8.60 7.79 8.66 6.99 8.56 8.42 6.44 7.77 7.80 7.68 7.80 6.93 7.96 6.34 5.84 7.54 6.48 7.96 9.18 7.85 10 7.33 8.05 8.14 7.76 6.62 7.63 7.75 8.93 7.33

Sample size (nj ) 767 336 320 312 382 284 378 418 735 575 315 440 642 341 281 399 254 503

Output Estimated users mean satisfaction Y1 6.29 6.80 7.21 6.60 5.39 6.27 6.79 6.81 6.02 6.44 6.29 5.90 6.41 6.21 7.18 6.74 6.97 6.09

BCC-O estimated efficiency score 0.97 0.94 1 0.93 0.78 0.87 1 0.96 0.88 1 1 0.86 0.90 0.89 1 1 0.99 0.87

Source: The Statistics of Health Establishments with Internship Regime 2009 and Health Barometer 2009.

Table 9 shows the results of the estimation point and confidence interval of the population efficiency score in each of Spain’s regions, considering variable returns–to-scale.

13

Table 9 Spain’s CCAAs efficiency scores estimated point and by optimistic/pessimistic and bootstrapping confidence interval using data from The Statistics of Health Establishments with Internship Regime and The Health Barometer of 2009. BCC-O model.

CCAA 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

ϕbj

0.973 0.943 1 0.932 0.788 0.872 1 0.970 0.882 1 1 0.867 0.903 0.894 1 1 0.992 0.877

OPCIE Inf. Sup. bound bound 0.936 1 0.898 0.990 0.955 1 0.884 0.982 0.744 0.833 0.826 0.920 0.947 1 0.928 1 0.846 0.920 0.961 1 0.943 1 0.823 0.914 0.866 0.943 0.853 0.937 0.953 1 0.960 1 0.942 1 0.837 0.919

BCIE 98% Inf. Sup. bound bound 0.936 1 0.893 0.982 0.994 1 0.885 0.974 0.745 0.828 0.826 0.912 0.981 1 0.930 1 0.851 0.913 1 1 1 1 0.822 0.905 0.867 0.932 0.858 0.929 0.970 1 0.995 1 0.946 1 0.838 0.913

Table 10 shows the classification of the CCAAs in the sets E ++ , E + and E − defined as in Despotis and Smirlis [13]: n o E ++ = j ∈ J = {1, . . . , n}/I Lj = 1 n o (19) E + = j ∈ J/I Lj < 1 and IUj = 1 n o E − = j ∈ J/IUj < 1 The set E ++ consists of the units that are efficient in any case. The set E + consists of units that are efficient in a maximal sense, but in which there are input/output adjustments under which they cannot maintain their efficiency; in this case, a ranking approach is needed to compare and rank the efficiencies of different DMUs. Finally, the set E − consists of the definitely inefficient units.

Table 10 Classification of the CCAAs in the sets E ++ efficient in any case, E + efficient in a maximal sense and E − inefficient units, according to the confidence interval efficiency methodology.

OPCIE E++ E+ E−

1, 3, 7, 8, 10, 11, 15, 16, 17 2, 4, 5, 6, 9, 12, 13, 14, 18

BCIE 98% 10, 11 1, 3, 7, 8, 15, 16, 17 2, 4, 5, 6, 9, 12, 13, 14,18

The CCAAs in which the efficiency hypothesis is rejected using any of the CIE methodologies are {2, 4, 5, 6, 9, 12, 13, 14, 18}. The CCAAs {10, 11} are considered efficient in any case with the bootstrap method, while they are considered efficient in a maximal sense in the optimistic/pessimistic method. 14

6. Conclusions In this paper, we have considered the DEA context with known inputs and unknown outputs estimated with a sample. This context is reasonable in many examples of making units that provide the same services, where their resources can explain the opinion of their users. The questionnaire is the most common tool to find out user opinion and its items can measure, for example, user satisfaction or the time the service is used (in libraries the time of stay or in airports the check-in time and waiting for baggage collection), or some measure associated to the offered service (in recreational areas such as attractions parks or casinos, the quantity spent by each user). Based on these sample opinions, the outputs in each DMU, mean satisfaction, mean time of use or wait, mean expense, etc., would be estimated. We obtain two types of confidence interval for the DEA population efficiency score, the efficiency that would be obtained if it were possible to have the exact value of the population output data. The DEA models to obtain the intervals of the first type are inspired in the optimistic/pessimistic point of view of the models of Despotis and Smirlis [13] and Wang et al. [35], to obtain a non-random interval efficiency in the context of interval data. Bootstrap is the methodology used to obtain the second type of confidence interval efficiency. To the best of our knowledge, the bootstrap methodology has not been used in the context presented in this paper. The comparison of the methods by simulation shows that the bootstrapping confidence interval efficiency is the best option for two principal reasons: first, because it allows the required confidence of the interval efficiency to be controlled, while also maintaining a very similar amplitude to that of the method based on the optimistic/pessimistic point of view. Second, because it detects DMUs that are efficient in any case, while Theorem 1 shows the impossibility of this happening when using confidence interval efficiency based on the optimistic/pessimistic point of view. The example with real data in the Spanish health environment shows the utility of the confidence interval efficiency to detect not efficient Spanish regions (CCAAs) in a more precise way than with a purely estimated value of the efficiency score. Acknowledgements The authors would like to thank the Editor and anonymous referees for their detailed reading that resulted in this improved version of the paper. References [1] H. Azizi, Y.-M. Wang, Improved DEA models for measuring interval efficiencies of decision-making units, Measurement, 46(3) (2013) 1325–1332. [2] R.D. Banker, A. Charnes, W.W. Cooper, Some models for estimating technical and scale inefficiencies in data envelopment analysis, Management Science, 30(9) (1984) 1078–1092. [3] E. Bayraktar, E. Tatoglu, A. Turkyilmaz, D. Delen, S. Zaim, S, Measuring the efficiency of customer satisfaction and loyalty for mobile phone brands with DEA, Expert Systems with Applications, 39(1) (2012) 99–106. [4] M.E. Ceyhan, J.C. Benneyan, Handling estimated proportions in public sector data envelopment analysis, Annals of Operations Research, 221 (1) (2014) 107–132. 15

[5] V. Charles, D. M. Kumar, Satisficing data envelopment analysis: An application to SERVQUAL efficiency, Measurement, 51 (2014) 71–80. [6] A. Charnes, W.W. Cooper, Chance-constrained programming, Management Science, 6 (1) (1959) 73–79. [7] A. Charnes , W.W. Cooper, Deterministic equivalents for optimizing and satisficing under chance constraints, Operations Research, 11 (1) (1963) 18–39. [8] A. Charnes, W.W. Cooper, E. Rhodes, Measuring the efficiency of decision making units, Journal of Operational Research, 2(6) (1978) 429–444.

European

[9] W.W. Cooper, Z. Huang, S.X. Li SX, Satisficing DEA models under chance constraints, Operations Research, 66 (4) (1996) 279–295.

Annals of

[10] W.W. Cooper, H. Deng, Z. Huang, S.X. Li, Chance constrained programming approaches to technical efficiencies and inefficiencies in stochastic data envelopment analysis, Journal of the Operational Research Society, 53(12) (2002) 1347–1356. [11] W.W. Cooper, L.M. Seiford, K. Tone, Introduction to data envelopment analysis and its uses: with DEA-solver software and references, Springer Science & Business Media (2006). [12] C. Daraio, M. Diana, F. Di Costa, C. Leporelli, G. Matteucci, A. Nastasi, Efficiency and effectiveness in the urban public transport sector: a critical review with directions for future research, European Journal of Operational Research, 248(1) (2016) 1–20. [13] D.K. Despotis, Y.G. Smirlis, Data envelopment analysis with imprecise data, Operational Research, 140 (1) (2002) 24–36.

European Journal of

[14] K. De Witte, B. Geys, Citizen coproduction and efficient public good provision: Theory and evidence from local public libraries, European Journal of Operational Research, 224(3) (2013) 592–602 [15] W.E. Diewert, Measuring productivity in the public sector: some conceptual problems, Productivity Analysis, 36(2) (2011) 177–191.

Journal of

[16] B. Efron, R.J. Tibshirani, An introduction to the bootstrap, CRC press (1994). [17] F.R. Førsund, Measuring effectiveness of production in the public sector, Omega, 73 (2017) 93-103 [18] C.J. Hammond, Efficiency in the provision of public services: a data envelopment analysis of UK public library systems, Applied Economics, 34(5) (2002) 649–657. [19] J.A. Hemmeter, Estimating public library efficiency using stochastic frontiers, Public Finance Review, 34(3) (2006) 328–348. [20] Z. Huang, S.X. Li, Stochastic DEA models with different types of input-output disturbances, Journal of Productivity Analysis, 15(2) (2001) 95–113. [21] P. Korhonen, R. Tainio, R., J. Wallenius, Value efficiency analysis of academic research, Journal of Operational Research, 130(1) (2001) 121–132. 16

European

[22] K.C. Land, C.A. Lovell, S. Thore, Chance-constrained data envelopment analysis, Decision Economics, 14 (6) (1993) 541–554.

Managerial and

[23] H. Lee, C. Kim, Benchmarking of service quality with data envelopment analysis, Expert Systems with Applications, 41(8) (2014) 3761–3768. [24] C.K. Lovell, J.T. Pastor, Target setting: An application to a bank branch network, European Journal of Operational Research, 98 (2) (1997) 290–299. [25] D.J. Mayston, Analysing the effectiveness of public service producers with endogenous resourcing, Journal of Productivity Analysis, 44(1) (2015) 115–126. [26] D.J. Mayston, Data envelopment analysis, endogeneity and the quality frontier for public services, Annals of Operations Research, 250(1) (2017) 185–203. [27] O.B. Olesen, N.C. Petersen, Chance constrained efficiency evaluation, (1995) 442–457.

Management Science, 41 (3)

[28] J. Ruggiero, On the measurement of technical efficiency in the public sector. Operational Research, 90(3) (1996) 553–565.

European Journal of

[29] D. Santín, G. Sicilia, Dealing with endogeneity in data envelopment analysis applications, Systems with Applications, 68 (2017) 173–184.

Expert

[30] M. Shwartz, J.F. Burgess, J. Zhu, A DEA based composite measure of quality and its associated data uncertainty interval for health care provider profiling and pay-for-performance, European Journal of Operational Research, 253(2) (2016) 489–502. [31] L. Simar, P.W. Wilson, Sensitivity analysis of efficiency scores: How to bootstrap in nonparametric frontier models, Management Science, 44 (1) (1998) 49–61. [32] P. Smith, D. Mayston, Measuring efficiency in the public sector, Omega, 15(3) (1987) 181–189. [33] J.A. Tapia, B. Salvador, J. M. Rodríguez, Data envelopment analysis in satisfaction survey research: sample size problem, Journal of the Operational Research Society, 69 (7) (2018) 1096-1104. [34] M. Tavana, R.K. Shiraz, A. Hatami-Marbini, A new chance-constrained DEA model with birandom input and output data, Journal of the Operational Research Society, 65(12) (2014)1824–1839. [35] Y. Wang, R. Greatbanks, J. Yang, Interval efficiency assessment using data envelopment analysis, Fuzzy Sets and Systems, 153 (3) (2005) 347–370. [36] D. Wu, D.L. Olson, A comparison of stochastic dominance and stochastic DEA for vendor evaluationand, International Journal of Production Research, 46(8) (2008) 2313–2327.

17

5HYLVHGPDQXVFULSWZLWKQRFKDQJHVPDUNHG

&OLFNKHUHWRYLHZOLQNHG5HIHUHQFHV

&OLFNKHUHWRDFFHVVGRZQORDG5HYLVHGPDQXVFULSWZLWKQR FKDQJHVPDUNHG0($6&,(6UHYGHISGI

Data Envelopment Analysis with Estimated Output Data: Confidence Intervals Efficiency Jesús A. Tapiaa , Bonifacio Salvadora , Jesús M. Rodríguezb a Department

of Statistic and Operative Research, University of Valladolid, Campus Miguel Delibes, Paseo Belén nº7,47011 Valladolid, Spain b Statistical & Budgetary General Management of the Regional Government of Castile & Leon, Regional Tax Office, José Cantalapiedra nº2, 47014 Valladolid, Spain

Abstract Data Envelopment Analysis (DEA) is very useful for measuring the relative efficiency of such service producing units as hospitals or state services, etc. In recent years, some efficiency analysis approaches have used customer opinion as data as they consider it essential for measuring the quality of the service received. This paper measures the DEA efficiency of services by taking known inputs that explain satisfaction-opinion indexes (outputs) estimated with a customer sample. In this situation the DEA has to be stochastic and the obtained efficiency will be an estimation of the population efficiency scores that would have been obtained if we had had the population output data. Specifically, a bootstrap is performed that derives the confidence intervals needed to estimate the population DEA efficiency. A simulation is carried out which allows the confidence intervals to be validated and compared and an empirical application to measure the efficiency in the Spanish health environment is also described. Keywords: Stochastic data envelopment analysis, Confidence interval Efficiency, Bootstrap, Survey sampling. 2000 MSC: 62D05, 90C99 1. Introduction Measuring the relative efficiency of service-producing units has been an important purpose of Data Envelopment Analysis (DEA) for many years ([32], [28]). When the available public services information is deterministic, the efficiency is evaluated with the original DEA models proposed in Charnes et al. [8] or Banker et al. [2]. A vast academic literature sets out the applications of the classic DEA models, considering public services as decision-making units (DMU); for instance, health care ([30]), universities and research institutes ([21]), government ([15]), public libraries ([18], [19]), schools ([29]), or public transport services ([12]). In recent years, some approaches have considered the opinion of the customer to be crucial to appropriately defining outputs for the measurement of public service efficiency ([3], [14]). A consumer satisfaction-opinion survey allows the quality of the service to be measured and can be used to define opinion indexes as output variables of the DEA ( [23], [5], [25, 26], [29] and [17]). We propose to carry out satisfaction-opinion surveys of consumers, synthesizing their answers in satisfactionopinion indexes. We then use these indexes as data output in an efficiency DEA of public services. For example, Email addresses: [email protected] (Jesús A. Tapia), [email protected] ( Bonifacio Salvador), [email protected] (Jesús M. Rodríguez)

Preprint submitted to Measurement

November 29, 2019

in libraries, the resources (books, personnel,...) can explain the monthly mean time that an individual uses the library or the mean loans per user, information that can be estimated with a user survey sampling in the face of the impossibility of carrying out a user census. Data estimated with a sample incorporate random variations into the DEA analysis and different solutions to this situation have been proposed in the last few decades. One of them is chance-constrained programming, linear programming (LP) problems subject to constraints defined in terms of probability, where an "expected value" (E-model) or a "most probable value" (P-model) of the DEA efficiency is obtained ([6, 7], [22], [27], [9, 10], [20], [36], [34], [5]). Chance-constrained programming gives a point estimation solution to the efficiency measure. The methodology is different here as we use linear programming DEA models subject to deterministic constraints to obtain confidence intervals. In this paper, we focus on the measuring efficiency problem as a statistical problem. First, in each serviceproducing (our DMU), we estimate the unknown outputs using the opinion of a sample of customers. Second, we estimate the population DEA efficiency score with confidence intervals. By efficiency population, we understand that which would be obtained if, in each DMU, the output data were obtained with the opinion of the census of customers instead of the opinion of a customer sample. We propose two confidence interval methodologies that, to our knowledge, are novel. One is inspired in the optimistic/pessimistic point of view of DEA models and the other in the use of bootstrap replications from the sample of customers in each DMU. Finally, we also propose a solution to the sample size problem of customers needed to guarantee the confidence of our intervals. Ceyhan and Benneyan [4] discuss several approaches (Monte Carlo (MC), bootstrapping as in Simar et al. [31], and optimistic/pessimistic as in Wang et al. [35]) to obtain intervals for the efficiency with each methodology in the DEA problem conducted on values that include such estimated proportions as defect, satisfaction, mortality or adverse event rates estimated from samples of individuals. By means of a simulation procedure, Ceyhan and Benneyan show that the confidence of the MC is very low. The two confidence interval efficiency methodologies proposed in our paper are different from used by Ceyhan and Benneyan as we use the information of each customer sample, not just the estimated output. In a simulation similar to that of Ceyhan and Benneyan, we evaluate the confidence of our DEA interval efficiency methodology proposals by checking the good behavior of the bootstrapping confidence interval efficiency methodology. Being able to measure the confidence makes our efficiency measurement methodology completely different from the proposed efficiency intervals in Despotis and Smirlis [13], Wang et al. [35] or Azizi and Wang [1]. From the statistical point of view of our approach, the efficiency is a parameter and the quality of the measurement of that efficiency depends on the sample size of customers used to estimate the data output. Ceyhan and Benneyan [4] investigated the impact of the sample size, but they did not propose a solution to the necessary sample size to control the error in the efficiency measurements. With the same assumptions as in this paper, Tapia et al. [33] determined the customer sample size necessary in each public service-producing unit so that the efficiency estimation error in the service-producing units will be smaller than a previously fixed value; while in this paper, the customer sample size necessary in each public service-producing unit is determined so as to guarantee the confidence of our intervals. The rest of the paper is organized as follows. In the second section, the proposal of the problem is set out. Section three discusses methodologies to obtain the confidence interval efficiency in the DEA; while section four describes a simulation study that compares and evaluates these methodologies. Section five illustrates the application of the proposed intervals in a real example, which allows the efficiency to be analyzed in the healthcare environment of the autonomous communities of Spain. A summary of the main conclusions is given in the last section.

2

2. Approaching the problem We formally consider n fixed DMUs, each one using m inputs to produce s outputs. We denote Xj =
X1 j , .., Xm j as the value of the m known inputs n in the jth DMU o ( j = 1, . . . , n) . The s outputs of the jth
DMU are represented by a finite population Uj = U1j , . . . , UNj j of size N j , where Uk j = Uk1 j , . . . , Uks j are s different measures that represent the kth individual of the jth DMU, for example, s indices that measure their opinion. The unknown population outputs are:  

Yj = f Uj j=1,...,n = Y1 j , . . . ,Ys j j=1,...,n (1)
where f is a function that, applied to the population Uj , determines the s unknown outputs Y1 j , . . . ,Ys j . If, in the jth population, we consider V1j , . . . , Vnj j to be a random sample u j of size n j from the finite population Uj , where Vkj is a random vector of dimension s, the estimated outputs are:

b j = f u j = Yˆ1 j , . . . , Yˆs j ; j = 1, . . . , n Y

(2)

Having fixed a sample in each DMU, it is very convenient to distinguish between the estimators defined in (2) and their estimated values represented by:
b y j = yb1 j , . . . , ybs j ; j = 1, . . . , n

(3)

For example, in this paper, we use u j as a simple random sampling without replacement (WOR) of Uj to estimate the vector of population means, then: ( !) Nj Nj Nj
∑k=1 Ukj ∑k=1 Uks j ∑k=1 Uk1 j = ,..., (4) Yj = f Uj = Nj Nj Nj j=1,...,n

(

bj = f u j Y




nj

∑ Vkj = = k=1 nj

n

n

j j ∑k=1 Vks j ∑k=1 Vk1 j ,..., nj nj

!)

(5) j=1,...,n

h i and a confidence interval of level 1 − α for the rth output (population mean) of the jth DMU is YˆrLj , YˆrUj .
Having observed a sample vk1 j , . . . , vk1 j k = 1, . . . , n j with WOR design, in each DMU, then the estimation of j = 1, . . . , n

the output




b y j = yb1 j , . . . , ybs j =

n

n

j j ∑ vks j ∑k=1 vk1 j , . . . , k=1 nj nj

!

; j = 1, . . . , n

(6)

i h and the observed interval is ybLrj , ybU rj . Table 1 shows the variable returns-to-scale DEA model (BCC-O) that we work with. The choice of the output orientation is justified by the interest of observing the units which, while maintaining the same resources (inputs), can improve the opinion; we only consider the BCC model because, in the presence of output data estimated with an opinion survey sampling, we discard constant returns-to-scale. In this situation, we denote: 3


• Population efficiency index ϕ j j=1,...,n BCC-O as the maximization of the Table 1 model, taking the

data Xj , Y j j=1,...,n (see Figure 1). As Y j are unknown, ϕ j j=1,...,n are unknown parameters that we aim to estimate.  
bj of the efficiency indexes ϕ j j=1,...,n • Abusing the notation, we represent the estimators Φ j=1,...,n   bj BCC-O as the maximization of the Table 1 model, taking the data Xj , Y , in the understanding j=1,...,n
yj j=1,...,n to obtain the estimated value of the estimator that the models are maximized with the data Xj , b b j that we denote by ϕb j (see Figure 1). Φ Population Sample

Estimated output data ෡ ܻ

Population output data ܻ

Known Inputs ܺ DEA model BCC-O

Estimated Efficiency ෡ Ȱ

Population Efficiency ߮

Figure 1 The definition process of the population efficiency and how it is estimated.

Table 1 Output-oriented BCC model.


− s + max ϕ + ε ∑m s + s ∑ i=1 i r=1 r s.t. ∑nj=1 λ j yr j − s+ r = ϕ yro , r = 1, . . . , s − n ∑ j=1 λ j xi j + si = xio , i = 1, . . . , m + λ j ≥ 0, s− i ≥ 0, sr ≥ 0; j = 1, . . . , n; i = 1, . . . , m; r = 1, . . . , s n ∑ j=1 λ j = 1

4

i h b U in each DMU b L, Φ The main objective of this paper is to determine a confidence interval efficiency Φ j j that will guarantee, with a confidence 1 − α , that the interval contains the population efficiency scores ϕ j , j = 1, ..., n; that is, having fixed α ∈ (0, 1): i h  b Uj b Lj , Φ ≥ 1 − α ; j = 1, . . . , n p ϕj ∈ Φ

3. Confidence Interval Efficiency

In this section, we propose confidence intervals for the population DEA efficiency scores (CIEs) when the data are known inputs and unknown outputs such as those described in Section 2. Two types of interval are defined, one adapting the optimistic/pessimistic point of view to our problem and another using bootstrap. Let us consider n DMUs, m known inputs and s outputs estimated with confidence intervals. 3.1. Methodology Optimistic/Pessimistic Despotis and Smirlis [13] and Wang el al. [35] obtain interval efficiency using data known to lie within a bounded interval and the optimistic/pessimistic point of view. In other words, they give the efficiency of each DMU by means of interval data. These intervals are not confidence intervals because, for these authors, there is neither a statistical nor a stochastic problem, so they have no parameters to estimate. Our problem is different because we consider the input data to be known, while the unknown outputs are estimated with customer samples and, with this information, we estimate the population efficiency parameter represented in Figure 1. In each DMU, we propose a confidence interval to the population efficiency score based on the optimistic/pessimistic point of view. bU bL Models (7) and (8) derive h the observed i lower, ϕoOP , and upper, ϕoOP , bounds of our proposal of the conbL ,Φ bU fidence interval efficiency Φ for the DMUo , based on the optimistic/pessimistic point of view oOP

oOP

(OPCIE). The constraints of this model are different from the constraints used by Despotis and Smirlis and Wang el al. in the context of hinterval idata. h i In the notation of Section 2, YbrLj , YbrUj is the confidence interval output data and ybLrj , ybU r j is the observed con-

L , fidence interval. To obtain the lower bound of the observed confidence interval efficiency of the DMUo , ϕboOP we take the lower extreme of its observed confidence interval for the output data, ybLro as the output data of this DMU, and the upper bounds of all the observed confidence intervals n ofothe outputs of all the DMUs as the . To obtain the upper bound output data of the production frontier, including the one evaluated, ybU rj j=1,...,n

U , we take the upper bound of its observed of the observed confidence interval efficiency for the DMUo , ϕboOP confidence interval of the output data, ybU ro as the output data of this DMU, and the lower bounds of all observed confidence intervals of the outputs of all the DMUs as the output data of the production frontier, including the n o one evaluated, ybLrj . j=1,...,n

To obtain the lower bound of the OPCIE, model (7) uses the same production frontier in all the DMUs and the same happens with model (8) to obtain the upper bound. Therefore, the OPCIE uses two production frontiers, one for each interval bound, which allows us to obtain comparable interval bounds.

5

Table 2 Bound multiplicative models of the optimistic/pessimistic confidence interval for the population efficiency scores. Lower bound

Upper bound

L maxur ,vi ϕboOP = ∑sr=1 ur ybLro + ko s.t. ∑m i=1 vi Xio = 1 m ∑sr=1 ur ybU r j − ∑i=1 vi X i j + ko ≤ 0; j = 1, . . . , n ur , vi ≥ ε ∀r, i, ko free in sign

U = s maxur ,vi ϕboOP ∑r=1 ur ybU ro + ko s.t. ∑m i=1 vi Xio = 1 ∑sr=1 ur ybLrj − ∑m i=1 vi X i j + ko ≤ 0; j = 1, . . . , n ur , vi ≥ ε ∀r, i, ko free in sign

(7)

(8)

From model (8), it is clear that the optimal weights cannot guarantee upper bound efficiencies less than or equal to unity. No matter what weights the models use, the efficiencies of the DMUs are all limited to less than or equal to one. Theorem 1. The lower bound of the confidence interval efficiency obtained with the optimistic/pessimistic model (7) is strictly smaller than 1 for all the DMUs:

ϕbLjOP < 1; j = 1, ..., n

(9)

Proof. Reasoning by reductio ad absurdum, let us consider that there exists one unit DMUo for which the optimum solutions of model (7), {u∗r }r=1,...,s and {v∗i }i=1,...,m , verify L ϕboOP =

therefore

s

∑ u∗r ybLro + ko = 1

(10)

r=1

m

∑ v∗i Xio = 1

(11)

i=1

As ybLro < ybU ro ; r = 1, ..., s then

s

m

r=1

i=1

∑ u∗r ybUro − ∑ v∗i X io + ko ≤ 0

s

∑ u∗r ybUro + ko >

r=1

and

s

∑ r=1

which contradicts (12).

(12)

s

∑ u∗r ybLro + ko = 1

(13)

r=1

u∗r ybU ro + ko −

6

m

∑ v∗i X io > 0 i=1

(14)

3.2. Bootstrap Methodology Bootstrap is a method to approach the distribution of statistics when this is difficult or impossible to determine analytically. The general idea is to use resampling to estimate an empirical distribution of the estimator of the parameter of interest (Efron [16]). In essence, in bootstrap, computational power is used as a substitute for calculation.
For the problem suggested in Section 2 with n DMUs and m known inputs X = X , . . . , X , a
sample m j 1 j j
u j of size n j to estimate the s outputs Yj = Y1 j , . . . ,Ys j and the output estimations b yj = yb1 j , . . . , ybs j , as in (3), are considered, j = 1, . . . , n, the steps of the observed bootstrapping confidence interval for the population efficiency scores procedure are: i. In the jth unit, let us consider abootstrap sample n o of size n j with which we obtain the bootstrap version of

the output estimations, b y∗j = yb∗1 j , . . . , yb∗s j . j=1,...,n   ii. With the data Xj ,b y∗j , using the Table 1 model, we obtain the bootstrap version of the estimated   j=1,...,n . efficiency scores ϕb∗j j=1,...,n

iii. Steps i. and ii. should be repeated B times n andothe B bootstrap version of the estimated efficiency scores ∗(b) for the jth DMU, j= 1, . . . , n, should be ϕb j . b=1,..., B

1−α 0 ,

the observed percentile bootstrap confidence interval for iv. Having fixed a coverage intention of level the population efficiency scores is h i ∗(α 0 /2) ∗(1−α 0 /2) b b ; j = 1, . . . , n (15) ϕj ,ϕj ∗(α)

where ϕb j

∗(b)

is the α -percentile of the B values ϕb j

.

The interval (15) observed interval of the bootstrapping confidence interval for the population efficiency i h is the 0 /2) 0 /2) ∗(α ∗(1−α b b ; j = 1, . . . , n. ,Φ score (BCIE) Φ j j

4. Evaluation of the methods

Simulation is the only method that allows the confidence to be verified and the length of the optimistic / pessimistic and bootstrapping confidence intervals efficiency to be compared, as the answers of all the customers (census) are available and, therefore, the true value of outputs and population efficiencies. Ceyhan and Benneyan [4] investigate the impact of the sample size on the measures of DEA efficiency if outputs and/or inputs are estimated with the answers from a survey sampling; though they do not propose a solution to the necessary sample size to control the error in the efficiency measures. We take a sample size in each DMU that guarantees an estimation error, e, of all their outputs with a probability, 1 − α , of making that error. Simulation study Table 3 corresponds to the data used in Cooper et al. [11] concerning the number of doctors and nurses (inputs) and the number of outpatients and inpatients (outputs) in 12 health centers. We use these data to generate the simulated population model as follows: in the jth DMU, j = 1, . . . , 12, a finite population 7

o n Pj = U1j , . . . , UNj j , of size N j is generated, where N j are independent random variables with uniform distribution in [10000, 50000], and    2  0 z1 j /4 z1 j Ukj ,→ N2 , z2 j 0 z22 j /4


and z1 j , z2 j are the original value outputs of the jth DMU, columns 4 and 5 of Table 3. Although there are other possibilities to generate the population data, we choose normal distribution because it is related to the use of the means to estimate the outputs. Sampling from the simulated population data reduces the effect that the distribution used in the simulation, in this case the normal distribution, has on the results. Table 3 Number of doctors, nurses, outpatients and inpatients in 12 health centers.

Doctor Nurse Outpatient X1 X2 Z1 1 2.0 15.1 10 2 1.9 13.1 15 3 2.5 16 16 4 2.7 16.8 18 5 2.2 15.8 9.4 6 5.5 25.5 23 7 3.3 23.5 22 8 3.1 20.6 15.2 9 3 24.4 19 10 5 26.8 25 11 5.3 30.6 26 12 3.8 28.4 25 Source: Table 1.5 Cooper et al. [11] DMU

Inpatient BCC-O Z2 efficiency score 9 1 5 1 5.5 0.925 7.2 1 6.6 0.767 9 0.955 8.8 1 8 0.826 10 0.990 10 1 14.7 1 12 1

Table 4 shows the simulated population model. In the second column, we have the population size of individuals of each DMU, in columns 3 and 4 the known inputs, the same as in Table 3. Columns 5 and 6 show the simulated values of the outputs, Y1 and Y2 , which correspond to the population mean: ! Nj Nj
∑k=1 Uk1 j ∑k=1 Uk2 j Y1 j ,Y2 j = ; j = 1, . . . , 12 , Nj Nj The last column shows the BCC-O population efficiency scores.

8

Table 4 Simulated population model.

DMU 1 2 3 4 5 6 7 8 9 10 11 12

Population size 43,341 24,438 45,606 12,578 19,314 21,782 19,024 36,271 30,691 28,385 28,005 49,077

Doctor Nurse Y1 Y2 X1 X2 2.0 15.1 9.98 9.01 1.9 13.1 14.98 5.01 2.5 16 15.99 5.50 2.7 16.8 17.96 7.18 2.2 15.8 9.40 6.58 5.5 25.5 22.96 8.97 3.3 23.5 21.99 8.77 3.1 20.6 15.17 8.01 3 24.4 19.02 10.04 5 26.8 24.89 10.01 5.3 30.6 26.11 14.69 3.8 28.4 25.09 11.98

BCC-O population efficiency score 1 1 0.926 1 0.766 0.957 0.998 0.826 0.991 1 1 1

The approximation to the confidence of the two CIE methodologies, as well as the analysis of their amplitudes, follows these steps: i. Fix the estimation error e = 0.9 and the confidence 1 − α = 0.9 in the jth DMU, then calculate, considering the WOR design, the sample size nr j , r = 1, 2; j = 1, ..., 12 as nr j ≥ 

where no = and

2 τ1−α/2

(er )2 φ −1 is

in order to verify

no no Nj

 +1

(16)

σr j 2 , τ1−α/2 = φ −1 (1 − α /2 ) the standard normal inverse cumulative distribution function
p Yˆr j −Yr j ≤ e ≥ 1 − α

Then, the sample size in the jth DMU is

 n j = max n1 j , n2 j , j = 1, . . . , 12

(17)

ii. Take the sample of size n j in the
jth DMU and estimate yˆr j ; r = 1, 2; j = 1, . . . , 12, as in (6). iii. With the data x1 j , x2 j , yˆ1 j , yˆ2 j j=1,...,12 , the optimistic/pessimistic and bootstrapping confidence intervals  efficiency, I j j=1,...,12 , are obtained. iv. Steps ii.-iii. are repeated 1000 times, and we obtain 1000 CIEs for each method o n (k) (k)L (k)U ] j = 1, . . . , 12 I j = [I j , I j k = 1, . . . , 1000

v. The confidence of the CIE methodology, for the jth DMU, is approximated with C j=

1 1000 ∑ I(ϕ j ∈I j (k)) 1000 k=1 9

vi. The expected value of the bounds of the CIEs, for the jth DMU, are approximated with E Ij

L




(k)L (k)U
∑1000 ∑1000 U k=1 I j k=1 I j and E I j = = 1000 1000

Simulation results: Table 5 shows the sample sizes obtained for each DMU, as in (16) and (17), for the value of output estimation error e = 0.9 and confidence 1 − α = 0.9. Table 5 Sample size obtained for each DMU considering the WOR design, a fixed output estimation error e=0.9 and a confidence 1−α=0.9.

DMU 1 2 3 4 5 6 7 8 9 10 11 12

nj 47 112 122 146 44 256 207 111 152 301 307 275

Table 6 shows the approximate confidence of the CIEs for the different methodologies and output oriented BCC model, e = 0.9 and α = 0.1 being fixed. We select two values, (1 − α 0 ) = 0.98 and (1 − α 0 ) = 0.9, for the confidence of the BCIEs. The simulated confidence of the optimistic/pessimistic and bootstrapping confidence intervals, (1 − α 0 ) = 0.98, are quite similar. Table 6 Simulated confidence of the optimistic/pessimistic and bootstrapping confidence intervals for the population efficiency scores. BCC-O model.

DMU

OPCIE

1 2 3 4 5 6 7 8 9 10 11 12

100 100 99.3 100 99.8 99.9 99.7 100 100 100 100 100

BCIE 0 1-α =0.98 100 100 98 99.9 96.9 98.7 98.8 98.2 98.9 99.9 100 100

BCIE 0 1-α =0.9 100 100 90.3 99.9 89.7 90.6 94.2 88.9 90.3 98.8 100 100

Figure 2 represents, with box-plots in each DMU , the length of the 1000 intervals for the two methodologies of CIEs, taking α = 0.1 and (1 − α 0 ) = 0.98, in the BCC-O model. In all the cases, the length of the bootstrap intervals is smaller than the length of the OPCIE intervals. 10

0.35

0.3

0.3

0.3

0.25

0.25

0.25

0.2 0.15

Interval length

0.35

Interval length

Interval length

0.35

0.2 0.15 0.1

0.1

0.05

0.05

0.05

[OPCIE]

0

[Bootst]

[OPCIE]

0.35

0.3

0.3

0.3

0.25

0.25

0.25

0.2 0.15

Interval length

0.35

0.2 0.15

0.15

0.1

0.1

0.05

0.05

0.05

[OPCIE]

0

[Bootst]

[OPCIE]

DMU4

0

[Bootst]

0.3

0.3

0.3

0.25

0.25

0.25 Interval length

0.35

Interval length

0.35

0.15

0.2 0.15

0.2 0.15

0.1

0.1

0.1

0.05

0.05

0.05

[OPCIE]

0

[Bootst]

[OPCIE]

DMU7

0

[Bootst]

0.3

0.3

0.3

0.25

0.25

0.25 Interval length

0.35

Interval length

0.35

0.15

0.2 0.15

0.2 0.15

0.1

0.1

0.1

0.05

0.05

0.05

[OPCIE]

[Bootst]

0

[OPCIE]

DMU10

[Bootst] DMU11

[Bootst] DMU9

0.35

0

[OPCIE]

DMU8

0.2

[Bootst] DMU6

0.35

0

[OPCIE]

DMU5

0.2

[Bootst]

0.2

0.1

0

[OPCIE] DMU3

0.35

Interval length

Interval length

0

[Bootst] DMU2

DMU1

Interval length

0.15

0.1

0

Interval length

0.2

0

[OPCIE]

[Bootst] DMU12

Figure 2 In each DMU, box-plot of the lengths of the optimistic/pessimistic and bootstrapping confidence intervals, α=0.1 and 0 1-α =0.98. BCC-O model.

11

Table 7 synthesizes the data from Figure 2 and shows the approximation of the expected values of the interval bounds. The expected values of the BCIEs are “better”, a smaller or identical upper bound and a bigger or identical lower bound being observed in all cases. If we look at the efficient population units {1, 2, 11, 12}, a null length in BCIEs can be observed, so this method detects the efficient units in any case, according to the classification that can be seen in (19). Table 7 Approximation of the expected values of the bounds of the optimistic/pessimistic and bootstrapping confidence intervals for 0 the population efficiency scores, α=0.1 and 1-α =0.98. BCC-O model.

OPCIE

DMU E Ij L E Ij U 1 0.837 1 2 0.891 1 3 0.843 0.998 4 0.903 1 5 0.647 0.917 6 0.881 0.998 7 0.911 1 8 0.743 0.929 9 0.887 1 10 0.927 1 11 0.933 1 12 0.932 1

BCIE

E Ij L E Ij U 1 1 1 1 0.850 0.995 0.964 1 0.685 0.882 0.887 0.994 0.927 1 0.774 0.888 0.919 1 0.949 1 1 1 1 1

In conclusion, the bootstrapping confidence interval method has three advantages: first, the a priori control of α 0 can lead to the achievement of the confidence of the interval efficiency required by the experimenter; second, it allows the efficient units to be detected in any case; and third, it has the smallest length and the "best" expected value of the bounds. 5. Illustrative example In this section, we apply the proposed confidence interval efficiency methodologies to a set of real data that fit the DEA model envisioned in this paper. We show how these intervals allow the DMUs to be classified in three defined efficiency classes, as in Despotis and Smirlis [13]. These data are public and the sample size of the health system users has been decided by the person responsible for the "Health Barometer" statistic survey. Spain’s Health Ministry has, for some time, compiled two statistics: “The Statistics of Health Establishments with Internship Regime (SHEWIR) and the "Health Barometer" (HB). In the first, a wide range of information is gathered from the health system, for example, the personnel’s indicators linked to the Spanish Health System and expense indicators from the public health system. In the second, starting from a sample design, a group of individuals is selected in each of Spain’s Autonomous Communities (CCAA) and a questionnaire is carried out to test the health system. In order to have some real data to apply the theory set out in this paper, we take as inputs the indicators of the personnel connected to the public health system (Doctors per 1000 inhabitants, Nurses per 1000 inhabitants, Nursing assistants per 1000 inhabitants) and the expense indicators of the public health system (Expenses for beds, outpatients, stays) obtained from the SHEWIR of 2009. To synthesize the information and reduce the number of inputs, a principal components analysis (PCA) is carried out in each of the groups of variables. The PCA1 and PCA2 are thus obtained, interpreted as size 12

of personnel and expenses, respectively. The mean answer of the sample of individuals, in each CCAA, to question 4 of the HB (2009) is used as output: “And, are you satisfied or dissatisfied with the manner in which the public health system operates in your Autonomous Community? Answer from 1 to 10, where 1 means ‘very dissatisfied’ and 10 means ‘very satisfied’”. Table 8 shows the sample sizes in each CCAA, (n j ), and the mean satisfaction of the user sample in each CCAA (Y1). The sample sizes have supposedly been determined with a certain estimation error, using the WOR design. The confidence interval for the mean satisfaction of the population of users in the jth CCAA is obtained as # " h i α r   α r   c Ybj , Ybj + φ −1 c Ybj ; j = 1, . . . 18; (18) YbjL , YbjU = Ybj − φ −1 Var Var 2 2   d Ybj is the variance estimator with WOR and φ −1 is the standard normal where Ybj is the sample mean, Var inverse cumulative distribution function, where α = 0.1. Table 8 Size of personnel and expenses of the Spanish Health System in 2009 (inputs), sample size to estimate the users’ mean satisfaction in Spain’s health system in 2009 (output), and estimated efficiency scores BCC-O.

CCAA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

Inputs Personnel Expense PCA1 PCA2 8.58 6.36 7.26 10 7.07 8.75 8.60 7.79 8.66 6.99 8.56 8.42 6.44 7.77 7.80 7.68 7.80 6.93 7.96 6.34 5.84 7.54 6.48 7.96 9.18 7.85 10 7.33 8.05 8.14 7.76 6.62 7.63 7.75 8.93 7.33

Sample size (nj ) 767 336 320 312 382 284 378 418 735 575 315 440 642 341 281 399 254 503

Output Estimated users mean satisfaction Y1 6.29 6.80 7.21 6.60 5.39 6.27 6.79 6.81 6.02 6.44 6.29 5.90 6.41 6.21 7.18 6.74 6.97 6.09

BCC-O estimated efficiency score 0.97 0.94 1 0.93 0.78 0.87 1 0.96 0.88 1 1 0.86 0.90 0.89 1 1 0.99 0.87

Source: The Statistics of Health Establishments with Internship Regime 2009 and Health Barometer 2009.

Table 9 shows the results of the estimation point and confidence interval of the population efficiency score in each of Spain’s regions, considering variable returns–to-scale.

13

Table 9 Spain’s CCAAs efficiency scores estimated point and by optimistic/pessimistic and bootstrapping confidence interval using data from The Statistics of Health Establishments with Internship Regime and The Health Barometer of 2009. BCC-O model.

CCAA 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

ϕbj

0.973 0.943 1 0.932 0.788 0.872 1 0.970 0.882 1 1 0.867 0.903 0.894 1 1 0.992 0.877

OPCIE Inf. Sup. bound bound 0.936 1 0.898 0.990 0.955 1 0.884 0.982 0.744 0.833 0.826 0.920 0.947 1 0.928 1 0.846 0.920 0.961 1 0.943 1 0.823 0.914 0.866 0.943 0.853 0.937 0.953 1 0.960 1 0.942 1 0.837 0.919

BCIE 98% Inf. Sup. bound bound 0.936 1 0.893 0.982 0.994 1 0.885 0.974 0.745 0.828 0.826 0.912 0.981 1 0.930 1 0.851 0.913 1 1 1 1 0.822 0.905 0.867 0.932 0.858 0.929 0.970 1 0.995 1 0.946 1 0.838 0.913

Table 10 shows the classification of the CCAAs in the sets E ++ , E + and E − defined as in Despotis and Smirlis [13]: n o E ++ = j ∈ J = {1, . . . , n}/I Lj = 1 n o (19) E + = j ∈ J/I Lj < 1 and IUj = 1 n o E − = j ∈ J/IUj < 1 The set E ++ consists of the units that are efficient in any case. The set E + consists of units that are efficient in a maximal sense, but in which there are input/output adjustments under which they cannot maintain their efficiency; in this case, a ranking approach is needed to compare and rank the efficiencies of different DMUs. Finally, the set E − consists of the definitely inefficient units.

Table 10 Classification of the CCAAs in the sets E ++ efficient in any case, E + efficient in a maximal sense and E − inefficient units, according to the confidence interval efficiency methodology.

OPCIE E++ E+ E−

1, 3, 7, 8, 10, 11, 15, 16, 17 2, 4, 5, 6, 9, 12, 13, 14, 18

BCIE 98% 10, 11 1, 3, 7, 8, 15, 16, 17 2, 4, 5, 6, 9, 12, 13, 14,18

The CCAAs in which the efficiency hypothesis is rejected using any of the CIE methodologies are {2, 4, 5, 6, 9, 12, 13, 14, 18}. The CCAAs {10, 11} are considered efficient in any case with the bootstrap method, while they are considered efficient in a maximal sense in the optimistic/pessimistic method. 14

6. Conclusions In this paper, we have considered the DEA context with known inputs and unknown outputs estimated with a sample. This context is reasonable in many examples of making units that provide the same services, where their resources can explain the opinion of their users. The questionnaire is the most common tool to find out user opinion and its items can measure, for example, user satisfaction or the time the service is used (in libraries the time of stay or in airports the check-in time and waiting for baggage collection), or some measure associated to the offered service (in recreational areas such as attractions parks or casinos, the quantity spent by each user). Based on these sample opinions, the outputs in each DMU, mean satisfaction, mean time of use or wait, mean expense, etc., would be estimated. We obtain two types of confidence interval for the DEA population efficiency score, the efficiency that would be obtained if it were possible to have the exact value of the population output data. The DEA models to obtain the intervals of the first type are inspired in the optimistic/pessimistic point of view of the models of Despotis and Smirlis [13] and Wang et al. [35], to obtain a non-random interval efficiency in the context of interval data. Bootstrap is the methodology used to obtain the second type of confidence interval efficiency. To the best of our knowledge, the bootstrap methodology has not been used in the context presented in this paper. The comparison of the methods by simulation shows that the bootstrapping confidence interval efficiency is the best option for two principal reasons: first, because it allows the required confidence of the interval efficiency to be controlled, while also maintaining a very similar amplitude to that of the method based on the optimistic/pessimistic point of view. Second, because it detects DMUs that are efficient in any case, while Theorem 1 shows the impossibility of this happening when using confidence interval efficiency based on the optimistic/pessimistic point of view. The example with real data in the Spanish health environment shows the utility of the confidence interval efficiency to detect not efficient Spanish regions (CCAAs) in a more precise way than with a purely estimated value of the efficiency score. Acknowledgements The authors would like to thank the Editor and anonymous referees for their detailed reading that resulted in this improved version of the paper. References [1] H. Azizi, Y.-M. Wang, Improved DEA models for measuring interval efficiencies of decision-making units, Measurement, 46(3) (2013) 1325–1332. [2] R.D. Banker, A. Charnes, W.W. Cooper, Some models for estimating technical and scale inefficiencies in data envelopment analysis, Management Science, 30(9) (1984) 1078–1092. [3] E. Bayraktar, E. Tatoglu, A. Turkyilmaz, D. Delen, S. Zaim, S, Measuring the efficiency of customer satisfaction and loyalty for mobile phone brands with DEA, Expert Systems with Applications, 39(1) (2012) 99–106. [4] M.E. Ceyhan, J.C. Benneyan, Handling estimated proportions in public sector data envelopment analysis, Annals of Operations Research, 221 (1) (2014) 107–132. 15

[5] V. Charles, D. M. Kumar, Satisficing data envelopment analysis: An application to SERVQUAL efficiency, Measurement, 51 (2014) 71–80. [6] A. Charnes, W.W. Cooper, Chance-constrained programming, Management Science, 6 (1) (1959) 73–79. [7] A. Charnes , W.W. Cooper, Deterministic equivalents for optimizing and satisficing under chance constraints, Operations Research, 11 (1) (1963) 18–39. [8] A. Charnes, W.W. Cooper, E. Rhodes, Measuring the efficiency of decision making units, Journal of Operational Research, 2(6) (1978) 429–444.

European

[9] W.W. Cooper, Z. Huang, S.X. Li SX, Satisficing DEA models under chance constraints, Operations Research, 66 (4) (1996) 279–295.

Annals of

[10] W.W. Cooper, H. Deng, Z. Huang, S.X. Li, Chance constrained programming approaches to technical efficiencies and inefficiencies in stochastic data envelopment analysis, Journal of the Operational Research Society, 53(12) (2002) 1347–1356. [11] W.W. Cooper, L.M. Seiford, K. Tone, Introduction to data envelopment analysis and its uses: with DEA-solver software and references, Springer Science & Business Media (2006). [12] C. Daraio, M. Diana, F. Di Costa, C. Leporelli, G. Matteucci, A. Nastasi, Efficiency and effectiveness in the urban public transport sector: a critical review with directions for future research, European Journal of Operational Research, 248(1) (2016) 1–20. [13] D.K. Despotis, Y.G. Smirlis, Data envelopment analysis with imprecise data, Operational Research, 140 (1) (2002) 24–36.

European Journal of

[14] K. De Witte, B. Geys, Citizen coproduction and efficient public good provision: Theory and evidence from local public libraries, European Journal of Operational Research, 224(3) (2013) 592–602 [15] W.E. Diewert, Measuring productivity in the public sector: some conceptual problems, Productivity Analysis, 36(2) (2011) 177–191.

Journal of

[16] B. Efron, R.J. Tibshirani, An introduction to the bootstrap, CRC press (1994). [17] F.R. Førsund, Measuring effectiveness of production in the public sector, Omega, 73 (2017) 93-103 [18] C.J. Hammond, Efficiency in the provision of public services: a data envelopment analysis of UK public library systems, Applied Economics, 34(5) (2002) 649–657. [19] J.A. Hemmeter, Estimating public library efficiency using stochastic frontiers, Public Finance Review, 34(3) (2006) 328–348. [20] Z. Huang, S.X. Li, Stochastic DEA models with different types of input-output disturbances, Journal of Productivity Analysis, 15(2) (2001) 95–113. [21] P. Korhonen, R. Tainio, R., J. Wallenius, Value efficiency analysis of academic research, Journal of Operational Research, 130(1) (2001) 121–132. 16

European

[22] K.C. Land, C.A. Lovell, S. Thore, Chance-constrained data envelopment analysis, Decision Economics, 14 (6) (1993) 541–554.

Managerial and

[23] H. Lee, C. Kim, Benchmarking of service quality with data envelopment analysis, Expert Systems with Applications, 41(8) (2014) 3761–3768. [24] C.K. Lovell, J.T. Pastor, Target setting: An application to a bank branch network, European Journal of Operational Research, 98 (2) (1997) 290–299. [25] D.J. Mayston, Analysing the effectiveness of public service producers with endogenous resourcing, Journal of Productivity Analysis, 44(1) (2015) 115–126. [26] D.J. Mayston, Data envelopment analysis, endogeneity and the quality frontier for public services, Annals of Operations Research, 250(1) (2017) 185–203. [27] O.B. Olesen, N.C. Petersen, Chance constrained efficiency evaluation, (1995) 442–457.

Management Science, 41 (3)

[28] J. Ruggiero, On the measurement of technical efficiency in the public sector. Operational Research, 90(3) (1996) 553–565.

European Journal of

[29] D. Santín, G. Sicilia, Dealing with endogeneity in data envelopment analysis applications, Systems with Applications, 68 (2017) 173–184.

Expert

[30] M. Shwartz, J.F. Burgess, J. Zhu, A DEA based composite measure of quality and its associated data uncertainty interval for health care provider profiling and pay-for-performance, European Journal of Operational Research, 253(2) (2016) 489–502. [31] L. Simar, P.W. Wilson, Sensitivity analysis of efficiency scores: How to bootstrap in nonparametric frontier models, Management Science, 44 (1) (1998) 49–61. [32] P. Smith, D. Mayston, Measuring efficiency in the public sector, Omega, 15(3) (1987) 181–189. [33] J.A. Tapia, B. Salvador, J. M. Rodríguez, Data envelopment analysis in satisfaction survey research: sample size problem, Journal of the Operational Research Society, 69 (7) (2018) 1096-1104. [34] M. Tavana, R.K. Shiraz, A. Hatami-Marbini, A new chance-constrained DEA model with birandom input and output data, Journal of the Operational Research Society, 65(12) (2014)1824–1839. [35] Y. Wang, R. Greatbanks, J. Yang, Interval efficiency assessment using data envelopment analysis, Fuzzy Sets and Systems, 153 (3) (2005) 347–370. [36] D. Wu, D.L. Olson, A comparison of stochastic dominance and stochastic DEA for vendor evaluationand, International Journal of Production Research, 46(8) (2008) 2313–2327.

17