- Email: [email protected]
Comparative analysis of UTA multicriteria methods
European Journal of Operational Research 130 (2001) 246±262
www.elsevier.com/locate/dsw
Comparative analysis of UTA multicriteria methods q Michel Beuthe *, Giuseppe Scannella Facult es Universitaires Catholiques de Mons GTM-Groupe Transport et Mobilite 151, Chauss ee de Binche, B-7000 Mons, Belgium Received 17 November 1997; accepted 2 March 1998
Abstract This paper reviews the main variants of the utility additive (UTA) multicriteria method, and systematically compares their predictive performance on two sets of data. It analyses both those cases where the model provides a ranking with errors and those without errors. First, it shows that the reference projects should be chosen carefully in order to elicit as much information as possible from the decision maker: a set of projects satisfying a fractional factorial plan is recommended. Second, it discusses the use of alternative post-optimality methods for solving the problem of multiple solutions and their dierent predictive performances. Third, it presents the results of simulations based on utility functions involving interdependence between criteria, and shows that UTA handles this problem eectively by an adjustment of its coecients. Finally, the in¯uence of the model's parameters on the predictive performance is also investigated. Ó 2001 Elsevier Science B.V. All rights reserved. Keywords: Linear programming; Multicriteria analysis; Utility theory; Optimisation
1. Introduction There are many dierent methods of multicriteria analysis which can be recommended de-
q This paper presents some results of the research program EUNET on ``Socio-Economic and Spatial Impacts of Transport'' commissioned by the European Community within the Fourth Framework Program for Research and Development. Part of the research was also founded by a FSRIU (Fonds Special de la Recherche inter-Universitaire) grant from the French Community of Belgium. * Corresponding author. Tel.: +32-65-32-32-97; fax: 32-65-3156-91. E-mail address: [email protected] (M. Beuthe).
pending upon the circumstances of decision making. For the assessment of investments in public infrastructure, it may be advisable to use a method of total aggregation in order to compute trade-o coecients between criteria. This is actually the basic motivation of this paper which will discuss a subset of total aggregation methods applied to the ranking of road infrastructure projects. Among these, the UTA method initially proposed by Jacquet-Lagreze and Siskos (1978, 1982) has several interesting features: it makes possible the estimation of a nonlinear additive function, which is obtained by the use of a linear program which provides a convenient piecewise linear ap-
0377-2217/01/$ - see front matter Ó 2001 Elsevier Science B.V. All rights reserved. PII: S 0 3 7 7 - 2 2 1 7 ( 0 0 ) 0 0 0 4 2 - 4
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
proximation of the function, and the only information required from the decision maker are global stated preferences between projects. This latter characteristics present some advantages as well as some problems when questioning the decision maker about his preferences, but, as will be shown in this paper, it makes it possible to solve to some extent the problem of interdependence between criteria while maintaining the utility function additivity. A number of variants of UTA have been proposed in the literature over the last 15 years. It appeared useful at this point to take stock of these models, and to compare their predictive performance on a few cases. Moreover, like all models, UTA and its variants have speci®c problems, the main one being the possibility that the linear program could have several solutions. When it comes to selecting one or several projects among the set of projects used for estimating the utility function, this situation is of no consequence, at least if the multiple functions are estimated without errors. But, if there remain errors in the estimation, or if the problem is to apply the utility function to a set of other projects, the use of dierent utility functions will lead to dierent rankings. The ®rst part of this paper presents an overview of the main variants of UTA: the basic model to start with, then its post-optimality analyses, and several dierent speci®cations of its error terms. The second part applies the methods thus reviewed to estimating three dierent utility functions on two sets of projectsÕ data. The ®rst set corresponds to an optimal fractional factorial plan used already in a paper by Beuthe and Scannella (1996). The other one is taken from a paper by Despotis et al. (1990). The cases of estimation without errors and with errors are treated separately. In both cases the relevant modelsÕ predictive performance are compared. The third part examines the case where the assumed utility function of the decision maker is not additive and presents some interdependence between the outcomes. The fourth part brie¯y investigates the in¯uence of the modelsÕ ®xed parameters on the predictive performance. From the results of all these simulations useful conclusions can be drawn for future use of this methodology.
247
2. The UTA model and its variants 2.1. The basic model The problem is to compare, rank and evaluate a set of actions, or projects, with respect to N different criteria which measure the favourable consequences of the projects. The measurements of these consequences are given by the vector g a g1 a; g2 a; . . . ; gN a for any project a belonging to A. As an example, for a highway project, the gi a could be the cost±bene®t ratio, its favourable impact on safety, on the environment, etc. The existence of an additive utility function is assumed: U g a
N X
ui gi a
1
i1
with ui gi P 0
and
dui > 0; dgi
which satis®es the classic axioms of decision theory, namely the axioms of comparability, re¯exivity, transitivity of choices, continuity and strict dominance. The additivity implies in particular that the partial utility of a criterion ui gi a depends only on the level of that particular criterion. For a discussion about the additive utility functions see for instance Keeney and Raia (1976). This function provides an aggregation of the criteria in a common index to compare, rank and assess the projects. UTA estimates the function U on a set of reference projects A0 by approximating the utility of each alternative a 2 A0 by U 0 g a
N X ui gi a r a; i1
with r(a) being a non-negative potential error relative to the utility of each alternative a. This is done by the method of linear goal programming proposed by Charnes and Cooper (1961,1977), which provides an approximation by linear intervals of the nonlinear functions. The objective
248
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
function of UTA, to be minimised, is the sum of these errors: X r a: F
N X ui gi 1 and ui gi 0
5
i1
0
8i; r a P 0 8a 2 A ;
ui gij P 0
8i; 8j:
a2A0
In order to apply that method, the ®eld of variation of each criterion gi ; gi , de®ned by its least favourable value of that criterion gi and its best value gi , is divided into ai equal intervals gij ; gij1 . The variables to be estimated by the program are the partial utilities at these bounds, say ui gij . The utility at intermediate values of the criteria are given by linear interpolation. For each pair of projects (a, b) belonging to A0 , the decision-maker, taking into account the set of criteria, must express his/her overall preferences or indierences. The results of these comparisons are introduced under the form of this constraint N X
fui gi a ÿ ui gi bg r a ÿ r b P d
i1
2
() aPb in the event of strict preferences, and N X
fui gi a ÿ ui gi bg r a ÿ r b 0
i1
3
() a I b; in the event of a strict preference. The constant d on the right-hand side of the inequality (2) must be strictly positive. Its value can very well in¯uence the solution of the program, so that it must not be given too high an initial value. The hypothesis that the partial utilities increase with the value of the criteria imposes a series of additional constraints: ui gij1 ÿ ui gij P si ; j 1; 2; . . . ; ai ; i 1; 2; . . . ; N ;
4
where si must be (strictly) positive. As for d, it is better to give it a small initial value. This question will be dealt with in Section 6. Finally there are normalisation and non-negativity conditions:
At this point we should mention that, in order to improve the accuracy of the estimation, more detailed speci®cations could also be adopted if additional information can be given by the decision maker, such as utilitiesÕ dierences between projects. However, the questionnaire addressed to the decision maker becomes then necessarily longer and more dicult. It may not be practical to go to such lengths, and therefore we will not do so. In UTA, a solution, namely the estimated additive utility function, may be of two types: ®rst, when F 0, any solution belonging to the convex set of solutions admitted by the constraints of the program provides a ranking identical with that given by the decision maker. In the event of positive errors, F > 0, no solution is admitted by the system of constraints, unless some error terms take a positive value. As a matter of fact, in this case, there does not exist a nonlinear additive utility function representing perfectly the preferences expressed by the decision maker. But, in both cases, the solutions may not be the only one as in any linear program and the set of partial utility functions may be very dierent from one solution to another. If we exclude the case of a decision maker who would be unable to reasonably compare the projects or who would be `irrational' in the sense of exhibiting intransitive preferences, the presence of errors may indicate that the decision makerÕs preferences are characterised by partial utilities which are not independent of each other or which are not monotonically increasing, in other words, which do not follow the assumptions made in the model. But, it may also be the case, more simply, that the intervals chosen should have been more numerous or de®ned in a dierent way. We should point out, however, that an additive function of nonlinear partial utility functions is already quite a ¯exible speci®cation; moreover, as will be shown, it can always provide an estimation of utilities which incorporates a certain degree of interdependence among criteria. Actually, as will be
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
shown below, it is our practical experience that it is dicult to simulate an assumed utility function with interdependence such that positive errors are obtained in its estimation. 2.2. Post-optimality analyses Naturally, UTA, like any other multicriteria method must be regarded only as a tool to help the decision maker. After a ®rst run it should always be possible to converse with the decision maker in order to specify or somewhat correct some stated preferences and obtain a better understanding of the estimated function. In the worst case, it may appear that the functionÕs speci®cation is not at all appropriate; in other cases, some adjustments of the program on the basis of additional or corrected information may produce an acceptable utility function. A software like UTA+ (Kostkowski and Slowinski, 1996) includes the possibility of such an interactive process. However, all eorts must be made to ®rst improve the solution in order to re¯ect his ®rst stated preferences as much as possible. For that reason, post-optimality analyses of the results have been proposed, which explore the set of computed optimal solutions, or solutions around that set. Even though the latter solutions correspond to larger F , experience has shown that they may produce a weak order of preferences closer to the stated one (see Despotis et al., 1990). A complete list of the possible post-optimality analyses can be found in Beuthe and Scannella (1996). In this paper, only the methods which could lead to a more limited set of solution, will be examined both cases, i.e., where F 0 and F > 0. The quality of the resulting utility functions will also be assessed in terms of their ranking performance. First of all, Jacquet-Lagreze and Siskos (1978) proposed that, when there is not a single solution, a function should simply be used which would be the average of the extreme optimal functions obtained from a sensitivity analysis applied on the last bounds of each criterion. The sensitivity analysis is made by adding this new constraint to the model
X r a 6 F h;
249
6
a2A0
where h is a small positive number. In the basic model, it was noted that the values given to d and s were to some extent arbitrary. Obviously, their level in¯uences the results as well as the predictive quality of the model, as will be illustrated by simulations in a later section. For that reason, and following a suggestion by Srinivasan and Shocker (1973), it is worthwhile to look for optimal values of d and/or s. First, it is possible to aim at accentuating the dierence between utilities by maximising d, the minimum dierence between the utility of two dierent actions. This model, we named UTAMP1 in order to point out that, on the basis of UTA, it maximises d to better identify the relations of preference between projects. The objective of this second-step program is then to maximise d, subject to constraints (2)±(6). For the case where F 0, this post-optimality analysis is included in the software Jacquet-Lagreze (1984). This case has been discussed and tested by Beuthe and Scannella (1996) with the purpose of reducing the set of solutions. In eect, by choosing the solution(s) which maximise(s) d among all the optimal solutions, the optimal set can only be reduced. A side eect is also to limit the convexity or concavity of the utility function. Beuthe and Scannella (1996) have also proposed to maximise the sum d s in order to stress not only the dierences of utilities between projects but also the dierences between utilities at successive bounds. Note that the simple addition of those two parameters is legitimate since both of them are de®ned in the same utility units. However, a weighted sum could also be considered. For this procedure, which was named UTAMP2, the constraints of the previous program, UTAMP1 remain unchanged but all the si become equal to s P 0. The maximisation of d s also reduces the set of optimal solutions. When F 0, these two post-optimality procedures are justi®ed by the dual relationship between the objective function of the UTA primal program and the objective function of its dual. Indeed, at the optimum
250
F
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
X a
r a d
X j
yj s
X xk z 0; k
where the yj Õs are the dual variables corresponding to the strict preference constraints (2), the xk 's are the dual variables of conditions (4), and z corresponds to the normalisation condition in (5). Given that d and s are strictly positive, all the dual variables must equal zero. Thus, in this case, marginal variations of d and s cannot have the eect of increasing F . However, in cases, where F > 0, some of the dual variables will be positive and variation of d and/or s could increase F , which would result in a ranking of lesser quality. Actually, when there are positive errors, at least one preference constraint will be saturated and at least one dual variable yj will be positive. Hence, the variation of d will automatically aect the level of F . This will be seen in some experiments related in Section 5. Another proposal by Despotis et al. (1990) for the case where F > 0 is to minimise the dierence between the minimum and maximum error of the ®rst estimation. This can be done by MInimising the Maximum individual Error z, z P r a for all a in A0 , in a linear model based on UTA and which we named UTAMIME.
2.3. Other speci®cations of the errors The errorsÕ speci®cation in the basic UTA model could be made more ¯exible. Whenever the value F obtained is zero, another more ¯exible speci®cation could not, obviously, reduce the problem of multiple solutions, but, when F > 0, another speci®cation could give dierent results. Two other speci®cations have been proposed in the literature: ®rst, Siskos and Yannacopoulos (1985) have introduced negative errors rÿ (a) beside the positive ones r (a), in a model named UTASTAR (see also Despotis et al., 1990). The objective function becomes the sum of all these errors, which are introduced in the UTA constraints. For instance, constraint (2) becomes
N X fui gi a ÿ ui gi bg r a ÿ rÿ a i1
ÿ r b rÿ b P d () a P b;
0
2
with r a P 0, rÿ a P 0 8a 2 A0 . Another possibility advanced by JacquetLagreze and Siskos (1982) is to introduce an error term r(ab) proper for each constraint comparing two projectsÕ utilities. This speci®cation is certainly needed when the subjective preferences obtained from pairwise judgements are not transitive. This model will be named UTA2. The objective function to be minimised is again the sum of all these errors. Since it is necessary to introduce a constraint for each comparison between two pairs of projects, m m ÿ 1=2 such constraints are needed if m projects are compared, while there are only (m ) 1) constraints in the basic UTA model. A typical UTA constraint is changed into N X fui gi a ÿ ui gi bg r ab P d () a P b; i1
200 with r ab P 0 8 ab 2 A0 . In the same spirit, Jacquet-Lagreze and Siskos (1982) have proposed another model which minimises the number of violated pairs of R in the weak order R0 given by U g. It is equivalent, in fact, to maximising KendallÕs s. The solution is given by a mixed linear program where the variables c(ab) of the objective function are discrete variables which take the value 1 whenever a preference relationship is violated, and the value 0 when it is satis®ed. The sum of all these errors is minimised under the following constraints: N X fui gi a ÿ ui gi bg M c ab P d i1
() a P b; N P i1 N P i1
2000 ;
9 > > fui gi a ÿ ui gi bg M c ab P 0 > = > > fui gi b ÿ ui gi ag M c ba P 0 > ; () a I b;
3000
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
with c ab 0 or 1 8 ab 2 R, and where M is a large number. Here again, the ®rst two sets of constraints must be applied to all pairs of projects. If the dierence U g a ÿ U g b for a pair a; b 2 R is greater than d, then c ab 0 and the judgement is respected, otherwise c ab 1 and it is violated. We named that model UTAMKEN. This presentation is slightly more general than in the original paper of Jacquet-Lagreze and Siskos (1982), because the indierence judgements are explicitly taken into account in the present paper. Unhappily, even in the best case, ®nding the solution may take a very large number of iterations. This is due to the large number of discrete variables in the model. The constraints (3000 ) include the possibility of indierences, hence errors in indierence constraints are taken into account when minimising the sum of errors. Note that KendallÕs usual index neglects such errors. 3. First set of simulations: Belgian road infrastructure projects 3.1. The basic data These simulations are applied to the case of 353 road projects under consideration for the building of the Belgian network during the period 1985±2010. After a ®rst cost±bene®t study (see Blauwens et al., 1982), the Road Administration, in collaboration with the Centre of Road Research (CRR), realised a multicriteria analysis (Ministere des Travaux Publics, 1985) that used 29 criteria regrouped in six main themes: the safety on the present road, the projectsÕ socioeconomic aspects, the impact on environment, the current and future trac, the problems of planning and urbanism, and ®nally the wear state of the current road. For each of the 29 criteria, the CRR built a discrete scale going from 1 to 5 in order to measure the positive eects of each project. Within each theme, an intermediate weighting was established by a team of experts: in order to aggregate the measurements of the 29 criteria in six indexes corresponding to the six main themes
251
(a discussion of the scales and methodology of the CRR is given by Delhaye (1993) and van Hecke (1993)). Finally, those six indexes were aggregated in a unique measurement of utility by a primary weighting ®xed by the political decision maker. The measurement scales as well as the intermediate weightings of the CRRs experts are taken for granted here, and the six indexes will be used as criteria for the simulations using the variants of the UTA model. However, in place of estimating the utility function on the basis of the CRRs preferences between some real projects, 25 ®ctitious projects will be used, which make up a fractional factorial plan (see Appendix A) according to a method suggested by Addelman (1962a,b). In eect, this matrix gives the maximum information which is possible to be obtained from the smallest number of projects for that number of criteria. Applying this information procedure on the above set of data, we show in the last section of the present paper that the quality of estimation in terms of forecasting power is substantially increased in this way with respect to the case with real projects. 3.2. Setting of the simulations For most simulations, a standard linear programming Simplex algorithm was used. But, UTAMKEN required a mixed linear programming Branch and Bound algorithm. As a software we used the OMP package of Beyers & Partners. The simulations were run on a PC 486 DX-4 100 MHz. Each LP simulation took about 5 seconds. The search of a set of optimal extreme solution in a post-optimality analysis took about 2 N 10 seconds, where N is the number of constraints. A simple FORTRAN program was used to compute the average solutions and the performance indices. Thus, a complete simulation took only a few minutes. Concerning the mixed LP based simulations, the algorithm takes much more time for ®nding the solution, due to the number of binary variables (see Table 1). The number of constraints and variables varies with the model. They are presented in Table 1.
252
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
Table 1 Dimensions of the UTA based modelsa Model UTA UTASTAR UTA2 UTAMKEN UTAMP1 UTAMP2 UTAMIME UTASTARMIME UTA2MIME
Number of constraints P Ac m P NI1 ai 1 c b m nI1 ai1 P C c m m ÿ 1=2 1 nI1 ai1 PN c D jP j 2 jIj 1 I1 ai 1 fAc ; Bc or C c g 1 fAc ; Bc or C c g 1 Ac 1 m Bc 1 2 m C c 1 jP j 2 jIj
Number of variables P Av m NI1 P ai 1 v B 2 m nI1 aP i 1 C c jP j 2 jIj NI1 ai 1 P Dv jP j 2 jIj NI1 ai 1 fAv ; Bv or C v g 1 fAv ; Bv or C v g 2 Av 1 Bv 1 Cv 1
a
The interpolation constraints are not included. jP j: number of strict preference constraints considering all the possible combinations. jIj: number of strict indierence constraints considering all the possible combinations. In UTAMKEN model, the jP j 2 jIj variables are binary variables.
3.3. Case where F 0 For this ®rst simulation, it is assumed that the decision maker's utility is additive and a linear combination of six concave partial utility functions. His/her preferences based on this function are introduced in constraints (2) and (3) of the UTA program. The solution obtained from the program and the corresponding ranking of the projects in A are then compared to the assumed utility function and the ranking it gives to all the projects. In this way, we can assess the UTAs performance in representing the decision makerÕs preferences. The assumed partial utility functions are illustrated in Fig. 1. Since in this case F is found equal to zero, there is no point to introduce another speci®cation of errors. Thus, only the post-optimality proce-
dures UTAMP1 and UTAMP2 are applied. Among the three methods, only UTAMP2 provides a unique solution. The average partial utility functions provided by UTA and UTAMP1 were also computed. The average functions of UTA and the functions of UTAMP2 are illustrated, respectively in Figs. 2 and 3. In both cases, we can observe that some of the functions are loosing their concavity. The presence of multiple solutions is detected by a sensitivity analysis on the utilities of the last bounds. Its results appear in Table 2, where the minimum and maximum values taken by the utilities are given. The gaps between the estimated functions and the assumed theoretical functions are also of interest and appear in Table 3. They appear as the average of the dierences between the estimated partial utilities at the ®ve bounds
Fig. 1. Concave assumed partial utility functions.
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
253
Fig. 2. Partial utility functions of the average solution UTA.
Fig. 3. Partial utility functions by UTAMP2.
and the corresponding theoretical utilities. The global average is the average of these average gaps over the six criteria. Another way to compare the three models is to analyse their performance in ranking all the projects in A. A useful performance index is the KendallÕs index s(m) (Kendall and Stuart, 1968)
which measures in this case the correlation (like all correlation index, s varies between )1 and 1) between the rankings produced by an estimated function and by the assumed theoretical function. Another useful index, called intersection density (see Despontin et al., 1986), d(m), consists in measuring the percentage of common elements in
Table 2 Sensitivity analysis (UTA±UTAMP1) Number of the criterion
Theoretical weights
UTA method: variation intervals
UTAMP1 method: variation intervals
Minimum
Maximum
Minimum
Maximum
1 2 3 4 5 6
0.25 0.20 0.15 0.20 0.10 0.10
0.231 0.106 0.124 0.192 0.057 0.051
0.314 0.224 0.172 0.239 0.131 0.133
0.269 0.195 0.142 0.215 0.071 0.069
0.295 0.196 0.142 0.229 0.085 0.094
0.0092 0.0100 0.0062 0.0052 0.0053 0.0134 0.0082 0.2588 0.1955 0.1495 0.1997 0.1036 0.0926 0.0150 0.0157 0.0057 0.0111 0.0211 0.0272 0.0160 0.25 0.20 0.15 0.20 0.10 0.10 1 2 3 4 5 6
0.2644 0.1695 0.1500 0.2135 0.1007 0.1017
0.0132 0.0275 0.0032 0.0053 0.0110 0.0188 0.0131
0.2881 0.1953 0.1418 0.2183 0.0812 0.0750
Solution Average solution
Average gap with respect to the theoretical functions
UTAMP1 method
Average solution
Average gap with respect to the theoretical functions UTA method
Theoretical weights Number of the criterion
Table 3 Solutions and average gaps (UTA, average solution-UTAMP1, average solution-UTAMP2)
Average gap with respect to the theoretical functions
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
UTAMP2 method
254
the reference core of two rankings, for instance among the 50 projects ranked ®rst in the compared rankings. Compared to the ÔaverageÕ UTA, the results provided by UTAMP1 show important shortcomings. First, if the variation intervals at the last bounds (see Table 2) are smaller for UTAMP1, they never contain the corresponding weight of the assumed function; second, the ÔgapsÕ of the average UTAMP1 function with respect to the assumed function are often larger than those of the average UTA function (see Table 3). Finally, if these two average functions are applied to all projects in A, the weakness of UTAMP1 is revealed by a performance of lower quality (see Table 4). However, when compared to the results of UTAMP2, the solution provided by average UTA appears inferior on all counts.
3.4. Case where F > 0 On the basis of the same data, a subjective ranking of the 25 projects belonging to the reference set A0 is assumed such that the rankings obtained from estimated utility function are not consistent with the assumed subjective ranking. In that case, the sum of errors F > 0. Three of the models explained above can be applied, namely UTA, UTASTAR and UTA2. Because of the presence of multiple solutions for each of them, only their average solution will be considered. But, the post-optimality procedures corresponding to UTAMP2 and UTAMIME will be applied to each of them. The corresponding solution will be named on the one hand UTAMIME, UTASTARMIME and UTA2MIME, respectively, and on the other UTAMP2, UTAMP2STAR and UTAMP22. The UTAMP1 post-optimality method is not taken into account here, because it is useless. Indeed, in the case of F > 0 at least one preference constraint of type (2) is satis®ed as an equality with an error equal to d. Thus, at that level of F it will be impossible with the UTAMP1 model to ®nd a d larger than the starting d. Table 5 below describes the performance of all these models.
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
255
Table 4 Statistics concerning the models UTA, UTAMP1 and UTAMP2 UTA (average solution) UTAMP1 (average solution) UTAMP2
d
s
r a
s 353
d 50
Fixed at 0.01 0.01591 0.00884
Fixed at 0.01 Fixed at 0.01 0.02122
0 ± ±
0.894 0.874 0.930
0.860 0.900 0.920
Table 5 Statistics concerning the average solutions of UTA, UTASTAR and UTA2 and their related post-optimality analyses P d s rmax a r a s 25 d 10 UTA UTASTAR UTA2 UTAMIME UTASTARMIME UTA2MIME UTAMP2 UTAMP2STAR UTAMP22
Fixed at Fixed at Fixed at Fixed at Fixed at Fixed at 0 0.00035 0.00088
0.001 0.001 0.001 0.001 0.001 0.001
Fixed at Fixed at Fixed at Fixed at Fixed at Fixed at 0.0012 0.00099 0.00012
0 0 0 0 0 0
First of all, it should be pointed out that the total errors of UTA2 cannot be compared to the other two modelsÕ sum of errors because they do not have the same meaning. We observe that the results produced by the UTA and UTAMIME are slightly superior to those of the other models. We should note also that, in this case, the post-optimality MIME procedure does not improve the ®nal results. On the other hand, the results produced by UTAMP2 are not good at all. This can be explained by the dual relationship between the UTAs primal problem and its dual. When F > 0, some of the dual variables must be positive, so that, when d s is maximised under the constraint of a ®xed sum of errors, d or s will tend to zero depending on the values taken by the dual variables associated with them. This is not likely to induce an improved solution, since it means that the estimated dierences of utility will tend to vanish. In this particular case, note that d 0 in the basic UTAMP2 procedure. Even though, it appears to work better when applied to UTASTAR and UTA2, we can conclude that this post-optimality procedure can only be recommended for the cases where F 0:
± ± ± 0.0120 0.0055 0.0042 ± ± ±
0.018 0.018 0.051 ± ± ± ± ± ±
0.913 0.904 0.905 0.913 0.857 0.902 0.738 0.890 0.898
1 1 1 1 1 1 0.80 1 1
4. Second set of simulations with F > 0 This is a case taken from a paper by Despotis et al. (1990). The set A0 consists of 14 alternatives, de®ned by four hypothetical quantitative criteria which can take four dierent values. Their values as well as the ranking given by the `decision maker' can be found in Appendix B. Here also, the three dierent models, UTA, UTASTAR and UTA2 have been applied and compared to each other (see Table 6). But for this smaller problems, the solution of UTAMKEN model was only found after 20 hours of iterations! As explained above, this model implies a very large number of constraints and binary variables, which lead to an extremely long computation. UTAMP2 post-optimality analysis is not presented here since F > 0 (see above). All these models give here a unique solution. Hence, a post-optimality analysis is useful only for exploring the set of solutions around the optimal solution. The model with the best predictive performance is UTAMKEN. This is only natural since it aims at maximising s. UTASTAR and its post-optimality analysis UTASTARMIME provide the second best results. Again, the MIME post-optimality analyses presented in the second
256
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
Table 6 Statistics concerning the (unique) solutions of UTA, UTASTAR and their post-optimality analyses MIME and UTAMKENa UTA UTASTAR UTA2 UTAMKEN UTAMIME UTASTARMIME UTA2MIME a
d ®xed at
s ®xed at
rmax a
Rr a
s 14
d 7
0.001 0.001 0.001 0.001 0.001 0.001 0.001
0 0 0 0 0 0 0
± ± ± ± 0.024 0.023 0.024
0.051 0.036 0.14 1 ± ± ±
0.472 0.670 0.769 0.780 0.450 0.670 0.549
0.714 0.857 0.430 0.857 0.571 0.857 0.430
Application from Despotis et al. (1990).
part of the table, does not improve the predictive performance of the models. Actually, given a level of F , this post-optimality analysis minimises the maximum error, but may increase at the same time the level of some other errors, and, as a consequence, may lower the predictive quality of the estimated utility function.
5. Utility aected by interdependence between criteria In the above simulations, the decision makerÕs assumed utility function was simply additive. But what would happen if the decision makerÕs subjective preferences were aected by some interdependence between criteria? In order to examine this problem, a set of estimations of UTA additive utilities was computed on the basis of stated preferences between projects resulting from utility functions characterised by some interdependence between outcomes. Thus, we assumed that the decision makerÕs utility for a project A was u A
6 6 Y X ui gi A a gi A i1
i1
b g 1 A g3 A c g1 A g 3 A g4 A d g1 A g6 A; where gi (A) is the outcome of A on the ith impact, ui gi A is the partial utility of the outcome of A on the ith impact and a±d are the coecients of the interdependence terms. The simulations are made on dierent combinations of these terms. The assumed partial utility functions of the decision
maker are the ones of Fig. 1. Table 7 summarises the results. The ®rst set of simulations is based on the same ®ctitious set of projects which satis®es a fractional factorial design. The second set is based on the same real projects A as above. In the second column, the combination of the interdependence terms is indicated by their coecients. The weights given to these terms appear in the third column. Let us note, to begin with that multiple solutions have been obtained in all these cases, so that the KendallÕs index has been computed with reference to an average solution. It is seen that, in all cases, the UTA estimation is made without error, while, in most cases, the KendallÕs s(353) remains at a relatively high level. Thus, the ranking is not much aected by the presence of interdependence in the decision makerÕs preferences. Actually, the estimated coecients are biased in order to compensate the presence of interdependence terms which are neglected by the UTA speci®cation. Some other forms of utility functions with interdependence were also considered, but we found out that it was very dicult indeed to generate cases with errors. In the few cases where some errors were generated, the utility functions of the decision maker had actually a sinusoidal shape with local saturation. Such a shape can hardly be accepted as anything normal and need not be considered in practice. Ex post facto, we realised that a similar aberration should explain the preference orders of the two cases with errors which were discussed above. Thus, even if there is interdependence in the decision makerÕs utility function, the UTA model
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
257
Table 7 Simulations cases with interdependence (d 0:001 and s 0) Reference projects
Interdependence terms
Weight factor of interdependence terms
Resulting value of F by UTA
KendallÕs s 353
Fictitious
± a, b, c, d a, b, c a, b, d a, b a, d c, d a b c d
± 0.01, 2, 4, 6 0.01, 1, 8 0.005, 1, 1 0.01, 1 0.003, 0.7 4, 5 0.003 7 5 7
0 0 0 0 0 0 0 0 0 0 0
0.874 0.853 0.821 0.890 0.886 0.865 0.820 0.840 0.735 0.874 0.811
Real
± a, b, c, d a, b, c a, b, d a, b a, d c, d a b c d
± 0.01, 2, 4, 6 0.01, 1, 8 0.005, 1, 1 0.01, 1 0.003, 0.7 4, 5 0.003 10 5 5
0 0 0 0 0 0 0 0 0 0 0
0.709 0.755 0.812 0.540 0.732 0.840 0.737 0.672 0.638 0.793 0.665
Table 8 Results of the simulation cases for the eect of d, s and h in case of F > 0 d ®xed at
s ®xed at
h ®xed at
Rr a
s 25
UTA 0.001 0.001 0.001 0.002 0.005 0.010 0.001 0.001 0.001 0.001
0 0.0005 0.001 0 0 0 0 0 0 0
0.00001 0.00001 0.00001 0.00001 0.00001 0.00001 0.00005 0.00010 0.00050 0.00095
0.018 0.025 0.032 0.036 0.090 0.180 ± ± ± ±
0.913 0.878 0.872 0.913 0.913 0.910 0.899 0.900 0.902 0.904
UTASTAR 0.001 0.001 0.001 0.002 0.005 0.010 0.001 0.001 0.001 0.001
0 0.0005 0.001 0 0 0 0 0 0 0
0.00001 0.00001 0.00001 0.00001 0.00001 0.00001 0.00005 0.00010 0.00050 0.00095
0.018 0.022 0.029 0.036 0.090 0.180 ± ± ± ±
0.904 0.858 0.832 0.897 0.896 0.892 0.904 0.904 0.905 0.900
258
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
provides in practice good results in terms of F and s.
6. Parameters in¯uence on predictive quality As we have noted, the choice of parameters d, s and h may in¯uence the solution of the linear program and, therefore, the ranking of projects. By means of simulations, their impacts on the (average) solution of dierent models are examined on the basis of the KendallÕs index. A number of cases is considered: when F 0 and F > 0, when there is interdependence in the utility function and when there is not, also when the reference set satis®es a fractional factorial plan and when the set is made up of real projects.
6.1. Case where F > 0 This is certainly not the most interesting case, if we keep in mind the results obtained in Section 5. Indeed, when there are positive errors, whenever possible, a dialogue should take place between the analyst and the decision maker in order to let her/him adjust her/his order of preferences and obtain a function estimated without errors. But it is relevant to pursue the analysis for the cases where such a dialogue is not possible. Here, the reference set of projects satis®es the same fractional factorial plan, and the stated preferences are those of Section 3.3. The sensitivity analysis is applied on both UTA and UTASTAR models. The KendallÕs index s is computed for the reference set A0 of projects jAj 25. The results are given in Table 8.
Table 9 Results of the simulation cases for the eect of d and s in case of F 0 UTA without interdependence
a
UTA with interdependence
d ®xed at
s ®xed at
s (353)
d ®xed at
s ®xed at
s 353
Orthogonal plan 0.001 0.001 0.001 0.001 0.001 0.001 0.002 0.005 0.010 0.015 0.0197b 0.0088c
0 0.0005 0.0010 0.0075 0.0150 0.0248a 0 0 0 0 0 0.021c
0.874 0.872 0.871 0.876 0.891 0.901 0.884 0.905 0.897 0.847 0.810 0.930
0.001 0.001 0.001 0.001 0.001
0 0.0005 0.0010 0.0075 0.0091a
0.820 0.821 0.822 0.810 0.818
0.002 0.005 0.0076b 0c
0 0 0 0.0103c
0.825 0.833 0.838 0.808
Set of real projects 0.001 0.001 0.001 0.001 0.001 0.002 0.0031b 0c
0 0.0005 0.0010 0.0075 0.0333a 0 0 0.0036c
0.709 0.776 0.717 0.774 0.877 0.818 0.771 0.858
0.001 0.001 0.001 0.001 0.001 0.0020 0.0036b 0.0028c
0 0.0005 0.0010 0.0075 0.0100a 0 0 0.008c
0.742 0.805 0.766 0.850 0.835 0.839 0.815 0.837
Maximal value of s for a given value of d. Maximal value of d for a given value of s (UTAMP1 procedure). c Maximal value of (d s) (UTAMP2 procedure). b
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
It is seen, ®rst of all, that, in most cases, the simple UTA average solution produces the most faithful ranking of the projects. Secondly, the dierent combinations of d and s provide dierent levels of F . This phenomenon is explained, as above in similar cases, by the primal-dual relationship in linear programming. Thirdly, higher levels of d and s have generally a rather negative in¯uence on the s statistics. It is particularly the case for s variations, while the eect of d is very weak in this case. This suggests that smaller values of these parameters will lead to better results. SRINIVASAN and SHOCKER recommend a d equal to 1= 100 Q, where Q is the number of indierence classes in the reference set. In the present case, d should then be equal to 0.0005. Our ®ndings tend to support this recommendation. Finally, the value of h, which is used in the dierent post-optimality analyses, appears to have a rather weak eect on the predictive quality.
259
However, on the basis of all our simulations, we would recommend the use of a very small value. 6.2. Case where F 0 Here, four dierent cases are examined: · the case where the decision makerÕs utility function is assumed additive and the case where there is interdependence between the criteria; · the case where the reference set satis®es a fractional factorial plan and when the set is made up of 25 real projects. The sensitivity analysis on d and s (h being ®xed at zero) is applied only to the UTA model, and the results are assessed by the s statistics over the full sample of 353 projects (see Table 9). With F 0, we know from the primal-dual relationship that small increases of d or s should have no impact on the level of errors. Hence, a tighter adjustment of the utility function with
Table 10 Fractional factorial plan: projectsÕ criteria and assumed rankings Projects 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
g1 1 1 1 1 1 2 2 2 2 2 3 3 3 3 3 4 4 4 4 4 5 5 5 5 5
g2 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
g3 1 2 3 4 5 2 3 4 5 1 3 4 5 1 2 4 5 1 2 3 5 1 2 3 4
g4 1 3 5 2 4 2 4 1 3 5 3 5 2 4 1 4 1 3 5 2 5 2 4 1 3
g5 1 4 2 5 3 2 5 3 1 4 3 1 4 2 5 4 2 5 3 1 5 3 1 4 2
g6 1 5 4 3 2 2 1 5 4 3 3 2 1 5 4 4 3 2 1 5 5 4 3 2 1
Ranking (F 0)
(F > 0)
25 23 19 16 14 24 17 20 6 10 20 13 12 8 14 7 22 9 3 3 1 18 5 10 2
25 23 4 18 8 24 20 18 6 16 16 13 6 15 11 22 21 13 11 10 1 2 5 3 8
260
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
Table 11 Multicriteria evaluation of 25 real projects belonging to the Belgian Road Infrastructure projects set: projectsÕ criteria and assumed ranking Projects
g1
g2
g3
g4
g5
g6
Ranking
327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351
4.00 3.60 3.35 3.00 4.00 3.00 3.00 3.00 3.00 3.30 3.00 2.75 3.00 3.00 3.00 3.60 3.20 2.90 3.15 3.30 3.55 3.00 3.65 3.85 3.25
2.60 3.30 2.95 3.30 3.25 3.70 3.20 3.20 3.60 3.45 3.10 3.10 3.30 3.45 4.10 3.10 3.20 2.75 3.00 2.65 2.95 2.75 3.10 2.95 3.50
2.00 1.90 3.45 2.85 2.85 2.85 3.35 2.60 2.65 3.35 2.50 3.25 2.65 3.35 3.35 3.00 3.00 2.25 3.10 3.45 2.85 3.10 2.90 2.55 2.50
3.95 3.30 3.30 3.00 3.10 3.00 3.15 3.00 3.30 2.50 3.00 3.00 3.00 3.30 3.30 3.00 3.00 3.55 1.65 3.55 2.70 3.00 2.30 3.50 3.70
3.95 3.95 3.60 4.35 3.25 4.05 4.05 3.85 4.80 4.25 3.80 3.00 3.50 3.75 3.25 3.50 3.70 3.50 4.00 3.55 3.30 2.15 4.20 2.00 3.50
3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.40 3.00 3.40 3.20 3.00
15 16 7 11 4 6 5 17 2 8 21 22 19 3 1 10 12 23 25 13 18 24 14 20 9
Table 12 Second set of simulation: projectsÕ criteria and assumed ranking Alternatives A B C D E F G H I J K L M N
g1 0.20 0.50 0.60 100 110 130 150 170 190 210 240 260 280 310
g2 1264 1420 3543 1928 1753 2345 2110 3712 1822 3654 1919 2213 2070 1055
larger values of s and d could very well improve the results. For that reason, we included the s and d values corresponding to their maximums with F 0. Indeed we observe that larger values of d
g3 )155 )322 )566 )488 )611 )477 )945 )623 )702 )90 )803 )847 )402 )965
g4
Ranking
640 512 499 473 465 409 364 322 218 200 190 174 162 143
8 1 2 11 5 13 6 14 9 12 4 10 3 7
and s have generally a positive eect on the KendallÕs s. Moreover, sometimes the highest s value is obtained for one of the maximal values of s or d.
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
It is noted also that applying the model on an orthogonal plan improves generally the quality of the results. This eect is stronger when assuming a simple additive utility function rather than a function with interdependence between criteria. An obvious reason for that phenomenon is that a fractional factorial plan includes only projects which are not correlated and which are characterised by an extreme combination of values for the criteria. Hence average projects with a satisfactory level on all counts are not included in the reference set and cannot reveal by their favourable ranking a phenomenon of interdependence in the utility function.
7. Conclusion This paper has reviewed the main variants of the UTA multicriteria analysis model, and compared systematically their predictive performance on two sets of data. The results of the simulations lead to several conclusions. First of all, when there is no error in the utility function estimation, i.e., F 0, the average utility function computed by the simple UTA model proposed by Jacquet-Lagreze and Siskos (1978) provides a practical and ecient method of estimation. However, the post-optimality method, proposed by Beuthe and Scannella (1996), which aims at accentuating the contrasts between utilities of dierent projects appears also as an ecient method for solving the problem of multiple solutions. Secondly, it appears that the UTA approach of estimating utility function on the basis of stated global preferences between projects provides in practice rather good results, even in cases where there is interdependence between criteria. Thirdly, when errors remain in the utility functionÕs estimation, i.e., F > 0, the UTASTAR model appears more reliable. However, the results obtained in cases of interdependent criteria, and our experience with many simulations indicates that the presence of errors is likely to reveal some `irrational' preference judgements by the decision maker. Whenever possible, it would then be advis-
261
able to verify with the decision maker whether some preference judgements should not be corrected. Fourthly, the reference projects should be chosen carefully in order to lead the decision maker to give as much information as possible on his/her preferences. From this point of view, the use of a set of projects satisfying a fractional factorial plan can be recommended, with the exception maybe of cases where there is suspicion of interdependence between criteria. Finally, the ®xed parameters of the linear program aect also the predictive quality of the models. The simulationsÕ results indicate that small values of d, s and h lead generally to better results in cases of F > 0. But, when F 0, larger values of d and s can provide better results. The use of the UTAMP1 or UTAMP2 procedures may then be used to ®nd the practical upper bounds of the values which can be given to these parameters. Appendix A See Tables 10 and 11.
Appendix B See Table 12.
References Addelman, S., 1962a. Orthogonal main-eect plans for asymmetrical factorial experiments. Technometrics 4 (1), 21±46. Addelman, S., 1962b. Symmetrical and asymmetrical fractional factorial plans. Technometrics 4 (1), 47±58. Beuthe, M., Scannella, G., 1996. Applications comparees des methodes dÕanalyse multicritere UTA. RAIRO Operations Research 30 (3), 293±315. Blauwens, G., Verthongen, R., de Donnea, F.X., Glejser, H., 1982. Poursuite de la construction du reseau routier belge: Analyse co^ uts±bene®ces, Secretariat General du Ministere des Travaux Publics. Charnes, A., Cooper, W.W., 1961. Management Models and Industrial Applications of Linear Programming, Wiley, New York. Charnes, A., Cooper, W.W., 1977. Goal programming and multiple objective optimizations part I. European Journal of Operational Research 1, 39±54.
262
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
Delhaye, O., 1993. La methode multicritere de l'utilite additive: Applications a des projets routiers, memoire d'ingeniorat commercial, Facultes Universitaires Catholiques de Mons, Mons. Despontin, M., Lehert, P., Roubens, M., 1986. Multi-attribute decision making by consumers associations. European Journal of Operational Research 23, 194±201. Despotis, D.K., Yannacopoulos, D., Zopounidis, C., 1990. A review of the UTA multicriteria method and some improvements. Foundations of Computing and Decision Sciences 15, 63±76. Jacquet-Lagreze, E., 1984. PREFCALC: Evaluation et decision multicritere. Revue de l'utilisateur de IBM-PC 3, 38±55. Jacquet-Lagreze, E., Siskos, J., 1978. Une methode de construction de fonctions d'utilite additives explicatives d'une preference globale, Cahier LAMSADE no. 6, Universite Paris-IX-Dauphine. Jacquet-Lagreze, E., Siskos, J., 1982. Assessing a set of additive utility functions for multicriteria decision-making the UTA method. European Journal of Operational Research 10, 151±164.
Keeney, R.L., Raia, H., 1976. Decisions with Multiple Objectives: Preferences and Value Trade-os. Wiley, New York. Kendall, M.G., Stuart, A., 1968. The Advanced Theory of Statistics, vol. III, Second ed., Hafner Publishing Company, New York. Kostkowski, M., Slowinski, R., 1996. `UTA+Application (v 1.20) userÕs manual', Cahier LAMSADE 95. Ministere des Travaux Publics-Administration des Routes, Analyse multicritere appliquee aux investissements routiers d'Etat 1985.. Siskos, J, Yannacopoulos, D., 1985. UTASTAR±An ordinal Regression Method for building additive value functions. Investigacßao Operacional 5/1, 39±53. Srinivasan, V., Shocker, A.D., 1973. Linear programming techniques for multi-dimensional analysis of preferences. Psychometrika 38, 337±369. van Hecke, C., 1993. Analyses multicriteres appliquees aux investissements routiers de l'Etat, memoire d'ingeniorat commercial, Facultes Universitaires Catholiques de Mons, Mons.
www.elsevier.com/locate/dsw
Comparative analysis of UTA multicriteria methods q Michel Beuthe *, Giuseppe Scannella Facult es Universitaires Catholiques de Mons GTM-Groupe Transport et Mobilite 151, Chauss ee de Binche, B-7000 Mons, Belgium Received 17 November 1997; accepted 2 March 1998
Abstract This paper reviews the main variants of the utility additive (UTA) multicriteria method, and systematically compares their predictive performance on two sets of data. It analyses both those cases where the model provides a ranking with errors and those without errors. First, it shows that the reference projects should be chosen carefully in order to elicit as much information as possible from the decision maker: a set of projects satisfying a fractional factorial plan is recommended. Second, it discusses the use of alternative post-optimality methods for solving the problem of multiple solutions and their dierent predictive performances. Third, it presents the results of simulations based on utility functions involving interdependence between criteria, and shows that UTA handles this problem eectively by an adjustment of its coecients. Finally, the in¯uence of the model's parameters on the predictive performance is also investigated. Ó 2001 Elsevier Science B.V. All rights reserved. Keywords: Linear programming; Multicriteria analysis; Utility theory; Optimisation
1. Introduction There are many dierent methods of multicriteria analysis which can be recommended de-
q This paper presents some results of the research program EUNET on ``Socio-Economic and Spatial Impacts of Transport'' commissioned by the European Community within the Fourth Framework Program for Research and Development. Part of the research was also founded by a FSRIU (Fonds Special de la Recherche inter-Universitaire) grant from the French Community of Belgium. * Corresponding author. Tel.: +32-65-32-32-97; fax: 32-65-3156-91. E-mail address: [email protected] (M. Beuthe).
pending upon the circumstances of decision making. For the assessment of investments in public infrastructure, it may be advisable to use a method of total aggregation in order to compute trade-o coecients between criteria. This is actually the basic motivation of this paper which will discuss a subset of total aggregation methods applied to the ranking of road infrastructure projects. Among these, the UTA method initially proposed by Jacquet-Lagreze and Siskos (1978, 1982) has several interesting features: it makes possible the estimation of a nonlinear additive function, which is obtained by the use of a linear program which provides a convenient piecewise linear ap-
0377-2217/01/$ - see front matter Ó 2001 Elsevier Science B.V. All rights reserved. PII: S 0 3 7 7 - 2 2 1 7 ( 0 0 ) 0 0 0 4 2 - 4
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
proximation of the function, and the only information required from the decision maker are global stated preferences between projects. This latter characteristics present some advantages as well as some problems when questioning the decision maker about his preferences, but, as will be shown in this paper, it makes it possible to solve to some extent the problem of interdependence between criteria while maintaining the utility function additivity. A number of variants of UTA have been proposed in the literature over the last 15 years. It appeared useful at this point to take stock of these models, and to compare their predictive performance on a few cases. Moreover, like all models, UTA and its variants have speci®c problems, the main one being the possibility that the linear program could have several solutions. When it comes to selecting one or several projects among the set of projects used for estimating the utility function, this situation is of no consequence, at least if the multiple functions are estimated without errors. But, if there remain errors in the estimation, or if the problem is to apply the utility function to a set of other projects, the use of dierent utility functions will lead to dierent rankings. The ®rst part of this paper presents an overview of the main variants of UTA: the basic model to start with, then its post-optimality analyses, and several dierent speci®cations of its error terms. The second part applies the methods thus reviewed to estimating three dierent utility functions on two sets of projectsÕ data. The ®rst set corresponds to an optimal fractional factorial plan used already in a paper by Beuthe and Scannella (1996). The other one is taken from a paper by Despotis et al. (1990). The cases of estimation without errors and with errors are treated separately. In both cases the relevant modelsÕ predictive performance are compared. The third part examines the case where the assumed utility function of the decision maker is not additive and presents some interdependence between the outcomes. The fourth part brie¯y investigates the in¯uence of the modelsÕ ®xed parameters on the predictive performance. From the results of all these simulations useful conclusions can be drawn for future use of this methodology.
247
2. The UTA model and its variants 2.1. The basic model The problem is to compare, rank and evaluate a set of actions, or projects, with respect to N different criteria which measure the favourable consequences of the projects. The measurements of these consequences are given by the vector g a g1 a; g2 a; . . . ; gN a for any project a belonging to A. As an example, for a highway project, the gi a could be the cost±bene®t ratio, its favourable impact on safety, on the environment, etc. The existence of an additive utility function is assumed: U g a
N X
ui gi a
1
i1
with ui gi P 0
and
dui > 0; dgi
which satis®es the classic axioms of decision theory, namely the axioms of comparability, re¯exivity, transitivity of choices, continuity and strict dominance. The additivity implies in particular that the partial utility of a criterion ui gi a depends only on the level of that particular criterion. For a discussion about the additive utility functions see for instance Keeney and Raia (1976). This function provides an aggregation of the criteria in a common index to compare, rank and assess the projects. UTA estimates the function U on a set of reference projects A0 by approximating the utility of each alternative a 2 A0 by U 0 g a
N X ui gi a r a; i1
with r(a) being a non-negative potential error relative to the utility of each alternative a. This is done by the method of linear goal programming proposed by Charnes and Cooper (1961,1977), which provides an approximation by linear intervals of the nonlinear functions. The objective
248
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
function of UTA, to be minimised, is the sum of these errors: X r a: F
N X ui gi 1 and ui gi 0
5
i1
0
8i; r a P 0 8a 2 A ;
ui gij P 0
8i; 8j:
a2A0
In order to apply that method, the ®eld of variation of each criterion gi ; gi , de®ned by its least favourable value of that criterion gi and its best value gi , is divided into ai equal intervals gij ; gij1 . The variables to be estimated by the program are the partial utilities at these bounds, say ui gij . The utility at intermediate values of the criteria are given by linear interpolation. For each pair of projects (a, b) belonging to A0 , the decision-maker, taking into account the set of criteria, must express his/her overall preferences or indierences. The results of these comparisons are introduced under the form of this constraint N X
fui gi a ÿ ui gi bg r a ÿ r b P d
i1
2
() aPb in the event of strict preferences, and N X
fui gi a ÿ ui gi bg r a ÿ r b 0
i1
3
() a I b; in the event of a strict preference. The constant d on the right-hand side of the inequality (2) must be strictly positive. Its value can very well in¯uence the solution of the program, so that it must not be given too high an initial value. The hypothesis that the partial utilities increase with the value of the criteria imposes a series of additional constraints: ui gij1 ÿ ui gij P si ; j 1; 2; . . . ; ai ; i 1; 2; . . . ; N ;
4
where si must be (strictly) positive. As for d, it is better to give it a small initial value. This question will be dealt with in Section 6. Finally there are normalisation and non-negativity conditions:
At this point we should mention that, in order to improve the accuracy of the estimation, more detailed speci®cations could also be adopted if additional information can be given by the decision maker, such as utilitiesÕ dierences between projects. However, the questionnaire addressed to the decision maker becomes then necessarily longer and more dicult. It may not be practical to go to such lengths, and therefore we will not do so. In UTA, a solution, namely the estimated additive utility function, may be of two types: ®rst, when F 0, any solution belonging to the convex set of solutions admitted by the constraints of the program provides a ranking identical with that given by the decision maker. In the event of positive errors, F > 0, no solution is admitted by the system of constraints, unless some error terms take a positive value. As a matter of fact, in this case, there does not exist a nonlinear additive utility function representing perfectly the preferences expressed by the decision maker. But, in both cases, the solutions may not be the only one as in any linear program and the set of partial utility functions may be very dierent from one solution to another. If we exclude the case of a decision maker who would be unable to reasonably compare the projects or who would be `irrational' in the sense of exhibiting intransitive preferences, the presence of errors may indicate that the decision makerÕs preferences are characterised by partial utilities which are not independent of each other or which are not monotonically increasing, in other words, which do not follow the assumptions made in the model. But, it may also be the case, more simply, that the intervals chosen should have been more numerous or de®ned in a dierent way. We should point out, however, that an additive function of nonlinear partial utility functions is already quite a ¯exible speci®cation; moreover, as will be shown, it can always provide an estimation of utilities which incorporates a certain degree of interdependence among criteria. Actually, as will be
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
shown below, it is our practical experience that it is dicult to simulate an assumed utility function with interdependence such that positive errors are obtained in its estimation. 2.2. Post-optimality analyses Naturally, UTA, like any other multicriteria method must be regarded only as a tool to help the decision maker. After a ®rst run it should always be possible to converse with the decision maker in order to specify or somewhat correct some stated preferences and obtain a better understanding of the estimated function. In the worst case, it may appear that the functionÕs speci®cation is not at all appropriate; in other cases, some adjustments of the program on the basis of additional or corrected information may produce an acceptable utility function. A software like UTA+ (Kostkowski and Slowinski, 1996) includes the possibility of such an interactive process. However, all eorts must be made to ®rst improve the solution in order to re¯ect his ®rst stated preferences as much as possible. For that reason, post-optimality analyses of the results have been proposed, which explore the set of computed optimal solutions, or solutions around that set. Even though the latter solutions correspond to larger F , experience has shown that they may produce a weak order of preferences closer to the stated one (see Despotis et al., 1990). A complete list of the possible post-optimality analyses can be found in Beuthe and Scannella (1996). In this paper, only the methods which could lead to a more limited set of solution, will be examined both cases, i.e., where F 0 and F > 0. The quality of the resulting utility functions will also be assessed in terms of their ranking performance. First of all, Jacquet-Lagreze and Siskos (1978) proposed that, when there is not a single solution, a function should simply be used which would be the average of the extreme optimal functions obtained from a sensitivity analysis applied on the last bounds of each criterion. The sensitivity analysis is made by adding this new constraint to the model
X r a 6 F h;
249
6
a2A0
where h is a small positive number. In the basic model, it was noted that the values given to d and s were to some extent arbitrary. Obviously, their level in¯uences the results as well as the predictive quality of the model, as will be illustrated by simulations in a later section. For that reason, and following a suggestion by Srinivasan and Shocker (1973), it is worthwhile to look for optimal values of d and/or s. First, it is possible to aim at accentuating the dierence between utilities by maximising d, the minimum dierence between the utility of two dierent actions. This model, we named UTAMP1 in order to point out that, on the basis of UTA, it maximises d to better identify the relations of preference between projects. The objective of this second-step program is then to maximise d, subject to constraints (2)±(6). For the case where F 0, this post-optimality analysis is included in the software Jacquet-Lagreze (1984). This case has been discussed and tested by Beuthe and Scannella (1996) with the purpose of reducing the set of solutions. In eect, by choosing the solution(s) which maximise(s) d among all the optimal solutions, the optimal set can only be reduced. A side eect is also to limit the convexity or concavity of the utility function. Beuthe and Scannella (1996) have also proposed to maximise the sum d s in order to stress not only the dierences of utilities between projects but also the dierences between utilities at successive bounds. Note that the simple addition of those two parameters is legitimate since both of them are de®ned in the same utility units. However, a weighted sum could also be considered. For this procedure, which was named UTAMP2, the constraints of the previous program, UTAMP1 remain unchanged but all the si become equal to s P 0. The maximisation of d s also reduces the set of optimal solutions. When F 0, these two post-optimality procedures are justi®ed by the dual relationship between the objective function of the UTA primal program and the objective function of its dual. Indeed, at the optimum
250
F
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
X a
r a d
X j
yj s
X xk z 0; k
where the yj Õs are the dual variables corresponding to the strict preference constraints (2), the xk 's are the dual variables of conditions (4), and z corresponds to the normalisation condition in (5). Given that d and s are strictly positive, all the dual variables must equal zero. Thus, in this case, marginal variations of d and s cannot have the eect of increasing F . However, in cases, where F > 0, some of the dual variables will be positive and variation of d and/or s could increase F , which would result in a ranking of lesser quality. Actually, when there are positive errors, at least one preference constraint will be saturated and at least one dual variable yj will be positive. Hence, the variation of d will automatically aect the level of F . This will be seen in some experiments related in Section 5. Another proposal by Despotis et al. (1990) for the case where F > 0 is to minimise the dierence between the minimum and maximum error of the ®rst estimation. This can be done by MInimising the Maximum individual Error z, z P r a for all a in A0 , in a linear model based on UTA and which we named UTAMIME.
2.3. Other speci®cations of the errors The errorsÕ speci®cation in the basic UTA model could be made more ¯exible. Whenever the value F obtained is zero, another more ¯exible speci®cation could not, obviously, reduce the problem of multiple solutions, but, when F > 0, another speci®cation could give dierent results. Two other speci®cations have been proposed in the literature: ®rst, Siskos and Yannacopoulos (1985) have introduced negative errors rÿ (a) beside the positive ones r (a), in a model named UTASTAR (see also Despotis et al., 1990). The objective function becomes the sum of all these errors, which are introduced in the UTA constraints. For instance, constraint (2) becomes
N X fui gi a ÿ ui gi bg r a ÿ rÿ a i1
ÿ r b rÿ b P d () a P b;
0
2
with r a P 0, rÿ a P 0 8a 2 A0 . Another possibility advanced by JacquetLagreze and Siskos (1982) is to introduce an error term r(ab) proper for each constraint comparing two projectsÕ utilities. This speci®cation is certainly needed when the subjective preferences obtained from pairwise judgements are not transitive. This model will be named UTA2. The objective function to be minimised is again the sum of all these errors. Since it is necessary to introduce a constraint for each comparison between two pairs of projects, m m ÿ 1=2 such constraints are needed if m projects are compared, while there are only (m ) 1) constraints in the basic UTA model. A typical UTA constraint is changed into N X fui gi a ÿ ui gi bg r ab P d () a P b; i1
200 with r ab P 0 8 ab 2 A0 . In the same spirit, Jacquet-Lagreze and Siskos (1982) have proposed another model which minimises the number of violated pairs of R in the weak order R0 given by U g. It is equivalent, in fact, to maximising KendallÕs s. The solution is given by a mixed linear program where the variables c(ab) of the objective function are discrete variables which take the value 1 whenever a preference relationship is violated, and the value 0 when it is satis®ed. The sum of all these errors is minimised under the following constraints: N X fui gi a ÿ ui gi bg M c ab P d i1
() a P b; N P i1 N P i1
2000 ;
9 > > fui gi a ÿ ui gi bg M c ab P 0 > = > > fui gi b ÿ ui gi ag M c ba P 0 > ; () a I b;
3000
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
with c ab 0 or 1 8 ab 2 R, and where M is a large number. Here again, the ®rst two sets of constraints must be applied to all pairs of projects. If the dierence U g a ÿ U g b for a pair a; b 2 R is greater than d, then c ab 0 and the judgement is respected, otherwise c ab 1 and it is violated. We named that model UTAMKEN. This presentation is slightly more general than in the original paper of Jacquet-Lagreze and Siskos (1982), because the indierence judgements are explicitly taken into account in the present paper. Unhappily, even in the best case, ®nding the solution may take a very large number of iterations. This is due to the large number of discrete variables in the model. The constraints (3000 ) include the possibility of indierences, hence errors in indierence constraints are taken into account when minimising the sum of errors. Note that KendallÕs usual index neglects such errors. 3. First set of simulations: Belgian road infrastructure projects 3.1. The basic data These simulations are applied to the case of 353 road projects under consideration for the building of the Belgian network during the period 1985±2010. After a ®rst cost±bene®t study (see Blauwens et al., 1982), the Road Administration, in collaboration with the Centre of Road Research (CRR), realised a multicriteria analysis (Ministere des Travaux Publics, 1985) that used 29 criteria regrouped in six main themes: the safety on the present road, the projectsÕ socioeconomic aspects, the impact on environment, the current and future trac, the problems of planning and urbanism, and ®nally the wear state of the current road. For each of the 29 criteria, the CRR built a discrete scale going from 1 to 5 in order to measure the positive eects of each project. Within each theme, an intermediate weighting was established by a team of experts: in order to aggregate the measurements of the 29 criteria in six indexes corresponding to the six main themes
251
(a discussion of the scales and methodology of the CRR is given by Delhaye (1993) and van Hecke (1993)). Finally, those six indexes were aggregated in a unique measurement of utility by a primary weighting ®xed by the political decision maker. The measurement scales as well as the intermediate weightings of the CRRs experts are taken for granted here, and the six indexes will be used as criteria for the simulations using the variants of the UTA model. However, in place of estimating the utility function on the basis of the CRRs preferences between some real projects, 25 ®ctitious projects will be used, which make up a fractional factorial plan (see Appendix A) according to a method suggested by Addelman (1962a,b). In eect, this matrix gives the maximum information which is possible to be obtained from the smallest number of projects for that number of criteria. Applying this information procedure on the above set of data, we show in the last section of the present paper that the quality of estimation in terms of forecasting power is substantially increased in this way with respect to the case with real projects. 3.2. Setting of the simulations For most simulations, a standard linear programming Simplex algorithm was used. But, UTAMKEN required a mixed linear programming Branch and Bound algorithm. As a software we used the OMP package of Beyers & Partners. The simulations were run on a PC 486 DX-4 100 MHz. Each LP simulation took about 5 seconds. The search of a set of optimal extreme solution in a post-optimality analysis took about 2 N 10 seconds, where N is the number of constraints. A simple FORTRAN program was used to compute the average solutions and the performance indices. Thus, a complete simulation took only a few minutes. Concerning the mixed LP based simulations, the algorithm takes much more time for ®nding the solution, due to the number of binary variables (see Table 1). The number of constraints and variables varies with the model. They are presented in Table 1.
252
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
Table 1 Dimensions of the UTA based modelsa Model UTA UTASTAR UTA2 UTAMKEN UTAMP1 UTAMP2 UTAMIME UTASTARMIME UTA2MIME
Number of constraints P Ac m P NI1 ai 1 c b m nI1 ai1 P C c m m ÿ 1=2 1 nI1 ai1 PN c D jP j 2 jIj 1 I1 ai 1 fAc ; Bc or C c g 1 fAc ; Bc or C c g 1 Ac 1 m Bc 1 2 m C c 1 jP j 2 jIj
Number of variables P Av m NI1 P ai 1 v B 2 m nI1 aP i 1 C c jP j 2 jIj NI1 ai 1 P Dv jP j 2 jIj NI1 ai 1 fAv ; Bv or C v g 1 fAv ; Bv or C v g 2 Av 1 Bv 1 Cv 1
a
The interpolation constraints are not included. jP j: number of strict preference constraints considering all the possible combinations. jIj: number of strict indierence constraints considering all the possible combinations. In UTAMKEN model, the jP j 2 jIj variables are binary variables.
3.3. Case where F 0 For this ®rst simulation, it is assumed that the decision maker's utility is additive and a linear combination of six concave partial utility functions. His/her preferences based on this function are introduced in constraints (2) and (3) of the UTA program. The solution obtained from the program and the corresponding ranking of the projects in A are then compared to the assumed utility function and the ranking it gives to all the projects. In this way, we can assess the UTAs performance in representing the decision makerÕs preferences. The assumed partial utility functions are illustrated in Fig. 1. Since in this case F is found equal to zero, there is no point to introduce another speci®cation of errors. Thus, only the post-optimality proce-
dures UTAMP1 and UTAMP2 are applied. Among the three methods, only UTAMP2 provides a unique solution. The average partial utility functions provided by UTA and UTAMP1 were also computed. The average functions of UTA and the functions of UTAMP2 are illustrated, respectively in Figs. 2 and 3. In both cases, we can observe that some of the functions are loosing their concavity. The presence of multiple solutions is detected by a sensitivity analysis on the utilities of the last bounds. Its results appear in Table 2, where the minimum and maximum values taken by the utilities are given. The gaps between the estimated functions and the assumed theoretical functions are also of interest and appear in Table 3. They appear as the average of the dierences between the estimated partial utilities at the ®ve bounds
Fig. 1. Concave assumed partial utility functions.
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
253
Fig. 2. Partial utility functions of the average solution UTA.
Fig. 3. Partial utility functions by UTAMP2.
and the corresponding theoretical utilities. The global average is the average of these average gaps over the six criteria. Another way to compare the three models is to analyse their performance in ranking all the projects in A. A useful performance index is the KendallÕs index s(m) (Kendall and Stuart, 1968)
which measures in this case the correlation (like all correlation index, s varies between )1 and 1) between the rankings produced by an estimated function and by the assumed theoretical function. Another useful index, called intersection density (see Despontin et al., 1986), d(m), consists in measuring the percentage of common elements in
Table 2 Sensitivity analysis (UTA±UTAMP1) Number of the criterion
Theoretical weights
UTA method: variation intervals
UTAMP1 method: variation intervals
Minimum
Maximum
Minimum
Maximum
1 2 3 4 5 6
0.25 0.20 0.15 0.20 0.10 0.10
0.231 0.106 0.124 0.192 0.057 0.051
0.314 0.224 0.172 0.239 0.131 0.133
0.269 0.195 0.142 0.215 0.071 0.069
0.295 0.196 0.142 0.229 0.085 0.094
0.0092 0.0100 0.0062 0.0052 0.0053 0.0134 0.0082 0.2588 0.1955 0.1495 0.1997 0.1036 0.0926 0.0150 0.0157 0.0057 0.0111 0.0211 0.0272 0.0160 0.25 0.20 0.15 0.20 0.10 0.10 1 2 3 4 5 6
0.2644 0.1695 0.1500 0.2135 0.1007 0.1017
0.0132 0.0275 0.0032 0.0053 0.0110 0.0188 0.0131
0.2881 0.1953 0.1418 0.2183 0.0812 0.0750
Solution Average solution
Average gap with respect to the theoretical functions
UTAMP1 method
Average solution
Average gap with respect to the theoretical functions UTA method
Theoretical weights Number of the criterion
Table 3 Solutions and average gaps (UTA, average solution-UTAMP1, average solution-UTAMP2)
Average gap with respect to the theoretical functions
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
UTAMP2 method
254
the reference core of two rankings, for instance among the 50 projects ranked ®rst in the compared rankings. Compared to the ÔaverageÕ UTA, the results provided by UTAMP1 show important shortcomings. First, if the variation intervals at the last bounds (see Table 2) are smaller for UTAMP1, they never contain the corresponding weight of the assumed function; second, the ÔgapsÕ of the average UTAMP1 function with respect to the assumed function are often larger than those of the average UTA function (see Table 3). Finally, if these two average functions are applied to all projects in A, the weakness of UTAMP1 is revealed by a performance of lower quality (see Table 4). However, when compared to the results of UTAMP2, the solution provided by average UTA appears inferior on all counts.
3.4. Case where F > 0 On the basis of the same data, a subjective ranking of the 25 projects belonging to the reference set A0 is assumed such that the rankings obtained from estimated utility function are not consistent with the assumed subjective ranking. In that case, the sum of errors F > 0. Three of the models explained above can be applied, namely UTA, UTASTAR and UTA2. Because of the presence of multiple solutions for each of them, only their average solution will be considered. But, the post-optimality procedures corresponding to UTAMP2 and UTAMIME will be applied to each of them. The corresponding solution will be named on the one hand UTAMIME, UTASTARMIME and UTA2MIME, respectively, and on the other UTAMP2, UTAMP2STAR and UTAMP22. The UTAMP1 post-optimality method is not taken into account here, because it is useless. Indeed, in the case of F > 0 at least one preference constraint of type (2) is satis®ed as an equality with an error equal to d. Thus, at that level of F it will be impossible with the UTAMP1 model to ®nd a d larger than the starting d. Table 5 below describes the performance of all these models.
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
255
Table 4 Statistics concerning the models UTA, UTAMP1 and UTAMP2 UTA (average solution) UTAMP1 (average solution) UTAMP2
d
s
r a
s 353
d 50
Fixed at 0.01 0.01591 0.00884
Fixed at 0.01 Fixed at 0.01 0.02122
0 ± ±
0.894 0.874 0.930
0.860 0.900 0.920
Table 5 Statistics concerning the average solutions of UTA, UTASTAR and UTA2 and their related post-optimality analyses P d s rmax a r a s 25 d 10 UTA UTASTAR UTA2 UTAMIME UTASTARMIME UTA2MIME UTAMP2 UTAMP2STAR UTAMP22
Fixed at Fixed at Fixed at Fixed at Fixed at Fixed at 0 0.00035 0.00088
0.001 0.001 0.001 0.001 0.001 0.001
Fixed at Fixed at Fixed at Fixed at Fixed at Fixed at 0.0012 0.00099 0.00012
0 0 0 0 0 0
First of all, it should be pointed out that the total errors of UTA2 cannot be compared to the other two modelsÕ sum of errors because they do not have the same meaning. We observe that the results produced by the UTA and UTAMIME are slightly superior to those of the other models. We should note also that, in this case, the post-optimality MIME procedure does not improve the ®nal results. On the other hand, the results produced by UTAMP2 are not good at all. This can be explained by the dual relationship between the UTAs primal problem and its dual. When F > 0, some of the dual variables must be positive, so that, when d s is maximised under the constraint of a ®xed sum of errors, d or s will tend to zero depending on the values taken by the dual variables associated with them. This is not likely to induce an improved solution, since it means that the estimated dierences of utility will tend to vanish. In this particular case, note that d 0 in the basic UTAMP2 procedure. Even though, it appears to work better when applied to UTASTAR and UTA2, we can conclude that this post-optimality procedure can only be recommended for the cases where F 0:
± ± ± 0.0120 0.0055 0.0042 ± ± ±
0.018 0.018 0.051 ± ± ± ± ± ±
0.913 0.904 0.905 0.913 0.857 0.902 0.738 0.890 0.898
1 1 1 1 1 1 0.80 1 1
4. Second set of simulations with F > 0 This is a case taken from a paper by Despotis et al. (1990). The set A0 consists of 14 alternatives, de®ned by four hypothetical quantitative criteria which can take four dierent values. Their values as well as the ranking given by the `decision maker' can be found in Appendix B. Here also, the three dierent models, UTA, UTASTAR and UTA2 have been applied and compared to each other (see Table 6). But for this smaller problems, the solution of UTAMKEN model was only found after 20 hours of iterations! As explained above, this model implies a very large number of constraints and binary variables, which lead to an extremely long computation. UTAMP2 post-optimality analysis is not presented here since F > 0 (see above). All these models give here a unique solution. Hence, a post-optimality analysis is useful only for exploring the set of solutions around the optimal solution. The model with the best predictive performance is UTAMKEN. This is only natural since it aims at maximising s. UTASTAR and its post-optimality analysis UTASTARMIME provide the second best results. Again, the MIME post-optimality analyses presented in the second
256
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
Table 6 Statistics concerning the (unique) solutions of UTA, UTASTAR and their post-optimality analyses MIME and UTAMKENa UTA UTASTAR UTA2 UTAMKEN UTAMIME UTASTARMIME UTA2MIME a
d ®xed at
s ®xed at
rmax a
Rr a
s 14
d 7
0.001 0.001 0.001 0.001 0.001 0.001 0.001
0 0 0 0 0 0 0
± ± ± ± 0.024 0.023 0.024
0.051 0.036 0.14 1 ± ± ±
0.472 0.670 0.769 0.780 0.450 0.670 0.549
0.714 0.857 0.430 0.857 0.571 0.857 0.430
Application from Despotis et al. (1990).
part of the table, does not improve the predictive performance of the models. Actually, given a level of F , this post-optimality analysis minimises the maximum error, but may increase at the same time the level of some other errors, and, as a consequence, may lower the predictive quality of the estimated utility function.
5. Utility aected by interdependence between criteria In the above simulations, the decision makerÕs assumed utility function was simply additive. But what would happen if the decision makerÕs subjective preferences were aected by some interdependence between criteria? In order to examine this problem, a set of estimations of UTA additive utilities was computed on the basis of stated preferences between projects resulting from utility functions characterised by some interdependence between outcomes. Thus, we assumed that the decision makerÕs utility for a project A was u A
6 6 Y X ui gi A a gi A i1
i1
b g 1 A g3 A c g1 A g 3 A g4 A d g1 A g6 A; where gi (A) is the outcome of A on the ith impact, ui gi A is the partial utility of the outcome of A on the ith impact and a±d are the coecients of the interdependence terms. The simulations are made on dierent combinations of these terms. The assumed partial utility functions of the decision
maker are the ones of Fig. 1. Table 7 summarises the results. The ®rst set of simulations is based on the same ®ctitious set of projects which satis®es a fractional factorial design. The second set is based on the same real projects A as above. In the second column, the combination of the interdependence terms is indicated by their coecients. The weights given to these terms appear in the third column. Let us note, to begin with that multiple solutions have been obtained in all these cases, so that the KendallÕs index has been computed with reference to an average solution. It is seen that, in all cases, the UTA estimation is made without error, while, in most cases, the KendallÕs s(353) remains at a relatively high level. Thus, the ranking is not much aected by the presence of interdependence in the decision makerÕs preferences. Actually, the estimated coecients are biased in order to compensate the presence of interdependence terms which are neglected by the UTA speci®cation. Some other forms of utility functions with interdependence were also considered, but we found out that it was very dicult indeed to generate cases with errors. In the few cases where some errors were generated, the utility functions of the decision maker had actually a sinusoidal shape with local saturation. Such a shape can hardly be accepted as anything normal and need not be considered in practice. Ex post facto, we realised that a similar aberration should explain the preference orders of the two cases with errors which were discussed above. Thus, even if there is interdependence in the decision makerÕs utility function, the UTA model
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
257
Table 7 Simulations cases with interdependence (d 0:001 and s 0) Reference projects
Interdependence terms
Weight factor of interdependence terms
Resulting value of F by UTA
KendallÕs s 353
Fictitious
± a, b, c, d a, b, c a, b, d a, b a, d c, d a b c d
± 0.01, 2, 4, 6 0.01, 1, 8 0.005, 1, 1 0.01, 1 0.003, 0.7 4, 5 0.003 7 5 7
0 0 0 0 0 0 0 0 0 0 0
0.874 0.853 0.821 0.890 0.886 0.865 0.820 0.840 0.735 0.874 0.811
Real
± a, b, c, d a, b, c a, b, d a, b a, d c, d a b c d
± 0.01, 2, 4, 6 0.01, 1, 8 0.005, 1, 1 0.01, 1 0.003, 0.7 4, 5 0.003 10 5 5
0 0 0 0 0 0 0 0 0 0 0
0.709 0.755 0.812 0.540 0.732 0.840 0.737 0.672 0.638 0.793 0.665
Table 8 Results of the simulation cases for the eect of d, s and h in case of F > 0 d ®xed at
s ®xed at
h ®xed at
Rr a
s 25
UTA 0.001 0.001 0.001 0.002 0.005 0.010 0.001 0.001 0.001 0.001
0 0.0005 0.001 0 0 0 0 0 0 0
0.00001 0.00001 0.00001 0.00001 0.00001 0.00001 0.00005 0.00010 0.00050 0.00095
0.018 0.025 0.032 0.036 0.090 0.180 ± ± ± ±
0.913 0.878 0.872 0.913 0.913 0.910 0.899 0.900 0.902 0.904
UTASTAR 0.001 0.001 0.001 0.002 0.005 0.010 0.001 0.001 0.001 0.001
0 0.0005 0.001 0 0 0 0 0 0 0
0.00001 0.00001 0.00001 0.00001 0.00001 0.00001 0.00005 0.00010 0.00050 0.00095
0.018 0.022 0.029 0.036 0.090 0.180 ± ± ± ±
0.904 0.858 0.832 0.897 0.896 0.892 0.904 0.904 0.905 0.900
258
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
provides in practice good results in terms of F and s.
6. Parameters in¯uence on predictive quality As we have noted, the choice of parameters d, s and h may in¯uence the solution of the linear program and, therefore, the ranking of projects. By means of simulations, their impacts on the (average) solution of dierent models are examined on the basis of the KendallÕs index. A number of cases is considered: when F 0 and F > 0, when there is interdependence in the utility function and when there is not, also when the reference set satis®es a fractional factorial plan and when the set is made up of real projects.
6.1. Case where F > 0 This is certainly not the most interesting case, if we keep in mind the results obtained in Section 5. Indeed, when there are positive errors, whenever possible, a dialogue should take place between the analyst and the decision maker in order to let her/him adjust her/his order of preferences and obtain a function estimated without errors. But it is relevant to pursue the analysis for the cases where such a dialogue is not possible. Here, the reference set of projects satis®es the same fractional factorial plan, and the stated preferences are those of Section 3.3. The sensitivity analysis is applied on both UTA and UTASTAR models. The KendallÕs index s is computed for the reference set A0 of projects jAj 25. The results are given in Table 8.
Table 9 Results of the simulation cases for the eect of d and s in case of F 0 UTA without interdependence
a
UTA with interdependence
d ®xed at
s ®xed at
s (353)
d ®xed at
s ®xed at
s 353
Orthogonal plan 0.001 0.001 0.001 0.001 0.001 0.001 0.002 0.005 0.010 0.015 0.0197b 0.0088c
0 0.0005 0.0010 0.0075 0.0150 0.0248a 0 0 0 0 0 0.021c
0.874 0.872 0.871 0.876 0.891 0.901 0.884 0.905 0.897 0.847 0.810 0.930
0.001 0.001 0.001 0.001 0.001
0 0.0005 0.0010 0.0075 0.0091a
0.820 0.821 0.822 0.810 0.818
0.002 0.005 0.0076b 0c
0 0 0 0.0103c
0.825 0.833 0.838 0.808
Set of real projects 0.001 0.001 0.001 0.001 0.001 0.002 0.0031b 0c
0 0.0005 0.0010 0.0075 0.0333a 0 0 0.0036c
0.709 0.776 0.717 0.774 0.877 0.818 0.771 0.858
0.001 0.001 0.001 0.001 0.001 0.0020 0.0036b 0.0028c
0 0.0005 0.0010 0.0075 0.0100a 0 0 0.008c
0.742 0.805 0.766 0.850 0.835 0.839 0.815 0.837
Maximal value of s for a given value of d. Maximal value of d for a given value of s (UTAMP1 procedure). c Maximal value of (d s) (UTAMP2 procedure). b
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
It is seen, ®rst of all, that, in most cases, the simple UTA average solution produces the most faithful ranking of the projects. Secondly, the dierent combinations of d and s provide dierent levels of F . This phenomenon is explained, as above in similar cases, by the primal-dual relationship in linear programming. Thirdly, higher levels of d and s have generally a rather negative in¯uence on the s statistics. It is particularly the case for s variations, while the eect of d is very weak in this case. This suggests that smaller values of these parameters will lead to better results. SRINIVASAN and SHOCKER recommend a d equal to 1= 100 Q, where Q is the number of indierence classes in the reference set. In the present case, d should then be equal to 0.0005. Our ®ndings tend to support this recommendation. Finally, the value of h, which is used in the dierent post-optimality analyses, appears to have a rather weak eect on the predictive quality.
259
However, on the basis of all our simulations, we would recommend the use of a very small value. 6.2. Case where F 0 Here, four dierent cases are examined: · the case where the decision makerÕs utility function is assumed additive and the case where there is interdependence between the criteria; · the case where the reference set satis®es a fractional factorial plan and when the set is made up of 25 real projects. The sensitivity analysis on d and s (h being ®xed at zero) is applied only to the UTA model, and the results are assessed by the s statistics over the full sample of 353 projects (see Table 9). With F 0, we know from the primal-dual relationship that small increases of d or s should have no impact on the level of errors. Hence, a tighter adjustment of the utility function with
Table 10 Fractional factorial plan: projectsÕ criteria and assumed rankings Projects 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
g1 1 1 1 1 1 2 2 2 2 2 3 3 3 3 3 4 4 4 4 4 5 5 5 5 5
g2 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
g3 1 2 3 4 5 2 3 4 5 1 3 4 5 1 2 4 5 1 2 3 5 1 2 3 4
g4 1 3 5 2 4 2 4 1 3 5 3 5 2 4 1 4 1 3 5 2 5 2 4 1 3
g5 1 4 2 5 3 2 5 3 1 4 3 1 4 2 5 4 2 5 3 1 5 3 1 4 2
g6 1 5 4 3 2 2 1 5 4 3 3 2 1 5 4 4 3 2 1 5 5 4 3 2 1
Ranking (F 0)
(F > 0)
25 23 19 16 14 24 17 20 6 10 20 13 12 8 14 7 22 9 3 3 1 18 5 10 2
25 23 4 18 8 24 20 18 6 16 16 13 6 15 11 22 21 13 11 10 1 2 5 3 8
260
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
Table 11 Multicriteria evaluation of 25 real projects belonging to the Belgian Road Infrastructure projects set: projectsÕ criteria and assumed ranking Projects
g1
g2
g3
g4
g5
g6
Ranking
327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351
4.00 3.60 3.35 3.00 4.00 3.00 3.00 3.00 3.00 3.30 3.00 2.75 3.00 3.00 3.00 3.60 3.20 2.90 3.15 3.30 3.55 3.00 3.65 3.85 3.25
2.60 3.30 2.95 3.30 3.25 3.70 3.20 3.20 3.60 3.45 3.10 3.10 3.30 3.45 4.10 3.10 3.20 2.75 3.00 2.65 2.95 2.75 3.10 2.95 3.50
2.00 1.90 3.45 2.85 2.85 2.85 3.35 2.60 2.65 3.35 2.50 3.25 2.65 3.35 3.35 3.00 3.00 2.25 3.10 3.45 2.85 3.10 2.90 2.55 2.50
3.95 3.30 3.30 3.00 3.10 3.00 3.15 3.00 3.30 2.50 3.00 3.00 3.00 3.30 3.30 3.00 3.00 3.55 1.65 3.55 2.70 3.00 2.30 3.50 3.70
3.95 3.95 3.60 4.35 3.25 4.05 4.05 3.85 4.80 4.25 3.80 3.00 3.50 3.75 3.25 3.50 3.70 3.50 4.00 3.55 3.30 2.15 4.20 2.00 3.50
3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.40 3.00 3.40 3.20 3.00
15 16 7 11 4 6 5 17 2 8 21 22 19 3 1 10 12 23 25 13 18 24 14 20 9
Table 12 Second set of simulation: projectsÕ criteria and assumed ranking Alternatives A B C D E F G H I J K L M N
g1 0.20 0.50 0.60 100 110 130 150 170 190 210 240 260 280 310
g2 1264 1420 3543 1928 1753 2345 2110 3712 1822 3654 1919 2213 2070 1055
larger values of s and d could very well improve the results. For that reason, we included the s and d values corresponding to their maximums with F 0. Indeed we observe that larger values of d
g3 )155 )322 )566 )488 )611 )477 )945 )623 )702 )90 )803 )847 )402 )965
g4
Ranking
640 512 499 473 465 409 364 322 218 200 190 174 162 143
8 1 2 11 5 13 6 14 9 12 4 10 3 7
and s have generally a positive eect on the KendallÕs s. Moreover, sometimes the highest s value is obtained for one of the maximal values of s or d.
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
It is noted also that applying the model on an orthogonal plan improves generally the quality of the results. This eect is stronger when assuming a simple additive utility function rather than a function with interdependence between criteria. An obvious reason for that phenomenon is that a fractional factorial plan includes only projects which are not correlated and which are characterised by an extreme combination of values for the criteria. Hence average projects with a satisfactory level on all counts are not included in the reference set and cannot reveal by their favourable ranking a phenomenon of interdependence in the utility function.
7. Conclusion This paper has reviewed the main variants of the UTA multicriteria analysis model, and compared systematically their predictive performance on two sets of data. The results of the simulations lead to several conclusions. First of all, when there is no error in the utility function estimation, i.e., F 0, the average utility function computed by the simple UTA model proposed by Jacquet-Lagreze and Siskos (1978) provides a practical and ecient method of estimation. However, the post-optimality method, proposed by Beuthe and Scannella (1996), which aims at accentuating the contrasts between utilities of dierent projects appears also as an ecient method for solving the problem of multiple solutions. Secondly, it appears that the UTA approach of estimating utility function on the basis of stated global preferences between projects provides in practice rather good results, even in cases where there is interdependence between criteria. Thirdly, when errors remain in the utility functionÕs estimation, i.e., F > 0, the UTASTAR model appears more reliable. However, the results obtained in cases of interdependent criteria, and our experience with many simulations indicates that the presence of errors is likely to reveal some `irrational' preference judgements by the decision maker. Whenever possible, it would then be advis-
261
able to verify with the decision maker whether some preference judgements should not be corrected. Fourthly, the reference projects should be chosen carefully in order to lead the decision maker to give as much information as possible on his/her preferences. From this point of view, the use of a set of projects satisfying a fractional factorial plan can be recommended, with the exception maybe of cases where there is suspicion of interdependence between criteria. Finally, the ®xed parameters of the linear program aect also the predictive quality of the models. The simulationsÕ results indicate that small values of d, s and h lead generally to better results in cases of F > 0. But, when F 0, larger values of d and s can provide better results. The use of the UTAMP1 or UTAMP2 procedures may then be used to ®nd the practical upper bounds of the values which can be given to these parameters. Appendix A See Tables 10 and 11.
Appendix B See Table 12.
References Addelman, S., 1962a. Orthogonal main-eect plans for asymmetrical factorial experiments. Technometrics 4 (1), 21±46. Addelman, S., 1962b. Symmetrical and asymmetrical fractional factorial plans. Technometrics 4 (1), 47±58. Beuthe, M., Scannella, G., 1996. Applications comparees des methodes dÕanalyse multicritere UTA. RAIRO Operations Research 30 (3), 293±315. Blauwens, G., Verthongen, R., de Donnea, F.X., Glejser, H., 1982. Poursuite de la construction du reseau routier belge: Analyse co^ uts±bene®ces, Secretariat General du Ministere des Travaux Publics. Charnes, A., Cooper, W.W., 1961. Management Models and Industrial Applications of Linear Programming, Wiley, New York. Charnes, A., Cooper, W.W., 1977. Goal programming and multiple objective optimizations part I. European Journal of Operational Research 1, 39±54.
262
M. Beuthe, G. Scannella / European Journal of Operational Research 130 (2001) 246±262
Delhaye, O., 1993. La methode multicritere de l'utilite additive: Applications a des projets routiers, memoire d'ingeniorat commercial, Facultes Universitaires Catholiques de Mons, Mons. Despontin, M., Lehert, P., Roubens, M., 1986. Multi-attribute decision making by consumers associations. European Journal of Operational Research 23, 194±201. Despotis, D.K., Yannacopoulos, D., Zopounidis, C., 1990. A review of the UTA multicriteria method and some improvements. Foundations of Computing and Decision Sciences 15, 63±76. Jacquet-Lagreze, E., 1984. PREFCALC: Evaluation et decision multicritere. Revue de l'utilisateur de IBM-PC 3, 38±55. Jacquet-Lagreze, E., Siskos, J., 1978. Une methode de construction de fonctions d'utilite additives explicatives d'une preference globale, Cahier LAMSADE no. 6, Universite Paris-IX-Dauphine. Jacquet-Lagreze, E., Siskos, J., 1982. Assessing a set of additive utility functions for multicriteria decision-making the UTA method. European Journal of Operational Research 10, 151±164.
Keeney, R.L., Raia, H., 1976. Decisions with Multiple Objectives: Preferences and Value Trade-os. Wiley, New York. Kendall, M.G., Stuart, A., 1968. The Advanced Theory of Statistics, vol. III, Second ed., Hafner Publishing Company, New York. Kostkowski, M., Slowinski, R., 1996. `UTA+Application (v 1.20) userÕs manual', Cahier LAMSADE 95. Ministere des Travaux Publics-Administration des Routes, Analyse multicritere appliquee aux investissements routiers d'Etat 1985.. Siskos, J, Yannacopoulos, D., 1985. UTASTAR±An ordinal Regression Method for building additive value functions. Investigacßao Operacional 5/1, 39±53. Srinivasan, V., Shocker, A.D., 1973. Linear programming techniques for multi-dimensional analysis of preferences. Psychometrika 38, 337±369. van Hecke, C., 1993. Analyses multicriteres appliquees aux investissements routiers de l'Etat, memoire d'ingeniorat commercial, Facultes Universitaires Catholiques de Mons, Mons.