ELECTRE-III-H: An outranking-based decision aiding method for hierarchically structured criteria

Expert Systems with Applications 42 (2015) 4910–4926

Contents lists available at ScienceDirect

Expert Systems with Applications journal homepage: www.elsevier.com/locate/eswa

ELECTRE-III-H: An outranking-based decision aiding method for hierarchically structured criteria Luis Del Vasto-Terrientes a,⇑, Aida Valls a, Roman Slowinski b,c, Piotr Zielniewicz b a

Departament d’Enginyeria Informàtica i Matemàtiques, Universitat Rovira i Virgili, Av Països Catalans, 26, 43007 Tarragona, Catalonia, Spain Institute of Computing Science, Poznan University of Technology, Piotrowo 2, 60-965 Poznan, Poland c Systems Research Institute, Polish Academy of Sciences, 01-447 Warsaw, Poland b

a r t i c l e

i n f o

Article history: Available online 23 February 2015 Keywords: Multi-criteria decision aiding Hierarchy of criteria Partial pre-order aggregation ELECTRE methods

a b s t r a c t This paper proposes a method for ranking a set of alternatives evaluated using multiple and conflicting criteria that are organised in a hierarchical structure. The hierarchy permits the decision maker to identify different intermediate sub-problems of interest. In that way, the analysis of the criteria is done according to the subsets defined in the hierarchy, and following the precedence relations in a bottomup approach. To deal with this type of hierarchical structures, an extension of the ELECTRE-III method, called ELECTRE-III-H, is presented. As all methods of ELECTRE family, this one also relies on building a binary outranking relation on the set of alternatives on the basis of concordance and discordance tests. The exploitation of this outranking relation generates a partial pre-order, establishing an indifference, preference or incomparability relation for each pair of alternatives. The idea of a bottom-up application of the classical ELECTRE-III method to sub-problems involving subsets of criteria at the intermediate levels of the hierarchy is infeasible because the evaluations of alternatives by criteria aggregating some sub-criteria have the form of partial pre-orders, and not complete pre-orders. Thus, we propose a new procedure for building outranking relations from a set of partial pre-orders, as well as a mechanism for propagating these pre-orders upwards in the hierarchy. With this method, the decision maker is able to analyse the problem in a decomposed way and gain information from the outputs obtained at intermediate levels. In addition, ELECTRE-III-H gives the decision maker the possibility to define a local preference model at each node of the hierarchy, according to his objectives and sub-problem characteristics. We show an application of this method to rank websites of tourist destination brands evaluated using a hierarchy with 4 levels. Ó 2015 Elsevier Ltd. All rights reserved.

1. Introduction In ranking problems, the decision maker (DM) wants to find an order structure on a set of alternatives taking into account his/her preference of one alternative over the others. This task is not straightforward when multiple conflicting criteria must be considered. The order structure of the alternatives depends on how well they perform on particular criteria and how important each criterion is to the DM. Thus, the performances of the alternatives on the whole set of criteria must be aggregated, taking into account the preferences of the DM expressed by the relative importance of each criterion (Rudas, Pap, & Fodor, 2013). To aggregate the performances and ⇑ Corresponding author. E-mail addresses: [email protected] (L. Del Vasto-Terrientes), aida. [email protected] (A. Valls), [email protected] (R. Slowinski), piotr. [email protected] (P. Zielniewicz). http://dx.doi.org/10.1016/j.eswa.2015.02.016 0957-4174/Ó 2015 Elsevier Ltd. All rights reserved.

formulate a recommendation, many decision aiding methods have been proposed, each with its own informational requirements and mathematical properties (Roy & Slowinski, 2013; Torra & Narukawa, 2007). This work focuses on solving the multiple criteria ranking problem with outranking methods (Figueira, Greco, & Ehrgott, 2005). In particular, we study the case of criteria that are organised in a hierarchy with different levels of generality. The utility-based and outranking approaches prevail nowadays in the Multiple Criteria Decision Aiding (MCDA), a discipline deriving from the field of Operational Research. Outranking methods have been very successful because they are easy to understand by DMs and they are based on realistic assumptions. The aim of the outranking method is to build a binary relation S, where aSb means ‘‘a is at least as good as b’’. It was proposed by Roy (1996) to establish a realistic representation of four basic situations of preference: indifference, weak preference, strict preference, and incomparability. The assertion aSb is considered to be true if there are sufficient arguments to affirm that a is not worse than b, and if

L. Del Vasto-Terrientes et al. / Expert Systems with Applications 42 (2015) 4910–4926

there is no essential reason to refuse this assertion. These concepts are formalised in the definition of concordance and discordance indices with respect to aSb. A well-known family of outranking methods is called ELECTRE (Figueira, Greco, Roy, & Slowinski, 2013). Since the proposal of the basic ELECTRE-I method, several subsequent versions have been developed for specific decision problems, e.g., ELECTRE-Is for the selection of the best alternatives, ELECTRE-II and ELECTRE-III/IV for constructing a ranking, and ELECTRE-TRI for sorting problems. ELECTRE methods have been widely acknowledged as effective and efficient decision aiding tools, with successful applications in different domains (Abedi, Torabi, Norouzi, & Hamzeh, 2012; Arondel & Girardin, 2000; Botti & Peypoch, 2013; Colson, 2000; Damaskos & Kalfakakou, 2005; Papadopoulos & Karagiannidis, 2008; Shanian, Milani, Carson, & Abeyaratne, 2008; Xu & Ouenniche, 2012). ELECTRE methods have strengths and weaknesses (Figueira et al., 2013). The strengths include the following:  ELECTRE methods are able to take into account the qualitative nature of some criteria, allowing the DM to consider the original data directly, without the need to make transformations into artificial numerical scales.  ELECTRE methods can deal with heterogeneous criteria scales, preserving the original scores of the alternatives on each criterion, without the need for normalisation techniques or the estimation of a value function. This heterogeneity of scales is usually an inconvenience for many decision support systems, which often require a common measurement scale for all criteria.  ELECTRE acknowledges the non-compensatory character in the aggregation, unlike other utility-based approaches such as AHP and MACBETH. In the outranking approach, if, on a certain criterion, an alternative is strongly opposed to the assertion aSb, this fact is enough to reject the assertion aSb. This characteristic is called ‘‘right to veto’’. The main weaknesses of ELECTRE methods are as follows:  When the aim is to calculate an overall score for each alternative, ELECTRE methods are not suitable and other scoring methods should be applied.  Classical ELECTRE methods assume that all the criteria are at the same level of generality and do not consider the possibility of working with subsets of criteria in a hierarchical structure. This paper addresses the second weakness of the classical ELECTRE method. In some real-world decision problems, criteria are naturally defined in a hierarchical structure with different levels of generality that model the implicit taxonomical relations between the criteria. This hierarchical approach is particularly suitable for complex problems with a large number of criteria. In such cases it may become cognitively difficult for the DM to consider all of the criteria together (Mustajoki, 2012). In many applications, the hierarchical structure has the form of a tree, where the root corresponds to the general goal of the DM, the nodes of the tree descending from the goal are sub-criteria, the nodes descending from these sub-criteria are the lower-level sub-criteria, and so on. Finally, the leaves correspond to the elementary criteria, in which the alternatives are directly evaluated. This process decomposes a complex goal into smaller problems with subsets of criteria, enabling the DM to analyse the alternatives with respect to single subsets of criteria at different levels of generality. The main contribution of this paper is a ranking method for a hierarchical set of criteria that extends the classical ELECTRE-III

4911

method (Figueira et al., 2013). ELECTRE-III follows two basic steps: (1) construction of a binary outranking relation based on partial concordance and discordance indices obtained from the consideration of a given set of criteria; these indices determine the aggregation conditions based on social choice models (majority rule and respect to minorities), and (2) exploitation of the outranking relation via distillation to obtain a ranking of alternatives in the form of a partial pre-order. The method proposed is called ELECTRE-III-H and is designed to generate and propagate the partial pre-orders calculated from the bottom level up to the root of the hierarchy tree. We first propose an iterative procedure that maintains the two steps of the classical ELECTRE method (construction and exploitation of the outranking relation) in all intermediate nodes of the hierarchy up to the root. First, the classical ELECTRE-III is applied at the bottom of the tree, aggregating the most specific elementary criteria to their direct parent and obtaining the first results in the form of partial preorders. These partial pre-orders are interpreted as inputs for the intermediate criteria from the upper level. The partial pre-orders are aggregated with the construction of a new pairwise credibility matrix. Next, the classical exploitation process (known as distillation) is applied to generate a partial pre-order at the parent node. With this approach, the DM obtains a result (i.e., a partial pre-order) at each of the intermediate levels of the tree, in addition to the overall partial pre-order at the root level. The second contribution is the definition of new partial concordance and discordance indices that take into account threshold values on partial pre-orders induced by criteria aggregated at intermediate levels of the hierarchy. The DM can also specify the relative importance of each criterion in the context of the same subset at each level of the hierarchy. The following subsection presents a range of application domains and concrete examples of decision problems involving a hierarchical structure of criteria, in which the DM needs to study the results at different levels of the tree. The rest of the paper is structured as follows: Section 2 reviews the classical ELECTRE-III model; Section 3 defines the concepts and notations with regard to the hierarchy of criteria; Section 4 describes the extension of the ELECTRE-III method, called ELECTRE-III-H; Section 5 presents a real-world application in which the DM considers a hierarchical structure of criteria to construct a preference ranking of tourist destination websites; finally, Section 6 presents the conclusions of the work. 1.1. Related works and applications Decisions based on a hierarchical structure of criteria appear in several fields. For example, environmental resource management discipline commonly involves conflicting interests such as economic, environmental impact and social criteria (Bobylev, 2011; Nordstrom, Eriksson, & Ohman, 2010; Sánchez-Lozano, TeruelSolano, Soto-Elvira, & García-Cascales, 2013; Valls et al., 2010). For example, in Nordstrom et al. (2010), a case study of a planning process for an urban forest in Sweden is addressed. The paper evaluates three alternative strategic forest plans for areas around the urban forest in Lycksele, Sweden. The interests of four social groups are considered (timber producers, environmentalists, recreationists and reindeer herders). For each group, different sub-criteria with differing preferences are taken into account (e.g., timber producers want to maximise the fertilised area, while reindeer herders wish to minimise it). Complex decision models appear also in medicine (Ahsan & Bartlema, 2004; Mendis & Gedeon, 2012; Reddy, Kelly, Thokala, Walters, & Duenas, 2014). In Reddy et al. (2014), it is presented a study for producing national guidance relating to the promotion of good health and the prevention and treatment of disease, at

4912

L. Del Vasto-Terrientes et al. / Expert Systems with Applications 42 (2015) 4910–4926

the Centre for Public Health (CPH) at the United Kingdom’s National Institute for Health and Care Excellence (NICE). The objective is to choose the most appropriate topics for this guidance taking into account a 3-level hierarchy of criteria with 3 main subcriteria: size of the problem, making the difference, and current variation in practice. Another area that is growing in popularity is the construction of rankings based on Quality Assessment, such as institution rankings (Aydin, Kahraman, & Kaya, 2012; Buyukozkan, Çifçi, & Guleryuz, 2011; Hsu & Pan, 2009; Torres-Salinas, Moreno-Torres, DelgadoLópez-Cózar, & Herrera, 2011). Complex sets of diverse criteria are used to build a ranking of alternatives taking into account different topics. For example, in Buyukozkan et al. (2011), a model to evaluate perceived service quality in the healthcare sector and to evaluate the performance of pioneering Turkish hospitals on criteria such as responsiveness, professionalism and empathy is presented, with a 3-level hierarchy of criteria. In Aydin et al. (2012), the European Foundation for Quality Management (EFQM) Excellence Award evaluates organisations on the basis of three main parameters: -leadership, strategy and processes- that are decomposed into a 3-level hierarchy of criteria. In Shen, Hermans, Brijs, and Wets (2012), a road safety performance evaluation for a group of European countries is presented. Several road safety performance criteria including speed, alcohol consumption and protective systems are structured hierarchically, allowing the analysis of each country’s performance for each one of this criterion based on index scores. Business management is based on strategic decisions that include complex criteria and therefore can be modelled in a hierarchical structure (Arbenz, Hummel, & Mainik, 2012; Chang, Kuo, Wu, & Tzeng, 2015; Kilic, Zaim, & Delen, 2015; Muerza, De Arcocha, Larrodé, & Moreno-Jiménez, 2014; Wang, Huang, & Dismukes, 2004; Yang, Chuang, & Huang, 2009). For example, in Wang et al. (2004), a manufacturing chain decision problem is analysed. The overall goal is to achieve optimal supplier efficiency with regard to a hierarchical structure of criteria, from basic indicators to four general measures of efficiency: delivery reliability, flexibility and responsiveness, cost, and assets. In Kilic et al. (2015) the aim is to select the best enterprise resource planning system using a hierarchy of criteria with 3 main branches: business, cost and technical issues. In the MCDA literature, very few methods consider the decomposition of decision problems using a hierarchy of criteria. The best known method for managing hierarchical structures is the Analytic Hierarchical Process (AHP) (Saaty, 1987), which belongs to the utility-based approach. AHP permits the DM to focus on specific sub-criteria to find the weights of each criterion depending on its position on the hierarchy by means of pairwise comparison of criteria having the same parent, which yields the relative trade-off weights. The pairwise comparison of alternatives and criteria is based on the judgment ratio scale from 1 to 9, in which 1 represents ‘‘Equally preferred’’ and 9 represents ‘‘Extremely preferred’’. Once the comparison matrix has been given by the DM, weights or priorities are derived finding the normalized eigenvector of the matrix. This requires the matrix to be consistent (or near consistent) to obtain meaningful priorities. Then, a numerical rating is obtained for each of the decision alternatives by means of an additive aggregation operator. AHP was applied to some of the case studied mentioned above (Ahsan & Bartlema, 2004; Bobylev, 2011; Reddy et al., 2014; Hsu & Pan, 2009; Muerza et al., 2014; Nordstrom et al., 2010). In Buyukozkan et al. (2011), the Analytic Network Process (ANP) method, which is a generalisation of AHP for networks instead of hierarchies, is applied. The difference between AHP and ANP, is such that ANP does not consider the alternatives as independent actions.

Despite the large literature and applications of AHP, the method has also received some critics. The consistency condition is difficult to achieve, several consistency indices have been proposed, as well as methods to obtain a transitive matrix (Bana e Costa & Vansnick, 2008). The additive nature of the aggregation has also been posed into question because it generates rank reversals (Ishizaka & Labib, 2011), but also because it is a compensative trade-off approach, which is not appropriate in some applications. Another weakness is the imprecision and uncertainty of the linguistic scale used for the construction of the pairwise comparison matrices. To overcome this weakness, the Fuzzy-AHP method has been proposed, which applies a range of value to incorporate possible DM’s uncertainty instead of merely crisp ratio values. In Aydin et al. (2012), Fuzzy-AHP is used to achieve a performance assessment of firms for EFQM Excellence Award using fuzzy scales to make pairwise comparisons. Several Fuzzy-AHP applications are presented in Mardani, Jusoh, and Zavadskas (2015) from 1994 to 2014. For ANP, a fuzzy approach has also been introduced (Chang et al., 2015). In other complex problems, AHP and ANP are combined with other methods to treat hierarchical structures of criteria. For example, in Sánchez-Lozano et al. (2013), Kilic et al. (2015), AHP and ANP respectively are applied only to establish the weights of the criteria in the hierarchy. There are some other utility-based approaches where aggregation operators are used to generate ratings of alternatives at different levels of generality. An interesting case is the method called Logic Scoring of Preference (LSP), where the operators are parametrized and can range from full conjunction, partial conjunction, partial disjunction and full disjunction. In addition, mandatory and optional criteria can be defined and treated accordingly in the different levels of the hierarchy. Some applications of this method for decision aiding are Dujmovic´ and Tré (2011), Hatch, Dragic´evic´, and Dujmovic´ (2014), Pijuan, Valls, Passuello, and Schuhmacher (2010). The main limitation is the complexity of the problem modeling using such high level operators on the basis of its logical properties. Moreover, all the values need to be in the same numerical scale, not allowing heterogeneity as in an outranking-based approach. Hierarchies of objectives are also considered in DEA (Data Envelopment Analysis) (Shen et al., 2012) and PGP (Preemptive Goal Programming) (Wang et al., 2004), but these methods concern a continuous space rather than a discrete set, as we study in this paper. In recent years, another methodology, called Multiple Criteria Hierarchy Process (MCHP), has been proposed to deal with hierarchical structures of criteria (Corrente, Greco, & Slowinski, 2012). It can be applied to any MCDA method, including utilitybased and outranking methods. For the outranking methods, this process is explained in Corrente, Greco, and Slowinski (2013b). It builds binary outranking relations at each node of the hierarchy. The selected ELECTRE method is applied first on the lowest level of the hierarchy to build a binary outranking preference relation for each subset of elementary criteria. Then, at upper levels, MCHP continues to construct binary outranking relations which are propagated up to the root. The preference information used to construct the outranking relations can be provided by the DM either directly (in form of outranking model parameters, like criteria weights and comparison thresholds) or indirectly (in form of pairwise comparisons of some alternatives). In the latter case, MCHP is combined with the Robust Ordinal Regression (ROR) (Corrente et al., 2013b). The ROR takes into account all sets of outranking model parameters compatible with the preference information provided by the DM to give a solution in terms of necessary and possible outranking relations, by applying all the compatible preference models on the considered alternatives. The authors present an illustrative example

L. Del Vasto-Terrientes et al. / Expert Systems with Applications 42 (2015) 4910–4926

regarding the evaluation of students who are competing for a scholarship based on Mathematics and Chemistry that decompose to more specific subjects. The approach based on indirect preference information relies on having a suitable set of decision examples, which may sometimes be hard to find when there is no historical data or the user is inexperienced. This is a potential shortcoming in some applications. Based on the advantages of the outranking approach, this paper proposes a novel procedure based on ELECTRE that allows decision analysis at different levels of the hierarchy tree. Given the limitations of an indirect approach, we use direct elicitation of preference information, in which the parameters of the method are given directly by the DM. The main difference between our proposal and the direct method described in Corrente et al. (2013b) for outranking methods is that we intend to apply the ELECTRE procedure (steps 1 and 2) at all levels of the hierarchy. With this approach we have two advantages: first, the DM obtains a result (i.e., a partial pre-order) at each of the intermediate levels of the tree and not only at the root level, thus the DM can analyse the ranking of the alternatives for each sub-problem, gaining more detailed knowledge of the complex decision problem; second, the DM can determine the level of concordance and the power of discordance at each level of the hierarchy. The model is therefore more complete and close to the DM needs because the aggregation conditions are different for each node depending on its particular meaning. 2. The ELECTRE-III method In this section, we review the basic concepts and steps of the ELECTRE-III method:  A = fa; b; c; . . .g is the finite set of alternatives,  n is the number of alternatives in A,  G = fg 1 ; g 2 ; . . . ; g m g is the finite set of criteria (it is assumed that m P 3),  g j ðaÞ represents the performance of alternative a on criterion g j 2 G. We assume, without loss of generality, that all the criteria are of the gain type, i.e., the greater the value, the better. A performance matrix M is built for A  G, where g j ðaÞ is the performance in row a and column j. Preferences in ELECTRE methods are modeled by a binary outranking relation S. Considering two alternatives, a and b, from the set A, four situations may occur:  aSb and not bSa : aPb (a is strictly preferred to b),  bSa and not aSb : aP  b (b is strictly preferred to a, which can be expressed as a being inversely preferred to b),  aSb and bSa : aIb (a is indifferent to b),  Not aSb and not bSa : aRb (a is incomparable to b). The outranking relation S is built taking into account the set G. Each criterion may support or be against the assertion aSb. Due to the imprecision and uncertainty inherent to complex human evaluation processes, criteria are modeled as pseudo-criteria (Rogers & Bruen, 1998). Consequently, in ELECTRE-III the outranking relation can be interpreted as a fuzzy relation. Each pseudo-criterion is associated with two discrimination thresholds, which may either be fixed or dependent on the evaluation g j ðaÞ:  indifference threshold qj ½g j ðaÞ, below which the DM is indifferent to two alternatives in terms of their evaluations on criterion gj ;

4913

 preference threshold pj ½g j ðaÞ, above which the DM shows a clear strict preference of one alternative over the other in terms of their evaluations on criterion g j . In addition, a veto threshold v j ½g j ðaÞ is also associated to the evaluation g j ðaÞ, where a discordant difference in favour of one alternative greater than this value will require the DM to negate any possible outranking relationship indicated by the other criteria. When comparing two alternatives, a; b 2 A, with respect to criterion g j 2 G, it is usually assumed that the thresholds are functions of the worst evaluation of the two alternatives. For the sake of simplicity, we will use the notation qj ðaÞ; pj ðaÞ; v j ðaÞ instead of qj ½g j ðaÞ; pj ½g j ðaÞ; v j ½g j ðaÞ in the rest of the paper. These thresholds define the preference structure that will be generated from the comparison of the alternatives, thus defining the decision model according to the DM requirements. The decision model also includes weights of criteria. A weight wj expresses the relative importance of criterion g j , as it can be interpreted as the voting power of each criterion to the outranking relation. The weights of criteria do not represent substitution rates as in the case of compensatory aggregation operators. Therefore, methods like AHP are not appropriate because they are designed to find tradeoff weights. In case of having a set of solved examples (i.e., supervised data set) a suitable way to find the weights is using the ROR method (Corrente, Greco, Kadzinski, & Slowinski, 2013a). When no solved examples are available, a well accepted procedure to help the DM to give the weights for ELECTRE methods is the Simos’ procedure (Figueira & Roy, 2002). This approach consists of associating a playing card with each criterion and rank these cards from the less to the most important with possibly ex aequo (cards or criteria with the same rank). The DM can put ‘‘white cards’’ between these ranks to express the relative power of each one, by making smaller or bigger the difference between ranks. The ELECTRE-III method has two steps: first, the construction of an outranking relation over all the possible pairs of alternatives; second, the exploitation of this outranking relation to solve the ranking decision problem (Figueira et al., 2013). 2.1. Construction of the outranking relation Given an ordered pair of alternatives ða; bÞ 2 A  A, alternative a outranks alternative b if a outperforms b on enough criteria of sufficient importance, and a is not outperformed by b with a significantly inferior performance on any single criterion. The outranking relation aSb is constructed on the basis of two tests:  Concordance test: A sufficient majority of criteria should be in favor of the assertion aSb.  Discordance test: In spite of the concordance condition, none of the criteria in the minority should be strongly against the assertion aSb, because very bad performance on one criterion may not be compensated by good performances on other criteria. To determine the credibility qðaSbÞ of the outranking relation, we calculate a partial concordance index cj ða; bÞ and a partial discordance index dj ða; bÞ for each criterion g j . 2.1.1. Concordance test Sometimes referred to as ‘‘the respect of the majority’’, the concordance test entails the calculation of a concordance index cða; bÞ that measures the strength of the coalition of criteria that support the hypothesis ‘‘a is at least as good as b’’. The overall concordance index is computed for each ordered pair a; b 2 A as follows:

4914

cða; bÞ ¼

L. Del Vasto-Terrientes et al. / Expert Systems with Applications 42 (2015) 4910–4926 m 1 X wj cj ða; bÞ W j¼1

where W ¼ defined as:

Pm

j¼1 wj ,

ð1Þ

and the partial concordance index cj ða; bÞ is

8 1 if g j ðaÞ P g j ðbÞ  qj ðbÞ > > < 0 if g j ðaÞ 6 g j ðbÞ  pj ðbÞ cj ða; bÞ ¼ > > g ðaÞg ðbÞþp ðbÞ j j : j otherwise: pj ðbÞqj ðbÞ

ð2Þ

2.1.2. Discordance test Sometimes referred to as ‘‘the respect of minorities’’, the discordance test entails the calculation of discordance index dj ða; bÞ that measures the strength of evidence provided by the jth criterion against the hypothesis aSb. The computation of the discordance index takes into account the criteria that disagree with the assertion aSb. In this case, each criterion is assigned a veto threshold v j . The veto is the maximum difference allowed between the evaluations of a pair of alternatives when g j ðaÞ  g j ðbÞ. The partial discordance index is defined as follows:

8 1 if g j ðaÞ  g j ðbÞ 6 v j ðaÞ > > < if g j ðaÞ  g j ðbÞ P pj ðaÞ dj ða; bÞ ¼ 0 > > : gj ðbÞgj ðaÞþpj ðaÞ otherwise: v j ðaÞpj ðaÞ

1. K 0 ¼ A n ðC 1 [ . . . [ C h1 Þ; i 1. 2. Using Si , construct K i as the subset of actions from K i1 whose qualification is maximum. i þ 1 and go to 3. If jK i j ¼ 1 or i ¼ r then C h ¼ K i and STOP, else i step 1. Note that two or more alternatives may belong to one distillate if they have the same qualification and none can be ranked better or worse than others. In this case, the alternatives are said to be indifferent and are assigned to the same ranking position. An illustrative example is shown in Fig. 1 for A ¼ fa; b; c; d; e; f ; g; h; i; j; k; lg. The intersection of the two complete pre-orders O # and O " gives the final partial pre-order O, shown in Fig. 2. This partial pre-order establishes a preference structure on the set of alternatives A. For each possible pair of alternatives, it assigns one of the following four binary relations fP; P ; I; Rg, so that for any two alternatives from set A, one may be preferred over the other,

ð3Þ

2.1.3. Degree of credibility of the outranking relation The overall concordance (1) and partial discordance (3) indices are combined to obtain a valued outranking relation with credibility qðaSbÞ 2 ½0; 1 defined by:

8 > < cða; bÞ Y qðaSbÞ ¼ cða; bÞ > : j2J

if dj ða; bÞ 6 cða; bÞ; 1dj ða;bÞ 1cða;bÞ

otherwise

8j ð4Þ

ða;bÞ

where Jða; bÞ is the set of criteria for which dj ða; bÞ > cða; bÞ. The credibility of outranking is equal to the overall concordance index when there is no discordant criterion.

Fig. 1. Complete pre-orders O # and O " obtained from a distillation process.

2.2. Ranking procedure The next step in the ELECTRE-III method is the exploitation of the credibility matrix to build a partial pre-order of the alternatives in A. This procedure is known as distillation. We assume that r outranking relations exist: S1  S2  . . . Sr (with r > 1). The exploitation procedure in ELECTRE-III consists in progressively refining the prescription by successively considering these relations S1 ; S2 ; . . . ; Sr (Vanderpooten, 1990). This refinement can be performed in two ways: from S1 to Sr or viceversa. ELECTRE-III considers both possibilities and ranks the alternatives in two complete pre-orders which are constructed in two different ways. The first complete pre-order is obtained in a descending manner (descending distillation), selecting the best rated alternatives initially, and finishing with the worst. The second complete pre-order is obtained in an ascending manner (ascending distillation), selecting the worst rated alternatives initially, and finishing with the best. Both distillations make an iterated choice based on a qualification index measured from Si . The procedure is as follows for the descending distillation (and analogously for the ascending one). At a certain iteration of the descending distillation procedure, we construct the class C h composed of ex aequo elements, having already constructed classes C 1 ; . . . ; C h1 (where class C 1 is the head class in the descending distillation and A n ðC 1 ; . . . ; C h1 Þ – /). The steps for building C h are:

Fig. 2. Partial pre-order O from the intersection of O" and O#.

L. Del Vasto-Terrientes et al. / Expert Systems with Applications 42 (2015) 4910–4926

or they may be indifferent, or incomparable. The incomparability of two alternatives occurs when one of these alternatives, let us say a, is ranked better than b in O ", but b is ranked better than a in O #. For example, in Fig. 2 alternatives e and h are incomparable in the final pre-order, because in Fig. 1 e is ranked better than h in O", but h is ranked better than e in O #. 3. Hierarchical structure of criteria In this section we define the concepts and notation with regard to a hierarchical structure of criteria. We will distinguish between three types of criteria depending on their level of generality in the taxonomy:  R is a set composed by a unique element that is the most general criterion. This corresponds to the root node, placed at the top of the tree. This criterion represents the main goal of the DM.  E is the set of the most specific criteria, called elementary criteria. They are placed at the lowest level of the hierarchical tree (i.e., the leaves). The performance of the alternatives is evaluated only in relation to these elementary criteria.  I is the set of intermediate criteria (or sub-criteria). They correspond to generalisations of other sub-criteria or elementary criteria. They are placed at intermediate levels of the tree, between R and E. Then, G is redefined with respect to the definition given in Section 2, so that it includes not only the elementary criteria but also the more general criteria, such that G = R [ I [ E. Let us set the number of criteria as m ¼ jGj, while 1 ¼ jRj; l ¼ jEj and h ¼ jI j, having that m ¼ 1 þ l þ h. Definition 1. The set of criteria is structured according to the following relations:    

The root criterion in R does not have any parent. Each intermediate criterion in I has a unique parent g j 2 R [ I . Each elementary criterion in E has a unique parent g j 2 R [ I . Criteria in I [ R may have multiple direct descendants gj 2 E [ I . An example of this structure is shown in Fig. 3: The structure presented in Fig. 3 contains as elements:

   

R = fg 1 g I = fg 1:1 ; g 1:2 ; g 1:2:1 ; g 1:2:2 g E = fg 1:1:1 ; g 1:1:2 ; g 1:1:3 ; g 1:2:1:1 ; g 1:2:1:2 ; g 1:2:2:1 ; g 1:2:2:2 g G ¼ fg 1 ; g 1:1 ; g 1:2 ; g 1:2:1 ; g 1:2:2 ; g 1:1:1 ; g 1:1:2 ; g 1:1:3 ; g 1:2:1:1 ; g 1:2:1:2 ; g 1:2:2:1 ; g 1:2:2:2 g

Fig. 3. Hierarchical tree of the set of criteria.

4915

All criteria in set G can be considered as pseudo-criteria, associated with indifference, preference and veto thresholds, except for the root criterion. These thresholds are defined as follows:  The indifference qj ðaÞ, preference pj ðaÞ and veto v j ðaÞ thresholds referring to elementary criteria from set E are fixed in the same way as in the ELECTRE-III method, based on the performance of the alternatives and depending on the scale of measurement of each criterion g j .  The indifference qj ðaÞ, preference pj ðaÞ, and veto v j ðaÞ thresholds referring to criteria from set I are functions of the difference of rank order value of the alternatives in a partial pre-order O. Having that jAj ¼ n, then 0 6 qj ðaÞ 6 pj ðaÞ 6 v j ðaÞ 6 n  1.

Definition 2. For D  E [ I , being the set of direct descendants of g i , each g j 2 D may have a weight wj that indicates its relative importance with respect to the rest of the descendants of g i (i.e., the rest of the elements of D). Fig. 4 shows the hierarchical tree of criteria with weights assigned to all criteria from set G. In the next section we introduce an extension of the ELECTRE-III method to deal with a hierarchical set of criteria, as defined above. 4. ELECTRE-III-H method In this section we propose the ELECTRE-III-H method, which calculates partial pre-orders at all levels of the hierarchy of criteria. The construction procedure is analogous to that of ELECTRE-III, presented in Section 2, in the sense that it also entails the calculation of the outranking relation (i.e., concordance and discordance tests) and the exploitation of this relation by distillation. However, this procedure also takes into account that evaluations of alternatives on intermediate criteria are given in the form of partial pre-orders, rather than complete pre-orders, as this is the case for elementary criteria. The procedure is presented in Algorithm 1. Algorithm 1. ELECTRE-III-H method 1: function ElectreIII-H (Criteria G, Alternatives A, PerformanceMatrix M) 2: X List of g j 2 I [ R with descendants all in E 3: Y List of g j 2 I [ R with descendants in E and I 4: Y sortCriteriaByLevels(Y) .Sort Y bottom-up 5: O = null .Set of partial pre-orders 6: for all xj 2 X do 7: Z get Children Criteria(xj ) q build Electre III Credibility(Z; A; M) 8: 9: Oj calculate Exploitation(q) 10: O ¼ O [ Oj 11: end for 12: for all yj 2 Y do 13: Z get Children Criteria(yj ) q build Electre III H Credibility(Z; A; M; O) 14: 15: Oj calculate Exploitation(q) 16: O ¼ O [ Oj 17: end for 18: return O 19: end function

ELECTRE-III-H distinguishes between two cases that lead to the construction of two lists of criteria that are treated differently:

4916

L. Del Vasto-Terrientes et al. / Expert Systems with Applications 42 (2015) 4910–4926

of the outranking relations, which results in a partial pre-order. In the following subsections we present the entire ELECTRE-III-H procedure in detail. This procedure introduces discrimination and veto thresholds with respect to partial pre-order positions. 4.1. Building and exploiting the credibility matrix with criteria in E

Fig. 4. Hierarchical tree of criteria with their weights.

 List X, in line 2, contains the intermediate criteria and the root criterion whose immediate descendants are all elementary criteria (i.e., all descendants belong to E).  List Y, in line 3, contains the intermediate criteria and the root criterion that have as immediate descendants other intermediate criteria, possibly including some, but not all, elementary criteria (at least one descendant must be an intermediate criterion). The procedure is based on a bottom-up approach, so that the criteria are analysed from the lowest level up to the root. Each of the two lists, X and Y, undergoes a different treatment because of the differences in the information given by the criteria. Note that the difference is in the calculation of the credibility matrix q that depends on the definition of the concordance and discordance indices.

The first part of the proposed algorithm corresponds to the calculation of the credibility matrix for nodes whose only descendants are elements in E. In this case the classical ELECTRE-III method is applied to aggregate evaluations of alternatives on the elementary criteria. For example, taking into account the hierarchy in Fig. 4, the original ELECTRE-III is applied separately for each subset of E with the same ancestor: fg 1:1:1 ; g 1:1:2 ; g 1:1:3 g; g 1:2:1:1 ; g 1:2:1:2 g; fg 1:2:2:1 ; g 1:2:2:2 g. The credibility is calculated directly from the evaluations in the performance matrix M, using the procedures explained in Section 2.1. This step corresponds to the first stage of Algorithm 1 (treatment of list X). The exploitation of the credibility matrix proceeds in the usual way, as explained in Section 2.2. This stage results in a partial preorder of alternatives for each immediate predecessor criterion. In our example, the results of the exploitation are three partial preorders fO1:1 ; O1:2:1 ; O1:2:2 g. These results are stored in the corresponding criterion, and are considered as evaluations of alternatives on these criteria fg 1:1 ; g 1:2:1 ; g 1:2:2 g. In the following section we explain how to aggregate partial pre-orders at intermediate levels of the hierarchical tree. 4.2. Building the credibility matrix with criteria in I For nodes in the hierarchy tree that have at least one direct descendant in I , the credibility cannot be calculated in the classical way. In this second stage of Algorithm 1, which must take into account the partial pre-orders resulting from prior evaluations in the hierarchy tree, we propose a new calculation of the partial concordance and discordance indices for ELECTRE-III-H. In the example presented in Fig. 4, ELECTRE-III-H is applied level by level up to R, as follows:

1. In the first stage, list X is treated (from line 6 to line 11) with the nodes placed at the bottom level of the hierarchy, i.e., the elementary criteria. This stage aggregates groups of elementary criteria by their direct ancestor xj to obtain the first result in the form of partial pre-orders. The credibility matrix is calculated using classical ELECTRE-III indices, using the performance scores stored in matrix M (line 8). The exploitation of the credibility matrix generates a partial pre-order for each node in Oj (line 9) and stored in the set O (line 10). Note that only a subset of criteria is considered in the credibility calculation, which contains the direct descendants of the current node xj (stored in Z in line 7). 2. In the second stage, the algorithm treats list Y (from line 12 to line 17). Then, the partial pre-orders obtained before (stored in O) are used as inputs for the upper level criteria. This requires the list to be ordered according to the precedence relations indicated by the tree structure of set G, from the lowest level up to the most general criterion (root). So, Y is sorted such that yj has no descendant in yjþ1 . . . ym (line 4). In line 14, the credibility is calculated using new formulas that will be defined in this section, which redefine the partial concordance and discordance indices in order to handle all of the binary preference relations that can be found in a partial pre-order (indifference, incomparability and preference). Therefore, in this case, both the performance matrix M and the list of partial pre-orders O are needed to calculate the credibility index according to the nature of the descendants of the current node xj (stored in Z in line 7). Finally, the exploitation procedure of the credibility is applied (line 15), resulting in a new partial pre-order Oj for each node g j 2 Y.

Considering any ordered pair of alternatives ða; bÞ, the goal is to propose a method for calculating cj ða; bÞ and dj ða; bÞ from Oj . For each pair of alternatives, we have only four possible binary relations: fP; P ; I; Rg. Thus, a partial pre-order can be represented in a matrix A  A containing the type of preference relation for each pair. Let us take an example with a set A ¼ fr; s; t; u; v ; w; x; y; zg of alternatives, structured in the partial pre-order O1 , as shown in Fig. 5. Let us suppose that this partial pre-order has been generated from a subset of E. The corresponding preference relations are given in Table 1. In the following subsections we explain how to calculate the partial concordances and discordances for each case of the four binary relations fP; P ; I; Rg. The definitions will be established using the information about the preference relations given in the preference matrix. The last column of Table 1 corresponds to the rank order value CðÞ, defined below.

Note that the procedure at each node is done following the two steps defined in ELECTRE methods: (1) construction of a credibility matrix based on the partial concordance and partial discordance indices, using (4); and (2) the distillation process for exploitation

Definition 3. Rank Order Value. The rank order value of an alternative a 2 A in a partial pre-order Oj is the number of alternatives that are preferred to a in this partial pre-order. It is denoted by Cj ðÞ.

 O1:2:1 ; O1:2:2 are aggregated to obtain O1:2  O1:1 ; O1:2 are aggregated to obtain O1

4917

L. Del Vasto-Terrientes et al. / Expert Systems with Applications 42 (2015) 4910–4926

 cj ðzP j wÞ = 1, and the partial discordance index dj ðzP j wÞ = 0, as the difference of Cj ðzÞ  Cj ðwÞ ¼ qj ðzÞ.  Case 2: When the difference between the rank order values Cj ðaÞ  Cj ðbÞ is greater than the indifference threshold qj ðaÞ and less than the preference threshold pj ðaÞ, the partial concordance index decreases, and discordance is zero. The calculation of the partial concordance index proceeds analogously to the classical ELECTRE-III in terms of the Cj ðÞ function:

8 p ðaÞðCj ðaÞCj ðbÞÞ if Cj ðaÞ  Cj ðbÞ > qj ðaÞ AND < cj ðaPj bÞ ¼ j pj ðaÞq j ðaÞ Cj ðaÞ  Cj ðbÞ 6 pj ðaÞ; then : dj ðaP bÞ ¼ 0 j ð10Þ uP j x

In the example presented in Fig. 5, for and Cj ðuÞ = 3, Cj ðxÞ = 1, qj ðuÞ = 1 and pj ðuÞ = 3, the partial discordance index  dj ðuP j xÞ = 0, whereas the partial concordance index cj ðuP j xÞ is calculated as follows:

Fig. 5. Example of a partial pre-order O1 generated in E.

Table 1 Matrix of preference relations in pre-order O1 .

r s t u v w x y z

r

s

t

u

v

w

x

y

z

C

I P R R P PP PP-

PI R PP PI PP-

R R I R P R R PR

R P R I P PP PP-

PPPPI PPPP-

P P R P P I P PP-

PI R PP PI PP-

P P P P P P P I P

P P R P P P P PI

3 1 1 3 0 5 1 8 6

cj ðuPj xÞ ¼

3  ð3  1Þ ¼ 0:5 31

 Case 3: When the difference between the rank order values Cj ðaÞ  Cj ðbÞ is greater than the preference threshold pj ðaÞ and less than or equal to the veto threshold v j ðaÞ, the partial concordance index is zero and the discordance increases. The calculation of the partial discordance index is analogous to the classical ELECTRE-III in terms of the Cj ðÞ function.

8  if Cj ðaÞ  Cj ðbÞ > pj ðaÞ AND < cj ðaPj bÞ ¼ 0 Cj ðbÞpj ðaÞ Cj ðaÞ  Cj ðbÞ 6 v j ðaÞ; then : dj ðaPj bÞ ¼ Cj ðaÞ v ðaÞp ðaÞ j

4.2.1. Preference and indifference relations, P and I The first situation we consider is when a is strictly preferred or indifferent to b in a partial pre-order Oj . Remember that the concordance index measures the support to the outranking relation defined as ‘‘a is at least as good as b’’, aSb. Since S ¼ I _ P, both preference aPj b and indifference aIj b relations in Oj indicate that Oj clearly supports the aSb. Therefore, the value of partial concordance index is set to 1:

cj ðaPj bÞ ¼ 1

ð5Þ

cj ðaIj bÞ ¼ 1

ð6Þ

Following the previous rationale, when aP j b and aIj b in Oj , we set the partial discordance index to 0:

dj ðaP j bÞ ¼ 0

ð7Þ

dj ðaIj bÞ ¼ 0:

ð8Þ

4.2.2. Inverse preference relation P When b is preferred over a in the partial pre-order Oj , the strength of the difference between b and a must be considered to calculate the degree of concordance or discordance with respect to the outranking relation aSb. We distinguish four cases:  Case 1: When the difference between the rank order values Cj ðaÞ  Cj ðbÞ is less than or equal to the indifference threshold qj ðaÞ. In this case the concordance with aSb is maximum and the discordance is zero.

( if Cj ðaÞ  Cj ðbÞ 6 qj ðaÞ;

then

cj ðaP j bÞ ¼ 1 dj ðaPj bÞ ¼ 0

ð9Þ

In our example, presented in Fig. 5, for zP  j w and Cj ðzÞ = 6, Cj ðwÞ = 5 and threshold qj ðzÞ = 1, the partial concordance index

j

ð11Þ In

the

example

presented

in

Fig.

and the partial concordance index cj ðzP j xÞ = 0, while the partial discordance

Cj ðzÞ ¼ 6; Cj ðxÞ = 1, qj ðzÞ = 1, pj ðzÞ = 3 and

5,

for

zP j x

v j ðzÞ = 6,

index dj ðzP j xÞ is calculated as follows:

dj ðzP j xÞ ¼

613 ¼ 0:667 63

 Case 4: When the difference between rank order values Cj ðaÞ  Cj ðbÞ is greater than veto threshold v j ðaÞ, the discordance is maximum and the concordance is zero.

( if Cj ðaÞ  Cj ðbÞ > v j ðaÞ;

then

cj ðaPj bÞ ¼ 0 dj ðaPj bÞ ¼ 1

ð12Þ

This case evaluates the partial concordance and discordance indices of yP j x in the pre-order given in Fig. 5, where Cj ðyÞ = 8, Cj ðxÞ = 1 and assuming that v j ðyÞ = 6. Note that, according to these definitions, the point that determines whether the difference on Cj ðaÞ  Cj ðbÞ is in support of (some degree of concordance) or against aSb (some degree of discordance) is the value of the preference threshold pj . 4.2.3. Incomparability relation R When, in partial pre-order Oj , alternative a is incomparable to alternative b, it is impossible to state whether this relation is closer to aP j b or aIj b or aP j b; thus, the partial pre-order gives no clear support to the outranking aSb. In this case, we take into account additional information about alternatives a and b given by the function Cj ðÞ. If the difference between the rank order values of Cj ðaÞ and Cj ðbÞ is negative or close to 0, then this should enforce the conviction that aRj b could turn to aP j b or aIj b rather than to aP j b, since the rank order value of a is less than the rank order

4918

L. Del Vasto-Terrientes et al. / Expert Systems with Applications 42 (2015) 4910–4926

value of b. Otherwise, if the difference between the rank order values of a and b were positive, then this should enforce the conviction that aRj b could turn to aP  j b rather than to aP j b or aIj b. In any case, for a pair of incomparable alternatives, the information is uncertain, so we will not set the maximum value for either c concordance or discordance. Instead, we use the base values k < 1 d

and k < 1 for partial concordance and discordance, respectively, which are tuned as described below. To establish the base values we can consider that aRj b could turn with an equal probability to aP j b; aIj b or aP  j b and that S ¼ I _ P, concluding that only two of the three possible relations support S. Thus, there is

2 3

d

aRj b would confirm aSb. We therefore propose k ¼ 23 and k ¼ 13. The proposed rules for the calculation of partial concordance and discordance indices in the case of incomparability are as follows:

if Cj ðaÞ  Cj ðbÞ 6 pj ðbÞ;

then

if Cj ðaÞ  Cj ðbÞ > pj ðbÞ;

then

cj ðaRj bÞ ¼ 0 d

dj ðaRj bÞ ¼ k þ ddj ða; bÞ dcj

if Cj ðaÞ  Cj ðbÞ < qj ðaÞ; if Cj ðaÞ  Cj ðbÞ ¼ qj ðaÞ;

then cj > k

c

ð17Þ

then cj ¼ k

c

ð18Þ

ð13Þ

ð14Þ

if Cj ðaÞ  Cj ðbÞ > qj ðaÞ AND if Cj ðaÞ  Cj ðbÞ 6 pj ðaÞ;

c

then cj < k :

ð19Þ

ddj

The rationale for adding the tuning factors and to the above formulas is that the partial concordance and discordance indices in these two rules should depend on the magnitude of the difference

Cj ðaÞ  Cj ðbÞ. The tuning factors dcj and ddj may be positive or negative, hence, they either increase or decrease the base partial concorc

Remark 1. If Cj ðaÞ  Cj ðbÞ 6 qj ðaÞ, this supports the conviction that the incomparability aRj b could turn to aP j b or aIj b rather than to aP  j b. This corresponds to the case of strong concordance, where, according to (13), the partial concordance behaves as follows:

Remark 2. If Cj ðaÞ  Cj ðbÞ > qj ðaÞ and Cj ðaÞ  Cj ðbÞ 6 pj ðaÞ, this supports the conviction that the incomparability aRj b could turn to aPj b or aIj b, rather than to aP  j b. So, this is a situation of weak concordance, where:

c

cj ðaRj bÞ ¼ k þ dcj ða; bÞ dj ðaRj bÞ ¼ 0

(

6 0:25 and 0:08 6 dj ðaRj bÞ 6 0:58. From this definition of concordance and discordance for the case of incomparability, we can find properties for the different cases determined by the thresholds.

chance that

c

(

0:42 6 cj ðaRj bÞ 6 0:92 and for discordance index 0:25 6 ddj ða; bÞ

d

Remark 3. If Cj ðaÞ  Cj ðbÞ > pj ðaÞ and Cj ðaÞ  Cj ðaÞ 6 v j ðaÞ, this supports the conviction that the incomparability aRj b could turn to aP  j b rather than to aP j b or aIj b. This corresponds to situation of weak discordance, where:

dance k or discordance k . More precisely, we propose: d

 dcj for the partial concordance index: We establish the following logical condition for concordance cj ðaRj bÞ: ‘‘If alternative a is incomparable to alternatives b and d in a partial pre-order Oj , and Cj ðaÞ  Cj ðbÞ < Cj ðaÞ  Cj ðdÞ 6 v j ðbÞ and 6 v j ðdÞ, then cj ðaRj bÞ should be greater than cj ðaRj dÞ’’. According to this condition, for each pair ða; bÞ 2 A  A, such that aRj b and Cj ðaÞ  Cj ðbÞ 6 v j ðaÞ in Oj , we propose:

dcj ða; bÞ

ðCj ðbÞ  Cj ðaÞ  qj ðaÞÞ  a ¼ ðpj ðaÞ  qj ðaÞÞ þ ðn  2Þ

d

ð20Þ ð21Þ

Remark 4. If Cj ðaÞ  Cj ðbÞ > v j ðaÞ, this supports the conviction that the incomparability aRj b could turn to aP  j b, rather than to aP j b or aIj b. This is the case of strong veto, where:

then dj > k

d

ð22Þ

ð15Þ

 ddj for the partial discordance index: We establish the following logical condition for concordance dj ðaRj bÞ: ‘‘If alternative a is incomparable to alternatives b and d in a partial pre-order Oj , and Cj ðaÞ  Cj ðbÞ > Cj ðaÞ  Cj ðdÞ P v j ðbÞ and P v j ðdÞ, then dj ðaRj bÞ should be greater than dj ðaRj dÞ’’. According to this condition, for each pair ða; bÞ 2 A  A, such that aRj b and Cj ðaÞ  Cj ðbÞ > v j ðaÞ in Oj , we propose:

ðCj ðaÞ  Cj ðbÞ  v j ðaÞÞ  a ðv j ðaÞ  pj ðaÞÞ þ ðn  2Þ

if Cj ðaÞ  Cj ðbÞ ¼ v j ðaÞ; then dj ¼ k

if Cj ðaÞ  Cj ðbÞ > v j ðaÞ;

where n is the number of alternatives in A, and 2 is subtracted from n in the denominator to account for the two incomparable alternatives considered (a and b). The value a has been introduced to control the maximum degree of change to the original partial concordance index for incomparability.

ddj ða; bÞ ¼

if Cj ðaÞ  Cj ðbÞ > pj ðaÞ AND if Cj ðaÞ  Cj ðbÞ < v j ðaÞ; then dj < k

ð16Þ

4.3. Exploitation of the credibility matrix for intermediate and root criteria At a given criterion g i , using the equations presented in Section 4.2, we can calculate a partial concordance cj and partial discordance dj indices from a given partial pre-order Oj obtained in a lower level, where j is a descendant of i. If g i has descendants that are elementary criteria (g j 2 E), then cj and dj are calculated as in classical ELECTRE-III using (2) and (3). Overall ci is calculated with (1). Then, the partial concordance indices obtained for each type of criterion are merged when the credibility matrix is calculated with (4). This credibility matrix is then exploited with the distillation algorithm, as defined in Section 2.2. Two complete pre-orders are generated from the ascending and descending distillation chain, which are merged to generate a partial pre-order Oi . 5. Case study

Again, the value a controls the maximum permitted degree of change to the original partial discordance index for incomparability. Note that a should be smaller than 13 in order to keep the concordance and discordance indices below 1. For example, we can set a to 0.25, so that for concordance index 0:25 6 dcj ða; bÞ 6 0:25 and

In this case study we wish to construct a priority ranking of websites designed to promote tourist destination brands. Websites have become crucial tools for communicating destination brands and for selling a variety of tourism services and related products (Fernández-Cavia & Huertas-Roig, 2009). However, as

L. Del Vasto-Terrientes et al. / Expert Systems with Applications 42 (2015) 4910–4926

some authors highlight, the tourism industry urgently needs to agree on sector-wide use of specific website evaluation techniques that are well established and can be used in the long term (Law, Qi, & Buhalis, 2010). The dataset considered in this section comes from a Spanish research project entitled ‘‘Online Communication for Destination Brands. Development of an Integrated Assessment Tool: Websites, Mobile Applications and Social Media (CODETUR)’’. The main aim of this project was to define a website evaluation framework to help destination managers to enhance and optimise the online communication of their brands. Once this framework has been defined, MCDA methods can be used to support the decision making process. A team of experts in tourism and communication constructed a set of indicators and defined a methodological procedure for using these indicators to evaluate tourist destination websites. A small set of websites was used as testing data. In this section we address the analysis of this dataset with the method proposed in the paper: ELECTRE-III-H. Due to the complexity of the problem, the experts organised the criteria into a hierarchy. The goal of the DM is not only to obtain a comprehensive ranking of the websites, but also to determine the position of the alternatives with respect to three different aspects of the problem. This will make it possible to identify the strengths and weaknesses of each website and to design appropriate strategies for addressing the weaknesses. The three main criteria considered are the following: 1. Usability and Accessibility: On the one hand, Usability evaluates the simplicity of using a website, assessing navigation indicators such as efficiency, coherence, loading speed or pleasantness. On the other hand, Accessibility refers to the set of recommendations, strategies and resources provided to users to facilitate navigation of the website, with a particular focus on the needs of users with disabilities. Accessibility also assesses the capacity of accessing to these resources by using limited devices such as smartphones. 2. Web Visibility: Nowadays, web search engines are a key tool for discovering tourist destinations. Therefore, the rank position of the websites in search engines is an important indicator of Web Visibility. Search engine optimisation is a field that studies the factors that influence the rank position of a website. Two dimensions are considered: internal and external factors. Internal factors include the use of appropriate keywords or special web programming techniques that may help to improve visibility in search engines, such as metadata. External factors are criteria that depend on users or marketing issues, such as the number of links to the website from tourism directories and portals, or the traffic rank measure.

4919

3. Brand Treatment: This criterion refers to the mental image that the user forms about the brand through using the website, which influences the perception of the benefits of a given tourist destination. This image is created implicitly by means of different communicative acts. Part of the mental image is given by the logotype and slogan, and the use of multimedia elements is also important to reinforce the message and create an identity associated with the brand (e.g., videos, photographs or stories). These three criteria are decomposed into several sub-criteria. To determine the weights of the criteria, the Simos method was applied with collaboration of the DM. The hierarchical structure defined is presented in Fig. 6. Using the notation introduced in Section 3, we have:  R = fg 1 g  I = fg 1:1 ; g 1:2 ; g 1:3 ; g 1:1:1 ; g 1:1:2 ; g 1:3:1 ; g 1:3:2 g  E = fg 1:1:1:1 ; g 1:1:1:2 ; g 1:1:1:3 ; g 1:1:2:1 ; g 1:1:2:2 ; g 1:1:2:3 ; g 1:2:1 ; g 1:2:2 ; g 1:3:1:1 ; g 1:3:1:2 ; g 1:3:2:1 ; g 1:3:2:2 g The corresponding elementary criteria (i.e., basic indicators) are defined in Table 2, together with the assigned weights. As explained above, the experts involved in the CODETUR project have defined a methodology for measuring the performance

Table 2 Elementary criteria names and weights. Criteria ID

Name

Brief description

w%

g 1:1:1:1

General usability indicators such as the use of a correct URL, website updates, etc Images quality, visual added services, etc The website offers help for complex tasks and provides help with online shopping

43

g 1:1:1:2 g 1:1:1:3

General indicators Multimedia Help

g 1:1:2:1 g 1:1:2:2

Font Navigation

30 37

g 1:1:2:3

Accessibility

Suitable font size and contrast Compatibility with different browsers, resolutions, and ease of navigation Accessibility for users with disabilities

g 1:2:1

Internal factors External factors

The website uses programming techniques that may help to improve web visibility Users or marketing factors such as traffic rank and links in tourism directories or portals

33

g 1:3:1:1 g 1:3:1:2

Slogan Logotype

Presence of the website slogan on home page Presence of the website logotype and visual brand identity on the website

20 80

g 1:3:2:1

Brand in images Brand in text

Communication of the emotional and functional identity of the brand Textual description and storytelling of the brand

55

g 1:2:2

g 1:3:2:2

Fig. 6. Hierarchy for website evaluation with weights for intermediate criteria.

36 21

33

67

45

4920

L. Del Vasto-Terrientes et al. / Expert Systems with Applications 42 (2015) 4910–4926

Table 3 Performance matrix of alternatives with regard to the criteria in E. Usability

Andalusia Catalonia Barcelona Madrid Santiago Rias Baixas Stockholm Wales Rome Switzerland

Andalusia Catalonia Barcelona Madrid Santiago Rias Baixas Stockholm Wales Rome Switzerland

Accessibility

g 1:1:1:1

g 1:1:1:2

g 1:1:1:3

g 1:1:2:1

g 1:1:2:2

g 1:1:2:3

100 74 100 88 100 50 100 88 100 76

100 30 60 100 80 30 100 70 100 100

60 0 80 60 20 0 100 0 100 100

25 75 75 50 50 75 100 100 100 100

62, 5 87, 5 62, 5 100 50 62, 5 62, 5 87, 5 62, 5 62, 5

16 38, 67 100 61,33 32 16 38, 67 0 16 16

Web visibility

Slogan and logotype

Brand image

g 1:2:1

g 1:2:2

g 1:3:1:1

g 1:3:1:2

g 1:3:2:1

g 1:3:2:2

72, 2 33, 3 74, 1 75, 9 100 46, 3 74, 1 92, 6 74, 1 83, 3

75, 59 85, 72, 68, 59 82, 65, 65, 90,

100 0 0 0 100 100 100 100 0 100

58 78 74 38 48 38 54 58 38 68

100 70 100 50 80 50 60 80 70 100

60 30 70 60 60 30 70 30 60 100

53 93 87 6 47 93 93 2

Table 4 Elementary criteria preference thresholds values.

of websites in relation to the elementary criteria in Table 2 (Fernández-Cavia et al., 2010). Following this methodology, 10 websites were analysed, including tourist brand sites for cities and autonomous communities in Spain and international destinations. The corresponding tourist destination brands are: Andalusia, Catalonia, Barcelona, Madrid, Santiago de Compostela, Rias Baixas, Stockholm, Wales, Rome and Switzerland. The data were collected in Summer 2012. The performance table obtained from the assessment of the elementary criteria is presented in Table 3. The range of values is 0–100; all criteria must be maximised.

5.1. ELECTRE-III-H at the level of elementary criteria In this section, we present the results of the first stage of Algorithm 1 at intermediate criteria whose descendants are all elementary criteria. The thresholds of the elementary criteria are constant, as defined in Table 4. Classical ELECTRE-III is applied to these intermediate criteria, which are based only on the elementary criteria. The corresponding partial pre-orders are shown in Fig. 7. These are the first partial pre-orders obtained with ELECTRE-IIIH. With this information, the DM can observe the position and relations of each website in the 5 lowest intermediate criteria. We proceed by propagating this information to the upper nodes in the hierarchy.

5.2. ELECTRE-III-H at the level of intermediate criteria

Criteria

qj

pj

vj

g 1:1:1:1 g 1:1:1:2 g 1:1:1:3

0 10 10

10 35 35

25 75 75

g 1:1:2:1 g 1:1:2:2 g 1:1:2:3

10 0 7

35 10 25

75 25 55

g 1:2:1 g 1:2:2

0 0

10 9

25 23

g 1:3:1:1 g 1:3:1:2

10 0

35 8

75 20

g 1:3:2:1 g 1:3:2:2

10 10

35 35

75 75

In this section, we analyse the results obtained by Algorithm 1 at the levels of intermediate criteria and root criterion. In this step, evaluations of partial pre-orders in Fig. 7 are propagated upwards the hierarchy tree to the root node, as shown in Fig. 8. Weights are

Fig. 8. Non-elementary criteria with weights.

Fig. 7. Case study of partial pre-orders obtained from E.

L. Del Vasto-Terrientes et al. / Expert Systems with Applications 42 (2015) 4910–4926

also indicated in Fig. 8. The preference threshold values for the intermediate criteria are: qj ðaÞ = 0, pj ðaÞ = 1 and v j ðaÞ ¼ 3. The case of the Santiago de Compostela destination website will be taken as example of the conclusions that the DM (i.e., the manager of the Santiago website) could obtain with the results given by ELECTRE-III-H. The results will be explained following the order in which they are obtained, according to Algorithm 1. After the calculation of a separate partial pre-order for each of the criteria ‘‘Usability’’ and ‘‘Accessibility’’, the partial concordance and discordance matrices are calculated at both sub-criteria (based on the rules defined in Section 4.2) and then aggregated into a credibility matrix at the level of ‘‘Usability and Accessibility’’, named qusability and accessibility . This process is shown in Fig. 9, where it can be seen that Stockholm outranks all of the other alternatives and Rome outranks 6 others, one with a credibility qRome;Madrid = 0.53. Moreover, Andalusia outranks many alternatives for ‘‘Usability’’, as shown in the concordance matrix, but for ‘‘Accessibility’’ it is ranked as the worst alternative and the discordance is activated, where v j ðaÞ = 3, vetoing Andalusia with respect to other alternatives and showing the non-compensation between criteria in the method. From this credibility matrix, the distillation process is used to calculate the partial pre-order for the criterion ‘‘Usability and Accessibility’’. The ‘‘Usability and Accessibility’’ partial pre-order is shown in Fig. 11.

4921

The DM can use this result to analyse not only the ranks but also the preference relations between the alternatives with respect to the ‘‘Usability and Accessibility’’ of the website. For example, in Fig. 11 we can see that overall, Santiago de Compostela (Sant) is better than Rias Baixas, incomparable to Wales and Catalonia, and worse than all the other tourist brand websites. Therefore, the manager of the Santiago de Compostela website should define and initiate some actions to improve the ‘‘Usability and Accessibility’’ in order to achieve a better overall position, taking into account that it is already performing well for the criteria ‘‘Brand Treatment’’ and ‘‘Web Visibility’’. The same procedure is applied for ‘‘Brand Treatment’’. First, we calculate the partial concordance and discordance indices from partial pre-orders obtained on the criteria ‘‘Slogan & Logotype’’ and ‘‘Brand Image’’, thus obtaining the credibility qbrand treatment (in Fig. 10). In this case, Switzerland outranks all of the other alternatives and Andalusia also outranks the rest except for Switzerland. Madrid and Rias Baixas are the alternatives with lowest outranking values. Next, another partial pre-order is obtained by distillation of this credibility matrix. Note that Santiago de Compostela is incomparable to Stockholm. Its main competitors with respect to ‘‘Brand Treatment’’ are Barcelona, Andalusia and Switzerland, which are ranked in better positions.

Fig. 9. Credibility building for Usability and Accessibility.

4922

L. Del Vasto-Terrientes et al. / Expert Systems with Applications 42 (2015) 4910–4926

Fig. 10. Credibility building for Brand Treatment.

Finally, at the root level we aggregate three partial pre-orders: ‘‘Usability and Accessibility’’, ‘‘Brand Treatment’’ and ‘‘Web Visibility’’. It should be noted that the concordance and discordance indices on the criterion ‘‘Web Visibility’’ have been calculated with classical ELECTRE-III, since they are obtained directly from elementary criteria. At the root level, the credibility matrix qgoal obtained is shown in Fig. 11. The partial pre-order generated at this node corresponds to the final assessment of the alternatives, at the most general level. The final ranking at the root criterion indicates that Barcelona is the best website overall, followed by Stockholm and Switzerland. At the intermediate criteria ‘‘Brand Treatment’’ and ‘‘Web Visibility’’, Switzerland is ranked as the best alternative; however, at ‘‘Usability and Accessibility’’ it is outranked by 4 alternatives, which affects its overall rank in the root. Stockholm, ranked 2nd overall, is the best alternative at ‘‘Usability and Accessibility’’, but at ‘‘Brand Treatment’’ it is ranked 4th. Barcelona is ranked 2nd or 3rd at all three intermediate criteria. Rias Baixas is clearly ranked as the worst website in both the overall and intermediate rankings. The Santiago de Compostela website is one of the lowest placed, ranking below 6 other websites. It is incomparable to Wales and Catalonia, so these are direct competitors that could be outranked if Santiago de Compostela improved at ‘‘Usability and Accessibility’’. This example shows that the knowledge that tourist destination managers can obtain at different levels of the hierarchy tree is hugely valuable in determining the most appropriate strategic decisions.

The following section presents a robustness analysis of ELECTRE-III-H for this case study. 5.3. Robustness analysis The results obtained with the ELECTRE-III-H method for this case study were validated with a robustness analysis. Three configuration scenarios of threshold values in the intermediate criteria are considered, including the ‘‘Central scenario’’ presented in Section 5.2 and detailed in Table 5: Note that when the preference and veto thresholds are increased, we are decreasing the strength of opposition to the assertion aSb (i.e., decreasing the discordance degree). Thus, we have defined strict and tolerant settings. In order to compare the partial pre-order results, we have assigned each alternative a ranking position according to the partial pre-order. This ranking corresponds to the position of the alternatives in the partial pre-order generated in the exploitation stage. Positions depend on the number of predecessors of each alternative in the partial pre-order (i.e., the Rank Order Value, Cj ðÞ). It is important to note that for rank ties we assigned a rank equal to the average of their positions, in ascending order of values, as applied in the Spearman correlation coefficient for rank ties. For example, if two alternatives are tied in 2nd position, the calculation of their rank position is 2þ3 ¼ 2:5. For this particular exam2 ple, the alternative ranked 3rd is then positioned 4th, and so on. We applied this calculation to normalise the ranking from 1 to 10.

L. Del Vasto-Terrientes et al. / Expert Systems with Applications 42 (2015) 4910–4926

Fig. 11. Credibility building and exploitation in goal.

Table 5 Robustness configuration scenarios for I. Scenario

qj ðaÞ

pj ðaÞ

v j ðaÞ

Strict Central Tolerant

0 0 1

0 1 2

2 3 4

Figs. 12–14 show the rankings of the alternatives in order to analyse their behaviour for each scenario. The ranking is relatively stable for the criteria ‘‘Goal’’, ‘‘Usability and Accessibility’’ and ‘‘Brand Treatment’’ in each scenario. ‘‘Web Visibility’’ has not been

Fig. 12. Goal rankings for the 3 scenarios studied.

4923

4924

L. Del Vasto-Terrientes et al. / Expert Systems with Applications 42 (2015) 4910–4926

Fig. 13. Usab. and access. rankings for the 3 scenarios studied.

Fig. 14. Brand treat. Rankings for the 3 scenarios studied.

included as it is evaluated directly with elementary criteria and is not affected by the threshold values applied for intermediate criteria. Barcelona clearly remains the best alternative overall, with the exception of the tolerant scenario, in which Barcelona and Stockholm share the best rank. Rias Baixas is clearly ranked as the worst website in all scenarios.

6. Conclusions In complex real-world multiple criteria ranking problems, a hierarchical structure of criteria facilitates more detailed analysis of preference relations in the set of alternatives at different levels of generality. In this paper, we have proposed an outranking method that extends the classical ELECTRE-III method for the case of a hierarchical structure of criteria, called ELECTRE-III-H. This method permits upward propagation of results, following the organisation of the criteria in the tree. New concordance and discordance measures have been defined to aggregate the partial pre-orders of the alternatives calculated at a lower level of the hierarchy. The proposal is based on the concept of Rank Order Value (CðÞ), which encompasses the four possible binary relations that can appear in a partial pre-order: Preference, Inverse Preference, Indifference and Incomparability fP; P ; I; Rg. This method, unlike other related proposals, generates a partial pre-order at each intermediate node of the hierarchy, thus giving the DM a solution for each subset of criteria. This has been illustrated in the case study, showing how the relation between websites can be studied from different viewpoints, as well as in a global integrated way. Other advantages following from the use of ELECTRE methodology are: the ability to handling incomparability between alternatives, while it is usually treated as indifference when using a compensatory model; the management of heterogeneous scales of measurements without requiring any

transformation technique that could cause a distortion of the information. In addition, the ELECTRE-III-H method for hierarchical structures of criteria allows the DM to design the decision model in a decomposed way, by defining locally the parameters of the preference structure (i.e., indifference, preference and veto thresholds, and relative weights). The modelisation focused on sub-goals of the main problem ease understanding of the complex problem, as well as permits the definition of a richer model. In that way, the final decision will be more adequate and closer to the user preferences than with a flat criteria model. The outranking hierarchical approach presented in this paper may be useful in diverse fields of application where the complexity of the problem requires a decomposition into multi-level subproblems. As reviewed in Section 1.1, fields related to management decisions are the most common, such as business, quality assessment or environmental management. Moreover, systems of this type can be integrated into other knowledge based systems, such as expert systems or recommender systems. Literature survey in expert systems reveals a lack of robust expert systems encompassing MCDA techniques (Jha, Kumar, Kumari, & Bepari, 2014). Generally, the application of MCDA techniques in expert systems for decision assessment relies on the use of AHP or assumes a flat criteria structure. For the case of personalised recommender systems, the use of MCDA techniques is also quite limited to basic aggregation methods, as reviewed in Valls, Moreno, and Borràs (2013). The integration of ELECTRE-III-H to a domain-specific expert system may be suitable for problems where the user is not evaluating a single object but a set of alternatives on the basis of a large set of criteria. In this case, obtaining a numerical score that summarises all the criteria is not as informative as a partial pre-order structure. For example, environmental management decisions commonly involve multiple conflicting criteria related to social, economical, health and environmental issues. The integration of ELECTRE-III-H in expert systems in this domain may be useful for the DM to understand the implications of multiple criteria evaluations referring to different sub-problems. In Chao, Del Vasto-Terrientes, Valls, Kumar, and Schuhmacher (2014), the ELECTRE-III-H method was applied to a decision support system for evaluating water allocation policies for a water stressed Mediterranean area in Northeastern Spain (Tarragona). In this paper, a simple real-world case study was presented, addressing the quality evaluation of websites for the promotion of tourist destination brands. For this problem, experts defined a hierarchical structure of criteria. ELECTRE-III-H was applied to solve the problem and a robustness analysis of the ranking results was performed with different scenarios. The robustness analysis revealed relatively stable results in the rankings for all scenarios. The main weakness of the method proposed is the definition of the threshold values for the set of intermediate and elementary criteria. The results depend, to some extend, on these parameters because they influence the construction of the preference relations. In the case study, the robustness analysis shows that the results are stable for a certain range of the parameters. Due to the reduced size of the sample, this conclusion is difficult to generalise. In addition, the interpretation of a partial pre-order structure may be difficult for some users. If that is the case, one could replace the partial pre-order by a median complete pre-order, as proposed in the classical ELECTRE-III method. However, note that some information is lost in this step (i.e., incomparabilities). The results obtained in this case study are encouraging, so future work will concentrate on testing the method with a larger number of criteria and alternatives. For future research, we also intend to propose a recommendation framework for the case of a hierarchical structure of criteria, including data pre-processing taking into account the user profile, and post-processing of the results

L. Del Vasto-Terrientes et al. / Expert Systems with Applications 42 (2015) 4910–4926

to give a more informative recommendation to the user. Another interesting point to be studied is the adaptation of an interactive tool to assist the domain expert to define the preference thresholds for non-elementary criteria. Some kind of background domain knowledge could be used in this task. Finally, an extension of the method for the case of group decision making (more than one stakeholder) would also be worth to study. In group decision making, an additional stage of consensus is required. Usually this consensus is achieved in result of iterative process of analysis of several recommendations provided by the MCDA method, until the different members accept one. In each iteration, the parameters of the model may be changed or some alternatives can be added or discarded. This is certainly an interesting line of future research. Acknowledgements This project has been funded by the Spanish research project SHADE (TIN-2012-34369: Semantic and Hierarchical Attributes in Decision Making). The first author is supported by a FI predoctoral Grant from Generalitat de Catalunya. The research of the last two authors has been supported by the Polish National Science Centre. The authors want to thank the collaboration of the experts in the case study, from the project CODETUR (CSO 2011-22691) funded by the Spanish Government. We especially thank Dr. Assumpció Huertas and Dr. José Fernández Cavia. We thank Moises Griño for his work in some programming tasks. References Abedi, M., Torabi, S. A., Norouzi, G. H., & Hamzeh, M. (2012). ELECTRE III: A knowledge-driven method for integration of geophysical data with geological and geochemical data in mineral prospectivity mapping. Journal of Applied Geophysics, 87, 9–18. Ahsan, M., & Bartlema, J. (2004). Monitoring healthcare performance by analytic hierarchy process: A developing-country perspective. International Transactions in Operational Research, 11, 465–478. Arbenz, P., Hummel, C., & Mainik, G. (2012). Copula based hierarchical risk aggregation through sample reordering. Insurance: Mathematics and Economics, 51, 122–133. Arondel, C., & Girardin, P. (2000). Sorting cropping systems on the basis of their impact on groundwater quality. European Journal of Operational Research, 127, 467–482. Aydin, S., Kahraman, C., & Kaya, I. (2012). A new fuzzy multicriteria decision making approach: An application for european quality award assessment. KnowledgeBased Systems, 32, 37–46. Bana e Costa, C. A., & Vansnick, J.-C. (2008). A critical analysis of the eigenvalue method used to derive priorities in AHP. European Journal of Operational Research, 187, 1422–1428. Bobylev, N. (2011). Comparative analysis of environmental impacts of selected underground construction technologies using the analytic network process. Automation in Construction, 20, 1030–1040. Botti, L., & Peypoch, N. (2013). Multi-criteria ELECTRE method and destination competitiveness. Tourism Management Perspectives, 6, 108–113. Buyukozkan, G., Çifçi, G., & Guleryuz, S. (2011). Strategic analysis of healthcare service quality using fuzzy AHP methodology. Expert Systems with Applications, 38, 9407–9424. Chang, B., Kuo, C., Wu, C.-H., & Tzeng, G.-H. (2015). Using fuzzy analytic network process to assess the risks in enterprise resource planning system implementation. Applied Soft Computing, 28, 196–207. Chao, T. C., Del Vasto-Terrientes, L., Valls, A., Kumar, V., & Schuhmacher, M. (2014). A hierarchical decision support system to evaluate the effects of climate change in water supply in a mediterranean river basin. In L. M. Cabedo, O. Pujol, & N. Agell (Eds.), Artificial intelligence research and development – Recent advances and applications, CCIA 2014, October 2014, Barcelona, Catalonia, Spain (pp. 77–86). Colson, G. (2000). The OR’s prize winner and the software ARGOS: How a multijudge and multicriteria ranking GDSS helps a jury to attribute a scientific award. Computers and Operations Research, 27, 741–755. Corrente, S., Greco, S., Kadzinski, M., & Slowinski, R. (2013a). Robust ordinal regression in preference learning and ranking. Machine Learning, 93, 381–422. Corrente, S., Greco, S., & Slowinski, R. (2012). Multiple criteria hierarchy process in robust ordinal regression. Decision Support Systems, 53, 660–674. Corrente, S., Greco, S., & Slowinski, R. (2013b). Multiple criteria hierarchy process with ELECTRE and PROMETHEE. Omega, 41, 820–846. Damaskos, X., & Kalfakakou, G. (2005). Application of ELECTRE III and DEA methods in the BRP of a bank branch network. Yugoslav Journal of Operations Research, 15, 259–276.

4925

Dujmovic´, J., & Tré, G. D. (2011). Multicriteria methods and logic aggregation in suitability maps. International Journal of Intelligent Systems, 26, 971–1001. Fernández-Cavia, J. C. R., Gómez, L., Gascó, M., Pedraza, R., Bonilla, S., Villaescusa, E., Luque, P.D., Huertas, A., Torres, T., Lozano, N., de San Eugenio, J., Míguez, M. I., Sicilia, M., & Barbero, S. M. (2010). Propuesta de diseño de una plantilla multidisciplinar para el análisis y evaluación de webs de destinos turísticos. In Proceedings of TURITEC-2010 (pp. 333–351). Málaga, Spain. Fernández-Cavia, J., & Huertas-Roig, A. (2009). City brands and their communication through web sites: Identification of problems and proposals for improvement. In M. Gascó-Hernandez & T. Torres-Coronas (Eds.), Information communication technologies and city marketing: Digital opportunities for cities around the world (pp. 26–49). IGI Global. Figueira, J. R., Greco, S., & Ehrgott, M. (2005). Multiple criteria decision analysis: State of the art surveys. Multiple criteria decision analysis: State of the art surveys (Vol. 8, pp. 859–890). Springer. Figueira, J. R., Greco, S., Roy, B., & Slowinski, R. (2013). An overview of ELECTRE methods and their recent extensions. Journal of Multi-Criteria Decision Analysis, 20, 61–85. Figueira, J., & Roy, B. (2002). Determining the weights of criteria in the ELECTRE type methods with a revised Simos’ procedure. European Journal of Operational Research, 139, 317–326. Hatch, K., Dragic´evic´, S., & Dujmovic´, J. (2014). Logic scoring of preference and spatial multicriteria evaluation for urban residential land use analysis. In M. Duckham, E. Pebesma, K. Stewart, & A. U. Frank (Eds.), Geographic information science. Lecture notes in computer science (8728, pp. 64–80). Springer International Publishing. Hsu, T. H., & Pan, F. F. C. (2009). Application of Monte Carlo AHP in ranking dental quality attributes. Expert Systems with Applications, 36, 2310–2316. Ishizaka, A., & Labib, A. (2011). Review of the main developments in the analytic hierarchy process. Expert Systems with Applications, 38, 14336–14345. Jha, N., Kumar, R., Kumari, A., & Bepari, B. (2014). Design, development and implementation of a robust decision support expert system (branDEC) in multi criteria decision making. Procedia engineering, 12th Global congress on manufacturing and management, 97 (pp. 1853 – 1865). Kilic, H. S., Zaim, S., & Delen, D. (2015). Selecting the best ERP system for SMEs using a combination of ANP and PROMETHEE methods. Expert Systems with Applications, 42, 2343–2352. Law, R., Qi, S., & Buhalis, D. (2010). Progress in tourism management: A review of website evaluation in tourism research. Tourism Management, 31, 297–313. Mardani, A., Jusoh, A., & Zavadskas, E. K. (2015). Fuzzy multiple criteria decisionmaking techniques and applications two decades review from 1994 to 2014. Expert Systems with Applications, 42, 4126–4148. Mendis, B. S. U., & Gedeon, T. D. (2012). Complex structured decision making model: A hierarchical frame work for complex structured data. Information Sciences, 194, 85–106. Muerza, V., De Arcocha, D., Larrodé, E., & Moreno-Jiménez, J. (2014). The multicriteria selection of products in technological diversification strategies: An application to the spanish automotive industry based on AHP. Production Planning and Control, 25, 715–728. Mustajoki, J. (2012). Effects of imprecise weighting in hierarchical preference programming. European Journal of Operational Research, 218, 193–201. Nordstrom, E. M., Eriksson, L. O., & Ohman, K. (2010). Integrating multiple criteria decision analysis in participatory forest planning: Experience from a case study in northern Sweden. Forest Policy and Economics, 12, 562–574. Papadopoulos, A., & Karagiannidis, A. (2008). Application of the multi-criteria analysis method ELECTRE III for the optimisation of decentralised energy systems. Omega, 36, 766–776. Pijuan, J., Valls, A., Passuello, A., & Schuhmacher, M. (2010). Evaluating the impact of sewage sludge application on agricultural soils. In XV Congreso español sobre tecnologías y lógica fuzzy (ESTYLF) (pp. 369–374). Huelva, Spain. Reddy, B., Kelly, M., Thokala, P., Walters, S., & Duenas, A. (2014). Prioritising public health guidance topics in the national institute for health and care excellence using the analytic hierarchy process. Public Health, 128, 896–903. Rogers, M., & Bruen, M. (1998). Choosing realistic values of indifference, preference and veto thresholds for use with environmental criteria within ELECTRE. European Journal of Operational Research, 107, 542–551. Roy, B. (1996). Multicriteria methodology for decision aiding. Dordrecht: Kluwer Academic Publishers. Roy, B., & Slowinski, R. (2013). Questions guiding the choice of a multicriteria decision aiding method. EURO Journal on Decision Processes, 1, 69–97. Rudas, I. J., Pap, E., & Fodor, J. (2013). Information aggregation in intelligent systems: An application oriented approach. Knowledge-Based Systems, 38, 3–13. Saaty, R. (1987). The analytic hierarchy process-what it is and how it is used. Mathematical Modelling, 9, 161–176. Sánchez-Lozano, J. M., Teruel-Solano, J., Soto-Elvira, P. L., & García-Cascales, M. S. (2013). Geographical information systems (GIS) and multi-criteria decision making (MCDM) methods for the evaluation of solar farms locations: Case study in south-eastern Spain. Renewable and Sustainable Energy Reviews, 24, 544–556. Shanian, A., Milani, A. S., Carson, C., & Abeyaratne, R. C. (2008). A new application of ELECTRE III and revised Simos’ procedure for group material selection under weighting uncertainty. Knowledge-Based Systems, 21, 709–720. Shen, Y., Hermans, E., Brijs, T., & Wets, G. (2012). Data envelopment analysis for composite indicators: A multiple layer model. Social Indicators Research, 114, 739–756.

4926

L. Del Vasto-Terrientes et al. / Expert Systems with Applications 42 (2015) 4910–4926

Torra, V., & Narukawa, Y. (2007). Modeling decisions: Information fusion and aggregation operators. Springer. Torres-Salinas, D., Moreno-Torres, J. G., Delgado-López-Cózar, E., & Herrera, F. (2011). A methodology for institution-field ranking based on a bidimensional analysis: The IFQ2A index. Scientometrics, 88, 771–786. Valls, A., Moreno, A., & Borràs, J. (2013). Multicriteria decision aid and artificial intelligence. In M. Doumpos & E. Grigoroudis (Eds.), Multicriteria decision aid and artificial intelligence chapter preference representation with ontologies (pp. 77–99). John Wiley & Sons. Valls, A., Pijuan, J., Schuhmacher, M., Passuello, A., Nadal, M., & Sierra, J. (2010). Preference assessment for the management of sewage sludge application on agricultural soils. Multicriteria Decision Making, 1, 4–24.

Vanderpooten, D. (1990). The construction of prescriptions in outranking methods. In C. Bana e Costa (Ed.), Readings in multiple criteria decision aid (pp. 184–215). Berlin: Springer. Wang, G., Huang, S. H., & Dismukes, J. P. (2004). Product-driven supply chain selection using integrated multi-criteria decision-making methodology. International Journal of Production Economics, 91, 1–15. Xu, B., & Ouenniche, J. (2012). Performance evaluation of competing forecasting models: A multidimensional framework based on MCDA. Expert Systems with Applications, 39, 8312–8324. Yang, C. L., Chuang, S. P., & Huang, R. H. (2009). Manufacturing evaluation system based on AHP/ANP approach for wafer fabricating industry. Expert Systems with Applications, 36, 11369–11377.