Supporting the intelligent MCDA user: A case study in multi-person multi-criteria decision support

European Journal of Operational Research 160 (2005) 172–189 www.elsevier.com/locate/dsw

Supporting the intelligent MCDA user: A case study in multi-person multi-criteria decision support Julie Hodgkin b

a,*

, Valerie Belton b, Anastasia Koulouri

b

a Department of Computing and Mathematics, University of Stirling, Stirling FK9 4LA, UK Department of Management Science, University of Strathclyde, 40 George Street, Glasgow G1 1QE, UK

Available online 19 May 2004

Abstract This paper stems from the conjunction of two research studies, the first investigating the provision of Intelligent User Support for multi-criteria decision analysis, the second seeking to provide decision support to small and medium enterprises in the Glasgow area. It describes two softwares for analysis and visual interactive display (AVIDs) designed to support expert MCDA users by extending the power of analysis of existing tools to enable the users to quickly make sense of complex multi-user, multi-criteria evaluations, possibly in the stressful and time pressured environment of a decision workshop. The use and subsequent evaluation of these softwares is described in the context of a study to explore potential futures, in the light of conflicting pressures, for a daycare centre for adults with cerebral palsy.
2004 Elsevier B.V. All rights reserved. Keywords: Multiple criteria decision analysis; Visual interactive modelling; Group decisions; Robustness and sensitivity analysis

1. Introduction The work discussed in this paper is part of a broader study investigating the potential for and benefits of ‘‘Intelligent User Support’’ in the context of multiple criteria decision analysis (MCDA). The foundations for this work were described by Belton and Hodgkin (1999), who highlight the need when designing such support to be clear about the knowledge and experience of the intended users and about the context of use. Following this, two streams of investigation evolved. *

Corresponding author. E-mail addresses: [email protected] (J. Hodgkin), [email protected] (V. Belton).

The first, which focused on the provision of support to ‘‘na€ıve’’ users (with respect to their knowledge of the underlying methodology and their familiarity with the supporting software), is described in part in Hodgkin et al. (2000). The second, concerned with the provision of support for ‘‘expert’’ users (i.e. knowledgeable about and experienced in the use of the method and supporting software), is the focus of this paper. The expert user may be facilitating a workshop involving a group of decision makers or stakeholders, they may be working off-line to support such a workshop, or they may be acting in an advisory capacity doing backroom analysis. It is assumed that they would be making use of one of the many generic MCDA softwares that are

0377-2217/$ - see front matter
2004 Elsevier B.V. All rights reserved. doi:10.1016/j.ejor.2004.03.007

J. Hodgkin et al. / European Journal of Operational Research 160 (2005) 172–189

available. Such users would not be expected to need support in structuring the problem, building an MCDA model, eliciting values, or in using the software of choice to capture this information, analyse it and display results. However, a comprehensive analysis requires extensive sensitivity/ robustness analysis which can be time-consuming, and if conducted under pressure in a workshop environment could possibly overlook important issues. Potential difficulties increase as the size of the problem increases––more options being examined, or a more detailed and complex set of criteria being used. If the process is one that initially seeks individual assessments, the complexity is magnified, according to the size of the group. Although many of the generic MCDA softwares provide visual interactive tools that facilitate sensitivity analysis and communication of results, there is still an onus on the facilitator to ensure that important issues are highlighted and adequately explored. The paper describes two softwares for analysis and visual interactive display (AVID), which were designed to provide additional support to facilitators or expert users of multi-attribute value function (MAVF models)––that is, to go beyond the usual facilities incorporated in generic softwares. The main objective in designing the softwares was to complement existing tools in providing an effective overview and highlighting important aspects of a problem. This can reduce stress on the facilitator in a workshop environment, providing reassurance that no potentially significant issues have been overlooked. In an off-line or backroom environment the same software enables the facilitator to carry out a more thorough analysis in the limited time available. In both instances this would ultimately also benefit the decision makers (DMs) by ensuring that the analysis is comprehensive and makes effective use of valuable time. The opportunity to use and evaluate these softwares arose as part of work with a small charitable organisation that provides care for adults with cerebral palsy. This intervention was itself part of a larger project, funded by Scottish Enterprise Glasgow (formerly Glasgow Development Agency), to provide decision support to small and medium enterprises in the Glasgow area. The case study is described in detail, thereby en-

173

abling us to demonstrate to MCDA facilitators the power of enhanced analytic tools and visual displays in supporting their part in an intervention as well as a means of communicating analysis to decision makers. In particular, we illustrate the use of the tools to facilitate the comparison of multiperson, multi-criteria data (without resorting to simplification by aggregation). In the case study this challenging task was done off-line, enabling the facilitators to choreograph an effective decision workshop involving the decision makers. Having demonstrated the value and success of the tools in this way, it opens up the way for more effective multi-user MCDA (both face-to-face and distributed), which allows for simultaneous independent evaluation of options by multiple decision makers. The structure of the paper is as follows. In Section 2 we describe the background to the case study and in Section 3 outline the softwares developed, illustrating the facilities provided through reference to the case study. In Section 4 we go on to describe in greater detail how the software was used and how it facilitated the process of analysis, reflecting on this in more depth in Section 5 as part of a formal process of evaluation. We conclude with a summary of the study and the learning derived from it, together with comments on directions for further research.

2. Background to the case study 2.1. Scottish Enterprise Glasgow and Glasgow Decision Support Project Scottish Enterprise Glasgow’s (SEG) aim is to promote economic growth within the city. Part of the support offered is the Small Business Development Service, which provides a range of assistance from start-up advice through to implementation of the Business Excellence model. The Glasgow Decision Support Project (GDSP), in conjunction with the Management Science Department at the University of Strathclyde, was initiated to provide support to such companies facing complex issues and difficult decisions. Its stated aim is ‘‘to help organisations in the Glasgow area make key business decisions by providing

174

J. Hodgkin et al. / European Journal of Operational Research 160 (2005) 172–189

access to leading edge decision support methods and expert advice on the process’’ (http:// www.mansci.strath.ac.uk/GDSP.html). The Rotary Residential Care Centre (RRCC) was one of the organisations to draw on this support; the problem it faced and a brief outline of the overall intervention are now described. 2.2. Problem description At the time of the study the client organisation owned and operated a residential unit, which was home for 11 adults with cerebral palsy, and a day centre, which provided facilities for these and other clients. Broadly, the issue facing the organisation was how these facilities should be developed in the future in order to best meet the needs and preferences of the clients and their carers at the same time as responding to pressures from the Social Work Department. The full study, which is described in detail elsewhere (Belton and Koulouri, in preparation) began with consideration of the broad issues facing the organisation, but following the first workshop it was decided that the residential unit and the day centre could be considered independently. It is the future of the day centre that is the focus of the case study described here. The GDSP team, comprising two facilitators and an observer, ran a total of six workshops with representatives of RRCC. As mentioned above, the first of these was concerned with problem structuring––understanding the issues involved, including the options open to the organisation and the factors that they felt to be important in guiding decision making. Workshops 2, 3, 4 and part of 5 focused on the evaluation of options for the residential unit, using a multi-attribute value function framework. Workshops 5 and 6 then moved on to consider alternative futures for the day centre and it was in conjunction with these that the AVID softwares were used. Each workshop involved a number of key stakeholders, including senior managers of the facilities, who are full time employees, and members of the Executive Committee, who are volunteers. It became apparent in the first workshops that the limited time available would not permit the whole analysis to be conducted in a ‘‘sharing’’

mode (Belton and Pictet, 1997) whereby a model is built, values are elicited and outcomes explored with the group as a whole. Thus a process which incorporates both ‘‘sharing’’ and ‘‘comparing’’ modes of working was developed. Each phase of the problem was initially structured as a group (a value tree defined and options identified). Scores were then provided by each participant, using customised scoresheets, in advance of the next workshop. The facilitators synthesised and carried out an exploratory analysis of this data in preparation for the subsequent workshop, during which they would review individual evaluations, compare these, and illustrate key sensitivities and robustness with a view to reaching consensus on the next steps. The process of exploratory data analysis, which was the basis for defining an agenda for the workshops, was found to be both time consuming and challenging. Aspects of this phase included: assimilation of the data provided by the individuals: getting a sense of each person’s evaluation of the options and assessing the robustness of the implied overall preferences: comparing and contrasting this information across participants, and so on. One of the facilitators, who was involved with the development of the AVID softwares, suggested that they could usefully support this exploratory analysis. The 2nd facilitator was introduced to the software using one of the problems considered in an earlier workshop to illustrate the process and potential. The systems were then used to support the analysis of alternative futures for the day centre. The day centre operated in a three-story townhouse, which was acquired by RRCC in 1958. However, only the ground floor was accessible to clients and the upper floors were not in use at the time of the study as a previous tenant had recently moved out. A variety of possible options were available to RRCC; these included continuing with the current operation, sale of the current centre, purchase or lease of new facilities, renting space in another community centre with appropriate facilities for the clients, a focus on outreach activities with or without a centre-based activity and complete withdrawal from the provision of day care. Nine specific options were defined. The value tree used for the evaluation of the options can be seen

J. Hodgkin et al. / European Journal of Operational Research 160 (2005) 172–189

175

Fig. 1. Full value tree.

in Fig. 1, which also gives the full scoring of the options by one of the decision makers and illustrates other aspects of the VISA displays that will be discussed later. The next section introduces the AVID softwares, first outlining the motivation for their development, and then using data from the case study to illustrate the nature of output provided. Section 4 goes on to discuss the actual use in the case study in greater detail.

3. Softwares for analysis and visual interactive display 3.1. Introduction In our view, the aim of MCDA is not to find the ‘‘right answer’’ to a problem. Rather, it is to facilitate decision makers in a process of learning about an issue and about their own and other stakeholders’ perspectives on and preferences relating to that issue. A MCDA model should provide an effective way of reflecting back and

synthesising judgments made, in a manner which aids the decision maker in understanding how these support different potential decisions. It should act as a catalyst for thinking, a sounding board against which intuition can be tested, and, as already mentioned, a vehicle for learning. Initial evaluations provided by a model should be seen not as the end of the analysis but as one stage in a process that continues with an extensive sensitivity or robustness analysis to further understanding (Belton and Stewart, 2002). However, the multi-dimensional complexity of a problem poses challenges in all these respects; what is the best way of achieving this? In particular, extensive sensitivity analyses are both time-consuming and difficult to communicate in a simple manner. Extensive work with decisionmaking groups over more than 20 years confirms our view that, for the majority of people, visual interactive displays are the most powerful means of communication. Displays must be easy to interpret, recognising that this is something that is dependent on the skills of the user. Our starting point was the existing software, VISA (Belton and Vickers, 1990) and Group VISA,

176

J. Hodgkin et al. / European Journal of Operational Research 160 (2005) 172–189

visual interactive softwares to support analysis using a multi-attribute value function (MAVF). Although we choose to work with the MAVF approach, and have done so for many years (see, for example, Belton, 1985, 1993), our use of this is informed by extensive knowledge of alternative approaches and an awareness of the complementary strengths of different approaches (Belton, 1986; Pictet and Belton, 2001; Belton and Stewart, 2002), which we draw on in our practice as appropriate. Visual interactive tools such as VISA and the ‘‘walking weights’’ in PROMETHEE (Brans et al., 1986) provide effective means for a user to explore sensitivity in an ad-hoc way, instantly seeing the effect of changes. However, systematic multidimensional sensitivity analysis is not well supported by these facilities and important features of a complex model may be overlooked, particularly in the more stressed and time-constrained environment of a decision workshop. A number of authors have discussed the need for, and value of, more comprehensive and systematic sensitivity analysis; for example: Rios Insua and French (1991) proposed a framework for sensitivity analysis in MCDA based on mathematical programming; Triantaphyllou and Sanchez (1997) introduce the concept of criticality of criteria and performance measures: Butler et al. (1997) propose a simulation based approach. The aim in developing the AVID softwares was to explore ways of enhancing the power of software for MCDA through the incorporation of more extensive and effective facilities for visualisation. The initial focus was on supporting the process of sensitivity and robustness analysis, with a focus on the expert user, but as the case study will illustrate the designed displays turned out to be of broader value. A review of techniques for the display of multidimensional data, within the field of MCDA and more generally, reveals many approaches. Focusing on displays that support the exploration of multi-dimensional data and of the variability in weights assigned to criteria in a MAVF analysis, we found it useful to group these in three categories. Firstly, approaches which seek to retain all information and to display this in some manner. The VISA profile graphs seen in Fig. 1 are one example of this. Others are Chernoff faces (Cher-

noff, 1973); Korhonen Houses (Korhonen, 1991), the use of radar plots as seen in Gestalt (Kasanen et al., 1989), Triple C’s sector graphs (Angehrn, 1991), and Star and Petal diagrams (Tan and Fraser, 1998). A second approach is to reduce the dimensionality of the data. Multi-variate statistical analyses provide a vehicle for this and have been employed in MCDA in the GAIA software (Mareschal and Brans, 1988) and by Stewart (1981) and in group decision support by Losa et al. (2001). A third approach is to focus on the outcome of an analysis rather than the input data and displays of sensitivity analyses fall into this category. The simplest form is the classic one-dimensional sensitivity graph incorporated in most softwares for MAVT (see Belton and Stewart, 2002, p. 149 or Butler et al., 1997, p. 533). If working with threedimensions a triangle can be used to represent weight space (a method used by Climaco and Antunes (1989) in the TRIMAP software and by Vetschera (1997) in his work on volume-based sensitivity analysis) and to display the performance of options in different regions of that space. Once again, this approach is illustrated in the context of MAVT by Belton and Stewart (2002, pp. 150–151) and Butler et al. (1997, pp. 534–535). Higher dimensional sensitivity analyses are both more time consuming to conduct and more difficult to communicate. Some analyses and displays focus on options, for example: potential optimality analyses (Hodgkin et al., 2000) or stability analyses (Wolters and Mareschal, 1995) show the change in criteria weights necessary to change the status of an option: cats whisker diagrams (Butler et al., 1997; Butler and Olson, 2000) show the range of rank attained by each option across the weight space (determined by simulation). As indicated above, the key driver in this work was to find more powerful and systematic means to support sensitivity analysis in a MAVT framework. Such analysis should be transparent, it should not distort or misrepresent the original data, and it should be amenable to visual display in a manner that is easy for the intended user to interpret. As VISA already incorporated facilities in the first category above we sought to extend its use-

J. Hodgkin et al. / European Journal of Operational Research 160 (2005) 172–189

fulness through the second and third approaches. This led us to focus on two types of analysis and associated display. From the second category above we identified Principal Components Analysis (PCA) as a potentially powerful tool and from the third category chose to explore the use of the triangular display to capture three key dimensions, as many of the ‘‘real world’’ analyses we have been involved in have focused around three high level criteria (see for example, Belton, 1993; Belton and Pratt, 1999). The two systems that evolved are now presented.

3.2. Principal components analysis-plot (PCAPlot) PCA is a non-parametric multi-variate statistical technique used to reduce the dimensionality of data (see, for example, Everitt and Dunn, 2001). The first principal component (defined by the eigenvector corresponding to the largest eigenvalue of the covariance matrix) defines a linear combination of the variables that accounts for as much as possible of the variation in the original data. The second principal component, orthogonal to the first, accounts for as much as possible of the remaining variability. PCA is useful only if there is some degree of correlation in the initial data, which has been the case in the majority of the MCDA models we have examined in our studies (Hodgkin, 2000). The input to the PCA is the matrix defining the performance of the options on each criterion, i.e. the scores elicited for the MAVF model. The visual output is a bi-plot having the two principal components as axes. This displays the position of the options, defined by their vector of scores, and the position of each criterion, defined by an identity vector. Relationships between the criteria and similarities between options may be discerned from the plot. A vector of criteria weights can also be shown in the bi-plot and our initial interest in the analysis and display was as a means of conveying the impact in variations in this vector on the preferred option. To illustrate the use of the software, let us take the preference information given by one of the decision makers involved in the case study (DM1);

177

this is the evaluation seen in Fig. 1 (options are evaluated against each criterion on a 0–100 scale, as seen in the window at the bottom for the figure). The bi-plot resulting from the PCA analysis of this data is seen in Fig. 2. The PCA-Plot depicts options (A–I), and criteria (1–9). The thick line stemming from the origin depicts the current weight vector, indicating the direction of the currently preferred option. The angle between two criteria lines indicates the degree of correlation between the scores for these criteria. From the plot we can see that the evaluations for criteria 2, 3 and 4 are highly correlated, as are those for criteria 5 and 6 (which are also similar to criteria 2–4). Criterion 1 is similarly oriented. Criteria 7 and 8 are also highly correlated but are in conflict with criteria 1–6. Criterion 9 appears to be different from all others. The closeness of options indicates their similarity and we see that options C, F, G and H are tightly clustered, performing well on criteria 1–4. Option B is highly differentiated from the others, as are E and I but to a lesser extent. The weight vector is aligned with criteria 2–4, suggesting that options which lie in this direction are preferred. However, note that the values assigned here, which appear across the bottom of the figure, are simply illustrative as decision makers were not asked to specify weights at this stage. In some circumstances it may be useful to see the impact of uncertainty about the value of the weights on the position of the weight vector. The black shaded area around the end of the weight vector shows how its position would change if the weights were allowed to vary by the amount shown by the hatched bars towards the bottom of Fig. 2 (approximately ±0.1). 3.3. ‘Triangle’-plot (M-Plot) The M-Plot software reveals the regions of weight space for which an option is preferred, which we denote as ‘‘preference regions’’. The multi-attribute value function model on which we base our analysis is defined as follows: X X V ðAÞ ¼ wi vi ðAÞ i ¼ 1 to N ; wi ¼ 1; i

i

178

J. Hodgkin et al. / European Journal of Operational Research 160 (2005) 172–189

Fig. 2. PCA-Plot.

where V ðAÞ is the overall value of option A, vi ðAÞ is the performance of option A on criterion i, wi is the weight (scaling factor) assigned to criterion i. A full exposition of the model and its use can be found in Belton and Stewart (2002). Thus, the preference region for option A is formally defined as follows: An option, A, is potentially preferred/optimal if there exist values of the weights, wi , such that V ðAÞ P V ðBÞ for all other options, B. The set of weights for which this holds, which define a convex sub-region in the multidimensional weight space, is denoted the preference region for A. The three-dimensional weight space can easily be displayed in two-dimensions as a consequence

of the normalisation constraint that requires weights to sum to 1. Each vertex corresponds to all weight being allocated to a single criterion. The preference regions are determined by linear programming. For each option, A, we first determine if it is potentially preferred, by solving the following linear program: X Max wi subject to:

V ðBÞ  V ðAÞ 6 0 for all other options; B; X wi ¼ 1:

Note that this LP simply determines if there is a feasible set of weights, satisfying the requirement of summing to unity, such that V ðAÞ P V ðBÞ. To achieve this any P dummy objective function of the form Min=Max ci wi ––for any values of ci ,––

J. Hodgkin et al. / European Journal of Operational Research 160 (2005) 172–189

may be used. The existence of a feasible solution demonstrates that option A is potentially preferred. For each potentially preferred option we then determine the vertices and adjoining edges of its preference region, as follows. Start with the initial solution to the above LP––this defines the initial set of vertices of the preference region; use the simplex method to determine all adjacent optimal solutions––add these solutions to the set of vertices; determine all adjacent optimal solutions to the new members of the set of vertices; continue until no new vertices are found. The procedure is described in detail in Hodgkin (2000). If the decision maker wishes to address a multicriteria problem that involves more than three criteria, then the M-Plot can be used to show weight space represented by three of the criteria

179

whilst fixing the weights of any remaining criteria relative to each other and in total. The decision maker can then interactively investigate the effect of increasing/decreasing the total value of the fixed weights. Essentially, this allows the decision maker to take slices through the tetrahedron that is the result of extending this display to three-dimensions representing 4 criteria. Fig. 3 shows the M-Plot derived from scores aggregated to the 4 top-level criteria in the value tree. The triangle displays the weight space determined by Clients, Organisational Issues and Funding Issues. The weight allocated to the fourth criterion, Carers’ View, is shown in the slide bar on the right. This can be interactively varied between 0 and 1. The black dot indicates an equal allocation of weights between the three criteria represented in the triangle.

Fig. 3. M-Plot.

180

J. Hodgkin et al. / European Journal of Operational Research 160 (2005) 172–189

The M-Plot shows that the only options that are potentially preferred are D, E and H. As the weight on Carers’ View is varied the preference region for option E grows a little larger (with decreased weight on Carers View) or disappears (with increased weight on Carers View), with the distribution of space allocated to options D and H remaining in the same proportion and pattern. No other options enter the plot. This result may be initially surprising if we refer back to the evaluations displayed in Fig. 1. The top bar chart shows the overall scores for options assuming the allocation of weights described and we can see that option E is ranked in 7th place. Four options, which receive higher overall scores, are not potentially preferred. It is evident that the preferences for option D and option H are robust across a wide range of criteria weights, with the latter being preferred over the former if the majority of the weight is allocated to Client considerations. It should be noted, however, that the plot does not show how much better D and H are compared to the options that are ranked 2nd or 3rd. Referring back to Fig. 1 we can see that options C, F and G also score highly, although, as commented above, they are not potentially optimal. We will return to this issue in discussion of the case study. The bar chart at the bottom of the screen allows the user to interactively change the current weight allocation between the three selected criteria and see the new position in weight space. This facility also allows the user to incorporate imprecision in the weight values; the shaded area to the centre of the plot shows the region in weight space covered by a variation in each of the three weights of ±0.1, 0.1 and 0.15. This area is contained within the preference region for option D, indicating that this is a robust choice if the user is confident that these bounds capture the range of acceptable trade-offs. If the weights are constrained in any way then the area of weight space that is ‘‘out of bounds’’ is easily illustrated. In the next section we describe how the new softwares were used in practice, in conjunction with Group VISA, to analyse multi-decision maker, multi-criteria evaluations of the 9 options in preparation for a decision workshop.

4. Case study As indicated earlier, the future of the day centre was considered in two workshops. The first of these, which was concerned with model building, agreed the value tree presented in Fig. 1 and a list of nine options for evaluation. Four representatives of the organisation were involved in the evaluation (three of these had been involved in the earlier workshops looking at the residential home, the fourth had specific responsibilities relating to the day centre) and each completed a scoresheet in advance of the second workshop. These customised scoresheets required the evaluators to use a visual analogue scale to position each option relative to ‘‘Status Quo’’ for each of the nine bottomlevel criteria in the value tree. These scores were first entered into Group VISA, the software that had been used for the exploratory data analysis in the earlier workshops. Group VISA allows the user to analyse the problem from the perspective of an individual participant and to compare results across participants at any node of the tree. Fig. 4 illustrates part of an analysis from the perspective of one of the participants, showing how the options are evaluated with respect to the sub-criteria of ‘‘Clients’’ (seen in the profile graph on the right of the display) and how these values are aggregated using the criteria weights (shown in the lower bar chart) to indicate performance with respect to ‘‘Clients’’ (seen in the bar chart to the top left). Fig. 5 shows the comparison of individual evaluations of the options aggregated to the level of ‘‘Clients’’ together with a detailed comparison of the evaluation of a selected option, E, on the sub-criteria of ‘‘Clients’’. The top chart (Fig. 5A) shows the aggregated score on ‘‘Clients’’, for each of the four decision makers (represented by four different shapes), for the nine options. There are some evident patterns and apparently common views reflected in this display. For example the decision maker represented by a star has the most ‘‘extreme’’ ratings for options C–I and all decision makers rate option B considerably lower that the status quo, option A. The second display (Fig. 5B), drills deeper into the information to explore possible reasons for some of the differences. It shows

J. Hodgkin et al. / European Journal of Operational Research 160 (2005) 172–189

181

Fig. 4. Individual evaluation.

Fig. 5. Comparing individual evaluations: (A) overall evaluation of all options, (B) evaluation of option E on sub-criteria of ‘‘clients’’.

the evaluation of option E, by the four decision makers, on the three sub-criteria of ‘‘Clients’’. It is interesting to note here that the scores of the evaluator represented by a star stand out as significantly different to those of the others. This may be an issue, which is worth pursing in the workshop. As already discussed, the aim of the exploratory analysis was for the facilitators to achieve a level of understanding of the individual evaluations and the comparison between these which enabled them to define a workshop agenda which would bring a

common understanding to the participants and enable them to agree on a way forward. This called for investigation of issues such as: • How each participant evaluated the options–– which ones are apparently preferred, which ones are highly rated, which ones are dominated or near-dominated, etc. • Which options are ‘‘potentially preferred’’ for each individual. • Are these individual evaluations robust or sensitive to the weights assigned to criteria.

182

J. Hodgkin et al. / European Journal of Operational Research 160 (2005) 172–189

• Where the individual evaluations differ from each other and where they are similar. • Whether differences follow a consistent pattern (for example one individual might use more extreme ratings than another) or a random one. • If it looks possible that a consensus ‘‘solution’’ might emerge. Although it is possible to answer all of these questions using Group VISA, doing so is a time consuming and painstaking task, even for a problem of this relatively small magnitude. The issue we wished to explore was whether or not the AVID softwares could facilitate the exploration. First of all we wished to get a sense of whether or not the four decision makers had scored the options against the bottom-level criteria in a similar way. Profile graphs showing the performance of the 9 options, across all 9 bottom-level criteria, for each of the individuals, offer one way of doing this. However, this is too much information to assimilate in this form at the same time. Furthermore, the format of the display is influenced by the order of presentation of the criteria, and patterns may be highlighted or overlooked simply as a consequence of this (for example, if two criteria on which the scores of the options are highly correlated are juxtaposed, then this will be apparent, but if they are separated in the display then it would not be so obvious). The PCA-Plot is a powerful means of ‘‘condensing’’ the data enabling us to see at a glance for each individual: which criteria are correlated and which are conflicting: the ‘‘position’’ of options with respect to criteria and which options are similarly positioned. Comparing the PCA plots for the 4 DMs provided a more immediate and holistic way to see how these basic evaluations compared. However, initial comparison of the four plots revealed one to be substantially different to the others, reflecting a very different pattern of scores. This belonged to the individual who had not been involved in the earlier workshops and on following this up it emerged that they had misunderstood the scoring process. This was clarified and a new scoresheet was completed. The four plots are now shown in Fig. 6A–D.

Several common features emerge from these plots. The key ones we identified were: • Criteria 2 and 3 are closely related for all DMs, as are criteria 1 and 4 (for DMs 1 and 4 all four criteria are closely related). These are the four criteria relating to quality of care. • Criteria 5 and 6, relating to organisational issues, are in conflict with criteria 1–4 for DMs 2, 3 and 4. • Criterion 7 is in conflict with criteria 1–4 for all DMs, as is criterion 8 for DMs 1, 2 and 3 (it is distinct but does not conflict to the same extent for DM4). Criteria 7 and 8 are very closely related for DMs 1 and 2, less so for the others. These are the two criteria relating to financial considerations. • Criterion 9, the need to meet the Social Work Department’s requirements, is distinct from other criteria for all DMs, lying somewhere between the 2 groups of criteria identified above. • For all DMs option B appears to be very different to the others. • Option E is also different from the others for all DMs as is option A for DMs 1, 2, and 3, apparently performing better with respect to organisational and financial factors rather than quality of care. • Options C and D are similarly positioned for DMs 1, 2 and 3. • Options F, G and H are similarly positioned for all DMs, with option I lying close to these for DMs 2, 3 and 4. This tentative information led us to go back to the profile plots to examine the raw data in greater detail. We first looked at options F, G and H, which were closely positioned in all the plots and saw that indeed all DMs had evaluated these very similarly, although there were small differences. In practice they represent three option ways of providing care but sharing a common core. We felt that the analysis and presentation of findings would be simplified by merging these into a single option (which will be displayed as option M), reducing the number under consideration to 7. We also looked at options C and D and found these to be rated identically by all DMs on the Clients and

J. Hodgkin et al. / European Journal of Operational Research 160 (2005) 172–189

183

Fig. 6. (A) Original PCA-Plot for decision maker 1, (B) original PCA-Plot for decision maker 2, (C) original PCA-Plot for decision maker 3, (D) original PCA-Plot for decision maker 4.

Carers criteria, and where they were differentiated on the other criteria option D was always rated above C; that is, option D dominated option C for all the DMs. On this basis we removed option C from our investigation at this point (the realization that this significant point and rather obvious point has not been easily spotted in the VISA profiles led us to incorporate a dominance analysis in that software). The PCA plots were redrawn with the remaining 6 options. No new insights were generated by these plots, which re-confirmed the other points noted above. However, it was again clear from its highly differentiated position that option B, which was to sell the centre and withdraw from day care, was very different from the others which represented different forms of care. On reflection we felt that this probably represented a higherlevel decision and that a better comparison of the options for care might be obtained if it were set aside.

A new set of PCA plots was drawn, now focusing on the remaining 5 options. These can be seen in Fig. 7 and it is clear that these options are highly differentiated; they are well scattered about the plots for all DMs. Option D and the new merged option (M) tend to be positioned towards the care related criteria, option E towards the organisational and financial criteria, whereas for options A and I there is no consistent positioning. Note that all these considerations are only tentative at this stage, we are trying to define an agenda for the next workshop and all issues would be discussed with the DMs. This initial investigation also highlighted that the Carers View (criterion 4) was very similar to one of Clients criteria (criterion 1, the social environment provided for the clients). This enabled us to focus on the 3 top-level criteria. To this stage no account had been taken of criteria weights. In order to explore the influence

184

J. Hodgkin et al. / European Journal of Operational Research 160 (2005) 172–189

Fig. 7. PCA-Plot of 4 decision makers with reduced number of options: (A) PCA-Plot for decision maker 1 with reduced number of options, (B) PCA-Plot for decision maker 2 with reduced number of options, (C) PCA-Plot for decision maker 3 with reduced number of options, (D) PCA-Plot for decision maker 4 with reduced number of options.

of criteria weighting in as flexible a manner as possible we turned to the M-Plots. Although the MPlot allows exploration of preferred options without specifying the high-level criteria weights, it does require values for lower level weights and we had to make some assumptions here, as we had not elicited values for these. As a first step we used the weight values, which had emerged in analysis of the options for residential care (which had utilized a very similar value tree), recognising that this was a significant assumption. However, at this stage we were only looking for insights into how to run and what might come out of the workshop, not ‘‘answers’’. The triangle plots for the 4 DMs, including option B (but omitting option C and merging F, G and H to produce M) are shown in Fig. 8. The merged option, M, dominates at this level for DM4 (however the three top level criteria are

Fig. 8. M-Plots of 4 decision makers.

J. Hodgkin et al. / European Journal of Operational Research 160 (2005) 172–189

weighted it is the preferred option) and so is the only one to feature in the plot. It is the only option, which features in all 4 plots, in all cases being preferred when a higher weight is given to ‘‘Clients’’. The plot for DM1 (already discussed in Section 3) brings in option D, which is preferred over most of the weight space and option E which is preferred if Funding issues are weighted highly. Option B is potentially preferred for DMs 2 and 3 for high weighting on Organisational and Funding issues. As the weight on the 4th criterion, Carers View, is increased options B and E disappear from the plots, highlighting the similarity of rating on this criterion with ‘‘Client’s’’. Option I appeared briefly in DM3’s plot as this weight was increased; at no point did option A appear. From this analysis we formed a tentative view that options D, E and the merged option were most likely to form the basis of a consensus. This led us to focus on the evaluations of these three options with a view to identifying where there were differences, to enable us to explore these in greater detail during the workshop. During the workshop itself we worked through the evaluations following the steps suggested by the above preliminary analysis. The participants agreed to simplify the decision by merging options F, G and H and it was confirmed that option C was dominated by option D and thus could also be removed. They wished to retain option B. A shared view was elicited on criteria weights and some scores were revised following discussion. Group VISA was used to facilitate these discussions. A final evaluation of the options, shown in Fig. 9, and accompanying sensitivity analysis showed that the merged option was preferred by everyone, with option D being similarly rated. Although option E was rated more highly by everyone in respect of Funding issues it was felt

Fig. 9. Final evaluation of options by the DMs.

185

that this did not outweigh its lower performance on Clients. Option I was also preferred or equivalent to option E, but for all DMs it was dominated or near-dominated by option D or the merged option, so was never the most preferred option. Following the workshop the process and outcomes were summarized in a detailed report to the Executive Committee and Board of Directors. The outcome of the case study is not a main concern of this paper, which is focused on the AVID softwares; a detailed evaluation of the participants’ perception of the intervention was carried out as part of the GDSC project and is discussed by Belton and Koulouri (in preparation). Nevertheless it is worthwhile noting that the participants felt that the process enabled them to clarify their thinking about the overall issue and their actions since have matched the preferred outcomes that emerged from the study of options both for the residential home and the day centre.

5. Evaluation of the software The softwares described in this paper had already undergone an extensive program of testing and evaluation in an experimental setting and this case study provided the first opportunity to assess their value in an action research context. One of the authors (referred to as the researcher in this section), who had no involvement in the case study, sought to conduct a systematic evaluation of their use, which addressed the following key questions: • How easy to interpret were the plots? • Did the facilitators find the AVID softwares useful? • What was the role of the AVID softwares? If the plots were found useful, in what ways were they useful? • To what environments and users are the AVID softwares best suited? The researcher conducted two sessions with the facilitators (referred to as F1 and F2). For F1, who had been involved in the development and initial

186

J. Hodgkin et al. / European Journal of Operational Research 160 (2005) 172–189

testing and evaluation of software it was the first opportunity to use the software in the context of a real study. F2 had no previous exposure to the software. The first of the sessions held with the facilitators was designed as a workshop where F2 was introduced to the AVID softwares, and taken through the basics of their use. The second session, held when involvement with the clients was over, was a semi-structured interview designed to seek answers to the above questions. 5.1. Results Responses were noted from both sessions and evaluated in terms of the research questions posed. These are summarized below: 5.1.1. How easy to interpret are the plots? The PCA-Plot was found to be the more difficult of the two plots to interpret. In Session 1, F2 stated that she found the plot a little confusing initially, however, after utilising the software she became more confident in interpreting the plot. In addition, F1, who is familiar with PCA, admitted that she still on occasions found the plot slightly confusing. The M-Plot was found to be intuitive and easy to interpret. This was especially true of models that consisted of only 3-criteria, where only the 2D version of the plot was required. 5.1.2. Are the prototype visualisation systems useful? As the case study indicates, the two systems were found useful in this context. It was interesting to note that the two plots were found useful in different ways. The PCA-Plot was found to be very much a heuristic device, in that the plot highlights information and then leaves the user to investigate the results more fully for themselves (using Group VISA). The M-Plot was found to be more of an analytical device, in that it is easy to see what the plot is showing and the user can quickly deduce whether this is what was expected. There is no ambiguity about the information being shown; nevertheless, the facilitators still found it useful to return to VISA to verify the information and explore this in greater depth.

5.1.3. What is the role of the plots? It would seem that the main role of the PCAPlot was in highlighting important issues within the data. The facilitators commented on the usefulness of being able to gain a quick overall view of the model, and find out the main areas of interest in the data, which could then be confirmed and investigated further. In particular, the plots highlighted clusters of similarly positioned options, options that were significantly differentiated from the others, highly correlated criteria and conflicting criteria. In this case they were particularly useful as a means of highlighting similarities and differences between the four DMs. The M-Plot was found to adopt a different role to that of the PCA-Plot. The M-Plot was found to be useful for giving a clear representation of the robustness of the model and of which options were potentially preferred, information which was difficult to determine from the PCA-Plot. Once again, the plot was found helpful for conducting comparisons between individuals. In designing this software it had been envisaged that it would be used for the display of high-level sensitivity analysis in a value tree once lower level weights were specified, rather than an exploratory tool. The use in this latter manner meant that assumptions about weight values had to be made and although it was possible to explore the impact of different assumptions this was done only in a minimal way. After the event the potential optimality of all options was checked using the facility incorporated in Intelligent VISA (Hodgkin et al., 2000), but for future use in this manner we would advocate that this was part of the exploratory analysis. Note that both displays were used in conjunction with the original VISA and Group VISA displays, adding to rather than replacing these. Each gave quick pointers to patterns or results that would not be immediately obvious in the VISA displays. However, following these suggestions the facilitators looked to the VISA displays to confirm or explore in greater depth. In essence, the displays provided new ways of ‘‘condensing’’ the data, complementing the existing facilities, which offer several ways of ‘‘slicing’’ the data.

J. Hodgkin et al. / European Journal of Operational Research 160 (2005) 172–189

Overall, the facilitators felt that the displays taken together enabled a more efficient and effective initial exploration of the data, allowing them to identify key similarities, to see ways of simplifying the issue at an early stage, and to focus on the important options. 5.1.4. What environments and users is the software suited to? The facilitators felt that for the near future they would still restrict the use of the PCA-Plot to backroom analysis. The primary reason for this was a desire to increase their own confidence in interpreting the plot before using it live with clients. Even then, they felt that the plot could potentially confuse clients and in a workshop environment it might be most appropriate to use it off-line. This could be by the one of the facilitators whilst the other is interacting with workshop participants; it could be during a break or whilst participants are in break-out groups; or if technology permits, the lead facilitator could be using a split screen with a ‘‘private’’ display of the output from the AVID software to inform interaction with participants. F1 felt that if she was working with clients who are familiar with the basic MCDA approach and supporting software, then it may be helpful to introduce them to the PCA display, but the decision to do so would be highly contingent on the skills of the group members. The facilitators did not feel that the software was appropriate for use by na€ıve users (e.g. clients) in an unfacilitated environment. The facilitators suggested that they would definitely make use of the M-Plot software in a live decision making environment. In addition, F1 stated that she would probably allow a na€ıve user to make use of the M-Plot, given the ease with which the plot can be interpreted. One point expressed by both facilitators with regard to allowing na€ıve users access to the systems, was that this decision is dependent on the individual client/na€ıve user. They stressed the importance of first identifying the client’s level of understanding and taking account of this then deciding whether or not these displays would be likely to enhance their understanding or their confusion.

187

5.2. Comment on the validity of the evaluation As indicated above, the main sources of data for the evaluation of the softwares were the initial training workshop and the semi-structured interview with the facilitators. Whilst this is a rich source of information, it clearly has some limitations, which should be acknowledged. Firstly, the time lag between the use of the softwares and the interview may influence responses. If this is too long then important aspects may be forgotten, if too short then there may not have been adequate opportunity for reflection. The time interval here was almost 4 months (as a consequence of the time to complete the intervention and intervening holiday period), which is longer than ideally preferred. On reflection a preferred approach would have been two interviews, one immediately following the intervention and the second following a period of reflection. Secondly, the focus was on one data collection method, the semi-structured interview. On reflection it would have been informative also to have more accurate and comprehensive record of actual use. However, the nature of use (two persons working both individually and collectively, in a fragmented way, over several days) would have made observation extremely difficult if not impossible and asking the facilitators to provide a detailed transcript of use would have inhibited that use. Thirdly, clearly the particular context (the facilitators, the problem, the clients, the environment) is of significance and will influence the evaluation; nevertheless, we feel that useful insights have been gained.

6. Concluding remarks The paper begins by motivating the need for extended software support for expert MCDA analysts, citing the need to be able to quickly interpret a large volume of data and decide on appropriate analyses to conduct, without overlooking important issues, possibly in the stressful and time pressured environment of a decision workshop. The complexity of this task increases many-fold with group-based systems, which elicit individual inputs from participants. If the future

188

J. Hodgkin et al. / European Journal of Operational Research 160 (2005) 172–189

sees a role for multi-criteria decision support tools in electronic democracy, allowing the participation of large populations in evaluation of options, then that complexity is further multiplied. We then go on to describe two softwares for analysis and visual display (AVIDs) that extend the power of analysis of existing softwares for multi-criteria analysis based on MAVT. Neither of these displays is in themselves innovative, both have been previously incorporated in other forms of multi-criteria analysis, it is their integrated use in practice, in the context of group MAVT which we believe has previously been unexplored. This is illustrated through a detailed description of their use in a case study. Interestingly the use in the case study differed somewhat from how we had imagined they would be used at the time of design. Our initial emphasis in designing the AVID softwares had been on supporting sensitivity and robustness analysis in a shared model in a workshop environment. However, the use in this case study focused on exploratory analysis of a multi-participant model prior to a workshop and was instrumental in setting the agenda for, or choreographing the workshop. The softwares proved to be extremely valuable in providing overviews of the participants’ evaluations and in highlighting key aspects of the data, which could be explored in greater detail using Group VISA. The two facilitators felt that the process of exploratory analysis was both expedited and enhanced by their use. At the same time, this backroom experience enabled the facilitators to gain confidence in the tools in the context of a ‘‘real’’ problem and to judge their potential in a workshop environment. Although it was felt that both would be useful in a workshop, there remains a question as to whether it would be helpful or confusing to expose participants to the PCA plot (as opposed to having available as an off-line tool for the facilitators’ use). A particularly valuable feature of the displays was their complementarity, with each other and with Group VISA. There is much scope for further work. This intervention and other investigations have suggested many ways of enhancing and extending the features of both softwares, including seamless integration with Group VISA. This could in part

address the need to make assumptions about the value criteria weights at lower levels in the value tree by allowing these to be interactively adjusted and updating the M-Plot to reflect the new values. There are many other dimensions to explore, the use of the softwares in a workshop environment, the use by other facilitators in order to explore the transferability of the perceived value, the response of na€ıve users to exposure to the softwares, and of course the use in different case studies, each of which is likely to bring its own unique issues and learning. As previously indicated, the use of the softwares in this comparative manner had not been envisaged originally and this also opens up areas for further investigation such as the use of quantitative measures, alongside the visualisations, to highlight areas of similarity or difference between individuals. The work began as part of a study investigating ‘‘Intelligent User Support’’ and it is worthwhile reflecting for a moment on whether that is what has been provided. The targeted user in this case is the expert MCDA analyst/facilitator and there is no doubt that these tools provide enhanced support for such users. But is it intelligent? Perhaps a more suitable description might be ‘‘Intelligent User’’ support––i.e. a tool to facilitate intelligent analysis by intelligent users! Finally, we would like to add a note that although the AVID softwares are discussed here in the context of their use with the Group VISA software, they can equally well complement the use of other approaches and softwares, such as the Analytic Hierarchy Process (Saaty, 1980) and Expert Choice, and can themselves be further enhanced by other forms of analysis and visual display such as the simulations presented by Butler et al. (1997).

Acknowledgements We would like to acknowledge the contribution of the following parties: Scottish Enterprise Glasgow (formerly the Glasgow Development Agency) who funded the Glasgow Decision Support Centre Project which led to this case study; the staff and directors of the Rotary Residential Care Centres

J. Hodgkin et al. / European Journal of Operational Research 160 (2005) 172–189

who were involved in the case study; Dstl who partly funded research leading to the development of the AVID softwares; and the anonymous referees for their helpful comments.

References Angehrn, A.A., 1991. Designing humanized systems for multiple criteria decision making. Human Systems Management 10 (3), 221–231. Belton, V., 1985. The use of a simple multiple criteria model to assist in selection from a shortlist. Journal of the Operational Research Society 36, 265–274. Belton, V., 1986. A comparison of the analytic hierarchy process and a simple multi-attribute value function. European Journal of Operational Research 26, 7–21. Belton, V., 1993. Project planning and prioritisation in the social services––an OR contribution. Journal of the Operational Research Society 44, 115–124. Belton, V., Hodgkin, J., 1999. Facilitators, decision makers, D.I.Y. users: Is intelligent multicriteria decision support for all feasible or desirable? European Journal of Operational Research 113, 247–260. Belton, V., Koulouri, A. Using multicriteria analysis in practice: A question of care, in preparation. Belton, V., Pictet, J., 1997. A framework for group decision using a MCDA model: Sharing, aggregating, or comparing individual information? Journal of Decision Systems (Special Issue on Group Decision and Negotiation) 6, 283–303. Belton, V., Pratt, R.L., 1999. Multi-criteria decision analysis: A structured approach to making choices. Journal of Defence Science 4, 47–55. Belton, V., Stewart, T.J., 2002. Multiple Criteria Decision Analysis: An Integrated Approach. Kluwer. Belton, V., Vickers, S.P., 1990. Use of a simple multi-attribute value function incorporating visual interactive sensitivity analysis for multiple criteria decision making. In: Bana e Costa, C.A. (Ed.), Readings in Multiple Criteria Decision Aid. Springer-Verlag, Berlin, pp. 319–334. Brans, J.P., Mareschal, B., Vincke, P.H., 1986. How to select and how to rank projects: The PROMETHEE method. European Journal of Operational Research 24, 228–238. Butler, J., Olson, D.L., 2000. Comparison of centroid and simulation approaches for selection sensitivity analysis. Journal of Multi-Criteria Decision Analysis 8, 146–161. Butler, J., Jia, J., Dyer, J., 1997. Simulation techniques for the sensitivity analysis of multi-criteria decision models. European Journal of Operational Research 103, 531– 546. Chernoff, H., 1973. The use of faces to represent points in kdimensional space graphically. Journal of the American Statistical Association 68 (342), 361–368.

189

Climaco, J.N., Antunes, C.H., 1989. Implementation of a userfriendly software package––a guided tour of TRIMAP. Mathematical Computer Modelling 12 (10/11), 1299–1309. Everitt, B., Dunn, G., 2001. Applied Multivariate Data Analysis. Arnold. Hodgkin, J., 2000. Intelligent user support in decision support systems. Ph.D. thesis, Department of Computing Science, University of Strathclyde. Hodgkin, J., Belton, V., Buchanan, J.T., 2000. Intelligent user support in multicriteria decision support. In: Haimes, Y.Y., Steuer, R.E. (Eds.), Proceedings of the 14th International Conference of Multi Criteria Decision Making. In: Research and Practice in Multiple Criteria Decision Making. Springer-Verlag, pp. 284–295. Kasanen, E., Ostermark, R., Zeleny, M., 1989. Gestalt system of holistic graphics: New management support view of MCDM. In: Lockett, A.G., Islei, G. (Eds.), Improving Decision Making in Organisations. In: Lecture Notes and Mathematical Systems. Springer-Verlag, pp. 143–156. Korhonen, P., 1991. Using harmonious houses for visual pairwise comparison of multiple criteria. Decision Support Systems 7, 47–54. Losa, F.B., Van den Honert, R., Joubert, A., 2001. The multivariate analysis biplot as a tool for conflict analysis in MCDA. Journal of Multi-Criteria Decision Analysis 10, 273–284. Mareschal, B., Brans, J.P., 1988. Geometrical representation for MCDA. European Journal of Operational Research 34, 69–77. Pictet, J., Belton, V., 2001. ACIDE: Analyse de la Compensation et de l’Incomparabilite dans la Decision. Vers une prise en compte pratique dans MAVT. In: Colorni, A., Paruccini, M., Roy, B. (Eds.), Aide Multi Critere a la Decision. European Commission, pp. 245–256. Rios Insua, D., French, S., 1991. A framework for sensitivity analysis in discrete multi-objective decision making. European Journal of Operational Research 54, 176–190. Saaty, T.L., 1980. The Analytic Hierarchy Process. McGraw Hill. Stewart, T.J., 1981. A descriptive approach to multiple criteria decision making. Journal of the Operational Research Society 32, 45–53. Tan, Y.S., Fraser, N.M., 1998. The modified Star graph and the Petal diagram: Two new visual aids for discrete alternative multicriteria decision making. Journal of Multi-Criteria Decision Analysis 7, 20–33. Triantaphyllou, E., Sanchez, A., 1997. A sensitivity analysis approach for some deterministic multi-criteria decision making methods. Decision Sciences 28, 151–194. Vetschera, R., 1997. A recursive algorithm for volume-based sensitivity analysis of linear decision models. Computers in Operational Research 24 (5), 477–491. Wolters, W.T.M., Mareschal, B., 1995. Novel types of sensitivity analysis for additive MCDM methods. European Journal of Operational Research 81, 281–290.