Strategic environmental assessment quality assurance: evaluating and improving the consistency of judgments in assessment panels

Environmental Impact Assessment Review 24 (2004) 3 – 25 www.elsevier.com/locate/eiar

Strategic environmental assessment quality assurance: evaluating and improving the consistency of judgments in assessment panels Bram F. Noble* Department of Geography, University of Saskatchewan, # 9 Campus Drive, Saskatoon, SK, Canada S7N 5A5 Received 1 January 2003; received in revised form 1 May 2003; accepted 1 May 2003

Abstract Assessment panels and expert judgment are playing increasing roles in the practice of strategic environmental assessment (SEA). Thus, the quality of an SEA decision rests considerably on the quality of the judgments of the assessment panel. However, there exists very little guidance in the SEA literature for practitioners concerning the treatment and integration of expert judgment into SEA decision-making processes. Subsequently, the performance of SEAs based on expert judgment is often less than satisfactory, and quality improvements are required in the SEA process. Based on the lessons learned from strategic- and project-level impact assessment practices, this paper outlines a number of principles concerning the use of assessment panels in SEA decision-making, and attempts to provide some guidance for SEA practitioners in this regard. Particular attention is given to the notion and value of consistency in assessment panel judgments. D 2003 Elsevier Inc. All rights reserved. Keywords: Expert judgment; Strategic environmental assessment; Consistency; Consensus; Assessment panels; Decision-making

1. Introduction Strategic environmental assessment (SEA) is gaining widespread recognition as a tool for supporting the sustainable development of the environment through

* Tel.: +1-306-966-1899. E-mail address: [email protected] (B.F. Noble). 0195-9255/$ – see front matter D 2003 Elsevier Inc. All rights reserved. doi:10.1016/S0195-9255(03)00118-5

4

B.F. Noble / Environ. Impact Asses. Rev. 24 (2004) 3–25

policy, plan and program decision-making processes. In recent years, we have witnessed a growing body of literature addressing SEA principles (e.g. Noble, 2000; Partidario, 2000; Therivel, 1993), methodology (e.g. Noble and Storey, 2001; Brown and Therivel, 2000; Verheem and Tonk, 2000), and performance criteria (e.g. Fischer, 2002; IAIA, 2002; Nitz and Brown, 2000). However, as Bonde and Cherp (2000) suggest, SEAs remain less than satisfactory and improvements in the quality of SEA decisions are required (see, e.g. Hazell and Benevides, 2000; Curran et al., 1998). The challenge is to consider how practitioners can ensure the quality of strategic decisions given the relatively broad-brush nature of SEA, and the increasing emphasis placed on the role of assessment panels, or groups of informed individuals, selected to assign impact assessment judgments based on experience and expertise. The argument presented here is that the limitations to improved SEA decisionmaking are largely due to the way in which assessment judgments are analyzed, treated, and applied in the SEA decision process. As more complex evaluation and decision-making methods and techniques are used in SEA, there is an increasing reliance placed on the judgments and expertise of assessment panels; however, there is very little guidance available to practitioners concerning its use and treatment. Based on the lessons learned from recent practice expert-based strategic- and project-level impact assessment case studies, and drawing particularly upon the results of an expert-based SEA of Canadian energy policy (Noble, 2002), this paper attempts to provide some guidance to practitioners on the way in which assessment judgments are solicited, evaluated, and integrated into SEA decision-making processes. The author elsewhere reports on the results of the SEA case study in detail (see Noble, 2002). What follows is a discussion of quality assurance in SEA decision-making, including guidelines for soliciting and analyzing expert judgment, and a detailed discussion of the notion and value of assessment consistency—an issue that has received insufficient attention (if at all any) in the SEA literature.

2. Assessment panels and SEA quality assurance Various innovative methods and techniques have been discussed in the impact assessment literature in recent years. Noble and Storey (2001), for example, presented a multi-criteria approach to SEA methodology, and Goyal and Desphande (2001) explored the value of the ‘‘importance scale matrix’’ in minimizing bias and subjectivity in EA decision-making. Pastakia and Jensen (1998) introduced the concept of a ‘‘rapid impact assessment matrix,’’ and Bonnell (1997) demonstrated an adaptation of the Delphi technique to address spatial variations in cumulative environmental effects. As more complex evaluation processes are being used in impact assessment that involve integral reasoning, there is, inherently, a stronger reliance on expert judgment (Kontic, 2000, p. 428). For example, in the SEA of the revised

B.F. Noble / Environ. Impact Asses. Rev. 24 (2004) 3–25

5

Lancashire Structure Plan, Lancashire, UK, the potential environmental impacts of policy statements were scored using an impact matrix with environmental scientists judging the impacts of each policy option on each environmental receptor. The SEA of the 1996 Neiafu Master Plan, Vava’u, Tonga, was similarly based on the expert judgment of planning officials and public administrators. The reliance on expert judgment in impact assessment is further illustrated by an environmental audit report of artificial waterway developments in Western Australia, where Bailey et al. (1992) found nearly two thirds of all impact predictions to be based on knowledge and experience; Culhane et al. (1987) report similar findings. Expert judgment is the backbone of most impact assessments, thus the quality of SEA depends, to a significant extent, on the credibility of the experts and the quality of the expert judgment (Kontic, 2000). The problem is that there is very little guidance to SEA practitioners on ensuring the quality of assessment panel judgments and, furthermore, the limitations of expert judgment in impact assessment are largely due to the way in which judgments are analyzed and applied in SEA decision processes. 2.1. Determining SEA panel characteristics One of the most challenging issues in any SEA involving the use of assessment panels is identifying the necessary size, composition, and expertise of the panel. The literature on impact assessment and environmental decisionmaking suggest no standard procedure for identifying an assessment panel; however, several principles and guidelines emerge from recent strategic and project-level impact assessment practices. These principles are not unique to SEA panels, but to selecting environmental assessment panels in general. 2.1.1. Size of an SEA panel Concerning assessment panel size, there is no set number of impact assessment decision-makers that is best suited to all situations. The size of an assessment panel depends on, among other things, the objectives of the impact assessment and available time and resources and is therefore context and case specific (Table 1). Turoff (1975), for example, suggests that as few as 10 people are sufficient for an assessment panel; however, actual panel size varies considerably in practice. Mar et al. (1985) identified 92 expert panellists for the development and assessment of an aquatic ecological monitoring scheme, of which 62 agreed to participate, while Ludlow (1975) identified 50 panellists to assess long-term resource management problems in Lake Michigan. In more recent practices, Bonnell (1997) identified 123 potential panellists to serve on a panel to assess the cumulative environmental effects of small-scale hydroelectric developments in Newfoundland, Canada, of which 49 agreed to participate, and Noble (2002), identified 141 potential panellists to evaluate the strategic impacts of Canadian energy policy alternatives, of

6

B.F. Noble / Environ. Impact Asses. Rev. 24 (2004) 3–25

Table 1 Selected criteria for determining assessment panel size and composition & Representation of those affected by SEA decisions

– affected interest groups – affected sectors, government departments, and industries & Representation of those who affect SEA decisions – public administrators – planners and policy-makers – scientists and researchers & Appropriate geographic representation & Inclusion of necessary expertise and experience & Practicality given available time and resources & Large enough to facilitate required statistical analysis & Credibility of panel size and membership

which 111 initially agreed to participate and 69 actually completed the assessment process. 2.1.2. Panel composition Concerning assessment panel composition, Scheele (1975) suggests that three types of panelists be considered, including experts, stakeholders, and facilitators. The relative proportion of each depends on the study objectives and the nature (i.e. complexity and technicality) of the impact assessment issue in question, and thus should be tailored to each individual assessment situation. Morgan and Onorio (2000), for example, suggested that public stakeholder involvement in the SEA of the Neiafu Master Plan, Vava’u, Tonga was neither feasible nor advisable given local resistances to public administration authorities. When it is clear who has to act, that is policy, plan or program decision-makers, but it is not clear how, that is the strategic direction, a panel of experts is often the preferred choice. In other cases, argues Partidario (2000), improving the added value of SEA requires increased attention to affected interests. 2.1.3. Identifying expertise While there is no standard procedure for identifying panellist expertise, several guidelines emerge from recent practices (Table 2). For example, Bonnell (1997), in assessing the cumulative effects of small-scale hydro, selected an assessment panel based on experience, reputation, and panellist’s involvement in hydroelectric developments and similar types of projects. Richey et al. (1985), in an environmental assessment study of effluent control systems, used several criteria for identifying and selecting assessment panellists, including: previous experience in two or more specialty areas that would be considered during the assessment; previous experience in at least one of the valued system components involved; and geographic representation. Gokhale (2001) adopted a similar approach for an environmental initiative prioritization in four major cities in India. Gokhale included such criteria as the total number of years of practice or

B.F. Noble / Environ. Impact Asses. Rev. 24 (2004) 3–25

7

Table 2 Expert panel selection criteria emerging from recent practice impact assessments & Experience in two or more of the specialty areas considered in the assessment & Current or previous leadership or management role in one or more of the specialty areas considered & Experience in at least one of the valued system components affected & Representation of a particular sector, interest, or affected geographic area & Seven to 10 years of combined education and professional experience in impact assessment and/or

one of the key assessment areas (disciplines) involved & Experience in similar types of assessments or decision-making processes & A high level of professional productivity as evidenced by:

– – – –

publications participation in professional meetings and symposia experience in project management current or previous membership on EA panels & Based on self-identified interest or expertise (those who simply wish to be involved) Source: Based on Noble (2002), Gokhale (2001), Bonnell (1997), Huylenbroeck and Coppens (1995), Richey et al. (1985), and Sobral et al. (1981).

experience in environmental management, the number of publications and presentations by each panellist on the topic in question, and leadership and management position in urban environmental organizations. 2.2. The relationship between accuracy and quality There has been much discussion in the impact assessment literature in recent years concerning the accuracy of impact predictions (e.g. Arts et al., 2001; Locke and Storey, 1997; Sadler, 1996; Bailey et al., 1992; CEAA, 1992). Experience has shown, however, that there are often practical difficulties in verifying impact predictions (e.g. Locke and Storey, 1997), and determination of accuracy as a measure of the quality of impact assessment judgments becomes problematic when the characteristics of the variables for which such judgments are made are constantly changing. Sadler (1996), in his final report on the international effectiveness of environmental assessment, Environmental Assessment in a Changing World, reports that 60% of the time assessments are unsuccessful to only marginally successful in making precise, verifiable impact predictions. Furthermore, Sadler reports that 75% of the time assessments are unsuccessful to only marginally successful in indicating confidence levels for data used in predicting impacts. This problem is only exacerbated as one moves from the project to the strategic levels of decision-making, since strategic-level decisions are often based on longer term actions and over a larger geographic area. Moreover, many of the impacts resulting from policy or plan-level decisions are indirect and difficult to measure with regard to the accuracy of impact predictions. Bailey et al. (1992) in an environmental audit report of EAs of artificial waterway developments in Australia found that the accuracy of impact predic-

8

B.F. Noble / Environ. Impact Asses. Rev. 24 (2004) 3–25

tions had no bearing on environmental management activities, and management responses to actual impacts were implemented for both inaccurately predicted impacts and for unforeseen impacts. Furthermore, it is perhaps misleading to assume a direct link between the quality of expert judgment, or expert credibility, and the accuracy of impact predictions. Parente et al. (1984, p. 180), Armstrong (1978, p. 86), and Welty (1972) agree, noting that a linear relationship between accuracy of prediction and expertise has not been consistently reported. The author agrees with Kontic (2000), in that expert credibility and the quality of SEA decisions does not necessarily reflect the accuracy of impact predictions. As Arts et al. (2001) suggest, it is not the predicted impacts that are important but the actual impacts. 2.3. Decision consensus Consensus is a common goal in resource and environmental group decisionmaking (e.g. Bryson and Joseph, 2000; Huylenbroeck and Coppens, 1995), and several authors have demonstrated group feedback and iteration to be extremely efficient in achieving consensus (e.g. Huylenbroeck and Coppens, 1995; Rohrbaugh, 1979; Scheibe et al., 1975). However, consensus should never be the primary goal of group feedback and iteration when assigning strategic impact assessment judgments. While an expert or analyst may contribute to a quantifiable or analytical estimation, it unlikely that a clear-cut (to all concerned) resolution will result from such an analysis (Turoff, 1970). ‘‘The same lack of knowledge that produced the need for a study that relied on expert judgement virtually assures that a group of ‘diverse experts’ will disagree’’ (Stewart and Glantz, 1985). Consensus through iteration and feedback does not necessarily increase the accuracy (or quality) of impact predictions (Parente et al., 1984). Rowe et al. (1991) explain the reasoning behind providing aggregate group feedback based on the so-called ‘‘theory of errors,’’ in that the aggregate of a group will provide an assessment that is generally superior to that of most of the individual assessments within the group. When the range of individual estimates includes the true answer, then the median is more accurate than one-half of the group. This does not mean, however, that the median is necessarily more accurate than the most accurate panellist. ‘‘If the individuals differ systematically among themselves with respect to the variables of interest, then such information is lost upon aggregation. . .and there is always the danger of ‘piecemeal’ distortion’’ (Coxon, 1982, p. 15). Thus, aggregating assessment data without examining individual differences may lead to a false sense of group consensus. Consensus is neither a necessary nor a sufficient condition for SEA decision quality and reliability, and is in no way a reflection of genuine agreement when dealing with diverse assessment panellists and complex environmental issues. SEA relies on increased integration, order, and congruity through the facilitation

B.F. Noble / Environ. Impact Asses. Rev. 24 (2004) 3–25

9

of horizontal decision-making and improved communication among agencies and organizations. It is because of these differing perceptions (and insufficient knowledge) that a lack of consensus exists (Gonzalez, 1992). The role of the assessment panel in SEA decision-making is not so much to obtain consensus as it is to expose all of the differing positions to improve the effectiveness of the decision-making process. SEA rests on the premise that the final decision-maker is not interested in having an assessment panel generate the final decision, but, rather, having the panel systematically evaluate and present all of the options such that the final decision-maker or decision-making agency has the necessary information to make an informed decision. SEA decision analysis should address pluralism and dynamism, rather than hide them, to address the multiplicity of different perceptions and often-conflicting interests involved in the assessment process. The resolution of a strategic question ‘‘must take into consideration the conflicting goals and values espoused by various interests as well as the facts’’ (Leitch and Leistritz, 1984). Thus, extreme judgments should be allowed to stand to the heterogeneity of the group, but only within the limits of tolerable inconsistency. 2.4. The importance of consistency Therivel et al. (1992) highlight as one of the main objectives of an SEA ‘‘to enable consistency to be developed across different policy sectors, especially where trade-offs need to be made between objectives.’’ Strategic decisions, particularly when dealing with issues as far reaching and as significant as environmental policy, should not be based on internally contradictory impact assessment judgments. The relationships in any SEA system should provide, to the greatest extent possible, a set of consistent and non-contradictory impact judgments. While there are several documents outlining the necessary components of quality SEA procedures and reports,1 current guidelines and ‘best’practices fail to address directly the notion and importance of consistency in assessment panel judgments. Consistency is a statistical measure of the extent to which an individual’s decision structure (i.e. set of assessment judgments) is closer to being logically related than randomly chosen. The consistency of impact assessment judgments reflects the extent to which the decision-maker(s) understands the problem, is knowledgeable of the decision variables involved, understands the assessment process, and is able to make a series of logically related impact assessment judgments based on uncertain and often incomplete information. The consistency 1

See, for example: the Canadian Department of Foreign Affairs and International Trade’s framework for the environmental assessment of trade agreements (DFAIT, 2002); IAIA’s (2002) list of SEA performance criteria; Transport Canada’s (2001) SEA framework requirements; Sections 2.2 through 2.5 of Canadian Cabinet Directive on SEA (CEAA, 1999); and the UK Department of Environment’s good practice guide to the environmental appraisal of development plans (DoE, 1993).

10

B.F. Noble / Environ. Impact Asses. Rev. 24 (2004) 3–25

of judgments in SEA is important not only because decisions are at the origin of SEA problems, but also because they are at the core of effective solutions to those problems. 2.4.1. Measuring consistency The most common measure of decision consistency is Saaty’s (1977) principle eigenvector approach. Saaty (1977) explains that for each pair of alternatives (i, j), an individual decision-maker can provide a comparison on the basis of each assessment criterion or environmental receptor (aij), such that aij = 1/aji. Thus, the consistency of a series of i–j comparisons across several assessment criteria can be determined by the consistency ratio (CR), defined as CI/RI, where CI is the consistency index of a matrix of assessment judgments and RI represents a random index.2 By comparing the CI of the individual’s impact assessment judgments to a matrix of randomly generated judgments, the CR can be determined. The generally accepted level of tolerable inconsistency is 0.10 (Saaty, 1977, 1997; Saaty and Vargas, 1982; Golden et al., 1989; Malczewski, 1999); in other words, a 10% chance that the judgments contained within the assessment matrix are generated at random. As the ratio of the individual’s assessment judgments to that of a randomly completed assessment matrix decreases (i.e. CR approaches zero), there is less of a chance that impact assessment judgments were randomly assigned. When the consistency ratio is indicative of inconsistent judgments, that is a consistency ratio greater than 0.10, the panellist should re-evaluate the impact assessment judgments. 2.4.2. Expectations for consistency From a theoretical standpoint, consistency is a necessary condition for representing a real-life problem; however, it is not sufficient. Perfect consistency in measurement is particularly difficult when dealing with an assessment panel of multiple decision-makers. Minimizing inconsistency does not mean getting an answer closer to the real solution, but that the ratio of estimates in the matrix of assessment judgments are closer to being logically related than to being randomly chosen (Saaty, 1977). Inconsistencies in assessment judgments may be a result of the lack of complete knowledge of the issue under consideration, the lack of experience with respect to the particular type of assessment, a misunderstanding of the assessment instructions, or the intentional misinterpretation of the information presented. The objective is to determine the extent to which inconsistencies in assessment judgments affect the overall strategic outcome, and to recognize that at the strategic level the consistency of assessment 2 The intent here is to provide only a brief overview of measuring decision consistency before the case study discussion. A more detailed discussion of decision consistency can be found in Saaty (1977), Saaty and Vargas (1982), Malczewski (1999), and Noble and Storey (2001). A simple example of consistency ratio calculation is included in Appendix A.

B.F. Noble / Environ. Impact Asses. Rev. 24 (2004) 3–25

11

judgments is not always related to expertise. As a prerequisite to taking decisions based on expert judgments, the quality (i.e. consistency) of those judgments should always be taken into consideration. While an expert decision-maker may not be able to provide an accurate impact assessment judgment in a complex and changing policy, plan, or program environment, any decision-maker with a clear understanding of the issue, decision variables, and the decision process should be able to demonstrate consistency in assessment judgments. SEA decisions, particularly when dealing with issues as far-reaching and as complex as environmental policy, should not be internally contradictory. The relationships in any SEA system should provide, to the greatest degree possible, a set of consistent and non-contradictory results—irrespective of the accuracy of assessment judgments. While the notion and value of the consistency of assessment judgments has received insufficient attention in the SEA literature, consistency is critical to ensuring the quality of SEA decisions.

3. Case study of SEA consistency The case study discussed here is based on an application reported by the author concerning an SEA of Canadian energy policy alternatives (see Noble, 2002). The SEA was implemented to identify the potential environmental implications of 5 energy policy options based on 11 assessment criteria (Table 3). Using an iterative survey assessment process, a panel of 69 self-identified

Table 3 Policy alternatives and assessment factors Alternatives (i) Status quo (ii) Hydroelectricity; natural gas co-generation with nuclear energy; off-grid renewable energy (iii) Electricity mosaic; increased contribution of renewable energy combined with equal increases in existing sources of hydroelectricity, natural gas, and clean coal technology (iv) Significant increase in clean coal technologies as a major energy source (v) Reliance on hydroelectricity and natural gas, with minor increases in renewable energy; decommissioning of existing nuclear plants Assessment factors (each defined by assessment criteria—see Noble, 2002) Environmental

Economic

Social

Atmospheric emissions Hazardous waste generation Habitat destruction Resource efficiency

Economic efficiency Market competitiveness

Security of supply Distributional equity Public health and safety Heritage preservation Pubic acceptability

Source: Summarized based on Noble (2002).

12

B.F. Noble / Environ. Impact Asses. Rev. 24 (2004) 3–25

experts (Fig. 1) were asked to evaluate the policy alternatives using a paired comparison assessment process (see Saaty, 1977; Noble, 2002) based on the criteria presented. The panel consisted of representatives from industry, government, consulting agencies, and non-government organizations. Members were selected based on their current or previous involvement in EA and the energy resource sector. The assessment process consisted of the initial impact assessment round, plus two iterations for consistency. The SEA methodology is detailed in Fig. 2. The focus of discussion here is limited to the evaluation of assessment judgment consistency. Consistency ratios were calculated based on Saaty’s (1977) analytical hierarchy process (see Appendix A) and using IDRISI multi-criteria decision support software. The median consistency ratios for the assessment panel’s initial evaluation are depicted in Fig. 3, and demonstrate overall inconsistency in impact assessment judgments. What follows is a discussion of the results of the consistency analysis and the implications of consistency in impact assessment judgments for SEA decision-making. 3.1. Individual iteration for consistency Individual assessment judgments from the initial impact assessment were reiterated and panellists were asked to reconsider those choice combinations for which analysis showed some inconsistencies in impact assessment judgment. Parente et al. (1984), in a series of experimental survey assessment applications, found that individual iteration more so than group feedback was the most important factor contributing to improvements in an individual’s evaluation over assessment rounds. Consistent with Bots and Hulshof (2000) and Saaty (1980), while panelists were asked to reconsider those assessments for which some inconsistencies existed, they were not forced to revise their initial judgments, but rather given the opportunity to do so. Barzilai (1998) explains that the

Fig. 1. Areas of expertise as self-identified by study panellists. Note: Responses ranged from those who identified themselves as generalists with no real expertise, to those who identified themselves as experts in areas relating to as many as seven of the assessment factors.

B.F. Noble / Environ. Impact Asses. Rev. 24 (2004) 3–25

Fig. 2. SEA framework.

13

14

B.F. Noble / Environ. Impact Asses. Rev. 24 (2004) 3–25

Fig. 2 (continued).

B.F. Noble / Environ. Impact Asses. Rev. 24 (2004) 3–25

15

Fig. 3. Median impact assessment consistency ratios: initial assessment.

assessment judgments are derived directly from the decision-maker’s input and are a true representation of the individual’s input regardless of the level of consistency. Forcing improvements in consistency, similar to forcing consensus, may distort the individual’s true answer, regardless of their understanding of the problem. Thus, inconsistent judgments were presented to the decision-makers as feedback from the analysis, and panelists were given the opportunity to confirm their initial assessment, regardless of inconsistencies, or to revise based on the information presented. Of the 69 responses received from the initial assessment round, 97% demonstrated some inconsistency in their decision structure and were asked to reconsider particular alternative-criterion assessment judgments. The number of inconsistent attributes returned to individual panelists ranged from 0 to 7, with the majority of panelists asked to reconsider four attributes. Approximately 92% of panelists revised their initial assessment judgments based on the individual feedback of consistency measures. Several panelists responded with only minor adjustments to their initial evaluations, indicating that they ‘‘had initially made an error in scoring’’ causing their inconsistency. One panelist indicated that there was no reason to adjust the initial assessment as no new significant information was received. An additional panelist responded by suggesting that they felt no need to change their initial assessment and that they would remain ‘‘consistently inconsistent.’’ Of the 92% of panelists who revised their initial assessments, all but one panelist improved their level of consistency. Fig. 4 depicts aggregate median consistency ratios for the first iteration of inconsistent assessment judgments. 3.2. Group iteration for consistency The second iteration provided panellists with an opportunity to review their individual assessment judgments in light of the group’s median responses. The purpose of this iteration was to allow the researcher to gain a better understanding of the group response and of the responses of those outside the

16

B.F. Noble / Environ. Impact Asses. Rev. 24 (2004) 3–25

Fig. 4. Median impact assessment consistency ratios after first (individual) iteration.

normal range (95% confidence interval for the group median) of the group who demonstrate a firm, consistent position. All panellists were returned their individual assessment judgments for each assessment factor and the group’s median assessment judgments, and asked to reconsider those assessments for which the analysis revealed some inconsistencies. Thirty-five panellists (57%) responded with changes to their evaluation in light of the group’s assessment judgments. To avoid the influence of sample size on group feedback, the upper and lower fence for the median of each of the group’s assessment judgments was provided (Tukey, 1977). Individuals outside the upper or lower fences for the median values are considered in conflict with the group. This does not mean that their assessments are not valid, rather that they be carefully evaluated for any inconsistencies in judgment. For alternatives (i) and (ii) based on criterion three (minimizing habitat destruction) (Table 3), for example, 60% of panellists outside the range of the group’s responses displayed consistency ratios significantly greater than 0.10 (i.e. inconsistent). For alternatives (i) and (iii), 68% of panellists outside the normal range of the group’s responses displayed consistency ratios indicative of inconsistent judgments. There were cases, however, where an individual was outside the median range of the group but remained consistent in their assessment judgments. Few of these individuals made any adjustments to their initial assessments during the group iteration. This is consistent with the findings of Saaty (1977) and Dalkey (1975) in that least consistent or least knowledgeable panellists tend to be drawn toward the median, while the most consistent or most knowledgeable panellists will be more confident and are less likely to adjust their individual judgments based on group feedback. Rowe et al. (1991) confirm that individuals outside the normal range of a group who are consistent in their judgments are unlikely to make any adjustments to their initial evaluations. On the other hand, individuals outside the group who are inconsistent in assigning assessment judgments are more likely to adjust their evaluations to be ‘‘on board’’ with the group.

B.F. Noble / Environ. Impact Asses. Rev. 24 (2004) 3–25

17

Fig. 5 illustrates the median consistency ratios for impact assessment judgments following the second iteration to be within the range of tolerable inconsistency (i.e. CR
0.10). Although it is possible to re-evaluate decisions until perfect consistency (i.e. CR = 0.0) is achieved, there is little change in assessment judgments once the CR falls below 0.10, and individuals with already consistent responses are unlikely to make any significant adjustments (Saaty, 1977). 3.3. Quality of the ‘experts’ Inconsistent judgements made without the benefit of detailed analysis may distort the final SEA decision. While Figs. 3 –5 depict improvements in the median consistency ratios over assessment rounds, and the median consistency ratios for the assessment panel are within the tolerable limit of inconsistency following the second iteration (Fig. 5), several individual panellists remained inconsistent in their assessment judgments on several alternative-criterion combinations (Fig. 6). The notion of an expert’s judgment being more consistent than that of a nonexpert seems almost tautological. However, of the 62 outlying consistency measures, indicated by the ‘ * ’ symbol in Fig. 6, representing expert judgments that are ‘extremely inconsistent’, over 60% are attached to assessment criteria for which panellists identified themselves as ‘‘experts.’’ Furthermore, at the 95% confidence interval for the median, no statistical difference was found to exist between ‘expert’ and ‘non-expert’ consistency ratios (CR) across all assessment criteria (Table 4). 3.3.1. Implications While the case study results cannot be directly extrapolated to other applications at different tiers of decision-making, they do raise some important issues for the SEA practitioner to consider concerning the notion of expertise and panel composition at the strategic level. Partidario (2000) suggests that

Fig. 5. Median impact assessment consistency ratios after second (group) iteration.

18

B.F. Noble / Environ. Impact Asses. Rev. 24 (2004) 3–25

Fig. 6. Individual impact assessment consistency measures. Note: Whiskers are present for C1, C10, C11, and C2 but are too small to be resolved in the output. Median consistency measures for each criterion C1 to C11 denoted by horizontal black lines in the box and whisker plot. ‘O’ denotes outliers; ‘ * ’ denotes extreme values of inconsistency.

increasing the value-added of SEA, and thus the quality of SEA decisions, requires increased sensitivity to communities and those affected by strategic outcomes. The results of consistency analysis may differ, favouring the experts, when dealing with technical program or project level issues. However, when dealing with higher order strategic issues, as demonstrated by the case study, the value of the expert panellist with regard to decision consistency may be overrated, and more attention might be given to stakeholders and facilitators.

Table 4 Experts versus non-experts median consistency ratios Expertise

Expert Non-expert a

Initial assessment

First iteration a

Median

95% CI

0.112 0.114

0.097 – 0.120 0.090 – 0.124

Second iteration

Median

95% CI

Median

95% CI

0.090 0.090

0.087 – 0.092 0.087 – 0.092

0.090 0.090

0.087 – 0.092 0.087 – 0.092

The 95% confidence interval for the median is a distribution free statistic. It is derived as p follows: Upper and lower fence = median ± (1.58  (H-spread)/ n). Where the H-spread is the difference between Tukey’s upper and lower hinges, represented by the box and whisker plot, and gives the range covered by the middle half of the data (approximately the 25th and 75th percentile) (Velleman and Hoaglin, 1981).

B.F. Noble / Environ. Impact Asses. Rev. 24 (2004) 3–25

19

One approach, as demonstrated by Huylenbroeck and Coppens (1995) in a multi-criteria analysis of rural development scenarios in the Gordon District, Scotland, is to incorporate the knowledge base and values of those with a stake in particular components of the policy, plan, or program issue under consideration. In the Gordon District, five alternative land use scenarios were assessed based on seven assessment criteria, categorized according to environmental preservation, recreation, and housing and community facilities. Three panels of local actors with respective interests in and knowledge of environmental conservation, recreation, and economic and social infrastructure were asked to evaluate the alternatives based on the criteria most closely associated with their area of knowledge and interest.

4. Conclusions One measure of the quality of SEA performance is the quality of the assessment judgments. As SEA practice continues, and more complex evaluation and decision-making techniques are used, the assessment panel is playing an increasingly important role. The problem relating to SEA quality assurance and decision consistency, however, is that there exists very little guidance for SEA practitioners concerning the use and treatment of expert judgment in strategic assessment processes. This paper set out to provide some direction in this regard. Based on the above discussion, and on the lessons learned from the author’s SEA application (Noble, 2002) and other similar expert-based assessments at the plan, program and project level, several guidelines and considerations for the use of assessment panels in SEA practices emerge: &

&

&

&

First, there is no best method for selecting an assessment panel. Panel size and composition rest on the SEA objectives, the socio-political assessment context, data and information requirements, available time and resources, what will be credible, and is thus dependent on setting and situation. The knowledge and experience of the assessment panel should be reflective of the technicality and complexity of the assessment issue. Second, the value of ‘expert’ judgment is often overrated in assessment panel decision-making at the strategic level. SEA practitioners must learn to stringently evaluate the ‘expert’ decision and to consider when the input of local parties and affected interests might be more valuable in leading to SEA decisions. Third, making accurate impact predictions is difficult at the project level. This is only exacerbated as one moves to the strategic-levels of decision-making. The accuracy of impact predictions, although important, is not a sufficient measure of SEA quality performance. Fourth, consensus is neither a necessary nor a sufficient condition for SEA decision-making, and should not be an indicator of SEA quality. The same lack of knowledge that required the use of an assessment panel likely means that

20

&

B.F. Noble / Environ. Impact Asses. Rev. 24 (2004) 3–25

panellists will disagree. SEA decision-making should highlight dynamism and pluralism, rather than hide them, such that the final SEA decision-maker has sufficient information to make an informed choice. Finally, the quality of impact assessment judgments in SEA rests significantly on the consistency of the assessment panellists. At the strategic level, particularly when dealing with broad-brush policy issues, consistency is not an accurate reflection of expertise, but rather of informed decisionmaking. SEA analysts should not assume that the expert judgment is more credible than that of the non-expert. Important to the quality of the SEA is that impact assessment judgments and the final SEA decision not be based on information that is inconsistent and contradictory. The relationships in any SEA system should generate, to the greatest extent possible, a consistent SEA decision structure.

In conclusion, the quality of SEA decisions rests significantly on the quality of assessment judgments. Thus, it is important that role and treatment of expert judgment receive more attention in SEA practice and in the impact assessment literature in general. There is no ‘best’-practice framework for SEA based on the use of assessment panels; however, several lessons can be learned from recent practices. This paper attempted to highlight these practices to provide some basic principles and guidelines to SEA practitioners and, ultimately, to contribute to quality improvements in the SEA process.

Acknowledgements The author wishes to acknowledge two anonymous reviewers for their useful comments and suggestions.

Appendix A. Simplified seven-step process for calculating consistency ratios Based on Saaty’s (1977) paired comparison assessment process, for any pair of alternatives (i and j), based on criterion Cx, an assessment score of:

9 = alternative i is extremely preferred to alternative j 7 = alternative i is very strongly preferred to alternative j 5 = alternative i is strongly preferred to alternative j 3 = alternative i is moderately preferred to alternative j 1 = alternative i is equally preferred to alternative j 1/3 = alternative j is moderately preferred to alternative i 1/5 = alternative j is strongly preferred to alternative i 1/7 = alternative j is very strongly preferred to alternative i 1/9 = alternative j is extremely preferred to alternative i

B.F. Noble / Environ. Impact Asses. Rev. 24 (2004) 3–25

21

Table A1 Initial paired comparison assessment matrix for criterion ‘Cx’ Cx

Alternative A

Alternative B

Alternative C

Alternative A Alternative B Alternative C

1 3 1/7

1/3 1 1/7

7 7 1

Step 1. Divide each cell entry of the paired comparison matrix by the sum of its corresponding column to normalize the matrix:

Table A2 Normalized paired comparison assessment matrix for criterion ‘Cx’ Cx

Alternative A

Alternative B

Alternative C

Alternative A Alternative B Alternative C

0.24 0.72 0.03

0.23 0.68 0.10

0.47 0.47 0.07

Step 2. Determine the priority vector of the matrix by averaging the row entries in the normalized assessment matrix. Priority vector: A ¼ 0:31 B ¼ 0:62 C ¼ 0:06 Step 3. Multiply each column of the initial assessment matrix by its relative priority, and sum the results: 1 0 1 0 0:88 6 C B C B C B C B C B C B C B C B C þ 0:62B 1 C þ 0:06B 7 C ¼ B 1:97 C 0:31B 3 C C B C B B C B A A @ A @ A @ @ 0:20 1 1=6 1=7 0

1

1

0

1=3

1

22

B.F. Noble / Environ. Impact Asses. Rev. 24 (2004) 3–25

Step 4. Divide the results by the original priorities: A ¼ 0:88=0:31 ¼ 2:84 B ¼ 1:97=0:62 ¼ 3:18 C ¼ 0:20=0:06 ¼ 3:33 Step 5. Average these results to obtain lmax: lmax ¼ 3:12 Step 6. Determine the consistency index: CI ¼ ðlmax  nÞ=ðn  1Þ ¼ ð3:12  3Þ=2 ¼ 0:06 Step 7. Calculate the consistency ratio: CR ¼ CI=RI; where RI for a three-factor matrix (i.e. 3  3) = 0.58 (see Saaty, 1977) CR ¼ 0:06=0:58 ¼ 0:10 Based on the CR, there is a 10% likelihood that assessment judgments in the initial paired comparison matrix (Table A1) are inconsistent (i.e. indicative of random responses). Inconsistency may be due to, among other things, the decisionmaker’s lack of understanding of the problem at hand, uncertainty in assigning assessment judgments, incomplete information, a poorly presented set of assessment rules and criteria, or intentional misrepresentation of assessment judgments.

References Armstrong JS. Long-range forecasting: from crystal ball to computer. New York: Wiley; 1978. Arts J, Caldwell P, Morrison-Saunders A. Environmental impact assessment follow-up: good practice and future directions—findings from a workshop at the IAIA 2000 conference. Impact Assessment and Project Appraisal 2001;19(3):175 – 85. Bailey J, Hobbs V, Saunders A. Environmental auditing: artificial waterway developments in Western Australia. Journal of Environmental Management 1992;34:1 – 13. Barzilai J. Consistency measures for pairwise comparison matrices. Journal of Multi-Criteria Decision Analysis 1998;7:123 – 32. Bonde J, Cherp A. Quality review package for strategic environmental assessments of land-use plans. Impact Assessment and Project Appraisal 2000;18(2):99 – 110. Bonnell SJ. The cumulative environmental effects of proposed small-scale hydroelectric developments in Newfoundland, Canada. MA Thesis. Memorial University of Newfoundland. St. John’s: Department of Geography; 1997. Bots P, Hulshof J. Designing multi-criteria decision analysis processes for priority setting in health policy. Journal of Multi-Criteria Decision Analysis 2000;9:56 – 75.

B.F. Noble / Environ. Impact Asses. Rev. 24 (2004) 3–25

23

Brown AL, Therivel R. Principles to guide the development of strategic environmental assessment methodology. Impact Assessment and Project Appraisal 2000;18(3):183 – 9. Bryson N, Joseph A. Generating consensus priority interval vectors for group decision-making in the AHP. Journal of Multi-Criteria Decision Analysis 2000;9:127 – 37. CEAA. The Canadian Environmental Assessment Act. Ottawa: Government of Canada; 1992. CEAA, Canadian Environmental Assessment Agency. Cabinet directive on the environmental assessment of policy, plan and program proposals. Ottawa: Minister of Supply and Services; 1999. Coxon AP. The user’s guide to multidimensional scaling. New Hampshire: Hienemann Educational Books; 1982. Culhane PJ, Friesma HP, Beecher JA. Forecasts and environmental decision-making: the content and predictive accuracy of environmental impact statements. Colorado: Westview Press; 1987. Curran JM, Wood C, Hilton M. Environmental appraisal of UK development plans: current practice and future directions. Environment and Planning. B, Planning and Design 1998;25:411 – 33. Dalkey NC. Towards a theory of group estimation. In: Linstone H, Turoff M, editors. The Delphi method: techniques and applications. London: Addison-Wesley; 1975. DFAIT. Handbook for conducting environmental assessments of trade negotiations. Ottawa: Department of Foreign Affairs and International Trade Canada; 2002. DoE. Environmental appraisal of development plans: a good practice guide. London: HMAO; 1993. Fischer T. Strategic environmental assessment performance criteria—the same requirements for every assessment? Journal of Environmental Assessment Policy and Management 2002;4(1):83 – 99. Gokhale AA. Environmental initiative prioritization with a Delphi approach: a case study. Environmental Management 2001;28(2):187 – 93. Golden BL, Wasil EA, Harker PT, editors. The analytic hierarchy process: applications and studies. New York: Springer-Verlag; 1989. Gonzalez MV. Environmental uncertainty, futures studies, and strategic planning. Technological Forecasting and Social Change 1992;42:335 – 49. Goyal SK, Desphande VA. Comparison of weight assignment procedures in evaluation of environmental impacts. Environmental Impact Assessment Review 2001;21:553 – 63. Hazell S, Benevides H. Toward a legal framework for SEA in Canada. In: Partidario MR, Clark C, editors. Perspectives on strategic environmental assessment. New York: Lewis; 2000. Huylenbroeck GV, Coppens A. Multicriteria analysis of conflicts between rural development scenarios in the Gordon District, Scotland. Journal of Environmental Planning and Management 1995; 38(3):393 – 407. IAIA. Strategic environmental assessment performance criteria. Special publication series, vol. 1; 2002. Kontic B. Why are some experts more credible than others? Environmental Impact Assessment Review 2000;20(1):427 – 34. Leitch JA, Leistritz FL. Delphi analysis: a technique for identifying and ranking environmental and natural resource policy issues. The Environmental Professional 1984;6:32 – 40. Locke J, Storey K. Socioeconomic assessment auditing: a Hibernia case study. In: Sinclair J, editor. Proceedings, Society of Petroleum Engineers 3rd international conference on health, safety, and environment in oil and gas exploration. New Orleans, LA, 9 – 12 June; 1997. Ludlow J. Delphi inquiries and knowledge utilization. In: Linstone HA, Turoff M, editors. The Delphi method: techniques and applications. London: Addison-Wesley; 1975. Malczewski G. GIS and multicriteria decision analysis. New York: Wiley; 1999. Mar BW, Lettenmaier DP, Horner R, Richey JS, Palmer RN, Millard SP, et al. Sampling design for aquatic ecological monitoring. Seattle, Washington: Department of Civil Engineering, University of Washington; 1985. Morgan RK, Onorio KR. SEA of the Neiafu Master Plan, Vava’u, Tonga. In: Partidario MR, Clark R, editors. Perspectives on strategic environmental assessment. New York: Lewis; 2000. p. 197 – 214. Nitz T, Brown AL. SEA must learn how policy making works. Journal of Environmental Assessment Policy and Management 2000;3(3):329 – 492.

24

B.F. Noble / Environ. Impact Asses. Rev. 24 (2004) 3–25

Noble BF. Strategic environmental assessment: what is it and what makes it strategic? Journal of Environmental Assessment Policy and Management 2000;2(2):203 – 24. Noble BF. Strategic environmental assessment of Canadian energy policy. Impact Assessment and Project Appraisal 2002;20(3):177 – 88. Noble BF, Storey K. Towards a structured approach to strategic environmental assessment. Journal of Environmental Assessment Policy and Management 2000;3(4):483 – 508. Parente FJ, Anderson JK, Myers P, O’Brien T. An examination of factors contributing to Delphi accuracy. Journal of Forecasting 1984;3:173 – 82. Partidario MR. Elements of an SEA framework—improving the added-value of SEA. Environmental Impact Assessment Review 2000;20:647 – 63. Pastakia CMR, Jensen A. The rapid impact assessment matrix (RIAM) for EIA. Environmental Impact Assessment Review 1998;18(5):461 – 80. Richey JS, Mar BW, Horner R. The Delphi technique in environmental assessment: implementation and effectiveness. Journal of Environmental Management 1985;21:135 – 46. Rohrbaugh J. Improving the quality of group judgment: social judgement analysis and the Delphi technique. Organizational Behaviour and Human Performance 1979;24:73 – 92. Rowe G, Wright G, Bolger F. Delphi: a reevaluation of research and theory. Technological Forecasting and Social Change 1991;39:139 – 55. Saaty TL. A scaling method for priorities in hierarchical structures. Journal of Mathematical Psychology 1977;15:243 – 81. Saaty TL. The analytic hierarchy process: planning, priority, setting, and resource allocation. New York: McGraw-Hill; 1980. Saaty TL. That is not the analytic hierarchy process: what the AHP is and what it is not. Journal of Multi-Criteria Decision Analysis 1997;6(6):324 – 35. Saaty TL, Vargas LG. The logic of priorities: applications in business, energy, health, and transportation. Boston: Kluwer-Nijhoff; 1982. Sadler B. Environmental assessment in a changing world. Final report of a study on the international effectiveness of environmental assessment. Canadian Environmental Assessment Agency and the International Association for Impact Assessment; 1996. Scheele DS. Reality construction as a product of Delphi interaction. In: Linstone HA, Turoff M, editors. The Delphi method: techniques and applications. Boston, MM: Addison-Wesley; 1975. Scheibe M, Skutsh M, Schofer J. Experiments in Delphi methodology. In: Linstone HA, Turoff M, editors. The Delphi method: techniques and applications. Boston, MA: Addison-Wesley, 1975. Sobral M, Hipel K, Farquhar GJ. A multi-criteria model for solid waste management. Journal of Environmental Management 1981;12(2):97 – 110. Stewart TR, Glantz MH. Expert judgment and climate forecasting: a methodological critique of ‘‘Climate Change to the Year 2000’’. Climatic Change 1985;7:159 – 83. Therivel R. Systems of strategic environmental assessment. Environmental Impact Assessment Review 1993;13(3):145 – 68. Therivel R, Wilson E, Thompson S, Heaney D, Pritchard D. Strategic environmental assessment. London: Earthscan Publications; 1992. Transport Canada. Strategic environmental assessment at Transport Canada: policy statement. Ottawa: Transport Canada; 2001. Tukey JW. Exploratory data analysis. Reading, MA: Addison-Wesley; 1977. Turoff M. The design of a policy Delphi. Technological Forecasting and Social Change 1970; 2(2):84 – 101. Turoff M. Delphi in government planning. In: Linstone HA, Turoff M, editors. The Delphi method: techniques and applications. Boston, MA: Addison-Wesley, 1975. Velleman PF, Hoaglin DC. Applications, basics, and computing of exploratory data analysis. Boston: Duxbury Press; 1981. Verheem R, Tonk J. Strategic environmental assessment: one concept, multiple forms. Impact Assessment and Project Appraisal 2000;19(3):177 – 82.

B.F. Noble / Environ. Impact Asses. Rev. 24 (2004) 3–25

25

Welty G. Problems of selecting experts for Delphi exercises. Academy of Management Journal 1972;15:121 – 4. Bram Noble is an Assistant Professor in the Department of Geography at the University of Saskatchewan, Canada. His research interests are in environmental impact assessment and management, particularly strategic environmental assessment and decision-making.