Evaluating participatory research: Framework, methods and implementation results

Journal of Environmental Management 157 (2015) 311e319

Contents lists available at ScienceDirect

Journal of Environmental Management journal homepage: www.elsevier.com/locate/jenvman

Evaluating participatory research: Framework, methods and implementation results Alex Smajgl a, b, *, John Ward c, d a

Mekong Region Futures Institute, 1655/340 Petchaburi Rd, Bangkok 10400, Thailand CSIRO Ecosystem Sciences, James Cook Drive James Cook University, Douglas Campus, Townsville QLD 4811, Australia c Mekong Region Futures Institute, Naga House, Vientiane, Lao Democratic People's Republic d CSIRO Ecosystem Sciences, 41 Boggo Road, Dutton Park QLD 4102, Australia b

a r t i c l e i n f o

a b s t r a c t

Article history: Received 11 December 2013 Received in revised form 26 February 2015 Accepted 9 April 2015 Available online 2 May 2015

This paper describes a structured participatory process and associated evaluation protocol developed to detect systems learning by decision makers involved in the management of natural resources. A series of facilitated participatory workshops were conducted to investigate learning when decision makers and influencers were confronted with the multiple, complex interactions arising from decisions concerned with the nexus of water, food and energy security. The participatory process and evaluation of learning were trialled in the Greater Mekong Subregion (GMS), where integrated scientific evidence was systematically presented to challenge existing beliefs concerned with the effectiveness of proposed policy actions and development investments. Consistent with theoretical propositions, individually held values, beliefs and attitudes were deployed as the primary factors (and psychometrics) that underpin and influence environmental management decision making. Observed and statistically significant changes in the three psychometrics expressed by decision makers in response to the facilitated presentation of scientific evidence during the participatory process, provided supportive evidence of systems learning and the evaluation protocol. Crown Copyright © 2015 Published by Elsevier Ltd. All rights reserved.

Keywords: Evaluation Monitoring Participatory research Water, food and energy nexus

1. Introduction The outcomes of participatory research and participatory modelling in particular are increasingly scrutinised to assess their influence on decision making processes (Chess, 2000; Jones et al., 2009; Kellert et al., 2000; Perez et al., 2011; Plummer and Armitage, 2007). This paper describes a structured participatory process and protocol developed to facilitate and evaluate systems learning by decision makers concerned with the management of environmental resources. The system investigated focussed on policy initiatives and development investments affecting, and affected by, the nexus of national water, food and energy security (Smajgl and Ward, 2013a, b). The evaluation protocol relies on the Challenge and Reconstruct Learning (ChaRL) Framework (Smajgl

* Corresponding author. Mekong Region Futures Institute, 1655/340 Petchaburi Rd, Bangkok 10400, Thailand. E-mail addresses: [email protected] (A. Smajgl), john.ward@ mekongfutures.com (J. Ward). http://dx.doi.org/10.1016/j.jenvman.2015.04.014 0301-4797/Crown Copyright © 2015 Published by Elsevier Ltd. All rights reserved.

and Ward, 2013a; see Fig. 3) and was trialled during a series of facilitated workshops attended by decision makers and influencers from government agencies, civil society, NGO's and the private sector operating in the Greater Mekong Subregion (GMS). The participatory process and evaluation utilised a mixed method approach to facilitate a formalised learning process for GMS decision makers. Decision maker learning was detected and evaluated via observed changes in individually held values, beliefs and attitudes, amended in response to the systematic presentation of scientific evidence. Scientific evidence, generated as part of the research, was integrated as an agent based simulation and used as a modelling device to challenge existing beliefs concerned with the effectiveness of proposed policy actions and development investments. We first summarise literature based insights on the status of participatory evaluation. We then describe the Greater Mekong Subregion, followed by a description of the research goals, process and theoretical underpinning as the three pillars necessary to design an effective research evaluation. A detailed explanation of the monitoring and evaluation methodology follows and the paper concludes with a discussion of the observed results.

312

A. Smajgl, J. Ward / Journal of Environmental Management 157 (2015) 311e319

Fig. 1. The wider Mekong region and the five local study areas.

2. Current evaluations of participatory processes Evaluations of participatory processes are essential to assess either the effectiveness of a specific participatory technique or to compare the relative effectiveness of methodological variants. However, the effectiveness of participatory processes is reliant on a positive, existing attitude towards learning among stakeholders. In most cases of participatory research multiple methods are combined to conduct the assessment defined by stakeholders (Jones et al., 2009), creating difficulties in attributing impact to a particular methodological element. The experimental testing of specific participatory protocols requires isolating all other influences and effects of a participatory process with a control group. But in reality deliberative participation does not provide these two conditions, which prohibits a formal comparison of outcomes with and without participatory research. Resolution requires an ex ante design for monitoring and evaluating research impacts. Most existing participatory research evaluation frameworks employ an ex post approach (Connick and Innes, 2003; Larson et al., 2010; Petts, 2001; Plummer and Armitage, 2007). Non-participatory research is generally characterised by less stakeholder

Fig. 2. The Challenge and Reconstruct Learning (ChaRL) framework (Smajgl et al., in press).

interactions allowing for a more controlled monitoring. In addition to (a) research goals, evaluation methods also depend on (b) the type of research process and (c) the underpinning theory, highlighted by the evaluation protocol described in Jones et al. (2009) and Perez et al. (2011). For instance, observationbased techniques, central to the work presented in this paper, require participatory processes and could not be carried out in a more traditional, non-participatory science processes. Theoretical underpinnings provide a third perspective for designing the monitoring approach by defining the target variables. For instance, if the underpinnings include specific theories on human cognition and the research goal is to facilitate learning among decision makers, theory identifies the measurement of values, beliefs and attitudes as critical for evaluating research impacts (Schwartz, 1992; Schwartz and Bilsky, 1987; Stern et al., 1999, 1998). In a meta study Boaz et al. (2008) reviewed 156 research publications finding examples of 17 categories of applied data gathering methods. In order of ranking, ex-post tracing (101 cases) was found to be the most commonly applied method to elicit data, followed by semi-structured exit interviews (57), case study analysis (56), documentary analysis (45), publication-related analysis (37), and surveys (30). Research impacts and data interpretation were evaluated according to 14 different types of frameworks. The most common framework relied on economic metrics. Kristjanson and Thornton (2004) focused on studies undertaken by the International Livestock Research Institute and found a similar array of 12 evaluation methods. In the domain of social simulation, semistructured interviews are often at the core of evaluation methods (Jones et al., 2009; Perez et al., 2011). However, a generic evaluation approach across all cases of participatory research or modelling may not readily correspond with the wide array of research foci and objectives of participatory research (Barreteau et al., 2010; Voinov and Bousquet, 2010). Research objectives can provide guidance for determining evaluation indicators, but according to Nagel and Aenis (2002) are not the only relevant dimension for designing evaluation.

A. Smajgl, J. Ward / Journal of Environmental Management 157 (2015) 311e319

313

Fig. 3. The participatory process and its 5 workshops following the ChaRL protocol.

3. Mekong region case study Australian Aid selected the Greater Mekong Subregion as the case study region as development initiatives being implemented or considered lacked evidence-based, cross-sectoral and transboundary coordination. The Greater Mekong Subregion (GMS) (see Fig. 1), comprised of Vietnam, Cambodia, Lao PDR, Thailand, Myanmar and Yunnan (China), is currently undergoing a period of rapid change largely facilitated through a new era of direct foreign investment. Traditionally most post-colonial investment was provided by donor organisations. In contrast, most contemporary GMS investment emanates from the economic surpluses of China, Thailand and Vietnam as those countries undergo rapid economic development (Ward et al., 2011). Development has elevated the demand for many commodities including electricity, natural resources and manufactured goods. Sourcing and satisfying commodity demand from neighbouring countries has contributed to an increasing connectivity within the GMS. High regional connectivity implies that decisions made in one part of the region are likely to have implications for other parts of the region. Thus, sustainable development of the GMS requires an improved understanding of trans-boundary and cross-sectoral consequences of (sub-) national decisions. This implies an improved understanding (or learning) by decision makers and decision influencers, facilitated by a participatory multi-level process. Critical from the outset was the development of a formal and replicable evaluation protocol capable of generating evidence that would indicate correlations between policy impacts, the participatory process and attendant research. As such, it is important to explain the formative phase of the participatory process and operational elements in more detail. The participatory process targeted an improved decision makers' understanding of the likely, unforeseen impacts of

impending development strategies in the Mekong region. First the participatory process had to engage and connect multiple tiers of governance (Smajgl, 2009) from all affected Mekong countries to capacitate intended dialogue and learning in a context of crosssectoral, cross-scale and transboundary dynamics. Most largescale investments in the Mekong region considered as part of this research are linked to decision making processes implemented at either the provincial or national level. In a scoping study conducted over several months in five Mekong countries, national policy makers, academics and practitioners, primarily concerned with natural resource management, agreed on one impending national decision with likely multi-sectoral and regional consequences. The impending decisions are summarised as: - Yunnan, China: Strategies to reduce rubber monocultures in Xishuangbanna. - Lao PDR: Water development options in the Nam Ngum catchment. - Thailand: Options for irrigated agriculture in Isaan. - Cambodia: Strategies for managing agencies and communities of the Tonle Sap lake to adapt to the consequences of upstream mainstream dams. - Vietnam: Strategies to adapt to sea-level rise in the Mekong Delta. In addition to the national perspective the research considered trans-boundary dynamic processes that connect the five Mekong countries, including migration patterns, hydrological flow, land use change and investment flows. The trans-boundary scale is not only influenced by national decisions but also targeted by supranational agencies (i.e. donor organisations, Mekong River Commission, Asian Development Bank, World Bank), albeit without executive or enforceable statutory powers.

314

A. Smajgl, J. Ward / Journal of Environmental Management 157 (2015) 311e319

4. Methods and case study application 4.1. Design of the participatory process linking science and decision-making The design of the participatory processes systematically followed the Challenge and Reconstruct Learning (ChaRL) framework and aimed to evaluate formalised and facilitated learning among decision makers and decision influencers at both national and supra-national policy levels (Smajgl et al., in print; Smajgl and Ward, 2013a). Fig. 2 illustrates the five workshop steps of the ChaRL framework. The participatory process was organised at two levels. At the local study level, workshops were conducted with district, provincial and national decision makers and decision influencers concerned with setting national policy for food, water and energy security. In parallel, supra-national policy decision makers were invited to participate in an independent but replicated workshop process focussed on the regional repercussions of national policy decisions. The design of the participatory processes is shown in Fig. 3. In the first workshop, participating decision makers developed future visions, articulated as plausible narratives, at both the local and the regional scale and iteratively revised in the second workshop to allow for the incorporation of expected impacts of actions to achieve those visions elsewhere (Foran et al., 2013), see Fig. 3. The resulting narratives were assumed to reflect what key decision makers and decision influencers perceive as desirable, plausible futures at the scale of their respective decision making domain. In workshops three and four participants were exposed to scientific evidence employing methods that introduced increasing levels of complexity. In a first step, participants were presented with the analysis of hydrological, agricultural, and livelihoodrelated consequences of the nominated and impending investment decision at the local scale. In a following step, implications for other parts of the Mekong region were addressed by an expert panel, who identified and analysed both regional implications and possible local study feedbacks (Smajgl and Ward, 2013b). Based on the arguments of Gilbert (2008) and Castellani and Hafferty (2009) social simulation was selected as one of the core methods to consider highly complex interactions and emergent phenomena. The social simulation approach is embedded in and informed by a participatory process to provide the required learning opportunities (Barreteau et al., 2010; Smajgl, 2010; Voinov and Bousquet, 2010). Approximately 70 per cent of participants in workshops one and two continued attendance in subsequent workshops. The social simulation model was also employed at the regional level to integrate hydrological, agricultural and livelihood related dynamics and simulate complex trans-boundary dynamics (see Smajgl et al. (2013) for a technical description of the model). The joint purpose of the mixed methods approach is to provide information to participating decision makers and to reveal prevailing, individually held causal beliefs (Smajgl, 2010). The purpose of the simulation model in particular was to provoke belief statements and was introduced during workshop 4 (Fig. 3). The final simulation-based analysis accounts for stochastic elements and was introduced during the fifth workshops, to illustrate the impacts of single and combined policies and investments proposed by participants to meet the future objectives articulated in the workshop 1 and 2 storylines. Consistent with Bankes (2002) and Sarewitz and Pielke Jr (2007) the modelling is not used as a predictive tool, presenting emerging patterns as alternative heuristics and not as forecasts. The key activity during the fifth workshop (Fig. 3) is the

presentation of all beliefs collated during the previous workshops. This step connects the key components and visualises the core principle of the ChaRL framework (Smajgl and Ward, 2013a; see Fig. 1). In the first workshop five session participants compared and discussed beliefs that were recorded during previous workshops. Beliefs were then compared with the scientific evidence provided earlier. The following discussions were focused on the question: ‘do pending decisions mean that your shared visions are more likely to be achieved, or not’. This step is critical, anchoring the previous ChaRL elements into the contemplated development investments and reframing the visions shared between the key sectors of water, food and energy. The visions provide the necessary normative and jointly agreed foundation to further stakeholder discussion. Prototype methodology for facilitating learning among decision makers has been tested in Indonesia (Smajgl, 2010). While in the case of Indonesia policy influences have been acknowledged by stakeholders, a robust evaluation was absent. The Mekong region project expands on the Prototype methodology and addresses the evaluation deficit. 4.2. Theoretical underpinnings of the evaluation protocol The underlying theoretical understanding is that of a complex adaptive system and that decision makers construct their own heuristics guiding their decision making in a constructed reality (Funtowicz and Ravetz, 1994; Gigerenzer et al., 2000). Furthermore we assume that existing beliefs (or intuitive knowledge) can be challenged and reconstructed through a deliberative process if the normative dimension inherent in proposed (and implemented) investment decisions are revealed and explicitly considered (Habermas, 1990, 2005; Pedersen, 2008). The ChaRL participatory framework combines future visions articulated by participants with the provision of scientific evidence specific to the research focus identified by national decision makers during the scoping study. Via an ensemble of mixed methods the combined effect potentially challenges existing beliefs and stimulates reconstructed learning. A formal and repeatable evaluation of decision maker learning is a critical aspect of this participatory research effort, intended to improve the objective comparison of the effectiveness of methods to facilitate knowledge reconstruction. Constructivism has introduced a paradigm shift in learning evaluation. Constructivism shifts the ‘support of multiple perspectives or interpretations of reality, knowledge construction, and context-rich, experience-based activities’ (Jonassen, 1992, p 137) to the centre of learning processes. The relevance of complexity and the nature of constructivism shape the potential for a growing gap between increasingly complex science and decision making geared to more simplified single sector solutions. The tenuous connection between research outcomes and development decisions implies that minimal contributions will be achieved if improved understanding of the likely, unintended consequences of proposed development is confined to the science community (Sarewitz and Pielke Jr, 2007). Improved systems understanding amongst decision makers, particularly in the context of the water, food and energy nexus, is likely to promote a raised awareness of cross sectoral interactions. The theoretical underpinnings of the research assumes that people construct their version of reality, that many system dynamics are non-linear and complex and that emergent phenomena can differ substantially from the predicted outcomes of linear interactions (Baas and Langton, 1994; Boschetti et al., 2010; Holland, 1998; Roling, 1992; Zellner et al., 2009). Sterman (2008) contends that people are cognitively too constrained to comprehend complex problems, forcing people to develop simplifying heuristics

A. Smajgl, J. Ward / Journal of Environmental Management 157 (2015) 311e319

(Castellani and Hafferty, 2009; Funtowicz and Ravetz, 1994; Todd, 2001). Concurrently, the question emerges as to how learning outcomes can be reliably aggregated and evaluated if they are individually constructed (Jonassen, 1992). Acknowledging the relevance of context in constructivism, an effective evaluation approach could try to trace new knowledge and attendant changes in value and belief orientations introduced by a specific research process. Cognitive psychology emphasises the importance and complex interaction of values, beliefs, norms and attitudes in influencing an individual's disposition towards learning (Ajzen, 1981, 1988; Barry and Oelschlaeger, 1998; Beratan, 2007; Fishbein and Ajzen, 1975; Kaplan, 2000; Schwartz and Bilsky, 1987; Schwartz and Tessler, 1972). Behavioural beliefs relate to the causeeeffect relationship between an individual's action and an observed outcome in the perceived social-ecological system. Causal beliefs are shaped against the backdrop of individual values expressed as attitudes and actions (Ajzen, 1981, 1988). These three components can be directly mapped into prominent conceptualisations of human behaviour, such as Ajzen's Theory of Planned Behaviour (Ajzen, 1981). As shown in Fig. 4 intentions to make particular decisions change if the underlying assumptions of causeeeffect relationships change (that is causal beliefs or behavioural beliefs), the attitude towards these behaviours changes (that is decision making options), underlying value orientations held by decision makers change (or normative beliefs) and as a consequence subjective norms are amended. Additionally, Ajzen argues that an individual's perceptions of how much of the action arena he/she controls is a key behavioural dimension. Schwartz and Bilsky (1990) developed a theory based method to reliably measure values. Empirically estimated from a set of 56 value items, Schwartz (1992) validated a parsimonious set of 10 value types, stable across sample cohorts and located in two dimensional space when grouped into four higher order clusters (Stern et al., 1998). For example, the value measures of the self transcendence (altruism) and self enhancement (egoism) clusters constitute the opposites of one axis, the conservatism and openness to change clusters the bounds of the orthogonal axis. Schwartz's human values theory has enabled theory based hypothesis testing of relationships within value constructs and between values and belief, attitudes and norm variables (Dietz et al., 1995; Stern et al., 1998). Based on empirical analysis and validity testing of the four clusters, Stern et al. (1998) argue for a brief 15 value item set, primarily focussed on environmental attitudes and behaviours and developed to address the resource demands of conducting a 56 item survey (Table 2). The five scale constructs of the brief value inventory are egoism, altruism, bio-centricity, openness to change and conservatism. Stern et al. (1999) postulate the value-belief-norm theory, which they argue integrates value theory and norm activation theory

315

through a causal chain of variables finally expressed as a propensity towards pro-environmental behaviour. When correlated with discussions regarding livelihood determinants, poverty reduction and behavioural responses to potential changes in livelihood circumstances, measurement of value orientations elicited via the 15 scale item inventory also acts a proxy measure of individual dispositions towards alternate resource development trajectories. Additionally, decision-oriented theories such as Image Theory (Beach, 1993; Beach and Mitchell, 1998) stress the importance of an articulated guiding vision, a factor generally implicit in the normative dimensions of other cognition-related theories. To translate the theoretical basis into practice, a measurable learning exercise requires: 1) explicit articulation of stakeholders' visions of a desirable, plausible future; 2) measurement and recording of extant beliefs; 3) controlled introduction of new knowledge; 4) measurement and recording of changes to beliefs, value orientations and attitudes as a measure of learning. 4.3. Evaluation methodology The key constructs chosen for the evaluation of the participatory process in the Greater Mekong Subregion were (1) beliefs, (2) values, and (3) attitudes. This keeps a narrow focus on individual learning without expanding into elements of organisation or social learning (see for evaluating social learning Albert et al. (2012)). Based on the tested theories presented above we assume that if during the participatory process, one or multiple constructs change, it is likely that the individual modified his/her understanding of likely decision impacts and would decide on an alternate development or investment option. The information treatments presented to stakeholders during workshops 3 and 4 constitute new evidence that either aims to ground-truth assumptions on critical system variables (i.e. how households would respond to prescribed changes in livelihood circumstances, such as new irrigation opportunities) or to introduce into the decision making process higher levels of complexity. Stated beliefs, expressed as key causal statements (if X occurs then Y occurs), were recorded by formally trained observers during workshops three and four. Training for all observers followed a specific, interactive and consistent protocol. All revealed beliefs were explicitly presented during the fifth workshop (see below) and compared with scientific evidence. The final session and the discussion on how prior and alternative heuristics compare allows for eliciting evidence for research impact. According to value-belief-norm theory (Stern et al., 1999) values act as the causal antecedent to behaviour and their elicitation provides an empirical basis for the individual beliefs articulated through the participatory methods described in Table 1. Individual value orientations of participants were elicited according to responses to the identical set of five value scales and 15 scale items

Fig. 4. Theory of planned behaviour (Ajzen, 1981).

316

A. Smajgl, J. Ward / Journal of Environmental Management 157 (2015) 311e319

Table 1 Examples taken from the reference data for stakeholder beliefs. Workshops 1 & 2

Workshops 3 & 4

Workshop 5

Hard adaptation measures improve livelihood

Most effective response to salinity intrusion combines hard and soft measures It is not necessary to do “big dykes” but small constructions are more important Small-scale infrastructure and land-use change most effective adaptation Riceeaquaculture is best along rivers/canals (mainly fish, less shrimp) Investments need to be prioritised and go to most vulnerable areas

Mekong Delta Provinces, Vietnam Sea dikes will reduce salinity and increase agricultural production New rice varieties help increase production under increasing salinity Sea dikes create risk for rice as storm surges increase

Sea dikes will reduce water quality

Shrimp-rice rotation farming increases farm income

Salinity increase triggers emigration

Upstream dams cause water shortage in the Mekong Delta Xishuangbanna, Yunnan, China PES is an effective conservation instrument

Improved education improves adaptive capacity and livelihoods PES is an effective conservation instrument

Droughts diminish rubber production Rubber plantations cause water shortages Conservation improves tourism income

Droughts diminish rubber production Rubber plantations cause water shortages Investments in NTFP will improve livelihoods

Urbanisation will replace rubber plantations

Planting rubber in downstream countries helps Yunnan's biodiversity

Soft adaptation measures improve livelihoods

PES can expand rubber, needs additional measure to help conservation Droughts diminish rubber production Rubber plantations cause water shortages People's training and education critical for achieving conservation goals PES should involve payments from downstream countries for converting rubber

Table 2 Summary and statistical tests of participant responses to the values questionnaire, workshops one and five. Scale and scale items

Biocentric Respecting the earth (harmony with other species) Unity with nature (fitting into nature) Protecting the environment (preserving nature) Open to change A varied life (filled with challenge, novelty and change) Curious (interested in everything, exploring) An exciting life (stimulating experiences) Conservative (family security) Honouring parents and elders (showing respect) Family security (safety for loved ones) Self discipline (self restraint and resistance to temptation) Altruism A world at peace (free of war and conflict) Social justice (correcting injustice, care for the weak) Equality (equal opportunity for all) Self interest Wealth (material possessions, money) Authority (the right to lead or command) Influential (having an impact on people and events) Sum supreme importance

Visions (workshop 1 n ¼ 141)

Challenge beliefs (workshop 5 n ¼ 109)

Test statistics

Meana (s.e)

Mean (s.e)

t value

Mean rank

16.51 (.167)

Mean rank

15.00 (.186) 146.00 147.13 145.86

15.07 (.216)

6.064** 98.98 97.52 99.16

13.39 (.190) 139.35 149.34 145.00

16.92 (.118)

5.635**

15.39 (.146)

16.83 (.130)

8.238**

15.36 (.182)

4323.5** 4282.5** 4813.5** 6.743**

103.95 94.62 101.57 12.22 (.273)

143.80 142.12 139.96 8.43 (.343)

4858.0** 4635.0** 5335.5**

99.57 100.06 94.29

142.16 149.37 144.00 14.04 (.256)

5076.5** 5731.0** 4793.5**

107.58 94.67 100.28

145.55 145.17 149.63

5104.0** 5340.5** 4319.0** 4.827**

101.83 104.00 106.79 3.42 (.242)

Mann Whitney U

5645.5** 4935.5** 4911.5** 11.09**

** indicates significant difference at a ¼ 0.05. a represents the mean of the sum of responses to the 3 scales items for each scale.

proposed by Stern et al. (1998) and described in Table 2. The selfcompleted values questionnaire was translated into Mandarin, Lao, Thai, Khmer and Vietnamese and back translated into English in all cases. The initial appraisal of the human values data was intended to detect differences in observed value orientations between workshops one (articulation of the vision narratives), four (introduction of science information) and five (challenged beliefs). Decision makers (n ¼ 388) participating in the first, fourth and fifth local and regional workshops were invited to complete the value item questionnaire. Responses were measured according to a 6 point numerical likert scale following the modification by Stern et al. (1998) of Schwartz (1992), ranging from 1:“opposed to my values” to 4: “of supreme importance” as a life guiding principle. Minus one was later transformed to 1,

0 to 2, 1 to 3, 2 to 4, 3 to 5 and 4 to 6 for statistical analysis. Questionnaire instructions suggested that there are generally only two scale items of supreme importance. The questionnaire response rate was approximately 75e95 percent, depending on the workshop location. Similar to beliefs, attitudes were recorded by trained observers during the entire five workshops. Attitude-type statements included normative statements regarding policy instruments, intervention options or related aspects. 5. Evaluation results Contingent on the constellation of learning objectives, participatory processes and the theoretical underpinnings, the ChaRL evaluation relies on three key components to measure the

A. Smajgl, J. Ward / Journal of Environmental Management 157 (2015) 311e319

influence of the research process on participant learning: beliefs, values and attitudes. The Vietnamese workshops included approximately 45 decision makers and planners from the Department of Natural Resources and the Environment and the Department of Agriculture and Rural Development from eleven Provinces in Vietnam's Mekong Delta, local decision influencers from local universities such as Can Tho University, and central Government officials from the Ministry of Natural Resources and the Environment and the Ministry of Agriculture and Natural Resources. The workshop series in Yunnan brought together around 30 decision makers and planners from the Xishuangbanna Prefecture, the Province Government representatives for Environment and for Agriculture, and central Government representatives from the Ministry of Environment. 5.1. Beliefs and attitudes Tables 1 and 3 provide samples for beliefs and attitudes elicited during the workshop process in the participatory process in the Greater Mekong Subregion. These examples were selected from the reference dataset and grouped into statements made during the visioning process (step 2 of the ChaRL framework, Fig. 2), during the presentation of research results (step 3 of the ChaRL Framework) and during the final workshop (step 4 and 5 of the ChaRL process). 5.2. Values Table 2 summarises the mean and variance of the aggregated likert scale scores of scale items for bio-centricity, openness to change, conservativeness, altruism and ego-centricity for the visions workshop (1) and the challenged beliefs workshop (5). The mean ranks of individual scale items are also described in Table 2. The t test statistics indicate that the visions workshop mean for all five scales is significantly higher (p < 0.05) than the challenged belief workshops and the standard error of the mean observed in the challenge workshop is higher than that estimated for the visions workshop. The variable “sum supreme importance” represents the number of scale items selected by respondents as of supreme importance. The mean of the sum supreme importance estimated for the visions workshop is significantly higher than that of the science and challenge workshops (t ¼ 11.09, p ¼ 0.000). These initial observations indicate that throughout the ChaRL workshop sequence, respondents have selected fewer scale items of supreme importance and been increasingly selective in assigning relative scale item importance. Non-parametric analysis comparing the ranked mean of individual scale item responses from the visions workshop and the challenge belief workshop also indicates significant differences (Mann Whiney U > 4300, p ¼ 0.000).

317

6. Discussion The results presented in Table 1 indicate that some causal beliefs held by workshop participants were successfully challenged. In workshop three held in Vietnam, the debate among policy makers across all tiers of governance and among academic and nonacademic decision influencers was extremely polarised between ‘hard solutions’ (largely targeting investments in water infrastructure such as sea dikes) and ‘soft solutions’ (largely focused on land use change related interventions). During the final workshop challenging held beliefs (or ‘myth-busting’) the same participants moved from articulating a polarised position to statements that revealed the increased benefits of combining hard and soft solutions. The polarisation in the policy debate had created an institutional barrier, impeding the development and implementation of coherent, integrated adaptation strategies to manage sea level rise in the Mekong Delta. In Xishuangbanna the evaluation approach revealed a clear preference for Payment of Ecosystem Services (PES) during workshops three and four, (see Table 1). PES was considered an effective instrument to achieve conservation goals and all participants involved made statements that endorsed the incentive scheme proposed by central and Provincial Governments. After scientific evidence was presented, suggesting that the current scheme could potentially contribute to a further expansion of monoculture rubber, due to deficits in monitoring and enforcement of land use change, beliefs and attitudes changed dramatically. During the final workshop the majority of participants recognised the monitoring and enforcement deficits and emphasised the risks of further rubber expansion inherent in the originally proposed PES scheme. As a control benchmark, the evaluation protocol relied on documented beliefs held by participants that were not challenged through the presentation of research evidence and simulation modelling. For example the belief “droughts diminish rubber production” (Table 1) was not analysed nor challenged and remained constant from workshop 1 through 5. The valency, significance and increased variance of values observed between workshops one and five suggests participants amended their held values from more generalised, ambit orientations (notably the decrease in the sum of supreme importance metric and increased variance) to increased levels of scale item differentiation and heterogenous value orientations. Increased differentiation of values coincides with the recognition of situational diversity and heterogeneity expressed as the amended beliefs in workshop five. The observed changes in values and correlated change in beliefs in the Yunnan and Vietnam workshops are consistent with the value-belief-norm theory postulated by Stern et al. (1999), the theory of planned behaviour (Azjen, 1981) and the psychological structure of human values (Schwartz and Blisky, 1990).

Table 3 Examples taken from the reference data for stakeholder attitudes. Regional group

Yunnan group

Vietnam group

Thailand does not need labour from other countries Science is crucial for policy decisions

The market determines rubber expansion

Impact assessments do not consider shrimp farming.

Rubber plantations in high altitudes should be changed into carbon sink forests. Planting rubber trees at top of mountains is unsuitable.

There is trade-off between rice and shrimp farming for any water structure measures Cost-benefit analysis should be done before the sluice gate in Tien River is built China is not a member of MRC, it is difficult to ask them to optimally operate dams in the upper Mekong Basin Summer-Autumn rice crop could be better adaptive to climate change It is necessary to do more research on tides and on flooding

Those involved in rubber do not consider conservation Individuals decide land use change not government nor society Labour is migrating to urban areas Rural people are conservative

Government needs to provide more technical services. Markets cannot solve the problem of high social cost brought by rubber expansion. The rubber industry should have diversified

318

A. Smajgl, J. Ward / Journal of Environmental Management 157 (2015) 311e319

In summary, the implementation of the described monitoring and evaluation methodology provided supportive evidence for assessing the impact of a learning-focused participatory research process held with natural resource decision makers in the Greater Mekong Subregion. Literature based insights indicate a deficit of standardised evaluation metrics and the absence of a systematic methodological approach (Poteete et al., 2010). Appropriate and effective elicitation methods and evaluation frameworks vary according to a vector of research questions, methodological process and underpinning theory (Boaz et al., 2008; Kristjanson and Thornton, 2004). We contend that the inclusion of monitoring and evaluation at the design phase of participatory research project is critical. Two reasons emphasise the need to design and implement coherent monitoring and evaluation protocols. First, many design aspects of participatory processes imply an analystdetermined choice of research methods. Monitoring provides the necessary data for comparing the effectiveness of alternative methods. Second, compiled evidence will enable a more formal appraisal of the decision influence of participatory processes, compared to traditional research and modelling approaches. The selection and eventual efficacy of monitoring and evaluation methods depend on the research goals, the type of research process and theoretical underpinnings, which defines the target, evaluative variables. As the overall research project goal was to facilitate improved systems learning among decision makers, the design of the ChaRL framework relied on salient theories on human cognition, which identify the measurement of values, beliefs and attitudes as critical for evaluating decision maker learning stimulated through the presentation of research analysis. 7. Conclusion In conclusion, the described evaluation methodology provided supportive evidence of systems learning, observed during a facilitated participatory process conducted with decision makers and decision influencers in the Greater Mekong sub-region. Design of the approach and implementation of the monitoring and evaluation methodology was feasible, demanded modest time and resource inputs and we argue is readily replicable in participatory processes involving multiple and diverse decision makers. The detection of changes in beliefs and value orientation observed for the majority of participants provided the empirical basis to infer that introduced scientific evidence, and the attributes of evidence presentation, influenced actual decision making. Analytical results and evidence produced during the research project were not presented as a singular definitive explanation. In contrast, evidence was integrated and presented as a device to challenge those widely held beliefs that when expressed as specific policy actions, did not align with or were unlikely to contribute to the visions and objectives stated by participating decision makers. The systematic challenging of individually held beliefs introduced sufficient doubt for stakeholders to reorient their held values, reappraise their own causal heuristics manifest as an increased willingness to consider and accept alternate cause-effect assumptions. In contrast to reviewed ex post evaluations, a key lesson arising from the case study analysis is that the tracing of psychometric changes requires monitoring protocols and evaluation metrics to be defined and implemented at process inception. However, learning in complex, natural resource decision environments is acknowledged as multi-factorial. The monitoring and evaluation approach described here does not impute all of these factors and excludes, for instance activities by other projects or stakeholders external to the workshop process. Structured interviews with influencers or observers external to the project could help identify potentially significant omitted factors and variables.

However, resource constraints prevented the monitoring and evaluation of external decision influencing factors, introducing caveats regarding the comprehensiveness of the monitoring approach. Implementing additional monitoring and evaluation learning dimensions, despite substantial resource demands, would have strengthened the evidence for evaluating science impacts even further. Based on the evidence presented, we do contend however that the values, beliefs and attitude psychometrics are important pre-requisites of alternate multi-factor evaluation indices. Acknowledgements This work was supported by the DFAT CSIRO Research for Development Alliance. The authors are deeply indebted to the research teams of the Kunming Institute of Botany and Can To University, who helped collating and translating the data presented in this paper. The authors wish to thank the 383 workshop participants for their sustained enthusiasm and support and two anonymous reviewers for their helpful comments. References Ajzen, I., 1981. The theory of planned behaviour. Organ. Behav. Hum. Decis. Process. 50, 179e211. Ajzen, I., 1988. Attitudes, Personality and Behavior. Open University Press, Milton Keynes. Albert, C., Zimmermann, T., Knieling, J., von Haaren, C., 2012. Social learning can benefit decision-making in landscape planning: Gartow case study on climate change adaptation, Elbe valley biosphere reserve. Landsc. Urban Plan. 105, 347e360. Baas, N., Langton, C., 1994. Emergence, Hierarchies and Hyperstructure 156. Addison-Wesley, New York. Bankes, S.C., 2002. Tools and techniques for developing policies for complex and uncertain systems. Proc. Natl. Acad. Sci. U. S. A. 99, 7263e7266. Barreteau, O., Bots, P.W.G., Daniell, K.A., 2010. A framework for clarifying “participation” in participatory research to prevent its rejection for the wrong reasons. Ecol. Soc. 15, 1. Barry, D., Oelschlaeger, M., 1998. A science for survival: values and conservation biology. Conserv. Biol. 10, 905e911. Beach, L.R., 1993. Image theory: an alternative to normative decision theory. In: McAlister, L., Rothschild, M.L. (Eds.). Advances in Consumer Research Association for Consumer Research, Provo, UT, pp. 235e238. Beach, L.R., Mitchell, T.R., 1998. The basics of image theory. In: Beach, L.R. (Ed.), Image Theory: Theoretical and Empirical Foundations. Lawrence Erlbaum Associates, Mahwah, NJ., pp. 3e18. Beratan, K.K., 2007. A cognition-based view of decision processes in complex socialeecological systems. Ecol. Soc. 12, 27. Boaz, A., Fitzpatrick, S., Shaw, B., 2008. Assessing the Impact of Research on Policy: a Review of the Literature for a Project on Bridging Research and Policy through Outcome Evaluation. King's College London. University of London, London, UK. Boschetti, F., Hardy, P.-Y., Grigg, N., Horwitz, P., 2010. Can we Learn How Complex Systems Work? Emergence. Complexity and Organization. Castellani, B., Hafferty, F., 2009. Sociology and Complexity Science: a New Field of Inquiry. Springer, Berlin, Heidelberg. Chess, C., 2000. Evaluating environmental public participation: methodological questions. J. Environ. Plan. Manag. 43, 769e784. Connick, S., Innes, J.E., 2003. Outcomes of collaborative water policy making: applying complexity thinking to evaluation. J. Environ. Plan. Manag. 46, 177e197. Dietz, T., Stirling Frisch, A., Kalof, L., Stern, P.C., Guagnano, G.A., 1995. Values and vegetarianism. An Explor. Analysis Rural Sociol. 60, 533e542. Fishbein, M., Ajzen, I., 1975. Belief, attitude, intention, and behaviour: an introduction to theory and research. Addison-Wesley, Reading. Foran, T., Ward, J., Kemp-Benedict, E., Smajgl, A., 2013. Developing detailed foresight narratives: a participatory technique from the Mekong region. Ecol. Soc. 18, 6. Funtowicz, S.O., Ravetz, J.R., 1994. Emergent complex systems. Futures 26, 568e582. Gigerenzer, G., Todd, P.M., Group, A.B.C.R., 2000. Simple Heuristics that make us Smart. Oxford University Press, Oxford. Gilbert, N., 2008. Agent-based Models. SAGE Publications, Los Angeles. Habermas, J., 1990. Moral Consciousness and Communicative Action. Polity Press, Cambridge. Habermas, J., 2005. Concluding comments on empirical approaches to deliberative politics. Acta Polit. 40, 384e392. Holland, J., 1998. Emergence: from Chaos to Order 495. Oxford University Press, Oxford. Jonassen, D.H., 1992. Evaluating constructivistic learning. In: Duffy, T.M., Jonassen, D.H. (Eds.), Constructivism and the Technology of Instruction: a

A. Smajgl, J. Ward / Journal of Environmental Management 157 (2015) 311e319 Conversation. Lawrence Erlbaum Associates, Hillsdale, New Jersey Hove and London, pp. 137e148. Jones, N.A., Perez, P., Measham, T.G., Kelly, G.J., D'Aquino, P., Daniell, K., Dray, A., Ferrand, N., 2009. Evaluating participatory modeling: developing a framework for cross-case analysis. Environ. Manag. 44, 1180e1195. Kaplan, S., 2000. Human nature and environmentally responsible behavior. J. Soc. Issues 56, 491e508. Kellert, S.R., Mehta, J.N., Ebbin, S.A., Lichtenfeld, L.L., 2000. Community natural resource management: promise, rhetoric, and reality. Soc. Nat. Resour. 13, 705e715. Kristjanson, P.M., Thornton, P.K., 2004. Methodological challenges in evaluating impact of cropelivestock interventions. In: Williams, T.O., Tarawali, S.A., Hiernaux, P., Fernandez-Rivera, S. (Eds.), Sustainable Crop-lifestock Production for Improved Livelihoods and Natural Resource Management in West Africa. ILRI, Ibadan, Nigeria. Larson, S., Measham, T.G., Williams, L.J., 2010. Remotely engaged? Towards a framework for monitoring the success of stakeholder engagement in remote regions. J. Environ. Plan. Manag. 53, 827e845. Nagel, U.J., Aenis, T., 2002. Towards Impact Assessment of Transdisciplinary Research Programs, Fifth IFSA European Symposium e Farming and Rural Systems Research and Extension. Local Identities and Globalisation, Florence, Italy, pp. 19e29. Pedersen, J., 2008. Habermas' method: rational reconstruction. Philosophy Soc. Sci. 38, 457e485. Perez, P., Aubert, S., Dare, W., Ducrot, R., Jones, N., Queste, J., Trebuil, G., Van Paassen, A., 2011. Assessment and monitoring of the effects of the ComMod approach. In: Etienne, M. (Ed.), Companion Modelling: a Participatory Approach to Support Sustainable Development (Editions Quae, Versaille Cedex, France). Petts, J., 2001. Evaluating the effectiveness of deliberative processes: waste management case-studies. J. Environ. Plan. Manag. 44, 207e226. Plummer, R., Armitage, D., 2007. A resilience-based framework for evaluating adaptive co-management: linking ecology, economics and society in a complex world. Ecol. Econ. 61, 62e74. Poteete, A.R., Janssen, M.A., Ostrom, E., 2010. Working Togeher: Collective Action, the Commons, and Multiple Methods in Practice. Princeton University Press, Princeton. Roling, N.G., 1992. The emergence of knowledge systems thinking 860. Knowl. Policy: Int. J. Knowl. Transf. Util. 5, 42e64. Sarewitz, D., Pielke Jr., R.A., 2007. The neglected heart of science policy: reconciling supply of and demand for science. Environ. Sci. Policy 10, 5e16. Schwartz, S.H., 1992. In: Zanna, M. (Ed.), Universals in the Content and Structure of Values: Theory and Empirical Tests in 20 Countries. Advances in Experimental Social Psychology Academic Press, New York, pp. 1e65. Schwartz, S.H., Bilsky, W., 1987. Toward a psychological structure of human values.

319

J. Personality Soc. Psychol. 53, 550e562. Schwartz, S.H., Bilsky, W., 1990. Toward a theory of the universal content and structure of values: extensions and cross cultural replications. J. Personality Soc. Psychol. 58, 878e891. Schwartz, S.H., Tessler, R.C., 1972. A test of a model for reducing measured attitudebehaviour discrepancies. J. Personality Soc. Psychol. 24, 225e236. Smajgl, A., 2009. Accounting for multiple governance scales in integrated modelling and decision support. In: Proctor, W., Van Kerkhoff, L., Hatfield Dodds, S. (Eds.), Reflecting on Integrated Mission Directed Research: Learning from Experience in Environmental and Natural Resource Management. CSIRO Publishing, Collingwood, pp. 1e15. Smajgl, A., 2010. Challenging beliefs through multi-level participatory modelling in Indonesia. Environ. Model. Softw. 25, 1470e1476. Smajgl, A., Egan, S., Kirby, M., Mainuddin, M., Ward, J., Kroon, F., 2013. The Mekong Region Simulation (Mersim) Model - Design Document CSIRO Climate Adaptation Flagship (Townsville). Smajgl, A., Foran, T., Dore, J., Ward, J., Larson, S., 2015. Visions, beliefs and transformation: exploring cross-sector and trans-boundary dynamics in the wider Mekong region. Ecol. Soc. (in press). Smajgl, A., Ward, J., 2013a. A framework for bridging science and decision making. Futures 52, 52e58. Smajgl, A., Ward, J., 2013b. The Water-food-energy Nexus in the Mekong Region. Springer (New York). Sterman, J.D., 2008. Risk communication on climate: mental models & mass balance. Science 322, 532e533. Stern, P.C., Dietz, T., Abel, T., Guagnano, G.A., Kalof, L., 1999. A value belief norm theory of support for social movements: the case of environmental concern. Hum. Ecol. Rev. 6, 81e97. Stern, P.C., Dietz, T., Guagnano, G.A., 1998. A brief inventory of values. Educ. Psychol. Meas. 58, 984e990. Todd, P.M., 2001. Fast and Frugal heuristics for environmentally bounded minds. In: Gigerenzer, G., Selten, R. (Eds.), Bounded Rationality: the Adaptive Toolbox. The MIT Press, Cambridge, Massachusetts, pp. 51e70. Voinov, A., Bousquet, F., 2010. Modelling with stakeholders. Environ. Model. Softw. 25, 1168e1281. Ward, J., Foran, T., Smajgl, A., Bouapao, L., Pech, S., Xing, L., 2011. Mekong energy metabolism: connecting energy demand into the Nexus of food-water-energy security. In: Moinuddin, H. (Ed.), International Conference on GMS 2020: Balancing Economic Growth and Environmental Sustainability. ADB, Manila, Bangkok. Zellner, M.L., Page, S.E., Rand, W., Brown, D.G., Robinson, D.T., Nassauer, J., Low, B., 2009. The emergence of zoning policy games in exurban jurisdictions: informing collective action theory. Land Use Policy 26, 256e367.