Evaluating Meaningful Watershed Educational Experiences: An exploration into the effects on participating students’ environmental stewardship characteristics and the relationships between these predictors of environmentally responsible behavior

G Model

JSEE-474; No. of Pages 14 Studies in Educational Evaluation xxx (2013) xxx–xxx

Contents lists available at ScienceDirect

Studies in Educational Evaluation journal homepage: www.elsevier.com/stueduc

Evaluating Meaningful Watershed Educational Experiences: An exploration into the effects on participating students’ environmental stewardship characteristics and the relationships between these predictors of environmentally responsible behavior Michaela Zint a,*, Anita Kraemer b, Giselle Kolenic c a

School of Natural Resources and Environment, University of Michigan, Dana Building, 440 Church Street, Ann Arbor, MI 48109-1041, United States eeEvaluations, 1605 Park Grove Ave., Catonsville, MD21228, United States c Center for Statistical Consulting and Research, University of Michigan, Rackham Building, 915 E. Washington Street, Ann Arbor, MI 48109-1070, United States b

A R T I C L E I N F O

A B S T R A C T

Article history: Received 5 June 2013 Accepted 25 July 2013

This study evaluated Meaningful Watershed Educational Experiences (MWEEs) funded by the National Oceanic and Atmospheric Administration’s Chesapeake Bay Watershed Education and Training (B-WET) grant program. It also empirically explores the relationships between predictors of environmentally responsible behavior (ERB) in the Hines, Hungerford, and Tomera (1986/1987) and Hungerford and Volk (1990) behavior models. Multilevel analyses identified associations between eight environmental stewardship characteristics, as well as between these outcomes and (1) participation in MWEEs (sample: 258 students in 20 treatment classes, 193 students in 12 comparison classes matched by grade) and (2) specific MWEE instructional practices (sample: 434 students in 29 treatment classes). Students who participated in MWEEs scored significantly higher in five of eight characteristics (i.e., knowledge of ecology, issues, and actions, individual locus of control, intention to act) than those in the comparison group. Students who were engaged in the science inquiry steps of analyzing data or reflection and those who participated in more of certain types of environmental actions also scored significantly higher in a greater number of environmental stewardship characteristics than students who did not have these experiences. Results suggest that MWEEs are likely to increase ERBs but are not reaching their full potential. Tests of the relationships between the variables in Hines et al. (1986/1987) and Hungerford and Volk (1990) models confirm that they predict a high amount of variance in intention to act and suggest that environmental stewardship characteristics are likely to interact in complex ways. ß 2013 Elsevier Ltd. All rights reserved.

Keywords: Program evaluation Environmental education Outdoor education Behavior Multilevel analysis Path analysis

Introduction Among the primary questions environmental educators, their funders, and supporters have is to what extent environmental education programs foster environmentally responsible behavior (ERB)1 and which instructional practices this outcome can be attributed to. The first question is of interest because the ultimate goal of environmental education is to foster behaviors that contribute to conserving, protecting, and restoring the

* Corresponding author. Tel.: +1 734 763 6961. E-mail address: [email protected] (M. Zint). 1 By environmental responsible behaviors we refer to any actions that directly or indirectly contribute to conserving, protecting, or restoring the environment. Environmental educators ‘‘foster’’ these behaviors by preparing individuals to independently analyze and respond to environmental issues in an informed manner. We are not suggesting the use of, or referring to persuasive, manipulative approaches to behavior change.

environment (UNESCO, 1978). Answers to the second question are critical to informing environmental education practice, by identifying the types of instruction most likely to lead to behavioral outcomes. Despite the interest in these questions, few studies are available to help answer them. Four syntheses of environmental education research and program evaluations, which reviewed work published between 1971 and 2008 (Leeming, Dwyer, Porter, & Coburn, 1993; Rickinson, 2001; Zelezny, 1999; Zint, 2012), identified only seventeen studies exploring the effects of environmental education on elementary and secondary students’ behavioral outcomes (Chawla & Derr, 2012). In addition, these studies often do not provide details about the environmental education programs they examined or how these programs were implemented (Zint, 2012). As a result, they offer few insights into the program characteristics or practices to which behavioral outcomes can be attributed. Nonetheless, based on the limited information that is provided in these studies, the authors of the aforementioned reviews conclude

0191-491X/$ – see front matter ß 2013 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.stueduc.2013.07.002

Please cite this article in press as: M. Zint, et al.. Evaluating Meaningful Watershed Educational Experiences: An exploration into the effects on participating students’ environmental stewardship characteristics and the relationships between these predictors of environmentally responsible behavior. Studies in Educational Evaluation (2014), http://dx.doi.org/10.1016/j.stueduc.2013.07.002

G Model

JSEE-474; No. of Pages 14 M. Zint et al. / Studies in Educational Evaluation xxx (2013) xxx–xxx

2

that longer programs are more effective in promoting ERBs than shorter programs (Rickinson, 2001; Zelezny, 1999; Zint, 2012), as are programs that engage students in active, experiential learning experiences including field trips, service learning, and investigations of local environmental issues (Leeming et al., 1993; Zelezny, 1999; Zint, 2012). Environmental education behavior theories Environmental education researchers have conducted few studies of the effects of environmental education on ERB or practices to which these outcomes can be attributed. However, they were among the first to identify potential determinants of ERB and explore the relationships between these predictors and ERB. Notably, almost 30 years ago, Hines et al. (1986/1987) published their influential meta-analysis of 128 studies assessing variables associated with ERBs. Their analysis identified reliable predictors of ERB and calculated the strengths between these variables and ERB (correlation range: .15 to .49). The model proposed by Hines et al. (1986/1987) based on their findings suggests that intention to act and situational factors directly determine ERB. Intention to act is, in turn, viewed as predicted directly by cognitive and other individual predictors (Fig. 1). Hungerford and Volk (1990) subsequently proposed a revised behavioral model for environmental educators based on Hines et al. (1986/1987) as well as research by their other students. This revised model does not include situational factors, as they are beyond the control of environmental educators and nests the remaining predictors of ERB within entry-level, ownership, and empowerment variables (Fig. 2). Alternative behavior theories have been introduced since that time (Heimlich & Ardoin, 2008). However, to the best of the authors’ knowledge the models proposed by Hines et al. (1986/ 1987) and Hungerford and Volk (1990) have most influenced environmental education practice. More recent evidence suggests that their popularity is warranted. For example, a recent metaanalysis of studies assessing variables associated with ERBs published since 1995 found mean correlations similar to those reported by Hines et al. (1986/1987) (Bamberg & Mo¨ser, 2007). In addition, research on the effectiveness of ‘‘Investigating and Evaluating Environmental Issues and Actions’’ (IEEIA), a form of instruction developed by Hungerford and Volk to target the variables in their behavior model, consistently finds improvements in students’ predictors of ERB and ERB itself (Marcinkowski, 2004; Volk & McBeth, 2012).

Environmental stewardship characteristics The predictors of ERB included in the models proposed by Hines et al. (1986/1987) and Hungerford and Volk (1990) are referred to as environmental stewardship characteristics in this article. Although some of their names differ, variables in both models include knowledge of issues and actions, environmental sensitivity, locus of control, personal responsibility, and intention to act. Some predictors, such as knowledge of ecology, however, are only included in one model and not the other. Knowledge of issues and actions refers to individuals’ awareness and understanding of environmental problems and how to engage in actions that help address these problems. Environmental sensitivity refers to individuals ‘‘empathic’’ feeling or attitude toward the environment (Hungerford & Volk, 1990, p. 11). Locus of control refers to individuals’ belief about the extent to which they can bring about change through their actions. As suggested by Hines et al. (1986/1987, p. 4), this variable may be described as individuals’ ‘‘efficacy;’’ i.e., the belief that their behaviors can help to address a particular environmental issue. Another distinction is made between individual and group locus of control with the former referring to individuals’ belief that they can make a difference on their own and the latter that they can make a difference by working collaboratively with others (Nowak, Wilke, Marcinkowski, Hungerford, & McKeown-Ice, 1995; Volk & McBeth, 2012). Personal responsibility refers to moral norms, or the feeling that one has a duty to protect the environment. Intention to act is an expression of willingness and commitment to engage in a particular behavior. Knowledge of ecology refers to the understanding individuals have about ecological and related natural science concepts and principles. The present study The present study was conducted to evaluate the effectiveness of the Bay Watershed Education and Training (B-WET) program administered by the National Oceanic and Atmospheric Administration (NOAA) Chesapeake Bay Office. NOAA is among the U.S. federal agencies with an environmental mission (http://www.ppi.noaa.gov/mission/) which it strives to achieve in part through funding environmental education. One of NOAA’s environmental education grant programs is B-WET. To date, organizations in seven US regions have received B-WET funding to provide Meaningful Watershed Educational Experiences (MWEEs) for students or professional development for teachers. As a result of these

Situational factors

Action skills

Knowledge of action strategies Knowledge of issues

Intention to act

Attitudes Locus of control (vs. Control) Personal responsibility (vs. Responsibiity)

Responsible environmental behavior

Personality factors

Fig. 1. ‘‘Proposed Model of Responsible Environmental Behavior’’. ß (2013) From analysis and synthesis of research on environmental behavior: A meta analysis by Hines, Hungerford & Tomera. Reproduced by permission of Taylor & Francis Group, LLC (http://www.tandfonline.com).

Please cite this article in press as: M. Zint, et al.. Evaluating Meaningful Watershed Educational Experiences: An exploration into the effects on participating students’ environmental stewardship characteristics and the relationships between these predictors of environmentally responsible behavior. Studies in Educational Evaluation (2014), http://dx.doi.org/10.1016/j.stueduc.2013.07.002

G Model

JSEE-474; No. of Pages 14 M. Zint et al. / Studies in Educational Evaluation xxx (2013) xxx–xxx

Entry-level variables

Major variable ______________________ Environmental sensitivity

Ownership variables

3

Empowerment variables

Major variables ______________________

Major variables ______________________

In-depth knowledge about issues Personal investment in issues and the environment

Knowledge of and skill in using environmental action strategies Locus of control (expectancy of reinforcement) Intention to act

Minor variables ______________________

Minor variables ______________________

Minor variable ______________________

Knowledge of ecology Androgyny Attitudes toward pollution, technology, and economics

Knowledge of the consequences of behavior –both positive and negative A personal commitment to issue resolution

In-depth knowledge about issues

C i t i z e n s h i p B e h a v i o r

Fig. 2. ‘‘Environmental Behavior Model’’. ß (2013) From changing learner behavior through environmental education by Hungerford & Volk. Reproduced by permission of Taylor & Francis Group, LLC (http:// www.tandfonline.com).

MWEEs, NOAA B-WET anticipates participants to become environmentally literate citizens who contribute to protecting and restoring watersheds and related ocean, coastal, and Great Lakes ecosystems (http://www.oesd.noaa.gov/grants/bwet.html). MWEEs are expected to include three phases, (1) a preparation phase during which participants are introduced to an environmental question or issue affecting their watershed or the Chesapeake Bay, (2) an action phase during which participants are immersed in one or more outdoor experiences and potentially, engage in environmental actions, and (3) a final phase during which participants have an opportunity for analysis and reflection as well as, possibly, the chance to communicate what they learned about their watershed/the Chesapeake Bay to others. As suggested by this description, MWEEs promote experiential, outdoor, science-based inquiries into environmental issues affecting students’ local watershed or the Chesapeake Bay. The NOAA B-WET evaluation assessed MWEEs for both students and teachers but only results for participating students are reported here. For this part of the evaluation, we were charged with assessing (1) the extent to which NOAA Chesapeake’s B-WET grant program is achieving its mission of fostering ERBs through MWEEs and (2) to which specific MWEE instructional practices outcomes toward this mission can be attributed. In light of the difficulties associated with measuring ERB directly, we selected a theory based approach to answering these questions. More specifically, we measured select predictors of ERB included in the Hines et al. (1986/1987) and Hungerford and Volk (1990) behavior models. Consistent with these models and barring constraining situational factors, increases in environmental stewardship characteristics should lead to higher levels of ERB. Therefore, our study’s research questions were (1) how does participation in B-WET funded MWEE programs relate to students’ environmental stewardship characteristics and (2) how do specific MWEE instructional practices relate to these qualities? By answering these questions, this study sought to contribute to the limited literature on the behavioral outcomes of environmental education programs and the specific instructional practices to

which these outcomes can be attributed. To the best of our knowledge, this is also one of only two other studies of watershed education programs to assess behavioral outcomes (Bodzin, 2008; Zint, Kraemer, Northway, & Lim, 2002) and the first study of behavioral outcomes of an environmental education grant program. In addition, this study sought to demonstrate how program evaluations designed to meet program decision makers’ needs can also make valuable contributions to advancing theory (Zint, 2012). To achieve this aim, results from exploratory investigations into the relationships between environmental stewardship characteristics are reported and discussed. To the best of our knowledge, few studies have measured the majority of the predicators in the Hines et al. (1986/1987) and Hungerford and Volk (1990) behavior models to allow such explorations. In addition, none have tested the simultaneous relationships between these variables, as suggested by these models.2 A better understanding of the relationships between environmental stewardship characteristics can identify which variables may be particularly important for environmental educators to target in light of their combined direct and indirect effects on ERB. Methods Design and measurement A quasi-experimental design was implemented to collect preand post-data from students who experienced MWEEs and a comparison group of students who did not. Questionnaires measured (1) students’ environmental stewardship characteris-

2 It is acknowledged that Bamberg and Mo¨ser (2007), building on the results of their meta-analysis, tested simultaneous relationships between predictors of ERB. However, the majority of their predictors were more closely related to the variables in the norm-activation model and theory of planned behavior than those in the Hines et al. (1986/1987) and Hungerford and Volk (1990) behavioral models

Please cite this article in press as: M. Zint, et al.. Evaluating Meaningful Watershed Educational Experiences: An exploration into the effects on participating students’ environmental stewardship characteristics and the relationships between these predictors of environmentally responsible behavior. Studies in Educational Evaluation (2014), http://dx.doi.org/10.1016/j.stueduc.2013.07.002

G Model

JSEE-474; No. of Pages 14 4

M. Zint et al. / Studies in Educational Evaluation xxx (2013) xxx–xxx

tics, (2) MWEE practices students believed they experienced, and (3) demographics. Eight stewardship characteristics associated with ERBs (Hines et al., 1986/1987; Hungerford & Volk, 1990) were measured: environmental sensitivity, knowledge of ecology, knowledge of environmental issues, sense of personal responsibility, knowledge of environmental action strategies, individual and group locus of control, and intention to act. Validated and reliable measures from past studies, including previous evaluations of environmental education programs, were used to measure these stewardship characteristics (Ajzen & Fishbein, 1980; Nowak et al., 1995; Zint et al., 2002). There were (1) four measures for actual knowledge of ecology (e.g., Which statement best describes the effect of sediment on water quality?), perceived knowledge of environmental issues (e.g., How much do you know about high levels of nutrients and where they come from?), and perceived knowledge of actions (e.g., I know how to save water at home to protect my local watershed/Bay), (2) three measures for environmental sensitivity (e.g., How much do you care about your local watershed/Bay?), personal responsibility (e.g., It is my personal responsibility to help protect my local watershed/Bay) and intention to act (e.g., In the future, I intend to clean up or take care of a local stream or waterway), and (3) two measures for individual locus of control (e.g., By working on my own, I can make a difference in solving environmental problems in my community) and group locus of control (e.g., By working with others, I can make a difference in solving environmental problems in my community). Students in elementary school (grades 3–5) were given three response options whereas secondary students (grades 6–12) had five. The latter were subsequently collapsed into three, to permit combined analyses of elementary and secondary student data. Scales were calculated by taking the mean of the items used to measure the respective constructs. The eight environmental stewardship characteristic scales had acceptable to good reliabilities (Nunnally & Bernstain, 1994) with Cronbach a values ranging from .60 to .79 for pre-test and .61 to .85 for post-test scales. In addition to measuring students’ environmental stewardship characteristics, we asked students to share how frequently, or if, they experienced the following MWEE practices: (1) learned outdoors, (2) learned hands-on, (3) learned about what they viewed as important to their lives, (4) reflected on how they helped their watershed/Bay and (5) engaged in a range of environmental actions during their MWEE (Table 1). Elementary students were again given three point response options whereas secondary students had four. We ultimately collapsed these response options into two, to identify students who reported experiencing these practices (regardless of frequency) and those who did not. We made this choice to facilitate reporting of results and after determining that it would not affect the substantive nature of our findings. The demographic questions in the instrument consisted of items to learn students’ sex (i.e., female, male), grade level (i.e., 3– 12), and academic performance (i.e., self reports of receiving mostly As through Ds or below). We relied on teachers to administer the student pre- and postquestionnaires but assisted them by providing relevant instructions. For example, teachers of MWEE students were asked to administer the pre-questionnaire immediately before the MWEE program began and the post-questionnaire on the last day of their MWEE. One of the other requests was that teachers distribute consent forms to obtain parental permission for their child’s participation. Only students for whom parental consent was obtained were included in this study. The teachers of participating students in the treatment group were also asked to complete a questionnaire at the end of their

MWEE to learn about the practices that they or other MWEE providers implemented (Table 1). Specifically, we asked teachers to report on the following: (1) select science inquiry practices (i.e., did students collect data, use field equipment, graphically display data, analyze watershed/Bay data, reflect on what they learned?) and additional inquiry practices included in environmental issue investigation (i.e., did students explore their local community for information about local watershed/Bay issues), (2) environmental action projects students engaged in during their MWEEs (i.e., did students implement a solution to a watershed/Bay problem, participate in a restoration, monitoring, pollution prevention project or communicate/share what they learned with others) and (3) MWEE instruction in different settings and their duration. Teachers were also asked to indicate if students had passive learning experiences (i.e., did students listen to talks about, or read about, local watershed/Bay environmental issues) to compare these results with those of active practices. Last, one exploratory measure asked if students learned about social issues within the context of the MWEE. This item was included because it was anticipated that those who also learned about these aspects of watershed/Bay issues would view them as more important and thus score higher in environmental stewardship characteristics (Kumler, 2011). Teachers were asked to report hours of instruction in different settings and whether they implemented the above practices or not. Sampling and participants To generate the evaluation’s sample, all 16 organizations who received NOAA Chesapeake B-WET funding during the grant year of our study were asked to provide names of participating teachers whom they expected to implement MWEEs. This list of teachers was randomized and an approximately equal number of elementary, middle, and high schools teachers were asked to take part in the study. The teachers who agreed to participate received instructions and instruments. If teachers indicated they could not participate, the next teacher on the randomized list was contacted until the desired sample size was reached. To obtain comparison classes (ones that did not experience MWEEs), MWEE teachers were asked to identify another teacher at their school willing to administer the questionnaire to their students at the same time MWEE students completed theirs. The MWEE teacher was not always able to identify such a comparison teacher. For example, in some schools, all students in a grade level participated in MWEEs. In other schools, the principal would not permit non-MWEE students to use class time for the questionnaire. In the end, 60 teachers returned questionnaires completed by 1,345 children from 60 classes across grades 3 through 12. Of those, 37 classes with 880 students experienced MWEEs and 23 classes with the remaining children did not. The final samples used for subsequent analyses were smaller, however, because we included only: (1) students with matching pre- and post-instruments, (2) MWEE classes that could be matched with comparison classes by grade, and (3) students whose teachers provided information about the MWEE practices their students experienced. Analyses to explore the effects of MWEEs were subsequently based on 20 treatment and 12 comparison classes with a total of 258 and 193 students, respectively in grades 3, 4, 6, 8, 9, and 10. The treatment and comparison groups did not differ in sex (45% boys, 55% girls, x2 = 1.64, df = 1, p = .20) or self-reported academic performance (50% indicated receiving mostly A’s and the other half mostly B’s through D’s, x2 = .03, df = 1 p = .87) but varied in their distribution across grades (x2 = 66.98, df = 5, p < .001). Analyses to explore the effects of specific MWEE instructional practices drew on data from 29 treatment classes with 434 students. This sample again consisted of about the same percent of boys and girls (48%

Please cite this article in press as: M. Zint, et al.. Evaluating Meaningful Watershed Educational Experiences: An exploration into the effects on participating students’ environmental stewardship characteristics and the relationships between these predictors of environmentally responsible behavior. Studies in Educational Evaluation (2014), http://dx.doi.org/10.1016/j.stueduc.2013.07.002

G Model

JSEE-474; No. of Pages 14 M. Zint et al. / Studies in Educational Evaluation xxx (2013) xxx–xxx

5

Table 1 Specific MWEE instructional practices experienced by students based on self-reports and their teachers’ feedback (n = 434). Practices students reported experiencing (abbreviation)

% Who experienced practice or mean

While studying about your local watershed or the Chesapeake Bay, how often did you . . . Do outdoor learning activities? (LearnOutdoors)

Do hands-on learning about the Bay, instead of just reading or hearing about it (LearnHandson)

Have time to talk about or write about how you helped your local watershed or the Bay? (LearnReflect) Learn things that are important to your life? (LearnImportant)

Did you do water testing on a local waterway? (EAWaterTest) Did you grow underwater grasses or wetland plants? (EAGrowSAV) Did you plant underwater grasses or wetland plants in a natural habitat? (EAPlantSAV) Did you make a presentation to a group of people? (EAPresent) Did you plant a tree? (EATree) Did you raise fish in your classroom? (EARaiseFish) Did you release fish into a local waterway? (EAReleaseFish) Did you put data on a web site? (EAWeb) Did you teach younger students about the watershed or the Bay? (only collected for secondary students) (EATeachChildren) Scale: number of above actions students reported engaging in, range: 0–8 (EA Scale)

88%

85%

80% 91%

55% 43% 21% 41% 30%

Practices that teachers reported their students experienced (abbreviation) During the unit on your local watershed or the Chesapeake Bay, did the students . . . Participate in a school classroom-based, local watershed or Bay curriculum? (ClassCurr) Learn about the local watershed or Bay outdoors on an onthe-water field trip? (On Water) Learn about the local watershed or Bay outdoors in the schoolyard? (Schoolyard) On average, how many hours did students spend learning in . . . during the Chesapeake Bay/watershed unit? In a classroom (Class Total) outdoors (Outdoor Total) inside a museum or nature center (Center Total) During the unit on your local watershed or the Chesapeake Bay, did the students . . . Listen to talks about, or read about, local watershed or Bay environmental issues? (TalkRead) Collect local watershed or Bay data? (Collect Data) Use field equipment, such as hand-held technology, for data collection? (UseEquip) Graphically display data (e.g., create charts, graphs)? (Graph Data) Analyze watershed or Bay data? (Analyze Data) Have an opportunity to reflect on their local watershed/Bay unit? (Reflect) Explore the local community (beyond the classroom) for information on local watershed or Bay environmental issues? (ExplComm) Study social, economic, historical, or archeological issues? (Social) Participate in a monitoring project (e.g., periodic water testing)? (Monitor) Participate in a restoration project (e.g., growing/planting wetland plants)? (Restore) Participate in a pollution prevention project (e.g., erosion control)? (Prevent) Participate in a communication/information-sharing action (e.g., making a presentation to the community)? (Commun) Implement a solution to a local watershed or Bay problem? (Action)

% Who experienced practice or mean

89% 82% 75%

Mean = 33 h Mean = 18.6 h Mean = 3.5 h

91% 71% 60% 58% 77% 94% 81%

71% 49% 73% 49% 43% 59%

24% 23% 17% 19%

Mean = 2.35 actions

and 52%, respectively), students who reported receiving mostly As versus other grades (45% and 55%, respectively) but the distribution across grades 3 through 12 differed (range: 5% in grade 10– 17% in grade 4). Analysis To test the effects of MWEE programs and specific MWEE instructional practices on students’ environmental stewardship outcomes as well as to explore the relationships between these characteristics, a series of multilevel analyses (Gelman & Hill, 2007; Raudenbush & Bryk, 2002) for students nested within classrooms were conducted using SPSS Version 20. Multilevel models (1) account for the shared, group nature of students’ MWEE experiences in classes and (2) allow for the investigation of fixedfactor effects both at the student and classroom level (here: respective pre-test, treatment, specific MWEE instructional

Scale: number of above action projects teachers reported implementing, range: 0–5 (EAScaleTeacher) Scale: did students implement at least one of the above action projects? 0 = no, 1 = yes (ActTeacher)

Mean = 2.7 actions 90%

practice, or environmental stewardship characteristic, sex, grade, self-reported academic performance) on the outcome of interest (here: environmental stewardship characteristics). Sex and grade were included as covariates because studies have shown differences in students’ environmental stewardship characteristics based on these variables (Zint, 2012). Similarly, selfreported academic performance was expected to influence how students experienced MWEEs and thus, also potentially their environmental stewardship characteristics. For the multilevel analysis and to facilitate reporting of results, we created dummy coded variables for treatment, female, receiving mostly As (vs. other grades), and experiencing specific MWEE instructional practices (1 = yes, 0 = no, respectively). We also verified that the pre- and posttest environmental stewardship characteristics were normally distributed. REML estimation was used to obtain model estimates. Initial multilevel analyses were conducted to obtain the relative contributions of within- and between group variability (Gelman &

Please cite this article in press as: M. Zint, et al.. Evaluating Meaningful Watershed Educational Experiences: An exploration into the effects on participating students’ environmental stewardship characteristics and the relationships between these predictors of environmentally responsible behavior. Studies in Educational Evaluation (2014), http://dx.doi.org/10.1016/j.stueduc.2013.07.002

Potential Values

FULL MODEL Treatment (Control = 0) Sex (Male=0) Academic Performance (0 = B trough F) Grade Pretest L1 Residual Variance L2 Residual Intercept Variance Mean Comparison Treatment Difference % Increase

b

Knowledge of issues

Personal responsibility

Knowledge of actions

Intention to act

0–4

1–3

1–3

1–3

1–3

1–3

1–3

0.36

1.18

0.31

0.41

0.26

0.29

0.50

0.33

0.09

0.56

0.08

0.04

0.07

0.02

0.06

0.07

21%

32%

20%

8%

21%

5%

10%

18%

SE 0.08

p 0.34

b

SE 0.21

p .02**

b

p <.001***

0.06

SE 0.08

p 0.46

b

0.38

SE 0.07

b

0.52

0.29

SE 0.06

p <.001***

0.06 0.05

0.05 0.05

0.22 0.34

0.02 0.14

0.10 0.11

0.82 0.17

0.01 0.00

0.05 0.05

0.81 1.00

0.05 0.02

0.06 0.06

0.35 0.67

0.04 0.02

0.04 0.04

0.37 0.73

0.19 0.48

0.04 0.65 0.04 <.001*** .91

0.01 0.53

0.02 0.47 0.45 <.001*** 0.23

0.03 0.58

0.02 0.13 0.04 <.001*** 0.29

0.03 0.52

0.01 0.01** 0.04 <.001*** 0.19

0.02

Individual locus of control

1–3

0.08

0.05 0.59

Group locus of control

0.02 0.01** 0.04 <.001*** 0.23

0.21

0.02

1.79 2.30 0.51 28%

0.02

1.84 2.22 0.38 21%

0.01

b

SE 0.06

p 0.44

b

0.05

0.17

SE 0.09

p 0.06*

0.01 0.03

0.05 0.05

0.83 0.57

0.09 0.09

0.06 0.06

0.17 0.17

0.02 0.48

0.02 0.26 0.04 <.001*** 0.39

0.02 0.42

0.01

2.22 2.50 0.29 13%

0.01 0.10 0.05 <.001*** 0.25

0.02

b 0.13

SE 0.07

p 0.06*

0.11 0.03

0.05 0.05

0.02** 0.51

0.01 0.71

0.01 0.65 0.04 <.001*** 0.21

0.01

1.80 1.97 0.17 9%

1.96 2.09 0.13 7%

M. Zint et al. / Studies in Educational Evaluation xxx (2013) xxx–xxx

UNCONDITIONAL MODEL L1 Residual Variance L2 Residual Intercept Variance Intraclass Correlation Coefficient (ICC)

Knowledge of ecology

G Model

Environmental sensitivity

JSEE-474; No. of Pages 14

6

Please cite this article in press as: M. Zint, et al.. Evaluating Meaningful Watershed Educational Experiences: An exploration into the effects on participating students’ environmental stewardship characteristics and the relationships between these predictors of environmentally responsible behavior. Studies in Educational Evaluation (2014), http://dx.doi.org/10.1016/j.stueduc.2013.07.002

Table 2 Multi-level analysis results to evaluate MWEE programs’ effects on students’ environmental stewardship characteristics (n = 451).

G Model

JSEE-474; No. of Pages 14 M. Zint et al. / Studies in Educational Evaluation xxx (2013) xxx–xxx

Hill, 2007; Raudenbush & Bryk, 2002). Results from these analyses were used to calculate the intraclass correlation coefficients (ICC) which were 10% or greater for six of the eight stewardship characteristics (Table 2), supporting the appropriateness of multilevel analysis. Next fixed effects were added to the model to determine whether they were significantly related to post-test ERB characteristics (***p  .001, **p  .05, *p  .10). For continuous variables, bs and standard errors are reported whereas means are provided for categorical variables. To explore the relationships between students’ environmental stewardship characteristics two path models were tested using AMOS Version 20. Model fit was assessed through several frequently used indicators (Kline, 2011): the x2 statistic, comparative fit index (CFI), Tucker Lewis Index (TLI), and root mean square error of approximation (RMSEA). The x2 should be low and non-significant to attest to a good fit between the sample and theoretical model (Kline, 2011), the CFI should be above 0.95 (Hu & Bentler, 1999), TLI not below 0.9 (Tucker & Lewis, 1973), and RMSEA less than 0.05 (MacCallum, Browne, & Sugawara, 1996). Results Relationships between MWEEs and students’ environmental stewardship characteristics Multilevel analysis indicated that students who experienced MWEEs changed significantly in three of the eight measured stewardship characteristics (Table 2, see p-values for ‘‘treatment’’ in full model). The three stewardship characteristics were knowledge of ecology, issues, and actions. In addition, two stewardship characteristics, individual locus of control and intention to act, closely approached significance. Students who experienced MWEEs had moderately higher mean scores for these stewardship characteristics compared to students who did not (Table 2). The largest relative differences occurred between students’ knowledge of ecology and knowledge of issues, followed by relatively smaller differences in knowledge of actions, internal locus of control, and intention to act. MWEE students did not differ significantly from the comparison group in the remaining three stewardship characteristics of environmental sensitivity, personal responsibility and group locus of control. As would be expected, pre-test score helped to significantly explain post stewardship outcomes, in the presence of the remaining covariates. Sex did so only in one instance, with girls scoring higher in intention to act than boys. Grade did so in two instances, with students’ in higher grades scoring lower in environmental sensitivity and knowledge of actions than those in lower grades. No significant associations were found between self-reported academic performance and any of the environmental stewardship characteristics. When the non-significant covariates self-reported academic performance and grade were removed from the model predicting post intention to act, the relationship between the treatment and this outcome became significant (p = .001). Similarly, when the non-significant covariate grade was removed from the model predicting post individual locus of control, the relationship between the treatment and this outcome also became significant (p = .03). Relationships between specific MWEE instructional practices and students’ environmental stewardship characteristics Based on students’ self-reports and teachers’ feedback, participants experienced the range of the expected MWEE instructional practices (Table 1). To explore the relationships between these practices and students’ post MWEE environmental stewardship

7

characteristics, a series of simple multilevel models with students nested within classrooms were conducted. Table 3 provides an overview of results for practices reported by students whereas Table 4 provides a synthesis of results for practices reported by teachers. Many of the specific MWEE instructional practices students reported experiencing or that their teachers reported that students experienced were statistically significantly related to students’ post intention to act, personal responsibility, knowledge of issues, individual locus of control, knowledge of actions and knowledge of ecology (Tables 3 and 4). Fewer of the specific MWEE instructional practices were statistically significantly related to environmental sensitivity and group locus of control. The following paragraphs identify the practices which were statistically significantly related the each of the eight respective environmental stewardship characteristics. Students scored higher in intention to act if (a) they reported they (1) learned about their watershed/Bay outdoors, (2) had time to talk or write about how they helped their watershed/Bay, (3) learned about their watershed/Bay hands-on, (4) learned about what they viewed as important to their life, (5) engaged in certain behaviors (i.e., planted a tree or SAVs, tested water, shared data through the Internet, gave a presentation) or more stewardship actions during their MWEEs, or (b) their teachers reported that students had the opportunity to (6) listen to talks or read about issues confronting their local watershed/Bay, (7) analyze data, (8) implement a solution to an environmental problem during their MWEE, (9) communicate with others about what they learned, or (10) reflect on what they had learned. Students scored higher in personal responsibility if (a) they reported that they (1) had time to talk or write about how they helped their watershed/Bay, (2) learned about their watershed/Bay hands-on, (3) engaged in certain behaviors (i.e., planted SAVs, tested water, shared data through the Internet) or more stewardship actions during their MWEEs or (b) their teachers reported that students had the opportunity to (4) use field equipment to collect data, (5) graph data, (6) analyze data, or (7) participate in a restoration project or at least one stewardship action during their MWEE. Students who explored their local community for information about local watershed/Bay issues, however, scored lower in personal responsibility. Students scored higher in knowledge of issues if (a) they reported they (1) learned about their watershed/Bay outdoors, (2) had time to talk or write about how they helped their watershed/Bay, (3) learned about their watershed/Bay hands-on, (4) learned about what they viewed as important to their life, (5) engaged in certain behaviors (i.e., grown or planted SAVs, tested water, gave a presentation) or more stewardship actions during their MWEEs, or (b) their teachers reported that students (6) spent time learning about watersheds/Bay at a museum or nature center. Students scored higher in individual locus of control if (a) they reported that they (1) had time to talk or write about how they helped their watershed/Bay, (2) learned about the Bay hands-on (3) engaged in certain behaviors (i.e., planted a tree or SAV, gave a presentation) or more stewardship actions during their MWEEs or (b) their teachers reported that students had the opportunity to (4) collect data or (5) analyze data, or that students (6) spent time learning about their watersheds/the Bay outdoors, or at a museum or nature center. Students scored higher in knowledge of actions if (a) they reported they (1) learned about their watershed/Bay outdoors, (2) had time to talk or write about how they helped their watershed/ Bay, (3) learned about their watershed/Bay hands-on, (4) learned about what they viewed as important to their life, (5) engaged in certain behaviors (i.e., grown or planted SAVs, gave a presentation) or more stewardship actions during their MWEEs, or (b)

Please cite this article in press as: M. Zint, et al.. Evaluating Meaningful Watershed Educational Experiences: An exploration into the effects on participating students’ environmental stewardship characteristics and the relationships between these predictors of environmentally responsible behavior. Studies in Educational Evaluation (2014), http://dx.doi.org/10.1016/j.stueduc.2013.07.002

G Model

JSEE-474; No. of Pages 14

***

SE 0.02

b

0.21 *

p

SE 0.02 0.04

b p

**

SE 0.02 0.04

b p

***

SE 0.02

b

0.05 *

b

0.04 ***

ns = not significant, means are reported when significant differences were found. * p  .10. ** p  .05. *** p  .001.

b ns

p ns SE

b SE Continuous Variable EA Scale

b

p ns

**

2.23 2.55

0.08

SE 0.02

p

SE 0.02

p

ns ns ns

2.54 2.39 ns ns ns ns ns

**

2.32 2.19

ns ns ns ns

*

2.32 2.30 2.14 2.19

**

p

ns ns ns

ns ns ns ns ns ns ns ns ns ns ns ns *

ns ns ns ns

2.35 2.33 **

*

2.26 2.31 2.29 2.28 2.13 2.15 2.12 2.16

**

*

**

*

2.16 2.21 2.23 2.06 2.06 2.09 **

2.02 1.85 ***

2.88 2.69 **

2.61 2.50 ns

*

** **

2.53 2.45 ns

2.46 2.49

ns ns ns

*

**

ns

2.03 1.84 ns

**

**

2.61 2.67

ns

2.72

2.82

*

1.84 ns ns

ns

**

ns ns ns

2.05

**

ns ****

2.76 2.47 **

***

*** **

**

2.45 2.44 **

***

2.30 2.28

Yes No p

**

2.24 2.26 2.24 2.23

Yes No

2.00 2.00 2.04 2.01 ns

No Yes

p No

Nominal Variables

LearnOutdoors LearnReflect LearnHandson LearnImportant EATree EAGrowSAV EAPlantSAV EAWaterTest EAPresent EAWEB EARaiseFish EAReleaseFish EATeachChildren

Mean Mean

Yes

p

ns ns ns ns ns ns ns ns ns ns ns

Mean Mean

**

***

**

2.16 2.16 2.14 2.21 *

1.98 1.97 1.83 1.81

No

ns ns ns

No p

**

2.55 2.57 2.56 2.55

Yes No

2.39 2.39 2.35 2.37

Mean

p

ns

Mean

Yes

p

*

2.01 1.94 1.96 2.06

Yes No Yes

p

ns

Mean Mean

Intention to act Individual locus of control Group locus of control Knowledge of actions Personal responsibility Knowledge of issues Know of ecology Environmental Sensitivity

Table 3 Results from multi-level analyses exploring relationships between students’ environmental stewardship characteristics and the specific MWEE instructional practices they reported experiencing (n = 434).

p

M. Zint et al. / Studies in Educational Evaluation xxx (2013) xxx–xxx ns

8

their teachers reported that students had the opportunity to (6) analyze data. Students scored higher in knowledge of ecology if their teachers reported that students had the opportunity to (1) learn about social issues within the context of learning about their local watershed/ Bay, (2) collect data, (3) analyze data, (4) reflect on what they had learned, or (5) spent more time learning about their watershed/Bay both in the classroom and outdoors. Students scored higher in environmental sensitivity if (a) they reported they (1) had time to talk or write about how they helped their watershed/Bay, (2) learned about their watershed/Bay handson, (3) engaged in water testing or (b) their teachers reported that students (4) learned about their local watershed/Bay through an on-the-water field trip. Students scored higher in group locus of control if (a) they reported they had (1) learned about what they viewed as important to their life or (2) engaged in planting SAVs, gave a presentation or implemented more stewardship actions during their MWEEs. None of the practices teachers reported that their students experienced were significantly related to students’ group locus of control. As suggested by these descriptions of the relationships between specific MWEE instructional practices and student post stewardship outcomes, some of these practices were related to more outcomes than others. More specifically, students who reported having active learning experiences, reflecting on their contribution, and engaging in more environmental actions during their MWEEs scored higher in six out of the eight stewardship characteristics. There were also more significant relationships between stewardship characteristics and engaging in certain environmental actions over others. For example, planting SAVs, giving presentations about what they learned, and testing water were related to more stewardship characteristics than planting trees, growing SAVs, and sharing data through the Internet. In some contrast, learning about what students felt was important to them was only significantly related to four stewardship characteristics and learning about their watershed/Bay outdoors to three stewardship characteristics. Although there were many significant differences in the stewardship characteristics of students who reported experiencing specific MWEE instructional practices and those who did not, differences in means were moderate on average with percent increases ranging from 9% to 29%. The largest difference in means was in the group locus of control between students who felt they were learning about something that was important to their life versus those who did not. In comparison to the number of statistically significant relationships that were found between students’ post stewardship characteristics and the specific MWEE instructional practices they reported experiencing, far fewer statistically significant relationships were found between these outcomes and the practices teachers reported that their students experienced. The teacher reported practice of analyzing data was related to the greatest number of students’ environmental stewardship characteristics (i.e., five). The remaining teacher reported practices were related to only two, one, or none of students’ environmental stewardship characteristics. Interestingly, while there were many relationships between students’ environmental stewardship characteristics and the environmental actions they reported engaging in, there were far fewer statistically significant relationships between these outcomes and the environmental projects teachers reported that their students participated in. In addition, the amount of hours of MWEE instruction in different settings was only related to one or two environmental stewardship characteristics. Of the three settings teachers were asked about, only one was related to one

Please cite this article in press as: M. Zint, et al.. Evaluating Meaningful Watershed Educational Experiences: An exploration into the effects on participating students’ environmental stewardship characteristics and the relationships between these predictors of environmentally responsible behavior. Studies in Educational Evaluation (2014), http://dx.doi.org/10.1016/j.stueduc.2013.07.002

Continuous Variables EAScale Teacher ClassTotal Outdoor Total CenterTotal

Know of ecology

Knowledge of issues

Personal responsibility

Knowledge of actions

Group locus of control

Indiv. locus of control

Intention to act

Mean

Mean

Mean

Mean

Mean

Mean

Mean

Mean

No

Yes

p ns ns ns ns ns ns ns ns ns ns ns ns ns ns ns

2.19

b

2.31

SE

No

Yes

p

No

Yes

ns ns 2.07 2.16

2.56 2.56

** *

ns ns 2.00

2.57

**

ns ns ns ns ns ns ns ns ns

*

ns

1.71

2.48

*

p ns ns ns ns

b

SE

p ns

.003 .003

*

.006 .009

b

SE

p ns ns ns

0.01

0.01

**

**

ns

p ns ns ns ns ns ns ns ns ns ns ns ns ns ns ns ns ns

No

Yes

p

No

Yes

ns 2.56

2.38

*

ns ns 2.29 2.32 2.15

2.49 2.49 2.49

** * **

2.40

2.56

ns 2.29

2.46

2.44

*

*

ns ns ns ns

b

SE

p ns ns ns ns

No

Yes

b

SE

**

p ns ns ns ns

p ns ns ns ns ns ns ns ns ns ns ns ns ns ns ns ns ns

ns ns ns ns ns ns ns ns ns ns

ns ns ns 2.21

p ns ns ns ns ns ns

b

SE

p ns ns ns ns

No

1.76

Yes

2.03

p

No

Yes

ns ns ns

1.91

2.15

2.01

**

**

ns ns ns ns ns ns ns ns ns ns

b

SE

p ns ns

0.01 0.02

0.05 0.01

** **

*

ns ns ns ns ns

ns ns 1.72

p

1.96 2.06

2.18 2.18

** *

ns ns ns 2.06

2.23

**

ns ns ns ns 1.67

2.16

***

b

SE

p ns ns ns ns

ns = not significant, means are reported when significant differences were found. * p  .10. ** p  .05. *** p  .001.

G Model

JSEE-474; No. of Pages 14

Categorical Variables TalkRead ExplComm Social CollectData UseEquip GraphData AnalyzData Action Restore Monitor Prevent Commun ActTeacher ClassCurr Schoolyard OnWater Reflect

Environmental sensitivity

M. Zint et al. / Studies in Educational Evaluation xxx (2013) xxx–xxx

Please cite this article in press as: M. Zint, et al.. Evaluating Meaningful Watershed Educational Experiences: An exploration into the effects on participating students’ environmental stewardship characteristics and the relationships between these predictors of environmentally responsible behavior. Studies in Educational Evaluation (2014), http://dx.doi.org/10.1016/j.stueduc.2013.07.002

Table 4 Results from multi-level analyses exploring relationships between students’ environmental stewardship characteristics and specific MWEE instructional practices teachers reported they experienced (n = 434).

9

G Model

JSEE-474; No. of Pages 14 M. Zint et al. / Studies in Educational Evaluation xxx (2013) xxx–xxx

10

environmental stewardship characteristics. Moreover, there was one negative relationship: students whose teachers reported that they explored their local community for information about the watershed/Bay scored lower in personal responsibility than students whose teachers reported that their students had not done so. Similar to the differences found in mean environmental stewardship characteristics based on the specific MWEE reported practices students reported experiencing, the differences in mean stewardship characteristics based on teachers’ reported practices were moderate on average, with percent increases ranging from 13% to 77%. The largest, outlying difference in means was in the knowledge of ecology between students whose teachers reported they had a chance to reflect on what they had learned and students whose teachers reported that they had not. Relationships between students’ environmental stewardship characteristics This study was designed to answer NOAA Chesapeake B-WET’s evaluation questions and not to test theoretical questions about the relationships between students’ environmental stewardship characteristics. Nonetheless, the data collected allow analyses to begin to explore these relationships. Two path models of ERB based on research conducted by Hines et al. (1986/1987) and Hungerford and Volk (1990) were tested first. These path models were based on a sample of 434 students. This sample size allows for testing for close fit for models up to 25 degrees of freedom to achieve a power of .80 (MacCallum et al., 1996). The first model, inspired by Hines et al. (1986/1987), tested how well post intention to act was directly predicted by six other post environmental stewardship characteristics (Fig. 3). Because this model was fully saturated, there were no degrees of freedom to generate a x2 statistic or many other regularly reported fit indices. However, an RMSEA of .35 suggested an unsatisfactory fit. Nonetheless, the model predicted intention to act very well (R2 = .61) and all six paths were statistically significant. The standardized path coefficients were small to moderate (range: .07– .38), with the highest coefficient between environmental sensitivity and intention to act.

Knowledge of Issues

.09**

Knowledge of Acons .14***

Environmental Sensivity

Personal Responsibility

.38***

.26***

Intenon to Act

.07**

Group Locus of Control

.07**

Individual Locus of Control Fig. 3. Path analysis of relationships between environmental stewardship characteristics inspired by the Hines et al. (1986/1987) model predicting environmental behavior. Note: ***p  .001, **p  .05 and all six exogenous variables were allowed to correlate (not shown).

The second model, building on work by Hungerford and Volk (1990), tested a more complex set of relationships between post intention to act the remaining seven other measured post environmental stewardship characteristics. Fit statistics for this model were also unsatisfactory (x2 = 210.74, df = 14, p < .001; CFI = .76, TLI = .52, RMSEA = .22). The model did not predict intention to act as well as the previous, more parsimonious model (R2 = .36) but all paths in this model were statistically significant, with small to moderate standardized path coefficients (range: .16– .50). In response to modification indices results, this model was revised by adding a direct path between environmental sensitivity and intention to act (Fig. 4). This modification substantially improved the model fit statistics (x2 = 174.99, df = 13, p < .001; CFI = .87, TLI = .72, RMSEA = .17) but overall fit remained unsatisfactory. It also greatly increased the prediction of intention to act (R2 = .53) and again, yielded small to moderate standardized path coefficients (range: .11–.51). The strongest relationships were between environmental sensitivity and intention to act, personal responsibility as well as knowledge of issues, suggesting that students expressing caring and concern for the environment were more likely to report an intention to act, a greater duty to protect the environment as well as improved knowledge of environmental issues. Because of the unsatisfactory fits of these path models, a series of multilevel analyses were conducted to further explore the relationships between the measured environmental stewardship characteristics (Table 5). Similar to earlier multilevel models, these analyses controlled for students nested in classes. In contrast to the earlier models, however, they did not include sex, grade or selfreported academic performance as covariates because they are not included in the behavior models proposed by Hines et al. (1986/ 1987) and Hungerford and Volk (1990). As would be expected, results for the multilevel model predicting intention to act were largely consistent with findings from the first path model. With the exception of individual locus of control, all of the remaining environmental stewardship characteristics significantly predicted intention to act. Students with higher environmental sensitivity and personal responsibility scored quite a bit higher in intention to act and students with higher knowledge of ecology scored slightly lower in intention to act. Many other significant relationships were found between the remaining environmental stewardship characteristics. Five of the stewardship characteristics significantly predicted knowledge of issues and actions, four predicted personal responsibility, and three predicted environmental sensitivity. In some contrast, only two of the environmental stewardship characteristics significantly predicted group locus of control and only one knowledge of ecology and individual locus of control, respectively. Alternatively, knowledge of actions significantly predicted five of the environmental stewardship characteristics, knowledge of issues and personal responsibility four each, environmental sensitivity three, group locus of control two, and knowledge of ecology as well as individual locus of control only one, respectively. In combination, findings from the path and multilevel analyses suggest that the environmental stewardship characteristics measured in this study all have a role to play in predicting intention to act and thus, ERB. Results also suggest the potential for a number of feedback loops between these variables and that the relative importance of certain stewardship characteristics may vary based on the number of significant relationships they have with other environmental stewardship characteristics as well as the size of these relationships. For example, multilevel analysis results suggest that knowledge of actions may play an important role in predicting intention to act not only directly, but also through a number of indirect paths.

Please cite this article in press as: M. Zint, et al.. Evaluating Meaningful Watershed Educational Experiences: An exploration into the effects on participating students’ environmental stewardship characteristics and the relationships between these predictors of environmentally responsible behavior. Studies in Educational Evaluation (2014), http://dx.doi.org/10.1016/j.stueduc.2013.07.002

G Model

JSEE-474; No. of Pages 14 M. Zint et al. / Studies in Educational Evaluation xxx (2013) xxx–xxx

11

.35***

2

R =.34

2

R =.36

2

R =.16

Environmental Sensivity

.45***

Knowledge of Issues

Group Locus of Control

.17***

Knowledge of Acons

.51*** .20***

.38*** .30*** .23*** .12***

.27***

Knowledge of Ecology

.49***

.19***

Personal Responsibility

2

R =.33

.27***

Individual Locus of Control

2

R =.15

.11***

Intenon to Act

2

R =.53

Fig. 4. Path analysis of relationships between environmental stewardship characteristics inspired by the Hungerford and Volk (1990) model predicting environmental behavior. Note: ***p  .001 and both exogenous variables were allowed to correlate (not shown).

Discussion This evaluative study of the NOAA Chesapeake B-WET program sought to determine (1) if participation in its funded programs related to students’ environmental stewardship characteristics and (2) which instructional practices were associated with changes in these qualities. The first goal was selected to help assess the effectiveness of the NOAA B-WET grant program in meeting its mission of fostering ERBs. The second goal was chosen to identify the contributions of practices promoted through MWEEs, within the context of this mission. In addition, we investigated the relationships between students’ environmental stewardship characteristics, to further enhance the theoretical contributions of this study. To evaluate the effectiveness of the NOAA B-WET grant program in meeting its mission of fostering ERBs, we assessed improvements in student participants’ environmental stewardship characteristics associated with ERBs (Hines et al., 1986/1987; Hungerford & Volk, 1990). We found evidence that students who participated in MWEEs increased in five of the eight characteristics we measured: knowledge of ecology, issues, and actions, individual locus of control and intention to act. Based on their meta-analysis, Hines et al. (1986/1987) found respective mean correlations of .37 for knowledge of issues and actions, .36 for locus of control, and .49 for intention to act and ERB. Other meta-analyses have found even higher relationships between intention to act and behavior. For example, Van den Putte (1991) as cited by Eagly and Chaiken (1993) found a mean correlation of .60. In light of (1) the significant improvements in five of the eight measured environmental stewardship characteristics, (2) the correlations between these variables and ERB identified through meta-analyses (Bamberg & Mo¨ser, 2007; Hines et al., 1986/1987), and (3) the fact that intention to act is consistently the best predictor of behavior, including ERBs (Bamberg & Mo¨ser, 2007), BWET funded MWEEs are likely to increase participating students’ ERBs. At the same time, NOAA B-WET funded programs are not reaching their full potential in fostering ERB. The increase in participating students’ intention to act was small relative to that of students in the comparison group. In addition, there were no statistically significant changes in three of the measured environmental stewardship characteristics: environmental sensitivity, personal responsibility and group locus of control. The lack of change in environmental sensitivity is not surprising. Research

suggests that environmental sensitivity tends to be developed through providing young students with extended positive outdoor experiences, guided by role models (Chawla & Derr, 2012). Participating students spent limited time outdoors and a few, none at all. We also do not know about the type of outdoor learning they experienced. For example, to what extent students participated in nature appreciation activities or were exposed to individuals they would consider role models. The lack of change in personal responsibility indicates that the connections students made between their actions and the environment were not strengthened in a way that increased their moral norms. Participating students may feel that environmental problems are others’ responsibility or too large for them to own. The lack of change in group locus of control suggests that participating students may not have had the opportunity to engage in collaborative environmental actions or that they did not feel these actions made a positive difference. We know that environmental sensitivity, personal responsibility, and group locus of control are also important predictors of ERB. The meta-analysis by Hines et al. (1986/1987) identified the respective mean correlations as .35 for environmental sensitivity, .33 for personal responsibility, and .36 for locus of control and ERB. Moreover, Bamberg and Mo¨ser’s (2007) test of a meta-analysis based ERB model suggests that in combination these variables explain 52% of the variance in intention to act. Analysis of the relationships between specific MWEE practices and students’ environmental stewardship characteristics shed insight into what types of instruction may be able to enhance these qualities and thus, ultimately students’ ERBs. As described in the results section, the analyses identified which specific instructional practices were associated with each of the eight respective student environmental stewardship characteristics and thus, which practices have the potential to enhance these qualities. For example, students’ sense of personal responsibility may be strengthened by engaging them in environmental action projects or select science inquiry steps. However, when students have the opportunity to investigate their community’s environmental problems, care must be taken that their feelings of personal responsibility do not decline. This could be the case if, as suggested earlier, what they learn leads them to believe that local environmental problems are others’ responsibility. Overall, these analyses also largely support the instructional practices NOAA B-WET promotes through its MWEEs. Students

Please cite this article in press as: M. Zint, et al.. Evaluating Meaningful Watershed Educational Experiences: An exploration into the effects on participating students’ environmental stewardship characteristics and the relationships between these predictors of environmentally responsible behavior. Studies in Educational Evaluation (2014), http://dx.doi.org/10.1016/j.stueduc.2013.07.002

Knowledge of ecology

FULL MODEL Environmental sensitivity Knowledge of ecology Knowledge of issues Personal responsibility Knowledge of actions Group Locus of control Individual locus of control L1 Residual variance L2 Residual intercept variance * ** ***

p  .10. p  .05. p  .001.

b

SE

0.22

0.46

0.31

0.30

0.05

0.06

0.04

0.03

0.13

0.12

12%

16%

14%

16%

10%

23%

28%

b SE p 0.163 0.053 0.002

0.729 0.179 <.001

b **

***

***

**

0.161 0.209 0.441

0.255 0.07

***

0.059 0.077 0.442

0.324 0.191 0.091

0.007

0.054 0.051 0.289

0.175 0.127 0.169

0.029 0.043 0.5

**

SE p 0.284 0.055 <.001

<.001

0.065 0.915 **

b ***

0.036 0.023 0.112 0.153 0.067 0.024

0.134 0.059 0.024

0.246 0.083 0.004

***

Intention to act

0.22

0.262 0.172 0.131

0.348 0.067 <.001

Individual locus of control

0.38

0.085 0.021 <.001 **

Group locus of control

0.28

0.014 0.026 0.594 0.223 0.074 0.003

Knowledge of actions

2.26

b SE p 0.095 0.158 0.549

p

Personal responsibility

b **

0.011 0.019 0.549 **

0.276 0.075 <.001

***

0.134 0.069 0.053

*

0.073 0.046 0.117

SE p 0.139 0.047 0.003

SE p 0.045 0.052 0.392

b SE p 0.084 0.079 0.285

b SE p 0.354 0.047 <.001

0.036 0.021 0.091

0.046 0.032 0.151

0.037 0.018 0.044

**

0.064 0.093 0.491

0.14

**

0.119 0.087 0.172

0.178 0.052 <.00***

0.198 0.055 <.001

***

0.018 0.061 0.769

0.188 0.051 <.001

***

0.118 0.057 0.04

**

0.302 0.067 <.001 0.252 0.055 <.001 0.103 0.038 0.007

***

**

***

0.278 0.104 0.008 0.021 0.096 0.823

0.008

0.042 0.852

**

0.055 0.012

0.249 0.063 <.001 0.068 0.056 0.228 0.009

0.038 0.812

0.31

1.84

0.23

0.26

0.18

0.21

0.47

0.18

0.02

0.25

0.00

0.01

0.00

0.02

0.04

0.00

***

***

M. Zint et al. / Studies in Educational Evaluation xxx (2013) xxx–xxx

UNCONDITIONAL MODEL LI Residual 0.34 Variance 0.12 L2 Residual Intercept Variance 27% Intraclass Correlation Coefficient (ICC)

Knowledge of issues

G Model

Environmental sensitivity

JSEE-474; No. of Pages 14

12

Please cite this article in press as: M. Zint, et al.. Evaluating Meaningful Watershed Educational Experiences: An exploration into the effects on participating students’ environmental stewardship characteristics and the relationships between these predictors of environmentally responsible behavior. Studies in Educational Evaluation (2014), http://dx.doi.org/10.1016/j.stueduc.2013.07.002

Table 5 Results from multi-level analyses exploring relationships between students’ environmental stewardship characteristics (n = 434).

G Model

JSEE-474; No. of Pages 14 M. Zint et al. / Studies in Educational Evaluation xxx (2013) xxx–xxx

who engaged in the science inquiry steps of analyzing data or reflection scored higher in more environmental stewardship characteristics than those who did not. In light of the MWEE practices students reported experiencing, this finding may be attributed to (1) these practices’ experiential nature, (2) students viewing what they learn through these types of instruction as relevant, and (3) the fact that these practices tend to incorporate opportunities for reflection. Engaging students in more, and certain ERBs (e.g., giving presentations about their findings to others) appears to be a particularly valuable environmental education practice. This conclusion is supported by the relatively high number of significant relationships between engaging in more and particular types of environmental actions. In some contrast, having opportunities to learn outdoors per se or the amount of time students’ spent learning outdoors did not seem as valuable with few significant relationships found between these practices and students’ environmental stewardship characteristics. Our study’s results are mostly consistent with prior environmental education and related research findings. For example, a review of the effectiveness of inquiry-based science instruction found this practice increased students’ understanding of science concepts when they were actively engaged in thinking about and drawing conclusions from data (Minner, Jurist, & Century, 2010). Reviews of the effectiveness of inquiry-based environmental issue investigation indicated that this type of instruction consistently enhances students’ environmental stewardship characteristics with the exception of individual and group locus of control (Marcinkowski, 2004; Volk & McBeth, 2012). In general, reviews of environmental education research and program evaluation suggest that students tend to report engaging in more ERBs when they are involved in such active learning processes than when information is passively shared with them (Leeming et al., 1993; Zelezny, 1999; Zint, 2012). Researchers have also highlighted the need for environmental educators to consider what is important to students and to help students make linkages between these values and the environment (Covitt, 2004). Similarly, opportunities to reflect have been identified as important because they help students integrate experiences (Rickinson, 2001). Engaging students in environmental actions has been found effective in enhancing environmental stewardship outcomes when conducted as part of service learning (Zint, 2012), including within watershed education contexts (Eflin & Sheaffer, 2006). In some contrast, a synthesis of research on outdoor learning (Rickinson et al., 2004) identified only two studies that were able to link outdoor instruction to ERB (Bogner, 2002; Mittelstaedt, Sanker, & Vanderveer, 1999). Moreover, there are only two studies of outdoor fieldwork and field trip programs within a watershed education context that found evidence connecting these practices to ERB (Bodzin, 2008; Zint et al., 2002). Finally, our study’s finding that program duration was associated with very few environmental stewardship characteristics is contrary to results from environmental education research and evaluation reviews. These reviews suggest that programs that are longer (e.g., two years) tend to be more effective in fostering ERBs than ones that are shorter (e.g., 10 h) (Rickinson, 2001; Zelezny, 1999; Zint, 2012). This study also provides an example of how evaluations studies can answer decision makers’ questions about their program as well as contribute to environmental education research and theory (Zint, 2012). For one, our study adds to the relatively limited body of research providing evidence that environmental education programs can foster ERBs (Leeming et al., 1993; Rickinson, 2001; Zelezny, 1999; Zint, 2012). Although our study does so indirectly, through a focus on changes in environmental stewardship characteristics, rather than ERB directly, the outcomes we measured are all consistent predictors of ERB (Bamberg & Mo¨ser,

13

2007; Hines et al., 1986/1987; Hungerford & Volk, 1990). Given the challenges and costs associated with measuring changes in actual ERB, we believe this theory based approach to evaluating programs for behavioral outcomes is more feasible in the majority of environmental education contexts. Indeed, we were also able to successfully apply this approach to evaluating the relative benefits of different types of environmental education interventions for a non-profit organization (Zint et al., 2002). In addition, this study provides insight into the practices that may be able to enhance individuals’ environmental stewardship characteristics and thus, increase their ERBs. For example, it highlights the potential benefits of engaging students in stewardship actions during their environmental education experiences, a practice for which there has been limited quantitative evidence of its effectiveness. Finally, our study contributes toward an improved theory of the relationships between environmental stewardship characteristics and thus, to improving environmental education models of ERB. To the best of our knowledge, we are the first to empirically test the simultaneous relationships between environmental stewardship characteristics through path analyses of models inspired by Hines et al. (1986/1987) and Hungerford and Volk (1990). Unsatisfactory fit indices suggest that the two models did not fully capture the relationships between the environmental stewardship characteristics. At the same time, however, they predicted a high amount of variance in intention to act and all of their paths were statistically significant. Moreover, results from the multi-level analyses exploring the relationships between the environmental stewardship characteristics, confirm the expectations by Hungerford and Volk (1990) that they operate in a ‘‘synergistic manner’’ (p. 11). In combination, these results provide a basis for future studies with larger sample sizes. Revised models can be tested using path analysis or ideally, structural equation modeling, to account for measurement error. Insights gained from such models will provide an improved understanding of how environmental educators can foster ERBs and thus, make meaningful contributions to the protection and restoration of our planet through their programs. Acknowledgements Funding for this evaluation was provided by the National Oceanic and Atmospheric Administration’s Chesapeake Bay Office, The Chesapeake Bay Trust, and The Keith Campbell Foundation for the Environment. We are also especially grateful to Nick Montgomery for the initial analyses he conducted of the evaluation’s data. References Ajzen, I., & Fishbein, M. (1980). Understanding attitudes and predicting social behavior. Englewood-Cliffs, NJ: Prentice-Hall. Bamberg, S., & Mo¨ser, G. (2007). Twenty years after Hines, Hungerford, and Tomera: A new meta-analysis of psycho-social determinants of pro-environmental behavior. Journal of Environmental Psychology, 27, 14–25, (2007). Bodzin, A. M. (2008). Integrating instructional technologies in a local watershed investigation with urban elementary learners. Journal of Environmental Education, 39(2), 47–57. Bogner, F. X. (2002). The influence of a residential outdoor programme to pupil’s environmental perception. European Journal of Psychology of Education, 17(1), 19– 34. Chawla, L., & Derr, V. (2012). The development of conservation behaviors in childhood and youth. In S. Clayton (Ed.), Handbook on Environmental and Conservation Psychology. New York, NY: Oxford University Press. Covitt, B. (2004). Motivation in environmental education: Supporting middle school students’ movtives for helping the Chesapeake Bay. Dissertation Abstracts International65(2) 409-A. Eagly, A. H., & Chaiken, S. (1993). The psychology of attitudes. Fort Worth, TX: Harcourt, Brace, & Janovich. Eflin, J., & Sheaffer, A. L. (2006). Service-learning in watershed-based initiatives: Keys to education for sustainability in geography? Journal of Geography, 105(1), 33–44. Gelman, A., & Hill, J. (2007). Data analysis using regression and multilevel/hierarchical models. Cambridge, UK: Cambridge University Press.

Please cite this article in press as: M. Zint, et al.. Evaluating Meaningful Watershed Educational Experiences: An exploration into the effects on participating students’ environmental stewardship characteristics and the relationships between these predictors of environmentally responsible behavior. Studies in Educational Evaluation (2014), http://dx.doi.org/10.1016/j.stueduc.2013.07.002

G Model

JSEE-474; No. of Pages 14 14

M. Zint et al. / Studies in Educational Evaluation xxx (2013) xxx–xxx

Heimlich, J., & Ardoin, N. (2008). Understanding behavior to understand behavior change: A literature review. Environmental Education Research, 14(3), 215–237. Hines, J. M., Hungerford, H. R., & Tomera, A. (1986/1987). Analysis and synthesis of research on environmental behavior: A meta analysis. Journal of Environmental Education, 18(2), 1–8. Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. Hungerford, H. R., & Volk, T. L. (1990). Changing learner behavior through environmental education. Journal of Environmental Education, 21(3), 8–21. Kline, R. B. (2011). Principles and practice of structural equation modeling (3rd ed.). New York, NY: The Guilford Press. Kumler, L. (2011). Students of action? A comparative investigation of secondary science and social studies students’ action repertoires in a land use context. Journal of Environmental Education, 42(1), 14–29. Leeming, F. C., Dwyer, W. O., Porter, B. E., & Coburn, M. K. (1993). Outcome research in environmental education. Journal of Environmental Education, 24(4), 8–21. MacCallum, R. C., Browne, M. W., & Sugawara, H. M. (1996). Power analysis and determination of sample size for covariance structure modeling. Psychological Methods, 1(2), 130–149. Marcinkowski, T. (2004). Using a logic model to review and analyze an environmental education program. In Volk, T. (Ed.). NAAEE Monograph Series (vol. 1). Washington, DC: North American Association for Environmental Education. Minner, D. D., Jurist, A., & Century, J. (2010). Inquiry-based science instruction – What is it and does it matter? Results from a research synthesis years 1984–2002. Journal of Research in Science Teaching, 47(4), 474–496. Mittelstaedt, R., Sanker, L., & Vanderveer, B. (1999). Impact of a week-long experiential education program on environmental attitude and awareness. Journal of Experiential Education, 22(3), 138–148.

Nowak, P., Wilke, R., Marcinkowski, T., Hungerford, H., & McKeown-Ice, R. (1995). The secondary school environmental literacy instrument. Ann Arbor, MI: National Consortium for Environmental Education and Training. Nunnally, J. C., & Bernstain, I. H. (1994). Psychometric theory. New York: McGraw-Hill. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks, CA: Sage Publications. Rickinson, M. (2001). Learners and learning in environmental education. Environmental Education Research, 7(3), 207–320. Rickinson, M., Dillon, J., Teamey, K., Morris, M., Choi, M. Y., Sanders, D., et al. (2004). A review of research on outdoor learning. London: National Foundation of Educational Research and King’s College London. Tucker, L. R., & Lewis, C. (1973). A reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38(1), 1–10. UNESCO. (1978). The Tbilisi declaration: Final report of the Intergovernmental Conference on Environmental Education. Paris: UNESCO. Van den Putte, B. (1991). 20 years of the theory of reasoned action of Fishbein and Ajzen: A. meta-analysis. University of Amsterdam (unpublished manuscript). Volk, T., & McBeth, W. (2012). Inquiry based learning: Leveraging students’ natural curiosity to learn about their environment. Conference Presentation http:// www.chesapeake.org/stac/presentations/213_Volk-Best%20Practices%202012. pdf. Zelezny, L. (1999). Educational interventions that improve environmental behaviors. Journal of Environmental Education, 31(1), 5–14. Zint, M. (2012). Advancing environmental education program evaluation: Insights from a review of behavioral outcome evaluations. In J. Dillon, M. Brody, B. Stephenson, & A. Wals (Eds.), International handbook of research in environmental education. New York: Routledge. Zint, M., Kraemer, A., Northway, H., & Lim, M. (2002). An evaluation of the Chesapeake Bay Foundation’s conservation education program. Conservation Biology, 16(3), 1–9.

Please cite this article in press as: M. Zint, et al.. Evaluating Meaningful Watershed Educational Experiences: An exploration into the effects on participating students’ environmental stewardship characteristics and the relationships between these predictors of environmentally responsible behavior. Studies in Educational Evaluation (2014), http://dx.doi.org/10.1016/j.stueduc.2013.07.002