A longitudinal assessment of the effectiveness of a school-based mentoring program in middle school

Contemporary Educational Psychology 38 (2013) 11–21

Contents lists available at SciVerse ScienceDirect

Contemporary Educational Psychology journal homepage: www.elsevier.com/locate/cedpsych

A longitudinal assessment of the effectiveness of a school-based mentoring program in middle school José Carlos Núñez a,⇑, Pedro Rosário b, Guillermo Vallejo a, Julio Antonio González-Pienda a a b

Department of Psychology, Plaza Feijo, s/n, 33003 Oviedo, University of Oviedo, Spain Department of Psychology, Campus de Gualtar, Braga, University of Minho, Portugal

a r t i c l e

i n f o

Article history: Available online 26 October 2012 Keywords: School-based mentoring program Self-regulated learning Self-efficacy Perceived usefulness Academic performance Middle school students

a b s t r a c t This work assessed the efficacy of a middle-school-based mentoring program designed to increase student use of self-regulated learning (SRL) strategies, self-efficacy for and the perceived usefulness of SRL as well as mathematics and language achievement. A longitudinal cluster randomized trial study design obtained evidence that found differential effects of a school-based mentoring program. Specifically, the performance of 94 seventh grade students naturally nested within four classrooms was measured at baseline and after 3, 6, and 9 months. Two classrooms were each randomly assigned to treatment or control conditions. First, the results indicated that participation in the mentoring program led to significant improvements with regard to all the dependent variables after the 9-month intervention, and significant effects had been observed at 6 months for some variables. Second, the program appears to play a more important role for SRL variables compared with academic variables. Third, the effect sizes were small, small-medium, or medium depending on academic mentoring, the type of variable used to assess the efficacy of the program, or the level of analysis considered, respectively. The effect size of this intervention was equal to or greater than those reported in prior studies. In conclusion, our findings underline the importance of academic mentoring programs that practice SRL strategies and emphasize the relevance of using study designs that provide both cross-sectional and longitudinal data. Ó 2012 Elsevier Inc. All rights reserved.

1. Introduction

1.1. Mentoring interventions

International reports (e.g., OECD, 2010) have repeatedly underlined the need to develop autonomy and responsibility in middle school students to improve their self-regulated learning (SRL) and academic achievement. A mentoring intervention is a relevant tool that can promote students school engagement and counter academic failure (e.g., Eby, Allen, Evans, Ng, & DuBois, 2008; DuBois, Holloway, Valentine, & Cooper, 2002; Rhodes, 2008). For example, approximately 2.5 million youths participate in mentoring programs in the United States each year (Karcher, 2005). However, more research is needed concerning the effectiveness of these programs because previous work taking an intervention perspective has been primarily cross-sectional and has varied by time. Although cross-sectional studies are able to compare across different age groups, they do not provide information as to how participants change during the corresponding time period. Thus, this paper uses a longitudinal study to analyze the efficacy of a school-based mentoring program across a school year with repeated measures.

In the Odyssey, Homer described Mentor as a sage and faithful friend of Ulysses, King of Ithaca. When Ulysses left for the Trojan War, he entrusted his son, Telemachus, and his wife, Penelope, to the care of Mentor. The term ‘‘mentoring’’ originates from this idea: a planned and guided pairing of a more expert and skilled person with one who is less skilled to achieve previously established goals. Both individuals develop personally as a result of the shared help that can occur at different stages of an individual’s life (Rhodes & DuBois, 2008) and assume different formats (e.g., youth mentoring, academic mentoring, and workplace mentoring; Eby et al., 2008; Spencer, 2007a). The present investigation shows the results of a school-based intervention program (e.g., Edmondson & White, 1998) assuming an academic mentoring format (Dorsey & Baker, 2004; Sambunjak, Straus, & Marusic, 2006). Following Eby et al. (2008), academic mentoring is an educational process in which an educator, frequently a teacher, counsels one or several students about issues from academic (e.g., study support) and nonacademic areas (e.g., relational problems with their classmates). Academic mentoring habitually focuses the intervention on academic achievement, and adjustment to academic life (Dorsey & Baker, 2004; Jacobi, 1991; Sambunjak et al., 2006).

⇑ Corresponding author. E-mail addresses: [email protected] (J.C. Núñez), [email protected] (P. Rosário), [email protected] (G. Vallejo), [email protected] (J.A. González-Pienda). 0361-476X/$ - see front matter Ó 2012 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.cedpsych.2012.10.002

12

J.C. Núñez et al. / Contemporary Educational Psychology 38 (2013) 11–21

Despite the recent promotion and expansion of mentoring programs, primarily in educational settings, there remain few and inconsistent data regarding the efficacy of this type of intervention (DuBois et al., 2002; Rhodes, 2008; Schwartz, Rhodes, Chan, & Herrera, 2011; Thompson & Kelly-Vance, 2001). One possible explanation is that these mentoring programs have different formats and characteristics (e.g., theoretical frameworks, frequency of contacts, target populations, and types of assessment), which greatly impair their ability to be compared and judged for their efficacy. An additional problem is that the investigation to identify the causes that lead to failure of such programs is limited (Spencer, 2007b). Recent meta-analyses and literature reviews focusing on mentoring argue that the deployed programs should fulfill a series of features or good practices to assure their positive effect. For example, DuBois et al. (2002) underlined five practices that predict the positive effects of mentoring programs: (a) providing ongoing mentor training; (b) structuring activities for mentors and mentees; (c) clarifying expectations concerning meeting frequency; (d) involving parents; and (e) monitoring program implementation. Eby et al. (2008) conducted a meta-analysis based on 116 research papers focusing on youth, academic, and workplace mentoring programs to calculate the effect size of these programs on the mentees’ behavioral, attitudinal, health-related, relational, motivational, and career outcomes. We underline two results from this study: first, mentoring is more strongly related to mentee attitudes and motivation/involvement than their behaviors (e.g., academic achievement), health, or career outcomes. Eby et al. reasoned that this finding might be because attitudes are more malleable than others (e.g., personality variables). Second, these authors found that, consistent with the majority of previous studies, the overall magnitude of the association between mentoring and outcomes was small; nevertheless, the results of their analysis also suggested that academic mentoring is more strongly associated with positive outcomes than youth and workplace mentoring. Based on the criteria established by Cohen (1988), the results indicated that the effect sizes were small-medium for academic mentoring, small for workplace mentoring, and not significant for youth mentoring. Thus, Eby et al. (2008) argued that, even in the case of academic and workplace mentoring programs, it is not possible to assert that positive outcomes are caused by the mentoring relationship because most of the studies included in the meta-analysis were cross-sectional or of a non-experimental nature. Therefore, these same authors suggest that future research should use experimental designs, investigate academic mentoring intervention outcomes over time, and should be anchored in clear frameworks. 1.2. Promoting SRL strategies and academic achievement The research of SRL emerged as a result of numerous attempts in responding to how students proactively control their learning, directing, and managing their cognitive and motivational processes in the direction of their goals (Zimmerman, 2008). The research, carried out up to date, contains a robust corpus of empirical data confirming a strong relation between SRL and academic success (e.g., Boekaerts & Corno, 2005; Liem, Lau, & Nie, 2008; Núñez, Cerezo, et al., 2011; Rosário, Núñez, et al., 2010; Zimmerman & Martinez-Pons, 1988). Furthermore, research shows that students who are trained in SRL strategies (e.g., goal setting, note taking and self-monitoring) activate high levels of motivation, are more engaged in academic tasks as well as achieve better results (e.g., Baker, Chard, Ketterlin-Geller, Apichatabutra, & Doabler, 2009; Boekaerts & Corno, 2005; Dignath, Buettner, & Langfeldt, 2008; Guthrie, McRae, & Klauda, 2007; Rosário, González-Pienda, et al., 2010; Wood, Bandura, & Bailey, 1990; Zimmerman, 2008). Recently, while aiming to capture students online efforts to learn, Zimmerman et al. developed a microanalytic methodology for targeting SRL processes

and learning strategies (Cleary & Zimmerman, 2001, 2004; Kitsantas & Zimmerman, 2002). This microanalytic methodology is rooted in the social cognitive framework and deals with students’ motivation and their role in SRL. Multiple assessment tools have been utilized to capture students’ motivational beliefs, feelings (e.g., self-efficacy) and self-regulation processes (e.g., self-monitoring) as well as their use of learning strategies (e.g., goal setting, keeping records, seeking help) (Zimmerman, 2002) while performing academic tasks or activities. This microanalytic methodology aims at collecting information which can be used to help students with their learning tasks, improve their study methods or help with the design of sensitive school based interventions for promoting students’ academic achievement (Cleary & Zimmerman, 2004; Zimmerman, 2008). Within the framework of SRL, Dignath et al.’s (2008) meta-analysis on the study of the efficacy of SRL interventions for primary school students, suggests that the most effective programs of training in self-regulation processes should: (a) present a solid theoretical base (the most efficient programs are based on the social cognitive theories); (b) explicitly deal with declarative knowledge of learning strategies (what they are and their utility to in achieving academic goals); (c) and also develop procedural knowledge, training in cognitive (e.g., organization of information and problem solving), metacognitive (e.g., planning and monitoring strategies), and motivational (e.g., feedback) processes. 1.3. Purpose of the present study Taking into consideration the recommendations regarding the nature and characteristics of mentoring programs (e.g., Karcher, 2005; Rhodes, 2008), the suggestions of Dignath et al. (2008) on the effectiveness of training programs of SRL strategies, as well as the methodological aspects that Eby et al. (2008) noted, an empirical study was conducted using an experimental design with an equivalent control group and four measurements across the school year to assess the efficacy of a school–based mentoring program and increase the use of SRL strategies and academic achievement. This school–based mentoring program was (a) theoretically based on a social cognitive framework; (b) dealt with and appraised three types of knowledge (declarative, procedural, and conditional); (c) was implemented during an academic school year; and (d) repeatedly measured the dependent variables to determine an effective time for intervention to begin (depending on the variable considered). Lastly, we assessed the efficacy of this program with regard to the comparison group. In summary, this study sought to evaluate and interpret the effectiveness of a school-based mentoring program designed to increase self-regulated learning and academic achievement of middle-school students (seventh graders). In particular, we examined the effects of the mentoring program on student SRL strategies, self-efficacy of SRL, and perceived utility of SRL as well as language and mathematics achievement at four different instances (times) across the school year using students nested within four seventh grade classes. Two classrooms were randomly assigned for treatment or control conditions. The above mentioned variables were chosen for their relevance with regard to the learning process (Liem et al., 2008). We hypothesize that students working within the mentoring program, compared to control students, via learning and training in the use of SRL strategies, would increase their selfefficacy for and the perceived usefulness of SRL, both of which lead to higher achievement in language and mathematics. Specifically, this study sought answers to the following questions: (a) Did the use of SRL strategies, self-efficacy for use of SRL strategies, perceived usefulness of SRL strategies, and language and mathematics achievement increase in students who enrolled in the mentoring program at the end of the intervention compared

J.C. Núñez et al. / Contemporary Educational Psychology 38 (2013) 11–21

to control students? (b) Did the effect size of academic mentoring vary as a function of the dependent variable? (c) Did the effect size of academic mentoring depend on the implementation time of the program (i.e., T1 through T4)? and (d) Did the effect size of the interaction between intervention type (mentoring vs. comparison group) and implementation time (T1 through T4) vary as a function of the dependent variable? The analyses of these four questions accounted for the potential effect of the variable study time. 2. Method 2.1. School and participant characteristics These investigators selected 7th grade because it is the beginning of middle school in the Portuguese educational system (7th to 9th grade). The transition from elementary to middle school represents a great challenge for many students because an adequate use of academic competences is expected in this setting of higher autonomy and responsibility. The two middle schools selected for this study are located in an urban school district in the north of Portugal and were randomly chosen from the pool of seven middle schools that agreed to participate. One school had six 7th grade classes, and the other had five 7th grade classes. The families of all students are lower-middle class as the high percentage of students (43% and 39.2% respectively) who receive free or reduced-price lunches demonstrates. These demographic data were collected from the offices of the participating schools. Two classes were randomly chosen for each participating school. One class from each school was randomly assigned as the experimental (mentoring) group, and one was randomly assigned as the comparison group. Thus, two classes comprised the experimental group (47 students), and the other two classes comprised the comparison group (47 students). Twenty-seven boys and 20 girls (57.4% and 42.6%, respectively), with ages ranging between 11–14 years (M = 12.25, SD = 0.70), participated as mentees in the experimental group. These 47 mentees were randomly divided into 12 groups of 4 students (one group had 3 students). A randomly assigned teacher/ mentor was responsible for each group. The comparison group was composed of 24 boys and 23 girls (51% and 49%) with ages ranging between 11 and 14 years (M = 12.55, SD = 0.85). All 94 students provided the required data at the four time points (T1–T4). Because differences in prior performance between the mentoring and comparison groups might significantly influence the effectiveness of the intervention, we conducted an ANOVA with regard to mathematics and Portuguese language grade marks from the end of the previous year. The results indicated there were no significant differences between the groups of students with regard to either Portuguese (F(1, 92) = .088, p = .767, g2p = .001) or mathematics (F(1, 92) = .374, p = .542, g2p = .004). Thus, the groups were confirmed to be equivalent with regard to academic achievement prior to the intervention (kwilks = .995, F(2, 91) = .214, p = .808, g2p = .005). The twelve teachers/mentors (8 women and 4 men), aged between 31–42 years, had an average of 8 years of teaching experience and were randomly chosen from the 20 volunteers across the two schools. All teachers/mentors fulfilled the following study requirements: (a) they volunteered for the project; (b) they had at least 5 years of middle school teaching experience; (c) they were available to participate in 3 workshops (one for each academic term) and in biweekly supervision sessions after class. 2.2. Measures Measures of SRL strategies, self-efficacy for and the perceived usefulness of SRL, and mathematics and Portuguese achievement

13

were used to assess the mentoring program. Four measurements of the first three variables were recorded during the school year in October, January, April, and June. Only three student achievement observations corresponded to the school term: January, April, and June. 2.2.1. SRL strategies The students’ SRL Strategies Inventory assesses nine self-regulated learning strategies concerning the three phases of the SRL process: Planning (i.e., ‘‘I make a plan before I begin writing. I think about what I want to say and how I need to write it.’’), Execution (i.e., ‘‘If I become distracted or lose concentration while I am in class or studying, then I usually try to regain my to achieve my goals.’’), and Evaluation (i.e., ‘‘I compare the grades I receive with the goals I set for that subject.’’) (Rosário, Núñez, et al., 2010). These items were rated on a 5-point Likert scale, ranging from 1 (never) to 5 (always).The reliability indices (i.e., Cronbach’s alpha) for this study were .80 for Planning, .85 for Execution, and .87 for Evaluation. The results of a confirmatory factor analysis were highly satisfactory, v2(24) = 57.9, p < .001, v2/df = 2.411, GFI = .984, AGFI = .970, CFI = .989, RMSEA = .043 (.029–.058), thereby obtaining evidence of the construct validity of this inventory (Rosário, González-Pienda, et al., 2010). 2.2.2. Self-efficacy for SRL Student self-efficacy for SRL assesses students’ beliefs in their capabilities to regulate their own learning by using a variety of learning strategies (e.g., taking notes and goal setting; Rosário et al., 2012). Following Zimmerman, Bandura, and Martinez-Pons (1992), the 10 items that assess student self-efficacy for SRL began with the phrase: ‘‘How well can you. . .’’ and were completed with statements such as ‘‘. . .take notes and later elaborate upon them to learn the material in detail’’ or ‘‘. . .use strategies to comprehensively memorize the study material’’. These items were rated on a 5-point Likert scale ranging from 1 (not very well) to 5 (very well). Factor analyses grouped the 10 items into two strategy factors: organization and assessment. The Cronbach’s alphas were .85 and .90 for the former and latter factors, respectively. The results of the confirmatory factor analysis were highly satisfactory, v2(35) = 119.6, p < .001, v2/df = 3.146, G9 FI = .969, AGFI = .951, CFI = .970, RMSEA = .057 (.046–.068), thereby obtaining evidence of the construct validity of this inventory (Rosário, González-Pienda, et al., 2010). 2.2.3. Perceived usefulness The perceived usefulness of SRL strategies in the academic setting assesses students’ perceived utilities regarding a variety of self-regulated learning strategies (Rosário et al., 2012). Ten items begin with the phrase, ‘‘How useful do you think it is to. . .’’ and were completed with statements such as ‘‘. . .take notes and later elaborate upon them to learn the material in detail’’ or ‘‘. . .use strategies to comprehensively memorize the study material’’. The items are presented using a 5-point Likert scale ranging from 1 (not very useful) to 5 (very useful) and are grouped into two strategy factors: organization and assessment. The Cronbach’s alphas were .89 and .91 for the former and latter factors, respectively. The results of the confirmatory factor analysis were highly satisfactory, v2(35) = 102.6, p < .001, v2/df = 2.930, GFI = .972, AGFI = .956, CFI = .982, RMSEA = .051 (.040–.062), thereby obtaining evidence of the construct validity of this inventory (Rosário, González-Pienda, et al., 2010). 2.2.4. Student study time Study time was assessed using an open-ended question concerning the number of hours students dedicated to studying math and language over 7 days. Following Plant, Ericsson, Hill, and As-

14

J.C. Núñez et al. / Contemporary Educational Psychology 38 (2013) 11–21

berg (2005), students responded to this question each day by completing a study time log. At the end of the week, the 7 daily logs were delivered in a sealed envelope. 2.2.5. Academic achievement Academic achievement was assessed using students’ school grades for Mathematics and Portuguese, which were collected from the schools’ secretariats at the end of the academic year. Portuguese Compulsory Education classifies the following grades as 1 and 2 (negative), 3 (fair), 4 (good), and 5 (excellent). 2.3. Procedure The enrolled schools sent invitations to their teachers and authorization requests to the parents of the seventh graders for the four selected classes. Students enrolled in the comparison group followed a weekly class oriented to promote study kills. This school activity will be detailed in the section 2.4. The mentoring program was conducted during an academic school year (mid- September through the end of June) with seventh grade students (12- to 13-year-olds) from public schools. The weekly 1-h mentoring sessions occurred after classes and lasted the entire school year. Twelve teachers/mentors participated in a 2-day workshop before the program was implemented. Teachers were trained in SRL processes, mentoring, and the procedural dynamics of the schoolbased mentoring program. They also participated in two 1-day workshops during the school year to monitor the intervention procedures and help them cope with difficulties found. These training sessions not only included theoretical presentations as well as the study and discussion of these concepts during the workshops but also role-playing sessions in which teachers assumed either the role of mentor or student. Teachers received a dossier with mentoring session record sheets, a list of activities and the procedure for each session. The members of the research team directed every workshop and coached the teachers enrolled in the mentoring program. In addition to these intensive training periods, the mentors met with the researchers biweekly for 1 h to address project issues and prevent the mentors to withdrawing from the planned protocol by adding new components based on their experience of what was working. Thus, tutors presented the previous mentoring session record sheets in each session. The procedures were monitored to assure that the twelve tutors were similarly following the program as planned. 2.4. Mentoring and comparison programs 2.4.1. Mentoring program The mentoring program was designed to develop student SRL strategies (e.g., goal-setting, self-monitoring, self-reflection, strategic planning, and organizational strategies) to increase motivation with regard to school tasks and academic achievement. Thus,

declarative, procedural, and conditional knowledge of SRL strategies is the central axis of the program. This program is based on the social cognitive framework that assumes that contextual variables and the learning setting play important roles in student motivation and self-regulation. Therefore, this program provides a clear structure that guides the work of the mentors; nevertheless, mentors are able to adjust the time they spend on certain activities depending on the group dynamics to facilitate their SRL (e.g., deeply analyze a behavior to facilitate strategic attributions and adaptive inferences). Zimmerman (2008) argues for the importance of microanalytic measures to assess student online efforts to learn. This investigation used this methodology to promote student assessment of and reflection upon the processes and strategies used while learning. Thus, the 14 self-regulation strategies that Zimmerman and Martinez-Pons (1986) identified (self-evaluating, organizing and transforming, goal-setting and planning, seeking information, keeping records and monitoring, environment structuring, selfconsequating, rehearsing and memorizing, seeking peer-, teacher-, or adult-assistance, reviewing tests, and notes and texts), were applied to the 6 SRL contexts that Zimmerman and Martinez-Pons (1988) identified (classroom situations, studying at home, completing writing assignments, completing mathematic assignments, preparing for and taking tests, and completing homework; see, Table 1). For each SRL strategy in each learning situation, the mentees reported their declarative knowledge, procedural knowledge regarding the strategy, and conditional knowledge (e.g., self-evaluations concerning classroom situations, studying at home, completed writing assignments). In the present study the goal of this microanalytic methodology was to help students to reflect upon their study practices, discuss how they usually implement their strategies with the group as well as other ways they could act (including how and when) across diverse learning settings and situations (see Table 1). Using the Testas story-tools is one example: Testas’ Misadventures (Rosário, 2004) promotes strategic learning via tales that reveal and promote SRL processes. Testas, the main character, is similar to the students who read this story (see Table 2). The story focuses on school tasks and activities as well as how Testas copes with learning challenges in realistic daily scenarios. Therefore, Testas’ experiences provide the students with the opportunity to learn and reflect upon a repertoire of learning strategies that might be useful not only for accomplishing school tasks but also in their daily lives (Rosário, Mourão, Núñez, González-Pienda, & Solano, 2008). For each learning strategy (e.g., self-evaluating, organizing and transforming, goal-setting and planning, and seeking information), the mentor helps his or her students to reflect upon their declarative, procedural, and conditional knowledge of these learning strategies across diverse learning contexts (e.g., classroom situations, studying at home, completing writing and mathematics assignments, preparing for and taking tests, and completing homework).

Table 1 Example of the SRL microanalytic methodology. Classroom situations

Studying at home

Completing writing assignments

Completing mathematics assignments

Preparing for and taking tests

Completing homework

Self-evaluations Declarative knowledge

What does ‘‘use a self-evaluation strategy’’ (e.g., in the classroom, at home, and so on) mean? Explain What must be taken into account to self-evaluate yourself (e.g., in the classroom, at home, and so on)? Explain

Procedural knowledge

How do you usually evaluate yourself (e.g., in the classroom, at home, and so on)? What do you do if you have difficulties? Explain Consider the self-evaluation of your work (e.g., in the classroom, at home, and so on). What do you conclude?

Conditional knowledge

When is it appropriate to evaluate yourself (e.g., in the classroom, at home, and so on) and when is it inappropriate? What can be performed in case of difficulties? Explain Why is it important to evaluate yourself (e.g., in the classroom, at home, and so on)? Elaborate

J.C. Núñez et al. / Contemporary Educational Psychology 38 (2013) 11–21

15

Table 2 Book extracts that illustrate SRL strategies. 2. Organization and transformation 4. Seeking information

5. Taking notes

‘‘I’ll outline the story that I learned. It’s easy to make an outline. In fact, it’s like following a recipe: First, I read or listen to the text until I manage to understand it enough to identify the main ideas. . .’’ (Rosário, 2002, p. 37) ‘‘To make sure, he sought information about the city from the local in-crowd. . . using a map they lent him, he searched for a possible itinerary. The distance on the map was akin to a well-extended palm, and the difficulties of this route were discouraging. This trip would take many, many days’’ (Rosário, 2002, p. 57) ‘‘At that moment, I remembered from my notebook that they looked like moon craters with all those blank spaces. . .. I understood that if I don’t listen, I can’t write. If I don’t write, I won’t read. If I don’t read afterwards, then I won’t understand. If I don’t understand, then I won’t learn. If I don’t learn, then I cannot respond correctly on the exam. These should not be the steps of how I learn’’ (Rosário, 2002, p. 71)

The mentor guides discussions, explains how students can expand their strategy repertoire, develops their senses of agency and personal control, helps them to project consequences onto their performance and develops their lifelong learning skills. The mentors discussed their work plan with the research team and conducted appropriate changes during the program sessions to meet their mentees’ self-regulation needs. The mentors used various examples to specify applications of the SRL to learning situations at all the sessions. Moreover, they provided feedback to their students when they used a strategy. In addition, they trained their students to record their learning results to increase their sense of control over learning and performance (e.g., graphing their test grades and keeping a diary). Each session ended with students setting a weekly personal goal, and the next session began with an analysis of the results achieved. 2.4.2. Comparison program In both participating middle schools, all students attend a study skills class, where they learn and practice learning strategies with their teachers, as well as do their homework or study. In the present study, students in the comparison program dedicated an hour per week in class to developing techniques for schoolwork, learning processes, and study skills while the students in mentoring group where in the library carrying out school activities. The two teachers followed the same booklet on learning strategies and a similar plan of activities in these study skills classes. The booklet was comprised of activities such as: small texts to train reading comprehension and organizing information; problem solving tasks and activities to teach and foster the use of a set of learning strategies (e.g., rehearsal strategies, goal setting, and note taking). Although the work plan of students in the comparison group, developed in this study skills classes, was not exactly the same as the one the mentoring group followed in their mentoring weekly sessions out of class, both groups spent an equal number of hours working on a set of learning strategies and SRL processes. 2.5. Analyses The present investigation applied a longitudinal cluster randomized trial study design. In cluster randomized trials, small and large groups of individuals (rather than individuals themselves) are randomly assigned to experimental conditions, and individuals from the same clusters are repeatedly measured over time. This design is a natural choice for testing many educational research questions. There are numerous options for analyzing data with this type of design, and no gold-standard approach exists. However, likelihood-based mixed-effects models (multivariate and univariate) provide appropriate general analytic frameworks to determine whether changes in response profiles over time differ among treatment groups. The mixed-effects model with repeated measures (MMRMs) implemented herein included an unstructured (UN) modeling of time and the within-participant error correlation

structure. The most parsimonious structures (e.g., compound symmetry and first-order autoregressive models) are based on strong and often unrealistic assumptions for most longitudinal data (Vallejo, Fernández, Livacic-Rojas, & Tuero-Herrero, 2011a). The version of MMRM herein used leads to a normal multivariate model and directly estimates the differences between treatment groups with regard to mean changes from baseline to the endpoint for all dependent variables. The specific model implementation was always conducted by fitting a UN structure with parameters estimated by the restricted maximum likelihood (REML) estimation as implemented in SAS PROC MIXED. Initially, we modeled the effect of the intervention by considering three different models in competition that each extended a prior model in a sensible and convenient way. The first model (hereafter, Model A) assumed that the 94 students selected from two middle schools were assigned to both treatment conditions (in which the mentoring program was denoted c1 and the comparison group was denoted c2) and measured across four time points (denoted by T) for five dependent variables: SRL strategies, selfefficacy for and the perceived usefulness of SRL, mathematics achievement, and Portuguese achievement. Model A does not include the mentor/teacher variable; thus, this analysis ignores data clustering at the teacher level. The second model (hereafter, Model B) analyzed the data with the 94 students nested within the fourteen teachers/mentors (twelve in the experimental condition and two in the control condition; denoted B/C), which were measured across time (i.e., T1–T4) for the five dependent variables. Finally, a third model (hereafter, Model C) conducted a multivariate regression after adjusting for the covariate study time measured at baseline. After selecting the most parsimonious model and without ignoring any relationship among the outcomes, we tested the effects of the fitted model. As will be shown later, all the multivariate effects were significant. Thus, the next step was to explore the data to interpret the nature of the specific differences, especially those related to the group by time interaction effects. Therefore, we concentrated on procedures that would identify whether the intervention program behaves differently at four different times using both multivariate and univariate models. In addition, partial etasquared, g2p , was used as an effect size because it is the most commonly used parameter in education research literature. p-Values are reported with effect sizes and their associated confidence intervals.

3. Results The results are presented in four sections. First, the fitting of three competing models section presents the results for Models A– C to choose the best model fit to address the study questions. Multivariate MMRM analyses compared the three models to avoid the assumption of a common covariance matrix. Vallejo and Ato (2012) provided sample SAS syntax for a multivariate MMRM analysis. Second, the multivariate MMRM analyses section describes the re-

16

J.C. Núñez et al. / Contemporary Educational Psychology 38 (2013) 11–21

Table 3 Estimated mean, standard deviations, and effect sizes for the observed data on five types of outcome variables and four measurement occasions. Mentoring group

Comparison group

Effect size

M

SD

M

SD

ES

SRL strategies T1 3.531 T2 3.559 T3 3.714 T4 3.901

.425 .469 .420 .323

3.648 3.570 3.661 3.610

.412 .473 .354 .268

.28 .02 .14 .98

Self-efficacy for SRL T1 3.609 T2 3.556 T3 3.713 T4 3.907

.449 .409 .375 .361

3.612 3.525 3.567 3.646

.431 .441 .391 .351

.00 .07 .38 .73

Perceived usefulness of SRL 3.757 .389 T1 T2 3.727 .495 T3 3.947 .470 T4 4.127 .455

3.727 3.580 3.608 3.682

.417 .601 .656 .412

.06 .27 .61 .90

Language achievement T2 2.511 T3 2.575 T4 2.766

.585 .683 .698

2.681 2.702 2.787

.755 .720 .720

.25 .18 .03

Mathematics achievement T2 2.617 .739 T3 2.618 .740 T4 3.021 .821

2.600 2.638 2.766

.712 .705 .698

.02 .02 .34

Note: M = mean; SD = standard deviation; ES = effect size (defined as the betweengroup difference divided by the standard deviation over time); T1 = October; T2 = January; T3 = April; T4 = June.

sults concerning the fitted model by simultaneously considering all dependent variables once an adequate model fit was selected. Third, the univariate MMRM analyses for each dependent variable section analyzes the main effects for treatment condition and time as well as their interaction. Fourth, to correctly interpret the results of this intervention, the last section, ancillary analyses, describes how we conducted additional MMRM data analyses that incorporated the study time measurements. The means, standard deviations and effect sizes corresponding to the five dependent variables are presented in Table 3 by group and measurement time (T1 = October; T2 = January; T3 = April; T4 = June). Their correlations with regard to mathematics and Portuguese language achievement (not shown) ranged between .535 and .835 for all six terms considered as well as between .034 and .842 for the three variables more directly related to SRL.

3.1. Fitting of three competing models Table 4 shows the multivariate MMRM SAS results of the different fits of Models A–C to the mentoring intervention data; all models have the same UN covariance structure but different mean structures. Neither B/C  T nor B/C were significant; thus, the mentors of the experimental group did not differ from those of the control group. In addition, there was no relationship between the covariate study time at baseline and the subsequent responses. The remaining fixed effects yielded similar results. Model A was chosen as our ‘‘final model’’ after assessing model fit using the AIC or BIC criteria based on the REML criterion. The same conclusion was obtained after comparing the three models using deviance statistics. The empirical results presented by Gurka (2006) as well as Vallejo, Fernández, Livacic-Rojas, and Tuero-Herrero (2011b) found the REML to be appropriate for selecting the best mean structure using information criteria.

Table 4 Results of fitting three multivariate mixed-effects model repeated measures analyses. Source

Model A

Model B

Model C

F(5, 76) = 3.15 p = .012 F(60, 229) = 0.98 p = .527 F(15, 66) = 18.76 p < .0001 F(15, 66) = 6.24 p < .0001 F(180, 356) = 0.94 p = .677

F(5, 56) = 1.17 p = .335 F(5, 74) = 2.94 p = .018 F(60, 212) = .97 p = .543 F(15, 66) = 18.76 p < .0001 F(15, 66) = 6.24 p < .0001 F(180, 356) = .96 p = .618

1712.9 426.0 2132.9 2666.9

1725.3 451.0 2143.3 2679.4

Fixed effects Time study C

F(5, 87) = 3.00 p = .0151

B/C T CT

F(15, 75) = 15.65 p < .0001 F(15, 75) = 5.48 p < .0001

B/C  T Fit statistics 2 REML-LF Number parameter AIC BIC

1613.8 76.0 2033.8 2567.9

Note: C = treatment conditions; B/C = teachers nested in treatment conditions; T = observation times; C  T = interaction of treatment conditions with observation times; B/C  T = interaction of teachers nested in treatment conditions with observation times; REML-LF = restricted/maximum likelihood log-likelihood function; AIC = Akaike Information Criteria; BIC = Bayesian Information Criteria.

3.2. Multivariate MMRM analyses According to Model A, there were significant [F(5, 87.6) = 3.00, p = 0.0151] differences between the mentoring and comparison groups averaged across the observation periods for the five dependent variables when considered simultaneously. In addition there was a significant [F(15, 74.8) = 15.65, p < .0001] increase in the mean response over time after averaging across the treatment groups and simultaneously considering all dependent variables. Importantly, there was a significant [F(15, 74.8) = 5.48, p < .0001] difference between the mentoring and comparison groups over time when considering the five dependent variables simultaneously. Therefore, participant performance changes over time; however, this pattern of change is not the same for the two treatment groups. The next step was to examine whether the change was different for the treatment groups from beginning to end. As with any factorial structure, the main analysis task regarding the effects of treatment and time is to explain their interaction in a manner consistent with the research objectives. For this purpose, we compared the change from the first through the last observation for both treatment groups and computed the corresponding tetrad contrasts. To control the family-wise error (FWE) rate for all possible tetrad contrasts on the five dependent variables analyzed simultaneously, the Hochberg (1988) step-up Bonferroni inequality was applied using the ESTIMATE statement in SAS PROC MIXED and the HOC option in SAS PROC MULTTEST. Table 5 shows that after applying Hochberg’s sequentially rejective Bonferroni procedure after controlling FWE at .05, four tetrad treatment-by-time interaction contrasts were significant: CT11 CT14 vs. CT21 CT24, CT12 CT14 vs. CT22 CT24, CT13 CT14 vs. CT23 CT24, and CT11 CT13 vs. CT21 CT23. On the other hand, CT12 CT13 vs. CT21 CT23 and CT11 CT12 vs. CT21 CT22 were not significant. Interestingly, changes in the five dependent variables (considering simultaneously) were not significant until approximately 6 months of intervention (T3), and they were highly significant at the last measurement (T4) compared with the first observation (T1). Similar results were obtained using a multivariate extension of the modified Brown–Forsythe (MBF) procedure that Vallejo and Ato (2006) developed. Table 5 also shows the proportion of the sample variance accounted for by the six tetrad contrasts. Applying Cohen’s (1988)

17

J.C. Núñez et al. / Contemporary Educational Psychology 38 (2013) 11–21 Table 5 Hochberg’s adjusted p values for all possible tetrad contrasts by simultaneously considering all dependent variables. Group C1 C1 C1 C1 C1 C1

vs. vs. vs. vs. vs. vs.

C2 C2 C2 C2 C2 C2

Time

dfN

dfD

F-value

Pr > F

Adj p

Partial g2 sample (lower–upper)

T1 T2 T3 T1 T2 T1

5 5 5 5 5 5

87.20 86.20 84.55 86.15 83.26 84.45

9.39 5.96 4.40 3.60 1.48 1.16

<.0001 <.0001 .00129 .00524 .20383 .33796

<.0001 .00051 .00519 .01573 .33791 .33792

.203 .139 .106 .089 .038 .031

T4 T4 T4 T3 T3 T2

.162 .078 .039 .030 .000 .000

.453 .341 .289 .258 .137 .111

Note: C1 = mentoring group; C2 = comparison group; T1 = response at Time 1; T2 = response at Time 2; T3 = response at Time 3; T4 = response at Time 4; dfN = numerator degrees of freedom (df); dfD = denominator df.

Table 6 Results of fitting two MMRM analysesa for each of the five outcome variables, with and without a time-varying covariate. Effects

Language achievement Conditions (C) Time C  Time Study time (ST) ST  Time Mathematics achievement Conditions (C) Time C  Time Study time (ST) ST  Time SRL strategies Conditions (C) Time C  Time Study time (ST) ST  Time Self-efficacy for SRL Conditions (C) Time C  Time Study time (ST) ST  Time Perceived usefulness of SRL Conditions (C) Time C  Time Study time (ST) ST  Time

With a longitudinal covariateb

No longitudinal covariates dfN

dfD

F value

Pr > F

dfN

dfD

1 2 2

90.4 90.6 90.6

0.66 9.32 1.48

1 2 2

91.5 84.1 84.1

1 3 3

F value

Pr > F

0.4180 0.0002 0.2315

1 1 1 1 1

88.3 88.8 89.2 137.0 82.7

0.69 5.79 2.28 0.92 0.70

0.4073 0.0182 0.1345 0.3396 0.4051

0.38 22.15 4.90

0.5413 <.00001 0.0097

1 1 1 1 1

91.0 90.6 90.1 127.0 88.8

0.59 7.10 4.68 2.87 0.02

0.4452 0.0091 0.0332 0.0930 0.8960

89.0 85.3 85.3

0.23 28.40 11.99

0.6317 <.00001 <.00001

1 1 1 1 1

91.0 85.4 90.9 104.0 81.0

0.20 4.92 19.17 2.62 0.33

0.6527 0.0291 <.0001 0.1083 0.5647

1 3 3

92.0 81.7 81.7

2.13 13.84 4.97

0.1476 <.00001 0.00032

1 1 1 1 1

89.0 81.4 85.4 113.0 72.9

2.35 12.50 14.51 3.21 2.24

0.1292 0.0007 <.0001 0.0757 0.1392

1 3 3

89.6 79.0 79.0

7.98 9.59 6.97

0.0058 <.00001 0.00003

1 1 1 1 1

91.5 90.9 090.8 113.0 87.8

8.79 1.41 18.74 0.19 0.40

0.0039 0.2382 <.0001 0.6608 0.5309

Note: See the note in Tables 4 and 5. a Based on a unstructured covariance matrix and putting time in the model as a repeated factor within subject. b Results based on first and last measurement.

classic work to this study’s interaction contrasts, a ‘‘small’’ association is defined as g2p = .010 (equivalent to Cohen’s d = .20), a ‘‘medium’’ association is g2p = .059 (equivalent to Cohen’s d = .50), and a ‘‘large’’ association is g2p = .138 (equivalent to Cohen’s d = .80). Although Cohen did not explicitly consider repeated measures designs, the same guidelines are also appropriate. 3.3. Univariate MMRM analyses for each dependent variable Follow-up univariate MMRM analyses were performed to determine which of the five dependent variables were responsible for the significant omnibus group-by-time interaction. Table 6 displays the results of the hypothesis tests for each dependent variable. Although we present the results for all fixed effects, we discuss only the interaction. Table 6 shows that the null hypothesis of parallel profiles for treatment conditions was rejected at the .01 level for all outcomes except for Portuguese language achievement.

In other words, changes to the other response variables (i.e., mathematics achievement, SRL strategies, self-efficacy for and the usefulness of SRL) are not the same across the treatment groups over time. We focus our interpretation on the significant interaction. As previously indicated, a useful method for assessing the interaction effect in a manner consistent with the research objectives is to perform a series of tetrad contrasts. Hochberg’s step-up procedure was adopted to assess statistical significance, and Table 7 shows the significant tetrad contrasts. In general, these results indicate that the effect of the intervention on mathematics achievement was only observed at T4. The intervention affected SRL strategies and self-efficacy for SRL at T3, and changes in the perceived usefulness of SRL strategies were observed at T2. Table 7 also shows the proportion of population variance accounted for the series of tetrad contrasts. The g2p values for the significant tetrad contrasts (after controlling the FWE at the .05 level) ranged from

18

J.C. Núñez et al. / Contemporary Educational Psychology 38 (2013) 11–21

Table 7 Hochberg’s adjusted p values based on considering all possible tetrad contrasts for each dependent variables. Group

Time

F-value

Pr > F

Adj p

Partial g2 sample (lower–upper)

dfN

dfD

Mathematics achievement C1 vs. C2 T3 T4 C1 vs. C2 T2 T4 C1 vs. C2 T2 T3

1 1 1

74.15 91.72 91.59

8.06 4.93 .13

.0059 .0287 .7184

.0176 .0573 .7184

.047 .026 .000

.012 .000 .000

.264 .138 .000

Language achievement T3 T4 C1 vs. C2 C1 vs. C2 T2 T4 C1 vs. C2 T2 T3

1 1 1

91.00 91.90 92.00

2.04 1.64 .13

.1557 .2038 .7178

.1557 .2038 .7178

.013 .008 .000

.000 .000 .000

.101 .081 .000

SRL strategies C1 vs. C2 C1 vs. C2 C1 vs. C2 C1 vs. C2 C1 vs. C2 C1 vs. C2

T1 T2 T3 T1 T2 T1

T4 T4 T4 T3 T3 T2

1 1 1 1 1 1

91.97 83.88 89.55 91.17 76.68 90.53

20.43 8.53 8.41 3.13 1.49 .96

<.0001 .0045 .0047 .0807 .2251 .3286

<.0001 .0224 .0188 .2423 .3286 .3286

.100 .045 .043 .022 .008 .000

.058 .010 .008 .000 .000 .000

.293 .197 .185 .149 .095 .000

Self-efficacy for SRL C1 vs. C2 T1 T2 C1 vs. C2 C1 vs. C2 T1 C1 vs. C2 T2 C1 vs. C2 T3 C1 vs. C2 T1

T4 T4 T3 T3 T4 T2

1 1 1 1 1 1

87.94 91.83 83.72 80.08 88.34 81.80

13.62 9.99 7.34 4.20 3.50 .39

.0004 .0021 .0082 .0435 .0644 .5340

.0023 .0106 .0327 .1288 .1288 .5340

.068 .051 .038 .022 .018 .000

.029 .013 .005 .000 .000 .000

.244 .198 .182 .143 .114 .000

Perceived usefulness of C1 vs. C2 T1 C1 vs. C2 T2 C1 vs. C2 T1 C1 vs. C2 T2 C1 vs. C2 T1 C1 vs. C2 T3

SRL T4 T4 T3 T3 T2 T4

1 1 1 1 1 1

92.00 91.50 85.48 84.88 76.54 66.47

19.89 9.86 7.51 2.99 1.51 1.17

<.0001 .0023 .0076 .0868 .2219 .2832

.0001 .0112 .0302 .2604 .2832 .2832

.097 .051 .039 .016 .008 .006

.057 .014 .006 .000 .000 .000

.288 .199 .181 .117 .095 .093

Note: See the note in Table 5.

.038 to .100. Although not shown in the table, the Observation Time main effect accounted for considerably more variance (g2p values ranged from .059 to .201) than the Treatment Condition  Observation Time interaction. Even so, in this paper, the interaction is the effect of most theoretical interest. To help visualize the interaction, we plotted the mean scores of the mentoring and comparison groups by time. Fig. 1 reveals that the growth profiles in the treatment conditions are not parallel to each other; thus, the two treatment conditions have different patterns of change over time. Specifically, the change is approximately linear and positive over time for both groups; however, the magnitude of the slope for the experimental group is greater than that of the control group. Therefore, the intervention positively affects the dependent variables over time compared with the control group. 3.4. Ancillary analyses Because achievement outcomes are generally correlated with time spent studying, and participant study time might influence student behavior in the treatment conditions, re-examining and clarifying the earlier effectiveness of the mentoring program (i.e., including time spent studying as a longitudinal covariate in the statistical analyses) becomes necessary. Importantly, whereas the previous analyses considered the outcomes across four measurements, the analyses of this section only include the first and last measurements because study time was not available at T2 and T3. To correctly interpret the results, we conducted a separate MMRM analysis that incorporated study time as a longitudinal covariate. In addition, this model also incorporated the interaction between time spent studying and time. In some cases, positing whether the relationships between the predictors and the outcomes increase or decrease over time can be of substantive interest. This task is clearly plausible in the present investigation

because the effectiveness of the program is not hypothesized to be immediate; rather, it is developed over time. In other words, although the covariate levels are initially minimally related or unrelated to the outcome variables, results indicate that group differences emerge over time. Thus, examining the degree to which the study time affects each outcome over time is of interest. Table 6 (right panel) indicates that study time did not produce the hypothesized effects. Specifically, this covariate was not effective as either a principal effect (p-values ranged from .076 to .661) or a secondary effect (p-values ranged from .139 to .896). In other words, more study time was not associated with greater improvements to the outcomes in either treatment group over time. Using a multilevel linear model that ignored data clustering at the teacher level (i.e., a linear mixed-effects regression model with a random intercept and slope), we found a similar pattern of results. These results are neither presented nor discussed; however, they are available from the authors upon request. 4. Discussion Despite obtaining evidence in favor of the implementation of these programs, diverse reviews and meta-analyses focused on mentoring indicate that the effect size of these interventions depends on the context in which they are deployed, their design, assessment methodology, and the measured outcomes. Furthermore, the effect size is usually small (see DuBois et al., 2002; Eby et al., 2008; Rhodes, 2008). Following the recommendations of previous authors, we used an experimental design to assess the efficacy of an academic mentoring program implemented during the school year in two seventh-grade classes. The purpose of this program was to increase student use of SRL strategies, self-efficacy, perceived usefulness of SRL strategies, and academic achievement. The efficacy of this program was assessed at 3, 6, and 9 months after its initiation.

J.C. Núñez et al. / Contemporary Educational Psychology 38 (2013) 11–21

SRL Strategies

Self-Efficacy for SRL

Perceived Usefulness of SRL

Math Achievement

19

Fig. 1. Means of each significant outcome across time by treatment condition. The significant interactions denote that the growth profiles between the treatment conditions are not parallel; consequently, the two treatment conditions display different patterns of change over time.

4.1. Was the school-based mentoring program effective? Our findings suggest that the mentoring program was effective. Students who participated in the academic mentoring program increased their SRL competences to meet school demands better than students from the comparison group. These data are in line with the findings of other investigators (e.g., Allen, Eby, Poteet, Lentz, & Lima, 2004; DuBois & Karcher, 2005; DuBois et al., 2002; Eby et al., 2008; Hansen, 2007; Johnson, 2007; Rhodes & DuBois, 2008). Although the interventions in the majority of previous studies (e.g., Eby et al., 2008; Herrera, Grossman, Kauh, Feldman, & McMaken, 2007) have had relatively small (or very small) effect size, the effect size of academic mentoring in our study were small, small-medium, or medium depending on the duration of this mentoring program, the variable used to assess the efficacy of the program, and the level of analysis considered, respectively. When considering the five dependent variables simultaneously, our findings show that differences between the mentoring and comparison groups were small after 3 months of implementation (Table 5, T1–T2, g2p = .031); the effect size was medium after 6 months (Table 5, T1–T3, g2p = .089); and the effect size was large after 9 months (Table 5, T1–T4, g2p = .203). Therefore, the effectiveness of academic mentoring depends on the measurement time after the program is implemented. These results have important implications not only for future meta-analyses but also with regard to educator and school administrator decisions and policies concerning the design and implementation of effective classroom interventions. Moreover our findings are aligned with other studies (see Eby et al., 2008) and indicate that the effect size of academic mentoring

differs as a function of the measured dependent variable. Thus, although the effect of the intervention was positive for all five dependent variables, Table 6 shows that, compared with mathematics and Portuguese achievement, the effect sizes are higher for the three variables that are directly related to the content of the intervention program (i.e., the use of SRL strategies, self-efficacy for SRL strategies, and the perceived usefulness of SRL strategies). This result is most likely due to the final academic results being considered for factors other than the efficient use of SRL strategies (e.g., classroom behavior, student attention, and student attitudes towards school and learning); with implications of higher level of transfer. Therefore, Table 7 displays that the effect sizes of academic mentoring in Portuguese and mathematics achievement were small, and significant group differences were obtained after elapse of 9 months (mathematics, T2–T4, g2p = .026; Portuguese, T3–T4, g2p = .013). However, the effect was small for the three remaining variables after 6 months (use of SRL strategies, T2–T4, g2p = .022; self-efficacy regarding the use of SRL strategies, T2–T4, g2p = .038; and the perceived usefulness of SRL strategies, T2–T4, g2p = .039) and medium after 9 months (use of SRL strategies, T2–T4, g2p = .100; self-efficacy regarding the use of SRL strategies, T2–T4, g2p = .068; and the perceived usefulness of SRL strategies, T2–T4, g2p = .097). Eby et al. (2008) also obtained a small effect size for academic mentoring on achievement (rc = .19), with a medium effect size concerning school attitudes (rc = .36). Finally, conclusions regarding the effectiveness of programs similar to the one implemented here may differ by the level of conducted analysis. Our academic mentoring program had a larger effect size when its data were analyzed from a multivariate perspective and the differences between the beginning and the

20

J.C. Núñez et al. / Contemporary Educational Psychology 38 (2013) 11–21

end of the program were considered (Table 5, T1–T4). Otherwise, a medium or small effect was computed when we adopted a univariate analysis to consider only two pre- and post test measurements. Table 7 displays a medium effect size for the use of SRL strategies (T1–T4) and a small effect size for mathematics achievement (T1–T4). In addition, we were unable to find an effect when all four measurements/observations were taken into account. Table 6 shows the results of the treatment conditions with regard to mathematics and Portuguese achievements. Therefore, to compare results across studies, we must provide all the details to identify diverse conditions (e.g., individual and group level analyses) affecting the study results. 4.2. Limitations, educational implications and future research Although numerous strengths mark this study including its randomized longitudinal design and the quality of its mentor training, it should also be considered in the context of its limitations. First, its findings are based on a middle-school mentoring program located in an urban school district in the north of Portugal. These findings must be replicated to determine their generalizability to other urban and rural areas of Portugal as well as other countries. Second, the current study sampled fourteen teachers. Future analyses based on data collected from an optimal number of teacherand student-level variables will illuminate more of the true benefits of a mentoring program. The current study addressed an important topic of effectiveness of a mentoring program and future investigations should diversify and increase the sample size of students and mentors to confirm these findings. Third, although the results suggest that this intervention was effective, some limitations concerning its design should be considered. Specifically, stating the effects of the program with certainty is impossible because this intervention contained both a microanalytic methodology and a story-tool without control groups for both conditions. Consequently, we cannot say whether the positive effects were due to the microanalytic methodology, the characteristics of the story-tool or both. Moreover, this research did not use a ‘‘placebo’’ treatment to rule out the ‘‘Hawthorn’’ effect. In fact, students in the mentoring condition might have performed better on all measures simply because they worked harder given the additional teacher attention they received throughout the year. Despite this design limitation, the absence of change in the dependent variables at T1 and T2 suggests that this study did not succumb to this effect. Even with these limitations, our results have clear implications for designing educational investigations to capture the complexities of mentoring programs and determine why these programs are effective under some conditions but not others. In accordance with the results of prior investigations, our findings suggest that the academic mentoring, described in the method section, effectively improves knowledge of SRL strategies as well as develops self-efficacy and the perceived usefulness of SRL strategies. Therefore, academic mentoring is a positive educational tool from an experimental perspective. Although our data do not allow us to make causal inferences, we speculate that the increase of declarative SRL strategy knowledge and its increased use throughout the mentoring program enhances student self-efficacy and the perceived usefulness of SRL strategies, thereby increasing academic achievement. According to the worrisome results regarding the grade retention of the Organization for Economic Co-operation and Development (OECD, Eurydice, 2011), many middle school teachers in Spain and Portugal frequently report that many of their students do not master the competences needed to apply SRL. For example, students have difficulty paying attention, avoiding distractions and focusing on the task; they cannot organize and interpret written

information; they do not display much skill when writing outlines; and they lack volition. Nevertheless, teachers and students consider that these competences are essential for successful learning. Therefore, middle school educators and administrators should increase the number of programs that improve these types of skills and strategies (Cleary & Zimmerman, 2004; Dignath et al., 2008; Simpson, Hynd, Nist, & Burrel, 1997). If these programs take the form of academic mentoring, then our work can contribute to many guidelines. Boekaerts and Corno (2005) suggest for preventive interventions to be urgently developed in class to increase students’ arrays of learning strategies and metacognitive skills in order to enable them to interpret school demands from a self-regulatory framework. The SRL contents discussed throughout our program (e.g., Testas’s story-tool) met students’ expectations and immediate academic challenges (e.g., time management, procrastination, note taking, academic distracters, and goal setting). This sense of usefulness might have increased students’ agent role in the learning process as well. Finally, this study’s data interpretation and generalization must be considered cautiously because its conclusions might differ if (a) the contents of the academic mentoring had not addressed SRL strategies, (b) the students were not seventh graders, (c) the dependent variables had not been measured via self-reports (e.g., event measures; Zimmerman, 2008), and (d) follow-up assessments had been employed. Therefore, although these data were consistent, the above limitations invite additional researchers to investigate the effects and efficacy of academic mentoring program implementation. Acknowledgments Grants awarded to the authors from the Spanish Ministry of Science and Innovation (Refs.: PSI2008-03624 and PSI2011-23395) supported this work. We gratefully acknowledge the insightful suggestions provided by the editor and by three anonymous reviewers on the previous version of this manuscript. References Allen, T. D., Eby, L. T., Poteet, M. L., Lentz, E., & Lima, L. (2004). Outcomes associated with mentoring protégés: A meta-analysis. Journal of Applied Psychology, 89, 127–136. http://dx.doi.org/10.1037/0021-9010.89.1.127. Baker, S. K., Chard, D. J., Ketterlin-Geller, L. R., Apichatabutra, C., & Doabler, C. (2009). Teaching writing to at-risk students: The quality of evidence for selfregulated strategy development. Exceptional Children, 75(3), 303–318. Boekaerts, M., & Corno, L. (2005). Self-regulation in the classroom: A perspective on assessment and intervention. Applied Psychology: An International Review, 54(2), 199–231. http://dx.doi.org/10.1111/j.1464-0597.2005.00205.x. Cleary, T., & Zimmerman, B. J. (2001). Self-regulation differences during athletic practice by experts, non-experts, and novices. Journal of Applied Sport Psychology, 13, 185–206. Cleary, T. J., & Zimmerman, B. J. (2004). Self-regulation empowerment program: A school-based program to enhance self-regulated and self-motivated cycles of student learning. Psychology in the Schools, 41, 537–550. http://dx.doi.org/ 10.1002/pits.10177. Cohen, J. (1988). Statistical power for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum. Dignath, C., Buettner, G., & Langfeldt, H. (2008). How can primary school students learn SRL strategies most effectively? A meta-analysis on self-regulation training programmes. Educational Research Review, 3, 101–129. http:// dx.doi.org/10.1016/j.edurev.2008.02.003. Dorsey, L. E., & Baker, C. M. (2004). Mentoring undergraduated nursing students. Nurse Educator, 29, 260–265. DuBois, D. L., Holloway, B. E., Valentine, J. C., & Cooper, H. (2002). Effectiveness of mentoring programs for youth: A meta-analytic review. American Journal of Community Psychology, 30, 157–197. http://dx.doi.org/10.1023/ A:1014628810714. DuBois, D. L., & Karcher, M. A. (2005). Handbook of youth mentoring. Thousand Oaks, CA: Sage. Eby, L. T., Allen, T. D., Evans, S. C., Ng, T., & DuBois, D. L. (2008). Does mentoring matter? A multidisciplinary meta-analysis comparing mentored and nonmentored individuals. Journal of Vocational Behavior, 72, 254–267. http:// dx.doi.org/10.1016/j.jvb.2007.04.005.

J.C. Núñez et al. / Contemporary Educational Psychology 38 (2013) 11–21 Edmondson, J. H., & White, J. (1998). A tutorial and counseling program: Helping students at risk of dropping out of school. Professional School Counseling, 1(4), 43–47. Eurydice, 2011. Grade retention during compulsory education in Europe: Regulations and statistics. Brussels: Eurydice. Gurka, M. J. (2006). Selecting the best linear mixed model under REML. The American Statistician, 60, 19–26. http://dx.doi.org/10.1198/000313006X90396. Guthrie, J. T., McRae, A. C., & Klauda, S. L. (2007). Contributions of Concept-Oriented Reading Instruction to knowledge about interventions for motivations in reading. Educational Psychologist, 42, 237–250. Hansen, K. (2007). One-to-one mentoring: Literature review. Philadelphia: Big Brothers Big Sisters of America. Herrera, C., Grossman, J. B., Kauh, T. J., Feldman, A. F., & McMaken, J. (2007). Making a difference in schools: The Big Brothers Big Sisters school-based mentoring impact study. Philadelphia, PA: Public/Private Ventures. Hochberg, Y. (1988). A sharper Bonferroni procedure for multiple tests of significance. Biometrika, 75, 800–802. http://dx.doi.org/10.1093/biomet/75.4.800. Jacobi, M. (1991). Mentoring and undergraduate academic success. A review of the literature. Review of Educational Research, 61, 505–532. Johnson, W. B. (2007). Student-faculty mentorship outcomes. In T. D. Allen, & L. T. Eby (Eds.), Blackwell handbook of mentoring (pp. 189–210). Oxford: Blackwell. http://dx.doi.org/10.1111/b.9781405133739.2007.00012.x. Karcher, M. (2005). The effects of developmental mentoring and high school mentors’ attendance on their younger mentees’ self-esteem, social skills, and connectedness. Psychology in the Schools, 42, 65–77. http://dx.doi.org/10.1002/ pits.20025. Kitsantas, A., & Zimmerman, B. J. (2002). Comparing self-regulatory processes among novice, non-expert, and expert volleyball players: A microanalytic study. Journal Of Applied Sport Psychology, 14, 91–105. Liem, A. D., Lau, S., & Nie, Y. (2008). The role of self-efficacy, task value, and achievement goals in predicting learning strategies, task disengagement, peer relationship, and achievement outcome. Contemporary Educational Psychology, 33, 486–512. http://dx.doi.org/10.1016/j.cedpsych.2007.08.001. Núñez, J. C., Cerezo, R., González-Pienda, J. A., Rosário, P., Valle, A., Fernández, E., et al. (2011). Implementation of training programs in self-regulated learning strategies in Moodle format: Results of a experience in higher education. Psicothema, 23, 274–281. OECD (2010). PISA 2009 results: What students know and can do – Student performance in reading, mathematics and science (Vol. I). http://dx.doi.org/ 10.1787/9789264091450-en. Plant, E. A., Ericsson, K. A., Hill, L., & Asberg, K. (2005). Why study time does not predict grade point average across college students: Implications of deliberate practice for academic performance. Contemporary Educational Psychology, 30, 96–116. http://dx.doi.org/10.1016/j.cedpsych.2004.06.001. Rhodes, J. (2008). Improving youth mentoring interventions through researchbased practice. American Journal of Community Psychology, 41, 35–42. http:// dx.doi.org/10.1007/s10464-007-9153-9. Rhodes, J. E., & DuBois, D. L. (2008). Mentoring relationships and programs for youth. Current Directions in Psychological Science, 17, 254–258. http://dx.doi.org/ 10.1111/j.1467-8721.2008.00585.x. Rosário, P. (2002). 007.° Ordem para estudar. Porto: Porto Editora. Rosário, P. (2004). Estudar o Estudar: As (Des)venturas do Testas [Studying study: Testas’s (mis)adventures]. Porto: Porto Editora. Rosário, P., Lourenço, A., Paiva, M.O., Núñez, J.C., González-Pienda, J.A., & Valle (2012). Self-efficacy and perceived utility as necessary conditions for selfregulated academic learning. Anales de Psicología, 28, 37–44. Rosário, P., González-Pienda, J. A., Cerezo, R., Pinto, R., Ferreira, P., Lourenço, A., et al. (2010). Efficacy of the program ‘‘Testas’s (mis)adventures’’ to promote the deep approach to learning. Psicothema, 22(4), 828–834. Rosário, P., Mourão, R., Núñez, J. C., González-Pienda, J., & Solano, P. (2008). Storytelling as a promotor of self-regulated learning (SRL) throughout

21

schooling. In A. Valle, J. C. Núñez, R. G. Cabanach, J. González-Pienda, & S. Rodríguez (Eds.), Handbook of instructional resources and their applications in the classroom (pp. 107–122). New York: Nova Science. Rosário, P., Núñez, J. C., González-Pienda, J., Valle, A., Trigo, L., & Guimarães, C. (2010). Enhancing self-regulation and approaches to learning in first-year college students: A narrative-based programme assessed in the Iberian Peninsula. European Journal of Psychology of Education, 25, 411–428. http:// dx.doi.org/10.1007/s10212-010-0020-y. Sambunjak, D., Straus, S. E., & Marusic, A. (2006). Mentoring in academic medicine. Journal of the American Medical Association, 296, 1103–1115. Schwartz, S., Rhodes, J., Chan, C., & Herrera, C. (2011). The impact of school-based mentoring on youths with different relational profiles. Developmental Psychology, 47(2), 450–462. http://dx.doi.org/10.1037/a0021379. Simpson, M., Hynd, C., Nist, C., & Burrel, K. (1997). College academic assistance programs and practices. Educational Psychology Review, 9, 39–87. http:// dx.doi.org/10.1023/A:1024733706115. Spencer, R. (2007a). Naturally occurring mentoring relationships involving youth. In: T. D. Allen, & L. T. Eby (Eds.), Handbook of mentoring: A multiple perspectives approach (pp. 99–117). Malden, MA: Blackwell Publishing. http://dx.doi.org/ 10.1002/9780470691960.ch9. Spencer, R. (2007b). ‘‘It’s not what I expected’’ A quantitative study of youth mentoring relationship failures. Journal of Adolescent Research, 22, 331–354. http://dx.doi.org/10.1177/0743558407301915. Thompson, L., & Kelly-Vance, L. (2001). The impact of mentoring on the academic achievement of at-risk youth. Children and Youth Services Review, 23(3), 227–242. http://dx.doi.org/10.1016/S0190-7409(01)00134-7. Vallejo, G., & Ato, M. (2006). Modified Brown–Forsythe procedure for testing interaction effects in split-plot designs. Multivariate Behavioral Research, 41, 549–578. http://dx.doi.org/10.1207/s15327906mbr4104_6. Vallejo, G., & Ato, M. (2012). Robust tests for multivariate factorial designs under heteroscedasticity. Behavior Research Methods, 44, 471–489. Vallejo, G., Fernández, M. P., Livacic-Rojas, P., & Tuero-Herrero, E. (2011a). Comparison of modern methods for analyzing unbalanced repeated measures data with missing values. Multivariate Behavioral Research, 46, 900–937. http:// dx.doi.org/10.1080/00273171.2011.625320. Vallejo, G., Fernández, M. P., Livacic-Rojas, P. E., & Tuero-Herrero, E. (2011b). Selecting the best unbalanced repeated measures model. Behavior Research Methods, 43, 18–36. http://dx.doi.org/10.3758/s13428-010-0040-1. Wood, R. E., Bandura, A., & Bailey, T. (1990). Mechanisms governing organizational performance in complex decision-making environments. Organizational Behavior and Human Decision Processes, 46, 181–201. http://dx.doi.org/ 10.1016/0749-5978(90)90028-8. Zimmerman, B. J. (2002). Becoming a self-regulated learner: An overview. Theory into Practice, 41(2), 64–70. Zimmerman, B. J. (2008). Investigating self-regulation and motivation: Historical background, methodological developments, and future prospects. American Educational Research Journal, 45, 166–183. http://dx.doi.org/10.3102/ 0002831207312909. Zimmerman, B. J., Bandura, A., & Martinez-Pons, M. (1992). Self-motivation for academic attainment: The role of self-efficacy beliefs and personal goal setting. American Educational Research Journal, 29, 663–676. http://dx.doi.org/10.3102/ 00028312029003663. Zimmerman, B. J., & Martinez-Pons, M. (1986). Development of a structured interview for assessing student use of self-regulated learning strategies. American Educational Research Journal, 23(4), 614–628. http://dx.doi.org/ 10.3102/00028312023004614. Zimmerman, B. J., & Martinez-Pons, M. (1988). Construct validation of a strategy model of student self-regulated learning. Journal of Educational Psychology, 80(3), 284–290. http://dx.doi.org/10.1037//0022-0663.80.3.284.