- Email: [email protected]
Comparing opportunity to learn and student achievement gains in southern African primary schools: A new approach
International Journal of Educational Development 33 (2013) 426–435
Contents lists available at SciVerse ScienceDirect
International Journal of Educational Development journal homepage: www.elsevier.com/locate/ijedudev
Comparing opportunity to learn and student achievement gains in southern African primary schools: A new approach Cheryl Reeves a,*, Martin Carnoy b,*, Nii Addy c a
Cape Peninsula University of Technology, South Africa Stanford University, United States c McGill University, Canada b
A R T I C L E I N F O
A B S T R A C T
Keywords: Opportunity to learn Student mathematics achievement gains Classroom fixed effects Southern Africa
A popular explanation for low student achievement in many developing countries’ primary schools is that students have relatively little opportunity to learn (OTL) the skills needed for academic success. However logical this explanation may be, surprisingly little empirical evidence has been presented to support it. In this paper we address this gap by estimating the effect of OTL on students’ academic performance using rich data we gathered on the teaching process in a large number of South African and Botswana Grade 6 classrooms. We use an innovative classroom fixed effects approach to estimate the impact of OTL on students’ mathematics achievement gains. We found statistically significant but very different results for our South Africa and Botswana samples. The discussion of those results in the context of differences in the two school systems gives us insights into the importance and limits of OTL as an explainer of student learning in low achievement schools. ß 2013 Elsevier Ltd. All rights reserved.
1. Introduction A popular explanation for low student achievement in many developing countries’ primary schools is that students have relatively little opportunity to learn the skills needed for academic success (Benavot and Amadio, 2004; Abadzi, 2009). However logical the explanation may be, surprisingly little empirical evidence has been presented to support it. We address this gap by estimating the effect of opportunity to learn (OTL) on students’ academic achievement using rich data we gathered on the teaching process in a large number of South African and Botswana classrooms. There is a long history of OTL as a key issue in student achievement. In its earlier incarnation in developed countries, OTL referred to the amount of time required for students with different abilities to learn the same curriculum (see, for example, Carroll, 1963; Berliner, 1990). In developing countries, OTL is defined more broadly as the amount of time that students are exposed to the curriculum in a given academic year. The educational effectiveness literature further breaks down OTL into ‘‘content coverage,’’ ‘‘content exposure’’ and ‘‘content emphasis.’’ The term ‘‘content
* Corresponding authors. E-mail addresses: [email protected] (C. Reeves), [email protected] (M. Carnoy). 0738-0593/$ – see front matter ß 2013 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.ijedudev.2012.12.006
coverage’’ is used to refer to the topics and sub-topics taught (Thompson and Senk, 2001; Porter and Smithson, 2001; Schmidt et al., 2001; Stigler and Hiebert, 2004). ‘‘Content exposure’’ refers to the amount of time devoted to a given subject overall (Rosenshine and Berliner, 1978; Floden, 2003; Lee, 1982; Wang, 1998; Porter and Smithson, 2001; Schmidt et al., 2001). ‘‘Content emphasis’’ refers to the relative amount of time students spend on the various topics (Wang, 1998). Most empirical research on OTL in developing countries has focused on measuring content coverage and content exposure (Gillies and Quijada, 2008; DeStefano et al., 2006; Moore et al., 2006). For example, the scheduled instructional time in developing country primary schools is often low compared to the minimum 850 h annually that experts claim is needed for a ‘‘quality education’’ (Benavot and Amadio, 2004). Schools in many developing countries are characterized by considerable teacher absenteeism (Chaudhury et al., 2005; Beteille, 2009) and student absenteeism may in part depend on teacher absenteeism (Marshall, 2003). Beyond absences, teachers are often present at school but not in their classrooms teaching, or are in their classrooms ‘‘teaching,’’ but are not teaching the required curriculum or are often not engaging the students in a meaningful way, especially on parts of the curriculum they feel uncomfortable teaching (Stallings, 1980). This growing literature has identified the generally poor and highly unpredictable delivery of the official curriculum to learners
C. Reeves et al. / International Journal of Educational Development 33 (2013) 426–435
as a potentially important problem, particularly in low and middleincome societies. Yet, analysts have done little to link variation in OTL to variation in student learning outcomes, providing limited evidence that increasing any of the usual measures of OTL improves student performance. There are a few exceptions. International cross section data from the Third International Mathematics and Science Survey (TIMSS) suggested that mathematics curriculum exposure has a positive effect on 8th grade student mathematics scores (Schmidt et al., 2001). Marshall in Guatemala (2003) and Reeves in the Western Cape Province of South Africa (2005), use student pre- and post-tests and classroom observations to estimate that teacher time use in primary school classrooms (Marshall, 2003) and the coverage of the required curriculum over a number of school years (Reeves, 2005) are associated with greater student learning gains. In this paper, we attempt to provide further empirical evidence for whether OTL contributes to student learning. The study is innovative in two ways. It compares the impact of OTL on the academic performance of very linguistically and socio-culturally similar students in the border regions of two educational systems—South Africa (North West Province) and Botswana—with different development histories but almost identical national curricula. Students in both systems are relatively low scoring, and low OTL has been identified as a problem, particularly in South Africa (Fleisch, 1999; Taylor et al., 2003). This allows us to test whether variation in OTL may have different impacts on student learning in different educational contexts. We use a classroom fixed effects approach to estimate the impact of OTL on students’ mathematics achievement gains in Grade 6 classrooms in a large sample of schools on either side of the border. We were able to get very detailed data on curriculum coverage (content coverage, content exposure, and content emphasis) from student notebooks, since in both countries teachers require students to record activities from all lessons in their individual notebooks. Using this detailed information on what took place in classrooms during the academic year, we were able to relate frequency of specific curriculum coverage in each classroom (content emphasis) to student achievement gains item-by-item on a 40-item mathematics test in that same classroom. The focus of our study in linking OTL to student mathematics performance is on the content (topics and sub-topics) made available in classrooms during the academic school year. In particular, we wanted to establish the extent to which Grade 6 students had the time and opportunity to learn the mathematics topics and sub-topics associated with the content in the items in the test we administered to students in our sample classrooms; that is, the ‘‘content emphasis’’ element of opportunity to learn. We found statistically significant but very different results for our South Africa and Botswana classroom samples. The discussion of those results in the context of differences in the two school systems provides insights into the importance and limits of OTL as an explainer of student learning in low achievement schools. 2. The context The school systems in Botswana and North West Province are imbedded in the same Setswana culture and language and both use English as the main teaching language in upper primary grades. However, Botswana gained independence from Britain in 1966 and developed its education in the context of national goals that were
427
designed to serve the entire population of Botswana equally. South African education was dominated by apartheid policies that separated Africans’ schooling from white and so-called ‘‘coloured’’ (mixed race and Asians) schools. North West Province was divided into an area that was part of ‘‘South Africa’’ and an African homeland, Bophutuswana, with its separate school system, but largely financed and controlled by the South African apartheid government. Ultimately, African schools in South Africa also became a primary site of struggle against apartheid, with students and teachers (through the teachers’ organization, SADTU) in the vanguard of the struggle (Hyslop, 1988). The meaning of education therefore differed in the two countries over the three decades before 1994, when apartheid ended and the Mandela government began to try to undo the apartheid legacy. Not surprisingly, however, fifteen years later, this legacy continues to shape the context of South African schools, through continued inequalities in resources, including the quality of teachers’ training, and through many teachers’ perception that their main role is as members of a political vanguard (through their teacher union organization) more than professionals responsible for student learning. These factors are much less of an issue in Botswana’s schools; hence, they have less of an influence on student’s opportunity to learn (Fuller et al., 1994). The other side of this coin is that students in Botswana score consistently higher on international tests of mathematics than students in South Africa (Reddy, 2006; van der Berg and Louw, 2008; SACMEQ, 2010). Despite relatively higher student achievement in Botswana, results of international comparative studies show that most students in the country are still learning at quite low levels. For example, in the 2003 and 2007 Trends in International Mathematics and Science Study (TIMSS), Botswana students were among the lowest scoring students worldwide. Even by African standards, Grade 6 students in Botswana and South Africa scored significantly lower on the SACMEQ tests in 2000 and 2007 than students in East Africa, despite much higher incomes per capita in Southern Africa (van der Berg and Louw, 2008; SACMEQ, 2010). 3. Data The data for the present study were collected as part of a larger study in a random sample of 116 public schools drawn from the population of no-fee public schools in both urban and rural areas located in the border region of the North West Province in South Africa and the Southeast region of Botswana (Carnoy et al., 2012). The study measured student learning gains, student family background variables, teacher background variables and mathematics knowledge, and data at the school level for 116 teachers teaching mathematics to more than 5000 Grade 6 students in 126 classrooms. As part of the study we also analyzed student notebooks twice during the school year and videotaped at least one and often two mathematics lessons for each teacher in the sample. We assessed students’ learning gains over the course of the academic year in 2009, through the use of a pre- and post-test based on the Grade 4–6 curriculum used in South African and Botswana schools. The student pre-test was administered toward the beginning of the school year in late March 2009, and the posttest, as near as possible to the end of the school year, in late October and early November 2009. The test was based on what students were supposed to have learned before entering Grade 6, and on what they should have learned during that academic year. The test results are reported below. From an analysis of the students’ notebooks, we estimated how many mathematics lessons teachers teach, what teachers cover in
428
C. Reeves et al. / International Journal of Educational Development 33 (2013) 426–435
those lessons, the grade level teaching of content, and how this relates to the curriculum that is supposed to be covered (the curriculum is national in both Botswana and South Africa) as represented by the 40 items on the curriculum based test that we applied to our sample of students. 3.1. Opportunity to learn instrument To ensure comparability of data collected on ‘‘content coverage’’ and ‘‘content emphasis’’ across classes (Porter and Smithson, 2001; Rowan et al., 2002), an opportunity to learn instrument was designed listing 110 mathematics topics associated with the study’s test items, which in turn, were based on the mandated national curriculum. These topics were organized on the instrument under five content areas: number, operations and relationships; patterns, functions and algebra; space and shape (geometry); measurement; and data handling. The list of topics and sub-topics used allowed for the collection of detailed data on content covered in class. For example, rather than simply recording coverage of ‘‘rounding off,’’ researchers needed to capture whether students have opportunities to ‘‘round off to the nearest 10, 100, or 1000.’’ 3.2. Opportunity to learn data sources and data collection In each of the classes observed, we reviewed a sample of the three most comprehensive student notebooks twice during the academic year—in May/June and again very near the end of the school year (October/November) 2009. Students in South Africa and Botswana are expected to write in their notebooks each time that they have a mathematics lesson. We used the dates of daily written work in students’ notebooks as evidence of students having had a lesson. For ‘‘content exposure,’’ researchers counted the number of ‘‘pieces’’ of dated work in the notebooks to estimate the number of mathematics lessons that had taken place between the beginning of the school year and October/November 2009. We were thus able to record the number of daily lessons during almost the entire academic year (all but the month of November and early December, a period of review before exams, when there is relatively less teaching of full topics). Detailed data on ‘‘content emphasis’’ and ‘‘content coverage’’ were collected only on topics associated with the content of the comparative study’s student test items. Fieldworkers used the topic list in the opportunity-to-learn instrument to indicate whether there was evidence, from observation of students’ notebooks, that a particular topic had been covered in each classroom during the school year 2009. To gauge the relative ‘‘content emphasis’’ given to each topic, fieldworkers also used dates of daily lessons in the students’ notebooks to estimate the number of lesson periods spent on each of the topics covered. The instrument allowed them to record whether less than one lesson had been spent on a particular topic. 4. Method A typical value-added model to estimate the effect of a classroom input on student outcomes relates the post-test score to the input, controlling for students’ pre-test score. Economists argue that this model assumes that students’ knowledge ‘‘decays’’ over the course of a year at a constant rate (an estimate of the decay is 1 minus the pre-test coefficient), with part of the treatment coefficient attributed to replacing the decay in knowledge. The model presents a problem in that the student’s initial achievement may be correlated with the error term (Ladd, 2008). One way to handle this problem is to estimate a value-added equation that uses test score differences as the measure of value added. That
model assumes no learning decay and eliminates the pre-test correlation with the error term, but poses other issues, namely whether differences in test scores are a statistically reliable measure of student learning gains.1 Such value added models are still subject to selection bias, in that teachers likely to offer more OTL may be self-selecting into classrooms with students likely to have larger gains in mathematics performance. Our model attempts to ‘‘correct’’ for that bias by estimating the OTL treatment effect across items in the same classroom with the same teacher (classroom fixed effects). We estimated two fixed effects models. The first model relates the number of lessons related to each test item in each classroom (OTL) to the average classroom post-test score on that item, controlling for the pre-test score on the same item. The second model relates OTL to the increase or decrease in the percentage of students getting an item correct that the teacher’s class made on that test item between the pre- and the posttest.2 For both models, we used 38 of the test items (two items were badly printed on the Botswana version of the initial test, so we left them out of this analysis both for Botswana and for North West Province). We had complete OTL and average achievement gain observations for 61 of the 62 classrooms in the North West Province sample and all 64 classrooms in Botswana. This made a matrix of 2318 observations of ‘‘content emphasis’’ in Northwest Province and 2432 observations of ‘‘content emphasis’’ in Botswana; a matrix of 2318 or 2432 observations of classroom-item average post-test scores, a matrix of 2318 or 2432 observations of classroom-item average initial test scores, and a matrix of 2318 or 2432 observations of classroom-item average test score gains (post-test score minus pre-test score). Our regression models are as follows: A posti j ¼ h þ aA prei j þ b1 OLT i j þ b2 OLTi2j þ e
(Models 1 and 2)
A posti j A prei j ¼ h þ b1 OLT i j þ b2 OLTi2j þ e
(Models 3 and 4)
where Apostij, average student post-test score on item i in classroom j; Apreij, average student pre-test score on item i in classroom j; OTLij, number of lessons touching on test item i in classroom j; Apostij Apreij, average student gain in post-test minus pre-test on item i in classroom j; h, intercept term; and e, an error term. The disadvantage of the method is that we cannot control for the many other data we collected on students and teachers, including student SES and teacher mathematics knowledge. The content emphasis measure of OTL we use (number of lessons given during the year on each of out student test items) was the only one 1 Cronbach and Furby (1970) argued that differences between pre-test and posttest scores are not very useful in social science research. Nevertheless, there has been widespread use of gain scores in educational research with reasonable and reproducible policy inferences (see, for example, Ladd’s review of estimating teacher effects with gain scores (Ladd, 2008), as well as Hill, Rowan, and Ball’s well known study of teacher effects (2005). Williams and Zimmerman (1996) reviewed the Cronbach and Furby critique from a statistical point of view and concluded that the reliability of the difference in post-test and pre-test is a function of the correlation between the post-test and pre-test and the ratio, l, of the standard deviations of the pre-test and post-tests. The closer the l is to 1.0, the greater the effect of the correlation between post-test and pre-test on reliability. In our case, according to the Williams and Zimmerman criterion (1996, Fig. 1), the reliability of the differences in test scores in both countries is not very high. As we show below, however, our results using test score differences produce results for our treatment variable that are consistent with those estimating the post-test. 2 There is considerable error associated with getting an item correct or incorrect from random marking on a multiple-choice test. However the error term is smaller for the average score for all students in the class. We took the average class pre-and post-test score on each item as the single best estimate of the proportion correct on that item in each class.
C. Reeves et al. / International Journal of Educational Development 33 (2013) 426–435
Fig. 1. North West Province and Botswana: average individual learner initial mathematics test score, by test item, late March, 2009. Source: North West Province and Botswana Schools Sample, 2009.
Fig. 2. (a) North West Province: average of individual learner pre-test and post-test mathematics score, by test item, late March and late October, 2009. Source: North West Province Schools Sample, 2009.
429
430
C. Reeves et al. / International Journal of Educational Development 33 (2013) 426–435
we could relate to particular test items. The method’s great advantage is that we can control for classroom fixed effects—we can estimate OTL effects across items within each classroom for a given teacher. 5. Results 5.1. Student mathematics achievement gains The average mathematics achievement in our sample of more than 3800 students in North West Province who took the initial test was 28.6%, with a standard deviation of 12.2. The average test score in our sample of 1774 Botswana students who took the initial test was 33.6% (including items 27 and 37, which had printing issues in the Botswana version of the test that made them difficult to read) with a standard deviation of 12.0.3 Thus, the difference in test score was a minimum of 5 points (including question 27 and 37 for both countries), a highly statistically significant difference. There was also great variation in the score from test item to test item in both countries. Fig. 1 shows this variation. The gains that students made between March and end October also varied greatly, with the gains on some items coming out reasonably large and positive, most gains small and positive, and two or three, negative (see Fig. 2a and b). In the North West sample, the average score on the post-test (round 2) for the sample as a whole was 31.6%, with a standard deviation of 12.4. The average gain for all of the 3485 students we could match in North West Province was therefore 3 points, with a standard deviation of 11.2 points. In Botswana, the gain was larger than in North West. The average score on the post-test for the sample as a whole was 38.6%, with a standard deviation of 14. Again, the difference between the average score in Botswana and North West was highly statistically significant. The average gain for all of the 1666 students we could match was 4.0 points (excluding gains on question 27 and 37), with a standard deviation of 10.2 points. The one point difference in gain between students in the two samples is also statistically significant at the 1% significance level. From pre- to post-test there were signs of improvement on a number of items. For example, in North West, learners scored between 40 and 49% on 6 items in the pre-test and on 8 items in the post-test and above 50% on 2 items in the pre-test and 5 items on the post-test. In Botswana, learners scored between 40 and 49% on 8 items in the pre-test and on 10 items in the post-test and above 50% on 8 items in the pre-test and 10 items on the post-test. North West Province learners showed overall negative gains in 10 of the 40 items on the learner test, with the biggest negative gain of about 18% on item 19. Botswana learners showed overall negative gains in 7 of the 40 items on the learner test, with the biggest negative gain of more than 20% also on item 19. That common fall in test score on item 19 resulted from a simple change in the problem, from a plus sign to a minus sign in an algebraic expression, suggesting that learners did not really understand the underlying mathematics of the expression in the pre-test. 5.2. Opportunity to learn across classrooms Overall, OTL in the classrooms we surveyed was low on all three measures used. Fig. 3 shows the number of mathematics lessons taught in the North West Province and Botswana classrooms in our samples (‘‘content exposure’’). Researchers counted the total number of days (by dates) students wrote work in their 3 If we exclude items 27 and 37, the average score on the initial test for Batswana learners was 34.6%, with a standard deviation of 12.4.
Table 1 North West Province and Botswana: estimated number of lessons that each class spent on topics per content area. Measure
North West Province
Botswana
Number, operations and relationships Measurement Space and shape (geometry) Data handling Patterns, functions and algebra
42.7 6.2 5.8 1.9 1.4
27.2 9.8 6.0 5.8 2.7
Total
58.0
52.0
Source: North West Province and Botswana Schools Sample, 2009.
mathematics notebooks. We found that teachers in the North West sample of classrooms gave 52 mathematics lessons, and in the Botswana sample, 78 mathematics lessons, from the beginning of the year to the date of the post-test (of a possible 130–140+ lessons). This covers all but the last month of school, suggesting that between competing teacher activities such as department meetings, professional development, and union meetings, teacher absenteeism, and replacing mathematics lessons with other student activities, students in both countries (especially South Africa) were exposed far less than planned to the mathematics curriculum. However, this does not, in itself, show that students achieved less because of this lower exposure. Fig. 4 shows that content coverage was also low. Only 14% of the sample of Grade 6 classes covered more than 50% of the test topics in North West, and in Botswana, only 3 of the 64 classrooms, or less than 5% of the sample of Grade 6 classes, covered more than 50% of the test topics.4 Our measure of interest, ‘‘content emphasis’’, suggests that teachers spent much more time on the topics covered in our student test related to number, operations and relationships than on the other content areas (Table 1). In the North West Province teachers spent 74% of the time spent on test topics overall on topics related to number, operations and relationships.5 In Botswana, only 52% of the lesson time was devoted to number, operations, and relationships (this is still a high fraction given that the Grade 6 curriculum was intended to be more oriented to the other, more advanced, topics), and much more time than in North West Province was devoted to data handling and measurement. When we use the ‘‘content emphasis’’ data to break down the average lessons per topic covered on the mathematics test we gave students, the North West teachers taught 2.5 and the Botswana teachers 2.2 lessons related to each test item from the beginning of the school year to the second observation of the student notebooks at the end October/early November. Especially in North West, the number of lessons taught on topics relating to each test item varied considerably—the standard deviation was 2.1 lessons across the 38 test items in North West Province and a lower standard deviation of 1.4 lessons in Botswana. 4 One explanation of the lower coverage in Botswana, even though Botswana classes met more frequently on average, is that teachers in Botswana were more likely to be covering topics that were not tested in the test we gave students. Our test covered a number of topics from the Grade 5 curriculum and did not cover all possible topics in Grade 6. Another explanation, which could parallel the first, is that researchers evaluating the notebooks for content exposure just had to count the number of lessons according to the notebooks’ dated entries. In the case of ‘‘content emphasis,’’ researchers were trying to estimate proportions of lessons spent on individual topics (within each lesson) and NW researchers may have overestimated (relative to Botswana researchers) the number of times that detailed topics were touched on. 5 In a meeting with 70 of the sampled teachers after the study was completed, they revealed that they do not teach parts of the math curriculum because they feel very insecure with and even incompetent to teach the material.
C. Reeves et al. / International Journal of Educational Development 33 (2013) 426–435
431
Fig. 3. North West Province and Botswana: Frequency of Observed Mathematics Lessons, 2009. Source: North West Province and Botswana Schools Sample, 2009.
Fig. 4. North West Province and Botswana: number of classrooms by percentage of test topics covered. Source: North West Province and Botswana Schools Sample, 2009.
Classrooms in which students initially scored less than 20 on the pre-test; Classrooms in which students initially scored 20–39.9 on pretest; and Classrooms in which students initially scored greater than 39.9 on pre-test.
classroom times test item (2318 or 2432 possible observations). The different specifications of independent variables used in the average item score regressions are: Model 1—post-test as a function of OTL (lessons per test item), controlling for pre-test score; Model 2—post-test score as a function of pre-test score, OTL, and OTLsquared; Model 3—average item score gain as a function of OTL’; Model 4—average item gain score as a function of OTL and OTL squared. The results in Tables 2a and 2b show that the relation between learning gains on items and the number of lessons given by teachers on each test item differs considerably between North West Province and Botswana. In North West classrooms, the gain is generally positively related to the number of lessons when modeled in quadratic form.6
This allowed us to test whether the opportunity to learn (OTL) influenced gains differentially across sub groups of students. Tables 2a and 2b show the regression results using the matrix of
6 As we would expect, the coefficient of OTL is somewhat smaller when we model value added assuming student learning does not decay between the pre-and posttest (dependent variable, gain score).
5.3. Estimating the regression model Using the model specified in the method section, we estimated our two models of student gains on each test item in each classroom as a function of the number of lessons on topics relating to each test item taught by each teacher. We also estimated the same models for the following sub-groups:
432
C. Reeves et al. / International Journal of Educational Development 33 (2013) 426–435
Table 2a North West Province: estimated gains on tests items per classroom related to opportunity to learn and average pre-test score, 2009. Variable
All classrooms Avg. pre-test score OTL
Dependent variable average item post-test score
Dependent variable average item gain score
Model 1
Model 2
Model 3
Model 4
0.7418*** (0.0363) 0.0018 (0.0009)
0.7366*** (0.0362) 0.0084*** (0.0018) 0.0005*** (0.0001) 0.0985*** 0.493 2318
0.0004 (0.0009)
0.0048*** (0.0018) 0.0004*** (0.0001) 0.0295*** 0.002 2318
OTL squared Intercept Adjusted R2 No. of observations Classrooms scoring 20 on pre-test Avg. pre-test score OTL
0.1046*** 0.490 2318 0.6276*** (0.1103) 0.0009 (0.0015)
OTL squared Intercept Adjusted R2 No. of observations Classrooms scoring 20–39.9 on pre-test Avg. pre-test score OTL
0.1270*** 0.058 912 0.9130*** (0.0905) 0.0028 (0.0015)
OTL squared Intercept adjusted R2 No. of observations Classrooms scoring 39.9 on pre-test Avg. pre-test score OTL
0.0500* 0.117 869 0.7226*** (0.0722) 0.0039 (0.0020)
OTL squared Intercept Adjusted R2 No. of observations
0.1109*** 0.243 537
0.6634*** (0.1089) 0.0034 (0.0024) 0.0003** (0.0001) 0.1213*** 0.061 912 0.9028*** (0.0900) 0.0138*** (0.0030) 0.0008*** (0.0002) 0.0419 0.131 869 0.7209*** (0.0534) 0.0070 (0.0049) 0.0003 (0.0003) 0.1081*** 0.242 537
0.0345*** 0.000 2318
0.0008 (0.0014)
0.0830*** .001 912
0.0027 (0.0015)
0.0259*** 0.003 869
0.0042* (0.0021)
0.0498** 0.005 537
0.0038 (0.0024) 0.0003** (0.0001) 0.0776*** 0.003 912
0.0136*** (0.0030) 0.0009*** (0.0002) 0.0150** 0.018 869
0.0060 (0.0047) 0.0002 (0.0003) 0.0520** 0.003 537
Source: North West School Sample, 2009. * p < 0.10. ** p < 0.05. *** p < 0.01.
The overall result in North West seems driven by what happens in classrooms averaging in the mid-range of pre-test scores (20– 39.9), although these average pre-test test scores are still very low (but not the lowest). This ‘‘mid-range’’ case is also one in which there is the least ‘‘decay’’ in the pre-test score (the coefficient of the pre-test score in the post-test estimate is greater than 0.9). For the entire sample, the average inflection point7 on the ‘‘parabola’’ in the quadratic OTL (lessons on topics related to test items) specification is 6–8 lessons per item, depending on how we model the value added in student learning. In other words, learning gains increase as average lessons taught on item test topics increase up to a rather high number of lessons touching on each test item. This is the case whether we assume learning decay (Model 2) or not (Model 4). However, at about 6–8 lessons per item, the gains begin to decrease, according to this estimate.8
7 The inflection point of a parabola is the coefficient of the X term divided by twice the coefficient of the i squared term. 8 One problem that often occurs in these types of regressions is serial correlation in the dependent variables—in this case, between item gains on the test across classrooms. We tested for this and found no significant problems of serial correlation between gains.
Whereas there are a few test items for which the average OTL is above or near the inflection point, the average lessons per item is 2.5, indicating that the inflection point is generally at a much greater number of lessons on a typical item than this rather low average. Yet, our estimates suggest that in many of the North West classrooms—namely, in the very low-scoring and in the higher scoring—the relation of additional lessons on test items to test score gains, although positive, is not statistically significant. The test score gain on items in Botswana classrooms is generally negatively related to the number of lessons given by teachers on each test item and the negative relation gets larger when the squared term is included. The negative coefficient is smaller in the value added model based on learning decay (Model 2). The overall result seems driven by what happens in the lower (<20) and mid-range of initial test scoring classrooms (20–39.9). The average inflection point on the ‘‘parabola’’ in the quadratic OTL (lessons on topics related to test items) specification is 16 lessons per item for the lowest initial scoring classrooms, about 6.4 lessons per topic for the middle scoring classrooms, and the relationship of lessons on topics to test score gains for the highest scoring classrooms (40 or higher percent on the initial test), depending on
C. Reeves et al. / International Journal of Educational Development 33 (2013) 426–435
433
Table 2b Botswana: estimated gains on tests items per classroom related to opportunity to learn and average pre-test score, 2009. Variable
All classrooms Avg. pre-test score OTL
Dependent variable average item post-test score
Dependent variable average item gain score
Model 1
Model 2
Model 3
Model 4
0.8329*** (0.0129) 0.0019 (0.0010)
0.8345*** (0.0128) 0.0047* (0.0019) 0.0002 (0.0002) 0.1120*** 0.574 2432
0.0027* (0.0020)
0.0066** (0.0019) 0.0003 (0.0002) 0.0580*** 0.004 2432
OTL squared Intercept Adjusted R2 No. of observations Classrooms scoring 20 on pre-test Avg. pre-test score OTL
0.1092*** 0.574 2432 0.8804*** (0.1063) 0.0066** (0.0020)
OTL squared Intercept Adjusted R2 No. of observations Classrooms scoring 20–39.9 on pre-test Avg. pre-test score OTL
0.1180*** 0.113 690 0.5980*** (0.1038) 0.0055** (0.0022)
OTL squared Intercept Adjusted R2 No. of observations Classrooms scoring 39.9 on pre-test Avg. pre-test score OTL
0.1830*** 0.053 912 0.8733*** (0.0302) 0.0053** (0.0024)
OTL squared Intercept Adjusted R2 No. of observations
0.0663** 0.401 830
0.8913*** (0.1081) 0.0130*** (0.0026) 0.0004*** (0.0001) 0.1257*** 0.121 690 0.5914*** (0.1030) 0.0176*** (0.0042) 0.0014** (0.0004) 0.1948*** 0.064 912 0.8674 *** (0.0297) 0.0114* (0.0046) 0.0006 (0.0004) 0.0626** 0.402 830
0.0535*** 0.002 2432
0.0066*** (0.0021)
0.1034*** 0.021 690
0.0050** (0.0019)
0.0659*** 0.005 912
0.0039 (0.0023)
0.0050 0.003 830
0.0131*** (0.0029) 0.0004*** (0.0001) 0.1126*** 0.030 690
0.0167*** (0.0039) 0.0013*** (0.0004) 0.0755*** 0.015 912
0.0086 (0.0046) 0.0004 (0.0004) 0.0104 0.004 830
Source: Botswana School Sample, 2009. * p < 0.10. ** p < 0.05. *** p < 0.01.
the model, is not significant or is positive and statistically significant. Thus, learning gains in Botswana are negatively related to test score gains up to a high number of lessons per topic (about 11–12), but the number of lessons up to which the relationship is negative becomes progressively smaller in classrooms with higher initial test scores, and may be positive in classrooms with higher average initial test scores. This suggests that the process of teaching in Botswana is such that classrooms with higher scoring students are benefitting more from more lessons than are classrooms with lower scoring students. 6. Discussion Our findings show that in North West Province schools, topic coverage matters for student learning, confirming what others have claimed in more general terms (Schmidt et al., 2001). We cannot argue definitively that lessons per item cause higher test score gains in North West Province, since, as we show elsewhere (Carnoy et al., 2012), the number of lessons given seems to be correlated with teacher mathematics knowledge across classrooms. Nevertheless, our estimates in Table 2a implicitly control
for teacher fixed effects and therefore provide considerable evidence that opportunity to learn does have a positive impact on gains in North West Province schools. This, in turn, suggests that teachers there could increase student learning by touching more often on the topics in the curriculum. As noted above, researchers in North West Province may have overestimated the number of lessons in which test topics were touched on. This could bias results if researcher errors were systematically correlated with those test items for which students had either larger or smaller gains. The most likely overestimate is in number, operations and relationships, yet when we estimate learning gains as a function of the number of lessons on that set of topics, the relationship is not statistically significant. Notwithstanding, possible recording errors could have biased estimates of the effect of OTL on student learning. The finding in Botswana that more lessons on items generally have a negative effect on gains and that the effect of more content emphasis is positive only in those classrooms with higher initial test scores is more difficult to explain. But because our study is comparative, we can use the differences between the two contexts to try to draw possible explanations of the differences in results.
434
C. Reeves et al. / International Journal of Educational Development 33 (2013) 426–435
The contrast in findings in the two contexts may result from differences in what parts of the curriculum teachers tend to teach in each country and the variation in content emphasis across classrooms. We pointed out that the standard deviation in our content emphasis measure is much larger across classrooms in our South African sample than in our Botswana sample. Botswana teachers emphasize content more uniformly, and, as we have seen, tend to put more emphasis on the more advanced subjects covered in the test we gave students. Botswana teachers are also more likely than South African teachers in our samples of public schools to teach to items that were not covered by our test, which focused more on 5th than 6th grade curriculum. In addition, other data we have show that the variation in average initial test scores across classrooms is much larger in our North West sample than in the Botswana sample (8.6 points standard deviation in North West versus 5.0 points standard deviation in Botswana). This means that Botswana teachers face 6th grade classrooms that tend to vary somewhat less in student mathematics ability. This may also help explain why Botswana teachers vary their content emphasis less across classrooms than in South Africa. The negative gains associated with more lessons given on each test item may therefore be associated with a tendency by Botswana teachers to aim their lessons ‘‘higher’’ than South African teachers regardless of the average initial test scores of the students in their classes. This ‘‘higher’’ average aim of content emphasis by Botswana teachers may favor students in classrooms where students have higher average initial test scores and may result in negative gains in classrooms with lower scoring students. Thus, the more homogeneous focus Botswana teachers put on somewhat more advanced mathematics topics may have a negative impact on learning gains in most classrooms and may favor students who already do better in mathematics. In our sample of lower-income North West province classrooms, the greater content emphasis on easier items and the somewhat greater heterogeneity of student mathematics ability across classrooms may result in increased content emphasis having a positive effect on learning gains, at least in classrooms above the lowest initial scores. The differences in OTL effects on learning gains between the North West Province and Botswana contexts are therefore instructive. They suggest that increased content emphasis per se does not uniformly result in greater student learning. Given the greater variation of student ability among classrooms, given what South African teachers tend to teach in their classrooms, and perhaps given the way they teach it, increased content emphasis contributes to their students learning gains, mainly in middle scoring classrooms. This suggests that in our sample of lower income South African schools, if teachers taught more mathematics lessons over the academic school year, most students’ mathematics learning would improve. However, given the smaller variation in initial student mathematics performance across Botswana classrooms, given what Botswana teachers tend to teach in their classrooms, and perhaps given the way they teach it, simply teaching more mathematics lessons may benefit higher scoring students but could have a negative impact on the learning gains of middle and lowest scoring students. It is important to emphasize that our results in this paper focus only on the contribution of OTL, as measured by content emphasis, to student learning. We noted earlier that overall, Botswana students in our study averaged greater gains than North West Province students on the mathematics test we gave them. Although our estimates suggest that increased content emphasis on test items tends to contribute to higher student learning gains in North West classrooms and lower learning gains in Botswana classrooms, other factors apparently work to improve average learning gains more in Botswana than in North West Province. This
suggests caution in jumping to conclusions about the size of OTL effects on student learning when compared to the effects of a composite of greater teacher skills, more effective teaching, and higher teacher expectations. Acknowledgements Thanks go to the schools, teachers, students and fieldworkers who participated in the study. We are grateful to the Spencer Foundation for their generous financial support and to Henry M. Levin for his insightful comments. References Abadzi, H., 2009. Instructional time loss in developing countries: concepts, measurement, and implications. World Bank Research Observer 24 (2), 267–290. Benavot, A, Amadio, M., 2004. A Global Study of Intended Instructional Time and Official School Curricula, 1980–2000. Background paper commissioned by the International Bureau of Education for the UNESCO-EFA Global Monitoring Report (2005). Berliner, D.C., 1990. ‘What’s all the fuss about instructional time?’. In: Ben BenPeretz, M., Bromme, R. (Eds.), The Nature of Time in Schools, Theoretical Concepts and Practitioner Perceptions. Teacher’s College Press, New York, NY. Beteille, T., 2009. Absenteeism, transfers, and patronage: the political economy of teacher labor markets in India. Unpublished Ph.D. dissertation, School of Education, Stanford University. Carroll, J., 1963. A model of school learning. Teachers College Record 64 (8), 723–733. Carnoy, M., Chisholm, L., Chilisa, B. (Eds.), 2012. The Low Achievement Trap: Comparing Schools in Botswana and South Africa. HSRC Press, Cape Town Available from http://www.hsrcpress.ac.za/product.php?productid=2293. Chaudhury, N., Hammer, J., Kremer, M., Muralidharan, K., Rogers, F.H., 2005. Missing in Action: Teacher and Health Worker Absence in Developing Countries. The World Bank, Washington, DC. Cronbach, L.J., Furby, L., 1970. How we should measure change: or should we? Psychological Bulletin 74 (1), 68–80. DeStefano, J., Schuh Moore, A., Hartwell, A., Balwanz, D., 2006. Reaching the Underserved through Complementary Models of Effective Schooling. EQUIP2, AED, and USAID, Washington, DC. Fleisch, B., 1999. School management and teacher accountability. In: Education Africa Forum, third ed. Education Africa, Johannesburg. Floden, R., 2003. The measurement of opportunity to learn. In: Porter, A.C., Gamoran, A. (Eds.), Methodological Advances in Cross-national Surveys of Educational Achievement. National Academy of Sciences, Washington, DC. Fuller, B., Hua, H., Snyder, C., 1994. When girls learn more than boys: the influence of time in school and pedagogy in Botswana? Comparative Education Review 38 (3), 347–376. Gillies, J., Quijada, J., 2008. Opportunity to Learn: A High Impact Strategy for Improving Educational Outcomes in Developing Countries. USAID, Washington, DC. Hill, H.C., Rowan, B., Ball, D.L., 2005. Effects of teachers’ mathematical knowledge for teaching on learner achievement. American Educational Research Journal 42, 371L 406. Hyslop, J., 1988. School student movements and state education policy 1972–1987. In: Cobbett, W., Cohen, R (Eds.), Popular Struggles in South Africa. James Currey, London. Ladd, H., 2008. Teacher effects: What do we know? In: Duncan, G., Spillane, J. (Eds.), Teacher Quality. Northwestern University, Evanston, IL. Lee, D., 1982. Exploring the construct of opportunity-to-learn. Integrated Education 20 (1–2), 62–63. Marshall, J., 2003. ‘If you build it, will they come? The effects of school quality on primary school attendance in rural Guatemala.’ Unpublished Ph.D. dissertation, School of Education, Stanford University. Moore, A.S., DeStefano, J., Gillies, J., 2006. Creating an opportunity to learn through complementary models of education. Journal of Education for International Development 3 (3), 1–15. Porter, A., Smithson, J., 2001. Are content standards being implemented in the classroom? A methodology and some tentative answers. In: Fuhrman, S. (Ed.), From Capitol to the Classroom: Standards-Based Reform in the States. University of Chicago Press, Chicago, pp. 60–80. Reddy, V., 2006. Mathematics and Science Achievement in South Africa in TIMMS 2003. HSRC, Cape Town. Reeves, C., 2005. The effect of ‘opportunity-to-learn’ and classroom pedagogy on mathematics achievement in schools serving low socio-economic status communities in the Cape Peninsula. Ph.D. dissertation, School of Education, Faculty of Humanities, University of Cape Town, Cape Town. Rosenshine, B., Berliner, D., 1978. Academic engaged time. British Journal of Teacher Education 4 (1), 3–26. Rowan, B., Correnti, R., Miller, J., 2002. What large-scale survey research tells us about teacher effects on student achievement: insights from the Prospects study of elementary schools. Teachers College Record 104 (8), 1525–1567. Schmidt, W., McKnight, C., Houang, R., Wang, H., Wiley, D., Cogan, L., Wolfe, R., 2001. Why Schools Matter: A Cross-National Comparison of Curriculum and Learning. Jossey-Bass, San Francisco.
C. Reeves et al. / International Journal of Educational Development 33 (2013) 426–435 SACMEQ (Southern African Consortium for Monitoring Education Quality), 2010. What are the levels and trends in reading and mathematics achievement? SACMEQ Policy Issue No. 2 (September). Stallings, J., 1980. Allocated academic learning time revisited, or beyond time on task. Educational Researcher 9 (11), 11–16. Stigler, J., Hiebert, J., 2004. Improving mathematics teaching. Educational Leadership 61 (5), 12–17. Taylor, N., Muller, J., Vinjevold, P., 2003. Getting Schools Working: Research and Systemic Reform in South Africa. Maskew Miller Longman/Pearson Education South Africa, Cape Town.
435
Thompson, D., Senk, S., 2001. The effects of curriculum on achievement in secondyear algebra. Journal for Research in Mathematics Education 32 (1), 58–84. van der Berg, S., Louw, M., 2008. South African student performance in regional context. In: Bloch, G., Chisholm, L., Fleisch, B., Mabizela, M. (Eds.), Investment Choices for South African Education. Wits University Press, Johannesburg. Wang, J., 1998. Opportunity to learn: the impacts and policy implications educational evaluation and policy analysis. American Educational Research Association 20 (3), 137–156. Williams., R.H., Zimmerman, D.W., 1996. Are simple gain scores obsolete? Applied Psychological Measurement 20, 59–69.
Contents lists available at SciVerse ScienceDirect
International Journal of Educational Development journal homepage: www.elsevier.com/locate/ijedudev
Comparing opportunity to learn and student achievement gains in southern African primary schools: A new approach Cheryl Reeves a,*, Martin Carnoy b,*, Nii Addy c a
Cape Peninsula University of Technology, South Africa Stanford University, United States c McGill University, Canada b
A R T I C L E I N F O
A B S T R A C T
Keywords: Opportunity to learn Student mathematics achievement gains Classroom fixed effects Southern Africa
A popular explanation for low student achievement in many developing countries’ primary schools is that students have relatively little opportunity to learn (OTL) the skills needed for academic success. However logical this explanation may be, surprisingly little empirical evidence has been presented to support it. In this paper we address this gap by estimating the effect of OTL on students’ academic performance using rich data we gathered on the teaching process in a large number of South African and Botswana Grade 6 classrooms. We use an innovative classroom fixed effects approach to estimate the impact of OTL on students’ mathematics achievement gains. We found statistically significant but very different results for our South Africa and Botswana samples. The discussion of those results in the context of differences in the two school systems gives us insights into the importance and limits of OTL as an explainer of student learning in low achievement schools. ß 2013 Elsevier Ltd. All rights reserved.
1. Introduction A popular explanation for low student achievement in many developing countries’ primary schools is that students have relatively little opportunity to learn the skills needed for academic success (Benavot and Amadio, 2004; Abadzi, 2009). However logical the explanation may be, surprisingly little empirical evidence has been presented to support it. We address this gap by estimating the effect of opportunity to learn (OTL) on students’ academic achievement using rich data we gathered on the teaching process in a large number of South African and Botswana classrooms. There is a long history of OTL as a key issue in student achievement. In its earlier incarnation in developed countries, OTL referred to the amount of time required for students with different abilities to learn the same curriculum (see, for example, Carroll, 1963; Berliner, 1990). In developing countries, OTL is defined more broadly as the amount of time that students are exposed to the curriculum in a given academic year. The educational effectiveness literature further breaks down OTL into ‘‘content coverage,’’ ‘‘content exposure’’ and ‘‘content emphasis.’’ The term ‘‘content
* Corresponding authors. E-mail addresses: [email protected] (C. Reeves), [email protected] (M. Carnoy). 0738-0593/$ – see front matter ß 2013 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.ijedudev.2012.12.006
coverage’’ is used to refer to the topics and sub-topics taught (Thompson and Senk, 2001; Porter and Smithson, 2001; Schmidt et al., 2001; Stigler and Hiebert, 2004). ‘‘Content exposure’’ refers to the amount of time devoted to a given subject overall (Rosenshine and Berliner, 1978; Floden, 2003; Lee, 1982; Wang, 1998; Porter and Smithson, 2001; Schmidt et al., 2001). ‘‘Content emphasis’’ refers to the relative amount of time students spend on the various topics (Wang, 1998). Most empirical research on OTL in developing countries has focused on measuring content coverage and content exposure (Gillies and Quijada, 2008; DeStefano et al., 2006; Moore et al., 2006). For example, the scheduled instructional time in developing country primary schools is often low compared to the minimum 850 h annually that experts claim is needed for a ‘‘quality education’’ (Benavot and Amadio, 2004). Schools in many developing countries are characterized by considerable teacher absenteeism (Chaudhury et al., 2005; Beteille, 2009) and student absenteeism may in part depend on teacher absenteeism (Marshall, 2003). Beyond absences, teachers are often present at school but not in their classrooms teaching, or are in their classrooms ‘‘teaching,’’ but are not teaching the required curriculum or are often not engaging the students in a meaningful way, especially on parts of the curriculum they feel uncomfortable teaching (Stallings, 1980). This growing literature has identified the generally poor and highly unpredictable delivery of the official curriculum to learners
C. Reeves et al. / International Journal of Educational Development 33 (2013) 426–435
as a potentially important problem, particularly in low and middleincome societies. Yet, analysts have done little to link variation in OTL to variation in student learning outcomes, providing limited evidence that increasing any of the usual measures of OTL improves student performance. There are a few exceptions. International cross section data from the Third International Mathematics and Science Survey (TIMSS) suggested that mathematics curriculum exposure has a positive effect on 8th grade student mathematics scores (Schmidt et al., 2001). Marshall in Guatemala (2003) and Reeves in the Western Cape Province of South Africa (2005), use student pre- and post-tests and classroom observations to estimate that teacher time use in primary school classrooms (Marshall, 2003) and the coverage of the required curriculum over a number of school years (Reeves, 2005) are associated with greater student learning gains. In this paper, we attempt to provide further empirical evidence for whether OTL contributes to student learning. The study is innovative in two ways. It compares the impact of OTL on the academic performance of very linguistically and socio-culturally similar students in the border regions of two educational systems—South Africa (North West Province) and Botswana—with different development histories but almost identical national curricula. Students in both systems are relatively low scoring, and low OTL has been identified as a problem, particularly in South Africa (Fleisch, 1999; Taylor et al., 2003). This allows us to test whether variation in OTL may have different impacts on student learning in different educational contexts. We use a classroom fixed effects approach to estimate the impact of OTL on students’ mathematics achievement gains in Grade 6 classrooms in a large sample of schools on either side of the border. We were able to get very detailed data on curriculum coverage (content coverage, content exposure, and content emphasis) from student notebooks, since in both countries teachers require students to record activities from all lessons in their individual notebooks. Using this detailed information on what took place in classrooms during the academic year, we were able to relate frequency of specific curriculum coverage in each classroom (content emphasis) to student achievement gains item-by-item on a 40-item mathematics test in that same classroom. The focus of our study in linking OTL to student mathematics performance is on the content (topics and sub-topics) made available in classrooms during the academic school year. In particular, we wanted to establish the extent to which Grade 6 students had the time and opportunity to learn the mathematics topics and sub-topics associated with the content in the items in the test we administered to students in our sample classrooms; that is, the ‘‘content emphasis’’ element of opportunity to learn. We found statistically significant but very different results for our South Africa and Botswana classroom samples. The discussion of those results in the context of differences in the two school systems provides insights into the importance and limits of OTL as an explainer of student learning in low achievement schools. 2. The context The school systems in Botswana and North West Province are imbedded in the same Setswana culture and language and both use English as the main teaching language in upper primary grades. However, Botswana gained independence from Britain in 1966 and developed its education in the context of national goals that were
427
designed to serve the entire population of Botswana equally. South African education was dominated by apartheid policies that separated Africans’ schooling from white and so-called ‘‘coloured’’ (mixed race and Asians) schools. North West Province was divided into an area that was part of ‘‘South Africa’’ and an African homeland, Bophutuswana, with its separate school system, but largely financed and controlled by the South African apartheid government. Ultimately, African schools in South Africa also became a primary site of struggle against apartheid, with students and teachers (through the teachers’ organization, SADTU) in the vanguard of the struggle (Hyslop, 1988). The meaning of education therefore differed in the two countries over the three decades before 1994, when apartheid ended and the Mandela government began to try to undo the apartheid legacy. Not surprisingly, however, fifteen years later, this legacy continues to shape the context of South African schools, through continued inequalities in resources, including the quality of teachers’ training, and through many teachers’ perception that their main role is as members of a political vanguard (through their teacher union organization) more than professionals responsible for student learning. These factors are much less of an issue in Botswana’s schools; hence, they have less of an influence on student’s opportunity to learn (Fuller et al., 1994). The other side of this coin is that students in Botswana score consistently higher on international tests of mathematics than students in South Africa (Reddy, 2006; van der Berg and Louw, 2008; SACMEQ, 2010). Despite relatively higher student achievement in Botswana, results of international comparative studies show that most students in the country are still learning at quite low levels. For example, in the 2003 and 2007 Trends in International Mathematics and Science Study (TIMSS), Botswana students were among the lowest scoring students worldwide. Even by African standards, Grade 6 students in Botswana and South Africa scored significantly lower on the SACMEQ tests in 2000 and 2007 than students in East Africa, despite much higher incomes per capita in Southern Africa (van der Berg and Louw, 2008; SACMEQ, 2010). 3. Data The data for the present study were collected as part of a larger study in a random sample of 116 public schools drawn from the population of no-fee public schools in both urban and rural areas located in the border region of the North West Province in South Africa and the Southeast region of Botswana (Carnoy et al., 2012). The study measured student learning gains, student family background variables, teacher background variables and mathematics knowledge, and data at the school level for 116 teachers teaching mathematics to more than 5000 Grade 6 students in 126 classrooms. As part of the study we also analyzed student notebooks twice during the school year and videotaped at least one and often two mathematics lessons for each teacher in the sample. We assessed students’ learning gains over the course of the academic year in 2009, through the use of a pre- and post-test based on the Grade 4–6 curriculum used in South African and Botswana schools. The student pre-test was administered toward the beginning of the school year in late March 2009, and the posttest, as near as possible to the end of the school year, in late October and early November 2009. The test was based on what students were supposed to have learned before entering Grade 6, and on what they should have learned during that academic year. The test results are reported below. From an analysis of the students’ notebooks, we estimated how many mathematics lessons teachers teach, what teachers cover in
428
C. Reeves et al. / International Journal of Educational Development 33 (2013) 426–435
those lessons, the grade level teaching of content, and how this relates to the curriculum that is supposed to be covered (the curriculum is national in both Botswana and South Africa) as represented by the 40 items on the curriculum based test that we applied to our sample of students. 3.1. Opportunity to learn instrument To ensure comparability of data collected on ‘‘content coverage’’ and ‘‘content emphasis’’ across classes (Porter and Smithson, 2001; Rowan et al., 2002), an opportunity to learn instrument was designed listing 110 mathematics topics associated with the study’s test items, which in turn, were based on the mandated national curriculum. These topics were organized on the instrument under five content areas: number, operations and relationships; patterns, functions and algebra; space and shape (geometry); measurement; and data handling. The list of topics and sub-topics used allowed for the collection of detailed data on content covered in class. For example, rather than simply recording coverage of ‘‘rounding off,’’ researchers needed to capture whether students have opportunities to ‘‘round off to the nearest 10, 100, or 1000.’’ 3.2. Opportunity to learn data sources and data collection In each of the classes observed, we reviewed a sample of the three most comprehensive student notebooks twice during the academic year—in May/June and again very near the end of the school year (October/November) 2009. Students in South Africa and Botswana are expected to write in their notebooks each time that they have a mathematics lesson. We used the dates of daily written work in students’ notebooks as evidence of students having had a lesson. For ‘‘content exposure,’’ researchers counted the number of ‘‘pieces’’ of dated work in the notebooks to estimate the number of mathematics lessons that had taken place between the beginning of the school year and October/November 2009. We were thus able to record the number of daily lessons during almost the entire academic year (all but the month of November and early December, a period of review before exams, when there is relatively less teaching of full topics). Detailed data on ‘‘content emphasis’’ and ‘‘content coverage’’ were collected only on topics associated with the content of the comparative study’s student test items. Fieldworkers used the topic list in the opportunity-to-learn instrument to indicate whether there was evidence, from observation of students’ notebooks, that a particular topic had been covered in each classroom during the school year 2009. To gauge the relative ‘‘content emphasis’’ given to each topic, fieldworkers also used dates of daily lessons in the students’ notebooks to estimate the number of lesson periods spent on each of the topics covered. The instrument allowed them to record whether less than one lesson had been spent on a particular topic. 4. Method A typical value-added model to estimate the effect of a classroom input on student outcomes relates the post-test score to the input, controlling for students’ pre-test score. Economists argue that this model assumes that students’ knowledge ‘‘decays’’ over the course of a year at a constant rate (an estimate of the decay is 1 minus the pre-test coefficient), with part of the treatment coefficient attributed to replacing the decay in knowledge. The model presents a problem in that the student’s initial achievement may be correlated with the error term (Ladd, 2008). One way to handle this problem is to estimate a value-added equation that uses test score differences as the measure of value added. That
model assumes no learning decay and eliminates the pre-test correlation with the error term, but poses other issues, namely whether differences in test scores are a statistically reliable measure of student learning gains.1 Such value added models are still subject to selection bias, in that teachers likely to offer more OTL may be self-selecting into classrooms with students likely to have larger gains in mathematics performance. Our model attempts to ‘‘correct’’ for that bias by estimating the OTL treatment effect across items in the same classroom with the same teacher (classroom fixed effects). We estimated two fixed effects models. The first model relates the number of lessons related to each test item in each classroom (OTL) to the average classroom post-test score on that item, controlling for the pre-test score on the same item. The second model relates OTL to the increase or decrease in the percentage of students getting an item correct that the teacher’s class made on that test item between the pre- and the posttest.2 For both models, we used 38 of the test items (two items were badly printed on the Botswana version of the initial test, so we left them out of this analysis both for Botswana and for North West Province). We had complete OTL and average achievement gain observations for 61 of the 62 classrooms in the North West Province sample and all 64 classrooms in Botswana. This made a matrix of 2318 observations of ‘‘content emphasis’’ in Northwest Province and 2432 observations of ‘‘content emphasis’’ in Botswana; a matrix of 2318 or 2432 observations of classroom-item average post-test scores, a matrix of 2318 or 2432 observations of classroom-item average initial test scores, and a matrix of 2318 or 2432 observations of classroom-item average test score gains (post-test score minus pre-test score). Our regression models are as follows: A posti j ¼ h þ aA prei j þ b1 OLT i j þ b2 OLTi2j þ e
(Models 1 and 2)
A posti j A prei j ¼ h þ b1 OLT i j þ b2 OLTi2j þ e
(Models 3 and 4)
where Apostij, average student post-test score on item i in classroom j; Apreij, average student pre-test score on item i in classroom j; OTLij, number of lessons touching on test item i in classroom j; Apostij Apreij, average student gain in post-test minus pre-test on item i in classroom j; h, intercept term; and e, an error term. The disadvantage of the method is that we cannot control for the many other data we collected on students and teachers, including student SES and teacher mathematics knowledge. The content emphasis measure of OTL we use (number of lessons given during the year on each of out student test items) was the only one 1 Cronbach and Furby (1970) argued that differences between pre-test and posttest scores are not very useful in social science research. Nevertheless, there has been widespread use of gain scores in educational research with reasonable and reproducible policy inferences (see, for example, Ladd’s review of estimating teacher effects with gain scores (Ladd, 2008), as well as Hill, Rowan, and Ball’s well known study of teacher effects (2005). Williams and Zimmerman (1996) reviewed the Cronbach and Furby critique from a statistical point of view and concluded that the reliability of the difference in post-test and pre-test is a function of the correlation between the post-test and pre-test and the ratio, l, of the standard deviations of the pre-test and post-tests. The closer the l is to 1.0, the greater the effect of the correlation between post-test and pre-test on reliability. In our case, according to the Williams and Zimmerman criterion (1996, Fig. 1), the reliability of the differences in test scores in both countries is not very high. As we show below, however, our results using test score differences produce results for our treatment variable that are consistent with those estimating the post-test. 2 There is considerable error associated with getting an item correct or incorrect from random marking on a multiple-choice test. However the error term is smaller for the average score for all students in the class. We took the average class pre-and post-test score on each item as the single best estimate of the proportion correct on that item in each class.
C. Reeves et al. / International Journal of Educational Development 33 (2013) 426–435
Fig. 1. North West Province and Botswana: average individual learner initial mathematics test score, by test item, late March, 2009. Source: North West Province and Botswana Schools Sample, 2009.
Fig. 2. (a) North West Province: average of individual learner pre-test and post-test mathematics score, by test item, late March and late October, 2009. Source: North West Province Schools Sample, 2009.
429
430
C. Reeves et al. / International Journal of Educational Development 33 (2013) 426–435
we could relate to particular test items. The method’s great advantage is that we can control for classroom fixed effects—we can estimate OTL effects across items within each classroom for a given teacher. 5. Results 5.1. Student mathematics achievement gains The average mathematics achievement in our sample of more than 3800 students in North West Province who took the initial test was 28.6%, with a standard deviation of 12.2. The average test score in our sample of 1774 Botswana students who took the initial test was 33.6% (including items 27 and 37, which had printing issues in the Botswana version of the test that made them difficult to read) with a standard deviation of 12.0.3 Thus, the difference in test score was a minimum of 5 points (including question 27 and 37 for both countries), a highly statistically significant difference. There was also great variation in the score from test item to test item in both countries. Fig. 1 shows this variation. The gains that students made between March and end October also varied greatly, with the gains on some items coming out reasonably large and positive, most gains small and positive, and two or three, negative (see Fig. 2a and b). In the North West sample, the average score on the post-test (round 2) for the sample as a whole was 31.6%, with a standard deviation of 12.4. The average gain for all of the 3485 students we could match in North West Province was therefore 3 points, with a standard deviation of 11.2 points. In Botswana, the gain was larger than in North West. The average score on the post-test for the sample as a whole was 38.6%, with a standard deviation of 14. Again, the difference between the average score in Botswana and North West was highly statistically significant. The average gain for all of the 1666 students we could match was 4.0 points (excluding gains on question 27 and 37), with a standard deviation of 10.2 points. The one point difference in gain between students in the two samples is also statistically significant at the 1% significance level. From pre- to post-test there were signs of improvement on a number of items. For example, in North West, learners scored between 40 and 49% on 6 items in the pre-test and on 8 items in the post-test and above 50% on 2 items in the pre-test and 5 items on the post-test. In Botswana, learners scored between 40 and 49% on 8 items in the pre-test and on 10 items in the post-test and above 50% on 8 items in the pre-test and 10 items on the post-test. North West Province learners showed overall negative gains in 10 of the 40 items on the learner test, with the biggest negative gain of about 18% on item 19. Botswana learners showed overall negative gains in 7 of the 40 items on the learner test, with the biggest negative gain of more than 20% also on item 19. That common fall in test score on item 19 resulted from a simple change in the problem, from a plus sign to a minus sign in an algebraic expression, suggesting that learners did not really understand the underlying mathematics of the expression in the pre-test. 5.2. Opportunity to learn across classrooms Overall, OTL in the classrooms we surveyed was low on all three measures used. Fig. 3 shows the number of mathematics lessons taught in the North West Province and Botswana classrooms in our samples (‘‘content exposure’’). Researchers counted the total number of days (by dates) students wrote work in their 3 If we exclude items 27 and 37, the average score on the initial test for Batswana learners was 34.6%, with a standard deviation of 12.4.
Table 1 North West Province and Botswana: estimated number of lessons that each class spent on topics per content area. Measure
North West Province
Botswana
Number, operations and relationships Measurement Space and shape (geometry) Data handling Patterns, functions and algebra
42.7 6.2 5.8 1.9 1.4
27.2 9.8 6.0 5.8 2.7
Total
58.0
52.0
Source: North West Province and Botswana Schools Sample, 2009.
mathematics notebooks. We found that teachers in the North West sample of classrooms gave 52 mathematics lessons, and in the Botswana sample, 78 mathematics lessons, from the beginning of the year to the date of the post-test (of a possible 130–140+ lessons). This covers all but the last month of school, suggesting that between competing teacher activities such as department meetings, professional development, and union meetings, teacher absenteeism, and replacing mathematics lessons with other student activities, students in both countries (especially South Africa) were exposed far less than planned to the mathematics curriculum. However, this does not, in itself, show that students achieved less because of this lower exposure. Fig. 4 shows that content coverage was also low. Only 14% of the sample of Grade 6 classes covered more than 50% of the test topics in North West, and in Botswana, only 3 of the 64 classrooms, or less than 5% of the sample of Grade 6 classes, covered more than 50% of the test topics.4 Our measure of interest, ‘‘content emphasis’’, suggests that teachers spent much more time on the topics covered in our student test related to number, operations and relationships than on the other content areas (Table 1). In the North West Province teachers spent 74% of the time spent on test topics overall on topics related to number, operations and relationships.5 In Botswana, only 52% of the lesson time was devoted to number, operations, and relationships (this is still a high fraction given that the Grade 6 curriculum was intended to be more oriented to the other, more advanced, topics), and much more time than in North West Province was devoted to data handling and measurement. When we use the ‘‘content emphasis’’ data to break down the average lessons per topic covered on the mathematics test we gave students, the North West teachers taught 2.5 and the Botswana teachers 2.2 lessons related to each test item from the beginning of the school year to the second observation of the student notebooks at the end October/early November. Especially in North West, the number of lessons taught on topics relating to each test item varied considerably—the standard deviation was 2.1 lessons across the 38 test items in North West Province and a lower standard deviation of 1.4 lessons in Botswana. 4 One explanation of the lower coverage in Botswana, even though Botswana classes met more frequently on average, is that teachers in Botswana were more likely to be covering topics that were not tested in the test we gave students. Our test covered a number of topics from the Grade 5 curriculum and did not cover all possible topics in Grade 6. Another explanation, which could parallel the first, is that researchers evaluating the notebooks for content exposure just had to count the number of lessons according to the notebooks’ dated entries. In the case of ‘‘content emphasis,’’ researchers were trying to estimate proportions of lessons spent on individual topics (within each lesson) and NW researchers may have overestimated (relative to Botswana researchers) the number of times that detailed topics were touched on. 5 In a meeting with 70 of the sampled teachers after the study was completed, they revealed that they do not teach parts of the math curriculum because they feel very insecure with and even incompetent to teach the material.
C. Reeves et al. / International Journal of Educational Development 33 (2013) 426–435
431
Fig. 3. North West Province and Botswana: Frequency of Observed Mathematics Lessons, 2009. Source: North West Province and Botswana Schools Sample, 2009.
Fig. 4. North West Province and Botswana: number of classrooms by percentage of test topics covered. Source: North West Province and Botswana Schools Sample, 2009.
Classrooms in which students initially scored less than 20 on the pre-test; Classrooms in which students initially scored 20–39.9 on pretest; and Classrooms in which students initially scored greater than 39.9 on pre-test.
classroom times test item (2318 or 2432 possible observations). The different specifications of independent variables used in the average item score regressions are: Model 1—post-test as a function of OTL (lessons per test item), controlling for pre-test score; Model 2—post-test score as a function of pre-test score, OTL, and OTLsquared; Model 3—average item score gain as a function of OTL’; Model 4—average item gain score as a function of OTL and OTL squared. The results in Tables 2a and 2b show that the relation between learning gains on items and the number of lessons given by teachers on each test item differs considerably between North West Province and Botswana. In North West classrooms, the gain is generally positively related to the number of lessons when modeled in quadratic form.6
This allowed us to test whether the opportunity to learn (OTL) influenced gains differentially across sub groups of students. Tables 2a and 2b show the regression results using the matrix of
6 As we would expect, the coefficient of OTL is somewhat smaller when we model value added assuming student learning does not decay between the pre-and posttest (dependent variable, gain score).
5.3. Estimating the regression model Using the model specified in the method section, we estimated our two models of student gains on each test item in each classroom as a function of the number of lessons on topics relating to each test item taught by each teacher. We also estimated the same models for the following sub-groups:
432
C. Reeves et al. / International Journal of Educational Development 33 (2013) 426–435
Table 2a North West Province: estimated gains on tests items per classroom related to opportunity to learn and average pre-test score, 2009. Variable
All classrooms Avg. pre-test score OTL
Dependent variable average item post-test score
Dependent variable average item gain score
Model 1
Model 2
Model 3
Model 4
0.7418*** (0.0363) 0.0018 (0.0009)
0.7366*** (0.0362) 0.0084*** (0.0018) 0.0005*** (0.0001) 0.0985*** 0.493 2318
0.0004 (0.0009)
0.0048*** (0.0018) 0.0004*** (0.0001) 0.0295*** 0.002 2318
OTL squared Intercept Adjusted R2 No. of observations Classrooms scoring 20 on pre-test Avg. pre-test score OTL
0.1046*** 0.490 2318 0.6276*** (0.1103) 0.0009 (0.0015)
OTL squared Intercept Adjusted R2 No. of observations Classrooms scoring 20–39.9 on pre-test Avg. pre-test score OTL
0.1270*** 0.058 912 0.9130*** (0.0905) 0.0028 (0.0015)
OTL squared Intercept adjusted R2 No. of observations Classrooms scoring 39.9 on pre-test Avg. pre-test score OTL
0.0500* 0.117 869 0.7226*** (0.0722) 0.0039 (0.0020)
OTL squared Intercept Adjusted R2 No. of observations
0.1109*** 0.243 537
0.6634*** (0.1089) 0.0034 (0.0024) 0.0003** (0.0001) 0.1213*** 0.061 912 0.9028*** (0.0900) 0.0138*** (0.0030) 0.0008*** (0.0002) 0.0419 0.131 869 0.7209*** (0.0534) 0.0070 (0.0049) 0.0003 (0.0003) 0.1081*** 0.242 537
0.0345*** 0.000 2318
0.0008 (0.0014)
0.0830*** .001 912
0.0027 (0.0015)
0.0259*** 0.003 869
0.0042* (0.0021)
0.0498** 0.005 537
0.0038 (0.0024) 0.0003** (0.0001) 0.0776*** 0.003 912
0.0136*** (0.0030) 0.0009*** (0.0002) 0.0150** 0.018 869
0.0060 (0.0047) 0.0002 (0.0003) 0.0520** 0.003 537
Source: North West School Sample, 2009. * p < 0.10. ** p < 0.05. *** p < 0.01.
The overall result in North West seems driven by what happens in classrooms averaging in the mid-range of pre-test scores (20– 39.9), although these average pre-test test scores are still very low (but not the lowest). This ‘‘mid-range’’ case is also one in which there is the least ‘‘decay’’ in the pre-test score (the coefficient of the pre-test score in the post-test estimate is greater than 0.9). For the entire sample, the average inflection point7 on the ‘‘parabola’’ in the quadratic OTL (lessons on topics related to test items) specification is 6–8 lessons per item, depending on how we model the value added in student learning. In other words, learning gains increase as average lessons taught on item test topics increase up to a rather high number of lessons touching on each test item. This is the case whether we assume learning decay (Model 2) or not (Model 4). However, at about 6–8 lessons per item, the gains begin to decrease, according to this estimate.8
7 The inflection point of a parabola is the coefficient of the X term divided by twice the coefficient of the i squared term. 8 One problem that often occurs in these types of regressions is serial correlation in the dependent variables—in this case, between item gains on the test across classrooms. We tested for this and found no significant problems of serial correlation between gains.
Whereas there are a few test items for which the average OTL is above or near the inflection point, the average lessons per item is 2.5, indicating that the inflection point is generally at a much greater number of lessons on a typical item than this rather low average. Yet, our estimates suggest that in many of the North West classrooms—namely, in the very low-scoring and in the higher scoring—the relation of additional lessons on test items to test score gains, although positive, is not statistically significant. The test score gain on items in Botswana classrooms is generally negatively related to the number of lessons given by teachers on each test item and the negative relation gets larger when the squared term is included. The negative coefficient is smaller in the value added model based on learning decay (Model 2). The overall result seems driven by what happens in the lower (<20) and mid-range of initial test scoring classrooms (20–39.9). The average inflection point on the ‘‘parabola’’ in the quadratic OTL (lessons on topics related to test items) specification is 16 lessons per item for the lowest initial scoring classrooms, about 6.4 lessons per topic for the middle scoring classrooms, and the relationship of lessons on topics to test score gains for the highest scoring classrooms (40 or higher percent on the initial test), depending on
C. Reeves et al. / International Journal of Educational Development 33 (2013) 426–435
433
Table 2b Botswana: estimated gains on tests items per classroom related to opportunity to learn and average pre-test score, 2009. Variable
All classrooms Avg. pre-test score OTL
Dependent variable average item post-test score
Dependent variable average item gain score
Model 1
Model 2
Model 3
Model 4
0.8329*** (0.0129) 0.0019 (0.0010)
0.8345*** (0.0128) 0.0047* (0.0019) 0.0002 (0.0002) 0.1120*** 0.574 2432
0.0027* (0.0020)
0.0066** (0.0019) 0.0003 (0.0002) 0.0580*** 0.004 2432
OTL squared Intercept Adjusted R2 No. of observations Classrooms scoring 20 on pre-test Avg. pre-test score OTL
0.1092*** 0.574 2432 0.8804*** (0.1063) 0.0066** (0.0020)
OTL squared Intercept Adjusted R2 No. of observations Classrooms scoring 20–39.9 on pre-test Avg. pre-test score OTL
0.1180*** 0.113 690 0.5980*** (0.1038) 0.0055** (0.0022)
OTL squared Intercept Adjusted R2 No. of observations Classrooms scoring 39.9 on pre-test Avg. pre-test score OTL
0.1830*** 0.053 912 0.8733*** (0.0302) 0.0053** (0.0024)
OTL squared Intercept Adjusted R2 No. of observations
0.0663** 0.401 830
0.8913*** (0.1081) 0.0130*** (0.0026) 0.0004*** (0.0001) 0.1257*** 0.121 690 0.5914*** (0.1030) 0.0176*** (0.0042) 0.0014** (0.0004) 0.1948*** 0.064 912 0.8674 *** (0.0297) 0.0114* (0.0046) 0.0006 (0.0004) 0.0626** 0.402 830
0.0535*** 0.002 2432
0.0066*** (0.0021)
0.1034*** 0.021 690
0.0050** (0.0019)
0.0659*** 0.005 912
0.0039 (0.0023)
0.0050 0.003 830
0.0131*** (0.0029) 0.0004*** (0.0001) 0.1126*** 0.030 690
0.0167*** (0.0039) 0.0013*** (0.0004) 0.0755*** 0.015 912
0.0086 (0.0046) 0.0004 (0.0004) 0.0104 0.004 830
Source: Botswana School Sample, 2009. * p < 0.10. ** p < 0.05. *** p < 0.01.
the model, is not significant or is positive and statistically significant. Thus, learning gains in Botswana are negatively related to test score gains up to a high number of lessons per topic (about 11–12), but the number of lessons up to which the relationship is negative becomes progressively smaller in classrooms with higher initial test scores, and may be positive in classrooms with higher average initial test scores. This suggests that the process of teaching in Botswana is such that classrooms with higher scoring students are benefitting more from more lessons than are classrooms with lower scoring students. 6. Discussion Our findings show that in North West Province schools, topic coverage matters for student learning, confirming what others have claimed in more general terms (Schmidt et al., 2001). We cannot argue definitively that lessons per item cause higher test score gains in North West Province, since, as we show elsewhere (Carnoy et al., 2012), the number of lessons given seems to be correlated with teacher mathematics knowledge across classrooms. Nevertheless, our estimates in Table 2a implicitly control
for teacher fixed effects and therefore provide considerable evidence that opportunity to learn does have a positive impact on gains in North West Province schools. This, in turn, suggests that teachers there could increase student learning by touching more often on the topics in the curriculum. As noted above, researchers in North West Province may have overestimated the number of lessons in which test topics were touched on. This could bias results if researcher errors were systematically correlated with those test items for which students had either larger or smaller gains. The most likely overestimate is in number, operations and relationships, yet when we estimate learning gains as a function of the number of lessons on that set of topics, the relationship is not statistically significant. Notwithstanding, possible recording errors could have biased estimates of the effect of OTL on student learning. The finding in Botswana that more lessons on items generally have a negative effect on gains and that the effect of more content emphasis is positive only in those classrooms with higher initial test scores is more difficult to explain. But because our study is comparative, we can use the differences between the two contexts to try to draw possible explanations of the differences in results.
434
C. Reeves et al. / International Journal of Educational Development 33 (2013) 426–435
The contrast in findings in the two contexts may result from differences in what parts of the curriculum teachers tend to teach in each country and the variation in content emphasis across classrooms. We pointed out that the standard deviation in our content emphasis measure is much larger across classrooms in our South African sample than in our Botswana sample. Botswana teachers emphasize content more uniformly, and, as we have seen, tend to put more emphasis on the more advanced subjects covered in the test we gave students. Botswana teachers are also more likely than South African teachers in our samples of public schools to teach to items that were not covered by our test, which focused more on 5th than 6th grade curriculum. In addition, other data we have show that the variation in average initial test scores across classrooms is much larger in our North West sample than in the Botswana sample (8.6 points standard deviation in North West versus 5.0 points standard deviation in Botswana). This means that Botswana teachers face 6th grade classrooms that tend to vary somewhat less in student mathematics ability. This may also help explain why Botswana teachers vary their content emphasis less across classrooms than in South Africa. The negative gains associated with more lessons given on each test item may therefore be associated with a tendency by Botswana teachers to aim their lessons ‘‘higher’’ than South African teachers regardless of the average initial test scores of the students in their classes. This ‘‘higher’’ average aim of content emphasis by Botswana teachers may favor students in classrooms where students have higher average initial test scores and may result in negative gains in classrooms with lower scoring students. Thus, the more homogeneous focus Botswana teachers put on somewhat more advanced mathematics topics may have a negative impact on learning gains in most classrooms and may favor students who already do better in mathematics. In our sample of lower-income North West province classrooms, the greater content emphasis on easier items and the somewhat greater heterogeneity of student mathematics ability across classrooms may result in increased content emphasis having a positive effect on learning gains, at least in classrooms above the lowest initial scores. The differences in OTL effects on learning gains between the North West Province and Botswana contexts are therefore instructive. They suggest that increased content emphasis per se does not uniformly result in greater student learning. Given the greater variation of student ability among classrooms, given what South African teachers tend to teach in their classrooms, and perhaps given the way they teach it, increased content emphasis contributes to their students learning gains, mainly in middle scoring classrooms. This suggests that in our sample of lower income South African schools, if teachers taught more mathematics lessons over the academic school year, most students’ mathematics learning would improve. However, given the smaller variation in initial student mathematics performance across Botswana classrooms, given what Botswana teachers tend to teach in their classrooms, and perhaps given the way they teach it, simply teaching more mathematics lessons may benefit higher scoring students but could have a negative impact on the learning gains of middle and lowest scoring students. It is important to emphasize that our results in this paper focus only on the contribution of OTL, as measured by content emphasis, to student learning. We noted earlier that overall, Botswana students in our study averaged greater gains than North West Province students on the mathematics test we gave them. Although our estimates suggest that increased content emphasis on test items tends to contribute to higher student learning gains in North West classrooms and lower learning gains in Botswana classrooms, other factors apparently work to improve average learning gains more in Botswana than in North West Province. This
suggests caution in jumping to conclusions about the size of OTL effects on student learning when compared to the effects of a composite of greater teacher skills, more effective teaching, and higher teacher expectations. Acknowledgements Thanks go to the schools, teachers, students and fieldworkers who participated in the study. We are grateful to the Spencer Foundation for their generous financial support and to Henry M. Levin for his insightful comments. References Abadzi, H., 2009. Instructional time loss in developing countries: concepts, measurement, and implications. World Bank Research Observer 24 (2), 267–290. Benavot, A, Amadio, M., 2004. A Global Study of Intended Instructional Time and Official School Curricula, 1980–2000. Background paper commissioned by the International Bureau of Education for the UNESCO-EFA Global Monitoring Report (2005). Berliner, D.C., 1990. ‘What’s all the fuss about instructional time?’. In: Ben BenPeretz, M., Bromme, R. (Eds.), The Nature of Time in Schools, Theoretical Concepts and Practitioner Perceptions. Teacher’s College Press, New York, NY. Beteille, T., 2009. Absenteeism, transfers, and patronage: the political economy of teacher labor markets in India. Unpublished Ph.D. dissertation, School of Education, Stanford University. Carroll, J., 1963. A model of school learning. Teachers College Record 64 (8), 723–733. Carnoy, M., Chisholm, L., Chilisa, B. (Eds.), 2012. The Low Achievement Trap: Comparing Schools in Botswana and South Africa. HSRC Press, Cape Town Available from http://www.hsrcpress.ac.za/product.php?productid=2293. Chaudhury, N., Hammer, J., Kremer, M., Muralidharan, K., Rogers, F.H., 2005. Missing in Action: Teacher and Health Worker Absence in Developing Countries. The World Bank, Washington, DC. Cronbach, L.J., Furby, L., 1970. How we should measure change: or should we? Psychological Bulletin 74 (1), 68–80. DeStefano, J., Schuh Moore, A., Hartwell, A., Balwanz, D., 2006. Reaching the Underserved through Complementary Models of Effective Schooling. EQUIP2, AED, and USAID, Washington, DC. Fleisch, B., 1999. School management and teacher accountability. In: Education Africa Forum, third ed. Education Africa, Johannesburg. Floden, R., 2003. The measurement of opportunity to learn. In: Porter, A.C., Gamoran, A. (Eds.), Methodological Advances in Cross-national Surveys of Educational Achievement. National Academy of Sciences, Washington, DC. Fuller, B., Hua, H., Snyder, C., 1994. When girls learn more than boys: the influence of time in school and pedagogy in Botswana? Comparative Education Review 38 (3), 347–376. Gillies, J., Quijada, J., 2008. Opportunity to Learn: A High Impact Strategy for Improving Educational Outcomes in Developing Countries. USAID, Washington, DC. Hill, H.C., Rowan, B., Ball, D.L., 2005. Effects of teachers’ mathematical knowledge for teaching on learner achievement. American Educational Research Journal 42, 371L 406. Hyslop, J., 1988. School student movements and state education policy 1972–1987. In: Cobbett, W., Cohen, R (Eds.), Popular Struggles in South Africa. James Currey, London. Ladd, H., 2008. Teacher effects: What do we know? In: Duncan, G., Spillane, J. (Eds.), Teacher Quality. Northwestern University, Evanston, IL. Lee, D., 1982. Exploring the construct of opportunity-to-learn. Integrated Education 20 (1–2), 62–63. Marshall, J., 2003. ‘If you build it, will they come? The effects of school quality on primary school attendance in rural Guatemala.’ Unpublished Ph.D. dissertation, School of Education, Stanford University. Moore, A.S., DeStefano, J., Gillies, J., 2006. Creating an opportunity to learn through complementary models of education. Journal of Education for International Development 3 (3), 1–15. Porter, A., Smithson, J., 2001. Are content standards being implemented in the classroom? A methodology and some tentative answers. In: Fuhrman, S. (Ed.), From Capitol to the Classroom: Standards-Based Reform in the States. University of Chicago Press, Chicago, pp. 60–80. Reddy, V., 2006. Mathematics and Science Achievement in South Africa in TIMMS 2003. HSRC, Cape Town. Reeves, C., 2005. The effect of ‘opportunity-to-learn’ and classroom pedagogy on mathematics achievement in schools serving low socio-economic status communities in the Cape Peninsula. Ph.D. dissertation, School of Education, Faculty of Humanities, University of Cape Town, Cape Town. Rosenshine, B., Berliner, D., 1978. Academic engaged time. British Journal of Teacher Education 4 (1), 3–26. Rowan, B., Correnti, R., Miller, J., 2002. What large-scale survey research tells us about teacher effects on student achievement: insights from the Prospects study of elementary schools. Teachers College Record 104 (8), 1525–1567. Schmidt, W., McKnight, C., Houang, R., Wang, H., Wiley, D., Cogan, L., Wolfe, R., 2001. Why Schools Matter: A Cross-National Comparison of Curriculum and Learning. Jossey-Bass, San Francisco.
C. Reeves et al. / International Journal of Educational Development 33 (2013) 426–435 SACMEQ (Southern African Consortium for Monitoring Education Quality), 2010. What are the levels and trends in reading and mathematics achievement? SACMEQ Policy Issue No. 2 (September). Stallings, J., 1980. Allocated academic learning time revisited, or beyond time on task. Educational Researcher 9 (11), 11–16. Stigler, J., Hiebert, J., 2004. Improving mathematics teaching. Educational Leadership 61 (5), 12–17. Taylor, N., Muller, J., Vinjevold, P., 2003. Getting Schools Working: Research and Systemic Reform in South Africa. Maskew Miller Longman/Pearson Education South Africa, Cape Town.
435
Thompson, D., Senk, S., 2001. The effects of curriculum on achievement in secondyear algebra. Journal for Research in Mathematics Education 32 (1), 58–84. van der Berg, S., Louw, M., 2008. South African student performance in regional context. In: Bloch, G., Chisholm, L., Fleisch, B., Mabizela, M. (Eds.), Investment Choices for South African Education. Wits University Press, Johannesburg. Wang, J., 1998. Opportunity to learn: the impacts and policy implications educational evaluation and policy analysis. American Educational Research Association 20 (3), 137–156. Williams., R.H., Zimmerman, D.W., 1996. Are simple gain scores obsolete? Applied Psychological Measurement 20, 59–69.