- Email: [email protected]
Private school vouchers and student achievement: A fixed effects quantile regression evaluation
Available online at www.sciencedirect.com
Labour Economics 15 (2008) 575 – 590 www.elsevier.com/locate/econbase
Private school vouchers and student achievement: A fixed effects quantile regression evaluation☆ Carlos Lamarche ⁎ Department of Economics, University of Oklahoma, 321 Hester Hall, 729 Elm Avenue, Norman, OK 73019, United States Available online 3 May 2008
Abstract Fundamental to the recent debate over school choice is the issue of whether voucher programs actually improve students' academic achievement. Using newly developed quantile regression approaches, this paper investigates the distribution of achievement gains in the first school voucher program implemented in the US. We find that while high-performing students selected for the Milwaukee Parental Choice program had a positive, convexly increasing gain in mathematics, low-performing students had a nearly linear loss. However, the program seems to prevent low-performing students from having an even bigger loss experienced by students in the public schools. © 2008 Elsevier B.V. All rights reserved. JEL classification: I21; I28 Keywords: School choice; Vouchers; Milwaukee; Fixed effects; Quantile regression
“While we celebrate those [students] doing well, we can't turn a blind eye to those who are not” Rod Paige, US secretary of Education, to the NY Times. ☆
Version: March 31, 2008. This paper is based on Chapter 2 of my dissertation “Quantile Regression for Panel Data” at the University of Illinois at Urbana-Champaign. I especially thank my advisor Roger Koenker for advice and detailed comments. I am grateful to Dan Bernhardt, Greg Burge, Todd Elder, Lynn Gottschalk, Kevin Hallock, Kangoh Lee, Darren Lubotsky, as well as labor lunch participants at the University of Illinois at Urbana-Champaign and seminar participants at Texas Tech and the 2007 Annual Meetings of the European Association of Labour Economists. I thank Professor Cecilia Rouse for providing the Milwaukee Parental Choice program's data. I would like to thank the Guest Editor as well as two anonymous referees for their very helpful comments. ⁎ Tel.: +1 405 325 5857. E-mail address: [email protected]. 0927-5371/$ - see front matter © 2008 Elsevier B.V. All rights reserved. doi:10.1016/j.labeco.2008.04.007
576
C. Lamarche / Labour Economics 15 (2008) 575–590
1. Introduction Fundamental to the recent debate over school choice is the issue of whether voucher programs actually improve students' academic achievement, while decreasing inequalities between the best and worst students. Milton Friedman's proposal to use vouchers as a method of improving the quality of education is based on the idea that private schools are more productive than public schools, which is still a highly controversial issue. If private schools are in fact more efficient than public schools, governments can improve the quality of education by offering tuition vouchers to families that want to send their children to private schools. The Milwaukee Parental Choice program, the first program implemented in the US, has been providing vouchers to low-income students to attend private school since 1990. The simplicity of the program's idea contrasts sharply with the complexities that plague the program's evaluation. The voucher programs' effect, for example from time t to t', is the difference between what would have happened at time t' if the student was selected and remained in a choice school during the time interval, and what would have happened at time t' if the student was not selected, and remained in the Milwaukee public school. This counterfactual exercise is impossible to obtain using observational data (Rubin, 1974). It may be possible, however, to construct groups using a randomized experiment (e.g., children randomly assigned to attend choice schools and to attend public schools). At time t', the difference between students' academic performances could be attributed to the type of school since the initial assignment was random. However, the Milwaukee Parental Choice program was not implemented under idealized conditions (Witte, 2000), and therefore the selection of the control group plays a major role. Given the lack of a valid control group, it is not surprising that previous empirical studies have delivered mixed findings. Rouse's (1998) seminal study used a sample of students in the Milwaukee public schools as a comparison group and individual fixed effects to control for latent characteristics, such as more motivated parents or student abilities that may differ between treatment and comparison groups. The presented approach builds upon Rouse by employing a newly developed fixed effects form of quantile regression that not only controls for unobserved individual heterogeneity, but also allows an examination of the program effects at different points of the educational attainment distribution. The empirical literature (e.g., Witte, 1997; Green et al., 1997; Rouse, 1998) has focused upon estimating how the selection to attend the Milwaukee choice schools affects mean test scores. This approach to evaluation may be incomplete for policy analysis of programs serving heterogeneous students. To illustrate, if a stated policy goal is to raise students' achievement to a predetermined minimum standard, it may not be optimal to pursue a program that benefits strong students while causing weaker students to fall further behind. Because education is expected to play an important role in mitigating inequality, “the distribution of achievement gains... constitutes an appropriate criterion for evaluating a school choice intervention” (Howell and Peterson, 2002). This paper focuses on the estimation of the selection to attend choice schools on the entire distribution of test scores, considering patterns of achievement in terms of quantiles. There are important reasons why economists, educators, and policy makers are interested in how voucher programs affect students' achievement beyond the mean effect, which is typically estimated with Ordinary Least Squares (OLS) or Instrumental Variables (IV). First, the standard methodology may miss how a policy affects achievement differently at different points of the conditional test score distribution, as illustrated in Eide and Showalter (1998). Second, the possibility that vouchers may increase the differences between high- and low-performing students in the private schools, concern notably reflected in Ascher, Fruchter, and Berne (1996) question
C. Lamarche / Labour Economics 15 (2008) 575–590
577
“What mechanisms ensure those students [in private schools] who need extra time and attention... receive this more costly instruction?”. Our analysis shows that being selected to participate in the program had an heterogeneous effect on educational attainment, which had not been uncovered by the previous work focusing on mean effects. First, we find that the selected students had an increase in math achievements that ranges from 3.1 to 1.2 percentile points per year across quantiles. The program seems to dramatically improve the academic achievements of the weak students, having a relatively modest effect on achievements among strong students. Second, although the mean effect suggests that the program has no effect on reading, we find some evidence suggesting that the program improved and reduced the achievements of the low- and high-performing students, respectively. Using a new instrumental variable estimator for quantile regression (Chernozhukov and Hansen, 2005), we find that the effect of being enrolled in choice schools ranges from 2.3 percentile points per year at the 0.1 quantile of the conditional distribution of reading scores, to − 2.4 at the 0.9 quantile. This result is important for policy analysis since the Milwaukee vouchers seem to have no effect at the mean, but indeed matter at the tails of the educational attainment distribution. Lastly, our results also reveal that students' gains, measured as the differences between test scores conditional on years since application to the program, are subtle in nature. While the “average” student in the program had a linear gain in mathematics, high-performing students had a positive, convexly increasing gain, and low-performing students had a nearly linear loss. The offer of vouchers seems to increase the inequalities between low- and high-performing students. However, as we mentioned above, weak students' scores increased dramatically, in levels, compared to public school students' scores. Therefore, the evidence suggests that being selected for the choice program prevented low-performing students from having an even bigger loss experienced by low-performing students in the public schools. The next section briefly introduces a simple behavioral framework, and Section 3 presents models and estimators. Section 4 describes the data and Section 5 the empirical results. Section 6 offers conclusions. 2. A simple behavioral framework We develop a simple variation of a model of mothers' decisions to enroll children in preschool (Behrman et al., 2004).1 The mother of student i maximizes a time separable utility function that depends on consumption C and the child's test score T. There is a technology that produces test scores depending on socioeconomic characteristics x, attendance to a choice school P, and stochastic factors εj, T = T(x, P,εj). We assume that εj is realized after the student attends school. For example, dissatisfaction with teachers, quality of education, school transportation problems associated either with choice school j = 1 or public school j = 0 attendance. The variable actual attendance to a choice school P is equal to aS− 1, taking value one if the student actually attends a choice school, 0 otherwise. The variable attendance to a choice school a takes the value one if the mother decides that the student should attend a choice school, and the variable selected to attend choice school in period t − 1, S− 1, take the value one if the student was selected. Lastly, each period the mother must spend assets A on consumption and, if she decides to send her child to a choice school, on a fixed cost K. 1 The framework models heterogeneity, which is unobserved by the econometrician, that can be recovered using quantile regression techniques (Koenker, 2005). Alternative strategies are considered in Bitler et al. (2006), Hastings et al. (2006), and Cullen et al. (2005).
578
C. Lamarche / Labour Economics 15 (2008) 575–590
It is possible to express the mother's dynamic problem simply as maxa{υ0, υ1}, where υ0 represents the value of not attending a choice school, and υ1 denotes the value of attending a choice school. If at time t the student was not selected, her mother trivially chooses a = 0 because she does not want to give up consumption to pay K. On the other hand, if at time t the student was selected to attend a choice school, the decision is simply, a = 1{υ1 ≥ υ0|S− 1 = 1}. This decision depends, among other factors, on the cost of attending a choice school, dissatisfaction with the choice schools, and mother's weight to her child educational achievements. These factors may be different among families, and consequently the probability of attendance may be different as well. 3. Evaluation of the Milwaukee voucher program The previous behavioral framework impose two restrictions on the evaluation of the voucher program, Tit ¼ T ðxit ; Pit ; eit Þ ¼ xitVA þ bPit eit
ð1Þ
Pit ¼ 1 υ1it zυ0it ÞSit1 ¼ d þ qit Sit1 :
ð2Þ
The assumption of linearity in Eqs. (1) and (2) gives a direct link to Rouse's (1998) structural equation model. Rouse estimates a reduced form equation assuming that the effect of selection into the voucher program on the probability of attendance is constant among families (ρit = ρ), yielding an “intention-to-treat” effect π1 that is equal to ρβ. An enormous difficulty in the evaluation of the Milwaukee Parental Choice program is the lack of a valid control group. Initial randomized treatment is needed to obtain an unbiased estimator of the voucher offer, but the program's randomization was based on the applicant group.2 Green et al. (1997) used the students not selected to attend choice schools as a comparison group, but this group may lead to selection bias (Rouse, 1998; Witte, 2000). Witte (1997) used a random sample of students from public schools as a comparison group, but as Rouse (1998) pointed out, there may be differences between the students that were selected to attend choice schools and the students that remained in the public schools.3 Only eligible parents who are more interested in their child's achievements apply to the program, which suggests that students who remained in the public system may have parents with a less strong preference for private schools. To get around these problems, Rouse (1998) used both the unsuccessful applicants and a random sample of students from the public schools as comparison groups, and individual fixed effects to control for unobservables such us more motivated parents or student ability. Our approach builds upon Rouse, estimating a fixed effects version of the quantile regression model, QTit sj jSit1 ; xit ; ai ¼ p0 sj þ p1 sj Sit1 þ xitVp2 sj þ ai ; ð3Þ where Q(·|·) is the τj-th conditional quantile function, and αi is an individual fixed effect. The non-selected students that are omitted for simplicity in Eq. (3) will be introduced in the regression 2 The Wisconsin legislature required choice schools to select students at random when the grade was oversubscribed. Although the best schools received more applications, the total number of applications exceeded the number of seats (Witte, 2000). 3 For instance, consider a family making a decision as to whether or not to apply to the program based on υi0j = xi0 ′ μ + αji, where the binary variable j takes the value 1 if the student applies to the program, and the value 0 otherwise. From Eq. (2), the participation in the program may depend entirely on parent's weight to her child educational achievements, Pi0 = 1{υi01 − υi00 ≥ 0} = 1{α1i − α0i ≥ 0}.
C. Lamarche / Labour Economics 15 (2008) 575–590
579
models estimated in Section 5. If we consider the possibility that the effect of selection to attend choice schools on the probability of attendance ρit is a linear function of ωi (e.g., individual or family reasons to leave the choice schools) and εit (e.g., choice program or private school factors), it is straightforward to show that the intention-to-treat effect, p1 sj ¼ b q Fe1 sj ¼ QTit sj jSit1 ¼ 1; xit ; ai QTit sj jSit1 ¼ 0; xit ; ai : The model, in its simplest version, assumes that the effect of the voucher offer π1(τj) is decreasing on the quantile τ, and depends on the effect of the selection on the probability of attendance and the differential school effect on achievement. We briefly consider two competing explanations. First, we may consider that the effect is a combination of a treatment effect β and a location-scale shift effect ρ(τj) = ρ − Fε− 1(τj).4 We conjecture that while only high-motivated students among low-performing students stay in the choice schools, most students among the high-performing students stay. Since those lower-performing students in the private schools are relatively more motivated than the lower-performing students in the public schools, the ‘intentionto-treat effect’ should be decreasing in terms of quantiles. Alternatively, we can hypothesize that it is a combination of a scale treatment effect β(τj) = − βFε− 1(τj) and a location shift effect β × ρ. It has been argued that private schools may work on all students equally, while public schools that are interested in raising mean achievement may give less amount of attention to lower-performing students rather than higher-performing students (Howell and Peterson, 2002, p. 156). Although the model now suggests that the voucher program has a different mechanism to affect achievement, we again expect to see a decreasing π1 over the quantiles. The empirical evidence will offer below (see, e.g., Figs. 1 and 2) provides some support for the latter explanation. 3.1. Estimation We estimate the ‘intention-to-treat’ effect π1(τj) using a quantile regression version of the classical fixed effects estimator introduced by Koenker (2004), J T N n o X X X J b sj j¼1 ; f abi gNi¼1 u arg min xj qsj Tit p0 sj p1 sj Sit1 ai p p;a
j¼1
t¼1
i¼1
where ρτj(u) = u(τj − I(u ≤ 0)) is the quantile loss function, and ωj is the weight (e.g., 1/J) given to the jth quantile. For simplicity, the vector of independent variables xit, defined below, is omitted. Parents' and students' unobserved characteristics that may differ between students' selected to attend choice schools and students in the public schools are captured by individual fixed effects αi's, independent of the quantiles. We advance the method in two directions. First, we propose to use panel-bootstrap to estimate the precision of the estimates of the parameters of the model. Our strategy accommodates to forms of heteroscedasticity replacing pairs {(Ti, Si)} : i = 1,…, N} over cross-sectional units i. For small and fixed number of time series observations, the standard errors are consistent provided that the It seems natural then to explore the differences between the treatment effect β and the intention-to-treat effect βρ at different quantiles of the educational attainment distribution. We first estimated a reduced form model based on Eqs. (1) b1 ¼ b and (2) to obtain p b q. We then estimated Eq. (1) to obtain βb considering the instrumental variable approach described in Section 3.1. We found that the percentage change between the treatment effect and the intention-to-treat effect has a tendency to increase suggesting that the effect of selection on the probability of attending choice schools is decreasing in τ. 4
580
C. Lamarche / Labour Economics 15 (2008) 575–590
Fig. 1. Estimated effects of being selected for the choice program times years since application on educational attainment. The panels present quantile regression estimates (solid line with dots), classical fixed effects (dashed line), and .95% confidence intervals for the point estimates.
Fig. 2. Estimates of the treatment effect. The panels present instrumental variable quantile regression estimates (solid line with dots), classical instrumental variable estimates (dashed line), and .95% confidence intervals.
C. Lamarche / Labour Economics 15 (2008) 575–590
581
number of cross-sectional units passes to infinity. Second, we develop a simple framework for testing, described in Appendix A. We also implement an instrumental variable form of quantile regression to estimate the causal effect of the program. By accommodating the function T(·) and the error term, Eqs. (1) and (2)are the system of equations considered by Chernozhukov and Hansen (2005, 3.1), who propose an instrumental variable (IV) method for estimating quantile regression treatment effects. We estimate the effect of choice schools on test scores considering, QTit ðsjPit ; xit Þ ¼ gðsÞ þ bðsÞPit þ xitVAðsÞ
ð4Þ
where β(τ) is the parameter of interest. The vector xit includes “applicant pool” dummy variables, a dummy variable for gender, family income, an indicator if family income is missing, and the grade level of the student when the student took the test. Since the selection to attend choice school is exogenous conditional on the school and grade to which the student applied, we use whether the child was randomly selected to attend choice schools Sit − 1 as an instrument for actual enrollment Pit. 4. Data We analyze data from the Milwaukee Parental Choice program.5 We consider a sample that contains information on applicants to the program including the students that were not selected for the choice schools and students from the Milwaukee public schools as in Rouse's paper (Table 1). There is data on reading and math test scores, based on the normal curve equivalent measure (NCE) from the Iowa Tests Basic Skills (ITBS). The empirical analysis is based on a sample of African-American and Hispanic students who applied to the choice program between 1990 and 1993, and a sample of students from the Milwaukee public schools.6 Test scores: The Normal Curve Equivalent (NCE) measure is a transformation of the ITBS that produces an integer-level measure, ranging from 1 to 99, with a national mean of 50 and a standard deviation of 21. We rely on math and reading NCE test scores because (a) the results can be compared with previous paper's findings (e.g., Witte, 1997; Rouse, 1998), and (b) the NCE measure has the advantage that it can be averaged. According to test makers, the NCE test score is the same if all the students make one year of progress after one academic year. If some students make more (less) progress in one year than the students in the population of interest, the NCE test score will be higher (lower).7
5 The Milwaukee Parental Choice program was targeted and limited, aimed to provide better educational opportunities to low income families. The program target was poor families living in the city of Milwaukee whose children were not attending private school that year. Families with income, at most, 1.75 times the national poverty line were eligible to apply (e.g., the threshold for a family of three was $21,000). 6 We use Rouse's sample of public school students assuring the comparability of results. As a robustness check, we estimated the effect of the voucher offer considering public school students with income less or equal to 1.75 times the national poverty line in 1990. The main empirical findings of this paper are robust to restricting the control group to include only low income students. 7 One-point increase at the quantile τ may not be translated to the same amount of knowledge in different grades, but addressing this possibility is beyond the scope of this paper. (Note that this same possibility is present when analyzing mean effects). Rather, the present exercise is focused on an investigation of the changes in the educational attainment conditional distribution for the selected students and public school students. Furthermore, we analyze trends over several years to avoid drawing conclusions based on one-year changes that are likely to be noisy measures of performance (Kane and Staiger, 2002).
582
C. Lamarche / Labour Economics 15 (2008) 575–590
Table 1 Descriptive statistics for the main variables in Rouse's sample analysis Variables
Students Selected
Not selected
MPS
Proportion currently enrolled in a choice school Math (NCE) scores Reading (NCE) scores Proportion female Family income (in thousands of 1994 dollars) Proportion missing income family Proportion African-American Proportion Hispanic Grade level test Proportion with imputed math test (Rouse, 1998 methodology) Number of students Number of observations
0.558 (0.497) 39.236 (18.706) 37.647 (16.265) 0.536 (0.499) 12.052 (5.915) 0.412 (0.492) 0.795 (0.404) 0.205 (0.404) 3.767 (2.250) 0.074 (0.262)
0.010 (0.102) 38.108 (18.777) 37.886 (17.090) 0.470 (0.499) 12.510 (5.901) 0.542 (0.498) 0.864 (0.343) 0.136 (0.343) 3.857 (2.088) 0.194 (0.396)
– 40.411 (18.483) 38.700 (16.461) 0.523 (0.500) 21.750 (8.658) 0.747 (0.435) 0.868 (0.338) 0.132 (0.338) 4.337 (2.122) 0.164 (0.371)
986 2462
358 859
2014 5408
MPS stands for Milwaukee public schools.
Year since application: This variable is defined as the first year in which a student applied if he either was selected the first time he applied or he was never selected to attend choice school. In case a student applied two or more times, the first year she was accepted is considered the year of application. In the case of the students in the Milwaukee public schools, the year of application was imputed considering year t the “year of application” for students who have a valid year t + 1 test score (Rouse, 1998). Application lotteries: The probability of being selected to attend choice school is random conditional on the school and grade to which the student applied, because the students were randomly selected when the school was oversubscribed for a particular grade. Consequently, we need controls in model (4) for the school and grade to which the student applied (see, e.g., Rouse, 1998). The application lotteries are time-invariant dummy variables, therefore they will not be included in model (3) because the variables are subsumed in the individual effects. 5. Results We shall present the empirical results. Instead of the focus on the mean achievement, we will report the effect of being selected to attend choice schools on the entire distribution of educational attainment. 5.1. The heterogeneous impact We will estimate a somewhat more complicated model than Eq. (3) containing the selected students, the non-selected students, and the sample of students from the public schools. We also introduce interactions between the indicator for whether or not the student was selected to attend a choice school and the years since application because previous researchers have argued that child's educational achievement may not improve immediately. We consider first the simplest version, assuming that gains are equal from year-to-year, Tit ¼ p0 þ p1 ðdit Sit Þ þ p2 ðdit Nit Þ þ
4 X k¼3
p3k ditk þ p4 Sit þ p5 Nit þ ai þ eit
ð5Þ
C. Lamarche / Labour Economics 15 (2008) 575–590
583
Table 2 Estimates of the selection to participate in the choice program on Math and Reading scores estimates for the full sample Quantiles 0.1
0.25
0.5
0.75
0.9
Mean
Fixed effects methods Dependent variable = math (NCE) test score Selected to attend choice school Not selected to attend choice school Selected × number of years Not selected × number of years
− 3.649 (1.866) −4.270 (1.429) − 3.321 (1.167) − 2.288 (1.192) −1.173 (1.633) − 3.391 (1.114) − 2.758 (3.192) −5.026 (2.109) − 3.621 (2.170) − 2.769 (2.150) −1.326 (2.707) − 3.052 (1.741) 2.323 (0.794)
3.143 (0.608)
2.321 (0.454)
1.685 (0.477)
1.178 (0.669)
2.294 (0.399)
0.919 (1.643)
1.681 (1.122)
1.526 (1.138)
0.844 (0.960)
−0.675 (1.026) 0.672 (0.783)
1.400 (1.169)
2.780 (1.412)
0.750 (1.027)
− 2.140 (3.479) −3.200 (2.070) − 0.555 (2.161) 1.350 (2.008)
2.605 (2.410)
− 0.644 (1.584)
Fixed effects methods Dependent variable = reading (NCE) test score Selected to attend choice school Not selected to attend choice school Selected × number of years Not selected × number of years
− 0.320 (1.625) −1.390 (1.372) 0.180 (1.254)
0.320 (0.687)
0.690 (0.630)
− 0.120 (0.541) − 0.700 (0.472) −1.160 (0.635) − 0.249 (0.368)
0.440 (1.555)
1.500 (1.069)
0.555 (0.898)
− 0.350 (0.813) −0.985 (1.015) 0.247 (0.706)
Math regressions include a dummy variable for whether the test score was imputed. The total number of observations are 8729 (Math) and 8751 (Reading). The standard errors (in parenthesis) are obtained after 1000 panel-bootstrap replications.
where dit − k indicates the number of years pre- or post-application to the program, (dit × Sit) is an interaction between the number of years since application and whether the student was selected to attend choice schools, and Nit is a dummy variable indicating whether the student applied to the program and was not accepted. 5.1.1. Fixed effects estimates We report estimates for the full sample in Table 2. First, the table shows negative estimates for the effect of being selected into a choice school, suggesting that the selected students had lower math test scores than the students in the public schools when they entered the program.8 Although we cannot provide a definitive explanation, it may be possible that program's applicants were not doing well in the public schools. Witte (2000) presents evidence on parental dissatisfaction with their kids' prior school within the Milwaukee public schools finding that (a) choice students seem to have less satisfied parents, and (b) the level of parents' dissatisfaction is statistically different between the choice students and the students in the public schools. He also argues that attitudes of parents toward their kids' prior school may be associated to their children's academic
8
Research had indicated that choice students have low test scores relative to low-income public school students (see, e.g., Witte, 2000), but the literature is relatively silent on explaining why the academic achievement of these students was low when they entered the program. As shown in Table 2, Rouse's (1998) estimated mean effect of being selected into the program on math test scores is negative and significant.
584
C. Lamarche / Labour Economics 15 (2008) 575–590
performance. Note that our hypothesis is also supported by the finding of negative and significant effects mainly among low-performers. The least squares fixed effect estimates suggests that the student selected for a choice school earn 2.3 additional percentile points per year in math relative to the student in the public schools. If instead of focusing on the mean effect, we consider various quantiles of the math score distribution, we see that the gains have some tendency to decrease as we go across the quantiles. For instance, the estimate changes from 3.1 additional percentile points per year at the 0.25 quantile to 1.2 points at the 0.9 quantile.9 Looking at the estimates of the gain in reading, the mean regression suggests that the students selected to attend choice schools did not improve relative to the students in the public schools. We get a bit more detail about the impact of the program by taking a look at various quantiles. Again, we see that there is a tendency to decrease as long as the quantile increases; from 0.7 additional percentile points per year at 0.25 quantile to − 1.2 additional percentile points per year at the 0.9 quantile. The program has a modest effect on the low-performing students, and a negative effect on the high-performing students, though the effects are insignificant at standard levels. Fig. 1 presents the effect of the selection to attend choice schools times years since application as a function of the quantiles of the conditional distribution of test scores. We observe that while the conditional mean effect seems to provide an incomplete summary of the effect of being selected to attend choice school on educational attainment, the quantile intention-to-treat effect provides a more detailed assessment of the gains among students. The left panel suggests that being selected to attend choice school has a positive effect on the lower tail of the math score distribution, therefore the program seems to improve the academic achievements of the low-performing students. For example, the students who did worse than the 0.3 quantile and were selected to attend choice school scored approximately 3.75 additional percentile points per year compared with the students who remained in the public school system. But, there is a modest effect on math and an adverse effect on reading in the right tail, not changing and reducing the achievements of the high-performing students. For instance, the students selected to the choice program earn approximately −1 percentile points per year relative to the students in the public schools at the 0.8 quantile of the reading score distribution.10 Earlier we noted that the voucher offer has a dramatic effect on low-performing students, which could be attributed to several channels. It may be simply argued that private schools are better than public schools, although the mechanism seems to be more complex. Another possible channel through which the program increased dramatically the achievement of the low-performing students in private schools could be that the program attracted the brightest students of the public schools. Since lower-performing students in the choice schools may be relatively more motivated than lowerperforming students in the public schools, we expect to see a dramatic effect in the lower tail of the educational attainment distribution. Finally, public schools may give less amount of attention to lower-performing students because they are evaluated on the basis of average test scores. 9 We evaluate the sensitivity of the results considering the 1991–1993 cohorts and a sample that excludes 1994 test scores for the 1990 cohort. Fixed effects quantile regression estimates again suggest a significant gain in mathematics with a tendency to decrease as we go across the quantiles. The results are available upon request. 10 We also estimated the achievement gains among the students selected to attend choice school, dividing this group of students in (a) those who are actually enrolled and (b) those who are not. We find, once again, both(i) significant effects in the lower tail of the math test score distribution, and (ii) a decreasing voucher effect after the 0.25 quantile. We also find that that the students who were selected to attend choice and remained in the program earn more percentile points than the students who were selected but dropped the program. For example, at the median of the math score distribution, the students who are enrolled in choice schools earn an increment of 2.9 percentile points per year, but the students who dropped choice schools earn just 1.3 percentile points. These results are available upon request.
C. Lamarche / Labour Economics 15 (2008) 575–590
585
5.1.2. IV estimates Table 3 presents IV estimates of the effect of enrollment on educational achievement. The least squares estimates suggest that the students enrolled in choice schools earn 1.7 additional percentile points per year in mathematics relative to the students enrolled in the public schools. Looking at the various quantiles of the math score distribution, we observe that the estimated effects are positive and different across quantiles. The estimated causal effect on math scores is positive and large for low-performing students, but small and statistically indistinguishable from zero for high-performing students. Note also that the mean effect suggests that the program has a negative, insignificant effect on reading. But this is an incomplete, possibly wrong, conclusion because there is a tendency to decrease as we go across the quantiles. The gain ranges from 2.3 additional percentile points per year at the 0.1 quantile to − 2.4 additional percentile points per year at the 0.9 quantile. The mean effect clearly misses relevant information for policy evaluation. Looking at the estimated effect presented in Fig. 2, we observe an heterogeneous impact that decreases as the quantile increases. The panels suggest that the Milwaukee Parental Choice program has a positive effect on the lower tail of math and reading conditional distributions, but no effect on math, and an adverse effect on reading in the right tail. The right panel suggests that the program improved and reduced reading achievements of the lower and higher attainment students, respectively. While students who did worse than the 0.3 quantile earned approximately 2 additional percentile points per year relative to the students in the public schools, the students who did better than the 0.7 quantile scored approximately − 2 percentile points per year. 5.2. Achievement gains In this section, we estimate the distribution of achievement gains considering, Tit ¼ p0 þ
4 X
p1k ðditk Sitk Þ þ
k¼1
4 X
p2k ðditk Nitk Þ þ
k¼1
4 X
p3k ditk þ ai þ eit
ð6Þ
k¼3
where, as before, dit − k is defined as before, Sit − k indicates whether the student was selected to attend a choice school, and Nit − k indicates whether the student was not selected. Considering a random sample of Milwaukee public schools students as a comparison group, we employ the fixed Table 3 Instrumental variables estimates of the causal effect of choice schools on math and reading test scores Quantiles 0.1 IV Methods
0.25
0.5
0.75
Currently enrolled 2.331 (1.701) 3.688 (1.399) 2.766 (1.090) 0.182 (1.206) in choice school IV Methods
0.9
Mean
0.605 (1.901)
1.701 (0.926)
Dependent variable = math (NCE) test score
Dependent variable = reading (NCE) test score
Currently enrolled 2.308 (1.378) 2.293 (1.080) 0.556 (0.971) −1.738 (1.126) − 2.417 (1.288) − 0.103 (0.810) in choice school Math regressions include a dummy variable for whether the test score was imputed. The total number of observations are 3177 (Math) and 3163 (Reading). The standard errors (in parenthesis) are obtained after 1000 panel-bootstrap replications.
586
C. Lamarche / Labour Economics 15 (2008) 575–590
effects method to estimate the panel data-quantile regression model, QTit (τj|dit − k, Sit − k, Nit − k, αi), evaluated by years since application to the program. bT (τj|·), and Table 4 shows coefficient estimates from the τj-th conditional quantile function Q it b (Tit|·). The students selected to attend choice schools had a roughly the conditional mean function E linear gain in mathematics, scoring approximately 2 additional points after four years (from 39.6 to 41.7 points). Part of the increase is explained by the performance of the students in the public schools, who had a negative 4.9 point gain, going from 40.4 to 35.5 points. We also see that lowperforming students in the public schools have a significant loss, while high-performing students have a negligible loss after four years. The relatively large and positive effect in the lower tail observed in Fig. 1 is therefore partially explained by the poor performance of the weakest students in the Milwaukee public schools. The table also shows reading scores, suggesting that there are no differences among the groups of students in the lower and upper tails of the score distribution. Although we cannot provide a definitive explanation for the poor performance of the public school students, two possibilities come to mind. First, the observed trend may be part of a decades-long trend. Witte (2000) presents historical data on achievement documenting that a decline of the public scores relative to the national norm occurred in the early 1970's. Second, the test scores could have been declining after the implementation of the voucher program due to the natural instability of the public school environment. In the first years, the changes in the educational institutions could affect the achievements of the students who needed more attention. The drop in the estimated public school test scores in the first years has been previously documented in charter schools in Washington D.C., Texas, Michigan, and in the Milwaukee voucher program at the mean level (see, e.g., Howell and Peterson, 2002; Rouse, 1998). Fig. 3 provides an efficient way to summarize the effect of being selected to attend choice school on achievement gains measured as differences in test scores levels. Rouse (1998) found that there is a linear gain in mathematics for students selected to attend choice schools (middle Table 4 Adjusted math and reading (NCE) test scores by year after application to the choice program Years since application to the program
Quantiles 0.1
0.25
0.5
0.75
0.9
Mean
Selected to attend choice schools Math (NCE) test score One year Four years
29.23 (1.25) 35.91 (0.96) 39.00 (0.76) 42.12 (0.79) 48.12 (1.03) 39.65 (0.73) 28.00 (2.88) 36.01 (2.06) 42.00 (1.46) 47.12 (1.34) 51.91 (1.98) 41.73 (1.28)
Milwaukee public school sample Math (NCE) test score One year Four years
31.91 (0.70) 37.00 (0.61) 40.00 (0.52) 42.73 (0.58) 47.91 (0.78) 40.39 (0.37) 22.51 (1.21) 27.05 (0.93) 35.00 (1.07) 42.91 (0.94) 46.70 (1.30) 35.53 (0.72)
Selected to attend choice schools Reading (NCE) test score One year Four years
29.00 (1.12) 35.00 (1.01) 39.00 (0.93) 43.00 (0.97) 48.00 (1.02) 39.04 (0.67) 22.00 (2.03) 29.00 (1.92) 36.00 (1.67) 41.00 (1.24) 43.00 (1.47) 35.06 (1.17)
Milwaukee public school sample Reading (NCE) test score One year Four years
29.00 (0.83) 35.00 (0.67) 39.00 (0.62) 42.00 (0.65) 46.00 (0.75) 38.39 (0.34) 23.00 (0.97) 29.00 (0.93) 36.00 (0.95) 42.00 (0.85) 46.00 (1.07) 35.62 (0.57)
The standard errors (in parenthesis) are obtained after 1000 panel-bootstrap replications.
C. Lamarche / Labour Economics 15 (2008) 575–590
Fig. 3. Adjusted math scores by years since application to the choice program. CS denotes choice schools.
587
588
C. Lamarche / Labour Economics 15 (2008) 575–590
Table 5 Tests for the equality of the adjusted test scores Years after application
Quantiles 0.1
0.25
0.5
0.75
0.9
0.178 0.054 0.000 0.000 0.000 0.000 0.000 0.000
0.001 0.000 0.001 0.848 0.000 0.002 0.000 0.009
0.287 0.064 0.021 0.380 0.053 0.141 0.021 0.024
0.302 0.132 0.383 1.000 0.397 0.502 0.334 0.540
0.015 0.003 0.581 1.000 0.080 0.029 0.090 0.087
Group of students
H0
Math (NCE) test score
Selected to attend choice schools
I II I II I II I II
0.926 0.679 0.000 0.000 0.000 0.000 0.000 0.000
Group of students
H0
Reading (NCE) test score
Selected to attend choice schools
I II I II I II I II
0.014 0.002 0.000 0.000 0.000 0.000 0.000 0.000
Milwaukee public school sample Selected and Milwaukee public school sample All groups
Milwaukee public school sample Selected and Milwaukee public school sample All groups
0.868 0.965 0.000 0.000 0.000 0.000 0.000 0.000
0.009 0.003 0.000 0.000 0.000 0.000 0.000 0.000
0.176 0.078 0.000 0.000 0.002 0.001 0.004 0.005
Hypothesis I considers the equality of coefficients across years since application. Hypothesis II considers the equality of coefficients between the first and fourth year since application. p-values are reported.
panel). We get more detail about the effect of being selected to attend choice schools over time by focusing on the math scores for the weakest and strongest students. While high-performing students have a positive, convexly increasing gain, low-performing students have a nearly linear loss. This evidence suggests the possibility that the Milwaukee vouchers increased the differences between high- and low-performing students in private schools. At first glance, one may find the estimates in Table 4 contradicting those presented in Table 2. The weak students selected to attend choice schools had roughly a point loss in math after four years since application to the program, but we concluded that these students were benefited by the voucher offer. This paradox is easily explained by the fact that we are now relaxing the assumption of equal gains over the years, considering differences in conditional test score levels instead of levels. We offer a simple numerical exercise to see the point. The average difference between the continuous line and the dashed line in the panels of Fig. 3 should be positive and decreasing beyond the 0.25 quantile as in Fig. 1. We find differences equal to {1.76, 3.66, 2.52, 1.43, 1.47} points at the {0.1, 0.25, 0.5, 0.75, 0.9} quantiles. Therefore, although low-performers in the choice schools were getting behind relative to the national norm after four years since application to the program, these students had significant gains relative to their counterfactual group. Finally, we test if the gains are significantly different considering the framework for hypothesis testing described in Appendix A. We test two hypotheses. First, we check whether the adjusted test scores in Table 4 are significantly different across several quantiles. We found that these differences were statistically significant across the quantiles of the math and reading score distributions (p-values were 0.000).11 Second, we test the equality of test scores across several 11
The results are available by the author upon request.
C. Lamarche / Labour Economics 15 (2008) 575–590
589
years since application to the program (Table 5). We find that (a) math scores are statistically different among groups, and (b) the difference between the scores in the first and fourth year of application for high-performing students selected to the program is significant at the standard 10% level (p-value equal to 0.064). Lastly, not surprisingly, there is weaker evidence that predicted reading scores are different across the years since application to the program. 6. Conclusions This paper investigates the distribution of achievement gains in the Milwaukee Parental Choice program. First, we find that the program seems to dramatically improve the academic achievements of weak students, having a relatively modest effect on achievements among strong students. Second, although the mean effect suggests that the program has no effect on reading, our results suggest that the program improved and reduced the achievements of the lower and higher-performing students, respectively. Third, the results also reveal that students' gains are subtle in nature. While the “average” student in the program had a linear gain in mathematics, we find that high attainment students had a positive, convexly increasing gain, and low attainment students had a nearly linear loss. Because education is expected to play an important role in mitigating inequality, the estimation of patterns of achievement in terms of quantiles may lead to a richer debate on school choice, a debate beyond the educational attainment of the “average” low-income student. Appendix A. General linear hypothesis We consider a general linear hypothesis on the vector ξ of the form H0 : Rξ = r, where R is a matrix that depends on the type of restrictions imposed. We evaluate the significance of differences across coefficient estimates from model (6) at different quantiles, and the significance of differences across coefficient estimates at different years since application on a vector with typical element, xkj ¼ QkT sj ; where QTk (τj) denotes the coefficients from the τj-th conditional quantile function k years after application to the choice program. For example, when we test whether the adjusted test score is significantly different across several quantiles one year after application, the vector is ξj1 = (QT1 (0.1),…, QT1(0.9))′, and when we test the equality of the adjusted test scores at the 0.1 quantile across several years since application, ξ1k = (QT1 (0.1),…, QT4 (0.1))′. The test statistics, h i1 b 1 R V R xb r T NT ¼ NT R xb r V R V b is the is asymptotically distributed as χq2 under H0, where q is the rank of the matrix R and V estimated covariance matrix. References Ascher, C., Fruchter, N., Berne, R., 1996. Hard Lessons: Public Schools and Privatization. Twentieth Century Fund Press, New York. Behrman, J., Cheng, Y., Todd, P., 2004. Evaluating preschool programs when length of exposure to the program varies: a nonparametric approach. Review of Economics and Statistics 86 (1), 108–132. Bitler, M., Gelbach, J., Hoynes, H., 2006. What means impact miss: distributional effects of welfare reform experiments. American Economic Review 96 (4), 988–1012.
590
C. Lamarche / Labour Economics 15 (2008) 575–590
Chernozhukov, V., Hansen, C., 2005. An IV model of quantile treatment effects. Econometrica 73 (1), 245–262. Cullen, J., Jacob, B., Levitt, S.D., 2005. The impact of school choice on student outcomes: an analysis of the Chicago public schools. Journal of Public Economics 89 (5–6), 729–760. Eide, E., Showalter, M., 1998. The effect of school quality on student performance: a quantile regression approach. Economic Letters 58, 345–350. Green, J.P., Peterson, P., Du, J., 1997. The effectiveness of school choice: the Milwaukee experiment. Harvard University Education Policy Paper 97-1. Hastings, J., Kane, T., Staiger, D., 2006. Preferences and heterogeneous treatment effects in a public school choice lottery. NBER Working Paper, W12145. Howell, W., Peterson, P., with Wolf, P., Campbell, D., 2002. The Education Gap: Voucher and Urban Schools Brookings Institution Press, Washington, D.C. Kane, T., Staiger, D., 2002. The promise and pitfalls of using imprecise school accountability measures. Journal of Economic Perspectives 16 (4), 91–114. Koenker, R., 2004. Quantile regression for longitudinal data. Journal of Multivariate Analysis 91, 74–89. Koenker, R., 2005. Quantile Regression. Cambridge University Press. Rouse, C., 1998. Private school vouchers and student achievement: an evaluation of the Milwaukee Parental Choice program. Quarterly Journal of Economics 553–602. Rubin, D., 1974. Estimating causal effects of treatments in randomized and non-randomized studies. Journal of Educational Psychology 66, 668–702. Witte, J., 1997. Achievements Effects of the Milwaukee Voucher Program, mimeo, University of Wisconsin. Witte, J., 2000. The Market Approach to Education: An Analysis of America's First Voucher Program. Princeton University Press.
Labour Economics 15 (2008) 575 – 590 www.elsevier.com/locate/econbase
Private school vouchers and student achievement: A fixed effects quantile regression evaluation☆ Carlos Lamarche ⁎ Department of Economics, University of Oklahoma, 321 Hester Hall, 729 Elm Avenue, Norman, OK 73019, United States Available online 3 May 2008
Abstract Fundamental to the recent debate over school choice is the issue of whether voucher programs actually improve students' academic achievement. Using newly developed quantile regression approaches, this paper investigates the distribution of achievement gains in the first school voucher program implemented in the US. We find that while high-performing students selected for the Milwaukee Parental Choice program had a positive, convexly increasing gain in mathematics, low-performing students had a nearly linear loss. However, the program seems to prevent low-performing students from having an even bigger loss experienced by students in the public schools. © 2008 Elsevier B.V. All rights reserved. JEL classification: I21; I28 Keywords: School choice; Vouchers; Milwaukee; Fixed effects; Quantile regression
“While we celebrate those [students] doing well, we can't turn a blind eye to those who are not” Rod Paige, US secretary of Education, to the NY Times. ☆
Version: March 31, 2008. This paper is based on Chapter 2 of my dissertation “Quantile Regression for Panel Data” at the University of Illinois at Urbana-Champaign. I especially thank my advisor Roger Koenker for advice and detailed comments. I am grateful to Dan Bernhardt, Greg Burge, Todd Elder, Lynn Gottschalk, Kevin Hallock, Kangoh Lee, Darren Lubotsky, as well as labor lunch participants at the University of Illinois at Urbana-Champaign and seminar participants at Texas Tech and the 2007 Annual Meetings of the European Association of Labour Economists. I thank Professor Cecilia Rouse for providing the Milwaukee Parental Choice program's data. I would like to thank the Guest Editor as well as two anonymous referees for their very helpful comments. ⁎ Tel.: +1 405 325 5857. E-mail address: [email protected]. 0927-5371/$ - see front matter © 2008 Elsevier B.V. All rights reserved. doi:10.1016/j.labeco.2008.04.007
576
C. Lamarche / Labour Economics 15 (2008) 575–590
1. Introduction Fundamental to the recent debate over school choice is the issue of whether voucher programs actually improve students' academic achievement, while decreasing inequalities between the best and worst students. Milton Friedman's proposal to use vouchers as a method of improving the quality of education is based on the idea that private schools are more productive than public schools, which is still a highly controversial issue. If private schools are in fact more efficient than public schools, governments can improve the quality of education by offering tuition vouchers to families that want to send their children to private schools. The Milwaukee Parental Choice program, the first program implemented in the US, has been providing vouchers to low-income students to attend private school since 1990. The simplicity of the program's idea contrasts sharply with the complexities that plague the program's evaluation. The voucher programs' effect, for example from time t to t', is the difference between what would have happened at time t' if the student was selected and remained in a choice school during the time interval, and what would have happened at time t' if the student was not selected, and remained in the Milwaukee public school. This counterfactual exercise is impossible to obtain using observational data (Rubin, 1974). It may be possible, however, to construct groups using a randomized experiment (e.g., children randomly assigned to attend choice schools and to attend public schools). At time t', the difference between students' academic performances could be attributed to the type of school since the initial assignment was random. However, the Milwaukee Parental Choice program was not implemented under idealized conditions (Witte, 2000), and therefore the selection of the control group plays a major role. Given the lack of a valid control group, it is not surprising that previous empirical studies have delivered mixed findings. Rouse's (1998) seminal study used a sample of students in the Milwaukee public schools as a comparison group and individual fixed effects to control for latent characteristics, such as more motivated parents or student abilities that may differ between treatment and comparison groups. The presented approach builds upon Rouse by employing a newly developed fixed effects form of quantile regression that not only controls for unobserved individual heterogeneity, but also allows an examination of the program effects at different points of the educational attainment distribution. The empirical literature (e.g., Witte, 1997; Green et al., 1997; Rouse, 1998) has focused upon estimating how the selection to attend the Milwaukee choice schools affects mean test scores. This approach to evaluation may be incomplete for policy analysis of programs serving heterogeneous students. To illustrate, if a stated policy goal is to raise students' achievement to a predetermined minimum standard, it may not be optimal to pursue a program that benefits strong students while causing weaker students to fall further behind. Because education is expected to play an important role in mitigating inequality, “the distribution of achievement gains... constitutes an appropriate criterion for evaluating a school choice intervention” (Howell and Peterson, 2002). This paper focuses on the estimation of the selection to attend choice schools on the entire distribution of test scores, considering patterns of achievement in terms of quantiles. There are important reasons why economists, educators, and policy makers are interested in how voucher programs affect students' achievement beyond the mean effect, which is typically estimated with Ordinary Least Squares (OLS) or Instrumental Variables (IV). First, the standard methodology may miss how a policy affects achievement differently at different points of the conditional test score distribution, as illustrated in Eide and Showalter (1998). Second, the possibility that vouchers may increase the differences between high- and low-performing students in the private schools, concern notably reflected in Ascher, Fruchter, and Berne (1996) question
C. Lamarche / Labour Economics 15 (2008) 575–590
577
“What mechanisms ensure those students [in private schools] who need extra time and attention... receive this more costly instruction?”. Our analysis shows that being selected to participate in the program had an heterogeneous effect on educational attainment, which had not been uncovered by the previous work focusing on mean effects. First, we find that the selected students had an increase in math achievements that ranges from 3.1 to 1.2 percentile points per year across quantiles. The program seems to dramatically improve the academic achievements of the weak students, having a relatively modest effect on achievements among strong students. Second, although the mean effect suggests that the program has no effect on reading, we find some evidence suggesting that the program improved and reduced the achievements of the low- and high-performing students, respectively. Using a new instrumental variable estimator for quantile regression (Chernozhukov and Hansen, 2005), we find that the effect of being enrolled in choice schools ranges from 2.3 percentile points per year at the 0.1 quantile of the conditional distribution of reading scores, to − 2.4 at the 0.9 quantile. This result is important for policy analysis since the Milwaukee vouchers seem to have no effect at the mean, but indeed matter at the tails of the educational attainment distribution. Lastly, our results also reveal that students' gains, measured as the differences between test scores conditional on years since application to the program, are subtle in nature. While the “average” student in the program had a linear gain in mathematics, high-performing students had a positive, convexly increasing gain, and low-performing students had a nearly linear loss. The offer of vouchers seems to increase the inequalities between low- and high-performing students. However, as we mentioned above, weak students' scores increased dramatically, in levels, compared to public school students' scores. Therefore, the evidence suggests that being selected for the choice program prevented low-performing students from having an even bigger loss experienced by low-performing students in the public schools. The next section briefly introduces a simple behavioral framework, and Section 3 presents models and estimators. Section 4 describes the data and Section 5 the empirical results. Section 6 offers conclusions. 2. A simple behavioral framework We develop a simple variation of a model of mothers' decisions to enroll children in preschool (Behrman et al., 2004).1 The mother of student i maximizes a time separable utility function that depends on consumption C and the child's test score T. There is a technology that produces test scores depending on socioeconomic characteristics x, attendance to a choice school P, and stochastic factors εj, T = T(x, P,εj). We assume that εj is realized after the student attends school. For example, dissatisfaction with teachers, quality of education, school transportation problems associated either with choice school j = 1 or public school j = 0 attendance. The variable actual attendance to a choice school P is equal to aS− 1, taking value one if the student actually attends a choice school, 0 otherwise. The variable attendance to a choice school a takes the value one if the mother decides that the student should attend a choice school, and the variable selected to attend choice school in period t − 1, S− 1, take the value one if the student was selected. Lastly, each period the mother must spend assets A on consumption and, if she decides to send her child to a choice school, on a fixed cost K. 1 The framework models heterogeneity, which is unobserved by the econometrician, that can be recovered using quantile regression techniques (Koenker, 2005). Alternative strategies are considered in Bitler et al. (2006), Hastings et al. (2006), and Cullen et al. (2005).
578
C. Lamarche / Labour Economics 15 (2008) 575–590
It is possible to express the mother's dynamic problem simply as maxa{υ0, υ1}, where υ0 represents the value of not attending a choice school, and υ1 denotes the value of attending a choice school. If at time t the student was not selected, her mother trivially chooses a = 0 because she does not want to give up consumption to pay K. On the other hand, if at time t the student was selected to attend a choice school, the decision is simply, a = 1{υ1 ≥ υ0|S− 1 = 1}. This decision depends, among other factors, on the cost of attending a choice school, dissatisfaction with the choice schools, and mother's weight to her child educational achievements. These factors may be different among families, and consequently the probability of attendance may be different as well. 3. Evaluation of the Milwaukee voucher program The previous behavioral framework impose two restrictions on the evaluation of the voucher program, Tit ¼ T ðxit ; Pit ; eit Þ ¼ xitVA þ bPit eit
ð1Þ
Pit ¼ 1 υ1it zυ0it ÞSit1 ¼ d þ qit Sit1 :
ð2Þ
The assumption of linearity in Eqs. (1) and (2) gives a direct link to Rouse's (1998) structural equation model. Rouse estimates a reduced form equation assuming that the effect of selection into the voucher program on the probability of attendance is constant among families (ρit = ρ), yielding an “intention-to-treat” effect π1 that is equal to ρβ. An enormous difficulty in the evaluation of the Milwaukee Parental Choice program is the lack of a valid control group. Initial randomized treatment is needed to obtain an unbiased estimator of the voucher offer, but the program's randomization was based on the applicant group.2 Green et al. (1997) used the students not selected to attend choice schools as a comparison group, but this group may lead to selection bias (Rouse, 1998; Witte, 2000). Witte (1997) used a random sample of students from public schools as a comparison group, but as Rouse (1998) pointed out, there may be differences between the students that were selected to attend choice schools and the students that remained in the public schools.3 Only eligible parents who are more interested in their child's achievements apply to the program, which suggests that students who remained in the public system may have parents with a less strong preference for private schools. To get around these problems, Rouse (1998) used both the unsuccessful applicants and a random sample of students from the public schools as comparison groups, and individual fixed effects to control for unobservables such us more motivated parents or student ability. Our approach builds upon Rouse, estimating a fixed effects version of the quantile regression model, QTit sj jSit1 ; xit ; ai ¼ p0 sj þ p1 sj Sit1 þ xitVp2 sj þ ai ; ð3Þ where Q(·|·) is the τj-th conditional quantile function, and αi is an individual fixed effect. The non-selected students that are omitted for simplicity in Eq. (3) will be introduced in the regression 2 The Wisconsin legislature required choice schools to select students at random when the grade was oversubscribed. Although the best schools received more applications, the total number of applications exceeded the number of seats (Witte, 2000). 3 For instance, consider a family making a decision as to whether or not to apply to the program based on υi0j = xi0 ′ μ + αji, where the binary variable j takes the value 1 if the student applies to the program, and the value 0 otherwise. From Eq. (2), the participation in the program may depend entirely on parent's weight to her child educational achievements, Pi0 = 1{υi01 − υi00 ≥ 0} = 1{α1i − α0i ≥ 0}.
C. Lamarche / Labour Economics 15 (2008) 575–590
579
models estimated in Section 5. If we consider the possibility that the effect of selection to attend choice schools on the probability of attendance ρit is a linear function of ωi (e.g., individual or family reasons to leave the choice schools) and εit (e.g., choice program or private school factors), it is straightforward to show that the intention-to-treat effect, p1 sj ¼ b q Fe1 sj ¼ QTit sj jSit1 ¼ 1; xit ; ai QTit sj jSit1 ¼ 0; xit ; ai : The model, in its simplest version, assumes that the effect of the voucher offer π1(τj) is decreasing on the quantile τ, and depends on the effect of the selection on the probability of attendance and the differential school effect on achievement. We briefly consider two competing explanations. First, we may consider that the effect is a combination of a treatment effect β and a location-scale shift effect ρ(τj) = ρ − Fε− 1(τj).4 We conjecture that while only high-motivated students among low-performing students stay in the choice schools, most students among the high-performing students stay. Since those lower-performing students in the private schools are relatively more motivated than the lower-performing students in the public schools, the ‘intentionto-treat effect’ should be decreasing in terms of quantiles. Alternatively, we can hypothesize that it is a combination of a scale treatment effect β(τj) = − βFε− 1(τj) and a location shift effect β × ρ. It has been argued that private schools may work on all students equally, while public schools that are interested in raising mean achievement may give less amount of attention to lower-performing students rather than higher-performing students (Howell and Peterson, 2002, p. 156). Although the model now suggests that the voucher program has a different mechanism to affect achievement, we again expect to see a decreasing π1 over the quantiles. The empirical evidence will offer below (see, e.g., Figs. 1 and 2) provides some support for the latter explanation. 3.1. Estimation We estimate the ‘intention-to-treat’ effect π1(τj) using a quantile regression version of the classical fixed effects estimator introduced by Koenker (2004), J T N n o X X X J b sj j¼1 ; f abi gNi¼1 u arg min xj qsj Tit p0 sj p1 sj Sit1 ai p p;a
j¼1
t¼1
i¼1
where ρτj(u) = u(τj − I(u ≤ 0)) is the quantile loss function, and ωj is the weight (e.g., 1/J) given to the jth quantile. For simplicity, the vector of independent variables xit, defined below, is omitted. Parents' and students' unobserved characteristics that may differ between students' selected to attend choice schools and students in the public schools are captured by individual fixed effects αi's, independent of the quantiles. We advance the method in two directions. First, we propose to use panel-bootstrap to estimate the precision of the estimates of the parameters of the model. Our strategy accommodates to forms of heteroscedasticity replacing pairs {(Ti, Si)} : i = 1,…, N} over cross-sectional units i. For small and fixed number of time series observations, the standard errors are consistent provided that the It seems natural then to explore the differences between the treatment effect β and the intention-to-treat effect βρ at different quantiles of the educational attainment distribution. We first estimated a reduced form model based on Eqs. (1) b1 ¼ b and (2) to obtain p b q. We then estimated Eq. (1) to obtain βb considering the instrumental variable approach described in Section 3.1. We found that the percentage change between the treatment effect and the intention-to-treat effect has a tendency to increase suggesting that the effect of selection on the probability of attending choice schools is decreasing in τ. 4
580
C. Lamarche / Labour Economics 15 (2008) 575–590
Fig. 1. Estimated effects of being selected for the choice program times years since application on educational attainment. The panels present quantile regression estimates (solid line with dots), classical fixed effects (dashed line), and .95% confidence intervals for the point estimates.
Fig. 2. Estimates of the treatment effect. The panels present instrumental variable quantile regression estimates (solid line with dots), classical instrumental variable estimates (dashed line), and .95% confidence intervals.
C. Lamarche / Labour Economics 15 (2008) 575–590
581
number of cross-sectional units passes to infinity. Second, we develop a simple framework for testing, described in Appendix A. We also implement an instrumental variable form of quantile regression to estimate the causal effect of the program. By accommodating the function T(·) and the error term, Eqs. (1) and (2)are the system of equations considered by Chernozhukov and Hansen (2005, 3.1), who propose an instrumental variable (IV) method for estimating quantile regression treatment effects. We estimate the effect of choice schools on test scores considering, QTit ðsjPit ; xit Þ ¼ gðsÞ þ bðsÞPit þ xitVAðsÞ
ð4Þ
where β(τ) is the parameter of interest. The vector xit includes “applicant pool” dummy variables, a dummy variable for gender, family income, an indicator if family income is missing, and the grade level of the student when the student took the test. Since the selection to attend choice school is exogenous conditional on the school and grade to which the student applied, we use whether the child was randomly selected to attend choice schools Sit − 1 as an instrument for actual enrollment Pit. 4. Data We analyze data from the Milwaukee Parental Choice program.5 We consider a sample that contains information on applicants to the program including the students that were not selected for the choice schools and students from the Milwaukee public schools as in Rouse's paper (Table 1). There is data on reading and math test scores, based on the normal curve equivalent measure (NCE) from the Iowa Tests Basic Skills (ITBS). The empirical analysis is based on a sample of African-American and Hispanic students who applied to the choice program between 1990 and 1993, and a sample of students from the Milwaukee public schools.6 Test scores: The Normal Curve Equivalent (NCE) measure is a transformation of the ITBS that produces an integer-level measure, ranging from 1 to 99, with a national mean of 50 and a standard deviation of 21. We rely on math and reading NCE test scores because (a) the results can be compared with previous paper's findings (e.g., Witte, 1997; Rouse, 1998), and (b) the NCE measure has the advantage that it can be averaged. According to test makers, the NCE test score is the same if all the students make one year of progress after one academic year. If some students make more (less) progress in one year than the students in the population of interest, the NCE test score will be higher (lower).7
5 The Milwaukee Parental Choice program was targeted and limited, aimed to provide better educational opportunities to low income families. The program target was poor families living in the city of Milwaukee whose children were not attending private school that year. Families with income, at most, 1.75 times the national poverty line were eligible to apply (e.g., the threshold for a family of three was $21,000). 6 We use Rouse's sample of public school students assuring the comparability of results. As a robustness check, we estimated the effect of the voucher offer considering public school students with income less or equal to 1.75 times the national poverty line in 1990. The main empirical findings of this paper are robust to restricting the control group to include only low income students. 7 One-point increase at the quantile τ may not be translated to the same amount of knowledge in different grades, but addressing this possibility is beyond the scope of this paper. (Note that this same possibility is present when analyzing mean effects). Rather, the present exercise is focused on an investigation of the changes in the educational attainment conditional distribution for the selected students and public school students. Furthermore, we analyze trends over several years to avoid drawing conclusions based on one-year changes that are likely to be noisy measures of performance (Kane and Staiger, 2002).
582
C. Lamarche / Labour Economics 15 (2008) 575–590
Table 1 Descriptive statistics for the main variables in Rouse's sample analysis Variables
Students Selected
Not selected
MPS
Proportion currently enrolled in a choice school Math (NCE) scores Reading (NCE) scores Proportion female Family income (in thousands of 1994 dollars) Proportion missing income family Proportion African-American Proportion Hispanic Grade level test Proportion with imputed math test (Rouse, 1998 methodology) Number of students Number of observations
0.558 (0.497) 39.236 (18.706) 37.647 (16.265) 0.536 (0.499) 12.052 (5.915) 0.412 (0.492) 0.795 (0.404) 0.205 (0.404) 3.767 (2.250) 0.074 (0.262)
0.010 (0.102) 38.108 (18.777) 37.886 (17.090) 0.470 (0.499) 12.510 (5.901) 0.542 (0.498) 0.864 (0.343) 0.136 (0.343) 3.857 (2.088) 0.194 (0.396)
– 40.411 (18.483) 38.700 (16.461) 0.523 (0.500) 21.750 (8.658) 0.747 (0.435) 0.868 (0.338) 0.132 (0.338) 4.337 (2.122) 0.164 (0.371)
986 2462
358 859
2014 5408
MPS stands for Milwaukee public schools.
Year since application: This variable is defined as the first year in which a student applied if he either was selected the first time he applied or he was never selected to attend choice school. In case a student applied two or more times, the first year she was accepted is considered the year of application. In the case of the students in the Milwaukee public schools, the year of application was imputed considering year t the “year of application” for students who have a valid year t + 1 test score (Rouse, 1998). Application lotteries: The probability of being selected to attend choice school is random conditional on the school and grade to which the student applied, because the students were randomly selected when the school was oversubscribed for a particular grade. Consequently, we need controls in model (4) for the school and grade to which the student applied (see, e.g., Rouse, 1998). The application lotteries are time-invariant dummy variables, therefore they will not be included in model (3) because the variables are subsumed in the individual effects. 5. Results We shall present the empirical results. Instead of the focus on the mean achievement, we will report the effect of being selected to attend choice schools on the entire distribution of educational attainment. 5.1. The heterogeneous impact We will estimate a somewhat more complicated model than Eq. (3) containing the selected students, the non-selected students, and the sample of students from the public schools. We also introduce interactions between the indicator for whether or not the student was selected to attend a choice school and the years since application because previous researchers have argued that child's educational achievement may not improve immediately. We consider first the simplest version, assuming that gains are equal from year-to-year, Tit ¼ p0 þ p1 ðdit Sit Þ þ p2 ðdit Nit Þ þ
4 X k¼3
p3k ditk þ p4 Sit þ p5 Nit þ ai þ eit
ð5Þ
C. Lamarche / Labour Economics 15 (2008) 575–590
583
Table 2 Estimates of the selection to participate in the choice program on Math and Reading scores estimates for the full sample Quantiles 0.1
0.25
0.5
0.75
0.9
Mean
Fixed effects methods Dependent variable = math (NCE) test score Selected to attend choice school Not selected to attend choice school Selected × number of years Not selected × number of years
− 3.649 (1.866) −4.270 (1.429) − 3.321 (1.167) − 2.288 (1.192) −1.173 (1.633) − 3.391 (1.114) − 2.758 (3.192) −5.026 (2.109) − 3.621 (2.170) − 2.769 (2.150) −1.326 (2.707) − 3.052 (1.741) 2.323 (0.794)
3.143 (0.608)
2.321 (0.454)
1.685 (0.477)
1.178 (0.669)
2.294 (0.399)
0.919 (1.643)
1.681 (1.122)
1.526 (1.138)
0.844 (0.960)
−0.675 (1.026) 0.672 (0.783)
1.400 (1.169)
2.780 (1.412)
0.750 (1.027)
− 2.140 (3.479) −3.200 (2.070) − 0.555 (2.161) 1.350 (2.008)
2.605 (2.410)
− 0.644 (1.584)
Fixed effects methods Dependent variable = reading (NCE) test score Selected to attend choice school Not selected to attend choice school Selected × number of years Not selected × number of years
− 0.320 (1.625) −1.390 (1.372) 0.180 (1.254)
0.320 (0.687)
0.690 (0.630)
− 0.120 (0.541) − 0.700 (0.472) −1.160 (0.635) − 0.249 (0.368)
0.440 (1.555)
1.500 (1.069)
0.555 (0.898)
− 0.350 (0.813) −0.985 (1.015) 0.247 (0.706)
Math regressions include a dummy variable for whether the test score was imputed. The total number of observations are 8729 (Math) and 8751 (Reading). The standard errors (in parenthesis) are obtained after 1000 panel-bootstrap replications.
where dit − k indicates the number of years pre- or post-application to the program, (dit × Sit) is an interaction between the number of years since application and whether the student was selected to attend choice schools, and Nit is a dummy variable indicating whether the student applied to the program and was not accepted. 5.1.1. Fixed effects estimates We report estimates for the full sample in Table 2. First, the table shows negative estimates for the effect of being selected into a choice school, suggesting that the selected students had lower math test scores than the students in the public schools when they entered the program.8 Although we cannot provide a definitive explanation, it may be possible that program's applicants were not doing well in the public schools. Witte (2000) presents evidence on parental dissatisfaction with their kids' prior school within the Milwaukee public schools finding that (a) choice students seem to have less satisfied parents, and (b) the level of parents' dissatisfaction is statistically different between the choice students and the students in the public schools. He also argues that attitudes of parents toward their kids' prior school may be associated to their children's academic
8
Research had indicated that choice students have low test scores relative to low-income public school students (see, e.g., Witte, 2000), but the literature is relatively silent on explaining why the academic achievement of these students was low when they entered the program. As shown in Table 2, Rouse's (1998) estimated mean effect of being selected into the program on math test scores is negative and significant.
584
C. Lamarche / Labour Economics 15 (2008) 575–590
performance. Note that our hypothesis is also supported by the finding of negative and significant effects mainly among low-performers. The least squares fixed effect estimates suggests that the student selected for a choice school earn 2.3 additional percentile points per year in math relative to the student in the public schools. If instead of focusing on the mean effect, we consider various quantiles of the math score distribution, we see that the gains have some tendency to decrease as we go across the quantiles. For instance, the estimate changes from 3.1 additional percentile points per year at the 0.25 quantile to 1.2 points at the 0.9 quantile.9 Looking at the estimates of the gain in reading, the mean regression suggests that the students selected to attend choice schools did not improve relative to the students in the public schools. We get a bit more detail about the impact of the program by taking a look at various quantiles. Again, we see that there is a tendency to decrease as long as the quantile increases; from 0.7 additional percentile points per year at 0.25 quantile to − 1.2 additional percentile points per year at the 0.9 quantile. The program has a modest effect on the low-performing students, and a negative effect on the high-performing students, though the effects are insignificant at standard levels. Fig. 1 presents the effect of the selection to attend choice schools times years since application as a function of the quantiles of the conditional distribution of test scores. We observe that while the conditional mean effect seems to provide an incomplete summary of the effect of being selected to attend choice school on educational attainment, the quantile intention-to-treat effect provides a more detailed assessment of the gains among students. The left panel suggests that being selected to attend choice school has a positive effect on the lower tail of the math score distribution, therefore the program seems to improve the academic achievements of the low-performing students. For example, the students who did worse than the 0.3 quantile and were selected to attend choice school scored approximately 3.75 additional percentile points per year compared with the students who remained in the public school system. But, there is a modest effect on math and an adverse effect on reading in the right tail, not changing and reducing the achievements of the high-performing students. For instance, the students selected to the choice program earn approximately −1 percentile points per year relative to the students in the public schools at the 0.8 quantile of the reading score distribution.10 Earlier we noted that the voucher offer has a dramatic effect on low-performing students, which could be attributed to several channels. It may be simply argued that private schools are better than public schools, although the mechanism seems to be more complex. Another possible channel through which the program increased dramatically the achievement of the low-performing students in private schools could be that the program attracted the brightest students of the public schools. Since lower-performing students in the choice schools may be relatively more motivated than lowerperforming students in the public schools, we expect to see a dramatic effect in the lower tail of the educational attainment distribution. Finally, public schools may give less amount of attention to lower-performing students because they are evaluated on the basis of average test scores. 9 We evaluate the sensitivity of the results considering the 1991–1993 cohorts and a sample that excludes 1994 test scores for the 1990 cohort. Fixed effects quantile regression estimates again suggest a significant gain in mathematics with a tendency to decrease as we go across the quantiles. The results are available upon request. 10 We also estimated the achievement gains among the students selected to attend choice school, dividing this group of students in (a) those who are actually enrolled and (b) those who are not. We find, once again, both(i) significant effects in the lower tail of the math test score distribution, and (ii) a decreasing voucher effect after the 0.25 quantile. We also find that that the students who were selected to attend choice and remained in the program earn more percentile points than the students who were selected but dropped the program. For example, at the median of the math score distribution, the students who are enrolled in choice schools earn an increment of 2.9 percentile points per year, but the students who dropped choice schools earn just 1.3 percentile points. These results are available upon request.
C. Lamarche / Labour Economics 15 (2008) 575–590
585
5.1.2. IV estimates Table 3 presents IV estimates of the effect of enrollment on educational achievement. The least squares estimates suggest that the students enrolled in choice schools earn 1.7 additional percentile points per year in mathematics relative to the students enrolled in the public schools. Looking at the various quantiles of the math score distribution, we observe that the estimated effects are positive and different across quantiles. The estimated causal effect on math scores is positive and large for low-performing students, but small and statistically indistinguishable from zero for high-performing students. Note also that the mean effect suggests that the program has a negative, insignificant effect on reading. But this is an incomplete, possibly wrong, conclusion because there is a tendency to decrease as we go across the quantiles. The gain ranges from 2.3 additional percentile points per year at the 0.1 quantile to − 2.4 additional percentile points per year at the 0.9 quantile. The mean effect clearly misses relevant information for policy evaluation. Looking at the estimated effect presented in Fig. 2, we observe an heterogeneous impact that decreases as the quantile increases. The panels suggest that the Milwaukee Parental Choice program has a positive effect on the lower tail of math and reading conditional distributions, but no effect on math, and an adverse effect on reading in the right tail. The right panel suggests that the program improved and reduced reading achievements of the lower and higher attainment students, respectively. While students who did worse than the 0.3 quantile earned approximately 2 additional percentile points per year relative to the students in the public schools, the students who did better than the 0.7 quantile scored approximately − 2 percentile points per year. 5.2. Achievement gains In this section, we estimate the distribution of achievement gains considering, Tit ¼ p0 þ
4 X
p1k ðditk Sitk Þ þ
k¼1
4 X
p2k ðditk Nitk Þ þ
k¼1
4 X
p3k ditk þ ai þ eit
ð6Þ
k¼3
where, as before, dit − k is defined as before, Sit − k indicates whether the student was selected to attend a choice school, and Nit − k indicates whether the student was not selected. Considering a random sample of Milwaukee public schools students as a comparison group, we employ the fixed Table 3 Instrumental variables estimates of the causal effect of choice schools on math and reading test scores Quantiles 0.1 IV Methods
0.25
0.5
0.75
Currently enrolled 2.331 (1.701) 3.688 (1.399) 2.766 (1.090) 0.182 (1.206) in choice school IV Methods
0.9
Mean
0.605 (1.901)
1.701 (0.926)
Dependent variable = math (NCE) test score
Dependent variable = reading (NCE) test score
Currently enrolled 2.308 (1.378) 2.293 (1.080) 0.556 (0.971) −1.738 (1.126) − 2.417 (1.288) − 0.103 (0.810) in choice school Math regressions include a dummy variable for whether the test score was imputed. The total number of observations are 3177 (Math) and 3163 (Reading). The standard errors (in parenthesis) are obtained after 1000 panel-bootstrap replications.
586
C. Lamarche / Labour Economics 15 (2008) 575–590
effects method to estimate the panel data-quantile regression model, QTit (τj|dit − k, Sit − k, Nit − k, αi), evaluated by years since application to the program. bT (τj|·), and Table 4 shows coefficient estimates from the τj-th conditional quantile function Q it b (Tit|·). The students selected to attend choice schools had a roughly the conditional mean function E linear gain in mathematics, scoring approximately 2 additional points after four years (from 39.6 to 41.7 points). Part of the increase is explained by the performance of the students in the public schools, who had a negative 4.9 point gain, going from 40.4 to 35.5 points. We also see that lowperforming students in the public schools have a significant loss, while high-performing students have a negligible loss after four years. The relatively large and positive effect in the lower tail observed in Fig. 1 is therefore partially explained by the poor performance of the weakest students in the Milwaukee public schools. The table also shows reading scores, suggesting that there are no differences among the groups of students in the lower and upper tails of the score distribution. Although we cannot provide a definitive explanation for the poor performance of the public school students, two possibilities come to mind. First, the observed trend may be part of a decades-long trend. Witte (2000) presents historical data on achievement documenting that a decline of the public scores relative to the national norm occurred in the early 1970's. Second, the test scores could have been declining after the implementation of the voucher program due to the natural instability of the public school environment. In the first years, the changes in the educational institutions could affect the achievements of the students who needed more attention. The drop in the estimated public school test scores in the first years has been previously documented in charter schools in Washington D.C., Texas, Michigan, and in the Milwaukee voucher program at the mean level (see, e.g., Howell and Peterson, 2002; Rouse, 1998). Fig. 3 provides an efficient way to summarize the effect of being selected to attend choice school on achievement gains measured as differences in test scores levels. Rouse (1998) found that there is a linear gain in mathematics for students selected to attend choice schools (middle Table 4 Adjusted math and reading (NCE) test scores by year after application to the choice program Years since application to the program
Quantiles 0.1
0.25
0.5
0.75
0.9
Mean
Selected to attend choice schools Math (NCE) test score One year Four years
29.23 (1.25) 35.91 (0.96) 39.00 (0.76) 42.12 (0.79) 48.12 (1.03) 39.65 (0.73) 28.00 (2.88) 36.01 (2.06) 42.00 (1.46) 47.12 (1.34) 51.91 (1.98) 41.73 (1.28)
Milwaukee public school sample Math (NCE) test score One year Four years
31.91 (0.70) 37.00 (0.61) 40.00 (0.52) 42.73 (0.58) 47.91 (0.78) 40.39 (0.37) 22.51 (1.21) 27.05 (0.93) 35.00 (1.07) 42.91 (0.94) 46.70 (1.30) 35.53 (0.72)
Selected to attend choice schools Reading (NCE) test score One year Four years
29.00 (1.12) 35.00 (1.01) 39.00 (0.93) 43.00 (0.97) 48.00 (1.02) 39.04 (0.67) 22.00 (2.03) 29.00 (1.92) 36.00 (1.67) 41.00 (1.24) 43.00 (1.47) 35.06 (1.17)
Milwaukee public school sample Reading (NCE) test score One year Four years
29.00 (0.83) 35.00 (0.67) 39.00 (0.62) 42.00 (0.65) 46.00 (0.75) 38.39 (0.34) 23.00 (0.97) 29.00 (0.93) 36.00 (0.95) 42.00 (0.85) 46.00 (1.07) 35.62 (0.57)
The standard errors (in parenthesis) are obtained after 1000 panel-bootstrap replications.
C. Lamarche / Labour Economics 15 (2008) 575–590
Fig. 3. Adjusted math scores by years since application to the choice program. CS denotes choice schools.
587
588
C. Lamarche / Labour Economics 15 (2008) 575–590
Table 5 Tests for the equality of the adjusted test scores Years after application
Quantiles 0.1
0.25
0.5
0.75
0.9
0.178 0.054 0.000 0.000 0.000 0.000 0.000 0.000
0.001 0.000 0.001 0.848 0.000 0.002 0.000 0.009
0.287 0.064 0.021 0.380 0.053 0.141 0.021 0.024
0.302 0.132 0.383 1.000 0.397 0.502 0.334 0.540
0.015 0.003 0.581 1.000 0.080 0.029 0.090 0.087
Group of students
H0
Math (NCE) test score
Selected to attend choice schools
I II I II I II I II
0.926 0.679 0.000 0.000 0.000 0.000 0.000 0.000
Group of students
H0
Reading (NCE) test score
Selected to attend choice schools
I II I II I II I II
0.014 0.002 0.000 0.000 0.000 0.000 0.000 0.000
Milwaukee public school sample Selected and Milwaukee public school sample All groups
Milwaukee public school sample Selected and Milwaukee public school sample All groups
0.868 0.965 0.000 0.000 0.000 0.000 0.000 0.000
0.009 0.003 0.000 0.000 0.000 0.000 0.000 0.000
0.176 0.078 0.000 0.000 0.002 0.001 0.004 0.005
Hypothesis I considers the equality of coefficients across years since application. Hypothesis II considers the equality of coefficients between the first and fourth year since application. p-values are reported.
panel). We get more detail about the effect of being selected to attend choice schools over time by focusing on the math scores for the weakest and strongest students. While high-performing students have a positive, convexly increasing gain, low-performing students have a nearly linear loss. This evidence suggests the possibility that the Milwaukee vouchers increased the differences between high- and low-performing students in private schools. At first glance, one may find the estimates in Table 4 contradicting those presented in Table 2. The weak students selected to attend choice schools had roughly a point loss in math after four years since application to the program, but we concluded that these students were benefited by the voucher offer. This paradox is easily explained by the fact that we are now relaxing the assumption of equal gains over the years, considering differences in conditional test score levels instead of levels. We offer a simple numerical exercise to see the point. The average difference between the continuous line and the dashed line in the panels of Fig. 3 should be positive and decreasing beyond the 0.25 quantile as in Fig. 1. We find differences equal to {1.76, 3.66, 2.52, 1.43, 1.47} points at the {0.1, 0.25, 0.5, 0.75, 0.9} quantiles. Therefore, although low-performers in the choice schools were getting behind relative to the national norm after four years since application to the program, these students had significant gains relative to their counterfactual group. Finally, we test if the gains are significantly different considering the framework for hypothesis testing described in Appendix A. We test two hypotheses. First, we check whether the adjusted test scores in Table 4 are significantly different across several quantiles. We found that these differences were statistically significant across the quantiles of the math and reading score distributions (p-values were 0.000).11 Second, we test the equality of test scores across several 11
The results are available by the author upon request.
C. Lamarche / Labour Economics 15 (2008) 575–590
589
years since application to the program (Table 5). We find that (a) math scores are statistically different among groups, and (b) the difference between the scores in the first and fourth year of application for high-performing students selected to the program is significant at the standard 10% level (p-value equal to 0.064). Lastly, not surprisingly, there is weaker evidence that predicted reading scores are different across the years since application to the program. 6. Conclusions This paper investigates the distribution of achievement gains in the Milwaukee Parental Choice program. First, we find that the program seems to dramatically improve the academic achievements of weak students, having a relatively modest effect on achievements among strong students. Second, although the mean effect suggests that the program has no effect on reading, our results suggest that the program improved and reduced the achievements of the lower and higher-performing students, respectively. Third, the results also reveal that students' gains are subtle in nature. While the “average” student in the program had a linear gain in mathematics, we find that high attainment students had a positive, convexly increasing gain, and low attainment students had a nearly linear loss. Because education is expected to play an important role in mitigating inequality, the estimation of patterns of achievement in terms of quantiles may lead to a richer debate on school choice, a debate beyond the educational attainment of the “average” low-income student. Appendix A. General linear hypothesis We consider a general linear hypothesis on the vector ξ of the form H0 : Rξ = r, where R is a matrix that depends on the type of restrictions imposed. We evaluate the significance of differences across coefficient estimates from model (6) at different quantiles, and the significance of differences across coefficient estimates at different years since application on a vector with typical element, xkj ¼ QkT sj ; where QTk (τj) denotes the coefficients from the τj-th conditional quantile function k years after application to the choice program. For example, when we test whether the adjusted test score is significantly different across several quantiles one year after application, the vector is ξj1 = (QT1 (0.1),…, QT1(0.9))′, and when we test the equality of the adjusted test scores at the 0.1 quantile across several years since application, ξ1k = (QT1 (0.1),…, QT4 (0.1))′. The test statistics, h i1 b 1 R V R xb r T NT ¼ NT R xb r V R V b is the is asymptotically distributed as χq2 under H0, where q is the rank of the matrix R and V estimated covariance matrix. References Ascher, C., Fruchter, N., Berne, R., 1996. Hard Lessons: Public Schools and Privatization. Twentieth Century Fund Press, New York. Behrman, J., Cheng, Y., Todd, P., 2004. Evaluating preschool programs when length of exposure to the program varies: a nonparametric approach. Review of Economics and Statistics 86 (1), 108–132. Bitler, M., Gelbach, J., Hoynes, H., 2006. What means impact miss: distributional effects of welfare reform experiments. American Economic Review 96 (4), 988–1012.
590
C. Lamarche / Labour Economics 15 (2008) 575–590
Chernozhukov, V., Hansen, C., 2005. An IV model of quantile treatment effects. Econometrica 73 (1), 245–262. Cullen, J., Jacob, B., Levitt, S.D., 2005. The impact of school choice on student outcomes: an analysis of the Chicago public schools. Journal of Public Economics 89 (5–6), 729–760. Eide, E., Showalter, M., 1998. The effect of school quality on student performance: a quantile regression approach. Economic Letters 58, 345–350. Green, J.P., Peterson, P., Du, J., 1997. The effectiveness of school choice: the Milwaukee experiment. Harvard University Education Policy Paper 97-1. Hastings, J., Kane, T., Staiger, D., 2006. Preferences and heterogeneous treatment effects in a public school choice lottery. NBER Working Paper, W12145. Howell, W., Peterson, P., with Wolf, P., Campbell, D., 2002. The Education Gap: Voucher and Urban Schools Brookings Institution Press, Washington, D.C. Kane, T., Staiger, D., 2002. The promise and pitfalls of using imprecise school accountability measures. Journal of Economic Perspectives 16 (4), 91–114. Koenker, R., 2004. Quantile regression for longitudinal data. Journal of Multivariate Analysis 91, 74–89. Koenker, R., 2005. Quantile Regression. Cambridge University Press. Rouse, C., 1998. Private school vouchers and student achievement: an evaluation of the Milwaukee Parental Choice program. Quarterly Journal of Economics 553–602. Rubin, D., 1974. Estimating causal effects of treatments in randomized and non-randomized studies. Journal of Educational Psychology 66, 668–702. Witte, J., 1997. Achievements Effects of the Milwaukee Voucher Program, mimeo, University of Wisconsin. Witte, J., 2000. The Market Approach to Education: An Analysis of America's First Voucher Program. Princeton University Press.