- Email: [email protected]
The effect of community college enrollment on bachelor's degree completion
Economics of Education Review 28 (2009) 199–206
Contents lists available at ScienceDirect
Economics of Education Review journal homepage: www.elsevier.com/locate/econedurev
The effect of community college enrollment on bachelor’s degree completion William R. Doyle ∗ Peabody College of Vanderbilt University, Peabody #514, 230 Appleton Place, Nashville, TN 37203-5721, United States
a r t i c l e
i n f o
Article history: Received 13 September 2006 Accepted 18 January 2008 JEL classification: I23 I28 C14 C41 Keywords: Efficiency Costs Higher education Community colleges
a b s t r a c t Rouse [Rouse, C. E. (1995). Democratization or diversion—the effect of community-colleges on educational-attainment. Journal of Business and Economic Statistics, 13(2), 217–224] finds that enrollment in a community college may divert students from attaining a bachelor’s degree. However, this result may be due to selection bias, as the population of community college students should be quite different from those who attend 4-year institutions in terms of both observable and unobservable characteristics. This study uses propensity score matching to non-parametrically balance a data set from the 1996 Beginning Postsecondary Students survey in order to overcome issues associated with selection bias. Results from a Cox proportional hazards model indicate that attendance at a community college lowers the hazard rate for completing a bachelor’s degree. The results are consistent with previous studies. © 2008 Elsevier Ltd. All rights reserved.
1. Introduction Nearly 80% of first time students who enter community colleges intend to complete a bachelor’s degree (United States Department of Education, 2005). While community colleges have a number of functions, it appears that most entering students intend to use them as a stepping stone to eventual bachelor’s degree completion. The vast majority of students who begin their postsecondary education in community colleges do not go on to attain a bachelor’s degree. Of those who begin at a community college with the intention of completing a bachelor’s degree, only 21% do so within 6 years (United States Department of Education, 2005). Many have suggested that community colleges actually function to push students away from their educational goals (Brint & Karabel, 1989;
∗ Tel.: +1 615 322 3904; fax: +1 615 343 7094. E-mail address: [email protected]. 0272-7757/$ – see front matter © 2008 Elsevier Ltd. All rights reserved. doi:10.1016/j.econedurev.2008.01.006
Clark, 1960). Rouse (1995) calls this the “diversion” effect of community colleges. On the other hand, many students who enroll at a community college would not otherwise have attended higher education, and some fraction of these students do go on to attain a bachelor’s degree. This is termed by Rouse the “democratization” effect of community colleges (Rouse, 1995). This debate has become quite important for policy purposes. Community colleges charge lower tuition and have lower per-student costs than their 4-year counterparts. Per student expenditures at public 4-year institutions average $9183 nationally, while per student expenditures at public 4-year institutions average $27,973 (United States Department of Education, 2004). The savings to both the student and the state are substantial when a student attends a community college. If students who begin at a community college are just as likely to complete a bachelor’s degree, then there is no reason for state policymakers not to encourage students
200
W.R. Doyle / Economics of Education Review 28 (2009) 199–206
to begin at such an institution. However, if students who begin at a community college are much less likely to attain a degree, then policymakers and institutional leaders would need to undertake a more serious examination of policies and practices to understand why exactly students who might have completed a bachelor’s degree were indeed diverted from their goals. Students who begin their postsecondary studies in community colleges are quite different from their counterparts who begin at a 4-year institution. These differences extend from observable features such as academic preparation and family background to possible unobservable features such as motivation or ability. Previous studies have dealt with this problem in various ways, either by using an instrumental variables approach to recover a quasi-experimental design or by using a selection on observables approach (Leigh & Gill, 2003; Rouse, 1995). This paper extends the selection on observables approach to balance the sample of students between a control (4-year enrollment) and a treatment (2-year enrollment) group. I am interested primarily in the question of diversion, that is, does starting at a community college lower the likelihood that a student will complete their desired level of schooling in a certain amount of time? To address the problem of selection bias in the estimation of treatment effects, I utilize propensity score matching in order to non-parametrically pre-process data. In order to accurately model the inherently timedependent process of degree attainment, I employ a Cox proportional hazards model (Kalbfleisch & Prentice, 1980; Singer & Willett, 2003). In using this approach, I am able to compare the experience of two very similar groups of students, one group which started at a community college and another group which began at a 4-year college. The educational outcomes of both groups after 6 years are compared. I find that the hazard rate for completion is lower for community college matriculants in both the full and matched sample.
Rouse (1995) used an instrumental variables approach to estimate the causal effect of community college attendance on bachelor’s degree attainment. Rouse uses two stage least squares to model the effect of community college attendance on both years of schooling and bachelor’s degree attainment. Using a sample from the High School and beyond dataset, Rouse finds a statistically significant negative coefficient on community college attendance when the dependent variable is years of attainment, with an effect of 0.6–1 fewer years of educational attainment for those who began at a community college. Rouse’s study has several shortcomings, most notably that it does not condition on students’ intentions. Many students enter community colleges without intending to attain a bachelor’s degree. Leigh and Gill (2003) make use of National Longitudinal Study of Youth data in order to condition on students’ expectations. Again using years of schooling as the dependent variable, they find that students who begin at a community college with the intention of receiving a bachelor’s degree end up with 0.7–0.8 fewer years of schooling than their peers. The authors of both studies acknowledge that they cannot account for many of the unobservable characteristics of students who enter 2-year as opposed to four colleges, a problem that could only be resolved through random assignment to institutions. However, this paper builds on these studies by matching the distributions of observable characteristics of students at community colleges with the distributions of the same characteristics of their peers. Several studies have examined the impact of institutional and state policies on bachelor’s degree completion. However, these studies have limited their samples to matriculants at 4-year colleges only (DesJardins, Ahlburg, & McCall, 1999). Adelman (1999) suggests that entrance into a 4-year college is the only reliable indicator of a student’s intention to complete a bachelor’s degree. While this may be reflective of the data, it does little to suggest whether community colleges can help students complete their educational objectives.
2. Previous studies The issue of students leaving college has long been of central concern to many researchers in the field of higher education. Why a student would not complete college is clearly an important policy issue at the institutional, state and federal level. One strand of the literature focuses primarily on the institution and the personal and environmental factors that may help or hinder students as they seek to complete their degrees (Braxton, Milem, & Sullivan, 2000; Tinto, 1975). A second strand of the literature, to which this study belongs, looks at the policy impact of differing institutional types on the eventual completion of students. I briefly review a few of the second type of study here. Two notable previous studies have looked at the issues of democratization of diversion in community colleges. The first, Rouse (1995) was the first to specify a model for democratization or diversion. The second, Leigh and Gill (2003) updates and refines Rouse’s model, using data that measure the intentions of students prior to entering higher education.
3. Functional form and student departure Traditional models of student departure have not taken into account that this is a process that is time-dependent, and therefore most appropriately modeled using event history analysis. This section briefly describes why the Cox proportional hazard model is well suited to modeling the hazard rate for graduation from college. Many authors1 have specified student departure from higher education as a standard a binary outcome, and model the departure process using standard logit or probit models. This type of model does not take into account the time-varying nature of the data, instead assuming that the hazard rate for graduation is constant over time. As DesJardins et al. (1999) point out, this is in many ways an unsatisfying approach to modeling event history data, since it assumes that in each year since matricula-
1
See for example Adelman (1999).
W.R. Doyle / Economics of Education Review 28 (2009) 199–206
tion, the overall probability of graduation is the same.2 The DesJardins et al. (1999) approach makes use of a prespecified form of the hazard rate to understand the effect of covariates on the binary dependent variable. An alternative form provided by Cox (1972) provides a more flexible functional form that does not rely on a pre-specified shape of the hazard curve. In the Cox proportional hazards model the non-parametric hazard rate is increased or decreased through multiplication by a scalar function of covariates (Therneau & Grambsch, 2001).
To proceed, an assumption of conditional independence must be made. That is, we need to assume that the information contained in x is sufficient to make the counterfactual outcome y0 independent of z (Smith & Todd, 2001): y0 ⊥ z|x Matching requires an assumption of common support, that is p(z = 1|x) < 1
3.1. Propensity score matching and causal inference Given the above-specified functional form for the process of student departure, I now turn to the use of propensity score matching to recover causal estimates of the effect of community college enrollment on the hazard rate for individuals’ attainment of a bachelor’s degree. All previous studies have dealt with a common issue: an experimental approach can yield causal estimates, but are infeasible in determining choice of college. In her analysis, Rouse (1995) employs an instrumental variable approach and finds a negative but marginally statistically significant effect of community colleges on attainment. Rouse addresses issues of exogeneity of these instrumental variables in her work, and finds “to the extent that observed measures of family background also reflect unobserved determinants of educational attainment, there is evidence that the instruments may be correlated with the error term in the educational attainment equation” (Rouse, 1995, p. 223). A different approach to this problem is selection on observables. This is the approach utilized by Leigh and Gill (2003). The selection on observables approach suggest that the conditioning variables contain sufficient information to provide a linear extrapolation of the effect of starting college on attainment for all types of students (footnote 2). Their results are consistent with Rouse (1995). Another approach to the problem of endogeneity of treatment is possible. Matching is in many ways a nonparametric form of selection on observables. In this study I divide students into two different groups, depending on whether the student in question began at a 2-year or 4year college. Group y1 denotes those students who were “treated”, that is, those students who began at a 2-year college. Group y0 denotes those students who began at a 4-year college (or, more neutrally, did not begin at a 2-year college). Student’s characteristics are described by a vector of covariates x and z, a binary variable equal to 1 if a student begins at a 2-year college and equal to 0 otherwise. What this paper seeks to establish is the effect of attending a 2-year college on those who did, in fact, attend a 2-year college—this is known as treatment on the treated (Smith & Todd, 2001): tt = E(y1 |x, z = 1) − E(y0 |x, z = 1)
2 Leigh and Gill (2003) do not directly address the same problem as this paper and the Rouse paper, instead focusing primarily on conditioning results on desired levels of schooling.
201
for all x
In our example, this would require that for all of the variables supposed to affect selection into the treatment groups, there is an area of common support among all groups in both the control and treatment group. However, instead of utilizing exact matching, it is also possible to match on x using the probability of treatment p = Pr(z = 1|x) (Heckman, Ichimura, & Todd, 1998). Probability of treatment can be estimated using standard models for binary outcomes, with treatment as the outcome in question. The predicted value for each individual is called the propensity score for treatment (Rosenbaum & Rubin, 1983a). In propensity score matching a vector of covariates x, is used to identify the propensity score pi for all individuals for attending a 2-year institution (Rosenbaum & Rubin, 1983a). Matching is accomplished by looking for students in the control group who are sufficiently similar to students in the treatment group in terms of this propensity score. The propensity score estimate of treatment on the treated is estimated as tt = E(y1 |x, z = 1) − E(y0 |x, z = 1) = E(E[y1 |p, z = 1] − E(y0 |p, z = 0]z = 1) In the case of this study the treatment on the treatment indicator will not be the mean differences in groups but the differences in proportional hazards between individuals who begin at community colleges and those who begin at 4-year institutions. Heckman, Ichimura, and Todd (1997) review the essential features necessary to reduce bias in evaluation studies and summarize them as (1) the treatment and control groups have identically distributed unobserved attributes; (2) observed attributes are identically distributed; (3) both treatment and control groups are given the same questionnaire; (4) treatment and control groups are in the same economic (or social) environment. Heckman et al. (1997) suggest that features (2–4) are much more important than has been previously suggested, and feature (1), selection bias, “is a relatively small part of bias as conventionally measured” (Heckman et al., 1997, p. 606). Features (2) and (3) are met quite easily in this study: propensity score matching will ensure identical distribution of observed characteristics, and all studied individuals completed a common questionnaire. Feature (4) common environments, is not met, since states vary considerably in the number of community colleges, and likelihood of attendance is predicated in part on proximity to a com-
202
W.R. Doyle / Economics of Education Review 28 (2009) 199–206
Table 1 Differences in means for complete sample, selected matching variables
Propensity score Composite test score Parental income Father’s education (range: 1 = less than H.S., 11 = doctorate) High school GPA (range: 1 = mostly A’s, 7 = mostly D’s)
Means treated
Means control
S.D.
t-Stat
Bias
0.54 755.99 42509.35 4.52 4.01
0.05 952.53 59503.61 5.78 3.56
0.2 217.03 57752.97 3.24 1.66
48.79 −26.21 −7.99 −10.09 5.85
1.93 −1.1 −0.33 −0.42 0.24
munity college (Rouse, 1995, 1998). Feature (1) should be minimized, but as always with observational studies, there can be no guarantee that treatments and controls do not have different distributions of unobserved characteristics.
4. Data This analysis makes use of data drawn from the National Center for Education Statistic’s Beginning Postsecondary Students study. Students in this study were identified at their initial institution of postsecondary enrollment through the National Postsecondary Student Aid Survey in 1996. Students were then followed through a series of follow up interviews in 2001 (United States Department of Education, 2002). The dependent variable in the study reflects the amount of time that the student took to complete a bachelor’s degree, measured in months. The data for this study are by necessity right-censored, since individuals who did not complete a bachelor’s degree may go on to do so in the future. The key independent variable of interest is a binary variable for community college entrance, coded to 1 if the respondent began her college career in a community college and coded to 0 otherwise. Five independent variables are used as controls. These are drawn from the relevant literature as important predictors of student success in completing a bachelor’s degree. These variables are defined as follows: • Age: student’s age in years at college entrance • Dependent: binary variable, coded to 1 if student is a dependent, 0 otherwise; • Hours worked: number of hours worked per week in academic year 1995–1996; • Full time: binary variable, coded to 1 if student enrolled full time at first institution; • Live on campus: binary variable, coded to 1 if student lived on campus at first institution. 5. Results for full sample Results for the Cox proportional hazard model for the full sample are available in a separate appendix on the author’s website. The results indicate that, after controlling for the above variables, beginning at a community college proportionally reduces the hazard rate for college completion by a factor of 0.7, with a 95% confidence interval bounded by [0.58, 0.86].
6. Matching for non-parametric pre-processing To obtain a better estimate for causal inference, this paper makes use of matching for non-parametric preprocessing, following the procedures suggested by Ho, Imai, King, and Stuart (2005). The computation for this paper was done in R, using the package Matchit described by Ho et al. (2005). To calculate the propensity score for each unit, I utilized a probit model for community college entrance using a total of 40 different characteristics of individuals, divided into four broad groupings: family characteristics (e.g. income, parent’s education), high school characteristics (e.g. percent free lunch, highest level of math offered) student information (e.g. delayed entry, gender, race), attitudinal data (e.g. what factors affected their choice of college) and financial aid and college cost data (tuition, aid from various sources). A summary of these variables, their coefficients in the matching model, and their means in both control and treatment groups in the full and matched sample is available upon request from the author. These variables are selected for their predictive power. The c statistic indicating area under the ROC curve for this model is 0.91, which indicates very good model fit (Hanley & McNeill, 1982). There is a severe pattern of missingness in the variable for college entrance exams—nearly half of the students who began at a community college did not take these exams. Rather than exclude these units from the calculation, I make use of multiple imputation to create several datasets with simulated values of the missing data. All results are adjusted using the techniques suggested by Little and Rubin (2002)3 . For the purposes of this paper, individuals who began at a community college are treated as the treatment group. Control group members are matched with each treatment group member using the Mahalanobis distance metric within calipers defined by the propensity score. A single control unit was selected for every treated individual. In selecting control units, a caliper width of 0.15 standard deviations in the propensity score is used. If no available matches were found within this width, the unit is excluded. The resulting dataset includes 818 units, with one 4year entrant matched to every 2-year entrant in the original dataset. A summary of the differences in means on the propensity score and the covariates used in the assignment model for the full dataset is available in Table 1. A summary
3 Little and Rubin (2002) has multiple proofs of bias in estimation that results from casewise deletion, and a description of the multiple imputation procedure used in this study.
W.R. Doyle / Economics of Education Review 28 (2009) 199–206
203
Table 2 Differences in means for matched sample, selected matching variables
Propensity score Composite test score Parental income Father’s education (range: 1 = less than H.S., 11 = doctorate) High school GPA (range: 1 = mostly A’s, 7 = mostly D’s)
Means treated
Means control
S.D.
t-Stat
Bias
Reduction
0.41 784.69 44452.97 4.71 4.11
0.39 777.85 43207.34 4.41 4.11
0.22 170.01 57367.12 2.99 1.79
1.32 0.58 0.31 1.46 0
0.09 0.04 0.02 0.1 0
1 1 1 1 1
Fig. 1. Distribution of covariates in complete and matched sample.
of the differences in means for the same variables for the matched dataset is shown in Table 2.4 Fig. 1 shows the distribution of the propensity score and selected covariates in the full and matched samples. The figure shows that the matched sample contains nearly identical distributions of the covariates for those who entered 2-year and 4-year institutions. 7. Results for models using matched data Table 3 shows the results of the Cox proportional hazards model for the matched sample. The effect of community college enrollment on bachelor’s degree attainment reported in the table shows that enrollment in a community college lowers the hazard rate to 0.68 of its baseline
4 The p-value listed in Table 3 refers to the likelihood ratio test statistic for models using multiply imputed datasets suggested by Little and Rubin (2002).
value, with a 95% confidence interval bounded by [0.5, 0.94]. This indicates that, even after matching on propensity scores and controlling for the relevant covariates, the estimated effect of community college enrollment is to lower the likelihood of bachelor’s degree attainment in any given time period. This effect is almost identical to the change in proportional hazard observed in the full sample. Fig. 2 displays the estimate of the effect of community college enrollment for each model using both the full sample and the matched sample. As the figure shows, the estimates of the negative effect of community college enrollment on bachelor’s degree completion are more consistently in the matched sample, but do not differ in the final specification of the model between the full and matched samples. From the results in the matched sample, I conclude that attendance at a community college does result in a lower hazard rate for bachelor’s degree attainment, even when comparing two groups with identically distributed observable characteristics.
204
W.R. Doyle / Economics of Education Review 28 (2009) 199–206
Table 3 Results of Cox proportional hazards model for bachelor’s degree completion—Matched sample (Linearized estimates of standard errors in parentheses) Model 1
Model 2
Model 3
Model 4
Coeff
Exp (coeff)
Coeff
Exp (coeff)
Coeff
Exp (coeff)
Coeff
Exp (coeff)
Community college Age, 10 Dependent Hours worked, 10 Full time Live on campus
−0.38 (0.16)
0.68
−0.39 (0.16) 0.01 (0.4) −0.11 (0.42)
0.68 1 0.9
−0.38 0.01 (0.4) −0.11 (0.42) −0.01 (0.05)
0.68 1 0.9 0.99
−0.38 (0.16) 0.01 (0.4) −0.11 (0.43) −0.01 (0.06) 0.04 (0.16) −0.31 (0.18)
0.68 1 0.9 0.99 1.04 .73
Log-likelihood (mean) Likelihood ratio (mean) DF p N
−767.7 6.2 1 0.01 818
−767.3 7.0 3 0.11 818
8. Sensitivity analysis The impact of community colleges on graduation rates in the matched sample can be considered to be a causal impact only if the conditional independence assumption has been met. It is entirely plausible that some other, unobserved characteristic of individuals could impact both selection into treatment and/or eventual outcomes in a way that would render the results for community colleges equivalent to 0. The goal of this sensitivity analysis is to determine what the characteristics of such a confounding variable would need to be in order to substantially alter the results. The original assumption of conditional independence, as stated earlier, assumes that the outcome is independent of treatment status conditional on the set of observed covariates x. For the purposes of this sensitivity analysis, I alter this assumption to state that the original assumption is false, and that the outcome is actually independent of treatment status conditional on the set of observed covariates x and an additional unobserved binary of confounder,
Fig. 2. Estimate of coefficient for community college attendance in all models (95% confidence interval in gray, 90% confidence interval in black).
−767.3 7.0 4 0.22 818
−765.5 14.43 6 0.02 818
U, that is: y0 ⊥ z|x, U The distribution of the unobserved confounding factor U can be characterized by the parameters pij where: Pr(U = 1|z = i, Y = j, x) = Pr(U = 1|z = i, Y = j) = pij where Y ∈ (0,1) is the outcome status, simplified to a binary variable for college graduation within 6-year, T ∈ (0,1) is a binary variable for the treatment of attendance at a community college, x is the set of covariates as previously defined, and U is the unobserved confounding variable. Following Rosenbaum and Rubin (1983b) and Ichino, Mealli, and Nannicini, (2006), the sensitivity analysis proceeds by imputing a value of the binary variable U to each subject, according to the subjects membership in each of the four groups defined by treatment status and outcomes (Ichino et al., 2006; Rosenbaum & Rubin, 1983b). As Ichino et al. (2006) discuss, the confounding variable U threatens the matching estimate when the confounder has a positive effect on the untreated outcome and on selection into treatment. If such an unobserved variable has this effect, the treatment estimate could in reality be 0, but be estimated to be positive. To determine the characteristics of such a variable, I undertook a sensitivity analysis in two parts. The first chooses parameters pij such that they mimic the distribution of a set of observed binary covariates. This part of the sensitivity analysis, described as sensitivity to “calibrated” confounders, shows how robust the matching estimates are to deviations form the conditional independence assumption which are brought about because we do not have information on other characteristics of individuals which are similar to their observed characteristics in the observable covariates x. The second part of the sensitivity analysis chooses values for U according to parameters pij that are designed to demonstrate the types of confounders that would have to exist to render the treatment effect to be 0. The parameters in this case are selected to satisfy conditions applied to d = p01 − p00 (the difference in the average value of U between graduates and non-graduates at 4-year institutions) and s = p1 − p0 (the difference in the average value of U among those who entered 2-year and 4-year institutions). To limit this part of the analysis, I differ from Ichino et al. in assuming that d > 0 and s < 0, so that the likely outcome
W.R. Doyle / Economics of Education Review 28 (2009) 199–206
205
Table 4 Effect of “calibrated” confounders
Baseline Female Race: non-white Father’s education > bachelor’s Full time Residential
p11
p10
p01
p00
0.60 0.11 0.14 0.61 0.15
0.52 0.33 0.06 0.66 0.05
0.59 0.23 0.26 0.92 0.79
0.51 0.34 0.14 0.87 0.48
Estimated coefficient for public 2-year attendance
S.E.
−0.38 −0.44 −0.43 −0.39 −0.42 −0.41
0.16 0.17 0.17 0.17 0.17 0.18
Note: pij indicate probability U = 1 in each subgroup defined by treatment and outcome status, with i ∈ (0,1) indicating treatment status with i = 1 as community college attendance, and j ∈ (0,1)indicating outcome status, with j = 1 indicating bachelor’s degree completion in 6 years.
effect of the unobserved covariate is positive and the likely selection effect of the same covariate is negative (Ichino et al., 2006). For the purposes of simplicity, and to identify the system of equations, I also follow Ichino et al. in imposing the constraint that the overall probability for the binary variable U = 0.4 and that p11 = p10 . Probabilities pij based on existing covariates are summarized in Table 4. These form the basis for imputing values of U for the sensitivity analysis in Table 4. The first row of Table 4 shows the coefficient for public 2-year attendance from the final model in the matched data set. This serves as the baseline estimate for the remainder of the sensitivity analysis. The second row of the table shows the effect of including a variable U with parameters pij identical to gender. The coefficient in this case is the average value over all 1000 iterations, while the standard is based on Rubin’s method for calculating variance estimates across multiple imputations (Rubin, 1987). The results indicate that the estimated coefficient when including a variable with similar distributional characteristics as gender results is slightly higher, with some increased variance. This is true for all of the other calibrated confounders, with the exception of Father’s education, which results in an estimate very close to the baseline. The estimate is therefore robust to deviations from the conditional independence assumption for variables that have similar properties to these observed covariates. Table 5 characterizes “killer” confounders based on two summaries of the parameters pij . Each cell in Table 5 represents a coefficient estimate for a configuration of parameters for both the difference in community college students and 4-year students in the value of U (s = p1 − p0 ) and the difference in graduates and non-graduates at 4-
year colleges in the value of the covariate U (d = p01 − p00 ). I assume the inequalities p0 > p1 and p00 < p01 , which are designed to threaten the observed negative treatment effect. The first assumption suggests that the unobserved covariate U will make it more likely that a prospective student will select a 4-year institution, while the second assumption states that the effect of that same covariate on the outcome among those who do attend a 4-year institution will be positive. As specified, this confounder could be thought of as something like drive for a bachelor’s degree. Such a student would be more likely to start at a 4-year college, and once in, would be more likely (conditioning on all other characteristics) to graduate. Table 5 shows the impact of such an unobserved confounding variable on the estimated impact of community college attendance on bachelor’s degree completion. The range of estimates for the coefficients goes from −0.47 to −0.36, with standard errors generally slightly larger than those estimated in the original model (this is due to the incorporation of both between and within imputation variance). In only four of the cells does the 95% confidence interval cross 0. This occurs when both d and s are relatively large. In the presence of an unobserved binary covariate which was both much more likely to occur in those who attend a 4-year institution (s < −0.6), and is much more prevalent in those who graduated (d > 0.4), would the estimate reported above be statistically indistinguishable from 0. In all other applications of this sensitivity analysis, the result remains both negative and statistically significant. The results from a sensitivity analysis utilizing both calibrated confounders and “killer” confounders shows that the estimate is robust to unobserved binary covariates
Table 5 Effect of “killer” confounders
Baseline d = 0.1 d = 0.2 d = 0.3 d = 0.4 d = 0.5 d = 0.6 d = 0.7
s = −0.1
s = −0.2
s = 0.3
s = 0.4
s = 0.5
s = 0.6
s = 0.7
−0.38 (0.16) −0.43 (0.17) −0.42 (0.17) −0.41 (0.17) −0.42 (0.17) −0.44 (0.17) −0.42 (0.17) −0.42 (0.17)
−0.43 (0.17) −0.42 (0.17) −0.41 (0.17) −0.43 (0.17) −0.4 (0.17) −0.42 (0.17) −0.43 (0.17)
−0.44 (0.17) −0.42 (0.17) −0.42 (0.17) −0.42 (0.17) −0.41 (0.17) −0.44 (0.17) −0.43 (0.17)
−0.43 (0.17) −0.44 (0.17) −0.44 (0.17) −0.43 (0.18) −0.41 (0.17) −0.42 (0.17) −0.43 (0.18)
−0.44 (0.17) −0.44 (0.17) −0.46 (0.18) −0.42 (0.18) −0.4 (0.18) −0.42 (0.18) −0.39 (0.19)
−0.45 (0.17) −0.45 (0.18) −0.44 (0.18) −0.4 (0.18) −0.42 (0.19) −0.41 (0.19) −0.39 (0.2)
−0.46 (0.18) −0.47 (0.18) −0.41 (0.18) −0.36 (0.19) −0.4 (0.2) −0.41 (0.2) −0.37 (0.2)
Note: s = p1 − p0 , the difference in the hypothetical binary variable U between those who attended and did not attend community colleges. d = p11 − p10 , the difference in the hypothetical binary variable U among those who did and did not graduate in 6 years. Estimates are the average of 1000 iterations repeating the previously specified “baseline” estimate with multiple imputations of the binary variable U. Standard errors combine both between- and within-imputation variance, following Rubin (1987). Numbers in bold indicate results are not significant.
206
W.R. Doyle / Economics of Education Review 28 (2009) 199–206
which are similar to observed binary covariates. The result would be threatened by a binary covariate which is much more likely to occur in students who attend a 4-year institution and graduate in 6 years. 9. Implications This analysis demonstrates the utility of propensity score matching for evaluating the impact of different attendance patterns in higher education. The use of nonparametric propensity score matching yielded results consistent with previous findings, all of which indicate a negative relationship between community college attendance and eventual degree completion. Policymakers who seek to shift enrollments to community colleges should be aware that lower graduation rates are a distinct possibility. Further research should investigate policy interventions at the community college and 4-year level that would aid both the transfer of credits between institutions and the time to degree for community college students who transfer to 4-year institutions. Acknowledgments I would like to thank Dale Ballot, Mark Dynarski, Stella Flores, Matthew Springer, Liang Zhang, and an anonymous reviewer for their comments and suggestions. Andrea Ichino and Tomas Nannicini were very generous in sharing data and STATA code that were used to implement their approach in the statistical programming language R. The author takes sole responsibility for the contents of this paper. References Adelman, C. (1999). Answers in the tool box: academic intensity, attendance patterns, and bachelor’s degree attainment, Technical report, United States Department of Education, National Center for Education Statistics. Braxton, J., Milem, J. F., & Sullivan, A. S. (2000). The influence of active learning on the college student departure process. Journal of Higher Education, 71(5), 569–590. Brint, S., & Karabel, J. (1989). The diverted dream. Oxford: Oxford University Press. Clark, B. A. (1960). The “cooling-out” function in higher education. The American Journal of Sociology, 65(6), 569–576.
Cox, D. R. (1972). Regression models and life tables. Journal of the Royal Statistical Society, Series B, 34(2). DesJardins, S. L., Ahlburg, D., & McCall, B. P. (1999). An event history model of student departure. Economics of Education Review, 18, 375–390. Hanley, J. A., & McNeill, B. J. (1982). The meaning and use of the area under a receiver operating curve. Radiology, 143, 29–36. Heckman, J. J., Ichimura, H., & Todd, P. E. (1997). Matching as an econometric evaluation estimator: evidence from evaluating a job training program. The Review of Economic Studies, 64(4), 605–654. Heckman, J. J., Ichimura, H., & Todd, P. E. (1998). Matching as an econometric evaluation estimator. The Review of Economic Studies, 65(2), 261–294. Ho, D. E., Imai, K., King, G., & Stuart, E. A. (2005). Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference1. Harvard University. Ichino, A. Mealli, F., & Nannicini, T. (2006). From temporary help jobs to permanent employment. What can we learn from matching estimators and their sensitivity?, Institute for the Study of Labor Discussion Paper #2149. Kalbfleisch, J., & Prentice, R. (1980). The Statistical analysis of failure time data. New York: John Wiley and Sons. Leigh, D. E., & Gill, A. M. (2003). Do community colleges really divert students from earning bachelor’s degrees? Economics of Education Review, 22(1), 23–30. Little, R. J., & Rubin, D. B. (2002). Statistical analysis with missing data (2nd ed.). Wiley-Interscience. Rosenbaum, P. R., & Rubin, D. B. (1983a). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55. Rosenbaum, P. R., & Rubin, D. B. (1983b). Assessing Sensitivity to an Unobserved Binary Covariate in an Observational Study with Binary Outcome. Journal of Royal Statistical Society Series B, 45(2),212–218. Rouse, C. E. (1995). Democratization or diversion—the effect of community-colleges on educational-attainment. Journal of Business and Economic Statistics, 13(2), 217–224. Rouse, C. E. (1998). Do two-year colleges increase overall educational attainment? Evidence from the States. Journal of Policy Analysis and Management, 17(4), 595–620. Rubin, D. (1987). Multiple imputation for nonresponse in surveys. New York: Springer. Singer, J. D., & Willett, J. B. (2003). Applied longitudinal data analysis: modeling change and event occurrence. New York: Oxford University Press. Smith, J. A., & Todd, P. E. (2001). Reconciling conflicting evidence on the performance of propensity score methods. American Economic Review, 91(2), 112–118. Therneau, T. M., & Grambsch, P. M. (2001). Modeling survival data: extending the Cox model. New York: Springer. Tinto, V. (1975). Dropout from higher education: a theoretical synthesis of recent research. Review of Educational Research, 45(1), 89–125. United States Department of Education (2002). Beginning postsecondary students longitudinal study: 1996–2001 (bps: 1996/2001) methodology report. Technical report, National Center for Education Statistics. United States Department of Education (2004). Digest of education statistics 2002. Technical report, National Center for Education Statistics. United States Department of Education (2005). Beginning postsecondary students: data analysis system. Technical report, National Center for Education Statistics.
Contents lists available at ScienceDirect
Economics of Education Review journal homepage: www.elsevier.com/locate/econedurev
The effect of community college enrollment on bachelor’s degree completion William R. Doyle ∗ Peabody College of Vanderbilt University, Peabody #514, 230 Appleton Place, Nashville, TN 37203-5721, United States
a r t i c l e
i n f o
Article history: Received 13 September 2006 Accepted 18 January 2008 JEL classification: I23 I28 C14 C41 Keywords: Efficiency Costs Higher education Community colleges
a b s t r a c t Rouse [Rouse, C. E. (1995). Democratization or diversion—the effect of community-colleges on educational-attainment. Journal of Business and Economic Statistics, 13(2), 217–224] finds that enrollment in a community college may divert students from attaining a bachelor’s degree. However, this result may be due to selection bias, as the population of community college students should be quite different from those who attend 4-year institutions in terms of both observable and unobservable characteristics. This study uses propensity score matching to non-parametrically balance a data set from the 1996 Beginning Postsecondary Students survey in order to overcome issues associated with selection bias. Results from a Cox proportional hazards model indicate that attendance at a community college lowers the hazard rate for completing a bachelor’s degree. The results are consistent with previous studies. © 2008 Elsevier Ltd. All rights reserved.
1. Introduction Nearly 80% of first time students who enter community colleges intend to complete a bachelor’s degree (United States Department of Education, 2005). While community colleges have a number of functions, it appears that most entering students intend to use them as a stepping stone to eventual bachelor’s degree completion. The vast majority of students who begin their postsecondary education in community colleges do not go on to attain a bachelor’s degree. Of those who begin at a community college with the intention of completing a bachelor’s degree, only 21% do so within 6 years (United States Department of Education, 2005). Many have suggested that community colleges actually function to push students away from their educational goals (Brint & Karabel, 1989;
∗ Tel.: +1 615 322 3904; fax: +1 615 343 7094. E-mail address: [email protected]. 0272-7757/$ – see front matter © 2008 Elsevier Ltd. All rights reserved. doi:10.1016/j.econedurev.2008.01.006
Clark, 1960). Rouse (1995) calls this the “diversion” effect of community colleges. On the other hand, many students who enroll at a community college would not otherwise have attended higher education, and some fraction of these students do go on to attain a bachelor’s degree. This is termed by Rouse the “democratization” effect of community colleges (Rouse, 1995). This debate has become quite important for policy purposes. Community colleges charge lower tuition and have lower per-student costs than their 4-year counterparts. Per student expenditures at public 4-year institutions average $9183 nationally, while per student expenditures at public 4-year institutions average $27,973 (United States Department of Education, 2004). The savings to both the student and the state are substantial when a student attends a community college. If students who begin at a community college are just as likely to complete a bachelor’s degree, then there is no reason for state policymakers not to encourage students
200
W.R. Doyle / Economics of Education Review 28 (2009) 199–206
to begin at such an institution. However, if students who begin at a community college are much less likely to attain a degree, then policymakers and institutional leaders would need to undertake a more serious examination of policies and practices to understand why exactly students who might have completed a bachelor’s degree were indeed diverted from their goals. Students who begin their postsecondary studies in community colleges are quite different from their counterparts who begin at a 4-year institution. These differences extend from observable features such as academic preparation and family background to possible unobservable features such as motivation or ability. Previous studies have dealt with this problem in various ways, either by using an instrumental variables approach to recover a quasi-experimental design or by using a selection on observables approach (Leigh & Gill, 2003; Rouse, 1995). This paper extends the selection on observables approach to balance the sample of students between a control (4-year enrollment) and a treatment (2-year enrollment) group. I am interested primarily in the question of diversion, that is, does starting at a community college lower the likelihood that a student will complete their desired level of schooling in a certain amount of time? To address the problem of selection bias in the estimation of treatment effects, I utilize propensity score matching in order to non-parametrically pre-process data. In order to accurately model the inherently timedependent process of degree attainment, I employ a Cox proportional hazards model (Kalbfleisch & Prentice, 1980; Singer & Willett, 2003). In using this approach, I am able to compare the experience of two very similar groups of students, one group which started at a community college and another group which began at a 4-year college. The educational outcomes of both groups after 6 years are compared. I find that the hazard rate for completion is lower for community college matriculants in both the full and matched sample.
Rouse (1995) used an instrumental variables approach to estimate the causal effect of community college attendance on bachelor’s degree attainment. Rouse uses two stage least squares to model the effect of community college attendance on both years of schooling and bachelor’s degree attainment. Using a sample from the High School and beyond dataset, Rouse finds a statistically significant negative coefficient on community college attendance when the dependent variable is years of attainment, with an effect of 0.6–1 fewer years of educational attainment for those who began at a community college. Rouse’s study has several shortcomings, most notably that it does not condition on students’ intentions. Many students enter community colleges without intending to attain a bachelor’s degree. Leigh and Gill (2003) make use of National Longitudinal Study of Youth data in order to condition on students’ expectations. Again using years of schooling as the dependent variable, they find that students who begin at a community college with the intention of receiving a bachelor’s degree end up with 0.7–0.8 fewer years of schooling than their peers. The authors of both studies acknowledge that they cannot account for many of the unobservable characteristics of students who enter 2-year as opposed to four colleges, a problem that could only be resolved through random assignment to institutions. However, this paper builds on these studies by matching the distributions of observable characteristics of students at community colleges with the distributions of the same characteristics of their peers. Several studies have examined the impact of institutional and state policies on bachelor’s degree completion. However, these studies have limited their samples to matriculants at 4-year colleges only (DesJardins, Ahlburg, & McCall, 1999). Adelman (1999) suggests that entrance into a 4-year college is the only reliable indicator of a student’s intention to complete a bachelor’s degree. While this may be reflective of the data, it does little to suggest whether community colleges can help students complete their educational objectives.
2. Previous studies The issue of students leaving college has long been of central concern to many researchers in the field of higher education. Why a student would not complete college is clearly an important policy issue at the institutional, state and federal level. One strand of the literature focuses primarily on the institution and the personal and environmental factors that may help or hinder students as they seek to complete their degrees (Braxton, Milem, & Sullivan, 2000; Tinto, 1975). A second strand of the literature, to which this study belongs, looks at the policy impact of differing institutional types on the eventual completion of students. I briefly review a few of the second type of study here. Two notable previous studies have looked at the issues of democratization of diversion in community colleges. The first, Rouse (1995) was the first to specify a model for democratization or diversion. The second, Leigh and Gill (2003) updates and refines Rouse’s model, using data that measure the intentions of students prior to entering higher education.
3. Functional form and student departure Traditional models of student departure have not taken into account that this is a process that is time-dependent, and therefore most appropriately modeled using event history analysis. This section briefly describes why the Cox proportional hazard model is well suited to modeling the hazard rate for graduation from college. Many authors1 have specified student departure from higher education as a standard a binary outcome, and model the departure process using standard logit or probit models. This type of model does not take into account the time-varying nature of the data, instead assuming that the hazard rate for graduation is constant over time. As DesJardins et al. (1999) point out, this is in many ways an unsatisfying approach to modeling event history data, since it assumes that in each year since matricula-
1
See for example Adelman (1999).
W.R. Doyle / Economics of Education Review 28 (2009) 199–206
tion, the overall probability of graduation is the same.2 The DesJardins et al. (1999) approach makes use of a prespecified form of the hazard rate to understand the effect of covariates on the binary dependent variable. An alternative form provided by Cox (1972) provides a more flexible functional form that does not rely on a pre-specified shape of the hazard curve. In the Cox proportional hazards model the non-parametric hazard rate is increased or decreased through multiplication by a scalar function of covariates (Therneau & Grambsch, 2001).
To proceed, an assumption of conditional independence must be made. That is, we need to assume that the information contained in x is sufficient to make the counterfactual outcome y0 independent of z (Smith & Todd, 2001): y0 ⊥ z|x Matching requires an assumption of common support, that is p(z = 1|x) < 1
3.1. Propensity score matching and causal inference Given the above-specified functional form for the process of student departure, I now turn to the use of propensity score matching to recover causal estimates of the effect of community college enrollment on the hazard rate for individuals’ attainment of a bachelor’s degree. All previous studies have dealt with a common issue: an experimental approach can yield causal estimates, but are infeasible in determining choice of college. In her analysis, Rouse (1995) employs an instrumental variable approach and finds a negative but marginally statistically significant effect of community colleges on attainment. Rouse addresses issues of exogeneity of these instrumental variables in her work, and finds “to the extent that observed measures of family background also reflect unobserved determinants of educational attainment, there is evidence that the instruments may be correlated with the error term in the educational attainment equation” (Rouse, 1995, p. 223). A different approach to this problem is selection on observables. This is the approach utilized by Leigh and Gill (2003). The selection on observables approach suggest that the conditioning variables contain sufficient information to provide a linear extrapolation of the effect of starting college on attainment for all types of students (footnote 2). Their results are consistent with Rouse (1995). Another approach to the problem of endogeneity of treatment is possible. Matching is in many ways a nonparametric form of selection on observables. In this study I divide students into two different groups, depending on whether the student in question began at a 2-year or 4year college. Group y1 denotes those students who were “treated”, that is, those students who began at a 2-year college. Group y0 denotes those students who began at a 4-year college (or, more neutrally, did not begin at a 2-year college). Student’s characteristics are described by a vector of covariates x and z, a binary variable equal to 1 if a student begins at a 2-year college and equal to 0 otherwise. What this paper seeks to establish is the effect of attending a 2-year college on those who did, in fact, attend a 2-year college—this is known as treatment on the treated (Smith & Todd, 2001): tt = E(y1 |x, z = 1) − E(y0 |x, z = 1)
2 Leigh and Gill (2003) do not directly address the same problem as this paper and the Rouse paper, instead focusing primarily on conditioning results on desired levels of schooling.
201
for all x
In our example, this would require that for all of the variables supposed to affect selection into the treatment groups, there is an area of common support among all groups in both the control and treatment group. However, instead of utilizing exact matching, it is also possible to match on x using the probability of treatment p = Pr(z = 1|x) (Heckman, Ichimura, & Todd, 1998). Probability of treatment can be estimated using standard models for binary outcomes, with treatment as the outcome in question. The predicted value for each individual is called the propensity score for treatment (Rosenbaum & Rubin, 1983a). In propensity score matching a vector of covariates x, is used to identify the propensity score pi for all individuals for attending a 2-year institution (Rosenbaum & Rubin, 1983a). Matching is accomplished by looking for students in the control group who are sufficiently similar to students in the treatment group in terms of this propensity score. The propensity score estimate of treatment on the treated is estimated as tt = E(y1 |x, z = 1) − E(y0 |x, z = 1) = E(E[y1 |p, z = 1] − E(y0 |p, z = 0]z = 1) In the case of this study the treatment on the treatment indicator will not be the mean differences in groups but the differences in proportional hazards between individuals who begin at community colleges and those who begin at 4-year institutions. Heckman, Ichimura, and Todd (1997) review the essential features necessary to reduce bias in evaluation studies and summarize them as (1) the treatment and control groups have identically distributed unobserved attributes; (2) observed attributes are identically distributed; (3) both treatment and control groups are given the same questionnaire; (4) treatment and control groups are in the same economic (or social) environment. Heckman et al. (1997) suggest that features (2–4) are much more important than has been previously suggested, and feature (1), selection bias, “is a relatively small part of bias as conventionally measured” (Heckman et al., 1997, p. 606). Features (2) and (3) are met quite easily in this study: propensity score matching will ensure identical distribution of observed characteristics, and all studied individuals completed a common questionnaire. Feature (4) common environments, is not met, since states vary considerably in the number of community colleges, and likelihood of attendance is predicated in part on proximity to a com-
202
W.R. Doyle / Economics of Education Review 28 (2009) 199–206
Table 1 Differences in means for complete sample, selected matching variables
Propensity score Composite test score Parental income Father’s education (range: 1 = less than H.S., 11 = doctorate) High school GPA (range: 1 = mostly A’s, 7 = mostly D’s)
Means treated
Means control
S.D.
t-Stat
Bias
0.54 755.99 42509.35 4.52 4.01
0.05 952.53 59503.61 5.78 3.56
0.2 217.03 57752.97 3.24 1.66
48.79 −26.21 −7.99 −10.09 5.85
1.93 −1.1 −0.33 −0.42 0.24
munity college (Rouse, 1995, 1998). Feature (1) should be minimized, but as always with observational studies, there can be no guarantee that treatments and controls do not have different distributions of unobserved characteristics.
4. Data This analysis makes use of data drawn from the National Center for Education Statistic’s Beginning Postsecondary Students study. Students in this study were identified at their initial institution of postsecondary enrollment through the National Postsecondary Student Aid Survey in 1996. Students were then followed through a series of follow up interviews in 2001 (United States Department of Education, 2002). The dependent variable in the study reflects the amount of time that the student took to complete a bachelor’s degree, measured in months. The data for this study are by necessity right-censored, since individuals who did not complete a bachelor’s degree may go on to do so in the future. The key independent variable of interest is a binary variable for community college entrance, coded to 1 if the respondent began her college career in a community college and coded to 0 otherwise. Five independent variables are used as controls. These are drawn from the relevant literature as important predictors of student success in completing a bachelor’s degree. These variables are defined as follows: • Age: student’s age in years at college entrance • Dependent: binary variable, coded to 1 if student is a dependent, 0 otherwise; • Hours worked: number of hours worked per week in academic year 1995–1996; • Full time: binary variable, coded to 1 if student enrolled full time at first institution; • Live on campus: binary variable, coded to 1 if student lived on campus at first institution. 5. Results for full sample Results for the Cox proportional hazard model for the full sample are available in a separate appendix on the author’s website. The results indicate that, after controlling for the above variables, beginning at a community college proportionally reduces the hazard rate for college completion by a factor of 0.7, with a 95% confidence interval bounded by [0.58, 0.86].
6. Matching for non-parametric pre-processing To obtain a better estimate for causal inference, this paper makes use of matching for non-parametric preprocessing, following the procedures suggested by Ho, Imai, King, and Stuart (2005). The computation for this paper was done in R, using the package Matchit described by Ho et al. (2005). To calculate the propensity score for each unit, I utilized a probit model for community college entrance using a total of 40 different characteristics of individuals, divided into four broad groupings: family characteristics (e.g. income, parent’s education), high school characteristics (e.g. percent free lunch, highest level of math offered) student information (e.g. delayed entry, gender, race), attitudinal data (e.g. what factors affected their choice of college) and financial aid and college cost data (tuition, aid from various sources). A summary of these variables, their coefficients in the matching model, and their means in both control and treatment groups in the full and matched sample is available upon request from the author. These variables are selected for their predictive power. The c statistic indicating area under the ROC curve for this model is 0.91, which indicates very good model fit (Hanley & McNeill, 1982). There is a severe pattern of missingness in the variable for college entrance exams—nearly half of the students who began at a community college did not take these exams. Rather than exclude these units from the calculation, I make use of multiple imputation to create several datasets with simulated values of the missing data. All results are adjusted using the techniques suggested by Little and Rubin (2002)3 . For the purposes of this paper, individuals who began at a community college are treated as the treatment group. Control group members are matched with each treatment group member using the Mahalanobis distance metric within calipers defined by the propensity score. A single control unit was selected for every treated individual. In selecting control units, a caliper width of 0.15 standard deviations in the propensity score is used. If no available matches were found within this width, the unit is excluded. The resulting dataset includes 818 units, with one 4year entrant matched to every 2-year entrant in the original dataset. A summary of the differences in means on the propensity score and the covariates used in the assignment model for the full dataset is available in Table 1. A summary
3 Little and Rubin (2002) has multiple proofs of bias in estimation that results from casewise deletion, and a description of the multiple imputation procedure used in this study.
W.R. Doyle / Economics of Education Review 28 (2009) 199–206
203
Table 2 Differences in means for matched sample, selected matching variables
Propensity score Composite test score Parental income Father’s education (range: 1 = less than H.S., 11 = doctorate) High school GPA (range: 1 = mostly A’s, 7 = mostly D’s)
Means treated
Means control
S.D.
t-Stat
Bias
Reduction
0.41 784.69 44452.97 4.71 4.11
0.39 777.85 43207.34 4.41 4.11
0.22 170.01 57367.12 2.99 1.79
1.32 0.58 0.31 1.46 0
0.09 0.04 0.02 0.1 0
1 1 1 1 1
Fig. 1. Distribution of covariates in complete and matched sample.
of the differences in means for the same variables for the matched dataset is shown in Table 2.4 Fig. 1 shows the distribution of the propensity score and selected covariates in the full and matched samples. The figure shows that the matched sample contains nearly identical distributions of the covariates for those who entered 2-year and 4-year institutions. 7. Results for models using matched data Table 3 shows the results of the Cox proportional hazards model for the matched sample. The effect of community college enrollment on bachelor’s degree attainment reported in the table shows that enrollment in a community college lowers the hazard rate to 0.68 of its baseline
4 The p-value listed in Table 3 refers to the likelihood ratio test statistic for models using multiply imputed datasets suggested by Little and Rubin (2002).
value, with a 95% confidence interval bounded by [0.5, 0.94]. This indicates that, even after matching on propensity scores and controlling for the relevant covariates, the estimated effect of community college enrollment is to lower the likelihood of bachelor’s degree attainment in any given time period. This effect is almost identical to the change in proportional hazard observed in the full sample. Fig. 2 displays the estimate of the effect of community college enrollment for each model using both the full sample and the matched sample. As the figure shows, the estimates of the negative effect of community college enrollment on bachelor’s degree completion are more consistently in the matched sample, but do not differ in the final specification of the model between the full and matched samples. From the results in the matched sample, I conclude that attendance at a community college does result in a lower hazard rate for bachelor’s degree attainment, even when comparing two groups with identically distributed observable characteristics.
204
W.R. Doyle / Economics of Education Review 28 (2009) 199–206
Table 3 Results of Cox proportional hazards model for bachelor’s degree completion—Matched sample (Linearized estimates of standard errors in parentheses) Model 1
Model 2
Model 3
Model 4
Coeff
Exp (coeff)
Coeff
Exp (coeff)
Coeff
Exp (coeff)
Coeff
Exp (coeff)
Community college Age, 10 Dependent Hours worked, 10 Full time Live on campus
−0.38 (0.16)
0.68
−0.39 (0.16) 0.01 (0.4) −0.11 (0.42)
0.68 1 0.9
−0.38 0.01 (0.4) −0.11 (0.42) −0.01 (0.05)
0.68 1 0.9 0.99
−0.38 (0.16) 0.01 (0.4) −0.11 (0.43) −0.01 (0.06) 0.04 (0.16) −0.31 (0.18)
0.68 1 0.9 0.99 1.04 .73
Log-likelihood (mean) Likelihood ratio (mean) DF p N
−767.7 6.2 1 0.01 818
−767.3 7.0 3 0.11 818
8. Sensitivity analysis The impact of community colleges on graduation rates in the matched sample can be considered to be a causal impact only if the conditional independence assumption has been met. It is entirely plausible that some other, unobserved characteristic of individuals could impact both selection into treatment and/or eventual outcomes in a way that would render the results for community colleges equivalent to 0. The goal of this sensitivity analysis is to determine what the characteristics of such a confounding variable would need to be in order to substantially alter the results. The original assumption of conditional independence, as stated earlier, assumes that the outcome is independent of treatment status conditional on the set of observed covariates x. For the purposes of this sensitivity analysis, I alter this assumption to state that the original assumption is false, and that the outcome is actually independent of treatment status conditional on the set of observed covariates x and an additional unobserved binary of confounder,
Fig. 2. Estimate of coefficient for community college attendance in all models (95% confidence interval in gray, 90% confidence interval in black).
−767.3 7.0 4 0.22 818
−765.5 14.43 6 0.02 818
U, that is: y0 ⊥ z|x, U The distribution of the unobserved confounding factor U can be characterized by the parameters pij where: Pr(U = 1|z = i, Y = j, x) = Pr(U = 1|z = i, Y = j) = pij where Y ∈ (0,1) is the outcome status, simplified to a binary variable for college graduation within 6-year, T ∈ (0,1) is a binary variable for the treatment of attendance at a community college, x is the set of covariates as previously defined, and U is the unobserved confounding variable. Following Rosenbaum and Rubin (1983b) and Ichino, Mealli, and Nannicini, (2006), the sensitivity analysis proceeds by imputing a value of the binary variable U to each subject, according to the subjects membership in each of the four groups defined by treatment status and outcomes (Ichino et al., 2006; Rosenbaum & Rubin, 1983b). As Ichino et al. (2006) discuss, the confounding variable U threatens the matching estimate when the confounder has a positive effect on the untreated outcome and on selection into treatment. If such an unobserved variable has this effect, the treatment estimate could in reality be 0, but be estimated to be positive. To determine the characteristics of such a variable, I undertook a sensitivity analysis in two parts. The first chooses parameters pij such that they mimic the distribution of a set of observed binary covariates. This part of the sensitivity analysis, described as sensitivity to “calibrated” confounders, shows how robust the matching estimates are to deviations form the conditional independence assumption which are brought about because we do not have information on other characteristics of individuals which are similar to their observed characteristics in the observable covariates x. The second part of the sensitivity analysis chooses values for U according to parameters pij that are designed to demonstrate the types of confounders that would have to exist to render the treatment effect to be 0. The parameters in this case are selected to satisfy conditions applied to d = p01 − p00 (the difference in the average value of U between graduates and non-graduates at 4-year institutions) and s = p1 − p0 (the difference in the average value of U among those who entered 2-year and 4-year institutions). To limit this part of the analysis, I differ from Ichino et al. in assuming that d > 0 and s < 0, so that the likely outcome
W.R. Doyle / Economics of Education Review 28 (2009) 199–206
205
Table 4 Effect of “calibrated” confounders
Baseline Female Race: non-white Father’s education > bachelor’s Full time Residential
p11
p10
p01
p00
0.60 0.11 0.14 0.61 0.15
0.52 0.33 0.06 0.66 0.05
0.59 0.23 0.26 0.92 0.79
0.51 0.34 0.14 0.87 0.48
Estimated coefficient for public 2-year attendance
S.E.
−0.38 −0.44 −0.43 −0.39 −0.42 −0.41
0.16 0.17 0.17 0.17 0.17 0.18
Note: pij indicate probability U = 1 in each subgroup defined by treatment and outcome status, with i ∈ (0,1) indicating treatment status with i = 1 as community college attendance, and j ∈ (0,1)indicating outcome status, with j = 1 indicating bachelor’s degree completion in 6 years.
effect of the unobserved covariate is positive and the likely selection effect of the same covariate is negative (Ichino et al., 2006). For the purposes of simplicity, and to identify the system of equations, I also follow Ichino et al. in imposing the constraint that the overall probability for the binary variable U = 0.4 and that p11 = p10 . Probabilities pij based on existing covariates are summarized in Table 4. These form the basis for imputing values of U for the sensitivity analysis in Table 4. The first row of Table 4 shows the coefficient for public 2-year attendance from the final model in the matched data set. This serves as the baseline estimate for the remainder of the sensitivity analysis. The second row of the table shows the effect of including a variable U with parameters pij identical to gender. The coefficient in this case is the average value over all 1000 iterations, while the standard is based on Rubin’s method for calculating variance estimates across multiple imputations (Rubin, 1987). The results indicate that the estimated coefficient when including a variable with similar distributional characteristics as gender results is slightly higher, with some increased variance. This is true for all of the other calibrated confounders, with the exception of Father’s education, which results in an estimate very close to the baseline. The estimate is therefore robust to deviations from the conditional independence assumption for variables that have similar properties to these observed covariates. Table 5 characterizes “killer” confounders based on two summaries of the parameters pij . Each cell in Table 5 represents a coefficient estimate for a configuration of parameters for both the difference in community college students and 4-year students in the value of U (s = p1 − p0 ) and the difference in graduates and non-graduates at 4-
year colleges in the value of the covariate U (d = p01 − p00 ). I assume the inequalities p0 > p1 and p00 < p01 , which are designed to threaten the observed negative treatment effect. The first assumption suggests that the unobserved covariate U will make it more likely that a prospective student will select a 4-year institution, while the second assumption states that the effect of that same covariate on the outcome among those who do attend a 4-year institution will be positive. As specified, this confounder could be thought of as something like drive for a bachelor’s degree. Such a student would be more likely to start at a 4-year college, and once in, would be more likely (conditioning on all other characteristics) to graduate. Table 5 shows the impact of such an unobserved confounding variable on the estimated impact of community college attendance on bachelor’s degree completion. The range of estimates for the coefficients goes from −0.47 to −0.36, with standard errors generally slightly larger than those estimated in the original model (this is due to the incorporation of both between and within imputation variance). In only four of the cells does the 95% confidence interval cross 0. This occurs when both d and s are relatively large. In the presence of an unobserved binary covariate which was both much more likely to occur in those who attend a 4-year institution (s < −0.6), and is much more prevalent in those who graduated (d > 0.4), would the estimate reported above be statistically indistinguishable from 0. In all other applications of this sensitivity analysis, the result remains both negative and statistically significant. The results from a sensitivity analysis utilizing both calibrated confounders and “killer” confounders shows that the estimate is robust to unobserved binary covariates
Table 5 Effect of “killer” confounders
Baseline d = 0.1 d = 0.2 d = 0.3 d = 0.4 d = 0.5 d = 0.6 d = 0.7
s = −0.1
s = −0.2
s = 0.3
s = 0.4
s = 0.5
s = 0.6
s = 0.7
−0.38 (0.16) −0.43 (0.17) −0.42 (0.17) −0.41 (0.17) −0.42 (0.17) −0.44 (0.17) −0.42 (0.17) −0.42 (0.17)
−0.43 (0.17) −0.42 (0.17) −0.41 (0.17) −0.43 (0.17) −0.4 (0.17) −0.42 (0.17) −0.43 (0.17)
−0.44 (0.17) −0.42 (0.17) −0.42 (0.17) −0.42 (0.17) −0.41 (0.17) −0.44 (0.17) −0.43 (0.17)
−0.43 (0.17) −0.44 (0.17) −0.44 (0.17) −0.43 (0.18) −0.41 (0.17) −0.42 (0.17) −0.43 (0.18)
−0.44 (0.17) −0.44 (0.17) −0.46 (0.18) −0.42 (0.18) −0.4 (0.18) −0.42 (0.18) −0.39 (0.19)
−0.45 (0.17) −0.45 (0.18) −0.44 (0.18) −0.4 (0.18) −0.42 (0.19) −0.41 (0.19) −0.39 (0.2)
−0.46 (0.18) −0.47 (0.18) −0.41 (0.18) −0.36 (0.19) −0.4 (0.2) −0.41 (0.2) −0.37 (0.2)
Note: s = p1 − p0 , the difference in the hypothetical binary variable U between those who attended and did not attend community colleges. d = p11 − p10 , the difference in the hypothetical binary variable U among those who did and did not graduate in 6 years. Estimates are the average of 1000 iterations repeating the previously specified “baseline” estimate with multiple imputations of the binary variable U. Standard errors combine both between- and within-imputation variance, following Rubin (1987). Numbers in bold indicate results are not significant.
206
W.R. Doyle / Economics of Education Review 28 (2009) 199–206
which are similar to observed binary covariates. The result would be threatened by a binary covariate which is much more likely to occur in students who attend a 4-year institution and graduate in 6 years. 9. Implications This analysis demonstrates the utility of propensity score matching for evaluating the impact of different attendance patterns in higher education. The use of nonparametric propensity score matching yielded results consistent with previous findings, all of which indicate a negative relationship between community college attendance and eventual degree completion. Policymakers who seek to shift enrollments to community colleges should be aware that lower graduation rates are a distinct possibility. Further research should investigate policy interventions at the community college and 4-year level that would aid both the transfer of credits between institutions and the time to degree for community college students who transfer to 4-year institutions. Acknowledgments I would like to thank Dale Ballot, Mark Dynarski, Stella Flores, Matthew Springer, Liang Zhang, and an anonymous reviewer for their comments and suggestions. Andrea Ichino and Tomas Nannicini were very generous in sharing data and STATA code that were used to implement their approach in the statistical programming language R. The author takes sole responsibility for the contents of this paper. References Adelman, C. (1999). Answers in the tool box: academic intensity, attendance patterns, and bachelor’s degree attainment, Technical report, United States Department of Education, National Center for Education Statistics. Braxton, J., Milem, J. F., & Sullivan, A. S. (2000). The influence of active learning on the college student departure process. Journal of Higher Education, 71(5), 569–590. Brint, S., & Karabel, J. (1989). The diverted dream. Oxford: Oxford University Press. Clark, B. A. (1960). The “cooling-out” function in higher education. The American Journal of Sociology, 65(6), 569–576.
Cox, D. R. (1972). Regression models and life tables. Journal of the Royal Statistical Society, Series B, 34(2). DesJardins, S. L., Ahlburg, D., & McCall, B. P. (1999). An event history model of student departure. Economics of Education Review, 18, 375–390. Hanley, J. A., & McNeill, B. J. (1982). The meaning and use of the area under a receiver operating curve. Radiology, 143, 29–36. Heckman, J. J., Ichimura, H., & Todd, P. E. (1997). Matching as an econometric evaluation estimator: evidence from evaluating a job training program. The Review of Economic Studies, 64(4), 605–654. Heckman, J. J., Ichimura, H., & Todd, P. E. (1998). Matching as an econometric evaluation estimator. The Review of Economic Studies, 65(2), 261–294. Ho, D. E., Imai, K., King, G., & Stuart, E. A. (2005). Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference1. Harvard University. Ichino, A. Mealli, F., & Nannicini, T. (2006). From temporary help jobs to permanent employment. What can we learn from matching estimators and their sensitivity?, Institute for the Study of Labor Discussion Paper #2149. Kalbfleisch, J., & Prentice, R. (1980). The Statistical analysis of failure time data. New York: John Wiley and Sons. Leigh, D. E., & Gill, A. M. (2003). Do community colleges really divert students from earning bachelor’s degrees? Economics of Education Review, 22(1), 23–30. Little, R. J., & Rubin, D. B. (2002). Statistical analysis with missing data (2nd ed.). Wiley-Interscience. Rosenbaum, P. R., & Rubin, D. B. (1983a). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55. Rosenbaum, P. R., & Rubin, D. B. (1983b). Assessing Sensitivity to an Unobserved Binary Covariate in an Observational Study with Binary Outcome. Journal of Royal Statistical Society Series B, 45(2),212–218. Rouse, C. E. (1995). Democratization or diversion—the effect of community-colleges on educational-attainment. Journal of Business and Economic Statistics, 13(2), 217–224. Rouse, C. E. (1998). Do two-year colleges increase overall educational attainment? Evidence from the States. Journal of Policy Analysis and Management, 17(4), 595–620. Rubin, D. (1987). Multiple imputation for nonresponse in surveys. New York: Springer. Singer, J. D., & Willett, J. B. (2003). Applied longitudinal data analysis: modeling change and event occurrence. New York: Oxford University Press. Smith, J. A., & Todd, P. E. (2001). Reconciling conflicting evidence on the performance of propensity score methods. American Economic Review, 91(2), 112–118. Therneau, T. M., & Grambsch, P. M. (2001). Modeling survival data: extending the Cox model. New York: Springer. Tinto, V. (1975). Dropout from higher education: a theoretical synthesis of recent research. Review of Educational Research, 45(1), 89–125. United States Department of Education (2002). Beginning postsecondary students longitudinal study: 1996–2001 (bps: 1996/2001) methodology report. Technical report, National Center for Education Statistics. United States Department of Education (2004). Digest of education statistics 2002. Technical report, National Center for Education Statistics. United States Department of Education (2005). Beginning postsecondary students: data analysis system. Technical report, National Center for Education Statistics.