- Email: [email protected]
Evaluating an alcohol and drug treatment program for the homeless: An econometric approach
Evaluation and Program Planning, Vol. 20, No. 2, pp. 205-215, 1997 0 1997 Elsevier Science Ltd. All rights reserved
Pergamon
Printed in Great Britain 0149-7189/97
PII: SO149-7189(96)00054-7
$17.00+0.00
EVALUATING AN ALCOHOL AND DRUG TREATMENT PROGRAM FOR THE HOMELESS: AN ECONOMETRIC APPROACH
JOEL A. DEVINE,
CHARLES
J. BRODY and JAMES D. WRIGHT
Tulane
University
ABSTRACT The New Orleans Homeless Substance Abusers Project (NOHSAP) was designedas a randomized field experiment to test the effectiveness of a residential alcohol and drug treatment program on the sobriety, employment, housing, andsocial integration of homeless substance abusers. However, program staff sabotaged randomization into treatment and control groups, and research attrition was also non-random. Non-random assignment to treatment and non-random research attrition threaten internal and external validity by biasing OLS estimates of the effects of treatment and necessitate use of econometric selection bias correction modeling techniques. Results of these corrected models are then used in subsequent estimates of treatment effects on a variety of outcome measures. After correction, positive treatment effects prove relatively modest. However, subsequent analysis suggests that NOHSAP exerted a critical indirect effect on outcomes by facilitating subject’s participation in outside substance abuse groups. We conclude with some observations on the policy implications of the substantive results. 0 1997 Elsevier Science Ltd
PROGRAM DESIGN AND IMPLEMENTATION
INTRODUCTION The New Orleans Homeless Substance Abusers Project (NOHSAP) was a residential, adult resocialization project targeting homeless alcoholics and drug abusers in the Greater New Orleans area. NOHSAP was a didactically-oriented therapeutic community designed to achieve four principal outcomes: (1) a drug and alcohol free existence; (2) residential stability; (3) economic independence; and (4) a reduction in family estrangement and an increase in general social functioning. This paper, which constitutes only a part of a much larger report, provides an outcome analysis of these four programmatic goals. ’
NOHSAP was designed as a multi-phase intervention: DetoxiJication: a seven-day pre-treatment screening program focused on sobering, introduction to Alcoholics Anonymous (AA), Narcotics Anonymous (NA), and Cocaine Anonymous (CA) principles, daily group meetings, and limited counselling and assessment. Clients completing detoxification and judged clinically suitable for treatment comprised the pool from which treatment clients were to be randomly chosen.
1. Social
Research reported here was funded by the National Institute of Alcohol Abuse and Alcoholism (l-UOl-AA0877501). We are grateful to our colleague, Dan McMillen, for his assistance. Requests for reprints should be addressed to Joel A. Devine, Department of Sociology, Tulane University, 220 Newcomb Hall, New Orleans, LA 701185698, U.S.A. ‘A fuller program description is available in Wright, Devine, and Eddington (1993) while a more extensive evaluation is available in Wright, Devine, and Joyner (1993).
205
206
JOEL
A. DEVINE
Transitional Care (TC): a 21-day didactic treatment program involving extensive client assessment and case management, twice-daily group meetings and placement in an off-campus alcohol or drug group. Clients successfully completing TC became eligible for an extended treatment program. Extended Care/Independent Living (ECIL): a 12month program that continued and amplified the interventions and strategies begun during TC. ECIL was intended to add a GED program, job training, and job placement, but the latter services were never fully developed, so ECIL was for the most part only a longer-term version of TC (Wright et al., 1993). Between February 199 1 and April 1992,670 clients were baselined during the detox phase. Of these, 164 were eventually placed in treatment conditions (TC only: N = 107; TC+ ECIL: N = 57), with ECIL clients remaining in treatment for periods ranging from less than two months to more than 12 months. The remaining 506 subjects were released back to the streets after detox and constitute the controls.*
NOHSAP Clients The demographic profile of NOHSAP clients is broadly comparable to the overall profile of the homeless in New Orleans (Rudegair Associates, 1990) and the nation at large (Wright, 1989): 75% male, 84% non-white, average age of 34, with 80% under the age of 40. Just under half of the clients (48%) did not complete high school, and their employment pattern since was irregular and discontinuous. Eighty-five percent of the clients were crack abusers while 48% were alcohol abusive. Drugs other than crack and alcohol were mentioned as a primary problem by only 4%. Just over half the sample were poly-substance abusive, with crack and alcohol overwhelmingly the most common combination; well more than half had previously been in treatment for alcohol or drug disorders. In sum, NOHSAP clients were young, poor, black, unemployed, homeless, and addicted-in short the urban underclass (Devine & Wright, 1993).
EVALUATION DESIGN AND IMPLEMENTATION The overall design for the outcome evaluation straightforward in theory. Clients completing each gram phase were to be screened for suitability for ther treatment then randomized into or out of the
was profurnext
‘The relative sizes of the treatment and control groups were fixed by the availability of treatment slots. After initial start-up, TC and ECIL ran at full capacity, so most clients leaving detox were released back to the community rather than placed in treatment.
et al.
program phase. Randomization would assure initial equivalence between treatments and controls, aggressive tracking and follow-up would minimize attrition, and changes from baseline to follow-up would provide valid outcome indicators. Thus, the design seemed to protect admirably against threats to internal validity (Cook & Campbell, 1979). And since the control condition amounted to seven days of detox followed by release back to the streets, which is the “customary treatment” usually made available to homeless and indigent substance abusers nation-wide, the external validity of the design also seemed acceptable. The outcome analyses are based on data obtained from six-month, post-treatment, follow-up interviews. Of the 670 clients originally baselined, a follow-up interview was obtained for 620, or 93%.3
Randomization The plan at program start-up was to place every suitable client in Transitional Care until capacity was reached, thereafter randomizing clients into treatment from the pool of detoxed eligibles. Eligibility for further treatment was a clinical assessment made by program staff. Placement in Extended Care was to be accomplished in much the same manner; initial placement to capacity was to be made on a non-random basis, and thereafter entry was to be determined via randomization from the pool of staff-determined eligibles who had successfully completed Transitional Care. The randomization plan allowed clinical staff to “cream” the client stream by only placing the names of clients they considered sufficiently motivated or potentially likely to succeed on the selection lists. However, given the limited resources available and the ineffectiveness of most alcohol and drug treatment programs (see e.g., Hubbard, Marsden, Rachal, Harwood, Cavanaugh, & Ginsburg, 1989; Gerstein & Harwood, 1990; Akers, 1992), “creaming” would avoid squandering resources on unsuitable clients. Once the lists of eligible clients had been determined, however, the random selection of clients from the lists was done by the research staff.
‘Follow-up interviews were scheduled at 3,6, 12, and 18 months posttreatment. We focus on the 6-month follow-ups because it is wellknown that relapse normally occurs in the first six months subsequent to treatment (Gerstein & Harwood, 1990). To maximize cases available for analysis and to minimize problems of attrition, we created a “synthetic six-month file” containing the follow-up interview occurring closest in time to the six-month anniversary of the client’s release from treatment. Although this means that any follow-up interview can be treated as the six-month follow-up for purposes of the outcome analysis, the “synthetic” six-month interview is in fact the actual sixmonth interview in 89.9% of the cases, the three-month interview in 5.5% of the cases, the 12-month interview in 2.9% of the cases, and the 18-month interview in the remaining 1.6%. These percentages do not differ for treatments and controls (x2 = 3.98, d’= 3, p > .20).
207
Evaluating an Alcohol and Drug Treatment Program Data on the randomization of clients into the two treatment groups are summarized in Table 1. Only 32% of the 164 clients entering TC and 25% of the 57 clients going on to treatment in ECIL were in fact actually randomized. Excluding the first 24 (14%) and 20 (35%) clients in TC and ECIL respectively, the figures improve marginally, but only to 38%. As the data in Table 1 further show, 8 (or 5%) and 5 (9%) of the clients entering TC and ECIL were actually randomized out of treatment only to enter after all when other clients who had been randomized in chose not to enter the treatment programs. Clients randomized in but not opting for treatment numbered 11 (or 7% of TC) and 9 (16% of ECIL). Thus, even allowing that randomization would not commence until full treatment capacity had been achieved, almost half (48.5%) of the persons receiving TC and almost a third (32%) of those entering ECIL were selected in a manner inconsistent with the experimental design. Just how and why the NOHSAP staff managed to subvert the randomization scheme is a long and complicated story elsewhere told and analyzed in depth (Devine, Wright, & Joyner, 1993; Devine, Wright, & Brody, 1995). To summarize briefly: when a “good” (favored) client was randomized into treatment, the program staff implemented the selection; otherwise randomization was largely ignored. This obviously compromises the research design and experimental integrity of the project by introducing potentially substantial selection bias. As such, the departure from randomization must be taken into account and modelled in subsequent outcome analyses. Attrition While non-random selection of clients for treatment and control conditions can threaten the baseline equivalence
RANDOMIZATION
of treatment and control groups, differential attrition over time can likewise produce outcome differences that reflect methods artifacts rather than true experimental effects. Two different sorts of attrition were of concern in the NOHSAP experiment: research attrition and treatment attrition. (a) Research Attrition. As reported, 670 clients were baselined and all but 50 had a follow-up interview, so the net research attrition amounted to 50/670 = 7% of all cases. Of course, the primary concern with attrition is whether treatments and controls differentially attrite, but in NOHSAP this did not appear to be the case; 93% follow-up rates were achieved with both treatments and controls. While T-tests on 500+ variables from the baseline survey showed no clear pattern of difference between the two groups, we have little confidence that attrition was truly random (Devine et al., 1994, 1995). Thus, we also take this attrition into account in the outcome analyses reported below. (b) Treatment Attrition-Survival in Treatment. As noted, Transitional Care theoretically provided 21 days of treatment and Extended Care could have added another 12 months. However, the actual time in treatment was highly variable. Among the 164 treatment clients, 107 (65%) exited after Transitional Care (TC only), and the remaining 57 went on to ECIL (TC+ECIL). (For convenience, we refer to these two groups as TC and ECIL clients respectively, emphasizing, however, that all 57 ECIL clients also went through the TC program.) Days in treatment (i.e., not including days in detox) averaged 20 days for TC clients but varied from 4 to 35 days (SD = 4.8 days). In fact, 14% of the TC clients actually completed two or fewer weeks of treatment, while 12% completed more than the allotted 21 days.
TABLE 1 STATUS AND INCLUSION
PROBABILITIES Treatment group Transitional care N % N
Number of clients (N=) Number/percent of clients randomized in and actually entering program Number/percent randomized in with inclusion probability < 1 .OO’ Number/percent randomized in with inclusion probability = 1 .OOt Number/percent of clients randomized out but still entering program* Number/percent of clients randomized in but not entering program Number/percent of initial clients entering prior to start of randomization Number/percent of other (non-initial) entering clients not randomized *Number of eligible clients exceeds number of spaces available. tNumber of available spaces exceeds number of eligible clients. *Previously unanticipated space becomes available. §Percent excluding initial clients entering prior to randomization. YPercent of enrolled N. 11 Percent of enrolled N excluding initial clients entering prior to randomization.
164 53 27 26 8 11 24 80
100 32l38§ 16/l 90 161185 5165 7W8ll 14/481575
57 14 10 4 5 9 20 18
ECIL % 100 25138s 181279 71115 9/l 4s 16lV2411 35l321493
208
JOEL A. DEVINE
In theory, ECIL clients could have received a maximum of 386 days in treatment (21 days of TC plus 365 days of ECIL), but among the 57 ECIL clients, treatment averaged only 166 days (median days in treatment = 135) and varied from 16 to 396 days (SD = 109 days). Only three ECIL clients actually completed 386 or more days in treatment. Clients left either TC and ECIL prematurely for any number of reasons. Some were administratively discharged for disciplinary reasons, rules infractions, or relapse; some left prematurely of their own accord, against staff advice. Others left because they (and staff) felt they were ready for re-entry into the “real world.” Clinical data on discharge statuses show that only 17 (30%) of 57 ECIL clients actually graduated from the program. Detailed analyses of retention in treatment produced only marginally significant results (data not presented, see Wright et al., 1993b). Race emerged as a marginally significant predictor (blacks averaged more days in treatment than whites), but other background variables such as age, gender, and the like were always insignificant. One variable concerning the client’s homelessness history and a few variables reflecting clients’ alcohol and drug histories proved to be marginally significant predictors of retention in treatment, but most were not. Among the Addiction Severity Index (ASI) composite scores (see McLellan, Luborsky, Woody, & O’Brien, 1980; Fureman, Parikh, Bragg, & McLellan, 1990; McGahan, Griffith, Parente, & McLellan, 1991) only the alcohol variable predicted retention in treatment. Retention in treatment was not significantly related to any measure of psychological status, despite some studies showing that clients with moderate psychopathology are more readily retained in treatment than others (Joe, Singh, Garland, Lehman, Sells, & Seder, 1983; McLellan, 1983; McLellan, Luborsky, & Cacciola, 1985; Joe, Simpson, & Hubbard, 1991). The lack of any consistent pattern in the attritionfrom-treatment analysis is fairly persuasive evidence that the clinical staff did nor attempt to deal with the most troubled or problematic clients by finding reasons to terminate their treatment (which they certainly could have done). On the other hand, these findings leave unexplained a rather curious anomaly, namely, that so many ECIL clients left the program well in advance of their allotted twelve (or thirteen) months. This is an important substantive issue to which we return later.
OUTCOME
EVALUATION
As noted, NOHSAP was designed to facilitate four basic treatment objectives: (1) a drug and alcohol free existence; (2) residential stability; (3) economic inde-
et al
pendence; and (4) a reduction in family estrangement and an increase in general social functioning. We undertake an outcome analysis of measures of these (and related) constructs in two parts. We initially analyze possible treatment effects for those who obtained any treatment (i.e., treatments equal TC or ECIL) versus those who did not without taking into account any selection bias owing to the failure to randomize and/or research attrition. Thereafter, we employ econometric methods for dealing with selection bias. Both selection into treatment and research attrition are modeled as probits. These equations are then used to construct new variables which are used as instruments in the outcome equations to estimate the “true” treatment effects (see Heckman, 1979; Barnow, Cain, & Goldberger, 1980; Greene, 1981). Dependent Variables
Throughout these analyses, six follow-up measures are used to estimate the NOHSAP treatment effects: 1.
2.
3.
4.
5.
6.
Substance Free (a measure of sobriety): the number of days in the last 30 days in which subjects did not consume any alcohol or illicit drugs. Days Housed (a measure of residential stability): the number of days in the last 30 in which subjects were domiciled in conventional types of housing (including the dwellings of friends and/or family but excluding shelters, nights on the street, in treatment or correctional facilities, etc.). Working (a measure of employment): the number of days in the last 30 days in which subjects were working for pay in legal activities. Good Days (a summary measure of the above three constructs): is the sum of the number of days “substance free”, “housed”, and “working” divided by three. ASI Family/Social Integration: a widely-used composite measure of the extent or severity of a client’s family/social problems, The Addiction Severity Index (ASI) was developed specifically as a multifaceted instrument to monitor recovery from addiction disorders (see McLellan et al., 1980; Fureman et al., 1990; McGahan et al., 1991). The composite score is computed according to a standardized formula from variables contained in the family and social functioning section of the AS1 (where higher scores represent more severe problems). Attends AA/NA/CA Meetings: a dichotomous variable indicating whether the subject is currently maintaining participation in an Alcoholics Anonymous, Narcotics Anonymous, or Cocaine Anonymous-type group. As part of the NOHSAP treatment plan, each client was encouraged to locate and attend “off-site” meetings. This measure is used as an indicator of subject’s continuing motivation and commitment (or lack thereof) to a substance-free lifestyle.
Evaluating an Alcohol and Drug Treatment Program ANALYSIS AND RESULTS Each of the first five outcome measures is regressed on a treatment dummy (coded “1” for those who received any treatment [TC or ECIL] and “0” for the controls) and the baseline value using ordinary least squares estimation. Since the baseline values are included as regressors, the coefficients for the other variables represent the predicted change in the dependent variables for a unit increase in the particular independent variable. For the final dependent variable, attendance at AA/NA/CA meetings, logistic regression is used since it is dichotomous. In addition, there is no baseline measure of the variable; the strong presumption being that there was no such participation at entry into detox. Treatments Versus Controls
The preliminary outcome analyses are presented in Table 2, Panel A. Quick perusal of these models suggests that treatment appeared to have a positive (i.e., beneficial) impact on all of the assessed outcomes except employment (equation 3). While the coefficient associated with family/social integration is not statistically significant (equation 5), the sign is in the “right” direction. Moreover, the coefficients indexing the impact of treatment on sobriety (equation l), housing (equation 2), good days (equation 4), and attendance at meetings (equation 6) are promising with respect to both sign and significance. Selection Bias and Correction Procedures
While the reported preliminary results suggest several benefits accruing from NOHSAP treatment, the estimates are biased owing to (1) the fact that random assignment into treatment was seriously compromised, and (2) the likelihood that research attrition was not random. To correct for these problems we follow the two-step estimation approach developed by Heckman (1979), Barnow, Cain, and Goldberger (1980), and others. We correct for the first problem, i.e., non-random assignment, by treating assignment to treatment as an endogenous variable in a probit regression. Thereafter, we include the resultant inverse Mills ratio from the probit equation as an additional variable (ATreatment) in estimates of the effect of treatment on the various dependent variables. Standard errors for the OLS coefficients are then corrected (see Greene, 198 1, 1992). The second potential biasing factor, research attrition, is dealt with in the same manner: a probit model for attrition from the sample is estimated and an inverse Mills ratio term from the model (AAttrition)is then included in the regression for the treatment effect. The combination of these two bias corrections leads to a three-equation each of system for the dependent variables-two probit equations (one for assignment to treat-
209
ment, the other for sample attrition) and a final equation for the effect of treatment. The latter includes the inverse Mills ratios from both probit equations. The specific model for selection into treatment was developed out of the larger project process evaluation (see footnote 1). Qualitative evidence from the process evaluation suggested that clinical staff favored women and clients with dependent children in their care when they circumvented the randomized research design. There was also some indication that clients with more extensive psychiatric histories were more likely to be deemed inappropriate for treatment. We also hypothesized that better educated clients and those with a history of crack and/or alcohol abuse would be perceived as more appropriate to the didactic orientation and substantive emphases of the NOHSAP program. Finally, as noted above, space availability initially dictated some non-random assignment and space continued to be a relevant factor throughout the assignment process. Therefore, we included a variable indicating weeks in which a higher than average proportion of clients entered the program. A number of other predictors (i.e., race, age, homelessness history, and a number of other behavioral events and familial traits) were also considered, but none were significant in the probit models we considered. Except for the space measure which is unrelated to follow-up, our initial attrition models employed the same factors described above. In addition, a number of other locational factors and measures related to the existence of family ties and subject’s residential, employment, and criminal/legal histories were tested based on the knowledge gained from the extensive tracking operation associated with the research project (see Wright, Allen, & Devine, 1995). Panel B of Table 2 presents the results of the probit equations. The final model for assignment to treatment is presented on the left. Overall, the model is highly significant. As surmised, the probability of being assigned to treatment is higher for women and those with children; education is positively related to the probability of receiving treatment; and those with a history of more than three previous psychiatric treatments are less likely to be assigned treatment status. In addition, the combination of positive coefficients for the dummy variables, “Crack” and “Booze,” indicate that those whose substance abuse problem was not owing to crack cocaine or alcohol were less likely to receive treatment than those who primarily abused these two substances. Finally, the “space available” variable proved to be a strong predictor of later assignment to treatment. Probit coefficients from the model of research attrition, i.e., whether or not a post-treatment followup interview was conducted, are reported on the right side of Panel B. Attrition was more likely for women, older clients, and those who had a legal matter (arraign-
210
JOEL A. DEVINE
EFFECTS OF TREATMENT
TABLE 2 WITHOUT CORRECTION FOR BIAS AND PROBIT BIAS CORRECTIONS
Panel A-preliminary (1)
Baseline Treatment
Adj. R2 N
treatment effects without correctionst,*
(2)
(3)
(4)
Days in last 30 Housed Working
Substance free Constant
et al.
21.168§ (0.596) 0.106” (0.040) 2.267”* (0.938)
22.534 (0.705) 0.156 (0.031) 1.797*** (0.780)
0.020 604
0.051 594
Good days
(5) ASI family/social integration
16.462 (0.543) 0.176* (0.041) 1.196’*’ (0.550)
0.113 (0.014) 0.193* (0.033) - 0.008 (0.020)
6.123 (0.513) 0.280 (0.049) -0.346 (0.864) 0.050 599
0.038 569
0.056 549
(6) Attends AAINAICA meetings - 0.549 (0.097) 1.135* (0.196) [OR=3.11]fl -
TOLS estimates, except for Logit estimation where “Attends AAINAKA meetings” is dependent variable. *Positive treatment coefficients indicate improvement, except with ASI composite score where negative indicates improved functioning. SCoefficient and standard error. IOdds ratio. Significance: l p < .OOl; **p < .Ol; l **p < .05; ****p < .lO (two-tailed).
Selection into treatment Independent variable Gendert Children* Education Psych. history Crack Booze Space available Constant
Panel B-probit (N=666)
Coefficient
Standard error
0.725* 1.089’ 0.081* - 0.963** 0.613 0.410* 0.854* -2.964 Model x2 (7 df): 192.89
0.149 0.213 0.030 0.439 0.223 0.131 0.132 0.433
610
coefficient
bias correction models Selection for research attrition (N= 664) Independent variable Coefficient Standard error Genderf Education Age Years in area Pend. legal5 Local address Constant
0.417’ - 0.042 0.165” - 0.028* 0.519” - 0.294 0.905
0.167 0.037 0.009 0.006 0.214 0.196 0.561
Model x2 (6 df): 38.86
TFemale = 1. *Has dependent children in facility with them = 1. gHas current pending legal problems. Significance: ‘p Q .Ol; “p c .05; l **p < .lO (two-tailed).
ment, trial, etc.) pending at time of intake. Education, length of time resident in the metro area, and having a local mailing address at intake were negatively related to attrition. Table 3, Panel A contains the results of the models estimating treatment effects with the corrections for both non-random assignment (;lrreatment)and attrition (AAttrition) derived from the above probit models. Critically, in the case of days substance free, days housed, good days, and attendance at meetings, the previous pattern of positive treatment effects (per Table 2, Panel A) is enhanced once selection into treatment and research attrition are taken into account. For instance, net of everything else, the coefficient associated with substance free indicates that treatments have 3.7 more days (out of 30) of sobriety than do controls, whereas the preliminary equation indicated only 2.3 more days.
Similarly, the coefficient associated with treatment and days housed is now 4.9 days versus the earlier reported 1.8 days, and the odds ratio (i.e., treatments to controls differential) of attending AA-type meetings has increased from 3.11 to 7.64. At the same time, however, the previously non-significant coefficients associated with working and family/social integration remain nonsignificant while the sign of the latter has also changed.
DAYS IN TREATMENT Having ascertained that the NOHSAP treatment is associated with several beneficial outcomes we next consider whether the quantity of treatment received-as measured by the number of days in treatment -also
Evaluating an Alcohol and Drug Treatment Program
eficial impact suggesting that longer stays in treatment result in more favorable outcomes. To illustrate this in more concrete terms, consider the case of an Extended Care client with the median number of days of treatment (i.e., 135). As such, the coefficient associated with, for instance, “Good Days” (i.e., b = 0.013) would translate into an additional 1.755 (135 x 0.013) good days (out of the possible 30). Similarly, the odds of attendance (equation 6) would increase by a factor of 1.O1 per day in treatment. Thus, 135 days of treatment raises the odds of attending meetings by a factor of 3.83 (1.01’35) compared to the controls - above and beyond the effects associated with the
conveys positive behavioral results. These models, which are the same as the models estimated in Panel A of Table 3 except for the addition of a variable indexing the actual number of days in treatment, are presented in Table 3, Panel B. The results indicate that the number of days in treatment does not confer very substantialthough not trivial-additional benefits, though four of the six coefficients (i.e., those associated with work, good days, family/social integration, and attendance at meetings) are in the favorable direction and significant at p < .lO. In addition, in the two equations where treatment itself appears to have had an adverse effect (i.e., equations 3 and 5) days in treatment has a ben-
TREATMENT
EFFECTS
TABLE 3 MEASURES, WITH CORRECTION BIASESt$
ON OUTCOME
Panel A-basic (1)
Substance free Constant 1Trelmenf &nricon Baseline Treatment
Adj. R2 N
&“ti,O. Baseline Treatment
Days in treatment Adj. R2 N
(4)
Good days
22.338 (0.835) -2.365*** (0.937) -2.546 (3.13) 0.146’ (0.031) 4.094* (I .43)
7.871 (0.825) 1.311 (1.10) -9.474”^ (3.88) 0.260’ (0.050) - 1.746 (1.68)
17.126 (0.666) -0.457 (0.686) -5.601 ** (2.45) 0.170 (0.041) 1.955”” (1.06)
0.021 604
0.058 594
0.061 599
0.045 569
Substance free
ATreatment
(3)
Days in last 30 Housed Working
(2)
l
(3)
Days in last 30 Housed Working
(4)
Good days
22.329 (0.835) -2.379”’ (0.938) -2.548 (3.13) 0.147” (0.031) 4.664” (1.50)
7.851 (0.822) 1.215 (1.09) -9.532’*’ (3.87) 0.259’ (0.049) - 3.055”” (1.74)
17.139 (0.664) - 0.494 (0.684) -5.646’*’ (2.45) 0.169’ (0.041) 1.201 (1.10)
0.012 (0.010)
0.004 (0.008)
0.023”’ (0.009)
0.013”’ (0.006)
0.057 594
0.070 599
0.052 569
Table 3 continued over page
(5) ASI family/social integration 0.088 (0.019) - 0.046’*’ I. (0.025) 0.087 (0.093) 0.192 (0.033) 0.047 (0.038) 0.061 549
(6) Attends ANNA/CA meetings - 0.749 (0.158) - 0.690*’ (0.243) -0.125 (0.811) 2.033’ (0.381) [OR = 7.6411 610
models with addition of number of days in treatment
21.642 (0.838) - 1.029s (1.15) - 5.539 (3.90) 0.103** (0.040) 3.04Y”’ (1.84)
0.022 604
AND A-I-TRITION
treatment effects with probit corrections
(2)
Panel B-corrected
Constant
FOR SELECTION
21.624 (0.839) -0.9815 (1.15) - 5.536 (3.90) O.lOV’ (0.040) 3.710”’ (1.76)
(1)
211
(5) ASI family/social integration 0.088 (0.019) -0.045**** (0.025) 0.089 (0.093) 0.191’ (0.033) 0.067 (0.040) - 0.345E - 03**‘* (0.205E - 03) 0.064 549
(6) Attends AAINAICA meetings - 0.758 (0.158) - 0.725** (0.247) -0.140 (0.815) I .783 (0.399) [OR =5.95]1 0.005**** (0.003) [OR=l.Ol]lj 610
212
JOEL A. DEVINE et al. TABLE 3-continued. Panel C-corrected (1)
Substance free
models with estimated effects of education (3)
(2)
(4)
Days in last 30 Housed Working
Good days
(5) ASI family/social integration
12.492 (1.40) 0.136 (0.666) -4.767”’ (2.36) 0.150* (0.041) 0.053 (1.11)
0.120 (0.050) - 0.046”” (0.026) 0.063 (0.093) 0.191* (0.033) 0.074**** (0.041)
13.679 (2.20) 0.0555 (1.16) -4.046 (3.63) 0.066’** (0.040) 1.074 (1.66)
21.045 (1.66) -2.210*** (0.962) -2.294 (3.14) 0.146’ (0.031) 4.360** (1.55)
2.534 (2.16) 1.991 *** (1.11) -6.490”’ (3.61) 0.246* (0.049) - 4.475*** (1.79)
Days in treatment
0.013 (0.010)
0.004 (0.006)
0.022*** (0.009)
0.014*** (0.006)
Education
0.756 (0.193)
0.120 (0.156)
0.502’* (0.166)
0.451* (0.119)
Adj. R* N
0.045 604
0.056 594
0.062 599
0.077 569
Constant ATreatment &I,,” Baseline Treatment
l
(6) Attends ANNA/CA meetings - 2.300 (0.505) - 0.534”’ (0.255) 0.151 (0.627) -
1.446* (0.411) [OR = 4.2511 -0.344E - 03**” 0.005*** (0.205E-03) (0.003) [OR = 1 .Ol]l -0.003 0.142* (0.004) (0.043) [OR = 1 .15]1 0.063 549
TOLS estimates, except for Logit estimation where “Attends AA/NA/CA Meetings” is dependent variable. $Positive treatment coefficients indicate improvement, except with ASI composite score where negative indicates improved functioning. 0 Coefficient and standard error. 1 Odds ratio. Significance: *p < ,001; “p < .Ol; ***p < .05; l ***p < .lO (two-tailed).
treatment dummy and any of the other variables in the equation.
independent
Education Inasmuch as NOHSAP was designed as a didactic program, plus the fact that education proved to be a significant factor in selection and attrition, we next sought to examine whether subject’s education influenced outcomes. A straight-forward test of this proposition entailed the addition of a single variable indexing the number of years of formal education to the previously specified models. These analyses are reported in Table 3, Panel C. Examination of these estimates is quite illuminating; education is significantly related to four of the six outcomes (substance-free, working, good days, and attends meetings). Moreover, the magnitude of the coefficients is generally impressive. For instance, each year of education means an additional three-quarters of a day of sobriety (out of 30), and half a day (out of 30) of employment. Similarly, assuming 11 years of education (the sample mean), the 1.15 odds-ratio for the effect of one year of education on attendance at meetings (equation 6) means that a subject with 11 years of schooling has odds 4.65 times as great of attending AA-type meetings than a subject without any schooling. At the same time,
610
coefficient
however, these results suggest some caution inasmuch as the inclusion of the education term renders the treatment effect non-significant with respect to sobriety (equation l), arguably the most direct and straightforward outcome expected from a program such as the one under study here. One additional caveat is in order: the effect of the dummy variable treatment on days working is now significant as well as negative, though the coefficient indexing the actual number of days in treatment is positive and significant. The combination of these two effects suggests that any positive benefit of treatment vis-a-vis employment ensues only after relatively long stays in treatment. While some readers may find the negative treatment effect counter-intuitive and somewhat disturbing, we speculate that this owes to the fact that treatment clients were strongly discouraged from seeking employment until after they had established sobriety and some degree of stability within the program. Thus, in the vast majority of cases, employment only manifests itself as a behavioral goal after numerous months in treatment. Alternative Specifications An extensive literature focuses on the difficulty of maintaining substance-abusers in long-term treatment.
213
Evaluating an Alcohol and Drug Treatment Program Though an over-simplification, it would not be inaccurate to conclude that the sum of our collective knowledge regarding the efficacy of substance-abuse treatment suggests that almost any treatment can prove “effective” if and when clients are successfully maintained in treatment (see, e.g., Hubbard et al., 1989; DeLeon, 1990; Gerstein & Harwood, 1990). Based on this literature and the findings represented throughout Table 3 -wherein treatment exhibited a somewhat mixed set of effects on the analyzed outcomes, but rather consistently had a beneficial impact on attendance at meetings-we sought to test further the proposition that one potential positive impact of NOHSAP (and perhaps other similiar programs) was less direct and simply operated to motivate or induce substance abusers to establish a much-needed longer-term commitment to their ongoing recovery. With this in mind, we undertook one further set of analyses whereby attendance at AA/NA/CA meetings was treated as an additional independent rather than dependent variable. As such, we simply added this variable to the models previously specified in Table 3, Panel C, equations 1-5. The results from these specifications are presented in Table 4. Examination of these equations indicates that treatment itself no longer exhibits a positive and significant effect except with respect to housing (equation 2). In fact (and bearing in mind that this is net of all other
variables), the sign of the other treatment coefficients are all in the “wrong” (i.e., non-beneficial) direction. Coefficients indexing the impact of days in treatment are correctly signed (i.e., in the beneficial direction), but only achieve statistical significance with respect to working, good days, and family/social integration. Education continues to exert the same pattern and magnitude of effects as before. Critically, attendance at meetings has a fairly substantial effect on sobriety, housing, and good days. (Moreover, while attendance is not significant vi.+a-vis working and family/social integration, the signs are in the beneficial direction.) With respect to sobriety (equation l), attendance adds more than five (out of 30) days. Similarly, with regard to housing (equation 2) and the good days summary measure (equation 4), attendance adds 1.25 and 2.36 days respectively. These results largely are consistent with the notion that the primary benefit of the NOHSAP program derived from its positive impact on attendance.
DISCUSSION The goal of NOHSAP was to demonstrate that it is possible to take homeless substance abusers and turn them into responsible and productive adult citizens. There is some, but only some, comfort in recognizing that this goal was unreasonably ambitious to begin with.
TABLE 4 AAKXNA EFFECTS ON VARIOUS OUTCOME MEASURES, OLS ESTlMATESt
(1) Substance free Constant
Baseline Treatment Days in treatment Education Attends AAICAINA Adj. R* N
I 3.758
(2.13) 0.681$ (1.13) - 3.909 (3.71) 0.067’*” (0.039) - 0.667 (1.a4) 0.008 (0.009) 0.599” (0.169) 5.359’ (0.812) 0.109 600
(2)
(3)
Days in last 30 Housed Working 21.040 (1.aa) -2.062*** (0.966) - 2.220 (3.14) 0.143’ (0.031) 3.963”’ (1.56) 0.003 (0.00s) 0.086 (0.159) 1.246”** (0.693) 0.060 590
2.602 (2.16) 2.044”*** (1.11) -8.670*** (3.82) 0.264’ (0.050) -4.683”’ (1.60) 0.022”’ (0.009) 0.478”’ (0.190) 0.560 (0.774) 0.065 596
(4)
Good days 12.654 (1.38) 0.41 a (0.678) - 4.903”’ (2.35) 0.134’ (0.040) - 0.729 (1.11) 0.011 l ** (0.005) 0.383* (0.119) 2.355’ (0.486) 0.112
(5) ASI family/social integration 0.120 (0.050) - 0.488**** (0.026) 0.082 (0.094) o.laa* (0.033) 0.075**** (0.042) -0342E-03***’ (0.206E -03) - 0.003 (0.004) -0.003 (0.019) 0.060 545
TPositive treatment coefficients indicate improvement, except with ASI composite score where negative coefficient indicates improved functioning. *Coefficient and standard error. Significance: ‘p < ,001; “p G .Ol;“‘p Q .05; l ***p < .lO (two-tailed).
214
JOEL A. DEVINE
The process evaluation showed that many problems were encountered in the implementation of the project (including the failure to implement the randomization scheme), and that this had some serious implications for the outcome analyses, necessitating the use of econometric bias-correcting modeling techniques. Once corrected for selection and attrition bias, the subsequent outcome analysis indicates that NOHSAP apparently achieved limited success in producing sober, stably housed, socially integrated, (and perhaps to a lesser extent) employed former addicts. In general, longer-term clients showed modest (though not always significant) improvements in several areas of functioning, but few clients remained in treatment for the duration of the planned program. Why so many Extended Care clients left prematurely remains an open question. It is certainly possible that program-specific factors drove clients away, but the results of exit questionnaires showed that most clients were highly satisfied with most aspects of the NOHSAP program (see Wright et al., 1993b). It is also possible that the clinical staff unloaded clients at the first sign of trouble, retaining only those deemed more-certain bets to succeed. But our analysis of retention in treatment suggested rather persuasively that the clinical staff did not attempt to deal with the most troubled or problematic clients by finding reasons to terminate treatment. As noted, the available research literature points unambiguously to long-term retention as a significant problem. In fact, the low retention of clients in Extended Care was actually a bit better than average for interventions of this sort (Gerstein & Harwood, 1990). Thus, we are left, largely by default, with the hypothesis that there may be something intrinsic to the circumstances, personalities, or conditions of homeless addicts that precludes most of them from being retained in a treatment program for much more than a few months. Unfortunately, if the “minimum retention” needed to establish beneficial long-term outcomes is indeed on the order of several months, and if the homeless substance abusers studied in this research are typical, then the conclusion is that most homeless addicts cannot be retained in treatment long enough for treatment to do them much good directly. However, our analyses also indicate that treatment per se and time in treatment serve indirectly to facilitate recovery by connecting clients to attendance at AA-type meetings and presumably fostering the motivation and life skills to continue participation in supportive recovery groups of this sort. Readers should bear in mind that we are not claiming that there is anything intrinsic to AA-type treatment that has greater efficacy than any other substance abuse recovery philosophy or program per se, only that residential treatment of the NOHSAP sort may well buy much-needed time and begin to build the necessary
et al.
behavioral, cognitive, affective, and associational frame-work and connections critical to the long-term treatment and recovery that takes place in a wide variety of non-residential treatment programs. If this finding is widely generalizable - a question that awaits much more extensive empirical research - it may allow more efficient if not more efficacious use of limited treatment resources. Finally, it may resolve part of the ongoing dilemma of retention in treatment by serving to alter perhaps somewhat unrealistic and overly-ambitious expectations regarding residential substance abuse treatment. CONCLUSION For all the problems encountered in NOHSAP, it seems that the principal lesson comes through with some clarity: homeless addicts who can be retained in a NOHSAP-style treatment program for more than a few months can usually profit from the experience, but most homeless addicts cannot be. Nonetheless, residential treatment of the sort provided in NOHSAP does appear to indirectly facilitate many of the desired outcomes by motivating, supporting, and fostering connections with ongoing out-patient treatment modes that themselves exert positive and direct effects on desired outcomes. This lends credence to the notion that recovery from substance abuse should not be thought of as a solitary event or “cure,” but as a long-term therapeutic process often facilitated by treatment. REFERENCES Akers, R. andpolicy.
(1992).Drugs, alcohol and society: Social structure, process, Belmont,
CA: Wadsworth.
Barnow, B., Cain, G., & Goldberger, A. (1980). Issues in the analysis of selectivity bias. In W. Stromsdorfer & G. Farkas (Eds.), Evaluation studies review annuals (pp. 43-59). Beverly Hills, CA: Sage. Cook, T., & Campbell, D. (1979). Quasi-experimentation: Design and analysis issuesforfield settings. Chicago, IL: Rand McNally. DeLeon, G. (1990). Treatment strategies. In J.C. Inciardi (Ed.), Handbook of drug control in the United States (pp. 115-l 38). New York: Greenwood. Devine, J. A., & Wright, J. D. (1993). The greatest of evils: Urban poverty and the American underclass. Hawthorne, NY: Aldine de Gruyter. Devine, J. A., Wright, J. D., & Brody, C. (1995). An evaluation of an alcohol and drug treatment program for homeless substance abusers. Evaluation Review, 19. 62&645. Devine, J. A., Wright, J. D., & Joyner. L. (1994). Issues in implementing randomized experiments in field settings: The New Orleans homeNew Directions for Program less substance abuse program. Evaluation, 63, 27-40.
Evaluating
an Alcohol
and Drug Treatment
Fureman, B., Parikh, G., Bragg, A., & McLellan, A. (1990). Addiction severity index: A guide to training and supervising ASI interviews based on the past ten years. Philadelphia, PA: University of Pennsylvania/ V. A. Center for Studies of Addiction.
Program
215
McLellan, A. (1983). Patient characteristics associated with outcome. In J. Cooper, F. Altman, B. Brown&D. Czechowicz (Eds.), Research on the treatment of narcotic addiction: State of the art. Washington, DC: USGPO.
Gerstein, D., & Harwood, H. (eds.). (1990). Treating drug problems (Vol. 1). Washington, DC: National Academy Press.
McLellan, A., Luborsky, L., Cacciola, J., Griffith, J., Evans, F., Barr, H., &O’Brien, C. (1985). New data from the addiction severity index: Reliability and validity in three centers. Journal of Nervous and
Greene, W. (1981). Sample selection bias as a specification error: A comment. Econometrica, 49, 795-798.
Mental Disease, 173.412423.
Greene, W. (1992). LIMDEP, version 6.0: User S manual and reference guide. Bellport, NY: Econometric Software, Inc.
McLellan, A., Luborsky, L., Woody, G., & O’Brien, C. (1980). An improved diagnostic instrument for substance abuse patients: the addiction severity index. Journal of Nervous and Mental Disease, 168, 2&33.
Heckman, J. (1979). Sample selection bias as a specification error. Econometrica,
47, 153-161.
Hubbard, R. L., Marsden, M., Rachal, J., Harwood, H. J., Cavanaugh, E., & Ginsburg, M. (1989). Drug abuse treatment: A national study of effectiveness. Chapel Hill, NC: University of North Carolina Press. Joe, G., Simpson, D., & Hubbard, R. L. (1991). Treatment predictors of tenure in methadone maintenance. Journal of Substance Abuse, 3,
Rudegair Associates. (1990). Searching for shelter: An analysis of New Orleans’ homeless programs. New Orleans, LA: Rudegair Assoc.
metropolitan
Wright, J. D. (1989). Address unknown. Hawthorne, NY: Aldine de Gruyter. Wright, J. D., Allen, T., 8c Devine, J. A. (1995). Tracking nontraditional populations in longitudinal studies. Evaluation and Program Planning, 18, 267-277.
73-84.
Joe, G., Singh, B., Garland, J., Lehman, W., Sells, S., & Seder, P. (1983). Retention in outpatient drug-free treatment clinics. Journal
Wright, J. D., Devine, J. A., & Eddington, N. (1993). The New Orleans homeless substance abusers project. Alcoholism Treatment Quarterly, 10. 51-64.
of Addictive Behaviors, 8, 219-234.
Wright, J. D., Devine, J. A., & Joyner, L. (1993). The least of mine: McGahan, P., Griffith, J., Parente, R., & McLellan, A. (1991). Composite scores from the ASI. Washington, DC: NIDA Project No. DA02254 and the Veterans Administration.
The New Orleans homeless substance
abusers project:
Final report
(Technical Report, September). Bethesda, MD: National Institute of Alcohol Abuse and Alcoholism.
Pergamon
Printed in Great Britain 0149-7189/97
PII: SO149-7189(96)00054-7
$17.00+0.00
EVALUATING AN ALCOHOL AND DRUG TREATMENT PROGRAM FOR THE HOMELESS: AN ECONOMETRIC APPROACH
JOEL A. DEVINE,
CHARLES
J. BRODY and JAMES D. WRIGHT
Tulane
University
ABSTRACT The New Orleans Homeless Substance Abusers Project (NOHSAP) was designedas a randomized field experiment to test the effectiveness of a residential alcohol and drug treatment program on the sobriety, employment, housing, andsocial integration of homeless substance abusers. However, program staff sabotaged randomization into treatment and control groups, and research attrition was also non-random. Non-random assignment to treatment and non-random research attrition threaten internal and external validity by biasing OLS estimates of the effects of treatment and necessitate use of econometric selection bias correction modeling techniques. Results of these corrected models are then used in subsequent estimates of treatment effects on a variety of outcome measures. After correction, positive treatment effects prove relatively modest. However, subsequent analysis suggests that NOHSAP exerted a critical indirect effect on outcomes by facilitating subject’s participation in outside substance abuse groups. We conclude with some observations on the policy implications of the substantive results. 0 1997 Elsevier Science Ltd
PROGRAM DESIGN AND IMPLEMENTATION
INTRODUCTION The New Orleans Homeless Substance Abusers Project (NOHSAP) was a residential, adult resocialization project targeting homeless alcoholics and drug abusers in the Greater New Orleans area. NOHSAP was a didactically-oriented therapeutic community designed to achieve four principal outcomes: (1) a drug and alcohol free existence; (2) residential stability; (3) economic independence; and (4) a reduction in family estrangement and an increase in general social functioning. This paper, which constitutes only a part of a much larger report, provides an outcome analysis of these four programmatic goals. ’
NOHSAP was designed as a multi-phase intervention: DetoxiJication: a seven-day pre-treatment screening program focused on sobering, introduction to Alcoholics Anonymous (AA), Narcotics Anonymous (NA), and Cocaine Anonymous (CA) principles, daily group meetings, and limited counselling and assessment. Clients completing detoxification and judged clinically suitable for treatment comprised the pool from which treatment clients were to be randomly chosen.
1. Social
Research reported here was funded by the National Institute of Alcohol Abuse and Alcoholism (l-UOl-AA0877501). We are grateful to our colleague, Dan McMillen, for his assistance. Requests for reprints should be addressed to Joel A. Devine, Department of Sociology, Tulane University, 220 Newcomb Hall, New Orleans, LA 701185698, U.S.A. ‘A fuller program description is available in Wright, Devine, and Eddington (1993) while a more extensive evaluation is available in Wright, Devine, and Joyner (1993).
205
206
JOEL
A. DEVINE
Transitional Care (TC): a 21-day didactic treatment program involving extensive client assessment and case management, twice-daily group meetings and placement in an off-campus alcohol or drug group. Clients successfully completing TC became eligible for an extended treatment program. Extended Care/Independent Living (ECIL): a 12month program that continued and amplified the interventions and strategies begun during TC. ECIL was intended to add a GED program, job training, and job placement, but the latter services were never fully developed, so ECIL was for the most part only a longer-term version of TC (Wright et al., 1993). Between February 199 1 and April 1992,670 clients were baselined during the detox phase. Of these, 164 were eventually placed in treatment conditions (TC only: N = 107; TC+ ECIL: N = 57), with ECIL clients remaining in treatment for periods ranging from less than two months to more than 12 months. The remaining 506 subjects were released back to the streets after detox and constitute the controls.*
NOHSAP Clients The demographic profile of NOHSAP clients is broadly comparable to the overall profile of the homeless in New Orleans (Rudegair Associates, 1990) and the nation at large (Wright, 1989): 75% male, 84% non-white, average age of 34, with 80% under the age of 40. Just under half of the clients (48%) did not complete high school, and their employment pattern since was irregular and discontinuous. Eighty-five percent of the clients were crack abusers while 48% were alcohol abusive. Drugs other than crack and alcohol were mentioned as a primary problem by only 4%. Just over half the sample were poly-substance abusive, with crack and alcohol overwhelmingly the most common combination; well more than half had previously been in treatment for alcohol or drug disorders. In sum, NOHSAP clients were young, poor, black, unemployed, homeless, and addicted-in short the urban underclass (Devine & Wright, 1993).
EVALUATION DESIGN AND IMPLEMENTATION The overall design for the outcome evaluation straightforward in theory. Clients completing each gram phase were to be screened for suitability for ther treatment then randomized into or out of the
was profurnext
‘The relative sizes of the treatment and control groups were fixed by the availability of treatment slots. After initial start-up, TC and ECIL ran at full capacity, so most clients leaving detox were released back to the community rather than placed in treatment.
et al.
program phase. Randomization would assure initial equivalence between treatments and controls, aggressive tracking and follow-up would minimize attrition, and changes from baseline to follow-up would provide valid outcome indicators. Thus, the design seemed to protect admirably against threats to internal validity (Cook & Campbell, 1979). And since the control condition amounted to seven days of detox followed by release back to the streets, which is the “customary treatment” usually made available to homeless and indigent substance abusers nation-wide, the external validity of the design also seemed acceptable. The outcome analyses are based on data obtained from six-month, post-treatment, follow-up interviews. Of the 670 clients originally baselined, a follow-up interview was obtained for 620, or 93%.3
Randomization The plan at program start-up was to place every suitable client in Transitional Care until capacity was reached, thereafter randomizing clients into treatment from the pool of detoxed eligibles. Eligibility for further treatment was a clinical assessment made by program staff. Placement in Extended Care was to be accomplished in much the same manner; initial placement to capacity was to be made on a non-random basis, and thereafter entry was to be determined via randomization from the pool of staff-determined eligibles who had successfully completed Transitional Care. The randomization plan allowed clinical staff to “cream” the client stream by only placing the names of clients they considered sufficiently motivated or potentially likely to succeed on the selection lists. However, given the limited resources available and the ineffectiveness of most alcohol and drug treatment programs (see e.g., Hubbard, Marsden, Rachal, Harwood, Cavanaugh, & Ginsburg, 1989; Gerstein & Harwood, 1990; Akers, 1992), “creaming” would avoid squandering resources on unsuitable clients. Once the lists of eligible clients had been determined, however, the random selection of clients from the lists was done by the research staff.
‘Follow-up interviews were scheduled at 3,6, 12, and 18 months posttreatment. We focus on the 6-month follow-ups because it is wellknown that relapse normally occurs in the first six months subsequent to treatment (Gerstein & Harwood, 1990). To maximize cases available for analysis and to minimize problems of attrition, we created a “synthetic six-month file” containing the follow-up interview occurring closest in time to the six-month anniversary of the client’s release from treatment. Although this means that any follow-up interview can be treated as the six-month follow-up for purposes of the outcome analysis, the “synthetic” six-month interview is in fact the actual sixmonth interview in 89.9% of the cases, the three-month interview in 5.5% of the cases, the 12-month interview in 2.9% of the cases, and the 18-month interview in the remaining 1.6%. These percentages do not differ for treatments and controls (x2 = 3.98, d’= 3, p > .20).
207
Evaluating an Alcohol and Drug Treatment Program Data on the randomization of clients into the two treatment groups are summarized in Table 1. Only 32% of the 164 clients entering TC and 25% of the 57 clients going on to treatment in ECIL were in fact actually randomized. Excluding the first 24 (14%) and 20 (35%) clients in TC and ECIL respectively, the figures improve marginally, but only to 38%. As the data in Table 1 further show, 8 (or 5%) and 5 (9%) of the clients entering TC and ECIL were actually randomized out of treatment only to enter after all when other clients who had been randomized in chose not to enter the treatment programs. Clients randomized in but not opting for treatment numbered 11 (or 7% of TC) and 9 (16% of ECIL). Thus, even allowing that randomization would not commence until full treatment capacity had been achieved, almost half (48.5%) of the persons receiving TC and almost a third (32%) of those entering ECIL were selected in a manner inconsistent with the experimental design. Just how and why the NOHSAP staff managed to subvert the randomization scheme is a long and complicated story elsewhere told and analyzed in depth (Devine, Wright, & Joyner, 1993; Devine, Wright, & Brody, 1995). To summarize briefly: when a “good” (favored) client was randomized into treatment, the program staff implemented the selection; otherwise randomization was largely ignored. This obviously compromises the research design and experimental integrity of the project by introducing potentially substantial selection bias. As such, the departure from randomization must be taken into account and modelled in subsequent outcome analyses. Attrition While non-random selection of clients for treatment and control conditions can threaten the baseline equivalence
RANDOMIZATION
of treatment and control groups, differential attrition over time can likewise produce outcome differences that reflect methods artifacts rather than true experimental effects. Two different sorts of attrition were of concern in the NOHSAP experiment: research attrition and treatment attrition. (a) Research Attrition. As reported, 670 clients were baselined and all but 50 had a follow-up interview, so the net research attrition amounted to 50/670 = 7% of all cases. Of course, the primary concern with attrition is whether treatments and controls differentially attrite, but in NOHSAP this did not appear to be the case; 93% follow-up rates were achieved with both treatments and controls. While T-tests on 500+ variables from the baseline survey showed no clear pattern of difference between the two groups, we have little confidence that attrition was truly random (Devine et al., 1994, 1995). Thus, we also take this attrition into account in the outcome analyses reported below. (b) Treatment Attrition-Survival in Treatment. As noted, Transitional Care theoretically provided 21 days of treatment and Extended Care could have added another 12 months. However, the actual time in treatment was highly variable. Among the 164 treatment clients, 107 (65%) exited after Transitional Care (TC only), and the remaining 57 went on to ECIL (TC+ECIL). (For convenience, we refer to these two groups as TC and ECIL clients respectively, emphasizing, however, that all 57 ECIL clients also went through the TC program.) Days in treatment (i.e., not including days in detox) averaged 20 days for TC clients but varied from 4 to 35 days (SD = 4.8 days). In fact, 14% of the TC clients actually completed two or fewer weeks of treatment, while 12% completed more than the allotted 21 days.
TABLE 1 STATUS AND INCLUSION
PROBABILITIES Treatment group Transitional care N % N
Number of clients (N=) Number/percent of clients randomized in and actually entering program Number/percent randomized in with inclusion probability < 1 .OO’ Number/percent randomized in with inclusion probability = 1 .OOt Number/percent of clients randomized out but still entering program* Number/percent of clients randomized in but not entering program Number/percent of initial clients entering prior to start of randomization Number/percent of other (non-initial) entering clients not randomized *Number of eligible clients exceeds number of spaces available. tNumber of available spaces exceeds number of eligible clients. *Previously unanticipated space becomes available. §Percent excluding initial clients entering prior to randomization. YPercent of enrolled N. 11 Percent of enrolled N excluding initial clients entering prior to randomization.
164 53 27 26 8 11 24 80
100 32l38§ 16/l 90 161185 5165 7W8ll 14/481575
57 14 10 4 5 9 20 18
ECIL % 100 25138s 181279 71115 9/l 4s 16lV2411 35l321493
208
JOEL A. DEVINE
In theory, ECIL clients could have received a maximum of 386 days in treatment (21 days of TC plus 365 days of ECIL), but among the 57 ECIL clients, treatment averaged only 166 days (median days in treatment = 135) and varied from 16 to 396 days (SD = 109 days). Only three ECIL clients actually completed 386 or more days in treatment. Clients left either TC and ECIL prematurely for any number of reasons. Some were administratively discharged for disciplinary reasons, rules infractions, or relapse; some left prematurely of their own accord, against staff advice. Others left because they (and staff) felt they were ready for re-entry into the “real world.” Clinical data on discharge statuses show that only 17 (30%) of 57 ECIL clients actually graduated from the program. Detailed analyses of retention in treatment produced only marginally significant results (data not presented, see Wright et al., 1993b). Race emerged as a marginally significant predictor (blacks averaged more days in treatment than whites), but other background variables such as age, gender, and the like were always insignificant. One variable concerning the client’s homelessness history and a few variables reflecting clients’ alcohol and drug histories proved to be marginally significant predictors of retention in treatment, but most were not. Among the Addiction Severity Index (ASI) composite scores (see McLellan, Luborsky, Woody, & O’Brien, 1980; Fureman, Parikh, Bragg, & McLellan, 1990; McGahan, Griffith, Parente, & McLellan, 1991) only the alcohol variable predicted retention in treatment. Retention in treatment was not significantly related to any measure of psychological status, despite some studies showing that clients with moderate psychopathology are more readily retained in treatment than others (Joe, Singh, Garland, Lehman, Sells, & Seder, 1983; McLellan, 1983; McLellan, Luborsky, & Cacciola, 1985; Joe, Simpson, & Hubbard, 1991). The lack of any consistent pattern in the attritionfrom-treatment analysis is fairly persuasive evidence that the clinical staff did nor attempt to deal with the most troubled or problematic clients by finding reasons to terminate their treatment (which they certainly could have done). On the other hand, these findings leave unexplained a rather curious anomaly, namely, that so many ECIL clients left the program well in advance of their allotted twelve (or thirteen) months. This is an important substantive issue to which we return later.
OUTCOME
EVALUATION
As noted, NOHSAP was designed to facilitate four basic treatment objectives: (1) a drug and alcohol free existence; (2) residential stability; (3) economic inde-
et al
pendence; and (4) a reduction in family estrangement and an increase in general social functioning. We undertake an outcome analysis of measures of these (and related) constructs in two parts. We initially analyze possible treatment effects for those who obtained any treatment (i.e., treatments equal TC or ECIL) versus those who did not without taking into account any selection bias owing to the failure to randomize and/or research attrition. Thereafter, we employ econometric methods for dealing with selection bias. Both selection into treatment and research attrition are modeled as probits. These equations are then used to construct new variables which are used as instruments in the outcome equations to estimate the “true” treatment effects (see Heckman, 1979; Barnow, Cain, & Goldberger, 1980; Greene, 1981). Dependent Variables
Throughout these analyses, six follow-up measures are used to estimate the NOHSAP treatment effects: 1.
2.
3.
4.
5.
6.
Substance Free (a measure of sobriety): the number of days in the last 30 days in which subjects did not consume any alcohol or illicit drugs. Days Housed (a measure of residential stability): the number of days in the last 30 in which subjects were domiciled in conventional types of housing (including the dwellings of friends and/or family but excluding shelters, nights on the street, in treatment or correctional facilities, etc.). Working (a measure of employment): the number of days in the last 30 days in which subjects were working for pay in legal activities. Good Days (a summary measure of the above three constructs): is the sum of the number of days “substance free”, “housed”, and “working” divided by three. ASI Family/Social Integration: a widely-used composite measure of the extent or severity of a client’s family/social problems, The Addiction Severity Index (ASI) was developed specifically as a multifaceted instrument to monitor recovery from addiction disorders (see McLellan et al., 1980; Fureman et al., 1990; McGahan et al., 1991). The composite score is computed according to a standardized formula from variables contained in the family and social functioning section of the AS1 (where higher scores represent more severe problems). Attends AA/NA/CA Meetings: a dichotomous variable indicating whether the subject is currently maintaining participation in an Alcoholics Anonymous, Narcotics Anonymous, or Cocaine Anonymous-type group. As part of the NOHSAP treatment plan, each client was encouraged to locate and attend “off-site” meetings. This measure is used as an indicator of subject’s continuing motivation and commitment (or lack thereof) to a substance-free lifestyle.
Evaluating an Alcohol and Drug Treatment Program ANALYSIS AND RESULTS Each of the first five outcome measures is regressed on a treatment dummy (coded “1” for those who received any treatment [TC or ECIL] and “0” for the controls) and the baseline value using ordinary least squares estimation. Since the baseline values are included as regressors, the coefficients for the other variables represent the predicted change in the dependent variables for a unit increase in the particular independent variable. For the final dependent variable, attendance at AA/NA/CA meetings, logistic regression is used since it is dichotomous. In addition, there is no baseline measure of the variable; the strong presumption being that there was no such participation at entry into detox. Treatments Versus Controls
The preliminary outcome analyses are presented in Table 2, Panel A. Quick perusal of these models suggests that treatment appeared to have a positive (i.e., beneficial) impact on all of the assessed outcomes except employment (equation 3). While the coefficient associated with family/social integration is not statistically significant (equation 5), the sign is in the “right” direction. Moreover, the coefficients indexing the impact of treatment on sobriety (equation l), housing (equation 2), good days (equation 4), and attendance at meetings (equation 6) are promising with respect to both sign and significance. Selection Bias and Correction Procedures
While the reported preliminary results suggest several benefits accruing from NOHSAP treatment, the estimates are biased owing to (1) the fact that random assignment into treatment was seriously compromised, and (2) the likelihood that research attrition was not random. To correct for these problems we follow the two-step estimation approach developed by Heckman (1979), Barnow, Cain, and Goldberger (1980), and others. We correct for the first problem, i.e., non-random assignment, by treating assignment to treatment as an endogenous variable in a probit regression. Thereafter, we include the resultant inverse Mills ratio from the probit equation as an additional variable (ATreatment) in estimates of the effect of treatment on the various dependent variables. Standard errors for the OLS coefficients are then corrected (see Greene, 198 1, 1992). The second potential biasing factor, research attrition, is dealt with in the same manner: a probit model for attrition from the sample is estimated and an inverse Mills ratio term from the model (AAttrition)is then included in the regression for the treatment effect. The combination of these two bias corrections leads to a three-equation each of system for the dependent variables-two probit equations (one for assignment to treat-
209
ment, the other for sample attrition) and a final equation for the effect of treatment. The latter includes the inverse Mills ratios from both probit equations. The specific model for selection into treatment was developed out of the larger project process evaluation (see footnote 1). Qualitative evidence from the process evaluation suggested that clinical staff favored women and clients with dependent children in their care when they circumvented the randomized research design. There was also some indication that clients with more extensive psychiatric histories were more likely to be deemed inappropriate for treatment. We also hypothesized that better educated clients and those with a history of crack and/or alcohol abuse would be perceived as more appropriate to the didactic orientation and substantive emphases of the NOHSAP program. Finally, as noted above, space availability initially dictated some non-random assignment and space continued to be a relevant factor throughout the assignment process. Therefore, we included a variable indicating weeks in which a higher than average proportion of clients entered the program. A number of other predictors (i.e., race, age, homelessness history, and a number of other behavioral events and familial traits) were also considered, but none were significant in the probit models we considered. Except for the space measure which is unrelated to follow-up, our initial attrition models employed the same factors described above. In addition, a number of other locational factors and measures related to the existence of family ties and subject’s residential, employment, and criminal/legal histories were tested based on the knowledge gained from the extensive tracking operation associated with the research project (see Wright, Allen, & Devine, 1995). Panel B of Table 2 presents the results of the probit equations. The final model for assignment to treatment is presented on the left. Overall, the model is highly significant. As surmised, the probability of being assigned to treatment is higher for women and those with children; education is positively related to the probability of receiving treatment; and those with a history of more than three previous psychiatric treatments are less likely to be assigned treatment status. In addition, the combination of positive coefficients for the dummy variables, “Crack” and “Booze,” indicate that those whose substance abuse problem was not owing to crack cocaine or alcohol were less likely to receive treatment than those who primarily abused these two substances. Finally, the “space available” variable proved to be a strong predictor of later assignment to treatment. Probit coefficients from the model of research attrition, i.e., whether or not a post-treatment followup interview was conducted, are reported on the right side of Panel B. Attrition was more likely for women, older clients, and those who had a legal matter (arraign-
210
JOEL A. DEVINE
EFFECTS OF TREATMENT
TABLE 2 WITHOUT CORRECTION FOR BIAS AND PROBIT BIAS CORRECTIONS
Panel A-preliminary (1)
Baseline Treatment
Adj. R2 N
treatment effects without correctionst,*
(2)
(3)
(4)
Days in last 30 Housed Working
Substance free Constant
et al.
21.168§ (0.596) 0.106” (0.040) 2.267”* (0.938)
22.534 (0.705) 0.156 (0.031) 1.797*** (0.780)
0.020 604
0.051 594
Good days
(5) ASI family/social integration
16.462 (0.543) 0.176* (0.041) 1.196’*’ (0.550)
0.113 (0.014) 0.193* (0.033) - 0.008 (0.020)
6.123 (0.513) 0.280 (0.049) -0.346 (0.864) 0.050 599
0.038 569
0.056 549
(6) Attends AAINAICA meetings - 0.549 (0.097) 1.135* (0.196) [OR=3.11]fl -
TOLS estimates, except for Logit estimation where “Attends AAINAKA meetings” is dependent variable. *Positive treatment coefficients indicate improvement, except with ASI composite score where negative indicates improved functioning. SCoefficient and standard error. IOdds ratio. Significance: l p < .OOl; **p < .Ol; l **p < .05; ****p < .lO (two-tailed).
Selection into treatment Independent variable Gendert Children* Education Psych. history Crack Booze Space available Constant
Panel B-probit (N=666)
Coefficient
Standard error
0.725* 1.089’ 0.081* - 0.963** 0.613 0.410* 0.854* -2.964 Model x2 (7 df): 192.89
0.149 0.213 0.030 0.439 0.223 0.131 0.132 0.433
610
coefficient
bias correction models Selection for research attrition (N= 664) Independent variable Coefficient Standard error Genderf Education Age Years in area Pend. legal5 Local address Constant
0.417’ - 0.042 0.165” - 0.028* 0.519” - 0.294 0.905
0.167 0.037 0.009 0.006 0.214 0.196 0.561
Model x2 (6 df): 38.86
TFemale = 1. *Has dependent children in facility with them = 1. gHas current pending legal problems. Significance: ‘p Q .Ol; “p c .05; l **p < .lO (two-tailed).
ment, trial, etc.) pending at time of intake. Education, length of time resident in the metro area, and having a local mailing address at intake were negatively related to attrition. Table 3, Panel A contains the results of the models estimating treatment effects with the corrections for both non-random assignment (;lrreatment)and attrition (AAttrition) derived from the above probit models. Critically, in the case of days substance free, days housed, good days, and attendance at meetings, the previous pattern of positive treatment effects (per Table 2, Panel A) is enhanced once selection into treatment and research attrition are taken into account. For instance, net of everything else, the coefficient associated with substance free indicates that treatments have 3.7 more days (out of 30) of sobriety than do controls, whereas the preliminary equation indicated only 2.3 more days.
Similarly, the coefficient associated with treatment and days housed is now 4.9 days versus the earlier reported 1.8 days, and the odds ratio (i.e., treatments to controls differential) of attending AA-type meetings has increased from 3.11 to 7.64. At the same time, however, the previously non-significant coefficients associated with working and family/social integration remain nonsignificant while the sign of the latter has also changed.
DAYS IN TREATMENT Having ascertained that the NOHSAP treatment is associated with several beneficial outcomes we next consider whether the quantity of treatment received-as measured by the number of days in treatment -also
Evaluating an Alcohol and Drug Treatment Program
eficial impact suggesting that longer stays in treatment result in more favorable outcomes. To illustrate this in more concrete terms, consider the case of an Extended Care client with the median number of days of treatment (i.e., 135). As such, the coefficient associated with, for instance, “Good Days” (i.e., b = 0.013) would translate into an additional 1.755 (135 x 0.013) good days (out of the possible 30). Similarly, the odds of attendance (equation 6) would increase by a factor of 1.O1 per day in treatment. Thus, 135 days of treatment raises the odds of attending meetings by a factor of 3.83 (1.01’35) compared to the controls - above and beyond the effects associated with the
conveys positive behavioral results. These models, which are the same as the models estimated in Panel A of Table 3 except for the addition of a variable indexing the actual number of days in treatment, are presented in Table 3, Panel B. The results indicate that the number of days in treatment does not confer very substantialthough not trivial-additional benefits, though four of the six coefficients (i.e., those associated with work, good days, family/social integration, and attendance at meetings) are in the favorable direction and significant at p < .lO. In addition, in the two equations where treatment itself appears to have had an adverse effect (i.e., equations 3 and 5) days in treatment has a ben-
TREATMENT
EFFECTS
TABLE 3 MEASURES, WITH CORRECTION BIASESt$
ON OUTCOME
Panel A-basic (1)
Substance free Constant 1Trelmenf &nricon Baseline Treatment
Adj. R2 N
&“ti,O. Baseline Treatment
Days in treatment Adj. R2 N
(4)
Good days
22.338 (0.835) -2.365*** (0.937) -2.546 (3.13) 0.146’ (0.031) 4.094* (I .43)
7.871 (0.825) 1.311 (1.10) -9.474”^ (3.88) 0.260’ (0.050) - 1.746 (1.68)
17.126 (0.666) -0.457 (0.686) -5.601 ** (2.45) 0.170 (0.041) 1.955”” (1.06)
0.021 604
0.058 594
0.061 599
0.045 569
Substance free
ATreatment
(3)
Days in last 30 Housed Working
(2)
l
(3)
Days in last 30 Housed Working
(4)
Good days
22.329 (0.835) -2.379”’ (0.938) -2.548 (3.13) 0.147” (0.031) 4.664” (1.50)
7.851 (0.822) 1.215 (1.09) -9.532’*’ (3.87) 0.259’ (0.049) - 3.055”” (1.74)
17.139 (0.664) - 0.494 (0.684) -5.646’*’ (2.45) 0.169’ (0.041) 1.201 (1.10)
0.012 (0.010)
0.004 (0.008)
0.023”’ (0.009)
0.013”’ (0.006)
0.057 594
0.070 599
0.052 569
Table 3 continued over page
(5) ASI family/social integration 0.088 (0.019) - 0.046’*’ I. (0.025) 0.087 (0.093) 0.192 (0.033) 0.047 (0.038) 0.061 549
(6) Attends ANNA/CA meetings - 0.749 (0.158) - 0.690*’ (0.243) -0.125 (0.811) 2.033’ (0.381) [OR = 7.6411 610
models with addition of number of days in treatment
21.642 (0.838) - 1.029s (1.15) - 5.539 (3.90) 0.103** (0.040) 3.04Y”’ (1.84)
0.022 604
AND A-I-TRITION
treatment effects with probit corrections
(2)
Panel B-corrected
Constant
FOR SELECTION
21.624 (0.839) -0.9815 (1.15) - 5.536 (3.90) O.lOV’ (0.040) 3.710”’ (1.76)
(1)
211
(5) ASI family/social integration 0.088 (0.019) -0.045**** (0.025) 0.089 (0.093) 0.191’ (0.033) 0.067 (0.040) - 0.345E - 03**‘* (0.205E - 03) 0.064 549
(6) Attends AAINAICA meetings - 0.758 (0.158) - 0.725** (0.247) -0.140 (0.815) I .783 (0.399) [OR =5.95]1 0.005**** (0.003) [OR=l.Ol]lj 610
212
JOEL A. DEVINE et al. TABLE 3-continued. Panel C-corrected (1)
Substance free
models with estimated effects of education (3)
(2)
(4)
Days in last 30 Housed Working
Good days
(5) ASI family/social integration
12.492 (1.40) 0.136 (0.666) -4.767”’ (2.36) 0.150* (0.041) 0.053 (1.11)
0.120 (0.050) - 0.046”” (0.026) 0.063 (0.093) 0.191* (0.033) 0.074**** (0.041)
13.679 (2.20) 0.0555 (1.16) -4.046 (3.63) 0.066’** (0.040) 1.074 (1.66)
21.045 (1.66) -2.210*** (0.962) -2.294 (3.14) 0.146’ (0.031) 4.360** (1.55)
2.534 (2.16) 1.991 *** (1.11) -6.490”’ (3.61) 0.246* (0.049) - 4.475*** (1.79)
Days in treatment
0.013 (0.010)
0.004 (0.006)
0.022*** (0.009)
0.014*** (0.006)
Education
0.756 (0.193)
0.120 (0.156)
0.502’* (0.166)
0.451* (0.119)
Adj. R* N
0.045 604
0.056 594
0.062 599
0.077 569
Constant ATreatment &I,,” Baseline Treatment
l
(6) Attends ANNA/CA meetings - 2.300 (0.505) - 0.534”’ (0.255) 0.151 (0.627) -
1.446* (0.411) [OR = 4.2511 -0.344E - 03**” 0.005*** (0.205E-03) (0.003) [OR = 1 .Ol]l -0.003 0.142* (0.004) (0.043) [OR = 1 .15]1 0.063 549
TOLS estimates, except for Logit estimation where “Attends AA/NA/CA Meetings” is dependent variable. $Positive treatment coefficients indicate improvement, except with ASI composite score where negative indicates improved functioning. 0 Coefficient and standard error. 1 Odds ratio. Significance: *p < ,001; “p < .Ol; ***p < .05; l ***p < .lO (two-tailed).
treatment dummy and any of the other variables in the equation.
independent
Education Inasmuch as NOHSAP was designed as a didactic program, plus the fact that education proved to be a significant factor in selection and attrition, we next sought to examine whether subject’s education influenced outcomes. A straight-forward test of this proposition entailed the addition of a single variable indexing the number of years of formal education to the previously specified models. These analyses are reported in Table 3, Panel C. Examination of these estimates is quite illuminating; education is significantly related to four of the six outcomes (substance-free, working, good days, and attends meetings). Moreover, the magnitude of the coefficients is generally impressive. For instance, each year of education means an additional three-quarters of a day of sobriety (out of 30), and half a day (out of 30) of employment. Similarly, assuming 11 years of education (the sample mean), the 1.15 odds-ratio for the effect of one year of education on attendance at meetings (equation 6) means that a subject with 11 years of schooling has odds 4.65 times as great of attending AA-type meetings than a subject without any schooling. At the same time,
610
coefficient
however, these results suggest some caution inasmuch as the inclusion of the education term renders the treatment effect non-significant with respect to sobriety (equation l), arguably the most direct and straightforward outcome expected from a program such as the one under study here. One additional caveat is in order: the effect of the dummy variable treatment on days working is now significant as well as negative, though the coefficient indexing the actual number of days in treatment is positive and significant. The combination of these two effects suggests that any positive benefit of treatment vis-a-vis employment ensues only after relatively long stays in treatment. While some readers may find the negative treatment effect counter-intuitive and somewhat disturbing, we speculate that this owes to the fact that treatment clients were strongly discouraged from seeking employment until after they had established sobriety and some degree of stability within the program. Thus, in the vast majority of cases, employment only manifests itself as a behavioral goal after numerous months in treatment. Alternative Specifications An extensive literature focuses on the difficulty of maintaining substance-abusers in long-term treatment.
213
Evaluating an Alcohol and Drug Treatment Program Though an over-simplification, it would not be inaccurate to conclude that the sum of our collective knowledge regarding the efficacy of substance-abuse treatment suggests that almost any treatment can prove “effective” if and when clients are successfully maintained in treatment (see, e.g., Hubbard et al., 1989; DeLeon, 1990; Gerstein & Harwood, 1990). Based on this literature and the findings represented throughout Table 3 -wherein treatment exhibited a somewhat mixed set of effects on the analyzed outcomes, but rather consistently had a beneficial impact on attendance at meetings-we sought to test further the proposition that one potential positive impact of NOHSAP (and perhaps other similiar programs) was less direct and simply operated to motivate or induce substance abusers to establish a much-needed longer-term commitment to their ongoing recovery. With this in mind, we undertook one further set of analyses whereby attendance at AA/NA/CA meetings was treated as an additional independent rather than dependent variable. As such, we simply added this variable to the models previously specified in Table 3, Panel C, equations 1-5. The results from these specifications are presented in Table 4. Examination of these equations indicates that treatment itself no longer exhibits a positive and significant effect except with respect to housing (equation 2). In fact (and bearing in mind that this is net of all other
variables), the sign of the other treatment coefficients are all in the “wrong” (i.e., non-beneficial) direction. Coefficients indexing the impact of days in treatment are correctly signed (i.e., in the beneficial direction), but only achieve statistical significance with respect to working, good days, and family/social integration. Education continues to exert the same pattern and magnitude of effects as before. Critically, attendance at meetings has a fairly substantial effect on sobriety, housing, and good days. (Moreover, while attendance is not significant vi.+a-vis working and family/social integration, the signs are in the beneficial direction.) With respect to sobriety (equation l), attendance adds more than five (out of 30) days. Similarly, with regard to housing (equation 2) and the good days summary measure (equation 4), attendance adds 1.25 and 2.36 days respectively. These results largely are consistent with the notion that the primary benefit of the NOHSAP program derived from its positive impact on attendance.
DISCUSSION The goal of NOHSAP was to demonstrate that it is possible to take homeless substance abusers and turn them into responsible and productive adult citizens. There is some, but only some, comfort in recognizing that this goal was unreasonably ambitious to begin with.
TABLE 4 AAKXNA EFFECTS ON VARIOUS OUTCOME MEASURES, OLS ESTlMATESt
(1) Substance free Constant
Baseline Treatment Days in treatment Education Attends AAICAINA Adj. R* N
I 3.758
(2.13) 0.681$ (1.13) - 3.909 (3.71) 0.067’*” (0.039) - 0.667 (1.a4) 0.008 (0.009) 0.599” (0.169) 5.359’ (0.812) 0.109 600
(2)
(3)
Days in last 30 Housed Working 21.040 (1.aa) -2.062*** (0.966) - 2.220 (3.14) 0.143’ (0.031) 3.963”’ (1.56) 0.003 (0.00s) 0.086 (0.159) 1.246”** (0.693) 0.060 590
2.602 (2.16) 2.044”*** (1.11) -8.670*** (3.82) 0.264’ (0.050) -4.683”’ (1.60) 0.022”’ (0.009) 0.478”’ (0.190) 0.560 (0.774) 0.065 596
(4)
Good days 12.654 (1.38) 0.41 a (0.678) - 4.903”’ (2.35) 0.134’ (0.040) - 0.729 (1.11) 0.011 l ** (0.005) 0.383* (0.119) 2.355’ (0.486) 0.112
(5) ASI family/social integration 0.120 (0.050) - 0.488**** (0.026) 0.082 (0.094) o.laa* (0.033) 0.075**** (0.042) -0342E-03***’ (0.206E -03) - 0.003 (0.004) -0.003 (0.019) 0.060 545
TPositive treatment coefficients indicate improvement, except with ASI composite score where negative coefficient indicates improved functioning. *Coefficient and standard error. Significance: ‘p < ,001; “p G .Ol;“‘p Q .05; l ***p < .lO (two-tailed).
214
JOEL A. DEVINE
The process evaluation showed that many problems were encountered in the implementation of the project (including the failure to implement the randomization scheme), and that this had some serious implications for the outcome analyses, necessitating the use of econometric bias-correcting modeling techniques. Once corrected for selection and attrition bias, the subsequent outcome analysis indicates that NOHSAP apparently achieved limited success in producing sober, stably housed, socially integrated, (and perhaps to a lesser extent) employed former addicts. In general, longer-term clients showed modest (though not always significant) improvements in several areas of functioning, but few clients remained in treatment for the duration of the planned program. Why so many Extended Care clients left prematurely remains an open question. It is certainly possible that program-specific factors drove clients away, but the results of exit questionnaires showed that most clients were highly satisfied with most aspects of the NOHSAP program (see Wright et al., 1993b). It is also possible that the clinical staff unloaded clients at the first sign of trouble, retaining only those deemed more-certain bets to succeed. But our analysis of retention in treatment suggested rather persuasively that the clinical staff did not attempt to deal with the most troubled or problematic clients by finding reasons to terminate treatment. As noted, the available research literature points unambiguously to long-term retention as a significant problem. In fact, the low retention of clients in Extended Care was actually a bit better than average for interventions of this sort (Gerstein & Harwood, 1990). Thus, we are left, largely by default, with the hypothesis that there may be something intrinsic to the circumstances, personalities, or conditions of homeless addicts that precludes most of them from being retained in a treatment program for much more than a few months. Unfortunately, if the “minimum retention” needed to establish beneficial long-term outcomes is indeed on the order of several months, and if the homeless substance abusers studied in this research are typical, then the conclusion is that most homeless addicts cannot be retained in treatment long enough for treatment to do them much good directly. However, our analyses also indicate that treatment per se and time in treatment serve indirectly to facilitate recovery by connecting clients to attendance at AA-type meetings and presumably fostering the motivation and life skills to continue participation in supportive recovery groups of this sort. Readers should bear in mind that we are not claiming that there is anything intrinsic to AA-type treatment that has greater efficacy than any other substance abuse recovery philosophy or program per se, only that residential treatment of the NOHSAP sort may well buy much-needed time and begin to build the necessary
et al.
behavioral, cognitive, affective, and associational frame-work and connections critical to the long-term treatment and recovery that takes place in a wide variety of non-residential treatment programs. If this finding is widely generalizable - a question that awaits much more extensive empirical research - it may allow more efficient if not more efficacious use of limited treatment resources. Finally, it may resolve part of the ongoing dilemma of retention in treatment by serving to alter perhaps somewhat unrealistic and overly-ambitious expectations regarding residential substance abuse treatment. CONCLUSION For all the problems encountered in NOHSAP, it seems that the principal lesson comes through with some clarity: homeless addicts who can be retained in a NOHSAP-style treatment program for more than a few months can usually profit from the experience, but most homeless addicts cannot be. Nonetheless, residential treatment of the sort provided in NOHSAP does appear to indirectly facilitate many of the desired outcomes by motivating, supporting, and fostering connections with ongoing out-patient treatment modes that themselves exert positive and direct effects on desired outcomes. This lends credence to the notion that recovery from substance abuse should not be thought of as a solitary event or “cure,” but as a long-term therapeutic process often facilitated by treatment. REFERENCES Akers, R. andpolicy.
(1992).Drugs, alcohol and society: Social structure, process, Belmont,
CA: Wadsworth.
Barnow, B., Cain, G., & Goldberger, A. (1980). Issues in the analysis of selectivity bias. In W. Stromsdorfer & G. Farkas (Eds.), Evaluation studies review annuals (pp. 43-59). Beverly Hills, CA: Sage. Cook, T., & Campbell, D. (1979). Quasi-experimentation: Design and analysis issuesforfield settings. Chicago, IL: Rand McNally. DeLeon, G. (1990). Treatment strategies. In J.C. Inciardi (Ed.), Handbook of drug control in the United States (pp. 115-l 38). New York: Greenwood. Devine, J. A., & Wright, J. D. (1993). The greatest of evils: Urban poverty and the American underclass. Hawthorne, NY: Aldine de Gruyter. Devine, J. A., Wright, J. D., & Brody, C. (1995). An evaluation of an alcohol and drug treatment program for homeless substance abusers. Evaluation Review, 19. 62&645. Devine, J. A., Wright, J. D., & Joyner. L. (1994). Issues in implementing randomized experiments in field settings: The New Orleans homeNew Directions for Program less substance abuse program. Evaluation, 63, 27-40.
Evaluating
an Alcohol
and Drug Treatment
Fureman, B., Parikh, G., Bragg, A., & McLellan, A. (1990). Addiction severity index: A guide to training and supervising ASI interviews based on the past ten years. Philadelphia, PA: University of Pennsylvania/ V. A. Center for Studies of Addiction.
Program
215
McLellan, A. (1983). Patient characteristics associated with outcome. In J. Cooper, F. Altman, B. Brown&D. Czechowicz (Eds.), Research on the treatment of narcotic addiction: State of the art. Washington, DC: USGPO.
Gerstein, D., & Harwood, H. (eds.). (1990). Treating drug problems (Vol. 1). Washington, DC: National Academy Press.
McLellan, A., Luborsky, L., Cacciola, J., Griffith, J., Evans, F., Barr, H., &O’Brien, C. (1985). New data from the addiction severity index: Reliability and validity in three centers. Journal of Nervous and
Greene, W. (1981). Sample selection bias as a specification error: A comment. Econometrica, 49, 795-798.
Mental Disease, 173.412423.
Greene, W. (1992). LIMDEP, version 6.0: User S manual and reference guide. Bellport, NY: Econometric Software, Inc.
McLellan, A., Luborsky, L., Woody, G., & O’Brien, C. (1980). An improved diagnostic instrument for substance abuse patients: the addiction severity index. Journal of Nervous and Mental Disease, 168, 2&33.
Heckman, J. (1979). Sample selection bias as a specification error. Econometrica,
47, 153-161.
Hubbard, R. L., Marsden, M., Rachal, J., Harwood, H. J., Cavanaugh, E., & Ginsburg, M. (1989). Drug abuse treatment: A national study of effectiveness. Chapel Hill, NC: University of North Carolina Press. Joe, G., Simpson, D., & Hubbard, R. L. (1991). Treatment predictors of tenure in methadone maintenance. Journal of Substance Abuse, 3,
Rudegair Associates. (1990). Searching for shelter: An analysis of New Orleans’ homeless programs. New Orleans, LA: Rudegair Assoc.
metropolitan
Wright, J. D. (1989). Address unknown. Hawthorne, NY: Aldine de Gruyter. Wright, J. D., Allen, T., 8c Devine, J. A. (1995). Tracking nontraditional populations in longitudinal studies. Evaluation and Program Planning, 18, 267-277.
73-84.
Joe, G., Singh, B., Garland, J., Lehman, W., Sells, S., & Seder, P. (1983). Retention in outpatient drug-free treatment clinics. Journal
Wright, J. D., Devine, J. A., & Eddington, N. (1993). The New Orleans homeless substance abusers project. Alcoholism Treatment Quarterly, 10. 51-64.
of Addictive Behaviors, 8, 219-234.
Wright, J. D., Devine, J. A., & Joyner, L. (1993). The least of mine: McGahan, P., Griffith, J., Parente, R., & McLellan, A. (1991). Composite scores from the ASI. Washington, DC: NIDA Project No. DA02254 and the Veterans Administration.
The New Orleans homeless substance
abusers project:
Final report
(Technical Report, September). Bethesda, MD: National Institute of Alcohol Abuse and Alcoholism.