Outcomes from CSAP's community partnership program: Findings from the national cross-site evaluation

Evaluation and Program Planning, Vol. 20, No. 3, pp. 345-355, 1991 Published by Elsevier Science Ltd

Pergamon

Printed in Great Britain 0149-7189/97 %17.00+0.00

PII: SO149-7189(97)00014-l

OUTCOMES FROM CSAP’S COMMUNITY PARTNERSHIP PROGRAM: FINDINGS FROM THE NATIONAL CROSS-SITE EVALUATION

ROBERT

K. YIN

COSMOS

SHAKEH Substance

Abuse

and

Mental

Health

Services

Corporation

J. KAFTARIAN

Administration,

Center

PING COSMOS

MARY Substance

Abuse and Mental

Health

for Substance

Abuse

Prevention

(SAMHSA-CSAP)

YU Corporation

A. JANSEN

Services Administration,

Center for Substance

Abuse Prevention

(SAMHSA-CSAP)

ABSTRACT The cross-site evaluation of SAMHSA-CSAPS community partnership program collected a broad variety of data from all partnerships and also from a stratljied, random sample of 24 partnerships and a set of matched comparison communities. The present article reports the results of the main outcome analysis, comparing substance abuse prevalence rates in the 24 partnerships and their matched comparisons, at two points in time. The prevalence rates were derivedfrom surveys of nearly 85,000 individuals in these 48 communities, divided into three age groups: adults, 10th graders, and 8th graders. The survey samples were selected to represent the entire community. All analyses were done by estimating regression models that accounted for individual confounders - i.e., the age, gender, race, education, employment status, and income of the adults, and the age, gender, and race of the youths. Further, the analyses were conducted two ways, a pooled analysis that assumes the same intervention was carried out by each partnership, and an individual analysis that assumes different interventions. (The two assumptions are competing interpretations of the community partnership program, and rather than favoring either one, the cross-site evaluation covered both.) The results showed weak but some statistically significant differences in the predicted direction: Partnerships’ prevalence rates were lower than the comparison communities’ rates. Ongoing analyses are further investigating these results and also examining the reasons why partnerships might have produced the results - work that needs to be continued before arriving at any dejnitive interpretation of the partnerships’ effects. Published by Elsevier Science Ltd

Under federal law, a work of the United States government is to be placed in the public domain and, neither the government, the author, nor anyone else may secure copyright in such a work or otherwise restrict its dissemination. Requests for reprints should be addressed to Robert K. Yin, COSMOS Corporation, 7475 Wisconsin Avenue, Suite 900, Bethesda, MD 20814, U.S.A.

345

346

ROBERT

ANALYTIC

INTRODUCTION The cross-site evaluation of the Community Partnership Program collected intensive outcome data from 24 partnership communities - a stratified, random sample of the entire universe of 251 partnership grants made by the SAMHSA-CSAP starting in 1990. Further, for each partnership community, a matched comparison community was identified, based on age, gender, ethnicity, size, and density of population; income levels; and geographic proximity of jurisdiction. Outcome data also were collected from these 24 comparison communities. In all, the surveys covered 48 communities and crosssectional samples of over 40,000 respondents each at two points in time (the first point was roughly mid-way through the partnership grants; the other was near or just after their ending). This article describes the initial outcome findings from the evaluation. Although analysis was still ongoing at the time the article was completed, such analysis was focused on the various independent variables and other explanations for the outcomes, and not the outcomes themselves, which are reported here in their definitive form. The main outcomes involve substance abuse prevalence, or the reported use of alcohol and illicit drugs, either during the past year or during the past month, on the part of three different age groups: adults queried in a random sample of households (telephone interviews); and representative samples of 8th and 10th graders attending schools in the community (classroom questionnaires). Table 1 shows the sample sizes for these three groups at both the earlier and later points in time (at both times, the samples were cross-sectional). To assess prevalence, questionnaire items were selected to replicate the two national substance abuse surveys sponsored by the U.S. Department of Health and Human Services - the National Household Survey (for adults) and the Monitoring the Future Survey (for youths) see (SAMHSA, 1995; and Johnston, O’Malley, Bachman, & National Institute on Drug Abuse, 1995). Other facets of the design and data collection procedures used in the cross-site evaluation are described more fully elsewhere (CSAP, June 1997).

SURVEYED

TABLE 1 POPULATIONS: COMMUNITY PROGRAM* tr

Adult (phone) Tenth graders (field) Eighth graders (field) *A total of 48 communities respectively.

14807 12842 14151 participated

K. YIN et al.

PARTNERSHIP

t2 12092 13042 16539 in the surveys at

Total 26 899 25 884 30 690

t, and f,

STRATEGY

The analysis followed a unique strategy, pursuing two different approaches: (1) an aggregate, pooled analysis of all partnership and comparison communities, and (2) an analysis of the individual partnerships compared to their matched comparison communities. Both approaches address the major, summative question of the entire evaluation - whetherpartnership communities show decreasedprevalence ratesfor substance abuse, over time, relative to their matched comparison communities.

However, the two approaches reflect two different ways of interpreting the design of the Community Partnership Program. Their juxtaposition in the same analysis represents a significant landmark in the methodology for analyzing community interventions, and not just substance abuse prevention. Pooled Analysis

The rationale for the pooled analysis rests on the claim that the Community Partnership Program prescribed the specific operational nature of the intervention to be followed in each partnership community. The program did indeed specify that partnerships had to consist of seven or more partners, spend only 10% of their resources for direct prevention services, and carry out a locally-supported evaluation. The pooled analysis therefore assumes that the same intervention was indeed carried out in each partnership community, and that the program implicitly followed the design of a community trial. As an example, a notable community trial in the prevention field was the COMMIT project, aimed at reducing smoking (COMMIT Research Group, 1991, 1995; and Gail, Byar, Pechacek, & Corle, 1992). From a methodological standpoint, the pooled analysis produces an “average” partnership and contrasts it with an “average” comparison community (the analysis ignores the linked identities between the specific pairs of partnership and comparison communities). Further, the analysis assumes a nested design: individual adults or youths were nested within 48 communities (“intact social groups”). Though the unit of data collection was the individual respondent in a survey, the unit of assignment and therefore unit of analysis was the community. As a result, the analysis of partnership outcomes needs to be conducted at the level of assignment - the communities - with adjustment for individual-level and community-level covariates as needed (Cornfield, 1979; and Murray, Hannan, & Baker, 1996). Among a family of models available for analyzing hierarchical data including hierarchical linear models (Bryk & Raudenbusch, 1987; Raudenbusch, 1988; and Perkins & Taylor, 1996) ML-n (Goldstein, 1987) and General Linear Mixed Model (Latour, Latour, & Wolfinger, 1994) - SAS Proc Mixed (Murray & Wolfinger, 1994; and SAS Institute, Inc., 1996) was chosen for the present

Community Partnership Program: Findings evaluation for its appropriateness, simplicity, and efficiency. SAS Proc Mixed employs a general linear “mixedmodel” regression. The mixed-model permits the analysis of both fixed and random effects, using information gathered from both individuals and communities, to account for the influence of the nested design. The nesting produces artificially low variance across individuals (assignment bias and intra-class correlation, or ICC, in the data), resulting in turn in potentially inflated statistical findings (Zucker, 1990; Murray & Hannan, 1990; Murray, Hannan, Jacobs, McGovern, Schmid, & Baker et al., 1994; and Murray & Wolfinger, 1994). The mixed-model: (a) produces a specific estimate of the ICC, and (b) arrives at an adjusted variance, offsetting the phenomenon defined by Kish (1965) as the “design effect” over 30 years ago. Thus, this first analytic strategy emerged in recognition of the potential biases resulting from the conventional fixed-effect models, which are only appropriate for individual-level data that are not nested. At the same time, mixed-model regression, due to what Cornfield termed as the dual penalties of variance inflation (across classes or communities) and reduced denominator degrees of freedom, severely limits the power to detect an intervention effect in an otherwise well-designed and wellexecuted community trial (Cornfield, 1979). The definition of the preferred mixed-model was as follows. First, accounting for individual confounders was important for removing the undesirable effects of other variables - in this case age, gender, race, education, employment status, and income - at the level of the individual respondent. (In the youth data, only three individual confounders were used: age, gender, and race.) Degrees of freedom were lost for each confounder used, but because the modifications were at the individual level, the overall effect of the loss on the large sample size was negligible. All of the confounders were therefore entered into the regressions together, not singly. Second, the choice of scaled weights followed the realization from previous research that using the original base weights so dramatically increases the variance (and hence standard error and confidence interval) as to render uninformative any inquiry into substantive relationships in a community trial - because the huge variance severely diminishes the power or ability to detect reasonable treatment effects (Kom & Graubard, 1991). The scaled weights take care of the needed adjustments, but without increasing variance, by converting the weights so that they have an average of one: Scaled Weight = Base Weight

1 Mean Base Weight

This method maintained the ratio of the sampling weights, but retained the same variance.

341

Third, taking community confounders into account was important for further removing the undesirable effects of other variance attributable to the community level. However, there are serious losses in degrees of freedom for every confounder used, because the unit of analysis is the community level, where there are only 48 degrees of freedom to start with. For instance, a race “variable” could actually consist of five types of racial and ethnic groups - which would result in the loss of five degrees of freedom from the initial set of 48. The present analysis, therefore, did not include any community confounding variables. Instead, these variables are part of ongoing investigations. Fourth, whereas the two waves of survey data reflected “two points in time”, the pooled analysis permitted the use of the more refined variable, “time-afteraward” - the exact month of the earlier and later data collection, relative to the onset of a partnership’s funding by SAMHSA-CSAP. Because of the extended time to conduct the surveys and because of the different initial funding periods by SAMHSA-CSAP, these months differed across partnerships. Finally, unlike the conventional fixed-effects model, the SAS Proc Mixed procedure does not differentiate the order in which the variables have been entered into the regression equation. Analysis of Individual Partnerships Comparisons

and Their Matched

The individual partnership analysis (24 pairs of partnership and comparison communities) was based on the premise that the Community Partnership Program supported different interventions in each partnership community. Given this assumption, each of the 24 interventions needed to be examined separately. Support for this interpretation of the program is based on the following claim: Although there were some uniform requirements imposed by the Community Partnership Program, each community was strongly encouraged to design and implement its own unique intervention, reflecting community needs and the desire to empower residents to make lasting changes. Because each of the partnerships had a matched comparison community, 24 separate analyses compared a partnership community’s performance with that of its matched comparison, over two points in time. Ordinary linear logistic-model regressions were used to analyze the data (Hanushek & Jackson, 1979), because the initial definition of the primary dependent variables of interest was based on a bivariate condition: the prevalence rate (“use” or “no use”) for substance abuse. Differences between partnership and comparison communities were expected to be observed as a function of two different points in time, controlling for influences of individual-level confounders, such as the age, gender, race, education, employment status, and income of the

348

ROBERT

K. YIN et al.

individual respondents. The theoretical specification led, for this initial analysis, to regressing in sequential order the binomial variables against: first, individuallevel confounders; second, the interaction term between community condition (partnership vs comparison community) and time (earlier or later survey); third, the community condition variable alone; and fourth, the time variable alone. The analysis was conducted using the SAS CATMOD procedure (SAS Institute, Inc., 1989-1995), which is used for the analysis of categorical dependent variables such as the binomial construction “use” or “no use”. The equations were fit to each of the 24 pairs of partnership and comparison communities for the adult, tenth, and eighth grade data separately.

RESULTS Pooled Analysis The mixed-model regressions were carried out 12 times - once for each of the three age groups, times each of the four dependent variables. Table 2 displays the results for all 12 models. All models compared the difference in regression slopes between partnership and comparison communities when individual confounders,

SUMMARY

scaled weights, and community condition were taken into account. (The degrees of freedom are lower for the tenth and eighth grade data, compared to the adult data, because data collection could not be completed for three pairs of communities for both of the grades.) The key outcome is found in the column entitled “difference in slopes” (partnership communities’ slope minus the comparison communities’ slope). The slope represents the difference in outcome (prevalence) per unit difference in the time variable. For the current evaluation, the predicted direction of this difference in slopes was negative that the partnership communities’ slope should have declined in contrast to the comparison communities’ slope. The results from these 12 models show that, for all 3 age groups and 4 different dependent variables, only one had a statistically significant difference at thep < .05 level: adult alcohol use in the past month. However, of the remaining 11 differences, all but the smallest one were in the predicted direction. Overall, the pattern and directionality of the aggregate results suggest that the Community Partnership Program was associated with change in the expected direction over a short period of time, although the effects are small and difficult to detect in the face of the high variance across partnership and comparison communities.

TABLE 2 OF MIXED-MODEL REGRESSION

RESULTS Covariance parameter estimate ratio 0

Difference in slopes

df

F

Illicit drug use in the last year Illicit drug use in the last month Alcohol use in the last year Alcohol use in the last month

- 0.00078 -0.00061 - 0.00040 -0.00159

46 46 46 46

1.72

0.00326522

1.81

0.00233931

0.31 4.33

10th grade

Illicit drug use in the last year Illicit drug use in the last month Alcohol use in the last year Alcohol use in the last month

-0.00104 -0.00114 - 0.00030 -0.00015

41 41 41 41

6th grade

Illicit drug use in the last year Illicit drug use in the last month Alcohol use in the last year Alcohol use in the last month

- 0.00047 - 0.00067 0.00006 - 0.00083

41 41 41 41

Age groups

Outcome variables

Adult

n

ICC

P

0.00191534 0.00152872

0.0033 0.0023 0.0019 0.0015

0.1961 0.1846 0.5817 0.0431

26 26 26 26

0.92 1.47 0.07 0.01

0.00362952 0.00313351 0.00540958 0.00721719

0.0036 0.0031 0.0054 0.0072

0.3433 0.2324 0.7982 0.9095

25159 25161 25135 25129

0.10 0.28 0.00 0.34

0.00351546 0.00261663 0.00403031 0.00297982

0.0035 0.0026 0.0040 0.0030

0.7550 0.5996 0.9715 0.5606

29 29 29 29

676 676 666 603

865 869 a31 a25

Key: df= degrees of freedom. F=Type Ill F. ICC = lntraclass Correlation Calculated as e/(1+0). p = probability of significance. n = size of sample used in this analysis. N/A = not applicable. Scaled weight, calculated as base weight (l/(MeanBaseWeight)), was used. Time, defined as months between grant award and survey administration date, was used. Community cluster was used. Individual confounders were used to control for the effects of age, gender, race, education, employment status, and income (only age, gender, and race were used in the youth data). Difference in slope was estimated as partnership communities’ slope minus comparison communities’ slope.

349

Community Partnership Program: Findings

tively, from t, to tz, relative to the comparison community’s prevalence rate. The results indicate that, among the 24 pairs (and the 12 outcomes examined for each pair), there were 22 statistically significant relationships at the p less than .05 level (p < .05), involving 8 of the 24 partnerships. An important concern was that this number of statistically significant relationships might have been due merely to the large number of regressions that had been run (the product of 24 pairs of communities multiplied by 3 age groups, multiplied by 4 outcomes, equals 288 regressions, minus 24 regressions due to missing data, and therefore a total of 264 regressions). The 22 significant relationships meant that significant results at thep less than .05 level (p < .05) had been found 8% of the time. Whether the margin was safely beyond what might be expected by chance - and whether the clustering of the significant relationships among a small group of partnerships rather than being spread randomly among a larger number of them was a further sign of “chance” results - was difficult to judge. To explore the issue further, Table 5 summarizes the eight partnerships that demonstrated the statistically

Analysis of Individual Partnerships

An initial observation of the raw (unadjusted) prevalence data was that the overall matching of paired partnership and comparison communities was fairly good for the adult sample and the tenth grade illicit drug use sample. However, small to slightly moderate levels of initial non-equivalence (2-l 5 percentage points in unadjusted prevalence rates, for 2 pairs) was observed for the tenth grade alcohol use sample and the eighth grade sample. Nevertheless, because the raw data were adjusted by accounting for individual confounders, the potential sources of non-equivalence were assumed to be minimized by the regression analyses. To test the difference within each partnership-comparison pair, an ordinary logistic-model regression was therefore carried out 12 times: once for each of the 3 age groups (adult, tenth, and eighth graders) times the 4 dependent variables. Tables 3 and 4 present the results, showing the value, sample size, and statistical significance for the coefficients of the critical interaction term, “community by year”. A negative sign for this term indicates that a partnership’s prevalence rate had changed in the expected direction - i.e., more nega-

ILLICIT DRUG USE: COMPARISON Site pairs

102l202 103l203 1041204 1051205 1061206 1071207 1091209 1111211 112/212 1141214 llW215 1201220 121/221 122/222 123/223 1241224 1251225 134J234 1351235 1361236 1371237 1381238

Adult 0.1689 -1.3645 0.2554 - 0.2826 0.1614 -0.0116 - 0.2624 -0.3129 -0.4164 0.7221” -0.4415 0.3590 - 0.4375 0.8207 0.1026 - 0.5495 -0.3394 -0.1750 0.7767 - 0.4245” - 0.3348 -0.1586 0.4974 0.5965

TABLE 3 OF PAIRS OF PARTNERSHIP

Illicit drug use in the past year N 10th grade N 8th grade 1147 1105 1116 1115 1126 1106 1151 1105 1106 1117 ii08 1144 1135 1125 1119 1132 1103 1119 1116 1091 i 084 1093 1086 1027

-0.2631 - 0.2273 -0.0140 -o.a982*** -0.0024 0.0806 0.1946 -0.6891” 0.0211 N/A -0.1997 -0.5097 - 0.3050** - 0.2204 -0.1894 - 0.2433 - 0.3760 N/A -0.0011 N/A 0.3572 0.1799 -0.5907 - 0.4022

i 048 1539 i 287 i i 07 372 1442 1100 459 1641 N/A 927 1186 1163 1637 1183 1497 1186 N/A 994 N/A 1361 1378 1609 a04

0.3153 0.5769 - 0.2942 N/A 0.3284 -0.0565 0.0914 0.0465 -0.5916 0.0255 - 0.0036 - 0.3242 0.0895 0.2819 -0.5001 -0.1964 N/A 0.0091 N/A 0.3753 0.0514 -0.1880 0.1607 -0.1326

N 639 la07 1678 N/A 968 1806 1658 966 1376 1331 ii83 1333 1571 1652 la20 1257 N/A 1575 N/A 1769 1554 1327 1362 945

AND COMPARISON

COMMUNITIES,

1995-1996

Illicit drug use in the past month N 10th grade N 8th grade

Adult 0.1638 - 2.8096’* 0.4947 - 0.2053 0.1178 - 0.2096 -0.5981 - 0.6489 - 0.4291 0.6519 -0.3601 - 0.0765 -0.2814 0.8176 - 0.2164 - 0.2397 0.4280 0.3258 1.2272 -0.3753 - 0.2002 - 0.4688 0.5451 1.0914

1147 1105 1116 1115 1126 1106 1151 1105 1106 1117 1108 1144 1135 1125 1119 1132 1103 1119 1116 1091 1084 1093 1086 1027

0.0649 -0.1541 -0.3250 -0.5904** -0.0478 0.1138 0.3352 -0.4712 -0.3446 N/A -0.3270 -0.3145 -0.3966*’ -0.2611 -0.3132 - 0.3477*’ -0.2872 N/A -0.1851 N/A 0.5360 0.3022 -0.8185 - 0.5925

1048 1530 1287 1107 372 1443 1100 459 1641 N/A 927 1186 1163 1637 1183 1497 1186 N/A 994 N/A 1361 1378 1609 804

0.5064 0.5852 -0.3175 N/A 0.3655 - 0.0769 - 0.2846 -0.0163 -0.1884 0.2517 -0.311 a 0.0523 -0.0860 0.1654 -0.6132 - 0.2520 N/A 0.3020 N/A 0.3847 -0.1464 -0.2112 0.3064 -0.1551

N 639 la07 1678 N/A 968 i 807 1658 966 1376 1331 1183 1334 1571 1652 1820 1257 N/A 1575 N/A 1769 1555 1328 1362 945

“p 5 .05 level; ***p 5 .Ol level; N/A = No data were collected. The variables in the logistic-model regressions were entered in the following order:

Adult Youth

Age 1 1

Gender 2 2

Race 3 3

Education 4

Emploment 5

Income 6

Population density

Site

Treat year 7 4

Treat a 5

Year 9 6

350

ROBERT

ALCOHOL Site pairs

Adult

101/201 102l202 1031203 1041204 1051205 106/206 1071207 1081208 1091209 111/211 112l212 1141214 115/215 1201220 1211221 122/222 1231223 1241224 1251225 1341234 1351235 1361236 1371237 1381238

USE:

N

-0.4766 -0.2373 0.3576 0.1188 0.1987 - 0.7247 -0.0752 -0.3513 -0.2035 0.2532 -0.3501 0.5192 -0.4697 0.4784 -0.1463 -0.1881 0.3341 0.1609 -0.3167 0.1731 0.2485 -0.1950 0.2661 0.2769

COMPARISON

OF PAIRS

K. YIN et al.

TABLE 4 OF PARTNERSHIP

Alcohol use in the past year 10th grade N 6th grade

1147 1105 1116 1117 1126 1106 1150 1104 1105 1117 1108 1144 1135 1124 1119 1130 1101 1118 1116 1092 1084 1093 1085 1024

0.3093 -0.7097** -0.3176 -0.7047”’ 0.8102 0.0439 0.1368 -0.5896*** -0.1855 N/A 0.2385 -0.5416 -0.6008*” -0.4794 0.2002 -0.5650*** -0.4081 N/A -0.0724 N/A 0.1868 0.4849 -0.3164 -0.4521

1046 1530 1286 1106 372 1442 1100 458 1639 N/A 927 1185 1160 1637 1184 1495 1183 N/A 992 N/A 1359 1373 1608 804

0.0202 0.1112 0.0371 N/A 1.1768 0.0569 0.2898 0.0520 - 0.3261 - 0.2920 -0.1985 -0.2481 0.1527 0.2915 - 0.2671 -0.3687** N/A 0.1782 N/A 0.1452 0.5955 -0.0190 0.3512 -0.1045

N 633 1803 1678 N/A 967 1807 1657 964 1376 1324 1181 1331 1570 1652 1819 1256 N/A 1574 N/A 1767 1555 1324 1360 945

l*p 5 .05 level; ‘*‘p < .Ol level; N/A = No data were collected. The variables in the logistic-model regressions were entered in the following Gender 2 2

Age 1 1

Adult Youth

INDIVIDUAL

Education 4

Race 3 3

PARTNERSHIPS:

AND COMPARISON

Employment 5

- 0.3096 - 0.4049 0.1470 0.0093 -0.3109 -0.6521 - 0.7998” - 0.3598 - 0.2824 0.1909 - 0.5989”’ 0.5750 - 0.4560*= 0.0663 0.1345 0.1915 0.5890 -0.1900 -0.2522 -0.0809 0.1480 -0.0409 - 0.0581 0.0506

1145 1101 1114 1114 1124 1102 1147 1103 1104 1117 1107 1137 1132 1118 1118 1129 1100 1116 1116 1089 1082 1088 1079 1021

0.4317 - 0.6350” -0.4475 -0.3902 0.2809 0.1794 -0.0984 -0.7634”” -0.0956 N/A 0.1170 -0.6064 -0.3518** -0.0220 0.1946 -0.8425’“’ -0.0943 N/A -0.1481 N/A 0.2340 0.3718 -0.6162 -0.1870

Partnership/comparison Eight partnerships 115/215

A with significant

122/222 102l202 1041204 1081208 1341234

8

A

reductions: **

Income 6

Population density

Site

OUTCOMES

** .

na

IN RELATION

8

A

Past year 10

**

***

*.

ttt

na

*** **

l*p

< .05;

with significant

8

na

na

na

reversal: _** na

na

na

***p -C.Ol;A = adult data; 10 = 10th grade data; 8 = 8th grade data: na = no data were collected.

Treat 8 5

Year 9 6

use A

l *

Past month 10

8

**

l **

*t

**

na

***

l

112/212 1071207 One partnership 111/211

635 1801 1677 N/A 966 1806 1635 965 1375 1324 1181 1331 1571 1654 1818 1256 N/A 1573 N/A 1769 1554 1324 1359 945

TO MATCHED

l *

**

l

tt

0.2550 0.1668 0.2234 N/A 0.8626 0.2312 -0.0098 0.0554 -0.1290 -0.1618 -0.0025 0.0653 -0.1351 0.1815 -0.4398 -0.2576 N/A 0.1477 N/A 0.2994 0.3899 -0.4170 0.2523 - 0.4374

Treat year 7 4

Alcohol Past month 10

**

l

1045 1529 1285 1106 372 1442 1099 457 1639 N/A 927 1187 1161 1635 1183 1493 1184 N/A 992 N/A 1357 1376 1607 804

N

order:

Illicit drug use Past year 10

1995-1996

Alcohol use in the past month N 10th grade N 8th grade

Adult

TABLE 5 OF STATISTICALLY SIGNIFICANT COMPARISON COMMUNITIES

SUMMARY

COMMUNITIES,

*** t*

na

na

na

Community Partnership Program: Findings significant reductions in the prevalence of substance abuse and the one partnership with the significant reversal (increase) in illicit drug use. The first five of the eight partnerships all showed statistically significant reductions for two or more outcome categories. A companion Table 6 examines the fuller pattern of results from these eight partnerships, comparing the pattern and consistency of results between “past year” and “past month”. The comparison checks for the possibility that one dependent variable might have been statistically significant but the other might have been reversed (though not statistically significant) and therefore contradictory. No such situations were found. In fact, the rankings in Table 6 show that even the rankings for “past year” and “past month” tended to be similar for all but one (No. 107) of the pertinent dependent variables. This entire pattern strengthens the possibility that the initial statistically significant results (8% of the regressions significant at the p less than .05 level (p < .05) might not have been merely due to the chance possibility of calculating such a large number of regressions.)

351 DISCUSSION

Findings From the Pooled Analysis Pending further investigation, the findings from the aggregate analysis have several tentative interpretations. First, because only 1 of the 12 models was statistically significant, the data may in fact reflect no particular direction or trends of any importance. Partnership communities were not markedly different from comparison communities, possibly because partnerships in the aggregate (the “average partnership”) did not produce prevention gains, relative to comparison communities in the aggregate (the “average comparison community”). Possibly, the one-year time difference between the two surveys was too short to surface a true multi-year trend (the desired survey interval would have resembled more closely the five-year length of the partnerships). Second, not reported in the tables is the fact that the unexplained variance (residuals) in these 12 models was generally high - 80% or more of the total variance (a couple were as high as 90 and 93 Oh).This high variance

TABLE 8 CONSISTENCY OF FINDINGS BETWEEN “PAST YEAR” AND “PAST MONTH” FOR EIGHT PARTNERSHIPS SIGNIFICANTLY DIFFERENT FROM THEIR COMPARISON COMMUNITIES

Partnership number and outcome variable

Size of coefficient in logistic model regression Past year Past month Coeff. Rankt Coeff. Rankt

No. 115: Adult alcohol use 10th grade illicit drug use 10th grade alcohol use

- .470 - .305** -.801*‘”

3 7 3

- ,458” - ,397” - .352**

4 5 8

No. 122: 10th grade illicit drug use 10th grade alcohol use 8th grade alcohol use

- .243 - .585*** - .389*

9 5

- .348** - .843- ,258

8 1 4

1

-2.810** - .835**

No. 102: Adult illicit drug use 10th grade alcohol use No. 104: 10th grade illicit drug use 10th grade alcohol use

- 1.385 - .710**

- ,898”’

- .590**

- ,705”’

2

- .390

No. 108: 10th grade illicit drug use 10th grade alcohol use

- .889** - .590*==

2 4

- .471 - .783’**

No. 134: Adult alcohol use

- ,425”

5

- .375

No. 112: Adult alcohol use

- .350

5

- .599***

No. 107: Adult alcohol use

- .075

12

- .800**

TSize of coefficient’s rank among all 24 partnerships. ““p < .05; l **p < .Ol

352

ROBERT K. YIN et al.

means that the specified models, covering outcome conditions and individual confounding factors only, was not very powerful. Many unspecified conditions, including an array of possible independent variables, could reduce the unexplained variance and offer more refined explanations of the results. This second possibility is therefore the topic of the continuing inquiries. Third, instead of pleading prematurely to substantive interpretations of the findings, other explanations related to the procedures used in the aggregate analysis - might need to be considered. Among these explanations are certain conditions that are beyond the control of the present state-of-the-art. For example, the mixed-model regression suffers from the two penalties previously cited (Cornfield, 1979): variance inflation due to cross-sectional differences in the data (in this case caused by the highly varied, national cross-section of communities represented by the partnerships), and reduced denominator degrees of freedom (in this case a sample size of 48). Both conditions are known to severely limit the power to detect an intervention effect in an otherwise well-designed and well-executed intervention (Cornfield, 1979), and no procedures have been developed to counteract these limitations (Murray et al., 1996). Along the same lines, a fourth possibility is that the data analysis might still require further procedural refinement. Several such refinements were considered and tested. First, the variables used in the models had alternative versions that could have produced different results. For instance, representing the prevalence rate in only binomial terms might have unnecessarily decreased the sensitivity of the measurement, and therefore the binomial was translated back into a scaled, ordinal prevalence rate (amount of drug taking reported, not just whether taking drugs or not). As another example, in case the “time-after-award” variable added unnecessary variance without additional explanatory power, the variable was translated back to two points in time (an earlier and a later time). However, neither of these substitutions altered the results.

Partitioning the Pooled Data Beyond all of these questions was a further lingering problem: The possibility that the pooled analysis was producing an “average” community that in fact represented no specific community at all - and that the mixed-model regressions effectively offset trends in two contrary directions, thereby producing an appearance of “no effect”. For instance, the use of the individual confounders stemmed from the common (methodological) problem that certain demographic differences (e.g., age, gender, and race) have usually been associated with differential prevalence rates. A fair comparison between the partnership and comparison communities

therefore required that the variance associated with these demographic factors first be controlled, and this procedure was the basis for the mixed-model results reported in Table 2. An alternative possibility is that, rather than “controlling” for such differences, they should be examined directly. In other words, an equally fair question is to compare one subgroup of individuals in the entire partnership sample with a matched subgroup in the comparison sample (Cook & Campbell, 1979). The procedure involves partitioning the original sample (of individuals) into subgroups and then making statistical comparisons between the subgroups. For instance, for race, the sample could be partitioned into “Black”, “White”, and “all other;” for gender, the sample could be partitioned into “males;” and “females;” and for age, the sample could be partitioned into “younger” and “older” age groups. Of all these possibilities, the most intriguing was the partitioning of the sample according to gender. This was because many substance abuse prevention programs are implicitly aimed at male populations, rather than males and females combined or females only (CSAP, 1991; and Dryfoos, 1990). For instance, “alternative youth usually involving youth camps or athletic programs”, activities, are implicitly designed for and involved many more males than females. Whereas the original mixedmodel regressions adjusted the gender differences between communities by controlling for the variance in prevalence rates attributable to gender, the alternative procedure is to compare the two types of communities (partnership and comparison) - first by the males in the sample and then by the females. A new set of mixed-model regressions was therefore estimated, dividing (or partitioning) the aggregate data into two groups: all males and all females. Instead of 12 dependent outcome models (3 age groups multiplied by 4 types of outcomes), there were now 24 models (the product of 3 age groups multiplied by 4 outcomes, multiplied by 2 sexes). Of course, the partitioning of the data also meant that the number of individuals in the final mixed-models was roughly halved, but this did not affect the degrees of freedom because they were defined by the number of communities-which did not change. Tables 7 and 8, covering first the mixed-models for males and then for females, show the results of the partitioned data. Among the models for males, statistically significant results at thep < .Ol level were found for 2 of the 12 dependent variables. The two models showed statistically significant reductions in eighth grade illicit drug use in the past month and alcohol use in the past month. Further, for one other modelladult illicit drug use in the past month - the differences were at the p I .05 level. In all but one of the remaining cases, the results were in the predicted direction (prevalence rates for males in the partnerships showed a declin-

Community

MALE SAMPLE:

SUMMARY

Partnership Program: Findings

353

TABLE 7 OF MIXED MODEL REGRESSION

RESULTS

Outcome variables

Difference in slopes

df

F

Covariance parameter estimate ratio 0

ICC

P

n

Adult

Illicit drug use in the last year Illicit drug use in the last month Alcohol use in the last year Alcohol use in the last month

-0.00159 -0.00148 - 0.00047 -0.00221

46 46 46 46

2.66 4.03 0.21 3.66

0.00721693 0.00486899 0.00489166 0.00387528

0.0072 0.0048 0.0049 0.0039

0.1063 0.0506 0.6523 0.0621

11836 11836 11826 11792

10th grade

Illicit drug use in the last year Illicit drug use in the last month Alcohol use in the last year Alcohol use in the last month

-0.00187 -0.00206 -0.00091 -0.00082

41 41 41 41

2.32 3.40 0.65 0.51

0.00000888 0.00000832 0.00000245 0.00000115

0.0000 0.0000 0.0000 0.0000

0.1355 0.0723 0.4259 0.4810

11920 11921 11906 11900

8th grade

Illicit drug use in the last year Illicit drug use in the last month Alcohol use in the last year Alcohol use in the last month

-0.00222 -0.00349 0.00023 -0.00370

41 41 41 41

2.45 7.48 0.02 7.40

0.00000000 0.00000000 0.00000000 0.00000000

0.0000 0.0000 0.0000 0.0000

0.1249 0.0092 0.8799 0.0095

14600 14601 14576 14575

Age groups

Key: df=degrees of freedom. f=Type Ill F. ICC = lntraclass Correlation Calculated as 0/( 1+ 0). p= probability of significance. n=size of sample used in this analysis N/A = not applicable. Scaled weight, calculated as base weight (l/(MeanBaseWeight)), was used. Time, defined as months between grant award and survey administration date, was used. Community cluster was used. Individual confounders were used to control for the effects of age, race, education, employment status, and income (only age, gender, and race were used in the youth data). Difference in slope was estimated as partnership communities’ slope minus comparison communities’ slope.

ing trend, compared to prevalence rates for males in the comparison communities). Among the mixed-models for females, there was only one statistically significant difference, covering illicit drug use in the past year by eighth graders, and it was in the reverse direction. Thus, the overall pattern reinforces the initial concern with the pooled, mixedmodel regressions - that they may have “averaged” competing trends, in this case among males and females. For the partnership communities, the results of the partitioned data analysis add substantially to the original picture. Although the aggregate models originally showed only 1 statistically significant difference among 12 dependent variables (Table 2), the partitioned data analysis reveals strong trends for several of the 12 variables - but for the males in the sample only. To the extent that partnership prevention activities were more oriented toward male populations, the results reflect potentially promising outcomes. Findings From the Analysis of Individual Partnerships The identification of eight partnerships with statistically significant reductions in substance abuse, relative to their comparison communities, may be taken as an alternative finding, also addressing the summative question of the entire evaluation. Further, the statistically significant results were complemented by similar ranked

patterns, though not statistically significant, when “past year” and “past month” outcomes were aligned. Also to be remembered is that the 24 partnerships had been selected as a random, stratified sample of the entire universe of 251 partnership grants - not as a sample of successful cases. Under these circumstances and pending further analysis, to identify 8 of 24 as having significant reductions also may be taken as a potentially encouraging sign for the Community Partnership Program.

SUMMARY This article has presented the preliminary outcome analysis from a cross-site evaluation of the Community Partnership Program. The analysis was based on data on substance abuse prevalence, collected from 24 partnership communities and their matched comparison communities, at 2 points in time. The data were analyzed two ways, reflecting alternative definitions of the partnership program: (1) a pooled analysis based on the assumption that the same intervention was implemented at all 24 partnership sites, and (2) individual paired comparisons based on the assumption that different and possibly unique interventions were implemented at each

ROBERT

354

FEMALE SAMPLE:

SUMMARY

et al.

K. YIN

TABLE 8 OF MIXED-MODEL

REGRESSION

RESULTS

Outcome variables

Difference in slopes

df

F

Covariance parameter estimateratio 0

ICC

P

n

Adult

Illicit drug use in the last year Illicit drug use in the last month Alcohol use in the last year Alcohol use in the last month

- 0.00006 0.00012 -0.00016 -0.00010

46 46 46 46

0.01 0.06 0.02 0.80

0.00395818 0.00327560 0.00493657 0.00517793

0.0039 0.0033 0.0049 0.0052

0.9191 0.8075 0.8783 0.3744

14840 14840 14840 14811

10th grade

Illicit drug use in the last year Illicit drug use in the last month Alcohol use in the last year Alcohol use in the last month

0.00054 0.00040 -0.00008 -0.00082

41 41 41 41

0.22 0.17 0.01 0.56

0.00000994 0.00000867 0.00000441 0.00000228

0.0000 0.0000 0.0000 0.0000

0.6388 0.6851 0.9422 0.4602

13239 13240 13229 13229

8th grade

Illicit drug use in the last year Illicit drug use in the last month Alcohol use in the last year Alcohol use in the last month

0.00306 0.00245 0.00059 0.00112

41 41 41 41

4.81 4.00 0.15 0.67

0.00000160 0.00000000 0.00000000 0.00000000

0.0000 0.0000 0.0000 0.0000

0.0341 0.0522 0.6974 0.4182

15265 15268 15255 15250

Age groups

Key: df= degrees of freedom. F=Type Ill F. ICC = lntraclass Correlation Calculated as e/(1+ 0). p= probability of significance. n=size of sample used in this analysis. N/A = not applicable. Scaled weight, calculated as base weight (l/(MeanBaseWeight)), was used. Time, defined as months between grant award and survey administration date, was used. Community cluster was used. Individual confounders were used to control for the effects of age, race, education, employment status, and income (only age, gender, and race were used in the youth data). Difference in slope was estimated as partnership communities’ slope minus comparison communities’ slope.

site. Both analyses used regression models, controlling for the effects of individual confounders such as age, gender, and race. The results of both analyses were weak, but with significant differences in the predicted direction: Partnerships’ prevalence rates were lower over the two points in time compared to the comparison communities’ prevalence rates. The analysis was preliminary in that few variables were investigated beyond the main outcomes and the main “treatment” condition - whether a community was a partnership community or a comparison community. Much further work needs to be continued before arriving at any definitive interpretation of the partnerships’ effects. However, important alternative explanations already have been addressed, in conducting both the pooled and individual analyses, in the present article. REFERENCES Bryk, A. S., & Raudenbusch, S. W. (1987). Application of hierarchial linear models to assessing change. Psyehologicaf Butfetin, JOJ(l), 147158. Center for Substance Abuse Prevention. (1991). Preuention plus III: Assessing alcohol and other drug prevention programs at the school and community level-A four-step guide to usefulprogram assessment.

Rockville, MD: Alcohol, Drug Abuse, and Mental Health istration, U.S. Department of Health and Human Services.

Admin-

Center for Substance Abuse Prevention. (1997, June). Fifth annual report of the national evaluation of the community partnership demonstration program. Rockville, MD: Substance Abuse and Mental Health Service Administration, U.S. Department of Health and Human Services. COMMIT Research Group. (1995). Community intervention trial for smoking cessation (COMMIT): I. Cohort results from a four-year community intervention. American Journal of Public Health, 85(February), 183-192. COMMIT Research Group. (199 I). Community intervention trial for smoking cessation (COMMIT): Summary of design and intervention. Journal of the National Cancer Institute, 83(22), 1620-1628. Cook, T. D., &Campbell, D. T. (1979). Q uasi-experimentation: Design and analysis issuesforJield settings. Boston, MA: Haughton Mifflin. Cornfield, J. (1979). Randomization by group: A formal American Journal of Epidemiology, ZO8(2), lOG102.

analysis.

Dryfoos, J. G. (1990). Adolescents at risk: Prevalence andprevention. New York: Oxford University Press. Gail, M. H., Byar, D. P., Pechacek, T. F., & Corle, D. K. for the COMMIT Study Group. (1992). Aspects of statistical design for the community intervention trial for smoking cessation (COMMIT). Controlled Clinical Trials, 13, 62 1.

Community

Partnership

Goldstein, H. (1987). Multilevel models in education and social research. New York: Oxford University Press. Hanushek, E. A., & Jackson, .I. E. (1979). social sciences. New York: Academic Press.

Program:

Findings

355

Murray, D. M., Hannan, P. J., & Baker, W. L. (1996). Monte Carlo study of alternative responses to intraclass correlation in community trials. Evah4ation Review, 20(3), 3 13-337.

Statistical methods for

Johnston, L. D., O’Malley, P. M., Bachman, J. G., & National Institute on Drug Abuse. (1995). National survey results on drug use from the monitoring the future study, 1975-1994. Rockville, MD: The University of Michigan Institute for Social Research and National Institute on Drug Abuse. Kish, L. (1965). Statistical designfor research. New York: Wiley. Korn, E. L., & Graubard, B. I. (1991). Epidemiologic studies utilizing surveys: Accounting for the sampling design. American Journal of Public Health, 81(9), 11661173. Latour, D., Latour, K., & Wolfinger, R. D. (1994). Getting started with PROC MIXED. Cary, NC: Software Sales and Marketing Department, SAS Institute, Inc. Murray, D. M., & Hannan, P. J. (1990). Planning for the appropriate analysis in school-based drug-use prevention studies. Journal of Constating and Clinical Psychology, 58(4), 458468. Murray, D. M., & Wolfinger, R. D. (1994). Analysis issues in the evaluation of community trials: Progress toward solutions in SAS/ STAT MIXED. Journal of Community Psychology, CSAP Special Issue, 14&l 54.

Murray, D. M., Hannan, P. J., Jacobs, D. R., McGovern, P. J., Schmid, L., Baker, W. L., & Gray, C. (1994). Assessing intervention effects in the Minnesota Heart Health Program. American Journal of Epidemiology, 139(l), 91-103. Perkins, D. D., & Taylor, R. B. (1996). Ecological assessments of community disorder: Their relationship to fear of crime and theoretical implications. American Journal of Community Psychology, 24(l), 63-107. Raudenbusch, linear models: 116.

S. W. (1988). Educational applications of hierarchial A review. Journal of Educational Statistics, 13(2), 85-

SAS Institute, Inc. (1989-1995). SAS/STATsoftware: for Windows: Release 6.11. Cary, NC: SAS Institute,

The SASsystem Inc.

SAS Institute, Inc. (1996). SASISTATsoftware: Change and enhancements through release 6.11. Cary, NC: SAS Institute, Inc. Substance Abuse and Mental Health Services Administration (SAMHSA). (1995). National household survey on drug abuse: Main findings 1993. Rockville, MD: SAMHSA, Office of Applied Studies. Zucker, D. M. (1990). Analysis of variance pitfall: The fixed effects analysis in a nested design. Educational and Psychological Measurement, 50, 731-738.