- Email: [email protected]
Dissemination of a Job Seekers' Workshop to drug treatment programs
BEHAVIORTHERAPY 19, 143-155, 1988
Dissemination of a Job Seekers' Workshop to Drug Treatment Programs JAMES L. SORENSEN SHARON M . HALL PETER LOEB TIM ALLEN
University of California, San Francisco EDWARD M . GLASER PAUL D . GREENBERG
Human Interaction Research Institute, Los Angeles To compare the effectiveness of techniques for disseminating an innovative job skills training program, 172 drug treatment programs in six USA states were randomly assigned to receive either printed materials only, printed materials plus on-site technical assistance, printed materials plus training at a conference, or no intervention. The treatment disseminated was the Job Seekers' Workshop, a skill training program designed to help drug treatment clients obtain employment. Adoption of the workshop was assessed at three-month follow-up. At three months, adoption rates were: site visit, 28%; conference, 19%; printed material, 4°70; and control, 007o. Dissemination methods that employed personal contact (site visits and conferences) produced significantly more adoption than did printed materials alone. Residential treatment programs were more likely to adopt the workshop than were outpatient drug-free or methadone maintenance programs.
When a new treatment technique is developed and proves to be beneficial, its diffusion throughout the treatment community is by no means automatic. An earlier version of this paper was presented at the joint meeting of the Evaluation Research Society and the Evaluation Network, San Francisco, October, 1984. This project was supported by a grant from the National Institute on Drug Abuse (1R01 DA 03058) to the Human Interaction Research Institute, and by Research Scientist Development Award application (l RO2 DA 0065) to Dr. Hall. The authors are grateful for the assistance of Joyce.Bird, Ph.D., in data analysis, and of William A. Hargreaves, Ph.D., for review of an earlier draft of the manuscript. Requests for reprints should be sent to Dr. Sorensen at Substance Abuse Services (Ward 92), University of California, San Francisco at San Francisco General Hospital, 1001 Potrero Avenue, San Francisco, CA 94110.
143 0005-7894/88/0143-015551.00/0 Copyright 1988 by Association for Advancementof Behavior Therapy All rights of reproduction in any form reserved.
144
SORENSEN ET AL.
Often innovators lack expertise or interest in promoting adoption of their improved treatments. Similarly, practioners frequently do not seek out improved methods, nor are they always eager to adopt innovations when they know about them (Fairweather, Sanders, and Tornatzky, 1974; Fairweather, 1980, Larsen and Nichols, 1972). Effective dissemination techniques are needed to encourage service providers to adopt promising innovations. Many dissemination techniques have been tried, including printed materials, film, tapes, consultation, organizational development, group enhancement, on-site technical assistance, network arrangements, training conferences and workshops, and participant observation. Glaser, Abelson, and Garrison (1983) surveyed more than 2,1300 published and unpublished manuscripts on dissemination. They noted the superiority of interactive consultation methods over such didactic materials as films or printed instructions as a means of promoting transfer and adoption of new knowledge. In a recent review Backer, Liberman, and Kuehnel (1986) conclude that "a key challenge for the future is to provide researchers with the needed skills and information to be able to design research studies that are more likely to lead to utilization" (pg. 117). Dissemination techniques have seldom been compared in rigorous experiments. Fairweather and colleagues completed a series of complex experiments in a large-scale effort to disseminate the Community Lodge concept to mental hospitals using written materials, an on-site workshop, or establishment of a demonstration ward (Fairweather et al., 1974; Fairweather, 1980; Tornatzky, Fergus, Avellar, and Fairweather, 1980). The results showed that hospitals were more likely to agree to accept the written materials or sponsor a workshop, but more likely to make a decision to adopt the lodge concept if they had agreed to establish a demonstration ward. However, self-selection affected the percentages of hospitals deciding to adopt in each condition. In a second phase of the study, programs that had decided to adopt were randomly assigned to receive either a written manual or three visits by a team of consultants. The results showed a significant difference favoring the consultant intervention. However, the measure of outcome was not adoption, but "progress toward adoption," (i.e., ratings of accomplishment in such tasks as acquiring a facility and funding, incorporating, establishing an employment activity for patients, and staff and patient group development). The number of hospitals that implemented a lodge was not reported. Glaser, Coffey, Marks, and Sarason (1967) compared printed materials, conferences, participant-observer site visits, and onsite consultation to disseminate a sheltered workshop training program. Representatives of agencies were interviewed to assess whether any features of the innovative program that they may have adopted were linked to the communications that they may have received. In both the community lodge and sheltered workshop studies, programs were randomly assigned to initial dissemination conditions (printed materials), and subsequent randomization occurred with only the programs that chose to participate in the successive phases of the studies that involved random assignment to more intensive interventions. Thus, the poor prospects could have chosen to not be in the more intensive interventions, which could have given an exaggerated picture of the effect size of the more intensive interventions.
DISSEMINATION OF JOB SEEKERS' WORKSHOP
145
Larsen, Aruntunian, and Finley (1974) investigated the dissemination of innovations to community mental health centers by using written materials, consultant visits, and visits by participants to other sites. The results uncovered no significant differences among dissemination conditions. Fergus (1979) tested the effectiveness of various telephone consultation techniques to supplement workshops on milieu therapy for geriatric programs. It is unclear from the reported results whether the health programs implemented milieu therapy, either as a result of the workshops or of the telephone consultation. Stevens and Tornatzky (1980) used a two-by-two factorial design to study the adoption of program evaluation methodology in substance abuse agencies, finding superior effectiveness for on-site versus telephone and group versus individual consultations. Only the study by Stevens and Tornatzky employed both random assignment of participants to all treatment conditions and used actual adoption of the innovation as an outcome measure. The innovation disseminated in the present study was the Job Seekers' Workshop, a behavioral skills training program designed to help drug treatment clients obtain employment. The workshop involves several sessions totaling 12 to 20 hours, in which drug abusers experience a series of job-seeking skill training activities. The primary activity is practicing job interviews with videotape feedback (an activity that consumes approximately 74°7o of the time of the sessions), but workshop participants also complete job applications, develop and follow up on job leads, and make telephone calls to prospective employers. Controlled studies have demonstrated the workshop's effectiveness in increasing the employment of clients in methadone maintenance (Hall, Loeb, Norton, & Yang, 1977), replicated its effects (Hall, Loeb, LeVois, & Cooper, 1981), and cross-validated the approach with recently paroled exoffenders with heroin abuse histories (Hall, Loeb, Coyne, & Cooper, 1981). In general, the results of dissemination studies support the idea that more intensive, in-person dissemination techniques are superior to using printed materials alone. The conclusions have been limited, however, by the selfselection of programs to be in the evaluations of the more intensive dissemination conditions, and the fact that outcome measures were not always clearly tied to adopting the innovation being disseminated. The present investigation built upon these studies, employing an experimental design that randomly assigned programs to all groups (control, written materials, and two in-person dissemination techniques) and employing adoption of the Job Seekers' Workshop as the primary outcome variable. The hypotheses were: (a) drug treatment programs exposed to the dissemination methods would be more likely to adopt the Job Seekers' Workshop than programs in the no-information control condition, and (b) programs exposed to dissemination methods that included in-person contact would be more likely to adopt the workshop than programs receiving printed materials only.
METHOD Subjects The subjects were 172 drug treatment programs in the states of California, Washington, Oregon, Illinois, Michigan, and Indiana. The criteria for state
146
SORENSEN ET AL.
selection were: geographical accessibility to the research project, availability of a complete state directory from which to develop recruitment lists, and the existence of enough discrete drug treatment programs (not combined with alcohol programs) to warrant recruitment efforts in that state. The criteria for selection of programs were that each: (a) had 20 or more adult clients in treatment; (b) provided separate drug treatment (not combined with alcohol, mental health, or health services); and (c) included one or more of three primary treatment modalities-outpatient drug-free, methadone maintenance, or residential treatment. A "separate" drug program was defined as having at least one staff member specifically assigned to serve drug treatment clients exclusively. These criteria eliminated programs for which the Job Seekers' Workshop would be least appropriate, i.e., those that had few clients, juvenile clients, or were primarily alcohol treatment programs.
Recruitment of Drug Treatment Programs Recruitment began with a mailing to 650 programs listed in state directories, to introduce the project and invite participation. The mailing included a cover letter, a project summary, a participation agreement, a "decline to participate" form, and a preaddressed stamped envelope. All programs that did not respond were telephoned to invite participation. A total of 299 programs listed in the directories were lost from the sample, by virtue of being closed, unresponsive, unlocatable, subsumed under other programs, or not being drug treatment programs. Another 104 programs declined to participate. All programs that returned the participation agreement received a mailed baseline questionnaire. Programs that did not return the questionnaire were telephoned, and if that failed to elicit its return, they were dropped from the project. Thirty-nine programs were dropped in this way. After the questionnaire had been returned, participating programs were screened for the inclusion criteria. Thirty-one did not meet the inclusion criteria. The remaining programs' primary treatment modality was identified (outpatient drug-free, methadone maintenance, or residential treatment), and defined as the program component that the program indicated as having the greatest number of clients. The baseline questionnaire also provided data on program background characteristics, estimates of the likelihood of their implementing the Job Seekers' Workshop, and judgments about potential barriers to its implementation.
Assignment of Programs to Dissemination Conditions The remaining 177 programs were stratified by primary drug treatment modality and randomly assigned on a two-to-one ratio to the less active experimental conditions (printed material, control) or the in-person conditions (technical assistance site visit, conference). Five programs were closed before the study was completed and were dropped from the sample. The number of remaining programs in each condition by primary modality appears in Table 1.
Experimental Conditions Three diffusion methods were developed for disseminating the Workshop:
147
DISSEMINATION OF JOB SEEKERS' WORKSHOP
TABLE 1 NUMBER OF PROGRAMS BY PRIMARY DRUG TREATMENT MODALITY AND EXPERIMENTAL CONDITION
Modality Condition
Printed material Site visit Conference Control Total
Outpatient drug-free
Methadone maintenance
Residential treatment
Total
n
n
n
n
29 16 15 27 87
15 l0 10 18 53
l0 6 6 l0 32
54 32 31 55 172
(1) printed materials (a descriptive booklet and leader's manual), (2) a oneday technical assistance site visit to the program, and (3) a two-day training conference attended by one staff member from each o f several drug treatment programs. Programs were randomly assigned to one of these three conditions or to a no-information control. The latter programs could have learned how to do the workshop from technical reports published in the drug treatment literature. Printed Materials Only: Programs received an illustrated 20-page booklet designed to introduce the Job Seekers' Workshop and summarize the outcome research that demonstrated its effectiveness. They also received the 52 page Leader's Manual (Loeb, LeVois, and Hall, 1982), containing detailed instructions on how to conduct a Job Seekers' Workshop. Technical Assistance Site Visit: Programs received the printed materials as well as one-day training in the Job Seekers' Workshop at their program site. The site visit was coordinated through each program's director with the request that as many staff as possible attend. When the date for the site visit had been arranged, a confirmation letter was mailed, along with the printed materials. The training followed a structured protocol, which featured a onehour demonstration videotape of an actual Job Seekers' Workshop,' role-play practice of the elements of the Job Seekers' Workshop with video feedback, training in the use of the videotape equipment, and discussion of issues related to implementation of the Job Seekers' Workshop. At the end of the training, all participants completed a feedback survey and were informed that collect calls would be accepted for telephone consultation about the Job Seekers' Workshop. All programs were called one week after the site visit to remind them of the availability of telephone consultation. Training Conference: Programs received the printed materials and were each invited to send a representative to a two-day conference at the University of ' The 23-page protocol for the site visit and conference is available from the authors, as is the demonstration videotape.
148
SORENSEN ET AL.
California, San Fransisco. Expenses were paid by the research project. Four conferences were held, each of which included one representative from each of five to eight programs. The conference followed a standard protocol, which featured a small group activity to facilitate group development, the demonstration videotape, role-play practice of the elements of the Job Seekers' Workshop with video feedback, training in the use of video equipment, and discussion of implementation planning in which participants developed a specific plan for the implementation of the Job Seekers' Workshop at their home programs. As in the site visit condition, participants completed an evaluation survey, the offer of additional telephone consultation was made, and all programs were called one week after the conference to remind them of the availability of such consultation. Control: Programs received a letter informing them of their assignment to this condition and indicating that they would receive the printed materials after the follow-up period.
Questionnaire Follow-up All programs were mailed a follow-up questionnaire three months postintervention. On the follow-up questionnaire, respondents indicated their action responses to the intervention. Levels of response included (1) read the booklet, (2) read the Leader's Manual, (3) expressed intent to implement the workshop, (4) made specific plans to implement the workshop (staff and date assigned), (5) adopted elements of the workshop, (6) conducted one workshop, and (7) conducted two or more workshops. In addition, respondents indicated their degree of satisfaction with the dissemination method to which they had been exposed, and any suggestions for improvement of the dissemination techniques.
Verification of Outcome A subsample of 73 programs received additional telephone follow-up after the conclusion of the questionnaire follow-up period. The telephone followup subsample included all programs in the site visit and conference conditions and ten programs in the printed material only condition. These latter ten programs included the five that had indicated adoption of the workshop or elements of it and five randomly selected programs that had indicated no adoption. None of the control group programs were included in the telephone follow-up as none of them had received information from the project about the Job Seekers' Workshop and none indicated any level of adoption. If a program had indicated adoption of the Job Seekers' Workshop or elements thereof on the written questionnaire, the telephone follow-up confirmed the adoption and clarified which elements of the workshop had been implemented. Programs that had previously indicated no adoption were asked to identify barriers to adoption of the workshop, suggest what else the project could have done to facilitate implementation, and make recommendations for future dissemination efforts. The confirmatory telephone calls changed the ratings of adoption for fourteen programs - upgrading them for eight programs and downgrading them for six.
DISSEMINATION OF JOB SEEKERS' WORKSHOP
149
Measurement of Adoption The criterion used for classifying a program as having conducted a Job Seekers' Workshop was that it conducted at least one workshop, using roleplay interview practice (the major activity of the workshop). This criterion of conducting an actual "trial" of the Job Seekers' Workshop goes beyond merely planning the intervention (but not so far as making the workshop a routine part of clinical practice), and reflects a middle-ground standard for adoption of an innovation (Roberts-Gray, 1986). Programs that implemented one or more other elements of the Job Seekers' Workshop (training in identifying vocational interests, finding job leads, completing job applications, and making telephone calls to prospective employers), but did not include roleplay interview practice, were classified as having "adopted elements" of the workshop. Thus, a consistent definition of the distinction between adopting the workshop and adopting elements was applied throughout the confirmed ratings of adoption. For various reasons, six programs did not receive the intervention to which they were assigned (withdrew, equipment failure, unable to attend conference). Another twelve programs withdrew after the intervention or did not cooperate with follow-up. Thus a total of 18 programs either did not receive the intervention to which they were assigned or withdrew after the intervention. Although the experimental groups did not differ [Z2 (3, N = 18) = 2.17, n.s.], in data analysis we made the conservative assumption that the absence of follow-up data or the lack of opportunity to adopt the workshop may have been related to the intervention to which programs were assigned. Therefore, these programs were included in the sample of 172 programs and were assumed in the data analysis not to have adopted the Job Seekers' Workshop. These 18 programs included seven that had been assigned to the printed material only condition, four to the site visit condition, four to the conference condition, and three to the control condition.
RESULTS Analysis o f Baseline Program Characteristics Baseline differences between the experimental groups on 18 background characteristics were examined to determine whether the groups were equivalent before the intervention. These variables reflected several major program characteristics including budget, staff size, treatment duration, client profile, availability of vocational services, and the importance attached to each of six treatment goals. The groups differed significantly on only one of 18 variables, the mean rank of the importance of finding and keeping employment as a treatment goal. The mean rank of this goal by experimental condition was: printed material, 3.5; site visit, 4.0; conference, 4.0; control, 4.3; F (3,166) = 3.26, p < .05 (1 = most important, 6 = least important). Based on the analysis of program characteristics at baseline, the groups can be considered generally equivalent. The effects of geographical location (state) and cohort (order in which pro-
150
SORENSEN ET AL.
grams were recruited and assigned) on the adoption measures were examined, but no significant differences appeared. Consequently, states and cohorts were combined in the outcome analyses. Of the 172 programs in the total sample, 87 (50.6%) were classified as outpatient drug-free, 53 (30.8%) as methadone maintenance, and 32 (18.6%) as residential treatment programs. Overall, 118 programs (68.6%) had only one modality of service; the other 54 programs (31.4%) were multi-modality. The most frequent combination of modalities was methadone maintenance and outpatient drug-free; 57 % o f the multimodality programs included these two modalities. Statistically significant differences (p < .05) were found among the program modalities on numerous background characteristics including budget, staff size, treatment duration, sex of clients, percent of black clients, client employment, availability of vocational services, and ranking of three of six treatment goals (drug abstinence, absence of criminal behavior, and improved family relations). These differences underscore the importance of stratifying by modality in the random assignment of programs to experimental conditions.
Outcome Analyses Eight significance tests were conducted in the key area of adoption of the Job Seekers' Workshop or its elements. To avoid interpreting results that may have occurred by chance, we employed Fisher's method of multiple comparisons (Lindman, 1974) and consequently, for these eight analyses, adopted .00625 as the alpha level. Adoption of the Job Seekers" Workshop. At the three-month follow-up the highest adoption rates were in the site visit condition (28.1%) and the conference condition (19.4%). Only 3.7 % of programs in the printed material condition adopted a workshop, and.. none of the control programs did [~2 (3, N = 172) = 23.42, p < .001]. Table 2 presents the number and percent of programs adopting the workshop a~ross the four experimental conditions. Differences in rate of adoption among the three intervention conditions were also statistically significant [;~2 (2, N -- 117) -- 10.44, p < .01, n.s.]. A test of the difference in rate of adoption between the printed material condition (3.7%) and the in-person interventions (23.8%) also was significant [2~ corrected for continuity (1, N = 117) = 7.92, p < .005]. The difference in rate of adoption between the two in-person interventions did not approach statistical significance. Adoption of Elements of the Job Seekers' Workshop. To assess the full impact of the interventions, differences among the experimental conditions were compared using a broader definition of adoption, which included programs that either had implemented the workshop or had adopted elements of the workshop. Rates of adoption of elements differed among the four experimental conditions in the expected direction: Site visit, 31.3°-/0; conference, 25.8°70; printed material, 5.6%; control, 0.0°70 [X2 (3, N = 172) = 26.05, p < .001]. A significant difference in rate of adoption (workshop or elements) again appeared among the three intervention conditions [g2 (2, N = 117) = 10.78, p < .005], as well as between printed material and the in-person interventions
DISSEMINATION OF JOB SEEKERS' WORKSHOP
15 i
TABLE 2 NUMBER AND PERCENT OF PROGRAMS ADOPTING TIlE JOB SEEKERS' WORKSHOP AT THREE-MONTH FOLLOW-UP Condition Adoption status Did not adopt n 070 Adopted n °70 Column total n 070
Printed material
Site visit
Conference
Control
Total programs
52 96.3
23 71.9
25 80.6
55 100
155 90.1
2 3.7
9 28.1
6 19.4
0 0
17 9.9
54 31.4
32 18.6
31 18.0
55 32.0
172 100
[X2 corrected to continuity (1, N = 117) = 8.95, p < .005]. The higher rate of adoption in the site visit condition as compared to the conference condition was not statistically significant [X2 corrected for continuity (1, N = 63) = 0.13, n.s.].
Additional Analyses: Rate of Adoption by Modality. When treatment modalities were differentiated, 24.1% of residential treatment programs in the sample adopted the workshop, compared with 9.4% of methadone maintenance programs and 5.7°70 of outpatient drug free programs. [X2 (2, N = 172) = 6.85, p < .05]. Usefulness of Job Seekers" Workshop. The 17 programs that implemented the Job Seekers' Workshop rated its usefulness in six areas, on a five-point scale, from "not at all useful" (1) to "extremely useful" (5). At three-month follow-up, "increasing clients' skills in completing job applications" received the highest mean rating (4.18; SD = 0.42) among the eleven programs that had adopted a workshop and also responded to this item, and was closely followed by "increasing client's job interviewing skills" (4.09; SD -~ 0.54; n = 11). In answer to the question, "How acceptable was the Job Seekers' Workshop to clients?", the mean score of acceptability was 4.09 (SD = 0.54; n = 11). Satisfaction with Services. At three-month follow-up, participating programs were asked about satisfaction with the services provided by the research project. The relationship between dissemination condition and satisfaction was significant IF (3,96) = 60.03, p < .001]. Highest rates of satisfaction were in the in-person interventions. On a scale o f 1 to 6, with 1 as the lowest rating and 6 as the highest, the mean satisfaction rating were: 5.52 (conference condition); 5.27 (site visit); 3.89 (printed material); and 2.09 (control). A comparison between programs in the printed material condition and those in the in-person interventions showed that the latter were more likely to respond positively [X2 (3, N = 117) = 46.65, p < .001].
152
SORENSEN ET AL.
DISCUSSION The results confirmed the two major hypotheses of the study. Significantly more drug treatment programs exposed to the experimental dissemination efforts adopted the Job Seekers' Workshop than did programs in the control group. Significantly more programs receiving printed materials plus in-person dissemination techniques (site visit or conference) adopted the workshop than did programs that received printed materials. In addition, a small, but consistent, directional difference favored one-day site visits over two-day centralized conferences. The follow-up finding that programs receiving in-person interventions expressed more satisfaction with the dissemination efforts than those receiving only the printed materials provides additional validation to the results. Indeed, some programs receiving in-person interventions cited a dramatic impact at their agency (Hall, Sorensen, & Loeb, in press). In summary, there was an experimental effect in the predicted direction, in the primary outcome measure of adoption and in some ancillary measures. The control condition, which received the written materials only after the follow-up information had been gathered, showed no adoption. This points out the relative merits of planned dissemination over the traditional academic channels alone. This study contributes to knowledge about dissemination principally in its use of a design that was not confounded by program self-selection for experimental groups. Most previous studies have been characterized by programs choosing to receive particular experimental conditions, which confounded motivation for change with dissemination condition. The present study also used adoption of the innovation as the major outcome variable (rather than intent to use the innovation or less direct measures of organizational change), and it addressed reliability of this major outcome variable. This study joins the work of Stevens and Tornatzky (1980) as a dissemination experiment that employed random assignment without self-selection to any experimental intervention, in which adoption of the innovation was the primary measure of outcome, and in which the results supported the initial hypotheses. The experiment confirmed the findings of studies that did not use random assignment to all experimental conditions (for example Glaser et al., 1967), with respect to the usefulness of in-person dissemination strategies. To our knowledge this is the first study to demonstrate the superior effectiveness of in-person dissemination over printed materials alone, which employed both random assignment to all treatment conditions and used adoption of the innovation as the outcome measure. The present study was limited by several boundary conditions that warrant further investigation. First, the target of the intervention was individual treatment programs, rather than high-level decision-makers or system planners who control organizational resources and influence policy. Adoption rates might be increased by using an alternative dissemination model that involves state and local coordinating or funding agencies. Also, little effort was made to screen programs on factors that might be related to their likelihood of adopting the innovation. Presumably, adoption rates could be improved by filtering out
DISSEMINATION OF JOB SEEKERS' WORKSHOP
153
programs that indicate little motivation to use the innovation and by selecting target programs that have characteristics conducive to adoption. Such an approach would, of course, further limit the generalizability of the results, which are limited already to the program set that agrees to participate in random assignment studies like the present one. Furthermore, the study used as its measure of "adoption" the self-report of program administrators, rather than directly observed use. To some extent adoption of a program is a matter of degree (Palumbo, Maynard-Moody, and Wright, 1984), and such a limited outcome measure does not address the issue of "manifest" versus "true" adoption that has been raised by Rappaport, Seidman, and Davidson (1979). Additionally we did not address cost-benefit issues, which could measure the relative expense of mounting various dissemination strategies balanced against their long-term effects on the social costs of drug abuse. In this study the inperson dissemination techniques apparently were more effective but also presumably were more costly to deliver, and future studies could investigate the relative benefits and costs to disseminators, recipients, and clients in programs. Other boundary conditions of the present study were its limited geographical scope and the fact that it did not measure the effectiveness with drug abusers or the durability of the Job Seekers' Workshops that were implemented by adopting programs. The study does not indicate the degree to which the programs were able to replicate the effects of the original Job Seekers' Workshop studies. As a follow-up to the work reported here, a second dissemination study is testing the effectiveness of intervening at the system level and is focusing on programs that seem more likely to adopt the Job Seekers' Workshop. Researchers should note that confirming the programs' written responses by additional follow-up improves the quality o f data. By making structured follow-up phone calls and probing carefully, this project confirmed ratings of adoption, which did not noticeably raise or lower the rate of adoption, but did alter the outcome for 8°7o of the sample. Had resources been available, it would have been desirable to undertake still more rigorous confirmation procedures, such as follow-up site visits to observe workshops in action. For those interested in the development of better skill training programs in the drug treatment system, this study extends information about the Job Seekers' Workshop. Although both methadone maintenance and outpatient drug-free programs adopted the innovation, residential treatment programs adopted it at a significantly higher rate. This is not surprising, since residential treatment programs usually include employment as a particular focus in later stages of treatment and the Job Seekers' Workshop more directly addresses their programmatic need. It may be that such an approach is most appropriate for residential treatment programs because virtually all will need a job at the end of treatment, but the behavioral basis of the Workshop may not be embraced by milieu-oriented treatment staff. The Job Seekers' Workshop was developed for and evaluated with outpatient addict populations, and there is a need to further investigate its effectiveness with clients of residential therapeutic communities. Finally, there are implications for those responsible for disseminating innovations to the human services. First, diffusion is not automatic, but it can be
154
SORENSEN ET AL.
encouraged. No programs in the control group adopted the Job Seekers' Workshop during the follow-up period, and no programs in the study, when initially contacted, were aware of the specific Job Seekers' Workshop despite the fact that it was the subject of previous research and publications. The conventional wisdom of many articles about dissemination was confirmed by finding that dissemination strategies that include in-person contact were more effective than written materials alone. The issue for system planners is to judge the relative costs of dissemination versus the benefits of programs adopting an improved treatment method. If the goal is to maximize the spread of an innovation, one should consider using in-person dissemination techniques.
REFERENCES Backer, T. E., Liberman, R. P., and Kuehnel, T. G. (1986). Dissemination and adoption of innovative psychosocial interventions. Journal o f Consulting and Clinical Psychology, 54, 111-118. Fairweather, G. W. (Ed.). (1980). The Fairweather Lodge." A twenty-five year retrospective. San Francisco: Jossey-Bass. Fairweather, G. W., Sanders, D. H., & Tornatzky, L. G. (1974). Creating change in mental health organizations. New York: Pergamon. Fergus, E. O. (1979). Telephone change agentry in the diffusion of a program for the elderly. Journal o f Community Psychology, 7, 270-277. Glaser, E. M., Abelson, H. H., & Garrison, K. N. (1983). Putting knowledge to use: Facilitating the diffusion o f knowledge and the implementation o f planned change. San Francisco: Jossey-Bass. Glaser, E. M., Coffey, H. S., Marks, J. B., and Sarason, 1. B. (1967). Utilization o f applicable research and demonstration results. Los Angeles: Human Interaction Research Institute. Hall, S. M., Loeb, P., Coyne, K., & Cooper, J. (1981). Increasing employment in ex-heroin addicts, I: Criminal justice sample. Behavior Therapy, 12, 443-452. Hall, S. M., Loeb, P., LeVois, M., & Cooper, J. (1981). Increasing employment in ex-heroin addicts Ii: Methadone maintenance sample. Behavior Therapy, 12, 453-460. Hall, S. M., Loeb, P., Norton, J., & Yang, R. (1977). Improving vocational placement in drug treatment clients: A pilot study. Addictive Behaviors, 15, 438-441. Hall, S. M., Sorensen, J. L., & Loeb, P. (1988). Development and diffusion of a skill training intervention. In T. G. Baker & D. S. Cannon (Eds.), Addictive disorders: Psychological research on assessment and treatment. New York: Praeger Scientific. Larsen, J. K., Aruntunian, C. A., & Finley, C. J. (1974). Diffusion o f innovations among community mental health centers. Palo Alto, CA: American Institutes for Research. Larsen, J. K., & Nichols, D. G. (1972). If nobody knows you've done it, have y o u . . . ?. Evaluation, 1, 39-44. Lindman, H. P. (1974). Analysis o f variance in complex experimental designs. San Francisco: W. H. Freeman. Loeb, P., LeVois, M., & Hall, S. M. (1982). Leader's manual: Job Seekers" Workshop. (Treatment Research Administrative Report). Rockville, MD: National Institute on Drug Abuse. Palumbo, D. J., Maynard-Moody, S., and Wright, P. (1984). Measuring degrees of successful implementation: Achieving policy versus statutory goals. Evaluation Review, 8, 45-74. Rappaport, J., Seidman, E., & Davidson, W. S. (1979). Demonstration research and manifest vs. true adoption: The natural history of a research project designed to divert adolescents from the legal system. In R. E Munoz, L. R. Snowden, & J. G. Kelly (Eds.) Social and psy-
DISSEMINATION OF JOB SEEKERS' WORKSHOP
155
chological research in community settings." Designing and conducting programs for social and personal well-being (pp. 101-144). San Francisco: Jossey-Bass. Roberts-Gray, C. (1986). Managing the implementation of innovations. Evaluation and Program Planning, 8, 261-269. Stevens, W. E, & Tornatzky, L. G. (1980). The dissemination of evaluation: An experiment. Evaluation Review, 4, 339-354. Tornatzky, L. G., Fergus, E. O., Avellar, J. W., & Fairweather, G. W. (1980). Innovation andsocial process. New York: Pergamon. RECEIVED: October 14, 1986
FINAL ACCEPTANCE: July 22, 1987
Dissemination of a Job Seekers' Workshop to Drug Treatment Programs JAMES L. SORENSEN SHARON M . HALL PETER LOEB TIM ALLEN
University of California, San Francisco EDWARD M . GLASER PAUL D . GREENBERG
Human Interaction Research Institute, Los Angeles To compare the effectiveness of techniques for disseminating an innovative job skills training program, 172 drug treatment programs in six USA states were randomly assigned to receive either printed materials only, printed materials plus on-site technical assistance, printed materials plus training at a conference, or no intervention. The treatment disseminated was the Job Seekers' Workshop, a skill training program designed to help drug treatment clients obtain employment. Adoption of the workshop was assessed at three-month follow-up. At three months, adoption rates were: site visit, 28%; conference, 19%; printed material, 4°70; and control, 007o. Dissemination methods that employed personal contact (site visits and conferences) produced significantly more adoption than did printed materials alone. Residential treatment programs were more likely to adopt the workshop than were outpatient drug-free or methadone maintenance programs.
When a new treatment technique is developed and proves to be beneficial, its diffusion throughout the treatment community is by no means automatic. An earlier version of this paper was presented at the joint meeting of the Evaluation Research Society and the Evaluation Network, San Francisco, October, 1984. This project was supported by a grant from the National Institute on Drug Abuse (1R01 DA 03058) to the Human Interaction Research Institute, and by Research Scientist Development Award application (l RO2 DA 0065) to Dr. Hall. The authors are grateful for the assistance of Joyce.Bird, Ph.D., in data analysis, and of William A. Hargreaves, Ph.D., for review of an earlier draft of the manuscript. Requests for reprints should be sent to Dr. Sorensen at Substance Abuse Services (Ward 92), University of California, San Francisco at San Francisco General Hospital, 1001 Potrero Avenue, San Francisco, CA 94110.
143 0005-7894/88/0143-015551.00/0 Copyright 1988 by Association for Advancementof Behavior Therapy All rights of reproduction in any form reserved.
144
SORENSEN ET AL.
Often innovators lack expertise or interest in promoting adoption of their improved treatments. Similarly, practioners frequently do not seek out improved methods, nor are they always eager to adopt innovations when they know about them (Fairweather, Sanders, and Tornatzky, 1974; Fairweather, 1980, Larsen and Nichols, 1972). Effective dissemination techniques are needed to encourage service providers to adopt promising innovations. Many dissemination techniques have been tried, including printed materials, film, tapes, consultation, organizational development, group enhancement, on-site technical assistance, network arrangements, training conferences and workshops, and participant observation. Glaser, Abelson, and Garrison (1983) surveyed more than 2,1300 published and unpublished manuscripts on dissemination. They noted the superiority of interactive consultation methods over such didactic materials as films or printed instructions as a means of promoting transfer and adoption of new knowledge. In a recent review Backer, Liberman, and Kuehnel (1986) conclude that "a key challenge for the future is to provide researchers with the needed skills and information to be able to design research studies that are more likely to lead to utilization" (pg. 117). Dissemination techniques have seldom been compared in rigorous experiments. Fairweather and colleagues completed a series of complex experiments in a large-scale effort to disseminate the Community Lodge concept to mental hospitals using written materials, an on-site workshop, or establishment of a demonstration ward (Fairweather et al., 1974; Fairweather, 1980; Tornatzky, Fergus, Avellar, and Fairweather, 1980). The results showed that hospitals were more likely to agree to accept the written materials or sponsor a workshop, but more likely to make a decision to adopt the lodge concept if they had agreed to establish a demonstration ward. However, self-selection affected the percentages of hospitals deciding to adopt in each condition. In a second phase of the study, programs that had decided to adopt were randomly assigned to receive either a written manual or three visits by a team of consultants. The results showed a significant difference favoring the consultant intervention. However, the measure of outcome was not adoption, but "progress toward adoption," (i.e., ratings of accomplishment in such tasks as acquiring a facility and funding, incorporating, establishing an employment activity for patients, and staff and patient group development). The number of hospitals that implemented a lodge was not reported. Glaser, Coffey, Marks, and Sarason (1967) compared printed materials, conferences, participant-observer site visits, and onsite consultation to disseminate a sheltered workshop training program. Representatives of agencies were interviewed to assess whether any features of the innovative program that they may have adopted were linked to the communications that they may have received. In both the community lodge and sheltered workshop studies, programs were randomly assigned to initial dissemination conditions (printed materials), and subsequent randomization occurred with only the programs that chose to participate in the successive phases of the studies that involved random assignment to more intensive interventions. Thus, the poor prospects could have chosen to not be in the more intensive interventions, which could have given an exaggerated picture of the effect size of the more intensive interventions.
DISSEMINATION OF JOB SEEKERS' WORKSHOP
145
Larsen, Aruntunian, and Finley (1974) investigated the dissemination of innovations to community mental health centers by using written materials, consultant visits, and visits by participants to other sites. The results uncovered no significant differences among dissemination conditions. Fergus (1979) tested the effectiveness of various telephone consultation techniques to supplement workshops on milieu therapy for geriatric programs. It is unclear from the reported results whether the health programs implemented milieu therapy, either as a result of the workshops or of the telephone consultation. Stevens and Tornatzky (1980) used a two-by-two factorial design to study the adoption of program evaluation methodology in substance abuse agencies, finding superior effectiveness for on-site versus telephone and group versus individual consultations. Only the study by Stevens and Tornatzky employed both random assignment of participants to all treatment conditions and used actual adoption of the innovation as an outcome measure. The innovation disseminated in the present study was the Job Seekers' Workshop, a behavioral skills training program designed to help drug treatment clients obtain employment. The workshop involves several sessions totaling 12 to 20 hours, in which drug abusers experience a series of job-seeking skill training activities. The primary activity is practicing job interviews with videotape feedback (an activity that consumes approximately 74°7o of the time of the sessions), but workshop participants also complete job applications, develop and follow up on job leads, and make telephone calls to prospective employers. Controlled studies have demonstrated the workshop's effectiveness in increasing the employment of clients in methadone maintenance (Hall, Loeb, Norton, & Yang, 1977), replicated its effects (Hall, Loeb, LeVois, & Cooper, 1981), and cross-validated the approach with recently paroled exoffenders with heroin abuse histories (Hall, Loeb, Coyne, & Cooper, 1981). In general, the results of dissemination studies support the idea that more intensive, in-person dissemination techniques are superior to using printed materials alone. The conclusions have been limited, however, by the selfselection of programs to be in the evaluations of the more intensive dissemination conditions, and the fact that outcome measures were not always clearly tied to adopting the innovation being disseminated. The present investigation built upon these studies, employing an experimental design that randomly assigned programs to all groups (control, written materials, and two in-person dissemination techniques) and employing adoption of the Job Seekers' Workshop as the primary outcome variable. The hypotheses were: (a) drug treatment programs exposed to the dissemination methods would be more likely to adopt the Job Seekers' Workshop than programs in the no-information control condition, and (b) programs exposed to dissemination methods that included in-person contact would be more likely to adopt the workshop than programs receiving printed materials only.
METHOD Subjects The subjects were 172 drug treatment programs in the states of California, Washington, Oregon, Illinois, Michigan, and Indiana. The criteria for state
146
SORENSEN ET AL.
selection were: geographical accessibility to the research project, availability of a complete state directory from which to develop recruitment lists, and the existence of enough discrete drug treatment programs (not combined with alcohol programs) to warrant recruitment efforts in that state. The criteria for selection of programs were that each: (a) had 20 or more adult clients in treatment; (b) provided separate drug treatment (not combined with alcohol, mental health, or health services); and (c) included one or more of three primary treatment modalities-outpatient drug-free, methadone maintenance, or residential treatment. A "separate" drug program was defined as having at least one staff member specifically assigned to serve drug treatment clients exclusively. These criteria eliminated programs for which the Job Seekers' Workshop would be least appropriate, i.e., those that had few clients, juvenile clients, or were primarily alcohol treatment programs.
Recruitment of Drug Treatment Programs Recruitment began with a mailing to 650 programs listed in state directories, to introduce the project and invite participation. The mailing included a cover letter, a project summary, a participation agreement, a "decline to participate" form, and a preaddressed stamped envelope. All programs that did not respond were telephoned to invite participation. A total of 299 programs listed in the directories were lost from the sample, by virtue of being closed, unresponsive, unlocatable, subsumed under other programs, or not being drug treatment programs. Another 104 programs declined to participate. All programs that returned the participation agreement received a mailed baseline questionnaire. Programs that did not return the questionnaire were telephoned, and if that failed to elicit its return, they were dropped from the project. Thirty-nine programs were dropped in this way. After the questionnaire had been returned, participating programs were screened for the inclusion criteria. Thirty-one did not meet the inclusion criteria. The remaining programs' primary treatment modality was identified (outpatient drug-free, methadone maintenance, or residential treatment), and defined as the program component that the program indicated as having the greatest number of clients. The baseline questionnaire also provided data on program background characteristics, estimates of the likelihood of their implementing the Job Seekers' Workshop, and judgments about potential barriers to its implementation.
Assignment of Programs to Dissemination Conditions The remaining 177 programs were stratified by primary drug treatment modality and randomly assigned on a two-to-one ratio to the less active experimental conditions (printed material, control) or the in-person conditions (technical assistance site visit, conference). Five programs were closed before the study was completed and were dropped from the sample. The number of remaining programs in each condition by primary modality appears in Table 1.
Experimental Conditions Three diffusion methods were developed for disseminating the Workshop:
147
DISSEMINATION OF JOB SEEKERS' WORKSHOP
TABLE 1 NUMBER OF PROGRAMS BY PRIMARY DRUG TREATMENT MODALITY AND EXPERIMENTAL CONDITION
Modality Condition
Printed material Site visit Conference Control Total
Outpatient drug-free
Methadone maintenance
Residential treatment
Total
n
n
n
n
29 16 15 27 87
15 l0 10 18 53
l0 6 6 l0 32
54 32 31 55 172
(1) printed materials (a descriptive booklet and leader's manual), (2) a oneday technical assistance site visit to the program, and (3) a two-day training conference attended by one staff member from each o f several drug treatment programs. Programs were randomly assigned to one of these three conditions or to a no-information control. The latter programs could have learned how to do the workshop from technical reports published in the drug treatment literature. Printed Materials Only: Programs received an illustrated 20-page booklet designed to introduce the Job Seekers' Workshop and summarize the outcome research that demonstrated its effectiveness. They also received the 52 page Leader's Manual (Loeb, LeVois, and Hall, 1982), containing detailed instructions on how to conduct a Job Seekers' Workshop. Technical Assistance Site Visit: Programs received the printed materials as well as one-day training in the Job Seekers' Workshop at their program site. The site visit was coordinated through each program's director with the request that as many staff as possible attend. When the date for the site visit had been arranged, a confirmation letter was mailed, along with the printed materials. The training followed a structured protocol, which featured a onehour demonstration videotape of an actual Job Seekers' Workshop,' role-play practice of the elements of the Job Seekers' Workshop with video feedback, training in the use of the videotape equipment, and discussion of issues related to implementation of the Job Seekers' Workshop. At the end of the training, all participants completed a feedback survey and were informed that collect calls would be accepted for telephone consultation about the Job Seekers' Workshop. All programs were called one week after the site visit to remind them of the availability of telephone consultation. Training Conference: Programs received the printed materials and were each invited to send a representative to a two-day conference at the University of ' The 23-page protocol for the site visit and conference is available from the authors, as is the demonstration videotape.
148
SORENSEN ET AL.
California, San Fransisco. Expenses were paid by the research project. Four conferences were held, each of which included one representative from each of five to eight programs. The conference followed a standard protocol, which featured a small group activity to facilitate group development, the demonstration videotape, role-play practice of the elements of the Job Seekers' Workshop with video feedback, training in the use of video equipment, and discussion of implementation planning in which participants developed a specific plan for the implementation of the Job Seekers' Workshop at their home programs. As in the site visit condition, participants completed an evaluation survey, the offer of additional telephone consultation was made, and all programs were called one week after the conference to remind them of the availability of such consultation. Control: Programs received a letter informing them of their assignment to this condition and indicating that they would receive the printed materials after the follow-up period.
Questionnaire Follow-up All programs were mailed a follow-up questionnaire three months postintervention. On the follow-up questionnaire, respondents indicated their action responses to the intervention. Levels of response included (1) read the booklet, (2) read the Leader's Manual, (3) expressed intent to implement the workshop, (4) made specific plans to implement the workshop (staff and date assigned), (5) adopted elements of the workshop, (6) conducted one workshop, and (7) conducted two or more workshops. In addition, respondents indicated their degree of satisfaction with the dissemination method to which they had been exposed, and any suggestions for improvement of the dissemination techniques.
Verification of Outcome A subsample of 73 programs received additional telephone follow-up after the conclusion of the questionnaire follow-up period. The telephone followup subsample included all programs in the site visit and conference conditions and ten programs in the printed material only condition. These latter ten programs included the five that had indicated adoption of the workshop or elements of it and five randomly selected programs that had indicated no adoption. None of the control group programs were included in the telephone follow-up as none of them had received information from the project about the Job Seekers' Workshop and none indicated any level of adoption. If a program had indicated adoption of the Job Seekers' Workshop or elements thereof on the written questionnaire, the telephone follow-up confirmed the adoption and clarified which elements of the workshop had been implemented. Programs that had previously indicated no adoption were asked to identify barriers to adoption of the workshop, suggest what else the project could have done to facilitate implementation, and make recommendations for future dissemination efforts. The confirmatory telephone calls changed the ratings of adoption for fourteen programs - upgrading them for eight programs and downgrading them for six.
DISSEMINATION OF JOB SEEKERS' WORKSHOP
149
Measurement of Adoption The criterion used for classifying a program as having conducted a Job Seekers' Workshop was that it conducted at least one workshop, using roleplay interview practice (the major activity of the workshop). This criterion of conducting an actual "trial" of the Job Seekers' Workshop goes beyond merely planning the intervention (but not so far as making the workshop a routine part of clinical practice), and reflects a middle-ground standard for adoption of an innovation (Roberts-Gray, 1986). Programs that implemented one or more other elements of the Job Seekers' Workshop (training in identifying vocational interests, finding job leads, completing job applications, and making telephone calls to prospective employers), but did not include roleplay interview practice, were classified as having "adopted elements" of the workshop. Thus, a consistent definition of the distinction between adopting the workshop and adopting elements was applied throughout the confirmed ratings of adoption. For various reasons, six programs did not receive the intervention to which they were assigned (withdrew, equipment failure, unable to attend conference). Another twelve programs withdrew after the intervention or did not cooperate with follow-up. Thus a total of 18 programs either did not receive the intervention to which they were assigned or withdrew after the intervention. Although the experimental groups did not differ [Z2 (3, N = 18) = 2.17, n.s.], in data analysis we made the conservative assumption that the absence of follow-up data or the lack of opportunity to adopt the workshop may have been related to the intervention to which programs were assigned. Therefore, these programs were included in the sample of 172 programs and were assumed in the data analysis not to have adopted the Job Seekers' Workshop. These 18 programs included seven that had been assigned to the printed material only condition, four to the site visit condition, four to the conference condition, and three to the control condition.
RESULTS Analysis o f Baseline Program Characteristics Baseline differences between the experimental groups on 18 background characteristics were examined to determine whether the groups were equivalent before the intervention. These variables reflected several major program characteristics including budget, staff size, treatment duration, client profile, availability of vocational services, and the importance attached to each of six treatment goals. The groups differed significantly on only one of 18 variables, the mean rank of the importance of finding and keeping employment as a treatment goal. The mean rank of this goal by experimental condition was: printed material, 3.5; site visit, 4.0; conference, 4.0; control, 4.3; F (3,166) = 3.26, p < .05 (1 = most important, 6 = least important). Based on the analysis of program characteristics at baseline, the groups can be considered generally equivalent. The effects of geographical location (state) and cohort (order in which pro-
150
SORENSEN ET AL.
grams were recruited and assigned) on the adoption measures were examined, but no significant differences appeared. Consequently, states and cohorts were combined in the outcome analyses. Of the 172 programs in the total sample, 87 (50.6%) were classified as outpatient drug-free, 53 (30.8%) as methadone maintenance, and 32 (18.6%) as residential treatment programs. Overall, 118 programs (68.6%) had only one modality of service; the other 54 programs (31.4%) were multi-modality. The most frequent combination of modalities was methadone maintenance and outpatient drug-free; 57 % o f the multimodality programs included these two modalities. Statistically significant differences (p < .05) were found among the program modalities on numerous background characteristics including budget, staff size, treatment duration, sex of clients, percent of black clients, client employment, availability of vocational services, and ranking of three of six treatment goals (drug abstinence, absence of criminal behavior, and improved family relations). These differences underscore the importance of stratifying by modality in the random assignment of programs to experimental conditions.
Outcome Analyses Eight significance tests were conducted in the key area of adoption of the Job Seekers' Workshop or its elements. To avoid interpreting results that may have occurred by chance, we employed Fisher's method of multiple comparisons (Lindman, 1974) and consequently, for these eight analyses, adopted .00625 as the alpha level. Adoption of the Job Seekers" Workshop. At the three-month follow-up the highest adoption rates were in the site visit condition (28.1%) and the conference condition (19.4%). Only 3.7 % of programs in the printed material condition adopted a workshop, and.. none of the control programs did [~2 (3, N = 172) = 23.42, p < .001]. Table 2 presents the number and percent of programs adopting the workshop a~ross the four experimental conditions. Differences in rate of adoption among the three intervention conditions were also statistically significant [;~2 (2, N -- 117) -- 10.44, p < .01, n.s.]. A test of the difference in rate of adoption between the printed material condition (3.7%) and the in-person interventions (23.8%) also was significant [2~ corrected for continuity (1, N = 117) = 7.92, p < .005]. The difference in rate of adoption between the two in-person interventions did not approach statistical significance. Adoption of Elements of the Job Seekers' Workshop. To assess the full impact of the interventions, differences among the experimental conditions were compared using a broader definition of adoption, which included programs that either had implemented the workshop or had adopted elements of the workshop. Rates of adoption of elements differed among the four experimental conditions in the expected direction: Site visit, 31.3°-/0; conference, 25.8°70; printed material, 5.6%; control, 0.0°70 [X2 (3, N = 172) = 26.05, p < .001]. A significant difference in rate of adoption (workshop or elements) again appeared among the three intervention conditions [g2 (2, N = 117) = 10.78, p < .005], as well as between printed material and the in-person interventions
DISSEMINATION OF JOB SEEKERS' WORKSHOP
15 i
TABLE 2 NUMBER AND PERCENT OF PROGRAMS ADOPTING TIlE JOB SEEKERS' WORKSHOP AT THREE-MONTH FOLLOW-UP Condition Adoption status Did not adopt n 070 Adopted n °70 Column total n 070
Printed material
Site visit
Conference
Control
Total programs
52 96.3
23 71.9
25 80.6
55 100
155 90.1
2 3.7
9 28.1
6 19.4
0 0
17 9.9
54 31.4
32 18.6
31 18.0
55 32.0
172 100
[X2 corrected to continuity (1, N = 117) = 8.95, p < .005]. The higher rate of adoption in the site visit condition as compared to the conference condition was not statistically significant [X2 corrected for continuity (1, N = 63) = 0.13, n.s.].
Additional Analyses: Rate of Adoption by Modality. When treatment modalities were differentiated, 24.1% of residential treatment programs in the sample adopted the workshop, compared with 9.4% of methadone maintenance programs and 5.7°70 of outpatient drug free programs. [X2 (2, N = 172) = 6.85, p < .05]. Usefulness of Job Seekers" Workshop. The 17 programs that implemented the Job Seekers' Workshop rated its usefulness in six areas, on a five-point scale, from "not at all useful" (1) to "extremely useful" (5). At three-month follow-up, "increasing clients' skills in completing job applications" received the highest mean rating (4.18; SD = 0.42) among the eleven programs that had adopted a workshop and also responded to this item, and was closely followed by "increasing client's job interviewing skills" (4.09; SD -~ 0.54; n = 11). In answer to the question, "How acceptable was the Job Seekers' Workshop to clients?", the mean score of acceptability was 4.09 (SD = 0.54; n = 11). Satisfaction with Services. At three-month follow-up, participating programs were asked about satisfaction with the services provided by the research project. The relationship between dissemination condition and satisfaction was significant IF (3,96) = 60.03, p < .001]. Highest rates of satisfaction were in the in-person interventions. On a scale o f 1 to 6, with 1 as the lowest rating and 6 as the highest, the mean satisfaction rating were: 5.52 (conference condition); 5.27 (site visit); 3.89 (printed material); and 2.09 (control). A comparison between programs in the printed material condition and those in the in-person interventions showed that the latter were more likely to respond positively [X2 (3, N = 117) = 46.65, p < .001].
152
SORENSEN ET AL.
DISCUSSION The results confirmed the two major hypotheses of the study. Significantly more drug treatment programs exposed to the experimental dissemination efforts adopted the Job Seekers' Workshop than did programs in the control group. Significantly more programs receiving printed materials plus in-person dissemination techniques (site visit or conference) adopted the workshop than did programs that received printed materials. In addition, a small, but consistent, directional difference favored one-day site visits over two-day centralized conferences. The follow-up finding that programs receiving in-person interventions expressed more satisfaction with the dissemination efforts than those receiving only the printed materials provides additional validation to the results. Indeed, some programs receiving in-person interventions cited a dramatic impact at their agency (Hall, Sorensen, & Loeb, in press). In summary, there was an experimental effect in the predicted direction, in the primary outcome measure of adoption and in some ancillary measures. The control condition, which received the written materials only after the follow-up information had been gathered, showed no adoption. This points out the relative merits of planned dissemination over the traditional academic channels alone. This study contributes to knowledge about dissemination principally in its use of a design that was not confounded by program self-selection for experimental groups. Most previous studies have been characterized by programs choosing to receive particular experimental conditions, which confounded motivation for change with dissemination condition. The present study also used adoption of the innovation as the major outcome variable (rather than intent to use the innovation or less direct measures of organizational change), and it addressed reliability of this major outcome variable. This study joins the work of Stevens and Tornatzky (1980) as a dissemination experiment that employed random assignment without self-selection to any experimental intervention, in which adoption of the innovation was the primary measure of outcome, and in which the results supported the initial hypotheses. The experiment confirmed the findings of studies that did not use random assignment to all experimental conditions (for example Glaser et al., 1967), with respect to the usefulness of in-person dissemination strategies. To our knowledge this is the first study to demonstrate the superior effectiveness of in-person dissemination over printed materials alone, which employed both random assignment to all treatment conditions and used adoption of the innovation as the outcome measure. The present study was limited by several boundary conditions that warrant further investigation. First, the target of the intervention was individual treatment programs, rather than high-level decision-makers or system planners who control organizational resources and influence policy. Adoption rates might be increased by using an alternative dissemination model that involves state and local coordinating or funding agencies. Also, little effort was made to screen programs on factors that might be related to their likelihood of adopting the innovation. Presumably, adoption rates could be improved by filtering out
DISSEMINATION OF JOB SEEKERS' WORKSHOP
153
programs that indicate little motivation to use the innovation and by selecting target programs that have characteristics conducive to adoption. Such an approach would, of course, further limit the generalizability of the results, which are limited already to the program set that agrees to participate in random assignment studies like the present one. Furthermore, the study used as its measure of "adoption" the self-report of program administrators, rather than directly observed use. To some extent adoption of a program is a matter of degree (Palumbo, Maynard-Moody, and Wright, 1984), and such a limited outcome measure does not address the issue of "manifest" versus "true" adoption that has been raised by Rappaport, Seidman, and Davidson (1979). Additionally we did not address cost-benefit issues, which could measure the relative expense of mounting various dissemination strategies balanced against their long-term effects on the social costs of drug abuse. In this study the inperson dissemination techniques apparently were more effective but also presumably were more costly to deliver, and future studies could investigate the relative benefits and costs to disseminators, recipients, and clients in programs. Other boundary conditions of the present study were its limited geographical scope and the fact that it did not measure the effectiveness with drug abusers or the durability of the Job Seekers' Workshops that were implemented by adopting programs. The study does not indicate the degree to which the programs were able to replicate the effects of the original Job Seekers' Workshop studies. As a follow-up to the work reported here, a second dissemination study is testing the effectiveness of intervening at the system level and is focusing on programs that seem more likely to adopt the Job Seekers' Workshop. Researchers should note that confirming the programs' written responses by additional follow-up improves the quality o f data. By making structured follow-up phone calls and probing carefully, this project confirmed ratings of adoption, which did not noticeably raise or lower the rate of adoption, but did alter the outcome for 8°7o of the sample. Had resources been available, it would have been desirable to undertake still more rigorous confirmation procedures, such as follow-up site visits to observe workshops in action. For those interested in the development of better skill training programs in the drug treatment system, this study extends information about the Job Seekers' Workshop. Although both methadone maintenance and outpatient drug-free programs adopted the innovation, residential treatment programs adopted it at a significantly higher rate. This is not surprising, since residential treatment programs usually include employment as a particular focus in later stages of treatment and the Job Seekers' Workshop more directly addresses their programmatic need. It may be that such an approach is most appropriate for residential treatment programs because virtually all will need a job at the end of treatment, but the behavioral basis of the Workshop may not be embraced by milieu-oriented treatment staff. The Job Seekers' Workshop was developed for and evaluated with outpatient addict populations, and there is a need to further investigate its effectiveness with clients of residential therapeutic communities. Finally, there are implications for those responsible for disseminating innovations to the human services. First, diffusion is not automatic, but it can be
154
SORENSEN ET AL.
encouraged. No programs in the control group adopted the Job Seekers' Workshop during the follow-up period, and no programs in the study, when initially contacted, were aware of the specific Job Seekers' Workshop despite the fact that it was the subject of previous research and publications. The conventional wisdom of many articles about dissemination was confirmed by finding that dissemination strategies that include in-person contact were more effective than written materials alone. The issue for system planners is to judge the relative costs of dissemination versus the benefits of programs adopting an improved treatment method. If the goal is to maximize the spread of an innovation, one should consider using in-person dissemination techniques.
REFERENCES Backer, T. E., Liberman, R. P., and Kuehnel, T. G. (1986). Dissemination and adoption of innovative psychosocial interventions. Journal o f Consulting and Clinical Psychology, 54, 111-118. Fairweather, G. W. (Ed.). (1980). The Fairweather Lodge." A twenty-five year retrospective. San Francisco: Jossey-Bass. Fairweather, G. W., Sanders, D. H., & Tornatzky, L. G. (1974). Creating change in mental health organizations. New York: Pergamon. Fergus, E. O. (1979). Telephone change agentry in the diffusion of a program for the elderly. Journal o f Community Psychology, 7, 270-277. Glaser, E. M., Abelson, H. H., & Garrison, K. N. (1983). Putting knowledge to use: Facilitating the diffusion o f knowledge and the implementation o f planned change. San Francisco: Jossey-Bass. Glaser, E. M., Coffey, H. S., Marks, J. B., and Sarason, 1. B. (1967). Utilization o f applicable research and demonstration results. Los Angeles: Human Interaction Research Institute. Hall, S. M., Loeb, P., Coyne, K., & Cooper, J. (1981). Increasing employment in ex-heroin addicts, I: Criminal justice sample. Behavior Therapy, 12, 443-452. Hall, S. M., Loeb, P., LeVois, M., & Cooper, J. (1981). Increasing employment in ex-heroin addicts Ii: Methadone maintenance sample. Behavior Therapy, 12, 453-460. Hall, S. M., Loeb, P., Norton, J., & Yang, R. (1977). Improving vocational placement in drug treatment clients: A pilot study. Addictive Behaviors, 15, 438-441. Hall, S. M., Sorensen, J. L., & Loeb, P. (1988). Development and diffusion of a skill training intervention. In T. G. Baker & D. S. Cannon (Eds.), Addictive disorders: Psychological research on assessment and treatment. New York: Praeger Scientific. Larsen, J. K., Aruntunian, C. A., & Finley, C. J. (1974). Diffusion o f innovations among community mental health centers. Palo Alto, CA: American Institutes for Research. Larsen, J. K., & Nichols, D. G. (1972). If nobody knows you've done it, have y o u . . . ?. Evaluation, 1, 39-44. Lindman, H. P. (1974). Analysis o f variance in complex experimental designs. San Francisco: W. H. Freeman. Loeb, P., LeVois, M., & Hall, S. M. (1982). Leader's manual: Job Seekers" Workshop. (Treatment Research Administrative Report). Rockville, MD: National Institute on Drug Abuse. Palumbo, D. J., Maynard-Moody, S., and Wright, P. (1984). Measuring degrees of successful implementation: Achieving policy versus statutory goals. Evaluation Review, 8, 45-74. Rappaport, J., Seidman, E., & Davidson, W. S. (1979). Demonstration research and manifest vs. true adoption: The natural history of a research project designed to divert adolescents from the legal system. In R. E Munoz, L. R. Snowden, & J. G. Kelly (Eds.) Social and psy-
DISSEMINATION OF JOB SEEKERS' WORKSHOP
155
chological research in community settings." Designing and conducting programs for social and personal well-being (pp. 101-144). San Francisco: Jossey-Bass. Roberts-Gray, C. (1986). Managing the implementation of innovations. Evaluation and Program Planning, 8, 261-269. Stevens, W. E, & Tornatzky, L. G. (1980). The dissemination of evaluation: An experiment. Evaluation Review, 4, 339-354. Tornatzky, L. G., Fergus, E. O., Avellar, J. W., & Fairweather, G. W. (1980). Innovation andsocial process. New York: Pergamon. RECEIVED: October 14, 1986
FINAL ACCEPTANCE: July 22, 1987