EarlyChildhood ISSN 0665-2006
ResearchQuarterly, l&407-424
@ Ablex Publishing Corp. All rights of reproduction reserved.
(1997)
FOCUS
Head Start leaching Center: Evaluation of a New Approach to Head Start Staff Development Diane M. Harm-Wingerd David A. Caruso Sheryl Gomes-Atwood Julianna Go/as University of Rhode /s/and
Head Start Teaching Centers are a national demonstration project designed to provide participatory training in all Head Start component areas within the context of an exemplary Head Start program. Each Teaching Center employs an independent evaluation to study this alternative approach to Head Start staff development. This paper presents the results of the outcome evaluation for the first year of the New England Head Start Teaching Center. The New England Head Start Teaching Center was designed to provide intensive training during a 3 or 5 day period of residence at the Teaching Center. This paper briefly describes the national Head Start Teaching Center model, the implementation of this model in the New England region, the outcome evaluation plan, and the results from the first year of training. The analyses of year one data indicated that training provided by the New England Head Start Teaching Center produced significant gains. As compared to similar Head Start employees who did not participate in training, both trainees and their supervisors reported significant gains in trainees’ knowledge, skills, and expertise after participating in the New England Head Start Teaching Center training.
For over 30 years Head Start has been the nation’s largest and most comprehensive program providing early childhood intervention to children and families living in poverty. Because of the unique size and scope of Head Start, the success of ongoing efforts to improve the quality of its programs and services is particularly significant. Recently, a broad consensus has emerged stressing quality improvements as a key initiative for Head Start (U.S. Department of Health and Human Services, 1993; Kassebaum, 1994). Direct all correspondence to: Diane M. Horn-Wingerd, HDF-Transition Road, University of Rhode Island, Kingston, RI 0288 l-08 18.
Center, 2 Lower College
407
408
Horm-Wingerd, Caruso, Gomes-Atwood, and Golas
One area that has been cited as crucial for improving the quality of Head Start is staff training (U.S. Department of Health and Human Services, 1993). A number of studies have documented that the quality of early childhood programs can be improved through staff training (Arnett, 1989; Berk, 1985; Goelman, 1992; Howes, 1983; Vandell & Powers, 1983; Vander Ven, 1994). Furthermore, training that involves learners in participatory, hands-on experience has recently begun to attract the interest of practitioners and investigators as a particularly effective approach. Several studies have found that participatory training is an effective model for a wide range of human services personnel (Franklin & Toifel, 1994; Manzo & Weich, 1994; McGonigle, 1992; Sims, 1990; Warshauer, 1988), including early childhood staff (Epstein, 1993; Jones, 1993; Weikart, 1994; Wolfe, 1994). Epstein (1993), for example, found that training programs that include active participation, observation and feedback, and practical, hands-on experience are most successful. In a recent study involving Head Start personnel, Fantuzzo, Childs, Stevenson, Cook&an, Ginsburg, Gay, Debnam, & Watson (1996), found that participatory training resulted in higher satisfaction with training and more positive classroom practices than the more typical didactic workshop training on similar topics. These findings highlight the importance of a recent Head Start initiative to enhance the quality of services for children and families. The Head Start Teaching Center, a federally funded national demonstration project, was conceived with the primary goal of enhancing staff development in an effort to improve the quality of services offered to Head Start children and their families. Specifically, the Head Start Teaching Center initiative was designed to test the efficacy of participatory training for staff in all components of Head Start services. When the 5 year project was initiated by the Administration for Children, Youth, and Families (ACYF) of the U.S. Department of Health and Human Services in 1992,14 Head Start Teaching Centers were funded including at least one in each Head Start region. As required by ACYF, each Teaching Center project includes an independent evaluation to ascertain the effectiveness of training. The 14 Head Start Teaching Centers are individual Head Start programs deemed by ACYF to provide exemplary Head Start services in all component areas and to be capable of providing quality training to visiting Head Start staff. Head Start Teaching Centers are expected to provide training that is “participatory in nature, relying on observation, guided practice and immediate feedback” (Federal Register, 1992) in each of the four major Head Start components: education, health, social services, and parent involvement. The teaching center concept is based on the teaching hospital model in medicine that provides quality medical services to patients and on-site training to medical students and visiting physicians. As in a teaching hospital setting, practical hands on experiences are supported and enhanced through the use of didactic methods, discussion, and various kinds of simulated situations. Weikart (1994) notes that the application of learned principles is the key to hands on training. He also notes that hands on training encourages questions, discussions, and immediate feedback. Thus in the Head Start Teaching Center model, hands on, participatory activities serve as the foundation of the
Head Start Teaching Center
409
training with accompanying didactic sessions, simulation activities, and discussion focused on the same topic. The 14 Head Start Teaching Centers funded by ACYF represent diverse approaches to implementing the teaching center model. Approaches range from the training of trainers to training individual Head Start direct service or administrative staff. While some programs provide for training a relatively small number of persons with ongoing follow-up activities over an extended period, others consist of 3 to 5 day training sessions without any follow-up training and therefore have the potential to provide training for several hundred trainees in a given year. Evaluation data collected over the 5 year demonstration project should add considerably to our knowledge of the efficacy of these different approaches to participatory training in early childhood education.
The New England Head Start Teaching Center The New England Head Start Teaching Center (NEHSTC), designed and implemented by CHILD, Inc., a Head Start grantee located in Rhode Island, provided intensive training during a relatively short 3 or 5 day period of residence at the Teaching Center. Typical training topics included: cultural diversity, individualizing services for teachers and family workers, home visitor training, home visiting program development for Head Start directors, food safety and sanitation, menu planning, and child initiated activities. All training topics were 3 day sessions except for home visitor training. Due to the logistics of scheduling two home visits for each trainee, the home visitor training was extended to 5 days. Each training session offered by the NEHSTC was composed of four types of learning activities: didactic, group discussion, simulation/role play, and participatory (typically observation and “hands-on” guided practice). The relative amount of time trainees spent in each of these four types of learning activities varied from topic to topic. Training at the NEHSTC was conducted by the Teaching Center Project Coordinator, a staff member hired to coordinate the project, and other component coordinators (e.g., Education, Parent Involvement, Health, and Social Services) as well as the Executive Director. Other Head Start staff assisted with the training through serving as “training partners” during participatory sessions. Training partners were direct service staff of the Head Start program (e.g. classroom teachers, home visitors, cooks) who facilitated the observational and guided practice training activities. Caruso, Horm-Wingerd, and Dickinson (1996) provide a more detailed description of the HSTC model and its implementation in the New England region. These authors describe the development and initiation of the NEHSTC during the 199293 pilot and planning year.
Evaluation Plan The request for proposals for the Head Start Teaching Center mandated an independent evaluation be included for the duration of the 5 year demonstration project (Federal Register, 1992). The independent evaluation team for the NEHSTC consisted of two faculty and one research assistant from a university in southern New England. The evaluation plan for the NEHSTC consisted of two major compo-
410
Horm-Wingerd, Caruso, Gomes-Atwood, and Golas
nents: (1) A formative evaluation of the development of the NEHSTC during the planning year (1992-93) and (2) An outcome evaluation to assess the impact of training during the implementation years of NEHSTC training (1993-97). The purpose of the formative evaluation was to examine the process of developing the Teaching Center including management issues, design of training strategies and topics, preparation of staff for training roles, and evaluation of pilot training. Data were collected from several sources including minutes of all planning meetings, observations of pilot training, self-reports from the Teaching Center staff, and summary evaluations of the pilot training provided by the trainees. The formative evaluation data regarding the process of developing and initiating the NEHSTC are presented and discussed by Caruso, Horm-Wingerd, and Dickinson (1996). In sum, these authors found the pilot training was high quality and that it was participatory in nature as distinct from the more typical didactic training format. It was also reported that the HSTC project had several positive impacts on the host Head Start program and the staff participating in delivering the training. The purpose of the outcome evaluation was to: (1) Document the training process and content; and (2) Evaluate the impact of training on the knowledge, skills, and expertise of the trainees. The outcome evaluation used a nonequivalent comparison group design (Cook & Campbell, 1979; Judd & Kenny, 1981) and involved multi-method techniques to enhance convergent validity (Judd & Kenny, 1981; Smith & Glass, 1987). This approach is consistent with recent literature that stresses the importance of multiple perspectives to increase the reliability and validity of results and decrease the possibility of bias (Jorde Bloom, 1994). The outcome evaluation design, outlined in Table 1, included a comparison group and longitudinal data. Comparison group subjects consisted of staff from several Head Start programs that had not participated in training at the NEHSTC in a given year. Validity of comparisons was enhanced by providing experiences for both groups. The training group received the 3 or 5 day training delivered by the NEHSTC staff. To minimize the Hawthorne effect (Huck, Corrnier, & Bounds, 1984) the comparison group received an alternative “treatment” consisting of a brochure about job related stress and a 3 hour presentation about ways to relieve job related stress. In order to evaluate training outcomes, pre-training, 1 month and 6 month posttraining data were collected. Prior to their participation in the training, trainees were asked to provide a variety of pre-training data including demographics, experience, education level, perceived competencies, perceived needs, and perceived program quality. A supervisor’s report of trainee competencies and needs was also collected prior to training. These data were necessary to make pre- to post-training comparisons and to control for background factors (i.e., demographics, education level, etc.) in the evaluation analysis. The comparison group provided the same pre-training information. Data on training outcomes was collected longitudinally at two time periods: 1 month and 6 months after training. Multiple methods used for assessing training included: (1) Self-report by trainee and comparison groups; (2) Supervisor’s
411
Head Start Teaching Center Table 1.
Outcome Evaluation
Design Group
Data Source 1.
2.
3.
Training Groups
Comparison Group
Pre-training data -trainee self-report -supervisor report
X X
X X
Evaluation of training sessions -observation -records review -trainee report
X X X
Evaluation of training outcomes -trainee self-report 1 month 6 months -supervisor report 1 month 6 months -observation in trainee program 6 months
X X
X X
X X
X X
X
reports on trainees and comparisons; and (3) On-site observation of randomly selected trainees at 6 months post-training. In essence the important questions that drove the outcome evaluation were: (1) What is the nature of training? and (2) Did the training make a difference for the trainees? The focus of this paper is to present the results of the outcome evaluation for the first year of training offered by the NEHSTC. METHOD
Subjects During the 1993-94 training year, 73 trainees participated in the NEHSTC training and 70 individuals participated in the comparison group “treatment,” a stress reduction workshop. Of the 73 potential subjects in the training group, 51 (69.86%) completed all aspects of the evaluation component; of the 70 potential comparisons, 31 (44.29%) completed all aspects of the evaluation component. In general, the subjects for the study were white females between the ages of 25 and 45 who were married with annual family incomes of $30,000 or less. The majority had some college or a B.S. degree and had worked in Head Start from 3 to 5 years. Analyses investigating possible differences between trainee and comparison subjects are presented in the Results section. Analyses indicated that no significant differences existed between individuals (both trainee and comparison subjects) who did and did not complete all portions of the evaluation component on the seven personal (sex, age, marital status, ethnic-
412
Horm-Wingerd, Caruso, Gomes-Atwood, and Galas
ity, education, income, number of children) and five employee characteristics (current Head Start position, years in current position, total years in Head Start, total years working with children in non-Head Start positions, training on similar topics) analyzed. Analyses also indicated no significant within group differences on these same variables for the comparison group. Within the training group on the other hand, individuals who did and did not complete all portions of the evaluation differed on only one variable. Those who did not complete all portions of the evaluation reported receiving less training on similar topics than individuals who completed all evaluation components. Due to the lack of significant differences within and between groups related to attrition, further analyses included only those trainees (5 1) and comparisons (3 1) with complete evaluation data. The decision to adopt stringent criteria for subject inclusion was made in order to capitalize on the power of the research design. To be included in the primary analyses of this study, trainee and comparison subjects must have had pre-training, 1 month and 6 month self and supervisor ratings. Failure to have one of these six data points resulted in elimination from primary analyses. Although these stringent criteria resulted in loss of subjects, especially comparison group subjects, they enabled the researchers to validly assess subjects over time and to assure that the multiple views (self and supervisor ratings) reported information on the same individuals. Alternate inclusion decisions could have enhanced the number of trainee and comparison subjects included in analyses. These alternative decisions, however, would have restricted the appropriate use of repeated measures analyses for the longitudinal assessment of the impact of training and resulted in different subjects being included in analyses using self and supervisor ratings as dependent variables. Additionally, the logic of eliminating subjects was bolstered by the finding of no differences between those who did and did not complete the entire evaluation component on the personal and employee characteristics reported above.
Procedure The names and addresses of trainees and their supervisors were obtained from the NEHSTC staff prior to training. The pre-training mailing, sent to both registered trainees and their supervisors, included: a brief description of the project, an informed consent letter, a demographic survey, and a survey specifically designed to measure knowledge, skills, and expertise related to each training topic. Selfaddressed, stamped return envelopes were provided for trainees and supervisors to independently return completed materials. If all materials were not returned prior to the first session of the training, a member of the evaluation team briefly met with trainees during the first 30 minutes of the training session to collect missing components. Follow-up phone calls were made to supervisors who did not return the necessary pre-training materials. At 1 and 6 months after training, trainees and supervisors were sent a cover letter and copy of the appropriate topic survey. They were asked to return the materials using the self-addressed, stamped envelopes provided by the evaluation team. Follow-up phone calls were used to remind participants to return the information. If
Head Start Teaching Center
413
needed, additional copies of surveys were mailed. Included in the 6 month mailing, participants randomly selected to be observed in their home location were sent a letter informing them of the date of the observation. They were requested to notify the evaluation team of a convenient time to facilitate the on-site observation. If they failed to return this information, a member of the evaluation team contacted the supervisor to inform them of the scheduled visit. At 6 months after training, a member of the evaluation team completed an on-site observation and rated trainees on the implementation of training concepts. A directory of Head Start programs in the New England Region was used to identify those programs that did not participate in NEHSTC training. These programs were contacted by the evaluation team and asked to participate in the project as part of the comparison group. As an incentive, these programs were offered the first opportunity to register employees for future training conducted by the NEHSTC. Comparison group subjects participated in a 3 hour stress reduction workshop designed as a pseudo treatment. While this treatment was considerably shorter than the HSTC training, it was included to minimize a potential Hawthorne effect. The stress reduction workshop was offered to comparison group subjects who held the same job titles as the trainees who participated in training at the NEHSTC. This ensured that the two groups were composed of roughly the same proportion of cooks, teachers, home visitors, and so forth. The comparison group participants completed the same pre-training and post-training materials as the trainees who received training at the NEHSTC. No 6 month on-site visit was scheduled for members of the comparison group. To document the nature and quality of training, a member of the evaluation team attended each training session to determine if the training syllabus was being followed and they completed a form to rate the quality of training within the four learning activities. Allocation of time devoted to each of the four types of learning activities was calculated by the evaluation team from the training syllabus and observations of training sessions. The NEHSTC staff designed and distributed an evaluation form at the end of training that was used to record trainees’ ratings of the quality of each day of the training as well as the overall quality. In addition, trainees were asked to provide ratings on the trainers. This form, therefore, was distributed to all trainees attending the NEHSTC regardless of their decision to participate in the formal evaluation. The NEHSTC staff shared these evaluations with the evaluation team. Instruments All instruments were developed for this study based on the specific training content and strategies used. Curriculum for each training topic, Head Start Performance Standards, and results of pilot testing were used to develop specific items. Instruments were piloted during the NEHSTC planning year (1992-93). Instruments were developed to document subjects’ demographic characteristics, the content and quality of training, and the outcomes associated with training.
414
1. A.
B.
Horm-Wingerd, Caruso, Gomes-Atwood, and Golas Instruments
2. Instruments A.
B.
C.
Information
to Document
the Content and Quality of Training
Types of Learning Activities: Based on the training syllabi and observation of each training session, the evaluation team used this instrument to calculate the percentage of each of the four types of learning activities (didactic, group discussion, simulation/ role play, participatory) included in each training session. Percentages were calculated as the total minutes for each type of learning activity, over the total minutes of the complete training session. Training Evaluation Form: Using a 10 point Likert scale, trainees evaluated the overall quality of the training, including trainers’ competence and skill. Trainees were asked to write explanations for what made them rate a training component as high as they did and what kept them from providing a higher rating. In addition, a 5 point Likert scale was used to rate the trainers’ knowledge, organization, enthusiasm, media use, and facilitation of discussion. The instrument was designed by the NEHSTC staff and was administered by them as the last activity of each training session. Mean scores were calculated to summarize the trainees’ ratings of training content and trainer competence. Training Session Observation Form: After extensive pilot training a member of the evaluation team used this 9 point Likert scale, while observing training, to rate the quality of training delivery within the four types of learning activities: didactic, group discussion, simulation/role play, participatory. The quality of training was operationalized by the specific items included on this scale. Items required the observer to rate the trainers’ enthusiasm, ability to create interest, speaking ability, ability to link hands-on tasks to key concepts, flexibility, facilitation of discussion, and delivery of feedback. Mean scores were calculated to summarize these observational ratings. Inter-rater agreement of 86% was established for this instrument prior to its use in data collection. The inter-rater agreement was calculated as a ratio of the number of agreements over total observations recorded by two independent observers.
3. Instruments A.
to Collect Demographic
Trainee Characteristics: Respondents’ background and demographic information including gender, age, education level and work experience was obtained via a short self report instrument that was distributed prior to training. Trainee Program Characteristics: Distributed to supervisors prior to training, this instrument provided information on the trainees’ Head Start programs. Items included information on the size of the program, the number of children served and the number of program sites.
to Evaluate the Outcome of Training
Trainee Self-Report: The Trainee Self Report, designed by the evaluation team during pilot training, was based on the Head Start performance standards relevant to the training topic, as well as on the specific content of each training session, Trainees were asked to respond to a series of questions using a Likert scale of 1 (very limited) to 5 (excellent), which rated their perceived knowledge, skills and expertise in a specific content area (i.e., food safety, celebrating diversity, child initiated activities). Trainee self reports were administered three times during the evaluation process. Trainees received the first survey prior to training. The second was administered 1 month after training and the third was administered 6 months after the initial training The score for this instrument was calculated as the sum of the Likert ratings for all individual items. Because the curriculum varied for different training topics, the number of items on the Trainee Self-Reports varied by topic. In order to compare
Head Start Teaching Center
B.
C.
415
scores across topics, scores were adjusted to be equivalent to the Trainee Self-Report with the most items. Supewisor Report: This instrument, designed by the evaluation team during pilot training, used the same questions found on the Trainee Self Report instrument. Supervisors were asked to rate their staff member’s knowledge, skills and expertise using a Likert rating scale of 1 (very limited) to 5 (excellent). Like the Trainee SelfReport, the Supervisor Report was administered three times; (1) prior to training; (2) 1 month after training; and (3) 6 months after training. As with the Trainee Self-Report, the score for this instrument was calculated as the sum of the Likert ratings for all individual items. Because the curriculum varied for different training topics, the number of items on the Supervisor Reports varied by topic. In order to compare scores across topics, scores were adjusted to be equivalent to the Supervisor Report with the most items. Home Program 6-Month Observation: Designed by the evaluation team, the instrument was used to collect observational data regarding trainees’ implementation of knowledge and skills covered in training. Prior to its use in data collection, inter-rater agreement of 86% was obtained. The inter-rater agreement was calculated as a ratio of the number of agreements over total observations recorded by two independent observers. Observations were made by a member of the evaluation team in the trainee’s home program 6 months after training. Based on a 9 point Likert scale, trainees were given a low (1 to 3), medium (4 to 6), or high (7 to 9) rating based on the degree (no evidence to strong evidence) to which they implemented concepts that were taught during training.
RESULTS Analyses were completed to investigate the comparability of the trainee and comparison groups, to describe the nature and quality of the training, and to investigate outcomes associated with training. Comparability of Trainee and Comparison Groups A series of chi-square or t-test analyses were completed to investigate the comparability of the trainee and comparison groups. Because a series of analyses were completed, the Bonferroni principle (Hays, 1973) was applied to control the familywise Type I error rate. Participants. Chi square analyses indicated no significant differences between the trainee and comparison group members who completed the evaluation component on the seven personal characteristics (sex, age, marital status, ethnicity, education, income, number of children) or the five employee characteristics (current Head Start position, years in current position, total years in Head Start, total years working with children in non-Head Start positions, training on similar topics). As described earlier, the trainees and comparisons were white females between the ages of 25 and 45 who were married with annual family incomes of $30,000 or less. The majority had some college or a B.S. degree and had worked in Head Start from 3 to 5 years. Program. Using Chi square analyses it was found that comparison and training subjects differed on only one out of the five program characteristics analyzed. Indi-
416
Horm-Wingerd, Caruso, Gomes-Atwood, and Golas
viduals in the comparison group came from programs that served a more urban population (85% of the comparisons; 66% of the training group). T-tests indicated that trainees and comparisons came from similar programs in terms of the average number of children served (216 for trainees; 223 for comparisons), average number of center-based (190 for trainees; 208 for comparisons) and home-based slots (25 for trainees; 15 for comparisons); and number of sites (8 for both).
Implementation of Training Nature of Training. Information
collected during training year 1 (1993-94) indicated that the NEHSTC was successful in capitalizing on the power of the teaching center model by including ample participatory training. The average percent of training time allocated to participatory activities was 37% with 23% didactic, 33% discussion, and 6% simulation. Table 2 lists the percent of time allocated to these four different learning activities for all training topics delivered during 1993-94. Quality of Training. All HSTC trainees, regardless of their decision to participate in the evaluation project, completed the Training Evaluation Form (N = 77) on the final day of each training session, Both these trainee ratings and independent observations completed by the evaluation team suggest the training was high quality. For example, across all training topics, the overall mean rating was 8.3 on a scale of 1 to 10 for training content. Similarly, high mean ratings for trainer competence across all training topics were found. On a scale from 1 to 5, the overall mean ratings were: 4.8 for knowledge, 4.8 for organization, 4.7 for enthusiasm, 4.6 for audio visual use, and 4.8 for facilitation of discussion. In narrative comments, trainees stated that the content of training was relevant and useful, the trainers were knowledgeable and organized, and the “laboratory” nature of the participatory training was invaluable. Additionally, narrative responses indicated the trainees
Table 2. Percent of Time Allocated 93-94 Trainings
to Different Learning Activities
for
Learning Activity Track
Didactic
Discussion
Simulation
Participatory
Celebrating Diversity Food Safety Individualizing Services/Social Serv Individualizing Services/Education Child Initiated Menu Planning Home Visitor Average
26 26 17
49 19 24
0
15 9
2.5 41 50
19
29
6
46
32 22 22 23
32 50 29 33
10
27 25 46 37
Note:
3 2 6
Numbers in table are percentages of time allocated to different learning activities.
Head Start Teaching Center
417
appreciated the opportunity to visit the NEHSTC to view how another program implemented Head Start policies and practices. Trainees reported the “hands-on” approach with “real activities” implemented in the comfortable atmosphere of the Teaching Center enhanced their learning and provided them with useful information that could be applied in their own work sites. Trainees also indicated the group discussions and exchange of ideas with other trainees enabled them to learn many new ideas and different approaches. On the negative side, some trainees reported the need for more “hands-on” activities, especially those that were meaningful and challenging. Some trainees reported the information presented was too generic and repetitive from day-to-day and that they were already familiar with some of the concepts included in the training. The evaluation team provided high ratings for the overall quality of the various learning activities. On a 9-point scale, the mean ratings provided by the evaluation team were 7.4 for participatory activities, 8.0 for didactic, and 7.7 for discussion. Because role play and simulation activities were allocated such a small percentage of time, there was inadequate opportunity to observe and rate this type of learning activity. Outcomes: Correlation of Ratings Overall, modest positive correlations were found between trainee and supervisor ratings at pre-training (I = .36), 1 month after training (Y= .31), and 6 months after training (I = .44). For trainees, the correlations between self and supervisor ratings ranged from little to low positive correlations at pretraining (r = .45), 1 month after training (r = .lS), and 6 months after training (r = .45). A similar range of correlations was found between comparison group self and supervisor ratings at pretraining (r = . lo), 1 month after training (r = .52), and 6 months after training (r = .29). Because these correlations were uniformly low to moderate, both trainee and supervisor ratings were used as dependent variables in further analyses. Outcomes: Impact of Training Table 3 displays the means for self and supervisor ratings for trainee and comparison subjects. A significant difference existed between the trainee and comparison subjects on their pretraining self ratings (t = -2.55, df= 80, p = .Ol), with the comparisons scoring higher. No significant difference was found for the pretraining supervisor ratings of trainees and comparisons (t = -1.35, df= 80. p = .18). Due to the significant difference in the pretraining self ratings, analysis of covariance was determined to be the most appropriate analysis strategy for investigating training outcomes (Stevens, 1992). Analysis of the outcomes for training year 1 (1993-94) indicated that the NEHSTC training had a significant impact. A 2 (group) X 2 (time) repeated measures analysis of covariance, using the self ratings collected prior to training as the covariate, indicated a significant difference between the groups over time, Wilks’ Lambda F(2,91) = 8.0, p = .OOl. Figure 1 shows the adjusted least square means. A Tukey’s HSD indicated significant gains from pretraining (M = 52.67) to posttraining at 1 month (M = 58.92) and at 6 months (M = 60.83) were reported by
418
Horm-Wingerd, Caruso, Gomes-Atwood, and Galas
Table 3. Means for Self and Supervisor Ratings for Trainee and Comparison Subject Ratings Self Group Trainee n=51 Comparison n=31 Note:
pre
1 month
Supervisor 6 months
pre
1 month
6 months
50.43 (10.88)
57.49 (10.23)
60.3 1 (10.12)
50.06 (9.64)
56.84 (9.40)
60.43 (10.73)
56.35 (8.90)
54.77 (9.92)
56.90 (8.25)
53.03 (9.72)
56.35 (9.88)
51.71 (8.38)
Standard deviations are reported in parentheses.
trainees but not comparisons (pretraining M = 52.67; 1 month M = 52.43; 6 month M = 56.05). At 1 month and 6 months after training, trainees reported significant gains in knowledge, skills, and expertise. The comparisons did not report significant gains from pretraining to 1 and 6 months. A similar pattern was found in the analysis of supervisor ratings. A 2 (group) X 2 (time) repeated measures analysis of covariance, using the supervisor pretraining ratings as the covariate, indicated a significant difference between the groups over time, Wilks’ Lambda F(2,78) = 10.1, p = .OOl. Figure 2 shows the adjusted least square means for the supervisor ratings of trainees and comparisons. Again using a Tukey’s HSD, significant gains were found from pretraining (M = 5 1.18) to posttraining at 1 month (M = 57.48) and at 6 months (M = 60.83). The comparisons did not report significant gains from pretraining (M = 5 1.18) to posttraining at 1 month (M = 55.31) and at 6 months (M = 51.06). Secondary analyses of covariance, using only the 1 month data as the dependent variables, were conducted to investigate if attrition between the 1 and 6 month data collection points impacted the outcomes. With the self-ratings, a significant difference was found between trainee (n = 65) and comparison (n = 60) groups at one month, F (1,122) = 24.30, p = .OOl. A similar significant difference was found for the supervisor ratings provided for trainee and comparison groups at 1 month, F( 1,122) = 13.94, p = ,001. These significant differences were consistent with the results of the ANCOVAs that included both the 1 and 6 month data. These consistencies suggest the significant differences found in the primary analyses were not related to attrition, including potential differential drop out as a function of subjects’ perceived competence. At 6 months after training, trainee and comparison participants were asked to respond to an open-ended question regarding any changes in their job performance and to identify factors responsible for promoting change. Both groups reported changes; however, trainees noted more changes regarding specific content taught at the HSTC (e.g., teachers including more cultural activities in curriculum, cooks checking refrigerator temperatures more frequently) and attributed these changes to the training they received. The comparisons reported changes that were generic in nature (e.g., received CDA, more accepting of own mistakes, increased organi-
419
Head Start Teaching Center 65.00
g
60.00
i5 2 P F ? 5 :
55.00 /
/
/
/
---_
50.00
I
pre-training
I
1 month
I
6 months
TIME Figure 1.
Adjusted Means for Self-Ratings of Trainee and Comparison Groups
zational skills, greater understanding of job) and they attributed these changes to factors such as staff support and increased experience gained over time. A random sample of 13 trainees were observed at their home Head Start sites 6 months after training by a member of the evaluation team. When rated on the implementation of the knowledge and skills covered in training, seven received high ratings, four received medium ratings, and two received low ratings. These 6 month observational ratings were found to correlate positively (r = S7) with 6 month supervisors’ ratings. These findings suggest that 6 months after training, these trainees were implementing at least a portion of the training concepts learned at the NEHSTC in their daily work. It should be noted that the 6 month observational data were based
420
Horm-Wingerd, Caruso, Gomes-Atwood, and Golas 65.00
$$I
60.00
i5 I w 3 Ei !!i c/I
55.00
50.00
I
pre-training
I
1 month
I
6 months
TIME Figure 2. Adjusted Means for Supervisor Ratings of Trainee and Comparison Groups Ratings
on a limited number of trainees, cannot be linked to pretraining observations, and were not collected for comparison subjects. However, these results substantiate the high ratings of skills, knowledge, and expertise reported by trainees and their supervisors at 6 months after training. DISCUSSION The findings of this evaluation study, based on the first training year of the NEHSTC, provide a wide range of information concerning the nature, quality, and impact of this approach to implementing the “participatory” training model envi-
Head Start Teaching Center
421
sioned in the ACYF national demonstration project. The results were based on 5 1 training and 31 comparison subjects who completed all instrumentation for the evaluation of the NEHSTC. Data for the study were based on self report, supervisor reports, and on-site observation methods. A key dimension of this evaluation involves the measurement of the specific types of training activities used at the NEHSTC. While training based on lecture and discussion can be delivered in more typical conference room training formats, participatory training requires the use of a “real world” setting in which to gain and analyze “hands-on” practice. Thus the overall objective of the NEHSTC was to design and deliver “participatory” training based on the FUV definition of training that relies on observation, guided practice, and immediate feedback. However, the NEHSTC approach also included didactic, discussion, and simulation training activities that do not require a real-world setting. The rationale for using didactic lessons, discussion, and simulation activities at the NEHSTC was to provide foundation information necessary to make the participatory training most meaningful. A central question for the evaluation, therefore, was whether a substantial amount of participatory training was included. It was found that on average, trainees at the NEHSTC spent more time in participatory activities than in didactic, discussion, or simulation activities. However, when averaged across all training topics, participatory accounted for only 37% of the total training time. The amount of time spent in participatory activities reached 50% on only one of the seven training topics delivered with a range of between 25% and 50%. While this amount of participatory activity clearly distinguishes the HSTC from the more traditional conference staff development approach that typically does not include participatory activity, this analysis does not answer the questions regarding the optimal amount of participatory activity. Given that the costs of sending a trainee for a period of residence at the Teaching Center is relatively expensive, it would seem important to address questions related to maximizing the potential of the “hands-on” participatory component that is unique to the teaching center model. Measures of the quality of training provide some additional insight into issues related to participatory activity and more traditional didactic and discussion sessions. In general, both trainee ratings and independent observations rated the training at the NEHSTC as high quality. In narrative comments most trainees rated the participatory activities as an invaluable component that enhanced their learning in ways that could be directly applied in their own places of work. However, some trainees reported a need for more “hands-on” participatory activity suggesting that the question of an appropriate or optimal amount remain to be addressed. A central question addressed in this evaluation concerned the impact of the NEHSTC model on trainee learning and performance outcomes. In order to examine this issue both trainee and supervisor reports were used as well as comparison group data. It was found that NEHSTC training had a significant impact. At 1 and 6 months after training trainees reported increased knowledge, skills and expertise and comparison group subjects did not report such gains. This self-report finding was corroborated by independent reports of the trainees’ supervisors and indepen-
422
Horm-Wingerd, Caruso, Gomes-Atwood, and Galas
dent observations of their job performance 6 months after training. In addition, in narrative comments 6 months after training, trainees themselves noted improvements in their job performance that were related to specific content from their NEHSTC experience. The pattern of data at 1 and 6 months after training is less consistent for the comparison group. At 1 month after training, the self-ratings of comparisons were similar to their pretraining ratings. However, the ratings provided by supervisors showed an increase at 1 month from pretraining. At 6 months, an opposite pattern was observed. The comparisons rated themselves higher than their pretraining score while their supervisors provided ratings that were similar to the pretraining ratings. The reasons for this inconsistent pattern of self and supervisor ratings with the comparison subjects are not apparent from the available data. Even though this pattern in the comparison data cannot be explained, when the two groups are directly compared in the analyses the results clearly demonstrate a significant difference for the training and comparison groups. These outcome findings clearly suggest that the NEHSTC experience had a positive impact on trainees’ perceptions of their competence and on their actual job performance as rated by supervisors and independently observed. It is important to note that the outcome evaluation focused on the impact of the overall training experience at the NEHSTC. Therefore, while trainees rated participatory activities positively, these findings cannot be attributed specifically to the participatory component of training. One potential limitation involves the high subject attrition rate. Two factors can be used to explain the subject attrition. For the comparison group, data often depended on one supervisor returning rating forms for numerous subjects. The failure of one supervisor to comply often led to incomplete data for many comparison subjects. The majority of the subject attrition in both groups, however, occurred at the 6 month period. One likely explanation for attrition at this period was related to the typical calendar of Head Start programs. For many subjects, the 6 month data collection period occurred during the Head Start programs’ Summer vacations. Subjects’ motivation to comply with study protocols may have been undermined by the fact that they were not working during this period. Also, it should be noted that this study used a comparison group instead of a true control group because of the inability to randomly assign subjects to groups. Due to these limitations, the findings should be interpreted with caution. Overall, the results of this evaluation of the NEHSTC suggest positive outcomes from 1 year of experience in implementing the teaching center participatory model. While questions related to maximizing the participatory nature of this approach remain, especially relative to costs of the model, the positive findings in terms of both training quality and impact suggest that the teaching center model has significant potential as a new approach to Head Start staff development. Data from future years of the NEHSTC evaluation, as well as evaluations of the 13 other national demonstration sites, should further clarify the potential of this model for Head Start or other types of early education staff development.
Head Start Teaching Center
423
Acknowledgment:
This project was supported in part by a grant from the U.S. Administration for Children, Youth, and Families, Head Start Bureau (Grant No. 90CD1047/02) to C.H.I.L.D., Inc. However, the opinions expressed do not necessarily reflect the policy of the ACYF or of C.H.I.L.D., Inc. and no official endorsement should be inferred. The authors are grateful for the cooperation of the staff of C.H.I.L.D., Inc.
REFERENCES Amett, J. (1989). Caregivers in day care centers: Does training matter? Journal of Applied Developmental Psychology, IO, 541-552. Berk, L. (1985). Relationship of caregiver education to child-oriented attitudes, job satisfaction, and behaviors toward children. Child Care Quarterly, 14, 103-109. Caruso, D.A., Horm-Wingerd, D.M., & Dickinson, L. (1996). Head Start Teaching Center: Describing the initiation of a new approach to Head Start staff development. Child and Youth Care Forum, 25(2), 89-99. Cook, T.D., & Campbell, D.T. (1979). Quasi-experimentation: Design and analysis issues forfield settings. Boston: Houghton-Mifflin. Epstein, A.S. (1993). Training for quality: Improving early childhood programs through systematic inservice training. Ypsilanti, MI: High Scope Press. Fantuzzo, J., Childs, S., Stevenson, H., Coolahan, K.C., Ginsburg, M., Gay, K., Debnam, D., & Watson, C. (1996). The Head Start Teaching Center: An evaluation of the experiential, collaborative training model for Head Start teachers and parent volunteers. Early Childhood Research Quarterly, 11, 79-99. Federal Register (1992). Head Start Teaching Centers 1.04, Vol. 7, No. 113, 13-14. Franklin, G., & Toifel, R.C. (1994). The effects of bibliographic instruction on library knowledge and skills among education students. Research Strategies, J2(4), 224-237. Goelman, H. (1992). Vision of program revision: A report on the early childhood education review project. University of British Columbia: Center for the Study for Curriculum and Instruction. Hays, W. (1973). Statistics, 4th ed. New York: Holt, Rinehart, & Winston. Howes, C. (1983). Caregiver behavior in center and family day care. Journal of Applied Developmental Psychology, 4, 99-107. Huck, S.W., Cormier, W.H., & Bounds, W.G. (1984). Reading statistics and research. New York: Harper & Row. Jones, E. (1993). Looking back: What we’ve learned about partnerships. In E. Jones (Ed.), Growing teachers: Partnerships in staffdevelopment (pp. 135-150). Washington, D.C.: National Association for the Education of Young Children. Jorde Bloom, P. (1994). Professional development for leaders: Lessons from Head Start. In J. Johnson & J. McCracken (Eds.), The early childhood career lattice: Perspectives on professional development (pp. 115-120). Washington, D. C.: NAEYC. Judd, C.M., & Kenny, D.A. (1981). Estimating the effects of social interventions. New York: Cambridge University Press. Kassebaum, N.L. (1994). Head Start : Only the best for American’s children. American Psychologist, 49, 123-126. Manzo, L., & Weich, L. (1994). Psychologist-in-trainingprogram evaluation, 1991-1992. Brooklyn, NY: New York City Board of Education. McGonigle, J.J. (1992). Staff training and community consultation. In V.B. Van Hasselt & D.J. Kolko (Eds.), Impatient behavior therapy for children and adolescents (pp. 23-51). New York: Plenum Press. Sims, R. R. (1990). An experiential learning approach to employee training systems. New York: Quorum books. Smith, M.L., & Glass, G.V. (1987). Research and evaluation in education and the social sciences. Englewood Cliffs, NJ: Prentice-Hall.
424
Horm-Wingerd, Caruso, Gomes-Atwood, and Golas
Stevens, J. (1992). Applied multivariate statistics for the social sciences, 2nd ed. Hillsdale, NJ: Erlbaum. U. S. Department of Health and Human Services, Administration on Children, Youth, and Families (1993). Creating a 21st century Head Start: Final report of the advisory committee on Head Start quality and expansion. Washington, D.C.: Administration on Children, Youth, and Families. Vandell, D., & Powers, C. (1983). Day care quality and children’s free play activities. American Journal of Orthopsychiatry, 53,493-500. Vander Ven, K. (1994). Professional development: A contextual model. In J. Johnson & J. McCracken (Eds.), The early childhood career lattice: Perspectives on professional development (pp. 79-88). Washington, D.C.: NAEYC. Warshauer, S. (1988). Inside training and development: Creating effective programs. San Diego, CA: University Associates. Weikart, D. (1994). The research shows: Issues in staff development. In J. Johnson & J. McCracken (Eds.), The early childhood career lattice: Perspectives on professional development (pp. 9699). Washington, D.C.: NAEYC. Wolfe, B. (1994). Effective practices in staff development: Head Start experiences. In J. Johnson & J. McCracken (Eds.), The early childhood career lattice: Perspectives on professional development (pp. 11 l-l 14). Washington, DC.: NAEYC.