- Email: [email protected]
Changing strategies in evaluations of preschool programs
Studies in EducationalEvaluation. Vol. 8, pp. 281-289,19~3
0191-491X/83/030281-09504.50/0
Printed in GreatBritain. All rights reserved.
CHANGING
Copyright© 1983PergamonPressLtd.
S T R A T E G I E S IN E V A L U A T I O N S PRESCHOOL PROGRAMS
OF
Edith H. Grotberg and Dennis J. Deloria Administration for Children, Youth and Families, Washington D.C., USA
A number of strategies relating to the evaluation of preschool programs have been changing. O n e c h a n g e has b e e n f r o m e v a l u a t i n g s e r v i c e d e l i v e r y s y s t e m s a n d child development outcomes independent of one another. A second area of change has b e e n f r o m e v a l u a t i n g t h e i m p a c t o f a p a r t i c u l a r i n t e r v e n t i o n on an i n d i v i d u a l child to evaluating t h e i m p a c t o f an i n t e r v e n t i o n w h i c h is p a r t o f a d y n a m i c ( e c o l o g i c a l ) s e t t i n g , on a c h i l d w h o is w i t h i n t h a t d y n a m i c ( e c o l o g i c a l ) s e t t i n g . A t h i r d c h a n g e has b e e n f r o m e v a l u a t i n g s i m p l y in t e r m s o f c h i l d d e v e l o p m e n t o u t comes t o e v a l u a t i n g b o t h o u t c o m e s as w e l l as t h o s e a s p e c t s o f a c h i l d d e v e l o p m e n t program w h i c h can be m a n i p u l a t e d o r c o n t r o l l e d by policy and management decisions. These are not mutually exclusive changes, and, indeed, a good deal o f o v e r l a p w i l l be e v i d e n t . H o w e v e r , t h e r e a r e s u f f i c i e n t d i f f e r e n c e s in a p p r o a c h to warrant separate treatment. Each o f t h e s e a r e a s o f c h a n g e w i l l be d e s c r i b e d largely within the context of programs and concerns of the Administration for Children, Youth and Families (ACYF).
Service Delivery Systems and Child Develolueent Outcomes Foster care systems were historically evaluated by determining the number of c h i l d r e n p l a c e d in f o s t e r c a r e , t h e n u m b e r o f f o s t e r p a r e n t s a v a i l a b l e , a n d t h e n u m b e r o f cases r e s o l v e d e i t h e r in t e r m s o f r e t u r n t o home o r p e r m a n e n t p l a c e ment. A l m o s t n e v e r was t h e i m p a c t o f t h e s e r v i c e d e l i v e r y s y s t e m a s s e s s e d in terms of child development outcomes. I n s t e a d , t h e r e was an i n c r e a s i n g l y s o p h i s ticated method for describing the components of the delivery system. Studies of t h e i m p a c t o f t h e f o s t e r c a r e s y s t e m on c h i l d d e v e l o p m e n t d i d n o t o c c u r t o a n y s i g n i f i c a n t e x t e n t u n t i l t h e w o r k o f D a v i d F a n s h e l a n d his c o l l e a g u e s ( 1 9 7 8 ) . The l o n g i t u d i n a l s t u d y t h e y i n i t i a t e d in 1968 a n d b e g a n r e p o r t i n g on in t h e m i d - 1 9 7 0 s contained sufficiently vigorous methodology to evaluate the impact of the foster c a r e s y s t e m on c h i l d d e v e l o p m e n t . This landmark work changed subsequent evaluation strategies for foster care programs to include both the system and child development outcomes. T h e r e s u l t i n g s y n t h e s i s is b e s t d e s c r i b e d in S y s t e m s o f Social S e r v i c e s f o r C h i l d r e n a n d T h e i r F a m i l i e s : An Overview (1978). T h e i m p a c t of t h e Head S t a r t p r o g r a m was h i s t o r i c a l l y e v a l u a t e d b y d e t e r m i n i n g child development outcomes, particularly those in t h e c o g n i t i v e and socialemotional realms. In a r e v i e w o f t h e H e a d S t a r t e v a l u a t i o n literature in 1 9 6 9 , Orotberg found that child development outcomes were not associated with specific c o m p o n e n t s of t h e Head S t a r t p r o g r a m b u t r a t h e r with t h e total p r o g r a m . Head Start was viewed as a unitary concept with a common impact on children. The 281
282
E. H. Grotberg ~ D. J. Deloria
p r o g r a m e l e m e n t s o f Head S t a r t w e r e n o t s u f f i c i e n t l y described for evaluation u n t i l t h e p u b l i c a t i o n o f t h e Head S t a r t P e r f o r m a n c e S t a n d a r d s in 1975. From that time to the present, Head S t a r t e v a l u a t i o n s h a v e r e l a t e d c h i l d d e v e l o p m e n t o u t comes w i t h s e r v i c e d e l i v e r y s y s t e m s ( p r o g r a m e l e m e n t s ) . A model to e v a l u a t e Head S t a r t is p r e s e n t e d h e r e t o i l l u s t r a t e t h e s y n t h e s i s o f program elements and child development outcomes (NTS, 1980). The model, frequently c a l l e d a l o g i c m o d e l , assumes a l o g i c a l s t r u c t u r e t o a p r o g r a m w h i c h included intended resource inputs, activities, outcomes or impacts. T h e model also assumes that there are causal relationships among resource inputs, program activites, and outcomes or impacts. The components of the conceptual framework o f t h e l o g i c model a r e d e s c r i b e d h e r e w i t h a s k e t c h o f t h e model p r e s e n t e d in F i g u r e 1. T h e model c o m p o n e n t s a r e as f o l l o w s : 1. Determinants of Implementation--the v a r i a b l e s a s s u m e d t o a f f e c t t h e success o f Head S t a r t p r o g r a m i m p l e m e n t a t i o n s a n d t h e f i d e l i t y of p r o g r a m e l e m e n t s t o t h e Performance Standards. 2. Implementation/Process--the variables that determine o f t h e c h a n g e s g e n e r a t e d b y Head S t a r t p r o g r a m e l e m e n t s . 3. Child Exposure/Opportunity child, distinguishing between c h i l d ' s use o f t h e m .
to Benefit--the the availability
the
nature
and
classroom experience of classroom resources
extent
of and
the the
4. Parent Exposure/Opportunity to Benefit--an analytic approach for correlating e x p o s u r e a n d o p p o r t u n i t y w i t h p a r e n t o u t c o m e s in a s y s t e m a t i c w a y . 5. Child Outcomes - the sions of child development relationships. 6. Parent Outcomes--the home, parent and program,
p h y s i c a l , c o g n i t i v e and e m o t i o n a l - m o t i v a t i o n a l d i m e n o r g a n i z e d i n t o a l o g i c a l s y s t e m of e x p o s u r e - o u t c o m e
relationships between and parent and self.
7. a n d 8. C o v a r i a t e s f o r C h i l d r e n relevant background variables.
parent
and Parents--pretest
and
child,
measures
parent
and
and theoretically
From Single Variables to Ecx)lo(jy T h e s e c o n d c h a n g e in e v a l u a t i o n has b e e n a slow s h i f t f r o m e v a l u a t i n g t h e i m p a c t o f a p a r t i c u l a r i n t e r v e n t i o n on a c h i l d t o e v a l u a t i n g t h e i m p a c t o f an i n t e r v e n t i o n w h i c h is p a r t o f a d y n a m i c ( e c o l o g i c a l ) s e t t i n g , on a c h i l d w h o is w i t h i n t h a t dynamic (ecological) setting. There are a number of reasons for the slowness of the shift. One is t h e i m p a c t on e v a l u a t i o n o f e x p e r i m e n t a l r e s e a r c h w h i c h d r e w h e a v i l y on s i n g l e f a c t o r v a r i a b l e s ( b o t h t h e d e p e n d e n t and t h e i n d e p e n d e n t v a r i ables) and which assumed linear causality between a given developmental condition a n d an o u t c o m e . A n o t h e r r e a s o n is an i n t e r v e n t i o n model w h i c h p o s i t s t h a t a given developmental condition results mainly from environmental inputs (emphasizing environmental inputs over genetic factors). T h e s e t w o m o d e l s p e r s i s t in developmental theory. The e m p h a s i s on i n d i v i d u a l developmental differences s t r e s s i n g g e n e t i c f a c t o r s is e x p o u n d e d b y r e s e a r c h e r s s u c h as W i l l e r m a n ( 1 9 8 0 ) , a n d t h e e m p h a s i s on t h e d e v e l o p m e n t a l s t i m u l a t i o n q u a l i t i e s o f t h e e n v i r o n m e n t e x p o u n d e d b y r e s e a r c h e r s s u c h as H u n t ( 1 9 8 0 ) . With t h e r e s u r g e n c e o f t h e use o f L e w i n i a n t h e o r y b y e c o l o g i c a l r e s e a r c h e r s s u c h as B r o n f e n b r e n n e r (1979), and
Determinants of Implementation
3
Implementation/Process
Child Exposure/ Opportunity to Benefit I
5
Child outcome
Classroom climate Parental involvement Continuity Curriculum tangible resources Curriculum intangible resources Staff: knowledge and understanding Staff: role behavior Staff: attitudes
2 Parent Exposure/ Opportunity to Benefit
Fig.
i.
Child ~ k ~ e v a r i a b l e s i
Child Child Covariates
t I
Parent Outcomes
Parent pre-measure Parent background variables
8 Parent Covariates
T
Attitude toward child Attitude toward program Attitude toware self Behavior: home Behavior: program Behavior: community Behavior: self Parent child interactions
6
Activities/time on activities Program content/outcome Measure overlap Treatment group membership Parent interactions
4
Treatment group |Conceptual abilities membership ~Communication abilities Activities/time on | A f f e c t toward learning, school activities School and community success Child interactions Situation learner interactions Program content/outcome Affect toward self Affect toward others Psychomotor abilities iPhysical health
Political complexity and external agency support Drganizational capacity and history of innovation 3pportunistic vs problem solving adoption philosoph ~doption philosophy Explicitness Somplexity Resource commitment Staff characteristics ]ommunity characteristics ~enter characteristics Program characteristics
1
~o
284
E. H. Grotberg Et D. J. Deloria
t h e use o f t h e t r a n s a c t i o n a l model b y r e s e a r c h e r s s u c h as S a m e r o f f ( 1 9 8 0 ) , a m o r e c o m p r e h e n s i v e base d e v e l o p e d f o r s t u d y i n g c h i l d d e v e l o p m e n t a n d t h e i n t e r a c t i o n p a t t e r n s g e n e r a t e d b y a r a n g e o f t r a n s a c t i o n s i m p a c t i n g on t h e c h i l d . In t h e t r a n s a c t i o n a l m o d e l , d e v e l o p m e n t is a s s u m e d t o r e s u l t f r o m a d y n a m i c i n t e r p l a y between a changing child and a changing environment. T h e t r a n s a c t i o n a l model is s e n s i t i v e to e a r l y d e v e l o p m e n t a l c o n d i t i o n s in c h i l d r e n , can c o m p e n s a t e f o r debilitating conditions which might exist and thereby prevent later conditions or deficits. In o t h e r w o r d s , c a u s a l c o n n e c t i o n s o f t h e s i n g l e f a c t o r m o d e l s can be broken, as can t h e a d d i t i v e c a u s e s o f t h e i n t e r a c t i o n a l m o d e l , b y t r a n s a c t i o n a l kinds of interventions. The ecological perspective interested the Federal Interagency Panel on E a r l y C h i l d h o o d R e s e a r c h a n d D e v e l o p m e n t in t h e e a r l y 1970's. This Panel, representi n g 28 F e d e r a l a g e n c i e s , r e c o g n i z e d t h e v a l u e of t h e e c o l o g i c a l p e r s p e c t i v e as b e i n g m o r e c o n s i s t e n t b o t h w i t h an i n t e r a g e n c y , interdisciplinary a p p r o a c h in addressing child development issues, and with the variety of i n t e r v e n t i o n p r o g r a m s a v a i l a b l e f r o m t h e g o v e r n m e n t in h e a l t h , e d u c a t i o n , w e l f a r e , t r a i n i n g , housing, and food programs. T h e Panel d e v e l o p e d a s t r a t e g y to e x a m i n e c u r r e n t and planned research and e v a l u a t i o n f r o m an e c o l o g i c a l p e r s p e c t i v e ( H e r t z , 1976). It i d e n t i f i e d t h e c a t e g o r i e s w h i c h d e p i c t m a j o r s y s t e m s in t h e c h i l d ' s e c o l o g y : the child development processes, the family, t h e b r o a d e r social and p h y s i c a l e n v i r o n m e n t , and the services and intervention programs. U n d e r each s y s t e m w e r e c r i t i c a l s u b s y s t e m s l e n d i n g t h e m s e l v e s to r e s e a r c h a n d e v a l u a t i o n ( s e e F i g u r e 2 ) .
After identifying the systems and subcategories, the Panel d e v e l o p e d two matrices: one examining interactions or transactions within major systems (See Figure 3); the other describing transactions between or among major systems within the child's ecology (See Figure 4). T h e Panel m e m b e r s u s e d t h e s e m a t r i c e s to e x a m i n e t h e i r o w n r e s e a r c h a n d to guide their research planning. The Administration for Children, Youth and Families, for example, adopted this ecological orientation for child and family research and evaluation, beginning in 1974. T h e A C Y F l o g i c model d e s c r i b e d a b o v e a n d u s e d f o r Head S t a r t e v a l u a t i o n c o u l d e a s i l y i n c o r p o r a t e t h e e c o l o g i c a l perspective.
From C h i l d D e v e l o p m e n t O u t r n ~ e s tO IlanacjementJIPolicy D e c i s i o n s T h e t h i r d c h a n g e in e v a l u a t i o n is a s h i f t f r o m f o c u s i n g o n l y on t h e o u t c o m e s o f a p r o g r a m f o r c h i l d r e n a n d t h e i r f a m i l i e s to f o c u s i n g on a s s e s s i n g b o t h o u t c o m e s p l u s t h o s e a s p e c t s of a p r o g r a m i m p a c t i n g on c h i l d r e n a n d t h e i r f a m i l i e s w h i c h can be m a n i p u l a t e d o r c o n t r o l l e d b y m a n a g e m e n t a n d p o l i c y d e c i s i o n s . Both C o n g r e s s a n d t h e v a r i o u s a g e n c i e s became i n c r e a s i n g l y c o n c e r n e d in t h e 1970's with evaluation for purposes of management and policy decisions. Head S t a r t a n d day care are two programs which illustrate this shift. F r o m 1965 to 1971, a l t h o u g h t h e Economic O p p o r t u n i t y Act required evaluations of programs, f e w s p e c i f i c s w e r e g i v e n a n d no r e p o r t s t o C o n g r e s s w e r e m a n d a t e d . B y 1972, h o w e v e r , Congress s p e c i f i e d t h e k i n d s o f e v a l u a t i o n s to be d o n e . Evaluations should "describe and measure, with appropriate means a n d to t h e e x t e n t f e a s i b l e , t h e i m p a c t o f s u c h p r o g r a m s , t h e i r e f f e c t i v e n e s s in a c h i e v i n g s t a t e d g o a l s , t h e i r i m p a c t on r e l a t e d p r o g r a m s and t h e i r s t r u c t u r e and m e c h a n i s m s for delivery of services, and including, where appropriate, comparisons with a p p r o p r i a t e c o n t r o l g r o u p s c o m p o s e d o f p e r s o n s w h o h a v e n o t p a r t i c i p a t e d in s u c h
Evaluation of Preschool Programs
Child Development
Broader Social and Physical Environment
The Family
Service~/ Intervention Programs
Kind of service Day care Educational services Special education Infant education Preschool Cultural factol K-6th grade Child rearing Health/medical Socioeconomic practices services factors Preventive Intrafamily (immunications, Religious relationships/ screening and factors family functioning diagnosis) Parental roles/ Pre- and postMass media behavior natal care Sibling relationships Social crises Drug abuse Parent-child treatment and (e.g., noise, relationships prevention crime, unNutrition programs Family health employment) Cognitive development Mental health Perception/attention services Family's interface Social change Thought processes Welfare services with society Government Reading policy changes Foster care Language Adoptive services Parent training Changing sex Handicaps roles Child abuse prevention and Socioemotional Changing family treatment development Emergency services Perception of self patterns Changing Advocacy Interpersonal relationships employment Delinquency Attitudes practices treatment Desegregation Recreation Values/morals Motivation Law enforcement Personality People in Place: Emotional developHome child's ment environment School Social behavior Community Group care center members/ Hospital Overall child Medical clinic citizens growth and (adults other Con~aunity center development than parents) Place of employPeers ment Residential Psychologist/ social worker/ institution counselor Child careworker Educational personnel Police/law enforcement officials I Medical/health care personnel The Federal Interagency Panel on Early Childhood Research and Development identified these categories as significant for ecological studies of child development.
Physical development Body growth Sensory-motor development Handicaps Deaf Blind Epilepsy Cerebral Palsy Orthopedic Communicative disorder Mental Retardation Learning Disability Autism Disease Nutrition Epidemiology Pregnancy and childbirth
Geographical location Housing conditions Facilities and equipment for children
Physical environment of home Family structure Nuclear family Single parent family Extended family Working mother
Fig.
2.
285
i
i,i
t ,
l l l l
ii
i
I
~
i i i i
i1~
i1!
i l l
I
' CHILD ~ i z // ; ~
I
Il
I
/
I I I
I
I I ! I,I
/
II
I
I
I I!
I
Iii
II/
I1~
.~
~ /
I l l , I
!
FAMILY"
I
i
II
/ '
The Family
i i
i i
i q
~
, ts
i
f
•
r
i ~
t
i
/
,
,s
ENVIRONMENT
i
i
'
s
t
i
l
Broader Social and Physical Environment
i •
~
i
,
/
i
i
~
!
,
,
l
s
/
j
i I i , p f ~ l i j
i J i j /
i
, i
PNSC~
,
! !
Services/ ~tervention Programs
The Federal Interagency Panel on Early Childhood Research and Development used this ~atrix to exmnil~e witJ]in major systems interactions,
Services/ Intervention pr<~rams
Broader Social and Physical Environment
The Family
Child Develo~nent
i
!
/
Child Development
Fig. 3.
FAMILy
CHILD
ENVI I ~ i ' , N E ~
FAN/~Y
,t
ENVIt~N~N~
CHILD
Broader Social and Physical Environmant
4.
P NOGARAMS
ENVI NONMENT
FAMILY
It
PROGP,AY~
CHILD
Services/ Intervention Prograr~
The Federal Interatency Panel on F~mrly C h i l d h o o d Research and Development used this M a t r i x to examine i~ajor systems transactions.
Intervention Programs
Services/
Broader Social and Physical Environment
The Family
Child Development
The Family
Systems Child Developmant
Fig,
m
~0 CO O~
Evaluation of Preschool Programs
287
p r o g r a m s " ( P u b l i c Law ( P . L . ) 92-424 Sec. 901 ( a ) ) . T h i s law also r e q u i r e d t h e d e v e l o p m e n t o f s t a n d a r d s f o r e v a l u a t i o n p w h i c h became k n o w n as Head S t a r t Performance Standards. B y 1975 C o n g r e s s r e q u i r e d t h a t " e v a l u a t i o n s shall be c o n d u c t e d b y p e r s o n s n o t directly i n v o l v e d in t h e a d m i n i s t r a t i o n o f t h e p r o g r a m o r p r o j e c t e v a l u a t i o n s " (P.L. 93-644 Sec. 5 2 5 ( a ) ) . A n d b y 1981 l a n g u a g e a p p e a r e d r e q u i r i n g the e s t a b l i s h m e n t of " w o r k i n g r e l a t i o n s h i p s w i t h t h e f a c u l t i e s o f c o l l e g e s o r u n i v e r s i t i e s l o c a t e d in t h e area in w h i c h a n y such e v a l u a t i o n is b e i n g c o n d u c t e d . " ( D r a f t 1982 Head S t a r t l e g i s l a t i o n . ) T h i s g r o w i n g i n v o l v e m e n t of C o n g r e s s in s p e c i f y i n g t h e e v a l u a t i o n s to be c o n ducted, guiding methodology and finally identifying the kinds of institutions from w h i c h e v a l u a t i o n s a r e to be d r a w n , has h a d a d r a m a t i c i m p a c t on r e s e a r c h e r s and evaluators. E v a l u a t i o n now is e x p e c t e d t o p r o v i d e i n f o r m a t i o n u s e f u l f o r m a n a g e m e n t and p o l i c y d e c i s i o n s . T h u s , w h e n C o n g r e s s was d e t e r m i n i n g , d u r i n g 1977 a n d 1978, w h e t h e r o r n o t i t s h o u l d e x p a n d Head S t a r t , r e s e a r c h e r r e s p o n d e d to this Congressional interest. A consortium of researchers, funded by ACYF, c o m p r i s e d some 14 e a r l y i n t e r v e n t i o n p r o g r a m s t h a t i n c l u d e d o r w e r e s i m i l a r t o Head S t a r t p r o g r a m s . T h e s e r e s e a r c h e r s g a t h e r e d and p o o l e d f o l l o w - u p d a t a f r o m i n t e r v e n t i o n p r o g r a m s as f a r b a c k as t h e e a r l y 1960s. The pooled data revealed r e m a r k a b l y c o n s i s t e n t f i n d i n g s a n d made a s t r o n g case f o r t h e b e n e f i t s o f p r e school p r o g r a m s s u c h as Head S t a r t . T h e Head S t a r t p r o g r a m was e x p a n d e d and t h e c o n s o r t i u m d a t a w e r e u s e d as one j u s t i f i c a t i o n f o r t h a t p o l i c y d e c i s i o n . A c l e a r - c u t m a n a g e m e n t and p o l i c y e v a l u a t i o n s t u d y was t h e N a t i o n a l Day C a r e S t u d y , b e g u n in 1974 and e n d e d in 1979. T h e i n i t i a l s t u d y was an e x a m i n a t i o n o f t h e q u a l i t y a n d c o s t o f c e n t e r b a s e d d a y c a r e f o r p r e s c h o o l c h i l d r e n , a n d , to a lesser extent, for infants and toddlers. B y 1975, a f t e r C o n g r e s s i n c o r p o r a t e d a m o d i f i e d v e r s i o n o f t h e 1968 F e d e r a l I n t e r a g e n c y Day C a r e R e g u l a t i o n s ( F I D C R ) i n t o T i t l e X X of t h e Social S e c u r i t y A c t , C o n g r e s s m a n d a t e d an e v a l u a t i o n of t h e a p p r o p r i a t e n e s s o f t h e FIDCR i m p o s e d b y T i t l e X X . T h e N a t i o n a l Day C a r e S t u d y was m o d i f i e d , a c c o r d i n g l y , to a d d r e s s t h e e v a l u a t i o n r e q u i r e m e n t s o f C o n g r e s s . T h e m a j o r o b j e c t i v e s o f t h e s t u d y t h e n became to d e t e r m i n e ( 1 ) t h e i m p a c t on t h e quality of care of different caregivers/child r a t i o s , g r o u p s i z e s , and q u a l i f i c a t i o n o f c l a s s r o o m s t a f f , and ( 2 ) t h e i m p a c t of t h e s e f a c t o r s on t h e c o s t of c a r e . These objectives l e n t t h e m s e l v e s to m a n i p u l a t i o n b y m a n a g e m e n t and p o l i c y makers. The study, conducted by Abt Associates, I n c . and SRI I n t e r n a t i o n a l ( 1 9 7 9 ) , combined a quasi-experimental methodology with research evaluation that would s a t i s f y t h e r e q u i r e m e n t s of C o n g r e s s and also meet t h e need f o r t h e most r e l i a b l e data possible. The findings of the research evaluation substantiated the general f i n d i n g s of t h e q u a s i - e x p e r i m e n t a l s t u d y , a f a c t t h a t p e r m i t t e d f i r m e r s t a t e m e n t s of findings. The findings indicated that new, modified federal day care standards were needed consistent with the evaluation results. The new standards were drafted, subj e c t e d to a f u l l y e a r o f p u b l i c c o m m e n t , and in M a r c h 1980 a p p r o v e d b y t h e S e c r e t a r y of H e a l t h and Human S e r v i c e s . I t was i n t e n d e d t h a t t h e s e s t a n d a r d s become a tool f o r s t a t e and local m a n a g e m e n t and p o l i c y d e c i s i o n s c o n c e r n i n g t h e q u a l i t y and cost o f d a y c a r e p r o g r a m s . However, with a growing trend toward d e c e n t r a l i z a t i o n and d e r e g u l a t i o n , C o n g r e s s p l a c e d a m o r a t o r i u m on t h e s t a n d a r d s in 1981. No a c t i o n s h a v e been t a k e n to d a t e .
288
E. H. Grotberg 8 D. J. Deloria
Sure.mat7 ~
Concluding Start
T h e c h a n g e in e v a l u a t i o n o f p r e s c h o o l p r o g r a m s r e s u l t i n g f r o m t h e s y n t h e s i s o f service delivery f o c u s a n d c h i l d d e v e l o p m e n t o u t c o m e s f o c u s has c o n t r i b u t e d significantly to more effective evaluations. D e l i v e r y s y s t e m s m a y be e f f i c i e n t a n d e a s y t o m a n a g e b u t i f t h e y a r e u n r e l a t e d to w h a t h a p p e n s to t h e d e v e l o p m e n t o f c h i l d r e n in t h e s y s t e m , t h e y a r e l i m i t e d . S i m i l a r l y , if t h e r e is no u n d e r s t a n d i n g of which attributes of a delivery system or which program elements account for p o s i t i v e c h i l d o u t c o m e s , t h e r e is no c l e a r w a y t o i m p r o v e t h e p r o g r a m . T h e c h a n g e f r o m f o c u s on s i n g l e v a r i a b l e s t o f o c u s on t h e i n t e r a c t i o n a n d t r a n s action of contextual variables helps evaluations identify what interventions are effective for which children in w h a t k i n d s o f f a m i l y , p r o g r a m a n d c o m m u n i t y settings. T h i s a p p r o a c h o p e n s u p a w i d e r a n g e of e n t r y p o i n t s f o r i n t e r v e n t i o n a n d can e n h a n c e t h o s e i n t e r v e n t i o n s w h i c h seem t o a c c o u n t f o r g r e a t e s t c h i l d development outcomes. T h e c h a n g e f r o m a s i n g u l a r f o c u s on c h i l d d e v e l o p m e n t o u t c o m e s t o effective program elements which are subject to management and policy h e l p e d in g u i d i n g d e c i s i o n m a k e r s . T h e f i n d i n g s of s u c h e v a l u a t i o n s amenable to making policy decisions which improve the quality of p r o g r a m a n d w h i c h can p r o v i d e m a n a g e r s w i t h t o o l s f o r m a i n t a i n i n g quality.
a f o c u s on c o n t r o l has are directly a preschool the desired
These changing s t r a t e g i e s r e f l e c t n o t o n l y t h e g e n e r a l g r o w t h in t h e v a r i o u s disciplines contributing t o p r o g r a m s a n d e v a l u a t i o n s b u t also t h e d e t e r m i n a t i o n o f c o n g r e s s i o n a l a n d o t h e r p o l i c y m a k e r s t o h a v e s o u n d i n f o r m a t i o n u p o n w h i c h to make decisions.
REFERENCES Abt
A s s o c i a t e s . Final report of the National Day Care Study: Center ( 5 v o l s . ) . C a m b r i d g e , M A , 1979.
Bronfenbrenner, U. The ecology of human H a r v a r d U n i v e r s i t y P r e s s , 1979. Fanshel, D., and Shinn, E.B. U n i v e r s i t y P r e s s , 1978. Grotberg, DC:
Children
development.
in foster
Children
at the
Cambridge,
care. New Y o r k :
MA:
Columbia
E.H. Review of research, 1966-1969 of Project Read Start. W a s h i n g t o n , O C D , 1969 ( E R I C D o c u m e n t R e p r o d u c t i o n S e r v i c e No. 2 8 3 0 8 ) .
T . W . The ecology of child development: A working paper f o r of research, development and evaluation. Washington, DC:
Hertz,
planners Contract
H EW-105-76-1120. Hunt,
J.McV. Implications of plasticity and hierarchical achievements for the a s s e s s m e n t o f d e v e l o p m e n t a n d r i s k of m e n t a l r e t a r d a t i o n . In D . B . S a w i n , R.C. Hawkins II, L.O. Walker and J.H. Penticuff (Eds.), Exceptional infant (Vol. 4). New Y o r k : Brunner/Mazel, 1980.
NTS
Research Corporation. Project Report 6: Description and analysis of program data: Basic Educational Skills Initiative. Durham, NC: NTS, 1980.
Evaluation of Preschool Programs
289
Sameroff, A.J. Issues in e a r l y r e p r o d u c t i o n and c a r e t a k i n g current status. In D . B . Sawin e t al. ( E d s . ) , op. cit.
risk:
Systems of social services for children and their families:
(DHEW Superintendent of
Publication Documents, Willerman,
L.
No. U.S.
An overview
OHDS 7 8 - 3 0 1 4 5 ) . Washington, DC: G o v e r n m e n t P r i n t i n g O f f i c e , 1978.
Genetics
of
psychosocial
risk.
In
D.B.
R e v i e w and
Sawin,
et
al.
(Eds.),
op. tit.
THE A U T H O R S E D I T H G R O T B E R G is Special A s s i s t a n t to t h e A s s o c i a t e C o m m i s s i o n e r f o r D e v e l o p mental S e r v i c e s in t h e A d m i n i s t r a t i o n f o r C h i l d r e n , Y o u t h a n d Families o f t h e U . S . D e p a r t m e n t o f H e a l t h a n d Human S e r v i c e s . DENNIS D E L O R I A is P r o g r a m A n a l y s t in t h e A d m i n i s t r a t i o n f o r C h i l d r e n , and Families o f t h e U . S . D e p a r t m e n t of H e a l t h and Human S e r v i c e s .
Youth
0191-491X/83/030281-09504.50/0
Printed in GreatBritain. All rights reserved.
CHANGING
Copyright© 1983PergamonPressLtd.
S T R A T E G I E S IN E V A L U A T I O N S PRESCHOOL PROGRAMS
OF
Edith H. Grotberg and Dennis J. Deloria Administration for Children, Youth and Families, Washington D.C., USA
A number of strategies relating to the evaluation of preschool programs have been changing. O n e c h a n g e has b e e n f r o m e v a l u a t i n g s e r v i c e d e l i v e r y s y s t e m s a n d child development outcomes independent of one another. A second area of change has b e e n f r o m e v a l u a t i n g t h e i m p a c t o f a p a r t i c u l a r i n t e r v e n t i o n on an i n d i v i d u a l child to evaluating t h e i m p a c t o f an i n t e r v e n t i o n w h i c h is p a r t o f a d y n a m i c ( e c o l o g i c a l ) s e t t i n g , on a c h i l d w h o is w i t h i n t h a t d y n a m i c ( e c o l o g i c a l ) s e t t i n g . A t h i r d c h a n g e has b e e n f r o m e v a l u a t i n g s i m p l y in t e r m s o f c h i l d d e v e l o p m e n t o u t comes t o e v a l u a t i n g b o t h o u t c o m e s as w e l l as t h o s e a s p e c t s o f a c h i l d d e v e l o p m e n t program w h i c h can be m a n i p u l a t e d o r c o n t r o l l e d by policy and management decisions. These are not mutually exclusive changes, and, indeed, a good deal o f o v e r l a p w i l l be e v i d e n t . H o w e v e r , t h e r e a r e s u f f i c i e n t d i f f e r e n c e s in a p p r o a c h to warrant separate treatment. Each o f t h e s e a r e a s o f c h a n g e w i l l be d e s c r i b e d largely within the context of programs and concerns of the Administration for Children, Youth and Families (ACYF).
Service Delivery Systems and Child Develolueent Outcomes Foster care systems were historically evaluated by determining the number of c h i l d r e n p l a c e d in f o s t e r c a r e , t h e n u m b e r o f f o s t e r p a r e n t s a v a i l a b l e , a n d t h e n u m b e r o f cases r e s o l v e d e i t h e r in t e r m s o f r e t u r n t o home o r p e r m a n e n t p l a c e ment. A l m o s t n e v e r was t h e i m p a c t o f t h e s e r v i c e d e l i v e r y s y s t e m a s s e s s e d in terms of child development outcomes. I n s t e a d , t h e r e was an i n c r e a s i n g l y s o p h i s ticated method for describing the components of the delivery system. Studies of t h e i m p a c t o f t h e f o s t e r c a r e s y s t e m on c h i l d d e v e l o p m e n t d i d n o t o c c u r t o a n y s i g n i f i c a n t e x t e n t u n t i l t h e w o r k o f D a v i d F a n s h e l a n d his c o l l e a g u e s ( 1 9 7 8 ) . The l o n g i t u d i n a l s t u d y t h e y i n i t i a t e d in 1968 a n d b e g a n r e p o r t i n g on in t h e m i d - 1 9 7 0 s contained sufficiently vigorous methodology to evaluate the impact of the foster c a r e s y s t e m on c h i l d d e v e l o p m e n t . This landmark work changed subsequent evaluation strategies for foster care programs to include both the system and child development outcomes. T h e r e s u l t i n g s y n t h e s i s is b e s t d e s c r i b e d in S y s t e m s o f Social S e r v i c e s f o r C h i l d r e n a n d T h e i r F a m i l i e s : An Overview (1978). T h e i m p a c t of t h e Head S t a r t p r o g r a m was h i s t o r i c a l l y e v a l u a t e d b y d e t e r m i n i n g child development outcomes, particularly those in t h e c o g n i t i v e and socialemotional realms. In a r e v i e w o f t h e H e a d S t a r t e v a l u a t i o n literature in 1 9 6 9 , Orotberg found that child development outcomes were not associated with specific c o m p o n e n t s of t h e Head S t a r t p r o g r a m b u t r a t h e r with t h e total p r o g r a m . Head Start was viewed as a unitary concept with a common impact on children. The 281
282
E. H. Grotberg ~ D. J. Deloria
p r o g r a m e l e m e n t s o f Head S t a r t w e r e n o t s u f f i c i e n t l y described for evaluation u n t i l t h e p u b l i c a t i o n o f t h e Head S t a r t P e r f o r m a n c e S t a n d a r d s in 1975. From that time to the present, Head S t a r t e v a l u a t i o n s h a v e r e l a t e d c h i l d d e v e l o p m e n t o u t comes w i t h s e r v i c e d e l i v e r y s y s t e m s ( p r o g r a m e l e m e n t s ) . A model to e v a l u a t e Head S t a r t is p r e s e n t e d h e r e t o i l l u s t r a t e t h e s y n t h e s i s o f program elements and child development outcomes (NTS, 1980). The model, frequently c a l l e d a l o g i c m o d e l , assumes a l o g i c a l s t r u c t u r e t o a p r o g r a m w h i c h included intended resource inputs, activities, outcomes or impacts. T h e model also assumes that there are causal relationships among resource inputs, program activites, and outcomes or impacts. The components of the conceptual framework o f t h e l o g i c model a r e d e s c r i b e d h e r e w i t h a s k e t c h o f t h e model p r e s e n t e d in F i g u r e 1. T h e model c o m p o n e n t s a r e as f o l l o w s : 1. Determinants of Implementation--the v a r i a b l e s a s s u m e d t o a f f e c t t h e success o f Head S t a r t p r o g r a m i m p l e m e n t a t i o n s a n d t h e f i d e l i t y of p r o g r a m e l e m e n t s t o t h e Performance Standards. 2. Implementation/Process--the variables that determine o f t h e c h a n g e s g e n e r a t e d b y Head S t a r t p r o g r a m e l e m e n t s . 3. Child Exposure/Opportunity child, distinguishing between c h i l d ' s use o f t h e m .
to Benefit--the the availability
the
nature
and
classroom experience of classroom resources
extent
of and
the the
4. Parent Exposure/Opportunity to Benefit--an analytic approach for correlating e x p o s u r e a n d o p p o r t u n i t y w i t h p a r e n t o u t c o m e s in a s y s t e m a t i c w a y . 5. Child Outcomes - the sions of child development relationships. 6. Parent Outcomes--the home, parent and program,
p h y s i c a l , c o g n i t i v e and e m o t i o n a l - m o t i v a t i o n a l d i m e n o r g a n i z e d i n t o a l o g i c a l s y s t e m of e x p o s u r e - o u t c o m e
relationships between and parent and self.
7. a n d 8. C o v a r i a t e s f o r C h i l d r e n relevant background variables.
parent
and Parents--pretest
and
child,
measures
parent
and
and theoretically
From Single Variables to Ecx)lo(jy T h e s e c o n d c h a n g e in e v a l u a t i o n has b e e n a slow s h i f t f r o m e v a l u a t i n g t h e i m p a c t o f a p a r t i c u l a r i n t e r v e n t i o n on a c h i l d t o e v a l u a t i n g t h e i m p a c t o f an i n t e r v e n t i o n w h i c h is p a r t o f a d y n a m i c ( e c o l o g i c a l ) s e t t i n g , on a c h i l d w h o is w i t h i n t h a t dynamic (ecological) setting. There are a number of reasons for the slowness of the shift. One is t h e i m p a c t on e v a l u a t i o n o f e x p e r i m e n t a l r e s e a r c h w h i c h d r e w h e a v i l y on s i n g l e f a c t o r v a r i a b l e s ( b o t h t h e d e p e n d e n t and t h e i n d e p e n d e n t v a r i ables) and which assumed linear causality between a given developmental condition a n d an o u t c o m e . A n o t h e r r e a s o n is an i n t e r v e n t i o n model w h i c h p o s i t s t h a t a given developmental condition results mainly from environmental inputs (emphasizing environmental inputs over genetic factors). T h e s e t w o m o d e l s p e r s i s t in developmental theory. The e m p h a s i s on i n d i v i d u a l developmental differences s t r e s s i n g g e n e t i c f a c t o r s is e x p o u n d e d b y r e s e a r c h e r s s u c h as W i l l e r m a n ( 1 9 8 0 ) , a n d t h e e m p h a s i s on t h e d e v e l o p m e n t a l s t i m u l a t i o n q u a l i t i e s o f t h e e n v i r o n m e n t e x p o u n d e d b y r e s e a r c h e r s s u c h as H u n t ( 1 9 8 0 ) . With t h e r e s u r g e n c e o f t h e use o f L e w i n i a n t h e o r y b y e c o l o g i c a l r e s e a r c h e r s s u c h as B r o n f e n b r e n n e r (1979), and
Determinants of Implementation
3
Implementation/Process
Child Exposure/ Opportunity to Benefit I
5
Child outcome
Classroom climate Parental involvement Continuity Curriculum tangible resources Curriculum intangible resources Staff: knowledge and understanding Staff: role behavior Staff: attitudes
2 Parent Exposure/ Opportunity to Benefit
Fig.
i.
Child ~ k ~ e v a r i a b l e s i
Child Child Covariates
t I
Parent Outcomes
Parent pre-measure Parent background variables
8 Parent Covariates
T
Attitude toward child Attitude toward program Attitude toware self Behavior: home Behavior: program Behavior: community Behavior: self Parent child interactions
6
Activities/time on activities Program content/outcome Measure overlap Treatment group membership Parent interactions
4
Treatment group |Conceptual abilities membership ~Communication abilities Activities/time on | A f f e c t toward learning, school activities School and community success Child interactions Situation learner interactions Program content/outcome Affect toward self Affect toward others Psychomotor abilities iPhysical health
Political complexity and external agency support Drganizational capacity and history of innovation 3pportunistic vs problem solving adoption philosoph ~doption philosophy Explicitness Somplexity Resource commitment Staff characteristics ]ommunity characteristics ~enter characteristics Program characteristics
1
~o
284
E. H. Grotberg Et D. J. Deloria
t h e use o f t h e t r a n s a c t i o n a l model b y r e s e a r c h e r s s u c h as S a m e r o f f ( 1 9 8 0 ) , a m o r e c o m p r e h e n s i v e base d e v e l o p e d f o r s t u d y i n g c h i l d d e v e l o p m e n t a n d t h e i n t e r a c t i o n p a t t e r n s g e n e r a t e d b y a r a n g e o f t r a n s a c t i o n s i m p a c t i n g on t h e c h i l d . In t h e t r a n s a c t i o n a l m o d e l , d e v e l o p m e n t is a s s u m e d t o r e s u l t f r o m a d y n a m i c i n t e r p l a y between a changing child and a changing environment. T h e t r a n s a c t i o n a l model is s e n s i t i v e to e a r l y d e v e l o p m e n t a l c o n d i t i o n s in c h i l d r e n , can c o m p e n s a t e f o r debilitating conditions which might exist and thereby prevent later conditions or deficits. In o t h e r w o r d s , c a u s a l c o n n e c t i o n s o f t h e s i n g l e f a c t o r m o d e l s can be broken, as can t h e a d d i t i v e c a u s e s o f t h e i n t e r a c t i o n a l m o d e l , b y t r a n s a c t i o n a l kinds of interventions. The ecological perspective interested the Federal Interagency Panel on E a r l y C h i l d h o o d R e s e a r c h a n d D e v e l o p m e n t in t h e e a r l y 1970's. This Panel, representi n g 28 F e d e r a l a g e n c i e s , r e c o g n i z e d t h e v a l u e of t h e e c o l o g i c a l p e r s p e c t i v e as b e i n g m o r e c o n s i s t e n t b o t h w i t h an i n t e r a g e n c y , interdisciplinary a p p r o a c h in addressing child development issues, and with the variety of i n t e r v e n t i o n p r o g r a m s a v a i l a b l e f r o m t h e g o v e r n m e n t in h e a l t h , e d u c a t i o n , w e l f a r e , t r a i n i n g , housing, and food programs. T h e Panel d e v e l o p e d a s t r a t e g y to e x a m i n e c u r r e n t and planned research and e v a l u a t i o n f r o m an e c o l o g i c a l p e r s p e c t i v e ( H e r t z , 1976). It i d e n t i f i e d t h e c a t e g o r i e s w h i c h d e p i c t m a j o r s y s t e m s in t h e c h i l d ' s e c o l o g y : the child development processes, the family, t h e b r o a d e r social and p h y s i c a l e n v i r o n m e n t , and the services and intervention programs. U n d e r each s y s t e m w e r e c r i t i c a l s u b s y s t e m s l e n d i n g t h e m s e l v e s to r e s e a r c h a n d e v a l u a t i o n ( s e e F i g u r e 2 ) .
After identifying the systems and subcategories, the Panel d e v e l o p e d two matrices: one examining interactions or transactions within major systems (See Figure 3); the other describing transactions between or among major systems within the child's ecology (See Figure 4). T h e Panel m e m b e r s u s e d t h e s e m a t r i c e s to e x a m i n e t h e i r o w n r e s e a r c h a n d to guide their research planning. The Administration for Children, Youth and Families, for example, adopted this ecological orientation for child and family research and evaluation, beginning in 1974. T h e A C Y F l o g i c model d e s c r i b e d a b o v e a n d u s e d f o r Head S t a r t e v a l u a t i o n c o u l d e a s i l y i n c o r p o r a t e t h e e c o l o g i c a l perspective.
From C h i l d D e v e l o p m e n t O u t r n ~ e s tO IlanacjementJIPolicy D e c i s i o n s T h e t h i r d c h a n g e in e v a l u a t i o n is a s h i f t f r o m f o c u s i n g o n l y on t h e o u t c o m e s o f a p r o g r a m f o r c h i l d r e n a n d t h e i r f a m i l i e s to f o c u s i n g on a s s e s s i n g b o t h o u t c o m e s p l u s t h o s e a s p e c t s of a p r o g r a m i m p a c t i n g on c h i l d r e n a n d t h e i r f a m i l i e s w h i c h can be m a n i p u l a t e d o r c o n t r o l l e d b y m a n a g e m e n t a n d p o l i c y d e c i s i o n s . Both C o n g r e s s a n d t h e v a r i o u s a g e n c i e s became i n c r e a s i n g l y c o n c e r n e d in t h e 1970's with evaluation for purposes of management and policy decisions. Head S t a r t a n d day care are two programs which illustrate this shift. F r o m 1965 to 1971, a l t h o u g h t h e Economic O p p o r t u n i t y Act required evaluations of programs, f e w s p e c i f i c s w e r e g i v e n a n d no r e p o r t s t o C o n g r e s s w e r e m a n d a t e d . B y 1972, h o w e v e r , Congress s p e c i f i e d t h e k i n d s o f e v a l u a t i o n s to be d o n e . Evaluations should "describe and measure, with appropriate means a n d to t h e e x t e n t f e a s i b l e , t h e i m p a c t o f s u c h p r o g r a m s , t h e i r e f f e c t i v e n e s s in a c h i e v i n g s t a t e d g o a l s , t h e i r i m p a c t on r e l a t e d p r o g r a m s and t h e i r s t r u c t u r e and m e c h a n i s m s for delivery of services, and including, where appropriate, comparisons with a p p r o p r i a t e c o n t r o l g r o u p s c o m p o s e d o f p e r s o n s w h o h a v e n o t p a r t i c i p a t e d in s u c h
Evaluation of Preschool Programs
Child Development
Broader Social and Physical Environment
The Family
Service~/ Intervention Programs
Kind of service Day care Educational services Special education Infant education Preschool Cultural factol K-6th grade Child rearing Health/medical Socioeconomic practices services factors Preventive Intrafamily (immunications, Religious relationships/ screening and factors family functioning diagnosis) Parental roles/ Pre- and postMass media behavior natal care Sibling relationships Social crises Drug abuse Parent-child treatment and (e.g., noise, relationships prevention crime, unNutrition programs Family health employment) Cognitive development Mental health Perception/attention services Family's interface Social change Thought processes Welfare services with society Government Reading policy changes Foster care Language Adoptive services Parent training Changing sex Handicaps roles Child abuse prevention and Socioemotional Changing family treatment development Emergency services Perception of self patterns Changing Advocacy Interpersonal relationships employment Delinquency Attitudes practices treatment Desegregation Recreation Values/morals Motivation Law enforcement Personality People in Place: Emotional developHome child's ment environment School Social behavior Community Group care center members/ Hospital Overall child Medical clinic citizens growth and (adults other Con~aunity center development than parents) Place of employPeers ment Residential Psychologist/ social worker/ institution counselor Child careworker Educational personnel Police/law enforcement officials I Medical/health care personnel The Federal Interagency Panel on Early Childhood Research and Development identified these categories as significant for ecological studies of child development.
Physical development Body growth Sensory-motor development Handicaps Deaf Blind Epilepsy Cerebral Palsy Orthopedic Communicative disorder Mental Retardation Learning Disability Autism Disease Nutrition Epidemiology Pregnancy and childbirth
Geographical location Housing conditions Facilities and equipment for children
Physical environment of home Family structure Nuclear family Single parent family Extended family Working mother
Fig.
2.
285
i
i,i
t ,
l l l l
ii
i
I
~
i i i i
i1~
i1!
i l l
I
' CHILD ~ i z // ; ~
I
Il
I
/
I I I
I
I I ! I,I
/
II
I
I
I I!
I
Iii
II/
I1~
.~
~ /
I l l , I
!
FAMILY"
I
i
II
/ '
The Family
i i
i i
i q
~
, ts
i
f
•
r
i ~
t
i
/
,
,s
ENVIRONMENT
i
i
'
s
t
i
l
Broader Social and Physical Environment
i •
~
i
,
/
i
i
~
!
,
,
l
s
/
j
i I i , p f ~ l i j
i J i j /
i
, i
PNSC~
,
! !
Services/ ~tervention Programs
The Federal Interagency Panel on Early Childhood Research and Development used this ~atrix to exmnil~e witJ]in major systems interactions,
Services/ Intervention pr<~rams
Broader Social and Physical Environment
The Family
Child Develo~nent
i
!
/
Child Development
Fig. 3.
FAMILy
CHILD
ENVI I ~ i ' , N E ~
FAN/~Y
,t
ENVIt~N~N~
CHILD
Broader Social and Physical Environmant
4.
P NOGARAMS
ENVI NONMENT
FAMILY
It
PROGP,AY~
CHILD
Services/ Intervention Prograr~
The Federal Interatency Panel on F~mrly C h i l d h o o d Research and Development used this M a t r i x to examine i~ajor systems transactions.
Intervention Programs
Services/
Broader Social and Physical Environment
The Family
Child Development
The Family
Systems Child Developmant
Fig,
m
~0 CO O~
Evaluation of Preschool Programs
287
p r o g r a m s " ( P u b l i c Law ( P . L . ) 92-424 Sec. 901 ( a ) ) . T h i s law also r e q u i r e d t h e d e v e l o p m e n t o f s t a n d a r d s f o r e v a l u a t i o n p w h i c h became k n o w n as Head S t a r t Performance Standards. B y 1975 C o n g r e s s r e q u i r e d t h a t " e v a l u a t i o n s shall be c o n d u c t e d b y p e r s o n s n o t directly i n v o l v e d in t h e a d m i n i s t r a t i o n o f t h e p r o g r a m o r p r o j e c t e v a l u a t i o n s " (P.L. 93-644 Sec. 5 2 5 ( a ) ) . A n d b y 1981 l a n g u a g e a p p e a r e d r e q u i r i n g the e s t a b l i s h m e n t of " w o r k i n g r e l a t i o n s h i p s w i t h t h e f a c u l t i e s o f c o l l e g e s o r u n i v e r s i t i e s l o c a t e d in t h e area in w h i c h a n y such e v a l u a t i o n is b e i n g c o n d u c t e d . " ( D r a f t 1982 Head S t a r t l e g i s l a t i o n . ) T h i s g r o w i n g i n v o l v e m e n t of C o n g r e s s in s p e c i f y i n g t h e e v a l u a t i o n s to be c o n ducted, guiding methodology and finally identifying the kinds of institutions from w h i c h e v a l u a t i o n s a r e to be d r a w n , has h a d a d r a m a t i c i m p a c t on r e s e a r c h e r s and evaluators. E v a l u a t i o n now is e x p e c t e d t o p r o v i d e i n f o r m a t i o n u s e f u l f o r m a n a g e m e n t and p o l i c y d e c i s i o n s . T h u s , w h e n C o n g r e s s was d e t e r m i n i n g , d u r i n g 1977 a n d 1978, w h e t h e r o r n o t i t s h o u l d e x p a n d Head S t a r t , r e s e a r c h e r r e s p o n d e d to this Congressional interest. A consortium of researchers, funded by ACYF, c o m p r i s e d some 14 e a r l y i n t e r v e n t i o n p r o g r a m s t h a t i n c l u d e d o r w e r e s i m i l a r t o Head S t a r t p r o g r a m s . T h e s e r e s e a r c h e r s g a t h e r e d and p o o l e d f o l l o w - u p d a t a f r o m i n t e r v e n t i o n p r o g r a m s as f a r b a c k as t h e e a r l y 1960s. The pooled data revealed r e m a r k a b l y c o n s i s t e n t f i n d i n g s a n d made a s t r o n g case f o r t h e b e n e f i t s o f p r e school p r o g r a m s s u c h as Head S t a r t . T h e Head S t a r t p r o g r a m was e x p a n d e d and t h e c o n s o r t i u m d a t a w e r e u s e d as one j u s t i f i c a t i o n f o r t h a t p o l i c y d e c i s i o n . A c l e a r - c u t m a n a g e m e n t and p o l i c y e v a l u a t i o n s t u d y was t h e N a t i o n a l Day C a r e S t u d y , b e g u n in 1974 and e n d e d in 1979. T h e i n i t i a l s t u d y was an e x a m i n a t i o n o f t h e q u a l i t y a n d c o s t o f c e n t e r b a s e d d a y c a r e f o r p r e s c h o o l c h i l d r e n , a n d , to a lesser extent, for infants and toddlers. B y 1975, a f t e r C o n g r e s s i n c o r p o r a t e d a m o d i f i e d v e r s i o n o f t h e 1968 F e d e r a l I n t e r a g e n c y Day C a r e R e g u l a t i o n s ( F I D C R ) i n t o T i t l e X X of t h e Social S e c u r i t y A c t , C o n g r e s s m a n d a t e d an e v a l u a t i o n of t h e a p p r o p r i a t e n e s s o f t h e FIDCR i m p o s e d b y T i t l e X X . T h e N a t i o n a l Day C a r e S t u d y was m o d i f i e d , a c c o r d i n g l y , to a d d r e s s t h e e v a l u a t i o n r e q u i r e m e n t s o f C o n g r e s s . T h e m a j o r o b j e c t i v e s o f t h e s t u d y t h e n became to d e t e r m i n e ( 1 ) t h e i m p a c t on t h e quality of care of different caregivers/child r a t i o s , g r o u p s i z e s , and q u a l i f i c a t i o n o f c l a s s r o o m s t a f f , and ( 2 ) t h e i m p a c t of t h e s e f a c t o r s on t h e c o s t of c a r e . These objectives l e n t t h e m s e l v e s to m a n i p u l a t i o n b y m a n a g e m e n t and p o l i c y makers. The study, conducted by Abt Associates, I n c . and SRI I n t e r n a t i o n a l ( 1 9 7 9 ) , combined a quasi-experimental methodology with research evaluation that would s a t i s f y t h e r e q u i r e m e n t s of C o n g r e s s and also meet t h e need f o r t h e most r e l i a b l e data possible. The findings of the research evaluation substantiated the general f i n d i n g s of t h e q u a s i - e x p e r i m e n t a l s t u d y , a f a c t t h a t p e r m i t t e d f i r m e r s t a t e m e n t s of findings. The findings indicated that new, modified federal day care standards were needed consistent with the evaluation results. The new standards were drafted, subj e c t e d to a f u l l y e a r o f p u b l i c c o m m e n t , and in M a r c h 1980 a p p r o v e d b y t h e S e c r e t a r y of H e a l t h and Human S e r v i c e s . I t was i n t e n d e d t h a t t h e s e s t a n d a r d s become a tool f o r s t a t e and local m a n a g e m e n t and p o l i c y d e c i s i o n s c o n c e r n i n g t h e q u a l i t y and cost o f d a y c a r e p r o g r a m s . However, with a growing trend toward d e c e n t r a l i z a t i o n and d e r e g u l a t i o n , C o n g r e s s p l a c e d a m o r a t o r i u m on t h e s t a n d a r d s in 1981. No a c t i o n s h a v e been t a k e n to d a t e .
288
E. H. Grotberg 8 D. J. Deloria
Sure.mat7 ~
Concluding Start
T h e c h a n g e in e v a l u a t i o n o f p r e s c h o o l p r o g r a m s r e s u l t i n g f r o m t h e s y n t h e s i s o f service delivery f o c u s a n d c h i l d d e v e l o p m e n t o u t c o m e s f o c u s has c o n t r i b u t e d significantly to more effective evaluations. D e l i v e r y s y s t e m s m a y be e f f i c i e n t a n d e a s y t o m a n a g e b u t i f t h e y a r e u n r e l a t e d to w h a t h a p p e n s to t h e d e v e l o p m e n t o f c h i l d r e n in t h e s y s t e m , t h e y a r e l i m i t e d . S i m i l a r l y , if t h e r e is no u n d e r s t a n d i n g of which attributes of a delivery system or which program elements account for p o s i t i v e c h i l d o u t c o m e s , t h e r e is no c l e a r w a y t o i m p r o v e t h e p r o g r a m . T h e c h a n g e f r o m f o c u s on s i n g l e v a r i a b l e s t o f o c u s on t h e i n t e r a c t i o n a n d t r a n s action of contextual variables helps evaluations identify what interventions are effective for which children in w h a t k i n d s o f f a m i l y , p r o g r a m a n d c o m m u n i t y settings. T h i s a p p r o a c h o p e n s u p a w i d e r a n g e of e n t r y p o i n t s f o r i n t e r v e n t i o n a n d can e n h a n c e t h o s e i n t e r v e n t i o n s w h i c h seem t o a c c o u n t f o r g r e a t e s t c h i l d development outcomes. T h e c h a n g e f r o m a s i n g u l a r f o c u s on c h i l d d e v e l o p m e n t o u t c o m e s t o effective program elements which are subject to management and policy h e l p e d in g u i d i n g d e c i s i o n m a k e r s . T h e f i n d i n g s of s u c h e v a l u a t i o n s amenable to making policy decisions which improve the quality of p r o g r a m a n d w h i c h can p r o v i d e m a n a g e r s w i t h t o o l s f o r m a i n t a i n i n g quality.
a f o c u s on c o n t r o l has are directly a preschool the desired
These changing s t r a t e g i e s r e f l e c t n o t o n l y t h e g e n e r a l g r o w t h in t h e v a r i o u s disciplines contributing t o p r o g r a m s a n d e v a l u a t i o n s b u t also t h e d e t e r m i n a t i o n o f c o n g r e s s i o n a l a n d o t h e r p o l i c y m a k e r s t o h a v e s o u n d i n f o r m a t i o n u p o n w h i c h to make decisions.
REFERENCES Abt
A s s o c i a t e s . Final report of the National Day Care Study: Center ( 5 v o l s . ) . C a m b r i d g e , M A , 1979.
Bronfenbrenner, U. The ecology of human H a r v a r d U n i v e r s i t y P r e s s , 1979. Fanshel, D., and Shinn, E.B. U n i v e r s i t y P r e s s , 1978. Grotberg, DC:
Children
development.
in foster
Children
at the
Cambridge,
care. New Y o r k :
MA:
Columbia
E.H. Review of research, 1966-1969 of Project Read Start. W a s h i n g t o n , O C D , 1969 ( E R I C D o c u m e n t R e p r o d u c t i o n S e r v i c e No. 2 8 3 0 8 ) .
T . W . The ecology of child development: A working paper f o r of research, development and evaluation. Washington, DC:
Hertz,
planners Contract
H EW-105-76-1120. Hunt,
J.McV. Implications of plasticity and hierarchical achievements for the a s s e s s m e n t o f d e v e l o p m e n t a n d r i s k of m e n t a l r e t a r d a t i o n . In D . B . S a w i n , R.C. Hawkins II, L.O. Walker and J.H. Penticuff (Eds.), Exceptional infant (Vol. 4). New Y o r k : Brunner/Mazel, 1980.
NTS
Research Corporation. Project Report 6: Description and analysis of program data: Basic Educational Skills Initiative. Durham, NC: NTS, 1980.
Evaluation of Preschool Programs
289
Sameroff, A.J. Issues in e a r l y r e p r o d u c t i o n and c a r e t a k i n g current status. In D . B . Sawin e t al. ( E d s . ) , op. cit.
risk:
Systems of social services for children and their families:
(DHEW Superintendent of
Publication Documents, Willerman,
L.
No. U.S.
An overview
OHDS 7 8 - 3 0 1 4 5 ) . Washington, DC: G o v e r n m e n t P r i n t i n g O f f i c e , 1978.
Genetics
of
psychosocial
risk.
In
D.B.
R e v i e w and
Sawin,
et
al.
(Eds.),
op. tit.
THE A U T H O R S E D I T H G R O T B E R G is Special A s s i s t a n t to t h e A s s o c i a t e C o m m i s s i o n e r f o r D e v e l o p mental S e r v i c e s in t h e A d m i n i s t r a t i o n f o r C h i l d r e n , Y o u t h a n d Families o f t h e U . S . D e p a r t m e n t o f H e a l t h a n d Human S e r v i c e s . DENNIS D E L O R I A is P r o g r a m A n a l y s t in t h e A d m i n i s t r a t i o n f o r C h i l d r e n , and Families o f t h e U . S . D e p a r t m e n t of H e a l t h and Human S e r v i c e s .
Youth