Family support programs: A quasi-experimental evaluation

Children and Youth Scnnccr Rnww, Vol. 1 I, pp. 239-263, Printed in the USA. All rights reserved.

1989 Copyright

0190-7409189 $3.00 + .oo C 1989 Maxwell Pergamon Macmillan plc

Family Support Programs: A Quasi-Experimental Evaluation

University

Janet Reis of Illinois at Urbana-Champaign

Susan Bennett Northwestern University

University

Systemetrics,

John Orme of Maryland at Baltimore

Elicia Hem Inc., Washington,

DC

The results are presented of quasi-experimental evaluation of three demonstration family support programs sponsored by a state child and family services agency. Three hundred and sixty-five family support participants and 265 comparison group participants matched at the community level completed surveys measuring four intermediate treatment outcomes, that is, level of depression, perceived social support, knowledge of child development, and punitive attitudes toward child-rearing. HiThis work was supported in part by, grants from the Illinois Department of Children and Family Services and the Pittway Charitable Fund to the Center for Health Services and Policy Research, Northwestern University. Requests for reprints should be sent to Janet Reis, College of Nursing, 1115 l/2 West Oregon, University of Illinois at Urbana-Champaign, Urbana, IL 61801.

239

240

Reis et al. erarchical linear regression was used to assess the contribution of participant’s age, parenting status, race. type of program from which services were received, and number of contacts with programs to these four intermediate outcomes. Across-site analyses revealed little relationship between receipt of family support services and measures of parenting. Analyses of each site with its matched comparison group found that q’pe of program rvas significantly related to depression and perceived social support (inner city site), perceived social support and punitive attitudes (small city site), and depression and perceived social support (rural site). The direction of these relationships were opposite of those originally hypothesized. These findings are discussed from methodological and service delivery perspectives.

The value of social support as a buffer for stressful events is widely recognized (Brody, 1985; Gottlieb, 1983, 1985). Increasingly, the helping professions are turning to the potential of naturally existing support systems in assisting people to cope with traumatic circumstances (Collins & Pancoast, 19i6). Most recently, the role of social support in optimizing family functioning is being examined. One barometer of interest is the increase in programs providing preventive support services to families under stress (Shonkoff, 1984). These programs typically provide a portfolio of services aimed at enhancing understanding of family dynamics, providing a place where parents can share experiences with other parents in similar circumstances, and referral services to help the family cope with broader economic or social problems. Some observers of this phenomenon suggest that “. . . these programs constitute a significant, if infant movement that will have a major impact on the health, development, and well being of American families” (Zigler, Weiss & Kagan, 1984, p. 1). Currently, there are an estimated 400 family support programs nationwide (Zigler et al., 1984). To date, there have been few published evaluations of the short-term impact of these programs on family functioning, or on more long-term programmatic outcomes such as community;wide rates of child abuse and neglect (Lutzker & Rice, 1984; Weiss 8c Jacobs, 1984). The purpose of this report is to summarize the results of a quasi-experimental evaluation of a family support program sponsored by the Illinois Department of Children and Family Services (IDCFS). Our summary begins with an overview of the theoretical framework used to select measures of short-term program impact. ,4n historical account of the Illinois program hereafter known as the Ounce of Prevention follows.

Family Support Programs

241

Discussion of the methodology includes an explanation of how the comparison sites were selected, a synopsis of each instrument, and explanation of the analytic strategy. The results of the evaluation are presented in a condensed form illustrative of the evaluation findings. The discussion considers issues relevant to future program evaluations and to basic research in child development and socialization. Some readers may prefer that the boundaries between these areas be kept more distinct. We believe that applied and basic research are most effectively advanced through interaction between these modes of inquiry. Thus, we address multiple perspectives in hopes of promoting further dialogue. Theoretical

Framework

From the sponsoring agency’s perspective, the ultimate objective of the experimental family support programs was to reduce community wide rates of child abuse and neglect. Thus, the ideal measure of program impact would be observed change in reported cases of child maltreatment in the treatment communities versus a comparable nontreated community. Several factors precluded use of abuse/neglect as an endpoint measure. First and foremost, the services providers were sincerely dedicated to the goals of primary prevention. Operationalization of this concept meant that their recruitment efforts were aimed at parents under stress who were not already identified as abusive or neglectful of their children. Concern with negative labelling of participants and therefore also of programs precluded almost any recording of atypical parenting behavior. Additionally, insufficient time had elapsed between implementation of the family support programs and the point at which IDCFS desired assessment of program impact to reasonably expect a detectable decrease in the number of child maltreatment cases. Since data on the ultimate treatment outcome were not available, instrumental measures of program impact were selected. In this context, instrumental is defined as a program effect which leads to other outcomes (Rosen & Proctor, 1981). Measurable attitudes and behaviors correlated with potentially preventable child abuse and neglect were identified through review of the cumulative evidence on the etiology of child maltreatment (Howze & Ketch, 1984; Wolfe, 1985). A process model of the determinants of parenting proved particularly useful in prioritizing the factors contributing to a family’s functional equilibrium (Belsky, 1980; 1984). In order of theoretical importance, the three factors posited to be most influential on parental functioning are the

242

Reis et al.

parents’ psychological resources, sources of stress and support, and the temperamental and behavioral characteristics of the child. Ideally, all three domains would be assessed simultaneously for families participating in a support program. Consideration of programmatic objectives, and evidence that the risk characteristics of the child are relatively easy to overcome, led to the choice of the first two elements - that is, parental psychological resources and level of social support as being most important for measurement in an evaluation. These global factors were further operationalized to address the specific needs and life circumstances of the average participant in the family support programs as determined by the first year of program operation. The typical participant in the first 12 months of program start-up was a young female who was not married but who was either pregnant for the first time or had one child. Given this constellation of background characteristics of the parent, two aspects of personal psychological resources, level of depressive symptomatology and parental knowledge, were selected for measurement. Previous research has shown that women are generally more depressed than men (Weissman 8c Klerman, 1977). Developmental theory suggests that this depressive state may further be exacerbated by the stresses of parenting (Pascoe, Loda, Jeffries dc Earp, 1981). Investigations of the correlates of child abuse have found that neglectful and abusive parents as compared to normal parents are more lonely, more rigid, less interactive with their children, and by inference, more depressed (Wolfe, 1985). Abusive and neglectful parenting has also been found to be related to unrealistic expectations of child development (Burgess & Conger, 1978; Twentymen & Plotkin, 1982). Parental age has in turn been documented to serve as a proxy for level of knowledge of developmental milestones, with younger parents being less knowlof family edgeable than older parents. Since the preponderance support program participants in this sample were teenagers, it was hypothesized that these pregnant and parenting adolescents would be less knowledgeable and less lenient, predisposing them to neglectful or abusive behavior. The third facet of the process model of parenting and child well-being measured in this evaluation assesses the type and amount of social support perceived as available to the parents. Theories on the context of parent-child relations postulate that the marital relationship is the first-order support system. Since most of the parents in the family support programs were single, it was hypothesized that the broader the network of kin, friends, and social and health care providers would constitute the principle source of

Family Support Programs

243

support. The hypothesis is bolstered by a substantial literature on the important buffering role of social networks for families under stress (Cochran & Brassard, 1979; Unger 8c Powell, 1980; Wilcox, 1981). Three maternal sociodemographic characteristics were also included in the conceptual model of predictors of parenting behavior and the final statistical model used to assess the impact of social support services. These characteristics (maternal age, parity, and race) are not amendable to modification through interventions, but nonetheless contribute to a parent’s psychological resources. Given their important role in the complex give-and-take between parent and child, it was thought necessary to explicitly analyze the relationships between these variables and the instrumental measures described above. Review-of several different bodies of literature suggests how each of these variables might be associated with maternal depression, child-rearing attitudes, and social support. Young mothers are generally less well-versed in the specifics of child care (Linde & Englehardt, 1979) and are more depressed than are older mothers (Colletta, 1983). Parity, as a proxy for previous experience with child caretaking, should be related to knowledge and attitude of child development. Ethnographic descriptions of parent
Reis et al.

244 Program History

The Ounce of Prevention demonstration effort was implemented in September 1982 under the auspices of the Illinois Department of Children and Family Services (IDCFS) and a private foundation. First-year funds to support the demonstration program described here were generated from matching grants of $400,000 from private and public partners. Funding in subsequent years was provided primarily by IDCFS. Each program is situated in a community identified as having disproportionate number of families at risk for malfunctioning. The criterion for selection of individual sites included documentation of community need for a new preventive service program, administrative capacity for implementing a new program, and a budget deemed reasonable and appropriate by the sponsoring agencies. Community need was operationalized to include communitv-wide rates of child abuse and neglect, teenage pregnancy, and unkmplovment rates. This index was a modification of previous work on indices of high-risk communities (Garbarino & Sherman, 1980). A total of six programs were chosen for participation from over 115 proposed service programs. Two of six programs were located in the City of Chicago in community areas with high rates of unemployment and teenage pregnancy (and anecdotally high rates of abuse and neglect). The remaining four sites were situated in northwest Illinois (City of Rockford), central Illinois (City of Peoria), and Southern Illinois (three geographically contiguous counties otherwise known as Shawnee). One Chicago site (Lawndale), the mid-state site (Peoria), and the three southern country sites had programs stabilized enough to permit evaluation. The other two sites experienced too many difficulties in program implementation to be included in the evaluation effort. The three sites participating in the evaluation are comparable both in terms of adult unemployment (roughlv 10%) and percentage of open IDCFS child cases as proportionate to the child population in their respective communities. Initiation of a trial demonstration program for the prevention of child abuse and neglect represented a departure from traditional IDCFS services. The agency’s $196 million budget (as of Fiscal Year 1985) was primarily dedicated to investigating reports of abuse or neglect, following up on indicated cases and arranging foster placement, providing day care, and other services to children found to be at risk for maltreatment (Illinois Department of Children and Family Services, 1984). A 700% growth in reported cases of abuse and neglect (from 9,183 cases in 19i’i to 67,256 cases in 1984)

Family Support Programs

245

prompted the agency to rethink its goals and re-examine potential points of intervention. When approached by a private foundation with similar concerns about testing ways to maximize family functioning, IDCFS agreed to pool resources and launch a multiple-site, three-year demonstration project for the improvement of family functioning. A summative evaluation of the demonstration projects was linked to introduction of the new services. The three social-support programs participating in the evaluation serve a total of 500 registered families’per year and typify the service philosophy of the Ounce of Prevention in seeking common objectives through different models of service delivery. Even though diversity in services was deliberately fostered by the Ounce administration, the program was built upon a set of core programs. Each has a drop-in center for parents and offers parent education classes. The parent education and support classes are modeled after the Minnesota Early Learning Design (MELD) Program Uunge & Ellwood, 1986). Social-support researchers such as Gottlieb (1985) would characterize the drop-in center and parent education programs as interventions aimed at providing participants with a broader network of similar peers who share common stressful predicaments. All programs relied on similar methods of advertising the availability of the new drop-in centers including presentations to local public school’s Parent-Teacher Associations and local community groups such as the Boys and Girls Clubs. Written materials were disseminated to other social service and health-care providers. Review of program records documented that 75% of drop-in center participants used this service a minimum of four times in a 12month period. Sixty percent of the mothers participating in the home visitation program were also involved in the drop-in center program. Two of the three sites (Peoria and Shawnee) also offered a home visitor’s program. In Peoria, all adolescents giving birth in the service area were identified through the community hospital and referred to the family-support program “Good Beginnings.” Each young mother was then approached regarding her interest in participating in the home visitation and related programs. Approximately 90% of all adolescents notified of the availability of familysupport services agreed to join the program. A similar procedure was followed in the Shawnee three county area with comparable results. In both programs, the home visitors were community-based women who were interested in promoting family well-being. Integral to start-up of the service demonstration programs was planning of a summative evaluation of program impact. A contract

Reis et al.

246

for such an evaluation was let by IDCFS to an independent, university-based group charged with identifying an evaluation design and short-term outcomes, carrying out the necessary data collection, and completing the analyses. An evaluation research associate was assigned to work half time at each of the demonstration sites for the duration of the demonstration period. Those experienced with assessments of innovative social service programs such as the Ounce of Prevention may well question the wisdom of attempting to document program effects only shortly after program implementation. A potent mixture of personalities, political pressures, and high expectations for the achievements of these innovative services made a summative or impact evaluation more appealing to funders than a formative, more qualitative effort. Whether this mindset may have placed the proverbial cart before the horse shall be considered later in the discussion. Methods

Table 1 presents the race, age, and parenting status of the 365 parents surveyed between March 1984 and June 1985 for the evaluation and the 265 parents constituting the nonequivalent comparison groups. Participants in the Ounce of Prevention programs were defined as mothers receiving service two or more times in a 12-month period. 1 Across the Ounce and comparison groups, all participants were women, and the majority (66%) had one child. Of this subgroup of one-child families, 12% of the mothers were pregnant. Sixty percent of the sample had less than a high-school education at the time of enrollment into the program. Seventy-two percent of the sample reported never being married; the same percentage were unemployed. Seventy-eight percent reported having no occupation. Seventy-four percent of the sample had an annual income of $15,000 or less, with 11%) falling into the $10,000 to $15,000 bracket, 23% falling into the $5,000 to $10,000 bracket, and 39%, falling into the $0 to $5,000 bracket. Comparison of Ounce ‘For purposes of this analvsis, Ounce participants were defined as mothers who had 2 or more contacts with an Ounce program in a 12-month period. This definition of participation was derived by the Ounce central administration and therefore adopted by the evaluation team. Several other operational definitions of Ounce Participants were tested in the statistical model, that is, restricting the definition of participants to four or more visits, and, six or more visits. Since these alternative operational definitions did not alter the results of the statistical analysis for Ounce participants versus the matched comparison, a decision was made to adhere to the original administrative defmition of program participation.

Family Support Programs

247

TABLE 1 Frequency Distribution of Demographic Characteristics of Ounce and Comparison Group Mothers

Demographic characteristic Race White Black Age Years <15 16 through 19 Years > 20 Mean Age SD

Parenting Status No children pregnant One child not pregnant One child pregnant Two or more not pregnant Two or more pregnant Total Number of Cases’

Lawndale ComparOunce

Peoria Compar-

ison Ounce

ison

Southern Illinois ComparOunce

Totals Compar-

ison Ounce

ison

1 a5

72

126

54 80

50 20

45 2

135 231

100 154

11 51 16 17.7 2.0

4 20 47 22.3 5.4

23 166 17 17.6 1.5

4 40 91 22.2 4.9

5 46 16 17.9 1.6

1 18 28 20.6 3.1

39 263 49 17.7 5.6

9 78 166 21.9 4.8

34

23

6

30

34

9

74

62

26

10

173

38

25

17

243

65

7

12

15

23

9

6

33

41

13

15

5

28

1

12

50

55

4

10

2

16

1

4

9

30

87

72

207

143

71

48

365

265

79

‘Deviations from the total number of Ounce and Comparison missing data.

group participants are due to

participants with individuals in the comparison group showed that proportionately more of the Ounce participants were nonpregnant mothers with one child. Seven sources of referral to the family-support programs were reported. The three most common sources of referral were schools (26% of program participants), friends (24% of program participants), and health care professionals (24% of program participants). In descending order of importance, the remaining referral sources came from social-service agencies (13% of program participants), family (7% of program participants), the media (2% of program

248

Reis et al.

participants), law enforcement officials pants), and the church (.3% of program

Evaluation

(.6% of program participants).

partici-

of Design

A noncomparable comparison group design with a crosssectional analysis was used in this evaluation. This design represents a compromise with the originally proposed pretest-posttest design with a noncomparable comparison group. The evaluation plan was altered as programmatic experience demonstrated that the resources necessary for client follow-up were not available. Therefore, data collection was staggered over time at each site in order that the samples include participants with a range of programmatic contacts. Approximately 50% of each Ounce sample had received 6 or more contacts at the time of data collection. Comparison communities were selected based on their standing on a composite risk index designed to give a crude measure of the quality of life for mothers and children in a given community and paralleling other methods of community risk assessment (Garbarino & Sherman, 1980). Five indices of maternal and child welfare were summed to create an overall risk index. These measures include community-wide birth rates, rates of teen births, prematuritv rates, excess infant deaths, and Aid to Families with Dependent Children. Definition of these parameters is given in Appendix ,4. A total risk score was generated for all communities with over 50,000 residents and the 77 community areas in Chicago. The familysupport programs fell into the top tenth percentile of risk (see ,4ppendix L4). Each comparison community fell within five units of its matched treatment site. Once an appropriate comparison community was identified, an agency within the community willing to participate in the evaluation effort was sought. Agencies were approached about the possibility of participating in the study given their reputation for willingness to sponsor research, and, the similarity of their clientele to the Ounce participants. In total, three comparison agencies participated. Each of three agencies provided routine health care services to its clientele. Review of the portfolio of clinic services at each site showed that there were no special programs such as offered through the Ounce of Prevention programs. Clinic staff often recognized the need for augmented social support programs but had neither the expertise nor resources to introduce such programs into their clinic settings. Table 2 summarizes comparative data on the services received by each treatment group and its comparison, and, total groups. The

249

Family Support Programs

TABLE 2 Summary of Primary Service and Number of Program Contacts for Ounce and Comparison Group Mothers Southern Illinois Lawndale Peoria Totals ComComComCompari parparparOunce ison Ounce ison Ounce ison Ounce ison

Primary Service’ Drop-in Center Visitation/ Outreach Day care Medical EducationSupport Number of Contacts Low (< 3) Medium (4 through 10) High (> 11) Mean No. of Contacts SD

12

30

18

5 72

178 1

13

9

1

130

30

47

10

69

192 1 30

14 249

111

32

16

42

60

51

27

26

103

119

13 36

17 5

85 61

55 13

16 27

16 4

114 124

88 22

29.8 40.2

4.1 4.8

10.2 11.5

‘Deviations from the total of Ounce and Comparison data.

5.2 4.0

16.3 25.9

4.4 4.2

14.8 23.9

4.7 4.3

group participants are due to missing

majority of comparison group mothers were recruited into the study through ambulatory health care clinics. These mothers were approached by a research assistant while they were waiting for a scheduled appointment with a health care provider. An effort was made to randomly select among the mothers coming to the clinic so that as representative a sample as possible was obtained. On average, these comparison group mothers had 4.7 contacts in the past 12 months with their clinic, in contrast to family support participants who had an average of 14.8 contacts. Based on the nature of services available at the health care clinics, it is safe to assume that the comparison group mothers were receiving well child care for their children and either prenatal or family planning services for themselves depending on whether they were pregnant. The average number of service contacts for the comparison group is similar to the average number of health care contacts made nationwide by

250

Reis et al. TABLE

3

Potential Range of Total Scores and Medians, Means, and Standard Deviations of Instrumental Measures

Measure Childrearing attitudes Developmental knowledge Wilcox social support CES-D

Potential range

Median

Mean

15-30

19.00

18.88

1.89

.33

10-20

15.00

13.53

1.52

.27

l&3(5 O-60

35.00 15.00

33.68 17.13

2.91 10.75

.80 .89

mothers of young children in Technology Assessment, 1988). Instnmentnl

public

health

Standard deviation

settings

Xlpha

(Office

of

Measures

Table 3 summarizes the potential range of the four instruments used in the evaluation along with the median, mode, standard deviation and, level of internal consistency (Cronbach’s alpha) fol each tool. Additional information on the psychometric characteristics of these tools is a\.ailable elsewhere (Reis, Orme, Barbera-Stein, & Herz, 1987). Suffice it to say here that each instrument was chosen after an extensive search for the most reliable and valid tool. In the absence of published data on the properties of these tools, the evaluation team was guided by expert opinion. Selection of a questionnaire measuring parental knowledge and attitudes toward child development proved to be especially challenging as virtuali!. no information is a\.ailable on the reliability or validity of the few instruments reported in the literature.? CES-D D cf-’TPSS>O~~ ::’ Scczlr. The CES-D is a self-report measure designed by the Center for Epidemiological Studies of the National Institute of Xlental Health to study depressive symptoms in the population (Radloff. 1977). The CES-D consists of 20 symptoms rated on a four-point scale in terms of frequency of occurrence during the previous week. .4 score of 3 means the symptom occurred most of the time. Total scores may range from 0 to 60. High scores suggest being at risk for depresslon. The usual cut-off

Family Support Programs to indicate clinical depression to be .89 for this sample.

251 is 16. Cronbach’s

alpha was calculated

Developmental Child-Rearing Attitudes and Expectations. The Field instrument is a self-report measure designed to evaluate knowledge of the average age of occurrence of developmental milestones and attitudes toward child-rearing (Field, Widmayer, Stringer 8c Ingatoff, 1982). The original scale was modified by the addition of four items (two items to each subscale) and format changes so that the parents were presented with clear choices. Parental attitudes are measured through presentation of 15 child-rearing scenarios. Total scores range from 15 to 30 with the higher scores indicating a punitive attitude. The last 10 items of the scale assess expectations of the age by which a baby attains certain developmental milestones “smile,” “ be fully potty trained, ” “obey when you say no”). (e.g., Respondents are asked to select one of seven age categories ranging from 0 to 5 months to 30 to 36 months corresponding to the period in which a child would be able to perform the specific task. Total scores range from 10 to 20. Scoring was done in accordance with the original standards used by test developers. Cronbach’s alpha for this sample was calculated to be .33 for child-rearing attitudes and .27 for developmental expectations. In view of these low reliabilities, the statistics derived from these measures should be regarded with caution. Perceived Social Support. The self-report social network scale used in the evaluation focusses on the extent to which parents have and use personal and community-based relationships for advice or assistance in their daily lives (Wilcox. 198 1). The instrument consists of 18 binary-choice items (True-False) regarding the types of things people feel they can ask of others. As a measure of each participant’s primary source of social support, respondents also are asked to indicate which person(s) thev most often had in mind while completing the survey. Total scores mav range from 18 to 36. This tool is considered to be one of the more internallv consistent measures of global functional social support (Cohen & wills, 1985). Cronbach’s alpha was calculated to be .80 for this sample. Analytic Strategy The analytic strategy employed in this assessment of the impact of family support services is deliberately conservative and is guided by the process model of parenting utilized to identify instrumental

Reis et al.

252

outcomes. The sociodemographic variables were entered first as potential predictors, followed by treatment status and number of programmatic contacts. This sequence of entry permitted assessment of the marginal effect of social support programs over and above maternal age, parenting status, and race. Hierarchical multiple regression was the analytic technique of choice. Dummy coding of the categorical independent predictors was carried out as follows: race (0 = white, 1 = black), parenting status (divided into four levels and coded 1 if parenting status existed at that particular level), and treatment status (0 = comparison group mothers, 1 = Ounce of Prevention mothers). A positive correlation or a positive regression coefficient indicates that the level of the variable coded as 1 displays a positive relationships with the dependent measure under consideration. Results A comparative analysis of Ounce and comparison group participants on each of the four scales according to race, age, and parenting status is summarized in Table 4. Tests of potential differences within the Ounce and comparison groups found that the Ounce mothers differed from each other on all four instrumental measures according to maternal race and age (all differences reported in Table 4 are significant at the .05 level or less). Parenting status was also a differentiating factor among Ounce participants on punitive attitudes, developmental knowledge, and level of depression. In brief, Ounce mothers who were black, young, and pregnant with no other children had a less favorable standing on the instrumental measures than did their white, older, or multiparous peers. As shown in Table 4, the comparison group mothers were generally more homogenous with the exception of maternal race which differentiated between mothers in the same direction as among Ounce participants. Separate hierarchical multiple regression analyses were conducted for each of the four outcome measures for each of the three combinations of demonstration and comparison groups. Analyses were performed across all sites as well, but since the results by in large did not achieve statistical significance, they are not reported here. Interactive terms between Treatment and number of contacts were also entered into the ana!yses, but are omitted from this report as the). \+‘ere not statistically significant. Table 5 presents a summary of the predictive role of maternal sociodemographics and intervention status with the four instrumental measures. Looking first at the three sociodemographic variables,

Family Support

Programs

253

TABLE 4 Scale Means for Ounce and Comparison Group to Race, Age, and Parenting

Participants

Instrumental measure’ developmental Social Knowledge sunnort ComComparparOunce ison Ounce ison

Demographic characteristics

Punitive attitudes ComparOunce ison

Race White Black

18.42* 18.09* 15.12* 15.04” 33.92* 33.12* 19.19 19.21 14.12 14.25 32.74 33.93

Age Years ~15 19.65* 16 through 19 18.95 Years > 20 18.17 Parenting Status No childrenpregnant 19.09* One childnot pregnant 18.97 One childpregnant 18.70 Two & more-not 18.03 pregnant Two or morepregnant 18.64

According

Status

Denression ComparOunce ison

16.30* 15.20* 19.30 16.97

18.34

14.34”

14.28

33.253 35.44”

18.14”

15.88

18.74 18.84

14.36 15.16

14.46 14.66

33.06 33.91

34.90 34.10

19.46 13.57

16.94 15.94

18.95

13.83* 14.18* 33.42

34.77

20.03*

16.68

18.45

4.56

14.78

33.30

34.58

18.15

16.05

18.90

4.69

14.29

32.37

34.78

17.48

16.38

18.76

15.42

14.28

33.77

33.90

14.22

14.43

18.95

14.67

15.27

33.32

33.74

20.56

18.86

‘Ounce and comparison group mothers were separately tested for potential differences on each instrumental measure within the categories of maternal race, age, and parenting status. *p<.o5.

13 statistically significant relationships at the .05 level were identified. Maternal age was related to knowledge of children’s developmental milestones at all three demonstration sites and to perceived social support and depression at the inner-city site. Parenting status and race were related to knowledge of developmental milestones at sites 2 and 3. Parenting status also accounted for a significant percent of the explained variance on perceived social support and

Reis et al.

254 TABLE

5

Summary of Effect of Sociodemographic Variables and Family Support Services on Maternal Knowledge, Attitudes, Depression and Social Support

Independent variables Age PGenting status Race Treatment status Number of contacts

Instrumental Attitudes

Knowledge of child

develonment 1 2 3 .19* .Il”

to%rard childrearing 1 2 3

.16* .12 .19

0

Measure’ Perceived

Maternal depression 1 2 3

social

SlIDDOt-t

1

2

3

.30* .03 .22* .24* ‘10

.I0

.23 .26

.23* .42* -27 .14 .I3 .34* .47* .29 .22* .19

.41* -11 .35 .43 313 .37

.36* .16 .20 .38 .21* .21

.27

.33

.48

.31 .25* .24

.49* .13 .39

.44* .31* .36*

.29

.35

.jO

.31 .25

.30* .49

.14 .39

.47* .32

.39

The three sites are Lawndale (site L), Peoria (site 2), and Shawnee (siie 3). ‘The Hierarchical multiple correlation coefficient is reported for each independent variable. Statistically significant changes in the percentage of variance accounted for as measured b! the F statistic are indicatd with an asterisk. All predictor variables designated as statisticalb significnat are significant at the .05 level or less. Parenting status was coded to include not pregnantit child, pregnant’1 child, or not pregnant/:! or more children, and pregnant/2 or more childen

depression at site 1. Maternal race emerged as a significant predictor of attitudes toward child-rearing and perceived social support at site 2. Examination of the correlational relationships between treatment status, number of programmatic contacts and the four instrumental measures identified another set of seven statisticall). significant R" values. Treatment status was associated with attitudes toward child-rearing at site 2, level of depression at site 1, and perceived social support at all three sites. Number of programmatic contacts made an independent contribution to attitudes toward child-rearing for site 3 participants and perceived social support for site 1 participants. The direction of the correlational relationships between the independent and dependent variables was assessed with standardized regression coefficients and the results of the t tests for testing the statistical significance of these coefficients. A listing of the standardized coefficients appears in Table 6.Since the focus of this discussion is on treatment effects, we limit our remarks to the

*p<.o5.

Age Parenting Status Not pregnant/ 1 child Pregnant/ 1 child Not pregnant/ 2 and more children Pregnant/ 2 and more children Race Treatment status Number of contacts

.27

.18 -.27*

- .08

-.05

- .03 .12

0

.14

-.08

.13

.07

- .05

- .05

2

-.04

.I0

1

-.17

- .02

.35 -.26*

.06

.27*

.41*

- .09

3

Regression

Knowledge of child development

Summary of Standardized

1

-.06

-.I3

.07 -.07

.19

.19

-.12

- .27

-.02

.19*

-.I9 .19”

-.09

-.08

-.Ol

.Ol

2

.21*

.lO

-.I3 -.I3

- .02

- .06

- .08

.06

.24*

.20* -.ll

.25*

.I 0

- .02

1 - .32*

3

-.03

0

- .05 .09

- .03

-.14

- .05

-.04

2

Maternal depression

measure

.08

Instrumental Attitudes toward childrearing

Variables

TABLE 6 Coefficients for Socioclemographic

3

0

.15

-.05 -.I4

.30

.06

0

-.29

1

.19*

-.35*

-.34 .07

-.28*

-.ll

-.I1

.30*

- .05

-.31*

.07 -.15*

-.06

.04

0

- .06

2

SuDDort

Perceived Social 3

-.I8

-.29*

- .04 -.05

-.12

-.05

-.ll

- .02

and Family Support Service

5 o

3 cd ;t 09

z

r

5 r: ‘c

r

256

Reis et al.

regression weights for treatment status and number of programmatic contacts. Ounce of Prevention participants displayed less perceived social support than members of the comparison group in each of the three sites. Ounce participants were also more punitive in their orientation toward child-rearing in site 2 than were the comparison group counterparts. Women in the inner-city Ounce program were more depressed than their comparison groups counterparts. Number of programmatic contacts was positively associated with punitive attitudes toward child rearing in site 3 and perceived social support in site 1.

Discussion The results of this quasi-experimental evaluation of three demonstration family-support programs found contradictory patterns of program impact for individual programs as compared to a noncomparable comparison group. Specifically, the regression analyses found no evidence of treatment impact of family-support services, although total amount of variance explained did achieve a medium effect size according to conventional standards (Cohen, 1977). A number of plausible alternative hypotheses can be summoned to explain the contradictory trends in treatment effect. Difficulties with obtaining truly comparable treatment and comparison groups through matching are discussed below. In retrospect. perhaps the most fundamental threat to this evaluation was a failure to design and implement a robust preventive treatment. Program planners operated from the untested assumption that the portfolio of services assembled was most responsive to their community’s needs. In fact, participants’ standing on dimensions of parenting and intensity of services provided suggest that the experimental programs were successful in recruiting participants who might benefit from additional information on family functioning and augmentation of their social support network. Yet as in previous assessments the results summarize here illusof primary prevention programs, trate the difficulties of documenting the payoffs from short-term, loM-intensit!, services (Leavy, 1983; Resnick, 1985). The Illinois experience recommends that more care be taken in pinpointing the psychological needs of a potential client population and matching these needs with well-defined interventions. Additional ethnographic work, particularly for minority families also appears to be in order. Black families caught in different cultural and economic cross currents must contend with a range of stresses that are important for program planners to understand (Staples, 1985). These kinds of

Family Support Programs

257

painstaking efforts may be viewed as unresponsive to local political pressures but would yield long term dividends in terms of building a knowledge base on program effectiveness. Methodologically, the outcomes of this evaluation verify once again the perils inherent in quasi-experimental designs. Of special concern is the fact that the process of self selection invariably produces groups of people who differ from each other in motivation to seek services and commitment to following through on professional advice (Cook & Campbell, 1979). The noncomparability between family support participants and parents recruited through ambulatory health care clinics is seen in their respective standing on measures of parenting. Relative to the comparison group, the Ounce mothers were more depressed, less knowledgeable about child development, and perceived themselves as having less social support. Other differences between the two groups undoubtedly exist. Thus, even though considerable effort was made to match the two groups of mothers on major background variables, and the kinds of communities they resided in, the results of the analysis verify the observation that matching more often than not undermatches. The results of this quasi-experimental evaluation also point to the necessity of ongoing psychometric work. The low level of internal consistency of the measures of knowledge of child development and attitudes toward child rearing were a surprising disappointment. These tools had previously been used in a program evaluation, and had been recommended for the current study by a well-known developmental psychologist. Nonetheless, these instruments failed to achieve an acceptable level of reliability, thus compromising, if not eliminating, their usefulness as interim outcome measures. Recently published data on the psychometric characteristics of the knowledge of child development inventory (Larsen & Juhasz, 1986) can perhaps be used to select better instruments for future evaluations. Instrument refinement of a different nature is also in order for measures of social support. A number of reviews have delineated the complexities and ambiguities inherent in this concept (Barrera & Ainlay, 1983; Hella, 1986; Leavy, 1983). The positively skewed responses of the sample reported on here attest further to the semantic problems encountered in arriving at valid definitions of social support. The self reports of the family-support program participants as having ample and satisfactory social support systems is particularly interesting in light of the original Ounce of Prevention program objectives of expanding and strengthening support systems. Data on the economic and educational circumstances as well

258

Reis et al.

as the level of depressive symptoms reported of the participants would suggest that the family-support program providers were successful in their recruitment of their target population. Yet, when queried directly about social support, participants claimed to be doing well. More work is needed to unravel the nuances of social support, as well as to determine precisely what type of support is necessary for optimizing child development. The state of knowledge on how to define social support and measure services designed to change available support must also be put in the context of program growth. The past few years have witnessed a virtual explosion of programs to strengthen families (Dunst, Trivette, SCCooper, 1986; Hobbs, Dekecki, Hoover-Dempsey, Moroney, Shayne & Weeks, 1984; McCubbin, Cauble 8c Patterson, 1982). Further, most would agree that the pressures facing families are unlikely to abate and that people working with families will continue to gravitate to the idea of enhancing support. As pointed out by Garbarino (1986) the proliferation of programs and the needs of American families driving this rapid expansion place a sharp challenge to evaluators to do a better job in documenting program accomplishments. Included on the future evaluation agenda are debates about the balance to be struck between process and outcome evaluations, and concurrent education of program administrators and politicians about what kinds of information different types of evaluations can reasonably be expected to provide. In light of these daunting tasks, we close this discussion with comments on a methodological issue long plaguing communitv based research, i.e., selection of the most appropriate evaluation design. Appropriateness in this context requires weighting methodological rigor against the acceptability of logistical and ethical features. Specifically, many evaluators continue to recommend that designs with random assignment be used in community-based research (Hormuth, Fitzgerald 8c Cook, 1985). The unpalatability of randomly withholding services from families thought to be in need continues to work against wider acceptance of this procedure which maximizes internal validity and therefore the interpretability of evaluation results. Further, the ever-prevalent phenomenon of self selection in self-help groups confounds the feasibility .of implementing a randomized design (Levy, 1984). Those indl~~lduals in most need are not necessarily the same people who seek assistance. Accumulation of equivocal evaluation findings may yet convince program administrators that the advantages of a randomized design outweigh ethical reservations. While the evaluation community compiles the strengths and weaknesses of quasi-experimental evaluations, are there alternative design options to be considered?

Family Support Programs

259

Specifically, are there designs more internally valid than the crosssectional approach employed here but which are also logistically feasible and politically palatable? One design/analytic strategy within reach if the necessary planning is done is a pretest-posttest design with a delayed treatment group and analyzed with an Analysisof-Covariance (ANCOVA) model. The prerequisite planning steps include establishment of mechanisms for pre- and posttesting of participants, and devising a schedule which delays entry into the program for a sufficiently long period of time so that appropriate measures can be obtained on those people interested in but not yet involved with the special program. Formation of a comparable, untreated group would help expiate the influence of self selection. The ANCOVA procedure would adjust further statistically for pretest differences between experimental and comparison groups thus optimizing the likelihood of detecting treatment effects (Huitema, 1980). Adoption of the type of evaluation design, use of more psychometrically sound measures, and clearer articulation of the intervention model would all contribute to superior evaluations, the results of which would enable more efficient delivery of more effective family support services. References Barrera, M. Jr., & Ainlay, S.L. (1983). The structure of social support: A conceptual and empirical analysis. Journal of Community Psycholog?, 11, 132-143. Belsky, J. (1980). Child maltreatment: An ecological integration. American Psychologrst, 35, 320-335. Belsky, J. (1984). The determinants of parenting: A process model. Child Deuelopment, 55, 83-96. Brady, J.G. (1985). Informal social networks: Possibilities and limitations for their usefulness in social policy. Journal of Community Psycholoa, 13, 338-349. Burgess, R.L., & Conger, R.D. (1978). Family interaction in abusive, neglectful, and normal families. Child Development, 49, 1163-I 173. Cochran, M.M., 8c Brassard, J.A. (1979)., Child d evelopment and personal social networks. Child Development, 50, 601616. Cohen, J. (1977). Statistical power anacysis for the behavioral sciences. i%ew York: Academic Press. Cohen, S., & Wills, T.A. (1985). Stress, social support, and the buffering hypothesis. Psychological Bulletin, 98(2), 3 10-357. Colletta, N.D. (1983). At risk for depression: A study of young mothers. The Journal of Genetic Psychology, 142, 30 l-3 10. Collins, A.H., 8c Pancoast, D.L. (1976). Natural helping networks: A strategy for prevention. Washington, DC: National Association of Social Workers. Cook, T., & Campbell, D.T. (1979). Qu as1‘-exp erimentation: Design and analysis issues forfield settings. Chicago: Rand McNally. Dunst, C.J., Trivette, C.M., & Cooper, C.S. (Eds.) (1986). Social support, family

Reis et al. functioning, and infant development. infant .Ilrntal Health Journal. I( 1). 6-105. Field, T.M., Widmayer. SM.. Stringer, S.. & Ingatoff, E. (1982). Teenage lo\\.erclass black mothers and their preterm infant: An intervention and developmental follow-up. Child Development, 51, 426-436. Garbarino, J. (1986). Can we measure success in preventing child abusei Issues in policy, programming and research. Child Abuse and ,l’eglect. 10. l-l?+ 1.56. Garbarino, J., & Sherman, D. (1980). High risk neighborhoods and high risk families: The human ecology of child maltreatment. Child Dmv~lopm~wt, 51. 188-198. Gottlieb, B.H. (1983). Social support as a focus for integrative research in psychology. American Psychologist, 38, 278-287. Gottlieb, B.H. (1985). Social networks and social support: An o\zerview of research, practice, and policy implications. Health Education f&arterl~, Z2( 1), .i-22. Hella, K. (Ed.) (1986). Desegregating the process of social support. ,]010.7/n/of Consulting and Clinical Psychology, 44, 4 1S-470. Hobbs, N., Dekecki, P.R., Hoover-Dempsey, K.\‘., hloroney. R.M., Shayne,. hl.\v.. & Weeks, K.H. (1984). Strengtheningfamilies. San Francisco: Jossey-Bass. methHormuth. S.E., Fitzgerald, N.M., & Cook, T.D. (1985). Q uasi-experimental ods for community-based research. In E.C. Susskind & D.C. Klein, (Eds.) Communic research: Methods, paradigm.s and applications (pp. 206-249). Nell York: Praeger. Howze, D.C., & Ketch, J.B. (1984). Disentangling life events, stress and social support: Implications for the primary prevention of child abuse and neglect. Child Abuse ti Neglect, 8, 401-409. Huitema, B.E. (1980). The anal@ ofcovariance and alternatzrvr. Seu I’ol-k: \j’ile\-. Illinois Department of Children and Family Services (1984). Human Service5 Data Report: Fiscal years 1983-1985. State of Illinois. Junge, M.. & Ellwood, A. (1986). MELD: Parent information and support groups. Infant ,\1ental Health Journal, 7(2), 146-155. Larsen, J.J., & Juhasz, A.M. (1986). The knowledge of child development inventor!.. .4dolescence, 21(8), 39-34. Leavy, R.L. (1983). Social support and psychological disorder: .A revie\\. .]ourwr~ oj Communi~ Psycholog?, 11, 3-2 1. Levy. L.H. (1984). Issues in research and evaluation. In A. Gartner ,Y-F. Riessman (Eds.) 7%r~self-help rezlolzttion (pp. 155-172). New York: Human Sciences Press. Linde. D.B., & Engelhardt, K.F. (1979). What do parents know about infant dekrelopment? Pediatric Nursing, 5, 32-36. Lundberg, h1.J. (1974). The incomplete adult: Social r1a.u con.strnznt.\ 07, pcrxntali~ der~elopmcnt. Westport, CT: Greenwood Press. Lutzker, J.R., & Rice, J.M. (1984). Project 12-ways: Measuring outcome of a large in-home service for treatment and prevention of child abuse and neglect. Chzld Abuse ti Neglect, 8, S19-S24. McCubbin, H.I., Cauble, A.E., & Patterson, J.M. (Eds.) (1982). Farnil> stress, cojxng, and social support. Springfield, IL: Charles C. Thomas. Office of Technology Assessment (1988). Healthy children: Inuesting in the future. Washington DC: U.S. Government Printing Office. Pascoe, J.M., Loda, F.A., Jeffries, V., & Earp, J.A. (198 1). The association between mother’s social support and provision of stimulation to their children. Deuelop-

Family Support Programs mental and Behavioral Pediatrics, Z(l), 15-19. Radloff, L.S. (1977). A self-report depression scale research in the general population. Journal of Applied Psychological Measurement, 1, 385-40 1. Reis, J., Orme, J., Barbera-Stein, L., & Herz, E. (1987). A multidimensional inventory for assessment of parental functioning. Evaluation and Program Planning,

10,

149-157.

Resnick, G. (1985). Enhancing parental competences for high risk mothers: An evaluation of prevention effects. Child Abuse and Neglect, 9, 479-489. Rosen, A.R., & Proctor, E.K. (1981). Distinctions between treatment outcomes and their implications for treatment evaluation. Journal of Clinical Psychology, 49, 418-425.

Shonkoff,

J.P. (1984). Social support

American

Journal

of Public Health,

and the development

of vulnerable

children.

74, 3 1O-3 12.

Silverstein, B.R. (1975). Children of the dark ghetto: A developmental p.sychology. New York: Praeger. Stack, C.C. (1974). All our kin: Strategies for survival in a black community. New York: Harper & Row. Staples, R. (1985). Changes in black family structure: The conflict between family ideology and structural conditions. Journal of Marriage and the Family, 47, 1005-1014.

Twentyman, C.T., & Plotkin, R.C. (1982). Unrealistic expectations of parents who maltreat their children: An educational deficit that pertains to child development. Journal of Clinical Psychology, 38, 497-503. Unger. D.G., & Powell, D.R. (1980). Supporting families under stress: The role of social networks. Family Relations, 29, 566-574. Weiss, H., & Jacobs, F. (1984). Effectiveness and evaluation of family support and education programs. Cambridge, MA: Harvard, Harvard Graduate School of Education. Weissman, M., & Klerman, G. (1977). Sex differences and the epidemiology of depression. Archives of General PsychiatT, 34, 98-l 11. Wilcox, B. (1981). Social support, life stress, and psychological adjustment: A test of the buffering hypothesis. American Journal of Community Psychology, 9, 371-386. Wolfe, D.A. (1985). Child-abusive parents: An empirical review and analysis. Psychological

Zigler,

Bulletin,

E.F., Weiss,

97(3),

462482.

H.B., & Kagan, S.L. (1984). Resource Guide. New Haven: Yale University.

Programs

to strengthen

families:

A

Reis et al.

262 APPENDIX A of Parameters for Community

Definition Birth

Rates

Risk

Total Area.

Live Births

for specificed

Total Area.

Population

for specified

Index’ Communit! x 1000

Percentage

teen births

Number of Births to Women Cnder 20 Years of Age for specified Community Area. x 100 Total Live Births for specified Communit) Area. Number of Births Community Area.

Prematurity

5 lbs. 8 oz. less for specified

Total Number of Live Births specified Community Area. Excess

.4DC

infant

deaths

Risk

Total number of persons Dependent Children. Profile

Ranking

Non-Ounce

Geographic

Area

Peoria (Peoria Co.)” Kankakee Co.

x 100 for

(all weights)

Excess deaths is the difference between the actual number of infant deaths that occurred and number of infant deaths that ~~1~1 have occurred if the “target rate” of l-1.85 deaths per 1000 live births in 1981 had occurred.

recipients

Community

Communit!

by Geographic Comparison

recei\.ing

Area

Xid to

Four

Ounce and

Sites

Teen birth rate

Excess birth rate

Prematuritv rate

Infant deaths

XFDC rate

Sum

Sum

Rank

Rank

Rank

Rnnl;

Rank

Rank’

Rank:’

83.5 773

52 75

83 92..5

9-f X8.5

18.5 -Hi.5

36 1 .O 380.0

3 2

Rockford Co.)* Sangamon

(Winnebago 73 Co. 86

60.5 58.j

77.5 87.3

9’7 95.5

18 1;

356.0 3i1.5

7 3

Shawnee” C0.Y

(Williamson 31

92

84

ill.5

-k2

338.5

1s

Family Support Programs

Marion Co. Rock Island Co. Saline Co.

263

Teen birth rate

Excess birth rate

Prematurity rate

Infant deaths

AFDC rate

Sum

Sum

100 54.5 95

90

58.5 80.5 67.5

65.5 91.5 48.5

28 46 33.5

342.0 344.5 344.5

12 10.5 10.5

72 100

Shawnee (Franklin co.)* Whiteside Co. LaSalle Co.

43.5 67 42

96.5 40 54

28 58.5 58.5

65.5 74.5 79.5

44 38 43

277.5 278.0 276.0

36 35 37

Shawnee Uackson co.)* Crawford Co. Randolph Co.

28.5 28.5 30

34 41 58.5

52.5 77.5 23

59.5 59.5 72

41.5 9.5 32.5

216.0 216.0 216.0

61 61 61

Lawndale (Chicago)* W. Garfield Park Fuller Park

66.5 70 58

71 70 76

54.5 70 68

65 57 72.5

73 65 34

293.5 299.5 219.5

9 8 10

Note: The potential range of the risk indices varied according to the geographic unit (i.e., county, city, or Chicago community area). Thus for Peoria, Rockford, and Shawnee as well as their matched communities, rankings within each columns 1 through 5 and for Aurora and its match, rankings could range from 1 to 25 for all cities over 50,000 population. Finally, for Lawndale and Near North as well as their matched communities, rankings could range for each of the five parameters, the higher the number, the greater the value of the index. *Ounce of Prevention site areas, listed adjacent to the Ounce sites, are the respective matched area(s). ‘The community risk index is adapted from a maternal and child health index developed by the Chicago Department of Health. ‘The hzgher the sum rank score in column six, the greater the level of community risk. sThe lower the risk rank in column seven, the greater the level of community risk. Thus, an inverse relationship exists betwen the sum rank (column 6) and the risk rank (column 7). “The Ounce demonstration service area for Shawnee was a three county area (Williamson. Franklin, and Jackson).