Political socialization and policy evaluation: The case of youth employment and training program

0149-7189/87 $3.00 + .OO Copyright Q 1987 Pergamon Journals Ltd

Evoluotion ondProgrom Planning, Vol. 10, pp. 35-42, 1987 Printed in the USA. All rights reserved.

POLITICAL SOCIALIZATION AND POLICY EVALUATION: THE CASE OF YOUTH EMPLOYMENT AND TRAINING PROGRAM BRIANP. NEDWEK St. Louis University

ABSTRACT The detection and assessment of side effects or unintended consequences in policy evaluation are seldom conducted in other than a casual manner. This paper investigates the unintended impact of an employment and training program on the political orientations of participants. Applying a political socialization framework, this study compared the postprogram attitudes toward citizenship of 286 CETA-eligible youth with a randomly assigned controlgroup of 161 youth. Entering minority status and gender as factors and preprogram attitudes toward citizenship as the covariate, the ANCOVA results indicate that political orientations of experimentals, especially program completers, are significantly higher compared with control group members. tion model in the field of youth employment and training may be enhanced alienation on the part of the client. As one observer of employment programs commented:

Socialization theory suggests that political orientations are the composite of attitudes, beliefs, values, and behaviors that are learned and passed from generation to generation by socialization agents (Almond & Verba, 1965; Dawson & Prewitt, 1969). While basic political orientations are established early in life by the strong influence of primary and secondary relationships (Easton & Dennis, 1969; Merelman, 1971), socialization experiences continue to penetrate the civic orientations of citizens throughout life. For example, encounters with police officials, school personnel and, among others, parapolitical clubs provide a rich source of political learning for adolescents (Beck, 1977; Beck & Jennings, 1982; Hess & Torney, 1967; Jennings, 1980; Merelman, 1980; Sigel8z Hoskins, 1981). An interesting and underresearched question is whether youth employment and training programs, as agents of political socialization, impact client political orientations. Socialization experiences may occur when the program or intervention itself provides learning stimuli in the form of material reinforcements that are ultimately converted into or replaced by symbolic ones (Merelman, 1966). Thus, for example, an unintended consequence that flows from a weak interven-

[I]f training proves to be essentially irrelevant or useless, and the client has taken the promises of the agency seriously, the main consequences will be an increase in aliena-

tion. (Archibald, 1968) Similarly, it has been illustrated that difficulties in overcoming early socialization to hierarchical role relations between staff and youth are not insignificant. These socialization incongruities were documented recently in a CETA evaluation (Fuller & Rapoport, 1984). In a related way, this study hoped to demonstrate changes in political attitudes as unintended consequences of program involvement. On the basis of the socialization literature, program participants (especially those who completed the treatment) should display a greater change in political attitudes when compared to their control group counterparts. Furthermore, changes in political attitudes should be more pronounced among early positive terminees compared to early negative terminees.

Requests for reprints should be sent to Brian P. Nedwek, PhD, Professor of Urban Affairs and Public Policy Analysis, St. Louis University, 221 North Grand Blvd., St. Louis, MD 63103.

35

36

BRIAN P. NEDWEK METHODOLOGY

The U.S. Department of Labor subcontracted a process and impact assessment of a vocational exploration demonstration project that was implemented in 13 cities across the nation. The project, known as the Vocational Exploration Demonstration Project (VEDP), enrolled 2,930 CETA-eligible youth in a preemployment and work maturity program. Depending upon the curriculum mix, participants received 320 or 400 hours of service.’ While each program applied the same eligibility criteria during the 1980-81 academic year program cycle, a classic experimental design was implemented in six of the thirteen sites, the remaining sites had neither control nor comparison groups. Experimental and control groups were established in the following manner:

In addition to the traditional measures of outcome assessment of employment and training programs, e.g., labor market successes, changes in knowledge of and attitudes toward the world-of-work were assessed. The Educational Testing Service (ETS) provided pretest and posttest surveys for measuring changes in seven areas that included: (1) vocational attitudes, (2) job knowledge, (3) job holding skills, (4) work relevant attitudes, (5) job seeking skills, (6) self-esteem, and (7) sex stereotyping of adult occupations. The survey instrument was administered in a group setting and re-

quired approximately one hour and fifteen minutes to complete. The outcome variable for this study is a citizenship index. This summated index was constructed on the basis of responses to five items selected from the ETS test battery. Of the seven subscales in the test battery, these five items were selected on the basis of explicit references to political objects, e.g., citizenship, judges, police, and the law. The first two items were taken from the Work Relevant Attitudes Inventory subscale and the remaining were part of the Self-Esteem battery. The latter subscale used pictorial scenarios to complement the items. The items and response categories appear in the appendix. Inter-item correlations were obtained for (1) individual items, (2) each item and the Citizenship score, and (3) two additional items for a predictive validation with the Citizenship lndex.3 The analysis yielded reasonably strong item-to-index correlations and modest correlations between the additional items and the index.4 The following analysis tests the influence of four independent variables. The first is group status, i.e., membership in the experimental or control group, and is used to assess treatment effects. The second variable is gender of the client. Previous research has documented sex differences for a variety of political orientations and behaviors (Milbrath, 1972; Verba & Nie, 1972). Conceptually, gender may be viewed as a kind of categoric group membership that provides perceptual filters through which political experiences are internalized. The third variable, race, has been used extensively in the analysis of political orientations. For this analysis, categoric group membership in one of three was used. The types were (1) white, (2) black, and (3) hispanic. While other racial and ethnic groups were in the data base, the limited number of such groups as Native American or Asian prohibited their inclusion in the analysis. The fourth independent variable was the type of termination status from the program. Termination before the end of the program cycle may have occurred because of an administrative separation, re-entry or return to school, entry into the labor market, personal illness or a variety of reasons. Each participant who

‘It should be noted that participants were compensated at the minimum hourly wage as a stipened for their participation. In addition, participants were eligible to receive supportive services during their tenure in the program. These supportive services included the traditional array of services provided to CETA clients. *While the random assignment process was successfully implemented, participants were added to the experimental group in a few sites when it became apparent that minimum slot allotments would not otherwise have been achieved.

‘Two items were used for assessing predictive validity. These items were selected because of their use in most efficacy and trust measures. The literature has well-documented the relationship between efficacy, trust, and general political orientations. 4The item-to-index correlations ranged from .26 to .70. Among experiments and controls, the two predictive validity items correlated with the index at .24 and .23. See the appendix for the entire interitem correlation matrix.

1. Program operators conducted an interest interview with all eligible applicants to develop a pool of up to two and one-half times the slot level; 2. Of those expressing an interest, the research agent drew a random sample and randomly assigned a youth to either a program slot or a control gr0up.l. 3. Both experimentals and controls completed a pretest battery and reading test and were notified immediately upon completion of the two test batteries of their status; 4. Program operators were responsible for the notification of the control group to participate in a posttest battery and a modified program completion survey; and 5. Control group members were compensated at a rate of $10.00 for their participation in each testing episode (Center for Urban Programs, 1982).

37

Political Socialization and Policy Evaluation variable, experimental/control as the explanatory variable, ethnic group as factors.

terminated early from the program was assessed a termination status by program personnel. The reason for termination was classified in this analysis as positive, neutral or negative.5 The data analysis routines included the following steps:

group membership and gender, race of

Folding-in the experimental design sites with other sites did yield significant pretest differences. For those analyses, the pretest was added as a covariate. Though only the data from the combined experimental sites are shown here, these data analysis routines were followed to strengthen the appropriateness of the ANCOVA approach. As has been recently recommended, the analyst should “delineate a logical rationale for the selection of variables as candidate covariates, preferably based on an overall theoretical framework” (DiConstanzo & Eichelberger, 1980).

1. Using the experimental design sites only, an analysis of variance of the Citizenship Index pretest experiments and controls was run for each site and for the combined sites; 2. Because significant differences in pretest scores were not found, the analysis of variance was performed for each group and for all the sites together, using the Citizenship Index posttest as the dependent

RESULTS Table 1 presents the analysis of variance of the Citizenship Index posttests. Group membership, race and gender were added as factors. In this phase of the analysis, experiments included both program completers and early terminees. In reviewing the main effects, it can be seen from Table 1 that significant treatment differences were detected (F = 5.958, p = .015). While gender differences were significant (F = 4.972, p = .026), racial characteristics were not statistically related to posttest scores. However, both two-way and three-way interactions yielded insignificant gender/factor combinations. The multiple classifications analysis (MCA) shown in Table 2 reports only 2.8 percent of the variance in

ANALYSIS

OF VARIANCE

the Citizenship Index posttest score is explained by treatment, race, and gender. Table 2 displays the unadjusted deviations from the grand posttest mean and the deviations adjusted for the factors. The results show program participants with substantially higher posttest mean scores compared to the control group. Adjusting for the factors, the Citizenship Index posttest means are 4.01 (3.97 + .04) and 3.89 - .08), respectively. Table 2 also shows that females held higher posttest scores than their male counterparts. The differences lessened when adjustments were made for the factors. Next, it was suspected that political attitudes toward citizenship would be more positive among program

TABLE 1 OF THE CITIZENSHIP

Sum of

Source

of Variation

Main Effects Group Status Race Gender P-way Interactions Group Status X Race Group Status X Gender Race X Gender 3-Way Interactions Group Status X Race X Gender Explained Residual Total

Squares

INDEX

POSTTEST

Mean

DF

Squares

F

Significance

3.343 1.567 0.453 1.306 0.756 0.049 0.152 0.499 0.570

4 1 2 1 5 2 1 2 2

0.836 1.567 0.226 1.308 0.151 0.025 0.152 0.249 0.285

3.177 5.958 0.861 4.972 0.575 0.093 0.577 0.948 1.083

,014 .015 ,424 .026 .719 ,911 .440 .388 .340

0.570 4.668 114.400 119.076

2

0.265

1.083

11 435 446

0.424 0.263 0.267

1.614

.340 .092

sTwenty-two different termination codes were possible. Typical positive reasons included (1) found a job. (2) entered military service, (3) entered academic or vocational school. (4) entered a training program, or simply (5) completed the program. Neutral termination

codes included reasons like (I) health, (2) pregnancy, (3) family or home responsibilities, or (4) transportation limitations. Negative terminations disproportionately included administrative separations, i.e., removed from the program because of conduct code violations.

38

BRIAN P. NEDWEK

MCA RESULTS: AGAINST Variable and Category

Group Status 1 Control 2 Experimental

TABLE 2 GROUP STATUS, RACE AND GENDER CITIZENSHIP INDEX POSTTEST Unadjusted Deviations

N

161 286

Eta

- 0.07 0.04

Adjusted for Independents

BETA

- 0.08 0.04 0.10

Race 1. White 2. Black 3. Hispanic

132 280 35

- 0.04 0.03 - 0.05

0.12 - 0.04 0.02 -0.06

0.07 Gender 1. Male 2. Female

- 0.08 0.04

154 293

0.06 -0.08 0.03

0.11

0.11

Multiple R Squared = 0.028 Multiple R = 0.168 Grand Mean = 3.97

tors. Significant main effects were found for progam completion status. Table 3 shows that completer/early terminee differences were significantly related to citizenship attitudes (F = 5.172, p = .024). The MCA shown in Table 4 reports that adjusting for the factors, program completers had a higher Citizenship Index posttest than early terminees. Completers had a mean Citizenship Index posttest score of 4.06, while early terminees averaged 3.92. When program completers were compared with early positive or neutral terminees and early negative terminees, an interesting pattern emerged. As expected, early negative terminees had a considerably lower

completers compared to early terminees, especially the early negatives terminees. To test this notion, analysis of variance of completer posttests against early terminees were performed in two steps. First, program completers were compared with early terminees. Next, a constructed termination status variable was developed. Termination status was trichotomized into (1) terminated at the end of the component, i.e., a progam completer, (2) early positive or neutral termination, and (3) early negative termination. The Citizenship Index postest score was entered as the dependent variable and the analysis of variance procedure was used with race and gender as added fac-

ANOVA

Source

of Variation

Main Effects Completion Status Race Gender 2-Way Interactions Completion Status X Race Completion Status X Gender Race X Gender 3-Way Interactions Completion Status X Race X Gender Explained Residual Total

TABLE 3 RESULTS: COMPLETION STATUS, RACE AND GENDER AGAINST CITIZENSHIP INDEX POSTTEST Sum of Squares

Mean DF

3.081 1.209 0.112 1.106 1.543

Squares

F

0.770 1.209 0.056 1.106 0.309

3.294 5.172 0.239 4.732 1.320

.012 .024 ,787 ,030 ,256

Significance

0.573

2

0.287

1.226

.295

0.222 0.689 0.274

1 2 2

0.222 0.345 0.137

0.952 1.474 0.586

.330 .231 .557

0.274 4.897 64.056 68.953

2 11 274 285

0.137 0.445 0.234 0.242

0.586 1.904

.557 ,039

Political Socialization and Policy Evaluation

39

TABLE 4 MCA RESULTS: COMPLETION STATUS, RACE AND GENDER AGAINST CITIZENSHIP INDEX POSTTEST Variable and Category

N

Completion Status 1 Completer 2 Early Terminee

188 98

Unadjusted Deviations

Eta

Adjusted for Independents

0.05

0.06 -0.11

-0.09 0.16

Race 1 White 2 Black 3 Hispanic

94 168 24

- 0.05 0.03 -0.03

0.14 - 0.02 0.02 0.05

0.08 Gender 1 Male 2 Female

103 183

BETA

-0.10 0.06

0.04 -0.08 0.05

0.15

0.13

Multiple R Squared = 0.045 Multiple R = 0.211 Grand Mean = 4.01 TABLE 5 MCA RESULTS: TERMINATION STATUS, AND GENDER AGAINST CITIZENSHIP INDEX Variable and Category

Termination Status 1 Completer 2 Early Positive or Neutral 3 Early Negative

N

Unadjusted Deviations

188

0.06

37 61

- 0.06 -0.14

Eta

RACE POSTTEST Adjusted for Independents

0.05 -0.03 -0.13 0.14

0.17 Race 1. White 2. Black 3. Hispanic

94 168 24

-0.02 0.02 0.05

- 0.05 0.03 -0.03 0.08

Gender 1. Male 2. Female

103 183

BETA

-0.10 0.06

0.05 - 0.08 0.05

0.15

0.13

Multiple R Squared = 0.047 Multiple R = 0.217 Grand Mean = 4.01

posttest average compared to completers (3.88 and 4.06, respectively). Table 5 shows that early positive or neutral terminees were little different compared with

the early negative terminations. However, the overall effect of termination status was not statistically significant (F = 2.930,~= .055).

DISCUSSION This study, guided by a political socialization analog and the substantive character of some items taken from a multivariate test battery, demonstrated that changed attitudes toward citizenship may be a side effeet of government employment and training programs.

While there exist a few notable exceptions (Klarman, 1974; Ribich, 1968; Striven, 1972), the measurement of spillover effects or unintended consequences seems to occur in a casual, serendipitious manner. The present study demonstrated some unintended consequences

BRIAN P NEDWEK

40

that became salient when concepts from the literature on political socialization were used. Some additional issues merit attention. First, it should be noted that ethnic characteristics failed to account for differences in citizenship attitudes. Program completers, regardless of personal characteristics, were more favorably disposed toward citizenship than were those not exposed to the treatment. Second, there is a matter of interpretation. While completers had a higher mean Citizenship Index posttest, perhaps the results are due to self-selection. For example, the pretest mean differences among completers, early positive terminees and early negative terminees were assessed using the Turkey a posteriori contrast procedure. While no significant differences in the pre-test means among early positive or negative terminees and program completers was found, both were somewhat higher when compared to early negative ter-

minees. The posttest increased for each group. More important, the adjusted posttest means between completers and early positive terminees were not significantly different. While self-selection could account for termination variability, i.e., those with higher pretest scores may be more efficacious and thus less likely to terminate, it would not account for experimentallcontrol group differences, Finally, the measurement of termination status is prey to program delivery agents. As with most multisite program implementations, variability in the application of administrative rules, e.g., administrative separation of a client, clouds the interpretability of findings. In some sites, assertive client behavior is rewarded, while in others, such behavior may be interpreted as aggressive. This validity problem is further compounded by the occurrence of staff turnover within sites.

CONCLUSION This study demonstrated how the literature on political learning can be used to structure the assessment of unintended consequences of government training programs. This study also demonstrated an empirical link between participation in an employment and training program and client political attitudes. It is hoped that the exploratory work reported here will serve as a baseline for future investigations of political learning. Finding changed citizenship attitudes to be an apparent product of government-sponsored training programs may be a refreshing departure from the litany of “no program effect” in the CETA evaluation literature.& These findings also have implications for more rigorous process assessment components in the evaluation strategy. Questions about what specific socialization agents are at work in the program should be explored. Little is known about what conditions and processes influence political learning. Finally, unintended consequences are part of a growing set of ambiguous and overlapping terms that describe unanticipated program outcomes. Secondary,

external, spillover or side effects are used interchangeably in the policy analysis literature. The primary area of agreement is that these phenomena are difficult to discover and measure (Rossi &z Freeman, 1982; Schneider, 1982). Technical as well as political factors have inhibited systematic examinations of program impacts other than those intended. It has been noted that “it is an open question . . . whether the problem for concern is the lack of measurement of external effects, or the tendency by administrators and others . . . to exaggerate their likely importance” (Cain & Hollister, 1972). These externalities are labeled as either beneficial or detrimental, depending upon the value complex or normative orientation of key stakeholders in the policy process, This research hopes to stimulte further investigations of methods for detecting side effects. Dumping unintended consequences into the error term does little to stimuiate the development of competing models or theories that undergird treatments. These results suggest the need for a more proactive stance in the search for actual effects of public policies.

REFERENCES ALMOND, G. A., & VERBA S. (1965). The civic culfure. Boston: Little, Brown. ARCHIBALD,

K. A. (1968). Unpublished

menting on the problems of evaluating manpower training programs. In E. S. Quade, Analysis for public decisions. New York: North Holland.

Rand trip report com-

6The evaluation of the Vocational Exploration Demonstration Project yiefded some interesting findings. First, there was an overall treatment effect on attitudes toward and knowledge of the world of work. Significant differences between program completers and controls were found in Vocational Attitudes, Job Holding Skills, Work Relevant Attitudes, Job Seeking Skills, reduced Sex Stereotyping, and Self Esteem. Second, while males continued to be far less likely

to be “at risk,” i.e., out of school or out of work, three months after program closeout than females, the differences were more pronounced among those who had demonstrated pre-employment or work maturity skills. Finally, minority participants remained twice as likely as whites to be outside the labor market, regardless of preemployment or work maturity skills.

Political

Socialization

and Policy Evaluation

41

BECK, P. A. (1977). The role of agents in political socialization. In S. A. Renshon (Ed.), Handbook of political socialization. New York: The Free Press.

KLARMAN, H. E. (1974). Application of cost-benefit analysis to the health services and the special case of technologic innovation. Journal of Health Services, 4, 325-352.

BECK, P. A., & JENNINGS, M. K. (1982). Pathways to participation. American Political Science Review, 76, 94-108.

MERELMAN, R. M. (1966). Learning and Legitimacy. American Political Science Review, 60, 548-561.

CAIN, G. G., & HOLLISTER, R. G. (1972). The methodology of evaluating social action programs. In P. H. Rossi and W. Williams (Eds.), Evaluating social programs: Theory, practice, and politics (pp. 109-137). New York: Seminar Press.

MERELMAN, R. M. (1971). Politicalsocialization and educational climates: A study of two school districts. New York: Holt, Rinehart and Winston.

CENTER FOR URBAN PROGRAMS (1982). The vocational exploration demonstration project: An analysis of the long term of VEDP II. Final report: Part A. A report submitted to the U.S. Department of Labor, Employment and Training Administration, Washington, DC. DAWSON, R., & PREWITT Boston: Little, Brown.

K. (1969). Political socialization.

DICOSTANZO, J. L., & EICHELBERGER R. T. (1980). Reporting ANCOVA results in evaluation settings. Evaluation Review, 4, 419-450. EASTON, D., & DENNIS J. (1%9). Children in the political system. New York: McGraw-Hill. PULLER, B., & RAPOPORT T. (1984). Indigenous evaluation: Distinguishing the formal and informal organizational structures of youth programs. Evaluation Review, 8, 25-44. HESS, R. D., & TORNEY J. V. (1967). The development ofpolitical attitudes in children. Chicago: Aldine. JENNINGS, M. K. (1980). Comment on Richard Merelman’s ‘democratic politics and the culture of american education’.” American Political Science Review, 74, 333-337.

MERELMAN, R. M. (1980) Democratic politics and the culture of american education. American Political Science Review, 74, 319-332. MILBRATH, L. W. (1972). Political participation. Chicago: Rand McNally. RIBICH, T. I. (1968). Education and Poverty. Washington, Brookings Institution.

DC:

ROSSI, P. J., &FREEMAN H. E. (1982). Evaluation: A systematic approach. Beverly Hills: Sage Publications. SCHNEIDER, ANNE L. (1982). Studying policy implementation: A conceptual framework, Evaluation Review, 6, 715-730. SIGEL, R., & HOSKINS M. (1981) The political involvement of adolescents. New Brunswick, NJ: Rutgers University Press. SCRIVEN, M. (1972). Pros and cons about goal-free evaluation. Evaluation Comment, 3, l-8. VERBA, S., & NIE N. H. (1972). Participation in America. New York: Harper and Row.

APPENDIX: INDEX ITEMS AND RESPONSE CATEGORIES Item 2: “It is hard to get ahead without breaking the law now and-then.” [l] Strongly agree, [2] somewhat agree, [3] somewhat disagree, [4] strongly disagree. Item 2: “Would you say that your chances of becoming a respected and law abiding member of your community are: [4] excellent, [3] reasonably good, [2] not very good, [l] very unlikely.” Item 3: “How do you think you’d make out if you were in court? [3] The judge would probably go easy with me - I’m worth giving a chance, [2] I would probably get a small fine- but nothing too bad, (11 I would probably get the worst possible punishment there is.” Item 4: “What would you do if someone came to your door and said this? [3] I would register and vote-it’s the only way I can make things better, [2] I’d register and vote-but It’s not much use for improving things, [l] No sense in even

bothering- my vote doesn’t change what happens to me.” Item 5: “How do you think you would make out if you were one of the people the cop was talking to? [3] If I didn’t do it, the cop wouldn’t blame me, [2] The cop might blame me-but I don’t think I would get in trouble if it wasn’t my fault. [l] I’m usually the one who would get blamed and arrested even if I didn’t do anything.” Item 6: * “You feel that you have little influence over the things that happen to you. [l] strongly agree, [2] somewhat agree, [3] somewhat disagree, and [4] strongly disagree.” Item 7:* “Most people cannot be trusted. [l] strongly agree, [2] somewhat agree, [3] somewhat disagree, and [4] strongly disagree.” *Items used for validation only.

BRIAN P. NEDWEK

APPENDIX: CORRELATIONS INDEX

INTER - ITEM OF CITIZENSHIP PRETEST Validation Items

Item 1 Item 2 Item 3 Item 4 Item 5 Index index

Item 2

Item 3

Item 4

Item 5

Item 6

Item 7

.Ol (656) .lO’ (654) .Ol (654) .51’ (647) 51 (377)

.02 (659) .07* (659) .26’ (647) .24’ (377)

.06 (657) .50’ (647) .51 (377)

.46’ (647) .47’ (377)

.24’ (641) .22’ (373)

.23’ (645)a .24’ (376)b

.22 (656) .04 (662) .14’ (660) .02 (660) .70’ (647) .70’ (377)

l

l

‘Significant at the .05 level. aIncludes experimental and control group members. blncludes only experimental group members.