- Email: [email protected]
Using Self Reports to Measure Program Impact*
Pergamon
@
Children andYoutiServices Review, Vol.19,No.7,pp.567–585, 1997 Copyright @1997EkvierScience Ltd Printed intheUSA.Allrightsresewed 019C-740%97 $17.00+.00 PII S0190-7409(97)00046-7
Using Self Reports to Measure Program Impact Freya L. Sonenstein The Urban Institute In order to demonstrate success, pregnancy prevention programs must show that their participants have changed their behavior. Accurately measuring levels of sexual activity and contraceptive use are important to determining program efficacy. These measures can be reliably obtained, but careful attention must be given to what is measured and how. In this paper, based on our experience conducting the National Survey of Adolescent Males, we offer several guidelines for measuring program effects on behavior. Successful evaluations should accurately describe program participants, including who is being targeted, who has participated in other programs, and who has already engaged in the behaviors that the program is intended to prevent; measure program outcomes by program logic and goals; employ research protocols that ensure strict confidentiality and the effective administration of surveys; and double-check responses to see that they correlate with other available evidence.
A recent review of research on programs to prevent teenage pregnancy concludes that although many programs have been created to reduce the incidence of teen pregnancy, few have been rigorously evaluated (Kirby, 1997). One of the common weaknesses of the studies reviewed is their failure to measure the effects of the program on behavior. In this paper, I describe what we have learned about measuring the sexual behavior of adolescents from our experience conducting the National Survey of Adolescent Males (NSAM). By providing practical advice on how to measure sexual behavior, I hope to encourage programs and their evaluators to document whether their programs produce behavioral change. Portionsof this articlearebasedon a paperpresentedat the conferenceResearchingSexual Behavior:MethodologicalIssues,whichwasheld on April26-28,1996,at the Universityof Indiana.The NationatSurveyof AdolescentMates has been supportedby grants from the NationatInstituteof ChildHealthand HumanDevelopment. Reprintsmaybe obtainedfromFreyaSonenstein,Ph.D.,Director,PopulationStudiesCenter, TheUrbanInstitute,2100M Street,N.W.,Washington,D.C.20037 [[email protected]]
567
568
Sonenstein
This discussion is particularly timely because the federal government is about to make a substantial new investment in teenage pregnancy prevention. Recent federal welfare reform legislation (P.L. 104-193) authorizes $50 million annually for states to implement prevention programs that promote sexual abstinence among unmarried teenagers. If states opt to use these funds, the matching-fund requirements will swell the resources devoted to abstinence education to almost half a billion dollars over the next five years. This sum does not include the $7.6 million dollars also authorized annually under the Adolescent Family Life Act. For the first time, significant federal and state funds will be invested in abstinence programs for teenagers (U.S. Department of Health and Human Services, 1997a;U.S. Department of Health and Human Services, 1997b). Unfortunately,no provision has been made in the legislation for program evaluation. This omission is particularly troubling because evidence on the effectiveness of abstinence programs is quite sketchy. Only a handful of abstinence programs have been evaluated. None of the published studies have demonstrated reductions in sexual activity levels, but each study suffers design flaws that prevent conclusions about either positive or negative effects (Kirby, 1997; Moore et al., 1995; Frost & Forrest, 1995; Miller & Paikoff, 1992). Given the forthcoming substantial investment of taxpayers’funds in these programs, it would be wise to focus research efforts on assessing whether abstinence and other types of prevention programs change teenagers’ behavior and reduce unintended pregnancies. To that end, this paper will describe lessons learned from our experience conducting the NSAM. First, I will discuss how to make decisions about what to measure within a program evaluation context. Then, I will turn to how to measure sexual behavior among teenagers. Choices about What to Measure
The measurement strategy for any program evaluation should be tailored to the theory of behavior that underlies the program approach. Programs are implemented because someone thinks that the intervention will achieve desired changes in behavior among a particular group of people. The form the intervention takes rests on assumptions about how and why people behave in the way they do. These assumptionsmay be expressed in a formallydeveloped theory of behavior such as the theory of reasoned behavior described below, they may bean experienced practitioner’s best guess about approaches likely to change the behavior of the target population, or they maybe some combi-
Using Self Reports
569
nation. Whatever form they take--formal or informal-- an understanding of such assumptions is critical for designing an evaluation and measurement strategy.The logic of the program rests on those assumptionsthat pinpoint the program levers expected to produce behavior change in participants. An evaluation tests whether the program logic works and whether participants’ behavior changes as a result. An example of a formal theory of behavior that has influenced many prevention programs is Fishbein and Ajzen’s theory of reasoned behavior (1975). In this model a behavior, like the decision to have intercourse for the first time, is seen as a product of a process in which various fainily and background characteristics lead teens, first, to hold particular attitudes about the consequences of having sex and, second, to perceive that people who are important to them have expectations about how they should behave. Together, these attitudes and normative expectations create an intention to actor not to act, which, in turn, leads to the actual behavior.] Figure 1 illustrates the major concepts in the theory. A program based on this theory will attempt to change behavior by trying to alter teenagers’ attitudes toward the consequences of sexual behavior and their perceptions about what people close to them expect. To assess whether the program has achieved its desired results, measures are needed of whether the program has produced changes in those attitudes and normative expectations as well as whether the participantschange their sexual behavior. This assessmentchecks whether the program logic has operated as expected and whether more participating teenagers are behaving in conformity with the program objectives. Thus, a program evaluation requires not just measures of behavior change but measures of the intermediate processes that produce those changes. The figure illustrating the theoretical model also shows that the characteristics of the programs’ participants are important to consider. Individuals enter programs with an initial set of attitudes and expectations that primarily reflect the families and the communities in which they live. It is important to record the characteristicsof the participants, because many programs target particular types of teenagers. A program effective with one type of teenager may not necessarily be effective with another. Therefore, a thorough description of program participants is important for understanding what types of programs work with particular types of teens. 1 For ease of presentationthe Fishbeinand Ajzenmodelhas been strippeddownto its most essentialelements.Readersare urgedto consultsourcedocumentsfor a fullerunderstanding of its complexity.(Fishbein& Ajzen, 1975;Fishbeinet al., 1991).
Sonenstein
Figurel
Simple Conceptual Model Example
Backgroundand FamilyCharacteristics
1
,
It is also important to know whether the program participants have participated in other interventions. Data from NSAM shows how crucial those measures can be. In 1988, almost all 15- to 19-year-old males in the U.S. reported receiving formal instruction in school about sexual topics and 58% reported having been taught “how to say no to sex” (Ku, Sonenstein and Pleck, 1992). Depending on the age of the program participants and the availability of other resources in the community, it is quite possible that program participantshave already experienced another intervention.This prior intervention may have influenced their attitudes and expectations. For these participants, the evaluation does not measure the effects of a single program on behavior but the effects of that program plus the effects of the other programs in which the participants have been involved. In addition, some participants maybe sexually active even though a program may be aimed at getting teens to delay the initiationof sex. In 1988, onethird of 15-year-old males reported that they had had intercourse, but there were substantial differences by ethnicity. More than two-thirds of black 15to 19-year-old males were sexually experienced, compared with one-third of Hispanic males and one-quarter of white males (Sonenstein, Pleck and Ku, 1989). Some program participants may have already engaged in the behavior targeted by the intervention. Indeed, in 19883890 of the 15- to 19-year-old
Using Self Reports
571
males reported that they had first received instruction about “how to say no to sex” afler they initiated sex (Ku, Sonenstein, & Pleck, 1992). It is therefore important to collect information about the program participants to be able to describe those targeted for the intervention, to differentiate those who have experienced other interventions, and to understand whether some of the participants have already engaged in the behaviors the program is trying to prevent. In addition, one would also want to measure how much of the program intervention the participants receive. Some teens attend every session, while others may attend sporadically. One would expect that those receiving more of the intervention will experience a greater change in attitudes, expectations, and behavior. This proposition can be tested if measures of program dosage are collected. The measures of program outcomes should be defined by the program logic. One would expect an abstinence program based on the theory of reasoned action to produce changes in participants’ attitudes toward the consequences of sex and in perceptions of normative expectations. These changes, in turn, would result in shifts in intentions to engage in sex. Finally, participants should be observed delaying the onset of sexual intercourse. While measures of all these concepts are needed to test the program logic, many evaluations of abstinence programs fall short because they measure shifts in mediating concepts like attitudes, expectations or behavioral intentions but neglect to measure actual behavior (Kirby, 1997). It is useful to know whether a program produces changes in attitudes, expectations, and behavioral intentions, but the bottom-line question is whether the program produces actual changes in behavior. To get at this question, the evaluation needs to ask teenagers about their sexual behavior. The choice of questions about sexual behavior should be tightly linked to the program logic. Some programs aim to delay sex until teens are older; some aim to delay sex until marriage. Some work with teens who have already had sex to get them to stop having sex or to reduce their number of partners. A few may promote alternative forms of sexual expression. Some emphasize better communication strategies between partners. Some focus on consistent contraception. The behavioral measures selected for evaluations of particular programs should reflect the behaviors targeted for change. These probably encompass more than just whether program participants have initiated vaginal intercourse by the end of the program. Depending on the program objectives, measures may be needed of the number of months participants delay initiating sex ●
572
Sonenstein
whether participants delay sex until marriage the number of sexual partners participants have ever had the number of sexual partners participantshave had in some recent period of time like the past three months the number of simultaneous sexual relationships the participant has ever had or had in some recent period of time communication strategies used by romantic partners other forms of sexual expression that participants engage in contraceptive methods used at last intercourse consistency of contraceptive method used in a given period of time, like the past three months or the past year We are not advocating that programs use all these measures. They should choose measures related to their program objectives. Programs that aim to delay sexual activity until marriage will need designs that assess participants’ sexual behavior prospectively until they marry. Programs that have shorter time frames can rely on shorter follow-up periods. It should be noted, however, that 12 months after program completion is generally considered a minimum follow-up period for evaluations of prevention programs, according to the guidelines established by the Program Archive on Sexuality and Adolescence (Sociometrics, 1997b). Programs that emphasize monogamy will probably select more measures of the types of sexual partnerships that participants enter. Programs that teach communication strategies for couples will need measures of those behaviors. Programs that include discussions of contraception will want to include measures of contraceptive behavior. In the development of a measurement strategy for a program evaluation, the main points to remember are Choices need to be made not just about the outcome measures but about the full range of concepts that the program logic defines as relevant. It is not enough to focus just on outcome behaviors. Measures are also needed of participants’ baseline characteristics, the major concepts—like attitudes and normative expectations-seen as affecting behavior, and the level of program participation. It is also not enough to measure attitudes or intent to behave, as many evaluations do, without examining whether behavioral changes actually occur. The choice of sexual behaviors to be measured should reflect the program logic and the types of behaviors targeted for change. ● ● ●
9
● ● ● ●
9
●
●
●
Using Self Reports
573
How to Measure Sexual Behavior
Having talked about the choices of what to measure, I now turn to what we have learned about how to measure sexual behavior. Measuring sexual behavior is inherently difficult because respondents are being asked to se~report on sensitive behavior. Measuring any behavior using self-reporting is challenging, but measuring sexual behavior poses special problems. The primary challenges of measuring sexual behavior among teenagers are that ●Sexual behavior is viewed as a private matter in our culture. There are strong norms about not discussing one’sown behavior with other people, especially with strangers. Respondents may be reluctant to answer these questions, or they may refuse to answer them, either actively or passively, by denying that they are engaging in the behavior. Sexual behavior among teenagers may also be tinged by social desirability issues. Teens may be reluctant to admit to engaging in behaviors that are disapproved of. For prevention programs, this issue may be particularly important if program participants do not honestly report their actions out of fear of disapproval. The corollary is that teens may be too eager to admit to engaging in behaviors that are socially approved. For reasons of both privacy norms and social desirability,estimates of the prevalence of sexual behaviors among teenagers maybe biased. Many sample surveys of U.S. teenagers conducted since the early 1970s have included measures of sexual behavior and have therefore dealt with these challenges. These surveys have included nationally representative samples of teenagers like the studies by Sorensen (1973), Zelnik and Kantner (1980), Mosher and Bachrach (1996), Sonenstein, Pleck, and Ku (1989), and Furstenburg, Moore, and Peterson (1985), as well as the new Adolescent Health Study, led by Richard Udry at the University of North Carolina. Studies of teenagers representing state or community populations have also been conducted, including those by Miller and Simon (1974) in Illinois, Vener and Stewart (1974) in three Michigan communities, Jessor and Jessor (1975) in a Rocky Mountain city, Udry and Billy (1987) in a North Carolina community, and Hingson, Strunin, Berlin, and Heeren in Massachusetts (1990). Many multipurpose surveys of teenagers, like the National Longitudinal Survey of Youth (Mott, 1986), High School and Beyond (Hanson, Morrison, & Ginsburg, 1989), and the Youth Risk Behavior Survey (CDC, 1995), include at least some questions about sexual behavior. These studies have developed approaches to address the problems associated with collecting information ●
574
Sonenstein
about sensitive behaviors. Moreover, this significant body of work demonstrates that teenagers can be asked about their sexual behavior. Our experience conducting NSAM, for example, indicates that most teenage males are willing to participate in a survey about reproductive health issues. Our response rates, even with a requirement of parental permission for those under 18, are similar to other household surveys. Three-quarters of all the males aged 15 to 19 identifih in the national household sampling frame participated in the surveysin 1988 and 1995. The follow-up rate was also high for the cohort first interviewed in 1988; we reinterviewed 8970 of them two and a half years later. Evidently, these teenagers were not troubled by their experience with the subject matter covered in the initial interviews. These surveys have also already dealt with many of the issues that an evaluator might face in phrasing questions about sexual behavior for teenagers. The surveys provide a useful source of questionnaire items for program evaluators, once decisions have been made about the evaluation’s general measurement strategy and the aspects of sexual behavior that should be appropriately included in the study. Some of these surveys have used selfadrninistered instruments, and some have relied on in-person interviews. In consultingthose instruments,evaluatomshould be cognizant of the differences between these two modes of data collection.2In selecting the measures to be used in NSAM, we consulted many instruments for collecting data. In addition, we ran many focus groups and qualitative interviews before the actual survey to make sure that teenagers were interpreting the questions in the intended manner (Sonenstein, Ku, & Pleck, forthcoming). A useful source for obtaining many of the instruments used in studies like these is the Data Archive on Adolescent Pregnancy Prevention, which is managed by the Sociometrics Corporation in Los Altos, California (Sociometrics, 1997a). In addition, this organization has published a guide to a minimum data set for evaluating programs aimed at preventing adolescent pregnancy. This guide includes a recommended core questionnaire and supplementary modules for use in evaluations of teenage pregnancy prevention programs (Brindis, Peterson, Card, & Eisen, 1996).
LThisdifferenceis discussedfurtherin a later sectionof this paper.
Using Self Reports
575
Confidentiality, Informed Consent, and Protections
When teenagers are asked to reveal information about sexual behavior, it is crucial that the evaluator pay strict attention to observing guidelines for protecting human subjeets (U.S. Department of Health and Human Services, 1993). One important component of the recommended procedures is to ensure that any information provided will be treated as confidential. Executed properly, such research protocols protect participants against the possibility that private information about them will be released inadvertently. This assurance has the added benefit of encouraging teenagers to provide truthful answers to questions about sensitive behaviors because they know that their answers will be treated confidentially. Providing these protections is a standard part of informed-consent procedures. In so doing, the researcher promises that the respondent’s answers will be seen only by the research staff and that all reports that describe findings will be aggregated in ways that prevent the respondent from being identified. Protective procedures should include storing information by identification numbers rather than by respondents’ names, storing master files and any information with personal identifiers in locked files, and requiring all staff, from interviewers and data gatherers to the dataprocessing personnel, to sign pledges not to release information about respondents to anyone other than authorized staff. In a program evaluation setting, this pledge means that program staff will not have access to the information provided to evaluators by individual program participants. When information is collected about sensitive behaviors like sex, other important aspects of protecting human subjects should also be followed. One such procedure is obtaining informed consent from the study participants. An important component of this process is providing complete information to the study participants about the kinds of questions that will be asked and assurances that they can refuse to answer any question without consequences. If a survey will ask questions about sex, the informed-consent procedures should prepare respondents for those questions. It is also a good idea to provide a rationale for these questions. Respondents need to know that their answers are important, and, when sampling procedures are used, respondents should be told that they represent many other teenagers like themselves. A particular issue for conducting surveys of teenagers is obtaining parental permission. In 1996, the U.S. Congress considered the Family Privacy Act, which would have barred any surveys of minor teenagers on sensitive topics without active parental permission; the legislation failed to pass. The surveys that would have been particularly affected by this legislation are those that
576
Sonenstein
collect data in schools and rely on classroom administration of paper and pencil instruments. Typically, these surveys of students in school use passive parental consent procedures. Parents are notified that the study will be conducted, but they must contact the school if they do not wish their child to participatein the study. Obtaining active consent from parents in these surveys is difficult because students are not very good about transporting permission slips home and then back to the school to the proper authorities. There are multiple opportunities for forms to get lost. Therefore, studies that rely on active consent procedures using the permission-slip approach are likely to have dismal response rates. When response rates are low, the resources put into the research effort are wasted. Active consent procedures are feasible and desirable when teenagers are to be interviewed in person. NSAM used active parental consent procedures for all teenage respondents under the age of 18. The parents of the teens had to sign an informed consent form before their sons were interviewed.The form had five main components: Frankness about the purpose of the study. The parental consent form said that the son had been randomly selected to participate in a survey whose purpose was “to get information about young men’s knowledge, attitudes and behaviors that relate to current health issues, such as alcohol and drug use, sexual behavior, sexually transmitted diseases and AIDS. ” Assurance that permission is required. The parent was told that the son would not be asked to participate if consent were not given. Assurance that the child still needed to provide his own consent. If parental permission were given, the teenager could still decide whether or not he wished to participate in the survey. Having started the interview, he could refuse to answer any question. Assurances about confidentiality. Answers to the interview would be treated as confidential and would not be associated with the son’sname in any reports. Indeed, parents were told that they would not have access to their own children’s answers. Contact names to obtain more information.The parent was given the name and telephone number of the study director and the Institutional Review Board chairperson. Although this form was quite long, study participation was excellent. In 1988, only 6.1% of households with eligible respondents did not participate because a parent withheld consent for his or her minor child; in 1995, this proportion was 5.6%. We believe that these high rates of cooperation are a tribute to the professional skills of the interviewers who did an excellent job ●
●
●
●
●
Using Self Reports
577
conveying the importance of the study. We also think that the high proportion of parents providing permission demonstrates the level of support that parents of teenagers have for public health efforts to address adolescent sexual health issues. A final protection issue for surveys about teenage sexual behavior is making provision for the rare instances in which a respondent reveals that he or she has been raped or sexually abused. Such provisions should include referrals to an appropriate agency for assistance and the reporting of the incident to children’s protective services if the state law requires it. While situations like these are unlikely, procedures should be in place so that staff know what to do if they occur and so that teenagers are supported to reduce the possibility of further harm. Questionnaire Formut
At the core of any survey about sexual behavior is a detailed questionnaire, administered by an interviewer (that is in person or by telephone) or self-administered by the respondent (a written or computer-assisted questionnaire, for example). The arguments in favor of interviewer-based questionnaires are that professionalinterviewers are more capable of handling complex questionnaires and skip logic and that personal rapport between the interviewer and subject can yield greater commitment to the interview process. Furthermore, the interviewer can provide additional explanation or guidance if the respondent has questions. The main arguments in favor of selfadministered questionnaires are that they ensure greater privacy for the respondent and reduce the risk that respondents will be reluctant to give responses to an interviewer who meets with the disapproval of the respondent. Self-administeredquestionnaires are recommended to reduce the respondents’ desire to give socially desirable answers (Bradburn, 1983) and to increase candid responses (Turner et al., 1992). This is particularly important for sensitive subjects, such as sexual behavior. In the 1988 and 1991 NSAM surveys, we elected to use both interviews and self-administered questionnaires. Many of the core behaviors, including most heterosexual behaviors and contraceptive use, were primarily explored in the face-to-face interviews. A paper-and-pencil self-administeredquestionnaire (SAQ) was used for the most sensitivebehaviors including illicit behavior, substance use, and very risky sexual practices. At the end of the in-person interview, which usually lasted one hour, interviewers explained that there were additional questions that we would like the respondent to answer in
578
Sonenstein
private. The respondentswere handed the self-administeredquestionnaire.The front page of the SAQ included assurances of confidentiality and instructions on how to complete the form. The interviewer reviewed these assurances and instructionswith the respondent before leaving the respondent to complete the form in private. To ensure privacy, the interviewerswere instructed not to read the questions in the SAQ to the respondents if they had difficulty reading. Instead, the interviewers were to note when a respondent had reading problems with the SAQ. When the respondent had completed the SAQ, he put it directly into a mailing envelope. The interviewer placed the completed interview schedule and the other study forms into the same envelope. Then the respondent sealed the envelope so that it was ready to be mailed to the survey research firm. In 1988 all but 21 of the 1,880 respondents (99%) completed the SAQ, and item nonresponse was never higher than 5%. Until recently, self-administered questionnaires generally took the form of a paper-and-pencilinstrument that a respondent filled out. These forms rely heavily on the literacy skills of the respondent, however, and are inhospitable to branching questions (Sonenstein et al., 1989).3These weaknesses can be particularly problematic in an evaluation of a program targeted to teens with academic difficulties. Great care needs to be taken in crafting a selfadministered paper-and-pencil questionnaire to make sure that the language is simple and easily comprehensible. Moreover, respondents sometimes have trouble following instructions that ask them to skip certain questions. As a result, the amount of information that can be collected in this paper-and-pencil format is inherently limited. To address these weaknesses, new audio approaches to collecting selfreports about sensitive behavior have been developed. Cannel and his colleagues have tried a “Walkman” approach in which respondents listen to an interview on a tape recorder and put their answers on an answer sheet (Cannel et al., 1991). More recently, audio computer assisted self-interviewing (ACASI) approaches have been developed (Johnston& Walton, 1992; O’Reilly et al., 1992). In this approach, a laptop computer administers the interview through headphones, and the respondent keys in answers. Every respondent hears questions exactly the same way. Poor readers can listen to the questions 3
Branchingquestionsare questionsthat differentiatethe pathsthat respondentstakethrough a questionnaire.For example,respondentswhosay theyhaveeverhad sex answerone set of questions-about numberof partners,recentnessof intercourse,etc.—whilethosethat have neverhad sex are askedanotherset of questions—aboutthe agetheyplanto havesexforthe firsttimeor the agetheyplan to marry.
Using Self Reports
579
rather than read them. Moreover, the administration of the interview may feel more private to the respondent because the answers are going into a machine rather than onto a form that someone will read. The A-CASI methodologyalso easily accommodates branching questions, and data quality improves when respondents are not trying to answer questions that are not relevant to them. We conducted an experiment in the 1995 NSAM to see if we got higher reports of sensitive behaviors using A-CASI compwd with paper-and-pencil SAQS. Preliminary results indicate that use of A-CASI has substantially increased reporting of stigmatized behaviors but made little difference to the reporting of behaviors like sexual intercourse and condom use (Turner, Ku, Sonenstein, & Pleck, 1996). These preliminary results suggest that adolescent males ages 15- to 19-years-oldgenerally do not underreport their heterosexual experience because of concerns about revealing this information to interviewers. Reliability and Validity of Reports About Sexual Experience
One of the most common criticisms of interview or questionnaire-based survey data is their credibility.Sexual behavior is one of the most private areas of human experience, and many are particularly skeptical of teenage males’ responses. There has been considerable speculation that teenage males in the U.S. exaggerate sexual experience (Newcomer & Udry, 1988). In the NSAM we can compare a respondent’s report to an interviewer with his answers on the SAQ, where presumably he is not as motivated to provide socially acceptable answers because of the privacy of the reporting situation. Our analyses indicate very high consistency between the respondents’ reports to the interviewer about engaging in sexual intercourse with a female and their SAQ reports. In 1988 there was 96~o consistency, and in 1995 there was 9570 consistency. Although the consistency of measures between the interviews and the SAQS demonstrates good reliability of results, it does not indicate validity to any external standards. A young man might say, for example, he has had sexual intercourse in the interview and the SAQ but still be dishonest on both occasions. Validation might be based on measures within the sample or inferred through comparisons with other population measures. When we initially reported that condom use by adolescent males more than doubled between 1979 and 1988, for instance, it was useful to be able to cite substantial growth in nationwide condom sales over that period in a consistent fashion (Sonenstein, Pleck, & Ku, 1989). This information greatly improved face
580
Sonenstein
validity for the survey results. It is more difllcult to provide external corroboration of sexual activity reports because the observed consequences of this behavior—births, abortions and sexually transmitted diseases--are affected by whether the teenagers use contraception. In our more recent analyses, we have found that levels of self-reported unprotected sex among our respondents in 1988 are strongly correlated with reports of pregnancies and births that occurred between 1988 and 1991. Other analyses comparing births reported by males in our survey are consistent with levels of births (children fathered by men of ages similar to our sample) reported by women in the 1988 National and Maternal Infant Health Survey (Lindberg et al., 1996). That is, not only can we demonstrate that males’ reports of births are broadly consistent with females’ but that the males’ fertility experiences can be predicted based on their earlier self-reported sexual and contraceptive behaviors. For the 1995 surveys,we have added yet another method of validation:we collect urine specimens from respondents over the age of 18 to conduct tests of infection by two common sexually transmitted diseases, Chlarnydia trachomatis and Neisseria gonorrhea. New DNA-based technologiesuse simple urine specimens and are highly sensitive and specific. If the diagnostic tests identify people who are infected but who claim they are not sexually active or have used condoms 100Yoof the time, then their responses need to be viewed as suspect. A recent study of clinic patients, for example, found that many of those who became infected with an STD reported 100% condom use; the study found that there was essentially no correlation between self-reported condom use and STD acquisition (Zenilman et al., 1995). In other areas of research, the use of biomarkers to validate self-reported behaviors has been common for a long time. Smoking researchers, for instance, routinely collect saliva or other biological specimens to measure tobacco metabolizesto verify smoking status. Our discussion of the reliability and validity of teens’ reports about their sexual behavior has relied on our own research with teenage males. Our work with this population demonstrates that reasonably good estimates of sexual activity levels can be obtained through close attention to the format of the questionnaire and to the setting in which it is used. We do not have evidence about how successful these methods are with female teenagers. It is likely that they work as well or better since young women are believed to be more ready participants in research.4. %ere is a fairlylargebodyof researchaboutproblemswiththe reliabilityof men’sreports abouttheir fertilityand fatheringbehavior(Cherlin& Griffith,1997).
581
Using Self Reports
Many of the approaches used in survey research can be adapted to program evaluation settings to improve the reliability and validity of participants’ reports about their sexual behavior. Evaluators conducting these studies should be alert to opportunities to crosscheck the reliability of their respondents’ reports using items within a single instrument or across instruments. Opportunities to corroborate reports with external data should also be sought.
Conclusion To demonstrate success, programs to prevent pregnancy must show that their participants have changed their behavior. Measures of whether participants have delayed the initiation of sexual intercourse, measures of their levels of sexual activity, or measures of their contraceptive use are crucial to providing proof of program efficacy. Although previous evaluations of such programs have often shied away from behavioral measures, a growing body of research shows that measures of teenagers’ sexual behavior can be reliably obtained. But to collect these data, researchers must pay careful attention to how the measures fit into the total design of the evaluation, how they reflect the program’s objectives, and how they are actually constructed and implemented. Because questions about sexual behavior ask young people to reveal aspects of their private lives, there has to be a strong rationale for making such inquiries. For this reason, we recommended developing a research design that is tightly linked to the program’s logic and that specifies behavioral measures suited to the program’s objectives. As noted earlier, questions about sexual behavior can take many forms. The measures selected must reflect aspects of this behavior that the program addresses. There needs to be a very clear reason why such personal questions are posed to teenagers. A good deal has been learned about how to improve the reliability and validity of teenagers’ reporting of their sexual behavior. Evaluators can turn to some of the studies and resources described earlier in this paper to obtain information about how to frame the actual questions that are used. A key decision is whether to use self-administered or interviewer-administered instruments.Although the privacy provided by the self-administeredquestionnaire may enhance full and complete reporting, this format has limitations if a paper-and-pencil version is used. Consideration should be given to developing “Walkman” or computer-administered questionnaires, since these ap-
582
Sonenstein
preaches can overcome the problems caused by poor literacy, which is especially common among many disadvantaged teenagers. Finally, protections must be put in place to ensure that the teenagers and their parents, when appropriate, understand the purpose of the study and the assurances of confidentiality that are made. When teenagers are asked about their sexual behavior, they must have given their informed consent to participate in the study, and they must understand that they can refuse to answer any question with impunity. In a program setting, this stipulation means that teenagers can continue in the program even if they do not participate in the research. Assurances of confidentiality also require carefully developed procedures to limit access to behavioral data with personal identifiers to research personnel. Program personnel will therefore not know what particular clients told the researchers. With the new infusion of federal funding for abstinence programs, the time is ripe for developing some rigorous evaluations of promising programs. To advance our knowledge of the types of programs that lead to reductions in teenage pregnancy, these evaluations must include assessments of the effects of the programs on teenagers’ sexual behavior. That behavior can be measured well if careful attention is paid to sound research principles and to the protection of human subjects.
References Bradburn, N.M. (1983). Response effects. In Rossi, P., Wright, J. & Anderson, A. (Eds.) Handbook of survey research. New York: Academic Press. Brindis, C., Peterson, J., Card, J., & Eisen, M. (1996). Prevention Minimum Evaluation Data Set (PMEDS). Los Altos,CA: SociometricsCorporation. Cannell,C., Cambum,D., Dykema,J., & Seltzer,S. (1991).Applied research on design and conduct of surveys of adolescent health behaviors and characteristics. Ann Arbor: Survey Research Center. Centers for Disease Control. (1995). Youth risk behavior surveillance-United States 1993 Morbidity and Mortality Weekly Report, 44(l), March 24. Cherlin, A., & Griffith, J. (1997). Methodological issues in improving data on fathers: A report of the working group on the methodology of studying fathers. Washington, D.C.: National Institute of Child Health and Human Development. Fishbein, M., & Ajzen, I. (1975). Belief attitude, intention and behavior. Reading, MA: Addison-Wesley. Fishbein, M., Bandura, A., Triandis, H.C., Kanfer, F.H., Becker, M.H., & Middlestadt, S.E. (1991). Factors influencing behavior and behavior change. Final
Using Self Reports
583
Report to the Theorist’s Workshop. Washington, D.C.: National Institute of Mental Health. Frost, J., & Forrest, J. (1995). Understanding the impact of effective teenage pregnancy prevention programs. Family Planning Perspectives, 27(5), 188-195. Furstenburg, F., Moore, K., & Peterson, J. (1985). Sex education and sexual experience among adolescents. American Journal of Public Health, 75(11), 1331-1332. Hanson, S., Morrison, D., & Ginsburg, A. (1989). The antecedents of teenage fatherhood. Demography, 26(4), 579-596. Hingson, R., Strunin, L., Berlin, B., & Heeren, T. (1990). Beliefs about AIDS, use of alcohol and drugs and unprotected sex among unprotected sex among Ivlassachusetts adolescents. American Journal of Public Health, 80, 295-299. Jessor, S.L., & Jessor, R. (1975). Transition from virginity to nonvirginityamong youth: A social-psychological study over time. Developmental Psychology, 11,473484. Johnston, K., & Walton, C. (1992). Reducing response effectsfor sensitive questions: The computer assisted self interview with audio. Ann Arbor: Institute for Social Research, University of Michigan. Kirby, D. (1997). No easy answers. Washington, D,C.: The National Campaign to Prevent Teenage Pregnancy. Ku, L., Sonenstein, F.L., & Pleck, J.H. (1992). The association of AIDS education and sex education with sexual behavior and condom use among teenage men. Family Planning Perspectives, 24(3), 100-106. Lindberg, L.D., Sonenstein, F.L., Martinez, G., & Marcotte, J. (1996). Accuracy of fathers’ reports offertility. Washington, D.C.: The Urban Institute. Miller, B.C., & Paikoff, R.L. (1992). Comparing adolescent pregnancy programs: methods and results. In Miller, B.C., Card, J.J., Paikoff, R.L., & Peterson, J.L. (Eds.) Preventing Adolescent Pregnancy. Newbury Park, CA: Sage Publications. Miller, P.Y., & Simon, W. (1974). Adolescent sexual behavior: context and change. Social Problems, 22(l), 58-75. Moore, K.A., Sugland, B.W., Blumenthal, C., Glei, D., & Snyder, N. (1995). Adolescent pregnancy prevention programs: Interventions and evaluations. Washington, D.C.: Child Trends Inc. Mosher, W.D., & Bachrach, C.A. (1996). Understanding U.S. fertility: Continuity and change in the National Survey of Family Growth. Family Planning Perspectives, 28( 1), 4-12. Mott, F.L. (1983). Fertility related data in the 1982 National Longitudinal Survey of Work Experience of Youth: An evaluation of data quality and some preliminary analyses. Columbus: Ohio State University. Newcomer, S., & Udry, R. (1988). Adolescent honesty in surveys of sexual behaviors. Journal of Adolescent Research, 3,4. O’Reilly, J., Hubbard, M., Lessler, J., & Biemer, P. (1992). Audio computer-assisted self-interviewing: new technology for data collection on sensitive
584
Sonenstein
issues and special populations. Paperpresentedat theAnnualMeetingof theAmeri-
can StatisticalAssociation,Boston,August,1992. SociometricsCorporation(1997~February).Announcing:NATASHAIV-The latest addition to the largest collection of adolescentpregnancyand pregnancy preventiondata!Archive Reporter. Los Altos,CA: SociometricsCorporation. SociometricsCorporation(1997b,February).Six more exemplaryprevention programsavailablein the PASHA collection.Archive Reporter. Los Altos, CA: Sociometrics Corporation. Sonenstein, F. L., Pleck, J. H., & Ku, L. (1989, August). Missing data on the missing male: Measuring fertility and paternity behavior among young men. Paper presented at the meeting of the American Statistical Association, Washington, D.C. Sonenstein, F.L., Ku, L., & Pleck, J.H. (Forthcoming). Measuring sexual behavior among teenage males in the US. In Bancroft, J. (Ed.) Researching sexual behavior: methodological issues. Bloomington, IN: The University of Indiana Press. Sonenstein, F.L. Pleck, J.H., & Ku, L.C. (1989). Sexual activity, condom use, and AIDS awareness among adolescent males. Family Planning Perspectives, 21, 152. Sorensen, R. (1973). Adolescent sexuality in contempora~ America. New York: World Pub. Turner, C.F., Ku, L., Sonenstein, F.L., & Pleck, J.H. (1995). Impact of AudioCASI on reporting of male-male sexual contacts: Preliminary findings from the National Survey of Adolescent Males, in Warnecke, R.B. (Ed.) Health survey research methods: Conference proceedings Hyattsville, MD: National Center for Health Statistics, pp. 171-76. Turner, C., Lessler, J., & Gffoerer, J. (Eds.). (1992). Survey measurement of drug use: Methodological studies. Washington D.C.: Government Printing Office. Udry, J.R., & Billy, J. (1987). Initiation of coitus in early adolescence. American Sociological Review, 52,841-855. U.S. Department of Health and Human Services (1997a). Block grant application guidance for The Abstinence IMucation Provision of the 1996 We~are Law P.L,.104193. Washington, D.C.: Office of State and Community Health, Maternal and Child Health Bureau, Health Resources and Services Administration, U.S. Department of Health and Human Services. U.S. Department of Health and Human Services (1997b). A national strategy to prevent teen pregnancy. Washington, D.C.: U.S. Department of Health and Human Services. U.S. Department of Health and Human Services (1993). Protecting human research subjects:Iinstitutional review board guidebook. Washington, D.C.: U.S. Government Printing Office. Vener, A.M., & Stewart, C.S. (1974). Adolescent sexual behavior in middle America revisited: 1973. Journal of Marriage and the Family, 36,728-735.
Using Self Reports
585
Zelnik, M., & Kantner, J.F. (1980). Sexual activity, contraceptive use, and pregnancy among metropolitan area teenagers: 1971-1979.Family Planning Perspectives, 12,230-237. Zenilman, J., Weisman, C., Rompalo, A., et al. (1995). Condom use to prevent incident STDS: the validity of self-reported condom use. Sexually Transmitted Diseases, 22(l), 15-21.
@
Children andYoutiServices Review, Vol.19,No.7,pp.567–585, 1997 Copyright @1997EkvierScience Ltd Printed intheUSA.Allrightsresewed 019C-740%97 $17.00+.00 PII S0190-7409(97)00046-7
Using Self Reports to Measure Program Impact Freya L. Sonenstein The Urban Institute In order to demonstrate success, pregnancy prevention programs must show that their participants have changed their behavior. Accurately measuring levels of sexual activity and contraceptive use are important to determining program efficacy. These measures can be reliably obtained, but careful attention must be given to what is measured and how. In this paper, based on our experience conducting the National Survey of Adolescent Males, we offer several guidelines for measuring program effects on behavior. Successful evaluations should accurately describe program participants, including who is being targeted, who has participated in other programs, and who has already engaged in the behaviors that the program is intended to prevent; measure program outcomes by program logic and goals; employ research protocols that ensure strict confidentiality and the effective administration of surveys; and double-check responses to see that they correlate with other available evidence.
A recent review of research on programs to prevent teenage pregnancy concludes that although many programs have been created to reduce the incidence of teen pregnancy, few have been rigorously evaluated (Kirby, 1997). One of the common weaknesses of the studies reviewed is their failure to measure the effects of the program on behavior. In this paper, I describe what we have learned about measuring the sexual behavior of adolescents from our experience conducting the National Survey of Adolescent Males (NSAM). By providing practical advice on how to measure sexual behavior, I hope to encourage programs and their evaluators to document whether their programs produce behavioral change. Portionsof this articlearebasedon a paperpresentedat the conferenceResearchingSexual Behavior:MethodologicalIssues,whichwasheld on April26-28,1996,at the Universityof Indiana.The NationatSurveyof AdolescentMates has been supportedby grants from the NationatInstituteof ChildHealthand HumanDevelopment. Reprintsmaybe obtainedfromFreyaSonenstein,Ph.D.,Director,PopulationStudiesCenter, TheUrbanInstitute,2100M Street,N.W.,Washington,D.C.20037 [[email protected]]
567
568
Sonenstein
This discussion is particularly timely because the federal government is about to make a substantial new investment in teenage pregnancy prevention. Recent federal welfare reform legislation (P.L. 104-193) authorizes $50 million annually for states to implement prevention programs that promote sexual abstinence among unmarried teenagers. If states opt to use these funds, the matching-fund requirements will swell the resources devoted to abstinence education to almost half a billion dollars over the next five years. This sum does not include the $7.6 million dollars also authorized annually under the Adolescent Family Life Act. For the first time, significant federal and state funds will be invested in abstinence programs for teenagers (U.S. Department of Health and Human Services, 1997a;U.S. Department of Health and Human Services, 1997b). Unfortunately,no provision has been made in the legislation for program evaluation. This omission is particularly troubling because evidence on the effectiveness of abstinence programs is quite sketchy. Only a handful of abstinence programs have been evaluated. None of the published studies have demonstrated reductions in sexual activity levels, but each study suffers design flaws that prevent conclusions about either positive or negative effects (Kirby, 1997; Moore et al., 1995; Frost & Forrest, 1995; Miller & Paikoff, 1992). Given the forthcoming substantial investment of taxpayers’funds in these programs, it would be wise to focus research efforts on assessing whether abstinence and other types of prevention programs change teenagers’ behavior and reduce unintended pregnancies. To that end, this paper will describe lessons learned from our experience conducting the NSAM. First, I will discuss how to make decisions about what to measure within a program evaluation context. Then, I will turn to how to measure sexual behavior among teenagers. Choices about What to Measure
The measurement strategy for any program evaluation should be tailored to the theory of behavior that underlies the program approach. Programs are implemented because someone thinks that the intervention will achieve desired changes in behavior among a particular group of people. The form the intervention takes rests on assumptions about how and why people behave in the way they do. These assumptionsmay be expressed in a formallydeveloped theory of behavior such as the theory of reasoned behavior described below, they may bean experienced practitioner’s best guess about approaches likely to change the behavior of the target population, or they maybe some combi-
Using Self Reports
569
nation. Whatever form they take--formal or informal-- an understanding of such assumptions is critical for designing an evaluation and measurement strategy.The logic of the program rests on those assumptionsthat pinpoint the program levers expected to produce behavior change in participants. An evaluation tests whether the program logic works and whether participants’ behavior changes as a result. An example of a formal theory of behavior that has influenced many prevention programs is Fishbein and Ajzen’s theory of reasoned behavior (1975). In this model a behavior, like the decision to have intercourse for the first time, is seen as a product of a process in which various fainily and background characteristics lead teens, first, to hold particular attitudes about the consequences of having sex and, second, to perceive that people who are important to them have expectations about how they should behave. Together, these attitudes and normative expectations create an intention to actor not to act, which, in turn, leads to the actual behavior.] Figure 1 illustrates the major concepts in the theory. A program based on this theory will attempt to change behavior by trying to alter teenagers’ attitudes toward the consequences of sexual behavior and their perceptions about what people close to them expect. To assess whether the program has achieved its desired results, measures are needed of whether the program has produced changes in those attitudes and normative expectations as well as whether the participantschange their sexual behavior. This assessmentchecks whether the program logic has operated as expected and whether more participating teenagers are behaving in conformity with the program objectives. Thus, a program evaluation requires not just measures of behavior change but measures of the intermediate processes that produce those changes. The figure illustrating the theoretical model also shows that the characteristics of the programs’ participants are important to consider. Individuals enter programs with an initial set of attitudes and expectations that primarily reflect the families and the communities in which they live. It is important to record the characteristicsof the participants, because many programs target particular types of teenagers. A program effective with one type of teenager may not necessarily be effective with another. Therefore, a thorough description of program participants is important for understanding what types of programs work with particular types of teens. 1 For ease of presentationthe Fishbeinand Ajzenmodelhas been strippeddownto its most essentialelements.Readersare urgedto consultsourcedocumentsfor a fullerunderstanding of its complexity.(Fishbein& Ajzen, 1975;Fishbeinet al., 1991).
Sonenstein
Figurel
Simple Conceptual Model Example
Backgroundand FamilyCharacteristics
1
,
It is also important to know whether the program participants have participated in other interventions. Data from NSAM shows how crucial those measures can be. In 1988, almost all 15- to 19-year-old males in the U.S. reported receiving formal instruction in school about sexual topics and 58% reported having been taught “how to say no to sex” (Ku, Sonenstein and Pleck, 1992). Depending on the age of the program participants and the availability of other resources in the community, it is quite possible that program participantshave already experienced another intervention.This prior intervention may have influenced their attitudes and expectations. For these participants, the evaluation does not measure the effects of a single program on behavior but the effects of that program plus the effects of the other programs in which the participants have been involved. In addition, some participants maybe sexually active even though a program may be aimed at getting teens to delay the initiationof sex. In 1988, onethird of 15-year-old males reported that they had had intercourse, but there were substantial differences by ethnicity. More than two-thirds of black 15to 19-year-old males were sexually experienced, compared with one-third of Hispanic males and one-quarter of white males (Sonenstein, Pleck and Ku, 1989). Some program participants may have already engaged in the behavior targeted by the intervention. Indeed, in 19883890 of the 15- to 19-year-old
Using Self Reports
571
males reported that they had first received instruction about “how to say no to sex” afler they initiated sex (Ku, Sonenstein, & Pleck, 1992). It is therefore important to collect information about the program participants to be able to describe those targeted for the intervention, to differentiate those who have experienced other interventions, and to understand whether some of the participants have already engaged in the behaviors the program is trying to prevent. In addition, one would also want to measure how much of the program intervention the participants receive. Some teens attend every session, while others may attend sporadically. One would expect that those receiving more of the intervention will experience a greater change in attitudes, expectations, and behavior. This proposition can be tested if measures of program dosage are collected. The measures of program outcomes should be defined by the program logic. One would expect an abstinence program based on the theory of reasoned action to produce changes in participants’ attitudes toward the consequences of sex and in perceptions of normative expectations. These changes, in turn, would result in shifts in intentions to engage in sex. Finally, participants should be observed delaying the onset of sexual intercourse. While measures of all these concepts are needed to test the program logic, many evaluations of abstinence programs fall short because they measure shifts in mediating concepts like attitudes, expectations or behavioral intentions but neglect to measure actual behavior (Kirby, 1997). It is useful to know whether a program produces changes in attitudes, expectations, and behavioral intentions, but the bottom-line question is whether the program produces actual changes in behavior. To get at this question, the evaluation needs to ask teenagers about their sexual behavior. The choice of questions about sexual behavior should be tightly linked to the program logic. Some programs aim to delay sex until teens are older; some aim to delay sex until marriage. Some work with teens who have already had sex to get them to stop having sex or to reduce their number of partners. A few may promote alternative forms of sexual expression. Some emphasize better communication strategies between partners. Some focus on consistent contraception. The behavioral measures selected for evaluations of particular programs should reflect the behaviors targeted for change. These probably encompass more than just whether program participants have initiated vaginal intercourse by the end of the program. Depending on the program objectives, measures may be needed of the number of months participants delay initiating sex ●
572
Sonenstein
whether participants delay sex until marriage the number of sexual partners participants have ever had the number of sexual partners participantshave had in some recent period of time like the past three months the number of simultaneous sexual relationships the participant has ever had or had in some recent period of time communication strategies used by romantic partners other forms of sexual expression that participants engage in contraceptive methods used at last intercourse consistency of contraceptive method used in a given period of time, like the past three months or the past year We are not advocating that programs use all these measures. They should choose measures related to their program objectives. Programs that aim to delay sexual activity until marriage will need designs that assess participants’ sexual behavior prospectively until they marry. Programs that have shorter time frames can rely on shorter follow-up periods. It should be noted, however, that 12 months after program completion is generally considered a minimum follow-up period for evaluations of prevention programs, according to the guidelines established by the Program Archive on Sexuality and Adolescence (Sociometrics, 1997b). Programs that emphasize monogamy will probably select more measures of the types of sexual partnerships that participants enter. Programs that teach communication strategies for couples will need measures of those behaviors. Programs that include discussions of contraception will want to include measures of contraceptive behavior. In the development of a measurement strategy for a program evaluation, the main points to remember are Choices need to be made not just about the outcome measures but about the full range of concepts that the program logic defines as relevant. It is not enough to focus just on outcome behaviors. Measures are also needed of participants’ baseline characteristics, the major concepts—like attitudes and normative expectations-seen as affecting behavior, and the level of program participation. It is also not enough to measure attitudes or intent to behave, as many evaluations do, without examining whether behavioral changes actually occur. The choice of sexual behaviors to be measured should reflect the program logic and the types of behaviors targeted for change. ● ● ●
9
● ● ● ●
9
●
●
●
Using Self Reports
573
How to Measure Sexual Behavior
Having talked about the choices of what to measure, I now turn to what we have learned about how to measure sexual behavior. Measuring sexual behavior is inherently difficult because respondents are being asked to se~report on sensitive behavior. Measuring any behavior using self-reporting is challenging, but measuring sexual behavior poses special problems. The primary challenges of measuring sexual behavior among teenagers are that ●Sexual behavior is viewed as a private matter in our culture. There are strong norms about not discussing one’sown behavior with other people, especially with strangers. Respondents may be reluctant to answer these questions, or they may refuse to answer them, either actively or passively, by denying that they are engaging in the behavior. Sexual behavior among teenagers may also be tinged by social desirability issues. Teens may be reluctant to admit to engaging in behaviors that are disapproved of. For prevention programs, this issue may be particularly important if program participants do not honestly report their actions out of fear of disapproval. The corollary is that teens may be too eager to admit to engaging in behaviors that are socially approved. For reasons of both privacy norms and social desirability,estimates of the prevalence of sexual behaviors among teenagers maybe biased. Many sample surveys of U.S. teenagers conducted since the early 1970s have included measures of sexual behavior and have therefore dealt with these challenges. These surveys have included nationally representative samples of teenagers like the studies by Sorensen (1973), Zelnik and Kantner (1980), Mosher and Bachrach (1996), Sonenstein, Pleck, and Ku (1989), and Furstenburg, Moore, and Peterson (1985), as well as the new Adolescent Health Study, led by Richard Udry at the University of North Carolina. Studies of teenagers representing state or community populations have also been conducted, including those by Miller and Simon (1974) in Illinois, Vener and Stewart (1974) in three Michigan communities, Jessor and Jessor (1975) in a Rocky Mountain city, Udry and Billy (1987) in a North Carolina community, and Hingson, Strunin, Berlin, and Heeren in Massachusetts (1990). Many multipurpose surveys of teenagers, like the National Longitudinal Survey of Youth (Mott, 1986), High School and Beyond (Hanson, Morrison, & Ginsburg, 1989), and the Youth Risk Behavior Survey (CDC, 1995), include at least some questions about sexual behavior. These studies have developed approaches to address the problems associated with collecting information ●
574
Sonenstein
about sensitive behaviors. Moreover, this significant body of work demonstrates that teenagers can be asked about their sexual behavior. Our experience conducting NSAM, for example, indicates that most teenage males are willing to participate in a survey about reproductive health issues. Our response rates, even with a requirement of parental permission for those under 18, are similar to other household surveys. Three-quarters of all the males aged 15 to 19 identifih in the national household sampling frame participated in the surveysin 1988 and 1995. The follow-up rate was also high for the cohort first interviewed in 1988; we reinterviewed 8970 of them two and a half years later. Evidently, these teenagers were not troubled by their experience with the subject matter covered in the initial interviews. These surveys have also already dealt with many of the issues that an evaluator might face in phrasing questions about sexual behavior for teenagers. The surveys provide a useful source of questionnaire items for program evaluators, once decisions have been made about the evaluation’s general measurement strategy and the aspects of sexual behavior that should be appropriately included in the study. Some of these surveys have used selfadrninistered instruments, and some have relied on in-person interviews. In consultingthose instruments,evaluatomshould be cognizant of the differences between these two modes of data collection.2In selecting the measures to be used in NSAM, we consulted many instruments for collecting data. In addition, we ran many focus groups and qualitative interviews before the actual survey to make sure that teenagers were interpreting the questions in the intended manner (Sonenstein, Ku, & Pleck, forthcoming). A useful source for obtaining many of the instruments used in studies like these is the Data Archive on Adolescent Pregnancy Prevention, which is managed by the Sociometrics Corporation in Los Altos, California (Sociometrics, 1997a). In addition, this organization has published a guide to a minimum data set for evaluating programs aimed at preventing adolescent pregnancy. This guide includes a recommended core questionnaire and supplementary modules for use in evaluations of teenage pregnancy prevention programs (Brindis, Peterson, Card, & Eisen, 1996).
LThisdifferenceis discussedfurtherin a later sectionof this paper.
Using Self Reports
575
Confidentiality, Informed Consent, and Protections
When teenagers are asked to reveal information about sexual behavior, it is crucial that the evaluator pay strict attention to observing guidelines for protecting human subjeets (U.S. Department of Health and Human Services, 1993). One important component of the recommended procedures is to ensure that any information provided will be treated as confidential. Executed properly, such research protocols protect participants against the possibility that private information about them will be released inadvertently. This assurance has the added benefit of encouraging teenagers to provide truthful answers to questions about sensitive behaviors because they know that their answers will be treated confidentially. Providing these protections is a standard part of informed-consent procedures. In so doing, the researcher promises that the respondent’s answers will be seen only by the research staff and that all reports that describe findings will be aggregated in ways that prevent the respondent from being identified. Protective procedures should include storing information by identification numbers rather than by respondents’ names, storing master files and any information with personal identifiers in locked files, and requiring all staff, from interviewers and data gatherers to the dataprocessing personnel, to sign pledges not to release information about respondents to anyone other than authorized staff. In a program evaluation setting, this pledge means that program staff will not have access to the information provided to evaluators by individual program participants. When information is collected about sensitive behaviors like sex, other important aspects of protecting human subjects should also be followed. One such procedure is obtaining informed consent from the study participants. An important component of this process is providing complete information to the study participants about the kinds of questions that will be asked and assurances that they can refuse to answer any question without consequences. If a survey will ask questions about sex, the informed-consent procedures should prepare respondents for those questions. It is also a good idea to provide a rationale for these questions. Respondents need to know that their answers are important, and, when sampling procedures are used, respondents should be told that they represent many other teenagers like themselves. A particular issue for conducting surveys of teenagers is obtaining parental permission. In 1996, the U.S. Congress considered the Family Privacy Act, which would have barred any surveys of minor teenagers on sensitive topics without active parental permission; the legislation failed to pass. The surveys that would have been particularly affected by this legislation are those that
576
Sonenstein
collect data in schools and rely on classroom administration of paper and pencil instruments. Typically, these surveys of students in school use passive parental consent procedures. Parents are notified that the study will be conducted, but they must contact the school if they do not wish their child to participatein the study. Obtaining active consent from parents in these surveys is difficult because students are not very good about transporting permission slips home and then back to the school to the proper authorities. There are multiple opportunities for forms to get lost. Therefore, studies that rely on active consent procedures using the permission-slip approach are likely to have dismal response rates. When response rates are low, the resources put into the research effort are wasted. Active consent procedures are feasible and desirable when teenagers are to be interviewed in person. NSAM used active parental consent procedures for all teenage respondents under the age of 18. The parents of the teens had to sign an informed consent form before their sons were interviewed.The form had five main components: Frankness about the purpose of the study. The parental consent form said that the son had been randomly selected to participate in a survey whose purpose was “to get information about young men’s knowledge, attitudes and behaviors that relate to current health issues, such as alcohol and drug use, sexual behavior, sexually transmitted diseases and AIDS. ” Assurance that permission is required. The parent was told that the son would not be asked to participate if consent were not given. Assurance that the child still needed to provide his own consent. If parental permission were given, the teenager could still decide whether or not he wished to participate in the survey. Having started the interview, he could refuse to answer any question. Assurances about confidentiality. Answers to the interview would be treated as confidential and would not be associated with the son’sname in any reports. Indeed, parents were told that they would not have access to their own children’s answers. Contact names to obtain more information.The parent was given the name and telephone number of the study director and the Institutional Review Board chairperson. Although this form was quite long, study participation was excellent. In 1988, only 6.1% of households with eligible respondents did not participate because a parent withheld consent for his or her minor child; in 1995, this proportion was 5.6%. We believe that these high rates of cooperation are a tribute to the professional skills of the interviewers who did an excellent job ●
●
●
●
●
Using Self Reports
577
conveying the importance of the study. We also think that the high proportion of parents providing permission demonstrates the level of support that parents of teenagers have for public health efforts to address adolescent sexual health issues. A final protection issue for surveys about teenage sexual behavior is making provision for the rare instances in which a respondent reveals that he or she has been raped or sexually abused. Such provisions should include referrals to an appropriate agency for assistance and the reporting of the incident to children’s protective services if the state law requires it. While situations like these are unlikely, procedures should be in place so that staff know what to do if they occur and so that teenagers are supported to reduce the possibility of further harm. Questionnaire Formut
At the core of any survey about sexual behavior is a detailed questionnaire, administered by an interviewer (that is in person or by telephone) or self-administered by the respondent (a written or computer-assisted questionnaire, for example). The arguments in favor of interviewer-based questionnaires are that professionalinterviewers are more capable of handling complex questionnaires and skip logic and that personal rapport between the interviewer and subject can yield greater commitment to the interview process. Furthermore, the interviewer can provide additional explanation or guidance if the respondent has questions. The main arguments in favor of selfadministered questionnaires are that they ensure greater privacy for the respondent and reduce the risk that respondents will be reluctant to give responses to an interviewer who meets with the disapproval of the respondent. Self-administeredquestionnaires are recommended to reduce the respondents’ desire to give socially desirable answers (Bradburn, 1983) and to increase candid responses (Turner et al., 1992). This is particularly important for sensitive subjects, such as sexual behavior. In the 1988 and 1991 NSAM surveys, we elected to use both interviews and self-administered questionnaires. Many of the core behaviors, including most heterosexual behaviors and contraceptive use, were primarily explored in the face-to-face interviews. A paper-and-pencil self-administeredquestionnaire (SAQ) was used for the most sensitivebehaviors including illicit behavior, substance use, and very risky sexual practices. At the end of the in-person interview, which usually lasted one hour, interviewers explained that there were additional questions that we would like the respondent to answer in
578
Sonenstein
private. The respondentswere handed the self-administeredquestionnaire.The front page of the SAQ included assurances of confidentiality and instructions on how to complete the form. The interviewer reviewed these assurances and instructionswith the respondent before leaving the respondent to complete the form in private. To ensure privacy, the interviewerswere instructed not to read the questions in the SAQ to the respondents if they had difficulty reading. Instead, the interviewers were to note when a respondent had reading problems with the SAQ. When the respondent had completed the SAQ, he put it directly into a mailing envelope. The interviewer placed the completed interview schedule and the other study forms into the same envelope. Then the respondent sealed the envelope so that it was ready to be mailed to the survey research firm. In 1988 all but 21 of the 1,880 respondents (99%) completed the SAQ, and item nonresponse was never higher than 5%. Until recently, self-administered questionnaires generally took the form of a paper-and-pencilinstrument that a respondent filled out. These forms rely heavily on the literacy skills of the respondent, however, and are inhospitable to branching questions (Sonenstein et al., 1989).3These weaknesses can be particularly problematic in an evaluation of a program targeted to teens with academic difficulties. Great care needs to be taken in crafting a selfadministered paper-and-pencil questionnaire to make sure that the language is simple and easily comprehensible. Moreover, respondents sometimes have trouble following instructions that ask them to skip certain questions. As a result, the amount of information that can be collected in this paper-and-pencil format is inherently limited. To address these weaknesses, new audio approaches to collecting selfreports about sensitive behavior have been developed. Cannel and his colleagues have tried a “Walkman” approach in which respondents listen to an interview on a tape recorder and put their answers on an answer sheet (Cannel et al., 1991). More recently, audio computer assisted self-interviewing (ACASI) approaches have been developed (Johnston& Walton, 1992; O’Reilly et al., 1992). In this approach, a laptop computer administers the interview through headphones, and the respondent keys in answers. Every respondent hears questions exactly the same way. Poor readers can listen to the questions 3
Branchingquestionsare questionsthat differentiatethe pathsthat respondentstakethrough a questionnaire.For example,respondentswhosay theyhaveeverhad sex answerone set of questions-about numberof partners,recentnessof intercourse,etc.—whilethosethat have neverhad sex are askedanotherset of questions—aboutthe agetheyplanto havesexforthe firsttimeor the agetheyplan to marry.
Using Self Reports
579
rather than read them. Moreover, the administration of the interview may feel more private to the respondent because the answers are going into a machine rather than onto a form that someone will read. The A-CASI methodologyalso easily accommodates branching questions, and data quality improves when respondents are not trying to answer questions that are not relevant to them. We conducted an experiment in the 1995 NSAM to see if we got higher reports of sensitive behaviors using A-CASI compwd with paper-and-pencil SAQS. Preliminary results indicate that use of A-CASI has substantially increased reporting of stigmatized behaviors but made little difference to the reporting of behaviors like sexual intercourse and condom use (Turner, Ku, Sonenstein, & Pleck, 1996). These preliminary results suggest that adolescent males ages 15- to 19-years-oldgenerally do not underreport their heterosexual experience because of concerns about revealing this information to interviewers. Reliability and Validity of Reports About Sexual Experience
One of the most common criticisms of interview or questionnaire-based survey data is their credibility.Sexual behavior is one of the most private areas of human experience, and many are particularly skeptical of teenage males’ responses. There has been considerable speculation that teenage males in the U.S. exaggerate sexual experience (Newcomer & Udry, 1988). In the NSAM we can compare a respondent’s report to an interviewer with his answers on the SAQ, where presumably he is not as motivated to provide socially acceptable answers because of the privacy of the reporting situation. Our analyses indicate very high consistency between the respondents’ reports to the interviewer about engaging in sexual intercourse with a female and their SAQ reports. In 1988 there was 96~o consistency, and in 1995 there was 9570 consistency. Although the consistency of measures between the interviews and the SAQS demonstrates good reliability of results, it does not indicate validity to any external standards. A young man might say, for example, he has had sexual intercourse in the interview and the SAQ but still be dishonest on both occasions. Validation might be based on measures within the sample or inferred through comparisons with other population measures. When we initially reported that condom use by adolescent males more than doubled between 1979 and 1988, for instance, it was useful to be able to cite substantial growth in nationwide condom sales over that period in a consistent fashion (Sonenstein, Pleck, & Ku, 1989). This information greatly improved face
580
Sonenstein
validity for the survey results. It is more difllcult to provide external corroboration of sexual activity reports because the observed consequences of this behavior—births, abortions and sexually transmitted diseases--are affected by whether the teenagers use contraception. In our more recent analyses, we have found that levels of self-reported unprotected sex among our respondents in 1988 are strongly correlated with reports of pregnancies and births that occurred between 1988 and 1991. Other analyses comparing births reported by males in our survey are consistent with levels of births (children fathered by men of ages similar to our sample) reported by women in the 1988 National and Maternal Infant Health Survey (Lindberg et al., 1996). That is, not only can we demonstrate that males’ reports of births are broadly consistent with females’ but that the males’ fertility experiences can be predicted based on their earlier self-reported sexual and contraceptive behaviors. For the 1995 surveys,we have added yet another method of validation:we collect urine specimens from respondents over the age of 18 to conduct tests of infection by two common sexually transmitted diseases, Chlarnydia trachomatis and Neisseria gonorrhea. New DNA-based technologiesuse simple urine specimens and are highly sensitive and specific. If the diagnostic tests identify people who are infected but who claim they are not sexually active or have used condoms 100Yoof the time, then their responses need to be viewed as suspect. A recent study of clinic patients, for example, found that many of those who became infected with an STD reported 100% condom use; the study found that there was essentially no correlation between self-reported condom use and STD acquisition (Zenilman et al., 1995). In other areas of research, the use of biomarkers to validate self-reported behaviors has been common for a long time. Smoking researchers, for instance, routinely collect saliva or other biological specimens to measure tobacco metabolizesto verify smoking status. Our discussion of the reliability and validity of teens’ reports about their sexual behavior has relied on our own research with teenage males. Our work with this population demonstrates that reasonably good estimates of sexual activity levels can be obtained through close attention to the format of the questionnaire and to the setting in which it is used. We do not have evidence about how successful these methods are with female teenagers. It is likely that they work as well or better since young women are believed to be more ready participants in research.4. %ere is a fairlylargebodyof researchaboutproblemswiththe reliabilityof men’sreports abouttheir fertilityand fatheringbehavior(Cherlin& Griffith,1997).
581
Using Self Reports
Many of the approaches used in survey research can be adapted to program evaluation settings to improve the reliability and validity of participants’ reports about their sexual behavior. Evaluators conducting these studies should be alert to opportunities to crosscheck the reliability of their respondents’ reports using items within a single instrument or across instruments. Opportunities to corroborate reports with external data should also be sought.
Conclusion To demonstrate success, programs to prevent pregnancy must show that their participants have changed their behavior. Measures of whether participants have delayed the initiation of sexual intercourse, measures of their levels of sexual activity, or measures of their contraceptive use are crucial to providing proof of program efficacy. Although previous evaluations of such programs have often shied away from behavioral measures, a growing body of research shows that measures of teenagers’ sexual behavior can be reliably obtained. But to collect these data, researchers must pay careful attention to how the measures fit into the total design of the evaluation, how they reflect the program’s objectives, and how they are actually constructed and implemented. Because questions about sexual behavior ask young people to reveal aspects of their private lives, there has to be a strong rationale for making such inquiries. For this reason, we recommended developing a research design that is tightly linked to the program’s logic and that specifies behavioral measures suited to the program’s objectives. As noted earlier, questions about sexual behavior can take many forms. The measures selected must reflect aspects of this behavior that the program addresses. There needs to be a very clear reason why such personal questions are posed to teenagers. A good deal has been learned about how to improve the reliability and validity of teenagers’ reporting of their sexual behavior. Evaluators can turn to some of the studies and resources described earlier in this paper to obtain information about how to frame the actual questions that are used. A key decision is whether to use self-administered or interviewer-administered instruments.Although the privacy provided by the self-administeredquestionnaire may enhance full and complete reporting, this format has limitations if a paper-and-pencil version is used. Consideration should be given to developing “Walkman” or computer-administered questionnaires, since these ap-
582
Sonenstein
preaches can overcome the problems caused by poor literacy, which is especially common among many disadvantaged teenagers. Finally, protections must be put in place to ensure that the teenagers and their parents, when appropriate, understand the purpose of the study and the assurances of confidentiality that are made. When teenagers are asked about their sexual behavior, they must have given their informed consent to participate in the study, and they must understand that they can refuse to answer any question with impunity. In a program setting, this stipulation means that teenagers can continue in the program even if they do not participate in the research. Assurances of confidentiality also require carefully developed procedures to limit access to behavioral data with personal identifiers to research personnel. Program personnel will therefore not know what particular clients told the researchers. With the new infusion of federal funding for abstinence programs, the time is ripe for developing some rigorous evaluations of promising programs. To advance our knowledge of the types of programs that lead to reductions in teenage pregnancy, these evaluations must include assessments of the effects of the programs on teenagers’ sexual behavior. That behavior can be measured well if careful attention is paid to sound research principles and to the protection of human subjects.
References Bradburn, N.M. (1983). Response effects. In Rossi, P., Wright, J. & Anderson, A. (Eds.) Handbook of survey research. New York: Academic Press. Brindis, C., Peterson, J., Card, J., & Eisen, M. (1996). Prevention Minimum Evaluation Data Set (PMEDS). Los Altos,CA: SociometricsCorporation. Cannell,C., Cambum,D., Dykema,J., & Seltzer,S. (1991).Applied research on design and conduct of surveys of adolescent health behaviors and characteristics. Ann Arbor: Survey Research Center. Centers for Disease Control. (1995). Youth risk behavior surveillance-United States 1993 Morbidity and Mortality Weekly Report, 44(l), March 24. Cherlin, A., & Griffith, J. (1997). Methodological issues in improving data on fathers: A report of the working group on the methodology of studying fathers. Washington, D.C.: National Institute of Child Health and Human Development. Fishbein, M., & Ajzen, I. (1975). Belief attitude, intention and behavior. Reading, MA: Addison-Wesley. Fishbein, M., Bandura, A., Triandis, H.C., Kanfer, F.H., Becker, M.H., & Middlestadt, S.E. (1991). Factors influencing behavior and behavior change. Final
Using Self Reports
583
Report to the Theorist’s Workshop. Washington, D.C.: National Institute of Mental Health. Frost, J., & Forrest, J. (1995). Understanding the impact of effective teenage pregnancy prevention programs. Family Planning Perspectives, 27(5), 188-195. Furstenburg, F., Moore, K., & Peterson, J. (1985). Sex education and sexual experience among adolescents. American Journal of Public Health, 75(11), 1331-1332. Hanson, S., Morrison, D., & Ginsburg, A. (1989). The antecedents of teenage fatherhood. Demography, 26(4), 579-596. Hingson, R., Strunin, L., Berlin, B., & Heeren, T. (1990). Beliefs about AIDS, use of alcohol and drugs and unprotected sex among unprotected sex among Ivlassachusetts adolescents. American Journal of Public Health, 80, 295-299. Jessor, S.L., & Jessor, R. (1975). Transition from virginity to nonvirginityamong youth: A social-psychological study over time. Developmental Psychology, 11,473484. Johnston, K., & Walton, C. (1992). Reducing response effectsfor sensitive questions: The computer assisted self interview with audio. Ann Arbor: Institute for Social Research, University of Michigan. Kirby, D. (1997). No easy answers. Washington, D,C.: The National Campaign to Prevent Teenage Pregnancy. Ku, L., Sonenstein, F.L., & Pleck, J.H. (1992). The association of AIDS education and sex education with sexual behavior and condom use among teenage men. Family Planning Perspectives, 24(3), 100-106. Lindberg, L.D., Sonenstein, F.L., Martinez, G., & Marcotte, J. (1996). Accuracy of fathers’ reports offertility. Washington, D.C.: The Urban Institute. Miller, B.C., & Paikoff, R.L. (1992). Comparing adolescent pregnancy programs: methods and results. In Miller, B.C., Card, J.J., Paikoff, R.L., & Peterson, J.L. (Eds.) Preventing Adolescent Pregnancy. Newbury Park, CA: Sage Publications. Miller, P.Y., & Simon, W. (1974). Adolescent sexual behavior: context and change. Social Problems, 22(l), 58-75. Moore, K.A., Sugland, B.W., Blumenthal, C., Glei, D., & Snyder, N. (1995). Adolescent pregnancy prevention programs: Interventions and evaluations. Washington, D.C.: Child Trends Inc. Mosher, W.D., & Bachrach, C.A. (1996). Understanding U.S. fertility: Continuity and change in the National Survey of Family Growth. Family Planning Perspectives, 28( 1), 4-12. Mott, F.L. (1983). Fertility related data in the 1982 National Longitudinal Survey of Work Experience of Youth: An evaluation of data quality and some preliminary analyses. Columbus: Ohio State University. Newcomer, S., & Udry, R. (1988). Adolescent honesty in surveys of sexual behaviors. Journal of Adolescent Research, 3,4. O’Reilly, J., Hubbard, M., Lessler, J., & Biemer, P. (1992). Audio computer-assisted self-interviewing: new technology for data collection on sensitive
584
Sonenstein
issues and special populations. Paperpresentedat theAnnualMeetingof theAmeri-
can StatisticalAssociation,Boston,August,1992. SociometricsCorporation(1997~February).Announcing:NATASHAIV-The latest addition to the largest collection of adolescentpregnancyand pregnancy preventiondata!Archive Reporter. Los Altos,CA: SociometricsCorporation. SociometricsCorporation(1997b,February).Six more exemplaryprevention programsavailablein the PASHA collection.Archive Reporter. Los Altos, CA: Sociometrics Corporation. Sonenstein, F. L., Pleck, J. H., & Ku, L. (1989, August). Missing data on the missing male: Measuring fertility and paternity behavior among young men. Paper presented at the meeting of the American Statistical Association, Washington, D.C. Sonenstein, F.L., Ku, L., & Pleck, J.H. (Forthcoming). Measuring sexual behavior among teenage males in the US. In Bancroft, J. (Ed.) Researching sexual behavior: methodological issues. Bloomington, IN: The University of Indiana Press. Sonenstein, F.L. Pleck, J.H., & Ku, L.C. (1989). Sexual activity, condom use, and AIDS awareness among adolescent males. Family Planning Perspectives, 21, 152. Sorensen, R. (1973). Adolescent sexuality in contempora~ America. New York: World Pub. Turner, C.F., Ku, L., Sonenstein, F.L., & Pleck, J.H. (1995). Impact of AudioCASI on reporting of male-male sexual contacts: Preliminary findings from the National Survey of Adolescent Males, in Warnecke, R.B. (Ed.) Health survey research methods: Conference proceedings Hyattsville, MD: National Center for Health Statistics, pp. 171-76. Turner, C., Lessler, J., & Gffoerer, J. (Eds.). (1992). Survey measurement of drug use: Methodological studies. Washington D.C.: Government Printing Office. Udry, J.R., & Billy, J. (1987). Initiation of coitus in early adolescence. American Sociological Review, 52,841-855. U.S. Department of Health and Human Services (1997a). Block grant application guidance for The Abstinence IMucation Provision of the 1996 We~are Law P.L,.104193. Washington, D.C.: Office of State and Community Health, Maternal and Child Health Bureau, Health Resources and Services Administration, U.S. Department of Health and Human Services. U.S. Department of Health and Human Services (1997b). A national strategy to prevent teen pregnancy. Washington, D.C.: U.S. Department of Health and Human Services. U.S. Department of Health and Human Services (1993). Protecting human research subjects:Iinstitutional review board guidebook. Washington, D.C.: U.S. Government Printing Office. Vener, A.M., & Stewart, C.S. (1974). Adolescent sexual behavior in middle America revisited: 1973. Journal of Marriage and the Family, 36,728-735.
Using Self Reports
585
Zelnik, M., & Kantner, J.F. (1980). Sexual activity, contraceptive use, and pregnancy among metropolitan area teenagers: 1971-1979.Family Planning Perspectives, 12,230-237. Zenilman, J., Weisman, C., Rompalo, A., et al. (1995). Condom use to prevent incident STDS: the validity of self-reported condom use. Sexually Transmitted Diseases, 22(l), 15-21.