- Email: [email protected]
Improving interview method and questionnaire design
BOOK REVIEWS
emplify the trend toward creative evaluation design based upon situational factors and needs rather than the dictates of science. Given the “challenging” environment evaluation will confront in the future, Conner believes the essays demonstrate an important lesson, “that evaluators need to be flexible in their methodological approaches if they want meaningful and useful results to present to policy and decision makers” (p. 9). The first five chapters describe new evaluation methods. Cooper and Huss discuss the construction and use of simulation models, illustrated by an application to gasoline line control policies. Simulations can be useful when natural conditions are hard to control, dangerous, yet to happen, or too numerous for simple hypotheses. Thompson, Rothrock, Strain, and Palmer argue that cost analysis is a subtle and sensitive art that has been neglected in evaluation. Yet decision makers need cost information more than ever. They provide an informative discussion of direct, budgetary and nonbudgetary costs; indirect costs; and opportunity costs; and illustrate with an example from medical quality assurance. Britan convincingly promotes contextual evaluation approaches, aimed at understanding the nature of programs, their meaning for participants, and their possibilities for improvement. Contextual evaluations are complex, change oriented, inductive, holistic, and use many methods. Two studies of large scale bureaucratic change provide examples. Yeaton and Redner present an important discussion of the rationale and methods for measuring the strength and integrity of social program interventions. Their example is from juvenile justice, but the ideas have far wider implications. The use of social indicators in evaluation is discussed by Bertrand, Mock, and Franklin. They present useful
guidelines for selecting indicators (availability, direct measurement, accuracy, cost, time frames), and argue that social indicators are easy, inexpensive, and unobtrusive evaluation tools. Part II contains four examples of combination evaluation approaches. Marsh describes the combination of time-series and interviews in an evaluation of the effects of a sexual assault law, showing both long-term effects and adding to the understanding of the ideological, implementation, and contextual factors which produced change. Hughes, Cordray, and Spiker discuss the combination of process and impact evaluation, using an example from a home care program for the elderly. This combination of approaches both demonstrates outcomes and provides evidence for the integrity and strength of treatment implementation and data for causal inferences. It also enhances the use of the findings. Hirst used archival searches through program records and household files with telephone surveys of service users and nonusers to evaluate energy information telephone services. This combination matched the complexity of the policy issues, triangulated methods, and established a knowledge base for a new program. In the final chapter, Fine and Akabas report on an extremely innovative effort in a relatively untouched area, the evaluation of trade union programs. They describe a sequence of archival analysis, survey and interview studies, and experimentally designed intervention to study the effects of early intervention in disability problems. Their work illustrates the analysis of multiple social levels (individual, family, work site, and benefit unit) in a single study, and is a promising example of research utilization. T. W. W.
Improving Interview Method and Questionnaire Design, by N. M. Bradburn, Francisco: Jossey-Bass, 1979, 214 pp., $15.95 (hardcover). This book presents the results of a research program on response effects in surveys, conducted jointly by NORC and the Survey Research Lab at the University of Illinois. The inquiry was guided by earlier conceptual work by the authors, which posited three influences on response effects, (1) the nature and structure of the tasks, (2) the characteristics of the interviewer, and (3) the characteristics of the respondent. Furthermore, task variables were grouped into three classes, (a) the method of administration and the structure of the task, (b) the self-presentation of the respondent, and (c) the saliency of the task for the respondent. These dimensions were discussed in relation to non-threatening behavioral questions, threatening behavioral questions, and attitudinal questions. This report focuses on
95
S. Sudman, and Associates. San
the response effects to threatening questions, and is based upon three large samples of the general population. The book both presents results, and suggests procedures that will yield the most accurate responses. The book contains 11 chapters, some of which are adaptations of previously published work. Most of the chapters describe sophisticated research designs, report complex quantitative analyses, and present full documentation in tables. Chapter conclusions and a conclusions-implications final chapter provide valuable summaries in less demanding styles. Two instructive appendices contain the actual questions used in the research. The authors compared face-to-face interviews, telephone surveys, self-administered questionnaires, and
96
BOOK REVIEWS
randomized response methods, and found that no one method is clearly superior. The choice of method can be guided by other considerations, such as cost, access, and feasibility. Changes in question form, however, can increase the amount of reporting of behaviors two or three times over standard formats. The authors assert that long introductions and open-ended response formats “should always” be used in asking about sensitive behavior. Also promising is asking respondents to report the behavior of their close friends. The presence of third parties in the interview situation had little effect. Assurances of confidentiality did improve responses, but signed consent forms should be used only at the end of the interview. Detailed introductions had no adverse impact. Interviewer effects are small in comparison to the importance of question wording. The research produced no evidence that the occurance of nonprogrammed speech (ad libs or probes) affected the quality of responses, even though it happened frequently. Welltrained interviewers might actually improve responses with flexibility and adjustments. On the other hand,
interviewer expectations of difficulty can have negative consequences, and negative thinking interviewers should be retrained or not used. The researchers found that respondents tend to deny engaging in certain behaviors when there are social norms against discussing such topics with strangers. Their test of the Marlowe-Crowne scale suggested that it is not a good measure of individual response effects, but its results should be treated as traits of the individual subjects and should be considered a part of the real differences among people. The details of this book will be interesting to methodologists and survey researchers. Some conventional wisdom about survey methods is confirmed while other beliefs are dismissed. Their conclusions are of general importance to everyone engaged in the use of interviews and questionnaires. They confirm that careful construction is important, but show that well trained and experienced interviewers can make adjustments without destroying the results. T. W. W.
emplify the trend toward creative evaluation design based upon situational factors and needs rather than the dictates of science. Given the “challenging” environment evaluation will confront in the future, Conner believes the essays demonstrate an important lesson, “that evaluators need to be flexible in their methodological approaches if they want meaningful and useful results to present to policy and decision makers” (p. 9). The first five chapters describe new evaluation methods. Cooper and Huss discuss the construction and use of simulation models, illustrated by an application to gasoline line control policies. Simulations can be useful when natural conditions are hard to control, dangerous, yet to happen, or too numerous for simple hypotheses. Thompson, Rothrock, Strain, and Palmer argue that cost analysis is a subtle and sensitive art that has been neglected in evaluation. Yet decision makers need cost information more than ever. They provide an informative discussion of direct, budgetary and nonbudgetary costs; indirect costs; and opportunity costs; and illustrate with an example from medical quality assurance. Britan convincingly promotes contextual evaluation approaches, aimed at understanding the nature of programs, their meaning for participants, and their possibilities for improvement. Contextual evaluations are complex, change oriented, inductive, holistic, and use many methods. Two studies of large scale bureaucratic change provide examples. Yeaton and Redner present an important discussion of the rationale and methods for measuring the strength and integrity of social program interventions. Their example is from juvenile justice, but the ideas have far wider implications. The use of social indicators in evaluation is discussed by Bertrand, Mock, and Franklin. They present useful
guidelines for selecting indicators (availability, direct measurement, accuracy, cost, time frames), and argue that social indicators are easy, inexpensive, and unobtrusive evaluation tools. Part II contains four examples of combination evaluation approaches. Marsh describes the combination of time-series and interviews in an evaluation of the effects of a sexual assault law, showing both long-term effects and adding to the understanding of the ideological, implementation, and contextual factors which produced change. Hughes, Cordray, and Spiker discuss the combination of process and impact evaluation, using an example from a home care program for the elderly. This combination of approaches both demonstrates outcomes and provides evidence for the integrity and strength of treatment implementation and data for causal inferences. It also enhances the use of the findings. Hirst used archival searches through program records and household files with telephone surveys of service users and nonusers to evaluate energy information telephone services. This combination matched the complexity of the policy issues, triangulated methods, and established a knowledge base for a new program. In the final chapter, Fine and Akabas report on an extremely innovative effort in a relatively untouched area, the evaluation of trade union programs. They describe a sequence of archival analysis, survey and interview studies, and experimentally designed intervention to study the effects of early intervention in disability problems. Their work illustrates the analysis of multiple social levels (individual, family, work site, and benefit unit) in a single study, and is a promising example of research utilization. T. W. W.
Improving Interview Method and Questionnaire Design, by N. M. Bradburn, Francisco: Jossey-Bass, 1979, 214 pp., $15.95 (hardcover). This book presents the results of a research program on response effects in surveys, conducted jointly by NORC and the Survey Research Lab at the University of Illinois. The inquiry was guided by earlier conceptual work by the authors, which posited three influences on response effects, (1) the nature and structure of the tasks, (2) the characteristics of the interviewer, and (3) the characteristics of the respondent. Furthermore, task variables were grouped into three classes, (a) the method of administration and the structure of the task, (b) the self-presentation of the respondent, and (c) the saliency of the task for the respondent. These dimensions were discussed in relation to non-threatening behavioral questions, threatening behavioral questions, and attitudinal questions. This report focuses on
95
S. Sudman, and Associates. San
the response effects to threatening questions, and is based upon three large samples of the general population. The book both presents results, and suggests procedures that will yield the most accurate responses. The book contains 11 chapters, some of which are adaptations of previously published work. Most of the chapters describe sophisticated research designs, report complex quantitative analyses, and present full documentation in tables. Chapter conclusions and a conclusions-implications final chapter provide valuable summaries in less demanding styles. Two instructive appendices contain the actual questions used in the research. The authors compared face-to-face interviews, telephone surveys, self-administered questionnaires, and
96
BOOK REVIEWS
randomized response methods, and found that no one method is clearly superior. The choice of method can be guided by other considerations, such as cost, access, and feasibility. Changes in question form, however, can increase the amount of reporting of behaviors two or three times over standard formats. The authors assert that long introductions and open-ended response formats “should always” be used in asking about sensitive behavior. Also promising is asking respondents to report the behavior of their close friends. The presence of third parties in the interview situation had little effect. Assurances of confidentiality did improve responses, but signed consent forms should be used only at the end of the interview. Detailed introductions had no adverse impact. Interviewer effects are small in comparison to the importance of question wording. The research produced no evidence that the occurance of nonprogrammed speech (ad libs or probes) affected the quality of responses, even though it happened frequently. Welltrained interviewers might actually improve responses with flexibility and adjustments. On the other hand,
interviewer expectations of difficulty can have negative consequences, and negative thinking interviewers should be retrained or not used. The researchers found that respondents tend to deny engaging in certain behaviors when there are social norms against discussing such topics with strangers. Their test of the Marlowe-Crowne scale suggested that it is not a good measure of individual response effects, but its results should be treated as traits of the individual subjects and should be considered a part of the real differences among people. The details of this book will be interesting to methodologists and survey researchers. Some conventional wisdom about survey methods is confirmed while other beliefs are dismissed. Their conclusions are of general importance to everyone engaged in the use of interviews and questionnaires. They confirm that careful construction is important, but show that well trained and experienced interviewers can make adjustments without destroying the results. T. W. W.