Respondent cooperation in telephone surveys

Evaluafion und Progrunt Plunning, Vol. Printed in the USA. All rights reserved.

RESPONDENT

I I, pp. 135-140,

1988 Copyright

COOPERATION

IN TELEPHONE

0149-7189/88 $3.00 + .oo ‘F: 1988 Pergamon Press plc

SURVEYS

The Effects of Using Volunteer Interviewers

MARC T. BRAVERMAN University

of California,

Davis

ABSTRACT This study reports the effects of using volunteer interviewers on respondent cooperation in telephone surveys. A random, countywide telephone survey focusing on a local youth services progrum was conducted using three subgroups of interviewers: professional program stafA volunteer udults, and volunteer adolescents. No differences were found among the groups on two ttariables: respondent refusals to be interviewed and respondent terminations of interviews in progress. The results support previous findings of modest interviewer effects in telephone surveys. Local evaluators may find the results particularly noteworthy in that volunteer assistance is ojien a significant program resource.

feasible way to conduct a community telephone survey may be to enlist the assistance of volunteers to serve as interviewers. But can this pragmatically based solution, assuming that it is supported by appropriate training and quality controls, be methodologically sound? The present report addresses this question from the perspective of respondent cooperation rate. Most of the previous research on interviewer effects has studied the face-to-face interview setting, and has found effects for nonsensitive material to be small to moderate (Freeman & Butler, 1976; Sudman & Bradburn, 1974). As several researchers (e.g., Dillman, 1978; Groves & Kahn, 1979) point out, there is reason to expect interviewer effects to be smaller in telephone interviews than in in-person interviews. First, the visual cues that operate in an in-person setting are absent in a telephone setting. Second, since the calling is typically done from a central location, there is opportunity for a supervisor to monitor the calls in progress. On the

As the use of telephone surveys gains popularity in evaluation and needs assessment studies, practitioners need to be alert to possible sources of bias associated with the technique. One important issue is the possibility of interviewer effects on respondent cooperation. Several recent studies have found interviewer effects in telephone surveys to be relatively modest (Groves & Magilavy, 1986; Tucker, 1983) but clearly the potential for bias exists. Therefore many survey researchers recommend that experienced interviewers be hired and paid to conduct the interviews (e.g., Fowler, 1984). This solution is not always feasible in evaluation settings. Many evaluators who work with local programs lack the personnel support to use program staff as interviewers, or the financial resources to hire professional interviewers. The resource to which evaluators frequently do have access, for programs that have served well-defined local clientele groups with some longevity, is volunteer support. In these cases, the only

The author wishes to thank Dr. Jana Kay Slater, Dr. Nick I.. Smith, and two anonymous reviewers for their valuable comments on earlier drafts of this paper. The author also wishes to acknowledge the contributions of Ralph Gay, Jeannette George, and Carl Schoner of Yolo Count) Cooperative Extension, in helping to develop and implement the survey study. An earlier version of this paper was presented at the annual meeting of the American Educational Research Association, San Francisco, April 19X6. Requests for reprint\ should be sent to Marc T. Braverman, PhD, Department of Applied Behavioral Sciences, llniversity of California, Da\i\, CA 95616.

135

136

MARC

T. BRAVERMAN

other hand, the generally large number of cases handled by each interviewer can magnify whatcvcr interviewer effects are indeed present. Pre\ ious reports of telephone interviewer effects have not compared the effectiveness of distinct groups of interviewers. Mosr studies have used relatively homogeneous interviewer pools, such as professional interviewers 01. college students enrolled in a course. These studies then examine refusal rates in terms of the natural variations occurring on demographic variables such as sex and age, or on individual difference \,ariables such as prior interviewing experience and voice quality. For instance, the se,ie of the telephone interviewer was found to be unrelated to respondent refusal rates over a scl-ies of studies reported by Dillman, Gallegos, and Frey (1976). The uge of the interviewer, however, was found to influence refusal rate in a study by Singer, Frankel, and Glassman (1983). Using age classifications of 18-21, 22-34, and 35 or older, they found that intervic\vers in the 18-21 age range were somewhat less successful at obtaining completed interviews than those in the two older categories, which did not differ from each other. The effects of prior inferviewing experience on refusal rate were also examined by Singer et al. (1983), using the classifications of no experience, less than one year of experience, and one or more years of experience. Very surprisingly, they found that intervie\vers in the third categorythe most experiencedhad a significantly lower cooperation rate than interviewers in the less experienced categories. They were not able to account for this counterintuitive result. The voice yuu/ir_b? of callers was found to be significantly related to refusal rate by Oksenberg, Coleman, and Cannel1 (1986), although the study considered only female interviewers. Lower refusals were linked with speakers who were judged to have higher auditory pitch, more inflections, faster speech, and more distinct pronunciation. The present study, rather than examining a single dimension, examines the effectiveness of telephone interviewers drawn from different sectors of the organization conducting the survey: professional program staff, adult volunteers, and teenage volunteers. Much of the importance of the comparison lies in its implications for evaluation practice: If empirical results indicate that respondent cooperation rates are robust with respect to these variations, practical possibilities that could be of benefit in numerous program settings would be strengthened. With greater reliance on volunteers, the need for careful task preparation and close supervision would undoubtedly increase. However, the telephone survey format is ideally suited to meet those requirements (e.g., Dillman, 1978). Several of the variables differentiating these three

groups might be hypothesized to have an impact on a respondent’s initial decision either to agree to participate or to refuse an interview. Comparing first the program staff and the adult colunteers, the Singer et al. (1983) demonstrated, may be a detriment to cooperation; (b) differences in Loice quality, which on average might be higher-pitched, le5.s assured, and less distinct -qualities which on balance might reduce cooperation (Oksenberp et al.. 1986); (c) a probably less sophisticated understanding of the whole scope of the program, which might be taxed during ans\vers to unexpected respondent questions; (d) status within the organization as a member rather than as a planner or probidcr of services. Most of these points might be hypothesized as working against the adolescents. On the other hand, several points could be hypothesized to exist in their favor. First, coopet-ation might be stimulated if a respondent reacts to a youthful caller with greater trust, tolerance, or indulgence than he or she would with an adult. Second, particularly \vith regard to the present survey, a young caller might be perceived as a highly appropriate interviewer for a survey about youth programs. Besides the attainment of initial cooperation. an area of potential difference is the ability of the interviewers to sustain an interview to completion. A poorly COIE ducted interview can be terminated by the respondent at any time, simply by hanging up. Dillman et al. (1976) estimate that the percentage of respondents who abandon an interview once it has begun might be as low as two percent. Nevertheless, it is possible that inexperienced interviewers, particularly teenagers, will be especially susceptible to early terminations, resulting in a loss of data and the threat of sample bias. The specific questions addressed in this study are the following: I. Are there differences among professional program staff, volunteer adults, and volunteer adolescents in eliciting the cooperation of telephone survey respondents? 2. Are there differences among these groups in the ability to sustain an interview once it has begun and carry it to completion?

Respondent

Cooperation

The most likely direction for differences in cooperation rate might be program staff (highest), followed by volunteer adults, followed by volunteer adolescents.

in Telephone

Surveys

137

However, as suggested in the analysis above, counterbalancing factors do exist that complicate attempts at prediction.

METHOD Project Description The survey on which the present analysis was based investigated the public’s knowledge and attitudes toward the county 4-H youth program in a predominantly rural county in central California (1985 population = 124,000). The 4-H program provides educational activities for youth from 9-19 years of age, utilizing adult volunteers to lead activities and teach subject matter. The interview asked for information on children in the home, followed by questions on the respondent’s awareness of local 4-H activity offerings, and questions on preferred program characteristics. Thus the content did not deal with respondent lifestyles or other sensitive topics, with the possible exception of questions on the presence of children in the household. The phone interview required from 5 to 10 minutes to complete, depending on the course of the skip patterns. Funds were not available to hire interviewers, and for the survey to be conducted it was necessary to use, as interviewers, volunteers involved with the program. Interviewers were drawn from three distinct groups: professional program staff, adult program volunteers, and teenage 4-H members. The first group had some prior interviewing experience, as well as experience in program planning and research methods. The latter two groups had no experience in these areas. Admittedly, there were reservations about the suitability of the latter two groups, but we decided to proceed with the survey and conduct post hoc analyses that could provide information on the performance of the interviewers. Survey Sample Calculations for desired level of sampling error led to the decision to obtain a sample of approximately 250 households from the county’s overall population. A systematic sample of households was drawn, using all of the phone directories in the county. Eligible respondents included any adult in the household. In the case of no answer or a busy signal, followups were made up to a total of three attempts. Interviewers and Procedure The telephone interviews were conducted in three sessions during a single week in November, 1984. The times chosen were Tuesday evening (6 p.m.-8 p.m.), Wednesday evening (6 p.m.-8 p.m.), and Saturday midday (11 a.m.-l p.m.). These periods present the

greatest probability that potential respondents will be at home, according to previous studies (Vigderhous, 1981; Weeks, Jones, Folsom, & Benrud, 1980). The interviewing team consisted of five professional 4-H staff members, 14 adult volunteer club leaders, and six 4-H members between the ages of 14 and 17. The staff members worked at all three sessions, whereas the adult leaders and youth typically worked at only one or two. The assignments to the three sessions were scheduled in advance. The calls were made from the county 4-H office. Because most of the interviewers were inexperienced, we were particularly careful in structuring the calling sessions. On each occasion we conducted a preliminary training session for the interviewers, familiarizing them with the written protocol, the procedure to follow, and the answers to questions that we anticipated from respondents. All interviewers strictly followed the written script for the introduction and body of the interviews. During calling periods, one of the staff members was always monitoring the calls in progress, to provide informational feedback to the interviewers when appropriate. Records were kept for each calling attempt and its results (completion, refusal, no answer, busy signal, etc.). By the end of the three calling sessions, 241 completed interviews had been obtained. Data Analysis Plan It was planned that the research questions listed above would be examined using chi square analysis. In the first question, the three levels of interviewer classification (program staff, adult volunteer, youth volunteer) would be crossed with 2 levels of respondent action (cooperation, refusal). In the second question, the three levels of interviewer classification would be crossed with 2 levels of completion (full completion, partial completion).’ In the event that either of these tests produced a significant finding, post hoc multiple comparison tests would be performed to identify which comparisons among the six cells were contributing to the significant result. These procedures, described by “‘Cooperation”

is here defined as an agreement by the re\pondcnt,

after the introductory

statement has been read, to Ict the intervieuer

begin questioning. A “refusal“ is a termination by the recpondcnt during or immediately completion”

following

is a termination

first question has been asked.

the introductory

rtatemcnt.

by the respondent

A “partial

sometime after the

13x

MARC

Marascuilo ally

and

bl/lcSwceney

s;traightfor\vard

to lhe ctandard

omnibus

Over

calling

all

made,

three

\+ hich and

answcm,

busy

adult tacts

tests typically

I here

and

The

(both

cases

and

plus

TABLE

COOPERATION

99 no-

but

defined

divided

no

as all

by total

was 2411’340

REFUSAL

OF TELEPHONE

con-

= 71%.

x 1HREE

INTERVIEWER

Respondent Type of Interviewer

Cal15

1

‘JERSUS

CL_ASSIFICATIONS

591

a child

rate,

partial)

used

included

where

refusal\)

in survey

powet

intcrview5,

which

cooperation

full

(complelion~

were

in 241 completed

lines,

\~a$ hotnc.

completions

chi square

I noncontacts,

25

are cornputationinterpretive

wssions,

rcsultcd

refusals,

(1977),

add much

and

‘I-. HRAVERMAN

Cooperntlon

Action Refusal

Total

-Program

staff

69 I72 600)

‘/OkJth

Note.

Percentages

Results

50 (28 7”<1)

174

48

167 6”“)

23 (32 4”n~

il

241

(70 9”0)

99 (29 1”“)

340

llsted

of chl square

Y5

(71 3”o)

Vdbrlk?r

~ntsl

26 (27 J1’,,j

!24

Adult volunteer

in the table refer

test

\.’

1:

525

to row totals NS at (k

05

TABLE 2 FULL VERSUS

PARTIAL

INTERVIEWS. CLASSIFICATIONS

OF TELEPHONE

Full

Type of Inver\/iewer --

Staff

1

INTERVIEWER

Completions

p Value, Totals

67

2

69

117

7

124

Total

184

9

193

Staff Youth

Volunteers

Total Comparison

3

2. Staff

Volunteers

67

2

69

47

1

48

114

3

117

Adult Volunteers

with Youth

117

7

Youth ?/oiunteers

47

1

48

164

8

172

Note. test

Statistical

p =

with Youth

Adult Volunteers

Total

Slgnlflcance

Staff with Adult Volunteers

Adult Volunteers

Comparison

OF

ACROSS

Partial

Completions Comp,lrison

COMPLETIONS

COMPARED

stgnlflcance

was computed

p =

315

(NS)

634

(NS)

Volunteers

124

p =

292

usmg the Fisher

(NS) exact

wwarch.

ha\ c not bwti

l-or

in general

the

mocl

uw-’

part.

houe\,cr,

!hc’\

Respondent Cooperation contrasts were performed using the Fisher exact test, also known as the Irwin-Fisher test (Marascuilo & McSweeney, 1977). This test, which makes no assumptions about expected cell frequencies, computes the probability under the null hypothesis of obtaining a result as extreme or more extreme than the given one. The analysis was broken down into pairwise contrasts because the te5t is most appropriate for 2 x 2 tables. The three planned contrasts resulted from cross-

in Telephone Surveys

139

ing each of the possible pairs of interviewer groups with the dichotomous outcome variables (completion vs. partial completion). To preserve the cy = .05 significance level, each test was performed at (Y = ,017. Table 2 presents results of the Fisher exact tests. None of the contrasts was significant, even at the less conservative (Y = .05 level. Thus, there were no differences found among the groups in the ability to sustain an interview to completion.

DISCUSSION In analyzing (I priori the variables that might influence respondent cooperation, several factors were identified which seemed to support predictions of greater effectiveness for program staff in comparison to adult volunteers. Other variables were identified with respect to adolescent volunteers that could influence this group’s relative effectiveness in either direction, compared to the two adult groups. Despite these group difference variables, no differences among the three groups were found. These results support and extend previous research that has found the telephone interview format to be relatively robust against serious interviewer effects, given adequate preparation and controls (e.g., Groves & Magilavy, 1986; Tucker, 1983). The findings suggest that under certain circumstances, evaluators can enlist volunteersteenagers as well as adults-to conduct the interviews without demonstrably impairing the participation rates. This knowledge can be of benefit to evaluators working with limited resources, where the use of volunteers t-nay represent the only means of making a survey possible. It. may be of particular interest to programs serving the public, in that information about community outreach effectiveness, public knowledge, or public attitudes might constitute important components of outcome evaluation, and these topics are well-suited for telephone surveys. Relevant programs might include youth-serving agencies, job-training programs, family service programs, or media-based community education campaigns. For all interviewing situations, it is essential that participants have adequate skills for the task at hand, and this study must not be interpreted as weakening that requirement. It does suggest that the necessary skills for a specific interviewing situation can be adequately developed with a relatively short, direct, taskappropriate training session. In addition, the opportunity to monitor the interview calls in progress provides an important safeguard against obvious errors. It was our experience, however, that the great majority of the volunteers did not require serious corrections during the course of the interviewing. Respondent cooperation is only one area in which interviewers may have an impact on the quality of a

telephone survey. In deciding whether a planned telephone survey project is appropriate for using volunteers, evaluators should also consider the possibility of increased measurement error due to interviewer effects in the data. The extent of this threat to data quality is largely dependent on the effectiveness of the preliminary training of interviewers, but it may also be affected by the content and format of the interview items themselves. As Tucker (1983) points out, the strength of interviewer effects is believed to be increased if the survey format requires interviewers to classify, interpret, or code the respondents’ answers. In addition, the possibility of interviewer effects may be higher for interview questions that deal with personal or sensitive information. However. recent investigations have rcduced these concerns somewhat, finding telephone interviewer effects due to item format or item sensitivity to be modest under conditions of careful training (Groves & Magilavy, 1986; Singer et al., 1983; Tucker, 1983). Nevertheless, some recommendations regarding survey format may be in order if volunteers, particularly adolescents, will be used as interviewers. The interview should probably be fairly high in structure, and call for straightforward factual information or structured response categories rather than free responses. Evaluators planning surveys that will require interpretation of respondent answers should address these tasks carefully in their preliminary training sessions, and, if possible, provide for a post-session assessment of interviewer skills. In addition, the survey items should deal with nonsensitive content. All of these considerations were met in the survey interview used in the present study; most sur\:eys that are used in program evaluations can probably meet them as well. Several other factors seem not to present serious concern. Groves and Magilavy (1986) found that attitudinal items were not subject to greater interviewer effects than were factual items. They also found that minor variations in feedback given to respondents by interviewers did not increase interviewer variance to a significant degree. An important topic that future studies should address is the effectiveness of volunteer interviewers compared to professional interviewers, such as those

140

MARC

T. BRAVERMAN

employed by survey research centers. Two major areas in which comparability should be studied are respondent cooperation and interviewer effects on data, both of which have already been discussed. A third area for comparing volunteers and professionals is interviewer dependability. Some researchers maintain that volunteers will have low project commitment, leading to problems such as high turnover rate or hurried training (e.g., Fowler, 1984). However, it is likely that the degree of commitment is contingent on several factors related to the specific research setting. Volunteers who

have established prior associations with a program may well demonstrate high dependability and commitment to project completion. This was our experience in the present study; the volunteers proved to be prompt and task-oriented. Furthermore, professional centers are not immune to difficulties in the recruitment and turnover of interviewers (see, for example, Singer et al., 1983). Therefore, generalized recommendations against the use of volunteers based on considerations of dependability may be unwarranted. This issue needs to be examined across a range of program settings.

CONCLUSIONS The possibility of using program participants as volunteer data collectors may be of high interest to evaluators working in settings with limited funding resources. The feasibility of such use for telephone interviews was strengthened by the present study, which indicated that respondent cooperation was not compromised by the use of volunteers-either adults or adolescents-in addition to program staff. However, each setting needs to be carefully examined for possible effects of the strategy on the study’s quality. For researchers planning telephone surveys, unique elements such as regional

characteristics, identity of the sponsoring agency, and the interview content and format will all bear on the appropriateness of using volunteers. In the present study, the survey was characterized by high item structure and nonsensitive item content, qualities which are probably shared by a high proportion of survey projects. These results are clearly encouraging for evaluators wishing to make use of volunteer availability and assistance, which can be a significant program resource.

REFERENCES h1.4RASC‘U1LO,

I .A.,

& McSWEENEY.

(.‘A:

rc~iences.

Uelmon

t,

Wadsworth.

OKSENBERG,

I..,

Interviewen Opinion

COLEMAN,

I:.,

50, Y7- I

FRANKEI

of interviewer

I’rthlic

Opinion

SUDMAN.

‘I IJC‘KER.

8r C;L.ASSMAN.

M.I..,

M.B.

and expectation\

N.M.

Interviewer

(19X3). The on teqx~nse.

(1974).

Ke.r~on.w

c@cts

,/,

Puhlk

JONES, Optimal

effects

in telephone

surceys.

Puh-

47, 84-Y5.

C;. (lY81).

patterns.

(1980).

Puhlfc,

Aldine

Quarlrrl_v,

VIGD~RHOUS,

opinion

, E.R.,

characteristics

Cr. (1983).

of wawnal

(1986).

survey\.

II

S., & BRADBURN,

lit, Opinion

C‘.f.

rates in telephone

Quuriert_v, 47. 6X-83.

r~,rvqvs. C’hicago:

WEEKS,

L.. Sr CANNELL,

vo~ccs and refuwl

Qucrrtertv.

SINGER, effect

C‘.H.

Nonpramer-

M. (lY77).

!-I(’ crt~i rti.srrrA~tlion-,fr~(~ methorf.~ .for [he soc%d

Scheduling Opinion

B.L.,

telephone

FOLSOM,

times to contact

Quur/er/y, 44, IO1 - I 14.

intenkws:

A study

Quurtert_v, 45, 250 -259. R.E., sample

JR., & BENRUD, houceholds.

Public