- Email: [email protected]
The overlooked ingredient in survey research
CHARLES
S. M A Y E R
The Overlooked Ingredient in Survey Research 75
A
by-product of the general acceptance of the marketing concept is an increasing reliance on marketing research. Since individual research requirements are not normally large, marketers rely on research suppliers for their informational needs. As research buyers, they encounter many problems, including the crucial question of the quality of the field work. Most research buyers agree with the adage that, no matter how sophisticated the research design, how elegant the analysis, The author is on leave in England during I966.67 ~ from the Amos Tuck School, Dartmouth, where he is Associate Professor of Business Admlnistra. tlon. The article is based on a presentation to a recent consumer marketing research semlnar ln Montreal.
FALL; 1966
or how incisive the final report, the results of marketing research are no better than the interviewing that generates the data. The accusation often aimed at the computergarbage-in, garbage-out-has a basic application to all research efforts. Yet, what are research buyers doing to ensure quality interviewing? What is meant by quality interviewing? How is such quality achieved? What can a research buyer do to be sure that the actual field work satisfies his quality requirements? This paper attempts to answer these questions by focusing on consumer surveys in which personal interviews are conducted in the home. Personal interviews are normally used if accurate and detailed data are needed. Accuracy is a function of both sampl6 design and the implementation of the study. The former affects sampling
CHARLES S. MAYER
76
error, while the latter is reflected in nonsampling error. Because of easy quantification, much of the imaginative work in research has been directed toward reduction of sampling error. Little attention has been paid to nonsampling error since no neat theoretical framework exists for its assessment. Unfortunately, nonsampling error can be, and often is, greater than sampling error. Interviewing errors are, in turn, an important component of nonsampling error. The conscientious researcher cannot assume an ostrich-like posture and make believe that interviewing errors are small, under control, and hence not worthy of attention. Interviewing errors may be large and are related to interviewing quality. As long as interviewing quality remains an overlooked ingredient in marketing research, the results obtained are likely to be faulty and hence may be misleading.
QUALITY INTERVIEWING Quality interviewing is easy to define, yet very dittlcult to implement. It simply means that the interviewer has followed instructions and conducted the interview as requested. Instructions in this instance refer to both general training on how to interview and specific instructions for a particular study. Such a definition of quality burdens the
research designer; instructions must be given keeping in mind who the interviewers are and what they have to do. Because many people at both the research design and buying stages have had no interviewing experience, they have little appreciation for the difficulties of interviewing and little sensitivity for its importance. Without such experience, both the instructions and the questionnaire may be unrealistic, and the quality of interviewing is bound to suffer. THE INTEnVIEWEI~S
The single outstanding characteristic of interviewing is that it is a part-time job. It also happens to be a very low-paid part-time job, with wages running around $1.60 per hour. 1 This figure compares with a median wage of approximately $2.25 for stenographers (general) or switchboard operators. 2 As a part-time job, interviewing mainly attracts housewives who use this means to supplement their family income, and others who look on interviewing as a secondary job, such as students or schoolteachers. In each locale, the interviewers are generally organized under one or more local resident supervisors who range in qualification from operators of an interviewing service with many years of research training and experience to interviewers with a few friends, Contact with interviewers is norreally made through these supervisors. The research buyers may not know, or care, who the interviewers are, how they are trained, or how well they conduct their assignments. It is no wonder that the interviewers are tempted to cut corners under such circumstances, since the iob is a dfl~cult one. 1 Charles S. Mayer, "An Exploratory Survey of the Compensation, Supervision and Training Praet_ices of Marketing Research Agencies," Amos Tuck School Working Paper, mimeographed, March, 1965; see also "Audit of Interviewer Pay Scales," Market Facts-New York, Inc., mimeographcxl, December, 1965. z U.S. Department of Labor, Nat{onal Survey of
Pro[essional, Administrative, Technical and Clerical Pay (Bulletin No. 1286; Washington: U.S. Gov't Printing Off]c_e, 1964), p. 16.
BUSINESS HORIZONS
OVERLOOKED INGREDIENT IN SURVEY RESEARCH
THE INTElWlEWEffS JOB In personal interviewing, the interviewer calls on households and attempts to complete interviews. She (acknowledging that most interviewers are female) must work under all conditions-rain, sleet, or snow. Her work may lead her to tough slum areas where she may risk abuse or even assault for very little information. To add to her frustrations, she may be looking for the proverbial needle in the haystack ff the study involves a product of low incidence, and usually she is under time pressure to complete her assignment. Given this setting, the interviewer is expected to obtain interviews from specifically designated respondents, ask questions exactly as worded, and record responses verbatim. Since she works in the field by herself, she has plenty of opportunity to compromise sampling instructions, to reword questions, or to record her own interpretation of the respondent's words. Obviously, she also has the opportunity to falsify part or all of the interview. If interviewers give in to these temptations, even if only sporadically, their actions can erode the value of the information. Interviewers are not a source of error because they are basically dishonest. On the contrary, they generally try to do a good job. But, if they get into a situation that is unclear to them, or ff they are asked to do an impossible job, they have no alternative but to take shortcuts. A/so, ff they feel that the research buyer does not care about how the interviews are conducted, they may develop bad habits that will carry over to all of their work. Unfortunately, many research buyers show very little concern for what goes on in the field.
THE EROSION OF INTERVIEWING QUALITY Interviewing quality is compromised ff the interviewer does not follow instructions and complete the interview as requested. There
F A L L , 1966
are numerous ways in which an interviewer may deviate from requirements. Some of these represent willful cheating while others reflect poor training or lack of understanding of the survey process; they all may result in meaningless survey findings. TOTAL FABRICATION
Although much of the literature on interviewer error focuses on outright fabrication of interviews, this problem is the least prevalent of all forms of cheating. Even very crude methods of validation, such as postcards, are sufficiently sensitive to detect the interviewer who fails to make contact with the respondent. For this reason, some interviewers have found much more subtle and insidious ways to cheat. PADDING OF BILLS
Padding expenses by falsifying the number of hours worked or number of miles traveled has been, and continues to be, a major factor in the rising cost of field work. This practice is widespread in interviewing for many reasons. First, the nature of the interviewing job dictates that the interviewer works without any direct supervision. It is very easy for her to add an hour here and there. When time standards are absent, which is the more frequent case, padding is a simple chore with minimum danger of detection. Moreover, if padding becomes widespread, meaningful time standards become difficult to establish. The low level of compensation only adds to the temptation to pad. The interviewer may feel, and perhaps rightly so, that her work is worth more than her pay. It is easy for her to rationalize that through padding she is merely getting what she deserves. Certainly, ff she is not challenged the first time, this tendency can become a permanent habit. Although the supervisor is best situated to challenge padding, she may not do so since her compensation is normally geared
77
CI-IABLES S. MAYER
to a percentage of the interviewers" charges. Thus, the higher the interviewers' bills, the m o r e money the supervisor tends to make. Such a compensatory system is not only illogical, but actually acts as a negative incentive on supervisory control of interviewer costs. While padding, as such, has no ciirect effect on the reliability of the information, both cost and reliability are used to evaluate the efficiency (value) of the information gathered. Padding of bills directly reduces the eificiency of the information because it increases the cost. Furthermore, padding of bills may be the first step toward the develOpment of a cheater syndrome. It is a form of dishonesty that may lead to more serious forms of cheating-cheating that affects the reliability of the information as well. Reed and Platten put it this way: the first time an interviewer falsifies a time record he is not only on the way to becoming a cheater, he is a cheater, The system of payment, fhetef0re, is likely to force the best interviewers into little compromises with complete honesty3 • . .
These compromises, ff undetected, have the tendency to grow.
tionnaires are long, the fatigue factor will result in incomplete or hurried responses. Rigidly specified and enforced respondent selection procedures, such as those used in probability sample designs, could eliminate much of this problem, but only at an increase in the cost of the research. However, another problem, which may also stem from interviewer overload, could still persist-asking key questions only. ASKING KEY QUESTIONSONLY
One way to avoid detection while cheating is to actually interview the desired respondent, but to ask only key questions in the questionnaire, filling in the rest at a later time. The traditional validation procedures, such as postcards or even felephone calls, are inadequate i n most circumstances to catch such cheating , Overloading t h e interviewer is one way of forcing her to try this technique. Boring, dull, or repetitious questionnaires are others. Yet, once t h e interviewer has tried using the short form questionnaire on one study, there is very little to stop her from continuing this practice~
WORKING ON MORE THAN ONE STUDY AT A TIME
CHANGING THE
In order to shelter themselvesffom the se-
Vere fluctuations that often occur, in thc~ Work load, interviewers generally w6rk f0r more~than one research organization, if two field jobs of two different companies a r e alerted for the same time, the temptatign is great to accept both. Working on more than one survey at a time creates too much work for the interviewer and can lead to serious problems. For example, thesame respondent can be asked to respond to two questionnaires back to back. Particularly if the ques~z
3 w. P. Beed and J. H. Platten, Jr.; "Smhceyon the problem of Interviewer Cheating," Intema:tiorial IournaI of Opinion and Attitude tteseamh
(September, 1947), p. 102.
;:.:°'
:":", ,
.SuRvEY VEttICLE
Another shortcut available to the interviewer is to change the method of interviewing. For example, an interviewer could conduct a persona] interview survey by telephone. Some questions may remain unasked, others may be altered, but an apparently complete interview could be obtained. Moreover, the interview would pass most validation testsi Such a change in the survey vehicle, how: ever, could substantially affect both the accuracy and the detail of the information collected. An alternative way of economizing interviewer time through changing the survey technique is to interview respondents in groups:.: Clearly, the opinions and attitudes of one individual in the group Can have a
BUSINESS HORIZONS
OVERLOOKED INGIIEDIENT IN SL!BVEY RESEABCH
strong influence on the reported opinions and attitudes of the others. A group interview will generally net less information in a structured interview situation than individual interviews with members of the group. Interviewers may also change survey results by selecting respondents on other qualifications than the survey requirements. For example, respondents may be selected because they were congenial on previous interviews or because they are friends of the interviewer. Such specially selected respondents are likely to be atypical, as are their responses.
T A L K I N G TO THE W R O N G RF~PONDENT
Part of the interviewer's assignment is to talk to specified persons. For example, she may have to talk to the person who is responsible for the maintenance of the family car. If she talks to another, more easily accessible member of the family, she may get answers, but they may not be meaningful. While respondent forcing-the process of making respondents fit assigned characteristics-could occur in both probability and quota samples, it is most prevalent toward the end of a difficult quota assignment. IMPROVISING
QUESTIONS
While not a form of overt cheating, question improvisation can also distort survey results. It usually occurs ff the questions are not well written, are diffcult or embarrassing to ask, or are simply repetitious. Improvisation occurs as much because of inadequate interviewer instruction as because of poor questionnaire design. For example, if the interviewer is not told the reason for asking two apparently similar questions, she may combine them into one. The interviewer may also alter questions for the convenience of the respondent. For example, if a respondent fails to comprehend a question the first time, the poorly trained interviewer may re-
FALL, 1966
phrase the question in a more simple, but different, form. The result-unknown to the analyst-will be an answer to a different question. FORCING OR EXPANDING ANSWERS
It is perfectly legitimate for respondents not to have opinions on certain subjects or not to be able to express what they mean by certain statements. Interviewers, knowing that they may be evaluated on completeness of the responses they obtain, may be tempted to force opinions, to put words into the respondent's mouth, or to expand the respondent's comments to reflect what they think the respondent meant to say. While they may do so to be helpful, the net result is still a meaningless and misleading answer. T H E USE OF UNTRAINED IN15~BVIEVCEIRS
Supervisors receive their compensation for two basic reasons-for having a trained staff and for actually supervising a job to be sure that study specifications are met. If either of these basic supervisory ingredients is missing, the study is unlikely to give accurate results. Training of interviewers varies from offce and on-the-job training by a competent supervisor to qualifying an interviewer as "trained" for sending her name (and money) to a service that then distributes (for a fee) lists of "trained interviewers." The latter practice can only exist ff the research buyers pay very little attention to the quality of field work done for them# Often it is not economical for the typical small research operation to investigate the quality of supervision they receive in each location. Thus they may unknowingly (or perhaps even knowingly) employ untrained, unqualified, and inexperienced interviewers. 4 H. M. Weisberg, "Empire Associates Indicted,"
View Points, VI (April, 1965), p. 8.
79
CHARLES S. MAYER
UNDONE BUT P.,~D-FoR SUPEnVISmN
Ford Sammis has listed the following s e r v ices that a supervisor should perform to earn her fee: 1. Line up and instruct competent interviewers whose ability can vouch for their personal knowledge. 2. Supervise their work thoroughly to see that sample and questionnaire requirements are met in a professional manner. 3. Edit all completed work and see that omissions and errors are corrected promptly. 4. Verify a reasonable percentage of all interviews by telephone or mail. 5. Pay all interviewers and send only one allinclusive bill to the client for the job.5
80 ¸
Yet many supervisors do no more than send work to untrained interviewers and charge a fee for supervision. This practice is only possible when the client allows it to happen.
when buyers do not verify the reliability of the information bought, research suppliers can develop potentially dangerous habits.
REQUIREMENTS FOR QUALITY INTERVIEWING
The two most obvious requirements for quality interviewing are capable personnel and effective controls. All the controls in the world cannot make incompetent people do good work. Therefore, a research buyer must first look at the people who do the interviewing. How were they recruited? How were they selected? How were they trained? Second, there must be evidence of an effective control system in operation. These features will be discussed in turn. THE INTERVIEWERS: H o w GOOD A~E THEY?
THE CURRENT TREND
Many of the above practices are becoming more prevalent in marketing research today. There are several reasons for this trend. First, as more people become aware of t h e potentialities of marketing research, they too want to avail themselves of its benefits. However, because they lack any real appreciation for research methodology, it has become possible for a number of marginal operators to establish themselves as research suppliers. The marginal operators have one distinct advantage over the more meticulous researchers-cost. As long as research buyers continue to "buy research by the pound," they help to perpetuate a system that produces inaccurate and unreliable marketing information. Second, research buyers have not insisted on examining the records of research suppliers to see that quality requirements are being met. In other words, 5 Ford Sammis, "Are the Field Mice Getting Foxy?" in W. K. Dolva, ed., Marketing Keys to Profits in the 1960"s (Chicago: American Marketing Association, 1959), p. 306.
It is important to find out who the interviewers are. Are they selected, trained, and supervised directly by the research supplier, or is the work merely subcontracted to local supervisors? There exists presently a pool of haphazardly selected and trained interviewers, who work for a coterie of local supervisors, and are used by many research suppliers. Work turned out by this group often falls substantially short of quality interviewing. A research buyer can quickly satisfy himself about the capabilities of a field staff by visiting the research supplier and learning about the methods of recruiting, selection, and training. While most suppliers will claim that they hire people individually, their files may only show the names of supervisors-an obvious discrepancy. Similarly, most suppliers will claim that they have a training program for interviewers. Yet they may only be able to produce an out-of-date training manual or, sometimes, no manual at all. The existence of a manual is not proof that training is given, but its absence is certairdy discrediting. A visit to a location to
BUSINESS HORIZONS
OVERLOOKED .INGREDIENTIN SURVEy RESEARC~
sit in on a briefing session or to accompany an interviewer will further confirm how capable and thoroughly trained the interviewers are. A closely related subject to selection, training, and control is the type of information kept by the research supplier on its interviewers. Some of the meaningful specifics are the amount and kind of information available, the date of the last entry, and the feedback to the interviewers on how they are performing. Again, a brief examination of the supplier's records can be enlightening. CONTROLS
Validation procedures are an important control on interviewing quality. Most research firms do no more than send out postcards to a sample of respondents to see ff the interview was actually conducted. This crude technique will only detect the interviewer who totally falsifies the interview. Some firms also insist that the local resident supervisor validate a proportion of the interviews by phone. While this system is not foolproof, since the supervisor may wish to cover up any cheating by her interviewers, its existence can be checked by asking to see validation records. The most effective method of validation available to a research supplier is to conduct telephone interviews from the home office. With the advent of low evening rates and Wide Area Telephone Service (WATS), validation from the home office is economically feasible. From the research buyer's point of view, the only unquestionable technique is to validate directly by direct telephone. In this way, the buyer can convince himself that specifications were followed exactly. Moreover, informing the supplier and the interviewers that this method of validation will be used should be a deterrent to poor quality in itself. An alternative to direct validation by the research buyer is to use an impartial third party to conduct the validation. 6
FALL, 1966
•
Other controls include the presence and use of cost data, call records, and listing sheets by the supplier. Through these forms, the supplier can trace the movements of individual interviewers. The research buyer can assure himself that these controls are being exercised through an inspection of the supplier's operating procedures. THE NEED FOR CONCERTED ACTION
Interviewing is an important ingredient in marketing research; unless the research buyer pays particular attention to it, interviewing quality can be eroded in many ways. All that the individual research buyer can do, however, is to specify research of high quality, to be willing to pay for it, and to look for symptons of its presence or absence. Most research buyers have too small a research requirement to exert any significant leverage on the over-all practices in the field. Only through concerted efforts can research buyers be sure of receiving quality interviewing on a continuous basis. They might, for example, give general support to the Advertising Research Foundation's proposed Field Audit and Completion Test (FACT). Under this system, the nth interview of each interviewer would be checked by an ~U~F-eontrolled telephone facility to determine whether the interview was made in the first place, and ff so, whether specific questions had been asked. If trouble were indicated, further checking would be initiated. The hoped-for end result would be a system of uniform validation and interviewer certification with the costs of the program built into every research budget, r Should such a program not materialize, the research buyers could set up a jointly owned, consolidated field service. A consolis Dik W. Twedt, "Is an 'Audit Bureau of Interviews' Needed Now?" Journal of Marketing, XXX (April, 1966), pp, 59-60. "Proposed ARF Audit of Field Interviews Controversial: Dodge," Advertising Age, XXXVII (May 23, 1966), p. 46.
81
CnAnLESS. MAYE~a
82
dated field service could benefit greatly from economies of scale. Such consolidation would make the work offered by the field service su~ciently important to exercise considerable leverage over supervisors and interviewers. Also, with sufficient work, the consolidated field service could afford to hire professional management and retain local resident supervisors on a permanent basis. The consolidated service could maintain a national pool of trained interviewers. These interviewers would be guaranteed a ~minimum amount of work per week and would be paid at a higher rate than interviewers receive currently, Raising rates without adequate controls would achieve little as far as padding is concerned. However, a consolidated service would also be able to establish standards and increase controls. It may even eliminate padding to such an extent that costs would be reduced, though a higher rate were paid to interviewers.
Such consolidation would not have to be anticompetitive. Competition could still be maintained in survey design and data analysis. Moreover, with careful planning, proprietary data couM be kept secret. OVEn THE past years many people have talked about the importance of quality interviewing. Unfo~unately, we are still in the talking stage. Marketing management and research directors have failed to realize the danger of inadequate field work. Until they do, and are prepared to pay for good work, the present interviewer problem will persist. Individual research buyers may be able to measure the presence of quality in interviewing, but they can do very little to cure the general problems in the field. Such cure can only come by concerted action through professional organizations , or through consolidating the research requirements of several research buyers.
i
W E CA~ never eliminate competition. We can only change the rules of the game, and the pay-off. Competition is to be found in the subdued and pious Quaker meeting just as surely as it is on the bloodiest of battlefields. The devices of love may be found in its arsenal side by side with weapons of steel. Our problem is not to avoid the unavoidable-competition-but to choose our weapons. In seeking the means that are most commensurate with human comfort, pleasure and dignity we cannot necessarily trust first impressions or traditional moral standards. We will need the deepest insights of psychology and anthropology to enable us to choose well. --Garrett Hardin NATURE AND MAN'S FATE
r d
.
.
.i
.
BUSINESS HORIZONS
S. M A Y E R
The Overlooked Ingredient in Survey Research 75
A
by-product of the general acceptance of the marketing concept is an increasing reliance on marketing research. Since individual research requirements are not normally large, marketers rely on research suppliers for their informational needs. As research buyers, they encounter many problems, including the crucial question of the quality of the field work. Most research buyers agree with the adage that, no matter how sophisticated the research design, how elegant the analysis, The author is on leave in England during I966.67 ~ from the Amos Tuck School, Dartmouth, where he is Associate Professor of Business Admlnistra. tlon. The article is based on a presentation to a recent consumer marketing research semlnar ln Montreal.
FALL; 1966
or how incisive the final report, the results of marketing research are no better than the interviewing that generates the data. The accusation often aimed at the computergarbage-in, garbage-out-has a basic application to all research efforts. Yet, what are research buyers doing to ensure quality interviewing? What is meant by quality interviewing? How is such quality achieved? What can a research buyer do to be sure that the actual field work satisfies his quality requirements? This paper attempts to answer these questions by focusing on consumer surveys in which personal interviews are conducted in the home. Personal interviews are normally used if accurate and detailed data are needed. Accuracy is a function of both sampl6 design and the implementation of the study. The former affects sampling
CHARLES S. MAYER
76
error, while the latter is reflected in nonsampling error. Because of easy quantification, much of the imaginative work in research has been directed toward reduction of sampling error. Little attention has been paid to nonsampling error since no neat theoretical framework exists for its assessment. Unfortunately, nonsampling error can be, and often is, greater than sampling error. Interviewing errors are, in turn, an important component of nonsampling error. The conscientious researcher cannot assume an ostrich-like posture and make believe that interviewing errors are small, under control, and hence not worthy of attention. Interviewing errors may be large and are related to interviewing quality. As long as interviewing quality remains an overlooked ingredient in marketing research, the results obtained are likely to be faulty and hence may be misleading.
QUALITY INTERVIEWING Quality interviewing is easy to define, yet very dittlcult to implement. It simply means that the interviewer has followed instructions and conducted the interview as requested. Instructions in this instance refer to both general training on how to interview and specific instructions for a particular study. Such a definition of quality burdens the
research designer; instructions must be given keeping in mind who the interviewers are and what they have to do. Because many people at both the research design and buying stages have had no interviewing experience, they have little appreciation for the difficulties of interviewing and little sensitivity for its importance. Without such experience, both the instructions and the questionnaire may be unrealistic, and the quality of interviewing is bound to suffer. THE INTEnVIEWEI~S
The single outstanding characteristic of interviewing is that it is a part-time job. It also happens to be a very low-paid part-time job, with wages running around $1.60 per hour. 1 This figure compares with a median wage of approximately $2.25 for stenographers (general) or switchboard operators. 2 As a part-time job, interviewing mainly attracts housewives who use this means to supplement their family income, and others who look on interviewing as a secondary job, such as students or schoolteachers. In each locale, the interviewers are generally organized under one or more local resident supervisors who range in qualification from operators of an interviewing service with many years of research training and experience to interviewers with a few friends, Contact with interviewers is norreally made through these supervisors. The research buyers may not know, or care, who the interviewers are, how they are trained, or how well they conduct their assignments. It is no wonder that the interviewers are tempted to cut corners under such circumstances, since the iob is a dfl~cult one. 1 Charles S. Mayer, "An Exploratory Survey of the Compensation, Supervision and Training Praet_ices of Marketing Research Agencies," Amos Tuck School Working Paper, mimeographed, March, 1965; see also "Audit of Interviewer Pay Scales," Market Facts-New York, Inc., mimeographcxl, December, 1965. z U.S. Department of Labor, Nat{onal Survey of
Pro[essional, Administrative, Technical and Clerical Pay (Bulletin No. 1286; Washington: U.S. Gov't Printing Off]c_e, 1964), p. 16.
BUSINESS HORIZONS
OVERLOOKED INGREDIENT IN SURVEY RESEARCH
THE INTElWlEWEffS JOB In personal interviewing, the interviewer calls on households and attempts to complete interviews. She (acknowledging that most interviewers are female) must work under all conditions-rain, sleet, or snow. Her work may lead her to tough slum areas where she may risk abuse or even assault for very little information. To add to her frustrations, she may be looking for the proverbial needle in the haystack ff the study involves a product of low incidence, and usually she is under time pressure to complete her assignment. Given this setting, the interviewer is expected to obtain interviews from specifically designated respondents, ask questions exactly as worded, and record responses verbatim. Since she works in the field by herself, she has plenty of opportunity to compromise sampling instructions, to reword questions, or to record her own interpretation of the respondent's words. Obviously, she also has the opportunity to falsify part or all of the interview. If interviewers give in to these temptations, even if only sporadically, their actions can erode the value of the information. Interviewers are not a source of error because they are basically dishonest. On the contrary, they generally try to do a good job. But, if they get into a situation that is unclear to them, or ff they are asked to do an impossible job, they have no alternative but to take shortcuts. A/so, ff they feel that the research buyer does not care about how the interviews are conducted, they may develop bad habits that will carry over to all of their work. Unfortunately, many research buyers show very little concern for what goes on in the field.
THE EROSION OF INTERVIEWING QUALITY Interviewing quality is compromised ff the interviewer does not follow instructions and complete the interview as requested. There
F A L L , 1966
are numerous ways in which an interviewer may deviate from requirements. Some of these represent willful cheating while others reflect poor training or lack of understanding of the survey process; they all may result in meaningless survey findings. TOTAL FABRICATION
Although much of the literature on interviewer error focuses on outright fabrication of interviews, this problem is the least prevalent of all forms of cheating. Even very crude methods of validation, such as postcards, are sufficiently sensitive to detect the interviewer who fails to make contact with the respondent. For this reason, some interviewers have found much more subtle and insidious ways to cheat. PADDING OF BILLS
Padding expenses by falsifying the number of hours worked or number of miles traveled has been, and continues to be, a major factor in the rising cost of field work. This practice is widespread in interviewing for many reasons. First, the nature of the interviewing job dictates that the interviewer works without any direct supervision. It is very easy for her to add an hour here and there. When time standards are absent, which is the more frequent case, padding is a simple chore with minimum danger of detection. Moreover, if padding becomes widespread, meaningful time standards become difficult to establish. The low level of compensation only adds to the temptation to pad. The interviewer may feel, and perhaps rightly so, that her work is worth more than her pay. It is easy for her to rationalize that through padding she is merely getting what she deserves. Certainly, ff she is not challenged the first time, this tendency can become a permanent habit. Although the supervisor is best situated to challenge padding, she may not do so since her compensation is normally geared
77
CI-IABLES S. MAYER
to a percentage of the interviewers" charges. Thus, the higher the interviewers' bills, the m o r e money the supervisor tends to make. Such a compensatory system is not only illogical, but actually acts as a negative incentive on supervisory control of interviewer costs. While padding, as such, has no ciirect effect on the reliability of the information, both cost and reliability are used to evaluate the efficiency (value) of the information gathered. Padding of bills directly reduces the eificiency of the information because it increases the cost. Furthermore, padding of bills may be the first step toward the develOpment of a cheater syndrome. It is a form of dishonesty that may lead to more serious forms of cheating-cheating that affects the reliability of the information as well. Reed and Platten put it this way: the first time an interviewer falsifies a time record he is not only on the way to becoming a cheater, he is a cheater, The system of payment, fhetef0re, is likely to force the best interviewers into little compromises with complete honesty3 • . .
These compromises, ff undetected, have the tendency to grow.
tionnaires are long, the fatigue factor will result in incomplete or hurried responses. Rigidly specified and enforced respondent selection procedures, such as those used in probability sample designs, could eliminate much of this problem, but only at an increase in the cost of the research. However, another problem, which may also stem from interviewer overload, could still persist-asking key questions only. ASKING KEY QUESTIONSONLY
One way to avoid detection while cheating is to actually interview the desired respondent, but to ask only key questions in the questionnaire, filling in the rest at a later time. The traditional validation procedures, such as postcards or even felephone calls, are inadequate i n most circumstances to catch such cheating , Overloading t h e interviewer is one way of forcing her to try this technique. Boring, dull, or repetitious questionnaires are others. Yet, once t h e interviewer has tried using the short form questionnaire on one study, there is very little to stop her from continuing this practice~
WORKING ON MORE THAN ONE STUDY AT A TIME
CHANGING THE
In order to shelter themselvesffom the se-
Vere fluctuations that often occur, in thc~ Work load, interviewers generally w6rk f0r more~than one research organization, if two field jobs of two different companies a r e alerted for the same time, the temptatign is great to accept both. Working on more than one survey at a time creates too much work for the interviewer and can lead to serious problems. For example, thesame respondent can be asked to respond to two questionnaires back to back. Particularly if the ques~z
3 w. P. Beed and J. H. Platten, Jr.; "Smhceyon the problem of Interviewer Cheating," Intema:tiorial IournaI of Opinion and Attitude tteseamh
(September, 1947), p. 102.
;:.:°'
:":", ,
.SuRvEY VEttICLE
Another shortcut available to the interviewer is to change the method of interviewing. For example, an interviewer could conduct a persona] interview survey by telephone. Some questions may remain unasked, others may be altered, but an apparently complete interview could be obtained. Moreover, the interview would pass most validation testsi Such a change in the survey vehicle, how: ever, could substantially affect both the accuracy and the detail of the information collected. An alternative way of economizing interviewer time through changing the survey technique is to interview respondents in groups:.: Clearly, the opinions and attitudes of one individual in the group Can have a
BUSINESS HORIZONS
OVERLOOKED INGIIEDIENT IN SL!BVEY RESEABCH
strong influence on the reported opinions and attitudes of the others. A group interview will generally net less information in a structured interview situation than individual interviews with members of the group. Interviewers may also change survey results by selecting respondents on other qualifications than the survey requirements. For example, respondents may be selected because they were congenial on previous interviews or because they are friends of the interviewer. Such specially selected respondents are likely to be atypical, as are their responses.
T A L K I N G TO THE W R O N G RF~PONDENT
Part of the interviewer's assignment is to talk to specified persons. For example, she may have to talk to the person who is responsible for the maintenance of the family car. If she talks to another, more easily accessible member of the family, she may get answers, but they may not be meaningful. While respondent forcing-the process of making respondents fit assigned characteristics-could occur in both probability and quota samples, it is most prevalent toward the end of a difficult quota assignment. IMPROVISING
QUESTIONS
While not a form of overt cheating, question improvisation can also distort survey results. It usually occurs ff the questions are not well written, are diffcult or embarrassing to ask, or are simply repetitious. Improvisation occurs as much because of inadequate interviewer instruction as because of poor questionnaire design. For example, if the interviewer is not told the reason for asking two apparently similar questions, she may combine them into one. The interviewer may also alter questions for the convenience of the respondent. For example, if a respondent fails to comprehend a question the first time, the poorly trained interviewer may re-
FALL, 1966
phrase the question in a more simple, but different, form. The result-unknown to the analyst-will be an answer to a different question. FORCING OR EXPANDING ANSWERS
It is perfectly legitimate for respondents not to have opinions on certain subjects or not to be able to express what they mean by certain statements. Interviewers, knowing that they may be evaluated on completeness of the responses they obtain, may be tempted to force opinions, to put words into the respondent's mouth, or to expand the respondent's comments to reflect what they think the respondent meant to say. While they may do so to be helpful, the net result is still a meaningless and misleading answer. T H E USE OF UNTRAINED IN15~BVIEVCEIRS
Supervisors receive their compensation for two basic reasons-for having a trained staff and for actually supervising a job to be sure that study specifications are met. If either of these basic supervisory ingredients is missing, the study is unlikely to give accurate results. Training of interviewers varies from offce and on-the-job training by a competent supervisor to qualifying an interviewer as "trained" for sending her name (and money) to a service that then distributes (for a fee) lists of "trained interviewers." The latter practice can only exist ff the research buyers pay very little attention to the quality of field work done for them# Often it is not economical for the typical small research operation to investigate the quality of supervision they receive in each location. Thus they may unknowingly (or perhaps even knowingly) employ untrained, unqualified, and inexperienced interviewers. 4 H. M. Weisberg, "Empire Associates Indicted,"
View Points, VI (April, 1965), p. 8.
79
CHARLES S. MAYER
UNDONE BUT P.,~D-FoR SUPEnVISmN
Ford Sammis has listed the following s e r v ices that a supervisor should perform to earn her fee: 1. Line up and instruct competent interviewers whose ability can vouch for their personal knowledge. 2. Supervise their work thoroughly to see that sample and questionnaire requirements are met in a professional manner. 3. Edit all completed work and see that omissions and errors are corrected promptly. 4. Verify a reasonable percentage of all interviews by telephone or mail. 5. Pay all interviewers and send only one allinclusive bill to the client for the job.5
80 ¸
Yet many supervisors do no more than send work to untrained interviewers and charge a fee for supervision. This practice is only possible when the client allows it to happen.
when buyers do not verify the reliability of the information bought, research suppliers can develop potentially dangerous habits.
REQUIREMENTS FOR QUALITY INTERVIEWING
The two most obvious requirements for quality interviewing are capable personnel and effective controls. All the controls in the world cannot make incompetent people do good work. Therefore, a research buyer must first look at the people who do the interviewing. How were they recruited? How were they selected? How were they trained? Second, there must be evidence of an effective control system in operation. These features will be discussed in turn. THE INTERVIEWERS: H o w GOOD A~E THEY?
THE CURRENT TREND
Many of the above practices are becoming more prevalent in marketing research today. There are several reasons for this trend. First, as more people become aware of t h e potentialities of marketing research, they too want to avail themselves of its benefits. However, because they lack any real appreciation for research methodology, it has become possible for a number of marginal operators to establish themselves as research suppliers. The marginal operators have one distinct advantage over the more meticulous researchers-cost. As long as research buyers continue to "buy research by the pound," they help to perpetuate a system that produces inaccurate and unreliable marketing information. Second, research buyers have not insisted on examining the records of research suppliers to see that quality requirements are being met. In other words, 5 Ford Sammis, "Are the Field Mice Getting Foxy?" in W. K. Dolva, ed., Marketing Keys to Profits in the 1960"s (Chicago: American Marketing Association, 1959), p. 306.
It is important to find out who the interviewers are. Are they selected, trained, and supervised directly by the research supplier, or is the work merely subcontracted to local supervisors? There exists presently a pool of haphazardly selected and trained interviewers, who work for a coterie of local supervisors, and are used by many research suppliers. Work turned out by this group often falls substantially short of quality interviewing. A research buyer can quickly satisfy himself about the capabilities of a field staff by visiting the research supplier and learning about the methods of recruiting, selection, and training. While most suppliers will claim that they hire people individually, their files may only show the names of supervisors-an obvious discrepancy. Similarly, most suppliers will claim that they have a training program for interviewers. Yet they may only be able to produce an out-of-date training manual or, sometimes, no manual at all. The existence of a manual is not proof that training is given, but its absence is certairdy discrediting. A visit to a location to
BUSINESS HORIZONS
OVERLOOKED .INGREDIENTIN SURVEy RESEARC~
sit in on a briefing session or to accompany an interviewer will further confirm how capable and thoroughly trained the interviewers are. A closely related subject to selection, training, and control is the type of information kept by the research supplier on its interviewers. Some of the meaningful specifics are the amount and kind of information available, the date of the last entry, and the feedback to the interviewers on how they are performing. Again, a brief examination of the supplier's records can be enlightening. CONTROLS
Validation procedures are an important control on interviewing quality. Most research firms do no more than send out postcards to a sample of respondents to see ff the interview was actually conducted. This crude technique will only detect the interviewer who totally falsifies the interview. Some firms also insist that the local resident supervisor validate a proportion of the interviews by phone. While this system is not foolproof, since the supervisor may wish to cover up any cheating by her interviewers, its existence can be checked by asking to see validation records. The most effective method of validation available to a research supplier is to conduct telephone interviews from the home office. With the advent of low evening rates and Wide Area Telephone Service (WATS), validation from the home office is economically feasible. From the research buyer's point of view, the only unquestionable technique is to validate directly by direct telephone. In this way, the buyer can convince himself that specifications were followed exactly. Moreover, informing the supplier and the interviewers that this method of validation will be used should be a deterrent to poor quality in itself. An alternative to direct validation by the research buyer is to use an impartial third party to conduct the validation. 6
FALL, 1966
•
Other controls include the presence and use of cost data, call records, and listing sheets by the supplier. Through these forms, the supplier can trace the movements of individual interviewers. The research buyer can assure himself that these controls are being exercised through an inspection of the supplier's operating procedures. THE NEED FOR CONCERTED ACTION
Interviewing is an important ingredient in marketing research; unless the research buyer pays particular attention to it, interviewing quality can be eroded in many ways. All that the individual research buyer can do, however, is to specify research of high quality, to be willing to pay for it, and to look for symptons of its presence or absence. Most research buyers have too small a research requirement to exert any significant leverage on the over-all practices in the field. Only through concerted efforts can research buyers be sure of receiving quality interviewing on a continuous basis. They might, for example, give general support to the Advertising Research Foundation's proposed Field Audit and Completion Test (FACT). Under this system, the nth interview of each interviewer would be checked by an ~U~F-eontrolled telephone facility to determine whether the interview was made in the first place, and ff so, whether specific questions had been asked. If trouble were indicated, further checking would be initiated. The hoped-for end result would be a system of uniform validation and interviewer certification with the costs of the program built into every research budget, r Should such a program not materialize, the research buyers could set up a jointly owned, consolidated field service. A consolis Dik W. Twedt, "Is an 'Audit Bureau of Interviews' Needed Now?" Journal of Marketing, XXX (April, 1966), pp, 59-60. "Proposed ARF Audit of Field Interviews Controversial: Dodge," Advertising Age, XXXVII (May 23, 1966), p. 46.
81
CnAnLESS. MAYE~a
82
dated field service could benefit greatly from economies of scale. Such consolidation would make the work offered by the field service su~ciently important to exercise considerable leverage over supervisors and interviewers. Also, with sufficient work, the consolidated field service could afford to hire professional management and retain local resident supervisors on a permanent basis. The consolidated service could maintain a national pool of trained interviewers. These interviewers would be guaranteed a ~minimum amount of work per week and would be paid at a higher rate than interviewers receive currently, Raising rates without adequate controls would achieve little as far as padding is concerned. However, a consolidated service would also be able to establish standards and increase controls. It may even eliminate padding to such an extent that costs would be reduced, though a higher rate were paid to interviewers.
Such consolidation would not have to be anticompetitive. Competition could still be maintained in survey design and data analysis. Moreover, with careful planning, proprietary data couM be kept secret. OVEn THE past years many people have talked about the importance of quality interviewing. Unfo~unately, we are still in the talking stage. Marketing management and research directors have failed to realize the danger of inadequate field work. Until they do, and are prepared to pay for good work, the present interviewer problem will persist. Individual research buyers may be able to measure the presence of quality in interviewing, but they can do very little to cure the general problems in the field. Such cure can only come by concerted action through professional organizations , or through consolidating the research requirements of several research buyers.
i
W E CA~ never eliminate competition. We can only change the rules of the game, and the pay-off. Competition is to be found in the subdued and pious Quaker meeting just as surely as it is on the bloodiest of battlefields. The devices of love may be found in its arsenal side by side with weapons of steel. Our problem is not to avoid the unavoidable-competition-but to choose our weapons. In seeking the means that are most commensurate with human comfort, pleasure and dignity we cannot necessarily trust first impressions or traditional moral standards. We will need the deepest insights of psychology and anthropology to enable us to choose well. --Garrett Hardin NATURE AND MAN'S FATE
r d
.
.
.i
.
BUSINESS HORIZONS