- Email: [email protected]
Effects of a final telephone reminder and questionnaire cover design in mail surveys
SOCIAL
SCIENCE
RESEARCH
17, 353-361 (1988)
Effects of a Final Telephone Reminder and Questionnaire Cover Design in Mail Surveys ANTON University
J. NEDERHOF of Leyden
To repair a major drawback in Dillman’s (1978) procedures for mail surveys, the final follow-up by certified mailing was replaced by a telephone reminder. Two samples of Dutch biotechnological researchers were surveyed. The telephone reminder was as effective as certified mailings in reducing nonresponse (twice by 34%, close to the predicted 33%), while improving the contact rate. The cost effectiveness of the telephone reminder was enhanced by timing the follow-up 10 days after the second follow-up, Based upon cognitive theory, a second experiment was aimed at increasing the response rate by enhancing the cognitive accessibility of the questionnaire. A largely black contrastive front cover yielded a 11% higher response rate than the same cover design that was white-spaced and hardly contrastive. 0 1988 Academic press. IIIC.
In recent years, important progress has been made regarding data collection by means of mail surveys. From a low quality data collection method characterized by low response rates and low data quality, mail surveys have been transformed into an effective method capable of yielding high quality data and competitive response rates (e.g., Dillman, 1978; Heberlein and Baumgartner, 1978; Nederhof, 1983a; Goyder, 1985). Two of the issues which have dominated recent mail survey research concern the international comparability of mail survey methods (Heberlein and Baumgartner, 1981; Goyder, 1985; Goyder and Leiper, 1985; Nederhof, 1985a) and further improvement of Dillmans’ total design method (TDM) for mail surveys (Dillman, 1978; Dillman and Moore, 1983; Nederhof, 1983b, 1985b). With regard to the former issue, Eichner and Habermehl (1981) found different impacts for a number of methodological variables for a sample of West German and Austrian surveys than Heberlein and Baumgartner (1978, 1981) obtained for a sample of North American surveys. Goyder and Leiper (1985) presented evidence that response rates of mail The research was supported by grants from the Ministery of Education and Science and the Ministery of Economic Affairs of the Netherlands. Address all communications and requests for reprints to Dr. Anton J. Nederhof, Leyden Institute for Social Policy Research, University of Leyden, Stationsplein 242, 2312 AR Leyden, The Netherlands. 353 0049-089X/88 $3.00 Copyright 0 19% by Academic Press. Inc. All rights of reproduction in any form reserved.
354
ANTON
J. NEDERHOF
surveys are different in the U.S., Canada, and Great Britain, even when largely comparable data collection techniques are employed. However, using a highly standardized set of implementation methods, the TDM, a number of studies (e.g., Eichner and Habermehl, 1982; Nederhof, 1985a, 1986) obtained similar results in West Germany and the Netherlands as Dillman’s U.S. surveys (Dillman, 1978). This shows that mail survey data collection methods can be exchanged between North America and parts of North Western Europe. In the present study, results of two Dutch surveys are compared to recent U.S. ones, both involving a telephone follow-up instead of the traditional TDM certified follow-up (Dillman, 1978). Thus, previous international comparisons are extended as a new mode of research is involved. The traditional TDM survey is characterized by the use of four mailings: the original questionnaire, followed 1 week later by a postcard reminder. a second copy of the questionnaire after 3 weeks, and finally, after 7 weeks, a third replacement questionnaire sent by certified mail. Past research has clearly shown that certified mail improves response rates considerably above the level obtained with regular mail (e.g., Dillman, 1978; Heberlein and Baumgartner, 1978; Dillman and Moore, 1983). In terms of social exchange theory, the substantial cost of certified mail indicates an investment on the part of the researcher to obtain the cooperation of the respondent, and thus underscores the importance of the study (cf. Dillman, 1978). In addition, the salience of this fourth contact is likely to be strongly enhanced because of the relatively rare incidence of receiving certified mail, as well as by requiring the signature of the respondent or someone else in the household. However, the use of certified mail has some important drawbacks. The requirement of signing for receipt is a coercive element, and the costs for the respondent will be even larger when he or she has to go to the post office in order to get the questionnaire. In addition to the costs of time and money which this involves, anxiety may have been created on the part of the respondent. Furthermore, this procedure may foster resentment, causing respondents to refuse to participate in future studies (cf. Nederhof, in press). The telephone follow-up attempts to eliminate these cost aspects while maintaining the high sense of importance and salience established by certified mail. Because of the high expenses and the disadvantages just noted, the final certified follow-up is frequently deleted or replaced by a regular mailing. As Dillman, Dillman, and Makela (1984) have shown, this substitution leads to substantial decreases in response rate. Therefore, if it could be shown that the telephone followup yields comparable results as the certified mailing, but with less costs both to researchers and respondents, this would be an important improvement, enlarging the repertoire of mail survey methods. First results suggest that the telephone reminder may be as effective
MAIL
355
SURVEYS
in enhancing response rates as is the certified mailing for samples taken from the general population (Dillman and Moore, 1983). As Dillman and Moore (1983) remark, however, their results may not necessarily be generalizable to other populations than the general public. In the present studies, an attempt will be made to generalize their findings to use in a special population. It was expected that the telephone reminder would reduce the nonresponse remaining after the first three mailings by a similar percentage as certified mailings have been found to do, namely 32.7% (Dillman, Dillman, and Makela, 1984). If this hypothesis is confirmed, additional evidence will have been gathered to show that mail survey methods can be used with comparable results both in the U.S. and the Netherlands. In addition, the effectiveness of the telephone reminder in screening ineligibles, its contribution to both the final response rate as well as to the percentage of the original sample which are contacted during data collection (the “contact rate”), was assessed. A second experimental study focused on the front cover of the questionnaire. In order to increase the response rate, the cognitive accessibility of the questionnaire was increased by improving front cover design. SALIENCE
AND COGNITIVE ACCESSIBILITY MAIL QUESTIONNAIRES
OF
Cognitive theories suggest an analysis of the response process in mail surveys in terms of salience and cognitive accessibility (how easily a thing comes to mind) (cf Fiske and Taylor. 1984; Bodenhauser and Wyer, 1987). Mail questionnaires should be made salient in the perception of potential respondents, and the cognitive accessibility of the questionnaire should be enhanced. Salience is important when the questionnaire is in view, while longer-term cognitive accessibility is potent when it is not in view. The front cover of the questionnaire is an important determinant of both salience and cognitive accessibility. The front covers of mail questionnaires have been discussed mainly along the lines of their immediate salience (e.g., Dillman, 1978) or, more specifically, their color (Pressley and Tullar, 1977). When, however, a questionnaire is not immediately completed and returned or is thrown away, it will be put aside for completion at a future moment or it will be decided at a later moment whether and/or when it will be completed. Then, it becomes important that the questionnaire is readily accessible for cognitive retrieval at an opportune moment for the respondent to complete the questionnaire. Although the various reminders serve to increase the salience of the questionnaire momentarily, they may fail for respondents who cannot or do not complete the questionnaire immediately, but await a more suitable moment. This analysis implies that the questionnaire should stand out among
356
ANTON
J. NEDERHOF
many other competing stimuli, such as other correspondence waiting to be answered or other printed matter. Also, it should avoid getting mixed up with mail likely to be thrown away or not attended. Thus, the use of colored pages may remind people of junk mail and may therefore be counterproductive, as various experiments showed (Pressley and Tullar, 1977; Jobber and Sanderson, 1983). The TDM pays attention to the outside of the envelope which contains the questionnaire by attempting to avoid looking like junk mail and to look instead as business mail, thus trying to make sure that the content of the envelope will be viewed at least briefly. Nederhof (1983b), for instance, has shown that personalizing the outside of the envelope by means of hand-written addresses increases response. The front cover of the questionnaire should be designed to optimize the longer-term cognitive accessibility of the questionnaire, which should increase the chance that the questionnaire will be attended to once again after having been put aside, and thus improve the chance that it will be completed and returned. As information is more likely to be retained in long-term memory if it has been processed more extensively, the questionnaire cover should be made more memorable by making it more distinctive (Bodenhauser and Wyer, 1987, p. 25) and more complex (Fiske and Taylor, 1984). METHOD
The telephone follow-up takes the tone of a friendly reminder and not of a reproach for not having answered before. It is explained to the respondent that he or she is being called to see if the questionnaire has arrived and to find out if there are any questions about the study. If the questionnaire is not available, the respondent is told that another copy will be sent to him or her today. A difference with the procedures followed by Dillman and Moore (1983) was that the telephone reminder did not follow closely upon the second follow-up, but was made after an interval of at least 9 days. This change was intended to reduce the number of persons to be called considerably, as the bulk of response to the second follow-up is usually registered within that interval. Another difference concerned the use of business reply envelopes instead of the possibly slightly more effective (2-3%) stamped return envelopes (cf. Dillman and Moore, 1983). For the. front cover experiment, two basically identical questionnaire covers were designed by ir. L. van Moll (see Fig. 1). However, one questionnaire was made to stand out more than the other by filling open spaces with black. This procedure enhanced both the contrast of the figures in the graphic illustration as well as the contrast of the letters forming the word “biotechnologie” (biotechnology) in the upper half of the design. The graphic design occupied nearly all of the front cover in order to increase salience and longer-term accessibility as much as possible.
MAIL
SURVEYS
FIGURE
357
I.
Both questionnaire covers only differed which black was used and in the resulting that the black questionnaire, being more complex, would lead to a higher response survey.
with regard to the extent to contrast. It was hypothesized distinctive and more visually rate in the later phases of the
SAMPLES Respondents were persons actively engaged in biotechnological research and living in the Netherlands. For the first survey, 336 names were drawn from two sources: from the membership list of the Dutch Biotechnological Association, and from a recent inventorization of biotechnological research in the Netherlands (Rip and Nederhof, 1986). Respondents were associated with universities, state research institutes, and industrial companies. The questionnaire was entitled “Biotechnology in the Netherlands: Developments in the Eighties.” It contained 18 pages of questions, mostly concerning attitudinal matters, as well as questions concerning the research projects of the respondents. A first copy of the questionnaire was mailed on October 12, 1984 and was followed 6 days later by a postcard reminder. Nonrespondents were sent a second copy of the questionnaire after 19 days. Finally, remaining nonrespondents were called 4 weeks after the first mailing or a few days later. At least three attempts were made to contact the respondents by phone. Otherwise, directions by Dillman (1978) and Nederhof (in press, pp. 58-59) were followed. Similar procedures were followed in the second study, among a second random sample, independently drawn from the sources mentioned above,
358
ANTON
J. NEDERHOF
TABLE I Response Rates of Two Questionnaire
Front Covers
Type of front cover White Respondents Nonrespondents Total
80 (75.5%) 26 106 x2 = 4.04; P < .05
Black
99 (86.1%) 16 I15
Total 179 42 221”
” Identified ineligibles have been removed.
of Dutch biotechnological researchers. This questionnaire was titled “Developments in Biotechnology in the Netherlands,” and also contained 18 pages of questions. This survey started on February 2, 1987 and was mailed to 320 biotechnologists. Half of this sample received a questionnaire with a graphic illustration largely in white, while the other half received a questionnaire with a largely black graphic on the front cover. Third and fourth mailings followed the schedules of survey one. One detail differed, however. A postcard reminder was sent to 10 respondents who had promised, in response to the fourth wave phone call, to return the questionnaire but failed to do so within 10 days after being contacted. Of the identified individuals, 20.7% were ineligible in the first study, and, due to a large increase of persons interested in biotechnology without doing research, 31.6% in the second study. RESULTS
In general, the telephone reminder appeared to be quite effective. First, the telephone call increased the response rate significantly in both cases: with 10 from 72% (after three mailings) to a final 82% in the first survey, and from 71 to 81% in the second survey (see Table 1). This represents in both cases a reduction of about 34.5% of the nonresponse remaining after three mailings. Although slightly higher than the 32.7% reduction predicted on the basis of the findings of Dillman and Moore (1983), this difference was not significant. In both surveys, the telephone reminder contributed 12% to the final response rate. These figures more than triple those obtained in mail surveys in the Netherlands using first class mail as a final third follow-up (Nederhof, 1985a). Second, 36 and 20% of the total number of detected ineligibles were found by means of the telephone reminder, an important contribution. Finally, another important advantage of the telephone reminder is the high contact rate: 95% (survey 1) and 97.7% (survey 2) of the original
MAIL
SURVEYS
359
sample were contacted, which considerably exceeds the average contact rate usually obtained in mail surveys using certified mailings (Dillman, 1978). A high contact rate is important for the identification of ineligibles. For reasons of comparability with the Dillman and Moore (1983) study, the above response rates have been computed without correcting for the number of ineligibles among nonrespondents. When corrections are made for ineligibles according to the definition of the CASRO (1982), the final response rate is estimated by assuming that of the nonrespondents also 20.7 or 3 1.6% were ineligible. This would increase the final response rate in the first survey from 82 to an estimated 85.4%, and in the second survey from 81 to an estimated 87.3%. Regarding the front cover experiment in the second survey, the black contrastive cover performed superiorly, netting a 10.6% higher response rate than the white cover, a significant difference (P < .05) (See Table 1). The significant increase in response rate was not evident early in the survey. After the first wave, the response rate for the white-covered questionnaires was 21.7%, whereas the response for the black covers was 19.1%, a nonsignificant difference. At the end of the second wave, the difference was already considerable, although nonsignificant: 47.2% (white) vs 57.4% (black) (x2 = 2.31; P > .lO). The third wave increased response rates to 66.0% (white) and 75.6% (black), still a nonsignificant difference (x2 = 2.48; P > .lO). DISCUSSION
It was examined whether a major drawback in an otherwise highly effective and efficient survey method could be repaired. For respondents and researchers, the use of certified mailings has important costs, both financially and in terms of exchange theoretical principles. On the other hand, the certified mailing is the most effective of the four mailings used in the TDM in reducing remaining nonresponse (Dillman, 1978). Replacement of the certified follow-up by a telephone reminder did not lead to a decrease in response rate. The nonresponse which remained after three mailings was reduced by the telephone call by a percentage (34.5%) highly comparable to that obtained in studies using follow-up by certified mail. These findings are in close accordance with the 33% predicted on the basis of Dillman and Moore (1983). This also suggests that the basic results obtained by means of survey methods developed in one part of the Western world can be replicated across modes of research (mail, telephone), across different types of populations (general population, professional groups), across time, and across research topics. However, a cross-continental experiment employing identical methods is needed to test these issues to a fuller extent. From a cost perspective, it is important to note that timing the telephone
360
ANTON
J. NEDERHOF
reminder about a week and a half after the second follow-up instead of almost immediately following it did not apparently lead to a decrease in response rate, but reduced the number of callbacks considerably, and thus also expenses. The second experiment shows the importance of the front cover of the questionnaire in increasing response rate in mail surveys, as evidenced by the 10.6% difference in final response rate. It also indicates the usefulness of theories from the cognitive perspective. That the white covers performed at least as well as the black covers early in the survey, but did significantly worse only after the fourth wave, shows that the initial salience of both covers is comparable. The predicted greater longer-term cognitive accessibility of the black questionnaire did indeed pay off later in the survey, with the difference in response rate reaching significance in the final wave. The difference in response rate between the two covers increased especially in response to mailings 2 (postcard follow-up) and 4 (telephone reminder), which do not include a copy of the questionnaire, and which require therefore retrieval by respondents. In contrast, in the mailings including a copy of the questionnaire (wave 1 and 3), and thus requiring less retrieval, the difference between both covers did not increase. A replication with larger samples is needed to test these interwave differences more thoroughly. These results also mean that now an additional element has been identified in the TDM procedure that is effective in the final phase of a survey, whereas all previous other attempts have failed. It should be noted that the increase was observed in the context of a final telephone reminder, where cognitive accessibility and salience of the questionnaire are somewhat more important (as the respondent has to search for the questionnaire) than in a fourth wave by certified mail where a (third) copy of the questionnaire is provided. However, we expect that a constrastive full-page graphic cover design would also enhance response in certified mailings due to the cover’s greater longerterm cognitive accessibility. REFERENCES Bodenhauser, G. V., and Wyer, R. S. (1987). Social cognition and social reality: Information acquisition and use in the laboratory and the real world, in Social Information Processing and Survey Methodology (H. J. Hippler, N. Schwarz, and S. Sudman, Eds.), Springer, New York. CASRO task force on completion rates, “On the definition of response 1982 rates,” CASRO, Port Jefferson, New York. Dillman, D. A. (1978). Mail and Telephone Surveys: The Total Design Method, Wiley, New York. Dillman, D. A., Dillman. J. J., and Makela, C. J. (1984). “The importance of adhering to details of the total design method (TDM) for mail surveys,” in Making Effective Use of Mailed Questionnaires (D. C. Lockhart, Ed.), Jossey-Bass, San Francisco. Dillman, D. A., and Moore, D. E. (1983). Improving Response Rates to Mail Surveys:
MAIL
SURVEYS
361
Results from Five Surveys. Paper presented at the 1983 Annual meeting of the American Association for Public Opinion Research at Hershey, PA. Eichner, K., and Habermehl, W. (1981). “Predicting response rates to mailed questionnaires.” American Sociological Review 46, 361-363. Eichner, K., and Habermehl, W. (1982). “Ergebnis einer empirischen Untersuchung zur Reprasentivitat postalischer Befragungen,” Kolner Zeitschrift fur Soziologie und Sozialpsychologie 34, 117-126. Fiske, S. A., and Taylor, S. E. (1984). Social Cognition, Addison-Wesley. Reading, MA. Goyder, J. (1985). “Face-to-face interviews and mailed questionnaires: The net difference in response rate,” Public Opinion Quarterly 49, 234-252. Goyder, J., and Leiper, J. M. (1985). “The decline in survey response: A social values interpretation,” Sociology 19, 55-71. Heberlein, T. A., and Baumgartner, R. (1978). “Factors affecting response rates to mailed questionnaires: A quantitative analysis of the published literature,” American Sociological Review 43, 447-462. Heberlein, T. A., and Baumgartner, R. (1981). “The effectiveness of the Heberlein-Baumgartner models for predicting response rates to mailed questionnaires: European and U.S. examples,” American Sociological Review 46, 363-367. Jobber, D., and Sanderson, S. M. (1983). “The effects of a prior letter and coloured questionnaires on mail survey response rates,” Journal of the Market Research Society 25. Nederhof, A. J. (1983a). “The effects of material incentives in mail surveys: Two studies,” Public Opinion Quarterly 47, 103-l 11. Nederhof, A. J. (1983b). “The effects of repetition and consistency of personalization treatments on response rates in mail surveys,” Social Science Research 12, l-9. Nederhof, A. J. (1985a). “A comparison of European and North American response patterns in mail surveys,” Journal of the Market Research Society 27, 55-63. Nederhof, A. J. (1985b). “A survey on suicide: Using a mail survey to study a highly threatening topic,” Quality and Quantity 19, 293-302. Nederhof, A. J. (1986). “Effects of research experience of subjects.” Quality and Quantity 20, 275-282. Nederhof, A. J. (in press). Artifact in Social Science Research: Coping with Sources of Bias in Data Collection, Irvington, New York. Pressley, M. M., and Tullar, W. (1977). “A factor interactive investigation of mail survey response rates from a commercial population,” Journal of Marketing Research 41, 108-112. Rip, A., and Nederhof, A. J. (1986). “Between dirigism and laissez-faire: Implementation of the science policy priority for biotechnology in the Netherlands,” Research Policy 15, 253-268.
SCIENCE
RESEARCH
17, 353-361 (1988)
Effects of a Final Telephone Reminder and Questionnaire Cover Design in Mail Surveys ANTON University
J. NEDERHOF of Leyden
To repair a major drawback in Dillman’s (1978) procedures for mail surveys, the final follow-up by certified mailing was replaced by a telephone reminder. Two samples of Dutch biotechnological researchers were surveyed. The telephone reminder was as effective as certified mailings in reducing nonresponse (twice by 34%, close to the predicted 33%), while improving the contact rate. The cost effectiveness of the telephone reminder was enhanced by timing the follow-up 10 days after the second follow-up, Based upon cognitive theory, a second experiment was aimed at increasing the response rate by enhancing the cognitive accessibility of the questionnaire. A largely black contrastive front cover yielded a 11% higher response rate than the same cover design that was white-spaced and hardly contrastive. 0 1988 Academic press. IIIC.
In recent years, important progress has been made regarding data collection by means of mail surveys. From a low quality data collection method characterized by low response rates and low data quality, mail surveys have been transformed into an effective method capable of yielding high quality data and competitive response rates (e.g., Dillman, 1978; Heberlein and Baumgartner, 1978; Nederhof, 1983a; Goyder, 1985). Two of the issues which have dominated recent mail survey research concern the international comparability of mail survey methods (Heberlein and Baumgartner, 1981; Goyder, 1985; Goyder and Leiper, 1985; Nederhof, 1985a) and further improvement of Dillmans’ total design method (TDM) for mail surveys (Dillman, 1978; Dillman and Moore, 1983; Nederhof, 1983b, 1985b). With regard to the former issue, Eichner and Habermehl (1981) found different impacts for a number of methodological variables for a sample of West German and Austrian surveys than Heberlein and Baumgartner (1978, 1981) obtained for a sample of North American surveys. Goyder and Leiper (1985) presented evidence that response rates of mail The research was supported by grants from the Ministery of Education and Science and the Ministery of Economic Affairs of the Netherlands. Address all communications and requests for reprints to Dr. Anton J. Nederhof, Leyden Institute for Social Policy Research, University of Leyden, Stationsplein 242, 2312 AR Leyden, The Netherlands. 353 0049-089X/88 $3.00 Copyright 0 19% by Academic Press. Inc. All rights of reproduction in any form reserved.
354
ANTON
J. NEDERHOF
surveys are different in the U.S., Canada, and Great Britain, even when largely comparable data collection techniques are employed. However, using a highly standardized set of implementation methods, the TDM, a number of studies (e.g., Eichner and Habermehl, 1982; Nederhof, 1985a, 1986) obtained similar results in West Germany and the Netherlands as Dillman’s U.S. surveys (Dillman, 1978). This shows that mail survey data collection methods can be exchanged between North America and parts of North Western Europe. In the present study, results of two Dutch surveys are compared to recent U.S. ones, both involving a telephone follow-up instead of the traditional TDM certified follow-up (Dillman, 1978). Thus, previous international comparisons are extended as a new mode of research is involved. The traditional TDM survey is characterized by the use of four mailings: the original questionnaire, followed 1 week later by a postcard reminder. a second copy of the questionnaire after 3 weeks, and finally, after 7 weeks, a third replacement questionnaire sent by certified mail. Past research has clearly shown that certified mail improves response rates considerably above the level obtained with regular mail (e.g., Dillman, 1978; Heberlein and Baumgartner, 1978; Dillman and Moore, 1983). In terms of social exchange theory, the substantial cost of certified mail indicates an investment on the part of the researcher to obtain the cooperation of the respondent, and thus underscores the importance of the study (cf. Dillman, 1978). In addition, the salience of this fourth contact is likely to be strongly enhanced because of the relatively rare incidence of receiving certified mail, as well as by requiring the signature of the respondent or someone else in the household. However, the use of certified mail has some important drawbacks. The requirement of signing for receipt is a coercive element, and the costs for the respondent will be even larger when he or she has to go to the post office in order to get the questionnaire. In addition to the costs of time and money which this involves, anxiety may have been created on the part of the respondent. Furthermore, this procedure may foster resentment, causing respondents to refuse to participate in future studies (cf. Nederhof, in press). The telephone follow-up attempts to eliminate these cost aspects while maintaining the high sense of importance and salience established by certified mail. Because of the high expenses and the disadvantages just noted, the final certified follow-up is frequently deleted or replaced by a regular mailing. As Dillman, Dillman, and Makela (1984) have shown, this substitution leads to substantial decreases in response rate. Therefore, if it could be shown that the telephone followup yields comparable results as the certified mailing, but with less costs both to researchers and respondents, this would be an important improvement, enlarging the repertoire of mail survey methods. First results suggest that the telephone reminder may be as effective
355
SURVEYS
in enhancing response rates as is the certified mailing for samples taken from the general population (Dillman and Moore, 1983). As Dillman and Moore (1983) remark, however, their results may not necessarily be generalizable to other populations than the general public. In the present studies, an attempt will be made to generalize their findings to use in a special population. It was expected that the telephone reminder would reduce the nonresponse remaining after the first three mailings by a similar percentage as certified mailings have been found to do, namely 32.7% (Dillman, Dillman, and Makela, 1984). If this hypothesis is confirmed, additional evidence will have been gathered to show that mail survey methods can be used with comparable results both in the U.S. and the Netherlands. In addition, the effectiveness of the telephone reminder in screening ineligibles, its contribution to both the final response rate as well as to the percentage of the original sample which are contacted during data collection (the “contact rate”), was assessed. A second experimental study focused on the front cover of the questionnaire. In order to increase the response rate, the cognitive accessibility of the questionnaire was increased by improving front cover design. SALIENCE
AND COGNITIVE ACCESSIBILITY MAIL QUESTIONNAIRES
OF
Cognitive theories suggest an analysis of the response process in mail surveys in terms of salience and cognitive accessibility (how easily a thing comes to mind) (cf Fiske and Taylor. 1984; Bodenhauser and Wyer, 1987). Mail questionnaires should be made salient in the perception of potential respondents, and the cognitive accessibility of the questionnaire should be enhanced. Salience is important when the questionnaire is in view, while longer-term cognitive accessibility is potent when it is not in view. The front cover of the questionnaire is an important determinant of both salience and cognitive accessibility. The front covers of mail questionnaires have been discussed mainly along the lines of their immediate salience (e.g., Dillman, 1978) or, more specifically, their color (Pressley and Tullar, 1977). When, however, a questionnaire is not immediately completed and returned or is thrown away, it will be put aside for completion at a future moment or it will be decided at a later moment whether and/or when it will be completed. Then, it becomes important that the questionnaire is readily accessible for cognitive retrieval at an opportune moment for the respondent to complete the questionnaire. Although the various reminders serve to increase the salience of the questionnaire momentarily, they may fail for respondents who cannot or do not complete the questionnaire immediately, but await a more suitable moment. This analysis implies that the questionnaire should stand out among
356
ANTON
J. NEDERHOF
many other competing stimuli, such as other correspondence waiting to be answered or other printed matter. Also, it should avoid getting mixed up with mail likely to be thrown away or not attended. Thus, the use of colored pages may remind people of junk mail and may therefore be counterproductive, as various experiments showed (Pressley and Tullar, 1977; Jobber and Sanderson, 1983). The TDM pays attention to the outside of the envelope which contains the questionnaire by attempting to avoid looking like junk mail and to look instead as business mail, thus trying to make sure that the content of the envelope will be viewed at least briefly. Nederhof (1983b), for instance, has shown that personalizing the outside of the envelope by means of hand-written addresses increases response. The front cover of the questionnaire should be designed to optimize the longer-term cognitive accessibility of the questionnaire, which should increase the chance that the questionnaire will be attended to once again after having been put aside, and thus improve the chance that it will be completed and returned. As information is more likely to be retained in long-term memory if it has been processed more extensively, the questionnaire cover should be made more memorable by making it more distinctive (Bodenhauser and Wyer, 1987, p. 25) and more complex (Fiske and Taylor, 1984). METHOD
The telephone follow-up takes the tone of a friendly reminder and not of a reproach for not having answered before. It is explained to the respondent that he or she is being called to see if the questionnaire has arrived and to find out if there are any questions about the study. If the questionnaire is not available, the respondent is told that another copy will be sent to him or her today. A difference with the procedures followed by Dillman and Moore (1983) was that the telephone reminder did not follow closely upon the second follow-up, but was made after an interval of at least 9 days. This change was intended to reduce the number of persons to be called considerably, as the bulk of response to the second follow-up is usually registered within that interval. Another difference concerned the use of business reply envelopes instead of the possibly slightly more effective (2-3%) stamped return envelopes (cf. Dillman and Moore, 1983). For the. front cover experiment, two basically identical questionnaire covers were designed by ir. L. van Moll (see Fig. 1). However, one questionnaire was made to stand out more than the other by filling open spaces with black. This procedure enhanced both the contrast of the figures in the graphic illustration as well as the contrast of the letters forming the word “biotechnologie” (biotechnology) in the upper half of the design. The graphic design occupied nearly all of the front cover in order to increase salience and longer-term accessibility as much as possible.
SURVEYS
FIGURE
357
I.
Both questionnaire covers only differed which black was used and in the resulting that the black questionnaire, being more complex, would lead to a higher response survey.
with regard to the extent to contrast. It was hypothesized distinctive and more visually rate in the later phases of the
SAMPLES Respondents were persons actively engaged in biotechnological research and living in the Netherlands. For the first survey, 336 names were drawn from two sources: from the membership list of the Dutch Biotechnological Association, and from a recent inventorization of biotechnological research in the Netherlands (Rip and Nederhof, 1986). Respondents were associated with universities, state research institutes, and industrial companies. The questionnaire was entitled “Biotechnology in the Netherlands: Developments in the Eighties.” It contained 18 pages of questions, mostly concerning attitudinal matters, as well as questions concerning the research projects of the respondents. A first copy of the questionnaire was mailed on October 12, 1984 and was followed 6 days later by a postcard reminder. Nonrespondents were sent a second copy of the questionnaire after 19 days. Finally, remaining nonrespondents were called 4 weeks after the first mailing or a few days later. At least three attempts were made to contact the respondents by phone. Otherwise, directions by Dillman (1978) and Nederhof (in press, pp. 58-59) were followed. Similar procedures were followed in the second study, among a second random sample, independently drawn from the sources mentioned above,
358
ANTON
J. NEDERHOF
TABLE I Response Rates of Two Questionnaire
Front Covers
Type of front cover White Respondents Nonrespondents Total
80 (75.5%) 26 106 x2 = 4.04; P < .05
Black
99 (86.1%) 16 I15
Total 179 42 221”
” Identified ineligibles have been removed.
of Dutch biotechnological researchers. This questionnaire was titled “Developments in Biotechnology in the Netherlands,” and also contained 18 pages of questions. This survey started on February 2, 1987 and was mailed to 320 biotechnologists. Half of this sample received a questionnaire with a graphic illustration largely in white, while the other half received a questionnaire with a largely black graphic on the front cover. Third and fourth mailings followed the schedules of survey one. One detail differed, however. A postcard reminder was sent to 10 respondents who had promised, in response to the fourth wave phone call, to return the questionnaire but failed to do so within 10 days after being contacted. Of the identified individuals, 20.7% were ineligible in the first study, and, due to a large increase of persons interested in biotechnology without doing research, 31.6% in the second study. RESULTS
In general, the telephone reminder appeared to be quite effective. First, the telephone call increased the response rate significantly in both cases: with 10 from 72% (after three mailings) to a final 82% in the first survey, and from 71 to 81% in the second survey (see Table 1). This represents in both cases a reduction of about 34.5% of the nonresponse remaining after three mailings. Although slightly higher than the 32.7% reduction predicted on the basis of the findings of Dillman and Moore (1983), this difference was not significant. In both surveys, the telephone reminder contributed 12% to the final response rate. These figures more than triple those obtained in mail surveys in the Netherlands using first class mail as a final third follow-up (Nederhof, 1985a). Second, 36 and 20% of the total number of detected ineligibles were found by means of the telephone reminder, an important contribution. Finally, another important advantage of the telephone reminder is the high contact rate: 95% (survey 1) and 97.7% (survey 2) of the original
SURVEYS
359
sample were contacted, which considerably exceeds the average contact rate usually obtained in mail surveys using certified mailings (Dillman, 1978). A high contact rate is important for the identification of ineligibles. For reasons of comparability with the Dillman and Moore (1983) study, the above response rates have been computed without correcting for the number of ineligibles among nonrespondents. When corrections are made for ineligibles according to the definition of the CASRO (1982), the final response rate is estimated by assuming that of the nonrespondents also 20.7 or 3 1.6% were ineligible. This would increase the final response rate in the first survey from 82 to an estimated 85.4%, and in the second survey from 81 to an estimated 87.3%. Regarding the front cover experiment in the second survey, the black contrastive cover performed superiorly, netting a 10.6% higher response rate than the white cover, a significant difference (P < .05) (See Table 1). The significant increase in response rate was not evident early in the survey. After the first wave, the response rate for the white-covered questionnaires was 21.7%, whereas the response for the black covers was 19.1%, a nonsignificant difference. At the end of the second wave, the difference was already considerable, although nonsignificant: 47.2% (white) vs 57.4% (black) (x2 = 2.31; P > .lO). The third wave increased response rates to 66.0% (white) and 75.6% (black), still a nonsignificant difference (x2 = 2.48; P > .lO). DISCUSSION
It was examined whether a major drawback in an otherwise highly effective and efficient survey method could be repaired. For respondents and researchers, the use of certified mailings has important costs, both financially and in terms of exchange theoretical principles. On the other hand, the certified mailing is the most effective of the four mailings used in the TDM in reducing remaining nonresponse (Dillman, 1978). Replacement of the certified follow-up by a telephone reminder did not lead to a decrease in response rate. The nonresponse which remained after three mailings was reduced by the telephone call by a percentage (34.5%) highly comparable to that obtained in studies using follow-up by certified mail. These findings are in close accordance with the 33% predicted on the basis of Dillman and Moore (1983). This also suggests that the basic results obtained by means of survey methods developed in one part of the Western world can be replicated across modes of research (mail, telephone), across different types of populations (general population, professional groups), across time, and across research topics. However, a cross-continental experiment employing identical methods is needed to test these issues to a fuller extent. From a cost perspective, it is important to note that timing the telephone
360
ANTON
J. NEDERHOF
reminder about a week and a half after the second follow-up instead of almost immediately following it did not apparently lead to a decrease in response rate, but reduced the number of callbacks considerably, and thus also expenses. The second experiment shows the importance of the front cover of the questionnaire in increasing response rate in mail surveys, as evidenced by the 10.6% difference in final response rate. It also indicates the usefulness of theories from the cognitive perspective. That the white covers performed at least as well as the black covers early in the survey, but did significantly worse only after the fourth wave, shows that the initial salience of both covers is comparable. The predicted greater longer-term cognitive accessibility of the black questionnaire did indeed pay off later in the survey, with the difference in response rate reaching significance in the final wave. The difference in response rate between the two covers increased especially in response to mailings 2 (postcard follow-up) and 4 (telephone reminder), which do not include a copy of the questionnaire, and which require therefore retrieval by respondents. In contrast, in the mailings including a copy of the questionnaire (wave 1 and 3), and thus requiring less retrieval, the difference between both covers did not increase. A replication with larger samples is needed to test these interwave differences more thoroughly. These results also mean that now an additional element has been identified in the TDM procedure that is effective in the final phase of a survey, whereas all previous other attempts have failed. It should be noted that the increase was observed in the context of a final telephone reminder, where cognitive accessibility and salience of the questionnaire are somewhat more important (as the respondent has to search for the questionnaire) than in a fourth wave by certified mail where a (third) copy of the questionnaire is provided. However, we expect that a constrastive full-page graphic cover design would also enhance response in certified mailings due to the cover’s greater longerterm cognitive accessibility. REFERENCES Bodenhauser, G. V., and Wyer, R. S. (1987). Social cognition and social reality: Information acquisition and use in the laboratory and the real world, in Social Information Processing and Survey Methodology (H. J. Hippler, N. Schwarz, and S. Sudman, Eds.), Springer, New York. CASRO task force on completion rates, “On the definition of response 1982 rates,” CASRO, Port Jefferson, New York. Dillman, D. A. (1978). Mail and Telephone Surveys: The Total Design Method, Wiley, New York. Dillman, D. A., Dillman. J. J., and Makela, C. J. (1984). “The importance of adhering to details of the total design method (TDM) for mail surveys,” in Making Effective Use of Mailed Questionnaires (D. C. Lockhart, Ed.), Jossey-Bass, San Francisco. Dillman, D. A., and Moore, D. E. (1983). Improving Response Rates to Mail Surveys:
SURVEYS
361
Results from Five Surveys. Paper presented at the 1983 Annual meeting of the American Association for Public Opinion Research at Hershey, PA. Eichner, K., and Habermehl, W. (1981). “Predicting response rates to mailed questionnaires.” American Sociological Review 46, 361-363. Eichner, K., and Habermehl, W. (1982). “Ergebnis einer empirischen Untersuchung zur Reprasentivitat postalischer Befragungen,” Kolner Zeitschrift fur Soziologie und Sozialpsychologie 34, 117-126. Fiske, S. A., and Taylor, S. E. (1984). Social Cognition, Addison-Wesley. Reading, MA. Goyder, J. (1985). “Face-to-face interviews and mailed questionnaires: The net difference in response rate,” Public Opinion Quarterly 49, 234-252. Goyder, J., and Leiper, J. M. (1985). “The decline in survey response: A social values interpretation,” Sociology 19, 55-71. Heberlein, T. A., and Baumgartner, R. (1978). “Factors affecting response rates to mailed questionnaires: A quantitative analysis of the published literature,” American Sociological Review 43, 447-462. Heberlein, T. A., and Baumgartner, R. (1981). “The effectiveness of the Heberlein-Baumgartner models for predicting response rates to mailed questionnaires: European and U.S. examples,” American Sociological Review 46, 363-367. Jobber, D., and Sanderson, S. M. (1983). “The effects of a prior letter and coloured questionnaires on mail survey response rates,” Journal of the Market Research Society 25. Nederhof, A. J. (1983a). “The effects of material incentives in mail surveys: Two studies,” Public Opinion Quarterly 47, 103-l 11. Nederhof, A. J. (1983b). “The effects of repetition and consistency of personalization treatments on response rates in mail surveys,” Social Science Research 12, l-9. Nederhof, A. J. (1985a). “A comparison of European and North American response patterns in mail surveys,” Journal of the Market Research Society 27, 55-63. Nederhof, A. J. (1985b). “A survey on suicide: Using a mail survey to study a highly threatening topic,” Quality and Quantity 19, 293-302. Nederhof, A. J. (1986). “Effects of research experience of subjects.” Quality and Quantity 20, 275-282. Nederhof, A. J. (in press). Artifact in Social Science Research: Coping with Sources of Bias in Data Collection, Irvington, New York. Pressley, M. M., and Tullar, W. (1977). “A factor interactive investigation of mail survey response rates from a commercial population,” Journal of Marketing Research 41, 108-112. Rip, A., and Nederhof, A. J. (1986). “Between dirigism and laissez-faire: Implementation of the science policy priority for biotechnology in the Netherlands,” Research Policy 15, 253-268.