- Email: [email protected]
Increasing survey response rates
Increasing Survey Response Rates Practical Instructions from the T0tal-Design Method by M. Chris Paxson
ospitality researchers, consultants, community-development groups, and others conduct surveys for many reasons. They survey guests to learn about their needs and predict trends. They learn about the availability of labor, supports for and barriers to doing business in the community, and why employees resign. Survey research makes important contributions to the industry and has informed managers, for example, about the development of convention-services management as
M. Chris Paxson, Ph.D., an assistant professor of hotel and restaurant administration at Washington State University, thanks Ken Clark for developing the graphicsfor this paper. © 1995,CornellUniversity
88
HOTELANDRESTAURANTADMINISTRATIONQUARTERLY
a career, corporate travel needs, and guests' views about tipping) Since 1990 more than a fourth o f the articles in the Cornell Quarterly have been based on the results o f organizational surveys. Surveying businesses, as opposed to the general public, however, has several problems. First, it may be difficult to get the names o f current owners or managers so that mailings can be personalized. Second, mailing lists o f companies from government sources or trade associations sometimes contain the name and address o f the accounting or auditing firm that handles a company's affairs and not the company's own street address. Third, the high failure rate for new businesses means that there are likely to be more obsolete addresses in a sample o f businesses than in a sample o f the general population o f comparable size. 2 Particular features o f the hospitality industry may present other problems for survey projects. Because most hospitality companies are small and have limited resources and time, respondents may be less likely to respond. U n d e r those conditions, survey salience and appeal are i m p o r t a n t . T i m i n g is also a concern because ofseasonality. N o t only must the survey researcher k n o w an address; he or she must also understand each business and its cycles. ~R.J. Montgomery and D.G. Rutherford, "A Profile of Convention-Services Professionals," Cornell Hotel and Restaurant Administration Quarterly, Vol. 35, No. 6 (December 1994), pp. 47-57; P,.A. Bell, "Corporate Travel-Management Trends--Implications for Hotel-Marketing Strategies," Cornell Hotel and Restaurant Administration Quarterly, Vol. 34, No. 2 (April 1993), pp. 30-39; and N.E. Brown and S.A. Rolle, "Tips versus Service Charges:The Iowa Scene," Cornell Hotel and Restaurant Administration Quarterly, Vol. 32, No. 1 (May 199l), pp. 74-81. 2M.C. Paxson, D.A. Dilhnan, and J. Tarnai, "Improving Response to Business Mail Surveys," in Survey Methods f~r Businesses, Farms, and Instimtians, ed. B.G. Cox and D.A. Binder (New York: Wiley--lnterscience, 1995), pp. 305-306.
T h e r e is also the individual's personal decision regarding whether to cooperate with the survey.We are asking respondents to do something extra, something that is probably not in their job description. Also, high turnover rates mean that respondents' personal identities may be less closely tied to their jobs, so that hospitality employees may be less likely to take the time to respond. For those and other reasons, organizational surveys typically experience low response rates. Because research findings may be considered part o f a firm's competitive advantage and therefore not be published, it is difficult to reliably estimate the extent o f nonresponse. Unofficial estimates put response rates at 10 to 20 percent, and some actually measure l to 10 percent. 3 In a recent issue o f a hospitality journal, response rates o f 3 l percent or less were c o m m o n . O n e researcher told me about an 18-percent response rate for a mail survey o f c o m m u n i t y businesses, explaining that such rates are typical. 4 Until recently, mail surveys o f the general population have also reported low response rates, even w h e n questionnaires were brief? Low response rates were accepted as inevitable. Also, it is c o m m o n l y believed to be too costly to increase response rates to surveys, even though such thinking is erroneous. It is therefore not surprising that researchers surveying hospitality establishments also considered low response rates inevitable and tolerable. ~See, for instance, Starch's comments reported in J.M. Converse, Survey Research in the United States (Berkeley; CA: University of" California Press, 1987), p. 98. 4This report was part o f a 1989 survey conducted by a market-research firm in Spokane, Washington. 5D.A. Dillman, Mail and "11,lephoneSurveys: The Total Desgn Method (New York: John Wiley and Sons, 1978), p. vii.
Low response rates are a concern. First, they introduce a major source o f error through nonresponse bias. N o matter how great the n u m b e r o f completed questionnaires, if only 20 percent o f the recipients fill out and return a questionnaire or complete a telephone interview, those opinions probably do not adequately represent the population. In other words, opinions o f the 80 percent w h o do not respond may differ significantly from those o f the 20 percent who do respond. This example illustrates nonresponse bias resulting from a low response rate: Two recent surveys were conducted in the same community at about the same time. Both asked respondents to estimate the percentage o f their customers located in one geographic region. The first survey, with a response rate o f 18 percent, concluded that 89 percent o f their customers were located in the region. T h e second survey, with a response rate o f 59 percent, concluded that only 73 percent o f customers were regional residents. Given the higher response rate for the second survey, its conclusions would be more reliable than those o£ the first, and the difference could affect a firm's marketing strategy. T h e second survey provides an important clue that in the first survey a disproportionate number o f managers answered whose perspective on the market was a local one. A second reason for concern about low response rates is the risk associated with making business decisions based on a small sample o f responses. What may meet the needs o f the 1 percent who c o n > pleted a guest c o m m e n t card may not reflect the views o f the 99 percent w h o did not. To maximize scarce resources and minimize risk, information should be as representative as possible.
Auoust 1995 •
Exhibit I _
Definition
To m a x i m i z e i m p a c t . . .
Saliency
Importance of the survey topic to the respondent
• focus on the research concept • design a good questionnaire, cover letter, and survey process
Sponsorship
Organization that develops or implements the survey
• have a legitimate, prestigious organization or agency sponsor the survey (making it clear that the sponsor's motive is not to sell a product or service) • use the sponsor's normal business stationery
Follow-up
Series of planned additional contacts by mail or telephone
• make courteous contacts to thank respondents and remind nonrespondents to participate • include a postcard, another mailing, and a telephone contact
Incentives
Tangible token rewards
• send cash or a small gift, preferably before the questionnaire is completed, or offer to send a copy of the results
Sources: Baumgartner & Heberlein; Goyder; and Yu and Cooper (see footnote 6 on this page).
Increasing ResponseRates Recent reviews summarize many research studies on responseincreasing techniques for mail surveys of the general population. 6 The reviews isolate four factors-saliency, sponsorship, follow-up, and incentives--and a possible fifth fiactor--p erson alization--found ~'See: R.M. Baumgartner and T.A. Heberlein, "Recent P, esearch on Mailed Questionnaire Response Rates," in Maleing E(fi,ctive Use of Mailed Questionnaires', ed. D.C. Lockhart (San Francisco: Jossey-Bass, 1984), p. 65; J.C. Goyder, "Further Evidence on Factors Affecting lZ,esponse Rates to Mailed Questionnaires," American Sociological Review, Vol. 47, No. 4 (1982), pp. 550-553; and J.Yu and H. Cooper, "A Quantitative Review of Research Design Effects on Response Rates to Questionnaires," Journal q[ Marketing Research, Vol. 20, No. l (1983), pp. 36-44.
68
most effective in increasing response rates in public opinion surveys (see Exhibit 1). Saliency. H o w important is the survey to the respondent? H o w well does it stand out from the average executive's daily blizzard of mail? The more salient the survey topic is to the respondent, the greater the likelihood of a completed questionnaire. To some extent, the saliency of the topic can be enhanced by the design and appearance of the questionnaire and its cover letter. Sponsorship. Surveys sponsored by universities, business schools, and hospitality programs may obtain better response rates than surveys sponsored by others. Respondents may return questionnaires to the
HOTELANDRESTAURANTADMINISTRATIONQUARTERLY
university researcher more readily because of psychological indebtedness; for example, they might want to return past favors for their education or for research that has aided them. Or the prestige and legitimacy associated with the sponsor may improve the response. Moreover, because of the sponsor, a supervisor may approve of a study and communicate that approval to the employees; that would increase the probability of its being returned. 7 Follow-up. Follow-up contacts can also increase response rates, and the type of follow-up affects the response rate. A follow-up reminder by telephone is more effective than a mailed follow-up, for instance. ~A possible explanation is that a followup by telephone underscores the importance of the study and makes the respondents feel rewarded by the attention they receive." Followups may also enhance the personalization aspect of the survey, which can increase response rates. Incentives. Providing monetary incentives is one of the most effective means of increasing response rates in surveys of the general public.>Monetary incentives may not be appropriate with business surveys, however, because the person who opens the envelope and pockets the cash may not be the person you'd like to complete the survey. Furthermore, the dean or director may not like the idea of faculty members stuffing surveys with 7Dillman, p. i6. ST.A. Heberlein and R.M. Baumgarmer, "Factors Affecting Response Rates on Mailed Questionnaires: A Quantitative Analysis of the Published Literature," American Sociological Revieu; Vol. 43, No. 4 (1978), pp. 447-462. 'q).A. Dillman and I).E. Moore, "hnproving Response to Mail Surveys: Results from Five Experiments" (working paper, Social and Ecunomic Sciences Research Center, Washington State University, 1983), p. 18. >R.J. Fox, M.R. Crask, and J. Kim, "Mail Survey Response Rate: A Meta-analysis of Selected Techniques for Inducing Response," Public Opiniotl Quarterly, Vol. 52, No. 4 (1988), pp. 467-491.
scarce budget dollars. It may be necessary to rely on other responseenhancing techniques, like datarequest cards. P e r s o n a l i z a t i o n . Some researchers report increased response rates with personalization, tr while others report decreased response rates. ~2 It appears that personalization may interact with other characteristics of the survey, but most experienced survey researchers do not believe it hurts. However, it may be difficult to obtain the names of current owners and managers to use in personalizing cover letters. For example, the membership directories of the American Hotel and Motel Association and the National Restaurant Association provide the names o f designated executives of member companies but not those of individual hotel or restaurant managers.
The Total-DesignMethod in his book Mail and Telephone Surveys: The Total Des~¢n Method, D o n A. Dillman proposes a theoretical and practical framework for mail questionnaires that integrates several of the above procedures for increasing response rates and adds a few of his own (see Exhibit 2, on the next page). ~ Dillman views the questionnaire as a social exchange between the researcher and the respondent.
According to social-exchange theory, the recipient will return the questionnaire if the costs of doing so are less than the perceived benefits.~4 Time and effort are perhaps the major costs. A recipient may reject a poorly designed, lengthy, or confusing questionnaire simply because it looks like it will take too long to complete or it brings into question the integrity of the research. For that reason, questionnaires and mailing processes must be designed carefully. Because researchers seldom can offer truly valuable rewards and questionnaires are often lengthy, the researchers must be concerned with all the controllable details of the survey design. Dillman's method, which he calls the total-design method, is based on techniques used by researchers at the Social and Economic Sciences Research Center in 26 business mail surveys between November 1989 and November 1992.The researchers surveyed business samples ranging in size from 74 to 4,300 (average: 1,604) and achieved average response rates of 52.6 percent.You too can achieve such results if you follow this method as I describe below. Cover letter. Address the cover letter individually, if possible, using the name of the business owner or
t JSee, for instance: E.H. Carpenter, "Personalizing Mail Surveys: A Replication and Reasscssment,"
Public Opinion Quarterly, Vo]. 38, No. 4 (1974), pp. 614-620; and I).A. l)ilhnan and J.H. Frey, "Contribution of" Personalizatiori to Mail Questionnaire P,.esponse as an Elemem of"a Previously Tested Method," Journal qfApplied Psyclwlt{qB Vol. 59, No. 3 (1974), pp. 297-301. *~-A.I<.Andreasen, "Personalizing Questiom~aire Correspondence," Public Opinio~t Quarterly, Vol. 34, No. 1 (1970), pp. 273-277; and M o. Houston and R.W. Jeff'erson, "The Negative Elf`cots of" Persunalization on Response Patterns in Mail Surveys,"Joumal (!fMarketil{~, Researck, Vol. 12, No. I (1975), pp. I14-117. ~l)un A. l)illman, Ph.D., is the director of the Social and Economic Sciences Research Center and a pro(essor in the departments of sociology and rural sociology at Washington State University. The center conducts about 50 survey-related projects each year using mail, telephone, and Eace-to-~ace survey methods, l)illman is also senior survey methodologist for the U.S. Bureau of the Census, where he conducts research on procedures for the census that will take place m the year 200t). 1)ilhnan is the author of` Mail and ~F~leplwtwSurveys: Tke ?atal Desl~n Metlwd (New York: John Wiley and Sons, 1978), identified by the Institute fbr Scientific lnfi)rmation as a "citation classic," in recognition of'having been cited more than 340 times in research papers, and co-author with Pricilla Salant of How to C'onduct Your Own Survey (New York: John Wiley and Sons, 1994). His active research program blends efforts to improve survey-research methods with the study of"issues atgecting rural America. Current projects concern impacts of information technologies on people and rural businesses and the improvement of data collected by mail, telephone, and Face-to-t~ce interviews. 14C.G. Homans, Social Bekal,ior (New York: Harcourt, Bracc, and Wor]d, 1961) p. 1 It).
August 1995 °
Exhibit 2
RESEARCH CONCEPT
"~
POPULATION RESEARCH DESIGN $
mm' n
- - mR " " ' n m" =~ - - - - m" m m= n
m= m m" I
SAMPLE SELECTION
LETTER COMPOSITION, QUESTIONNAIRE DESIGN, SURVEY PLAN PRENOTIFICATION PRETEST
FIRST MAILING Cover letter 1
QUestionnaire
Datacard
FOLLOW-UP Postcard to thank or remind
~--
SECOND MAILING Cover letter
Oestionna
THIRD FOLLOW-UP Telephone call
70
HOTELAND RESTAURANTADMINISTRATIONQUARTERLY
manager, and include a signature in ballpoint or fountain pen. In the letter, which should be as brief as possible (two or '~ three short paragraphs maxi"2 mum), present the purposes / and potential benefits of collecting the survey data. Explain the importance of completing the survey, emphasizing the limited size and selectivity o f the sample. Pinally, point out that the data will be held in confidence; respondents' names will not be associated with their answers to questions, and the data will be reported in aggregate form only. Questionnaire. Make the questionnaire clear, concise, and easy to fill out. A questionnaire that looks formidable is less likely to be completed, because respondents think it will take too much time. Focus on what you want to find out; don't use it as a fishing expedition for your next seven research projects. To make the task appear brief, reduce the size of the questionnaire, use an attractive layout, minimize the nmnber of questions, and eliminate repetition. Minimize confusion by simplifying complex questions and adding directions. Pretesting the cover letter and the questionnaire with a small sample of survey respondents is a good idea and will increase the response in the long run. A pretest will also highlight questions that respondents won't want to answer because of privacy or embarrassment concerns, or because their meaning is difficult to decipher.
!
L
Return envelope. Include a stamped, addressed return envelope with first-class postage. 1)oing so saves the respondent time. Although the time saved may be small, the benefits of responding are also small. This is another way to reduce the apparent cost of responding below
cONFtDENTIAsuRVEY L / _ .~do1~nt ol Gonven
~I I !
/~
j/
\, # ' '
that o£ the perceived benefits. Supplying the return envelope also demonstrates, at least subliminally, that you are sensitive to the issue of cost-benefit and have spent your money to save the respondent 32 cents. Moreover, the postage also indicates that the researcher considers each response so important that he or she is willing to pay extra for it. To make sure the technique is not lost on the respondent, use a letterhead business envelope with a large, colorful, commemorative stamp. Data-request card. Given the general lack of resources to provide incentives to potential respondents, one way to promote value in participation is to offer to share the results of the research with respondents. That is easily accomplished by enclosing in the mailing package a brightly colored self-addressed postcard asking the researchers to send to the respondent a copy o£ the data. Respondents .just have to fill it out and mail it (separately from the questionnaire to preserve anonymity) at their expense.
Main characteristics of the cover letter • Personalized address Address the letter to a specific
individual in the organization • S e l e c t i v e n e s s of a u d i e n c e
Point out that the participants were carefully chosen for their experience or expertise • I m p o r t a n c e of participation
Explain that each reply will enhance the value of the results and therefore benefit the industry • W h o is s p o n s o r i n g the s u r v e y
Identify the primary sponsor as a way to increase the survey's credibility and importance • P u r p o s e a n d benefits of the s t u d y
Describe why such a study is important and who will benefit from its findings • Confidentiality
Ensure in writing the confidential nature of each returned survey • D a t a - e x c h a n g e offer
Offer to share the results of the study upon request • Original s i g n a t u r e of s e n d e r
Sign each letter by hand
August 1995 •
\Vi~>tll i~ I I December 12, 1994 Mr Matt Barge Convention ServicesManager The Westin Hotel Seattle Seattte, WA 98121 Dear Mr Berge, _ ,= ~rc1e
Last week we mailed you a questionnaire about the career and organizational development of convention services managers. If you already have completed and returned the questionnaire please accept our sincere thanks. If not, please do so today. We are especially grateful for your help because we believe your responses will increase our collective knowledge about the convention services profession. If by chance you did not receive the questionnaire or it was misplaced, please cell us at (509) 335-5766, and we will get one in the mail to you today, Sincerely,
Rhonda J. Montgomery Assistant Professor University of South Carolina
Denney G. Rutherford Professor Washington State University
About three weeks ago we wrote to you seekingyour opimons about the career and organizationaldevelopmentof conventionservices managers. As of today, we have not received your completed questionnaire We realize that you may not have had the time to comptete it However,we woutd greatly appreciate hearing from you The only way we know of gathering informationabout conventionservices managers is to ask people like yourself It would be helpful if you would take a few minutesto answer the enclosed questionnaire. We are writing to you again because of the significanceeach questionnaire has to the usefulnessof this study. Yournamewas drawn through a scientific sampling process in which every conventionservice manager in the United States had an equal chanceof being selected. In order for this study to be truly representative, it is essential that each managerin the sample return the questionnaire. in the event that your questionnairehas been misplaced, we have enclosed another copy. If you have any questions about the survey, please call us at (509) 335-5766 Your participation is greatly appreciated, Sincerely,
Rhonda J. Montgomery Assistant Professor University of South Carolina
DenneyG Rutherford Professor Washington State University
Enclosures
The data-request cards can provide useful information, and data from the cards can reinforce a study's reliability. For example, two researchers surveyed the top 100 hotel firms (in terms of total rooms) on a legal topic and received a 48-percent return. An analysis o f the data-request cards, however, suggested that the respondents accounted for over two-thirds of the hotel rooms in the United States, so they felt more comfortable drawing conclusions from the respondents' data. *sAlthough some questionnaire recipients may request the results ~SD.G. Rutherford and J.P. McConnell, "'De Facto Security Standards: Operators at Risk;' Cornell Hotel and Restaurant Administration Quarterly, VoL 31, No. 4 (February 1991), pp. 106-1 t7.
72
and not return the questionnaire, most will conform to the ideas of social-exchange theory and send in the data-request card only if they also send in the questionnaire. For your part, as soon as you have data, reprint your questionnaire supplying frequencies, means, and ranges and mail it to those who returned data-request cards. That fulfills your social-exchange obligation and may even generate some surprising offers of help. F o U o w - u p . Carefully designed follow-ups are essential, The mailings and telephone calls are more than simple reminders. Each followup contact is an opportunity to appeal for completion and return o f the questionnaire. To that end, each
HOTELANDRESTAURANTADMINISTRATIONQUARTERLY
follow-up should use a slightly different approach and include three carefully timed contacts. O n e week after the initial mailing, send all respondents a postcard. It should thank those w h o have responded and courteously remind those w h o have not. It may be plain, or it may include a picture or drawing; there is some evidence that postcards that repeat a theme or central idea of the questionnaire or otherwise visually remind the respondent to complete it increase response rates. " I~'A.J. Nederhof, "Effects of Final Telephone Reminder and Questionnaire Cover Design in Mail Surveys" (Leiden, the Netherlands: Center for Science and Technology Studies, University of Leiden, 1989; unpublished manuscript).
Three weeks after the initial mailing, send nonrespondents a letter and replacement questionnaire. This mailing is nearly the same in design and appearance as the original. However, the cover letter informs nonrespondents that you have not received their questionnaire and appeals for its return. ~7 Make the third follow-up contact by telephone. The personal nature of the telephone contact underscores the importance of the survey. Politely ask whether the potential respondent has received the prior mailings and try to persuade her or him to complete the questionnaire. Offer to send a replacement questionnaire.
Cost Considerations Some critics argue that the costs of mailing to an adequate sample of firms and using the total-design method are prohibitive) s Obviously, first-class mail and following up with nonrespondents by telephone costs more than third-class mail; and multiple follow-ups cost more than single mailings. However, the costs of obtaining the same number of completed questionnaires using a single mailing and using the total-design method are almost identical, because of the difference in response rates (see Exhibit 3). If you assume that 15 percent of the questionnaires ~7Each potential respondent is assigned a nunrber and that number is written on the questiomlaire (usually by hand). As the questionnaires are returned, the researcher keeps track of the surveys using those numbers. In that way, potential participants who havei~'t returned their questionnaires can be identified and targeted for a follow-up mailing or telephone call. When all the mailings and follow-ups are completed, the list of numbers is discarded and the anonymity of respondents is preserved. ~XE.]). deLeeuw and J. Hox, "The Effects of Response-Stimulating Factors on l~_esponse Rates and Data Quality in Mail Surveys," Journal q/ Q].-ficial Stattstics, Vol. 4, No. 3 (I 988), pp. 241-249.
Exhibit 3
Total-design method
Single-mailing method
784 85 60 400
2,353 85 20 400
$3,920 $1,568 $784
$11,765 0 0
Required beginning sample size Estimated percentage of questionnaires deliverable Estimated response rate (percentage) Number of completed questionnaires Cost of initial mailing ($5 each) Cost of personalization ($2 each) Cost of postcard follow-up ($1 each) Cost of second mailing to 50 percent of the beginning sample ($7 each) Cost of telephone call to 25 percent of the beginning sample ($8 each) Total cost
are undeliverable, to obtain 400 completed questionnaires using the single-mailing method, which has a response rate of about 20 percent, you would have to contact 2,353 businesses. At a hypothetical if not typical $5 for each eight-page questionnaire packet, the total cost for the mailing would be $l 1,765. To obtain 400 completed questionnaires using the total-design method, which has a response rate of about 60 percent, you would have to contact only 784 businesses. In this example, the total cost of the survey would be $10,584, or about 10 percent less than the single-cont a c t method. The remaining monies can be invested in a fourth followup, if desired, to raise the response rate even lnore. Thus, although the total-design method involves more mailings and even pays for return postage, it is no more expensive than a single-mailing survey for the same number of completed responses and has a distinct advantage: the high response
$2,744
0
$1,568
0
$10,584
$11,765
rate. The higher the response rate, the lower the error in estimating true conditions in the surveyed hospitality environment.
Conclusion Researchers who base their conclusions on surveys with low response rates run the risk of drawing incorrect conclusions and contributing misinformation to the industry's knowledge base. If, however, we regularly achieve survey response rates of about 60 percent, we will draw more-accurate conclusions that will strengthen that knowledge base. A major step in achieving that goal is the development of the totaldesign method. If researchers will personalize their cover letters, use questionnaires that are easy to complete, and make follow-up contacts, they will achieve higher response rates and reduce the bias that results from low response. The result will be research reports that are more academically sound.
August1995 •
ospitality researchers, consultants, community-development groups, and others conduct surveys for many reasons. They survey guests to learn about their needs and predict trends. They learn about the availability of labor, supports for and barriers to doing business in the community, and why employees resign. Survey research makes important contributions to the industry and has informed managers, for example, about the development of convention-services management as
M. Chris Paxson, Ph.D., an assistant professor of hotel and restaurant administration at Washington State University, thanks Ken Clark for developing the graphicsfor this paper. © 1995,CornellUniversity
88
HOTELANDRESTAURANTADMINISTRATIONQUARTERLY
a career, corporate travel needs, and guests' views about tipping) Since 1990 more than a fourth o f the articles in the Cornell Quarterly have been based on the results o f organizational surveys. Surveying businesses, as opposed to the general public, however, has several problems. First, it may be difficult to get the names o f current owners or managers so that mailings can be personalized. Second, mailing lists o f companies from government sources or trade associations sometimes contain the name and address o f the accounting or auditing firm that handles a company's affairs and not the company's own street address. Third, the high failure rate for new businesses means that there are likely to be more obsolete addresses in a sample o f businesses than in a sample o f the general population o f comparable size. 2 Particular features o f the hospitality industry may present other problems for survey projects. Because most hospitality companies are small and have limited resources and time, respondents may be less likely to respond. U n d e r those conditions, survey salience and appeal are i m p o r t a n t . T i m i n g is also a concern because ofseasonality. N o t only must the survey researcher k n o w an address; he or she must also understand each business and its cycles. ~R.J. Montgomery and D.G. Rutherford, "A Profile of Convention-Services Professionals," Cornell Hotel and Restaurant Administration Quarterly, Vol. 35, No. 6 (December 1994), pp. 47-57; P,.A. Bell, "Corporate Travel-Management Trends--Implications for Hotel-Marketing Strategies," Cornell Hotel and Restaurant Administration Quarterly, Vol. 34, No. 2 (April 1993), pp. 30-39; and N.E. Brown and S.A. Rolle, "Tips versus Service Charges:The Iowa Scene," Cornell Hotel and Restaurant Administration Quarterly, Vol. 32, No. 1 (May 199l), pp. 74-81. 2M.C. Paxson, D.A. Dilhnan, and J. Tarnai, "Improving Response to Business Mail Surveys," in Survey Methods f~r Businesses, Farms, and Instimtians, ed. B.G. Cox and D.A. Binder (New York: Wiley--lnterscience, 1995), pp. 305-306.
T h e r e is also the individual's personal decision regarding whether to cooperate with the survey.We are asking respondents to do something extra, something that is probably not in their job description. Also, high turnover rates mean that respondents' personal identities may be less closely tied to their jobs, so that hospitality employees may be less likely to take the time to respond. For those and other reasons, organizational surveys typically experience low response rates. Because research findings may be considered part o f a firm's competitive advantage and therefore not be published, it is difficult to reliably estimate the extent o f nonresponse. Unofficial estimates put response rates at 10 to 20 percent, and some actually measure l to 10 percent. 3 In a recent issue o f a hospitality journal, response rates o f 3 l percent or less were c o m m o n . O n e researcher told me about an 18-percent response rate for a mail survey o f c o m m u n i t y businesses, explaining that such rates are typical. 4 Until recently, mail surveys o f the general population have also reported low response rates, even w h e n questionnaires were brief? Low response rates were accepted as inevitable. Also, it is c o m m o n l y believed to be too costly to increase response rates to surveys, even though such thinking is erroneous. It is therefore not surprising that researchers surveying hospitality establishments also considered low response rates inevitable and tolerable. ~See, for instance, Starch's comments reported in J.M. Converse, Survey Research in the United States (Berkeley; CA: University of" California Press, 1987), p. 98. 4This report was part o f a 1989 survey conducted by a market-research firm in Spokane, Washington. 5D.A. Dillman, Mail and "11,lephoneSurveys: The Total Desgn Method (New York: John Wiley and Sons, 1978), p. vii.
Low response rates are a concern. First, they introduce a major source o f error through nonresponse bias. N o matter how great the n u m b e r o f completed questionnaires, if only 20 percent o f the recipients fill out and return a questionnaire or complete a telephone interview, those opinions probably do not adequately represent the population. In other words, opinions o f the 80 percent w h o do not respond may differ significantly from those o f the 20 percent who do respond. This example illustrates nonresponse bias resulting from a low response rate: Two recent surveys were conducted in the same community at about the same time. Both asked respondents to estimate the percentage o f their customers located in one geographic region. The first survey, with a response rate o f 18 percent, concluded that 89 percent o f their customers were located in the region. T h e second survey, with a response rate o f 59 percent, concluded that only 73 percent o f customers were regional residents. Given the higher response rate for the second survey, its conclusions would be more reliable than those o£ the first, and the difference could affect a firm's marketing strategy. T h e second survey provides an important clue that in the first survey a disproportionate number o f managers answered whose perspective on the market was a local one. A second reason for concern about low response rates is the risk associated with making business decisions based on a small sample o f responses. What may meet the needs o f the 1 percent who c o n > pleted a guest c o m m e n t card may not reflect the views o f the 99 percent w h o did not. To maximize scarce resources and minimize risk, information should be as representative as possible.
Auoust 1995 •
Exhibit I _
Definition
To m a x i m i z e i m p a c t . . .
Saliency
Importance of the survey topic to the respondent
• focus on the research concept • design a good questionnaire, cover letter, and survey process
Sponsorship
Organization that develops or implements the survey
• have a legitimate, prestigious organization or agency sponsor the survey (making it clear that the sponsor's motive is not to sell a product or service) • use the sponsor's normal business stationery
Follow-up
Series of planned additional contacts by mail or telephone
• make courteous contacts to thank respondents and remind nonrespondents to participate • include a postcard, another mailing, and a telephone contact
Incentives
Tangible token rewards
• send cash or a small gift, preferably before the questionnaire is completed, or offer to send a copy of the results
Sources: Baumgartner & Heberlein; Goyder; and Yu and Cooper (see footnote 6 on this page).
Increasing ResponseRates Recent reviews summarize many research studies on responseincreasing techniques for mail surveys of the general population. 6 The reviews isolate four factors-saliency, sponsorship, follow-up, and incentives--and a possible fifth fiactor--p erson alization--found ~'See: R.M. Baumgartner and T.A. Heberlein, "Recent P, esearch on Mailed Questionnaire Response Rates," in Maleing E(fi,ctive Use of Mailed Questionnaires', ed. D.C. Lockhart (San Francisco: Jossey-Bass, 1984), p. 65; J.C. Goyder, "Further Evidence on Factors Affecting lZ,esponse Rates to Mailed Questionnaires," American Sociological Review, Vol. 47, No. 4 (1982), pp. 550-553; and J.Yu and H. Cooper, "A Quantitative Review of Research Design Effects on Response Rates to Questionnaires," Journal q[ Marketing Research, Vol. 20, No. l (1983), pp. 36-44.
68
most effective in increasing response rates in public opinion surveys (see Exhibit 1). Saliency. H o w important is the survey to the respondent? H o w well does it stand out from the average executive's daily blizzard of mail? The more salient the survey topic is to the respondent, the greater the likelihood of a completed questionnaire. To some extent, the saliency of the topic can be enhanced by the design and appearance of the questionnaire and its cover letter. Sponsorship. Surveys sponsored by universities, business schools, and hospitality programs may obtain better response rates than surveys sponsored by others. Respondents may return questionnaires to the
HOTELANDRESTAURANTADMINISTRATIONQUARTERLY
university researcher more readily because of psychological indebtedness; for example, they might want to return past favors for their education or for research that has aided them. Or the prestige and legitimacy associated with the sponsor may improve the response. Moreover, because of the sponsor, a supervisor may approve of a study and communicate that approval to the employees; that would increase the probability of its being returned. 7 Follow-up. Follow-up contacts can also increase response rates, and the type of follow-up affects the response rate. A follow-up reminder by telephone is more effective than a mailed follow-up, for instance. ~A possible explanation is that a followup by telephone underscores the importance of the study and makes the respondents feel rewarded by the attention they receive." Followups may also enhance the personalization aspect of the survey, which can increase response rates. Incentives. Providing monetary incentives is one of the most effective means of increasing response rates in surveys of the general public.>Monetary incentives may not be appropriate with business surveys, however, because the person who opens the envelope and pockets the cash may not be the person you'd like to complete the survey. Furthermore, the dean or director may not like the idea of faculty members stuffing surveys with 7Dillman, p. i6. ST.A. Heberlein and R.M. Baumgarmer, "Factors Affecting Response Rates on Mailed Questionnaires: A Quantitative Analysis of the Published Literature," American Sociological Revieu; Vol. 43, No. 4 (1978), pp. 447-462. 'q).A. Dillman and I).E. Moore, "hnproving Response to Mail Surveys: Results from Five Experiments" (working paper, Social and Ecunomic Sciences Research Center, Washington State University, 1983), p. 18. >R.J. Fox, M.R. Crask, and J. Kim, "Mail Survey Response Rate: A Meta-analysis of Selected Techniques for Inducing Response," Public Opiniotl Quarterly, Vol. 52, No. 4 (1988), pp. 467-491.
scarce budget dollars. It may be necessary to rely on other responseenhancing techniques, like datarequest cards. P e r s o n a l i z a t i o n . Some researchers report increased response rates with personalization, tr while others report decreased response rates. ~2 It appears that personalization may interact with other characteristics of the survey, but most experienced survey researchers do not believe it hurts. However, it may be difficult to obtain the names of current owners and managers to use in personalizing cover letters. For example, the membership directories of the American Hotel and Motel Association and the National Restaurant Association provide the names o f designated executives of member companies but not those of individual hotel or restaurant managers.
The Total-DesignMethod in his book Mail and Telephone Surveys: The Total Des~¢n Method, D o n A. Dillman proposes a theoretical and practical framework for mail questionnaires that integrates several of the above procedures for increasing response rates and adds a few of his own (see Exhibit 2, on the next page). ~ Dillman views the questionnaire as a social exchange between the researcher and the respondent.
According to social-exchange theory, the recipient will return the questionnaire if the costs of doing so are less than the perceived benefits.~4 Time and effort are perhaps the major costs. A recipient may reject a poorly designed, lengthy, or confusing questionnaire simply because it looks like it will take too long to complete or it brings into question the integrity of the research. For that reason, questionnaires and mailing processes must be designed carefully. Because researchers seldom can offer truly valuable rewards and questionnaires are often lengthy, the researchers must be concerned with all the controllable details of the survey design. Dillman's method, which he calls the total-design method, is based on techniques used by researchers at the Social and Economic Sciences Research Center in 26 business mail surveys between November 1989 and November 1992.The researchers surveyed business samples ranging in size from 74 to 4,300 (average: 1,604) and achieved average response rates of 52.6 percent.You too can achieve such results if you follow this method as I describe below. Cover letter. Address the cover letter individually, if possible, using the name of the business owner or
t JSee, for instance: E.H. Carpenter, "Personalizing Mail Surveys: A Replication and Reasscssment,"
Public Opinion Quarterly, Vo]. 38, No. 4 (1974), pp. 614-620; and I).A. l)ilhnan and J.H. Frey, "Contribution of" Personalizatiori to Mail Questionnaire P,.esponse as an Elemem of"a Previously Tested Method," Journal qfApplied Psyclwlt{qB Vol. 59, No. 3 (1974), pp. 297-301. *~-A.I<.Andreasen, "Personalizing Questiom~aire Correspondence," Public Opinio~t Quarterly, Vol. 34, No. 1 (1970), pp. 273-277; and M o. Houston and R.W. Jeff'erson, "The Negative Elf`cots of" Persunalization on Response Patterns in Mail Surveys,"Joumal (!fMarketil{~, Researck, Vol. 12, No. I (1975), pp. I14-117. ~l)un A. l)illman, Ph.D., is the director of the Social and Economic Sciences Research Center and a pro(essor in the departments of sociology and rural sociology at Washington State University. The center conducts about 50 survey-related projects each year using mail, telephone, and Eace-to-~ace survey methods, l)illman is also senior survey methodologist for the U.S. Bureau of the Census, where he conducts research on procedures for the census that will take place m the year 200t). 1)ilhnan is the author of` Mail and ~F~leplwtwSurveys: Tke ?atal Desl~n Metlwd (New York: John Wiley and Sons, 1978), identified by the Institute fbr Scientific lnfi)rmation as a "citation classic," in recognition of'having been cited more than 340 times in research papers, and co-author with Pricilla Salant of How to C'onduct Your Own Survey (New York: John Wiley and Sons, 1994). His active research program blends efforts to improve survey-research methods with the study of"issues atgecting rural America. Current projects concern impacts of information technologies on people and rural businesses and the improvement of data collected by mail, telephone, and Face-to-t~ce interviews. 14C.G. Homans, Social Bekal,ior (New York: Harcourt, Bracc, and Wor]d, 1961) p. 1 It).
August 1995 °
Exhibit 2
RESEARCH CONCEPT
"~
POPULATION RESEARCH DESIGN $
mm' n
- - mR " " ' n m" =~ - - - - m" m m= n
m= m m" I
SAMPLE SELECTION
LETTER COMPOSITION, QUESTIONNAIRE DESIGN, SURVEY PLAN PRENOTIFICATION PRETEST
FIRST MAILING Cover letter 1
QUestionnaire
Datacard
FOLLOW-UP Postcard to thank or remind
~--
SECOND MAILING Cover letter
Oestionna
THIRD FOLLOW-UP Telephone call
70
HOTELAND RESTAURANTADMINISTRATIONQUARTERLY
manager, and include a signature in ballpoint or fountain pen. In the letter, which should be as brief as possible (two or '~ three short paragraphs maxi"2 mum), present the purposes / and potential benefits of collecting the survey data. Explain the importance of completing the survey, emphasizing the limited size and selectivity o f the sample. Pinally, point out that the data will be held in confidence; respondents' names will not be associated with their answers to questions, and the data will be reported in aggregate form only. Questionnaire. Make the questionnaire clear, concise, and easy to fill out. A questionnaire that looks formidable is less likely to be completed, because respondents think it will take too much time. Focus on what you want to find out; don't use it as a fishing expedition for your next seven research projects. To make the task appear brief, reduce the size of the questionnaire, use an attractive layout, minimize the nmnber of questions, and eliminate repetition. Minimize confusion by simplifying complex questions and adding directions. Pretesting the cover letter and the questionnaire with a small sample of survey respondents is a good idea and will increase the response in the long run. A pretest will also highlight questions that respondents won't want to answer because of privacy or embarrassment concerns, or because their meaning is difficult to decipher.
!
L
Return envelope. Include a stamped, addressed return envelope with first-class postage. 1)oing so saves the respondent time. Although the time saved may be small, the benefits of responding are also small. This is another way to reduce the apparent cost of responding below
cONFtDENTIAsuRVEY L / _ .~do1~nt ol Gonven
~I I !
/~
j/
\, # ' '
that o£ the perceived benefits. Supplying the return envelope also demonstrates, at least subliminally, that you are sensitive to the issue of cost-benefit and have spent your money to save the respondent 32 cents. Moreover, the postage also indicates that the researcher considers each response so important that he or she is willing to pay extra for it. To make sure the technique is not lost on the respondent, use a letterhead business envelope with a large, colorful, commemorative stamp. Data-request card. Given the general lack of resources to provide incentives to potential respondents, one way to promote value in participation is to offer to share the results of the research with respondents. That is easily accomplished by enclosing in the mailing package a brightly colored self-addressed postcard asking the researchers to send to the respondent a copy o£ the data. Respondents .just have to fill it out and mail it (separately from the questionnaire to preserve anonymity) at their expense.
Main characteristics of the cover letter • Personalized address Address the letter to a specific
individual in the organization • S e l e c t i v e n e s s of a u d i e n c e
Point out that the participants were carefully chosen for their experience or expertise • I m p o r t a n c e of participation
Explain that each reply will enhance the value of the results and therefore benefit the industry • W h o is s p o n s o r i n g the s u r v e y
Identify the primary sponsor as a way to increase the survey's credibility and importance • P u r p o s e a n d benefits of the s t u d y
Describe why such a study is important and who will benefit from its findings • Confidentiality
Ensure in writing the confidential nature of each returned survey • D a t a - e x c h a n g e offer
Offer to share the results of the study upon request • Original s i g n a t u r e of s e n d e r
Sign each letter by hand
August 1995 •
\Vi~>tll i~ I I December 12, 1994 Mr Matt Barge Convention ServicesManager The Westin Hotel Seattle Seattte, WA 98121 Dear Mr Berge, _ ,= ~rc1e
Last week we mailed you a questionnaire about the career and organizational development of convention services managers. If you already have completed and returned the questionnaire please accept our sincere thanks. If not, please do so today. We are especially grateful for your help because we believe your responses will increase our collective knowledge about the convention services profession. If by chance you did not receive the questionnaire or it was misplaced, please cell us at (509) 335-5766, and we will get one in the mail to you today, Sincerely,
Rhonda J. Montgomery Assistant Professor University of South Carolina
Denney G. Rutherford Professor Washington State University
About three weeks ago we wrote to you seekingyour opimons about the career and organizationaldevelopmentof conventionservices managers. As of today, we have not received your completed questionnaire We realize that you may not have had the time to comptete it However,we woutd greatly appreciate hearing from you The only way we know of gathering informationabout conventionservices managers is to ask people like yourself It would be helpful if you would take a few minutesto answer the enclosed questionnaire. We are writing to you again because of the significanceeach questionnaire has to the usefulnessof this study. Yournamewas drawn through a scientific sampling process in which every conventionservice manager in the United States had an equal chanceof being selected. In order for this study to be truly representative, it is essential that each managerin the sample return the questionnaire. in the event that your questionnairehas been misplaced, we have enclosed another copy. If you have any questions about the survey, please call us at (509) 335-5766 Your participation is greatly appreciated, Sincerely,
Rhonda J. Montgomery Assistant Professor University of South Carolina
DenneyG Rutherford Professor Washington State University
Enclosures
The data-request cards can provide useful information, and data from the cards can reinforce a study's reliability. For example, two researchers surveyed the top 100 hotel firms (in terms of total rooms) on a legal topic and received a 48-percent return. An analysis o f the data-request cards, however, suggested that the respondents accounted for over two-thirds of the hotel rooms in the United States, so they felt more comfortable drawing conclusions from the respondents' data. *sAlthough some questionnaire recipients may request the results ~SD.G. Rutherford and J.P. McConnell, "'De Facto Security Standards: Operators at Risk;' Cornell Hotel and Restaurant Administration Quarterly, VoL 31, No. 4 (February 1991), pp. 106-1 t7.
72
and not return the questionnaire, most will conform to the ideas of social-exchange theory and send in the data-request card only if they also send in the questionnaire. For your part, as soon as you have data, reprint your questionnaire supplying frequencies, means, and ranges and mail it to those who returned data-request cards. That fulfills your social-exchange obligation and may even generate some surprising offers of help. F o U o w - u p . Carefully designed follow-ups are essential, The mailings and telephone calls are more than simple reminders. Each followup contact is an opportunity to appeal for completion and return o f the questionnaire. To that end, each
HOTELANDRESTAURANTADMINISTRATIONQUARTERLY
follow-up should use a slightly different approach and include three carefully timed contacts. O n e week after the initial mailing, send all respondents a postcard. It should thank those w h o have responded and courteously remind those w h o have not. It may be plain, or it may include a picture or drawing; there is some evidence that postcards that repeat a theme or central idea of the questionnaire or otherwise visually remind the respondent to complete it increase response rates. " I~'A.J. Nederhof, "Effects of Final Telephone Reminder and Questionnaire Cover Design in Mail Surveys" (Leiden, the Netherlands: Center for Science and Technology Studies, University of Leiden, 1989; unpublished manuscript).
Three weeks after the initial mailing, send nonrespondents a letter and replacement questionnaire. This mailing is nearly the same in design and appearance as the original. However, the cover letter informs nonrespondents that you have not received their questionnaire and appeals for its return. ~7 Make the third follow-up contact by telephone. The personal nature of the telephone contact underscores the importance of the survey. Politely ask whether the potential respondent has received the prior mailings and try to persuade her or him to complete the questionnaire. Offer to send a replacement questionnaire.
Cost Considerations Some critics argue that the costs of mailing to an adequate sample of firms and using the total-design method are prohibitive) s Obviously, first-class mail and following up with nonrespondents by telephone costs more than third-class mail; and multiple follow-ups cost more than single mailings. However, the costs of obtaining the same number of completed questionnaires using a single mailing and using the total-design method are almost identical, because of the difference in response rates (see Exhibit 3). If you assume that 15 percent of the questionnaires ~7Each potential respondent is assigned a nunrber and that number is written on the questiomlaire (usually by hand). As the questionnaires are returned, the researcher keeps track of the surveys using those numbers. In that way, potential participants who havei~'t returned their questionnaires can be identified and targeted for a follow-up mailing or telephone call. When all the mailings and follow-ups are completed, the list of numbers is discarded and the anonymity of respondents is preserved. ~XE.]). deLeeuw and J. Hox, "The Effects of Response-Stimulating Factors on l~_esponse Rates and Data Quality in Mail Surveys," Journal q/ Q].-ficial Stattstics, Vol. 4, No. 3 (I 988), pp. 241-249.
Exhibit 3
Total-design method
Single-mailing method
784 85 60 400
2,353 85 20 400
$3,920 $1,568 $784
$11,765 0 0
Required beginning sample size Estimated percentage of questionnaires deliverable Estimated response rate (percentage) Number of completed questionnaires Cost of initial mailing ($5 each) Cost of personalization ($2 each) Cost of postcard follow-up ($1 each) Cost of second mailing to 50 percent of the beginning sample ($7 each) Cost of telephone call to 25 percent of the beginning sample ($8 each) Total cost
are undeliverable, to obtain 400 completed questionnaires using the single-mailing method, which has a response rate of about 20 percent, you would have to contact 2,353 businesses. At a hypothetical if not typical $5 for each eight-page questionnaire packet, the total cost for the mailing would be $l 1,765. To obtain 400 completed questionnaires using the total-design method, which has a response rate of about 60 percent, you would have to contact only 784 businesses. In this example, the total cost of the survey would be $10,584, or about 10 percent less than the single-cont a c t method. The remaining monies can be invested in a fourth followup, if desired, to raise the response rate even lnore. Thus, although the total-design method involves more mailings and even pays for return postage, it is no more expensive than a single-mailing survey for the same number of completed responses and has a distinct advantage: the high response
$2,744
0
$1,568
0
$10,584
$11,765
rate. The higher the response rate, the lower the error in estimating true conditions in the surveyed hospitality environment.
Conclusion Researchers who base their conclusions on surveys with low response rates run the risk of drawing incorrect conclusions and contributing misinformation to the industry's knowledge base. If, however, we regularly achieve survey response rates of about 60 percent, we will draw more-accurate conclusions that will strengthen that knowledge base. A major step in achieving that goal is the development of the totaldesign method. If researchers will personalize their cover letters, use questionnaires that are easy to complete, and make follow-up contacts, they will achieve higher response rates and reduce the bias that results from low response. The result will be research reports that are more academically sound.
August1995 •