- Email: [email protected]
Commentary on “Common Method Bias in Marketing: Causes, Mechanisms, and Procedural Remedies”
Journal of Retailing 88 (4, 2012) 563–566
Commentary
Commentary on “Common Method Bias in Marketing: Causes, Mechanisms, and Procedural Remedies” Hans Baumgartner a,∗ , Bert Weijters b,c a
Smeal College of Business at The Pennsylvania State University, Department of Marketing, 482 Business Building, University Park, PA 16802, United States b Vlerick Business School, Reep 1, B-9000 Ghent, Belgium c Ghent University, Tweekerkenstraat 2, B-9000 Ghent, Belgium
Abstract MacKenzie and Podsakoff (M&P) have written a very useful guide for researchers in retailing and marketing on how to deal with the problem of common method bias in survey research. We applaud their effort to provide procedural remedies that can be implemented before the data are collected in an attempt to counter satisficing and prevent the damaging effects of method bias in survey data. In this comment we identify several issues that merit further research, including the need to formulate a multi-motive model of survey response, the need to identify determinants of method bias other than satisficing, the need to evaluate the relative importance of different sources of method bias (as well as in-depth research on the major sources), the need for more extensive pretesting before administering a survey, and the challenge of how to enhance systematic processing in the age of internet panels. © 2012 New York University. Published by Elsevier Inc. All rights reserved. Keywords: Method bias; Survey methods; Satisficing
MacKenzie and Podsakoff (M&P) have written a very useful guide for researchers in retailing and marketing on how to deal with the problem of common method bias in survey research. Their article builds on and extends their earlier writings on the subject (Podsakoff et al. 2003; Podsakoff, MacKenzie, and Podsakoff 2012) by focusing on procedural remedies that can be implemented before the data are collected in order to prevent the damaging effects of nonsubstantive responding on observed responses. This is in contrast to post hoc correction procedures, which have been the focus of much of prior research. The eminent statistician Ronald Fisher once remarked, “To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of. To utilise this kind of experience he must be induced to use his imagination, and to foresee in advance the difficulties and uncertainties with which, if they are not foreseen, his investigations will be beset” (Fisher 1938, p. 17).
∗
Corresponding author. Tel.: +1 814 863 3559; fax: +1 814 865 3015. E-mail addresses: [email protected] (H. Baumgartner), [email protected] (B. Weijters).
Although methodologically oriented researchers in marketing have emphasized the importance of controlling measurement error, particularly systematic measurement error, it is our impression that common method biases have been taken much more seriously in other fields (e.g., management) and that empirical researchers in marketing (apart from a ritualistic concern with such things as nonresponse bias or social desirability bias) often do not seem to pay much attention to method artifacts. Since the post hoc procedures described in the literature are not always effective and usually cumbersome to implement, empirical researchers’ reticence to follow the advice offered by methodologists may be somewhat understandable. The approach proposed by M&P, which consists of critically evaluating, during the design of the study, a checklist of factors that might lead to common method bias (organized around the researcher’s goals to increase respondents’ ability and motivation to respond accurately, and increase the difficulty of satisficing) should go a long way toward preventing the occurrence of method bias. In this note, we would like take the opportunity to draw researchers’ attention to several other issues related to method bias that we believe have been underemphasized in the literature. The comment is written in the spirit of increasing researchers’ concern with common method bias, which will hopefully contribute to more valid research findings.
0022-4359/$ – see front matter © 2012 New York University. Published by Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jretai.2012.10.003
564
H. Baumgartner, B. Weijters / Journal of Retailing 88 (4, 2012) 563–566
The need to formulate a multi-motive model of survey response Building on earlier work by Krosnick (1991) and others, M&P view the respondent’s task as a tradeoff between optimizing and satisficing. Furthermore, as stated by M&P (p. 6), “when respondents are satisficing rather than optimizing they will be more likely to respond stylistically, and their responses will be more susceptible to method bias.” Since satisficing occurs when respondents lack the ability or motivation to respond accurately, which depends on both personal characteristics (e.g., respondents’ capabilities) and situational factors (e.g., the difficulty of the response), the researcher’s task is to select respondents who are sufficiently able and motivated to provide accurate responses and to design and administer the survey in such a way as to stimulate and sustain ability and motivation. Based on M&P’s article, readers might be led to expect that inaccurate responding only occurs when ability and/or motivation are low, which may be true when only accuracy goals are considered. However, once it is acknowledged that there are motives other than accuracy in a survey context, it is apparent that high motivation and high ability can also encourage inaccurate responding. Researchers in other areas have found it useful to consider a variety of motives that may guide behavior in a given situation. Such a multi-motive view may also be beneficial in a survey context. For example, in decision making, Bettman, Luce, and Payne (1998) distinguish the goals of maximizing decision accuracy, minimizing decision effort, minimizing the experience of negative emotions during decision making, and maximizing the ease of justification of a decision. In the attitude area, researchers have distinguished between multiple goals which may influence attitude formation, including accuracy motivation, defense motivation, and impression motivation (Chaiken, Giner-Sorolla, and Chen 1996). Such an expanded list of motives should also prove beneficial in the context of surveys. A well-known example from survey research in which motives other than accuracy are salient is the phenomenon of socially desirable responding (SDR). SDR means that respondents provide answers that make them look good (Steenkamp, De Jong, and Baumgartner 2010). Respondents high in SDR may be strongly motivated to manage a certain impression, and they may also be able to do so, but high motivation and ability do not lead to valid responses. In M&P’s terms, one might argue that respondents’ ability and motivation to provide accurate responses are low in this case, and M&P discuss issues related to social desirability under factors that decrease the motivation to respond accurately. However, research shows that SDR is not always a conscious and deliberate impression-management tactic and instead may reflect honestly held but exaggerated selfperceptions. In this situation, it is not very clear how meaningful it is to equate respondents who are high in ability and high in motivation with respondents who respond accurately. Although less research has dealt with goals such as ease of justification or defense motivation in a survey context, it is easy to see how a desire to justify certain responses or to uphold deeply held beliefs may lead to biased responses even though ability and
motivation are high. It would be useful if future research investigated a broader set of motives that guide respondents when they provide answers to survey questions, because research in other areas (e.g., in the attitude area) has shown quite convincingly that the degree of effort expended on a task is orthogonal to the type of motivation that is salient during the response process (which implies that high ability and high motivation should not be equated with accuracy). The need to consider determinants of method bias other than satisficing Although method bias frequently occurs under peripheral processing conditions, when respondents are trying to satisfice, it has to be acknowledged that satisficing is not the only antecedent of method bias. M&P are careful to focus their discussion of satisficing and method bias on the first three stages of Tourangeau, Rips, and Rasinski’s (2000) model of the survey response process (comprehension, retrieval, and judgment), where the effects of satisficing may be expected to be most pronounced. However, the response stage, in which respondents have to map the judgment onto the response categories provided in order to answer the question, is also a source of numerous method biases, which may not be due to satisficing. The issue of socially desirable responding, which may be viewed as an instance of response editing, has already been mentioned. Another example is a situation in which different respondents have different response mapping functions. To illustrate, consider a task where satisfaction ratings are collected on a 10-point scale. As pointed out by M&P, labeling all response options may reduce response mapping problems, but even then some respondents may think that a score of 7 is pretty good, whereas others may believe that a rating of 8 is what one normally should expect. Although this difference in the use of the response scale would result in method bias, no lack of ability or motivation and no attempt to satisfice are involved (see Weijters, Cabooter, and Schillewaert 2010, esp. Figure 3, for recommendations on which response scale formats to use in different contexts). Cross-cultural research is one area where the problem of noncomparability of responses is acute, because mapping functions have been shown to vary substantially across cultures, in part due to differences in cultural values. For example, it is well-known that collectivist cultures are more prone to acquiescent responding and are more likely to use the midpoint of the scale due to their greater emphasis on conformity and modesty, whereas individualistic cultures have a greater preference for the extremes of the rating scale because decisiveness is a virtue (see Baumgartner and Weijters forthcoming, for an extended discussion of response biases in cross-cultural measurement). There are also more subtle differences in cross-cultural survey response behavior that arise from the fact that the response categories used in rating scales have to be translated (e.g., ‘strongly agree’ in English versus ‘tout à fait d’accord’ in French) and may not be equivalent across cultures. In a recent program of research, Weijters, Geuens, and Baumgartner (2012) investigated whether differences in the intensity (response categories are endorsed less frequently when their labels are more intense) or familiarity
H. Baumgartner, B. Weijters / Journal of Retailing 88 (4, 2012) 563–566
(response categories are endorsed more frequently when their labels are more commonly used in day-to-day language) of endpoint category labels (like strongly agree or completely agree) were more likely to result in differences in response behavior in different languages. Their results indicate that label familiarity is a more important driver of differential use of the extreme response categories. The need to evaluate the relative importance of different sources of method bias M&P discuss a total of 26 contributors to method bias and the list could probably be expanded. Although Tables 2 to 4 can serve as a useful checklist for researchers designing a survey, and the potential remedies proposed provide some guidance on how to prevent method bias, it is likely that researchers will find it challenging to implement all the recommendations in practice. It is also likely that the various sources of method bias will vary in their seriousness and that the proposed remedies will differ in their effectiveness. Researchers would greatly benefit from more explicit counsel on which factors might be most detrimental in different situations and which correction procedures are most likely to increase the validity of the findings. In particular, it would be very useful if there were a more tightly organized structure of the variables that encourage method bias. The conditions identified by M&P as causing method bias are all related to some of the following categories: characteristics of the respondent (e.g., verbal ability, various personality dimensions); factors associated with the instructions, individual questions (e.g., complexity, ambiguity), and the response scales used; more general characteristics of the survey (e.g., grouping of the items, length of the survey, reactions toward the sponsor of the survey); and mode of survey administration (e.g., auditory vs. written presentation of items, presence of an interviewer). Future research should try to develop a model that catalogs the various influences on stylistic responding in a more systematic fashion and, based on this framework, derive recommendations for improving survey research practices. The need to examine important sources of method bias in greater depth Once important contributors to method bias have been identified, it will be necessary to discuss the issues engendered by the bias in question in some depth, as illustrated in the recent article by Tourangeau and Yan (2007) on sensitive questions. Researchers would probably prefer simple procedural solutions to common problems, but unfortunately this is frequently impossible. Our recent research on reversed item method bias is a case in point (see Weijters and Baumgartner 2012). The use of reversed items in surveys is sometimes recommended to inhibit stylistic responding and control for acquiescence, but reversed items also lead to many problems (e.g., low item reliabilities, complex factor structures). It is not meaningful to answer the question of whether or not to use reversed items with a simple yes or no. One important consideration is how an item is reversed – by negating a regular (non-reversed) item or by using
565
an item whose meaning is the polar opposite of a regular item. Based on an extensive literature review, Weijters and Baumgartner suggest that negated items should be used sparingly, if at all, especially when a very similar or identical regular item is reversed by adding the particle ‘not’, because negations are confusing to people. Polar opposite reversals are recommended, but only when they satisfy certain conditions (e.g., respondents should be explicitly or implicitly made aware of the presence of reversed items; antonyms have to be contradictory, rather than contrary; even contradictories may be problematic with East Asian respondents; polar opposite reversals are most useful when they encourage broad coverage of the target construct’s domain of content; regular and reversed items should be dispersed randomly throughout the questionnaire; fully labeled response scales should be used because respondents do not always interpret scale categories correctly; etc.). As can be seen from this long but incomplete list of caveats, although we definitely recommend the use of reversed items in questionnaires, there is an art to writing good reversals, and it appears that many researchers have not mastered this art (and may not be aware of all of the science undergirding the art). The need to engage in more extensive pretesting before administering a survey Being aware of the multitude of conditions that can lead to method bias and thinking about potential remedies is a valuable first step in collecting more valid data. However, it is unlikely that this will be sufficient for averting most problems. Although experts may be able to detect common shortcomings in the design and administration of surveys, there is no substitute for indepth pretesting. It is our impression that academic researchers do relatively little pretesting before they launch a survey, and even if they conduct a pretest, the pretests are not as informative as they could be. For example, a researcher may conduct a prestudy with a small sample of respondents in order to investigate whether the items intended to measure a certain construct yield internally consistent responses with adequate reliabilities. Although such a prestudy may identify some problems, it probably does not provide much information about why a problem occurs, and it is unlikely to identify all major problems. Furthermore, since method bias often inflates correlations, high internal consistency may signal a problem, instead of reassuring the researcher that none exists. Cognitive interviewing is a particularly valuable technique for improving questionnaires. Respondents from the target population are asked to verbalize their reactions to the survey as they answer questions, and the researcher can assess how participants comprehend both the literal and pragmatic meaning of the question, whether and how they retrieve information from memory before providing an answer, what strategies they use to form a judgment and how responses to previous items influence their judgments, and how the answer is formatted to fit the response scale. One disadvantage of cognitive interviewing is that responding to questions when one has to verbalize one’s thought processes may not correspond to the way in which respondents normally complete a survey. If this is a
566
H. Baumgartner, B. Weijters / Journal of Retailing 88 (4, 2012) 563–566
concern, less intrusive techniques such as eye movement recording may be used to gain insight into people’s response behavior. We acknowledge that these procedures are effortful and that researchers may be reluctant to adopt them as a routine component of the survey process. Ideally, methodologically oriented researchers would conduct the necessary studies and provide practically relevant recommendations about how surveys can be made less prone to method biases.
comment, including the need to formulate a multi-motive model of survey response, the need to consider determinants of method bias other than satisficing, the need to evaluate the relative importance of different sources of method bias, the need to examine important sources of method bias in greater depth, the need to engage in more extensive pretesting before administering a survey, and the need to encourage systematic processing in the age of internet panels.
The need to encourage systematic processing in the age of internet panels
References
Some of the recommendations made by M&P involve telling respondents why the survey and their personal opinions are important, stressing the value of accuracy, and encouraging participants to pay attention to all aspects of the survey. Unfortunately, the empirical evidence about how attentive respondents are to instructions and questions is sobering. For example, in Meade and Craig’s (2012) study about careless responding, only 90 percent of respondents disagreed or strongly disagreed with the statements that they slept less than one hour per night or did not understand a word of English; for statements such as ‘I have never brushed my teeth’, ‘All my friends are aliens’, and ‘All my friends say I would make a great poodle’, the percentages were even lower, although one could argue that asking a strange question may entice people to provide a strange answer. There is also solid evidence that respondents often do not read instructions. For example, in their work on Instructional Manipulation Checks, Oppenheimer, Meyvis, and Davidenko (2009) found that between 14 and 46 percent of respondents ignored the instruction not to respond to a question about participation in sports activities and to instead click on the title at the top of the screen in order to proceed to the next screen. The problems of careless responding are probably exacerbated when respondents complete surveys on the internet, often for rather small payments. In addition, there are problems of non-representative sampling, self-selection, and other errors in surveys besides measurement error that may pose serious threats to the validity of surveys. Conclusion In conclusion, we hope that the guidelines provided by M&P on how to deal with the problem of common method bias in survey research will get the attention they deserve. Substantive researchers in retail and marketing need to use more procedural remedies that can be implemented before the data are collected in order to prevent the damaging effects of method bias. Methodological researchers can address a number of remaining gaps in the survey literature, some of which we highlighted in this
Baumgartner, Hans and Bert Weijters (forthcoming), “Response Biases in Crosscultural Measurement,” in Handbook of Culture and Consumer Psychology, Sharon Ng and Angela Y. Lee, eds. Oxford University Press. Bettman, James R., Mary Frances Luce and John W. Payne (1998), “Constructive Consumer Choice Processes,” Journal of Consumer Research, 25 (December), 187–217. Chaiken, Shelly, Roger Giner-Sorolla and Serena Chen (1996), “Beyond Accuracy: Defense and Impression Motives in Heuristic and Systematic Information Processing,” in The Psychology of Action: Linking Cognition and Motivation to Behavior, Peter M. Gollwitzer and John A. Bargh, eds. New York: Guilford, 553–578. Fisher, Ronald A. (1938), “Presidential Address,” Sankhy¯a: The Indian Journal of Statistics, 4 (1), 14–7. Krosnick, Jon A. (1991), “Response Strategies for Coping with the Cognitive Demands of Measures in Surveys,” Applied Cognitive Psychology, 5 (3), 213–36. Meade, Adam W. and Bartholomew S. Craig (2012), “Identifying Careless Responses in Survey Data,” Psychological Methods, 17 (September), 437–55. Oppenheimer, Daniel A., Tom Meyvis and Nicolas Davidenko (2009), “Instructional Manipulation Checks: Detecting Satisficing to Increase Statistical Power,” Journal of Experimental Social Psychology, 45, 867–72. Podsakoff, Philip M., Scott B. MacKenzie, Jeong-Yeon Lee and Nathan P. Podsakoff (2003), “Common Method Biases in Behavioral Research: A Critical Review of the Literature and Recommended Remedies,” Journal of Applied Psychology, 88 (5), 879–903. Podsakoff, Philip M., Scott B. MacKenzie and Nathan P. Podsakoff (2012), “Sources of Method Bias in Social Science Research and Recommendations on How to Control It,” Annual Review of Psychology, 63, 539–69. Steenkamp, Jan-Benedict E.M., Martijn De Jong and Hans Baumgartner (2010), “Socially Desirable Response Tendencies in Survey Research,” Journal of Marketing Research, 47 (April), 199–214. Tourangeau, Roger, Lance J. Rips and Kenneth Rasinski (2000), The Psychology of Survey Response, Cambridge, UK: Cambridge University Press. Tourangeau, Roger and Ting Yan (2007), “Sensitive Questions in Surveys,” Psychological Bulletin, 133 (5), 859–83. Weijters, Bert and Hans Baumgartner (2012), “Misresponse to Reversed and Negated Items in Surveys: A Review,” Journal of Marketing Research, 49 (5), 737–47. Weijters, Bert, Elke Cabooter and Niels Schillewaert (2010), “The Effect of Rating Scale Format on Response Styles: The Number of Response Categories and Response Category Labels,” International Journal of Research in Marketing, 27, 236–47. Weijters, Bert, Maggie Geuens and Hans Baumgartner (2012), “The Effect of Response Category Label Familiarity on Item Response to Likert Scales,” Unpublished working paper, Ghent University.
Commentary
Commentary on “Common Method Bias in Marketing: Causes, Mechanisms, and Procedural Remedies” Hans Baumgartner a,∗ , Bert Weijters b,c a
Smeal College of Business at The Pennsylvania State University, Department of Marketing, 482 Business Building, University Park, PA 16802, United States b Vlerick Business School, Reep 1, B-9000 Ghent, Belgium c Ghent University, Tweekerkenstraat 2, B-9000 Ghent, Belgium
Abstract MacKenzie and Podsakoff (M&P) have written a very useful guide for researchers in retailing and marketing on how to deal with the problem of common method bias in survey research. We applaud their effort to provide procedural remedies that can be implemented before the data are collected in an attempt to counter satisficing and prevent the damaging effects of method bias in survey data. In this comment we identify several issues that merit further research, including the need to formulate a multi-motive model of survey response, the need to identify determinants of method bias other than satisficing, the need to evaluate the relative importance of different sources of method bias (as well as in-depth research on the major sources), the need for more extensive pretesting before administering a survey, and the challenge of how to enhance systematic processing in the age of internet panels. © 2012 New York University. Published by Elsevier Inc. All rights reserved. Keywords: Method bias; Survey methods; Satisficing
MacKenzie and Podsakoff (M&P) have written a very useful guide for researchers in retailing and marketing on how to deal with the problem of common method bias in survey research. Their article builds on and extends their earlier writings on the subject (Podsakoff et al. 2003; Podsakoff, MacKenzie, and Podsakoff 2012) by focusing on procedural remedies that can be implemented before the data are collected in order to prevent the damaging effects of nonsubstantive responding on observed responses. This is in contrast to post hoc correction procedures, which have been the focus of much of prior research. The eminent statistician Ronald Fisher once remarked, “To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of. To utilise this kind of experience he must be induced to use his imagination, and to foresee in advance the difficulties and uncertainties with which, if they are not foreseen, his investigations will be beset” (Fisher 1938, p. 17).
∗
Corresponding author. Tel.: +1 814 863 3559; fax: +1 814 865 3015. E-mail addresses: [email protected] (H. Baumgartner), [email protected] (B. Weijters).
Although methodologically oriented researchers in marketing have emphasized the importance of controlling measurement error, particularly systematic measurement error, it is our impression that common method biases have been taken much more seriously in other fields (e.g., management) and that empirical researchers in marketing (apart from a ritualistic concern with such things as nonresponse bias or social desirability bias) often do not seem to pay much attention to method artifacts. Since the post hoc procedures described in the literature are not always effective and usually cumbersome to implement, empirical researchers’ reticence to follow the advice offered by methodologists may be somewhat understandable. The approach proposed by M&P, which consists of critically evaluating, during the design of the study, a checklist of factors that might lead to common method bias (organized around the researcher’s goals to increase respondents’ ability and motivation to respond accurately, and increase the difficulty of satisficing) should go a long way toward preventing the occurrence of method bias. In this note, we would like take the opportunity to draw researchers’ attention to several other issues related to method bias that we believe have been underemphasized in the literature. The comment is written in the spirit of increasing researchers’ concern with common method bias, which will hopefully contribute to more valid research findings.
0022-4359/$ – see front matter © 2012 New York University. Published by Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jretai.2012.10.003
564
H. Baumgartner, B. Weijters / Journal of Retailing 88 (4, 2012) 563–566
The need to formulate a multi-motive model of survey response Building on earlier work by Krosnick (1991) and others, M&P view the respondent’s task as a tradeoff between optimizing and satisficing. Furthermore, as stated by M&P (p. 6), “when respondents are satisficing rather than optimizing they will be more likely to respond stylistically, and their responses will be more susceptible to method bias.” Since satisficing occurs when respondents lack the ability or motivation to respond accurately, which depends on both personal characteristics (e.g., respondents’ capabilities) and situational factors (e.g., the difficulty of the response), the researcher’s task is to select respondents who are sufficiently able and motivated to provide accurate responses and to design and administer the survey in such a way as to stimulate and sustain ability and motivation. Based on M&P’s article, readers might be led to expect that inaccurate responding only occurs when ability and/or motivation are low, which may be true when only accuracy goals are considered. However, once it is acknowledged that there are motives other than accuracy in a survey context, it is apparent that high motivation and high ability can also encourage inaccurate responding. Researchers in other areas have found it useful to consider a variety of motives that may guide behavior in a given situation. Such a multi-motive view may also be beneficial in a survey context. For example, in decision making, Bettman, Luce, and Payne (1998) distinguish the goals of maximizing decision accuracy, minimizing decision effort, minimizing the experience of negative emotions during decision making, and maximizing the ease of justification of a decision. In the attitude area, researchers have distinguished between multiple goals which may influence attitude formation, including accuracy motivation, defense motivation, and impression motivation (Chaiken, Giner-Sorolla, and Chen 1996). Such an expanded list of motives should also prove beneficial in the context of surveys. A well-known example from survey research in which motives other than accuracy are salient is the phenomenon of socially desirable responding (SDR). SDR means that respondents provide answers that make them look good (Steenkamp, De Jong, and Baumgartner 2010). Respondents high in SDR may be strongly motivated to manage a certain impression, and they may also be able to do so, but high motivation and ability do not lead to valid responses. In M&P’s terms, one might argue that respondents’ ability and motivation to provide accurate responses are low in this case, and M&P discuss issues related to social desirability under factors that decrease the motivation to respond accurately. However, research shows that SDR is not always a conscious and deliberate impression-management tactic and instead may reflect honestly held but exaggerated selfperceptions. In this situation, it is not very clear how meaningful it is to equate respondents who are high in ability and high in motivation with respondents who respond accurately. Although less research has dealt with goals such as ease of justification or defense motivation in a survey context, it is easy to see how a desire to justify certain responses or to uphold deeply held beliefs may lead to biased responses even though ability and
motivation are high. It would be useful if future research investigated a broader set of motives that guide respondents when they provide answers to survey questions, because research in other areas (e.g., in the attitude area) has shown quite convincingly that the degree of effort expended on a task is orthogonal to the type of motivation that is salient during the response process (which implies that high ability and high motivation should not be equated with accuracy). The need to consider determinants of method bias other than satisficing Although method bias frequently occurs under peripheral processing conditions, when respondents are trying to satisfice, it has to be acknowledged that satisficing is not the only antecedent of method bias. M&P are careful to focus their discussion of satisficing and method bias on the first three stages of Tourangeau, Rips, and Rasinski’s (2000) model of the survey response process (comprehension, retrieval, and judgment), where the effects of satisficing may be expected to be most pronounced. However, the response stage, in which respondents have to map the judgment onto the response categories provided in order to answer the question, is also a source of numerous method biases, which may not be due to satisficing. The issue of socially desirable responding, which may be viewed as an instance of response editing, has already been mentioned. Another example is a situation in which different respondents have different response mapping functions. To illustrate, consider a task where satisfaction ratings are collected on a 10-point scale. As pointed out by M&P, labeling all response options may reduce response mapping problems, but even then some respondents may think that a score of 7 is pretty good, whereas others may believe that a rating of 8 is what one normally should expect. Although this difference in the use of the response scale would result in method bias, no lack of ability or motivation and no attempt to satisfice are involved (see Weijters, Cabooter, and Schillewaert 2010, esp. Figure 3, for recommendations on which response scale formats to use in different contexts). Cross-cultural research is one area where the problem of noncomparability of responses is acute, because mapping functions have been shown to vary substantially across cultures, in part due to differences in cultural values. For example, it is well-known that collectivist cultures are more prone to acquiescent responding and are more likely to use the midpoint of the scale due to their greater emphasis on conformity and modesty, whereas individualistic cultures have a greater preference for the extremes of the rating scale because decisiveness is a virtue (see Baumgartner and Weijters forthcoming, for an extended discussion of response biases in cross-cultural measurement). There are also more subtle differences in cross-cultural survey response behavior that arise from the fact that the response categories used in rating scales have to be translated (e.g., ‘strongly agree’ in English versus ‘tout à fait d’accord’ in French) and may not be equivalent across cultures. In a recent program of research, Weijters, Geuens, and Baumgartner (2012) investigated whether differences in the intensity (response categories are endorsed less frequently when their labels are more intense) or familiarity
H. Baumgartner, B. Weijters / Journal of Retailing 88 (4, 2012) 563–566
(response categories are endorsed more frequently when their labels are more commonly used in day-to-day language) of endpoint category labels (like strongly agree or completely agree) were more likely to result in differences in response behavior in different languages. Their results indicate that label familiarity is a more important driver of differential use of the extreme response categories. The need to evaluate the relative importance of different sources of method bias M&P discuss a total of 26 contributors to method bias and the list could probably be expanded. Although Tables 2 to 4 can serve as a useful checklist for researchers designing a survey, and the potential remedies proposed provide some guidance on how to prevent method bias, it is likely that researchers will find it challenging to implement all the recommendations in practice. It is also likely that the various sources of method bias will vary in their seriousness and that the proposed remedies will differ in their effectiveness. Researchers would greatly benefit from more explicit counsel on which factors might be most detrimental in different situations and which correction procedures are most likely to increase the validity of the findings. In particular, it would be very useful if there were a more tightly organized structure of the variables that encourage method bias. The conditions identified by M&P as causing method bias are all related to some of the following categories: characteristics of the respondent (e.g., verbal ability, various personality dimensions); factors associated with the instructions, individual questions (e.g., complexity, ambiguity), and the response scales used; more general characteristics of the survey (e.g., grouping of the items, length of the survey, reactions toward the sponsor of the survey); and mode of survey administration (e.g., auditory vs. written presentation of items, presence of an interviewer). Future research should try to develop a model that catalogs the various influences on stylistic responding in a more systematic fashion and, based on this framework, derive recommendations for improving survey research practices. The need to examine important sources of method bias in greater depth Once important contributors to method bias have been identified, it will be necessary to discuss the issues engendered by the bias in question in some depth, as illustrated in the recent article by Tourangeau and Yan (2007) on sensitive questions. Researchers would probably prefer simple procedural solutions to common problems, but unfortunately this is frequently impossible. Our recent research on reversed item method bias is a case in point (see Weijters and Baumgartner 2012). The use of reversed items in surveys is sometimes recommended to inhibit stylistic responding and control for acquiescence, but reversed items also lead to many problems (e.g., low item reliabilities, complex factor structures). It is not meaningful to answer the question of whether or not to use reversed items with a simple yes or no. One important consideration is how an item is reversed – by negating a regular (non-reversed) item or by using
565
an item whose meaning is the polar opposite of a regular item. Based on an extensive literature review, Weijters and Baumgartner suggest that negated items should be used sparingly, if at all, especially when a very similar or identical regular item is reversed by adding the particle ‘not’, because negations are confusing to people. Polar opposite reversals are recommended, but only when they satisfy certain conditions (e.g., respondents should be explicitly or implicitly made aware of the presence of reversed items; antonyms have to be contradictory, rather than contrary; even contradictories may be problematic with East Asian respondents; polar opposite reversals are most useful when they encourage broad coverage of the target construct’s domain of content; regular and reversed items should be dispersed randomly throughout the questionnaire; fully labeled response scales should be used because respondents do not always interpret scale categories correctly; etc.). As can be seen from this long but incomplete list of caveats, although we definitely recommend the use of reversed items in questionnaires, there is an art to writing good reversals, and it appears that many researchers have not mastered this art (and may not be aware of all of the science undergirding the art). The need to engage in more extensive pretesting before administering a survey Being aware of the multitude of conditions that can lead to method bias and thinking about potential remedies is a valuable first step in collecting more valid data. However, it is unlikely that this will be sufficient for averting most problems. Although experts may be able to detect common shortcomings in the design and administration of surveys, there is no substitute for indepth pretesting. It is our impression that academic researchers do relatively little pretesting before they launch a survey, and even if they conduct a pretest, the pretests are not as informative as they could be. For example, a researcher may conduct a prestudy with a small sample of respondents in order to investigate whether the items intended to measure a certain construct yield internally consistent responses with adequate reliabilities. Although such a prestudy may identify some problems, it probably does not provide much information about why a problem occurs, and it is unlikely to identify all major problems. Furthermore, since method bias often inflates correlations, high internal consistency may signal a problem, instead of reassuring the researcher that none exists. Cognitive interviewing is a particularly valuable technique for improving questionnaires. Respondents from the target population are asked to verbalize their reactions to the survey as they answer questions, and the researcher can assess how participants comprehend both the literal and pragmatic meaning of the question, whether and how they retrieve information from memory before providing an answer, what strategies they use to form a judgment and how responses to previous items influence their judgments, and how the answer is formatted to fit the response scale. One disadvantage of cognitive interviewing is that responding to questions when one has to verbalize one’s thought processes may not correspond to the way in which respondents normally complete a survey. If this is a
566
H. Baumgartner, B. Weijters / Journal of Retailing 88 (4, 2012) 563–566
concern, less intrusive techniques such as eye movement recording may be used to gain insight into people’s response behavior. We acknowledge that these procedures are effortful and that researchers may be reluctant to adopt them as a routine component of the survey process. Ideally, methodologically oriented researchers would conduct the necessary studies and provide practically relevant recommendations about how surveys can be made less prone to method biases.
comment, including the need to formulate a multi-motive model of survey response, the need to consider determinants of method bias other than satisficing, the need to evaluate the relative importance of different sources of method bias, the need to examine important sources of method bias in greater depth, the need to engage in more extensive pretesting before administering a survey, and the need to encourage systematic processing in the age of internet panels.
The need to encourage systematic processing in the age of internet panels
References
Some of the recommendations made by M&P involve telling respondents why the survey and their personal opinions are important, stressing the value of accuracy, and encouraging participants to pay attention to all aspects of the survey. Unfortunately, the empirical evidence about how attentive respondents are to instructions and questions is sobering. For example, in Meade and Craig’s (2012) study about careless responding, only 90 percent of respondents disagreed or strongly disagreed with the statements that they slept less than one hour per night or did not understand a word of English; for statements such as ‘I have never brushed my teeth’, ‘All my friends are aliens’, and ‘All my friends say I would make a great poodle’, the percentages were even lower, although one could argue that asking a strange question may entice people to provide a strange answer. There is also solid evidence that respondents often do not read instructions. For example, in their work on Instructional Manipulation Checks, Oppenheimer, Meyvis, and Davidenko (2009) found that between 14 and 46 percent of respondents ignored the instruction not to respond to a question about participation in sports activities and to instead click on the title at the top of the screen in order to proceed to the next screen. The problems of careless responding are probably exacerbated when respondents complete surveys on the internet, often for rather small payments. In addition, there are problems of non-representative sampling, self-selection, and other errors in surveys besides measurement error that may pose serious threats to the validity of surveys. Conclusion In conclusion, we hope that the guidelines provided by M&P on how to deal with the problem of common method bias in survey research will get the attention they deserve. Substantive researchers in retail and marketing need to use more procedural remedies that can be implemented before the data are collected in order to prevent the damaging effects of method bias. Methodological researchers can address a number of remaining gaps in the survey literature, some of which we highlighted in this
Baumgartner, Hans and Bert Weijters (forthcoming), “Response Biases in Crosscultural Measurement,” in Handbook of Culture and Consumer Psychology, Sharon Ng and Angela Y. Lee, eds. Oxford University Press. Bettman, James R., Mary Frances Luce and John W. Payne (1998), “Constructive Consumer Choice Processes,” Journal of Consumer Research, 25 (December), 187–217. Chaiken, Shelly, Roger Giner-Sorolla and Serena Chen (1996), “Beyond Accuracy: Defense and Impression Motives in Heuristic and Systematic Information Processing,” in The Psychology of Action: Linking Cognition and Motivation to Behavior, Peter M. Gollwitzer and John A. Bargh, eds. New York: Guilford, 553–578. Fisher, Ronald A. (1938), “Presidential Address,” Sankhy¯a: The Indian Journal of Statistics, 4 (1), 14–7. Krosnick, Jon A. (1991), “Response Strategies for Coping with the Cognitive Demands of Measures in Surveys,” Applied Cognitive Psychology, 5 (3), 213–36. Meade, Adam W. and Bartholomew S. Craig (2012), “Identifying Careless Responses in Survey Data,” Psychological Methods, 17 (September), 437–55. Oppenheimer, Daniel A., Tom Meyvis and Nicolas Davidenko (2009), “Instructional Manipulation Checks: Detecting Satisficing to Increase Statistical Power,” Journal of Experimental Social Psychology, 45, 867–72. Podsakoff, Philip M., Scott B. MacKenzie, Jeong-Yeon Lee and Nathan P. Podsakoff (2003), “Common Method Biases in Behavioral Research: A Critical Review of the Literature and Recommended Remedies,” Journal of Applied Psychology, 88 (5), 879–903. Podsakoff, Philip M., Scott B. MacKenzie and Nathan P. Podsakoff (2012), “Sources of Method Bias in Social Science Research and Recommendations on How to Control It,” Annual Review of Psychology, 63, 539–69. Steenkamp, Jan-Benedict E.M., Martijn De Jong and Hans Baumgartner (2010), “Socially Desirable Response Tendencies in Survey Research,” Journal of Marketing Research, 47 (April), 199–214. Tourangeau, Roger, Lance J. Rips and Kenneth Rasinski (2000), The Psychology of Survey Response, Cambridge, UK: Cambridge University Press. Tourangeau, Roger and Ting Yan (2007), “Sensitive Questions in Surveys,” Psychological Bulletin, 133 (5), 859–83. Weijters, Bert and Hans Baumgartner (2012), “Misresponse to Reversed and Negated Items in Surveys: A Review,” Journal of Marketing Research, 49 (5), 737–47. Weijters, Bert, Elke Cabooter and Niels Schillewaert (2010), “The Effect of Rating Scale Format on Response Styles: The Number of Response Categories and Response Category Labels,” International Journal of Research in Marketing, 27, 236–47. Weijters, Bert, Maggie Geuens and Hans Baumgartner (2012), “The Effect of Response Category Label Familiarity on Item Response to Likert Scales,” Unpublished working paper, Ghent University.