- Email: [email protected]
A tale of two cultures: Revisiting journal editors' views of replication research
Journal of Business Research 66 (2013) 1457–1459
Contents lists available at SciVerse ScienceDirect
Journal of Business Research
A tale of two cultures: Revisiting journal editors' views of replication research Richard W. Easley a,⁎, Charles S. Madden a, 1, Van Gray b, 2 a b
Baylor University, Department of Waco, One Bear Place, Waco 76798-8007, TX, United States Hankamer School of Business, One Bear Place, Baylor University, 98007, Waco, TX 76798, United States
a r t i c l e
i n f o
Article history: Received 1 January 2011 Received in revised form 1 July 2011 Accepted 1 February 2012 Available online 31 May 2012 Keywords: Replication Variety Meta-analysis Business Research
a b s t r a c t While replications are an important and integral component of the scientific method, and are common in the natural science literature, the usefulness of replication research is not as widely practiced in the social sciences. The authors previously (1995, 2000) investigated the prevalence of replication research by soliciting journal editors' perceptions of their disciplines' attitudes toward such work. Originally, two studies questioned editors — first in the natural and social sciences and, later, editors in advertising, communications, and marketing journals. Findings included that natural science editors have generally endorsed replication as a necessary part of research, while social science editors have been less than enthusiastic about its adoption. Marketing communications and advertising editors responded consistently with that of most other social science editors. This paper revisits journal editors' views of replication research and updates the state of research in the social sciences. © 2012 Elsevier Inc. All rights reserved.
1. Introduction Madden, Easley, and Dunn (1995) investigate social science and natural science journal editors' perceptions of their disciplines' attitudes toward replication research. In the years between that investigation and today's use of replication research in the social sciences, this area of research is an area of some debate, and the usefulness of replication research still draws concerns and differences of opinion. Investigators have always recognized replication as an important aspect of scientific method (Simon, 1978). Certainly, scholars understand replication and the important role replication plays in the scientific process of verification, and most scholars acknowledge the need for a tradition of verification in their discipline (e.g., Berthon, Pitt, Ewing, & Carr, 2002). However, somewhat paradoxically, strict replications rarely appear in business, marketing, and management research journals. For example, the broad study of replications in management research by Hubbard and Armstrong (1996) found these studies constitute less than 10% of the articles in the fields of accounting, economics, and finance and less than 5% in the fields of management and marketing. Still, interest in replication in the social sciences remains stable and significant, and an awareness of the importance of replication as a contributor to the knowledge base is strong. (e.g., Kane, 1984; Rosenthal & Rosnow, 1984; Collins, 1985; Ehrenberg, 1990).
⁎ Corresponding author. Tel./fax: + 1 254 710 6178. E-mail addresses: [email protected] (R.W. Easley), [email protected] (C.S. Madden), [email protected] (V. Gray). 1 Tel.: + 1 254 710 6168. 2 Tel.: + 1 254 710 8468. 0148-2963/$ – see front matter © 2012 Elsevier Inc. All rights reserved. doi:10.1016/j.jbusres.2012.05.013
In 2000, a special issue of the Journal of Business Research (Easley and Madden, 2000) – the first within the discipline of marketing – covered only the topic of replication. Since this earlier work, conditions and technologies for all disciplines have changed; in some cases, access to information has undergone dramatic change. The number of new and subspecialized journals, electronic journals and new outlets for investigative productivity, including podcasts and other electronic forms of research productivity and communication, is growing. Modern society seems comfortable with blogs, tweets, and their lack of review, professional scrutiny and scientific repeatability. More importantly, recent publications have indicated that changes in editorial polices of leading marketing and management science journals are encouraging and are more receptive in facilitating replication studies (Evanschitzky, Baumgart, Hubbard, & Armstrong, 2007). Replication research in the social sciences may be valued now even more than before. Accordingly, the authors reasoned that revisiting this area to see if any changes have occurred would be appropriate. This paper provides a brief look back followed by a current update, via an additional survey of journal editor's opinions, on the state of replication research in the natural and social science disciplines. 2. Background In the original article referenced previously, a central theme or question emerges: “If replication is so firmly established as necessary for good science, why is there such a disparity between the actual state of replication in our related disciplines and the ideal state that so many researchers espouse?” (Reid, Soley, & Wimmers, 1981). Additionally, other key questions addressed in the original work included but were not limited to the following issues. Are researchers in the advertising, communications, and marketing disciplines out of
1458
R.W. Easley et al. / Journal of Business Research 66 (2013) 1457–1459
the scientific mainstream, as numerous authors suggest, because researchers have never fully honored the tradition of replication? Do strongly held values, including a rich tradition of replication, guide other sciences in a more orderly pursuit of knowledge? Have the advertising, communications, and marketing disciplines found strategies to foster a rich replication tradition? What are reasonable expectations for replication?
As part of the original investigation, editors responded to additional questions specific to marketing. In this discipline-specific component, the respondents agreed that replication studies do not often appear in marketing, advertising and related disciplines unless they are in controversial areas. Additionally, while editors agreed about the lack of rewards for conducting replications and the difficulty of publishing replications, most respondents believed replication research should play a much larger role.
3. The original procedure 5. The revised study The earlier paper included two survey studies conducted to find answers to such questions (Madden et al., 1995). Originally, investigators chose 205 journals for the sample frame. They selected the journals randomly from the social and natural science holdings of a major university library and divided them equally between the social and natural sciences. Additionally, they explained the rationale for selecting journal editors as follows (Madden et al., 1995): “Journal editors were chosen as respondents because of their potential ability to provide insightful commentary about replication research and its role in science. They are obviously exposed to more papers in their respective disciplines than most other scholars. Editors act as a “channel member,” (and some would argue, gatekeeper) who assembles product assortments (academic papers) from various producers (researchers) and delivers them to appraisers (reviewers) who deem them acceptable or unacceptable for public consumption (publication). They are assumed to be leaders in their respective areas of research. Further, editors' viewpoints were assumed to reflect a mainstream disciplinary bias in their respective areas.” At that time, current editors of the journals received a two-page questionnaire. Investigators asked the editors to respond to three statements by indicating agreement or disagreement on a fourpoint scale. The questionnaire also asked them to provide explanatory comments for their responses. One hundred and seven editors responded to the survey, resulting in a response rate of 52%. The study categorizes editors' responses into a social/natural science dichotomy, and analyzed the responses by calculating a t-test to determine whether the responses differed significantly between the two groups.
The objectives of this paper are similar to those in the original study. In essence, this paper is a replication of the former replication investigation—a replication of a replication study. The central theme of why social sciences do not more widely embrace replication research, as well as the original four questions comparing replication research in the natural and social sciences, still seem timely and important. Additionally, this research highlights the following questions. What has changed? Why has change occurred? What possible problems persist in the area of replication? How has the focus changed since publication of the earlier paper? 6. Methodology For this revised study, investigators sent (electronically) a survey to 362 journal editors, across all social and natural sciences. The survey consisting of several sections of questions. Some of the questions asked editors' opinions about replication and the degree to which they agreed or disagreed with the statements, using a standard fivepoint Likert-type scale. Additionally, a series of open-ended questions provided a basis for comparison, much like those found in the original study. This second investigation achieved a response rate of 15.47% (56 editors). In the original 1995 study, before electronic surveys were widely available, the surveys went out by mail to editors and this, along with possibly some unfamiliarity with electronic surveys among some journal editor recipients, might account for the difference in response rates between the two studies. 7. Analysis
4. The original results In general, the editor comments provided significant insight about differences regarding replication research between the natural and social sciences. The results, described in the original paper, are as follows: “Within the natural sciences, replication evidently plays a significant role in some disciplines. One editor commented, “replication is rarely an issue for us – perhaps never – since we publish them.” Within the social sciences, replication seems to play a less significant role and many editors recognized that the role of replication should be enhanced. The underlying basis for the neglect of replication appears to be current research norms that encourage researchers to seek recognition by conducting original research. One editor commented that tenure is not achieved by conducting replication research. Such behavior is not likely to change unless the disciplines, norms and values for the research process also change.The natural science editors provided a very clear explanation for their ratings. They valued the rigor of the review process and the checks it provides. They viewed effective peer review and academic integrity as important mechanisms for ensuring that published research is valid and reliable. However, they felt that if studies deviate from accepted wisdom or look too good to be true, the findings can be considered suspect. The same opinions were expressed by the social science editors, but some of them felt that studies should be considered suspect prior to being replicated.”
Consistent with the previous study, investigators used a t-test to determine whether the responses differed significantly between the two groups. As depicted in the Table 1 below, only two significant differences appear between natural science and social science editors, suggesting a greater convergence between the two editors' groups' views regarding replication research during the 1995 to 2010 period. Consequently, this change seems to indicate a shift in attitudes toward replication in the social sciences, in that they are getting closer to natural sciences on “ideals for replication.” Table 2 compares the questions and their means from the marketing and management journal editors in the original paper and all editors included in this most recent survey. The table illustrates the results of conducting a pooled t-test. At a .05 significance level, the means for questions 9, 10, 12, 13, and 14 are different, demonstrating a clear difference in editor perceptions during the intervening years between the original study and the current study. For example, questions 9 (. . . more attention should be paid to replication) and 10 (. . . importance of replication . . . is given lip service only) show a statistically significant decrease in means, suggesting that various disciplines are giving more attention to replication and less lip service to the importance of replication, both indicating more emphasis on the use of replication in academic research. Similarly, Questions 12 (. . . giant leap forward . . .), 13 (. . . replication researchers . . . lack creativity) and 14 (Failure to replicate . . . does little to reduce article status) suggest the same outcome: that disciplines, as reflected by the opinion leaders (read: editors) of the
R.W. Easley et al. / Journal of Business Research 66 (2013) 1457–1459 Table 1 Group statistics by type of science.
Table 2 Comparison of the original study and current study means. Meana Std. Std. Deviation Error Mean
Question
N
1. Replication studies are usually not Natural published in my journal except in sciences controversial areas Social Sciences 2. Most reviewers of the journal require Natural new theory development sciences Social Sciences 3. Few strict replications are submitted Natural to the journal sciences Social Sciences Natural 4. Replication manuscripts sciences are frequently published in the journal Social Sciences 5. The editorial philosophy of the journal Natural is to avoid replications sciences Social Sciences 6. Something new must be in the Natural manuscript for it to be publishable sciences in the journal Social Sciences
18 3.17
.618
.146
31 2.97
1.016
.182
18 2.89
.832
.196
31 2.74
.729
.131
18 3.22
.808
.191
31 3.32
.748
.134
18 1.61
.608
.143
31 1.81
.833
.150
18 2.72b
.895
.211
31 2.03b
.948
.170
18 3.00
.970
.229
31 3.06
.854
.153
a b
1459
5-point Likert-type scale, where 5 = strongly agree. Significant at the .05 level.
disciplines are valuing replication research and replication within new research as more important to the advancement of science. 8. Conclusions from the original study and new discussion As this paper discussed earlier, one might conclude that the views and opinions of editors in the natural and social sciences on the topic of replication research are converging. The comparison in data between the original studies and the current study seems to indicate that exact conclusion. And, certainly, the social sciences show progress in the integration of replications and the need for replication. Still, the social sciences have much progress to make to be at the point of the natural sciences regarding replication. Essentially, this is a story of two cultures and the chasm that exists between them; one (the natural sciences) that views replication as a necessary and respected component in the quest for knowledge, and one (the social sciences) that views replication in a more negative context. Social science editors think of replication as an uncreative process that unfairly displaces “original” and important studies. While the two cultures may demonstrate concurrence in some of their views on the role of replication, problem areas remain in the social sciences and their perspective on the importance and necessity of replication. References Berthon, P., Pitt, L., Ewing, M., & Carr, C. (2002, December). Potential research space in MIS: a framework for envisioning and evaluating research replication, extension, and generation. INFORMS, 13(4), 416–427. Collins, H. M. (1985). Changing Order: Replication and Induction in Scientific Practice. London: Sage.
Questions
New study Old study t-pooled meana meana
1. Replication studies are usually not published in my journal except in controversial areas 2. Most reviewers of the journal require new theory development 3. Few strict replications are submitted to the journal 4. Replication manuscripts are frequently published in the journal 5. The editorial philosophy of the journal is to avoid replications 6. Something new must be in the manuscript for it to be publishable in the journal 7. My discipline devotes adequate attention to the replication of published research 8. Researchers avoid the replication process due to the difficulty of publishing these types of articles 9. There should be more attention paid to replication in the my discipline's literature 10. The importance of replication in the research process is given lip service only 11. Novelty is more prized by researchers in my discipline than “mere” replications 12. Most researchers look too much for the giant leap forward rather than the steady march of science 13. A researcher who spent his or her time attempting to replicate previously published work would be thought to be lacking creativity 14. Failure to replicate does little to reduce the status of a published article 15. Published studies in the my discipline are “suspect” until they are replicated and these results are published in literature 16. Researchers avoid replication because the rewards for doing those types of investigations are considered less than those of conducting original research 17. How often does your journal publish replications of previously published studies?
2.970
2.800
− 0.596
2.758
2.667
− 0.373
3.091
3.267
0.691
1.697
1.600
− 0.428
2.288
2.333
0.153
2.985
3.214
0.839
2.293
1.933
− 1.438
2.776
3.067
1.120
2.534
3.143
2.281b
2.586
3.143
2.133b
3.190
3.200
0.041
2.397
3.000
2.485b
2.759
3.333
2.645b
2.397
2.933
2.333b
2.052
2.000
− 0.204
3.017
3.214
0.773
1.773
1.667
− 0.515
a 5-point Likert-type scale, where 5 = strongly agree; means are combined means (i.e., natural and social sciences). b Significant at the .05 level.
Easley, R. W., & Madden, C. S. (2000). Replications and extensions in marketing and management research. Journal of Business Research, 48, 1–3. Ehrenberg, A. S. C. (1990). A Hope for the Future of Statistics: MSoD. The American Statistician, 44, 195–196. Evanschitzky, H., Baumgarth, C., Hubbard, R., & Armstrong, J. S. (2007). Replication research’s disturbing trend. Journal of Business Research, 60, 411–415. Hubbard, R., & Armstrong, J. (1996). Publication bias against null results. Psychological Reports, 80, 337–338. Kane, E. J. (1984). Why Journal Editors Should Encourage the Replication of Applied Econometric Research. Quarterly Journal of Business and Economics, 23, 3–8. Madden, C. S., Easley, R. W., & Dunn, M. G. (1995). How journal editors view replication research. Journal of Advertising, 24(4), 77–87. Reid, L., Soley, L. C., & Wimmers, R. D. (1981). Replication in advertising research: 1977, 1978, 1979. Journal of Advertising, 10(1), 3–13. Rosenthal, R., & Rosnow, R. L. (1984). Applying Hamlet’s question to the ethical conduct of research. American Psychologist, 39, 561–563. Simon, J. L. (1978). Basic Research Methods in Social Science. New York: Random House.
Contents lists available at SciVerse ScienceDirect
Journal of Business Research
A tale of two cultures: Revisiting journal editors' views of replication research Richard W. Easley a,⁎, Charles S. Madden a, 1, Van Gray b, 2 a b
Baylor University, Department of Waco, One Bear Place, Waco 76798-8007, TX, United States Hankamer School of Business, One Bear Place, Baylor University, 98007, Waco, TX 76798, United States
a r t i c l e
i n f o
Article history: Received 1 January 2011 Received in revised form 1 July 2011 Accepted 1 February 2012 Available online 31 May 2012 Keywords: Replication Variety Meta-analysis Business Research
a b s t r a c t While replications are an important and integral component of the scientific method, and are common in the natural science literature, the usefulness of replication research is not as widely practiced in the social sciences. The authors previously (1995, 2000) investigated the prevalence of replication research by soliciting journal editors' perceptions of their disciplines' attitudes toward such work. Originally, two studies questioned editors — first in the natural and social sciences and, later, editors in advertising, communications, and marketing journals. Findings included that natural science editors have generally endorsed replication as a necessary part of research, while social science editors have been less than enthusiastic about its adoption. Marketing communications and advertising editors responded consistently with that of most other social science editors. This paper revisits journal editors' views of replication research and updates the state of research in the social sciences. © 2012 Elsevier Inc. All rights reserved.
1. Introduction Madden, Easley, and Dunn (1995) investigate social science and natural science journal editors' perceptions of their disciplines' attitudes toward replication research. In the years between that investigation and today's use of replication research in the social sciences, this area of research is an area of some debate, and the usefulness of replication research still draws concerns and differences of opinion. Investigators have always recognized replication as an important aspect of scientific method (Simon, 1978). Certainly, scholars understand replication and the important role replication plays in the scientific process of verification, and most scholars acknowledge the need for a tradition of verification in their discipline (e.g., Berthon, Pitt, Ewing, & Carr, 2002). However, somewhat paradoxically, strict replications rarely appear in business, marketing, and management research journals. For example, the broad study of replications in management research by Hubbard and Armstrong (1996) found these studies constitute less than 10% of the articles in the fields of accounting, economics, and finance and less than 5% in the fields of management and marketing. Still, interest in replication in the social sciences remains stable and significant, and an awareness of the importance of replication as a contributor to the knowledge base is strong. (e.g., Kane, 1984; Rosenthal & Rosnow, 1984; Collins, 1985; Ehrenberg, 1990).
⁎ Corresponding author. Tel./fax: + 1 254 710 6178. E-mail addresses: [email protected] (R.W. Easley), [email protected] (C.S. Madden), [email protected] (V. Gray). 1 Tel.: + 1 254 710 6168. 2 Tel.: + 1 254 710 8468. 0148-2963/$ – see front matter © 2012 Elsevier Inc. All rights reserved. doi:10.1016/j.jbusres.2012.05.013
In 2000, a special issue of the Journal of Business Research (Easley and Madden, 2000) – the first within the discipline of marketing – covered only the topic of replication. Since this earlier work, conditions and technologies for all disciplines have changed; in some cases, access to information has undergone dramatic change. The number of new and subspecialized journals, electronic journals and new outlets for investigative productivity, including podcasts and other electronic forms of research productivity and communication, is growing. Modern society seems comfortable with blogs, tweets, and their lack of review, professional scrutiny and scientific repeatability. More importantly, recent publications have indicated that changes in editorial polices of leading marketing and management science journals are encouraging and are more receptive in facilitating replication studies (Evanschitzky, Baumgart, Hubbard, & Armstrong, 2007). Replication research in the social sciences may be valued now even more than before. Accordingly, the authors reasoned that revisiting this area to see if any changes have occurred would be appropriate. This paper provides a brief look back followed by a current update, via an additional survey of journal editor's opinions, on the state of replication research in the natural and social science disciplines. 2. Background In the original article referenced previously, a central theme or question emerges: “If replication is so firmly established as necessary for good science, why is there such a disparity between the actual state of replication in our related disciplines and the ideal state that so many researchers espouse?” (Reid, Soley, & Wimmers, 1981). Additionally, other key questions addressed in the original work included but were not limited to the following issues. Are researchers in the advertising, communications, and marketing disciplines out of
1458
R.W. Easley et al. / Journal of Business Research 66 (2013) 1457–1459
the scientific mainstream, as numerous authors suggest, because researchers have never fully honored the tradition of replication? Do strongly held values, including a rich tradition of replication, guide other sciences in a more orderly pursuit of knowledge? Have the advertising, communications, and marketing disciplines found strategies to foster a rich replication tradition? What are reasonable expectations for replication?
As part of the original investigation, editors responded to additional questions specific to marketing. In this discipline-specific component, the respondents agreed that replication studies do not often appear in marketing, advertising and related disciplines unless they are in controversial areas. Additionally, while editors agreed about the lack of rewards for conducting replications and the difficulty of publishing replications, most respondents believed replication research should play a much larger role.
3. The original procedure 5. The revised study The earlier paper included two survey studies conducted to find answers to such questions (Madden et al., 1995). Originally, investigators chose 205 journals for the sample frame. They selected the journals randomly from the social and natural science holdings of a major university library and divided them equally between the social and natural sciences. Additionally, they explained the rationale for selecting journal editors as follows (Madden et al., 1995): “Journal editors were chosen as respondents because of their potential ability to provide insightful commentary about replication research and its role in science. They are obviously exposed to more papers in their respective disciplines than most other scholars. Editors act as a “channel member,” (and some would argue, gatekeeper) who assembles product assortments (academic papers) from various producers (researchers) and delivers them to appraisers (reviewers) who deem them acceptable or unacceptable for public consumption (publication). They are assumed to be leaders in their respective areas of research. Further, editors' viewpoints were assumed to reflect a mainstream disciplinary bias in their respective areas.” At that time, current editors of the journals received a two-page questionnaire. Investigators asked the editors to respond to three statements by indicating agreement or disagreement on a fourpoint scale. The questionnaire also asked them to provide explanatory comments for their responses. One hundred and seven editors responded to the survey, resulting in a response rate of 52%. The study categorizes editors' responses into a social/natural science dichotomy, and analyzed the responses by calculating a t-test to determine whether the responses differed significantly between the two groups.
The objectives of this paper are similar to those in the original study. In essence, this paper is a replication of the former replication investigation—a replication of a replication study. The central theme of why social sciences do not more widely embrace replication research, as well as the original four questions comparing replication research in the natural and social sciences, still seem timely and important. Additionally, this research highlights the following questions. What has changed? Why has change occurred? What possible problems persist in the area of replication? How has the focus changed since publication of the earlier paper? 6. Methodology For this revised study, investigators sent (electronically) a survey to 362 journal editors, across all social and natural sciences. The survey consisting of several sections of questions. Some of the questions asked editors' opinions about replication and the degree to which they agreed or disagreed with the statements, using a standard fivepoint Likert-type scale. Additionally, a series of open-ended questions provided a basis for comparison, much like those found in the original study. This second investigation achieved a response rate of 15.47% (56 editors). In the original 1995 study, before electronic surveys were widely available, the surveys went out by mail to editors and this, along with possibly some unfamiliarity with electronic surveys among some journal editor recipients, might account for the difference in response rates between the two studies. 7. Analysis
4. The original results In general, the editor comments provided significant insight about differences regarding replication research between the natural and social sciences. The results, described in the original paper, are as follows: “Within the natural sciences, replication evidently plays a significant role in some disciplines. One editor commented, “replication is rarely an issue for us – perhaps never – since we publish them.” Within the social sciences, replication seems to play a less significant role and many editors recognized that the role of replication should be enhanced. The underlying basis for the neglect of replication appears to be current research norms that encourage researchers to seek recognition by conducting original research. One editor commented that tenure is not achieved by conducting replication research. Such behavior is not likely to change unless the disciplines, norms and values for the research process also change.The natural science editors provided a very clear explanation for their ratings. They valued the rigor of the review process and the checks it provides. They viewed effective peer review and academic integrity as important mechanisms for ensuring that published research is valid and reliable. However, they felt that if studies deviate from accepted wisdom or look too good to be true, the findings can be considered suspect. The same opinions were expressed by the social science editors, but some of them felt that studies should be considered suspect prior to being replicated.”
Consistent with the previous study, investigators used a t-test to determine whether the responses differed significantly between the two groups. As depicted in the Table 1 below, only two significant differences appear between natural science and social science editors, suggesting a greater convergence between the two editors' groups' views regarding replication research during the 1995 to 2010 period. Consequently, this change seems to indicate a shift in attitudes toward replication in the social sciences, in that they are getting closer to natural sciences on “ideals for replication.” Table 2 compares the questions and their means from the marketing and management journal editors in the original paper and all editors included in this most recent survey. The table illustrates the results of conducting a pooled t-test. At a .05 significance level, the means for questions 9, 10, 12, 13, and 14 are different, demonstrating a clear difference in editor perceptions during the intervening years between the original study and the current study. For example, questions 9 (. . . more attention should be paid to replication) and 10 (. . . importance of replication . . . is given lip service only) show a statistically significant decrease in means, suggesting that various disciplines are giving more attention to replication and less lip service to the importance of replication, both indicating more emphasis on the use of replication in academic research. Similarly, Questions 12 (. . . giant leap forward . . .), 13 (. . . replication researchers . . . lack creativity) and 14 (Failure to replicate . . . does little to reduce article status) suggest the same outcome: that disciplines, as reflected by the opinion leaders (read: editors) of the
R.W. Easley et al. / Journal of Business Research 66 (2013) 1457–1459 Table 1 Group statistics by type of science.
Table 2 Comparison of the original study and current study means. Meana Std. Std. Deviation Error Mean
Question
N
1. Replication studies are usually not Natural published in my journal except in sciences controversial areas Social Sciences 2. Most reviewers of the journal require Natural new theory development sciences Social Sciences 3. Few strict replications are submitted Natural to the journal sciences Social Sciences Natural 4. Replication manuscripts sciences are frequently published in the journal Social Sciences 5. The editorial philosophy of the journal Natural is to avoid replications sciences Social Sciences 6. Something new must be in the Natural manuscript for it to be publishable sciences in the journal Social Sciences
18 3.17
.618
.146
31 2.97
1.016
.182
18 2.89
.832
.196
31 2.74
.729
.131
18 3.22
.808
.191
31 3.32
.748
.134
18 1.61
.608
.143
31 1.81
.833
.150
18 2.72b
.895
.211
31 2.03b
.948
.170
18 3.00
.970
.229
31 3.06
.854
.153
a b
1459
5-point Likert-type scale, where 5 = strongly agree. Significant at the .05 level.
disciplines are valuing replication research and replication within new research as more important to the advancement of science. 8. Conclusions from the original study and new discussion As this paper discussed earlier, one might conclude that the views and opinions of editors in the natural and social sciences on the topic of replication research are converging. The comparison in data between the original studies and the current study seems to indicate that exact conclusion. And, certainly, the social sciences show progress in the integration of replications and the need for replication. Still, the social sciences have much progress to make to be at the point of the natural sciences regarding replication. Essentially, this is a story of two cultures and the chasm that exists between them; one (the natural sciences) that views replication as a necessary and respected component in the quest for knowledge, and one (the social sciences) that views replication in a more negative context. Social science editors think of replication as an uncreative process that unfairly displaces “original” and important studies. While the two cultures may demonstrate concurrence in some of their views on the role of replication, problem areas remain in the social sciences and their perspective on the importance and necessity of replication. References Berthon, P., Pitt, L., Ewing, M., & Carr, C. (2002, December). Potential research space in MIS: a framework for envisioning and evaluating research replication, extension, and generation. INFORMS, 13(4), 416–427. Collins, H. M. (1985). Changing Order: Replication and Induction in Scientific Practice. London: Sage.
Questions
New study Old study t-pooled meana meana
1. Replication studies are usually not published in my journal except in controversial areas 2. Most reviewers of the journal require new theory development 3. Few strict replications are submitted to the journal 4. Replication manuscripts are frequently published in the journal 5. The editorial philosophy of the journal is to avoid replications 6. Something new must be in the manuscript for it to be publishable in the journal 7. My discipline devotes adequate attention to the replication of published research 8. Researchers avoid the replication process due to the difficulty of publishing these types of articles 9. There should be more attention paid to replication in the my discipline's literature 10. The importance of replication in the research process is given lip service only 11. Novelty is more prized by researchers in my discipline than “mere” replications 12. Most researchers look too much for the giant leap forward rather than the steady march of science 13. A researcher who spent his or her time attempting to replicate previously published work would be thought to be lacking creativity 14. Failure to replicate does little to reduce the status of a published article 15. Published studies in the my discipline are “suspect” until they are replicated and these results are published in literature 16. Researchers avoid replication because the rewards for doing those types of investigations are considered less than those of conducting original research 17. How often does your journal publish replications of previously published studies?
2.970
2.800
− 0.596
2.758
2.667
− 0.373
3.091
3.267
0.691
1.697
1.600
− 0.428
2.288
2.333
0.153
2.985
3.214
0.839
2.293
1.933
− 1.438
2.776
3.067
1.120
2.534
3.143
2.281b
2.586
3.143
2.133b
3.190
3.200
0.041
2.397
3.000
2.485b
2.759
3.333
2.645b
2.397
2.933
2.333b
2.052
2.000
− 0.204
3.017
3.214
0.773
1.773
1.667
− 0.515
a 5-point Likert-type scale, where 5 = strongly agree; means are combined means (i.e., natural and social sciences). b Significant at the .05 level.
Easley, R. W., & Madden, C. S. (2000). Replications and extensions in marketing and management research. Journal of Business Research, 48, 1–3. Ehrenberg, A. S. C. (1990). A Hope for the Future of Statistics: MSoD. The American Statistician, 44, 195–196. Evanschitzky, H., Baumgarth, C., Hubbard, R., & Armstrong, J. S. (2007). Replication research’s disturbing trend. Journal of Business Research, 60, 411–415. Hubbard, R., & Armstrong, J. (1996). Publication bias against null results. Psychological Reports, 80, 337–338. Kane, E. J. (1984). Why Journal Editors Should Encourage the Replication of Applied Econometric Research. Quarterly Journal of Business and Economics, 23, 3–8. Madden, C. S., Easley, R. W., & Dunn, M. G. (1995). How journal editors view replication research. Journal of Advertising, 24(4), 77–87. Reid, L., Soley, L. C., & Wimmers, R. D. (1981). Replication in advertising research: 1977, 1978, 1979. Journal of Advertising, 10(1), 3–13. Rosenthal, R., & Rosnow, R. L. (1984). Applying Hamlet’s question to the ethical conduct of research. American Psychologist, 39, 561–563. Simon, J. L. (1978). Basic Research Methods in Social Science. New York: Random House.