- Email: [email protected]
Editorial: Journal peer review in integrative medicine discipline
1
Seminars in Integrative Medicine, Vol 2, No 1 (March), 2004: pp 1-4
Manuscript reviewers are described as the “gatekeepers of science” because they play an important role in dissemination of scientific information. Their responsibility is particularly great toward emergent fields such as integrative medicine that conduct investigations in a new research territory. In this editorial, we briefly describe several pitfalls that can impede reviewers’ impartial judgment and that appear particularly important in regard to manuscripts concerning complementary and alternative medicine and integrative medicine. We suggest that increasing reviewers’ awareness of the pitfalls can improve fairness in the peer review process. The ultimate responsibilities rest on the journal editors to ensure impartiality in the peer review process.
From the Jefferson Medical College of Thomas Jefferson University, Philadelphia, PA. Mohammadreza Hojat, PhD: Research Professor of Psychiatry and Human Behavior and Director of Jefferson Longitudinal Study, Center for Research in Medical Education and Health Care and Department of Psychiatry and Human Behavior, Jefferson Medical College of Thomas Jefferson University, Philadelphia, PA. Steven Rosenzweig, MD: Clinical Associate Professor of Emergency Medicine, Medical and Academic Director of JeffersonMyrna Brind Center of Integrative Medicine, Jefferson Medical College of Thomas Jefferson University, Philadelphia, PA. Address reprint requests to Mohammadreza Hojat, PhD, Jefferson Medical College of Thomas Jefferson University, Center for Research in Medical Education and Health Care, 1025 Walnut Street, Philadelphia, PA 19107. Phone: 215-955-9459; Fax: 215-923-6939; E-mail: [email protected]
© 2004 Elsevier Inc. All rights reserved. 1543-1150/04/0201-0001$30.00/0 10.1016/j.sigm.2004.05.005
EDITORIAL: JOURNAL PEER REVIEW IN INTEGRATIVE MEDICINE DISCIPLINE MOHAMMADREZA HOJAT
D
AND
STEVEN ROSENZWEIG
issemination of scientific information through professional journals (and assignment of funding priority by funding organizations) is determined to a large extent by a peer review process, defined as the expert assessment of the submitted materials.1 Because of the reviewers’ important role in publication decisions, they have been aptly described as the “gatekeepers of science.”2,3 Peer review outcomes influence the career of individual authors (e.g., academic promotion, acquisition of research grants, etc), the direction of future scientific research in a particular discipline, and the mainstream acceptance of innovative therapies. It is a clear imperative that fairness and impartiality of journal review must be safeguarded. Nevertheless, potential sources of bias have been identified in the peer review process. These pitfalls may be even more prominent in the review of manuscripts related to complementary and alternative medicine (CAM) and integrative medicine. Several attempts have been made to address issues involved in the peer review process. In particular, there have been four International Congresses on Peer Review in Biomedical Publications in 1989, 1993, 1997, and 2001, organized by the Journal of the American Medical Association (JAMA) and the British Medical Journal Publishing Group. Selected presentations in these International Congresses addressing a broad spectrum
2
HOJAT AND ROSENZWEIG
of issues in scientific publication have been published in four thematic issuesof JAMA published after each congress (JAMA, March 6, 1990; 263(10):1317-1441; JAMA, July 13, 1994; 272(2):279-174; and JAMA, July 15, 1998; 280(3):203-306; and JAMA, June 5, 2002; 287(21):2759-2871). A few books are also devoted to publication issues and editorial peer review in medicine.4-6
BIAS
S
IN
PEER REVIEW
everal pitfalls can contribute to the outcomes of the peer review process. Details of these pitfalls have been described elsewhere.7,8 In this article, we briefly highlight those that are particularly relevant to CAM and integrative medicine. One pitfall is a “confirmatory bias” defined as a tendency of some reviewers to accept outcomes that agree with commonly accepted theories and to reject those that do not.9 This bias seems to work against innovative studies10 that are needed in newly developing disciplines. Studies in CAM are especially at risk of encountering confirmatory bias when submitted to conventional medical journals because such studies often challenge the accepted conventional notions. For example, positive studies of certain homeopathic (infinitely diluted) medicines seemingly defy basic principles of physical chemistry.11 Acupuncture theory assumes the existence of vital energy (Qi) and a subtle energy anatomy of the body (meridians); both of these concepts are held highly suspect by biomedicine. CAM studies may suggest a mind– body causality that is inconsistent with mainstream assumptions.12 Confirmatory bias may extend to study methodology as well: proof by randomized, placebo controlled clinical trial is itself a commonly accepted theory. However, there is no adequate placebo for certain CAM interventions. For example, European mistletoe injections in cancer patients induce a tell-tale local inflammation. Also, subjects may simply be unwilling to be randomized between two, dramatically different, active treatments. This is the case with a current, NIH funded study of pancreatic cancer patients who receive either standard chemotherapy or a regimen of pancreatic enzymes, micronutrient supplements, and diet therapy: Investigators were forced to abandon a randomized design.
Another bias is the “file drawer” defined as the reviewer’s tendency to recommend studies that report positive results and reject those with negative findings.13 The studies in which the hypothesis of no relationship (null hypothesis) is accepted have a much lower chance of being recommended for publication.14,15 Consideration of this bias is important in new disciplines, because it increases the probability of “false positive” conclusions. This may lead to the premature acceptance and application of either a conventional therapy or an unconventional therapy. A “reverse” file drawer bias may very well be a problem among reviewers of conventional medical journals, where the publication of negative studies may further a scientific agenda of disproving CAM. Some raised this concern when JAMA chose to publish a child’s 4th grade science project as a refutation of therapeutic touch.16 In a sense, favoring the publication of a negative CAM study may be considered a form of confirmatory bias. Another bias is called “the Matthew effect”17 defined as the reviewers’ tendency to recommend manuscripts submitted by famous investigators from prestigious institutions regardless of the manuscripts’ scientific and technical merit.18 This bias decreases the publication chance of new investigators who are not affiliated with prestigious research centers. The Matthew effect may be a concern when CAM studies are conducted by new investigators at nonallopathic institutions, such as colleges of chiropractic, naturopathy, or Oriental Medicine. A final issue is bias due to the reviewer’s theoretical persuasion and ideological orientation. The potential for this pitfall in CAM was explored in an unusual study by Resch and colleagues.19 They prepared two near-identical manuscript versions of a fictitious positive study. Version A demonstrated the efficacy of a conventional therapy (pharmaceutical agent), while version B demonstrated efficacy of a CAM therapy (homeopathic remedy) in treatment of obesity. Journal reviewers, randomly assigned to receive either version, were more likely to favorably review the conventional treatment.
SUGGESTIONS
S
TO
REDUCE BIAS
everal approaches have been used to alleviate the effects of some of the reviewers’ biases. These approaches include the “single-blind” re-
EDITORIAL
view, in which the reviewer’s identity is concealed from the author(s), and the “doubleblind” review, in which the authors’ identity and affiliation are concealed from the reviewer’s, and the reviewer’s identity is concealed from the author(s). These approaches cannot completely conceal the authors’ or the affiliated institutions’ identities.20 Studies have shown that neither concealing the authors’ identities nor requiring the reviewers to sign their reports had any measurable effect on the review quality.21 In contrast, McNutt and colleagues22 found that blinding improved the quality of the reviews. Blind review (double- or singleblind) may help to mitigate the Matthew effect, but it is extremely difficult to resolve confirmation bias, the file drawer problem, and theoretical persuasion issues. Some journals ask authors to prepare their manuscript by starting the introduction, methods, results, and conclusions on new pages, so that different reviewers can assess different sections of the manuscript independent of other sections. In this case, emphasis is placed on the assessment of methodology independent of the results and their interpretations.23 Of course, reviewers cannot be blinded to the use of CAM versus conventional medicine in a particular study. Editors must therefore be particularly careful to clearly define review criteria and to examine reviews for evidence of confirmatory and ideological biases. Human services professionals are committed to comply with a set of professional ethical codes, and their lack of awareness of the codes of conduct cannot be a defense against charges of misconduct. For example, ethical code number 9.06 developed by the American Psychological Association (APA) specifies that psychologists must take into account the purpose of assessments and also be aware of various pitfalls that might affect their judgments and reduce the accuracy of their assessments.24 Similarly, we propose that, for the sake of impartiality and fairness in journal peer review process, it would be desirable to develop a set of ethical guidelines for reviewers. The pitfalls in the peer review process that may influence the reviewers’ judgments can be described and approaches to avoid bias can be suggested in the guidelines. Journal reviewers, as well as editors, should be required to comply with the reviewers’
3
ethical guidelines to safeguard impartiality in peer review publication.
CONCLUSION
M
anuscript review is a time-demanding task that brings neither fame nor fortune to journal reviewers who volunteer their time and make sincere efforts to improve dissemination of scientific information.25 Qualified reviewers are assets to the journals, but the pool of qualified reviewers is quite small.26 Biases in the peer review process are rarely publicized, but evidence suggests that they do exist.27 While we are convinced that the peer review system is working reasonably well, we strongly believe that the system can and must be improved and strengthened. Although editors have discretion in editorial decision making, they are often guided by the reviewers’ comments. Because editors have the ultimate responsibilities in the outcomes of peer review process, they should be as vigilant as possible to assure that the reviewers’ assessments of the submitted manuscripts remain as impartial as possible. The power of a reviewer is evident in the editor’s final publication decision. It has been reported that papers receiving a single negative review have a low probability of being accepted.23 We believe that the key to generating proper changes lies mostly within the journal editors. They should make sure that the new reviewers are aware of the pitfalls of peer review process, as well as the journal’s goal, guidelines and expectations. Awareness of the issues is half of the solution. The editors and reviewers, as well as journal readers and manuscript contributors, should participate in assessing the peer review process to assure its integrity and fairness. This is particularly crucial for new journals in new multidisciplinary areas such as integrative medicine, which ventures into a relatively new territory of scientific inquiry and may be particularly vulnerable to reviewer bias. Improvement of the peer review process requires input from journal readers, manuscript contributors, as well as reviewers, and more importantly, the journal editors. Editors of peer review journals are in a position to watch over the gatekeepers of science, and thus can ultimately influence the flow of scientific information.
4
HOJAT AND ROSENZWEIG
REFERENCES 1. Bailar JC, Patterson K: Journal peer review: The need for a research agenda. N Engl J Med 312:654-657, 1985 2. Crane D: The gatekeepers of science: Some factors affecting the selection of articles for scientific journals. Am Sociol 32:195-201, 1967 3. DeGrazia A: The scientific reception system and Dr. Velikovsky. Am Behav Sci 7:38-56, 1963 4. Lock S: A Difficult Balance: Editorial Peer Review in Medicine. Philadelphia, PA ISI Press, 1985 5. Lock S: The Future of Medical Journals. Plymouth, England, British Medical Journal, 1992 6. Jefferson T, Godlee F: Peer Review in Health Care. London, England, British Journal Publishing Group, 1999 7. Hojat M, Gonnella JS, Caelleigh AS: Impartial judgment by the “gatekeepers” of science: Fallibility and accountability in the peer review process. Adv Health Sci Educ 8:75-96, 2003 8. Caelleigh AS, Hojat M, Steinecke A, Gonnella JS: Effects of reviewers’ gender on assessments of a gender-related standardized manuscript. Teach Learn Med 15:163-167, 2003 9. Mahoney M: Publication prejudices: An experimental study of confirmatory bias in peer review system. Cogn Ther Res 1:161-175, 1977 10. Olson CM: Peer review of biomedical literature. Am J Emerg Med 8:356-358, 1990 11. Linde K, Clausius N, Ramirez G, et al: Are the clinical effects of homeopathy placebo effects? A meta-analysis of placebo-controlled trails. Lancet 350(9081):834-843, 1997 12. Relman A, Angell M: Resolved: Psychosocial interventions can improve clinical outcomes in organic disease: Rebuttals. Psychosom Med 64:564-567, 2002 13. Rosenthal R: The “file drawer problem” and tolerance of null results. Psychol Bull 86:638-541, 1979 14. Greenwald AG: Consequences of prejudice against the null hypothesis. Psychol Bull 82:1-20, 1975
15. Dickersin K: The existence of publication bias and risk factors for its occurrence. JAMA 263:1385-1389, 1990 16. Carpenter J, Hagemaster J, Joiner B: An even closer look at therapeutic touch. JAMA 280:1905 & 1908, 1998 17. Merton RK: The Matthew effect in science. Science 159:56-63, 1968 18. Peters DP, Ceci SJ: Peer review practices of psychological journals: The fate of published articles, submitted again. Behav Brain Sci 5:187-195, 1982 19. Resch KI, Ernst E, Garrow J: A randomized controlled study of reviewer bias against an unconventional therapy. J R Soc Med 93:164-167, 2000 20. Fisher M, Friedman SB, Strauss B: The effects of blinding on acceptance of research papers by peer review. JAMA 272: 143-146, 1994 21. Godlee F, Gale C, Martyn CN: Effect on the quality of peer review of blinding reviewers and asking them to sign their names. JAMA 280:237-240, 1998 22. McNutt RA, Evans AT, Fletcher RH, et al: The effects of blinding on the quality of peer review. JAMA 263:1371-1376, 1990 23. Armstrong SJ: We need to rethink the editorial role of peer reviewers. Chron Higher Educ 43(9):B3, October 1996 24. American Psychological Association (APA): Ethical Principles of Psychologists and Code of Conduct. Washington, DC, APA, 2002 25. Colman AM: Editorial role in author-referee disagreements. Bull Br Psychol Soc 32:390-391, 1979 26. Bloom FE: The importance of reviewers. Science 283: 789, 1999 27. Altman DG, Chalmers I, Herxheimer A: Is there a case for an international medical scientific press council? JAMA 272:166-167, 1994
Seminars in Integrative Medicine, Vol 2, No 1 (March), 2004: pp 1-4
Manuscript reviewers are described as the “gatekeepers of science” because they play an important role in dissemination of scientific information. Their responsibility is particularly great toward emergent fields such as integrative medicine that conduct investigations in a new research territory. In this editorial, we briefly describe several pitfalls that can impede reviewers’ impartial judgment and that appear particularly important in regard to manuscripts concerning complementary and alternative medicine and integrative medicine. We suggest that increasing reviewers’ awareness of the pitfalls can improve fairness in the peer review process. The ultimate responsibilities rest on the journal editors to ensure impartiality in the peer review process.
From the Jefferson Medical College of Thomas Jefferson University, Philadelphia, PA. Mohammadreza Hojat, PhD: Research Professor of Psychiatry and Human Behavior and Director of Jefferson Longitudinal Study, Center for Research in Medical Education and Health Care and Department of Psychiatry and Human Behavior, Jefferson Medical College of Thomas Jefferson University, Philadelphia, PA. Steven Rosenzweig, MD: Clinical Associate Professor of Emergency Medicine, Medical and Academic Director of JeffersonMyrna Brind Center of Integrative Medicine, Jefferson Medical College of Thomas Jefferson University, Philadelphia, PA. Address reprint requests to Mohammadreza Hojat, PhD, Jefferson Medical College of Thomas Jefferson University, Center for Research in Medical Education and Health Care, 1025 Walnut Street, Philadelphia, PA 19107. Phone: 215-955-9459; Fax: 215-923-6939; E-mail: [email protected]
© 2004 Elsevier Inc. All rights reserved. 1543-1150/04/0201-0001$30.00/0 10.1016/j.sigm.2004.05.005
EDITORIAL: JOURNAL PEER REVIEW IN INTEGRATIVE MEDICINE DISCIPLINE MOHAMMADREZA HOJAT
D
AND
STEVEN ROSENZWEIG
issemination of scientific information through professional journals (and assignment of funding priority by funding organizations) is determined to a large extent by a peer review process, defined as the expert assessment of the submitted materials.1 Because of the reviewers’ important role in publication decisions, they have been aptly described as the “gatekeepers of science.”2,3 Peer review outcomes influence the career of individual authors (e.g., academic promotion, acquisition of research grants, etc), the direction of future scientific research in a particular discipline, and the mainstream acceptance of innovative therapies. It is a clear imperative that fairness and impartiality of journal review must be safeguarded. Nevertheless, potential sources of bias have been identified in the peer review process. These pitfalls may be even more prominent in the review of manuscripts related to complementary and alternative medicine (CAM) and integrative medicine. Several attempts have been made to address issues involved in the peer review process. In particular, there have been four International Congresses on Peer Review in Biomedical Publications in 1989, 1993, 1997, and 2001, organized by the Journal of the American Medical Association (JAMA) and the British Medical Journal Publishing Group. Selected presentations in these International Congresses addressing a broad spectrum
2
HOJAT AND ROSENZWEIG
of issues in scientific publication have been published in four thematic issuesof JAMA published after each congress (JAMA, March 6, 1990; 263(10):1317-1441; JAMA, July 13, 1994; 272(2):279-174; and JAMA, July 15, 1998; 280(3):203-306; and JAMA, June 5, 2002; 287(21):2759-2871). A few books are also devoted to publication issues and editorial peer review in medicine.4-6
BIAS
S
IN
PEER REVIEW
everal pitfalls can contribute to the outcomes of the peer review process. Details of these pitfalls have been described elsewhere.7,8 In this article, we briefly highlight those that are particularly relevant to CAM and integrative medicine. One pitfall is a “confirmatory bias” defined as a tendency of some reviewers to accept outcomes that agree with commonly accepted theories and to reject those that do not.9 This bias seems to work against innovative studies10 that are needed in newly developing disciplines. Studies in CAM are especially at risk of encountering confirmatory bias when submitted to conventional medical journals because such studies often challenge the accepted conventional notions. For example, positive studies of certain homeopathic (infinitely diluted) medicines seemingly defy basic principles of physical chemistry.11 Acupuncture theory assumes the existence of vital energy (Qi) and a subtle energy anatomy of the body (meridians); both of these concepts are held highly suspect by biomedicine. CAM studies may suggest a mind– body causality that is inconsistent with mainstream assumptions.12 Confirmatory bias may extend to study methodology as well: proof by randomized, placebo controlled clinical trial is itself a commonly accepted theory. However, there is no adequate placebo for certain CAM interventions. For example, European mistletoe injections in cancer patients induce a tell-tale local inflammation. Also, subjects may simply be unwilling to be randomized between two, dramatically different, active treatments. This is the case with a current, NIH funded study of pancreatic cancer patients who receive either standard chemotherapy or a regimen of pancreatic enzymes, micronutrient supplements, and diet therapy: Investigators were forced to abandon a randomized design.
Another bias is the “file drawer” defined as the reviewer’s tendency to recommend studies that report positive results and reject those with negative findings.13 The studies in which the hypothesis of no relationship (null hypothesis) is accepted have a much lower chance of being recommended for publication.14,15 Consideration of this bias is important in new disciplines, because it increases the probability of “false positive” conclusions. This may lead to the premature acceptance and application of either a conventional therapy or an unconventional therapy. A “reverse” file drawer bias may very well be a problem among reviewers of conventional medical journals, where the publication of negative studies may further a scientific agenda of disproving CAM. Some raised this concern when JAMA chose to publish a child’s 4th grade science project as a refutation of therapeutic touch.16 In a sense, favoring the publication of a negative CAM study may be considered a form of confirmatory bias. Another bias is called “the Matthew effect”17 defined as the reviewers’ tendency to recommend manuscripts submitted by famous investigators from prestigious institutions regardless of the manuscripts’ scientific and technical merit.18 This bias decreases the publication chance of new investigators who are not affiliated with prestigious research centers. The Matthew effect may be a concern when CAM studies are conducted by new investigators at nonallopathic institutions, such as colleges of chiropractic, naturopathy, or Oriental Medicine. A final issue is bias due to the reviewer’s theoretical persuasion and ideological orientation. The potential for this pitfall in CAM was explored in an unusual study by Resch and colleagues.19 They prepared two near-identical manuscript versions of a fictitious positive study. Version A demonstrated the efficacy of a conventional therapy (pharmaceutical agent), while version B demonstrated efficacy of a CAM therapy (homeopathic remedy) in treatment of obesity. Journal reviewers, randomly assigned to receive either version, were more likely to favorably review the conventional treatment.
SUGGESTIONS
S
TO
REDUCE BIAS
everal approaches have been used to alleviate the effects of some of the reviewers’ biases. These approaches include the “single-blind” re-
EDITORIAL
view, in which the reviewer’s identity is concealed from the author(s), and the “doubleblind” review, in which the authors’ identity and affiliation are concealed from the reviewer’s, and the reviewer’s identity is concealed from the author(s). These approaches cannot completely conceal the authors’ or the affiliated institutions’ identities.20 Studies have shown that neither concealing the authors’ identities nor requiring the reviewers to sign their reports had any measurable effect on the review quality.21 In contrast, McNutt and colleagues22 found that blinding improved the quality of the reviews. Blind review (double- or singleblind) may help to mitigate the Matthew effect, but it is extremely difficult to resolve confirmation bias, the file drawer problem, and theoretical persuasion issues. Some journals ask authors to prepare their manuscript by starting the introduction, methods, results, and conclusions on new pages, so that different reviewers can assess different sections of the manuscript independent of other sections. In this case, emphasis is placed on the assessment of methodology independent of the results and their interpretations.23 Of course, reviewers cannot be blinded to the use of CAM versus conventional medicine in a particular study. Editors must therefore be particularly careful to clearly define review criteria and to examine reviews for evidence of confirmatory and ideological biases. Human services professionals are committed to comply with a set of professional ethical codes, and their lack of awareness of the codes of conduct cannot be a defense against charges of misconduct. For example, ethical code number 9.06 developed by the American Psychological Association (APA) specifies that psychologists must take into account the purpose of assessments and also be aware of various pitfalls that might affect their judgments and reduce the accuracy of their assessments.24 Similarly, we propose that, for the sake of impartiality and fairness in journal peer review process, it would be desirable to develop a set of ethical guidelines for reviewers. The pitfalls in the peer review process that may influence the reviewers’ judgments can be described and approaches to avoid bias can be suggested in the guidelines. Journal reviewers, as well as editors, should be required to comply with the reviewers’
3
ethical guidelines to safeguard impartiality in peer review publication.
CONCLUSION
M
anuscript review is a time-demanding task that brings neither fame nor fortune to journal reviewers who volunteer their time and make sincere efforts to improve dissemination of scientific information.25 Qualified reviewers are assets to the journals, but the pool of qualified reviewers is quite small.26 Biases in the peer review process are rarely publicized, but evidence suggests that they do exist.27 While we are convinced that the peer review system is working reasonably well, we strongly believe that the system can and must be improved and strengthened. Although editors have discretion in editorial decision making, they are often guided by the reviewers’ comments. Because editors have the ultimate responsibilities in the outcomes of peer review process, they should be as vigilant as possible to assure that the reviewers’ assessments of the submitted manuscripts remain as impartial as possible. The power of a reviewer is evident in the editor’s final publication decision. It has been reported that papers receiving a single negative review have a low probability of being accepted.23 We believe that the key to generating proper changes lies mostly within the journal editors. They should make sure that the new reviewers are aware of the pitfalls of peer review process, as well as the journal’s goal, guidelines and expectations. Awareness of the issues is half of the solution. The editors and reviewers, as well as journal readers and manuscript contributors, should participate in assessing the peer review process to assure its integrity and fairness. This is particularly crucial for new journals in new multidisciplinary areas such as integrative medicine, which ventures into a relatively new territory of scientific inquiry and may be particularly vulnerable to reviewer bias. Improvement of the peer review process requires input from journal readers, manuscript contributors, as well as reviewers, and more importantly, the journal editors. Editors of peer review journals are in a position to watch over the gatekeepers of science, and thus can ultimately influence the flow of scientific information.
4
HOJAT AND ROSENZWEIG
REFERENCES 1. Bailar JC, Patterson K: Journal peer review: The need for a research agenda. N Engl J Med 312:654-657, 1985 2. Crane D: The gatekeepers of science: Some factors affecting the selection of articles for scientific journals. Am Sociol 32:195-201, 1967 3. DeGrazia A: The scientific reception system and Dr. Velikovsky. Am Behav Sci 7:38-56, 1963 4. Lock S: A Difficult Balance: Editorial Peer Review in Medicine. Philadelphia, PA ISI Press, 1985 5. Lock S: The Future of Medical Journals. Plymouth, England, British Medical Journal, 1992 6. Jefferson T, Godlee F: Peer Review in Health Care. London, England, British Journal Publishing Group, 1999 7. Hojat M, Gonnella JS, Caelleigh AS: Impartial judgment by the “gatekeepers” of science: Fallibility and accountability in the peer review process. Adv Health Sci Educ 8:75-96, 2003 8. Caelleigh AS, Hojat M, Steinecke A, Gonnella JS: Effects of reviewers’ gender on assessments of a gender-related standardized manuscript. Teach Learn Med 15:163-167, 2003 9. Mahoney M: Publication prejudices: An experimental study of confirmatory bias in peer review system. Cogn Ther Res 1:161-175, 1977 10. Olson CM: Peer review of biomedical literature. Am J Emerg Med 8:356-358, 1990 11. Linde K, Clausius N, Ramirez G, et al: Are the clinical effects of homeopathy placebo effects? A meta-analysis of placebo-controlled trails. Lancet 350(9081):834-843, 1997 12. Relman A, Angell M: Resolved: Psychosocial interventions can improve clinical outcomes in organic disease: Rebuttals. Psychosom Med 64:564-567, 2002 13. Rosenthal R: The “file drawer problem” and tolerance of null results. Psychol Bull 86:638-541, 1979 14. Greenwald AG: Consequences of prejudice against the null hypothesis. Psychol Bull 82:1-20, 1975
15. Dickersin K: The existence of publication bias and risk factors for its occurrence. JAMA 263:1385-1389, 1990 16. Carpenter J, Hagemaster J, Joiner B: An even closer look at therapeutic touch. JAMA 280:1905 & 1908, 1998 17. Merton RK: The Matthew effect in science. Science 159:56-63, 1968 18. Peters DP, Ceci SJ: Peer review practices of psychological journals: The fate of published articles, submitted again. Behav Brain Sci 5:187-195, 1982 19. Resch KI, Ernst E, Garrow J: A randomized controlled study of reviewer bias against an unconventional therapy. J R Soc Med 93:164-167, 2000 20. Fisher M, Friedman SB, Strauss B: The effects of blinding on acceptance of research papers by peer review. JAMA 272: 143-146, 1994 21. Godlee F, Gale C, Martyn CN: Effect on the quality of peer review of blinding reviewers and asking them to sign their names. JAMA 280:237-240, 1998 22. McNutt RA, Evans AT, Fletcher RH, et al: The effects of blinding on the quality of peer review. JAMA 263:1371-1376, 1990 23. Armstrong SJ: We need to rethink the editorial role of peer reviewers. Chron Higher Educ 43(9):B3, October 1996 24. American Psychological Association (APA): Ethical Principles of Psychologists and Code of Conduct. Washington, DC, APA, 2002 25. Colman AM: Editorial role in author-referee disagreements. Bull Br Psychol Soc 32:390-391, 1979 26. Bloom FE: The importance of reviewers. Science 283: 789, 1999 27. Altman DG, Chalmers I, Herxheimer A: Is there a case for an international medical scientific press council? JAMA 272:166-167, 1994