- Email: [email protected]
Suspended judgment
SUSPENDED JUDGMENT Is the Scientific Paper a Fraud? William A. Silverman, MD
In 1662, when the Royal Society of London for Improving Natural Knowledge was incorporated, the members chose Robert Hooke for the post of Curator of Experiments, assigned to perform experiments suggested by members. Daniel Boorstin has noted [1] that at the time of this appointment, Hooke sounded a keynote that marked the beginning of a new scientific age: "The truth is [that] the science of Nature has been already too long made only a work of the brain and the fancy. It is now high time that it should return to the plainness and soundness of observations on material and obvious things." The needs to disseminate the new information and to provide a public record documenting the details of the experiments in this scientific revolution were recognized at the very outset. Many of the early "natural scientists" recorded their new observations and concepts in letters sent to an international network of correspondents. The letters came to be called "epistolary discourses" [2]: they were the forerunners of a new written form the scientific paper. More than 25 years ago Sir Peter Medawar attacked the format of the m o d e m scientific paper as fraudulent [3]. Articles in the form in which they are communicated to learned journals are, he charged, notorious for misrepresenting the processes of thought that led to whatever discoveries they describe. More recently, scientific communications have been criticized because they project a misleading image of perfection. Bob Coleman, a novelist and a perceptive critic of the written word, has observed that scientific writing often seems too good to be true [4]. He quoted a widely published biochemist who said, "Nowadays, journals don't even want you to discuss your negative experiments. The very hint that you might have erred makes you insufficiently Olympian to be in print." The m o d e m artificiality is odd because an open confession of difficulties, of failures, and of errors was considered to be part and parcel of the earliest scientific reports of efforts to obtain synthetic knowledge ("covered with the fingerprints and grime of the material world"), as distinct from analytical knowledge ("glowing with the radiance of disembodied thought") [5]. In 1661, Robert Boyle published two essays entitled "Of the Unsuccessfulness of Experiments" and "Of Unsucceeding Experiments" [6]. In the esAddress reprint requests to: William A. Silverman, MD, 90 La Cuesta Drive, Greenbrae, CA 94904.
February 1, 1990; revised October4, 1990. Controlled Clinical Trials 12:273-276 (1991) O Elsevier Science Publishing Co., Inc. 1991 655 Avenue of ~he Americas, New York, New York 10010
273 0197-2456/91/$3.50
274
W.A. Silverman
says, he told the reader, . you will meet with several Observations and Experiments, which though communicated for true by Candid Authors or undistrusted Eye-witnesses, or perhaps recommended to you by your own experience, may upon further tryal disappoint your expectation, either not at all succeeding constantly, or at least varying much from what you expected." And he counseled p a t i e n c e , " . . , the despondency [my words are] apt to produce [may be lessened] by letting you see, that though some of your Experiments should not always prove constant, you have divers Partners in that infelicity, who have not been discouraged." Coleman's review of early volumes of the Transactions of the American Philosophical Society (these began to be published in 1771), turned up numerous frank accounts of troubles in the carrying out of "experiments that let light into the nature of things" [4]. During the Society's early years, failure, or difficulty, he found, was a fundamental rhetorical device used to both start and end reports. However, by the mid-1800s, with the burgeoning growth of scientific activity, the lively, accurate accounts in the Society's journal were rapidly replaced by the familiar modern form: a colorless progression of abstracts, abbreviated introductions, methods, results, and interpretations. The change in writing style was accompanied by a difference in attitude about failure to achieve predicted results. Prior to the change, in 1750, the Abbe Nollet remarked about one of his experiments, "When I found my Attempts were fruitless, I without any difficulty communicated it to all other Philosophers with w h o m I corresponded." This type of open dedaration was replaced by what Stephen Gould has called "a perverse silence about 'no effect found'" [7]. Null results---the failure to find any effect in any direction-came to be viewed as an experiment gone awry. And scientists became reluctant to "expose" their failures to public view. For example, a research assistant of the exalted Ernest Rutherford, who was awarded the 1908 Nobel prize in physics, reported that his chief worked incessantly, always in search of something new [8]. But he reported or published only those works which had a positive result; these were a very small percent of the whole mass of experimental work he carried out. The remainder were not published, and were unknown, even to his students. In the current era, a discrete system of peer review of papers before publication has evolved; it hides most of the discussion and disputes about research problems from the reader's view. The modern gate-keeping process is highly regarded, but it has been subjected to surprisingly few rigorously designed tests. John Bailar and Kay Patterson point out a paradox [9]: "the arbiters of rigor, quality, and innovation in scientific reports submitted for publication do not apply to their own work the standards they use in judging the work of others." They have called for a research agenda to examine the journal peer-review process. Results of the first organized effort to encourage such research were reported at an international congress convened in 1989, and another meeting is planned for 1992 [10]. Surveys of methodologic rigor in published reports of clinical trials suggest that advance editorial review of trial design and analysis leaves much to be desired [11,12]. Less attention has been paid to the prejudicial influence of referees" mind-set. The survey of publication bias in clinical trials by Kay Dickersin and her associates [13] indicated that a bias against pub"
.
.
275
Is the Scientific Paper a Fraud?
lication of "negative results" does exist. But the disinclination to publish found in this survey seemed to originate most often with the investigators' low opinion of disappointing study results (the "file drawer problem") and less often because of rejection by journal reviewers after papers were submitted for publication. A controlled trial conducted by Michael Mahoney focused directly on publication bias that takes place at the level of peer review [14]. He set out to examine the role of confirmatory bias (the tendency to emphasize and believe experiences that support one's view and to ignore or discredit those which do not) among 75 reviewers of a psychology journal. The referees were divided (by random allocation) into five groups and they were asked to review manuscripts which described identical (fictitious) experiments. In each group, the "results" sections of the test manuscripts were systematically altered to report one of five different outcomes: results that were either positive, negative, null, mixed with a positive interpretation, or mixed with a negative interpretation. The reviewers were found to be strongly biased against manuscripts that reported results contrary to their theoretical perspective. Mahoney suggests that one possible approach to the problem of referees' outcome-bias might be to ask for masked reviews: an evaluation of the relevance and methodology in a manuscript describing an experiment without seeing either the results or their interpretation. "Given that the researched question is relevant and the experimental methodology adequate," he argues, "the obtained results--whatever they might be--should be of interest to the scientific community." The suggestion is not new, a similar proposal for journal reviews in the social sciences was made by William Walster and Ann Cleary 20 years ago [15], and the logic was straightforward. Since the determination to publish the results of a research study is a particular treatment of data, they opined, it makes sense that the same limitations should be imposed on publication judgments as are demanded in all experimental design decisions. A basic rule of scientific procedure is that any determination regarding the management of data must be made before the findings are inspected. When this rule is extended to publication verdicts, it requires that w h e n a report is submitted for review the results and the conclusions should be withheld. What is missing in the recommendations for outcome-masked reviews is evidence from comparative field trials using real-life manuscripts submitted to, say, medical journals, and an agreement concerning the objective outcome of interest (for example, a sizable increase in the proportion of methodologically sound manuscripts accepted for publication after outcome-masked as compared with unmasked reviews). Nonetheless, Mahoney's question cannot be dismissed lightly: "Have we presumed that [peer review] is 'naturally' reliable and objective? . [H]ave we assumed that scientists are somehow unaffected by the processes which appear to be so common in other members of the species?" Cognitive bias was recognized as a potential problem at the very beginning of the scientific revolution. In 1621, Francis Bacon observed: [16] .
.
[T]he human intellect.., is more moved and excited by affirmatives than by negatives.
276
W.A. Silverman
REFERENCES
1. 2. 3. 4.
Boorstin DJ: The Discoverers. New York, Random House, 1983 Cohen 113:Revolution in Science. Cambridge, MA, Harvard University Press, 1985 Medawar PB: Is the scientific paper a fraud? The Listener 70:377-378, 1963 Coleman B: Science writing: Too good to be true? New York Times, September 27, 1987 5. Heilbroner RL: The missing link. NY Review of Books, November 5, 1981. 6. Boyle R: Certain Philosophical Essays. London, 1661 7. Gould SJ: Cardboard Darwinism. NY Review of Books, September 25, 1986. 8. Wilson D: Rutherford: Simple Genius. London, Hodder & Stoughton, 1983 9. Baillar JC IlI, Patterson K: Journal peer review. The need for a research agenda. N Engl J Med 312:654--657, 1985 10. Rennie D: Selected proceedings from the First International Congress on Peer Review in Biomedical Publication. JAMA 263:1317-1441, 1990 11. DerSimonian R, Charette LJ, McPeek B, Mostellar F: Reporting on methods in clinical trials. N Engl J Med 306:1332-1337, 1982 12. Pocock SJ, Hughes MD, Lee RJ: Statistical problems in the reporting of clinical trials. A survey of three medical journals. N Engl J Med 317:426-432, 1987 13. Dickersin K, Chart S, Chalmers TC, Sacks HS, Smith H Jr: Publication bias and clinical trials. Controlled Clin Trials 8:34,3-,353, 1987 14. Mahoney MJ: Publication prejudices: An experimental study of confirmatory process bias in the peer review system. Cogn Ther Res 1:161-175, 1977 15. Walster GW, Cleary TA: A proposal for a new editorial policy in the social sciences. Am Stat 28:16-19, 1970 16. Bacon F: Novum Orgsnum. New York, Bobbs-Merrill, 1960 (originally published in 1621)
In 1662, when the Royal Society of London for Improving Natural Knowledge was incorporated, the members chose Robert Hooke for the post of Curator of Experiments, assigned to perform experiments suggested by members. Daniel Boorstin has noted [1] that at the time of this appointment, Hooke sounded a keynote that marked the beginning of a new scientific age: "The truth is [that] the science of Nature has been already too long made only a work of the brain and the fancy. It is now high time that it should return to the plainness and soundness of observations on material and obvious things." The needs to disseminate the new information and to provide a public record documenting the details of the experiments in this scientific revolution were recognized at the very outset. Many of the early "natural scientists" recorded their new observations and concepts in letters sent to an international network of correspondents. The letters came to be called "epistolary discourses" [2]: they were the forerunners of a new written form the scientific paper. More than 25 years ago Sir Peter Medawar attacked the format of the m o d e m scientific paper as fraudulent [3]. Articles in the form in which they are communicated to learned journals are, he charged, notorious for misrepresenting the processes of thought that led to whatever discoveries they describe. More recently, scientific communications have been criticized because they project a misleading image of perfection. Bob Coleman, a novelist and a perceptive critic of the written word, has observed that scientific writing often seems too good to be true [4]. He quoted a widely published biochemist who said, "Nowadays, journals don't even want you to discuss your negative experiments. The very hint that you might have erred makes you insufficiently Olympian to be in print." The m o d e m artificiality is odd because an open confession of difficulties, of failures, and of errors was considered to be part and parcel of the earliest scientific reports of efforts to obtain synthetic knowledge ("covered with the fingerprints and grime of the material world"), as distinct from analytical knowledge ("glowing with the radiance of disembodied thought") [5]. In 1661, Robert Boyle published two essays entitled "Of the Unsuccessfulness of Experiments" and "Of Unsucceeding Experiments" [6]. In the esAddress reprint requests to: William A. Silverman, MD, 90 La Cuesta Drive, Greenbrae, CA 94904.
February 1, 1990; revised October4, 1990. Controlled Clinical Trials 12:273-276 (1991) O Elsevier Science Publishing Co., Inc. 1991 655 Avenue of ~he Americas, New York, New York 10010
273 0197-2456/91/$3.50
274
W.A. Silverman
says, he told the reader, . you will meet with several Observations and Experiments, which though communicated for true by Candid Authors or undistrusted Eye-witnesses, or perhaps recommended to you by your own experience, may upon further tryal disappoint your expectation, either not at all succeeding constantly, or at least varying much from what you expected." And he counseled p a t i e n c e , " . . , the despondency [my words are] apt to produce [may be lessened] by letting you see, that though some of your Experiments should not always prove constant, you have divers Partners in that infelicity, who have not been discouraged." Coleman's review of early volumes of the Transactions of the American Philosophical Society (these began to be published in 1771), turned up numerous frank accounts of troubles in the carrying out of "experiments that let light into the nature of things" [4]. During the Society's early years, failure, or difficulty, he found, was a fundamental rhetorical device used to both start and end reports. However, by the mid-1800s, with the burgeoning growth of scientific activity, the lively, accurate accounts in the Society's journal were rapidly replaced by the familiar modern form: a colorless progression of abstracts, abbreviated introductions, methods, results, and interpretations. The change in writing style was accompanied by a difference in attitude about failure to achieve predicted results. Prior to the change, in 1750, the Abbe Nollet remarked about one of his experiments, "When I found my Attempts were fruitless, I without any difficulty communicated it to all other Philosophers with w h o m I corresponded." This type of open dedaration was replaced by what Stephen Gould has called "a perverse silence about 'no effect found'" [7]. Null results---the failure to find any effect in any direction-came to be viewed as an experiment gone awry. And scientists became reluctant to "expose" their failures to public view. For example, a research assistant of the exalted Ernest Rutherford, who was awarded the 1908 Nobel prize in physics, reported that his chief worked incessantly, always in search of something new [8]. But he reported or published only those works which had a positive result; these were a very small percent of the whole mass of experimental work he carried out. The remainder were not published, and were unknown, even to his students. In the current era, a discrete system of peer review of papers before publication has evolved; it hides most of the discussion and disputes about research problems from the reader's view. The modern gate-keeping process is highly regarded, but it has been subjected to surprisingly few rigorously designed tests. John Bailar and Kay Patterson point out a paradox [9]: "the arbiters of rigor, quality, and innovation in scientific reports submitted for publication do not apply to their own work the standards they use in judging the work of others." They have called for a research agenda to examine the journal peer-review process. Results of the first organized effort to encourage such research were reported at an international congress convened in 1989, and another meeting is planned for 1992 [10]. Surveys of methodologic rigor in published reports of clinical trials suggest that advance editorial review of trial design and analysis leaves much to be desired [11,12]. Less attention has been paid to the prejudicial influence of referees" mind-set. The survey of publication bias in clinical trials by Kay Dickersin and her associates [13] indicated that a bias against pub"
.
.
275
Is the Scientific Paper a Fraud?
lication of "negative results" does exist. But the disinclination to publish found in this survey seemed to originate most often with the investigators' low opinion of disappointing study results (the "file drawer problem") and less often because of rejection by journal reviewers after papers were submitted for publication. A controlled trial conducted by Michael Mahoney focused directly on publication bias that takes place at the level of peer review [14]. He set out to examine the role of confirmatory bias (the tendency to emphasize and believe experiences that support one's view and to ignore or discredit those which do not) among 75 reviewers of a psychology journal. The referees were divided (by random allocation) into five groups and they were asked to review manuscripts which described identical (fictitious) experiments. In each group, the "results" sections of the test manuscripts were systematically altered to report one of five different outcomes: results that were either positive, negative, null, mixed with a positive interpretation, or mixed with a negative interpretation. The reviewers were found to be strongly biased against manuscripts that reported results contrary to their theoretical perspective. Mahoney suggests that one possible approach to the problem of referees' outcome-bias might be to ask for masked reviews: an evaluation of the relevance and methodology in a manuscript describing an experiment without seeing either the results or their interpretation. "Given that the researched question is relevant and the experimental methodology adequate," he argues, "the obtained results--whatever they might be--should be of interest to the scientific community." The suggestion is not new, a similar proposal for journal reviews in the social sciences was made by William Walster and Ann Cleary 20 years ago [15], and the logic was straightforward. Since the determination to publish the results of a research study is a particular treatment of data, they opined, it makes sense that the same limitations should be imposed on publication judgments as are demanded in all experimental design decisions. A basic rule of scientific procedure is that any determination regarding the management of data must be made before the findings are inspected. When this rule is extended to publication verdicts, it requires that w h e n a report is submitted for review the results and the conclusions should be withheld. What is missing in the recommendations for outcome-masked reviews is evidence from comparative field trials using real-life manuscripts submitted to, say, medical journals, and an agreement concerning the objective outcome of interest (for example, a sizable increase in the proportion of methodologically sound manuscripts accepted for publication after outcome-masked as compared with unmasked reviews). Nonetheless, Mahoney's question cannot be dismissed lightly: "Have we presumed that [peer review] is 'naturally' reliable and objective? . [H]ave we assumed that scientists are somehow unaffected by the processes which appear to be so common in other members of the species?" Cognitive bias was recognized as a potential problem at the very beginning of the scientific revolution. In 1621, Francis Bacon observed: [16] .
.
[T]he human intellect.., is more moved and excited by affirmatives than by negatives.
276
W.A. Silverman
REFERENCES
1. 2. 3. 4.
Boorstin DJ: The Discoverers. New York, Random House, 1983 Cohen 113:Revolution in Science. Cambridge, MA, Harvard University Press, 1985 Medawar PB: Is the scientific paper a fraud? The Listener 70:377-378, 1963 Coleman B: Science writing: Too good to be true? New York Times, September 27, 1987 5. Heilbroner RL: The missing link. NY Review of Books, November 5, 1981. 6. Boyle R: Certain Philosophical Essays. London, 1661 7. Gould SJ: Cardboard Darwinism. NY Review of Books, September 25, 1986. 8. Wilson D: Rutherford: Simple Genius. London, Hodder & Stoughton, 1983 9. Baillar JC IlI, Patterson K: Journal peer review. The need for a research agenda. N Engl J Med 312:654--657, 1985 10. Rennie D: Selected proceedings from the First International Congress on Peer Review in Biomedical Publication. JAMA 263:1317-1441, 1990 11. DerSimonian R, Charette LJ, McPeek B, Mostellar F: Reporting on methods in clinical trials. N Engl J Med 306:1332-1337, 1982 12. Pocock SJ, Hughes MD, Lee RJ: Statistical problems in the reporting of clinical trials. A survey of three medical journals. N Engl J Med 317:426-432, 1987 13. Dickersin K, Chart S, Chalmers TC, Sacks HS, Smith H Jr: Publication bias and clinical trials. Controlled Clin Trials 8:34,3-,353, 1987 14. Mahoney MJ: Publication prejudices: An experimental study of confirmatory process bias in the peer review system. Cogn Ther Res 1:161-175, 1977 15. Walster GW, Cleary TA: A proposal for a new editorial policy in the social sciences. Am Stat 28:16-19, 1970 16. Bacon F: Novum Orgsnum. New York, Bobbs-Merrill, 1960 (originally published in 1621)