- Email: [email protected]
Peer commentary on peer review: A case study in scientific quality control
Book reviews
38
understanding in the introduction would be helpful. Several review collections now exist from cognitive psychology and artificial intelligence which could serve as a basis for an introductory attempt to relate the topics of the various papers to current discussions within more basic research. To sum up, Sime and Coombs’s book presents a collection of papers which can be useful to illustrate the state of affairs in the design of computer systems to newcomers. As a guide on design it leaves something to be desired. We are still waiting for an essay on design of computer systems, like C. Alexanders’ On design (Harvard University Press, 1964). the synthesis off orm for architectural This, however, clearly has not been the aim of the editors.
J. Rasmussen
Ris# National Laboratory Roskilde, Denmark
S. Harnad
(Editor).
University
Press,
Peer commentary on peer review: A case study in scientafic quality control. Cambridge
1982. 71 pp. ISBN
0 521 27306 4.
This is a most unusual publication. It contains a reprint of an article by D. P. Peters and Stephen J. Ceci ‘Peer-review practices of psychology journals: the fate of published articles, submitted again’ first published in Behavioral and Brain Sciences, volume 5, number 2. The authors selected twelve published research articles by authors from prestigious and very productive American psychology departments, originally published in highly regarded and widely read American psychology journals, with rejection rates of about 80 per cent, and non-blind refereeing practices. Peters and Ceci substituted fictitious names and institutions and re-submitted the articles to journals that had originally refereed them between 18 and 32 months earlier. This process involved 38 editors and reviewers. Only 3 of the 12 articles were detected. Thus 9 of the resubmitted articles went through a second review process. Eight of these were rejected. Sixteen of the eighteen referees (89 per cent) recommended against publication and the editors agreed. In many cases the grounds for rejection related to serious methodological flaws in the experiments reported in the articles. The rejections were unanimous-on average, two referees per article, plus the journal editor. Peters and Ceci concluded that the peer-review system, at least in psychology, is biased against authors from low-status institutions and that a blind-review procedure would eliminate this bias. In the past few decades much attention has been paid to the peer-review system, especially in science, and to the ways in which scientists develop consensus. What makes the present volume particularly interesting and unusual is that the article, after publication in Behavioral and Brain Sciences was sent to over 50 commentators, including editors in the social and physical sciences, grant-given bodies, bibliometricians, sociologists of science, investigators, advocates, philosophers of science, as well as experimental critics, and reformers of the peer-review system. Commentators cover just about every aspect of the peer-review system. Some are particularly critical of the Peters and Ceci experiment from an ethical point of view, and others were critical of its methodology. Perhaps the most telling methodological criticism concerns the high rejection
Book reviews
39
rate (about 80 per cent) of the psychology journals that featured in the study. Journal editors and most referees are of course aware that approximately 80 per cent of the manuscripts they receive will be rejected. Therefore, both editors and referees are set to reject manuscripts because of this management The slightest problem alone, irrespective of the quality of the manuscript. error, disagreement, inconsistency, methodological softness, and so forth, is likely to be seized upon as a reason for rejection. Before this study can be taken seriously, it is necessary to repeat it in a subject with a low rejection rate-(say) in the order of 1O-l 5 per cent. The Peters and Ceci experiment was not definitive. There is no agreement amongst the commentators about the conclusion reached by Peters and Cecithat the peer-review system is unduly influenced by institutional status of authors. We can agree that it is essential to acquire a better understanding of the journal review systems. If it is not objective and reliable it is necessary to show why. An understanding of this matter will require further study of the way in which knowledge evolves in the social sciences. This in turn will involve a questioning of the dominant paradigm of the scientific model of progress and its applicability to psychology. Few commentators deal with these issues which include: 1. The reasons why the social sciences have had little success in solving social, economic, political and anthropological problems. 2. The reasons for low consensus in the social sciences. 3. The appropriateness of the so-called classical scientific method (drawn mainly from physics) for the social sciences, and psychology in particular. For those interested in the philosophy and sociology of science the short and often pithy and informative remarks of the commentators will be of interest. In order to whet the appetite, we can refer to just one or two. Michael Moravcsik of the Institute of Theoretical Science at the University of Oregon suggests that the rejection rate in physics journals is about 20-30 per cent and that it is no accident that it is so low, because there is a high degree of consensus in physics concerning what is right and what is wrong; and furthermore, physics is much more cumulative than psychology, in the sense that even small contributions are judged useful additions to the structure of knowledge. Moravcsik also suggests that plagiarism and the resubmission of manuscripts going undetected would be unlikely to arise in physics. One of the reasons is that many fields in physics are well delineated, and the community of physicists within each subfield is highly interactive. Hence, bogus people with fictitious institutional affiliations would be easily recognizable as fake, and plagiarism would be more obvious. Many of the commentators discuss differences between science and social science. Ziman suggests that the reviewers involved in the ‘experiment’ were grossly incompetent. They were so ignorant of their subjects that at least 75 per cent of them did not even know that the very same work had been done before. Furthermore, they had overlooked serious methodological errors, noted by their reviewers on the second time round. Ziman says that in his experience in physics he has never come across anything like such widespread incompetence or irresponsibility. He is surprised that such a thing could happen in psychology and suggests that it needs to be tested further. Surely he is going too far? Other commentators (e.g., Manwell and Baker, of the Department of Zoology, University of Adelaide) refer to the considerable literature on plagiarism in science. For example, a recent widely publicized case which
Book reviews
40
involved a Jordanian researcher who pirated a number of already published papers and recycled these under his own name with minimal alterations that (Broad, 1981; Manwell and Baker, 1981). M anwell and Baker conclude because this plagiarist was able to amass a considerable publication list based upon other scientists’ work many referees and editors in science cannot detect resubmissions based upon already published research. There is also a considerable literature on research on referee agreement. Agreement tends to be low in many fields (Whitehurst, 1983, 1984), and even when the data from research on referee agreement has been reworked to take account of criticism of the statistical techniques improvements in agreement have been small (Finn, 1972; Whitehouse, 1983, 1984). The commentary on the Peters and Ceci article makes very interesting especially so since well-known scientists obviously disagree reading, fundamentally about the nature of peer-review, and also about the way in which knowledge is structured and created in science as well as in the social sciences. This is essential reading for all those concerned with the sociology and philosophy of science; the nature of research and scholarship in the social sciences; and indeed, for information providers who are no longer willing to accept information and documents at their face value. There can be few who now hold to the belief that scientific research, scholarship, and writing is rational, structured and orderly. But a reading of this monograph may still today surprise some that there is so much disagreement. J.
M. Brittain
Department of Library and Information Studies Loughborough Uniuersity of TechnoloQ
REFERENCES Fraud and the structure of science. Science, 212, 137-141. Effects of some variations in rating scale characteristics on the means and reliabilities of ratings. Educational and Psychological Measurement, 35, 255-265. MANWELL., c. and BAKER, c. M A. (1979). The double helix: science and myth in the act of creation. BioScience, 29, 742-746. WHITEHURST, G. J. (1983). Interrater agreement for reviews for ‘Development Reviews’. BROAD,
w. J. (1981).
FINN, R. H. (1972).
Developmental Reviews, 3, 73-78. WHITEHURST,
G. J.
(1984).
Interrater
American Psychologist, 39(l), 22-28.
Euan
Blauvelt
and Jennifer
Durlacher
Sources of European economic information.
agreement
for journal
(Editors). 4th edn. Gower,
1983.
manuscript
reviews.
644 pp.
The 4th edition of Sources of European economic information builds on the foundations laid in the first three editions, that is, to identify within the covers of one volume the major sources of European economic and statistical information together with the names and addresses of the relevant publishers. This edition of the guide runs to almost 650 pages and contains approximately 2000 entries. In the last six years the guide has been updated three times which suggests that regular revision is a feature of this publication. A significant development with the 4th edition is the inclusion for the first time of source publications from East European states. Nine countries are
38
understanding in the introduction would be helpful. Several review collections now exist from cognitive psychology and artificial intelligence which could serve as a basis for an introductory attempt to relate the topics of the various papers to current discussions within more basic research. To sum up, Sime and Coombs’s book presents a collection of papers which can be useful to illustrate the state of affairs in the design of computer systems to newcomers. As a guide on design it leaves something to be desired. We are still waiting for an essay on design of computer systems, like C. Alexanders’ On design (Harvard University Press, 1964). the synthesis off orm for architectural This, however, clearly has not been the aim of the editors.
J. Rasmussen
Ris# National Laboratory Roskilde, Denmark
S. Harnad
(Editor).
University
Press,
Peer commentary on peer review: A case study in scientafic quality control. Cambridge
1982. 71 pp. ISBN
0 521 27306 4.
This is a most unusual publication. It contains a reprint of an article by D. P. Peters and Stephen J. Ceci ‘Peer-review practices of psychology journals: the fate of published articles, submitted again’ first published in Behavioral and Brain Sciences, volume 5, number 2. The authors selected twelve published research articles by authors from prestigious and very productive American psychology departments, originally published in highly regarded and widely read American psychology journals, with rejection rates of about 80 per cent, and non-blind refereeing practices. Peters and Ceci substituted fictitious names and institutions and re-submitted the articles to journals that had originally refereed them between 18 and 32 months earlier. This process involved 38 editors and reviewers. Only 3 of the 12 articles were detected. Thus 9 of the resubmitted articles went through a second review process. Eight of these were rejected. Sixteen of the eighteen referees (89 per cent) recommended against publication and the editors agreed. In many cases the grounds for rejection related to serious methodological flaws in the experiments reported in the articles. The rejections were unanimous-on average, two referees per article, plus the journal editor. Peters and Ceci concluded that the peer-review system, at least in psychology, is biased against authors from low-status institutions and that a blind-review procedure would eliminate this bias. In the past few decades much attention has been paid to the peer-review system, especially in science, and to the ways in which scientists develop consensus. What makes the present volume particularly interesting and unusual is that the article, after publication in Behavioral and Brain Sciences was sent to over 50 commentators, including editors in the social and physical sciences, grant-given bodies, bibliometricians, sociologists of science, investigators, advocates, philosophers of science, as well as experimental critics, and reformers of the peer-review system. Commentators cover just about every aspect of the peer-review system. Some are particularly critical of the Peters and Ceci experiment from an ethical point of view, and others were critical of its methodology. Perhaps the most telling methodological criticism concerns the high rejection
Book reviews
39
rate (about 80 per cent) of the psychology journals that featured in the study. Journal editors and most referees are of course aware that approximately 80 per cent of the manuscripts they receive will be rejected. Therefore, both editors and referees are set to reject manuscripts because of this management The slightest problem alone, irrespective of the quality of the manuscript. error, disagreement, inconsistency, methodological softness, and so forth, is likely to be seized upon as a reason for rejection. Before this study can be taken seriously, it is necessary to repeat it in a subject with a low rejection rate-(say) in the order of 1O-l 5 per cent. The Peters and Ceci experiment was not definitive. There is no agreement amongst the commentators about the conclusion reached by Peters and Cecithat the peer-review system is unduly influenced by institutional status of authors. We can agree that it is essential to acquire a better understanding of the journal review systems. If it is not objective and reliable it is necessary to show why. An understanding of this matter will require further study of the way in which knowledge evolves in the social sciences. This in turn will involve a questioning of the dominant paradigm of the scientific model of progress and its applicability to psychology. Few commentators deal with these issues which include: 1. The reasons why the social sciences have had little success in solving social, economic, political and anthropological problems. 2. The reasons for low consensus in the social sciences. 3. The appropriateness of the so-called classical scientific method (drawn mainly from physics) for the social sciences, and psychology in particular. For those interested in the philosophy and sociology of science the short and often pithy and informative remarks of the commentators will be of interest. In order to whet the appetite, we can refer to just one or two. Michael Moravcsik of the Institute of Theoretical Science at the University of Oregon suggests that the rejection rate in physics journals is about 20-30 per cent and that it is no accident that it is so low, because there is a high degree of consensus in physics concerning what is right and what is wrong; and furthermore, physics is much more cumulative than psychology, in the sense that even small contributions are judged useful additions to the structure of knowledge. Moravcsik also suggests that plagiarism and the resubmission of manuscripts going undetected would be unlikely to arise in physics. One of the reasons is that many fields in physics are well delineated, and the community of physicists within each subfield is highly interactive. Hence, bogus people with fictitious institutional affiliations would be easily recognizable as fake, and plagiarism would be more obvious. Many of the commentators discuss differences between science and social science. Ziman suggests that the reviewers involved in the ‘experiment’ were grossly incompetent. They were so ignorant of their subjects that at least 75 per cent of them did not even know that the very same work had been done before. Furthermore, they had overlooked serious methodological errors, noted by their reviewers on the second time round. Ziman says that in his experience in physics he has never come across anything like such widespread incompetence or irresponsibility. He is surprised that such a thing could happen in psychology and suggests that it needs to be tested further. Surely he is going too far? Other commentators (e.g., Manwell and Baker, of the Department of Zoology, University of Adelaide) refer to the considerable literature on plagiarism in science. For example, a recent widely publicized case which
Book reviews
40
involved a Jordanian researcher who pirated a number of already published papers and recycled these under his own name with minimal alterations that (Broad, 1981; Manwell and Baker, 1981). M anwell and Baker conclude because this plagiarist was able to amass a considerable publication list based upon other scientists’ work many referees and editors in science cannot detect resubmissions based upon already published research. There is also a considerable literature on research on referee agreement. Agreement tends to be low in many fields (Whitehurst, 1983, 1984), and even when the data from research on referee agreement has been reworked to take account of criticism of the statistical techniques improvements in agreement have been small (Finn, 1972; Whitehouse, 1983, 1984). The commentary on the Peters and Ceci article makes very interesting especially so since well-known scientists obviously disagree reading, fundamentally about the nature of peer-review, and also about the way in which knowledge is structured and created in science as well as in the social sciences. This is essential reading for all those concerned with the sociology and philosophy of science; the nature of research and scholarship in the social sciences; and indeed, for information providers who are no longer willing to accept information and documents at their face value. There can be few who now hold to the belief that scientific research, scholarship, and writing is rational, structured and orderly. But a reading of this monograph may still today surprise some that there is so much disagreement. J.
M. Brittain
Department of Library and Information Studies Loughborough Uniuersity of TechnoloQ
REFERENCES Fraud and the structure of science. Science, 212, 137-141. Effects of some variations in rating scale characteristics on the means and reliabilities of ratings. Educational and Psychological Measurement, 35, 255-265. MANWELL., c. and BAKER, c. M A. (1979). The double helix: science and myth in the act of creation. BioScience, 29, 742-746. WHITEHURST, G. J. (1983). Interrater agreement for reviews for ‘Development Reviews’. BROAD,
w. J. (1981).
FINN, R. H. (1972).
Developmental Reviews, 3, 73-78. WHITEHURST,
G. J.
(1984).
Interrater
American Psychologist, 39(l), 22-28.
Euan
Blauvelt
and Jennifer
Durlacher
Sources of European economic information.
agreement
for journal
(Editors). 4th edn. Gower,
1983.
manuscript
reviews.
644 pp.
The 4th edition of Sources of European economic information builds on the foundations laid in the first three editions, that is, to identify within the covers of one volume the major sources of European economic and statistical information together with the names and addresses of the relevant publishers. This edition of the guide runs to almost 650 pages and contains approximately 2000 entries. In the last six years the guide has been updated three times which suggests that regular revision is a feature of this publication. A significant development with the 4th edition is the inclusion for the first time of source publications from East European states. Nine countries are