Measuring the quality of publications: new methodology and case study

Information Processing and Management 36 (2000) 551±570

www.elsevier.com/locate/infoproman

Measuring the quality of publications: new methodology and case study Jack P.C. Kleijnen*, Willem Van Groenendaal Department of Information Systems (BIK)/Center for Economic Research (CentER), School of Management and Economics (FEW), Tilburg University (KUB), Postbox 90153, 5000 LE, Tilburg, The Netherlands Received 21 July 1999; accepted 6 December 1999

Abstract In practice, it is important to evaluate the quality of research, in order to make decisions on tenure, funding, and so on. This article develops a methodology using citations to measure the quality of journals, proceedings, and book publishers. (Citations are also used by the Science and Social Science Citation Indexes, published by the Institute for Scienti®c Information (ISI), but these Indexes do not cover proceedings, books, and certain journals.) The novel methodology uses statistical sampling, bootstrapping, and classi®cation. This methodology is applied to the ®eld of Information Systems. In this case-study, class-1 turns out to consist of three journals Ð MIS Quarterly, Management Science, and Communications of the ACM Ð and two proceedings Ð VLDB and SIGMOD. The class-1 publishers are Springer, Wiley, and Addison-Wesley. Moreover, hundreds of other journals etc. are classi®ed into a small number of classes. 7 2000 Elsevier Science Ltd. All rights reserved. Keywords: Citation analysis; Clustering; Ranking; Statistics; Resampling

1. Introduction In practice, it is important to evaluate the quality of research, in order to make decisions on tenure, funding, a library's acquisition policy, and so on. Our approach uses citations, not other quantitative measures (such as rejection rates or number of book copies sold) or subjective measures (such as peer review, also called opinion surveys). As pointed out by Tahai * Corresponding author. Tel.: +31-13-4662029; fax: +31-13-4663377. E-mail address: [email protected] (J.P.C. Kleijnen). 0306-4573/00/$ - see front matter 7 2000 Elsevier Science Ltd. All rights reserved. PII: S 0 3 0 6 - 4 5 7 3 ( 9 9 ) 0 0 0 7 6 - X

552

J.P.C. Kleijnen, W. Van Groenendaal / Information Processing and Management 36 (2000) 551±570

and Rigsby (1998), citation studies of overall journal quality provide the most objective means to evaluate journals. Citations are also used by the Science Citation Index (SCI) and the Social Science Citation Index (SSCI), published by the Institute for Scienti®c Information (ISI). These indexes, however, do not cover all journals judged to be important for a speci®c ®eld; moreover, they do not cover any proceedings or books! How serious are these shortcomings? In relatively new ®elds Ð such as Information Systems Ð some researchers claim that proceedings are important. Moreover, Van Damme (1999, p. 22) mentions that the percentage of articles by Dutch Information Systems authors that is published in ISI journals is only 12%, whereas that percentage is 58% for Dutch scholars in econometrics: Van Damme (1999, p. 35) concludes that some 1000 journals are not listed by ISI and yet may be relevant for Dutch authors. Finally, books may form basic knowledge in a discipline (also see Van Damme 1999, pp. 40±42). Holsapple, Johnson, Manakyan and Tanner (1993, p. 234) quanti®es the relative number of citations for Information Systems: journals 53.7%, books 34.8%, and proceedings 11.5% (for Arti®cial Intelligence; Cheng, Holsapple & Lee (1996) give the following percentages for 1993: 43, 24, 26). To rank proceedings and journals not listed by ISI Ð given time and personnel constraints Ð we use a sampling approach: we take a sample of articles, collect their reference lists, and count how many times a speci®c journal (e.g., MIS Quarterly (MISQ )), a yearly proceedings (e.g., VLDB ), or book publisher (e.g., Springer) is cited. Table 1 gives an example showing the number of references by citing (exporting) articles in journal or proceedings to cited (importing) journals, proceedings, or book publishers. This example illustrates that some journals may be autarkic (see the row denoted A[1]), nearly autarkic (A[2]), or diversi®ed (A[3]). Autarky is related to `inbreeding': authors refer relatively often to articles in the journal or proceedings that publishes their papers. Therefore we sample articles from those journals Ð ISI and non-ISI Ð and proceedings that we think to be relevant for the ®eld studied. Note that Holsapple et al. (1993) uses a sample of ®ve journals only, when evaluating journals, proceedings, and books. Our methodology implies sampling error. To quantify this error we use bootstrapping or resampling (described in Section 3), which gives con®dence intervals. Next we use these intervals to cluster journals and proceedings into a number of homogeneous quality

Table 1 Number of references by citing article A in journal or proceedings to cited journal J, proceedings P, or book publisher B: an example

Journal A[1] A[2] A[3] Proceedings A[4] A[5]

J[1]

J[2]

J[3]

J[4]

P[1]

P[2]

100 7 12

95 9

51

2

3

1

6 13

5 9

1 3

2 3

15 4

3 33

P[3]

5

B[1]

B[2]

2 4

1 2

1 2

1

B[3]

Total

3

100 105 117 33 85

J.P.C. Kleijnen, W. Van Groenendaal / Information Processing and Management 36 (2000) 551±570

553

classes. We cluster book publishers separately (collecting references to books hardly requires any extra e€ort, so we collect that information; however, we do not use books as citing publications). We illustrate our methodology by a case study: we apply the methodology to the ®eld of Information Systems Ð also known as management information systems (MIS) or business computing. We will de®ne this ®eld by specifying which set of journals and proceedings to sample (see Section 3). We selected this particular ®eld, because our School of Management and Economics Ð including our Department of Information Systems Ð wishes to evaluate its faculty members in an objective way. Our school's approach is expected to be followed by all other Dutch schools. Obviously, similar problems must be addressed by research institutes and funding agencies all over the world! Before we started, our school had already decided that an important measure is the ISI impact factor (besides other important factors, such as software developed, and student evaluations of teacher performance). ISI (1993, pp. 10±11) de®nes this factor for journal j in year t as the ratio of: (i) the total number of citations received by all the articles published in this journal j in the preceding 2 years t-1 or t-2, from all journals (including j ) that are included in the ISI data bases; and (ii) the total number of articles in that same journal j in the same time window [t-1, t-2]. What does this de®nition imply? This de®nition implies that general journals may have a higher impact factor than specialist journals. Similar idiosyncrasies hold for proceedings and books. Likewise, survey articles receive more citations than technical articles. Further, the denominator corrects for the size of journals (Cheng et al. (1996) also correct for size). Obviously, the impact factor varies with time t. Finally, ISI uses a time window of 2 years, but ISI (1993, p. 6) itself states: ``In some ®elds, 5-year impact factors may be more appropriate''. Indeed, we claim that a 2-year window is too small for many ®elds: many top journals have long publication delays. Therefore we will focus on results for an in®nite time window: we count all references to articles published in a particular journal (e.g., MISQ ) Ð whatever the publication date of that cited article is (these references are collected from the reference list at the end of the citing article sampled from the most recent issue of a journal or proceedings). More comments are collected in Kleijnen and Van Groenendaal (1999). In summary, our approach is inspired by ISI's impact factors: we use citations. We, however, also measure: (1) references from and to proceedings; (2) references from all journals that we think are relevant for a certain ®eld; and (3) references to books. Unlike ISI, we do not correct for the size of journals etc. So our measure is simpler (we hope that our results are still valid; see Section 4), requires less time (sampling only, no data collected on the size of journals, proceedings, and book series), and so it is cheaper. We focus on an unlimited time window (smaller time windows give smaller citation numbers that are more erratic). The remainder of this article is organized as follows. Section 2 summarizes the literature on measuring the quality of publications. Section 3 presents our methodology, consisting of sampling, bootstrapping, and classi®cation. Section 4 presents a case study, applying our methodology to the Information Systems ®eld. Section 5 gives conclusions.

554

J.P.C. Kleijnen, W. Van Groenendaal / Information Processing and Management 36 (2000) 551±570

2. Literature on measuring quality of publications The search for the `holy grail' of measurable quality is not new. For example, Holsapple et al. (1993) lists seven studies Ð published between 1982 and 1991 Ð on the quality of Information Systems publications. We add: Anbar (1997), Cheng et al. (1996), Hardgrave and Walstrom (1997), Nord and Nord (1995), Robey, Walstrom, Adams and Swanson (1998), and Walstrom, Hardgrave and Wilson (1995). The journals in many other research areas have been classi®ed also; for example, Holsapple et al. (1995) classify journals on Decision Support Systems, and Tahai and Rigsby (1998) review and classify the ®eld of Accounting. In general, this discussion (on the measurement of the quality of scienti®c publications and research) has intensi®ed over these last years, as witnessed by articles in the scienti®c and the popular press; see Van Damme (1999). Various reasons for identifying the `best' publication outlets are discussed in Holsapple et al. (1993), Walstrom et al. (1995), and Tahai and Rigsby (1998); as examples we have already mentioned decisions on tenure and funding. In both the literature and practice, ISI's impact factors play an important role. ISI has three data bases, which together cover 6000 journals from 60 countries! The impact factors are computed from all three data bases. In our case-study, however, we will use only two ISI data bases (and some more sources), namely the SSCI and the SCI, because the third data base does not seem to contain journals that refer to Information Systems publications. The quality of journals might be evaluated through peer review. We will indeed use peer review to validate the outcomes of our case study; that is, we will use the list of the top-®ve conferences in Hardgrave and Walstrom (1997). (Other publications mentioning peer review are Cooper, Blair and Pao (1993), Nord and Nord (1995), and Robey et al. (1998).) If publications are to be classi®ed, the next question is: how many classes should be distinguished? For example, Holsapple et al. (1993, pp. 238±242) and Nord and Nord (1995) distinguish only two quality classes, whereas the Dutch universities distinguish ®ve classes. We, however, do not apriori decide on the number of classes: our methodology automatically yields a number of homogeneous classes (see Section 3.3). Note that it makes no sense to test the correlation between the rankings resulting from our procedure and from the ISI impact values: in our case-study we collect citations from Information Systems publications only, whereas ISI collects citations from all ®elds. 3. Methodology: sampling, bootstrapping, and classi®cation We discuss the following three steps of our methodology: sampling (Section 3.1), bootstrapping (Section 3.2), and classi®cation (Section 3.3). 3.1. Sampling An important practical problem in our sampling approach is: how to de®ne the population of citing publications (also see the `base set' in Cheng et al. (1996))? For our Information Systems case-study we use the following sources. ISI distinguishes subject categories or (sub)®elds. Within the category `Computer science' we

J.P.C. Kleijnen, W. Van Groenendaal / Information Processing and Management 36 (2000) 551±570

555

consider the following two subcategories to be relevant: Information Systems and Arti®cial Intelligence. So we exclude ®ve subcategories: Cybernetics, Hardware and Architecture, Interdisciplinary Applications, Software Graphics Programming, and Theory and Methods. We also exclude categories such as Information Science and Library Science and Telecommunications, because these categories do not seem to contain journals that refer to Information Systems publications. Note that impact factors vary drastically over subcategories (and categories): for example, in 1996 the highest impact factor was 1.777 for Information Systems, whereas it was 2.654 for Arti®cial Intelligence. Hence, scientists working on the `borderline' of a subdiscipline, may `shop around' among subdisciplines. Because the SCI and SSCI do not include all Information Systems journals that are judged to be important by Information Systems experts, we also use the list of journals in Holsapple et al. (1993, pp. 236±237). Altogether, our journal population to be sampled consists of 170 Information Systems journals; that list is given in Kleijnen and Van Groenendaal (1999). Because the SCI and SSCI contain both high quality and low quality citing journals, we sample each journal in our journal population, except for the following. To save time, we do not sample from those journals that we expect to be cited very rarely in Information Systems. An example is Cognitive Brain Research (CBR ). If we kept CBR in our sample, we would ®nd out that the publications that it references will end at the bottom of our ranked list of Information Systems publications. But we are not interested in such low quality publications on Information Systems, even though these publications may be high quality publications in brain research. Some other journals are eliminated because they ceased to exist. Furthermore, some journals have no references at all. Finally, some journals we could simply not locate. Altogether we sample 123 journal articles. Publications do exhibit particular citation patterns (see Table 4, further discussed in Section 4). ISI computes its impact factors from both high quality and low quality citing journals. Therefore we decide to sample articles Ð not only from ISI and non-ISI journals Ð but also from proceedings. As our proceedings population we use the 50 proceedings that are cited most often by the 123 sampled journal articles. Moreover, we add nine proceedings listed by expert colleagues. We sample two articles Ð instead of one Ð from each proceedings, for two reasons: (i) articles in proceedings tend to have fewer references (see Section 4); and (ii) we have fewer proceedings than journals (59 vs 170). As with journals, we had some practical problems when trying to obtain speci®c proceedings. These problems implied that we did not sample 59 proceedings, but only 41 (resulting in 82 citing proceedings articles). 3.2. Bootstrapping We wish to estimate the accuracy of our sample results. We measure accuracy through 1-a con®dence intervals for the number of citations per journal, proceedings, or book publisher; a denotes the type I error probability per interval. Such intervals are provided by bootstrapping, without assuming a speci®c (say, Gaussian) distribution. So bootstrapping is a very attractive technique; Efron and Tibshirani (1993, pp. 115, 383), however, warn: ``bootstrapping is not a

556

J.P.C. Kleijnen, W. Van Groenendaal / Information Processing and Management 36 (2000) 551±570

uniquely de®ned concept . . . alternative bootstrap methods may coexist''. For our situation we interpret bootstrapping as follows. The basic idea of bootstrapping is: what happens if a speci®c citing (not cited) article is `forgotten'; what if this article is counted twice; what if it is counted three times; and so on. In other words, what happens if citing article 1 gets a weight of zero, whereas article 2 gets a weight of two, three, etc.? The total sample size, however, is kept ®xed. This `resampling' technique and its implementation using pseudorandom numbers is further explained in Kleijnen and Van Groenendaal (1999). Bootstrapping gives estimated density functions (EDFs, cumulative frequency distributions) for the total number of times a particular journal, proceedings, or book publisher is cited in our sample. Fig. 1 gives an example: it displays the EDFs of the class-1 journals and proceedings, plus the EDF of the best publication in class 2 (namely, Administrative Science Quarterly or ASQ ). Bootstrapping suggests the following natural way to account for the fact that we sample only one article per journal, whereas we sample two articles per proceedings (which biases the results because of citation patters): when bootstrapping we count the citing journal articles twice! Such bootstrapping implies that the total sample size increases from 205 (=123  1+41  2) to 328 (=123  2+41  2).

3.3. Classi®cation The bootstrap EDFs give 1-a con®dence intervals, which we use to cluster the individual journals and proceedings (and book publishers Ð see below) into classes. We develop the following heuristic. All members of a class should be able to compete with the leader of that class. We start with the list of cited journals and proceedings sorted from most often cited to least often cited; Table 2 shows this for the case study (further discussed in Section 4). We compute the a quantile of each EDF; this quantile estimates the lower limit of the (one-sided) 1-a con®dence interval. We ®nd which cited journal or proceedings has the highest a quantile. In the case study, a=0.1 gives as leader of the top class the journal Communications of the ACM (CACM ). Class leaders are printed in bold in Table 2. We place the next best journal or proceedings in the same class if its upper value of the 1-a con®dence interval exceeds the lower value of the 1-a con®dence interval for the class leader. Fig. 1 shows that CACM's estimated 0.1 quantile is circa 100 and Management Science's estimated 0.9 quantile is circa 160, so both journals belong to the same class. We proceed in this way, until a journal or proceedings does not pass this test: ASQ's estimated 0.9 quantile is circa 70 (<100) so ASQ is the leader of the next lower class Ð see Table 2, class 2. Obviously, we can decrease the number of classes by decreasing the value of a; for example replacing 0.1 by 0.05 and 0.9 by 0.95 gives ®ve classes instead of six classes. We also classify book publishers, in a similar way; see Table 3. Let us consider the case study in some detail.

Fig. 1. Cumulative distribution of top journals and proceedings.

J.P.C. Kleijnen, W. Van Groenendaal / Information Processing and Management 36 (2000) 551±570

557

558

J.P.C. Kleijnen, W. Van Groenendaal / Information Processing and Management 36 (2000) 551±570

Table 2 Journals and proceedings ranked on number of citations received in sample of 123 journal and 82 proceedings articles (bold denotes class leader when a=0.1; only top-four classes displayed) Median Class 1 MIS Quarterly Management Science Communications of the ACM Proceedings of the International Conference on Very Large Databases (VLDB) Proceedings of the ACM Conference on Management of Data (SIGMOD) Class 2 Administrative Science Quarterly Arti®cial Intelligence Journal of Management Information Systems Journal of Personality and Social Psychology Harvard Business Review European Journal of Operational Research Journal of Accounting Research Proceedings of the National Conference on Arti®cial Intelligence (AAAI) Academy of Management Review Journal of the ACM IEEE Transactions on Software Engineering Econometrica IEEE Transactions on Pattern Analysis and Machine Intelligence Proceedings of the ACM Symposium on Principles of Database Systems (PODS) Accounting, Organizations, and Society Academy of Management Journal Information and Computation IEEE Computer Information Systems Research Journal of Computer and Systems Science Accounting Review, The ACM Transactions on Database Systems International Journal of Human-Computer Studies IEEE Transactions on Computers Decision Sciences Proceedings of the ACM Symposium on Principles of Programming Languages (POPL) Human Relations Journal of the American Statistical Association Pattern Recognition Review of Economic Studies Memory and Cognition Class 3 ACM Computing Surveys Proceedings of the International Joint Conference on Arti®cial Intelligence (IJCAI) Psychological Review IEEE Transactions on Knowledge and Data Engineering

135 132 123 87 87 63 55 47 47 45 44 44 44 43 42 42 42 41 41 40 39 38 36 36 35 35 32 32 31 31 31 30 30 27 26 24 34 34 32 31 (continued on next page)

J.P.C. Kleijnen, W. Van Groenendaal / Information Processing and Management 36 (2000) 551±570

559

Table 2 (continued ) Median Information and Management ACM SIGMOD Record IEEE Transactions on Systems, Man and Cybernetics Proceedings of the ACM Symposium on the Theory of Computing (STOC) Proceedings of the IEEE International Conference on Data Engineering (ICDE) Proceedings of the International Conference on Logic Programming (ICLP) Proceedings of the International Conference on Information Systems (ICIS) Journal of Applied Psychology Psychological Bulletin Theoretical Computer Science Journal of Experimental Psychology: Learning, Memory and Cognition IEEE Software Machine Learning Neural Networks Proceedings of the Workshop on Case-Based Reasoning (DARPA) Journal of Automated Reasoning Sloan Management Review American Psychologist Organizational Behavior and Human Decision Processes Cognitive Science Cognitive Psychology Proceedings of the International Symposium on Logic Programming (ILPS) Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Operations Research Journal of the American Society for Information Science (ASIS) Journal of the Operational Research Society Organization Science Proceedings of the IEEE Symposium on Logic in Computer Science (LICS) Proceedings of the International Conference on Information and Communication Technologies in Tourism Journal of Logic Programming Research in Organizational Behavior Journal of Economic Theory Proceedings of the IEEE International Conference on Robotics and Automation Proceedings of the USENIX Conference Information Systems Vision Research AI Magazine American Journal of Psychology Proceedings of the ACM Symposium on Principles of Distributed Computing SIAM Journal on Computing Annals of Mathematics and Arti®cial Intelligence Biometrika Cognition Journal of Consumer Research Proceedings of the ACM Conference on Research and Development on Information Retrieval (SIGIR)

31 30 30 30 30 29 29 28 27 26 26 24 24 24 24 22 22 22 22 21 21 20 20 19 19 19 19 19 19 18 18 18 18 18 17 17 16 16 16 15 15 15 15 15 15 (continued on next page)

560

J.P.C. Kleijnen, W. Van Groenendaal / Information Processing and Management 36 (2000) 551±570

Table 2 (continued ) Median Class 4 Interfaces ACM Transactions on Information Systems Journal of Management Proceedings of the IEEE Symposium on Foundations of Computer Science (FOCS) Proceedings of the IEEE Information Processing and Management International Journal of Production Research Proceedings of the International Conference on Machine Learning (ICML) Annual Review of Psychology Personnel Psychology Journal of Marketing Neural Computation Proceedings of the International Symposium on Computer Architecture Proceedings of the IEEE International Computer Conference (COMPCON) Proceedings of the International Conference on Principles of Knowledge Representation and Reasoning Proceedings of the IEEE Real-Time Systems Symposium American Economic Review International Journal of Distributed and Parallel Databases Pattern Recognition Letters Philosophical Transactions of the Royal Society of London IEEE Transactions on Engineering Management American Sociological Review Journal of Business Journal of Accounting and Economics Journal of Educational Psychology Human Factors Journal of Marketing Research New Generation Computing International Journal of Clinical and Experimental Hypnosis Proceedings of the European Conference on Arti®cial Intelligence (ECAI) Proceedings of EFOC Proceedings of the American Educational Research Association Proceedings of the Microprogramming Workshop (MICRO) Nature ACM Transactions on Programming Languages and Systems International Journal of Computer Vision ACM SIGSOFT Software Engineering Notes Science of Computer Programming Decision Support Systems Journal of Experimental Social Psychology Journal of Organizational Behavior Long Range Planning IEEE Transactions on Robotics and Automation Journal of Law and Economics

17 16 16 16 16 15 15 15 14 14 14 14 14 14 14 14 13 13 13 12 12 12 12 12 12 12 12 12 12 12 12 12 12 11 11 11 11 11 11 11 11 11 11 11 (continued on next page)

J.P.C. Kleijnen, W. Van Groenendaal / Information Processing and Management 36 (2000) 551±570

561

Table 2 (continued ) Median Journal of Finance 11 Journal of Management Studies 11 Journal of Information Systems 11 Proceedings of the Conference on Automated Deduction (CADE) 11 Proceedings of the ACM Conference on OO Programming Systems, Languages, and Applications 11 (OOPSLA) Journal of Sound and Vibrations 10 Information Processing Letters 10 BIT 10 Information Systems Management 10 CVGIP: Image Understanding 10 Data Base 10 IEEE Expert 10 Computer Vision, Graphics, and Image Processing 10 Behaviour and Information Technology 10 Quarterly Journal of Economics, The 10 California Management Review 10 Organization Studies 10 Journal of Management Accounting Research 10 Information Week 10 Advances in Consumer Research 10 Journal of Public Policy and Marketing 10 Personality and Individual Di€erences 10 Arti®cial Intelligence in Engineering 10 Transactions IECE Japan 10 Proceedings of the Royal Society of London 10 Proceedings of the International Conference on Computer Vision (ICCV) 10 Proceedings of the Australasian Database Conference 10 Proceedings of the International Conference on Measurement and Modelling of Comp. Syst. 10 (SIGMETRICS) Proceedings of the International Workshop on Infrastructure for Temporal Databases 10 IEEE Transactions on Communications 9 Journal of the Royal Statistical Society 9 IEEE Transactions on Information Theory 9 Organizational Dynamics 9 VLDB Journal 9 Scienti®c American 9 Proceedings of the International Conference on Advanced Information Systems Engineering 9 (CAiSE) Proceedings of the International Conference on Architectural Support for Progr. Languages 9 and Op. Sys Proceedings of the International Workshop on Logic Programming and Non-Monotonic 9 Reasoning (LPNMR) Academy of Management Best Paper Proceedings 9 Proceedings of the Hawaii International Conference on System Sciences (HICSS) 9 Proceedings of the Annual Conference of the Cognitive Science Society 9 Astronomical Journal 8 (continued on next page)

562

J.P.C. Kleijnen, W. Van Groenendaal / Information Processing and Management 36 (2000) 551±570

Table 2 (continued ) Median ACM Transactions on Computer Systems Computer Networks and ISDN Systems Discrete and Computational Geometry Science Journal of the Optical Society of America Proceedings of the International World Wide Web Conference Business Week Quarterly Journal of Experimental Psychology, The USSR Computing Mathematics and Mathematical Physics Omega Information Systems and Operational Research (INFOR) Research Strategies Software Engineering Journal Journal of Consumer A€airs Statistics in Medicine Methods of Information in Medicine Proceedings of the ACM Symposium on Discrete Algorithms (SODA) Proceedings of the IEEE Symposium on Visual Languages Proceedings of the IEEE Workshop on Visual Languages Proceedings of the ACM Symposium of Computational Geometry Proceedings of the International Workshop on Advanced Visual Interfaces Proceedings of the International Conference on Cooperative Multimodal Communication ABA Banking Journal AIIE Transactions Annals of Statistics International Journal of Computational Geometry & Applications Lisp and Symbolic Computation Journal of Systems and Software Behavioral and Brain Sciences, The Annals of Tourism Research Biological Cybernetics Computer Aided Design Journal of Real-Time Systems CSVA Acta Technica Proceedings of the Text Retrieval Conference (TREC) International Journal of Operational Research ASCE Journal of Water Resources Planning and Management Engineering Economist, The Journal of Consumer Policy Computers and Chemical Engineering Journal of Sport Psychology Journal of Supercomputing Proceedings of the European Workshop on Learning Robots Proceedings of the Workshop on Functional Programming Proceedings of the International Conference on Genetic Algorithms

8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 7 7 7 7 7 7 7 7 7 6 6 6 6 6 6 6 6 6 6 6 6 6 6

J.P.C. Kleijnen, W. Van Groenendaal / Information Processing and Management 36 (2000) 551±570

563

Table 3 Book publishers ranked on number of citations received in sample of 123 journal and 82 proceedings articles (bold denotes class leader when a=0.1; top-six classes displayed) Median Class 1 Springer-Verlag Wiley Addison-Wesley Class 2 Prentice Hall MIT Press Cambridge University Press Class 3 Morgan Kaufmann Publishers Sage Publications Academic Press McGraw Hill Elsevier Lawrence Erlbaum Associates Class 4 Oxford University Press North-Holland MacMillan Kluwer Jossey-Bass Free Press Benjamin-Cummings JAI Press Houghton Mi‚in Class 5 University of California Press Heinemann Harvard Business School Press University of Chicago Press Freeman Blackwell, Basil Harvard University Press Routledge Harper and Row Pitman AAAI Press Plenum Press Irwin Princeton University Press Tavistock Van Nostrand Reinhold Co. Random House Rand-McNally Guilford Press

169 143 131 101 83 76 57 58 55 51 39 39 34 31 31 29 27 31 25 24 21 19 18 19 20 16 18 17 17 16 15 14 11 14 12 12 12 12 12 12 (continued on next page)

564

J.P.C. Kleijnen, W. Van Groenendaal / Information Processing and Management 36 (2000) 551±570

Table 3 (continued ) Median Microsoft Press Stanford University Press Class 6 Simon & Schuster Norton Computer Science Press World Scienti®c Publishing Greenwood Press Basic Books University of Michigan Gorcum, Van Taylor & Francis Clarendon Press Pergamon Press Airlines Electronic Engineering Committee (AEEC) GMD National Research Center for Information Technology Praeger Goodyear Ellis Horwood Little Brown & Co. ACM Press Kent Publishing Co. Harvester American Psychological Association Ablex Dekker, Marcel Yale University Press VDI-Verlag Hogarth Knopf ILR Press IOS Press IEEE Computer Society Press Penguin Ballinger Longman Putnam Lexington Books University of Waterloo National Institute of Standards and Technology Holt, Rinehart and Wilson Scott, Foresman and Co. Batsford, B.T. Doubleday Manchester Business School Harcourt Brace (Javanovich) Century VII Publishing Company

6 10 10 9 9 10 8 6 7 8 8 7 7 4 4 6 6 5 6 5 6 6 6 6 4 6 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 (continued on next page)

J.P.C. Kleijnen, W. Van Groenendaal / Information Processing and Management 36 (2000) 551±570

565

Table 3 (continued ) Median Gower West Publishing Cornell University Press Harper Collins Nijho€, Martinus Methuen Chapman and Hall Research Psychologists Press Mouton Appleton-Century-Crofts In®x

4 4 4 4 4 3 4 4 4 4 4

4. Case study: an information systems application of the methodology Our case study's sample gives many data on citing and cited publications. These data we store in a data base (using the Access software). This yields the following information. We have 123 citing journal articles and 82 proceedings articles. These citing articles refer to 6901 publications, namely 3128 journal articles, 1532 proceedings articles, 1577 books, and 664 other publications (such as working papers). Altogether we cluster the cited journals and proceedings into six classes (using a=0.1). To save space, Table 2 lists only the four highest classes. The lowest class (class 6) includes journals and proceedings that receive no citations in our sample; the other journals and proceedings in this class do receive citations, but so few that their 1 - a con®dence intervals cover the value zero. How valid are our results? Table 2 gives rankings for journals that seem to have good face validity. Clearly there are three top journals: MISQ, Management Science, and CACM. The ®rst and last journal also feature among the top-®ve US journals selected by Holsapple et al. (1993, p. 234). Management Science is not in this top-®ve; we think the explanation is that this journal is not restricted to Information Systems. All three journals feature among the top journals in Cooper et al. (1993). Some colleagues (including a referee) commented that CACM has changed character in recent years: instead of publishing detailed research papers, it now publishes short `lightweight' articles. We conclude that periodically updating quality measurements is necessary! Further, the well-known journal Information Systems Research (ISR ) is found in class 2 only, whereas many colleagues seem to rank ISR higher than MISQ. Is ISR overrated by some researchers, or does our methodology give wrong classi®cations for some journals (also see Section 5, Conclusions)? Table 2 also gives two top proceedings, VLDB and SIGMOD. These two are not among the top-®ve proceedings in Hardgrave and Walstrom (1997, p.123). One explanation may be that those authors use a questionnaire, not citations. Another explanation may be that VLDB and SIGMOD report on generalist conferences, not on meetings focused on information systems. Anyhow, it is interesting to see that there are proceedings that can compete with top journals! Hardgrave and Walstrom's top-®ve proceedings further include ICIS, which we ®nd in our

566

J.P.C. Kleijnen, W. Van Groenendaal / Information Processing and Management 36 (2000) 551±570

class 3 (that class has 49 members). Their top proceedings HICSS features in our class 4 (132 members). Their top proceedings IFIP turns up in our class 5 (225 members). Their top proceedings DSS and DSI we cannot ®nd in our table! The explanation may be that either these two proceedings are never cited in our sample or their names are misspelled; also see class 6 with its 621 members plus all members not explicitly mentioned because they were never mentioned. (Furthermore, Dutch colleagues provided us with a list of 17 important proceedings; all except one occur in our list: two in class 1, none in class 2, four in class 3, two in class 4, ®ve in class 5, and three in class 6.) We conclude that altogether we classify 1063 journals and proceedings, and that unfortunately our ranking of proceedings seems to have less face validity! Causes may be: sampling error (our sample size of 205 is too small), bias (our citing population is not representative for Information Systems), or generalist vs specialist proceedings. It is interesting that proceedings occur in all quality classes, including class 1. However, in the top classes most publications are journals, not proceedings. Our interpretation is: articles in proceedings do have impact, but they should be considered as preliminary publications that should be followed by publication in journals. (Of course, our ranking is based on an in®nite time window, no correction for size of cited publication, particular de®nition of the population of citing Information Systems journals and proceedings.) What is the validity of Table 3's results for book publishers? The top-three publishers turn out to be Springer, Wiley, and Addison-Wesley. These publishers are indeed well known in Information Systems. Springer's top position may be explained by the many Lecture Notes and books reporting on conferences that are not organized yearly. Note that we list Elsevier and North-Holland as two separate publishers (see classes 3 and 4), whereas these publishers are parts of a single company (Reed±Elsevier). In general, it is not always clear how to de®ne a particular book publisher (Wiley US vs Wiley UK?), whereas it is very clear how to de®ne a speci®c journal (MISQ ). Therefore Table 3 should be interpreted with care. Querying our data base gives some interesting results. Table 4 lists the relative number of references by (citing) articles in journals and proceedings respectively to the four types of (cited) publications distinguished. This table shows that journals refer more to journals than proceedings do: 48.7% vs 33.5%. Yet, proceedings refer (slightly) more often to journals than to proceedings: 33.5% vs 32.0%. (We also test statistically whether the two distributions of references by journals and proceedings to journals, proceedings, books, and others are equal; we ®nd a highly signi®cant di€erence; see Kleijnen and Van Groenendaal (1999).) Further querying shows that the median number of references is 25 in our sampled journals;

Table 4 Citation patterns of journals and proceedings References from/to

Journal (%)

Proceedings (%)

Book (%)

Remainder (%)

Journal Proceedings

48.7 33.5

19.4 32.0

23.4 20.9

8.5 13.5

Fig. 2. Detecting outliers.

J.P.C. Kleijnen, W. Van Groenendaal / Information Processing and Management 36 (2000) 551±570

567

568

J.P.C. Kleijnen, W. Van Groenendaal / Information Processing and Management 36 (2000) 551±570

it is 16 for proceedings. The actual numbers vary between 1 and 125 for journals, and between 1 and 49 for proceedings. The overall median is 19. Another query shows whether a particular proceedings refers mainly to journals or not; for example, HICSS articles cite many journals. Also, the true impact of publications reaches its peak after more than 2 years (ISI's time window is only two). A closer look at Table 2 shows that Ð rather surprisingly Ð Econometrica turns up in class 2. This journal is a prestigious journal in the ®eld of econometrics, but it does not publish articles on Information Systems! Querying our database shows that all 42 (=2  21) references that Econometrica originally receives in our sample, are received from a single (citing) article (namely, an article sampled from the Journal of Group Decision and Negotiation, JGDN ). Fig. 2 shows that Econometrica's EDF is clearly less smooth than the EDFs of many other journals. IEEE Computer has the same number of citations (namely, 42), but this number results from fourteen (instead of a single) sampled citing publications. Consequently, IEEE Computer has a much smoother EDF. Likewise, the Journal of the ACM (JACM ) has the same median as Econometrica (namely, 42; see last column in Table 2), but JACM is cited by twelve sampled articles. In other words, outliers are easily identi®ed by `ragged' EDFs. In general, outliers in statistics need special handling. In our case, we may decide to scrutinize any cited journal that receives all its citations from a single sampled article. Fig. 3 illustrates another interesting characteristic of our methodology: three journals receive the same number of citations (namely 16), and yet they are placed in di€erent classes. Indeed, ACM Transactions on Information Systems and the Journal of Management are in class 4, whereas the American Journal of Psychology (AJP) falls into class 3, which is led by Proceedings of IJCAI. However, since AJP is an outlier, we should not let our methodology routinely decide on this journal!

5. Conclusions Measuring the quality of publications is an important issue in practice, and it is a methodological challenge! We propose a novel methodology that samples articles from journals and proceedings, followed by bootstrapping to obtain con®dence intervals and by a clustering heuristic to form quality classes. This methodology gives ranked lists of journals (ISI and nonISI) and proceedings, and book publishers respectively. For example, in our Information Systems case-study, class-1 turns out to consist of three journals Ð MISQ, Management Science, and CACM Ð and two proceedings Ð VLDB and SIGMOD. More work and research are needed to improve our results and methodology: smaller sampling error (increase current sample size of 205), smaller time windows (now in®nite), correction for size of cited publication medium (now not used), revised de®nition of citing journals and proceedings relevant for Information Systems (e.g., include ISI's `Telecommunications' category and `Interdisciplinary Applications' subcategory), addition of books as sources of citations, inclusion of `non-English' journals, etc. We further hope that our methodology will be applied by others who have the resources needed to measure the quality of publications. Such measuring is a task for an institute, not an individual (to collect and store data we used the services of two research-assistants during

Fig. 3. Classifying outliers.

J.P.C. Kleijnen, W. Van Groenendaal / Information Processing and Management 36 (2000) 551±570

569

570

J.P.C. Kleijnen, W. Van Groenendaal / Information Processing and Management 36 (2000) 551±570

approximately 50 working days). Our methodology may also be applied to other ®elds: Accounting, Finance, Marketing, Simulation, etc. (a good starting point is the following web page with approximately 150 OR journals: http://www.informs.org/Biblio/ACI.html). Obviously, the empirical results of our case study are not ®nal (actually, di€erent methods give di€erent lists; see Cheng et al. 1996 for examples). Nevertheless, we hope that our results and methodology are accepted as better than any alternative currently available! Acknowledgements We thank an anonymous referee and our colleagues Drs. Eric van Damme (Tilburg) and Eric van Heck (Rotterdam) for their comments on an earlier version of this article, which prompted us to clarify a number of issues. References Anbar Management Magazine. (1997). no. 2 (http://www.anbar.co.uk/anbar/excellence/league.htm). Cheng, C. H., Holsapple, C. W., & Lee, A. (1996). Citation-based journal rankings for AI research; a business perspective. AI Magazine, 17,(2),87±97. Cooper, R. B., Blair, D., & Pao, M. (1993). Communicating MIS research: a citation study of journal in¯uence. Information Processing & Management, 29(1), 113±127. Efron, B., & Tibshirani, R. J. (1993). Introduction to the bootstrap. New York: Chapman & Hall. Hardgrave, B. C., & Walstrom, K. A. (1997). Forums for MIS scholars. Communications of the ACM, 40(11), 119± 124. Holsapple, C. W., Johnson, L., Manakyan, H., & Tanner, J. (1993). A citation analysis of business computing research journals. Information & Management, 25(5), 231±244. ISI. (1993). Journal citation reports. Science Citation Index, Institute for Scienti®c Information. Kleijnen, J. P. C., & Van Groenendaal, W. (1999). Measuring the quality of publications: new methodology and results (unabridged version). Discussion Paper, CentER, Tilburg University. Nord, J. H., & Nord, G. D. (1995). MIS research: journal status assessment and analysis. Information & Management, 29, 29±42. Robey, D., Walstrom, K. A., Adams, D., & Swanson, B. (1998). Target journals for information systems research: current practices and controversies. In R. Hirschheim, M. Newman, & J. L. DeGross, Proceedings of the 19th International Conference on Information Systems (ICIS), Helsinki, Finland (pp. 420±424). Tahai, A., & Rigsby, J. T. (1998). Information processing using citations to investigate journal in¯uence in accounting. Information Processing & Management, 34(2/3), 341±359. Van Damme, E. (1999). Towards a new system of output measurement in Dutch economics. Commissie Herziening Outputmeting Economie, CentER, Tilburg University. Walstrom, K. A., Hardgrave, B. C., & Wilson, R. L. (1995). Forums for Management Information Systems scholars. Communications of the ACM, 38(3), 93±107.