- Email: [email protected]
A methodological issue concerning the use of Social Sciences Citation Index Journal Citation Reports impact factor data for journal ranking
Library Acquisitions: Practice & Theory, Vol. 18, No. 4, pp. 447-458, 1994 Copyright 0 1994 Elsevier Science Ltd Printed in the USA. All rights reserved 0364-6408/94 $6.00 + .OO
Pergamon
0364-6408(94)ooo30-1
A METHODOLOGICAL ISSUE CONCERNING THE USE OF SOCIAL SCIENCES CITATION INDEX JOURNAL CITATION REPORTS IMPACT FACTOR DATA FOR JOURNAL RANKING THOMAS E. NISONGER Associate
Professor
Indiana School
of Library
University and Information
Bloomington, E-mail:
Science
IN 47405
[email protected]
Abstract- Following a brief introduction of citation-based journal rankings as potential serials management tools, the most frequently used citation measureimpact factor - is explained. This paper then demonstrates a methodological bias inherent in averaging Social Sciences Citation Index Journal Citation Reports (SSCI JCR) impact factor data from two or more consecutive years. A possible method for correcting the bias, termed adjusted impact factor, is proposed. For illustration, a set of political science journals is ranked according to three different methods (crude averaging, weighted averaging, and adjusted impact factor) for combining SSCI JCR impact factor data from successive years. Although the correlations among the three methods are quite high, one can observe noteworthy differences in the rankings that could impact on collection development decisions. Keywords-Journal
ranking, Journal Citation Reports, Impact factor, Political
science journals. INTRODUCTION In the mid-1990s the management of serials has assumed paramount importance for collection development librarians because of the exceedingly high serials inflation rate. The so-called serials crisis has had a direct impact on traditional acquisitions as libraries have shifted funds away from the monographic budget in order to pay for their serials. As countless libraries face serials review and cancellation projects, many librarians are seeking new tools to help them 447
448
T. E. NISONGER
evaluate journals. Journal ranking studies based on Journal Citation Reports data represent useful collection management tools whose full potential is beginning to be appreciated by librarians. Since the publication of Gross and Gross’s famous study of chemistry journals in 1927, journal rankings or ratings have been offered in numerous disciplines [l]. A bibliography of the last decade’s major journal ranking studies is included in Nisonger [2]. In recent years, perception studies based on subjective judgment by experts and citation data, especially from the Institute for Scientific Information (ISI), represent the two most frequently used ranking methods. This article’s focus is on the latter approach. The Institute for Scientific Information publishes three citation indexes covering each of the broad areas of human knowledge: the Social Sciences Citation Index (SSCI), the Science Citation Index (SCZ), and the Arts & Humanities Citation Index. Both the SSCI and the SCZ contain a section called the Journal Citation Reports (JCR) which contains citation data on thousands of journals [3]. The Arts & Humanities Citation Index does not have a JCR as it is the conventional wisdom of collection development that serials are less important in that area. The JCR ranks journals according to five criteria [4]. The two most significant for serials collection management are total citations received and impact factor, a measure to normalize for journal size that will be explained in detail in the next section. Citation analysis is based on the assumption that if an author cites a journal, he or she has found it useful. Thus, it follows that the more frequently a journal is cited, the greater its role in the scholarly communication process. The precise meaning of citation is unclear and beyond this paper’s scope. Nevertheless, a journal’s citation record is indicative of some amorphous positive attribute that is often called quality, value, or impact. Moreover, it offers the advantage of an objective measure. For practical decision-making purposes in collection management, the primary concern is with a journal’s relative ranking within its discipline rather than the raw citation data. A number of previous authors have advocated the use of citation data for serials collection management. Eugene Garfield, the ISI’s founder and an illustrious advocate of citation studies, contends that “. . . citation analysis would appear to be of great potential value in management of library journal collections” [5]. Smith states that “. . . the JCR is a helpful objective tool to augment other methods of journal deselection” [6]. Broadus argues that “. . . one might begin the process of identifying titles for cancellation by reviewing Journal Citation Reports (JCR)” [7], while Dombrowski asserts, “Citation analysis cannot be used as the sole method of journal evaluation, but if it is used in conjunction with other evaluative methods, it can provide a fairly accurate picture of a journal’s value . . .” [8]. Leavy concludes that “. . . judicious use of the [JCR’s] data can benefit any academic library in managing its scholarly periodical collection” [9]. Line has assumed a contrarian position, arguing that the critical decision-making criterion should be a journal’s use in a particular library rather than citation data [lo]. Ranking studies are obviously applicable to journal selection, cancellation, and weeding decisions. While the application to librarianship is evident, it should be noted that journal ranking studies are of considerable interest to scholars in other disciplines. These rankings can provide guidance in article submission decisions and assist in evaluation of a faculty member’s research output. In fact, the vast majority of journal rankings have been conducted by scholars outside the field of library and information science. The author’s unpublished research indicates that, as one might intuitively expect, a journal’s citation data can vary significantly from year to year. Such fluctuation has practical implications. Consequently, many researchers have averaged impact factor data from two or
A Methodological Issue
449
more adjacent years in order to compensate for yearly fluctuation. This paper’s objective is to demonstrate that simply averaging impact factor data, while seemingly a valid procedure, introduces a subtle methodological bias.
IMPACT FACTOR As developed by the ISI, impact factor represents a ratio of citations received to citable items, based on citations made in the current year (by all journals in the IS1 database) to articles published during the previous two years. In essence, it indicates the mean number of times a so-called average article has been cited. A journal’s 1990 impact factor would be calculated by the following formula. 1990 impact factor =
No. of 1990 citations to 1989 + 1988 items No. of citable items published in 1989 + 1988 ’
Impact factor compensates for the advantage that older, larger, or more frequently published journals would enjoy if rankings were based on total citations received. Although somewhat controversial, impact factor has been used for citation-based journal rankings more than any other citation measure, and many researchers have asserted it is the most valid citation measure for journal evaluation purposes. For example, Buffardi and Nichols stress that impact factor “. . . is probably the best single measure of journal quality we have at present” Ill]. Dometrius, while acknowledging inadequacies, terms impact factor “. . . a robust indicator of journal stature” [ 121. Vinkler declares, “There is at present no better indicator applicable in practice characterizing the scientific impact of journals than the impact factors given by Garfield” [Founder of the JCR] [13]. However, a number of methodological and practical difficulties with journal rankings based on impact factor have been pointed out. Obviously, citation data constitute only one of numerous journal evaluation criteria, including subjective judgment, use, indexing, relevance to a library’s mission, and cost. Only a portion of the total journal universe is covered in the Journal Citation Reports (although generally the most important scholarly journals are included) and many librarians do not have access to the JCR in any case. Impact factor is generally recognized to be invalid for cross-disciplinary comparisons, new journals (which have not had time to establish a citation record), and nonresearch journals. Impact factor does not consider citations to articles published more than two years previously. Anecdotal evidence abounds of errors in the IS1 database, and while Ribbe has noted errors in JCR data, he concludes that most impact factors are correct within 5% [14]. Tomer has questioned the significance of impact factor because he found a ranking based on impact factor correlated highly with a ranking of the same journal set according to total citations received [15]. Research by Woodward and Hensman has shown that review journals tend to have higher impact factors [16]. Finally, Scanlan has offered a harsh criticism of impact factor from the perspective of publishers, noting that the type and length of articles and a research field’s size, style, and citation tradition, as well as journal self-citations can influence impact factor [17]. Several scholars have proposed modifications or alternatives to the IS1 impact factor to correct perceived deficiencies. Van Raan and Hartmann developed a measure, termed “comparative impact,” which graphically depicts the citation record for each type of publication in a journal, e.g., letters, editorials, and “normal” articles [18]. To compensate for the fact that JCR rankings are constructed on citations from all disciplines, i.e., every journal covered in
450
T. E. NISONGER
the IS1 database, Hirst and Talent advanced the “computer science impact factor” [19], Hirst the “discipline impact factor” [20], and He and Pao the “discipline influence score” [21], all utilizing the manipulation of IS1 data to rank a discipline’s journals based solely on citations from the discipline itself. Methods for estimating the random fluctuation of impact factor have been developed by Nieuwenhuysen and Rosseau [22], based on Poisson distribution, and by Nieuwenhuysen, using a probabilistic model [23]. Both these advanced statistical techniques will probably not be of much concern to most practicing librarians. Bonitz has suggested a “selective impact” measure that combines a journal’s impact factor and productivity from SD1 (Selective Dissemination of Information) runs [24]. A number of efforts have been undertaken to develop impact measures that allow valid comparisons across subfields and disciplines. For example, Vinkler has proposed an “average subfield impact factor,” based on his analysis of chemistry journals, which indicates the average citation rate for various subfields [25]. Vinkler also introduced the “citation strategy indicator,” which relates a journal’s impact factor to the mean impact factor of other journals in its specialty [26,27]. Schubert and Braun advanced a “relative citation rate,” which relates citations received to citations expected [28]. Sen recently proposed a “normalized impact factor,” which utilizes a multiplier based on the current year’s highest journal impact factor in each subject category covered by the JCR [29]. The preceding survey of methodological issues concerning impact factor, although by no means comprehensive, illustrates that impact factor has been the focus of considerable discussion in the literature. It is obvious that much more effort has been devoted to the problems of cross disciplinary comparison than to using impact factors from a range of years-the subject of this paper. Yet, ongoing controversy continues concerning how impact factor may be utilized most effectively.
PREVIOUS JOURNAL RANKINGS BASED ON AVERAGING IMPACT FACTOR Several major journal ranking or rating studies have averaged SSCZ JCR impact factor data from two or more consecutive years in order to account for yearly fluctuations in the data. Christenson and Sigelman averaged impact factor data for the years 1977, 1978, and 1979 to rate 56 political science and 61 sociology journals in their citation-based ranking study [30] undertaken for the purpose of comparison with the subjective rankings of political science journals by Giles and Wright [3 l] and sociology journals by Glenn [32]. Although Christenson and Sigelman did not offer an explicit ranking, a ranking may be constructed from their ratings. Feingold averaged SSCZ impact factor data for 1985 and 1986 to rank 52 journals in eight subfields of social science psychology: applied, clinical/abnormal, developmental, experimentallearning and memory, experimental-perception, personality, quantitative, and social [33]. (The journals were ranked within each subfield rather than within the 52 total journals.) Colson ranked 35 public administration journals by averaging SSCZ impact factor data for the years 1981 through 1986 [34]. To rank 26 criminology and criminal justice journals, Stack averaged the 1984 and 1985 impact factors [35]. Multiple-year impact factors have also been averaged for purposes other than formal journal rankings. Ribbe used the mean 1983-1985 and mean 1984-1987 Science Citation Index JCR impact factors for sets of 17 and 20 mineralogy, petrology, and geochemistry journals in his study, which demonstrated that federally funded research is more likely to be published in high impact factor journals [36,37].
451
A Methodological Issue
As noted above, researchers have used multi-year data to compensate for yearly fluctuation. To quote Stack, “I used a two-year period to increase the reliability of the index” j38J. Colson observed, “Citation scores are subject to certain annual ~uctuations, so a fair ordering of journal rank requires more than one year of data” [39]. Christenson and Sigelman used three years “To mitigate the possibilities of yearly fluctuations” [40]. These researchers have not explicitly addressed the method they used to average impact factors from different years. More fundamentally, they were apparently unaware that averaging IS1 impact factors from consecutive years, regardless of method, introduces a subtle bias into the results. The next section discusses alternate methods for averaging impact factor. It is followed by a section that explains the bias inherent in averaging impact factor data from adjacent years.
CRUDE VERSUS WEIGHTED
AVERAGING
In “crude” averaging, one simply calculates the mean of a journal’s raw impact factors for the years being combined. For “weighted” averaging, one calculates the ratio of total citations (to the previous two years) to total source items (pub~sh~ during the previous two years) for the years being averaged. Table 1, which summ~izes 1987, 1988, and 1989 SSCf JCR data for the journal World PO&tics, can be used to illustrate the difference. In crude averaging, the three final yearly impact factors as reported in JCR (1.973, 1.625, and 1.900) would simply be averaged, resulting in a mean of 1.833. In weighted averaging, the total of each year’s citations to the previous two years would be divided by the total of the source articles published during the preceding two years, e.g., 18249, obtaining a figure of 1.838. The crude vs. weighted averaging issue is a rather self-evident point that should not surprise anyone versed in statistics. The author believes it should be mentioned because previous researchers have not explicitly stated which averaging method they used, and, as illustrated in this paper, it does make some difference in the results obtained.
BIAS INTRODUCED
BY AVERAGING IMPACT FACTOR
The averaging of impact factor data for consecutive years, regardless of which method is used, introduces a subtle bias. When a range of years is combined, data for the middle years
TABLE 1 SSCI JCR DATA FOR THE JOURNAL World Politics Citations to Last 2 Years
Source Items Last 2 Years
Impact Factor
1988 1989
73 52 57
37 32 30
1.973 1.625 1.900
Total
182
99
Year 1987
452
T. E. NISONGER
will be counted twice, while the extreme years will be counted only once. The following figures illustrate this bias when the impact factors for the years 1990 and 1989 are averaged, using either the crude or weighted method. 1990 1. F. = 1990 citations to 1989 + 1988/1989 + 1988 source items, 1989 I. F. = 1989 citations to 1988 + 1987/1988 + 1987 source items. It is apparent that 1988 is counted twice, while 1989 and 1987 only once. If additional years were added to the range to be averaged, e.g., 1991 and 1987, the extreme years in the preceding illustration (1989 and 1987) would then be counted twice, but a new set of extreme years would develop. This is illustrated below. 1991 I. F. = 1991 citations to 1990 + 1989/1990 + 1989 source items. 1990 I. F. = 1990 citations to 1989 + 1988/1989 + 1988 source items. 1989 I. F. = 1989 citations to 1988 + 1987/1988 + 1987 source items. 1988 I. F. = 1988 citations to 1987 + 1986/1987 + 1986 source items. We now see that the three middle years (1989, 1988, and 1987) are counted twice, while a new set of extreme years (1986 and 1990) are considered only once in the final average figure. No matter how many consecutive years are averaged, there will always be middle years considered twice, and extreme years that are counted only one time. However, averaging a greater number of years will reduce the bias’s influence. One might intuitively suspect this effect would be self-canceling, as it applies equally to all journals in the set. However, as illustrated below, notable differences between alternate rankings (one corrected for this effect, the other uncorrected) are apparent. This would not be an issue if there were no difference between one year of a journal and another. However, both logic and intuition indicate that the quality of a journal’s articles (and thus its citation record) will vary from year to year just as a collegiate sports team will have a different record each year. The question of year-to-year consistency of impact factor data and rankings has not been fully investigated, although a number of researchers, e.g., Buffardi and Nichols [41], Gordon [42], Vlachy [43], and Sievert [44], have briefly addressed the issue. However, their research is only tangentially relevant to the issue under examination here. One would logically expect considerable year-to-year stability between impact factors because citations to the same year would be included in both, i.e., the number of citations to articles published in 1988 would be used in calculating both the 1989 and 1990 impact factors.
ADJUSTED
IMPACT FACTOR
To compensate for this bias, the author proposes an adjustment to the JCR data, termed here adjusted impact factor, to be used when averaging impact factor data from successive years. In this approach, for each year in the range to be averaged, the number of citations made (by all journals in the ISI’s database) that year to the journal’s issues published two years earlier is divided by the number of citable items published two years previously. For exam-
A Methodological
453
Issue
ple, when combining a journal’s 1990 data with those of an adjacent year, one would divide the number of citations made in 1990 to the journal’s 1988 issues by the number of citable items it contained in 1988. The following formula illustrates the averaging of data from the 1989, 1988, and 1987 JCR using the adjusted approach. (1989 tits. to 1987) + (1988 tits. to 1986) + (1987 tits. to 1985) 1987 citable items + 1986 citable items + 1985 citable items ’ In this method, each year is counted only once. No complicated algorithm is necessary, while the data are easily gathered from the JCR. A two-year rather than a one-year period between the citation and the cited item’s publication data is used to allow more time for a journal to be cited.
DEMONSTRATION
WITH SET OF POLITICAL
SCIENCE JOURNALS
To illustrate the influence of these issues on actual journal rankings, a set of 64 political science journals were rated and ranked according to the crude averaging, the weighted averaging, and the adjusted impact factor methods for combining SSCI JCR data for three years. The results are displayed in Table 2. Parenthetically, it should be noted that a rating assigns an absolute evaluative measure to a journal, while a ranking places it in rank order among a journal set. These 64 journals were chosen because they represent the set covered in the SSCI JCR from among the 78 journals included in the most recent major perception-based ranking of political science journals by Giles, Mizell, and Patterson [45,46]. The years 1987, 1988, and 1989 were combined because they most closely correspond to Giles, Mizell, and Patterson’s 1988 survey date. Some of these journals, e.g., American Sociological Review, are outside the discipline of political science, although this point is irrelevant to the issues addressed in this paper. Examination of columns 1 and 2 in Table 2, comparing crude and weighted averaging, reveals that only 14 of the 64 journals (21.9Vo) obtain an identical rating by both methods, thus indicating we are dealing with a genuine issue. Obviously, for practical purposes a journal’s ranking is more significant than its rating. The correlation between rankings based on crude versus weighted averaging is almost identical (r = .9999), although some differences between the rankings produced by the two methods are apparent. Only 46 of the 64 journals (71.9%) are ranked in either the same position or tied for the same position regardless of which averaging method is used. Nevertheless, the rankings of just two titles vary by more than one position. Journal of International Affairs moved from 38th under crude averaging to a tie for 35th with weighted averaging, while Journal of Peace Research dropped from 35th with crude averaging to a 37th place tie by means of weighted averaging. It is noteworthy that the rankings of the top 19 journals were identical under both methods, whereas those with divergent positions almost invariably fell within the bottom two-thirds. One can conclude that for most decision-making purposes it makes little or no practical difference whether crude or weighted averaging is used. However, from the theoretical perspective weighted averaging clearly represents a more accurate measure. Inspection of Table 2’s second and third columns allows comparison of the weighted averaging with the adjusted impact factor methods. Not surprisingly, only one journal (Journal of Black Studies) has an identical score under both. Noteworthy differences can be observed between the two rankings, although the correlation between these two approaches is also
454
T. E. NISONGER
TABLE 2 RANKINGS OF A SET OF 64 POLITICAL SCIENCE JOURNALS USING THREE METHODS OF COMBINING SSCI JCR DATA FROM 1987, 1988, and 1989 Journal Title
Journal of Political Economy American Sociological Review American Journal of Sociology American Political Science Review International Organization World Politics Administrative Science Quarterly Foreign Affairs Law and Society Review American Journal of International Law American Journal of Political Science International Studies Quarterly Social Forces Public Administration Review Public Opinion Quarterly Journal of Politics Public Interest Judicature China Quarterly Daedalus British Journal of Political Science Public Choice Political Science Quarterly Behavioral Science International Affairs Journal of Conflict Resolution Legislative Studies Quarterly Journal of Asian Studies Comparative Political Studies Urban Affairs Quarterly American Politics Quarterly PS: Political Science & Politics Political Theory Soviet Studies Journal of International Affairs Canadian Journal of Political Science Comparative Politics Journal of Peace Research Political Studies Social Science Quarterly Publius Simulation and Games European Journal of Political Research Western Political Quarterly Administration and Society Politics and Society Government and Opposition Asian Survey
Crude Ave. Impact Factor 2.192 2.141 1.972 1.936 1.904 1.833 1.749 1.635 1.459 1.370 1.176 1.047 0.981 0.849 0.840 0.786 0.774 0.756 0.739 0.649 0.647 0.613 0.623 0.570 0.568 0.552 0.553 0.539 0.529 0.527 0.501 0.488 0.476 0.470 0.415 0.420 0.420 0.421 0.404 0.398 0.352 0.355 0.346 0.335 0.316 0.332 0.306 0.295
(1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) (14) (15) (16) (17) (18) (19) (20) (21) (23) (22) (24) (25) (27) (26) (28) (29) (30) (31) (32) (33) (34) (38) (36) (36) (35) (39) (40) (42) (41) (43) (44) (46) (45) (47) (49)
Weighted Ave. Impact Factor 2.192 2.144 1.979 1.937 1.888 1.838 1.758 1.637 1.469 1.367 1.166 1.045 0.980 0.839 0.837 0.789 0.774 0.759 0.739 0.645 0.645 0.613 0.580 0.574 0.570 0.553 0.549 0.538 0.533 0.527 0.508 0.482 0.476 0.476 0.426 0.426 0.421 0.421 0.403 0.403 0.352 0.346 0.344 0.335 0.315 0.314 0.305 0.295
(1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) (14) (15) (16) (17) (18) (19) (20) (20) (22) (23) (24) (25) (26) (27) (28) (29) (30) (31) (32) (33) (33) (35) (35) (37) (37) (39) (39) (41) (42) (43) (44) (45) (46) (47) (48)
Adjusted Impact Factor
2.602 2.809 2.450 2.804 2.303
(4)
1.392 1.343 1.163 0.884 0.950 1.034 0.655 0.775 0.887 0.658 0.646 0.841 0.609 0.746 0.487 0.663 0.735 0.556 0.733 0.736 0.597 0.474 0.689 0.543 0.563 0.390 0.609 0.402 0.510 0.507 0.400 0.414 0.430 0.411 0.500 0.389 0.304 0.300
(10) (11) (13) (17) (15) (14) (27) (19) (16) (26) (28) (18) (29) (20) (38) (25) (22) (33) (23) (21) (31) (39) (24) (34) (32) (45) (29) (43) (35) (36) (44) (41) (40) (42) (37) (46) (49) (50)
(1)
(5) (2) (6) 1.889 (7) 2.778 (3) 1.582 (9) 1.857 (8) 1.270 (12)
(Continued)
A Methodological
Issue
45s
TABLE 2 continued Journal Title
Journal of Latin American Studies Policy Sciences Political Quarterly Journal of InterAmerican Studies and World Affairs Political Science American Behavioral Scientist Review of Politics Polity Policy Studies Journal Annals of the American Academy of Political and Social Science Dissent International Social Science Journal Journal of Black Studies Social Science Journal Journal of Developing Areas Orbis
Crude Ave. Impact Factor
Weighted Ave. Impact Factor
Adjusted Impact Factor
0.297 (48) 0.291 (50) 0.278 (51)
0.294 (49) 0.286 (50) 0.279 (5 1)
0.318 (48) 0.231 (55) 0.233 (54)
0.254 0.252 0.218 0.206 0.190 0.181
(52) (53) (54) (55) (56) (57)
0.241 0.240 0.217 0.202 0.189 0.182
(52) (53) (54) (55) (56) (57)
0.215 0.324 0.277 0.190 0.247 0.241
(57) (47) (5 1) (60) (52) (53)
0.178 0.176 0.165 0.165 0.148 0.138 0.029
(58) (59) (60) (60) (62) (63) (64)
0.178 0.176 0.163 0.162 0.148 0.137 0.029
(58) (59) (60) (61) (62) (63) (64)
0.218 0.096 0.192 0.162 0.207 0.109 0.011
(56) (63) (59) (61) (58) (62) (64)
extremely high (r = .9771). The top-ranked journal using weighted averaging, Journal of Political Economy, actually ranked fourth after the impact factor was adjusted. International Affairs dropped 13 places, from 25th to 3&h, while two journals changed by 10 places: Canadian Journal of Political Science (from 35th to 45th) and Public Interest (17th to 27th). Two titles varied by nine places: Urban Affairs Quarterfy increased from 30th to 21st, while Political Theory changed from 33th to 24th. Only four journals (Social Forces, Public Opinion Quarterly, American Politics Quarterly, and Politics and Society) displayed the identical ranking under both methods. When the top ranked journals under weighted averaging are compared with the leading journals using adjusted impact factor, nine of the top ten and 17 of the top 20 overlap. Thus, it is apparent that the differences between the rankings by weighted averaging and adjusted impact factor are significant and for some journals quite striking, despite the high correlation between the two methods. These differences could certainly have an impact on the decision-making process. It is noteworthy that for 46 journals, the adjusted impact factors are higher than those obtained by crude averaging and higher than the weighted averaging result for 48 journals (with one journal having an identical rating for both the adjusted impact factor and weighted averaging). The mean adjusted impact factor for the entire set was 0.790, contrasted to a mean of 0.668 by crude averaging and 0.666 by weighted averaging. This phenomenon undoubtedly reflects the fact that calculating the adjusted impact factor based on a two-year lag allows a greater opportunity for citations to be received.
CONCLUSION This research paper’s primary objective has been to point out the methodological error inherent in averaging JCR impact factor data, as earlier researchers who have done so have
456
T. E. NISONGER
not addressed the hidden bias this introduces into the results. In fact, because this bias is probably not intuitively obvious, there is a danger future researchers or practitioners could repeat and compound the error. While the supporting evidence used in this paper is from the Social Sciences Citation Index JCR, the bias would also apply to the Science Citation Index JCR, as its impact factor is calculated in an identical manner. The “adjusted impact factor” formula proposed here represents one possible strategy for correcting the bias. This particular correction is not endorsed as infallible. It too may contain hidden biases, e.g., journals that tend to be cited quickly could be disadvantaged due to the two-year lag period. Other options are undoubtedly available. For example, averaging impact factor data from alternate years would also correct the bias. There is an obvious trade-off in terms of effort versus accuracy when using the SSCI JCR impact factor data. To achieve maximum precision requires considerably more time spent recalculating the statistics, while the most efficient utilization of the available statistics sacrifices some degree of accuracy. Researchers and libraries will have to determine their own priorities regarding efficiency versus accuracy. The author presumes that rigorous researchers would definitely wish to compensate for the bias discussed in this paper. Practicing librarians, who would normally consider impact factor as only one of many variables (e.g., local usage, curricular relevance, indexing, cost, subjective judgment, etc.) in reaching journal subscription and cancellation decisions, may not require as high a degree of accuracy, although they should be aware of this methodological bias. Questions for further research include: 1. the degree of year-to-year fluctuation in JCR data-an issue the author is presently investigating; 2. what alternate approaches (in addition to the ones suggested here) are available for correcting the bias analyzed in this paper; 3. whether a different type of adjustment, such as a one-year rather than a two-year lag, to the impact factor might be more appropriate for scientific disciplines in which immediacy of citation would be more important; 4. what would be the implications of using a one-year rather than a two-year lag in calculating the adjusted impact factor; 5. the extent to which the results of earlier journal rankings would differ if a correction were made for the bias; and 6. what proportion of academic collection development librarians use the JCR, how they are using it, and for what purposes. This issue is significant because it relates to the effective utilization of JCR citation data and the accurate evaluation of scholarly journals- both important topics in contemporary library and information science. As researchers and library practitioners will undoubtedly continue to use JCR impact factor data for journal evaluation and ranking, an ongoing dialogue concerning the accurate use of this tool is imperative.
AcknowledgementsStephen P. Harter, Professor, Indiana University, School of Library and is gratefully acknowledged for providing the inspiration to write this research note by pointing methodological issue analyzed here. Blaise Cronin, Dean; Charles H. Davis, Visiting Scholar; ciate Professor; and Judith Serebnick, Associate Professor (all at Indiana University’s SLIS) able input. The author’s graduate assistants, Karen Fouchereaux and Cindy Pierard, photocopied JCR.
Information Science, out to the author the Debora Shaw, Assoare thanked for valupages from the SSCI
A Methodological Issue
457
NOTES 1. Gross, P.L.K. and Gross, E.M. “College Libraries and Chemical Education,” Science, 66 (October 28, 1927), 385-389. 2. Nisonger, Thomas E. Collection Evaluation in Academic Libraries: A Literature Guide and Annotated Bibliography. Englewood, CO: Libraries Unlimited, 1992. 3. Through 1988 the JCR was issued annually as a bound volume in both the SSCI and the XI. Beginning in 1989 both JCRs were issued separately in microfiche format. 4. The five criteria are: (1) total citations received; (2) impact factor; (3) immediacy index, i.e., how quickly a journal is cited; (4) items published in the current year; and (5) citations in the current year to issues of the journal published during the two preceding years. 5. Garfield, Eugene. “Citation Analysis as a Tool in Journal Evaluation,” Science, 178 (November 3, 1972), 477. 6. Smith, Thomas E. “The Journal Citation Reports as a Deselection Tool,” Bulletin of the Medical Library Association, 73 (October 1985), 388. 7. Broadus, Robert N. “A Proposed Method for Eliminating Titles from Periodical Subscription Lists,” College & Research Libraries 46 (January 1985), 30-35. 8. Dombrowski, Theresa. “Journal Evaluation Using Journal Citation Reports as a Collection Development Tool,” Collection Management, 10, nos. 3/4 (1988), 175-180. 9. Leavy, Marvin D. “The Journal Citation Reports as a Tool in Periodical Subscription Decisions: An Illustration and an Evaluation,” delivered at the Annual Fall Meeting of the Kansas Library Association/College and University Libraries Section in Hutchinson Holidome, Friday, October 10, 1980. ERIC Document 197 732, pp. 10. 10. Line, Maurice B. “Use of Citation Data for Periodicals Control in Libraries: A Response to Broadus,” College & Research Libraries, 46 (January 1985), 36-37. 11. Buffardi, Louis C. and Nichols, Julia A. “Citation Impact, Acceptance Rate, and APA Journals,” American Psychologist, 36 (November 1981), 1456. 12. Dometrius, Nelson C. “Subjective and Objective Measures of Journal Stature,” Social Science Quarterly, 70 (March 1989), 202. 13. Vinkler, P. “Bibliometric Features of Some Scientific Subfields and the Scientometric Consequences Therefrom,” Scientometrics, 14, nos. 5/6 (1988), 453. 14. Ribbe, Paul H. “Assessment of Prestige and Price of Professional Publications,” American Mineralogist, 73 (May/June 1988), 449-469. 15. Tomer, Christinger. “A Statistical Assessment of Two Measures of Citation: the Impact Factor and the lmmediacy Index,” Information Processing & Management, 22, no. 3 (1986), 251-258. 16. Woodward, A.M. and Hensman, Sandy. “Citations to Review Serials,” Journal of Documentation, 32 (December 1976), 290-293. 17. Scanlan, Brian D. “Coverage by Current Contents and the Validity of Impact Factors: IS1 from a Journal Publisher’s Perspective,” Serials Librarian, 13 (October/November 1987), 57-66. 18. Van Raan, A.F.J. and Hartmann, D. “The Comparative Impact of Scientific Publications and Journals: Methods of Measurement and Graphical Display,” Scientometrics, 11, nos. 5/6 (1987), 325-33 1. 19. Hirst, Graeme and Talent, Nadia. “Computer Science Journals-An Iterated Citation Analysis,” IEEE Transactions on Professional Communication, PC-20 (December 1977), 233-238. 20. Hirst, Graeme. “Discipline Impact Factor? A Method for Determining Core Journal Lists,” Journal of the American Society for Information Science, 29 (July 1978), 171-172. 21. He, Chunpei and Pao, Miranda Lee. “A Discipline-Specific Journal Selection Algorithm,” Information Processing & Management, 22, no. 5 (1986), 405-416. 22. Nieuwenhuysen, P. and Rosseau, R. “A Quick and Easy Method to Estimate the Random Effect on Citation Measures,” Scientometrics, 13, nos. l/2 (1988), 45-52. 23. Nieuwenhuysen, Paul. “Journal Citation Measures: Taking into Account Their Fluctuations from Year to Year,” Journal of Information Science, 15, no. 3 (1989), 175-178. 24. Bonitz, M. “Journal Ranking by Selective Impact: New Methods Based on SD1 Results and Journal Impact Factors,” Scientometrics, 7, nos. 3/6 (1985), 471-485. 25. Vinkler, P. “Evaluation of Some Methods for the Relative Assessment of Scientific Publications,” Scientometrics, 10, nos. 3/4 (1986), 157-177. 26. Vinkler, P. “A Quasi-Quantitative Citation Model,” Scientometrics, 12, nos. l/2 (1987), 47-72. 27. Vinkler, “Bibliometric Features of Some Scientific Subfields,” 453-474. 28. Schubert, A. and Braun, T. “Relative Indicators and Relational Charts for Comparative Assessment of Publication Output and Citation Impact,” Scientometrics, 9, nos. 5/6 (1986), 281-291.
458
T. E. NISONGER
29. Sen, B.K. “Normalized Impact Factor,” Journal of Documentation, 48 (September 1992), 318-325. 30. Christenson, James A. and Sigelman, Lee. “Accrediting Knowledge: Journal Stature and Citation Impact in Social Science,” Social Science Quarterly, 66 (December 1985), 964-975. 31. Giles, Michael W. and Wright, Gerald C., Jr. “Political Scientists’ Evaluations of Sixty-Three Journals,” PS: Political Science & Politics, 8 (Summer 1975), 254-256. 32. Glenn, Norval D. “American Sociologists’ Evaluations of Sixty-Three Journals,” American Sociologist, 6 (November 1971), 298-303. 33. Feingold, Alan. “Assessment of Journals in Social Science Psychology,” American Psycholog&, 44 (June 1989), 961-964. 34. Colson, Harold. “Citation Rankings of Public Administration Journals,” Administration & Society, 21 (February 1990), 452-471. 35. Stack, Steven. “Measuring the Relative Impacts of Criminology and Criminal Justice Journals: A Research Note,” Justice Quarterly, 4 (September 1987), 475-484. 36. Ribbe, “Assessment of Prestige,” 449-469. 37. Ribbe, Paul H. “A Scientist’s Assessment of a Microcosm of the Serials Universe,” Serials Librarian, 17, nos. 3/4 (1990), 121-142. 38. Stack, “Measuring the Relative Impacts,” p. 478. 39. Colson, “Citation Rankings,” p. 454. 40. Christenson and Sigelman, “Accrediting Knowledge,” p. 967. 41. Buffardi and Nichols, “Citation Impact,” p. 1454. 42. Gordon, Michael D. “Citation Ranking Versus Subjective Evaluation in the Determination of Journal Hierarchies in the Social Sciences,” Journal of the American Society for Information Science, 33 (January 1982), 57. 43. Vlachy, Jan. “Trends in the Citation Impact of Physics Journals, 1974-1983,” Czechoslovak Journal of Physics, B 35, no. 2 (1985), 187-190. 44. Sievert, MaryEllen C. “Ten Years of the Literature of Online Searching: An Analysis of Online and Online Review,” Journal of the American Society for Information Science, 41 (December 1990), 560-568. 45. Giles, Michael W., Mizell, Francie, and Patterson, David. “Political Scientists’ Journal Evaluations Revisited,” PS Political Science & Politics, 22 (September 1989), 613-617. 46. The “adjusted impact factor” ranking was used in Nisonger, Thomas E. “A Ranking of Political Science Journals Based on Citation Data,” Serials Review, 19, no. 4 (1993), 7-14.
Pergamon
0364-6408(94)ooo30-1
A METHODOLOGICAL ISSUE CONCERNING THE USE OF SOCIAL SCIENCES CITATION INDEX JOURNAL CITATION REPORTS IMPACT FACTOR DATA FOR JOURNAL RANKING THOMAS E. NISONGER Associate
Professor
Indiana School
of Library
University and Information
Bloomington, E-mail:
Science
IN 47405
[email protected]
Abstract- Following a brief introduction of citation-based journal rankings as potential serials management tools, the most frequently used citation measureimpact factor - is explained. This paper then demonstrates a methodological bias inherent in averaging Social Sciences Citation Index Journal Citation Reports (SSCI JCR) impact factor data from two or more consecutive years. A possible method for correcting the bias, termed adjusted impact factor, is proposed. For illustration, a set of political science journals is ranked according to three different methods (crude averaging, weighted averaging, and adjusted impact factor) for combining SSCI JCR impact factor data from successive years. Although the correlations among the three methods are quite high, one can observe noteworthy differences in the rankings that could impact on collection development decisions. Keywords-Journal
ranking, Journal Citation Reports, Impact factor, Political
science journals. INTRODUCTION In the mid-1990s the management of serials has assumed paramount importance for collection development librarians because of the exceedingly high serials inflation rate. The so-called serials crisis has had a direct impact on traditional acquisitions as libraries have shifted funds away from the monographic budget in order to pay for their serials. As countless libraries face serials review and cancellation projects, many librarians are seeking new tools to help them 447
448
T. E. NISONGER
evaluate journals. Journal ranking studies based on Journal Citation Reports data represent useful collection management tools whose full potential is beginning to be appreciated by librarians. Since the publication of Gross and Gross’s famous study of chemistry journals in 1927, journal rankings or ratings have been offered in numerous disciplines [l]. A bibliography of the last decade’s major journal ranking studies is included in Nisonger [2]. In recent years, perception studies based on subjective judgment by experts and citation data, especially from the Institute for Scientific Information (ISI), represent the two most frequently used ranking methods. This article’s focus is on the latter approach. The Institute for Scientific Information publishes three citation indexes covering each of the broad areas of human knowledge: the Social Sciences Citation Index (SSCI), the Science Citation Index (SCZ), and the Arts & Humanities Citation Index. Both the SSCI and the SCZ contain a section called the Journal Citation Reports (JCR) which contains citation data on thousands of journals [3]. The Arts & Humanities Citation Index does not have a JCR as it is the conventional wisdom of collection development that serials are less important in that area. The JCR ranks journals according to five criteria [4]. The two most significant for serials collection management are total citations received and impact factor, a measure to normalize for journal size that will be explained in detail in the next section. Citation analysis is based on the assumption that if an author cites a journal, he or she has found it useful. Thus, it follows that the more frequently a journal is cited, the greater its role in the scholarly communication process. The precise meaning of citation is unclear and beyond this paper’s scope. Nevertheless, a journal’s citation record is indicative of some amorphous positive attribute that is often called quality, value, or impact. Moreover, it offers the advantage of an objective measure. For practical decision-making purposes in collection management, the primary concern is with a journal’s relative ranking within its discipline rather than the raw citation data. A number of previous authors have advocated the use of citation data for serials collection management. Eugene Garfield, the ISI’s founder and an illustrious advocate of citation studies, contends that “. . . citation analysis would appear to be of great potential value in management of library journal collections” [5]. Smith states that “. . . the JCR is a helpful objective tool to augment other methods of journal deselection” [6]. Broadus argues that “. . . one might begin the process of identifying titles for cancellation by reviewing Journal Citation Reports (JCR)” [7], while Dombrowski asserts, “Citation analysis cannot be used as the sole method of journal evaluation, but if it is used in conjunction with other evaluative methods, it can provide a fairly accurate picture of a journal’s value . . .” [8]. Leavy concludes that “. . . judicious use of the [JCR’s] data can benefit any academic library in managing its scholarly periodical collection” [9]. Line has assumed a contrarian position, arguing that the critical decision-making criterion should be a journal’s use in a particular library rather than citation data [lo]. Ranking studies are obviously applicable to journal selection, cancellation, and weeding decisions. While the application to librarianship is evident, it should be noted that journal ranking studies are of considerable interest to scholars in other disciplines. These rankings can provide guidance in article submission decisions and assist in evaluation of a faculty member’s research output. In fact, the vast majority of journal rankings have been conducted by scholars outside the field of library and information science. The author’s unpublished research indicates that, as one might intuitively expect, a journal’s citation data can vary significantly from year to year. Such fluctuation has practical implications. Consequently, many researchers have averaged impact factor data from two or
A Methodological Issue
449
more adjacent years in order to compensate for yearly fluctuation. This paper’s objective is to demonstrate that simply averaging impact factor data, while seemingly a valid procedure, introduces a subtle methodological bias.
IMPACT FACTOR As developed by the ISI, impact factor represents a ratio of citations received to citable items, based on citations made in the current year (by all journals in the IS1 database) to articles published during the previous two years. In essence, it indicates the mean number of times a so-called average article has been cited. A journal’s 1990 impact factor would be calculated by the following formula. 1990 impact factor =
No. of 1990 citations to 1989 + 1988 items No. of citable items published in 1989 + 1988 ’
Impact factor compensates for the advantage that older, larger, or more frequently published journals would enjoy if rankings were based on total citations received. Although somewhat controversial, impact factor has been used for citation-based journal rankings more than any other citation measure, and many researchers have asserted it is the most valid citation measure for journal evaluation purposes. For example, Buffardi and Nichols stress that impact factor “. . . is probably the best single measure of journal quality we have at present” Ill]. Dometrius, while acknowledging inadequacies, terms impact factor “. . . a robust indicator of journal stature” [ 121. Vinkler declares, “There is at present no better indicator applicable in practice characterizing the scientific impact of journals than the impact factors given by Garfield” [Founder of the JCR] [13]. However, a number of methodological and practical difficulties with journal rankings based on impact factor have been pointed out. Obviously, citation data constitute only one of numerous journal evaluation criteria, including subjective judgment, use, indexing, relevance to a library’s mission, and cost. Only a portion of the total journal universe is covered in the Journal Citation Reports (although generally the most important scholarly journals are included) and many librarians do not have access to the JCR in any case. Impact factor is generally recognized to be invalid for cross-disciplinary comparisons, new journals (which have not had time to establish a citation record), and nonresearch journals. Impact factor does not consider citations to articles published more than two years previously. Anecdotal evidence abounds of errors in the IS1 database, and while Ribbe has noted errors in JCR data, he concludes that most impact factors are correct within 5% [14]. Tomer has questioned the significance of impact factor because he found a ranking based on impact factor correlated highly with a ranking of the same journal set according to total citations received [15]. Research by Woodward and Hensman has shown that review journals tend to have higher impact factors [16]. Finally, Scanlan has offered a harsh criticism of impact factor from the perspective of publishers, noting that the type and length of articles and a research field’s size, style, and citation tradition, as well as journal self-citations can influence impact factor [17]. Several scholars have proposed modifications or alternatives to the IS1 impact factor to correct perceived deficiencies. Van Raan and Hartmann developed a measure, termed “comparative impact,” which graphically depicts the citation record for each type of publication in a journal, e.g., letters, editorials, and “normal” articles [18]. To compensate for the fact that JCR rankings are constructed on citations from all disciplines, i.e., every journal covered in
450
T. E. NISONGER
the IS1 database, Hirst and Talent advanced the “computer science impact factor” [19], Hirst the “discipline impact factor” [20], and He and Pao the “discipline influence score” [21], all utilizing the manipulation of IS1 data to rank a discipline’s journals based solely on citations from the discipline itself. Methods for estimating the random fluctuation of impact factor have been developed by Nieuwenhuysen and Rosseau [22], based on Poisson distribution, and by Nieuwenhuysen, using a probabilistic model [23]. Both these advanced statistical techniques will probably not be of much concern to most practicing librarians. Bonitz has suggested a “selective impact” measure that combines a journal’s impact factor and productivity from SD1 (Selective Dissemination of Information) runs [24]. A number of efforts have been undertaken to develop impact measures that allow valid comparisons across subfields and disciplines. For example, Vinkler has proposed an “average subfield impact factor,” based on his analysis of chemistry journals, which indicates the average citation rate for various subfields [25]. Vinkler also introduced the “citation strategy indicator,” which relates a journal’s impact factor to the mean impact factor of other journals in its specialty [26,27]. Schubert and Braun advanced a “relative citation rate,” which relates citations received to citations expected [28]. Sen recently proposed a “normalized impact factor,” which utilizes a multiplier based on the current year’s highest journal impact factor in each subject category covered by the JCR [29]. The preceding survey of methodological issues concerning impact factor, although by no means comprehensive, illustrates that impact factor has been the focus of considerable discussion in the literature. It is obvious that much more effort has been devoted to the problems of cross disciplinary comparison than to using impact factors from a range of years-the subject of this paper. Yet, ongoing controversy continues concerning how impact factor may be utilized most effectively.
PREVIOUS JOURNAL RANKINGS BASED ON AVERAGING IMPACT FACTOR Several major journal ranking or rating studies have averaged SSCZ JCR impact factor data from two or more consecutive years in order to account for yearly fluctuations in the data. Christenson and Sigelman averaged impact factor data for the years 1977, 1978, and 1979 to rate 56 political science and 61 sociology journals in their citation-based ranking study [30] undertaken for the purpose of comparison with the subjective rankings of political science journals by Giles and Wright [3 l] and sociology journals by Glenn [32]. Although Christenson and Sigelman did not offer an explicit ranking, a ranking may be constructed from their ratings. Feingold averaged SSCZ impact factor data for 1985 and 1986 to rank 52 journals in eight subfields of social science psychology: applied, clinical/abnormal, developmental, experimentallearning and memory, experimental-perception, personality, quantitative, and social [33]. (The journals were ranked within each subfield rather than within the 52 total journals.) Colson ranked 35 public administration journals by averaging SSCZ impact factor data for the years 1981 through 1986 [34]. To rank 26 criminology and criminal justice journals, Stack averaged the 1984 and 1985 impact factors [35]. Multiple-year impact factors have also been averaged for purposes other than formal journal rankings. Ribbe used the mean 1983-1985 and mean 1984-1987 Science Citation Index JCR impact factors for sets of 17 and 20 mineralogy, petrology, and geochemistry journals in his study, which demonstrated that federally funded research is more likely to be published in high impact factor journals [36,37].
451
A Methodological Issue
As noted above, researchers have used multi-year data to compensate for yearly fluctuation. To quote Stack, “I used a two-year period to increase the reliability of the index” j38J. Colson observed, “Citation scores are subject to certain annual ~uctuations, so a fair ordering of journal rank requires more than one year of data” [39]. Christenson and Sigelman used three years “To mitigate the possibilities of yearly fluctuations” [40]. These researchers have not explicitly addressed the method they used to average impact factors from different years. More fundamentally, they were apparently unaware that averaging IS1 impact factors from consecutive years, regardless of method, introduces a subtle bias into the results. The next section discusses alternate methods for averaging impact factor. It is followed by a section that explains the bias inherent in averaging impact factor data from adjacent years.
CRUDE VERSUS WEIGHTED
AVERAGING
In “crude” averaging, one simply calculates the mean of a journal’s raw impact factors for the years being combined. For “weighted” averaging, one calculates the ratio of total citations (to the previous two years) to total source items (pub~sh~ during the previous two years) for the years being averaged. Table 1, which summ~izes 1987, 1988, and 1989 SSCf JCR data for the journal World PO&tics, can be used to illustrate the difference. In crude averaging, the three final yearly impact factors as reported in JCR (1.973, 1.625, and 1.900) would simply be averaged, resulting in a mean of 1.833. In weighted averaging, the total of each year’s citations to the previous two years would be divided by the total of the source articles published during the preceding two years, e.g., 18249, obtaining a figure of 1.838. The crude vs. weighted averaging issue is a rather self-evident point that should not surprise anyone versed in statistics. The author believes it should be mentioned because previous researchers have not explicitly stated which averaging method they used, and, as illustrated in this paper, it does make some difference in the results obtained.
BIAS INTRODUCED
BY AVERAGING IMPACT FACTOR
The averaging of impact factor data for consecutive years, regardless of which method is used, introduces a subtle bias. When a range of years is combined, data for the middle years
TABLE 1 SSCI JCR DATA FOR THE JOURNAL World Politics Citations to Last 2 Years
Source Items Last 2 Years
Impact Factor
1988 1989
73 52 57
37 32 30
1.973 1.625 1.900
Total
182
99
Year 1987
452
T. E. NISONGER
will be counted twice, while the extreme years will be counted only once. The following figures illustrate this bias when the impact factors for the years 1990 and 1989 are averaged, using either the crude or weighted method. 1990 1. F. = 1990 citations to 1989 + 1988/1989 + 1988 source items, 1989 I. F. = 1989 citations to 1988 + 1987/1988 + 1987 source items. It is apparent that 1988 is counted twice, while 1989 and 1987 only once. If additional years were added to the range to be averaged, e.g., 1991 and 1987, the extreme years in the preceding illustration (1989 and 1987) would then be counted twice, but a new set of extreme years would develop. This is illustrated below. 1991 I. F. = 1991 citations to 1990 + 1989/1990 + 1989 source items. 1990 I. F. = 1990 citations to 1989 + 1988/1989 + 1988 source items. 1989 I. F. = 1989 citations to 1988 + 1987/1988 + 1987 source items. 1988 I. F. = 1988 citations to 1987 + 1986/1987 + 1986 source items. We now see that the three middle years (1989, 1988, and 1987) are counted twice, while a new set of extreme years (1986 and 1990) are considered only once in the final average figure. No matter how many consecutive years are averaged, there will always be middle years considered twice, and extreme years that are counted only one time. However, averaging a greater number of years will reduce the bias’s influence. One might intuitively suspect this effect would be self-canceling, as it applies equally to all journals in the set. However, as illustrated below, notable differences between alternate rankings (one corrected for this effect, the other uncorrected) are apparent. This would not be an issue if there were no difference between one year of a journal and another. However, both logic and intuition indicate that the quality of a journal’s articles (and thus its citation record) will vary from year to year just as a collegiate sports team will have a different record each year. The question of year-to-year consistency of impact factor data and rankings has not been fully investigated, although a number of researchers, e.g., Buffardi and Nichols [41], Gordon [42], Vlachy [43], and Sievert [44], have briefly addressed the issue. However, their research is only tangentially relevant to the issue under examination here. One would logically expect considerable year-to-year stability between impact factors because citations to the same year would be included in both, i.e., the number of citations to articles published in 1988 would be used in calculating both the 1989 and 1990 impact factors.
ADJUSTED
IMPACT FACTOR
To compensate for this bias, the author proposes an adjustment to the JCR data, termed here adjusted impact factor, to be used when averaging impact factor data from successive years. In this approach, for each year in the range to be averaged, the number of citations made (by all journals in the ISI’s database) that year to the journal’s issues published two years earlier is divided by the number of citable items published two years previously. For exam-
A Methodological
453
Issue
ple, when combining a journal’s 1990 data with those of an adjacent year, one would divide the number of citations made in 1990 to the journal’s 1988 issues by the number of citable items it contained in 1988. The following formula illustrates the averaging of data from the 1989, 1988, and 1987 JCR using the adjusted approach. (1989 tits. to 1987) + (1988 tits. to 1986) + (1987 tits. to 1985) 1987 citable items + 1986 citable items + 1985 citable items ’ In this method, each year is counted only once. No complicated algorithm is necessary, while the data are easily gathered from the JCR. A two-year rather than a one-year period between the citation and the cited item’s publication data is used to allow more time for a journal to be cited.
DEMONSTRATION
WITH SET OF POLITICAL
SCIENCE JOURNALS
To illustrate the influence of these issues on actual journal rankings, a set of 64 political science journals were rated and ranked according to the crude averaging, the weighted averaging, and the adjusted impact factor methods for combining SSCI JCR data for three years. The results are displayed in Table 2. Parenthetically, it should be noted that a rating assigns an absolute evaluative measure to a journal, while a ranking places it in rank order among a journal set. These 64 journals were chosen because they represent the set covered in the SSCI JCR from among the 78 journals included in the most recent major perception-based ranking of political science journals by Giles, Mizell, and Patterson [45,46]. The years 1987, 1988, and 1989 were combined because they most closely correspond to Giles, Mizell, and Patterson’s 1988 survey date. Some of these journals, e.g., American Sociological Review, are outside the discipline of political science, although this point is irrelevant to the issues addressed in this paper. Examination of columns 1 and 2 in Table 2, comparing crude and weighted averaging, reveals that only 14 of the 64 journals (21.9Vo) obtain an identical rating by both methods, thus indicating we are dealing with a genuine issue. Obviously, for practical purposes a journal’s ranking is more significant than its rating. The correlation between rankings based on crude versus weighted averaging is almost identical (r = .9999), although some differences between the rankings produced by the two methods are apparent. Only 46 of the 64 journals (71.9%) are ranked in either the same position or tied for the same position regardless of which averaging method is used. Nevertheless, the rankings of just two titles vary by more than one position. Journal of International Affairs moved from 38th under crude averaging to a tie for 35th with weighted averaging, while Journal of Peace Research dropped from 35th with crude averaging to a 37th place tie by means of weighted averaging. It is noteworthy that the rankings of the top 19 journals were identical under both methods, whereas those with divergent positions almost invariably fell within the bottom two-thirds. One can conclude that for most decision-making purposes it makes little or no practical difference whether crude or weighted averaging is used. However, from the theoretical perspective weighted averaging clearly represents a more accurate measure. Inspection of Table 2’s second and third columns allows comparison of the weighted averaging with the adjusted impact factor methods. Not surprisingly, only one journal (Journal of Black Studies) has an identical score under both. Noteworthy differences can be observed between the two rankings, although the correlation between these two approaches is also
454
T. E. NISONGER
TABLE 2 RANKINGS OF A SET OF 64 POLITICAL SCIENCE JOURNALS USING THREE METHODS OF COMBINING SSCI JCR DATA FROM 1987, 1988, and 1989 Journal Title
Journal of Political Economy American Sociological Review American Journal of Sociology American Political Science Review International Organization World Politics Administrative Science Quarterly Foreign Affairs Law and Society Review American Journal of International Law American Journal of Political Science International Studies Quarterly Social Forces Public Administration Review Public Opinion Quarterly Journal of Politics Public Interest Judicature China Quarterly Daedalus British Journal of Political Science Public Choice Political Science Quarterly Behavioral Science International Affairs Journal of Conflict Resolution Legislative Studies Quarterly Journal of Asian Studies Comparative Political Studies Urban Affairs Quarterly American Politics Quarterly PS: Political Science & Politics Political Theory Soviet Studies Journal of International Affairs Canadian Journal of Political Science Comparative Politics Journal of Peace Research Political Studies Social Science Quarterly Publius Simulation and Games European Journal of Political Research Western Political Quarterly Administration and Society Politics and Society Government and Opposition Asian Survey
Crude Ave. Impact Factor 2.192 2.141 1.972 1.936 1.904 1.833 1.749 1.635 1.459 1.370 1.176 1.047 0.981 0.849 0.840 0.786 0.774 0.756 0.739 0.649 0.647 0.613 0.623 0.570 0.568 0.552 0.553 0.539 0.529 0.527 0.501 0.488 0.476 0.470 0.415 0.420 0.420 0.421 0.404 0.398 0.352 0.355 0.346 0.335 0.316 0.332 0.306 0.295
(1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) (14) (15) (16) (17) (18) (19) (20) (21) (23) (22) (24) (25) (27) (26) (28) (29) (30) (31) (32) (33) (34) (38) (36) (36) (35) (39) (40) (42) (41) (43) (44) (46) (45) (47) (49)
Weighted Ave. Impact Factor 2.192 2.144 1.979 1.937 1.888 1.838 1.758 1.637 1.469 1.367 1.166 1.045 0.980 0.839 0.837 0.789 0.774 0.759 0.739 0.645 0.645 0.613 0.580 0.574 0.570 0.553 0.549 0.538 0.533 0.527 0.508 0.482 0.476 0.476 0.426 0.426 0.421 0.421 0.403 0.403 0.352 0.346 0.344 0.335 0.315 0.314 0.305 0.295
(1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) (14) (15) (16) (17) (18) (19) (20) (20) (22) (23) (24) (25) (26) (27) (28) (29) (30) (31) (32) (33) (33) (35) (35) (37) (37) (39) (39) (41) (42) (43) (44) (45) (46) (47) (48)
Adjusted Impact Factor
2.602 2.809 2.450 2.804 2.303
(4)
1.392 1.343 1.163 0.884 0.950 1.034 0.655 0.775 0.887 0.658 0.646 0.841 0.609 0.746 0.487 0.663 0.735 0.556 0.733 0.736 0.597 0.474 0.689 0.543 0.563 0.390 0.609 0.402 0.510 0.507 0.400 0.414 0.430 0.411 0.500 0.389 0.304 0.300
(10) (11) (13) (17) (15) (14) (27) (19) (16) (26) (28) (18) (29) (20) (38) (25) (22) (33) (23) (21) (31) (39) (24) (34) (32) (45) (29) (43) (35) (36) (44) (41) (40) (42) (37) (46) (49) (50)
(1)
(5) (2) (6) 1.889 (7) 2.778 (3) 1.582 (9) 1.857 (8) 1.270 (12)
(Continued)
A Methodological
Issue
45s
TABLE 2 continued Journal Title
Journal of Latin American Studies Policy Sciences Political Quarterly Journal of InterAmerican Studies and World Affairs Political Science American Behavioral Scientist Review of Politics Polity Policy Studies Journal Annals of the American Academy of Political and Social Science Dissent International Social Science Journal Journal of Black Studies Social Science Journal Journal of Developing Areas Orbis
Crude Ave. Impact Factor
Weighted Ave. Impact Factor
Adjusted Impact Factor
0.297 (48) 0.291 (50) 0.278 (51)
0.294 (49) 0.286 (50) 0.279 (5 1)
0.318 (48) 0.231 (55) 0.233 (54)
0.254 0.252 0.218 0.206 0.190 0.181
(52) (53) (54) (55) (56) (57)
0.241 0.240 0.217 0.202 0.189 0.182
(52) (53) (54) (55) (56) (57)
0.215 0.324 0.277 0.190 0.247 0.241
(57) (47) (5 1) (60) (52) (53)
0.178 0.176 0.165 0.165 0.148 0.138 0.029
(58) (59) (60) (60) (62) (63) (64)
0.178 0.176 0.163 0.162 0.148 0.137 0.029
(58) (59) (60) (61) (62) (63) (64)
0.218 0.096 0.192 0.162 0.207 0.109 0.011
(56) (63) (59) (61) (58) (62) (64)
extremely high (r = .9771). The top-ranked journal using weighted averaging, Journal of Political Economy, actually ranked fourth after the impact factor was adjusted. International Affairs dropped 13 places, from 25th to 3&h, while two journals changed by 10 places: Canadian Journal of Political Science (from 35th to 45th) and Public Interest (17th to 27th). Two titles varied by nine places: Urban Affairs Quarterfy increased from 30th to 21st, while Political Theory changed from 33th to 24th. Only four journals (Social Forces, Public Opinion Quarterly, American Politics Quarterly, and Politics and Society) displayed the identical ranking under both methods. When the top ranked journals under weighted averaging are compared with the leading journals using adjusted impact factor, nine of the top ten and 17 of the top 20 overlap. Thus, it is apparent that the differences between the rankings by weighted averaging and adjusted impact factor are significant and for some journals quite striking, despite the high correlation between the two methods. These differences could certainly have an impact on the decision-making process. It is noteworthy that for 46 journals, the adjusted impact factors are higher than those obtained by crude averaging and higher than the weighted averaging result for 48 journals (with one journal having an identical rating for both the adjusted impact factor and weighted averaging). The mean adjusted impact factor for the entire set was 0.790, contrasted to a mean of 0.668 by crude averaging and 0.666 by weighted averaging. This phenomenon undoubtedly reflects the fact that calculating the adjusted impact factor based on a two-year lag allows a greater opportunity for citations to be received.
CONCLUSION This research paper’s primary objective has been to point out the methodological error inherent in averaging JCR impact factor data, as earlier researchers who have done so have
456
T. E. NISONGER
not addressed the hidden bias this introduces into the results. In fact, because this bias is probably not intuitively obvious, there is a danger future researchers or practitioners could repeat and compound the error. While the supporting evidence used in this paper is from the Social Sciences Citation Index JCR, the bias would also apply to the Science Citation Index JCR, as its impact factor is calculated in an identical manner. The “adjusted impact factor” formula proposed here represents one possible strategy for correcting the bias. This particular correction is not endorsed as infallible. It too may contain hidden biases, e.g., journals that tend to be cited quickly could be disadvantaged due to the two-year lag period. Other options are undoubtedly available. For example, averaging impact factor data from alternate years would also correct the bias. There is an obvious trade-off in terms of effort versus accuracy when using the SSCI JCR impact factor data. To achieve maximum precision requires considerably more time spent recalculating the statistics, while the most efficient utilization of the available statistics sacrifices some degree of accuracy. Researchers and libraries will have to determine their own priorities regarding efficiency versus accuracy. The author presumes that rigorous researchers would definitely wish to compensate for the bias discussed in this paper. Practicing librarians, who would normally consider impact factor as only one of many variables (e.g., local usage, curricular relevance, indexing, cost, subjective judgment, etc.) in reaching journal subscription and cancellation decisions, may not require as high a degree of accuracy, although they should be aware of this methodological bias. Questions for further research include: 1. the degree of year-to-year fluctuation in JCR data-an issue the author is presently investigating; 2. what alternate approaches (in addition to the ones suggested here) are available for correcting the bias analyzed in this paper; 3. whether a different type of adjustment, such as a one-year rather than a two-year lag, to the impact factor might be more appropriate for scientific disciplines in which immediacy of citation would be more important; 4. what would be the implications of using a one-year rather than a two-year lag in calculating the adjusted impact factor; 5. the extent to which the results of earlier journal rankings would differ if a correction were made for the bias; and 6. what proportion of academic collection development librarians use the JCR, how they are using it, and for what purposes. This issue is significant because it relates to the effective utilization of JCR citation data and the accurate evaluation of scholarly journals- both important topics in contemporary library and information science. As researchers and library practitioners will undoubtedly continue to use JCR impact factor data for journal evaluation and ranking, an ongoing dialogue concerning the accurate use of this tool is imperative.
AcknowledgementsStephen P. Harter, Professor, Indiana University, School of Library and is gratefully acknowledged for providing the inspiration to write this research note by pointing methodological issue analyzed here. Blaise Cronin, Dean; Charles H. Davis, Visiting Scholar; ciate Professor; and Judith Serebnick, Associate Professor (all at Indiana University’s SLIS) able input. The author’s graduate assistants, Karen Fouchereaux and Cindy Pierard, photocopied JCR.
Information Science, out to the author the Debora Shaw, Assoare thanked for valupages from the SSCI
A Methodological Issue
457
NOTES 1. Gross, P.L.K. and Gross, E.M. “College Libraries and Chemical Education,” Science, 66 (October 28, 1927), 385-389. 2. Nisonger, Thomas E. Collection Evaluation in Academic Libraries: A Literature Guide and Annotated Bibliography. Englewood, CO: Libraries Unlimited, 1992. 3. Through 1988 the JCR was issued annually as a bound volume in both the SSCI and the XI. Beginning in 1989 both JCRs were issued separately in microfiche format. 4. The five criteria are: (1) total citations received; (2) impact factor; (3) immediacy index, i.e., how quickly a journal is cited; (4) items published in the current year; and (5) citations in the current year to issues of the journal published during the two preceding years. 5. Garfield, Eugene. “Citation Analysis as a Tool in Journal Evaluation,” Science, 178 (November 3, 1972), 477. 6. Smith, Thomas E. “The Journal Citation Reports as a Deselection Tool,” Bulletin of the Medical Library Association, 73 (October 1985), 388. 7. Broadus, Robert N. “A Proposed Method for Eliminating Titles from Periodical Subscription Lists,” College & Research Libraries 46 (January 1985), 30-35. 8. Dombrowski, Theresa. “Journal Evaluation Using Journal Citation Reports as a Collection Development Tool,” Collection Management, 10, nos. 3/4 (1988), 175-180. 9. Leavy, Marvin D. “The Journal Citation Reports as a Tool in Periodical Subscription Decisions: An Illustration and an Evaluation,” delivered at the Annual Fall Meeting of the Kansas Library Association/College and University Libraries Section in Hutchinson Holidome, Friday, October 10, 1980. ERIC Document 197 732, pp. 10. 10. Line, Maurice B. “Use of Citation Data for Periodicals Control in Libraries: A Response to Broadus,” College & Research Libraries, 46 (January 1985), 36-37. 11. Buffardi, Louis C. and Nichols, Julia A. “Citation Impact, Acceptance Rate, and APA Journals,” American Psychologist, 36 (November 1981), 1456. 12. Dometrius, Nelson C. “Subjective and Objective Measures of Journal Stature,” Social Science Quarterly, 70 (March 1989), 202. 13. Vinkler, P. “Bibliometric Features of Some Scientific Subfields and the Scientometric Consequences Therefrom,” Scientometrics, 14, nos. 5/6 (1988), 453. 14. Ribbe, Paul H. “Assessment of Prestige and Price of Professional Publications,” American Mineralogist, 73 (May/June 1988), 449-469. 15. Tomer, Christinger. “A Statistical Assessment of Two Measures of Citation: the Impact Factor and the lmmediacy Index,” Information Processing & Management, 22, no. 3 (1986), 251-258. 16. Woodward, A.M. and Hensman, Sandy. “Citations to Review Serials,” Journal of Documentation, 32 (December 1976), 290-293. 17. Scanlan, Brian D. “Coverage by Current Contents and the Validity of Impact Factors: IS1 from a Journal Publisher’s Perspective,” Serials Librarian, 13 (October/November 1987), 57-66. 18. Van Raan, A.F.J. and Hartmann, D. “The Comparative Impact of Scientific Publications and Journals: Methods of Measurement and Graphical Display,” Scientometrics, 11, nos. 5/6 (1987), 325-33 1. 19. Hirst, Graeme and Talent, Nadia. “Computer Science Journals-An Iterated Citation Analysis,” IEEE Transactions on Professional Communication, PC-20 (December 1977), 233-238. 20. Hirst, Graeme. “Discipline Impact Factor? A Method for Determining Core Journal Lists,” Journal of the American Society for Information Science, 29 (July 1978), 171-172. 21. He, Chunpei and Pao, Miranda Lee. “A Discipline-Specific Journal Selection Algorithm,” Information Processing & Management, 22, no. 5 (1986), 405-416. 22. Nieuwenhuysen, P. and Rosseau, R. “A Quick and Easy Method to Estimate the Random Effect on Citation Measures,” Scientometrics, 13, nos. l/2 (1988), 45-52. 23. Nieuwenhuysen, Paul. “Journal Citation Measures: Taking into Account Their Fluctuations from Year to Year,” Journal of Information Science, 15, no. 3 (1989), 175-178. 24. Bonitz, M. “Journal Ranking by Selective Impact: New Methods Based on SD1 Results and Journal Impact Factors,” Scientometrics, 7, nos. 3/6 (1985), 471-485. 25. Vinkler, P. “Evaluation of Some Methods for the Relative Assessment of Scientific Publications,” Scientometrics, 10, nos. 3/4 (1986), 157-177. 26. Vinkler, P. “A Quasi-Quantitative Citation Model,” Scientometrics, 12, nos. l/2 (1987), 47-72. 27. Vinkler, “Bibliometric Features of Some Scientific Subfields,” 453-474. 28. Schubert, A. and Braun, T. “Relative Indicators and Relational Charts for Comparative Assessment of Publication Output and Citation Impact,” Scientometrics, 9, nos. 5/6 (1986), 281-291.
458
T. E. NISONGER
29. Sen, B.K. “Normalized Impact Factor,” Journal of Documentation, 48 (September 1992), 318-325. 30. Christenson, James A. and Sigelman, Lee. “Accrediting Knowledge: Journal Stature and Citation Impact in Social Science,” Social Science Quarterly, 66 (December 1985), 964-975. 31. Giles, Michael W. and Wright, Gerald C., Jr. “Political Scientists’ Evaluations of Sixty-Three Journals,” PS: Political Science & Politics, 8 (Summer 1975), 254-256. 32. Glenn, Norval D. “American Sociologists’ Evaluations of Sixty-Three Journals,” American Sociologist, 6 (November 1971), 298-303. 33. Feingold, Alan. “Assessment of Journals in Social Science Psychology,” American Psycholog&, 44 (June 1989), 961-964. 34. Colson, Harold. “Citation Rankings of Public Administration Journals,” Administration & Society, 21 (February 1990), 452-471. 35. Stack, Steven. “Measuring the Relative Impacts of Criminology and Criminal Justice Journals: A Research Note,” Justice Quarterly, 4 (September 1987), 475-484. 36. Ribbe, “Assessment of Prestige,” 449-469. 37. Ribbe, Paul H. “A Scientist’s Assessment of a Microcosm of the Serials Universe,” Serials Librarian, 17, nos. 3/4 (1990), 121-142. 38. Stack, “Measuring the Relative Impacts,” p. 478. 39. Colson, “Citation Rankings,” p. 454. 40. Christenson and Sigelman, “Accrediting Knowledge,” p. 967. 41. Buffardi and Nichols, “Citation Impact,” p. 1454. 42. Gordon, Michael D. “Citation Ranking Versus Subjective Evaluation in the Determination of Journal Hierarchies in the Social Sciences,” Journal of the American Society for Information Science, 33 (January 1982), 57. 43. Vlachy, Jan. “Trends in the Citation Impact of Physics Journals, 1974-1983,” Czechoslovak Journal of Physics, B 35, no. 2 (1985), 187-190. 44. Sievert, MaryEllen C. “Ten Years of the Literature of Online Searching: An Analysis of Online and Online Review,” Journal of the American Society for Information Science, 41 (December 1990), 560-568. 45. Giles, Michael W., Mizell, Francie, and Patterson, David. “Political Scientists’ Journal Evaluations Revisited,” PS Political Science & Politics, 22 (September 1989), 613-617. 46. The “adjusted impact factor” ranking was used in Nisonger, Thomas E. “A Ranking of Political Science Journals Based on Citation Data,” Serials Review, 19, no. 4 (1993), 7-14.