- Email: [email protected]
Conclusions
14
Conclusions No a priori law exists. There are individuals, groups of individuals and things with common and particular properties. Relationships between individuals or things may be regarded as interactions between the corresponding characteristics. A law may be derived by the regular repetition of the same interaction. In scientometrics – as in several other fields of science and social science – ‘laws’ cannot be regarded as strict rules; they rather represent probabilities. On the basis of scientific knowledge, however, predictions are possible. The question arises as to whether predictions can also be made based on scientometric knowledge. The answer is yes. If a paper is published in a journal with high Garfield (Impact) Factor (GF), we may estimate with high probability that it will obtain a higher number of citations than a paper in a journal with low GF. Nevertheless, one should always take into account the skewed distribution of citations among the papers in a journal. Or, a paper of a scientist with relatively high π-index (or Hirsch index) (Chapter 7) will be more frequently cited than a paper of an individual with low π-index (or Hirsch index). Naturally, the probability of the prediction is the higher the higher the difference between the π-indices (or Hirsch indices). Similarly, the higher the hierarchical level of the study, the higher the probability of the prediction. This feature can be attributed to the statistical character of scientometric relationships. According to E. Kant science is a system of organised knowledge. Defining things, phenomena and correlations, parallel with the classification of items of the study, is inevitable in any scientific discipline. Finding common and specific characteristics of the items belonging to the same system, and distinguishing these items from those belonging to another system, is a prerequisite of classification. The lack of generally accepted theories, definitions, rules and classification of knowledge may impede the development of scientific fields. Several authors in scientometrics have suggested the need for
257
The Evaluation of Research by Scientometric Indicators
coherent, organised knowledge in the field (see Chapter 2). I proposed a uniform description of aspects, elements, sets, systems, measures, units and indicators of evaluative scientometrics, and gave a review on the possibilities of applying the most frequently used indicators (Chapter 3). A great number of scientometric indicators have been suggested in the literature, and the number of indices is continuing to grow. According to Chapter 3 we may distinguish three main types of indicators based on formal and functional aspects, namely gross, complex and composite indices. Complex indicators generally apply reference standards. The most frequently used indices are the specific (e.g. number of citations by paper) and relative indices (e.g. Relative Subfield Citedness, RW). The suitability of the indicators in evaluation depends strongly on the reliability and relevance of the data and calculation method used. The dynamic change of information in science can be followed by the change of the number of journal papers, references and citations (Chapters 4 and 11). This can reveal the development, stagnation or decline of individual topics or even fields. Different models (linear, exponential, logistic) are used to show the growth of the scientific literature (Chapter 4). The Relative Publication Growth (RPG) index has been introduced to characterise the publication increase. RPG is equal to the total number of papers published in a year related to the sum of papers published in a preceding period of 2, 5 or 10, etc., years. Changes in the mean RPG index and Mean Annual Percentage Rate of the increase in the number of journal papers run in parallel. The publication development of different chemical fields (e.g. biochemistry, organic chemistry and polymer chemistry), for example, was found to be significantly different in the period 1970–2000. The main research trends in the fields of science can be detected by scientometric mapping. However, individual creativity and knowledge are required to select individual research topics with promising future results. Science is an information-producing activity, the essence of which is communication. The majority of information in the sciences is published in journals. Consequently, determination of the eminence of journals and impact of the individual publications in the journals represent the most important issues in evaluative scientometrics. Several traditional and recently introduced indicators are available in the scientometric literature. The GF as suggested by Garfield (1979) is a traditional and frequently applied index, which may be regarded as the philosopher’s stone (‘ultima ratio’) in scientometrics. Chapter 5 shows that the GF corresponds to the mean probability of the citedness of journal papers, which can be calculated as the product of
258
Conclusions
the respective RPG index and the mean number of the corresponding references in journal papers. A new interpretation of the GF is given, which verifies the use of this indicator for characterising the eminence of scientific journals. It has been verified that the normalized GF and the normalized Specific Impact Contribution (SIC) indices are identical measures. The SIC index relates the percentage share of citations obtained by a journal within the total citations received by all journals in the field to its share in publications. It has also been shown that the RW indicator corresponds to the SIC index within any set of journals. From the above, the conclusion may be drawn that the GF of journals should not be assumed as the mean citedness of papers in a given journal. In contrast, the GF of a journal measures the relative contribution of the journal to the total impact of the journals devoted to the respective scientific field. Consequently, GF characterises the journal as a whole entity and it is only formally equal to the citedness of the ‘mean’ paper therein. It is shown further (Chapter 5) that the mean global Chance for Citedness (CC) index of the papers (i.e. possible number of citations per paper) in a field is constant if the Relative Publication Growth of the field is constant (‘steady-state field’, i.e. the numbers of papers published yearly are identical), while the CC index increases in rapidly developing fields (e.g. yearly number of papers: 100, 105, 115, 130). In contrast, it decreases in constantly developing fields (e.g. yearly number of papers: 100, 110, 120). One of the most serious problems in evaluation of the eminence of journals is how to consider the different effects of the bibliometric factors on the number of citations in different science fields. The answer is that appropriate reference standards should be found (see Chapters 5 and 11). The eminence of journals can be represented by short-term indicators (e.g. GF). The Current Contribution Index (CCI) was suggested for assessing long-term impact (Chapter 5). CCI represents the relative contribution of a journal (in terms of its share in citations received) to the total recent, relevant knowledge (RRK) of the corresponding field or subfield. RRK is defined as the body of information required for generating new information. It may be approximated by the total number of references recently published in journals devoted to the given field. No significant correlation was found between the GF and CCI of journals in several fields of science. CCI data showed significant correlation with the number of papers published, while GF indices significantly correlated with the number of total citations obtained. More recently, the h-index and π-index (see Chapter 7) have been applied for characterising the eminence of journals.
259
The Evaluation of Research by Scientometric Indicators
After disclosing scientific information in journals, a period of complex revelation and evaluation may begin with participation of fellow scientists (Chapters 6 and 13). The length and results of these processes depend primarily on the inherent scientific values and practical usefulness of the information presented. Several models describing the ageing of publications are known. Studies on the Annual Citedness Rate of journals revealed that the highest rate of impact could be obtained in the 2nd to 4th year after the publication year, on average. The ageing rate of information in chemical journals was found to be about 6 per cent annually. This means that the information value of journal papers (assuming 100 per cent in the peak year) may be only several per cent after 17–20 years. Naturally, the ageing rate of information depends strongly on the field. Applying the function of ageing, we can predict the possible number of citations available during a given period from the year with the maximum value of Annual Citedness (number of citations in a year). It is often claimed in the literature that journals or scientists of relatively high standard may obtain more citations than deserved. The phenomenon has been termed the Matthew effect. Nevertheless, a reverse effect can also be observed. Namely, many papers published in journals of high GF will be cited less frequently than the ‘average’ paper in that journal. This regularity has been termed as Invitation Paradox. Metaphorically: ‘For many are called, but few are chosen’ (The Gospel according to St Matthew) (Chapter 7). The Publication Strategy (PS) indicator (Chapter 7) characterises the average impact of the publication channels applied by the authors. By measuring the eminence of journals through the GF, the PS index gives the (weighted) mean GF of the journals where the papers studied are published. The indicator should not be regarded as an impact index of the corresponding papers or authors. Instead, the PS indicator shows the potential of the authors in selecting journals. The selection is orientated primarily based on the quality and overlap of information in the paper to be published with that of the publishing journal. Nevertheless, several nonscientific (‘connectional’ or ‘social’) factors may also influence the decision on the place of publication and acceptance of the paper by the journal. The PS index depends highly on the different bibliometric factors of the field. Therefore, application of the Relative Publication Strategy (RPS) indicator may be preferred. The RPS index relates the (weighted) mean GF of the publishing journals to the (weighted) mean GF of the journals devoted to the corresponding field. The first relative impact indicator (Relative Citation Rate, RCR) was introduced by Schubert et al. (1983) (Chapter 7). The index is used to
260
Conclusions
provide comparative assessment of the international impact of journal papers in similar or different fields. The reference standard of the RCR index is selected by the authors themselves, and is equal to the PS indicator. In this way, the index relates the Journal Paper Citedness (JPC, i.e. citations per paper) value of the set of papers studied to the PS index (i.e. mean GF of the publishing journals) of the corresponding authors. In contrast, the Relative Subfield Citedness (RW) indicator suggested by me (Vinkler, 1986) relates the mean number of citations obtained by a set of papers to the mean citation rate of the journal papers in the corresponding field. In this way, the RW index applies an objective standard, which is independent of the publishing authors. The following relationship exists between the above-mentioned relative impact indicators (RW, RCR and RPS), within a given scientometric system: RW can be obtained as the product of RPS and RCR (Chapter 7). The relative indicators of some countries may show a particular feature. There are countries which publish a substantial number of their papers in local journals with relatively low GF, and thus their PS index is low. Accordingly, their RCR indicator will be relatively high (about 0.5–0.8) compared with their RW index (0.3–0.4). The RW index is believed to give more reliable information on international impact than RCR. In calculating relevant relative indicators, the selection of an appropriate standard is essential. Scientific progress is made primarily by information acknowledged by a high number of citations. Consequently, highly cited papers represent the most important category of journal papers. The set of papers that are highly cited can be termed ‘the elite set’. The number of papers in the elite set (Pπ) may be approximated by the square root of total papers (√P) (Chapter 7). For sets consisting of a very high number of items (e.g. papers in a journal), the following equation was introduced: Pπ = 10(log P) – 10. For representing scientific eminence, a new impact index (π-index) was introduced, which measures the number of citations, C(Pπ), obtained by the most influential papers (i.e. papers in the elite set): π-index = 0.01 C(Pπ). The elite set concept offers new potential for introducing comparable impact indicators for individuals, teams, countries or journals. Evaluative scientometrics has been threatened recently by publications published without substantial scientometric knowledge or without appropriate mathematical and statistical analyses, or without considering the physical meaning of the data and indicators applied, as well as possible consequences. The introduction of the Hirsch index exploded the repository of sleeping scientometric indices. Several indices, which are often artefacts,
261
The Evaluation of Research by Scientometric Indicators
are used without preliminary verification and this fact raises the question of responsibility among professional scientometricians. Many indicators can be calculated but only a few are worth applying for evaluation (see, Invitation Paradox, Chapter 7). Scientometric impact indicators can be verified directly or indirectly. The former process refers to relating the indices to accepted non-scientometric (social) eminence indices (peer review or number of PhD dissertations made in the laboratory assessed, memberships and awards of the researchers, etc.). Possible accordance of the data may be regarded as verification. Indirect proof of the indices involves the accordance of the new index with an ‘old’, verified index (i.e. similar results obtained by the πv-index or π-index and total number of citations). Evaluation of the publications of individuals may, in particular, be inconsistent (see, for example, the different rates of dependent, i.e. self-, citations of researchers and different shares of the contribution of co-authors). Only those indicators that are in accordance with some basic scientometric considerations may be recommended for scientometric evaluation. Such basic requirements include: the citation shows impact; the highly cited papers exert greater impact than other papers. Consequently, an eminence index should be higher if highly cited papers receive more citations (see h-index and π-index, Chapter 7). The study of citations to scientific publications became available with the establishment of the Science Citation Index (Garfield, 1979). The citations may represent a category of evaluation of scientific impact, while references may be regarded as a category of information. The mean impact of the input information of scientific research may be characterised by the mean of the (weighted) impact indicators (e.g. GF) of the journals referenced (Reference Strategy, RS) (Chapter 8). By relating the RS index of the authors selected to an appropriate standard (e.g. mean GF of the journals in the field), the Relative Reference Strategy (RRS) index can be calculated. It has been found that authors preferably refer to journals of higher international influence (first law of Reference Strategy). It may be concluded further that the international impact of information sources preferably referred to is more homogeneous than the impact of the set of journals referencing (third law of Reference Strategy). The communication process in science may also proceed through publications. The model of Manifested Communication through Publications (MCP) postulates the following consecutive steps in a single communication cycle for a bilateral case: information emission, absorption, impact and reflection (Chapter 8). A complete communication cycle contains seven steps. As a proof for the ‘manifested
262
Conclusions
information impact’ made by author A on author B, the reference in a paper of author B to the paper of A is regarded. A manifested communication cycle assumes that author A reveals the corresponding paper of author B, and publishes a new paper, which refers to that paper. The frequency of the communication can be calculated by an information matrix (INFOTRIX Model). The citation may be regarded as the scientometric unit of impact in evaluative scientometrics. Consequently, it is highly important to show that the referencing process represents a particular professional peer review, which may show the influence of the results in the publication on the referencing author. Considering relative frequency and strength of motivations of the authors referencing, the Reference Threshold Model was established (Chapter 9). The study verifies that the references in natural science publications are the results of a thorough evaluation process and the decisive share of references is made, partly or exclusively, for professional motives with relatively low Reference Threshold. In contrast, the Reference Threshold of references made for ‘connectional’ (i.e. nonscientific, social) motivations is significantly higher. According to my study about 73 per cent of the total relevant publications are referenced directly or indirectly (e.g. Planck constant), while the share of publications that are discarded after consideration may be about 20 per cent. The share of publications referenced for exclusively connectional reasons was found to be very low (1.2 per cent). In modern natural sciences the number of researchers participating in research projects and thus the number of co-authors of journal papers is steadily increasing (Chapter 10). In applying for grants, positions or awarding prizes, etc., determination of individual merit is highly relevant. The present study reveals that two types of activity (experimental work and data analysis) may represent about 55–60 per cent of the total efforts in creating a chemical paper. According to the Correct Credit Distribution Scores model, the rank of co-authors may reflect the approximate measure of contribution. The individual share of credit of the second co-authors decreases, for example, as follows: 71.4, 56.3, 47.4, 41.6 and 37.4 per cent for a 2-, 3-, 4-, 5- or 6-authored publication, respectively. Scientific cooperation between laboratories is considered to yield mutual advantages. Several authors are of the opinion that papers made in cooperation may attract higher numbers of independent citations. Other authors disagree. In my opinion, the impact of a multi-authored paper is primarily influenced by the merit of the paper and by the individual
263
The Evaluation of Research by Scientometric Indicators
eminence of the participating authors and no additional (synergetic) effect can be attributed only to the higher number of cooperating partners. In evaluative scientometrics standardisation refers to eliminating or at least decreasing the effect of factors influencing scientometric indicators, which are beyond the control of the individuals or teams assessed. Bibliometric factors [e.g. type of research results published (i.e. basic, applied, methodological, theoretical); type of publication (i.e. journal papers, books, conference proceedings); ageing rate of information; and progress and coherence of the field], which may influence the value of the scientometric indicators, may be highly different by field or even by topic (Chapter 11). The bibliometric factors strongly influence, for example, the mean number of references and the GF of journals. Consequently, to calculate appropriate standards is an important part of any scientometric evaluation. The Cross-field Factor (CF) describes how many times the mean GF of the journals in a field should be multiplied to obtain the highest mean GF among the fields studied. In 2004, for example, the following CF values were calculated (selecting the mean GF of journals in biochemistry and molecular biology as unity): organic chemistry, 1.78; analytical chemistry, 2.11; chemical engineering, 3.96; mathematics, 6.56. To obtain standards by research field or subfield, the journals belonging to those fields or subfields should be classified first. The ‘Item-by-item’ subject classification (Glänzel and Schubert, 2003) combined with the ‘Characteristic scores and scales’ method (Braun et al., 1990, Glänzel et al., 2007) is highly appropriate for this purpose (Chapter 11). Determining cognitive resemblance between referencing and referenced publications seems to be the best method for classifying journals or sets of papers by topic. By analysing profile similarities (Peters et al. 1995), we may obtain sets of publications which are thematically more or less homogeneous. Scientometric assessment cannot be performed as a routine, as in analytical measurements in chemistry. According to Moravcsik (1988): ‘It is not possible to prescribe an assessment procedure, which holds universally’. In most cases assessment may be regarded as a study, which requires at a minimum a basic knowledge of the methods of evaluative scientometrics. Therefore, it is of paramount interest that science managers, science officers, science politicians and even scientists learn the fundamental issues behind scientometrics. Direct comparison of the performance indicators of teams working in different fields is not possible – except the application of relative indicators. It is possible, however, to detect the ordinal rank number of the team assessed in the corresponding field by a selected indicator (e.g. JPC; citations per paper), and to compare this rank number with the rank
264
Conclusions
of a team working in another field (standardising between fields). The total number of the teams working in the corresponding fields, however, may greatly influence the value of the rank number. Standardisation of indicators within a field can be carried out by calculating, for example, Z-scores. Scientometrics may yield indispensable data and indicators for the science policy of each hierarchical level (individuals, teams, institutes, countries, topics, disciplines, etc.) in the following fields:
monitoring the performance of research organisations, obtaining information for starting, granting or closing research projects, selecting research priorities, studying the structure of research in a country, studying the relationships between science and society or science and technology, etc.
It has been shown (Chapter 12) that both evaluation methods (peer review and quantitative methods) which are performed by fellow scientists are based primarily on the data of publications and citations. Peer assessments are limited in time and number of evaluators, and they are planned and orientated. In contrast, evaluation by citations is unlimited in time and number of evaluators, and the process is not centralised or orientated (Chapter 12). With the increasing hierarchical level of scientometric assessment, the number of items analysed increases in general. Parallel with this increase, the reliability of the results increases (Chapters 11 and 12). This can be traced back to the statistical nature of scientometric analyses. The chances of obtaining appropriate standards are also higher at higher hierarchical levels. However, several regularities, which may be valid for greater sets, do not hold for lower levels. Consequently, there are several methodological difficulties in applying scientometric relationships and indicators at lower hierarchical levels (e.g. individuals). This means that applicability and validity of the data and indicators used in the assessment should be investigated on the level of individuals. In scientometrics only a whole set (e.g. all journals in a field) may represent a ‘meta set’ (‘meta journal’), which may correctly represent all features of the corresponding scientometric system. Indicators calculated for individual (‘real’) sets, which should be analysed in practice, always differ from the indicators calculated for a theoretically ideal ‘whole’ set. Before performing scientometric analyses of publications, some basic assumptions should be made: 265
The Evaluation of Research by Scientometric Indicators
the citation is regarded as the scientometric unit of impact of scientific information,
higher scientific impact is revealed by a higher number of citations, the journal paper is regarded as the scientometric unit of information in the fields of natural and life sciences,
the amount of information produced is related to the number of journal papers published,
publishing more frequently in periodicals of high eminence means a greater impact may be attained. For correct assessment, reliable data, appropriate methods and relevant indicators should be obtained, constructed and used (Chapter 12). Concerning the evaluation of scientific eminence, one group of scientists is of the opinion that:
Scientific eminence cannot be or, even should not be, represented by quantitative indicators. Scientific results can be evaluated only by experts in the specific field. Scientometric indicators are not reliable for assessing individuals or teams; only publications and patents, etc., of countries might be analysed by indicators. The second group of scientists believes that:
Quantitative indicators may represent scientific eminence ‘more or less’ correctly. According to some experts of evaluation methods (and also scientists in different fields), it would be possible to apply a single indicator (e.g. total number of citations or h-index), which could appropriately characterise the scientific eminence of individuals, teams, journals or even countries. Most scientometricians are, however, of the view that scientific eminence could be approximated only by several indicators describing different aspects. The application of composite indicators (Chapter 7) may be regarded as a synthesis of the above views. The classical works of evaluative scientometrics by Martin and Irvine (1983, 1984, 1985) and Irvine and Martin (1984) show a wide spectrum of input and output indicators. According to these authors partial indicators [(papers per researcher), (citations per paper), (number of highly cited papers related to the total), etc.] reflecting adequate aspects of the research activity assessed may yield convergent results. Convergence of the indicators shows that the conclusion drawn is correct (Chapter 12).
266
Conclusions
Science and scientific research is multifaceted, and consequently several indicators may be applied for characterising the different aspects. However, in order to obtain reasonable results, applicable in practice, one should select a limited number of indicators characterising the most important aspects only. Science politicians can apply only definite answers [as binary digits: yes (1) or no (0)] to practical questions, such as: should the project be granted or not, should we start with or stop research activity in a given field, may this person be promoted to this position or not, etc? Therefore, application of Occam’s principle in scientometrics is recommended (Chapter 12). Accordingly, the number of indicators applied in evaluations should be reduced to the possible lowest but still sufficient number of indicators. With regard to Occam’s principle in scientometrics, a Composite Publication Indicator is suggested (Chapter 12) for evaluating the publications of teams, laboratories or countries. The indicator consists of four part-indices: Journal Paper Productivity (JPP), Relative Publication Strategy (RPS), Relative Subfield Citedness (RW) and Highly Cited Paper Productivity (HCPP). JPP takes into account the amount of information by each researcher produced during a year, RPS characterises the mean relative eminence of the publication channels used and RW reflects the mean relative impact of the journal papers published. The HCPP indicator represents the specific amount of most important publications. Weighting of the individual part-indices depends on the purposes of the assessment. A part-index can be calculated by dividing the corresponding index (e.g. RW) of a team by the sum of the indices of the teams evaluated. The composite indicator may represent a linear combination of the partindices selected. The study of publications at laboratory or team level may offer useful information for local science policy-makers and also for the researchers themselves. For example, using appropriate scientometric indicators, the grant offered for basic research can be distributed among the research teams of an institute (Chapter 12). Scientometric assessment made without taking necessary measures with consequences is useless (or even frustrating). According to an empirical investigation (Chapter 12) the structure of science in a country was found to be characteristic of the developmental grade of science in that country. The Mean Structural Difference (MSD) values calculated for several countries show that life sciences represent a significantly greater share within science in highly developed countries
267
The Evaluation of Research by Scientometric Indicators
than in transitional or underdeveloped countries. In contrast, physics and chemistry are overrepresented in Central and Eastern European countries. From a study of the relationship between GDP and development of science, the conclusion could be drawn that up to a certain level of economic development the production of basic science information does not increase the wealth of an underdeveloped country, but on an advanced economic and social level, further development will not be possible without increasing the level of maintenance of fundamental research. Research priorities should be selected by taking into account primarily the requirements of the national economy and society, traditions and results previously attained, possible present and future human and financial potential, international relationships, trends in the world’s economic and social growth, and trends of science. In revealing past performance, present potentials and trends in science, scientometrics offers considerable help. Maps of research fields and subfields dynamically studied (by revealing frequency and links of publications by topic) may help science politicians to select research priorities on a country scale. Science policy is practised in most countries, however, rather as policy than as science. Nevertheless, I hope that in most countries, scientometrics may soon contribute to turning qualitative science policy into an activity substantiated by quantitative indicators. The core idea of the Scientometric Model of Institutionalisation of Scientific Information (ISI-S model) is that the higher the number of citations obtained by a publication and the longer the citing time period, the deeper the incorporation of information in the publication into the body of knowledge of the field (Chapter 13). In the development of science, processes of generating, evaluating and modifying information are most important. Converting information into knowledge and integrating the pieces of information into the relevant knowledge body are primarily done by distinguished authors, who also publish reviews, monographs or books, are editors or members of editorial boards, and deliver plenary or invited lectures at scientific meetings. The ISI-S model describes the possible development of scientific information published through different evaluation and modification processes toward a cognitive consensus of distinguished authors of the corresponding scientific field. The ISI-S model assumes sets of relevant information with a short- or long-term impact, aged information, faulty or redundant information and information that is integrated into basic scientific knowledge or common knowledge. According to the ISI-S model, the rate of development of science is significantly slower than the rate of increase in the number of publications.
268
Conclusions
Making progress in any branch of science requires knowledge and creativity in research, appropriate conditions for the work and opportunity. Development in a selected period depends greatly also on the internal potential of the field. Science and scientometrics (and even art) may be advanced also by external factors, i.e. the interests, initiatives or even demands of individuals or society. Evaluative scientometrics has two aspects: basic and applied. Science politicians may profit primarily from the applicable results of scientometrics. In several countries increasing emphasis is being laid on public accountability of science. There is an increasing demand for the application of scientometric methods also in the nomination of individuals for a position or award, for distributing grants or for selecting national priorities. Are evaluative scientometrics in vigorous development or in a period of stagnation? The extensive trend of the corresponding publications can easily be demonstrated. Searching for the items ‘scientometric*’ and ‘indicator*’, for example, referenced by Google Scholar, the following trend was found: 1970: 0; 1975: 2; 1980: 38; 1985: 64; 1990: 100; 1995: 137; 2000: 243; 2005: 510; 2008: 760. I have tried here to report on all important results attained in evaluative scientometrics. Nevertheless, I am aware that there may be relevant publications that I have omitted. There are several valuable papers, however, which were not available at the time of writing. The available scientometrics literature shows that great efforts and intensive development are being made. As a result, scientometrics is significantly contributing to revealing the changing mechanism of information processes in science and the methods of evaluative scientometrics have become an indispensible tool for science policy-makers. I hope that this publication offers a modest contribution to the development and application of evaluative scientometrics.
269
Conclusions No a priori law exists. There are individuals, groups of individuals and things with common and particular properties. Relationships between individuals or things may be regarded as interactions between the corresponding characteristics. A law may be derived by the regular repetition of the same interaction. In scientometrics – as in several other fields of science and social science – ‘laws’ cannot be regarded as strict rules; they rather represent probabilities. On the basis of scientific knowledge, however, predictions are possible. The question arises as to whether predictions can also be made based on scientometric knowledge. The answer is yes. If a paper is published in a journal with high Garfield (Impact) Factor (GF), we may estimate with high probability that it will obtain a higher number of citations than a paper in a journal with low GF. Nevertheless, one should always take into account the skewed distribution of citations among the papers in a journal. Or, a paper of a scientist with relatively high π-index (or Hirsch index) (Chapter 7) will be more frequently cited than a paper of an individual with low π-index (or Hirsch index). Naturally, the probability of the prediction is the higher the higher the difference between the π-indices (or Hirsch indices). Similarly, the higher the hierarchical level of the study, the higher the probability of the prediction. This feature can be attributed to the statistical character of scientometric relationships. According to E. Kant science is a system of organised knowledge. Defining things, phenomena and correlations, parallel with the classification of items of the study, is inevitable in any scientific discipline. Finding common and specific characteristics of the items belonging to the same system, and distinguishing these items from those belonging to another system, is a prerequisite of classification. The lack of generally accepted theories, definitions, rules and classification of knowledge may impede the development of scientific fields. Several authors in scientometrics have suggested the need for
257
The Evaluation of Research by Scientometric Indicators
coherent, organised knowledge in the field (see Chapter 2). I proposed a uniform description of aspects, elements, sets, systems, measures, units and indicators of evaluative scientometrics, and gave a review on the possibilities of applying the most frequently used indicators (Chapter 3). A great number of scientometric indicators have been suggested in the literature, and the number of indices is continuing to grow. According to Chapter 3 we may distinguish three main types of indicators based on formal and functional aspects, namely gross, complex and composite indices. Complex indicators generally apply reference standards. The most frequently used indices are the specific (e.g. number of citations by paper) and relative indices (e.g. Relative Subfield Citedness, RW). The suitability of the indicators in evaluation depends strongly on the reliability and relevance of the data and calculation method used. The dynamic change of information in science can be followed by the change of the number of journal papers, references and citations (Chapters 4 and 11). This can reveal the development, stagnation or decline of individual topics or even fields. Different models (linear, exponential, logistic) are used to show the growth of the scientific literature (Chapter 4). The Relative Publication Growth (RPG) index has been introduced to characterise the publication increase. RPG is equal to the total number of papers published in a year related to the sum of papers published in a preceding period of 2, 5 or 10, etc., years. Changes in the mean RPG index and Mean Annual Percentage Rate of the increase in the number of journal papers run in parallel. The publication development of different chemical fields (e.g. biochemistry, organic chemistry and polymer chemistry), for example, was found to be significantly different in the period 1970–2000. The main research trends in the fields of science can be detected by scientometric mapping. However, individual creativity and knowledge are required to select individual research topics with promising future results. Science is an information-producing activity, the essence of which is communication. The majority of information in the sciences is published in journals. Consequently, determination of the eminence of journals and impact of the individual publications in the journals represent the most important issues in evaluative scientometrics. Several traditional and recently introduced indicators are available in the scientometric literature. The GF as suggested by Garfield (1979) is a traditional and frequently applied index, which may be regarded as the philosopher’s stone (‘ultima ratio’) in scientometrics. Chapter 5 shows that the GF corresponds to the mean probability of the citedness of journal papers, which can be calculated as the product of
258
Conclusions
the respective RPG index and the mean number of the corresponding references in journal papers. A new interpretation of the GF is given, which verifies the use of this indicator for characterising the eminence of scientific journals. It has been verified that the normalized GF and the normalized Specific Impact Contribution (SIC) indices are identical measures. The SIC index relates the percentage share of citations obtained by a journal within the total citations received by all journals in the field to its share in publications. It has also been shown that the RW indicator corresponds to the SIC index within any set of journals. From the above, the conclusion may be drawn that the GF of journals should not be assumed as the mean citedness of papers in a given journal. In contrast, the GF of a journal measures the relative contribution of the journal to the total impact of the journals devoted to the respective scientific field. Consequently, GF characterises the journal as a whole entity and it is only formally equal to the citedness of the ‘mean’ paper therein. It is shown further (Chapter 5) that the mean global Chance for Citedness (CC) index of the papers (i.e. possible number of citations per paper) in a field is constant if the Relative Publication Growth of the field is constant (‘steady-state field’, i.e. the numbers of papers published yearly are identical), while the CC index increases in rapidly developing fields (e.g. yearly number of papers: 100, 105, 115, 130). In contrast, it decreases in constantly developing fields (e.g. yearly number of papers: 100, 110, 120). One of the most serious problems in evaluation of the eminence of journals is how to consider the different effects of the bibliometric factors on the number of citations in different science fields. The answer is that appropriate reference standards should be found (see Chapters 5 and 11). The eminence of journals can be represented by short-term indicators (e.g. GF). The Current Contribution Index (CCI) was suggested for assessing long-term impact (Chapter 5). CCI represents the relative contribution of a journal (in terms of its share in citations received) to the total recent, relevant knowledge (RRK) of the corresponding field or subfield. RRK is defined as the body of information required for generating new information. It may be approximated by the total number of references recently published in journals devoted to the given field. No significant correlation was found between the GF and CCI of journals in several fields of science. CCI data showed significant correlation with the number of papers published, while GF indices significantly correlated with the number of total citations obtained. More recently, the h-index and π-index (see Chapter 7) have been applied for characterising the eminence of journals.
259
The Evaluation of Research by Scientometric Indicators
After disclosing scientific information in journals, a period of complex revelation and evaluation may begin with participation of fellow scientists (Chapters 6 and 13). The length and results of these processes depend primarily on the inherent scientific values and practical usefulness of the information presented. Several models describing the ageing of publications are known. Studies on the Annual Citedness Rate of journals revealed that the highest rate of impact could be obtained in the 2nd to 4th year after the publication year, on average. The ageing rate of information in chemical journals was found to be about 6 per cent annually. This means that the information value of journal papers (assuming 100 per cent in the peak year) may be only several per cent after 17–20 years. Naturally, the ageing rate of information depends strongly on the field. Applying the function of ageing, we can predict the possible number of citations available during a given period from the year with the maximum value of Annual Citedness (number of citations in a year). It is often claimed in the literature that journals or scientists of relatively high standard may obtain more citations than deserved. The phenomenon has been termed the Matthew effect. Nevertheless, a reverse effect can also be observed. Namely, many papers published in journals of high GF will be cited less frequently than the ‘average’ paper in that journal. This regularity has been termed as Invitation Paradox. Metaphorically: ‘For many are called, but few are chosen’ (The Gospel according to St Matthew) (Chapter 7). The Publication Strategy (PS) indicator (Chapter 7) characterises the average impact of the publication channels applied by the authors. By measuring the eminence of journals through the GF, the PS index gives the (weighted) mean GF of the journals where the papers studied are published. The indicator should not be regarded as an impact index of the corresponding papers or authors. Instead, the PS indicator shows the potential of the authors in selecting journals. The selection is orientated primarily based on the quality and overlap of information in the paper to be published with that of the publishing journal. Nevertheless, several nonscientific (‘connectional’ or ‘social’) factors may also influence the decision on the place of publication and acceptance of the paper by the journal. The PS index depends highly on the different bibliometric factors of the field. Therefore, application of the Relative Publication Strategy (RPS) indicator may be preferred. The RPS index relates the (weighted) mean GF of the publishing journals to the (weighted) mean GF of the journals devoted to the corresponding field. The first relative impact indicator (Relative Citation Rate, RCR) was introduced by Schubert et al. (1983) (Chapter 7). The index is used to
260
Conclusions
provide comparative assessment of the international impact of journal papers in similar or different fields. The reference standard of the RCR index is selected by the authors themselves, and is equal to the PS indicator. In this way, the index relates the Journal Paper Citedness (JPC, i.e. citations per paper) value of the set of papers studied to the PS index (i.e. mean GF of the publishing journals) of the corresponding authors. In contrast, the Relative Subfield Citedness (RW) indicator suggested by me (Vinkler, 1986) relates the mean number of citations obtained by a set of papers to the mean citation rate of the journal papers in the corresponding field. In this way, the RW index applies an objective standard, which is independent of the publishing authors. The following relationship exists between the above-mentioned relative impact indicators (RW, RCR and RPS), within a given scientometric system: RW can be obtained as the product of RPS and RCR (Chapter 7). The relative indicators of some countries may show a particular feature. There are countries which publish a substantial number of their papers in local journals with relatively low GF, and thus their PS index is low. Accordingly, their RCR indicator will be relatively high (about 0.5–0.8) compared with their RW index (0.3–0.4). The RW index is believed to give more reliable information on international impact than RCR. In calculating relevant relative indicators, the selection of an appropriate standard is essential. Scientific progress is made primarily by information acknowledged by a high number of citations. Consequently, highly cited papers represent the most important category of journal papers. The set of papers that are highly cited can be termed ‘the elite set’. The number of papers in the elite set (Pπ) may be approximated by the square root of total papers (√P) (Chapter 7). For sets consisting of a very high number of items (e.g. papers in a journal), the following equation was introduced: Pπ = 10(log P) – 10. For representing scientific eminence, a new impact index (π-index) was introduced, which measures the number of citations, C(Pπ), obtained by the most influential papers (i.e. papers in the elite set): π-index = 0.01 C(Pπ). The elite set concept offers new potential for introducing comparable impact indicators for individuals, teams, countries or journals. Evaluative scientometrics has been threatened recently by publications published without substantial scientometric knowledge or without appropriate mathematical and statistical analyses, or without considering the physical meaning of the data and indicators applied, as well as possible consequences. The introduction of the Hirsch index exploded the repository of sleeping scientometric indices. Several indices, which are often artefacts,
261
The Evaluation of Research by Scientometric Indicators
are used without preliminary verification and this fact raises the question of responsibility among professional scientometricians. Many indicators can be calculated but only a few are worth applying for evaluation (see, Invitation Paradox, Chapter 7). Scientometric impact indicators can be verified directly or indirectly. The former process refers to relating the indices to accepted non-scientometric (social) eminence indices (peer review or number of PhD dissertations made in the laboratory assessed, memberships and awards of the researchers, etc.). Possible accordance of the data may be regarded as verification. Indirect proof of the indices involves the accordance of the new index with an ‘old’, verified index (i.e. similar results obtained by the πv-index or π-index and total number of citations). Evaluation of the publications of individuals may, in particular, be inconsistent (see, for example, the different rates of dependent, i.e. self-, citations of researchers and different shares of the contribution of co-authors). Only those indicators that are in accordance with some basic scientometric considerations may be recommended for scientometric evaluation. Such basic requirements include: the citation shows impact; the highly cited papers exert greater impact than other papers. Consequently, an eminence index should be higher if highly cited papers receive more citations (see h-index and π-index, Chapter 7). The study of citations to scientific publications became available with the establishment of the Science Citation Index (Garfield, 1979). The citations may represent a category of evaluation of scientific impact, while references may be regarded as a category of information. The mean impact of the input information of scientific research may be characterised by the mean of the (weighted) impact indicators (e.g. GF) of the journals referenced (Reference Strategy, RS) (Chapter 8). By relating the RS index of the authors selected to an appropriate standard (e.g. mean GF of the journals in the field), the Relative Reference Strategy (RRS) index can be calculated. It has been found that authors preferably refer to journals of higher international influence (first law of Reference Strategy). It may be concluded further that the international impact of information sources preferably referred to is more homogeneous than the impact of the set of journals referencing (third law of Reference Strategy). The communication process in science may also proceed through publications. The model of Manifested Communication through Publications (MCP) postulates the following consecutive steps in a single communication cycle for a bilateral case: information emission, absorption, impact and reflection (Chapter 8). A complete communication cycle contains seven steps. As a proof for the ‘manifested
262
Conclusions
information impact’ made by author A on author B, the reference in a paper of author B to the paper of A is regarded. A manifested communication cycle assumes that author A reveals the corresponding paper of author B, and publishes a new paper, which refers to that paper. The frequency of the communication can be calculated by an information matrix (INFOTRIX Model). The citation may be regarded as the scientometric unit of impact in evaluative scientometrics. Consequently, it is highly important to show that the referencing process represents a particular professional peer review, which may show the influence of the results in the publication on the referencing author. Considering relative frequency and strength of motivations of the authors referencing, the Reference Threshold Model was established (Chapter 9). The study verifies that the references in natural science publications are the results of a thorough evaluation process and the decisive share of references is made, partly or exclusively, for professional motives with relatively low Reference Threshold. In contrast, the Reference Threshold of references made for ‘connectional’ (i.e. nonscientific, social) motivations is significantly higher. According to my study about 73 per cent of the total relevant publications are referenced directly or indirectly (e.g. Planck constant), while the share of publications that are discarded after consideration may be about 20 per cent. The share of publications referenced for exclusively connectional reasons was found to be very low (1.2 per cent). In modern natural sciences the number of researchers participating in research projects and thus the number of co-authors of journal papers is steadily increasing (Chapter 10). In applying for grants, positions or awarding prizes, etc., determination of individual merit is highly relevant. The present study reveals that two types of activity (experimental work and data analysis) may represent about 55–60 per cent of the total efforts in creating a chemical paper. According to the Correct Credit Distribution Scores model, the rank of co-authors may reflect the approximate measure of contribution. The individual share of credit of the second co-authors decreases, for example, as follows: 71.4, 56.3, 47.4, 41.6 and 37.4 per cent for a 2-, 3-, 4-, 5- or 6-authored publication, respectively. Scientific cooperation between laboratories is considered to yield mutual advantages. Several authors are of the opinion that papers made in cooperation may attract higher numbers of independent citations. Other authors disagree. In my opinion, the impact of a multi-authored paper is primarily influenced by the merit of the paper and by the individual
263
The Evaluation of Research by Scientometric Indicators
eminence of the participating authors and no additional (synergetic) effect can be attributed only to the higher number of cooperating partners. In evaluative scientometrics standardisation refers to eliminating or at least decreasing the effect of factors influencing scientometric indicators, which are beyond the control of the individuals or teams assessed. Bibliometric factors [e.g. type of research results published (i.e. basic, applied, methodological, theoretical); type of publication (i.e. journal papers, books, conference proceedings); ageing rate of information; and progress and coherence of the field], which may influence the value of the scientometric indicators, may be highly different by field or even by topic (Chapter 11). The bibliometric factors strongly influence, for example, the mean number of references and the GF of journals. Consequently, to calculate appropriate standards is an important part of any scientometric evaluation. The Cross-field Factor (CF) describes how many times the mean GF of the journals in a field should be multiplied to obtain the highest mean GF among the fields studied. In 2004, for example, the following CF values were calculated (selecting the mean GF of journals in biochemistry and molecular biology as unity): organic chemistry, 1.78; analytical chemistry, 2.11; chemical engineering, 3.96; mathematics, 6.56. To obtain standards by research field or subfield, the journals belonging to those fields or subfields should be classified first. The ‘Item-by-item’ subject classification (Glänzel and Schubert, 2003) combined with the ‘Characteristic scores and scales’ method (Braun et al., 1990, Glänzel et al., 2007) is highly appropriate for this purpose (Chapter 11). Determining cognitive resemblance between referencing and referenced publications seems to be the best method for classifying journals or sets of papers by topic. By analysing profile similarities (Peters et al. 1995), we may obtain sets of publications which are thematically more or less homogeneous. Scientometric assessment cannot be performed as a routine, as in analytical measurements in chemistry. According to Moravcsik (1988): ‘It is not possible to prescribe an assessment procedure, which holds universally’. In most cases assessment may be regarded as a study, which requires at a minimum a basic knowledge of the methods of evaluative scientometrics. Therefore, it is of paramount interest that science managers, science officers, science politicians and even scientists learn the fundamental issues behind scientometrics. Direct comparison of the performance indicators of teams working in different fields is not possible – except the application of relative indicators. It is possible, however, to detect the ordinal rank number of the team assessed in the corresponding field by a selected indicator (e.g. JPC; citations per paper), and to compare this rank number with the rank
264
Conclusions
of a team working in another field (standardising between fields). The total number of the teams working in the corresponding fields, however, may greatly influence the value of the rank number. Standardisation of indicators within a field can be carried out by calculating, for example, Z-scores. Scientometrics may yield indispensable data and indicators for the science policy of each hierarchical level (individuals, teams, institutes, countries, topics, disciplines, etc.) in the following fields:
monitoring the performance of research organisations, obtaining information for starting, granting or closing research projects, selecting research priorities, studying the structure of research in a country, studying the relationships between science and society or science and technology, etc.
It has been shown (Chapter 12) that both evaluation methods (peer review and quantitative methods) which are performed by fellow scientists are based primarily on the data of publications and citations. Peer assessments are limited in time and number of evaluators, and they are planned and orientated. In contrast, evaluation by citations is unlimited in time and number of evaluators, and the process is not centralised or orientated (Chapter 12). With the increasing hierarchical level of scientometric assessment, the number of items analysed increases in general. Parallel with this increase, the reliability of the results increases (Chapters 11 and 12). This can be traced back to the statistical nature of scientometric analyses. The chances of obtaining appropriate standards are also higher at higher hierarchical levels. However, several regularities, which may be valid for greater sets, do not hold for lower levels. Consequently, there are several methodological difficulties in applying scientometric relationships and indicators at lower hierarchical levels (e.g. individuals). This means that applicability and validity of the data and indicators used in the assessment should be investigated on the level of individuals. In scientometrics only a whole set (e.g. all journals in a field) may represent a ‘meta set’ (‘meta journal’), which may correctly represent all features of the corresponding scientometric system. Indicators calculated for individual (‘real’) sets, which should be analysed in practice, always differ from the indicators calculated for a theoretically ideal ‘whole’ set. Before performing scientometric analyses of publications, some basic assumptions should be made: 265
The Evaluation of Research by Scientometric Indicators
the citation is regarded as the scientometric unit of impact of scientific information,
higher scientific impact is revealed by a higher number of citations, the journal paper is regarded as the scientometric unit of information in the fields of natural and life sciences,
the amount of information produced is related to the number of journal papers published,
publishing more frequently in periodicals of high eminence means a greater impact may be attained. For correct assessment, reliable data, appropriate methods and relevant indicators should be obtained, constructed and used (Chapter 12). Concerning the evaluation of scientific eminence, one group of scientists is of the opinion that:
Scientific eminence cannot be or, even should not be, represented by quantitative indicators. Scientific results can be evaluated only by experts in the specific field. Scientometric indicators are not reliable for assessing individuals or teams; only publications and patents, etc., of countries might be analysed by indicators. The second group of scientists believes that:
Quantitative indicators may represent scientific eminence ‘more or less’ correctly. According to some experts of evaluation methods (and also scientists in different fields), it would be possible to apply a single indicator (e.g. total number of citations or h-index), which could appropriately characterise the scientific eminence of individuals, teams, journals or even countries. Most scientometricians are, however, of the view that scientific eminence could be approximated only by several indicators describing different aspects. The application of composite indicators (Chapter 7) may be regarded as a synthesis of the above views. The classical works of evaluative scientometrics by Martin and Irvine (1983, 1984, 1985) and Irvine and Martin (1984) show a wide spectrum of input and output indicators. According to these authors partial indicators [(papers per researcher), (citations per paper), (number of highly cited papers related to the total), etc.] reflecting adequate aspects of the research activity assessed may yield convergent results. Convergence of the indicators shows that the conclusion drawn is correct (Chapter 12).
266
Conclusions
Science and scientific research is multifaceted, and consequently several indicators may be applied for characterising the different aspects. However, in order to obtain reasonable results, applicable in practice, one should select a limited number of indicators characterising the most important aspects only. Science politicians can apply only definite answers [as binary digits: yes (1) or no (0)] to practical questions, such as: should the project be granted or not, should we start with or stop research activity in a given field, may this person be promoted to this position or not, etc? Therefore, application of Occam’s principle in scientometrics is recommended (Chapter 12). Accordingly, the number of indicators applied in evaluations should be reduced to the possible lowest but still sufficient number of indicators. With regard to Occam’s principle in scientometrics, a Composite Publication Indicator is suggested (Chapter 12) for evaluating the publications of teams, laboratories or countries. The indicator consists of four part-indices: Journal Paper Productivity (JPP), Relative Publication Strategy (RPS), Relative Subfield Citedness (RW) and Highly Cited Paper Productivity (HCPP). JPP takes into account the amount of information by each researcher produced during a year, RPS characterises the mean relative eminence of the publication channels used and RW reflects the mean relative impact of the journal papers published. The HCPP indicator represents the specific amount of most important publications. Weighting of the individual part-indices depends on the purposes of the assessment. A part-index can be calculated by dividing the corresponding index (e.g. RW) of a team by the sum of the indices of the teams evaluated. The composite indicator may represent a linear combination of the partindices selected. The study of publications at laboratory or team level may offer useful information for local science policy-makers and also for the researchers themselves. For example, using appropriate scientometric indicators, the grant offered for basic research can be distributed among the research teams of an institute (Chapter 12). Scientometric assessment made without taking necessary measures with consequences is useless (or even frustrating). According to an empirical investigation (Chapter 12) the structure of science in a country was found to be characteristic of the developmental grade of science in that country. The Mean Structural Difference (MSD) values calculated for several countries show that life sciences represent a significantly greater share within science in highly developed countries
267
The Evaluation of Research by Scientometric Indicators
than in transitional or underdeveloped countries. In contrast, physics and chemistry are overrepresented in Central and Eastern European countries. From a study of the relationship between GDP and development of science, the conclusion could be drawn that up to a certain level of economic development the production of basic science information does not increase the wealth of an underdeveloped country, but on an advanced economic and social level, further development will not be possible without increasing the level of maintenance of fundamental research. Research priorities should be selected by taking into account primarily the requirements of the national economy and society, traditions and results previously attained, possible present and future human and financial potential, international relationships, trends in the world’s economic and social growth, and trends of science. In revealing past performance, present potentials and trends in science, scientometrics offers considerable help. Maps of research fields and subfields dynamically studied (by revealing frequency and links of publications by topic) may help science politicians to select research priorities on a country scale. Science policy is practised in most countries, however, rather as policy than as science. Nevertheless, I hope that in most countries, scientometrics may soon contribute to turning qualitative science policy into an activity substantiated by quantitative indicators. The core idea of the Scientometric Model of Institutionalisation of Scientific Information (ISI-S model) is that the higher the number of citations obtained by a publication and the longer the citing time period, the deeper the incorporation of information in the publication into the body of knowledge of the field (Chapter 13). In the development of science, processes of generating, evaluating and modifying information are most important. Converting information into knowledge and integrating the pieces of information into the relevant knowledge body are primarily done by distinguished authors, who also publish reviews, monographs or books, are editors or members of editorial boards, and deliver plenary or invited lectures at scientific meetings. The ISI-S model describes the possible development of scientific information published through different evaluation and modification processes toward a cognitive consensus of distinguished authors of the corresponding scientific field. The ISI-S model assumes sets of relevant information with a short- or long-term impact, aged information, faulty or redundant information and information that is integrated into basic scientific knowledge or common knowledge. According to the ISI-S model, the rate of development of science is significantly slower than the rate of increase in the number of publications.
268
Conclusions
Making progress in any branch of science requires knowledge and creativity in research, appropriate conditions for the work and opportunity. Development in a selected period depends greatly also on the internal potential of the field. Science and scientometrics (and even art) may be advanced also by external factors, i.e. the interests, initiatives or even demands of individuals or society. Evaluative scientometrics has two aspects: basic and applied. Science politicians may profit primarily from the applicable results of scientometrics. In several countries increasing emphasis is being laid on public accountability of science. There is an increasing demand for the application of scientometric methods also in the nomination of individuals for a position or award, for distributing grants or for selecting national priorities. Are evaluative scientometrics in vigorous development or in a period of stagnation? The extensive trend of the corresponding publications can easily be demonstrated. Searching for the items ‘scientometric*’ and ‘indicator*’, for example, referenced by Google Scholar, the following trend was found: 1970: 0; 1975: 2; 1980: 38; 1985: 64; 1990: 100; 1995: 137; 2000: 243; 2005: 510; 2008: 760. I have tried here to report on all important results attained in evaluative scientometrics. Nevertheless, I am aware that there may be relevant publications that I have omitted. There are several valuable papers, however, which were not available at the time of writing. The available scientometrics literature shows that great efforts and intensive development are being made. As a result, scientometrics is significantly contributing to revealing the changing mechanism of information processes in science and the methods of evaluative scientometrics have become an indispensible tool for science policy-makers. I hope that this publication offers a modest contribution to the development and application of evaluative scientometrics.
269