- Email: [email protected]
Quantification of the performance of research units: A simple mathematical model
477
Quantification of the performance of research units: A simple mathematical model *
K-J. dkadetnie
CZERWUN der Wissenschaften der DDR, Wissenschaftliches informationszentmm. SrhifShauerdamm 19, DDR-IQ4O &rlin, <;. .&R.
Final version
received
August
1989
In the present study a formal model is proposed to quantify the performance of research units. A system of coupled linear equations is derived in which “objective” (bihtiometric) indicators as welt as peer rating are taken into account, i.e. this appraach involves the combined use of several indicators, each reflecting a different aspect of recognition by &he scientific community. To overcome the dichotomy of bibliomettic measures versus peer review, the weighting factors of peer-rating parameters include in a self-consistent way all other evaluation parameters.
1. Introduction ConsiderabIe effort has been expended to overcome the problem of finding appropriate output measures of the performance of research units (i-e_ universities, university departments, faculties, research institutes, research groups, industrial R&D laboratories, etc.). Such output indicators to quantify the results of R&D activities are, among others, the following: annual number of publications in international scientific key journals; publications per researcher; number of citations (excluding self-citations) to these publications as a measure of &he actual
* The authors express their thanks to the referees for critica! remarks. One of the aythors (W.E.) thanks M. Gebei for helpful discussions. Research Policy 19 (1990) 477-480 North-Holland 1X)48-7333/90/$3.50
0 1990 - Elsevier Science Publishers
influence of research output on surrounding scientific activities; citations per researcher or per paper; ratio of observed and expected number of citations (this indicator reflects the influence of publications in comparison with an average publication on a given subject field); number af patents; patents per researcher; peer evaluation (including or excluding selfranking): invitations to attend important international conferences (number of plenary lectures): international co-operation; number of awards. These and others measures are now being used in an increasing number of studies as indicators of the assessment of basic research, and partially as decision-making tools for research policy (project evaluation, establishment of research priorities, allocation of resources to different research institutions, etc.). Especially bibliometric, i.e. publication and citation indicators, in some cases complemented by peer-evaluation data have been applied by several authors (e.g. [l-7]) for such studies. Although bibliometric measures and peer ranking reflect only some relevant aspects of the performance of research units, we can comprehend these indicators as a system of “converging partial factors” [S] which provides a reliable estimate of the contribution to scientific progress made by different research units.
B.V. (North-Holland)
H. Englisch und H. -.I. Crenvon
478
Subsequently we propose a simple mathematical model of the annual assessment of a set of research units which includes “objective”, i.e. quantitative parameters such as bibliometric, as well as “subjective” ones, i.e. a mutual peer evaluation of the research units, where the opinion of a more successful unit gets a higher weight for the evaluation. ’ In the scientometric literature it was, to our knowledge, assumed that all peers possess the same competence. But the practice of awarding prizes [9] or the allocation of funds often proceeds from the expectation that higher ranked scientists or groups are more competent for the research evaluation.
2. Model and discussion The set of research units should contain k elements. Let us denote the evaluation parameter of the performance of the i th unit in the n th year of quantification by p,(n) 2 0. q,,> 0 is the quality parameter for the research of the ith unit according to the judgement of the jth unit. Furthermore, let us denote the part of p,(n) which is calculated from “objective” parameters by o,(n) 2 0. Then the weight w,(n) of the judgement of the jth unit in the nth year shall comprehend, besides o,(n), the mean value of the quality parameter of former years, i.e. +
P,(l)
+ ... +t,(n
W,(4
= O/(4
Thus,
our first formulation
p,(n)
= o,(n)
+
- 1)
n-l
4,l WI(~)
(1)
for p,(n) 2 0 will be: +
...
+q,kWk(n).
(2)
Which parameters can be incorporated in to o,? The kind of parameters and their weights depend, of course, on the specific subject. Bibliometric indicators have long been accepted as output indicators in basic research. In applied research one can ask for the number of patents. The determination of the weights of the several components of o, and of the relationship between “objective” and
A measure of the relative standing of journals in stratified networks proposed by Doreian possesses analogous properties: citations from journals with a higher standing contribute more than citations from journals of a lower standing; and higher interjournal citation rates count more than lower rates [El.
/ Performance
of
research units
“subjective” parameters is especially a task for peers. In our view, for instance, the weighting of the “objective” parameters depends on the special publication and citation habits in a subfield of basic research, i.e. we must take into account in each case the typical communication network. One could hope that the ratios between the p,(n) do not strongly depend on these weighting factors since in some case studies a relatively high correlation between peer judgements and citation indicators in reported [5,10]. A disadvantage of the algorithm (l))(2) becomes obvious for n = 1 where a mean qualitative parameter of the former years does not exist. A simple way out would be to identify w,(l) with o, (1). Another variant is to fix the weight as the mean value of the qualitative parameter of the previous years including the actual year, i.e. cqn)
=
P,(l)
+ ... +p,(n) n
(3)
This formula is not only interesting for n = 1; also for small n it can yield better founded values for the weights. Then the desired parameter p,(n) is, in a self-consistent way, given by P, ( n ) = 0, ( n ) + iq ,=,
(~-l)q”-l)+P,(4 I’
n
.
(4)
Let us denote the matrix with elements q,, by Q and the vectors with elements o,(n), p,(n) and G,(, - 1) by o(n), p(n) and iG(n - l), respectively. The vector p( n ) can now be represented by p(n)=(nZ-Q)-‘(no(n)+(n-l)QK+r-l)), (5) where Z is the identity matrix. Contrary to the ansatz (l)-(2), the matrix Q must now fulfil some additional conditions. This will be obvious with the choice Q = nZ. For example, it is sufficient that the eigenvector of Q with positive elements has an eigenvalue less than n since, owing to the theorem by Perron and Frobenius [ll], every eigenvalue is then less than n, which implies that the series expansion of (nZ - Q) -’ converges to a matrix with non-negative matrix elements. But non-negative values of p,(n) are, in many cases, desirable, for example if a research grant with total amount r was to be divided among the units
H. Englixh
proportional
to
their
P,(n)-r/fP,(@)+ p;(n) is at least as not reclaimed for For small Q the proximated by I/n
reputation according to (The positivity of ... +p,(n)). long as necessary, as funds are an insufficient perfarmance.) expression (nl - Q)--l is ap+ Q/n2; thus we get from (5):
This expression has a form similar transformation of (1) leads to p(n)
=o(n)
and H. J. Czetwon
+ QO(E> + QZ(n
to (I), since a
- 1).
(7)
For small Q the difference between the algorithms (l)--(2) and (3)-(4) consist only in the weights before the second and third summand in (6) and (7). As an illustration of our method we apply it to the data published by Martin and Irvine [S] in an empirical study of four radio astronomy centres. Let us assume we want to estimate the magnitude of the cont~butions of the investigated centres over the ten-year period 1969-.I978 only from the mutual peer evaluation (table 13) and the 1978 citations to 1969-1978 publications (table 8). Therefore, k = 4, n = 1, oT(l) = (1120, 340, 1030, 1340) (T denotes the transposition) and
/ Performunce
o/reseurch
units
479
price that the inversion of a (k x ~)~matrix inconvenient for k r 4 on a pocket calculator.
is
3. Conclusions Which main conclusions may be drawn from the methodology described in this paper? First, the principal advantage of the lneth~ology lies in its ability to unify the relative merits of both bibliometric measures and peer review in one quantitative model. Future studies could point out what combination of peer evaIuation and bibliometric indicators provides the best method for science policy makers. Generally speaking, we think that more injection of quantitative methods of research evaluation into science policy would be good. Second, the present model raises the possibility of systematically tracking the performance of research institutions over time. Last, not least, the application of a single index of performance seems to be desirable, especially in case that only some few research units (as in small countries) act as peers. Then in our approach the influence of misjudgements can be diminished.
References
Q=
0.3 0.115 0.085 j 0.275
0.245 0.21 0.16 0.235
0.32 0.095 0.135 0.265
0.305 0.095 0.185 ’ 0.32 1
R.C. Anderson, P. Narin and P. M~AIlister. Publication Ratings versus Peer Ratings of Universities, Journul OJ
where the entries in Q are the differences to 8 (the maximal value in table 13) from the published figures divided by 20 such that the best research institute gets, in our scale, the highest values, and that the entries are distinctly less than 1. We neglect the fact that the figures in table 13 of ref. [S] are built up by integer ranks between 1 and 9, and do not indicate that the difference in quality between successive ranks is not equidistant. The aIgorithm (2) with w(l) = o(l) leads, after normalization to percent, to ~~(1) = 32.4%, 10.9%, 22.3%, 34.5%), whereas the refined method (5) yields p“(l) = (34.68, 13.2%, f&2%, 34.0%). In comparison with the percentage of citations (29.2%, 8.9%, X.9%, 35-O%) it shows that the second method indeed takes the peer evaluation better into account; however, we must pay the
So&&for
Information
Sciewe
29 (1978) 91-103.
Endfer, J.P. Rushton and H.L. Roediger Hi, Productivity and Scholarly impact (~itatjons) of British, Canadian. and U.S. Departments of Psychology (1975). Americcm Psychohoiogisr 33 (1978) 1064-1082. T. Roche and J.L. Smith, Frequency of Citations as a Criterion for the Ranking of Departments, Journals, and Individuals, Sociologicul Inquiry 48 (1978) 49-59. A. Schubert and T. Braun, Some Scientometric Measures of Publishing Performance for 85 Hungarian Research Institutes, S~ie~f~~~efries 3 (1981) 379-388. B.R. Martin and J. Irvine, Assessing Basic Research. Some Partial Indicators of Scientific Progress in Radio Astronomy, Reseurch Policy 12 (1983) 61-90. HF. Moed. W.J.M. Burger, J.G. Frankfort and A.F.J. van Raan. A Camparative Study of Bibiiometric Past Performance Analysis and Peer Jud~ement, ~~rieffi~~err~c.~8 (1985) 149--259. H.F. Moed. W.J.M. Burger. J.G. Frankfort and A.F.J. van Raan, The Use of BibIiometric Data for the Measurement of University Research Performance, Reseurch Pdicy 14 (1985) 131-149. N.S.
480
H. Englisch and H. J. Czerwon / Performunce
[8] P. Doreian, A Measure of Standing of Journals in Stratified Networks, Screntometrm 8 (1985) 341-363. [9] G. Kiippers, P. Weingart and N. Ulitzka. Die Nohelpreise wx Physlk und Chemie 1901~ 1929. Muter&en mm Nominterungsproress (B.-Kleine-Verlag, Bielefeld, 1982). [lo] S.M. Lawani and A.E. Bayer, Validity of Citation Criteria
of research unirs
for Assessing the Influence of Scientific Publications: New Evidence with Peer Assessment, Journal o/ the Ameruxn Society /or Informuiion Science 34 (1983) 59-66. [ll] 0. Perron. Zur Theorie der Matrices, Mathemafische Ann&n 64 (1907) 261.
Quantification of the performance of research units: A simple mathematical model *
K-J. dkadetnie
CZERWUN der Wissenschaften der DDR, Wissenschaftliches informationszentmm. SrhifShauerdamm 19, DDR-IQ4O &rlin, <;. .&R.
Final version
received
August
1989
In the present study a formal model is proposed to quantify the performance of research units. A system of coupled linear equations is derived in which “objective” (bihtiometric) indicators as welt as peer rating are taken into account, i.e. this appraach involves the combined use of several indicators, each reflecting a different aspect of recognition by &he scientific community. To overcome the dichotomy of bibliomettic measures versus peer review, the weighting factors of peer-rating parameters include in a self-consistent way all other evaluation parameters.
1. Introduction ConsiderabIe effort has been expended to overcome the problem of finding appropriate output measures of the performance of research units (i-e_ universities, university departments, faculties, research institutes, research groups, industrial R&D laboratories, etc.). Such output indicators to quantify the results of R&D activities are, among others, the following: annual number of publications in international scientific key journals; publications per researcher; number of citations (excluding self-citations) to these publications as a measure of &he actual
* The authors express their thanks to the referees for critica! remarks. One of the aythors (W.E.) thanks M. Gebei for helpful discussions. Research Policy 19 (1990) 477-480 North-Holland 1X)48-7333/90/$3.50
0 1990 - Elsevier Science Publishers
influence of research output on surrounding scientific activities; citations per researcher or per paper; ratio of observed and expected number of citations (this indicator reflects the influence of publications in comparison with an average publication on a given subject field); number af patents; patents per researcher; peer evaluation (including or excluding selfranking): invitations to attend important international conferences (number of plenary lectures): international co-operation; number of awards. These and others measures are now being used in an increasing number of studies as indicators of the assessment of basic research, and partially as decision-making tools for research policy (project evaluation, establishment of research priorities, allocation of resources to different research institutions, etc.). Especially bibliometric, i.e. publication and citation indicators, in some cases complemented by peer-evaluation data have been applied by several authors (e.g. [l-7]) for such studies. Although bibliometric measures and peer ranking reflect only some relevant aspects of the performance of research units, we can comprehend these indicators as a system of “converging partial factors” [S] which provides a reliable estimate of the contribution to scientific progress made by different research units.
B.V. (North-Holland)
H. Englisch und H. -.I. Crenvon
478
Subsequently we propose a simple mathematical model of the annual assessment of a set of research units which includes “objective”, i.e. quantitative parameters such as bibliometric, as well as “subjective” ones, i.e. a mutual peer evaluation of the research units, where the opinion of a more successful unit gets a higher weight for the evaluation. ’ In the scientometric literature it was, to our knowledge, assumed that all peers possess the same competence. But the practice of awarding prizes [9] or the allocation of funds often proceeds from the expectation that higher ranked scientists or groups are more competent for the research evaluation.
2. Model and discussion The set of research units should contain k elements. Let us denote the evaluation parameter of the performance of the i th unit in the n th year of quantification by p,(n) 2 0. q,,> 0 is the quality parameter for the research of the ith unit according to the judgement of the jth unit. Furthermore, let us denote the part of p,(n) which is calculated from “objective” parameters by o,(n) 2 0. Then the weight w,(n) of the judgement of the jth unit in the nth year shall comprehend, besides o,(n), the mean value of the quality parameter of former years, i.e. +
P,(l)
+ ... +t,(n
W,(4
= O/(4
Thus,
our first formulation
p,(n)
= o,(n)
+
- 1)
n-l
4,l WI(~)
(1)
for p,(n) 2 0 will be: +
...
+q,kWk(n).
(2)
Which parameters can be incorporated in to o,? The kind of parameters and their weights depend, of course, on the specific subject. Bibliometric indicators have long been accepted as output indicators in basic research. In applied research one can ask for the number of patents. The determination of the weights of the several components of o, and of the relationship between “objective” and
A measure of the relative standing of journals in stratified networks proposed by Doreian possesses analogous properties: citations from journals with a higher standing contribute more than citations from journals of a lower standing; and higher interjournal citation rates count more than lower rates [El.
/ Performance
of
research units
“subjective” parameters is especially a task for peers. In our view, for instance, the weighting of the “objective” parameters depends on the special publication and citation habits in a subfield of basic research, i.e. we must take into account in each case the typical communication network. One could hope that the ratios between the p,(n) do not strongly depend on these weighting factors since in some case studies a relatively high correlation between peer judgements and citation indicators in reported [5,10]. A disadvantage of the algorithm (l))(2) becomes obvious for n = 1 where a mean qualitative parameter of the former years does not exist. A simple way out would be to identify w,(l) with o, (1). Another variant is to fix the weight as the mean value of the qualitative parameter of the previous years including the actual year, i.e. cqn)
=
P,(l)
+ ... +p,(n) n
(3)
This formula is not only interesting for n = 1; also for small n it can yield better founded values for the weights. Then the desired parameter p,(n) is, in a self-consistent way, given by P, ( n ) = 0, ( n ) + iq ,=,
(~-l)q”-l)+P,(4 I’
n
.
(4)
Let us denote the matrix with elements q,, by Q and the vectors with elements o,(n), p,(n) and G,(, - 1) by o(n), p(n) and iG(n - l), respectively. The vector p( n ) can now be represented by p(n)=(nZ-Q)-‘(no(n)+(n-l)QK+r-l)), (5) where Z is the identity matrix. Contrary to the ansatz (l)-(2), the matrix Q must now fulfil some additional conditions. This will be obvious with the choice Q = nZ. For example, it is sufficient that the eigenvector of Q with positive elements has an eigenvalue less than n since, owing to the theorem by Perron and Frobenius [ll], every eigenvalue is then less than n, which implies that the series expansion of (nZ - Q) -’ converges to a matrix with non-negative matrix elements. But non-negative values of p,(n) are, in many cases, desirable, for example if a research grant with total amount r was to be divided among the units
H. Englixh
proportional
to
their
P,(n)-r/fP,(@)+ p;(n) is at least as not reclaimed for For small Q the proximated by I/n
reputation according to (The positivity of ... +p,(n)). long as necessary, as funds are an insufficient perfarmance.) expression (nl - Q)--l is ap+ Q/n2; thus we get from (5):
This expression has a form similar transformation of (1) leads to p(n)
=o(n)
and H. J. Czetwon
+ QO(E> + QZ(n
to (I), since a
- 1).
(7)
For small Q the difference between the algorithms (l)--(2) and (3)-(4) consist only in the weights before the second and third summand in (6) and (7). As an illustration of our method we apply it to the data published by Martin and Irvine [S] in an empirical study of four radio astronomy centres. Let us assume we want to estimate the magnitude of the cont~butions of the investigated centres over the ten-year period 1969-.I978 only from the mutual peer evaluation (table 13) and the 1978 citations to 1969-1978 publications (table 8). Therefore, k = 4, n = 1, oT(l) = (1120, 340, 1030, 1340) (T denotes the transposition) and
/ Performunce
o/reseurch
units
479
price that the inversion of a (k x ~)~matrix inconvenient for k r 4 on a pocket calculator.
is
3. Conclusions Which main conclusions may be drawn from the methodology described in this paper? First, the principal advantage of the lneth~ology lies in its ability to unify the relative merits of both bibliometric measures and peer review in one quantitative model. Future studies could point out what combination of peer evaIuation and bibliometric indicators provides the best method for science policy makers. Generally speaking, we think that more injection of quantitative methods of research evaluation into science policy would be good. Second, the present model raises the possibility of systematically tracking the performance of research institutions over time. Last, not least, the application of a single index of performance seems to be desirable, especially in case that only some few research units (as in small countries) act as peers. Then in our approach the influence of misjudgements can be diminished.
References
Q=
0.3 0.115 0.085 j 0.275
0.245 0.21 0.16 0.235
0.32 0.095 0.135 0.265
0.305 0.095 0.185 ’ 0.32 1
R.C. Anderson, P. Narin and P. M~AIlister. Publication Ratings versus Peer Ratings of Universities, Journul OJ
where the entries in Q are the differences to 8 (the maximal value in table 13) from the published figures divided by 20 such that the best research institute gets, in our scale, the highest values, and that the entries are distinctly less than 1. We neglect the fact that the figures in table 13 of ref. [S] are built up by integer ranks between 1 and 9, and do not indicate that the difference in quality between successive ranks is not equidistant. The aIgorithm (2) with w(l) = o(l) leads, after normalization to percent, to ~~(1) = 32.4%, 10.9%, 22.3%, 34.5%), whereas the refined method (5) yields p“(l) = (34.68, 13.2%, f&2%, 34.0%). In comparison with the percentage of citations (29.2%, 8.9%, X.9%, 35-O%) it shows that the second method indeed takes the peer evaluation better into account; however, we must pay the
So&&for
Information
Sciewe
29 (1978) 91-103.
Endfer, J.P. Rushton and H.L. Roediger Hi, Productivity and Scholarly impact (~itatjons) of British, Canadian. and U.S. Departments of Psychology (1975). Americcm Psychohoiogisr 33 (1978) 1064-1082. T. Roche and J.L. Smith, Frequency of Citations as a Criterion for the Ranking of Departments, Journals, and Individuals, Sociologicul Inquiry 48 (1978) 49-59. A. Schubert and T. Braun, Some Scientometric Measures of Publishing Performance for 85 Hungarian Research Institutes, S~ie~f~~~efries 3 (1981) 379-388. B.R. Martin and J. Irvine, Assessing Basic Research. Some Partial Indicators of Scientific Progress in Radio Astronomy, Reseurch Policy 12 (1983) 61-90. HF. Moed. W.J.M. Burger, J.G. Frankfort and A.F.J. van Raan. A Camparative Study of Bibiiometric Past Performance Analysis and Peer Jud~ement, ~~rieffi~~err~c.~8 (1985) 149--259. H.F. Moed. W.J.M. Burger. J.G. Frankfort and A.F.J. van Raan, The Use of BibIiometric Data for the Measurement of University Research Performance, Reseurch Pdicy 14 (1985) 131-149. N.S.
480
H. Englisch and H. J. Czerwon / Performunce
[8] P. Doreian, A Measure of Standing of Journals in Stratified Networks, Screntometrm 8 (1985) 341-363. [9] G. Kiippers, P. Weingart and N. Ulitzka. Die Nohelpreise wx Physlk und Chemie 1901~ 1929. Muter&en mm Nominterungsproress (B.-Kleine-Verlag, Bielefeld, 1982). [lo] S.M. Lawani and A.E. Bayer, Validity of Citation Criteria
of research unirs
for Assessing the Influence of Scientific Publications: New Evidence with Peer Assessment, Journal o/ the Ameruxn Society /or Informuiion Science 34 (1983) 59-66. [ll] 0. Perron. Zur Theorie der Matrices, Mathemafische Ann&n 64 (1907) 261.