- Email: [email protected]
A fuzzy set approach to the evaluation of journal grades
Fuzzy Sets and Systems 131 (2002) 63 – 74
www.elsevier.com/locate/fss
A fuzzy set approach to the evaluation of journal grades Duanning Zhoua; b , Jian Mac; ∗ , Efraim Turbanc , Narasimha Bollojuc b College
a Computer Science Department, Zhongshan University, Guangzhou, China of Business and Public Administration, Eastern Washington University Spokane, WA 99202-1660, USA c Information Systems Department, City University of Hong Kong, Kowloon, Hong Kong
Abstract Many universities, research institutions and government agencies are continuously attempting to grade or rank journals for their academic value. Such grading is needed for personnel decisions and for funding and resource allocation purposes. The grading of journals is dependent both on objective information, such as the impact ratios of the journals, and on subjective information, such as experts’ judgments about the journals. Also, the available information for decision may be incomplete or imprecise due to the problem nature and complexity of the decision process. This paper proposes a fuzzy set approach that integrates objective and subjective information for evaluating grades of journals. It provides a comprehensive method for dealing with incomplete and imprecise information to support the whole evaluation process. A Web-based decision support c 2002 Elsevier Science B.V. All rights system that is based on the proposed fuzzy set approach has been implemented. reserved. Keywords: Group decision-making; Fuzzy preference; Evaluating academic journals; Subjective evaluation; Objective evaluation; Decision support system
1. Introduction Evaluating the quality of academic journals is required for several personnel decisions, such as recruiting, promotion, tenure and retention. Such evaluation is also done for merit increases and for determination of research funding. Many institutions use formal grading or point systems for such evaluations. In most cases the evaluation is done by a group (a committee or a panel, in which members are called experts) which complicates the evaluation process since a consensus is required. For years, researchers ∗ Corresponding author. Tel.: +852-278-8514; fax: +852-27844198. E-mail address: [email protected] (J. Ma).
in several disciplines attempted to Bnd the most appropriate formula for such evaluations. Research on this topic was conducted in: economics [12], accounting and Bnance [1], management [3,7], information systems [8,9,10,14], real estate [5] and construction management [15]. Unfortunately, there is no consensus on how best to conduct the journal evaluation. The methodologies for journal evaluation can be classiBed as either subjective or objective, depending on how the decision information is obtained and used. The subjective approach, also called the perception analysis approach, uses questionnaires to solicit subjective information from experts such as deans, department heads, renowned practitioners, and=or academic staG members. The collected information is compiled and a ranking of the journals is done based on the
c 2002 Elsevier Science B.V. All rights reserved. 0165-0114/02/$ - see front matter PII: S 0 1 6 5 - 0 1 1 4 ( 0 1 ) 0 0 2 5 5 - X
64
D. Zhou et al. / Fuzzy Sets and Systems 131 (2002) 63 – 74
collective respondents’ perceptions. DiGerent researchers used diGerent models for aggregating subjective opinions. The work of Gillenson and Stutz [8] is an example of ranking information systems journals based on respondents’ perceptions. The objective approach, also called citation analysis approach, determines the rankings of journals based on various forms of citation counts, by tabulating citations from a set of base journals, or a set of selected articles, over a certain time period. For example, Liebowitz and Palmer [12] used a citation approach to rank economics journals, and Holsapple et al. [10] employed a citation analysis methodology to rank business computing system journals. Citation information is publicly available in a form of “total cites”, “immediacy index”, “total articles”, cited half-life and “impact ratio (or factor)”. Information about these measures is available in sources such as the Journal Citation Reports, a CD-ROM database available in most libraries and updated annually. This paper uses the impact ratio which is based on “the average number of times articles published in a speciBc journal in the two previous years were cited in a particular year”. However, the proposed methodology will accept any other quantitative measures. Evaluations and rankings of journals determined by subjective approaches alone can be inOuenced by biases, or by lack of suPcient knowledge or experience. Objective approaches completely disregard subjective judgement of experts. It is obvious that neither the subjective, nor the objective approaches by themselves are best for the evaluation. Despite the considerable research conducted on evaluation, there are few researches on how to integrate the two approaches. Also, few researches can be found with the following two issues: (1) how to deal with incomplete subjective or objective information, and (2) how to transform the evaluations to a tangible journal grade. Due to incomplete and uncertain objective information (e.g., citation information may not be available for some journals), as well as lack of suPcient knowledge, experts may Bnd it diPcult to express their preference precisely. Fuzzy set theory [16] is applied here as a tool for solving the problem of imprecise subjective judgments and incomplete objective information. Fuzzy set methods for evaluation were attempted before. For example, Biswas [2] describes a fuzzy method to evaluate students’ answerscripts; Ross [13]
provides a fuzzy method for a generic multi-criteria evaluation. However, none of the previous methods can solve our research problems. This paper proposes a fuzzy set approach to deal with the above research issues. It analyzes the past grades assigned to journals in an attempt to identify the relationship between the historical assigned grades and the impact ratios of journals at the time the grades were assigned. Based on the discovered relationship, the model produces a suggested current grading by integrating experts’ opinions with the current impact ratio of the journals. ‘A’ computerized group decision support system based on the proposed evaluation method has been developed and it is available on the Web (http://144.214.54.91/zdn). The method is designed to Bt the process of funding research in Hong Kong. However, it can be easily modiBed to cover other situations. The Hong Kong process (see Section 3 for details), involves a combination of objective information which is based on impact ratios and on historical grades, and on subjective information solicited from experts, who are organized in disciplinary panels. It assigns grades of A; B; C or zero to each journal in the evaluation process. This paper is organized as follows. Section 2 describes the proposed fuzzy set approach. Section 3 presents an application, and a summary is given in Section 4. The appendix illustrates a Web-based group decision support system developed for the evaluation of grade of journals in Hong Kong.
2. The proposed fuzzy set approach 2.1. Overview The proposed evaluation procedure is composed of 6 steps. The Brst and second steps are conducted for all journals in one discipline. The last four are done for each journal independently. The steps are: Step 1: Analyze the relationship between past journal grades and past impact ratios. Certain relationships are evidenced between the two variables. Our model expresses these relationships quantitatively. Once established, this relationship can be used as an objective component for future evaluations.
D. Zhou et al. / Fuzzy Sets and Systems 131 (2002) 63 – 74
Step 2: DeBne qualitative guidelines for expressing subjective judgments. Experts often are unable to express their subjective judgments the way they really want to do it. Therefore, the expressions may be inaccurate. By providing optional qualitative judgment statements to the experts, they are able to express their views in a more practical way. Step 3: Assign weights that express the importance the panel wishes to place on the objective and subjective portions of the evaluation. Step 4: Integrate the subjective information of all panel members and the objective information to create a fuzzy evaluation matrix. Step 5: Add the weights and aggregate the fuzzy evaluation matrix. Step 6: The aggregated results can be used as a reference point for determining the journal’s grade by the evaluators. 2.2. General notations Let G = {g1 ; g2 ; : : : ; gd } be a set of grades of journals (without loss of generality, we assume g1 is the best grade and gd the worst grade). Let P = {P1 ; P2 ; : : : ; Pt } be the set of experts and J = {J1 ; J2 ; : : : ; Jn } be the set of journals. The grades g1 ; g2 ; : : : ; gd are fuzzy sets deBned on journal set J with membership values between 0 and 1, and they can be represented as J1 J2 Jn gi = ; ;:::; a1i a2i ani (aji ∈ [0; 1]; 1 6 i 6 d and j = 1; 2; : : : ; n); where aij is the membership degree that represents the objective and subjective ranking of journal j in the fuzzy set gi . For example, A, given below, speciBes the membership values of journals J1 ; J2 ; : : : ; Jn as 0:8; 0:5; : : : ; 0:1; respectively, for grade A. J1 J2 Jn A= : ; ;:::; 0:8 0:5 0:1 2.3. Step 1: analysis of the relationship between grades and impact ratios of journals The relationships between impact ratios and grades of journals can be expressed by using the concepts
65
of membership degree and membership function in fuzzy set theory. Assume that there exist membership functions g1 ; g2 ; : : : ; gd which reOect the relationship between impact ratios and grades of journals. If we denote the impact ratio of journal Jl as Jlx (16l6n); then we can determine fuzzy set g1 ; g2 ; : : : ; gd on the consideration of impact ratios of journals, which have the following format: J2 Jn J1 gi = ; ;:::; gi (J1x ) gi (J2x ) gi (Jnx ) (i = 1; : : : ; d and gi (J1x ); : : : ; gi (Jnx ) ∈ [0; 1]): Membership functions g1 ; g2 ; : : : ; gd can be obtained by analyzing historical data at one point of time (e.g., the latest data) of impact ratios and grades of journals. The membership functions can be computed, for example, by counting how many journals received certain grade in certain intervals of the impact ratio (frequency distribution). Section 3 illustrates how such membership functions are computed using historical data of impact ratios and associated journal grades. With membership functions g1 ; g2 ; : : : ; gd ; from the point view of journal Jk ; vector (g1 (Jkx ); g2 (Jkx ); : : : ; gd (Jkx )) expresses the grade of journal Jk . This means that journal Jk belongs to grades g1 ; g2 ; : : : ; gd withmembershipdegreesg1 (Jkx ); g2 (Jkx ); : : : ; gd (Jkx ), respectively. In other words, the vector (g1 (Jkx ); g2 (Jkx ); : : : ; gd (Jkx )) represents fuzzy set “grade of journal Jk ” deBned on the grade set g1 ; g2 ; : : : ; gd . In general, with membership functions, we can deBne a fuzzy mapping as follows: f : x → (g1 (x); g2 (x); : : : ; gd (x)); where x belongs to the impact ratio set. With the membership functions g1 ; g2 ; : : : ; gd , the fuzzy mapping f gives a journal (e.g., Jk ) with an impact ratio, x; the degree to which the speciBc journal belongs to the grades of journals g1 ; g2 ; : : : ; gd , respectively. We can explain that the degrees form a fuzzy set “grade of journal Jk ” deBned on the grades set g1 ; g2 ; : : : ; gd . For example, if the impact ratio of a journal is known to be 0.9, a designation of g1 (0:9) = 0:7; g2 (0:9) = 0:5; : : : ; gd (0:9) = 0:2; means the journal belongs to the set of grades g1 ; g2 ; : : : ; gd with 0:7; 0:5; : : : ; 0:2 degrees, respectively.
66
D. Zhou et al. / Fuzzy Sets and Systems 131 (2002) 63 – 74
For each journal Jk ∈ J (16k6n); the degrees to which the journal belongs to diGerent grades g1 ; g2 ; : : : ; gd form a vector, denoted as impact ratio vector V k : V k = (g1 (Jkx ); g2 (Jkx ); : : : ; gd (Jkx )); k = 1; : : : ; n:
(1)
The vector V k reOects the objective information of journal Jk ; i.e. impact ratio of journal Jk . Next we show how the subjective information of journals is solicited and how it is combined with the objective information of vector V k . 2.4. Step 2: expression of the experts’ subjective judgments individually Assume that a group of experts, P; (P = {P1 ; P2 ; : : : ; Pt }); participates in a panel. Usually, panel members express their opinions, Brst, independent of each other. Then, they try to reach a consensus. As mentioned above, the grades g1 ; g2 ; : : : ; gd are fuzzy sets deBned on journal set, J; where membership is between 0 and 1, where J1 J2 Jn gi = ; ;:::; a1i a2i ani (aji ∈ [0; 1]; 1 6 i 6 d and j = 1; 2; : : : ; n): Such information can be expressed in the following matrix, experts need to assign values column-wise: Grades J1
g1
g2
:::
gd
a11
a12
:::
a1d
a22
:::
an2
:::
.. .
a21 :::
Jn
an1
Journals J2
a2d and
However, in practice, it may be unpractical or not easy for experts to specify their subjective judgments using fuzzy sets g1 ; g2 ; : : : ; gd and associated degrees of memberships for a set of journals. For this reason, we expect speciBcation row-wise (instead column-wise)
from experts for each journal, which is much easier for experts to estimate journals by assigning degrees on grades. Furthermore, since each expert need to Bll in information for several, sometimes dozens of journals, we provide them with an option of expressing their estimates in judgment terms. A judgment term is an ordered d-tuple a1 ; a2 ; : : : ; ad ; where ai ∈ [0; 1]; 16i6d. The values of the judgment term, i.e., d-tuple, of a journal represent the degrees that a speciBc journal belongs to the grades g1 ; g2 ; : : : ; gd ; respectively. For example, let G = {A; B; C; D}; then a statement: “The grade of the journal is a somewhat less than grade A” can be expressed as 0:9; 0:3; 0; 0 and “The grade of the journal is better than grade B” can be expressed as 0:2; 0:8; 0:1; 0 Although we provide a set of predeBned judgment terms, experts can alternatively specify the d-tuple by directly inserting numerical values instead of using the predeBned qualitative terms. 2.5. Step 3: assignment of weights The experts themselves or the panel leader can assign weights to impact ratios of diGerent journals. In addition, the panel leader can assign a weight to the opinion of the group or to the experts (either as a group, or for each expert), for each journal. In this subsection, we consider how the weight assignments are used to calculate a “weight vector” which is used to evaluate the grades of journals. Let the weight of impact ratio for journal Jk be w0k ; and the weight of the panel of experts for that Jk journal, be wek ; where w0k + wek = 1; w0k ¿0; wek ¿0. The reason that the sum of the weights is one, is that we allow this way a trade oG between the subjective and objective assessments. Suppose we assign weights wek1 ; wek2 ; : : : ; wekt one each to experts p1 ; p2 ; : : : ; pt ; re t spectively, where j=1 wekj = 1; wekj ¿0; j = 1; : : : ; t. The assignment of weights to impact ratios and to experts is presented below. Each expert t pj is assigned a weight of wjk = wek wekj ; where j=1 wjk = wek . The weight vector corresponding to journal Jk is W k = (w0k ; w1k ; : : : ; wtk ). If any expert does not give judgment information of a journal, then the weight corresponding to that expert is redistributed among
D. Zhou et al. / Fuzzy Sets and Systems 131 (2002) 63 – 74
other weights. This normalization process satisBes t the condition j=1 wekj = 1. 2.6. Step 4: integration of the experts’ opinions and objective information For expert Pi , the subjective judgment (i.e. judgment term) on the journal Jk forms a vector, denoted as Vi k = (vi1k ; vi2k ; : : : ; vidk ), where each vijk ∈ [0; 1]; (16i6t; j = 1; : : : ; d). For a panel of t experts involved in evaluating the journal Jk ; a subjective expert evaluation relation ma trix, denoted as E k = [vijk ; i = 1; : : : ; t; j = 1; : : : ; d; is formed by integrating the individual experts’ judg ment vectors Vi k ; i = 1; : : : ; t. The relation E k consist of the experts’ subjective judgments information on journal Jk . To evaluate the journal Jk both subjectively and objectively, we add the impact ratio vector V k in (1) as the Brst row elements of the matrix E k , then get the integrated evaluation relation matrix denoted as E k = [vijk ]; i = 0; 1; : : : ; t; j = 1; : : : ; d.
67
k k k ) ⊕ c(w1k • e1j ) ⊕ · · · ⊕ (wnk • enj ) where yjk = (w0k • e0j and ‘•’, ‘⊕’ are deBned as
(a) algebraic product, a • b: c = ab (b) bounded sum, a ⊕ b: c = a ⊕ b = min{1; a + b}. 2.8. Step 6: determination of the grades of the journals According to the principles of fuzzy classiBcation [6], we have yik = Max(y1k ; y2k ; : : : ; ydk ). Thus the corresponding grade gi is the Bnal grade of the journal Jk . In practical evaluation process, we set a paramek ter " to check if |yik − yi+1 |¡"; i¡d; if it happens, then we can say this journal is between grades gi and gi+1 . If |yik − yjk |¡"; and i¿j; j = i + 1; i¡d − 1; that means the experts have strong contradicting opinions on the grade of the journal. Delphi method [4] or GDSS [11] may be used to reach consensus. Alternatively, other consensus reaching methods can be applied. 3. Application in Hong Kong
2.7. Step 5: aggregation of the impact ratio and experts’ judgments For a journal Jk ; we derived the fuzzy weight W k and the fuzzy evaluation relation E k . Then, the process of determining the Bnal grade of the journal Jk is equivalent to the process of determining a membership value for the journal Jk in each of the evaluation grades g1 ; g2 ; : : : ; gd . This process can be implemented through the composition operation W k ◦ E k . The result is a fuzzy vector (evaluation vector), denoted as Y k ; containing the membership values for the journal Jk in each of evaluation grades g1 ; g2 ; : : : ; gd : Y k = W k ◦ Ek
= (w0k ; w1k ; : : : ; wtk )
k e01
k e 11 ◦ ::: k et1
= (y1k ; y2k ; : : : ; ydk );
k e02
:::
k e12
:::
::: k et2
:::
k e0d
k e1d ::: k etd (2)
Research is always one of the most important functions of a university. To improve the research performance, the University Grant Committee (UGC) in Hong Kong conducts research assessment exercise (RAE) once in every 3 years. For the purpose of this exercise, local universities form cost centers according to disciplines suggested by the UGC. Academic staG members are assigned to cost centers according to their research disciplines. A staG member submits up to 5 recently (3 years) published or accepted journal articles for RAE assessment. UGC forms disciplinary panels to review the journals and assign a grade: A: top-tier, B: mid-tier, C: lower-tier, to each journal. When an academic staG member is assessed to have 3 “B” grade or better articles published during the assessment period, he/she is most likely to become an active researcher (AR). In the previous two RAE exercises, an AR meant an allocation of research funds to the cost center equivalent to at least the researcher’s salary for the forthcoming 3 years. Thus universities in Hong Kong pay a lot of attention to the RAE exercises. From the perspective of the universities, it is very important to estimate well in advance the likely
68
D. Zhou et al. / Fuzzy Sets and Systems 131 (2002) 63 – 74
Fig. 1. The frequency of grade A on S.
Fig. 2. The frequency of grade B on S.
Fig. 3. The frequency of grade C on S.
grades of diGerent journals. These estimations can be communicated to the academic staG members so that they can identify a set of journals to publish in, as well as to decide which journals to include as the Bve “best” (in case they have more than Bve). Our methodology can be used by the universities to optimize the above decisions. It can also be used by diGerent panels to arrive at their decisions. The proposed fuzzy evaluation method was tested in evaluating the grades of journals in Science and Engineering in the RAE of UGC of Hong Kong. The detail steps are: (1) Analysis of the relationship between grades of journals and impact ratios
A frequency analysis method was used to Bnd the relationship between grades of journals and impact ratios in the Science and Engineering discipline. List of Journals in Science & Engineering of the year 1996 (the last time the evaluation was conducted) including the impact ratio and suggested grade was used for this purpose. The set of grades of the journals is G = (A; B; C; 0); and the impact ratio set was found to be S ⊂ [0; 6:4]. Because the sample size (285) may not be large enough, we considered the interval [x; x +0:2] as the analysis unit. The results of frequency distribution for every element of G are shown in Fig. 1. (The frequency of grade A on S.), in Fig. 2. (The frequency of grade B on S.), and in Fig. 3. (The frequency of grade C on S.).
D. Zhou et al. / Fuzzy Sets and Systems 131 (2002) 63 – 74
Fig. 4. The Percentages of grades A; B; C on S.
Fig. 5. Straight lines from Fig. 4.
Fig. 6. The relationship functions of grades A; B; C on S.
69
70
D. Zhou et al. / Fuzzy Sets and Systems 131 (2002) 63 – 74
Table 1 The judgment terms
Judgment term
4-tuple
Absolutely belongs to A a Somewhat less than A Less than A
1; 0; 0; 0 0:9; 0:1; 0; 0 0:7; 0:3; 0; 0
Much better than B Better than B Absolutely belongs to B a Somewhat less than B Less than B
0:5; 0:5; 0; 0 0:2; 0:8; 0; 0 0; 1; 0; 0 0; 0:9; 0:1; 0 0; 0:7; 0:3; 0
Much better than C Better than C Absolutely belongs to C a Somewhat less than C Less than C
0; 0:5; 0:5; 0 0; 0:2; 0:8; 0 0; 0; 1; 0 0; 0; 0:9; 0:1 0; 0; 0:7; 0:3
Much better than 0 Better than 0 Absolutely belongs to 0
0; 0; 0:5; 0:5 0; 0; 0:2; 0:8 0; 0; 0; 1
No information
(&; &; &; &)
Based on the distribution of the grades, the percentages of grades A; B; C are calculated as shown in Fig. 4. (The Percentages of grades A; B; C on S.) Fig. 4 reOects the relationship between impact ratios and grades A; B; and C. Using fuzzy set theory, we explain the relationship as the possibilities (likelihood) of journals with impact ratios belonging to the grade set. We approximated the curves in Fig. 4 by using straight lines to connect the main points (see Fig. 5) and received the membership functions representing grades A; B, C as shown in Fig. 6. The membership functions were computed as the following mathematical formulas: x ¡ 0:75; 0 A (x) = (x − 0:75)=0:75 0:75 6 x 6 1:5; 1 1:5 ¡ x: 0 x ¡ 0:1; (x − 0:1)=0:45 0:1 6 x 6 0:55; 0:55 ¡ x ¡ 0:75; B (x) = 1 (1:5 − x)=0:75 0:75 6 x 6 1:5; 0 1:5 ¡ x:
0 x ¡ 0:1; C (x) = (0:55 − x)=0:45 0:1 6 x 6 0:55; 0 0:55 ¡ x: 0 (x) =
1
x ¡ 0:1;
0
Otherwise:
(2) Experts’ subjective judgments The judgment terms are 4-tuples corresponding to the grade set G, which are shown in Table 1. The values on the right-hand side of the table were set by the authors. These values can be changed by the members of each panel. (3) Integration of the experts’ subjective options Suppose there are six experts participating in the journal evaluation, the objective and subjective data for journal Jk are as follows: Journal name: Jk ; Impact ratio: 1.258 Judgment of expert 1: Much better than B Judgment of expert 2: a Somewhat less than A Judgment of expert 3: Absolutely belongs to B Judgment of expert 4: Absolutely belongs to A Judgment of expert 5: No information Judgment of expert 6: 0:9; 0:3; 0; 0 (used numerical data) From the membership functions (Fig. 6 and its mathematical expression), we compute an impact ratio vector as V k = (0:6773; 0:3227; 0; 0). And by converting the judgment terms, using Table 1, we get the evaluation matrix of experts’ subjective judgments Ek .
Ek
0:5
0:9 0 = 1 & 0:9
0:5 0:1 1 0 & 0:3
0
0
0 0 0 : 0 0 & & 0
0
0
Now we add V k as the Brst row elements in E k , and get evaluation matrix E k .
D. Zhou et al. / Fuzzy Sets and Systems 131 (2002) 63 – 74
71
Fig. 7. The page of expert judgments on journals.
0:6773
0:5 0:9 k E = 0 1 & 0:9
0:3227 0:5 0:1 1 0 & 0:3
0
0
0 0 0 0 0 0 : 0 0 & & 0 0
(4) Assignment of weights of impact ratio and experts Assume the assigned weight for the impact ratio of the journal is 0.5. Thus, the weight of the experts subjective opinions is 1 − 0:5 = 0:5. According to the method of calculating the weights, we get W k = (0:5; 0:1; 0:1; 0:1; 0:1; 0; 0:1): (5) Aggregation of information of impact ratio and experts’ judgments
According to Eq. (2), we aggregate the impact ratio and the experts’ judgments: Y k = W k ◦ E k = (0:6687; 0:3513; 0; 0): (6) Determination of the grade of the journal From Y k , we can infer that the grade of the journal Jk is most likely to be A. As stated earlier, if the group does not accept this grade, they need to use some consensus-reaching methods to resolve the conOict. 4. Summary Establishing grades of journals is very important yet a diPcult task. Two distinct approaches are used in evaluating grades of journals: one is subjective approach, where journal grades are determined by experts’ subjective judgments; the other is objective approach which determines journal grades using objective information such as impact ratios. This paper
72
D. Zhou et al. / Fuzzy Sets and Systems 131 (2002) 63 – 74
Fig. 8. The page of determining grade of journals.
presents a fuzzy set approach which integrates the subjective and objective approaches. This synthetic approach comprises six steps, namely, (1) analyzing the relationship between past journal grades and the impact ratios; (2) deBning judgment terms for experts to elicit their subjective judgments; (3) assigning weights to impact ratio and experts; (4) integrating the experts’ subjective opinions; (5) aggregating the information of impact ratios and experts’ judgments to get fuzzy evaluation vector; (6) analyzing the fuzzy vector to decide the Bnal grades of journals. A decision support system has been developed to demonstrate the feasibility and eGectiveness of this application using a case study in Hong Kong and it is posted in the Internet (see appendix for details).
79600006), and Research Grant Council of Hong Kong and NSFC joint research scheme (Project No. 9050137). Two anonymous referees also gave useful comments and suggestions on the content and presentation of the paper. Appendix Web-based decision support system for the evaluation of journal grades
Acknowledgements
A group decision support system has been developed as GDSS Server on the Web. The underlying technology of the GDSS Server includes Microsoft NT Server, Microsoft SQL Server, and Microsoft Internet Information Server. The GDSS Server is built using Microsoft Active Server Page. The GDSS provides four main functions:
This research is partly supported by the Competitive Earmarked Research Grant (CERG) of Hong Kong (Project No. 9040375), National Natural Science Foundation of China (NSFC) (Project No.
(1) Journal information: This function is designed to display information on journals. (2) Judgment term information: This function is designed to display information on judgment terms.
D. Zhou et al. / Fuzzy Sets and Systems 131 (2002) 63 – 74
73
Fig. 9. The page of facilitator functions.
(3) Expert judgment: This function is designed for the experts to express their subjective judgments. An expert Brst selects a journal, then gives the judgment opinion on the journal (see Fig. 7 for an example). As an option, several experts can give their judgments simultaneously. (4) Facilitator functions: After experts provide their independent judgments on journals, a facilitator can use the aggregation function to integrate the impact ratio information and expert judgment information to compute the aggregated results. Fig. 8 shows the aggregation result of the example of the journal discussed in Section 4. The facilitator functions also include Journal, Judgment Term, Weight, User, etc. These functions are designed to manage journal information, judgment term information, weight information, and expert information, respectively (see Fig. 9).
References [1] V.A. Beattie, R.J. Ryan, The impact of non-serial publications on research in accounting and Bnance, ABACUS 27 (1) (1991) 32–50. [2] R. Biswas, An Application of fuzzy sets in students’ evaluation, Fuzzy Sets and Systems 74 (1995) 187–194. [3] C.H. Cheng, A. Kumar, J.G. Motwani, A. Reisman, M.S. Madam, A citation analysis of the technology innovation management journals, IEEE Trans. Eng. Management 46 (1) (1999) 4–13. [4] N.C. Dalkey, The Delphi Method: An Experimental Study of Group Opinion, The Rand Corporation, Santa Monica, 1969. [5] J. Diaz, R.T. Black, J. Rabianski, A note on the ranking of real estate research journals, Real Estate Econom. 24 (4) (1996) 551–563. [6] D. Dubois, H. Prade, Fuzzy Sets and Systems: Theory and Applications, Academic, New York, 1980. [7] M.M. Extejt, J.E. Smith, The behavioral sciences and management: an evaluation of relevant journals, J. Management 16 (3) (1990) 539–551.
74
D. Zhou et al. / Fuzzy Sets and Systems 131 (2002) 63 – 74
[8] M.L. Gillenson, J.D. Stutz, Academic issues in MIS: journals and books, MIS Quart. 15 (4) (1991) 447–452. [9] C.W. Holsapple, L.E. Johnson, H. Manakyan, J.T. Tanner, A citation analysis of business computing research journal, Inform. Management 25 (5) (1993) 231–244. [10] C.W. Holsapple, L. Johnson, H. Manakyan, J. Tanner, Business computer research journals: a normalized citation analysis, J. Management Inform. Systems 11 (1) (1994) 131– 140. [11] L.M. Jessup, J. Valacich, Group Support Systems: New Perspectives, Macmillan, New York, 1993.
[12] S.J. Liebowitz, J.P. Palmer, Assessing the relative impacts of economic journals, J. Econom. Literature 22 (1984) 77–88. [13] T.J. Ross, Fuzzy Logic With Engineering Application, McGraw-Hill, Singapore, 1997 pp. 315 –317. [14] K.A. Walstrom, B.C. Hardgrave, R.I. Wilson, Forums for management information systems scholars, Comm. ACM 38 (3) (1995) 93–102. [15] C.K. Wing, The ranking of construction management journals, Constr. Management Econom. 15 (1997) 387–398. [16] L.A. Zadeh, Fuzzy sets, Inform. Control 8 (1965) 338–353.
www.elsevier.com/locate/fss
A fuzzy set approach to the evaluation of journal grades Duanning Zhoua; b , Jian Mac; ∗ , Efraim Turbanc , Narasimha Bollojuc b College
a Computer Science Department, Zhongshan University, Guangzhou, China of Business and Public Administration, Eastern Washington University Spokane, WA 99202-1660, USA c Information Systems Department, City University of Hong Kong, Kowloon, Hong Kong
Abstract Many universities, research institutions and government agencies are continuously attempting to grade or rank journals for their academic value. Such grading is needed for personnel decisions and for funding and resource allocation purposes. The grading of journals is dependent both on objective information, such as the impact ratios of the journals, and on subjective information, such as experts’ judgments about the journals. Also, the available information for decision may be incomplete or imprecise due to the problem nature and complexity of the decision process. This paper proposes a fuzzy set approach that integrates objective and subjective information for evaluating grades of journals. It provides a comprehensive method for dealing with incomplete and imprecise information to support the whole evaluation process. A Web-based decision support c 2002 Elsevier Science B.V. All rights system that is based on the proposed fuzzy set approach has been implemented. reserved. Keywords: Group decision-making; Fuzzy preference; Evaluating academic journals; Subjective evaluation; Objective evaluation; Decision support system
1. Introduction Evaluating the quality of academic journals is required for several personnel decisions, such as recruiting, promotion, tenure and retention. Such evaluation is also done for merit increases and for determination of research funding. Many institutions use formal grading or point systems for such evaluations. In most cases the evaluation is done by a group (a committee or a panel, in which members are called experts) which complicates the evaluation process since a consensus is required. For years, researchers ∗ Corresponding author. Tel.: +852-278-8514; fax: +852-27844198. E-mail address: [email protected] (J. Ma).
in several disciplines attempted to Bnd the most appropriate formula for such evaluations. Research on this topic was conducted in: economics [12], accounting and Bnance [1], management [3,7], information systems [8,9,10,14], real estate [5] and construction management [15]. Unfortunately, there is no consensus on how best to conduct the journal evaluation. The methodologies for journal evaluation can be classiBed as either subjective or objective, depending on how the decision information is obtained and used. The subjective approach, also called the perception analysis approach, uses questionnaires to solicit subjective information from experts such as deans, department heads, renowned practitioners, and=or academic staG members. The collected information is compiled and a ranking of the journals is done based on the
c 2002 Elsevier Science B.V. All rights reserved. 0165-0114/02/$ - see front matter PII: S 0 1 6 5 - 0 1 1 4 ( 0 1 ) 0 0 2 5 5 - X
64
D. Zhou et al. / Fuzzy Sets and Systems 131 (2002) 63 – 74
collective respondents’ perceptions. DiGerent researchers used diGerent models for aggregating subjective opinions. The work of Gillenson and Stutz [8] is an example of ranking information systems journals based on respondents’ perceptions. The objective approach, also called citation analysis approach, determines the rankings of journals based on various forms of citation counts, by tabulating citations from a set of base journals, or a set of selected articles, over a certain time period. For example, Liebowitz and Palmer [12] used a citation approach to rank economics journals, and Holsapple et al. [10] employed a citation analysis methodology to rank business computing system journals. Citation information is publicly available in a form of “total cites”, “immediacy index”, “total articles”, cited half-life and “impact ratio (or factor)”. Information about these measures is available in sources such as the Journal Citation Reports, a CD-ROM database available in most libraries and updated annually. This paper uses the impact ratio which is based on “the average number of times articles published in a speciBc journal in the two previous years were cited in a particular year”. However, the proposed methodology will accept any other quantitative measures. Evaluations and rankings of journals determined by subjective approaches alone can be inOuenced by biases, or by lack of suPcient knowledge or experience. Objective approaches completely disregard subjective judgement of experts. It is obvious that neither the subjective, nor the objective approaches by themselves are best for the evaluation. Despite the considerable research conducted on evaluation, there are few researches on how to integrate the two approaches. Also, few researches can be found with the following two issues: (1) how to deal with incomplete subjective or objective information, and (2) how to transform the evaluations to a tangible journal grade. Due to incomplete and uncertain objective information (e.g., citation information may not be available for some journals), as well as lack of suPcient knowledge, experts may Bnd it diPcult to express their preference precisely. Fuzzy set theory [16] is applied here as a tool for solving the problem of imprecise subjective judgments and incomplete objective information. Fuzzy set methods for evaluation were attempted before. For example, Biswas [2] describes a fuzzy method to evaluate students’ answerscripts; Ross [13]
provides a fuzzy method for a generic multi-criteria evaluation. However, none of the previous methods can solve our research problems. This paper proposes a fuzzy set approach to deal with the above research issues. It analyzes the past grades assigned to journals in an attempt to identify the relationship between the historical assigned grades and the impact ratios of journals at the time the grades were assigned. Based on the discovered relationship, the model produces a suggested current grading by integrating experts’ opinions with the current impact ratio of the journals. ‘A’ computerized group decision support system based on the proposed evaluation method has been developed and it is available on the Web (http://144.214.54.91/zdn). The method is designed to Bt the process of funding research in Hong Kong. However, it can be easily modiBed to cover other situations. The Hong Kong process (see Section 3 for details), involves a combination of objective information which is based on impact ratios and on historical grades, and on subjective information solicited from experts, who are organized in disciplinary panels. It assigns grades of A; B; C or zero to each journal in the evaluation process. This paper is organized as follows. Section 2 describes the proposed fuzzy set approach. Section 3 presents an application, and a summary is given in Section 4. The appendix illustrates a Web-based group decision support system developed for the evaluation of grade of journals in Hong Kong.
2. The proposed fuzzy set approach 2.1. Overview The proposed evaluation procedure is composed of 6 steps. The Brst and second steps are conducted for all journals in one discipline. The last four are done for each journal independently. The steps are: Step 1: Analyze the relationship between past journal grades and past impact ratios. Certain relationships are evidenced between the two variables. Our model expresses these relationships quantitatively. Once established, this relationship can be used as an objective component for future evaluations.
D. Zhou et al. / Fuzzy Sets and Systems 131 (2002) 63 – 74
Step 2: DeBne qualitative guidelines for expressing subjective judgments. Experts often are unable to express their subjective judgments the way they really want to do it. Therefore, the expressions may be inaccurate. By providing optional qualitative judgment statements to the experts, they are able to express their views in a more practical way. Step 3: Assign weights that express the importance the panel wishes to place on the objective and subjective portions of the evaluation. Step 4: Integrate the subjective information of all panel members and the objective information to create a fuzzy evaluation matrix. Step 5: Add the weights and aggregate the fuzzy evaluation matrix. Step 6: The aggregated results can be used as a reference point for determining the journal’s grade by the evaluators. 2.2. General notations Let G = {g1 ; g2 ; : : : ; gd } be a set of grades of journals (without loss of generality, we assume g1 is the best grade and gd the worst grade). Let P = {P1 ; P2 ; : : : ; Pt } be the set of experts and J = {J1 ; J2 ; : : : ; Jn } be the set of journals. The grades g1 ; g2 ; : : : ; gd are fuzzy sets deBned on journal set J with membership values between 0 and 1, and they can be represented as J1 J2 Jn gi = ; ;:::; a1i a2i ani (aji ∈ [0; 1]; 1 6 i 6 d and j = 1; 2; : : : ; n); where aij is the membership degree that represents the objective and subjective ranking of journal j in the fuzzy set gi . For example, A, given below, speciBes the membership values of journals J1 ; J2 ; : : : ; Jn as 0:8; 0:5; : : : ; 0:1; respectively, for grade A. J1 J2 Jn A= : ; ;:::; 0:8 0:5 0:1 2.3. Step 1: analysis of the relationship between grades and impact ratios of journals The relationships between impact ratios and grades of journals can be expressed by using the concepts
65
of membership degree and membership function in fuzzy set theory. Assume that there exist membership functions g1 ; g2 ; : : : ; gd which reOect the relationship between impact ratios and grades of journals. If we denote the impact ratio of journal Jl as Jlx (16l6n); then we can determine fuzzy set g1 ; g2 ; : : : ; gd on the consideration of impact ratios of journals, which have the following format: J2 Jn J1 gi = ; ;:::; gi (J1x ) gi (J2x ) gi (Jnx ) (i = 1; : : : ; d and gi (J1x ); : : : ; gi (Jnx ) ∈ [0; 1]): Membership functions g1 ; g2 ; : : : ; gd can be obtained by analyzing historical data at one point of time (e.g., the latest data) of impact ratios and grades of journals. The membership functions can be computed, for example, by counting how many journals received certain grade in certain intervals of the impact ratio (frequency distribution). Section 3 illustrates how such membership functions are computed using historical data of impact ratios and associated journal grades. With membership functions g1 ; g2 ; : : : ; gd ; from the point view of journal Jk ; vector (g1 (Jkx ); g2 (Jkx ); : : : ; gd (Jkx )) expresses the grade of journal Jk . This means that journal Jk belongs to grades g1 ; g2 ; : : : ; gd withmembershipdegreesg1 (Jkx ); g2 (Jkx ); : : : ; gd (Jkx ), respectively. In other words, the vector (g1 (Jkx ); g2 (Jkx ); : : : ; gd (Jkx )) represents fuzzy set “grade of journal Jk ” deBned on the grade set g1 ; g2 ; : : : ; gd . In general, with membership functions, we can deBne a fuzzy mapping as follows: f : x → (g1 (x); g2 (x); : : : ; gd (x)); where x belongs to the impact ratio set. With the membership functions g1 ; g2 ; : : : ; gd , the fuzzy mapping f gives a journal (e.g., Jk ) with an impact ratio, x; the degree to which the speciBc journal belongs to the grades of journals g1 ; g2 ; : : : ; gd , respectively. We can explain that the degrees form a fuzzy set “grade of journal Jk ” deBned on the grades set g1 ; g2 ; : : : ; gd . For example, if the impact ratio of a journal is known to be 0.9, a designation of g1 (0:9) = 0:7; g2 (0:9) = 0:5; : : : ; gd (0:9) = 0:2; means the journal belongs to the set of grades g1 ; g2 ; : : : ; gd with 0:7; 0:5; : : : ; 0:2 degrees, respectively.
66
D. Zhou et al. / Fuzzy Sets and Systems 131 (2002) 63 – 74
For each journal Jk ∈ J (16k6n); the degrees to which the journal belongs to diGerent grades g1 ; g2 ; : : : ; gd form a vector, denoted as impact ratio vector V k : V k = (g1 (Jkx ); g2 (Jkx ); : : : ; gd (Jkx )); k = 1; : : : ; n:
(1)
The vector V k reOects the objective information of journal Jk ; i.e. impact ratio of journal Jk . Next we show how the subjective information of journals is solicited and how it is combined with the objective information of vector V k . 2.4. Step 2: expression of the experts’ subjective judgments individually Assume that a group of experts, P; (P = {P1 ; P2 ; : : : ; Pt }); participates in a panel. Usually, panel members express their opinions, Brst, independent of each other. Then, they try to reach a consensus. As mentioned above, the grades g1 ; g2 ; : : : ; gd are fuzzy sets deBned on journal set, J; where membership is between 0 and 1, where J1 J2 Jn gi = ; ;:::; a1i a2i ani (aji ∈ [0; 1]; 1 6 i 6 d and j = 1; 2; : : : ; n): Such information can be expressed in the following matrix, experts need to assign values column-wise: Grades J1
g1
g2
:::
gd
a11
a12
:::
a1d
a22
:::
an2
:::
.. .
a21 :::
Jn
an1
Journals J2
a2d and
However, in practice, it may be unpractical or not easy for experts to specify their subjective judgments using fuzzy sets g1 ; g2 ; : : : ; gd and associated degrees of memberships for a set of journals. For this reason, we expect speciBcation row-wise (instead column-wise)
from experts for each journal, which is much easier for experts to estimate journals by assigning degrees on grades. Furthermore, since each expert need to Bll in information for several, sometimes dozens of journals, we provide them with an option of expressing their estimates in judgment terms. A judgment term is an ordered d-tuple a1 ; a2 ; : : : ; ad ; where ai ∈ [0; 1]; 16i6d. The values of the judgment term, i.e., d-tuple, of a journal represent the degrees that a speciBc journal belongs to the grades g1 ; g2 ; : : : ; gd ; respectively. For example, let G = {A; B; C; D}; then a statement: “The grade of the journal is a somewhat less than grade A” can be expressed as 0:9; 0:3; 0; 0 and “The grade of the journal is better than grade B” can be expressed as 0:2; 0:8; 0:1; 0 Although we provide a set of predeBned judgment terms, experts can alternatively specify the d-tuple by directly inserting numerical values instead of using the predeBned qualitative terms. 2.5. Step 3: assignment of weights The experts themselves or the panel leader can assign weights to impact ratios of diGerent journals. In addition, the panel leader can assign a weight to the opinion of the group or to the experts (either as a group, or for each expert), for each journal. In this subsection, we consider how the weight assignments are used to calculate a “weight vector” which is used to evaluate the grades of journals. Let the weight of impact ratio for journal Jk be w0k ; and the weight of the panel of experts for that Jk journal, be wek ; where w0k + wek = 1; w0k ¿0; wek ¿0. The reason that the sum of the weights is one, is that we allow this way a trade oG between the subjective and objective assessments. Suppose we assign weights wek1 ; wek2 ; : : : ; wekt one each to experts p1 ; p2 ; : : : ; pt ; re t spectively, where j=1 wekj = 1; wekj ¿0; j = 1; : : : ; t. The assignment of weights to impact ratios and to experts is presented below. Each expert t pj is assigned a weight of wjk = wek wekj ; where j=1 wjk = wek . The weight vector corresponding to journal Jk is W k = (w0k ; w1k ; : : : ; wtk ). If any expert does not give judgment information of a journal, then the weight corresponding to that expert is redistributed among
D. Zhou et al. / Fuzzy Sets and Systems 131 (2002) 63 – 74
other weights. This normalization process satisBes t the condition j=1 wekj = 1. 2.6. Step 4: integration of the experts’ opinions and objective information For expert Pi , the subjective judgment (i.e. judgment term) on the journal Jk forms a vector, denoted as Vi k = (vi1k ; vi2k ; : : : ; vidk ), where each vijk ∈ [0; 1]; (16i6t; j = 1; : : : ; d). For a panel of t experts involved in evaluating the journal Jk ; a subjective expert evaluation relation ma trix, denoted as E k = [vijk ; i = 1; : : : ; t; j = 1; : : : ; d; is formed by integrating the individual experts’ judg ment vectors Vi k ; i = 1; : : : ; t. The relation E k consist of the experts’ subjective judgments information on journal Jk . To evaluate the journal Jk both subjectively and objectively, we add the impact ratio vector V k in (1) as the Brst row elements of the matrix E k , then get the integrated evaluation relation matrix denoted as E k = [vijk ]; i = 0; 1; : : : ; t; j = 1; : : : ; d.
67
k k k ) ⊕ c(w1k • e1j ) ⊕ · · · ⊕ (wnk • enj ) where yjk = (w0k • e0j and ‘•’, ‘⊕’ are deBned as
(a) algebraic product, a • b: c = ab (b) bounded sum, a ⊕ b: c = a ⊕ b = min{1; a + b}. 2.8. Step 6: determination of the grades of the journals According to the principles of fuzzy classiBcation [6], we have yik = Max(y1k ; y2k ; : : : ; ydk ). Thus the corresponding grade gi is the Bnal grade of the journal Jk . In practical evaluation process, we set a paramek ter " to check if |yik − yi+1 |¡"; i¡d; if it happens, then we can say this journal is between grades gi and gi+1 . If |yik − yjk |¡"; and i¿j; j = i + 1; i¡d − 1; that means the experts have strong contradicting opinions on the grade of the journal. Delphi method [4] or GDSS [11] may be used to reach consensus. Alternatively, other consensus reaching methods can be applied. 3. Application in Hong Kong
2.7. Step 5: aggregation of the impact ratio and experts’ judgments For a journal Jk ; we derived the fuzzy weight W k and the fuzzy evaluation relation E k . Then, the process of determining the Bnal grade of the journal Jk is equivalent to the process of determining a membership value for the journal Jk in each of the evaluation grades g1 ; g2 ; : : : ; gd . This process can be implemented through the composition operation W k ◦ E k . The result is a fuzzy vector (evaluation vector), denoted as Y k ; containing the membership values for the journal Jk in each of evaluation grades g1 ; g2 ; : : : ; gd : Y k = W k ◦ Ek
= (w0k ; w1k ; : : : ; wtk )
k e01
k e 11 ◦ ::: k et1
= (y1k ; y2k ; : : : ; ydk );
k e02
:::
k e12
:::
::: k et2
:::
k e0d
k e1d ::: k etd (2)
Research is always one of the most important functions of a university. To improve the research performance, the University Grant Committee (UGC) in Hong Kong conducts research assessment exercise (RAE) once in every 3 years. For the purpose of this exercise, local universities form cost centers according to disciplines suggested by the UGC. Academic staG members are assigned to cost centers according to their research disciplines. A staG member submits up to 5 recently (3 years) published or accepted journal articles for RAE assessment. UGC forms disciplinary panels to review the journals and assign a grade: A: top-tier, B: mid-tier, C: lower-tier, to each journal. When an academic staG member is assessed to have 3 “B” grade or better articles published during the assessment period, he/she is most likely to become an active researcher (AR). In the previous two RAE exercises, an AR meant an allocation of research funds to the cost center equivalent to at least the researcher’s salary for the forthcoming 3 years. Thus universities in Hong Kong pay a lot of attention to the RAE exercises. From the perspective of the universities, it is very important to estimate well in advance the likely
68
D. Zhou et al. / Fuzzy Sets and Systems 131 (2002) 63 – 74
Fig. 1. The frequency of grade A on S.
Fig. 2. The frequency of grade B on S.
Fig. 3. The frequency of grade C on S.
grades of diGerent journals. These estimations can be communicated to the academic staG members so that they can identify a set of journals to publish in, as well as to decide which journals to include as the Bve “best” (in case they have more than Bve). Our methodology can be used by the universities to optimize the above decisions. It can also be used by diGerent panels to arrive at their decisions. The proposed fuzzy evaluation method was tested in evaluating the grades of journals in Science and Engineering in the RAE of UGC of Hong Kong. The detail steps are: (1) Analysis of the relationship between grades of journals and impact ratios
A frequency analysis method was used to Bnd the relationship between grades of journals and impact ratios in the Science and Engineering discipline. List of Journals in Science & Engineering of the year 1996 (the last time the evaluation was conducted) including the impact ratio and suggested grade was used for this purpose. The set of grades of the journals is G = (A; B; C; 0); and the impact ratio set was found to be S ⊂ [0; 6:4]. Because the sample size (285) may not be large enough, we considered the interval [x; x +0:2] as the analysis unit. The results of frequency distribution for every element of G are shown in Fig. 1. (The frequency of grade A on S.), in Fig. 2. (The frequency of grade B on S.), and in Fig. 3. (The frequency of grade C on S.).
D. Zhou et al. / Fuzzy Sets and Systems 131 (2002) 63 – 74
Fig. 4. The Percentages of grades A; B; C on S.
Fig. 5. Straight lines from Fig. 4.
Fig. 6. The relationship functions of grades A; B; C on S.
69
70
D. Zhou et al. / Fuzzy Sets and Systems 131 (2002) 63 – 74
Table 1 The judgment terms
Judgment term
4-tuple
Absolutely belongs to A a Somewhat less than A Less than A
1; 0; 0; 0 0:9; 0:1; 0; 0 0:7; 0:3; 0; 0
Much better than B Better than B Absolutely belongs to B a Somewhat less than B Less than B
0:5; 0:5; 0; 0 0:2; 0:8; 0; 0 0; 1; 0; 0 0; 0:9; 0:1; 0 0; 0:7; 0:3; 0
Much better than C Better than C Absolutely belongs to C a Somewhat less than C Less than C
0; 0:5; 0:5; 0 0; 0:2; 0:8; 0 0; 0; 1; 0 0; 0; 0:9; 0:1 0; 0; 0:7; 0:3
Much better than 0 Better than 0 Absolutely belongs to 0
0; 0; 0:5; 0:5 0; 0; 0:2; 0:8 0; 0; 0; 1
No information
(&; &; &; &)
Based on the distribution of the grades, the percentages of grades A; B; C are calculated as shown in Fig. 4. (The Percentages of grades A; B; C on S.) Fig. 4 reOects the relationship between impact ratios and grades A; B; and C. Using fuzzy set theory, we explain the relationship as the possibilities (likelihood) of journals with impact ratios belonging to the grade set. We approximated the curves in Fig. 4 by using straight lines to connect the main points (see Fig. 5) and received the membership functions representing grades A; B, C as shown in Fig. 6. The membership functions were computed as the following mathematical formulas: x ¡ 0:75; 0 A (x) = (x − 0:75)=0:75 0:75 6 x 6 1:5; 1 1:5 ¡ x: 0 x ¡ 0:1; (x − 0:1)=0:45 0:1 6 x 6 0:55; 0:55 ¡ x ¡ 0:75; B (x) = 1 (1:5 − x)=0:75 0:75 6 x 6 1:5; 0 1:5 ¡ x:
0 x ¡ 0:1; C (x) = (0:55 − x)=0:45 0:1 6 x 6 0:55; 0 0:55 ¡ x: 0 (x) =
1
x ¡ 0:1;
0
Otherwise:
(2) Experts’ subjective judgments The judgment terms are 4-tuples corresponding to the grade set G, which are shown in Table 1. The values on the right-hand side of the table were set by the authors. These values can be changed by the members of each panel. (3) Integration of the experts’ subjective options Suppose there are six experts participating in the journal evaluation, the objective and subjective data for journal Jk are as follows: Journal name: Jk ; Impact ratio: 1.258 Judgment of expert 1: Much better than B Judgment of expert 2: a Somewhat less than A Judgment of expert 3: Absolutely belongs to B Judgment of expert 4: Absolutely belongs to A Judgment of expert 5: No information Judgment of expert 6: 0:9; 0:3; 0; 0 (used numerical data) From the membership functions (Fig. 6 and its mathematical expression), we compute an impact ratio vector as V k = (0:6773; 0:3227; 0; 0). And by converting the judgment terms, using Table 1, we get the evaluation matrix of experts’ subjective judgments Ek .
Ek
0:5
0:9 0 = 1 & 0:9
0:5 0:1 1 0 & 0:3
0
0
0 0 0 : 0 0 & & 0
0
0
Now we add V k as the Brst row elements in E k , and get evaluation matrix E k .
D. Zhou et al. / Fuzzy Sets and Systems 131 (2002) 63 – 74
71
Fig. 7. The page of expert judgments on journals.
0:6773
0:5 0:9 k E = 0 1 & 0:9
0:3227 0:5 0:1 1 0 & 0:3
0
0
0 0 0 0 0 0 : 0 0 & & 0 0
(4) Assignment of weights of impact ratio and experts Assume the assigned weight for the impact ratio of the journal is 0.5. Thus, the weight of the experts subjective opinions is 1 − 0:5 = 0:5. According to the method of calculating the weights, we get W k = (0:5; 0:1; 0:1; 0:1; 0:1; 0; 0:1): (5) Aggregation of information of impact ratio and experts’ judgments
According to Eq. (2), we aggregate the impact ratio and the experts’ judgments: Y k = W k ◦ E k = (0:6687; 0:3513; 0; 0): (6) Determination of the grade of the journal From Y k , we can infer that the grade of the journal Jk is most likely to be A. As stated earlier, if the group does not accept this grade, they need to use some consensus-reaching methods to resolve the conOict. 4. Summary Establishing grades of journals is very important yet a diPcult task. Two distinct approaches are used in evaluating grades of journals: one is subjective approach, where journal grades are determined by experts’ subjective judgments; the other is objective approach which determines journal grades using objective information such as impact ratios. This paper
72
D. Zhou et al. / Fuzzy Sets and Systems 131 (2002) 63 – 74
Fig. 8. The page of determining grade of journals.
presents a fuzzy set approach which integrates the subjective and objective approaches. This synthetic approach comprises six steps, namely, (1) analyzing the relationship between past journal grades and the impact ratios; (2) deBning judgment terms for experts to elicit their subjective judgments; (3) assigning weights to impact ratio and experts; (4) integrating the experts’ subjective opinions; (5) aggregating the information of impact ratios and experts’ judgments to get fuzzy evaluation vector; (6) analyzing the fuzzy vector to decide the Bnal grades of journals. A decision support system has been developed to demonstrate the feasibility and eGectiveness of this application using a case study in Hong Kong and it is posted in the Internet (see appendix for details).
79600006), and Research Grant Council of Hong Kong and NSFC joint research scheme (Project No. 9050137). Two anonymous referees also gave useful comments and suggestions on the content and presentation of the paper. Appendix Web-based decision support system for the evaluation of journal grades
Acknowledgements
A group decision support system has been developed as GDSS Server on the Web. The underlying technology of the GDSS Server includes Microsoft NT Server, Microsoft SQL Server, and Microsoft Internet Information Server. The GDSS Server is built using Microsoft Active Server Page. The GDSS provides four main functions:
This research is partly supported by the Competitive Earmarked Research Grant (CERG) of Hong Kong (Project No. 9040375), National Natural Science Foundation of China (NSFC) (Project No.
(1) Journal information: This function is designed to display information on journals. (2) Judgment term information: This function is designed to display information on judgment terms.
D. Zhou et al. / Fuzzy Sets and Systems 131 (2002) 63 – 74
73
Fig. 9. The page of facilitator functions.
(3) Expert judgment: This function is designed for the experts to express their subjective judgments. An expert Brst selects a journal, then gives the judgment opinion on the journal (see Fig. 7 for an example). As an option, several experts can give their judgments simultaneously. (4) Facilitator functions: After experts provide their independent judgments on journals, a facilitator can use the aggregation function to integrate the impact ratio information and expert judgment information to compute the aggregated results. Fig. 8 shows the aggregation result of the example of the journal discussed in Section 4. The facilitator functions also include Journal, Judgment Term, Weight, User, etc. These functions are designed to manage journal information, judgment term information, weight information, and expert information, respectively (see Fig. 9).
References [1] V.A. Beattie, R.J. Ryan, The impact of non-serial publications on research in accounting and Bnance, ABACUS 27 (1) (1991) 32–50. [2] R. Biswas, An Application of fuzzy sets in students’ evaluation, Fuzzy Sets and Systems 74 (1995) 187–194. [3] C.H. Cheng, A. Kumar, J.G. Motwani, A. Reisman, M.S. Madam, A citation analysis of the technology innovation management journals, IEEE Trans. Eng. Management 46 (1) (1999) 4–13. [4] N.C. Dalkey, The Delphi Method: An Experimental Study of Group Opinion, The Rand Corporation, Santa Monica, 1969. [5] J. Diaz, R.T. Black, J. Rabianski, A note on the ranking of real estate research journals, Real Estate Econom. 24 (4) (1996) 551–563. [6] D. Dubois, H. Prade, Fuzzy Sets and Systems: Theory and Applications, Academic, New York, 1980. [7] M.M. Extejt, J.E. Smith, The behavioral sciences and management: an evaluation of relevant journals, J. Management 16 (3) (1990) 539–551.
74
D. Zhou et al. / Fuzzy Sets and Systems 131 (2002) 63 – 74
[8] M.L. Gillenson, J.D. Stutz, Academic issues in MIS: journals and books, MIS Quart. 15 (4) (1991) 447–452. [9] C.W. Holsapple, L.E. Johnson, H. Manakyan, J.T. Tanner, A citation analysis of business computing research journal, Inform. Management 25 (5) (1993) 231–244. [10] C.W. Holsapple, L. Johnson, H. Manakyan, J. Tanner, Business computer research journals: a normalized citation analysis, J. Management Inform. Systems 11 (1) (1994) 131– 140. [11] L.M. Jessup, J. Valacich, Group Support Systems: New Perspectives, Macmillan, New York, 1993.
[12] S.J. Liebowitz, J.P. Palmer, Assessing the relative impacts of economic journals, J. Econom. Literature 22 (1984) 77–88. [13] T.J. Ross, Fuzzy Logic With Engineering Application, McGraw-Hill, Singapore, 1997 pp. 315 –317. [14] K.A. Walstrom, B.C. Hardgrave, R.I. Wilson, Forums for management information systems scholars, Comm. ACM 38 (3) (1995) 93–102. [15] C.K. Wing, The ranking of construction management journals, Constr. Management Econom. 15 (1997) 387–398. [16] L.A. Zadeh, Fuzzy sets, Inform. Control 8 (1965) 338–353.