Understanding relevance judgment in the view of perceived value

Library and Information Science Research 41 (2019) 100982

Contents lists available at ScienceDirect

Library and Information Science Research journal homepage: www.elsevier.com/locate/lisres

Understanding relevance judgment in the view of perceived value a,b

Jianping Liu a b

, Jian Wang

a,b,⁎

a,b

, Guomin Zhou

T

Agricultural Information Institute, Chinese Academy of Agricultural Sciences, 12 Zhongguancun South Street, Beijing 100081, China Key Laboratory of Agricultural Big Data, Ministry of Agriculture and Rural Affairs, 12 Zhongguancun South Street, Beijng 100081, China

ABSTRACT

While researchers have explored and proposed dozens of user relevance judgment criteria (RJC) in various situations, there is a lack of empirical research on the effects of RJC on user relevance judgment and how. The study sought to develop relations between RJC and situational relevance (SR) via four perceived values, epistemic value, functional value, conditional value, and social value, by introducing multi-factors perceived value (PV) theory and structural equation modeling (SEM). The study developed a RJC model basing on multi-factors PV and derived seven hypotheses to be verified. The data of the use of RJC and self-estimation of PVs from 453 people who were all participants of a national data sharing competition were collected by questionnaire, and then were verified and analyzed by SEM to test the hypotheses. The results verify the effectiveness of four PVs on SR with different levels. Meanwhile, it also suggests that RJC can function as the measurements of PVs and reduce RJC as well. Based on the above process, the study puts forward a new definition of SR based on four information values and verifies the SR judgment model—information value-utility model. The research provides theoretical basis and measurement dimensions for understanding and measuring SR. Finally, the theoretical and practical implications of this study are discussed.

1. Introduction From the very beginning of Vannevar Bush's 1945 article “As We May Think,” the domain of information science has accumulated dozens of research studies and has made progress in relevance connotation, manifestation, relevance judgment criteria (RJC), relevance judgment behavior, etc. (Barry & Schamber, 1998; Mizzaro, 1997; Saracevic, 2007, 2016; Schamber, 1994; Taylor, 2013; Wang & Soergel, 1998). However, researchers till now cannot give a clear explanation of how people reach their relevance judgment, mainly how they use various RJC to draw a final decision of relevance. There are at least two main reasons for this. The first relates to the diversity and complexity of manifestations of user relevance. Saracevic (1996) summarized four types of user relevance according to existing research: topical, cognitive or pertinence, situational, and affective or motivational relevance. Social-cognitive relevance and other perspectives also emerged (Cosijn & Ingwersen, 2000; Hjørland, 2002; Ingwersen & Järvelin, 2011). However, from the perspective of measurability, only situational relevance (SR) takes a pragmatic and measurable perspective and is operationalized as the utility of the information objects to the user's situational task at hand (Borlund, 2003; Cosijn & Ingwersen, 2000; Xu & Chen, 2006); the second relates to the limitations of measurement dimensions of user relevance. Researchers identified large numbers of RJC in different situations (Barry, 1994; Barry & Schamber, 1998; Maglaughlin & Sonnenwald, 2002). However, RJC as measurement dimensions of SR still faces many limitations (Xu & Chen, 2006).

⁎

Are there higher constructions than RJC that can function as measurement dimensions of SR, or are there constructions that connect RJC on one side and SR on the other side? This study aims at such constructs by introducing multi-factors perceived value (PV) theory and developing relations between SR and RJC via differential PVs. For this purpose, multi-factors PV was explored. 1.1. Multi-factors perceived value theory PV theory is the main theory in the field of customer consumption. According to this theory, the decision-making behavior of users in purchasing commodities is the result of users' perception of commodity value. Zeithaml (1988) put forward a definition that “perceived value is the consumer's overall assessment of the utility of a product based on perceptions of what is received and what is given” (p. 14). Based on Zeithaml (1988) cost-benefit perspective, Sheth, Newman, and Gross (1991) expanded the dimensions that affect users' perception of commodity value to five independent dimensions that comprehensively cover the factors that affect users' perception of commodity value (see 3.2 for detailed discussion of the connotation of the five PVs), and three fundamental propositions as the axiomatic of the theory as follows and verified by empirical methods.

• “Consumer choice is a function of multiple consumption values. • The consumption values make differential contributions in any given choice situation.

Corresponding author at: Agricultural Information Institute, Chinese Academy of Agricultural Sciences, 12 Zhongguancun South Street, Beijing 100081, China. E-mail address: [email protected] (J. Wang).

https://doi.org/10.1016/j.lisr.2019.100982 Received 27 March 2019; Received in revised form 19 August 2019; Accepted 6 September 2019 Available online 19 October 2019 0740-8188/ © 2019 Elsevier Inc. All rights reserved.

Library and Information Science Research 41 (2019) 100982

J. Liu, et al.

• The consumption values are independent” (Sheth et al., 1991, p.

important basis for taking the PVs as the measurement dimension of SR. SR associates information objects with tasks and problems in the situation perceived by users. Utility is used as the operational definition of SR, which makes it more possible to measure user relevance from the pragmatic perspective. However, there is still a lack of empirical research on the effective measurement dimensions of SR (utility).

160)

These three propositions define the premise for the theory to become a mature theory from the aspects of the function, influence, and interrelation of the five PVs. The theory of multi-factors PV has been widely applied in the fields of marketing, consumption, and information sciences (Ladhari & Morales, 2008; Misilei & Liew, 2018; Pihlström & Brush, 2010; Wang & Soergel, 1998; Wang & Wang, 2010). Specific to relevance study, it is worth noting that there are many similarities between SR judgment and commodity value perception regarding aspects of scenario, decision processes and connotations. For example, researchers all use “utility” as the operational concept in both concepts (Borlund, 2003; Sheth et al., 1991; Zeithaml, 1988). The PV theory shows similarities to SR judgment and commodity value perception which makes it ideal for use in analysis.

2.2. User relevance judgment criteria 2.2.1. Identification of relevance judgment criteria The original intention of RJC research is to find the measurement dimensions of user relevance judgment. Schamber, Eisenberg, and Nilan (1990) stated that “an understanding of relevance criteria, or the reasons underlying relevance judgment, as observed from the user's perspective, may contribute to a more complete and useful understanding of the dimensions of relevance” (p. 771). RJC identification is the premise and foundation of user relevance judgment research. Taking Schamber (1991) and Barry (1994) as representative studies, researchers identified dozens of RJC including topicality, quality, authority, availability, novelty, etc. (Cool, Belkin, & Kantor, 1993; Hirsh, 1999; Maglaughlin & Sonnenwald, 2002; Wang & Soergel, 1998).

1.2. Problem statement Although there are many similarities between SR judgment and commodity value perception, whether PVs can be used as effective measurement dimensions of SR and establish a relationship with RJC, the core variable in SR, needs verification. The research addressed the following questions:

2.2.2. Summary of core cross-situational relevance judgment criteria Therefore, based on seven classic independent studies, 11 core RJC are the basis for exploring effective user relevance measurement dimensions in this study. The summary principles of the core RJC set are as follows: First, the RJC are all from situational experiment originally, in which RJC was summarized by coding the interview content. Second, in this study, RJC is defined as the concept of “cognitive tool” on which users rely for relevance judgment, and it is also a certain level of judgment made by users (e.g. in order to judge the authority of current information object, users usually take the producers and affiliation of the authors as the evidences or clues, not RJC itself). Third, it summarizes the RJC from research focusing on different information carriers. Among them, as shown in Table 1, Cool et al. (1993), Barry (1994), Wang and Soergel (1998), and Maglaughlin and Sonnenwald (2002) took the texts and documents, Choi and Rasmussen (2002) took images and Hirsh (1999) took both texts and images as information carriers.

1. Based on the similarity of both definitions and decision processes, can PVs be empirically verified as measurement dimensions of SR judgment? If so, 2. How do differential PVs affect SR judgment? Based on this, can a more operational definition be given for SR? 3. Based on the similarity of both connotations, can RJC be empirically verified as a measurement index of PVs? 2. Literature review 2.1. Situational relevance 2.1.1. The importance of situation Situation has been a key factor in relevance, both for user-oriented and system-oriented, study for a long time. Saracevic (1996) pointed out that “relevance cannot be considered without a situation” (p. 206). Relevance occurs in the real situation in which users pursue, evaluate and use information (Anderson, 2005). The term situation expresses two characteristics of user relevance judgment: “interactivity” and “situational dependency”. Interaction is a necessary process for user relevance judgment, and situation is the product of interaction. “Situational dependency” refers to the impact of situation on users. User relevance judgment always depends on specific tasks or problems perceived from a situation.

2.2.3. Limitations of relevance judgment criteria as dimensions of user relevance RJC identification research provides many possible dimensions for measuring user relevance. However, RJC has limitations when functioning as measurement dimensions of user relevance judgment: (1) RJC has a big set, ranging from a dozen to dozens (Barry, 1994; Barry & Schamber, 1998; Maglaughlin & Sonnenwald, 2002). If every RJC is considered, there is no operational significance in the practice of information retrieval. (2) Consensus has been reached that topicality has a primary and basic role to user relevance judgment (Barry, 1994; Schamber, 1991; Wang & Soergel, 1998). However, other RJC have the same level of influence on SR judgment. (3) The lack of functional definition of RJC leads to different levels of concepts in RJC sets, such as information attributes (author, origin) and demographic characteristics (age) as RJC (Fitzgerald & Galloway, 2001; Sedghi, Sanderson, & Clough, 2013). (4) RJC terms overlap severely, the same meaning is expressed in different terms (e.g., recency, currency, temporal issues).

2.1.2. Utility as an operational concept of situational relevance Utility, as an operational concept of SR, is mainly embodied in the classic definitions. SR was first defined by Wilson (1973) as “the relation between an information object and the information recipient's individual and personal view of the world and his or her situation in it” (p. 548), which emphasized the importance of situation to the user's perceived relevance. In the definition by Borlund (2003), “situational relevance is understanding as the utility or usefulness of the viewed and assessed information object(s) by pointing to the relationship between such retrieved object(s) and the work task at hand underlying the information need as perceived by the user” (p. 915). Other studies also explicitly used utility (or usefulness alternatively) in the definition of SR (Borlund & Ingwersen, 1997; Cosijn & Ingwersen, 2000; Saracevic, 1975, 1996). Meanwhile, “utility” also functions as the operational concept in the classic definitions of PV in the context of commodity purchasing (Sheth et al., 1991; Zeithaml, 1988), which provides an

2.2.4. Exploration of effective measurement dimensions of user relevance Barry and Schamber (1998) summarized 10 core RJC in different situations through comparative study. Wang and Soergel (1998, p. 130, Table 8) compared the 11 RJC obtained from research results with Barry (1993, 1994), Cool et al. (1993), and Schamber (1991). The results showed that only 4 RJC (topicality, recency, quality, and authority) appeared in all four studies. Based on the information seeking process (ISP) model proposed by Kuhlthau (1991) and taking the task 2

Library and Information Science Research 41 (2019) 100982

J. Liu, et al.

Table 1 Core RJC of Cross-situation. RJC

Wang and White Schamber (1999) (1991)

Cool et al. (1993)

Barry (1994)

Hirsh (1999)

Maglaughlin and Sonnenwald (2002)

Choi and Rasmussen (2002)

Topicality Intelligibility/Ability to understand Novelty Quality Recency/Currency Special request Completeness Convenience Accessibility/Availability/Obtainability Authority Normative/Format

✓

✓ ✓

✓

✓

✓

✓

✓ ✓

✓ ✓

✓ ✓ ✓ ✓ ✓ ✓

✓ ✓ ✓

✓ ✓ ✓

✓ ✓ ✓

Criteria

mapping

Values

✓ ✓ ✓ ✓

✓ ✓

✓ ✓

✓

decomposing

✓ ✓

✓ ✓

Situational relevance (Utility)

✓ ✓ ✓ ✓

✓ ✓ ✓

✓

Fig. 1. Conceptual model. affecting

phase of user information query as the control variable, the researchers studied the dynamic characteristics of RJC (Levene, Bar-Ilan, & Zhitomirsky-Geffet, 2018; Taylor, 2009, 2013; Taylor, Cool, Belkin, & Amadio, 2007; Vakkari, 2000; Wang & White, 1999; ZhitomirskyGeffet, Bar-Ilan, & Levene, 2017). The research results validate Saracevic's (2016) conclusion that “user's relevance judgment criteria for inferences are fairly stable” (p. 59). The existence of core RJC and the stability of RJC using both reveal the existence of core factors that may be the measurement dimension of user relevance but lacks further verification. Greisdorf (2000, 2003) discussed RJC as “lower level of relevance” (e.g. current, authority, etc.) form “higher level of relevance” (systematic, topicality, pertinence, utility, and motivation) in the way of aggregation (e.g. based on conjunction and disjunction rules). However, these “higher level of relevance” actually correspond to the different types of relevance summarized by Saracevic (1996) and are not suitable as aggregation dimensions of RJC. Each of the five types of relevance is an umbrella concept with rich connotation, and there is a high degree of overlap among them. Xu and Chen (2006) verified five RJC (topicality, reliability, scope, understandability, and novelty) as measurement dimensions of SR by comparing RJC with five principles of communication theory (Grice, 1991). However, other RJC that are as important as the five RJC are not discussed in their study. As shown in Table 1, RJC such as quality and recency are not considered in Xu and Chen's model. Wang and Soergel (1998) introduced PV theory and explored the cooccurrence of PVs and RJC by coding the interview content which concluded that RJC formed PVs through combination. But the research did not discuss the relationship between PVs and SR. Wang and Soergel's research focused on the influence of the five values on the actual document selection behavior (acceptance, maybe, or rejection) under the guidance of decision rules.

Behavior intention

3. Methodology 3.1. Conceptual model and assumptions Based on the similarity between SR judgment and commodity value perception, this study summarizes the core variables involved in the study and their relationships as shown in Fig. 1. This conceptual model mainly explains the core assumptions and basis of this study at three levels. First, user relevance is the psychological and subjective relevance perceived by users (Cosijn & Ingwersen, 2000; Harter, 1992; Ingwersen and Peter, 1992; Saracevic, 1975; Schamber et al., 1990), and SR expresses it as the user's perception of the utility of information objects in solving context problems. Therefore, SR judgment is essentially an expression of the user's beliefs that affects the user's behavioral intentions, and the intention is the factor directly affecting actual behaviors (Azjen, 1991). Second, based on the similarity of the definition and decision process of SR and commodity value perception, the SR (utility) is decomposed into users' perception of the four PVs of information, assuming that they have a positive impact on the SR judgment, thus forming the structural model to be verified in this study (structural relationship of latent variables). Third, based on the similarity of the definitions of SR and PV, the mapping relationship between RJC and PVs is established. The specific mapping relationship is based on the comparison between the definition of Sheth's PVs and the consistency of connotation of the 11 crosssituational RJC summarized in this study (the relationship between measurement index and latent variables). 3.2. Operational model and assumptions Based on the conceptual model, this study proposes an operational model to be verified as shown in Fig. 2. Four information PVs affect SR judgment, while 11 RJC and PVs have a reflective relationship. In order to compare the consistency of RJC and PVs in concept connotation in detail, the original definitions of five PVs proposed by Sheth et al. (1991) are as follows:

2.3. Summary Researchers have identified a number of RJC, but RJC has many limitations as measurement dimensions of SR. Researchers explored the possible dimensions of SR measurement, providing important reference for this study. In this study the similarity between SR judgment and commodity value perception were compared (see 1.1 for detailed discussion of the multi-factors PV), and verified the effectiveness of PVs as the dimension of SR measurement through hypothesis testing.

• “Epistemic • 3

value—the perceived utility acquired from an alternative's capacity to arouse curiosity, provide novelty, and/or satisfy a desire for knowledge; Functional value—the perceived utility acquired from an

Library and Information Science Research 41 (2019) 100982

J. Liu, et al.

Quality Recency Completeness

Topicality

Special request Accessibility

H5

Functional Value

H6 H7

H2

Epistemic value

Novelty

H1

Intelligibility

Conditional Value

H3

Convenience

PU1 Authority

Social Value

Situational Relevance (Perceived utility)

H4

Normative

PU2

PU3 PU4

Fig. 2. Research model and hypothesis. Note: hypothesis testing result with SmartPLS3; SRMR = 0.088; *P < 0.05, **P < 0.01, ***p < 0.001.

• • •

alternative's capacity for functional, utilitarian, or physical performance; Conditional value—the perceived utility acquired by an alternative as the result of the specific situation or set of circumstances facing the choice maker. Social value—the perceived utility acquired from an alternative's association with one or more specific social groups; Emotional value—the perceived utility acquired from an alternative's capacity to arouse feelings or affective states.” (p. 160–163)

functional value were extracted. Hypothesis 3. Accessibility and convenience reflect the connotation of conditional value and have a positive effect on SR judgment. This study also redefined conditional value to fit information context (see Appendix A). By comparing the definition with the connotation of core RJC (as shown in Table 1), 2 RJC corresponding to conditional value were extracted. Hypothesis 4. Authority, normative reflect the connotation of social value and have a positive effect on SR judgment.

3.2.1. Exclusion of emotional value This study did not measure emotional value for two reasons: firstly, compared to a problem-solving context, emotional factors are more likely to affect user relevance judgment in a non-problem-solving context (such as surfing the internet for entertainment) (Xu, 2007). This study takes scientific data users with real tasks of completing a competition work within limited time which is a typical problem-solving task. In addition, the RJC expressing emotional state does not appear in the 11 types of cross-situational RJC summarized in this research. To some extent, it also proves that emotional factors are not the core factors. Therefore, this study did not verify the influence of emotional factors on SR judgment. The mechanism of the influence of emotional factors on user relevance alone could be considered in future research.

This study also redefined social value to fit information context (see Appendix A). By comparing the definition with the connotation of core RJC (as shown in Table 1), 2 RJC corresponding to social value were extracted. Hypotheses H5–H7. Epistemic value has a positive effect on functional value, conditional value and social value. Hypotheses of H5–H7 are proposed based on the three aspects. Firstly, based on the hierarchical characteristics of user information needs, Taylor (1968) divided the user's information needs into four progressive levels: visceral (Q1), conscious (Q2), formalized (Q3) and compromised (Q4) information need. In the early stage of Q1 to Q2, users are generally in an anomalous state of knowledge (topics and problems not well defined/topics often unfamiliar) (Belkin, Oddy, & Brooks, 1982). Returning to the four PVs of information, epistemic value expresses the user's perception of unknown knowledge and new knowledge. Secondly, based on the RJC corresponding to epistemic in H1, intelligibility, topicality, and novelty reflect the connotation of epistemic value, while users pay more attention to these three RJC at the initial stage of user relevance judgment. In addition, Wang and Soergel (1998) asserted that “epistemic value is the prerequisite for all other types of value, and document without epistemic value will definitely be rejected” (p. 21). Therefore, this study assumes that epistemic value plays a prerequisite role in SR judgment and affects other values.

3.2.2. Research hypotheses Hypothesis 1. Topicality, novelty, and intelligibility reflect the connotation of epistemic value and have a positive effect on SR judgment. According to the similarity between SR judgment and commodity value perception, the research assumed that epistemic value has a positive effect on SR judgment. As a borrowed concept from Sheth et al. (1991), this study redefined epistemic value to fit the information context (see Appendix A). By comparing the definition with the connotation of core RJC (as shown in Table 1), three RJC corresponding to epistemic value were extracted. Hypothesis 2. Quality, recency, completeness, and special request reflect the connotation of functional value and have a positive effect on SR judgment.

3.3. Participants and data collection 3.3.1. Scientific data users In order to verify the hypotheses, scientific data users with real tasks were recruited as subjects in this study. In order to better explain the purpose and significance of this study, it is necessary to discuss the

This study also redefined functional value to fit the information context (see Appendix A). By comparing the definition with the connotation of core RJC (as shown in Table 1), 4 RJC corresponding to 4

Library and Information Science Research 41 (2019) 100982

J. Liu, et al.

characteristics of scientific data and their users. In a broad sense, data refers to the symbolic expression of target information objects (Borgman, 2013). Scientific data, also known as research data, refers to data in scientific research situations and is the basic material supporting research activities. With the ever-growing open science and data sharing, the curation, retrieval, and sharing of scientific data are of great importance. For example, Google released its Google dataset search1 in 2018, which is a dataset search service that is similar to Google search but was released almost 19 years later. The research of relevance judgment of scientific data users is very important to explain and predict the retrieval behavior of these people, affecting the design and development of user-oriented scientific data retrieval systems as well. Looking at the existing user relevance studies, most of them use documents (Barry, 1994; Wang & Soergel, 1998, etc.) as information carriers. There is also research focused on the relevance for images (Markkula & Sormunen, 2000), audio (Laplante, 2010), and web pages (Tombros, Ruthven, & Jose, 2003), but there is a lack of attention on scientific data.

epistemic value has a significant impact on the SR judgment) and further explain some phenomena and theories.

3.3.2. Data collection process The data collection of this study is based on a national competition (Innovation Competition of Science and Technology Resources Sharing Service for College Student, “Sharing Cup”2for short) of scientific data in China. The purpose of the competition is a national science and technology activity aimed at promoting the reuse and efficiency of scientific data through competition. Twenty-three scientific data sharing platforms (responsible for the management, organization and sharing of scientific data in different fields) provide scientific data used in competitions and related research issues (such as, the competition topic of “Chinese medicine data mining” given by the Scientific Data Sharing Platform of Population and Health3). Competitors submit possible works in the form of research papers, multimedia, website systems, business plans, etc. based on given scientific data and competition topics. Competitors can retrieve relevant data from 23 platforms and can also expand data from other scientific data resources. This study investigated the participants of the sixth year of competition (May 2018–December 2018). Students from universities and research institutes all over China (mainly undergraduates and postgraduates) are all encouraged to participate. The feedback task lasted for three months from December 2018 to February 2019 after the competition was closed. The participants received questionnaires from official notices and emails, (see Appendix A). Users scored each measurement variable according to its importance using a seven-level scale (the importance increases continuously from 1 to 7). Four hundred and fifty-three valid feedback questionnaires (excluding 18 invalid questionnaires) were received, with a recovery rate of 96% (see 4.1 for detailed demographic information).

The measurement model mainly verifies the structural validity of the construction. Structural validity mainly tests the internal consistency, convergence validity, and discrimination validity of construction. In this study, SmartPLS3 was used to evaluate the structural validity of the measurement model (Ringle, Wende, & Becker, 2015). Cronbach's alpha (α) and composite reliability (CR) are important indicators to measure internal consistency. In exploratory research, CR and α require greater than 0.6 (Chin, 1998; Höck & Ringle, 2006), while standardized loading (SL) requires greater than 0.7 (Hair, Ringle, & Sarstedt, 2011). Convergence validity is mainly verified by average variance extracted (AVE). AVE should be greater than 0.5 in exploratory research. As shown in Table 2, CR, SL, α, and AVE all meet the above requirements. The discrimination validity is verified by Fornel-Larcker Criterium. Table 3 shows that if the top value (square root of AVE) in each column is greater than other values in that column, the discrimination validity is valid (Fornell & Larcker, 1981). As shown in Table 2 and Table 3, the measurement model in this study meets all requirements. At the same time, the measurement variables corresponding to latent variables are all significant at the level of p < 0.001, which verifies the relationship between RJC and four PVs in this study. To further illustrate the validity of convergent and discriminative of each sub-item to the intended latent variable, Table 4 lists the cross loadings values for each measurement index. For the intended latent variable, if the load value is greater than 0.7, then it reflects the connotation of the latent variable and has convergent validity. At the same time, for the non-intended latent variables, the load value is required to be less than 0.6 (0.5 in some studies), that is, the index has discriminative validity (Garson, 2016). From Table 4, the convergence and discrimination validity of the measurement index meet the basic requirements.

4. Results 4.1. Demographic information This study received 453 valid feedback questionnaires (excluding 18 invalid questionnaires). The gender ratio of the subjects was balanced (M = 57.8%, F = 42.2%), the majority of subjects were postgraduate students (postgraduate = 82.4%, undergraduate = 17.6%). The age range was mainly 18–30 (18–30 = 91.6%, other = 8.4%). At the same time, users' scientific data retrieval frequency is above 1–2 times per week (Never = 0, 1–2 Per Week = 37.7, 2–5 Per Week = 28.5; More than 1 per day = 34.8). Ten users who never used scientific data were deleted as invalid data. 4.2. Measurement model

3.4. Approach to data analysis

4.3. Structural model

A strict psychological measurement method-structural equation model (SEM) was used in this study. Anderson and Gerbing (1988) proposed this method to develop and verify theoretical assumptions. As an effective psychometric analysis method, SEM is widely used in behavioral science, marketing, education and other fields (Garson, 2016). The analysis process of SEM is mainly divided into two steps: measurement model and structural model. The measurement model is used to verify the structural stability between the measurement index and the latent variable (where topicality is statistically significant as a measurement index of epistemic value). The structural model is used to verify the stability of the relationship between latent variables (whether

Based on the measurement model, the structural model mainly tests the research hypothesis (the relationship between latent variables) and the interpretation and prediction ability of the model. As shown in Fig. 3 of the structural equation model verified in this study, the path coefficients are all normalized coefficients. The validity of latent variable relation is mainly through bootstrap resampling technique (5000 bootstrap samples; no sign changes) provides p-values and CLs to evaluate the significance of paths (Nevitt & Hancock, 2001). The results showed that H1, H5, H6, and H7 were significant at the level of p < 0.001, while H2 and H3 were significant at the level of p < 0.05. H4 is not significant and the research hypothesis is not valid. The interpretation and prediction capabilities of the model are mainly verified by the following indicators: R2, SRMR. R2 is an important indicator to explain the predictive ability of the model, the

1

https://toolbox.google.com/datasetsearch http://share.escience.net.cn 3 http://www.ncmi.cn/ 2

5

Library and Information Science Research 41 (2019) 100982

J. Liu, et al.

Composite-based standardized root mean square residual (SRMR) is an index to evaluate the overall fitting degree of the model. It measures the difference between the observed correlation matrix and model-implied correlation matrix (Henseler et al., 2014). The smaller SRMR, the better the fitting, and the threshold is less than 0.10. As shown in Fig. 3, R2 of latent variables in this study are all greater than0.25. Meanwhile, SRMR = 0.088 is within the acceptable range as shown in the note of Fig. 3. The results show that this research model has strong interpretation and prediction ability.

Table 2 Assessment results of the measurement model. Construct

Items

Epistemic value

Functional value

Conditional value Social value Perceived utility

Mean

SD

SL

Topicality Intelligibility Novelty

5.642 5.673 5.499

1.269 1.158 1.271

0.807*** 0.779*** 0.781***

Quality Recency Special request Completeness

6.024 5.766 5.581 5.709

1.157 1.244 1.341 1.188

0.854*** 0.824*** 0.601*** 0.785***

Accessibility Convenience

5.669 5.726

1.265 1.141

0.886*** 0.810***

Authority Normative

5.936 5.773

1.144 1.212

0.924*** 0.925***

PU1 PU2 PU3 PU4

5.996 5.768 5.874 5.804

0.954 1.088 1.057 1.117

0.865*** 0.774*** 0.806*** 0772***

α

C.R

AVE

0.697

0.837

0.720

0.770

0.853

0.596

5. Discussion and implications 0.616

0.837

0.720

0.819

0.880

0.648

0.830

0.922

0.854

5.1. Perceived values as effective measurement dimensions of situational relevance The empirical results of this study show that the four information values are statistically significant as measurement dimensions of SR, and the structural model is effective (SRMR = 0.088 < 0.1, RFV2 = 0.601, RCV2 = 0.577, RSV2 = 0.457, RSR2 = 0.364 > 0.25). Therefore, a new operational definition for SR is given: “SR is the user's comprehensive perception of the utility of information objects in solving current tasks/problems based on the epistemic value, functional value, social value, and conditional value of information objects”. This definition follows the classic definition of utility as the operational concept of SR (Cosijn & Ingwersen, 2000; Saracevic, 1975, 1996), and provides the measurement dimensions of SR. Information users make comprehensive judgments according to the differences of situations and considering multiple or all information values. Comprehensive judgments, as a component of users' behavioral intentions, further affect users' actual information behaviors (selecting, reading and using).

Note: ***Significant at 0.001 (two-tailed); SL = standardized loading; C.R = composite reliability; α = Cronbach's alpha; and AVE = average variance extracted. Table 3 Fornell-Larcker Criterium. Latent variable correlations(LVC) CV CV EV FV PU SV

0.849 0.760 0.772 0.529 0.632

EV 0.789 0.775 0.565 0.676

FV

0.772 0.556 0.737

PU

0.805 0.487

Discriminant validity met? (Square root of AVE > LVC?)

5.2. The effect of perceived values on situational relevance

SV

0.769

The study verified that different information values have different effects on the user SR judgment. Epistemic value has a prerequisite effect on the SR judgment in this study context. As shown in Fig. 3, the influence of epistemic value on user SR judgment is significant at the level of p < 0.001, while the influence of functional value and conditional value on the user SR judgment is significant at the level of p < 0.05. However, social value has no significant influence on user SR judgment.

Yes Yes Yes Yes Yes

Note: CV = Conditional value; EV = Epistemic value; FV = Functional value; SV = Social value; PU = Perceived utility. Table 4 Cross Loadings.

Accessibility Convenience Topicality Novelty Intelligibility Completeness Special request Recency Quality Authority Normative PU1 PU3 PU2 PU4

Conditional value

Epistemic value

Functional value

Social value

Perceived_utility

0.886 0.810 0.628 0.535 0.627 0.646 0.421 0.609 0.679 0.594 0.575 0.481 0.395 0.418 0.403

0.726 0.549 0.807 0.781 0.779 0.554 0.409 0.667 0.717 0.611 0.638 0.534 0.382 0.431 0.458

0.677 0.634 0.587 0.618 0.63 0.785 0.601 0.824 0.854 0.698 0.665 0.532 0.391 0.441 0.412

0.57 0.499 0.534 0.467 0.591 0.528 0.304 0.599 0.765 0.924 0.925 0.44 0.375 0.375 0.373

0.478 0.417 0.503 0.371 0.455 0.435 0.354 0.437 0.484 0.466 0.435 0.865 0.774 0.806 0.772

The bold items are the measurement index of intended latent variable in each column

At the same time, this study verified the “special” position of epistemic value in SR judgment. Epistemic value has a significant effect on the other three values (H5, H6, and H7 are all significant at the level of p < 0.001). It verified that, in the early stage of information need from

bigger the value of the R-square, the stronger the model predictor for the variance explanation of the endogenous variable (Latan & Ramli, 2013). The R2 values of 0.25, 0.40, and 0.75 respectively indicate the weak, medium, and strong states of the prediction ability of the model. 6

Library and Information Science Research 41 (2019) 100982

J. Liu, et al.

0.775*** p=0.000)

Functional Value RFV 2=0.601

Epistemic Value

0.760*** p=0.000) 0.676*** p=0.000)

0.188* p=0.022)

0.259*** p=0.000)

Conditional Value RCV2=0.577 0.130* p=0.046) Situational Relevance (Perceived Utility) RSR2=0.364

Social Value RSV2=0.457 0.092 p=0.153) Fig. 3. Information Value-Utility model.

5.5. Limitations and future direction

Q1 to Q2 (Taylor, 1968), a user needs to judge the epistemic value of the information object. The user's perception of the epistemic value of information is the premise of perceiving other values. Moreover, this result also confirms the primary and prerequisite role of topicality (Barry, 1994; Cool et al., 1993; Schamber, 1991; Wang & Soergel, 1998).

Some limitations should be mentioned. Firstly, this research is based on a typical problem-solving situation (searching scientific data for a competition work), the research results may not have sufficient explanatory power for the cognitive process of information relevance judgment in a non-problem-solving situation (e.g., browsing news in web pages for entertainment, etc.), which involves emotions and needs further exploration. Secondly, with college students as samples of scientific data users, the generalization of results must be approached very carefully. In the future, research with more types of situations and data users should be conducted to further verify the results.

5.3. The mapping relationship between relevance judgment criteria and perceived values According to the research results in Table 2, the mapping relationship between different RJC and four PVs is significant at the level of p < 0.001. Epistemic value reflects users' exploration of unknown or new knowledge, and the corresponding RJC are topicality, intelligibility, and novelty. The function value reflects the user's perception of the specific function (objective attributes) of information, and the corresponding RJC are quality, recency, special request, and completeness. Conditional value reflects the user's perception of the restriction of the objective environment of information, and the corresponding RJC are accessibility and convenience. Social value reflects users' perception of the social attributes of the information, and the corresponding RJC are authority and normal. RJC are not limited to the core set of RJC summarized in this study; other RJC can be mapped to PV as long as their connotations correspond to the PVs.

5.6. The implication for understanding relevance This study has important theoretical implications for understanding the nature of user relevance and measurement. The new operational definition for SR mainly explains the characteristics of user relevance from the following four aspects. First, user relevance is essentially a kind of psychological perception (belief to some extent) of users. Utility, as an operational concept of SR, emphasizes the importance of context. Second, the utility can be obtained through user's perception of four PVs of information objects. Third, the importance of the situation, relevance judgment cannot be separated from the situation (Anderson, 2005; Saracevic, 1996). Fourth, PVs are the higher constructions of RJC. Four information PVs comprehensively cover the dimensions that affect the user SR judgment.

5.4. Why social value has no significant effect on situational relevance in this study It is unexpected that social value has no significant effect on users' perceived information utility in this study. This may be related to the problem-solving situation (searching scientific data for a competition work) as the research situation in this study. In problem-solving situations when information resources are limited, users are likely to consider the epistemic value and functional value of information first, and social value may be used by users more in selecting relevant information. Although social value has no significant influence on users' perceived information utility in this study, as shown in Table 2, authority and normative are significant as measurement variables of social value. Perhaps social value plays a significant role in non-problemsolving situations (e.g., browsing news in web pages for entertainment, etc.), which requires further exploration.

5.7. The implication for scientific data retrieval and discovery It is true that information retrieval (IR) benefits more from early rather than modern relevance research, partly because the huge barriers between research outputs to their applications, and partly because we have not understood how people make their relevance judgment. This research tries to make one step toward application by providing a comparison between SR judgment and commodity value perception. People have developed many effective techniques and tools to help customers make their judgment, including information representation and discovery. People can also improve searching and utilizing scientific data based on and inspired by the understanding of the relationship between RJC and SR via PVs. 7

Library and Information Science Research 41 (2019) 100982

J. Liu, et al.

6. Conclusion

Acknowledgements

By introducing the theory of multi-factors PV, the research develops a value-utility model for relevance judgment aiming to provide an explanation of how SR judgment is drawn from RJC. In the model, four PVs function as a bridge from RJC to SR. The research verifies such a relationship by testing seven hypotheses derived from the value-utility model with SEM method. With the relations between RJC and PVs, and with the relations between PVs and SR, this study gives a possible path of how relevance drew from PVs and RJC. Besides the path suggested, the study also contributes to the understanding of the nature of SR from the viewpoint of PV, and proposes a more operational definition for SR. The new understanding has implications for both relevance research and scientific data retrieval.

This work was supported by grants from National Social Science fund project titled “Scientific Data User Relevance Criteria and Use Model Empirical Study (14BTQ056)”, National Science & Technology Infrastructure of China fund project titled “Research on Quality Evaluation of Scientific Data”, and Agricultural Science, Technology Innovation Project of CAAS (Project No.CAAS-ASTIP-2016-AII), and the project of Fundamental Research Funds for CAAS, “The Construction of Platform and Data Mining for Crop Breeding (Y2016XK08)” and The Establishment of Data Collection Standard and System Development for Agricultural Experimental Stations (Y2019PT27)”. The authors also acknowledge the data collection support from National Science & Technology Infrastructure of China (http://www. escience.net.cn).

Appendix A. Major items of the questionnaire 1. In the process of completing the competition works, please evaluate the importance of the following relevance criteria in judging the epistemic value of scientific data, and the importance gradually increases from 1 to 7. Topicality:the data are consistent with my research topic Intelligibility:I can understand the subject and content of data Novelty:data are novel in content, processing methods, etc. 2. In the process of completing the competition works, please evaluate the importance of the following relevance criteria in judging the functional value of scientific data, and the importance gradually increases from 1 to 7. Quality:the data meet the requirements in terms of precision, accuracy, verifiability, etc. Recency:the data is relatively recent in time to complete my task Special request:the data meet some specific need, such as the description language of the data, etc. Completeness:the data shows the necessary data elements, and missing data will not influence the completion of this task. 3. In the process of completing the competition works, please evaluate the importance of the following relevance criteria in judging the conditional value of scientific data, and the importance gradually increases from 1 to 7. Accessibility:the data can be downloaded directly and all data can be viewed Convenience:It is convenient to acquire the data and requires no unnecessary cost. 4. In the process of completing the competition works, please evaluate the importance of the following relevance criteria in judging the social value of scientific data, and the importance gradually increases from 1 to 7. Authority:The source of the data is reliable Normative:Data collection methods, processing methods and formats are recognized by the academic community. 5. In the process of completing the competition work, please evaluate the following statements describing the role of scientific data in your work, and the importance gradually increasing from 1 to 7. PU1: Scientific data as research evidence PU2: Scientific data can verify research theories PU3: Scientific data is the basis of my research PU4: Scientific data filled up my cognitive gap

1 1 1

2 2 2

3 3 3

4 4 4

5 5 5

6 6 6

7 7 7

1 1 1 1

2 2 2 2

3 3 3 3

4 4 4 4

5 5 5 5

6 6 6 6

7 7 7 7

1 1

2 2

3 3

4 4

5 5

6 6

7 7

1 1

2 2

3 3

4 4

5 5

6 6

7 7

1 1 1 1

2 2 2 2

3 3 3 3

4 4 4 4

5 5 5 5

6 6 6 6

7 7 7 7

Note: The definitions of the 4 types of PVs are described as follows. Epistemic value: the perceived utility of information to satisfy a desire for knowledge, curiosity for new thing or information that is unknown. Functional value: the perceived utility of information making the contribution to the specific task at hand. Conditional value: the perceived utility of information is yet to be decided circumstantially. Social value: the perceived utility of information in association with specific social groups or with individuals such as academic advisor famous figures in the field etc.

Chin, W. W. (1998). The partial least squares approach to structural equation modeling. In G. A. Marcoulides (Ed.). Modern methods for business research (pp. 295–336). Mahwah, NJ: Lawrence Erlbaum Associates. Choi, Y., & Rasmussen, E. M. (2002). Users' relevance criteria in image retrieval in American history. Information Processing & Management, 38, 695–726. Cool, C., Belkin, N. J., & Kantor, P. B. (1993). Characteristics of texts affecting relevance judgments. In M. E. Williams (Ed.). Proceedings of the 14th national online meeting (pp. 77–84). Medford, NJ: Learned Information. Cosijn, E., & Ingwersen, P. (2000). Dimensions of relevance. Information Processing & Management, 36, 533–550. Fitzgerald, M. A., & Galloway, C. (2001). Relevance judging, evaluation, and decision making in virtual libraries: A descriptive study. Journal of the American Society for Information Science and Technology, 52, 989–1010. Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18(1), 39–50. Garson, G. D. (2016). Partial least squares: Regression and structural equation models. Asheboro, NC: Statistical Associates Publishers. Greisdorf, H. (2003). Relevance thresholds: A multi-stage predictive model of how users evaluate information. Information Processing & Management, 39, 403–423. Greisdorf, H. F. (2000). Relevance thresholds: A conjunctive/disjunctive model of end-user cognition as an evaluative process (Unpublished doctoral dissertation) Denton, TX: University of North Texas. Grice, H. P. (1991). Studies in the way of words. Cambridge, MA: Harvard University Press.

References Anderson, J. C., & Gerbing, D. W. (1988). Structural equation modeling in practice: A review and recommended two-step approach. Psychological Bulletin, 103, 411–423. Anderson, T. D. (2005). Relevance as process: Judgements in the context of scholarly research. Information Research: An International Electronic Journal, 10. Azjen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50, 179–211. Barry, C. L. (1993). The identification of user criteria of relevance and documents characteristics: Beyond the topical approach to information retrieval. (Unpublished doctoral dissertation) Syracuse, NY: Syracuse University. Barry, C. L. (1994). User-defined relevance criteria: An exploratory study. Journal of the American Society for Information Science, 45, 149–159. Barry, C. L., & Schamber, L. (1998). Users' criteria for relevance evaluation: A crosssituational comparison. Information Processing & Management, 34, 219–236. Belkin, N. J., Oddy, R. N., & Brooks, H. M. (1982). Ask for information retrieval: Part II. Results of a design study. Journal of Documentation, 38, 145–164. Borgman, C. L. (2013). Scholarship in the networked world: Big data, little data, no data. Oliver Smithies Lecture, Balliol College, Oxford. Borlund, P. (2003). The concept of relevance in IR. Journal of the American Society for Information Science and Technology, 54, 913–925. Borlund, P., & Ingwersen, P. (1997). The development of a method for the evaluation of interactive information retrieval systems. Journal of Documentation, 53, 225–250.

8

Library and Information Science Research 41 (2019) 100982

J. Liu, et al.

Toward a dynamic, situational definition∗. Information Processing & Management, 26, 755–776. Sedghi, S., Sanderson, M., & Clough, P. (2013). How do health care professionals select medical images they need? Aslib Proceedings New Information Perspectives, 65, 54–72. Sheth, J. N., Newman, B. I., & Gross, B. L. (1991). Why we buy what we buy: A theory of consumption values. Journal of Business Research, 22, 159–170. Taylor, A. (2009). Relevance criterion choices in relation to search progress. Dissertations & Theses - Gradworksvol. 26, 203–208. Taylor, A. (2013). Examination of work task and criteria choices for the relevance judgment process. Journal of Documentation, 69, 523–544. Taylor, A. R., Cool, C., Belkin, N. J., & Amadio, W. J. (2007). Relationships between categories of relevance criteria and stage in task completion. Information Processing & Management, 43, 1071–1084. Taylor, R. S. (1968). Question-negotiation and information seeking in libraries. College & Research Libraries, 29, 178–194. Tombros, A., Ruthven, I., & Jose, J. M. (2003). Searchers criteria for assessing web pages. Proceedings from the 26th annual international ACM SIGIR conference on research and development in information retrieval (pp. 385–386). Toronto, Canada: ACM. Vakkari, P. (2000). Relevance and contributing information types of searched documents in task performance. Proceedings of the 23rd annual international ACM SIGIR conference on research and development in information retrieval (pp. 2–9). Athens, Greece, New York: ACM (July 24–28). Wang, H. Y., & Wang, S. H. (2010). Predicting mobile hotel reservation adoption: Insight from a perceived value standpoint. International Journal of Hospitality Management, 2, 598–608. Wang, P., & Soergel, D. (1998). A cognitive model of document use during a research project: Study I. document selection. Journal of the American Society for Information Science, 49, 115–133. Wang, P., & White, M. D. (1999). A cognitive model of document use during a research project: Study II. Decisions at the reading and citing stages. Journal of the American Society for Information Science, 50, 98–144. Wilson, P. (1973). Situational relevance. Information Storage & Retrieval, 9, 457–471. Xu, Y. (2007). Relevance judgment in epistemic and hedonic information searches. Journal of the American Society for Information Science and Technology, 58, 179–189. Xu, Y., & Chen, Z. (2006). Relevance judgment: What do information users consider beyond topicality? Journal of the Association for Information Science and Technology, 57, 961–973. Zeithaml, V. A. (1988). Consumer perceptions of price, quality, and value: A means-end model and synthesis of evidence. Journal of Marketing, 52, 2–22. Zhitomirsky-Geffet, M., Bar-Ilan, J., & Levene, M. (2017). Analysis of change in users' assessment of search results over time. Journal of the Association for Information Science and Technology, 68, 1137–1148.

Hair, J. F., Ringle, C. M., & Sarstedt, M. (2011). PLS-SEM: Indeed a silver bullet. Journal of Marketing Theory and Practice, 19, 139–152. Harter, S. P. (1992). Psychological relevance and information science. Journal of the Association for Information Science and Technology, 43, 602–615. Henseler, J., Dijkstra, T. K., Sarstedt, M., Ringle, C. M., Diamantopoulos, A., Straub, D., et al. (2014). Common beliefs and reality about partial least squares: Comments on Rönkkö & Evermann (2013). Organizational Research Methods, 17, 182–209. Hirsh, S. G. (1999). Children's relevance criteria and information seeking on electronic resources. Journal of the American Society for Information Science, 50, 1265–1283. Hjørland, B. (2002). Epistemology and the socio-cognitive perspective in information science. Journal of the Association for Information Science and Technology, 53, 257–270. Höck, M., & Ringle, C. M. (2006). Strategic networks in the software industry: An empirical analysis of the value continuum. Social Science Electronic Publishing, 4, 132–151. Ingwersen & Peter (1992). Information retrieval interaction. Journal of the American Society for Information Science, 44, 303–304. Ingwersen, P., & Järvelin, K. (2011). The turn: Integration of information seeking and retrieval in context. Berlin, Germany: Springer Publishing Company, Incorporated. Kuhlthau, C. C. (1991). Inside the search process: Information seeking from the user's perspective. Journal of the American Society for Information Science, 42, 361–371. Ladhari, R., & Morales, M. (2008). Perceived service quality, perceived value and recommendation: A study among Canadian public library users. Library Management, 29, 352–366. Laplante, A. (2010). Users' relevance criteria in music retrieval in everyday life: An exploratory study. Proceedings of the 11th international society for music information retrieval conference (ISMIR’10). Utrecht, The Netherlands: ISMIR. Latan, H., & Ramli, N. (2013). The results of partial least squares-structural equation modelling analyses. PLS-SEM. Retrieved from http://ssrn.com/abstract=2364191. Levene, M., Bar-Ilan, J., & Zhitomirsky-Geffet, M. (2018). Categorical relevance judgment. Journal of the Association for Information Science and Technology, 69, 1084–1094. Maglaughlin, K. L., & Sonnenwald, D. H. (2002). User perspectives on relevance criteria: A comparison among relevant, partially relevant, and not-relevant judgments. Journal of the American Society for Information Science and Technology, 53, 327–342. Markkula, M., & Sormunen, E. (2000). End-user searching challenges indexing practices in the digital newspaper photo archive. Information Retrieval, 1, 259–285. Misilei, J., & Liew, C. L. (2018). Perceived value of digital components in library programmes: The case of Auckland Libraries' dare to explore summer reading programme. Library & Information Science Research, 40, 2019–2236. Mizzaro, S. (1997). Relevance: The whole history. Journal of the American Society for Information Science, 48, 810–832. Nevitt, J., & Hancock, G. R. (2001). Performance of bootstrapping approaches to model test statistics and parameter standard error estimation in structural equation modeling. Structural Equation Modeling, 8, 353–377. Pihlström, M., & Brush, G. J. (2010). Comparing the perceived value of information and entertainment mobile services. Psychology & Marketing, 25, 732–755. Ringle, C. M., Wende, S., & Becker, J. M. (2015). Smart PLS 3. Bönningstedt: SmartPLS. Retrieved from http://www.smartpls.com. Saracevic, T. (1975). Relevance: A review of and a framework for the thinking on the notion in information science. Journal of the Association for Information Science and Technology, 26, 1915–1933. Saracevic, T. (1996). Relevance reconsidered. In P. Ingewersen, & N. O. Pors (Eds.). Proceedings of second international conference on conceptions of library and information science, (CoLIS 1996) (pp. 201–218). Copenhagen: The Royal School of Librarianship. Saracevic, T. (2007). Relevance: A review of the literature and a framework for thinking on the notion in information science. Part III: Behavior and effects of relevance. Journal of the American Society for Information Science and Technology, 58, 2126–2144. Saracevic, T. (2016). The notion of relevance in information science: Everybody knows what relevance is. But, what is it really? Synthesis Lectures on Information Concepts Retrieval, and Services, 8(3) (i-109). Schamber, L. (1991). Users' criteria for evaluation in multimedia information seeking and use situations. (unpublished doctoral dissertation) Syracuse, NY: Syracuse University. Schamber, L. (1994). Relevance and information behavior. Annual Review of Information Science and Technology, 29, 3–48. Schamber, L., Eisenberg, M. B., & Nilan, M. S. (1990). A re-examination of relevance:

Jianping Liu is a graduate research associate at the Agricultural Information Institute, Chinese Academy of Agricultural Sciences. He is a PhD student in information retrieval technology in Chinese Academy of Agricultural Sciences. His research focuses on cognitive mechanisms and modeling of user information relevance judgment process. His research has been published in The 3rd International Conference on Computer Science and Application Engineering (2019) and Man-Machine-Environment system engineering:Proceedings of the 19th international conference on MMESE (2019). Jian Wang is a professor in information sciences at the Cognitive Computing Research Laboratory of the Agricultural Information Institute, Chinese Academy of Agricultural Sciences. He holds a PhD in geoinformatics from the Chinese Academy of Sciences. He is the editor of the Journal of Agricultural Big Data in China. His research has been published in International Symposium on Intelligent Information Technology in Agriculture (2007) and Sensor Letters (2010). Guomin Zhou is a chancellor's professor in information sciences and deputy director of Agricultural Information Institute, Chinese Academy of Agricultural Sciences. He holds a PhD in information sciences from the Chinese Academy of Sciences. He is the executive editor of the Journal of Agricultural Big Data in China. His research has been published in Intelligent Automation and Soft Computing (2010) and Telkomnika Indonesian Journal of Electrical Engineering (2014).

9