The effects of contextualized access to knowledge on judgement

Int. J. Human-Computer Studies (2001) 55, 787}814 doi:10.1006/ijhc.2001.0507 Available online at http://www.idealibrary.com on

The effects of contextualized access to knowledge on judgement JI-YE MAO Department of Management Sciences, University of Waterloo, Waterloo, Ontario, Canada N2L 3G1. email: [email protected] IZAK BENBASAT Faculty of Commerce and Business Administration, University of British Columbia, Vancouver, Canada V6T 1Z2. email: [email protected] (Received 24 July 2000, and accepted in revised form 15 August 2001) This research conceptualizes contextualized access to knowledge, i.e. the ability to access task domain knowledge within the context of problem-solving and investigates its e!ects on knowledge dissemination. Two informationally equivalent versions of a "nancial analysis knowledge-based system (KBS) were compared in a laboratory experiment, one with contextualized access to the underlying task domain knowledge (deep explanations) via hypertext-style links and the other without such access. Results indicate that contextualized access had signi"cant advantages. It a!orded a major portion of the requests for deep explanations to occur in the context of problem-solving, as opposed to in the abstract, and led to a signi"cant increase in the number of requests. The increased utilization of deep explanations and contextualized use were associated with a greater degree of congruence between users' judgement and KBS. The conclusion is that availability of knowledge alone is not su$cient; contextualized accessibility is the key for knowledge dissemination and for in#uencing performance.  2001 Academic Press KEYWORDS: contextualized access to knowledge; knowledge-based systems; explanation; hypertext; e!ort-accuracy tradeo!; knowledge management.

1. Introduction As the foundation of economies has shifted from natural resources to intellectual assets, organizations are being compelled to enhance their knowledge management practices. Meanwhile, advances in computer technologies have made it easier and more coste!ective to codify, store and share certain kinds of knowledge. The trend has been to make more knowledge, such as best practices, success stories, policy books and training manuals available on-line through corporate Intranets or stand-alone systems. For example, major banks, insurance companies and utilities "rms are among the organizations that have built computer-based workbenches for customer service representatives (e.g. Desmarais, Leclair, Fiset & Talbi, 1997). However, successful knowledge management requires not only knowledge acquisition and knowledge organization, but also 1071-5819/01/110787#28 $35.00/0

 2001 Academic Press

788

J.-Y. MAO AND I. BENBASAT

access to relevant knowledge in a timely manner (Schwartz & Te'eni, 2000). One of the bigger challenges to be tackled next is to further enhance access to on-line knowledge. Knowledge-based systems (KBS, considered the same as expert systems herein) as a branch of applied arti"cial intelligence not only predate the recent surge of interest in knowledge management, but also stand out as a well-established means for knowledge management. KBS are programs that achieve expert-level competence in solving problems by utilizing knowledge of the task domain (Feigenbaum, McCorduck & Nii, 1988). A fundamental feature of KBS is the ability to provide explanations that are descriptions of what a system does, how it works and why its actions are appropriate (Swartout, 1987). Through the advice and explanations that they provide, KBS can make highly specialized domain knowledge widely available to their users. The objectives of this paper are two-fold. First, drawing upon relevant literature, we de"ne the concept &&contextualized access to knowledge'', and argue that it can be highly e!ective for encouraging and facilitating access to knowledge. Contextualized access means access to relevant task knowledge is immediate and within the problem-solving context rather than via searching in a separate context. Thus, there is no need to break the continuity of task performance to seek relevant domain knowledge. The concept is illustrated with a detailed example of a KBS that uses hypertext-style links to access its underlying domain knowledge. Second, we report on a lab experiment that has investigated the following two questions: Does contextualized access to explanations of underlying KBS knowledge in-uence explanation use and judgement by users, and how? Since explanation use may depend on one's prior knowledge and experience, the second question is: =hat are the e+ects of KBS users' task domain knowledge on their explanation use and judgement? Our focus is on the particular style of accessing knowledge, rather than the means of providing such access. Results from this research have implications for a broad range of systems designed for knowledge management or performance support. Typically, these systems need to provide contextualized access to task domain knowledge as explanation or other forms of support. The importance of explanation has been reinforced in recent years by the development of other types of intelligent systems, such as intelligent agents, interfaces and tutoring systems, because explanation plays a crucial role in the interaction between users and complex systems (cf. Gregor & Benbasat, 1999). Prior research has also explored the use of hypertext for accessing knowledge in systems such as decision support systems (e.g. Bieber & Kimbrough, 1992), performance support systems (e.g. Desmarais et al., 1997) and organizational memory (Schwartz & Te'eni, 2000). However, the concept of contextualized access is not well de"ned and little research has been conducted to investigate its theoretical foundation and empirical basis. This paper is structured as follows. Section 2 de"nes and illustrates contextualized access to knowledge, along with KBS explanations and contextualized access to KBS explanations. Section 3 presents the theoretical foundation of this research, relevant literature and the hypotheses to be tested. Section 4 describes the research method, including the experimental systems and procedures. Data analysis and results are presented in Section 5. Lastly, Section 6 discusses the main conclusions and limitations of this research, and directions for future research.

CONTEXTUALIZED ACCESS TO KNOWLEDGE

789

2. Background 2.1. CONTEXTUALIZED ACCESS TO KNOWLEDGE

Although making helpful information available on-line a!ords many bene"ts that might not be practical in printed form, there are also serious usability problems (cf. Shneiderman, 1998). Most on-line help remains an electronic version of a printed manual, and users must dig through indices "lled with technical gibberish. It is di$cult for users to "nd useful information and to switch between the help and their work context (Head, 1998). Therefore, it is not surprising that on-line help is greatly under utilized. Research in human-computer interactions has identi"ed many reasons for the low usage of on-line help. First, users may not be able to formulate queries e!ectively, i.e. to give precise and di!erentiating descriptions of things they lack knowledge about (Blair & Maron, 1985; Nickerson, 1999). Users are left alone to navigate through the query results (Hertzum & Frokjaer, 1996; Horvitz, 1999), and often need to re-formulate the queries on a trial-and-error basis. This process can be fruitless and frustrating. Second, much empirical evidence has shown that most users put a lot more emphasis on getting their work done than seeking help to optimize their work (Fisher, Lemke & Schwab, 1985; Carroll & Rosson, 1987; Desmarais, Larochelle & Giroux, 1987; Furman & Spyridakis, 1992). One potential solution to these problems is to make on-line help proactive. However, several major challenges have hindered the e!ectiveness of this approach, e.g. correctly inferring a user's task and delivering relevant advice at the right time (Furman & Spyridakis, 1992; Beaumont, 1994; Wolfe & Eichmann, 1997; Agah & Tanie, 2000). Moreover, users like predictability and to be in control, but they do not like surprises (Shneiderman, 1998; Hook, 2000), which are associated with system-initiated help. A more practical approach to enhancing the relevance of on-line help is to make it sensitive to context, e.g. to display appropriate topics according to the context when the F1 key is pressed. For example, it was found that contextualized terminological support, e.g. suggesting a synonym of a term newly added to the query, was important for user interfaces to information retrieval systems, and this feature was frequently requested by users (Brajnik, Mizzaro & Tasso, 1996). Arti"cial intelligence has been applied to the design of context-sensitive help for over two decades (cf. Kearsley, 1988). There has been a great deal of research in providing explanations based on context. The word &&context'' is considered the environment or setting in which something exists or occurs (Mittal & Paris, 1995). Mittal and Paris (1995) proposed a comprehensive framework of context, including the problem-solving situation, participants involved, mode of interaction in which communication occurs, discourse taking place and external world. Providing explanation in context essentially means customizing computer system's output based on these factors. In this research, the concept of contextualized access to knowledge can be illustrated with the following examples. Desmarais et al. (1997) reported on the development of a graphical user interface to a large utility company's customer database. Customer billing information appearing on the computer screen of customer service representatives was hyperlinked to the details of how the amounts were computed. This made the knowledge to explain electricity bills to customers conveniently available, since providing this explanation could be a di$cult task for certain payment plans and required a substantial amount of task domain knowledge.

790

J.-Y. MAO AND I. BENBASAT

A more recent example is a knowledge-enhanced email system, kMail (Schwartz & Te'eni, 2000). The goal is to use organizational knowledge (memory), which can be in the form of de"nitions, graphic images, policy guidelines, product speci"cations or opinions, to enhance email messages. From an email message, hypertext links to segments of organizational memory are provided to create a context for the message to be properly understood. In both of the above examples, relevant domain knowledge was integrated into the output of an information system in a natural and seamless way allowing immediate access to knowledge within the same context of problem-solving, as opposed to via searching in a di!erent context. Contextualized access to knowledge implies two basic features: (1) there is no need for users to leave the task context and break the continuity of their task to access domain knowledge; and (2) there is no, or little, need to search for relevant knowledge, because the applicability of the knowledge is signi"ed and access is immediate through explicitly highlighted references such as hypertext link markers (buttons, coloured or underlined text). Contextualized access satis"es the three conditions speci"ed by Schwartz and Te'eni (2000) for e!ective knowledge dissemination as a key aspect of knowledge management: (1) awareness, i.e. users must be made aware that relevant knowledge is available, (2) identi"cation, which suggests that users must be able to readily identify the useful knowledge, and (3) delivery, which must be to the point of need and in a timely manner. In contrast, access to on-line help in most windows-based systems is not contextualized, including conventional context-sensitive help, although it may use hypertext. It is di!erent from the contextualized access de"ned in this paper, because there exist two separate contexts: the task context vs. the help context, where information has to be searched based on keywords or from indices. Additionally, the user, who has a primary task to complete, has the burden of identifying applicable information from potentially large amounts of irrelevant information from a separate part of the system. The notion of contextualized learning is based on the argument that learning is e!ective when it does not occur in a separate phase and place from problem-solving (Fischer, Lemke & McCall, 1990). A system supporting contextualized learning must help the problem-solvers to see where their knowledge is inadequate (to perceive breakdowns), to "nd the problem-solving knowledge they need for such situations, and to understand how generalized principles relate to particular situations. All of these requirements can be satis"ed by contextualized access to knowledge through hypertextstyle links, which is expected to lead to better task performance. Contextualized access to knowledge can be particularly e!ective for promoting workplace learning. The notion of &&production paradox'' refers to the con#icts between learning and working constantly present in work settings (Carroll & Rosson, 1987). ¸earning is inhibited by lack of time and working is inhibited by lack of knowledge. Consequently, productivity su!ers. However, the &&cost'' of learning may be reduced through the design of better learning support facilities (Carroll & Rosson, 1987), such as contextualized access to knowledge. As a result, problem-solvers may develop a better understanding of the domain, which may result in both better quality and more e$cient task performance. Hypertext is a natural choice for implementing contextualized access, because it can enhance both knowledge representation (organization) and access (interactivity)

CONTEXTUALIZED ACCESS TO KNOWLEDGE

791

(Conklin, 1987). As a knowledge representation scheme, hypertext preserves the rich associations among domain concepts that corresponds to an expert's knowledge schemata (Jonassen, 1990). Domain knowledge about concepts and relationships can be accessed in the context of other related ones. As a user interface modality, hypertext can make domain knowledge more accessible in the context of problem-solving to users from multiple perspectives (Marshall & Shipman, 1995), allowing for di!erent levels of prior knowledge, and encourage explorations of di!erent levels of di$culty, detail and granularity.

2.2. KBS EXPLANATIONS AND CONTEXTUALIZED ACCESS

Explanations are typically generated based on the problem-solving process of a KBS, and are referred to as reasoning-trace explanations, e.g. the How and Why explanations in MYCIN (Shortli!e, 1976). Such explanations are di$cult to understand for several reasons (Moore & Swartout, 1991; Southwick, 1991; Wick & Thompson, 1992), with the most frequently cited one being the lack of knowledge. For example, in a rule-based KBS, problem-solving knowledge is represented implicitly as a set of rules in a compiled form with the underlying justi"cation removed. Moreover, domain objects and static relationships among them are not described, nor are the terminologies de"ned, due to the inherent inadequacy of production rules for knowledge representation (Fikes & Kehler, 1985). Therefore, it is di$cult to justify the conclusions and reasoning process of a KBS solely with reasoning-trace explanations. Research has suggested that explanations need to go beyond reasoning-trace to provide backing or "rst principles for KBS action. For instance, GUIDON (Clancey, 1983) and XPLAIN (Swartout, 1983) provided explanations based on explicitly represented domain knowledge. To respond to the range of questions that the user may ask, the explanation must draw from several di!erent knowledge sources, such as terminological knowledge, factual domain knowledge, problem-solving knowledge (Swartout, Paris & Moore, 1991). Deep explanations justify KBS output by linking it to a causal model of the underlying knowledge, i.e. deep knowledge (Southwick, 1991). In this research, deep knowledge is considered to include the three types of knowledge identi"ed by Swartout and Smoliar (1987) (Table 1). Appendix A has several examples of deep explanations drawn from the experimental KBS used in this research. TABLE 1 Deep knowledge in KBS (Swartout & Smoliar, 1987) Terminological knowledge. Knowledge of concepts and relationships of a domain that domain experts use to communicate with each other. In order for one to understand a domain, one must understand the terms to describe the domain. Domain descriptive knowledge. &&Textbook rudiments'' which are required before one can solve problems. It provides abstract factual knowledge about a domain, typically represented declaratively. Problem-solving knowledge. Knowledge about how tasks can be accomplished. It can be represented as plans and methods that consist of a sequence of steps to accomplish a goal.

792

J.-Y. MAO AND I. BENBASAT

The one-shot nature of MYCIN-style explanation has also been criticized as clearly inconsistent with the inherently incremental and interactive process occurring between human advice-seeker and giver (Moore & Swartout, 1991; Carenini & Moore, 1993). Moore and Swartout developed an explanation mechanism that was able to clarify and elaborate on previous explanations, and to respond to follow-up questions, recognizing the need for the explanation to be interactive (cf. Moore & Swartout, 1991). They considered this a reactive approach, which accepts feedback from the user and alerts its plans for explanation. This approach requires #exible explanation strategies with many varied plans for achieving given discourse goals. Another line of research involves enhancing explanations with user modelling and natural language generation. A great deal of work has been done by Paris to tailor the content, organization and phrasing of the text of explanation (cf. Paris, 1991, 1993). The main argument is that meaningful explanation should be provided based on the user's goal, level of domain knowledge, and the context, to provide informative, coherent, understandable and relevant explanation. A user model contains a variety of information about the user, including the user's domain knowledge, goals and plans, speci"c beliefs and preferences or interests. However, the success of this approach depends on the feasibility to build a complete and correct user model, and the tractability of utilizing such a model in a system's reasoning and explanation generation (cf. Moore & Swartout, 1991). This research focuses on contextualized access to deep explanations from KBS output. In the experimental KBS employed in this research, deep explanations are represented and linked to each other with hypertext-style links, and then linked to KBS output (intermediate results, recommendations and reasoning-trace explanations). The approach in this research is more consistent with the reactive approach advocated by Moore and Swartout (1991), although it does not involve sophisticated strategies to generate an explanation. Details of the experimental system are illustrated later in Section 4.

3. Theories and hypotheses The e!ects of contextualized access to knowledge on users' behaviour and performance are investigated in the context of a KBS. The independent variables are contextualized access to deep explanations and the users' level of domain knowledge (see Figure 1), since explanation use is in#uenced by the users' level of domain knowledge and by the explanation access method, which in turn a!ects users' understanding of KBS output and learning (Dhaliwal & Benbasat, 1996). The dependent variables are explanation use and judgement. Explanation use refers to the use of knowledge given in both deep and reasoning-trace explanations, which in#uence users' judgement. A method of measuring the e!ectiveness of a KBS and contextualized access to its underlying knowledge is by examining its impact on users' judgement. This study focuses on judgement congruence, which refers to the extent to which a user's judgement converges to the inferences made by those experts whose judgements were used to develop the KBS. It is expected that the more the KBS in#uences the user, the higher judgement congruence will be. In this section, we review the underlying theories for the linkages among the variables of interest, and generate the hypotheses to be tested. The &&e!ort-accuracy tradeo! ''

793

CONTEXTUALIZED ACCESS TO KNOWLEDGE

H3

Domain knowledge H2a

H2b

Use of deep explanations

H4a

Judgement congruence H4b H1a

Use of trace explanations H1b

Contextualized access FIGURE 1. The research model.

framework (Payne, Bettman & Johnson, 1993) will be used to predict the in#uence of contextualized access on explanation use. A comprehensive review of the literature on explanation use and alternative theoretical perspectives can be found in Dhaliwal and Benbasat (1996), and Gregor and Benbasat (1999).

3.1. EFFECTS OF CONTEXTUALIZED ACCESS ON EXPLANATION USE

One of the basic arguments made in this research is that having information available on-line does not mean users will access it. In the context of KBS, it is known from earlier research that not all available explanations are requested or used (Dhaliwal, 1993). In their review of the literature, Gregor and Benbasat (1999) identify several reasons for explanation use. Explanations are important to users in a number of circumstances, e.g. when a user perceives an anomaly in the advice provided, wants to learn more about the domain or about the reasoning approach of a KBS, or needs a speci"c piece of knowledge to participate properly in problem-solving. Will contextualized access lead to increased explanations use? To answer this question, we refer to the &&e!ort-accuracy tradeo! framework'' (Payne et al., 1993). According to this theory, an individual typically considers the cost of taking a particular course of action against the bene"ts that will accrue from taking that action. Empirical research has con"rmed that decision-makers attend more to e!ort reduction than to decision quality maximization (e.g. Russo & Dosher, 1983), i.e. one is motivated to pursue the strategy requiring the least e!ort and yet providing an acceptable solution (e.g. Beach & Mitchell, 1978). Gregor and Benbasat (1999) have extended the &&e!ort-accuracy tradeo! '' to the domain of explanation use. They argue both from a theoretical point of view, and by providing empirical evidence, that cost}bene"t considerations in#uence KBS explanation use.

794

J.-Y. MAO AND I. BENBASAT

If KBS output leaves domain concepts, causal relationships or procedures used unexplained, users have the choice of "lling in the missing knowledge either with their own or by requesting assistance, i.e. explanations from the KBS. The latter may require less cognitive e!ort, since explanations can alleviate the need for searching and retrieving knowledge from long-term memory. Moreover, users may lack some knowledge that is needed, but do not realize it. Contextualized access could alert users about their lack of knowledge, and provide convenient access to the knowledge in explanations and within the problem-solving context. According to the &&e!ort-accuracy tradeo! framework'', since contextualized access reduces the cost of obtaining explanations, it encourages increased use of explanations. Conversely, without contextualized access, i.e. if the knowledge is available but takes additional e!ort to "nd and access, users are far less likely to make e!ort to search for the needed knowledge. Hypothesis H1a follows directly the anticipated bene"ts of contextualized access to deep knowledge. H1a: Users with contextualized access will request more deep explanations than users without such access.

H1b is based on the assumption that contextualized access to deep explanations from within reasoning-trace explanations will make the latter more understandable and useful, thus leading to their higher use. The theoretical support for H1b is somewhat weaker, since in this study, contextualized access is associated only with deep explanations and all users have the same access to reasoning-trace explanations. H1b: Users with contextualized access will request more reasoning-trace explanations than users without such access.

3.2. EFFECTS OF DOMAIN KNOWLEDGE ON EXPLANATION USE AND JUDGEMENT CONGRUENCE

It is generally agreed that domain-related knowledge plays a crucial role in problemsolving and skilled-performance in many "elds. It takes a signi"cant amount of learning and practice to achieve a reasonable degree of pro"ciency (Anderson, 1983). A person who has gone to school and obtained book knowledge is considered a novice. Through experience with using the knowledge, one learns how it applies to both common and exceptional cases (Kolodner, 1983). The comprehension of a KBS recommendation involves establishing a causal relationship between the raw data and the conclusion, which is a knowledge-intensive problem-solving process. It can be argued that novices have a stronger need for deep explanations than those with practical experience (H2a) because novices' knowledge may not be complete. Deep explanations can provide much needed background domain knowledge. In contrast, experienced professions will likely have at least some specialized procedures and a much weaker need for the general background knowledge in deep explanations. Moreover, due to the lack of experience on the appropriate application of general domain knowledge, novices will request more reasoning-trace explanations to help understand what knowledge is applicable to the situation and how it is applied (H2b). H2: Novices will request more deep explanations (H2a) and more reasoning-trace explanations (H2b) than experienced professionals.

CONTEXTUALIZED ACCESS TO KNOWLEDGE

795

Prior research has also shown that experts tend to be less in#uenced by the provision of decision rules than those with only moderate knowledge in the task domain (Arkes, Dawes & Christensen, 1986). The use of KBS had a greater impact on the performance of low-skill employees than on high-skill employees in terms of increased judgement accuracy (Lamberti & Wallace, 1990). Experienced professionals are expected to be more likely to apply their own special problem-solving procedures and reach their own judgement in a highly autonomous manner. As a result, novices' will be more heavily in#uenced than the experienced professionals because novices are more likely to follow the line of reasoning and apply the same knowledge and procedures used by the KBS. Unlike contextualized access to knowledge, which is posited to in#uence judgement congruence indirectly through explanation use, the level of domain knowledge is expected to have a direct impact on judgement congruence as a result of using KBS. H3: Judgement congruence will be higher for novices than for experienced professionals. 3.3. EFFECTS OF EXPLANATION USE ON JUDGEMENT CONGRUENCE

Whereas contextualized access to deep explanations is expected to signi"cantly increase explanation use, the increase is meaningful and important if it enhances judgement congruence. To be in#uenced by a KBS, users must understand KBS recommendations, which in turn is largely determined by the comprehension of the underlying concepts and procedures embedded in KBS output. According to the level-of-processing view (e.g. Craik & Lockhart, 1972), the act of elaboration, which can result from explanation use, induces learners to process information more deeply, potentially enabling better comprehension of the key concepts and their role in the problem domain. Such elaboration can be particularly useful for novices to reduce the arbitrariness of domain concepts and their relationships. Greater elaboration about the meaning of new material increases both the number and strength of the links among the related information (Anderson, 1983). Empirical research in psychology indicates that explanations that provide a detailed reason for an action facilitate subsequent recall better than explanations providing some additional facts but not a detailed rationale for that action (e.g. Pressley, McDaniel, Turnure & Wood, 1987). For example, detailed procedural explanations of how to execute various computer program commands are better than abstract explanations about their purpose and functions (Reder, Charney & Morgan, 1986). Explanations that provide justi"cations for the relevance of new material are especially helpful. For example, novices wrote better computer programs after they saw examples of not only how an expert programmer solved similar problems, but also an explanation that justi"ed the speci"c design choices by the expert (Linn & Clancy, 1992). Similarly, explanations that provide a rationale for using general statistical inference techniques increase the tendency for them to be applied (Cheng & Holyoak, 1985). Therefore, there is reason to believe that explanation use may a!ect users' judgements. A key assumption of this research is that if users' judgement is congruent with the underlying reasoning of the KBS, their decision quality should be better than otherwise. Hypothesis H4a posits the direct e!ect of deep explanation use on judgement congruence. By virtue of the information they provide (see Tables 1 and 2), deep explanations help make the comprehension of KBS output deeper than it would be without such explanations. Similarly, Hypothesis H4b suggests that the use of reasoning-trace explanations

796

J.-Y. MAO AND I. BENBASAT

can facilitate the comprehension of the problem-solving process by KBS, by elaborating the causal links between the speci"c inputs to a reasoning process and its conclusions. H4: The use of deep explanations (H4a) and reasoning-trace explanations (H4b) will be positively related to judgement congruence.-

4. Research method A laboratory experiment was conducted to test the hypotheses. The experimental task involved a realistic "nancial analysis. Subjects were asked to assume the role of a corporate loan evaluation o$cer of a large "nancial institution, and to evaluate an application for a major commercial loan by a hypothetical "rm. Subjects were told to use a KBS designed for loan evaluation to assess various aspects of the company's "nancial health. Then, based on the assessment, they would make a recommendation as to whether the loan should be approved, and if so, in what amount. Whereas commercial loan evaluation is a very complex process, and goes much beyond ratio analysis, the experimental systems and tasks were focused on ratio analysis only, which is an important part of loan evaluation and other common tasks in "nance and accounting. The reason that loan evaluation was introduced to frame the task was merely to make the task more interesting and engaging. 4.1. DEVELOPMENT AND VALIDATION OF THE EXPERIMENTAL SYSTEMS

This research used a simulated KBS developed for a prior research (Dhaliwal, 1993), due to the lack of access to a commercial or proprietary KBS for "nancial analysis. A panel of six senior "nancial analysts, whose experiences ranged from 12 to 23 years, was recruited for knowledge acquisition. They were given the same commercial loan evaluation case to analyse as the one used in this research. Concurrent verbal protocols were collected to determine the types of analysis that they performed along with their detailed reasoning processes and explanations. The results, which consisted of the assessment of six aspects of the "nancial health of the hypothetical "rm based on "nancial ratio analysis, became the basis of the simulated KBS, FINALYZER. The system, including all explanations, has gone through multiple rounds of validation involving accounting and "nancial analysis professionals. It was considered highly realistic and useful, and the level of expertise displayed by the system was rated highly. By enhancing the access to deep explanations with hypertext-style links, a new version named Hyper-FINALYZER was also built. The experts whose knowledge was used to develop the KBS were also asked to provide a &&solution'', consisting of six judgmental questions involved in the experimental task. The "nal set of scores agreed upon by the panel, after a two-round Delphi process, was taken as a benchmark of the &&correct'' judgements for the case (Dhaliwal, 1993). - This direct e!ect is strengthened by the indirect e!ect via the combination of Hypotheses H2 and H4, which predict that novices will use more explanations than experts (H2) and that explanation use will lead to improvement in judgement congruence (H4).

797

CONTEXTUALIZED ACCESS TO KNOWLEDGE

4.2. EXPERIMENTAL SYSTEMS AND EXPLANATION ACCESS METHODS

FINALYZER performs "nancial analysis in terms of several subanalyses such as liquidity, capital structure and pro"tability. It displays three types of screens for each of the subanalyses: (1) an information screen containing an index of domain concepts ("nancial ratios and procedures), for which deep explanations can be requested (Figure A1), (2) a data screen of relevant "nancial ratios calculated from the "nancial statements of the "rm to be evaluated (Figure A2), and (3) recommendation screens presenting results of the &&evaluation'' of the "nancial statements and ratios, in the form of recommendations (Figure A3). Reasoning-trace explanations can also be requested for each of the recommendations. The sequence of the screens is shown in Figure 2, following the normal procedure of "nancial analysis, i.e. calculating "nancial ratios "rst and then yielding judgements for making decisions and predictions. In FINALYZER, users can request deep explanations only from the information screen. By default (following the input-process-output sequence), deep explanations (Figures A4 and A5) are accessible prior to the presentation of data and recommendations, although it is possible to navigate back later to obtain deep explanations, through the &&previous-screen'' and &&next-screen'' buttons in a linear manner. Therefore, access to task domain knowledge is in abstract (as labelled in Figure 2) or out of context, because it is not integrated into the speci"c KBS output, i.e. data, recommendations and their reasoning-trace explanations (Figure A6). Domain knowledge accessed from the information screen appears general to the KBS. In contrast, Hyper-FINALYZER allows contextualized access to deep explanations via hypertext-style links (note that the hypertext-style link buttons in Figures A2 }A6 are not available in FINALYZER), e.g. in the context of problem-solving (identi"ed as Problem-Context in Figure 3). In other words, deep explanations are seamlessly integrated into KBS output, and can be requested from three di!erent contexts: (1) abstract refers to the requests made directly through the information screens (Figure A1) only, as in FINALYZER, separated from the speci"c context of problem-solving; (2) knowledgecontext refers to the requests via hypertext-style links from within other related deep explanations initially invoked from information screens (Figures A4 and A5); (3) problem-context refers to the requests via hypertext-style links originated from data (Figure A2), recommendations (Figure A3) and reasoning-trace explanations (Figure A6), i.e. within the speci"c context of problem-solving. In both (2) and (3), access is immediate,

Select a subanalysis

Information screen

Data screen

Recommendation screen

Abstract

Deep explanations

FIGURE 2. Flow chart of FINALYZER.

Reasoning-trace explanations

798

J.-Y. MAO AND I. BENBASAT

Select a subanalysis

Information screen

Data screen

Recommendation screen

Abstract Problem-context Knowledgecontext

Deep explanations

Reasoning-trace explanations

FIGURE 3. Hyper-FINALYZER with contextualized access to deep explanations.

and within the context of other KBS output, i.e. advice, reasoning-trace and deep explanations. They are the treatment and the only di!erence between the two systems. In summary, the information contents are identical in the two versions of the experimental KBS. The only di!erence resides in the contextualized accessibility of deep knowledge enabled by the extra hypertext-style links in Hyper-FINALYZER. Such contextualized access is considered more e!ective than the conventional on-line help or context-sensitive help as discussed in Section 2.1.

4.3. DOMAIN KNOWLEDGE AND SUBJECTS

Subjects with two di!erent levels of domain knowledge, namely novices and experienced professionals, participated in this study. Novices may have considerable knowledge but lack experience in a given area, and they are di!erent from laypersons who have little if any skill in problem-solving (Camerer & Johnson, 1991). Our novice subjects included 20 undergraduate students specializing in accounting, and nine MBA students who either had taken accounting courses extensively or had an undergraduate degree in accounting. None of them belonged to any professional associations, and only four of them had limited "nancial analysis work experience. They were &&educated novices'', similar to entry-level employees for "nancial analysis positions. An expert is a person who is experienced in a particular domain and has some professional or social credentials (Camerer & Johnson, 1991). However, it is di$cult to recruit a large number of expert subjects similar to those who helped develop the experimental KBS. Therefore, this study targeted experienced professionals with professional quali"cations such as Certi"ed General Accountants (CGA) and Certi"ed Financial Analysts (CFA) who had at least 3 years of post-qualifying work experience directly related to "nancial analysis. Twenty-six quali"ed individuals participated including 14 CGAs and 12 CFAs. On average, they had 9.6 years of "nancial-analysis-related work experience. They were not necessarily specialized in major commercial loan evaluation tasks such as the one used in this study, which occur relatively infrequently and at a high organizational level. Nonetheless, the experimental task was clearly job-related, because it was primarily the "nancial ratio analysis part of loan evaluation, for which the experienced professionals had the required general background and

799

CONTEXTUALIZED ACCESS TO KNOWLEDGE

experience. Moreover, these professionals were likely to possess local expertise (Paris, 1987), i.e. they have substantial knowledge and experience in some sub-area of the task. 4.4. EXPERIMENTAL PROCEDURES

The experiment was administered individually to each subject randomly assigned to a treatment condition as shown in Table 2. A $50 prize was promised to the top 20% of individuals in each experimental condition, based on the quality of their judgement. Each subject completed a tutorial on the use of a sample KBS that was similar in structure and general functionality to the experimental one, but designed for evaluating consumer credit applications. This tutorial was intended to ensure that subjects were comfortable using the experimental KBS, and to eliminate potential novelty e!ects. Subjects were then given the commercial loan evaluation case to analyse manually and make judgements. This manual analysis was useful not only for the subjects to become familiar with the details of the task and data, but also for controlling the e!ect of individual di!erences in the measurement of judgement congruence. Next, subjects were given the experimental KBS, to help them re-analyse the same case and make their judgements again. Subjects were allowed as much time as they needed. The total time for the experiment typically ranged from one and a half to 2 h, with the average being slightly less than 2 h. 4.5. MEASUREMENT OF DEPENDENT VARIABLES

Judgement congruence is a performance-based measure. It is composed of six speci"c judgmental questions, each of which deals with one aspect of the "nancial health of the hypothetical "rm assessed by the panel of six experts, in terms of current liquidity, capital structure, asset utilization, market valuation, "nancial management and operating management. For example, the question related to current liquidity was Based on your analysis and under current economic and interest-rate conditions, rate Canacom's current liquidity position. Please circle the correct answer. Very Weak Position: 1!2!3!4!5!6!7!8!9!10: very Strong Position

Answers given to these six judgmental questions before using the KBS are labelled Pre-KBS scores; and after using the KBS, Post-KBS scores. It is important to note that the KBS provided only some basic assessment of strengths and weaknesses of the company described in the commercial loan evaluation case along TABLE 2 The experimental design Access to deep explanations Non-contextualized (FINALYZER) Task domain knowledge

Novices Experienced

14 13

Contextualized (Hyper-FINALYZER) 15 13

800

J.-Y. MAO AND I. BENBASAT

Knowledge-source experts

Experts’ judgement (canacom case)

Knowledge

KBS (FINALYZER)

Judgement congruence

Advice and explanations

KBS users

User’s judgement (canacom case)

FIGURE 4. Operationalization of judgement congruence.

with explanations for its assessment, without direct answers to the six judgemental questions posed to the users. As described earlier in Section 4.1, the panel of experts upon whose evaluations the KBS was based provided a benchmark &&solution'' to these six questions. Absolute deviations from the benchmark were calculated for the Pre- and Post-KBS scores, leading to a measure: Judgement congruence"" Pre-KBS !Benchmark"

scores!Benchmark"!"Post-KBS

scores

A positive number indicates that after using the KBS, subjects' judgements were brought closer to the underlying expert judgements of the KBS. Such convergence to expert judgements could mean that the knowledge of the experts was transferred to the subjects through KBS recommendations and explanations (see Figure 4). Computer logs captured detailed data on each request for deep and reasoning-trace explanations. For instance, in the contextualized access condition, the number of requests was recorded separately for each of the di!erent contexts where deep explanations could be accessed (see Figure 3), because their contribution to judgement congruence might vary.

5. Data analysis 5.1. ANTECEDENTS AND OUTCOMES OF EXPLANATION USE

The research model was tested using a partial least squares (PLS) analysis with PLSGraph (Chin & Frye, 1996). PLS is a multivariate analysis technique ideal for testing structural models with latent variables (cf. Barclay, Higgins & Thompson, 1995). It

801

CONTEXTUALIZED ACCESS TO KNOWLEDGE

– 0.26

Domain knowledge – 0.18

Use of deep explanations

0.28*

0.36*

2

R = 0.42 Judgement congruence 0.02 0.62**

2

R = 0.23

Use of trace explanations 2

R = 0.08 Contextualized access

0.06 ** p < 0.01

* p < 0.05

FIGURE 5. Estimated model of explanation use and its consequences.

simultaneously assesses the reliability and validity of the measures of theoretical constructs, and estimates the relationship among theoretical constructs. Thus, the overall predictive power (in a regression sense) of a research model can be examined, while minimizing the measurement error involved. PLS is more appropriate than LISREL in this case because this research is mainly exploratory in nature, rather than con"rming a particular model based on strong theory, and it involves formative constructs that cannot be adequately modelled using covariance structure analysis (Chin, 1998). Another appealing feature of PLS is its ability to work with small sample sizes, which made this analysis possible. The three endogenous constructs, judgement congruence, use of deep and of reasoningtrace explanations, are considered being &&formed'' as a weighted linear combination of observed variables. Such a formative relationship implies that the construct is expressed as a function of the observable variables, which &&form'', cause or precede the construct. Judgement congruence is formed with the six judgmental items (see Section 4.5) as surrogate measures, because it was speci"c to the experimental task rather than derived from a general theory. The Use of Deep Explanations measure consists of the number of requests in the three di!erent contexts (abstract, knowledge-context, problem-context), whereas the use of reasoning-trace explanations is measured based on the number of requests of the three more speci"c types, i.e. the How, Why and Strategic explanations. The results of the structural modelling are shown in Figure 5, including path coe$cients representing the hypothesized causal relationships between the variables, and multiple Rs. The path coe$cients can be interpreted as standardized coe$cients ('s) in multiple regression analysis. Their signi"cance was tested using t-tests based on the built-in Jackkni"ng function in PLS-Graph. A multiple R is interpreted, as in multiple regression, to indicate the percentage of the variance in the respective latent construct that is explained by the exogenous constructs.

802

J.-Y. MAO AND I. BENBASAT

TABLE 3 Summary of results Hypothesis

Std. path coe!. (direct e!ect)A

H1a: AccessPdeep explanation use 0.62** H1b: AccessPtrace explanation use 0.06 H2a: ExpertisePdeep explanation use !0.18 H2b: expertisePtrace explanation use 0.28 H3: expertisePjudgement congruence !0.26 H4a: deep explanation useP Judgement congruence 0.36* H4b: trace explanation useP Judgement congruence 0.02

Total e!ect-

0.62 0.06 !0.18 0.28 !0.32 0.36 0.02

Variance accounted for by the direct e!ectB

Total Results variance? accounted for by the model (R)

38% 0.5% 3.5% 8% 8.5%

42% 8% 42% 8% 23%

15%

23%

Supported

23%

N.S.

0.3%

Supported

N.S# N.S.

N.S.-N.S.

-Total e!ect"direct e!ect#indirect e!ect. ? The total variance in the endogenous construct accounted for by the PLS model, i.e. multiple Rs. A**p(0.01, *p(0.05. B Variance accounted for by direct e!ect"path coe$cient;correlation. # N.S. stands for &¬ supported''. -- The positive path coe$cient is in fact opposite to the predicated direction, not supporting H2b.

The amount of variance accounted for by each of the exogenous constructs is shown in Table 3. While path coe$cients only represent the direct e!ect of each of the antecedent constructs in the model, it is important to consider the total e!ects (sum of the direct and indirect e!ects) which are the overall indicators of the relative importance of antecedent constructs (Barclay et al., 1995). For example, Expertise has a direct e!ect on judgement congruence (!0.26), and the two indirect e!ects via the use of deep and reasoning-trace explanations being the product of the two path coe$cients on each path (!0.18;0.36 and 0.28;0.02, respectively). Table 4 presents the means of the dependent measures to put the results in Table 3 in perspective. For example, the judgement of novices who had contextualized access to deep explanations was much more congruent with the expert panel's benchmark, improved by 19% after using KBS (see the de"nition of judgement congruence in Section 4.5). Computer logs showed that on average these subjects requested 17 deep explanations out of 113 unique ones, and 16 reasoning-trace explanations out of 57 unique ones. The number of explanation requests in Table 4 are higher than the number of unique explanations accessed, as some of the explanations were requested more than once. It is interesting to note that without contextualized access, there was essentially no repeated access to deep explanations. Only two subjects had one repeated access. Moreover, once they started reading KBS recommendations and reasoning-trace explanations, only four out of the 25 subjects backtracked to the Index screen to access deep explanations. This result combined with the relatively high number of explanation access suggests that the deep explanations were understood with one reading and considered useful.

803

CONTEXTUALIZED ACCESS TO KNOWLEDGE

TABLE 4 Means of dependent measuresWithout contextualized access Novices

Experienced professionals

With contextualized access

Deep explanation requests Reasoning-trace explanation requests Judgement congruence (improvement after KBS use)

6.6 18.1 0.54 (6%) (n"13)

17.3 16.3 2.13 (19%) (n"15)

Deep explanation requests Reasoning-trace explanation requests Judgement congruence (improvement after KBS use)

2.4 16.7 !0.08 (!1%) (n"12)

9.9 15.6 0.85 (9%) (n"13)

- One subject's data was excluded from all analyses because he experienced some technical di$culties in accessing explanations, and "nished the experiment without using any at all. A second subject's data was also excluded because the case was deemed to be too extreme from &&normal'' use. She requested a total of 63 deep explanations, resulting in a z-score of 4.33, much beyond the p"0.001 criterion of 3.67 (two-tailed) (Tabachnick & Fidell, 1989).

TABLE 5 Repeated access to deep explanations

Novices Experienced professionals Overall

Total no. of access

Total no. of unique access

Explanations accessed once

Explanations accessed twice

Explanations accessed more than twice

17.33 9.85

14.80 8.69

12.80 7.69

1.60 0.92

0.40 0.08

13.86

11.96

10.43

1.29

0.25

Given the contextualized access, novice subjects on average accessed two deep explanations more than once, and professionals had one repeated access, as shown in Table 5. Furthermore, computer logs also show that the 28 subjects only gave nine explanations repeated access prior to reading KBS recommendations and reasoningtrace explanations, whereas all of the remaining repeated access occurred afterwards, i.e. in the context of problem-solving. It appears as though contextualized access allowed subjects to obtain useful knowledge at di!erent stages of the task, which sometimes resulted in repeated access to what they had already seen previously.

5.2. DETERMINANTS OF EXPLANATION USE

Contextualized access signi"cantly increased deep explanation use (13.9 vs. 4.6 for non-contextualized access). It accounts for 38% out of 42% of the variance in deep explanation use captured by the model (see Table 4). Therefore, H1a is supported.

804

J.-Y. MAO AND I. BENBASAT

Domain knowledge accounts for the remaining 3.5% of the variance. Novices used more deep explanations than experienced professionals (12.3 vs. 6.3). However, this di!erence is not statistically signi"cant at the 0.05 level, although the p-value is smaller than 0.10. Therefore, H2a is not strongly supported. The model accounts for only 8% of the variance in the use of reasoning-trace explanations. Contextualized access to deep explanations did not in#uence the use of reasoning-trace explanations, thus lending no support to H1b. Contrary to the direction predicted by H2b there appears to be a positive relationship between the level of domain knowledge and the number of reasoning-trace explanation requests. Whereas the numbers in Table 4 do not indicate a di!erence between experienced professionals and novices in terms of reasoning-trace explanation use, we took a closer look at PLS data to identify how the various sub-components contributed to the formation of the reasoningtrace explanation requests. First, we observed that the How explanations were most in#uential (i.e. had a larger weight) in the formation of the Use of Reasoning-Trace Explanations construct. Second, computer logs on explanation requests show that experienced professionals used more How explanations than novices (55 vs. 40% of the total use of reasoning-trace explanations). These two facts explain the positive relationship between the level of domain knowledge and the number of reasoning-trace explanation requests. 5.3. DETERMINANTS OF JUDGEMENT CONGRUENCE

The direct e!ect of users' domain knowledge accounts for 8.5% of the variance in judgement congruence. As hypothesized, experienced professionals had a lower level of judgement congruence than novices (see Table 4). However, this di!erence is not statistically signi"cant at the 0.05 level, although the p-value is smaller than 0.10. H3 is not strongly supported. The use of deep explanations was far more e!ective than reasoning-trace explanations for judgement congruence. There is a signi"cant positive relationship between the use of deep explanations and judgement congruence. Deep explanation use accounts for the remaining 15% of the variance in judgement congruence out of a total of 23% (Table 4), while reasoning-trace explanation use has an insigni"cant contribution in explaining the variance. This result is consistent with the judgement congruence scores in Table 4 that after KBS use novices improved their scores by 19 and 6%, with and without contextualized access; the corresponding scores for experienced professionals are 9 and !1%, respectively. Thus, H4a is supported, but H4b is not. The lack of support for H4b may be attributed to two factors: "rst, understanding the meaning of KBS output (enhanced by deep explanations) seems to be more important than requesting reasoning-trace explanations without a good understanding of their implications. Second, there was little variance in the use of reasoning-trace explanations (Table 4). Overall, contextualized access to deep explanations was positively associated with judgement congruence, due to the indirect e!ect of increasing the use of deep explanations, which in turn contributed to the enhanced judgement congruence. 5.4. CONTEXT OF DEEP EXPLANATION USE

A post hoc analysis was performed on users' preferred context of deep explanation use. Each cell in Table 6 represents the total number (frequency) of explanations requested in

805

CONTEXTUALIZED ACCESS TO KNOWLEDGE

TABLE 6 E+ect of domain knowledge on the context of deep explanation use Context of deep explanation requests No. of requests Domain knowledge

Novices (n"15) Experienced professionals (n"13) Total (n"28)

Abstract

Knowledgecontext

Problemcontext

129 (85) 49.6% 61 (29) 47.7%

44 16.9% 12 9.4%

87 33.5% 55 43.0%

190 (114) 49.0%

56 14.4%

142 36.6%

Total 260 67.0% 128 33.0% 388 100%

a particular context (see Figure 3) by all of the novices or experienced professionals with contextualized access to deep explanations. Since the 13 novices and 12 experienced professionals without contextualized access could request explanations in abstract only, their numbers are presented in parentheses. Overall, about half of the explanation requests occurred in abstract through information screens. The Pearson's chi-square for Table 6 is not statistically signi"cant (p"0.06), which tests the di!erence between the way novices and experienced professionals used contextualized access. The signi"cance of the chi-square test is apparently weakened by the similarity between experienced professionals and novices in the proportion of deep explanations requested in abstract. If explanation use in abstract is excluded from the analysis, the di!erence between novices and experienced professionals would be highly signi"cant (p"0.02). This result indicates that the preferred context of explanation use was associated with the level of domain knowledge. Experienced professionals made few explanation requests but a higher percentage of the requests were in the context of problem-solving, whereas novices were more likely to follow hypertext links to explore domain concepts in the context of deep knowledge. Interestingly, the overall proportion of contextualized requests for deep explanations (via both the knowledge- and problem-contexts) was about the same for both novices and experienced professionals, indicating that they were equally willing to take advantage of contextualized access. In fact, contextualized access also led to more deep explanation requests in abstract through the information screens, with an increase from 4.6 to 6.8. The e!ect is slightly bigger for the experienced professionals (chi-square"11.1, p(0.001). This increase is presumably due to subjects' awareness that deep explanations on other related domain concepts and relationships can be accessed through hypertext links, i.e. exploring the knowledge-context, as illustrated in Figure 3. Therefore, explanations accessed from the information screens also become more useful, which in#uenced the cognitive e!ort}bene"t tradeo!. The fact that a major portion of the use of deep explanations remained in abstract indicates that contextualized access enhanced knowledge access, but did not completely eliminate the need to have access to domain knowledge prior to

806

J.-Y. MAO AND I. BENBASAT

the task, which is recognized as cognitive input to task in the literature (Dhaliwal & Benbasat, 1996).

6. Discussion and conclusions 6.1. DISCUSSION OF THE RESULTS

The results show that contextualized access to deep knowledge has several bene"ts. It increased the average number of deep explanation requests by a factor of three, from 4.6 to 13.9, and enabled about 37% of deep explanation requests to occur in the problemsolving context rather than in the abstract. More importantly, the increased utilization of deep explanations and contextualized use were associated with a higher degree of judgement congruence with the source experts whose knowledge was represented in the KBS. Both novices and experienced professionals appeared to have taken advantage of contextualized access. These results are signi"cant from a theoretical perspective, as they seem to suggest that contextualized access to knowledge is a highly e!ective method to reduce the &&cost'' of learning and in#uence the e!ort-accuracy tradeo! involved in accessing domain knowledge. Contrary to our expectation, the amount of domain expertise made little di!erence in judgement congruence and explanation use. There are several plausible factors that might have led to the lack of di!erence between novices and experienced professionals. For example, it could be a result of training e!ect in our experimental design. Subjects learned quickly that explanations could help and relied upon them, and this e!ect could be so strong that it obliterated any di!erence due to prior experience. In particular, the consistently high usage of reasoning-trace explanation by all subject groups seems to support this argument. A control condition without any explanation could have helped con"rm this. It could also be the case that the way judgement congruence was operationalized (Figure 4) might not be speci"c enough to discern any performance di!erence between novices and experienced professionals. Moreover, since the sample size is relatively small, constrained by the available experienced processionals willing to participate, it is likely that the statistical tests are not powerful enough to detect potentially signi"cant di!erences between the two groups. Users did not use more reasoning-trace explanations, despite the enhanced accessibility of deep explanations from within the context of reasoning-trace explanations. On the other hand, increased deep explanation requests did not o!set the use of reasoning-trace explanations, despite users' propensity to reduce e!ort as predicted by the cognitive e!ort perspective. This result, along with the relatively high use of reasoning-trace explanations, suggests that users considered reasoning-trace explanations useful, with or without enhanced access to deep explanation. It is likely that di!erent factors in#uenced deep and reasoning-trace explanation request. It is interesting to note that experienced professionals used the How type of reasoning-trace explanation more than the Why type (55 vs. 40%). This di!erence might be due to the functional di!erences between the two types of reasoning-trace: the How type focuses on subordinate goals and causal antecedents, facilitating the understanding of KBS conclusions in terms of previously established conclusions and facts, whereas the Why type reveals the superordinate, i.e. the goal hierarchy KBS reasoning. It is likely

CONTEXTUALIZED ACCESS TO KNOWLEDGE

807

that experienced professionals had less need for the goal hierarchy because it is relatively easier for them to infer, thus their use of explanation was mostly for clarifying or con"rming subordinate goals and facts.

6.2. LIMITATIONS AND FUTURE STUDIES

Several limitations of this research should be kept in mind when considering the implications of the results. First, the "ndings are based on the initial use of the experimental KBS. There could also have been potential novelty e!ects due to the use of hypertext-style links for knowledge representation and contextualized access, which was new to the subjects. Despite the training provided to minimize novelty e!ects, subjects' behaviour might still be di!erent from &&natural'' behaviour that had evolved over long periods of exposure to the technology. This limitation was due to the fact that we did not have a functional KBS with the necessary features for a longitudinal study. Second, whereas contextualized access to deep knowledge was associated with a higher degree of judgement congruence, it is not certain if users actually learned more from the KBS, because explanation access is a proxy for information use. For example, contextualized access might have made the KBS appear more convincing, causing some users, novices in particular, to &&go-along''. Novices might be more willing to accept the advice of the KBS because of its perceived credibility. KBS use has in#uenced the judgement of novices, but it is not necessarily the case that learning has taken place. Since our main performance measure, judgement congruence, does not reveal the cognitive e!ect of the explanation, more direct measures would be needed in future studies, e.g. memory recall or post-task interview. These direct measures will help us "nd out if contextualized access to knowledge enables better comprehension of key concepts, and what knowledge is actually used in the task, and how. Third, all of the subjects analysed the case manually "rst and then used the KBS, whereas normally the groups should have been split in half to counterbalance the sequence. This is a #aw in the experimental design, although the initial manual analysis was deemed necessary to measure the treatment e!ect on judgement. It has likely weakened our results because the manual analysis could lead to "rm judgement, thus subjects would be in#uenced by the subsequent KBS use and contextualized access to knowledge to a lesser degree than otherwise. Finally, in this study, contextualized access to domain knowledge was implemented based on hypertext-style links from keywords appearing in KBS output. It could be more bene"cial to use more sophisticated strategies, e.g. by actually anticipating the kind of knowledge needed depending on the type of tasks and user experience, and then prioritizing the &&nearness'' or accessibility of relevant knowledge. This line of research could bene"t from prior research in user modelling and explanation in context reviewed in Section 2. Another direction of exploration is for the KBS to anticipate the knowledge need and provide relevant knowledge proactively rather than relying upon the users' queries.

6.3. CONCLUSIONS

The most important contribution of this research is the conceptualization of contextualized access to task domain knowledge, which highlights the distinction between

808

J.-Y. MAO AND I. BENBASAT

availability and contextualized accessibility. Our results accentuate the importance of convenience access to the underlying domain knowledge for understanding the meaning of the output of intelligent systems: access to problem-solving rules (i.e. reasoning-trace) may not be su$cient for in#uencing user judgement; instead, understanding the meaning of the rules is the key. The use of hypertext can be an e!ective strategy for integrating the underlying problem-solving knowledge into the output of intelligent systems.

This research is supported by a research grant from the Natural Sciences and Engineering Research Council of Canada (NSERC). The authors wish to thank Professor A.G. Sutcli!e and two anonymous reviewers for their thorough and constructive comments. Shirley Gregor and Fiona F.H. Nah have read earlier versions of this paper and o!ered helpful comments.

References AGAH, A. & TANIE, K. (2000). Intelligent graphical user interface design utilising multiple fuzzy agent. Interacting with Computers, 12, 529}542. ANDERSON, J. R. (1983). A spreading activation theory of memory. Journal of
CONTEXTUALIZED ACCESS TO KNOWLEDGE

809

DESMARAIS, M., LAROCHELLE, S. & GIROUX, L. (1987). The diagnosis of user strategies. In H. BULLINGER & B. SHACKEL, Eds. Human-Computer Interaction2IN¹ERAC¹+87. Amsterdam: Elsevier. DESMARAIS, M. C., LECLAIR, R., FISET, J. Y. & TALBI, H. (1997). Cost-justifying electronic performance support systems. Communications of the ACM, 40, 39}48. DHALIWAL, J. S. (1993). An experimental investigation of the use of explanations provided by knowledge-based systems. Unpublished Doctoral Dissertation, University of British Columbia. DHALIWAL, J. S. & BENBASAT, I. (1996). The use and e!ects of knowledge-based system explanations: theoretical foundations and a framework for empirical evaluation. Information Systems Research, 7, 342}362. FALK, R. F. & MILLER, N. 1992. A Primer for Soft Modelling. Akron, OH: The University of Akron Press. FEIGENBAUM, E. A. & MCCORDUCK, P. (1983). ¹he Fifth Generation. Reading, MA: AddisonWesley Publishing Company. FEIGENBAUM, E. A., MCCORDUCK, P. & NII, H. P. (1988). ¹he Rise of Expert Company. New York: Times Books. FIKES, R. & KEHLER, T. (1985). The role of frame-based representation in reasoning. Communications of the ACM, 28, 904}920. FININ, T. W. (1983). Providing help and advice in task oriented systems. Proceedings of International Joint Conference on Arti,cial Intelligence, pp. 288}294. Karlsruhe, Germany. FISCHER, G., LEMKE, A. C. & MCCALL, R. (1990). Towards a system architecture supporting contextualised learning. Proceedings of 8th National Conference on Arti,cial Intelligence, Vol. 1, 420}425. Cambridge, MA: AAAI Press/The MIT Press. FISHER, G., LEMKE, A. & SCHWAB, T. (1985). Knowledge-based help systems. Proceedings of the CHI+85 Human Factors in Computing Systems. N>: ACM. FURMAN, A. & SPYRIDAKIS, J. (1992). The e!ect of system-initiated advice on the use of menu navigation shortcuts, IEEE ¹ransactions on Professional Communication, 35, 112}119. GREGOR, S. & BENBASAT, I. (1999). Explanations from intelligent systems: a review of theoretical foundations and empirical work. MIS Quarterly, 23, 497}530. HEAD, A. J. (1998). Are Microsoft's animated interface agents helpful? Online, 22, 19}28. HERTZUM, M. & FROKJAER, E. (1996). Browsing and querying in online documentation: a study of user interface and the interaction process. ACM ¹ransactions on Computer-Human Interaction, 3, 136}161. HOOK, K. (2000). Steps to take before intelligent user interfaces become real, Interacting with Computers, 12, 409}426. HORVITZ, E. (1999). Principles of mixed-initiative user interfaces. CHI +99 Conference Proceedings: ¹he CHI is the ¸imit. N>: ACM. JONASSEN, D. H. (1990). Semantic network elicitation: tools for structuring hypertext. In: R. MCALEESE & C. GREEN, Eds. Hypertext: ¹he State of the Art. London: Intellect. KEARSLEY, G. (1988). Online Help Systems: Design and Implementation. Norwood, NJ: Ablex Publishing KOLODNER, J. L. (1983). Towards an understanding of the role of experience in the evolution from novice to expert. International Journal of Man-Machine Studies, 19, 497}518. LAMBERTI, D. M. & WALLACE, W. A. (1990). Intelligent interface design: an empirical assessment of knowledge presentation in expert systems. MIS Quarterly, 14, 279}311. LINN, M. C. & CLANCY, M. J. (1992). Can experts' explanations help students develop program design skills. International Journal of Man-Machine Studies, 28, 511}551. MARSHALL, C. C. & SHIPMAN, III, F. M. (1995). Spatial hypertext: designing for change. Communications of the ACM, 38, 88}97. MITTAL, V. O. & PARIS, C. L. (1995). Generating explanation in context: the system perspective. Expert Systems with Applications, 8, 491}503. MOORE, J. D. & SWARTOUT, W. R. (1991). A reactive approach to explanation: taking the user's feedback into account. In: C. L. PARIS, W. R. SWARTOUT, R. & W.C. MANN Eds. Natural ¸anguage Generation in Arti,cial Intelligence and Computational ¸inguistics, pp. 3}48. Boston: Kluwer Academic Publishers. NICKERSON, R. (1999) Why interactive computer systems are sometimes not used by people who might bene"t from them? International Journal of Human-Computer Studies, 51, 307}321.

810

J.-Y. MAO AND I. BENBASAT

PARIS, C. L. (1987). Combining discourse strategies to generate descriptions to users along a naive/expert spectrum. Proceedings of the 10th International Joint Conference on Arti,cial Intelligence, Vol. 2, pp. 626}632. Milan, Italy. Los Altos; CA: Morgan Kaufmann. PARIS, C. L. (1991). Generation and explanation: building an explanation facility for the explainable expert systems framework. In: C. L. PARIS, W. R. SWARTOUT & MANN, W. C. Eds. Natural ¸anguage Generation in Arti,cial Intelligence and Computational ¸inguistics, pp. 49}82. Boston: Kluwer Academic Publishers. PARIS, C. L. (1993). ;ser Modelling in ¹ext Generation. London, UK: Printer Publishers. PAYNE, J. W., BETTMAN, J. R. & JOHNSON, E. J. (1993). ¹he Adaptive Decision Maker. Cambridge: Cambridge University Press. PRESSLEY, M., MCDANIEL, M. A., TURNURE, J. E. & WOOD, E. (1987). Generation and precision of elaboration: e!ects on intentional and incidental learning. Journal of Experimental Psychology: ¸earning, Memory, and Condition, 13, 291}300. REDER, L. M., CHARNEY, D. H. & MORGAN, K. I. (1986). The role of elaborations in learning a skill from an instructional text. Memory and Cognition, 14, 64}78. RUSSO, J. & DOSHER, B. (1983). Strategies for multiattribute binary choice. Journal of Experimental Psychology: ¸earning, Memory and Cognition, 9, 676}696. SCHWARTZ, D. G. & TE'ENI, D. (2000). Tying knowledge to action with kMail. IEEE Intelligent Systems and ¹heir Applications, 15, 33}39. SHNEIDERMAN, B. (1998). Designing the ;ser Interface: Strategies for E+ective Human-Computer Interaction (3rd edn). Reading, MA: Addison-Wesley. SHORTLIFFE, E. H. (1976). Computer-Based Medical Consultations: M>CIN. NY: Elsevier/NorthHolland Inc. SOUTHWICK, R. W. (1991). Explaining reasoning: an overview of explanation in knowledge-based systems. Knowledge Engineering Review, 6, 1}19. SWARTOUT, W. R. (1983). XPLAIN: a system for creating and explaining expert consulting programs. Arti,cial Intelligence, 21, 285}325. SWARTOUT, W. R. (1987). Explanation. In S. C. SHAPIRIO & D. ECKROTH, Eds. Encyclopedia of Arti,cial Intelligence, Vol. 1 pp. 298}300. New York: John Wiley & Sons. SWARTOUT, W. R. & SMOLIAR, S. (1987). On making expert systems more like experts. Expert Systems, 4, 196}207. SWARTOUT, W. R, PARIS, C. L., & MOORE, J. D. (1991). Design for explainable expert systems. IEEE Expert, 6, 58}64. TABACHNICK, B. G. & FIDELL, L. S. (1989). ;sing Multivariate Statistics (2nd edn). NY: Harper & Row Publishers. WICK, M. R. & THOMPSON, W. B. (1992). Reconstructive expert system explanation. Arti,cial Intelligence, 54, 33}70. Wolfe, B., & Eichmann, D. (1997). A neural network approach to tracking eye position. International Journal of Human-Computer Interaction, 9, 59}79. Paper accepted for publication by Associate Editor, Dr A. Sutcli!e

Appendix A: Illustrations of the experimental systems Some screen-shots from FINALYZER and Hyper-FINALYZER (Figure A1) are provided herein to illustrate their basic features and di!erences. Due to the space limitation, not all of the di!erent types of explanations are shown. Deep explanations on each domain concept and procedure to be used in a subanalysis can be accessed from this screen. Users need to click on a ratio-button (small circle) "rst to select a particular concept or procedure "rst, and then press on one of the push-buttons (rectangles labelled with WHY, HOW, and STRATEGIC) corresponding to the three types of deep explanations. Figures A2}A6 show other screens in Hyper-FINALYZER. Their counterparts in FINALYZER screens are identical to them, except that the embedded radio buttons are not available, thus contextualized access to knowledge is not possible.

CONTEXTUALIZED ACCESS TO KNOWLEDGE

FIGURE A1. Example of information screens in both FINALYZER and Hyper-FINALYZER.

FIGURE A2. Example of data screen in Hyper-FINALYZER.

811

812

J.-Y. MAO AND I. BENBASAT

FIGURE A3. Example of recommendation screens in Hyper-FINALYZER.

FIGURE A4. Example of deep explanations (how) in Hyper-FINALYZER.

CONTEXTUALIZED ACCESS TO KNOWLEDGE

FIGURE A5. Example of deep explanation (strategic) in Hyper-FINALYZER.

FIGURE A6. Example of reasoning-trace explanations (how) in Hyper-FINALYZER.

813

814

J.-Y. MAO AND I. BENBASAT

Appendix B: The measurement model and other details The measurement model represents the relationships between the observed variables and the constructs they measure. In PLS analysis, item loadings and internal consistencies for the constructs are examined as a test of reliability. Loadings can be considered &&"rst principle component loadings within the context of the model'' (Falk & Miller, 1992, p. 64). While loadings are more appropriate for examination of constructs with re#ective measures, weights are more appropriate for interpreting the formative relationships (by comparing the weights of di!erent indicators). Table B1 shows individual item loadings on the corresponding constructs, along with weights. In the structural model, a reasonable index of the percentage of variance accounted for by a particular predictor construct can be obtained by multiplying the path coe$cient by the corresponding correlation (Falk & Miller, 1992). For example, the path coe$cient from use of deep explanations to judgement congruence is 0.36 (Figure 3), multiplied by the corresponding correlation 0.41 (Table B1), equals 0.15. Therefore, 15% out of 23% of the total variance in judgement congruence accounted for by the structural model was due to the use of deep explanations. As Falk and Miller (1992) suggested, a predictor variable should account for at least 1.5% of the variance in the predicted variable. TABLE B1 Construct measures Measure

Construct

Judgement congruence E1: Based on your analysis and under current economic and interest-rate conditions, rate Canacom's current liquidity position. E2: 2, rate Canacom's long-term solvency position. E3: 2, rate Canacom's asset utilization. E4: 2, rate the value of Canacom's stock as loan collateral. E5: 2, rate the quality of Canacom's ,nancial management. E6: 2, rate the quality of Canacom's operating management. Use of deep explanations Abstract: deep explanations requested through information screens, in the abstract of problem solving. Knowledge-context: deep explanations requested via hypertext links from other related deep explanations invoked from information screen only. Problem-context: deep explanations requested via hypertext links in the context of problem-solving.

LoadingsWeights

0.36 0.20 0.07 0.07 0.35 0.02 0.69 0.81 !0.48 !0.62 0.19 0.31

0.47

0.12

0.60

0.33

0.92

0.80

Use of reasoning-trace explanations =hy: rationalizing why a particular conclusion that has been reached is important for the task. !0.49 !0.86 How: revealing how a particular conclusion has been reached by presenting a trace of the evaluations. 0.35 0.75 Strategic: clarifying the overall goal structure used by the KBS to reach a conclusion, and specifying the manner in which each assessment leading to the conclusion "ts into the overall plan of assessments that have been performed. !0.56 !0.56