- Email: [email protected]
Enhancing knowledge elicitation using the cognitive interview
Expert Systems With Applications, Vol. 10, No. 1, pp. 127-133, 1996 Copyright© 1996 Elsevier Science Ltd Printed in Great Britain. All rights reserved 0957-4174/96 $15.00 + 0.00
Pergamon
0957-4174(95)00039-9
Enhancing Knowledge Elicitation using the Cognitive Interview JANETrE W. MOODY* The Citadel,Charleston,SC
R I C H A R D P. W I L L University of South Florida, Tampa, FL
J. ELLIS BLANTON University of South Florida, Tampa, FL
Abstract--Various knowledge elicitation techniques have been recommended for the development of expert systems, but a review of these techniques reveals that most are designed to elicit rules (i.e. procedural knowledge)from the expert in order to build rule-based expert systems. With the growing interest in case-based expert systems comes the need to introduce and evaluate new knowledge elicitation techniques designed specifically to capture knowledge in the form of cases. Cases represent the unique combination of situational variables and solutions experienced by the expert, i.e. episodic knowledge. This paper reports the finding of a study that investigated the applicability of the Cognitive Interview for eliciting the detail-rich cases required for the development of case-based expert systems. The Cognitive Interview is a technique that was developed specifically to capture episodic knowledge and has been extensively tested for use in other disciplines. This theoretically grounded technique has practical implications for enhanced knowledge elicitation in the development of expert systems.
1. INTRODUCTION
based expert systems. Episodic knowledge, which is highly autobiographical and experimential, is organized in memory by time and place (Tulving, 1983, 1985, 1986; Slade, 1991). The purpose of this study is to investigate the applicability of the Cognitive Interview in the knowledge elicitation phase of expert systems development. The cognitive Interview is a technique (Fisher & Geiselman, 1992) developed and extensively tested for use in criminal and medical investigations, specifically to capture episodic knowledge. This theoretically grounded technique has practical implications for enhanced knowledge elicitation.
VARIOUS KNOWLEDGE ELICITATION TECHNIQUES have been recommended for the development of expert systems (Byrd, Cossick & Zmud, 1992). A review of these techniques reveals that most are designed to elicit rules (i.e. procedural knowledge) from the expert in order to build rule-based expert systems. For a variety of reasons, there is growing interest in case-based expert systems (Slade, 1991; Yoon & Guimaraes, 1993). This new representation requires the introduction and evaluation of new knowledge elicitation techniques designed specifically to capture knowledge in the form of cases. Cases represent the unique combinations of situational variables and solutions experienced by the expert. These experiences are part of the expert's episodic knowledge which is recalled to provide the training set for case-
2. THE COGNITIVE INTERVIEW The categories of knowledge relevant to the development of expert systems, expert systems knowledge representation schemes and suggested knowledge elicitation techniques are presented in Table 1.
*Requests for reprints should be sent to: ProfessorJ. Moody,MSC 142-A, 171 MoultrieSt., Charleston,SC 29409, USA. 127
128
J. W. Moody et al.
TABLE 1 Framework of Knowledge Catagodes, Expert System Representations, end Knowledge EIIcltatlon Techniques
Knowledge Categories
Representation in Expert Systems
Procedural
Rule-Based
Declarative
Frame-Based
Semantic
Semantic Networks
KnowledgeElicitation Techniques Protocol Analysis, Standard KE Interviews Repertory Grids, MDS, Standard KE Interviews Card Sorting, MDS, Repertory Grids, Standard KE Interviews
iiiiiiiiiiiiiiiiiiiiiiii The Cognitive Interview is structured around five principles of memory retrieval: context reinstatement, focused retrieval, extensive retrieval, varied retrieval, and multiple representations. Each principle is based upon theoretical and empirical support, and is enacted during the Cognitive Interview as presented below. The principle of context reinstatement attests that an event is more likely to be recalled if the stimuli surrounding that event are recreated. The term context refers to all stimuli (physical and psychological) concurrent with the event (Best, 1989). Physical context reinstatement can be applied by either conducting the interview in the expert's principle place of employment, or by having the expert mentally visualize and describe the work setting, equipment, etc. Psychological context reinstatement involves having the expert think about and describe his/her feelings during the episodes being recalled (rushed, relaxed, anxious, confident, etc.). The second principle is the principle of focused retrieval, based upon the theories of Kahneman (1973). Kahneman's theories explain why physical (noise, movements, etc.) and psychological (interviewer interruptions of responses) distractions can deteriorate the memory retrieval process. The principle of focused retrieval is facilitated by eliminating distractions, such as noise, extraneous people milling about, or interruptions caused by the interviewer asking questions or filling in pauses. The principle of extensive retrieval is based on a phenomenon called hypermnesia (Roediger & Payne, 1982). Hypermnesia is when people demonstrate increased recall through repeated tests. Effective memory retrieval often requires effort and many interviewees quit after a cursory attempt. Extensive retrieval is enacted by having the expert search through memory even after he/she feels as though everything has been recalled. This cannot be accomplished by having the same question repeated over and over but requires creativity on the part of the interviewer to rephrase and vary the form of the questions to encourage additional searches through memory. Varied retrieval applies the concept that "memories not activated by one retrieval probe may be accessed
with another probe" (Fisher & Quigley, in press). The principle of varied retrieval requires changing the questions from categorical to temporal, and vice versa, as in "Did you have any patron inquire about government documents?" (categorically cued) to "What was the last request you handled that day?" (temporally cued). The multiple representations principle is based on the premise that an event may be stored and recalled in two forms (Fisher & Chandler, 1984). The two-code theory states that when an event occurs, it establishes both an episodic code (i.e. a unique ordinal position) as well as a thematic code (i.e. its commonality with other events). These different codes affect how the complete event will be recalled. Temporally cued recall decreases when retrieval attempts are delayed while thematically cued recall improves after a delay vs no delay. In addition, details of an event may be recalled based on sensory characteristics such as visual images, sounds, and tactile characteristics. In order to use the principle of multiple representations, the interview would begin with the request that the expert provide an open-ended description of all aspects of the events under review. This provides the interviewer with some of the expert's images and mental representations of the items to be recalled. The interviewer can then probe these images separately, for example, by having the expert recall the sound of the patron's voice requesting assistance, images of the request form as it sat on the desk, or images of the customer walking to the desk with a request. The five principles underlying the Cognitive Interview provide guidance and structure for the interviewer who can then customize the interview to the specific expert and case domain. These five principles are summarized in Table 2. 3. M E T H O D O L O G Y An experiment was conducted at eight academic library sites, consisting of private and public colleges and universities, including a law school. Libraries in general and reference services in particular represent a knowl-
Cognitive Interview Enhancement
129
TABLE 2 The Cognitive Interview: Principles of Memory Retrieval
Principle
TABLE 3 Example of Standard Knowledge EUcitation Interview Waldron (1986)
Description DO'S
Context Reinstatement Recall is enchance by recreating the event stimuli (physical and psychological).
A knowledge acquisition interview should be structured and goal oriented.
Focused Retrieval
Distractions deteriorate the memory retrieval process.
It is up to the knowledge engineer to ensure that the knowledge acquisition interview remains directed.
Extensive Retrieval
Recall is increased by increasing the number of retrieval attempts.
The funnel technique of general-to-specific questions is the recommended structure of the interview.
Varied Retrieval
Recall may be activated by different probes.
The knowledge engineer follows an outline of prepared questions in this order:
Multiple Representations
Events may be stored and recalled in two forms (episodic and thematic).
edge domain well-established as appropriate for expert systems (Carrington, 1990; Johnston & Weckert, 1990; Young, 1990; Butkovich, Browning & Taylor, 1991). Reference librarians handle diverse patron inquiries that are often unpredictable due to interrelationships among "numerous situational, system-related and personal variables" (White, 1981). White (1981) has noted that in handling a patron inquiry, the reference librarian may address the following: "--the problem creating the original question; ---the subject of the request; ----the nature of the service to be provided, i.e., the answer requirements; --situational constraints likely to affect selection or use of information, such as a deadline; --personal variables that constitute long-term constraints, such as intelligence and attitude; --prior search history, i.e., what the user has already done to locate the information." (p. 374). Yet, for all of its complexity, "the traditional reference interview disappears without a trace" (Lynch, 1983). Reference librarians seldom have the time to reflect on an inquiry once it has been resolved, so typically there is no specific memory encoding for later recall. In addition, consistent with the "paradox of expertise" (Johnson, 1983), the more proficient they become, the less able they are to articulate specific past encounters. Reference librarians provide the episodic knowledge which would be appropriate for representation in an expert system. This research examines the effectiveness and efficiency of the Cognitive Interview for eliciting episodic knowledge from professional librarians. Each library patron inquiry constitutes an event whose resolution, coupled with complete details of the event, represents the type of training set examples used in building expert systems (Hart, 1987; Yoon & Guimaraes, 1993). In this situation, the "event" which begins with the patron inquiry and ends with the librarian's resolution of that inquiry, is the episodic knowledge.
Open questions are used to obtain a general description of the expert's approach to the task. Probing questions are used to delve into a specific section of the expert's description. Closed questions are used to verify and clarify the knowledge base. The interview is summarized and a transition moves the interview to the next subject area. DON'TS Leading questions designate an appropriate response and should be avoided. Closed questions are restrictive and have limited information value. They should only be used when the knowledge engineer is trying to clarify information.
3.1. Independent Variable The independent variable is the interview technique used to elicit knowledge about the events handled by the reference librarian. The reference librarians (interviewees) were randomly assigned to receive either the standard knowledge elicitation interview (SI) or the Cognitive Interview (CI) by interviewers trained in both techniques. Demographics collected on interviewees confirmed no significant differences between the recipients of the SI vs recipients of the CI. In order to avoid the threat of inter-interviewer differences, an interviewer conducted an equal number of Sis and CIs. At each interview site, a coin was tossed to see which interview treatment (SI or CI) would be conducted first. The treatments were alternated thereafter for that interviewer. Interviewers were students enrolled in a senior-level expert systems class who received training on the SI and on the CI during regularly scheduled class time. The SI is typically presented in the literature in very general terms, as illustrated in Tables 3 and 4. Text books for expert systems development often provide even less direction. Practicing knowledge engineers confirm that knowledge engineers receive little or no formal training in interviewing techniques, and often learn by simply observing other knowledge engineers. This study
130 TABLE 4 Example of Standard Knowledge Elicltatien Interview. Scott, Clayton end Gibson (1991)
J. W. Moody et al. TABLE 5 Interview Efficiency: Number of Events
Tre=ment
N
Median
CI
21
10
SI
21
5
Average
SD
1. INTRODUCE A NEW TOPIC 2. ASK A QUESTION Identify the information objective Put the question into words - - Voice the question to the expert - - Correct the question if necessary --
3. LISTEN TO THE ANSWER 4. RESPOND TO THE ANSWER Ask a follow-up question - - Confirm your understanding - - Resolve contradictions - - Learn how the expert uses unfamiliar terms - - Update your agenda 5. CLOSE THE TOPIC
adhered to the SI guidelines outlined in Tables 3 and 4. Also, as indicated by Fisher and Quigley (in press), a standard interview contains no mnemonic instructions. The CI technique concentrates on reinstating the context of the original event, followed by a variety of retrieval routes to activate recall (Fisher, Geiselman & Amador, 1989; Fisher & Geiselman, 1992). The CI incorporates many of the principles of the SI (i.e. avoid interruptions, establish rapport, use open-ended and closed questions as appropriate) plus an overlay of memory retrieval aides based on the five cognitive principles outlined earlier.
3.2. Dependent Variables Efficiency as measured by the number of events elicited per interview, and effectiveness as measured by the degree of completeness of details recalled per event are the dependent variables examined. The topic, length of encounter, type of inquiry, evaluation of the patron's level of expertise, and the steps taken to provide assistance, comprise the details of an event. A Completeness of Detail (COD) instrument was developed to measure the effectiveness of the interview technique. This instrument consists of a 7-point rating scale to indicate the degree of completeness of detail for each event (7 = "very complete", 1 = "very incomplete") recalled by the interviewees. A panel of two Senior Reference Librarians performed the ratings by independently evaluating each event using the COD instrument. The Senior Reference Librarians, one male and one female, had a total of 25 years of library reference work experience and were from different academic institutions with no affiliation to the interview collection sites. The events were taken from the full-length transcribed interviews, coded for identification purposes, and pre-
10.81
5.619
4.650
2.974
*Significant = p=0.0003.
sented to the panel in random order as stand-alone events. Rater "leaming" was addressed by using different raters between the pilot and final study.
4. RESULTS Interview efficiency was measured by counting the number of events elicited per interview session. Due to a relatively small sample size (n = 21), and to avoid the assumption of homogenity or normally distributed populations, the nonparametric Mann-Whitney test was used to compare the number of events elicited between the CI and the SI. The results are shown in Table 5. The statistical significance supports the position that the CI helps interviewees recall more events. Interviewees receiving the CI also made supportive comments such as "as we continue talking, I keep recalling more" or "keep asking--more things are coming back now". While recalling more events is considered more efficient, the completeness of details recalled is critical for showing the CI to be more effective. Interview effectiveness was measured by comparing the ratings of 32 randomly selected events for both the SI and the CI. The Mann-Whitney test was used again to compare the completeness of detail generated by using the different interview approaches. Using the COD instrument, two raters coded the 32 events with no significant differences between raters. The results are shown in Table 6. The completeness of detail also supports a significantly greater number of details elicited from the CI. This finding indicates that the CI is an important knowledge elicitation aid for capturing episodic knowledge.
5. DISCUSSION There are two major areas of interest from this research project. The first concerns the applicability of the CI to the development of specific types of expert systems, i.e. case-based expert systems. The second addresses the wider applicability of the CI to gathering domain knowledge in general, independent of expert system type. First, this research supports the efficacy of the CI as a knowledge elicitation technique for the development of case-based expert systems. Rule-based, frame-based, and
Cognitive Interview Enhancement
131 TABLE 6 Interview Effectiveness: Completeness of Detail Rater 1"
Rater 2**
Treatment
N
Median
Avg.
SD
Median
Avg.
SD
CI Sl
32 32
6.15 4.85
5.93 4.02
1.43 1.87
6.0 5.0
5.19 4.06
1.51 1.41
*Significant at p = 0.0001 **Significant at p = 0.0009.
semantic network expert systems have experienced many years of development. Based on this maturity, it is not surprising that various knowledge elicitation techniques have been recommended to assist the knowledge engineer in their development. However, as expert systems have become more widespread throughout business organizations, so have the expectations of users regarding the quality of their output (Yoon & Guimaraes, 1993). Users of expert systems place a high degree of faith in the technology (Will, 1991) and can be misled by ill-conceived and poorly developed systems. Although expert systems built using explicit rules, frames, and networks still predominate, there is a growing awareness of limitations in both the development process and the resulting system (Slade, 1991; Yoon & Guimaraes, 1993). Once the development process is complete, an even greater challenge is faced: how to maintain the expert system to keep pace with the dynamic environment it addresses (Will, McQuaig & Hardaway, 1994). Unlike human experts, "most conventional expert systems don't learn from experience" (Yoon & Guimaraes, 1993). Thus a modification of the rule base becomes necessary, accompanied by repercussions on the interrelated rules, resulting in further exposure to rule errors. As a result of these concems, both researchers (Slade, 1991; Yoon & Guimaraes, 1993) and practitioners (Will, McQuaig & Hardaway, 1994) have advocated the use of case-based expert systems. (Note: see Expert Systems With Applications, Vol. 6, No. 1, 1993 for an extensive review of case-based expert systems.) Ultimately, all expert systems rely upon "rules", but in the case-based system the rules are derived through induction, not by having the expert state them explicitly. Not surprisingly, a greater deal of effort has gone into refinement of the algorithmic modeling of this process (Liang, 1992; Mukhopadhyay, Vicinanza & Prietula, 1992; Gaines & Shaw, 1993). It would appear that an equal amount of effort is warranted in improving the repertoire of techniques available to collect information in the form of cases from the expert. Previous developments of casebased expert systems have relied on cases from pre-written sources, such as Harvard Business School Case Services (Chadha, Mazlack & Pick, 1991), patient histories, and legal precedents (Slade, 1991). Unfortunately, reliance on existing sources not only pre-supposes
that all relevant attributes of the cases were captured, it also fails to address those knowledge domains without existing sources. The CI provides a knowledge elicitation technique that specifically addresses the case-building needs of a case-based expert system. The second major conclusion of this study is the CI's wider applicability to the expert system development process in general. There are several characteristics associated with some of the knowledge elicitation techniques currently in use that might be raised as points of concern. For example, multi-dimensional scaling, card sorting, and repertory grids may confine the information elicited from the expert by predetermining what will be useful. Such confinement may leave out important aspects of the knowledge domain under review. In addition, there are drawbacks associated with the use of interviews. Open interviews are seen as ineffective in obtaining the necessary details (Byrd, Cossick & Zmud, 1992). On the other hand, structured interviewing techniques previously tested (Agarwal & Tanniru, 1990) are, by necessity, tied to a specific knowledge domain. As such, they are not applicable to the development of expert systems for other knowledge domains. However, the use of these techniques need not be dysfunctionally confining if, prior to their use, the knowledge engineer has been able to survey the knowledge domain with the expert to determine an overall intelligence of that domain. The CI provides a good candidate technique for making this initial survey of the domain. In other words, the CI provides a theoretically grounded, yet domain independent, technique to guide the expert into providing relevant information. By assisting the expert in the recall of events that utilized his expertise, the CI, used early in the development process, can increase the effectiveness of other elicitation techniques better suited for use later in the process. Figure 1 depicts elicitation techniques in the development process. Several practical implications have emerged from this study. First, the CI uses theoretical principles that have an intuitive appeal in general. They represent techniques that many individuals have used in attempting to recall information for themselves, often without knowing consciously why the techniques were effective. It seems reasonable to suggest that this intuitive appeal makes training in their use easier and their enactment more
132
J. W. Moody et al.
KnowledgeElieitation Teclmiques
Development Interaction
Knowledge Representation
i!iii;i;~iiiii!i~;ilili!i:i:i!iiii~
ii!iii!ii!i!i!i!•iiiiiii!Protocol iiiil Analysis
Knowledge Engineer
~ii:iiii~iiii!~!iiiililUii!ii~!ili!~!i~i,iii!iii
MultidimensionalScaling
Rule-Based
Frame-Based
Repertory Grids ~ii'i~!!!i~iiiil]ii!iii!i!iiiiiiiiiiiiii,iii!iiii '
•
Card Sorting
SemanticNet
iiii!ii!!;i!ili!i~ii!i!!!!iii:i~iiil Expert
Case-Based
FIGURE 1. Elicitatlon techniques in the development process.
effective than other interviewing techniques. Interview actions that are appropriate to the Cognitive Interview are defined in Table 7. Secondly, no special artifacts or peripherals are required to conduct a CI, as are required by other knowledge elicitation techniques such as multi-dimensional scaling, card sorting, etc. This feature increases the portability of the technique for a variety of situations and minimizes the intrusions in the knowledge engineer--expert relationship. Finally, one defining feature of an expert system is that it addresses "relatively limited and narrowly scoped areas of expertise" (Turban, 1990). The CI's principles of memory retrieval are applicable to episodic knowledge
in general, making it useful regardless of the knowledge domain under investigation. 6. LIMITATIONS AND FUTURE W O R K Several limitations of this field study must be noted. For example, it was not possible to determine the accuracy of the elicited information. However, since respondents were quite willing to say when they could not recall events and their compensation was not tied to the number of events recalled, there is no reason to believe that the events recalled were fabricated. In addition, the interview sessions were limited to one hour in order to minimize participant fatigue while still allowing sufficient time to collect detailed events.
TABLE 7 Cognitive Interview: Application of Principles
Principle
Application
Context Reinstatement
Ask interviewee to think back to the original event, recalling the physical surroundings (time of day, workspace area, weather conditions) as well as emotional/physiological (bored, rushed, relaxed, etc.)
Focused Retrieval
Interviewee may close eyes to minimize distractions; body movements should be minimized. Interviewer must avoid interrupting and eliminate external intrusions to the session.
Extensive Retrieval
Interviewer does not let interviewee stop after a cursory search of memory but encourages multiple attempts.
Varied Retrieval
It is common to recall details of an event in chronological order and from an egocentric perspective. Ask the interviewee to recall details in reverse order or starting in the middle and working to either end. In addition, having the interviewee recall the event from the perspective of a third party witnessing the event may elicit details not previously recalled.
Multiple Representations
Have interviewee recall details considered unusual, humorous, etc. (i.e. those that share a theme). Other details may be recalled by having interviewee use multiple senses, such as sounds, visual images, and tactile representations.
Cognitive Interview Enhancement A l t h o u g h this m a y in fact be the m a x i m u m a m o u n t of time an expert c a n give the k n o w l e d g e e n g i n e e r at one time, in practice there would need to be m o r e than one interview session over the course o f systems development. It m a y be that, by the third or fourth session of the interview series, n o appreciable difference exists in the n u m b e r a n d / o r quality of details elicited through the CI versus the SI. Furthermore, although it was not observed d u r i n g this study, it could be that the CI is perceived as too p r o b i n g for interviewees, causing them discomfort. If this were so, the c o n t e n t o f subsequent interviews might be adversely affected due to d i m i n i s h e d interviewee cooperation. Therefore, another limitation o f the study m a y be that the favorable statistical findings were the result of an abbreviated interview process rather than the overall value of the treatment itself. Future research on use of the CI as a k n o w l e d g e elicitation technique should address several research questions. F o r example, while the CI produced a statistically significant higher n u m b e r of events and details recalled, there is m u c h r o o m for i m p r o v e m e n t in the scores obtained, i.e. from an average score of 5.6 for the degree o f completeness of details recalled to something closer to 7.0. Several factors probably contributed to the relatively low scores. First, the interviewers used were students in an Expert Systems class with little real world experience in i n t e r v i e w i n g strangers on any subject. Therefore, their limited basic interviewing skills probably affected the outcomes of both the SI and CI equally. Higher scores should be obtained with the use of experienced k n o w l e d g e engineers w h o have been trained in using the CI. Finally, the experimental nature of this project m a d e it necessary to keep the c o m p o n e n t s of the CI hidden from interviewees. F u t u r e research might explore whether or not m a k i n g interviewees aware of the m e m o r y retrieval techniques d u r i n g the interview process enhances their participation a n d results in increased recall of events.
REFERENCES Agarwal, R. & Tanniru, M. (1990). Knowledge acquisition using structured interviewing. Journal of Management Information Systems, 7(1), 123-140. Best, J. (1989). Cognitive Psychology. 2nd Edn, St. Paul, MN: West Publishing Co. Butkovich, N., Browning, M. & Taylor, K. (1991). The reference expert: a computerized database utilizing INMAGIC and a WORM drive. DATABASE, 35-38. Byrd, T., Cossick, K. & Zmud, R. (1992). A Synthesis of research on requirements analysis and knowledge acquisition techniques. MIS Quarterly, 117-128. Carrington, B. (1990). Expert systems: power to the experts. DATABASE 47-50. Chadha, S., Mazlack, L. & Pick, R. (1991). Using existing knowledge sources (cases) to build an expert system. Expert Systems, 8(1), 3-12. Fisher, R. & Chandler, C. (1984). Dissociations between temporally cued and thematically cued recall. Bulletin of the Psychonomic
133 Society, 22(5), 395-397. Fisher, R., Chin, D. & McCauley, M. (1990). Enhancing eyewitness recollection with the cognitive interview. National Police Research Unit, 6, 3-11. Fisher, R. & Geiselman, R. (1992). Memory Enhancing Techniquesfor Investigative Interviewing. Springfield, IL: Charles Thomas Publishers. Fisher, R., Geiseiman, R. & Amador, M. (1989). Field test of the cognitive interview: enhancing the recollection of actual victims and witnesses of crime. Journal of Applied Psychology, 74(5), 722-727. Fisher, R. & Quigley, K. (1995). Applying cognitive theory in public health investigations: enhancing food recall with the Cognitive Interview. In Tanur, J. (Ed.), Questions About Questions. New York: Sage. Gaines, B. & Shaw, M. (1983). Eliciting knowledge and transferring it effectively to a knowledge-based system. IEEE Tansactions on Knowledge and Data Engineering, 5(1), 4-13. Hart, A. (1987). Role of induction in knowledge elicitation. In A. L. Kidd (Ed.), Knowledge Acquisition for Expert Systems: A Practical Handbook, pp. 165-189. London: Plenum Press. Johnson, P. (1983) What kind of expert should a system be? Journal of Medicine and Philosophy, 8, 77-97. Johnston, M. & Weckert, J. (1990). Selection advisor: an expert system for collection development. Information Technology and Libraries, 9(3), 219-225. Kahneman, D. (1973). Attention and Effort, Englewood Cliffs, NJ: Prentice Hall. Liang, T. (1992). A composite approach to inducing knowledge for expert systems design. Management Science, 38(1), 1-17. Lynch, M. (1983). Research in library reference/information service. Library Trends, 12. Mukhopadhyay, T., Vicinanza, S. & Prietula, M. (1992). Examining the feasibility of a case-based reasoning model for software effort estimation. MIS Quartelry, 155-171. Noether, G. (1967). Elements of Nonparametric Statistics, New York: Wiley. Roediger, H. & Payne, D. (1982). Hypermnesia: the role of repeated testing. Journal of Experimental Psychology: Learning, Memory, and Cognition, 8(1), 66-72. Scott, A., Clayton, J. & Gibson, E. (1991). A Practical Guide to Knowledge Acquisition, Reading, Massachusetts: Addison-Wesley Publishing Co. Slade, S. (Spring, 1991). Case-based reasoning: a research paradigm. AI Magazine, 42-55. Tulving, E. (1983). Elements of Episodic Memory. Oxford: Clarendon Press. Tulving, E. (1985). How many memory systems are there? American Psychologist, 40, 385-398. Tulving, E. (1986). What kind of a hypothesis is the distinction between episodic and semantic memory? Journal of Experimental Psychology: Learning, Memory and Cognition, 12(2), 307-311. Turban, E. (1990) Decision Support and Expert Systems, New York: Macmillan Publishing Co. Waldron, V. (1986). Interviewing for knowledge. IEEE Transactions on Professional Communications, 29(2), 31-34. White, M (1981). The dimensions of the reference interview. RQ, 20(4), 373-381. Will, R. (1991). True and false dependence on technology: evaluation with an expert system. Computers in Human Behavior, 17, 171-183. Will, R., McQuaig, M. & Hardaway, D. (1994). Identifying long-term success issues of expert systems. Expert Systems with Applications, 7(2), 273-280. Yoon, Y. & Guimaraes, T. (1993). Selecting expert system development techniques. Information and Management, 24, 209-223. Young, R. (1990). Artificial intelligence and school library media centers. School Library Media Quarterly, 18(3), 150-157.
Pergamon
0957-4174(95)00039-9
Enhancing Knowledge Elicitation using the Cognitive Interview JANETrE W. MOODY* The Citadel,Charleston,SC
R I C H A R D P. W I L L University of South Florida, Tampa, FL
J. ELLIS BLANTON University of South Florida, Tampa, FL
Abstract--Various knowledge elicitation techniques have been recommended for the development of expert systems, but a review of these techniques reveals that most are designed to elicit rules (i.e. procedural knowledge)from the expert in order to build rule-based expert systems. With the growing interest in case-based expert systems comes the need to introduce and evaluate new knowledge elicitation techniques designed specifically to capture knowledge in the form of cases. Cases represent the unique combination of situational variables and solutions experienced by the expert, i.e. episodic knowledge. This paper reports the finding of a study that investigated the applicability of the Cognitive Interview for eliciting the detail-rich cases required for the development of case-based expert systems. The Cognitive Interview is a technique that was developed specifically to capture episodic knowledge and has been extensively tested for use in other disciplines. This theoretically grounded technique has practical implications for enhanced knowledge elicitation in the development of expert systems.
1. INTRODUCTION
based expert systems. Episodic knowledge, which is highly autobiographical and experimential, is organized in memory by time and place (Tulving, 1983, 1985, 1986; Slade, 1991). The purpose of this study is to investigate the applicability of the Cognitive Interview in the knowledge elicitation phase of expert systems development. The cognitive Interview is a technique (Fisher & Geiselman, 1992) developed and extensively tested for use in criminal and medical investigations, specifically to capture episodic knowledge. This theoretically grounded technique has practical implications for enhanced knowledge elicitation.
VARIOUS KNOWLEDGE ELICITATION TECHNIQUES have been recommended for the development of expert systems (Byrd, Cossick & Zmud, 1992). A review of these techniques reveals that most are designed to elicit rules (i.e. procedural knowledge) from the expert in order to build rule-based expert systems. For a variety of reasons, there is growing interest in case-based expert systems (Slade, 1991; Yoon & Guimaraes, 1993). This new representation requires the introduction and evaluation of new knowledge elicitation techniques designed specifically to capture knowledge in the form of cases. Cases represent the unique combinations of situational variables and solutions experienced by the expert. These experiences are part of the expert's episodic knowledge which is recalled to provide the training set for case-
2. THE COGNITIVE INTERVIEW The categories of knowledge relevant to the development of expert systems, expert systems knowledge representation schemes and suggested knowledge elicitation techniques are presented in Table 1.
*Requests for reprints should be sent to: ProfessorJ. Moody,MSC 142-A, 171 MoultrieSt., Charleston,SC 29409, USA. 127
128
J. W. Moody et al.
TABLE 1 Framework of Knowledge Catagodes, Expert System Representations, end Knowledge EIIcltatlon Techniques
Knowledge Categories
Representation in Expert Systems
Procedural
Rule-Based
Declarative
Frame-Based
Semantic
Semantic Networks
KnowledgeElicitation Techniques Protocol Analysis, Standard KE Interviews Repertory Grids, MDS, Standard KE Interviews Card Sorting, MDS, Repertory Grids, Standard KE Interviews
iiiiiiiiiiiiiiiiiiiiiiii The Cognitive Interview is structured around five principles of memory retrieval: context reinstatement, focused retrieval, extensive retrieval, varied retrieval, and multiple representations. Each principle is based upon theoretical and empirical support, and is enacted during the Cognitive Interview as presented below. The principle of context reinstatement attests that an event is more likely to be recalled if the stimuli surrounding that event are recreated. The term context refers to all stimuli (physical and psychological) concurrent with the event (Best, 1989). Physical context reinstatement can be applied by either conducting the interview in the expert's principle place of employment, or by having the expert mentally visualize and describe the work setting, equipment, etc. Psychological context reinstatement involves having the expert think about and describe his/her feelings during the episodes being recalled (rushed, relaxed, anxious, confident, etc.). The second principle is the principle of focused retrieval, based upon the theories of Kahneman (1973). Kahneman's theories explain why physical (noise, movements, etc.) and psychological (interviewer interruptions of responses) distractions can deteriorate the memory retrieval process. The principle of focused retrieval is facilitated by eliminating distractions, such as noise, extraneous people milling about, or interruptions caused by the interviewer asking questions or filling in pauses. The principle of extensive retrieval is based on a phenomenon called hypermnesia (Roediger & Payne, 1982). Hypermnesia is when people demonstrate increased recall through repeated tests. Effective memory retrieval often requires effort and many interviewees quit after a cursory attempt. Extensive retrieval is enacted by having the expert search through memory even after he/she feels as though everything has been recalled. This cannot be accomplished by having the same question repeated over and over but requires creativity on the part of the interviewer to rephrase and vary the form of the questions to encourage additional searches through memory. Varied retrieval applies the concept that "memories not activated by one retrieval probe may be accessed
with another probe" (Fisher & Quigley, in press). The principle of varied retrieval requires changing the questions from categorical to temporal, and vice versa, as in "Did you have any patron inquire about government documents?" (categorically cued) to "What was the last request you handled that day?" (temporally cued). The multiple representations principle is based on the premise that an event may be stored and recalled in two forms (Fisher & Chandler, 1984). The two-code theory states that when an event occurs, it establishes both an episodic code (i.e. a unique ordinal position) as well as a thematic code (i.e. its commonality with other events). These different codes affect how the complete event will be recalled. Temporally cued recall decreases when retrieval attempts are delayed while thematically cued recall improves after a delay vs no delay. In addition, details of an event may be recalled based on sensory characteristics such as visual images, sounds, and tactile characteristics. In order to use the principle of multiple representations, the interview would begin with the request that the expert provide an open-ended description of all aspects of the events under review. This provides the interviewer with some of the expert's images and mental representations of the items to be recalled. The interviewer can then probe these images separately, for example, by having the expert recall the sound of the patron's voice requesting assistance, images of the request form as it sat on the desk, or images of the customer walking to the desk with a request. The five principles underlying the Cognitive Interview provide guidance and structure for the interviewer who can then customize the interview to the specific expert and case domain. These five principles are summarized in Table 2. 3. M E T H O D O L O G Y An experiment was conducted at eight academic library sites, consisting of private and public colleges and universities, including a law school. Libraries in general and reference services in particular represent a knowl-
Cognitive Interview Enhancement
129
TABLE 2 The Cognitive Interview: Principles of Memory Retrieval
Principle
TABLE 3 Example of Standard Knowledge EUcitation Interview Waldron (1986)
Description DO'S
Context Reinstatement Recall is enchance by recreating the event stimuli (physical and psychological).
A knowledge acquisition interview should be structured and goal oriented.
Focused Retrieval
Distractions deteriorate the memory retrieval process.
It is up to the knowledge engineer to ensure that the knowledge acquisition interview remains directed.
Extensive Retrieval
Recall is increased by increasing the number of retrieval attempts.
The funnel technique of general-to-specific questions is the recommended structure of the interview.
Varied Retrieval
Recall may be activated by different probes.
The knowledge engineer follows an outline of prepared questions in this order:
Multiple Representations
Events may be stored and recalled in two forms (episodic and thematic).
edge domain well-established as appropriate for expert systems (Carrington, 1990; Johnston & Weckert, 1990; Young, 1990; Butkovich, Browning & Taylor, 1991). Reference librarians handle diverse patron inquiries that are often unpredictable due to interrelationships among "numerous situational, system-related and personal variables" (White, 1981). White (1981) has noted that in handling a patron inquiry, the reference librarian may address the following: "--the problem creating the original question; ---the subject of the request; ----the nature of the service to be provided, i.e., the answer requirements; --situational constraints likely to affect selection or use of information, such as a deadline; --personal variables that constitute long-term constraints, such as intelligence and attitude; --prior search history, i.e., what the user has already done to locate the information." (p. 374). Yet, for all of its complexity, "the traditional reference interview disappears without a trace" (Lynch, 1983). Reference librarians seldom have the time to reflect on an inquiry once it has been resolved, so typically there is no specific memory encoding for later recall. In addition, consistent with the "paradox of expertise" (Johnson, 1983), the more proficient they become, the less able they are to articulate specific past encounters. Reference librarians provide the episodic knowledge which would be appropriate for representation in an expert system. This research examines the effectiveness and efficiency of the Cognitive Interview for eliciting episodic knowledge from professional librarians. Each library patron inquiry constitutes an event whose resolution, coupled with complete details of the event, represents the type of training set examples used in building expert systems (Hart, 1987; Yoon & Guimaraes, 1993). In this situation, the "event" which begins with the patron inquiry and ends with the librarian's resolution of that inquiry, is the episodic knowledge.
Open questions are used to obtain a general description of the expert's approach to the task. Probing questions are used to delve into a specific section of the expert's description. Closed questions are used to verify and clarify the knowledge base. The interview is summarized and a transition moves the interview to the next subject area. DON'TS Leading questions designate an appropriate response and should be avoided. Closed questions are restrictive and have limited information value. They should only be used when the knowledge engineer is trying to clarify information.
3.1. Independent Variable The independent variable is the interview technique used to elicit knowledge about the events handled by the reference librarian. The reference librarians (interviewees) were randomly assigned to receive either the standard knowledge elicitation interview (SI) or the Cognitive Interview (CI) by interviewers trained in both techniques. Demographics collected on interviewees confirmed no significant differences between the recipients of the SI vs recipients of the CI. In order to avoid the threat of inter-interviewer differences, an interviewer conducted an equal number of Sis and CIs. At each interview site, a coin was tossed to see which interview treatment (SI or CI) would be conducted first. The treatments were alternated thereafter for that interviewer. Interviewers were students enrolled in a senior-level expert systems class who received training on the SI and on the CI during regularly scheduled class time. The SI is typically presented in the literature in very general terms, as illustrated in Tables 3 and 4. Text books for expert systems development often provide even less direction. Practicing knowledge engineers confirm that knowledge engineers receive little or no formal training in interviewing techniques, and often learn by simply observing other knowledge engineers. This study
130 TABLE 4 Example of Standard Knowledge Elicltatien Interview. Scott, Clayton end Gibson (1991)
J. W. Moody et al. TABLE 5 Interview Efficiency: Number of Events
Tre=ment
N
Median
CI
21
10
SI
21
5
Average
SD
1. INTRODUCE A NEW TOPIC 2. ASK A QUESTION Identify the information objective Put the question into words - - Voice the question to the expert - - Correct the question if necessary --
3. LISTEN TO THE ANSWER 4. RESPOND TO THE ANSWER Ask a follow-up question - - Confirm your understanding - - Resolve contradictions - - Learn how the expert uses unfamiliar terms - - Update your agenda 5. CLOSE THE TOPIC
adhered to the SI guidelines outlined in Tables 3 and 4. Also, as indicated by Fisher and Quigley (in press), a standard interview contains no mnemonic instructions. The CI technique concentrates on reinstating the context of the original event, followed by a variety of retrieval routes to activate recall (Fisher, Geiselman & Amador, 1989; Fisher & Geiselman, 1992). The CI incorporates many of the principles of the SI (i.e. avoid interruptions, establish rapport, use open-ended and closed questions as appropriate) plus an overlay of memory retrieval aides based on the five cognitive principles outlined earlier.
3.2. Dependent Variables Efficiency as measured by the number of events elicited per interview, and effectiveness as measured by the degree of completeness of details recalled per event are the dependent variables examined. The topic, length of encounter, type of inquiry, evaluation of the patron's level of expertise, and the steps taken to provide assistance, comprise the details of an event. A Completeness of Detail (COD) instrument was developed to measure the effectiveness of the interview technique. This instrument consists of a 7-point rating scale to indicate the degree of completeness of detail for each event (7 = "very complete", 1 = "very incomplete") recalled by the interviewees. A panel of two Senior Reference Librarians performed the ratings by independently evaluating each event using the COD instrument. The Senior Reference Librarians, one male and one female, had a total of 25 years of library reference work experience and were from different academic institutions with no affiliation to the interview collection sites. The events were taken from the full-length transcribed interviews, coded for identification purposes, and pre-
10.81
5.619
4.650
2.974
*Significant = p=0.0003.
sented to the panel in random order as stand-alone events. Rater "leaming" was addressed by using different raters between the pilot and final study.
4. RESULTS Interview efficiency was measured by counting the number of events elicited per interview session. Due to a relatively small sample size (n = 21), and to avoid the assumption of homogenity or normally distributed populations, the nonparametric Mann-Whitney test was used to compare the number of events elicited between the CI and the SI. The results are shown in Table 5. The statistical significance supports the position that the CI helps interviewees recall more events. Interviewees receiving the CI also made supportive comments such as "as we continue talking, I keep recalling more" or "keep asking--more things are coming back now". While recalling more events is considered more efficient, the completeness of details recalled is critical for showing the CI to be more effective. Interview effectiveness was measured by comparing the ratings of 32 randomly selected events for both the SI and the CI. The Mann-Whitney test was used again to compare the completeness of detail generated by using the different interview approaches. Using the COD instrument, two raters coded the 32 events with no significant differences between raters. The results are shown in Table 6. The completeness of detail also supports a significantly greater number of details elicited from the CI. This finding indicates that the CI is an important knowledge elicitation aid for capturing episodic knowledge.
5. DISCUSSION There are two major areas of interest from this research project. The first concerns the applicability of the CI to the development of specific types of expert systems, i.e. case-based expert systems. The second addresses the wider applicability of the CI to gathering domain knowledge in general, independent of expert system type. First, this research supports the efficacy of the CI as a knowledge elicitation technique for the development of case-based expert systems. Rule-based, frame-based, and
Cognitive Interview Enhancement
131 TABLE 6 Interview Effectiveness: Completeness of Detail Rater 1"
Rater 2**
Treatment
N
Median
Avg.
SD
Median
Avg.
SD
CI Sl
32 32
6.15 4.85
5.93 4.02
1.43 1.87
6.0 5.0
5.19 4.06
1.51 1.41
*Significant at p = 0.0001 **Significant at p = 0.0009.
semantic network expert systems have experienced many years of development. Based on this maturity, it is not surprising that various knowledge elicitation techniques have been recommended to assist the knowledge engineer in their development. However, as expert systems have become more widespread throughout business organizations, so have the expectations of users regarding the quality of their output (Yoon & Guimaraes, 1993). Users of expert systems place a high degree of faith in the technology (Will, 1991) and can be misled by ill-conceived and poorly developed systems. Although expert systems built using explicit rules, frames, and networks still predominate, there is a growing awareness of limitations in both the development process and the resulting system (Slade, 1991; Yoon & Guimaraes, 1993). Once the development process is complete, an even greater challenge is faced: how to maintain the expert system to keep pace with the dynamic environment it addresses (Will, McQuaig & Hardaway, 1994). Unlike human experts, "most conventional expert systems don't learn from experience" (Yoon & Guimaraes, 1993). Thus a modification of the rule base becomes necessary, accompanied by repercussions on the interrelated rules, resulting in further exposure to rule errors. As a result of these concems, both researchers (Slade, 1991; Yoon & Guimaraes, 1993) and practitioners (Will, McQuaig & Hardaway, 1994) have advocated the use of case-based expert systems. (Note: see Expert Systems With Applications, Vol. 6, No. 1, 1993 for an extensive review of case-based expert systems.) Ultimately, all expert systems rely upon "rules", but in the case-based system the rules are derived through induction, not by having the expert state them explicitly. Not surprisingly, a greater deal of effort has gone into refinement of the algorithmic modeling of this process (Liang, 1992; Mukhopadhyay, Vicinanza & Prietula, 1992; Gaines & Shaw, 1993). It would appear that an equal amount of effort is warranted in improving the repertoire of techniques available to collect information in the form of cases from the expert. Previous developments of casebased expert systems have relied on cases from pre-written sources, such as Harvard Business School Case Services (Chadha, Mazlack & Pick, 1991), patient histories, and legal precedents (Slade, 1991). Unfortunately, reliance on existing sources not only pre-supposes
that all relevant attributes of the cases were captured, it also fails to address those knowledge domains without existing sources. The CI provides a knowledge elicitation technique that specifically addresses the case-building needs of a case-based expert system. The second major conclusion of this study is the CI's wider applicability to the expert system development process in general. There are several characteristics associated with some of the knowledge elicitation techniques currently in use that might be raised as points of concern. For example, multi-dimensional scaling, card sorting, and repertory grids may confine the information elicited from the expert by predetermining what will be useful. Such confinement may leave out important aspects of the knowledge domain under review. In addition, there are drawbacks associated with the use of interviews. Open interviews are seen as ineffective in obtaining the necessary details (Byrd, Cossick & Zmud, 1992). On the other hand, structured interviewing techniques previously tested (Agarwal & Tanniru, 1990) are, by necessity, tied to a specific knowledge domain. As such, they are not applicable to the development of expert systems for other knowledge domains. However, the use of these techniques need not be dysfunctionally confining if, prior to their use, the knowledge engineer has been able to survey the knowledge domain with the expert to determine an overall intelligence of that domain. The CI provides a good candidate technique for making this initial survey of the domain. In other words, the CI provides a theoretically grounded, yet domain independent, technique to guide the expert into providing relevant information. By assisting the expert in the recall of events that utilized his expertise, the CI, used early in the development process, can increase the effectiveness of other elicitation techniques better suited for use later in the process. Figure 1 depicts elicitation techniques in the development process. Several practical implications have emerged from this study. First, the CI uses theoretical principles that have an intuitive appeal in general. They represent techniques that many individuals have used in attempting to recall information for themselves, often without knowing consciously why the techniques were effective. It seems reasonable to suggest that this intuitive appeal makes training in their use easier and their enactment more
132
J. W. Moody et al.
KnowledgeElieitation Teclmiques
Development Interaction
Knowledge Representation
i!iii;i;~iiiii!i~;ilili!i:i:i!iiii~
ii!iii!ii!i!i!i!•iiiiiii!Protocol iiiil Analysis
Knowledge Engineer
~ii:iiii~iiii!~!iiiililUii!ii~!ili!~!i~i,iii!iii
MultidimensionalScaling
Rule-Based
Frame-Based
Repertory Grids ~ii'i~!!!i~iiiil]ii!iii!i!iiiiiiiiiiiiii,iii!iiii '
•
Card Sorting
SemanticNet
iiii!ii!!;i!ili!i~ii!i!!!!iii:i~iiil Expert
Case-Based
FIGURE 1. Elicitatlon techniques in the development process.
effective than other interviewing techniques. Interview actions that are appropriate to the Cognitive Interview are defined in Table 7. Secondly, no special artifacts or peripherals are required to conduct a CI, as are required by other knowledge elicitation techniques such as multi-dimensional scaling, card sorting, etc. This feature increases the portability of the technique for a variety of situations and minimizes the intrusions in the knowledge engineer--expert relationship. Finally, one defining feature of an expert system is that it addresses "relatively limited and narrowly scoped areas of expertise" (Turban, 1990). The CI's principles of memory retrieval are applicable to episodic knowledge
in general, making it useful regardless of the knowledge domain under investigation. 6. LIMITATIONS AND FUTURE W O R K Several limitations of this field study must be noted. For example, it was not possible to determine the accuracy of the elicited information. However, since respondents were quite willing to say when they could not recall events and their compensation was not tied to the number of events recalled, there is no reason to believe that the events recalled were fabricated. In addition, the interview sessions were limited to one hour in order to minimize participant fatigue while still allowing sufficient time to collect detailed events.
TABLE 7 Cognitive Interview: Application of Principles
Principle
Application
Context Reinstatement
Ask interviewee to think back to the original event, recalling the physical surroundings (time of day, workspace area, weather conditions) as well as emotional/physiological (bored, rushed, relaxed, etc.)
Focused Retrieval
Interviewee may close eyes to minimize distractions; body movements should be minimized. Interviewer must avoid interrupting and eliminate external intrusions to the session.
Extensive Retrieval
Interviewer does not let interviewee stop after a cursory search of memory but encourages multiple attempts.
Varied Retrieval
It is common to recall details of an event in chronological order and from an egocentric perspective. Ask the interviewee to recall details in reverse order or starting in the middle and working to either end. In addition, having the interviewee recall the event from the perspective of a third party witnessing the event may elicit details not previously recalled.
Multiple Representations
Have interviewee recall details considered unusual, humorous, etc. (i.e. those that share a theme). Other details may be recalled by having interviewee use multiple senses, such as sounds, visual images, and tactile representations.
Cognitive Interview Enhancement A l t h o u g h this m a y in fact be the m a x i m u m a m o u n t of time an expert c a n give the k n o w l e d g e e n g i n e e r at one time, in practice there would need to be m o r e than one interview session over the course o f systems development. It m a y be that, by the third or fourth session of the interview series, n o appreciable difference exists in the n u m b e r a n d / o r quality of details elicited through the CI versus the SI. Furthermore, although it was not observed d u r i n g this study, it could be that the CI is perceived as too p r o b i n g for interviewees, causing them discomfort. If this were so, the c o n t e n t o f subsequent interviews might be adversely affected due to d i m i n i s h e d interviewee cooperation. Therefore, another limitation o f the study m a y be that the favorable statistical findings were the result of an abbreviated interview process rather than the overall value of the treatment itself. Future research on use of the CI as a k n o w l e d g e elicitation technique should address several research questions. F o r example, while the CI produced a statistically significant higher n u m b e r of events and details recalled, there is m u c h r o o m for i m p r o v e m e n t in the scores obtained, i.e. from an average score of 5.6 for the degree o f completeness of details recalled to something closer to 7.0. Several factors probably contributed to the relatively low scores. First, the interviewers used were students in an Expert Systems class with little real world experience in i n t e r v i e w i n g strangers on any subject. Therefore, their limited basic interviewing skills probably affected the outcomes of both the SI and CI equally. Higher scores should be obtained with the use of experienced k n o w l e d g e engineers w h o have been trained in using the CI. Finally, the experimental nature of this project m a d e it necessary to keep the c o m p o n e n t s of the CI hidden from interviewees. F u t u r e research might explore whether or not m a k i n g interviewees aware of the m e m o r y retrieval techniques d u r i n g the interview process enhances their participation a n d results in increased recall of events.
REFERENCES Agarwal, R. & Tanniru, M. (1990). Knowledge acquisition using structured interviewing. Journal of Management Information Systems, 7(1), 123-140. Best, J. (1989). Cognitive Psychology. 2nd Edn, St. Paul, MN: West Publishing Co. Butkovich, N., Browning, M. & Taylor, K. (1991). The reference expert: a computerized database utilizing INMAGIC and a WORM drive. DATABASE, 35-38. Byrd, T., Cossick, K. & Zmud, R. (1992). A Synthesis of research on requirements analysis and knowledge acquisition techniques. MIS Quarterly, 117-128. Carrington, B. (1990). Expert systems: power to the experts. DATABASE 47-50. Chadha, S., Mazlack, L. & Pick, R. (1991). Using existing knowledge sources (cases) to build an expert system. Expert Systems, 8(1), 3-12. Fisher, R. & Chandler, C. (1984). Dissociations between temporally cued and thematically cued recall. Bulletin of the Psychonomic
133 Society, 22(5), 395-397. Fisher, R., Chin, D. & McCauley, M. (1990). Enhancing eyewitness recollection with the cognitive interview. National Police Research Unit, 6, 3-11. Fisher, R. & Geiselman, R. (1992). Memory Enhancing Techniquesfor Investigative Interviewing. Springfield, IL: Charles Thomas Publishers. Fisher, R., Geiseiman, R. & Amador, M. (1989). Field test of the cognitive interview: enhancing the recollection of actual victims and witnesses of crime. Journal of Applied Psychology, 74(5), 722-727. Fisher, R. & Quigley, K. (1995). Applying cognitive theory in public health investigations: enhancing food recall with the Cognitive Interview. In Tanur, J. (Ed.), Questions About Questions. New York: Sage. Gaines, B. & Shaw, M. (1983). Eliciting knowledge and transferring it effectively to a knowledge-based system. IEEE Tansactions on Knowledge and Data Engineering, 5(1), 4-13. Hart, A. (1987). Role of induction in knowledge elicitation. In A. L. Kidd (Ed.), Knowledge Acquisition for Expert Systems: A Practical Handbook, pp. 165-189. London: Plenum Press. Johnson, P. (1983) What kind of expert should a system be? Journal of Medicine and Philosophy, 8, 77-97. Johnston, M. & Weckert, J. (1990). Selection advisor: an expert system for collection development. Information Technology and Libraries, 9(3), 219-225. Kahneman, D. (1973). Attention and Effort, Englewood Cliffs, NJ: Prentice Hall. Liang, T. (1992). A composite approach to inducing knowledge for expert systems design. Management Science, 38(1), 1-17. Lynch, M. (1983). Research in library reference/information service. Library Trends, 12. Mukhopadhyay, T., Vicinanza, S. & Prietula, M. (1992). Examining the feasibility of a case-based reasoning model for software effort estimation. MIS Quartelry, 155-171. Noether, G. (1967). Elements of Nonparametric Statistics, New York: Wiley. Roediger, H. & Payne, D. (1982). Hypermnesia: the role of repeated testing. Journal of Experimental Psychology: Learning, Memory, and Cognition, 8(1), 66-72. Scott, A., Clayton, J. & Gibson, E. (1991). A Practical Guide to Knowledge Acquisition, Reading, Massachusetts: Addison-Wesley Publishing Co. Slade, S. (Spring, 1991). Case-based reasoning: a research paradigm. AI Magazine, 42-55. Tulving, E. (1983). Elements of Episodic Memory. Oxford: Clarendon Press. Tulving, E. (1985). How many memory systems are there? American Psychologist, 40, 385-398. Tulving, E. (1986). What kind of a hypothesis is the distinction between episodic and semantic memory? Journal of Experimental Psychology: Learning, Memory and Cognition, 12(2), 307-311. Turban, E. (1990) Decision Support and Expert Systems, New York: Macmillan Publishing Co. Waldron, V. (1986). Interviewing for knowledge. IEEE Transactions on Professional Communications, 29(2), 31-34. White, M (1981). The dimensions of the reference interview. RQ, 20(4), 373-381. Will, R. (1991). True and false dependence on technology: evaluation with an expert system. Computers in Human Behavior, 17, 171-183. Will, R., McQuaig, M. & Hardaway, D. (1994). Identifying long-term success issues of expert systems. Expert Systems with Applications, 7(2), 273-280. Yoon, Y. & Guimaraes, T. (1993). Selecting expert system development techniques. Information and Management, 24, 209-223. Young, R. (1990). Artificial intelligence and school library media centers. School Library Media Quarterly, 18(3), 150-157.