- Email: [email protected]
Applying tacit knowledge management techniques for performance assessment
Computers & Education 41 (2003) 173–189 www.elsevier.com/locate/compedu
Applying tacit knowledge management techniques for performance assessment Michel Mitri* James Madison University, CIS/OM Program, College of Business, 309 Zane Showker Hall, Harrisonburg, VA 22807, USA Received 1 December 2002; accepted 25 March 2003
Abstract Performance assessment is an important task in all levels of education, both as input for identifying remedial needs of individual students and for improving general quality of education. Although explicit assessment measures can be obtained through objective standardized testing, it is much more difficult to capture fuzzier, or tacit, performance assessment measures. The problem of tacit knowledge capture is a central theme in the field of knowledge management, and assessment management can be thought of as a form of knowledge management. Therefore, tacit assessment management can be facilitated through technologies commonly used in knowledge management systems such as databases, Internet architectures, artificial intelligence, and decision support techniques. This paper describes tacit performance assessment in the context of knowledge management and presents a prototype decision support system for managing tacit assessment knowledge using knowledge management techniques. # 2003 Elsevier Ltd. All rights reserved. Keywords: Architectures for educational technology system; Evaluation methodologies; Cooperative/collaborative learning; Authoring tools and methods; Post-secondary education
1. The need for tacit performance assessment Educational quality has become a major topic in recent years. This has led to political ramifications, such as President Bush’s call for increased accountability in primary and secondary schools. In addition, accreditation agencies such as AASCB require systematic monitoring of university degree programs, with measurable assessment results leading to program revisions (AACSB, 2001). Education providers have found this to be a particularly time-consuming and * Tel.: +1-540-568-3019; fax: +1-540-568-3017. E-mail address: [email protected] (M. Mitri). 0360-1315/03/$ - see front matter # 2003 Elsevier Ltd. All rights reserved. doi:10.1016/S0360-1315(03)00034-4
174
M. Mitri / Computers & Education 41 (2003) 173–189
labor-intensive task, and efforts often get bogged down due to political opposition, bureaucratic inertia, and faculty disinterest. Proposed solutions to this problem often emphasize standardized testing. But standardized tests usually fail to capture higher-order learning objectives. For example, consider Boom’s six levels of cognitive learning objectives (Bloom, 1956). These include: content knowledge (simple recall and recognition), comprehension (understanding/interpretation/paraphrasing), application (of principles to concrete situations), analysis (clarification/ decomposition), synthesis (creation of new patterns or structure), and evaluation (value judgment). Of these, objective tests (which are easily automated) are most effective at measuring lower-order skills such as content knowledge and comprehension (Scouller & Prosser, 1994). But assessment of higher-order tasks, such as application, analysis, synthesis, and evaluation, requires significant faculty involvement, and is much more difficult to automate. In essence, the assessment of these skills require faculty to use intuition, judgment, and feeling. Much more thought must go into this type of evaluation, which we will term tacit assessment, than is necessary for judging students’ knowledge of simple facts or principles (explicit assessment). Tacit assessment is the area in which curriculum assessment efforts often encounter roadblocks. Yet, it is this type of evaluation that is most likely to truly measure the effectiveness of an educational system. Anyone can spit out facts and figures. It takes true understanding and even wisdom to apply these facts to real-world problems, and this is the skill that faculty hope to impart on their students. How can an institution develop a systematic methodology for measuring a type of knowledge that seems to be inherently unmeasurable, or at least unquantifiable?
2. How knowledge management techniques can support tacit assessment Fortunately, there is a recent stream of research in the information technology sector that may prove helpful to the problem of assessing higher-order learning. The field of knowledge management (KM) is exploring methods of discovering, codifying, storing, and automating knowledge (Alavi & Leidner, 1999; Gregory, 2000) that bears significant resemblance to the problems stated above. The information gleaned from a university’s assessment procedures can certainly be thought of as knowledge to be managed, in the sense that it resides in the minds and documents of multiple members of the organization. Furthermore, it is the type of knowledge that can greatly benefit from KM techniques, because it is actionable (Sallis & Jones, 2002). Actionable knowledge is knowledge that is applied toward solving real-world problems. This is contrasted with knowledge gained solely for the purpose of increasing a person’s understanding or expertise. In the case of performance assessment, the problem-solving efforts are two-fold: (1) encouraging students to improve their study habits and (2) making corrective adjustments to curricula in order to improve the quality of education. Of particular relevance to the problems stated in the previous section, a major focus of KM involves capturing tacit knowledge and transforming it into explicit knowledge. Explicit knowledge can be defined as that which is easily expressible, verbal, and simple to codify, whereas tacit knowledge involves more gut feeling, experience, and intuition, and is therefore much more difficult to articulate and express to others (Nonaka & Takeouchi; 1995, Sallis & Jones, 2002). The type of assessment concerned with Bloom’s higher-order objectives can be considered to be tacit
M. Mitri / Computers & Education 41 (2003) 173–189
175
assessment because it is not easily represented in highly structured formats conducive to multiplechoice, standardized tests. The conundrum faced by educators when performing assessment is how to quantify or at least codify their perceptions, beliefs, and perspectives of students’ subjective expertise, mental models, and interpersonal skills. As mentioned earlier, this is the reason that assessment of higher-order skills is so time-consuming, which therefore tends to make widespread assessment efforts an anathema to many faculty members. KM’s potential to aid in performance assessment has been addressed by other KM authors. Many authors describe KM as a technique for facilitating learning in organizations (Gregory, 2000; Senge, 1990). Frequently KM is discussed in terms of auditing the quality of learning and/ or providing corrective feedback (Argyris, 1980; Garvin, 1993). KM techniques have been directly associated with Bloom’s hierarchy of learning objectives (Rademacher, 1999), and of course these objectives form the backbone of many assessment efforts (Miller, Imrie, & Cox, 1998; Nordvall & Braxton, 1996). KM’s direct applicability to improvement of curriculum development efforts have also been documented (Nordvall and Braxton, 1996). An important theme in KM involves the concept of double-loop learning, in which organizations make fundamental structural and policy changes based on the information gained in the KM practices. This is contrasted with single-loop learning, in which minor modifications to the knowledge base are performed without altering underlying structures and assumptions. Again, this applies directly to the problem of educational assessment, which as mentioned above, is utilized both for the purpose of improving student performance (single-loop learning) and revising curricula (double-loop learning), as illustrated in Fig. 1.
3. Applying information technology to tacit assessment The central problem being addressed in this article involves capture, representation, and effective use of performance assessments for higher-order learning tasks; assessments which by their nature are difficult to codify and which do not easily lend themselves to standardized objective testing methods. In the previous section, this problem was shown to be analogous to efforts in the
Fig. 1. Performance assessment as a contributor to study practice (single-loop learning) and curriculum development (double-loop learning).
176
M. Mitri / Computers & Education 41 (2003) 173–189
knowledge management community for capturing tacit knowledge and transforming it into explicit knowledge. This section describes the information technologies that are utilized in such capture and transformation efforts. In the sections that follow, a computer system is described which applies these technologies to the problem of tacit performance assessment. Knowledge management systems (KMS) typically utilize familiar technologies, including: telecommunications tools such as the internet, WWW, FTP, email, search engines and portals (Black, 2000; Garvin, 1993); data storage mechanisms such as relational and object-oriented databases and document management systems; and technologies for supporting human decisionmaking such as expert systems and decision support systems (Sahasrabudhe, 2000). Some of these technologies (e.g. email, video conferencing, collaborative work systems) are applied to capturing and sharing tacit knowledge. Others (knowledge acquisition, decision support, expert systems) are applied to transforming tacit knowledge into explicit knowledge. And still others (search engines, databases, document management systems) are used for acquiring, codifying, storing and disseminating explicit knowledge (Nemati, Steiger, Lakshmi, & Herschel, 2002). The application of technologies for different types of knowledge and knowledge transformation can be characterized in this way. Tacit knowledge is inherently unstructured, and its capture is typically supported by technologies that provide free-form communication and data input capabilities (writing, drawing, talking, networking). Explicit knowledge is highly structured, so the technologies applied are more codified (relational database models, object-oriented classifications, hyperlink and keyword representations, data structures). The transformation of tacit to explicit knowledge requires systems that straddle both the unstructured and structured domains of information processing; these have traditionally been addressed by decision support and artificial intelligence techniques. Indeed, AI has a large role to play in most KM systems (Liebowitz, 2001). Knowledge acquisition techniques such as interviewing, protocol analysis, and personal construct theory are useful for eliciting tacit knowledge from domain experts. Knowledge discovery techniques involving data mining and induction can help to create new knowledge by finding patterns and relationships in data repositories. Knowledge ontologies and representations such as rules, cases, frames/ objects, and semantic networks are useful for knowledge codification. Intelligent agent technology facilitates search and retrieval. Natural language and speech understanding front ends improve query tools for KM systems. In the context of tacit assessment management, AI and DSS technologies are particularly promising. Knowledge representations including rule-bases (O’Leary & Selferidge, 2000), semantic networks (Kuwata & Yatsu, 1997), and frame representations (Gaines & Shaw, 1993) have been utilized in KM systems, and can provide a qualitative representation of tacit assessment knowledge. Multi-attribute utility models, commonly used in decision support technologies, can contribute quantitative assessment codifications, effectively creating scores and weights that can be used to measure performance. The challenge being addressed in this research, then, is to develop an information architecture that allows users to enter unstructured, tacit assessments (in the form of written paragraphs of text), and associate these unstructured assessments to structured ontological categories and quantitative scores. This data entry process then leads to data structuring utilizing AI knowledge representations, inference using AI reasoning engines, and scoring using decision support formulae. It is through this process that the tacit-to-explicit assessment transformation will be
M. Mitri / Computers & Education 41 (2003) 173–189
177
accomplished. Once tacit assessments are encoded in more explicit forms, they can be shared, manipulated, and mined for the purpose of supporting both remedial services of individual students and curriculum enhancement in educational institutions. The following sections describe the theoretical underpinnings of this architecture and present a prototype system implementation.
4. SEMNET-MAU: combining semantic networks with multi-attribute utility models In the following section, a prototype system called ASSESS is described, with which evaluators can perform tacit assessment on students in their curricula. The architecture underlying this system is based on a combination of two technologies: semantic network knowledge representations and multi-attribute utility models. This combination is referred to as SEMNET-MAU (Mitri, 1995). A semantic network is a form of knowledge representation based on a graph structure of nodes and links. Nodes represent objects in the world and links represent relationships between these objects. Semantic networks are useful representations in tacit assessment domains for two reasons. First, they explicitly represent semantics or meaning by describing relationships between individual objects and concepts. Second, they allow inferences about knowledge that may not be explicitly represented, via a mechanism called spreading activation, a process in which the ‘‘attention’’ or ‘‘focus’’ of the inference travels from node to node via the links that connect them. This fosters ‘‘reasoning by association,’’ where associations are the links in the network. For a tacit assessment system, this kind of reasoning can provide intuitive connections between seemingly disparate judgments, as will be shown later. Semantic networks were first developed as models of human memory and natural language representations (Quillian, 1985; Schank & Rieger, 1974). Subsequently, their use was extended to include knowledge representations used in expert systems (Brachman, 1979) and database indexing (Cohen & Kjeldsen, 1987; Roussopoulos & Mylopoulos, 1975). More recently, semantic network representations have been applied in knowledge management systems (Gaines & Shaw, 1997; Kuwata & Yatsu, 1997), as well as many educational applications (Lambiotte, Dansereau, Cross, & Reynolds, 1989; Liu, 1994). Thus, there is significant precedent that supports the application of this technology to the problem of tacit assessment in educational institutions. Decision models involving evaluation and assessment processes are frequently implemented via weighted algebraic expressions such as the additive linear model shown below: Y¼
n X i¼1
WX i i
where Y is the final judgment score, the Xis are the scores of criteria, and Wis are the weights of importance assigned to the criteria. Linear models are effective measures of human performance in many domains (Dawes, 1988; Slovic and Lichtenstein, 1971), particularly when attributes or criteria have conditionally monotone relationships with the final score; therefore, they are ubiquitous in the decision sciences literature. One often-used version of weighted additive modeling is the Analytic Hierarchy Process
178
M. Mitri / Computers & Education 41 (2003) 173–189
(Saaty, 1980), implemented in the software Expert Choice. AI systems also make frequent use of multi-attribute utility models, in domains as diverse as game-playing (Berliner, 1979), civil engineering (Maher, 1988), financial risk assessment (Pinson, 1991), and international marketing (Mitri, 1995). The potential contributions of semantic networks for knowledge management and multi-attribute utility models for evaluation and assessment motivate a merger of these methods for the purpose of assessment management. This merger results in a model called SEMNET-MAU (Mitri, 1995). The SEMNET-MAU model provides a mechanism for linking judgments to combinations of concepts that are in turn associated with each other through relationship links. The model includes the following components: concept types, concepts, relationship links, judgments, and inference strategies. Because of the association of judgments with conceptual networks, and the merger of semantic net and multi-attribute models, SEMNET-MAU provides an inference capability that allows prediction of tacit judgments in the absence of explicit judgments, as described below. Concepts are nodes in the semantic network, and pertain to the objects being represented in the database. Concepts are grouped into concept types, which each pertain to a general category of concept. In the context of the ASSESS prototype, there are two concept types: topics and skills; these will be described later in the paper. A user’s database query consists of a concept combination, which includes one concept from each concept type. In addition, each judgment in the database is uniquely identified by a single concept combination. The semantic relationships between the concepts of a concept type are represented via relationship links. Several types of relationship links may exist, but the most obvious one is a parent– child link. This describes a tree structure of concepts, where general concepts appear at the top and more specific concepts appear at lower levels. This kind of link is often called an ‘‘is-a’’ link in semantic net jargon. Another possible link is a similarity link that associated concepts in different hierarchies based on their similarity to each other. In principle, there is no limit to the types and numbers of links that can exist between concepts. The evaluative nature of the SEMNET-MAU model is represented via judgment records. These kinds of records contain several fields for representing an evaluation. One field contains a score between 0 and 100. Each judgment also includes a confidence-level, indicating the certainty of the evaluation. The evaluation and confidence level are used for inference purposes, as described below. A comment field allows evaluators to enter free-form text pertaining to the judgment. Judgments are also time-stamped with the date that the judgment was made. Because judgments include quantitative scores and are associated with concepts structured in a semantic network, SEMNET-MAU is capable of performing inferential assessment by combining spreading activation with multi-attribute calculations. Inference in SEMNET-MAU is used to predict or estimate judgment values based on other related judgments. An example of this will be described later. The inference mechanism is based on the following principles: 1. Conceptual proximity: judgments that are ‘‘conceptually close’’ to the user’s query have more influence than those that are ‘‘conceptually far’’. 2. Temporal proximity: judgments that are ‘‘temporally close’’ to the user’s query have more influence than those that are ‘‘temporally far’’.
M. Mitri / Computers & Education 41 (2003) 173–189
179
3. Confidence: judgments with higher confidence levels have more influence than those with low confidence levels. The principle of confidence is implemented through the confidence levels that evaluators select when entering their judgments. If an evaluator is highly certain of the accuracy of her judgment, she will enter a high confidence level (in the 80–100 range). If an evaluator is very unsure of the judgment’s accuracy, she will enter a low confidence level (10–30). These confidence levels will be used in calculating the relative weights of judgments when inference processing occurs, based on formula (3) below. To enforce the principles of conceptual and temporal proximity, an inference strategy specifies a scope of inference that constrains the number of steps allowed for the spreading activation search, and thus defines conceptual and temporal ‘‘boundaries’’ of the spreading activation search. This guarantees that judgments that are too far removed conceptually and temporally from the query will not be included in the search. In addition, the inference strategy specifies conceptual and temporal attenuation factors, which determine the dampening of effect that occurs as the search gets further and further from the concepts or time frame of a user’s query. The attenuation factors ensure that closely related judgments have the strongest impact on the inference. Formulas (1) and (2) below implement these impacts. Inference processing in the SEMNET-MAU model takes place when a user initiates a query looking for judgments pertaining to a particular concept combination. A SEMNET-MAU system performs a search using spreading activation, bounded by the scope of inference, which results in a list of related judgments. These related judgments are then assigned comparative weights based on the attenuations that are applied to them. This process involves four main steps: First, attenuation values are calculated for the concepts within the inference strategy’s scope in each conceptual category. This is how the principle of conceptual proximity is implemented. These values are determined by raising the attenuation factor (a number between 0 and 1) for a relationship to an exponent value based on the number of steps between the inferred concept and the concept found from the spreading activation, as in this expression: AttenuationValue ¼ AttenuationFactorSTEPS
ð1Þ
Similarly, temporal attenuation is calculated for each judgment located via the spreading activation; in this case the STEPS exponent refers to the number of time units (e.g. days) between the judgment being inferred and the related judgment located, thus implementing the principle of temporal proximity. The smaller the attenuation factor, the greater the dampening of influence as distance increases. For example, an attenuation factor of 0.5 causes the attenuation value to be 0.5 at one step away, 0.25 and 2 steps away, and 0.125 at three steps away. On the other hand, an attenuation factor of 0.9, causes the attenuation value to be 0.9 at one step away, 0.81 at two steps away, and 0.729 at three steps away. Second, attenuation values of concepts, along with temporal attenuation values are combined to form a judgment attenuation value for each judgment. Each judgment’s attenuation value is calculated as the product of its conceptual and temporal attenuations: JudgmentAttenuationValue ¼ ðConceptAttenuationValuei Þ TemporalAttenValue
ð2Þ
180
M. Mitri / Computers & Education 41 (2003) 173–189
Once judgment attenuations have been calculated, they are combined with judgment confidences and normalized into weights according to the following formula: Judgment Attenuation Valuei Judgment Confidencei Judgment Weighti ¼ Pn j Judgment Attenuation Valuej Judgment Confidencej
ð3Þ
where n is the total number of judgments found via the spreading activation process. Here, the principle of proximity (temporal and conceptual), which were implemented in steps (1) and (2) is combined with the principle of confidence in order to determine the weight of influence that each related judgment will have when estimating the inferred judgment value. Finally, the inferred judgment score is determined using a weighted sum, thus implementing the multi-attribute utility portion of SEMNET-MAU:
Inferred Judgment Score ¼
n X Judgment Scorei Judgment Weighti
ð4Þ
i1
These calculations will be demonstrated in later sections of this article.
5. Applying SEMNET-MAU to tacit assessment management The remainder of this paper describes a specific implementation of SEMNET-MAU in a prototype tacit assessment management system called ASSESS. The following sections describe the architecture and algorithms involved in ASSESS, and illustrate typical user interactions and screen images. This primary purpose of ASSESS is to input tacit assessments of student performance from evaluators and transform these tacit assessments to explicit assessments via the structures of the judgment and semantic network framework; in this way it serves as a knowledge codification and management tool. In addition, ASSESS makes predictions by using spreading activation and multi-attribute inference algorithms, thus serving as a knowledge discovery tool. It also serves as a knowledge dissemination tool by providing a variety of query techniques and data visualizations for viewing and manipulating data. Because it is designed for use by many evaluators, ASSESS serves as a group decision support system, facilitating collaboration assessment efforts. With ASSESS, students are judged according to their performance in a particular topic using a particular skill. For example, a student may be judged for his or her oral communication skills with regard to the topic of systems analysis. Alternatively, an evaluator may judge a student on his or her computation skills in the accounting area. Judgments are tacit; that is, they involve entry of the evaluator’s perceptions in a free-form text field. Judgments are also explicit in that the evaluator enters a specific score for a judgment and assigns it to specific topic and skill keywords.
M. Mitri / Computers & Education 41 (2003) 173–189
181
6. Stepping through ASSESS 6.1. Developing the topics and skills ontology Curriculum developers and assessment modelers can begin with the ASSESS tool to generate the ontology of topics and skills upon which evaluations are to be performed. This ontology is implemented as a semantic network, enabling the SEMNET-MAU processes to conduct inferential evaluations when needed. The structure of the semantic network also provides a conceptual framework with which evaluators can organize their assessments. Fig. 2 shows the screen used for ontology development. This screen provides the ability for users to insert concepts from the two main concept types of ASSESS, topics and skills. In this figure, the topics and skill sets are both displayed in terms of their parent–child relationships. As described earlier, any number of relationships can exist between concepts of a concept type. In ASSESS, there is also a similarity relationship that holds between concepts from different ancestral paths in the parent-child hierarchy. The screen shown in Fig. 2 provides an authoring interface that allows curriculum designers to describe the learning objectives to assess. 6.2. Creating a judgment Evaluators create judgments by identifying the student to be judged, and the topic and skill set that the judgment is for. The evaluator then indicates a number from 0 to 100 for the score, and selects a confidence level. Recall from the discussion of SEMNET-MAU that the confidence level is used to assist in inference processing. In addition, system users can view a judgment’s confidence level in order ascertain how much credence to give a particular evaluation. The evaluator also types in a comment or explanation of the judgment. Although the comment does not have any bearing in score calculations or inference processing, it provides valuable tacit knowledge to users, and essentially forms the justification for a judgment. All entered judgments are timestamped and stored in the database. The screen for this process is shown in Fig. 3.
Fig. 1. Performance assessment as a contributor to study practice (single-loop learning) and curriculum development (double-loop learning).
182
M. Mitri / Computers & Education 41 (2003) 173–189
Fig. 2. Author screen for developing ontology of topics and skill sets.
Fig. 3. Screen for entering an evaluation.
M. Mitri / Computers & Education 41 (2003) 173–189
183
6.3. Performing standard queries Users can perform standard database queries in ASSESS. For example, in Fig. 4 the top left portion of the screen indicates a query of all judgments from evaluators in the CIS/OM program for all computer information students on the topic Programming. The list in the upper right portion shows all the judgments satisfying the query, in descending order by date. The bottom right portion shows details of the selected judgment from the list, which can be altered or deleted. 6.4. Visualizing data and analyzing performance trends In order to support decision-making in an assessment environment, ASSESS includes a graphic visualization feature with seamless integration between differing views of the same data, and includes drill-down capabilities for summary-to-detail transitions. Fig. 5 shows a grid/chart view whereby evaluators can filter data according to any combination of topics, skills, students and/or majors, evaluators and/or programs and date ranges, and view a pictorial (chart or grid) view summarizing the assessment results. In the example of Fig. 5, we see a summary of a particular student’s trend of judgments over the academic year on a month by month basis for four major topics: database, programming, networks, and systems analysis. This trend shows improvement in database scores, steadily high scores in systems analysis, low programming scores, and deterioration of network performance.
Fig. 4. A standard assessment query.
184
M. Mitri / Computers & Education 41 (2003) 173–189
Fig. 5. Graphical view of one student’s performance over time.
A user can view the specific judgments for a particular data point in the chart by clicking on the data point itself. This drill-down process takes the user directly to the list of judgments, as shown in Fig. 6. Here we see the screen that results when the user clicked on the lowest data point in the month of April 2002 for the ‘‘network’’ topic, which consists of three judgments made in that month. By selecting a judgment from the list, the student can see the details of the judgment. Fig. 7 shows an aggregation of judgments for all IS-related courses during the academic year, displaying the aggregation by topic area and skill set. This illustrates the variety of filters and associations that can be displayed in the graph view. ASSESS helps evaluators ascertain common
Fig. 6. Drilling down to see details of judgments for a particular topic and time period.
M. Mitri / Computers & Education 41 (2003) 173–189
185
Fig. 7. Viewing trends in student performance as aggregate organized by topic and skill set.
strengths and weaknesses of all students, thus supporting faculty efforts for improving the quality of education at their institution. 6.5. Assessment knowledge discovery through inference ASSESS includes an inference algorithm based on the SEMNET-MAU model described earlier. This helps predict student performance on topics and skill sets when no explicit judgment exists. In ASSESS, the semantic network is composed of two conceptual categories: topics and skill sets. Each conceptual category includes a number of concepts related to each other via parent–child (hierarchy) links and similarity (cross-hierarchy) links. Recall that ASSESS provides a screen for managing these topics, as shown in Fig. 2. As described earlier, the inference algorithm estimates a judgment based on the principles of temporal and conceptual proximity and judgment confidence, implemented via an inference strategy that determines the scope of inference and the attenuation factors. This is shown in Fig. 8. In this case the user performed a query searching for a judgment of the student by the evaluator for the skill of written communication in the topic of programming. No judgment was found. So the user requested an inference by clicking the Inference button (lower left quadrant of the screen). The lower left quadrant of the screen also shows the inference strategy, the scope of inference and the attenuation factors. As shown in Fig. 2, the topic called ‘‘programming’’ is a direct descendent of ‘‘information systems’’, which is in turn a direct descendent of ‘‘business knowledge’’. Thus, as shown in Fig. 8, the spreading activation has caused the topics scope of inference, being bounded by a steps limitation of 2, to include programming, information systems, and business knowledge as the allowable topics to search. In addition, the skill ‘‘written communication’’ has a parent–child link to ‘‘communication’’, and a similarity link to ‘‘oral communication’’. Thus, these three skill sets are found by spreading activation, based on the parent–child and similarity linkages and the step constraints of the relationship types in the inference strategy. Therefore, for this inference,
186
M. Mitri / Computers & Education 41 (2003) 173–189
Fig. 8. The result of an inferential evaluation via SEMNET-MAU model.
spreading activation in the semantic network has produced nine possible combinations of the three topics and the three skill sets within the scope of inference. Fig. 8 also shows the temporal steps constraint of 9999 days, which in effect places no constraint on the time period. Note that the user is able to set the steps constraint to his or her discretion. The top right quadrant of Fig. 8 shows that the inference algorithm has constructed a query using these nine topic-skill combinations to obtain every judgment satisfying the step constraints. The query has resulted in four actual judgments. These judgments are ranked based on their relative weights, which were determined based on formulas of the SEMNET-MAU model. Note that the inference strategy shown in Fig. 8 displays the attenuations associated with the topics and skill sets via different types of relationship links. For parent–child links the attenuation factor is 0.9 and for similarity links the attenuation factor is also 0.9. In addition, the temporal attenuation factor is set at 0.9999. Users can set these attenuation factors (which should be a number between 0 and 1) at their discretion to affect the inference results. From formula (1) above, the attenuation values for the topics and skills within the scope were calculated. For example, the business knowledge attenuation value is 0.81. This is because the number parent-child attenuation factor is 0.9 and there are two steps between programming and business knowledge, hence the conceptual attenuation value for business knowledge is 0.92=0.81. These conceptual attenuations, along with temporal attenuations and confidence levels were combined in formula (3) to determine the weight of each judgment, shown in the list of the top right quadrant of Fig. 8.
M. Mitri / Computers & Education 41 (2003) 173–189
187
The inferred judgment value is shown in the bottom left quadrant of Fig. 8, near the Inference button. This value of 60 was determined based on formula (4). In this way new knowledge was discovered where none existed before, another knowledge management contribution of the system.
7. Conclusion This paper argues that performance assessment is an activity that can benefit from knowledge management techniques. In particular, tacit assessment, which involves evaluator intuition and judgment rather than objective standardized testing, and which therefore is considerably harder to codify and manage, is a prime candidate for the types of knowledge management that involve transformation from tacit to explicit knowledge. Considering the wealth of useful information in tacit assessments, and the importance of this information for the purposes of student remediation and curriculum design, there is sufficient motivation to warrant an exploration of applying knowledge management principles to tacit assessment. The assessment management system described in this article is one implementation of this application of knowledge management to tacit assessment. ASSESS organizes knowledge through an ontology of topics and skill sets via a semantic network. It provides an input mechanism for evaluators to create tacit judgments and transform them to explicit judgments based on the structure of judgment records and their relationships to concepts in the semantic network. ASSESS provides views of summary data and allows drill-down transformations from general to specific information. Knowledge discovery is supported through the use of spreading activation and multi-attribute utility calculations for the purpose of inferring judgments. ASSESS offers a potential solution to difficult assessment problems, and illustrates the applicability of knowledge management to higher education. In its prototype form, the system is written in MS Access and VBA. This proof-of-concept system next will be tested in a classroom setting. Beyond that, several enhancements to the system are being considered. These include providing automatic links to exam results, tutorial components, and eLearning modules. Data mining and visualization techniques can be extended, and collaborative filtering techniques can be used to provide enhanced functionality.
References Alavi, & Leidner, D. E. (1999). Knowledge management systems: issues, challenges, and benefits, Communications of the AIS, 1, No. 7. Argyris, C., & Schon, D. (1978). Organizational learning: a theory of action perspective. Reading, PA: Addison Wesley. Association to Advance Collegiate Schools of Business (AACSB) (2001), Standards for business accreditation. Available: www.aacsb.edu. Berliner, H. (1979) On the construction of evaluation functions for large domains. Proc. Intl. Joint Conf on Artificial Intelligence. pp 53–55. Black, K. (2000). Internet search engines and knowledge management. In T. K. Srikantaiah, & M. E. D. Koenig (Eds.), Knowledge management for the information professional (pp. 255–268). Medford, NJ: Information Today, Inc.
188
M. Mitri / Computers & Education 41 (2003) 173–189
Bloom, B. S. (1956). Taxonomy of educational objectives handbook I: cognitive domain. New York: David McKay Co 255-268. Brachman, R. J. (1979) On the epistemological status of semantic networks. In: Brachman and Levesque (Eds.) Readings in Knowledge Representation. Morgan Kaufmann Publishers, Inc. 1985. pp 191–216. Cohen, P., & Kjeldsen, R. (1987) Information retrieval by constrained spreading activation in semantic networks. COINS Technical Report 87-66. University of Massachusetts at Amherst. Dawes, R. (1988). Rational choice in an uncertain world. Orlando, FL: Harcourt Bruce Jovanovich Publishers, Inc. Gaines, B. R., & Shaw, M. L. G. (1993). Open architecture multimedia documents. Proceedings of ACM Multimedia, 93, 137–146. Gaines, B. R., & Shaw, M. L. G. (1997), Knowledge management for research communities. Proceedings of the AAAI Spring Symposium on Artificial Intelligence in Knowledge Management, Stanford University, (March 24–26, 1997), pp 55–62. Garvin, D. A. (1993). Building a learning organization. Harvard Business Review, 72(4), 78–86. Gregory, V. (2000). Knowledge management and building the learning organization. In: T. K. Srikantaiah, & M. E. D. Koenig (Eds.), Knowledge management for the information professional information today, Inc. Medford, NJ 2000 pp 161–179. Kuwata, Y., Yatsu, M. (1997), Managing knowledge using a semantic-network. In: Proceedings of the AAAI Spring Symposium on Artificial Intelligence in Knowledge Management, Stanford University, (March 24–26, 1997), pp 94–98. Lambiotte, J. G., Dansereau, D. F., Cross, D. R., & Reynolds, S. B. (1989). Multirelational semantic maps. Educational Psychology Review, 1(4), 331–367. Liebowitz, J. (2001). Knowledge management and its link to artificial intelligence. Expert Systems with Applications, 20(1), 1–6. Liu, M. (1994). Hypermedia-assisted-instruction and second language learning: a semantic-network-based approach. Computers in the Schools, 10(3), 293–312. Maher, M. (1988). HI-RISE: and expert system for preliminary structural design. In M. D. Rychener (Ed.), Expert systems for engineering design (pp. 37–52). USA: Academic Press, Inc. Miller, A. H., Imrie, B. W., & Cox, K. (1998). Student assessment in higher education: a handbook for assessing performance. London: Kogan Page, Ltd. Mitri, M. (1995). Combining semantic networks with multi-attribute utility models: an evaluative database indexing method. Expert Systems with Applications: An International Journal, 9(3), 283–294. Nemati, H. R., Steiger, D. M., Lakshmi, S. I., & Herschel, R. T. (2002). Knowledge warehouse: an architectural integration of knowledge management, decision support, artificial intelligence and data warehousing. Decision Support Systems, 33(2), 143–161. Nonaka, I., & Takeouchi, H. (1995). The knowledge-creating company. NY: Oxford University Press. Nordvall, R. C., & Braxton, J. M. (1996). An alternative definition of quality of undergraduate college education: toward usable knowledge for improvement. Journal of Higher Education, 67(5), 483–497. O’Leary, D. E., & Selferidge, P. (2000), Knowledge management for best practices. Communications of the ACM, 43(11). Pinson, S. (1991). Integrating surface knowledge and multiattribute decision models in a credit risk assessment system. AAAI-1991 AI in Business Workshop, Anaheim, CA. Quillian, M. R. (1967) Word concepts: a theory and simulation of some basic semantic capabilities. In: Brachman and Levesque (Eds.), Readings in knowledge representation. Morgan Kaufmann Publishers, Inc. 1985. pp 97–118. Rademacher, R. (1999), Applying bloom’s taxonomy of cognition to knowledge management systems, SIGCPR ’99, New Orleans, pp. 276–278. Roussopoulos, N., & Mylopoulos, J. (1975) Using semantic networks for database management. In: Mylopolopus, & Brodie (Eds.), Readings in artificial intelligence and databases. Morgan Kaufmann Publishers, Inc., pp. 112–137. Saaty, T. L. (1980). The analytic hierarchy process. New York: McGraw-Hill. Sahasrabudhe, V. (2000). Information technology in support of knowledge management. In T. K. Srikantaiah, & M. E. D. Koenig (Eds.), Knowledge management for the information professional (pp. 255–268). Medford, NJ: Information Today, Inc. Sallis, E., & Jones, G. (2002). Knowledge management in education. London: Kogan Page.
M. Mitri / Computers & Education 41 (2003) 173–189
189
Schank, R. C., & Rieger, C. J. (1974) Inference and computer understanding of natural language. In: Brachman and Levesque (Eds.), Readings in knowledge representation. Morgan Kaufmann Publishers, Inc. 1985. pp 119–140. Scouller, K. M., & Prosser, M. (1994). Students’ experiences in studying for multiple-choice question examinations. Studies in Higher Education, 19, 267–279. Senge, P. M. (1990). The fifth discipline: the art and practice of the learning organization. NY: Doubleday/Currency. Slovic, P., & Lichtenstein, S. (1971). Comparison of bayesian and regression approaches to the study of information processing in judgement. Organizational Behavior and Human Performance, 6, 649–744.
Applying tacit knowledge management techniques for performance assessment Michel Mitri* James Madison University, CIS/OM Program, College of Business, 309 Zane Showker Hall, Harrisonburg, VA 22807, USA Received 1 December 2002; accepted 25 March 2003
Abstract Performance assessment is an important task in all levels of education, both as input for identifying remedial needs of individual students and for improving general quality of education. Although explicit assessment measures can be obtained through objective standardized testing, it is much more difficult to capture fuzzier, or tacit, performance assessment measures. The problem of tacit knowledge capture is a central theme in the field of knowledge management, and assessment management can be thought of as a form of knowledge management. Therefore, tacit assessment management can be facilitated through technologies commonly used in knowledge management systems such as databases, Internet architectures, artificial intelligence, and decision support techniques. This paper describes tacit performance assessment in the context of knowledge management and presents a prototype decision support system for managing tacit assessment knowledge using knowledge management techniques. # 2003 Elsevier Ltd. All rights reserved. Keywords: Architectures for educational technology system; Evaluation methodologies; Cooperative/collaborative learning; Authoring tools and methods; Post-secondary education
1. The need for tacit performance assessment Educational quality has become a major topic in recent years. This has led to political ramifications, such as President Bush’s call for increased accountability in primary and secondary schools. In addition, accreditation agencies such as AASCB require systematic monitoring of university degree programs, with measurable assessment results leading to program revisions (AACSB, 2001). Education providers have found this to be a particularly time-consuming and * Tel.: +1-540-568-3019; fax: +1-540-568-3017. E-mail address: [email protected] (M. Mitri). 0360-1315/03/$ - see front matter # 2003 Elsevier Ltd. All rights reserved. doi:10.1016/S0360-1315(03)00034-4
174
M. Mitri / Computers & Education 41 (2003) 173–189
labor-intensive task, and efforts often get bogged down due to political opposition, bureaucratic inertia, and faculty disinterest. Proposed solutions to this problem often emphasize standardized testing. But standardized tests usually fail to capture higher-order learning objectives. For example, consider Boom’s six levels of cognitive learning objectives (Bloom, 1956). These include: content knowledge (simple recall and recognition), comprehension (understanding/interpretation/paraphrasing), application (of principles to concrete situations), analysis (clarification/ decomposition), synthesis (creation of new patterns or structure), and evaluation (value judgment). Of these, objective tests (which are easily automated) are most effective at measuring lower-order skills such as content knowledge and comprehension (Scouller & Prosser, 1994). But assessment of higher-order tasks, such as application, analysis, synthesis, and evaluation, requires significant faculty involvement, and is much more difficult to automate. In essence, the assessment of these skills require faculty to use intuition, judgment, and feeling. Much more thought must go into this type of evaluation, which we will term tacit assessment, than is necessary for judging students’ knowledge of simple facts or principles (explicit assessment). Tacit assessment is the area in which curriculum assessment efforts often encounter roadblocks. Yet, it is this type of evaluation that is most likely to truly measure the effectiveness of an educational system. Anyone can spit out facts and figures. It takes true understanding and even wisdom to apply these facts to real-world problems, and this is the skill that faculty hope to impart on their students. How can an institution develop a systematic methodology for measuring a type of knowledge that seems to be inherently unmeasurable, or at least unquantifiable?
2. How knowledge management techniques can support tacit assessment Fortunately, there is a recent stream of research in the information technology sector that may prove helpful to the problem of assessing higher-order learning. The field of knowledge management (KM) is exploring methods of discovering, codifying, storing, and automating knowledge (Alavi & Leidner, 1999; Gregory, 2000) that bears significant resemblance to the problems stated above. The information gleaned from a university’s assessment procedures can certainly be thought of as knowledge to be managed, in the sense that it resides in the minds and documents of multiple members of the organization. Furthermore, it is the type of knowledge that can greatly benefit from KM techniques, because it is actionable (Sallis & Jones, 2002). Actionable knowledge is knowledge that is applied toward solving real-world problems. This is contrasted with knowledge gained solely for the purpose of increasing a person’s understanding or expertise. In the case of performance assessment, the problem-solving efforts are two-fold: (1) encouraging students to improve their study habits and (2) making corrective adjustments to curricula in order to improve the quality of education. Of particular relevance to the problems stated in the previous section, a major focus of KM involves capturing tacit knowledge and transforming it into explicit knowledge. Explicit knowledge can be defined as that which is easily expressible, verbal, and simple to codify, whereas tacit knowledge involves more gut feeling, experience, and intuition, and is therefore much more difficult to articulate and express to others (Nonaka & Takeouchi; 1995, Sallis & Jones, 2002). The type of assessment concerned with Bloom’s higher-order objectives can be considered to be tacit
M. Mitri / Computers & Education 41 (2003) 173–189
175
assessment because it is not easily represented in highly structured formats conducive to multiplechoice, standardized tests. The conundrum faced by educators when performing assessment is how to quantify or at least codify their perceptions, beliefs, and perspectives of students’ subjective expertise, mental models, and interpersonal skills. As mentioned earlier, this is the reason that assessment of higher-order skills is so time-consuming, which therefore tends to make widespread assessment efforts an anathema to many faculty members. KM’s potential to aid in performance assessment has been addressed by other KM authors. Many authors describe KM as a technique for facilitating learning in organizations (Gregory, 2000; Senge, 1990). Frequently KM is discussed in terms of auditing the quality of learning and/ or providing corrective feedback (Argyris, 1980; Garvin, 1993). KM techniques have been directly associated with Bloom’s hierarchy of learning objectives (Rademacher, 1999), and of course these objectives form the backbone of many assessment efforts (Miller, Imrie, & Cox, 1998; Nordvall & Braxton, 1996). KM’s direct applicability to improvement of curriculum development efforts have also been documented (Nordvall and Braxton, 1996). An important theme in KM involves the concept of double-loop learning, in which organizations make fundamental structural and policy changes based on the information gained in the KM practices. This is contrasted with single-loop learning, in which minor modifications to the knowledge base are performed without altering underlying structures and assumptions. Again, this applies directly to the problem of educational assessment, which as mentioned above, is utilized both for the purpose of improving student performance (single-loop learning) and revising curricula (double-loop learning), as illustrated in Fig. 1.
3. Applying information technology to tacit assessment The central problem being addressed in this article involves capture, representation, and effective use of performance assessments for higher-order learning tasks; assessments which by their nature are difficult to codify and which do not easily lend themselves to standardized objective testing methods. In the previous section, this problem was shown to be analogous to efforts in the
Fig. 1. Performance assessment as a contributor to study practice (single-loop learning) and curriculum development (double-loop learning).
176
M. Mitri / Computers & Education 41 (2003) 173–189
knowledge management community for capturing tacit knowledge and transforming it into explicit knowledge. This section describes the information technologies that are utilized in such capture and transformation efforts. In the sections that follow, a computer system is described which applies these technologies to the problem of tacit performance assessment. Knowledge management systems (KMS) typically utilize familiar technologies, including: telecommunications tools such as the internet, WWW, FTP, email, search engines and portals (Black, 2000; Garvin, 1993); data storage mechanisms such as relational and object-oriented databases and document management systems; and technologies for supporting human decisionmaking such as expert systems and decision support systems (Sahasrabudhe, 2000). Some of these technologies (e.g. email, video conferencing, collaborative work systems) are applied to capturing and sharing tacit knowledge. Others (knowledge acquisition, decision support, expert systems) are applied to transforming tacit knowledge into explicit knowledge. And still others (search engines, databases, document management systems) are used for acquiring, codifying, storing and disseminating explicit knowledge (Nemati, Steiger, Lakshmi, & Herschel, 2002). The application of technologies for different types of knowledge and knowledge transformation can be characterized in this way. Tacit knowledge is inherently unstructured, and its capture is typically supported by technologies that provide free-form communication and data input capabilities (writing, drawing, talking, networking). Explicit knowledge is highly structured, so the technologies applied are more codified (relational database models, object-oriented classifications, hyperlink and keyword representations, data structures). The transformation of tacit to explicit knowledge requires systems that straddle both the unstructured and structured domains of information processing; these have traditionally been addressed by decision support and artificial intelligence techniques. Indeed, AI has a large role to play in most KM systems (Liebowitz, 2001). Knowledge acquisition techniques such as interviewing, protocol analysis, and personal construct theory are useful for eliciting tacit knowledge from domain experts. Knowledge discovery techniques involving data mining and induction can help to create new knowledge by finding patterns and relationships in data repositories. Knowledge ontologies and representations such as rules, cases, frames/ objects, and semantic networks are useful for knowledge codification. Intelligent agent technology facilitates search and retrieval. Natural language and speech understanding front ends improve query tools for KM systems. In the context of tacit assessment management, AI and DSS technologies are particularly promising. Knowledge representations including rule-bases (O’Leary & Selferidge, 2000), semantic networks (Kuwata & Yatsu, 1997), and frame representations (Gaines & Shaw, 1993) have been utilized in KM systems, and can provide a qualitative representation of tacit assessment knowledge. Multi-attribute utility models, commonly used in decision support technologies, can contribute quantitative assessment codifications, effectively creating scores and weights that can be used to measure performance. The challenge being addressed in this research, then, is to develop an information architecture that allows users to enter unstructured, tacit assessments (in the form of written paragraphs of text), and associate these unstructured assessments to structured ontological categories and quantitative scores. This data entry process then leads to data structuring utilizing AI knowledge representations, inference using AI reasoning engines, and scoring using decision support formulae. It is through this process that the tacit-to-explicit assessment transformation will be
M. Mitri / Computers & Education 41 (2003) 173–189
177
accomplished. Once tacit assessments are encoded in more explicit forms, they can be shared, manipulated, and mined for the purpose of supporting both remedial services of individual students and curriculum enhancement in educational institutions. The following sections describe the theoretical underpinnings of this architecture and present a prototype system implementation.
4. SEMNET-MAU: combining semantic networks with multi-attribute utility models In the following section, a prototype system called ASSESS is described, with which evaluators can perform tacit assessment on students in their curricula. The architecture underlying this system is based on a combination of two technologies: semantic network knowledge representations and multi-attribute utility models. This combination is referred to as SEMNET-MAU (Mitri, 1995). A semantic network is a form of knowledge representation based on a graph structure of nodes and links. Nodes represent objects in the world and links represent relationships between these objects. Semantic networks are useful representations in tacit assessment domains for two reasons. First, they explicitly represent semantics or meaning by describing relationships between individual objects and concepts. Second, they allow inferences about knowledge that may not be explicitly represented, via a mechanism called spreading activation, a process in which the ‘‘attention’’ or ‘‘focus’’ of the inference travels from node to node via the links that connect them. This fosters ‘‘reasoning by association,’’ where associations are the links in the network. For a tacit assessment system, this kind of reasoning can provide intuitive connections between seemingly disparate judgments, as will be shown later. Semantic networks were first developed as models of human memory and natural language representations (Quillian, 1985; Schank & Rieger, 1974). Subsequently, their use was extended to include knowledge representations used in expert systems (Brachman, 1979) and database indexing (Cohen & Kjeldsen, 1987; Roussopoulos & Mylopoulos, 1975). More recently, semantic network representations have been applied in knowledge management systems (Gaines & Shaw, 1997; Kuwata & Yatsu, 1997), as well as many educational applications (Lambiotte, Dansereau, Cross, & Reynolds, 1989; Liu, 1994). Thus, there is significant precedent that supports the application of this technology to the problem of tacit assessment in educational institutions. Decision models involving evaluation and assessment processes are frequently implemented via weighted algebraic expressions such as the additive linear model shown below: Y¼
n X i¼1
WX i i
where Y is the final judgment score, the Xis are the scores of criteria, and Wis are the weights of importance assigned to the criteria. Linear models are effective measures of human performance in many domains (Dawes, 1988; Slovic and Lichtenstein, 1971), particularly when attributes or criteria have conditionally monotone relationships with the final score; therefore, they are ubiquitous in the decision sciences literature. One often-used version of weighted additive modeling is the Analytic Hierarchy Process
178
M. Mitri / Computers & Education 41 (2003) 173–189
(Saaty, 1980), implemented in the software Expert Choice. AI systems also make frequent use of multi-attribute utility models, in domains as diverse as game-playing (Berliner, 1979), civil engineering (Maher, 1988), financial risk assessment (Pinson, 1991), and international marketing (Mitri, 1995). The potential contributions of semantic networks for knowledge management and multi-attribute utility models for evaluation and assessment motivate a merger of these methods for the purpose of assessment management. This merger results in a model called SEMNET-MAU (Mitri, 1995). The SEMNET-MAU model provides a mechanism for linking judgments to combinations of concepts that are in turn associated with each other through relationship links. The model includes the following components: concept types, concepts, relationship links, judgments, and inference strategies. Because of the association of judgments with conceptual networks, and the merger of semantic net and multi-attribute models, SEMNET-MAU provides an inference capability that allows prediction of tacit judgments in the absence of explicit judgments, as described below. Concepts are nodes in the semantic network, and pertain to the objects being represented in the database. Concepts are grouped into concept types, which each pertain to a general category of concept. In the context of the ASSESS prototype, there are two concept types: topics and skills; these will be described later in the paper. A user’s database query consists of a concept combination, which includes one concept from each concept type. In addition, each judgment in the database is uniquely identified by a single concept combination. The semantic relationships between the concepts of a concept type are represented via relationship links. Several types of relationship links may exist, but the most obvious one is a parent– child link. This describes a tree structure of concepts, where general concepts appear at the top and more specific concepts appear at lower levels. This kind of link is often called an ‘‘is-a’’ link in semantic net jargon. Another possible link is a similarity link that associated concepts in different hierarchies based on their similarity to each other. In principle, there is no limit to the types and numbers of links that can exist between concepts. The evaluative nature of the SEMNET-MAU model is represented via judgment records. These kinds of records contain several fields for representing an evaluation. One field contains a score between 0 and 100. Each judgment also includes a confidence-level, indicating the certainty of the evaluation. The evaluation and confidence level are used for inference purposes, as described below. A comment field allows evaluators to enter free-form text pertaining to the judgment. Judgments are also time-stamped with the date that the judgment was made. Because judgments include quantitative scores and are associated with concepts structured in a semantic network, SEMNET-MAU is capable of performing inferential assessment by combining spreading activation with multi-attribute calculations. Inference in SEMNET-MAU is used to predict or estimate judgment values based on other related judgments. An example of this will be described later. The inference mechanism is based on the following principles: 1. Conceptual proximity: judgments that are ‘‘conceptually close’’ to the user’s query have more influence than those that are ‘‘conceptually far’’. 2. Temporal proximity: judgments that are ‘‘temporally close’’ to the user’s query have more influence than those that are ‘‘temporally far’’.
M. Mitri / Computers & Education 41 (2003) 173–189
179
3. Confidence: judgments with higher confidence levels have more influence than those with low confidence levels. The principle of confidence is implemented through the confidence levels that evaluators select when entering their judgments. If an evaluator is highly certain of the accuracy of her judgment, she will enter a high confidence level (in the 80–100 range). If an evaluator is very unsure of the judgment’s accuracy, she will enter a low confidence level (10–30). These confidence levels will be used in calculating the relative weights of judgments when inference processing occurs, based on formula (3) below. To enforce the principles of conceptual and temporal proximity, an inference strategy specifies a scope of inference that constrains the number of steps allowed for the spreading activation search, and thus defines conceptual and temporal ‘‘boundaries’’ of the spreading activation search. This guarantees that judgments that are too far removed conceptually and temporally from the query will not be included in the search. In addition, the inference strategy specifies conceptual and temporal attenuation factors, which determine the dampening of effect that occurs as the search gets further and further from the concepts or time frame of a user’s query. The attenuation factors ensure that closely related judgments have the strongest impact on the inference. Formulas (1) and (2) below implement these impacts. Inference processing in the SEMNET-MAU model takes place when a user initiates a query looking for judgments pertaining to a particular concept combination. A SEMNET-MAU system performs a search using spreading activation, bounded by the scope of inference, which results in a list of related judgments. These related judgments are then assigned comparative weights based on the attenuations that are applied to them. This process involves four main steps: First, attenuation values are calculated for the concepts within the inference strategy’s scope in each conceptual category. This is how the principle of conceptual proximity is implemented. These values are determined by raising the attenuation factor (a number between 0 and 1) for a relationship to an exponent value based on the number of steps between the inferred concept and the concept found from the spreading activation, as in this expression: AttenuationValue ¼ AttenuationFactorSTEPS
ð1Þ
Similarly, temporal attenuation is calculated for each judgment located via the spreading activation; in this case the STEPS exponent refers to the number of time units (e.g. days) between the judgment being inferred and the related judgment located, thus implementing the principle of temporal proximity. The smaller the attenuation factor, the greater the dampening of influence as distance increases. For example, an attenuation factor of 0.5 causes the attenuation value to be 0.5 at one step away, 0.25 and 2 steps away, and 0.125 at three steps away. On the other hand, an attenuation factor of 0.9, causes the attenuation value to be 0.9 at one step away, 0.81 at two steps away, and 0.729 at three steps away. Second, attenuation values of concepts, along with temporal attenuation values are combined to form a judgment attenuation value for each judgment. Each judgment’s attenuation value is calculated as the product of its conceptual and temporal attenuations: JudgmentAttenuationValue ¼ ðConceptAttenuationValuei Þ TemporalAttenValue
ð2Þ
180
M. Mitri / Computers & Education 41 (2003) 173–189
Once judgment attenuations have been calculated, they are combined with judgment confidences and normalized into weights according to the following formula: Judgment Attenuation Valuei Judgment Confidencei Judgment Weighti ¼ Pn j Judgment Attenuation Valuej Judgment Confidencej
ð3Þ
where n is the total number of judgments found via the spreading activation process. Here, the principle of proximity (temporal and conceptual), which were implemented in steps (1) and (2) is combined with the principle of confidence in order to determine the weight of influence that each related judgment will have when estimating the inferred judgment value. Finally, the inferred judgment score is determined using a weighted sum, thus implementing the multi-attribute utility portion of SEMNET-MAU:
Inferred Judgment Score ¼
n X Judgment Scorei Judgment Weighti
ð4Þ
i1
These calculations will be demonstrated in later sections of this article.
5. Applying SEMNET-MAU to tacit assessment management The remainder of this paper describes a specific implementation of SEMNET-MAU in a prototype tacit assessment management system called ASSESS. The following sections describe the architecture and algorithms involved in ASSESS, and illustrate typical user interactions and screen images. This primary purpose of ASSESS is to input tacit assessments of student performance from evaluators and transform these tacit assessments to explicit assessments via the structures of the judgment and semantic network framework; in this way it serves as a knowledge codification and management tool. In addition, ASSESS makes predictions by using spreading activation and multi-attribute inference algorithms, thus serving as a knowledge discovery tool. It also serves as a knowledge dissemination tool by providing a variety of query techniques and data visualizations for viewing and manipulating data. Because it is designed for use by many evaluators, ASSESS serves as a group decision support system, facilitating collaboration assessment efforts. With ASSESS, students are judged according to their performance in a particular topic using a particular skill. For example, a student may be judged for his or her oral communication skills with regard to the topic of systems analysis. Alternatively, an evaluator may judge a student on his or her computation skills in the accounting area. Judgments are tacit; that is, they involve entry of the evaluator’s perceptions in a free-form text field. Judgments are also explicit in that the evaluator enters a specific score for a judgment and assigns it to specific topic and skill keywords.
M. Mitri / Computers & Education 41 (2003) 173–189
181
6. Stepping through ASSESS 6.1. Developing the topics and skills ontology Curriculum developers and assessment modelers can begin with the ASSESS tool to generate the ontology of topics and skills upon which evaluations are to be performed. This ontology is implemented as a semantic network, enabling the SEMNET-MAU processes to conduct inferential evaluations when needed. The structure of the semantic network also provides a conceptual framework with which evaluators can organize their assessments. Fig. 2 shows the screen used for ontology development. This screen provides the ability for users to insert concepts from the two main concept types of ASSESS, topics and skills. In this figure, the topics and skill sets are both displayed in terms of their parent–child relationships. As described earlier, any number of relationships can exist between concepts of a concept type. In ASSESS, there is also a similarity relationship that holds between concepts from different ancestral paths in the parent-child hierarchy. The screen shown in Fig. 2 provides an authoring interface that allows curriculum designers to describe the learning objectives to assess. 6.2. Creating a judgment Evaluators create judgments by identifying the student to be judged, and the topic and skill set that the judgment is for. The evaluator then indicates a number from 0 to 100 for the score, and selects a confidence level. Recall from the discussion of SEMNET-MAU that the confidence level is used to assist in inference processing. In addition, system users can view a judgment’s confidence level in order ascertain how much credence to give a particular evaluation. The evaluator also types in a comment or explanation of the judgment. Although the comment does not have any bearing in score calculations or inference processing, it provides valuable tacit knowledge to users, and essentially forms the justification for a judgment. All entered judgments are timestamped and stored in the database. The screen for this process is shown in Fig. 3.
Fig. 1. Performance assessment as a contributor to study practice (single-loop learning) and curriculum development (double-loop learning).
182
M. Mitri / Computers & Education 41 (2003) 173–189
Fig. 2. Author screen for developing ontology of topics and skill sets.
Fig. 3. Screen for entering an evaluation.
M. Mitri / Computers & Education 41 (2003) 173–189
183
6.3. Performing standard queries Users can perform standard database queries in ASSESS. For example, in Fig. 4 the top left portion of the screen indicates a query of all judgments from evaluators in the CIS/OM program for all computer information students on the topic Programming. The list in the upper right portion shows all the judgments satisfying the query, in descending order by date. The bottom right portion shows details of the selected judgment from the list, which can be altered or deleted. 6.4. Visualizing data and analyzing performance trends In order to support decision-making in an assessment environment, ASSESS includes a graphic visualization feature with seamless integration between differing views of the same data, and includes drill-down capabilities for summary-to-detail transitions. Fig. 5 shows a grid/chart view whereby evaluators can filter data according to any combination of topics, skills, students and/or majors, evaluators and/or programs and date ranges, and view a pictorial (chart or grid) view summarizing the assessment results. In the example of Fig. 5, we see a summary of a particular student’s trend of judgments over the academic year on a month by month basis for four major topics: database, programming, networks, and systems analysis. This trend shows improvement in database scores, steadily high scores in systems analysis, low programming scores, and deterioration of network performance.
Fig. 4. A standard assessment query.
184
M. Mitri / Computers & Education 41 (2003) 173–189
Fig. 5. Graphical view of one student’s performance over time.
A user can view the specific judgments for a particular data point in the chart by clicking on the data point itself. This drill-down process takes the user directly to the list of judgments, as shown in Fig. 6. Here we see the screen that results when the user clicked on the lowest data point in the month of April 2002 for the ‘‘network’’ topic, which consists of three judgments made in that month. By selecting a judgment from the list, the student can see the details of the judgment. Fig. 7 shows an aggregation of judgments for all IS-related courses during the academic year, displaying the aggregation by topic area and skill set. This illustrates the variety of filters and associations that can be displayed in the graph view. ASSESS helps evaluators ascertain common
Fig. 6. Drilling down to see details of judgments for a particular topic and time period.
M. Mitri / Computers & Education 41 (2003) 173–189
185
Fig. 7. Viewing trends in student performance as aggregate organized by topic and skill set.
strengths and weaknesses of all students, thus supporting faculty efforts for improving the quality of education at their institution. 6.5. Assessment knowledge discovery through inference ASSESS includes an inference algorithm based on the SEMNET-MAU model described earlier. This helps predict student performance on topics and skill sets when no explicit judgment exists. In ASSESS, the semantic network is composed of two conceptual categories: topics and skill sets. Each conceptual category includes a number of concepts related to each other via parent–child (hierarchy) links and similarity (cross-hierarchy) links. Recall that ASSESS provides a screen for managing these topics, as shown in Fig. 2. As described earlier, the inference algorithm estimates a judgment based on the principles of temporal and conceptual proximity and judgment confidence, implemented via an inference strategy that determines the scope of inference and the attenuation factors. This is shown in Fig. 8. In this case the user performed a query searching for a judgment of the student by the evaluator for the skill of written communication in the topic of programming. No judgment was found. So the user requested an inference by clicking the Inference button (lower left quadrant of the screen). The lower left quadrant of the screen also shows the inference strategy, the scope of inference and the attenuation factors. As shown in Fig. 2, the topic called ‘‘programming’’ is a direct descendent of ‘‘information systems’’, which is in turn a direct descendent of ‘‘business knowledge’’. Thus, as shown in Fig. 8, the spreading activation has caused the topics scope of inference, being bounded by a steps limitation of 2, to include programming, information systems, and business knowledge as the allowable topics to search. In addition, the skill ‘‘written communication’’ has a parent–child link to ‘‘communication’’, and a similarity link to ‘‘oral communication’’. Thus, these three skill sets are found by spreading activation, based on the parent–child and similarity linkages and the step constraints of the relationship types in the inference strategy. Therefore, for this inference,
186
M. Mitri / Computers & Education 41 (2003) 173–189
Fig. 8. The result of an inferential evaluation via SEMNET-MAU model.
spreading activation in the semantic network has produced nine possible combinations of the three topics and the three skill sets within the scope of inference. Fig. 8 also shows the temporal steps constraint of 9999 days, which in effect places no constraint on the time period. Note that the user is able to set the steps constraint to his or her discretion. The top right quadrant of Fig. 8 shows that the inference algorithm has constructed a query using these nine topic-skill combinations to obtain every judgment satisfying the step constraints. The query has resulted in four actual judgments. These judgments are ranked based on their relative weights, which were determined based on formulas of the SEMNET-MAU model. Note that the inference strategy shown in Fig. 8 displays the attenuations associated with the topics and skill sets via different types of relationship links. For parent–child links the attenuation factor is 0.9 and for similarity links the attenuation factor is also 0.9. In addition, the temporal attenuation factor is set at 0.9999. Users can set these attenuation factors (which should be a number between 0 and 1) at their discretion to affect the inference results. From formula (1) above, the attenuation values for the topics and skills within the scope were calculated. For example, the business knowledge attenuation value is 0.81. This is because the number parent-child attenuation factor is 0.9 and there are two steps between programming and business knowledge, hence the conceptual attenuation value for business knowledge is 0.92=0.81. These conceptual attenuations, along with temporal attenuations and confidence levels were combined in formula (3) to determine the weight of each judgment, shown in the list of the top right quadrant of Fig. 8.
M. Mitri / Computers & Education 41 (2003) 173–189
187
The inferred judgment value is shown in the bottom left quadrant of Fig. 8, near the Inference button. This value of 60 was determined based on formula (4). In this way new knowledge was discovered where none existed before, another knowledge management contribution of the system.
7. Conclusion This paper argues that performance assessment is an activity that can benefit from knowledge management techniques. In particular, tacit assessment, which involves evaluator intuition and judgment rather than objective standardized testing, and which therefore is considerably harder to codify and manage, is a prime candidate for the types of knowledge management that involve transformation from tacit to explicit knowledge. Considering the wealth of useful information in tacit assessments, and the importance of this information for the purposes of student remediation and curriculum design, there is sufficient motivation to warrant an exploration of applying knowledge management principles to tacit assessment. The assessment management system described in this article is one implementation of this application of knowledge management to tacit assessment. ASSESS organizes knowledge through an ontology of topics and skill sets via a semantic network. It provides an input mechanism for evaluators to create tacit judgments and transform them to explicit judgments based on the structure of judgment records and their relationships to concepts in the semantic network. ASSESS provides views of summary data and allows drill-down transformations from general to specific information. Knowledge discovery is supported through the use of spreading activation and multi-attribute utility calculations for the purpose of inferring judgments. ASSESS offers a potential solution to difficult assessment problems, and illustrates the applicability of knowledge management to higher education. In its prototype form, the system is written in MS Access and VBA. This proof-of-concept system next will be tested in a classroom setting. Beyond that, several enhancements to the system are being considered. These include providing automatic links to exam results, tutorial components, and eLearning modules. Data mining and visualization techniques can be extended, and collaborative filtering techniques can be used to provide enhanced functionality.
References Alavi, & Leidner, D. E. (1999). Knowledge management systems: issues, challenges, and benefits, Communications of the AIS, 1, No. 7. Argyris, C., & Schon, D. (1978). Organizational learning: a theory of action perspective. Reading, PA: Addison Wesley. Association to Advance Collegiate Schools of Business (AACSB) (2001), Standards for business accreditation. Available: www.aacsb.edu. Berliner, H. (1979) On the construction of evaluation functions for large domains. Proc. Intl. Joint Conf on Artificial Intelligence. pp 53–55. Black, K. (2000). Internet search engines and knowledge management. In T. K. Srikantaiah, & M. E. D. Koenig (Eds.), Knowledge management for the information professional (pp. 255–268). Medford, NJ: Information Today, Inc.
188
M. Mitri / Computers & Education 41 (2003) 173–189
Bloom, B. S. (1956). Taxonomy of educational objectives handbook I: cognitive domain. New York: David McKay Co 255-268. Brachman, R. J. (1979) On the epistemological status of semantic networks. In: Brachman and Levesque (Eds.) Readings in Knowledge Representation. Morgan Kaufmann Publishers, Inc. 1985. pp 191–216. Cohen, P., & Kjeldsen, R. (1987) Information retrieval by constrained spreading activation in semantic networks. COINS Technical Report 87-66. University of Massachusetts at Amherst. Dawes, R. (1988). Rational choice in an uncertain world. Orlando, FL: Harcourt Bruce Jovanovich Publishers, Inc. Gaines, B. R., & Shaw, M. L. G. (1993). Open architecture multimedia documents. Proceedings of ACM Multimedia, 93, 137–146. Gaines, B. R., & Shaw, M. L. G. (1997), Knowledge management for research communities. Proceedings of the AAAI Spring Symposium on Artificial Intelligence in Knowledge Management, Stanford University, (March 24–26, 1997), pp 55–62. Garvin, D. A. (1993). Building a learning organization. Harvard Business Review, 72(4), 78–86. Gregory, V. (2000). Knowledge management and building the learning organization. In: T. K. Srikantaiah, & M. E. D. Koenig (Eds.), Knowledge management for the information professional information today, Inc. Medford, NJ 2000 pp 161–179. Kuwata, Y., Yatsu, M. (1997), Managing knowledge using a semantic-network. In: Proceedings of the AAAI Spring Symposium on Artificial Intelligence in Knowledge Management, Stanford University, (March 24–26, 1997), pp 94–98. Lambiotte, J. G., Dansereau, D. F., Cross, D. R., & Reynolds, S. B. (1989). Multirelational semantic maps. Educational Psychology Review, 1(4), 331–367. Liebowitz, J. (2001). Knowledge management and its link to artificial intelligence. Expert Systems with Applications, 20(1), 1–6. Liu, M. (1994). Hypermedia-assisted-instruction and second language learning: a semantic-network-based approach. Computers in the Schools, 10(3), 293–312. Maher, M. (1988). HI-RISE: and expert system for preliminary structural design. In M. D. Rychener (Ed.), Expert systems for engineering design (pp. 37–52). USA: Academic Press, Inc. Miller, A. H., Imrie, B. W., & Cox, K. (1998). Student assessment in higher education: a handbook for assessing performance. London: Kogan Page, Ltd. Mitri, M. (1995). Combining semantic networks with multi-attribute utility models: an evaluative database indexing method. Expert Systems with Applications: An International Journal, 9(3), 283–294. Nemati, H. R., Steiger, D. M., Lakshmi, S. I., & Herschel, R. T. (2002). Knowledge warehouse: an architectural integration of knowledge management, decision support, artificial intelligence and data warehousing. Decision Support Systems, 33(2), 143–161. Nonaka, I., & Takeouchi, H. (1995). The knowledge-creating company. NY: Oxford University Press. Nordvall, R. C., & Braxton, J. M. (1996). An alternative definition of quality of undergraduate college education: toward usable knowledge for improvement. Journal of Higher Education, 67(5), 483–497. O’Leary, D. E., & Selferidge, P. (2000), Knowledge management for best practices. Communications of the ACM, 43(11). Pinson, S. (1991). Integrating surface knowledge and multiattribute decision models in a credit risk assessment system. AAAI-1991 AI in Business Workshop, Anaheim, CA. Quillian, M. R. (1967) Word concepts: a theory and simulation of some basic semantic capabilities. In: Brachman and Levesque (Eds.), Readings in knowledge representation. Morgan Kaufmann Publishers, Inc. 1985. pp 97–118. Rademacher, R. (1999), Applying bloom’s taxonomy of cognition to knowledge management systems, SIGCPR ’99, New Orleans, pp. 276–278. Roussopoulos, N., & Mylopoulos, J. (1975) Using semantic networks for database management. In: Mylopolopus, & Brodie (Eds.), Readings in artificial intelligence and databases. Morgan Kaufmann Publishers, Inc., pp. 112–137. Saaty, T. L. (1980). The analytic hierarchy process. New York: McGraw-Hill. Sahasrabudhe, V. (2000). Information technology in support of knowledge management. In T. K. Srikantaiah, & M. E. D. Koenig (Eds.), Knowledge management for the information professional (pp. 255–268). Medford, NJ: Information Today, Inc. Sallis, E., & Jones, G. (2002). Knowledge management in education. London: Kogan Page.
M. Mitri / Computers & Education 41 (2003) 173–189
189
Schank, R. C., & Rieger, C. J. (1974) Inference and computer understanding of natural language. In: Brachman and Levesque (Eds.), Readings in knowledge representation. Morgan Kaufmann Publishers, Inc. 1985. pp 119–140. Scouller, K. M., & Prosser, M. (1994). Students’ experiences in studying for multiple-choice question examinations. Studies in Higher Education, 19, 267–279. Senge, P. M. (1990). The fifth discipline: the art and practice of the learning organization. NY: Doubleday/Currency. Slovic, P., & Lichtenstein, S. (1971). Comparison of bayesian and regression approaches to the study of information processing in judgement. Organizational Behavior and Human Performance, 6, 649–744.