- Email: [email protected]
Reflections on 25+ years of knowledge acquisition
Available online at www.sciencedirect.com
Int. J. Human-Computer Studies 71 (2013) 211–215 www.elsevier.com/locate/ijhcs
Reflections on 25 þ years of knowledge acquisition Bob J. Wielinga Vrije Universiteit Amsterdam, Netherlands Accepted 25 September 2012 Available online 22 October 2012
Abstract In this paper I give a short reflection on Knowledge Acquisition as a subfield of AI and Knowledge Engineering over the last 25 years or so. My major message is that knowledge modeling is an underrated but still important method to reduce the complexity problems that arise in constructing knowledge-based applications. Scale – as apparent in the Semantic Web – is another important parameter in recent developments in Knowledge Acquisition: it requires other techniques than those of the 1980s. Natural Language Processing is the most promising way forward, but also the most difficult source of the acquisition of formalized knowledge. I will argue that some of the lessons learned in building knowledge-based systems may carry over to reasoning in the Semantic Web and to knowledge acquisition from natural language web sources. & 2012 Elsevier Ltd. All rights reserved. Keywords: Knowledge acquisition; Knowledge engineering; CommonKads; Knowledge patterns
1. Lessons from the past Mankind has been concerned with knowledge and knowledge processes for many thousands of years. Gaines (this issue) discusses in this issue evidence of knowledge capture as early as 30,000 years ago. Philosophers have been concerned with the nature of knowledge over the last two and a half millennia. However it is only a relatively short time ago that questions arose how to represent knowledge in such a way that it can be used by computer programs and – more importantly in the context of this special issue – how to acquire such formalized knowledge. Knowledge representation has been a topic in AI ever since the field emerged in the 1950s, but knowledge acquisition only became an issue in the early 1980s with the advent of computer systems that could use considerable amounts of knowledge to solve problems: expert systems. My personal interest in knowledge acquisition for computer systems started in 1983 with one of the first ICT (Esprit) projects of the European Community: P12 (Hayward et al., 1987). The goal of that project was to develop engineering principles for the acquisition and implementation of expert knowledge. Using techniques
from psychology – interviews, repertory grids, modeling – and AI we attacked the knowledge acquisition problem (Wielinga and Breuker, 1984). Early lessons learned were that verbal data – as acquired by various interviewing techniques – do not speak for themselves, they need interpretation to become usable in computer systems. Such interpretation requires modeling (Wielinga and Breuker, 1986). These early exercises in knowledge acquisition spawned a series of European projects aimed at developing a structured methodology for engineering knowledgebased systems resulting in the CommonKads methodology (Schreiber et al., 1999). The starting point of CommonKads was the notion of the knowledge level (Newell, 1982). Modeling at the knowledge level enabled a decoupling of the implementation details and the actual content of the knowledge. CommonKads advocated the following principles for knowledge modeling:
E-mail address: [email protected] 1071-5819/$ - see front matter & 2012 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.ijhcs.2012.10.007
Knowledge acquisition requires interpretation and modeling at the knowledge level (Newell, 1982). Knowledge comes in three types: domain knowledge (domain schemata and instances thereof), inference knowledge and task (control) knowledge. Knowledge templates are a crucial tool for knowledge acquisition and knowledge reuse.
212
B.J. Wielinga / Int. J. Human-Computer Studies 71 (2013) 211–215
In particular problem solving methods (PSM’s) (Benjamins and Fensel, 1998) are template models for generic reasoning processes coupled to domain knowledge schemata.
The metaphysical stance of CommonKads was based on pragmatic considerations. Domain knowledge is clearly the field of the domain expert and can be elicited by fairly simple techniques. Domain schemata and inference knowledge are the domain of the knowledge engineer who interprets reasoning steps of the human expert in terms of cognitive processes, the task level specifies a control strategy which closely relates to AI strategies for search and problem solving. The distinctions made in CommonKads and comparable approaches (Clancey, 1985; Steels, 1990) were useful in building knowledge-based systems and libraries of reusable components successfully supported the knowledge engineering process (Valente et al., 1998; Breuker, 1999; Fensel et al., 1999). What happened to the ideas that CommonKads advocated? Fox (2011), in his reflection on 25 years of the ‘‘Knowledge Engineering Review’’, wondered whether the idea of reusable components, such as PSM’s, as building blocks for knowledge-based systems that was so prominent in the 1990s has failed or that it has just gone out of fashion, and if so why? There is no simple answer to this question. The world has changed with the Semantic Web (Gil, 2011; d’Aquin et al., 2008). With the Semantic Web the domain of KA has become much larger, but the tasks have become simpler. Knowledge engineering projects that are just concerned with single user, stand alone knowledge based systems in a closed world are not a major concern of the KA community anymore, albeit that such systems are still being developed in commercial contexts (see for example Zhou et al, 2007; Yang et al., 2006). The tasks in the expert systems era, such as diagnosis, assessment, planning, configuration and design, were complex but well defined, while the tasks in the Semantic Web context are much more difficult to classify. We are now concerned with tasks like semantic search, information extraction, the representation of large knowledge bases and knowledge integration. However, the powerful notions of reusable knowledge structures as defined in CommonKads have not disappeared, but come back in a somewhat different form, as we will discuss in subsequent sections. The special issue of AI EDAM (Brown, 2009) shows other evidence that the notion of PSM is still relevant in Knowledge Engineering. In particular O’Connor et al. (2009) and Domingue and Fensel (2009) argue in that issue that the PSM approach is still a promising line of research also in the context of the web, but that there is a lack of tools to support the configuration of PSMs into practical applications. This lack of tools to support knowledge level modeling and PSM configuration is remarkable. In the 1990s much interest and effort was dedicated to the development of tools for KA and many new tools were
developed and presented for example during the Banff KAW conferences. Only one of all these tools appears to have survived: Protege (Musen, 1989). In the early days of KADS the knowledge model contained a fourth layer: the strategic layer. The knowledge involved in this layer was subject of an extensive research project (REFLECT, Reinders et al., 1991), but was removed from the CommonKads knowledge model since it concerned knowledge about reasoning processes and not about the domain or task itself. Meta-level reasoning in the context of knowledge-based systems has not been a topic of major interest in the KA community. However, issues like how to select or configure a problem solving strategy or how to deal with inconsistent results of a reasoning process seem still to be valid in current research (O’Connor et al., 2009). The IBROW project (Benjamins et al., 1998) aimed at the automatic configuration of problem solving methods from PSM components distributed on the web. A specification language for defining problem solving competence, UPML (Fensel et al., 1999), was developed and several implementations of a brokering service for problem solving were created (Benjamins et al., 1999; Crube´zy et al., 2003; Fensel et al., 1999). Work on semantic web services could potentially benefit from these techniques for integrating and combining distributed software components. Unfortunately the results of the REFLECT and IBROW projects have not been taken up, possibly because the issues tackled in these projects were somewhat ahead of their time. In the 1990s the focus on small scale, complex knowledge bases shifted somewhat to larger knowledge repositories which had a less complex structure, but were better understood and carried the promise of reusability: ontologies. In philosophy an ontology is a theory of being, but in Computer Science an ontology is a sharable conceptualization of some domain (Gruber, 1993). van Heijst et al. (1997) add to that definition that ontologies should be at the knowledge level and that the viewpoint taken in the conceptualization may be affected by the particular domain and the task that the ontology should support. As a consequence a distinction between different types of ontologies was introduced: generic ontologies, domain ontologies and application ontologies. In the 1990s it was gradually realized that linguistic databases (Fellbaum, 1998) and thesauri such as the Art and Architecture Thesaurus (The Getty Foundation, 2000), could become valuable resources for the Knowledge Acquisition community (Wielinga et al., 2001). At the same time it became clear that such sources were not perfect: the notion of concept was somewhat overloaded, relations were not used consistently and coverage was imbalanced. Dealing with these problems became a major research issue (Gangemi et al., 2003). In spite of these problems such resources are now part of the LOD cloud and are being used in many applications. In conclusion, 25 years of Knowledge Acquisition research has highlighted some useful concepts: the importance of
B.J. Wielinga / Int. J. Human-Computer Studies 71 (2013) 211–215
the knowledge level, the typology of knowledge types, reusability of knowledge at different levels of abstraction and the advantages of libraries of knowledge components. 2. Problems of the present Although CommonKads has been widely adopted in the Knowledge Engineering community (see for example Prat et al., 2012; Yang et al., 2006; Zhou et al, 2007), in Semantic Web research the use of modeling for reasoning has been almost nonexistent, with a few exceptions (Arguello et al., 2008; van Aart et al., 2010). The Semantic Web community has largely ignored the lessons learned in engineering knowledge based systems. van Harmelen et al. (2009); Gil and Noy (2009) have noted this problem and present an extensive analysis of a number of Semantic Web applications. Although they use a set of primitive inferences which is more logic based than the usual set of inferences used in CommonKads, their work clearly shows an added value of modeling Semantic Web applications at the knowledge level using a structured approach. They conclude that many Semantic Web applications can be generalized to a small set of generic tasks that involve only limited forms of reasoning. The question ‘‘where is the reasoning in the Semantic Web and what forms does it take?’’ is still a very valid one. The importance of modeling becomes even more evident when one considers the increasing complexity of standard representation schemata arising in the Semantic Web context. The relations in RDF(S) are limited and the inferences that can be made over these relations, such as inheritance, are fairly well-understood. The introduction of more complex schemata such as SKOS, introduces relations with a much more fuzzy semantics and hence much less clear inferences that can be made with them. What can we derive from the statement that A skos:hasBroader B? One obvious inference is that B is a more general concept than A, hence the set of instances of A may overlap with those of B. More subtle inferences could be that certain properties of B carry over to A, but maybe not all of them. Alternatively, narrower concepts of B can have some similarity to A. Making inferences over relations that have a more intuitive than a pure logic interpretation explicit was one of the goals of CommonKads. Complex schemata, such as the Europeana Data Model (Hennicke et al., 2011) for integrating large amounts of heterogeneous data sets, introduce an even larger set of relations that require complex interpretation processes to be used in intelligent processes. Modeling large and complex datasets at the schema level is well underway in the Semantic Web community, but modeling the reasoning processes over these schemata is still very much an open problem. Knowledge Acquisition in the last two decades of the previous century was largely a small scale enterprise. Knowledge bases consisted of a few thousands concepts at most. Current efforts in knowledge research aim at capturing much larger sets of knowledge elements and
213
attempt to build applications that employ very large knowledge bases. Elicitation techniques developed in the 1980s are just not efficient enough to cater for the creation of such large scale knowledge repositories. Where will the knowledge of even larger knowledge bases come from? The Linked Open Data initiative (Bizer et al., 2009) shows that large amounts of knowledge can be made accessible. The problem however is how to link the millions of knowledge elements of individual sources to each other and how to integrate all these pieces of knowledge into a coherent whole. A first step in this integration process is that of mapping ontologies. The work of Tordai (Tordai, 2012; Tordai et al., 2011; Tordai, 2010) shows that mapping large scale ontologies such as Wordnet (Fellbaum, 1998) and the Art and Architecture Thesaurus (The Getty Foundation, 2000) can be done automatically to a certain extent, but many problems remain:
coverage even between fairly general ontologies differs considerably; ambiguity in concept labels causes major problems in ontology mapping; hierarchical structures modeling similar domains are often very different, because of different viewpoints on the world that ontology builders adhere to; concepts are often fuzzy rather than Aristotelian, well defined entities.
The latter point has been well documented in psycholinguistic studies (for example Rosch et al., 1976; Lakoff, 1987). Lakoff argues that concepts as cognitive constructs in human thinking and language should be seen as conceptual clouds rather than as single points of meaning. For example the concept ‘‘mouth’’ has a central (cf. Rosch prototypes) meaning of an entrance through which some substance can be transported, but it has also many derived – instrumental, metaphorical, metonymic – meanings such as the mouth of a river or a mouth as a communication device. This complexity of concepts makes mapping and integration of ontologies a difficult task. Complex relational networks and even more complex reasoning processes will be needed to produce behavior that really mimics human capabilities. Whereas the sources of knowledge in the expert systems era were human experts, for large scale Knowledge Acquisition we now have to resort to other source materials, notably texts representing knowledge in natural language. Extracting knowledge from text remains a very difficult task. Parsers that produce parse trees for sentences are widely available, but the interpretation of such tree structures into knowledge is still a difficult area. Certain types of structures such as Named Entities, dates and simple events can be extracted from parsed texts, but more complex concepts and relations are still very difficult to detect.
214
B.J. Wielinga / Int. J. Human-Computer Studies 71 (2013) 211–215
3. Future directions Assigning meaning to paths in RDF graphs can be achieved by using semantic patterns that group sequences of relations into abstract relations, much in the same way as inferences in CommonKads generate new knowledge by combining simple domain facts. Semantic relation search will greatly improve the usefulness of the large amount of simple relations currently present in the Semantic Web. In recent work on ontology engineering a similar notion, that of knowledge patterns is emerging (Clark et al., 2000; Schreiber and Aroyo, 2008; Gangemi, 2005; Gangemi and Presutti, 2010). Knowledge patterns are units of meaning that can be used as building blocks of ontologies and domain knowledge bases. The idea of knowledge patterns is similar to domain schemata in CommonKads, although the latter patterns are closely connected to PSM’s and the importance of patterns as stand alone meta-level structures was not prominent in CommonKads. Building libraries of reusable knowledge and inference patterns appears to be a promising way forward to make sense of the large amount of implicit knowledge in the Semantic Web. The potential of the huge amounts of knowledge in the form of natural language documents is even greater, but more difficult to explicate. Language technology – lexical databases, parsers, information extraction techniques – has reached a point where it is usable on a large scale. However, interpreting the structures that such technology produces is still very difficult. Early attempts to guide natural language interpretation using template models such as scripts (Schank and Abelson, 1977) were successful in small domains, but did not scale up. Some research projects already explore similar lines. For example, the DART database (Clark and Harrison, 2009; Gil and Noy, 2009) contains a large amount of simple propositions of the type ‘‘an airplane can fly’’. It would be very interesting to see whether generalizations of these propositions can lead to interesting relational patterns. Framenet (Baker and Sato, 2003) and its translation into knowledge patterns (Nuzzolese et al., 2011) may be another source of interesting, reusable knowledge structures. A main challenge for today’s research in KA is to develop semantic patterns which can fulfill a similar role as generic models in problem solving. 4. Conclusions Allen Newell’s notion of the Knowledge Level has been widely acknowledged in the AI community, but at the same time its implications appear to have been largely forgotten. CommonKads has attempted to provide a framework to describe knowledge structures and corresponding reasoning processes in semi-formal models at the knowledge level. Such models have been successful in small scale knowledge-based system applications, but have scarcely been applied in large scale Semantic Web applications. Developing and testing such models and abstracting them into templates (cf. inference patterns and PSM’s) will – in
my view – be a crucial advance to the development of Semantic Web technologies. Scaling up Knowledge Acquisition to sizes that are promised by the Semantic Web, will be a major challenge both to researchers in the Knowledge Sciences and to those in the Natural Language Processing communities. Attacking these challenges could benefit from taking some of the lessons of Knowledge Acquisition research from the previous century more seriously than has been done so far.
References Arguello, M., Des, J., Fernandez-Prieto, M., Perez, R., Paniagua, H., 2008. Enabling reasoning on the web: introducing a test-bed simulation framework. In: Second UKSIM European Symposium on Computer Modeling and Simulation, 2008. EMS ’08, pp. 469–475. Baker, C.F., Sato, H., 2003. The FrameNet data and software. In: Matsumoto, Y. (Ed.), The Companion Volume to the Proceedings of 41st Annual Meeting of the Association for Computational Linguistics, pp. 161–164. Benjamins, R., Decker, S., Fensel, D., Motta, E., Schreiber, G., Studer, R., Wielinga, B., 1998. Ibrow3—an intelligent brokering service for knowledge-component reuse on the world wide web. In: Proceedings of the 11th Banff Knowledge Acquisition for Knowledge-Based Systems Workshop. Benjamins, V.R., Fensel, D., 1998. Editorial: problem-solving methods. International Journal of Human-Computer Studies 49 (4), 305–313. Benjamins, V.R., Wielinga, B.J., Wielemaker, J., Fensel, D., 1999. Towards brokering problem-solving knowledge on the internet. In: Fensel, D., Studer, R. (Eds.), Knowledge Acquisition, Modeling and Management, 11th European Workshop, EKAW ’99, Dagstuhl Castle, Germany, May 26–29, 1999, Proceedings. Lecture Notes in Computer Science, vol. 1621. Springer, pp. 33–48. Bizer, C., Heath, T., Berners-Lee, T., 2009. Linked data—the story so far. International Journal on Semantic Web and Information Systems 5 (3), 1–22. Breuker, J., 1999. Indexing problem solving methods for reuse. In Fensel, D., Studer, R. (Eds.), Knowledge Acquisition, Modeling and Management, 11th European Workshop, EKAW ’99, Dagstuhl Castle, Germany, May 26–29, 1999, Proceedings. Lecture Notes in Computer Science, vol. 1621. Springer, pp. 315–322. Brown, D.C., 2009. Problem solving methods: past, present, and future. AI EDAM 23 (4), 327–329. Clancey, W.J., 1985. Heuristic classification. Artificial Intelligence 27 (3), 289–350. Clark, P., Harrison, P., 2009. Large-scale extraction and use of knowledge from text. In: Gil, Y., Noy, N.F. (Eds.), Proceedings of the 5th International Conference on Knowledge Capture (K-CAP 2009), September 1–4, 2009. ACM, Redondo Beach, CA, USA, pp. 153–160. Clark, P., Thompson, J.A., Porter, B.W., 2000. Knowledge patterns. In: Cohn, A.G., Giunchiglia, F., Selman, B. (Eds.), KR. Morgan Kaufmann, pp. 591–600. Crube´zy, M., Musen, M.A., Motta, E., Lu, W., 2003. Configuring online problem-solving resources with the internet reasoning service. IEEE Intelligent Systems 18 (2), 34–42. d’Aquin, M., Motta, E., Sabou, M., Angeletou, S., Gridinoc, L., Lopez, V., Guidi, D., 2008. Toward a new generation of semantic web applications. IEEE Intelligent Systems 23 (3), 20–28. Domingue, J., Fensel, D., 2009. Problem solving methods in a global networked age. AI EDAM 23 (4), 373–390. Fellbaum, C. (Ed.), 1998. WordNet: An Electronic Lexical Database. MIT Press. Fensel, D., Benjamins, V.R., Motta, E., Wielinga, B.J., 1999. Upml: a framework for knowledge system reuse. In: Dean, T. (Ed.), IJCAI. Morgan Kaufmann, pp. 16–23.
B.J. Wielinga / Int. J. Human-Computer Studies 71 (2013) 211–215 Fensel, D., Studer, R. (Eds.), Knowledge Acquisition, Modeling and Management, 11th European Workshop, EKAW ’99, Dagstuhl Castle, Germany, May 26–29, 1999, Proceedings. Lecture Notes in Computer Science, vol. 1621. Springer. Fox, J., 2011. Formalizing knowledge and expertise: Where have we been and where are we going?. Knowledge Engineering Review 26 (1), 5–10. Gaines, B. Knowledge acquisition: past, present and future. International Journal of Human Computer Studies, http://dx.doi.org/10.1016/j.ijhcs. 2012.10.010, this issue. Gangemi, A., 2005. Ontology design patterns for semantic web content. In: Gil, Y., Motta, E., Benjamins, V.R., Musen, M.A. (Eds.), International Semantic Web Conference. Lecture Notes in Computer Science, vol. 3729. Springer, pp. 262–276. Gangemi, A., Guarino, N., Masolo, C., Oltramari, A., 2003. Sweetening wordnet with dolce. AI Magazine 24 (3), 13–24. Gangemi, A., Presutti, V., 2010. Towards a pattern science for the semantic web. Semantic Web 1 (1–2), 61–68. Gil, Y., 2011. Interactive knowledge capture in the new millennium: how the semantic web changed everything. Knowledge Engineering Review 26 (1), 45–51. Gil, Y., Noy, N.F. (Eds.), Proceedings of the 5th International Conference on Knowledge Capture (K-CAP 2009), September 1–4, 2009. ACM, Redondo Beach, CA, USA. Gruber, T.R., 1993. A translation approach to portable ontology specifications. Knowledge Acquisition 5 (2), 199–220. Hayward, S.A., Wielinga, B.J., Breuker, J.A., 1987. Structured analysis of knowledge. International Journal of Man-Machine Studies 26 (4), 487–498. Hennicke, S., Olensky, M., de Boer, V., Isaac, A., Wielemaker, J., 2011. A data model for cross-domain data representation. In: Proceedings of the 12th International Symposium on Information Science, Hildesheim, Germany, pp. 136–147. Lakoff, G., 1987. Women, Fire and Dangerous Things; What Categories Reveal About the Mind. University of Chicago Press. Musen, M.A., 1989. An editor for the conceptual models of interactive knowledge-acquisition tools. International Journal of Man-Machine Studies 31 (6), 673–698. Newell, A., 1982. The knowledge level. Artificial Intelligence 18 (1), 87–127. Nuzzolese, A.G., Gangemi, A., Presutti, V., 2011. Gathering lexical linked data and knowledge patterns from framenet. In: K-CAP 2011, the Sixth International Conference on Knowledge Capture, Banff, Canada. Proceedings, 2011. O’Connor, M.J., Nyulas, C., Tu, S.W., Buckeridge, D.L., Okhmatovskaia, A., Musen, M.A., 2009. Software-engineering challenges of building and deploying reusable problem solvers. AI EDAM 23 (4), 339–356. Prat, N., Akoka, J., Comyn-Wattiau, I., 2012. An mda approach to knowledge engineering. Expert Systems with Applications 39 (12), 10420–10437. Reinders, M., Vinkhuyzen, E., Voß, A., Akkermans, H., Balder, J., Bartsch-Sporl, B., Bredeweg, B., Drouven, U., van Harmelen, F., ¨ Karbach, W., Karssen, Z., Schreiber, G., Wielinga, B.J., 1991. A conceptual modelling framework for knowledge-level reflection. AI Communications 4 (2/3), 74–87. Rosch, E., Mervis, C., Gray, W., Johnson, D., Boyes-Braem, P., 1976. Basic objects in natural categories. Cognitive Psychology 8, 382–439. Schank, R.C., Abelson, R.P., 1977. Scripts, Plans, Goals and Understanding: an Inquiry into Human Knowledge Structures. L. Erlbaum, Hillsdale, NJ.
215
Schreiber, A.T., Akkermans, J.M., Anjewierden, A.A., de Hoog, R., Shadbolt, N.R., Van de Velde, W., Wielinga, B.J., 1999. Knowledge Engineering and Management: The CommonKADS Methodology. MIT Press, Cambridge, MA. Schreiber, G., Aroyo, L., 2008. Principles for knowledge engineering on the web. In: AAAI Spring Symposium: Symbiotic Relationships between Semantic Web and Knowledge Engineering, AAAI, pp. 78–82. Steels, L., 1990. Components of expertise. AI Magazine 11 (2), 28–49. The Getty Foundation, 2000. Aat: Art and architecture thesaurus. / http://www.getty.edu/research/tools/vocabulary/aat/S. Tordai, A., 2012. On Ontology Alignment Techniques. Ph.D. Thesis, Amsterdam, Vrije Universiteit Amsterdam. Tordai, A., van Ossenbruggen, J., Schreiber, G., Wielinga, B., 2011. Let’s agree to disagree: on the evaluation of vocabulary alignment. In: Musen, M.A., Corcho, O´. (Eds.), Proceedings of the Sixth International Conference on Knowledge Capture, K-CAP ’11. ACM, New York, NY, USA, pp. 65–72. Tordai, A., van Ossenbruggen, J., Schreiber, G., Wielinga, B.J., 2010. Aligning large SKOS-like vocabularies: two case studies. In: Aroyo, L., Antoniou, G., Hyvonen, E., ten Teije, A., Stuckenschmidt, H., ¨ Cabral, L., Tudorache, T. (Eds.), Proceedings of the 7th Extended Semantic Web Conference (ESWC2010). Lecture Notes in Computer Science, vol. 6088. Springer, pp. 198–212. Valente, A., Breuker, J., de Velde, W.V., 1998. The Commonkads library in perspective. International Journal of Human-Computer Studies 49 (4), 391–416. van Aart, C.J., Wielinga, B.J., van Hage, W.R., 2010. Mobile cultural heritage guide: location-aware semantic search. In: Cimiano, P., Pinto, H.S. (Eds.), EKAW. Lecture Notes in Computer Science, vol. 6317. Springer, pp. 257–271. van Harmelen, F., ten Teije, A., Wache, H., 2009. Knowledge engineering rediscovered: towards reasoning patterns for the semantic web. In: Gil, Y., Noy, N.F. (Eds.), Proceedings of the 5th International Conference on Knowledge Capture (K-CAP 2009), September 1–4, 2009. ACM, Redondo Beach, CA, USA, pp. 81–88. van Heijst, G., Schreiber, A.T., Wielinga, B.J., 1997. Using explicit ontologies in kbs development. International Journal of HumanComputer Studies 46 (2), 183–292. Wielinga, B.J., Breuker, J.A., 1984. Interpretation of verbal data for knowledge acquisition. In: ECAI, pp. 3–12. Wielinga, B.J., Breuker, J.A., 1986. Models of expertise. In: ECAI, pp. 497–509. Wielinga, B.J., Schreiber, A.T., Wielemaker, J., Sandberg, J., 2001. From thesaurus to ontology. In: K-CAP. ACM, pp. 194–201. Yang, D., Tong, L., Ye, Y., Wu, H., 2006. Applying commonkads and semantic web technologies to ontology-based e-government knowledge systems. In: Mizoguchi, R., Shi, Z., Giunchiglia, F. (Eds.), The Semantic Web, ASWC 2006. Lecture Notes in Computer Science, vol. 4185. Springer, Berlin, Heidelberg, pp. 336–342. http://dx.doi.org/10. 1007/11836025_34. Zhou, M., Ren, J., Qi, J., Niu, D., Li, G., 2007. Commonkads methodology for developing power grid switching orders systems. In: Washio, T., Zhou, Z.-H., Huang, J., Hu, X., Li, J., Xie, C., He, J., Zou, D., Li, K.-C., Freire, M. (Eds.), Emerging Technologies in Knowledge Discovery and Data Mining, Lecture Notes in Computer Science, vol. 4819. Springer, Berlin, Heidelberg, pp. 87–98. http://dx. doi.org/10.1007/978-3-540-77018-3_10.
Int. J. Human-Computer Studies 71 (2013) 211–215 www.elsevier.com/locate/ijhcs
Reflections on 25 þ years of knowledge acquisition Bob J. Wielinga Vrije Universiteit Amsterdam, Netherlands Accepted 25 September 2012 Available online 22 October 2012
Abstract In this paper I give a short reflection on Knowledge Acquisition as a subfield of AI and Knowledge Engineering over the last 25 years or so. My major message is that knowledge modeling is an underrated but still important method to reduce the complexity problems that arise in constructing knowledge-based applications. Scale – as apparent in the Semantic Web – is another important parameter in recent developments in Knowledge Acquisition: it requires other techniques than those of the 1980s. Natural Language Processing is the most promising way forward, but also the most difficult source of the acquisition of formalized knowledge. I will argue that some of the lessons learned in building knowledge-based systems may carry over to reasoning in the Semantic Web and to knowledge acquisition from natural language web sources. & 2012 Elsevier Ltd. All rights reserved. Keywords: Knowledge acquisition; Knowledge engineering; CommonKads; Knowledge patterns
1. Lessons from the past Mankind has been concerned with knowledge and knowledge processes for many thousands of years. Gaines (this issue) discusses in this issue evidence of knowledge capture as early as 30,000 years ago. Philosophers have been concerned with the nature of knowledge over the last two and a half millennia. However it is only a relatively short time ago that questions arose how to represent knowledge in such a way that it can be used by computer programs and – more importantly in the context of this special issue – how to acquire such formalized knowledge. Knowledge representation has been a topic in AI ever since the field emerged in the 1950s, but knowledge acquisition only became an issue in the early 1980s with the advent of computer systems that could use considerable amounts of knowledge to solve problems: expert systems. My personal interest in knowledge acquisition for computer systems started in 1983 with one of the first ICT (Esprit) projects of the European Community: P12 (Hayward et al., 1987). The goal of that project was to develop engineering principles for the acquisition and implementation of expert knowledge. Using techniques
from psychology – interviews, repertory grids, modeling – and AI we attacked the knowledge acquisition problem (Wielinga and Breuker, 1984). Early lessons learned were that verbal data – as acquired by various interviewing techniques – do not speak for themselves, they need interpretation to become usable in computer systems. Such interpretation requires modeling (Wielinga and Breuker, 1986). These early exercises in knowledge acquisition spawned a series of European projects aimed at developing a structured methodology for engineering knowledgebased systems resulting in the CommonKads methodology (Schreiber et al., 1999). The starting point of CommonKads was the notion of the knowledge level (Newell, 1982). Modeling at the knowledge level enabled a decoupling of the implementation details and the actual content of the knowledge. CommonKads advocated the following principles for knowledge modeling:
E-mail address: [email protected] 1071-5819/$ - see front matter & 2012 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.ijhcs.2012.10.007
Knowledge acquisition requires interpretation and modeling at the knowledge level (Newell, 1982). Knowledge comes in three types: domain knowledge (domain schemata and instances thereof), inference knowledge and task (control) knowledge. Knowledge templates are a crucial tool for knowledge acquisition and knowledge reuse.
212
B.J. Wielinga / Int. J. Human-Computer Studies 71 (2013) 211–215
In particular problem solving methods (PSM’s) (Benjamins and Fensel, 1998) are template models for generic reasoning processes coupled to domain knowledge schemata.
The metaphysical stance of CommonKads was based on pragmatic considerations. Domain knowledge is clearly the field of the domain expert and can be elicited by fairly simple techniques. Domain schemata and inference knowledge are the domain of the knowledge engineer who interprets reasoning steps of the human expert in terms of cognitive processes, the task level specifies a control strategy which closely relates to AI strategies for search and problem solving. The distinctions made in CommonKads and comparable approaches (Clancey, 1985; Steels, 1990) were useful in building knowledge-based systems and libraries of reusable components successfully supported the knowledge engineering process (Valente et al., 1998; Breuker, 1999; Fensel et al., 1999). What happened to the ideas that CommonKads advocated? Fox (2011), in his reflection on 25 years of the ‘‘Knowledge Engineering Review’’, wondered whether the idea of reusable components, such as PSM’s, as building blocks for knowledge-based systems that was so prominent in the 1990s has failed or that it has just gone out of fashion, and if so why? There is no simple answer to this question. The world has changed with the Semantic Web (Gil, 2011; d’Aquin et al., 2008). With the Semantic Web the domain of KA has become much larger, but the tasks have become simpler. Knowledge engineering projects that are just concerned with single user, stand alone knowledge based systems in a closed world are not a major concern of the KA community anymore, albeit that such systems are still being developed in commercial contexts (see for example Zhou et al, 2007; Yang et al., 2006). The tasks in the expert systems era, such as diagnosis, assessment, planning, configuration and design, were complex but well defined, while the tasks in the Semantic Web context are much more difficult to classify. We are now concerned with tasks like semantic search, information extraction, the representation of large knowledge bases and knowledge integration. However, the powerful notions of reusable knowledge structures as defined in CommonKads have not disappeared, but come back in a somewhat different form, as we will discuss in subsequent sections. The special issue of AI EDAM (Brown, 2009) shows other evidence that the notion of PSM is still relevant in Knowledge Engineering. In particular O’Connor et al. (2009) and Domingue and Fensel (2009) argue in that issue that the PSM approach is still a promising line of research also in the context of the web, but that there is a lack of tools to support the configuration of PSMs into practical applications. This lack of tools to support knowledge level modeling and PSM configuration is remarkable. In the 1990s much interest and effort was dedicated to the development of tools for KA and many new tools were
developed and presented for example during the Banff KAW conferences. Only one of all these tools appears to have survived: Protege (Musen, 1989). In the early days of KADS the knowledge model contained a fourth layer: the strategic layer. The knowledge involved in this layer was subject of an extensive research project (REFLECT, Reinders et al., 1991), but was removed from the CommonKads knowledge model since it concerned knowledge about reasoning processes and not about the domain or task itself. Meta-level reasoning in the context of knowledge-based systems has not been a topic of major interest in the KA community. However, issues like how to select or configure a problem solving strategy or how to deal with inconsistent results of a reasoning process seem still to be valid in current research (O’Connor et al., 2009). The IBROW project (Benjamins et al., 1998) aimed at the automatic configuration of problem solving methods from PSM components distributed on the web. A specification language for defining problem solving competence, UPML (Fensel et al., 1999), was developed and several implementations of a brokering service for problem solving were created (Benjamins et al., 1999; Crube´zy et al., 2003; Fensel et al., 1999). Work on semantic web services could potentially benefit from these techniques for integrating and combining distributed software components. Unfortunately the results of the REFLECT and IBROW projects have not been taken up, possibly because the issues tackled in these projects were somewhat ahead of their time. In the 1990s the focus on small scale, complex knowledge bases shifted somewhat to larger knowledge repositories which had a less complex structure, but were better understood and carried the promise of reusability: ontologies. In philosophy an ontology is a theory of being, but in Computer Science an ontology is a sharable conceptualization of some domain (Gruber, 1993). van Heijst et al. (1997) add to that definition that ontologies should be at the knowledge level and that the viewpoint taken in the conceptualization may be affected by the particular domain and the task that the ontology should support. As a consequence a distinction between different types of ontologies was introduced: generic ontologies, domain ontologies and application ontologies. In the 1990s it was gradually realized that linguistic databases (Fellbaum, 1998) and thesauri such as the Art and Architecture Thesaurus (The Getty Foundation, 2000), could become valuable resources for the Knowledge Acquisition community (Wielinga et al., 2001). At the same time it became clear that such sources were not perfect: the notion of concept was somewhat overloaded, relations were not used consistently and coverage was imbalanced. Dealing with these problems became a major research issue (Gangemi et al., 2003). In spite of these problems such resources are now part of the LOD cloud and are being used in many applications. In conclusion, 25 years of Knowledge Acquisition research has highlighted some useful concepts: the importance of
B.J. Wielinga / Int. J. Human-Computer Studies 71 (2013) 211–215
the knowledge level, the typology of knowledge types, reusability of knowledge at different levels of abstraction and the advantages of libraries of knowledge components. 2. Problems of the present Although CommonKads has been widely adopted in the Knowledge Engineering community (see for example Prat et al., 2012; Yang et al., 2006; Zhou et al, 2007), in Semantic Web research the use of modeling for reasoning has been almost nonexistent, with a few exceptions (Arguello et al., 2008; van Aart et al., 2010). The Semantic Web community has largely ignored the lessons learned in engineering knowledge based systems. van Harmelen et al. (2009); Gil and Noy (2009) have noted this problem and present an extensive analysis of a number of Semantic Web applications. Although they use a set of primitive inferences which is more logic based than the usual set of inferences used in CommonKads, their work clearly shows an added value of modeling Semantic Web applications at the knowledge level using a structured approach. They conclude that many Semantic Web applications can be generalized to a small set of generic tasks that involve only limited forms of reasoning. The question ‘‘where is the reasoning in the Semantic Web and what forms does it take?’’ is still a very valid one. The importance of modeling becomes even more evident when one considers the increasing complexity of standard representation schemata arising in the Semantic Web context. The relations in RDF(S) are limited and the inferences that can be made over these relations, such as inheritance, are fairly well-understood. The introduction of more complex schemata such as SKOS, introduces relations with a much more fuzzy semantics and hence much less clear inferences that can be made with them. What can we derive from the statement that A skos:hasBroader B? One obvious inference is that B is a more general concept than A, hence the set of instances of A may overlap with those of B. More subtle inferences could be that certain properties of B carry over to A, but maybe not all of them. Alternatively, narrower concepts of B can have some similarity to A. Making inferences over relations that have a more intuitive than a pure logic interpretation explicit was one of the goals of CommonKads. Complex schemata, such as the Europeana Data Model (Hennicke et al., 2011) for integrating large amounts of heterogeneous data sets, introduce an even larger set of relations that require complex interpretation processes to be used in intelligent processes. Modeling large and complex datasets at the schema level is well underway in the Semantic Web community, but modeling the reasoning processes over these schemata is still very much an open problem. Knowledge Acquisition in the last two decades of the previous century was largely a small scale enterprise. Knowledge bases consisted of a few thousands concepts at most. Current efforts in knowledge research aim at capturing much larger sets of knowledge elements and
213
attempt to build applications that employ very large knowledge bases. Elicitation techniques developed in the 1980s are just not efficient enough to cater for the creation of such large scale knowledge repositories. Where will the knowledge of even larger knowledge bases come from? The Linked Open Data initiative (Bizer et al., 2009) shows that large amounts of knowledge can be made accessible. The problem however is how to link the millions of knowledge elements of individual sources to each other and how to integrate all these pieces of knowledge into a coherent whole. A first step in this integration process is that of mapping ontologies. The work of Tordai (Tordai, 2012; Tordai et al., 2011; Tordai, 2010) shows that mapping large scale ontologies such as Wordnet (Fellbaum, 1998) and the Art and Architecture Thesaurus (The Getty Foundation, 2000) can be done automatically to a certain extent, but many problems remain:
coverage even between fairly general ontologies differs considerably; ambiguity in concept labels causes major problems in ontology mapping; hierarchical structures modeling similar domains are often very different, because of different viewpoints on the world that ontology builders adhere to; concepts are often fuzzy rather than Aristotelian, well defined entities.
The latter point has been well documented in psycholinguistic studies (for example Rosch et al., 1976; Lakoff, 1987). Lakoff argues that concepts as cognitive constructs in human thinking and language should be seen as conceptual clouds rather than as single points of meaning. For example the concept ‘‘mouth’’ has a central (cf. Rosch prototypes) meaning of an entrance through which some substance can be transported, but it has also many derived – instrumental, metaphorical, metonymic – meanings such as the mouth of a river or a mouth as a communication device. This complexity of concepts makes mapping and integration of ontologies a difficult task. Complex relational networks and even more complex reasoning processes will be needed to produce behavior that really mimics human capabilities. Whereas the sources of knowledge in the expert systems era were human experts, for large scale Knowledge Acquisition we now have to resort to other source materials, notably texts representing knowledge in natural language. Extracting knowledge from text remains a very difficult task. Parsers that produce parse trees for sentences are widely available, but the interpretation of such tree structures into knowledge is still a difficult area. Certain types of structures such as Named Entities, dates and simple events can be extracted from parsed texts, but more complex concepts and relations are still very difficult to detect.
214
B.J. Wielinga / Int. J. Human-Computer Studies 71 (2013) 211–215
3. Future directions Assigning meaning to paths in RDF graphs can be achieved by using semantic patterns that group sequences of relations into abstract relations, much in the same way as inferences in CommonKads generate new knowledge by combining simple domain facts. Semantic relation search will greatly improve the usefulness of the large amount of simple relations currently present in the Semantic Web. In recent work on ontology engineering a similar notion, that of knowledge patterns is emerging (Clark et al., 2000; Schreiber and Aroyo, 2008; Gangemi, 2005; Gangemi and Presutti, 2010). Knowledge patterns are units of meaning that can be used as building blocks of ontologies and domain knowledge bases. The idea of knowledge patterns is similar to domain schemata in CommonKads, although the latter patterns are closely connected to PSM’s and the importance of patterns as stand alone meta-level structures was not prominent in CommonKads. Building libraries of reusable knowledge and inference patterns appears to be a promising way forward to make sense of the large amount of implicit knowledge in the Semantic Web. The potential of the huge amounts of knowledge in the form of natural language documents is even greater, but more difficult to explicate. Language technology – lexical databases, parsers, information extraction techniques – has reached a point where it is usable on a large scale. However, interpreting the structures that such technology produces is still very difficult. Early attempts to guide natural language interpretation using template models such as scripts (Schank and Abelson, 1977) were successful in small domains, but did not scale up. Some research projects already explore similar lines. For example, the DART database (Clark and Harrison, 2009; Gil and Noy, 2009) contains a large amount of simple propositions of the type ‘‘an airplane can fly’’. It would be very interesting to see whether generalizations of these propositions can lead to interesting relational patterns. Framenet (Baker and Sato, 2003) and its translation into knowledge patterns (Nuzzolese et al., 2011) may be another source of interesting, reusable knowledge structures. A main challenge for today’s research in KA is to develop semantic patterns which can fulfill a similar role as generic models in problem solving. 4. Conclusions Allen Newell’s notion of the Knowledge Level has been widely acknowledged in the AI community, but at the same time its implications appear to have been largely forgotten. CommonKads has attempted to provide a framework to describe knowledge structures and corresponding reasoning processes in semi-formal models at the knowledge level. Such models have been successful in small scale knowledge-based system applications, but have scarcely been applied in large scale Semantic Web applications. Developing and testing such models and abstracting them into templates (cf. inference patterns and PSM’s) will – in
my view – be a crucial advance to the development of Semantic Web technologies. Scaling up Knowledge Acquisition to sizes that are promised by the Semantic Web, will be a major challenge both to researchers in the Knowledge Sciences and to those in the Natural Language Processing communities. Attacking these challenges could benefit from taking some of the lessons of Knowledge Acquisition research from the previous century more seriously than has been done so far.
References Arguello, M., Des, J., Fernandez-Prieto, M., Perez, R., Paniagua, H., 2008. Enabling reasoning on the web: introducing a test-bed simulation framework. In: Second UKSIM European Symposium on Computer Modeling and Simulation, 2008. EMS ’08, pp. 469–475. Baker, C.F., Sato, H., 2003. The FrameNet data and software. In: Matsumoto, Y. (Ed.), The Companion Volume to the Proceedings of 41st Annual Meeting of the Association for Computational Linguistics, pp. 161–164. Benjamins, R., Decker, S., Fensel, D., Motta, E., Schreiber, G., Studer, R., Wielinga, B., 1998. Ibrow3—an intelligent brokering service for knowledge-component reuse on the world wide web. In: Proceedings of the 11th Banff Knowledge Acquisition for Knowledge-Based Systems Workshop. Benjamins, V.R., Fensel, D., 1998. Editorial: problem-solving methods. International Journal of Human-Computer Studies 49 (4), 305–313. Benjamins, V.R., Wielinga, B.J., Wielemaker, J., Fensel, D., 1999. Towards brokering problem-solving knowledge on the internet. In: Fensel, D., Studer, R. (Eds.), Knowledge Acquisition, Modeling and Management, 11th European Workshop, EKAW ’99, Dagstuhl Castle, Germany, May 26–29, 1999, Proceedings. Lecture Notes in Computer Science, vol. 1621. Springer, pp. 33–48. Bizer, C., Heath, T., Berners-Lee, T., 2009. Linked data—the story so far. International Journal on Semantic Web and Information Systems 5 (3), 1–22. Breuker, J., 1999. Indexing problem solving methods for reuse. In Fensel, D., Studer, R. (Eds.), Knowledge Acquisition, Modeling and Management, 11th European Workshop, EKAW ’99, Dagstuhl Castle, Germany, May 26–29, 1999, Proceedings. Lecture Notes in Computer Science, vol. 1621. Springer, pp. 315–322. Brown, D.C., 2009. Problem solving methods: past, present, and future. AI EDAM 23 (4), 327–329. Clancey, W.J., 1985. Heuristic classification. Artificial Intelligence 27 (3), 289–350. Clark, P., Harrison, P., 2009. Large-scale extraction and use of knowledge from text. In: Gil, Y., Noy, N.F. (Eds.), Proceedings of the 5th International Conference on Knowledge Capture (K-CAP 2009), September 1–4, 2009. ACM, Redondo Beach, CA, USA, pp. 153–160. Clark, P., Thompson, J.A., Porter, B.W., 2000. Knowledge patterns. In: Cohn, A.G., Giunchiglia, F., Selman, B. (Eds.), KR. Morgan Kaufmann, pp. 591–600. Crube´zy, M., Musen, M.A., Motta, E., Lu, W., 2003. Configuring online problem-solving resources with the internet reasoning service. IEEE Intelligent Systems 18 (2), 34–42. d’Aquin, M., Motta, E., Sabou, M., Angeletou, S., Gridinoc, L., Lopez, V., Guidi, D., 2008. Toward a new generation of semantic web applications. IEEE Intelligent Systems 23 (3), 20–28. Domingue, J., Fensel, D., 2009. Problem solving methods in a global networked age. AI EDAM 23 (4), 373–390. Fellbaum, C. (Ed.), 1998. WordNet: An Electronic Lexical Database. MIT Press. Fensel, D., Benjamins, V.R., Motta, E., Wielinga, B.J., 1999. Upml: a framework for knowledge system reuse. In: Dean, T. (Ed.), IJCAI. Morgan Kaufmann, pp. 16–23.
B.J. Wielinga / Int. J. Human-Computer Studies 71 (2013) 211–215 Fensel, D., Studer, R. (Eds.), Knowledge Acquisition, Modeling and Management, 11th European Workshop, EKAW ’99, Dagstuhl Castle, Germany, May 26–29, 1999, Proceedings. Lecture Notes in Computer Science, vol. 1621. Springer. Fox, J., 2011. Formalizing knowledge and expertise: Where have we been and where are we going?. Knowledge Engineering Review 26 (1), 5–10. Gaines, B. Knowledge acquisition: past, present and future. International Journal of Human Computer Studies, http://dx.doi.org/10.1016/j.ijhcs. 2012.10.010, this issue. Gangemi, A., 2005. Ontology design patterns for semantic web content. In: Gil, Y., Motta, E., Benjamins, V.R., Musen, M.A. (Eds.), International Semantic Web Conference. Lecture Notes in Computer Science, vol. 3729. Springer, pp. 262–276. Gangemi, A., Guarino, N., Masolo, C., Oltramari, A., 2003. Sweetening wordnet with dolce. AI Magazine 24 (3), 13–24. Gangemi, A., Presutti, V., 2010. Towards a pattern science for the semantic web. Semantic Web 1 (1–2), 61–68. Gil, Y., 2011. Interactive knowledge capture in the new millennium: how the semantic web changed everything. Knowledge Engineering Review 26 (1), 45–51. Gil, Y., Noy, N.F. (Eds.), Proceedings of the 5th International Conference on Knowledge Capture (K-CAP 2009), September 1–4, 2009. ACM, Redondo Beach, CA, USA. Gruber, T.R., 1993. A translation approach to portable ontology specifications. Knowledge Acquisition 5 (2), 199–220. Hayward, S.A., Wielinga, B.J., Breuker, J.A., 1987. Structured analysis of knowledge. International Journal of Man-Machine Studies 26 (4), 487–498. Hennicke, S., Olensky, M., de Boer, V., Isaac, A., Wielemaker, J., 2011. A data model for cross-domain data representation. In: Proceedings of the 12th International Symposium on Information Science, Hildesheim, Germany, pp. 136–147. Lakoff, G., 1987. Women, Fire and Dangerous Things; What Categories Reveal About the Mind. University of Chicago Press. Musen, M.A., 1989. An editor for the conceptual models of interactive knowledge-acquisition tools. International Journal of Man-Machine Studies 31 (6), 673–698. Newell, A., 1982. The knowledge level. Artificial Intelligence 18 (1), 87–127. Nuzzolese, A.G., Gangemi, A., Presutti, V., 2011. Gathering lexical linked data and knowledge patterns from framenet. In: K-CAP 2011, the Sixth International Conference on Knowledge Capture, Banff, Canada. Proceedings, 2011. O’Connor, M.J., Nyulas, C., Tu, S.W., Buckeridge, D.L., Okhmatovskaia, A., Musen, M.A., 2009. Software-engineering challenges of building and deploying reusable problem solvers. AI EDAM 23 (4), 339–356. Prat, N., Akoka, J., Comyn-Wattiau, I., 2012. An mda approach to knowledge engineering. Expert Systems with Applications 39 (12), 10420–10437. Reinders, M., Vinkhuyzen, E., Voß, A., Akkermans, H., Balder, J., Bartsch-Sporl, B., Bredeweg, B., Drouven, U., van Harmelen, F., ¨ Karbach, W., Karssen, Z., Schreiber, G., Wielinga, B.J., 1991. A conceptual modelling framework for knowledge-level reflection. AI Communications 4 (2/3), 74–87. Rosch, E., Mervis, C., Gray, W., Johnson, D., Boyes-Braem, P., 1976. Basic objects in natural categories. Cognitive Psychology 8, 382–439. Schank, R.C., Abelson, R.P., 1977. Scripts, Plans, Goals and Understanding: an Inquiry into Human Knowledge Structures. L. Erlbaum, Hillsdale, NJ.
215
Schreiber, A.T., Akkermans, J.M., Anjewierden, A.A., de Hoog, R., Shadbolt, N.R., Van de Velde, W., Wielinga, B.J., 1999. Knowledge Engineering and Management: The CommonKADS Methodology. MIT Press, Cambridge, MA. Schreiber, G., Aroyo, L., 2008. Principles for knowledge engineering on the web. In: AAAI Spring Symposium: Symbiotic Relationships between Semantic Web and Knowledge Engineering, AAAI, pp. 78–82. Steels, L., 1990. Components of expertise. AI Magazine 11 (2), 28–49. The Getty Foundation, 2000. Aat: Art and architecture thesaurus. / http://www.getty.edu/research/tools/vocabulary/aat/S. Tordai, A., 2012. On Ontology Alignment Techniques. Ph.D. Thesis, Amsterdam, Vrije Universiteit Amsterdam. Tordai, A., van Ossenbruggen, J., Schreiber, G., Wielinga, B., 2011. Let’s agree to disagree: on the evaluation of vocabulary alignment. In: Musen, M.A., Corcho, O´. (Eds.), Proceedings of the Sixth International Conference on Knowledge Capture, K-CAP ’11. ACM, New York, NY, USA, pp. 65–72. Tordai, A., van Ossenbruggen, J., Schreiber, G., Wielinga, B.J., 2010. Aligning large SKOS-like vocabularies: two case studies. In: Aroyo, L., Antoniou, G., Hyvonen, E., ten Teije, A., Stuckenschmidt, H., ¨ Cabral, L., Tudorache, T. (Eds.), Proceedings of the 7th Extended Semantic Web Conference (ESWC2010). Lecture Notes in Computer Science, vol. 6088. Springer, pp. 198–212. Valente, A., Breuker, J., de Velde, W.V., 1998. The Commonkads library in perspective. International Journal of Human-Computer Studies 49 (4), 391–416. van Aart, C.J., Wielinga, B.J., van Hage, W.R., 2010. Mobile cultural heritage guide: location-aware semantic search. In: Cimiano, P., Pinto, H.S. (Eds.), EKAW. Lecture Notes in Computer Science, vol. 6317. Springer, pp. 257–271. van Harmelen, F., ten Teije, A., Wache, H., 2009. Knowledge engineering rediscovered: towards reasoning patterns for the semantic web. In: Gil, Y., Noy, N.F. (Eds.), Proceedings of the 5th International Conference on Knowledge Capture (K-CAP 2009), September 1–4, 2009. ACM, Redondo Beach, CA, USA, pp. 81–88. van Heijst, G., Schreiber, A.T., Wielinga, B.J., 1997. Using explicit ontologies in kbs development. International Journal of HumanComputer Studies 46 (2), 183–292. Wielinga, B.J., Breuker, J.A., 1984. Interpretation of verbal data for knowledge acquisition. In: ECAI, pp. 3–12. Wielinga, B.J., Breuker, J.A., 1986. Models of expertise. In: ECAI, pp. 497–509. Wielinga, B.J., Schreiber, A.T., Wielemaker, J., Sandberg, J., 2001. From thesaurus to ontology. In: K-CAP. ACM, pp. 194–201. Yang, D., Tong, L., Ye, Y., Wu, H., 2006. Applying commonkads and semantic web technologies to ontology-based e-government knowledge systems. In: Mizoguchi, R., Shi, Z., Giunchiglia, F. (Eds.), The Semantic Web, ASWC 2006. Lecture Notes in Computer Science, vol. 4185. Springer, Berlin, Heidelberg, pp. 336–342. http://dx.doi.org/10. 1007/11836025_34. Zhou, M., Ren, J., Qi, J., Niu, D., Li, G., 2007. Commonkads methodology for developing power grid switching orders systems. In: Washio, T., Zhou, Z.-H., Huang, J., Hu, X., Li, J., Xie, C., He, J., Zou, D., Li, K.-C., Freire, M. (Eds.), Emerging Technologies in Knowledge Discovery and Data Mining, Lecture Notes in Computer Science, vol. 4819. Springer, Berlin, Heidelberg, pp. 87–98. http://dx. doi.org/10.1007/978-3-540-77018-3_10.