A semantic metrics suite for evaluating modular ontologies

Information Systems 38 (2013) 745–770

Contents lists available at SciVerse ScienceDirect

Information Systems journal homepage: www.elsevier.com/locate/infosys

A semantic metrics suite for evaluating modular ontologies Faezeh Ensan a,n, Weichang Du b a b

Sauder School of Business, University of British Columbia, Vancouver, BC, Canada Faculty of Computer Science, University of New Brunswick, Fredericton, NB, Canada

a r t i c l e i n f o

abstract

Available online 3 January 2013

Ontologies, which are formal representations of knowledge within a domain, can be used for designing and sharing conceptual models of enterprises information for the purpose of enhancing understanding, communication and interoperability. For representing a body of knowledge, different ontologies may be designed. Recently, designing ontologies in a modular manner has emerged for achieving better reasoning performance, more efficient ontology management and change handling. One of the important challenges in the employment of ontologies and modular ontologies in modeling information within enterprises is the evaluation of the suitability of an ontology for a domain and the performance of inference operations over it. In this paper, we present a set of semantic metrics for evaluating ontologies and modular ontologies. These metrics measure cohesion and coupling of ontologies, which are two important notions in the process of assessing ontologies for enterprise modeling. The proposed metrics are based on semantic-based definitions of relativeness, and dependencies between local symbols, and also between local and external symbols of ontologies. Based on these semantic definitions, not only the explicitly asserted knowledge in ontologies but also the implied knowledge, which is derived through inference, is considered for the sake of ontology assessment. We present several empirical case studies for investigating the correlation between the proposed metrics and reasoning performance, which is an important issue in applicability of employing ontologies in real-world information systems. & 2012 Elsevier Ltd. All rights reserved.

Keywords: Ontologies Modular ontologies Ontology design Ontology evaluation Measurements Semantic metrics Cohesion Coupling Description logics Reasoning performance

1. Introduction Semantic Web techniques have been increasingly applied to real-world problems in the recent years. The Ontology for Biomedical Investigations (OBI) [45], the Gene and Gene products ontology [1] and the Earth and environmental terminologies (SWEET Ontologies) [43] are among the large ontologies used for describing complex domains. So far, numerous techniques, frameworks and methodologies for ontology development and management [15,17,14] and ontology description [37,30] have been proposed. Among these languages, Description Logics (DLs)-based languages have shown to be able to

n Correspondence to: Suite 303, 18 Pemberton Avenue, Toronto, ON, Canada. Tel.: þ1 647 348 8750. E-mail addresses: [email protected] (F. Ensan), [email protected] (W. Du).

0306-4379/$ - see front matter & 2012 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.is.2012.11.012

provide a strong basis for defining and formalizing ontologies. OWL-DL is an instance of DL-based languages, which has been widely used for representing ontologies. Various reasoning algorithms [4], query answering techniques [21] and reasoning engines [27,44] facilitate the employment of DL-based ontologies. For describing a given domain of discourse, different ontology designs are often feasible. One approach to ontology design is monolithic design, i.e., all captured concepts, roles, axioms and assertions of the domain are gathered in one monolithic ontology. For instance all the ontology concepts and roles are placed in a single OWL file in the case of OWL-DL ontologies. The second approach is to use a modularization strategy, i.e., the domain is collectively described using a set of ontologies, called ontology modules, each of which describes a subdomain of discourse. Sub-domains of a domain may have dependencies on one another. Hence, their corresponding

746

Faezeh Ensan, Weichang Du / Information Systems 38 (2013) 745–770

ontology modules cannot be represented independently. An ontology module may need to refer to symbols (classes, roles and individuals) of other modules. OWLDL has an ‘‘import’’ feature that can be used for integrating different ontology modules together and allows them to have references to external symbols from other modules. Using the import feature, an ontology module is able to import the whole knowledge base of one or more ontology modules. Modular ontology formalisms such as DDL [8], E-connections [33], P-DL [5] and IBF [16] provide alternative ways for defining and integrating ontology modules. These formalisms provide new extensions to existing Description Logics syntax and semantics. The objective of these formalisms is to allow ontology modules to evolve independently and the reasoning to be performed only on the relevant parts of the knowledge base. The emergence of new technologies for developing modular ontologies [49] highlights the increasing need for appropriate metrics for evaluating modular ontologies. A clear set of such metrics for ontology evaluation can facilitate the comparison and selection of ontologies for different applications. It also enhances the process of modular ontology design and development by providing unambiguous guidelines for ontology developers. As yet, no widely accepted metric suites have been introduced for assessing modular ontologies. 1.1. Related works Considerable amount of research has been conducted on the evaluation of ontologies [9]. One important approach for evaluating ontologies is to investigate their completeness, correctness and coverage with regards to a domain of discourse. This investigation can be accomplished by comparing ontologies with a gold standard, which is manually designed for the domain of discourse [36,10]. In Spyns et al. [46], ontology triples are evaluated by analyzing how appropriately they cover the topic of a corpus. The notions of precision and recall are employed for measuring domain coverage. The work of Brewster et al. [11] follows a similar approach for investigating how well a given ontology fits the domain knowledge. In their work, the concepts and properties of the ontology are compared with the terms in domain-specific documents, and the overlaps are analyzed. In addition, a probabilistic method is employed for finding the best ontology for a given domain-specific corpus. Another approach for ontology evaluation is taskbased evaluation. According to this approach, an ontology is analyzed to find out how effective it is in the performance of a well-defined task, i.e., whether the use of a given ontology in a specific task effects the efficiency of the tasks in that context or not [41]. In Lozano-Tello and Gomez-Perez [34], a different approach for ontology evaluation is proposed. There, a set of characteristics of ontologies, which should be considered by human experts for assessment and evaluation, is introduced and discussed. These characteristics are about the content of an ontology, the language in which the ontology is implemented, and the methodology and software tool that are

employed for developing the ontology and the cost of the usage of the ontology. The importance of metrics for measuring quality of ontologies is widely acknowledged in the literature. Gangemi et al. [20] propose measures for assessing structural, functional and usability-profiling dimensions of ontologies. The functional dimension focuses on the application of an ontology in a context and the usabilityprofiling dimension focuses on profiling and metadata. The structural dimension is concerned with the syntax and formal semantics of ontologies and is based on the representation of ontologies as graphs. Some of the proposed structural metrics are the depth and breath of the ontology graph, the cardinality of sibling nodes and leaf nodes, the average rate of axioms per class, consistency ratio, cycle ratios and inverse relation ratio. In Burton-Jones et al. [12], a metric suite is proposed for ontology assessment. These metrics measure the syntactic, semantic, pragmatic and social qualities of an ontology. Syntactic quality is concerned with the proportion of the features of the ontology language that are utilized for representing an ontology and the degree of incorrect syntax within that ontology. Semantic quality focuses on the meaning of the ontology terms. Through semantic quality metrics, the meanings of the ontology terms are investigated in a lexical database such as WordNet to see if the utilized terms are meaningful and clear. In addition, the inconsistent usage of terms are measured. Pragmatic quality evaluates the usefulness of a given ontology through its true statements, its relativity to the application requirements and also its size. Finally, social qualities measure the number of times that a given ontology is accessed by other ontologies. In Tartir et al. [50], a set of metrics are introduced for assessing the schema and the entire knowledge base (including individuals) of an ontology. Some of these metrics include: relationship richness, which measures the placement of the defined roles in the ontology; inheritance richness, which measures the average number of subclasses per class; attribute richness, which measures the number of attributes per class; and class richness, which measures the distribution of instances between classes. Also, in Tartir et al. [50], a metric for measuring cohesion is introduced, which is equal to the number of separate connected components of the graph that represents a given ontology. It is assumed that two nodes (corresponding to two instances) are connected if there is a relationship between their instances. Following a similar approach in considering ontology graphs for the sake of assessment, the work by Zhang et al. [55] proposes a set of metrics for measuring ontology complexity. These metrics measure the number of concepts, relations, paths and the average number of relations per concepts, the average path per concept, and the average connectivity degree of each concept. The work in Wang et al. [53] is among few existing research on modular ontology evaluation in the literature. It introduces a set of characteristics that modular ontologies need to have. These characteristics include reusability, encapsulation, loose coupling, authorization, self-containment, scalability, and reasoning support. It analyzes different

Faezeh Ensan, Weichang Du / Information Systems 38 (2013) 745–770

modular ontology formalisms to find out how they support these characteristics. Cohesion and coupling are two important measures that can be employed for evaluating modular ontologies. Cohesion refers to the uniformity of an ontology module and coupling refers to the interdependencies that exists between ontology modules. Cohesion and coupling are well-known measurements for evaluating object oriented designs. In Chidamber and Kemerer [13], two objects are defined as coupled if at least one of them affects the other. Cohesion for an object is defined based on the degree of similarity of its methods. In Kramer and Kaindl [32], a set of metrics are introduced for measuring coupling and cohesion in knowledge-based systems. These metrics measure the cohesion of a frame and the coupling of two or more frames by means of the relations between the slots of the frames that are induced through their common references in rules. There are a few proposals in the literature that investigate the notions of cohesion and coupling for ontologies. In Yao et al. [54], the cohesion of an ontology is investigated through some structural metrics such as the number of root classes, the number of leaf classes, and the average depth of inheritance. In Stuckenschmidt and Klein [47], the authors introduce the notion of coherent modules and propose a methodology for partitioning a monolithic ontology to a set of coherent modules, while a coherent module contains a set of concepts that are dependant on each other. In this specific work, only those dependencies are considered that can be derived from the structure of a given ontology (not from its semantic). Authors claim this structural method, contrary to any semantic method, can scale up to large ontologies. In Orme et al. [40], the coupling of an ontology is measured by counting the number of external classes that are used for defining classes and properties in the ontology, the number of references to external classes, and the number of includes in the ontology. In Ma et al. [35], a set of metrics are introduced for measuring cohesion of ontologies. These metrics are: Number of Ontology Partitions (NOP), Number of Minimally Inconsistent Subsets (NMIS) and Average Value of Axiom Inconsistencies (AVAI). These metrics are obtained based on the semantics of a given ontology rather than its syntax. Since these metrics are introduced for changing ontologies, their focus is mostly on inconsistencies that may be induced by ontology axioms.

1.2. Motivation Despite the increasing demand for developing ontologies in a modular manner and the considerable amount of research on modular ontologies in the recent years [49], only a few works have been reported on evaluating modular ontologies and even less accomplished works on metrics for evaluating modular ontologies. In the following, we describe the main arguments that motivate our work:

 The main contributions of a modular approach in the process of ontology development are to bring flexibility





747

for component reuse, support for more efficient query answering and enhance component change and evolution. These important perspectives have not received enough attention in the literature for evaluating ontologies. For assessing the quality of a modular ontology, in addition to the currently discussed criteria for evaluating ontologies in the literature such as domain coverage or possible inconsistencies and syntax errors, we need to evaluate ontologies from a ‘‘design-based’’ perspective, i.e., attempt to perform the following analyses: (understandability) How well can ontologies be understood by a third-party observer? (reusability) How well can a portion of the ontology be understood and shared for being reused in developing ontologies for other domains of discourse? How much of nonrelated knowledge is protected when shared with external users? (evolvability and maintainability) When a portion of the ontology evolves or versioned, how much are the other parts of the ontology affected? (reasoning performance) How efficient is query answering over the ontology when queries are only concerned with a small portion of the ontology? For representing the same body of knowledge, ontologies may be designed in monolithic or modular ways. There may be even more than one modular designs where the number of modules, distribution of concepts, roles and axioms and the relationships between ontology modules are organized differently. Although the advantages of modular designs for ontologies over the monolithic designs have been well discussed in the literature [48], it is hard to find measurements and metrics for evaluating and comparing various ontology designs. For example, assume that there are two modular designs with different number of modules, which represent the same body of knowledge. It is desirable to have a set of metrics that enable us to evaluate these deigns and to investigate which one is better with regards to design-based perspectives such as understandability or evolvability. Most existing metrics for assessing ontologies are syntax based, i.e., they are based on the graph representation of ontologies or only consider the explicitly asserted axioms in the ontology knowledge bases and do not take the semantics of ontologies into account properly [52]. For evaluating modular ontologies and ontology designs, it would be desirable to have a set of semantic metrics that are independent of the representation and the syntax of the knowledge base. Semantic metrics have this considerable advantage that consider not only the explicitly asserted axioms in an ontology but also the implied axioms, which are obtained by reasoning.

1.3. Contributions The objective of this paper is to propose a set of metrics for evaluating DL-based modular and monolithic ontologies from a design-based perspective. For this purpose, we define a set of metrics for ontologies, modular ontologies and also ontology designs. We investigate

748

Faezeh Ensan, Weichang Du / Information Systems 38 (2013) 745–770

the notions of cohesion and coupling, which are highly attached to the design-based perspective for the evaluation of ontologies. This way, highly cohesive and lowly coupled ontology modules can be understood, reused and evolved more easily. We define a set of metrics for assessing cohesion and coupling in ontologies and ontology designs. These metrics are defined based on the definitions of dependencies and relativeness between local symbols of ontologies (as a basis for cohesion) and between their local and external symbols (as a basis for coupling). In view of the fact that the local and external dependencies are defined semantically, all relationships between ontological symbols, either explicitly asserted or not, will be taken into account for the sake of assessment and measurement. The main contributions of this paper can be enumerated as follows:

 We provide a comprehensive basis for evaluating DL-





based ontologies in modular and monolithic designs. We formalize the notion of metrics for evaluating ontology modules and comparing ontology designs. We define a set of metrics for measuring cohesion and coupling in ontologies. These metrics are based on precise semantic-based definitions for dependencies between local and external symbols in ontologies. Based on the cohesion and coupling metrics, we define metrics for evaluating different ontology designs. We present several studies for analyzing the association between the introduced metrics and the time required for consistency checking and running conjunctive queries over ontology modules. We show that consistency checking and query execution are considerably more efficient in highly cohesive ontology modules.

1.4. Outline The rest of this paper is organized as follows: Section 2 gives some preliminaries about DL based ontologies, Interface-Based modular ontology Formalism (IBF) [16,18], and the query dispatching and integrating policy that we employed for running and testing conjunctive queries over modular designs. We use IBF and also ‘‘owl:imports’’ mechanism to model modular designs in our case studies. Among the presented sections, Section 5 is the only dependent section on the content of Sections 2.2 and 2.3 for presenting the case studies. In Section 3, we provide definitions for ontology modules, monolithic and modular ontology designs and metrics for evaluating ontology modules and comparing ontology designs. Section 4 introduces cohesion and coupling for evaluating ontology modules and provides a set of metrics for measuring cohesion and coupling in ontology modules and ontology designs. In Section 5 the proposed metrics are investigated with regards to the reasoning performance. Section 6 provides some discussion on the metrics and directions for future work. Finally, Section 7 concludes the paper.

2. Background 2.1. DL-based ontologies An ontology (a DL knowledge base) is defined as M ¼ /T ,AS; where T denotes TBox, which comprises a set of General Inclusion Axioms (GCIs); and A stands for ABox, which is the assertional part of the knowledge base. A GCI is of the form of ALB, where A and B can be name concepts (also called atomic concepts) or complex concept descriptions [2]. Complex descriptions are built based on concept constructors, role constructors, and named concepts and roles. Different Description Logics provide different sets of concept and role constructors. For example, the Description Logic ALC has these concept constructors: concept conjunction ðuÞ, concept disjunction ðtÞ, concept complement ð:Þ, universal restriction ð8Þ, and existential restriction ð(Þ. For a more complete discussion about DLs constructors see Baader et al. [3]. The signature of an ontology M, which is denoted as SigðMÞ, is defined as a set of all concept names, role names, and individual names in its TBox and ABox. The semantics of an ontology is given by means of interpretations. An interpretation I ¼ ðDI ,I Þ consists of a non-empty set DI and a mapping function I , which maps each concept C 2 SigðMÞ to C I D DI , each role R 2 SigðMÞ to RI D DI  DI and each individual a 2 SigðMÞ to aI 2 DI . Function I is extended to map complex concepts and roles that are defined based on the constructors of different Description Logics (a more complete explanation on the various DLs concept and role constructors can be found in Baader et al. [3]). An interpretation I satisfies a general inclusion axiom CLD iff C I DDI , satisfies an assertion C(a) iff aI 2 C I and an assertion Rðx,yÞ iff /xI ,yI S 2 RI . The satisfaction relation between I and the other axioms, which are defined by more expressive Description Logics, such as those that support role hierarchy and inverse roles, can be found in [3,31]. An interpretation I is a model of a knowledge base M if it satisfies every axiom and assertion in the TBox and ABox of M. An axiom a is implied by a knowledge base M and denoted as MFa iff all models of M satisfies a. 2.2. Interface-based modular ontology formalism The Interface-Based modular ontology Formalism (IBF) [16,18] is a formalism for developing modular ontologies. Through this formalism, a modular ontology is defined as a set of ontology modules and interfaces. In IBF, the dependencies between modules have been modeled through interfaces. An ontology module can either realize or utilize an interface. In fact, realizer modules export the public part of their TBox through interfaces to be reused by utilizer modules. A utilizer module augments its knowledge base with the definitions and assertions provided by the realizer modules. In the definition of modular ontologies, a configuration function exists that specifies which utilizer and realizer modules should be connected to each other. Based on IBF, ontology modules can communicate through interfaces and not directly, which is an important step to reduce the tightly coupled-ness of ontologies. IBF follows an augmentation approach for sharing ABox assertions through ontology

Faezeh Ensan, Weichang Du / Information Systems 38 (2013) 745–770

749

Fig. 1. An IBF modular ontology for academic people and publications.

modules. Accordingly, IBF uses conjunctive or epistemic queries to retrieve all individuals of interface concepts and roles from a realizer module and augments the domain of the utilizer module with these individuals. Formal definitions for IBF syntax and semantics can be found in Ensan and Du [18]. Here, we just describe its application for defining modular ontologies through an example. Fig. 1 shows an IBF modular ontology. This modular ontology is comprised two ontology modules: OMPublication and OM-Person and one interface: Inf-Person. OM-Person is an ontology module, which describes people in an academic context. OM-Publication describes academic publications such as conference papers, books and manuals.1 OM-Person realizes the interface while OM-Publication utilizes it. OM-Publication uses the symbol in Inf-Person in its TBox in the axiom: >L8 publicationAuthor.Inf-Person:Person. According to IBF, for augmentation, OM-Publication should pose queries over OM-Person for the concepts and roles of the interface (in this case there is only one interface concept, which is Person). After retrieving individuals, the following assertions are inserted into the OM-Publication ontology module: Inf-Person:Person(John), Inf-Person:Person(Mary), Inf- Person:Person(Elizabeth). 2.3. Query execution Conjunctive queries are expressive query languages for searching ABoxes of knowledge bases and can be formulated 1 The content of OM-Person and OM-Publication is inspired from the Univ-Bench ontology of the LUBM [26] framework.

in different language such as SPARQL [42]. In Section 6, we will analyze the required time for answering conjunctive queries over modular ontologies. Here, we briefly explain our methodology for executing conjunctive queries over modular ontologies. Consider the following simple query: SELECT ?X WHERE f ?X rdf:type Publication. g

This query has one symbol: Publication which is a local symbol of OM-Publication ontology module. Hence, this query is posed to OM-Publication. Now consider the following query, a more complex conjunctive query that has symbols that belong to local symbols of different ontology modules. SELECT ?X ?Y WHERE f ?X rdf:type Publication. ?X publicationAuthor ?Y. ?Y rdf:type Person g

This query has three symbol: Publication and publicationAuthor, which are local symbols of OM-Publication ontology module; and Person, which is a local concept of OM-Person module. In this case, the query is broken down into two smaller queries: SELECT ?X ?Y WHERE f ?X rdf:type Publication. ?X publicationAuthor ?Y g

750

Faezeh Ensan, Weichang Du / Information Systems 38 (2013) 745–770

SELECT ?Y WHERE f ?Y rdf:type Person g

where the first query is posed to OM-Publication and the second query is posed to OM-Person. The result set of the original query is formed by joining the result sets of these sub-queries. 3. Evaluation of ontologies: modular and monolithic designs In this section, we first clarify our interpretation of notions of ontology modules and ontology designs. We give precise definitions of metrics for evaluating ontology modules and ontology designs. Up until now, ontology modules and modular ontologies have been defined differently in the literature [48,18,6,24]. The basic notion in which most existing definitions coincide is that ontology modules have references to external symbols from other ontologies. More formally, an ontology module M is a DL knowledge base whose signature consists of two disjoint sets of local symbols ðLocðMÞÞ, and external symbols ðExtðMÞ). External symbols are the local symbols of other ontology modules, and SigðMÞ  LocðMÞ ] ExtðMÞ. This definition is inclusive for ordinary ontologies that are developed based on the monolithic approach. Accordingly, monolithic ontologies are ontology modules whose set of external symbols are empty. In the following, we present a definition for numerical metrics that measure ontology modules. Since the definition of ontology modules includes cases where ontologies are designed monolithically, the following definition can also be used for defining metrics for evaluating monolithic ontologies. For defining ontology metrics, we follow the approach that is employed in Chidamber and Kemerer [13] for defining software metrics for object oriented systems. Definition 1. Let fM1 , . . . ,Mn g be a set of ontology modules. Further, assume P is a binary relation on these ontology modules that corresponds to a specific perspective for measuring ontology modules. Further, let fq1 , . . . ,qn g be a set of numbers and S be a binary relation on these numbers, such as 4, o or ¼. A numerical metric m maps a tuple of ontology modules and the relationship between them ð/fM1 , . . . ,Mn gÞ,PS to a set of numbers and a binary relation on numbers /fq1 , . . . ,qn g,SS. As an explanatory example, assume there are three ontology modules M1 , M2 and M3 . Further, assume P is a relationship between these ontology modules from the reusability perspective such that PðM1 ,M2 Þ means that M1 is more reusable than M2 . Consider three arbitrary numbers q1 ¼ 10, q2 ¼ 5 and q3 ¼ 2 and the relation S on these numbers be ‘‘less-than’’ or ‘‘ o ’’. m is a numerical metric that assigns M1 to q1, M2 to q2, M3 to q3 and P to S. Based on m, since q3 o q2 oq1 , M3 is more reusable than M2 and M2 is more reusable than M1 .

For introducing a numerical metric for measuring ontology modules, it should be shown how the corresponding numbers are produced for each ontology module. In addition, it should be justified how the relation between numbers (S) corresponds to the relationship between ontology modules from a specific perspective (P). Example 1. Different ontology modules have different levels of understandability. Hence, there is a relationship P between ontology modules from the understandability perspective such that PðM1 ,M2 Þ means that ontology module M1 is more understandable than M2 . As a simple example for numerical metrics, assume that the size of ontology modules is defined for measuring ontology modules from the understandability perspective. The justification would be that ontology modules with smaller sizes represent a more focused portion of the domain of discourse and can be understood more easily. The size of an ontology module is defined as the number of symbols in its signature, i.e., for ontology module M, SizeðMÞ ¼ 9SigðMÞ9. For ontology modules Mi , SizeðMi Þ is the number based on which the relationship S (‘‘lessthan-equal’’ ( r )) can be defined. The numerical metric Size assigns a number to each ontology module. The relationship S that exists between these numbers corresponds to the relationship P between ontology modules. Definition 2 formally defines ontology designs: Definition 2. An ontology design D is defined as a tuple /M,SymS where M is the set of ontology modules. For each M 2 M, there is a set of ontology modules N DM, S such that M= 2N and ExtðMÞ D Mi 2N LocðMi Þ. Sym is the set of all local symbols of ontology modules in the design: S Sym  Mi 2M LocðMi Þ. The following example shows an ontology design and its components for representing the Tourism domain. Example 2. Let OM-Accommodation and OM-Attractions be two ontology modules for describing accommodations and attractions of the Tourism domain, respectively. The knowledge base of OM-Accommodation is fHotelL Accommodation, MotelLAccommodationg. The knowledge base of OM-Attraction is fBeachLSightseeing, MuseumL Sightseeing, AppropriateSightseeingLSightseeing u ( isCloseTo. OM-Accommodation: Accommodation g. D ¼ /M,SymS is an ontology design for the Tourism domain where M ¼ fOM  Accommodation; OM  Attractiong and Sym ¼ fHotel; Motel; Accommodation; Beach; SightSeeing; Museum; AppropriateSightSeeing; isCloseTog. In Definition 2, 9M9 denotes the number of ontology modules that the ontology design D has. If 9M9 is equal to one, D is a monolithic design and its only module does not have any references to any external ontologies. Example 3. Consider the Tourism domain that is described in Example 2. This domain can be represented through the monolithic ontology design D0 ¼ /M0 ,Sym0 S, where M0 includes just one ontology module: OM-Tourism.

Faezeh Ensan, Weichang Du / Information Systems 38 (2013) 745–770

The knowledge base of the OM-Tourism is: fHotelL Accommodation, MotelLAccommodation, BeachL Sightseeing, MuseumLSightseeing, AppropriateSightseeingL Sightseeingu (isCloseTo. Accommodation g. Here Sym0 is equal to Sym in Example 2. Similar to the definition of metrics for ontology modules, we can define metrics for ontology designs as follows:

Definition 3. Let fD1 , . . . ,Dn g be a set of ontology designs. Assume P to be a binary relation on ontology designs that corresponds to a specific perspective such as maintainability, reusability and reasoning performance. Further, let fq1 , . . . ,qn g be a set of numbers and S be a binary relation on these numbers, e.g., 4, o or ¼. A numerical metric m maps a tuple of ontology designs and the relationships between them /fmathcalD1 , . . . ,Dn g,PS to a set of numbers and a binary relation on numbers /fq1 , . . . ,qn g,SS. For example, assume that metric m measures the average size of ontology modules in an ontology design for evaluating their understandability. For an ontology design D ¼ /M,SymS, m ¼ 9Sym9=9M9. This metric is based on the hypothesis that an ontology design that includes ontology modules with smaller average size can be understood better by a third-party observer. Based on this metric, mðD1 Þ o mðD2 Þ implies that ontology design D1 and its ontology modules are more understandable than ontology design D2 and its constituent ontology modules. For instance, the value of m for the ontology designs D and D0 , the ontology designs that are described in Examples 2 and 3, is equal to 9Sym9=9M9 ¼ 9=2 ¼ 4:5 and 9Sym0 9=9M0 9 ¼ 9=1 ¼ 9, respectively. This metric implies that ontology design D is more understandable than D0 .

4. A set of metrics for evaluating ontology modules In this section, we introduce cohesion and coupling measurements for evaluating ontology modules and ontology designs. Cohesion and coupling are two important concepts for evaluating software designs that have been thoroughly investigated in the Software Engineering community [13,29]. Analogously, these notions can be employed for evaluating ontologies and ontology designs. Good ontology designs should aim at minimizing coupling and maximizing cohesion. An ontology module is considered to be highly cohesive if the information conveyed by that module are all describing a very specific sub-domain of discourse, i.e., they are focusing on the description or employment of a focused group of related concepts. An ontology module is considered to be lowly coupled if it has the least amount of dependency on external information. In other words, the concepts and roles of a low-coupled ontology module are defined based on or use the minimum possible external concepts and roles from other ontology modules.

751

4.1. Semantic dependencies In this section, we provide methods for measuring cohesion and coupling in ontology modules and ontology designs. Cohesion in an ontology module is concerned with how ‘‘related’’ and ‘‘dependent’’ its concepts and roles are. Coupling means how ‘‘related’’ and ‘‘dependent’’ its concepts and roles are on the external symbols from external ontology modules. For a better explanation, consider the following three ontology modules: M1 ¼ fA,Bg, A and B are local concepts. M2 ¼ fALBg, A and B are local concepts. M3 ¼ fALBg, A is a local concept and B is an external concept form an external ontology module M0 3.

In this example, in the first situation, A and B are two unrelated concepts that are collocated in the ontology module M1. They may be diverse pieces of information that describe unrelated subjects. Hence, ontology module M1 is not highly cohesive. In the second situation, A is defined as a subclass of B and hence A and B are strongly related to each other. Here, ontology module M2 is highly cohesive, i.e., it describes a related sub-domain of discourse. In the third situation, A is strongly dependent on B, which is an external symbol from the foreign ontology module M0 3. Hence, in this ontology design, M is highly coupled on M0 3. Conclusively, the more dependencies are between the local symbols of an ontology module, the more cohesive it is. Further, the more dependencies are between local symbols of an ontology module and external symbols from foreign modules, the more coupled it is on foreign ontology modules. In the following, in order to precise the definition of cohesion and coupling, we first provide formal definitions for dependencies between concepts and roles in ontology modules. Based on these definitions, we can quantify the dependencies between local symbols and between local and external symbols in an ontology module. Subsequently and based on the quantities of the dependency relationships, we measure the cohesion and coupling of ontology modules. Finally, we define cohesion and coupling for ontology designs based on the values of the coupling and cohesion of their ontology modules. A concept or a role in an ontology module is dependent on the other, if its semantics are limited or bounded, once the semantic of the other has been set up. For example, assume an ontology module with two concepts A, B and an axiom, which asserts ALB. In this knowledge base, A is dependant on B, i.e., once B has been interpreted to be a subset of the domain elements, the valid interpretations of A are limited to those that interpret A as a subset of B. As another example, consider an ontology module with the axiom AL(R:C. In this ontology the interpretations of A are limited, once the semantic of C has been established. The dependency of a concept on a role happens when the semantics of the concept are limited by the domain or the range of that role. For instance, back to the previous example of an ontology with the axiom: AL(R:C, once R has been interpreted to be a set of the ordered pairs of

752

Faezeh Ensan, Weichang Du / Information Systems 38 (2013) 745–770

elements, the semantics of A are bounded to the set of domain elements of R. Roles can have dependencies on each other, as well. For example, in an ontology with a role axiom R1 LR2 , R1 is dependent on R2. The dependency of a role on a concept happens when its interpretations are limited once the semantic of the concept is specified. For example, in an ontology with the axiom (R:>LC, R is dependent on C. In order to technically formalize dependencies in ontology modules, we define two types of dependencies: strong dependencies and moderate dependencies. Definition 4 provides a definition for strong dependencies in ontology modules: Definition 4. Let A and B be two concepts and P and R two roles where A,B,P,R 2 SigðMÞ and M is an ontology module. Further, assume MjA  ?, MjA  >, MjB  ?, MjB  >, Mj(P:>  ?, and Mj(P:>  >. Let MB be obtained by adding the axiom BL ? to M. Further let M þ B be obtained by adding the axiom >LB to M. A is strongly dependent on B, if one of the following happens: 1. 2. 3. 4.

MB FA  ? M þ B FA  ? MB FA  > M þ B FA  >

Moreover, the role R is strongly dependent on B if either MB F(R:>  ?, MB F(R:>  >, M þ B F(R:>  ?, or M þ B F(R:>  >. Let MP be obtained by adding the axiom (P:>L ? to M and M þ P be obtained by adding the axiom >L(P:> to M. A is strongly dependant on P if either MP FA  ?, MP FA  >, M þ P FA  ?, or M þ P FA  >. Further, the role R is strongly dependent on P if either MP F(R:>  ?, MP F(R:>  >, M þ P F(R:>  ?, or M þ P F(R:>  >. Definition 5. We define SDepM ðAÞ as the set of all symbols in M on which A is strongly dependent. Intuitively, a strong dependency occurs on the extreme boundaries: when a concept B or the domain of a role P is interpreted as an empty set or as the top concept. In these situations, a concept A, which is strongly dependant on B or P, is interpreted as an empty set or the top concept. Example 4. In the following, we investigate strong dependencies in ontology modules in different situations based on Definition 4:

1. Let M ¼ fALBg, MB ¼ fALB,BL ?g, MB FAL ?, consequently, A is strongly dependent on B. 2. Let M ¼ fALBg, M þ A ¼ f>LA,ALBg, M þ A F>LB, consequently, B is strongly dependent on A. 3. Let M ¼ fAL:Bg, M þ B ¼ fAL:B,>LBg, M þ B FAL ?, consequently, A is strongly dependent on B. 4. Let M ¼ fAL(R:Cg, MC ¼ fAL(R:C,CL ?g, MC F AL ?, consequently, A is strongly dependent on C. 5. Let M ¼ fAL(R:Cg, MR ¼ fAL(R:C,(R:>L ?g, MR F AL ?, consequently, A is strongly dependent on R.

6. Let M ¼ fR1 LR2 g, MR2 ¼ fR1 LR2 ,(R2 :>L ?g, MR2 F (R1 :>L ?, consequently, R1 is strongly dependent on R2. 7. Let M ¼ f(R:>L:Ag, M þ A ¼ f(R:>L:A,>LAg, M þ A F (R:>L ?, consequently, R is strongly dependent on A.

Moderate dependency is concerned with situations where the interpretations of a concept or a role are conditionally limited by the interpretations of the other symbols. For instance, assume an ontology module with three concepts A, B and C and an axiom: ALB t C. In this knowledge base, once the interpretations of the concept B and C have been set up and they have been interpreted as a set of domain elements, the valid interpretations of A are limited to those that interpret A as a subset of B or C. Now, assume that C is interpreted as an empty set. In this condition, the semantics of A are bounded to the semantics of B. We say A is moderately dependent on B and C. Definition 6 defines moderate dependencies between symbols of an ontology module. Definition 6. Let P and Q be two symbols in ontology module M, and MQ and M þ Q be defined similarly to Definition 4. P is moderately dependent on Q, if P is not strongly dependent on Q and either SDepM ðPÞ differs from SDepMQ ðPÞ or SDepM ðPÞ differs from SDepM þ Q ðPÞ. Definition 7. We define MDepM ðPÞ as a set of all symbols in M on which P is moderately dependent. Intuitively, when a concept or domain of a role is interpreted as an empty set or as the top concept, the interpretations of some concepts and roles may be affected in such a way that they find new strong dependencies to other symbols. For instance, in ontology module M with ALB t C, SDepM ðAÞ ¼ |, i.e., A does not have any strong dependency on other symbols. However in this knowledge base, if B is interpreted as an empty set, A will have strong dependencies on C, so SDepMB ðAÞ ¼ fCg, which differs from SDepM ðAÞ. Hence, according to Definition 6, A has a moderate dependency on B. The following examples show moderate dependencies in other knowledge bases. Example 5. Let M be an ontology module with this axiom A u CLD. In this ontology module, A is strongly dependent on no symbol (SDepM ðAÞ ¼ |). In this knowledge base, M þ C implies that ALD. Accordingly, SDepM þ C ðAÞ is equal to fDg, which differs from SDepM ðAÞ; and hence A is moderately dependent on C. Further, MD implies that AL:C. Accordingly, SDepMD ðAÞ is equal to fCg, which differs from SDepM ðAÞ; and hence A is moderately dependent on D. In this ontology module, MDepM ðAÞ ¼ fC,Dg. Example 6. Let M be an ontology module with following axioms: ALB ALðB u (R:CÞ t D ALB t :K

Faezeh Ensan, Weichang Du / Information Systems 38 (2013) 745–770

In this ontology, A is strongly dependent on B, so SDepM ðAÞ ¼ fBg. Now, we investigate the moderate dependency of A on C, R, K and D.

 MC implies that ALB,ALD,ALB t :K. Accordingly,  



SDepMC ðAÞ ¼ fB,Dg, which differs from SDepM ðAÞ. Consequently, A is moderately dependent on C. MR implies that ALB,ALD,ALB t :K. Accordingly, SDepMR ðAÞ ¼ fB,Dg, which differs from SDepM ðAÞ. Consequently, A is moderately dependent on R. MK implies that ALB,ALðB u (R:CÞ t D, AL>. Accordingly, SDepMK ðAÞ ¼ fBg which does not differ from SDepM ðAÞ. Additionally, MþK implies that ALB,ALðB u (R:CÞ t D. Accordingly, SDepMK ðAÞ ¼ fBg which does not differ from SDepM ðAÞ. Consequently, A is not moderately dependent on K. MD implies that ALB,ALðB u (R:CÞ,ALB t :K. Accordingly, SDepMD ðAÞ ¼ fB,R,Cg, which differs from SDepM ðAÞ. Consequently, A is moderately dependent on D.

The notion of dependency between symbols of an ontology module, in the sense that we defined earlier in this section, is more general than those dependencies that are captured by strong and moderate definitions. For example, in an ontology module with the axiom ALB t C t D, the semantics of A are affected by the semantics of B. Nonetheless, there is neither a strong nor a moderate dependency between A and B according to Definitions 4 and 6, respectively. In fact, in this situation, A has a ‘‘moderately’’ moderate dependency on B, i.e., the semantics of A are limited by the semantics of B if the semantics of both C and D have been already set up. Theoretically it is possible to find more semantic dependencies in an ontology module than those that have been captured by Definitions 4 and 6. Definition 8 provides a basis for finding more complex moderate dependencies—e.g., a dependency between A and Bi in a module with the axiom ALB1 t B2 t    tBi t    t Bn . Definition 8. Let P and Q be two symbols in ontology module M and MQ and M þ Q be defined similarly to Definition 4. The moderate dependency, which is defined in Definition 6, is a first-degree moderate dependency and is denoted as P1 Q . Furthermore, MDep1M ðPÞ is the set of all symbols in M on which P is first-degree-moderately dependent. For every n Z 2, we define Pn Q if P is not strongly dependent on Q and P is not moderately dependent on Q of the degree of m on and also either n1 n1 MDepn1 M ðPÞ differs from MDepMQ ðPÞ or MDepM ðPÞ differs from MDepn1 ðPÞ. MþQ Example 7. Let M be an ontology module with the axiom A u B u CLD. In this ontology module MDep1M ðAÞ ¼ |. But, MDep1M þ B ðAÞ ¼ fC,Dg (see Example 5), which differs from MDep1M ðAÞ, so A is second-degree-moderately dependant on B, i.e., A2 B. Obviously, Definition 8 induces a considerable high computational cost. In this paper, we restrict the definition of the evaluation metrics just to be

753

based on the dependencies that are defined in Definitions 4 and 6 and leave the investigation of low-cost algorithms for finding more complex semantic dependencies between ontological elements for future work.

4.2. Semantic metrics In order to define semantic metrics based on semantic dependencies, we distinguish between the dependencies between local symbols and the dependencies of local symbols on the external ones. The former represents the cohesion of an ontology module while the latter shows its coupling on other modules. In the following, we formally define local and external strong and moderate dependencies. Definition 9. For ontology module M: SDepLocðMÞ ¼ f/P,Q S9P 2 LocðMÞ, Q 2 SDepM ðPÞ and Q 2 LocðMÞg. SDepExtðMÞ ¼ f/P, Q S9P 2 LocðMÞ,Q 2 SDepM ðPÞ and Q 2 ExtðMÞg. MDepLocðMÞ ¼ f/P,Q S9P 2 LocðMÞ,Q 2 MDepM ðPÞ and Q 2 LocðMÞg. MDepExtðMÞ ¼ f/P,Q S9P 2 LocðMÞ,Q 2 MDepM ðPÞ and Q 2 ExtðMÞg.

Based on the definitions of dependency sets, we introduce two metrics for measuring cohesion and two metrics for measuring coupling in ontology modules. Definition 10. For an ontology module M, NSLD and NMLD are two metrics for measuring its cohesion that are defined as follows: NSLD

NMLD

is the Number of Strong Local Dependencies that exist between local symbols of an ontology module. NSLDðMÞ ¼ 9SDepLocðMÞ9, i.e., the size of the SDepLoc set for ontology module M. is the Number of Moderate Local Dependencies that exist between local symbols of an ontology module. NMLDðMÞ ¼ 9MDepLocðMÞ9, i.e., the size of the MDepLoc set for ontology module M.

NSLD and NMLD are numerical metrics that assign numbers to ontology modules according to Definition 10. There is a ‘‘less-than-or-equal’’, ‘‘ Z ’’, relationship between the numbers that have been assigned to ontology modules. This relationship associates with the cohesion of ontology modules. The larger the values assigned by NSLD and NMLD to the ontology modules are, the more cohesive the ontology modules would be. Definition 11 introduces two metrics for measuring coupling in ontology modules.

754

Faezeh Ensan, Weichang Du / Information Systems 38 (2013) 745–770

Definition 11. For an ontology module M, NSED and NMED are two metrics for measuring its coupling on external ontology modules and are defined as follows: NSED

NMED

is the Number of Strong External Dependencies that exist between local symbols of an ontology module on its external symbols. NSEDðMÞ ¼ 9SDepExtðMÞ9, i.e., the size of the SDepExt set for ontology module M. is the Number of Moderate External Dependencies that exist between local symbols of an ontology module on its external symbols. NMEDðMÞ ¼ 9MDepExtðMÞ9, i.e., the size of the MDepExt set for ontology module M.

NSED and NMED are numerical metrics that assign numbers to ontology modules according to Definition 11. There is a ‘‘less-than-or-equal’’, ‘‘ r ’’, relationship between the assigned numbers by these metrics. This is related to the coupling of ontology modules. The lower values that are assigned by these metrics to ontology modules, the less coupled the ontology modules are on foreign ontology modules. Let us remark an important point here: SDepLoc, MDepLoc, SDepExt and MDepExt are defined as sets and therefore do not have any redundant ordered pairs of symbols. Observably, the definitions for metrics for cohesion and coupling prevent double counting:

of ontology module M and is defined as following: 8 if 9LocðMÞ9 ¼ 1 > <1 COHðMÞ ¼ g  NSLDþ d  NMLD > otherwise : ðg þ dÞ  MLD

ð1Þ

In Definition 13, cohesion is measured by a weighted average of the number of all strong and moderate dependencies that are between local symbols in the ontology module over the number of all potential local dependencies that could possibly exist. A highly cohesive ontology module has a larger number of local dependencies relative to the maximum number of local dependencies that may exist. When there is just one local symbol, when MLD is zero, the value of the metric is defined to be equal to 1. COH is an absolute metric which assigns a value between zero and one to ontology modules. Since MLD is always more than NSLD and NMLD, the upper bound of COH is one. An inclusive metric for measuring coupling of ontology modules, based on NSED and NMED metrics is defined as follows: Definition 14. Let M be an ontology module and g and d be two real numbers such that g, d 4 0. COPðMÞ, the coupling of ontology module M is defined as follows: COPðMÞ ¼

8 > <0

if 9ExtðMÞ9 ¼ 0

g  NSED þ d  NMED

> : ðg þ dÞ  ð9LocðMÞ9  9ExtðMÞ9Þ

otherwise

ð2Þ

MLD is a numerical metric that assigns numbers to ontology modules. Ontology modules with smaller local size, which leads to less value for MLD, are more probable to describe a focused sub-domain of discourse. They are more understandable and are more appropriate for reuse. Observably, there is a ‘‘less-than-or-equal’’, ‘‘ r’’, relationship between the assigned numbers by MLD that associates with cohesion of ontology modules. Definition 13 provides an inclusive metric for measuring the cohesion of ontology modules based on the values of the NSLD, NMLD and MLD metrics.

Coupling is calculated based on a weighted average of the number of all strong and moderate dependencies that exist between local symbols on the external ones in the ontology module over the number of all potential external dependencies that could possibly exist. Given an ontology module M, at most 9LocðMÞ9  9ExtðMÞ9 dependencies can exist between its local symbols and the external ones. COP has a value between 0 and 1. According to these definitions, strong and moderate dependencies have different impacts on the cohesion and coupling of an ontology module. This difference is driven by different values of their corresponding coefficients in Eqs. (1) and (2). Since strong dependencies (measured by NSLD and NSED) represent a stronger type of relationship between ontological symbols (such as concept subsumption) comparing with a moderate type (measured by NMLD and NMED), it looks reasonable to assign a higher value to g comparing with d. In different domains and applications, the exact values of the coefficients for strong and moderate dependencies are indicated based on the discretion of domain experts and ontology evaluators. In the following and for our examples and case studies, we consider g be twice larger than d (in our experiments, we assigned g ¼ 2 and d ¼ 1). Based on the introduced measures, we now define a cohesion and a coupling metric for evaluating different ontology designs as follows:

Definition 13. Let M be an ontology module that has at least one local symbol and g and d be two real numbers such that g, d 4 0. COH is a metric for measuring cohesion

Definition 15. For an ontology design D ¼ /M,SymS: P COHðMÞ  9LocðMÞ9 COHDes ðDÞ ¼ M2MP ð3Þ M2M 9LocðMÞ9

Remark 1. Definitions 10 and 11 for cohesion and coupling metrics prevent double counting. In addition to the number of strong and moderate dependencies, Definition 12 introduces another metric for measuring cohesion in ontology modules. According to this definition, MLD is the Maximum number of Local Dependencies, which can exist between local symbols of an ontology module. Definition 12. MLD

Let n ¼ 9LocðMÞ9, MLD is equal to n  ðn1Þ.

Faezeh Ensan, Weichang Du / Information Systems 38 (2013) 745–770

P COPDes ðDÞ ¼

 9LocðMÞ9 9LocðMÞ9 M2M

M2M COHðMÞ

P

ð4Þ

Intuitively, the cohesion of an ontology design is dependent on the cohesion values of its contributing ontology modules. Similarly, the coupling of an ontology design is dependent on the coupling values of its ontology modules. COHDes and COPDes are numerical metrics for assessing ontology designs. They assign numbers to each ontology design. There is a ‘‘greater-than-or-equal-relationship’’ between the numbers which are assigned by COHDes. This relationship associates with the cohesion of an ontology design such that higher value for the metric means that the ontology design is more cohesive. There is a ‘‘less-than-or-equal-relationship’’ between the numbers which are assigned by COPDes which means that a lower value for the metric shows that the ontology design is less coupled with the foreign ontology modules. 5. Empirical study of the modular ontology metrics w.r.t. query answering performance In this section, we empirically investigate the association between the introduced metrics in the previous sections and the time that a reasoning engine needs for consistency checking and answering conjunctive queries over ontology modules.

755

measures of the number of strong and moderate dependencies that exist between local and external symbols in an ontology module. The more dependent the local symbols of an ontology are to each other, the larger values NSLD and NMLD have. Moreover, the more dependent the local symbols of an ontology are on its external symbols, the larger values NSED and NMED have. Hence, the more cohesive an ontology module is, i.e., related knowledge of a specific sub-domain is gathered in one or few modules and is not spread all over the knowledge base, the more values NSLD and NMLD would have. MLD is a metric that measures the most number of strong and moderate dependencies that can exist between local symbols in an ontology module. A large value for a fraction like NSLD=MLD for a given module implies that the local symbols of the module are well dependant (there is a large number of dependencies between them comparing with all decencies that could exit in the best case), which means that the module is describing a focused sub-domain. Finally, COH and COP, COHDes , COPDes are compound metrics, which are defined based on the previously mentioned metrics. COHDes , COPDes have theory-based models that an ontology design is more cohesive when its constituent ontology modules are more cohesive and it is more coupled when its constituent ontology modules have coupling on each others.

5.2. Hypotheses 5.1. Cohesion and coupling metrics and query answering time Cohesion and coupling can be considered to be associated with the time that reasoning engines need for query answering and reasoning. The performance of reasoning in ontologies can be affected by various parameters. Different DLs, such as ALC, SHION , SHIQ, and SHOIQ, induce different time complexities. In addition, the number of classes, roles, axioms and assertions in ontologies affects the time required for query answering and reasoning tasks. Nonetheless, all these parameters being equal, the design of an ontology also can have influence on the reasoning performance. As an explanatory example, assume two designs for an ontology, first the ontology is comprised a couple of highly cohesive, low-coupled ontology modules and second, a monolithic design, where there is just one complex ontology module that has the whole knowledge base. In the first case, an incoming query may be answered by applying reasoning algorithms on one or a limited number of interrelated ontology modules. On the other hand in the second case, for answering all queries the whole knowledge base should be processed even though the query is related to a small portion of the ontology module. The introduce metrics in this paper represent cohesion of ontology modules and coupling between them in the sense important for measuring query answering time. NSLD and NMLD are measures of the number of strong and moderate dependencies that exist between local symbols of an ontology module. NSED and NMED are

Cohesion represents the degree of similarity between local concepts and roles in an ontology module. Consider two ontology designs that describe the same domain and represent the same set of concepts, axioms and assertions, while they differ in the number of ontology modules and the distribution of domain knowledge in their modules. In this setting, we expect that the time required for answering a query over a highly cohesive ontology module from an ontology design to be less than when the query is posed to a lowly cohesive ontology module from the other design. By assuming that ontology designs represent the same domain information, we ensure that query answering time is just affected by the quality of the design and not influenced by other criteria such as size and complexity of the entire domain. We refer to ontology designs that represent the same body of knowledge as comparable designs. Hypothesis 1 formally describes the association between the cohesion metric and the time required for answering queries over ontology modules. Hypothesis 1 Assume D1 ¼ /M1,Sym1S and D2 ¼ /M2,Sym2S be two comparable ontology designs, where M1 2 M1 and M2 2 M2. Assume Q be a query that can be answered by both ontology modules M1 and M2. If COHðM1Þ 4 COHðM2Þ, the time required for answering Q over M1 is less than the time required for answering Q over M2 when the system and tool setting are the same in both cases.

756

Faezeh Ensan, Weichang Du / Information Systems 38 (2013) 745–770

The exact definition for comparable designs depends on the modular formalisms that are employed for representing modular ontologies. In case of OWL imports and IBF, Definition 16 gives a method for finding comparable designs. There is some other works in the literature that can be used to define comparable designs in other formalisms such as E-Connections [25]. Definition 16. Let D1 ¼ /M1,Sym1S and D2 ¼ /M2, Sym2S be two ontology designs. Assume O1 to be an ontology module that includes all symbols, axioms and assertions in all ontology modules in D1. Further, assume O2 to be an ontology that includes all symbols, axioms and assertions in all ontology modules in D2. We say D1 and D2 are comparable designs if Sym1  Sym2 and for each axiom a in TBox or ABox of O1, O2Fa and for each axiom b in TBox or ABox of O2, O1Fb. For this definition, it is assumed that the sets of local symbols of ontology modules in ontology designs are disjoint. Based on Definition 16, all symbols of different ontology modules are mapped into one monolithic ontology, and then the monolithic ontologies related to different designs are compared with each other. For running a conjunctive query over an ontology module, its ABox consistency should be checked first. We expect that an ontology module with a low cohesion, which represents various concepts, roles and ABox assertions besides the subject of the query, needs more time for ABox consistency checking rather than a highly cohesive one, which focuses on representing the subject of the query. Hypothesis 2 For the conjunctive query Q and ontology modules M1 and M2 that are described in Hypothesis 1, the time required for answering Query Q including ABox consistency checking over M1 is less than this time over M2, when the system and tool setting are the same in both cases. Coupling represents how dependent an ontology module is on others for representing a sub-domain of discourse. The more coupled a modular design is, the higher the probability for involving a larger number of ontology modules for a given query would be. The number of involved modules is important especially in distributed designs, where ontology modules are deployed on different servers, and the cost of the communication between modules and the integration of results is considerable. 5.3. Evaluation of ontologies and queries For the sake of evaluation, we analyze query answering over three ontologies in different designs as follows: 1. Lehigh University Benchmark (LUBM) [26]: LUBM is a benchmark for the evaluation of Semantic Web techniques. LUBM includes an ontology, which is called Univ-Bench, which describes universities, departments and academic people and activities. Based on

Univ-Bench and using the IBF formalism, we designed two comparable modular ontology designs for describing universities and academic activities. LUBM provides 14 extensional queries for test and evaluation. In order to make the difference between query answering times over different ontology modules more significant, we modify the LUBM queries. We also analyzed two more queries over LUBM from Tzoganis et al. [51] (Query 15 and Query 16). Appendix A gives the queries that we used for the evaluation. 2. VICODI2: VICODI is an ontology about European History, which is developed as a component of the VICODI contextualization system [39]. Using the Neon [28] toolkit and its plugins for ontology modularization, we designed two comparable modular designs for VICODI. For analyzing query answering time, we evaluated the following five queries over VICODI, some of which are taken from Motik and Sattler [38]: Query1 (x) Location(x) Query2 (x,y,z)

Military-Person(x), hasRole(y,x), related(x,z)

Query3 (x,y)

Time-Dependent-Relation(x), hasRelationMember(x,y), Event(y)

Query4 (x,y)

Object(x), hasRole(x,y), Symbol(y)

Query5 (x)

Individual(x), hasRole(x,y), Scientist(y), hasRole(y,z), Discoverer(z), hasRole(z,m), Inventor(M)

3. DBPedia [7]: DBPedia is an ontology for making the structured information of Wikipedia available on the Web. Using Neon toolkit, we designed two comparable modular designs for DBPedia. Appendix B gives the four queries that we designed based on the sample queries provided by the DBPedia project.3 In developing modular designs based on the above mentioned ontologies, we checked modular designs for being comparable. For this purpose, we checked the following actions after designing a modular design:

 Each symbol in a monolithic ontology is represented in  

exactly one module of its corresponding modular design. Each TBox axiom in a monolithic ontology is either explicitly or implicitly represented in at least one module of its corresponding modular design. Each ABox assertion in a monolithic ontology is represented in at least one module of its corresponding modular design.

5.3.1. Ontology designs In this section, we describe the modular ontology designs that we developed for the TBox of LUBM Univ-Bench, VICODI, 2

http://www.vicodi.org See http://wiki.dbpedia.org/Datasets and http://wiki.dbpedia.org/ OnlineAccess 3

Faezeh Ensan, Weichang Du / Information Systems 38 (2013) 745–770

757

Fig. 2. LUBM modular ontology design 1: an IBF modular ontology for representing the represented knowledge of Univ-Bench ontology.

and DBPedia. Figs. 2 and 3 show two modular ontology designs for representing the TBox of Univ-Bench ontology. In the rest of this paper, we refer to the Univ-Bench ontology as LUBM monolithic design, the modular ontology in Fig. 2 as LUBM Modular Design 1, and the modular ontology in Fig. 3 as LUBM Modular Design 2. In modular design 1 there are two ontology modules: OM-Publication and OM-Person-Organization. OMPublication has all concepts, roles and TBox axioms related to the publication notion. OM-Person-Organization includes all concepts, roles and TBox axioms of Univ-Bench except those that have already been represented in OM-Publication. These two ontology modules are connected through two interfaces: Inf-Person-Org and Inf-Pub. Inf-Person-Org is realized by OM-PersonOrganization and has two concepts: Person and Research. Person and Research are used by OM-Publication ontology modules for defining the range of the publicationAuthor and publicationResearch properties, respectively. The prefix of Inf-Person-Org indicates that the concepts Research and Person are utilized from an external ontology module through the Inf-Person-Org interface. Inf-Pub has the concept Publication, which is used by OM-PersonOrganization ontology module for defining the range of orgPublication property. In Modular Design 2, there are three ontology modules: OM-Organization, OM-Person, and OM-Publication. OM-Publication in this design has the same knowledge base as it has in design 1, except that it utilizes concepts Person and Research from OM-Person through interface Inf-Person, and these concepts are preceded by the prefix Inf-Person. OM-Organization has all concepts and properties related to organizations like University, Program, College and so on, and OM-Person includes all remaining concepts and properties of Univ-Bench, which are mostly

related to the notion of person. OM-Person and OMOrganization are related through two interfaces: Inf-Org and INF-Person. OM-Person realizes Inf-Person and utilizes Inf-Org, while OM-Organization realizes Inf-Org and utilizes Inf-Person. Additionally, OM-Organization utilizes the concept Publication from OM-Publication though InfPub. In order to create modular designs for TBoxes of VICODI and DBPedia ontologies, we used the Neon toolkit and its Ontology Partitioning and Module Extraction plugins. Ontology partitioning plugin supports decomposing an ontology into smaller modules. Using this plugin, a user can select an ontology, specify some parameters and execute a partitioning algorithm. The result of the algorithm is a set of OWL ontologies, while their dependencies are modeled by means of ‘‘owl:imports’’. Algorithm parameters are minimum size of modules (Min-size), and level of transitive co-inclusions (level). The Module Extraction plugin supports the extraction of smaller modules from an ontology or from a module that is created by the ontology partitioning algorithm. Fig. 4 shows two modular ontology designs for VICODI, which are created by Neon. In the first design, which is referred to as VICODI Modular Design 1, there are 10 modules. This design is created by executing the Neon partitioning algorithm with these parameters: Minsize ¼5 and level ¼3. In this design, the largest module has 40 symbols and 424 axioms on which seven other modules are directly dependant, i.e., they import this ontology. The smallest module has five symbols and five axioms. The second design, which is referred to as VICODI modular design 2, is created by executing the Neon partitioning algorithm with these parameters: Minsize ¼3 and level¼10. This design has 22 modules. The largest ontology module has 25 symbols and 42 axioms

758

Faezeh Ensan, Weichang Du / Information Systems 38 (2013) 745–770

Fig. 3. LUBM modular ontology design 2: an IBF modular ontology for representing the represented knowledge of Univ-Bench ontology.

Fig. 4. VICODI modular ontology designs: two modular designs for VICODI ontology that are created by Neon toolkit.

and 11 other modules are dependent on it, while it is not dependent on any module. The smallest module has three symbols and three axioms and is dependent on another module, while no module is dependent on it. Fig. 5 shows two ontology designs for the DBPedia TBox. The first one, which is referred as DBPedia Modular Design 1, is created by the Neon partitioning algorithm that is initialized by the parameters Min-size¼10 and level ¼4, and also some slight modification on ontology module #10 for making it more focused on the notions of Film and Movie Artists. This design has 10 ontology module whose largest module has 1358 symbols and 2181 axioms. This module depends on no other module, while nine other modules depend on it. The smallest module has 10 symbols and 12 axioms. For designing the second modular design, DBPedia Modular Design 2,

we used the module extraction plugin in Neon and extracted symbols related to these notions: Organization, Person, Place, Work, Biology, and Event. The corresponding modules are Org_Module, Person_Module, Place_Module, Work_Module, Bio_Module, and Event_Module. The links in Fig. 5 show how these ontologies import the others.4 5.3.2. Metric values Table 1 shows the ontology modules to whom test queries are posed in LUBM, VICODI, and DBPedia datasets. Obviously, in monolithic designs, all queries are posed to 4 All modular designs and ontology modules can be found in http:// www.filedropper.com/isdata

Faezeh Ensan, Weichang Du / Information Systems 38 (2013) 745–770

759

Fig. 5. Two DBPedia modular ontology designs, which are created by Neon toolkit.

Table 1 Ontology modules over which queries are posed. Dataset

Query

Monolithic design Modular design 1

LUBM

Queries 1, 4, 5, 6, 7, 9, 10, 14, 15, 16 Univ-Bench Queries 2, 8, 12, 13 Univ-Bench Query 3 Univ-Bench Query 11 Univ-Bench

OM-Person-Organization OM-Person-Organization OM-Publication OM-Person-Organization

OM-Person OM-Person; OM-Organization OM-Publication OM-Organization

VICODI

Query Query Query Query Query

1 2 3 4 5

VICODI VICODI VICODI VICODI VICODI

Module Module Module Module Module

0 1 0; Module 7 2; Module 3; Module 8 0; Module 1

Module Module Module Module Module

DBPedia Query Query Query Query

1 2 3 4

DBPedia DBPedia DBPedia DBPedia

Module Module Module Module

0; Module 4; Module 10 Person-Module 0 Person-Module; Org-Module; Place-Module 1 Work-Module 0; Module 7 Org-Module

the only one ontology that exists. In modular designs, based on the constituting symbols in a query, the queries are posed to one or more ontology modules. Table 2 in Appendix C shows the value of the cohesion metric (COH) for monolithic ontologies and also for those ontology modules in modular designs over which the queries are posed. For calculating cohesion and coupling, we set d ¼ 1 and g ¼ 2. In this table, W-Avg stands for a weighted average of the values of COH and is calculated P ¼n as W-AvgðCOHðM1Þ, . . . ,COHðMnÞÞ ¼ ii ¼ 1 COHðMiÞ Pi ¼ n 9LocðMi Þ9= i ¼ 1 9LocðMi Þ9. This table also shows the value of the coupling metric for the ontology designs. 5.3.3. Datasets and test environment LUBM has a data generator for creating scalable extensional data for its ontology. LUBM(n, s) denotes a dataset that has n universities and is generated using a seed value of s. The LUBM dataset is represented through a set of small files. For the sake of our evaluation, we created a datasets, LUBM(5,0), with 5 universities and 129 533 individuals. We developed a Java program that assigns the exact same individuals that the LUBM benchmark creates for the monolithic design to the modular designs.

Modular design 2

0; Module 7; Module 9 1 0; Module 19 2; Module 12; Module 18; Module 20 1; Module 6

Using this application, we generated a set of files corresponding to ABox assertions of the monolithic design and two set of files for ABox assertions corresponding to OMPerson-Organization and OM-Publication concepts and roles, and also three set of files for ABox assertions of OM-Person, OM-Organization, and OM-Publication concepts and roles. Each individual in each of the files of the modular designs has a corresponding instance, which has been asserted in a file in the set of files for the monolithic design. Hence, the monolithic and modular designs have the same extensional data and they can be compared to each other. We used the ontology provided by the VICODI project as the monolithic dataset. For DBPedia, we used the datasets provided in the project website5 and created a single monolithic dataset. Since the sizes of the monolithic ontology files were large, the Neon toolkit was not able to modularize VICODI and DBPedia ontologies. Hence, we used Neon for modularizing only the TBox of these ontologies, and we developed a java application that

5

http://wiki.dbpedia.org/Ontology

760

Faezeh Ensan, Weichang Du / Information Systems 38 (2013) 745–770

assigned to each module appropriate instances. For an instance a and ontology module M, the application inserts the axiom A(a) to M if the class A is a symbol in the TBox of M and A(a) is in the monolithic ontology. In addition, for a property R, Rða,bÞ is inserted to M if R is a symbol in the TBox of M. Following the Neon approach, we used the same namespace for all classes, roles, and individuals, even if they are in different modules. We employed Pellet 1.0 [44] as the OWL-DL reasoner for running queries over ontology modules. The machine that we used is a PC with 2.13 GHz Intel Core 2 Duo, 2 GB Memory and Windows XP 2002 Service Pack 3. We set the maximum of heap size to 1 GB for running queries. In our evaluation, we distinguish between the time required for loading knowledge base (the time needed for loading ontologies), the time required for ABox consistency checking, and the time required for running a query. Pellet checks ABox consistency in the first run of a query. We consider it as the time for both consistency checking and query execution. For the time required for running a query (excluding ABox consistency checking), we run each query 10 times consecutively and compute the average time disregarding the first run. In our evaluation, we deployed all modules on one server. In this deployment, the time needed for joining the result sets of queries, which are posed over more than one module, is ignorable compared with the query execution time. 5.4. Results In this section, we represent test results for the LUBM, VICODI, and DBPedia ontologies and the modular designs. 5.4.1. LUBM All instance retrieval queries over the monolithic design are posed to Univ-Bench ontology, the only ontology module which exists in this design. In modular design 1, conjunctive queries are posed to either OMPublication or OM-Person-Organization; whereas in modular design 2, queries are posed over OM-Publication, OM-Person, OM-Organization, or both OM-Person and

OM-Organization. For instance, consider two Queries 3 and 11. In modular design 1, Query 3; whose constituent symbols are Publication and publicationAuthor, which are both local symbols of the OM-Publication ontology module; is posed over OM-Publication. Similarly, in modular design 2, this query is posed to OM-Publication. The constituent symbols of Query 11 are ResearchGroup, subOrganization, and University. In modular design 1, all these symbols belong to OM-Person-Organization, and in modular design 2 all these symbols belong to OMOrganization. Hence, Query 11 is posed to OM-PersonOrganization and OM-Organization in modular designs 1 and 2, respectively. Fig. 6 shows the time required for running Query 3 and Query 11 over the monolithic and two modular designs for the LUBM(5,0) dataset. OM-Publication is dedicated to representing the notion of academic publications and has the same knowledge base in both modular designs. It is more cohesive compared to the Univ-Bench, which represent academic activities and publications. The value of COH metric for OM-Publication and Univ-Bench also confirm this observation. COH(Univ-Bench) is 0.056 that is less than COH (OM-Publication), which is 0.098. As Fig. 6 shows the time required for running Query 3 over monolithic design is more than the time required for running Query 3 over modular designs. According to Fig. 6, the time required for running Query 11 over the monolithic and modular designs is less than 10 ms, that is, too low to show a meaningful correlation with the values of cohesion metric. Fig. 7 shows the time required for running and the time required for consistency checking and running Queries 1, 4, 5, 6, 7, 9, 10, 14, 15 and 16. In modular design 1, all these queries are posed to OM-Person-Organization. In modular design 2, all of them are posed to OM-Person. As the figure shows, query execution time and also consistency checking and query execution time over the monolithic design with COH of 0.056 is considerably more than over two modular designs 1 and 2 with COH of 0.085 and 0.11, respectively. In addition, the running time of queries in modular design 1 is more compared to modular design 2 whose COH metric has a larger value.

Fig. 6. The time required for running query 3 over the LUBM(5,0) dataset in the monolithic and modular designs.

Faezeh Ensan, Weichang Du / Information Systems 38 (2013) 745–770

761

Fig. 7. ABox consistency checking and query execution time over LUBM monolithic and modular designs.

Fig. 8 shows the time required for running and the time required for consistency checking and running Queries 2, 8, 12 and 13. In modular design 1, all these queries are posed to OM-Person-Organization, while in modular design 2, they are more complex. They are first broken into two sub-queries and are posed to both OM-Person, and OMOrganization, and then the result sets of sub-queries are integrated. As the figure shows, query execution time and also consistency checking and query execution times over the monolithic design with COH of 0.056 are more than three time in two modular designs 1 and 2 with COH of 0.085 and 0.1043, respectively. In addition, the modular design 2, which has a larger value for its COH metric, has a better performances in both query execution and ABox consistency checking.

5.4.2. VICODI Fig. 9 shows the time that the reasoning engine needed for running five VICODI queries, for loading the knowledge base and running queries, and for ABox consistency checking and running queries over the VICODI dataset in the monolithic and modular designs. This figure shows that these times are considerably more in the monolithic design whose value for the COH metric is noticeably less than the value of COH metrics in modular designs. In addition, this figure shows that modular design 2, whose COH metric has a larger value in all queries compared with modular design 1, shows a roughly better performance in all aspects of query execution, ABox consistency checking and knowledge base loading.

762

Faezeh Ensan, Weichang Du / Information Systems 38 (2013) 745–770

Fig. 8. ABox consistency checking and query execution time over LUBM monolithic and modular designs.

5.4.3. DBPedia Fig. 10 shows the time that the reasoning engine needed for running queries 1, 2, and 4, for ABox consistency checking and running queries over DBPedia dataset in the monolithic and modular designs. DBPedia is a very large dataset including about 1,478,000 instances. Checking consistency and running queries over this huge ontology was impossible in our computer. Hence, the execution time of all queries is unmeasurable in the monolithic design. Modular design 1 has a large ontology module (Module 0) to which the other small ones are connected (see Fig. 5). The size of this module is not much smaller than the DBPedia ontology in the monolithic design. Hence, all queries that are posed to Module 0 in this design are unmeasurable in our environment. On the other hand, these queries were successfully run in the modular design 2, which has ontology modules with higher cohesion.

Fig. 11 shows the running time of Query 3 over two DBPedia modular designs. Query 3 is not posed to Module 0 (the largest ontology module in the DBPedia modular design 1), but instead it is posed to the small cohesive Module 1 with COH¼ 0.0375. Query 3 in modular design 2 is posed to Person_Module and Work_Module. The weighted average of COH metrics of these modules is 0.011246, which is lower than COH of Module 1 in modular design 1. As the figure shows, query execution has a significantly better performance in modular design 1 compared to design 2 for this query. Hypotheses 1 and 2 are supported by the results presented in this section. Accordingly, the time required for running queries over LUBM, VICODI and DBPedia and also the time needed for ABox consistency checking are less in the ontology modules that have larger values for their COH metric.

Faezeh Ensan, Weichang Du / Information Systems 38 (2013) 745–770

763

Fig. 9. The time required for query execution, for ABox consistency checking and query execution, and for knowledge base loading and query execution over VICODI dataset in the monolithic and modular designs.

764

Faezeh Ensan, Weichang Du / Information Systems 38 (2013) 745–770

Fig. 10. The time required for query execution and for ABox consistency checking and query execution of queries 1, 2, and 4 over DBPedia dataset in the monolithic and modular designs.

5.5. Discussion As we mentioned earlier in this section, coupling can intuitively be correlated with the number of modules that are involved with a given query on a specific subject.

In order to investigate this hypothesis, we need to have a wide range of queries provided for each dataset in order to be able to find a reliable value for the probable number of involved modules for each query. Among the datasets that we explored, LUBM has a wider range of queries.

Faezeh Ensan, Weichang Du / Information Systems 38 (2013) 745–770

765

Fig. 11. The time required for query execution and for ABox consistency checking and query execution of query 3 over DBPedia dataset in modular designs.

Fig. 12. The relationship between COP of a design and the number of modules to be involved for queries.

Fig. 12 shows the relationship between the number of involved modules for queries and the value of the COP metric for two modular designs of the LUBM dataset. Fig. 12 shows that queries over the LUBM modular design 2 (with a larger value for its COPDES metric) that are more probable to be posed to more than one module. Obviously, in a distributed deployment, where modules are deployed on different servers, this design faces more complexities in posing queries and integrating their results. Unfortunately, we were not able to find a query log, a set of queries that are posed to the ontologies in realworld applications, for any of the datasets, and hence we leave a more general investigation of the correlation between coupling metrics and the number of involved modules for queries for future work. For calculating cohesion and coupling metrics for different ontology modules in this section, we just consider their TBoxes, i.e., we considered the semantic dependencies between roles and concepts but not individuals. Our intuition was that the TBox of an ontology module can be a good representative of the semantic dependencies that exist between the symbols of the whole

knowledge base, including its ABox. Obviously, for the ontologies whose ABoxes have asymmetrically broadened, we need to consider dependencies between all symbols.

6. Discussion and future work There is an extensive body of work in the literature for formalizing and finding the relevant axioms of an ontology to a set of terms in order to provide solutions for the extraction and integration of ontology modules [22–24]. These solutions mostly focus on finding the relevancy of axioms to symbols. Nonetheless, we can find some commonalities in their approach with the approach that is employed in this paper for finding semantic dependencies between symbols. Given a set of symbols S, Grau et al. [22] defines local( S) as all axioms that are local w.r.t. S. In this definition, an axiom a is local w.r.t. S if it is possible to take any interpretation for the symbols in S and extend it to a model of a while the interpretation interprets the additional symbols as the empty set. Based on this approach, a local axiom w.r.t. S must be a tautology when all its additional symbols are replaced by ?.

766

Faezeh Ensan, Weichang Du / Information Systems 38 (2013) 745–770

Both the approaches utilize the idea of replacing some symbols in the ontology with ? (or alternatively >) for finding dependencies. The approach that is employed in this paper analyzes and formalizes the impact of the replacement of every symbol on the other ontological symbols for finding symbolic interdependencies, while the other approach focuses on extracting a subset of axioms of an ontology that is related to a set of symbols. For future work, we would like to analyze the employment of the presented metrics in this paper in the process of module extraction for extracting more cohesive and less coupled modules from an ontology, related to a set of symbols. Furthermore in this paper, we showed the relationship between the cohesion and coupling metrics and the reasoning performance. Intuitively, we can observe that these metrics can be employed for evaluating ontologies from other design-based criteria such as understandability, reusability and maintainability. Highly cohesive ontology modules can be easily understood by a third-party observer. This is because the intent and the information presented by the module are limited to specific information. Similarly, low-coupling facilitates understandability and reusability of ontology modules. For understanding and reusing an ontology module, least amount of external knowledge should be explored and understood. High-cohesive modules enhance more effective reusability and maintainability by allowing their users to utilize a related group of concepts and roles and avoid the import of non-related subjects and sub-domains of discourse. Low-coupling facilitates change propagations and inconsistency resolutions. When a low-coupled ontology module is modified, the least possible number of the other ontology modules and the least portion of their knowledge bases are affected. Observably in low-coupled ontology modules, the less effort is needed for inconsistency resolution and applying new revisions. We leave the precise analysis of the relationship between the cohesion and coupling metrics and these designs based criteria for future work. For this purpose, we intend to analyze the cohesion and coupling metrics in the context of different empirical studies with different domains and different groups of domain experts and ontology designers. Even though we defined higher-degree moderate dependencies, we based the definitions of the coupling and cohesion metrics on the strong and first-degree moderate dependencies. The reason is that finding all higher-degree moderate dependencies is a very computationally expensive

task and hardly achievable for most ontologies and systems. Theoretically, the definitions of the cohesion and coupling metrics match the intuitive notion of dependencies between ontological terms when they capture all dependencies. However, we kept their definitions as computationally achievable as possible for now and left the exploration of the efficient algorithms and methods for capturing other types of semantics dependencies for future work.

7. Conclusion In this paper, we proposed a set of semantic metrics for evaluating cohesion and coupling of ontologies in monolithic and modular designs. Through these metrics, we are able to compare different ontology modules and also different ontology designs that may possibly exist for representing a body of knowledge. We empirically investigated several case studies of conjunctive query answering time for both monolithic and modular ontology designs. The investigations showed that the time required for answering queries and ABox consistency checking over ontology modules with higher value for their cohesion metric is less compared with ontology modules with lower values for their cohesion metric. The main aspects of our work for evaluating modular ontologies are as follows:

 The introduced metrics for assessing modular ontolo-





gies are based on semantic definitions of dependencies between local symbols and between local and external symbols of ontology modules. This semantic approach has the advantage that it considers all asserted and implied axioms for ontology evaluation. Our work for assessing ontologies focuses on both internal and external attributes of ontologies (internal and external attributes are introduced in Fenton [19]). We introduced metrics for measuring cohesion and coupling that are internal attributes of ontologies. We also investigated reasoning performance: the external attribute of ontologies, and we showed the association between the metrics and this external attribute. The evaluation framework, which is introduced in this paper, supports evaluating ontology modules and ontology designs. Based on this framework, different ontology designs that may exist for representing the same body of knowledge can be evaluated and compared with regards to their cohesion and coupling.

Appendix A The following are the conjunctive queries that we used for evaluating reasoning performance over LUBM knowledge bases. Query1

SELECT ?X ?C WHEREf ?X rdf : type ub : GraduateStudent: ?X ub : takesCourse ?C g

Faezeh Ensan, Weichang Du / Information Systems 38 (2013) 745–770

Query2

SELECT ?X ?Y ?Z WHEREf ?X rdf : type ub : GraduateStudent: ?Y rdf : type ub : University: ?Z rdf : type ub : Department: ?X ub : memberOf ?Z: ?Z ub : subOrganizationOf ?Y: ?X ub : undergraduateDegreeFrom ?Y g

Query3

SELECT ?X ?Y WHERE f ?X rdf : type ub : Publication: ?Y rdf : type ub : Professor: ?X ub : publicationAuthor ?Y g

Query4

SELECT ?X ?Y1 ?Y2 ?Y3 WHERE f ?X rdf : type ub : Professor: ?X ub : name ?Y1: ?X ub : emailAddress ?Y2: ?X ub : telephone ?Y3 g

Query5

SELECT ?X ?Y WHEREf ?X rdf : type ub : Person: ?X ub : memberOf ?Y g

Query6

SELECT ?X WHERE f ?X rdf : type ub : Student g

Query7

SELECT ?X ?Y ?Z WHERE f ?X rdf : type ub : Student ?Y rdf : type ub : Course: ?X ub : takesCourse ?Y: ?Z ub : teacherOf ?Y g

Query8

SELECT ?X ?Y ?Z ?M WHERE f ?X rdf : type ub : Student: ?Y rdf : type ub : Department: ?Y ub : subOrganizationOf ?M: ?X ub : emailAddress ?Z g

Query9

SELECT ?X ?Y ?Z WHERE f ?X rdf : type ub : Student: ?Y rdf : type ub : Faculty: ?Z rdf : type ub : Course: ?X ub : advisor ?Y: ?Y ub : teacherOf ?Z: ?X ub : takesCourse ?Z g

Query10

SELECT ?XWHERE f ?X rdf : type ub : Student: ?Y rdf : type ub : GraduateCourse: ?X ub : takesCourse ?Y g

Query11

SELECT ?X WHERE f ?X rdf : type ub : ResearchGroup: ?X ub : subOrganizationOf ?Y: ?Y rdf : type ub : University g

767

768

Faezeh Ensan, Weichang Du / Information Systems 38 (2013) 745–770

Query12

SELECT ?X ?Y ?M WHERE f ?X rdf : type ub : Chair: ?Y rdf : type ub : Department: ?X ub : worksFor ?Y: ?Y ub : subOrganizationOf ?M g SELECT ?X ?Y WHERE f

Query13

?X rdf : type ub : Person: ?Y ub : hasAlumnus ?X g SELECT ?X WHERE f

Query14

?X rdf : type ub : UndergraduateStudent g SELECT ?X ?C WHERE f

Query15

?X rdf : type ub : Student: ?X rdf : type ?C: ?C rdfs : subClassOf ub : Employeeg SELECT ?X ?C WHEREf

Query16

?X rdf : type ub : Student: ?X rdf : type ?C: ?C directSubClassOf ub : Employeeg

Appendix B The following are the queries that we used for evaluating reasoning performance over DBPedia knowledge bases. Query1

SELECT ?name ?birth ?death ?person WHERE f ?person dbo : birthPlace o http : ?person dbo : birthDate ?birth:

//

dbpedia:org=resource=Berlin 4 :

?person dbo : deathDate ?death FILTER ð?birth o ‘‘1900-01-01’’44 xsd : dateÞ: g ORDER BY ?name Query2

SELECT ?player, ?place, ?cap, pop WHERE f ?s foaf : page ?player: ?s rdf : type o http : ?s o http :

//

?s o http : ?club o http :

//

?s o http :

//

//

dbpedia:org=ontology=SoccerPlayer 4 :

dbpedia:org=property=position 4 ?position: //

dbpedia:org=property=clubs 4 ?club: dbpedia:org=ontology=capacity 4 ?cap: dbpedia:org=ontology=birthPlace 4 ?place:

?place ?population ?pop: OPTIONAL f?s o http :

//

dbpedia:org=ontology=number 4 ?tricot:g

Filter ðxsd : intð?popÞ4 10000000Þ: Filter ðxsd : intð?capÞo 40000Þ: Filter ð?position ¼ ’’Goalkeeper’’@en 99 ?position ¼ o http :

//

dbpedia:org=resource=Goalkeeper_%28associationf ootball%29 4 99

?position ¼ o http :

//

dbpedia:org=resource=Goalkeeper_%28football%29 4 Þ

g Limit 1000

Query3

SELECT ?subject ?label ?released ?abstract WHERE f ?subject rdf : type o http : ?subject o http :

//

//

dbpedia:org=ontology=Film 4:

dbpedia:org=property=starring 4 o http :

?subject rdfs : comment?abstract: ?subject rdfs : label ?label: FILTERðlangð?abstractÞ ¼ ‘‘en’’ &&langð?labelÞ ¼ ‘‘en’’Þ: ?subject o http :

//

dbpedia:org=ontology=releaseDate 4 ?released:

FILTERðxsd : dateð?releasedÞ o ‘‘2000-01-01’’44 xsd : dateÞ: g ORDER BY ?released LIMIT 20

//

dbpedia:org=resource=TomC ruise 4 :

Faezeh Ensan, Weichang Du / Information Systems 38 (2013) 745–770

769

Table 2 Cohesion and coupling of ontology modules. Dataset

Ontology design

LUBM

Monolithic Univ-Bench design Modular design 1 OM-Publication OM-Person-Organization Modular design 2 OM-Publication OM-Person OM-Organization OM-Organization; OM-Person

VICODI

Module

COH

COP

9Loc9

COHW-Avg COPDes

0.0566

0

68

–

0

0.0984 0.0851 0.0984 0.1102 0.0879 –

0.0666 0.0125 0.0666 0.2051 0.1632 –

15 53 15 39 14 14; 39

– – – – – 0.1043

0.0245

0

194

–

0

0.3333 0.6666 – – – – 0.6666 – – –

30 40 30; 12; 30; 22; 21 22; 19; 21;

– – 0.08477 0.0960 0.08053 0.1937 – 0.0988 0.2052 0.1366

0

861

–

0

– 0 0.666 – 0.063 – – 0.0243

674; 3; 28 674 13 674; 6 204 204; 192; 49 204; 65 192

0.00488 – – 0.0033 – 0.00914 0.01234 –

–

Monolithic VICODI 0.03044 design Modular design 1 Module 0 0.1003 Module 1 0.0606 Module 0; Module 7 0.1003; 0.0634 Module 2; Module 3; Module 8 0.202; 0.2666; 0.01111 Module 0; Module 1 0.202; 0.266; 0.0111 Modular design 2 Module 0; Module 7; Module 9 0.1341; 0.4444; 0.333 Module 1 0.0920 Module 0; Module 19 0.1341; 0.0634 Module 2; Module 12; Module 18; Module 20 0.1676; 0.266; 0.266; 0.222 Module 1; Module 6 0.0920; 0.444

DBPedia Monolithic DBPedia design Modular design 1 Module 0; Module 4; Module 10 Module 0 Module 1 Module 0; Module 7 Modular design 2 Person-Module Person-Module; Org-Module; Place-Module Person-Module; Work-Module Org-Module

Query4

0.0025 0.0025; 0.0025 0.1538 0.0025; 0.0061 0.0061; 0.0061; 0.0089

0.2222; 0.037

0.0888 0.0089; 0.022 0.0318

22 5; 25 40 3; 4 22 5; 5; 6 3

0.16596

0.5257

0.58169

0.0436

0.04788

SELECT n WHERE f ?company a o http :

//

?company o http : ?product o http :

// //

:?producta o http :

//

dbpedia:org=ontology=Organisation 4: dbpedia:org=ontology=foundationPlace 4 o http : dbpedia:org=ontology=developer 4 ?company

//

dbpedia:org=resource=California4 :

dbpedia:org=ontology=Software 4

g

Appendix C Cohesion and coupling of ontology modules are shown in Table 2.

References [1] M. Ashburner, C. Ball, J. Blake, D. Botstein, H. Butler, J. Cherry, A. Davis, K. Dolinski, S. Dwight, J. Eppig, et al., Gene ontology: tool for the unification of biology, Nature Genetics 25 (1) (2000) 25–29. [2] F. Baader, Appendix: description logic terminology, The Description Logic Handbook: Theory, Implementation, and Applications (2003) 485–495. [3] F. Baader, D. Calvanese, D.L. McGuinness, D. Nardi, P.F. PatelSchneider (Eds.), The Description Logic Handbook: Theory, Implementation, and Applications, Cambridge University Press, 2003. [4] F. Baader, U. Sattler, An overview of tableau algorithms for description logics, Studia Logica 69 (1) (2001) 5–40. [5] J. Bao, D. Caragea, V. Honavar, Modular ontologies—a formal investigation of semantics and expressivity, in: R. Mizoguchi, Z. Shi, F. Giunchiglia (Eds.), ASWC, vol. 4185, 2006, pp. 616–631.

[6] J. Bao, G. Slutzki, V. Honavar, A semantic importing approach to knowledge reuse from multiple ontologies, in: Proceedings of the National Conference on Artificial Intelligence, vol. 22, 1999, AAAI Press, MIT Press, Menlo Park, CA, Cambridge, MA, London, 2007, p. 1304. [7] C. Bizer, J. Lehmann, G. Kobilarov, S. Auer, C. Becker, R. Cyganiak, S. Hellmann, Dbpedia-a crystallization point for the web of data, Web Semantics: Science, Services and Agents on the World Wide Web, 2009. [8] A. Borgida, L. Serafini, Distributed description logics: assimilating information from peer sources, Journal of Data Semantics 1 (2003) 153–184. [9] J. Brank, M. Grobelnik, D. Mladenic, A survey of ontology evaluation techniques, in: Proceedings of the Conference on Data Mining and Data Warehouses (SiKDD 2005), Citeseer, 2005. [10] J. Brank, D. Mladenic, M. Grobelnik, Gold standard based ontology evaluation using instance assignment, in: Proceedings of the EON 2006 Workshop, Citeseer, 2006.

770

Faezeh Ensan, Weichang Du / Information Systems 38 (2013) 745–770

[11] C. Brewster, H. Alani, S. Dasmahapatra, Y. Wilks, Data driven ontology evaluation, in: Proceedings of LREC, vol. 2004, Citeseer, 2004. [12] A. Burton-Jones, V. Storey, V. Sugumaran, P. Ahluwalia, A semiotic metrics suite for assessing the quality of ontologies, Data & Knowledge Engineering 55 (1) (2005) 84–102. [13] S. Chidamber, C. Kemerer, A metrics suite for object oriented design, IEEE Transactions on Software Engineering 20 (6) (1994) 476–493. [14] O. Corcho, M. Fernandez-Lopez, A. Gomez-Perez, Methodologies, tools and languages for building ontologies: where is their meeting point, Data & Knowledge Engineering 46 (1) (2003) 41–64. [15] F. Ensan, W. Du, Towards domain-centric ontology development and maintenance frameworks, in: Proceedings of the Nineteenth International Conference on Software Engineering & Knowledge Engineering (SEKE 2007), Citeseer, 2007. [16] F. Ensan, W. Du, An interface-based ontology modularization framework for knowledge encapsulation, in: Proceedings of the 7th International Conference on the Semantic Web, Springer, 2008, p. 532. [17] F. Ensan, W. Du, Formalizing the role of goals in the development of domain-specific ontological frameworks, in: Proceedings of the 41st Annual Hawaii International Conference on System Sciences, IEEE Computer Society, 2008, p. 120. [18] F. Ensan, W. Du, A knowledge encapsulation approach to ontology modularization, Knowledge and Information Systems 20 (3) (2009) 249–283. [19] N. Fenton, Software measurement: a necessary scientific basis, IEEE Transactions on Software Engineering 20 (3) (1994) 199–206. [20] A. Gangemi, C. Catenacci, M. Ciaramita, J. Lehmann, Modelling ontology evaluation and validation, in: Proceedings of the 3rd European Semantic Web Conference (ESWC2006), vol. 4011, Springer, 2006. [21] B. Glimm, I. Horrocks, C. Lutz, U. Sattler, Conjunctive query answering for the description logic, Journal of Artificial Intelligence Research 31 (2008) 157–204. [22] B. Grau, I. Horrocks, Y. Kazakov, U. Sattler, Just the right amount: extracting modules from ontologies, in: Proceedings of the 16th International Conference on World Wide Web, ACM, 2007, pp. 717–726. [23] B. Grau, I. Horrocks, Y. Kazakov, U. Sattler, Modular reuse of ontologies: theory and practice, Journal of Artificial Intelligence Research 31 (1) (2008) 273–318. [24] B. Grau, Y. Kazakov, I. Horrocks, U. Sattler, A logical framework for modular integration of ontologies, in: Proceedings of the 20th International Joint Conference on Artificial Intelligence (IJCAI 2007), Citeseer, 2007, pp. 298–303. [25] B. Cuenca-Grau, B. Parsia, E.Sirin, A.Kalyanpur, Automatic partitioning of owl ontologies using e-connections, Tech. rep., UMIACS, available at http://www.mindswap.org/2004/multipleOnt/papers/ Partition.pdf (2005). [26] Y. Guo, Z. Pan, J. Heflin, LUBM: a benchmark for OWL knowledge base systems, Web Semantics: Science, Services and Agents on the World Wide Web 3 (2–3) (2005) 158–182. [27] V. Haarslev, R. Moller, Racer: a core inference engine for the semantic web, in: Proceedings of the 2nd International Workshop on Evaluation of Ontology-based Tools, Citeseer, 2003, pp. 27–36. [28] P. Haase, H. Lewen, R. Studer, T. Tran, M. Erdmann, M. d’Aquin, E. Motta, The neon ontology engineering toolkit, in: WWW, 2008. [29] M. Hitz, B. Montazeri, Measuring coupling and cohesion in objectoriented systems, in: Proceedings of the International Symposium on Applied Corporate Computing, vol. 50, 1995, pp. 75–76. [30] I. Horrocks, DAMLþ OIL: a description logic for the semantic web, IEEE Data Engineering Bulletin, 25 (1) (2002) 4–9. [31] I. Horrocks, U. Sattler, S. Tobies, Practical reasoning for expressive description logics, in: Proceedings of the 6th International Conference on Logic Programming and Automated Reasoning, LPAR ’99, Springer-Verlag, London, UK, 1999, pp. 161–180. [32] S. Kramer, H. Kaindl, Coupling and cohesion metrics for knowledgebased systems using frames and rules, ACM Transactions on

[33]

[34]

[35]

[36] [37] [38]

[39]

[40] [41]

[42] [43]

[44]

[45]

[46] [47]

[48]

[49]

[50]

[51]

[52] [53]

[54] [55]

Software Engineering and Methodology 13 (July) (2004) 332–358. URL: /http://doi.acm.org/10.1145/1027092.1027094S. O. Kutz, C. Lutz, F. Wolter, M. Zakharyaschev, E-connections of abstract description systems, Artificial Intelligence 156 (1) (2004) 1–73. A. Lozano-Tello, A. Gomez-Perez, Ontometric: a method to choose the appropriate ontology, Journal of Database Management 15 (2) (2004) 1–18. Y. Ma, B. Jin, Y. Feng, Semantic oriented ontology cohesion metrics for ontology-based systems, The Journal of Systems and Software (2009). A. Maedche, S. Staab, Measuring similarity between ontologies, in: Lecture Notes in Computer Science, 2002, pp. 251–263. D. McGuinness, F. Van Harmelen, OWL web ontology language overview, W3C Recommendation 10 (2004). 2004-03. B. Motik, U. Sattler, A comparison of reasoning techniques for querying large description logic aboxes, in: Logic for Programming, Artificial Intelligence, and Reasoning, Springer, 2006, pp. 227–241. G. Nagypa´l, R. Deswarte, J. Oosthoek, Applying the Semantic Web: the VICODI experience in creating visual contextualization for history, Literary and Linguistic Computing 20 (3) (2005) 327. A. Orme, H. Tao, L. Etzkorn, Coupling metrics for ontology-based system, IEEE Software 23 (2) (2006) 102–108. R. Porzel, R. Malaka, A task-based approach for ontology evaluation, in: ECAI Workshop on Ontology Learning and Population, Valencia, Spain, Citeseer, 2004. E. Prud-Hommeaux, A. Seaborne, et al., SPARQL query language for RDF, W3C Working Draft 4, 2006. R. Raskin, M. Pan, Semantic Web for Earth and Environmental Terminology (SWEET), in: Proceedings of the Workshop on Semantic Web Technologies for Searching and Retrieving Scientific Data, Citeseer, 2003. E. Sirin, B. Parsia, B.C. Grau, A. Kalyanpur, Y. Katz, Pellet: a practical owl-dl reasoner, Web Semantics: Science, Services and Agents on the World Wide Web 5 (2) (2007) 51–53. B. Smith, M. Ashburner, C. Rosse, J. Bard, W. Bug, W. Ceusters, L. Goldberg, K. Eilbeck, A. Ireland, C. Mungall, et al., The OBO foundry: coordinated evolution of ontologies to support biomedical data integration, Nature Biotechnology 25 (11) (2007) 1251–1255. P. Spyns, et al., EvaLexon: Assessing Triples Mined from Texts, Technical Report 09, 2005. H. Stuckenschmidt, M. Klein, Structure-based partitioning of large concept hierarchies, The Semantic Web-ISWC 2004, 2004, pp. 289– 303. H. Stuckenschmidt, M. Klein, Reasoning and change management in modular ontologies, Data & Knowledge Engineering 63 (2) (2007) 200–223. H. Stuckenschmidt, C. Parent, S. Spaccapietra, Modular Ontologies: Concepts, Theories and Techniques for Knowledge Modularization, Springer-Verlag, New York, Inc., 2009. S. Tartir, I. Arpinar, M. Moore, A. Sheth, B. Aleman-Meza, OntoQA: metric-based ontology quality analysis, in: Proceedings of the Workshop on Knowledge Acquisition from Distributed, Autonomous, Semantically Heterogeneous Data and Knowledge Sources (KADASH), Citeseer, 2006. G. Tzoganis, D. Koutsomitropoulos, T. Papatheodorou, Querying Ontologies: Retrieving Knowledge from Semantic Web Documents, Tech-Report, Available at: /http://www.hpclab.ceid.upatras.gr/vio grafika/kotsomit/pubs/eureka09.pdfS, 2009. D. Vrandecic, Y. Sure, How to design better ontology metrics, Lecture Notes in Computer Science, vol. 4519, , 2007, p. 311. Y. Wang, J. Bao, P. Haase, G. Qi, Evaluating formalisms for modular ontologies in distributed information systems, Lecture Notes in Computer Science, vol. 4524, , 2007, pp. 178–193. H. Yao, A. Orme, L. Etzkorn, Cohesion metrics for ontology design and application, Journal of Computer Science 1 (1) (2005) 107–113. D. Zhang, C. Ye, An evaluation method for ontology complexity analysis in ontology evolution, Lecture Notes in Computer Science vol. 4248 (2006)., p. 214.