- Email: [email protected]
Semantic oriented ontology cohesion metrics for ontology-based systems
The Journal of Systems and Software 83 (2010) 143–152
Contents lists available at ScienceDirect
The Journal of Systems and Software journal homepage: www.elsevier.com/locate/jss
Semantic oriented ontology cohesion metrics for ontology-based systems Yinglong Ma a,b,*, Beihong Jin b, Yulin Feng b a b
Department of Computer Sciences and Technology, North China Electric Power University, Beijing 102206, PR China Technology Center of Software Engineering, Institute of Software, Chinese Academy of Sciences, Beijing 100190, PR China
a r t i c l e
i n f o
Article history: Received 27 August 2008 Received in revised form 23 July 2009 Accepted 24 July 2009 Available online 6 August 2009 Keywords: Ontology Semantic metrics Ontology cohesion metrics Semantic inconsistency
a b s t r a c t Ontologies play a core role to provide shared knowledge models to semantic-driven applications targeted by Semantic Web. Ontology metrics become an important area because they can help ontology engineers to assess ontology and better control project management and development of ontology based systems, and therefore reduce the risk of project failures. In this paper, we propose a set of ontology cohesion metrics which focuses on measuring (possibly inconsistent) ontologies in the context of dynamic and changing Web. They are: Number of Ontology Partitions (NOP), Number of Minimally Inconsistent Subsets (NMIS) and Average Value of Axiom Inconsistencies (AVAI). These ontology metrics are used to measure ontological semantics rather than ontological structure. They are theoretically validated for ensuring their theoretical soundness, and further empirically validated by a standard test set of debugging ontologies. The related algorithms to compute these ontology metrics also are discussed. These metrics proposed in this paper can be used as a very useful complementarity of existing ontology cohesion metrics. Ó 2009 Elsevier Inc. All rights reserved.
1. Introduction With the development of the Semantic Web (Berners-Lee et al., 2001), ontologies play a core role to provide shared knowledge models to semantic-driven applications on Web. Ontologies involve agreed terms and their relationships in different domains. Different users can agree on the use of a common ontology in order to annotate their content or resolve their differences through interactions and negotiations. With the promising benefits provided by ontologies, domain ontology construction and reuse have become more and more important in all kinds of semantic-driven applications. There also are many domain ontologies such as the gene ontology (GO) and other ontologies from the Open Biology Ontologies (OBO, 2008). Although ontology construction is one of core issues for development of semantic-driven applications, it is rather important to assess the quality of an ontology. Assessing ontology can help ontology engineers to better control project management and development of ontology based systems, and therefore reduce the risk of project failures. From the viewpoint of ontology developers, by assessing quality of ontology, ontology developers can automatically recognize areas that might need more work and specify what parts of the ontology might cause problems. In the
* Corresponding author. Address: Department of Computer Sciences and Technology, North China Electric Power University, Beijing 102206, PR China. Tel.: +86 10 51963385.. E-mail addresses: [email protected] (Y. Ma), [email protected] (B. Jin), [email protected] (Y. Feng). 0164-1212/$ - see front matter Ó 2009 Elsevier Inc. All rights reserved. doi:10.1016/j.jss.2009.07.047
case, they can revise the ontology and further improve the quality of the ontology. From the viewpoint of ontology users, assessing qualities of ontologies allows them to compare between these different ontologies and select from these ontologies a higher quality ontology that is going to be used. As in many other related fields, you can only control what you can measure (DeMarco, 1982). Measuring ontology for ontology assessing has become an active area. In the last years, many ontology metrics and measures have been proposed and some principal work has been done to study the nature of measures for ontologies in general (Gangemi et al., 2005; Gangemi et al., 2006; Lozano-Tello and Gomez-Perez, 2004; Tartir et al., 2005; Haining et al., 2005). These metrics and principles provide some useful guides for ontology engineers about what methods are considered (e.g. structure or semantics), and how useful the method is (e.g. it is useful for ontology developers and ontology users). However, most metrics still are based on structural notions without taking into account the semantics which leads to incomparable measurement results (Vrandecic and Sure, 2007). First, most metrics are based on RDF graph structure. Only a very small number of metrics is considering ontology semantics such as subsumption. Second, these proposed metrics are unstable without considering possible additions of further axioms to an ontology because they have not taken the open world assumption (OWA) into account. OWA can satisfy the requirements of ontologies in the context of dynamic and changing Web. Furthermore, just because of dynamic and changing characters of ontologies on Web, a consistent ontology probably becomes inconsistent. Few ontology metrics are considering inconsistency of ontology. Third, most important, some ontology metrics are very
144
Y. Ma et al. / The Journal of Systems and Software 83 (2010) 143–152
often proposed to capture elusive concepts or characteristics such as cohesion (Tartir et al., 2005; Haining et al., 2005; Orme et al., 2007), complexity (Dalu et al., 2006; Orme et al., 2007), coupling (Orme Anthony et al., 2006), connectivity (Tartir et al., 2005), usability (Lozano-Tello and Gomez-Perez, 2004), etc. But most of the work except the work in the literatures (Haining et al., 2005; Orme Anthony et al., 2006; Orme et al., 2007), have not given the precise definitions and their theoretical soundness of these related concepts or characteristics. As mentioned in the literature (Briand et al., 1996), the theoretical soundness of a measure is the obvious prerequisite for its acceptability and use. Looking at the work in the literatures (Haining et al., 2005; Orme Anthony et al., 2006; Orme et al., 2007), although these metrics have theoretical soundness, it seems that these metrics still consider only ontology structure and seldom measure ontology semantics. Even if some of them are considering ontology semantics, they are unstable. In this paper, we propose a set of ontology cohesion metrics to measure the modular relatedness of ontologies in the context of dynamic and changing Web. We concentrate on measuring inconsistent in ontologies and fully consider the ontological semantics rather than ontological structure. These metrics are: Number of Ontology Partitions (NOP), Number of Minimally Inconsistent subsets (NMIS) and Average Value of Axiom Inconsistencies (AVAI). We also give the related algorithms computing these measurements. These metrics are theoretically validated by using standard metrics validation frameworks, and empirically validated by using a prototype implementing the metrics and algorithms presented in this paper. At last, we present a complete process and explain how the ontology cohesion metrics proposed in this paper can be reasonably used as a cogent complementarity of existing ontology cohesion metrics in order to measure ontology cohesion degree in the context of dynamic and changing Web. This paper is organized as follows. In Section 2, we introduce the preliminaries about ontology representation and criteria of analyzing metrics. In Section 3, we propose the set of our ontology cohesion metrics. The related algorithms computing these ontology metrics also are discussed. Section 4 gives the theoretical validation of these ontology cohesion metrics. In Section 5, we discuss the empirical validation of our ontology cohesion metrics. In Section 6, we briefly discuss the process of assessing quality of ontologies by combining our ontology cohesion metrics with other existing semantic cohesion metrics. Section 7 is the related work. We overview most of existing ontology metrics (including our metrics proposed in the paper) from different points of view, and give a comprehensive comparison of ontology cohesion metrics. Section 8 is conclusion.
2. Preliminaries 2.1. Ontology representation An ontology is defined as a formal, explicit specification of a shared conceptualization (Gruber, 1993). Ontologies can be represented in some ontology description languages such as Resource Description Framework (RDF) (Manola and Miller, 2004), Web Ontology Language (OWL) (Dean and Schreiber, 2004) and Description logic (DL) (Baader et al., 2003). OWL is an extension of RDF and a machine-readable language for sharing and reasoning information on the Web. It has been recommended as the standard web ontology language. An OWL ontology representation consists of axioms and facts. Axioms are the semantic knowledge defined by building relationships between classes and properties. Facts represent the individual assertions. There are three forms of OWL languages related to each other: OWL Lite, OWL DL and OWL Full. OWL Full contains OWL DL, and OWL DL contains OWL Lite. OWL
DL and OWL Lite correspond semantically with certain Description Logic languages. For examples, OWL DL is partially based on the DL SHOIN(D), which includes special constructors such as oneOf, transitive properties, inverse properties and datatype properties. OWL Lite is based on the less expressive DL SHIF(D). Ontology reasoning based on OWL Full is undecidable and existing reasoners do not support reasoning with all features of OWL Full. In this paper, the DL languages that we work on are those for which semantic reasoning for consistency checking is decidable. Generally speaking, a concept in DL is referred to as a class in OWL. A role in DL is a property in OWL. The terminological axioms and individuals have the same meaning in DL and OWL. Just because of the close connection between OWL and DLs, in this paper, we will make no distinction between ontologies (in OWL) and knowledge bases (in DL), and the related definition and examples also are given mainly in DL syntax. DL represents the knowledge of a domain by first defining the relevant concepts of the domain. Then these defined concepts are used to specify the properties of the objects in the domain. A knowledge base K can be defined as K ¼ ðT; AÞ, where T and A are TBox and Abox, respectively. TBox and ABox represent the sets of terminological axioms and individual assertions, respectively. In the TBox, basic descriptions are atomic concepts and atomic roles. Atomic concepts are designated by unary predicates. Atomic roles are designated by binary predicates to express relationships between individuals. Concept descriptions can be built on atomic concepts by iteratively applying constructors such as intersection ðuÞ, union ðtÞ, negation ðqÞ, value restriction ð8R CÞ and existential quantification ð9R CÞ, etc. Axioms express how concepts and roles are related to each other. Generally, An axiom is of the form C v D or C D, where C and D are concept descriptions. C v D if concept C is subsumed by concept D. C D if C v D and D v C. An ABox is a set of individual assertions of the form CðaÞ or Rða; bÞ, where R is a role, and a, b are individuals. For an interpretation I ¼ ðDI ; I Þ; I maps every atomic concept A to a subset AI # DI , and every atomic role R to a binary relation RI # DI DI , where DI is the domain, and I is the interpretation function. I satisfies C v D if C I # DI , and satisfies C D if C I ¼ DI . I satisfies CðaÞ if aI 2 C I , and satisfies Rða; bÞ if I ðaI ; b Þ 2 RI . I is a model of K if I is the model of T and A. I is a model of TðAÞ if it satisfies all axioms (assertions) in TðAÞ. A concept C is satisfiable if there is a model of T such that C T – ;. An knowledge base K has two forms of inconsistency: incoherent and inconsistent. The former refers that there is an unsatisfiable concept in TBox. The latter refers that K has no any model. For a knowledge base K ¼ ðT; AÞ, the tableau algorithm can be used to check the satisfiability of a concept in T and consistency of A w.r.t T. Its basic principle of checking the satisfiability of a concept C is to gradually build a model I of C such that C I is not empty. As a result, a tree-like model of concept can be built by decomposing concept C by using tableau expansion rules. The tableau algorithm terminates when either no more rules can be applicable, or when a clash occurs. 2.2. Cohesion metrics and validation framework It is desirable to have a formal model and precise theoretical foundation of metrics evaluation criteria, through which we can assess the usefulness and correctness of measures within well-defined contexts. In traditional software measurement, the concept of cohesion refers to the degree to which the elements in a module belong together. Especially for object-oriented software, cohesion refers to the degree of the relatedness or consistency in functionality of the members in a class. Cohesion measures separation of responsibilities, independence of components and control of com-
145
Y. Ma et al. / The Journal of Systems and Software 83 (2010) 143–152
plexity (Chae et al., 2000). Classes with strong cohesion is desirable for object oriented systems. One of the most widely known objectoriented cohesion metrics was proposed by Chidamber and Kemerer Chidamber and Kemerer (1994): Lack of Cohesion of Methods (LCOM). Because object-oriented conceptual model has a close association with ontology representation, software metric evaluation criteria (including such measurement concepts as cohesion, complexity and coupling, etc.) for object-oriented software can be regarded as a candidate evaluation framework for ontology quality assessing. Kitchenham et al. (1995) proposed a general framework for software measurement validation. They described the structure of any measure as containing the entities being analyzed, the attribute being measured, the unit used, and the data scale. In order to have any meaning for a value, they specified the entity, the attribute, and the units. The measure must be defined over a specified set of permissible values (discrete or continuous). A direct measure within the framework also should have four aspects to be theoretically valid, respectively. Briand et al. (1996) further presented a set of criteria for clarifying software measurement concepts such as complexity and cohesion. Different measurement concepts are involved in different characteristics and criteria. For example, as for cohesion metrics, the four aspects include: (1) nonnegativity and normalization, representing that the value is nonnegative and the values are comparable between different modules. (2) null value, i.e. the value is zero if there is no intramodule relationship within a module. (3) monotonicity, refers that the value never decreases when adding intramodule relationships into a module. And (4) cohesive module, i.e. the value after merging two unrelated modules is never greater than the maximum cohesion of the original modules. Here, intramodule relationships can be regarded as axioms in ontology representation. In this paper, our ontology cohesion metrics will be fitted into the frameworks of Kitchenham et al. (1995);Briand et al. (1996) for ensuring the metrics proposed theoretically correct. 3. Proposal of ontology cohesion metrics We first introduce some basic definitions about ontologies, which are the prerequisite of our ontology cohesion metrics. Then we will specifically define our metrics. 3.1. Basic definitions A knowledge base K can be defined as K ¼ ðT; AÞ, where T and A are TBox and Abox, respectively. In the following, we introduce some basic definitions. Definition 1. For K0 ¼ ðT0 ; A0 Þ and K ¼ ðT; AÞ, the followings hold:
K0 #K iff T0 #T and A0 #A: K0 K iff T0 T and A0 A: K0 \ K iff T0 \ T andA0 \ A: K0 [ K iff T0 [ T and A0 [ A: Definition 2. For a knowledge base K ¼ ðT; AÞ, its cardinality is denoted jKj, and jKj ¼ jT [ Aj. Proposition 1. For any K1 ; K2 # K, if K1 [ K2 ¼ K, then jKj ¼ jK1 j þ jK2 j jK1 \ K2 j. In the case, if K1 \ K2 ¼ ;, then jKj ¼ jK1 j þ jK2 j. In the TBox of knowledge base K, a concept is either an atomic one or a complex concept expression. Atomic concepts are named ones. For example, the term Student defined by < owl : Classrdf : ID ¼ Student= > is an atomic class name. In contrast, complex concepts have not named and are called anony-
mous classes in general. For example, some complex classes containing properties are sometimes defined by some annotations such as owl:Restriction, etc. Definition 3. For a knowledge base K ¼ ðT; AÞ, the set of defined concepts is denoted C ¼ fC 1 ; . . . ; C n g, where each C i ð1 6 i 6 nÞ is either an atomic concept or a complex concept defined in K. In this paper, we assume that all axioms in an TBox T are unfoldable ones because we want to avoid the so-called cycle axiom problem (Baader et al., 2003). Although blocking techniques can be applied to the tableau algorithm to resolve reasoning problem with cycle axioms, it is beyond the scope of this paper. We use the definition of unfoldable axioms from the literature (Schlobach et al., 2007). Definition 4. For any axiom a of the form A v B or A B, a is unfoldable iff A is a named concept that is atomic and unique, and B does not contain direct or indirect reference to A. As mentioned in Section 2.1, a knowledge base is inconsistent if and only if it has no model. Theoretically speaking, the tableau algorithm can be used to check the satisfiability of a concept in T and consistency of A w.r.t T. KAON21 adopts algorithms different from tableau algorithm for consistency checking (Motik, 2006). In practice, the existing reasoners (e.g. RACER Haarslev and Moller, 2006 and Sirin et al. (2007)) with visual ontology editors (e.g. Protege Stanford (2008) and SWOOP Kalyanpur et al., 2006), support inconsistency checking for ontology users and developers. Especially, Pellet supported by SWOOP has a strong capability for inconsistency checking based on ontologies. Different ontologies probably have different inconsistency degrees. Measuring the degree of ontology inconsistency can help ontology users and developers to appropriately modify and select ontologies for future reuse. In order to measure ontology inconsistency, we use the inconsistency value defined in the literature (Deng et al., 2007). Definition 5. Let K be a knowledge base. The function iv : 2K ! f0; 1g, assigns a value to each K0 . For any K0 # K, the inconsistency value of K0 can be defined as:
iv ðK0 Þ ¼
0;
if K0 is consistent or K0 is empty
1; otherwise
:
Example 1. Look at a simple ontology example in DL syntax. In knowledge base K ¼ ðT; AÞ; T contains axioms as follows: 1: A v B, 2: A v qB, 3: C v E u F; 4: E v 8s:G u D; 5: F v 9s:qG; A contains an individual assertion: 6: AðaÞ; 7: EðbÞ. For the sake of simplicity, we refer the axioms and assertions by their numbers. There are only K0 ¼ ðf1; 2g; f6gÞ and K00 ¼ ðf3; 4; 5g; f7gÞ such that iv ðK0 Þ ¼ 1 and iv ðK00 Þ ¼ 1. Because not all axioms are the source of ontology inconsistency in an inconsistent ontology. It is necessary to determine the set of axioms causing ontology inconsistency. We introduce the minimally inconsistent subsets (MIS) of K defined by the literature (Deng et al., 2007). Definition 6. For any subset K0 # K, K0 is the minimally inconsistent subset (MIS) of K if the following conditions hold: 1. iv ðKÞ ¼ 1, and 2. iv ðK00 Þ ¼ 0 for every K00 such that K00 K
1
http://www.kaon2.semanticweb.org/.
146
Y. Ma et al. / The Journal of Systems and Software 83 (2010) 143–152
Then we can obtain the set of all MISs in an inconsistent ontology, which is denoted SMIS. Furthermore, based on the set SMIS, we can define the inconsistency impact value of each axiom causing ontology inconsistency. Definition 7. For K ¼ ðT; AÞ, if its SMIS ¼ fmis1 ; mis2 ; . . . ; misn g. Then inconsistency impact value of each axiom or assertion a in K can be defined by a function mapping impv : ðT [ AÞ ! N, where
impv ðaÞ ¼
m þ 1; if for all i1 ; . . . ; ij ; . . . ; im ; a 2 misij 1;
if for each misk 2 SMIS; a R misk
:
where 1 6 j 6 m; 1 6 m; ij 6 n; 1 6 k 6 n and N is the set of natural number. We also need to consider that an ontology probably contains multiple parts which are semantically unrelated to each other. We called them semantic partitions. Ontology inconsistencies impossibly occur across multiple semantic partitions.
Fig. 1. computePart.
Definition 8. For an axiom a of the form A v B or A B, if B is a concept name, then the signature of the axiom a is denoted SignatureðaÞ ¼ fA; Bg. If B is a complex concept expression and contains concept names C 1 ; . . . ; C n in the expression, then the signature of the axiom a is SignatureðaÞ ¼ fA; C 1 ; . . . ; C n g. For the assertions of the form AðaÞ; SignatureðAðaÞÞ ¼ fAg.
ple and unrelated topics, and the concept organization and aggregation within the ontology are loose relatively. In the case, in order to enhance ontology cohesion, either the ontology should be decomposed into multiple ontologies with a single topic, or new semantic relationships should be exploited and added between these partitions.
Definition 9. If two axioms (assertions) a and b are semantically relevant, we denote SemRelev antða; bÞ. SemRelev antða; bÞ iff or 9A; BðA 2 SignatureðaÞ ^ B 2 SignatureðaÞ \ SignatureðbÞ – ; SignatureðbÞÞ such that there exists a subsumption relation between A and B by using certain reasoning paths of concept subsumption in knowledge base.
3.2.2. Definition of Number of Minimally Inconsistent Subsets (NMIS) Number of Minimally Inconsistent Subsets (NMIS) is the number of all minimally inconsistent subsets in a knowledge base. The definition of minimally inconsistent subset has been introduced in the previous section. NMIS of a knowledge base K can be formulated as follows:
Definition 10. For a knowledge base K ¼ ðT; AÞ; K0 # K is a semantic partition of K, where K0 ¼ ðT0 ; A0 Þ if and only if a and b are semantically irrelevant for any a 2 T0 [ A0 and for b 2 ðT [ AÞ n ðT0 [ A0 Þ.
NMISðKÞ ¼ jSMISj
3.2. Definitions of ontology cohesion metrics In this section, we present the following ontology cohesion metrics: Number of Ontology Partitions (NOP), Number of Minimally Inconsistent Subsets (NMIS) and Average Value of Axiom Inconsistencies (AVAI). We also give the related algorithms to compute the number of differen entities of these metrics. 3.2.1. Definition of Number of Ontology Partitions (NOP) Number of Ontology Partitions (NOP) is the number of semanticalpartitions of a knowledge base. NOP of a knowledge base K can be formulated as follows:
NOPðKÞ ¼ jPartSj
ð1Þ
where PartS is the set of semantic partitions in knowledge base K.
ð2Þ
where SMIS refers to the set of all minimally inconsistent subsets (MISs) in the knowledge base K. Here, we will give an algorithm computeSMIS to obtain the SMIS of the knowledge base K, which is shown in Fig. 2. In the algorithm, the possible subsets of a knowledge base first are traversed according to their size. Once a subset is decided as an MIS, then other subsets containing the subset will be ignored and no longer perform consistency checking. All found MISs are moved into the set SMIS. Number of Minimally Inconsistent Subsets (NMIS) can be used to measure the scope of inconsistency impacts of a knowledge base. Inconsistent axioms and assertions will impede the understanding and sharing of the whole knowledge base (ontology). Within an ontology with a large number of MISs, some ontological modules cannot be effectively congregated together and achieve a close and unambiguous sharing. The more there are MISs in an knowledge base, the more the knowledge base is difficult to share. Meanwhile, more MISs mean that ontology engineers have to take more time and efforts to revise these inconsistencies. Thus in the
Example 2. Look back to Example 1. K includes two partitions as follows: Part1 ¼ f1; 2; 6g and Part2 ¼ f3; 4; 5; 7g, i.e. PartS ¼ fPart1; Part2g. According to our definition of NOP, we know that NOPðKÞ ¼ jPartSj ¼ 2. The algorithm computePart obtaining ontology partitions of the knowledge base K is illustrated in Fig. 1. In the algorithm, all axioms will be tagged and compared with each other with respect to their semantic relevance. Those axioms semantically relevant will have the same partition. The variable NOP is used to record number of partitions, and then is returned. Number of Ontology Partitions (NOP) can be used to measure whether the contents of an ontology have a common topic. If there are multiple partitions in an ontology, then the ontology has multi-
Fig. 2. computeSMIS.
147
Y. Ma et al. / The Journal of Systems and Software 83 (2010) 143–152
phase of ontology design and construction, it is desirable to reduce axiom inconsistencies, which means a high ontology cohesion. Example 3. We revise K in Example 1 by adding the following axiom into K, i.e. 8:H v A. We will find that K1;2 ; 6 are K3;4;5 ; 7 are the minimally inconsistent subsets. But the subset like K1;2;8 ; 6 is not. Proposition 2. The algorithm computeSMIS can prevent double counting of MISs. In order to explain that the algorithm computeSMIS can prevent double counting of MISs, we only need to show that for any MISi ; MISj 2 SMIS and jMISi j 6 jMISj j; MISi – MISj and MISi å MISj hold, where i – j. On the one hand, according to the algorithm, all subsets of K are sorted by cardinality of K0 . Each subset is traversed only once in the whole iteration process. The subsets with same cardinality must be distinct, so MISi – MISj . On the other hand, in the algorithm, the supersets of all detected MISs will be ignored and are not considered for inconsistency checking at all, so MISi å MISj . According to the analysis above, this algorithm computeSMIS can effectively prevent double counting during collecting MISs. Preventing double counting is necessary to ensure the protocol validity of the NMIS metric. It is worth noting that NOP value of an ontology will probably help separate the ontology into multiple parts for future inconsistency checking and obtaining MISs. For example, if an ontology has a large volume of data, then it will cause high reasoning complexity. In the case, we can separate it into multiple subontologies according to their semantic partitions. Then we only need to find MISs from each partitions. This will reduce complexity of consistency checking and semantic reasoning. The reason is that the same MIS cannot exist across multiple partitions. It is impossible for two sets of axioms semantically irrelevant to have semantic inconsistency between them. 3.2.3. Definition of Average Value of Axiom Inconsistencies (AVAI) Average Value of Axiom Inconsistencies (AVAI) is the ratio of the sum of inconsistency impact values of all axioms and assertions to the cardinality of the knowledge base. But if the ontology is empty, the AVAI will encounter the problem that the denominator of the ratio is zero. To avoid this problem but not to cause new problems, the size of the knowledge base plus 1 will be used as the denominator. Specifically, the definition of AVAI metric of a knowledge base K ¼ ðT; AÞ can be formulated as follows:
P
AVAIðKÞ ¼
a2ðT[AÞ
impv ðaÞ
jKj þ 1
ð3Þ
We give a simple algorithm computeAVAI to compute the Average Value of Axiom Inconsistencies, which is shown in Fig. 3. The algorithm calculates inconsistency impact value in accord with the Definition 7. During the course of inner iteration, the inconsistency impact values of each axiom a is registered in the variable i. The sum of inconsistency impact value of all axioms is stored in variable AVAIvalue. AVAIvalue is further is transformed according to Eq. 3. AVAI ontology cohesion metric can be used to measure the inconsistent degree of ontology to which the axioms in the ontology belong together. Although the axioms in an ontology with a higher AVAI are possibly closely related to each other, but the ontology is more difficult to understand and eliminate inconsistencies. This situation reflects the fact that the ontology developers do not understand the domain knowledge very well. For ontology developers and users, they have to take more time and efforts to correctly understand and eliminate these inconsistencies. Obviously, a lower AVAI value in an ontology is desirable.
Fig. 3. computeAVAI.
Note that different axioms of an ontology possibly have different inconsistency impact values. The axioms with higher inconsistency impact values have more contribution to ontology inconsistencies than those with lower inconsistency impact values. Considering that, we should first select and revise the axioms with higher inconsistency impact values to improve ontology quality. In fact, it is not difficult to obtain the axiom(s) with the highest inconsistency impact value by mildly revising the algorithm in Fig. 3. Example 4. Look back to example in Example 3. We find impv ðiÞ ¼ 2 for any 1 6 i 6 7, and impv ð8Þ ¼ 1. Then AVAIðKÞ ¼ P impv ðaÞ a2ðT[AÞ 5 ¼ 14þ1 8þ1 ¼ 3. jKjþ1 4. Theoretical analysis and validation of ontology cohesion metrics In this section, each metric is examined and validated theoretically by the Kitchenham et al.’s framework (Kitchenham et al., 1995) and Briand et al.’s framework (Briand et al., 1996), which are briefly introduced in Section 2.2. 4.1. Analysis of NOP The metric NOP is a direct measure, in this case, to count the semantic partitions in ontologies. For NOP, the entity, attribute, unit and data scale are ontology, number of semantic partitions, semantic partition and interval, respectively. NOP satisfies Kitchenham, et al.’s four properties of for measurement validation as follows: (1) Attribute validity: the entity has the attribute (number of semantic partitions), which can be obtained by logical reasoning and algorithm in Fig. 1. (2) Unit validity: the attribute is measured by counting the number of semantic partitions. (3) Instrumental validity: The algorithm in Fig. 1 can be used to automatically collect measurement units. The instrument is valid as long as the metrics collecting tool correctly counts the number of semantic partitions in K. (4) Protocol validity: The calculation process can prevent double counting and be free from counting errors by counting the number of semantic partitions. We use specific criteria for Briand et al.’s cohesion metrics for validation of the NOP metric: (1) Nonnegativity and normalization: The value of NOP is never negative and the values can be compared between different versions of evolving ontologies. (2) Null value: Typically, if K is empty, the value of NOP is zero.
148
Y. Ma et al. / The Journal of Systems and Software 83 (2010) 143–152
(3) Monotonicity: is unapplicable because NOP value may decrease after adding an axiom into K. (4) Cohesive Modules: It is obvious that the value of NOP of merged modules is never greater than the maximum NOP of the original modules.
(2) Unit validity: the attribute is measured by counting the inconsistency impact values of all axioms (assertions). (3) Instrumental validity: The underlying theoretical model and algorithm is valid. Logic reasoning based inconsistency checking is used for detecting axiom consistencies, and algorithm in Fig. 3 can be used to automatically collect measurement units. The instrument is valid as long as the metric collecting tool correctly counts the inconsistency impact values of all axioms (assertions) in K. (4) Protocol validity: The calculation process is consistent and unambiguous, and will be free from counting errors.
4.2. Analysis of NMIS The metric NMIS is a direct measure, in this case, to count the minimally inconsistent subsets (MISs) in ontologies. For the NMIS measurement, the entity, attribute, unit and data scale are ontology, number of MISs, MIS and interval, respectively. NMIS satisfies Kitchenham et al.’s four properties of for measurement validation as follows:
We further use specific criteria for Briand et al.’s cohesion metrics for validation of the AVAI metric: (1) Nonnegativity and normalization: The value of AVAI is never negative and the values can be compared between different versions of evolving ontologies. (2) Null value: Typically, if K is consistent, the value of AVAI is zero. (3) Monotonicity: The value of AVAI will never decrease when adding an axiom into K. Assume the AVAI value of K is n . The AVAI value of K0 after adding an axiom AVAIðKÞ ¼ jKjþ1
(1) Attribute validity: the entity has the attribute (number of MISs), which can be obtained and calculated by logical reasoning and the algorithm in Fig. 2. (2) Unit validity: the attribute is measured by counting the number of MISs. (3) Instrumental validity: The underlying theoretical model and algorithm are valid. Logic reasoning based consistency checking is used for detecting axiom inconsistencies, and the algorithm in Fig. 2 can be used to automatically collect measurement units. The instrument is valid as long as the metric collecting tool correctly counts the number of MISs in the ontology (knowledge base) K. (4) Protocol validity: Calculations will be performed according to the formal inconsistent checking model and algorithm. The calculation process can prevent double counting and be free from counting errors by counting the number of MISs.
nþj . In fact, AVAIðKÞ < AVAIðK0 Þ into K, AVAIðK0 Þ ¼ jKjþ2
always holds because
The AVAI metric is a direct measure to count inconsistency impact values of all axioms or assertions. All MISs in K are used to obtain AVAI value. For AVAI, the entity, attribute, unit and data scale are ontology, Average value of axiom inconsistence, Inconsistent impact value per axiom and interval, respectively. AVAI satisfies Kitchenham, et al’s four aspects for measurement criteria as follows:
b2ðT2 [A2 Þ
impv ðdÞ
jKjþ1
impv ðaÞ
P jK1 jþjK2 jþ1
þ
¼ P
a2ðT1 [A1 Þ
impv ðaÞþ
b2ðT2 [A2 Þ
impv ðbÞ
jK1 jþjK2 jþ1 b2ðT2 [A2 Þ
P
impv ðbÞ
jK1 jþjK2 jþ1
6
a2ðT1 [A1 Þ
impv ðaÞ
jK1 jþ1
¼ þ
impv ðbÞ
jK2 jþ1
. That is, AVAIðKÞ 6 AVAIðK1 Þ þ AVAIðK2 Þ
always holds. It is not difficult to find that AVAI satisfies the aspect of cohesive modules
5. Empirical validation of our ontology cohesion metrics To evaluate whether these ontology cohesion metrics are useful for ontologies, we implemented the metrics and algorithms in Java using KAON2, and have performed some preliminary experiments. Four ontologies are use as the data sets to calculate the ontology cohesion metrics. They are object ontology,2 Koala ontology,3 university ontology4 and mini-tambis ontology.5 We obtain all the MISs of an ontology using the algorithm proposed in this paper. The related cohesion metrics for these ontologies can be calculated by the corresponding algorithms described in this paper. In order to generalize our ontology cohesion measurement for possible inconsistent ontologies, we need to revise some of these debugging ontologies to form a relevant complete knowledge bases, that is, both TBoxs and ABoxs of these knowledge bases are not empty. Some of these ontologies have no individuals, i.e. their ABoxs are empty. We added some new individuals for some unsatisfiable concepts in these (inconsistent) ontologies. Some 2
(1) Attribute validity: the entity has the attribute (inconsistent impact value), which can be calculated by Definition 7 and Eq. 3.
d2ðT[AÞ
AVAIðKÞ ¼ P a2ðT1 [A1 Þ
4.3. Analysis of AVAI
nþj 6 jKjþ2 always holds, where
n 6 jKj and j ¼ impv ðaÞ P 1. (4) Cohesive modules: Assume that K1 and K2 are unrelated. This means that K1 \ K2 ¼ ;. Thus inconsistencies of them are completely distinct. That is, if the sets of MISs of K1 and K2 be denoted as SMIS1 and SMIS2 , respectively, then SMIS1 \ SMIS2 ¼ ;. If the modular ontology after merging K1 and K2 is denoted K, then it is not difficult to find that jKj ¼ jK1 j þ jK2 j according to Proposition 1. P P P
The specific criteria for Briand et al.’s cohesion metrics are used for validation of the NMIS metric: (1) Nonnegativity and normalization: The value of NMIS is never negative and the values can be compared between different versions of evolving ontologies. (2) Null value: Typically, if K is consistent, the value of NMIS is zero. (3) Monotonicity: The value of NMIS will never decrease when adding an axiom into K because it is impossible to make some MIS consistent after adding an axiom into K. (4) Cohesive modules: The value of NMIS of merged modules is never greater than the maximum NMIS of the original modules. The number of MISs of merged modules will never be greater than the maximum number of MISs in the original modules because it is impossible for two unrelated ontologies to form new MISs after they are merged.
n jKjþ1
3 4 5
http://www.flacp.fujitsulabs.com/tce/ontologies/2004/03/object.owl. http://www.protege.stanford.edu/plugins/owl/owl-library/koala.owl. http://www.mindswap.org/ontologies/debugging/university.owl. http://www.mindswap.org/2005/debugging/ontologies/miniTambis.owl.
Y. Ma et al. / The Journal of Systems and Software 83 (2010) 143–152
149
Fig. 4. Some information about datasets.
information about these ontologies is given in Fig. 4. From the second to the fifth column of the table, we list the four ontologies. For each example ontology, we count its number of class names, number of all classes (including anonymous classes), number of properties, number of individuals, number of axioms (assertions). We compute number of all classes because the example ontologies have been transformed for stable and comparable measurement. More experimental evaluation to these ontology cohesion metrics is encouraged by using more debugging ontologies. Using the NOP ontology cohesion metric to compare the above four ontologies clearly shows how they are intended to be used. Fig. 5 shows the specific NOP values in each ontology. From the figure it can be clearly seen that the mini-tambis ontology has a higher NOP value than other ontologies. This means that the mini-tambis ontology has more topics than other ontologies, and the concept organization and aggregation within the ontology are loose relatively. The NOP value of the Koala ontology is equivalent to 3, so contents of the ontology have thee topics. To certain extent, it has higher ontology cohesion than other ontologies, and is relatively easy to understand and reuse. For ontology engineers, either the ontology should be decomposed into multiple ontologies with a single topic, or new semantic relationships should be exploited and added between these partitions and form a relatively complete topic. Using the NMIS ontology cohesion metric to compare the above four ontologies clearly shows how they are intended to be used. Fig. 6 shows the specific NMIS values in each ontology. According
Fig. 5. NOP comparison between datasets.
Fig. 7. AVAI comparison between datasets.
to the figure, it can be clearly seen that the mini-tambis ontology has higher NMIS value than other ontologies. This shows that the mini-tambis ontology has a bigger scope of contents in conflict, and more modules of the ontology possibly cannot be effectively congregated together and achieve a close and unambiguous sharing. So it is rather difficult to understand and reuse. The Object ontology has a zero NMIS value, which shows that the concepts and properties of the ontology are well organized, and the ontology is easy to understand and reuse. In another aspect, in order to eliminate inconsistencies in MISs of the mini-tambis ontology, ontology developers also have to take more time and efforts to revise these inconsistencies in the ontology than the revision to other ontologies. In the following, we use the AVAI ontology cohesion metric to compare the above the four ontologies. Fig. 7 shows the specific AVAI values in each ontology. According to the figure, the University ontology has the highest average value of axiom inconsistencies, which means that the axioms in University ontology have a higher inconsistency degree to which these axioms belong to together. At first sight, it appears that a higher AVAI value in the University ontology shows the close relations between the axioms of the ontology. But in fact, this situation reflects that the ontology developers do not fully understand the domain knowledge very well. It will be difficult for the ontology to understand and revise. In order to revise all inconsistencies in the ontology, ontology developers have to take more time and efforts to eliminate ontology inconsistencies. The AVAI value of the Object ontology is 1. This shows that all axioms in the ontology have no any inconsistency. The ontology is designed in a good manner and easy to reuse. 6. Complete process of measuring ontology cohesion
Fig. 6. NMIS comparison between datasets.
In this section, we attempt to present a complete process and explain how the ontology cohesion metrics proposed in this paper can be reasonably used as a cogent complementarity of existing ontology cohesion metrics in order to measure ontology cohesion degree in the context of dynamic and changing Web. This process can generate stable ontology cohesion metrics and further measure cohesion degree including inconsistent ontologies. The process is illustrated as follows.
150
Y. Ma et al. / The Journal of Systems and Software 83 (2010) 143–152
(1) In order to measure ontology cohesion degree and assess quality of an ontology, we first select those ontology cohesion metrics with the precise definitions and the theoretical soundness of cohesion because such metrics reflect the nature of cohesion. We do this because the theoretical soundness of a measure is the prerequisite for its acceptability and use. This is similar to the following situation. Suppose that one defines a new measure, claiming it is a distance measure. However, the distance measure do not satisfy the triangle inequality characterizing the property of distance measures. Then we can say that it in essence is not a distance measure, and should not be selected to measure distances. (2) It is not enough that the selected ontology cohesion metrics have the theoretical soundness. We still need to inspect whether these metrics are stable. So called stable metrics are metrics that are stable with regards to possible additions of further axioms to the ontology (Vrandecic and Sure, 2007). For example, the metrics related to depth of inheritance tree of ontology are possible unstable. Taking the example in (Vrandecic and Sure, 2007), the knowledge base (ontology) is K ¼ ðfC P 1R >; D P 2R >; E P 3R >g; ;Þ. Authors inspect the maximal depth measure, and find that the maximal depth of the ontology is 1 according to the definition of the maximal depth measure. But in fact, it is not difficult to find that D v C and E v D. Intuitively, the maximal depth of the ontology should be 3. This means that the maximal depth measure is unstable. Similarly, Yao’s ADIT-LN metric (Haining et al., 2005) and Orme’s AF-RC metric (Orme Anthony et al., 2006) are unstable. Fortunately, Vrandecic and Sure (Vrandecic and Sure, 2007) proposed an ontology normalization approach to make an unstable metric stable. To address unstable metrics, the original ontologies being analyzed should be transformed in a semantic preserving manner. Then the selected metrics are stable with regards to the transformed ontologies. (3) Because ontologies in dynamic and changing environments are possibly inconsistent, we must judge whether the ontology being analyzed is consistent. If the ontology is consistent, we use those metrics selected from the existing ontology cohesion metrics according to the former steps. If it is inconsistent, we further need to inspect whether the selected metrics can be used to measure cohesion degree of inconsistent ontologies. If they cannot measure inconsistent ontologies, then our ontology cohesion metrics presented in this paper can be used for measuring cohesion of inconsistent ontologies. (4) Assessing quality of an consistent ontology is to help ontology users to select best ontology candidate for their applications, and also to guide ontology developers to improve the design of the ontology. Measuring an inconsistent ontology possibly can help ontology developers to decide the degree of inconsistencies in the ontology and to predict how much time and efforts will be taken for repairing inconsistencies and improving the design of the ontology. This may be done in an iterative manner.
7. Related work In the last years, many ontology metrics and measures have been presented and some principal work has been done to study the nature of metrics and measures for ontologies in general. With the development of Semantic Web technology, assessing quality of ontologies has become a key problem for semantic-driven applications.
Some foundational work has been done to study the theoretical validation frameworks. A quality oriented ontology description framework (QOOD) (Gangemi et al., 2006) and the O2 and oQual models (Gangemi et al., 2005) were proposed. The authors create semiotic models for ontology evaluation and validation, and described how measures should be built in order to actually assess quality. A framework for metrics called OntoMetric (Lozano-Tello and Gomez-Perez, 2004) was provided, which defines the relations between the different metrics, their attributes, and the quality attributes they capture. However, the use of the OntoMetric tool is not clearly defined, and the large number of characteristics makes the model difficult to understand (Tartir et al., 2005). Furthermore, it seems that the two frameworks mentioned above do not analyze the ontology cohesion. In contrast, Kitchenham et al. (1995) and Briand et al. (1996) proposed two general frameworks for software measurement validation from different viewpoints. The former described the structure of any measure as containing the entities, attribute, unit and the data scale. The latter presented the precise definitions of measurement concepts and characteristics such as size, length, complexity, cohesion and coupling. Many specific ontology metrics are proposed. oQual introduces distinct measure sets such as Measure types, the structural dimension, the functional dimension, NLP-driven evaluation, measuring the usability-profile of ontologies. Ontometric propose a complex framework consisting of 160 characteristics spread across five dimensions: content, language, methodology, tools, and costs (Lozano-Tello and Gomez-Perez, 2004). OntoQA tool implements a number of metrics (Tartir et al., 2005). It allows for the automatic measurement of ontologies. Some metrics are defined such as richness, population, or cohesion, and so on. But authors fail to define if these metrics are structurally or semantically defined (Vrandecic and Sure, 2007). Yao et al. proposed three ontology cohesion metrics such as Number of Root Classes, Number of Leaf Classes and Average Depth of Inheritance Tree, which are theoretically validated using Kitchenham et al.’s framework and Briand et al.’s criteria. Some ontology metrics also are proposed for coupling, complexity and other characteristics associated with favorable outcomes for an application (Orme Anthony et al., 2006; Orme et al., 2007; Burton-Jones et al., 2005). However, most of these metrics do not consider the stable ontology metrics which is one of the natures of ontology metrics. Most of these metrics still consider only ontology structure and seldom measure ontology semantics. Even if some of them are considering ontology semantics, they are unstable. Considering the dynamic and changing Web, ontologies on Web will continuously evolve. In the situation, stable ontology metrics are desirable. An approach for normalization of ontologies is proposed to explicate some features of the semantics of an ontology within its structure, and thus the structural metrics actually capture the semantics they are supposed to capture (Vrandecic and Sure, 2007). The ontology metrics designed by this approach are regarded as stable metrics. Another problem also is paid attention to: ontologies in the context of changing and dynamic Web will inevitably bring about internal inconsistencies with the evolution of ontologies. Deng et al. (2007) proposed a method of measuring inconsistencies in ontologies based so called Shapley value. Application of this approach can improve the quality of ontology diagnosis and repair in general. Qi and Hunter (2007) proposed three metrics to measuring so called ontology incoherence in ontologies but it seems that they failed to analyze characteristics of metrics. At last, we briefly discuss and compare theoretical validation to some ontology cohesion metrics proposed in other work. They include Yao et al.’s metrics, Orme et al.’s cohesion metrics, the Coh metric from OntoQA and Qi et al.’s metrics (Qi et al.’s metrics are compared because we find that they satisfies the properties of cohesion metrics). Meanwhile, we compare the difference among
Y. Ma et al. / The Journal of Systems and Software 83 (2010) 143–152
151
Fig. 8. Overview between different ontology cohesion metrics.
validation criteria and properties. The specific comparison is shown in Fig. 8. In the figure, ‘+’ (‘’) represents that some metric satisfies (does not satisfy) some criteria of the validation framework. ‘ST’ and ‘SE’ represent that some metric is used to measure ontology structure and ontology semantics, respectively. ‘S’ and ‘I’ denote Schema and individual, respectively. ‘Y’–Yes and ‘N’–No. 8. Conclusion In this paper, we propose a set of ontology cohesion metrics to measure the modular relatedness of ontologies in the context of dynamic and changing Web. We concentrate on inconsistent ontologies and fully consider the ontological semantics rather than ontological structure. These metrics proposed in this paper are stable and can be used as a very useful complementarity of existing ontology cohesion metrics. These metrics are theoretically and empirically validated. These ontology cohesion metrics can indicate cohesion quality of ontologies from different perspectives, and can help ontology developers and users to effectively assess the qualities of ontologies. Acknowledgement This work is supported by the Chinese National ‘‘863” High-Tech Program under Grant (Nos. 2006AA01Z231 and 2007AA04Z148). References Baader, Franz, Calvanese, Diego, McGuinness, Deborah, Nardi, Daniele, PatelSchneider, Peter, 2003. The description logic handbook: theory. Implementation and Applications. Cambridge University Press. Berners-Lee, T., Hendler, J., Lassila, O., 2001. The semantic web. Scientific American 284 (5), 34–43. Briand, Lionel C., Morasca, Sandro, Basili, Victor R., 1996. Property-based software engineering measurement. IEEE Transaction on Software Engineering 22 (1), 68–86. Burton-Jones, A., Storey, V.C., Sugumaran, V., Ahluwalia, P., 2005. A semiotic metrics suite for assessing the quality of ontologies. Data and Knowledge Engineering 55 (1), 84–102.
Chae, H.S., Kwon, Y.R., Bae, D.H., 2000. A cohesion measure for object-oriented classes. Software Practice and Experience 30 (12), 1405–1431. Chidamber, S.R., Kemerer, C.F., 1994. A metrics suite for object oriented design. IEEE Transaction on Software Engineering 20 (6), 476–493. Dalu, Zhang, Chuan, Ye, Zhe, Yang, 2006. An evaluation method for ontology complexity analysis in ontology evolution. In: Proceedings of EKAW2006, pp. 214–221. Dean, Mike, Schreiber, Guus, 2004. OWL Web Ontology Language Reference. W3C Recommendation 10 February 2004,
152
Y. Ma et al. / The Journal of Systems and Software 83 (2010) 143–152
Qi, Guilin, Hunter, Anthony, 2007. Measuring incoherence in description logicbased ontologies. In: Proceedings of ESWC2007, pp. 381–394. Sirin, E., Parsia, B., Grau, B.C., Kalyanpur, A., Pellet, Katz Y., 2007. A practical OWL-DL reasoner. Journal of Web Semantics 5 (2), 51–53. Tartir, S., Arpinar, I.B., Moore, M., Sheth, A.P., Aleman-Meza, B., 2005. OntoQA, metric-based ontology quality analysis. In: Proceedings of IEEEWorkshop on Knowledge Acquisition from Distributed, Autonomous, Semantically Heterogeneous Data and Knowledge Sources. Vrandecic, Denny, Sure, York, 2007. How to design better ontology metrics. In: Proceedings of ESWC2007, pp. 311–325.
Yinglong Ma received his Ph.D. degree in Institute of Software, Chinese Academy of Sciences, Beijing, PR China, in 2006, and his M.S. degree in computer science from Northwestern University, Xi’an, PR China, in 2002. Currently, he is an associate professor in Department of Computer Science, North China Electric Power, Beijing, PR China. His research interests include Semantic Web, ontology and knowledge engineering, and artificial intelligence and formal method. He has published over 30 papers in some international conferences and journals of these areas.
Beihong Jin received her B.S. degree in 1989 from Tsinghua University, Beijing, PR China, and her M.S. degree in 1992 and her Ph.D. degree in 1999 from Institute of Software, Chinese Academy of Sciences, Beijing, PR China, all in computer science. Currently she is a Professor at Institute of Software, Chinese Academy of Sciences. Her research interests include mobile and pervasive computing, middleware and distributed systems. She has published over 60 research papers in international conferences and journals of these areas and holds one China patent. She is a senior member of the CCF (China Computer Federation) and a member of the ACM.
Yulin Feng received his M.S. degree in 1967 from Department of Mathematics, Wuhan University, Wuhan, PR China, and his Ph.D. degree in 1982 from Institute of Computing Technology, Chinese Academy of Sciences, Beijing, PR China. From 1982 to 1985, he was in Carnegie Mellon University and Stanford University to do postdoctoral research work. Currently he is a principal professor at Technology Center of Software Engineering, Institute of Software, Chinese Academy of Sciences. His research interests include software engineering, middleware and distributed systems, and system formal specification and model verification. He has published over 80 research papers in international conferences and journals of these areas.
Contents lists available at ScienceDirect
The Journal of Systems and Software journal homepage: www.elsevier.com/locate/jss
Semantic oriented ontology cohesion metrics for ontology-based systems Yinglong Ma a,b,*, Beihong Jin b, Yulin Feng b a b
Department of Computer Sciences and Technology, North China Electric Power University, Beijing 102206, PR China Technology Center of Software Engineering, Institute of Software, Chinese Academy of Sciences, Beijing 100190, PR China
a r t i c l e
i n f o
Article history: Received 27 August 2008 Received in revised form 23 July 2009 Accepted 24 July 2009 Available online 6 August 2009 Keywords: Ontology Semantic metrics Ontology cohesion metrics Semantic inconsistency
a b s t r a c t Ontologies play a core role to provide shared knowledge models to semantic-driven applications targeted by Semantic Web. Ontology metrics become an important area because they can help ontology engineers to assess ontology and better control project management and development of ontology based systems, and therefore reduce the risk of project failures. In this paper, we propose a set of ontology cohesion metrics which focuses on measuring (possibly inconsistent) ontologies in the context of dynamic and changing Web. They are: Number of Ontology Partitions (NOP), Number of Minimally Inconsistent Subsets (NMIS) and Average Value of Axiom Inconsistencies (AVAI). These ontology metrics are used to measure ontological semantics rather than ontological structure. They are theoretically validated for ensuring their theoretical soundness, and further empirically validated by a standard test set of debugging ontologies. The related algorithms to compute these ontology metrics also are discussed. These metrics proposed in this paper can be used as a very useful complementarity of existing ontology cohesion metrics. Ó 2009 Elsevier Inc. All rights reserved.
1. Introduction With the development of the Semantic Web (Berners-Lee et al., 2001), ontologies play a core role to provide shared knowledge models to semantic-driven applications on Web. Ontologies involve agreed terms and their relationships in different domains. Different users can agree on the use of a common ontology in order to annotate their content or resolve their differences through interactions and negotiations. With the promising benefits provided by ontologies, domain ontology construction and reuse have become more and more important in all kinds of semantic-driven applications. There also are many domain ontologies such as the gene ontology (GO) and other ontologies from the Open Biology Ontologies (OBO, 2008). Although ontology construction is one of core issues for development of semantic-driven applications, it is rather important to assess the quality of an ontology. Assessing ontology can help ontology engineers to better control project management and development of ontology based systems, and therefore reduce the risk of project failures. From the viewpoint of ontology developers, by assessing quality of ontology, ontology developers can automatically recognize areas that might need more work and specify what parts of the ontology might cause problems. In the
* Corresponding author. Address: Department of Computer Sciences and Technology, North China Electric Power University, Beijing 102206, PR China. Tel.: +86 10 51963385.. E-mail addresses: [email protected] (Y. Ma), [email protected] (B. Jin), [email protected] (Y. Feng). 0164-1212/$ - see front matter Ó 2009 Elsevier Inc. All rights reserved. doi:10.1016/j.jss.2009.07.047
case, they can revise the ontology and further improve the quality of the ontology. From the viewpoint of ontology users, assessing qualities of ontologies allows them to compare between these different ontologies and select from these ontologies a higher quality ontology that is going to be used. As in many other related fields, you can only control what you can measure (DeMarco, 1982). Measuring ontology for ontology assessing has become an active area. In the last years, many ontology metrics and measures have been proposed and some principal work has been done to study the nature of measures for ontologies in general (Gangemi et al., 2005; Gangemi et al., 2006; Lozano-Tello and Gomez-Perez, 2004; Tartir et al., 2005; Haining et al., 2005). These metrics and principles provide some useful guides for ontology engineers about what methods are considered (e.g. structure or semantics), and how useful the method is (e.g. it is useful for ontology developers and ontology users). However, most metrics still are based on structural notions without taking into account the semantics which leads to incomparable measurement results (Vrandecic and Sure, 2007). First, most metrics are based on RDF graph structure. Only a very small number of metrics is considering ontology semantics such as subsumption. Second, these proposed metrics are unstable without considering possible additions of further axioms to an ontology because they have not taken the open world assumption (OWA) into account. OWA can satisfy the requirements of ontologies in the context of dynamic and changing Web. Furthermore, just because of dynamic and changing characters of ontologies on Web, a consistent ontology probably becomes inconsistent. Few ontology metrics are considering inconsistency of ontology. Third, most important, some ontology metrics are very
144
Y. Ma et al. / The Journal of Systems and Software 83 (2010) 143–152
often proposed to capture elusive concepts or characteristics such as cohesion (Tartir et al., 2005; Haining et al., 2005; Orme et al., 2007), complexity (Dalu et al., 2006; Orme et al., 2007), coupling (Orme Anthony et al., 2006), connectivity (Tartir et al., 2005), usability (Lozano-Tello and Gomez-Perez, 2004), etc. But most of the work except the work in the literatures (Haining et al., 2005; Orme Anthony et al., 2006; Orme et al., 2007), have not given the precise definitions and their theoretical soundness of these related concepts or characteristics. As mentioned in the literature (Briand et al., 1996), the theoretical soundness of a measure is the obvious prerequisite for its acceptability and use. Looking at the work in the literatures (Haining et al., 2005; Orme Anthony et al., 2006; Orme et al., 2007), although these metrics have theoretical soundness, it seems that these metrics still consider only ontology structure and seldom measure ontology semantics. Even if some of them are considering ontology semantics, they are unstable. In this paper, we propose a set of ontology cohesion metrics to measure the modular relatedness of ontologies in the context of dynamic and changing Web. We concentrate on measuring inconsistent in ontologies and fully consider the ontological semantics rather than ontological structure. These metrics are: Number of Ontology Partitions (NOP), Number of Minimally Inconsistent subsets (NMIS) and Average Value of Axiom Inconsistencies (AVAI). We also give the related algorithms computing these measurements. These metrics are theoretically validated by using standard metrics validation frameworks, and empirically validated by using a prototype implementing the metrics and algorithms presented in this paper. At last, we present a complete process and explain how the ontology cohesion metrics proposed in this paper can be reasonably used as a cogent complementarity of existing ontology cohesion metrics in order to measure ontology cohesion degree in the context of dynamic and changing Web. This paper is organized as follows. In Section 2, we introduce the preliminaries about ontology representation and criteria of analyzing metrics. In Section 3, we propose the set of our ontology cohesion metrics. The related algorithms computing these ontology metrics also are discussed. Section 4 gives the theoretical validation of these ontology cohesion metrics. In Section 5, we discuss the empirical validation of our ontology cohesion metrics. In Section 6, we briefly discuss the process of assessing quality of ontologies by combining our ontology cohesion metrics with other existing semantic cohesion metrics. Section 7 is the related work. We overview most of existing ontology metrics (including our metrics proposed in the paper) from different points of view, and give a comprehensive comparison of ontology cohesion metrics. Section 8 is conclusion.
2. Preliminaries 2.1. Ontology representation An ontology is defined as a formal, explicit specification of a shared conceptualization (Gruber, 1993). Ontologies can be represented in some ontology description languages such as Resource Description Framework (RDF) (Manola and Miller, 2004), Web Ontology Language (OWL) (Dean and Schreiber, 2004) and Description logic (DL) (Baader et al., 2003). OWL is an extension of RDF and a machine-readable language for sharing and reasoning information on the Web. It has been recommended as the standard web ontology language. An OWL ontology representation consists of axioms and facts. Axioms are the semantic knowledge defined by building relationships between classes and properties. Facts represent the individual assertions. There are three forms of OWL languages related to each other: OWL Lite, OWL DL and OWL Full. OWL Full contains OWL DL, and OWL DL contains OWL Lite. OWL
DL and OWL Lite correspond semantically with certain Description Logic languages. For examples, OWL DL is partially based on the DL SHOIN(D), which includes special constructors such as oneOf, transitive properties, inverse properties and datatype properties. OWL Lite is based on the less expressive DL SHIF(D). Ontology reasoning based on OWL Full is undecidable and existing reasoners do not support reasoning with all features of OWL Full. In this paper, the DL languages that we work on are those for which semantic reasoning for consistency checking is decidable. Generally speaking, a concept in DL is referred to as a class in OWL. A role in DL is a property in OWL. The terminological axioms and individuals have the same meaning in DL and OWL. Just because of the close connection between OWL and DLs, in this paper, we will make no distinction between ontologies (in OWL) and knowledge bases (in DL), and the related definition and examples also are given mainly in DL syntax. DL represents the knowledge of a domain by first defining the relevant concepts of the domain. Then these defined concepts are used to specify the properties of the objects in the domain. A knowledge base K can be defined as K ¼ ðT; AÞ, where T and A are TBox and Abox, respectively. TBox and ABox represent the sets of terminological axioms and individual assertions, respectively. In the TBox, basic descriptions are atomic concepts and atomic roles. Atomic concepts are designated by unary predicates. Atomic roles are designated by binary predicates to express relationships between individuals. Concept descriptions can be built on atomic concepts by iteratively applying constructors such as intersection ðuÞ, union ðtÞ, negation ðqÞ, value restriction ð8R CÞ and existential quantification ð9R CÞ, etc. Axioms express how concepts and roles are related to each other. Generally, An axiom is of the form C v D or C D, where C and D are concept descriptions. C v D if concept C is subsumed by concept D. C D if C v D and D v C. An ABox is a set of individual assertions of the form CðaÞ or Rða; bÞ, where R is a role, and a, b are individuals. For an interpretation I ¼ ðDI ; I Þ; I maps every atomic concept A to a subset AI # DI , and every atomic role R to a binary relation RI # DI DI , where DI is the domain, and I is the interpretation function. I satisfies C v D if C I # DI , and satisfies C D if C I ¼ DI . I satisfies CðaÞ if aI 2 C I , and satisfies Rða; bÞ if I ðaI ; b Þ 2 RI . I is a model of K if I is the model of T and A. I is a model of TðAÞ if it satisfies all axioms (assertions) in TðAÞ. A concept C is satisfiable if there is a model of T such that C T – ;. An knowledge base K has two forms of inconsistency: incoherent and inconsistent. The former refers that there is an unsatisfiable concept in TBox. The latter refers that K has no any model. For a knowledge base K ¼ ðT; AÞ, the tableau algorithm can be used to check the satisfiability of a concept in T and consistency of A w.r.t T. Its basic principle of checking the satisfiability of a concept C is to gradually build a model I of C such that C I is not empty. As a result, a tree-like model of concept can be built by decomposing concept C by using tableau expansion rules. The tableau algorithm terminates when either no more rules can be applicable, or when a clash occurs. 2.2. Cohesion metrics and validation framework It is desirable to have a formal model and precise theoretical foundation of metrics evaluation criteria, through which we can assess the usefulness and correctness of measures within well-defined contexts. In traditional software measurement, the concept of cohesion refers to the degree to which the elements in a module belong together. Especially for object-oriented software, cohesion refers to the degree of the relatedness or consistency in functionality of the members in a class. Cohesion measures separation of responsibilities, independence of components and control of com-
145
Y. Ma et al. / The Journal of Systems and Software 83 (2010) 143–152
plexity (Chae et al., 2000). Classes with strong cohesion is desirable for object oriented systems. One of the most widely known objectoriented cohesion metrics was proposed by Chidamber and Kemerer Chidamber and Kemerer (1994): Lack of Cohesion of Methods (LCOM). Because object-oriented conceptual model has a close association with ontology representation, software metric evaluation criteria (including such measurement concepts as cohesion, complexity and coupling, etc.) for object-oriented software can be regarded as a candidate evaluation framework for ontology quality assessing. Kitchenham et al. (1995) proposed a general framework for software measurement validation. They described the structure of any measure as containing the entities being analyzed, the attribute being measured, the unit used, and the data scale. In order to have any meaning for a value, they specified the entity, the attribute, and the units. The measure must be defined over a specified set of permissible values (discrete or continuous). A direct measure within the framework also should have four aspects to be theoretically valid, respectively. Briand et al. (1996) further presented a set of criteria for clarifying software measurement concepts such as complexity and cohesion. Different measurement concepts are involved in different characteristics and criteria. For example, as for cohesion metrics, the four aspects include: (1) nonnegativity and normalization, representing that the value is nonnegative and the values are comparable between different modules. (2) null value, i.e. the value is zero if there is no intramodule relationship within a module. (3) monotonicity, refers that the value never decreases when adding intramodule relationships into a module. And (4) cohesive module, i.e. the value after merging two unrelated modules is never greater than the maximum cohesion of the original modules. Here, intramodule relationships can be regarded as axioms in ontology representation. In this paper, our ontology cohesion metrics will be fitted into the frameworks of Kitchenham et al. (1995);Briand et al. (1996) for ensuring the metrics proposed theoretically correct. 3. Proposal of ontology cohesion metrics We first introduce some basic definitions about ontologies, which are the prerequisite of our ontology cohesion metrics. Then we will specifically define our metrics. 3.1. Basic definitions A knowledge base K can be defined as K ¼ ðT; AÞ, where T and A are TBox and Abox, respectively. In the following, we introduce some basic definitions. Definition 1. For K0 ¼ ðT0 ; A0 Þ and K ¼ ðT; AÞ, the followings hold:
K0 #K iff T0 #T and A0 #A: K0 K iff T0 T and A0 A: K0 \ K iff T0 \ T andA0 \ A: K0 [ K iff T0 [ T and A0 [ A: Definition 2. For a knowledge base K ¼ ðT; AÞ, its cardinality is denoted jKj, and jKj ¼ jT [ Aj. Proposition 1. For any K1 ; K2 # K, if K1 [ K2 ¼ K, then jKj ¼ jK1 j þ jK2 j jK1 \ K2 j. In the case, if K1 \ K2 ¼ ;, then jKj ¼ jK1 j þ jK2 j. In the TBox of knowledge base K, a concept is either an atomic one or a complex concept expression. Atomic concepts are named ones. For example, the term Student defined by < owl : Classrdf : ID ¼ Student= > is an atomic class name. In contrast, complex concepts have not named and are called anony-
mous classes in general. For example, some complex classes containing properties are sometimes defined by some annotations such as owl:Restriction, etc. Definition 3. For a knowledge base K ¼ ðT; AÞ, the set of defined concepts is denoted C ¼ fC 1 ; . . . ; C n g, where each C i ð1 6 i 6 nÞ is either an atomic concept or a complex concept defined in K. In this paper, we assume that all axioms in an TBox T are unfoldable ones because we want to avoid the so-called cycle axiom problem (Baader et al., 2003). Although blocking techniques can be applied to the tableau algorithm to resolve reasoning problem with cycle axioms, it is beyond the scope of this paper. We use the definition of unfoldable axioms from the literature (Schlobach et al., 2007). Definition 4. For any axiom a of the form A v B or A B, a is unfoldable iff A is a named concept that is atomic and unique, and B does not contain direct or indirect reference to A. As mentioned in Section 2.1, a knowledge base is inconsistent if and only if it has no model. Theoretically speaking, the tableau algorithm can be used to check the satisfiability of a concept in T and consistency of A w.r.t T. KAON21 adopts algorithms different from tableau algorithm for consistency checking (Motik, 2006). In practice, the existing reasoners (e.g. RACER Haarslev and Moller, 2006 and Sirin et al. (2007)) with visual ontology editors (e.g. Protege Stanford (2008) and SWOOP Kalyanpur et al., 2006), support inconsistency checking for ontology users and developers. Especially, Pellet supported by SWOOP has a strong capability for inconsistency checking based on ontologies. Different ontologies probably have different inconsistency degrees. Measuring the degree of ontology inconsistency can help ontology users and developers to appropriately modify and select ontologies for future reuse. In order to measure ontology inconsistency, we use the inconsistency value defined in the literature (Deng et al., 2007). Definition 5. Let K be a knowledge base. The function iv : 2K ! f0; 1g, assigns a value to each K0 . For any K0 # K, the inconsistency value of K0 can be defined as:
iv ðK0 Þ ¼
0;
if K0 is consistent or K0 is empty
1; otherwise
:
Example 1. Look at a simple ontology example in DL syntax. In knowledge base K ¼ ðT; AÞ; T contains axioms as follows: 1: A v B, 2: A v qB, 3: C v E u F; 4: E v 8s:G u D; 5: F v 9s:qG; A contains an individual assertion: 6: AðaÞ; 7: EðbÞ. For the sake of simplicity, we refer the axioms and assertions by their numbers. There are only K0 ¼ ðf1; 2g; f6gÞ and K00 ¼ ðf3; 4; 5g; f7gÞ such that iv ðK0 Þ ¼ 1 and iv ðK00 Þ ¼ 1. Because not all axioms are the source of ontology inconsistency in an inconsistent ontology. It is necessary to determine the set of axioms causing ontology inconsistency. We introduce the minimally inconsistent subsets (MIS) of K defined by the literature (Deng et al., 2007). Definition 6. For any subset K0 # K, K0 is the minimally inconsistent subset (MIS) of K if the following conditions hold: 1. iv ðKÞ ¼ 1, and 2. iv ðK00 Þ ¼ 0 for every K00 such that K00 K
1
http://www.kaon2.semanticweb.org/.
146
Y. Ma et al. / The Journal of Systems and Software 83 (2010) 143–152
Then we can obtain the set of all MISs in an inconsistent ontology, which is denoted SMIS. Furthermore, based on the set SMIS, we can define the inconsistency impact value of each axiom causing ontology inconsistency. Definition 7. For K ¼ ðT; AÞ, if its SMIS ¼ fmis1 ; mis2 ; . . . ; misn g. Then inconsistency impact value of each axiom or assertion a in K can be defined by a function mapping impv : ðT [ AÞ ! N, where
impv ðaÞ ¼
m þ 1; if for all i1 ; . . . ; ij ; . . . ; im ; a 2 misij 1;
if for each misk 2 SMIS; a R misk
:
where 1 6 j 6 m; 1 6 m; ij 6 n; 1 6 k 6 n and N is the set of natural number. We also need to consider that an ontology probably contains multiple parts which are semantically unrelated to each other. We called them semantic partitions. Ontology inconsistencies impossibly occur across multiple semantic partitions.
Fig. 1. computePart.
Definition 8. For an axiom a of the form A v B or A B, if B is a concept name, then the signature of the axiom a is denoted SignatureðaÞ ¼ fA; Bg. If B is a complex concept expression and contains concept names C 1 ; . . . ; C n in the expression, then the signature of the axiom a is SignatureðaÞ ¼ fA; C 1 ; . . . ; C n g. For the assertions of the form AðaÞ; SignatureðAðaÞÞ ¼ fAg.
ple and unrelated topics, and the concept organization and aggregation within the ontology are loose relatively. In the case, in order to enhance ontology cohesion, either the ontology should be decomposed into multiple ontologies with a single topic, or new semantic relationships should be exploited and added between these partitions.
Definition 9. If two axioms (assertions) a and b are semantically relevant, we denote SemRelev antða; bÞ. SemRelev antða; bÞ iff or 9A; BðA 2 SignatureðaÞ ^ B 2 SignatureðaÞ \ SignatureðbÞ – ; SignatureðbÞÞ such that there exists a subsumption relation between A and B by using certain reasoning paths of concept subsumption in knowledge base.
3.2.2. Definition of Number of Minimally Inconsistent Subsets (NMIS) Number of Minimally Inconsistent Subsets (NMIS) is the number of all minimally inconsistent subsets in a knowledge base. The definition of minimally inconsistent subset has been introduced in the previous section. NMIS of a knowledge base K can be formulated as follows:
Definition 10. For a knowledge base K ¼ ðT; AÞ; K0 # K is a semantic partition of K, where K0 ¼ ðT0 ; A0 Þ if and only if a and b are semantically irrelevant for any a 2 T0 [ A0 and for b 2 ðT [ AÞ n ðT0 [ A0 Þ.
NMISðKÞ ¼ jSMISj
3.2. Definitions of ontology cohesion metrics In this section, we present the following ontology cohesion metrics: Number of Ontology Partitions (NOP), Number of Minimally Inconsistent Subsets (NMIS) and Average Value of Axiom Inconsistencies (AVAI). We also give the related algorithms to compute the number of differen entities of these metrics. 3.2.1. Definition of Number of Ontology Partitions (NOP) Number of Ontology Partitions (NOP) is the number of semanticalpartitions of a knowledge base. NOP of a knowledge base K can be formulated as follows:
NOPðKÞ ¼ jPartSj
ð1Þ
where PartS is the set of semantic partitions in knowledge base K.
ð2Þ
where SMIS refers to the set of all minimally inconsistent subsets (MISs) in the knowledge base K. Here, we will give an algorithm computeSMIS to obtain the SMIS of the knowledge base K, which is shown in Fig. 2. In the algorithm, the possible subsets of a knowledge base first are traversed according to their size. Once a subset is decided as an MIS, then other subsets containing the subset will be ignored and no longer perform consistency checking. All found MISs are moved into the set SMIS. Number of Minimally Inconsistent Subsets (NMIS) can be used to measure the scope of inconsistency impacts of a knowledge base. Inconsistent axioms and assertions will impede the understanding and sharing of the whole knowledge base (ontology). Within an ontology with a large number of MISs, some ontological modules cannot be effectively congregated together and achieve a close and unambiguous sharing. The more there are MISs in an knowledge base, the more the knowledge base is difficult to share. Meanwhile, more MISs mean that ontology engineers have to take more time and efforts to revise these inconsistencies. Thus in the
Example 2. Look back to Example 1. K includes two partitions as follows: Part1 ¼ f1; 2; 6g and Part2 ¼ f3; 4; 5; 7g, i.e. PartS ¼ fPart1; Part2g. According to our definition of NOP, we know that NOPðKÞ ¼ jPartSj ¼ 2. The algorithm computePart obtaining ontology partitions of the knowledge base K is illustrated in Fig. 1. In the algorithm, all axioms will be tagged and compared with each other with respect to their semantic relevance. Those axioms semantically relevant will have the same partition. The variable NOP is used to record number of partitions, and then is returned. Number of Ontology Partitions (NOP) can be used to measure whether the contents of an ontology have a common topic. If there are multiple partitions in an ontology, then the ontology has multi-
Fig. 2. computeSMIS.
147
Y. Ma et al. / The Journal of Systems and Software 83 (2010) 143–152
phase of ontology design and construction, it is desirable to reduce axiom inconsistencies, which means a high ontology cohesion. Example 3. We revise K in Example 1 by adding the following axiom into K, i.e. 8:H v A. We will find that K1;2 ; 6 are K3;4;5 ; 7 are the minimally inconsistent subsets. But the subset like K1;2;8 ; 6 is not. Proposition 2. The algorithm computeSMIS can prevent double counting of MISs. In order to explain that the algorithm computeSMIS can prevent double counting of MISs, we only need to show that for any MISi ; MISj 2 SMIS and jMISi j 6 jMISj j; MISi – MISj and MISi å MISj hold, where i – j. On the one hand, according to the algorithm, all subsets of K are sorted by cardinality of K0 . Each subset is traversed only once in the whole iteration process. The subsets with same cardinality must be distinct, so MISi – MISj . On the other hand, in the algorithm, the supersets of all detected MISs will be ignored and are not considered for inconsistency checking at all, so MISi å MISj . According to the analysis above, this algorithm computeSMIS can effectively prevent double counting during collecting MISs. Preventing double counting is necessary to ensure the protocol validity of the NMIS metric. It is worth noting that NOP value of an ontology will probably help separate the ontology into multiple parts for future inconsistency checking and obtaining MISs. For example, if an ontology has a large volume of data, then it will cause high reasoning complexity. In the case, we can separate it into multiple subontologies according to their semantic partitions. Then we only need to find MISs from each partitions. This will reduce complexity of consistency checking and semantic reasoning. The reason is that the same MIS cannot exist across multiple partitions. It is impossible for two sets of axioms semantically irrelevant to have semantic inconsistency between them. 3.2.3. Definition of Average Value of Axiom Inconsistencies (AVAI) Average Value of Axiom Inconsistencies (AVAI) is the ratio of the sum of inconsistency impact values of all axioms and assertions to the cardinality of the knowledge base. But if the ontology is empty, the AVAI will encounter the problem that the denominator of the ratio is zero. To avoid this problem but not to cause new problems, the size of the knowledge base plus 1 will be used as the denominator. Specifically, the definition of AVAI metric of a knowledge base K ¼ ðT; AÞ can be formulated as follows:
P
AVAIðKÞ ¼
a2ðT[AÞ
impv ðaÞ
jKj þ 1
ð3Þ
We give a simple algorithm computeAVAI to compute the Average Value of Axiom Inconsistencies, which is shown in Fig. 3. The algorithm calculates inconsistency impact value in accord with the Definition 7. During the course of inner iteration, the inconsistency impact values of each axiom a is registered in the variable i. The sum of inconsistency impact value of all axioms is stored in variable AVAIvalue. AVAIvalue is further is transformed according to Eq. 3. AVAI ontology cohesion metric can be used to measure the inconsistent degree of ontology to which the axioms in the ontology belong together. Although the axioms in an ontology with a higher AVAI are possibly closely related to each other, but the ontology is more difficult to understand and eliminate inconsistencies. This situation reflects the fact that the ontology developers do not understand the domain knowledge very well. For ontology developers and users, they have to take more time and efforts to correctly understand and eliminate these inconsistencies. Obviously, a lower AVAI value in an ontology is desirable.
Fig. 3. computeAVAI.
Note that different axioms of an ontology possibly have different inconsistency impact values. The axioms with higher inconsistency impact values have more contribution to ontology inconsistencies than those with lower inconsistency impact values. Considering that, we should first select and revise the axioms with higher inconsistency impact values to improve ontology quality. In fact, it is not difficult to obtain the axiom(s) with the highest inconsistency impact value by mildly revising the algorithm in Fig. 3. Example 4. Look back to example in Example 3. We find impv ðiÞ ¼ 2 for any 1 6 i 6 7, and impv ð8Þ ¼ 1. Then AVAIðKÞ ¼ P impv ðaÞ a2ðT[AÞ 5 ¼ 14þ1 8þ1 ¼ 3. jKjþ1 4. Theoretical analysis and validation of ontology cohesion metrics In this section, each metric is examined and validated theoretically by the Kitchenham et al.’s framework (Kitchenham et al., 1995) and Briand et al.’s framework (Briand et al., 1996), which are briefly introduced in Section 2.2. 4.1. Analysis of NOP The metric NOP is a direct measure, in this case, to count the semantic partitions in ontologies. For NOP, the entity, attribute, unit and data scale are ontology, number of semantic partitions, semantic partition and interval, respectively. NOP satisfies Kitchenham, et al.’s four properties of for measurement validation as follows: (1) Attribute validity: the entity has the attribute (number of semantic partitions), which can be obtained by logical reasoning and algorithm in Fig. 1. (2) Unit validity: the attribute is measured by counting the number of semantic partitions. (3) Instrumental validity: The algorithm in Fig. 1 can be used to automatically collect measurement units. The instrument is valid as long as the metrics collecting tool correctly counts the number of semantic partitions in K. (4) Protocol validity: The calculation process can prevent double counting and be free from counting errors by counting the number of semantic partitions. We use specific criteria for Briand et al.’s cohesion metrics for validation of the NOP metric: (1) Nonnegativity and normalization: The value of NOP is never negative and the values can be compared between different versions of evolving ontologies. (2) Null value: Typically, if K is empty, the value of NOP is zero.
148
Y. Ma et al. / The Journal of Systems and Software 83 (2010) 143–152
(3) Monotonicity: is unapplicable because NOP value may decrease after adding an axiom into K. (4) Cohesive Modules: It is obvious that the value of NOP of merged modules is never greater than the maximum NOP of the original modules.
(2) Unit validity: the attribute is measured by counting the inconsistency impact values of all axioms (assertions). (3) Instrumental validity: The underlying theoretical model and algorithm is valid. Logic reasoning based inconsistency checking is used for detecting axiom consistencies, and algorithm in Fig. 3 can be used to automatically collect measurement units. The instrument is valid as long as the metric collecting tool correctly counts the inconsistency impact values of all axioms (assertions) in K. (4) Protocol validity: The calculation process is consistent and unambiguous, and will be free from counting errors.
4.2. Analysis of NMIS The metric NMIS is a direct measure, in this case, to count the minimally inconsistent subsets (MISs) in ontologies. For the NMIS measurement, the entity, attribute, unit and data scale are ontology, number of MISs, MIS and interval, respectively. NMIS satisfies Kitchenham et al.’s four properties of for measurement validation as follows:
We further use specific criteria for Briand et al.’s cohesion metrics for validation of the AVAI metric: (1) Nonnegativity and normalization: The value of AVAI is never negative and the values can be compared between different versions of evolving ontologies. (2) Null value: Typically, if K is consistent, the value of AVAI is zero. (3) Monotonicity: The value of AVAI will never decrease when adding an axiom into K. Assume the AVAI value of K is n . The AVAI value of K0 after adding an axiom AVAIðKÞ ¼ jKjþ1
(1) Attribute validity: the entity has the attribute (number of MISs), which can be obtained and calculated by logical reasoning and the algorithm in Fig. 2. (2) Unit validity: the attribute is measured by counting the number of MISs. (3) Instrumental validity: The underlying theoretical model and algorithm are valid. Logic reasoning based consistency checking is used for detecting axiom inconsistencies, and the algorithm in Fig. 2 can be used to automatically collect measurement units. The instrument is valid as long as the metric collecting tool correctly counts the number of MISs in the ontology (knowledge base) K. (4) Protocol validity: Calculations will be performed according to the formal inconsistent checking model and algorithm. The calculation process can prevent double counting and be free from counting errors by counting the number of MISs.
nþj . In fact, AVAIðKÞ < AVAIðK0 Þ into K, AVAIðK0 Þ ¼ jKjþ2
always holds because
The AVAI metric is a direct measure to count inconsistency impact values of all axioms or assertions. All MISs in K are used to obtain AVAI value. For AVAI, the entity, attribute, unit and data scale are ontology, Average value of axiom inconsistence, Inconsistent impact value per axiom and interval, respectively. AVAI satisfies Kitchenham, et al’s four aspects for measurement criteria as follows:
b2ðT2 [A2 Þ
impv ðdÞ
jKjþ1
impv ðaÞ
P jK1 jþjK2 jþ1
þ
¼ P
a2ðT1 [A1 Þ
impv ðaÞþ
b2ðT2 [A2 Þ
impv ðbÞ
jK1 jþjK2 jþ1 b2ðT2 [A2 Þ
P
impv ðbÞ
jK1 jþjK2 jþ1
6
a2ðT1 [A1 Þ
impv ðaÞ
jK1 jþ1
¼ þ
impv ðbÞ
jK2 jþ1
. That is, AVAIðKÞ 6 AVAIðK1 Þ þ AVAIðK2 Þ
always holds. It is not difficult to find that AVAI satisfies the aspect of cohesive modules
5. Empirical validation of our ontology cohesion metrics To evaluate whether these ontology cohesion metrics are useful for ontologies, we implemented the metrics and algorithms in Java using KAON2, and have performed some preliminary experiments. Four ontologies are use as the data sets to calculate the ontology cohesion metrics. They are object ontology,2 Koala ontology,3 university ontology4 and mini-tambis ontology.5 We obtain all the MISs of an ontology using the algorithm proposed in this paper. The related cohesion metrics for these ontologies can be calculated by the corresponding algorithms described in this paper. In order to generalize our ontology cohesion measurement for possible inconsistent ontologies, we need to revise some of these debugging ontologies to form a relevant complete knowledge bases, that is, both TBoxs and ABoxs of these knowledge bases are not empty. Some of these ontologies have no individuals, i.e. their ABoxs are empty. We added some new individuals for some unsatisfiable concepts in these (inconsistent) ontologies. Some 2
(1) Attribute validity: the entity has the attribute (inconsistent impact value), which can be calculated by Definition 7 and Eq. 3.
d2ðT[AÞ
AVAIðKÞ ¼ P a2ðT1 [A1 Þ
4.3. Analysis of AVAI
nþj 6 jKjþ2 always holds, where
n 6 jKj and j ¼ impv ðaÞ P 1. (4) Cohesive modules: Assume that K1 and K2 are unrelated. This means that K1 \ K2 ¼ ;. Thus inconsistencies of them are completely distinct. That is, if the sets of MISs of K1 and K2 be denoted as SMIS1 and SMIS2 , respectively, then SMIS1 \ SMIS2 ¼ ;. If the modular ontology after merging K1 and K2 is denoted K, then it is not difficult to find that jKj ¼ jK1 j þ jK2 j according to Proposition 1. P P P
The specific criteria for Briand et al.’s cohesion metrics are used for validation of the NMIS metric: (1) Nonnegativity and normalization: The value of NMIS is never negative and the values can be compared between different versions of evolving ontologies. (2) Null value: Typically, if K is consistent, the value of NMIS is zero. (3) Monotonicity: The value of NMIS will never decrease when adding an axiom into K because it is impossible to make some MIS consistent after adding an axiom into K. (4) Cohesive modules: The value of NMIS of merged modules is never greater than the maximum NMIS of the original modules. The number of MISs of merged modules will never be greater than the maximum number of MISs in the original modules because it is impossible for two unrelated ontologies to form new MISs after they are merged.
n jKjþ1
3 4 5
http://www.flacp.fujitsulabs.com/tce/ontologies/2004/03/object.owl. http://www.protege.stanford.edu/plugins/owl/owl-library/koala.owl. http://www.mindswap.org/ontologies/debugging/university.owl. http://www.mindswap.org/2005/debugging/ontologies/miniTambis.owl.
Y. Ma et al. / The Journal of Systems and Software 83 (2010) 143–152
149
Fig. 4. Some information about datasets.
information about these ontologies is given in Fig. 4. From the second to the fifth column of the table, we list the four ontologies. For each example ontology, we count its number of class names, number of all classes (including anonymous classes), number of properties, number of individuals, number of axioms (assertions). We compute number of all classes because the example ontologies have been transformed for stable and comparable measurement. More experimental evaluation to these ontology cohesion metrics is encouraged by using more debugging ontologies. Using the NOP ontology cohesion metric to compare the above four ontologies clearly shows how they are intended to be used. Fig. 5 shows the specific NOP values in each ontology. From the figure it can be clearly seen that the mini-tambis ontology has a higher NOP value than other ontologies. This means that the mini-tambis ontology has more topics than other ontologies, and the concept organization and aggregation within the ontology are loose relatively. The NOP value of the Koala ontology is equivalent to 3, so contents of the ontology have thee topics. To certain extent, it has higher ontology cohesion than other ontologies, and is relatively easy to understand and reuse. For ontology engineers, either the ontology should be decomposed into multiple ontologies with a single topic, or new semantic relationships should be exploited and added between these partitions and form a relatively complete topic. Using the NMIS ontology cohesion metric to compare the above four ontologies clearly shows how they are intended to be used. Fig. 6 shows the specific NMIS values in each ontology. According
Fig. 5. NOP comparison between datasets.
Fig. 7. AVAI comparison between datasets.
to the figure, it can be clearly seen that the mini-tambis ontology has higher NMIS value than other ontologies. This shows that the mini-tambis ontology has a bigger scope of contents in conflict, and more modules of the ontology possibly cannot be effectively congregated together and achieve a close and unambiguous sharing. So it is rather difficult to understand and reuse. The Object ontology has a zero NMIS value, which shows that the concepts and properties of the ontology are well organized, and the ontology is easy to understand and reuse. In another aspect, in order to eliminate inconsistencies in MISs of the mini-tambis ontology, ontology developers also have to take more time and efforts to revise these inconsistencies in the ontology than the revision to other ontologies. In the following, we use the AVAI ontology cohesion metric to compare the above the four ontologies. Fig. 7 shows the specific AVAI values in each ontology. According to the figure, the University ontology has the highest average value of axiom inconsistencies, which means that the axioms in University ontology have a higher inconsistency degree to which these axioms belong to together. At first sight, it appears that a higher AVAI value in the University ontology shows the close relations between the axioms of the ontology. But in fact, this situation reflects that the ontology developers do not fully understand the domain knowledge very well. It will be difficult for the ontology to understand and revise. In order to revise all inconsistencies in the ontology, ontology developers have to take more time and efforts to eliminate ontology inconsistencies. The AVAI value of the Object ontology is 1. This shows that all axioms in the ontology have no any inconsistency. The ontology is designed in a good manner and easy to reuse. 6. Complete process of measuring ontology cohesion
Fig. 6. NMIS comparison between datasets.
In this section, we attempt to present a complete process and explain how the ontology cohesion metrics proposed in this paper can be reasonably used as a cogent complementarity of existing ontology cohesion metrics in order to measure ontology cohesion degree in the context of dynamic and changing Web. This process can generate stable ontology cohesion metrics and further measure cohesion degree including inconsistent ontologies. The process is illustrated as follows.
150
Y. Ma et al. / The Journal of Systems and Software 83 (2010) 143–152
(1) In order to measure ontology cohesion degree and assess quality of an ontology, we first select those ontology cohesion metrics with the precise definitions and the theoretical soundness of cohesion because such metrics reflect the nature of cohesion. We do this because the theoretical soundness of a measure is the prerequisite for its acceptability and use. This is similar to the following situation. Suppose that one defines a new measure, claiming it is a distance measure. However, the distance measure do not satisfy the triangle inequality characterizing the property of distance measures. Then we can say that it in essence is not a distance measure, and should not be selected to measure distances. (2) It is not enough that the selected ontology cohesion metrics have the theoretical soundness. We still need to inspect whether these metrics are stable. So called stable metrics are metrics that are stable with regards to possible additions of further axioms to the ontology (Vrandecic and Sure, 2007). For example, the metrics related to depth of inheritance tree of ontology are possible unstable. Taking the example in (Vrandecic and Sure, 2007), the knowledge base (ontology) is K ¼ ðfC P 1R >; D P 2R >; E P 3R >g; ;Þ. Authors inspect the maximal depth measure, and find that the maximal depth of the ontology is 1 according to the definition of the maximal depth measure. But in fact, it is not difficult to find that D v C and E v D. Intuitively, the maximal depth of the ontology should be 3. This means that the maximal depth measure is unstable. Similarly, Yao’s ADIT-LN metric (Haining et al., 2005) and Orme’s AF-RC metric (Orme Anthony et al., 2006) are unstable. Fortunately, Vrandecic and Sure (Vrandecic and Sure, 2007) proposed an ontology normalization approach to make an unstable metric stable. To address unstable metrics, the original ontologies being analyzed should be transformed in a semantic preserving manner. Then the selected metrics are stable with regards to the transformed ontologies. (3) Because ontologies in dynamic and changing environments are possibly inconsistent, we must judge whether the ontology being analyzed is consistent. If the ontology is consistent, we use those metrics selected from the existing ontology cohesion metrics according to the former steps. If it is inconsistent, we further need to inspect whether the selected metrics can be used to measure cohesion degree of inconsistent ontologies. If they cannot measure inconsistent ontologies, then our ontology cohesion metrics presented in this paper can be used for measuring cohesion of inconsistent ontologies. (4) Assessing quality of an consistent ontology is to help ontology users to select best ontology candidate for their applications, and also to guide ontology developers to improve the design of the ontology. Measuring an inconsistent ontology possibly can help ontology developers to decide the degree of inconsistencies in the ontology and to predict how much time and efforts will be taken for repairing inconsistencies and improving the design of the ontology. This may be done in an iterative manner.
7. Related work In the last years, many ontology metrics and measures have been presented and some principal work has been done to study the nature of metrics and measures for ontologies in general. With the development of Semantic Web technology, assessing quality of ontologies has become a key problem for semantic-driven applications.
Some foundational work has been done to study the theoretical validation frameworks. A quality oriented ontology description framework (QOOD) (Gangemi et al., 2006) and the O2 and oQual models (Gangemi et al., 2005) were proposed. The authors create semiotic models for ontology evaluation and validation, and described how measures should be built in order to actually assess quality. A framework for metrics called OntoMetric (Lozano-Tello and Gomez-Perez, 2004) was provided, which defines the relations between the different metrics, their attributes, and the quality attributes they capture. However, the use of the OntoMetric tool is not clearly defined, and the large number of characteristics makes the model difficult to understand (Tartir et al., 2005). Furthermore, it seems that the two frameworks mentioned above do not analyze the ontology cohesion. In contrast, Kitchenham et al. (1995) and Briand et al. (1996) proposed two general frameworks for software measurement validation from different viewpoints. The former described the structure of any measure as containing the entities, attribute, unit and the data scale. The latter presented the precise definitions of measurement concepts and characteristics such as size, length, complexity, cohesion and coupling. Many specific ontology metrics are proposed. oQual introduces distinct measure sets such as Measure types, the structural dimension, the functional dimension, NLP-driven evaluation, measuring the usability-profile of ontologies. Ontometric propose a complex framework consisting of 160 characteristics spread across five dimensions: content, language, methodology, tools, and costs (Lozano-Tello and Gomez-Perez, 2004). OntoQA tool implements a number of metrics (Tartir et al., 2005). It allows for the automatic measurement of ontologies. Some metrics are defined such as richness, population, or cohesion, and so on. But authors fail to define if these metrics are structurally or semantically defined (Vrandecic and Sure, 2007). Yao et al. proposed three ontology cohesion metrics such as Number of Root Classes, Number of Leaf Classes and Average Depth of Inheritance Tree, which are theoretically validated using Kitchenham et al.’s framework and Briand et al.’s criteria. Some ontology metrics also are proposed for coupling, complexity and other characteristics associated with favorable outcomes for an application (Orme Anthony et al., 2006; Orme et al., 2007; Burton-Jones et al., 2005). However, most of these metrics do not consider the stable ontology metrics which is one of the natures of ontology metrics. Most of these metrics still consider only ontology structure and seldom measure ontology semantics. Even if some of them are considering ontology semantics, they are unstable. Considering the dynamic and changing Web, ontologies on Web will continuously evolve. In the situation, stable ontology metrics are desirable. An approach for normalization of ontologies is proposed to explicate some features of the semantics of an ontology within its structure, and thus the structural metrics actually capture the semantics they are supposed to capture (Vrandecic and Sure, 2007). The ontology metrics designed by this approach are regarded as stable metrics. Another problem also is paid attention to: ontologies in the context of changing and dynamic Web will inevitably bring about internal inconsistencies with the evolution of ontologies. Deng et al. (2007) proposed a method of measuring inconsistencies in ontologies based so called Shapley value. Application of this approach can improve the quality of ontology diagnosis and repair in general. Qi and Hunter (2007) proposed three metrics to measuring so called ontology incoherence in ontologies but it seems that they failed to analyze characteristics of metrics. At last, we briefly discuss and compare theoretical validation to some ontology cohesion metrics proposed in other work. They include Yao et al.’s metrics, Orme et al.’s cohesion metrics, the Coh metric from OntoQA and Qi et al.’s metrics (Qi et al.’s metrics are compared because we find that they satisfies the properties of cohesion metrics). Meanwhile, we compare the difference among
Y. Ma et al. / The Journal of Systems and Software 83 (2010) 143–152
151
Fig. 8. Overview between different ontology cohesion metrics.
validation criteria and properties. The specific comparison is shown in Fig. 8. In the figure, ‘+’ (‘’) represents that some metric satisfies (does not satisfy) some criteria of the validation framework. ‘ST’ and ‘SE’ represent that some metric is used to measure ontology structure and ontology semantics, respectively. ‘S’ and ‘I’ denote Schema and individual, respectively. ‘Y’–Yes and ‘N’–No. 8. Conclusion In this paper, we propose a set of ontology cohesion metrics to measure the modular relatedness of ontologies in the context of dynamic and changing Web. We concentrate on inconsistent ontologies and fully consider the ontological semantics rather than ontological structure. These metrics proposed in this paper are stable and can be used as a very useful complementarity of existing ontology cohesion metrics. These metrics are theoretically and empirically validated. These ontology cohesion metrics can indicate cohesion quality of ontologies from different perspectives, and can help ontology developers and users to effectively assess the qualities of ontologies. Acknowledgement This work is supported by the Chinese National ‘‘863” High-Tech Program under Grant (Nos. 2006AA01Z231 and 2007AA04Z148). References Baader, Franz, Calvanese, Diego, McGuinness, Deborah, Nardi, Daniele, PatelSchneider, Peter, 2003. The description logic handbook: theory. Implementation and Applications. Cambridge University Press. Berners-Lee, T., Hendler, J., Lassila, O., 2001. The semantic web. Scientific American 284 (5), 34–43. Briand, Lionel C., Morasca, Sandro, Basili, Victor R., 1996. Property-based software engineering measurement. IEEE Transaction on Software Engineering 22 (1), 68–86. Burton-Jones, A., Storey, V.C., Sugumaran, V., Ahluwalia, P., 2005. A semiotic metrics suite for assessing the quality of ontologies. Data and Knowledge Engineering 55 (1), 84–102.
Chae, H.S., Kwon, Y.R., Bae, D.H., 2000. A cohesion measure for object-oriented classes. Software Practice and Experience 30 (12), 1405–1431. Chidamber, S.R., Kemerer, C.F., 1994. A metrics suite for object oriented design. IEEE Transaction on Software Engineering 20 (6), 476–493. Dalu, Zhang, Chuan, Ye, Zhe, Yang, 2006. An evaluation method for ontology complexity analysis in ontology evolution. In: Proceedings of EKAW2006, pp. 214–221. Dean, Mike, Schreiber, Guus, 2004. OWL Web Ontology Language Reference. W3C Recommendation 10 February 2004,
152
Y. Ma et al. / The Journal of Systems and Software 83 (2010) 143–152
Qi, Guilin, Hunter, Anthony, 2007. Measuring incoherence in description logicbased ontologies. In: Proceedings of ESWC2007, pp. 381–394. Sirin, E., Parsia, B., Grau, B.C., Kalyanpur, A., Pellet, Katz Y., 2007. A practical OWL-DL reasoner. Journal of Web Semantics 5 (2), 51–53. Tartir, S., Arpinar, I.B., Moore, M., Sheth, A.P., Aleman-Meza, B., 2005. OntoQA, metric-based ontology quality analysis. In: Proceedings of IEEEWorkshop on Knowledge Acquisition from Distributed, Autonomous, Semantically Heterogeneous Data and Knowledge Sources. Vrandecic, Denny, Sure, York, 2007. How to design better ontology metrics. In: Proceedings of ESWC2007, pp. 311–325.
Yinglong Ma received his Ph.D. degree in Institute of Software, Chinese Academy of Sciences, Beijing, PR China, in 2006, and his M.S. degree in computer science from Northwestern University, Xi’an, PR China, in 2002. Currently, he is an associate professor in Department of Computer Science, North China Electric Power, Beijing, PR China. His research interests include Semantic Web, ontology and knowledge engineering, and artificial intelligence and formal method. He has published over 30 papers in some international conferences and journals of these areas.
Beihong Jin received her B.S. degree in 1989 from Tsinghua University, Beijing, PR China, and her M.S. degree in 1992 and her Ph.D. degree in 1999 from Institute of Software, Chinese Academy of Sciences, Beijing, PR China, all in computer science. Currently she is a Professor at Institute of Software, Chinese Academy of Sciences. Her research interests include mobile and pervasive computing, middleware and distributed systems. She has published over 60 research papers in international conferences and journals of these areas and holds one China patent. She is a senior member of the CCF (China Computer Federation) and a member of the ACM.
Yulin Feng received his M.S. degree in 1967 from Department of Mathematics, Wuhan University, Wuhan, PR China, and his Ph.D. degree in 1982 from Institute of Computing Technology, Chinese Academy of Sciences, Beijing, PR China. From 1982 to 1985, he was in Carnegie Mellon University and Stanford University to do postdoctoral research work. Currently he is a principal professor at Technology Center of Software Engineering, Institute of Software, Chinese Academy of Sciences. His research interests include software engineering, middleware and distributed systems, and system formal specification and model verification. He has published over 80 research papers in international conferences and journals of these areas.