- Email: [email protected]
Dynamic fuzzy data analysis based on similarity between functions
'f;
"
t
FUZZY
sets and systems ELSEVIER
Fuzzy Sets and Systems 105 (1999) 81 90
Dynamic fuzzy data analysis based on similarity between functions A. Joentgen a,., L. Mikenina a, R. Weber b, H.-J. Zimmermann a a
Lehrstuhlfiir Unternehmensforsehung, R WTH Aachen, Templeroraben 64, 52062 Aachen, Germany b MIT GmbH, Promenade 9, 52076 Aachen, Germany
Received April 1998; revised July 1998
Abstract
In data analysis, objects are usually represented by feature vectors, each describing a state of an object at a point of time. Most methods for data analysis use only these feature vectors and do not take into account changes over time. They can therefore be called static, But often a "dynamic" approach, which utilizes the feature changes over time, seems to be more appropriate (e.g. supervision of patients in medical care, state-dependent maintenance of machines, classification of shares). In this paper, different criteria for structuring the field of "dynamic data analysis (DDA)" are proposed and one of the relevant approaches is investigated in more detail. This approach considers possible ways to handle dynamics within static methods for data analysis. In doing this, different types of similarity measures for trajectories are defined, which can be used to modify static methods for data analysis. One of the proposed similarity measures has been integrated into the fuzzy c-means. An application example is used to demonstrate the applicability of the modified fuzzy c-means. (~) 1999 Elsevier Science B.V. All rights reserved. Keywords: Cluster analysis; Pattern recognition; Dynamic data analysis; Acoustic quality control
I. M o t i v a t i o n
Classification o f objects using known features is one o f the most frequent tasks in data analysis [1,2,6,9]. Usually, objects are represented by feature vectors, which often consist o f numbers, describing the object at a certain moment. The development o f objects over time (and therefore the development o f the features) is not considered explicitly or is taken into account by just using single values o f the past in the feature vector. Methods that use this type o f feature vectors can be called static. In many applications, however, explicit * Corresponding author.
consideration o f trajectories rather than single points is desirable, e.g.: • monitoring o f patients in medicine, e.g. during narcosis, where the development of the patients' condition is essential; • state-dependent machine maintenance; • rating o f shares: the examination o f the development o f share prices and other characteristics allows better estimates than just considering the current share price. In all cases, where a dynamic viewpoint is desirable, the momentary snapshot for some components o f the feature vector may be replaced by a trajectory o f this feature. Thus, dynamic objects are represented by multi-dimensional trajectories in the feature
0165-0114/99/$ - see front matter (~) 1999 Elsevier Science B.V. All rights reserved. PII: S0165-0114(98)00337-6
82
A. Joentgen et aL /Fuzzy Sets and Systems 105 (1999) 81-90
40 30
40
feature 1 +
+
feature 1
30
4"
20
20
.0,
10.
10
i'x_.-.,__.
; j,f
feature 2 o" "" "ib' ' ' ~,b' "' "3U" "' ~ " "
"eb'" "'eb
(a)
J" "--"---,-. feature 2
o"'" 'ib'"""2o"""W ' " ;~'" "~d" ~b (b)
Fig. 1. (a) States of objects at a point of time; (b) projectionsof trajectories over time into the feature space. space. Since most methods for data analysis are not suited to classify objects described by trajectories, new methods for dynamic data analysis have to be developed. Fig. 1 illustrates the difference between classical (static) and dynamic data analysis. Consider a twodimensional feature space with one additional time dimension and suppose that a set of objects is observed over time. States of objects at a point of time can be seen in the cut of this three-dimensional space at the current moment (Fig. la). Two classes of objects can easily be distinguished in this plane. However, if the trace of each object from the initial to its current state (i.e. its trajectory) is considered and projected into the feature space (Fig. lb), other classes may seem more reasonable. This paper is structured as follows: In Section 2, a taxonomy for the classification of different topics related to dynamic data analysis is proposed and illustrated with some examples. Section 3 describes two types of similarity between trajectories, which could be relevant for different problems, and presents methods to determine the similarity measure in both cases. In Section 4, two approaches are proposed, which allow the classification of dynamic objects by static methods. The modification of classical methods, in particular the fuzzy c-means algorithm, is illustrated by an application example.
2. Taxonomy for dynamic data analysis In the following, different criteria are listed, which are suitable to structure the field of dynamic data
analysis. The criteria themselves are divided into two groups: problem-related criteria and methodrelated criteria. Problem related criteria depend on the structure of the observed system and the desired results. Method-related criteria refer to methods and techniques, which are used in classifier design and classification. Essentially they include selection and processing of relevant features. The field of dynamic data analysis can be structured using the following Taxonomy: Problem-related criteria:
1. Classification object: (a) classification of several systems, (b) classification of states of one system. 2. Type of class structure: (a) static class structure, (b) dynamic class structure. 3. Observation period: (a) point of time, (b) rolling horizon. Method-related criteria:
1. Handling of dynamics (trajectories): (a) during preprocessing, (b) within data analysis methods. 2. Type of preprocessing of the trajectories: (a) directly measured trajectories (no preprocessing), (b) aggregated trajectories, (c) ratio between trajectories. 3. Time dependence of the feature set: (a) constant feature set, (b) variable feature set. Table 1 shows some examples of combinations of problem-related criteria.
83
A. Joentgen et al./Fuzzy Sets and Systems 105 (1999) 81-90 Table 1 Examples of combinations of problem related criteria Condition rating for a known machine
lb, 2a, 3a
Preventive inspection/state dependent maintenance of a known machine
l b, 2a, 3b
Monitoring and classifier adaptation for an unknown machine, machine installation
lb, 2b, 3b
Comparison of machines using a given class structure, quality control
l a, 2a, 3a
Continuous parallel monitoring of machines
l a, 2a, 3b
Determination and dynamic adaptation of the state structure of a group of unknown machines
l a, 2b, 3b
natural (depending on the chosen type o f structural similarity). The following two sections describe these two types o f similarity and the relationships between them.
35
253° is
i Y
i
10
>.ffl
s 0
i ! ix,
0
20
40
60
80
100 120 140 160 180 200
Fig. 2. Structural and pointwise similarity.
3. Structural and pointwise similarity As stated before, the components o f feature vectors describing dynamic objects are trajectories. Starting from the fact that most methods for data analysis use a distance measure or a similarity measure as a criteflon to classify objects, one way to handle dynamic objects is to define the similarity measure for trajectories (functions) and to use it within existing or perhaps completely new methods. Similarity of trajectories can be defined in different ways. Basically, two viewpoints can be distinguished. The more similar two trajectories are • the better they match in form/evolution/characteristics (structural similarity). • the smaller their (pointwise) distance in feature space is (pointwise similarity). Fig. 2 gives an example o f the differences between structural and pointwise similarity. In terms of pointwise similarity A and B would be grouped together as well as C and D. But in terms o f structural similarity the grouping {A, D} and {B, C} seems to be more
3.1. Definition of structural similarity between functions Structural similarity relates to a variety o f aspects of the trajectories (functions) under consideration: form, evolution, size or orientation (of trajectories in ~R") are some examples. Depending on the chosen aspect, different criteria may be relevant to describe similarity, e.g. slope, curvature, position and values of extremal points or other information like smoothness or monotonicity (as a degree o f membership of a trajectory to the set of monotone functions). Here some examples of structural similarity are given for illustration: (A) Slope and curvature of trajectories are relevant, but their position in the feature space is not relevant: The functions y = x and y -- 1.001 • x + 100 are similar (both describe straight lines with approximately equal slope and a curvature of zero), whereas y = x and y = x + 0.001 * sin(x) are not similar (despite the fact that they are much closer in terms of Euclidean distance). This type o f definition o f similarity can be applied to classify, e.g., shares as "decreasing" (A, E, H), 'fincreasing" (B, D, G, J, K), "constant" (F, I ) or "fluctuating" (C), depending on the trajectories of their share prices (Fig. 3). (B) Form of trajectories is relevant, but their size and position in the feature space are not relevant: The unit circle (center at (0, 0) and radius 1 ) and the circle with center at (100,0) and radius 17.4 are similar to
84
A. Joentgen et aL /Fuzzy Sets and Systems 105 (1999) 81-90 80 70
60
B
50 o o ....
.~ 40 t~
3O
m m
G H 20 ~
m
m
m
m
~
10
Time
Fig. 3. Fictitious developments of share prices.
degree 1, whereas the unit circle and the unit square are much less similar. This type of definition may be applied to classify engines, using the airborne sound they emit during operation: amplitude and position of characteristic patterns change with speed, independent of the state of the engine. However, the characteristic pattern remains the same depending on the fact, whether an engine is intact or damaged (Fig. 4). In some cases, structural similarity can be reduced to pointwise similarity, for instance in the first example, by considering the pointwise similarity for the first and the second derivatives of the functions, respectively. One method to define structural similarity between functions is to consider relevant characteristics of these functions (e.g. integral, extrema), which contain the information about the specific structure of functions. The following algorithm to determine a measure for structural similarity between arbitrary functions f and g is proposed: 1. A set of relevant characteristics Ki, i = 1,... ,m, describing structural similarity is chosen.
2. A fuzzy set Ai labeled "admissible difference for characteristic Ki" with membership function ]2i is defined. 3. All characteristics K i ( f ) for the function f and Ki(g) for the function g are calculated. 4. For each characteristic Ki the difference A K i = I K i ( f ) - Ki(g)], i = 1. . . . . m, is calculated. 5. The degree of membership si --/~i(z3K/) of the difference AKi to the fuzzy set A i is calculated for each characteristic Ki. These membership values can be interpreted as similarities between functions f and g with respect to the chosen characteristics. 6. Finally the vector [sl,s2 . . . . . Sm] of partial similarities is transformed using specific transformations (e.g. y-operator, fuzzy integral, minimum, maximum) into a real number s ( f , g) expressing the overall degree of similarity. To define structural similarity between functions, the following possible characteristic values can be used: 1. Integral. 2. Global minimum, maximum. 3. Position of minima, maxima, zeros, inflection points.
A+Joentgen et al./Fuzzy Sets and Systems 105 (1999) 81-90 2.*
I I,
85
,,,
01, 01,
,i
08 01, 00, +01 .06, +00
9+~ ,l~ ,,to ~,+, ~+l* +~I, +10 +
t+|. +0'+,
(a)
l
~+10 zl
~tl, +,Io
el,
l,+ +
~I
do
,Io
do
I
l
++10 Io *'+
(b) Fig. 4. Idealizedcharacteristicpatterns of time signals for (a) an intact engine; (b) an engine with some defect.
4. Number of minima, maxima, zeros, inflection points. 5. Statistical characteristics. 6. Parameters (if a family of parametric functions is under consideration). 7. Spline parameters (if spline approximation is used). 8. Fourier/Taylor/Wavelet coefficients. 9. Range of function values. 10. Median of function values. 11. Center of gravity. All these characteristic values may be calculated for the original function (trajectory) as well as for any derived function (e.g. derivatives, transformations, etc). Furthermore, characteristics may be defined over the whole domain or just over parts of it (e.g. maximum of the first derivative in the domain 5 < x < 8). The definition of structural similarity as well as the choice of relevant characteristics can be simplified if the class of possible functions (trajectories) is restricted. 3.2. Definition of pointwise similarity between functions Pointwise similarity between functions is concerned with the closeness of functions in the feature space and is based on considering functional values directly (function's characteristics or derived functions are not relevant in this case). The proposed method uses similarity of the difference of two functions to the zero-function as a measure of similarity for a pair of functions. That is, similarity between functions 9(x) and h(x) defined on the universe X is deter-
mined as similarity between the difference function f ( x ) = g(x) - h(x), x EX, and the zero-function: s(g,h) =s(g - h,O)=s(f,O). The following algorithm to determine a measure for pointwise similarity between an arbitrary function f ( x ) and the zero-function is proposed: 1. A fuzzy set A "approximately zero" with a membership function # is defined (Fig. 5a). To emphasize the time focus, the variable x is taken to be time (t). 2. The degree of membership I t ( f (x)) of the function f ( x ) to the fuzzy set A is calculated for each point x E X. These degrees of membership can be interpreted as (pointwise) similarities of the function f ( x ) to the zero-function (Fig. 5b). 3. The function # ( f ( x ) ) is eventually transformed by using specific transformations (e.g. 7-operator, fuzzy integral, minimum, maximum) into a real number s ( f , 0) expressing the overall degree of being zero. All similarity measures obtained with the help of this algorithm are invariant with respect to the addition of a function, i.e. s(y,h ) =s(9 + c,h + c) holds for all functions 9, h and c. On the other hand, every similarity measure satisfying the above equation can be described by defining pointwise similarity between an arbitrary function and the zero function. 3.3. The case o f multi-dimensional functions The two algorithms presented above were formulated to determine structural and pointwise similarity between one-dimensional functions. The extension of these definitions for n-dimensional functions 9(x) and h(x), x E X 1 × )(2 × " " × X,, is straightforward, and
86
A. Joentgen et al./Fuzzy Sets and Systems 105 (1999) 81 90
o.e~v(fft))
~(y)
[~
l
1
o.,,1-,,r,,,,-,..,,,,,.) \_,/, / \,./ .,_, o.21
t
o (a)
(b)
Fig. 5. (a) The fuzzy set "approximately zero" (#(y)), the function f ( t ) and the resulting pointwise similarity #(f(t)); (b) projection of pointwise similarity into the plane (t, p(f(t))).
will be explained based on the algorithm for pointwise similarity. The modification of the algorithm can be performed in two ways: 1. Fuzzy sets A~ "approximately zero" are defined on each subuniverse X,., i = 1.... , n, and the similarity measures, sx,(g, h ), i = 1..... n, are determined according to the described algorithm for projections of functions g(x) and h(x) on subuniverses. The result is the n-dimensional vector of similarities [sx,,sx2,
.... sx,,]. 2. The n-dimensional fuzzy set A "approximately zero" is defined onXl ×X2 ×... ×Xn and the similarity measure sx, ×x2 ×.--xx,, is obtained for n-dimensional functions analogously to the one-dimensional case. For some classification methods it could be desirable to transform the similarity measure into the distance measure using, e.g. the relation
the Euclidean norm:
d(g,h) = ~/ ~
d~.
V i=l,...,n
2. The n-dimensional vector [sxL,sx2,... ,sx,] is transformed by using some transformations (e.g. 7operator, fuzzy integral, minimum, maximum) into an overall similarity s(g, h). Thereafter the distance measure is calculated e.g. by Eq. (3.1). The obtained distance measure between ndimensional functions g and h can be used as a criterion within classical methods for data analysis, allowing the classification of multi-dimensional trajectories. This topic will be discussed in the next section in more detail.
4. Approaches to handle dynamics (trajectories) 1 s(a, h)
d(g,h)- - -
1.
(3.1)
In the first case, when n one-dimensional fuzzy sets are given, the transformation can be performed in two ways: 1. The distance measure is calculated for the components of the n-dimensional vector [sx,, sx2 ..... sx,,] resulting in the vector [dx~, dx: .... , dx,, ] . The latter is then transformed into an overall distance using e.g.
In the following, two different methods for the handling of dynamics within existing methods for data analysis are considered (see point 1 of method related criteria in the taxonomy, Section 2): (a) During preprocessing: feature vectors containing trajectories are preprocessed as to become valid inputs for classical methods such as e.g. fuzzy c-means. (b) Within the data analysis methods: classical methods are modified, so that they can process feature vectors containing trajectories directly.
A. Joentgen et al./Fuzzy Sets and Systems 105 (1999) 81-90
XI
X~
xl x~ x~
I
Xml
2 X ha2
*
•
*
XN
x~
N X InN
Fig. 6. Transformationof a featurevector containingtrajectories into a usual featurevector. Since the modifications of the classical methods do not directly affect the way clusters are built, the resulting methods are basically static. But they are suited to process dynamic objects. Each approach is handled separately in the next two sections. 4.1. Handling o f trajectories during preprocessing
The goal of preprocessing is the preparation and representation of the measured data in order to make the classification possible and improve classification results [3]. In many data analysis tools, methods for preprocessing are integrated [7]. These methods include transformations of data such as calculation of the power spectrum from the time signal, computation of different characteristics or scaling/standardization of the data. Thus, usually preprocessing is performed along with feature selection. The easiest way to integrate dynamic features into existing methods for static data analysis is to transform trajectories into real numbers (characteristic values) and to use the latter instead of the original trajectories, i.e. vector valued features are replaced by one or more real numbers. This leads to conventional feature vectors, which can be processed by classical methods. This idea is illustrated in Fig. 6, where X1,X2 . . . . . XN denote features represented as trajectories or vectors and G(Xj), i-- 1. . . . . Lj, j = 1. . . . . N, is the ith characteristic value for feature j. It should be noted that the number L j, j = 1,..., N, and type of characteristic values can vary for differ-
87
ent features. Since this approach does not require any modifications of the classification methods used, it can very easily be used in conjunction with different methods for data analysis. The following approach requires a modification of the classification methods, but does not use any characteristic values. 4.2. Handling o f trajectories within data analysis methods
In the previous section, the problem of using trajectories is circumvented by reducing each trajectory to a vector of characteristic values. In the following, another approach to handle dynamics is proposed, which is based on similarity between functions. First, some basic remarks related to the notions of distance and similarity are given. Many data analysis methods (e.g. fuzzy c-means [2], possibilistic c-means [6], (fuzzy-) Kohonen networks [8]) use the distance between pairs of feature vectors describing objects as a measure of similarity between these objects. Starting with a distance d(g, h) between objects, a similarity relation can be defined by s(g, h ) = 1/(1 + d(g, h)) [1]. Conversely, each strictly positive similarity relation defines a distance measure d(g,h) = 1/s(g,h) - 1. All data analysis methods mentioned above use nothing else but the distance between objects and class representatives to calculate degrees of membership of objects to classes. The positions of objects in the feature space are used to determine representatives of each class. Therefore, it is sufficient to provide a distance for pairs of objects and/or class representatives to be able to calculate degrees of class membership. These considerations were used to develop a modified version of the fuzzy c-means algorithm, which is called the functional fuzzy c-means (FFCM) and is able to classify dynamic objects (i.e. objects described by trajectories). Since the features are trajectories, the class centers calculated by the FFCM are not just points in the feature space, as in classical fuzzy c-means, but consist themselves of trajectories. This idea is illustrated in Fig. 7, where for the sake of simplicity objects are represented by only one feature. The functional fuzzy c-means algorithm (FFCM) is very similar to the standard fuzzy c-means (FCM). In the following, we present the FFCM and point at the differences to the FCM.
88
A. Joentgen et aL /Fuzzy Sets and Systems 105 (1999) 8 1 - 9 0
object 1
object n
objects
featur!s of objects =
trajectories clustering of trajectories
FFCM ~ , ~
J
1
~
class centers
=trajectories
•..
centerof class 1
center of class c
Fig. 7. Input and output of the functional fuzzy c-means. The problem of finding fuzzy clusters of trajectories in the feature space can be formulated as the minimization of an objective function J(B, U;X) of the form
J(B, U ; X ) =
Z(Itij)md2(xj, bi), i=l j=l
where c is the number of clusters, N the number of objects, m the fuzzifier (weighting exponent),/hi the degree of membership of object j to class i, dZ(xj, bi) the distance between object j and the class center of class i, xj the feature vector describing object j, bi the class center of class i. It should be noted that in the case of the FFCM the components of the feature vector of object xj and of class center bi are trajectories in the feature space. The distance measure proposed in Section 3.2 is used for the calculation of d2(xj, bi ). The algorithm for solving the described problem consists of the following steps: 1. Initialization: Generate values I~ij for i = 1. . . . . c and j = 1. . . . . N such that c Z#ij=l V j = I ..... N. i=1 2. Determination of class centers b i --
~;=l(~ij)mxj N
Remark. The product and the sum are calculated for each component of each trajectory of the feature vectors. 3. Recalculation of membership values #ij: This is the main difference between the FFCM and the FCM. The FFCM calculates the distances dij and dkj using the distance measure proposed in Section 3.2.
m
i = 1..... c.
bi:
k=l \ dkj,I i = 1 ..... c , j = l ..... N. 4. Stopping criterion: There exist many possible stopping criteria. One is to repeat steps 2 - 4 until the changes in the membership values between two iterations are smaller than a fixed threshold. The next section describes an application example and the results, which were obtained using the functional fuzzy c-means algorithm for classification. Another example from the field of automatic fault detection can be found in [5].
4.3. Application The functional fuzzy c-means was applied to a problem from the field of acoustic quality control, in particular to automatic bearing diagnosis. The function p(y) = e -ay2 was used as a pointwise membership function representing the fuzzy set "approximately zero". For the aggregation of pointwise similarities into the overall similarity the y-operator [9] was applied. Antifriction bearings are used in a variety of machines. Their operability is essential for the availability of machines. Therefore, early detection of antifriction bearing defects is a major task within preventive machine diagnosis. One of the most common methods used for diagnostic purposes is based on the analysis of the vibration signal generated by an operating machine. As a rule, damaged machine elements cause additional vibrations, which lead to additional peaks in the frequency spectrum of the structure-bome sound emitted by a machine in operation. In the example at hand, two antifriction bearings were observed and compared, one of them being
A. Joentgen et aL /Fuzzy Sets and Systems 105 (1999) 81-90
89
Fig, 8. Centers of classes (spectra),
intact, the other having an outer raceway error. The structure-borne sound of each bearing was measured 20 times and the corresponding frequency spectra were calculated. The task was to recognize the damaged bearing using nothing but these frequency spectra. This problem is described in more detail in [4]. The test conducted consisted of two steps. In the first step, the spectra were classified using the FFCM. In this case, each bearing was described by one feature, namely the spectrum for this bearing. Therefore, the resulting class centers are also spectra. These class centers are shown in Fig. 8. In the second step, in order to compare the results with the classical fuzzy c-means, three relevant features were extracted from each spectrum using expert knowledge. Then, the three-dimensional feature vectors were classified using the classical fuzzy c-means. In both cases the rate of correct classification was 100%, showing that the functional fuzzy c-means is suitable for the classification of frequency spectra. However, the advantage of using the new method is that the classification can be carried out without any feature selection and therefore without expert knowledge, necessary for the static classification.
idea behind the functional fuzzy c-means is to use a distance measure generated from a similarity measure for trajectories. To evaluate the functional fuzzy c-means, it was applied to the classification of frequency spectra. For the purpose of comparison, frequency spectra were also classified using the static fuzzy c-means with feature selection. Both algorithms provided 100% correct classification. However, using the functional fuzzy c-means, the step of feature selection can be eliminated, reducing the requirements on expert knowledge. This is an advantage for many real world applications, since the selection of relevant features often presents difficulties and must be performed depending on the problem. Further research will show, whether the proposed method is suitable for other types of functions and will be focused on the definition of other similarity measures.
5. Conclusion
References
In this paper, a taxonomy of dynamic data analysis was suggested and illustrated exemplarily. Two approaches to deal with dynamics by existing (static) methods for data analysis were investigated. One possible way to modify classical methods was proposed and applied to the fuzzy c-means algorithm. The main
[1] H. Bandemer, W. N~ither, Fuzzy Data Analysis, Kluwer Academic Publishers, Dordrecht, 1992. [2] J.C. Bezdek, Pattern Recognition with Fuzzy Objective Function Algorithms, Plenum Press, New York, 1981. [3] A. Famili, W.-M. Shen, R. Weber, E. Simoudis, Data preprocessing and intelligent data analysis, Intelligent Data Anal. 1 (1) (1997). http://www.elsevier.com/locate/ida.
Acknowledgements This research was supported by Project DFGZi104/27-1.
90
A. Joentgen et al./ Fuzz)' Sets and Systems 105 (1999) 81-90
[4] M. Fochem, A. Joentgen, B. Geropp, Automatic antifriction bearing diagnosis with fuzzy-logic and neural networks, Proc. of ESIT 1997, Aachen, Germany, 1997, pp. 19-24. [5] A. Joentgen, L. Mikenina, R. Weber, A. Zeugner, H.-J. Zimmermann, Automatic fault detection in gearboxes by dynamic fuzzy data analysis, Fuzzy Sets and Systems 105 (1999) 123-132.
[6] R. Krishnapuram, J.M. Keller, A possibilistic approach to clustering, IEEE Trans. Fuzzy Syst. 1 (2) (1993) 98-I10. [7] MIT, Data Engine 2.1 Manual, Aachen, 1997. [8] D.E. Rumelhart, J.L. McClelland, Parallel Distributed Processing, 8th. ed., MIT Press, Cambridge, MA, 1988. [9] H.-J. Zimmermann, Fuzzy Set Theory - and its Applications, 3rd ed., Kluwer Academie Publishers, Dordrecht, 1996.
"
t
FUZZY
sets and systems ELSEVIER
Fuzzy Sets and Systems 105 (1999) 81 90
Dynamic fuzzy data analysis based on similarity between functions A. Joentgen a,., L. Mikenina a, R. Weber b, H.-J. Zimmermann a a
Lehrstuhlfiir Unternehmensforsehung, R WTH Aachen, Templeroraben 64, 52062 Aachen, Germany b MIT GmbH, Promenade 9, 52076 Aachen, Germany
Received April 1998; revised July 1998
Abstract
In data analysis, objects are usually represented by feature vectors, each describing a state of an object at a point of time. Most methods for data analysis use only these feature vectors and do not take into account changes over time. They can therefore be called static, But often a "dynamic" approach, which utilizes the feature changes over time, seems to be more appropriate (e.g. supervision of patients in medical care, state-dependent maintenance of machines, classification of shares). In this paper, different criteria for structuring the field of "dynamic data analysis (DDA)" are proposed and one of the relevant approaches is investigated in more detail. This approach considers possible ways to handle dynamics within static methods for data analysis. In doing this, different types of similarity measures for trajectories are defined, which can be used to modify static methods for data analysis. One of the proposed similarity measures has been integrated into the fuzzy c-means. An application example is used to demonstrate the applicability of the modified fuzzy c-means. (~) 1999 Elsevier Science B.V. All rights reserved. Keywords: Cluster analysis; Pattern recognition; Dynamic data analysis; Acoustic quality control
I. M o t i v a t i o n
Classification o f objects using known features is one o f the most frequent tasks in data analysis [1,2,6,9]. Usually, objects are represented by feature vectors, which often consist o f numbers, describing the object at a certain moment. The development o f objects over time (and therefore the development o f the features) is not considered explicitly or is taken into account by just using single values o f the past in the feature vector. Methods that use this type o f feature vectors can be called static. In many applications, however, explicit * Corresponding author.
consideration o f trajectories rather than single points is desirable, e.g.: • monitoring o f patients in medicine, e.g. during narcosis, where the development of the patients' condition is essential; • state-dependent machine maintenance; • rating o f shares: the examination o f the development o f share prices and other characteristics allows better estimates than just considering the current share price. In all cases, where a dynamic viewpoint is desirable, the momentary snapshot for some components o f the feature vector may be replaced by a trajectory o f this feature. Thus, dynamic objects are represented by multi-dimensional trajectories in the feature
0165-0114/99/$ - see front matter (~) 1999 Elsevier Science B.V. All rights reserved. PII: S0165-0114(98)00337-6
82
A. Joentgen et aL /Fuzzy Sets and Systems 105 (1999) 81-90
40 30
40
feature 1 +
+
feature 1
30
4"
20
20
.0,
10.
10
i'x_.-.,__.
; j,f
feature 2 o" "" "ib' ' ' ~,b' "' "3U" "' ~ " "
"eb'" "'eb
(a)
J" "--"---,-. feature 2
o"'" 'ib'"""2o"""W ' " ;~'" "~d" ~b (b)
Fig. 1. (a) States of objects at a point of time; (b) projectionsof trajectories over time into the feature space. space. Since most methods for data analysis are not suited to classify objects described by trajectories, new methods for dynamic data analysis have to be developed. Fig. 1 illustrates the difference between classical (static) and dynamic data analysis. Consider a twodimensional feature space with one additional time dimension and suppose that a set of objects is observed over time. States of objects at a point of time can be seen in the cut of this three-dimensional space at the current moment (Fig. la). Two classes of objects can easily be distinguished in this plane. However, if the trace of each object from the initial to its current state (i.e. its trajectory) is considered and projected into the feature space (Fig. lb), other classes may seem more reasonable. This paper is structured as follows: In Section 2, a taxonomy for the classification of different topics related to dynamic data analysis is proposed and illustrated with some examples. Section 3 describes two types of similarity between trajectories, which could be relevant for different problems, and presents methods to determine the similarity measure in both cases. In Section 4, two approaches are proposed, which allow the classification of dynamic objects by static methods. The modification of classical methods, in particular the fuzzy c-means algorithm, is illustrated by an application example.
2. Taxonomy for dynamic data analysis In the following, different criteria are listed, which are suitable to structure the field of dynamic data
analysis. The criteria themselves are divided into two groups: problem-related criteria and methodrelated criteria. Problem related criteria depend on the structure of the observed system and the desired results. Method-related criteria refer to methods and techniques, which are used in classifier design and classification. Essentially they include selection and processing of relevant features. The field of dynamic data analysis can be structured using the following Taxonomy: Problem-related criteria:
1. Classification object: (a) classification of several systems, (b) classification of states of one system. 2. Type of class structure: (a) static class structure, (b) dynamic class structure. 3. Observation period: (a) point of time, (b) rolling horizon. Method-related criteria:
1. Handling of dynamics (trajectories): (a) during preprocessing, (b) within data analysis methods. 2. Type of preprocessing of the trajectories: (a) directly measured trajectories (no preprocessing), (b) aggregated trajectories, (c) ratio between trajectories. 3. Time dependence of the feature set: (a) constant feature set, (b) variable feature set. Table 1 shows some examples of combinations of problem-related criteria.
83
A. Joentgen et al./Fuzzy Sets and Systems 105 (1999) 81-90 Table 1 Examples of combinations of problem related criteria Condition rating for a known machine
lb, 2a, 3a
Preventive inspection/state dependent maintenance of a known machine
l b, 2a, 3b
Monitoring and classifier adaptation for an unknown machine, machine installation
lb, 2b, 3b
Comparison of machines using a given class structure, quality control
l a, 2a, 3a
Continuous parallel monitoring of machines
l a, 2a, 3b
Determination and dynamic adaptation of the state structure of a group of unknown machines
l a, 2b, 3b
natural (depending on the chosen type o f structural similarity). The following two sections describe these two types o f similarity and the relationships between them.
35
253° is
i Y
i
10
>.ffl
s 0
i ! ix,
0
20
40
60
80
100 120 140 160 180 200
Fig. 2. Structural and pointwise similarity.
3. Structural and pointwise similarity As stated before, the components o f feature vectors describing dynamic objects are trajectories. Starting from the fact that most methods for data analysis use a distance measure or a similarity measure as a criteflon to classify objects, one way to handle dynamic objects is to define the similarity measure for trajectories (functions) and to use it within existing or perhaps completely new methods. Similarity of trajectories can be defined in different ways. Basically, two viewpoints can be distinguished. The more similar two trajectories are • the better they match in form/evolution/characteristics (structural similarity). • the smaller their (pointwise) distance in feature space is (pointwise similarity). Fig. 2 gives an example o f the differences between structural and pointwise similarity. In terms of pointwise similarity A and B would be grouped together as well as C and D. But in terms o f structural similarity the grouping {A, D} and {B, C} seems to be more
3.1. Definition of structural similarity between functions Structural similarity relates to a variety o f aspects of the trajectories (functions) under consideration: form, evolution, size or orientation (of trajectories in ~R") are some examples. Depending on the chosen aspect, different criteria may be relevant to describe similarity, e.g. slope, curvature, position and values of extremal points or other information like smoothness or monotonicity (as a degree o f membership of a trajectory to the set of monotone functions). Here some examples of structural similarity are given for illustration: (A) Slope and curvature of trajectories are relevant, but their position in the feature space is not relevant: The functions y = x and y -- 1.001 • x + 100 are similar (both describe straight lines with approximately equal slope and a curvature of zero), whereas y = x and y = x + 0.001 * sin(x) are not similar (despite the fact that they are much closer in terms of Euclidean distance). This type o f definition o f similarity can be applied to classify, e.g., shares as "decreasing" (A, E, H), 'fincreasing" (B, D, G, J, K), "constant" (F, I ) or "fluctuating" (C), depending on the trajectories of their share prices (Fig. 3). (B) Form of trajectories is relevant, but their size and position in the feature space are not relevant: The unit circle (center at (0, 0) and radius 1 ) and the circle with center at (100,0) and radius 17.4 are similar to
84
A. Joentgen et aL /Fuzzy Sets and Systems 105 (1999) 81-90 80 70
60
B
50 o o ....
.~ 40 t~
3O
m m
G H 20 ~
m
m
m
m
~
10
Time
Fig. 3. Fictitious developments of share prices.
degree 1, whereas the unit circle and the unit square are much less similar. This type of definition may be applied to classify engines, using the airborne sound they emit during operation: amplitude and position of characteristic patterns change with speed, independent of the state of the engine. However, the characteristic pattern remains the same depending on the fact, whether an engine is intact or damaged (Fig. 4). In some cases, structural similarity can be reduced to pointwise similarity, for instance in the first example, by considering the pointwise similarity for the first and the second derivatives of the functions, respectively. One method to define structural similarity between functions is to consider relevant characteristics of these functions (e.g. integral, extrema), which contain the information about the specific structure of functions. The following algorithm to determine a measure for structural similarity between arbitrary functions f and g is proposed: 1. A set of relevant characteristics Ki, i = 1,... ,m, describing structural similarity is chosen.
2. A fuzzy set Ai labeled "admissible difference for characteristic Ki" with membership function ]2i is defined. 3. All characteristics K i ( f ) for the function f and Ki(g) for the function g are calculated. 4. For each characteristic Ki the difference A K i = I K i ( f ) - Ki(g)], i = 1. . . . . m, is calculated. 5. The degree of membership si --/~i(z3K/) of the difference AKi to the fuzzy set A i is calculated for each characteristic Ki. These membership values can be interpreted as similarities between functions f and g with respect to the chosen characteristics. 6. Finally the vector [sl,s2 . . . . . Sm] of partial similarities is transformed using specific transformations (e.g. y-operator, fuzzy integral, minimum, maximum) into a real number s ( f , g) expressing the overall degree of similarity. To define structural similarity between functions, the following possible characteristic values can be used: 1. Integral. 2. Global minimum, maximum. 3. Position of minima, maxima, zeros, inflection points.
A+Joentgen et al./Fuzzy Sets and Systems 105 (1999) 81-90 2.*
I I,
85
,,,
01, 01,
,i
08 01, 00, +01 .06, +00
9+~ ,l~ ,,to ~,+, ~+l* +~I, +10 +
t+|. +0'+,
(a)
l
~+10 zl
~tl, +,Io
el,
l,+ +
~I
do
,Io
do
I
l
++10 Io *'+
(b) Fig. 4. Idealizedcharacteristicpatterns of time signals for (a) an intact engine; (b) an engine with some defect.
4. Number of minima, maxima, zeros, inflection points. 5. Statistical characteristics. 6. Parameters (if a family of parametric functions is under consideration). 7. Spline parameters (if spline approximation is used). 8. Fourier/Taylor/Wavelet coefficients. 9. Range of function values. 10. Median of function values. 11. Center of gravity. All these characteristic values may be calculated for the original function (trajectory) as well as for any derived function (e.g. derivatives, transformations, etc). Furthermore, characteristics may be defined over the whole domain or just over parts of it (e.g. maximum of the first derivative in the domain 5 < x < 8). The definition of structural similarity as well as the choice of relevant characteristics can be simplified if the class of possible functions (trajectories) is restricted. 3.2. Definition of pointwise similarity between functions Pointwise similarity between functions is concerned with the closeness of functions in the feature space and is based on considering functional values directly (function's characteristics or derived functions are not relevant in this case). The proposed method uses similarity of the difference of two functions to the zero-function as a measure of similarity for a pair of functions. That is, similarity between functions 9(x) and h(x) defined on the universe X is deter-
mined as similarity between the difference function f ( x ) = g(x) - h(x), x EX, and the zero-function: s(g,h) =s(g - h,O)=s(f,O). The following algorithm to determine a measure for pointwise similarity between an arbitrary function f ( x ) and the zero-function is proposed: 1. A fuzzy set A "approximately zero" with a membership function # is defined (Fig. 5a). To emphasize the time focus, the variable x is taken to be time (t). 2. The degree of membership I t ( f (x)) of the function f ( x ) to the fuzzy set A is calculated for each point x E X. These degrees of membership can be interpreted as (pointwise) similarities of the function f ( x ) to the zero-function (Fig. 5b). 3. The function # ( f ( x ) ) is eventually transformed by using specific transformations (e.g. 7-operator, fuzzy integral, minimum, maximum) into a real number s ( f , 0) expressing the overall degree of being zero. All similarity measures obtained with the help of this algorithm are invariant with respect to the addition of a function, i.e. s(y,h ) =s(9 + c,h + c) holds for all functions 9, h and c. On the other hand, every similarity measure satisfying the above equation can be described by defining pointwise similarity between an arbitrary function and the zero function. 3.3. The case o f multi-dimensional functions The two algorithms presented above were formulated to determine structural and pointwise similarity between one-dimensional functions. The extension of these definitions for n-dimensional functions 9(x) and h(x), x E X 1 × )(2 × " " × X,, is straightforward, and
86
A. Joentgen et al./Fuzzy Sets and Systems 105 (1999) 81 90
o.e~v(fft))
~(y)
[~
l
1
o.,,1-,,r,,,,-,..,,,,,.) \_,/, / \,./ .,_, o.21
t
o (a)
(b)
Fig. 5. (a) The fuzzy set "approximately zero" (#(y)), the function f ( t ) and the resulting pointwise similarity #(f(t)); (b) projection of pointwise similarity into the plane (t, p(f(t))).
will be explained based on the algorithm for pointwise similarity. The modification of the algorithm can be performed in two ways: 1. Fuzzy sets A~ "approximately zero" are defined on each subuniverse X,., i = 1.... , n, and the similarity measures, sx,(g, h ), i = 1..... n, are determined according to the described algorithm for projections of functions g(x) and h(x) on subuniverses. The result is the n-dimensional vector of similarities [sx,,sx2,
.... sx,,]. 2. The n-dimensional fuzzy set A "approximately zero" is defined onXl ×X2 ×... ×Xn and the similarity measure sx, ×x2 ×.--xx,, is obtained for n-dimensional functions analogously to the one-dimensional case. For some classification methods it could be desirable to transform the similarity measure into the distance measure using, e.g. the relation
the Euclidean norm:
d(g,h) = ~/ ~
d~.
V i=l,...,n
2. The n-dimensional vector [sxL,sx2,... ,sx,] is transformed by using some transformations (e.g. 7operator, fuzzy integral, minimum, maximum) into an overall similarity s(g, h). Thereafter the distance measure is calculated e.g. by Eq. (3.1). The obtained distance measure between ndimensional functions g and h can be used as a criterion within classical methods for data analysis, allowing the classification of multi-dimensional trajectories. This topic will be discussed in the next section in more detail.
4. Approaches to handle dynamics (trajectories) 1 s(a, h)
d(g,h)- - -
1.
(3.1)
In the first case, when n one-dimensional fuzzy sets are given, the transformation can be performed in two ways: 1. The distance measure is calculated for the components of the n-dimensional vector [sx,, sx2 ..... sx,,] resulting in the vector [dx~, dx: .... , dx,, ] . The latter is then transformed into an overall distance using e.g.
In the following, two different methods for the handling of dynamics within existing methods for data analysis are considered (see point 1 of method related criteria in the taxonomy, Section 2): (a) During preprocessing: feature vectors containing trajectories are preprocessed as to become valid inputs for classical methods such as e.g. fuzzy c-means. (b) Within the data analysis methods: classical methods are modified, so that they can process feature vectors containing trajectories directly.
A. Joentgen et al./Fuzzy Sets and Systems 105 (1999) 81-90
XI
X~
xl x~ x~
I
Xml
2 X ha2
*
•
*
XN
x~
N X InN
Fig. 6. Transformationof a featurevector containingtrajectories into a usual featurevector. Since the modifications of the classical methods do not directly affect the way clusters are built, the resulting methods are basically static. But they are suited to process dynamic objects. Each approach is handled separately in the next two sections. 4.1. Handling o f trajectories during preprocessing
The goal of preprocessing is the preparation and representation of the measured data in order to make the classification possible and improve classification results [3]. In many data analysis tools, methods for preprocessing are integrated [7]. These methods include transformations of data such as calculation of the power spectrum from the time signal, computation of different characteristics or scaling/standardization of the data. Thus, usually preprocessing is performed along with feature selection. The easiest way to integrate dynamic features into existing methods for static data analysis is to transform trajectories into real numbers (characteristic values) and to use the latter instead of the original trajectories, i.e. vector valued features are replaced by one or more real numbers. This leads to conventional feature vectors, which can be processed by classical methods. This idea is illustrated in Fig. 6, where X1,X2 . . . . . XN denote features represented as trajectories or vectors and G(Xj), i-- 1. . . . . Lj, j = 1. . . . . N, is the ith characteristic value for feature j. It should be noted that the number L j, j = 1,..., N, and type of characteristic values can vary for differ-
87
ent features. Since this approach does not require any modifications of the classification methods used, it can very easily be used in conjunction with different methods for data analysis. The following approach requires a modification of the classification methods, but does not use any characteristic values. 4.2. Handling o f trajectories within data analysis methods
In the previous section, the problem of using trajectories is circumvented by reducing each trajectory to a vector of characteristic values. In the following, another approach to handle dynamics is proposed, which is based on similarity between functions. First, some basic remarks related to the notions of distance and similarity are given. Many data analysis methods (e.g. fuzzy c-means [2], possibilistic c-means [6], (fuzzy-) Kohonen networks [8]) use the distance between pairs of feature vectors describing objects as a measure of similarity between these objects. Starting with a distance d(g, h) between objects, a similarity relation can be defined by s(g, h ) = 1/(1 + d(g, h)) [1]. Conversely, each strictly positive similarity relation defines a distance measure d(g,h) = 1/s(g,h) - 1. All data analysis methods mentioned above use nothing else but the distance between objects and class representatives to calculate degrees of membership of objects to classes. The positions of objects in the feature space are used to determine representatives of each class. Therefore, it is sufficient to provide a distance for pairs of objects and/or class representatives to be able to calculate degrees of class membership. These considerations were used to develop a modified version of the fuzzy c-means algorithm, which is called the functional fuzzy c-means (FFCM) and is able to classify dynamic objects (i.e. objects described by trajectories). Since the features are trajectories, the class centers calculated by the FFCM are not just points in the feature space, as in classical fuzzy c-means, but consist themselves of trajectories. This idea is illustrated in Fig. 7, where for the sake of simplicity objects are represented by only one feature. The functional fuzzy c-means algorithm (FFCM) is very similar to the standard fuzzy c-means (FCM). In the following, we present the FFCM and point at the differences to the FCM.
88
A. Joentgen et aL /Fuzzy Sets and Systems 105 (1999) 8 1 - 9 0
object 1
object n
objects
featur!s of objects =
trajectories clustering of trajectories
FFCM ~ , ~
J
1
~
class centers
=trajectories
•..
centerof class 1
center of class c
Fig. 7. Input and output of the functional fuzzy c-means. The problem of finding fuzzy clusters of trajectories in the feature space can be formulated as the minimization of an objective function J(B, U;X) of the form
J(B, U ; X ) =
Z(Itij)md2(xj, bi), i=l j=l
where c is the number of clusters, N the number of objects, m the fuzzifier (weighting exponent),/hi the degree of membership of object j to class i, dZ(xj, bi) the distance between object j and the class center of class i, xj the feature vector describing object j, bi the class center of class i. It should be noted that in the case of the FFCM the components of the feature vector of object xj and of class center bi are trajectories in the feature space. The distance measure proposed in Section 3.2 is used for the calculation of d2(xj, bi ). The algorithm for solving the described problem consists of the following steps: 1. Initialization: Generate values I~ij for i = 1. . . . . c and j = 1. . . . . N such that c Z#ij=l V j = I ..... N. i=1 2. Determination of class centers b i --
~;=l(~ij)mxj N
Remark. The product and the sum are calculated for each component of each trajectory of the feature vectors. 3. Recalculation of membership values #ij: This is the main difference between the FFCM and the FCM. The FFCM calculates the distances dij and dkj using the distance measure proposed in Section 3.2.
m
i = 1..... c.
bi:
k=l \ dkj,I i = 1 ..... c , j = l ..... N. 4. Stopping criterion: There exist many possible stopping criteria. One is to repeat steps 2 - 4 until the changes in the membership values between two iterations are smaller than a fixed threshold. The next section describes an application example and the results, which were obtained using the functional fuzzy c-means algorithm for classification. Another example from the field of automatic fault detection can be found in [5].
4.3. Application The functional fuzzy c-means was applied to a problem from the field of acoustic quality control, in particular to automatic bearing diagnosis. The function p(y) = e -ay2 was used as a pointwise membership function representing the fuzzy set "approximately zero". For the aggregation of pointwise similarities into the overall similarity the y-operator [9] was applied. Antifriction bearings are used in a variety of machines. Their operability is essential for the availability of machines. Therefore, early detection of antifriction bearing defects is a major task within preventive machine diagnosis. One of the most common methods used for diagnostic purposes is based on the analysis of the vibration signal generated by an operating machine. As a rule, damaged machine elements cause additional vibrations, which lead to additional peaks in the frequency spectrum of the structure-bome sound emitted by a machine in operation. In the example at hand, two antifriction bearings were observed and compared, one of them being
A. Joentgen et aL /Fuzzy Sets and Systems 105 (1999) 81-90
89
Fig, 8. Centers of classes (spectra),
intact, the other having an outer raceway error. The structure-borne sound of each bearing was measured 20 times and the corresponding frequency spectra were calculated. The task was to recognize the damaged bearing using nothing but these frequency spectra. This problem is described in more detail in [4]. The test conducted consisted of two steps. In the first step, the spectra were classified using the FFCM. In this case, each bearing was described by one feature, namely the spectrum for this bearing. Therefore, the resulting class centers are also spectra. These class centers are shown in Fig. 8. In the second step, in order to compare the results with the classical fuzzy c-means, three relevant features were extracted from each spectrum using expert knowledge. Then, the three-dimensional feature vectors were classified using the classical fuzzy c-means. In both cases the rate of correct classification was 100%, showing that the functional fuzzy c-means is suitable for the classification of frequency spectra. However, the advantage of using the new method is that the classification can be carried out without any feature selection and therefore without expert knowledge, necessary for the static classification.
idea behind the functional fuzzy c-means is to use a distance measure generated from a similarity measure for trajectories. To evaluate the functional fuzzy c-means, it was applied to the classification of frequency spectra. For the purpose of comparison, frequency spectra were also classified using the static fuzzy c-means with feature selection. Both algorithms provided 100% correct classification. However, using the functional fuzzy c-means, the step of feature selection can be eliminated, reducing the requirements on expert knowledge. This is an advantage for many real world applications, since the selection of relevant features often presents difficulties and must be performed depending on the problem. Further research will show, whether the proposed method is suitable for other types of functions and will be focused on the definition of other similarity measures.
5. Conclusion
References
In this paper, a taxonomy of dynamic data analysis was suggested and illustrated exemplarily. Two approaches to deal with dynamics by existing (static) methods for data analysis were investigated. One possible way to modify classical methods was proposed and applied to the fuzzy c-means algorithm. The main
[1] H. Bandemer, W. N~ither, Fuzzy Data Analysis, Kluwer Academic Publishers, Dordrecht, 1992. [2] J.C. Bezdek, Pattern Recognition with Fuzzy Objective Function Algorithms, Plenum Press, New York, 1981. [3] A. Famili, W.-M. Shen, R. Weber, E. Simoudis, Data preprocessing and intelligent data analysis, Intelligent Data Anal. 1 (1) (1997). http://www.elsevier.com/locate/ida.
Acknowledgements This research was supported by Project DFGZi104/27-1.
90
A. Joentgen et al./ Fuzz)' Sets and Systems 105 (1999) 81-90
[4] M. Fochem, A. Joentgen, B. Geropp, Automatic antifriction bearing diagnosis with fuzzy-logic and neural networks, Proc. of ESIT 1997, Aachen, Germany, 1997, pp. 19-24. [5] A. Joentgen, L. Mikenina, R. Weber, A. Zeugner, H.-J. Zimmermann, Automatic fault detection in gearboxes by dynamic fuzzy data analysis, Fuzzy Sets and Systems 105 (1999) 123-132.
[6] R. Krishnapuram, J.M. Keller, A possibilistic approach to clustering, IEEE Trans. Fuzzy Syst. 1 (2) (1993) 98-I10. [7] MIT, Data Engine 2.1 Manual, Aachen, 1997. [8] D.E. Rumelhart, J.L. McClelland, Parallel Distributed Processing, 8th. ed., MIT Press, Cambridge, MA, 1988. [9] H.-J. Zimmermann, Fuzzy Set Theory - and its Applications, 3rd ed., Kluwer Academie Publishers, Dordrecht, 1996.