- Email: [email protected]
Multivariate statistical methods for monitoring continuous processes: assessment of discrimination power of disturbance models and diagnosis of multiple disturbances
Chemometrics and intelligent laboratory systems
ELSEVIER
Chemometrics and Intelligent Laboratory Systems 30 (1995) 37-48
Multivariate statistical methods for monitoring continuous processes: assessment of discrimination power of disturbance models and diagnosis of multiple disturbances A.C. Raich, A. Cinar
*
Chemical Engineering Department, Illinois Institute of Technology, Chicago, IL 60616, USA
Received 21 December 1994; accepted 31 March 1995
Abstract A new methodology was reported [1,2] for integrated use of principal components analysis (PCA) and discriminant analysis in order to determine out-of-control status of a continuous process and to diagnose the source causes for abnormal behav-
ior. Most of the disturbances were identified with good rates of success, with a higher success rate for step or ramp type of disturbances. Quantitative tools that evaluate overlap and similarity between high-dimensional PCA models are proposed in this communication, and their implications on determining the discrimination power of PCA models of processes operating under disturbances are discussed. Diagnosis of several disturbances occurring simultaneously is also investigated. The criterion developed provide upper limits of discrimination power of various single and multiple process disturbances. The techniques developed are illustrated by assessing the process described by the Tennessee Eastman Control Challenge problem [3]. Keywords: Pattern recognition; Discriminant analysis; Fault diagnosis; Principal components analysis
1. Introduction Statistical process control (SPC) has gained importance in chemical process industries as a means to improve quality and yield, and to reduce costs. Conventional univariate SPC methods do not make the best use of data available, and often yield misleading results on multivariate chemical processes. The use of several univariate charts simultaneously to detect the out-of-control status of a multivariable process does
* Corresponding author, email: [email protected] 0169-7439/95/$09.50
not have a theoretical foundation and yields large Type I and Type II errors. Hotelling’s T2 charts provide reliable and correct tools for detecting that a multivariable process has gone out-of-control, but they cannot provide clues about the process variables that signaled the out-of-control behavior and the source cause(s) for the abnormal behavior. Chemometric methods and principal components analysis (PCA) have been utilized for merging detection of out-of-control status [4], identification of the process variables with significant variations [5], and diagnosis of source causes [1,2,4,6]. For small-dimensional systems which can be represented adequately by a
0 1995 Elsevier Science B.V. All rights reserved
SSDI 0169-7439(95)00035-6
38
A.C. Raich, A. Cinar/Chemometrics
and Intelligent Laboratory Systems 30 (1995) 37-48
few principal components, graphical techniques have been proposed [4]. In our previous work, novel methods using PCA and discriminant analysis were reported for monitoring of multivariable continuous processes. These statistical methods have integrated detection and diagnosis into an automated, quantitative analysis framework [1,2]. Process monitoring centers around a PCA model built with data collected at nominal operating conditions. When an outof-control situation is detected, the observation is compared to PCA models for known disturbances. Using refinements of statistical distances, discriminant analysis then selects the most likely cause(s) for the current out-of-control condition. Successful diagnosis depends on the discrimination potential of these disturbance models, as high similarity between models would limit the potential to distinguish between the corresponding disturbances. Other questions can be posed about the usefulness of particular PCA models when applied to continuous processes. Comparison of models built from data at different operating points, non-stationary conditions, or even different sampling frequencies may be of interest for monitoring process performance. In diagnosis of source causes for disturbances, a basis for comparison of models can help identify misdiagnosis patterns. Attempts to diagnose multiple faults further highlights the need for a deeper understanding of the similarities between models for contributing disturbances and masking of secondary faults. Also, for a process that has a moving target, such as a batch process, it may be useful to have a statistical basis to identify changes in the process model. In disturbance diagnosis, where there are many potential models, it is useful to have a quantitative measure of similarity or overlap between models, and to identify the overall likelihood of successful diagnosis.
2. Assessment of model similarity by overlap of
covariance Eigenvalue-based techniques such as PCA describe a set of data with two basic entities: a central point and directions of spread around the center. In statistical terms, these correspond to the mean and covariance, which completely characterize a multivariate normal probability distribution.
In comparing multivariate models, much work has been done on testing of significant differences between means with the same covariance. It is much more difficult to test for differences in covariance, especially when there is a difference in the means. However, especially in disturbance diagnosis, it is the difference in covariance which can be most crucial; diagnosis can be successfully done, whether or not means are different, as long as there is a difference in covariance [7]. Testing for eigenvalue models of covariance adds new complications, as the statistical characteristics are not well known, even for the most common distributions. Simplifying assumptions for special cases can be made, with significant loss of generality [8]. Using geometry concepts rather than statistics with known distributions, the angles between the principal components of different models can be utilized to develop a useful criteria. 2.1. Angles between coordinate directions A general treatment of between-groups comparisons of principal components [9] describes the derivation of angles between coordinate axes from different models. Using eigenvalue decomposition, the angles (Ye between PCA coordinate direction sets P, and Pz correspond to eigenvalues s:12: P; P, P; P, = L’SL
(1)
(Yk= cos-y
(2)
$2)
where L gives a set of consensus coordinates and sk is the kth largest eigenvalue in S, corresponding to the smallest angle CQ between the k-dimensional coordinate subspaces [9]. Angles between covariance coordinate systems are visually interpretable in 2 or 3 dimensions, and computationally extend to higher dimensional systems in a straightforward manner. An ideal statistic would describe conical confidence thresholds for angles between coordinate axes of different models. However, such a statistical analysis is currently prohibitively complex beyond 2 or 3 dimensions [lo]. Instead, possible simple geometric benchmarks include the minimum angle between models and the sum of squares of the cosines of angles between model axes. For high-dimensional models, the minimum angle between models can quickly become trivial. It is generally recognized that
A.C. Raich, A. Qnar/Chemometrics
and Intelligent Laboratory Systems 30 (1995) 37-48
considerable similarity between models is often found within the first few dimensions and that successful discrimination generally requires consideration of higher dimensions [7]. This necessitates the use of higher dimension models and can make the minimum angle as small as computational inaccuracy allows. 2.2. Similarity factor The sum of cosines of angles between model axes can be used to define a fraction of similarity between models. First describe exact similarity between two models. Models of data with a variance of one for each of the same number of independent variables would correspond to coordinate axes P, and P, both equal to the identity matrix. The eigenvector decomposition described earlier would result in a set of eigenvalues {sJ all equal to one with corresponding angles all equal to zero. The sum of squares of cosines would then be equal to the number of dimensions considered (p). A general measure of similarity f is then defined as the sum of squares of cosines divided by the number of dimensions, p: f=
;k$lcos2(.,) =1 i
Sk
P k=l
The similarity factor f ranges from zero (lack of similarity) to one (exact similarity). As a description of overall geometric similarity in spread, the similarity factor f provides a quantitative measure of difference in covariance directions between models. The similarity factor was used to assess the differences among various disturbance models. For example, classification of disturbances to the Tennessee Eastman Industrial Challenge Problem was observed to correspond to similarity factors greater than 0.6 for 85% of the observations. While a high similarity factor does not guarantee misclassifications, low similarity between models generally indicated a low probability for misclassifications. A low similarity factor value corresponds to fewer misclassifications. Besides providing comparisons between models of disturbances, the similarity factor could provide a way to check if PCA models for different stages of a batch process change their orientation around a moving mean (in time).
39
3. Assessment of model similarity by overlap of means Another statistical test for comparing multivariate models is based on differences in means. Geometrically, this corresponds to comparison of origin of coordinates rather than the coordinate directions that were used in testing the overlap of covariance. Many statistical tests have been developed for testing means, but these can become numerically unstable when significant correlation occurs. Working around the instability issue, the interest is to look at overlap between eigenvalue-based models. In a technique described to identify chemical species from chemometric data of multicomponent mixtures, target factor analysis sets a confidence level on the presence of contributing features [ll]. In a broader application, the method can place a likelihood on whether or not a suspected vector is a contributor to the model of a multivariate data set. The statistically and numerically viable approach [ll] is based on the distribution of quadratic forms, such as Mahalanobis weighing. A statistic can be defined to test if a specified vector is contributing to the description by an eigenvalue-based model, that is, if the vector is significantly inside the confidence region containing the data. With respect to overlap of means, the test can be applied to determine whether the mean from one model, pi, significantly overlaps the region of data from another model. This involves estimating p1 with respect to the other model, which is centered around p2. In this case, the test statistic is
with t=(pI-p2)Pandr=tP’-p1
(5)
where p is the fraction of variation in the data accounted for by the model, t is the approximation of pi by the model, P is the eigenvector model (directions) transforming p measurements to k dimensions, 2 is the covariance of the k new variables, pi is the mean of measurements for disturbance tested, pa is the mean of n modeled observations, and r is the residual error in pi unexplained by the model. The test indicates significant overlap of the means
A.C. Raich, A. @tar/Chemometrics
40
and Intelligent Laboratory Systems 30 (1995) 37-48
pr with the model around p2 when the statistic m is smaller than a threshold m, from the F distribution: m, = Fol;(p-
k),(n-
k)
natorial counting techniques used in the binomial and hypergeometric distributions can estimate the overall similarity or overlap between groups as:
(6)
d = (total number of overlaps between means and other groups)
In chemometric applications, a confidence level of 95% has been satisfactorily used. In general use, combined with the similarity factor technique, mean overlap analysis could be useful to test if an existing PCA model fits a new set of observations or if two PCA models are analogous. Used by itself, mean overlap was not as successful an indicator of pairwise misclassification as the similarity factor.
/l&g-111
(7)
The count of overlaps is done using the F test described in (6) and g is the number of groups under consideration. Similarly, an estimate of the best success rate between several groups can be simply stated as a percentile: 1 = lOO( 1 - d)
(8)
4. Overlap between many groups
5. Diagnosis of multiple disturbances A crucial consideration in formulating the rules for discriminating between multiple groups is the overall similarity between them. If the probability density functions of each group were exactly known, an upper bound for success in classification could be formulated [7]. Different rules could be devised and evaluated to balance their approach to this minimum error rate. The increase in complexity of the rules may not pay off in terms of reduction in the error rate. In real world situations where the density functions are not known exactly, such a rigorous evaluation is not feasible, but it is still useful to approximate the best success rate for correct discrimination of disturbances. This analysis is generally related to a Mahalanobis-type distance between groups, with increasing distance corresponding to greater success in classification. The difficulties related to Mahalanobis inversion with correlated data, especially with different covariance structures, makes characterization of separation between groups difficult. Only some special cases with a few groups, a few variables, or special characteristics can be handled directly [8]. An additional important consideration is over-fitting of the model to data used in training. When too complex a model is used, inflated success rates may be found, yet the use of the model with new data will not be as successful. This effect, which is generally attributed to the inclusion of noise, is countered in practice by evaluating the rate of success with new data, often referred to as cross-validation. Rather than a detailed statistical approach, a general guideline reminiscent of the form of the combi-
Diagnosis of a source cause for an out-of-control condition has been discussed extensively using a variety of distance-based methods. However, such methods were designed for use with single disturbances. Their adaptability to handle combinations of disturbances is also of interest. Quantitative measures of overlap between singledisturbance models can be generalized to analyze diagnosis accuracy of multiple disturbances. If there were no overlap, two alternative schemes might handle multiple disturbances within an existing parallel discriminant framework. Similar to a fuzzy logic approach, idealizing the combination of disturbances as being located between the regions of the underlying component disturbances, allocations of membership to the different independent disturbances contributing to the combination might provide diagnosis of underlying disturbances. More generally, a successful extension of the discrimination scheme might also be done via the introduction of new models for each combination of interest; this approach could describe combinations of disturbances that produce models that are not simply the consensus of component models. Using the measures of similarity in model center and direction of spread described earlier, the hypothesis of independence can be checked. Besides describing misclassifications between single disturbances, similarity measures can give insight into treatment of multiple disturbances, including masking of some conditions.
A.C. Raich, A. @zar/
Chemometrics
and Intelligent Laboratory Systems 30 (1995) 37-48
5.1. Masking of multiple disturbances It is realistic to expect considerable similarity between models for the same process. Besides inherent physical correlation, it is desirable for a controlled process to maintain certain levels and relationships between variables. Inspection of the PCA models may point out an interesting effect: a few combinations of variables may be generally excluded, corresponding to these inherent relationships. Since such overlap is likely to exist for many controlled processes, the multiple disturbance scenario is further complicated. When the region spanned by the model for one component disturbance (outer disturbance) contains the region spanned by the model for another disturbance (inner disturbance), their combination will not be perfectly diagnosed. Instead, only the outer disturbance will be diagnosed and the inner disturbance will be masked. The type of disturbances may play a role in the success rate of disturbance diagnosis. Often, disturbances of deterministic nature (step, ramp) are masked less than random (noise) disturbances. The combinations of deterministic (step and ramp) disturbances may show various masking patterns. In some cases they follow the simple inner and outer
Table 1 Tennessee
41
disturbance pattern. In other cases, while the models of single disturbances have mutually overlapping means and highly similar spreads, the models of their combined disturbance may not overlap with the single-disturbance means. A quantitative indicator of when this type of effect might occur is the angle between the means of the disturbances, a and b, using a nominal in-control operating point as the vertex: cos( 0) = (db)/(
Ilallllbll)
where llall = Ja’;;
(9)
As the angle 8 approaches 90”, this effect becomes more pronounced and the overlap of the combination with the component single-disturbance models is reduced. Hence, while the success of multiple disturbance diagnosis using only single disturbances in the discrimination scheme is low, the success of a scheme including combination disturbances is improved. 5.2. Misdiagnosis of additional disturbances when single disturbances occur In the analysis of random or noisy disturbances, it was seen that some single disturbance models masked secondary disturbances. Alternatively, with step or ramp disturbances, when the combination distur-
Eastman process disturbances
Tag
Disturbance
description
1 2 3 4 5 6 7 8 9 A B C C&F D E F G H I J K
Feed ratio Feed composition Feed temperature Reactor coolant inlet temperature Condenser coolant inlet temperature Feed loss Feed header pressure loss Feed composition Feed temperature Feed temperature Reactor coolant inlet temperature Condenser coolant inlet temperature Disturbance C and sticking condenser coolant valve Drift in reaction kinetics Sticking reactor coolant valve Sticking condenser coolant valve Unknown fault Unknown fault Unknown fault Unknown fault Unknown fault
Stream
Type
4 4 2 12 13 1 4 4 4 2 12 13 13
Step Step Step Step Step Step Step Random Random Random Random Random Random Ramp Random Random Random Ramp Ramp Random Random
12 13
42
A.C. Raich, A. Cinar / Chemometrics
and Intelligent Laboratory Systems 30 (1995) 37-48
bance model is outer to the models for the single disturbances that are included in the combination disturbance, additional disturbances, i.e., the other disturbances in the combination model, may be diagnosed when not actually present. A quantitative measure of the likelihood of such misdiagnosis can be defined by the angle between means of the combination and its respective components.
6. Assessment of model similarity and disturbance diagnosis capability with simulated plant data The Tennessee Eastman Industrial Challenge problem [3] was presented as a realistic, plant-scale process simulator to use in process control research. With multiple process units, feed, exit and recycle streams in gas and liquid phase, the process provides considerable correlation between several measurements. In addition, the software has 21 programmed disturbances, including step, ramp, random, and noisy changes in flow rates, temperatures, pressures, and concentrations (Table 1). Data sets generated from the simulator for this study included 21 groups of 22 correlated variables, generally with over 200 observations in each group. A set of such large dimensions is beyond the scope of most statistical methods for diagnosis and evaluation of similarity. Trajectories of various disturbances are plotted on the plane
of the first two PC scores in Figs. l-3. Fig. 1 displays 21 different trajectories resulting from disturbances listed in Table 1. It is impossible to visually discriminate most of them, and the use of additional scores plots do not provide enough separation. Hence, numerical tools for discrimination are needed. A successful integrated monitoring and disturbance diagnosis scheme based on PCA models was developed previously [1,2]. Such a scheme is limited by the similarity between component models. The methodology presented in this paper is applied to Tennessee Eastman plant data for assessing the overlap and generating set confidence factors about the disturbance discrimination capability of the diagnosis methods. PCA models of the same set of original input variables are compared using slightly different scaling, as indicated by reference to separate means (Eq. (5)). Data were scaled to unit variance within each type of disturbance for similarity factor computations. The outcome of the diagnosis efforts are summarized in Table 2. The disturbance introduced to the process is listed in the first column. The other columns of Table 2 indicate the disturbance diagnosed by the proposed methodology. For example, the first row shows that disturbance 1 was diagnosed 230 times as disturbance 1, 7 times as disturbance 15, and once as disturbance 21. The outcome of similarity factor computations is
First principal Component of Normal Operation
Fig. 1. Display of Tennessee Eastman process responses to 21 different disturbances. mal operation data.
Principal components
computations
are based on nor-
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
2
0 185 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
3
0 0
0 0
0 241 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0
68 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0
57 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
9
10
0
0 0 0
0 1 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0
113 0 0 0 0 0 1 0 0 0 0 0 0 1
0 194 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0
0 0 0 0 0 0 1 126 0 0 0 0 0 0 39 0 0 0 0
0
0
0 0
0
problem
0
0
0
8
Control Challenge
0
0
0
0 0
7 0
6 0
5 0
Eastman
0
to Tennessee
4
of disturbances 11 0 1 0 167 0 0 0 0 0 3 77 0 0 5 0 0 3 4 0 0 0
12
1 0 0 63 0 0 0 0 2 0 45 45 13 0 0 0 2 42 0 5
0
0
53 0 0 0 0 0 0 0
0
0
0
0
0
0
0
0
0
0 0
0 108 0 0 0 0 2 2 169 169 25 0 0 0 0 17 0 5
0
0
14 0
13 0
15 7 6 11 1 0 0 0 0 22 3 10 0 0 4 4 3 15 5 17 3 2
16
16 22 0 0 0 0 16 27 2 6 3 3 14 6 6 3 11 1 6 7
0
17 0 0 0 1 0 0 0 0 0 24 1 0 0 0 0 0 151 1 0 0 0
18 0 0 0 8 0 0 0 0 0 0 0 0 0 0 0 0 0 184 0 0 0
19 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 9 0 0
0 16 22 0 0 0 0 16 27 2 6 3 3 14 6 6 3 11 1 6 7
20
1 16 0 0 1 0 0 0 0 0 2 2 2 7 0 0 0 1 3 0 192
21
Entries indicate number of observations from disturbance indicated by row which were diagnosed as belonging to disturbance indicated by column: entries off the diagonal are misdiagnosed. Overall success in diagnosis for these new observations is around 65%, which concurs with the estimate of 35% overlap between all disturbances, derived using Table 4.
1
230 0 0 0 0 0 0 71 0 0 0 0 0 0 0 0 0 0 0 0 0
#
Table 2 Matrix of diagnosis
1 0.93 0.71 0.92 0.91 0.97 0.93 0.98 0.69 0.66 0.89 0.89 0.90 0.89 0.06 0.00 0.81 0.91 0.85 0.00 0.88
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
0.93 1 0.74 0.89 0.78 0.93 0.92 0.93 0.73 0.77 0.91 0.89 0.89 0.94 0.06 0.00 0.85 0.88 0.93 0.00 0.84
2
0.71 0.74 1 0.73 0.73 0.69 0.70 0.71 0.71 0.68 0.75 0.71 0.71 0.75 0.32 0.25 0.73 0.73 0.74 0.25 0.69
3
0.92 0.89 0.73 1 0.92 0.92 0.93 0.92 0.68 0.74 0.92 0.94 0.94 0.93 0.07 0.00 0.81 0.93 0.88 0.00 0.88
4 0.91 0.78 0.73 0.92 1 0.91 0.93 0.90 0.65 0.68 0.88 0.94 0.94 0.88 0.00 0.00 0.76 0.90 0.82 0.00 0.91
5
6 0.97 0.93 0.69 0.92 0.91 1 0.96 0.98 0.69 0.67 0.89 0.90 0.90 0.89 0.00 0.00 0.77 0.90 0.87 0.00 0.88
7 0.93 0.92 0.70 0.93 0.93 0.96 1 0.94 0.66 0.55 0.89 0.93 0.93 0.89 0.00 0.00 0.76 0.91 0.89 0.00 0.89
8 0.98 0.93 0.71 0.92 0.90 0.98 0.94 1 0.68 0.67 0.91 0.89 0.89 0.92 0.00 0.00 0.77 0.92 0.88 0.00 0.87
9 0.69 0.73 0.71 0.68 0.65 0.69 0.66 0.68 1 0.68 0.71 0.66 0.66 0.71 0.35 0.22 0.76 0.76 0.74 0.22 0.65
10
0.84 0.80 0.78 0.00 0.67
h.78 0.72 0.72 0.79
0.66 0.77 0.68 0.74 0.68 0.67 0.55 0.67 0.68
11 0.89 0.91 0.75 0.92 0.88 0.89 0.89 0.91 0.71 0.78 1 0.92 0.92 0.95 0.00 0.00 0.85 0.95 0.90 0.00 0.88
12 0.89 0.89 0.71 0.94 0.94 0.90 0.93 0.89 0.66 0.72 0.92 1 0.99 0.88 0.09 0.07 0.80 0.94 0.85 0.07 0.91
13 0.90 0.89 0.71 0.94 0.94 0.90 0.93 0.89 0.66 0.72 0.92 0.99 1 0.88 0.11 0.07 0.80 0.94 0.84 0.07 0.91
14 0.89 0.94 0.75 0.93 0.88 0.89 0.89 0.92 0.71 0.79 0.95 0.88 0.88 1 0.09 0.10 0.87 0.94 0.93 0.10 0.85
15 0.06 0.06 0.32 0.07 0.00 0.00 0.00 0.00 0.35 0.84 0.00 0.09 0.11 0.09 1 0.88 0.00 0.00 0.06 0.88 0.00
16 0.00 0.25 0.00 0.00 0.00 0.00 0.00 0.22 0.80 0.00 0.07 0.07 0.10 0.88 1 0.07 0.00 0.00 1.00 0.00
0.00
17 0.81 0.85 0.73 0.81 0.76 0.77 0.76 0.77 0.76 0.78 0.85 0.80 0.80 0.87 0.00 0.07 1 0.88 0.91 0.07 0.77
18 0.91 0.88 0.73 0.93 0.90 0.90 0.91 0.92 0.76 0.00 0.95 0.94 0.94 0.94 0.00 0.00 0.88 1 0.89 0.00 0.90
19 0.85 0.93 0.74 0.88 0.82 0.87 0.89 0.88 0.74 0.67 0.90 0.85 0.84 0.93 0.06 0.00 0.91 0.89 1 0.00 0.84
20 0.00 0.00 0.25 0.00 0.00 0.00 0.00 0.00 0.22 0.00 0.00 0.07 0.07 0.10 0.88 1.00 0.07 0.00 0.00 1 0.00
21 0.88 0.84 0.69 0.88 0.91 0.88 0.89 0.87 0.65 0.00 0.88 0.91 0.91 0.85 0.00 0.00 0.77 0.90 0.84 0.00 1
Factors close to 1 indicate high similarity behveen directions of models. The matrix is symmetric since comparison of models designated in rows and columns is insignificant. Note that the diagonal is ones since every model is identical to itself.
1
#
Table 3 Similarity factor between disturbance models for Tennessee Eastman Control Challenge problem
A.C. Raich, A. Qnar/Chemometrics
and Intelligent Laboratory Systems 30 (1995) 37-48
45
46
A.C. Raich, A. @zar/
Chemometrics
and Intelligent Laboratory Systems 30 (1995) 37-48
given in Table 3. Comparison to the diagnosis of observations from each disturbance listed in Table 2 shows that 85% of observations were diagnosed as fitting a model with a similarity factor above 0.6 to the model of the true disturbance. For example, row 5 in Table 2 lists 68 points correctly diagnosed as disturbance 5, 63 incorrectly diagnosed as disturbance 12, 108 misdiagnosed as disturbance 13, and 1 point misclassified as disturbance 21. The corresponding similarity factors are 1,0.94,0.94, and 0.91, which are all higher than the 0.6 cutoff and among the models most similar to the model for disturbance 5. Notable misclassifications in the absence of high similarity factors include misdiagnosis of observations as belonging to groups 15, 16, and 20, which had low similarity factors to most other models. These upsets were poorly detected and consequently the groups did not have an appropriate amount of data available. Insufficient sample size causes these models to be statistically unreliable and affects interpretation of the similarity factor as an indicator of misclassification between these and other models. An example can be seen in comparing the ninth row in Table 2 and Table 3, where 1 point is misdiagnosed as disturbance 10, 22 points as disturbance 15, 27 points as disturbance 16, and 27 points as disturbance 20. Of these misclassifications, Table 3 shows
First
only model 10 to have a high similarity factor. Adding up the number of observations classified along rows in Table 2, it can be seen that groups 15, 16, and 20 had less than 20, only one tenth of the average number of samples per group. To avoid the resulting misinterpretation of similarity factor, models can be screened to be sure that similar numbers of observations are used for all groups. Results of the F-test for mean overlap for this system are shown in Table 4. Using these results as a building block, the overlap between all disturbance models can be estimated as 35%. Both similarity measures indicate that models of virtually all single disturbances produce some overlap with each other. As expected, all the models excluded a few combinations of variables, which probably describe inherent physical singularities. Similarity measures indicate that the random or noisy disturbances have more overlap with other models, particularly with each other. The similarity measures could also systematically indicate which random disturbance in a pair would be masked. In general, the inner disturbance is masked; its mean overlaps the model for the combination, which was most similar in spread to the outer disturbance model. In combination with each other, step and ramp disturbances show less predictable masking patterns. Although some cases follow the simple inner and
Principal Component of Normal Operation
Fig. 2. Tennessee Eastman process response to disturbances *.
6 and 7. Disturbance
6: x, disturbance
7:
+ ,
simultaneous
disturbance
6 and 7:
A.C. Raich, A. Cinar/ Chemometrics and Intelligent Laboratory Systems 30 (1995) 37-48
47
First Principal Component of Normal Operation Fig. 3. Tennessee and 19: *.
Eastman process response to disturbances
14 and 19. Disturbance
outer pattern, sometimes models of single disturbances have mutually overlapping means and highly similar spreads, but the model of the combination disturbance does not overlap with the single-disturbance means. In Fig. 2, the trajectories for two different step disturbances (loss of feed stream and loss of header pressure) are shown when each occur separately or simultaneously. The trajectories are clearly different from each other, and the individual disturbance trajectories do not overlap with the combined disturbance trajectory. Also, a measured in Eq. (9), the angle between these trajectories is approximately 30”. Hence, unusual effects due to angle around the nominal operating point are unlikely. Fig. 3 shows the trajectories for a slow drift in reaction kinetics (ramp) and for another unknown disturbance. The trajectories for the individual disturbances as well as the trajectory for their combined occurrence have significant overlap. Incorporating combinations of random variations with ramp or step disturbances into the diagnosis scheme leads to misdiagnosis of a second disturbance when only a noisy disturbance is present. When the use of only single disturbance models was considered, inner random disturbances are masked. As might be expected, ramp or step disturbances tend to be outer models; this is consistent with moving the process farther off its control target or nominal oper-
14: x, disturbance
19: + , simultaneous
disturbance
14
ating point. As the outer model, single ramp and step disturbances mask secondary random noise disturbances. Combinations of only step or ramp disturbances are diagnosed successfully in schemes using either single disturbances or combinations. The percentage of successful diagnosis of various disturbance combinations based on score distances are: Disturbances 6-7 (step/step) 96%, 7-D (step/ramp) 96%, D-H (ramp/ramp) 72%, and C-F (random/random) 33%. In the single disturbance scheme, the outer disturbance is generally diagnosed, and only rarely a second disturbance is indicated when not actually preseni.
7. Conclusions New techniques for quantifying the overlap of PCA models are proposed. Interpretation of these measures and their implications to misdiagnosis and masking of multiple faults are demonstrated by using the Tennessee Eastman Challenge Problem as an example process. These methods may be useful in future work to evaluate similarity between process models derived under different conditions, to monitor process behavior, or to diagnose multiple disturbances for a broad spectrum of chemical processes.
48
A.C. Raich, A. Qnar/Chemometrics
and Intelligent Laboratory Systems 30 (1995) 37-48
References [l] A. Raich and A. Cinar, 1993 Am. Inst. Chem. Eng. Annu. Meet., St. Louis, MO, 7-12 November 1993, paper 148~. [2] A. Raich and A. Cinar, Proc. ADCHEM 1994, pp. 4.52. [3] J.J. Downs and E.F. Vogel, 1990 Am. Inst. Chem. Eng. Annu. Meet., Chicago, IL, November 1990, paper 24a. [4] J. Kresta, J.F. MacGregor and T. Marlin, Can. J. Chem. Eng., 69 (1991) 35. [5] P. Miller, R.E. Swanson and C.F. Heckler, submitted.
[6] W. Ku and C. Georgakis, 1993 Am. Inst. Chem. Eng. Annu. Meet., St. Louis, MO, 7-12 November 1993, paper 149g. [7] K. Fukunaga, Introduction to Statistical Pattern Recognition, 2nd edn., Academic Press, New York, 1990. [8] G.J. McLachlan, Discriminant Analysis and Statistical Pattern Recognition, Wiley, New York, 1992. [9] W.J. Krzanowski, J. Am. Stat. Assoc., 74 (1979) 703. [lo] G.S. Watson, Statistics on Spheres, Wiley, New York, 1983. [ll] E.R. Malinowski, J. Chemom., 3 (1988) 49.
ELSEVIER
Chemometrics and Intelligent Laboratory Systems 30 (1995) 37-48
Multivariate statistical methods for monitoring continuous processes: assessment of discrimination power of disturbance models and diagnosis of multiple disturbances A.C. Raich, A. Cinar
*
Chemical Engineering Department, Illinois Institute of Technology, Chicago, IL 60616, USA
Received 21 December 1994; accepted 31 March 1995
Abstract A new methodology was reported [1,2] for integrated use of principal components analysis (PCA) and discriminant analysis in order to determine out-of-control status of a continuous process and to diagnose the source causes for abnormal behav-
ior. Most of the disturbances were identified with good rates of success, with a higher success rate for step or ramp type of disturbances. Quantitative tools that evaluate overlap and similarity between high-dimensional PCA models are proposed in this communication, and their implications on determining the discrimination power of PCA models of processes operating under disturbances are discussed. Diagnosis of several disturbances occurring simultaneously is also investigated. The criterion developed provide upper limits of discrimination power of various single and multiple process disturbances. The techniques developed are illustrated by assessing the process described by the Tennessee Eastman Control Challenge problem [3]. Keywords: Pattern recognition; Discriminant analysis; Fault diagnosis; Principal components analysis
1. Introduction Statistical process control (SPC) has gained importance in chemical process industries as a means to improve quality and yield, and to reduce costs. Conventional univariate SPC methods do not make the best use of data available, and often yield misleading results on multivariate chemical processes. The use of several univariate charts simultaneously to detect the out-of-control status of a multivariable process does
* Corresponding author, email: [email protected] 0169-7439/95/$09.50
not have a theoretical foundation and yields large Type I and Type II errors. Hotelling’s T2 charts provide reliable and correct tools for detecting that a multivariable process has gone out-of-control, but they cannot provide clues about the process variables that signaled the out-of-control behavior and the source cause(s) for the abnormal behavior. Chemometric methods and principal components analysis (PCA) have been utilized for merging detection of out-of-control status [4], identification of the process variables with significant variations [5], and diagnosis of source causes [1,2,4,6]. For small-dimensional systems which can be represented adequately by a
0 1995 Elsevier Science B.V. All rights reserved
SSDI 0169-7439(95)00035-6
38
A.C. Raich, A. Cinar/Chemometrics
and Intelligent Laboratory Systems 30 (1995) 37-48
few principal components, graphical techniques have been proposed [4]. In our previous work, novel methods using PCA and discriminant analysis were reported for monitoring of multivariable continuous processes. These statistical methods have integrated detection and diagnosis into an automated, quantitative analysis framework [1,2]. Process monitoring centers around a PCA model built with data collected at nominal operating conditions. When an outof-control situation is detected, the observation is compared to PCA models for known disturbances. Using refinements of statistical distances, discriminant analysis then selects the most likely cause(s) for the current out-of-control condition. Successful diagnosis depends on the discrimination potential of these disturbance models, as high similarity between models would limit the potential to distinguish between the corresponding disturbances. Other questions can be posed about the usefulness of particular PCA models when applied to continuous processes. Comparison of models built from data at different operating points, non-stationary conditions, or even different sampling frequencies may be of interest for monitoring process performance. In diagnosis of source causes for disturbances, a basis for comparison of models can help identify misdiagnosis patterns. Attempts to diagnose multiple faults further highlights the need for a deeper understanding of the similarities between models for contributing disturbances and masking of secondary faults. Also, for a process that has a moving target, such as a batch process, it may be useful to have a statistical basis to identify changes in the process model. In disturbance diagnosis, where there are many potential models, it is useful to have a quantitative measure of similarity or overlap between models, and to identify the overall likelihood of successful diagnosis.
2. Assessment of model similarity by overlap of
covariance Eigenvalue-based techniques such as PCA describe a set of data with two basic entities: a central point and directions of spread around the center. In statistical terms, these correspond to the mean and covariance, which completely characterize a multivariate normal probability distribution.
In comparing multivariate models, much work has been done on testing of significant differences between means with the same covariance. It is much more difficult to test for differences in covariance, especially when there is a difference in the means. However, especially in disturbance diagnosis, it is the difference in covariance which can be most crucial; diagnosis can be successfully done, whether or not means are different, as long as there is a difference in covariance [7]. Testing for eigenvalue models of covariance adds new complications, as the statistical characteristics are not well known, even for the most common distributions. Simplifying assumptions for special cases can be made, with significant loss of generality [8]. Using geometry concepts rather than statistics with known distributions, the angles between the principal components of different models can be utilized to develop a useful criteria. 2.1. Angles between coordinate directions A general treatment of between-groups comparisons of principal components [9] describes the derivation of angles between coordinate axes from different models. Using eigenvalue decomposition, the angles (Ye between PCA coordinate direction sets P, and Pz correspond to eigenvalues s:12: P; P, P; P, = L’SL
(1)
(Yk= cos-y
(2)
$2)
where L gives a set of consensus coordinates and sk is the kth largest eigenvalue in S, corresponding to the smallest angle CQ between the k-dimensional coordinate subspaces [9]. Angles between covariance coordinate systems are visually interpretable in 2 or 3 dimensions, and computationally extend to higher dimensional systems in a straightforward manner. An ideal statistic would describe conical confidence thresholds for angles between coordinate axes of different models. However, such a statistical analysis is currently prohibitively complex beyond 2 or 3 dimensions [lo]. Instead, possible simple geometric benchmarks include the minimum angle between models and the sum of squares of the cosines of angles between model axes. For high-dimensional models, the minimum angle between models can quickly become trivial. It is generally recognized that
A.C. Raich, A. Qnar/Chemometrics
and Intelligent Laboratory Systems 30 (1995) 37-48
considerable similarity between models is often found within the first few dimensions and that successful discrimination generally requires consideration of higher dimensions [7]. This necessitates the use of higher dimension models and can make the minimum angle as small as computational inaccuracy allows. 2.2. Similarity factor The sum of cosines of angles between model axes can be used to define a fraction of similarity between models. First describe exact similarity between two models. Models of data with a variance of one for each of the same number of independent variables would correspond to coordinate axes P, and P, both equal to the identity matrix. The eigenvector decomposition described earlier would result in a set of eigenvalues {sJ all equal to one with corresponding angles all equal to zero. The sum of squares of cosines would then be equal to the number of dimensions considered (p). A general measure of similarity f is then defined as the sum of squares of cosines divided by the number of dimensions, p: f=
;k$lcos2(.,) =1 i
Sk
P k=l
The similarity factor f ranges from zero (lack of similarity) to one (exact similarity). As a description of overall geometric similarity in spread, the similarity factor f provides a quantitative measure of difference in covariance directions between models. The similarity factor was used to assess the differences among various disturbance models. For example, classification of disturbances to the Tennessee Eastman Industrial Challenge Problem was observed to correspond to similarity factors greater than 0.6 for 85% of the observations. While a high similarity factor does not guarantee misclassifications, low similarity between models generally indicated a low probability for misclassifications. A low similarity factor value corresponds to fewer misclassifications. Besides providing comparisons between models of disturbances, the similarity factor could provide a way to check if PCA models for different stages of a batch process change their orientation around a moving mean (in time).
39
3. Assessment of model similarity by overlap of means Another statistical test for comparing multivariate models is based on differences in means. Geometrically, this corresponds to comparison of origin of coordinates rather than the coordinate directions that were used in testing the overlap of covariance. Many statistical tests have been developed for testing means, but these can become numerically unstable when significant correlation occurs. Working around the instability issue, the interest is to look at overlap between eigenvalue-based models. In a technique described to identify chemical species from chemometric data of multicomponent mixtures, target factor analysis sets a confidence level on the presence of contributing features [ll]. In a broader application, the method can place a likelihood on whether or not a suspected vector is a contributor to the model of a multivariate data set. The statistically and numerically viable approach [ll] is based on the distribution of quadratic forms, such as Mahalanobis weighing. A statistic can be defined to test if a specified vector is contributing to the description by an eigenvalue-based model, that is, if the vector is significantly inside the confidence region containing the data. With respect to overlap of means, the test can be applied to determine whether the mean from one model, pi, significantly overlaps the region of data from another model. This involves estimating p1 with respect to the other model, which is centered around p2. In this case, the test statistic is
with t=(pI-p2)Pandr=tP’-p1
(5)
where p is the fraction of variation in the data accounted for by the model, t is the approximation of pi by the model, P is the eigenvector model (directions) transforming p measurements to k dimensions, 2 is the covariance of the k new variables, pi is the mean of measurements for disturbance tested, pa is the mean of n modeled observations, and r is the residual error in pi unexplained by the model. The test indicates significant overlap of the means
A.C. Raich, A. @tar/Chemometrics
40
and Intelligent Laboratory Systems 30 (1995) 37-48
pr with the model around p2 when the statistic m is smaller than a threshold m, from the F distribution: m, = Fol;(p-
k),(n-
k)
natorial counting techniques used in the binomial and hypergeometric distributions can estimate the overall similarity or overlap between groups as:
(6)
d = (total number of overlaps between means and other groups)
In chemometric applications, a confidence level of 95% has been satisfactorily used. In general use, combined with the similarity factor technique, mean overlap analysis could be useful to test if an existing PCA model fits a new set of observations or if two PCA models are analogous. Used by itself, mean overlap was not as successful an indicator of pairwise misclassification as the similarity factor.
/l&g-111
(7)
The count of overlaps is done using the F test described in (6) and g is the number of groups under consideration. Similarly, an estimate of the best success rate between several groups can be simply stated as a percentile: 1 = lOO( 1 - d)
(8)
4. Overlap between many groups
5. Diagnosis of multiple disturbances A crucial consideration in formulating the rules for discriminating between multiple groups is the overall similarity between them. If the probability density functions of each group were exactly known, an upper bound for success in classification could be formulated [7]. Different rules could be devised and evaluated to balance their approach to this minimum error rate. The increase in complexity of the rules may not pay off in terms of reduction in the error rate. In real world situations where the density functions are not known exactly, such a rigorous evaluation is not feasible, but it is still useful to approximate the best success rate for correct discrimination of disturbances. This analysis is generally related to a Mahalanobis-type distance between groups, with increasing distance corresponding to greater success in classification. The difficulties related to Mahalanobis inversion with correlated data, especially with different covariance structures, makes characterization of separation between groups difficult. Only some special cases with a few groups, a few variables, or special characteristics can be handled directly [8]. An additional important consideration is over-fitting of the model to data used in training. When too complex a model is used, inflated success rates may be found, yet the use of the model with new data will not be as successful. This effect, which is generally attributed to the inclusion of noise, is countered in practice by evaluating the rate of success with new data, often referred to as cross-validation. Rather than a detailed statistical approach, a general guideline reminiscent of the form of the combi-
Diagnosis of a source cause for an out-of-control condition has been discussed extensively using a variety of distance-based methods. However, such methods were designed for use with single disturbances. Their adaptability to handle combinations of disturbances is also of interest. Quantitative measures of overlap between singledisturbance models can be generalized to analyze diagnosis accuracy of multiple disturbances. If there were no overlap, two alternative schemes might handle multiple disturbances within an existing parallel discriminant framework. Similar to a fuzzy logic approach, idealizing the combination of disturbances as being located between the regions of the underlying component disturbances, allocations of membership to the different independent disturbances contributing to the combination might provide diagnosis of underlying disturbances. More generally, a successful extension of the discrimination scheme might also be done via the introduction of new models for each combination of interest; this approach could describe combinations of disturbances that produce models that are not simply the consensus of component models. Using the measures of similarity in model center and direction of spread described earlier, the hypothesis of independence can be checked. Besides describing misclassifications between single disturbances, similarity measures can give insight into treatment of multiple disturbances, including masking of some conditions.
A.C. Raich, A. @zar/
Chemometrics
and Intelligent Laboratory Systems 30 (1995) 37-48
5.1. Masking of multiple disturbances It is realistic to expect considerable similarity between models for the same process. Besides inherent physical correlation, it is desirable for a controlled process to maintain certain levels and relationships between variables. Inspection of the PCA models may point out an interesting effect: a few combinations of variables may be generally excluded, corresponding to these inherent relationships. Since such overlap is likely to exist for many controlled processes, the multiple disturbance scenario is further complicated. When the region spanned by the model for one component disturbance (outer disturbance) contains the region spanned by the model for another disturbance (inner disturbance), their combination will not be perfectly diagnosed. Instead, only the outer disturbance will be diagnosed and the inner disturbance will be masked. The type of disturbances may play a role in the success rate of disturbance diagnosis. Often, disturbances of deterministic nature (step, ramp) are masked less than random (noise) disturbances. The combinations of deterministic (step and ramp) disturbances may show various masking patterns. In some cases they follow the simple inner and outer
Table 1 Tennessee
41
disturbance pattern. In other cases, while the models of single disturbances have mutually overlapping means and highly similar spreads, the models of their combined disturbance may not overlap with the single-disturbance means. A quantitative indicator of when this type of effect might occur is the angle between the means of the disturbances, a and b, using a nominal in-control operating point as the vertex: cos( 0) = (db)/(
Ilallllbll)
where llall = Ja’;;
(9)
As the angle 8 approaches 90”, this effect becomes more pronounced and the overlap of the combination with the component single-disturbance models is reduced. Hence, while the success of multiple disturbance diagnosis using only single disturbances in the discrimination scheme is low, the success of a scheme including combination disturbances is improved. 5.2. Misdiagnosis of additional disturbances when single disturbances occur In the analysis of random or noisy disturbances, it was seen that some single disturbance models masked secondary disturbances. Alternatively, with step or ramp disturbances, when the combination distur-
Eastman process disturbances
Tag
Disturbance
description
1 2 3 4 5 6 7 8 9 A B C C&F D E F G H I J K
Feed ratio Feed composition Feed temperature Reactor coolant inlet temperature Condenser coolant inlet temperature Feed loss Feed header pressure loss Feed composition Feed temperature Feed temperature Reactor coolant inlet temperature Condenser coolant inlet temperature Disturbance C and sticking condenser coolant valve Drift in reaction kinetics Sticking reactor coolant valve Sticking condenser coolant valve Unknown fault Unknown fault Unknown fault Unknown fault Unknown fault
Stream
Type
4 4 2 12 13 1 4 4 4 2 12 13 13
Step Step Step Step Step Step Step Random Random Random Random Random Random Ramp Random Random Random Ramp Ramp Random Random
12 13
42
A.C. Raich, A. Cinar / Chemometrics
and Intelligent Laboratory Systems 30 (1995) 37-48
bance model is outer to the models for the single disturbances that are included in the combination disturbance, additional disturbances, i.e., the other disturbances in the combination model, may be diagnosed when not actually present. A quantitative measure of the likelihood of such misdiagnosis can be defined by the angle between means of the combination and its respective components.
6. Assessment of model similarity and disturbance diagnosis capability with simulated plant data The Tennessee Eastman Industrial Challenge problem [3] was presented as a realistic, plant-scale process simulator to use in process control research. With multiple process units, feed, exit and recycle streams in gas and liquid phase, the process provides considerable correlation between several measurements. In addition, the software has 21 programmed disturbances, including step, ramp, random, and noisy changes in flow rates, temperatures, pressures, and concentrations (Table 1). Data sets generated from the simulator for this study included 21 groups of 22 correlated variables, generally with over 200 observations in each group. A set of such large dimensions is beyond the scope of most statistical methods for diagnosis and evaluation of similarity. Trajectories of various disturbances are plotted on the plane
of the first two PC scores in Figs. l-3. Fig. 1 displays 21 different trajectories resulting from disturbances listed in Table 1. It is impossible to visually discriminate most of them, and the use of additional scores plots do not provide enough separation. Hence, numerical tools for discrimination are needed. A successful integrated monitoring and disturbance diagnosis scheme based on PCA models was developed previously [1,2]. Such a scheme is limited by the similarity between component models. The methodology presented in this paper is applied to Tennessee Eastman plant data for assessing the overlap and generating set confidence factors about the disturbance discrimination capability of the diagnosis methods. PCA models of the same set of original input variables are compared using slightly different scaling, as indicated by reference to separate means (Eq. (5)). Data were scaled to unit variance within each type of disturbance for similarity factor computations. The outcome of the diagnosis efforts are summarized in Table 2. The disturbance introduced to the process is listed in the first column. The other columns of Table 2 indicate the disturbance diagnosed by the proposed methodology. For example, the first row shows that disturbance 1 was diagnosed 230 times as disturbance 1, 7 times as disturbance 15, and once as disturbance 21. The outcome of similarity factor computations is
First principal Component of Normal Operation
Fig. 1. Display of Tennessee Eastman process responses to 21 different disturbances. mal operation data.
Principal components
computations
are based on nor-
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
2
0 185 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
3
0 0
0 0
0 241 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0
68 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0
57 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
9
10
0
0 0 0
0 1 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0
113 0 0 0 0 0 1 0 0 0 0 0 0 1
0 194 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0
0 0 0 0 0 0 1 126 0 0 0 0 0 0 39 0 0 0 0
0
0
0 0
0
problem
0
0
0
8
Control Challenge
0
0
0
0 0
7 0
6 0
5 0
Eastman
0
to Tennessee
4
of disturbances 11 0 1 0 167 0 0 0 0 0 3 77 0 0 5 0 0 3 4 0 0 0
12
1 0 0 63 0 0 0 0 2 0 45 45 13 0 0 0 2 42 0 5
0
0
53 0 0 0 0 0 0 0
0
0
0
0
0
0
0
0
0
0 0
0 108 0 0 0 0 2 2 169 169 25 0 0 0 0 17 0 5
0
0
14 0
13 0
15 7 6 11 1 0 0 0 0 22 3 10 0 0 4 4 3 15 5 17 3 2
16
16 22 0 0 0 0 16 27 2 6 3 3 14 6 6 3 11 1 6 7
0
17 0 0 0 1 0 0 0 0 0 24 1 0 0 0 0 0 151 1 0 0 0
18 0 0 0 8 0 0 0 0 0 0 0 0 0 0 0 0 0 184 0 0 0
19 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 9 0 0
0 16 22 0 0 0 0 16 27 2 6 3 3 14 6 6 3 11 1 6 7
20
1 16 0 0 1 0 0 0 0 0 2 2 2 7 0 0 0 1 3 0 192
21
Entries indicate number of observations from disturbance indicated by row which were diagnosed as belonging to disturbance indicated by column: entries off the diagonal are misdiagnosed. Overall success in diagnosis for these new observations is around 65%, which concurs with the estimate of 35% overlap between all disturbances, derived using Table 4.
1
230 0 0 0 0 0 0 71 0 0 0 0 0 0 0 0 0 0 0 0 0
#
Table 2 Matrix of diagnosis
1 0.93 0.71 0.92 0.91 0.97 0.93 0.98 0.69 0.66 0.89 0.89 0.90 0.89 0.06 0.00 0.81 0.91 0.85 0.00 0.88
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
0.93 1 0.74 0.89 0.78 0.93 0.92 0.93 0.73 0.77 0.91 0.89 0.89 0.94 0.06 0.00 0.85 0.88 0.93 0.00 0.84
2
0.71 0.74 1 0.73 0.73 0.69 0.70 0.71 0.71 0.68 0.75 0.71 0.71 0.75 0.32 0.25 0.73 0.73 0.74 0.25 0.69
3
0.92 0.89 0.73 1 0.92 0.92 0.93 0.92 0.68 0.74 0.92 0.94 0.94 0.93 0.07 0.00 0.81 0.93 0.88 0.00 0.88
4 0.91 0.78 0.73 0.92 1 0.91 0.93 0.90 0.65 0.68 0.88 0.94 0.94 0.88 0.00 0.00 0.76 0.90 0.82 0.00 0.91
5
6 0.97 0.93 0.69 0.92 0.91 1 0.96 0.98 0.69 0.67 0.89 0.90 0.90 0.89 0.00 0.00 0.77 0.90 0.87 0.00 0.88
7 0.93 0.92 0.70 0.93 0.93 0.96 1 0.94 0.66 0.55 0.89 0.93 0.93 0.89 0.00 0.00 0.76 0.91 0.89 0.00 0.89
8 0.98 0.93 0.71 0.92 0.90 0.98 0.94 1 0.68 0.67 0.91 0.89 0.89 0.92 0.00 0.00 0.77 0.92 0.88 0.00 0.87
9 0.69 0.73 0.71 0.68 0.65 0.69 0.66 0.68 1 0.68 0.71 0.66 0.66 0.71 0.35 0.22 0.76 0.76 0.74 0.22 0.65
10
0.84 0.80 0.78 0.00 0.67
h.78 0.72 0.72 0.79
0.66 0.77 0.68 0.74 0.68 0.67 0.55 0.67 0.68
11 0.89 0.91 0.75 0.92 0.88 0.89 0.89 0.91 0.71 0.78 1 0.92 0.92 0.95 0.00 0.00 0.85 0.95 0.90 0.00 0.88
12 0.89 0.89 0.71 0.94 0.94 0.90 0.93 0.89 0.66 0.72 0.92 1 0.99 0.88 0.09 0.07 0.80 0.94 0.85 0.07 0.91
13 0.90 0.89 0.71 0.94 0.94 0.90 0.93 0.89 0.66 0.72 0.92 0.99 1 0.88 0.11 0.07 0.80 0.94 0.84 0.07 0.91
14 0.89 0.94 0.75 0.93 0.88 0.89 0.89 0.92 0.71 0.79 0.95 0.88 0.88 1 0.09 0.10 0.87 0.94 0.93 0.10 0.85
15 0.06 0.06 0.32 0.07 0.00 0.00 0.00 0.00 0.35 0.84 0.00 0.09 0.11 0.09 1 0.88 0.00 0.00 0.06 0.88 0.00
16 0.00 0.25 0.00 0.00 0.00 0.00 0.00 0.22 0.80 0.00 0.07 0.07 0.10 0.88 1 0.07 0.00 0.00 1.00 0.00
0.00
17 0.81 0.85 0.73 0.81 0.76 0.77 0.76 0.77 0.76 0.78 0.85 0.80 0.80 0.87 0.00 0.07 1 0.88 0.91 0.07 0.77
18 0.91 0.88 0.73 0.93 0.90 0.90 0.91 0.92 0.76 0.00 0.95 0.94 0.94 0.94 0.00 0.00 0.88 1 0.89 0.00 0.90
19 0.85 0.93 0.74 0.88 0.82 0.87 0.89 0.88 0.74 0.67 0.90 0.85 0.84 0.93 0.06 0.00 0.91 0.89 1 0.00 0.84
20 0.00 0.00 0.25 0.00 0.00 0.00 0.00 0.00 0.22 0.00 0.00 0.07 0.07 0.10 0.88 1.00 0.07 0.00 0.00 1 0.00
21 0.88 0.84 0.69 0.88 0.91 0.88 0.89 0.87 0.65 0.00 0.88 0.91 0.91 0.85 0.00 0.00 0.77 0.90 0.84 0.00 1
Factors close to 1 indicate high similarity behveen directions of models. The matrix is symmetric since comparison of models designated in rows and columns is insignificant. Note that the diagonal is ones since every model is identical to itself.
1
#
Table 3 Similarity factor between disturbance models for Tennessee Eastman Control Challenge problem
A.C. Raich, A. Qnar/Chemometrics
and Intelligent Laboratory Systems 30 (1995) 37-48
45
46
A.C. Raich, A. @zar/
Chemometrics
and Intelligent Laboratory Systems 30 (1995) 37-48
given in Table 3. Comparison to the diagnosis of observations from each disturbance listed in Table 2 shows that 85% of observations were diagnosed as fitting a model with a similarity factor above 0.6 to the model of the true disturbance. For example, row 5 in Table 2 lists 68 points correctly diagnosed as disturbance 5, 63 incorrectly diagnosed as disturbance 12, 108 misdiagnosed as disturbance 13, and 1 point misclassified as disturbance 21. The corresponding similarity factors are 1,0.94,0.94, and 0.91, which are all higher than the 0.6 cutoff and among the models most similar to the model for disturbance 5. Notable misclassifications in the absence of high similarity factors include misdiagnosis of observations as belonging to groups 15, 16, and 20, which had low similarity factors to most other models. These upsets were poorly detected and consequently the groups did not have an appropriate amount of data available. Insufficient sample size causes these models to be statistically unreliable and affects interpretation of the similarity factor as an indicator of misclassification between these and other models. An example can be seen in comparing the ninth row in Table 2 and Table 3, where 1 point is misdiagnosed as disturbance 10, 22 points as disturbance 15, 27 points as disturbance 16, and 27 points as disturbance 20. Of these misclassifications, Table 3 shows
First
only model 10 to have a high similarity factor. Adding up the number of observations classified along rows in Table 2, it can be seen that groups 15, 16, and 20 had less than 20, only one tenth of the average number of samples per group. To avoid the resulting misinterpretation of similarity factor, models can be screened to be sure that similar numbers of observations are used for all groups. Results of the F-test for mean overlap for this system are shown in Table 4. Using these results as a building block, the overlap between all disturbance models can be estimated as 35%. Both similarity measures indicate that models of virtually all single disturbances produce some overlap with each other. As expected, all the models excluded a few combinations of variables, which probably describe inherent physical singularities. Similarity measures indicate that the random or noisy disturbances have more overlap with other models, particularly with each other. The similarity measures could also systematically indicate which random disturbance in a pair would be masked. In general, the inner disturbance is masked; its mean overlaps the model for the combination, which was most similar in spread to the outer disturbance model. In combination with each other, step and ramp disturbances show less predictable masking patterns. Although some cases follow the simple inner and
Principal Component of Normal Operation
Fig. 2. Tennessee Eastman process response to disturbances *.
6 and 7. Disturbance
6: x, disturbance
7:
+ ,
simultaneous
disturbance
6 and 7:
A.C. Raich, A. Cinar/ Chemometrics and Intelligent Laboratory Systems 30 (1995) 37-48
47
First Principal Component of Normal Operation Fig. 3. Tennessee and 19: *.
Eastman process response to disturbances
14 and 19. Disturbance
outer pattern, sometimes models of single disturbances have mutually overlapping means and highly similar spreads, but the model of the combination disturbance does not overlap with the single-disturbance means. In Fig. 2, the trajectories for two different step disturbances (loss of feed stream and loss of header pressure) are shown when each occur separately or simultaneously. The trajectories are clearly different from each other, and the individual disturbance trajectories do not overlap with the combined disturbance trajectory. Also, a measured in Eq. (9), the angle between these trajectories is approximately 30”. Hence, unusual effects due to angle around the nominal operating point are unlikely. Fig. 3 shows the trajectories for a slow drift in reaction kinetics (ramp) and for another unknown disturbance. The trajectories for the individual disturbances as well as the trajectory for their combined occurrence have significant overlap. Incorporating combinations of random variations with ramp or step disturbances into the diagnosis scheme leads to misdiagnosis of a second disturbance when only a noisy disturbance is present. When the use of only single disturbance models was considered, inner random disturbances are masked. As might be expected, ramp or step disturbances tend to be outer models; this is consistent with moving the process farther off its control target or nominal oper-
14: x, disturbance
19: + , simultaneous
disturbance
14
ating point. As the outer model, single ramp and step disturbances mask secondary random noise disturbances. Combinations of only step or ramp disturbances are diagnosed successfully in schemes using either single disturbances or combinations. The percentage of successful diagnosis of various disturbance combinations based on score distances are: Disturbances 6-7 (step/step) 96%, 7-D (step/ramp) 96%, D-H (ramp/ramp) 72%, and C-F (random/random) 33%. In the single disturbance scheme, the outer disturbance is generally diagnosed, and only rarely a second disturbance is indicated when not actually preseni.
7. Conclusions New techniques for quantifying the overlap of PCA models are proposed. Interpretation of these measures and their implications to misdiagnosis and masking of multiple faults are demonstrated by using the Tennessee Eastman Challenge Problem as an example process. These methods may be useful in future work to evaluate similarity between process models derived under different conditions, to monitor process behavior, or to diagnose multiple disturbances for a broad spectrum of chemical processes.
48
A.C. Raich, A. Qnar/Chemometrics
and Intelligent Laboratory Systems 30 (1995) 37-48
References [l] A. Raich and A. Cinar, 1993 Am. Inst. Chem. Eng. Annu. Meet., St. Louis, MO, 7-12 November 1993, paper 148~. [2] A. Raich and A. Cinar, Proc. ADCHEM 1994, pp. 4.52. [3] J.J. Downs and E.F. Vogel, 1990 Am. Inst. Chem. Eng. Annu. Meet., Chicago, IL, November 1990, paper 24a. [4] J. Kresta, J.F. MacGregor and T. Marlin, Can. J. Chem. Eng., 69 (1991) 35. [5] P. Miller, R.E. Swanson and C.F. Heckler, submitted.
[6] W. Ku and C. Georgakis, 1993 Am. Inst. Chem. Eng. Annu. Meet., St. Louis, MO, 7-12 November 1993, paper 149g. [7] K. Fukunaga, Introduction to Statistical Pattern Recognition, 2nd edn., Academic Press, New York, 1990. [8] G.J. McLachlan, Discriminant Analysis and Statistical Pattern Recognition, Wiley, New York, 1992. [9] W.J. Krzanowski, J. Am. Stat. Assoc., 74 (1979) 703. [lo] G.S. Watson, Statistics on Spheres, Wiley, New York, 1983. [ll] E.R. Malinowski, J. Chemom., 3 (1988) 49.