- Email: [email protected]
Reconstruction and Observability, a Survey
Copyright 10 IFAC Nonlinear Control Systems Design, Enschede, The Netherlands, 1998
RECONSTRUCTION AND OBSERVABILITY, A SURVEY Floris Takens •
* Department of Mathematics, University of Groningen. Netherlands
It is worth mentioning that autocorrelations (or power spectra) are not capable to detect the difference between determistic and stochastic processes: with the well known Logistic system (xn+ I 4xn (1 - xn), with Xn E [0, 1]) one generates orbits, which have, as time series, zero autocorrelations, just like a completely random (i.i.d.) time series.
1. INTRODUCTION
=
Reconstruction, as this term is used in the theory of non linear dynamical systems and their time series, is very closely related with the notion of observability in system and control theory: the basic idea is that the state of a deterministic system is uniquely determined by a finite set of successive observations; the necessary number of observations depends on the order of the system. This idea was well established in the theory of linear systems, and the notion is basic in the theory of Kalman filtering which was devoIloped first in (Kalman, 1960).
2. STATEMENT AND DISCUSSION OF THE
MAIN RESULT Here we give the statement of the reconstruction theorem. This theorem deals with time series generated by a dynamical system, given by
Around 1980, the problem of observability was reconsidered. From the systems and controIl point of view it was Aeyels who attacked and solved the problem for general nonlinear systems, i.e. without assumptions of analyticity, see (Aeyels, 1978; Aeyels, 1981 ; Aeyels, 1982) and the references mentioned in these papers. The main result is, as in the linear case, that observability is a generic property - a more precise formulation will be given in section 2.
- a state space X; - a map tp : X -+ X, which assigns to each state the next state; successions of states of the form {Xi = tp(xi-d} are called orbits; in the following we assume that X is a finite dimensional vector space or a submanifold of such a vector space and that tp is at least Cl; - a read out map y : X -+ R which assigns to each state X the value y(x) which is measured if the system is in that state; so an orbit {Xi = tpi(XO)} leads to a sequence of measured values, i.e. to a time series, {Yi = Y(Xi)}; also we assume the read out function Y to be Cl .
In that same period. researchers in the theory of nonlinear dynamical systems (in the tradition of S. Smale and R. Thorn) had realized that nonlinear systems, and in particular the systems which are called nowadays chaotic, can behave in a way which is seemingly random. This raised the question how to detect in these cases whether there is an underlying deterministic mechanism. This problem was solved, at least in principle. and its solution was based on this same result of generic observability. combined with a numericaVgeometrical methodology of analysis of large samples of high dimensional vectors which we will discuss in a later section. A treatment of these problems was given in (packard et al., 1980) from an experimental point of view and in (Takens, 1981) from a mathematical point of view. For a more recent and more didactical exposition see (Sauer et al., 1991).
With these ingredients we define the reconstruction maps: Reck : X -+ Rk, which assign to each state X
E X
the vector
(y(x), y(tp(x)), . . . , y(~-l (x))) E Rk.
In other words, Reck assigns to each state x the k successive measurements which would have resulted from tp and y if the system would have started at state x.
Next we mention that both tp and y belong to function spaces in which there is a topology. the so called
427
which is generated by a deterministic system whose state space has dimenion d (or whose state space has a higher dimension, but whose orbits are attracted to ad-dimensional attractor). Then the reconstruction vectors (Yi , " ' , Xi+I-l), which are the images of the points of the orbit, generating the time series, under the reconstruction map Recl , are all on the image, under the reconstruction map, of the state space (respectively attracor); this image has dimension at most d, because, by assumption, the recostruction map is differentiable. So if the embedding dimension I is bigger than d, the reconstruction vectors are concentrated on a subset of RI whose dimension is lower than l. (This holds even without the genericity asssumptions in the reconstruction theorem being satisfied.) This can be detected numerically by estimating the probability pl(£) that two ran~mly chosen reconstruction vectors in dimension I are, within distance £; the quantity pi (£) is called the I-dimensional correlation integral at length scale £ . If the density of the reconstruction vectors in RI is continuous, then pi (£) is propotional to £1 for small £; if these reconstruction vectors are concentrated on a subset of dimension d < I, then pi (£) is proportional to £d for small £.
strong Cl-topology. This means that we have the notion of generic elements of these function spaces: we say that a property P is generic for dynamical systems like (qJ, y) if there is an open and dense subset U of Cl (X, X) X Cl (X, R), such that whenever the dynamical system (Cp, ji) E U , it has property P. This is also expressed by saying that generic qJ and y have property P. Theorem J. For a state space X as above, and for generic qJ : X ~ X and y : X ~ R, the reconstruction maps Reck : X ~ R* are injective for k > 2· dim(X),
A few remarks are here in order: - This notion of genericity was introduced as an analogue of 'valid with probability one' in cases where there is no natural (probability) measure, but where there is a topological structure available. The actual status of genericity as a formalization of 'almost always valid' is rather complicated and dubious, but in the above theorem, this interpretation seems to be justified. This point has not yet been formalized mathematically in a satisfactory way, e.g. see (Sauer et aI., 1991). - From the point of view of the linear theory, it may be somewhat surprizing that we have to require that in the above therem k > 2 · dim(X) . This can be explained as follows . For every x E X it is a generic property for a set of dim(X) functions to form a good coordinate system in a small neighbourhood of x in the sense that there are no two points with the same coordinates. In the linear case, .this implies that these functions form globally a good coordinate system. This is not the case in the nonlinear situation. Stronger, there are examples of 2-dimensional surfaces, e.g. the projective plane, on which it is even not possible to have three smooth functions which form a global coordinate system in the sense that they distinguish any two points. - There are various generalizations and adaptations of this theorem. One can consiter systems with continuous time and one can consider read out functions with values in a higher dimensional vector space. Also one can consider infinite dimension state spaces if the (asymptotic) dynamics takes place on a finite dimensional attractor. See (Sauer et aI., 1991) and the references mentioned in that paper.
For those y, for which the generic conditions on qJ and y in the reconstruction theorem are satisfied, this dimension is independent of y , so that it gives information about the dynamics which is independent of the the way it is observed, e.i. which is independent of y so far as 'y is generic' . The above description of the methodology of detecting deteminism in a time series needs much more details which we cannot give here. For example there are many different notions of dimension, and not for all of them the above statements are correct. For more details see (Kantz and Schreiber, 1997). In general one can say that the test for determinism along these lines is mainly a theoretical possibility: 'real' time series are essentially always contaminated with noise. On the other hand, the analysis of time series in terms of the correlation integrals, as defined above, provides information which cannot be obtained from an the ususal analysis in terms of power spectra and autocorrelations. This information is useful when assessing the predictability of a time series.
4. RECONSTRUCTION AND THE DISCRIMINATION BETWEEN DIFFERENT DYNAMICAL REGIMES
Independent of whether time series are generated by a deterministic system or not. it is often an important question to deceide whether two time series might have been generated by the same mechanism. One can think for example of compairing a measured time series with a time series generated by a mathematical model for the process under consideration. For this problem of discremination one can of course use
3. RECONSTRUCTION AND THE DETECTION OF DETERMINISTIC DYNAMICS
As we mentioned before, the method of reconstruction was introduced as a way to detect whether a given stationary time series was generated by a deterministic system or not. The idea can be explained in a simple manner: suppose we have a time series {Yd
428
power spectra and autocorrelations, but as we have seen, with these we even cannot see the difference between deterministic and stochastic processes. So additional methods are needed.
Aeyels, D. (1981). Generic observability of differentiable systems. SIAM 1. Control and Optimization 19, 595-603. Aeyels, D. (1982). Global observability of MorseSmale vector fields. 1. Diff. Equ. 45, 1-15. Diks, C, WR. van Zwet, F. Takens and 1. DeGoede (1996). Detecting differences between delay vector distributions. Phys. Rev. E 53,2169-2176. Kalman, R.E. (1960). A new approach to linear filtering and prediction problems. Trans. ASME, 1. Basic Eng 82, 35-45. Kantz, H. and T. Schreiber (1997) . Nonlinear time series analysis. number 7 In: Cambridge nonlinear science series. Cambridge University Press. Packard, N.H., 1.P. Crutchfield, 1.D. Farmer and R.S . Shaw (1980). Geometry from time series. Phys. Rev. Letters 45, 712-716. Sauer, T., 1.A. Yorke and M. Casdagli (1991). Embeddology.l. Stat. Phys. 65,579-616. Takens, F. (1981). Detecting strange attractors in turbulence. In: Dynamical systems and turbulence (D. Rand and L.-S Young, Eds.) . Number 898 In: Lecture Notes in Mathematics. Springer-VerJag.
Recently a method was devolloped by Diks et al (Diks et al., 1996) to distinguish time series by detecting possible differences between their reconstruction measures. The I-dimensional reconstruction measure J.LI is the probability measure on RI which corresponds to the density of the I-dimensional reconstruction vectors in RI . (For stationary time series, this can be made into a mathematically rigorous definition, but for nonstationary time series this notion does not make sense.) This is a natural approach for stationary time series. All (dynamical) information about such times series is contained in the reconstruction measures. In many cases one can even restrict to reconstruction measures in relatively low dimensions. This method of Diks is based of a smoothed version of correlation integrals and mixed correlation integrals which we can define as follows . First we note that the usual correlation integral in dimension I at length scale e is the expectation value of Ht(Yi , Yj ) , where we take the average over all pairs of I-dimensional reconstruction vectors (Yi , Yj), and where Ht stands for the function which is 1 if the distance between the arguments is less than e and is zero otherwise. We obtain a smoothed correlation integral Spl(e) if we replace the function Ht by Ft = (27re 2)-1/2 . exp( _d2 /2e 2 ), where d is the Euclidean distance between the two arguments. When considering two time series {Yi} and {zd, with I-dimensional reconstruction vectors {Yd and {Zi}' we define the mixed smoothed correlation integral M Spl(e) as the expectation value of Ft(Yi , Zj) . If there are various time series we indicate the time series to which the definition is applied as subscript, so the above mixed smoothed correlation integral can be denoted by SM pfr,z)(£)' The test for the time series Y = {yd and Z = (zd to be different is based on the fact that the quantity SP~(£) + SI1(£) - 2MSptr,z) (e) is always nonnegative, and it is positive if and only if the [dimensional reconstruction measures of the time series Y and Z are different. This is based on the fact that this expression is the integral, with respect to the Lebesgue measure on RI, of (p~ - p~)2, where p~, respectively p~, is the density of the measure on RI, obtained by convoluting the [-dimensional reconstruction measure of Y, respectively Z, with the Gauss normal distribution with average 0 and variance ~ £2 •
5. REFERENCES Aeyels, D. (1978). Global observability for nonlinear autonomous differential equations. PhD thesis. Wahington University. St. Louis, Missouri.
429
RECONSTRUCTION AND OBSERVABILITY, A SURVEY Floris Takens •
* Department of Mathematics, University of Groningen. Netherlands
It is worth mentioning that autocorrelations (or power spectra) are not capable to detect the difference between determistic and stochastic processes: with the well known Logistic system (xn+ I 4xn (1 - xn), with Xn E [0, 1]) one generates orbits, which have, as time series, zero autocorrelations, just like a completely random (i.i.d.) time series.
1. INTRODUCTION
=
Reconstruction, as this term is used in the theory of non linear dynamical systems and their time series, is very closely related with the notion of observability in system and control theory: the basic idea is that the state of a deterministic system is uniquely determined by a finite set of successive observations; the necessary number of observations depends on the order of the system. This idea was well established in the theory of linear systems, and the notion is basic in the theory of Kalman filtering which was devoIloped first in (Kalman, 1960).
2. STATEMENT AND DISCUSSION OF THE
MAIN RESULT Here we give the statement of the reconstruction theorem. This theorem deals with time series generated by a dynamical system, given by
Around 1980, the problem of observability was reconsidered. From the systems and controIl point of view it was Aeyels who attacked and solved the problem for general nonlinear systems, i.e. without assumptions of analyticity, see (Aeyels, 1978; Aeyels, 1981 ; Aeyels, 1982) and the references mentioned in these papers. The main result is, as in the linear case, that observability is a generic property - a more precise formulation will be given in section 2.
- a state space X; - a map tp : X -+ X, which assigns to each state the next state; successions of states of the form {Xi = tp(xi-d} are called orbits; in the following we assume that X is a finite dimensional vector space or a submanifold of such a vector space and that tp is at least Cl; - a read out map y : X -+ R which assigns to each state X the value y(x) which is measured if the system is in that state; so an orbit {Xi = tpi(XO)} leads to a sequence of measured values, i.e. to a time series, {Yi = Y(Xi)}; also we assume the read out function Y to be Cl .
In that same period. researchers in the theory of nonlinear dynamical systems (in the tradition of S. Smale and R. Thorn) had realized that nonlinear systems, and in particular the systems which are called nowadays chaotic, can behave in a way which is seemingly random. This raised the question how to detect in these cases whether there is an underlying deterministic mechanism. This problem was solved, at least in principle. and its solution was based on this same result of generic observability. combined with a numericaVgeometrical methodology of analysis of large samples of high dimensional vectors which we will discuss in a later section. A treatment of these problems was given in (packard et al., 1980) from an experimental point of view and in (Takens, 1981) from a mathematical point of view. For a more recent and more didactical exposition see (Sauer et al., 1991).
With these ingredients we define the reconstruction maps: Reck : X -+ Rk, which assign to each state X
E X
the vector
(y(x), y(tp(x)), . . . , y(~-l (x))) E Rk.
In other words, Reck assigns to each state x the k successive measurements which would have resulted from tp and y if the system would have started at state x.
Next we mention that both tp and y belong to function spaces in which there is a topology. the so called
427
which is generated by a deterministic system whose state space has dimenion d (or whose state space has a higher dimension, but whose orbits are attracted to ad-dimensional attractor). Then the reconstruction vectors (Yi , " ' , Xi+I-l), which are the images of the points of the orbit, generating the time series, under the reconstruction map Recl , are all on the image, under the reconstruction map, of the state space (respectively attracor); this image has dimension at most d, because, by assumption, the recostruction map is differentiable. So if the embedding dimension I is bigger than d, the reconstruction vectors are concentrated on a subset of RI whose dimension is lower than l. (This holds even without the genericity asssumptions in the reconstruction theorem being satisfied.) This can be detected numerically by estimating the probability pl(£) that two ran~mly chosen reconstruction vectors in dimension I are, within distance £; the quantity pi (£) is called the I-dimensional correlation integral at length scale £ . If the density of the reconstruction vectors in RI is continuous, then pi (£) is propotional to £1 for small £; if these reconstruction vectors are concentrated on a subset of dimension d < I, then pi (£) is proportional to £d for small £.
strong Cl-topology. This means that we have the notion of generic elements of these function spaces: we say that a property P is generic for dynamical systems like (qJ, y) if there is an open and dense subset U of Cl (X, X) X Cl (X, R), such that whenever the dynamical system (Cp, ji) E U , it has property P. This is also expressed by saying that generic qJ and y have property P. Theorem J. For a state space X as above, and for generic qJ : X ~ X and y : X ~ R, the reconstruction maps Reck : X ~ R* are injective for k > 2· dim(X),
A few remarks are here in order: - This notion of genericity was introduced as an analogue of 'valid with probability one' in cases where there is no natural (probability) measure, but where there is a topological structure available. The actual status of genericity as a formalization of 'almost always valid' is rather complicated and dubious, but in the above theorem, this interpretation seems to be justified. This point has not yet been formalized mathematically in a satisfactory way, e.g. see (Sauer et aI., 1991). - From the point of view of the linear theory, it may be somewhat surprizing that we have to require that in the above therem k > 2 · dim(X) . This can be explained as follows . For every x E X it is a generic property for a set of dim(X) functions to form a good coordinate system in a small neighbourhood of x in the sense that there are no two points with the same coordinates. In the linear case, .this implies that these functions form globally a good coordinate system. This is not the case in the nonlinear situation. Stronger, there are examples of 2-dimensional surfaces, e.g. the projective plane, on which it is even not possible to have three smooth functions which form a global coordinate system in the sense that they distinguish any two points. - There are various generalizations and adaptations of this theorem. One can consiter systems with continuous time and one can consider read out functions with values in a higher dimensional vector space. Also one can consider infinite dimension state spaces if the (asymptotic) dynamics takes place on a finite dimensional attractor. See (Sauer et aI., 1991) and the references mentioned in that paper.
For those y, for which the generic conditions on qJ and y in the reconstruction theorem are satisfied, this dimension is independent of y , so that it gives information about the dynamics which is independent of the the way it is observed, e.i. which is independent of y so far as 'y is generic' . The above description of the methodology of detecting deteminism in a time series needs much more details which we cannot give here. For example there are many different notions of dimension, and not for all of them the above statements are correct. For more details see (Kantz and Schreiber, 1997). In general one can say that the test for determinism along these lines is mainly a theoretical possibility: 'real' time series are essentially always contaminated with noise. On the other hand, the analysis of time series in terms of the correlation integrals, as defined above, provides information which cannot be obtained from an the ususal analysis in terms of power spectra and autocorrelations. This information is useful when assessing the predictability of a time series.
4. RECONSTRUCTION AND THE DISCRIMINATION BETWEEN DIFFERENT DYNAMICAL REGIMES
Independent of whether time series are generated by a deterministic system or not. it is often an important question to deceide whether two time series might have been generated by the same mechanism. One can think for example of compairing a measured time series with a time series generated by a mathematical model for the process under consideration. For this problem of discremination one can of course use
3. RECONSTRUCTION AND THE DETECTION OF DETERMINISTIC DYNAMICS
As we mentioned before, the method of reconstruction was introduced as a way to detect whether a given stationary time series was generated by a deterministic system or not. The idea can be explained in a simple manner: suppose we have a time series {Yd
428
power spectra and autocorrelations, but as we have seen, with these we even cannot see the difference between deterministic and stochastic processes. So additional methods are needed.
Aeyels, D. (1981). Generic observability of differentiable systems. SIAM 1. Control and Optimization 19, 595-603. Aeyels, D. (1982). Global observability of MorseSmale vector fields. 1. Diff. Equ. 45, 1-15. Diks, C, WR. van Zwet, F. Takens and 1. DeGoede (1996). Detecting differences between delay vector distributions. Phys. Rev. E 53,2169-2176. Kalman, R.E. (1960). A new approach to linear filtering and prediction problems. Trans. ASME, 1. Basic Eng 82, 35-45. Kantz, H. and T. Schreiber (1997) . Nonlinear time series analysis. number 7 In: Cambridge nonlinear science series. Cambridge University Press. Packard, N.H., 1.P. Crutchfield, 1.D. Farmer and R.S . Shaw (1980). Geometry from time series. Phys. Rev. Letters 45, 712-716. Sauer, T., 1.A. Yorke and M. Casdagli (1991). Embeddology.l. Stat. Phys. 65,579-616. Takens, F. (1981). Detecting strange attractors in turbulence. In: Dynamical systems and turbulence (D. Rand and L.-S Young, Eds.) . Number 898 In: Lecture Notes in Mathematics. Springer-VerJag.
Recently a method was devolloped by Diks et al (Diks et al., 1996) to distinguish time series by detecting possible differences between their reconstruction measures. The I-dimensional reconstruction measure J.LI is the probability measure on RI which corresponds to the density of the I-dimensional reconstruction vectors in RI . (For stationary time series, this can be made into a mathematically rigorous definition, but for nonstationary time series this notion does not make sense.) This is a natural approach for stationary time series. All (dynamical) information about such times series is contained in the reconstruction measures. In many cases one can even restrict to reconstruction measures in relatively low dimensions. This method of Diks is based of a smoothed version of correlation integrals and mixed correlation integrals which we can define as follows . First we note that the usual correlation integral in dimension I at length scale e is the expectation value of Ht(Yi , Yj ) , where we take the average over all pairs of I-dimensional reconstruction vectors (Yi , Yj), and where Ht stands for the function which is 1 if the distance between the arguments is less than e and is zero otherwise. We obtain a smoothed correlation integral Spl(e) if we replace the function Ht by Ft = (27re 2)-1/2 . exp( _d2 /2e 2 ), where d is the Euclidean distance between the two arguments. When considering two time series {Yi} and {zd, with I-dimensional reconstruction vectors {Yd and {Zi}' we define the mixed smoothed correlation integral M Spl(e) as the expectation value of Ft(Yi , Zj) . If there are various time series we indicate the time series to which the definition is applied as subscript, so the above mixed smoothed correlation integral can be denoted by SM pfr,z)(£)' The test for the time series Y = {yd and Z = (zd to be different is based on the fact that the quantity SP~(£) + SI1(£) - 2MSptr,z) (e) is always nonnegative, and it is positive if and only if the [dimensional reconstruction measures of the time series Y and Z are different. This is based on the fact that this expression is the integral, with respect to the Lebesgue measure on RI, of (p~ - p~)2, where p~, respectively p~, is the density of the measure on RI, obtained by convoluting the [-dimensional reconstruction measure of Y, respectively Z, with the Gauss normal distribution with average 0 and variance ~ £2 •
5. REFERENCES Aeyels, D. (1978). Global observability for nonlinear autonomous differential equations. PhD thesis. Wahington University. St. Louis, Missouri.
429