- Email: [email protected]
Drift instabilities and chaos in forecasting and adaptive decision theory
Physica D 72 (1994) 309-323 North-Holland SSD1 0167-2789(93)E0315-3
Drift instabilities and chaos in forecasting and adaptive decision theory * M e l i n d a G o l d e n K u s c h a and B. Erik Y d s t i e b, a Simulation Sciences Inc., 601 South Valencia Ave., Brea, CA 92621, USA b Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh, PA 15213, USA
Received 14 December 1992 Revised manuscript received 17 June 1993 Accepted 11 November 1993 Communicatedby H. Flaschka
Bifurcationtheory showsthat policyadaptation and the rationalexpectationshypothesisof macro-economiescan be used to explain unpredictability,rapid changesin solutionstructure and chaos in decisionproblems with uncertainty. Structural errors lead to catastrophic instability and forecasts become irrelevant. Short evaluation horizon and the application of measures designedto give quick response give multiperiodicityand chaos. Finally,wronginterpretations of the context lead to global bifurcations, laminar drift (complacency) and chaotic bursting. The discrete map representing these dynamicsis of interest in its own right. It is non-invertibleand displaysbifurcationbehaviour not commonly seen in systemsderived from physical considerations.
1. Introduction It is difficult to model and control a complex macro-economic system, like a national economy. Cause and effect relationships are poorly understood. The global context changes and policy implementations made today influence the future direction of the economy and make yesterday's forecasts obsolete. To make rational policy decisions it is necessary to experiment and to adapt the decision mechanisms to the changing global environment. This concept is the basis for the rational expectations hypothesis; a hypothesis which has been met with considerable interest and enthusiasm in the field of econometrics [ 2 ]. The idea, simply stated, is this: "forecasts and models are continually adjusted and updated to * This research was supported by the National Science Foundation (NSF~-'TS-8903160) and the US Department of Energy (DOE# DEFG-0285-ER-13318). l Correspondingauthor.
avoid consistent errors". The application of rational expectations, not only brings forecasts better in line with our experiences, it also improves our decision making procedures and results in more optimal system performance. But, the coupling of forecast adjustment and policy implementation can give instability. The problem is the following. Poor forecasting results in ill adapted policies and large swings in system performance. These swings excitate the system. Cause and effect relationships are then highlighted and the forecasting models can be adapted to the current trends. These adapted models give better policies allowing the system to stabilize and achieve a new steady state. The forecasting errors are now small and mostly determined by secondary effects and the decision mechanism enters a complacent state. Previously successful policies become increasingly ill-adapted. Eventually this leads to destabilization, swings and excitement. The circle is
0167-2789/94/$ 07.00 ~) 1994-Elsevier Science B.V. All fights reserved
310
M.G. Kusch, B.E. Ydstie / Drift instabilities and chaos in forecasting
now complete and the system re-adapts. This interchange of stability and instability displays intermittcncy and chaotic bursting. In adaptive systems there is a conflict between the model builder and the policy maker. The model builder wants to excite the system to gain new experiences. The policymaker attempts to suppress swings and make the future predictable. In an uncertain environment this conflict leads to drifting and chaos. In order to focus the discussion, we investigate a simple, adaptive decision problem which mimics the behaviour described above in a surprisingly accurate manner. A parameter identifier is used for policy adaptation and to improve the forecast. The objective is stabilization. Earlier investigations along the lines discussed here have been performed by Ydstie [ 19 ], Marcels and Bitmead [ 14 ], Praly and Pomet [ 15 ], and Golden and Ydstic [9]. It has recently been shown that adaptive systems may generate deterministic chaos [ 10 ].
y ~uy
Fig. I. A representativeadaptive control system. this paper it was shown that a non-adaptive decision mechanism based on the separation principle converges and that in the limit optimal decisions are taken if the system is linear and the performance objective quadratic. Consider the inflation model [ 13 ]
P(t + 1) = 2 - 1 p ( t ) - ~h(t) + v(t), where the constants and variables are defined so that t = 0, 1,2,...
- the set of positive integers, denoting time instant,
2. An adaptive decision problem A plausible structure for decision theory is shown in fig. 1. There is a system (S) with outputs (y) that needs to be stabilized using instruments (u). The policy maker (P) uses an estimate of the current state (~b) and an abstract model ( M ) to produce forecasts and evaluate alternative policies. After a policy has been chosen and implemented the policymaker will "wait and see" while comparisons are made between forecasts (3) and actual outcomes. Discrepancies (e = y - Y) are used to revise and update the forecasting model. The refined model is used to update the policy. In this model for decision making the tasks of updating the forecast and implementing the policy are separated. This approach, referred to as certainty equivalence control or simply the separation principle, was first discussed in the context of economic decision theory by Simon [ 16 ]. In
+Y
P
- price level,
h
- per capita level of currency,
V
-
a sequence of r a n d o m shocks,
0<7, 0<2<1
- parameters.
The constant 2 - t is a measure of the gross inflation rate. In addition we have the government budget constraint
h(t + 1 ) -
where
1 1 + n h(t)
M.G. Kusch, B.E, Ydstie / Drift instabilities and chaos in forecasting
311
- government financing through currency creation, n
- population growth rate.
By using the budget constraint we can eliminate h (t) from the equation for the price level and we can write it as an auto-regression: P ( t + 2) + a l P ( t + 1) + a2P(t) = bou(t-d)
+ c l v ( t + 1) + C2V(t),
(1)
where d = 2 is a delay and u(t) = (yh(t) + P ( t ) ) ~ ( t ) can be viewed as a control variable (instrument). The control problem consists of choosing ~(t), and hence u(t), so that the price level is stabilized. In the absence of additional constraints we achieve this objective by deficit financing and choose ~ (t) so that E ( P ( t + 2)[~-(t)) = P*, where P* is the desired price level and E (. [Jr (t)) denotes the conditional expectation. If all system parameters are known and v (t) is a martingale difference sequence, then a Kalman filter can be used to estimate the noise component. The separation principle applies and the system achieves asymptotic optimal performance. In an adaptive decision problem some or all of the parameters al, a2, b0, Cl, c2 are estimated and the control is implemented using the estimated model. Goodwin, Ramadge and Caines [ 11 ] showed that if the stochastic approximation algorithm is used to estimate the parameters, then the adaptive algorithm converges in the Cesaro sense. In the limit we get t
lim 1 ~ - ~ E ( ( P ( t - 2) - e*)2lJr(t)) t--*oo t = r
iffil a.s.,
where r corresponds to the minimum achievable using any non-anticipatory control. Chen and
Fig. 2. Interaction with the environment.
Guo [3] have shown the same result with the least squares estimate. In this context the least squares estimate can be viewed as a Kalman filter for parameter estimation. The Kalman fdter produces the optimal or "rational" expectation of future price levels. The aim of this paper is to study the dynamics of the adaptive decision mechanism when it is set in a a wider context. In our example the environment (E) shown in fig. 2 generates two types of perturbations. The exogenous component (v) is independent of actions and changes made by the system S. The reactive component (w) describes how the environment reacts to policy changes in S. We will focus on the stabilization problem. Policy objective. Choose u (t) = f (y (t)) so that ~(t + 1 ) ~- y*, where ~(t + 1 ) is the forecast of y (t + 1 ) and y (t + 1 ) is generated so that y ( t + 1) = a l y ( t ) + b o u ( t - 1) + z(t).
(2)
The following policy will be used: 1
*
u(t) = ~oo(Y -
Oy(t)
-
"£(t)).
(3)
Here ~(t) is the estimate of z, bo is the estimate of bo and 0 is the estimate of - a . In the context equation ( 1 ) we have y (t) = P ( t ) , z (t) = - a 2 y ( t - 1) + c l v ( t ) + c 2 v ( t - 1) a n d d = 1. It is possible to estimate all the parameters and noises in this simple model. This was done in the works of Goodwin, Ramadge and Caines [ 11 ] and Chen and Guo [ 3 ]. The point we want to make is that there will be a residual term that cannot be modelled using a finite dimensional
312
M.G. Kusch, B.E. Ydstie / Drift instabilities and chaos in forecasting
linear (or nonlinear) regression driven by sequentially uncorrelated noise. This residual is captured by z (t) in the model given above. In order to reduce the dimension of the problem we discuss the case (4)
z ( t ) = a 2 y ( t - 1) + v.
We model the environment E so that it contributes a fixed, but unknown bias v and a delayed reaction to policy implementations made by system S. This reaction is metered by the parameter a2. This represents the simplest structure that can be used to study problems of this type. We fix the parameter b0 and use the following recursive algorithm to estimate parameter - a :
Estimation algorithm: Let O(t - 1 ) be the estimate o f - a at time t - 1, then compute O(t) = 0 " ( t - 1) + p y ( t 1)e(t) c + y ( t - 1) 2. Here p > 0 adjusts the rate of learning, c is,an estimate of the noise level and e is the forecast error, i.e.,
e(t) = y ( t ) - O(t - 1 ) y ( t -
1) - ' b u ( t - 1).
I f p = 1 and c = 0 the result is orthogonal projections onto hyperplanes. If c increases, for example c = t, then the result is the stochastic approximation algorithm.
y ~ - O y + v, 0 ~ 0 + Py (-Oy + v) c+y
2
We have re-parameterized so that 0 = kO(t) + am with k = bo/'bo and P = kp. With v = 0 the set
A s = {y = O, 0 = Oo s.t.
IOol <
1}
represents a stable equilibrium and the set A u = {y = 0,0 :
00 s,t. 1001
1}
represents a critical or unstable equilibrium. The stability condition follows from the fact that det(DF1 )0,00 = - 0 0 so 1001 < 1 is necessary for the map to be dissipative and contracting in the neighborhood of A = A s t3 A u. This paper will show that structural perturbations set up a drift along the equilibrium set. This gives rise to a type of behaviour normally associated with saddle nodes. However, nonlinear interaction, due to the estimation, prevents global instability and the solution then oscillates between the stable and the unstable equilibrium set. The solution bifurcates when the environment exerts a constant, but unknown perturbation, v, on the system. The degenerate solution simply disappears and is replaced by a period-two fixed point: Pl = (Yl = 0,01 = 1),
3. The set A
In this section we will show that adaptive forecasting may lead to biassing and optimal performance when the learning rate is low. High learning rates give chaos.
P2 = (Y2 = V, 02 = 1).
The eigenvalues of the linearized mapping DG 2 at the period-two fixed point are given by e v 2 ( v 2 + 2c) ~1,2 = 1 --
Example 1 [15]. With a2 = y* = 0, the dynamics of the adaptive system ( 2 ) - ( 4 ) can be represented by the mapping G1 : R2 --* R2:
2C(V 2 + C)
1 4/ p~)2(~2 + 2c) )2 _ 4. 4-~V(
2C(V2 + c )
M.G. Kusch, B.E. Ydstie / Drift instabilities and chaos in forecasting
With 0 < P < 4c(v 2 + c ) / ( v 2 + 2c)v 2, the period-two f'Lxed point is elliptic, By adding an additional bifurcation parameter (for example the environmental reaction parameter a2) we have hyperbolic attraction. The basin of attraction remains as a2 ~ 0 and it follows that we converge to the period-two solution. Close to the elliptic point the average forecasting error approaches v/2. This is optimal for the type of controller discussed here and only half of that obtained by using the true parameters a and b in the forecast. Stability is ensured by adaptation despite the non-hyperbolic nature of the fixed point. Thus, policy improvement is achieved by biassing the model as predicated by the rational expectations. However, convergence is extremely slow and it is not global. This is because in addition to the period-two attractor there are higher order attractors. When P = 4c(v 2 + c) / (v 2 + 2c)v 2, we obtain 21 = 22 = - 1. In this case a period-doubling bifurcation occurs which results in a hyperbolic period-four fixed point. The process o f period doubling continues as the learning rate increases giving rise to significant deterioration in performance. The main conclusions that can be drawn from this example are that adaptive policies using the separation principle do not necessarily give optimality. High learning rates lead to multiple equilibrium states and bifurcation. Optimal performance, when it is achieved, is achieved by biassing the model. Thus model building and control are in conflict when there are unmodelled perturbations acting on the system. Finally, the stability properties of solutions cannot be deduced from linearization since the equilibrium set, in the absence of perturbations, is a nonhyperbolic set.
4. Complacency and chaotic bursting The map discussed in the previous section has simple limitingbehaviour when the learning rate is low. It follows that this m a p cannot be used
313
to explain how persistent drifting comes about in adaptive systems. In an attempt to account more explicitly for how instabilities and chaos are generated by the environment, we consider the following example. Example 2118]. Let a2 = 0 and y* # 0. We then get the map G2 : R 2 ~ R2: y ~-Oy 040
+ y,
+ py-Oy + 7- 1
#+y2
(5)
where 0 = kO(t) + a l , y = y / y * , P = kp, fl = c/y .2 and ? = k + v/y*. This map, which at first sight appears to be very similar to G1, differs in a number of important respects. First, there is a period-one fixed point: Pl = { Y =
1,0=y-I}.
By noting that d e t ( D F ) = 1 - 7 at the fixed point we see that the map is (locally) dissipative if and only if 0 < y < 2. In the case b = b this gives
I 1<1. This condition is referred to as sufficient excitation and it implies that the signal y* should dominate the perturbation v. This situation does not normally occur in a practical application of adaptive control. The reason for this is that v would normally be a sequence of large shocks with unpredictable magnitude and arrival times and y* is quite small relative to these perturbations. In other words, the sufficient excitation condition is not normally satisfied. The fixed point is locally stable provided [2i[ < 1 where
~.~,2=~
2-7
,8+1
M.G. Kusch, B.E. Ydstie / Drift instabilities and chaos in forecasting
314
~
~ ~.~.I-
supercridcal period-doubling
__..~
Hopf bifurcation limit cycle appears
P (13+1)
Stable Period-One Fixed Point
~ ~
"~'
subcritical period-doubling bifurcation
N
period-two%
® @©
limit point
linear instability
period-two and period-one fixed points coexist
Fig. 3. Stability boundaries for the period-one fixed point in Example 3.2.
~'~
1.8
L~ o
1.4~--
2 e~ •~
1.05
-
~
.6~
-
• ......,
II ,r .,........-""" ....,.."
~ 0
~
.3
-.07~
.,,...
c.._
--
- -
'.28
144
:6
.'76
J.92
~.08
1.24
P [bifurcationparameter] Fig. 4. Bifurcation diagram showing the asymptotic normalized process output y (o¢) as a function of the tunable parameter P for map G2 with {p = 0.25, 7 -- 1.7}. This and the following bifurcation diagrams is a projection of the two-dimensional system on the y - P plane and is obtained by incrementing the bifurcation parameter and perturbing the system from its previous steady-state value. The map is then iterated a minimum of 5 000 times and the final 1000 iterations are plotted at each new value of P.
M.G. Kusch, B.E. Ydstie / Drift instabilities and chaos in forecasting
±
(2-~,
p fl~l
)2 + 4(~,_ 1) )
are the eigenvalues ofDG2 (x) at the fixed point. Figure 3 shows the bifurcation boundaries of the period-one fixed point for different values of the parameters P, fl, and ~. From this figure and the analysis below it becomes clear that there exists only three routes to instability, namely
- Linear drift due to the wrong sign of b. - Period doubling due to high adaptive or control gain. - Hopf and global bifurcations due to environmental bias.
Thus it is natural to use P
and
7
as independent parameters. Choosing the wrong sign for b gives P < 0. This leads to infinite drift of the parameter estimate, overall system instability and renders the forecasts irrelevant. This instability can be corrected by changingthe structural error, thus reversing the sign of b. The high gain instability results from underestimating the magnitude of b (high control gain) or choosing p large (fast adaptation). By relating the discretized mapping to a continuous time system these problems can be seen as stemming from short evaluation horizon and a desire for quick results. To investigate this type of instability it is convenient to use P as a bifurcation parameter. Starting out with a stable fixed point and following line A in fig. 3 we get the bifurcation diagram shown in fig. 4. When we increase P, we lose stability by subcritical perioddoubling bifurcation at P = 0.75 as we cross the hypothenus
P fl+l
-
2(2
-
315
y).
The bifurcation diagram contains a hysteresis loop as indicated in fig. 4. Over a certain range of P there are two stable solutions. The location of the unstable period-two fixed point was calculated using AUTO [4 ] and is given by the dashed line. Thus adaptive policies have co-existing solutions and their limiting behaviour may depend on the choice of initial conditions. The region where hysteresis is located is indicated in fig. 3. The change from subcritical to supercritical at P/(fl + 1 ) = 1.176 and ~ = 1.412. A further increase in P leads to a second period doubling at P = 1.02. The process of period doubling continues until, at some value of P, the map G2 becomes "chaotic". Eventually at P = 2 the solutions are unbounded. A further discussion is given by Golden and Ydstie [9 ]. This example shows that desire to rapidly reach equilibrium or to rapidly change policy is counterproductive and leads to multi-periodicity and chaos. The fixed point undergoes a supercritical Hopf bifurcation along the line 7 = 0, forming an invariant orbit topologically similar to a circle. An unstable invariant circle also forms and these meet at a saddle-node bifurcation causing both to vanish. Figure 5 (corresponding to point B in fig. 3) shows both invariant circles for 7 = - 0 . 0 4 , P = 0.8,fl = 1.2. The unstable invariant circle also vanishes at ~, = 0 where it collides with the degenerate solution A2 = {y = 0,0 = 00}. The criterion [00l < 1 is necessary for this solution to be locally stable and attracting. It follows that the dynamics associated with the stable period-one fixed point can only be followed over a limited range of parameters. The set A2 corresponds to the set A described in Example 1 and, analogous to the previous example, this set can be viewed as a bifurcation of a period-two point which is established for ~, < 0. The significance
316
M.G. Kusch, B.E. Ydstie /Drift instabilities and chaos in forecasting
-0.4-
'~
e~
-0.8-
-1.2-1.6-
o
-2
0
//
//
•
[l ~
unstablefixedpoint ~../ ((1 11 04) , - 1 . 0~ 4/ ..... ~ ~"/
~ 015
unstableinvariant circle
,,-'
;
115
2
215
~1/ (normalized process output)
Fig. 5. Stable and unstable invariant circles following the Hopf bifurcation. The unstable invariant circle is constructed by iterating the map G2 backwards.
of the two solutions, which co-exist for a limited range of 7, can be explained as follows. The desired fixed point (y = 1 ) results from the forecast error going to zero, whereas the undesired fixed point (y = 0) results from the adaptation mechanism being turned off. The latter leads to parameter drift. Other periodic attractors can also be observed on and near the line 7 = 0. For P > 1 and fl = P - 1 a period-three fixed point (y, 0) : {(1,1) ~ ( - 1 , - 1 ) ~ ( - 1 , 1 ) --. (1,1)} is attracting for some initial conditions. Figure 6 shows the basins of attraction for P = 1.5 and fl = 0.5 (point C in fig. 3). The area colored gray denotes the set of initial conditions (y, 0) which are mapped onto the line, y = 0. The area colored white denotes the set of initial conditions which are mapped onto the fixed point ( 1, - 1 ). Finally the area colored black denotes the set of initial conditions which are mapped onto the period-three attractor. A more detailed study can be found in Frouzakis et al. [ 7 ]. By continuing the period-three fixed point using AUTO, we determine that it comes from a saddle-node bifurcation and it exists over a significant range of the parameters P, and ?. The evolution of higher-order attractors from
Fig. 6. Basins of attraction constructed by simulation.
both fixed points is shown in fig. 7 for {P = 0.8, fl = 1.2} as 7 is varied along line D in fig. 3. Here it is seen that the stable invariant circle, which bifurcates from the fixed point ( 1 , - 1 ), eventually vanishes due to a saddle-node bifurcation of invariant circles. The stable period-two fixed point which terminates at the point (0,1) bifurcates into a pair of stable invariant circles. This type of bifurcation is typical of what can be observed in more complex adaptive mechanisms. As 7 is decreased, the invariant circles undergo frequency-locking causing period-n attractors to appear on the invariant circles. The phenomenon of frequency-locking occurs in discrete systems when eigenvalues cross the unit circle at the nth root of unity. Regions in the parameter space, where locking occurs, are referred to as Arnold tongues [1 ]. A description of this phenomenon in the context of adaptive control is given by Frouzakis et al. [7]. As the parameter 7 is further decreased the adaptive
M.G. Kusch, B.E. Ydstie / Drift instabilities and chaos in forecasting
317
2.5 O
o
1.5
0.
•-~
0.5
"~ 0
-.5
[
~
-1.5
~-
-2.5
Hc
Cos
-11
-',s
-
'2
y [bifurcation parameter] Fig. 7. Bifurcation diagram constructed by simulation. The parameter ~ represents the noise to signal ratio.
system alternates between quasiperiodic (unlocked) and periodic (locked) states as it traverses these regions and the periodic attractor undergoes a series of simple period-doubling and saddle-node bifurcations similar to those in the logistic map. An understanding of this process can be obtained from a magnified picture showing the intersecting stable (W s (B)) and unstable ( W u (B)) manifolds of two neighboring saddles of a period-28 f'Lxed point (fig. 8). The period-28 point gives rise to frequency locking while the manifold crossings create a homoclinic tangle. This affects the transients to the attractor to a considerable extent and continues to do so after the period-28 point loses its stability. In particular, we note the presence of a hyperbolic fixed point (B) and a (transversal) intersection of the respective manifolds W s (B) and WU(B) at the point (A). Since A # B it follows that G2 has a hyperbolic invariant limit set. The presence of this structure, known as a horseshoe, indicates the existence of a strange hyperbolic attractor. The dynamics associated with such structures are in general remarkably stable, albeit complicated [ 17 ]. As ~ is further decreased the saddles and nodes collide and disappear. What remains is the com-
plex geometric structure which forms the chaotic attractor as shown in fig. 9 (7 = -0.2). This phenomenon has been extensively investigated in the literature [8,5 ]. It is clear that the dynamics in the drift and burst phase are controlled by the period-two fixed point observed in Example 1 and not by the period-one fixed point. From this we conclude that linearization around the fixed point is irrelevant. Figure 10 shows the time series of the output and the parameter estimate when the system is chaotic. During quiescent periods the system is stabilized and the output and the parameter estimate drift slowly in response to the biassing influence of the environment. The forecast eventually becomes irrelevant. Eventually, a burst in the output supplies excitation and large forecast errors. This causes the parameter estimate to readapt to what appears to be a "good" estimate. This process, which represents the problem described in the introduction, can be duplicated with great regularity in experimental studies.
5. Persistent instability and conflicts In this section we show that adaptive decision making processes are destabilized when there is
M.G. Kasch, B.E. Ydstie / Drift instabilities and chaos in forecasting
318
i
• unstoble perlod-2 • stcble period-28
i
~
o
~
i 1.1
c
~L .
0.6
o
0.1
-1.5
-1.0
-0.5
V (normalized process output) Fig. 8. E n l a r g e m e n t o f t h e s t a b l e a n d u n s t a b l e m a n i f o l d s o f a p e r i o d - 2 8 p o i n t .
chaotic ohase
2.375
.-
1.75 e~
.~
1 .I 25
-.125
-'3
4.75
-'.5
'.75
V [normalizedprocess output] Fig. 9. Simulation showing the phase plane of an adaptive system undergoing parameter drift and output bursting. u n m o d e l l e d interaction with surrounding processes. T h e m e c h a n i s m adapts, h o w e v e r , multiperiodicity and c h a o s yield lower performance.
y --~-Oy - a2x + "0, 0~0
+
py-Oy
- a2x + v c+y2
Example 3 [18]. With a2 ~ 0 and y* = 0) w e h a v e the m a p G3 : R 3 ~ R3:
where 0 = kO(t) + al and P = kp. T h e set
x---~y,
A~ = {x = O,y = 0 , 0 = Oo s.t. IOo[ < 1}
M.G. Kusch, B.E. Ydatie / Drift instabilities and chaos in forecasting
319
2.5
-i o
1.5
¢¢J
8 "d -.5
--
- ! .~5 - -
"output bursting"
'o
%0
~oo
~oo
koo
koo
,~oo
sampling interval
2.3
--
t~
~.s-
0.7
•.~
-
-o.1
-
-0.9
--
"parameter drift"
'o
',oo
~,oo sampling interval
Fig. 10. The top figure shows the process output while the bottom figure shows the estimated parameter during parameter drift and bursting.
represents a stable equilibrium when v = 0. There is an obvious connection between this and the set ,4. The criterion 100[ < 1 is necessary if the map is to be dissipative and contracting in the neighborhood of ,43. Similar to the map G~, there are no periodone fixed points when v # 0. However, there is a period-two fixed point which is given by #l
#1 A period-I fixed point can be introduced by setting y* ~ 0. It was shown, using map G2 that this point does not control the dynamics in the region of interest so this idea is not pursued here.
P l = (Xl = O, Yl = v / ( 1
+ a 2 ) , O l = 1 -I- a 2 ) ,
P2 = (Y2 = O, x2 --~ v / ( 1
..I- a 2 ) , 0 2 = 1 -I- a 2 ) .
We have det (DG3) = a2. A mapping with a constant, non-zero determinant is a global diffemorphism. The map is (globally) dissipative if and only if ]a2[ < 1. This sets an upper bound for the amount of unmodelled feedback that can be tolerated from the global context. In other words, one cannot stabilize a system if the external environment dominates. In such situations it is necessary to weaken the influence of the global con-
M.G. Kusch, B.E. Ydstie / Drift instabilities and chaos in forecasting
320
.75
t~
025
'15
a 2 [bifurcationparameter] Fig. I 1. Bifurcation diagram constructed by simulation. text by decoupling and isolation through the application of policies aimed towards neutralizing the effect of external influences. In the extreme, barriers can be introduced to negate feedback once the source has been identified. Such barriers are effective until countermeasures are taken. Countermeasures, of course, introduce feedback bringing us back to where we started. The condition for local stability of the periodtwo fixed point is
- 1 < a2 < O,
O
4c(1 + a2)3('v 2 + c(1 + a2) 2) v2(v 2 + 2 c ( 1 + a 2 ) 2)
A Hopfbifurcation occurs along the line a2 = 0. This corresponds, not coincidentally, to the first asymptotic solution of the map GI, a stable elliptic period-two fixed point. The period-two fixed point is hyberbolic and locally stable if - 1 < a2 < 0. In this region, we obtain a forecasting error of v / ( 2 + 2a2). At a2 = 0, corresponding to map GI, the attractor undergoes a Hopfbifurcation and two invariant circles form, one about each elliptic point of the period two fixed point at a2 = 0. This phenomenon is similar to the behavior observed for map G2 as the external per-
turbations become increasingly important. Figure 11 shows a bifurcation diagram which illustrates the asymptotic behavior of this map as a2 is increased. The sharp transitions from periodic behavior to chaotic behavior are indicative of global bifurcations as saddles and nodes of the periodic attractor collide and vanish. Complex tangles of connected stable and unstable manifolds remain and can be analyzed using the same techniques as those used in the study of map G2. The pathway to chaos is similar even though G3 is three dimensional. One way to explain this is that the phase space contracts and becomes two dimensional when la21 < 1. One significant point to notice is that although the performance of the adaptive system deteriorates quickly as a2 is increased from 0 to 0.025, no significant decrease in performance is noticed as the external feedback increases. The dynamics are similar to those observed in mapping G2. It follows that even very small unmodelled feedback from the environment may lead to significant biassing of the policy. As a2 is increased further, the laminar region of the attractor shrinks indicating, in a loose sense, a stronger attractiveness of the line y* = 0. Figure 12 shows a comparison of the time
M.G. Kusch, B.E. Ydstie / Drift instabilities and chaos in forecasting
321
2.4
.~
1.9
i
-
1.4 --
.~
~
0.9
-
0.4
-
-0.1 -
y (process output)
2.4
0 (parameter estimate) 1.775-
1.15
0.525
. .................
s _ta_bJ]_iv_]_h9.it . . . . . . . . .
-
0.5
--
-0.125
-
y (process output) it,_,,,lildk,,~..,ilil~i,...,dlLh ...... dihl..h.iL.,,,,llh~,illL~illh,dllL,,ih,...,litla ......... .......
o
,i'"'"",''"
~ r ~ , .......
;bo
, ~ _
~l~'"'"""r""~"""~ ~lrr~ ......... ~ffrr,''-'~
2bo
3~o
:~::":'
4~o
sampling interval
Fig. 12. Parameter drift and output bursting due to the addition of unmodelled dynamics a2 = 0.85. Note that trajectories are more irregular but that the performance of the adaptive system has not changed significantly.
322
M.G. Kasch, B.E. Ydstie /Drift instabilities and chaos in forecasting
evolution plots of 0 and y with the phase plane plot of such a system. Now the model drifts slowly, but randomly, near, and typically above, the stability limit of the parameter estimate of the period-two fixed point, 0 = 1. The process output has become very erratic. In fact, the local feedback process is unstable most of the time. The main objective of policy adaptation is to "play catch up" with the external environment and this leads to biassing and a conflict. This type of instability, which is qualitatively similar to the dynamical behaviour of some macroeconomic systems, can bc explained using the rational expectations hypothesis and bchaviour close to saddle points [2]. However, the arguments presented here are based on linear theory and are incomplete since they do not explain how the system recovers. We maintain that the system quickly recovers since it adapts. In a more complex system there is no reason to believe that the system state returns to the state it previously occupied. An example where the adaptive system, after a burst, returns to a different state has been reported by Huberman [12]. Small parameter changes, unstructured perturbations and stochastic noise have little influence on the dynamics of these systems. If we consider a2 to be the coefficient resulting from the discrctization of a second order, stable, continuous-time process, then a2 = e - d h , where d > 0 and h is the sampling period. From this relationship it is obvious that by increasing the sampling period, we can reduce the degree of complexity of the drift. In other words, the coupling can be made less important by extending the forecasting horizon.
6. Conclusions In this paper we have shown that unpredictability and randomness in adaptive decision theory is generated by a conflict between forecasting and policy making. This conflict leads to intermittcncy observed as complacent drift and
chaotic bursting originating from a degenerate set A. In one case the degenerate set collapses to an elliptic period-two point and drift originates from a Hopf bifurcation of this. In the other case the degenerate set collapses to a period-one point and we observe global bifurcations from this. The evolution of these distinct attractors has been traced and it has been shown that they coalesce in a saddle node bifurcation. It is the period-two point which continues to control the dynamic behaviour of the adaptive policy. The presence of a horseshoc has been determined and shows that the results we obtain are structurally stable and reproducible in simulations and experiments. The analysis of these simple, low-order systems provides a new point of view for the understanding of how chaos and unpredictability can be generated in adaptive decision processes. The analysis may also be used to explain origins and evolution of complacency and chaotic bursting in macro-economic models.
Acknowledgements The authors aprcciate numerous helpful discussions with Professor Ioannis Kevrikidis, Dr Ray Adomaitis and Dr Christos Frouzakis of the Department of Chemical Engineering at Princeton University. W c found the use of their computer package, S C I G M A , invaluable in showing the existence of the hyperbolic scts.
References [I] V.I. Arnold, Geometrical Methods in the Theory of Ordinary DifferentialEquations (Springer,N e w York, 1983). [2] D.K.H. Beg8, The Rational Expectations Revolution in Macro-Economics (The Johns Hopkins Univ. Press, Baltimore, M D , 1982). [3]L. Guo and H.F. Chen, Revisit to ~,stremWittenmark's self-tuning regulator and ELS based adaptive trackers, IEEE Trans. Auto. Control. 36 (1991) 802-812.
M.G. Kasch, B.E. Ydstie / Drift instabilities and chaos in forecasting [4]E. Doedel, AUTO: Software for Continuation and Bifurcation Problems in Ordinary Differential Equations (California Institute of Technology, 1986). [5]R.E. Ecke and I.G. Kevrekidis, Interactions of resonances and global bifurcations in RayleighBernard convection, Phys. Lett. A 131 (1988). [6] C.E. Frouzakis, R.A. Adomaitis and I.G. Kevrikidis, Resonance phenomena in adaptively controlled system, Int. J. Bifur. Chaos 1 (1990) 83-106. [7] C.E. Frouzakis, R.A. Adomaitis, I.G. Kevrikidis, M.P. Golden and B.E. Ydstie, The structure of basin boundaries in a simple adaptive control system, in: Chaotic Dynamics: Theory and Practice, T. Bountis, ed. (Plenum, New York, 1992) pp. 195-210. [8] L. Glass and R. Perez, Fine structure of phase locking, Phys. Rev. Lett. 48 (1982) 1772-1775. [9] M.P. Golden and B.E. Ydstie, Bifurcation in model reference adaptive control systems, Syst. Control Lett. 11 (1988) 413-430. [10]M.P. Golden and B.E. Ydstie, Small amplitude chaos and ergodicity in adaptive control systems, Automatica 28 (1992) 11-25. [11] G.C. Goodwin, P.J. Ramadge and P.E. Caines, Discrete time stochastic adaptive control, SIAM J. Control Optimiz. 19 (1981) 829-853.
323
[12] B.A. Huberman, Controlling Chaos, Paper presented at Annual Meeting AIChE, Los Angeles, CA, (November, 1991 ). [ 13] A. Marcet and T.J. Sargent, Least squares learning and the dynamics of hyperinflation, in: Economic Complexity: Chaos, Sunspots, Bubbles and Nonlinearity (Cambridge Univ. Press, Cambridge, 1989) pp. 119-137. [14]I.M.Y. Mareels and R.R. Bitmead, Non-linear dynamics in adaptive control: chaotic and Paeriodic stabilization, Automatica 22 (1986) 641-655. [ 15 ] L. Praly and M. Pomet, Periodic Solutions in Adaptive Systems: The Regular Case, IFAC Xth Triennal World Congress (Munich, FRG, 1987) Vol. 10, pp. 40-44. [ 16 ] H.A. Simon, Dynamic programming under uncertainty with quadratic criterion function, Econometrica 24 (1956) 74. [17] M. Shub, Global Stability of Dynamical Systems (Springer, New York, 1987). [18] B.E. Ydstie and M.P. Golden, Chaos and Strange Attractors in Adaptive Control Systems, IFAC Xth Triennal World Congress (Munich, FRG, 1987) Vol. 10, pp. 127-132. [ 19] B.E. Ydstie, Bifurcation and Complex Dynamics in Adaptive Control Systems, Proc. 25th IEEE-CDC (Athens, Greece, December, 1986), pp. 2232-2236.
Drift instabilities and chaos in forecasting and adaptive decision theory * M e l i n d a G o l d e n K u s c h a and B. Erik Y d s t i e b, a Simulation Sciences Inc., 601 South Valencia Ave., Brea, CA 92621, USA b Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh, PA 15213, USA
Received 14 December 1992 Revised manuscript received 17 June 1993 Accepted 11 November 1993 Communicatedby H. Flaschka
Bifurcationtheory showsthat policyadaptation and the rationalexpectationshypothesisof macro-economiescan be used to explain unpredictability,rapid changesin solutionstructure and chaos in decisionproblems with uncertainty. Structural errors lead to catastrophic instability and forecasts become irrelevant. Short evaluation horizon and the application of measures designedto give quick response give multiperiodicityand chaos. Finally,wronginterpretations of the context lead to global bifurcations, laminar drift (complacency) and chaotic bursting. The discrete map representing these dynamicsis of interest in its own right. It is non-invertibleand displaysbifurcationbehaviour not commonly seen in systemsderived from physical considerations.
1. Introduction It is difficult to model and control a complex macro-economic system, like a national economy. Cause and effect relationships are poorly understood. The global context changes and policy implementations made today influence the future direction of the economy and make yesterday's forecasts obsolete. To make rational policy decisions it is necessary to experiment and to adapt the decision mechanisms to the changing global environment. This concept is the basis for the rational expectations hypothesis; a hypothesis which has been met with considerable interest and enthusiasm in the field of econometrics [ 2 ]. The idea, simply stated, is this: "forecasts and models are continually adjusted and updated to * This research was supported by the National Science Foundation (NSF~-'TS-8903160) and the US Department of Energy (DOE# DEFG-0285-ER-13318). l Correspondingauthor.
avoid consistent errors". The application of rational expectations, not only brings forecasts better in line with our experiences, it also improves our decision making procedures and results in more optimal system performance. But, the coupling of forecast adjustment and policy implementation can give instability. The problem is the following. Poor forecasting results in ill adapted policies and large swings in system performance. These swings excitate the system. Cause and effect relationships are then highlighted and the forecasting models can be adapted to the current trends. These adapted models give better policies allowing the system to stabilize and achieve a new steady state. The forecasting errors are now small and mostly determined by secondary effects and the decision mechanism enters a complacent state. Previously successful policies become increasingly ill-adapted. Eventually this leads to destabilization, swings and excitement. The circle is
0167-2789/94/$ 07.00 ~) 1994-Elsevier Science B.V. All fights reserved
310
M.G. Kusch, B.E. Ydstie / Drift instabilities and chaos in forecasting
now complete and the system re-adapts. This interchange of stability and instability displays intermittcncy and chaotic bursting. In adaptive systems there is a conflict between the model builder and the policy maker. The model builder wants to excite the system to gain new experiences. The policymaker attempts to suppress swings and make the future predictable. In an uncertain environment this conflict leads to drifting and chaos. In order to focus the discussion, we investigate a simple, adaptive decision problem which mimics the behaviour described above in a surprisingly accurate manner. A parameter identifier is used for policy adaptation and to improve the forecast. The objective is stabilization. Earlier investigations along the lines discussed here have been performed by Ydstie [ 19 ], Marcels and Bitmead [ 14 ], Praly and Pomet [ 15 ], and Golden and Ydstic [9]. It has recently been shown that adaptive systems may generate deterministic chaos [ 10 ].
y ~uy
Fig. I. A representativeadaptive control system. this paper it was shown that a non-adaptive decision mechanism based on the separation principle converges and that in the limit optimal decisions are taken if the system is linear and the performance objective quadratic. Consider the inflation model [ 13 ]
P(t + 1) = 2 - 1 p ( t ) - ~h(t) + v(t), where the constants and variables are defined so that t = 0, 1,2,...
- the set of positive integers, denoting time instant,
2. An adaptive decision problem A plausible structure for decision theory is shown in fig. 1. There is a system (S) with outputs (y) that needs to be stabilized using instruments (u). The policy maker (P) uses an estimate of the current state (~b) and an abstract model ( M ) to produce forecasts and evaluate alternative policies. After a policy has been chosen and implemented the policymaker will "wait and see" while comparisons are made between forecasts (3) and actual outcomes. Discrepancies (e = y - Y) are used to revise and update the forecasting model. The refined model is used to update the policy. In this model for decision making the tasks of updating the forecast and implementing the policy are separated. This approach, referred to as certainty equivalence control or simply the separation principle, was first discussed in the context of economic decision theory by Simon [ 16 ]. In
+Y
P
- price level,
h
- per capita level of currency,
V
-
a sequence of r a n d o m shocks,
0<7, 0<2<1
- parameters.
The constant 2 - t is a measure of the gross inflation rate. In addition we have the government budget constraint
h(t + 1 ) -
where
1 1 + n h(t)
M.G. Kusch, B.E, Ydstie / Drift instabilities and chaos in forecasting
311
- government financing through currency creation, n
- population growth rate.
By using the budget constraint we can eliminate h (t) from the equation for the price level and we can write it as an auto-regression: P ( t + 2) + a l P ( t + 1) + a2P(t) = bou(t-d)
+ c l v ( t + 1) + C2V(t),
(1)
where d = 2 is a delay and u(t) = (yh(t) + P ( t ) ) ~ ( t ) can be viewed as a control variable (instrument). The control problem consists of choosing ~(t), and hence u(t), so that the price level is stabilized. In the absence of additional constraints we achieve this objective by deficit financing and choose ~ (t) so that E ( P ( t + 2)[~-(t)) = P*, where P* is the desired price level and E (. [Jr (t)) denotes the conditional expectation. If all system parameters are known and v (t) is a martingale difference sequence, then a Kalman filter can be used to estimate the noise component. The separation principle applies and the system achieves asymptotic optimal performance. In an adaptive decision problem some or all of the parameters al, a2, b0, Cl, c2 are estimated and the control is implemented using the estimated model. Goodwin, Ramadge and Caines [ 11 ] showed that if the stochastic approximation algorithm is used to estimate the parameters, then the adaptive algorithm converges in the Cesaro sense. In the limit we get t
lim 1 ~ - ~ E ( ( P ( t - 2) - e*)2lJr(t)) t--*oo t = r
iffil a.s.,
where r corresponds to the minimum achievable using any non-anticipatory control. Chen and
Fig. 2. Interaction with the environment.
Guo [3] have shown the same result with the least squares estimate. In this context the least squares estimate can be viewed as a Kalman filter for parameter estimation. The Kalman fdter produces the optimal or "rational" expectation of future price levels. The aim of this paper is to study the dynamics of the adaptive decision mechanism when it is set in a a wider context. In our example the environment (E) shown in fig. 2 generates two types of perturbations. The exogenous component (v) is independent of actions and changes made by the system S. The reactive component (w) describes how the environment reacts to policy changes in S. We will focus on the stabilization problem. Policy objective. Choose u (t) = f (y (t)) so that ~(t + 1 ) ~- y*, where ~(t + 1 ) is the forecast of y (t + 1 ) and y (t + 1 ) is generated so that y ( t + 1) = a l y ( t ) + b o u ( t - 1) + z(t).
(2)
The following policy will be used: 1
*
u(t) = ~oo(Y -
Oy(t)
-
"£(t)).
(3)
Here ~(t) is the estimate of z, bo is the estimate of bo and 0 is the estimate of - a . In the context equation ( 1 ) we have y (t) = P ( t ) , z (t) = - a 2 y ( t - 1) + c l v ( t ) + c 2 v ( t - 1) a n d d = 1. It is possible to estimate all the parameters and noises in this simple model. This was done in the works of Goodwin, Ramadge and Caines [ 11 ] and Chen and Guo [ 3 ]. The point we want to make is that there will be a residual term that cannot be modelled using a finite dimensional
312
M.G. Kusch, B.E. Ydstie / Drift instabilities and chaos in forecasting
linear (or nonlinear) regression driven by sequentially uncorrelated noise. This residual is captured by z (t) in the model given above. In order to reduce the dimension of the problem we discuss the case (4)
z ( t ) = a 2 y ( t - 1) + v.
We model the environment E so that it contributes a fixed, but unknown bias v and a delayed reaction to policy implementations made by system S. This reaction is metered by the parameter a2. This represents the simplest structure that can be used to study problems of this type. We fix the parameter b0 and use the following recursive algorithm to estimate parameter - a :
Estimation algorithm: Let O(t - 1 ) be the estimate o f - a at time t - 1, then compute O(t) = 0 " ( t - 1) + p y ( t 1)e(t) c + y ( t - 1) 2. Here p > 0 adjusts the rate of learning, c is,an estimate of the noise level and e is the forecast error, i.e.,
e(t) = y ( t ) - O(t - 1 ) y ( t -
1) - ' b u ( t - 1).
I f p = 1 and c = 0 the result is orthogonal projections onto hyperplanes. If c increases, for example c = t, then the result is the stochastic approximation algorithm.
y ~ - O y + v, 0 ~ 0 + Py (-Oy + v) c+y
2
We have re-parameterized so that 0 = kO(t) + am with k = bo/'bo and P = kp. With v = 0 the set
A s = {y = O, 0 = Oo s.t.
IOol <
1}
represents a stable equilibrium and the set A u = {y = 0,0 :
00 s,t. 1001
1}
represents a critical or unstable equilibrium. The stability condition follows from the fact that det(DF1 )0,00 = - 0 0 so 1001 < 1 is necessary for the map to be dissipative and contracting in the neighborhood of A = A s t3 A u. This paper will show that structural perturbations set up a drift along the equilibrium set. This gives rise to a type of behaviour normally associated with saddle nodes. However, nonlinear interaction, due to the estimation, prevents global instability and the solution then oscillates between the stable and the unstable equilibrium set. The solution bifurcates when the environment exerts a constant, but unknown perturbation, v, on the system. The degenerate solution simply disappears and is replaced by a period-two fixed point: Pl = (Yl = 0,01 = 1),
3. The set A
In this section we will show that adaptive forecasting may lead to biassing and optimal performance when the learning rate is low. High learning rates give chaos.
P2 = (Y2 = V, 02 = 1).
The eigenvalues of the linearized mapping DG 2 at the period-two fixed point are given by e v 2 ( v 2 + 2c) ~1,2 = 1 --
Example 1 [15]. With a2 = y* = 0, the dynamics of the adaptive system ( 2 ) - ( 4 ) can be represented by the mapping G1 : R2 --* R2:
2C(V 2 + C)
1 4/ p~)2(~2 + 2c) )2 _ 4. 4-~V(
2C(V2 + c )
M.G. Kusch, B.E. Ydstie / Drift instabilities and chaos in forecasting
With 0 < P < 4c(v 2 + c ) / ( v 2 + 2c)v 2, the period-two f'Lxed point is elliptic, By adding an additional bifurcation parameter (for example the environmental reaction parameter a2) we have hyperbolic attraction. The basin of attraction remains as a2 ~ 0 and it follows that we converge to the period-two solution. Close to the elliptic point the average forecasting error approaches v/2. This is optimal for the type of controller discussed here and only half of that obtained by using the true parameters a and b in the forecast. Stability is ensured by adaptation despite the non-hyperbolic nature of the fixed point. Thus, policy improvement is achieved by biassing the model as predicated by the rational expectations. However, convergence is extremely slow and it is not global. This is because in addition to the period-two attractor there are higher order attractors. When P = 4c(v 2 + c) / (v 2 + 2c)v 2, we obtain 21 = 22 = - 1. In this case a period-doubling bifurcation occurs which results in a hyperbolic period-four fixed point. The process o f period doubling continues as the learning rate increases giving rise to significant deterioration in performance. The main conclusions that can be drawn from this example are that adaptive policies using the separation principle do not necessarily give optimality. High learning rates lead to multiple equilibrium states and bifurcation. Optimal performance, when it is achieved, is achieved by biassing the model. Thus model building and control are in conflict when there are unmodelled perturbations acting on the system. Finally, the stability properties of solutions cannot be deduced from linearization since the equilibrium set, in the absence of perturbations, is a nonhyperbolic set.
4. Complacency and chaotic bursting The map discussed in the previous section has simple limitingbehaviour when the learning rate is low. It follows that this m a p cannot be used
313
to explain how persistent drifting comes about in adaptive systems. In an attempt to account more explicitly for how instabilities and chaos are generated by the environment, we consider the following example. Example 2118]. Let a2 = 0 and y* # 0. We then get the map G2 : R 2 ~ R2: y ~-Oy 040
+ y,
+ py-Oy + 7- 1
#+y2
(5)
where 0 = kO(t) + a l , y = y / y * , P = kp, fl = c/y .2 and ? = k + v/y*. This map, which at first sight appears to be very similar to G1, differs in a number of important respects. First, there is a period-one fixed point: Pl = { Y =
1,0=y-I}.
By noting that d e t ( D F ) = 1 - 7 at the fixed point we see that the map is (locally) dissipative if and only if 0 < y < 2. In the case b = b this gives
I 1<1. This condition is referred to as sufficient excitation and it implies that the signal y* should dominate the perturbation v. This situation does not normally occur in a practical application of adaptive control. The reason for this is that v would normally be a sequence of large shocks with unpredictable magnitude and arrival times and y* is quite small relative to these perturbations. In other words, the sufficient excitation condition is not normally satisfied. The fixed point is locally stable provided [2i[ < 1 where
~.~,2=~
2-7
,8+1
M.G. Kusch, B.E. Ydstie / Drift instabilities and chaos in forecasting
314
~
~ ~.~.I-
supercridcal period-doubling
__..~
Hopf bifurcation limit cycle appears
P (13+1)
Stable Period-One Fixed Point
~ ~
"~'
subcritical period-doubling bifurcation
N
period-two%
® @©
limit point
linear instability
period-two and period-one fixed points coexist
Fig. 3. Stability boundaries for the period-one fixed point in Example 3.2.
~'~
1.8
L~ o
1.4~--
2 e~ •~
1.05
-
~
.6~
-
• ......,
II ,r .,........-""" ....,.."
~ 0
~
.3
-.07~
.,,...
c.._
--
- -
'.28
144
:6
.'76
J.92
~.08
1.24
P [bifurcationparameter] Fig. 4. Bifurcation diagram showing the asymptotic normalized process output y (o¢) as a function of the tunable parameter P for map G2 with {p = 0.25, 7 -- 1.7}. This and the following bifurcation diagrams is a projection of the two-dimensional system on the y - P plane and is obtained by incrementing the bifurcation parameter and perturbing the system from its previous steady-state value. The map is then iterated a minimum of 5 000 times and the final 1000 iterations are plotted at each new value of P.
M.G. Kusch, B.E. Ydstie / Drift instabilities and chaos in forecasting
±
(2-~,
p fl~l
)2 + 4(~,_ 1) )
are the eigenvalues ofDG2 (x) at the fixed point. Figure 3 shows the bifurcation boundaries of the period-one fixed point for different values of the parameters P, fl, and ~. From this figure and the analysis below it becomes clear that there exists only three routes to instability, namely
- Linear drift due to the wrong sign of b. - Period doubling due to high adaptive or control gain. - Hopf and global bifurcations due to environmental bias.
Thus it is natural to use P
and
7
as independent parameters. Choosing the wrong sign for b gives P < 0. This leads to infinite drift of the parameter estimate, overall system instability and renders the forecasts irrelevant. This instability can be corrected by changingthe structural error, thus reversing the sign of b. The high gain instability results from underestimating the magnitude of b (high control gain) or choosing p large (fast adaptation). By relating the discretized mapping to a continuous time system these problems can be seen as stemming from short evaluation horizon and a desire for quick results. To investigate this type of instability it is convenient to use P as a bifurcation parameter. Starting out with a stable fixed point and following line A in fig. 3 we get the bifurcation diagram shown in fig. 4. When we increase P, we lose stability by subcritical perioddoubling bifurcation at P = 0.75 as we cross the hypothenus
P fl+l
-
2(2
-
315
y).
The bifurcation diagram contains a hysteresis loop as indicated in fig. 4. Over a certain range of P there are two stable solutions. The location of the unstable period-two fixed point was calculated using AUTO [4 ] and is given by the dashed line. Thus adaptive policies have co-existing solutions and their limiting behaviour may depend on the choice of initial conditions. The region where hysteresis is located is indicated in fig. 3. The change from subcritical to supercritical at P/(fl + 1 ) = 1.176 and ~ = 1.412. A further increase in P leads to a second period doubling at P = 1.02. The process of period doubling continues until, at some value of P, the map G2 becomes "chaotic". Eventually at P = 2 the solutions are unbounded. A further discussion is given by Golden and Ydstie [9 ]. This example shows that desire to rapidly reach equilibrium or to rapidly change policy is counterproductive and leads to multi-periodicity and chaos. The fixed point undergoes a supercritical Hopf bifurcation along the line 7 = 0, forming an invariant orbit topologically similar to a circle. An unstable invariant circle also forms and these meet at a saddle-node bifurcation causing both to vanish. Figure 5 (corresponding to point B in fig. 3) shows both invariant circles for 7 = - 0 . 0 4 , P = 0.8,fl = 1.2. The unstable invariant circle also vanishes at ~, = 0 where it collides with the degenerate solution A2 = {y = 0,0 = 00}. The criterion [00l < 1 is necessary for this solution to be locally stable and attracting. It follows that the dynamics associated with the stable period-one fixed point can only be followed over a limited range of parameters. The set A2 corresponds to the set A described in Example 1 and, analogous to the previous example, this set can be viewed as a bifurcation of a period-two point which is established for ~, < 0. The significance
316
M.G. Kusch, B.E. Ydstie /Drift instabilities and chaos in forecasting
-0.4-
'~
e~
-0.8-
-1.2-1.6-
o
-2
0
//
//
•
[l ~
unstablefixedpoint ~../ ((1 11 04) , - 1 . 0~ 4/ ..... ~ ~"/
~ 015
unstableinvariant circle
,,-'
;
115
2
215
~1/ (normalized process output)
Fig. 5. Stable and unstable invariant circles following the Hopf bifurcation. The unstable invariant circle is constructed by iterating the map G2 backwards.
of the two solutions, which co-exist for a limited range of 7, can be explained as follows. The desired fixed point (y = 1 ) results from the forecast error going to zero, whereas the undesired fixed point (y = 0) results from the adaptation mechanism being turned off. The latter leads to parameter drift. Other periodic attractors can also be observed on and near the line 7 = 0. For P > 1 and fl = P - 1 a period-three fixed point (y, 0) : {(1,1) ~ ( - 1 , - 1 ) ~ ( - 1 , 1 ) --. (1,1)} is attracting for some initial conditions. Figure 6 shows the basins of attraction for P = 1.5 and fl = 0.5 (point C in fig. 3). The area colored gray denotes the set of initial conditions (y, 0) which are mapped onto the line, y = 0. The area colored white denotes the set of initial conditions which are mapped onto the fixed point ( 1, - 1 ). Finally the area colored black denotes the set of initial conditions which are mapped onto the period-three attractor. A more detailed study can be found in Frouzakis et al. [ 7 ]. By continuing the period-three fixed point using AUTO, we determine that it comes from a saddle-node bifurcation and it exists over a significant range of the parameters P, and ?. The evolution of higher-order attractors from
Fig. 6. Basins of attraction constructed by simulation.
both fixed points is shown in fig. 7 for {P = 0.8, fl = 1.2} as 7 is varied along line D in fig. 3. Here it is seen that the stable invariant circle, which bifurcates from the fixed point ( 1 , - 1 ), eventually vanishes due to a saddle-node bifurcation of invariant circles. The stable period-two fixed point which terminates at the point (0,1) bifurcates into a pair of stable invariant circles. This type of bifurcation is typical of what can be observed in more complex adaptive mechanisms. As 7 is decreased, the invariant circles undergo frequency-locking causing period-n attractors to appear on the invariant circles. The phenomenon of frequency-locking occurs in discrete systems when eigenvalues cross the unit circle at the nth root of unity. Regions in the parameter space, where locking occurs, are referred to as Arnold tongues [1 ]. A description of this phenomenon in the context of adaptive control is given by Frouzakis et al. [7]. As the parameter 7 is further decreased the adaptive
M.G. Kusch, B.E. Ydstie / Drift instabilities and chaos in forecasting
317
2.5 O
o
1.5
0.
•-~
0.5
"~ 0
-.5
[
~
-1.5
~-
-2.5
Hc
Cos
-11
-',s
-
'2
y [bifurcation parameter] Fig. 7. Bifurcation diagram constructed by simulation. The parameter ~ represents the noise to signal ratio.
system alternates between quasiperiodic (unlocked) and periodic (locked) states as it traverses these regions and the periodic attractor undergoes a series of simple period-doubling and saddle-node bifurcations similar to those in the logistic map. An understanding of this process can be obtained from a magnified picture showing the intersecting stable (W s (B)) and unstable ( W u (B)) manifolds of two neighboring saddles of a period-28 f'Lxed point (fig. 8). The period-28 point gives rise to frequency locking while the manifold crossings create a homoclinic tangle. This affects the transients to the attractor to a considerable extent and continues to do so after the period-28 point loses its stability. In particular, we note the presence of a hyperbolic fixed point (B) and a (transversal) intersection of the respective manifolds W s (B) and WU(B) at the point (A). Since A # B it follows that G2 has a hyperbolic invariant limit set. The presence of this structure, known as a horseshoe, indicates the existence of a strange hyperbolic attractor. The dynamics associated with such structures are in general remarkably stable, albeit complicated [ 17 ]. As ~ is further decreased the saddles and nodes collide and disappear. What remains is the com-
plex geometric structure which forms the chaotic attractor as shown in fig. 9 (7 = -0.2). This phenomenon has been extensively investigated in the literature [8,5 ]. It is clear that the dynamics in the drift and burst phase are controlled by the period-two fixed point observed in Example 1 and not by the period-one fixed point. From this we conclude that linearization around the fixed point is irrelevant. Figure 10 shows the time series of the output and the parameter estimate when the system is chaotic. During quiescent periods the system is stabilized and the output and the parameter estimate drift slowly in response to the biassing influence of the environment. The forecast eventually becomes irrelevant. Eventually, a burst in the output supplies excitation and large forecast errors. This causes the parameter estimate to readapt to what appears to be a "good" estimate. This process, which represents the problem described in the introduction, can be duplicated with great regularity in experimental studies.
5. Persistent instability and conflicts In this section we show that adaptive decision making processes are destabilized when there is
M.G. Kasch, B.E. Ydstie / Drift instabilities and chaos in forecasting
318
i
• unstoble perlod-2 • stcble period-28
i
~
o
~
i 1.1
c
~L .
0.6
o
0.1
-1.5
-1.0
-0.5
V (normalized process output) Fig. 8. E n l a r g e m e n t o f t h e s t a b l e a n d u n s t a b l e m a n i f o l d s o f a p e r i o d - 2 8 p o i n t .
chaotic ohase
2.375
.-
1.75 e~
.~
1 .I 25
-.125
-'3
4.75
-'.5
'.75
V [normalizedprocess output] Fig. 9. Simulation showing the phase plane of an adaptive system undergoing parameter drift and output bursting. u n m o d e l l e d interaction with surrounding processes. T h e m e c h a n i s m adapts, h o w e v e r , multiperiodicity and c h a o s yield lower performance.
y --~-Oy - a2x + "0, 0~0
+
py-Oy
- a2x + v c+y2
Example 3 [18]. With a2 ~ 0 and y* = 0) w e h a v e the m a p G3 : R 3 ~ R3:
where 0 = kO(t) + al and P = kp. T h e set
x---~y,
A~ = {x = O,y = 0 , 0 = Oo s.t. IOo[ < 1}
M.G. Kusch, B.E. Ydatie / Drift instabilities and chaos in forecasting
319
2.5
-i o
1.5
¢¢J
8 "d -.5
--
- ! .~5 - -
"output bursting"
'o
%0
~oo
~oo
koo
koo
,~oo
sampling interval
2.3
--
t~
~.s-
0.7
•.~
-
-o.1
-
-0.9
--
"parameter drift"
'o
',oo
~,oo sampling interval
Fig. 10. The top figure shows the process output while the bottom figure shows the estimated parameter during parameter drift and bursting.
represents a stable equilibrium when v = 0. There is an obvious connection between this and the set ,4. The criterion 100[ < 1 is necessary if the map is to be dissipative and contracting in the neighborhood of ,43. Similar to the map G~, there are no periodone fixed points when v # 0. However, there is a period-two fixed point which is given by #l
#1 A period-I fixed point can be introduced by setting y* ~ 0. It was shown, using map G2 that this point does not control the dynamics in the region of interest so this idea is not pursued here.
P l = (Xl = O, Yl = v / ( 1
+ a 2 ) , O l = 1 -I- a 2 ) ,
P2 = (Y2 = O, x2 --~ v / ( 1
..I- a 2 ) , 0 2 = 1 -I- a 2 ) .
We have det (DG3) = a2. A mapping with a constant, non-zero determinant is a global diffemorphism. The map is (globally) dissipative if and only if ]a2[ < 1. This sets an upper bound for the amount of unmodelled feedback that can be tolerated from the global context. In other words, one cannot stabilize a system if the external environment dominates. In such situations it is necessary to weaken the influence of the global con-
M.G. Kusch, B.E. Ydstie / Drift instabilities and chaos in forecasting
320
.75
t~
025
'15
a 2 [bifurcationparameter] Fig. I 1. Bifurcation diagram constructed by simulation. text by decoupling and isolation through the application of policies aimed towards neutralizing the effect of external influences. In the extreme, barriers can be introduced to negate feedback once the source has been identified. Such barriers are effective until countermeasures are taken. Countermeasures, of course, introduce feedback bringing us back to where we started. The condition for local stability of the periodtwo fixed point is
- 1 < a2 < O,
O
4c(1 + a2)3('v 2 + c(1 + a2) 2) v2(v 2 + 2 c ( 1 + a 2 ) 2)
A Hopfbifurcation occurs along the line a2 = 0. This corresponds, not coincidentally, to the first asymptotic solution of the map GI, a stable elliptic period-two fixed point. The period-two fixed point is hyberbolic and locally stable if - 1 < a2 < 0. In this region, we obtain a forecasting error of v / ( 2 + 2a2). At a2 = 0, corresponding to map GI, the attractor undergoes a Hopfbifurcation and two invariant circles form, one about each elliptic point of the period two fixed point at a2 = 0. This phenomenon is similar to the behavior observed for map G2 as the external per-
turbations become increasingly important. Figure 11 shows a bifurcation diagram which illustrates the asymptotic behavior of this map as a2 is increased. The sharp transitions from periodic behavior to chaotic behavior are indicative of global bifurcations as saddles and nodes of the periodic attractor collide and vanish. Complex tangles of connected stable and unstable manifolds remain and can be analyzed using the same techniques as those used in the study of map G2. The pathway to chaos is similar even though G3 is three dimensional. One way to explain this is that the phase space contracts and becomes two dimensional when la21 < 1. One significant point to notice is that although the performance of the adaptive system deteriorates quickly as a2 is increased from 0 to 0.025, no significant decrease in performance is noticed as the external feedback increases. The dynamics are similar to those observed in mapping G2. It follows that even very small unmodelled feedback from the environment may lead to significant biassing of the policy. As a2 is increased further, the laminar region of the attractor shrinks indicating, in a loose sense, a stronger attractiveness of the line y* = 0. Figure 12 shows a comparison of the time
M.G. Kusch, B.E. Ydstie / Drift instabilities and chaos in forecasting
321
2.4
.~
1.9
i
-
1.4 --
.~
~
0.9
-
0.4
-
-0.1 -
y (process output)
2.4
0 (parameter estimate) 1.775-
1.15
0.525
. .................
s _ta_bJ]_iv_]_h9.it . . . . . . . . .
-
0.5
--
-0.125
-
y (process output) it,_,,,lildk,,~..,ilil~i,...,dlLh ...... dihl..h.iL.,,,,llh~,illL~illh,dllL,,ih,...,litla ......... .......
o
,i'"'"",''"
~ r ~ , .......
;bo
, ~ _
~l~'"'"""r""~"""~ ~lrr~ ......... ~ffrr,''-'~
2bo
3~o
:~::":'
4~o
sampling interval
Fig. 12. Parameter drift and output bursting due to the addition of unmodelled dynamics a2 = 0.85. Note that trajectories are more irregular but that the performance of the adaptive system has not changed significantly.
322
M.G. Kasch, B.E. Ydstie /Drift instabilities and chaos in forecasting
evolution plots of 0 and y with the phase plane plot of such a system. Now the model drifts slowly, but randomly, near, and typically above, the stability limit of the parameter estimate of the period-two fixed point, 0 = 1. The process output has become very erratic. In fact, the local feedback process is unstable most of the time. The main objective of policy adaptation is to "play catch up" with the external environment and this leads to biassing and a conflict. This type of instability, which is qualitatively similar to the dynamical behaviour of some macroeconomic systems, can bc explained using the rational expectations hypothesis and bchaviour close to saddle points [2]. However, the arguments presented here are based on linear theory and are incomplete since they do not explain how the system recovers. We maintain that the system quickly recovers since it adapts. In a more complex system there is no reason to believe that the system state returns to the state it previously occupied. An example where the adaptive system, after a burst, returns to a different state has been reported by Huberman [12]. Small parameter changes, unstructured perturbations and stochastic noise have little influence on the dynamics of these systems. If we consider a2 to be the coefficient resulting from the discrctization of a second order, stable, continuous-time process, then a2 = e - d h , where d > 0 and h is the sampling period. From this relationship it is obvious that by increasing the sampling period, we can reduce the degree of complexity of the drift. In other words, the coupling can be made less important by extending the forecasting horizon.
6. Conclusions In this paper we have shown that unpredictability and randomness in adaptive decision theory is generated by a conflict between forecasting and policy making. This conflict leads to intermittcncy observed as complacent drift and
chaotic bursting originating from a degenerate set A. In one case the degenerate set collapses to an elliptic period-two point and drift originates from a Hopf bifurcation of this. In the other case the degenerate set collapses to a period-one point and we observe global bifurcations from this. The evolution of these distinct attractors has been traced and it has been shown that they coalesce in a saddle node bifurcation. It is the period-two point which continues to control the dynamic behaviour of the adaptive policy. The presence of a horseshoc has been determined and shows that the results we obtain are structurally stable and reproducible in simulations and experiments. The analysis of these simple, low-order systems provides a new point of view for the understanding of how chaos and unpredictability can be generated in adaptive decision processes. The analysis may also be used to explain origins and evolution of complacency and chaotic bursting in macro-economic models.
Acknowledgements The authors aprcciate numerous helpful discussions with Professor Ioannis Kevrikidis, Dr Ray Adomaitis and Dr Christos Frouzakis of the Department of Chemical Engineering at Princeton University. W c found the use of their computer package, S C I G M A , invaluable in showing the existence of the hyperbolic scts.
References [I] V.I. Arnold, Geometrical Methods in the Theory of Ordinary DifferentialEquations (Springer,N e w York, 1983). [2] D.K.H. Beg8, The Rational Expectations Revolution in Macro-Economics (The Johns Hopkins Univ. Press, Baltimore, M D , 1982). [3]L. Guo and H.F. Chen, Revisit to ~,stremWittenmark's self-tuning regulator and ELS based adaptive trackers, IEEE Trans. Auto. Control. 36 (1991) 802-812.
M.G. Kasch, B.E. Ydstie / Drift instabilities and chaos in forecasting [4]E. Doedel, AUTO: Software for Continuation and Bifurcation Problems in Ordinary Differential Equations (California Institute of Technology, 1986). [5]R.E. Ecke and I.G. Kevrekidis, Interactions of resonances and global bifurcations in RayleighBernard convection, Phys. Lett. A 131 (1988). [6] C.E. Frouzakis, R.A. Adomaitis and I.G. Kevrikidis, Resonance phenomena in adaptively controlled system, Int. J. Bifur. Chaos 1 (1990) 83-106. [7] C.E. Frouzakis, R.A. Adomaitis, I.G. Kevrikidis, M.P. Golden and B.E. Ydstie, The structure of basin boundaries in a simple adaptive control system, in: Chaotic Dynamics: Theory and Practice, T. Bountis, ed. (Plenum, New York, 1992) pp. 195-210. [8] L. Glass and R. Perez, Fine structure of phase locking, Phys. Rev. Lett. 48 (1982) 1772-1775. [9] M.P. Golden and B.E. Ydstie, Bifurcation in model reference adaptive control systems, Syst. Control Lett. 11 (1988) 413-430. [10]M.P. Golden and B.E. Ydstie, Small amplitude chaos and ergodicity in adaptive control systems, Automatica 28 (1992) 11-25. [11] G.C. Goodwin, P.J. Ramadge and P.E. Caines, Discrete time stochastic adaptive control, SIAM J. Control Optimiz. 19 (1981) 829-853.
323
[12] B.A. Huberman, Controlling Chaos, Paper presented at Annual Meeting AIChE, Los Angeles, CA, (November, 1991 ). [ 13] A. Marcet and T.J. Sargent, Least squares learning and the dynamics of hyperinflation, in: Economic Complexity: Chaos, Sunspots, Bubbles and Nonlinearity (Cambridge Univ. Press, Cambridge, 1989) pp. 119-137. [14]I.M.Y. Mareels and R.R. Bitmead, Non-linear dynamics in adaptive control: chaotic and Paeriodic stabilization, Automatica 22 (1986) 641-655. [ 15 ] L. Praly and M. Pomet, Periodic Solutions in Adaptive Systems: The Regular Case, IFAC Xth Triennal World Congress (Munich, FRG, 1987) Vol. 10, pp. 40-44. [ 16 ] H.A. Simon, Dynamic programming under uncertainty with quadratic criterion function, Econometrica 24 (1956) 74. [17] M. Shub, Global Stability of Dynamical Systems (Springer, New York, 1987). [18] B.E. Ydstie and M.P. Golden, Chaos and Strange Attractors in Adaptive Control Systems, IFAC Xth Triennal World Congress (Munich, FRG, 1987) Vol. 10, pp. 127-132. [ 19] B.E. Ydstie, Bifurcation and Complex Dynamics in Adaptive Control Systems, Proc. 25th IEEE-CDC (Athens, Greece, December, 1986), pp. 2232-2236.