Chapter 7 Learning dynamics

Chapter 7

LEARNING DYNAMICS GEORGE W. EVANS University of Oregon SEPPO HONKAPOHJA

University of Helsinki Contents

Abstract Keywords 1. Introduction 1.1. Expectations and the role of learning 1.1.1. Background 1.1.2. Role of learning in macroeconomics 1.1.3. Alternative reduced forms 1.2. Some economic examples 1.2.1. The Muth model 1.2.2. A linear model with multiple REE 1.2.3. The overlapping generations model with money 1.3. Approaches to learning 1.3.1. Rational learning 1.3.2. Eductive approaches 1.3.3. Adaptive approaches 1.4. Examples of statistical learning rules 1.4.1. Least squares learning in the Muth model 1.4.2. Least squares learning in a linear model with multiple REE 1.4.3. Learning a steady state 1.4.4. The seignorage model of inflation 1.5. Adaptive learning and the E-stability principle 1.6. Discussion of the literature 2. General m e t h o d o l o g y : recursive stochastic algorithms 2.1. General setup and assumptions 2.1.1. Notes on the technical literature 2.2. Assumptions on the algorithm 2.3. Convergence: the basic results 2.3.1. ODE approximation Handbook of Macroeconomics, Volume 1, Edited by JB. Taylor and M. WoodJbrd © 1999 Elsevier Science B. V All rights reserved 449

452 452 453 453 453 454 455 456 456 457 458 461 461 462 464 465 465 467 468 471 472 473 475 475 476 476 478 478

450

G. W. Evans and S. Honkapohja

2.3.2. Asymptotic analysis 2.4. Convergence: further discussion 2.4.1. Immediate consequences 2.4.2. Algorithms with a projection facility 2.5. Instability results 2.6. Further remarks 2.7. Two examples 2.7.1. Learning noisy steady states 2.7.2. A model with a unique REE 2.8. Global convergence 3. Linear e c o n o m i c m o d e l s 3.1. Characterization of equilibria 3.2. Learning and E-stability in univariate models 3.2.1. A leading example 3.2.1.1. A characterization of the solutions 3.2.1.2. E-stability of the solutions 3.2.1.3. Strong E-stability 3.2.1.4. E-stability and indeterminacy 3.2.2. The leading example: adaptive learning 3.2.2.1. Adaptive and statistical learning of MSV solution 3.2.2.2. Learning non-MSV solutions 3.2.2.2.1. Recursive least squares learning: the AR(1) case 3.2.2.2.2. Learning sunspot solutions 3.2.3. Lagged endogenous variables 3.2.3.1. A characterization of the solutions 3.2.3.2. Stability under learning of the AR(1) MSV solutions 3.2.3.3. Discussion of examples 3.3. Univariate models further extensions and examples 3.3.1. Models with t dating of expectations 3.3.1.1. Alternative dating 3.3.2. Bubbles 3.3.3. A monetary model with mixed datings 3.3.4. A linear model with two forward leads 3.4. Multivariate models 3.4.1. MSV solutions and learning 3.4.2. Multivariate models with time t dating 3.4.3. Irregular models 4. L e a r n i n g in nonlinear m o d e l s 4.1. Introduction 4.2. Steady states and cycles in models with intrinsic noise 4.2.1. Some economic examples 4.2.2. Noisy steady states and cycles 4.2.3. Adaptive learning algorithms

479 480 480 480 481 482 483 483 484 486 487 487 488 488 489 490 491 491 493 493 493 494 494 495 496 496 496 497 497 499 499 5O0 501 502 503 505 5O5 5O6 5O6 5O7 5O7 509 510

Ch. 7: Learning Dynamics

451

4.2.4. E-stability and convergence 4.2.4.1. Weak and strong E-stability 4.2.4.2. Convergence 4.2.4.3. The case of small noise 4.2.5. Economic models with steady states and cycles 4.2.5.1. Economic examples continued 4.2.5.2. Other economic models 4.3. Learning sunspot equilibria 4.3.1. Existence of sunspot equilibria 4.3.2. Analysis of learning 4.3.2.1. Fornmlation of the learning rule 4.3.2.2. Analysis of convergence 4.3.3. Stability of SSEs near deterministic solutions 4.3.4. Applying the results to OG and other models 5. E x t e n s i o n s a n d r e c e n t d e v e l o p m e n t s 5.1. Genetic algorithms, classifier systems and neural networks 5.1.1. Genetic algorithms 5.1.2. Classifier systems 5.1.3. Neural networks 5.1.4. Recent applications of genetic algorithms 5.2. Heterogeneity in learning behavior 5.3. Learning in misspecified models 5.4. Experimental evidence 5.5. Further topics 6. C o n c l u s i o n s References

511 512 513 513 513 513 514 515 516 517 517 518 520 520 521 521 521 523 524 525 527 528 530 531 533 533

452

G. W. Evans and S. Honkapohja

Abstract

This chapter provides a survey of the recent work on learning in the context of macroeconomics. Learning has several roles. First, it provides a boundedly rational model of how rational expectations can be achieved. Secondly, learning acts as a selection device in models with multiple REE (rational expectations equilibria). Third, the learning dynamics themselves may be of interest. While there are various approaches to learning in macroeconomics, the emphasis here is on adaptive learning schemes in which agents use statistical or econometric techniques in self-referential stochastic systems. Careful attention is given to learning in models with multiple equilibria. The methodological tool is to set up the economic system under learning as a SRA (stochastic recursive algorithm) and to analyze convergence by the method of stochastic approximation based on an associated differential equation. Global stability, local stability and instability results for SRAs are presented. For a wide range of solutions to economic models the stability conditions for REE under statistical learning rules are given by the expectational stability principle, which is treated as a unifying principle for the results presented. Both linear and nonlinear economic models are considered and in the univariate linear case the full set of solutions is discussed. Applications include the Muth cobweb model, the Cagan model of inflation, asset pricing with risk neutrality, the overlapping generations model, the seignorage model of inflation, models with increasing social returns, IS-LM-Phillips curve models, the overlapping contract model, and the Real Business Cycle model. Particular attention is given to the local stability conditions for convergence when there are indeterminacies, bubbles, multiple steady states, cycles or sunspot solutions. The survey also discusses alternative approaches and recent developments, including Bayesian learning, eductive approaches, genetic algorithms, heterogeneity, misspecifled models and experimental evidence.

Keywords expectations, learning, adaptive learning, least squares learning, eductive learning, multiple equilibria, expectational stability, stochastic recursive algorithms, sunspot equilibria, cycles, multivariate models, MSV solutions, stability, instability, ODE aproximation, stochastic approximation, computational intelligence, dynamic expectations models J E L classification: E32, D83, D84, C62

Ch. 7: Learning Dynamics

453

1. Introduction

1.1. Expectations and the role of learning 1.1.1. Background In modern macroeconomic models the role o f expectations is central. In a typical reduced form model a vector o f endogenous variables yt depends on lagged values yt-1, on expectations o f the next period's values, Yt+l, and perhaps on a vector o f exogenous shocks ut, e.g. taking the form Yt = F ( y t 1,Yt+l, ut), where for the moment assume F to be linear. O f course, in some models the dependence on Yt-l or ut m a y be absent. The information set available when Yt+l is formed typically includes {Yt-i, ut-i, i = 1,2, 3 . . . . } and may or may not also include the contemporaneous values Yt and ut. A useful notation, i f y t , ut are in the information set, is E{yt+l and we write the reduced form as

Yt - F ( y t l,EtYt+l, ut).

(1)

I f y t and ut are not included in the information set then we write yet+l as ET_yt+l. In the economic models we consider in this survey, these expectations are those held by the private agents in the economy, i.e. o f the households or the firms. Models in which policy makers, as well as private agents, must form expectations raise additional strategic issues which we do not have space to explore l . Following the literature, we restrict attention to models with a large number o f agents in which the actions o f an individual agent have negligible effect on the values yr. Closing the model requires a theory o f how expectations are formed. In the 1950s and 1960s the standard approach was to assume adaptive expectations, in which expectations were adjusted in the direction o f the most recent forecast error, e.g. in the scalar case, and assuming Yt is in the information set, ETyt+l = ET_lyt + Y(yt - ET_lyt) for some value o f 0 < ~/ ~< 1. Though simple and often well-behaved, a well known disadvantage o f adaptive expectations is that in certain environments it will lead to systematic forecast errors, which appears inconsistent with the assumption o f rational agents. The rational expectations revolution o f the 1970s has led to the now standard alternative assumption that expectations are equal to the true conditional expectations in the statistical sense. Rational expectations (in this standard interpretation used in macroeconomics) is a strong assumption in various ways: it assumes that agents know the true economic model generating the data and implicitly assumes coordination o f expectations by the agents 2. It is, however, a natural benchmark assumption and is widely in use.

1 See Sargent (1999) for some models with learning by policy makers. 2 A rational expectations equilibrium can be interpreted as a Nash equilibrium, a point made in Townsend (1978) and Evans (1983). It is thus not rational for an individual to hold "rational expectations" unless all other agents are assumed to hold rational expectations. See the discussion in Frydman and Phelps (1983).

454

G. 14(Evans and S. Honkapohja

More recently a literature has developed in which the RE (rational expectations) assumption has been replaced by the assumption that expectations follow a learning rule, either a stylized or a real-time learning rule, which has the potential to converge to RE. An example of a learning rule is one in which agents use a linear regression model to forecast the variables of interest and estimate the required parameters by least squares, updating the parameter estimates each period to incorporate new data. Modeling expectations in this fashion puts the agents in the model in a symmetric position with the economic analyst, since, when studying real economies, economists use econometrics and statistical inference. In contrast, under RE the agents in the model economy have much more information than the outside observer. It is worth emphasizing that most of the literature on learning reviewed in this paper has followed standard practice in macroeconomics and postulates the assumption of a representative agent as a simplification. This implies that the expectations and the learning rules of different agents are assumed to be identical. Some recent papers allow for heterogeneity in learning and this work is discussed below. 1.1.2. Role o f learning in macroeconomics

Introducing learning into dynamic expectations models has several motivations. First, learning has been used to address the issue of the plausibility of the RE assumption in a particular model: could boundedly rational agents arrive at RE through a learning rule? This issue is of interest as it provides a justification for the RE hypothesis. The early work by DeCanio (1979), Bray (1982) and Evans (1983) focused on this, and some further papers are Bray and Savin (1986), Fourgeaud, Gourieroux and Pradel (1986), Marcet and Sargent (1989b), and Guesnerie (1992). This view is forcefully expressed by Lucas (1986), though he views the adjustment as very quick. Secondly, there is the possibility of models with multiple REE (rational expectations equilibria). If some REE are locally stable under a learning rule, while others are locally unstable, then learning acts as a selection device for choosing the REE which we can expect to observe in practice. This point was made in Evans (1985) and Grandmont (1985) and developed, for example, in Guesnerie and Woodford (1991) and Evans and Honkapohja (1992, 1994b, 1995a). Extensive recent work has been devoted to obtaining stability conditions for convergence of learning to particular REE and this work is discussed in detail in the later sections of this paper. A particular issue of interest is the conditions under which there can be convergence to exotic solutions, such as sunspot equilibria. This was established by Woodford (1990). Thirdly, it may be of interest to take seriously the learning dynamics itself, e.g. during the transition to RE. Dynamics with learning can be qualitatively different from, say, fully rational adjustment after a structural change. This has been the focus of some policy oriented papers, e.g. Taylor (1975), Frydman and Phelps (1983), Currie, Garratt and Hall (1993) and Fuhrer and Hooker (1993). It has also been the focus of some recent work on asset pricing, see Timmermann (1993, 1996) and Bossaerts (1995). Brian Arthur [see e.g. papers reprinted in Arthur (1994)] has emphasized path-

Ch. 7: Learning Dynamics

455

dependence of adaptive learning dynamics in the presence of multiple equilibria. If the model is misspecified by the agents, then this can effectively lead to persistent learning dynamics as in Evans and Honkapohja (1993a), Marcet and Nicolini (1998) and Timmermann (1995). Even if the model is not misspecified, particular learning dynamics may not fully converge to an REE and the learning dynamics may be of intrinsic interest. This arises, for example, in Arifovic (1996), Evans and Ramey (1995), Brock and Hommes (1996, 1997), and Moore and Schaller (1996, 1997) 3. The theoretical results on learning in macroeconomics have begun to receive some support in experimental work [e.g. Marimon and Sunder (1993, 1994) and Marimon, Spear and Sunder (1993)] though experimental work in macroeconomic set-ups has so far been less than fully studied. We review this work in Section 5.4. The implications of these results have led also to one further set of issues: the effects of policy, and appropriate policy design, in models with multiple REE. For example, if there are multiple REE which are stable under learning, then policy may play a role in which equilibrium is selected, and policy changes may also exhibit hysteresis and threshold effects. The appropriate choice of policy parameters can eliminate or render unstable inefficient steady states, cycles or sunspot equilibria. For examples, see Evans and Honkapohja (1993a,b, 1995b). Howitt (1992) provides examples in which the stability under learning of the REE is affected by the form of the particular monetary policy 4. A further application of learning algorithms is that they can also be used as a computational tool to solve a model for its REE. This point has been noted by Sargent (1993). An advantage of such algorithms is that they find only "learnable" REE. A well-known paper illustrating a computational technique is Marimon, McGrattan and Sargent (1989). A related approach is the method of parameterized expectations, see Marcet (1994) and Marcet and Marshall (1992).

1.1.3. Alternative reduced forms The models we will consider have various reduced forms, and some preliminary comments are useful before turning to some economic examples. The form (1) assumed that contemporaneous information is available when expectations are formed. If alternatively the information set is {Yt-i, ut-i, i = 1,2, 3,...} then Yt may also (or

3 These lines of research in macroeconomicscorrespond to parallel developments in game theory. For a survey of learning in economics which gives a greater role to learning in games, see Marimon (1997). See also Fudenberg and Levine (1998). 4 Herepolicy is modeled as a rule which atomistic private agents take as part of the economicstructure. Modeling policy as a game is a different approach, see e.g. Cho and Sargent (1996a) and Sargent (1999) for the latter in the context of learning.

456

G. W. Evans and S. Honkapohja

instead) depend on E[_lyt, the expectation o f y t formed at t - 1, so that the reduced form is Yt = F ( y t - l , E t * _ l y t , E t lYt+l,/At) o r Yt = F ( y t 1,E t lYt, ut).

The Muth model, below, is the special case Yt = F(E[_,yt, ut). In nonlinear stochastic models a point requiring some care is the precise quantity about which expectations are formed. Even assuming that lagged values of Yt-i are not present, the required reduced form might be Yt = H ( E [ G ( y t + l , Hi+l), Ut).

In many of the early examples the model is nonstochastic. The reduced form then becomes Yt = H ( E t G ( y t + I ) ) . If H is invertible then by changing variables to Y~ = I-I-~ (yt) the model can be transformed to ~t = E [ f ( ~ t + l ) ,

(2)

w h e r e f ( ~ ) = G(H(.~)). The form (2) is convenient when one considers the possibility of stochastic equilibria for models with no intrinsic randomness, see Section 4.3. In nonstochastic models, if agents have point expectations, these transformations are unnecessary and the model can again simply be analyzed in the form yt - f ( E t Y~+I),

(3)

where f ( y ) = H ( G ( y ) ) . This is standard, for example, in the study of learning in Overlapping Generations models. Finally, mixed datings of expectations appear in some models. For example, the seignorage model of inflation often considers a formulation in which

1.2. Some economic examples

It will be helpful at this stage to give several economic examples which we will use to illustrate the role of learning. 1.2.1. The Muth model

The "cobweb" model of a competitive market in which the demand for a perishable good depends on its price and the supply, due to a production lag, depends on its

457

Ch. 7: Learning Dynamics

expected price, was originally solved under rational expectations by Muth (1961). Consider the structural model qt = m l - m2Pt + Olt, q¢ = r l E ~ lpt + r ~ w t - i + vzt,

where m2, rl > 0, Vlt and vzt are unobserved white noise shocks and wt 1 is a vector o f exogenous shocks, also assumed white noise for convenience, qt is output, Pt is price, and the first equation represents demand while the second is supply. The reduced form for this model is (4)

Pt = ~ + a E t lPt + Y l w t 1 + ~lt,

where ~ = m l / m 2 , ] / = - r 2 / m 2 , and a = - r l / m 2 There is a unique REE in this model given by

~t = (vlt - 02~)/m2. Note that a < 0.

pt = a + blwt-1 + ~/t, where a=(1-a) ~/~, b = ( 1 a ) - l ) ,. Under RE, E t _ l P t = ~l + [Jwt_ 1. L u c a s a g g r e g a t e s u p p l y m o d e l . A n identical reduced form arises from the following simple macroeconomic model in the spirit o f Lucas (1973). Aggregate output is given

by qt = 7t+ O(pt - E t _ l P t ) + ~t,

while aggregate demand is given by the quantity theory equation mt + vt = P t + qt,

and the money supply follows the policy rule mt = ff~ +[9out + p l Wt 1.

Here 0 > 0 and the shocks ~t, vt, ut and wt are assumed for simplicity to be white noise. Solving for Pt in terms o f E ~ l P t , wt-1 and the white noise shocks yields the reduced form (4). For this model 0 < a = 0(1 + 0) -1 < 1. 1.2.2. A l i n e a r m o d e l w i t h m u l t i p l e R E E

Reduced form models o f the form Yt = a + [3Ei* yt+l + 6 y t - , + x w t + vt

(5)

arise from various economic models. Here Yt is a scalar, vt is a scalar white noise shock and wt is an exogenous vector o f observables which we will assume follows a stationary first-order VAR (vector auto-regression) wt = p w t 1 + et.

(6)

Variables dated t are assumed in the time t information set. An example is the linearquadratic market model described in Sections XIV.4 and XIV.6 o f Sargent (1987).

458

G. W. Evans and S. Honkapohja

The standard procedure is to obtain solutions of the form Yt = ?t + byl-I + Uwt + [tvt,

(7)

where D satisfies the quadratic 82 - / 3 18 +/3-1,~ = 0. For many parameter values there will be a unique stationary solution with 181 < 1. However, if externalities or taxes are introduced into the model as in Section XIV.8 of Sargent (1987), then for appropriate parameter values both roots of the quadratic are real and have absolute value less than unity, so that there are two stationary solutions of the form (7). (In this case there also exist solutions that depend on sunspots.) 1.2.3. The overlapping generations model with m o n e y

The standard Overlapping Generations model with money provides an example of a model with REE cycles. 5 Assume a constant population of two-period lived agents. There are equal numbers of young and old agents, and at the end of each period the old agents die and are replaced in the following period by young agents. In the simple version with production, the utility function of a representative agent born at the beginning of period t is U ( C t + l ) - W(nt), where Ct+l is consumption when old and nt is labor supplied when young. U is assumed increasing and concave and W is assumed increasing and convex. We assume that output of the single perishable good qt for the representative agent is given by qt = nt, and that there is a fixed stock of money M. The representative agent produces output in t, trades the goods for money, and then uses the money to buy output for consumption in t + 1. The agent thus chooses n t , M t and ct+~ subject to the budget constraints ptnt = Mt = Pt+lCt+l, where Pt is the price of goods in year t. In equilibrium ct = nt, because the good is perishable, and N(t = M. The first-order condition for the household is ~ ' ( n 3 = Et. (p,+~ pt g'(ct+l)). Using the market clearing condition ct+l = nt+l and the relation Pt/Pt+l = nt+l/nt, which follows from the market clearing condition p t n t = M , we obtain the univariate equation n t W ' ( n t ) = E t (nt+l U'(nt+O). Since w(n) =- n W ' ( n ) is an increasing function we can invert it to write the reduced form as nt = H ( E t G(nt+l)),

(8)

where H(-) - w 1(.) and G(n) = nU'(n). I f one is focusing on nonstochastic solutions, the form (8) expresses the model in terms of labor supply (or equivalently 5 Overlapping generations models are surveyed, for example, in Geanakoplos and Polemarchakis (1991).

Ch. 7." Learning Dynamics

459

nt

rl t

~) t+l) I

I

n~

n t+l

n2

(a)

I

;,

n 3

at+ 1

(b)

rl t

f(nt+l)

nt

f(nt+< g)

i nL

n H

I

nL nu

nt+l

(c)

I

nH

~-

nt+~

(d) Fig. 1.

real balances) and assuming point expectations one has E[G(nL+I) = G(E[nt+l) and one can write nt = f ( E [ n t + l ) f o r f = H o G. The model can also be expressed in terms of other economically interpretable variables, such as the inflation rate :vt = p t / P t - l . One takes the budget constraint and the household's first-order condition which, under appropriate assumptions, yield savings (real balances) as a function mt ==-M/pt = S(E[Yf;t+I). Then the identity mt~5t = mt t yields ~t = S(E[ l~t)/S(E[Jvt+~). Depending on the utility functions U and W, the reduced form function f can have a wide range of shapes. If the substitution effect dominates everywhere then f will be increasing and there will be a single interior steady state (see Figure 1a), but if the income effect dominates over part of the range t h e n f can be hump-shaped. (An

G W. Evans and N Honkapohja

460

autarkic steady state state can also exist for the model.) In consequence the OG model can have perfect foresight cycles as well as a steady state (see Figure lb). Grandmont (1985) showed that for some choices o f preferences there coexist perfect foresight cycles of every order 6. Whenever there are cycles in the OG model, there are multiple equilibria, so that the role o f learning as a selection criterion becomes important. Various extensions o f the OG model can give rise to multiple (interior) steady states. Extension 1 (seignorage model). We briefly outline here two extensions o f the OG model which lead to the possibility o f multiple steady states. In the first extension we introduce government purchases financed by seignorage. Assuming that there is a fixed level o f real government purchases g financed entirely by printing money then g = (Mr - M t i)/pt. The first-order condition for the household is the same as in the basic model. Using the market clearing conditions p t n t = Mr, pt+lct+~ = Mt and Ct+l = nt+l - g we have pt/pt+l = (nt+l - g ) / n t which yields nt = H(E[((nt+l - g)U'(nt+l - g))) or nt = f(E[(nt+l - g)) for nonstochastic equilibria 7. In the case where the substitution effect dominates and where f is an increasing concave function which goes through the origin, this model has two interior steady states provided g > 0 is not too large. See Figure lc. It can be verified that the steady state n = nH corresponds to higher employment and lower inflation relative to n = nL. Extension 2 (increasing social returns). Assume again that there is no government spending and that the money supply is constant. However, replace the simple production function qt = nt by the function qt - 5C(nt, Art), where Aft denotes aggregate labor effort and represents a positive production externality. We assume U1 > 0, U2 > 0 and 5ell < 0. Here Art = Knt where K is the total number of agents in the economy. The first-order condition is now W'(nt) = E? Pt .T'l(nt,Knt) Pt+l

Ul(Ct+l).

Using pt~ot+l = qt+l/qt and Ct+l = qt+l we have W' (nt ).T'(nt, Knt ) .Ul(nt,Knt)

= Et.F(nt+~,Knt+l) U'(.U(nt+l,Knt+l)).

Letting #(nt) denote the left-hand-side function, it can be verified that #(nt) is a strictly increasing function o f nt. Solving for nt and assuming point expectations

6 Conditions for the existence of k-cycles are discussed in Grandmont (1985) and Guesnerie and Woodford (1992). 7 Alternatively, the model can be expressed in terms of the inflation rate in the form S(E; l:r,) S(E?:rt+l) - g

Ch. 7: Learning Dynamics

461

yields nt = f~(E[nt+l) for a suitable j~. For appropriate specifications of the utility functions and the production functions it is possible to obtain reduced-form functionsj7 which yield three interior steady states, as in Figure ld. Examples are given in Evans and Honkapohja (1995b). Employment levels nL < n v < nH correspond to low, medium and high output levels, and the steady states nL and n v can be interpreted as coordination failures. 1.3. Approaches to learning

Several distinct approaches have been taken to learning in dynamic expectations models. We will broadly classify them into (i) Rational learning, (ii) Eductive approaches and (iii) Adaptive learning. The focus of this chapter is on adaptive learning, but we will provide an overview of the different approaches. 1.3.1. Rational learning

A model of rational learning, based on Bayesian updating, was developed by Townsend (1978) in the context of the cobweb model. In the simplest case, it is supposed that agents know the structure of the model up to one unknown parameter, ml, the demand intercept. There are a continuum of firms and each firm has a prior distribution for m I. The prior distributions of each firm are common knowledge. Townsend shows that there exist Nash equilibrium decision rules, in which the supply decision of each firm depends linearly on its own mean belief about ml and the mean beliefs of others. Together with the exogenous shocks, this determines aggregate supply qt and the price level Pt in period t, and finns use time t data to update their priors. It also follows that for each agent the mean belief about ml converges to ml as t --+ oc, and that the limiting equilibrium is the REE. Townsend extends this approach to consider versions in which the means of the prior beliefs of other agents are unknown, so that agents have distributions on the mean beliefs of others, as well as distributions on the mean of the markets distributions on the mean beliefs of others, etc. Under appropriate assumptions, Townsend is able to show that there exist Nash equilibrium decision rules based on these beliefs and that they converge over time to the REE. This approach is explored further in Townsend (1983). Although this general approach does exhibit a process of learning which converges to the REE, it sidesteps the issues raised above in our discussion of the role of learning. In particular, just as it was asked whether the REE could be reached by a boundedly rational learning rule, so it could be asked whether the Nash equilibrium strategies could be reached by a learning process. In fact the question of how agents could ever coordinate on these Nash equilibrium decision rules is even more acute, since they are based on ever more elaborate information sets. The work by Evans and Ramey (1992) on expectation calculation can also be regarded as a kind of rational learning, though in their case there is not full convergence to the REE (unless calculation costs are 0). Here agents are endowed with calculation

462

G. W. Evans and S. Honkapohja

algorithms, based on a correct structural model, which agents can use to compute improved forecasts. Agents balance the benefits o f improved forecasts against the time and resource costs o f calculation and are assumed to do so optimally. Formally, since their decisions are interdependent, they are assumed to follow Nash equilibrium decision rules in the number o f calculations to make at each time. Because o f the costs of expectation calculation, the calculation equilibrium exhibits gradual and incomplete adjustment to the REE. In a "Lucas supply-curve" or "natural rate" macroeconomic model, with a reduced form close to that o f the "cobweb" model, they show how monetary nonneutrality, hysteresis and amplification effects can arise. As with Townsend's models, the question can be raised as to how agents learn the equilibrium calculation decision rules 8. 1.3.2. E d u c t i v e a p p r o a c h e s

Some discussions o f learning are "eductive" in spirit, i.e. they investigate whether the coordination o f expectations on an REE can be attained by a mental process o f reasoning 9. Some o f the early discussions o f expectational stability, based on iterations o f expectation functions, had an eductive flavor, in accordance with the following argument. Consider the reduced form model (4) and suppose that initially all agents contemplate using some (nonrational) forecast rule E°t_lPt = a ° + b°'wt 1.

(9)

Inserting these expectations into Equation (4) we obtain the actual law o f motion which would be followed under this forecast rule: Pt = (t~ + a a °) + ( a b ° + 7)'wt x + tit,

and the true conditional expectation under this law o f motion: E t - l p t = (~ +ota °) + (orb ° + y ) ' w t 1.

Thus if agents conjecture that other agents form expectations according to Equation (9) then it would instead be rational to form expectations according to E11pt = a l

+bltwt_l,

where a 1 = ~ + a a ° and b 1 = y + ab °.

8 Evans and Ramey (1998) develop expectation calculation models in which the Nash equilibrium calculation decision rules are replaced by adaptive decision rules based on diagnostic calculations. This framework is then more like the adaptive learning category described below, but goes beyond statistical learning in two ways: (i) agents balance the costs and benefits of improved calculations, and (ii) agents employ a structural model which allows them to incorporate anticipated structural change. 9 The term "eductive" is due to Binmore (1987).

Ch. 7: Learning Dynamics

463

Continuing in this way, if agents conjecture that all other agents form expectations according to the rule EN1pt = a N + bNtwt_l, then it would be rational to instead form expectations according to

EN~lpt = (t~ + aaN) + (]I + (ybN)twt 1. Letting 0 N! = (a N, bNt), the relationship between Nth-order expectations and (N + 1)thorder expectations is given by

~)N+a = T(oN),

N = 1,2,3,...,

(10)

where

T(O)' : (Ta(a, b), Tb(a, b) I) = (l~ + aa, g' + ab').

(11)

One might then say that the REE is "expectationally stable" if limk~o~ q~N=~= (fi, ~)~)/. The interpretation is that if this stability condition is satisfied, then agents can be expected to coordinate, through a process o f reasoning, on the REE 10. Clearly for the problem at hand the stability condition is lal < 1, and if this condition is met then there is convergence globally from any initial ~b°. For the Lucas supply model example above, this condition is always satisfied. For the cobweb model, satisfaction of the stability condition depends on the relative slopes o f the supply and demand curves. In fact we shall reserve the term "expectational stability" for a related concept based on the corresponding differential equation. The differential equation version gives the appropriate condition for convergence to an REE under the adaptive learning rules. To distinguish the concepts clearly we will thus refer to stability under the iterations (10) as iterative expectational stability or iterative E-stability. The concept can be and has been applied to more general models. Let q~ denote a vector which parameterizes the expectation fimction and suppose that T(¢) gives the parameters o f the true conditional expectation when all other agents follow the expectation function with parameters q~. An REE will be a fixed point ~} of T (and in general there may be multiple REE o f this form). The REE is said to be iteratively E-stable if q~N __+ ~} for all q~0 in a neighborhood o f ~}.

10 Interpreting convergence of iterations of (10) as a process of learning the REE was introduced in DeCanio (1979) and was one of the learning rules considered in Bray (1982). [Section 6 of Lucas (1978) also considered convergence of such iterations.] DeCanio (1979) and Bray (1982) give an interpretation based on real time adaptive learning in which agents estimate the parameters of the forecast rule, but only alter the parameters used to make forecasts after estimates converge in probability. The eductive argument presented here is based on Evans (1983), where the term "expectational stability" was introduced. Evans (1985, 1986) used the iterative E-stability principle as a selection device in models with multiple REE. Related papers include Champsaur (1983) and Gottfries (1985).

464

G. W. Evans and S. Honkapohja

An apparent weakness of the argument just given is that it assumes homogeneous expectations of the agents. In fact, the eductive argument based on iterative E-stability is closely related to the concept of rationalizability used in game theory, which allows for heterogeneity of the expectations of agents. The issue of rationalizability in the cobweb model was investigated by Guesnerie (1992). In Guesnerie's terminology the REE is said to be strongly rational if for each agent the set of rationalizable strategies is unique and corresponds to the REE, Guesnerie showed that if lal < 1 in Equation (4) then the REE is strongly rationalizable, so that in this case the eductive arguments are indeed compelling. Guesnerie (1992) shows that the strong rationality argument can be extended to allow also for heterogeneity in the economic structure, e.g. a different supply curve for each agent, due to different cost functions. The argument can also be extended to cases with multiple REE by making the argument local. In Evans and Guesnerie (1993) the argument is extended to a multivariate setting and the relationship between strong rationality and iterative E-stability is further examined. I f the model is homogeneous in structure, then (even allowing for heterogeneity in beliefs) an REE is strongly rational if and only if it meets the iterative E-stability condition. However, if heterogeneity in the structure is permitted, then iterative E-stability is a necessary but not sufficient condition for strong rationality of the REE. For an investigation of strong rationality in univariate models with expectations of future variables, see Guesnerie (1993). Guesnerie (1996) develops an application to Keynesian coordination problems. 1.3.3. Adaptive approaches

We come now to adaptive approaches to learning, which have been extensively investigated over the last 15 years. In principle, there is a very wide range of adaptive formulations which are possible. As Sargent (1993) has emphasized, in replacing agents who are fully "rational" (i.e. have "rational expectations") with agents who possess bounded rationality, there are many ways to implement such a concept 11. One possibility is to extend the adaptive expectations idea by considering generalized expectation functions, mapping past observations of a variable into forecasts of future values of that variable, where the expectation function is required to satisfy certain reasonable axioms (including bounded memory in the sense of a fixed number of past observations). This approach was taken, in the context of nonstochastic models, in the early work by Fuchs (1979) and Fuchs and Laroque (1976), and the work was extended by Grandmont (1985) and Grandmont and Laroque (1986). Under appropriate assumptions it can be shown that the resulting dynamic systems can converge to perfect

It Sargent (1993) provides a wide-ranging overviewof adaptive learning. See Honkapohja(1996) for a discussion of Sargent's book. Adaptive learning is also reviewed in Evans and Honkapohja (1995a) and Marimon (1997). Marcet and Sargent (1988) and Honkapohja (1993) provide concise introductions to the subject.

Ch. 7: Learning Dynamics

465

foresight steady states or cycles. Using a generalization of adaptive expectations, the conditions under which learning could converge to perfect foresight cycles were also investigated by Guesnerie and Woodford (1991). A second approach is to regard agents as statisticians or econometricians who estimate forecasting models using standard statistical procedures and who employ these techniques to form expectations of the required variables. This line of research has naturally focussed on stochastic models, though it can also be applied to nonstochastic models. Perhaps the greatest concentration of research on learning in macroeconomics has been in this area, and this literature includes, for example, Bray (1982), Bray and Savin (1986), Fourgeaud, Gourieroux and Pradel (1986), Marcet and Sargent (1989c), and Evans and Honkapohja (1994b,c, 1995c). A third possibility is to draw on the computational intelligence 12 literature. Agents are modeled as artificial systems which respond to inputs and which adapt and learn over time. Particular models include classifier systems, neural networks and genetic algorithms. An example of such an approach is Arifovic (1994). Cho and Sargent (1996b) review the use of neural networks, and the range of possibilities is surveyed in Sargent (1993) 13. We discuss these approaches in the final section of this paper. Finally, we remark that not all approaches fall neatly into one of the classes we have delineated. For example, Nyarko (1997) provides a framework which is both eductive and adaptive. Agents have hierarchies of beliefs and actions are consistent with Bayesian updating. For a class of models which includes the cobweb model, conditions are given for convergence to the Nash equilibrium of the true model. The focus of this survey is on adaptive learning and the main emphasis is on statistical or econometric learning rules for stochastic models. We now illustrate this approach in the context of the economic examples above. 1.4. Examples o f statistical learning rules 1.4.1. Least squares learning in the Muth model

Least squares learning in the context of the Muth (or cobweb) model was first analyzed by Bray and Savin (1986) and Fourgeaud, Gourieroux and Pradel (1986). They ask whether the REE in that model is learnable in the following sense. Suppose that firms believe prices follow the process Pt = a + b'wt 1 + rh,

(12)

corresponding to the unique REE, but that a and b are unknown to them. Suppose that firms act like econometricians and estimate a and b by running least squares

12 This term is now more common than the equivalent term "artificial intelligence". 13 Spear (1989) takes yet another viewpoint and looks at bounded rationality and learning in terms of computational constraints.

G.W.Evansand S. Honkapohja

466

regressions ofpt on wt 1 and an intercept using data {Pi, wi}i=o. t l Letting (at l, bt-1) denote their estimates at t - 1, their forecasts are given by

E[_]pt = at i + bl_~wt-i.

(13)

The values for (at 1,bt-1) are given by the standard least-squares formula / ~-1

\ 1

( aZt z- 1i l)z=; l[) b , \i=1

t-1

"~

(~Zi-lPi,~ /

\i=1

where

z, = (1 w , ) .

(14)

/

Equations (4), (13) and (14) form a fully specified dynamic system, and we can ask: Will (at, bt) I ---+ (gl, t~l)I as t --+ oo.9 The above papers showed that if a < 1 then convergence occurs with probability 1. It is notable that this stability condition is weaker than the condition lal < 1 obtained under the eductive arguments. Since ct < 0 always holds in the Muth model (provided only that supply and demand curves have their usual slopes), it follows that least squares learning always converges with probability one in the Muth model. The stability condition can be readily interpreted using the expectational stability condition, formulated as follows. As earlier, we consider the mapping (11) from the perceived law of motion (PLM), parameterized by q~t = (a, bt), to the implied actual law of motion (ALM) which would be followed by the price process if agents held those fixed perceptions and used them to form expectations. Consider the differential equation dO aT - T(q~) - ~b, where z- denotes "notional" or "artificial" time. We say that the REE ~ = (fi, b~)~ is expectationally stable or E-stable if ~} is locally asymptotically stable under this differential equation. Intuitively, E-stability determines stability under a stylized learning rule in which the PLM parameters (a, b) are slowly adjusted in the direction of the implied A L M parameters. It is easily verified that for the Muth model the E-stability condition is simply a < 1, the same as the condition for stability under least-squares learning. The formal explanation for the reason why E-stability provides the correct stability condition is based upon the theory of stochastic approximation and will be given in later sections. It should be emphasized that in their role as econometricians the agents treat the parameters of Equation (12) as constant over time. This is correct asymptotically, provided the system converges. However, during the transition the parameters of the A L M vary over time because of the self-referential feature of the model. Bray and Savin (1986) consider whether an econometrician would be able to detect the transitional misspecification and find that in some cases it is unlikely to be spotted 14. 14 Bullard (1992) considers some recursive learning schemes with time-varying parameters. However, the specification does not allow the variation to die out asymptotically.

467

Ch. 7: Learning Dynamics 1.4.2. Least squares learning in a linear model with multiple R E E

Consider now the model (5) and suppose that agents have a PLM (perceived law of motion) of the form (15)

Yt = a + byt 1 + c'wt + ~k,

and that they estimate the parameters a, b, and c by a least squares regression ofyt on Yt-I, wt and an intercept. Letting = (a,,b,c;),

z; = ( 1 , y ,

1,w;),

the estimated coefficients are given by

(~t=~i~oZiZi)

(16)

(i~oZiYi)

and expectations are given by Etyt+l - at + byt + e;pwt,

where for convenience we are assuming that p is known. For simplicity, estimates q~t are based only on data through t - 1. Substituting this expression for E;yt+l into Equation (5) and solving for Yt yields the ALM (actual law of motion) followed by Yt under least squares learning. This can written in terms of the T-map from the PLM to the ALM: Yt = T(~)t)'zt + W((~t) vt, where

T(0 ) = r

W

= [ (1-[3b)-'6 ] , and \ (1 -/3b)-i (/f + [3cp) /

= (1-/3b) i.

(17) (18)

(19)

Note that fixed points ~ = (fi, b, U) ~ of T(~b) correspond to REE. The analysis in this and more general models involving least squares learning is facilitated by recasting Equation (16) in recursive form. It is well known, and can easily be verified by substitution, that the least squares formula (16) satisfies the recursion g)t = ~)t-1 + ~tR/Igt-l(Yt-I - ~ 1zt l), Rt = Rt I + ~'t(zt lZ; I -Rt-~),

(20)

for Yt = 1/t and suitable initial conditions 15. Using this RLS (Recursive Least Squares) set-up also allows us to consider more general "gain sequences" Yr. The dynamic system to be studied under RLS learning is thus defined by Equations (17)-(20).

15

Rt is an estimate of the moment matrix for zt. For suitable initial conditionsR t = t 1 ~ti=l 0 ZiZ~"

G. W. Evans and S. Honkapohja

468

Marcet and Sargent (1989c) showed that such dynamic systems fit into the framework o f stochastic recursive algorithms which could be analyzed using the stochastic approximation approach. This technique, which is described in the next section, associates with the system an ordinary differential equation (ODE) which controls the motion o f the system. In particular, only asymptotically stable zeros ~} o f the differential equation are possible limit points o f the stochastic dynamic system, such that Ct --+ ~}. In the case at hand the ODE is dO

dT R-IMz(O)(T(())-O)'

dz " - Mz(0)-R'

where M~(O) = lim E [zt(O)zt(O)'] t-~oo

for zt(O)' = (1,yt l(0),w~) and Y,(0) = T(~)'zt(O)+ W(O)vt. Here T(0 ) is given by Equation (18). Furthermore, as Marcet and Sargent (1989c) point out, local stability o f the ODE is governed by d 0 / d r = T(0) - 0. It thus follows that E-stability governs convergence of RLS learning to an REE o f the form (7). For the model at hand it can be verified that if there are two stationary REE o f the form (7), then only one o f them is E-stable, so that only one o f them is a possible limit point o f RLS learning. This is an example of how RLS learning can operate as a selection criterion when there are multiple REE 16

1.4.3. Learning a steady state We now consider adaptive learning of a steady state in nonstochastic nonlinear models of the form (3):

Yt =f(E[yt+l). The basic OG model and the extensions mentioned above fit this framework. One natural adaptive learning rule is to forecast Yt+l as the average o f past observed values, * = (t - - 1)-l ~ i = t-I oYi for t = 1,2,3 . . . . . Since the model is nonstochastic, i.e. Etyt+l the traditional adaptive expectations formula E[yt+x = Et*_yt + Y(Yt 1 - E[_lYt) for fixed 0 < ~/~< 1 also has the potential to converge to a perfect foresight steady state.

16 Marcet and Sargent (1989c) focused on set-ups with a unique REE. Evans and Honkapohja (1994b) showed how to use this framework in linear models with multiple REE and established the connection between RLS learning and E-stability in such models. In these papers convergence with probability 1 is shown when a "Projection Facility" is employed. Positive convergence results when learning does not incorporate a projection facility are given in Evans and Honkapohja (1998b), which also gives details for this example. See also Section 2.4.2 for discussion.

Ch. 7: LearningDynamics

469

Both of these cases are covered by the following recursive formulation, in which for convenience we use q~t to denote the forecast at time t ofyt+l: E[yt+l = q~t, where Ot = ()t-i + Yt(yt 1 -q)t-l), and where the gain sequence 7t satisfies

0
and

~yt=+oo. t-I

The choice gt = t 1 gives the adaptive rule in which the forecast ofyt+l is the simple average o f past values. The "fixed-gain" choice Yt = Y, for 0 < y ~< 1, corresponds to adaptive expectations. For completeness we will give the adaptive learning results both for the "fixed-gain" case and for the "decreasing-gain" case in which lim 7t

t--~oo

= 0,

which is obviously satisfied by Yt = t -I . In specifying the learning framework as above, we have followed what we will call the "standard" timing assumption, made in the previous subsection, that the parameter estimate q}t depends only on data through t - 1. This has the advantage of avoiding simultaneity between Yt and ~bt. However, it is also worth exploring here the implications of the "alternative" assumption in which q~t = ~ t - 1 q- Yt(Yt -q~t-1). We will see that the choice of assumptions can matter in the fixed-gain case, but is not important if Yt = t -1 or if Yt = Y > 0 is sufficiently small. Standard timing assumption. Combining equations, and noting that Yt = f(~t), the system under adaptive learning follows the nonlinear difference equation Ot = q~t-1 + Yt(f(q~t-~)- q}t-l).

(21)

Note that in a perfect foresight steady state, y¢ = ~ = ~, where ~ = f ( ~ ) . In the constantgain case we have ~t = (1 - y)(bt 1 + Yf(~t-~) and it is easily established that a steady state ~ = ~ is locally stable under Equation (21) if and only if ll + y ( f ' ( ¢ } ) - 1)l < 1, i.e. iff 1 - 2/7 < f / ( ~ ) < 1. Note that 1 - 2 / y --~ - o o as y -+ 0.17 Under the decreasing-gain assumption limt--+oo Yt = 0, it can be shown that a steady state ¢} = ~ is locally stable under (21) if and only i f f ' ( ~ ) < 1. Thus the stability condition under decreasing-gain corresponds to the small-gain limit of the condition for constant gain.

17 Guesnerie and Woodford (1991) show how to generalize this condition for equilibrium k-cycles.

470

G. W Evans and X Honkapohja

Alternative timing assumption. Now instead we have the implicit equation ¢ , = ¢,-~ + Y , ( J ( ~ , )

- ¢,-1).

In general there need not be a unique solution for Ot given Ot 1, though this will be assured if gt is sufficiently small and ifff(q~) is bounded. Assuming uniqueness and focusing on local stability near ~ we can approximate this equation by Ot - ~ = (1 - yt.fl(@))-l(1 - Yt)((Pt-I - ¢)). Under the fixed-gain assumption this leads to the stability condition that either f ' ( ~ ) < 1 o r f ' ( ~ ) > 2 / y - 1 (these possibilities correspond to the cases yf'(~) < 1 and y f f ( ~ ) > 1). Under decreasing gain the condition is again simplyf1(~) < 1 (which again is also the small-gain limit for the constant-gain case) 18. S u m m a r y under small gain. Thus for the small-gain case, i.e. assuming either decreasing gain or a sufficiently small constant gain, the condition for local stability under adaptive learning is not affected by the timing assumption and is simply f t ( ~ ) < 1. Returning to our various examples, it follows that the steady states in Figures la and lb are stable under adaptive learning. In Figure lc, the high-output, low-inflation steady state is locally stable, while the low-output, high-inflation steady state is locally unstable. Finally, in Figure ld the high- and low-output steady states nL and nH are locally stable, while nu is locally unstable. As is clear from the above discussion, in the case of sufficiently large constant gains the stability condition is more complex and can depend sensitively on timing assumptions. [Lettau and Van Zandt (1995) analyze the possibilities in detail for some frameworks.] Our treatment concentrates on the decreasing-gain case in large part because in stochastic models, such as the linear models discussed above, decreasing gain is required to have the possibility of convergence to an REE. This also holds if intrinsic noise is introduced into the nonlinear models of this section, e.g. changing model (2) to Yt = E t f ( Y t + l ) + vt. Even if Ut is iid with arbitrarily small support, the above learning rules with constant gain cannot converge to an REE while with decreasing gain and appropriate assumptions we still obtain (local) convergence to the (noisy) steady state if the stability condition is met 19. Finally, we remark that the key stability condition, f f ( ~ ) < 1 for stability of a steady state under adaptive learning with small gain, corresponds to the E-stability condition. In this case the PLM is taken to be simply Yt = ~ for an arbitrary ~b. Under this PLM the appropriate forecast Ei*f(Yt+l) is f(~b) and the implied A L M is Yt = f(q~). The

18 The formal results for the decreasing-gain case can be established using the results of Section 2. Alternatively, for a direct argument see Evans and Honkapohja (1995b). 19 However, we do think the study of constant-gain learning is important also for stochastic models. For example, Evans and Honkapohja (1993a) explore its possible value if either (i) the model is misspecified or (ii) other agents use constant-gain learning.

Ch. 7: LearningDynamics

471

T-map from the PLM to the A L M is just f ( 0 ) , so the E-stability differential equation is dqVdT = f ( q ~ ) - q~, giving the stability conditionf'(~}) < 1. Stability under adaptive learning and E-stability for cycles and sunspots in nonlinear models are reviewed in Section 4. 1.4.4. The seignorage model o f inflation

The preceding analysis of learning a steady state is now illustrated by a summary discussion of the implications of adaptive learning for the seignorage model of inflation. This model is chosen because of its prominence in the macroeconomic literature. The seignorage model is discussed, for example, in Bruno (1989) and in Blanchard and Fischer (1989), pp. 195-201. In this model there is a fixed level of government expenditures g financed by seignorage, i.e. g = (Mt - Mt O/pt or g = Mt/pt - (Pt 1/Pt)(Mt-l/Pt-1). The demand for real balances is given by Mt/pt = S(E[pt+I/pt) and it is assumed that S t < 0. The model can be solved for inflation :rt = pt~ot-i as a function of Et*:rt+l ~ E[pt+l/pt and El_ 1srt oi" equivalently (in the nonstochastic case) can be written in terms o f M S p t and Et(Mt+l~Ot+l ). For the Overlapping Generations version of this model, given as Extension 1 of Section 1.2.3, nt = Mt/pt and the model was written as nt = f ( E [ n t + l ) . In order to apply directly the previous section's analysis of adaptive learning, we initially adopt this formulation. The central economic point, illustrated in Figure lc, is that for many specifications there can be two steady states: a high real balances (low-inflation) steady state nt = nil, err = :q satisfying 0 < f ( n H ) < 1, and a low real balances (high-inflation) steady state nt = hE, Jrt = Sr2 satisfying f~(nL) > 1. In the economic literature [e.g. Bruno (1989)] the possibility has been raised that convergence to nL provides an explanation o f hyperinflation. The analysis of the previous section shows that this is unlikely unless the gain parameter is large (and the alternative timing assumption used). In the smallgain case, the low-inflation/high real balances steady state nH is locally stable under learning and the high-inflation/low real balances steady state nL is locally unstable, in line with E-stability. Suppose instead that the model is formulated in terms of inflation rates. In this case the reduced form is :rt -

S(Et l,Tgt) S(E?ev~+I) - g"

In the usual cases considered in the literature we now have that h(~) -= S ( ~ ) / ( S ( s r ) - g ) is increasing and convex (for Jr not too large), and of course for small deficits g we have the two steady-state inflation rates as fixed points of h(~). The low-inflation (high real balances) steady state ~1 then satisfies 0 < h~(~l) < 1 and the high-inflation (low real balances) steady state ~2 satisfies h~(~2) > 1. E-stability for the PLM ~t = q) is determined by d 0 / d r = h(q~) - q~, so that again the low-inflation steady state is E-stable and the high-inflation steady state is not.

472

G. W, Evans and S. Honkapohja

Under adaptive learning of the inflation rate we assume that E[~t+l = 0t with either the standard timing Ot = Or-1 + gt(~t 1 - q~t-1) or the alternative timing assumption Ot = Ot-1 + gt(J~t - Ct-1). This set-up has been examined in detail by Lettau and Van Zandt (1995). They find that in the constant-gain case, for some values of y, the high-inflation steady state can be stable under learning under the alternative timing assumption. However, their analysis confirms that with small constant gain or decreasing gain the low-inflation steady state is always locally stable under adaptive learning and the high-inflation steady state is always locally unstable under adaptive learning. To conclude the discussion we make two further points. First, in some papers the learning is formulated in terms of price levels, rather than real balances or inflation rates, using least squares regressions of prices on lagged prices. Such a formulation can be problematic since under systematic inflation the price level is a nonstationary variable 2o. Second, the seignorage model has been subject to experimental studies, see Marimon and Sunder (1993) and Arifovic (1995). Their results suggest convergence to the low-inflation, high-output steady state. Such results accord with the predictions of decreasing or small-constant-gain learning. 1.5. Adaptive learning and the E-stability principle

We have seen that when agents use statistical or econometric learning rules (with decreasing gain), convergence is governed by the corresponding E-stability conditions. This principle, which we will treat as a unifying principle throughout this paper, can be stated more generally. Consider any economic model and consider its REE solution. Suppose that a particular solution can be described as a stochastic process with a particular parameter vector } (e.g. the parameters of an autoregressive process or the mean values over a k-cycle). Under adaptive learning our agents do not know ~} but estimate it from data using a statistical procedure such as least squares. This leads to estimates ~bt at time t and the question is whether q~t ---+~ as t ---, c~. For a wide range of economic examples and learning rules we will find that convergence is governed by the corresponding E-stability condition, i.e. by local asymptotic stability of ~}under the differential equation

dO dr

T(~0) - q},

(22)

where T is the mapping from the PLM q~ to the implied ALM T(q~). The definition of E-stability based on the differential equation (22) is the formulation used in Evans (1989) and Evans and Honkapohja (1992, 1995a). This requirement of E-stability is less strict than the requirement of iterative E-stability based on

20 See also the discussion in Section 5.1.4.

Ch. 7: LearningDynamics

473

Equation (10)21. As became evident from the results o f Marcet and Sargent (1989c), it is the differential equation formulation (22) which governs convergence of econometric learning algorithms. This form o f E-stability has been systematically employed as a selection rule with multiple REE in linear models by Evans and Honkapohja (1992, 1994b) and Duffy (1994), and in nonlinear models by Evans (1989), Marcet and Sargent (1989a), and Evans and Honkapohja (1994c, 1995b,c). O f course, there may be alternative ways to parameterize a solution and this may affect stability under learning. In particular, agents may use perceived laws of motion that have more parameters than the REE o f interest, i.e. overparameterization o f the REE may arise. This leads to a distinction between weak vs. strong E-stability. An REE is said to be weakly E-stable if it is E-stable as above, with the perceived law o f motion taking the same form as the REE. Correspondingly, we say that an REE is strongly E-stable if it is also locally E-stable for a specified class o f overparameterized perceived laws o f motion. (The additional parameters then converge to zero.) 22 We remark that, since it may be possible to overparameterize solutions in different ways, strong E-stability must always be defined relative to a specified class o f PLMs 23. Finally, as a caveat it should be pointed out that, although the bulk of work suggests the validity o f the E-stability principle, there is no fully general result which underpins our assertion. It is clear from the preceding section that the validity o f the principle may require restricting attention to the "small-gain" case (gain decreasing to zero or, if no intrinsic noise is present, a sufficiently small constant gain). Another assumption that will surely be needed is that the information variables, on which the estimators are based, remain bounded. To date only a small set o f statistical estimators has been examined. We believe that obtaining precise general conditions under which the E-stability principle holds is a key subject for future work.

1.6. Discussion of the literature In the early literature the market model o f Muth (1961), the overlapping generations model and some linear models were the most frequently used frameworks to analyze learning dynamics. Thorough treatments o f learning dynamics in the Muth model were given by Bray and Savin (1986) and Fourgeaud, Gourieroux and Pradel (1986). Interestingly, without mentioning rational expectations, Carlson (1968) proposed that price expectations be formed as the mean o f observed past prices in study the linear

21 There is a simple connection between E-stability based on Equation (22) and the stricter requirement of iterative E-stability. An REE ~ is E-stable if and only if all eigenvalues of the derivative map DT(~) have real parts less than one. For iterative E-stability the requirement is that all eigenvalues of DT(~) lie inside the unit circle. 22 Early applications of the distinction between weak and strong stability, introduced for iterative E-stability in Evans (1985), include Evans and Honkapohja (1992), Evans (1989) and Woodford(1990). 23 In an analogous way, E-stability can also be used to analyze non-REE solutions which are tmderparameterized. See Section 5.3 below.

474

(7. W. Evans and S. Honkapohja

non-stochastic cobweb (or Muth) model. Auster (1971) extends the convergence result for the corresponding nonlinear setup. Lucas (1986) is an early analysis of the stability of steady states in an OG model. Grandmont (1985) considers the existence of deterministic cycles for the basic OG model. He also examines learning using the generalizations of adaptive expectations to finite-memory nonlinear forecast functions. Guesnerie and Woodford (1991) propose a generalization to adaptive expectations allowing possible convergence to deterministic cycles. Convergence of learning to sunspot equilibria in the basic OG model was first discovered by Woodford (1990). Linear models more general than the Muth model were considered under learning in the early literature. Marcet and Sargent (1989c) proposed a general stochastic framework and technique for the analysis of adaptive learning. This technique, studied e.g. in Ljung (1977), is known as recursive stochastic algorithms or stochastic approximation. (Section 2 discusses this methodology.) Their paper includes several applications to well-known models. Margaritis (1987) applied Ljung's method to the model of Bray (1982). Grandmont and Laroque (1991) examined learning in a deterministic linear model with a lagged endogenous variable for classes of finite memory rules. Evans and Honkapohja (1994b) considered extensions of adaptive learning to stochastic linear models with multiple equilibria. Other early studies of learning include Taylor (1975) who examines learning and monetary policy in a natural rate model, the analysis of learning in a model of the asset market by Bray (1982), and the study Blume and Easley (1982) of convergence of learning in dynamic exchange economies. Bray, Blume and Easley (1982) provide a detailed discussion of the early literature. The collection Frydman and Phelps (1983) contains several other early papers on learning. Since the focus of this survey is on adaptive learning in stochastic models we will not comment here on the more recent work in this approach. The comments below provide references to approaches and literature that will not be covered in detail in later sections. For Bayesian learning the first papers include Turnovsky (1969), Townsend (1978, 1983), and McLennan (1984). Bray and Kreps (1987) discuss rational learning and compare it to adaptive approaches. Nyarko (1991) shows in a monopoly model that Bayesian learning may fail to converge if the true parameters are outside the set of possible prior beliefs. Recent papers studying the implications of Bayesian learning include Feldman (1987a,b), Vives (1993), Jun and Vives (1996), Bertocchi and Yong (1996) and the earlier mentioned paper by Nyarko (1997). A related approach is the notion of rational beliefs introduced by Kurz (1989, 1994a,b). The collection Kurz (1997) contains many central papers in this last topic. The study of finite-memory learning rules in nonstochastic models was initiated in Fuchs (1977, 1979), Fuchs and Laroque (1976), and Tillmann (1983) and it was extended in Grandmont (1985) and Grandmont and Laroque (1986). These models can be viewed as a generalization of adaptive expectations. A disadvantage is that the finite-memory learning rules cannot converge to an REE in stochastic models, cf.

475

Ch. 7: Learning Dynamics

e.g. Evans and Honkapohja (1995c). Further references of expectation formation and learning in nonstochastic models are Grandmont and Laroque (1990, 1991), Guesnerie and Woodford (1991), Moore (1993), B6hm and Wenzelburger (1995), and Chatterji and Chattopadhyay (1997). Learning in games has been subject to extensive work in recent years. A small sample of papers is Milgrom and Roberts (1990, 1991), Friedman (1991), Fudenberg and Kreps (1993, 1995), Kandori, Mailath and Rob (1993), and Crawford (1995). Recent surveys are given in Marimon and McGrattan (1995), Marimon (1997), and Fudenberg and Levine (1998). Kirman (1995) reviews the closely related literature on learning in oligopoly models. Another related recent topic is social learning, see e.g. Ellison and Fudenberg (1995) and Gale (1996).

2. General methodology: recursive stochastic algorithms 2.1. General setup and assumptions In the first papers on adaptive learning, convergence was proved directly and the martingale convergence theorem was the basic toot, see e.g. Bray (1982), Bray and Savin (1986), and Fourgeand, Gourieroux and Pradel (1986). Soon it was realized that it is necessary to have a general technique to analyze adaptive learning in more complex models. Marcet and Sargent (1989b,c) and Woodford (1990) introduced a method, known as stochastic approximation or recursive stochastic algorithms, to analyze the convergence of learning behavior in a variety of macroeconomic models. A general form of recursive algorithms can be described as follows. To make economic decisions the agents in the economy need to forecast the current and/or future values of some relevant variables. The motions of these variables depend on parameters whose true values are unknown, so that for forecasting the agents need to estimate these parameters on the basis of available information and past data. Formally, let Ot E N ~ be a vector of parameters and let Ot = Ot-I q- ytQ(t, Ot-l,Xt)

(23)

be an algorithm describing how agents try to learn the true value of 0. It is written in a recursive form since learning evolves over time. Here Yt is a sequence of"gains", often something like Yt = t 1. Xt C tRk is a vector of state variables. Note that in general the learning rule depends on the vector of state variables. This vector is taken to be observable, and we will postulate that it follows the conditionally linear dynamics Y t = A ( O t - l ) Y t l + B(Ot

l)mt,

(24)

where Wt is a random disturbance term. The detailed assumptions on this interrelated system will be made below 24. 24 Note that somewhat different timing conventions are used in the literature. For example, in some expositions Wt 1 may be used in place of Wt in Equation (24). The results are unaffected as long as W~ is an iid exogenousprocess.

476

G. W. Evans and S. Honkapohja

Note that the least squares learning systems in Section 1.4 can be written in the form (23) and (24). For example, consider the system given by Equations (17) and (20). Substituting Equation(17) into Equation (20) and setting St-i = Rt yields an equation of the form (23), with O[ = (at, bt, c;, vec(St)') and X/ = (1,yt 1, w~,yt-2, w;_ 1, oi-0, and it can be checked that Xt follows a process of the form (24) 2s.

2.1.1. Notes on the technical literature The classical theory of stochastic approximation, see Robbins and Monro (1951) and Kiefer and Wolfowitz (1952), was developed for models without full state variable dynamics and feedback from parameter estimates. Recent expositions of stochastic approximation are given e.g. in Benveniste, Metivier and Priouret (1990), Ljung, Pflug and Walk (1992), and Kushner and Yin (1997). A widely cited basic paper is Ljung (1977), which extended stochastic approximation to setups with dynamics and feedback. Ljung's results are extensively discussed in the book by Ljung and S6derstr6m (1983). A further generalization of Ljung's techniques is presented in Benveniste, Metivier and Priouret (1990). A somewhat different approach, based on Kushner and Clark (1978), is developed in Kuan and White (1994). An extension of the algorithms to infinite-dimensional spaces is given in Chen and White (1998). Stochastic approximation techniques were used by Arthur, Ermoliev and Kaniovski (1983, 1994) to study generalized urn schemes. Evans and Honkapohj a (1998a,b) and the forthcoming book Evans and Honkapohj a (1999a) provide a synthesis suitable for economic theory and applications. The exposition here is based primarily on these last-mentioned sources. Other useful general formulations are Ljung (1977), Marcet and Sargent (1989c), the appendix of Woodford (1990), and Kuan and White (1994).

2.2. Assumptions on the algorithm Let Ot c IRd be a vector of parameters and Xt E IRk be a vector of state variables. At this stage it is convenient to adopt a somewhat specialized form of Equation (23), so that the evolution of 0t is assumed to be described by the difference equation 0 t = Or_ 1 q- ]/tJ-~(Ot 1,Xt) + ]/2tPt(Ot-l,~t ).

(25)

Here ~(.) and Pt(') are two functions describing how the vector 0 is updated (the second-order term Pt(') is often not present). Note that in Equation (25) the function Q(t, Ot-l,Xt) appearing in Equation (23) has been specialized into first- and secondorder terms in the gain parameter Yr.

25 Here vec denotes the matrix operatorwhich forms a column vector from the matrix by stacking in order the columns of the matrix.

Ch. 7: Learning Dynamics

477

Next we come to the dynamics for the vector of state variables. In most economic models the state dynamics are assumed to be conditionally linear, and we postulate here that Xt follows Equation (24). Without going into details we note here that it is possible to consider more general situations, where Xt follows a Markov process dependent on 0t-l. This is needed in some applications, and the modifications to the analysis are presented in detail in Evans and Honkapohja (1998a). For local convergence analysis one fixes an open set D C N J around the equilibrium point of interest. The next step is to formulate the assumptions on the learning rule (25) and the state dynamics (24). We start with the former and postulate the following: (A.1). 7t is a positive, nonstochastic, nonincreasing sequence satisfying O(3

O<3

t--I

t-,1

(A.2). For any compact Q c D there exist C1, C2, ql and q2 such that VO E Q and Vt: (i) lT-{(0,x)l ~< c1(1 + Ix]q1),

O0 I;,(0,x)l ~< c2(1 + Ixlq2) (A.3). For any compact Q c D the function 7-/(0,x) satisfies V0, 0 ~ E Q and Xl,X2 E ~ k .

(i)

107-[(O,xl)/Ox-O~(O,xz)/Ox I <~L~ txl -x2l,

(ii) I~(0,0)-~(0',0)1 ~
Note that (A.1) is clearly satisfied for Yt = C/t, C constant. (A.2) imposes polynomial bounds on 7-/(.) and Pt('). (A.3) holds provided 7-[(O,x) is twice continuously differentiable (denoted as C 2) with bounded second derivatives on every Q. For the state dynamics one makes the assumptions (B.1). Wt is iid with finite absolute moments. (B.2). For any compact subset Q c D :

sup Ig(0)l ~ M,

O~Q

sup IA(0)I ~ p < 1,

OcQ

for some matrix norm 1-1, and A(O) and B(O) satisfy Lipschitz conditions on Q. R e m a r k : In (B.2) the condition on A(O) is a little bit stronger than stationarity.

However, if at some 0* the spectral radius (the maximum modulus of eigenvalues)

478

G W.Evans and S. Honkapohja

satisfies r(A(O*)) < 1 then the condition on A(O) in (B.2) holds in a neighborhood of 0". These are pretty general assumptions. In specific models the situation may be a great deal simpler. One easy case arises when the state dynamics Xt do not depend on the parameter vector 0~ 1. A classical special case in stochastic approximation, first discussed by Robbins and Monro (1951), arises when the distribution of the state variable Xt+l can depend on 0t but is otherwise independent of the history Xt,Xt-l,.

. . , Ot, Ot 1 . . . . .

In general the recursive algorithm consisting of Equations (25) and (24) for 0~ and Xt, respectively, is a nonlinear, time-varying stochastic difference scheme. At first sight the properties of such systems may seem hard to analyze. It turns out that, due the special structure of the equation for the parameter vector, the system can be studied in terms of an associated ordinary differential equation which is derived as follows: (i) Fix 0 and define the corresponding state dynamics

fft( O) : A( O)~_I (0) + B( O)Wt. (ii) Consider the asymptotic behavior of the mean of 7-{(0,Xt(0)), i.e.

h(O)

=

lim t---+O0

ET~(O,2,(O)).

The associated differential equation is then defined as dO dr - h(O). Given assumptions (A. 1)-(A.3) and (B. 1)-(B.2) it can be shown that the fimction h(O) is well-defined and locally Lipschitz.

2.3. Convergence: the basic results 2.3.1. ODE approximation The basic idea in the "ODE method" is to write the algorithm in the form

Ot+l = Ot + yt+lh( Ot) + et,

where

e, = 7,+1 [~(0,,X~+i) - h(O,) + ~',+~p,+l(O,,X,+l)]. Thus, et is essentially the approximation error between the algorithm and a standard discretization of the associated ODE. In proving the validity of the method the main difficulty is in showing that the cumulated approximation errors ~ et are bounded. The precise details of proof are very lengthy indeed [see Evans and Honkapohja (1998a) for an exposition], and they must be omitted here.

Ch. 7: Learning Dynamics

479

2.3.2. Asymptotic analysis We now make the assumption that we have an equilibrium point o f the associated ODE 0* which is locally asymptotically stable for the ODE. It turns out that, in a particular sense, the time path o f 0t generated by the algorithm will converge to 8", provided for its starting point (x, a) the component a is sufficiently close to 0". In doing this the Lyapunov stability theory for ODEs in the form of so-called converse Lyapunov theorems is needed m6. Suppose that 0* is locally asymptotically stable for the associated ODE d g / d r = h(0(r)). Then a version of the converse Lyapunov theorems states that on the domain of attraction ~ o f 0* for the ODE there exists a C e Lyapunov function U(O) having the properties U(8)>OforallgE~, 8~8", (a) U(8*)=O, 0~8", (b) U'(8) h ( 8 ) < O f o r a l l g E ~ , (c) U(8) --~ oc if 8 ---+ 0 ~ or 181 -+ 0<3.27 Introduce now the notation K(c) = {0; U(8) ~< c}, c > 0 for the contour sets of the Lyapunov function. Also let P~ ..... be the probability distribution of (X~, 8t)t>~n with X~ = x, On = a. The following theorem is the basic convergence result for recursive stochastic algorithms (25) and (24): T h e o r e m 1. Let 8" be an asymptotically stable equilibrium point of the ODE d 8 / d r = h(0(r)). Suppose assumptions (A) and (B) are satisfied on D = int(K(c)) for some c > O. Suppose that for 0 < cl < c2 we have K(c2) C D. Then (i) Va E K(Cl),n >/O,x one has

Pn, x,a{Ot leaves K(c2) infinite time or Ot ---+8*} = 1, and (ii) .for any compact Q c D there exist constants' B2 and s such that Va E Q, n >~ 0, x"

P~ .... (8, --+ 8*} ~ x -O2 (a + fxlS)j(n), where J(n) is a positive decreasing sequence with limn~o~ J(n) = O.

Remark:J(n) isinfactgivenbyJ(n)=

(

1+

~ t=n+l

Z2

~ t=n+l

)

7~ •

To interpret the results one first fixes the contour sets K(cl) C K(c2). The theorem states two things. First, the algorithm either converges to 0* or diverges outside K(c2). Second, the probability o f converging to 8" is bounded from below by a sequence o f

26 For converses of the Lyaptmov stability theorems see Hahn (1963, 1967). 27 0~ denotes the boundary of ~.

G.W. Evans and S. Honkapohja

480

numbers which tends to unity as n ~ oc. In other words, if at some large value of t the algorithm has not gone outside K(c2) then it will converge to 0* with high probability. 2.4. Convergence: fi~rther discussion 2.4.1. Immediate consequences The following two results are special cases for obtaining statements about convergence when starting at time 0. The first result is an immediate consequence of the second part of Theorem 1: Corollary 2. Suppose Yt = ~g[, where ~ satisfies (A.1). Let the initial value o f 0 belong to some compact Q c D. Then V6 > 0 : 3~* such that V0 < ~ < ~* and aEQ:

Po, x,a{ot ~ o*} >1 1 - 6 . This is the case of slow adaption. For slow enough adaption the probability of convergence can be made "very close" to one. For general adaption speeds and with additional assumptions it is possible to obtain convergence with positive probability: Corollary 3. Assume that O* is locally asymptotically stable for the ODE. Assume that each component of Wt is' either a random variable with positive continuous density or else is constant. Fix a compact set Q c D, such that O* E int(Q), and a compact set J c II{k. Suppose that for every Oo E Qo and Xo E Jo in some sets Qo and Jo, and for every n > O, there exists a sequence Wo. . . . . WT, with T ) n, such that Or E int(Q) and X r E int(J). Then

P0..... { 0 ~ 0 " } > 0 for all a C Qo and x c Jo. It must be emphasized that it is not in general possible to obtain bounds close to unity even for the most favorable initial conditions at this level of generality. The reason is that for small values of t the ODE does not approximate well the algorithm. For early time periods sufficiently large shocks may displace Ot outside the domain of attraction of the ODE. 2.4.2. Algorithms with a projection facility In the earlier literature [e.g., Marcet and Sargent (1989b,c), Evans and Honkapohja (1994b,c, 1995c)] this problem was usually avoided by an additional assumption, which is called the Projection Facility (PF). It is defined as follows: For some 0 < Cl < c2, with K(c2) c D, the algorithm is followed provided Ot c int(K(c2)).

Ch. 7." Learning Dynamics

481

Otherwise, it is projected to some point in K(ci). An alternative to PF, see e.g. Ljung (1977), is to introduce the direct boundedness assumption that the algorithm visits a small neighborhood o f the equilibrium point infinitely often. This condition is often impossible to verify. The hypothesis of a PF has been criticized as being inappropriate for decentralized markets [see Grandmont (1998), Grandmont and Laroque (1991) and Moreno and Walker (1994)]. The basic results above do not invoke the projection facility which has in fact a further strong implication. With a PF the probability for convergence to a stable equilibrium point can be made equal to unity: Corollary 4. Consider the general algorithm augmented by a projection facility. Then

Vx, a : Po, x,a{Ot --+ 0"} = 1.

We omit the proof, which is a straightforward consequence of the main theorems, see Evans and Honkapohja (1998a). Finally, we note here that almost sure local convergence can be obtained in some special models, provided that the support o f the random shock is sufficiently small, see Evans and Honkapohja (1995c). Also for nonstochastic models there is no need to have a PF when one is interested in local stability. However, for some nonstochastic models problems with continuity o f the functions in the learning algorithm may arise 28.

2.5. Instability results We will now consider the instability results which will, broadly speaking, state the following: (i) The algorithm cannot converge to a point which is not an equilibrium point o f the associated ODE, and (ii) the algorithm will not converge to an unstable equilibrium point o f the ODE. We will have to adopt a new set o f conditions 29. Let again Ot C IRa be a vector o f parameters and adopt the general form (23) for the algorithm, i.e. Ot = Ot 1 + ~/tQ(t, Ot-I ,~t). Below we will impose assumptions directly on Q(.). Again, Xt E 1Rk is the vector of observable state variables with the conditionally linear dynamics (24), i.e. Xt = A(Ot_i)Xt_l + B(Ot_l)Wt. Select now a domain D* C IRd such that all the eigenvalues o f A(O) are strictly inside the unit circle V0 c D*. The final domain o f interest will be an open and connected set

28 For example, the moment matrix in recursive least squares can become singular asymptotically. See Grandmont and Laroque (1991) and Grandmont (1998) for a discussion. Evans and Honkapohja (1998b) and Honkapohja (1994) discuss the differences between stochastic and nonstuchastic models. 29 The main source for the instability results is Ljung (1977). (We will adopt his assumptions A.) A slightly different version of Ljung's results is given in the appendix of Woodford (1990). For an instability result with decreasing gain in a nonstochastic setup see Evans and Honkapohja (1999b).

G. W. Evans and S. Honkapohja

482

D C D* and the conditions below will be postulated for D. We introduce the following assumptions: (C.1). Wt & a sequence o f independent random variables with IWtl < C with probability one f o r all t. (C.2). Q(t, O,x) is C 1 in (O,x) f o r 0 E D. For fixed (O,x) the derivatives are bounded in t. (C.3). The matrices A(O) and B(O) are Lipschitz on D. (C.4). limt~oo EQ(t, O,f(t(O))= h(O) exists f o r 0 E D, where Xt(O) = A(O)Xt_l(O) + B(O)Wt. (C.5). Yt is a decreasing sequence with the properties ~ - ~ Yt = oo, ~-2~ ~t < oc Jbr

It 1] < oo.

some p, and lim suPt~o ~ ~ - ~

With these assumptions the following theorem holds [see Ljung (1977) for a proof]: T h e o r e m 5. Consider the algorithm with assumptions (C). Suppose at some point O* E D we also have the validity o f the conditions (i) Q(t, 0*,Xt(0*)) has a couariance matrix that is bounded below by a positive definite matrix, and (ii) EQ(t, O, Xt(O)) is C 1 in 0 in a neighborhood o f O* and the derivatives converge uniformly in t. Then if h(O*) ~ 0 or if Oh(O*)/O0 ~ has an eigenualue with positive real part, Pr(O~ --~ 0 " ) = 0. In other words, the possible rest points of the recursive algorithm consist of the locally stable equilibrium points of the associated ODE 30. It is worth mentioning the role of condition (i) in the theorem. It ensures that at even large values of t some random fluctuations remain, and the system cannot stop at an unstable point or nonequilibrium point. For example, if there were no randomness at all, then with an initial value precisely at an trustable equilibrium the algorithm would not move off that point. If the system is nonstochastic the usual concept of instability, which requires divergence from nearby starting points, is utilized instead. 2.6. Further remarks" The earlier stability and these instability results are the main theorems from the theory of recursive algorithms that are used in the analysis o f adaptive learning in economics. 30 This assumes that the equilibrium points are isolated. There are more general statements of the result.

Ch. 7: LearningDynamics

483

We note here that there exist some extensions yielding convergence to more general invariant sets of the ODE under further conditions. I f the invariant set consists of isolated fixed points and the dynamics can be shown to remain in a compact domain, then it is possible to prove a global result that learning dynamics converges to the set of locally stable fixed points 31. Another global convergence result for a unique equilibrium under rather strong conditions will be discussed in a moment. As already mentioned in Section 1.5 in the Introduction a simpler way of obtaining the appropriate convergence condition for adaptive learning is the concept of expectational stability. The method for establishing the connection between E-stability and convergence of real-time learning rules is obviously dependent on the type of the PLM that agents are presumed to use. For nonlinear models one usually has to be content with specific types of REE, whereas for linear models the entire set of REE can be given an explicit characterization and one can be more systematic. We remark that the parameterization of the REE and the specification of who is learning and what (i.e. perceived law of motion) can in principle affect the stability conditions. This situation is no different from other economic models of adjustment outside a full equilibrium. However, it is evident that the local stability condition that the eigenvalues of T(.) have real parts less than one is invariant to 1-1 transformations q~ --+ /3 = f ( ~ ) , where f and f-1 are both C 1 . Recall, however, that if agents overparameterize the solution this may affect the stability condition, which is captured by the distinction between weak and strong E-stability.

2.7. Two examples 2.7.1. Learning noisy steady states We consider univariate nonlinear models of the form

Yt = H(E[ G(yt+,, or+l), or),

(26)

where ut is an iid shock. Here E[G(yt+t, vt+l) denotes subjective expectations of a (nonlinear) function of the next period's value of Yt+l and the shock ut+l. Under REE E[ G(yt+ I, Vt+l) = Et G(y~+l, Vt+l), the true conditional expectation. As mentioned previously, various overlapping generations models provide standard examples that fit these frameworks.

31 Woodford(1990) and Evans, Honkapohjaand Marimon (1998a) are examples of the use of this kind of result.

484

G. W. Evans and S. Honkapohja

A noisy steady state for Equation (26) is given by 0 = E G ( y ( o ) , v), such that O = EG(H((), v), o),

Yt = H(O, v~).

Note that Yt is an iid process. For learning a steady state the updating rule is O, = Ot-i +t l [G(yt, v t ) - O t - l ] ,

(27)

which is equivalent to taking sample means. Contemporaneous observations are omitted for simplicity, so that we set E'[G(yt+l,vt+l) = 0t 132. Thus yt = H(Ot l,Vt) which is substituted into Equation (27) to obtain a stochastic approximation algorithm o f the form (24), (25). The convergence condition for such a "noisy" steady state is

~--~E(G(H(O, v), v) < 1.

This condition can also be obtained from the E-stability equation, since the T-map is in fact T(O) = E ( G ( H ( O , v), v). The extension to the E-stability of cycles is discussed in Evans and Honkapohja (1995c) and in Section 4 below.

2. 7.2. A model with a unique R E E The market model of Muth (196 l) was introduced in Section 1.2.1 above. We consider briefly its generalization to simultaneous equations, e.g. to multiple markets, discussed earlier in Evans and Honkapohja (1995a, 1998b): Yt = ~ + AEt_lYt + Cwt,

wt = Bwt 1 + ot.

Here Yt is an n x 1 vector o f endogenous variables, wt is an observed p x 1 vector o f stationary exogenous variables, and vt is a p x 1 vector of white noise shocks with finite moments. The eigenvalues of Bp×p are assumed to lie inside the unit circle. For simplicity, the matrix B is assumed to be known. E[_jyt denotes the expectations o f agents held at time t - 1 based on their perceived law o f motion. Assume also that I - A is invertible.

32 This avoids a simultaneity betweeny~ and 0t, see Section 1.4.3 for further discussion and references.

485

Ch. 7: Learning Dynamics

This model has a unique RISE Yt = ?t + [~wt 1 + ~h,

where fi = ( I - A ) 1~, ~ = ( / _ A ) - I C B and ~Tt = Cut. Is this REE expectationally stable? Consider perceived laws of motion of the form Yt = a + bwt-1 + ~lt

for arbitrary n × 1 vectors a and n x p matrices b. The corresponding expectation function is E ; l Y t = a + bwt 1, and one obtains the actual law of motion Yt = (t~ + A a ) + ( A b + CB)wt_l + ~t,

where ~t = Cot. The T mapping is thus T ( a , b) = ( # + A a , A b + CB).

E-stability is determined by the differential equation da

db

dv - Iz+(A-I)a,

dv - CB+(A-I)b.

This system is locally asymptotically stable if and only if all eigenvalues of A negative real parts, i.e. if the roots of A have real parts less than one. In real-time learning the perceived law of motion is time-dependent:

I have

yt = at-i + bt-l Wt-1 + ~]t,

where the parameters at and bt are updated running recursive least squares (RLS). Letting ¢ = (a, b), z; = (1, w;), et = Y t - (9t-tzt-1, RLS can be written 1

-1

t

Rt = Rt 1 + t l(zt_lz~_l - Rt-I).

This learning rule is complemented by the short-run determination of the value for Yt which is Yt = T(¢t 1)zt 1 + Cot,

where T(¢) = T ( a , b) as given above. In order to convert the system into standard form (25) we make a timing change in the system governing Rt. Thus we set SL-1 Rt, so that =

st =sL 1 +t-'(ztz;-S,

1 ) + t -2

(')

(ztz;-st_l)

The last term is then of the usual form with pt(St 1, zt) = -TZT(ztzt t - St-l). The model is of the form (25) with Ot = vec(q);, St) and X / = (1, w;, w; I, vt). The dynamics for

486

G. W. Evans and S. Honkapohja

the state variable are driven by the exogenous processes and one can verify that the basic assumptions for the convergence analysis are met. The associated ODE can be obtained as follows. Substituting in for et and Yt one obtains for Ot

O; = O ; l + t ls,~z,-~ [T(0, 1)z, 1 + C v , - 0 t

~z~_~]'

= ~)~-1 -}- t-latllzt-lZ~l [Z(~t-l) - ~t-1]' -t- t-lStllzt lVffC'. Taking expectations and limits one obtains the ODE as d q ~ ' - R 1Mz[T(q))-q)]', dT

dR-Mz-R, dr

where Mz denotes the positive definite matrix Mz = Eztz~. The second equation is independent of q~ and it is clearly globally asymptotically stable. Moreover, since R -+ Mz, the stability of the first equation is governed by the E-stability equation

d~ - ~r(0) - 0. dr Its local stability condition is that the eigenvalues of A have real parts less than one, see above. Thus the E-stability condition is the convergence condition for the RLS learning algorithm in this model. In the next section we establish a global result that is applicable for this model.

2.8. Global convergence In this section we provide a stronger set of conditions than conditions (A) and (B) of Section 2.2, which guarantees global convergence of the recursive algorithm

0, : 0t ~ + z t ~ ( 0 , - 1 , x , ) + yTp,(O,-1,x,). The new assumptions are: (D.1). The functions 7-[(0, x) and pt( O,x) satisfy for all O, O' E iRd and all x, x' c iRk: (i) [7-{(O,x 1 ) - 7-[(O,x2)l <~LI(1 + t0[)Ix1 -x21 (1 + [Xl[p' + Ixzlp2), (/0 ]7-/(0,0)-7-/(0',0)1 ~< L2 I0 - O'l, (iii) IO~(Oox)/Ox - O~(O',x)/Oxl ~< L2 10 - 0'l (1 + IxlP2), Or) Ipt(O,x)l <<.L2(1 + 101)(1 + fxl q)

for some constants LI,L2,Pl,P2 and q. (D.2). The dynamics for the state variable Xt E iRk is" independent of 0 and satisfies

(B.1) and (B.2) above. With these conditions one has the following global result:

Ch. 7." Learning Dynamics

487

Theorem 6. Under assumptions (A.1), (D.1) and (D.2) assume that there exists a unique equilibrium point O* E W~a o f the associated ODE. Suppose that there exists a positive C 2 function U(O) on ~d with bounded second derivatives satisfying (i) U'(O)h(O) < O f o r all 0 ~ 0", (i 0 U(O) = 0 iff O = 0", (ii 0 U(O) ~ al012 for all 0 with 101 >> po for some a, po > O. Then the sequence On converges Po ..... almost surely to 0".

A proof is outlined in Evans and Honkapohja (1998b). In that paper it is also shown how this theorem can be used to establish global convergence in the multivariate linear model of Section 2.7.2. 3. Linear economic models

3.1. Characterization o f equilibria Many linear rational expectations models have multiple solutions, and this is one o f the reasons why the study o f learning in such models is o f considerable interest, as previously noted. Consider the following specification: g k Yt = a + ~-~OiYt i+ Z [ 3 i E t i-I

lYt+i+Ut,

(28)

i-O

in which a scalar endogeneous variable Yt depends on its lagged values, on expectations o f its current and future values, and on a white noise shock yr. Here Et-lYt+i denotes the expectation o f yt+i based on the time t - 1 information set. For this model it is possible to give a complete characterization o f the solutions, using the results o f Evans and Honkapohja (1986) and Broze, Gourieroux and Szafarz (1985). The technique is based on the method o f undetermined coefficients, but rather than guessing a solution of a particular kind it is applied systematically to find a representation for all possible solutions. Every solution can be written in the form

Yt

~ - 1' ~[Jk i 1--[30y t k ~e 6i -Z..~ffYt [3~nlOt- ¢....~ [jk Yt i + [,4k i= l lak i=1 k k -

k i+Vt

(29)

q-~CiOt-i+Edict-i, i-1 i-1

where et is an arbitrary martingale difference sequence, i.e. a stochastic process satisfying Et Iet = 0, and where c i , . . . , ck, dl . . . . . di¢ are arbitrary 33. Various particular 33 There is an extensive literature on solution techniques to linear RE models and different possible representations of the solutions. Some central references are Gourieroux, Laffont and Monfort (1982), Broze, Gourieroux and Szafarz (1990), Whiteman (1983), McCallum (1983), Pesaran (1981), d'Autume (1990), and Taylor (1986).

488

G. W. Evans and S. Honkapohja

solutions can be constructed from Equation (29) by choosing the values for the ci and di and the et process appropriately.

In the literature attention is most often focused on so-called minimal state variable (MSV) solutions to Equation (28) 34. These solutions are of the form g Yt = a + Z

biYt-i + Or.

i-I

Many macroeconomic models have expectations for which the information set includes the current values o f the variables. A characterization similar to Equation (29) is available for such models. Some models in the literature have mixed datings of expectations and/or incorporate exogenous processes other than white noise. Although there exists a general characterization for the set o f solutions in Broze, Gourieroux and Szafarz (1985), it is often easier to be creative and derive the representation by the principles outlined above. The references in the footnote above provide detailed discussions of the methods in particular frameworks. 3.2. L e a r n i n g and E-stability in unit)ariate models

In this section we give a comprehensive analysis o f adaptive learning dynamics for some specific linear setups. Although these models appear to be relatively simple, they cover a large number o f standard macroeconomic models that have been developed in the literature. Another advantage o f focusing initially on simple models is that we can study the learning dynamics for the full set o f solutions and obtain complete analytic results. It is possible to generalize the analysis o f learning to more general setups (including various multivariate models) and derive, for example, the conditions for stability of specific solutions under learning. However, these conditions become easily abstract, so that analytic results are limited and it becomes necessary to resort to numerical methods. 3.2.1. A leading example

Consider the univariate model Yt = a + [3oEt* ly t + [31E[~lYt+l + t~t,

(30)

where vt is assumed to be an exogenous process satisfying Et_l c~t = O.

34 The terminology is due to McCallum (1983), but our usage differs from his in that we only use his primary solution principle to define MSV solutions. McCallum also introduces a subsidiary principle, and he defines MSV solutions as those that satisfy both principles. McCallum (1997) argues that his MSV criterion provides a classification scheme for delineating the bubble-free solution.

489

Ch. 7: Learning Dynamics

E x a m p l e 3.1. Sargent a n d Wallace (1975) "ad-hoc" model: qt = al + a2(Pt - Et_lPt) + ult, qt = bl + b2(rt - ( E t lPt+l

where a2 > 0;

Et_lPt)) + u2t,

m = co + P t + c l q t + c 2 r t + u 3 t ,

where b2 < 0;

where Cl > 0, C2 < 0.

Here q,p, and m are the logarithms o f output, the price level and the money stock, respectively, and the money stock is assumed constant, r is the nominal interest rate. This fits the reduced form (30) with yt = p t , and/31 > 0 and/3o +/31 < 1. E x a m p l e 3.2. R e a l balance m o d e l [Taylor (1977)]: qt = al + a2(m - P t ) + ult,

where a2 > 0;

qt = bl + b 2 ( r t - ( E ? _ l p t + l - E [ _ l p t ) ) + b 3 ( m - p t ) + u 2 t , m = co +Pt + qt + c2rt + c 3 ( m - p t ) + u3t,

where b2 < 0, b3 > 0;

where c2 < 0, 0 < c3 < 1.

The reduced form is Equation (30) with Yt = Pt and/31 = -rio, where /30 = b2(b3 + b2(1

a2 - c3)c~ ~ - a2) -l.

For appropriate choice o f structural parameters, any value/30 ~ 0 is possible. 3.2.1.1. A characterization o f the solutions. The set o f stochastic processes Yt = -/311a + fijl (1 - /3o) yt 1 +or + C l Ut-I +diet-1

(31)

characterizes the possible REE. cl and dl are free, and et is an arbitrary process satisfying Et let = 0. et is often referred to as a "sunspot", since it can be taken to be extrinsic to the model. We will refer to Equation (31) as the ARMA(1,1) set of solutions. These solutions can either be stochastically (asymptotically) stationary or explosive, depending on the parameter values. The ARMA(1,1) solutions are stationary if 1/311(1 --/30)] < 1. Choosing d l = 0 and cl = -/311(1 -/3o) gives an ARMA(1,1) process with a common factor for the autoregressive and moving average lag polynomials. When cancelled this yields the MSV solution 35 a Y t -- 1 -- /30 -- [31

+ yr.

(32)

The MSV solution is, o f course, often the solution chosen in applied work, and it is the unique non-explosive solution if I/3~-1(1 -/30)[ > 1. Various terminologies are in use

35 See Evans and Honkapohja (1986, 1994b) for details of this technique in general setups.

490

G. 14(Evans and S. Honkapohja

for this situation: the model is equivalently said to be "saddle-point stable" or "regular" and the MSV solution is said to be "locally determinate". If instead l[311(1 -/30)l < 1 then the model is said to be "irregular" and the MSV solution is described as "locally indeterminate". It is precisely in this case that the A R M A solutions are stationary. We will now consider the E-stability of the various solutions, taking no account of whether the ARMA(1,1) solutions are stationary or explosive (an issue to which we will return). Obviously for this model the MSV solution is always stationary. 3.2.1.2. E-stability o f the solutions. Posit a PLM (Perceived Law of Motion) of the same form as the MSV solution:

(33)

yt = a + yr.

Under this PLM we obtain y t = a + ([3o + [31)a + vt

as the Actual Law of Motion implied by the PLM (33). For E-stability one examines the differential equation da d r - a + ([3o +[3,) a - a

(34)

with unique equilibrium ~ = a/(1 - [ 3 0 - [31). The E-stability condition is

[30 +[31 < 1.

(35)

Next consider PLMs of the ARMA(1,1) form Yt = a + b y t - l +cvt l +det-1 +vt,

(36)

where et is an arbitrary process satisfying Et l gt = O, assumed observable at t. The implied A L M is Yt = a + [3oa +/31a(1 + b) + ([3ob + [3lb2)yt-1 + (flOC + [31be)or 1 + ([3od + [31bd)et-t + yr.

(37)

The mapping from PLM to A L M thus takes the form T ( a , b , c , d ) = (a+[30a+[31a(1 +b),[3ob+[31b2,[[3oc+[31bc,[3od+[31bd),

(38)

and we therefore consider the differential equation d d~(a, b, c, d) = T(a, b, c, d) - (a, b, c, d).

(39)

Note first that (a, b) form an independent subsystem, d ( a , b) = Tab(a, b) - (a, b). Evaluating the roots of DTab - I at the ARMA(1,1) solution values a = -/311a, b = [311(1 -[30), it follows that E-stability for the ARMA(1,1) solutions requires [30 > 1,

/31 < 0.

(40)

It is then further possible to show that if (a, b) converge to the ARMA(1,1) solution values, then under Equation (39) (c, d) also converge to some value [see Evans and

Ch. 7.. Learning Dynamics

491

Honkapohja (1992) for details]. Hence Equations (40) are the conditions for the ARMA(1,1) solution set to be E-stable. 3.2.1.3. StrongE-stability. Reconsider the MSV solution. Suppose agents allow for the possibility that Yt might depend on Yt l, vt j and et 1 as well as an intercept and yr. Is the MSV solution locally stable under the dynamics (39)? Evaluating D T - I at ( a , b , c , d ) = ( a / ( 1 - / 3 o - / 3 1 ) , 0 , 0 , 0 ) one obtains for the MSV solution the strong E-stability conditions:

/30+/31 < 1,

/3o < 1.

(41)

These conditions are stronger than the weak E-stability condition (35). For the ARMA(1,1) solution class one obtains that they are never strongly E-stable, if one allows for PLM o f the form Yt = a + blyt-i + b2Yt 2 + c o t 1 + det 1 +yr.

(42)

The argument here is more difficult (since the linearization Of the differential equation subsystem in (b~, b2) has a zero eigenvalue), and is given in Evans and Honkapohja (1999a). See Evans and Honkapohja (1992, 1994b) for related arguments. (In fact the lack o f strong E-stability is also delicate, since the differential system based on Equation (42) exhibits one-sided stability/instability)36. 3.2.1.4. E-stability and indeterminacy. The overall situation for the model (30) is

shown in Figure 2 37. In terms o f E-stability, there are 4 regions of the parameter space. Iffl0 +/31 > 1 and fil > 0 then none of the REE are E-stable. I f Equation (41) holds then the MSV solution is strongly E-stable, while the ARMA(1,1) solution is E-unstable. If fi0 + fil < 1 and/3o > 1 then the MSV solution is weakly but not strongly E-stable, and the ARMA(1,1) solutions are also weakly E-stable. Finally if/3o +/31 > 1 and /31 < 0 then the ARMA(1,1) solution set is weakly E-stable, while the MSV solution is E-unstable. In Figure 2 the region o f indeterminacy (in which there are multiple stationary solutions) is marked by the shaded cones extending up and down from the point (1, 0). Outside this region, the MSV solution is the unique stationary solution, while inside the indeterminacy region, the A R M A solutions as well as the MSV solution are stationary. For this framework the connection between indeterminacy and E-stability can be summarized as follows. In the special case/3o = 0, indeterminacy arises iff

36 Under Equation (42) the strong E-stability condition for the MSV solution remains (4i). 37 We comment briefly on the relationship of the results given here and those in Evans (1985) and Evans and Honkapobja (1995a). The results in Evans (1985) are based on iterative E-stability, which is a stronger stability requirement. In addition both Evans (1985) and Evans and Honkapohja (1995a) used a stronger definition of weak E-stability for the MSV solution, using PLMs with yt_1 included.

G. W. Evans and S. Honkapohja

492

MSV solution strongly E - sta )le, ARMA solution explosive and [ i - unstable

All solutions E - unstable, ARMA solutions stationary .

All solutions E - unstable, ARMA solutions explosive

.13o MSV solution E - unstable, A R M A solutions explosive and weakly E - stable "[I ~' solution I solution '1 I strongly weakly but E - stable, not strongly 1 ARMA solutions E - stable, stationary and A R M A solutions E - unstable stationary and weakly E - stable

Fig. 2. I /31 l> 1, but the A R M A solutions are never E-stable. However if/30 > 1, cases of (weakly) E-stable A R M A solutions arise in the right-hand half o f the lower cone of indeterminacy. Thus in general there is no simple connection between weak E-stability and determinacy 3s. Applying these results to examples 3.1 and 3.2, we have the following. In the Sargent-Wallace "ad hoc" model, the MSV solution is uniquely stationary and it is strongly E-stable, while the other solutions are E-unstable. In the Taylor real-balance model we have/31 = -/3o. There are three cases: (i) if/3o < ½ then the MSV solution is uniquely stationary and is strongly E-stable, while the other solutions are E-unstable; (ii) if ~l
38 However, we know of no cases in which a set of ARMA solutions is strongly E-stable. See Evans and Honkapohja (1994b).

Ch. 7:

493

Learning Dynamics

(iii) if/30 > 1 then the MSV solution is stationary and weakly (but not strongly) E-stable and the ARMA(1,1) solutions are also stationary and weakly (but not strongly) E-stable. 3.2.2. The leading example: adaptive learning 3.2.2.1. Adaptive and statistical learning o f M S V solution. Since the MSV solution is an lid process, the natural statistical estimate is the sample mean, t at = t l ~ - ~ Y t - i , i-1

which is, in recursive form, at = at-i

+ t-l(yt

-

(43)

at-l).

Inserting Yt = ol + ([30 q-/31) at 1 + ot into the recursive equation we obtain the dynamic equation at = a t - 1 - t - t

l(a-]-(/30-}-/31)

at 1--at 1 + Ut).

(44)

Thus the model (30) with PLM (33) and learning rule (43) leads to the stochastic recursive algorithm (44), which can be analyzed using the tools of Section 2. The associated ODE is just the E-stability equation (34). It follows that if the E-stability condition/30 +/31 < 1 is met, then there will be convergence of at to fi = a/(1 -/3o -/31 ) and hence of the process followed by yt to the MSV solution. Indeed, for this set-up there is a tmique zero of the ODE and under the E-stability condition it is globally stable. Thus if/3o +/31 < 1 then at ~ fi with probability 1 globally, i.e. for any initial conditions. 3.2.2.2. Learning non-MSV solutions. Consider next whether suitable statistical learning rules are capable of learning the non-MSV solutions. Since there are technical complications which arise from the continua of solutions, we start with a particular solution from the set (31) in which cl = d l = 0:

Yt = --/311a +/311( 1 --/30)Yt-1 +Vt.

(45)

We also restrict attention to the "irregular" case 1/311(1 -/30)[ < 1, so that we are considering an asymptotically stationary solution. We remark that if the model (30) is regarded as defined for t /> 1, then the solution set (45) has an arbitrary initial condition Yo, the influence of which dies out asymptotically (in the irregular case). In nonstochastic models, vt - 0 and the MSV solution is the steady state Yt = a/(1 -/30 -/31). The solutions (45) then constitute a set of paths, indexed by

494

G.W. Evans and X H o n k a p o h j a

the initial Y0, converging to the steady state, and, as mentioned above, the steady state is then said to be "indeterminate". Thus the question we are now considering is the stochastic analogue to whether an adaptive learning rule can converge to an REE in the "indeterminate" case. 3.2.2.2.1. Recursiue least squares learning: the AR(1) case. We thus assume that agents have a PLM o f the AR(1) form yt = a + byt

l + Ut .

Agents estimate a and b statistically and at time t - 1 forecast Yt and Yt+l using the PLM Yt = at 1 + bt-lYt-~ + Or, where at-l, bt 1 are the estimates o f a and b at time t - 1. Inserting the corresponding forecasts into the model (30) it follows that Yt is given by Yt = Ta(at-l,bt 1)+ Tb(at l,bt-1)Yt

l+Ut,

(46)

where Ta(a,b) = a +/3oa+/31a(1 +b),

Tb(a,b) = fiob+[31b 2.

We assume that (at, bt) are estimated by ordinary least squares. Letting ¢~ = (at, bt),

zt' 1 : (1,yt 1),

least squares can be written in recursive form as Ot = Ot-i + t IRtlz t l(yt - z ~ Rt = Rt l + t

1

10t-I),

t

(47)

(zt 1Zt_l - Rt-l).

Equations (47) and (46) define a stochastic recursive algorithm, and the tools of Section 2 can be applied. In particular, for regions o f the parameter space in which Ibl < 1 we obtain the associated differential equation de d r - R - ' M z ( ¢ ) ( T ( O ) - ¢)'

dR dT - Mz(¢) - R ,

(48)

where M~(¢) = E[zt(¢)zt(¢)'] and zt(¢) is defined as the process for zt under Equation (46) with fixed Ct = ¢. Here ¢' = (a, b) and T(¢) = (Ta(a,b), Tb(a, b)). Provided that /30 +/31 < 1 and /30 > 1, the A R ( I ) solution (45) is stationary and weakly E-stable and it can be shown that the ODE (48) is locally stable at (a, b) = (-/311 a,/311 (1 -/30))39. It follows that under these conditions the solution (45) is locally stable under least squares learning. 3.2.2.2.2. Learning sunspot solutions. Consider now the full class o f ARMA(1,1) solutions (31). Assuming that ut and the sunspot variable el are observable at t, we

39 That stability of ODEs of the form (48) is governed by the stability of the differential equation de -- T(¢) - ¢ is shown in Marcet and Sargent (1989c). ~7

Ch. 7:

495

Learning Dynamics

can consider least squares learning which allows for this more general dependence. We now set (9~ = (at, bt, ct, dr),

z~l = (1,yt-1, vt-1, et 1),

(49)

and we continue to assume that agents use recursive least squares to update their coefficient estimates, ~bt. Thus under least squares learning the dynamic system is given by Equations (47), (49) and the equation Yt = T(Ot)'zt 1 + ut, where T(q)) is given by Equation (38). This again defines a stochastic recursive algorithm and for q~ = ( a , b , c , d ) with [b I < 1 the ODE is again o f the form (48). It is again straightforward to show that local stability of the ODE is governed by the differential equation dqVdr = T(~b) - q~ defining E-stability. There is a technical problem in applying the stochastic approximation tools, however: the assumptions are not satisfied at the ARMA(1,1) set of solutions since they include an unbounded continuum. Although this prevents a formal proof of convergence to the ARMA(1,1) set under least-squares learning, simulations appear to show that there is indeed convergence to an ARMA(1,1) solution if the E-stability condition is met 4°. See Evans and Honkapohja (1994b) for an illustration of convergence to sunspot solutions in a related model. 3.2.3. L a g g e d e n d o g e n o u s variables

In many economic models, the economy depends on lagged values as well as on expectations o f the future. We therefore extend our model (30) to allow direct feedback from Yt-~ and consider models of the form Yt = a + 6yt-1 + [3oE?_lyt + [31E[_lyt+l + or.

(50)

This reduced form is analyzed in Evans and Honkapohja (1992). E x a m p l e 3.3. Taylor (1980) o v e r l a p p i n g c o n t r a c t model: i 1 * Xt = ~ X t 1 + " ~ E t - l X t + l

+

i * * ~Y(Et-lqt + E~-lqt+~) + Ul~,

w, = ½(xt +xt 1), qt = k + mt - wt + uet, mt = Fn + (1 - ~ ) wt,

where xt is the (log) contract wage at time t, wt is the (log) average wage level, qt is aggregate output, and mt is money supply. 0 < ~ < 1, and 1 - cp is a measure of accommodation of price shocks. The reduced form is: xt = a + ½(1 - ½cPY)xt 1 1 ~cP~Et_~xt * + ½(1 1 ~q~y)E t* lXt+l + yr.

4o Recently Heinemaun (1997b) has looked at the stability under learning of the solutions to this model when agents use a stochastic gradient algorithm.

496

G. W. Evans and S. Honkapohja

E x a m p l e 3.4. Real balance model with p o l i c y feedback: Augment Example 3.2 with a policy feedback mt = m + dpt 1 + u4t. In this example /31 = --/30 and any value of fi0 ~ 0 can arise. 3.2.3.1. A characterization o f the solutions. The MSV solutions are o f the AR(1) form.

Guess a solution of the form (51)

Yt = ~P+PYt 1 +vt.

A solution o f this form must satisfy /31p 2 +([30 - 1)/9+ b = 0,

a(1 -[30 -/31(1 + p ) ) - I = ~fl.

If ( / 3 0 - 1) 2 -4/316 > 0 there are two solutions Pl and P2 o f the form (51). One also has the ARMA(2,1) class Yt = -/31~ a + /311( 1 - /3o)Yt 1 - (~/311yt-2 +

Ut + C l O t

1 + d, et-1,

(52)

where ~t is an arbitrary sunspot and where cl and dl are arbitrary. 3.2.3.2. Stability under learning o f the AR(1) M S V solutions. Assume that agents have a PLM of the AR(1) form yt = a + blyt 1 + or. The (weak) E-stability conditions are /3o+fil-l+/31bl

<0,

fio-l+2/31bl <0,

(53)

which are to be evaluated at the A R ( I ) REE bl = Pl and bl =/32. For real-time learning agents are regressing Yt on yt-1 and an intercept. Applying the results o f the previous section, it can be shown that an MSV AR(1) solution is locally stable under least squares learning if and only if it is E-stable. It is also easily verified that only one o f the two MSV solutions can be E-stable 41 . Hence local stability under least squares learning operates as a selection criterion which chooses between the two MSV solutions 42. Strong E-stability o f the MSV solutions and weak E-stability for the ARMA(2,1) class is analyzed in Evans and Honkapohja (1992). 3.2.3.3. Discussion o f examples. Example 3.3, Taylor's overlapping contracts model,

illustrates a situation which often arises. The model is saddle-point stable, i.e. there is a unique non-explosive solution and this solution is an MSV solution. In addition,

41 Recall that our use here of the term "MSV" solutions does not require them to satisfy the subsidiary principle of McCallum (1983). For a discussion of the relationship between E-stability and McCallum's MSV criterion see Evans and Honkapohja (1992). 42 Moore (1993) has this outcome in a model of aggregate externalities.

497

Ch. 7: Learning Dynamics

this is the solution which would be selected by learning dynamics: only the unique stationary MSV solution can be strongly or even weakly E-stable. Example 3.4, the real balances model augmented with a monetary policy feedback, illustrates the broader range of phenomena which can arise more generally. In particular, for appropriate choices o f (5 and /30 (with/31 = -[3o) one can obtain (i) a (weakly) E-stable ARMA(2,1) class o f solutions, (ii) an explosive AR(1) solution which is strongly E-stable with all other solutions E-unstable 43, or (iii) all solutions stationary, with a unique strongly E-stable AR(1) solution and all other solutions E-unstable. See Evans and Honkapohja (1992) for details. 3.3. Univariate models - f u r t h e r extensions and examples

The general principles developed in the preceding example can be readily extended to other univariate models which alter or extend the framework. 3.3.1. Models with t dating o f expectations

In many economic models the variable of interest Yt depends on expectations of future variables which are formed at time t. This means that these expectations may depend on exogenous variables dated at time t and also on Yt itself. The simplest example is the "Cagan model": (54)

yt = [3Etyt+l + 3,wt + vt,

where wt is an exogenous stochastic process which is observed at time t and vt is an unobserved white noise shock. For simplicity we will focus on the case in which wt follows a stationary AR(1) process (55)

Wt = a + ~PWt I + ut,

where ut is white noise and [~Pl < 1. Example 3.5. Cagan model o f inflation: The demand for money is a linear function o f expected inflation mt - p t = -Y(E[pt+l - P t ) + tk,

y > O,

where mt is the log o f the money supply at time t, and Pt is the log of the price level at time t. This can be solved for the above form with yt =--pt, wt =- mr, [3 = 7/(1 + y), and )~ = 1/(1 + 7).

43 Adaptive learning methods can be extended to show convergenceto explosive REE such as this one see Evans and Honkapohja (1994a). For further analysis of learning in nonstationary setups, see Zenner (1996).

498

G. W. Evans and S. Honkapohja

E x a m p l e 3.6. P P P model o f the exchange rate: In a small open economy with flexible exchange rates, perfect capital mobility, exogenous output and purchasing power parity (PPP) we have m t - P t - a - c i t + ot, c > O it = tt + E t el+l - et Pt = fix + ex. Here it is the nominal interest rate, it is the foreign nominal interest rate, et is the log of the exchange rate, mt is the log of the money snpply, pt is the log of the price level, and/Sx is the log of the foreign price level, rot, Or, 19t and ix are treated as exogenous. The

first equation represents monetary equilibrium, the second is the open parity condition and the third is the PPP equation. Solving for et we arrive at the form (54), with Yt =- et, wt a linear combination of the exogenous variables, and/3 = c/(1 + c).

Example 3.7.

Asset p r i c i n g with risk neutrality: Under risk neutrality and appropriate assumptions, all assets earn expected rate of return 1 + r, where r > 0 is the real net interest rate, assumed constant. If an asset pays dividend de at the end of period t then its price pt at t is given by Pt = (1 + r ) - I ( E t p t + l +dr).

We again have the form (54), with yt =--pt, wt =- dt and [3 = (1 + r ) 1. There is a unique M S V solution to the model given by Yt = a + bwt + vt,

f i - (1 - / 3 ) - l a / 3 b

where and

b = (1 -/3~p)-1/t.

In the context of the model (54) and particularly in the case of the asset pricing application, the MSV solution is often referred to as the f u n d a m e n t a l solution 44. Is the fundamental solution stable under least squares learning? We first obtain the (weak) E-stability conditions. The T-map is T,(a, b) =/3a + ot/3b,

Tb(a, b) -/3~flb + )~,

and it is easily verified that the fundamental solution is E-stable if [3 < 1 and/3~p < 1. We have assumed [~Pl < 1 and in all three of our economic examples above 0
44 The solution can also be computed as the present value of expected dividends.

Ch. 7: Learning Dynamics

499

Let (at, bt) denote the least squares estimates using data on (wi,Yi), i = 1. . . . , t - 145. Expectations are then given by E[yt+j = at + btEtWt+l = (at + bta) + bt~Pwt, where for simplicity we treat ~p and a as known, and under learning yt is given by Yt = Ta(at, bt) + Tb(at, bt)wt + yr. Applying the standard stochastic approximation results, it follows that the fundamental solution is locally stable under least squares learning provided the E-stability condition above is met. This holds for each of the above economic examples.

3.3.1.1. Alternative dating. In the above models we have treated EtYt+l as an expectation formed at time t using all information dated t or earlier. The analysis of the fundamental solution and its stability under learning is little changed if we replace this by the assumption that the value of current variables are u n k n o w n when the expectation is formed. Thus if we consider instead Yt = [3E[ lYt+l + )~wt + vt, the fundamental solution is o f the form yt = ct + [)wt i + Or. Although the T-mapping is somewhat different, it is readily determined that the E-stability conditions are identical.

3.3.2. Bubbles 46 There are other solutions to (54) besides the fundamental solution. For simplicity let us focus on the case in which wt - 1, i.e. the model

Yt = [3EtYt+l + )~ + Or, and we assume 0 < [3 < 1. The fundamental solution is Yt = (1 -[3)-1)~ + vt and the general solution takes the form

Yt = - [ 3 1)~ +[3-1yt l-[3-1Vt_l + d e t , where et is an arbitrary sunspot and d is arbitrary. Solutions other than the fundamental solution are explosive and are often referred to as "rational bubbles" or "explosive bubbles". Under the alternative dating the model is

Yt = [3Et_lYt+l + 2~+ Yr.

45 For technical simplicity we assume that the data point (wt,Yt) is not available for the least squares estimates at t of the coefficients(a~,bt), though we do allow the time t forecasts to depend on w, This avoids simultaneity between Yt and b, With additional technical complexity this simultaneity can be permitted, e.g. Marcet and Sargent (1989c). 46 Salge (1997) is a recent review of the literature on asset pricing and bubbles.

G. W. Evans and S. Honkapohja

500

Note that this model is a special case o f the Leading Example (30) with/30 = 0 and /31 =/3. The bubbles solutions take the form (31) i.e. Yt = -/3-1~, + /3-1Yt 1 -t-ot-}- ClUt-1 + det 1.

Are the bubble solutions stable under learning? We consider the case with t - 1 dating o f expectations 47 and confine attention to E-stability. The results are apparent from Figure 2. Since/30 = 0 and 0 < /31 < 1 the bubble solutions are never stable under learning. Note that the fundamental solution is strongly E-stable in this model. Least squares learning will locally converge to the fundamental solution, but not to a bubble solution. Although this analysis casts doubt on the possibility o f adaptive learning converging to explosive bubbles, several points should be borne in mind. First, given initial expectations near an explosive path, least squares learning will not necessarily converge to the fundamental solution. Least squares learning may instead evolve along nonrational explosive paths as the economy is pushed further and further from any REE. Secondly, this section has focused on a simple set-up, which includes the standard asset pricing model in which the issue o f bubbles is most often discussed. The previous section showed that in models with a lagged dependent variable, it is possible for learning to converge to an explosive REE, even when there is a unique non-explosive path 48. A more complex model of asset pricing may thus generate bubbles which are stable under adaptive learning. The issue o f learning in asset pricing models with feedback has been investigated by Timmermann (1994).

3.3.3. A monetary model with mixed datings

In some cases reduced forms include a mixture o f dates at which expectations are formed. An example is the Duffy (1994) analysis o f the Farmer (1991) model 49. A version of the OG production model is considered in which the output o f the single perishable good depends on current and lagged labor input. This gives two ways for agents to store wealth: holding money and holding inventories of goods in process. As in the basic OG model, one way to summarize the model is to obtain the demand for real balances as a function o f expected inflation, mt = f ( E [ p t + l / p t ) . If the money stock is constant, then mt = M/pt and the reduced form can be solved forpt in terms o f

47 With t dating bubbles are also unstable under learning, see Evans (1989). 48 Adam and Szafarz (1992) also emphasize the importance of lagged endogenous variables. 49 Another example is the analysis of Muth's inventory model presented in Evans (1989). This paper also shows how to apply E-stability to models involving conditional variances as well as conditional expectations.

Ch. 7: Learning Dynamics

501

E[pt+l. Alternatively the model can be written in terms o f the inflation rate ¢c~ = Pt/Pt-l as

f(E~l~)

f(E?~,+~) Note that the model has a perfect foresight steady state o f ~t = 1 (this is the MSV solution). Linearizing around ~ = 1 yields ~t = 1 +fl0E[_l¢[, -fioEtcCt+a,

where fi0 -

if(l) f(1)

While in the standard OG f r a m e w o r k f ' ( 1 ) / f ( 1 ) can be positive or negative but cannot exceed 1, Farmer shows that in his m o d e l f ' ( 1 ) / f ( 1 ) > 1 is possible for appropriate specification o f technology and preferences. The listing o f solutions and the analysis of learning can proceed along the lines presented in Section 3.2; the principal difference is that E~a~t+i appears here in place o f E t lart+l (the model here also has no random shock, but this could be easily introduced). In addition to the MSV solution there are perfect foresight AR(1) solutions o f the form ;rt = /301 q - / 3 0 1 ( / 3 0 - 1);rt_l, and if/30 > ½, these constitute a family o f paths (indexed by the initial ~0) converging to ~t = 1 (i.e., if fi0 > ½ the steady state is indeterminate). It is easily verified that the MSV solution ~ = 1 is E-stable under the PLM ~:t = a for any value of/30. If instead the PLM allows for the possibility of an AR(1) path ;rt = a + c3rt 1 then the solution Jrt = 1 remains E-stable only if/30 < 1. Thus, this constitutes a strong E-stability condition for the MSV solution. However, the AR(1) continuum is weakly E-stable if/30 > 1 50. This model provides an example, based on a version of the OG model with fully specified microfoundations, o f the possibility o f learning converging to an indeterminate steady state. If random noise were present, the convergence would be to a member o f the ARMA(1,1) class o f solutions as in our Leading Example. 3.3.4. A linear model with two f o r w a r d leads

In more complicated linear models it is possible for there to be multiple strongly E-stable solutions. In this set-up there will be two or more solutions which are locally stable under least squares learning. To see that this is possible, consider a model in which E[_lyt+2 is also included: Yt =

a

+

(~Yt-I +/30E~lyt +/31Et~lYt+~ +/32Et-lYt+2 + or.

(56)

There are again various classes o f solutions: an ARMA(3,2) class of solutions, 1 to 3 ARMA(2,1) classes o f solutions (which in general depend on sunspot variables, 50 Strong E-stability of the AR(1) solutions would look at PLMs of the form ~t = a + cJvt 1+ d~t 2. As in the "Leading Example" of the earlier section, it can be shown that the corresponding strong E-stability equation at the AR(1) solutions has a zero root and exhibits one-sided stability/instability.

502

G. W. Evans and S. Honkapohja

i.e. on arbitrary martingale difference sequences), and up to 3 AR(1) solutions. We focus on these latter solutions - the MSV solutions. [See Evans and Honkapohja (1994b) for a discussion o f the other solutions and a full treatment o f details.] AR(1) solutions o f the form (51), i.e. Yt = ~P +PYt 1 + vt, must satisfy fl = di +/30/9 +/31/02 +/32/93. Depending on whether or not the cubic has a pair o f complex roots, there are 1 or 3 AR(1) solutions. The main new result is that for an open set o f parameter values two o f the AR(1) solutions are strongly E-stable. For weak E-stability we consider PLMs o f the form Yt = a + blyt 1 + Or.

This leads to the T-map which specifies the A L M Yt = Ta(a, bl) + Tbl (a, bl)Yt-1 + vt: T.(a, hi) = a +/3oa +/31 (abl + a) +/32(bl (bl a + a) + a), Tb~(a, bl) = O+flobl +fl, b~ +fl2b~.

The key to the result is that stability o f the bl parameter is determined by a cubic which may have two stable roots, depending on the parameters. For strong E-stability we allow the A L M to depend on additional lags yt-i as well as on lags vt i and et-i. For the strong E-stability conditions and for conditions under which there are two strongly E-stable stationary AR(1) solutions, see Evans and Honkapohja (1994b). E x a m p l e 3.8. Dornbusch-type model with p o l i c y feedback: The equations are Pt-Pt

1 = ~ E t 1dr,

dt = - y ( r t - Etpt+l +Pt) + tl(et - P t ) , r, = )~ l(pt - Opt-l), rt = Etet+l - et.

The first equation is a Phillips curve, the second the IS curve for an open economy, the third is the LM curve in which monetary policy reacts to pt-1, and the last equation is the open parity condition. The reduced form for Pt can be put in the required form (56). Evans and Honkapohja (1994b) show that the case o f two strongly E-stable AR(1) solutions, as well as weakly E-stable A R M A solution classes depending on a sunspot variable, can arise for appropriate parameter values. 3.4. Multivariate models

Systematic examination o f univariate linear models is illuminating because they can be used to illustrate the wide range o f outcomes for macroeconomic models with

Ch. 7:

503

Learning Dynamics

adaptive learning and because numerous textbook examples fit this framework. For serious applied work a multivariate set-up is needed. Consider the following example. E x a m p l e 3.9. A d h o c s t i c k y - p r i c e m o d e l w i t h p o l i c y f e e d b a c k : Pt -Pt-I

= ao + a l q t + (Et-lpt+l - E t - l p t )

+ Ult

qt = bo - b l ( r t - E t lPt+~ + E t - l p t ) + v2t mt - P t = co + c l q t - c2rt + O3t mt = do + d l p t - , + d2qt 1 + d3rt 1 + d4mt 1 + v 4 ,

The first equation is a standard but a d h o c rational expectations Phillips curve in which inflation, P t - P t 1, depends on expected inflation, E t - l p t + l - E t - i P t , and on aggregate real output, qt. The second equation is the IS curve, relating qt to the e x a n t e rt - E t lPt+l + E t lPt. The third equation is the LM curve, equating the supply of real money balances mt - P t to the demand for them. Here rt is the nominal interest rate. The last equation is the monetary policy feedback rule, relating the nominal supply of money to lagged prices Pt 1 and lagged values of the other variables. Each equation is also subject to an unobservable iid random shock, vit. We assume al, bl, cl, ca > 0. Letting Yt = (Pt, qt, rt, m f f and vt = (Ult, v2t, o3t, u4t) t this model can be put in the form Yt = a + f i o E t - l y t +[3lEt lYt+l + 6yt 1 + ~vt,

where the coefficients are appropriately sized matrices. More generally, allowing for a vector of exogenous variables wt we can consider the following general form: Yt = a + f i o E t lYt + ]3lEt lYt+l + ~Yt-1 -]- l~Wt + ~Ut,

(57)

Wt = p W t 1 + et.

Here Yt is an n × 1 vector of endogenous variables and wt is a vector of exogenous variables which we assume to follow a stationary VAR, so that et is white noise. This set-up, apart from the dating of expectations, is close to McCallum (1983). 3.4.1. M S V s o l u t i o n s a n d l e a r n i n g

We consider the MSV (minimal state variable) solutions which are of the following form: y¢ = a + b y ¢ 1 + c w t I + fret + ~vt,

(58)

wt = p w t | + et,

(59)

where a, b, and c are to be determined by the method of undetermined coefficients. Note that the solutions are in the form of a VAR (vector autoregression). Computing

G.W. Evans and S. Honkapohja

504

Et-~yt and Et-lyt+l and inserting into Equation (57) it follows that REE of this form must satisfy the matrix equations (I - [30 - [3~ b - [31)a

= a,

(60)

[31b2 + ([30 - I)b + 6

= 0,

(61)

=

(62)

(I - [30

-

[31b)c [31ep -

K'p.

In general there are multiple solutions and various techniques are available for solving models of this form. For example, in the regular case there is a unique stationary solution and a modification of the Blanchard and Kahn (1980) technique can be used. See Evans and Honkapohja (1999a) and Christiano and Valdivia (1994). It is also possible to use the E-stability algorithm itself to find a solution. Sargent (1993) noted the possibility of using learning rules to solve RE models, and an advantage of this procedure is that it yields only solutions which are stable under learning. Finally, using numerical techniques one can directly compute E-stable equilibria in applied large-scale macroeconomic models, see Garratt and Hall (1997) and Currie, Garratt and Hall (1993) who also investigate dynamics of learning transitions during regime changes. With PLMs of the form (58), the mapping from the PLM to the ALM is

(!) r

/

°l+(fl°+fll+[31b)a

~

= [ [3~b2 + [30b + 6 1. \ [30c + fllbC + fllCp + tgp /I

(63)

Expectational stability is determined by the matrix differential equation

d~

=T

b

-

.

(64)

c

To analyze the local stability of (64) at an RE solution 2, b, ~ one linearizes the system at that RE solution. In Evans and Honkapohja (1999a) we show the following result:

Proposition 7. An MSV solution gt, f), ~ to Equation (57) is E-stable if (i)

all the eigenvalues o¢ °vecr"-Cvecb) have real parts less than 1, : / O(vec b)' '-

Oi) the eigenvalues of:1 0(vec o vecc)' r~ (vec b, vec ~) have real parts less than 1, and (iii) the eigenvalues of the matrix [3o + [31 + [31b have real parts less than 1. Under least squares learning the perceived law of motion

Yt = at 1 + bt-lYt-1 + ct-lwt-1 q-Ifet + ~vt is used by agents to make forecasts, where the parameters at, bt and ct are updated by running multivariate recursive least squares (RLS). (Although t¢ and ~ may be

Ch. 7:

Learning Dynamics

505

unknown, these do not affect the agents' forecasts). Letting q~r = ( a , b , c ) and ! / / z t = ( 1 , y t , w t ) , RLS can still be written as Equation (47), and Yt is determined by the ALM Yt = T(e)t-i)1zt-1 + t c e t + ~vt. The tools already developed extend to the multivariate case and can be used to show that an E-stable solution is locally stable under recursive least squares learning. 3.4.2. Multivariate models with time t dating

Many multivariate models, such as the RBC (Real Business Cycle) model and those described in Farmer (1993), have Yt depend on expectations Etyt+l, i.e. where the vector Yt is part of the information set. Example 3.10. R e a l Business Cycle Model: The equilibrium equations of the standard RBC model, once linearized around the steady state, can be written in the form Ct = [311EtCt+l q- [~13EtSt+l, kt = (~21ct 1 -t- ~22kt 1 -}- (523st 1,

(65)

st = 10st-1 + vt.

where ct is consumption, kt is the capital stock and st is the productivity shock (with all variables in log deviation from the mean form). This is a special case of the general set-up Yt = Ot + fiEtYt+ 1 + (Sy t 1 -t- K'W t -]- ~Ut, wt = pwt

(66)

l + et,

and the MSV solutions now have the form Yt = a + byt-] + cwt + dvt.

(67)

Solutions can be computed using the Blanchard-Kahn technique, e.g. see Farmer (1993). In Evans and Honkapohja (1999a) we obtain corresponding E-stability conditions and show that they govern convergence of least squares learning. It is straightforward to use iterations of the E-stability algorithm to compute a solution in VAR form for numerical specifications of the RBC model. 3.4.3. Irregular models

I f the model (57) or (66) is not regular then there exist multiple stationary solutions and the solutions may depend on sunspot variables. For example, Benhabib and Farmer (1994), Farmer (1993) and Farmer and Guo (1994) have emphasized variations of the RBC model incorporating increasing returns which can lead to irregular models. In

G. W. Evans and S. Honkapohja

506

the irregular case, models of the form (66), for example, can have solutions of the form Yt = a + byt-1 + cwt + dvt + f et,

where et is a sunspot variable and f ~ 0. The general techniques for studying least-squares learning can be extended to this set-up. The detailed information assumptions on expectation formation plays an important role in determining the conditions for stability under learning in these cases. Based on the results in Evans and Honkapohja (1999a) it appears that requiring a solution to be locally stable under adaptive learning imposes additional substantive requirements on the reduced form, and hence the underlying structural, parameter values.

4. Learning in nonlinear models 4.1. I n t r o d u c t i o n

Our aim here is to discuss in detail how the analysis of learning behavior is carried out for various nonlinear models. As a vehicle we make use of a few simple overlapping generations (OG) models. These models are convenient illustrations of learning and adjustment of expectations, since basic OG models usually have a one-step ahead forward-looking reduced form. OG models are usually nonlinear, and they often have multiple REE. The models may have different types of equilibria, such as steady states, indeterminate paths to steady states, cycles, and sunspots. Thus these models can exhibit phenomena known as indeterminacy and endogenous fluctuations, see e.g. Boldrin and Woodford (1990) and Guesnerie and Woodford (1992) for a review 51. It should be noted, though, that such phenomena are not restricted to OG models. For example, Howitt and McAfee (1992) obtain multiple steady states and endogenous fluctuations in a model of search externalities in the labor market. These fluctuations are stable under a learning process. Evans, Honkapohja and Romer (1998b) consider an endogenous growth model with complementarities between different types of capital goods. Their model has equilibria with self-fulfilling fluctuations which are stable under learning. There are numerous other nonlinear models in the literature 52, and below we discuss two other models in some detail.

51 Note that these fluctuations are self-fulfilling. A different view is in e.g. Heymann and Sanguinetti (1997) who develop models in which (nonrational) transitory fluctuations can arise when agents asymptoticallylearn a steady state. s2 For example, Balasko (1994) and Balasko and Royer (1996) consider expectational stability in some Walrasian models.

Ch. 7: Learning Dynamics

507

Although a large part of our focus is on stability of equilibria under adaptive learning, it should be noted that instability of REE has also been analyzed. This can play an important role for economic results. Woodford (1990) develops an OG model in which the steady state is unstable under certain conditions, and the economy converges to a sunspot solution as a result of learning behavior. Howitt (1992) shows the instability of the steady state under learning behavior when monetary policy is carried out by means of interest rate control in a conventional macroeconomic model. Bullard (1994) shows that in an OG model non-rational limit-cycle trajectories can emerge from learning with sufficiently rapid money growth. Instability results for nonstochastic economies are emphasized by B6nassy and Blad (1989), Grandmont and Laroque (1991) and Grandmont (1998). Interestingly, Chatterji and Chattopadhyay (1997) show that global stability may prevail in spite of local instability 53. 4.2. Steady states and cycles in models with intrinsic noise

We will mostly consider models with intrinsic noise. These arise, for example when a stochastic taste or productivity shock is introduced into the basic OG model. We first present a general one-step ahead forward-looking model with random shocks, after which we provide several economic examples of nonlinear models with and without noise. Consider univariate models of the form y¢ = E;G(yt+l) + vt,

(68)

where G is a nonlinear function and vt is an exogenous shock. We assume that v¢ is iid with mean E(vt) = 0. Here E~G(yt+I) denotes the expectation of G(yt+l) formed at time t (which is nonrational outside an equilibrium). These forms are often inadequate for economic models, see below. In fact, adding productivity shocks into even simple versions of the basic OG model requires the more general reduced form Yt = H ( E { G(yt+l, vt+l), vt).

(69)

We will make the assumptions that the mappings G and H are twice continuously differentiable on some open rectangles (possibly infinite). The analysis of learning a steady state for models (68) and (69) was already considered in Section 2.7.1. Before taking up the issue of adaptive learning we consider some RE solutions to economic models taking the above form. 4.2.1. Some economic examples

Example 4.1. In the basic OG model with production introduced in Section 1.2.3 agents supply labor nt and produce (perishable) output when young and consume G+I 53 Evans and Honkapohja (1998b) and Honkapohja (1994) discuss the differences between the nonstochastic and stochastic economies.

508

G. W. Evans and S. Honkapohja

when old. Output is equal to labor supply and there is a fixed quantity o f money M. Holding money is the only mechanism for saving and thus the budget constraints are pt+lct+l = M and p t n t = M . We now introduce a random taste shock by making the utility function U(Ct+l) - V ( n t ) + et ln(nt).

Here ct is an iid positive random shock to the disutility of labor and we assume that et is known to the agents who are young at time t 54. The first-order condition for a maximum thus is V ' ( n t ) - 6t/nt = E t P t U'(ct+l). Pt+l

Combining with the market clearing condition ct+l = nt+l and using Pt/pt+l = nt+l/nt we get *

!

n t V ' ( n t ) - et = E t (nt+l U (nt+l)).

Finally, if we change variables from n to y = va(n), where O ( n ) =- n V ' ( n ) (note that O(n) is increasing for all n ) 0), we obtain Equation (68) where vt ~ c t - E ( e t ) . E x a m p l e 4.2. The technique used in example 4.1 to transform the model to the form (68) cannot always be used when there are intrinsic shocks. (It should be apparent that the technique required very special assumptions on utility.) The more general form (69) is usually required. As an illustration consider the case o f additive productivity shocks. We return to the assumption that utility is given by U(Ct+l) - V ( n t ) , but now assume that output qt is given by qt = nt + )~t,

where ~ is an iid positive productivity shock. The budget constraints are now and P t q t = M , and the first-order condition plus the market clearing condition q~+l = Ct+l and Pt/Pt+l = qt+l/qt yields

Pt+lCt+l = M

(nt + )~t)V1(nt) = E t ((nt+l + )Lt+l) Ut(nt+l + .~t+l)).

Since (n + ~ ) V 1 ( n ) is strictly increasing in n, and letting vt - 3 , t - E(J,t), this equation can be solved for nt and put in the form (69) where Yt - nt.

54 Letting ~'(n) - V(n) - ~ ln(n) we have ~'/(n) - V'(n) - e/n and Vn(n) = Vn(n) + c/n 2. Under the standard assumptions V/, Vn > 0, with e > 0, we have P'~(n) > 0 for n sufficiently large and fen(n) > 0 for all n ~> 0. Thus the marginal disutility of labor fZ/(n) may be negative at small n but we have the required assumptions needed for a well-defined interior solution to the household maximization problem.

Ch. 7: Learning Dynamics

509

E x a m p l e 4.3. Increasing social returns: A different extension o f the basic OG model to incorporate increasing social returns is obtained in Evans and Ho1~kapohja (1995b). This model was already sketched in Section 1.2.3. Assuming that substitution effects dominate in consumption behavior, this model can have up to three interior steady states. E x a m p l e 4.4. Hyperinflation or seignorage: Consider the basic OG model with government consumption financed by money creation. This model was also introduced in Section 1.2.3. Using the first-order condition nt V ~(nt) = E7 ((nt+l - gt+l) U ~(nt+l - gt+l)) and assuming further that gt = g + vt, where vt is iid with Evt = 0 and "small" compact support, one obtains a special case o f the reduced form (69): n, = H ( E t G(nt+b vt+l )), where the parameter g has been absorbed into the function G. We note here two extensions of this model. First, we sketch below the analysis of Bullard (1994) which is based on the same model but with the alternative assumption that money growth is constant while government expenditure adjusts to satisfy the budget constraint. Second, Evans, Honkapohja and Marimon (1998a) consider the same model but with a constitutional limitation on government consumption which cannot exceed a given fraction o f GDR This extension leads to the possibility o f a further constrained steady state which is usually stable trader steady-state learning behavior. 4.2.2. Noisy steady states and cycles For models with intrinsic noise we consider the REE which are analogs o f perfect foresight steady states and cycles. We start with the simplest case: a noisy steady state for the model (68). Under rational expectations we have Yt = EtG(yt+l) + vt, and we look for a solution o f the form Yt = Y + Or. It follows that.P must satisfy.P = E G ( ~ + vt). In general, a solution o f this form may not exist even if G has a fixed point 33, i.e. if 33 = G(33), but an existence result is available for the case o f "small noise". More generally, consider noisy cycle REE for the model (69). A noisy k-cycle is a stochastic process o f the form Yt - y i ( v t ) for t m o d k = i,

i = 1,... , k - 1,

(70)

Yt =yk(vt) for t m o d k - O, where the k functions yi(ot) satisfy

yi(vt) = H(EG(yi+l(Vt+l), Ot+l), Ot) if t mod k = i, yk(vt) = H ( E G ( y l (vt+l), vt+l), vt) for t mod k = 0.

i-1 ..... k

1,

(71)

G.W. Evans and S. Honkapohja

510

In a noisy k-cycle, the expectations EG(yt+I, vt+l) follow a regular cycle. We will use the notation

Oi = EG(yi(vt), or) if t mod k = i,

for i = 1 , . . . , k - 1,

Ok = EG(yk(vt), vt) if t mod k = 0, so that yi(vt) = H(~+I, or), i = 1.... , k - 1, and yk(vt) = H(OI, Or). Thus a noisy k-cycle is equivalently defined by (01 . . . . . 0k) such that

Oi = EG(H(Oi+I, Or), vt) for i = 1. . . . . k - 1,

(72)

Ok = EG(H(O~, vt), 03. A noisy steady state corresponds to the case k = 1 and is thus defined by a function

y(vt) such that y(vt) = H(Ea(y(vt+~), vt+x), vt). Letting 0 = EG(y(vt+l), vt+l) it is seen that a noisy steady state is equivalently defined by a value 0 satisfying 0 = EG(H(O, or), vt). It can be shown that noisy k-cycles exist near k-cycles of the corresponding nonstochastic model, provided the noise is sufficiently small in the sense that it has small bounded support 55.

4.2.3. Adaptive learning algorithms We now introduce adaptive learning for noisy k-cycles (for the analysis o f stability o f a noisy steady state see Section 2.7.1 above). Suppose agents believe they are in a noisy k-cycle. They need to make 1-step ahead forecasts o f G(yt, vt) and at time t have estimates (01t,..., 0kt) for the expected values o f G(yt, vt) at the different points o f the k-cycle, k ~> 1. That is, 0it is their estimate at t for EG(yt, vt) if t mod(k) = i, for i = 1 , . . . , k - 1 and 0kt is their estimate at t for EG(yt, vt) if t mod(k) = 0. We assume that G(yt, vt) is observable and used to update their estimates. Since vt is iid, in an REE noisy k-cycle the values o f G(yt, vt) are independently distributed across time and are identically distributed for the same values o f t mod k.

55 For a rigorous statement and proof see Evans and Honkapohja (1995c).

Ch. 7: Learning Dynamics

511

A natural estimator o f ( 0 1 , . . . , 01,) is then given by separate sample means for each stage o f the cycle:

Oit= (#Ni) a Z G(yj, oj), jCN~ Ni - { j = l . . . . , t - 1

where

(73)

Itmodk=i}fori=l,.--,k-1,

Nk = { j = 1. . . . , t - 1 [ t m o d k = 0}. Here #N~ denotes the cardinality of the set Ni, i.e. the number o f elements in the set. Given the estimates the A L M takes the form

Yt = H(Oi+l,t, Vt) if t mod k = i,

for i = 1.... , k - 1,

Yt = H(Olt, Vt) if t m o d k = O.

(74)

The system consisting o f Equations (73) and (74) can be put into a recursive form, and the standard techniques o f Section 2 can be applied 56. The details are given in Evans and Honkapohja (1995c).

4.2.4. E-stability and convergence Before stating the formal convergence results under adaptive learning, we derive the appropriate stability condition using the E-stability principle. Recall that under this principle we focus on the mapping from a vector o f parameters characterizing the Perceived Law o f Motion (PLM) to the implied parameter vector characterizing the Actual Law o f Motion (ALM). Although in a noisy k-cycle the solution is given by k functions, yi(ot), i = 1 . . . . . k, what matters to the agents are only the expected values o f G(yt+~, vt+l). If agents believe they are in a noisy k-cycle, then their PLM is adequately summarized by a vector 0 = (01 . . . . . Ok), where

Oi = G(yt, vt) e if t m o d k

= i,

f o r / = 1....

, k - 1,

Ok = G(yt, vt) e if t mod k = O. If agents held these (in general nonrational) perceptions fixed, then the economy would follow an actual (generally nonrational) k-cycle

Yt = H ( O i + l , v t ) i f t m o d k = i ,

f o r i = l .... ,k

1,

56 Guesnerie and Woodford (1991) look at analogous fixed-gain learning rules in a nonstochastic system.

G.W. Evans and S. Honkapohja

512

Yt = H(O1, vt) if t m o d k = 0. The corresponding parameters 0* = ( 0 ~ , . . . , 0k*) of the ALM induced by the PLM are given by the expected values of G(yt, vt) under this law of motion:

O[ ~ EG(H(Oi+l,vt),vt) if t m o d k = i, O~ = EG(H(01, vt), or) if t mod k = 0.

for i = 1.... , k - 1,

Thus the mapping 0* = T(O) from the PLM to the A L M is given by

T(O) = (R(02),..., R(Olc),R(O1)), where

R(Oi) = E(G(H(Oi, or), vt)), assuming k > 1. For k = 1 we have simply T(O) = R(O) = E(G(H(O, or), vt). With this formulation of the T mapping the definition of E-stability is based on the differential equation (22) with ~0 = 0. It is easily verified using Equation (72) that fixed points of T(O) correspond to REE noisy k-cycles. A REE noisy k-cycle 0 is said to be E-stable if Equation (22) is locally asymptotically stable at 0 57 Proposition 8. Consider an REE noisy k-cycle of the model (69) with expectation parameters 0 = (01 Ok). Let ~ = 1-[I~_1R'(Oi). Then 0 is E-stable if and only if ....

,

~<1 -(cos(Jr/k)) -k < ~ < 1

ifk-l

ork=2

i f k > 2.

4.2.4.1. Weak and strong E-stability. In the context of k-cycles the distinction between weak and strong stability, discussed in Section 1.5, arises naturally as follows. A k-cycle can always be regarded as a degenerate nk-cycle for any integer n > 1. Thus the 2-cycle (01, 02) is also a 4-cycle taking values (01,02, 01, 02), a 6-cycle taking values (01, 02, 01, 02, 01, 02) , etc. Define k as the primitive period of the cycle if it is not an m-cycle for any order m < k (e.g. the primitive period is 2 in the example just given if 01 ¢ 02). Consider now a noisy k-cycle REE of the model (69) with primitive period k and expectation parameters 0 = (01 . . . . . Ok). 0 is said to be weakly E-stable if it is E-stable when regarded as a k-cycle and strongly E-stable if it is E-stable when

57 The next two propositions about E-stability of steady states and cycles are proved in Evans and Honkapohja (1995c).

Ch. 7: LearningDynamics

513

regarded as an nk-cycle for every positive integer n. Conditions for strong E-stability are given by the following result. Proposition 9. Consider an REE noisy k-cycle of the model (69), with primitive period k, and with expectation parameters 0 = (01 .... , Ok). 0 is strongly E-stable if and only if I~[ < 1.

4.2.4.2. Convergence. For this framework the preceding E-stability conditions provide the appropriate convergence condition under adaptive learning for Equations (73) and (74). The associated differential equation turns out to be just the equation defining E-stability d 0 / d r = T(O) - O, so that we have Proposition 10. Consider an REE noisy k-cycle of the model (69), with primitive period k, and with expectation parameters 0 = (01 . . . . . Ok). Suppose that 0 is weakly E-stable. Then 0 is locally stable under adaptive learning. If instead 0 is not weakly E-stable then 0t = (01t,..., Okt) converges to 0 with probability 0 58. Of course by "locally stable" under adaptive learning we mean various more specific statements made explicit in Section 2: (1) Convergence with positive probability for nearby initial points. (2) Convergence with probability close to 1 for sufficiently low adaption rates. (3) Convergence with probability 1 if a sufficiently small Projection Facility is used.

4.2.4.3. The case of small noise. If vt = 0, then we have Yt = F(yt+t) under perfect foresight, where F ( y ) =_H(G(y, 0), 0). The E-stability condition is then determined in terms of ~ = Ft(~I)F'(~2)... F'(~k) for a perfect foresight k-cycle (~31,... ,.vk). Recall that noisy k-cycles exist nearby if the noise is sufficiently "small". It can be shown that the E-stability conditions are "inherited" from the perfect foresight case. Moreover, for small enough noise it is also possible to show convergence from nearby initial points with probability 1 without a projection facility, see Evans and Honkapohja (1995c). 4.2.5. Economic models with steady states and cycles 4.2.5.1. Economic examples continued. These general convergence and non-convergence results can be easily applied to the four economic examples in Section 4.2.1. The technique is to convert each model to the form (69) or its simpler versions. For Examples 4.1 and 4.2 describing two formulations of preference shocks it may be shown that with "small enough" noise there exists a stable noisy steady state in the neighborhood of the corresponding steady state in the nonstochastic model. (However, with sufficient noise the stability condition can be altered.) In Example 4.3, the model of increasing social returns, the multiple interior steady states can be divided into

58 A correspondingresult holds also for strong E-stabilitywhenthe learning rule is overparameterized.

514

G. l/Y E v a n s a n d X H o n k a p o h j a

locally stable and unstable ones. For the hyperinflation model (Example 4.4) the lowinflation steady state is stable under learning, whereas the high-inflation steady state is not. The same models furnish examples of RE cycles that are stable or unstable under learning. For example, consider the basic OG model in Example 4.1 without any shocks. It is well-known that this model can have deterministic cycles as REE, provided that over a suitable range the offer curve slopes downwards sufficiently steeply 59. The stability results of learning behavior in Propositions 8-10 provide stability conditions for cycles in the basic OG model. Moreover, an E-stable deterministic cycle remains E-stable in a model in which a sufficiently small preference shock has been added.

4.2.5.2. Other economic models. Example 4.5. Instability of Interest Rate Pegging: The argument that tight interest rate control is not a feasible monetary policy has recently been re-examined by Howitt (1992) for some alternative economies with learning behavior. One of the models has both short- and far-sighted agents. The former live for two periods, selling their endowment e when young and consume only at old age the proceeds ePt 1. The latter have a constant endowment y and an objective function E~ ~/~o[3Ju(et+j). (Here E[ denotes expectations.) They face a finance constraint implying that Mt = PrY, since current consumption and investment in bonds is paid for by initial money, a transfer and initial bonds (with interest). M~ is end-of-period money holding. Denoting the nominal interest factor on bonds by R , the first-order condition for the far-sighted agent is

u'(e,) = R.1E:

P., L art+, 1' a~,+,- Pt Market clearing for goods yields ct = y + e(1 - 1/¢~t). The finance constraint implies that inflation equals money growth. With a pegged interest factor R the model has a unique perfect foresight steady state with inflation factor ¢c* = Re. The analysis of this model proceeds by defining the variable fiut{y + e[1 (1/art)l} Xt ~ ¢tt SO that the first-order condition gives -

u'[y + e(1 - 1/~)] =/~t+l,Xt+l ~ E[xt+b This equation defines a function a~, = ¢c(~t+l). Introducing the notation h(x) = fiR/~(x) the model yields the dynamic equation

xt = 5ct+lh(~ct+l), where h,h' > O,h(x*) = 1, and where x* = a~-~(a~*). 59 See Grandmont (1985) for details.

Ch. 7: LearningDynamics

515

It is easy to verify that if, for example, agents try to learn a steady state, then x* will be unstable 6o. This is easily seen by noting that the derivative o f F(X,+l ) = 2,+ih(.~,+l ) is greater than unity at x*. E x a m p l e 4.6. Learning Equilibria: The model of Bullard (1994) is obtained from the model o f Example 4.4 o f Section 4.2.1 by replacing the assumption o f constant (real) government spending by constant nominal money growth 0 = M,/Mt-1. Government spending is then made endogenous, so that the budget constraint is satisfied. Bullard's model can be described in terms o f the savings (or money demand) function MJP, = S(Pt/E[Pt+I) and forecasting o f the inflation rate fit = E[Pt+I/P,. For the latter it is postulated that agents run a first-order autoregression using data through t - 1. This system can be written as a system o f three nonlinear difference equations: /3, =/3,1 + g , - I

rs(/3'-12) /3. 1 ,

/3., =/3, 1, g, =

21 t, S ( /3,_ll ) ll

+

11, •

Bullard shows that if the money growth rate 0 is not too large, this system is stable with inflation given by/3" - 0. However, if 0 is increased beyond a critical value, the steady state becomes unstable. In fact, the system undergoes a H o p f bifurcation, and the learning dynamics converges to a limit cycle 61. There is a multiplicity o f these "learning equilibria" depending on the starting point. When the estimates o f agents, attempting to learn a steady state, converge to a (nonrational) limit cycle, it is possible that forecasting errors becomes large and exhibit some regularities. If such a regularity is found, then agents would try to exploit such a regularity and stop using the previous learning rule. However, Bullard shows that for carefully chosen savings functions the forecast errors can exhibit a complex pattern, so that agents do not necessarily find regularities they could exploit. Sch6nhofer (1996) examines this issue further and shows that the forecast errors can even be chaotic.

4.3. Learning sunspot equilibria In this subsection our interest lies in the analysis o f learning o f equilibria which are influenced by extraneous random phenomena, often referred to as "sunspots". To

6o We note here that Howitt (1992) considers also other more general learning rules. This is possible, since the system is one-dimensional and relatively simple. 61 The possibility of convergence to a non-REE limit cycle, under learning with decreasing gain, arises because the regressor, i.e. past price level, is nonstationary.

516

G. W. Evans and X Honkapohja

simplify the discussion we will assume that preference or technology shocks do not appear in the model 62 and focus on the class o f models Yt = E[ f (yt+l).

(75)

The rational expectations equilibria for Equation (75) satisfy yt = Et[ f(yt+l)], where Et denotes the conditional expectation, given information at time t. The rational expectations equilibria which are dependent on extraneous random phenomena or 'sunspots' have received a great deal o f attention in the recent literature after the initial investigations by Shell (1977), Azariadis (1981), and Cass and Shell (1983). The existence o f such sunspot equilibria has, in particular, been much studied 63. The seminal work o f Woodford (1990) demonstrates that, for appropriate specifications, learning can converge to sunspot solutions in the basic OG model. The Woodford (1990) representation o f learning in the OG monetary model provides a careful treatment which precisely reflects the information available to households 64. Our presentation here follows Evans and Honkapohja (1994c) which is developed in terms o f the simple reduced form (75). These rational solutions, together with deterministic cycles studied above, can be viewed as a modern formulation of a long tradition in economics which emphasizes the possibility o f endogenous fluctuations in market economies. The nonlinearity of the economic model is a key element in generating the possibility o f these equilibria even if extraneous variables can sometimes appear as part of rational solutions in linear models as well (see Section 3). 4.3.1. Existence o f sunspot equilibria We begin by reviewing some results on the existence o f sunspot equilibria. The definition o f a sunspot equilibrium involves the idea that economic agents in the model condition their expectations on some (random) variable st which otherwise does not have any influence on the model economy. Although different types of sunspot solutions have been considered in the literature we will focus here on REE that take the form o f a finite Markov chain. For most o f the analysis we simplify even further by assuming that the extraneous random variable is a 2-state Markov chain with a constant transition matrix H = (Jrij), 0 < ~.j < 1, i,j = 1,2. Here ~ j denotes the probability that St+l = j given that currently in state st = i 65. A 2-state Markov chain is defined by probabilities ~11 and Yg22since ~12 = 1 --~ll and ~21 = 1 - Jv22.

62 Intrinsic shocks could be easily introduced, see Evans and Honkapohja (1998a). 63 See Chiappori and Guesnerie (1991) and Guesnerie and Woodford (1992) for recent surveys. 64 An interpretation of the Woodford (1990) results in terms of E-stability was presented in Guesnerie and Woodford (1992). 65 Guesnerie and Woodford (1992) discuss possible interpretations of s t.

Ch. 7: LearningDynamics'

517

One defines a (2-state) Stationary Sunspot Equilibrium (SSE) ( y ~, y~ ) with transition probabilities 7r/j by means o f the equations Y~' = ~1 lf(Y~') + (1 - ~ll)f(Y~),

y~ = (1 - ~22)f(YT) + ~22f(Y~).

(76)

In the SSE Yt = Y~l if st = i. These equations are clearly a special case o f an REE for Equation (75), where Yt+~ = )~7 with probability Jr/j given that yt = y[. These equations have the geometric interpretation that the two values (YT,Y~) must be convex combinations o f f ( y ~ ) and f(y~). This observation makes it possible to construct economic examples o f SSEs as follows. Assume that f(331) < J'(332) for two points 331 and 332. Then there exist 0 < zcU < 1 such that (331,332) is an SSE with transition probabilities zc~jif and only if the points 331 and 332 both lie in the open interval (f(331),f(332)). Note that in this construction the two points 33! and 332 need not be near any deterministic equilibria. A large part of the literature has focused on the existence o f SSEs in small neighborhoods around deterministic cycles or steady states for model (75). To make this notion precise we say that an SSE y = (Yl,Y2) is an c-SSE relative to 2 = (21 ,Y2) if y lies in an e-neighborhood o f ~. Next, it may be noted that a deterministic equilibrium 2-cycle (Yl,22), with Yl = f(Y2) and Y2 = f(21), is a limiting case o f an SSE when Jrll, Jr22 ---+ 0. Similarly, a pair o f distinct steady states, i.e. a pair (Yl,Y2) satisfying 21 ~ Y2, Yl = f ( Y l ) and Y2 =f(Y2), is a limiting case o f SSEs with Zrll, 7c22 ~ 1. It is easy to derive the following results for e-SSEs near deterministic equilibria: (i) I f f ' ( ~ l ) f ' ( ~ 2 ) ~ 1 holds for a 2-cycle (21,22), there is an e > 0 such that for all 0 < c ~ < c there exists an e'-SSE relative to (21,Y2). (ii) I f f ' ( 2 1 ) ~ 1 andf'(j32) ~ 1 at a pair o f distinct steady states (21,;v2), there is an e such that for all 0 < e' < e there exists an e'-SSE relative to (21,22) 66 (iii) There is an e > 0 such that for all 0 < e ~ < e there exists an e'-SSE relative to a single steady state 33 if and only if [f'(33)[ > 1. The overlapping generations models sketched above provide simple examples o f SSEs and e-SSEs since these models can exhibit multiple steady states, steady states with [f~(33)[ > 1 and cycles. To conclude the discussion on the existence of SSEs we remark here that for fully specified models it is sometimes possible to utilize arguments based on global analysis (such as the index theorem o f Poincar6 and Hopf) to prove the existence o f SSEs, see the surveys cited previously.

4.3.2. Analysis of learning 4.3.2.1. Formulation of the learning rule. For learning sunspot equilibria the agents must have a perceived law o f motion that in principle can enable them to learn such 66 This kind of SSE may be called an "animal spirits" cycle in accordance with Howitt and McAfee (1992).

G. W. Evans and S. Honkapohja

518

an REE: If agents believe that the economy is in an SSE, a natural estimator for the value o f y t in the two different sunspot states is, for each state of the sunspot process in the past, the computation o f the average o f the observations ofyt which have arisen in that state of the sunspot st. This is a form o f state-contingent averaging. Thus let q}t = (~blt, O2t) be the estimates o f the values that Yt takes in states 1 and 2 o f the sunspot. Let also ~ t = 1 if& = j and ~Pjt = 0 otherwise be the indicator function for state j o f the sunspot. The learning rules based on state-contingent averaging can be written in the form Ojt =Oj, t-I + t ll])/,t-lq/lt_l(Yt-l--Oj, t l-t-et 1),

qjt = qj, t l + t

l(~#/,t-1--qj, t-1),

(77)

Yt = lPlt[Ygllf(Olt) + (1 -- ~11)f(02t)] + /P2t [(1 -- a't'22)f(01t) + a'g22f(02t)]

for j = 1,2. We note here that in the learning rules agents are assumed to use observations only through period t - 1. This is to avoid a simultaneity between Yt and expectations E t f ( y,+l ). Equations (77) are interpreted as follows, tqj, t_ 1 is the number o f times state j has occurred up to time t - 1. The recursion for the fraction o f observations o f state j is the second of Equations (77). The first equation is then a recursive form for the state averages, with one modification to be discussed shortly. Finally, the third of Equations (77) gives the temporary equilibrium for the model, since the right-hand side is the expectation o f the value o f f ( y t + O given the forecasts ~jt. We make a small modification in the learning rule by including a random disturbance et to the algorithm. This can be interpreted as a measurement or observation error, and it is assumed to be iid with mean 0 and bounded support (tetl < C, C > 0, with probability 1) 67.

4.3.2.2. Analysis o f convergence. We now show that, under a stability condition, the learning rule (77) above converges locally to an SSE. For this we utilize the local convergence results reviewed in Section 2. First introduce the variables

O[=(01t, e)2t,qlt,q2t),

X / = (~Pl,t 1, ~P2,t-l, et-l)

and the functions

~J'(0t l , ~ t ) = q~j,t-lq/,lt-l(Yt 1-Oj, t 1 +Et 1), ~2+i(0t 1,Xt) = *Pi,t 1 - q i , t-1,

j=

1,2,

i = 1,2.

For state dynamics we note simply that Xt is a Markov process independent of 0t. The system is then in a standard form for recursive algorithms 6s. 67 The observation error is needed only for the instability result. 68 The formal analysis requires an extension of the basic framework of Section 2 to non-iid shocks or alternatively to Markovian state dynamics as summarized in the appendix of Evans and Honkapohja (1998b) and treated in detail in Evans and Honkapohja (1998a). The formal details for the former approach are given in Woodford (1990) and Evans and Honkapohja (1994c).

Ch. 7: LearningDynamics

519

The associated differential equation governing local convergence is d 0 / d r = h(O), where

hi(0) = Yglql[~ll/(~l)+( 1 - ~ 1 1 ) / ( ~ 2 ) - ~ 1 ] , h2(0) =~2q2[(1 13g22)f(01)w,7g22]c(~2)-~2], h3(0) = ~ l - q l , h4(0) = f g 2 - q 2 .

Here (~l, £c2) is the limiting distribution of the states of the Markov chain. Clearly, at the equilibrium point ql = Yrl, q2 = ~2 and (~1, q~2)is an SSE. In the ODE d 0 / d r = h(O) the subsystem consisting of the last two components of h(O) is independent of (Ol, q52) and one has global stability for it in the domain qi E (0, 1), i = 1, 2. It follows that the entire ODE is locally stable provided DT(e)I, ~2) has all eigenvalues with real parts less than unity, where

T(qJI' q~2) =

( T I ( ~ I , ~2) ~ = ( ; r l l f ( q ) l ) + (1 -- Jrl 1)f(~2) "~ T2(~bl,q~2)J (1-~22)f(Ol)+~22f(O2)J"

(78)

Note that the function T(q~I, 02) = [T1 (¢1, ~2), T2(~l, ¢2)] defines the mapping from the perceived law of motion [Yt+x = ~ if st+l = 1, yt+l = ¢2 if st+l = 2] to the actual law of motion [Yt+i = ¢~ ifst+l = 1, Yt+l = 0~ if St+l = 2], where (¢~,~b~) = T(¢~,¢2). The condition on the eigenvalues can thus be used to define the concept of E-stability for sunspot equilibria. We have obtained the following result:

Proposition 11. The learning rule (77) converges locally to an SSE (y~,y~) provided it is weakly E-stable, i.e. the eigenvalues of DT(y~,y~) have real parts less than one.

Remark: The notion of convergence is as in Theorem 1 in Section 2. If the algorithm is augmented with a projection facility, almost sure convergence is obtained. It is also possible to derive an instability result along the lines of Evans and Honkapohja (1994c) for SSEs which are not weakly E-stable:

Proposition 12. Suppose that an SSE (y~,y~) is weakly E-unstable, so that DT(y~,y~) has an eigenvalue with real part greater than unity. Then the learning dynamics (77) converge to (Yl ,Y2) with probability zero. The stability result can also be developed for the general model (26) or (69). In this framework sunspot equilibria are noisy, because the equilibrium is influenced by both the sunspot variable as well as the exogenous preference or technology shock. This is discussed in Evans and Honkapohja (1998a).

G.W.Evans and S. Honkapohja

520 4.3.3. Stability o f SSEs near deterministic solutions

The preceding result shows that local convergence to SSEs can be studied using E-stability based on Equation (78). Computing D T we have

DT(y) =

a-gllf'(yl) (1 - ~ll)f'(Y2) ) (1 - ~22)f'(Yl) a-g22f'(y2) "

The analysis of E-stability of SSEs near deterministic solutions (e-SSEs) is based on two observations. First, D T ( y ) can be computed for the deterministic solutions, which are limiting cases for e-SSEs. Second, under a regularity condition, the fact that eigenvalues are continuous functions o f the matrix elements provides the E-stability conditions for e-SSEs in a neighborhood o f the deterministic solution. This approach yields the following results: (i) Given a 2-cycle ~ = (YI,.~2) w i t h f ' ( ~ l ) f ' ( ~ 2 ) -~ 0, there is an e > 0 such that for all 0 < c' < e all e'-SSEs relative to ~ are weakly E-stable if and only if P is weakly E-stable, i.e., it s a t i s f i e s f ' ( ~ 0 f ' ( ~ 2 ) < 1. (ii) Given two distinct steady states Yl ¢ Y2 there is an e > 0 such that for all 0 < e ~ < e all e~-SSEs relative to ~ = (Yl,Y2) are weakly E-stable if and only if both steady states are weakly E-stable, i . e . , f ' ( ~ l ) < 1 a n d f ' ( ~ 2 ) < 1. Analogous results are available when a 2-cycle is strongly E-stable or a pair of distinct steady states are strongly E-stable [see Evans and Honkapohja (1994c) for the definition and details]. For the case o f a single steady state the situation is more complex, but the following partial result holds: Let ~ be a weakly E-unstable steady state, i.e. S ( . v ) > 1. Then there exists an e > 0 such that for all 0 < e' < e all c'-SSEs relative to .~ are weakly E-unstable. One may recall from Proposition 3 that SSEs near a single steady state ~ also exist when f ' ( ~ ) < -1. For this case it appears that both E-stable and E-trustable e-SSEs relative to 33 may exist. However, it can be shown that there is a neighborhood of such that SSEs in the neighborhood are E-unstable in a strong sense. 4.3.4. Applying the results to OG and other models The hyperinflation model, Example 4.4 in Section 4.2.1, has often been used as an economic example for sunspot equilibria. This construction relies on the two distinct steady states o f the model. The application o f the results above shows that such equilibria near a pair o f steady states are unstable under learning. In order to construct a robust example o f such "animal spirits" sunspot solutions it is necessary to have a pair o f steady states that are both stable when agents try to learn them. Since under certain regularity conditions two stable steady states are separated by an unstable one, the construction o f a robust example o f sunspot equilibria, which is based on distinct steady states, normally requires the existence o f three steady states at a minimum.

Ch. 7: LearningDynamics

521

The model of increasing social returns, Example 4.3 in Section 4.2.1, is a simple OG model with this property. Evans and Honkapohja (1993b) develop this extension and provide simulations illustrating convergence to such an SSE. Other similar robust examples of these endogenous fluctuations are the "animal spirits" equilibria in Howitt and McAfee (1992) in a model of search externalities, and equilibrium growth cycles in Evans, Honkapohja and Romer (1998b) in a model of endogenous growth with complementary capital goods. Alternatively, stable sunspot solutions can be obtained when the model exhibits a k-cycle which is by itself stable under learning. If such a k-cycle is found, then normally there also exist stable sunspot solutions nearby, provided agents allow for the possibility of sunspots in their learning behavior. OG models with downward-sloping offer curves provide simple examples of sunspot equilibria near deterministic cycles. In addition to these local results, the original analysis of Woodford (1990) showed how to use index theorem results to obtain global stability results for SSEs in the OG model.

5. Extensions and recent developments

In this section we take up several further topics that have been analyzed in the area of learning dynamics and macroeconomics. These include some alternative learning algorithms, heterogeneity of learning rules, transitions and speed of convergence results, and learning in misspecified models. 5.1.

Genetic algorithms, classifier systems and neural networks

Some of the models for learning behavior have their origins in computational intelligence. Genetic algorithms and classifier systems have found some applications in economics. 5.1.1.

Genetic algorithms

Genetic algorithms (GA) were initially designed for finding optima in non-smooth landscapes. We describe the main features of GAs using the Muth market model which is one of the very first applications of GAs to economics. The exposition follows Arifovic (1994). We thus consider a market with n firms with quadratic cost functions Cit = x q i t + 1 2 ~ynqit, where qit is the production by firm i, and x and y are parameters. Given 1 2 price expectations Pte the expected profit of firm i is l i f t = P te qit - xqit - ~ynqit, and one obtains the supply function for firm i as qit = ( y n ) - l ( P ~ - x ) . The demand function is taken to be pt - A - B ~ - 1 qit, and the RE solution Pt = P~ yields qit = qt = (A - x ) / [ n ( B +y)].

G. W. E v a n s a n d S. H o n k a p o h j a

522

Arifovic (1994) considers some alternative GA's. We outline here her "singlepopulation" algorithm. Formally, there is a population A t of 'chromosomes' Ait which are strings o f length ~ o f binary characters 0, 1 : a I

g

Air = ( it, . . . ~ ait)~

k

where

ait

= 0 or 1.

To each chromosome Air one associates a production decision by firm i by the formula g

xit qit = ~ ,

where

= V" ak 2 k 1 xit ~ it • k=l

Here k is a norming factor 69. Short-run profits/~t = Hit = Ptqit - Cit provide a measure of 'fitness' for alternative chromosomes (production decisions). Here P t is the shortrun equilibrium price, given a configuration o f n chromosomes. The basic idea in a genetic algorithm is to apply certain genetic operators to different chromosomes in order to produce new chromosomes. In these operators the fitness measure provides a criterion o f success, so that chromosomes with higher fitness have a better chance o f producing offsprings to the population. The following operators are used by Arifovic (1994): (1) R e p r o d u c t i o n : Each chromosome Air produces copies with a probability which depends on its fitness. The probability o f a copy Cit is given by P ( c i t ) = ~ A i t / ( ~ ; = 1 [Ait)" The resulting n copies constitute a 'mating pool'. (2) C r o s s o v e r : Two strings are selected randomly from the pool. Next, one selects a random cutoff point, and the tails o f the selected chromosomes are interchanged to obtain new chromosome strings. Example. If there are two strings [ 110101111 ] and [001010010], and tails o f length 4 are interchanged, then the new strings are [110100010] and [001011111]. Altogether n / 2 pairs are selected (assume that n is even, for simplicity). (3) M u t a t i o n : For each string created in step 2, in each position 0 and 1 is changed to the alternative value with a small probability. These are standard genetic operations. In her analysis Arifovic (1994) adds another operator which is not present in standard G A s 70. (4) E l e c t i o n : The new 'offsprings' created by the preceding three operators are tested against their 'parents' using the profit measured at the previous price as the fitness criterion. The rules for replacement are if one offspring is better than both parents, replace the less-fit parent, -

69 Note that for large g the expressions xit can approximate any real number over the range of interest. 70 The market model does not converge when this operator is absent. Since mutation is always occurring, unless it is made to die off asymptotically, something like the election operator must be utilized to get convergence.

Ch. 7: Learning Dynamics

523

if both offsprings are better, replace both parents, - if parents are better than offsprings, they stay in the population. These four operations determine a new population of size n and, given this configuration, a new short-run equilibrium price is determined by the equality of demand and output. After this the genetic operators are applied again using the new market price and profits as the fitness measure. Arifovic (t994) shows by simulations that this algorithm converges to the RE solution irrespective of the model parameter values 71. This result is remarkable, since it happens in spite of the myopia in the fitness criterion. (The system, however, has no stochastic shocks.) For some specifications it also turns out that the time paths of the GA corresponds reasonably well with certain experimental results for the market model. These genetic operations can be given broad interpretations in terms of economic behavior. First, reproduction corresponds to imitation of those who have done well. Second, crossover and mutation are like testing new ideas and making experiments. Finally, election means that only promising ideas are in fact utilized. To conclude this discussion we remark that as a model of learning the genetic algorithm is probably best interpreted as a framework of social rather than individual learning, cf. Sargent (1993). Indeed, individual firms are like individual chromosomes who are replaced by new ones according to the rules of the algorithm. -

5.1.2. Classifier systems Classifier Systems provide a different variety of learning algorithms which can be made more akin to thought processes of individuals than a GA. This allows a direct behavioral interpretation with individual economic agents doing the learning. A classifier system consists of an evolving collection of 'condition-action statements' (i.e. decision rules) which compete with each other in certain specified ways. The winners become the active decisions in the different stages. The strengths (or utility and costs) of the possible classifiers are a central part of the system and accounts are kept of these strengths. When a 'message' indicating current conditions arrives, one or more classifiers are activated as the possible decisions given the signal. Next, the competition stage starts to select the active classifier. The strengths are updated according to the performance of the active classifier. (The updating rules in fact mimic the updating of parameter estimates in stochastic approximation.) Typically, there are also ways for introducing new classifiers 72. A well-known economic application of classifier systems is Marimon, McGrattan and Sargent (1989). They introduce classifier system learning into the model of money

71 Thisfinding is consistent with the E-stabilitycondition and corresponds to the Least Squares learning results, see Sections 1.4.1 and 2.7.2: downward sloping demand and upward sloping supply is sufficient for global convergence. 72 Sargent (1993), pp. 77-81, and Dawid (1996), pp. 13-171 provide somewhat more detailed descriptions of classifier systems.

524

G. W. Evans and S. Honkapohja

and matching due to Kiyotaki and Wright (1989). Using simulations Marimon et al. show that learning converges to a stationary Nash equilibrium in the Kiyotaki-Wright model, and that, when there are multiple equilibria, learning selects the fundamental low-cost solution. Another recent application is Lettau and Uhlig (1999). They utilize a classifier system as a rule-of-thumb decision procedure in the usual dynamic programming setup for consumption-saving decisions. The system does not fully converge to the dynamic programming solution, and Lettau and Uhlig suggest that this behavior can account for the 'excess' sensitivity of consumption to current income. 5.1.3. Neural networks

Another very recent approach to learning models based on computational intelligence has been the use o f neural networks 73. The basic idea in neural networks is to represent an unknown functional relationship between inputs and outputs in terms o f a network structure. In general the networks can consist o f several layers o f nodes, called neurons, and connections between these neurons. The simplest example o f a network is the perceptron which is a single neuron receiving several input signals and sending out a scalar output. Infeedforward networks information flows only forward from one layer o f neurons to a subsequent one. Such a network usually has several layers o f neurons, organized so that neurons at the same layer are not connected to each other, and neurons in later layers do not feed information back to earlier layers in the structure. In network structures signals are passed along specified connections between the different neurons in the network. In each neuron input signals are weighted by some weights and the aggregate is processed through an activation function of that neuron. The processed signal is the output from that neuron, and it is sent to further neurons connected to it or if at the terminal layer as a component o f the output o f the whole network. A n important property o f these networks is that they can provide good approximations o f the unknown functional relation between the inputs and the outputs. To achieve this the networks must be 'trained': the weights for inputs at each neuron must be determined so that, given the training data, the network approximates well the functional relation present in the input and output data. This training is often based on numerical techniques such as the gradient method, and in fact many training schemes can be represented as stochastic approximation algorithms. The training can be done with a fixed data set, so that it is then an 'off-line' algorithm, or it may been done 'on-line' as a recursive scheme. In the latter case the basic setup corresponds closely to adaptive learning.

73 The use of neural networks in economics is discussed e.g. in Beltratti, Margarita and Terna (1996), Cho and Sargent (1996b), and Sargent (1993). White (1992) is an advanced treatise discussing the relationship of neural networks to statistics and econometrics.

Ch. 7: Learning Dynamics

525

In economic theory, neural networks have very recently been utilized as representations of approximate functional forms, as computational devices and as an approach to bounded rationality and learning. One use of neural networks has been the computation of (approximate) solutions to economic models, see e.g. Beltratfi, Margarita and Terna (1996) for various illustrations from economics and finance. Another use of neural networks has been in modelling bounded rationality and learning. Cho (1995) uses perceptrons in the repeated prisoner's dilemma game, so that the perceptrons classify the past data and through a threshold this leads to a decision in accordance with the output of the perceptron. Such strategies are quite simple, and thus the modeled behavior is very much boundedly rational. Nevertheless, the efficient outcomes of the game can be recovered by use of these simple strategies. Cho and Sargent (1996a) apply this approach to study reputation issues in monetary policy. Other papers using neural networks as a learning device in macroeconomic models include Barucci and Landi (1995), Salmon (1995), Packal~n (1997) and Heinemann (1997a). The last two studies look at connections to E-stability in the Muth model. 5.1.4. Recent applications o f genetic algorithms

The paper by Arifovic (1994) demonstrated the potential of GAs to converge to the REE, and a natural question is whether such convergence occurs in other models, and whether, when there are multiple equilibria, there is a one-to-one correspondence between solutions which are stable under statistical or econometric learning rules and solutions which are stable under GAs. The expectational stability principle, which states that there is a close connection between stability under adaptive learning rules and expectational stability, would argue for a tight correspondence between stability under econometric learning and under GAs. One setup in which this question can be investigated is the OG model with seignorage, in which a fixed real deficit is financed by printing money. Recall that, provided the level of the deficit is not too large, there are two REE monetary steady states. E-stability and stability under adaptive learning was discussed in Sections 1.4.3 and 1.4.4. Under small-gain adaptive learning of the inflation rate, the lowinflation steady state is locally stable while the high-inflation steady state is locally unstable, consistent with the E-stability results. Learning in this model was actually first investigated under least-squares learning by Marcet and Sargent (1989a). They assumed that agents forecast inflation according to the perceived law of motion Pt+l = [3tPt, where fit is given by the least squares regression (without intercept) of prices on lagged prices. They showed that there could be convergence only to the lowinflation steady state, never to the high-inflation steady state. In addition, in simulations they found some cases with unstable paths leading to expected inflation rates at which there was no temporary equilibrium (i.e., at which it was impossible to finance the deficit through money creation). Arifovic (1995) sets up the GA so that the chromosome level represents the first period consumption of the young. Using GA simulations (with an election operator),

526

G. W. Evans and S. Honkapohja

she also finds convergence to the low-inflation steady state and never to the highinflation steady state. There are some differences in detail from Least Squares learning. From some starting points which lead to unstable paths under (Marcet-Sargent) leastsquares learning there was convergence under GA learning. It is possible that some of these apparent discrepancies arise from the particular least-squares learning scheme followed. Since the price level in either steady state is a trended series, whereas the inflation rate is not, it would be more natural to an econometrician to estimate the inflation rate by its sample mean rather than by a regression of prices on past prices. In any case, there does appear to be a close connection in this model between the local stability properties of statistical and GA learning, and the key features of learning dynamics are revealed by E-stability. In Bullard and Duffy (1998a), GAs are used to look at the issue of convergence to cycles in the standard deterministic OG endowment model with money. Recall that Grandmont (1985) showed that for appropriate utility functions it is straightforward to construct models in which there are regular perfect foresight cycles. Recall also that Guesnerie and Woodford (1991) and Evans and Honkapohja (1995c) provide local stability conditions for the convergence of adaptive and statistical learning rules to particular RE k-cycles. For "decreasing-gain" rules these are the E-stability conditions which are given in the above section on nonlinear models. It is therefore of interest to know whether GAs exhibit the same stability conditions. In Bullard and Duffy (1998a) agent i uses the following simple rule for forecasting next period's price: F [ [ P ( t + 1)] = P ( t - ki - 1). Different values of ki are consistent with different perfect foresight cycles. (Note that every value of ki is consistent with learning steady states). The value of ki used by agent i is coded as a bit string of length 8, so that the learning rule is in principle capable of learning cycles up to order 39. Given their price forecast, each agent chooses its optimal level of saving when young and total saving determines the price level. A GA is used to determine the values o f k i used in each generation. Note that in this setup [in contrast to the approach in Arifovic (1994, 1995)] the GA operates on a forecast rule used by the agent, rather than directly on its decision variable 74. The question they ask is: starting from a random assignment of bit strings, will the GA converge to cycles? To answer this question they conduct GA simulations for a grid of values of the parameter specifying the relative risk aversion parameter of the old. Their central finding is that, with only a handful of exceptions, there is convergence either to steady states or 2-cycles, but not to higher-order cycles. This finding raises the possibility that GAs may have somewhat different stability properties than other learning rules. However, the results are based on simulations using a GA

74 This makes GA learning closer in spirit to least squares and other adaptive learning of forecast rules. Using GAs to determine forecast rules was introduced in Bullard and Duffy (1994). Bullard and Duffy (1998b) show how to use GAs to directly determine consumption plans in n-period OG endowment economies.

Ch. 7: Learning Dynamics

527

with a particular specification of the initial conditions and the forecast rule. Thus many issues concerning stability under GAs remain to be resolved 75. We close this section with a brief description of two other recent papers which use GAs in macroeconomic learning models. Arifovic (1996) considers an OG model with two currencies. This model possesses a continuum of stationary perfect foresight solutions indexed by the exchange rate. In the GA set-up each agent has a bit string which determines the consumption level and the portfolio fractions devoted to the two currencies. Fitness of string i used by a member of generation t - 1 is measured by its ex-post utility and is used to determine the proportion of bit strings in use in t + 1 according to genetic operator updating rules. The central finding is that the GA does not settle down to a nonstochastic stationary perfect foresight equilibrium, but instead exhibits persistent fluctuations in the exchange rate driven by fluctuations in portfolio fractions. Arifovic, Bullard and Duffy (1997) incorporate GA learning in a model of economic development based on Azariadis and Drazen (1990). This model, which emphasizes the roles of human capital and threshold externalities, has two perfect foresight steady states: a low-income zero-growth steady state and a highincome positive-growth steady state. In the GA set-up the bit strings encode the fraction of their time young agents spend in training and the proportion of their income they save 76. The central finding, based on simulations, is that, starting from the low-income steady state, economies eventually make a transition to the high-income steady state after a long, but unpredictable length of time. These examples illustrate that GAs can be readily adapted to investigate a wide range of macroeconomic models. An advantage of GAs in economics is that they automatically allow for heterogeneity. A disadvantage is that there are no formal convergence results. Although in some cases there are supporting theoretical arguments, the findings in economics to date rely primarily on simulations. This literature is growing fast. Dawid (1996) provides an overview of GAs and discusses their applications to both economic models and evolutionary games. Lettau (1997) considers the effects of learning via genetic algorithms in a model of portfolio choice. 5.2. Heterogeneity in learning behavior

In most of the literature on statistical and econometric learning it is assumed that the learning rules of economic agents are identical. This is a counterpart and an addition to the assumption of the existence of a representative agent. Some studies have considered models in which agents have different learning rules. An early example is Bray and Savin (1986), who allow for agents to have heterogeneous priors in the context of the Muth model. Howitt (1992) incorporates different learning rules in his analysis of the instability of interest rate pegging. Evans, Honkapohja and Marimon (1998a)

75 GA learning of 2-cycles has also recentlybeen investigatedin Arifovic (1998). 76 In this model all of the standard genetic operators are used except the election operator.

528

G. W. Evans and S. Honkapohja

extend the deficit financing inflation model to include a continuum of agents with identical savings functions but different learning rules. Marcet and Sargent (1989b) consider a model in which two classes of agents with different information form different expectations. Soerensen (1996) looks at adaptive learning with heterogeneous expectations in a nonstochastic OG model. In this literature there are two techniques for setting up and analyzing models with heterogenous learning. First, as pointed out by Marcet and Sargent (1989c), when setting up the problem as a recursive algorithm it is straightforward to allow for a finite range of possibly heterogeneous expectations by expanding the state vector accordingly. This is easily done when there are a finite number of different agent types. Second, in some models it may be possible to aggregate the different learning rules and obtain for mean expectations a rule that is amenable to standard techniques. Evans, Honkapohja and Marimon (1998a) is an example of this latter methodology. The stability conditions for learning are in general affected by behavioral heterogeneity. However, many models with heterogeneous agents make the assumption that the dynamics of endogenous variables in the reduced form depend only on average expectations 77. It turns out that, when the basic framework is linear, the stability condition for convergence of learning with heterogeneous expectations is identical to the corresponding condition when homogeneous expectations are imposed, see Evans and Honkapohja (1997). Finally, we remark that the models based on GAs and classifier systems discussed above can incorporate heterogeneity in learning behavior, as can the approach developed in Brock and Hommes (1997). Using the latter approach, Brock and de Fontnouvelle (1996) obtain analytical results on expectational diversity.

5.3. L e a r n i n g

in m i s s p e c i f i e d m o d e l s

In most of the literature it has been assumed that agents learn based on a PLM (perceived law of motion.) that is well specified, i.e. nests an REE of interest. However, economic agents, like econometricians, may fail to correctly specify the actual law of motion, even asymptotically. It may still be possible to analyze the resulting learning dynamics. An early example of this idea, in the context of a duopoly model, is Kirman (1983). Maussner (1997) is a recent paper focusing on monopolistic competition. As an illustration, consider the Muth model of Sections 1.2.1 and 1.4.1 with reduced form (4). Agents were assumed to have a PLM of the form P t = a + btwt_l + l"lt, corresponding to the REE. Suppose that instead their PLM is Pt = a + rh, so that

77 Frydman (1982) and some papers in the volume Frydman and Phelps (1983) have stressed the importance of average opinions.

Ch. 7: Learning Dynamics

529

agents do not recognize the dependence o f price on wt-1, and that they estimate a by least squares. Then at = at-i + t l ( p t - a t - l ) , and the PLM at time t - 1 is Pt = at-1 + ~/t with corresponding forecasts E~_lpt = at-l. Thus the A L M is Pt = t.t + aat-i + yl wt 1 + 17t and the corresponding stochastic recursive algorithm is at = at-1 + t 1(~+ (a - 1)at-1 + g'wt-1 + ~Tt). The associated ODE is d a / d z = ~t + (a - 1)a, and thus from Section 2 it follows that at --+ fi = (1 - a ) - l g almost surely. (We remark that the ODE da/d'c can also be interpreted as the E-stability equation for the underparameterized class o f PLMs here considered). In this case we have convergence, but it is not to the unique REE which is Pt = (1 - a) 1# + (1 - a)-ly~wt I + rh. Agents make systematic forecast errors since their forecast errors are correlated with wt-i and they would do better to condition their forecasts on this variable. However, we have ruled this out by assumption: we have restricted PLMs to those which do not depend on wt-1. Within the restricted class o f PLMs we consider, agents in fact converge to one which is rational given this restriction. The resulting solution when the forecasts are Ei*lpt = fi is pt = (1 - a ) - l ~ + Y'w,-1 + ~,. We might describe this as a restricted perceptions equilibrium since it is generated by expectations which are optimal within a limited class o f PLMs. The basic idea o f a restricted perceptions equilibrium is that we permit agents to fall short o f rationality specifically in failing to recognize certain patterns or correlations in the data. Clearly, for this concept to be "reasonable" in a particular application, the pattern or correlation should not be obvious. In a recent paper, Hommes and Sorger (1998) have proposed the related, but in general more stringent, concept o f consistent expectations equilibria. This requires that agents correctly perceive all autocorrelations o f the process. The restricted perceptions equilibrium concept is closely related to the notion o f reduced order limited information R E E introduced in Sargent (1991). Sargent considers the Townsend (1983) model in which two classes o f agents have different information sets and each class forms expectations based on a PLM which is a fixed-order vector A R M A process, e.g. a first-order A R process. This gives a mapping from the PLM to the A L M and a fixed point o f this map is a limited information REE, which was studied under learning in Marcet and Sargent (1989b). Sargent shows that this solution has reduced order, i.e. agents could make better forecasts using a higher-order

530

G. W. Evans and S. Honkapohja

ARMA process. In Sargent (1991), agents use an ARMA process, which is shown to yield full-order equilibrium 78. Some recent literature has explored learning dynamics in economies which are subject to recurrent structural shifts. As pointed out in Evans and Honkapohja (1993a), there are in principle two approaches if agents understand that these shifts will recur. One approach is for them to construct a hypermodel which allows for the structural shifts. If the agents misspecify such a model, they may converge to a restricted perceptions equilibrium, as above. An alternative approach is to allow for the structural shifts using a constant- or nondecreasing-gain learning algorithm which can potentially track the structural change. The constant-gain procedure was followed in Evans and Honkapohja (1993a). The choice of gain parameter involves a trade-off between its tracking ability and forecast variance, and an equilibrium in this class of learning rules was obtained numerically. In this kind of framework, policy can exhibit hysteresis effects if the model has multiple steady states. The recent analysis of Sargent (1999) also employs a constant-gain algorithm. In two recent papers the agents use algorithms in which the gain parameter is reset as a result of structural change. Timmermann (1995) looks at an asset pricing model with decreasing gain between structural breaks. It is assumed that agents know when a structural change has occurred and reset their gain parameters accordingly. This leads to persistent learning dynamics with greater asset price volatility 79. Marcet and Nicolini (1998) consider the inflation experience in some Latin American countries. Using an open economy version of the seignorage model in which the level of seignorage is exogenous and random, they assume that agents use decreasing gain unless recent forecast errors are high, in which case they revert to a higher fixed gain. They show that under this set-up the learning rule satisfies certain reasonable properties. Under their framework, recurrent bouts of hyperinflation are possible, and are better explained than under rational expectations. 5.4. Experimental evidence

Since adaptive learning can have strong implications for economic dynamics, experimental evidence in dynamic expectations models is of considerable interest. However, to date only a relatively small number of experiments have been undertaken. The limited evidence available seems to show that, when convergent, time paths from experimental data converge towards steady states which are stable under smallgain adaptive learning. Perhaps the clearest results are from experiments based on

78 Evans, Honkapohjaand Sargent (1993) consider an equilibrium in which a proportion of agents have perfect foresight and the rest, econometricians,have the optimal model from a restricted class of PLMs. Mitra (1997) considers a model with these two types of agents in which the econometricianschoose an optimal memory length. 79 In Timmermann(1993, 1996) excess asset price volatility is shown during the learning transition in a model with no structural breaks.

Ch. 7: Learning Dynamics

531

the hyperinflation (seignorage) OG model. Recall that in this model the high real balance/low-inflation steady state is E-stable, and thus stable under adaptive learning, whereas the low real balance/high-inflation steady state is unstable 8o. This theoretical result is strongly supported by the experiments described in Marimon and Sunder (1993) [related experiments are reported in Arifovic (1995)]: convergence is always to the high real balance steady state and never to the low real balance steady state. Marimon, Spear and Sunder (1993) consider endogenous fluctuations (2-cycles and sunspot equilibria) in the basic OG model. Their results are mixed: persistent, beliefdriven cycles can emerge, but only after the pattern has been induced by corresponding fundamental shocks. These papers also consider some aspects of transitional learning dynamics. One aspect that clearly emerges is that heterogeneity of expectations is important: individual data show considerable variability. Arifovic (1996) conducts experiments in the 2-currency OG model in which there is a continuum of equilibrium exchange rates. These experiments exhibit persistent exchange rate fluctuations, which are consistent with GA learning. For the same model, using a Newton method for learning decision rules, simulations by Sargent (1993), pp. 107-112, suggest path-dependent convergence to a nonstochastic REE. These results raise several issues. First, it would be useful to simulate learning rules like the Newton method with heterogeneous agents and alternative gain sequences. Second, given the existence of sunspot equilibria in models of this type one should also investigate whether such solutions are stable under adaptive learning. Finally, Marimon and Sunder (1994) and Evans, Honkapohja and Marimon (1998a) introduce policy changes into experimental OG economies with seignorage. The former paper considers the effects of prealmounced policy changes. The results are difficult to reconcile with rational expectations but the data are more consistent with an adaptive learning process. The latter paper introduces a constitutional constraint on seignorage which can lead to three steady states, two of which are stable under learning. The experiments appear to confirm that these are the attractors. The learning rules in this paper incorporate heterogeneity with random gain sequences, inertia and experimentation. This generates considerable diversity and variability during the learning transition which has the potential to match many aspects of experimental data. 5.5. Further topics

The speed of convergence for learning algorithms is evidently an important issue for the study of learning behavior. The self-referential nature of many learning models invalidates the direct application of the corresponding results from classical statistics. At present very few studies exist on this subject. An analytic result on asymptotic speed of convergence for stochastic approximation algorithms is provided in Benveniste,

80 At least providedthe gain is sufficientlysmall. See Sections 1.4.3 and 1.4.4.

532

G. W. Evans and S. Honkapohja

Metivier and Priouret (1990), on pp. 110 and 332. In particular, suppose that the gain sequence is 7t = C/t. Then, provided the real parts of all eigenvalues of the derivative of the associated ODE are less than -0.5, asymptotic convergence occurs at rate v't. (No analytic results are available in this case if the eigenvalue condition fails.) Marcet and Sargent (1995) have applied this result to adaptive learning in a version of the Cagan inflation model. They also carried out Monte Carlo simulations. The numerical results appear to accord with the analytics if the model satisfies the eigenvalue condition. However, the speed of convergence can be very slow when the eigenvalue condition fails 81. In the discussion of statistical learning procedures it is a standard assumption that the PLM can be specified parametrically. However, just as an econometrician may not know the appropriate functional form it may be reasonable to assume that agents face the same difficulty. In this case a natural procedure is to use nonparametric techniques. This is discussed in Chen and White (1998). As an illustration consider learning a noisy steady state in a nonlinear model (26) in Section 2.7.1 which we repeat here for convenience: Yt = H(E[G(yt+I, Vt+l), vt). Previously, the shock was assumed to be iid and in this case a noisy steady state y(vt) could be described in terms of a scalar parameter 0* = EG(y(v), v) (here the expectation is taken with respect to the distribution of v). Chen and White (1998) instead consider the case where vt is an exogenous, stationary and possibly nonlinear AR(1) process. A natural PLM is now of the form Ei*G(yt+l, vt+~) = O(vt), and under appropriate assumptions there exists an REE O(vt) in this class. Agents are assumed to update their PLM using recursive kernel methods of the form

Or(o)

=

Ot 1(0) + t -1 [G(yt, v,) - Ot-i (v)] 91((vt - vt 1)/ht)/ht,

where 9l(.) is a kernel function (i.e. a density which is symmetric around zero) and {ht} is a sequence of bandwidths (i.e. a sequence of positive numbers decreasing to zero). Chen and White establish that under a number of technical assumptions and an E-stability-like condition the learning mechanism converges to O(vt) almost surely, provided a version of the projection facility is employed. Another new approach employs models in which agents choose a predictor from some class of expectation functions. Brock and Hommes (1997) suggest the notion of an adaptively rational expectations equilibrium in which agents make a choice among finitely many expectations functions on the basis of past performance. This choice is coupled with the dynamics of endogenous variables, and the resulting dynamics can sometimes lead to complicated global dynamics. A related paper is Hommes and Sorger (1998). The approach is similar in spirit to models of choice of forecasting functions in the presence of nonlinear dynamics or structural shifts, cf. Evans and Honkapohja (1993a), Marcet and Nicolini (1998), and Mitra (1997).

gl Vives (1993) has establisheda similar asymptoticspeed of convergenceresult for Bayesianlearning.

Ch. 7: LearningDynamics

533

6. Conclusions

Increasingly, macroeconomists are investigating models in which multiple rational expectations equilibria can arise. Traditionally, this was considered theoretically awkward: which solution would the economy follow? Examining adaptive learning in such circumstances is particularly fruitful. Requiring stability of equilibria under adaptive learning can greatly reduce the degree of multiplicity. In some models there is a unique equilibrium which is (locally) stable under learning, while other models can have more than one stable equilibrium. Even in the latter case, incorporating learning dynamics provides a resolution of the indeterminacy issue, since models with multiple stable equilibria are converted into models with path dependence. The dynamics of such an economy are determined by its initial conditions (including expectations) and by the equations of motion which include the learning rules as well as the usual structural equations of the model. In particular, the ultimate equilibrium can in part be determined by the sequence of random shocks during the transition. As was indicated above, there is some experimental evidence supporting the important role played by adaptive learning in models with multiplicity. A number of important policy issues can arise in such models, and learning dynamics need to be taken into account in formulating economic policies. In some cases policy rules can lead to unstable economic systems even though the equilibria themselves may seem satisfactory. In cases with multiple stable equilibria, the path dependence exhibited in models with adaptive learning can lead to hysteresis effects with changes in policy. In addition, temporarily inefficient policies may be necessary to guide the economy to a superior equilibrium. Finally, even in cases with a unique equilibrium, learning dynamics can be important in characterizing data in situations where there are sudden changes in policy regimes. The dynamics with learning can be very different from fully rational adjustments after such a change. Although our discussion has focused most heavily on asymptotic convergence to REE, some of these other issues, which have been less studied, are likely to receive more attention in the future. Learning dynamics is a new area of research where many issues are still open and new avenues no doubt remain to be discovered. We look forward to future work with excitement.

References

Adam, M., and A. Szafarz (1992), "Speculativebubbles and financial markets", Oxford EconomicPapers 44:626--640. Amman, H.M., D.A. Kendrick and J. Rust, eds (1996), Handbook of Computational Economics, vol. 1 (Elsevier, Amsterdam). Arifovic, J. (1994), "Genetic algorithmlearning and the cobweb model", Journal of Economic Dynamics and Control 18:3-28.

534

G. W. Evans and S. Honkapohja

Arifovic, J. (1995), "Genetic algorithms and inflationary economies", Journal of Monetary Economics 36:219-243. Arifovic, J. (1996), "The behavior of the exchange rate in the genetic algorithm and experimental economies", Journal of Political Economy 104:510-541. Arifovic, J. (1998), "Stability of equilibria under genetic algorithm adaption: an analysis", Macroeconomic Dynamics 2:1-21. Arifovic, J., J. Bullard and J. Duffy (1997), "The transition from stagnation to growth: an adaptive learning approach", Journal of Economic Growth 2:185-209. Arthur, W.B. (1994), Increasing Returns and Path Dependence in the Economy (The University of Michigan Press, Ann Arbor, MI). Arthur, W.B., Y.M. Ermoliev and Y.M. Kaniovski (1983), "On generalized urn schemes of the Polya kind", Kibernetica 19:49-56. Arthur, W.B., Y.M. Ermoliev and Y.M. Kaniovski (1994), "Strong laws for a class of path-dependent stochastic processes with applications", in: Arthur (1994), chap. 10, pp. 185-201. Auster, R. (1971), "The invariably stable cobweb model", Review of Economic Studies 38:117-121. Azariadis, C. (1981), "Self-fialfilling prophecies", Journal of Economic Theory 25:380-396. Azariadis, C., and A. Drazen (1990), "Threshold externalities in economic development", The Quarterly Journal of Economics 104:501-526. Balasko, Y. (1994), "The expectational stability of Walrasian equilibria", Journal of Mathematical Economics 23:179-203. Balasko, Y., and D. Royer (1996), "Stability of competitive equilibrium with respect to recursive and learning processes", Journal of Economic Theory 68:319 348. Barnett, W., J. Geweke and K. Shell, eds (1989), Economic Complexity: Chaos, Sunspots, Bubbles, and Nonlinearity (Cambridge University Press, Cambridge). Barnett, W., et al., eds (1991), Equilibrium Theory and Applications, Proceedings of the Sixth International Symposium in Economic Theory and Econometrics (Cambridge University Press, Cambridge). Barucci, E., and L. Landi (1995), "Non-parametric versus linear learning devices: a procedural perspective", Working paper (University of Florence). Beltratti, A., S. Margarita and E Terna (1996), Neural Networks for Economic and Financial Modelling (International Thompson Computer Press, London). B~nassy, J., and M. Blad (1989), "On learning and rational expectations in an overlapping generations model", Journal of Economic Dynamics and Control 13:379400. Benhabib, J., and R.E. Farmer (1994), "Indeterminacy and increasing returns", Journal of Economic Theory 63:194 1. Benveniste, A., M. Metivier and P. Priouret (1990), Adaptive Algorithms and Stochastic Approximations (Springer, Berlin). Bergstr6m, V, and A.E. Vredin, eds (1994), Measuring and Interpreting Business Cycles (Oxford University Press, Oxford). Bertocchi, G., and W. Yong (1996), "Imperfect information, Bayesian learning and capital accumulation", Journal of Economic Growth 1:487-503. Binmore, K. (1987), "Modeling rational players", Economics and Philosophy 3:179-214. Blanchard, O.J., and S. Fischer (1989), Lectures on Macroeconomics (MIT Press, Cambridge, MA). Blanchard, O.J., and C.M. Kahn (1980), "The solution of linear difference models under rational expectations", Econometrica 48:1305-1311. Blume, L.E., and D. Easley (1982), "Learning to be rational", Journal of Economic Theory 26:340-351. B6hm, V, and J. Wenzelburger (1995), "Expectations, forecasting, and perfect foresight - a dynamical systems approach", Discussion Paper 307 (University of Bielefeld). Boldrin, M., and M. Woodford (1990), "Equilibrium models displaying endogenous fluctuations and chaos", Journal of Monetary Economics 25:189 222. Bossaerts, E (1995), "The econometrics of learning in financial markets", Econometric Theory 11: 151-189.

Ch. 7: LearningDynamics

535

Bray, M. (1982), "Learning, estimation, and the stability of rational expectations equilibria", Journal of Economic Theory 26:318-339. Bray, M., and D.M. Kreps (1987), "Rational learning and rational expectations", in: Feiwel (1987), chap. 19, pp. 597-625. Bray, M., and N. Savin (1986), "Rational expectations equilibria, learning, and model specification", Econometrica 54:1129-1160. Bray, M., L.E. Blume and D. Easley (1982), "Introduction to the stability of rational expectations", Journal of Economic Theory 26:313-317. Brock, W.A., and E de Fontnouvelle (1996), "Expectational diversity in monetary economics", Working Paper SSRI 9624 (University of Wisconsin-Madison). Brock, W.A., and C.H. Hornmes (1996), "Models of complexity in economics and finance", Working paper (University of Wisconsin, Madison). Brock, WA., and C.H. Hommes (1997), "A rational route to randomness", Econometrica 65:1059-1095. Broze, L., C. Gourieroux and A. Szafarz (1985), "Solutions of dynamic linear rational expectations models", Econometric Theory 1:341-368. Broze, L., C. Gourieroux and A. Szafarz (1990), Reduced Forms of Rational Expectations Models. Fundamental of Pure and Applied Economics (Harwood Academic Publishers). Bruno, M. (1989), "Econometrics and the design of economic reform", Econometrica 57:275 306. Bullard, J. (1992), "Time-varying parameters and nonconvergence to rational expectations under least squares learning", Economics Letters 40:159-166. Bullard, J. (1994), "Learning equilibria", Journal of Economic Theory 64:468-485. Bullard, J., and J. Duffy (1994), "Using genetic algorithms to model the evolution of heterogeneous beliefs", Working paper (Federal Reserve Bank of St. Louis). Bullard, J., and J. Duffy (1998a), "Learning and the stability of cycles", Macroeconomic Dynamics 2:22-48. Bullard, J., and J. Duffy (1998b), "A model of learning and emulation with artificial adaptive agents", Journal of Economic Dynamics and Control 22:179-207. Carlson, J. (1968), "An invariably stable cobweb model", Review of Economic Studies 35:360-363. Cass, D., and K. Shell (1983), "Do sunspots matter?", Journal of Political Economy 91:193-227. Champsaur, E (1983), "On the stability of rational expectations equilibria", Working Paper 8324 (CORE). Champsaur, P., et al., eds (1990), Essays in Honor of Edmond Malinvaud, vol. 1. Microeconomics (MIT Press, Cambridge, MA). Chatterji, S., and S.K. Cbattopadhyay (1997), "Global stability in spite of 'local instability' with learning in general equilibrium models", Working Paper WP-AD 97-11 (IVIE). Chen, X., and H. White (1998), "Nonparametric adaptive learning with feedback", Journal of Economic Theory 82:190522. Chiappori, EA., and R. Guesnerie (1991), "Sunspot equilibria in sequential market models", in: Hildenbrand and Sormenschein (1991) 1683-1762. Cho, I.-K. (1995), "Perceptrons play the repeated prisoner's dilemma", Journal of Economic Theory 67:266-284. Cho, I.-K., and T.J. Sargent (1996a), "Learning to be credible", Working paper (Brown University). Cho, I.-K., and TA. Sargent (1996b), "Neural networks for encoding and adapting in dynamic economies", in: Amman, Kendriek and Rust (1996) 441-470. Christiano, L.J., and V. Valdivia (1994), "Notes on solving models using a linearization method", mimeograph (Northwestern University). Crawford, V.E (1995), "Adaptive dynamics in coordination games", Econometrica 63:103-143. Currie, D., A. Garratt and S. Hall (1993), "Consistent expectations and learning in large scale macroeconometric models", in: Honkapohja and ingberg (1993), pp. 21-42. d'Autume, A. (1990), "On the solution of linear difference equations with rational expectations", Review of Economic Studies 57:672688.

536

G. W. Evans and S. Honkapohja

Dawid, H. (1996), Adaptive Learning by Genetic Algorithms: Analytical Results and Applications to Economic Models (Springer, Berlin). DeCanio, S. (1979), "Rational expectations and learning from experience", The Quarterly Journal of Economics 94:47-57. Dixon, H., and N. Rankin, eds (1995), The New Macroeconomics: Imperfect Markets and Policy Effectiveness (Cambridge University Press, Cambridge). Duffy, J. (1994), "On learning and the nonuniqueness of equilibrium in an overlapping generations model with fiat money", Journal of Economic Theory 64:541 553. Ellison, G., and D. Fudenberg (1995), "Word-of-mouth corunmnication and social learning", Quarterly Journal of Economics 110:93-125. Evans, G.W. (1983), "The stability &rational expectations in macroeeonomic models", in: Frydman and Phelps (1983), chap. 4, pp. 67-94. Evans, G.W. (1985), "Expectational stability and the multiple equilibria problem in linear rational expectations models", The Quarterly Journal of Economics 100:1217-1233. Evans, G.W. (1986), "Selection criteria for models with non-uniqueness", Journal of Monetary Economics 18:147-157. Evans, G.W. (1989), "The fragility of sunspots and bubbles", Journal of Monetary Economics 23: 297-317. Evans, G.W., and R. Guesnerie (1993), "Rationalizability, strong rationality, and expectational stability", Games and Economic Behaviour 5:632-646. Evans, G.W., and S. Honkapohja (1986), "A complete characterization of ARMA solutions to linear rational expectations models", Review of Economic Studies 53:227-239. Evans, G.W., and S. Honkapohja (1992), "On the robustness of bubbles in linear RE models", International Economic Review 33:1-14. Evans, G.W., and S. Honkapohja (1993a), "Adaptive forecasts, hysteresis and endogenous fluctuations", Federal Reserve Bank of San Francisco Economic Review 1993(1):3-13. Evans, G.W., and S. Honkapohja (1993b), "Learning and economic fluctuations: using fiscal policy to steer expectations", European Economic Review 37:595-602. Evans, G.W., and S. Honkapohja (1994a), "Convergence of least squares learning to a non-stationary equilibrium", Economic Letters 46:131-136. Evans, G.W., and S. Honkapohja (1994b), "Learning, convergence, and stability with multiple rational expectations equilibria", European Economic Review 38:1071-1098. Evans, G.W., and S. Honkapohja (1994c), "On the local stability of sunspot equilibria under adaptive learning rules", Journal of Economic Theory 64:142-161. Evans, G.W., and S. Honkapohja (1995a), "Adaptive learning and expectational stability: an introduction", in: Kirrnan and Salmon (1995), chap. 4, pp. 102 126. Evans, G.W., and S. Honkapohja (1995b), "Increasing social returns, learning and bifurcation phenomena", in: Kirman and Salmon (1995), chap. 7, pp. 216-235. Evans, G.W., and S. Honkapohja (1995e), "Local convergence of recursive learning to steady states and cycles in stochastic nonlinear models", Econometrica 63:195-206. Evans, G.W., and S. Honkapohja (1997), "Least squares learning with heterogeneous expectations", Economic Letters 52:197~01. Evans, G.W., and S. Honkapohja (1998a), "Convergence of learning algorithms without a projection facility", Journal of Mathematical Economics 30:59-86. Evans, G.W., and S. Honkapohja (1998b), "Economic dynamics with learning: new stability results", Review of Economic Studies 65:23~44. Evans, G.W., and S. Honkapohja (1999a), Learning and Expectations in Macroeconomics, book manuscript (Eugene, OR and Helsinki). Evans, G.W., and S. Honkapohja (1999b), "Convergence for difference equations with vanishing time dependence, with applications to adaptive learning", Economic Theory, forthcoming.

Ch. 7." Learning Dynamics

537

Evans, G.W., and G. Ramey (1992), "Expectations calculation and currency collapse", American Economic Review 82:207-224. Evans, G.W, and G. Ramey (1995), "Expectation calculation, hyperinflation and currency collapse", in: Dixon and Rankin (1995), chap. 15, pp. 307-336. Evans, G.W, and G. Ramey (1998), "Calculation, adaptation and rational expectations", Macroeconomic Dynamics 2:156 182. Evans, G.W, S. Honkapohja and T.J. Sargent (1993), "On the preservation of deterministic cycles when some agents perceive them to be random fluctuations", Journal of Economic Dynamics and Control 17:705-721. Evans, G.W, S. Honkapohja and R. Marimon (1998a), "Convergence in monetary inflation models with heterogenous learning rules", Discussion paper 386 (Department of Economics, University of Helsinki). Evans, G.W, S. Honkapohja and EM. Romer (1998b), "Growth cycles", American Economic Review 88:495-515. Farmer, R.E. (1991), "Sticky prices", The Economic Journal 101:1369-1379. Farmer, R.E. (1993), The Economics of Self-Fulfilling Prophecies (MIT Press, Cambridge, MA). Farmer, R.E., and J.-T. Guo (1994), "Real business cycles and the animal spirits hypothesis", The Journal of Economic Theory 63:42-72. Feiwel, G.R., ed. (1987), Arrow and the Ascent of Modern Economic Theory (New York University Press, New York). Feldman, M. (1987a), "Bayesian learning and convergence to rational expectations", Journal of Mathematical Economics 16:297-313. Feldman, M. (1987b), "An example of convergence to rational expectations with heterogeneous beliefs", International Economic Review 28(3):635-650. Fourgeand, C., C. Gourieroux and J. Pradel (1986), "Learning procedures and convergence to rationality", Econometrica 54:845-868. Friedman, D. (1991), "Evolutionary games in economics", Econometrica 59:632665. Frydman, R. (1982), "Towards an understanding of market procesesses: individual expectations, learning, and convergence to rational expectations equilibrium", American Economic Review 72:652-668. Frydman, R., and E.E. Phelps (1983), Individual Forecasting and Aggregate Outcomes, "Rational Expectations" Reexamined (Cambridge University Press, Cambridge). Fuchs, G. (1977), "Formation of expectations: a model in temporary general equilibrium theory", Journal of Mathematical Economics 4:167 187. Fuchs, G. (1979), "Is error learning behavior stabilizing?", Journal of Economic Theory 20:300-317. Fuchs, G., and G. Laroque (1976), "Dynamics of temporary equilibria and expectations", Econometrica 44:1157-1178. Fudenberg, D., and D.M. Kreps (1993), "Learning mixed equilibria", Games and Economic Behaviour 5:320-367. Fudenberg, D., and D.M. Kreps (1995), "Learning in extensive-form games I. Self-confirming equilibria", Games and Economic Behaviour 8:20-55. Fudenberg, D., and D.K. Levine (1998), Theory of Learning in Games (MIT Press, Cambridge, MA). Fuhrer, J.C., and M.A. Hooker (1993), "Learning about monetary regime shifts in an overlapping wage contract model", Journal of Economic Dynamics and Control 17:531-553. Gale, D. (1996), "What have we learned from social learning?", European Economic Review 40: 617-628. Garratt, A., and S. Hall (1997), "E-equilibria and adaptive expectations: output and inflation in the LBS model", Journal of Economic Dynamics and Control 21:1149-1171. Geanakoplos, J.D., and H.M. Polemarchakis (1991), "Overlapping generations", in: Hildenbrand and Sonnenschein (1991), chap. 35, pp. 1899-1960. Gottfries, N. (1985), "Multiple perfect foresight equilibriums and convergence of learning processes", Journal of Money, Credit and Banking 17:111-117.

538

(7. W. Evans and S. Honkapohja

Gourieroux, C., J. Laffont and A. Monfort (1982), "Rational expectations in dynamic linear models: analysis of the solutions", Econometrica 50:409-425. Grandmont, J.-M. (1985), "On endogenous competitive business cycles", Econometrica 53:995 1045. Grandmont, J.-M. (1998), "Expectations formation and stability of large socioeconomic systems", Econometrica 66:741~81. Grandmont, J.-M., and G. Laroque (1986), "Stability of cycles and expectations", Journal of Economic Theory 40:138-151. Grandmont, J.-M., and G. Laroque (1990), "Stability, expectations, and predetermined variables", in: Champsaur et al. (1990), chap. 3, pp. 71-92. Grandmont, J.-M., and G. Laroque (1991), "Economic dynamics with learning: some instability examples", in: Barnett et al. (1991), chap. 11, pp. 247-273. Griliches, Z., and M. Intriligator (1986), Handbook of Econometrics, vol. 3 (North-Holland, Amsterdam). Guesnerie, R. (1992), "An exploration of the eductive justifications of the rational-expectations hypothesis", American Economic Review 82:1254 1278. Guesnerie, R. (1993), "Theoretical tests of the rational-expectations hypothesis in economic dynamical models", Journal of Economic Dynamics and Control 17:847-864. Guesnerie, R. (1996), "Coordination problems with fixed and flexible wages: the role of the Keynesian multiplier", mimeograph (DELTA, ENS). Guesnerie, R., and M. Woodford (1991), "Stability of cycles with adaptive learning rules", in: Barnett et al. (1991), pp. 111 134. Guesnerie, R., and M. Woodford (1992), "Endogenous fluctuations", in: Laffont (1992), chap. 6, pp. 289-412. Hahn, W (1963), Theory and Application of Liapunov's Direct Method (Prentice-Hall, Englewood Cliffs, N J). Hahn, W. (1967), Stability of Motion (Springier, Berlin). Heinemann, M. (1997a), "Adaptive learning of rational expectations using neural networks", Working paper (University of Hannover). Heinemann, M. (1997b), "Convergence of adaptive learning and expectational stability: the case of multiple rational expectations equilibria", Working paper (University of Hannover). Heymann, D., and P. Sanguinetti (1997), "Business cycles from misperceived trends", Working paper (University of Buenos Aires). Hildenbrand, W., and H. Sonnenschein, eds (1991), Handbook of Mathematical Economics, vol. IV (North-Holland, Amsterdam). Hommes, C.H., and G. Sorger (1998), "Consistent expectations equilibria", Macroeconomic Dynamics 2:287-321. Honkapohja, S. (1993), "Adaptive learning and bounded rationality: an introduction to basic concepts", European Economic Review 37:587-594. Honkapohja, S. (1994), "Expectations driven nonlinear business cycles: comments", in: Bergstr6m and Vredin (1994), chap. 19, pp. 256~62. Honkapohja, S. (1996), "Bounded rationality in macroeconomics: a review essay", Journal of Monetary Economics 35:509 518. Honkapohja, S., and M. Ingberg, eds (1993), Macroeconomie Modeling and Policy Implications (NorthHolland, Amsterdam). Howitt, R (1992), "Interest rate control and nonconvergence to rational expectations", Journal of Political Economy 100:776-800. Howitt, E, and R.R McAfee (1992), "Animal spirits", American Economic Review 82:493-507. Jun, B., and X. Vives (1996), "Learning and convergence to a full-information expectations are not equivalent", Review of Economic Studies 63:653-674. Kandori, M., G.J. Mailath and R. Rob (1993), "Learning, mutation, and long run equilibria in games", Econometrica 61:29-56.

Ch. 7: Learning Dynamics

539

Kiefer, J., and J. Wolfowitz (1952), "Stochastic estimation of the modulus of a regression function", Annals of Mathematical Statistics 23:462~466. Kirman, A.E (1983), "On mistaken beliefs and resultant equilibria", in: Frydman and Phelps (1983), chap. 8, pp. 147-166. Kirman, A.P. (1995), "Learning in oligopoly: theory, simulation, and experimental evidence", in: Kirman and Salmon (1995), chap. 5, pp. 127-178. Kirman, A.E, and R Salmon, eds (1995), Learning and Rationality in Economies (Basil Blackwell, Oxford). Kiyotaki, N., and R. Wright (1989), "On money as a medium of exchange", Journal of Political Economy 97:922954. Kreps, D.M., and K. Wallis, eds (1997), Advances in Economics and Econometrics: Theory and Applications, vol. I (Cambridge University Press, Cambridge). Kuan, C.-M., and H. White (1994), "Adaptive learning with nonlinear dynamics driven by dependent processes", Econometrica 62:1087 1114. Kurz, M. (1989), "Bounded ability of agents to learn the equilibrium price process of a complex economy", Technical report 540 (IMSSS, Stanford University). Kurz, M. (1994a), "Asset prices with rational beliefs", Working paper (Stanford University). Kurz, M. (1994b), "On the structure and diversity of rational beliefs", Economic Theory 4:1-24. Kurz, M., ed. (1997), Endogenous Economic Fluctuations. Studies in the Theory of Rational Beliefs (Springer, Berlin). Kushner, H., and D. Clark (1978), Stochastic Approximation Methods for Constrained and Unconstrained Systems (Springer, Berlin). Kushner, H.J., and G.G. Yin (1997), Stochastic Approximation Algorithms and Applications (Springer, Berlin). Laffont, J.-J., ed. (1992), Advances in Economic Theory: Sixth World Congress, vol. 2 (Cambridge University Press, Cambridge). Lettau, M. (1997), "Explaining the facts with adaptive agents: the case of mutual funds flows", Journal of Economic Dynamics and Control 21:111 21147. Lettau, M., and H. Uhlig (1999), "Rules of thumb and dynamic programming", American Economic Review 89:148-174. Lettau, M., and T. Van Zandt (1995), "Robustness of adaptive expectations as an equilibrium selection device", Working paper (Northwestern University). Ljung, L. (1977), "Analysis of recursive stochastic algorithms", IEEE Transactions on Automatic Control 22:551-575. Ljung, L., and T. Srderstrrm (1983), Theory and Practice of Recursive Identification (MIT Press, Cambridge, MA). Ljung, L., G. Pflug and H. Walk (1992), Stochastic Approximation and Optimization of Random Systems (Birkhauser, Basel). Lucas Jr, R.E. (1973), "Some international evidence on output-inflation tradeoffs", American Economic Review 63:326-334. Lueas Jr, R.E. (1978), "Asset prices in an exchange economy", Eeonometrica 46:1429-1445. Lucas Jr, R.E. (1986), "Adaptive behavior and economic theory", Journal of Business 59(Suppl.): $401-$426. Marcet, A. (1994), "Simulation analysis of dynamic stochastic models: applications to theory and estimation", in: Sims (1994) 81-118. Marcet, A., and D.A. Marshall (1992), "Convergence of approximate model solutions to rational expectations equilibria using the method of parameterized expectations", Working Paper WP73 (Northwestern University). Marcet, A., and J.R Nicolini (1998), "Recurrent hyperinflations and learning", Working Paper 1875

(CEPR).

540

G. W. Eoans and S. Honkapohja

Marcet, A., and T.J. Sargent (1988), "The fate of systems with 'adaptive' expectations", AEA Papers and Proceedings, 78(2): 168-172. Marcet, A., and T.J. Sargent (1989a), "Convergence of least squares learning and the dynamic of hyperinflation", in: Barnett et al. (1989), pp. 119-137. Marcet, A., and T.J. Sargent (1989b), "Convergence of least-squares learning in environments with hidden state variables and private information", Journal of Political Economy 97:1306-1322. Marcet, A., and T.J. Sargent (1989e), "Convergence of least-squares learning mechanisms in selfreferential linear stochastic models", Journal of Economic Theory 48:337068. Marcet, A., and T.J. Sargent (1995), "Speed of convergence of recursive least squares: learning with autoregressive moving-average perceptions", in: Kirman and Salmon (1995), chap. 6, pp. 179-215. Margaritis, D. (1987), "Strong convergence of least squares learning to rational expectations", Economics Letters 23:157 161. Marimon, R. (1997), "Learning from learning in economics", in: Kreps and Wallis (1997), chap. 9, pp. 278-315. Marimon, R., and E.R. McGrattan (1995), "On adaptive learning in strategic games", in: Kirman and Salmon (1995), chap. 3, pp. 63 101. Marimon, R., and S. Sunder (1993), "Indeterminacy of equilibria in a hyperinflationary world: experimental evidence", Econometriea 61:1073-1107. Marimon, R., and S. Sunder (1994), "Expectations and learning under alternative monetary regimes: an experimental approach", Economic Theory 4:131 - 162. Marimon, R., E.R. McGrattan and T. Sargent (1989), "Money as medium of exchange with artificially intelligent agents", Journal of Economic Dynamics and Control 14:329 373. Marimon, R., S.E. Spear and S. Sunder (1993), "Expectationally driven market volatility: an experimental study", Journal of Economic Theory 61:74-103. Maussner, A. (1997), "Learning to believe in nonrational expectations that support Pareto-superior outcomes", Journal of Economics 65:235-256. McCallum, B.T. (1983), "On nonuniqueness in linear rational expectations models: an attempt at perspective", The Journal of Monetary Economics 11:134-168. McCallum, B.T. (1997), "Alternative criteria for identifying bubble-free solutions in rational expectations models", Working paper (Carnegie Mellon and NBER). McLennan, A. (1984), "Price dispersion and incomplete learning in the long run", Journal of Economic Dynamics and Control 7:331-347. Milgrom, P., and J. Roberts (1990), "Rationalizability, learning, and equilibrium in games with strategic complementarities", Econometrica 58:1255-1277. Milgrom, P., and J. Roberts (1991), "Adaptive and sophisticated learning in normal form games", Games and Economic Behavior 3:82 100. Mitra, K. (1997), "Least squares prediction and nonlinear dynamics under non-stationarity", mimeograph (University of Helsinki). Moore, B.J. (1993), "Least-squares learning and the stability of equilibria with externalities", Review of Economic Studies 60:197-208. Moore, B.J., and H. Schaller (1996), "Learning, regime switches, and equilibrium asset pricing dynamics", Journal of Economic Dynamics and Control 20:979-1006. Moore, B.J., and H. Schaller (1997), "Persistent and transitory shocks, learning and investment dynamics", Working paper (Carleton University). Moreno, D., and M. Walker (1994), "Two problems in applying Ljung's 'projection algorithms' to the analysis of decentralized learning", Journal of Economic Theory 62:42(~427. Muth, J.E (1961), "Rational expectations and the theory of price movements", Econometrica 29:315-335. Nyarko, Y. (1991), "Learning in mis-specified models and the possibility of cycles", Journal of Economic Theory 55:416-427.

Ch. 7: Learning Dynamics

541

Nyarko, Y. (1997), "Convergence in economic models with Bayesian hierarchies of beliefs", Journal of Economic Theory 74:266-296. Paekal6n, M. (1997), "Adaptive learning of rational expectations: a neural network approach", mimeograph (University of Helsinki). Pesaran, H. (1981), "Identification of rational expectations models", Journal of Econometrics 16:375-398. Robbins, H., and S. Monro (1951), "A stochastic approximation method", Annals of Mathematical Statistics 22:400-407. Salge, M. (1997), Rational Bubbles. Theoretical Basis, Economic Relevance and Empirical Evidence with Special Emphasis on the German Stock Market (Springer, Berlin). Salmon, M. (1995), "Bounded rationality and learning; procedural learning", in: Kirman and Salmon (1995), chap. 8, pp. 236~75. Sargent, T.J. (1987), Macroeeonomic Theory, 2nd edition (Academic Press, New York). Sargent, T.J. (1991), "Equilibrium with signal extraction from endogenous variables", Journal of Economic Dynamics and Control 15:245-273. Sargent, T.J. (1993), Bounded Rationality in Macroeconomics (Oxford University Press, Oxford). Sargent, T.J. (1999), The Conquest of American Inflation (Princeton University Press, Princeton, NJ). Sargent, T.J., and N. Wallace (1975), "'Rational expectations', the optimal monetary instrument and the optimal money supply rule", Journal of Political Economy 83:241~54. Sch6nhofer, M. (1996), "Chaotic learning equilibria", Discussion Paper 317 (University of Bielefeld). Shell, K. (1977), "Monnaie et allocation intertemporelle", Working paper (CNRS Seminaire de E. Malinvaud, Paris). Sims, C.A., ed. (1994), Advances in Econometrics, Sixth World Congress, vol. 2 (Cambridge University Press, Cambridge). Soerensen, J.P. (1996), "An economy with heterogeneous agents", Working paper (University of Edinburgh). Spear, S.E. (1989), "Learning rational expectations under computability constraints", Econometrica 57:889-910. Taylor, J.B. (1975), "Monetary policy during a transition to rational expectations", Journal of Political Economy 83:1009-1021. Taylor, J.B. (1977), "Conditions for unique solutions in stochastic macroeconomic models with rational expectations", Econometrica 45:1377-1386. Taylor, LB. (1980), "Aggregate dynamics and staggered contracts", Journal of Political Economy 88: 1~3. Taylor, J.B. (1986), "New approaches to stabilization policy in stochastic models of macroeconomic fluctuations", in: Griliches and Intriligator (1986), chap. 34, pp. 1997-2055. Tillmaun, G. (1983), "Stability in a simple pure consumption loan model", Journal of Economic Theory 30:315-329. Timmermann, A.G. (1993), "How learning in financial markets generates excess volatility and predictability in stock prices", Quarterly Journal of Economics 108:1135-1145. Timmermann, A.G. (1994), "Can agents learn to form rational expectations? Some results on convergence and stability of learning in the UK stock market", Economic Journal 104:777-797. Timmermann, A.G. (1995), "Volatility clustering and mean reversion of stock returns in an asset pricing model with incomplete learning", Working paper (University of California, San Diego). Timmermann, A.G. (1996), "Excessive volatility and predictability of stock prices in autoregressive dividend models with learning", Review of Economic Studies 63:523 557. Townsend, R.M. (1978), "Market anticipations, rational expectations, and Bayesian analysis", International Economic Review 19:481-494. Townsend, R.M. (1983), "Forecasting the forecast of others", Journal of Political Economy 91:546-588. Turnovsky, S. (1969), "A Bayesian approach to the theory of expectations", Journal of Economic Theory 1:220-227.

542

G. W. Evans and S. Honkapohja

Vives, X. (1993), "How fast do rational agents learn?", Review of Economic Studies 60:329-347. White, H. (1992), Artificial Neural Networks: Approximation and Learning Theory (Basil Blackwell, Oxford). Whiteman, C. (1983 ), Linear Rational Expectations Models. University of Minnesota Press (Minneapolis, MN). Woodford, M. (1990), "Learning to believe in sunspots", Econometriea 58:277 307. Zenner, M. (1996), Learning to Become Rational. The Case of Self-Referential Autoregressive and Non-Stationary Models (Springer, Berlin).