Alternative algorithms for the macRae OLCV strategy

Journal of Economic Dynamics and Control 7 (1984) 21-37. North-Holland

ALTERNATIVE ALGORITHMS FOR THE MacRAE OLCV STRATEGY * Alfred L. NORMAN University of Texas at Austin, TX 78712, USA Received October 1982, final version received September 1983 Using dynamic programming MacRae obtained a linear feedback law for her open loop constrained variance (OLCV) strategy. To compute the control law requires solving a two point boundary value problem. The dynamic programming formulation does not admit gradient related algorithms. This paper presents an alternative augmented Lagrangian formulation which can be solved as a deterministic constrained optimization problem. The new approach is superior because it admits gradient related algorithms plus all the algorithms which could be employed using the original approach. Three examples demonstrate the superiority of the new approach.

I. Introduction

Aoki (1967) demonstrated that the general case of the Bayesian formulation of the linear quadratic estimation and control problem cannot be solved analytically. Subsequently, many investigators have proposed approximations in an effort to reduce the computational problems or to obtain analytic results. For her open loop constrainted variance (OLCV) strategy, MacRae (1972, 1975) simplifies the statistical assumptions of the unknown coefficients. Using dynamic programming MacRae obtains a linear feedback law. Computational difficulties arise in determining the gains for the OLCV linear feedback law as their computation requires solving a two point boundary problem, x It is the purpose of this paper to present an augmented Lagrangian formulation of the OLCV strategy which can be solving using a wide variety of software for deterministic optimization. The MacRae OLCV strategy is a stochastic control problem. To employ software for deterministic optimization, the stochastic control problem must be formulated as an equivalent deterministic optimization problem. By partitioning the stochastic variables into their stochastic and deterministic components, *This material is based upon work supported by the National Science Foundation under Grant SOC 76-11187. tTo obtain the recursive relationships in the dynamic programming formulation, the reduced gradient is set to zero. The types of algorithms which can be applied to solve the two point boundary problem has been greatly restricted. 0165-1889/84/$3.00 © 1984, Elsevier Science Publishers B.V. (North-Holland)

A. L Norman, AIternativealgorithmsfor the MacRae OLC V strategy

22

an equivalent deterministic optimization problem with side conditions is obtained in section 2. 2 The new formulation is shown to be equivalent to the original MacRae formulation. In section 3 algorithms to solve the necessary conditions are considered. The alternative formulation can be solved by code for gradient related algorithms as well as any algorithm for the original formulation. One approach to the original formulation is successive substitution which is a block Gauss-Seidel algorithm. Computation results are shown for three sample problems. The author's code for successive substitution and two variable metric and one conjugate gradient codes from the 1973 HarweU library, which were kindly supplied by D.M. Himmelblau, were considered. The results indicate the effectiveness of the new formulation.

2. An alternative formulation of the MacRae OLCV strategy The MacRae OLCV strategy is

(I)

rain J, u N

where N

J=

E E ( ( I + ~ ) - " ' t:Xk~kXk '" ' + Uktk)[W* ' + :'u a' R k u * + XkS, },

(2)

k~1

subject to

x k =A,_xXk_: + Bk_lU * + Ck_lZ * + e,, ----Dk_lW k + ek , F , 1= F~'_ll + I2-x ® E( wkw~ },

k = 1,2 . . . . . N,

(3)

k = 1,2 . . . . . N,

(4)

given initial conditions A, B, C, F0-1 and x0, where Qk, R,, Sk, t, = time varying coefficients of the quadratic loss function; Q, and R k are positive semidefinite; N = time horizon; = discount rate; x, = (n × 1) vector of state variables; u, = (m × 1) vector of policy or control variables; u N'

t . = ( u £ u ~ I . . . . . UN),

z~. W,

= ( p - n - m) vector of exogenous variables; Xp t = ( k-l, Uk, Z,), and W, is a perfectly observed ( p × 1) vector;

w N'

= ( w ; , w~ . . . . .

p

p

w~,);

2This approach is an extension of Chow (1975) wherein he solved the linear quadratic control problem with known coefficientsby Lagrangian multipliers. Using Norman (1981) the results herein could be extended to structural models.

23

A. L Norman, A Iternative algorithms for the MacRae OLC V strategy

D,

Ek

= (Ak, Bk, Ck) is a matrix of intertemporal independent random variables with mean (A, B, C) and where /'k given by (4) represents the covariance matrix of D k with elements arranged by rows; the dimensions of A, B, C are n × n, n × m, n × (p n - m), respectively; = independent identically distributed vector of disturbances with E(ek}----0 and E(eke~} =~2.

The derivations herein will utilize the notation, identities and theorems for matrix derivatives developed by MacRae (1974). The symbols which will be introduced are Kroneker product, ®, pack operator, P ( ) , star product, *, and permuted identity matrix, 1(,.,,). To make the derivations self-contained, a brief summary is presented in appendix A. Additional algorithms to solve the stochastic control problem are obtained by converting the original problem into an equivalent deterministic optimization problem. First a linear feedback law on observing the previous state is assumed, (5)

Uk = G k X k - 1 + gk"

Next the variables are partitioned into their deterministic (-) and stochastic (-) components; for example, x k = ~k + 5Ck"The system dynamics become Uk = Gk'~k-1 + gk,

(6)

fq. = GkSCk_l,

(7)

= (;I +

+

(8)

+ Cz,,

5ck = ( Ak_ I + Bk_xGk)SCk_ I + .4k_12k_l

+ B,_x(Gk~,_lgk) + (~k_lZk + ek.

(9)

Given the statistical assumptions on the coefficients, E{~k} = 0 .

(10)

Now using (6), (7), and (10), J can be evaluated as N

J = Y'. (1 + 8) -k [½2'kQk,2k + ½(Gk~k_ 1 + gk)'Rl,(Gk~k_l + gk) k=l

+ Y , ' k S k + ( G k ~ k _ l + g k ) ' t ~ + !2Qk* ~k + ½ G ~ R k G ~ k - 1 ] ,

where @k = E(:2k~ }.

(11)

24

A.L. Norman. A Iternatit,e algorithms for the MacRae OLC V strate~"

Note this derivation differs slightly from the usual in that the dynamics for x k, that is ~k and ~k are constrained to the objective function rather than substituted into the objective function. To obtain the equivalent deterministic formulation requires an expression for ~k- The evaluation of ~ requires the evaluation of terms such as

'Pl = E{ Ak-lxk-lxk-lGkBk-1 }. Using identity 2a of appendix A, xr,, =

~;,_,O,:

* E( P(,4~_,)e'(B~_,)}.

To express E( '/'1 ) in terms of F~,s requires the following identity, which is established in appendix A for a ( p × q) matrix a,

P( a) = l~q.,~P( a').

(12)

Defining *sYk_ 1 as E{ P(Ak_I)P'(Bk_I) }, one obtains

ABet-1 = Ic...f~-s~I~...,~. Using this approach, ~t is obtained as

= (.~ + BG~)~_ ,(,i +Bo~)' + (~_1 + ~,- 1~- 1)*"%-,

+ ( a~_la;, +( a~,~_, + g~)( G,.~ + g~)').~ r~_, +( ~k-,a'k + ~k-,( Ok-~k-, + gk)')*AsFk-, + (a~,~_, + ( a ~ _ , + g~)~-1) *"%-1 + ~k-lz~ *'4cFk -1

-' * C A F k - I + gkXk-1

+zk(ak~k-~ + g)'*csrk_~ +(a~k_, + g~)z'k ,"cr~_l +z~z~ , c c r , _ 1 + fL

(13)

Thus, the deterministic optimization problem becomes minimize (11) subject to (8), (13) and (4), given the initial conditions. Note (4) is the original covariance updating relationship which is required to compute (13). Once this

A. 1. Norman. A Iternati~,e algorithms for the MacRae OLC V strategy

25

problem has been solved the controls can be obtained from auxiliary equation, (5). This is a matrix form of the classical maxima and minima problem with side conditions. The number of unknown variables associated with .~:~, q~, G~, g~, F~ is nN, n2N, nmN, nN, n2p2N, respectively, which when summed is (n2( p2 + 1) + n(m + 2))N. For computation purposes the symmetry of @~ and F~ can be utilized. This problem can be solved using matrix Lagrangian multiplier methods. To augment (8), (13) and (4) appropriately dimensioned matrix Lagrangian multipliers M k, K k and A~ can be introduced; however, as was the case in MacRae (1976), it is desirable to utilize undiscounted multipliers M k, K, and ~,, where M k = (1 + 8)*.,~,,

K, = (1 + 8 ) * K , ,

h, = (1 + 6)kAy.

(14)

Assuming the Jacobian of the constraints is of rank ( N Z ( p 2 + 1)+ n)N, the constrained optimization problem becomes

minL([.~,_l,G,,g,,K,,M,,F,_l,~,,h,]k=l,N

),

(15)

where N

L=J+

E

+ 8)-'

k-I

N

+ E ½(1 + 8)-kM** [F,-l - Fk-J, - ~-1 @ E{ ww'}] k~l

N

+ E k=l

The (2n2(p2+ 1 ) + n ( m + 3))N necessary conditions to minimize L constitute a two point boundary value problem. The proposed method of solution is to solve the necessary conditions to obtain backward recursive relationships for K k, M k and h k -- ~, - Kkxk, forward relationships for ~,, Fk and xk, and to use these relationships to obtain the generalized reduced gradient for Gk and gk. In obtaining these relationships their equivalence to the earlier work of MacRae will be demonstrated. In this paper h k is used where MacRae uses g~. To obtain MacRae's results the following definitions are required:

H k = ~ , K k ~ a .-v".*r s s"k-1 _ f g - I * m~S + R k,

(16)

FI" = ~ , K k y + K~F~A1 _ ~ - 1 . M~A,

(17)

.f, = [ B ' K k C + K k * F~. c, - J~-' * M : c] z k + B'h k + t~.

(18)

26

A. L Norman, AIternative algorithms for the MacRae OLC V strategy

Using these expressions the first-order necessary conditions can be simplified to .

OL --=O=[Hfik+Fk](~k_d~_l+~k_i)+[Hdk+fk]~', OGk

(19)

.

~L --=

(20)

o = [,qdk + F~]Xk_, + [/4~g~ + A ] ,

Ogk

.

OL O~-----~= O = -- Kk + Qk

for

k = N,

"h --..1_ Kk+/;-(~-' , AA , =-/~+Q~+[A~+,A

M A~.+,) A

(21)

-- FlP.+ IHk+llFk + I]i/(I "i-8), .

OL

at,zX--=0=-ha+s~

for k = N ,

= - h k + s x. + [ ( A +Ahk+ 1 .

Kk+iC+(K~t.+,*F~c)-(~-'* MAC))zk+l

Fk + IH k-1+ / / k + , ] / (

1 +3)

for k < N,

(22)

OL

--~0~ 01"y x

= M u if

k=N,

= Mk-[Mk+l + rk(~+, * E( w~+,w;+, }) r~] /(1+6) 6.

7.

8.

OL

if k < N ,

(23)

0~----~-= 0 gives (8),

(24)

OL OK---~k=

(25)

OL

0 gives (13),

0--'~k= 0 gives (4).

(26)

A.L. Norman. AIternative algorithmsfor the MacRae OLC V strategy

27

Noting Xk and ~k are arbitrary in (19) and (20) gives G k = - H k 1Fk

and

gk = - H k lfk.

Thus the alternatives formulation is equivalent to MacRae's (1975) original stochastic dynamic programming formulation. The (n2(p2 + 1) + n ( m + 2))N necessary conditions are (4), (8), (13), (19), (20), (21), (22) and (23).

3. Algorithms, code and examples To present a generalized reduced gradient algorithm to minimize J, G k is defined as G

.....

The reduced gradient of J with respect to G k, which will be symbolized by ~TJ(G k), is defined by stacking OL/OGk, O L / O g k into a vector corresponding to (46). The generalized reduced gradient algorithm is: 1.

Obtain an estimate for G k. This might be done by setting Fk = F0 for k = 1 , 2 ..... N.

2.

C o m p u t e ~ k , ~ k , F k f o r k = l , 2 . . . . . N. Compute Mk, K k, h k for k = N, N - 1..... 1. Compute XTJ(GU).

3. 4. 5. 6. 7. 8.

Determine search direction d. Determine a scalar step size a*. This is often done by approximate minimization of the function r ( a) = J( G N + ad ). Replace G N by G N + a*d. Repeat steps 2-7 until convergence is obtained.

There are a large number of alternative procedures for steps 5 and 6. Step 5 might be obtained from any gradient related algorithm such as a variable metric or conjugate gradient algorithm. Step 6 might be obtained from a quadratic or cubic polynomial fit or a Fibinacci search. An alternative procedure of obtaining V J ( G k) would be the one proposed by Fair (1974) in which case step 3 is eliminated. In the original formulation of MacRae (1975) the reduced gradient V J ( G k) is not available. The most obvious algorithm is a successive substitution or iterative linear quadratic algorithm obtained by replacing steps 3-8 with: 3'. 4'.

Compute Mk, Kk, hk, G k, gk for k = N , N - 1 . . . . . 1. Repeat steps 2 and 3 until convergence is obtained.

28

A.L. Norman, AIternative algorithms for the MacRae OLC V strategy

A relaxation factor can be inserted as step 3a as: 3a'. G ( i ) N = G ( i - 1) N + a * ( G ( i ) s - G ( i - 1)N), where the parentheses refer to the iteration, a* can be obtained by approximate minimization of the function r( a) = J( G( i - 1)N+ ct*( G( i ) N - G( i - 1)U)). Note this algorithm is obtainable from the necessary conditions of the alternative formulation. Also this algorithm is a block form of the Gauss-Seidel algorithm to solve the necessary conditions. At each step for each time period the matrix being computed is based on the solved values for all previous matrices. With the relaxation factor the algorithm is a block form of the SOR algorithm. Currently there is a growing number of good codes which could be used for steps 5 and 6. For this study the following variable metric and conjugate gradient algorithms on the 1973 Harwell Library were selected for study: 1. 2. 3.

Powell's (1970) variable metric, VA06ADF; Fletcher's (1972) variable metric, VA09ADF; Fletcher's (1964) conjugate gradient, VA08ADF.

These three codes will be compared with the author's code for the successive substitution algorithm. To evaluate alternative codes for the MacRae OLCV strategy, three sample problems were examined. The first problem examined is a problem previously examined by MacRae (1972). For this problem the successive substitution code is much more efficient than the three codes from the Harwell Library. Two other problems were created to determine whether the performance of the successive substitution code is problem specific. The computation results obtained in this section were obtained on the University of Texas Control Data Cyber 170/750. The various codes were compiled using the MNF 5.4 compiler. The computations were made in interactive mode. The initial guess for the controls for each case was obtained by assuming F k = F o for all k. The problem previously studied by MacRae (1972) is as follows: 8

min E k=l

1

2

I

2

E(~-Xk +-~Uk },

29

A.L. Norman, Alternatit,e algorithms for the MacRae OLCV strategr

subject to x k = 0.7xk_ l + bk_tu k + 3.5 + ek, F~7' -= FIT', + E( u] }/~22, eg = iid(0, 0.2), b0 = - 0.5, /'0=0.5, xo =

0.0.

The computational results for problem 1 are as follows: Problem 1 results

Successive substitution with VA02 linear search

Powell VM VA06

Iter

Func

Obj

Iter

Func

1 19

1 19

132.309 127.492

1 20 40 60 80 86

1 20 40 60 80 86

Fletcher VM VA09

Obj 132.309 127.569 127.492 127.492 127.492 127.492

Fletcher CG VA08

Iter

Func

Obj

Iter

Func

Obj

1 20 37

1 25 42

132.309 127.493 127.492

1 20 40 52

1 4 79 100

132.309 127.500 127.492 127.492

Notes: .

2.

These results agree with the previous results of MacRae. i t e r - n u m b e r of iterations; f u n c - - t h e number of times the objective function subject to the constraints was evaluated; o b j - v a l u e of the constrainted objective function.

JEDC--,8

A. L Norman. AIternative algorithms for the MacRae OLC V strategy

30

3.

The convergence criterion for the VA06 code was the sum of the squares of the generalized reduced gradients < 10 -4. The convergence criterion for the other three was the change in each control between successive periods < 10 -.4. VA08 stopped with 100 func calls.

As obj is 127.4967 after five iterations with the SS code, it is clear for this problem the SS code dominates the three codes from the Harwell library. For problem 1 the first guess to the control gains is a close approximation to the optimal sequence of controls gains. In constructing problem 2 and 3, an effort was made to ensure that the initial guess would mat be a good guess. Problem 2 is as follows: 8

min Y', k=l

subject to xk = a ~ - ~ x k - 1 + bk-~uk + ck-1 + ek,

/~k'-1 = Yk._ll + ~2_

iE xk-x I //k [Xk-l/dka], 1

where x 0 = 0.0,

a 0 = 0.1,

b0 = 0.1,

c o = 3.5,

to=

0.5 0.0 0.0

0.0 0.5 0.0

0.0, 0.0 0.5

ek -- lid(O, 0.2),

a l, a 2 . . . . . a s = 10.0. Now, before examining the computational results, some explanation would be helpful. For this problem the track, a,, equals 0 until the final period when a s = 10. This means there is a strong incentive to reduce the covariance matrix Ft for the final decision. As all of the coefficients are random, the sequence of controls which optimally reduces the covariance matrix is not obvious. The computational results are as follows:

A. L Norman, A lternatiPe algorithms for the MacRae OLC V strategy

31

Problem 2 results

Successive substitution with VA02 linear search Iter 1 9

Func 1 9

Powell VM VA06

Obj

Iter

Func

Obj

72.5102 28.7396

1 20 40 60 80 100 120 140 160 163

1 20 40 60 80 100 120 140 160 163

72.5102 49.2864 33.9783 28.0261 25.8961 23.9474 22.9377 22.0475 22.0424 22.0424

Fletcher VM VA09 Iter 1 20 40 60 80 100 120

Fletcher CG VA08

Func

Obj

Iter

Func

Obj

1 20 40 60 80 100 120

72.5102 30.6258 24.0842 22.7501 22.1831 22.0542 22.0424

1 20 40 60 80 100 120 140 146

1 43 83 133 174 213 253 290 303

72.5102 32.7668 26.2349 24.6236 24.1111 24.0826 24.0821 24.0821 24.0821

Notes:

1. Convergence criterion same as before. 2. The linear search VA02 is employed with the successive substitution code as defined in step 3a'. For this problem the ranking of the codes is substantially different from problem 1. The successive substitution algorithm without a linear search diverges. With a cubic linear search using first derivatives, VA02, the code stops short of the minima obtained by VA06 and VA09. This problem indicates there exist problems for which the Harwell variable metric code is superior at minimizing the objective function than the code based on the original algorithm.

32

A.L. Norman, AIternative algorithms for the MacRae OLC V strategl'

Problem 3 is the same as problem 2 except a s = 100.0 instead of 10.0. This problem places even greater emphasis on learning than problem 2. The computational results are as follows: Problem 3 results

Successive substitution with VA02 linear search

Powell VM VA06

Iter

Func

Obj

Iter

Func

Obj

1 10

1 10

4513.22 4427.19

1 50 100 150 200 250 300 350 400 450 500

1 50 100 150 200 250 300 350 400 450 500

4513.22 207.57 160.20 150.39 148.55 147.00 143.09 141.68 140.30 134.42 131.59

Fletcher VM VA09

Fletcher CG VA08

Iter

Func

Obj

Iter

Func

Obj

1 3

1 10

4513.22 720.06

1 8

1 43

4513.22 582.95

Notes:

1. Convergence criterion are the same as before. 2. VA06 stops after 500 function calls. The largest generalized reduced gradient is 0.1 For this problem, only the Power VM VA06 code appears to be converging to the minima. The point at which VA09 and VA08 stop the generalized reduced gradient have at least one component whose absolute value is greater than 20. For the iteration at which VA09 and VA08 respectively stop, the search direction does not lead to a reduction of the constrained objective function. It should be noted both VA09 and VA08 are far superior, to the SS code. To interpret the empirical results an examination of Ortega and Rheinbolt is useful. To use Ortega and Rheinbolt (1970), (4), (9) and (14) are substituted

A. L Norman, AIternative algorithms for the MacRae OLC V strategl'

33

into J so that J is a composite function of G~. The global convergence proof of the SOR algorithm 14.6.7 assumes J(G N) is uniformly convex. Given (4), J(G N) is not convex. While weaker conditions may be discovered for global convergence of the SOR algorithm, the empirical results indicate they would not apply to the MacRae OLCV strategy. From theorem 14.3.4 it can be seen that one can expect global convergence from a gradient related with a Curry-Altman scalar search. The key to applying this theorem is the observation that the level sets for the MacRae OLCV strategy are contained within the level sets for the certainty equivalence, CE, strategy. The advantage of the new formulation is that the computational problem can be addressed from a much wider set of algorithms and codes. The empirical results suggest that when learning, that is covariance reduction, is important the new algorithms will perform much better than the old approach.

Appendix A: Summary of matrix operations

1. Definitions a.

The symbol ® represents the well known Kronecker product [see Theil (1971, p. 303)].

b.

The pack operator P forms columns of a (m x n) matrix A into a (mn x 1) vector P( A ) = [a u, a2x ..... alz, a2z ..... a,,m].

c.

The star product * of A [a (m x n) matrix] and B [ a m p x np) matrix], C = A * B, is defined as C = F.iyaijB U where aii is the 0"th element of A and Bu is the 0th submatrix of B.

d.

The permutation matrix Iu,,.,, ~ is defined as a square mn dimensional matrix with I¢,,,,,),,=1

for

k=(j-1)m+i, r = ( i - 1)n + j , i = 1 , 2 . . . . . m,

=0 e.

j=l,2

. . . . . n,

otherwise.

Given two matrices A and B, then F An and AB£ are defined as

£An=E(P(A')P'(B')}

and

AnF=E{P(A)P'(B)}.

Note: In the paper Ft° ° is abbreviated to Ft.

34

A.L. Norman, Alternative algorithmsfor the MacRae OLCV strategy

2. Identities a,

The MacRae (1974) relationship for the matrix product of conformable matrices ABC is

ABC = B * P( A)P'( C'). b. Given an(m × n) matrix A, a (p x q) matrix B and a (pro x qn) matrix C, the notation above can be utilized to show that .~*(B* C) = (A* cT)* B, where * is the star operator defined in MacRae (1974), and C r = Ic.,.p)CI~o,,,),

Ij

l,k

~ ij

~lk

= E Ea~o,,,F~,-,,,,+I.(j-~).+~. 1.k /j

(A*Cr)=~_,al~C r, lk B*(A*

r C T ) - ~ E E bi talkC(1-1)n+j,(k-1)m+i" q lk

Using the index functions above, it can be shown that C(i-l)m+l.(j-l)n+k ~ cT (l-l)p+i,(k-l)q+j" Hence,

A*(B*C)= (A*CT)* B. C.

From the definition of the star product we obtain

( Kt * r A ° ] w , -- ( K,*

.4C +( K, ~ Fi4-])u, +( Kt* Ft-1)z, •

d. From eq. (A.7) of MacRae (1974) we obtain

M, ,(~-~, z{ w,w,,}) = z{ w;(a, M,)w,},

A.L. Norman, AIternative algorithms for the MacRae OLC V strategy

35

3. Matrix differentiation

MacRae (1974) defined matrix differentiation as OY/OX=

Y® O/Ox.

For a scalar Y and an n colunm vector X this leads to the same definition as in Goldberger (1963); for an m column vector Y and an n column vector X the MacRae (1974) definition leads to an mn column vector, whereas the Goldberger (1963) definition leads to an (n × m) matrix. All that is required in this section is the derivation of a scalar with a vector. A useful result, obtainable using MacRae (1974), is that if L = C * A G ' B where C, A, G, B are (r × n), (n × n), (n × r), (r × r) matrices, respectively, then O L / O G = B C ' A . Also, if L = C" * B ' G A , then O L / R G = B C ' A .

Appendix B The derivation of (19)-(23) requires some manipulation. To demonstrate the steps the derivation of (19) will be considered herein. From (15), a L / O G k = O, = [ R k G k + ( K k * Fks__nl)G k + ( K k * F ~ I ) - (I2-1 * MffA).

-(~-~, g~"")Gk]x~_l ~_1 --

~t

-(~-~, g~")g~.-(n-', g~.<)z~ + B%]~_~ + [R~G~ + ~ ' K ~ 6 ~ + (Kk * rL"l) G~ +

+(K~., r L ~ )

-B'K~Y

- ( ~ - ~ • M~") - (~-~ • M~") Gk] ~ . _ , .

(B.1) The derivative is obtained using the following identity. If a, fl, T are (p x q), ( m × n ), ( mp × nq ) matrices, respectively, then a*( fl * "/ ) = (u*Z,p,,,,TI,,,.q,) * ft.

(B.2)

36

A.L. Norman, AIternative algorith#u" for the MacRae OLC V strategy

This identity, which is established in appendix A, is implied by the example of MacRae (1974, p. 343). The identity is utilized to evaluate expressions such as

't"2 = ( Kh *( G,.¢k_~G ~ )* ssFk_l)® O/OGk.

(B.3)

Using (B.2),

'P2 = (( Kx * F:_B, )* Gx.q~k_,G~, ) ) ® O/OG,..

(B.4)

Utilizing section 3 of appendix A, g'z = 2( Kk * rks._sl) Gk~k_ 1.

(B.5)

Secondly, an alternative expression for '/'3 = Mk . ( _ 5 2 - 1 ® E{wkw/)) is required. Utilizing identity d of appendix A, 'P3 can be written as

,t,3 = _ ( 52-1 * M : A ) * ( ~ : ; - ,

+

_(52-1 * M : B ) * ( X ~ - , ( G : k - ,

¢k-1) + gk)' +

Ck-,G;)

_(52-1 * M : C ) * ( . ~ _ : ; )

_ (52-1 * M : ~ ) * ( ( G : ' ~ - I --(52 -1

+

g~)~--, + G:~_,)

* M:")* ((G:~_, + g~)(G:k-1 + g~)' + G:~_,G~)

_(52-1 * M U ) * ( ( o : k _ l + gk)zi _ (52-1 * MCA)*(zk.2~._,) _(52-1

, M['), (z~(o:k_l + g~)')

- ( 5 2 - ' * MCC)*( zkz~.).

(B.6)

To obtain MacRae's expression for Gk: a.

Add BPKkXkX~_ 1 tO the first term on the right-hand side of (B.1) and subtract the same expression from the second term.

b.

Substitute (8) for -~k in the first term and B'KkCzk)ff~._ 1 to the third term.

c.

Define h k --- )~k - Kk-~k and use in the second term.

transfer (B'K~.Bg k +

A.L. Norman. A hernativealgorithmsfor the MacRae OLC V strateg)' d.

37

Combine the first and the third terms,

OL/OG k = [( R,. + B'KkB +( Kk . F ~ . ° , ) - ( ~ - ' , M ~ a ) ) G k + B'KA + ( K k * F B f f , ) - - ( I 2 - ' * M,O'~)] (.2k_ t.2Z_ t + ~ k - 1 ) + [( R,. + B ' K k B + ( K k * F:_ot ) - ( I 2 - ' *

M~.n))gk + t k

+ B'KkCz k + ( K k * r:Cl ) z,. - ( g2-' • M,e. c ) z k + Bh k ] ff'~._l. (B.7) Using (16), (17), and (18),

OL/OG

= [HkGk +

+

+ [Hx.gk + f , ] x ' ,

(B.8)

as (~k_lff~_l + ~ k - 1 ) and x k - t are arbitrary, and assuming H -1 exists,

G~. = - H [ 1 F k ,

gk = -Hi-try.

(B.9)

(B.10)

The derivations of (20)-(23) require a similar amount of work.

References Aoki, M, 1967, Optimization of stochastic systems (Academic Press, New York). Chow, G., 1975, Analysis and control of dynamic economic systems (Wiley, New York). Fair, R.C., 1974, On the solution of optimal control problems as ma~ximizationproblems, Annals of Economic and Social Measurement 3, 125-154. Fletcher, R. and C.M. Reeves, 1964, Function minimization by conjugate gradients, Computer Journal 7, 263-278. Goldberger, A.S., 1974, Econometric theory (Wiley, New York). Powell, M.J.D., 1970, A.E.R.E. report 6469. MacRae, E.C., 1972, Linear decision with experimentation, Annals of Economic and Social Measurement 4, 437-447. MacRae, E.C., 1974, Matrix derivatives with an application to an adaptive linear decision problem, Journal of Statistics, 337-346. MacRae, E.C., 1975, An adaptive learning rule for multiperiod decision problems, Econometrica 43, 5-6, 893-906. Norman, A.U, 1981, On the control of structural models, Journal of Econometrics 15, 13-24. Ortega, J. and W. Rheinboldt, 1970, Iterative solution of nonlinear equations in several variables (Academic Press, New York). Theil, H., 1971, Principles of econometrics (Wiley/Hamilton, New York).