- Email: [email protected]
Generalizations and applications of a class of dynamic programming problems
European Journal of Operational Research 31 (1987) 127-131 North-Holland
127
Generalizations and applications . of a class of dynam ic program m ing problems Hans Ulrich BUHL Institut fir Wirtschaftstheorie und Operations Research der UniversittitKarlsruhe, am SchloB, Bau IV, 7500 Karlsruhe I, Germany, Fed. Rep.
KaiserstraDe 12, Kollegium
Abstract: In Buhl and Siedersleben(1984) it has been shown, that a class of Dynamic Programming
Problems satisfying certain independenceand reachability conditions has interesting properties and rather simple solutions. This class is generalized here in such a way to be applicable to problems frequently encountered in economic models employing the concept of utility functions. Although the properties used may seem restrictive, they are satisfied for a number of relevant problems. Keywords: Dynamic programming, inventory, investment, growth
1. A class of dynamic programming problems
A general formulation of the dynamic programming problem with one-state-dependentfunctions is given by Maximize ’
i
&h-l,
u,>s
r-1
subject to m
-%=f,b,-l~ U,EG,, xg E gJ
4, X,EZ,,
t=l.,...,T,
1 (specified).
For t = 1,. . . , T, the state space E, and the control space 52, are contained in some supersets ,” and 0. I3+ =ny,,z, is called the global state space and s2’= I-I TBIL?, the global control space. The functions f, and g, are defined on 3 X 52and take on values in ,” and R, respectively. The product sets ZT and OT are defined in the usual way. In Buhl and Siedersleben[6] we have shown that if there exist functions G, and H, such that (IA) and
given
g,(x,-,,
u,>~G,(x,-,)+H,(f,(x,-,,
u,>),
t=l,...,T,
certain reachability conditions precisely specified there 2, the solution(s) of problem (P) can be
Received April 1985; revised June 1986 The author is indebted to anonymus referees for very helpful suggestions. ’ Throughout the paper, ah maxima are assumed to exist; by convention, maxB= - co. * It suffices that some sequenceof optimal states ($ ),-t,....r obtained from problem (R) constitutes a feasible path for problem (P), i.e. there exists some sequence( u,” ),-I ,.,,, r such that x,* =/,(x,5,, a,*) for all t. 0377-2217/87/$3.50 0 1987, Bkevier Science Publishers B.V. (North-Holland)
128
H. U. Buhl / Dynamic programming problems
obtained by solving the simple unconstrained problem (R)
Maximize
i
@ ,(x,),
t-1 x, E q,
where
H,(x,)+G,+,b,), H,b,)s
Qirbt)=
t=l,...,T-1, t = T.
2. Generalizations
.A generalization of dynamic programmin g problems (P), frequently encountered in economic models employing the concept of utility functions is hhbize
U(yl,...,yT),
subject to
(UP> Y,=&h-l> x, =frh-+
4 4
t=l
y,=R,
x,Eq:“,,
xg E 20
(specified).
,-**, T,
u, E fit
Usually, the function 17: lRT+ [w is assumed to be continuously differentiable, concave and increasing, which is to be assumed further. A quite strong generalization of the independenceproperty (IA) is the following (I) For all t there exists some function F, such that %> = F,b,-19 f,(x,-1,
gth-1,
u,>>.
The following theorem shows, how problem (UP) may be solved in special but relevant cases if conditions (I) are satisfied. Theorem 1. Let Z,,CR, 52,GR, Z, be convex and conditions (I) be satisfied for all t = 1,. . . , T. Assume further that the function F,: E2 + II2 and the increasing function U: RT + If2 are concave and differentiable. If satisfying the equation system there exists a feasible sequence {x~},-,,.,~
$2(x,-1,x,)+$+(x,. (0)
au ayT
a& ’ a,T(xT-l.
xT)
=
x,+1)=0, t=l, . . . . T-1,’ O,
then it constitutes an optimal solution to problem (UP). ’ By this abbreviation we mean ~(Yl9....Udl
,-F&r ,-I, x,).t-I,...,
T.
2 If XTEST is specified the equation for t= T vanishes.
t=T,2
H. U. Buhl / Dynamic programming problems
129
Proof. From the problem (UP)‘s formulation and conditions (I) it follows for all feasible sequences
{XtLO,...,T U(Y,,...9 YT) = q&(x,,
4,*-.>
= U(F,(X,> fdxo, = qqx,,
gTbT-*,
UT)>
~,>)Y.JT(xT-,,
fT(XT--l> 4)
x1) ,... 3F,(x,-1, x,)3..., FT(XT-1, XT).
As necessaryfirst order conditions for an interior maximum one obtains condition (0). Since all F, are concave and U is both increasing and concave with respect to all yl,. . . , J+, U is also concave with respect to all x0,. . . , xr. Formally this is shown for some x := (x0,. . . , xr), letting h, p E [0, 11, h + p = 1. Then
The first inequality follows from the concavity of all F, and the monotonicity of U, while the second inequality follows from the concavity of U with respect to all arguments. Due the concavity of U: Rr-, R’ with respect to all x0,. . ., xr each feasible stationary solution satisfying the first order conditions for optimality (0) maximizes the function U: IF4T + R. 0 Corollary 2. If in Theorem 1 the stronger independenceproperty (IA) is satisfied, then the optimality conditions (0) are given by
(04
+yx,)+
5.
3; ‘lqXT) aYT
= 0,
G:+I(x,> =O,
t=l,...,T-1, t = T,
2. If the function U: lRT + IR is given by WY,,...,
YT) = lil
u,(Y,)~@~ (1 + ii)-‘,
where U,: R + IR and i, E [0, co), then the preceding theorem’s equation system reducesto
(01)
U,‘(H,(x,) + G,(x~-I))(~+~,)&‘(x,) + ur:IW,+I(x,+I )+G,+,(x,))G,‘,,(x,)=O, U~(HT(XT)+GT(~T-,))H~(~T)=~,
t=l,...,T-1, t=T.’
3. If we have V,( y,) = y,, then we obtain the equationsystem (02)
(1 + i,)H,‘(x,) &(xT) = O,
+ G,‘+,(x,) = 0,
’ Again, if xT is specified, this equation vanishes.
t = l,..., t = T.
T- 1,
130
H. U. Buhl / Dynamic programming problems
3. Applications Both problem (UP)‘s formulation and the independence property (I) are far more general than the respective ones of [6]. Still, one might consider the independence property (I) too restrictive for practical application. Thus, in the remaining we will describe cases,where this property is satisfied. In [6], we have given one application and briefly discussed five others, where conditions (IA) were satisfied. The discussion of these applications from inventory control, optimal dividend policy and optimal economic growth related to [2] will not be repeated here. Additional applications of the special case in Corollary 2.3 are given in [l]. Many of them could be generalized to problems (UP) with conditions (I) satisfied by introducing concave utility functions instead of the additive functions employed there. In [3] and [4] a number of special problems (UP) are discussed, for all of which the independence property is satisfied. Via application of Theorem 1, in the models are determined - optimal cooperative and noncooperative investment policies, - optimal wages and shares of workers, - optimal saving rates, - optimal tax rates. Further, the economic effects of these policies with respect to distribution and economic growth are analyzed there. For reasons of brevity, we conclude the paper with a generalization of the application given in [6]: Example. Optimal inventory control. Consider the following optimization problem: Maximize
U(Y,,...,
yT),
subject to 4) = (1 -UN - Gib,-1)~ A= &h-1, x, =fr(x,-1, 4 =%h-1) + W(XI-A CUE) x,,, xr specified, t=l,...,T. OGX,, y,, O,
t=L...,T.
After sales in period t, the remaining stock from period t - 1 still has some stock value s,(x,-J. the inventory stock’s value x, is given by x, =h(x,+ For t=l,...,T, ~,b,-d
4
=%(&-I)
Hence
+ vfb,-1).
the term =4x,-d
+ rrh-d
expressesthe firm’s operating value, i.e. the sum of revenuesplus stock value after sales of the inventory stock x,-r. Obviously the independenceproperty (I) is satisfiable by defining F,(n,b)=(l-u,)(s,(a)+r,(a)-b),
Application of Theorem 1 now yields:
t-l
,..., T.
H. U. Buhl / Dynamic programming problems
131
Theorem 3. Let the functions z,:’ [w+ --f R + and the increasing function U: RT + If3 be concave and differentiable. If there exists a feasible sequence { xf” },-O,.. , T satis-ing the equation system
(OE)
Z,‘+,(x,)=(~,&)++, ,
t=l,...,
T-l,
1+1
then it constitutes an optimal solution to problem (UE).
As a numerical example, again along with [6] we specify Sk%1) v, = 0,
= ix,-19 x,=x,=1,
r,(x,-l)
= ix:/‘,,
and assume U: IwT + R to be symmetric. Then x*=1 t=l , . . . , T, is an optimal path of inventory stocks and u,? = l/2 are the corresponding reinvestmlnt radios. In each period t, the owner receives dividends y, = l/4, t = 1,. . . , T, thus
-no
matter how U: !R’, + Iw is precisely specified.
References [l] Buhl, H.U., “Dynamic programming solutions for economic models requiring little information about the future”, in: W. Eichhom and R. Henn (eds.), Mathematical Systems in Economics 86, Athenaurn, Konigstein, 1983. [2] Buhl, H.U., “A discrete model of optimal economic growth”, Journal of Macroeconomics (1984). [3] Buhl, H.U., “Workers’ optimal shares and tax rates in a neoclassical model of distribution and wealth”, Disc. paper No. 250, Institut ft Wirtschaftstheorie und Operations Research, Universitiit Karlsruhe, F.R.G., 1985. [4] Buhl, H.U., “A neoclassical theory of distribution and wealth”, in: M. Beckmann and W. Krelle (eds.) Lecture Notes in Economics and Mathematical Systems 262, Springer, Berlin, 1986. [5] Buhl, H.U., and Eichhorn, W. “Optimal growth for resource-dependent open economies”, in G. Hammer and D. Pallaschke (eds.), Selected Topics in Operations Research and Mathemaiical Economics, Springer, Berlin, 1984. [6] Buhl, H.U., and Siedersleben, J., “On a class of dynamic programming problems, whose optimal controls and states are independent of the future”, European Journal of Operational Research 18 (3) (1984) 364-368. . [7] Dreyfus, S., and Law, A., The Art and Theory of Dynamic Programming, Academic Press, New York, 1977. [8] Neumann, K., “Operations Research Ver/ahren, Band II, Hanser, Miinchen, 1977.
127
Generalizations and applications . of a class of dynam ic program m ing problems Hans Ulrich BUHL Institut fir Wirtschaftstheorie und Operations Research der UniversittitKarlsruhe, am SchloB, Bau IV, 7500 Karlsruhe I, Germany, Fed. Rep.
KaiserstraDe 12, Kollegium
Abstract: In Buhl and Siedersleben(1984) it has been shown, that a class of Dynamic Programming
Problems satisfying certain independenceand reachability conditions has interesting properties and rather simple solutions. This class is generalized here in such a way to be applicable to problems frequently encountered in economic models employing the concept of utility functions. Although the properties used may seem restrictive, they are satisfied for a number of relevant problems. Keywords: Dynamic programming, inventory, investment, growth
1. A class of dynamic programming problems
A general formulation of the dynamic programming problem with one-state-dependentfunctions is given by Maximize ’
i
&h-l,
u,>s
r-1
subject to m
-%=f,b,-l~ U,EG,, xg E gJ
4, X,EZ,,
t=l.,...,T,
1 (specified).
For t = 1,. . . , T, the state space E, and the control space 52, are contained in some supersets ,” and 0. I3+ =ny,,z, is called the global state space and s2’= I-I TBIL?, the global control space. The functions f, and g, are defined on 3 X 52and take on values in ,” and R, respectively. The product sets ZT and OT are defined in the usual way. In Buhl and Siedersleben[6] we have shown that if there exist functions G, and H, such that (IA) and
given
g,(x,-,,
u,>~G,(x,-,)+H,(f,(x,-,,
u,>),
t=l,...,T,
certain reachability conditions precisely specified there 2, the solution(s) of problem (P) can be
Received April 1985; revised June 1986 The author is indebted to anonymus referees for very helpful suggestions. ’ Throughout the paper, ah maxima are assumed to exist; by convention, maxB= - co. * It suffices that some sequenceof optimal states ($ ),-t,....r obtained from problem (R) constitutes a feasible path for problem (P), i.e. there exists some sequence( u,” ),-I ,.,,, r such that x,* =/,(x,5,, a,*) for all t. 0377-2217/87/$3.50 0 1987, Bkevier Science Publishers B.V. (North-Holland)
128
H. U. Buhl / Dynamic programming problems
obtained by solving the simple unconstrained problem (R)
Maximize
i
@ ,(x,),
t-1 x, E q,
where
H,(x,)+G,+,b,), H,b,)s
Qirbt)=
t=l,...,T-1, t = T.
2. Generalizations
.A generalization of dynamic programmin g problems (P), frequently encountered in economic models employing the concept of utility functions is hhbize
U(yl,...,yT),
subject to
(UP> Y,=&h-l> x, =frh-+
4 4
t=l
y,=R,
x,Eq:“,,
xg E 20
(specified).
,-**, T,
u, E fit
Usually, the function 17: lRT+ [w is assumed to be continuously differentiable, concave and increasing, which is to be assumed further. A quite strong generalization of the independenceproperty (IA) is the following (I) For all t there exists some function F, such that %> = F,b,-19 f,(x,-1,
gth-1,
u,>>.
The following theorem shows, how problem (UP) may be solved in special but relevant cases if conditions (I) are satisfied. Theorem 1. Let Z,,CR, 52,GR, Z, be convex and conditions (I) be satisfied for all t = 1,. . . , T. Assume further that the function F,: E2 + II2 and the increasing function U: RT + If2 are concave and differentiable. If satisfying the equation system there exists a feasible sequence {x~},-,,.,~
$2(x,-1,x,)+$+(x,. (0)
au ayT
a& ’ a,T(xT-l.
xT)
=
x,+1)=0, t=l, . . . . T-1,’ O,
then it constitutes an optimal solution to problem (UP). ’ By this abbreviation we mean ~(Yl9....Udl
,-F&r ,-I, x,).t-I,...,
T.
2 If XTEST is specified the equation for t= T vanishes.
t=T,2
H. U. Buhl / Dynamic programming problems
129
Proof. From the problem (UP)‘s formulation and conditions (I) it follows for all feasible sequences
{XtLO,...,T U(Y,,...9 YT) = q&(x,,
4,*-.>
= U(F,(X,> fdxo, = qqx,,
gTbT-*,
UT)>
~,>)Y.JT(xT-,,
fT(XT--l> 4)
x1) ,... 3F,(x,-1, x,)3..., FT(XT-1, XT).
As necessaryfirst order conditions for an interior maximum one obtains condition (0). Since all F, are concave and U is both increasing and concave with respect to all yl,. . . , J+, U is also concave with respect to all x0,. . . , xr. Formally this is shown for some x := (x0,. . . , xr), letting h, p E [0, 11, h + p = 1. Then
The first inequality follows from the concavity of all F, and the monotonicity of U, while the second inequality follows from the concavity of U with respect to all arguments. Due the concavity of U: Rr-, R’ with respect to all x0,. . ., xr each feasible stationary solution satisfying the first order conditions for optimality (0) maximizes the function U: IF4T + R. 0 Corollary 2. If in Theorem 1 the stronger independenceproperty (IA) is satisfied, then the optimality conditions (0) are given by
(04
+yx,)+
5.
3; ‘lqXT) aYT
= 0,
G:+I(x,> =O,
t=l,...,T-1, t = T,
2. If the function U: lRT + IR is given by WY,,...,
YT) = lil
u,(Y,)~@~ (1 + ii)-‘,
where U,: R + IR and i, E [0, co), then the preceding theorem’s equation system reducesto
(01)
U,‘(H,(x,) + G,(x~-I))(~+~,)&‘(x,) + ur:IW,+I(x,+I )+G,+,(x,))G,‘,,(x,)=O, U~(HT(XT)+GT(~T-,))H~(~T)=~,
t=l,...,T-1, t=T.’
3. If we have V,( y,) = y,, then we obtain the equationsystem (02)
(1 + i,)H,‘(x,) &(xT) = O,
+ G,‘+,(x,) = 0,
’ Again, if xT is specified, this equation vanishes.
t = l,..., t = T.
T- 1,
130
H. U. Buhl / Dynamic programming problems
3. Applications Both problem (UP)‘s formulation and the independence property (I) are far more general than the respective ones of [6]. Still, one might consider the independence property (I) too restrictive for practical application. Thus, in the remaining we will describe cases,where this property is satisfied. In [6], we have given one application and briefly discussed five others, where conditions (IA) were satisfied. The discussion of these applications from inventory control, optimal dividend policy and optimal economic growth related to [2] will not be repeated here. Additional applications of the special case in Corollary 2.3 are given in [l]. Many of them could be generalized to problems (UP) with conditions (I) satisfied by introducing concave utility functions instead of the additive functions employed there. In [3] and [4] a number of special problems (UP) are discussed, for all of which the independence property is satisfied. Via application of Theorem 1, in the models are determined - optimal cooperative and noncooperative investment policies, - optimal wages and shares of workers, - optimal saving rates, - optimal tax rates. Further, the economic effects of these policies with respect to distribution and economic growth are analyzed there. For reasons of brevity, we conclude the paper with a generalization of the application given in [6]: Example. Optimal inventory control. Consider the following optimization problem: Maximize
U(Y,,...,
yT),
subject to 4) = (1 -UN - Gib,-1)~ A= &h-1, x, =fr(x,-1, 4 =%h-1) + W(XI-A CUE) x,,, xr specified, t=l,...,T. OGX,, y,, O,
t=L...,T.
After sales in period t, the remaining stock from period t - 1 still has some stock value s,(x,-J. the inventory stock’s value x, is given by x, =h(x,+ For t=l,...,T, ~,b,-d
4
=%(&-I)
Hence
+ vfb,-1).
the term =4x,-d
+ rrh-d
expressesthe firm’s operating value, i.e. the sum of revenuesplus stock value after sales of the inventory stock x,-r. Obviously the independenceproperty (I) is satisfiable by defining F,(n,b)=(l-u,)(s,(a)+r,(a)-b),
Application of Theorem 1 now yields:
t-l
,..., T.
H. U. Buhl / Dynamic programming problems
131
Theorem 3. Let the functions z,:’ [w+ --f R + and the increasing function U: RT + If3 be concave and differentiable. If there exists a feasible sequence { xf” },-O,.. , T satis-ing the equation system
(OE)
Z,‘+,(x,)=(~,&)++, ,
t=l,...,
T-l,
1+1
then it constitutes an optimal solution to problem (UE).
As a numerical example, again along with [6] we specify Sk%1) v, = 0,
= ix,-19 x,=x,=1,
r,(x,-l)
= ix:/‘,,
and assume U: IwT + R to be symmetric. Then x*=1 t=l , . . . , T, is an optimal path of inventory stocks and u,? = l/2 are the corresponding reinvestmlnt radios. In each period t, the owner receives dividends y, = l/4, t = 1,. . . , T, thus
-no
matter how U: !R’, + Iw is precisely specified.
References [l] Buhl, H.U., “Dynamic programming solutions for economic models requiring little information about the future”, in: W. Eichhom and R. Henn (eds.), Mathematical Systems in Economics 86, Athenaurn, Konigstein, 1983. [2] Buhl, H.U., “A discrete model of optimal economic growth”, Journal of Macroeconomics (1984). [3] Buhl, H.U., “Workers’ optimal shares and tax rates in a neoclassical model of distribution and wealth”, Disc. paper No. 250, Institut ft Wirtschaftstheorie und Operations Research, Universitiit Karlsruhe, F.R.G., 1985. [4] Buhl, H.U., “A neoclassical theory of distribution and wealth”, in: M. Beckmann and W. Krelle (eds.) Lecture Notes in Economics and Mathematical Systems 262, Springer, Berlin, 1986. [5] Buhl, H.U., and Eichhorn, W. “Optimal growth for resource-dependent open economies”, in G. Hammer and D. Pallaschke (eds.), Selected Topics in Operations Research and Mathemaiical Economics, Springer, Berlin, 1984. [6] Buhl, H.U., and Siedersleben, J., “On a class of dynamic programming problems, whose optimal controls and states are independent of the future”, European Journal of Operational Research 18 (3) (1984) 364-368. . [7] Dreyfus, S., and Law, A., The Art and Theory of Dynamic Programming, Academic Press, New York, 1977. [8] Neumann, K., “Operations Research Ver/ahren, Band II, Hanser, Miinchen, 1977.