- Email: [email protected]
Optimal control of semilinear stochastic evolution equations
Nonhnear
Analyws,
Theory,
Methods
Pergamon
OPTIMAL
& Applrcatms, Vol. 23, NO. 1, pp. 15-U. ,994 Copyright S 1994 Elsevier Science Ltd Printed in Great Britain. All nghts reserved 0362-546X/94 $7.00+ .OO
CONTROL OF SEMILINEAR STOCHASTIC EVOLUTION EQUATIONS V. S. BORKAR~~ and T. E. GOVINDAN§~(
t Department
of Electrical Engineering and Nonlinear Studies Group, Indian Institute of Science, Bangalore 560012; and 8 University Department of Chemical Technology, University of Bombay, Matunga, Bombay 400019, India (Received 6 February
1993; received for publication
Key words and phrases: Stochastic evolution equation, selection, verification theorem, Nisio semigroup.
optimal
16 June 1993)
control,
Markov
control,
Markov
I. INTRODUCTION
THE AIM of this paper is to initiate a semigroup theory-based approach to characterization of optimal Markov controls for controlled semilinear stochastic evolution equations. (It may be recalled that Markov controls are those that depend only on the current state at each time.) For finite dimensional controlled stochastic differential equations with a nondegenerate diffusion matrix, this task is traditionally achieved through the Hamilton-Jacobi-Bellman equation of dynamic programming associated with the problem and an accompanying “verification theorem”. The latter states that an optimal Markov control can be explicitly obtained by the pointwise minimization of a “Hamiltonian” derivable from the solution of the HJB equation. Moreover, any optimal Markov control is obtainable in this manner. Thus, one simultaneously obtains the existence of, and the necessary and sufficient conditions for, optimal Markov controls [l]. (An alternative approach to necessary conditions proceeds through the stochastic maximum principle, see [2].) In finite dimensional problems with degeneracy or infinite dimensional problems, the HJB equation presents some problems. One, namely possible nonuniqueness of solutions, has been successfully tackled in recent years through the concept of viscosity solutions [3]. A good verification theorem, however, is still missing due to the absence of sufficient regularity of the viscosity solutions (see, however, [4]) and other even more formidable problems, such as the ill-posedness of a general Markov-controlled stochastic differential equation in the absence of nondegeneracy. (The only exception is the case when the dependence of the Markov control on the current state is sufficiently smooth, which is an unreasonable assumption to make.) In the case of the former difficulty, nonsmooth analysis [5] gives some hope, but it still leaves the latter issues wide open. In the case of finite dimensional controlled diffusions with degeneracy, one way these issues were handled, at least insofar as the existence of optimal Markov controls is concerned, was by adopting Krylov’s Markov selection procedure [6], originally developed for obtaining a Markov solution of ill-posed Martingale problems, to select a specific extremal solution measure of the control system which is both optimal and Markov [7, 81. Associated with this Markov process, of course, is its transition semigroup. On the other hand, the control problem leads to the construction of another,
$ Research 11Research
supported supported
by Grant No. 2611192-G from the Department by Chimanlal Choksi Award from the UDCT, 15
of Atomic Energy, Government University of Bombay.
of India.
16
V. S. BORKAR and T. E. GOVINDAN
nonlinear semigroup of operators, called the Nisio semigroup. This semigroup may be viewed as an abstract statement of the dynamic programming principle and is to the HJB equation what the transition semigroup is to the backward Kolmogorov equation [9]. Our central idea is to compare the Nisio semigroup with a modified transition semigroup corresponding to the optimal Markov solution in order to extract necessary and sufficient conditions for optimality of a Markov control. The paper is organized as follows: the next section establishes the existence of a solution to the controlled evolution equation for a prescribed admissible control process and proves the compactness of solution measures over all admissible controls under a fixed initial law, proving thereby the existence of an optimal admissible control. These are proved essentially along standard lines. Section 3 constructs the associated Nisio semigroup. Section 4 uses the Markov selection procedure to extract an optimal Markov solution. Section 5 compares the transition semigroup (to be precise, a modification thereof) of this Markov process with the Nisio semigroup to derive the necessary and sufficient conditions for optimality of a Markov solution. The rest of this section sets up the notation and describes the problem. Let H, H’ be real separable Hilbert spaces and U c H’ the closed unit ball. The control system under consideration will be described by the evolution equation dX(t)
= [AX(t)
+ F(X(t))
+ Bu(t)] dt + dw(t), (1.1)
X(O) = &, for t 2 0. Here: (i) A is the infinitesimal generator of a differentiable semigroup S(t), I > 0, on H (see [lo, p. 731 for definition and properties) with domain a)(A); (ii) F: H --* H, a bounded Lipschitz map with Lipschitz constant K; (iii) B: H’ -+ H, a continuous linear operator; (iv) w( 0) is an H-valued Wiener process with incremental covariance given by a trace class operator Q; (v) X0 an H-valued random variable independent of w( *) and with a prescribed law rrOwhich has bounded moments; and (vi) u( *) is a U-valued control process with measurable paths satisfying the following “nonanticipativity” condition: for t 5 s, w(t) - w(s) is independent of (w(y), u(y),y 5 sJ. The process u(*) as above will be called an admissible control. Call it a Markov control if if it is a timeu( -) = v(X( a)) for a measurable u: H -+ U. Call X( *) a Markov solution homogeneous Markov process. If X( .) is a Markov solution, it follows as in [ll, corollary 1.1.1, p. 131, that u(a) is a Markov control. The converse is not true in general. What is more, for a given measurable u : H -+ U, (1.1) with u( *) set equal to u(X( *)) need not in general have a solution or have a unique one if it does. The cost we seek to minimize over all admissible controls U( .) is m
J(no, 4.)) = E
ewcrrk(X(t), 0
u(t)) dt 1
where cx > 0 and k E C,(H x H’) (the space of bounded continuous maps from H x H’ to R with the topology of uniform convergence on compacts). Let U be endowed with the weak topology of H’ relativized to U. We further assume that k( *, u) is Lipschitz uniformly in u
17
Semilinear stochastic evolution equations
and k(x, a) is convex and, therefore, weakly lower semicontinuous. compact metrizable with the metric d(x,,x,)
= E 2-‘lC~,,e,)
Topologized as above, U is
- (x,,e,)l
n=l
where (e,) is a complete orthonormal basis of H’ and (., * > is the inner particular, U is Polish and k(x, e): U + R lower semicontinuous. It should be added that we consider the weak (in the probabilistic of the above control problem. That is, we seek to minimize the cost (X(a), u( *), w( *)) as above, possibly defined on different probability spaces. for a discussion of these issues. 2. EXISTENCE
OF OPTIMAL
product in H’. In sense) formulation over all processes See [ 11, pp. 17-191
CONTROLS
This section establishes the existence of an optimal admissible control. First, we recall the relevant solution concepts for (1.1). Definition 2.1. An H-valued process X( *) with continuous paths is a strong solution of (1.1) for a prescribed pair of processes (u(v), w(e)) on a probability space (Q, 5, P) with u(e) admissible, if: (i) it is adapted to the filtration (Z,], where 5, = the P-completion of ,? @NY), &% Y 5 4; (ii) X(t) E a)(A) for a.e. t, a.s.; T
(iii) I’0
IIAx(t)II dt < * t
f
(iv)
X(t) = X0
AX(s)ds
+
a.s. for all T 2 0; and
+
F(x(s))ds
+
~~Bu(s)ds
+
i:dw(.s),
t z- 0a.s.
i0
i0
This concept is in general too strong even in the linear case and the equations of delay or hyperbolic type will not usually have strong solutions [12]. This motivates the following definition. Definition (u( -), w( *))
2.2. An H-valued process X( *) is a mild solution of (1 .l) for a prescribed pair as above if (i) above holds, and
(ii)’ t + X(t) is measurable with ji IIx(s)~~~ds < 00 for all t 2 0 a.s.; (iii)’
X(t)
= s(t)&
+
t
t + c qt Jo
t - s)F(X(s)) ds +
S(t - s) dw(s),
Jo
S(t - s)Bu(s) ds
t 1 0; a.s.
0
We shall call (iii)’ the mild version of (1.1). A strong solution is also a mild solution. The conditions for the converse to hold are given by the following lemma which follows as in [13, proposition 2.31.
18
V. S.BORKARandT.E. GOVINDAN
LEMMA 2.1. uEu,t>S; (a) S(t (b) IIAS(t (c) llAS(t Then a mild
Suppose
the following
conditions
hold:
X0 E D(A)
a.s.
and
for each x E H,
s)F(x), S(t - s)Bu E D(A); - s)F(x)II 5 g,(t - s)llxll for some g,: R, -+ R which is locally integrable; - s)BuII I g,(t - s)llxll for some g,: R, 4 R which is locally integrable. solution X(e) of (1.1) is also a strong solution.
The next result proves the existence (u(.), IV(*)) as above.
of a unique
mild solution
to (1.1) for a prescribed
THEOREM 2.1.Given a Wiener process w(a), a random variable X,, and an admissible above on some probability space (Q, 5, P), (1.1) has an a.s. unique mild solution. Proof. Note that IlS(t)II I M for t E [0, T], T L 0, and M < 05 which may depend (see [lo, theorem 3.2.31). Let x(t), t 2 0, i = 1,2 be H-valued processes with continuous on (Q, 5, P) and define ‘r Zi(t)
= S(t)X”
+
pair
u( .) as
on T paths
f [S(t - s)F(yi(s))
S(t - s)dw(s),
+ S(t - s)Bu(s)] ds +
! ,0
i .O
i = 1, 2, t r 0. Fix T > 0 and let II * II denote the supremum norm on C([O, T]; H). Consider the Banach space e of H-valued processes X(t), t E [0, T] on (Q, 5, P) with continuous sample paths under the norm II . II-= [EII. //2]1’2. Then ’ [S(t - s)F(Y,(s)) II.0
- S(t - s)F(y,(s))]
ds
5 TMKIIY,(.)
- yZ (.)11-.
II-
Thus, /lZ,(*) For sufficiently small a unique fixed point repeated on [T, 2T], map and, therefore,
Z2(911-5
TMKIIy,(.)
-
W)11-.
on C and, hence, has T > 0, the map yi( *) + Zi( *) is thus a contraction X( .) which will then be a mild solution of (1.1). The argument may be [2T, 3T] and so on. Conversely, any mild solution is a fixed point of this the unique one.
as in [lo, 141 to cover locally Lipschitz F(.). Remark 2.1. This result may be “localized” See [15] for extensions of the Lipschitz condition. From now on, a solution of (1.1) is always meant in the mild sense unless otherwise indicated. We shall now establish the compactness of the laws of (X(e), u( .)) as u(a) varies over all admissible controls, the law of X0 being kept fixed at no. We shall use Oe(...) to denote “the law of . ..“. Also, for a function f(e)on [0, a), let f([O, T]), T > 0, denote its restriction to [0, T]. We topologize the path space of u(e) as follows: for T > 0, denote by B, the closed unit ball of L,[O, T] with the weak topology of &[O, T] relativized to it. B, is compact metrizable and, hence, Polish. Let B denote the closed unit ball in L,[O, KJ) with the coarsest topology required to render continuous maps B + B, that map f E B to f([O, T]) E B,. Recall the orthonormal basis (ei) of H’. Let ai = (ei, u(e)>, i 2 1. Then ai E B for each i and a(*) = [o(t(*), (Yz(.), ... ] E B”. B, B” are compact Polish spaces as well. The map lo: u E CJ -+ between I/ and u?(U) by virtue of being 1% a 20, (e,, u>, . . .] E [-1, 11” is a homeomorphism
19
Semilinear stochastic evolution equations
a continuous injection with compact domain. We identify u(s), (Y(*) via this homeomorphism and use the notation u(e) to denote either, depending on the context. Similarly, u([O, T]) may be viewed as a B,-valued random variable for T > 0. For any Polish space X, P(X) will denote the Polish space of probability measures on X with the Prohorov topology. Let JGO= ($(X(a), 4.)) I N-1 a d missible, .$(X0) = no] c P(C([O, 00); H) x B”). rTO is tight.
LEMMA 2.2.
Proof. Since B” is compact, it suffices to prove that (&Z(X(*)) /u(e) is admissible, 2(X,) = n,) is tight. Furthermore, it suffices to fix an arbitrary finite T > 0 and show that td:(X(]O, TI)) Id*> a dmissible, d=(X,,) = rrO) is tight in P(C([O, T); H)). Fix t2 > t, in [0, T]. Then
X(t*) - X(1,) = Lw,>& - w,KI1
+
‘fl
[W, - s) - S(f, - ~)lKm))
\ ,O
fl
‘fl
+
.0 I
[S(t, - s) - S(t, - s)]Bu(s) d.s +
’ f2 +
We now obtain II~(~)II, K2 =
-
s)F(X(s))
ds
+ i
i fl U
fl
estimates for each term on the right-hand side. Let K, be a bound on in the proof of lemma 2.1. Then for a suitable constant C > 0,
II1 2n
[II!
[s(t, - s) - S(t, - s)lF(X(s))
ds
0
f1
- t2”-lE 1 <
< -
S(t, - s) dw(s).
s)Bu(s) ds +
-
IIBIIand~4as
IfI
E
[S(t, - s) - S(t, - s)] dw(s) f2
S(t2
t1
moment
i0
n f2 qt,
1
ds
K2”t2”1
ll
II[S(t, - 4 - s(t, -
W'W(s))l12"
1
ds
.0
‘E I
IIS(t,
-
s)
-
S(t,
(by Jensen’s
inequality)
s)112”ds
-
1
I (K,)2”t;“-1E
C2”(t2 - tJ2” ds
(by [lo, theorem
3.2.31)
1 5 (CK,)2”T3n(t2
- tJ,
(2.1) 2n
E
[s(t, - s) - S(t, - s)] dw(s) II 1 I [n(2n - l)]“T”-’
(trace Q)” E
‘I [IS@, - s) - S(t, - s)/[~~ ds 0
1
(by [13, proposition I
[n(2n - l)]“(CT)2”
(trace Q)” (t2 - tl)n,
1.91) (2.2)
V. S. BORKAR and T. E. GOVINDAN
1
II1 2n
fl
E
[S(t, - s) - S(t, - s)]Bu(s) ds 0
dl12n ds
I K;“T2”-1E I (CK2)2”T3n(t2 -
E[llWMo
-
(2.3)
W,)Xo112”l 5 C2"T"E[II~0112"l(t,
td”.
(2.4)
- tl)“,
(2.5)
-
Similarly 2n
t2
E
illi
set, - w(m))
E and E (2.1)-(2.7),
- s)Bu(s) ds
t2
(MKJ2”T”(t2
II1
I (MK2)2”Tn(t2
- tJ,
(2.6)
2n
S(t, - s) dw(s)
I
[n(2n - 1)M2 trace Q]” (t2 - tf.
(2.7)
we obtain E]]tXt,)
for a constant claim.
I
2n
qt, [IIS 11
t1 Combining
II1 II1
ds
- -W,)ii2”l 5 alt, - t,I”
a = a(T, M, K, , K2, C, n). For n = 2, a standard
tightness
criterion
implies
the
LetX”(*), w”(a), u”(e), n = 1,2, . . . . be, respectively, H, Hand U-valued processes on some probability spaces (fi2,, 5,) P,,), n = 1,2, . . ., such that w”(m) is a Wiener process with incremental covariance Q and independent of X”(O), u”(*) is admissible with respect to w”(e), g(X”(0)) = ;rcoand (X”(v), w”(e), u”(e)) satisfy the mild version of (1.1). By the above lemma, d=(X”(*), w”(o), c?(e)), n 2 1 is a tight sequence in P(C([O, 03); H) x C([O, 00); H) x B”), theorem, it where c?(e) = [01;(e), a!;(*), . . . ] with ar = (ei, u”( *)) for i L 1. By Prohorov’s converges along a subsequence. By dropping to this subsequence, we suppose that .JZ(X”( *), w”( *), ofl( *)) + 2(X-(
*), w”( *), a-( *)).
Clearly, .$(X”(O)) = rco and w”(o) is a Wiener process with incremental covariance Q and independent of X”(O). By Skorokhod’s theorem [16, p. 91, we may suppose that space (!A, ‘5, P) and X”(e), W”(‘),CYn(*), n = 1,2, . . . . 00 are defined on a common probability (X”(e), w”(o), (u”(a)) -+ (X”(.), w”(e), cum(*)) in P(C([O, 00); H) x C([O, w); H) x B”) outside a set N E 5 of zero probability. LEMMA
2.3. There exists a U-valued
process
u”(o) such that
ocj”(t) = (ej, u”(t)), Proof. Fix a T > 0 and a sample to satisfy
Ii
point
T (aj@‘(t)
0
jl
1.
o $ N. Let n(1) = 1 and define
- c$“(t))(a,+‘(t)
- a;(t))
dt
(n(k)) inductively
< l/k.
Semilinear stochastic evolution equations
This is possible because $( *) -+ cUj(-) in B. Argue conclude that for each j 2 1,
21
as in the proof
strongly in &[O, T] and, hence, a.e. along a subsequence. extract a subsequence (m(k)) of ]m) such that
of [ll,
By a diagonal
lemma
argument,
II. 1.21 to
we may
a.e. t, j 2 1. Define
u,Jt) E U by Q(t)
p-1
=
(_!_ y
a”“)(t)
m(k) I=1
1
=
y
UNo(t),
t 2
m(k) I=1
>
0.
Then any limit point u(t) of (u,Jt)) in U must satisfy u(t) = olj”(t), j 1 1 for a.e. t. Thus, a”(t) E p(U) for a.e. t where the qualification “a.e.” may be dropped by modifying CY”(0) on a set of zero Lebesgue measure. Set u”(e) = ~-‘(cY~(*)). LEMMA2.4. The process
u”(e) above is an admissible
control
Proof. Fix t 1 s. Then w”(t) - W’(S) is independent n = 1,2,.... Since independence is preserved under convergence n = 03. The claim follows.
with respect
to w”(*).
of (w”([O, s]), c?([O, s])) for in law, this also holds true for
LEMMA2.5. For t 2 0, t
t S(t - s)Bu”(s) ds --f
i
0
S(t - s)Bu”(.s) ds
a.s.
0
Proof. Fix o $ N as above. Recall t -+ u,Jt) defined as in the proof of lemma 2.3. Then uk(t) + u”(t) in U, that is, weakly in H’, for a.e. t. Thus, Buk(t) + Bum(t) weakly in H for a.e. t and in turn, S(t - s)Buf(s) + ,S(t - s)Bu”(s) weakly for a.e. s in [0, t], t L 0. Thus, t
t S(t - s)Bv,(s)
ds -+
weakly.
At the same time, we have x”(t)
tqtI 0
and
S(t - s)Bu”(s) ds s0
i0
s(t)x”(o) s)F(X”(s))
+ X”(t),
t
+ s(t)x”(o), ds -+
i
qt
0
w”(t) -+ w”(t).
- s)F(X”(.s)) ds,
V. S. BORKAR and T. E. GOVINDAN
22
Thus,
equation
(1.1) in the mild form implies
that
‘I
!
S(t - s)Bu”(s) ds + h(t)
,0 for an h E C([O, co); H).
But then
“f
I
S(t - s)Bu”“‘(s)
S(t - s)Bu,(s) ds =
ds + h(t),
,I0 implying -1
h(t)
!
qt
=
-
s)Bu”(s) ds.
.O
This immediately
leads to the following
theorem.
2.2. I_ is compact.
THEOREM
Proof. It suffices to prove that (X”( *), u”(e), w”(a)) satisfies But this is immediate from the proof of the above lemma. Let /3 = inf J(n,, u(a)) where the infimum is over all admissible THEOREM
J(n,,
u”(a))
Proof.
2.3. If (X”(o), u”(e)), n = 1, 2, . . . above = p = lim J(7to, u”(e)). n-m
From
the uniform
Lipschitz
continuity
are
such
the mild
and,
e-“‘[k(X”(t),
that
J(7co, u”(-)) 1 /I,
of k( *, u),
u”(t)) - k(X”(t),
u”(t))] dt + 0
hence, 1, lim E rl
Thus,
for (n(j)),
(m(k)),
u”(t)) dt 1 = fi.
u,(e) as before, &
By convexity
em”’ &Y”(t),
m!‘~[ /
I
1-e -“‘k(X”(t),
u”“‘(t))
e-“‘k(X”(t),
uk(t)) dt
k-m
1
that is, BsE[ proving
dt
.O
of k(x, +), lim sup E
the claim.
<[:e -“‘k@?‘(t),
u”(t)) dt
1
of (1.1).
u(e).
‘Im I,I 0
version
I /3,
a.s.
then
Semilinear stochastic evolution equations COROLLARY 2.1. The set (d=(X( e), u( 0)) E lYr,,1J(n,,
u( *)) = /?I is compact
Proof. That this set is nonempty is already proved above. arguments by taking (X”(.), u”(*)) with J(n,, u”(e)) = p.
nonempty.
Compactness
Theorem 2.3 guarantees the existence of an optimal admissible the existence of an optimal Markov control in Section 4, following semigroup in the next section. 3. THE NISI0
23
follows by the above
control. We improve this to the construction of the Nisio
SEMIGROUP
In this section we shall construct the Nisio semigroup associated with the control system (1.1). This is a semigroup of nonlinear operators which captures the essence of the dynamic programming principle. Let C,(H) denote the Banach space of bounded uniformly continuous functions mapping H + R with the supremum norm, denoted by I(- 11.For f E C,(H), t 2 0, define Qtf(x)
=
u(.) admissible
LEMMA 3.1. Q,f(*)
bf emcus /@X(s), u(s)) ds + e-“‘f(X(t))/X(O)
E
inf
x E H.
=
U “0
E C,(H).
To prove this, we need another technical lemma. Consider prescribed probability space and for x E H, let X(x, t), t z 0, denote the mild solution
u( *), IV(*) on some of (1.1) for X0 = x.
for x,y E H, 3.2. E[llX(x, t) - X(y,s)/12] I c,(llx - y112+ It - .s12+ Jt - sl) t, s E [0, T] for some prescribed T > 0 and some constant C, > 0 which may depend on T.
LEMMA
Proof. Write out explicitly X(x, t) - X(y, s) using the mild versions them and then use the estimates as in lemma 2.2 to deduce E[llX(x,
for some inequality.
constants
0 - X0,
C’,
C”
of (1 .l) satisfied
~)~~21 5 C’(llx - _dz + it - ~1’ + It - ~1) ”T + C" EtllX(x, z) - KY, z)tf21dz /0
depending
on
T. The claim
now
follows
from
the Gronwall
Proofoflemma3.1.Let&>O.Forx,yEH,t>O, IQlf(x)
- Qrf(y)I ‘I I sup E emuS(k(X(x, s), W) UC.)I il
5 tcilx
- ;;
+ wE[lf(X(x,
- k(X(y, 9, 4s))) ds + e-“’ (f(X(x,
t)) - f(X(y,
t)) iIIIIX(x,
t) - XY,
UC.) +
;ly,yElif(X(x,
by
t)) - f(X(y,
t)) 1IiIIX@, 0 - WY,
t)ll<
all
1)) - f(X(y,
011 2 all
0))
V. S. BORKAR and T.E. GOVINDAN
24
for any 6 > 0, C being a positive constant. (The first term bounds the integral involving k( *, 0) by virtue of the uniform Lipschitz continuity of k(. , u) and lemma 3.2.) Pick 6 > 0 above such that If(x) -f(y)\ < E whenever 1(x - ~11 < 6. Then
IQrfC4 - Qrf(y)I 5 tCIlx -YII + E + ~E[IlXx, tc(Jx - yll +
I
where C’ is a bound on llfll and x + Q,f(x) is uniformly continuous. This completes the proof.
& +
$g IIX - Yl12,
C” = C,C’. Since E > 0 was arbitrary, it follows that It is clearly bounded by llfll + tC2 for any C2 L Ik(*, *)I.
Thus, Qr: C,(H) + C,(H), t 2 0, is a family of nonlinear in particular, that it forms a semigroup.
THEOREM
(i) (ii) (iii) (iv)
0 - NY, WI
operators.
The next result proves,
3.1. Q,, t 2 0, satisfying the following: Q,, = Identity; if f, g E C,(H) satisfy f L g, then Q, f L Q,g, t L 0; forf,g E H, t 2 0, IIQtf - Qrgll 5 ewa’llf - gll; IIQ,f - Q,f II--f 0 as t + s;
(v) QtQ, = Q,Qr = Qr+s; (vi) IIQtf II 5 t(l - e--01’)“,“,p(k(x, 2.4)(/a+ e-*‘ll f II.
Proof.
(i), (ii), (vi) are easy. To prove (iii), note that IQtf(4
- Q,g(x)l 5 T;
iE[ ”
-
[~e-“‘k(X(x,
t))]
r
E
[.i 0 5 eC”’
4, 4s)) do + e-“‘f(X(x,
emasW%G 4, u(s)) ds +
e-“’ g(X,(x, 0)
II
I(f - g(l.
To prove (iv), note that for t > s, “f
IQrf(4
- Q,f(x)l
I [! H 5aif -~1 +b 5 y,y
0
s
-E
,O
emayk(X(x, Y), U(Y)) dy + e-“sf(Xx,
wE[If(X(x, t)) - f(X(x, $)I1 UC.1
1 II t))
e?’ &X(x, Y), U(Y)) dy + e-aff(X(x,
E
9)
25
Semilinear stochastic evolution equations
for suitable constants a, b > 0 independent of x. Argue as in the proof of lemma 3.1 to conclude that for any E > 0 and a suitable 6 > 0, JQrf(x) - Q,f(x)\
I aJt - sJ + &b + ?(\I
- sj2 + It - sl),
where C” is as in lemma 3.1. Since E > 0 was arbitrary, (iv) follows. (v) follows from standard dynamic programming arguments as in [ 11, Sections III. 1, 111.31. Properties (i), (v) identify the semigroup property of lQ,, t r 0). This is the Nisio semigroup associated with system (1 .l). We now characterize its infinitesimal generator. Define the operator G on C,(H) by
for h E C,(H) whenever the limit exists in C,(H) and set a>(G) = (h E C,(H) ) the above limit exists]. We call G the infinitesimal generator of (Qt, t 2 0) and d)(G) its domain. Let 6 = (h E C,(H) 1h is twice continuously Frechet differentiable with bounded support and its gradient h’ and Hessian h” are uniformly continuous and uniformly bounded in norm and satisfy h’(x) E 9(A*) for all x E H, where A* is the adjoint of A]. THEOREM
3.2. 6 C a)(G) and for h E 6, Gh = inf [(Lh)(*, U) - olh(*) + k(*, u)] U
where (Lh)(x, u) = (A*h’(x),x)
+ (h’(x),F(x)
+ Bu) + t trace (Qh”(x)).
Proof. The proof depends on an approximation argument adapted from [13]. For A E p(A), the resolvent set of A, and R(I) = AR@, A), where R(A, A) = (AI - A)-‘, let X,(a) satisfy the evolution equation dXh(t) = [AX,(t)
+ R(A)F(X,(t))
+ R(A)Bu(t)] dt + R(A) dw(t), (3.1)
X,(O) = R(A)X,. By lemma 2.1, this has a unique strong solution. We now prove that E[l/X(t) - X,(t)/12] -+ 0 as A -+ 00, t 1 0. Consider t X(t)
- Xi(l)
= s(t)[x,
- R(A)&]
+
.i0
s(t - wvw)
-
t +
I,
+
ds
t qt
s 0 =
WF(X,(~))l
-
s)[Bu(s)
-
R(l)Bu(s)]
ds
+
S(t .i 0
I,,
-
s)[l
-
R(l)]
dw(.s)
26
V. S. BORKAR and T.E. GOVINDAN
where
,f I
‘f
S(t - s)[l - R(A)]F(X(s)) ds +
+
0
1
S(t - s)[Z - R(A)] dw(s).
.O
x,(ol121 5 ww,21 + ml) with E[Z:] and for a suitable E[Z;]
I
constant
M2E[II(Z
c
I 2(MK)‘t
- X,(s)I12] ds
E > 0, - R(A))Xo(12]+ M2t
(
‘E[I(I- R(A))F(X(S))~~~] ds
.i 0 t
+
M=t
E[II(Z - R(A))Bu(s)~~~] ds + M2 trace Q i .O
for M, K as in theorem dominated convergence
2.1, the last term being a consequence theorem,
E[llX(t)
- X,(t)([=] I 2(MK)2r
The claim now follows
from Gronwall’s
‘E[llX(s) i0
inequality.
E]I]Z -
W1121 ds >
of [13, proposition
1.41. By the
- X,(s)ll=] ds + o(h).
Let
I
Q:f(z)
= inf E UC.)
e --OLs k(X,(s),
u(s)) ds + e-“‘f(X,(t))/X,(O)
0
= z
, I
For h E 2, define (L,h)(x,
U) = (A*h’(x),
x> + (h’(x), R@)(F(x)
Applying Ito’s formula of [13, corollary solution of (3.1) for X0 = x, we have
+ Bu)) + * trace(R(A)QR@)h”(x)).
1.21 to h(X,(x,
t)), where h E $j and X,(x,
*) is the
f = infE 4.)
e-as ((k,h)(X,(x, 0
s), u(s)) - oh(X,(x,
s)) + &X,(x,
s), u(s))) ds
. I
Semilinear
Letting A + 00, the foregoing above) to
stochastic
evolution
and our hypothesis
27
equations
on k, h lead (after a small computation
t Q,h(x)
- h(x) = inf E UC.)
emols((Lh)(X(x, [.i
Let X0(x, t), t 2 0, denote
s), 4s))
- cuh(X(x, s)) + k(X(x,
s), u(s))) ds
0
the mild solution
- h(x) 5 inf E aeu
emols((Lh)(X”(x, ij
QtW
- W) 5
for ,any arbitrarily prescribed in x. On the other hand, Q,h(x)
-
t
h(x) z E
s), a) - cuh(X”(x, s)) + k(X’(x,
s), a))) ds
0
Using lemma 3.2 and the uniform 3.1 to conclude
1 .
to (1.1) when X0 = x and u( *) = a E U. Then
‘t Qth(x)
as
continuity
of k, h, h’, h”, argue as in the proof
1 .
of theorem
inf [(Lh)(x, a) - &z(x) + k(x, a)] + E t + o(t) >
( aeu
E > 0 independent
of x and with o(t)/t
e --Ois,‘rf, ((Lh)(X(x,
+ 0 as t 10 uniformly
s), a) - a/2(X(x, s))
0
1
inf [(,55)(x, a) - c&(x) + k(x, a)] - E (1 - e-“‘)/a new >
by analogous arguments, E > 0 again being o(t)/t + 0 as t 10 uniformly in x. Combining ‘I’&?f (QtW
arbitrarily the two,
prescribed
- W)) = SW)
and independent
‘ewus k(X”(s),
= E H
where X”( *) satisfies [ll, theorem 111.3.81.
a) ds + e-“‘f(X”(t))/X”(O)
= x
0
the mild
version
of (1.1)
4. MARKOV
for
of x and
in C,(H).
Remark 3.1. One can also show that Qt, t 2 0, is the “lower envelope” [ll, p. 811 of the affine semigroups Qf, t 2 0, a E U, defined by QPf(x)
- o(t)
u( .) = a. This
can
in the sense of
1 ,
be proved
as in
SELECTION
We shall now set up an analog of Krylov’s Markov selection procedure to extract an optimal Markov control. Let (Si] C (0, 00) be the rationals and (fi) a countable subset of C,(H) which is a separating class for P(H). For n = 0, 1,2, . . . , define F,, = C([O, co); H) x B” + R by m &(x(e),
~(a)) =
e-“’ &x(t), u(t)) dt i
0
V. S. BORKAR and T. E. GOVINDAN
28
and let [Fr , F2, . . .) be a relabelled
enumeration
of the maps
m e+‘fj(x(t))
(x(*)9 Y(.)) + i For rc E P(H),
i,j=
dt,
1,2 ,....
0
let
Jo(n, M.1) = EFo(X-), (that is, Jo(rr, u( *)) = J(n, u(e)) defined
with 2(X(O))
M*)>l
= rr,
earlier),
&I(@ = i$ Jo(n, U(‘)), A,(n) andfori=
Jitrc,
U(‘)) E r, I Job, a*))
x E H,
=
6(n)
= Amf) Ji(n, d’)), 1 IT
let S, denote
E[F,(X(‘)v
u(o))l
U(‘))
Ai For
= lUX(.)Y
= I/o(n)19
1,2,..., with $(X(O))
= loe(x(o>> u(‘)) E Ai-l(n) the
= n,
I Jj(r9 u(*)) = Y(n))*
Dirac
measure at x. Let Al(x) = Ai(6,), v’(x) = I$(6,), 2.1, it follows that Ai( i 2 0, ... . As in corollary is a family of compact nonempty subsets of P(C([O, 00); H) x B”) for each rc. It is clearly nested A,(n) = fliAi(7r) (or A;(X) = A/(X)) (that is, Ai C A,_l(lr) for all i). Thus, are compact nonempty. Let B(n) = (S(X(*)) 1Oe(X(*), u(a)) E A,(n)} (respectively, B’(x) = (JZ(X(*)) 12(X(*), u(e)) E AL(x)]). We shall prove that these are in fact singletons containing a single Markov solution and furthermore, if B’(x) = (P,], then (P,, x E Hj is a Markov family (i.e. satisfies the Chapman-Kolmogorov equation). We start with some technical lemmas. Let (X(e), w(e), u(e)) satisfy the mild version of (1.1) with 2(X(O)) = rr. Let ,u~ E P(C([O, m>; H) x C([O, co); H) x B”) denote the regular conditional law of (X(e), W(D), u(*)) given X(0) = x, defined n- a.s. uniquely. Consider the probability space (d, 5, p) defined as follows: d = C([O, 03); H) x C([O, co); H) x B”, 5 its Bore1 a-field and p = px. Define on this probability space the “canonical” processes X,(e), w,(e), u,(o) as follows: for w = (x(m), u(a), z( *)) E fi, Xx(w, t) = x(t), w,(w, t) = u(t), u,(w, t) = z(t), t > 0. The following is easily proved. Ji'(Xl U(')) = Ji(S,,U(')),i = 0, 1,2,
ni
LEMMA 4.1. For z- a.s. x, w,(e) is a Wiener process with incremental covariance admissible control w.r.t. w,( *) and X,( 0) the corresponding (unique) mild solution X,(O) = x. 4.2. If
LEMMA
da))
E
Proof.
$(X(e),
u(e)) EAT
for
some
i z 0,
then
for
II-
a.s.
A;(x). Let i = 0. Then vo(n) =
n(dx) I/d(x) = E K,‘W(O))l .
Q, u,(e) an to (1.1) with
x,
.33(X,(*),
Semilinear
stochastic
evolution
equations
29
Let E > 0 and E, = 2-“~, n L 1. Let u”( .), n h 1, be admissible controls such that for n 2 1, u”( *) is &,-optimal for initial law rr. Let X”(a) be the corresponding mild solution of (1.1). Let 2, = lx E H ( u,“(a) is c-optimal for the initial condition xl = lx E HI J,(x, uJ(.)) I V;(x) + E),
n 2 1.
Then ~(2,‘) 5 2-“, otherwise the c,-optimality of u”(e) for initial law rr is contradicted. Let B, = Zi, B, = .Z,\(Uk:‘, Z,), n 2 2. Then n(U, B,) = 1. Consider the probability space (a’, 3’, P’) defined as follows: Q’ = H x C([O, co); H) x B”, 5’ = its Bore1 a-field and P’ the probability measure defined as follows: letting (x, y, z), where y = y( *), z = z( *), denote the typical elements of H, C([O, 00); H) and B”, respectively, let P’(dx, dy, dz) = Ir(dz) v(x, dy, dz), where v(x, dy, dz) E P(C([O, co); H) x B”), x E H, is simply 2(X,“(.), u,“(e)) for x E B,, n 2 1 and arbitrary for x $ lJ, B,. Define the “canonical processes” (X(e), ~(a)) on (Sz’, F’, P’) as follows: if UJ = (a, x(.),y(*)) denotes a typical element of Q’, then _%(a, t) = x(t), U(o, t) = y(t), t > 0. It is routine to check that this is an admissible pair satisfying the mild version of (1 .l) with C(X(0)) = rc. Also,
Since E > 0 was arbitrary, it follows that v,(n) = E[&‘(X(O))] = j n(dx)E[F,(X,(*), u,(e))]. The claim now follows for i = 0. Suppose the claim holds for i = 0, 1, 2, . . . ,j. Let $(X(e), u(e)) E Aj+l(n). By hypothesis, S(X,(*), u,(e)) E A;(x) for 7~ a.s. x. Now mimick the above argument to conclude that I$+i(rr) = E[v+i(X(O))] = 1 rc(d_x)E[Fj+,(X,(*), u,(e))] and, therefore, $3(X,(*), u,(+)) E AJ+i(x) for rr- a.s. x. The claim follows by induction. Let S(X,(*), u,(a)) E Ai for some i 2 0, rc E P(H). Let 5, = the completion of n S>f 0(X,(y), ui(y), y 5 s) with respect to the underlying probability measure for t 2 0. Let q be an a.s. finite (S,)-stopping time. Let .53(X,(~)) = P and S(X,(*), u,(e)) E A,(p) for i as above. Construct (2(e), 1?(e)) satisfying the mild version of (1 .l) as follows: let f: C([O, a); H) x B” -+ [0, co) denote the measurable function such that q = j&Y,(*), ui(*)). Let Fj = f@(e), z&o)). The process (x(*), t?(e)) is constructed (say, on the canonical path space C([O, a~); H) x B” endowed with its Bore1 a-field) such that (i) .=W(Q A e), 6(1 A 0)) = J&W7 A *), U(VA e)), (ii) the regular conditional law of (_%(Q+ *), tl(J + *)) given (??(e A *), ti(ij A -)) = its regular con_ditional law given z(e) = the regular conditional law of (X,(m), u,( *)) given X,(O). Thus, (X(e), 17(*)) is a replica in law of (Xi(*), ui(.)) up to the stopping time 3 - q and (X(4 + *), $0 + 0)) is a replica in law of (X,( *), u2( m)). Let 5, = the completion of nS, f a(z(y), u”(y), y I S) with respect to the underlying probability measure for t 2 0. Let 6, (. . .) denote the “regular conditional law of . . .” given gi. LEMMA 4.3.-Ci(_@fj + *),ii(ij + *))E Al(x)(,=gcGj Proof.
a.s.
This is immediate from the preceeding lemma and our construction
of (z( *), c(e)).
V. S. BORKAR and T. E. GOVINDAN
30
LEMMA 4.4. .$(X(*),
Proof. Suppose
G(e)) E Ai(
i = 0. Then
E[F,(z( -),
U( -))I = E
+
eecrt&z(t), eealiE
?
=E
0
C(t)) dt
e-““-“’ k@?(t), C(t)) dt/$,
11
eear k(Xl(t), u,(t)) dt + emaq4(x, u~~(~))I~=~~~~.
(4.1)
1
If the claim were false, then (4.1) is strictly
greater
than
“‘I = E
b(n)
e-“‘k(X,(t), II II0
ul(t)) dt
m +
e-““E
eC”(‘-‘) k(X,(t),
u,(t)) dt/F,
li ‘I
11
‘1
=E
em”’k(X,(t), [S,O
u,(t)) dt + e-“” J&G ti,(-)) Ix=x,cllj I
(4.2)
where fi(*) = u,(q + a). Letting 4 = $(X,(q)), we have J;(x, G,(a)) 2 Vi(x) = J$x, u,,(a)) for $-- a.s. x by lemma 4.2. Thus, (4.2) is greater than or equal to (4.1). Since the opposite inequality is true in any case by the very definition of Vo(n), equality must hold. The claim J,‘(x, ti,( *)) = follows for i = 0. Suppose it holds for i = 0, 1,2, . . . . j, and furthermore, Vi(x) = J/(x, u,,(e)) for 4- a.s. x and 0 5 I 5 j. Now mimick the above argument to show that the claim is true for i = j + 1 and Jjl+,(x, c,(e)) = vj’+,(x) = J’+,(x, u,,(.)) for $- a.s. x. The claim follows by induction. COROLLARY4.1. .G,,(X,(q
+ a), U,(V + .)) E Ai(X)Ix=X,(II) a.s.
Proof. This is equivalent contained
in the proof
to: J/(x, ci,(*)) = c’(x) for dof the above lemma.
LEMMA 4.5. Let AZ(X,(*), u,(e)) E A,(n). process and ur(*) a Markov control.
Proof. The second claim follows that from corollary 4.1, &&X,(v where II is as above.
Then
X,(e)
a.s. x, the proof of which is already
is a time-homogeneous
Markov
from the first as in [II, pp. 12-131. To prove the first, note
+ .A u,(rl + -)) E ~i(~)l,=X,~s~
a.s.,
i 1 0,
Thus,
E]Fi(X,(rl
strong
+ e), u,(ul + -)W,l
= vi’(X,h))
a.s.,
i 2 0.
Semilinear
Taking
conditional
expectations
E]Fi(X,(r
stochastic
with respect
+ *), u,(rl +
-))/m)l
evolution
to X,(q)
31
equations
on both sides
= vi’(X,(rl)) = E[Fi(X,(q + -), u,(q + -))/F,] as.
Thus,
for (si], {f,j as in the beginning
(4.3)
of this section,
1, e --5f1E[f,(X,(q
+ t))/s,]
dr
\ ,0 ‘m =
e -““E[fj(X,(q
+ t))/X,(q)]
dt a.s.,
il
l,jz
1.
I0 Since (Si) are dense in (0, co), it follows E]fj(Xi(11
+
W5,l
from [ll,
= E[fj(X,(r
lemma
+ Wx,(rl)l
IV.l.51,
that
a.s.,
j 2 1, for a.e. t.
of X,(o) Use Fubini’s theorem to interchange “a.s.” and “a.e. t” and invoke path continuity to drop the “a.e”. Since (A] is a separating class for P(H), the strong Markov property follows. Time-homogeneity follows from the observation that the middle term in (4.3) does not have an explicit q-dependence. THEOREM 4.1. Also, if for x ] rc(dx)P,(da) corresponding law II.
For each x E H, B’(x) is a singleton containing the law of a Markov solution. E H, B’(x) = {Px], then (P,, x E HI is a Markov family. Finally, if P,(dw) = for some 71 E P(H) and $(X( *), u( *)) is such that .$(X(e)) = P, and u( *) the Markov control implied by lemma 4.5, then (X(e), u( *)) is optimal for initial
Proof. Suppose JZ(X(.), u(*)), Z(X’( *), u’(.)) E A L(x), for some x E H. Then the foregoing with q = 0 shows that E[Fi(X(*), ~(a))] = E[&(X’(.), u’(e))], i 2 1, leading toE[fi(X(t))] = E’[fi(X’(t))], t > 0, i 2 1, by arguments similar to those of the preceeding lemma. Since cfi) is a separating class for P(H), the laws of X(t), X’(t) agree for each t. Denote their common law of ~(x, t, dy) E P(H). By corollary 4.1, &(X(t + -)) E B(X(t)) a.s. Thus, for i 2 1, t, s 1 0,
i
fi(Z)p(X,
t + S,
dz) = E[fi(X(t
+ s))/X(O) = X]
= E[E[fi (X(t + s))/X(t)ll =E
fi(Z)P(X(t),
S, dZ)
l.i >I
= I// I
fi (z)P(Y, ~3 dz)P(x,
1 t, dY)*
By our choice of (fi], the Chapman-Kolmogorov equation follows. Thus, (p(t, x, dy), t 2 0, x E HJ are indeed the transition probabilities of a Markov process. Since transition probabilities and the initial law completely specify a Markov process in law, each B’(x) must be a singleton. From lemma 4.2, it follows that each B(n) is a singleton. The last claim is obvious.
‘TJ 3 J .103 : dq 2 + 2 : ‘J aur3ap ‘0 z 1 god ‘LLLIOUunu_ua.Idns ayl IJ$!M g + ff suo!wIn3 a~qt?.InsEauI [aloH papunoq 30 aat?ds yXue$~ ayl alouap TJ Ia7 *ssaDo.Id ~oy.~em Zkupuodsa.LIo3 ayi Zu!aq (.)x ‘(( .&)n = (.)n ~O.IWO~~oyni?n B 01 %npuodsal.IoD (1.1) 30 sanweaw uoynlos 30 (H 3 x ‘“d) 1cpu1~?3AO~IEIAJ 1? .Iap!suoD MON '0 < 1 ‘!a 30 uyod paxg E si A ‘snyL ‘0 7 I .103
~yc2wdp p.wpuws
‘LIIEU!~ *(~)“3
01 peal [I’III uorvag ‘1 I] y se ssuawnXn? 8uyuuue.Go.Id 3 h ‘snyL ~~UBISUO~aIqv!ns v 8uIaq 0 < 3 ‘Z’E r?wuraI dq
(‘)n il I (.)n i’lI 114- xl12 +
3 5
0’
IP ((W
‘(I ‘OxPi
- (W
‘(I ‘x)x)Y) ,,_a
2
dns + ? 5
_LL,
I[
0
2~ Wn
‘0 ‘43~
- HIP ‘0 ‘xlxh)
IL
2
,,_a
dns 5 [((c)h - (x)h I
a%IEl 0 < _z YXd ‘0 < 3 pur! H 3 ic ‘x la-I *papunoq ICl.IEap s! hi .anbyun aq .r!ayl ‘alo3alayl ‘pue suoyXwu0~ a.w ‘0 < .t “6 wqi ~~01103 II ‘(~!)I’E uraloayi ‘0 < _I “0
30 lu!od paxy anbrun 0
.
x=
(O)X/IP
(‘)?I
(W ‘(~lxh ,,_a
B
11~~
sw!od pax!3 tu0.1~ *Joor&
ays s! h ‘I’S vruns?
qq!ss!mpr!
= (x)/h
34
,=<11
[
Icq pauyap
s! s!y~ wa[qold
.iuapyAa aq 111~ se ‘uogsas ixau ayi u! waloayi uogez!3Ila.h ayi 30 uogejnw~o3 Ino spage s!y~ *s1oguo3 AO~.IE~J .Iaylo 103 uoyyos au0 uaAa 30 asualsrxa ayl alnsua li saop .ION ~suop!puos pi?!l~u! 1112 10 XLIOS.103 suoyn[os ‘Ipwydouou dIq!ssod ‘laylo ar\ey llew ui pa%nId ((.)x), n = (.)n qil~ (1-1) ieq Aiyq~ssod ayi aicu!uuIa $0~ = (.)*n aq ~0.1iuo~ saop s!yi w?yl pu!tu u! lday aq pinoys 11 ~a~qwns~atu n + jg f *n ‘((.)x)*0 9wpuodsaJlo3 ayl lay ‘Ieurgdo ale (1’ 1) 30 (.)_y Suo~lnIos ~0y.q~~ %upuodsal.IoD ysns {H 3 x cxd) dpwe3 ~oy.qy e 30 aauais!xa ayi away aki ‘az!.n?wuns 01
~oyqq
ayi
wyl
NV(INIAOD
‘3
‘J
pLI?2 XVXXOH
‘S
‘A
ZE
33
Semilinear stochastic evolution equations
semigroup of operators where k,(x) = k(x, v(x)), x E H, and (T,‘, t 1 0) is the transition E -+ E associated with the Markov process X( *). For A > 0, define the resolvent operators R,(A): E + E by m f E E. e-” T,‘f(x) dx, R,(L)f(x) = i0 These satisfy the resolvent
identity
%(A)&(P)
= R&)R,@)
= -(A - P)-~(R,@)
- R,(P)),
for A, fi E (0, a~), implying in particular that range (R,(A)) is independent of A. Let E, denote its closure in E. Then E. is a closed subspace of E given by [17, p. 1101 Eo=(f~E~AR,(A)f+finEas,I-+co).
(5.1)
From (5.1), it follows that T,’ maps E, + E. for t 2 0 and thus may be viewed as a semigroup of operators on E,. Define an operator G: with domain a)(G:) c E, as follows: ZIJ(G:) = R,(A)E, (which is again independent of A), which is then dense in E, by (5.1), and G:: D(G:) + E. is unambiguously defined by (A - G:)R,(A)f = f for all 1 > 0, f E E, [17, p. 1111. Then the following hold [17, pp. llO-1131: (i) R,(A) = (A - G/,-l on E,; (ii) for f E E,, f E a>(G:) if and only if the limit
exists in E, and then G;f LEMMA 5.2.
infinitesimal
= g.
t 1 0) restricts to a semigroup of bounded affine operators generator G, satisfying D(G,) = a)(G:) and for f E D(G:),
(T,,
G,f
E, + E, with
= G:f - af + (a! - G:)R,(a)k,.
(5.2)
Remark 5.1. If k, E E,, the last term on the right in (5.2) is simply k,. Proof.
For f E E, , t > 0, 7;f(x)
- f(x)
= eC”’ T,‘f(x)
- f(x)
+
m emcuS T,’ k,(x) ds 0
=
ema’
q’f(x)
- f(x)
me-“$ T; k,(x) ds i t
+ R,(cr)k,(x)
- em*’ T,’ R,(ol)k,(x).
From (5.3), it follows that 7; maps E, to E,. Semigroup property follows property. (q) are clearly affine. Finally, divide (5.3) on both sides by t and as t 10. Note that R,(oc)f E ‘B(G:) and thus the limit on the right-hand side the left-hand side) exists if and only if f E a)(G:). If it does, the left-hand tends to G,f and the right-hand side is seen to tend to the right-hand side LEMMA 5.3. For t > 0, T,’ : E. + Ho is a contraction
V,(X) = E
co emartk,(X(t)) 0
and has a unique
1
dt 1X(0) = x .
(5.3)
from the Markov take the limit in E, (and, therefore, on side by definition of (5.2).
fixed point
wv given by
34
V. S. BORKAR and T. E. GOVINDAN
Proof.
For
f, g E E,, t > 0, one easily checks that 1T’f(x)
- T,’ &)I
= ~E[e-“‘(fW(O) 5
The first claim follows.
By the Markov
eCatllf - gll.
property, m
*t w,(x) = E
eFas k,(X(s))
ds + e-“‘E
[! 0
e-“‘“-” WWI
ds + emcuty(X(t))/X(O)
[!.O
proving
the second
Our verification
ds/X(0]/WX
= x]
i.i t
'I emolsk,(X(s))
= E
- sW(t)))li
= x
, 1
claim. theorem
is the following
theorem.
5.1. Let {Px, x E H) be a Markov family corresponding to a Markov control u( *) = v(X( e)), X( *) being the corresponding Markov process. Define (T,‘, t L 01, E, as above. (a) The above Markov family is optimal for any initial law if and only if there exists a I?/ E E, fl C,(H) which is the common fixed point of (Qt, t > 0) and (c’, t > 0). (b) Alternatively, the above Markov family is optimal if and only if the value function (I/ defined at the beginning of this section satisfies w E D(G) n a)(G,) and THEOREM
G,i,u=t?=Gv where 19denotes the zero vector in H. (c) At least one optimal Markov family
(5.4)
exists.
Proof. The first part is immediate from the observation that I,Y= wV implies optimality of the Markov family for all initial conditions. The second part follows immediately from the first. The third is simply a restatement of the results of Section 3 and ensures that the first two statements are not vacuous. Note the clear similarity between (5.4) and the “pointwise minimization of the Hamiltonian” that occurs in the finite dimensional verification theorem. In conclusion, it should be noted that this work leaves a few loose ends and should, therefore, be considered only as a first step towards obtaining a good verification theorem for controlled stochastic evolution equations. For one thing, we have made several restrictive assumptions. Another problem is the fact that we do not have a good characterization of the space E,. Also, the uniqueness issue for the evolution of the Nisio semigroup needs to be studied (see, for example, [18]). We hope, however, to have successfully underscored the following point: any statement of verification theorem for controlled stochastic evolution equations in infinite dimensions should involve a Markov family of solution measures rather than an individual Markov control because of the possible ill-posedness of a general Markov-controlled stochastic evolution equation.
Semilinear
stochastic
evolution
equations
35
REFERENCES 1. BENSOUSSANA., Stochastic Control by Functional Analysis Methods. North-Holland, Amsterdam (1982). 2. HAUSSMANNU. G., A stochastic maximum principle for optimal control of diffusions, Pifman Research Notes in Mathematics, Vol. 151. Longman Scientific and Technical, Harlow (1986). 3. LIONS P. L., Viscosity solutions of fully nonlinear second-order equations and optimal stochastic control in infinite dimensions, Part 1: The case of bounded stochastic evolutions, Acta Math. 161, 243-278 (1988). 4. CANNARSAP. & DA PRATOG., Second-order Hamilton-Jacobi equations in infinite dimensions, SIAM J. Control
Optim. 29, 414-492 (1991). 5. CLARKE F. H., Optimization and Nonsmooth Analysis. Wiley, New York (1983). 6. KRYLOV N. V., Selection
of a Markov
process
from a Markov
system of processes,
Izv. Akad. Nauka USSR Ser.
Math. 37, 691-708 (1973). (In Russian.) 7. EL KAROUI N., NGUYEN D. H. & JEANBLANC-PIQUE M., Compactification methods in the control of degenerate diffusions: existence of an optimal control, Sfochastics 20, 169-219 (1987). 8. HAUSSMANN, U. G., Existence of optimal Markovian controls for degenerate diffusions, in Stochastic Differential Systems (Edited by N. CHRISTOPEIT, K. HELMES and M. KOHLMANN), Lecture Notes in Control and Information Sciences, Vol. 78, pp. 171-186. Springer, Berlin (1986). 9. NISIO M., Lectures on Stochastic Control Theory, ISI Lecture Notes, Vol. 9, Macmillan, India (1983). 10. AHMED N. U., Semigroup theory with applications to systems and control, Pitman Research Notes in Mathematics, Vol. 246. Longman Scientific and Technical, Harlow (1991). 11. BORKAR V. S., Optimal control of diffusion processes, Pitman Research Notes in Mathematics, Vol. 203. Longman Scientific and Technical, Harlow (1989). 12. ARNOLD L., CURTAIN R. F. & KOTELENEZ P., Nonlinear stochastic evolution equations in Hilbert space, Report No. 17, University of Bremen (1980). 13. ICHIKAWA A., Stability of semilinear stochastic evolution equations, J. math. Analysis Applic. 90, 12-44 (1982). 14. DA PRATO G. & ZABCZYK J., Nonexplosion, boundedness and ergodicity for stochastic semilinear equations, J. diff Eqns 98, 181-195 (1992). 15. GOVINDAN T. E. & JOSHI M. C., Stability and optimal control of stochastic functional differential equations with memory, Num. funct. Anal. Optim. 13, 249-265 (1992). 16. IKEDA N. & WATANABE S., Stochastic Differential Equations and Diffusion Processes. North-Holland, Tokyo (1981). 17. WILLIAMS D., Diffusions, Markov Processes and Martingales, Vol. 1: Foundations. Wiley, Chichester (1979). 18. NISIO M., Note on uniqueness of semigroup associated with Bellman operator, in Advances in Filtering and Optimal Stochastic Control (Edited by W. FLEMING and L. E. GOROSTIZA), Lecture Notes in Control and Information Sciences, Vol. 42, pp. 267-275. Springer, New York (1982).
Analyws,
Theory,
Methods
Pergamon
OPTIMAL
& Applrcatms, Vol. 23, NO. 1, pp. 15-U. ,994 Copyright S 1994 Elsevier Science Ltd Printed in Great Britain. All nghts reserved 0362-546X/94 $7.00+ .OO
CONTROL OF SEMILINEAR STOCHASTIC EVOLUTION EQUATIONS V. S. BORKAR~~ and T. E. GOVINDAN§~(
t Department
of Electrical Engineering and Nonlinear Studies Group, Indian Institute of Science, Bangalore 560012; and 8 University Department of Chemical Technology, University of Bombay, Matunga, Bombay 400019, India (Received 6 February
1993; received for publication
Key words and phrases: Stochastic evolution equation, selection, verification theorem, Nisio semigroup.
optimal
16 June 1993)
control,
Markov
control,
Markov
I. INTRODUCTION
THE AIM of this paper is to initiate a semigroup theory-based approach to characterization of optimal Markov controls for controlled semilinear stochastic evolution equations. (It may be recalled that Markov controls are those that depend only on the current state at each time.) For finite dimensional controlled stochastic differential equations with a nondegenerate diffusion matrix, this task is traditionally achieved through the Hamilton-Jacobi-Bellman equation of dynamic programming associated with the problem and an accompanying “verification theorem”. The latter states that an optimal Markov control can be explicitly obtained by the pointwise minimization of a “Hamiltonian” derivable from the solution of the HJB equation. Moreover, any optimal Markov control is obtainable in this manner. Thus, one simultaneously obtains the existence of, and the necessary and sufficient conditions for, optimal Markov controls [l]. (An alternative approach to necessary conditions proceeds through the stochastic maximum principle, see [2].) In finite dimensional problems with degeneracy or infinite dimensional problems, the HJB equation presents some problems. One, namely possible nonuniqueness of solutions, has been successfully tackled in recent years through the concept of viscosity solutions [3]. A good verification theorem, however, is still missing due to the absence of sufficient regularity of the viscosity solutions (see, however, [4]) and other even more formidable problems, such as the ill-posedness of a general Markov-controlled stochastic differential equation in the absence of nondegeneracy. (The only exception is the case when the dependence of the Markov control on the current state is sufficiently smooth, which is an unreasonable assumption to make.) In the case of the former difficulty, nonsmooth analysis [5] gives some hope, but it still leaves the latter issues wide open. In the case of finite dimensional controlled diffusions with degeneracy, one way these issues were handled, at least insofar as the existence of optimal Markov controls is concerned, was by adopting Krylov’s Markov selection procedure [6], originally developed for obtaining a Markov solution of ill-posed Martingale problems, to select a specific extremal solution measure of the control system which is both optimal and Markov [7, 81. Associated with this Markov process, of course, is its transition semigroup. On the other hand, the control problem leads to the construction of another,
$ Research 11Research
supported supported
by Grant No. 2611192-G from the Department by Chimanlal Choksi Award from the UDCT, 15
of Atomic Energy, Government University of Bombay.
of India.
16
V. S. BORKAR and T. E. GOVINDAN
nonlinear semigroup of operators, called the Nisio semigroup. This semigroup may be viewed as an abstract statement of the dynamic programming principle and is to the HJB equation what the transition semigroup is to the backward Kolmogorov equation [9]. Our central idea is to compare the Nisio semigroup with a modified transition semigroup corresponding to the optimal Markov solution in order to extract necessary and sufficient conditions for optimality of a Markov control. The paper is organized as follows: the next section establishes the existence of a solution to the controlled evolution equation for a prescribed admissible control process and proves the compactness of solution measures over all admissible controls under a fixed initial law, proving thereby the existence of an optimal admissible control. These are proved essentially along standard lines. Section 3 constructs the associated Nisio semigroup. Section 4 uses the Markov selection procedure to extract an optimal Markov solution. Section 5 compares the transition semigroup (to be precise, a modification thereof) of this Markov process with the Nisio semigroup to derive the necessary and sufficient conditions for optimality of a Markov solution. The rest of this section sets up the notation and describes the problem. Let H, H’ be real separable Hilbert spaces and U c H’ the closed unit ball. The control system under consideration will be described by the evolution equation dX(t)
= [AX(t)
+ F(X(t))
+ Bu(t)] dt + dw(t), (1.1)
X(O) = &, for t 2 0. Here: (i) A is the infinitesimal generator of a differentiable semigroup S(t), I > 0, on H (see [lo, p. 731 for definition and properties) with domain a)(A); (ii) F: H --* H, a bounded Lipschitz map with Lipschitz constant K; (iii) B: H’ -+ H, a continuous linear operator; (iv) w( 0) is an H-valued Wiener process with incremental covariance given by a trace class operator Q; (v) X0 an H-valued random variable independent of w( *) and with a prescribed law rrOwhich has bounded moments; and (vi) u( *) is a U-valued control process with measurable paths satisfying the following “nonanticipativity” condition: for t 5 s, w(t) - w(s) is independent of (w(y), u(y),y 5 sJ. The process u(*) as above will be called an admissible control. Call it a Markov control if if it is a timeu( -) = v(X( a)) for a measurable u: H -+ U. Call X( *) a Markov solution homogeneous Markov process. If X( .) is a Markov solution, it follows as in [ll, corollary 1.1.1, p. 131, that u(a) is a Markov control. The converse is not true in general. What is more, for a given measurable u : H -+ U, (1.1) with u( *) set equal to u(X( *)) need not in general have a solution or have a unique one if it does. The cost we seek to minimize over all admissible controls U( .) is m
J(no, 4.)) = E
ewcrrk(X(t), 0
u(t)) dt 1
where cx > 0 and k E C,(H x H’) (the space of bounded continuous maps from H x H’ to R with the topology of uniform convergence on compacts). Let U be endowed with the weak topology of H’ relativized to U. We further assume that k( *, u) is Lipschitz uniformly in u
17
Semilinear stochastic evolution equations
and k(x, a) is convex and, therefore, weakly lower semicontinuous. compact metrizable with the metric d(x,,x,)
= E 2-‘lC~,,e,)
Topologized as above, U is
- (x,,e,)l
n=l
where (e,) is a complete orthonormal basis of H’ and (., * > is the inner particular, U is Polish and k(x, e): U + R lower semicontinuous. It should be added that we consider the weak (in the probabilistic of the above control problem. That is, we seek to minimize the cost (X(a), u( *), w( *)) as above, possibly defined on different probability spaces. for a discussion of these issues. 2. EXISTENCE
OF OPTIMAL
product in H’. In sense) formulation over all processes See [ 11, pp. 17-191
CONTROLS
This section establishes the existence of an optimal admissible control. First, we recall the relevant solution concepts for (1.1). Definition 2.1. An H-valued process X( *) with continuous paths is a strong solution of (1.1) for a prescribed pair of processes (u(v), w(e)) on a probability space (Q, 5, P) with u(e) admissible, if: (i) it is adapted to the filtration (Z,], where 5, = the P-completion of ,? @NY), &% Y 5 4; (ii) X(t) E a)(A) for a.e. t, a.s.; T
(iii) I’0
IIAx(t)II dt < * t
f
(iv)
X(t) = X0
AX(s)ds
+
a.s. for all T 2 0; and
+
F(x(s))ds
+
~~Bu(s)ds
+
i:dw(.s),
t z- 0a.s.
i0
i0
This concept is in general too strong even in the linear case and the equations of delay or hyperbolic type will not usually have strong solutions [12]. This motivates the following definition. Definition (u( -), w( *))
2.2. An H-valued process X( *) is a mild solution of (1 .l) for a prescribed pair as above if (i) above holds, and
(ii)’ t + X(t) is measurable with ji IIx(s)~~~ds < 00 for all t 2 0 a.s.; (iii)’
X(t)
= s(t)&
+
t
t + c qt Jo
t - s)F(X(s)) ds +
S(t - s) dw(s),
Jo
S(t - s)Bu(s) ds
t 1 0; a.s.
0
We shall call (iii)’ the mild version of (1.1). A strong solution is also a mild solution. The conditions for the converse to hold are given by the following lemma which follows as in [13, proposition 2.31.
18
V. S.BORKARandT.E. GOVINDAN
LEMMA 2.1. uEu,t>S; (a) S(t (b) IIAS(t (c) llAS(t Then a mild
Suppose
the following
conditions
hold:
X0 E D(A)
a.s.
and
for each x E H,
s)F(x), S(t - s)Bu E D(A); - s)F(x)II 5 g,(t - s)llxll for some g,: R, -+ R which is locally integrable; - s)BuII I g,(t - s)llxll for some g,: R, 4 R which is locally integrable. solution X(e) of (1.1) is also a strong solution.
The next result proves the existence (u(.), IV(*)) as above.
of a unique
mild solution
to (1.1) for a prescribed
THEOREM 2.1.Given a Wiener process w(a), a random variable X,, and an admissible above on some probability space (Q, 5, P), (1.1) has an a.s. unique mild solution. Proof. Note that IlS(t)II I M for t E [0, T], T L 0, and M < 05 which may depend (see [lo, theorem 3.2.31). Let x(t), t 2 0, i = 1,2 be H-valued processes with continuous on (Q, 5, P) and define ‘r Zi(t)
= S(t)X”
+
pair
u( .) as
on T paths
f [S(t - s)F(yi(s))
S(t - s)dw(s),
+ S(t - s)Bu(s)] ds +
! ,0
i .O
i = 1, 2, t r 0. Fix T > 0 and let II * II denote the supremum norm on C([O, T]; H). Consider the Banach space e of H-valued processes X(t), t E [0, T] on (Q, 5, P) with continuous sample paths under the norm II . II-= [EII. //2]1’2. Then ’ [S(t - s)F(Y,(s)) II.0
- S(t - s)F(y,(s))]
ds
5 TMKIIY,(.)
- yZ (.)11-.
II-
Thus, /lZ,(*) For sufficiently small a unique fixed point repeated on [T, 2T], map and, therefore,
Z2(911-5
TMKIIy,(.)
-
W)11-.
on C and, hence, has T > 0, the map yi( *) + Zi( *) is thus a contraction X( .) which will then be a mild solution of (1.1). The argument may be [2T, 3T] and so on. Conversely, any mild solution is a fixed point of this the unique one.
as in [lo, 141 to cover locally Lipschitz F(.). Remark 2.1. This result may be “localized” See [15] for extensions of the Lipschitz condition. From now on, a solution of (1.1) is always meant in the mild sense unless otherwise indicated. We shall now establish the compactness of the laws of (X(e), u( .)) as u(a) varies over all admissible controls, the law of X0 being kept fixed at no. We shall use Oe(...) to denote “the law of . ..“. Also, for a function f(e)on [0, a), let f([O, T]), T > 0, denote its restriction to [0, T]. We topologize the path space of u(e) as follows: for T > 0, denote by B, the closed unit ball of L,[O, T] with the weak topology of &[O, T] relativized to it. B, is compact metrizable and, hence, Polish. Let B denote the closed unit ball in L,[O, KJ) with the coarsest topology required to render continuous maps B + B, that map f E B to f([O, T]) E B,. Recall the orthonormal basis (ei) of H’. Let ai = (ei, u(e)>, i 2 1. Then ai E B for each i and a(*) = [o(t(*), (Yz(.), ... ] E B”. B, B” are compact Polish spaces as well. The map lo: u E CJ -+ between I/ and u?(U) by virtue of being 1% a 20, (e,, u>, . . .] E [-1, 11” is a homeomorphism
19
Semilinear stochastic evolution equations
a continuous injection with compact domain. We identify u(s), (Y(*) via this homeomorphism and use the notation u(e) to denote either, depending on the context. Similarly, u([O, T]) may be viewed as a B,-valued random variable for T > 0. For any Polish space X, P(X) will denote the Polish space of probability measures on X with the Prohorov topology. Let JGO= ($(X(a), 4.)) I N-1 a d missible, .$(X0) = no] c P(C([O, 00); H) x B”). rTO is tight.
LEMMA 2.2.
Proof. Since B” is compact, it suffices to prove that (&Z(X(*)) /u(e) is admissible, 2(X,) = n,) is tight. Furthermore, it suffices to fix an arbitrary finite T > 0 and show that td:(X(]O, TI)) Id*> a dmissible, d=(X,,) = rrO) is tight in P(C([O, T); H)). Fix t2 > t, in [0, T]. Then
X(t*) - X(1,) = Lw,>& - w,KI1
+
‘fl
[W, - s) - S(f, - ~)lKm))
\ ,O
fl
‘fl
+
.0 I
[S(t, - s) - S(t, - s)]Bu(s) d.s +
’ f2 +
We now obtain II~(~)II, K2 =
-
s)F(X(s))
ds
+ i
i fl U
fl
estimates for each term on the right-hand side. Let K, be a bound on in the proof of lemma 2.1. Then for a suitable constant C > 0,
II1 2n
[II!
[s(t, - s) - S(t, - s)lF(X(s))
ds
0
f1
- t2”-lE 1 <
< -
S(t, - s) dw(s).
s)Bu(s) ds +
-
IIBIIand~4as
IfI
E
[S(t, - s) - S(t, - s)] dw(s) f2
S(t2
t1
moment
i0
n f2 qt,
1
ds
K2”t2”1
ll
II[S(t, - 4 - s(t, -
W'W(s))l12"
1
ds
.0
‘E I
IIS(t,
-
s)
-
S(t,
(by Jensen’s
inequality)
s)112”ds
-
1
I (K,)2”t;“-1E
C2”(t2 - tJ2” ds
(by [lo, theorem
3.2.31)
1 5 (CK,)2”T3n(t2
- tJ,
(2.1) 2n
E
[s(t, - s) - S(t, - s)] dw(s) II 1 I [n(2n - l)]“T”-’
(trace Q)” E
‘I [IS@, - s) - S(t, - s)/[~~ ds 0
1
(by [13, proposition I
[n(2n - l)]“(CT)2”
(trace Q)” (t2 - tl)n,
1.91) (2.2)
V. S. BORKAR and T. E. GOVINDAN
1
II1 2n
fl
E
[S(t, - s) - S(t, - s)]Bu(s) ds 0
dl12n ds
I K;“T2”-1E I (CK2)2”T3n(t2 -
E[llWMo
-
(2.3)
W,)Xo112”l 5 C2"T"E[II~0112"l(t,
td”.
(2.4)
- tl)“,
(2.5)
-
Similarly 2n
t2
E
illi
set, - w(m))
E and E (2.1)-(2.7),
- s)Bu(s) ds
t2
(MKJ2”T”(t2
II1
I (MK2)2”Tn(t2
- tJ,
(2.6)
2n
S(t, - s) dw(s)
I
[n(2n - 1)M2 trace Q]” (t2 - tf.
(2.7)
we obtain E]]tXt,)
for a constant claim.
I
2n
qt, [IIS 11
t1 Combining
II1 II1
ds
- -W,)ii2”l 5 alt, - t,I”
a = a(T, M, K, , K2, C, n). For n = 2, a standard
tightness
criterion
implies
the
LetX”(*), w”(a), u”(e), n = 1,2, . . . . be, respectively, H, Hand U-valued processes on some probability spaces (fi2,, 5,) P,,), n = 1,2, . . ., such that w”(m) is a Wiener process with incremental covariance Q and independent of X”(O), u”(*) is admissible with respect to w”(e), g(X”(0)) = ;rcoand (X”(v), w”(e), u”(e)) satisfy the mild version of (1.1). By the above lemma, d=(X”(*), w”(o), c?(e)), n 2 1 is a tight sequence in P(C([O, 03); H) x C([O, 00); H) x B”), theorem, it where c?(e) = [01;(e), a!;(*), . . . ] with ar = (ei, u”( *)) for i L 1. By Prohorov’s converges along a subsequence. By dropping to this subsequence, we suppose that .JZ(X”( *), w”( *), ofl( *)) + 2(X-(
*), w”( *), a-( *)).
Clearly, .$(X”(O)) = rco and w”(o) is a Wiener process with incremental covariance Q and independent of X”(O). By Skorokhod’s theorem [16, p. 91, we may suppose that space (!A, ‘5, P) and X”(e), W”(‘),CYn(*), n = 1,2, . . . . 00 are defined on a common probability (X”(e), w”(o), (u”(a)) -+ (X”(.), w”(e), cum(*)) in P(C([O, 00); H) x C([O, w); H) x B”) outside a set N E 5 of zero probability. LEMMA
2.3. There exists a U-valued
process
u”(o) such that
ocj”(t) = (ej, u”(t)), Proof. Fix a T > 0 and a sample to satisfy
Ii
point
T (aj@‘(t)
0
jl
1.
o $ N. Let n(1) = 1 and define
- c$“(t))(a,+‘(t)
- a;(t))
dt
(n(k)) inductively
< l/k.
Semilinear stochastic evolution equations
This is possible because $( *) -+ cUj(-) in B. Argue conclude that for each j 2 1,
21
as in the proof
strongly in &[O, T] and, hence, a.e. along a subsequence. extract a subsequence (m(k)) of ]m) such that
of [ll,
By a diagonal
lemma
argument,
II. 1.21 to
we may
a.e. t, j 2 1. Define
u,Jt) E U by Q(t)
p-1
=
(_!_ y
a”“)(t)
m(k) I=1
1
=
y
UNo(t),
t 2
m(k) I=1
>
0.
Then any limit point u(t) of (u,Jt)) in U must satisfy u(t) = olj”(t), j 1 1 for a.e. t. Thus, a”(t) E p(U) for a.e. t where the qualification “a.e.” may be dropped by modifying CY”(0) on a set of zero Lebesgue measure. Set u”(e) = ~-‘(cY~(*)). LEMMA2.4. The process
u”(e) above is an admissible
control
Proof. Fix t 1 s. Then w”(t) - W’(S) is independent n = 1,2,.... Since independence is preserved under convergence n = 03. The claim follows.
with respect
to w”(*).
of (w”([O, s]), c?([O, s])) for in law, this also holds true for
LEMMA2.5. For t 2 0, t
t S(t - s)Bu”(s) ds --f
i
0
S(t - s)Bu”(.s) ds
a.s.
0
Proof. Fix o $ N as above. Recall t -+ u,Jt) defined as in the proof of lemma 2.3. Then uk(t) + u”(t) in U, that is, weakly in H’, for a.e. t. Thus, Buk(t) + Bum(t) weakly in H for a.e. t and in turn, S(t - s)Buf(s) + ,S(t - s)Bu”(s) weakly for a.e. s in [0, t], t L 0. Thus, t
t S(t - s)Bv,(s)
ds -+
weakly.
At the same time, we have x”(t)
tqtI 0
and
S(t - s)Bu”(s) ds s0
i0
s(t)x”(o) s)F(X”(s))
+ X”(t),
t
+ s(t)x”(o), ds -+
i
qt
0
w”(t) -+ w”(t).
- s)F(X”(.s)) ds,
V. S. BORKAR and T. E. GOVINDAN
22
Thus,
equation
(1.1) in the mild form implies
that
‘I
!
S(t - s)Bu”(s) ds + h(t)
,0 for an h E C([O, co); H).
But then
“f
I
S(t - s)Bu”“‘(s)
S(t - s)Bu,(s) ds =
ds + h(t),
,I0 implying -1
h(t)
!
qt
=
-
s)Bu”(s) ds.
.O
This immediately
leads to the following
theorem.
2.2. I_ is compact.
THEOREM
Proof. It suffices to prove that (X”( *), u”(e), w”(a)) satisfies But this is immediate from the proof of the above lemma. Let /3 = inf J(n,, u(a)) where the infimum is over all admissible THEOREM
J(n,,
u”(a))
Proof.
2.3. If (X”(o), u”(e)), n = 1, 2, . . . above = p = lim J(7to, u”(e)). n-m
From
the uniform
Lipschitz
continuity
are
such
the mild
and,
e-“‘[k(X”(t),
that
J(7co, u”(-)) 1 /I,
of k( *, u),
u”(t)) - k(X”(t),
u”(t))] dt + 0
hence, 1, lim E rl
Thus,
for (n(j)),
(m(k)),
u”(t)) dt 1 = fi.
u,(e) as before, &
By convexity
em”’ &Y”(t),
m!‘~[ /
I
1-e -“‘k(X”(t),
u”“‘(t))
e-“‘k(X”(t),
uk(t)) dt
k-m
1
that is, BsE[ proving
dt
.O
of k(x, +), lim sup E
the claim.
<[:e -“‘k@?‘(t),
u”(t)) dt
1
of (1.1).
u(e).
‘Im I,I 0
version
I /3,
a.s.
then
Semilinear stochastic evolution equations COROLLARY 2.1. The set (d=(X( e), u( 0)) E lYr,,1J(n,,
u( *)) = /?I is compact
Proof. That this set is nonempty is already proved above. arguments by taking (X”(.), u”(*)) with J(n,, u”(e)) = p.
nonempty.
Compactness
Theorem 2.3 guarantees the existence of an optimal admissible the existence of an optimal Markov control in Section 4, following semigroup in the next section. 3. THE NISI0
23
follows by the above
control. We improve this to the construction of the Nisio
SEMIGROUP
In this section we shall construct the Nisio semigroup associated with the control system (1.1). This is a semigroup of nonlinear operators which captures the essence of the dynamic programming principle. Let C,(H) denote the Banach space of bounded uniformly continuous functions mapping H + R with the supremum norm, denoted by I(- 11.For f E C,(H), t 2 0, define Qtf(x)
=
u(.) admissible
LEMMA 3.1. Q,f(*)
bf emcus /@X(s), u(s)) ds + e-“‘f(X(t))/X(O)
E
inf
x E H.
=
U “0
E C,(H).
To prove this, we need another technical lemma. Consider prescribed probability space and for x E H, let X(x, t), t z 0, denote the mild solution
u( *), IV(*) on some of (1.1) for X0 = x.
for x,y E H, 3.2. E[llX(x, t) - X(y,s)/12] I c,(llx - y112+ It - .s12+ Jt - sl) t, s E [0, T] for some prescribed T > 0 and some constant C, > 0 which may depend on T.
LEMMA
Proof. Write out explicitly X(x, t) - X(y, s) using the mild versions them and then use the estimates as in lemma 2.2 to deduce E[llX(x,
for some inequality.
constants
0 - X0,
C’,
C”
of (1 .l) satisfied
~)~~21 5 C’(llx - _dz + it - ~1’ + It - ~1) ”T + C" EtllX(x, z) - KY, z)tf21dz /0
depending
on
T. The claim
now
follows
from
the Gronwall
Proofoflemma3.1.Let&>O.Forx,yEH,t>O, IQlf(x)
- Qrf(y)I ‘I I sup E emuS(k(X(x, s), W) UC.)I il
5 tcilx
- ;;
+ wE[lf(X(x,
- k(X(y, 9, 4s))) ds + e-“’ (f(X(x,
t)) - f(X(y,
t)) iIIIIX(x,
t) - XY,
UC.) +
;ly,yElif(X(x,
by
t)) - f(X(y,
t)) 1IiIIX@, 0 - WY,
t)ll<
all
1)) - f(X(y,
011 2 all
0))
V. S. BORKAR and T.E. GOVINDAN
24
for any 6 > 0, C being a positive constant. (The first term bounds the integral involving k( *, 0) by virtue of the uniform Lipschitz continuity of k(. , u) and lemma 3.2.) Pick 6 > 0 above such that If(x) -f(y)\ < E whenever 1(x - ~11 < 6. Then
IQrfC4 - Qrf(y)I 5 tCIlx -YII + E + ~E[IlXx, tc(Jx - yll +
I
where C’ is a bound on llfll and x + Q,f(x) is uniformly continuous. This completes the proof.
& +
$g IIX - Yl12,
C” = C,C’. Since E > 0 was arbitrary, it follows that It is clearly bounded by llfll + tC2 for any C2 L Ik(*, *)I.
Thus, Qr: C,(H) + C,(H), t 2 0, is a family of nonlinear in particular, that it forms a semigroup.
THEOREM
(i) (ii) (iii) (iv)
0 - NY, WI
operators.
The next result proves,
3.1. Q,, t 2 0, satisfying the following: Q,, = Identity; if f, g E C,(H) satisfy f L g, then Q, f L Q,g, t L 0; forf,g E H, t 2 0, IIQtf - Qrgll 5 ewa’llf - gll; IIQ,f - Q,f II--f 0 as t + s;
(v) QtQ, = Q,Qr = Qr+s; (vi) IIQtf II 5 t(l - e--01’)“,“,p(k(x, 2.4)(/a+ e-*‘ll f II.
Proof.
(i), (ii), (vi) are easy. To prove (iii), note that IQtf(4
- Q,g(x)l 5 T;
iE[ ”
-
[~e-“‘k(X(x,
t))]
r
E
[.i 0 5 eC”’
4, 4s)) do + e-“‘f(X(x,
emasW%G 4, u(s)) ds +
e-“’ g(X,(x, 0)
II
I(f - g(l.
To prove (iv), note that for t > s, “f
IQrf(4
- Q,f(x)l
I [! H 5aif -~1 +b 5 y,y
0
s
-E
,O
emayk(X(x, Y), U(Y)) dy + e-“sf(Xx,
wE[If(X(x, t)) - f(X(x, $)I1 UC.1
1 II t))
e?’ &X(x, Y), U(Y)) dy + e-aff(X(x,
E
9)
25
Semilinear stochastic evolution equations
for suitable constants a, b > 0 independent of x. Argue as in the proof of lemma 3.1 to conclude that for any E > 0 and a suitable 6 > 0, JQrf(x) - Q,f(x)\
I aJt - sJ + &b + ?(\I
- sj2 + It - sl),
where C” is as in lemma 3.1. Since E > 0 was arbitrary, (iv) follows. (v) follows from standard dynamic programming arguments as in [ 11, Sections III. 1, 111.31. Properties (i), (v) identify the semigroup property of lQ,, t r 0). This is the Nisio semigroup associated with system (1 .l). We now characterize its infinitesimal generator. Define the operator G on C,(H) by
for h E C,(H) whenever the limit exists in C,(H) and set a>(G) = (h E C,(H) ) the above limit exists]. We call G the infinitesimal generator of (Qt, t 2 0) and d)(G) its domain. Let 6 = (h E C,(H) 1h is twice continuously Frechet differentiable with bounded support and its gradient h’ and Hessian h” are uniformly continuous and uniformly bounded in norm and satisfy h’(x) E 9(A*) for all x E H, where A* is the adjoint of A]. THEOREM
3.2. 6 C a)(G) and for h E 6, Gh = inf [(Lh)(*, U) - olh(*) + k(*, u)] U
where (Lh)(x, u) = (A*h’(x),x)
+ (h’(x),F(x)
+ Bu) + t trace (Qh”(x)).
Proof. The proof depends on an approximation argument adapted from [13]. For A E p(A), the resolvent set of A, and R(I) = AR@, A), where R(A, A) = (AI - A)-‘, let X,(a) satisfy the evolution equation dXh(t) = [AX,(t)
+ R(A)F(X,(t))
+ R(A)Bu(t)] dt + R(A) dw(t), (3.1)
X,(O) = R(A)X,. By lemma 2.1, this has a unique strong solution. We now prove that E[l/X(t) - X,(t)/12] -+ 0 as A -+ 00, t 1 0. Consider t X(t)
- Xi(l)
= s(t)[x,
- R(A)&]
+
.i0
s(t - wvw)
-
t +
I,
+
ds
t qt
s 0 =
WF(X,(~))l
-
s)[Bu(s)
-
R(l)Bu(s)]
ds
+
S(t .i 0
I,,
-
s)[l
-
R(l)]
dw(.s)
26
V. S. BORKAR and T.E. GOVINDAN
where
,f I
‘f
S(t - s)[l - R(A)]F(X(s)) ds +
+
0
1
S(t - s)[Z - R(A)] dw(s).
.O
x,(ol121 5 ww,21 + ml) with E[Z:] and for a suitable E[Z;]
I
constant
M2E[II(Z
c
I 2(MK)‘t
- X,(s)I12] ds
E > 0, - R(A))Xo(12]+ M2t
(
‘E[I(I- R(A))F(X(S))~~~] ds
.i 0 t
+
M=t
E[II(Z - R(A))Bu(s)~~~] ds + M2 trace Q i .O
for M, K as in theorem dominated convergence
2.1, the last term being a consequence theorem,
E[llX(t)
- X,(t)([=] I 2(MK)2r
The claim now follows
from Gronwall’s
‘E[llX(s) i0
inequality.
E]I]Z -
W1121 ds >
of [13, proposition
1.41. By the
- X,(s)ll=] ds + o(h).
Let
I
Q:f(z)
= inf E UC.)
e --OLs k(X,(s),
u(s)) ds + e-“‘f(X,(t))/X,(O)
0
= z
, I
For h E 2, define (L,h)(x,
U) = (A*h’(x),
x> + (h’(x), R@)(F(x)
Applying Ito’s formula of [13, corollary solution of (3.1) for X0 = x, we have
+ Bu)) + * trace(R(A)QR@)h”(x)).
1.21 to h(X,(x,
t)), where h E $j and X,(x,
*) is the
f = infE 4.)
e-as ((k,h)(X,(x, 0
s), u(s)) - oh(X,(x,
s)) + &X,(x,
s), u(s))) ds
. I
Semilinear
Letting A + 00, the foregoing above) to
stochastic
evolution
and our hypothesis
27
equations
on k, h lead (after a small computation
t Q,h(x)
- h(x) = inf E UC.)
emols((Lh)(X(x, [.i
Let X0(x, t), t 2 0, denote
s), 4s))
- cuh(X(x, s)) + k(X(x,
s), u(s))) ds
0
the mild solution
- h(x) 5 inf E aeu
emols((Lh)(X”(x, ij
QtW
- W) 5
for ,any arbitrarily prescribed in x. On the other hand, Q,h(x)
-
t
h(x) z E
s), a) - cuh(X”(x, s)) + k(X’(x,
s), a))) ds
0
Using lemma 3.2 and the uniform 3.1 to conclude
1 .
to (1.1) when X0 = x and u( *) = a E U. Then
‘t Qth(x)
as
continuity
of k, h, h’, h”, argue as in the proof
1 .
of theorem
inf [(Lh)(x, a) - &z(x) + k(x, a)] + E t + o(t) >
( aeu
E > 0 independent
of x and with o(t)/t
e --Ois,‘rf, ((Lh)(X(x,
+ 0 as t 10 uniformly
s), a) - a/2(X(x, s))
0
1
inf [(,55)(x, a) - c&(x) + k(x, a)] - E (1 - e-“‘)/a new >
by analogous arguments, E > 0 again being o(t)/t + 0 as t 10 uniformly in x. Combining ‘I’&?f (QtW
arbitrarily the two,
prescribed
- W)) = SW)
and independent
‘ewus k(X”(s),
= E H
where X”( *) satisfies [ll, theorem 111.3.81.
a) ds + e-“‘f(X”(t))/X”(O)
= x
0
the mild
version
of (1.1)
4. MARKOV
for
of x and
in C,(H).
Remark 3.1. One can also show that Qt, t 2 0, is the “lower envelope” [ll, p. 811 of the affine semigroups Qf, t 2 0, a E U, defined by QPf(x)
- o(t)
u( .) = a. This
can
in the sense of
1 ,
be proved
as in
SELECTION
We shall now set up an analog of Krylov’s Markov selection procedure to extract an optimal Markov control. Let (Si] C (0, 00) be the rationals and (fi) a countable subset of C,(H) which is a separating class for P(H). For n = 0, 1,2, . . . , define F,, = C([O, co); H) x B” + R by m &(x(e),
~(a)) =
e-“’ &x(t), u(t)) dt i
0
V. S. BORKAR and T. E. GOVINDAN
28
and let [Fr , F2, . . .) be a relabelled
enumeration
of the maps
m e+‘fj(x(t))
(x(*)9 Y(.)) + i For rc E P(H),
i,j=
dt,
1,2 ,....
0
let
Jo(n, M.1) = EFo(X-), (that is, Jo(rr, u( *)) = J(n, u(e)) defined
with 2(X(O))
M*)>l
= rr,
earlier),
&I(@ = i$ Jo(n, U(‘)), A,(n) andfori=
Jitrc,
U(‘)) E r, I Job, a*))
x E H,
=
6(n)
= Amf) Ji(n, d’)), 1 IT
let S, denote
E[F,(X(‘)v
u(o))l
U(‘))
Ai For
= lUX(.)Y
= I/o(n)19
1,2,..., with $(X(O))
= loe(x(o>> u(‘)) E Ai-l(n) the
= n,
I Jj(r9 u(*)) = Y(n))*
Dirac
measure at x. Let Al(x) = Ai(6,), v’(x) = I$(6,), 2.1, it follows that Ai( i 2 0, ... . As in corollary is a family of compact nonempty subsets of P(C([O, 00); H) x B”) for each rc. It is clearly nested A,(n) = fliAi(7r) (or A;(X) = A/(X)) (that is, Ai C A,_l(lr) for all i). Thus, are compact nonempty. Let B(n) = (S(X(*)) 1Oe(X(*), u(a)) E A,(n)} (respectively, B’(x) = (JZ(X(*)) 12(X(*), u(e)) E AL(x)]). We shall prove that these are in fact singletons containing a single Markov solution and furthermore, if B’(x) = (P,], then (P,, x E Hj is a Markov family (i.e. satisfies the Chapman-Kolmogorov equation). We start with some technical lemmas. Let (X(e), w(e), u(e)) satisfy the mild version of (1.1) with 2(X(O)) = rr. Let ,u~ E P(C([O, m>; H) x C([O, co); H) x B”) denote the regular conditional law of (X(e), W(D), u(*)) given X(0) = x, defined n- a.s. uniquely. Consider the probability space (d, 5, p) defined as follows: d = C([O, 03); H) x C([O, co); H) x B”, 5 its Bore1 a-field and p = px. Define on this probability space the “canonical” processes X,(e), w,(e), u,(o) as follows: for w = (x(m), u(a), z( *)) E fi, Xx(w, t) = x(t), w,(w, t) = u(t), u,(w, t) = z(t), t > 0. The following is easily proved. Ji'(Xl U(')) = Ji(S,,U(')),i = 0, 1,2,
ni
LEMMA 4.1. For z- a.s. x, w,(e) is a Wiener process with incremental covariance admissible control w.r.t. w,( *) and X,( 0) the corresponding (unique) mild solution X,(O) = x. 4.2. If
LEMMA
da))
E
Proof.
$(X(e),
u(e)) EAT
for
some
i z 0,
then
for
II-
a.s.
A;(x). Let i = 0. Then vo(n) =
n(dx) I/d(x) = E K,‘W(O))l .
Q, u,(e) an to (1.1) with
x,
.33(X,(*),
Semilinear
stochastic
evolution
equations
29
Let E > 0 and E, = 2-“~, n L 1. Let u”( .), n h 1, be admissible controls such that for n 2 1, u”( *) is &,-optimal for initial law rr. Let X”(a) be the corresponding mild solution of (1.1). Let 2, = lx E H ( u,“(a) is c-optimal for the initial condition xl = lx E HI J,(x, uJ(.)) I V;(x) + E),
n 2 1.
Then ~(2,‘) 5 2-“, otherwise the c,-optimality of u”(e) for initial law rr is contradicted. Let B, = Zi, B, = .Z,\(Uk:‘, Z,), n 2 2. Then n(U, B,) = 1. Consider the probability space (a’, 3’, P’) defined as follows: Q’ = H x C([O, co); H) x B”, 5’ = its Bore1 a-field and P’ the probability measure defined as follows: letting (x, y, z), where y = y( *), z = z( *), denote the typical elements of H, C([O, 00); H) and B”, respectively, let P’(dx, dy, dz) = Ir(dz) v(x, dy, dz), where v(x, dy, dz) E P(C([O, co); H) x B”), x E H, is simply 2(X,“(.), u,“(e)) for x E B,, n 2 1 and arbitrary for x $ lJ, B,. Define the “canonical processes” (X(e), ~(a)) on (Sz’, F’, P’) as follows: if UJ = (a, x(.),y(*)) denotes a typical element of Q’, then _%(a, t) = x(t), U(o, t) = y(t), t > 0. It is routine to check that this is an admissible pair satisfying the mild version of (1 .l) with C(X(0)) = rc. Also,
Since E > 0 was arbitrary, it follows that v,(n) = E[&‘(X(O))] = j n(dx)E[F,(X,(*), u,(e))]. The claim now follows for i = 0. Suppose the claim holds for i = 0, 1, 2, . . . ,j. Let $(X(e), u(e)) E Aj+l(n). By hypothesis, S(X,(*), u,(e)) E A;(x) for 7~ a.s. x. Now mimick the above argument to conclude that I$+i(rr) = E[v+i(X(O))] = 1 rc(d_x)E[Fj+,(X,(*), u,(e))] and, therefore, $3(X,(*), u,(+)) E AJ+i(x) for rr- a.s. x. The claim follows by induction. Let S(X,(*), u,(a)) E Ai for some i 2 0, rc E P(H). Let 5, = the completion of n S>f 0(X,(y), ui(y), y 5 s) with respect to the underlying probability measure for t 2 0. Let q be an a.s. finite (S,)-stopping time. Let .53(X,(~)) = P and S(X,(*), u,(e)) E A,(p) for i as above. Construct (2(e), 1?(e)) satisfying the mild version of (1 .l) as follows: let f: C([O, a); H) x B” -+ [0, co) denote the measurable function such that q = j&Y,(*), ui(*)). Let Fj = f@(e), z&o)). The process (x(*), t?(e)) is constructed (say, on the canonical path space C([O, a~); H) x B” endowed with its Bore1 a-field) such that (i) .=W(Q A e), 6(1 A 0)) = J&W7 A *), U(VA e)), (ii) the regular conditional law of (_%(Q+ *), tl(J + *)) given (??(e A *), ti(ij A -)) = its regular con_ditional law given z(e) = the regular conditional law of (X,(m), u,( *)) given X,(O). Thus, (X(e), 17(*)) is a replica in law of (Xi(*), ui(.)) up to the stopping time 3 - q and (X(4 + *), $0 + 0)) is a replica in law of (X,( *), u2( m)). Let 5, = the completion of nS, f a(z(y), u”(y), y I S) with respect to the underlying probability measure for t 2 0. Let 6, (. . .) denote the “regular conditional law of . . .” given gi. LEMMA 4.3.-Ci(_@fj + *),ii(ij + *))E Al(x)(,=gcGj Proof.
a.s.
This is immediate from the preceeding lemma and our construction
of (z( *), c(e)).
V. S. BORKAR and T. E. GOVINDAN
30
LEMMA 4.4. .$(X(*),
Proof. Suppose
G(e)) E Ai(
i = 0. Then
E[F,(z( -),
U( -))I = E
+
eecrt&z(t), eealiE
?
=E
0
C(t)) dt
e-““-“’ k@?(t), C(t)) dt/$,
11
eear k(Xl(t), u,(t)) dt + emaq4(x, u~~(~))I~=~~~~.
(4.1)
1
If the claim were false, then (4.1) is strictly
greater
than
“‘I = E
b(n)
e-“‘k(X,(t), II II0
ul(t)) dt
m +
e-““E
eC”(‘-‘) k(X,(t),
u,(t)) dt/F,
li ‘I
11
‘1
=E
em”’k(X,(t), [S,O
u,(t)) dt + e-“” J&G ti,(-)) Ix=x,cllj I
(4.2)
where fi(*) = u,(q + a). Letting 4 = $(X,(q)), we have J;(x, G,(a)) 2 Vi(x) = J$x, u,,(a)) for $-- a.s. x by lemma 4.2. Thus, (4.2) is greater than or equal to (4.1). Since the opposite inequality is true in any case by the very definition of Vo(n), equality must hold. The claim J,‘(x, ti,( *)) = follows for i = 0. Suppose it holds for i = 0, 1,2, . . . . j, and furthermore, Vi(x) = J/(x, u,,(e)) for 4- a.s. x and 0 5 I 5 j. Now mimick the above argument to show that the claim is true for i = j + 1 and Jjl+,(x, c,(e)) = vj’+,(x) = J’+,(x, u,,(.)) for $- a.s. x. The claim follows by induction. COROLLARY4.1. .G,,(X,(q
+ a), U,(V + .)) E Ai(X)Ix=X,(II) a.s.
Proof. This is equivalent contained
in the proof
to: J/(x, ci,(*)) = c’(x) for dof the above lemma.
LEMMA 4.5. Let AZ(X,(*), u,(e)) E A,(n). process and ur(*) a Markov control.
Proof. The second claim follows that from corollary 4.1, &&X,(v where II is as above.
Then
X,(e)
a.s. x, the proof of which is already
is a time-homogeneous
Markov
from the first as in [II, pp. 12-131. To prove the first, note
+ .A u,(rl + -)) E ~i(~)l,=X,~s~
a.s.,
i 1 0,
Thus,
E]Fi(X,(rl
strong
+ e), u,(ul + -)W,l
= vi’(X,h))
a.s.,
i 2 0.
Semilinear
Taking
conditional
expectations
E]Fi(X,(r
stochastic
with respect
+ *), u,(rl +
-))/m)l
evolution
to X,(q)
31
equations
on both sides
= vi’(X,(rl)) = E[Fi(X,(q + -), u,(q + -))/F,] as.
Thus,
for (si], {f,j as in the beginning
(4.3)
of this section,
1, e --5f1E[f,(X,(q
+ t))/s,]
dr
\ ,0 ‘m =
e -““E[fj(X,(q
+ t))/X,(q)]
dt a.s.,
il
l,jz
1.
I0 Since (Si) are dense in (0, co), it follows E]fj(Xi(11
+
W5,l
from [ll,
= E[fj(X,(r
lemma
+ Wx,(rl)l
IV.l.51,
that
a.s.,
j 2 1, for a.e. t.
of X,(o) Use Fubini’s theorem to interchange “a.s.” and “a.e. t” and invoke path continuity to drop the “a.e”. Since (A] is a separating class for P(H), the strong Markov property follows. Time-homogeneity follows from the observation that the middle term in (4.3) does not have an explicit q-dependence. THEOREM 4.1. Also, if for x ] rc(dx)P,(da) corresponding law II.
For each x E H, B’(x) is a singleton containing the law of a Markov solution. E H, B’(x) = {Px], then (P,, x E HI is a Markov family. Finally, if P,(dw) = for some 71 E P(H) and $(X( *), u( *)) is such that .$(X(e)) = P, and u( *) the Markov control implied by lemma 4.5, then (X(e), u( *)) is optimal for initial
Proof. Suppose JZ(X(.), u(*)), Z(X’( *), u’(.)) E A L(x), for some x E H. Then the foregoing with q = 0 shows that E[Fi(X(*), ~(a))] = E[&(X’(.), u’(e))], i 2 1, leading toE[fi(X(t))] = E’[fi(X’(t))], t > 0, i 2 1, by arguments similar to those of the preceeding lemma. Since cfi) is a separating class for P(H), the laws of X(t), X’(t) agree for each t. Denote their common law of ~(x, t, dy) E P(H). By corollary 4.1, &(X(t + -)) E B(X(t)) a.s. Thus, for i 2 1, t, s 1 0,
i
fi(Z)p(X,
t + S,
dz) = E[fi(X(t
+ s))/X(O) = X]
= E[E[fi (X(t + s))/X(t)ll =E
fi(Z)P(X(t),
S, dZ)
l.i >I
= I// I
fi (z)P(Y, ~3 dz)P(x,
1 t, dY)*
By our choice of (fi], the Chapman-Kolmogorov equation follows. Thus, (p(t, x, dy), t 2 0, x E HJ are indeed the transition probabilities of a Markov process. Since transition probabilities and the initial law completely specify a Markov process in law, each B’(x) must be a singleton. From lemma 4.2, it follows that each B(n) is a singleton. The last claim is obvious.
‘TJ 3 J .103 : dq 2 + 2 : ‘J aur3ap ‘0 z 1 god ‘LLLIOUunu_ua.Idns ayl IJ$!M g + ff suo!wIn3 a~qt?.InsEauI [aloH papunoq 30 aat?ds yXue$~ ayl alouap TJ Ia7 *ssaDo.Id ~oy.~em Zkupuodsa.LIo3 ayi Zu!aq (.)x ‘(( .&)n = (.)n ~O.IWO~~oyni?n B 01 %npuodsal.IoD (1.1) 30 sanweaw uoynlos 30 (H 3 x ‘“d) 1cpu1~?3AO~IEIAJ 1? .Iap!suoD MON '0 < 1 ‘!a 30 uyod paxg E si A ‘snyL ‘0 7 I .103
~yc2wdp p.wpuws
‘LIIEU!~ *(~)“3
01 peal [I’III uorvag ‘1 I] y se ssuawnXn? 8uyuuue.Go.Id 3 h ‘snyL ~~UBISUO~aIqv!ns v 8uIaq 0 < 3 ‘Z’E r?wuraI dq
(‘)n il I (.)n i’lI 114- xl12 +
3 5
0’
IP ((W
‘(I ‘OxPi
- (W
‘(I ‘x)x)Y) ,,_a
2
dns + ? 5
_LL,
I[
0
2~ Wn
‘0 ‘43~
- HIP ‘0 ‘xlxh)
IL
2
,,_a
dns 5 [((c)h - (x)h I
a%IEl 0 < _z YXd ‘0 < 3 pur! H 3 ic ‘x la-I *papunoq ICl.IEap s! hi .anbyun aq .r!ayl ‘alo3alayl ‘pue suoyXwu0~ a.w ‘0 < .t “6 wqi ~~01103 II ‘(~!)I’E uraloayi ‘0 < _I “0
30 lu!od paxy anbrun 0
.
x=
(O)X/IP
(‘)?I
(W ‘(~lxh ,,_a
B
11~~
sw!od pax!3 tu0.1~ *Joor&
ays s! h ‘I’S vruns?
qq!ss!mpr!
= (x)/h
34
,=<11
[
Icq pauyap
s! s!y~ wa[qold
.iuapyAa aq 111~ se ‘uogsas ixau ayi u! waloayi uogez!3Ila.h ayi 30 uogejnw~o3 Ino spage s!y~ *s1oguo3 AO~.IE~J .Iaylo 103 uoyyos au0 uaAa 30 asualsrxa ayl alnsua li saop .ION ~suop!puos pi?!l~u! 1112 10 XLIOS.103 suoyn[os ‘Ipwydouou dIq!ssod ‘laylo ar\ey llew ui pa%nId ((.)x), n = (.)n qil~ (1-1) ieq Aiyq~ssod ayi aicu!uuIa $0~ = (.)*n aq ~0.1iuo~ saop s!yi w?yl pu!tu u! lday aq pinoys 11 ~a~qwns~atu n + jg f *n ‘((.)x)*0 9wpuodsaJlo3 ayl lay ‘Ieurgdo ale (1’ 1) 30 (.)_y Suo~lnIos ~0y.q~~ %upuodsal.IoD ysns {H 3 x cxd) dpwe3 ~oy.qy e 30 aauais!xa ayi away aki ‘az!.n?wuns 01
~oyqq
ayi
wyl
NV(INIAOD
‘3
‘J
pLI?2 XVXXOH
‘S
‘A
ZE
33
Semilinear stochastic evolution equations
semigroup of operators where k,(x) = k(x, v(x)), x E H, and (T,‘, t 1 0) is the transition E -+ E associated with the Markov process X( *). For A > 0, define the resolvent operators R,(A): E + E by m f E E. e-” T,‘f(x) dx, R,(L)f(x) = i0 These satisfy the resolvent
identity
%(A)&(P)
= R&)R,@)
= -(A - P)-~(R,@)
- R,(P)),
for A, fi E (0, a~), implying in particular that range (R,(A)) is independent of A. Let E, denote its closure in E. Then E. is a closed subspace of E given by [17, p. 1101 Eo=(f~E~AR,(A)f+finEas,I-+co).
(5.1)
From (5.1), it follows that T,’ maps E, + E. for t 2 0 and thus may be viewed as a semigroup of operators on E,. Define an operator G: with domain a)(G:) c E, as follows: ZIJ(G:) = R,(A)E, (which is again independent of A), which is then dense in E, by (5.1), and G:: D(G:) + E. is unambiguously defined by (A - G:)R,(A)f = f for all 1 > 0, f E E, [17, p. 1111. Then the following hold [17, pp. llO-1131: (i) R,(A) = (A - G/,-l on E,; (ii) for f E E,, f E a>(G:) if and only if the limit
exists in E, and then G;f LEMMA 5.2.
infinitesimal
= g.
t 1 0) restricts to a semigroup of bounded affine operators generator G, satisfying D(G,) = a)(G:) and for f E D(G:),
(T,,
G,f
E, + E, with
= G:f - af + (a! - G:)R,(a)k,.
(5.2)
Remark 5.1. If k, E E,, the last term on the right in (5.2) is simply k,. Proof.
For f E E, , t > 0, 7;f(x)
- f(x)
= eC”’ T,‘f(x)
- f(x)
+
m emcuS T,’ k,(x) ds 0
=
ema’
q’f(x)
- f(x)
me-“$ T; k,(x) ds i t
+ R,(cr)k,(x)
- em*’ T,’ R,(ol)k,(x).
From (5.3), it follows that 7; maps E, to E,. Semigroup property follows property. (q) are clearly affine. Finally, divide (5.3) on both sides by t and as t 10. Note that R,(oc)f E ‘B(G:) and thus the limit on the right-hand side the left-hand side) exists if and only if f E a)(G:). If it does, the left-hand tends to G,f and the right-hand side is seen to tend to the right-hand side LEMMA 5.3. For t > 0, T,’ : E. + Ho is a contraction
V,(X) = E
co emartk,(X(t)) 0
and has a unique
1
dt 1X(0) = x .
(5.3)
from the Markov take the limit in E, (and, therefore, on side by definition of (5.2).
fixed point
wv given by
34
V. S. BORKAR and T. E. GOVINDAN
Proof.
For
f, g E E,, t > 0, one easily checks that 1T’f(x)
- T,’ &)I
= ~E[e-“‘(fW(O) 5
The first claim follows.
By the Markov
eCatllf - gll.
property, m
*t w,(x) = E
eFas k,(X(s))
ds + e-“‘E
[! 0
e-“‘“-” WWI
ds + emcuty(X(t))/X(O)
[!.O
proving
the second
Our verification
ds/X(0]/WX
= x]
i.i t
'I emolsk,(X(s))
= E
- sW(t)))li
= x
, 1
claim. theorem
is the following
theorem.
5.1. Let {Px, x E H) be a Markov family corresponding to a Markov control u( *) = v(X( e)), X( *) being the corresponding Markov process. Define (T,‘, t L 01, E, as above. (a) The above Markov family is optimal for any initial law if and only if there exists a I?/ E E, fl C,(H) which is the common fixed point of (Qt, t > 0) and (c’, t > 0). (b) Alternatively, the above Markov family is optimal if and only if the value function (I/ defined at the beginning of this section satisfies w E D(G) n a)(G,) and THEOREM
G,i,u=t?=Gv where 19denotes the zero vector in H. (c) At least one optimal Markov family
(5.4)
exists.
Proof. The first part is immediate from the observation that I,Y= wV implies optimality of the Markov family for all initial conditions. The second part follows immediately from the first. The third is simply a restatement of the results of Section 3 and ensures that the first two statements are not vacuous. Note the clear similarity between (5.4) and the “pointwise minimization of the Hamiltonian” that occurs in the finite dimensional verification theorem. In conclusion, it should be noted that this work leaves a few loose ends and should, therefore, be considered only as a first step towards obtaining a good verification theorem for controlled stochastic evolution equations. For one thing, we have made several restrictive assumptions. Another problem is the fact that we do not have a good characterization of the space E,. Also, the uniqueness issue for the evolution of the Nisio semigroup needs to be studied (see, for example, [18]). We hope, however, to have successfully underscored the following point: any statement of verification theorem for controlled stochastic evolution equations in infinite dimensions should involve a Markov family of solution measures rather than an individual Markov control because of the possible ill-posedness of a general Markov-controlled stochastic evolution equation.
Semilinear
stochastic
evolution
equations
35
REFERENCES 1. BENSOUSSANA., Stochastic Control by Functional Analysis Methods. North-Holland, Amsterdam (1982). 2. HAUSSMANNU. G., A stochastic maximum principle for optimal control of diffusions, Pifman Research Notes in Mathematics, Vol. 151. Longman Scientific and Technical, Harlow (1986). 3. LIONS P. L., Viscosity solutions of fully nonlinear second-order equations and optimal stochastic control in infinite dimensions, Part 1: The case of bounded stochastic evolutions, Acta Math. 161, 243-278 (1988). 4. CANNARSAP. & DA PRATOG., Second-order Hamilton-Jacobi equations in infinite dimensions, SIAM J. Control
Optim. 29, 414-492 (1991). 5. CLARKE F. H., Optimization and Nonsmooth Analysis. Wiley, New York (1983). 6. KRYLOV N. V., Selection
of a Markov
process
from a Markov
system of processes,
Izv. Akad. Nauka USSR Ser.
Math. 37, 691-708 (1973). (In Russian.) 7. EL KAROUI N., NGUYEN D. H. & JEANBLANC-PIQUE M., Compactification methods in the control of degenerate diffusions: existence of an optimal control, Sfochastics 20, 169-219 (1987). 8. HAUSSMANN, U. G., Existence of optimal Markovian controls for degenerate diffusions, in Stochastic Differential Systems (Edited by N. CHRISTOPEIT, K. HELMES and M. KOHLMANN), Lecture Notes in Control and Information Sciences, Vol. 78, pp. 171-186. Springer, Berlin (1986). 9. NISIO M., Lectures on Stochastic Control Theory, ISI Lecture Notes, Vol. 9, Macmillan, India (1983). 10. AHMED N. U., Semigroup theory with applications to systems and control, Pitman Research Notes in Mathematics, Vol. 246. Longman Scientific and Technical, Harlow (1991). 11. BORKAR V. S., Optimal control of diffusion processes, Pitman Research Notes in Mathematics, Vol. 203. Longman Scientific and Technical, Harlow (1989). 12. ARNOLD L., CURTAIN R. F. & KOTELENEZ P., Nonlinear stochastic evolution equations in Hilbert space, Report No. 17, University of Bremen (1980). 13. ICHIKAWA A., Stability of semilinear stochastic evolution equations, J. math. Analysis Applic. 90, 12-44 (1982). 14. DA PRATO G. & ZABCZYK J., Nonexplosion, boundedness and ergodicity for stochastic semilinear equations, J. diff Eqns 98, 181-195 (1992). 15. GOVINDAN T. E. & JOSHI M. C., Stability and optimal control of stochastic functional differential equations with memory, Num. funct. Anal. Optim. 13, 249-265 (1992). 16. IKEDA N. & WATANABE S., Stochastic Differential Equations and Diffusion Processes. North-Holland, Tokyo (1981). 17. WILLIAMS D., Diffusions, Markov Processes and Martingales, Vol. 1: Foundations. Wiley, Chichester (1979). 18. NISIO M., Note on uniqueness of semigroup associated with Bellman operator, in Advances in Filtering and Optimal Stochastic Control (Edited by W. FLEMING and L. E. GOROSTIZA), Lecture Notes in Control and Information Sciences, Vol. 42, pp. 267-275. Springer, New York (1982).