On sets of occupational measures generated by a deterministic control system on an infinite time horizon

Nonlinear Analysis 88 (2013) 27–41

Contents lists available at SciVerse ScienceDirect

Nonlinear Analysis journal homepage: www.elsevier.com/locate/na

On sets of occupational measures generated by a deterministic control system on an infinite time horizon V. Gaitsgory a,∗ , M. Quincampoix b a

Mathematical Sciences Laboratory, Flinders University, GPO Box 2100, Adelaide, SA 2001, Australia

b

Laboratoire de Mathématiques de Bretagne Atlantique, unité CNRS UMR6205, Université de Brest, 6 Avenue Victor Le Gorgeu, 29200 Brest, France

article

abstract

info

Article history: Received 10 September 2012 Accepted 28 March 2013 Communicated by S. Carl Keywords: Deterministic control systems Differential inclusions Infinite time horizon Discounting and long run time average criteria Linear constraints representations of occupational measures sets

We give a representation for the closed convex hull of the set of discounted occupational measures generated by control-state trajectories of a deterministic control system. We also investigate the limit behavior of the latter when the discount factor tends to zero and compare it with the limit behavior of the long run time average occupational measures set. The novelty of our results is in that we allow the control set dependence on the state variables that make the results to be applicable to differential inclusions. © 2013 Published by Elsevier Ltd

1. Introduction and preliminaries It is well known that nonlinear optimal control problems can be equivalently reformulated as infinite dimensional linear programming problems considered on spaces of occupational measures generated by control-state trajectories. Having many attractive features and being applicable in both stochastic and deterministic settings, the linear programming (LP) based approaches to optimal control problems have been intensively studied in the literature. Important results justifying the use of LP formulations in dealing with various problems of optimal control of stochastic systems were obtained in [1–5] (see also more recent developments in [6–9]). Various aspects of LP based analysis and solution of deterministic optimal control problems on a finite time horizon were studied in [10–15] (and in some earlier works mentioned therein). The LP approach to deterministic optimal control problems with long run time average and time discounting criteria was developed in [16–20]. This paper continues the line of research started in [16–18], with the main focus (and novelty) of the results presented here being that the control set is allowed to be dependent on the state variables (which extends the applicability of the LP approach to differential inclusions). We consider the control system y′ (t ) = f (u(t ), y(t )),

t > 0,

y(0) = y0 ,

(1)

where the controls are measurable functions satisfying the inclusion u(t ) ∈ U (y(t )).

∗

Corresponding author. E-mail addresses: [email protected], [email protected] (V. Gaitsgory), [email protected] (M. Quincampoix).

0362-546X/$ – see front matter © 2013 Published by Elsevier Ltd http://dx.doi.org/10.1016/j.na.2013.03.015

(2)

28

V. Gaitsgory, M. Quincampoix / Nonlinear Analysis 88 (2013) 27–41

Here f (u, y) : M × Rm → Rm is a continuous function of (u, y) that satisfies local Lipschitz conditions in y uniformly with respect to u ∈ M, M is a compact metric space and U (·) is an upper semicontinuous set-valued function that maps points in Rm into closed subsets of M (U (y) ⊂ M). Solutions of the system (1) will be assumed to satisfy the inclusion y(t ) ∈ Y

∀t ∈ [0, ∞),

(3) m

where Y is a given compact subset of R , with (3) being interpreted as a state constraint. The paper presents results about sets of occupational measures generated by the solutions of the system (1)–(2). Firstly, we obtain a representation formula for the closed convex hull of the set of discounted occupational measures generated by the control-state trajectories of the system (1)–(2). This result is stated in Section 2.1 (see Theorem 2.2) and proved in Section 3 (the idea of the proof being similar to that of [15], where optimal control problems on a finite time horizon were considered). Secondly, we use the above mentioned result to obtain a representation formula for the limit of the closed convex hull of the set of ‘‘non-discounted’’ occupational measures (these being simply called occupational measures) generated by the control-state trajectories of the system (1)–(2) on a finite time interval [0, S ] when S → ∞. These results are stated in Section 2.2 (see Theorems 2.5, 2.6 and Proposition 2.7) and proved in Section 4. Let us introduce some notations and definitions that are going to be used in the paper. Given a compact metric space X , the space of probability measures defined on Borel subsets of X will be denoted as P (X ). Note that, being endowed with a metric consistent with its weak∗ topology, P (X ) becomes a compact metric space. Given such a metric ρ , one can define the ‘‘distance’’ ρ(γ , Γ ) between γ ∈ P (X ) and Γ ⊂ P (X ) and the Hausdorff metric ρH (Γ1 , Γ2 ) between Γ1 ⊂ P (X ) and Γ2 ⊂ P (X ) as follows: def

def

ρ(γ , Γ ) = inf ρ(γ , γ ′ ), ′

ρH (Γ1 , Γ2 ) = max{ sup ρ(γ , Γ2 ), sup ρ(γ , Γ1 )}.

γ ∈Γ

γ ∈Γ1

γ ∈Γ2

(4)

A pair (u(t ), y(t )) will be called y0 -admissible on a finite time interval [0, S ] (S > 0), or on the infinite time interval [0, ∞) if (1)–(2) are satisfied for almost all t ∈ [0, S ] and y(t ) ∈ Y for all t ∈ [0, S ] (respectively, for almost all and all t ∈ [0, ∞)). Let def

K = Graph(U ) = {(u, y) : u ∈ U (y), y ∈ Y }.

(5)

Note that, due to the upper semicontinuity of U (·), K is a compact subset of M × Y . Given an y0 -admissible pair (u(·), y(·)), the probability measure γ ∈ P (K ) is called the occupational measure generated by this pair on an interval [0, S ] if



h(u, y)γ (du, dy) = K

1

S



S

h(u(t ), y(t ))dt

(6)

0

for any h(·) ∈ C (K ) (the space of continuous functions on K ). The probability measure γ ∈ P (K ) is called the discounted occupational measure generated by this pair if



h(u, y)γ (du, dy) = C

∞



K

e−Ct h(u(t ), y(t ))dt

(7)

0

for any h(·) ∈ C (K ). Along with the system (1)–(2), let us consider a relaxed version of this system y′ (t ) = f¯ (µ(t ), y(t )),

t > 0,

y(0) = y0 ,

(8)

where the controls are measurable functions µ(t ) = µ(t , du) ∈ P (U (y(t ))) and def

f¯ (µ(t ), y(t )) =

 U (y(t ))

f (u, y(t ))µ(t , du).

(9)

A pair (µ(t ), y(t )) will be called relaxed y0 -admissible on a finite time interval [0, S ] (S > 0), or on the infinite time interval [0, ∞) if (8)–(9) are satisfied for almost all t ∈ [0, S ] and y(t ) ∈ Y for all t ∈ [0, S ] (respectively, for almost all and all t ∈ [0, ∞)). Given a relaxed y0 -admissible pair (µ(·), y(·)), the probability measure γ ∈ P (K ) is called the occupational measure generated by this pair on an interval [0, S ] if     1 S h(u, y)γ (du, dy) = h(u, y(t ))µ(t , du) dt (10) S

K

U (y(t ))

0

for any h(·) ∈ C (K ). The probability measure γ ∈ P (K ) is called the discounted occupational measure generated by this pair if



h(u, y)γ (dy, du) = C K

for any h(·) ∈ C (K ).



∞

e 0

−Ct

 U (y(t ))

 h(u, y(t ))µ(t , du) dt ,

(11)

V. Gaitsgory, M. Quincampoix / Nonlinear Analysis 88 (2013) 27–41

29

In the following, Γ (S , y0 ) and Γ r (S , y0 ) will stand for the sets of occupational measures generated by all y0 -admissible r and, respectively, by all relaxed y0 -admissible pairs on [0, S ]. Also, Γdis (C , y0 ) and Γdis (C , y0 ) will stand for the sets of discounted occupational measures generated by all y0 -admissible and, respectively, by all relaxed y0 -admissible pairs. Note that every y0 -admissible pair is also relaxed y0 -admissible (with µ(t ) being a Dirac measure ∀t > 0). Hence

Γ (S , y0 ) ⊂ Γ r (S , y0 ),

r Γdis (C , y0 ) ⊂ Γdis (C , y0 ),

with Γ (S , y0 ) and (C , y0 ) being closed (and, hence, compact) subsets of P (K ). Under the assumption that Filippov–Wazewski type theorems are applicable on Y (see [21]), the equalities r Γdis

r

clΓ (S , y0 ) = Γ r (S , y0 ),

r clΓdis (C , y0 ) = Γdis (C , y0 ),

(12)

are valid, where cl stands for the closure of the corresponding set. These equalities are valid, for example, if Y is a forward invariant set with respect to the solutions of the system (1)–(2) and if the map U (·) satisfies Lipschitz conditions on Y (see, e.g., Theorems 10.4.3 and 10.4.4 in [22]). 2. Main results 2.1. A representation of the set of discounted occupational measures Lemma 2.1. The following inclusion is valid r ¯ Γdis co (C , y0 ) ⊂ W (C , y0 ),

(13)

¯ stands for the closed convex hull, and where co    γ ∈ P (K ) : (∇φ(y)T f (u, y) + C (φ(y0 ) − φ(y)))γ (du, dy) = 0 ∀φ ∈ C 1 .

def

W (C , y0 ) =

(14)

K r Proof. Take arbitrary γ ∈ Γdis (C , y0 ). By definition, there exists a relaxed y0 -admissible pair (µ(·), y(·)) such that γ is the discounted occupational measure generated by this pair. Using the fact that (11) is valid for any continuous function h(u, y), one can obtain



∇φ(y) f (u, y)γ (du, dy) = C T

K



∞

e

0 ∞

−Ct

(∇φ(y(t )))

T





U (y(t ))

f (u, y(t ))µ(t , du) dt



∞

(∇φ(y(t ))) y (t )dt = −C φ(y0 ) + C e−Ct φ(y(t ))dt 0  0  ∞ = −C 2 e−Ct (φ(y0 ) − φ(y(t )))dt = −C (φ(y0 ) − φ(y))γ (dy, du) ∀φ ∈ C 1

= C

e

−Ct

0

T ′

2

K

r ⇒ γ ∈ W (C , y0 ) ⇒ Γdis (C , y0 ) ⊂ W (C , y0 ).

As can be readily seen, the set W (C , y0 ) is convex and weak∗ compact. Hence, the last inclusion implies (13).



The main result of this paper is the following Theorem 2.2. Assume that there exists at least one relaxed y0 -admissible pair. Then r ¯ Γdis co (C , y0 ) = W (C , y0 ).

(15)

Proof. The proof of the theorem is given in Section 3. Note here only that the idea of the proof is similar to that of [15], where the validity of a similar statement was established for problems considered on a finite time interval. To be able to exploit the idea of [15] for problems on the infinite time horizon, we use Theorem 4.4 of [17] establishing that the equality (15) is valid in the case the control set is independent of the state variables.  Let g (u, y) ∈ C (K ) and let def

g¯ (µ, y) =

 U (y)

g (u, y)µ(du).

(16)

Consider the problem



∞

inf

(µ(·),y(·)) 0

def

e−Ct g¯ (µ(t ), y(t ))dt = VC (y0 ),

(17)

where inf is over all relaxed y0 -admissible pairs. Consider also the infinite dimensional (ID) linear programming (LP) problem

 min

γ ∈W (C ,y0 ) K

def

g (u, y)γ (du, dy) = G∗C (y0 ),

(18)

30

V. Gaitsgory, M. Quincampoix / Nonlinear Analysis 88 (2013) 27–41

Corollary 2.3. The optimal value of the optimal control problem (17) and the optimal value of the IDLP problem (18) are related by the equality CVC (y0 ) = G∗C (y0 ).

(19)

r Proof. From the fact that, by definition, Γdis (C , y0 ) is the set of discounted occupational measures generated by all relaxed y0 admissible pairs and from (11) (considered with h(·) = g (·)) it follows that

CVC (y0 ) =

 min

r (C ,y ) γ ∈Γdis 0 K

g (u, y)γ (du, dy) =

The latter imply (19) (due to (15)).

 min

r (C ,y ) ¯ Γdis γ ∈co 0 K

g (u, y)γ (du, dy).

(20)



2.2. Convergence of the sets of occupational measures Let W ⊂ P (K ) be defined by the equation def

W = {γ ∈ P (K ) :



∇φ(y)T f (u, y)γ (du, dy) = 0 ∀φ ∈ C 1 }.

(21)

K

Observe that W can be obtained by formally taking C = 0 in (14). In Theorem 2.1 of [23] it has been established that, in the case U (·) is a constant valued map (that is U (y) = U ∀y ∈ Y ),

¯ Γ (S ), W ) = 0, lim ρH (co

(22)

S →∞

where def



Γ (S ) =

{Γ (S , y0 )}.

(23)

y0 ∈Y

In this section, we use the result of the previous section to show that the convergence (22) remains valid (under certain conditions) for a non-constant valued map U (·) as well. Note that, if the relationships (12) are valid, then by (15),

¯ Γdis (C , y0 ) = W (C , y0 ), co

(24)

and, consequently, by (19) and (20), CVC (y0 ) =

 inf

γ ∈Γdis (C ,y0 ) K

g (u, y)γ (du, dy) = G∗C (y0 ).

(25)

The validity of (22) is established by Theorems 2.5 and 2.6 that are proved (under different sets of assumptions; see below) with the help of the following lemma. Lemma 2.4. Assume the relationships (12) are valid. Then (22) is true if

¯ (C ), W ) = 0, lim ρH (coW

(26)

C →0

where def

W (C ) = ∪y0 ∈Y W (C , y0 ).

(27)

Proof. The proof of the lemma is given in Section 4.1. It is based on (25) and it is similar to the proof of Proposition 6.2 in [17].  Note that the validity of (26) is equivalent to that, for any continuous g (·), lim inf G∗C (y0 ) = G∗ ,

(28)

C →0 y 0 ∈ Y

where G∗C (y0 ) is the optimal value in (18) and G∗ is the optimal value of the following IDLP problem



def

g (u, y)γ (du, dy) = G∗ .

min γ ∈W

(29)

K

A pair (u(t ), y(t )) will be called (y0 , δ)-admissible on [0, ∞) if (1)–(2) are satisfied for almost all t ∈ [0, ∞) and y(t ) ∈ Y + δ B¯ for all t ∈ [0, ∞), where δ > 0 and B¯ is the closed unit ball in Rm . Consider the problem

 inf

(u(·),y(·)) 0

+∞

e−Ct g (u(t ), y(t ))dt = VCδ (y0 ), def

where inf is over all (y0 , δ)-admissible pairs.

(30)

V. Gaitsgory, M. Quincampoix / Nonlinear Analysis 88 (2013) 27–41

31

Assumption 1. For any δ > 0 small enough, the set Y + δ B¯ is viable, and also, for any locally Lipschitz continuous g (·), the function VCδ (·) is continuous on Y + δ B¯ and the function VC (·) is continuous on Y . Assumption 2. (i) The map U (·) is continuous on Y + δ0 B¯ for some δ0 > 0; (ii) For any y¯ ∈ Y + δ0 B (B standing for the open unit ball in Rm ) and for any u¯ ∈ U (¯y), there exists a single-valued continuous selection u(y) ∈ U (y) defined on y¯ + r B¯ (r > 0 may depend on y¯ ) such that u(¯y) = u¯ . Remark. Note that Assumption 1 is satisfied if the set Y is convex and if there exist δ0 > 0 and r > 0 such that r B¯ ∈ f (y, U (y))

∀y ∈ Y + δ0 B¯ .

In fact, as can be readily verified, under these conditions, the value functions VCδ (·) and VC (·) will be Lipschitz continuous on the corresponding sets (see, e.g., p. 101 and Corollary III.2.10 in [24]). Note also that Assumption 2 is satisfied if Assumption 4 introduced below is satisfied (due to a corollary from the Michael selection Theorem; see Corollary 9.1.3 in [22]). Theorem 2.5. If the relationships (12) are valid and Assumptions 1 and 2 are satisfied, then (22) is true. Proof. The proof is given in Section 4.2.



Let us now introduce another set of assumptions. Assumption 3. (i) The set Y is forward invariant with respect to the solutions of the system (1) obtained with the controls satisfying (2). (ii) There exists an open neighborhood N of Y (Y ⊂ N ) such that all solutions of the system (1) obtained with the initial conditions from N and with the controls satisfying (2) do not leave a compact set Yˆ ⊂ Rm . Assumption 4. (i) M is a compact Banach space and U (y) is a convex and closed subset of M for any y ∈ Yˆ , where Yˆ is the compact set from Assumption 1. (ii) U (·) satisfies Lipschitz conditions on Yˆ . That is, dH (U (y′ ), U (y′′ )) ≤ c ∥y′ − y′′ ∥,

∀y′ , y′′ ∈ Yˆ ,

(31)

where dH is the Hausdorff metric generated by a norm in M. Theorem 2.6. If Assumptions 3 and 4 are satisfied, then (22) is true. Proof. The proof is given in Section 4.2.



Remark. As has been mentioned above, the fact that Y is assumed to be forward invariant implies the validity of the relationships (12). Note that, due to the definition of the occupational measures (see (6)), the validity of (22) is equivalent to that, for any continuous g (·), lim inf ΘS (y0 ) = G∗ ,

(32)

S →∞ y0 ∈Y

where def

ΘS (y0 ) =

1

S



g (u(t ), y(t ))dt =

inf

S (u(·),y(·))

0

 inf

γ ∈Γ (S ,y0 ) K

g (u, y)γ (du, dy)

(33)

the minimization being over all y0 -admissible pairs in the first case and over all occupational measures generated by these pairs in the second. ¯ Γdis (C , y0 ) (with C → 0) Let us now introduce an additional assumption, which will allow us to prove that the limit of co ¯ Γ (S , y0 ) (with S → ∞) exist and are equal for any y0 ∈ Y and that W is equal to the closed convex hull and the limit of co of the union of these limits. Assumption 5. Corresponding to any continuous function h : K → R, there exists a function νh : R → R+ with limr →0+ νh (r ) = 0 such that for arbitrary y′0 , y′′0 ∈ Y and for an arbitrary y′0 -admissible pair (u′ (·), y′ (·)), there exists a y′′0 -admissible pair (u′′ (·), y′′ (·)) such that

  1  S

S

h(u′ (t ), y′ (t ))dt − 0

1 S

S

 0

 

h(u′′ (t ), y′′ (t ))dt  ≤ νh (∥y′0 − y′′0 ∥).

(34)

Proposition 2.7. Let Assumptions 3–5 be satisfied. Then, for any y0 ∈ Y , there exists a convex and compact set Γ (y0 ) ⊂ P (K ) such that

¯ Γdis (C , y0 ), Γ (y0 )) = lim ρH (co ¯ Γ (S , y0 ), Γ (y0 )) = 0, lim ρH (co

C →0

S →∞

(35)

32

V. Gaitsgory, M. Quincampoix / Nonlinear Analysis 88 (2013) 27–41

with the convergence being uniform with respect to y0 ∈ Y . Also,

¯ ∪y0 ∈Y Γ (y0 ). W = co

(36)

Proof. Using Assumption 3 with h(·) = g (·), one can readily verify that the optimal value function ΘS (·) defined in (33) satisfies the inequality

|ΘS (y′0 ) − ΘS (y′′0 )| ≤ νg (∥y′0 − y′′0 ∥) ∀S > 0 & ∀ y′0 , y′′0 ∈ Y .

(37)

Using now exactly the same argument as that used in the proof of the corresponding result in [25], one can come to the conclusion that, for any y0 ∈ Y , there exists a limit def

lim ΘS (y0 ) = lg (y0 ),

(38)

S →∞

with the convergence being uniform with respect to y0 ∈ Y . The fact that the convergence in (38) is uniform allows one to conclude (see Theorem 6 in [26]) that, for any y0 ∈ Y , there exists the limit lim CVC (y0 ) = lg (y0 ),

(39)

C →0

with the convergence being also uniform with respect to y0 ∈ Y . The relationships (38) and (39) are valid for any continuous g (·) and hence imply (35) with

   def Γ (y0 ) = γ ∈ P (K ) : ∀g (·) ∈ C (K ), g (u, y)dγ ≤ lg (y0 )

(40)

K

(see, e.g., Section 2.6 in [22]). The fact that (36) is valid can now be readily established on the basis of (22) and (35).



Note, in conclusion, that, in the case U (y) is a constant valued map (U (y) = U), Assumption 3 can be verified to be valid if ∃a ≥ 0 such that

(f (u, y1 ) − f (u, y2 ))T (y1 − y2 ) ≤ −a∥y1 − y2 ∥2 ∀ u ∈ U & ∀ (y1 , y2 ) ∈ Y × Y .

(41)

For a > 0, the above is a well known dissipativity condition and, for a = 0, it is a ‘‘non-expansivity’’ condition introduced in [25]. 3. Proof of Theorem 2.2 We will divide the proof into four steps. (i) Auxiliary relaxed admissible pairs. Let the multivalued function F (·) be defined by the equation def

F (y) =



v:v=

 U (y)

f (u, y)µ(du), µ ∈ P (M ), supp(µ) ⊂ U (y)



∀ y ∈ Y.

(42)

It is easy to verify that F (·) is upper semicontinuous. Hence, its graph def

Graph(F ) = {(v, y) : v ∈ F (y), y ∈ Y }

(43)

is compact. Let D and Q be closed balls in Rm such that Y ⊂ D and F (y) ⊂ Q

∀y ∈ Y ⇒ Graph(F ) ⊂ Q × D.

(44)

Let ν(t , dv) : [0, ∞) → P (Q ) be measurable and let y(t ) satisfy the equation y′ (t ) =



vν(t , dv) for a.e. t > 0; y(0) = y0 .

(45)

Q

The pair (ν(·), y(·)) will be called auxiliary relaxed y0 -admissible if y(t ) ∈ D ∀t ∈ [0, ∞). Given such a pair, let the measure γ ∈ P (Q × D) be defined by the equation     ∞ −Ct q(v, y)γ (dv, dy) = C e q(v, y(t ))ν(t , dv) dt , (46) Q ×D

0

Q

the latter being satisfied for any q ∈ C (Q × D). This measure will be called the discounted occupational measure generated by the pair (ν(·), y(·)) and will be denoted as γ (ν(·),y(·)) .

V. Gaitsgory, M. Quincampoix / Nonlinear Analysis 88 (2013) 27–41

33

Denote by V (C , y0 ) the set of occupational measures generated by all auxiliary relaxed y0 -admissible pairs. That is, def

{γ (ν(·),y(·)) },



V (C , y0 ) =

(47)

(ν(·),y(·))

where the union is over all auxiliary relaxed y0 -admissible pairs. Note that, similarly to Theorem 1.2 in [15], it can be shown that V (C , y0 ) is a compact subset of P (Q × D). Lemma 3.1. The following equality is valid

¯ V (C , y0 ) = Ω (C , y0 ), co

(48)

where

  Ω (C , y0 ) = γ ∈ P (Q × D) : def

(∇φ(y) v + C (φ(y0 ) − φ(y)))γ (dv, dy) = 0 ∀φ ∈ C T

P (Q ×D)

1



.

(49)

Proof. Let q(v, y) : rB × Rm → R1 be Lipschitz continuous and let Vq (C , y0 ) be defined by the equation +∞



def

Vq (C , y0 ) =

e−Ct

inf

(ν(·),y(·)) 0





q(v, y(t ))ν(t , dv) dt ,

(50)

Q

where the inf is over all auxiliary relaxed y0 -admissible pairs. It is easy to verify that the function Vq (C , ·) satisfies Lipschitz conditions on D (see, e.g., Proposition III.2.3 and p. 398 in [24]). This ensures that the conditions of Theorem 4.4 in [17] are satisfied, and, hence, the equality (49) is implied by this theorem.  (ii) Comparison of the optimal values of two infinite dimensional LP problems. Let g (u, y) : M × Rm → R1 be continuous and let q(v, y) : Q × D → [0, ∞] be defined by the equations:



def

q(v, y) =

min

µ∈P (U (y))

U (y)

g (u, y)µ(du)|

 U (y)

f (u, y)µ(du) = v



∀ (v, y) ∈ Graph(F );

(51)

def

q(v, y) = ∞ ∀ (v, y) ∈ (Q × D)/Graph(F ).

(52)

Note that from (51) it follows that q(f (u, y), y) ≤ g (u, y) ∀u ∈ U (y), y ∈ Y .

(53)

It can be shown that the function q(v, y) is lower semicontinuous in (v, y) on Q × D (see the proof of Lemma 3.4). It also can be readily verified that q(v, y) is convex in v for every y ∈ D. Let q∗C (y0 ) be the optimal value of the following IDLP problem



def

q∗C (y0 ) =

min

γ ∈ω(C ,y0 ) Q ×D

q(v, y)γ (dv, dy),

(54)

where def



ω(C , y0 ) = γ ∈ Ω (C , y0 ) :



dist (y, Y )γ (dv, dy) = 0; Q ×D





dist (v, F (y))γ (dv, dy) = 0 .

(55)

Q ×D

Note that the set ω(C , y0 ) is compact, and, hence, an optimal solution of the problem (54) exists. Lemma 3.2. The following inequality is valid q∗C (y0 ) ≤ G∗C (y0 ),

(56)

where GC (y0 ) is the optimal value in (18). ∗

Proof. Take arbitrary γ ∈ W (C , y0 ) (note that W (C , y0 ) ̸= ∅ due to Lemma 2.1 and due to the assumption that there exists r at least one relaxed y0 -admissible pair, which implies that Γdis (C , y0 ) ̸= ∅). Define a measure γ¯ ∈ P (Q × D) as one that satisfies the equation

 Q ×D

def

h(v, y)γ¯ (dv, dy) =



h(f (u, y), y)γ (du, dy)

(57)

K

for any h(·, ·) ∈ C (Q × D). Note that from the fact that (57) is valid for any continuous functions h(·, ·) it follows that it is valid for any lower (upper) semicontinuous h(·, ·) (see Corollary 1.5 in [15] and p. 222 in [27]).

34

V. Gaitsgory, M. Quincampoix / Nonlinear Analysis 88 (2013) 27–41

By definition (see (5) and (43)), ∪(u,y)∈K (f (u, y), y) ∈ Graph(F ). Hence,

γ¯ ((Q × D)/Graph(F )) =



1(Q ×D)/Graph(F ) (f (u, y), y)γ (du, dy) = 0 K

⇒ supp(γ¯ ) ⊂ Graph(F ). The latter implies that



dist (y, Y )γ¯ (dv, dy) = 0,



Q ×D

dist (v, F (y))γ¯ (dv, dy) = 0.

(58)

Q ×D

Also, by (57),



(∇φ(y)T v + C (φ(y0 ) − φ(y)))γ¯ (dv, dy)  = (∇φ(y)T f (u, y) + C (φ(y0 ) − φ(y)))γ (du, dy) = 0 ∀φ ∈ C 1 .

Q ×D

K

Consequently (see (49) and (55)),

γ¯ ∈ ω(C , y0 ).

(59)

Since the function q(v, y) is lower semicontinuous, one obtains that



q(v, y)γ¯ (dv, dy) =



Q ×D

q(f (u, y), y)γ (du, dy) ≤



K

g (u, y)γ (du, dy), K

the second equality being due to (53). From (59) it follows that q∗C (y0 ) ≤



g (u, y)γ (du, dy) ∀γ ∈ W (C , y0 ). K

This proves (56).



(iii) Two technical lemmas. Lemma 3.3. There exists a solution of the differential inclusion y′ (t ) ∈ F (y(t )),

y(0) = y0

(60)

such that y(t ) ∈ Y

∀t ∈ [0, ∞)

(61)

and ∞



e−Ct q(y′ (t ), y(t ))dt ≤ q∗C (y0 ).

C

(62)

0

Proof. The set ω(C , y0 ) is not empty (since W (C , y0 ) is not empty; see the proof of Lemma 3.2). Also, as has been mentioned above, ω(C , y0 ) is compact and, hence, the set of optimal solutions of the IDLP problem (54) is not empty. Let γ ∗ be one of such optimal solutions. That is, γ ∗ ∈ Ω (C , y0 ),



dist (y, Y )γ ∗ (dv, dy) = 0,



Q ×D

dist (v, F (y))γ ∗ (dv, dy) = 0

(63)

Q ×D

and



q(v, y)γ ∗ (dv, dy) = q∗C (y0 ).

(64)

Q ×D

Using the validity of (48), one can establish (similarly to Corollaries 1.4 and 1.5 in [15]) that there exists a measure η ∈ P (V (C , y0 )) such that for every continuous (and lower semicontinuous) function h(v, y) : Q × D → R1 ,



h(v, y)γ (dv, dy) = ∗

Q ×D

 V (C ,y0 )



 h(v, y)γ (dv, dy) η(dγ ). Q ×D

(65)

V. Gaitsgory, M. Quincampoix / Nonlinear Analysis 88 (2013) 27–41

35

Hence, (63) and (64) can be rewritten as



 V (C ,y0 )



V (C ,y0 )

Q ×D



 V (C ,y0 )





dist (y, Y )γ (dv, dy) η(dγ ) = 0,



dist (v, F (y))γ (dv, dy) η(dγ ) = 0

(66)

Q ×D

 q(v, y)γ (dv, dy) η(dγ ) = q∗C (y0 ).

(67)

Q ×D

Denote by V ′ a subset of V (C , y0 ) such that





dist (y, Y )γ (dv, dy) = 0, Q ×D

dist (v, F (y))γ (dv, dy) = 0.

(68)

Q ×D

Note that, since V ′ ⊂ V (C , y0 ) ⊂ Ω (C , y0 ),

V ′ ⊂ ω(C , y0 ).

(69)

Due to the fact that dist (·, ·) ≥ 0, the validity of equalities (66) implies that

 

η(V ) = 1 ⇒ ′

V′

 q(v, y)γ (dv, dy) η(dγ ) = q∗C (y0 ).

(70)

Q ×D

Also, by (69),



q(v, y)γ (dv, dy) ≥ q∗C (y0 ) ∀γ ∈ V ′ .

(71)

Q ×D

Let V ′′ ⊂ V ′ be such that



q(v, y)γ (dv, dy) = q∗C (y0 )

∀γ ∈ V ′′ .

(72)

Q ×D

By (70) and (71), η(V ′′ ) = 1. Take arbitrary γ ∈ V ′′ . Since V ′′ ⊂ V (C , y0 ), γ is a discounted occupational measure generated by some auxiliary relaxed y0 -admissible pair (ν(·), y(·)). That is, γ = γ (ν(·),y(·)) . Now the fact that γ satisfies (68) is equivalent to the validity of the following equations (see (46)): ∞

 0=

e

−Ct



0

dist (y(t ), Y )ν(t , dv) dt = Q

∞





−Ct

0=



e−Ct dist (y(t ), Y )dt ⇒ y(t ) ∈ Y

∀t ≥ 0;

0



dist (v, F (y(t )))ν(t , dv) dt ⇒ supp ν(t , dv) ∈ F (y(t )),

e 0

∞



for a.e. t ≥ 0.

(73)

Q

Also, from (72) it follows that ∞



e−Ct

C



0



q(v, y(t )) ν(t , dv)dt = q∗C (y0 ).

(74)

Q

From the convexity of F (·) and from (73) (see also (45)) it follows that y (t ) = ′



vν(t , dv) ∈ F (y(t )) for a.e. t ∈ [0, ∞). Q

That is, y(t ) is a solution of the differential inclusion (60). Also, due to the convexity of q(·, y), ∞



e−Ct q(y′ (t ), y(t ))dt = 0

∞



e−Ct q 0

This and (74) imply the validity of (62).



∞

  vν(t , dv), y(t ) dt ≤ Q

e−Ct 0





q(v, y(t )) ν(t , dv)dt . Q



Lemma 3.4. There exists a relaxed y0 -admissible pair (µ(·), y(·)) such that ∞



e−Ct

C 0

 U (y(t ))

g (u, y(t ))µ(t , du)dt ≤ q∗C (y0 ).

(75)

36

V. Gaitsgory, M. Quincampoix / Nonlinear Analysis 88 (2013) 27–41

Proof. For (v, y) ∈ Graph(F ), let

  µ ∈ P (M ) : supp(µ) ⊂ U (y),

def

A(v, y) =

U (y)

f (u, y)µ(du) = v

 (76)

and let D (v, y) be the set of solutions of the problem defined in the right-hand side of (51). That is, def

D (v, y) =

  µ ¯ ∈ A(v, y) :

  = µ ¯ ∈ A(v, y) :

U (y)

g (u, y)µ( ¯ du) =

 min

µ∈A(v,y) U (y)



g (u, y)µ(du)



U (y)

g (u, y)µ( ¯ du) = q(v, y) .

(77)

Note that D(v, y) is not empty and closed since A(v, y) is not empty and closed. The fact that A(v, y) is closed is verified as follows. Assume that µl ∈ A(v, y) and that µl → µ (convergence being weak∗ in P (M )). To show that µ ∈ A(v, y), it is enough to show that supp(µ) ⊂ U (y), which is implied by the following relationships: 1 = lim µl (U (y)) ≤ µ(U (y)) ⇒ µ(U (y)) = 1. l

(78)

Let us now prove that the map (v, y) → D (v, y) has a measurable selection on Graph(F ). To prove this, it is sufficient to establish that D (·, ·) is measurable as a set-valued map with closed nonempty images (see Theorem 8.1.3 in [22]). Note that def

the function h(v, y, µ) = M g (u, y)µ(du) is continuous on Graph(F ) × P (M ). Hence, by the theorem about measurability of the marginal map (see Theorem 8.2.11 in [22]), to prove the measurability of D (·, ·), it is enough to prove that A(·, ·) is measurable, which, in turn, will be established if one shows that A(·, ·) is upper semicontinuous on Graph(F ). Take arbitrary sequences (vl , yl ) ∈ Graph(F ) and µl ∈ A(vl , yl ) converging to some (v, y) ∈ Graph(F ) and to µ: vl → v, yl → y, µl ⇀ µ. The upper semicontinuity of A(·, ·) will be established if one shows that µ ∈ A(v, y). Since U is upper semicontinuous, corresponding to any δ > 0, there exists N such that



U (yl ) ⊂ U (y) + δ B ∀l ≥ N ⇒ µl (U (y) + δ B) = 1 ∀l ≥ N , where B is a closed unit ball in M. Similarly to (78), one can obtain that 1 = lim µl (U (y) + δ B) ≤ µ(U (y) + δ B) ⇒ µ(U (y) + δ B) = 1 ⇒ µ(U (y)) = 1, l

the latter equality being due to the fact that δ can be arbitrary small. Also, since f is continuous, we have

   v = lim vl = lim f (u, yl )µl (du) = lim f (u, yl )µl (du) = lim f (u, y)µl (du) l l l l U (yl ) M M   = f (u, y)µ(du) = f (u, y)µ(du). U (y)

M

Thus µ ∈ A(v, y), and the map D (·, ·) is measurable on Graph(F ). Let µ ¯ (v,y) ∈ D (v, y) be a measurable selection of D (v, y) for (v, y) ∈ Graph(F ). Note that, by definition,

 U (y)

f (u, y)µ ¯ (v,y) (du) = v,

 U (y)

g (u, y)µ ¯ (v,y) (du) = q(v, y).

(79)

Take now a solution y(t ) of the differential inclusion (60) that satisfies (61) and (62), and take def

µ( ¯ t) = µ ¯ (y′ (t ),y(t )) .

(80)

Then, by (79),

 U (y(t ))

f (u, y(t ))µ( ¯ t , du) = y′ (t ),

 U (y(t ))

g (u, y(t ))µ( ¯ t , du) = q(y′ (t ), y(t )).

(81)

The first equality and (61) imply that the pair (µ(·), y(·)) is relaxed y0 -admissible, while the second one and (62) imply the validity of (75).  (IV) The final step of the proof. From Lemma 3.4 it follows that

 inf

r (C ,y ) γ ∈Γdis 0 K

g (u, y)γ (du, dy) ≤ q∗C (y0 ).

(82)

V. Gaitsgory, M. Quincampoix / Nonlinear Analysis 88 (2013) 27–41

37

Hence, by (56),

 inf

r (C ,y ) γ ∈Γdis 0 K

g (u, y)γ (du, dy) ≤ G∗C (y0 ).

(83)

Since it is valid for any continuous function g (u, y), it follows that r ¯ Γdis W (C , y0 ) ⊂ co (C , y0 ),

(84)

which together with (13) prove (15).



4. Proofs for Section 2.2 4.1. Proof of Lemma 2.4 To prove (22), it is enough to show that (28) implies (32). Note that from (28) (and from (25)) it follows that, for any sequence Ci → 0, there exist a sequence of yi0 ∈ Y and a sequence of yi0 -admissible pairs (ui (·), yi (·)) such that lim ζi = 0,

ζi = Ci

i→∞

+∞



def

e−Ci t g (ui (t ), yi (t ))dt − G∗ .

(85)

0

From Lemma 3.5(ii) in [28] it follows that there exists a sequence Si , i = 1, 2, . . . , such that Si ≥ √KC (K > 0 being a i

constant) and such that 1

Si



Si

g (ui (t ), yi (t ))dt ≤ G∗ + ζi +



Ci

0

⇒ inf ΘSi (y0 ) ≤ ΘSi (yi0 ) ≤ G∗ + ζi +



y0 ∈Y

Ci

⇒ limS →∞ inf ΘS (y0 ) ≤ G∗ .

(86)

y0 ∈Y

Since (as can be readily verified) the inequality limS →∞ inf ΘS (y0 ) ≥ G∗ , y0 ∈Y

is valid too, (86) implies that limS →∞ inf ΘS (y0 ) = G∗ .

(87)

y0 ∈Y

The latter implies, in turns, that def

lim ηi = 0,

ηi =

i→∞

1 Si

Si



g (ui (t ), yi (t ))dt − G∗ .

(88)

0

√

From Lemma 3.8 in [29] it follows that, for any i, there exists a non-negative ti ≤ Si − 1 S

S

 0

1 g (ui (ti + t ), yi (ti + t ))dt ≤ G∗ + ηi + √ Si

def

Si L

(L > 0 being a constant) such that

 √  Si ∀S ∈ 0 , .

(89)

L

def

Let u˜ i (·) = ui (ti + ·), y˜ i (·) = yi (ti + ·). Note that (˜ui (·), y˜ i (·)) is a yi (ti )-admissible pair. Hence, by (89), inf ΘS (y0 ) ≤

y0 ∈Y

1 S



S

1

g (˜u (t ), y˜ (t ))dt ≤ G + ηi + √ i

i

0

∗

Si



∀S ∈ 0,

√  Si

L

⇒ limS →∞ inf ΘS (y0 ) ≤ G∗ . y0 ∈Y

The latter and (87) prove (32).



4.2. Proofs of Theorems 2.5 and 2.6 We will start with the proof of Theorem 2.6, which is based on two lemmas stated and proved below. Lemma 4.1. Let Assumptions 3 and 4 be satisfied. Then, for any locally Lipschitz continuous function g (u, y), the value of the objective function VC (·) (see (17)) is continuous on N .

38

V. Gaitsgory, M. Quincampoix / Nonlinear Analysis 88 (2013) 27–41

Proof. The proof of the lemma follows exactly the same lines as the proof of Proposition 2.1 on p. 99 in [24]. Note that the above cited result is established under the assumption that the function g (u, y) is bounded. In our case it is not needed since, by Assumption 1, the trajectories that start in N do not leave a compact set Yˆ . Also the result in [24] is obtained for a constant valued map U (y) = U. However, the extension to the case when U (·) satisfies Lipschitz condition (31) (see Assumption 4) is straightforward.  Consider the Hamilton–Jacobi–Bellman (HJB) equation C α + H (y, Dα) = 0,

(90)

where def

H (y, ξ ) = max {−ξ T f (u, y) − g (u, y)}.

(91)

u∈U (y)

Lemma 4.2. Let Assumptions 3 and 4 be satisfied. Then, for any locally Lipschitz continuous function g (u, y), the value function VC (·) is a viscosity subsolution of the HJB (90) in N . Proof. Take an arbitrary y0 ∈ N and choose r > 0 in such a way that y0 + r B¯ ⊂ N , where B¯ is a closed unit ball in Rm . Since the map y → f (y, U (y)) is continuous and bounded on cl(N ), there exists h¯ > 0 such that the solution of the system (1) obtained with the initial condition y(0) = y0 and with any control u(·) satisfying (2) (denote such a solution as y(t , y0 , u(·))) will be contained in y0 + r B¯ for all t ∈ [0, h¯ ]. That is, y(t , y0 , u(·)) ∈ y0 + r B¯ ∀t ∈ [0, h¯ ]. Let now ϕ(·) be a differentiable function on Rm such that

ϕ(y0 ) = VC (y0 ) and ϕ(y) ≥ VC (y) ∀y ∈ y0 + r B¯ . Take an arbitrary u¯ ∈ U (y0 ). From a consequence of the Michael selection Theorem it follows (see Corollary 9.1.3 in [22]) that there exists a continuous single-valued selection y → uˆ of U (·) defined on y0 + r B¯ such that uˆ (y0 ) = u¯ . Let y¯ (·) be a solution of the system y¯ ′ (t ) = f (ˆu(¯y(t )), y¯ (t )) that satisfies the initial condition y¯ (0) = y0 . Let ∆ ∈ (0, h¯ ) and let y(t ) be a solution of the system y′ (t ) = f (u(t ), y(t )) that satisfies the initial condition y¯ (0) = y¯ (∆), with u(·) being an arbitrary control satisfying the inclusion u(t ) ∈ U (y(t )). Define u˜ (t ) and y˜ (t ) by the equations u˜ (t ) := y˜ (t ) :=

uˆ (¯y(t )) if t ∈ [0, ∆), u(t − ∆) if t ≥ ∆;

 

y¯ (t ) if t ∈ [0, ∆), y(t − ∆) if t ≥ ∆.

As can be readily seen, the following relationships are valid:

 +∞ e−Ct g (˜y(t ), u˜ (t ))dt ϕ(y0 ) = VC (y0 ) ≤ 0  ∆  +∞ −Ct = e g (˜u(t ), y˜ (t ))dt + e−Ct g (˜u(t ), y˜ (t ))dt ∆

0

(and, by a change of variable) ∆

 =

e−Ct g (ˆu(¯y(t )), y¯ (t ))dt + e−C ∆

+∞



e−Ct g (u(t ), y(t ))dt . 0

0

Since u(·) and y(·) are arbitrary, the latter imply (by taking inf over u(·) and y(·)) that

ϕ(y0 ) ≤

∆



e−Ct g (ˆu(¯y(t )), y¯ (t ))dt + e−C ∆ VC (¯y(∆)) ≤

∆



e−Ct g (ˆu(¯y(t )), y¯ (t ))dt + e−C ∆ ϕ(¯y(∆)).

0

0

Thus, 1

∆

(ϕ(y0 ) − e−C ∆ ϕ(¯y(∆))) ≤

1

∆

∆



e−Ct g (ˆu(¯y(t )), y¯ (t ))dt ,

0

and, by letting ∆ → 0+ , one can obtain that

−∇ϕ(y0 ).f (¯u, y0 ) + C ϕ(y0 ) ≤ g (¯u, y0 , ). Since u¯ is an arbitrary element of U (y0 ), we infer that max {−∇ϕ(y0 ).f (y0 , u) − g (y0 , u)} + C ϕ(y0 ) ≤ 0.

u∈U (y0 )

That is, VC satisfies the viscosity subsolution inequality at y0 . The proof of the lemma is complete.



V. Gaitsgory, M. Quincampoix / Nonlinear Analysis 88 (2013) 27–41

39

Proof of Theorem 2.6. By Lemma 2.4, to prove the theorem, it is sufficient to show that (26) is valid. As can be readily seen, limC →0 W (C ) ⊂ W ,

(92)

which (due to the fact that W is convex and compact) implies that max ρ(γ , W ) = 0.

lim

(93)

C →0 γ ∈coW ¯ (C )

Thus, to establish the validity of (26), it is enough to show that

¯ (C ). W ⊂ coW

(94)

The latter, in turn, will be established if



g (u, y)γ (du, dy) ≤ min

min



γ ∈W

¯ (C ) γ ∈coW

g (u, y)γ (du, dy) = G∗

(95)

for any locally Lipschitz continuous function g (u, y). By the definitions of G∗C (·) and W (C ) (see (18) and (92)),



g (u, y)γ (du, dy) = inf

min

¯ (C ) γ ∈coW

 min

y∈Y γ ∈W (C ,y)

g (u, y)γ (du, dy) = inf G∗C (y). y0 ∈Y

Hence, due to (19), the inequality (95) is equivalent to C min VC (y) ≤ G∗ ,

(96)

y∈Y

which we are going to prove below. Denote by VCϵ (·) the ϵ sup-convolution of VC (·) defined as follows



VCϵ (x) = sup VC (y) − def

y∈N

1 2ϵ

 ∥x − y∥2 ,

(97)

where ϵ > 0 is a small parameter. The function VCϵ (·) is semiconvex on N and it converges to VC (·) as ϵ → 0, with the convergence being uniform on any compact subset of N and, in particular, on Y (see Lemma 4.11, p. 72 in [24]). That is, max |VC (y) − VCϵ (y)| ≤ ν1 (ϵ), y∈Y

lim ν1 (ϵ) = 0.

ϵ→0

(98)

By Lemma 4.2, the function VC (·) is a viscosity subsolution of the HJB (90). Also, it can be readily verified that, due to the continuity of the map U (·) (as well as the continuity of the function f (·, ·)) the inequality

|H (x, ξ ) − H (y, ξ )| ≤ ω(∥x − y∥(1 + ∥ξ ∥))

(99)

is satisfied for all x, y ∈ cl(N ) and for all ξ ∈ R , where ω(θ ) → 0 when θ → 0. This implies that all conditions of Proposition 4.13 of [24] (see p. 74 in [24]) are satisfied, and, on the basis of the latter, one concludes that the function VCϵ (·) is a viscosity subsolution of the equation m

C α + H (y, Dα) = ρϵ (y)

(100)

in N , where ρϵ (y) tends to zero with ϵ tending to zero (uniformly with respect to y from any compact subset of N ). Being semiconvex, the function VCϵ (·) satisfies local Lipschitz conditions in N (Proposition 4.6, p. 66 in [24]). Hence, by Rademacher’s theorem, it is differentiable almost everywhere in N and from the fact that VCϵ (·) is a subsolution of (100) it follows that CVCϵ (y) + H (y, ∇ VCϵ (y)) ≤ ρϵ (y) a.e. in N .

(101)

For any y ∈ N , let DVC ϵ (y) = {ξ ∈ Rm : ξ = lim ∇ VCϵ (y′ )}. def

Y

y ′ →y

ϵ

ϵ

Note that coDVC (y) = ∂ VC (y) (Clarke’s generalized gradient). Let also def

ρˆ ϵ (y) = lim sup ρϵ (y′ ). y′ →y

From (101) it follows that CVCϵ (y) + H (y, ξ ) ≤ ρˆ ϵ (y) ∀ξ ∈ DVCϵ (y), ∀y ∈ N

⇒ CVCϵ (y) + max H (y, ξ ) ≤ ρˆ ϵ (y) ∀y ∈ N ϵ ξ ∈DVC (y)

ϵ

⇒ CVC (y) + max H (y, ξ ) ≤ ρˆ ϵ (y) ∀y ∈ N , ϵ ξ ∈∂ VC (y)

(102)

40

V. Gaitsgory, M. Quincampoix / Nonlinear Analysis 88 (2013) 27–41

the latter inequality being true due to the fact that H (y, ξ ) is convex in ξ . Taking into account the definition of the Hamiltonian (91), one can continue as follows CVCϵ (y) − min

min {ξ T f (u, y) + g (u, y)} ≤ ρˆ ϵ (y) ∀y ∈ N

ξ ∈∂ VCϵ (y) u∈U (y)

⇒ min CVCϵ (y) ≤ min min

min {ξ T f (u, y) + g (u, y)} + sup ρˆ ϵ (y).

y∈Y u∈U (y) ξ ∈∂ V ϵ (y)

y∈Y

(103)

y∈Y

C

Since min min

min {ξ T f (u, y) + g (u, y)} ≤ sup min min

y∈Y u∈U (y) ξ ∈∂ V ϵ (y)

min {ξ T f (u, y) + g (u, y)},

ψ(·)∈Lip y∈Y u∈U (y) ξ ∈∂ψ(y)

C

where sup is over all locally Lipschitz continuous functions ψ(·), and since by Lemma 3.2 in [17] and Theorem 4.1(ii) in [16] (see also Theorem 3.1(ii) in [17]),1 sup min min

min {ξ T f (u, y) + g (u, y)} = G∗ ,

ψ(·)∈Lip y∈Y u∈U (y) ξ ∈∂ψ(y)

from (103) it follows that min CVCϵ (y) ≤ G∗ + sup ρˆ ϵ (y). y∈Y

(104)

y∈Y

As was mentioned above, ρϵ (y) tends to zero with ϵ tending to zero uniformly in y from any compact subset of N . This implies that ρˆ ϵ (y) defined in (102) also tends to zero with ϵ tending to zero uniformly in y from any compact subset of N . Hence, lim sup ρˆ ϵ (y) = 0.

ϵ→0 y∈Y

Having this and (98) in mind, one can prove (96) by passing to the limit in (104) with ϵ → 0.



Proof of Theorem 2.5. As in the proof of Theorem 2.6, it is sufficient to prove that (96) is valid. Due to the validity of Assumptions 1 and 2, one can use the same argument as in the proof of Lemma 4.2 to establish that VCδ (·) is a viscosity subsolution of the HJB (90) for every δ > 0 small enough. Note that, due to continuity of U (·), the inequality (99) is valid. Hence, proceeding as in the proof of Theorem 2.6 with the replacement of VC (·) by VCδ (·), one can establish that C min VCδ (y) ≤ G∗

(105)

y∈Y

for every δ > 0 small enough. From the validity of the relationships (12) it follows that lim VCδ (y0 ) = VC (y0 ) ∀y0 ∈ Y .

(106)

δ→0

The fact that the relationships (12) imply (106) has been established in [17] for constant valued U (·) (see Lemma 4.3 in [17]). The proof of this fact for continuous U (·) follows similar lines. δ δ Since VC 1 (y) ≥ VC 2 (y) ∀y ∈ Y if δ1 < δ2 and since VCδ (·) and VC (·) are continuous on Y , the convergence in (106) is uniform on Y (by Dini’s theorem). Hence, lim | min VCδ (y) − min VC (y)| = 0.

δ→0

y∈Y

y∈Y

Consequently, passing to the limit in (105) with δ → 0, one proves (96).



Acknowledgments The work of the first author was partially funded by the Australian Research Council Discovery-Project Grants DP0664330, DP120100532, and DP130104432 and by the Linkage International Grant LX0560049. The second author was partially supported by project SADCO, FP7-PEOPLE-2010-ITN, No. 264735 and he was also supported partially by the French National Research Agency ANR-10-BLAN 0112. References [1] A.G. Bhatt, V.S. Borkar, Occupation measures for controlled markov processes: characterization and optimality, Annals of Probability 24 (1996) 1531–1562. [2] W.H. Fleming, D. Vermes, Convex duality approach to the optimal control of diffusions, SIAM Journal on Control and Optimization 27 (5) (1989) 1136–1155. 1 Note that the proofs of the above cited results in [17,16] are given for a constant valued map U (y) = U; the proofs of these results in the general case when the map U (·) is upper semicontinuous are exactly the same.

V. Gaitsgory, M. Quincampoix / Nonlinear Analysis 88 (2013) 27–41

41

[3] T.G. Kurtz, R.H. Stockbridge, Existence of Markov controls and characterization of optimal markov controls, SIAM Journal on Control and Optimization 36 (2) (1998) 609–653. [4] R.H. Stockbridge, Time-Average Control of a Martingale Problem. Existence of a Stationary Solution, Annals of Probability 18 (1990) 190–205. [5] R.H. Stockbridge, Time-average control of a martingale problem: a linear programming formulation, Annals of Probability, 18. [6] V. Borkar, A. Budhiraja, Ergodic control for constrained diffusions: characterization using HJB equations. (English summary), SIAM Journal on Control and Optimization 43 (4) (2004/05) 1467–1492. [7] R. Buckdahn, D. Goreac, M. Quincampoix, Stochastic optimal control and linear programming approach, Applied Mathematics and Optimization 63 (2) (2011) 257–276. [8] F. Dufour, R. Stockbridge, On the existence of strict optimal controls for constrained, controlled Markov processes in continuous-time, Stochastics 84 (1) (2012) 55–78. [9] D. Goreac, O.-S. Serea, A note on linearization methods and dynamic programming principles for stochastic discontinuous control problems, Electronic Communications in Probability 17 (12) (2012). [10] D. Hernandez-Hernandez, O. Hernandez-Lerma, M. Taksar, The linear programming approach to deterministic optimal control problems, Applicationes Mathematicae 24 (1) (1996) 17–33. [11] O. Hernandez-Lerma, J.B. Lasserre, Markov Chains and Invariant Probabilities, Birkhauser-Verlag, Basel, 2003. [12] R. Klötzler, On a general conception of duality in optimal control, in: Lecture Notes in Math., vol. 703, Springer-Verlag, New York, 1979, pp. 189–196. [13] J.B. Lasserre, D. Henrion, C. Prieur, E. Trélat, Nonlinear optimal control via occupation measures and LMI-relaxations, SIAM Journal on Control and Optimization 47 (2008) 1643–1666. [14] J.E. Rubio, Control and Optimization. The Linear Treatment of Nonlinear Problems, Manchester University Press, Manchester, 1985. [15] R. Vinter, Convex duality and nonlinear optimal control, SIAM Journal on Control and Optimization 31 (2) (1993) 518–538. [16] L. Finlay, V. Gaitsgory, I. Lebedev, Duality in linear programming problems related to deterministic long run average problems of optimal control, SIAM Journal on Control and Optimization 47 (2008) 1667–1700. [17] V. Gaitsgory, M. Quincampoix, Linear programming approach to deterministic infinite horizon optimal control problems with discounting, SIAM Journal on Control and Optimization 48 (4) (2009) 2480–2512. [18] V. Gaitsgory, S. Rossomakhine, Linear programming approach to deterministic long run average problems of optimal control, SIAM Journal on Control and Optimization 44 (6) (2005/2006) 2006–2007. [19] D. Goreac, O.-S. Serea, Linearization techniques for L∞ -control problems and dynamic programming principles in classical and L∞ control problems, ESAIM: Control, Optimization and Calculus of Variations (2011) http://dx.doi.org/10.1051/cocv/201183. [20] M. Quincampoix, O. Serea, The problem of optimal control with reflection studied through a linear optimization problem stated on occupational measures, Nonlinear Anal. 72 (6) (2010) 2803–2815. [21] H. Frankowska, F. Rampazzo, Filippov’s and Filippov–Wazewski’s theorems on closed domains, Journal of Differential Equations 161 (2) (2000) 449–478. [22] J.P. Aubin, H. Frankowska, Set-Valued Analysis, Birkhauser, 1990. [23] V. Gaitsgory, On representation of the limit occupational measures set of a control systems with applications to singularly perturbed control systems, SIAM Journal on Control and Optimization 43 (1) (2004) 325–340. [24] M. Bardi, I. Capuzzo-Dolcetta, Optimal control and viscosity solutions of Hamilton–Jacobi–Bellman equations, in: Systems and Control: Foundations and Applications, Birkhäuser, Boston, 1997, xvii, 570 pp. [25] M. Quincampoix, J. Renault, On the existence of a limit value in some non expansive optimal control problem, SIAM Journal on Control and Optimization 49 (5) (2011) 2118–2132. [26] M. Oliu-Barton, G. Vigeral, A uniform Tauberian Theorem in Optimal Control, in: Advances in Dynamic Game Theory and Applications (Annals of ISDG), vol. 12, Birkhauser, 2012. [27] R.B. Ash, Measure, Integration and Functional Analysis, Academic Press, New York, 1972. [28] L. Grüne, Asymptotic controllability and exponential stabilization of nonlinear control systems at singular points, SIAM Journal on Control and Optimization 36 (5) (1998) 1485–1503. [29] L. Grüne, On the relation between discounted and average optimal value functions, Journal of Differential Equations 148 (1998) 65–99.