Bilinear Optimal Control of the Fokker-Planck Equation*

2nd IFAC Workshop on Control of Systems Governed 2nd IFAC Workshop on Control of Systems Governed by Partial Differentialon Equations 2nd IFAC Workshop Control of Systems Governed by Partial Differentialon Equations 2nd IFAC Workshop Control Governed June 13-15, 2016. Bertinoro, Italyof Systems by Partial Differential Equations Available online at www.sciencedirect.com June 13-15, 2016. Bertinoro, Italy by Partial Differential Equations June 13-15, 2016. Bertinoro, Italy June 13-15, 2016. Bertinoro, Italy

ScienceDirect

IFAC-PapersOnLine 49-8 (2016) 254–259

Bilinear Optimal Control Bilinear Optimal Control Bilinear Optimal Control  Bilinear Optimal Control  of the Fokker-Planck Equation of the Fokker-Planck Equation   of the Fokker-Planck Equation of the Fokker-Planck Equation ∗ ∗∗

Arthur Arthur Fleig Fleig ∗∗ Roberto Roberto Guglielmi Guglielmi ∗∗ ∗∗ Arthur Fleig Roberto Guglielmi ∗ Arthur Fleig Roberto Guglielmi ∗∗ ∗ Department of Mathematics, University of Bayreuth, Germany, ∗ Department of Mathematics, University of Bayreuth, Germany, ∗ Department of Mathematics, University of Bayreuth, Germany, (e-mail: [email protected]). ∗ Department of (e-mail: [email protected]). Mathematics, University of Bayreuth, Germany, ∗∗ Radon (e-mail: [email protected]). for Computational and ∗∗ Radon Institute for Computational and Applied Applied Mathematics, Mathematics, Austria Austria (e-mail: [email protected]). ∗∗ Radon Institute for Computational and Applied (e-mail: [email protected]) ∗∗ Radon Institute (e-mail: [email protected]) Institute for Computational and Applied Mathematics, Mathematics, Austria Austria (e-mail: (e-mail: [email protected]) [email protected])

Abstract: Abstract: The The optimal optimal tracking tracking problem problem of of the the probability probability density density function function of of aa stochastic stochastic process process can can Abstract: The optimal tracking problem of the probability density function of a stochastic process be expressed in terms of an optimal bilinear control problem for the Fokker-Planck equation, with the be expressed terms oftracking an optimal bilinear control problem for the Fokker-Planck equation, with can the Abstract: Theinoptimal problem of the probability density function of a stochastic process can be expressed in terms of an optimal bilinear control problem for the Fokker-Planck equation, with the control in the coefficient of the divergence term. We analyze the case of time and space dependent control in theincoefficient of optimal the divergence analyzeforthe of time andequation, space dependent be expressed terms of an bilinear term. controlWe problem thecase Fokker-Planck with the control coefficient the term. analyze the time dependent controls. order to of solutions state require suitable controls. Inthe order to deduce deduceof existence of nonnegative nonnegative solutions for thecase stateof equation wespace require suitable control in inIn the coefficient ofexistence the divergence divergence term. We We analyzefor thethe case ofequation time and andwe space dependent controls. In order to deduce existence of nonnegative solutions for the state equation we require integrability assumptions on the coefficients of the Fokker-Planck equation and thus on the control integrability assumptions the coefficients of the solutions Fokker-Planck and thus on the suitable control controls. In order to deduceonexistence of nonnegative for theequation state equation we require suitable integrability assumptions on the coefficients of the Fokker-Planck equation and thus on the function. Furthermore, we establish the existence of optimal controls and we derive the associated first function. Furthermore, we on establish the existence of optimal controlsequation and we derive the associated first integrability assumptions the coefficients of the Fokker-Planck and thus on the control control function. Furthermore, we establish the existence of optimal controls and we derive the associated order necessary optimality conditions. order necessary optimality conditions. function. Furthermore, we establish the existence of optimal controls and we derive the associated first first order necessary optimality order necessary optimality conditions. conditions. © 2016, IFAC (International Federation of Automatic Control) Hosting by Elsevier Ltd. All rights reserved. Keywords: Keywords: control control system system analysis, analysis, optimal optimal control, control, bilinear bilinear control, control, Fokker-Planck Fokker-Planck equation, equation, Keywords: control system analysis, optimal control, bilinear control, stochastic control stochastic Keywords:control control system analysis, optimal control, bilinear control, Fokker-Planck Fokker-Planck equation, equation, stochastic stochastic control control 1. Assuming the 1. INTRODUCTION INTRODUCTION Assuming for for simplicity simplicity that that the state state variable variable X Xtt evolves evolves in in d with 1. INTRODUCTION Assuming for simplicity that the state variable X evolves in a bounded domain Ω of R smooth boundary, we define d t a bounded for domain Ω of Rthat with smooth boundary, we define 1. INTRODUCTION Assuming simplicity the state variable X evolves in t d with smooth boundary, aaQ bounded domain Ω of R we define := Ω × (0, T ), Σ := ∂ Ω × (0, T ), and a := σ σ /2, i, j d i j ik k j The study of the Fokker-Planck (FP) equation, also known as Q bounded := Ω × (0, T ), Σ Ω :=of ∂ ΩR× with (0, T ), and aiboundary, /2,define i, j = = domain smooth j := σik σk jwe The study of the Fokker-Planck (FP) equation, also known as 1, Q Ω (0, ), (0, T σ σthe i, .. .. d, where and in use Einstein j := k j /2, The the (FP) equation, also as Kolmogoroff equation, received great and increasd, where here and∂∂ Ω in× the following we the Einstein Q ..:= := Ω× × (0, T T here ), Σ Σ := := Ω ×the (0,following T ), ), and and aaiiwe σik i, jj = = j :=use Kolmogoroff forward equation, has has received great andknown increasik σ k j /2, The study study of of forward the Fokker-Planck Fokker-Planck (FP) equation, also known as 1, 1, .. .. .. d, here following use summation convention. We denote by the partial Kolmogoroff forward equation, has great ing interest the by (1931), summation convention. Wethe denote by ∂∂iiwe and theEinstein partial d, where where here and and in in the following weand use∂∂ttthe the Einstein ing interest starting starting from the work work by Kolmogoroff Kolmogoroff (1931), 1, Kolmogoroff forwardfrom equation, has received received great and and increasincreassummation convention. We ∂∂i and ∂t where the derivative respect and respectively, ii = ing starting from the work (1931), owing to of derivative with with respect to to and t, t, by respectively, = convention. Wexxiidenote denote by the partial partial i and ∂t where owing to its its relation relation with the description of the the time time evolution evolution ing interest interest starting with fromthe thedescription work by by Kolmogoroff Kolmogoroff (1931), summation derivative with respect to x and t, respectively, where 1, . . . , d. i owing to its relation with the description of the time evolution of the Probability Density Function (PDF) of the velocity of a 1, . . . , d. with respect to xi and t, respectively, where ii = = of the Probability Density Function (PDF)ofofthe thetime velocity of a derivative owing to its relation with the description evolution 1, .. .. .. ,, d. of Density (PDF) of particle. In the well-posedness the FP d.suitable assumptions on the coefficients b and σ , it is Under particle. In recent recent years, years, theFunction well-posedness ofthe thevelocity FP equation equation of the the Probability Probability Density Function (PDF) of ofof the velocity of aa 1, Under suitable assumptions on the coefficients b and σ , it is particle. In recent the of equation under low assumptions on been Under suitable coefficients and σ (Primak al., p. under low regularity assumptions on the the coefficients coefficients has been well particle. In regularity recent years, years, the well-posedness well-posedness of the the FP FP has equation well known, seeassumptions (Primak et eton al.,the 2004, p. 227) 227) bband and (Protter, Underknown, suitablesee assumptions on the2004, coefficients and(Protter, σ ,, it it is is under low regularity assumptions on the coefficients has been studied, see Le Lions also in with known, see (Primak et al., 2004, p. 227) and (Protter, 2005, p. 297) that, given an initial distribution ρ , the PDF studied, seeregularity Le Bris Bris and and Lions (2008), (2008), also in connection connection with well under low assumptions on the coefficients has been 0 2005,known, p. 297)see that, given an initial distribution ρ0 , the PDF well (Primak et al., 2004, p. 227) and (Protter, studied, see Le Bris and Lions (2008), also in connection with existence, uniqueness and stability of martingale solutions to p. given initial PDF associated with the (1) to existence, and stability of martingale solutions to 2005, studied, seeuniqueness Le Bris and Lions (2008), also in connection with 0 ,, the associated withthat, the stochastic stochastic process (1) evolves evolves ρ according to 2005, p. 297) 297) that, given an an process initial distribution distribution ρaccording 0 the PDF existence, and of solutions to the related stochastic equation, see (2008). associated with the stochastic process (1) evolves according to following FP equation: the related uniqueness stochastic differential differential equation, see Figalli Figalli (2008). existence, uniqueness and stability stability of martingale martingale solutions to the the following FP equation: associated with the stochastic process (1) evolves according to the stochastic differential equation, see (2008). Moreover, properties of FP have following FP equation: Moreover, control properties of the the FP equation equation have become become the related related control stochastic differential equation, see Figalli Figalli (2008). the 2 the following FP equation: − ∂ (a ρ) + ∂ (b ρ) = 0 , in Q , (2) ∂∂t ρ 2 Moreover, control properties of the FP equation have become of main interest in mean field game theory, see Porretta (2015) ij i i in Q , (2) of main interest in properties mean field of game see Porretta (2015) t ρ − ∂ii2jj (ai j ρ) + ∂i (bi ρ) = 0 , Moreover, control the theory, FP equation have become − ∂∂i= ρ) + ∂∂i (b = 00 ,, in Ω Q ,, (2) ∂∂ρ(x, 2j (a of interest in mean game see (2015) for further insight this connection. In eere the tρ i j(x) i ρ) , in . (3) 0) ρ ρ − (a ρ) + (b ρ) = Q (2) 0 for further insight on thisfield connection. In Blaqui` Blaqui` re (1992), (1992), the of main main interest in on mean field game theory, theory, see Porretta Porretta (2015) t i j i i in Ω . (3) ρ(x, 0) i=j ρ0 (x) , for further insight on this connection. In Blaqui` e re (1992), the analysis of the controllability properties of the FP equation (x) , in Ω . (3) 0) = ρ 0 analysis ofinsight the controllability properties of theereFP equation for further on this connection. In Blaqui` (1992), the We refer ρ(x, to and in Ω . numer(3) 0) = ρ(1989) 0 (x) , for analysis the properties of equation has been in with system and refer ρ(x, to Risken Risken (1989) for an an exhaustive exhaustive theory theory and numerhas been of developed in connection connection with quantum quantum system and We analysis ofdeveloped the controllability controllability properties of the the FP FP equation We refer to Risken (1989) for an exhaustive theory and numerical methods for the FP equation. A solution ρ to (2)-(3) shall has been developed in connection with quantum system and stochastic control. ical refer methods for the(1989) FP equation. A solutiontheory ρ to (2)-(3) shall to Risken for an exhaustive and numerstochastic control. in connection with quantum system and We has been developed ical methods for FP equation. A to furthermore properties of i.e., stochastic control. furthermore satisfy the standard properties of aaρ PDF, i.e., shall ical methodssatisfy for the thethe FPstandard equation. A solution solution ρ PDF, to (2)-(3) (2)-(3) shall stochastic control. In satisfy the standard properties of aa, PDF, i.e., In aa similar similar way, way, our our main main interest interest in in the the optimal optimal control control of of the the furthermore ρ(x,t) ≥ 0 , (x,t) ∈ Q furthermore satisfy the standard properties of PDF, i.e., ρ(x,t) ≥ 0 , (x,t) ∈ Q , In similar our main interest in of FP derives connection with evolution of  FP equation derives from its connection with the thecontrol evolution of In aaequation similar way, way, our from main its interest in the the optimal optimal control of the the  ρ(x,t) ≥ 00 ,, (x,t) ∈ ρ(x,t) ≥ ∈TQ Q) ,,. FP equation derives from its connection with the evolution of the PDF associated with a stochastic process. Given T > 0, let  dx = 11 ,, tt(x,t) ∈ (0, the PDF associated stochastic process. T > 0, let FP equation deriveswith froma its connection with Given the evolution of  ρ(x,t) ρ(x,t) dx = ∈ (0, T . Ω ρ(x,t) dx = 1 , t ∈ (0, T ) the PDF with process. Given T let us aa continuous-time stochastic process described .. Ω ρ(x,t) dx = 1 , t ∈ (0, T ) us consider continuous-time stochastic process described by theconsider PDF associated associated with aa stochastic stochastic process. Given T> > 0, 0, by let ) Ω us consider a continuous-time stochastic process described by the (Itˆ o ) stochastic differential equation Consider now the presence of a control function the (Itˆo) stochastic differential equation us consider a continuous-time stochastic process described by Consider now theΩpresence of a control function acting acting on on (1) (1) the (Itˆ o ) stochastic differential equation Consider now the presence of a control function acting on (1) through the drift term b, dX = b(X ,t) dt + σ (X ,t) dW , t ∈ (0, T ) , the (Itˆo) dX stochastic equation through the drift term b, t = b(Xtdifferential t ,t) dWt , Consider now the presence of a control function acting on (1) ,t) dt + σ (X t ∈ (0, T ) , t t t t (1) through the drift term b, = b(X ,t) dt + σ (X ,t) dW , t ∈ (0, T ) , dX (1) t t t t dX = b(X ,t; u) dt + σ (X ,t) dW , (4) through the drift term b, t t t t X(t = 0) = X , b(X ,t) dt + σ (X ,t) dW , t ∈ (0, T ) , dX 0 dXt = b(Xt ,t; u) dt + σ (Xt ,t) dWt , (4) t t (1) X(tt = 0) =t X0 , (1) where the control dX b(X u) dt + σ (X dW (4) t = t ,t; t ,t) t ,, X(t = 0) = X , 0 d m has to be chosen from a suitable class of dX = b(X ,t; u) dt + σ (X ,t) dW (4) t t t t X(t = 0) = X , where X ∈ R is the initial condition, d ≥ 1, W ∈ R is 0 d m where the control has to be chosen from a suitable class of where X00 ∈ Rd is the initial condition, d ≥ 1, Wtt ∈ Rm is admissible where the control has to be chosen from a suitable class of functions in a way to minimize a certain cost funcwhere X ∈ R is the initial condition, d ≥ 1, W ∈ R is an m−dimensional Wiener process, m ≥ 1, b = (b , . . . , b ) admissible functions in a way to minimize a certain cost funcd m t 0 where the control has to be chosen from a suitable class of m 1 an m−dimensional Wiener process, m ≥ d1,≥b 1, =W (bt 1∈ , . .R . , bmis) tional. where X0 ∈ R is the initial condition, admissible functions in a way to minimize a certain cost funcIn the non-deterministic case of (4), the state evolution an m−dimensional Wiener process, m ≥ 1, b = (b , . . . , b ) is a vector valued drift function, and the dispersion matrix tional. In the non-deterministic case of (4), the state evolution m 1 admissible functions in a way to minimize a certain cost funcis am−dimensional vector valued d×m drift function, the1,dispersion an Wiener process,and m≥ b = (b1 , . .matrix . , bm ) X tional. In of the evolution random Therefore, dealing with is aa t ,t) vector drift the dispersion σ = ∈ Rd×m assumed to rank. represents random variable. variable.case Therefore, when dealing with Xtt represents In the the aanon-deterministic non-deterministic case of (4), (4), when the state state evolution i j) σ (X (X = (σ (σvalued isfunction, assumed and to have have full rank. matrix is vector valued drift is function, and the full dispersion matrix tional. t ,t) i j ) ∈ Rd×m represents a random variable. Therefore, when dealing X stochastic optimal control, usually the average of the cost funct σ (X ,t) = (σ ) ∈ R is assumed to have full rank. stochastic optimal control, usually the average of the cost funcd×m t i j with Xt represents a random variable. Therefore, when dealing with σ (Xt ,t) = (σi j ) ∈ R is assumed to have full rank. stochastic optimal usually average of the function see Fleming and (1975).  This work was partially supported by the EU under the 7th Framework Protion is is considered, considered, see for for example example Fleming and Rishel (1975). stochastic optimal control, control, usually the the average ofRishel the cost cost func This work was partially supported by the EU under the 7th Framework Protion is see for Fleming (1975). In the functional is the form  In particular, particular, the cost cost functional is usually usually ofand the Rishel form gram, Initial Training Network SADCO, tion is considered, considered, see for example example Flemingof and Rishel (1975). ThisMarie work Curie was partially supported by the FP7-PEOPLE-2010-ITN EU under the 7th Framework Pro  gram, Initial Training Network SADCO,  ThisMarie work Curie was partially supported by the FP7-PEOPLE-2010-ITN EU under the 7th Framework ProIn particular, the cost functional is usually of the form   T  GA 264735-SADCO, by the DFG project Model Predictive Control for the gram, Marie Curie Initial Training Network FP7-PEOPLE-2010-ITN SADCO, In particular, the cost functional is usually of the form T GA 264735-SADCO, by Training the DFGNetwork project FP7-PEOPLE-2010-ITN Model Predictive ControlSADCO, for the   gram, Marie Curie Initial J(X, Fokker-Planck Equation, GR DFG 1569/15-1, by Predictive the INdAM through GA 264735-SADCO, by the project and Model Control for the L(t, X Xtt ,, u(t))dt u(t))dt + + ψ(X ψ(XTT )) ,, J(X, u) u) = =E E 0TT L(t, Fokker-Planck Equation, GR DFG 1569/15-1, by Predictive the INdAM through GA 264735-SADCO, by the project and Model Control for the L(t, X , u(t))dt + ψ(X ) J(X, u) = E GNAMPA Research Project 2015 ”Analisi e controllo di equazioni a derivate 0 t T Fokker-Planck Equation, GR 1569/15-1, by thediINdAM through the J(X, u) = E 0 L(t, Xt , u(t))dt + ψ(XT ) ,, GNAMPA Research Project 2015 ”Analisi eand controllo equazioni a derivate Fokker-Planck Equation, GR 1569/15-1, and by the INdAM through the parziali nonlineari”. for suitable GNAMPA Research Project 2015 ”Analisi e controllo di equazioni a derivate parziali nonlineari”. for suitable running running cost cost0L L and and terminal terminal cost cost ψ. ψ. GNAMPA Research Project 2015 ”Analisi e controllo di equazioni a derivate parziali nonlineari”. for suitable running cost L and terminal parziali nonlineari”. for suitable running cost L and terminal cost cost ψ. ψ.

Copyright © 2016, 2016 International Federation of 256Hosting by Elsevier Ltd. All rights reserved. 2405-8963 © IFAC (International Federation of Automatic Control) Copyright © 2016 International Federation of 256 Automatic Control Copyright © 2016 International Federation of 256Control. Peer review under responsibility of International Federation of Automatic Automatic © Control Copyright 2016 International Federation of 256 Automatic Control 10.1016/j.ifacol.2016.07.450 Automatic Control

256 256 256 256

IFAC CPDE 2016 June 13-15, 2016. Bertinoro, Italy

Arthur Fleig et al. / IFAC-PapersOnLine 49-8 (2016) 254–259

On the other hand, the state of a stochastic process can be characterized by the shape of its statistical distribution, which is represented by the PDF. Therefore, controlling the PDF provides an accurate and flexible control strategy that can accommodate to a wide class of objectives, cf. (Brockett, 2001, Section 4). In this direction, in Forbes et al. (2004); Jumarie (1992); K´arn´y (1996); Wang (1999) PDF-control schemes were proposed, where the cost functional depends on the PDF of the stochastic state variable. In this way, a deterministic objective results and no average is needed. However, in these references, stochastic methods were still adopted in order to approximate the state variable Xt of the random process. On the other hand, in Annunziato and Borz`ı (2010, 2013) the authors approach directly the problem of tracking the PDF associated with the stochastic process. If the control acts through the drift term as in (4), the evolution of the PDF is controlled through the advection term of equation (2). This is a rather weak action of the controller on the system, usually called of bilinear type, since the control takes action as a coefficient of the state variable. Indeed, few controllability results are known for such a kind of control system (e.g. Blaqui`ere, 1992; Porretta, 2014). Concerning the existence of bilinear optimal control, a first result was given by Addou and Benbrik (2002) for a control function that only depends on time. Relying on this result, in Annunziato and Borz`ı (2010, 2013) the tracking of a PDF governed by (2) has been studied with a time dependent control function. Notice that, in general, the space domain in (2) is Rd instead of Ω. However, if localized SDEs are under consideration, or if the objective is to keep the PDF within a given compact set of Ω and the probability to find Xt outside of Ω is negligible, we might focus on the description of the evolution of the PDF in the bounded domain Ω ⊂ Rd . Assuming that the physical structure of the problem ensures the confinement of the stochastic process within Ω, it is reasonable to employ homogeneous Dirichlet boundary conditions ρ(x,t) = 0

in Σ ,

also known as absorbing boundary conditions (Primak et al., 2004, page 231) (see also Feller (1954) for a complete characterization of possible boundary conditions in dimension one). The aim of this work is to extend the theoretical study on the existence of bilinear optimal controls of the FP equation by Addou and Benbrik (2002) to the case of more general control functions, i.e., to the case of a bilinear control that depends both on time and space. In connection with our motivation from stochastic optimal control, on the one hand, a simpler controller u = u(t) would be easier to implement in some applications. On the other hand, in certain situations it could be handier or even required to act on the space variable as well. In general, the richer structure of a control u = u(x,t) allows to substantially improve the tracking performance of a PDF, as shown in Fleig et al. (2014). For a more detailed presentation of the results in the current work and their proofs, we refer to Fleig and Guglielmi (2016). In the sequel, following Aronson (1968), we introduce proper assumptions on the functional framework to ensure existence of solutions to state equation of the form (2) in Section 2. Section 3 is devoted to recast the FP equation in an abstract setting and to deduce useful a-priori estimates on its solution. The main result on existence of solutions to the optimal control problem is Theorem 9 presented in Section 4, whereas in Section 5 we 257

255

deduce the system of first order necessary optimality conditions that characterizes the optimal solutions, recast in Corollary 16. Section 6 concludes. 2. EXISTENCE OF SOLUTIONS TO THE FP EQUATION In this section, we describe the functional framework that we will use to ensure the existence of solutions to ∂t y − ∂i2j (ai j y) + ∂i (bi (u)y) = f in Q , (5)

which, assuming ai j ∈ C1 (Q) for all i, j = 1, . . . , d, and setting b˜ j (u) := ∂i ai j − b j (u), can be recast in the flux formulation   ∂t y − ∂ j ai j ∂i y + b˜ j (u)y = f in Q , (6)

with initial and boundary conditions y(x,t) = 0 ,

(x,t) ∈ Σ ,

2

x ∈ Ω, y(x, 0) = y0 (x) ∈ L (Ω) , and associated weak formulation       fv = ∂t yv − ∂ j ai j ∂i y + b˜ j (u)y v Q

=−

 Q

Q

y∂t v −



Q

y(·, 0)v(·, 0) +

Ω

(8)

  Q

∈ W21,1 (Q)

(7)

 ai j ∂i y + b˜ j (u)y ∂ j v

with v|∂ Ω = 0 and v(·, T ) = for any test function v 0, where the differentials dx and dt have been omitted for readability. Here and in the following sections we assume the subsequent hypotheses: Assumption 1. (1) For all i, j = 1, . . . , d, the coefficients ai j are constant and there exists 0 < θ < ∞ such that ai j ξi ξ j ≥ θ |ξ |2 ∀ξ ∈ Rd , (2) f , b˜ j (u) ∈ Lq (0, T ; L∞ (Ω)), j = 1, ..., d, with 2 < q ≤ ∞. We assume for simplicity the coefficients ai j to be constant in order to focus more specifically on the bilinear action of the control through the divergence term. However, it is possible to extend the analysis to the case of space-dependent positive definite diffusion coefficients, see Fleig and Guglielmi (2016). Under Assumption 1, a result by (Aronson, 1968, Thm. 1, p. 634) ensures the existence and uniqueness of (nonnegative) solutions to equation (6). Theorem 2. (Existence of nonnegative solutions). Suppose that Assumption 1 holds and let y0 ∈ L2 (Ω). Then there exists a unique y ∈ L2 (0, T ; H01 (Ω)) ∩ L∞ (0, T ; L2 (Ω)) satisfying       −y∂t v + ai j ∂i y + b˜ j (u)y ∂ j v − f v = y0 v(·, 0) Ω

Q

∈ W21,1 (Q)

with v|∂ Ω = 0 and v(·, T ) = 0, i.e., y is for every v the unique weak solution of the Fokker-Planck initial boundary value problem (6)-(8). Moreover, if f ≡ 0 and 0 ≤ y0 ≤ m almost everywhere in Ω, then 0 ≤ y(x,t) ≤ m(1 +CFP k) almost everywhere in Q , where d   k := ∑ ˜b j (u)Lq (0,T ;L∞ (Ω)) j=1

and the constant CFP > 0 depends only on T, Ω, and the structure of the FP equation.

IFAC CPDE 2016 256 June 13-15, 2016. Bertinoro, Italy

Arthur Fleig et al. / IFAC-PapersOnLine 49-8 (2016) 254–259

Remark 3. If the right-hand-side in (6) is of the form f = div(F) with F : Q → Rd , Theorem 2 remains true assuming that Fi ∈ L2 (Q), i = 1, ..., d, see Aronson (1968). Remark 4. In the works Le Bris and Lions (2008); Porretta (2015), the well-posedness of the FP equation has been established even for drift coefficients b ∈ L2 (Q), in the context of renormalized solutions. These papers could describe the right framework for studying the optimal control problem of the FP equation in a Hilbert setting. The solution obtained by Theorem 2 is more regular. To this end, let us consider the Gelfand triple V → H → V  , with H := L2 (Ω), V := H01 (Ω), and V  = H −1 (Ω) the dual space of V , endowed with norms   y2H

2

:=

y dx ,

Ω

LV  :=

yV2

sup

y∈V,yV =1

:=

2

|∇y| dx ,

Ω   L, yV  ,V  ,

respectively, where . , . V  ,V represents the duality map between V and V  . We remind that W (0, T ) := {y ∈ L2 (0, T ;V ) : y˙ ∈ L2 (0, T ;V  )} ⊂ C([0, T ]; H) , y˙ denoting the time derivative of y. Proposition 5. Under the assumptions of Theorem 2, the solution y to problem (6)-(8) belongs to W (0, T ), possibly after a modification on a set of measure zero. In this way, we gain regularity of the time derivative of the state solution, which also allow us to deduce that y ∈ C([0, T ]; H). 3. A-PRIORI ESTIMATES In this section, we deduce a-priori estimates of solutions to the FP equation (5),(7),(8) with f ∈ L2 (0, T ;V  ). For the sake of clarity, we recast it in abstract form  y(t) ˙ + Ay(t) + B(u(t), y(t)) = f (t) in V  , t ∈ (0, T ) (9) y(0) = y0 , where y0 ∈ H, A : V → V  is a linear and continuous operator such that  Az, ϕV  ,V =

Ω

ai j ∂i z ∂ j ϕ dx

∀z, ϕ ∈ V,

and the operator B : L∞ (Ω; Rd ) × H → V  is defined by B(u, y), ϕV  ,V = −



Ω

bi (u)y ∂i ϕ dx = −



yb(u). ∇ϕ dx

Ω

for all u ∈ L∞ (Ω; Rd ), y ∈ H, ϕ ∈ V . In the following, E (y0 , u, f ) refers to (9) whenever we want to point out the data (y0 , u, f ). Given q > 2, admissible controls are functions u ∈ U := Lq (0, T ; L∞ (Ω; Rd )) ⊂ L2 (0, T ; L∞ (Ω; Rd )) . Moreover, we assume the following property. Assumption 6. There exist functions γi ∈ C1 (Ω) such that the components of the function b : Rd+1 × U → Rd , (x,t; u) → b(x,t; u(x,t)) are given by bi (x,t; u) = γi (x) + ui (x,t) , (x,t) ∈ Q , i = 1, . . . , d . (10) From this section on, we denote by M and C generic positive constants that might change from line to line. In this setting, relation (10) ensures that B(u, y)V  ≤ M(1 + uL∞ (Ω;Rd ) ) yH for any u ∈ L∞ (Ω; Rd ) and y ∈ H.

258

For brevity, in the following we will refer to the space L p (0, T ; X) simply by L p (X), for any p ∈ [1, +∞] and X Banach space. To ease the notation, we still denote by A and B the operators A : L2 (V ) → L2 (V  ) and B : U × L∞ (H) → Lq (V  ) with 1/q + 1/q = 1, such that, respectively, Az = −∂ j (ai j ∂i z)

∀z ∈ L2 (V )

and B(u, y) = ∂i (bi (u)y) = div(b(u)y) ∀u ∈ U , y ∈ L∞ (H) . Indeed, for every u ∈ U and y ∈ L∞ (H) we have that div(b(u)y) ∈ Lq (V  ) and B(u, y)Lq (V  ) = div(b(u)y)Lq (V  ) ≤ M(1 + uU ) yL∞ (H) . The next result gives some useful a-priori estimates on the solution to (9). Lemma 7. Let y0 ∈ H, f ∈ L2 (V  ) and u ∈ U . Then a solution y to (9) satisfies the estimates   y2L∞ (H) ≤ M(u) y0 2H +  f 2L2 (V  ) ,   y2L2 (V ) ≤ (1 + u2U )M(u) y0 2H +  f 2L2 (V  ) , y ˙ 2L2 (V  ) ≤ 2  f 2L2 (V  ) +

  (1 + u2U )M(u) y(0)2H +  f 2L2 (V  ) , 2

where M(u) := Cec(1+uU ) , for some positive constants c, C. These a-priori estimates could be used to prove existence and uniqueness of solutions to (9) by a Galerkin approximation method, see, for example, Evans (2010). However, we prefer to rely on the well-posedness result by Aronson (1968), since it also ensures the positivity of the solutions, which is a natural property for a PDF. On the other hand, the estimates given by Lemma 7 are crucial to prove Theorem 9 and Lemma 14, see Fleig and Guglielmi (2016) for the proofs of these results. 4. EXISTENCE OF OPTIMAL CONTROLS In this section, we consider the minimization of a cost func˜ u), where the state y is subject to equation (9) with tional J(y, control u and source f ≡ 0. We require Assumptions 1 and 6 to hold in this and the following section. Fixing y0 ∈ H, we introduce the control-to-state operator Θ : U → C([0, T ]; H) such that u → y ∈ C([0, T ]; H) solution of E (y0 , u, 0). Thus, the optimization problem turns into mini˜ mizing the so-called reduced cost functional J(u) := J(Θ(u), u), which we assume to be bounded from below, over a suitable non-empty subset of admissible controls Uad . Without loss of generality, we assume the existence of a control u˜ ∈ Uad such that J(u) ˜ < ∞. In the following, we consider the usual box constraints for the space of admissible controls, i.e., Uad := {u ∈ U : ua ≤ u(x,t) ≤ ub for a.e. (x,t) ∈ Q} , (11) where ua , ub ∈ Rd and ua ≤ ub is to be understood componentwise. In the proof of the main theorem we also need the following compactness result (see Aubin (1963), (Lions, 1969, Th´eor`eme 5.1, page 58) or (Simon, 1987, Corollary 4)). Theorem 8. Let X,Y, Z be three Banach spaces, with dense and continuous inclusions Y → X → Z, the first one being compact, and let I be an open interval in R. Then, for every p ∈ [1, +∞) and r > 1 we have the compact inclusions L p (I;Y ) ∩W 1,1 (I; Z) → L p (I; X)

IFAC CPDE 2016 June 13-15, 2016. Bertinoro, Italy

Arthur Fleig et al. / IFAC-PapersOnLine 49-8 (2016) 254–259

and

L∞ (I;Y ) ∩W 1,r (I; Z) → C(I; X). Theorem 9. Let y0 ∈ H and let Assumptions 1 and 6 hold. ˜ Consider the reduced cost functional J(u) = J(Θ(u), u), to be minimized over the controls u ∈ Uad . Assume that J is bounded from below and (sequentially) weakly-star lower semicontinuous. Then there exists a pair (y, ¯ u) ¯ ∈ C([0, T ]; H) × Uad such that y¯ solves E (y0 , u, ¯ 0) and u¯ minimizes J in Uad . Remark 10. Requiring box constraints as in (11) might seem a too restrictive choice. However, we note that in case of bilinear action of the control into the system, for state equations different from those considered in this paper, even box constraints might not suffice to ensure the existence of optimal controls, see for example (Lions, 1971, Section 15.3, p. 237). Theorem 9 clearly also holds for any Uad that is a bounded weakly-star closed subset of U . However, note that in the unconstrained case Uad ≡ U , asking only J(u) ≥ λ uU for some λ > 0 is not enough. Instead, a condition of the type J(u) ≥ λ uL∞ (Q) would allow to prove the existence of optimal controls. However, this kind of condition is not very practical in applications. Corollary 11. Let y0 ∈ H and let Assumptions 1 and 6 hold. Let yd ∈ L2 (0, T ; H), yΩ ∈ H, α, β , λ ≥ 0 with max{α, β } > 0. Then an optimal pair (y, ¯ u) ¯ ∈ C([0, T ]; H) × Uad exists for the reduced cost functional J(u) defined by β λ α y − yd 2L2 (H) + y(T ) − yΩ 2H + u2L2 (H) , (12) 2 2 2 where y = Θ(u). Remark 12. If one wants to use the cost functional (12) without imposing box constraints on the control, e.g., Uad ≡ U , one shall require more regularity on the state y and on the control u, in order to gain the same level of compactness required in the proof of Theorem 9. Indeed, further regularity of y can be ensured by standard improved regularity results, see for example (Wloka, 1987, Theorems 27.2 and 27.5) and (Ladyzhenskaya et al., 1967, Theorem 6.1 and Remark 6.3). However, these results come at the price of requiring more regularity of the coefficients in the PDE, which, in our case, translates to more regularity of the control. In particular, one would need to require differentiability of u both in time and space, which is a feature that is scarcely ever satisfied in the numerical simulations. Remark 13. Corollary 11 applies analogously to the case of time-independent controls in the admissible space U˜ad := {u ∈ L∞ (Ω) : ua ≤ u(x) ≤ ub for a.e. x ∈ Ω} (13) for some ua , ub ∈ Rd such that ua ≤ ub (component-wise), and the reduced cost functional J2 (u) given by α β λ y − yd 2L2 (H) + y(T ) − yΩ 2H + u2H , 2 2 2 where y = Θ(u). 5. ADJOINT STATE AND OPTIMALITY CONDITIONS In this section, we consider b such that b(u) = u, thus B is given by B(u, y) = div(uy) ∀u ∈ U , y ∈ L∞ (0, T ; H) . This choice does not affect the generality of the problem. Indeed, for b as in Assumption 6, assuming maxi {γi , γi } sufficiently small, we can include the contribution div (γy) in the operator A, which becomes Aγ z := Az + div(γz) and still satisfies the assumptions required on A. 259

257

Thanks to the estimates given by Lemma 7, we deduce the following result. Lemma 14. Let y0 ∈ H. Then the control-to-state map Θ is differentiable in the Fr´echet sense, and for every u, ¯ h ∈ U the function Θ (u)h ¯ satisfies  z˙(t) + Az(t) + B(u(t), ¯ z(t)) = −B(h(t), y(t)) ¯ in V  , (14) z(0) = 0 , where y¯ = Θ(u). ¯ Thanks to Remark 3, Theorem 2 ensures the existence of a unique weak solution of equation (14). We introduce the operator B˜ : L2 (V ) → L2 (L2 (Ω; Rd )) such that ˜ B(v) = ∇x v for all v ∈ L2 (V ), where ∇x denotes the gradient with respect to the space variable x ∈ Rd . For every u ∈ U , v ∈ L2 (V ), and w ∈ L∞ (H), we have that T 



˜ b(u). B(v), w

0

H

dt =



bi (u)w ∂i v dxdt

Q

=−

T 0

B(u(t), w(t)), vV  ,V dt

and the above integrals are well-defined. In the sequel, we give the first order necessary optimality conditions for the cost functional J as in (12). We start by deducing an explicit representation formula for the derivative of J. Incidentally, let us point out that J is one of the objective functionals most commonly used in the numerical simulations, see, for example, Annunziato and Borz`ı (2013); Fleig et al. (2014). Proposition 15. Let yd ∈ Lq (0, T ; L∞ (Ω)), yΩ ∈ L2 (Ω), and y0 ∈ L∞ (Ω). Then the functional J given by (12) is differentiable in U and, for all u, h ∈ U , dJ(u)h =



hi (t) [y(t)∂i p(t) + λ ui (t)] dxdt

(15)

Q

holds, where hi , i = 1, . . . , d, are the components of h ∈ U , y ∈ W (0, T ) ∩ L∞ (Q) is the solution of E (y0 , u, 0) and p ∈ W (0, T ) is the solution of the adjoint equation  ˜ − p(t) ˙ + Ap(t) − b(u(t)). Bp(t) = α [y(t) − yd (t)] in V  , p(T ) = β [y(T ) − yΩ ] .

(16)

Let us observe that system (16) is an Hamilton-Jacobi-Bellman equation (HJB), where the function hi ∂i p, yV  ,V : (0, T ) → R belongs to L1 (0, T ) for all i = 1, . . . , d, owing to hi ∈ Lq (L∞ (Ω)) with q > 2, y ∈ L2 (V ) and ∂i p ∈ L∞ (V  ). Moreover, y0 ∈ L∞ (Ω) implies y ∈ L∞ (Q), thus y − yd ∈ Lq (L∞ (Ω)) as required by Assumption 1. Furthermore, y(T ) − yΩ ∈ L2 (Ω). Therefore, by the change of variable q(t) = p(T − t), v(t) = u(T − t) and f (t) = α[y(T −t) − yd (T −t)], equation (16) is recast in a form similar to (9), for which a similar analysis as in Section 2 can be performed, based on (Aronson, 1968, Theorem 1, p. 634), in order to ensure existence and uniqueness of solutions for equation (16). In addition, if yΩ ∈ L∞ (Ω) we conclude that p ∈ W (0, T ) ∩ L∞ (Q). We note that, a priori, for every u ∈ U , dJ(u) is defined in U . However, thanks to the representation formula (15), it may be extended to a map defined on L2 (L2 (Ω; Rd )).

IFAC CPDE 2016 258 June 13-15, 2016. Bertinoro, Italy

Arthur Fleig et al. / IFAC-PapersOnLine 49-8 (2016) 254–259

As a consequence of Proposition 15 and the variational inequality dJ(u)(u ¯ − u) ¯ ≥ 0 for any u ∈ Uad and locally optimal solution u, ¯ we deduce the first order necessary optimality conditions, formulated in the next result. Corollary 16. Let yd ∈ Lq (0, T ; L∞ (Ω)), yΩ ∈ L2 (Ω), and y0 ∈ L∞ (Ω). Consider the cost functional J defined by (12) with α, β , γ ≥ 0 and max{α, β } > 0. Then an optimal pair (y, ¯ u) ¯ ∈ C([0, T ]; H) × Uad for J with corresponding adjoint state p¯ is characterized by the following necessary conditions:   ∂t y¯ − ai j ∂i2j y¯ + ∂i u¯i y¯ = 0 , in Q , −∂t p¯ − ai j ∂i2j p¯ − u¯i ∂i p¯ = α[y¯ − yd ] , in Q , y¯ = p¯ = 0

on Σ ,

y(0) ¯ = y0 ,

p(T ¯ ) = β [y(T ¯ ) − yΩ ] , in Ω ,

 Q

[y∂ ¯ i p¯ + λ u¯i ] (ui − u¯i ) dxdt ≥ 0

(17)

∀u ∈ Uad .

We observe that optimality systems of similar structure, that is, coupling a FP equation with an HJB equation, often appear in the optimal control of mean field games dynamics, see for example (Bensoussan et al., 2013, Chapter 4). Remark 17. In the case of time-independent control as in Remark 13, the only modification needed in the optimality system (17) is the variational inequality, which changes to    T y∂ ¯ i p¯ dt + λ u¯i (ui − u¯i ) dx ≥ 0 ∀u ∈ U˜ad , Ω

0

where U˜ad is given by (13).

6. CONCLUSION For the controlled Fokker-Planck equation with a spacedependent control u(x,t) acting on the drift term we have established theoretical results regarding the existence of optimal controls and necessary optimality conditions. Compared to just time-dependent controls u(t), where the PDF can only be moved as a whole, space-dependent control allows to consider a much wider class of objectives. When applying the calculated optimal control directly to the stochastic process, this results in a feedback loop, which may be interesting to a variety of applications, e.g., fluid flow, quantum control, or finance. ACKNOWLEDGEMENTS The authors wish to express their gratitude to Lars Gr¨une for suggesting them this very interesting subject and for many helpful comments. They would also like to thank Alfio Borz`ı for very helpful discussions. The authors also thank the referees for their suggestions that improved the readability of the paper. REFERENCES Addou, A., Benbrik, A., 2002. Existence and uniqueness of optimal control for a distributed-parameter bilinear system. J. Dynam. Control Systems 8 (2), 141–152. Annunziato, M., Borz`ı, A., 2010. Optimal control of probability density functions of stochastic processes. Math. Model. Anal. 15 (4), 393–407. 260

Annunziato, M., Borz`ı, A., 2013. A Fokker-Planck control framework for multidimensional stochastic processes. J. Comput. Appl. Math. 237 (1), 487–507. Aronson, D. G., 1968. Non-negative solutions of linear parabolic equations. Ann. Scuola Norm. Sup. Pisa (3) 22, 607–694. Aubin, J.-P., 1963. Un th´eor`eme de compacit´e. C. R. Acad. Sci. Paris 256, 5042–5044. Bensoussan, A., Frehse, J., Yam, P., 2013. Mean field games and mean field type control theory. Springer Briefs in Mathematics. Springer, New York. Blaqui`ere, A., 1992. Controllability of a Fokker-Planck equation, the Schr¨odinger system, and a related stochastic optimal control (revised version). Dynam. Control 2 (3), 235–253. Brockett, R., 2001. New issues in the mathematics of control. In: Mathematics unlimited—2001 and beyond. Springer, Berlin, pp. 189–219. Evans, L. C., 2010. Partial differential equations, 2nd Edition. Vol. 19 of Graduate Studies in Mathematics. American Mathematical Society, Providence, RI. Feller, W., 1954. Diffusion processes in one dimension. Trans. Amer. Math. Soc. 77, 1–31. Figalli, A., 2008. Existence and uniqueness of martingale solutions for SDEs with rough or degenerate coefficients. J. Funct. Anal. 254 (1), 109–153. Fleig, A., Gr¨une, L., Guglielmi, R., 2014. Some results on model predictive control for the Fokker-Planck equation. In: MTNS 2014 : 21st International Symposium on Mathematical Theory of Networks and Systems, July 7-11, 2014. University of Groningen, The Netherlands. pp. 1203–1206. Fleig, A., Guglielmi, R., 2016. Optimal Control of the Fokker-Planck Equation with StateDependent Controls. preprint, available online at https://epub.uni-bayreuth.de/2794/. Fleming, W. H., Rishel, R. W., 1975. Deterministic and stochastic optimal control. Springer-Verlag, Berlin-New York. Forbes, M. G., Forbes, J. F., Guay, M., 2004. Regulating discrete-time stochastic systems: focusing on the probability density function. Dyn. Contin. Discrete Impuls. Syst. Ser. B Appl. Algorithms 11 (1-2), 81–100. Jumarie, G., 1992. Tracking control of nonlinear stochastic systems by using path cross-entropy and Fokker-Planck equation. Internat. J. Systems Sci. 23 (7), 1101–1114. K´arn´y, M., 1996. Towards fully probabilistic control design. Automatica J. IFAC 32 (12), 1719–1722. ¨ Kolmogoroff, A., 1931. Uber die analytischen Methoden in der Wahrscheinlichkeitsrechnung. Math. Ann. 104 (1), 415–458. Ladyzhenskaya, O., Solonnikov, V., Ural’tseva, N., 1967. Linear and quasilinear equations of parabolic type. Moskva: Izdat. ’Nauka’. 736 pp. Le Bris, C., Lions, P.-L., 2008. Existence and uniqueness of solutions to Fokker-Planck type equations with irregular coefficients. Comm. Partial Differential Equations 33 (7-9), 1272–1317. Lions, J.-L., 1969. Quelques m´ethodes de r´esolution des probl`emes aux limites non lin´eaires. Dunod; GauthierVillars, Paris. Lions, J.-L., 1971. Optimal control of systems governed by partial differential equations. Translated from the French by S. K. Mitter. Die Grundlehren der mathematischen Wissenschaften, Band 170. Springer-Verlag, New York-Berlin. Porretta, A., 2014. On the planning problem for the mean field games system. Dyn. Games Appl. 4 (2), 231–256.

IFAC CPDE 2016 June 13-15, 2016. Bertinoro, Italy

Arthur Fleig et al. / IFAC-PapersOnLine 49-8 (2016) 254–259

Porretta, A., 2015. Weak solutions to Fokker-Planck equations and mean field games. Arch. Ration. Mech. Anal. 216 (1), 1–62. Primak, S., Kontorovich, V., Lyandres, V., 2004. Stochastic methods and their applications to communications. John Wiley & Sons, Inc., Hoboken, NJ. Protter, P. E., 2005. Stochastic integration and differential equations. Vol. 21 of Stochastic Modelling and Applied Probability. Springer-Verlag, Berlin. Risken, H., 1989. The Fokker-Planck equation, 2nd Edition. Vol. 18 of Springer Series in Synergetics. Springer-Verlag, Berlin. Simon, J., 1987. Compact sets in the space L p (0, T ; B). Ann. Mat. Pura Appl. (4) 146, 65–96. Wang, H., 1999. Robust control of the output probability density functions for multivariable stochastic systems with guaranteed stability. IEEE Trans. Automat. Control 44 (11), 2103–2107. Wloka, J., 1987. Partial differential equations. Cambridge University Press, Cambridge.

261

259