- Email: [email protected]
Chapter I Convex Functions
CHAPTER I
Convex Functions
Introduction This chapter assumes a basic knowledge of topological vector spaces. Moreover, we shall recall in Section I several fundamental aspects of this theory which will be constantly used in what follows. These reminders are in no way systematic and are centred on the notion of a convex set. We go on to consider convex functions (Sections 2-4) and their differentiability (Sections 5 and 6). All the vector spaces studied here are real.
1. CONVEX SETS AND THEIR SEPARAnON 1.1. Convex sets Let V be a vector space over R. If u and v are two points of V, u and v are called the endpoints of the line-segment denoted by [u,v] where
[u, v] = { AU
+ (1
- A)v
I°~
A ~ 1 }.
A set .91 c V is said to be convex if and only if for every pair of elements (u, v) ofd the segment [u,v] is contained in V. We know that a set .91 c V is convex if and only if for every finite subset of elements Ul, ••• , Un of .91, and for every family of real positive numbers AI, ..., An with sum unity, we have n
I
i= 1
AiUiEd.
The whole of the space V is convex and, conventionally, so is the empty set. Every intersection of convex sets is convex, but in general the union of convex sets is not convex. If .91 is any subset of V, the intersection of all the sets containing .91 is a convex set, and it is the smallest convex set containing d. It is called the convex hull ofd and is denoted by co d. It is also the set of all the convex combinations of the elements of .91, i.e., n
co d =
{
i~1
n
A;U;
I n E N, t~1
Ai
3
= 1,
Ai ~ 0, u, Ed, 1
~i~n}
4
FUNDAMENTALS OF CONVEX ANALYSIS
Let J't' be an affine hyperplane with equation I(u) linear form on V and IX E R. The sets { U
E V
{ UE
If(u) <
I
V f(u) ~
IX },
{ U
IX},
{ U
= IX where I
I E V If(u) ~
E
V f(u) >
is a non-zero
IX }, IX}
are called respectively open and closed half-spaces bounded by J't'. These are two convex sets which depend only on J't', and not on the particular I and IX chosen for its equation. 1.2. Separation of convex sets
We recall that a topological vector space (t.v.s.) is defined as a vector space V endowed with a topology for which the operations
(u, v) >-+ u
+v
(A., u) >-+ A.U
of V x V into V, of R x V into V
are continuous. The neighbourhoods of any point may then be deduced from those of the origin by translation. A t.v.s. is said to be a locally convex space (I.c.s.) if the origin possesses a fundamental system of convex neighbourhoods. This is the case with normed spaces: it is sufficient to take the set of neighbourhoods formed by the balls centred on the origin. All the usual t.v.s. encountered in analysis are locally convex. Now let V be a t.v.s. and J't' an affine hyperplane with equation I(u) = IX where I is a non-zero linear functional on V and IX E R. It can be shown that the set J't' is topologically closed if and only if the function I is continuous. Under these conditions the open (closedjhalf-spaces determined by J't' will be topologically open (closed). In a t.V.S. V, the closure of a convex set is convex and the interior of a convex set is also convex (possibly empty). More generally, if d c V is convex, if u E d (the interior of d) and if...!' E.si (the closure of d), then [u,v[ c d, from which we deduce that .si = d, whenever d 'f; 0. This suggests to us the introduction of the following definition: a point u E d will be called internal if every line passing through u meets d in a segment [v 10 V2] sueh that u E ]VI' V2[' Hence every interior point is internal, and by the above argument, if .si 'f; 0 every internal point is interior. If d is any subspace of V, the intersection of all the closed convex subsets containing d is the smallest closed convex subset containing d. It is also the closure of the convex hull of d (and not the convex hull of the closure!); it is called the closed convex hull of d and is denoted by co d.
5
CONVEX FUNCTIONS
An affine hyperplane Yf is said to separate (strictly separate) two sets .91 and ffl if each of the closed (open) half-spaces bounded by Yf contains one of them. This may be written analytically as follows. If t(u) = ex is the equation of Yf, then we have t(u) ~ ex, Vu E d,
t(v)
~
ex, "Iv E ffl,
for separation, t(u) < (x, Vu E sl/, t(v) > lX, "Iv E ffl. for strict separation. We now recall the Hahn-Banach theorem in its geometric form, and its consequences for the separation of convex sets. Naturally it is in the context of I.c.s. that the most precise results are obtained.
Hahn-Banach Theorem. Let V be a real t.v.s., .91 an open non-empty convex set, and vi{ a non-empty affine subspace which does not intersect d. Then there exists a closed affine hyperplane Yf which contains vi{ and does not intersect d. Corollary 1.1. Let V be a real t.v.s., .91 an open non-empty convex set, ffl a non-empty convex set which does not intersect d. Then there exists a closed affine hyperplane Yf which separates .91 and fJ8. Corollary 1.2. Let V be a reall.c.s., CC and ffl two non-empty disjoint convex sets with one compact and the other closed. Then there exists a closed affine hyperplane Yf which strictly separates CC and ffl. Here is an example of application of Corollary 1.1. Let .91 be a subset of V and Yf a closed affine hyperplane which contains at least one point u E .91, such that .91 is completely contained in one of the closed half-spaces determined by :Yl'; we say that :Yl' is a supporting hyperplane and u a supporting point of d. Then
Corollary 1.3. In a real t.V.S. V, let .91 be a convex set with non-empty interior. Then every boundary point ofd is a supporting point ofd. An application of Corollary 1.2 is:
Corollary 1.4. In a real l.c.s. V, every closed convex set is the intersection of the closed half-spaces which contain it. All these results have a fundamental importance in analysis because they allow a convenient theory of duality. Thus, if V is a Hausdorff I.c.s., the Hahn-Banach theorem allows us to assert the existence of non-zero continuous linear forms over V: it is sufficient to consider two points Ul and U2 of V, and to separate then by a closed affine hyperplane :Yl' (Cor. 1.2); if the equation of
6 :If is t(u)
FUNDAMENTALS OF CONVEX ANALYSIS
= el, the
non-zero linear form t is continuous since :Yf is closed and
t(Ut) =I t(U2)' The vector space V* of continuous linear functionals over V is
said to be the topological dual, or more simply the dual of V. The elements of V* will be, in general, denoted by u* or v*, and-- -
Mazur's Lemma. Let V be a normed space and (Un)nEN a sequence converging weakly to it. Then there is a sequence of convex combinations (vn)nEN such that N
where
L
k=n
Ak = I and Ak ~ 0,
which converges to u in norm. Proof For every n E N, U belongs to the weak closure of Uf=n {Uk}, and afortiori, to the weak closure of co {Uk}' But this is exactly the weakly closed convex hull of U~=n {Uk} which coincides with its closed convex hull by Corollary 104. Finally,
ur=n
ro
U E co
U
{Uk},
VnEN,
k =n
and it suffices to choose Vn
E
co U~=n
{ud such that Ilvn - unll ~ lin.
•
7
CONVEX FUNCTIONS
1.3. Analytical form of the Hahn-Banach theorem The analytical form of the Hahn-Banach theorem is more precise than the geometrical form. We consider a real vector space Vand a sub-linear function j over V, i.e. a mappingj of V into R which satisfies
j(AU) = Aj(U), j(u + v) ~ j(u)
vu e V, I::/A > 0,
+ j(v),
I::/(u, v) E V
X
v.
Hahn-Banach Theorem. Let V be a real vector space, j a sub-linear function over V, .It a vector subspace of V, t a linear functional over .It which is everywhere less than j. Then there exists a linear functional? over V which extends t and is everywhere less than j. Corollary 1.5. Let V be a normed space, .It a topological vector subspace, t a continuous linear functional over .It. Then t can be extended into a continuous linear functional over V with the same norm. 2. CONVEX FUNCTIONS
2.1. Definitions As before, we take a real vector space Vand consider mappings of d c V into R, that is, the values +ro and -ro are allowed to the functions under consideration.
Definition 2.1. Letd be a convex subspace of V, and Fa mapping of d into R. F is said to be convex if, for every u and v in d, we have:
(2.1)
F(AU
+ (1 - A)V)
~
AF(u)
+ (1 - A)F(v)
I::/A
E
[0, IJ
whenever the right-hand side is defined. The inequality (2.1) must therefore be valid unless F(u) = -F(v) = ±ro. By induction, it can be shown that if Fis convex, for every finite set Ul, .•• , u; of points of Vand for every family A1 , ••• , An of real positive numbers with sum unity, then
(2.2) whenever the right-hand side is defined.
8
FUNDAMENTALSOFCONVEXANALY~S
It is easy to see that, if F: V -+ II is convex, the sections
(2.3)
{ u I F(u) ::;;; a} and {u I F(u) < a}
are, for each a E II, convex sets of V. The converse is, however, false. For instance, if F is convex and if ¢: II -+ II is an increasing function then ¢ 0 F: V -+ II will also have convex sections but will not be convex in general. For every mapping F: V -+ II, we call the section:
(2.4)
dom F = { u I F(u) <
+
00 }
the effective domain of F. The effective domain of a convex function is thus convex. Why do we allow the value +00 ? If F is a mapping of d c V into R, we can associate with it the function F defined throughout V by:
(2.5)
I
~(u) = F(u) if u E .9'1, F(u) = + 00 if u¢d.
Thus F is convex if and only if d c V is convex and F: d -+ R is convex. In the theory of convex functions, because of this extension by +00, we need only consider those functions defined everywhere. There is a further advantage. If d is a subset of V, the indicatorfunction XJI1 of d is defined as:
(2.6)
X,r,1(u) I X,r,1(u)
= 0 =
+
if 00
u e sd, if u ¢ d.
Clearly d is a convex subset if and only if XJI1 is a convex function. Thus the study of convex sets is naturally reduced to the study of convex functions. On the other hand, convex functions which assume the value -00 are very special. If F(u) = -00, then on every half-line starting from ii, either F is identically equal to -00, or Ftakes the value -00 between ii and a point ii, any value at ii, and +00 beyond ii. To distinguish these very special cases, we shall say that a convex function F of V in II is proper if it nowhere takes the value -00 and is not identically equal to +00.
Definition 2.2. The epigraph of a function F: V......,.. II is the set: (2.7)
I
epi F = {(u, a) E V x R f(u) ::;;; a}.
It is the set of points of V x R which lie above the graph of F. The projection of epi F on V is none other than dom F. The epigraph will be found a most
9
CONVEX FUNCTIONS
useful concept in the study of convex functions because of the following result: Proposition 2.1. A function F: V -)0 R is convex
if and only if its epigraph
is convex. Proof Let F be convex and take (u,a) and (v,b) in epi F. Then, necessarily, F(u) ~ a < +00 and F(v) ~ b < +00, and for all A E [0,1] from (2.1) we have F(AU + (1 - A)V)
~
AF(u) + (1 - A)F(v)
~
Aa + (1 - A)b,
which means precisely that A(U, a) + (1 - A)(v,b) E epi F. Conversely, let epi F be convex. Its projection dom F is therefore convex and it is sufficient to verify (2.1) over dom F. Let us therefore take u and v in dom F, a ~ F(u) and b ~ F(v). By hypothesis, A(u,a) + (1 - A)(v,b) E epi F for every A E [0,1] so that:
F(AU
+ (1
- A)V)
~
Aa + (1 - A)b.
If F(u) and F(v) are finite, it is sufficient to take a = F(u) and b = F(v). If either F(u) or F(v) is equal to -00 it is sufficient to allow a or b to tend to -00 and (2.1) is obtained in both cases. _ It remains for us to consider the usual manipulations of convex functions. The results, which are trivial, have been collected together in the following proposition. In particular we infer from them that the set of convex functions is a convex cone. Proposition 2.2. (i) IfF: V -)0 R is convex and if A is a realpositive number, then AF is convex. (ii) IfF and G are convexfunctionsfrom V into R, then F + G is convex. We stipulate that (F + G)(u) = +00 if F(u) = -G(u) = too. (iii) !f(FI)leI is any family of convex functions of V into R, their pointwise supremum F = sup! eI F, is convex. Also, let us recall the concept of a strictly convex function. Definition 2.3. Let d be a convex set of V and F a mapping of d into R. F is said to be strictly convex if it is convex and the strict inequality holds in (2.1), 'Vu,vEd,u:;=vand'VAE ]0,1[. 2.2. Lower semi-continuous functions We now pass on to topological properties, so V will be a real I.c.s. We recall that a function F: V -?- R is said to be lowersemi-continuous on V (I.s.c.),
10
FUNDAMENTALS OF CONVEX ANALYSIS
if it satisfies the two equivalent conditions:
(2.8) (2.9)
VaER,
{ U
E
V I F(u) ~ a}
is closed,
lim F(u) ~ F(u). u-u
WiE V,
Naturally F will be called upper semi-continuous (u.s.c.) if -F is I.s.c. Thus for example the indicator function x",i') of a set .91 c V will be I.s.c. (or u.s.c.) if and only if .91 is closed (or open). More generally, we can at once state: Proposition 2.3. A function F: V ~ R is l.s.c. if and only if its epigraph is closed.
Proof Let us define on V x R a function ¢ by ¢(u, a) = F(u) - a. Then the statements that F is I.c.s. on V and that ¢ is I.c.s. on V x R are equivalent to each other. Now for every r E R, the section ¢([-oo, r Dis the set obtained from epi F by the translation of vector (r,O) and it is therefore closed if and only if epi F is closed. • Let us further recall that a pointwise supremum of I.s.c. functions is l.s.c, This then leads to the following definition: for every mapping F: V ~ R, the largest I.s.c. minorant of F will be called the I.s.c. regularization of F and will be denoted by P. It exists as the pointwise supremum of those I.s.c. functions everywhere less than F, and is characterized by: Corollary 2.1. Let F: V ~ Rand P be its l.s.c. regularization. We have epi
(2.10) (2.11 )
VUE V,
F= F(u)
epi F, = lim F(v). V-'u
Proof Since P is a I.s.c. function everywhere less than F, epi P is a closed set containing epi F, and hence epi F. Conversely, epi F is an epigraph. (1) Let G be a function such that epi G = epi F; it is a I.s.c. function everywhere less than F, hence G :::; P and epi G = epi F contains epi P. Thus (2.10) is proved and (2.11) follows directly from it. • (1) If (u,a) E epi F, there exists a filter (u.,a.) E epi F which converges to (u,a). If b > a, then for some suitable ()(, a.";; b and since F(u.)";; a. it follows that (u.,b) E epi F, and in the limit (u,b) E epi F. _ _ The intersection of epi Fwith a straight line {u} x R is an empty set or an interval [a, +00[; if we set G(u) = +00 in the former and G(u) = a in the latter case, we see that epi G = epi F.
11
CONVEX FUNCTIONS
The case of convex functions assumes a special interest since lower semicontinuity exists when the topology of V is weakened. This is an extremely valuable property. Corollary 2.2. Every l.s.c. convex function F of V in 'R remains l.s.c. when V is supplied with its weak topology a(V, V'). In fact, the epigraph of F being convex, this is equivalent to saying that it is closed and hence weakly closed (Cor. 1.4). The case of l.s.c. convex functions which assume the value -00 (improper functions) is more specialized: Proposition 2.4. If F: V -+ 'R is a l.s.c. convex function and assumes the value -00, it cannot take any finite value.
Proof Let us suppose that there exists u E V a point such that F(u) E R. We then take a E R such that a < F(u), and we strictly separate (u,a) from the closed convex set epi F. Then there exists a continuous non-zero linear form t over V and pER such that:
V(u,a)EepiF,
(2.12)
f(u)
+ IXQ <
f(u)
+ aa.
Taking u = ii and a = F(u), we get IX(F(u) - a) > 0, and hence IX> O. The two members of (2.12) can thus be divided by IX:
Vu E V,
(2.13)
-.!.f(u - u) IX
+
Ii
< F(u),
which is impossible, since the first member is everywhere finite and the second member takes the value -00 at one point at least. _ 2.3. Continuity of convex functions The study of the continuity of convex functions is based on the following lemma: Lemma 2.1. If in the neighbourhood of a point u E V, a convex function F is bounded above by a finite constant, then F is continuous at u.
Proof We reduce the problem by translation to the case where u = 0 and F(O) = O. Let "Y be a neighbourhood of the origin such that F(v) ~ a < +00 for all v of "Y. Let us define "If/' = "Y n -"Y (which is a symmetric neighbourhood of the origin), and let us take s E ]0, 1[. If v E eir, we have, due to the convexity of F:
!:.E"Y B
- !:.e E
'
"Y
'
+ eF(v/e) ~ ea,
hence
F(v) ~ (1 - e)F(O)
hence
F(v) ~ (1 + e)F(O) - eF(- v/e) ~ - ea.
12
FUNDAMENTALS OF CONVEX ANALYSIS
Then IF(v)l:::; ea for every v in e"f/l, whence we have the required continuity. • A general conclusion may be drawn: Proposition 2.5. Let F: V ~ R be a convex function. The following statements are equivalent to each other: (i) there exists a non-empty open set (IJ on which F is not everywhereequal to
and is boundedabove by a constant a < +00 ; (ii) F is a properfunction, and it is continuousover the interior of its effective domain, which is non-empty.
-00
0
~
Proof Clearly (ii) implies (i). Conversely, if (i) is true, (IJ c dom F. Let us take u E (IJ such that F(u) > -00. From Lemma 2.1, Fwill be continuous at u, and hence finite in a neighbourhood of u, and hence proper. For ~ry v E d~, there exists p > 1 such that w = u + p(v - u) also belongs to dom F. The homothety h with centre wand ratio 1 - lip transforms u into v and (IJ into an open set h«(IJ) containing v. For every v' E h«(IJ), we have by convexity:
F(v')::::; p -1 Foh-1(v') +.!.F(w)::::; p -1 a +.!-F(w). p p P P o .-----.....
To sum up: every point v E dom F possesses a neighbourhood h«(IJ) where F is bounded above by a finite constant. From Lemma 2.1, F is continuous at v, which concludes the proof. • We can use this result more precisely in numerous special cases-spaces of finite dimension, normed spaces and barrelled spaces. Corollary 2.3. Every proper convex function on a space offinite dimension is continuouson the interior ofits effective domain. o .-----.....
Proof If dom F is non-empty, it contains n + 1 affinely independent points u1 , 1 :::; t « n + 1, where n is the dimension of V. From the inequality defining convexity, F is bounded above by maXl,,;t";n+lF(Ul) over the open set:
Vi} •
Corollary 2.4. Let F be a proper convex function on a normed space. The following properties are equivalent to each other: (i) the~e exists a non-empty open set over which F is bounded above; (ii) dQ;F:f: 0 and F is locally Lipschitz there.
13
CONVEX FUNCTIONS
Proof It is obvious that (ii) over ~F
=>
(i). Conversely, if (i) is true, F is continuous
(Prop. 2.5). Taking u E .?4(u; r)
~,
for every r >
= {v Illv
- ull
By the continuity of F at u, there exists an r« >
"Iv
E
-
.?4(u; ro),
< m
00
~ r}.
°
~
such that: F(v) ~ M < +
Let us now suppose that r E ]O,ro[, and let us take VI G(w) = F(w
+
v1 )
-
°we define:
E
00.
.?4(u,r). Let us set
F(v 1 ) ,
so that G(O) = 0, and 1f/ = {wlllwll::S; ro - r}. Then G is bounded above by M - mover 1f/, and due to the proof of Lemma 1.1:
VI' E [0, IJ, Vw
(2.14)
E
e«,
IG(w)1 ~ e(M - m).
If Ilv - VIII::S; ro - r, then w = v - VI substituting back into (2.14): (2.15)
E
e1f/, with
I'
= Ilv - vIII/(ro - r), and
"Iv E .?4(v 1 ; ro - r),
If finally V2 E PJ(u; r), we take equidistant points UI = V1> U2' . . ., Un-1> Un=V2 on the segment [V I,V2]cPJ(u;r), in sufficient number so that Ilun - un+111 ::s; ro - r for I ::s; k ::s; n. From (2.15), we have IF(uk )
-
F(Uk+ 1)1 ~
M-m ro - r
Ilu k
-
uk+ 111
for
1~ k ~ n
and by adding each member, we obtain the local Lipschitz condition:
• Remark 2.1. Clearly the above proof gives an estimate of the Lipschitz constant for (ii). Corollary 2.5. Every l.s.c. convex function over a barrelled space (in particular a Banach space) is continuous over the interior of its effective domain.
Proof Let u E d~, which is assumed to be non-empty. By a suitable translation, we can shift u to the origin. Then let a > F(O). The set ~ = {u E VIF(u) ::s; a} is closed and convex. Also it is absorbent, since the restriction of F to every straight line passing through the origin is continuous in the neighbourhood of the origin (Cor. 2.3). Thus C{/ n -C{/ is a barrel and
14
FUNDAMENTALS OF CONVEX ANALYSIS
hence a neighbourhood of the origin. As Fis bounded by a over rrJ, Fis continuous at 0 (Prop. 2.5). 3. POINTWISE SUPREMUM OF CONTINUOUS AFFINE FUNCTIONS 3.1. Definition of T(V) As usual, V is a real I.c.s. The affine continuous functions over V are functions of the type v> t(v) + Ol, where t is a continuous linear functional over V and Ol E R. Definition 3.1. The set of functions F: V -7 R which are pointwise supremum of a family of continuous affine functions is denoted by r( V). r o(V) denotes the subset of FE r(V) other than the constants +00 and -00. It follows immediately from this definition that all the functions FE r(V) are convex and I.s.c. Conversely: Proposition 3.1. The following properties are equivalent to each other:
(i) FEr(V) (ii) F is a convex l.s.c. function from V into then F is identically equal to -00.
R, and if F takes the value -00
Proof. Note that the pointwise supremum of an empty family is -00 and that if the family under consideration is non-empty, F cannot take the value -00. Therefore we have (i) => (ii). Conversely, suppose that Fis a convex l.s.c. function of VintoR not taking the value -00. If F is the constant +00, it is the pointwise supremum of all the continuous affine functions of V into R. If FEr o(V), for every UE V and for every a < F(u) we will show that there is a continuous affine function of V into R whose value at u is located between a and F(u), which establishes the result. Now epi F is a closed convex set which does not contain the point (u,a). We can strictly separate them by a closed affine hyperplane :/f of V x R with equation:
(3.1)
:/f
= { (u, a) E
I
V x R f(u)
+ aa = {3 }
where t is a continuous non-zero linear functional over V, thus have:
(3.2)
f(u)
+ Olii <
{3
IY.
and (3 E R. We
CONVEX FUNCTIONS
(3.3)
\i(u, a) E epi F,
15
f(u) + aa > 13.
If F(u) < + 00, we can take u = u and a = F(u) which gives (X(F(u) - ii) > 0 where (X > O. When (3.2) and (3.3) are divided by (x, we obtain:
a < J!.. < F(u). (X - If(u) (X
(3.4)
The continuous affine function ~(X - ..!. (X t(.) (whose graph is nothing other than Yf) therefore answers the problem. If F(u) = +00, either (X ¥- 0 and we are back with the preceding case, or (X = O. In this case, (3.2) and (3.3) mean that the continuous affine function 13 - t(.) is >0 at u and <0 over dom F. The above case allows us to construct a continuous affine function everywhere less than F, e.g. y - m(.). Then for every c> 0, y - m(.) + c(f3 - t(.» is always a continuous affine function everywhere less than F, and it only remains to choose c sufficiently large so that
y - m(u)
(3.5) 3.2.
+
c(f3 - ((u)) > ii. •
r -regularization
Definition 3.2. Let F and G be two functions of V into R. The following are equivalent to each other: (i) G is the pointwise supremum of the continuous affine functions everywhere less than F; (ii) G is the largest minorant ofFin reV). G is then called the r-regularization ofF.
We shall now show the equivalence of (i) and (ii). Let us call G1 (or G2 ) the pointwise supremum of continuous affine functions (or functions of reV» everywhere less than F. Then G1 and G2 belong to reV), as the pointwise supremum of functions of reV), and G1 ~ G2 • Conversely, every continuous affine minorant of G2 is a function of reV) everywhere less than F. By definition it must be less than G1 everywhere. The functions G1 and G2 have the same set of affine continuous minorants. As they belong both to reV), they must coincide. _ In particular, if F E r( V), it coincides with its r-regularization. In general, we can construct the epigraph of the r-regularization as the closed convex hull of the epigraph of the function.
16
FUNDAMENTALS OF CONVEX ANALYSIS
Proposition 3.2. Let F: V -+ Rand G be its r-regularization. If there exists a continuous affinefunction everywhere less than F, we have: epi G = co epi
F.
Proof Let ¢ be a continuous affine function everywhere less than F. It is easy to see that the closed convex set co epi F is the epigraph of a convex I.s.c. function G. Since epi F c co epi Fe epi ¢
we have Fj» G ~ ¢, and therefore G E r(V). If, finally, G':;;; Fwhere G' E r(V) then epi G' is a closed convex set containing epi F, and hence containing co epi F= epi G, which means that G':;;; G. • Thus, for example, if d c V, the r-regularization of its indicator function XsI is none other than the indicator function of its closed convex envelope. We may wonder what ordering relations exist between F, its r-regularization G and its I.s.c. regularization F as defined in Section 2. Corollary 2.1 and Proposition 3.2 immediately give us the result:
Proposition 3.3. Let F: V -+ R, and G be its r-regularization.
(i) G ~ F ~ F; (ii) ifF is convex and admits a continuous affine minorant,
F = G.
4. POLAR FUNCTIONS
4.1. Definition In this paragraph, as in those which follow, we shall designate by Vand V* two vector spaces placed in duality by a bilinear pairing denoted by The spaces Vand V* will be supplied with topologies O'(V, V*) and O'(V*, V) which render them I.c.s. and Hausdorff. Let F be a function of V into R. If u* E V* and Q( E R, the continuous affine function u f-+ u,u*) - Q( is everywhere less than F if and only if
<.,.).
<
'VUE
V,
Q( ;;.:
- F(u),
or again:
(4.1)
Q( ;;.:
F*(u*)
if we agree to set
(4.2)
F*(u*) = sup {
17
CONVEX FUNCTIONS
The consideration of the continuous affine minorants of F thus leads us to define by (4.2) a function F*: V* --+ R. Definition 4.1. If F: V --+ R,formula (4.2) definesafunctionfrom V* into R, denotedby F*, and called the polar (or conjugate) function ofF.
It is obvious that in (4.2) we can confine ourselves to those u in dom F:
(4.3)
F*(u*) =
UE
sup {< u, u* ) - F(u) } domF
which enables us to see that F* is the pointwise supremum of the family of continuous affine functions (u,.> - F(u), for u E dom F, of V* into R. We therefore conclude that F* E F(V*), and in particular that F is l.s.c. and convex. Note that if F is the constant +00, dom F= 0 and F* is the constant -00. This immediately results in the following properties:
F*(O) = - inf F(u);
(4.4) (4.5) (4.6)
UEV
u r«
G, we have F*
~
G*;
(inf F.)* = sup F~, iEI'
iEI'
(sup F i )* :::;; inf Ft, iel
iet
for everyfamily (Fi ) , EI offunctions over V; (4.7)
(AF)*(u*) = AF*(u*/A),
for every A> 0; (4.8)
(F + IX)* = F* -
IX,
for every IX E R; (4.9) for every a E V, we denote by Fa the translatedfunction FaCv) = F(v - a). Then (Fa)*(u*)
=
F*(u*)
+
4.2. Bipolars. Dual convex functions We can repeat the process, thereby leading to the bipolar F**, which is now a function of V into R:
(4.10)
F**(u) = sup { ( u, u* >- F*(u*) }. u*eP'*
18
FUNDAMENTALS OF CONVEX ANALYSIS
From the above, F** E F(V), and we can compare F and F**, which are defined over the same space. The result is as follows:
Proposition 4.1. Let F be afunction of V into R. Then its bipolar F** is none other than its F-regularization. In particular, ifF E F(V), F** = F. Proof. By definition, the F-regularization of F is the pointwise supremum of all continuous affine minorants of F. We can restrict ourselves to those which are maximal, i.e., from (4.1), those functions:
(4.1 1)
u
>-+
-
F*(u*).
But their pointwise supremum is none other than F**, from (4.10). Whence the result. • The repetition of this process is limited:
Corollary 4.1. For every F: V
--?-
R,
we have F* = F***.
Proof. As F** is the F-regularization of F, we have F** (4.5): F*
~
~
F, and so from
F***.
Alternatively, from (4.10), for every u E C:
-
F**(u) ~ F*(u*)
whence F***(u*) = sup { ueV
-
F**(u)} ~ F*(u*).
We have seen that FE F(V) if and only if F** = F. We thus arrive at the following definition:
Definition 4.2. The polarity establishes a bijection between r( V) and F( V*). FE F( V) and G E V*) are said to be in duality if they corres pond in the bijection:
rc
(4.12)
F = G*
and
G = F*.
The constants too on V and too on V* are in duality. Thus FE Fo(V) if and only if F* E r o(V*): the polarity establishes a one-to-one correspondence between Fo(V) and Fo(V*).
19
CONVEX FUNCTIONS
4.3. Examples Let d be a subset of V and Xd its indicator function. Let us seek its polar:
<
X;(u*) = sup { u, u* ) - i. (u)} UEV
= sup
~;(u*)
UEd
It is a convex function, l.s.c. and positively homogeneous on V*, termed the support/unction of d. We have seen in Section 3 that X~= Xcod. In particular, d and co d will have the same support function. Here is an extremely useful example of dual convex functions. We will take for V a normed space, for v* its topological dual, and denote by 11.11 the norm of V and by 11.11* the norm of V*. Then Vand V* are in duality and we supply them with the weak topologies a(V, V*) and a(V*, V). We take an even function
F(u) =
Proof It is obvious that FEr o(V) and G E r o(V*). It is thus sufficient to prove that F* = G. To do this, we write: F*(u*) = ~~.P
= ~~~
{- cp( Ilull) } ~~? {-
=
~~~ {t
= (as
Ilu*ll* -
cp is even)
= Sup {t tER Remark 4.1. Let
IX, IX* E
cp(t)}
Ilu*ll* -
]1, 00[, satisfying
1/1X
+
1/1X* = 1.
It is easy to verify that the functions
= .!./t/", IX
1
20
FUNDAMENTALS OF CONVEX ANALYSIS
belong to Fo(R) and are conjugate. We therefore deduce that
F(u) =
.!.llulla
and
tX
are conjugate convex functions.
G(u*) =
~tX Ilu*II~·
_
Remark 4.2. Proposition 4.2 is no longer valid if tp is not even. In that case we have
where cpT is the conjugate function of the function CPt defined by (i = I, 2):
CPl(t) CP2(t)
R,
=
cp(ltl),
t
=
+
t < 0,
00,
E
t
0. •
~
Remark 4.3. More generally, Proposition 4.2 is valid under the following hypothesis: for all m > 0, the function t -+ cp(t) - mt attains its minimum at a point t ~ 0.
I
This condition is satisfied if cP is even or equally if cP
~
0 and cp(O)
= O.
_
5. SUBDIFFERENTIABILITY 5.1. Definition
<., .)
Henceforth V will designate a I.c.s., V* its topological dual, the bilinear canonical pairing over V x V* and F a mapping of V into R. We shall say that a continuous affine function t everywhere less than F is exact at the point u E V if t(u) = F(u). Necessarily, F(u) will be finite and t will have the form:
f(v) Necessarily (5.1)
t
= =
+ F(u) -
is maximal: its constant term is the greatest possible, whence:
F(u) -
=
- F*(u*).
Definition 5.1. A function F of V into R is said to be subdifferentiable at the point u E V if it has a continuous affine minorant which is exact at u. The slope u* E v* of such a minorant is called a subgradient of F at u, and the set of subgradients at u is called the subdifferential at u and is denoted of(u).
21
CONVEX FUNCTIONS
If Fis not subdifferentiable at u, we have of(u) = 0. We have the following characterization:
(5.2)
u* E of(u)
if and only if
F(u)
+ F(u) ::>; F(v),
"tIVE
is finite and V
If t is a continuous affine function bounded above by F, t is everywhere less than F**, the r-regularization. If, furthermore, t(u) = F(u), we obtain t(u)::>; F**(u)::>; F(u), whence t(u) = F**(u), from which these results follow: (5.3)
if of(u) i=
(5.4)
if
0,
F(u)
F(u) = F**(u),
=
F**(u),
of(u) = of**(u).
We now have a direct consequence of this definition; it already makes us anticipate the role of subdifferentiation in optimization problems:
(5.5)
F(u) = min F(v) veV
if and only if
0 E of(u).
We will now bring in polar functions. We take from (5.1) the following characterization: u*
Proposition 5.1. Let F be a function of V into R: and F* its polar. Then E of(u) if and only if:
(5.6)
F(u)
+ F*(u*) =
Proof The necessary condition has been established in (5.1). Conversely, if (5.6) is satisfied, the continuous affine function
<., u* ) + F(u)
-
is everywhere less than F (since its constant term is equal to -F*(u*)) and is exact at u. •
Corollary 5.1. The set of(u) (possibly empty) is convex and a(V*, V)-closed in V*. Proof By the definition (4.2) of F*, we always have F*(u*) -
~
- F(u).
Then Proposition 5.1 can be written: of(u)
= { u* E
V*
I F*(u*)
-
and the second term is closed and convex since F*
E
::>;; -
F(u) }
r(v*).
•
22
FUNDAMENTALS OF CONVEX ANALYSIS
Corollary 5.2. For every function F of V into R, we have (5.7)
u*
E
8F(u)
=> U E
8F*(u*).
If,furthermore, FE reV) we have (5.8)
u"
Proof Since F**
~
E
8F(u)
o¢>
8F*(u*).
UE
F, if u* E of(u), (5.6) entails that:
(5.9)
F**(u)
+ F*(u*)
::::;
Since the inverse inequality is always satisfied, (5.9) is in fact an equality, which means that u E of*(u*). If F E reV), F** = F, and (5.8) therefore is a result of (5.7). • For convex functions, we have at our disposal a very simple criterion for subdifferentiability: Proposition 5.2. Let F be a convex function of V into Ii,finite and continuous at the point u E V. Then of(v) '# 0 for all v~, and in particular of(u)'# 0.
Proof Since F is finite and continuous at u, it is bounded above oin a neighbourhood of u and so is finite and continuous at each point of ~ (Prop. 2.5). Thus it is sufficient to show that of(u) '# 0. Since F is convex, epi F is a convex subset of V x R. Since F is continuous, the interior of epi F is non-empty. To understand this, we need only take an open neighbourhood (9 of u over which F is bounded above by the constant c E R: the set (9 x Jc, +oo[ is an open subset of V x R contained in epi F: Since (u, F(u)) belongs to the boundary of epi F, we can separate it from by a closed affine hyperplane (Cor. 1.1). We thus obtain a supporting hyperplane Yf' of epi F, containing (u, F(u)). Let us write its equation:
epfp Yf' =
{(v, a) E V x R
I
= 13 } where u* E V*, ex and 13 E R
where the coefficients, not all zero, are linked by: V(v, a) E epi F, v, u* )
<
< u, u* )
+ aa
~
13
+ exF(u) = p.
If ex = 0, we will have 0, and dividing through by Ct:
Vv E dom F,
la -
lCt -
= F(u).
::::; F(v)
23
CONVEX FUNCTIONS
Finally,
v» E V,
U, - U*/rx
>+ F(u)
::;; F(v)
which proves that -u*/a E of(u), which is therefore non-empty.
•
Remark 5.1. Corollary 6.1 below shows that a proper l.s.c. convex function F defined on a complete normed space is subdifferentiable "almost everywhere" (more precisely, over a dense subset) inside dom F.
5.2. Relation with Gateaux-differentiability We shall now complete our demonstration that, at least in the context of convex functions, subdifferentiability constitutes a generalization of differentiability. Definition 5.2. Let F be a function of V into R. We call the limit as A-.,.O+, if it exists, of F(u
(5.10)
+ AV)
- F(u)
A
the directional derivative of Fat u in the direction v and denote it by F'(u;v). If there exists u* E V* such that:
F'(u; v) =
Vv E V,
we say that F is Gateaux-differentiable at u, call u* the Gateaux-differential at u of F, and denote it by F'(u). The uniqueness of the Gateaux-differential follows directly; it is characterized by: (5.11)
VVE
V,
- F(u) = 1l·Ifl F(u + AV) 1
A..,. 0+
IL
.
The case of convex functions is particularly interesting since the expression (5.10) is in that instance an increasing function of A. Thus, when A-.,. 0+, this expression always possesses a limit, which, however, can be ±oo. We will show that essentially the case of Gateaux-differentiability is the same as that of the uniqueness of the subgradient. Proposition 5.3. Let F be a convex function of V into R. If F is Gateauxdifferentiable at u E V, it is subdifferentiable at u and of(u) = {F'(u)}. Conversely, if at the point u E V, F is continuous and finite and has only one subgradient, then Fis Gateaux-differentiable at u and of(u) = {F'(u)}.
24
FUNDAMENTALS OF CONVEX ANALYSIS
Proof If F is Gateaux-differentiable at u, it is obvious that F'(u) indeed v E Vand w = v - u, we have F(u + w) - F(u) ~ F'(u; w) = F(v) - F(u) ~
F(u
+ AW) - F(u)
~
WE
E
of(u); if
0:
<
A w, u* )
on dividing through by A and passing to the limit we get:
F(u)
+ AF'(u; v)
~
F(u
+ AV).
Geometrically, this means that in V x R, the straight line: .2
= {(u + AV, F(u) + AF'(u; v» I A E R},
epiF
does not pass through the interior of epi F. But is an open convex set since epi F is convex, and it is non-empty since F is continuous and finite. From the Hahn-Banach theorem, there is a closed affine hyperplane yt' containing .2 which does not intersect It is easy to see that yt' is the graph of a continuous affine function everywhere less than F and exact at u. Since the subgradient u* of F at u has been supposed unique, the slope of yt' is u* and since yt' contains .2:
epiF.
F'(u; v)
=
which proves that F is Gateaux-differentiable at u with differential u*.
•
The convexity of a Gateaux-differentiable function may be characterized in the following way:
Proposition 5.4. Let F be a Gateaux-differentiable function of d c V, d convex, into R. Then the following are equivalent to each other:
(5.12) (5.13)
F is convex over d F(v)
~
F(u)
+
v u, vEd.
2S
CONVEX FUNCTIONS
Similarly, thefollowing are equivalent to each other: (5.14) F is strictly convex over d F(v) > F(u)
(5.15)
+
vu, v Ed, u "# v.
Proof From the preceding proposition, (5,12) implies (5.13). Conversely, the inequality (5.13) written with U and (l-A)u+Av(u,vEd,AE]O,l[) necessitates (5.18)
F(u)
~
F(u + A(V - u)) + A
Similarly,
(5.19) F(v)
~
Fiu + A(V - u)) + (1 - A)
+ A(V - u)), v - U ).
By multiplying (5.18) by (1- A), (5.19) by A and adding the inequalities, we obtain:
F((l - A)U
+ AV) :::;
(1 - A)F(u)
+ AF(v).
To show that (5.14) implies (5.15) we note that, as in Proposition 5.3,
F(u
+ A(V - u)) <
(1 - A)F(u)
+ AF(v),
if u,v E d and A E ]0,1 [. Whence, F being convex:
u ) :::; F(u
+ A(V ~
un - F(u) < F(v) -
F(u).
To show that (5.15) implies (5.14) we proceed as for the convexity, noting that (5.18) and (5.19) are strict inequalities if u "# v. • The convexity of a function is expressed by the monotonicity of its Gateauxdifferential: Proposition 5.5. Let F be a Gateaux-differentiable function of d c V, d convex, into R. It will be convex if and only if its differential F' is a monotone mapping of V into V*, i.e. if:
(5.20)
Proof From Proposition 5.3, if F is convex, F'(Ul) and F'(U2) are the subgradients of F at U1 and at U2:
u 1 ' F'(u 1 ) u z, F'(u z)
)
+ F(u 1 ) :::; + F(u z) :::;
Adding these terms together, we obtain (5.20).
F(u z) F(u 1 ) ·
26
FUNDAMENTALS OF CONVEX ANALYSIS
Conversely, if F is Gateaux-differentiable and if F' is monotone, for all u and v in V, the function ¢ of [0, I] into R defined by ¢>(A) = F(u
+
A(V - u))
is differentiable with derivative: ¢>'(A) =
u, F'(u
+ A(V
- u)) ).
But, because of (5.20), ¢' is increasing: thus ¢ is convex over [0, I] and in particular ¢>(A) ~ (1 - A)¢>(O)
+ A¢>(I),
IiAE[O,I],
which is the desired inequality: F((I - A)U
+ AV)
~
(1 - A)F(u)
+ AF(v),
5.3. Subdifferential calculus It only remains for us to examine to what degree the ordinary differential calculus can be extended into a subdifferential calculus. Some results follow directly: (5.21)
(5.22)
let F: V -+ R and A> 0. At every point U E V, we have J(AF)(u)
=
A JF(u),
let F1 and F2 : V -+ R. At every point u E V, we have J(F 1
+
F 2)(U) ~ JF 1 (u)
+
JF 2(U).
The equality in (5.22) is far from being always realized. Here, however, is a simple case where it holds: Proposition 5.6. If F1 and F2 E F(V), and dom F2 where F 1 is continuous, we have: (5.23)
liu
E
if there is a point UE dom F1 n
V,
Proof We have to show that the inverse inclusion of (5.22) is true, that is, that each u* E o(F1 + F2 )(u) can be decomposed into ut + uf, with uf E oF1 (u) and u~ E oF2 (u). Our hypothesis is that F1 and F2 have finite values at u and that (5.24)
v» E V,
27
CONVEX FUNCTIONS
Consider the convex sets in V x R: C 1 = { (v, a) I F 1 (v) -
The inequality (5.24) implies that they only have boundary points common. But C 1 is the epigraph of the function G: G( v)
= F 1 ( v) -
F 1 ( u)
+
In
which is convex and continuous at u. Thus C1 is a convex set with a non-empty interior. We can thus separate (;1 and Cz by a closed affine hyperplane :Yf (Cor. 1.1). It is easily verified that :Yf is "non-vertical" (see Prop. 5.2) and is thus the graph of a continuous affine function.
v
-+
where
v* E V* and ex
E
R.
The separation can be written:
v» E V. In setting v = u, we obtain ex = -
V,
VVE
V,
Hence -v* E oFz(u) and u* + v* E oF1(u). Whence we have the desired decomposition, u* = uT + ut with uT = u* + v* and u! = -v*. • Finally, after considering the subdifferential of a sum of functions, let us examine the subdifferential of a composite function. We take two I.c.s. Vand Y with topological duals V* and y* and a continuous linear mapping A: V --+ Y with transpose A *: y* --+ V*. Let FE F( Y); the function F 0 A: V --+ R belong to F(V). Proposition 5.7. Let there be a point Au where F is continuous and finite. Then for all points u of V, we have
a(p
(5.25)
Proof Letp*
E
0
A)(u) = A * aF(Au).
of(Au). By definition:
vp E Y,
< p - Au, p* )
+ F(Au)
:::; pep)
28
FUNDAMENTALS OF CONVEX ANALYSIS
and a fortiori: \:Iv
V,
E
>+ F 0
0
A(v)
0
A(v).
a(Fo A)(u) which proves that:
E
(5.26)
A* of(Au) c o(F
Conversely, take u* \:Iv
(5.27)
A(u) ~ F
+FoA(u)~FOA(v).
\:IvEV,
Thus A*p*
- Au, p*
E
E
0
A)(u).
a(F 0 A)(u).
V,
-
u, u*
>+ F 0
A(u) ~ F
Let us consider the affine subspace in Y x R.
2
= {(Av,
u, u*
>+ F 0
A(u)) I v E V }.
The inequality (5.27) shows that 2 and epi F only have boundary points in common. Since F has been assumed to be convex and continuous at Au, o
ePiF =I 0, and
there o is a closed affine hyperplane :!It' containing 2 which
does not intersect ep1'F. As usual, :!It' is non-vertical, and therefore is the graph of a continuous affine function of Y into R:
+
p ~
0(,
where
< Av, p*
>+
Since :!It' contains 2 : \:Iv
V,
E
(5.28)
0(
v» E V,
(5.29)
and
p* E Y* 0(
=
u, u*
o(ER.
>+ F
= .
0
A(u)
= F 0 A(u) -
o
Thus u* = A *p*. Finally, since :!It' does not intersect epfF: \:Ip (5.30)
Thusp* (5.31)
E
Y,
\:Ip E
E
Y,
+ F 0 A(u) - ~ F(p)
Convex Functions
Introduction This chapter assumes a basic knowledge of topological vector spaces. Moreover, we shall recall in Section I several fundamental aspects of this theory which will be constantly used in what follows. These reminders are in no way systematic and are centred on the notion of a convex set. We go on to consider convex functions (Sections 2-4) and their differentiability (Sections 5 and 6). All the vector spaces studied here are real.
1. CONVEX SETS AND THEIR SEPARAnON 1.1. Convex sets Let V be a vector space over R. If u and v are two points of V, u and v are called the endpoints of the line-segment denoted by [u,v] where
[u, v] = { AU
+ (1
- A)v
I°~
A ~ 1 }.
A set .91 c V is said to be convex if and only if for every pair of elements (u, v) ofd the segment [u,v] is contained in V. We know that a set .91 c V is convex if and only if for every finite subset of elements Ul, ••• , Un of .91, and for every family of real positive numbers AI, ..., An with sum unity, we have n
I
i= 1
AiUiEd.
The whole of the space V is convex and, conventionally, so is the empty set. Every intersection of convex sets is convex, but in general the union of convex sets is not convex. If .91 is any subset of V, the intersection of all the sets containing .91 is a convex set, and it is the smallest convex set containing d. It is called the convex hull ofd and is denoted by co d. It is also the set of all the convex combinations of the elements of .91, i.e., n
co d =
{
i~1
n
A;U;
I n E N, t~1
Ai
3
= 1,
Ai ~ 0, u, Ed, 1
~i~n}
4
FUNDAMENTALS OF CONVEX ANALYSIS
Let J't' be an affine hyperplane with equation I(u) linear form on V and IX E R. The sets { U
E V
{ UE
If(u) <
I
V f(u) ~
IX },
{ U
IX},
{ U
= IX where I
I E V If(u) ~
E
V f(u) >
is a non-zero
IX }, IX}
are called respectively open and closed half-spaces bounded by J't'. These are two convex sets which depend only on J't', and not on the particular I and IX chosen for its equation. 1.2. Separation of convex sets
We recall that a topological vector space (t.v.s.) is defined as a vector space V endowed with a topology for which the operations
(u, v) >-+ u
+v
(A., u) >-+ A.U
of V x V into V, of R x V into V
are continuous. The neighbourhoods of any point may then be deduced from those of the origin by translation. A t.v.s. is said to be a locally convex space (I.c.s.) if the origin possesses a fundamental system of convex neighbourhoods. This is the case with normed spaces: it is sufficient to take the set of neighbourhoods formed by the balls centred on the origin. All the usual t.v.s. encountered in analysis are locally convex. Now let V be a t.v.s. and J't' an affine hyperplane with equation I(u) = IX where I is a non-zero linear functional on V and IX E R. It can be shown that the set J't' is topologically closed if and only if the function I is continuous. Under these conditions the open (closedjhalf-spaces determined by J't' will be topologically open (closed). In a t.V.S. V, the closure of a convex set is convex and the interior of a convex set is also convex (possibly empty). More generally, if d c V is convex, if u E d (the interior of d) and if...!' E.si (the closure of d), then [u,v[ c d, from which we deduce that .si = d, whenever d 'f; 0. This suggests to us the introduction of the following definition: a point u E d will be called internal if every line passing through u meets d in a segment [v 10 V2] sueh that u E ]VI' V2[' Hence every interior point is internal, and by the above argument, if .si 'f; 0 every internal point is interior. If d is any subspace of V, the intersection of all the closed convex subsets containing d is the smallest closed convex subset containing d. It is also the closure of the convex hull of d (and not the convex hull of the closure!); it is called the closed convex hull of d and is denoted by co d.
5
CONVEX FUNCTIONS
An affine hyperplane Yf is said to separate (strictly separate) two sets .91 and ffl if each of the closed (open) half-spaces bounded by Yf contains one of them. This may be written analytically as follows. If t(u) = ex is the equation of Yf, then we have t(u) ~ ex, Vu E d,
t(v)
~
ex, "Iv E ffl,
for separation, t(u) < (x, Vu E sl/, t(v) > lX, "Iv E ffl. for strict separation. We now recall the Hahn-Banach theorem in its geometric form, and its consequences for the separation of convex sets. Naturally it is in the context of I.c.s. that the most precise results are obtained.
Hahn-Banach Theorem. Let V be a real t.v.s., .91 an open non-empty convex set, and vi{ a non-empty affine subspace which does not intersect d. Then there exists a closed affine hyperplane Yf which contains vi{ and does not intersect d. Corollary 1.1. Let V be a real t.v.s., .91 an open non-empty convex set, ffl a non-empty convex set which does not intersect d. Then there exists a closed affine hyperplane Yf which separates .91 and fJ8. Corollary 1.2. Let V be a reall.c.s., CC and ffl two non-empty disjoint convex sets with one compact and the other closed. Then there exists a closed affine hyperplane Yf which strictly separates CC and ffl. Here is an example of application of Corollary 1.1. Let .91 be a subset of V and Yf a closed affine hyperplane which contains at least one point u E .91, such that .91 is completely contained in one of the closed half-spaces determined by :Yl'; we say that :Yl' is a supporting hyperplane and u a supporting point of d. Then
Corollary 1.3. In a real t.V.S. V, let .91 be a convex set with non-empty interior. Then every boundary point ofd is a supporting point ofd. An application of Corollary 1.2 is:
Corollary 1.4. In a real l.c.s. V, every closed convex set is the intersection of the closed half-spaces which contain it. All these results have a fundamental importance in analysis because they allow a convenient theory of duality. Thus, if V is a Hausdorff I.c.s., the Hahn-Banach theorem allows us to assert the existence of non-zero continuous linear forms over V: it is sufficient to consider two points Ul and U2 of V, and to separate then by a closed affine hyperplane :Yl' (Cor. 1.2); if the equation of
6 :If is t(u)
FUNDAMENTALS OF CONVEX ANALYSIS
= el, the
non-zero linear form t is continuous since :Yf is closed and
t(Ut) =I t(U2)' The vector space V* of continuous linear functionals over V is
said to be the topological dual, or more simply the dual of V. The elements of V* will be, in general, denoted by u* or v*, and
Mazur's Lemma. Let V be a normed space and (Un)nEN a sequence converging weakly to it. Then there is a sequence of convex combinations (vn)nEN such that N
where
L
k=n
Ak = I and Ak ~ 0,
which converges to u in norm. Proof For every n E N, U belongs to the weak closure of Uf=n {Uk}, and afortiori, to the weak closure of co {Uk}' But this is exactly the weakly closed convex hull of U~=n {Uk} which coincides with its closed convex hull by Corollary 104. Finally,
ur=n
ro
U E co
U
{Uk},
VnEN,
k =n
and it suffices to choose Vn
E
co U~=n
{ud such that Ilvn - unll ~ lin.
•
7
CONVEX FUNCTIONS
1.3. Analytical form of the Hahn-Banach theorem The analytical form of the Hahn-Banach theorem is more precise than the geometrical form. We consider a real vector space Vand a sub-linear function j over V, i.e. a mappingj of V into R which satisfies
j(AU) = Aj(U), j(u + v) ~ j(u)
vu e V, I::/A > 0,
+ j(v),
I::/(u, v) E V
X
v.
Hahn-Banach Theorem. Let V be a real vector space, j a sub-linear function over V, .It a vector subspace of V, t a linear functional over .It which is everywhere less than j. Then there exists a linear functional? over V which extends t and is everywhere less than j. Corollary 1.5. Let V be a normed space, .It a topological vector subspace, t a continuous linear functional over .It. Then t can be extended into a continuous linear functional over V with the same norm. 2. CONVEX FUNCTIONS
2.1. Definitions As before, we take a real vector space Vand consider mappings of d c V into R, that is, the values +ro and -ro are allowed to the functions under consideration.
Definition 2.1. Letd be a convex subspace of V, and Fa mapping of d into R. F is said to be convex if, for every u and v in d, we have:
(2.1)
F(AU
+ (1 - A)V)
~
AF(u)
+ (1 - A)F(v)
I::/A
E
[0, IJ
whenever the right-hand side is defined. The inequality (2.1) must therefore be valid unless F(u) = -F(v) = ±ro. By induction, it can be shown that if Fis convex, for every finite set Ul, .•• , u; of points of Vand for every family A1 , ••• , An of real positive numbers with sum unity, then
(2.2) whenever the right-hand side is defined.
8
FUNDAMENTALSOFCONVEXANALY~S
It is easy to see that, if F: V -+ II is convex, the sections
(2.3)
{ u I F(u) ::;;; a} and {u I F(u) < a}
are, for each a E II, convex sets of V. The converse is, however, false. For instance, if F is convex and if ¢: II -+ II is an increasing function then ¢ 0 F: V -+ II will also have convex sections but will not be convex in general. For every mapping F: V -+ II, we call the section:
(2.4)
dom F = { u I F(u) <
+
00 }
the effective domain of F. The effective domain of a convex function is thus convex. Why do we allow the value +00 ? If F is a mapping of d c V into R, we can associate with it the function F defined throughout V by:
(2.5)
I
~(u) = F(u) if u E .9'1, F(u) = + 00 if u¢d.
Thus F is convex if and only if d c V is convex and F: d -+ R is convex. In the theory of convex functions, because of this extension by +00, we need only consider those functions defined everywhere. There is a further advantage. If d is a subset of V, the indicatorfunction XJI1 of d is defined as:
(2.6)
X,r,1(u) I X,r,1(u)
= 0 =
+
if 00
u e sd, if u ¢ d.
Clearly d is a convex subset if and only if XJI1 is a convex function. Thus the study of convex sets is naturally reduced to the study of convex functions. On the other hand, convex functions which assume the value -00 are very special. If F(u) = -00, then on every half-line starting from ii, either F is identically equal to -00, or Ftakes the value -00 between ii and a point ii, any value at ii, and +00 beyond ii. To distinguish these very special cases, we shall say that a convex function F of V in II is proper if it nowhere takes the value -00 and is not identically equal to +00.
Definition 2.2. The epigraph of a function F: V......,.. II is the set: (2.7)
I
epi F = {(u, a) E V x R f(u) ::;;; a}.
It is the set of points of V x R which lie above the graph of F. The projection of epi F on V is none other than dom F. The epigraph will be found a most
9
CONVEX FUNCTIONS
useful concept in the study of convex functions because of the following result: Proposition 2.1. A function F: V -)0 R is convex
if and only if its epigraph
is convex. Proof Let F be convex and take (u,a) and (v,b) in epi F. Then, necessarily, F(u) ~ a < +00 and F(v) ~ b < +00, and for all A E [0,1] from (2.1) we have F(AU + (1 - A)V)
~
AF(u) + (1 - A)F(v)
~
Aa + (1 - A)b,
which means precisely that A(U, a) + (1 - A)(v,b) E epi F. Conversely, let epi F be convex. Its projection dom F is therefore convex and it is sufficient to verify (2.1) over dom F. Let us therefore take u and v in dom F, a ~ F(u) and b ~ F(v). By hypothesis, A(u,a) + (1 - A)(v,b) E epi F for every A E [0,1] so that:
F(AU
+ (1
- A)V)
~
Aa + (1 - A)b.
If F(u) and F(v) are finite, it is sufficient to take a = F(u) and b = F(v). If either F(u) or F(v) is equal to -00 it is sufficient to allow a or b to tend to -00 and (2.1) is obtained in both cases. _ It remains for us to consider the usual manipulations of convex functions. The results, which are trivial, have been collected together in the following proposition. In particular we infer from them that the set of convex functions is a convex cone. Proposition 2.2. (i) IfF: V -)0 R is convex and if A is a realpositive number, then AF is convex. (ii) IfF and G are convexfunctionsfrom V into R, then F + G is convex. We stipulate that (F + G)(u) = +00 if F(u) = -G(u) = too. (iii) !f(FI)leI is any family of convex functions of V into R, their pointwise supremum F = sup! eI F, is convex. Also, let us recall the concept of a strictly convex function. Definition 2.3. Let d be a convex set of V and F a mapping of d into R. F is said to be strictly convex if it is convex and the strict inequality holds in (2.1), 'Vu,vEd,u:;=vand'VAE ]0,1[. 2.2. Lower semi-continuous functions We now pass on to topological properties, so V will be a real I.c.s. We recall that a function F: V -?- R is said to be lowersemi-continuous on V (I.s.c.),
10
FUNDAMENTALS OF CONVEX ANALYSIS
if it satisfies the two equivalent conditions:
(2.8) (2.9)
VaER,
{ U
E
V I F(u) ~ a}
is closed,
lim F(u) ~ F(u). u-u
WiE V,
Naturally F will be called upper semi-continuous (u.s.c.) if -F is I.s.c. Thus for example the indicator function x",i') of a set .91 c V will be I.s.c. (or u.s.c.) if and only if .91 is closed (or open). More generally, we can at once state: Proposition 2.3. A function F: V ~ R is l.s.c. if and only if its epigraph is closed.
Proof Let us define on V x R a function ¢ by ¢(u, a) = F(u) - a. Then the statements that F is I.c.s. on V and that ¢ is I.c.s. on V x R are equivalent to each other. Now for every r E R, the section ¢([-oo, r Dis the set obtained from epi F by the translation of vector (r,O) and it is therefore closed if and only if epi F is closed. • Let us further recall that a pointwise supremum of I.s.c. functions is l.s.c, This then leads to the following definition: for every mapping F: V ~ R, the largest I.s.c. minorant of F will be called the I.s.c. regularization of F and will be denoted by P. It exists as the pointwise supremum of those I.s.c. functions everywhere less than F, and is characterized by: Corollary 2.1. Let F: V ~ Rand P be its l.s.c. regularization. We have epi
(2.10) (2.11 )
VUE V,
F= F(u)
epi F, = lim F(v). V-'u
Proof Since P is a I.s.c. function everywhere less than F, epi P is a closed set containing epi F, and hence epi F. Conversely, epi F is an epigraph. (1) Let G be a function such that epi G = epi F; it is a I.s.c. function everywhere less than F, hence G :::; P and epi G = epi F contains epi P. Thus (2.10) is proved and (2.11) follows directly from it. • (1) If (u,a) E epi F, there exists a filter (u.,a.) E epi F which converges to (u,a). If b > a, then for some suitable ()(, a.";; b and since F(u.)";; a. it follows that (u.,b) E epi F, and in the limit (u,b) E epi F. _ _ The intersection of epi Fwith a straight line {u} x R is an empty set or an interval [a, +00[; if we set G(u) = +00 in the former and G(u) = a in the latter case, we see that epi G = epi F.
11
CONVEX FUNCTIONS
The case of convex functions assumes a special interest since lower semicontinuity exists when the topology of V is weakened. This is an extremely valuable property. Corollary 2.2. Every l.s.c. convex function F of V in 'R remains l.s.c. when V is supplied with its weak topology a(V, V'). In fact, the epigraph of F being convex, this is equivalent to saying that it is closed and hence weakly closed (Cor. 1.4). The case of l.s.c. convex functions which assume the value -00 (improper functions) is more specialized: Proposition 2.4. If F: V -+ 'R is a l.s.c. convex function and assumes the value -00, it cannot take any finite value.
Proof Let us suppose that there exists u E V a point such that F(u) E R. We then take a E R such that a < F(u), and we strictly separate (u,a) from the closed convex set epi F. Then there exists a continuous non-zero linear form t over V and pER such that:
V(u,a)EepiF,
(2.12)
f(u)
+ IXQ <
f(u)
+ aa.
Taking u = ii and a = F(u), we get IX(F(u) - a) > 0, and hence IX> O. The two members of (2.12) can thus be divided by IX:
Vu E V,
(2.13)
-.!.f(u - u) IX
+
Ii
< F(u),
which is impossible, since the first member is everywhere finite and the second member takes the value -00 at one point at least. _ 2.3. Continuity of convex functions The study of the continuity of convex functions is based on the following lemma: Lemma 2.1. If in the neighbourhood of a point u E V, a convex function F is bounded above by a finite constant, then F is continuous at u.
Proof We reduce the problem by translation to the case where u = 0 and F(O) = O. Let "Y be a neighbourhood of the origin such that F(v) ~ a < +00 for all v of "Y. Let us define "If/' = "Y n -"Y (which is a symmetric neighbourhood of the origin), and let us take s E ]0, 1[. If v E eir, we have, due to the convexity of F:
!:.E"Y B
- !:.e E
'
"Y
'
+ eF(v/e) ~ ea,
hence
F(v) ~ (1 - e)F(O)
hence
F(v) ~ (1 + e)F(O) - eF(- v/e) ~ - ea.
12
FUNDAMENTALS OF CONVEX ANALYSIS
Then IF(v)l:::; ea for every v in e"f/l, whence we have the required continuity. • A general conclusion may be drawn: Proposition 2.5. Let F: V ~ R be a convex function. The following statements are equivalent to each other: (i) there exists a non-empty open set (IJ on which F is not everywhereequal to
and is boundedabove by a constant a < +00 ; (ii) F is a properfunction, and it is continuousover the interior of its effective domain, which is non-empty.
-00
0
~
Proof Clearly (ii) implies (i). Conversely, if (i) is true, (IJ c dom F. Let us take u E (IJ such that F(u) > -00. From Lemma 2.1, Fwill be continuous at u, and hence finite in a neighbourhood of u, and hence proper. For ~ry v E d~, there exists p > 1 such that w = u + p(v - u) also belongs to dom F. The homothety h with centre wand ratio 1 - lip transforms u into v and (IJ into an open set h«(IJ) containing v. For every v' E h«(IJ), we have by convexity:
F(v')::::; p -1 Foh-1(v') +.!.F(w)::::; p -1 a +.!-F(w). p p P P o .-----.....
To sum up: every point v E dom F possesses a neighbourhood h«(IJ) where F is bounded above by a finite constant. From Lemma 2.1, F is continuous at v, which concludes the proof. • We can use this result more precisely in numerous special cases-spaces of finite dimension, normed spaces and barrelled spaces. Corollary 2.3. Every proper convex function on a space offinite dimension is continuouson the interior ofits effective domain. o .-----.....
Proof If dom F is non-empty, it contains n + 1 affinely independent points u1 , 1 :::; t « n + 1, where n is the dimension of V. From the inequality defining convexity, F is bounded above by maXl,,;t";n+lF(Ul) over the open set:
Vi} •
Corollary 2.4. Let F be a proper convex function on a normed space. The following properties are equivalent to each other: (i) the~e exists a non-empty open set over which F is bounded above; (ii) dQ;F:f: 0 and F is locally Lipschitz there.
13
CONVEX FUNCTIONS
Proof It is obvious that (ii) over ~F
=>
(i). Conversely, if (i) is true, F is continuous
(Prop. 2.5). Taking u E .?4(u; r)
~,
for every r >
= {v Illv
- ull
By the continuity of F at u, there exists an r« >
"Iv
E
-
.?4(u; ro),
< m
00
~ r}.
°
~
such that: F(v) ~ M < +
Let us now suppose that r E ]O,ro[, and let us take VI G(w) = F(w
+
v1 )
-
°we define:
E
00.
.?4(u,r). Let us set
F(v 1 ) ,
so that G(O) = 0, and 1f/ = {wlllwll::S; ro - r}. Then G is bounded above by M - mover 1f/, and due to the proof of Lemma 1.1:
VI' E [0, IJ, Vw
(2.14)
E
e«,
IG(w)1 ~ e(M - m).
If Ilv - VIII::S; ro - r, then w = v - VI substituting back into (2.14): (2.15)
E
e1f/, with
I'
= Ilv - vIII/(ro - r), and
"Iv E .?4(v 1 ; ro - r),
If finally V2 E PJ(u; r), we take equidistant points UI = V1> U2' . . ., Un-1> Un=V2 on the segment [V I,V2]cPJ(u;r), in sufficient number so that Ilun - un+111 ::s; ro - r for I ::s; k ::s; n. From (2.15), we have IF(uk )
-
F(Uk+ 1)1 ~
M-m ro - r
Ilu k
-
uk+ 111
for
1~ k ~ n
and by adding each member, we obtain the local Lipschitz condition:
• Remark 2.1. Clearly the above proof gives an estimate of the Lipschitz constant for (ii). Corollary 2.5. Every l.s.c. convex function over a barrelled space (in particular a Banach space) is continuous over the interior of its effective domain.
Proof Let u E d~, which is assumed to be non-empty. By a suitable translation, we can shift u to the origin. Then let a > F(O). The set ~ = {u E VIF(u) ::s; a} is closed and convex. Also it is absorbent, since the restriction of F to every straight line passing through the origin is continuous in the neighbourhood of the origin (Cor. 2.3). Thus C{/ n -C{/ is a barrel and
14
FUNDAMENTALS OF CONVEX ANALYSIS
hence a neighbourhood of the origin. As Fis bounded by a over rrJ, Fis continuous at 0 (Prop. 2.5). 3. POINTWISE SUPREMUM OF CONTINUOUS AFFINE FUNCTIONS 3.1. Definition of T(V) As usual, V is a real I.c.s. The affine continuous functions over V are functions of the type v> t(v) + Ol, where t is a continuous linear functional over V and Ol E R. Definition 3.1. The set of functions F: V -7 R which are pointwise supremum of a family of continuous affine functions is denoted by r( V). r o(V) denotes the subset of FE r(V) other than the constants +00 and -00. It follows immediately from this definition that all the functions FE r(V) are convex and I.s.c. Conversely: Proposition 3.1. The following properties are equivalent to each other:
(i) FEr(V) (ii) F is a convex l.s.c. function from V into then F is identically equal to -00.
R, and if F takes the value -00
Proof. Note that the pointwise supremum of an empty family is -00 and that if the family under consideration is non-empty, F cannot take the value -00. Therefore we have (i) => (ii). Conversely, suppose that Fis a convex l.s.c. function of VintoR not taking the value -00. If F is the constant +00, it is the pointwise supremum of all the continuous affine functions of V into R. If FEr o(V), for every UE V and for every a < F(u) we will show that there is a continuous affine function of V into R whose value at u is located between a and F(u), which establishes the result. Now epi F is a closed convex set which does not contain the point (u,a). We can strictly separate them by a closed affine hyperplane :/f of V x R with equation:
(3.1)
:/f
= { (u, a) E
I
V x R f(u)
+ aa = {3 }
where t is a continuous non-zero linear functional over V, thus have:
(3.2)
f(u)
+ Olii <
{3
IY.
and (3 E R. We
CONVEX FUNCTIONS
(3.3)
\i(u, a) E epi F,
15
f(u) + aa > 13.
If F(u) < + 00, we can take u = u and a = F(u) which gives (X(F(u) - ii) > 0 where (X > O. When (3.2) and (3.3) are divided by (x, we obtain:
a < J!.. < F(u). (X - If(u) (X
(3.4)
The continuous affine function ~(X - ..!. (X t(.) (whose graph is nothing other than Yf) therefore answers the problem. If F(u) = +00, either (X ¥- 0 and we are back with the preceding case, or (X = O. In this case, (3.2) and (3.3) mean that the continuous affine function 13 - t(.) is >0 at u and <0 over dom F. The above case allows us to construct a continuous affine function everywhere less than F, e.g. y - m(.). Then for every c> 0, y - m(.) + c(f3 - t(.» is always a continuous affine function everywhere less than F, and it only remains to choose c sufficiently large so that
y - m(u)
(3.5) 3.2.
+
c(f3 - ((u)) > ii. •
r -regularization
Definition 3.2. Let F and G be two functions of V into R. The following are equivalent to each other: (i) G is the pointwise supremum of the continuous affine functions everywhere less than F; (ii) G is the largest minorant ofFin reV). G is then called the r-regularization ofF.
We shall now show the equivalence of (i) and (ii). Let us call G1 (or G2 ) the pointwise supremum of continuous affine functions (or functions of reV» everywhere less than F. Then G1 and G2 belong to reV), as the pointwise supremum of functions of reV), and G1 ~ G2 • Conversely, every continuous affine minorant of G2 is a function of reV) everywhere less than F. By definition it must be less than G1 everywhere. The functions G1 and G2 have the same set of affine continuous minorants. As they belong both to reV), they must coincide. _ In particular, if F E r( V), it coincides with its r-regularization. In general, we can construct the epigraph of the r-regularization as the closed convex hull of the epigraph of the function.
16
FUNDAMENTALS OF CONVEX ANALYSIS
Proposition 3.2. Let F: V -+ Rand G be its r-regularization. If there exists a continuous affinefunction everywhere less than F, we have: epi G = co epi
F.
Proof Let ¢ be a continuous affine function everywhere less than F. It is easy to see that the closed convex set co epi F is the epigraph of a convex I.s.c. function G. Since epi F c co epi Fe epi ¢
we have Fj» G ~ ¢, and therefore G E r(V). If, finally, G':;;; Fwhere G' E r(V) then epi G' is a closed convex set containing epi F, and hence containing co epi F= epi G, which means that G':;;; G. • Thus, for example, if d c V, the r-regularization of its indicator function XsI is none other than the indicator function of its closed convex envelope. We may wonder what ordering relations exist between F, its r-regularization G and its I.s.c. regularization F as defined in Section 2. Corollary 2.1 and Proposition 3.2 immediately give us the result:
Proposition 3.3. Let F: V -+ R, and G be its r-regularization.
(i) G ~ F ~ F; (ii) ifF is convex and admits a continuous affine minorant,
F = G.
4. POLAR FUNCTIONS
4.1. Definition In this paragraph, as in those which follow, we shall designate by Vand V* two vector spaces placed in duality by a bilinear pairing denoted by The spaces Vand V* will be supplied with topologies O'(V, V*) and O'(V*, V) which render them I.c.s. and Hausdorff. Let F be a function of V into R. If u* E V* and Q( E R, the continuous affine function u f-+ u,u*) - Q( is everywhere less than F if and only if
<.,.).
<
'VUE
V,
Q( ;;.:
- F(u),
or again:
(4.1)
Q( ;;.:
F*(u*)
if we agree to set
(4.2)
F*(u*) = sup {
17
CONVEX FUNCTIONS
The consideration of the continuous affine minorants of F thus leads us to define by (4.2) a function F*: V* --+ R. Definition 4.1. If F: V --+ R,formula (4.2) definesafunctionfrom V* into R, denotedby F*, and called the polar (or conjugate) function ofF.
It is obvious that in (4.2) we can confine ourselves to those u in dom F:
(4.3)
F*(u*) =
UE
sup {< u, u* ) - F(u) } domF
which enables us to see that F* is the pointwise supremum of the family of continuous affine functions (u,.> - F(u), for u E dom F, of V* into R. We therefore conclude that F* E F(V*), and in particular that F is l.s.c. and convex. Note that if F is the constant +00, dom F= 0 and F* is the constant -00. This immediately results in the following properties:
F*(O) = - inf F(u);
(4.4) (4.5) (4.6)
UEV
u r«
G, we have F*
~
G*;
(inf F.)* = sup F~, iEI'
iEI'
(sup F i )* :::;; inf Ft, iel
iet
for everyfamily (Fi ) , EI offunctions over V; (4.7)
(AF)*(u*) = AF*(u*/A),
for every A> 0; (4.8)
(F + IX)* = F* -
IX,
for every IX E R; (4.9) for every a E V, we denote by Fa the translatedfunction FaCv) = F(v - a). Then (Fa)*(u*)
=
F*(u*)
+
4.2. Bipolars. Dual convex functions We can repeat the process, thereby leading to the bipolar F**, which is now a function of V into R:
(4.10)
F**(u) = sup { ( u, u* >- F*(u*) }. u*eP'*
18
FUNDAMENTALS OF CONVEX ANALYSIS
From the above, F** E F(V), and we can compare F and F**, which are defined over the same space. The result is as follows:
Proposition 4.1. Let F be afunction of V into R. Then its bipolar F** is none other than its F-regularization. In particular, ifF E F(V), F** = F. Proof. By definition, the F-regularization of F is the pointwise supremum of all continuous affine minorants of F. We can restrict ourselves to those which are maximal, i.e., from (4.1), those functions:
(4.1 1)
u
>-+
F*(u*).
But their pointwise supremum is none other than F**, from (4.10). Whence the result. • The repetition of this process is limited:
Corollary 4.1. For every F: V
--?-
R,
we have F* = F***.
Proof. As F** is the F-regularization of F, we have F** (4.5): F*
~
~
F, and so from
F***.
Alternatively, from (4.10), for every u E C:
F**(u) ~ F*(u*)
whence F***(u*) = sup { ueV
F**(u)} ~ F*(u*).
We have seen that FE F(V) if and only if F** = F. We thus arrive at the following definition:
Definition 4.2. The polarity establishes a bijection between r( V) and F( V*). FE F( V) and G E V*) are said to be in duality if they corres pond in the bijection:
rc
(4.12)
F = G*
and
G = F*.
The constants too on V and too on V* are in duality. Thus FE Fo(V) if and only if F* E r o(V*): the polarity establishes a one-to-one correspondence between Fo(V) and Fo(V*).
19
CONVEX FUNCTIONS
4.3. Examples Let d be a subset of V and Xd its indicator function. Let us seek its polar:
<
X;(u*) = sup { u, u* ) - i. (u)} UEV
= sup
~;(u*)
UEd
It is a convex function, l.s.c. and positively homogeneous on V*, termed the support/unction of d. We have seen in Section 3 that X~= Xcod. In particular, d and co d will have the same support function. Here is an extremely useful example of dual convex functions. We will take for V a normed space, for v* its topological dual, and denote by 11.11 the norm of V and by 11.11* the norm of V*. Then Vand V* are in duality and we supply them with the weak topologies a(V, V*) and a(V*, V). We take an even function
F(u) =
Proof It is obvious that FEr o(V) and G E r o(V*). It is thus sufficient to prove that F* = G. To do this, we write: F*(u*) = ~~.P
= ~~~
{
=
~~~ {t
= (as
Ilu*ll* -
cp is even)
= Sup {t tER Remark 4.1. Let
IX, IX* E
cp(t)}
Ilu*ll* -
]1, 00[, satisfying
1/1X
+
1/1X* = 1.
It is easy to verify that the functions
= .!./t/", IX
1
20
FUNDAMENTALS OF CONVEX ANALYSIS
belong to Fo(R) and are conjugate. We therefore deduce that
F(u) =
.!.llulla
and
tX
are conjugate convex functions.
G(u*) =
~tX Ilu*II~·
_
Remark 4.2. Proposition 4.2 is no longer valid if tp is not even. In that case we have
where cpT is the conjugate function of the function CPt defined by (i = I, 2):
CPl(t) CP2(t)
R,
=
cp(ltl),
t
=
+
t < 0,
00,
E
t
0. •
~
Remark 4.3. More generally, Proposition 4.2 is valid under the following hypothesis: for all m > 0, the function t -+ cp(t) - mt attains its minimum at a point t ~ 0.
I
This condition is satisfied if cP is even or equally if cP
~
0 and cp(O)
= O.
_
5. SUBDIFFERENTIABILITY 5.1. Definition
<., .)
Henceforth V will designate a I.c.s., V* its topological dual, the bilinear canonical pairing over V x V* and F a mapping of V into R. We shall say that a continuous affine function t everywhere less than F is exact at the point u E V if t(u) = F(u). Necessarily, F(u) will be finite and t will have the form:
f(v) Necessarily (5.1)
t
= =
is maximal: its constant term is the greatest possible, whence:
F(u) -
- F*(u*).
Definition 5.1. A function F of V into R is said to be subdifferentiable at the point u E V if it has a continuous affine minorant which is exact at u. The slope u* E v* of such a minorant is called a subgradient of F at u, and the set of subgradients at u is called the subdifferential at u and is denoted of(u).
21
CONVEX FUNCTIONS
If Fis not subdifferentiable at u, we have of(u) = 0. We have the following characterization:
(5.2)
u* E of(u)
if and only if
F(u)
+ F(u) ::>; F(v),
"tIVE
is finite and V
If t is a continuous affine function bounded above by F, t is everywhere less than F**, the r-regularization. If, furthermore, t(u) = F(u), we obtain t(u)::>; F**(u)::>; F(u), whence t(u) = F**(u), from which these results follow: (5.3)
if of(u) i=
(5.4)
if
0,
F(u)
F(u) = F**(u),
=
F**(u),
of(u) = of**(u).
We now have a direct consequence of this definition; it already makes us anticipate the role of subdifferentiation in optimization problems:
(5.5)
F(u) = min F(v) veV
if and only if
0 E of(u).
We will now bring in polar functions. We take from (5.1) the following characterization: u*
Proposition 5.1. Let F be a function of V into R: and F* its polar. Then E of(u) if and only if:
(5.6)
F(u)
+ F*(u*) =
Proof The necessary condition has been established in (5.1). Conversely, if (5.6) is satisfied, the continuous affine function
<., u* ) + F(u)
-
is everywhere less than F (since its constant term is equal to -F*(u*)) and is exact at u. •
Corollary 5.1. The set of(u) (possibly empty) is convex and a(V*, V)-closed in V*. Proof By the definition (4.2) of F*, we always have F*(u*) -
~
- F(u).
Then Proposition 5.1 can be written: of(u)
= { u* E
V*
I F*(u*)
-
and the second term is closed and convex since F*
E
::>;; -
F(u) }
r(v*).
•
22
FUNDAMENTALS OF CONVEX ANALYSIS
Corollary 5.2. For every function F of V into R, we have (5.7)
u*
E
8F(u)
=> U E
8F*(u*).
If,furthermore, FE reV) we have (5.8)
u"
Proof Since F**
~
E
8F(u)
o¢>
8F*(u*).
UE
F, if u* E of(u), (5.6) entails that:
(5.9)
F**(u)
+ F*(u*)
::::;
Since the inverse inequality is always satisfied, (5.9) is in fact an equality, which means that u E of*(u*). If F E reV), F** = F, and (5.8) therefore is a result of (5.7). • For convex functions, we have at our disposal a very simple criterion for subdifferentiability: Proposition 5.2. Let F be a convex function of V into Ii,finite and continuous at the point u E V. Then of(v) '# 0 for all v~, and in particular of(u)'# 0.
Proof Since F is finite and continuous at u, it is bounded above oin a neighbourhood of u and so is finite and continuous at each point of ~ (Prop. 2.5). Thus it is sufficient to show that of(u) '# 0. Since F is convex, epi F is a convex subset of V x R. Since F is continuous, the interior of epi F is non-empty. To understand this, we need only take an open neighbourhood (9 of u over which F is bounded above by the constant c E R: the set (9 x Jc, +oo[ is an open subset of V x R contained in epi F: Since (u, F(u)) belongs to the boundary of epi F, we can separate it from by a closed affine hyperplane (Cor. 1.1). We thus obtain a supporting hyperplane Yf' of epi F, containing (u, F(u)). Let us write its equation:
epfp Yf' =
{(v, a) E V x R
I
= 13 } where u* E V*, ex and 13 E R
where the coefficients, not all zero, are linked by: V(v, a) E epi F, v, u* )
<
< u, u* )
+ aa
~
13
+ exF(u) = p.
If ex = 0, we will have
Vv E dom F,
la -
lCt -
= F(u).
::::; F(v)
23
CONVEX FUNCTIONS
Finally,
v» E V,
U, - U*/rx
>+ F(u)
::;; F(v)
which proves that -u*/a E of(u), which is therefore non-empty.
•
Remark 5.1. Corollary 6.1 below shows that a proper l.s.c. convex function F defined on a complete normed space is subdifferentiable "almost everywhere" (more precisely, over a dense subset) inside dom F.
5.2. Relation with Gateaux-differentiability We shall now complete our demonstration that, at least in the context of convex functions, subdifferentiability constitutes a generalization of differentiability. Definition 5.2. Let F be a function of V into R. We call the limit as A-.,.O+, if it exists, of F(u
(5.10)
+ AV)
- F(u)
A
the directional derivative of Fat u in the direction v and denote it by F'(u;v). If there exists u* E V* such that:
F'(u; v) =
Vv E V,
we say that F is Gateaux-differentiable at u, call u* the Gateaux-differential at u of F, and denote it by F'(u). The uniqueness of the Gateaux-differential follows directly; it is characterized by: (5.11)
VVE
V,
- F(u) = 1l·Ifl F(u + AV) 1
A..,. 0+
IL
The case of convex functions is particularly interesting since the expression (5.10) is in that instance an increasing function of A. Thus, when A-.,. 0+, this expression always possesses a limit, which, however, can be ±oo. We will show that essentially the case of Gateaux-differentiability is the same as that of the uniqueness of the subgradient. Proposition 5.3. Let F be a convex function of V into R. If F is Gateauxdifferentiable at u E V, it is subdifferentiable at u and of(u) = {F'(u)}. Conversely, if at the point u E V, F is continuous and finite and has only one subgradient, then Fis Gateaux-differentiable at u and of(u) = {F'(u)}.
24
FUNDAMENTALS OF CONVEX ANALYSIS
Proof If F is Gateaux-differentiable at u, it is obvious that F'(u) indeed v E Vand w = v - u, we have F(u + w) - F(u) ~ F'(u; w) = F(v) - F(u) ~
F(u
+ AW) - F(u)
~
WE
E
of(u); if
<
A w, u* )
on dividing through by A and passing to the limit we get:
F(u)
+ AF'(u; v)
~
F(u
+ AV).
Geometrically, this means that in V x R, the straight line: .2
= {(u + AV, F(u) + AF'(u; v» I A E R},
epiF
does not pass through the interior of epi F. But is an open convex set since epi F is convex, and it is non-empty since F is continuous and finite. From the Hahn-Banach theorem, there is a closed affine hyperplane yt' containing .2 which does not intersect It is easy to see that yt' is the graph of a continuous affine function everywhere less than F and exact at u. Since the subgradient u* of F at u has been supposed unique, the slope of yt' is u* and since yt' contains .2:
epiF.
F'(u; v)
=
which proves that F is Gateaux-differentiable at u with differential u*.
•
The convexity of a Gateaux-differentiable function may be characterized in the following way:
Proposition 5.4. Let F be a Gateaux-differentiable function of d c V, d convex, into R. Then the following are equivalent to each other:
(5.12) (5.13)
F is convex over d F(v)
~
F(u)
+
v u, vEd.
2S
CONVEX FUNCTIONS
Similarly, thefollowing are equivalent to each other: (5.14) F is strictly convex over d F(v) > F(u)
(5.15)
+
vu, v Ed, u "# v.
Proof From the preceding proposition, (5,12) implies (5.13). Conversely, the inequality (5.13) written with U and (l-A)u+Av(u,vEd,AE]O,l[) necessitates (5.18)
F(u)
~
F(u + A(V - u)) + A
Similarly,
(5.19) F(v)
~
Fiu + A(V - u)) + (1 - A)
+ A(V - u)), v - U ).
By multiplying (5.18) by (1- A), (5.19) by A and adding the inequalities, we obtain:
F((l - A)U
+ AV) :::;
(1 - A)F(u)
+ AF(v).
To show that (5.14) implies (5.15) we note that, as in Proposition 5.3,
F(u
+ A(V - u)) <
(1 - A)F(u)
+ AF(v),
if u,v E d and A E ]0,1 [. Whence, F being convex:
u ) :::; F(u
+ A(V ~
un - F(u) < F(v) -
F(u).
To show that (5.15) implies (5.14) we proceed as for the convexity, noting that (5.18) and (5.19) are strict inequalities if u "# v. • The convexity of a function is expressed by the monotonicity of its Gateauxdifferential: Proposition 5.5. Let F be a Gateaux-differentiable function of d c V, d convex, into R. It will be convex if and only if its differential F' is a monotone mapping of V into V*, i.e. if:
(5.20)
Proof From Proposition 5.3, if F is convex, F'(Ul) and F'(U2) are the subgradients of F at U1 and at U2:
u 1 ' F'(u 1 ) u z, F'(u z)
)
+ F(u 1 ) :::; + F(u z) :::;
Adding these terms together, we obtain (5.20).
F(u z) F(u 1 ) ·
26
FUNDAMENTALS OF CONVEX ANALYSIS
Conversely, if F is Gateaux-differentiable and if F' is monotone, for all u and v in V, the function ¢ of [0, I] into R defined by ¢>(A) = F(u
+
A(V - u))
is differentiable with derivative: ¢>'(A) =
u, F'(u
+ A(V
- u)) ).
But, because of (5.20), ¢' is increasing: thus ¢ is convex over [0, I] and in particular ¢>(A) ~ (1 - A)¢>(O)
+ A¢>(I),
IiAE[O,I],
which is the desired inequality: F((I - A)U
+ AV)
~
(1 - A)F(u)
+ AF(v),
5.3. Subdifferential calculus It only remains for us to examine to what degree the ordinary differential calculus can be extended into a subdifferential calculus. Some results follow directly: (5.21)
(5.22)
let F: V -+ R and A> 0. At every point U E V, we have J(AF)(u)
=
A JF(u),
let F1 and F2 : V -+ R. At every point u E V, we have J(F 1
+
F 2)(U) ~ JF 1 (u)
+
JF 2(U).
The equality in (5.22) is far from being always realized. Here, however, is a simple case where it holds: Proposition 5.6. If F1 and F2 E F(V), and dom F2 where F 1 is continuous, we have: (5.23)
liu
E
if there is a point UE dom F1 n
V,
Proof We have to show that the inverse inclusion of (5.22) is true, that is, that each u* E o(F1 + F2 )(u) can be decomposed into ut + uf, with uf E oF1 (u) and u~ E oF2 (u). Our hypothesis is that F1 and F2 have finite values at u and that (5.24)
v» E V,
27
CONVEX FUNCTIONS
Consider the convex sets in V x R: C 1 = { (v, a) I F 1 (v) -
The inequality (5.24) implies that they only have boundary points common. But C 1 is the epigraph of the function G: G( v)
= F 1 ( v) -
F 1 ( u)
+
In
which is convex and continuous at u. Thus C1 is a convex set with a non-empty interior. We can thus separate (;1 and Cz by a closed affine hyperplane :Yf (Cor. 1.1). It is easily verified that :Yf is "non-vertical" (see Prop. 5.2) and is thus the graph of a continuous affine function.
v
-+
where
v* E V* and ex
E
R.
The separation can be written:
v» E V. In setting v = u, we obtain ex = -
V,
VVE
V,
Hence -v* E oFz(u) and u* + v* E oF1(u). Whence we have the desired decomposition, u* = uT + ut with uT = u* + v* and u! = -v*. • Finally, after considering the subdifferential of a sum of functions, let us examine the subdifferential of a composite function. We take two I.c.s. Vand Y with topological duals V* and y* and a continuous linear mapping A: V --+ Y with transpose A *: y* --+ V*. Let FE F( Y); the function F 0 A: V --+ R belong to F(V). Proposition 5.7. Let there be a point Au where F is continuous and finite. Then for all points u of V, we have
a(p
(5.25)
Proof Letp*
E
0
A)(u) = A * aF(Au).
of(Au). By definition:
vp E Y,
< p - Au, p* )
+ F(Au)
:::; pep)
28
FUNDAMENTALS OF CONVEX ANALYSIS
and a fortiori: \:Iv
V,
E
>+ F 0
0
A(v)
0
A(v).
a(Fo A)(u) which proves that:
E
(5.26)
A* of(Au) c o(F
Conversely, take u* \:Iv
(5.27)
A(u) ~ F
\:IvEV,
Thus A*p*
- Au, p*
E
E
0
A)(u).
a(F 0 A)(u).
V,
-
u, u*
>+ F 0
A(u) ~ F
Let us consider the affine subspace in Y x R.
2
= {(Av,
u, u*
>+ F 0
A(u)) I v E V }.
The inequality (5.27) shows that 2 and epi F only have boundary points in common. Since F has been assumed to be convex and continuous at Au, o
ePiF =I 0, and
there o is a closed affine hyperplane :!It' containing 2 which
does not intersect ep1'F. As usual, :!It' is non-vertical, and therefore is the graph of a continuous affine function of Y into R:
p ~
0(,
where
< Av, p*
>+
Since :!It' contains 2 : \:Iv
V,
E
(5.28)
0(
v» E V,
(5.29)
and
p* E Y* 0(
=
u, u*
o(ER.
>+ F
0
A(u)
= F 0 A(u) -
o
Thus u* = A *p*. Finally, since :!It' does not intersect epfF: \:Ip (5.30)
Thusp* (5.31)
E
Y,
\:Ip E
E
Y,
+ F 0 A(u) ~ F(p).
aF(Au). Whence the desired result: u* o(F
0
= A*p* E A*aF(Au), and
A)(u) c A * of(Au). •
6. ll-SUBDIFFERENTIABILITY Henceforth we shall assume that V is a Banach space.
6.1. An ordering relation over V x R Let us take a number m > 0, and consider in V x R the closed convex cone with non-empty interior, defined by:
~(m),
29
CONVEX FUNCTIONS
(6.1)
= {(u, a) E V
~(m)
x R Ia + m Ilull
~ O}.
We can associate with it an ordering relation over V x R for which it will be the cone of positive elements. We shall denote if by ~. By definition: (6.2)
(u, a)
~
(v, b) <;> (v - u, b - a) E ~(m).
Proposition 6.1. Let S be a closed subset of V x R such that inf{ a I (u, a) E S} > - 00.
(6.3)
Then S has a maximal element for the given ordering. Proof It is sufficient to show that every totally ordered family of S has an upper bound in S; the conclusion will then follow from Zorn's lemma. Hence let (at> Ui), eI be a totally ordered family of S, and ff the filter over S formed by its starting sections:
(6.4)
A E$i ¢'>3iEI:
A=>{ (a j, uj)IJ ~ i}.
The family (a')'eI is decreasing by (6.1) and, from (6.3), it is bounded. It is thus convergent in R, and from the inequality which defines the given ordering:
(6.5)
m
Ilu
i -
ujll ~ la, -
Vi,jEI
ajl
we deduce that $i is a Cauchy filter over V x R. It therefore converges to a limit (ii,l1) which belongs to S since the latter is closed. It only remains to show that (ii,l1) is the required maximal element. For this we take any i E I and write the inequality
(6.6)
v]
i
~
and, passing to the limit inj:
(6.7)
(a -
aJ
+ m
I it
- v.
I
~ O.
Thus (ii, 11) ~ (a" U,) for all i, which establishes the result.
•
6.2. Application to non-convex functions Let F be a mapping of V into R with -00 < inf F < +00. To say that F(u) = inf F amounts to saying that 0 is the subgradient of F at u. We can enquire what property of differentiability is related to the fact that F(u) ~ inf F+ B. Theorem 6.1. Let F be a l.s.c. function of V into and let there be a point u where
(6.8)
F(u)
~
inf F
+ B.
R,
with -00 < inf F < +00,
30
FUNDAMENTALS OF CONVEX ANALYSIS
For all A> 0, there is a UJ" E V such that (6.9)
(6.10)
Ilu - uJ,,11 ~ A and F(u,l.) ~ F(u) epi F n {(U,l.' F(u;.))
+ C6'(eIA)}
=
(u... F(u,l.))'
Proof Let us apply Proposition 6.1 to the closed set S = epi F, for the ordering relation associated with the cone C6'(ejA). There is a maximal element (a...u,l.) which is greater than (F(u),u). Since (a,l.,u;) is maximal, aJ" = F(uJ,,) and (6.10) is satisfied. To verify (6.9), we simply write (F(u),u) :::;; (F(uJ,,), uJ,,):
l lu- u,l. I ~
F(u) - F(u,l.)'
The second term is bounded bye due to (6.8). Whence we obtain (6.9).
•
To understand this proposition more clearly, we may note that if F is furthermore Gateaux-differentiable, condition (6.10) implies that, for all v E V: s (6.11) Vt E [0,1], F(u,l.) -;: t [e] ~ F(u,l. + tv)
(6.12)
-]llvll
~
whence it immediately follows that:
(6.13) If we further specify that A be corollary:
0, we obtain the
following more striking
Corollary 6.1. Let F be a Gateaux-differentiable l.s.c. function of V into R and u a point where:
(6.14) F(u) ~ inf F Then there is a point v such that:
(6.15) (6.16)
(6.17)
F(v)
~
+ e.
F(u)
Ilu - vii ~ J; IIF'(v)ll* ~ J;.
6.3. Application to convex fnnctions
Let V still be a Banach space, V* its dual, FEr o(V) and F* its polar. We know (definition of F*) that inf {F(u) + F*(u*) -
31
CONVEX FUNCTIONS
Definition 6.1. We call the set ofu*
o ::;;
(6.18)
+
F(u)
E
V* such that:
F*(u*) -
a
the e-subdifferential ofF at the point u E V, and denote it by o,F(u).
The relation (6.18) being symmetrical in u and u* u*
E
o.F(u) <::>
oEF*(u*).
U E
This also implies that the function v -+ (v - u,u*) + F(u) - 8 is bounded above by F. We immediately deduce from this that, for all a> 0, 0EF(u) is non-empty if and only if F(u) is finite. The sets 0EF(u) decrease with a and their intersection for a > 0 is the subdifferential of(u). The principal result concerning s-subdifferentials is the following:
Theorem 6.2. Let V be a Banach space and V* its topological dual. Let FE ro(V), F* its polar, 1I E V, u* E V* with u* E 0EF(u). For all A> 0, there exists a; E V and u* E v* such that
(6.19)
Ilu - u;.11 ::;; A,
u! E of(u;.).
(6.20) In particular (A = that
(6.21)
Ilu* - u!11 :::; a/A,
0),
if u*
E
a,F(u), there exists
Ilu - uE11 :::; j;"
Ilu* -
uill :::;
UE E
V and u~
E
V* such
j;"
ui E of(u.). Proof Consider the function over V
(6.22)
G(v)
=
F(v) -
F*(u*).
By hypothesis (6.18), we have G(u) ::; inf G + a. We can thus apply Theorem 6.1 to G. There exists U;. E V such that
Ilu - u;.11 :::;
(6.23) (6.24)
epi G (\ {(u;., G(u;.)) + ~(a/A)}
A
= (u;., G(u;.)).
For greater simplicity, we shall denote by C(};. the cone (u;.,G(u;.)) + ~(eIA) in V x R. It is a closed convex set with non-empty interior and epi G is a closed convex set. From (6.24) and the Hahn-Banach theorem, we can separate ii;. and epi G by a closed affine hyperplane JIf of V x R with equation:
(6.25)
+
b = 0,
h*E V*,
a
and
be R
32
FUNDAMENTALS OF CONVEX ANALYSIS
Since C(J A is closed and convex, it is the closure of its (non-empty) interior and so;/t' also separates C(JA and epi G. We cannot allow a = 0 in (6.25) otherwise
(6.26) (6.27)
V(v, r) E epi G V(v,r)EC(J".
Because of (6.24), the inequalities (6.26) and (6.27) are equalities at the point (uM G(uJ), which gives us: (6.28)
b
-
= -
G(uJ.
The relation (6.27) can then be written: (6.29)
U A'
h * > + r - G(u,,) ~ 0
V(v, r) E C(JA'
Returning to the definition of C(J A:
(6.30)
+s
~
V(w, s) E C(J(G/A).
0
Thus, taking s = -Gj A, and returning to the definition (6.1) of C(J( Gj A), we get: sup
(6.31 )
II w II ., 1
~
lOlA
(6.32)
It only remains to put uj = u* - h* E V*. From (6.32) we obtain: (6.33)
and by substituting (6.22) and (6.28) into (6.26): (6.34) (6.35)
Vv E V, Vv
E
V,
+ G(v) -
F(v) - F(u,,) -
But the latter demonstrates that
ut E of(u
A) ,
0
and so concludes the proof.
•
Corollary 6.2. Let V be a Banach space, and FEr o(V). The set of points where F is subdifferentiable is dense in dom F. Proof Let
Uo E
dom F and
F(uo) = ~~?
10
> 0 be fixed: since F = F**,
[< uo, u* >- F*(u*)J,
CONVEX FUNCTIONS
and there exists u~
E
33
V* such that:
F*(U6)
~
F(uo) - e,
that is u~ E o,F(uo). Applying Theorem 6.2 with A = 0, we deduce that F is subdifferentiable at a point u, E V such that Iluo - u,11 ~ 0.