- Email: [email protected]
The method of projections for finding the common point of convex sets
THE METHOD OF PROJECTIONS FOR FINDING TNE POINT OF CONVEX SETS* COMMON L. G.
GUBIN .
B. T.
POLYAK and E.V.
RAIK
MOscow (Aeceived
29
November
1966)
MANYmathematical and applied problems can be reduced to finding some common point of a system (finite or infinite) of convex sets. Usually each of the sets is such that it is not difficult to find the projection of any point on to this set. In this paper we shall consider various methods of finding points from the intersection of sets, using projection on to a separate set as an elementary operation. The strong convergence of the sequences obtained in this way is proved. Applications are given to various problems, including the problem of best approximation and problems of optimal control. Particular attention is paid in the latter case to problems with restrictions on the phase coordinates.
1. Methods of successive projection A prqject point .+(x)
ion
E
of the point x on to the set ;; of a normed space I’? is a .:, such that (see Fig. 1) 112--R(z)ll=
Theproperties of projections
infib-yll= p(z,R). tlER
of projections will be given in Section 2, and examples for actual sets will be given in Section 4.
Hereafter we shall assume that f? is a Hilbert space, although some of the results given below are also valid for a wider class of spaces. Suppose we are given
l
Zh.
vychisl.
Mat.
a system of sets I:)~ E E, a E
mat.
Fiz.
7,
6,
A, where :1 is a
1211 - 1226, 196’7. 1
2
L.G.
Gubin
et
finite or infinite set of subscripts. point x of this system of sets, XEQ=
al.
The problem n Qa.
is to find
some common successive
The methods of
GA
FIG.
1.
projection consists in constructing a sequence x0, x1, . . . , xn, . . . , where the point x o is arbitrary, and in order to obtain the point znt 1 from x* a set Qacn, is chosen and a step is made in the direction of projecting the point xn on to this set, i. e. Pfl Hereafter
=
39 + h, (PC+)(P)
Pacn)(z)
=
Po,(,)
-
zn) (
In particular,
(x). zn+l =
o< if
h, <
2.
(1)
A, 3 1, we have
Pa(n) (rn) 7
U’)
i.e. for xntl we take the projection xn on to gc(,,). The order in which the sets are selected (i.e. the method of choosing a(n)), can vary. If A is a finite set, 3 = {Ccl, . . ., a,), a cyclic order of projection can be used CL(n)
=
h(mod
(2)
m)+iv
where n(mod m) represents the remainder obtained from dividing n by m. In the general case the method of projecting on to the most remote set can be used, i e. p(xn,
Qacn,)= sup
p(xn, Qa)=
@(zn).
(3’)
CUSA
Since, however, such a set may not exist or it may be difficult to find, it is sufficient to solve the problem of finding the remotest set approximately, namely to choose an arbitrary a(n) satisfying the following condition:
The
fromP (xn,QCC ( R,) --t 0 it
method
of
follows
projections
thata
3
(rn) =
The convergence of these methods follows which will be proved in Section 2.
sup p (P, Qa)-+ 0. aEA
from Theorem 1 given
(3)
below,
Theorem 1 Let all
sets
Cj, be closed
and convex,
Q =
fl Qa
be non-empty and
CLEA 0
<
&i <
Let any of the following (a)
Q, n
( aO_
h,
<
2 -
conditions
Q,)" is
E2 >
E2,
0.
(4)
be satisfied:
non-empty
(here ?o
is a fixed
set
from the
a&
system f&t if0 denotes
the interior
of ‘I) ;
(b) all Oar, with the possible
exception of one (f&J, vex* with the common function S(T) ;
(c) B is
finite
(d) A = {a,, (2, = {X1
con-
dimensional;
. . . . a,,,) is finite, (Ci,
X><
Then in methods (l), strongly
are uniformly
convergent
and all
&
are half-spaces,
i.e.
/3i).
(2)
and (11,
to sane point
(3) for x0 E Q =
any x0 the semence xn is fl Qz.. In the cases (a) WEA
and (d)
in method (11,
(3’)
the rate
of convergence
is geometrical.
The methods (l), (2) and (11, (3) are suggested in [2, 31 for the finite dimensional case. Their convergence is proved in the same place. Some generalizations of the results of [2, 31 for the finite dimensional case are given in [41. In [51 the weak convergence of the methods in the general case is proved, and no conditions of the type (a) - (d) are required. The assertion of Theorem 1 concerning the strong convergence of the method and an estimate of the rate of convergence for the case (a), are obviously new.
l
A set R is called uniformly convex if there is a function E(T), S(T) > 0 when -r > 0, such that from z ER, y E 9 it follows that z E R for all z, for which lb - (X + Y) / 211< 6(11X- yll) [ll .
4
L.G.
Gubin
et
al.
Let us now consider the case when the sets ‘So do not necessarily have a common point. Here we.shall confine ourselves to finite A and to the method Cl’), (2). i.e. to the case of cyclic projection. Theorem
2
Let all them (for
Oai,
i = 1, . ..,
explicitness,
points
2; E
m - li
;j_,G,)
$oi,
m be closed
o,,)
and convex and at least
be finite.
Then it
is possible
i = 1, . . . , m such that I)ci,,Cx while
= x1,
in the method (l’),
(2)
) = Xi+l,
one of
to find i = 1, . . . ,
we have xkmtitl
-
.km+l
ana xkmti weakly converges to Xi as k -+ co. If in -* %i+l - Xi, the convergence addition any of the conditions (b) - (d) is satisfied, will be strong. If all sets C&.., apart from possibly one, are also convex*,
and
;Q,
the rate
of a geometrical
i=i
is empty, the sequence xkmti converges
1
to Xi at
progression.
Note 1. For the case of two sets such that
llZ_i - ZAl =
m = 2 we obtain
the points
x1, xz
inf IIII:- YII= P (Qan Qd.
=Qa, VEQ, .Note 2. For the strong convergence require that at least one of the sets be uniformly marks also
of the method it
is sufficient
to
(Joj) should Qai (for explicitness, convex and that Qa. n Qa, should be empty (similar re-
apply to the geometric
The method (1’).
J-1 rate of convergence).
(2) with m = 2, go,
n Q,,
= 9 has been suggested
[sI only when one of the in [sI . However its convergence is proved in sets is compact or finite dimensional. In addition, for m = 2 the result of Theorem 2 has been obtained by different methods in IIT]. The case of an arbitrary m (but assuming the compactness of one of the sets) has been considered in Ltd. The proof of Theorem 2 is given below in Section 2. Let us now consider *
yet
another version
A set H is called strongly convex, the function S(T) = y-r2, y > 0.
if
of the method, using a it
is uniformly
convex with
The
different
way of choosing zn+* =
P,(5n D (zn)
method
of
the length
of the step,
+ yn (0 (In) )--i (Pa(*) (2”) =
IIPcLc,,, (xn)
-
5
projections
Al
=
-
namely sn)),
a,;q+a
yn 2
P (xn,
(1”)
0,
Qa) .
(3”)
In other words, during each iteration a step of length yn is made in the direction of projecting on to the remotest set, and the point obtained is projected on to some selected set Q,. Theorem
3
If all and vn+O, finite
Qa are closed
sets
condition
g yn = 00, the method (1”).
number of steps,
Indeed
and convex,
Q =
zn E Q =
i.e.
Qa fl (x : g(x)
<
g(s) = The supporting
functional
nn has the form
En =
0}, sup
n Qa c&s4
gives for
is satisfied,
a solution
within
a
some 72.
where
ac54, a&
to the set
(3”)
(a)
P (2, Qa),.. {r : g(x)
(@(z~~))--~(P~(~)(s~)
-C).
<
g(z”)}
After
at the point this
it
is clear
that Theorem 3 follows from the general results of the paper [91, since the method (1’3, (3”) is a special case of the method considered in [91, and all conditions of Theorem 1 in [91 are satisfied.
2.
Proof of Theorems 1 and 2
1. The properties of projections. We shall nw describe some general properties of the operation of projection, which will be needed subsequently. It will always be understood that R is a convex closed set, and the projection of x on to R is denoted by P(x).
First of all P(x) exists [lo], p. 286). and therefore
and is unique for every x (see method (1) is meaningful.
for
example
Lemma 1 The vector
z - P(X) is supporting
to .9 at the point
P(x),
i.e.
L.G.
Gubin
et
so,
@-J%),Y--P4) If R is strongly
Proof.
we have
---P(x)) (0.
(x-P(X),Y
convex with the function
For 0 < h < 1 we have
convex nature
(5)
y E R.
x@R,~#P(x),~ER,
convex,
If R is uniformly
al.
hy +
(5')
E(T),
(1 -
we have
E R
h)i(z)
from the definition
of R. Therefore,
in view of the
of projection
lb - P(x) II2 < II5 - hy - (1 - h)P(x) 112= = lb - P(x) II2+ AW(z) - YIP + h(z - P(x), P(x) - y). Hence (x-P(x), (5).
P(x))
y-
< hljP(x) - ~112, and taking h + 0, we obtain
In the case of a strongly
that
1/2(P(x)+y)+z~R
llx -
p (4 II,
(5),
we obtain
then
convex set there when
exists
We takea=s(s-p(z))/
tk.llS:.
l/z(P(x) + y) + z E R.
w =
Replacing
0 2 (x -P(x), w -4(4)=‘/2(~--(4,Y + i.e.
(3 -
P(x),
Y -
,y by zu in
-P(x))+
,lp(x;_x,,
P(x))
such an E > 0,
(~-p(49x--(xh
-2ellx - P(x)11 c 0.
<
Finally, for a uniformly convex set, equality, since we can take e = ,S(lly -
(5”)
follows
from the last
in-
P(&)ll).
Lemma2 The projection
operator
is non-stretching,
i.e.
IIP(x) - P(y)II < 112- Yll. If R is strongly
convex,
x # y and x e
IIP(x) -P(y) If R is strongly
convex,
we have
(6)
R (or y 6?? RI, we have
II c 115- Yll.
03’)
The nethod
of projections
P(s) - P(y)!1 < dlz - yll, de Q =
7
1 ~w~~+llv--~iv)ll)
1+2Y(lls-
(6”) Roof.
Applying
(5)
twice,
(5 - W), JYY)-Jw) Adding these
inequalities
we obtain
G 0,
b--P(Y),+)
-WY))
we have
0 > (x - $4,P(x) - P(y) ) + I!P (x) - WY) II2 2 2 -lb - yll IP(z) -P(Y) II + IIW) Hence
IIP(z) -P(y)
II <
< 0.
11x-- yll.
Similarly,
using,
formula (5’) for a strictly convex set, and (5”) set (with 5(-r) = y-r?), we obtain (6’) and (6”).
for
---p(Y) IF.
instead
of
a strongly
(5),
the
convex
2. scheme of the proof for Theorem 1. It is easy to see that the statement of Theorem 1 concerning the convergence of xn follows from the following chain of lemmas.
In method (1) ZEQ=
n&
for
any choice
of a(n)
for
every point
on to
andforalln
%A
IIZn+i- 2116 IIP - zll.
(7)
Lemma 4
In the method Cl),
(2) and (l),
lim 0 (zn) =
R-kCS
0,
(3) with the condition
(4) on h,
CD(x)= SUPP(Gu).
(8)
%SA
Lemma5 If any of the conditions (a) - (d) are satisfied, (8). the following condition sequence xn, satisfying lim p (xn, Q) = 0. Tl-rco
for any bounded is satisfied (9)
8
L.G.
Gubin
et
al.
Lemma 6
If for a convex closed set Q end sequence a? the conditions (9) axe satisfied, xn converges to some point x* E Q.
(7) and
The proofs of Lemmas3 to 6 will be given below. 3. Proof of Lemma .3. Using formula (5) for the sets condition O< h, ,12, we obtain for x EZ Q
4. hoof
of I!&URU4,
p(xn, 0) = 11~ - P,(P) THUS,
p(S’,
Using Lemma3 with n = k$(xn),
II >
Q) monotonically
Ilc+i - P&P) = p (P+‘, Q) . decreases,
Qatn f and the
we obtain
ll 2 ll~+i - pc, (xn+i) ll =
and therefore
there exists
PRO*
P =Iimp(x*,Q), n-+“W ~~he~ore,
Q) - p2(xn+l, Q) > 11x*- PQ(x”) 112 - IIxn+i - PQ(xn) 112 = = llxfl - PQ(xn) II2- IW + hn (Pct(72,(2”) - Zn) - PQ (@) II2 =
p2(x”,
-- llxn - P&y12 - b" -
-
Z?w(z=
+
%I(~cZ,?~)(~
-~Q(2n),&(?&)(z") -
z",PQ(Zn)
Using Lemma1 and condition p2W
pQ(z")!i2 -
xn)
=
k2ik(n)(xn) %&
-
-
d12
-
~~)llP~~~~(~n)
-
d2
+
-PC&~*)).
(4). we find
Q) - @(x~+*, Q) a ~32tl~~n,(~~)
But, as has been shown, p (~9, Q) --f p, that
therefore
-
d12.
(W
from (10) it follows
P(@, Qacn,)= IF - Pa(n)(sn) 11 + o. For the method (I), For the method (l),
(3) in accordance with (3) it follows that @(a?) 4 0. (2) and a certain E > 0 we choose N such that
The method
p (xn, Qw+,) <
e / 2m for
all
II > I’. Then for n >N
= hnp(zn, Qacn)) II e / m.
IIZn+* - znll = 3LnIIPcqn)(zn) - Al For every l< for for n>N
one can find
i
9
of projections
k < m
that a(n + k) = ai.
such
There-
p (P, Qai) < 11~ - ~~+~ll + [IP+~ - Pa(n+k)(P+k) II < 112”- zn+‘II + . * * . . . + ll~+k--i - ~n+k ll + ll~n+k - Pacn+k)(znfk) II < ke I m + e / 2m -C e. Therefore # (xn) = max p (xn, Qai) < e, i
which proves
Lemma 4 for
the method (I),
5. Proof of Lemma.5. Let condition
=Q,fl U-IQ~)~ end a&
llz -
xl} <
such that
(E + 6) E
0,
for
all
of all
that
a # z.
z E Qol, P,(y)
We take .y = P--o(P)
We choose
all
a # a and all
is 811 arbitrary
w =
ex/
point
(e + 6) +
6,~/
Indeed,
But lb - 41 = (6 / e) Ily - P,(Y) II < 6,
z E Qa. Thus UI is an internal
end points
if ,y
e for all a # a, then
where z = x + @/E) (.y - pa(y)). therefore
(a) be satisfied.
6 > 0 such that t E Q, for
6. We remark first p(y, QOL) <
(2).
E Qa,
point
therefore
and put E = 20(x”).
P (Y, Qa) G lb -&W)
il G IIu - Al
= P (xn, Q$+ Therefore if we form w as above, since x E &, y E Qa, therefore Therefore
+++11xn-
of sn interval
having
w E Q,. Then for a # z
+ 11~~- Pa(xn) II =
P (xn, Q,) <%
+ G = e.
we have w E Qa for all a # or. But also w E Qa. Consequently, w E 3).
XII< e(-$+f)
=$ce.
the
L.C. Cubin et al.
10
Here we have used inequality condition
that
(6)
the sequence
(whence
rn(IIrn -
IIy -
XII <
R)
xl1 <
llz’l -
211 ) and the
is bounded.
Thus,
(11) which in view of
(8) proves
Lemma 5.
Let condition
(b) be satisfied.
that there
is a subsequence
i.e.
We assume that Lemma 5 is invalid, xnk such that
kll_P(Z”k, We choose Q, (r”)
E=
min {p/4,6
E, p (rN, Q) >,
<
Let us consider
that
6.(ljPa($N)
PQ(zN) II 2
i.e.
y E
Q, for
YII =
lb--“‘ -
dicts
all
‘12(Po(rN)
hand,
with its
centre
ipa
at
E/2
+
*/2(PQ(~N)
+
1/2p(~N, Q) f
&(xN)).
It is obvious
E,
Pa(zN)
i-e.
to Q,.
(p/2)
- E a
‘/21lp,-
and
But’ Il’Pa(~N) --
6(P/4) >, 5
II s
con-
p/2
-
ami Ily(xN)
--~W
+
i. e. y E Qa. Since y E Q- and
N E
Pu (z”,)
2
+ PQ(x”))
*/2(Pa(sN)
- ZNII 3
c/2 =
I/zE,
f
entirely
P&T)ll>
-
a f a, therefore
the assumption
= N such that
nk
for a # iii, in view of the uniform
II = ‘/2II&(sN)
zNII <
and find
0,
PQ(s~) II) belongs
G([IP,(xN)
+PQp)
1/211Pol(~N)-
f
y =
hN - PQ (x”) 11-
P/4 = P/4, 1/2(Pa(zN)
-
e >
o > 0.
P/Z.
the point
of Qo, a sphere
radius
p,(s)jl
(p/4)},
N E QG. ~1 the other
vexity
Q) =
0. Hence
II d pbNI
p(rN,
Q)
\(
llxN -
lt211~N - p&N) Ii + Y211~NQ,s E
that pCxN, Q) ap/2.
When condition (c) is satisfied Lemma 5 is easily proved by contradiction. Let there be a subsequence xnk, for which p(x*k, Q) > p > 0. In view of the boundedness of the sequence {x”k), another subsequence can be separated out from it (which for simplicity will also be denoted converging to some point x*. Since p(xnk, Q,) -+ 0 when k + co by ~“~1, for every a, and Q, is closed, therefore x* E Qo for all a. Therefore z* E Q. which contradicts the assumption that p(xnk, Q)a p > 0. Finally, if condition (d) is satisfied, we denote by ,!, a finite dimensional subspace going through x0 and spanned by the vectors c r, . . . ,c,,,, and we denote by in the projection of xn on L. It is easy to prove that
The method
P WYQi)
P (zn,Qi),
=
P
w, Q) =
to show Lemma 5 over the finite the following estimate is true
from which the statement
p (Sn, Q),
for
all
s m+1 =
Let xf E IF
-
and in view of
7.
Estimating
satisfied definition
; S,,. n=o
X*1/ < (9),
closed
Q) for
But a sequence
%ll.
section.
are convex, p(z”,
all
pQ(xcn)
rate
p) a sphere with its
11 +
i.e.
(since
in view of
xm E S,)
x* ES(P~(Z”), ilpQ(xn)
(12))
-
and the estimate
P (r”, Q) Q cm (Xn) =
cp (Xn.pQqn)) < c
p(xn, Q)),
x*11
<
2p
<
(lo),
p2(xna Q)- p2(xnfi,
(I -e$$)‘h P(XnY Q).
But from (13) 2p(xn,
Q),
we obtain
Therefore
<
we have
(13)
(a) or (d) is Using the
whence &
(xn,
&(xn).
8182
~‘11
(71,
and in addition
when condition
p(sn,Q)
of method (3’)
llxn -
at
0.
of convergence.
(11) or
P(p+i,Q)
centre
P(cQ)).
n
x.11=
n-
the
follows.
of such sets must have a non-empty inter-
lim (lxn -
we have (see
case
(12)
and non-empty
Then since
km -
is sufficient
cases.
&n =n~OS(P~(~n),
Ii <
it
space L. But in this
i
6. Proof of Lemma 6. We denote by S(x, the point x and of radius p and consider
11~m-Po(zn)
therefore
dimensional [21
of the lemma immediately
Lemma 5 is proved
Then the sets S,
11
Q)< cmaxp(m,Qi),
~(5~~
Thus
of projections
Q) < 4
I-
E$)n’2
-
Q)
L.G.
12
Gubin
et
al.
We remark that the $81118 proof can be extended also to the more general method (l), (3) with some refinement of the condition (3). At the expense of some complication it is also possible to prove the geometric rate of convergence for the method (l), (2). of Theorem 2. For explicitness,
8. Proof
let Qol be bounded. Consider
the operator
pa,(Pad. *-(PmPa,(x) ) 1.- .) ),
P(x) =
W
which successively projects on to all m sets. Using inequality (6) m times, we obtain that the operator p(x) is non-stretching, i.e. IIP(s) - P(y) II
YII.
In addition it takes a closed convex bounded set Qa into itself.
Therefore [ll,
putting 5%= Pa,(E),
of a Hilbert space
121 it is a stationary ioint x1 E QaJ.
. . . , F,, =
f’,,
from the condition PG,)
(Zm-~),
=
x1 we obtain Pa,(&) =%I, i.e. the points xl, ._., FIRsatisfy the conditions of the theorem. Furthermore, for the method cl’), (2) we have ~lxitkm
-_ fill2
-
p+km+i
_
112=
his,,
-
xi+km+lII
_ x:itkm+i,
-
_
IJZ~+~ _
+
2(x’+km+’
X
IIZi+l - iT;;_ilI - (Xi+km+i - Xifkm, Ei+i - Zi) ] >
>
zjl(2) _
2 (xi+km
(llxi+k”
+
zifi,
fi+i
_ zi)
(llxi+km - xi+km+iIj -
+
xi+kmfi
2[ IIxi+km
z&+~)
+
_ xi+km+iII
x
lIZi+* .- ,ill)“.
Hence ll~i+~m- %ll > 112 i+kn+i - Zi+iII. Consequently, the quantity llri+knr_ ziII monotonically decreases and as k -, OI it approaches a limit which is independent of i. Therefore in the inequality obtained above the left-hand part approaches zero as k -. co, the thereforellzi+km Zi+km+iII+ IIz~+~- iill, and also llri+km- ri+km+lIIIl~i+i- ~?=uill -
(2’r+km+i _
xi+km
,
Zi+l
Xi+kmlla ~~;~k:zp._
Xi+kmll
(xi+km+i-
Xi+km,
-
+ -
Zi)
--t
IIs~+~ IIfi-i
fi+i
-
-Zi)
0. But II (Xi+km+i ~~112
-
Z(x”+km+i
-
Xi+km)
-
_
Xi+km,
zi+i
(fi+i _
zi)
Z;;ui) 11’ = =
+2[ll&+km+i -- xi+kml\(\5i+i - ~~(10, therefore xi+km+i- xi*km--t Ei+l - Zi.
Zi11)2
] 4
Furthermore. the sequence xltknt, k = 0. 1, . . . , is bounded; hence there exists a subsequence which is weakly convergent to some point X,. Since Qal is convex and closed, and therefore it is weakly closed, and xltkm E
Qa,, therefore
also i, E Q,,. Using the convergence $+-km-
xi+km-t& - fi, proved above it is easy to show that the corresponding
The
nethod
of projections
subsequence x Ilfkn weakly converges to
13
A = Pa, (21) = 5~4+ i!g - fi.
weak convergence of the subsemence x i+km to 4i = P,,(&_t) “i
-
;;-I
is proved similarly.
Pa, (%=J = %Fz+ zi -
The
= f+r +
Hence it follows that
zm = 5Ti+ “r: pi+* -
Z$) + 2%-
z,
= ff.
i=i
Thus, the points x”i satigfy . . . , m. -
1, Pa,(&)
=
&,
the conditions
Pa i+i$Zi) = i?i+t, i =z 1,
snd W@ can put Xui= Xi.
IFheweak convergence
-55;;uill Of all sequencesXi+km t0 Xi follows from the fact that 1/$9+km monotonically
decreases with increasing k
(which has been shown earlier).
Now let all sets Qai be uniformly convex. If all points xl, . . . , x,,, coincide, the strong convergence of the method follows from Theorem 1. Otherwise there is a uuifonnlv convex set C&., for which Xj f Gj-1 (this happens in the case remarked on in note 2 ofjthe theorem). Then Xj is a boundary point of the set Qo. (since fj = Paj (Sj-i), Sj-i@ Qaj). But if the sequence of points .xi’km’ 1s ’ weakly convergent to a boundary point of the uniformly convex set, it is also strongly convergent ([131, Theorem 3). Therefore .j+k= * _j_ But in view of the continuity of the projection operator (6)) x itkm + xi for all i. Finally if all sets Qoi, except possibly
one, are strongly convex,
among these sets there is one such that xj # xj-l* close to “j-1, x, Y s Qaj I, sufficiently
Then for
ofrc, Qocj) > p > 0, p(y, Qaj) 2 p > 0; llPaj (x) -Pa,
(y) II <
qlla: -
yll, q <
therefore
1. As a result
i.e.
%,j-l
@
‘~,i
*
we shall have from (6”) we obtain operator (14) is com-
pressing in the neighbourhood of the point Zj_1, But all xj-ltka are near “j-1 for fairly large k. Therefore the principle of compressed mappings is applicable, from which it follows that .jtkm -( Xi at the rate of a geometric progression. But then (in view of (6)) for every i xitkm will convergeto Xi at the sme rate,
3. Rate of convergence As we shall see from the examples given below, each step of the successive projection method can be realized very simply: the amount of calculation associated with one iteration is usually small. Apart from this the method is stable with respect to error. However, a serious
14
L.C. Cubin et at.
drawback of the method may appear to be its slow convergence. As can be seen from the simplest examples the rate of convergence is sometimes actually small. For example, for set over the two-dimensional plane
FIG.
2.
{(xi, x2) : 22 < 0). ft can be al 01, Q2= Qi = { (51, x2) :x2 a ai??, proved that for any initial approximation the method converges to the point (0, 0) at a rate slower than that of sny geometrical progression (namely. at the rate of the order of l/n). This case, however, is fairly typical. Indeed, if the original problem is to minimize the functional
f(x)
over the set Q1, while f =
in1 f(rf
is known, the problem is re-
duoed to finding any point of the set Q = Qt f-j Qg, where Q2 = {x: j(3) < j”} . Here normally Q consists of a unique point, so that condi tio;i fa) of Theorem I is not applicable and the geometric progression rate of convergence cannot be asserted. However, even when the geometric progression rate of convergence follows from Theorem.1, the denominator
FIG. 3. of the progression may be nearly unity= This is Fig. 2 (two hyperplanes intersecting at a small is the minimization problem considered earlier, from 2, taking Q2 = (X : f(x) f f” + E, E > O>.
the case for example in angle). Another example when it is solved apart In this case QI fl Qza is usually non-empty and the rate of convergence is geometrical, However*
The
method
of
15
projections
as follows from the estimates obtained in proving Theorem 1, the denominator of the progression is nearer unity, the smaller 6, i.e. the smaller E. The situation is similar with method (1”)) (3’): if Theorem 3 is applicable, the number of sets is finite, but may be very large. Because of this the problem of accelerating the rate of convergence is important. We shall consider one possible way of achieving this. We shall only deal with the case of two sets. Thus, we want to find x* E Q = Q1 fl Q2, We form, Lc3=
beginning
from x0,
the sequence
ri = P;(9),
22 =
p2(r1),
Pl(X"), Lz+ = xi+ a(s3-xi), 11x*-
A=
(51 -
x2112
x3, xi
-
(151
x2) *
Here Pi is a projection on to ?i, the first three points coincide with those obtained in method (l’), (2) and the value of A is so chosen that of the straight line (x1, x3) with the point x4 lies at the intersection a hyperplane which supports Q2 at the x2 (see Fig. 3). Further points are obtained in a similar manner: x5 =
x’ = Pi(X6),
x6 = PP (x5),
Pi(b),
11x5-
h=------
x8=x5+h(x’-x5),
x6112
(x5-27,x5-56)
etc.
Let us now prove
Theorem
the convergence
of this
VI n Q20 (01Q10 n
(b) Q2 (or (c)
method.
4
Let Q1 and Q2 be convex and closed, Q = Q1 any of the following conditions be satisfied: (a)
,’
Q2)
Q1) is uniformly
E is finite
flQ2 be non-empty and let
is non-empty; convex;
dimensional.
Then in the method described
above xn + X* E 3.
Proof. We shall use the ssme scheme as in proving Theorem 1. At first we shall prove that I[xR+~-xl! < llxn -xz(I~ for all n and for any x E (2. For n = 0. 1,
2, 4, 5, 6. 3,
. . . this
follows
from Lemma 3. It remSinS
L.G.
16
to prove peatedly,
NI inequality we obtain
Gubin
of the type
et
(Iti -
(Xi-
x3, xi
-
=
x2) ,
=
llxi-
x2112-
(x2
511<
119 -211.
Using (6)
re-
llxi - x2112
11x*- X2112
a=
al.
-
(x’
-
x2, xi
IIS’
-
x2112
x3lx2
-
x3)
-
x2)+(x2
-
x3,
xi
-
x2)
=
51 +
(x2
-
~3,
XI
-
~3)
>‘I ’
IIS’
z2112
-
52112 =
”
+ h(x3 -x’)) = lld - lc2ll2+ h(z’ 2’) = 0; -9, ti (xi-9, 3+-z) = (39-a+, x4 - 2) + (x2 - x3, x4 - 5) = (2’ - 9, 33 Sk-2) = (9” - x2) + (2’ -x2, 9 - cz) + ( d-3.3, 33-3) > (1-x3, - Lz+_+--9) + (33-33, 9-z) > (h-l)(P9,s - 2’) B 0; 1133 - zp = llti - 3jp + lb.+- 54112 + 2(x4 - 5, 33 - 9) = Ilti - 412 + + (Iti - xqp + 2(a - 1) (x’ - ti, 33 - 3) > lIti - xl12. (x’
-
9,
&
-
x2)
Furthermore
=
(x’
-
x2, .xf -x2
we show that
0(x”)
In the same way as in the proof pw(xn, Q) Q2) <
-
P~(P+~,
=
msx(p(sn,
Qt), p(P,
of Lemma4 we obtain,
Q) = L2, that 6, + 0 and p (x0,
Q2)) + 0.
putting QI)
<
61,
p(z’,
P (x2, Vi) < 63, p(ti, Q) < 64, P (x5, Q) < 6?, etc., therefore into account that p (~3, @) = ~(“3, Qi) = p(x5, Qi) = . . . = 0 and
62,
taking
p (x2, Q2) = p (x6, Q2) =
@(x4) < p (ti,
. . . = 0, we obtain
(D(9)
< 62, CD(S) < da,
Qt) + p (x5, Qz) < 64 + (65,@(x5) < 65, i. e. in fact
uJ(x”) -a 0. With this we have proved (7) and (8) for this method. Lemmas 5 and 6 are applicable, which proves the theorem.
Therefore
We shall not obtain an estimate of the rate of convergence for this method but will merely show with two examples that it does in fact accelerate the convergence. The first example is a case of linear constraints. It is easy to see that if Q1 and !& are hyperplsnes (or halfspaces), the point x4 is already a solution (as we have seen the original method may converge very slowly for this case, see Fig. 2). In the case of sets over a two-dimensional plane Qi = { (xi,x2) : ~2 > aq2, a > 0), : x2 < 0}, considered earlier, it is not difficult to obQz = {(aix2) tain
the estimate
the convergence Thus it
Il~+ll <
(l/2n)
of the original
lls”ll,
for the accelerated
method is extremely
is to be hoped that this
simple method of
method, whilst
slow. accelerating
of projections
The nethod
17
convergence is fairly effective. We also remark that the idea of this method can be carried over to the case when the intersection of m (and not two) sets is required. In method (l’), (2) (or (l’), (3’)) for this purpose so-called accelerating steps of the type (15) have to be made for various pairs of sets. The order of choosing these pairs may be of various kinds. In particular they can be chosen such that for the case of linear constraints an accurate solution is obtained in a finite number of steps.
4. Examples and applications We shall consider the various forms of the method of successive j ections through several problems. 1.
Solution
are given
where co E is a set of The problem ing a point a half-space. h,(x)=
of
a set
of
linear
. In a Hilbert
inequalities
linear
scalar product, aa: is a number, and A infinite). We shall assume that c&O. inequalities is equivalent to finawhere each
It is easy to verify
ha,
li, = ((h,
Quc=
5) -
if if
k)llhl12,
The method (l’), (3’) - the method of projecting restriction - takes the form xn+i =
xn-( (Cqn),xn) -
{CT: (ca, Z) <
a,}
is
that
x, x -
space ,F suppose we
inequalities
E, (c, x) denotes the subscripts (finite or of solving this set of of the set Q =,?p,
(
pro-
aqn,)Cqn)/llCqn)ll?
xn,
(ca, z) < aa, (C,,Z)>
tZa;
on to the furthest
if
(Cw, x*1 > aq4t
if
(h(n),
Xn)
<
%nb
(17) where a(n)
is determined
from the condition
(3’):
(cmq P(x~,Q~~))=@(X~)=W& (here
we assume that the supremum is reached).
II4l
- act
(18)
18
L.C.
Gubin
et
al.
Let us now consider another variant, that of projecting on to the restriction which is least satisfied, where xntl is computed according to (17), and a(n) is chosen from the condition ( Cqn),
P) -
sup 1(Cat zn) - 4.
Uqn) =
(j9)
USA We show that choosing
if
n(n)
0 <
condition
$c,ll <
(3) will
<
BEA
(cqn),
zn) -
a%
m Applying Theorem
with such a method of
M, a ~-4,
be satisfied.
Indeed,
(cav ;I;,,aa SUP (Ca,P)m -
a-J(z”) = sup =
m <
<
=
WEA
M -
@w,
\rn
Theorem 1 we obtain
f&c
4
-
acr(,)
IIcqn,ll
the following
=$4w2atn,).
theorem.
5
Let the system (16) have a solution and let any of the following conditions be satisfied: (a) Q” is non-empty; (b) E is finite dimensional; (c) A is finite. Then the method (17), (18) converges to a solution, and if
m <
llcall <
(a) and (c)
the method (17))
M,
the rate of convergence
(19) also
converges.
In the cases
is geometrical.
The problem considered in this section has numerous applications. Two of these will be given below in examples 2 and 4. In addition, linear programming and the solution of sets of linear equations can be reduced to this problem. We shall not however discuss these here since these applications are well known (see for example [2 - 5, 141). 2. The problem of the best uniform approximation. Over the interval [a, bl let a continuous function p(t) be given. We are required to find its best approximation (in the Chebyshev sense) by a set of known functions cpl(t), . . . . q+,,(t), continuous over [a, bl. In other words, we seek the coefficients x1, . . . , x,, minimizing
f(z)= We specify equalities
$a&
/rp(~~&~i(~) i=i
some number A > 0, and we shall
--
&i’pi(%h, i=i
1.
seek the solution
a
of the in-
(20)
The
method
of
19
projections
If the solution of (20) exists, A> f * = inf f(x); if it does not exist, case h can be reduced, and in the second it can be A < f*. In the first increased. In this wsy we can obtain the solution of the original problem with any accuracy. Thus the problem has been reduced to the solution of an auxiliary problem of the kind (20). But such a problem is a specia case of problem (16) considered above, if for A we take the double interval de[a, bl, a = t, ca.= (vi(t) ,..., (pm(t)) or ca.= --(vi(t), . . . . q,(t)), pending on whether the leftor right-hand part of the inequality in Therefore the method (17) takes the form (20) is under consideration.
I Cl Xin
Xi
nil
xsian(q(L)--i
where t,
is
xin(Pi(tn)
I]
!
5
($Ji2(t))-L
xi”rFi(t7z)),
if
1 q(L)-~
found from a condition
$j
Xi”C@(tn)
I( $
Xi”(&(tn)
1 >h,
of the type
(18)
~iZ(t*))p”*=
i=i
i=l
= or of the type
X
iI,
i=i
] Cp(tn)-
cpi(tn)
i=i
i=i
I
=
Ii
TCtn)-
lb-
max
a
[I q(t)-iXi”qi(t) / ( $j
q$(t))+]
i=i
i=i
(19)
m
If
2
q$(L) >
0,
a S t<
b, the convergence
of the methods follows
i=i
from Theorem 5 (E is finite dimensional), and if h 7 f*, the methods converge at a geometric rate (since in this case Q” is non-empty). This method of solution is extremely simple: within each step it requires only that the maximum of a numerical function over sn interval should be found.
L.C.
20
3.
The problem
function strictions 1)
o,f optimal
u(t) ELz’[O, T],
Gubin
control.
u(t) =
et
al.
It is required to find a vector
(ui(t), . . . ,u,(t)),
satisfying
re-
of the following kind:
Gi = {u : IlUll Q C}, where IlUll = j ri; n?(t)dt, 0
c > 0;
i=i
2) Qz = {U : u(t) EM for almost all t. O< 2
where M is a closed
d E Em 31 Q3= {zz : s(T) = d}, where z(t) = (s,(t), . . .,zm(t)), x(t) and u(t) are related through the differential equation
dx
Ax+Bu,
-= dt
x (0) = x0,
while
(21)
where A is an m x m matrix, and B is an r x m matrix. Certain problems of optimal control csn be reduced to this problem. For example, in a problem concerning operating speed with restrictions Qr, QS, QJ we can fix the value of T and seek a control function u(t), satisfying the restrictions. If such a u(t) is found, T is not less than the optimum, while if it is not found, it is greater than the optimum. In the other problem - that of minimizing fi urz(t)dt 0 +=i
under the restrictions Qz, Q3 - it is possible to specify sn approximate value of the functional, equal to c, and to reduce the problem to finding the intersections Q1 n QZ n Q3. We write down the projections Qz, QJ in a Hilber space Lzr(O, are lengthy, although simple) :
Pi(U) = p2(u) = bfd(U(t)), where set M at each instant t.
. . . . r, CF< t
of an arbitrary u(t) on to the set Q1, (we shall omit the proofs, since they
T)
U,
if
C~/IMI,
if
PM is a projection
For example, if
we have P*(U) = u(t),
Ilull G C, II4 > c, on to a finite
dimensional
Q2 = {u : \ur(t) 1 < 1, i = i, where
The method
1, pi(t)
=
W(t)9
i
Pa(u) = z>(t) = u-B**,
of
21
projections
if if
@i(t) > 1, (uj@)
-4, if
I< 1,
w(t) < -1,
where i&I&=
--A’*,
and v(t)
condition a(T) = d, where dx/dt = Ax t &I, x(O) = c is possible to solve the linear boundary problem
d9
dx
-A*$,
t=
In other words, it
x(0) = c,
z=Ax+Bu-BB'\p,
is chosen from
x(T)
=
d.
Thus the projection on to each of the sets Q1,Qz,QS is found without difficulty. Therefore, it is possible to use the method of successive project ion (l), (2) or (1). (3), each step of which can be realized simply. Let us now give a result concerning the convergence of the method for various sets of restrictions (we shall denote by Q the set to which u must belong). Theoren
6
Let any of the following and there exists u(t),
conditions be satisfied:
for which
/lUll
(1) C?= Qr n QJ
u Z(T) = d;(3) Q = Qz
n Q3,
and the system (21) is not degenerate [131, A is bounded and there exists u(t), for which v(t) E M for some 6 > 0 for almost all t, if Ilc(t) -
E(t)11 6 6, Z(T) = d; (3) Q =
to condition
Q1 ~Q~~Q~
and, in addition
(2), IlUll< c.
Then in the method of successive projection lim /IV - ball = 0, where u* EZQ. ?%In
fact,
Q.1,Qz, and Q3 are convex closed sets in L,‘.
In case (1)
QIO n QJ is non-empty, i.e.
condition (a) of Theorem 1 is satisfied. In cases (2) and (3) condition (a) is not satisfied, but it can be shown that Lemma5 is true. Since conditions (a) - (d) of Theorem 1 are used only in proving Lemma5, Theorem 1 is valid also in cases (2) and (3). In this way Theorem 6 is proved. 4. The pigpen
of optimaI
control
with
restrictions
over
the
phase
22
L.G.
Gubin
et
al.
coordinates. Let it be required, apart from the restrictions that the control u(t) should belong to the set Q;=
where given
{u: (q(t),
z(t))
q(t) = {e(t), . . . , q,,‘(t)} numerical
<
b(t),
is a given
0
<
Q1, Q2, Q3,
V,
function,
b(t)
is
a
function, m
(W)~s(t))= 2
Qi(l)X&),
while x(t) and u(t) are related as before through equation (21). The projection on to Q4 is difficult to find, but a different approach is possible. The set Q4 can be represented as the intersection of a continuum of sets Qt of the kind
Qt={u:(q(t), x(t)) Each set Qt is a half-space.
x(t)= where F(t) e-At,
b(t)},
4
ct =
Ct (4
at =
matrix),
of the system dx/dt
=xt (2)8’ (4 z) q(z)
b(t)-(q(t),
=
Therefore
)
{,‘I ;,“;;
xtw=
F(qxo);
uw, u(z)=
(F(t)
mere
Thus Q4 has the form (KS), where a = t, on to Qt has the form
=
= As
@(t, 2) =B’F(t)F-l(t).
0
pQ t (u)
Q~. Tl
i s(t,T)U(t)dt 0
+
is the fundamental matrix
QI = {u: (ct, U)L, < at},
s[O:
Indeed,
F(t)xo
if A is a constant
n
Q=
u(~)-[(q(t),x(t))--b(t)l[
x xt(+w+7(~),
i=l
A = LO, ~1. The projection
if
(q(t),X(t))
j ll~‘(t,r)q(t)l12d~]-i 0
if
(q(%W)>
G b(t),
x
W).
The
Therefore
cal
of
confine
22
projections
the method of successive
Qz 0 n Qt (we shall since
method
for Q = Qz
projection
ourselves
to these
cases
for
simplicity,
the presence of the sets Q1 and Q3 does not introduce cifficulties) takes the form
p+i
P2(4
=
{ where t,
PC+
is determined
9 n
tun19
if
p(un,Q2)>
p(umQtn)
if
p(u”,
p(@,
from a condition
[(q(ln),zn(tn))-b(ln)l
Q2) <
of the type
fl Q4 =
any theoreti-
Qt,),
(18):
( (jlie*(ln,r)q(ln),,2dr)-1~= 0
w%
end the problem is (if q(t) and b(t) are continuous, (q(O),x0) -K More simply the instant non-degenerate [131, this maximum is achieved). on to the least satisfied t, is determined in the method of projection restriction (see (19)):
Finally,
b(t),
if O,C
1 follows
E(t) EM,
0 <
t <
T,
exists,
such that
(q(t),
x(t))
<
t ,(T, it follows that Q2 n Q4” is non-empty and from Theorem the strong convergence of the methods described.
projection method given in [151 The version of the successive _ _ is more complicated for this problem. Apart fromthis, the paper 1151 proves only the weak convergence of that method. Tram
lated
by
H.F.
Cleaves
REFERENCES 1.
POLYAK, B.T. Existence theorems and convergence of minimizing sequences in extremum problems with restrictions. Dokl. Akad Nauk SSSR, 166, 8, 287 - 290. 1966.
2.
AGMON.S. The relaxation 6. 3, 392 - 393, 1954.
method for
linear
inequalities.
Cnn.J.~~nth.
24
3.
4.
L.G.
MOTZKIN. T.S. equalities. EREMIN, Usp.
al.
and SBOENBERG, I. The relaxation J. Math. 6, 3, 393 - 404,
Can.
I. I. mat.
et
Gubin
Generalization 20. 2,
of the Motskin-Agmon 183 - 188, 1965.
Nauk,
5.
BREGMAN, L;M. Finding the of successive projection. 490. 1965.
6.
CHENN, 1. and GOLDSTEIN, A.A. Proximity Am. Math. Sot. 10, 3, 448 - 450, 1959.
7.
LEVITIN, E.S. strictions.
linear
relaxation
maps for
convex
sets.
and PGLYAK, B.T. Methods of minimizing under vychisl. .Mat. mat. Fiz. 6, 5, 78’7 - 823,
Closest-point 3, 212 - 225.
9.
POLYAK, B.T. A general method of solving Akad. Nauk SSSR, 114, 33 - 36, 1967.
10.
BOURBAKI, N. Topological 1959.
11. BROWDER, F.E. Ftoc. natn.
maps and their 1965.
vector
spaces,
Nonexpansive nonlinear Acad. Sci. U.S.A. 54,
GDBDE, D. Zum Prinzip - 4, 251
der
- 258,
in-
method,
products.
Nieuw.
extremum
problems.
Izd.-vo
in.
Arch.
Do& 1.
Moscow,
lit.
operators in a Banach 4, 1041 - 1044, 1965.
kontraktiven
Abbildung.
Proc.
re1966.
Zh.
STILES, W.J. Wisk., 13,
3
for
common point of convex sets by the method Dokl. A&ad. Nauk SSSR, 162, 3, 487 -
8.
12.
method 1954.
space.
Nachr.
lath.
30,
1965.
13.
LEVITIN, E.S. and POLYAK, B.T. On the convergence of minimizing sequences in conditional extremum problems. Dokl. Akad. Nauk SSSR, 168, 5, 997 - 1000, 1966.
14.
TOMPKINS, C. Projection linear
15.
Progr.,
2,
methods 425
VAISBORD, E.M. The method mate solution mat.
Fiz.
6,
of 6,
-
448,
in calculation. Proc. Washington, 1955.
of successive
an optimal
971 - 980,
control
1966.
projection problem.
Zh.
for
II
Symp.
the
vychis
approxi-
1.
!vlat .
GUBIN .
B. T.
POLYAK and E.V.
RAIK
MOscow (Aeceived
29
November
1966)
MANYmathematical and applied problems can be reduced to finding some common point of a system (finite or infinite) of convex sets. Usually each of the sets is such that it is not difficult to find the projection of any point on to this set. In this paper we shall consider various methods of finding points from the intersection of sets, using projection on to a separate set as an elementary operation. The strong convergence of the sequences obtained in this way is proved. Applications are given to various problems, including the problem of best approximation and problems of optimal control. Particular attention is paid in the latter case to problems with restrictions on the phase coordinates.
1. Methods of successive projection A prqject point .+(x)
ion
E
of the point x on to the set ;; of a normed space I’? is a .:, such that (see Fig. 1) 112--R(z)ll=
Theproperties of projections
infib-yll= p(z,R). tlER
of projections will be given in Section 2, and examples for actual sets will be given in Section 4.
Hereafter we shall assume that f? is a Hilbert space, although some of the results given below are also valid for a wider class of spaces. Suppose we are given
l
Zh.
vychisl.
Mat.
a system of sets I:)~ E E, a E
mat.
Fiz.
7,
6,
A, where :1 is a
1211 - 1226, 196’7. 1
2
L.G.
Gubin
et
finite or infinite set of subscripts. point x of this system of sets, XEQ=
al.
The problem n Qa.
is to find
some common successive
The methods of
GA
FIG.
1.
projection consists in constructing a sequence x0, x1, . . . , xn, . . . , where the point x o is arbitrary, and in order to obtain the point znt 1 from x* a set Qacn, is chosen and a step is made in the direction of projecting the point xn on to this set, i. e. Pfl Hereafter
=
39 + h, (PC+)(P)
Pacn)(z)
=
Po,(,)
-
zn) (
In particular,
(x). zn+l =
o< if
h, <
2.
(1)
A, 3 1, we have
Pa(n) (rn) 7
U’)
i.e. for xntl we take the projection xn on to gc(,,). The order in which the sets are selected (i.e. the method of choosing a(n)), can vary. If A is a finite set, 3 = {Ccl, . . ., a,), a cyclic order of projection can be used CL(n)
=
h(mod
(2)
m)+iv
where n(mod m) represents the remainder obtained from dividing n by m. In the general case the method of projecting on to the most remote set can be used, i e. p(xn,
Qacn,)= sup
p(xn, Qa)=
@(zn).
(3’)
CUSA
Since, however, such a set may not exist or it may be difficult to find, it is sufficient to solve the problem of finding the remotest set approximately, namely to choose an arbitrary a(n) satisfying the following condition:
The
fromP (xn,QCC ( R,) --t 0 it
method
of
follows
projections
thata
3
(rn) =
The convergence of these methods follows which will be proved in Section 2.
sup p (P, Qa)-+ 0. aEA
from Theorem 1 given
(3)
below,
Theorem 1 Let all
sets
Cj, be closed
and convex,
Q =
fl Qa
be non-empty and
CLEA 0
<
&i <
Let any of the following (a)
Q, n
( aO_
h,
<
2 -
conditions
Q,)" is
E2 >
E2,
0.
(4)
be satisfied:
non-empty
(here ?o
is a fixed
set
from the
a&
system f&t if0 denotes
the interior
of ‘I) ;
(b) all Oar, with the possible
exception of one (f&J, vex* with the common function S(T) ;
(c) B is
finite
(d) A = {a,, (2, = {X1
con-
dimensional;
. . . . a,,,) is finite, (Ci,
X><
Then in methods (l), strongly
are uniformly
convergent
and all
&
are half-spaces,
i.e.
/3i).
(2)
and (11,
to sane point
(3) for x0 E Q =
any x0 the semence xn is fl Qz.. In the cases (a) WEA
and (d)
in method (11,
(3’)
the rate
of convergence
is geometrical.
The methods (l), (2) and (11, (3) are suggested in [2, 31 for the finite dimensional case. Their convergence is proved in the same place. Some generalizations of the results of [2, 31 for the finite dimensional case are given in [41. In [51 the weak convergence of the methods in the general case is proved, and no conditions of the type (a) - (d) are required. The assertion of Theorem 1 concerning the strong convergence of the method and an estimate of the rate of convergence for the case (a), are obviously new.
l
A set R is called uniformly convex if there is a function E(T), S(T) > 0 when -r > 0, such that from z ER, y E 9 it follows that z E R for all z, for which lb - (X + Y) / 211< 6(11X- yll) [ll .
4
L.G.
Gubin
et
al.
Let us now consider the case when the sets ‘So do not necessarily have a common point. Here we.shall confine ourselves to finite A and to the method Cl’), (2). i.e. to the case of cyclic projection. Theorem
2
Let all them (for
Oai,
i = 1, . ..,
explicitness,
points
2; E
m - li
;j_,G,)
$oi,
m be closed
o,,)
and convex and at least
be finite.
Then it
is possible
i = 1, . . . , m such that I)ci,,Cx while
= x1,
in the method (l’),
(2)
) = Xi+l,
one of
to find i = 1, . . . ,
we have xkmtitl
-
.km+l
ana xkmti weakly converges to Xi as k -+ co. If in -* %i+l - Xi, the convergence addition any of the conditions (b) - (d) is satisfied, will be strong. If all sets C&.., apart from possibly one, are also convex*,
and
;Q,
the rate
of a geometrical
i=i
is empty, the sequence xkmti converges
1
to Xi at
progression.
Note 1. For the case of two sets such that
llZ_i - ZAl =
m = 2 we obtain
the points
x1, xz
inf IIII:- YII= P (Qan Qd.
=Qa, VEQ, .Note 2. For the strong convergence require that at least one of the sets be uniformly marks also
of the method it
is sufficient
to
(Joj) should Qai (for explicitness, convex and that Qa. n Qa, should be empty (similar re-
apply to the geometric
The method (1’).
J-1 rate of convergence).
(2) with m = 2, go,
n Q,,
= 9 has been suggested
[sI only when one of the in [sI . However its convergence is proved in sets is compact or finite dimensional. In addition, for m = 2 the result of Theorem 2 has been obtained by different methods in IIT]. The case of an arbitrary m (but assuming the compactness of one of the sets) has been considered in Ltd. The proof of Theorem 2 is given below in Section 2. Let us now consider *
yet
another version
A set H is called strongly convex, the function S(T) = y-r2, y > 0.
if
of the method, using a it
is uniformly
convex with
The
different
way of choosing zn+* =
P,(5n D (zn)
method
of
the length
of the step,
+ yn (0 (In) )--i (Pa(*) (2”) =
IIPcLc,,, (xn)
-
5
projections
Al
=
-
namely sn)),
a,;q+a
yn 2
P (xn,
(1”)
0,
Qa) .
(3”)
In other words, during each iteration a step of length yn is made in the direction of projecting on to the remotest set, and the point obtained is projected on to some selected set Q,. Theorem
3
If all and vn+O, finite
Qa are closed
sets
condition
g yn = 00, the method (1”).
number of steps,
Indeed
and convex,
Q =
zn E Q =
i.e.
Qa fl (x : g(x)
<
g(s) = The supporting
functional
nn has the form
En =
0}, sup
n Qa c&s4
gives for
is satisfied,
a solution
within
a
some 72.
where
ac54, a&
to the set
(3”)
(a)
P (2, Qa),.. {r : g(x)
(@(z~~))--~(P~(~)(s~)
-C).
<
g(z”)}
After
at the point this
it
is clear
that Theorem 3 follows from the general results of the paper [91, since the method (1’3, (3”) is a special case of the method considered in [91, and all conditions of Theorem 1 in [91 are satisfied.
2.
Proof of Theorems 1 and 2
1. The properties of projections. We shall nw describe some general properties of the operation of projection, which will be needed subsequently. It will always be understood that R is a convex closed set, and the projection of x on to R is denoted by P(x).
First of all P(x) exists [lo], p. 286). and therefore
and is unique for every x (see method (1) is meaningful.
for
example
Lemma 1 The vector
z - P(X) is supporting
to .9 at the point
P(x),
i.e.
L.G.
Gubin
et
so,
@-J%),Y--P4) If R is strongly
Proof.
we have
---P(x)) (0.
(x-P(X),Y
convex with the function
For 0 < h < 1 we have
convex nature
(5)
y E R.
x@R,~#P(x),~ER,
convex,
If R is uniformly
al.
hy +
(5')
E(T),
(1 -
we have
E R
h)i(z)
from the definition
of R. Therefore,
in view of the
of projection
lb - P(x) II2 < II5 - hy - (1 - h)P(x) 112= = lb - P(x) II2+ AW(z) - YIP + h(z - P(x), P(x) - y). Hence (x-P(x), (5).
P(x))
y-
< hljP(x) - ~112, and taking h + 0, we obtain
In the case of a strongly
that
1/2(P(x)+y)+z~R
llx -
p (4 II,
(5),
we obtain
then
convex set there when
exists
We takea=s(s-p(z))/
tk.llS:.
l/z(P(x) + y) + z E R.
w =
Replacing
0 2 (x -P(x), w -4(4)=‘/2(~--(4,Y + i.e.
(3 -
P(x),
Y -
,y by zu in
-P(x))+
,lp(x;_x,,
P(x))
such an E > 0,
(~-p(49x--(xh
-2ellx - P(x)11 c 0.
<
Finally, for a uniformly convex set, equality, since we can take e = ,S(lly -
(5”)
follows
from the last
in-
P(&)ll).
Lemma2 The projection
operator
is non-stretching,
i.e.
IIP(x) - P(y)II < 112- Yll. If R is strongly
convex,
x # y and x e
IIP(x) -P(y) If R is strongly
convex,
we have
(6)
R (or y 6?? RI, we have
II c 115- Yll.
03’)
The nethod
of projections
P(s) - P(y)!1 < dlz - yll, de Q =
7
1 ~w~~+llv--~iv)ll)
1+2Y(lls-
(6”) Roof.
Applying
(5)
twice,
(5 - W), JYY)-Jw) Adding these
inequalities
we obtain
G 0,
b--P(Y),+)
-WY))
we have
0 > (x - $4,P(x) - P(y) ) + I!P (x) - WY) II2 2 2 -lb - yll IP(z) -P(Y) II + IIW) Hence
IIP(z) -P(y)
II <
< 0.
11x-- yll.
Similarly,
using,
formula (5’) for a strictly convex set, and (5”) set (with 5(-r) = y-r?), we obtain (6’) and (6”).
for
---p(Y) IF.
instead
of
a strongly
(5),
the
convex
2. scheme of the proof for Theorem 1. It is easy to see that the statement of Theorem 1 concerning the convergence of xn follows from the following chain of lemmas.
In method (1) ZEQ=
n&
for
any choice
of a(n)
for
every point
on to
andforalln
%A
IIZn+i- 2116 IIP - zll.
(7)
Lemma 4
In the method Cl),
(2) and (l),
lim 0 (zn) =
R-kCS
0,
(3) with the condition
(4) on h,
CD(x)= SUPP(Gu).
(8)
%SA
Lemma5 If any of the conditions (a) - (d) are satisfied, (8). the following condition sequence xn, satisfying lim p (xn, Q) = 0. Tl-rco
for any bounded is satisfied (9)
8
L.G.
Gubin
et
al.
Lemma 6
If for a convex closed set Q end sequence a? the conditions (9) axe satisfied, xn converges to some point x* E Q.
(7) and
The proofs of Lemmas3 to 6 will be given below. 3. Proof of Lemma .3. Using formula (5) for the sets condition O< h, ,12, we obtain for x EZ Q
4. hoof
of I!&URU4,
p(xn, 0) = 11~ - P,(P) THUS,
p(S’,
Using Lemma3 with n = k$(xn),
II >
Q) monotonically
Ilc+i - P&P) = p (P+‘, Q) . decreases,
Qatn f and the
we obtain
ll 2 ll~+i - pc, (xn+i) ll =
and therefore
there exists
PRO*
P =Iimp(x*,Q), n-+“W ~~he~ore,
Q) - p2(xn+l, Q) > 11x*- PQ(x”) 112 - IIxn+i - PQ(xn) 112 = = llxfl - PQ(xn) II2- IW + hn (Pct(72,(2”) - Zn) - PQ (@) II2 =
p2(x”,
-- llxn - P&y12 - b" -
-
Z?w(z=
+
%I(~cZ,?~)(~
-~Q(2n),&(?&)(z") -
z",PQ(Zn)
Using Lemma1 and condition p2W
pQ(z")!i2 -
xn)
=
k2ik(n)(xn) %&
-
-
d12
-
~~)llP~~~~(~n)
-
d2
+
-PC&~*)).
(4). we find
Q) - @(x~+*, Q) a ~32tl~~n,(~~)
But, as has been shown, p (~9, Q) --f p, that
therefore
-
d12.
(W
from (10) it follows
P(@, Qacn,)= IF - Pa(n)(sn) 11 + o. For the method (I), For the method (l),
(3) in accordance with (3) it follows that @(a?) 4 0. (2) and a certain E > 0 we choose N such that
The method
p (xn, Qw+,) <
e / 2m for
all
II > I’. Then for n >N
= hnp(zn, Qacn)) II e / m.
IIZn+* - znll = 3LnIIPcqn)(zn) - Al For every l< for for n>N
one can find
i
9
of projections
k < m
that a(n + k) = ai.
such
There-
p (P, Qai) < 11~ - ~~+~ll + [IP+~ - Pa(n+k)(P+k) II < 112”- zn+‘II + . * * . . . + ll~+k--i - ~n+k ll + ll~n+k - Pacn+k)(znfk) II < ke I m + e / 2m -C e. Therefore # (xn) = max p (xn, Qai) < e, i
which proves
Lemma 4 for
the method (I),
5. Proof of Lemma.5. Let condition
=Q,fl U-IQ~)~ end a&
llz -
xl} <
such that
(E + 6) E
0,
for
all
of all
that
a # z.
z E Qol, P,(y)
We take .y = P--o(P)
We choose
all
a # a and all
is 811 arbitrary
w =
ex/
point
(e + 6) +
6,~/
Indeed,
But lb - 41 = (6 / e) Ily - P,(Y) II < 6,
z E Qa. Thus UI is an internal
end points
if ,y
e for all a # a, then
where z = x + @/E) (.y - pa(y)). therefore
(a) be satisfied.
6 > 0 such that t E Q, for
6. We remark first p(y, QOL) <
(2).
E Qa,
point
therefore
and put E = 20(x”).
P (Y, Qa) G lb -&W)
il G IIu - Al
= P (xn, Q$+ Therefore if we form w as above, since x E &, y E Qa, therefore Therefore
+++11xn-
of sn interval
having
w E Q,. Then for a # z
+ 11~~- Pa(xn) II =
P (xn, Q,) <%
+ G = e.
we have w E Qa for all a # or. But also w E Qa. Consequently, w E 3).
XII< e(-$+f)
=$ce.
the
L.C. Cubin et al.
10
Here we have used inequality condition
that
(6)
the sequence
(whence
rn(IIrn -
IIy -
XII <
R)
xl1 <
llz’l -
211 ) and the
is bounded.
Thus,
(11) which in view of
(8) proves
Lemma 5.
Let condition
(b) be satisfied.
that there
is a subsequence
i.e.
We assume that Lemma 5 is invalid, xnk such that
kll_P(Z”k, We choose Q, (r”)
E=
min {p/4,6
E, p (rN, Q) >,
<
Let us consider
that
6.(ljPa($N)
PQ(zN) II 2
i.e.
y E
Q, for
YII =
lb--“‘ -
dicts
all
‘12(Po(rN)
hand,
with its
centre
ipa
at
E/2
+
*/2(PQ(~N)
+
1/2p(~N, Q) f
&(xN)).
It is obvious
E,
Pa(zN)
i-e.
to Q,.
(p/2)
- E a
‘/21lp,-
and
But’ Il’Pa(~N) --
6(P/4) >, 5
II s
con-
p/2
-
ami Ily(xN)
--~W
+
i. e. y E Qa. Since y E Q- and
N E
Pu (z”,)
2
+ PQ(x”))
*/2(Pa(sN)
- ZNII 3
c/2 =
I/zE,
f
entirely
P&T)ll>
-
a f a, therefore
the assumption
= N such that
nk
for a # iii, in view of the uniform
II = ‘/2II&(sN)
zNII <
and find
0,
PQ(s~) II) belongs
G([IP,(xN)
+PQp)
1/211Pol(~N)-
f
y =
hN - PQ (x”) 11-
P/4 = P/4, 1/2(Pa(zN)
-
e >
o > 0.
P/Z.
the point
of Qo, a sphere
radius
p,(s)jl
(p/4)},
N E QG. ~1 the other
vexity
Q) =
0. Hence
II d pbNI
p(rN,
Q)
\(
llxN -
lt211~N - p&N) Ii + Y211~NQ,s E
that pCxN, Q) ap/2.
When condition (c) is satisfied Lemma 5 is easily proved by contradiction. Let there be a subsequence xnk, for which p(x*k, Q) > p > 0. In view of the boundedness of the sequence {x”k), another subsequence can be separated out from it (which for simplicity will also be denoted converging to some point x*. Since p(xnk, Q,) -+ 0 when k + co by ~“~1, for every a, and Q, is closed, therefore x* E Qo for all a. Therefore z* E Q. which contradicts the assumption that p(xnk, Q)a p > 0. Finally, if condition (d) is satisfied, we denote by ,!, a finite dimensional subspace going through x0 and spanned by the vectors c r, . . . ,c,,,, and we denote by in the projection of xn on L. It is easy to prove that
The method
P WYQi)
P (zn,Qi),
=
P
w, Q) =
to show Lemma 5 over the finite the following estimate is true
from which the statement
p (Sn, Q),
for
all
s m+1 =
Let xf E IF
-
and in view of
7.
Estimating
satisfied definition
; S,,. n=o
X*1/ < (9),
closed
Q) for
But a sequence
%ll.
section.
are convex, p(z”,
all
pQ(xcn)
rate
p) a sphere with its
11 +
i.e.
(since
in view of
xm E S,)
x* ES(P~(Z”), ilpQ(xn)
(12))
-
and the estimate
P (r”, Q) Q cm (Xn) =
cp (Xn.pQqn)) < c
p(xn, Q)),
x*11
<
2p
<
(lo),
p2(xna Q)- p2(xnfi,
(I -e$$)‘h P(XnY Q).
But from (13) 2p(xn,
Q),
we obtain
Therefore
<
we have
(13)
(a) or (d) is Using the
whence &
(xn,
&(xn).
8182
~‘11
(71,
and in addition
when condition
p(sn,Q)
of method (3’)
llxn -
at
0.
of convergence.
(11) or
P(p+i,Q)
centre
P(cQ)).
n
x.11=
n-
the
follows.
of such sets must have a non-empty inter-
lim (lxn -
we have (see
case
(12)
and non-empty
Then since
km -
is sufficient
cases.
&n =n~OS(P~(~n),
Ii <
it
space L. But in this
i
6. Proof of Lemma 6. We denote by S(x, the point x and of radius p and consider
11~m-Po(zn)
therefore
dimensional [21
of the lemma immediately
Lemma 5 is proved
Then the sets S,
11
Q)< cmaxp(m,Qi),
~(5~~
Thus
of projections
Q) < 4
I-
E$)n’2
-
Q)
L.G.
12
Gubin
et
al.
We remark that the $81118 proof can be extended also to the more general method (l), (3) with some refinement of the condition (3). At the expense of some complication it is also possible to prove the geometric rate of convergence for the method (l), (2). of Theorem 2. For explicitness,
8. Proof
let Qol be bounded. Consider
the operator
pa,(Pad. *-(PmPa,(x) ) 1.- .) ),
P(x) =
W
which successively projects on to all m sets. Using inequality (6) m times, we obtain that the operator p(x) is non-stretching, i.e. IIP(s) - P(y) II
YII.
In addition it takes a closed convex bounded set Qa into itself.
Therefore [ll,
putting 5%= Pa,(E),
of a Hilbert space
121 it is a stationary ioint x1 E QaJ.
. . . , F,, =
f’,,
from the condition PG,)
(Zm-~),
=
x1 we obtain Pa,(&) =%I, i.e. the points xl, ._., FIRsatisfy the conditions of the theorem. Furthermore, for the method cl’), (2) we have ~lxitkm
-_ fill2
-
p+km+i
_
112=
his,,
-
xi+km+lII
_ x:itkm+i,
-
_
IJZ~+~ _
+
2(x’+km+’
X
IIZi+l - iT;;_ilI - (Xi+km+i - Xifkm, Ei+i - Zi) ] >
>
zjl(2) _
2 (xi+km
(llxi+k”
+
zifi,
fi+i
_ zi)
(llxi+km - xi+km+iIj -
+
xi+kmfi
2[ IIxi+km
z&+~)
+
_ xi+km+iII
x
lIZi+* .- ,ill)“.
Hence ll~i+~m- %ll > 112 i+kn+i - Zi+iII. Consequently, the quantity llri+knr_ ziII monotonically decreases and as k -, OI it approaches a limit which is independent of i. Therefore in the inequality obtained above the left-hand part approaches zero as k -. co, the thereforellzi+km Zi+km+iII+ IIz~+~- iill, and also llri+km- ri+km+lIIIl~i+i- ~?=uill -
(2’r+km+i _
xi+km
,
Zi+l
Xi+kmlla ~~;~k:zp._
Xi+kmll
(xi+km+i-
Xi+km,
-
+ -
Zi)
--t
IIs~+~ IIfi-i
fi+i
-
-Zi)
0. But II (Xi+km+i ~~112
-
Z(x”+km+i
-
Xi+km)
-
_
Xi+km,
zi+i
(fi+i _
zi)
Z;;ui) 11’ = =
+2[ll&+km+i -- xi+kml\(\5i+i - ~~(10, therefore xi+km+i- xi*km--t Ei+l - Zi.
Zi11)2
] 4
Furthermore. the sequence xltknt, k = 0. 1, . . . , is bounded; hence there exists a subsequence which is weakly convergent to some point X,. Since Qal is convex and closed, and therefore it is weakly closed, and xltkm E
Qa,, therefore
also i, E Q,,. Using the convergence $+-km-
xi+km-t& - fi, proved above it is easy to show that the corresponding
The
nethod
of projections
subsequence x Ilfkn weakly converges to
13
A = Pa, (21) = 5~4+ i!g - fi.
weak convergence of the subsemence x i+km to 4i = P,,(&_t) “i
-
;;-I
is proved similarly.
Pa, (%=J = %Fz+ zi -
The
= f+r +
Hence it follows that
zm = 5Ti+ “r: pi+* -
Z$) + 2%-
z,
= ff.
i=i
Thus, the points x”i satigfy . . . , m. -
1, Pa,(&)
=
&,
the conditions
Pa i+i$Zi) = i?i+t, i =z 1,
snd W@ can put Xui= Xi.
IFheweak convergence
-55;;uill Of all sequencesXi+km t0 Xi follows from the fact that 1/$9+km monotonically
decreases with increasing k
(which has been shown earlier).
Now let all sets Qai be uniformly convex. If all points xl, . . . , x,,, coincide, the strong convergence of the method follows from Theorem 1. Otherwise there is a uuifonnlv convex set C&., for which Xj f Gj-1 (this happens in the case remarked on in note 2 ofjthe theorem). Then Xj is a boundary point of the set Qo. (since fj = Paj (Sj-i), Sj-i@ Qaj). But if the sequence of points .xi’km’ 1s ’ weakly convergent to a boundary point of the uniformly convex set, it is also strongly convergent ([131, Theorem 3). Therefore .j+k= * _j_ But in view of the continuity of the projection operator (6)) x itkm + xi for all i. Finally if all sets Qoi, except possibly
one, are strongly convex,
among these sets there is one such that xj # xj-l* close to “j-1, x, Y s Qaj I, sufficiently
Then for
ofrc, Qocj) > p > 0, p(y, Qaj) 2 p > 0; llPaj (x) -Pa,
(y) II <
qlla: -
yll, q <
therefore
1. As a result
i.e.
%,j-l
@
‘~,i
*
we shall have from (6”) we obtain operator (14) is com-
pressing in the neighbourhood of the point Zj_1, But all xj-ltka are near “j-1 for fairly large k. Therefore the principle of compressed mappings is applicable, from which it follows that .jtkm -( Xi at the rate of a geometric progression. But then (in view of (6)) for every i xitkm will convergeto Xi at the sme rate,
3. Rate of convergence As we shall see from the examples given below, each step of the successive projection method can be realized very simply: the amount of calculation associated with one iteration is usually small. Apart from this the method is stable with respect to error. However, a serious
14
L.C. Cubin et at.
drawback of the method may appear to be its slow convergence. As can be seen from the simplest examples the rate of convergence is sometimes actually small. For example, for set over the two-dimensional plane
FIG.
2.
{(xi, x2) : 22 < 0). ft can be al 01, Q2= Qi = { (51, x2) :x2 a ai??, proved that for any initial approximation the method converges to the point (0, 0) at a rate slower than that of sny geometrical progression (namely. at the rate of the order of l/n). This case, however, is fairly typical. Indeed, if the original problem is to minimize the functional
f(x)
over the set Q1, while f =
in1 f(rf
is known, the problem is re-
duoed to finding any point of the set Q = Qt f-j Qg, where Q2 = {x: j(3) < j”} . Here normally Q consists of a unique point, so that condi tio;i fa) of Theorem I is not applicable and the geometric progression rate of convergence cannot be asserted. However, even when the geometric progression rate of convergence follows from Theorem.1, the denominator
FIG. 3. of the progression may be nearly unity= This is Fig. 2 (two hyperplanes intersecting at a small is the minimization problem considered earlier, from 2, taking Q2 = (X : f(x) f f” + E, E > O>.
the case for example in angle). Another example when it is solved apart In this case QI fl Qza is usually non-empty and the rate of convergence is geometrical, However*
The
method
of
15
projections
as follows from the estimates obtained in proving Theorem 1, the denominator of the progression is nearer unity, the smaller 6, i.e. the smaller E. The situation is similar with method (1”)) (3’): if Theorem 3 is applicable, the number of sets is finite, but may be very large. Because of this the problem of accelerating the rate of convergence is important. We shall consider one possible way of achieving this. We shall only deal with the case of two sets. Thus, we want to find x* E Q = Q1 fl Q2, We form, Lc3=
beginning
from x0,
the sequence
ri = P;(9),
22 =
p2(r1),
Pl(X"), Lz+ = xi+ a(s3-xi), 11x*-
A=
(51 -
x2112
x3, xi
-
(151
x2) *
Here Pi is a projection on to ?i, the first three points coincide with those obtained in method (l’), (2) and the value of A is so chosen that of the straight line (x1, x3) with the point x4 lies at the intersection a hyperplane which supports Q2 at the x2 (see Fig. 3). Further points are obtained in a similar manner: x5 =
x’ = Pi(X6),
x6 = PP (x5),
Pi(b),
11x5-
h=------
x8=x5+h(x’-x5),
x6112
(x5-27,x5-56)
etc.
Let us now prove
Theorem
the convergence
of this
VI n Q20 (01Q10 n
(b) Q2 (or (c)
method.
4
Let Q1 and Q2 be convex and closed, Q = Q1 any of the following conditions be satisfied: (a)
,’
Q2)
Q1) is uniformly
E is finite
flQ2 be non-empty and let
is non-empty; convex;
dimensional.
Then in the method described
above xn + X* E 3.
Proof. We shall use the ssme scheme as in proving Theorem 1. At first we shall prove that I[xR+~-xl! < llxn -xz(I~ for all n and for any x E (2. For n = 0. 1,
2, 4, 5, 6. 3,
. . . this
follows
from Lemma 3. It remSinS
L.G.
16
to prove peatedly,
NI inequality we obtain
Gubin
of the type
et
(Iti -
(Xi-
x3, xi
-
=
x2) ,
=
llxi-
x2112-
(x2
511<
119 -211.
Using (6)
re-
llxi - x2112
11x*- X2112
a=
al.
-
(x’
-
x2, xi
IIS’
-
x2112
x3lx2
-
x3)
-
x2)+(x2
-
x3,
xi
-
x2)
=
51 +
(x2
-
~3,
XI
-
~3)
>‘I ’
IIS’
z2112
-
52112 =
”
+ h(x3 -x’)) = lld - lc2ll2+ h(z’ 2’) = 0; -9, ti (xi-9, 3+-z) = (39-a+, x4 - 2) + (x2 - x3, x4 - 5) = (2’ - 9, 33 Sk-2) = (9” - x2) + (2’ -x2, 9 - cz) + ( d-3.3, 33-3) > (1-x3, - Lz+_+--9) + (33-33, 9-z) > (h-l)(P9,s - 2’) B 0; 1133 - zp = llti - 3jp + lb.+- 54112 + 2(x4 - 5, 33 - 9) = Ilti - 412 + + (Iti - xqp + 2(a - 1) (x’ - ti, 33 - 3) > lIti - xl12. (x’
-
9,
&
-
x2)
Furthermore
=
(x’
-
x2, .xf -x2
we show that
0(x”)
In the same way as in the proof pw(xn, Q) Q2) <
-
P~(P+~,
=
msx(p(sn,
Qt), p(P,
of Lemma4 we obtain,
Q) = L2, that 6, + 0 and p (x0,
Q2)) + 0.
putting QI)
<
61,
p(z’,
P (x2, Vi) < 63, p(ti, Q) < 64, P (x5, Q) < 6?, etc., therefore into account that p (~3, @) = ~(“3, Qi) = p(x5, Qi) = . . . = 0 and
62,
taking
p (x2, Q2) = p (x6, Q2) =
@(x4) < p (ti,
. . . = 0, we obtain
(D(9)
< 62, CD(S) < da,
Qt) + p (x5, Qz) < 64 + (65,@(x5) < 65, i. e. in fact
uJ(x”) -a 0. With this we have proved (7) and (8) for this method. Lemmas 5 and 6 are applicable, which proves the theorem.
Therefore
We shall not obtain an estimate of the rate of convergence for this method but will merely show with two examples that it does in fact accelerate the convergence. The first example is a case of linear constraints. It is easy to see that if Q1 and !& are hyperplsnes (or halfspaces), the point x4 is already a solution (as we have seen the original method may converge very slowly for this case, see Fig. 2). In the case of sets over a two-dimensional plane Qi = { (xi,x2) : ~2 > aq2, a > 0), : x2 < 0}, considered earlier, it is not difficult to obQz = {(aix2) tain
the estimate
the convergence Thus it
Il~+ll <
(l/2n)
of the original
lls”ll,
for the accelerated
method is extremely
is to be hoped that this
simple method of
method, whilst
slow. accelerating
of projections
The nethod
17
convergence is fairly effective. We also remark that the idea of this method can be carried over to the case when the intersection of m (and not two) sets is required. In method (l’), (2) (or (l’), (3’)) for this purpose so-called accelerating steps of the type (15) have to be made for various pairs of sets. The order of choosing these pairs may be of various kinds. In particular they can be chosen such that for the case of linear constraints an accurate solution is obtained in a finite number of steps.
4. Examples and applications We shall consider the various forms of the method of successive j ections through several problems. 1.
Solution
are given
where co E is a set of The problem ing a point a half-space. h,(x)=
of
a set
of
linear
. In a Hilbert
inequalities
linear
scalar product, aa: is a number, and A infinite). We shall assume that c&O. inequalities is equivalent to finawhere each
It is easy to verify
ha,
li, = ((h,
Quc=
5) -
if if
k)llhl12,
The method (l’), (3’) - the method of projecting restriction - takes the form xn+i =
xn-( (Cqn),xn) -
{CT: (ca, Z) <
a,}
is
that
x, x -
space ,F suppose we
inequalities
E, (c, x) denotes the subscripts (finite or of solving this set of of the set Q =,?p,
(
pro-
aqn,)Cqn)/llCqn)ll?
xn,
(ca, z) < aa, (C,,Z)>
tZa;
on to the furthest
if
(Cw, x*1 > aq4t
if
(h(n),
Xn)
<
%nb
(17) where a(n)
is determined
from the condition
(3’):
(cmq P(x~,Q~~))=@(X~)=W& (here
we assume that the supremum is reached).
II4l
- act
(18)
18
L.C.
Gubin
et
al.
Let us now consider another variant, that of projecting on to the restriction which is least satisfied, where xntl is computed according to (17), and a(n) is chosen from the condition ( Cqn),
P) -
sup 1(Cat zn) - 4.
Uqn) =
(j9)
USA We show that choosing
if
n(n)
0 <
condition
$c,ll <
(3) will
<
BEA
(cqn),
zn) -
a%
m Applying Theorem
with such a method of
M, a ~-4,
be satisfied.
Indeed,
(cav ;I;,,aa SUP (Ca,P)m -
a-J(z”) = sup =
m <
<
=
WEA
M -
@w,
\rn
Theorem 1 we obtain
f&c
4
-
acr(,)
IIcqn,ll
the following
=$4w2atn,).
theorem.
5
Let the system (16) have a solution and let any of the following conditions be satisfied: (a) Q” is non-empty; (b) E is finite dimensional; (c) A is finite. Then the method (17), (18) converges to a solution, and if
m <
llcall <
(a) and (c)
the method (17))
M,
the rate of convergence
(19) also
converges.
In the cases
is geometrical.
The problem considered in this section has numerous applications. Two of these will be given below in examples 2 and 4. In addition, linear programming and the solution of sets of linear equations can be reduced to this problem. We shall not however discuss these here since these applications are well known (see for example [2 - 5, 141). 2. The problem of the best uniform approximation. Over the interval [a, bl let a continuous function p(t) be given. We are required to find its best approximation (in the Chebyshev sense) by a set of known functions cpl(t), . . . . q+,,(t), continuous over [a, bl. In other words, we seek the coefficients x1, . . . , x,, minimizing
f(z)= We specify equalities
$a&
/rp(~~&~i(~) i=i
some number A > 0, and we shall
--
&i’pi(%h, i=i
1.
seek the solution
a
of the in-
(20)
The
method
of
19
projections
If the solution of (20) exists, A> f * = inf f(x); if it does not exist, case h can be reduced, and in the second it can be A < f*. In the first increased. In this wsy we can obtain the solution of the original problem with any accuracy. Thus the problem has been reduced to the solution of an auxiliary problem of the kind (20). But such a problem is a specia case of problem (16) considered above, if for A we take the double interval de[a, bl, a = t, ca.= (vi(t) ,..., (pm(t)) or ca.= --(vi(t), . . . . q,(t)), pending on whether the leftor right-hand part of the inequality in Therefore the method (17) takes the form (20) is under consideration.
I Cl Xin
Xi
nil
xsian(q(L)--i
where t,
is
xin(Pi(tn)
I]
!
5
($Ji2(t))-L
xi”rFi(t7z)),
if
1 q(L)-~
found from a condition
$j
Xi”C@(tn)
I( $
Xi”(&(tn)
1 >h,
of the type
(18)
~iZ(t*))p”*=
i=i
i=l
= or of the type
X
iI,
i=i
] Cp(tn)-
cpi(tn)
i=i
i=i
I
=
Ii
TCtn)-
lb-
max
a
[I q(t)-iXi”qi(t) / ( $j
q$(t))+]
i=i
i=i
(19)
m
If
2
q$(L) >
0,
a S t<
b, the convergence
of the methods follows
i=i
from Theorem 5 (E is finite dimensional), and if h 7 f*, the methods converge at a geometric rate (since in this case Q” is non-empty). This method of solution is extremely simple: within each step it requires only that the maximum of a numerical function over sn interval should be found.
L.C.
20
3.
The problem
function strictions 1)
o,f optimal
u(t) ELz’[O, T],
Gubin
control.
u(t) =
et
al.
It is required to find a vector
(ui(t), . . . ,u,(t)),
satisfying
re-
of the following kind:
Gi = {u : IlUll Q C}, where IlUll = j ri; n?(t)dt, 0
c > 0;
i=i
2) Qz = {U : u(t) EM for almost all t. O< 2
where M is a closed
d E Em 31 Q3= {zz : s(T) = d}, where z(t) = (s,(t), . . .,zm(t)), x(t) and u(t) are related through the differential equation
dx
Ax+Bu,
-= dt
x (0) = x0,
while
(21)
where A is an m x m matrix, and B is an r x m matrix. Certain problems of optimal control csn be reduced to this problem. For example, in a problem concerning operating speed with restrictions Qr, QS, QJ we can fix the value of T and seek a control function u(t), satisfying the restrictions. If such a u(t) is found, T is not less than the optimum, while if it is not found, it is greater than the optimum. In the other problem - that of minimizing fi urz(t)dt 0 +=i
under the restrictions Qz, Q3 - it is possible to specify sn approximate value of the functional, equal to c, and to reduce the problem to finding the intersections Q1 n QZ n Q3. We write down the projections Qz, QJ in a Hilber space Lzr(O, are lengthy, although simple) :
Pi(U) = p2(u) = bfd(U(t)), where set M at each instant t.
. . . . r, CF< t
of an arbitrary u(t) on to the set Q1, (we shall omit the proofs, since they
T)
U,
if
C~/IMI,
if
PM is a projection
For example, if
we have P*(U) = u(t),
Ilull G C, II4 > c, on to a finite
dimensional
Q2 = {u : \ur(t) 1 < 1, i = i, where
The method
1, pi(t)
=
W(t)9
i
Pa(u) = z>(t) = u-B**,
of
21
projections
if if
@i(t) > 1, (uj@)
-4, if
I< 1,
w(t) < -1,
where i&I&=
--A’*,
and v(t)
condition a(T) = d, where dx/dt = Ax t &I, x(O) = c is possible to solve the linear boundary problem
d9
dx
-A*$,
t=
In other words, it
x(0) = c,
z=Ax+Bu-BB'\p,
is chosen from
x(T)
=
d.
Thus the projection on to each of the sets Q1,Qz,QS is found without difficulty. Therefore, it is possible to use the method of successive project ion (l), (2) or (1). (3), each step of which can be realized simply. Let us now give a result concerning the convergence of the method for various sets of restrictions (we shall denote by Q the set to which u must belong). Theoren
6
Let any of the following and there exists u(t),
conditions be satisfied:
for which
/lUll
(1) C?= Qr n QJ
u Z(T) = d;(3) Q = Qz
n Q3,
and the system (21) is not degenerate [131, A is bounded and there exists u(t), for which v(t) E M for some 6 > 0 for almost all t, if Ilc(t) -
E(t)11 6 6, Z(T) = d; (3) Q =
to condition
Q1 ~Q~~Q~
and, in addition
(2), IlUll< c.
Then in the method of successive projection lim /IV - ball = 0, where u* EZQ. ?%In
fact,
Q.1,Qz, and Q3 are convex closed sets in L,‘.
In case (1)
QIO n QJ is non-empty, i.e.
condition (a) of Theorem 1 is satisfied. In cases (2) and (3) condition (a) is not satisfied, but it can be shown that Lemma5 is true. Since conditions (a) - (d) of Theorem 1 are used only in proving Lemma5, Theorem 1 is valid also in cases (2) and (3). In this way Theorem 6 is proved. 4. The pigpen
of optimaI
control
with
restrictions
over
the
phase
22
L.G.
Gubin
et
al.
coordinates. Let it be required, apart from the restrictions that the control u(t) should belong to the set Q;=
where given
{u: (q(t),
z(t))
q(t) = {e(t), . . . , q,,‘(t)} numerical
<
b(t),
is a given
0
<
Q1, Q2, Q3,
V,
function,
b(t)
is
a
function, m
(W)~s(t))= 2
Qi(l)X&),
while x(t) and u(t) are related as before through equation (21). The projection on to Q4 is difficult to find, but a different approach is possible. The set Q4 can be represented as the intersection of a continuum of sets Qt of the kind
Qt={u:(q(t), x(t)) Each set Qt is a half-space.
x(t)= where F(t) e-At,
b(t)},
4
ct =
Ct (4
at =
matrix),
of the system dx/dt
=xt (2)8’ (4 z) q(z)
b(t)-(q(t),
=
Therefore
)
{,‘I ;,“;;
xtw=
F(qxo);
uw, u(z)=
(F(t)
mere
Thus Q4 has the form (KS), where a = t, on to Qt has the form
=
= As
@(t, 2) =B’F(t)F-l(t).
0
pQ t (u)
Q~. Tl
i s(t,T)U(t)dt 0
+
is the fundamental matrix
QI = {u: (ct, U)L, < at},
s[O:
Indeed,
F(t)xo
if A is a constant
n
Q=
u(~)-[(q(t),x(t))--b(t)l[
x xt(+w+7(~),
i=l
A = LO, ~1. The projection
if
(q(t),X(t))
j ll~‘(t,r)q(t)l12d~]-i 0
if
(q(%W)>
G b(t),
x
W).
The
Therefore
cal
of
confine
22
projections
the method of successive
Qz 0 n Qt (we shall since
method
for Q = Qz
projection
ourselves
to these
cases
for
simplicity,
the presence of the sets Q1 and Q3 does not introduce cifficulties) takes the form
p+i
P2(4
=
{ where t,
PC+
is determined
9 n
tun19
if
p(un,Q2)>
p(umQtn)
if
p(u”,
p(@,
from a condition
[(q(ln),zn(tn))-b(ln)l
Q2) <
of the type
fl Q4 =
any theoreti-
Qt,),
(18):
( (jlie*(ln,r)q(ln),,2dr)-1~= 0
w%
end the problem is (if q(t) and b(t) are continuous, (q(O),x0) -K More simply the instant non-degenerate [131, this maximum is achieved). on to the least satisfied t, is determined in the method of projection restriction (see (19)):
Finally,
b(t),
if O,C
1 follows
E(t) EM,
0 <
t <
T,
exists,
such that
(q(t),
x(t))
<
t ,(T, it follows that Q2 n Q4” is non-empty and from Theorem the strong convergence of the methods described.
projection method given in [151 The version of the successive _ _ is more complicated for this problem. Apart fromthis, the paper 1151 proves only the weak convergence of that method. Tram
lated
by
H.F.
Cleaves
REFERENCES 1.
POLYAK, B.T. Existence theorems and convergence of minimizing sequences in extremum problems with restrictions. Dokl. Akad Nauk SSSR, 166, 8, 287 - 290. 1966.
2.
AGMON.S. The relaxation 6. 3, 392 - 393, 1954.
method for
linear
inequalities.
Cnn.J.~~nth.
24
3.
4.
L.G.
MOTZKIN. T.S. equalities. EREMIN, Usp.
al.
and SBOENBERG, I. The relaxation J. Math. 6, 3, 393 - 404,
Can.
I. I. mat.
et
Gubin
Generalization 20. 2,
of the Motskin-Agmon 183 - 188, 1965.
Nauk,
5.
BREGMAN, L;M. Finding the of successive projection. 490. 1965.
6.
CHENN, 1. and GOLDSTEIN, A.A. Proximity Am. Math. Sot. 10, 3, 448 - 450, 1959.
7.
LEVITIN, E.S. strictions.
linear
relaxation
maps for
convex
sets.
and PGLYAK, B.T. Methods of minimizing under vychisl. .Mat. mat. Fiz. 6, 5, 78’7 - 823,
Closest-point 3, 212 - 225.
9.
POLYAK, B.T. A general method of solving Akad. Nauk SSSR, 114, 33 - 36, 1967.
10.
BOURBAKI, N. Topological 1959.
11. BROWDER, F.E. Ftoc. natn.
maps and their 1965.
vector
spaces,
Nonexpansive nonlinear Acad. Sci. U.S.A. 54,
GDBDE, D. Zum Prinzip - 4, 251
der
- 258,
in-
method,
products.
Nieuw.
extremum
problems.
Izd.-vo
in.
Arch.
Do& 1.
Moscow,
lit.
operators in a Banach 4, 1041 - 1044, 1965.
kontraktiven
Abbildung.
Proc.
re1966.
Zh.
STILES, W.J. Wisk., 13,
3
for
common point of convex sets by the method Dokl. A&ad. Nauk SSSR, 162, 3, 487 -
8.
12.
method 1954.
space.
Nachr.
lath.
30,
1965.
13.
LEVITIN, E.S. and POLYAK, B.T. On the convergence of minimizing sequences in conditional extremum problems. Dokl. Akad. Nauk SSSR, 168, 5, 997 - 1000, 1966.
14.
TOMPKINS, C. Projection linear
15.
Progr.,
2,
methods 425
VAISBORD, E.M. The method mate solution mat.
Fiz.
6,
of 6,
-
448,
in calculation. Proc. Washington, 1955.
of successive
an optimal
971 - 980,
control
1966.
projection problem.
Zh.
for
II
Symp.
the
vychis
approxi-
1.
!vlat .