Weak and norm convergence of a parallel projection method in Hilbert spaces

Weak and Norm Convergence of a Parallel Projection Method in Hilbert Spaces G. Crombez University of Ghent Kr&$aan 281 / S9 B-9000 Gent, Belgium

Transmitted

by Melvin R. Scott

ABSTRACT We

that for

convex sets

parallel projection

a Hilbert

the variable to

parameters,

point of

for finding

under very

intersection

of

under additional

conditions

the

common

about

sequence

is

sets. We

prove norm

on the

sets.

point

variable weights convergent of the

1. Methods sets

for

a common

a Hilbert

have

their applicability control

of a

attracted

such fields

problems

2, 8,

number

attention image recovery, More precisely,

in

of

convex

years, due optimization and a (real

complex)

Hilbert H are r nonempty convex sets = 1,. , r) with nonempty intersection C* = n r= ,Ci, and it is desirable to find a point in C* in an iterative way, starting from a point X. In the mathematical literature this problem is known under the name of the convex feasibility problem. The idea for solving this problem is to use the projections Pi onto the different sets Ci, and to combine them appropriately to form an operator T such that, starting from an element x in H, the sequence T”x converges (weakly or strongly) to an element in C*; hence the name, method of Projections Onto Convex Sets (POCS), or just projection method for short. Originally, the operator T was formed in a sequential manner with the then relaxation constants pure projections Pi, i.e., T = P,.P,_l *a* P,P,;

APPLIED MATHEMATICS

AND COMPUTATION

0 1993 6. Crombez Published by Elsevier Science Publishing

56:35-48

Co., Inc., 1993

35

0096-3003/93/$0.00

36

G. CROMBEZ

hi > 0 were introduced to form from each Pi the corresponding T, given by Ti = 1 + hi( Pi - 1)with 1 denoting the identity operator on H, and T was formed by T = T,.T,_, **. T2TI; instead of relaxation constants Ai that keep their

fixed

constant

value

during

the

iteration,

relaxation

functions

A{()

whose value may change at each iteration step could be used. With the appearance of parallel computers, the interest went to parallel methods (also called block-iterative methods) of combining the projections. In its most elementary form, positive constants (called weights) czi are chosen such that Cl= r oi = 1, and then T is constructed as T = Cr= 1ai Pi. Here also, relaxation constants Ai may be introduced, such that T is given by T =

Cl= ,cqT,, with Ti given as above by T, = 1 + hi(Pi- 1). For

the

sequential

as well

as for the

{T “x)z= o is usually only weakly convergent

parallel method, to a point of C*.

the sequence Hence, a first

problem, for which partial solutions have been found, was to find additional conditions concerning the sets Ci to assure convergence in norm of the sequence {T”x}~=,. A second problem that has attracted the attention of mathematicians has to do with the speed of convergence in the parallel method since the way in which the weight functions and relaxation parameter functions are chosen (variable)

at each iteration

step provides a tool of dynamic parallel processing

that may influence the speed of convergence. In Section 3 we show that for a slightly adapted version of the parallel method, under very mild conditions concerning the weight functions aiO and under a general condition concerning the relaxation functions Ai( ), the corresponding sequence is weakly convergent to a point of C *. In Section 4 it is shown that under appropriate conditions about the sets Ci, the convergence of that sequence is in fact convergence in norm of the Hilbert space.

2.

MATHEMATICAL

PRELIMINARIES

H is a complex Hilbert space with inner product (, > and norm )I 11derived from that inner product. 1 denotes the identity operator on H. C,(i = 1,2,. . .) t-1 are a finite number of nonempty closed convex sets in H with nonempty intersection C* = n r= ,Ci, and Pi (i = 1,. , r> denoting the corresponding (shortest distance) projection operators onto Cj. A,( ), h2( ), . , A,( ) are t- real valued strictly positive functions defined on H; they are called relaxation parameter functions. a,( >, (Y,( >, . . , a,( ) are r + 1 real valued strictly positive functions defined on H such that C;_, C$X) = I, Vx E H, the so-called weight functions. From Pi and Ajo, the operator Ti is formed, given by

Ti=l+Ai()(Pi-l),

i=l

,...,r,

(1)

37

Weak and Norm Convergence and these are combined

s=

ao( )l

to form the operator

+ i

2

q( )Ti,

operator

+)

= 1,

Vx E H.

(2)

j=o

i=l

(The

S as given by

S in (2) slightly deviates

from the parallel

operator

T that

usually appears in the parallel method, namely T = CL = 1 &Tk with C; = 1 & = 1 and each Pk being a strictly positive constant or function. This deviation may disappear if we think of introducing H as another closed convex set C, with corresponding

projection

operator

P, =

1;then the corresponding

(1) is the identity operator 1,and (2) has precisely For any given x in H, Sx is given by

SX = a,(

T, in

the usual parallel form).

C ai( X)(X + hj( x)( pjx - ‘))

X)X +

i=l

= i aj(“)” j=O

+

k cq(X)hj(X)(PjX

-r),

i=l

and hence for each x in H, Sx in (2) may also be written in a form in which (Y,&) does not appear explicitly,

sx =x

+

namely,

2 cq(X)hj(X)(PjX

-x),

(3)

i=l

but with C;=,C+Y) < 1. When, in particular, all Ai0 obtain the expression

Sx =x

are equal and we denote

+ A(x)

c

a,(x)(P,x

-x),

them by A(\(>,we

(4)

i=l

in which only one relaxation function is present. For a given starting point x, we form the sequence x, SX, S’x, . . . , S”X, . . and we want to find conditions on a& Xj = 0, 1,.. . , r> and Ai( Hi = 1 . . 1r) for which the sequence (S”x) is (at least weakly) convergent to a p&t of C*. To this end, use will be made of the following theorem, the proof of which may be derived from the proof of Theorem 2.3-4 in [9]. For further reference we state it as Theorem 1.

38

G. CROMBEZ

THEOREM 1. starting point x:

Let T: H -+ H be a map with the following

(i) The set F offixed points of T is nonempty. (ii) F is closed and convex. (iii) IIT”+ ’ x - yll < (IT”x - yll, Vy E F, Vn = 0,1,2,. (iv) T”x - T”+l x+Oa.sn-++m. (v) Whenever a subsequence {T”‘x) of {T”x} element y1 in H, then y1 is a fixed point of T. Then the sequence

(T “~}n+=“~is weakly convergent

We will suppose that for the starting point true.

properties for a

..

is weakly convergent

to a fired

to an

point of T.

x, the following Proposition

is

PROPOSITION1

l[TiSnx - yjl < IIS”x - yll,

At the end of Section

tly E C*,

3 it wiIl be shown that Proposition

when, for each relaxation function n = 0, 1,2,. . . 3.

WEAK

Vi = 1,. . . , r, Vn = 0,1,2,,

CONVERGENCE

OF

We prove the following theorem {S”x}, with starting point x.

1 is certainly

Ai( >, we have that 0 < &(S”x)

THE

true

< 2 for all

SEQUENCE

about the convergence

of the sequence

THEOREM 2. When, for the starting point x, Proposition 1 is true, and when none of the sequences a;.(S”xxj = O,l,. . . , r) and h,(S”x)(i = 1,. , r> is convergent to zero, then the sequence {S”r} is weakly convergent to a point of C*.

The

conditions

on the weights

oj( > in Theorem

2 are very mild when

compared to conditions already appearing in the literature (as in [6] and [7]), especially since they also guarantee convergence in norm under appropriate conditions on the Ci, as will be shown in Section 4.

39

Weak and Norm Convergence The proof of Theorem that, for the operator

2 will follow from a series of lemmas

S of Theorem

LEMMA 1. The set offixed

PROOF. For constant

oj(j

2, the conditions

of Theorem

that show

1 are true.

points of S is C*

= 0,

1,...,r) and constant

hi(i =

1,. .,r>,

the proof appeared in [3]. We repeat it here (with the necessary adaptations) for the sake of convenience. It is clear that for y E C* we have that Sy = y, as follows from (3). Conversely, when y E H is a fured point of S, then again from (3) we derive that then necessarily Cy= iq( y)h,(yXP, y - y> = 0. Denoting cu,(y)h,( y) = a,(y) > 0, we show that the condition CL= iai( y)(P; y - y) = 0 implies that Pi y = y for i = 1,. ,r, leading to the result that y E c*. Indeed, since then P,y - y = - Cy, :( ai( ~)/a,.( y>X Pi y - y ), we have for an element z E C * = fJ I= iCi, and denoting by Re(, > the real part of (, >

Re(y

- Pry, z - pry> r-1 ui(Y>

j~ln(o(f’iy-y),~-y r

and the last term may be rewritten

‘-‘*1)P,y c i=l

4

as:

r-* - yll’ + ,sl

Y>

%(Y> mRe(P,y r

Hence,

Re(y

- f’,y, z - Pry>

+c

r-1

4Y) -Re(P,y i=l %(Y)

- y, z - P,y>,

- y, z - Pjy>.

40

G. CROMBEZ

and in the right-hand

side, considered

as a sum of three terms, both the first

and the last term are nonnegative. This leads to the conclusion that if at least one of the vectors P, y - y is different from zero for some i with 1 < i < r - 1, then Re( y - P, y, z - I’, y) would be strictly positive, violating a well-known property of projection operators. Hence, we necessarily must have that P, y - y = 0 for all i = 1,2,

. , r - 1,which also leads to P,. y -

y = 0; so y E c*.

Since

C*,

n

as the intersection

of the closed

and convex

sets Ci is itself

closed and convex, and C* is also supposed to be nonempty, we see that conditions (i> and (ii) of Theorem 1 are true for the operator S. Also condition (iii) of Theorem 1 is true for S, as the following lemma shows.

LEMMA 2. llS”+r x - yll G IIS”x - yll, Vy E C*, the starting point x.

PROOF. This is an easy consequence

Vn = 0,1,2,.

of Proposition

,J;w

1 since

IIs”+ lx - yll = IlS( S”x) - yll = q)( S”r)S”x I/

+ 2 q( S”x)T,S”x

<

-

a,(

S”x)llS”r

- cq)( S”x) y

yll +

ii

‘yi(S”X)Y

i=l

i=l

2

q(

S”X)llTjS”X

II

- yll

i=l <

llS”x - yll.

n

LEMMA 3. Let pl() and p2( ) be real-valued, strictly positive functions defined on H, such that pi(z) + pp(z) = 1, Vz E H. Let R, V and Q be operator,s on H such that R = kl( >V + p2( >Q, and such that the following conditions are trme: (i) the set F offixed points of R is f $3 (ii) for some fixed point y E F and for some x E H we have

IIVR”x - yll G IIR”x - yll, llQR”x

-

yll < IlR”x - yli,

Vn = 0,1,2

,...,

tin = 0, 1,2,.

and

Weak and Norm Convergence (iii> the sequence

pul( R”x)

Then IIVR” - Rn”xll

PROOF.

Let

41 does not tend to zero for n -+ + ~0.

+ 0 as n +

+m.

y be the point of F as in (ii). We have

IIRR”x - ~11’ = 11,u~( R”x)VR”x = (/+(

+ p2( R”x)QR”x

R”x))~IIVR”

+ 2p1( R”x)pJ

- pl( R”r)

y - PU~(R”X)Y~I~

x - yll” + ( p2( R”x))~~~QR”~

R”x)Re(VR”x

- yi12

- y, QR”x - y),

and also IIVR”x - QR”rll” = IIVR”x - y112 + IIQR”x -

yll’ - 2Re(VR”x

- y, QR”x - y>

Multiplying both members of the latter equality with /-~~(R”x)p,(R”x) adding to the former, and taking into account that ,uJR”x) + /JJR”~)

and = 1,

we obtain

QR"d2

/+( R”x) p2( R”x) IIVR”x -

IIVR” x - y/l” + p2( R”x)llQR”x

= pl(R”x)

and, due to the assumptions

- yl12 - IIRR”x - yll”,

in (ii), we derive

pl( R”x) pz( R”x)llVR”x

- QW12

< IIR”x - yll” - IIRR”x - ylt Since Q = (l/l

-

PJ >X R - pl( )V),

VR”x - QR”x =

we have

1 _ pf( R”x)

(VR”x

- Rn+%

(5)

G. CROMBEZ

42 from which we conclude

IIVR”x - w+l

that

cdl2=

For N a positive integer,

(1 -

/A~(A”x))211VA”x - QR”xl12.

this last result together

= E ,_L~( wx)(l

with inequality

(5) leads to

- /.L~( R”x))llVR”~ - Q~“~~?

n=O ( (Ix - y112-

llRNflX - yl12

< Ilx - yllZ, from which we derive that

Hence, the genera1 term of this series tends to zero for n going to infinity. By assumption (iii), this necessarily implies that IIVR"x- R"+lxll + 0 as n+ +m. n

LEMMA 4.

For the starting point x we have

lIS”x - s”+‘xlJ -+ 0 as rL-+ +m.

PROOF. Putting Q = XI= r( CI$ )/l - (Y~( ))Ti, S may be written as S = a,( )l + (1 - cq,<>>Q; h ence, S has the form of the operator R in Lemma 3, with V = 1 and pr() = (~~0. Since the set of the fixed points of S is nonempty (Lemma 11, and since the sequence cx,,(S”x) does not tend to zero for n * +m (assumption of Theorem 21, the result of Lemma 4 will follow from Lemma 3 as soon as also condition (ii) in Lemma 3 is true.

43

Weak and Norm Convergence

Since the first inequality in (ii> of Lemma 3 is trivially true (since V z l), it is sufficient to check that lleS”x - yll < I~s”x - yll for some y E C* and for n = 0, 1,2, . . . . We show, in fact, that llQS no - yll < 1lS”x - yll for all y in C*, and all n = 0, 1,2, . . . . Indeed, since, by assumption in Theorem 2, Proposition 1 is true, we have that llTiSnx - yll Q llS”x - y/l for all y E C*, 1 Vi = 1,2,. . . , r and for all n = 0, 1,2, . . Also, C[, ,(czi(Snr)/

a,(S”x))y

= y, for each nonnegative

IIQS”x- yll =

IIk i=l

<

n. Hence,

dSXn)py* -

1 - cQ(YX)

y)

II

IIS”x - yll.

n

Lemma 4 shows that also condition (iv> of Theorem 1 is true for the operator S. The last condition of Theorem 1 for S, condition (v), will be shown to be true in Lemma

LEMMA 5. n+

+m,Vi=1,2

7, for which Lemma

For the starting ,...,

5 and Lemma

point x we have

6 are needed.

llTiSnx - Sn+lxll

+ 0 as

r.

PROOF. For k = 1,2,. . . , r put

i#k

Then

S may be written as S = c+( )Tk + (1 -

q( ))&,

which again has the

form of the operator fi in Lemma 3 with V = Tk, Q = Qk and pl( ) 3 (Yk( 1. The result of Lemma 3 is applicable as soon as llQkS”x - yll G IIS”x - yll, Vn = 0, 1,2,. . . , Vk = 1,2,. . , r, and this is shown with a for yEC*, calculation in the same manner as for Q in Lemma 4. n

In the next lemma, we denote S”x E H to the set Ci c H.

by &S”x,

Ci) the distance

of the point

44

G. CROMBEZ LEMMA 6.

d( S”x, CJ = llS”x - PiS”Xll + 0

as n -+

+a,

Vi = 1,.

. , r.

PROOF. Replacing, in the statement of Lemma 5, T, by its expression, i.e., Ti = 1 + Ai()(Pj - 11, we get Ai(Snx)(PzS’“x - Snx) + (S”X - S”+lx) + 0. Since also S”X - Sn+ix + 0 (Lemma 4), and since by assumption in Theorem

2, hi(S”x)

is not convergent

to zero, we immediately

obtain

the

result.

n

Let {Sf”‘x} be a subsequence of { S”x} that weakly converges LEMMA 7. an element y, in H. Then y1 is a fixed point of S.

to

PROOF.

We show that

y, must belong

to C,. The same procedure

may

then be repeated to show that y, must also belong to C,, . . . , C,, hence to c*. Suppose that yi E C,, then Pi yi # yl, By weak convergence of {S”‘X} to yi it must be true that (Sri’‘’

(sn’x - p, y1,

- yl, y1 - I’, yi)

y1 - PlYI)

+

(Pl Yl

-

+ 0, and hence

also

y1> y1 - p, y1) - 0.

(6)

Given ,s > 0, it follows by applying L emma 6 to the subsequence {S”‘x) that there exists N E Z+ such that 11S”‘r - P,S”‘xll < 6, for all indices n’ of the subsequence

bigger than N. Introducing

(Y’x -

P,S”‘x,

y, - p, yJ

P,S”‘x

into (61, we get

(P,S”‘x- p, y1,

+

y1 - p, yJ

- IIP, y1 - y#+o.

(7)

Taking the real part of (7) we have, respectively, Re(S”‘x

- P,S”‘x,

y1 - P, yl)

for the first and second term

< IIS”‘x - P,S”‘xll <

IIyl - P, yll(

&II y1 - p, y1ll

for n’ > N, and Re( P,S”‘x: - P, yl, yl - P, y,) < 0 for all n’, by a wellknown property of projection operators. Hence, choosing .s < all yi - P, ylll

Weak and Norm

from

(7),

necessarily

45

Convergence

we immediately

arrive

at a contradiction;

this

shows

that

belongs to C,.

Since from the foregoing

lemmas it follows that all conditions

yi n

in Theorem

1 are true for the operator S fulfilling the conditions of Theorem 2, we conclude that Theorem 2 is true; i.e., the sequence {S”x} is weakly convergent to a point x* E C*. We remark that Proposition

1 in Theorem

2 is not the same as nonexpan-

sivity of the operators Ti, for which it is needed that llT,u - T,vll Q 11~ - ~11 for all U, u E H. When the relaxation function hi( > is not a constant function, nonexpansivity is not guaranteed, even for values of hi( > smaller than 1. In the next theorem we show that Proposition 1 is certainly true when the relaxation functions Aj( > have positive values not exceeding 2.

THEOREM3. When for i = 1,2,. . . , r we have that 0 < Ai the starting point x and for all n = 0, 1,2, . , then Proposition

Q 2 for 1 is true.

PROOF. For shortness, we denote a general element S”x by the letter Z. So we have to show that llT,z - yll < llz - yll for all y E C*. We remember that, for y E C*, we always have Pi y = y. We consider three cases for the values of A,(z). (i) When 0 < A,(z)

< 1, then

Tjz - y = (1 -

A<(z))@

- y) + Ai(z)(Piz

- P,y),

and hence,

llT
- yII + A\i(z)IIpiz

- ‘iyII.

Since llP,z - P,yll Q 11.~- yll by a well-known property of projections, immediately derive that, in this case, lITi - yll G II2 - yll. (ii) When hi(z) = 1, then Tjz = P,z; hence, lITi (iii) When

-

yll = IIP,z - yll = IIP,z - P,yll G II2 - yll.

1 < hi(z)

Q 2, then as in (i) we may write

T~z - y = (1 -

Ai( z))( z - y) + Ai( z)( piz - Piy)>

we

46

G. CROMBEZ

but now 1 -

hi(z)

< 0. We have

= (1 XRe(z

Ai(z))z/IZ

- y112 + 2A,(z)(l

- y, Pj.z

- F’,y)

-

hi(Z))

+ (Aj( z))~]]P,-

- P,y]]‘.

Now another known property of projections states that IIP,z - Pi yl12 < Re (z - y, P{Z - pj y ). This, combined with the fact that 1 - hi(z) < 0, leads to

lIT,z - yl12 =G(1

-

Ai( =))‘\lz

+[2A,(~)(l = (1 -

-

Ai( z))“IIz

-

yI12

hi(~))

+

(Ai(z))‘]IIP,z

- y112 + Ai( 2)(2

-

-f’iyI12

Aj( z))llPjz

- piyl12 (8)

For Ai

= 2, we immediately

have that

IIT,z - yl12G II2 - ~11~. For 1 < that IIP,z - piyll G llz - yll

Ai( z) < 2 we get from (81, when we remember

llTiZ - yl12 < 11~- yl12[1 - 2Ai( 2) + (Ai( z))” + 2’i(z) and so we again obtain ]]T,z -

4.

NORM

CONVERGENCE

yl12 G II2 - yl12. OF

THE

- (A,(z))~]’ n

SEQUENCE

In the method of projections onto convex sets, either sequential or parallel but with constant weights and constant relaxation parameters, it has been shown that under appropriate conditions about the closed convex sets Ci, the constructed weakly convergent sequence will also be convergent in norm (also called strongly convergent) [4, 5, 7, 91. Essentially, we can say that under the same additional conditions about the sets Ci, also the sequence {S”x] constructed above, which is weakly convergent to a point x * E C *, is in fact convergent in norm to x *. As an example of this assertion, we prove a theorem in which for three different assumptions about the sets Ci it is shown that the sequence {Snr] is really convergent

in norm.

47

Weak and Norm Convergence

THEOREM4. i = 1,. , . , t-:

Ifany

of th e f 0 11owing conditions

is satisfied for the sets Ci,

(a) There exists an index k E (1,2, . . . , r} such that

(b)

All Ci, with the possible

convex. (c) Each

Ci is a half-space,

exception i.e.,

of one of them,

Ci = {x E H:

(x,

are uniformly

zi> Q pj for

some

zi E H and some pi E R} then the sequence

{ Yx}

of Theorem

2 is convergent

in norm to a point in C *.

PROOF. The sequence (S’ x} is weakly convergent to a point x * E C *, and hence it is bounded. According to Lemma 6, we also have that lim, ~ + m

SUPl.ciG ,.d(S”x,

Cj) = 0. It then follows from 15, Lemma

d(S”x,C*)

= /(S”x - Pc,S”xll

where Pc. z denotes the projection

--f 0

as

n +

51 that +w,

of a point z in H onto C*. Now it follows

easily that this last property is sufficient to guarantee the norm convergence of the sequence { S”x} to x*. Indeed, from Lemma 2 we have, for any positive integers n and k and for any y E C*

IIS“+kx - yll < IISn+k-lX - yll < *** < IIYX - yll, and hence,

since

IISn+kx - S”xll Q llSn+kX - yll + II y - SflXII, we derive that 11sn+kX - S”xll < 2llS”x - yll. In particular, taking for y the point PC* S”X, we have that

IISn+k~ -

S”xll < 2llS”x

- Pc,S”xll

as seen above. This implies that the sequence and hence it converges

in norm to the point x*.

+ Oas n -+ +m {Yx}

is a Cauchy

sequence, W

48

G. CROMBEZ

REFERENCES D.

Butnariu

method

and Y. Censor,

for solving

puter Mathematics G. Crombez,

4379-94

(1990).

recovery

155:413-419

by convex

G. Crombez,

Parallel

Math. J , 42:445-450

common

Pierro

behavior

of a block-iterative

problems,

International

combinations

methods

in image recovery

and A. N. Iusem,

A parallel sets,

projection

Pesquisa

B. T. Polyak, and E. V. Raik, The method point

of convex

sets,

U.S.S.R.

Strong

convergence

of finding

5:1-20

of projections

Comput.

of projection-like

Optim. Theory Appl. 56:433-461 Pierra,

Decomposition

Programming

Math.

a

(1985).

for finding

and Math.

Phys.

28:96-115

2-applications

Imaging

1:95-102

D. C. Youla, projections, York, 29-77

in Hilbert

spaces,

through

formalization

in a product

space,

Math.

(1984). and

numerical

by the method

results,

IEEE

of convex projections:

Transactions

on

Medical

(19821.

Mathematical in Image (1987).

methods

(1988).

M. I. Sezan and H. Stark, Image restoration Part

method

(1967).

N. Ottavy, G.

J, Math.

onto convex sets,

Operucional

L. G. Gubin,

J.

of projections,

by projections

the

7:1-24

of Com-

(1992).

point of a family of convex

common

projection

Journal

(1991).

Czechoslovak A. R. De

the

feasibility

Image

Anal. A&.

On

convex

theory

Recovery:

of image restoration Theory

by the method

and Application,

Academic

of convex Press,

New