- Email: [email protected]
Chaotic relaxation
LINEAR
ALGEBRA
Chaotic
AND
ITS
199
APPLICATIOSS
Relaxation*
D. CHAZAN
AND
IBM
Research Center,
Watson
Communicated
W. MIRANKER Yorktown
Heights,
New
York
by Alan J. Hoffman
ABSTRACT In this paper we consider relaxation methods for solving linear systems of equations.
These methods are suited for execution on a parallel system of processors.
They have the feature of allowing a minimal amount of communication of computational status between the computers, so that the relaxation on a chaotic appearance, reduces programming nature.
We give a precise characterization
of divergence,
and conditions guaranteeing
process, while taking
and processor time of a bookkeeping of chaotic relaxation,
some examples
convergence.
INTRODUCTION
In this paper we discuss for solving capable case,
linear
systems
of execution
there
a preliminary
of equations.
on a set of parallel
is a minimum
between the computers,
study These
of relaxation methods
processors.
of communication
so that the relaxation
methods
are of a form
In the most general
of computational process takes
status
on a chaotic
appearance. Let
the linear
system
be given
by
Ax=& where x is an n vector. (1) by relaxation usually
If A is symmetric
may be described
in coordinate
directions,
(0.1) and positive
as successive
definite,
univariate
of the quadratic
solving
minimization,
form
Q(x)= :(x, Ax) - (6 4. * Dedicated
to Professor
A. XI. Ostrowski on his 75th birthday.
Linear Copyright
0
(0.2)
1969
Algebra and Its Applications by American
Elsevier
2(1969), 199-222
Publishing
Company,
Inc.
200
D. CHAZAN
For
the
Gauss-Seidel
method
prior to commencing available effect
a univariate
any other.
determining
Each
However,
what displacement
Thus,
the displacement
point
x may be made at some other
tion
is a familiar
displacements
displacements
J.
decreased
[3].
the
point
x.
at the
apparently
arbitrary minimiza-
x, but
only one of the ?z
relaxation
quality
get out of
or the specific
difficulty.
was suggested
extensive
to the authors
numerical
found that
experiments
The
programming
on the relative
chaotic
form
and computer
cessors and the algorithm.
number
of the time
His experiments
of overrelaxation
with
The precise
of processors,
relaxation
r,
eliminated
in coordinating
also exhibited
allowable
by
the use of more processors
time for solving the linear system.
depended
dimension.
will rapidly
may be of different
Rosenfeld
the computation
in the amount
is to be effected,
by a processor
This
at a point
who has conducted
considerable
is
scheme n univariate
may be of varying
form of the decrease to n, the
point.
to
a processor
displace
case, the r processors
of chaotic
relaxation
that
are
assigned
at x.
performed
Rosenfeld
chaotic
and
determined
For the Jacobi
chaotic
The problem
tasks
to a minimum
since they themselves
relaxations
their
are computed
occurs
In the general phase,
complete
one.
the time
at a point x in n-space
processors
is completed r processors
may be sequentially
during
other
pattern
may
minimization
Now we suppose that
for this computation.
a minimization.
AND W. MIRANKER
the pro-
a diminution
for convergence
in a chaotic
mode. Chaotic
relaxation
by A. Ostrowski because
is related
from a number
his free steering
of predecessor
results
Schechter
for so-called
monotone
considered
block relaxation, convergence
statements
relaxation
is specialized
Sect.
1 chaotic
notion.
In addition,
subclass
of chaotic
Algebra
and
relaxation
A.
Its
[6], Ostrowski the matrix
obtained A be a so-
and obtain-
For simplicity,
for chaotic
block
to ordinary
we have not
is defined
along
2(1969),
with
and introduce
called periodic schemes.
Applications
relaxation.
free steering,
When
our results
of Ostrowski.
we give several examples schemes
In that
although the methods we use may be extended
some of the statements
In
states.
values
by compo-
[5] considered free block relaxation
to obtain recover
more general
free but the antecedent
under the condition
ed convergence
the chaotic
which has been studied
It is however
is based are chosen freely component
called H matrix.
Linear
[5].
not only is the order of relaxation
upon which a relaxation nent
to free steering
[6] and S. Schechter
199-222
a consistency an important
In Section
2 we give
CHAOTIC RELAXATION a variational
201
characterization
to chaotic
the periodic case which diverge. to a difference equation
equation
variational
convergence
characterization
to the fully chaotic and
converge.
A number
of matrices
and two examples
state
schemes space.
two convergence
result in the periodic
conditions
of corollaries
for which
chaotic
The
difference In Sect.
case by exploiting
derived in Sect. 2. Finally
sufficient
for a chaotic
4 the
in Sect. 5, we turn This theorem
iteration
gives
scheme
are then given which produce
relaxation
in
are reduced
theorems.
case and give our main theorem.
necessary
1.
in an augmented
is then used to obtain
we give another
schemes
In Sect. 3 periodic
to
classes
converges.
DEFINITION OF TERJIS In this section, we define a chaotic iteration
the notion
of consistency
then give several
examples
is a class of schemes, subsequent
scheme.
scheme
of chaotic
schemes.
periodic
schemes,
called
We then introduce
with a linear system. One of these which
We
examples
we will study
in
sections.
DEFINITION. vectors,
of a chaotic
xi, i = 1,
X chaotic iterative scheme is a class of sequences of n. . Each sequence in this class is defined recursively by
x,ii-l =
1) (1.1) E&l
b&eJj)
+
i =
(C",d),
kn+l(i).
i
The initial
n-vector
(2.1) depends
x1 and the n-vector
on a triple
(B, C, P’).
The ith rows of B and C are denoted by ci.
Y
is a sequence
For
some fixed
(i) (ii) Moreover,
integer
properties
1 < k,&,(j)
by b” = (bil,. . . , b,“) and
Y = {K,(j),.
., k,+,(j)},
i =
:
i = 1,.
< fi,
k, tl (j) = i infinitely
with the implication
The mapping
s> 0
0 < k,(j) < s,
We will identify
respectively
of 12 + l-vectors;
1,. . .) with the following
d are arbitrary.
Here B and C are a x n matrices.
a chaotic
j=l,... often
,i=l,...
.,?Z,
.
.
for each i, s < i < n.
relaxation
that once B, C, and 9
scheme
by the triple
are fixed a particular
(B, C, Y), sequence
of the class is chosen. Linear Algebra and Its Applicatiom
2(1969), 199- 222
D. CHAZAN AND W. MIRANKER
202 This time
definition
may
i, the k,+,(j)th
n -
1 components
of kkl(j),
updated infinitely Since system
as follows:
At each
of ~j is updated
are unchanged.
the second
was produced
be interpreted
component
The updating
component
instant
uses the first component
of ~j-~~(j), etc.
Every
component
often, and no update uses a value of a component
by an update
s or more steps
we will use chaotic
iteration
Ax = d, we introduce
DEFINITION.
A chaotic
iteration
system Ax = d, with solution
is
which
previously.
procedures
a consistency
of
while the remaining
to solve
notion
procedure
the
linear
in the following
is consistent
with a linear
z, if z is a fixed point of the mapping
(l.l),
i.e., z = Bz + Cd. Notice
that
From
(1.2) is independent
of Y.
now on we will assume
with the linear system the y3 are given
that
the chaotic
scheme
is consistent
Ax = d. We note then that if yj = xj -
by the following
recursive
=
c b~yaJ-k,(l’,
z, then
scheme i #
(YiJ>
Y, If1
(1.2)
h+l(j)
(1.3)
i = h‘,~_l(j).
n
(1.3)
is a chaotic
system may
iteration Since
Ax = 0.
set d = 0 without
Obtaining
Consistent
A consistent
scheme
Then
consistent
scheme
may be obtained
scheme.
the choosing
(9,
matrix
schemes,
B = D-IE
with parameter
B”, Cw). Here B” = I -
wD-lA
w.
by choosing
whose iith
A larger class of consistent
and under relaxation
homogeneous we
loss of generality.
as follows. Let D be the diagonal A.
to the consistent
Schemes.
chaotic
E = D -
corresponding
we are only considering
and
B and C
element is aii and let C = D-l
produces
schemes correspond
We denote
these
to over
schemes
and C” = wD. Note that
a by
B1 = B
and Cl = C. We will now give several
examples
= 0 and k,+,(j)
= (i -
each i, i = 1,. . ., n exactly precisely Linear
the
Algebra
Gauss-Seidel
schemes.
Let Y be defined by k,(j) = k,(j) = * . . =
Example 1 (Gauss-Seidel). k,(j)
of chaotic
l)(mod
n).
Here s = 1 and k,+,(j)
once in every relaxation
and Its Applications
n updates.
procedure.
2(1969),
199-222
This
= i, for scheme
is
CHAOTIC RELAXATION 2 (Jacobi).
Example and
k,+,(j)
203
k,+,(j)
-
Let (j -
Y
be
defined
l)(mod
n).
Here
for each i, i = 1,. . ., n exactly precisely
the
Jacobi
Here a number
as it becomes
available
time j a processor other
Schemes).
is assigned
x, are either
P is still calculating available
to relax
the
no lapping
Each
processor At
x, say. We imagine
that each of the compo-
or assigned x,.
to be updated Ultimately,
while
a newly
all assignments
until the component
For this procedure
P.
at xi
we see that
are relaxed
infinitely
subclass
of schemes
s =
often.
Schemes
periodic
schemes,
an important
These
do not correspond
schemes
corresponds
Periodic
processors repeating
compute
are suitable
to any of the classical are an orderly
simultaneously
periodic pattern is given
schemes
to the case where k,+,(j)
schemes
DEFINITION. scheme
is
to relax components.
the component
all components
We now introduce
scheme
introduce
this calculation
by processor
1 and again
Periodic
j.
= i,
processor would be assigned to update xi. The no lapping conven-
has been updated
but
K,+,(j)
This scheme
are at work.
sequentially
updated
tion rules this out and stops 2n -
We
of processors
P is assigned to relax the component
than
= . * * = k,(j) = and
procedure.
that it takes so long to perform nents
k,(j)
s = n
once in every rt updates.
relaxation
3 (No Lapping
Example convention.
by
Corresponding scheme
k,(j)
methods.
This
is a periodic
=
case.
fashion
Formally,
of The
that
a
a periodic
definition.
to each integer*
7,
1 <
k,(j) = . . . = k,(j),
7 <
n, a
periodic
and
j” - ’
(I 1 ” 1
class of
function
of the chaotic
but in such a regular
with
we call
computation
exists in the computation.
by the following
is a chaotic
version
which
for parallel
1,
j
= (i -
l)(modn).
and with k,+,(j) A periodic vertices
scheme
may be described
are the integers,
* r is the number
by a dependency
graph
whose
j = 1, 2, . . . .
of processors Linear
assigned to the relaxation Algebra
and Its Applications
calculation.
2(1969), 199-222
Il.
204 The axis is indexed being written
by j.
At the instant
into memory.
of a particular
processor.
CHAZAN
Each
AND
W.
j, the vector
MIRANKER
xi is viewed as
curve in the graph traces
For ii < ja, j2 can be reached
the activity
from ji along a
trajectory in the graph, if and only if the updated component of xJ2 (i.e., i. ci _-1j) depends on the jist iterate formally (in the sense that the %i.l 2 right-hand side of 1.1 contains a term with superscript ii). (Alternatively, in terms
of the parallel
xJ2 are successive
processors,
updates
this happens
if and only if xi1 and
of the same processor.)
The graph
for Y = 3
and n = 6 is shown in Fig. 1.1 where the label from ji to jZ is k,+, (i2 -
1).
.
4 FIG.
1.1.
Each
Graph
6
5
of periodic
case
wlth
time a vector is written,
in the computing dated
system,
components
processors
j is increased
the actual
are written
zero to some positive
three
into
number
and
by unity,
dimension
although
interval
of time between
memory
will vary
S.
in reality which up-
from essentially
in a more or less unpredictable
manner.
Conventions Unless
otherwise
symmetric
specified,
the remainder The
and A = MM*.
columns
of this paper deals with A of M are m’, i = 1, . . . , n.
D is the diagonal matrix (a,jd,i), E = D - .4, R = D-IE, B” = I - wD-IA, and C” = wD-l. In Sect. 5, A will not be
In addition,
C = D-l, required
2.
TWO
to be symmetric.
DIVERGENT
EXAMPLES
In this section we will derive a representation a sequence
of minimizations
representation acterize
to explore
the difficulties
A variational
of a quadratic two divergent
in studying
characterization
theorem.
Linear
and Its Applications
2(1969),
relaxation
as
Then we will use this which
help to char-
relaxation.
of chaotic
in the following
Algebra
examples
chaotic
case is given
of chaotic
form.
199-222
relaxation
in the periodic
CHAOTIC
THEOREM
2.1.
Let 9
scheme parametrized M,
205
TIET,AXATIOS
be a sequence
b?l Y and w.
oj iterates
Produced
by a periodic
I,t ml is the ith column of
I,et _vi = MY.
then \’i'l = J”
where
(2.1)
C.C~ minimizes j-r
:-I
-
i13’
Here
kl, -I’ll
cdm
-
!/ 1~denotes
Proof.
the E,uclidean
cd?%
%4,, (2.2)
norm.
\I’e will first set o = 1.
In this case we have
(2.3)
Let v3 be the vector whose ith component
is a,,. Then (2.3) may be written
as
_
,j
&% - l(T)
_
(2.4)
Now
xi
,ILl(‘i
4;,7:(:,
=
(2.5)
since k,,,(i) # k,+,(j), i - n < i < j (i.e., no component is updated twice before each of the n - 1 remaining ones are updated once). Using (2.5) in (2.4)
we may
write
crj as
When w # 1 it is easy to verify that cc3is replaced by u&. It is also easy k 1 (i) A,+ r(i) to verify that C$ZJ” ’ IS the displacement along v which minimizes (p+l
In terms
of yi
(2.4)
_
,i,%iIyq"p+l
and (2.7)
Lmear
_
become
Algebra
and
&)kn+q),
(2.7)
respectively
Its
Applications
2(1969), 199-222
I). CHAZAN
206 ljyj-rt
This
concludes
I _
W. MIRANKER
g,kn.il(j)112,
(2.9)
the proof of the theorem.
We note here that u3 can thus be completely of Yi-‘-rl. We now consider Exawajde 1.
AND
two divergent
determined
from knowledge
examples.
Let
and
M=
mi = (l,O, O)T,
i=
1,2,3.
Let w=1+s.
Y = 2, Then Y J+i = yJ Since yi = Md, is described
$~+,(/I
wu’m
.
y3 has the form yj = (9, 0, 0). Thus, the iteration
equivalently
by the following
iteration
scheme
scheme for the scalar
zj,
,i+1 r ,j - c()c(?. Using
(2.2),
this can be written
as
,i-i-1 = zi The
eigenvalues
of this
shown to be no smaller Fig.
2.1 where
the
(2.10) is graphically -
second-order than
evolution
of a solution
illustrated.
Algebra
and Its Afiplications
(2.10)
iterative
1 if E > 0.
1 and blows up exponentially.
Linear
e&i, scheme
may
It is instructive of the
difference
be easily to consider equation
The solution has initial values z1 = z2 = zi is located 2(1969),
199-222
at the point
labeled
j.
CHAO’l’lC
FIG.
2.1.
Evolution
axis and iterate
Example writing
207
KEIAXATION
of solution of Eq. (3.1).
Let A and M be as above.
2.
everything
in terms
This iterative
scheme
(2.11)
=
FIG. 2.2. Evolution axis and iterate
Let w = 1 and Y = 3. Again
zi-l,
(2.11) greater
in Fig.
4 II 0
2.2.
5 I 8 I
of solution of Eq. (3.3).
than one. The way the
6,7 II 2
Iteration
number 1s labeled above the
values below.
examples
with the matrix
show
that
underrelaxation
A given here.
why both of these processes examples
_
also has eigenvalues
1,2,3,9 t I -I
I I -3
or relaxation
y
blows up is illustrated
IO
These
number is labeled above the
of zi we obtain z z+l
process
Iteration
values below.
(W < 1) must
The examples
are unstable.
step as a motion
Since we view each iteration
in the direction
given here the directions
be used
also point out the reasons
of motion
mi, we can say that in the were so close together
that
tasks were duplicated. It seems as though the event when ml = m2 = . . . = m” is, in a sense, the worst case for such schemes. A%conjecture is therefore above,
that
they converge
to guarantee these
if OJ is such that
The
apparently
are arbitrarily A
In particular,
for the specific
A
(1, < 1 should be sufficient
for the two-processor
(Y = 2) case.
worst
case
None of
(ml = m2 = . . . = m”) corresponds
It seems clear that there exist nonsingular
matrices
close to the worst case form and which exhibit
DIFFERENCE
EQUATION
In this section as a first-order
converge
has been settled.
singular matrix.
3.
for all A.
convergence
questions
the schemes
AND
CONVERGENCE
IN
we will derive a representation
difference
equation
Lirwar
Algebra
in Euclidean and
Its
THE
to a which
divergence.
PERIODIC
CASE
for the periodic scheme, n +
Applications
7 -
l-space.
2(1969),
We
199- 222
208
D.
illustrate
this for I = 2.
present
The general
two cases for which
Note
that
,&,+,(I -
... =
for the
1) # k,+,(j)
scheme
W.
MIRANKER
similar.
We then
is demonstrated. k,+,(j)
for i = 1,. . ., n -
= (j -
1. Thus,
1) mod n so that
Xin+l(j) = ~~~~~~~~ =
1-(n-11 this fact can be represented by the graph 'k n+l(i) . Schematically, (for n = 7) : where the vertices of the graph are labeled (i, j).
below
I
I I
3.1.
Graph
of the
(ir, jr) is connected
periodic
case
for
Y =
4 and
**
l
n =
7
to (is, ja) if i, = is and no update of xi, occurs for j with
jr < j < ja. Referring updating
-j
l **2n
123***nnti FIG.
AND
case is quite
convergence
periodic
CHAZAN
to this graph it is easy to see that
is given by the following
set of relations
,~ (Y 7ljR -3 2
=
(b2, x kl+lk+l)
=;.$
XQ(l+l)fi+J
=
(b3,
=
,#+1)~+2)
by2X:,yfi
for Y = 2, the
(3.1).
‘-I+;
b13X1(4'-l)n+% +
k 3,’
b$,;P+l+ 2
la-2 x
n
(q+l)n+l+n
=
(b”,
.$+lh+*-1)
=
2
b;X;,(q+l)n+l+Y
:,= 1
(3.1) where bi is the ith row of B and q is any positive I.ineau
.4lgebra
and
Its
Applications
2(1969),
199-222
integer.
CHAOTIC
209
RELAXATION
We will explain how the first equation in (3.1) comes about. line segment
in the Fig. 3.1 represents
The segment
begins
The segment
terminates
memory
of that
relaxation,
the
memory
When
contents
which
this
processors,
the first component are then
left endpoint vertical
line meets
of segments
forming
with the exception
predecessor equation
group.
The
remaining
equations
Finally
a vertical
in (3.1)
in the right
the
In Fig. 3.1 this of horizontal
line
which belongs
to a
member
of the first
In a similar
two groups.
may
line through
are thus seen to belong
of the last component these
of the
case, Y = 2 The memory
line segment. contents
by
that processor
component.
a parallelogram
superscripts
in (3.1) fall into
are representative
by drawing horizontal
a
Those horizontal
receives for its computation
determinable
write into to effect
is determinable
hence in the present
line is drawn and the memory
segments
is assigned
available
the penultimate
of a penultimate
to one group
into memory.
prior to the next
a processor
in memory;
itself has just written
contents
value is written
it has
vertical
value currently
A horizontal
value of a component.
line at the value j of the assignment.
component which
computed
one unit of index
component.
drawing a vertical line segments
when that
a computed
be deduced
manner
the
from the figure.
we add the equation x (q+21n= x (q+l)n+n = x qn+lJrn n n n
Let
zq be the n-dimensional
vector
whose components
24 = Xk-l)n+l+i
I
a recurrence
Dzqtl = L 0zqfl
i = 1,. . .,a,
z
and let wq = x,,~“. Multiplying we obtain
the ith equation
relation
are
in (3.1) by aii, i = 1, . . . , n,
for (z”, wq) :
+ L,.zq+l + (L, +
L,*)zq + L,*zq -
ul,nzp,w (3.2)
r&q+i = 2,“. Here Lo, L,, L, are n x n matrices
whose elements
(Lp)i,i are:
i=l
i = n, otherwise
(L&i
=
i
I-
Linear
0,
i = 12,
0,
jai-1
aij,
otherwise
Algebra
and
Its
(3.3)
j=l
Applications
2(1969),
199-222
D.
210
CHAZAN
AND
W.
MIRANKER
and i=i--l otherwise. Our first case for which convergence theorem. 3.1.
THEOREM
definite,
is obtained
is given in the following
If n = 3, w = 1, r = 2, and A is symmetric
then the periodic
positive
scheme converges.
Prooj.
that
Without loss of generality we may select m,, m2, and ma such m, = (1, 0, 0), m2 = (wr, wz, 0), ma = (or, vs, va) and
and furthermore we may assume that wr2 + wa”- = ZJ~~+ va2 + va2 = I. Then ai, i = (m,, mj) and a,,, = (m,, mi) = 1. If we let Sq = (z”, wp), we obtain from (3.2) -and (3.3): 0 0
1 sq+’
Sqt’=
i
-VI
0
0 0 00
0 0
through
Reordering the equation ZL~O = Saq, we have:
11441 = ~ i
0
0
0
v1
0
0
0
0
0
0
0
0
sq
-
0 0
Algebra
and Ifs
Applications
0
0
0
-10
(3.4)
urq = Siq, u2q = Ssq, us4 = S;,
Let
Linear
(w,v)
2(1969),
199-222
and
CHAOTIC
211
RELAXATION
Then
,249+l
+(Z
If u is an eigenvector
$9+1+(;
Qo)
(3.5)
@=O,
of (3.59, we have
Au, + iiPu, + Qu2 = 0 Au, + Ru, = 0 where u is split into two two-dimensional
QRu, =
,12u1+ i12Pu, Convergence
Thus,
In order to show that
to show that the roots p of !p(I + A) -
/ 1 denotes
ur and us.
0.
of the scheme will follow if 111< 1.
Ilj < 1, it is enough
where
components
a determinant,
BCI = 0,
are smaller than
1 in magnitude.
YOU,
b *
4c
vb2 -
p=
2
~~
where b = (q”
+ (w, u>” + vi2 -
If b2 < 4c, then
lpi2 = c < 1.
v,q(w,
and
v))
Suppose
b2 > 4c.
c = zUrvr(w, v). We must show then
that !b + l/b2 -
4cj < 2.
But b = w12 + viz + (w, v)w2v2 = VI2 + WI2 + qv1w92 Then b is positive
+ (~z~2)2.
since WI2 + vi2 >, 2/VJ,l> Linear
Algebra
l(~lS)(~2~2)l.
and
Its
Af$lications
2(1969),
199-222
212
D.
CHAZAN
AND
W.
MIRANKER
Also b < 2, since
Then Vb2 -
2 -
<
2.
(1%
-
1~2/12
only to show that
the proof there remains
to conclude
the positivity
Using
&.
<
of (2 -
(2 -
b) >
b) we wish to have
4-4b+b2>b2-4c or l>b-cc. But
which
concludes
the proof.
Our second case in which we obtain
convergence
is given in Theorem
3.2. THEOREM
Let Y = 2 and suppose
3.2.
and A are positive
definite.
a,,, = 0 and D + L, + L2*
Then the periodic
scheme described by (3.2)
and (3.3) converges. Proof.
The proof was suggested
along lines proposed Linear
Algebra
and Its
by P. Stein Applications
by W. Kahn
[l].
[2] is also possible.
2(1969),
199-
222
A similar
proof
CHAOTIC
213
RELAXATION
a,,, = 0,
Since
L,, vanishes,
and
the
iteration
scheme
reduces
to
Dzqfl= L,z’+’ + (L, + L2*)zq + L,*zq. Let A be the eigenvalue corresponding
of largest modulus of this scheme and let z be the
eigenvector;
i.e.,
ADZ = IL,z + (L, + L,*)z + L,*z. Then
i =
(ZP(4 + L,*)z) + (ZJL,z)* ~. (2, Dz) - (2, L,z)
We abbreviate
this expression
as follows:
NOW y + CI> 0, since D + L, + L,* is positive definite. > 0,
since
17 = y -
A is positive
definite.
Let P = pi + i&
y -
tc -
p -
8 = tl + Pi,
p* and
/3i. Then
(3.6) Since
r -
jAj < 1. 4.
ONE
6 > 0 and q + 6 > 0, then This
concludes
MORE
SPECIAL
In this schemes The
section
obtained
angle between
It is easily
the proof
Iv/ > 181, and (3.6) implies
that
of the theorem.
CASE
we use the in Sect.
variational
2 to obtain
characterization
another
convergence
of periodic statement.
m’ and mi is given by
seen that
if
Linear
Algebra
and
Its
Applications
2(1969),
199-222
214
I>. CHAZAN
then D + L, + L2* is positive Theorem
3.2 applies.
definite.
In this section
AND W. MIRANKER
If, in addition,
a,,
we shall show that if
= 0, then
’
ai,i+l
(4.1)
Va;,iai+lZ< + ’ then
it is sufficient
to have al n
(4.2)
j/,,
to hold.
We formulate
this as Theorem
4.1.
If the inequalities
THEOREM 4.1.
(4.1) and (4.2) are satisfied then the
scheme comerges.
periodic
Proof.
We normalize
We rewrite
(2.8)
rn’ so that
(lrnil1= 1.
as Y
i+i = Yj _ .jmkn+l(i),
(4.1)
where (4.2) Combining Y
(4.1) and (4.2) gives it-1 = yj _
If we let zj = c&l,
(mk,+l(j),
yj + ,j-lmkn+i(i-l))mk,;~(,),
this relation
,j+r -_ (mkn+i(i),Yj) Using this expression Y
may
be written
+ Zj(mkn+l(f),
for u3 (= zj+‘),
as
mkn+l(i-l)).
(4.3)
(4.1) may be written
as
j+i = Yj _ (mkn+l(i))(mkn+l(l))Y~ -
.zi(m
kn+l(i), m~~ll+r)~m~nil(i)
(4.4)
Let w = rnkn+‘(‘) and let /? = (mkn+l(i’, mkn+l”-l)) = a++JG Then
the Eqs.
(4.3) and (4.4) may be written
Linear
Algebra and Its Applications
2(1969), 199-222
in matrix
form:
CHAOTIC
215
RELAXATION
y3
i
71
,j+l
)i
I
=
-
w)(w w,I‘
We shall now define a quadratic the map (4.5) contracting. Let
(4.5)
vector norm which, when [PI < $, makes
This norm will be seen to be independent
ZV)(W, so that
P = I -
y’ )(J f N(g.
-Pw p
P2 = P and PW = 0.
Then
of W.
N may be
written as
\,wTP h
N = The quadratic
norm which we are seeking
tensor
with an appropriate
is determined
scalar
a.
We
first
by the metric determine
a so
that
is negative ipi < fr.
The
semi-definite. Using
quadratic
form (a -
This
l), (1 -
are all positive. The positivity
is equivalent
to this last
y)” ~ 2P(w, y).z + (1 -
matrix
is
(a + l)P2b2.
(4.6)
for all (y, z) iff (a + 1)B2), -
(4P2 -
The last expression of this fi2 -
This
this under the hypothesis
of P, we have
corresponding
l)(~,
is nonnegative (a -
We will accomplish
the properties
1) -
I)(1 -
(a t
l)P2))
in (4.7) is the discriminant
discriminant ((a -
4(a -
may
(a2 -
be written
(4.7) of (4.6).
as:
l))p2) < 0.
to
(4.8) Linear
plgebra
and
Its
Applzcations
2(1969),
199-222
216
D. CHAZAN
The maximum
of (a -
AND
1)/a” over all a is t and is attained
/zl< $ and a = 2 satisfies
(4.8).
Moreover,
in (4.7) to be positive.
by a slightly
using the nonsingularity
that
the product
5. A GENERAL
of all the N -
s is also (strictly)
As can be seen from the above discussion, in general
statements
about
even
1, the matrix
In this section,
we shall show that not satisfied
there
(B, C, 9’)
as in Sect. that
these
scheme
azj-
of the full class of chaotic
For example,
procedures stronger
Furthermore, i.e., if they
the chaotic
are
relaxation
that
for the matrix
the assumptions
which
if B is defined in terms of A
E, then a sufficient
condition
A+ converge.
for convergence
is
A+ is the matrix
with ai,;,
lai,jj,
if the norm of the Jacobi than
1 a certain
on the value of the norm is allowed. In what follows the absolute the matrix
so that
it is surprising
1
is smaller
Y
to make
to satisfy
are also necessary,
a sequence
““=I__ Furthermore,
In order
diverges.
1, B = D -
with entries
not much can be said about schemes.
B must be required
conditions
exists
are so weak.
the Jacobi
to A+
of A, which shows contracting.
we shall state such conditions.
In view of the examples, will be required
for periodic
the convergence
in Sect.
conditions.
scheme
The proof is completed
THEOREM
convergence described
at a = 2. Thus,
these values of a and ,8 also
cause the first two expressions finer argument
W. MIRANKER
IlMl (or vector
zrfl, matrix
amount
I -
D-lA+
corresponding
of overrelaxation
depending
We shall now state our main result.
value of a matrix
1~1)whose components
M (or a vector
V) will be
are the absolute
value
of M (or v). A vector z, will be said to be positive if each of its components is positive. THEOREM.
(u)
Let (B, C, 9)
be a chaotic relaxatioti
scheme.
The sequence of iterates xi converges if there exists a positive vector
v and a number cc, LX< 1 such that lB]v
This huppercs if the spectral radius of 1B 1, p( IBI) < 1.
(c)
If no such v exists then there exists a sequence 9,
for which the iterates corresponding Linear
Algebra
and Its Applications
to (B, C, 9,)
2(1969). 199-222
depending on B
do not converge.
CHAOTIC
proof. three
217
RELAXATION
(a)
The
proof
of the
sufficiency
is divided
into
parts : The set {WI Iwi < a} (a rectangular
(i) into itself
by a factor
CI under
parallelopiped)
multiplication
The
sequence
of iterates
x3 is bounded.
(iii)
The
sequence
of iterates
tends
(ii)
Suppose
101 < U.
is contracted
by B.
(ii)
(i)
to zero.
Then
We shall now show that if 1XJl+r!, . . .) p+s1
for some constant
1x3( < av for all j 3 jr + S.
which occurs in the definition
This may be seen by induction jr + s. Then either9
or p = k, +i(j,J,
< av
a > 0 and a given ji then
(Here s is the parameter j,,>
condition
on j.
For suppose
# k,_,(j,),
in which
of a chaotic scheme.)
lxjl < av for all j < j,,,
in which case lx>+‘1
G 1x21 <
UV,
case 3 ‘07’ = (P, z),
where zI = x13oPk”‘0) and therefore then by the contracting
property
/zl(< uvl for each 1, 1 < 1 < 12. But of B (with v replaced
by av), we have
(bp, z) < uuvp
.. .
xfiJo+’ < avp for all 9. (iii)
notethatif
To conclude
completes
the proof of part condition
(ii).
(a), we may
ix31i-sl < av then bv what was said above lxjl< ,
/G+‘j,...,
for all j > ji.
This
the proof of the sufficiency
If xp is updated
at time j2, jz > ji, we have
uv
for all j >, jZ
ixp.l/ = (bp, z) for some z with components
Let ja be an instant j < ja with k,+,(j) j 2 ja. argument
In particular above,
so that = i.
z, < uv,.
Hence,
for each 1 < i < n there
From
what
was said above,
exists
~jl < auv for j3 + 1 < j < j3 + s.
and using Linear
the
fact
Algebra
that and
each Its
Repeating
component
Applications
j, j0 + s <
lxj\ < ccuv for all the
is updated
2(1969),
199-222
218
D. CHAZAN
infinitely there
often, we may observe that if
exists
ii+i with
the proof
We shall first
show that
where for some positive any E > 0 there
exists
let v = (vi, yva).
By choosing
C&Jfor all i 3 i, for some i,
Ix’~+‘/ < C?“V for all j 3 iii-,
lxil --f 0 and completes (b)
jxi 1<
AND W. MIRANKER
This implies
if a matrix
F has the form
vi and zj2 F1ivl < uivi, F,,v,
< cqs.
v with Fv < (K + E)U, cc = max(cl,,
Then
exists
y sufficiently the proof
v > 0 with
CQ). Indeed,
small it is easy to see how uivi + yFlzvz can
of (b) we wish to show that
lBjv<
CIV, CI< 1.
This
follows
by induction
in the normal form of IBI, and the auxilliary
just established
Indeed, this is true if IBI is irreducible
above.
it is shown for any \BI w h ose normal
Now suppose that
jBl has n + 1 components.
iB1 where F,, has n components that F,,v,
(c) Linear
1.
We write IL? in the form
by the induction
jBjv <
than
Hence,
UV,
From hypothesis
completes
F,,v,
< uzvz
u
Hence, the spectral
the proof
this it follows
for some y > 0, v = (vi, ~vs),
if IBlv < uv, v > 0, cc< 1, B is a contraction
This
result
([4], p. 30).
form has n components
and F,, is irreducible.
when cc, = p(lBI) < 1 and us < 1.
Conversely,
on the
=
< cclvI. Furthermore,
defined by /Ix/l = max( lx&J.
< YLXQ.
if p(lBI) < 1 there
number of components Suppose
for
Then
be made smaller then (tci + F)ZI~,while it is always true that ycqs To complete
that
of sufficiency.
in the norm
radius of ]Bi is smaller
of (b).
Suppose there exists no v with 1B/v < v. Then by (b) @(I B 1) > 1. Algebra
and
Its Applications
3(1969),
199-222
CHAOTIC
219
RELAXATIOK
Let v be the eigenvector eigenvalue.
jB/ corresponding
of
to the largest
Then v may be chosen to be real ([4], p. 76) and nonnegative.
Using this fact we shall now show how to construct that the sequence diverges
real
of iterates
for some initial
corresponding
a sequence
to the scheme
9,
so
(B, C, Ypo)
state z.
Let z be such that Bz= If B, C is consistent now constructed
or
z+ v
(B -
with an invertible
I)z = v.
matrix
A, then z exists.
9,
is
sequence
of
as follows:
For 1 < j < 9s we let k,(j) = j If the iteration iterates
2,
i=
l,...,n,
k,+,(j)
= i -
is started at an initial state z, the following
1.
is produced :
21 + 2’1
21 + Z’1 21 + 2/l z2
22
+
22
v2
z,
...
+
02
:
where the fact that (b+ z) = z, + vi was used repetitively. may be written
as the sum of two sequences
uil,
This sequence
and cr21,
Now let K,+,(n + 1) = 1, and select k,(n + l), i = 1, . . ., n, so that sgn(x~~‘l-k’(nfl))
= sgn bil.
Lznear
It follows
Algebra
that
and
Its
A$@ications
2(1969),
199-222
220
D.
where p is the spectral
radius
of IBI.
CHAZAN
We may
AND
continue
letting k,,+i 1 + ) I j mod n and again choosing that sgn(,i’i) ’ )=sgnbii for n+l
W.
MIRANKER
this process,
1 < KJj)
sgn bii, for 2n + 1 < j < 3n + 1. The result of this iteration
up to time
Where
i-kL(i’) = procedure
3n + 1 is:
ui2 and oi3 are the sequence
of updatings
= n + 1 so
andsgn(x,
identical
as2 and os3 have
of iterates
to the one described
generated
by a sequence
above but applied to oil, and
the form: +
P”! 2
PVl 2
FL! 2
V,
%
P"fi
2
2
2
and -
P”1 2
-
P"1 2
PU3
2
Note now that
the last column
which is identical Linear
Algebra
of crz2 together
with 021 but with
and Its Applications
-
2(1969).
with os3 form an array
pvi replacing 199-222
vi.
If we now let
CHAOTIC
221
RELAXATION
ki(j + 2n) = k,(j), i = 1, . . ., n + 1, j > n + 1 it is easy to see that the resulting p > 1.
oai cannot
sequence Suppose
a chaotic
now that
relaxation
converge
the sequence
sequence
a sequence
with initial
starting
proof
state
converges.
with the initial
on the other hand ali diverges to produce
if p >, 1 and at
state
must
diverge
if
Then oil + (~a*is
z which diverges.
If
it is easy to see how Y may be modified
z + v/2 which
diverges.
This
concludes
the
of (c) and the theorem.
It may be noted that of freedom sufficient
in proving
was used in selecting to guarantee
convergence
then the full class described chaotic
schemes
certainly
yielding
part (c) of the theorem
Y,,.
a great deal
Clearly weaker conditions of a smaller
by Definition
successively
1.
could be
class of chaotic
schemes
Any finer classification
stronger
convergence
of
results would
be of some interest. A be written in the form ‘4 = D -
E
when D is a diagonal matrix with elements d,,$ = a,,l. and ei,j = ai,i, i f
j.
Let the matrix
COROLLARY.
If
B” = I ~ wD-‘A,
the scheme (B”‘, C”, 9’)
C = wD-I,
converges fo7
1 and w = 1 if p( IBI) = u < 1.
all Y satisfying on assumptions of Definition
If 0 < co < 2/(1 + M) the scheme must converge as well. Proof.
We must
exists
v > 0 so that
there
exists
show that
u > 0 so that
lB”lv < (I(1 Let /l = (11 -
p(lB”l) < 1 or alternatively
lB”lv < ,4?v,p < 1.
031 + WE).
But
lBl(v < uv.
w) + wlBll)v < I(1 It remains
that
By (b) of the theorem
there
we know
then
cu)lv + WCS = (11 to show that
/3< 1.
WI + ocr)v. If 1 < w <
AlsoifO
we may note that it follows from the above theorem
one of the
following
(1)
A symmetric
and strictly
(2)
A irreducibly
diagonally Linear
conditions, diagonally
dominant
Algebra
and
convergence dominant
that
still holds.
( [4], p. 23).
( [4], p. 23). Its
Applications
2(1969), 199-222
222
D. CHAZAN
A symmetric
(3)
positive
definite
with
AND
W. MIRANKER
nonpositive
offdiagonal
entries. We may also note that greater
than
by simply theorem
if in case (3) the spectral
one, the iteration
turning
process
is unstable.
the proof of the theorem
around.
is as strong as we may wish it to be.
of L are not all negative,
radius
of D-l_L is
This can be shown In that
If, however,
case our
the elements
it is not clear what happens.
REFERENCES 1 W. Kahan,
private
communication.
2 P. Stein, Some general theorems on iterants, J. Res. Natl. BUY. Standards 3 J. Rosenfeld, Report
A case study
RC-1864.
1967. 4 R. Varga,
on programming
IBM Watson
Research
Matrix Iterative AnaZysis,
1962. 5 S. Schechter,
Relaxation
methods
12(1959), 313-335. 6 4. Ostrowski, Determinanten
for parallel processors,
Center, Yorktown
Prentice-Hall,
Heights,
Englewood
for linear equations,
48(1952). Research New York,
Cliffs, New Jersey,
Comm. Pure
and Appl.
Math.
Konvergenz
van
linearen
mit iiberwiegender Iterationsprozessen,
Hauptdiagonale Comm.
175-210. Received
Limar
June
Algebra
4,
and
1968
Its
Applicatiom
2(1969),
199- 222
Math.
und die absolute Helu.
30(1955),
ALGEBRA
Chaotic
AND
ITS
199
APPLICATIOSS
Relaxation*
D. CHAZAN
AND
IBM
Research Center,
Watson
Communicated
W. MIRANKER Yorktown
Heights,
New
York
by Alan J. Hoffman
ABSTRACT In this paper we consider relaxation methods for solving linear systems of equations.
These methods are suited for execution on a parallel system of processors.
They have the feature of allowing a minimal amount of communication of computational status between the computers, so that the relaxation on a chaotic appearance, reduces programming nature.
We give a precise characterization
of divergence,
and conditions guaranteeing
process, while taking
and processor time of a bookkeeping of chaotic relaxation,
some examples
convergence.
INTRODUCTION
In this paper we discuss for solving capable case,
linear
systems
of execution
there
a preliminary
of equations.
on a set of parallel
is a minimum
between the computers,
study These
of relaxation methods
processors.
of communication
so that the relaxation
methods
are of a form
In the most general
of computational process takes
status
on a chaotic
appearance. Let
the linear
system
be given
by
Ax=& where x is an n vector. (1) by relaxation usually
If A is symmetric
may be described
in coordinate
directions,
(0.1) and positive
as successive
definite,
univariate
of the quadratic
solving
minimization,
form
Q(x)= :(x, Ax) - (6 4. * Dedicated
to Professor
A. XI. Ostrowski on his 75th birthday.
Linear Copyright
0
(0.2)
1969
Algebra and Its Applications by American
Elsevier
2(1969), 199-222
Publishing
Company,
Inc.
200
D. CHAZAN
For
the
Gauss-Seidel
method
prior to commencing available effect
a univariate
any other.
determining
Each
However,
what displacement
Thus,
the displacement
point
x may be made at some other
tion
is a familiar
displacements
displacements
J.
decreased
[3].
the
point
x.
at the
apparently
arbitrary minimiza-
x, but
only one of the ?z
relaxation
quality
get out of
or the specific
difficulty.
was suggested
extensive
to the authors
numerical
found that
experiments
The
programming
on the relative
chaotic
form
and computer
cessors and the algorithm.
number
of the time
His experiments
of overrelaxation
with
The precise
of processors,
relaxation
r,
eliminated
in coordinating
also exhibited
allowable
by
the use of more processors
time for solving the linear system.
depended
dimension.
will rapidly
may be of different
Rosenfeld
the computation
in the amount
is to be effected,
by a processor
This
at a point
who has conducted
considerable
is
scheme n univariate
may be of varying
form of the decrease to n, the
point.
to
a processor
displace
case, the r processors
of chaotic
relaxation
that
are
assigned
at x.
performed
Rosenfeld
chaotic
and
determined
For the Jacobi
chaotic
The problem
tasks
to a minimum
since they themselves
relaxations
their
are computed
occurs
In the general phase,
complete
one.
the time
at a point x in n-space
processors
is completed r processors
may be sequentially
during
other
pattern
may
minimization
Now we suppose that
for this computation.
a minimization.
AND W. MIRANKER
the pro-
a diminution
for convergence
in a chaotic
mode. Chaotic
relaxation
by A. Ostrowski because
is related
from a number
his free steering
of predecessor
results
Schechter
for so-called
monotone
considered
block relaxation, convergence
statements
relaxation
is specialized
Sect.
1 chaotic
notion.
In addition,
subclass
of chaotic
Algebra
and
relaxation
A.
Its
[6], Ostrowski the matrix
obtained A be a so-
and obtain-
For simplicity,
for chaotic
block
to ordinary
we have not
is defined
along
2(1969),
with
and introduce
called periodic schemes.
Applications
relaxation.
free steering,
When
our results
of Ostrowski.
we give several examples schemes
In that
although the methods we use may be extended
some of the statements
In
states.
values
by compo-
[5] considered free block relaxation
to obtain recover
more general
free but the antecedent
under the condition
ed convergence
the chaotic
which has been studied
It is however
is based are chosen freely component
called H matrix.
Linear
[5].
not only is the order of relaxation
upon which a relaxation nent
to free steering
[6] and S. Schechter
199-222
a consistency an important
In Section
2 we give
CHAOTIC RELAXATION a variational
201
characterization
to chaotic
the periodic case which diverge. to a difference equation
equation
variational
convergence
characterization
to the fully chaotic and
converge.
A number
of matrices
and two examples
state
schemes space.
two convergence
result in the periodic
conditions
of corollaries
for which
chaotic
The
difference In Sect.
case by exploiting
derived in Sect. 2. Finally
sufficient
for a chaotic
4 the
in Sect. 5, we turn This theorem
iteration
gives
scheme
are then given which produce
relaxation
in
are reduced
theorems.
case and give our main theorem.
necessary
1.
in an augmented
is then used to obtain
we give another
schemes
In Sect. 3 periodic
to
classes
converges.
DEFINITION OF TERJIS In this section, we define a chaotic iteration
the notion
of consistency
then give several
examples
is a class of schemes, subsequent
scheme.
scheme
of chaotic
schemes.
periodic
schemes,
called
We then introduce
with a linear system. One of these which
We
examples
we will study
in
sections.
DEFINITION. vectors,
of a chaotic
xi, i = 1,
X chaotic iterative scheme is a class of sequences of n. . Each sequence in this class is defined recursively by
x,ii-l =
1) (1.1) E&l
b&eJj)
+
i =
(C",d),
kn+l(i).
i
The initial
n-vector
(2.1) depends
x1 and the n-vector
on a triple
(B, C, P’).
The ith rows of B and C are denoted by ci.
Y
is a sequence
For
some fixed
(i) (ii) Moreover,
integer
properties
1 < k,&,(j)
by b” = (bil,. . . , b,“) and
Y = {K,(j),.
., k,+,(j)},
i =
:
i = 1,.
< fi,
k, tl (j) = i infinitely
with the implication
The mapping
s> 0
0 < k,(j) < s,
We will identify
respectively
of 12 + l-vectors;
1,. . .) with the following
d are arbitrary.
Here B and C are a x n matrices.
a chaotic
j=l,... often
,i=l,...
.,?Z,
.
.
for each i, s < i < n.
relaxation
that once B, C, and 9
scheme
by the triple
are fixed a particular
(B, C, Y), sequence
of the class is chosen. Linear Algebra and Its Applicatiom
2(1969), 199- 222
D. CHAZAN AND W. MIRANKER
202 This time
definition
may
i, the k,+,(j)th
n -
1 components
of kkl(j),
updated infinitely Since system
as follows:
At each
of ~j is updated
are unchanged.
the second
was produced
be interpreted
component
The updating
component
instant
uses the first component
of ~j-~~(j), etc.
Every
component
often, and no update uses a value of a component
by an update
s or more steps
we will use chaotic
iteration
Ax = d, we introduce
DEFINITION.
A chaotic
iteration
system Ax = d, with solution
is
which
previously.
procedures
a consistency
of
while the remaining
to solve
notion
procedure
the
linear
in the following
is consistent
with a linear
z, if z is a fixed point of the mapping
(l.l),
i.e., z = Bz + Cd. Notice
that
From
(1.2) is independent
of Y.
now on we will assume
with the linear system the y3 are given
that
the chaotic
scheme
is consistent
Ax = d. We note then that if yj = xj -
by the following
recursive
=
c b~yaJ-k,(l’,
z, then
scheme i #
(YiJ>
Y, If1
(1.2)
h+l(j)
(1.3)
i = h‘,~_l(j).
n
(1.3)
is a chaotic
system may
iteration Since
Ax = 0.
set d = 0 without
Obtaining
Consistent
A consistent
scheme
Then
consistent
scheme
may be obtained
scheme.
the choosing
(9,
matrix
schemes,
B = D-IE
with parameter
B”, Cw). Here B” = I -
wD-lA
w.
by choosing
whose iith
A larger class of consistent
and under relaxation
homogeneous we
loss of generality.
as follows. Let D be the diagonal A.
to the consistent
Schemes.
chaotic
E = D -
corresponding
we are only considering
and
B and C
element is aii and let C = D-l
produces
schemes correspond
We denote
these
to over
schemes
and C” = wD. Note that
a by
B1 = B
and Cl = C. We will now give several
examples
= 0 and k,+,(j)
= (i -
each i, i = 1,. . ., n exactly precisely Linear
the
Algebra
Gauss-Seidel
schemes.
Let Y be defined by k,(j) = k,(j) = * . . =
Example 1 (Gauss-Seidel). k,(j)
of chaotic
l)(mod
n).
Here s = 1 and k,+,(j)
once in every relaxation
and Its Applications
n updates.
procedure.
2(1969),
199-222
This
= i, for scheme
is
CHAOTIC RELAXATION 2 (Jacobi).
Example and
k,+,(j)
203
k,+,(j)
-
Let (j -
Y
be
defined
l)(mod
n).
Here
for each i, i = 1,. . ., n exactly precisely
the
Jacobi
Here a number
as it becomes
available
time j a processor other
Schemes).
is assigned
x, are either
P is still calculating available
to relax
the
no lapping
Each
processor At
x, say. We imagine
that each of the compo-
or assigned x,.
to be updated Ultimately,
while
a newly
all assignments
until the component
For this procedure
P.
at xi
we see that
are relaxed
infinitely
subclass
of schemes
s =
often.
Schemes
periodic
schemes,
an important
These
do not correspond
schemes
corresponds
Periodic
processors repeating
compute
are suitable
to any of the classical are an orderly
simultaneously
periodic pattern is given
schemes
to the case where k,+,(j)
schemes
DEFINITION. scheme
is
to relax components.
the component
all components
We now introduce
scheme
introduce
this calculation
by processor
1 and again
Periodic
j.
= i,
processor would be assigned to update xi. The no lapping conven-
has been updated
but
K,+,(j)
This scheme
are at work.
sequentially
updated
tion rules this out and stops 2n -
We
of processors
P is assigned to relax the component
than
= . * * = k,(j) = and
procedure.
that it takes so long to perform nents
k,(j)
s = n
once in every rt updates.
relaxation
3 (No Lapping
Example convention.
by
Corresponding scheme
k,(j)
methods.
This
is a periodic
=
case.
fashion
Formally,
of The
that
a
a periodic
definition.
to each integer*
7,
1 <
k,(j) = . . . = k,(j),
7 <
n, a
periodic
and
j” - ’
(I 1 ” 1
class of
function
of the chaotic
but in such a regular
with
we call
computation
exists in the computation.
by the following
is a chaotic
version
which
for parallel
1,
j
= (i -
l)(modn).
and with k,+,(j) A periodic vertices
scheme
may be described
are the integers,
* r is the number
by a dependency
graph
whose
j = 1, 2, . . . .
of processors Linear
assigned to the relaxation Algebra
and Its Applications
calculation.
2(1969), 199-222
Il.
204 The axis is indexed being written
by j.
At the instant
into memory.
of a particular
processor.
CHAZAN
Each
AND
W.
j, the vector
MIRANKER
xi is viewed as
curve in the graph traces
For ii < ja, j2 can be reached
the activity
from ji along a
trajectory in the graph, if and only if the updated component of xJ2 (i.e., i. ci _-1j) depends on the jist iterate formally (in the sense that the %i.l 2 right-hand side of 1.1 contains a term with superscript ii). (Alternatively, in terms
of the parallel
xJ2 are successive
processors,
updates
this happens
if and only if xi1 and
of the same processor.)
The graph
for Y = 3
and n = 6 is shown in Fig. 1.1 where the label from ji to jZ is k,+, (i2 -
1).
.
4 FIG.
1.1.
Each
Graph
6
5
of periodic
case
wlth
time a vector is written,
in the computing dated
system,
components
processors
j is increased
the actual
are written
zero to some positive
three
into
number
and
by unity,
dimension
although
interval
of time between
memory
will vary
S.
in reality which up-
from essentially
in a more or less unpredictable
manner.
Conventions Unless
otherwise
symmetric
specified,
the remainder The
and A = MM*.
columns
of this paper deals with A of M are m’, i = 1, . . . , n.
D is the diagonal matrix (a,jd,i), E = D - .4, R = D-IE, B” = I - wD-IA, and C” = wD-l. In Sect. 5, A will not be
In addition,
C = D-l, required
2.
TWO
to be symmetric.
DIVERGENT
EXAMPLES
In this section we will derive a representation a sequence
of minimizations
representation acterize
to explore
the difficulties
A variational
of a quadratic two divergent
in studying
characterization
theorem.
Linear
and Its Applications
2(1969),
relaxation
as
Then we will use this which
help to char-
relaxation.
of chaotic
in the following
Algebra
examples
chaotic
case is given
of chaotic
form.
199-222
relaxation
in the periodic
CHAOTIC
THEOREM
2.1.
Let 9
scheme parametrized M,
205
TIET,AXATIOS
be a sequence
b?l Y and w.
oj iterates
Produced
by a periodic
I,t ml is the ith column of
I,et _vi = MY.
then \’i'l = J”
where
(2.1)
C.C~ minimizes j-r
:-I
-
i13’
Here
kl, -I’ll
cdm
-
!/ 1~denotes
Proof.
the E,uclidean
cd?%
%4,, (2.2)
norm.
\I’e will first set o = 1.
In this case we have
(2.3)
Let v3 be the vector whose ith component
is a,,. Then (2.3) may be written
as
_
,j
&% - l(T)
_
(2.4)
Now
xi
,ILl(‘i
4;,7:(:,
=
(2.5)
since k,,,(i) # k,+,(j), i - n < i < j (i.e., no component is updated twice before each of the n - 1 remaining ones are updated once). Using (2.5) in (2.4)
we may
write
crj as
When w # 1 it is easy to verify that cc3is replaced by u&. It is also easy k 1 (i) A,+ r(i) to verify that C$ZJ” ’ IS the displacement along v which minimizes (p+l
In terms
of yi
(2.4)
_
,i,%iIyq"p+l
and (2.7)
Lmear
_
become
Algebra
and
&)kn+q),
(2.7)
respectively
Its
Applications
2(1969), 199-222
I). CHAZAN
206 ljyj-rt
This
concludes
I _
W. MIRANKER
g,kn.il(j)112,
(2.9)
the proof of the theorem.
We note here that u3 can thus be completely of Yi-‘-rl. We now consider Exawajde 1.
AND
two divergent
determined
from knowledge
examples.
Let
and
M=
mi = (l,O, O)T,
i=
1,2,3.
Let w=1+s.
Y = 2, Then Y J+i = yJ Since yi = Md, is described
$~+,(/I
wu’m
.
y3 has the form yj = (9, 0, 0). Thus, the iteration
equivalently
by the following
iteration
scheme
scheme for the scalar
zj,
,i+1 r ,j - c()c(?. Using
(2.2),
this can be written
as
,i-i-1 = zi The
eigenvalues
of this
shown to be no smaller Fig.
2.1 where
the
(2.10) is graphically -
second-order than
evolution
of a solution
illustrated.
Algebra
and Its Afiplications
(2.10)
iterative
1 if E > 0.
1 and blows up exponentially.
Linear
e&i, scheme
may
It is instructive of the
difference
be easily to consider equation
The solution has initial values z1 = z2 = zi is located 2(1969),
199-222
at the point
labeled
j.
CHAO’l’lC
FIG.
2.1.
Evolution
axis and iterate
Example writing
207
KEIAXATION
of solution of Eq. (3.1).
Let A and M be as above.
2.
everything
in terms
This iterative
scheme
(2.11)
=
FIG. 2.2. Evolution axis and iterate
Let w = 1 and Y = 3. Again
zi-l,
(2.11) greater
in Fig.
4 II 0
2.2.
5 I 8 I
of solution of Eq. (3.3).
than one. The way the
6,7 II 2
Iteration
number 1s labeled above the
values below.
examples
with the matrix
show
that
underrelaxation
A given here.
why both of these processes examples
_
also has eigenvalues
1,2,3,9 t I -I
I I -3
or relaxation
y
blows up is illustrated
IO
These
number is labeled above the
of zi we obtain z z+l
process
Iteration
values below.
(W < 1) must
The examples
are unstable.
step as a motion
Since we view each iteration
in the direction
given here the directions
be used
also point out the reasons
of motion
mi, we can say that in the were so close together
that
tasks were duplicated. It seems as though the event when ml = m2 = . . . = m” is, in a sense, the worst case for such schemes. A%conjecture is therefore above,
that
they converge
to guarantee these
if OJ is such that
The
apparently
are arbitrarily A
In particular,
for the specific
A
(1, < 1 should be sufficient
for the two-processor
(Y = 2) case.
worst
case
None of
(ml = m2 = . . . = m”) corresponds
It seems clear that there exist nonsingular
matrices
close to the worst case form and which exhibit
DIFFERENCE
EQUATION
In this section as a first-order
converge
has been settled.
singular matrix.
3.
for all A.
convergence
questions
the schemes
AND
CONVERGENCE
IN
we will derive a representation
difference
equation
Lirwar
Algebra
in Euclidean and
Its
THE
to a which
divergence.
PERIODIC
CASE
for the periodic scheme, n +
Applications
7 -
l-space.
2(1969),
We
199- 222
208
D.
illustrate
this for I = 2.
present
The general
two cases for which
Note
that
,&,+,(I -
... =
for the
1) # k,+,(j)
scheme
W.
MIRANKER
similar.
We then
is demonstrated. k,+,(j)
for i = 1,. . ., n -
= (j -
1. Thus,
1) mod n so that
Xin+l(j) = ~~~~~~~~ =
1-(n-11 this fact can be represented by the graph 'k n+l(i) . Schematically, (for n = 7) : where the vertices of the graph are labeled (i, j).
below
I
I I
3.1.
Graph
of the
(ir, jr) is connected
periodic
case
for
Y =
4 and
**
l
n =
7
to (is, ja) if i, = is and no update of xi, occurs for j with
jr < j < ja. Referring updating
-j
l **2n
123***nnti FIG.
AND
case is quite
convergence
periodic
CHAZAN
to this graph it is easy to see that
is given by the following
set of relations
,~ (Y 7ljR -3 2
=
(b2, x kl+lk+l)
=;.$
XQ(l+l)fi+J
=
(b3,
=
,#+1)~+2)
by2X:,yfi
for Y = 2, the
(3.1).
‘-I+;
b13X1(4'-l)n+% +
k 3,’
b$,;P+l+ 2
la-2 x
n
(q+l)n+l+n
=
(b”,
.$+lh+*-1)
=
2
b;X;,(q+l)n+l+Y
:,= 1
(3.1) where bi is the ith row of B and q is any positive I.ineau
.4lgebra
and
Its
Applications
2(1969),
199-222
integer.
CHAOTIC
209
RELAXATION
We will explain how the first equation in (3.1) comes about. line segment
in the Fig. 3.1 represents
The segment
begins
The segment
terminates
memory
of that
relaxation,
the
memory
When
contents
which
this
processors,
the first component are then
left endpoint vertical
line meets
of segments
forming
with the exception
predecessor equation
group.
The
remaining
equations
Finally
a vertical
in (3.1)
in the right
the
In Fig. 3.1 this of horizontal
line
which belongs
to a
member
of the first
In a similar
two groups.
may
line through
are thus seen to belong
of the last component these
of the
case, Y = 2 The memory
line segment. contents
by
that processor
component.
a parallelogram
superscripts
in (3.1) fall into
are representative
by drawing horizontal
a
Those horizontal
receives for its computation
determinable
write into to effect
is determinable
hence in the present
line is drawn and the memory
segments
is assigned
available
the penultimate
of a penultimate
to one group
into memory.
prior to the next
a processor
in memory;
itself has just written
contents
value is written
it has
vertical
value currently
A horizontal
value of a component.
line at the value j of the assignment.
component which
computed
one unit of index
component.
drawing a vertical line segments
when that
a computed
be deduced
manner
the
from the figure.
we add the equation x (q+21n= x (q+l)n+n = x qn+lJrn n n n
Let
zq be the n-dimensional
vector
whose components
24 = Xk-l)n+l+i
I
a recurrence
Dzqtl = L 0zqfl
i = 1,. . .,a,
z
and let wq = x,,~“. Multiplying we obtain
the ith equation
relation
are
in (3.1) by aii, i = 1, . . . , n,
for (z”, wq) :
+ L,.zq+l + (L, +
L,*)zq + L,*zq -
ul,nzp,w (3.2)
r&q+i = 2,“. Here Lo, L,, L, are n x n matrices
whose elements
(Lp)i,i are:
i=l
i = n, otherwise
(L&i
=
i
I-
Linear
0,
i = 12,
0,
jai-1
aij,
otherwise
Algebra
and
Its
(3.3)
j=l
Applications
2(1969),
199-222
D.
210
CHAZAN
AND
W.
MIRANKER
and i=i--l otherwise. Our first case for which convergence theorem. 3.1.
THEOREM
definite,
is obtained
is given in the following
If n = 3, w = 1, r = 2, and A is symmetric
then the periodic
positive
scheme converges.
Prooj.
that
Without loss of generality we may select m,, m2, and ma such m, = (1, 0, 0), m2 = (wr, wz, 0), ma = (or, vs, va) and
and furthermore we may assume that wr2 + wa”- = ZJ~~+ va2 + va2 = I. Then ai, i = (m,, mj) and a,,, = (m,, mi) = 1. If we let Sq = (z”, wp), we obtain from (3.2) -and (3.3): 0 0
1 sq+’
Sqt’=
i
-VI
0
0 0 00
0 0
through
Reordering the equation ZL~O = Saq, we have:
11441 = ~ i
0
0
0
v1
0
0
0
0
0
0
0
0
sq
-
0 0
Algebra
and Ifs
Applications
0
0
0
-10
(3.4)
urq = Siq, u2q = Ssq, us4 = S;,
Let
Linear
(w,v)
2(1969),
199-222
and
CHAOTIC
211
RELAXATION
Then
,249+l
+(Z
If u is an eigenvector
$9+1+(;
Qo)
(3.5)
@=O,
of (3.59, we have
Au, + iiPu, + Qu2 = 0 Au, + Ru, = 0 where u is split into two two-dimensional
QRu, =
,12u1+ i12Pu, Convergence
Thus,
In order to show that
to show that the roots p of !p(I + A) -
/ 1 denotes
ur and us.
0.
of the scheme will follow if 111< 1.
Ilj < 1, it is enough
where
components
a determinant,
BCI = 0,
are smaller than
1 in magnitude.
YOU,
b *
4c
vb2 -
p=
2
~~
where b = (q”
+ (w, u>” + vi2 -
If b2 < 4c, then
lpi2 = c < 1.
v,q(w,
and
v))
Suppose
b2 > 4c.
c = zUrvr(w, v). We must show then
that !b + l/b2 -
4cj < 2.
But b = w12 + viz + (w, v)w2v2 = VI2 + WI2 + qv1w92 Then b is positive
+ (~z~2)2.
since WI2 + vi2 >, 2/VJ,l> Linear
Algebra
l(~lS)(~2~2)l.
and
Its
Af$lications
2(1969),
199-222
212
D.
CHAZAN
AND
W.
MIRANKER
Also b < 2, since
Then Vb2 -
2 -
<
2.
(1%
-
1~2/12
only to show that
the proof there remains
to conclude
the positivity
Using
&.
<
of (2 -
(2 -
b) >
b) we wish to have
4-4b+b2>b2-4c or l>b-cc. But
which
concludes
the proof.
Our second case in which we obtain
convergence
is given in Theorem
3.2. THEOREM
Let Y = 2 and suppose
3.2.
and A are positive
definite.
a,,, = 0 and D + L, + L2*
Then the periodic
scheme described by (3.2)
and (3.3) converges. Proof.
The proof was suggested
along lines proposed Linear
Algebra
and Its
by P. Stein Applications
by W. Kahn
[l].
[2] is also possible.
2(1969),
199-
222
A similar
proof
CHAOTIC
213
RELAXATION
a,,, = 0,
Since
L,, vanishes,
and
the
iteration
scheme
reduces
to
Dzqfl= L,z’+’ + (L, + L2*)zq + L,*zq. Let A be the eigenvalue corresponding
of largest modulus of this scheme and let z be the
eigenvector;
i.e.,
ADZ = IL,z + (L, + L,*)z + L,*z. Then
i =
(ZP(4 + L,*)z) + (ZJL,z)* ~. (2, Dz) - (2, L,z)
We abbreviate
this expression
as follows:
NOW y + CI> 0, since D + L, + L,* is positive definite. > 0,
since
17 = y -
A is positive
definite.
Let P = pi + i&
y -
tc -
p -
8 = tl + Pi,
p* and
/3i. Then
(3.6) Since
r -
jAj < 1. 4.
ONE
6 > 0 and q + 6 > 0, then This
concludes
MORE
SPECIAL
In this schemes The
section
obtained
angle between
It is easily
the proof
Iv/ > 181, and (3.6) implies
that
of the theorem.
CASE
we use the in Sect.
variational
2 to obtain
characterization
another
convergence
of periodic statement.
m’ and mi is given by
seen that
if
Linear
Algebra
and
Its
Applications
2(1969),
199-222
214
I>. CHAZAN
then D + L, + L2* is positive Theorem
3.2 applies.
definite.
In this section
AND W. MIRANKER
If, in addition,
a,,
we shall show that if
= 0, then
’
ai,i+l
(4.1)
Va;,iai+lZ< + ’ then
it is sufficient
to have al n
(4.2)
j/,,
to hold.
We formulate
this as Theorem
4.1.
If the inequalities
THEOREM 4.1.
(4.1) and (4.2) are satisfied then the
scheme comerges.
periodic
Proof.
We normalize
We rewrite
(2.8)
rn’ so that
(lrnil1= 1.
as Y
i+i = Yj _ .jmkn+l(i),
(4.1)
where (4.2) Combining Y
(4.1) and (4.2) gives it-1 = yj _
If we let zj = c&l,
(mk,+l(j),
yj + ,j-lmkn+i(i-l))mk,;~(,),
this relation
,j+r -_ (mkn+i(i),Yj) Using this expression Y
may
be written
+ Zj(mkn+l(f),
for u3 (= zj+‘),
as
mkn+l(i-l)).
(4.3)
(4.1) may be written
as
j+i = Yj _ (mkn+l(i))(mkn+l(l))Y~ -
.zi(m
kn+l(i), m~~ll+r)~m~nil(i)
(4.4)
Let w = rnkn+‘(‘) and let /? = (mkn+l(i’, mkn+l”-l)) = a++JG Then
the Eqs.
(4.3) and (4.4) may be written
Linear
Algebra and Its Applications
2(1969), 199-222
in matrix
form:
CHAOTIC
215
RELAXATION
y3
i
71
,j+l
)i
I
=
-
w)(w w,I‘
We shall now define a quadratic the map (4.5) contracting. Let
(4.5)
vector norm which, when [PI < $, makes
This norm will be seen to be independent
ZV)(W, so that
P = I -
y’ )(J f N(g.
-Pw p
P2 = P and PW = 0.
Then
of W.
N may be
written as
\,wTP h
N = The quadratic
norm which we are seeking
tensor
with an appropriate
is determined
scalar
a.
We
first
by the metric determine
a so
that
is negative ipi < fr.
The
semi-definite. Using
quadratic
form (a -
This
l), (1 -
are all positive. The positivity
is equivalent
to this last
y)” ~ 2P(w, y).z + (1 -
matrix
is
(a + l)P2b2.
(4.6)
for all (y, z) iff (a + 1)B2), -
(4P2 -
The last expression of this fi2 -
This
this under the hypothesis
of P, we have
corresponding
l)(~,
is nonnegative (a -
We will accomplish
the properties
1) -
I)(1 -
(a t
l)P2))
in (4.7) is the discriminant
discriminant ((a -
4(a -
may
(a2 -
be written
(4.7) of (4.6).
as:
l))p2) < 0.
to
(4.8) Linear
plgebra
and
Its
Applzcations
2(1969),
199-222
216
D. CHAZAN
The maximum
of (a -
AND
1)/a” over all a is t and is attained
/zl< $ and a = 2 satisfies
(4.8).
Moreover,
in (4.7) to be positive.
by a slightly
using the nonsingularity
that
the product
5. A GENERAL
of all the N -
s is also (strictly)
As can be seen from the above discussion, in general
statements
about
even
1, the matrix
In this section,
we shall show that not satisfied
there
(B, C, 9’)
as in Sect. that
these
scheme
azj-
of the full class of chaotic
For example,
procedures stronger
Furthermore, i.e., if they
the chaotic
are
relaxation
that
for the matrix
the assumptions
which
if B is defined in terms of A
E, then a sufficient
condition
A+ converge.
for convergence
is
A+ is the matrix
with ai,;,
lai,jj,
if the norm of the Jacobi than
1 a certain
on the value of the norm is allowed. In what follows the absolute the matrix
so that
it is surprising
1
is smaller
Y
to make
to satisfy
are also necessary,
a sequence
““=I__ Furthermore,
In order
diverges.
1, B = D -
with entries
not much can be said about schemes.
B must be required
conditions
exists
are so weak.
the Jacobi
to A+
of A, which shows contracting.
we shall state such conditions.
In view of the examples, will be required
for periodic
the convergence
in Sect.
conditions.
scheme
The proof is completed
THEOREM
convergence described
at a = 2. Thus,
these values of a and ,8 also
cause the first two expressions finer argument
W. MIRANKER
IlMl (or vector
zrfl, matrix
amount
I -
D-lA+
corresponding
of overrelaxation
depending
We shall now state our main result.
value of a matrix
1~1)whose components
M (or a vector
V) will be
are the absolute
value
of M (or v). A vector z, will be said to be positive if each of its components is positive. THEOREM.
(u)
Let (B, C, 9)
be a chaotic relaxatioti
scheme.
The sequence of iterates xi converges if there exists a positive vector
v and a number cc, LX< 1 such that lB]v
This huppercs if the spectral radius of 1B 1, p( IBI) < 1.
(c)
If no such v exists then there exists a sequence 9,
for which the iterates corresponding Linear
Algebra
and Its Applications
to (B, C, 9,)
2(1969). 199-222
depending on B
do not converge.
CHAOTIC
proof. three
217
RELAXATION
(a)
The
proof
of the
sufficiency
is divided
into
parts : The set {WI Iwi < a} (a rectangular
(i) into itself
by a factor
CI under
parallelopiped)
multiplication
The
sequence
of iterates
x3 is bounded.
(iii)
The
sequence
of iterates
tends
(ii)
Suppose
101 < U.
is contracted
by B.
(ii)
(i)
to zero.
Then
We shall now show that if 1XJl+r!, . . .) p+s1
for some constant
1x3( < av for all j 3 jr + S.
which occurs in the definition
This may be seen by induction jr + s. Then either9
or p = k, +i(j,J,
< av
a > 0 and a given ji then
(Here s is the parameter j,,>
condition
on j.
For suppose
# k,_,(j,),
in which
of a chaotic scheme.)
lxjl < av for all j < j,,,
in which case lx>+‘1
G 1x21 <
UV,
case 3 ‘07’ = (P, z),
where zI = x13oPk”‘0) and therefore then by the contracting
property
/zl(< uvl for each 1, 1 < 1 < 12. But of B (with v replaced
by av), we have
(bp, z) < uuvp
.. .
xfiJo+’ < avp for all 9. (iii)
notethatif
To conclude
completes
the proof of part condition
(ii).
(a), we may
ix31i-sl < av then bv what was said above lxjl< ,
/G+‘j,...,
for all j > ji.
This
the proof of the sufficiency
If xp is updated
at time j2, jz > ji, we have
uv
for all j >, jZ
ixp.l/ = (bp, z) for some z with components
Let ja be an instant j < ja with k,+,(j) j 2 ja. argument
In particular above,
so that = i.
z, < uv,.
Hence,
for each 1 < i < n there
From
what
was said above,
exists
~jl < auv for j3 + 1 < j < j3 + s.
and using Linear
the
fact
Algebra
that and
each Its
Repeating
component
Applications
j, j0 + s <
lxj\ < ccuv for all the
is updated
2(1969),
199-222
218
D. CHAZAN
infinitely there
often, we may observe that if
exists
ii+i with
the proof
We shall first
show that
where for some positive any E > 0 there
exists
let v = (vi, yva).
By choosing
C&Jfor all i 3 i, for some i,
Ix’~+‘/ < C?“V for all j 3 iii-,
lxil --f 0 and completes (b)
jxi 1<
AND W. MIRANKER
This implies
if a matrix
F has the form
vi and zj2 F1ivl < uivi, F,,v,
< cqs.
v with Fv < (K + E)U, cc = max(cl,,
Then
exists
y sufficiently the proof
v > 0 with
CQ). Indeed,
small it is easy to see how uivi + yFlzvz can
of (b) we wish to show that
lBjv<
CIV, CI< 1.
This
follows
by induction
in the normal form of IBI, and the auxilliary
just established
Indeed, this is true if IBI is irreducible
above.
it is shown for any \BI w h ose normal
Now suppose that
jBl has n + 1 components.
iB1 where F,, has n components that F,,v,
(c) Linear
1.
We write IL? in the form
by the induction
jBjv <
than
Hence,
UV,
From hypothesis
completes
F,,v,
< uzvz
u
Hence, the spectral
the proof
this it follows
for some y > 0, v = (vi, ~vs),
if IBlv < uv, v > 0, cc< 1, B is a contraction
This
result
([4], p. 30).
form has n components
and F,, is irreducible.
when cc, = p(lBI) < 1 and us < 1.
Conversely,
on the
=
< cclvI. Furthermore,
defined by /Ix/l = max( lx&J.
< YLXQ.
if p(lBI) < 1 there
number of components Suppose
for
Then
be made smaller then (tci + F)ZI~,while it is always true that ycqs To complete
that
of sufficiency.
in the norm
radius of ]Bi is smaller
of (b).
Suppose there exists no v with 1B/v < v. Then by (b) @(I B 1) > 1. Algebra
and
Its Applications
3(1969),
199-222
CHAOTIC
219
RELAXATIOK
Let v be the eigenvector eigenvalue.
jB/ corresponding
of
to the largest
Then v may be chosen to be real ([4], p. 76) and nonnegative.
Using this fact we shall now show how to construct that the sequence diverges
real
of iterates
for some initial
corresponding
a sequence
to the scheme
9,
so
(B, C, Ypo)
state z.
Let z be such that Bz= If B, C is consistent now constructed
or
z+ v
(B -
with an invertible
I)z = v.
matrix
A, then z exists.
9,
is
sequence
of
as follows:
For 1 < j < 9s we let k,(j) = j If the iteration iterates
2,
i=
l,...,n,
k,+,(j)
= i -
is started at an initial state z, the following
1.
is produced :
21 + 2’1
21 + Z’1 21 + 2/l z2
22
+
22
v2
z,
...
+
02
:
where the fact that (b+ z) = z, + vi was used repetitively. may be written
as the sum of two sequences
uil,
This sequence
and cr21,
Now let K,+,(n + 1) = 1, and select k,(n + l), i = 1, . . ., n, so that sgn(x~~‘l-k’(nfl))
= sgn bil.
Lznear
It follows
Algebra
that
and
Its
A$@ications
2(1969),
199-222
220
D.
where p is the spectral
radius
of IBI.
CHAZAN
We may
AND
continue
letting k,,+i 1 + ) I j mod n and again choosing that sgn(,i’i) ’ )=sgnbii for n+l
W.
MIRANKER
this process,
1 < KJj)
sgn bii, for 2n + 1 < j < 3n + 1. The result of this iteration
up to time
Where
i-kL(i’) = procedure
3n + 1 is:
ui2 and oi3 are the sequence
of updatings
= n + 1 so
andsgn(x,
identical
as2 and os3 have
of iterates
to the one described
generated
by a sequence
above but applied to oil, and
the form: +
P”! 2
PVl 2
FL! 2
V,
%
P"fi
2
2
2
and -
P”1 2
-
P"1 2
PU3
2
Note now that
the last column
which is identical Linear
Algebra
of crz2 together
with 021 but with
and Its Applications
-
2(1969).
with os3 form an array
pvi replacing 199-222
vi.
If we now let
CHAOTIC
221
RELAXATION
ki(j + 2n) = k,(j), i = 1, . . ., n + 1, j > n + 1 it is easy to see that the resulting p > 1.
oai cannot
sequence Suppose
a chaotic
now that
relaxation
converge
the sequence
sequence
a sequence
with initial
starting
proof
state
converges.
with the initial
on the other hand ali diverges to produce
if p >, 1 and at
state
must
diverge
if
Then oil + (~a*is
z which diverges.
If
it is easy to see how Y may be modified
z + v/2 which
diverges.
This
concludes
the
of (c) and the theorem.
It may be noted that of freedom sufficient
in proving
was used in selecting to guarantee
convergence
then the full class described chaotic
schemes
certainly
yielding
part (c) of the theorem
Y,,.
a great deal
Clearly weaker conditions of a smaller
by Definition
successively
1.
could be
class of chaotic
schemes
Any finer classification
stronger
convergence
of
results would
be of some interest. A be written in the form ‘4 = D -
E
when D is a diagonal matrix with elements d,,$ = a,,l. and ei,j = ai,i, i f
j.
Let the matrix
COROLLARY.
If
B” = I ~ wD-‘A,
the scheme (B”‘, C”, 9’)
C = wD-I,
converges fo7
1 and w = 1 if p( IBI) = u < 1.
all Y satisfying on assumptions of Definition
If 0 < co < 2/(1 + M) the scheme must converge as well. Proof.
We must
exists
v > 0 so that
there
exists
show that
u > 0 so that
lB”lv < (I(1 Let /l = (11 -
p(lB”l) < 1 or alternatively
lB”lv < ,4?v,p < 1.
031 + WE).
But
lBl(v < uv.
w) + wlBll)v < I(1 It remains
that
By (b) of the theorem
there
we know
then
cu)lv + WCS = (11 to show that
/3< 1.
WI + ocr)v. If 1 < w <
AlsoifO
we may note that it follows from the above theorem
one of the
following
(1)
A symmetric
and strictly
(2)
A irreducibly
diagonally Linear
conditions, diagonally
dominant
Algebra
and
convergence dominant
that
still holds.
( [4], p. 23).
( [4], p. 23). Its
Applications
2(1969), 199-222
222
D. CHAZAN
A symmetric
(3)
positive
definite
with
AND
W. MIRANKER
nonpositive
offdiagonal
entries. We may also note that greater
than
by simply theorem
if in case (3) the spectral
one, the iteration
turning
process
is unstable.
the proof of the theorem
around.
is as strong as we may wish it to be.
of L are not all negative,
radius
of D-l_L is
This can be shown In that
If, however,
case our
the elements
it is not clear what happens.
REFERENCES 1 W. Kahan,
private
communication.
2 P. Stein, Some general theorems on iterants, J. Res. Natl. BUY. Standards 3 J. Rosenfeld, Report
A case study
RC-1864.
1967. 4 R. Varga,
on programming
IBM Watson
Research
Matrix Iterative AnaZysis,
1962. 5 S. Schechter,
Relaxation
methods
12(1959), 313-335. 6 4. Ostrowski, Determinanten
for parallel processors,
Center, Yorktown
Prentice-Hall,
Heights,
Englewood
for linear equations,
48(1952). Research New York,
Cliffs, New Jersey,
Comm. Pure
and Appl.
Math.
Konvergenz
van
linearen
mit iiberwiegender Iterationsprozessen,
Hauptdiagonale Comm.
175-210. Received
Limar
June
Algebra
4,
and
1968
Its
Applicatiom
2(1969),
199- 222
Math.
und die absolute Helu.
30(1955),