Statistical regularization of systems of algebraic equations

STATISTICAL

REGULARIZATION OF SYSTEMS OF ALGEBRAIC EQUATIONS * E. L. ZHUKOVSKII Moscow (Received

THE QUESTION system

of the construction

AZ = u, is considered

experimental

realization

9 February 1971)

of the pseudo

when the vector

of it with a normally

solution

of the linear

u is not known exactly distributed

algebraic

but

error and covariance

matrix uoT are known. The method of solution used is the method of filtering of Markov processes. In the case where the error of the vector u is distributed arbitrarily, a filter

using

the regularization

of the solution

method of A. N. Tikhonov

[l], we present

of this problem.

1. Formulation of the problem Let A = lUijI be an m x I matrix, specifying the linear transformation m > 1, and z = {zr, . . . , z’} and ii = (3, . . . , P} be vectors

A : Rr +Rm,

the real spaces R, and R, respectively. In this paper we consider of the solution of the algebraic system of equations AZ = ii,

(1)

of

the question

iiERm,

when the vector ii is not known exactly, but its A-approximations z,, . . . . z’,... are known; a A-approximation ;);1 is a vector which is the sum of the two expectation, the vectors u”, = E + Au”,. Using M.O. to denote the mathematical expectation of the vectors of unknown vector ii = P.o.~“, is the mathematical the samples z,,; && is a vector, each coordinate of which is a normally distributed quantity with zero mathematical expectation and variance known in advance

(A&,h)2 = M.O.1ii,k - ii”1 2. ~~2 = 111.0.

The following measurements

problem

is solved:

of the coordinates

asymptotically unbiased “weighted” discrepancy

as a result

of the vector

of repeated ii E R,,,

estimate for the vector IIAZ; - u llB, deviating

*Zh. ujkchisl.Mat. mat. Fiz., 12, 1, 185-191. 1972.

230

experimental

find the consistent

z; E R,,,, minimizing the least from the given vector

and

Statistical

regularization

of systems

of algebmic

231

equations

m. E R, in the sense of the norm II- Ilc, which is defined by the scalar product (Cz, t), where B = (a~~)” and C is the matrix of weights, which will be defined in section 2, and oT is the transport matrix of cx Therefore,

we estimate

121. In this formulation

the pseudo

of the problem

solution

of problem

it is assumed

(1) in the sense

of

that the matrix A may have

rank r < I. The existence of some further information about the error AZ;, makes it possible to construct a stable statistical algorithm for the solution of problem (1) with an estimate

of the variance

of each coordinate

This information is naturally the assumption of the vector zn is normally distributed.

of the solution

found.

that the error AZ’ of the coordinates

The filter approach [3, 41 to the solution of problem (1) presented in this paper, based on the sequential Bayesian method of estimating the unknown vector Z, includes [5, 61, and enables In section

the method of statistical solution previously considered the pseudo solution of problem (1) to be constructed.

2 the formulation

problem

(1) is given

and the form of the

is indicated, in section 3 in Theorems 2 and 3 the main paper are formulated, in section 4 difference schemes for

regularizing algorithm results of the present solving

of problem

in

(1) are presented.

2.

Statistical

regularizing

algorithm

1. Let us have a sufficiently large number of realizations of the vector zn about which we suppose that E, = ii + w(n), where n is the number of the realization dimension

of the vector

m, and CJis an arbitrary

With these stochastic

chain

42)

w(n) E N(0, E)

ii,

assumptions

is a normal random vector

non-singular

about the solution

of

matrix. of problem

(1) the following

model may be used.

On the probability (z,,, &), n=l, Gl+i

=

space

&I+, =&a+

&a,

where both equations

are understood

of two random quantities. 0, we obtain

equation

(a, 3, 9) we consider

2, . . . . which controls

(l).)

aw(n + 1), in the sense

(It is obvious

the two-dimensional

the stochastic 20 =

difference 2,

a0

of the stochastic

that when the covariance

=

Markov system

u, equivalence matrix 00~ I

232

E. L- Zhukbvskii

We will interpret the sequence Z, as the *unobserved” component, and 2’ as the ‘observed” component. We denote by m,(o) = m,,(&(~), . . . . Z,(o)) the optimal

estimate

minimizing

,M. 0. ]]_zR- m, I$+ = s

(5 - m,,)T (z - m,) P (z,, E dz ) 5,

_ ), UI..... u,

Rl

where MO. llzn~lRlz < 00, for the sequence Z, which is controlled by the system (2), where M. 0. denotes the mathematical expectation subject to the conditional probability Q{zn E ds]9r;1, .... ;;, }, which will be calculated in Theorem variables

is the x-algebra of the wsets It is known [7], that

1, FZ,. ..$, & . . . . u,.

where rn is an estimate The calculation on the sequential connection

of the covariance

of these Bayesian

Theorem

generated

matrix of the vector

conditional

mathematical

method of estimating

by the random

z,.

expectations

the unknown

is based

vector Z. In this

1 holds.

Theorem 1 If z,, E N( mo, ro), and the sequence zn is controlled by the system estimates m, and rn exist, are normal and satisfy the system m,+i = m, + ~,,.AT(uu~ + AI’J*)-‘(~~++ (3)

(2), the

-Am,),

r,A=((JO= + ArnAT) -‘Ar,,

r ,,+1 = r, -

where mR is the vector

of conditional mathematical expectation: m, = r+~a.[z,]fi,, matrix of the unknown solution 2, with . ..) E*]; rh = jyiil is the covariance the elements vij = 111.0.[(zni- mni) (z,jm,,j) /u”,,, . . . , ai], i, j = 1,. . . , I; To = C’

i nitial

is the initial

approximation

Proof. normal

(4)

We consider

vector

covariance

approximation

~.o.(~n+*l&l+~) (s”+l cov(zn+I

-

P.O.

I f-L+,)

;h).

with mathematical

l3 = II”,

oo:II

By hypothesis expectation

a =

Zn)

-

E

( A > zn theorem

~u+t)[COV(Hn+r, fL+~)]-’

a,), cov(z,,

m, is the

it has a two-dimensional

_ By the normal correlation

= Bf.0. 2” + cov(zn+l,

=

matrix c

Z.

the pair (z,,

distribution

matrix

for the positive-definite

for the vector

C0V(Z,+1,U”n+1)

x

x

and [g],

Statistical

regularization

of systems

of algebmic

equations

233

{cov(Z*+i, Zn+i)l-‘[cov (Zn+i, u”n+i)P, which proves the existence r n = Cov(z,+il&+l). In order to calculate P (r,+,,

&+i I L+I).

m,+i = ~0. (z,+i) iL+l)

of the estimates

the quantities &+1

Vn+i =

Let

occurring

and

here we find the distribution

be a random variable;

we show that

( &+i 1

it is normally distributed. For this we write down the characteristic a two-dimensional random variable: f(t) =

rd. 0.

[nt.o.(exp

ar.o.[exp {iPa -

{it%+i} 1u’,, . . . (6, zn)l =

‘/#j3t} 1u”,,. . . , iii] =

116. o.(exp (itTaiz,} / E,, . . . , Qexp - i/#aiTrnait

exp (itaim, Thereby calculate

{-‘/zt’Br)

- ‘/#fit},

=

where

the normality of the distribution of the quantity the moments of v~+~. They are given by

cov(ii,,+i,

cov(z?l+i,

zn+1 I an) = ra,

cov(z,+L,

u”n+i1ii,) = I-,.@.

Note 1.

these

Solving

values

al = un+I is proved.

We

M.O.(u”,+~i E,) = Am,,,

1y.o.(z,+i I En) = mnr

Substituting

function

of the moments

the system

&+I 1ii,,) = AI’,A= + (J(T~‘,

in (4), we obtain

(3).

(3), we obtain n

mn = [E +

45)

ro_4=((J@)-' AR]-’

mo + I’oAT(wT)-’ El a=,

In the particular

case

where u is commutative _

UUTO-’ + ATA

(‘3)

-

a.

with the matrix A, we obtain

ugTro-*rno + AT-

a=, r, = &{ro-i

(au*) + ATAn]-‘.

As will be shown below, the estimate

the solution

m,, Tn for the pseudo

solution

of the difference

system

(3) gives

for

E. L. Zhukovskii

234

2; = lim m, 78400 and its covariance Note 2.

matrix r.

The statistical

regularizing

algorithm

of the solution

of problem

(1) is constructed similarly when we require that not only z~+~ t z,, but also the k-th difference of the solution sn with respect to n remain constant from realization of the input vector zn. Then the Bayesian filtering problem is written in the form ii n-l-l

(2’)

4, + uw(n + I),

=

Gl+i

=

(1) Gl+i

=

2,

2, +

(i)+ 2 ? . . . . . . . . . . . &a

z’“:,--n z’R’ 11

with the initial

conditions

z(r))

=

zp, . . . , z’A)(o) = 0.

z(i) (0) =

zo,

Then the estimate mk(n) of the regularized solution z of problem (l), the k-th difference of which equals zero from experiment to experiment, will be the solution

of the system Amo(n) = roo(n)AT(aa')-'(u"n+l-AAm~(n)), -Am,(n)), Ami = m,(n) + I’~(n)A~(~d-‘(~n+i . * * . . . . . . . . . . . . . . . * * * AmA

(3’)

AI’(n) =

-[(E+

= mA_,(n)

+ rA,(n)AT(~UT)-'(~7t+i -Am~(n))v

ar(n)

d(n) + rbw -

a)r(n)aT]r~~*+Ar(n)A7-1[(E+4rwm

with the initial conditions rno (0)

=

no, rni(0) = mi,

...q

mA(0) =

0;

r(o)

=

rO*

Here 0 E 0 ...O OOE...O a= . . . . . , OO...E oo...o

are matrices vector,

of dimension

whose elements

kl x kl, and A = (A, 0, . . . . 0) is a (12+ l)-component are matrices

of dimension

m x 1. The regularizing

Stutisticcd regularization of systems of algebraic equations

algorithms

(3) and (3’) of the solution

controlled

by the system

2.

We consider

the random parametric

T”[z, G] = Il.42- k2

(7)

a > 0 is a parameter functional

of problem

(2) and (2’) are thereby

(1) for the case where it is constructed.

functional

+ alb- mb2,

which consists

is the square

235

of the sum of two functionals:

of the deviation

the first

of the vector

I/Z - m,ljc From AZ in the norm of the matrix B = (00. T)*‘; the second functional is the square of the deviation of the unknown solution z from m, in the norm \I - \I,, where C = r,‘It is easy to prove that there exists a vector, unique with probability ma = hi. G.z*, realizing the minimum of the smoothing functional (7) and identical with (5) for a = a, = l/n.

1,

This can easily be verified if Euler’s equation for the functional T’lIz,^ul is used and the system (4) is summed. It is also obvious that if we assume that the matrices A and 0 are commutative and in (5) put P = (oaT) r,“, we obtain a Tikhonov regularization for problem (1) with a = a, = l/n. This

shows

that the sequential

Bayesian

method of estimating

the unknown

vector zU in problem (1) or the problem of the linear unsteadxfiltering of the ‘unobserved” component zn from the uobserved” component U, of the Markov by the system (3), in the case where the sequence (2% 2 n) t which is controlled input vector U, has a normally distributed error, defines a Tikhonov regularizing algorithm with a = l/n. The solution Bayesian

sequential

of problem solution

(1) realized

by equation

(5) will be called

the

below.

The identity of the Tikhonov regularized solution and the Bayesian sequential solution in the case of a Gaussian error makes it possible to establish some statistical properties of the solution obtained by the Tikhonov regularization

method (asymptotic

unbiasedness

and independence

as b- + 0); and

conversely, since the Bayesian sequential solution is a solution of Euler’s equation for the functional T’?z, ~1,it will be shown in Theorem 2 that for a = l/n + 0 it is a pseudo solution of problem (1) and is calculated as the limit

236

E. L. Zhukouskii

zu = lim 7nn. *-cQ 3.

As is obvious

from the statement

of section

2, the parameter

in the

statistical regularization is the number of realizations of the input vector zh. It is known that one of the fundamental criteria of the accuracy of the approximations

pR (As, G), 1 which in our case may be the first term of the functional Ta[z, u^l, that is, (A~ - &)~(oo~)-i(~z - ^u), which has a x2 distribution with (I - 1) degrees of freedom. statistical

of z obtained

is the magnitude

And therefore, the ‘discrepancy check of the hypotheses. 3.

Properties

principle”

of the Bayesian

Let To and uoT be the matrices holds.

of the discrepancy

defined

in this case

sequential previously.

will be a

solution The following

lemma

Lemma. The condition

~‘~A~(ocr~)-l~x = 0

is equivalent

to the condition

A=AZ =

0, x E RI.

We denote by L the subspace in R, of vectors such that L = ix E R[ : Gx = 0) = ker G, where G = iToAT(uu~) -*A, and N=(~ER~:~IL}~=I~G,

and

L CBN = R1.

Theorem2 The sequence m, = [E + I',,AT(uuT)--l An]-’

mo + I’oAT(uuT)-’ [

Cl S=i u”,

with probability 1 in R, to the vector iii = mO'+ fi",wheremo’ = np,mo, z; = E = lim m, = mp’ + ro%(u-tAro’lz)+u-‘~ is a pseudo and iii”= np.,+i n solution of problem (1).

converges

We calculate -t + roAT(uuT)-LA

x I

Statistical

regularization

of systems

ro-’ + AT(UCP) -I A

of algebmic

237

equations

--I 1ro-‘mo + 1 1 n

n

--i

r,,-’ + rl~(cw)-’

A

a=,

Because (~a’)-r

of the positive =

definiteness

of the matrices

r0 and B we can write

Q2, I’,,-’ = ‘P, Q, 0 > 0.

Writing D=

V = QA,

VW’

and

f=-

ii,,

we obtain lim m, =

n-rm

lim CP-~ $

n-m

which is satisfied inverse

i

E +DT D]-'D,Q~ =

TI-WCI 1

J n( +

r;JZ (o-lAr~~2)+a-1ii

with probability

1.

= zlb’

The symbol ( )+ denotes

the pseudo-

matrix.

Lemma

1 of [2] was used in the calculation

Note 3. and m,lker defines

E + DTD]-~~ @,,'ro-lmo+ lim@-i

[

In particular, ATA,

if problem (1) has a solution

the theorem proved implies

an estimate

of the second

m, converging

limit.

in the ordinary

that the iterative

to the normal solution

scheme

sense (3)

in the mean-square

sense. We return to the question of the estimate

of the asymptotic

of the Bayesian

m,=~.o.(z,/n,

sequential

,...,

unbiasedness

and independence

solution

Ei).

Theorem3 The Bayesian independent

sequential

estimate

The asymptotic

solution

for the exact unbiasedness

m, is an asymptotically

solution

unbiased

and

of problem (1).

of the estimate

m, follows from the proof of

238

E. L. Zhukouekii

Theorem

2.

We prove the independence of the estimate m,. For this we show that cov(m,-zz;)+O_We have cov(m,-zz;;) =~.o.(m,-i&,)(m,,-a,,)~+ (FE,,-zz;;:) (iii” - 2;;)‘. where 1 = mn

n

z&o.-

ii ,. n c 8-l

Using

of Theorem 2, we obtain

the notation

then ~.o.(m,--Eiii,)(m,-iiiiii,)T=-Q)-i

It follows tends

1 n

from-the proof used in Theorem

Ln f

1

-1

DTD

DTD@-‘.

2 that (iii, - zi) (iii, - 2;)~.

as n + m

to 0 in the norm 11- llRl. It is obvious from this that the matrix c& (m, also proves the independence of the estimate m,, for the pseudo

2;) + 0, and this

solution

2;.

.4. Numerical method of solution The approximate method of solving problem (1) in the case of a Gaussian error reduces to the solution of equation (S), requiring only one inversion of the when n realizations of the vector are known. matrix [E + l?oAT(aaT)-lAn]-i, The solution z’, can also be obtained by minimizing functional (7) by direct methods of minimizing quadratic

the smoothing functionals.

We consider problem (1) with the error of the vector z,, distributed arbitrarily. In this case a filter of the solution of problem (1) can be constructed by means of Tikhonov’s regularization method. Let the vector zrr have an arbitrarily distributed error with a non-singular covariance matrix uuT. Let equation

za = (c& + AT(oar) -‘A) -~A=(ooT) -l^u of problem (1) for I0 = E, where UC-

calculated

be the Tikhonov

regularized

1 +& P/l *=1

for some realizations

of the vector

z?, and a > 0 is the parameter

Statistical

defined, Then

for example,

the following

regularization

of systems

as the solution iteration

of algebraic

of the equation

algorithm

is proposed

It is obvious

that as the initial

za, and at subsequent Numerical

steps

calculations

approximation

we will improve performed

ljdz=-^ulls (Tikhonov

(.:X1

z,,+! = zn +(a,!? + AT(~uT)-iA)-‘AT(uuT)-i

equations

239

= lla, _ ills.

filter):

u”,+i -dAz,

. 1

*=*

Z, = 0, at the first

step we obtain

it.

at the Moscow University

Computer

on model problems have demonstrated the efficiency and good agreement Bayesian sequential solution with the given theoretical solution. In conclusion I wish to thank A. N. Tikhonov for supervising A. N. Shiryaev and R. Sh. Liptser for their interest and assistance paper. The author thanks V. A. Morozov for reading making a number of valuable comments.

Centre of the

the paper, with the

the paper in manuscript

and

and

Translated by J. Berry REFERENCES 1.

TIKHONOV, A. N. On incorrect problems of linear algebra and a stable method of solving

them.

Do&l. Akad. Nauk SSSR,

163. 3, 591-595,

1965.

2.

MOROZOV, V. A. On pseudo solutions. 1387-1391.1969.

3.

LIPTSER, R. SH. and SHIRYAEV. A. N. Non-linear filtering of diffusion processes. Tr. Mat. in-ta Akad. Nauk SSSR, 104, 135-180, 1968.

4.

GLONTI. 0. A. Sequential Markov chain. Litovskii

5.

PETROV, A. P. Estimates of linear functionals for the solution problems. Zh. uychisl. Mat. mat. Fiz., 7, 3, 648-654.1967.

6.

TURCHIN, V. F. Solution of a Fredholm equation of the first kind in a statistical ensemble of smooth functions. Zh. uychisl. Mat. mat. Fiz., 7, 6,1270-1284, 1967.

7.

GIKHMAN, I. I. and SKOROKHOD. Processes (Vvedenie v teoriyu

8.

ANDERSON, T. W. An Introduction to Multiuariate v mogomernyi statisticheskii analiz). Fizmatgiz,

Zh. vychisl.

Mat. mat. Fiz.,

filtering and the interpolation mat. sb., 2, 263-279, 1969.

9, 6,

of components

Markov

of a

of some inverse

A. V. Introduction to the Theory of Random sluchainykh protsessov), ‘Naukam, Moscow 1965.

Statistical Moscow.

Analysis 1963.

(Bvedenie