Entropy-convergence in Stieltjes and Hamburger moment problem

NORTH- H ( X l A N D

Entropy-Convergence in Stieltjes and Hamburger Moment Problem Marco Frontini and Aldo Tagliani

Dipartmento di Matematica Politecnico di Milano Piazza L. da Vinci, 32 20133 Milano, Italy

ABSTRACT In the classical Stieltjes and Hamburger moment problem the sequence of maximum entropy approximants, whose first M moments are equal to given ones, is considered. It is proved that whenever an infinite moment problem is determined, then maximum entropy approximants converge in entropy to the function characterized by given moments. Entropy-convergence is proved by using exclusively existence and uniqueness conditions. © Elsevier Science Inc., 1997

1.

INTRODUCTION

The Maximum Entropy (ME) Principle is widely used in diverse scientific contexts to estimate an unknown density from measured or calculated moment data. The idea of ME is simply to choose the density which maximizes some measure of entropy, subject to given moment constraints. Two desirable features of this methodology are: 1. Existence of such an ME density given by first M assigned moments (being unicity guaranteed by the form of ME solution); 2. The ME estimate should converge to unknown density as the number of given moments increases. Existence problem has been completely solved in three classical Hausdorff [1], Stieltjes and Hamburger [2] cases. The convergence problem has been widely treated only in the Hausdorff case [3]. In Stieltjes and Hamburger cases, only a sufficient condition of entropy-convergence has been given [2].

APPLIED MATHEMATICSAND COMPUTATION88:39-51 (1997) 0096-3003/97/$17.00 PII S0096-3003(96)00305-0

© Elsevier Science Inc., 1997 655 Avenue of the Americas, New York, NY 10010

40

M. FRONTINI AND A. TAGLIANI

In this paper, entropy-convergence (and so Ll-norm convergence) is studied in Stieltjes and Hamburger cases, by proving that whenever the moment problem is determined, then ME approximants converge in entropy to unknown density. 2.

KNOWN RESULTS ABOUT S T I E L T J E S AND HAMBURGER MOMENT PROBLEM

We recall some classical results about the Stieltjes and Hamburger moment problem which will be used in the convergence proof. Such results are drawn essentially from [4]. Given a set of moments/zj, j = 0 . 1 , . . . , Hankel determinants are defined as

--

--

,:

/z o

...

/.tf

~N

"'"

/-~ N

I

,... ,

~1 ~2

~2

]~3

"'"

~N+I

(2.2)

h(1) = ,~N

'

~N+I

"'"

~2N+l

i) Existence Existence of a solution of (infinite) moment problem is related to the positivity of Hankel determinants, more precisely the following Theorems hold.

THEOREM 2.1. ([4], p. 5, Theorem 1.2) problem

~n= f+f _ tnd¢,

For the Hamburger m o m e n t

n= 0,1,...

to have a solution, it is necessary that A n >t 0, n = 0, 1 , . . . . In order for a solution to exist whose spectrum is not reducible to a finite set of points, it is necessary and sufficient that A n > O,

n=0,1,....

Entropy- Convergence and Moments Problem

41

In order for a solution to exist whose spectrum consists of precisely k + 1 distinct points, it is necessary and sufficient that A o > 0,...,Ak

>

O,

Ak+ 1 = Ak+ 2 =

...

= 0.

The moment problem is determined in this case.

THEOREM 2.2. ([4], p. 6, T h e o r e m 1.3):

I z , = fo

t"d~b'

For the Stieltjes moment problem

n=0,1,...

to have a solution, it is necessary that A n > / 0 , and A(~) > / 0 , n = 0, 1 , . . . . In order for a solution to exist whose spectrum is not reducible to a finite set of points, it is necessary and sufficient that A n > 0, A(~) > 0

n = 0, 1 , . . . .

In order for a solution to exist whose spectrum consists of precisely k + 1 points distinct from t = O, it is necessary and sufficient that A o > 0,...,A h ~ ) > O,

• '',

k > 0, Ak+ 1 = A k + 2 = . . . = 0 5(~,) > 0, A(1) ~ k + l

= h (1) k+2

~

"''

= 0

While in order for a solution to exist whose spectrum consists of precisely k + 1 points, one of them being t = 0, it is necessary and sufficient that A 0 > 0,...,Ak i(ol) ~> 0 , . . . ,

> 0, Ak+ 1 ----Ak+ 2 = . . . ---- 0

A(1)_ 1 > 0, A(1) = A(~)+I -~-

. . .

= 0.

In the last two cases, the moment problem is determined ii) Uniqueness T h e o r e m s 2.1 a n d 2.2 are n o t , i n general, sufficient to g u a r a n t e e u n i q u e ness of s o l u t i o n in t h e m o m e n t p r o b l e m ( d e t e r m i n e d p r o b l e m ) e v e n if i n f i n i t e m o m e n t s te k are assigned.

42

M. FRONTINI AND A. TAGLIANI

T h e results on uniqueness require knowledge of a s y m p t o t i c behavior of the following quantities: A~

p.(0)

= --

(2.3)

A~ A('2

pO)(0) = A(1)-

(2.4)

where A ~ and A°,)- are obtained respectively from A n and A(1) by deleting the first row and the first column. Let / ~ , , and /z~, n be values for which A n = 0 and A(ln) = 0 respectively, then the conditions of T h e o r e m s 2.1 and 2.2 m a y be interpreted, in the determined cases, as follows.

REMARK 2.1. W h e n /z 0 - /~, n, H a m b u r g e r and Stieltjes m o m e n t problems are determined as dO degenerates in n masses ( T h e o r e m 2.1 and 2.2 with k + 1 = n). I f / z I = IZl, ,, Stieltjes m o m e n t p r o b l e m is determined as dO degenerates in n + 1 masses ( T h e o r e m 2.2 with k = n). W e recall the following results ([4], p. 69, 75).

THEOREM 2.3. A necessary and sufficient condition that Hamburger moment problem be determined is that at least one of the quantities p ( 0 ) = lim p , ( 0 ) ,

p(Z)(0) = lim p(2)(0)

(2.5)

be equal to zero, where p(2)(0) = A(2)/A(2)-n, , , with

~2

AT) = iz 2 , A(~) = I ~P'2

~3 , ~4

"'"

~N+2

~2) = "'',

(2.6)

~ N +2

"'"

~2N+2

THEOREM 2.4. A necessary and sufficient condition for the Stieltjes moment problem to be determined is that at least one of quantities

Entropy- Convergence and Moments Problem p(O) = lim p.(O), n..-}

43

p(1)(O) = lim pO)(O)

oo

(2.7)

n ......} o o

be equal to zero. (Theorem 2.4 is a reformulation in terms of p~(O) and p°)(O) of a result given in [4], p. 75).

REMARK 2.2. It is interesting to observe that whenever p(0) = 0, then both the Hamburger and Stieltjes moment problems, corresponding to the same moments sequence, are determined. On the other hand if p(0) > 0 and p(1)(0) = 0, then the Stieltjes moment problem, corresponding to a given moments sequence, is determined, unlike the Hamburger moment problem.

iii) Geometrical interpretation Quantities p,(0) and p(~l)(0) benefit from an interesting geometrical interpretation as they represent respectively the distance between it0 and /~, ~ and between te I and pe~,~, where teo, ~ and /~, ~ are allowable lower bounds of P~0 and t~l, when first 2n and 2n + 1 moments are assigned. More precisely p.(o)

= go -

p(.')(o)

= ,o

,o.

(2.8)

(2.9)

-

As stated by Theorems 2.3 and 2.4, unicity conditions imply, in moment space, the relationships lim /~, ~ = /~0,

(2.10)

lim /.~,,

(2.11)

n---~ a¢

=

~£1,

n - - - ~ oo

which will be used in next convergence proof. 3.

PROBLEM FORMULATION

Let R be the definition interval of an unknown density function f ( x ) ( R -= [0, + ~) in the Stieltjes case and R -= ( - ~ , + oo) in the Hamburger case) and

H[ f] = - fRf( x)ln f ( x ) dx,

(3.1)

44

M. FRONTINI AND A. TAGLIANI

the Boltzmann-Shannon entropy of density f(x). The ME Principle represents a formal procedure to obtain approximant fM(X) of f ( x ) with first M + 1 moments /zj, j = 0 , . . . , M equal to the given ones. Through Lagrange multipliers technique, one seeks maximization of functional entropy S = S ( f ) defined as

S ( f ) = -- fRf( x) ln f( x)dx + j~__o)tj

xJf( x)dx - I~ •

(3.2)

Functional variation with respect to unknown density f ( x ) yields

dS( f ) = 0 ---) f u ( x)

exp

Ajz ~

(3.3)

df

to be supplemented by constraints condition

tt n = fR x"exp

- Y'~Ajx j , n = O , . . . , M j=0 /

(3.4)

where )~j are Lagrange multipliers. Integrating by parts (3.4), next moments /zj, j = M + 1, M + 2 , . . . are function of the first M + 1 ones and of Lagrange multipliers, and may be obtained in both Stieltjes and Hamburger cases

as

M

( n + 1)t~, -

E jAjlz,+j = 0, j=l

n >/ 1.

(3.5)

From (3.1) and (3.4) we have an explicit formula in terms of Aj and /zj of H[ fM] entropy

H[ fM] = --

f fM(x)ln fM(X)

M

dx = ~_~ Ajtx j. j=o

(3.6)

Necessary and sufficient conditions for existence of an ME solution of a finite Stieltjes and Hamburger moment problems are given [2] by the following theorem.

Entropy- Convergence and Moments Problem

45

THEOREM 3.1. Whenever an upper bound for the moments does not exist, necessary and sufficient conditions for existence of a M E solution are the following: i) Hamburger case ( M = 2 N) A 0 > 0, A 1 > 0 , . . . , A

N > 0.

(3.7)

ii) Stieltjes case M=2N A 0 > 0, A(ol) > 0, A 1 > O, A(1) > 0,...,~N_IA(1) > 0, A N > 0

M= 2N+

1

A o > 0, A(~) > 0, A 1 > 0, A(1) > 0 , . . . , A N > 0, A(~ > O.

4.

(3.8)

(3.9)

ENTROPY-CONVERGENCE

It is evident that entropy-convergence implies uniqueness of moment problem solution f ( x ) . F u r t h e r m o r e , entropy-convergence means lim m ~ I - l [ fm] = H[ f]. i) Hamburger case ( M = 2 N). L e t /xl,... , ~£M be assigned and /z 0 is varied continuously. Then from (3.4), Lagrange multipliers Aj are functions of /x0. By differentiating (3.4) with respect to be0 we have the differential linear system

I

(4.1)

(where unknown moments ~t~M+1,---, ~I'2M are obtained from (3.5)). From (3.6), by taking into account the first equation of (4.1), d --H[fM] d/£0

M = A0 • j= 0

d}tj /zj-7--- = A0 -- 1, a~£0

(4.2)

46

M. FRONTINI AND A. TAGLIANI

and by (2.3),

d2

dA o

dix2 H[

dg----~o -

1

PM(O-------~ <0

(4.3)

hold. T h u s H[ fu] is a differentiable concave function o f / x 0. W e are in position to prove t h a t limu_, ~ H[fM ] = H [ f ] . Let us suppose the relationship

lim H[

fM] = Htim > H[ f ]

(4.4)

M-,

holds. When, for given re0, M -* ~, we have from geometrical interpretation of unicity conditions lim M ~ ~/Z~, M = /x0 while, fixed M, by varying tt 0

lim

H[fu]

= -- ~

(4.5)

/.to--' ~o,, M

as fM(X) degenerates in a set of N masses. Let us consider the following sequences of ME solutions:

1. fM(X) with m o m e n t s /x0 . . . . ,/z M. 2. f ~ ( x ) with m o m e n t s tt 0 - e , / Z l , . . . , J[£M with 8 ~ R + and /z 0 > J[~O,M SO t h a t f~t(x) exists and /-/[ f~] < /-/[ f ] holds. As M--* ~, then from (2.10) e ---) 0 so t h a t the m o m e n t /x 0 of fM(X) and f ~ ( x ) differ by an arbitrarily small quantity, while their entropies differ by a finite q u a n t i t y greater t h a n Hli m -- HI f ] (see Figure 1; take note t h a t /z 0 admits only a lower b o u n d given by the positivity of determinants (2.1)). Taking into account (4.4) we have reached a contradiction on the continuity of/-/[ fM] as a function of /z 0. T h e n H,m = HI f ] holds and so the result

lim H[

M - ~ oo

is proved.

fM] = H [ f ]

(4.6)

Entropy- Convergence and Moments Problem

47

0 -10 -20

-3O -4O Hlim -50

-6O H[f]

-70 -80 -90 -100

1 0

0.5

1

1.5

' 2

' 2.5

3

' 3.5

4

4.5

5

FIG. 1. HI fM] as function of /z0 for different M values. The obtained result can be formulated by:

THEOREM 4.1. In the Hamburger case, whenever the sequence of assigned moments determines uniquely a density, then ME approximants converge in entropy to ](x).

ii) Stieltjes case Once moments sequence is assigned, the Stieltjes problem should be determined, unlike the Hamburger moment (Remark 2.2). Thus it is necessary to consider the two cases separately. a) Stieltjes and Hamburger problems determined. The formal procedure is similar to the Hamburger case. By differentiating (3.4) with respect /%, (4.1)-(4.5) are obtained again and conclusion (4.6) is reached. b) Stieltjes problem determined and Hamburger problem undetermined. From Remark 2.2 it follows p(O) = lim p~(O) = e > O,

p(')(O) = lira p(~l)(O) = 0

48

M. FRONTINI AND A. TAGLIANI

where c is a constant. Let re0, re2, . . . , ~[~M be assigned and t¢1 varied continuously. Then from (3.4), the Lagrange multipliers A/are functions of tt 1. By differentiating (3.4) with respect to ~1 we have the differential linear system

lit1

~['tM- 1 ~M

]'£2

"'"

~/'M ~t£M+ 1

~M+I

"'"

"

"'" "'"

~2 M- 1 ~b2M

]

dA1/d~l dA2/ d~i

= -

(4.7)

[ dAM/d~l

(where unknown moments ~M+ 1 , ' ' ' , ~[£2M fire obtained from (3.5)). From (3.6), by taking into account the first equation of (4.7),

d M dhj d~l H[ .fM] = E tey-7-j= o a~l

+ X1

(4.8)

~1,

and

d2 dh 1 1 d,~ H I / , , ] = d ' l = - A---M"

~"£0

~'£2

~[~3

"'*

]A'M

~'L2

~"f~4

Jt£5

"*"

]A'M+ 2

tt£3

~5

['£6

"'"

~[~M+ 3

~L~M

JtKM+ 2

~/'M+ 3

"'"

tU'2 M

<0

(4.9)

hold. By algebraic properties of definite positive matrices, the second determinant in (4.9) is positive, thus I-l[fM] is a differentiable concave function of tL1. We are in position to prove that lira i-~ =/-/[ fro] ---- /t[ f]. Let us suppose the relationship

lira H[ fu] = Ulim > H[ f ]

M-* :~

(4.10)

Entropy- Convergenceand Moments Problem

49

holds. When, for given/zl, M -~ ~, we have from geometrical interpretation of unicity conditions limM~®lZ~, M ----/1'1 while, fixed M, by varying ~1 aim

U [ f M ] ---- - - ~

(4.11)

~'1-'* ~ , M

as .fM(X) degenerates in a set of N masses. Let us consider the following sequences of ME solutions:

1. fM(X) with moments ~0 . . . . , IZM" 2. f~(X) with moments IZ0, ~1 - 8, IZ2,..., ~t£M with E ~ R + and ~1 -- E > ~t£1,M SO t h a t f ~ ( x ) exists and /-/[ f~] ~
lim H I fM] = H I [ ]

(4.12)

M--~

is proved.

-5 -10 Hlirn -15 -20

H[f| -25 -30

M+2

-35 M -4O

-45

0.5

-

1 FIG. 2.

1.5

2

HIfM] as function of ~1 for different M values.

2.5

50

M. FRONTINI AND A. TAGLIANI

THEOREM 4.2. In Stieltjes cas~ whenever the sequence of assigned moments determines a density uniquely, then M E approximants converge in entropy to f(x).

REMARK 4.1.

Once direct divergence /[ f, fM] is defined as

(4.13)

convergence in entropy is equivalent [5] to convergence in direct divergence, being tej(f) = IZj(fM), j = 0 , . . . , M, then

I [ f , fM] = H[fM] -- H [ f ] .

(4.14)

Also the following inequality is known to hold [6]:

i[ f, fM]

If(x) - f . ( x )

(4.15)

Thus convergence in direct divergence (or equivalently in entropy) previously proved implies convergence in Ll-norm. 5.

CONCLUSIONS

The classical Stieltjes and Hamburger moment problem of maximum entropy approximants, whose first M moments are equal to given ones, has been considered and it has been proved that, whenever an infinite moment problem is determined, maximum entropy approximants converge in entropy to a function characterized by given moments. This result is similar to the one obtained in the Hausdorff case [5], where the infinite moment problem is always determined if the existence conditions are verified. With out technique, the entropy-convergence has been proved by using exclusively existence and uniqueness conditions. If additional assumptions on regularity of an unknown function are added we conjecture that, as in Hausdorff case [3], a stronger convergence holds. Numerical experiments concerning the inversion of Laplace transform (strictly connected to Stieltjes moment problem) (results not yet published) support such a conjecture.

Entropy-Convergence and Moments Problem

51

REFERENCES 1. Csiszar, I-divergence geometry of probability distributions and maximization problems. Annals of Probability3:146-158, (1975). 2. M. Frontini, A. Tagliani, Maximum entropy in the finite Stieltjes and Hamburger moment problem. J. Math. Physics 35:6748-6756, (1994). 3. J. M. Borwein, A. S. Lewis, Convergence of best entropy estimates. SIAM J. Optimization 1:191-205, (1991). 4. J. A. Shohat, J. D. Tamarkin, The Problem of Moment~ Amer. Math. Soc., (1963). 5. B. Forte, W. Hughes, Z. Pales, Maximum entropy estimators and the problem of moments. Rcndiconti di Matematicc4 Serie V I I 9:689-699, (1989). 6. S. Kullback, Information Theory and Statistic~ Dover, New York, (1962).