Modes of convergence of Markov chain transition probabilities

JOURNAL

OF MATHEMATICAL

Modes

ANALYSIS

of Convergence

AND

APPLICATIONS

of Markov

of Mathematics

Chain

Transition

Probabilities

L. TWEEDIE

RICHARD

Division

60, 280-291 (1977)

and Statistics,

CSIRO,

Canberra,

Australia

Submitted by T. T. Soong

Suppose {P”(x, A)} denotes the transition law of a general state space Markov chain {XJ. We find conditions under which weak convergence of {X,} to a random variable X with law L (essentially defined by s P(x, dy) g(y) + JL(dy)g(y) for bounded continuous g) implies that (X,,} tends to X in total variation (in the sense that /I P(x, a) - L 11+ 0), which then shows that L is an invariant measure for IX,}. The conditions we find involve some irreducibility assumptions on {X,,} and some continuity conditions on the one-step transition law {P(x, A)}.

1.

INTRODUCTION

We suppose that {X,} is a discrete parameter Markov chain on a measure space (Z”, 9) with temporally homogeneous transition probabilities P(x,

A) = Pr(&+,

Two of the fundamental

E A 1X,

= x),

AE~,

XE.%-.

questions for such chains are:

(i)

Do the transition

(ii)

if so, is the limit

probabilities

converge in any sense as n + co; and

an invariant

probability

distribution

for (X,} ?

Many of the approaches to this problem have involved thinking of the transition law P(*, *) as an operator on one of the L, spaces of (x, F), and recently several authors (e.g., [l, 151) have investigated the related problem: (iii) If the transition probabilities converge in any stronger sense ?

converge in some weak sense, do they

Typically, these authors show that if {P”} converges in the weak operator topology, then “averages” (usually CCsaro means) of the (P”} converge in the strong operator topology under certain conditions. In this paper we look at these questions when convergence is in the weak and strong measure-theoretic topologies. If pn(x, .) -L(e) in the weak (measuretheoretic) topology, then we identify conditions which ensure that L is the unique invariant measure for (X,) and that the convergence is, in fact, in total 280 Copyright All rights

8 1977 by Academic Press, Inc. of reproduction in any form reserved.

ISSN

0022-247X

MARKOV

CHAIN

TRANSITION

PROBABILITIES

281

variation. Less stringent conditions are found which continue to ensure invariance of L, but only give convergence of averages in total variation. Our conditions involve some degree of irreducibility and continuity on the transition probabilities. We do not think they are as weak as possible: Rather, we have tried to give conditions that will be readily verifiable at the same order of practicality that might enable the original weak convergence criterion to be verified. In the final section we give some easy examples which illustrate the need for conditions such as ours to ensure that weak convergence of transition probabilities implies anything further. Our examples show that assertions of invariance of a weak limit, such as those in [6, Theorem 21 or [5, p. 3111, need not always be true.

2.

PRELIMINARIES

Suppose that (9?, s) is a topological space, with F the Bore1 u-field of 3’. Let {XJ denote a fixed Markov chain on 3 with temporally homogeneous transition probabilities {P”(x, A)), w here as usual, for each 3, P”(zc, .) is a probability measure on s, and for each A, P”(‘, A) is a measurable function on %. We write @, V, and Y for, respectively, the set of bounded measurable functions from 3 to the real line and the subsets of continuous and lowersemicontinuous elements of 9. For any measure 1” on 9 we write pg for .fg( y) MY), and P”(x, g) for J PYx, dr) g(y), for any g E 9. The basic assumption in most of our theorems is that {X,} converges zzeakly to a random variable X on 3. If X has the distribution Pr(X E A) = L(A),

AEF,

then this weak convergence is defined following Topsee [17] by demanding that, for all r E S and g E Y, there exists L such that liF+,“f Pn(x, g) > Lg. For g E V, (1) implies

i% P”(x,g) = Lg, which is the more usual definition of weak convergence (cf. [2, 121); on spaces which are sufficiently rich in continuous functions, (1) and (2) are of course equivalent. (Our assumption of the existence of L as a measure on 9 in (1) or (2) is often redundant. When % is metric, for example, (1) holds if we assume merely that for each g E %‘, Pn(x, g) tends to the same limit K(g) for every

282

RICHARD L. TWEEDIE

x E 9). We say that (X,} convergesto X in total variation if, as n -+oo, for each x II qx,

-1 -L

II + 0,

(3)

where 11Ij denotes total variation of a signed measure. Since (3) implies P(x, g) --+ Lg uniformly for g E Bk = (g E 95 1g(x)1 f k, x E %}, it is clearly stronger than (1); in this paper, we find conditions on {X,} which imply the converse, and also conditions which suffice for CCsaro weak convergence liy+rrf ‘, i Pm(x, g) > Lg, m-1

gE2

(4)

to imply CCsaro total variation convergence 11; i P”(x, -) - L /I+ 0. 1

We introduce two forms of irreducibility.

We call {X,} irreducible if the measures

are equivalent (have the same null sets), and &irreducible if there is some nontrivial u-finite measure (b on 9 such that +(A) > 0 implies C Pn(x, A) > 0 for every x ES. Irredu’cibility is used by Sidak [16], and is stronger than + irreducibility, used by Orey [lo]. We write 9+ for the set of .F sets of positive y,-measure for all x when (X,) is irreducible, and for the set of 9r sets of positive $-measure when (X,} is +-irreducible: No confusion should result. We also introduce three forms of continuity for the one-step transition laws {P(x, .)}. We say that {P(x, a)) is we&y continuous if, for every g E 9, P(x, g) is also in 9; that {P(x, .)} is strongly continuous if, for every g E 5$?‘, P(x, g) is in %:,which is equivalent to asking that, for every A E 9, P(x~ , A) -+ P(x, A) whenever x,-+x; and that {P(x, .)} is strongly equicontinuous if the set {P(x, A), A E .Fj is an equicontinuous set of functions of x.

3. THE MAIN RESULTS Suppose either (i) (X,} is $-irreducible and (P(x, *)} is strongly continuous or (ii) {X,,) is irreducible, and there exists a measure4 on 9, equivalent to each of the measuresyz , such that 4 is regular and the support of 4 is of second category in the relativixed topology; and {P(x, .)} is weakly continuous. Then (4) implies (5), and L is invariant fm {X,}. THEOREM 1.

MARKOV

CHAIN

TRANSITION

PROBABILITIES

283

THEOREM 2. Suppose{X,,> is $-irreducible and (P(x, .)) is strongly continuous. Then (1) implies (3) and L is inwariant for (X,}.

The proofs of these results are given in a sequence of lemmas in Section 3. The basic dea is to show that, under the conditions of Theorem 1, (4) implies that (a) &I must be recurrent rather than transient; (b) IS,,) must be +-recurrent (which is a stronger condition recurrence);

than

(c) {*Y,J must be positive, rather than null, recurrent. These conditions enable one to use results in [IO] to show that (5) holds. To prove Theorem 2, we further show that strong continuity and $-irreducibility ensure, with (I), that (d) {A-J is aperiodic. That (3) holds then follows again from [IO]. Without some form of irreducibility, results from [lo] are not applicable. Before proving (a)-(d), however, we prove the following result, which shows that with a stronger continuity condition, irreducibility can often be removed. It also illustrates that much stronger continuity assumptions than those of Theorems 1 and 2 are needed to enable us to deduce the equivalence of (I) and (3), or (4) and (5), from known results (such as those in [3] or [14]) about the equivalence of weak and strong convergence of general sequences of measures. THEOREM 3. Suppose F is a separable metric space, with 9 the Bore1 o-jeld on Y. If {P(x, .)} is strongly equicontinuous, then (1) implies (3) and (4) implies (5); and L is invariant for {X,,).

Proof.

.%ssumethat (1) holds. For any g E E, (1) implies Lg = ;+NJP*(x, g) = $E Pn-l(x, P(., g))

= LP(*, g>, since P( ., g) is continuous. Hence the measures L and LP agree as operators on V, and so, since S is metric, are identical [12]; that is, L is invariant for {X,). Now set cd = {P(x, A), A E S}, so by assumption, & is a uniformly bounded and equicontinuous class of functions of x. Write, for fixed .1cE SY”,,un = Pn(x, .). From (3.3) of [14], (1) implies that, as n--f CO,

RICHARD L. TWEEDIE

284

But pnP(-, A) = P+l(x, A), and L is invariant, and so (3) holds for any x; and {X,> converges in total variation to X. A similar proof with pn = n-l g shows that (4) implies (5).

pyx,

.),

1

To amplify further the connections between Theorems I and 2 and the results of [14], we give the following corollary. COROLLARY. Suppose that 55 is a separable metric space, with 9 a-field. If(i) of Theorem 1 holds, and, as n + co,

the Bore1

for all x and for every class ~2 of uniformly bounded equicontinuous functions, then the same uniform convergence holds for any class & of umformly bounded measurable functions. If(i) or (ii) of Theorem 1 holds, and rf as n -+ CO,

for every x and every class JZ? of unrformly bounded equicontinuous functions, then again the convergence is uniform over any class d of uniformly bounded functions. Proof.

This follows immediately from our results andTheorem3.1 of [14].

1

4. PROOFS OF THE MAIN THEOREMS

Our first step is to notice that, from &irreducibility, we can classify {X,} as either transient (if for some A E F+, 1 Pn(x, A) < co for all x) or recurrent (if for all A E F+, C P”(x, A) = co for all x), analogously with the definitions when SY is countable (cf. [18, Theorem 11; similar results are shown in [9] or [16]). We prove LEMMA A. Suppose (4) holds and {X,} is +irreducible, and that either (i) {P(x, .)} is strongly continuous or (ii) {P(x, .)} is weakly continuous, and 4 is regular, with support of second category in the relativized topology. Then {X,} is recurrent. Proof. Suppose that {X,} is transient. If (ii) holds, it follows from [13] that there is a sequence (U,} of open sets with lJ, t EZ and, for each fixed K

MARKOV

CHAIN

TRANSITION

285

PROBABILITIES

Pn(x, UK) -+ 0 as n-+ CO,for all x. Arguments similar to those of [13J show that if {P(x, .)) is strongly continuous, such a sequence of open sets also exists. Now if (4) holds, it follows that L(s) = 1, and so there exists K such that L(C;,J > 0. Let Ik be the indicator function of Uk ; then 1k E 2, and LI, .b 0; but for every .Y, Pn(x, Ik) + 0 as 12-+ co. This contradicts (4), and so (A:n) is recurrent. 1 The definition of recurrence is easily shown to imply that, for any rl E .F+, F(x, A) = Pr{Xn E A for some 711X(, L- 2) is unity for all x not in some set in 5 - s+ (cf. [9, Theorem 21). IfF(x, A) = 1 for all A E 3+ and all .2:E 3, then we call {XJ +recurrent [lo, p. 41. We now show that the conditions of Theorem 1 imply that /X,> is actually $-recurrent. LEMMA

Proof. theorem.

B. If {Xn} is irreducible, then recurrence implies qkecurrence. This is shown in [9, Theorem 83, as remarked after the end of that 1

It is easy to prove Lemma B directly. Rather harder is to show: LEMMA C. If {X,) is +-irreducible, (P(x, .)} is holds, then {X,,} is +recurrent.

PYOO~. Suppose (X,} is not $-recurrent, NA = {y: F(y, A) < 1) # O. We note that

strongly

continuous, and (4)

and choose A E F+

F(x, A) = P(x, ‘4) + jACP(.r, dy) F(y, a);

such that

(6)

since P(x, -4) is continuous, and since, because [F(., A) IAC(.)J is bounded, P(x, [F(., A) IAC(.)]) is continuous, (6) shows that F(*, A) is continuous. We distinguish three cases. I. The limit measureL has L(Ni) = 1, and, for x E N.A , F(r, A) has at least two vaZuesp, and p, > p, . Let N&J,) = {,K F(x, A) ,( pB}. Since F(., A) is continuous, N,(p,) is closed, and hence, its complement is an open set containing N; . Take g, to be the indicator function of [N4(p2)]c so that g, E Y: We have Lg, = I, and so from (4), for all x

(7)

286

RICHARD

L.

TWEEDIE

But for any x, and any m,

so that

w, 4 2 Pz‘,

i

P”@, [N‘4(P,)l”)

??I=1

-P2

from (7). Since we have assumed the existence of x E NA withF(x, A) = pi < pa, this is a contradiction; so in this case, NA is empty. II. L(N:) = 1 and F(x, A) = p for x E NR . Since NA = {x: F(x, A) < p}, it follows that NA is itself closed. Let g, be the indicator of Nz ; as before, Lg, = l.Nowforx~N,,andalln

Prn(%dY)F(Y, A)

and we have a contradiction with (4); so NA is empty. III. L(N,.,) > 0. Since NA = {x: F(x, A) < l}, NA is open, so that, if g, is the indicator of NA , g, E 0: and in this case,Lg, > 0. Now Lemma A shows that {X,J is recurrent, and so from [9], for $-almost all x, we have Q(x, A) = Pr(X, E A infinitely often ( X,, = x) = 1. For any such x, we have from

[l - Q(x,41 3 jNAJYx, dy) [I - qy, A)] that Pn(x, NA) = P”(x, gN) = 0; so again we contradict (4), and NA is empty. 1 Since A was arbitrary, this completes the proof of $-recurrence of {X,). For a #-recurrent chain we know [lo, Theorem 7.21 that an invariant measure exists. Further, we have a classification of {X,) as either null or positive recurrent, again analogously with the integers. In the former case [lo, Theorem 7.31 holds; in the latter, [lo, Theorem 7.1(i)] holds. LEMMA D. If (X,} is +-recurrent, and either of the continuity or (ii) of Lemma A hold, then {XJ is positive recurrent if (4) holds. Proof.

Suppose that fX,> is actually null recurrent.

conditions (i)

MARKOV CHAIN TRANSITION

287

PROBABILITIES

If (ii) holds, then from [13], we can again find a sequence of open sets U, t Z such that Pn(x, U,) + 0 for all x. As in Lemma A, this contradicts (4), and so the assumption of null-recurrence is not tenable. If(i) holds, a similar sequence can be constructed; and so in both cases, (X,} is positive recurrent. 1 Suppose now either of the conditions of Theorem 1 hold. From the lemmas, we can apply [IO, Theorem 7.1(i)] to see that there is an invariant probability measure r such that

But (4), together with (8), means r = L and so (5) holds, and L is invariant; and thus the proof of Theorem 1 is complete. To complete the proof of Theorem 2, we need to show that (1) implies that {X,) is aperiodic [IO, p. 151, and this we can do under the assumption of strong continuity. LEMMA E, If (X,} is $-irreducible, (P(x, .)} is strongly continuous and (1) holds, then (X,} is aperiodic. Proof. Suppose {X,} has period d > 1 and let (C, , ..., C,) be a cycle [IO, p. 131, so that for Y < d, c, = {x: P(x, C,,,) = l}

and

Cd

=

{x:

P(x,

Cl)

q=

1).

Since P(x, C,) is continuous for each I, each of the sets C, is closed. Moreover, for some Y, L(C,) > 0, for L is given by r in (8), since the conditions of the lemma imply that Theorem 1 holds; r is stronger than 4, from [I 0, Theorem 7.2(iii)J; and #(SY - u%, C,) = 0. Write I, for the indicator of (C,-,)” (or Ci , if I = 1), so that LI, > 0, and I, E 2. Now for every x E C, , we have lim inf Pn(x, IT) = lim Pnd-l(x, I,) == 0, n-tm n-)x since Pnd-l(x, C,-,) = 1, x E C, . This contradicts (I), and so {X,) is aperiodic. Hence under the conditions of Theorem 2, we can apply the aperiodic form [IO, Theorem 7.1.(i)] and (3) holds: So the proof of Theorem 2 is complete.

5. COMMENTS AND EXAMPLES 5.1. Unless we assume some connection between {X,} and the topology Y, such as weak continuity, we should not expect to conclude much from (1). 409/6011-19

288

RICHARD

L. TWEEDIE

In particular, we cannot conclude that L is in any sense a stationary measure for {X,,}. For consider a chain on LIY= [0, a) with, for tt = 1,2,... P(n-1, 1 + (n + 1)-i) = OL,, P(n-1, (n + 1)-l) = 1 - a, , P(1 + n-1, (n + 1)-l) = 1, and for other x. P(x, x - 1) = 1,

x > 1,

P(x, x) = 1,

1 >x>o.

Then {X,} always converges weakly in the CCsaro sense to a random variable concentrated on (0, 1}, while if a,,-+ 0, {X,} converges weakly without averaging to the degenerate variable at 0. Clearly, L is not stationary for (X,}. By choosing 01,--f 0 sufficiently slowly, this example can be made to satisfy all of Condition A of [6], and hence provides a counterexample to the main result of the paper; one such choice is 01, = 1 - (nj(n + l))2. The example in [19, Sect. 61 shows that the claim that Condition A of [6] implies stochastic boundedness of the n-step transition probabilities is also false. To see that the weak convergence of {P”} in stochastically monotone chains [S] does not imply the existence of a stationary distribution, one can look at a similar model with 55 = [0, 1] u (2) and P(x, 1 - n-1) = 1,

x E [1 - (n - 1)-l, 1 - n-l),

n = 2, 3,...,

P(1, 2) = P(2,2) = 1; this has a stationary distribution concentrated at {2}, but for x < 1, P(x, .) converges weakly to a measure concentrated at {I), which is not invariant for {X,}. Hence, this example contradicts the lemma of [5, Sect. 51. 5.2. Similarly, one can show that (1) need not imply any setwise convergence of {P(x, .)> even when (P(x, *)} is weakly continuous, unless some irreducibility holds. A trivial example is supplied by setting 5? = (0, n-r n > I}, with P(0, 0) = 1, P(n-i, (n + 1)-l) = 1. Here, though, L is concentrated at zero and so is invariant for {X,}, and as in the proof of Theorem 3, it can be shown that (1) and weak continuity of {P(x, .)> together imply that L is invariant at least in the weak sense that Lg = LP( ., g) for every g E 5~7.Under suitable regularity conditions, (for example, when Z” is a metric space [12]), it is then possible to deduce that L satisfies the usual measure-theoretic invariant equations. One could then use the results of [I l] to derive Theorem 1, with the regularity conditions replacing our condition on Q.

MARKOV

CHAIN

TRANSITION

PROBABILITIES

289

5.3. If S is countable, with F the discrete topology, and if {X,) is irreducible in the classical sense, then our theorem asserts that, if there exists a measure L such that P(i, j) *L(j), i, Jo 3 (that is, if {X,) is aperiodic and positive recurrent [4]), then

1 I Pn(i,j)-L(j)[+O,n-+

co.

This result can be deduced from [7, IV.8.141. 5.4. Inspection of the proofs of Theorems 1 and 2 shows that, rather than (1) or (4), it is enough to assume that there is a nonnegative function L on the open sets in fl such that either liy+$f

P(x,

U) = L(U)

(9)

or l$ipf

n-l f Pm@, U) = L(U) 1

holds for every open U, and L satisfies

for every sequence {U,} of open sets increasing to %, and

L(u')=

1

(11)

for every open set U’ > UC whenever L(U) = 0. In the next section, we give an example of a chain such that (9), (lo), and (11) hold; and where, instead of aperiodicity and positive recurrence, we have a chain which is periodic and null. Moreover, the transition law of the chain is weakly continuous, and the chain is irreducible, with regular irreducibility measure: Hence, it is clear that the second category assumption in our conditions is not unnecessary. 5.5. Let S be a circle of circumference 2(2)‘/“, and let % be the union of S, , the rational points in S, with S, + 2112= ( y E S: y = x + 2ii2, x E S,}; and let {t! , j = 1,2,...} be an ordering of S, . Note that .‘Z”is countable, but not second category in the relativized topology of S. Let (Pj , j = I,...} be a probability distribution on the integers, with Pj > 0 for all j; and for any x E 9, let

290

RICHARD L. TWEEDIE

Then {P(x, .)> is the transition law of a random walk {X,) which is clear11 $-irreducible with 4 as counting measure on L3?. Further, Feller [S, pp. 270-2741 shows that {P(x, *)} is weakly continuous and that, as 12+ co, P(x, .) (considered as a measure on the Bore1 sets of S converges weakly to the uniform distribution of S. Now any open set U in S i the restriction to 3Y of an open set U’ in S; and it follows that (9), (IO), and (11 hold with L( U) = p( U’), where p is uniform probability measure in S. But {X,} is periodic, with periodic classes S, and S,. + 2112;and (X,) is nul since if it were positive, we would have lit:y

P (0, (0)) = v. > 0,

which would contradict the uniformity

of the limiting distribution

of P"(O, -:

ACKNOWLEDGMENTS I am grateful paper.

to D. B. Pollard

and C. C. Heyde for some useful conversations

on thi

REFERENCES 1. M. A. AKCOGLU AND L. SUCHESTON, Weak convergence of positive contraction implies strong convergence of averages, Z. Wahrscheinlichk’eitiei’bstheorie .oerw. Geb. 3 (1975), 139-145. 2. P. BILLINGSLEY, “Convergence of Probability Measures,” Wiley, New York, 1968 3. P. BILLINGSLEY AND F. TOPSBE, Uniformity in weak convergence, 2. Wahrscheinlich keitstheorie were. Geb. 7 (1967), 1-16. 4. K. L. CHUNG, “Markov Chains with Stationary Transition Probabilities,” 2nd ed Springer-Verlag. Berlin. 1967. 5. D. J. DALEY, Stochastically monotone Markov chains, Z. FVahrsckeinZichkeitstheori WYW. Geb. 10 (1968), 305-317. 6. L. E. N. DELBROUCK, On stochastic boundedness and stationary measures for Marko processes, /. Math. Anal. Appl. 33 (1971), 149-162. 7. N. DUNFORD AND J. SCHWARTZ, “Linear Operators, I: General Theory,” Wiley Interscience, New York, 1958. 8. W. FELLER, “An Introduction to Probability Theory and Its Applicatinos,” 2nd ed. Vol. II. Wiley, New York, 1971. 9. N. JAIN AND B. JAMISON, Contributions to Doeblin’s theory of Markov processer Z. Wahrscheinlichkeitstheorie oerw. Geb. 8 (1967), 19-40, 10. S. OREY, “Lecture Notes on Limit Theorems for Markov Chain Transition Prob abilities,” Van Nostrand, London, 1971. 11. F. PAPANGELOU, “A Martingale Approach to the Convergence of the Iterates of Transition Function,” Manchester-Sheffield Research Report 55/FP/l, 1975. 12. K. R. PARTHASARATHY, “Probability Measures on Metric Spaces,” Academic Press London/New York, 1967.

MARKOV

CHAIN

TRANSITION

PROBABILITIES

291

13. D. B. POLLARD AND R. L. TWEEIUE, R-theory for Markov chains on a topological state space, II, 2. Wahrscheinlrchkeitstheorie verw. Geb. 34 (1976), 269-278. 14. R. RANCA RAO, Relations between weak and uniform convergence of measures with applications, Am. Math. Statist. 33 (1962), 659-680. 15. R. SATO, On mean ergodic theorems for positive operators in Lebesgue space, J. Math. Sot. Japan 27 (1975). 207-212. 16. Z. SID& Classification of Markov chains with a general state space, in “Transactions Theory, Statistical Dec. Functions, of the Fourth Prague Conference on. Information and Random Processes, 1965,” pp. 547-571, Academia, Prague, 1967. and Measure,” Springer-Verlag, Berlin, 1970. 17. F. TOPWE, “Topology 18. R. L. TWEEDIE, R-theory for Markov chains on a general state space, I, Ann. ProbabiIity 2 (1974), 840-864. 19. R. L. TWFEDIE, Sufficient conditions for ergodicity and recurrence of Markov chains on a general state space, Stochastic Proc. Appl. 3 (1975), 385-403.