- Email: [email protected]
Singularly Perturbed Markov Chains with Two Small Parameters: A Matched Asymptotic Expansion
Journal of Mathematical Analysis and Applications 267, 281 – 309 (2002) doi:10.1006/jmaa.2001.7772, available online at http://www.idealibrary.com on
Singularly Perturbed Markov Chains with Two Small Parameters: A Matched Asymptotic Expansion Q. G. Liu and G. Yin1 Department of Mathematics, Wayne State University, Detroit, Michigan 48202
and Q. Zhang2 Department of Mathematics, University of Georgia, Athens, Georgia 30602 Submitted by S. P. Sethi Received March 31, 2000
This work is concerned with asymptotic properties of solutions to forward equations for singularly perturbed Markov chains with two small parameters. It is motivated by the model of a cost-minimizing firm involving production planning and capacity expansion and a two-level hierarchical decomposition. Our effort focuses on obtaining asymptotic expansions of the solutions to the forward equation. Different from previous work on singularly perturbed Markov chains, the inner expansion terms are constructed by solving certain partial differential equations. The methods of undetermined coefficients are used. The error bound is obtained. 2002 Elsevier Science (USA) Key Words: manufacturing system; singularly perturbed Markov chain; multiple scale; forward equation; asymptotic expansion.
1. INTRODUCTION Much of the current interest stems from the motivation of various applications in manufacturing systems. In such applications, one aims to 1 This research was supported in part by the National Science Foundation under Grant DMS-9877090. 2 The research of this author was supported in part by the Office of Naval Research Grant N00014-96-1-0263.
281 0022-247X/02 $35.00 2002 Elsevier Science (USA)
All rights reserved.
282
liu, yin, and zhang
find the optimal production-planning strategies and to use Markov jump structures in the modeling of uncertainty (e.g., machine capacity). In many applications, one often needs to deal with large-scale systems. Even for a seemingly not so complex system, the optimal control can be very difficult to obtain (see [12, Sect. 5.2]), not to mention the added difficulty of a very large dimensional state space involved. In fact, not only is it an impossible task to obtain analytic solutions, but also the direct numerical approximation can be overwhelming and the straightforward use of numerical methods may not be adequate. Naturally, one seeks to break the large job into small pieces with the hope that certain decompositions and aggregations will lead to a simplification of the intractable systems; see Simon and Ando [13]. A viable alternative calls for taking advantage of the high contrast rates (some states vary an order of magnitude faster than the rest) of changes in the physical systems and using a singular perturbation approach as a tool to reduce the complexity of the underlying systems. To have a thorough understanding of the problems, it is of the utmost importance to learn the intrinsic structures and asymptotic properties of the underlying Markov chains. Continuing effort has been devoted to studying singularly perturbed Markov chains; see [3, 9–12] among others. Recently, in [6], Khasminskii et al. used matched asymptotic expansion to establish the convergence of a sequence of the probability vectors. Singularly perturbed Markov chains with recurrent states, naturally divisible into a number of classes, are then treated in [7]. This line of work has been continued in [15], in which we have further derived asymptotic expansions for Markov chains with the inclusion of transient states and absorbing states, and asymptotic distributions for Markov chains having recurrent states. In this work, we examine a continuous-time model with two small parameters. The use of multiple small parameters stems from consideration of hierarchical structures of various stochastic systems in manufacturing as well as in communication networks. The motivation comes from a production-marketing system (see [Chap. 11]). The underlying problem is concerned with a cost-minimizing firm involving production planning and capacity expansion and a two-level hierarchical decomposition. In [12], Sethi and Zhang have found the asymptotic optimal strategies by using the limit distribution of the Markov chains. In this work, our main effort is devoted to obtaining asymptotic properties of the solutions to the forward equation satisfied by the probability vector. In previous work on singularly perturbed Markov chains, the asymptotic expansions are obtained by solving appropriate algebraic and ordinary differential equations, whereas in this paper, partial differential equations are also involved. The results are useful in the subsequent studies on Markov decision processes and controlled Markovian systems involving multiple small parameters. To proceed, let us begin with an example of the manufacturing system.
singularly perturbed markov chains
283
Example 1.1. Consider a production-marketing system that produces a single product type using a two-state production capacity αt ∈ 0 1. Let ut denote the production rate subject to the production constraints 0 ≤ ut ≤ αt. Let xt ∈ 1 denote the total surplus and let zt ∈ z1 z2 denote a two-state stochastic demand rate. The system is given by xt ˙ = ut − zt x0 = x 1
(1.1)
1
where x ∈ is the initial surplus. Let wt ∈ denote the rate of advertising satisfying 0 ≤ wt ≤ K, with K representing an upper bound on the advertising rate. The profit functional J· is defined by Jx α z u· w· ∞ =E e−ρt πzt − h1 xt + cut + wt dt 0
(1.2)
where ρ > 0 is the discount rate, π is the revenue per unit sale, h1 · is the surplus cost function, and c < π is the unit production cost. The problem is to find a control ut wt t ≥ 0, that maximizes Jx α z u· w·. Let = 0 z1 1 z1 0 z2 1 z2 denote the state space of the pair αt zt. Then as in [12], the process αt zt can be formulated as a Markov chain generated by 1 1 Au + Bw ε δ −µ1 µ1 0 0 1 λ u −λ1 u 0 0 = 1 0 0 −µ1 µ1 ε 0 0 λ1 u −λ1 u −µ2 w 0 µ2 w 0 1 0 µ2 w 0 −µ2 w + λ2 w 0 −λ2 w 0 δ 0 λ2 w 0 −λ2 w
Qεδ u w =
Here ε and δ are small parameters signifying the frequency of jumps of α and z. Note here µ1 represents the machine repair rate and hence is independent of production rate u. It is easily seen that
µ1 Q with Q = −µ1 Au = diag Q λ1 u −λ1 u −µ2 w µ2 w Bw = Qw ⊗ I2 with Qw = λ2 w −λ2 w where I2 denotes the 2 × 2 identity matrix, and “⊗” denotes the usual Kronecker product (i.e., with X = xij and Y = yij , the ijth entry of X ⊗ Y
284
liu, yin, and zhang
is defined to be X ⊗ Y ij = xij Y ). In what follows, we shall generalize this model to include many states and weak and strong interactions. The rest of the paper is arranged as follows. Section 2 gives the precise formulation of the problem. Section 3 proceeds with the construction of the asymptotic expansion. Section 4 concentrates on the error analysis. Finally, Section 5 concludes the paper with further remarks.
2. FORMULATION 2.1. Initial Formulation We work with a finite horizon 0 T for some T > 0. Recall that a generam×m is said to be weakly irreducible (see [15]), if f tQt = 0, tor Qt
m ∈ and i=1 fi t = 1 has a unique nonnegative solution. The unique solution is termed a quasi-stationary distribution. The structure of the generator Qε δ to be considered is motivated by Example 1.1. To proceed, first generalize the setup in the example to include many states. For ease of presentation, consider the stationary case. Let αε δ t be a Markov chain with αε δ t = αε1 t αδ2 t and state space = a1 b1 am0 b1 a1 bl am0 bl where l and m0 are some positive integers. Denote m = lm0 . Suppose the generator is given by Qε δ = 1/εA + 1/δB, where
Q ⊗ Im and B = Q (2.1) A = diag Q 0
= q˜ ∈ m0 ×m0 being a generator and Q = q¯ ij ∈ l×l being with Q ij m0 ×m0 a generator and Im0 ∈ is an m0 × m0 dimensional identity matrix. Henceforth, we use the symbol diag to denote a diagonal block matrix with and Q are both weakly irreducible. appropriate entries. Suppose that Q Then it is easy Let ν˜ denote the stationary distribution corresponding to Q. ˜ = Q/δ. is the to see that diag˜ν ν˜ Qε δ | This demonstrates that Q/δ generator of the Markov chain αδ2 t with state space b1 bl . Similarly, with state space we can show αε1 t is a Markov chain generated by Q/ε a1 am0 . We next generalize this idea further for nonstationary cases and to include weak and strong interactions. 2.2. More General Formulation Suppose ε > 0 and δ > 0 are small parameters, αε δ t is a finite state Markov chain with state space = 1 m generated by Qε δ t, and the row vector pε δ t = Pαε δ t = 1 Pαε δ t = m ∈ 1×m denotes the probability distribution of the Markov chain at time t. It is well
singularly perturbed markov chains
285
known that pε δ · is a solution to the forward equation dpε δ t = pε δ tQε δ t pε δ 0 = p0 (2.2) dt
where p0i ≥ 0 for each i and p0i = 1 is the initial probability distribution. Assume that the generator Qε δ t ∈ m×m has the form 1 1 At + Bt + Qt (2.3) ε δ Assume that At and Bt have the same partitioned form and the same and Q replaced by time form of Kronecker product as in (2.1) with Q dependent Qt and Qt, respectively. The slowly changing motion is described by the generator qˆ 11 tIm0 qˆ 1l tIm0 = (2.4) Qt qˆ l1 tIm0 qˆ ll tIm0 Qε δ t =
Throughout the rest of the paper, unless otherwise noted, we always work with (2.3) with time-varying generators. In addition, we often use the notion |j = 1 1 ∈ j×1 , for some integer j. We make the following assumptions: = q¯ ij t are weekly irre(A1) For each t ∈ 0 T , Qt and Qt ducible. (A2) For some n ≥ 0, A·, B·, and Q· are n + 1-time continuously differentiable on [0, T]. In addition, d n+1 /dt n+1 A·, are Lipschitz on [0,T]. d n+1 /dt n+1 B·, and d n+1 /dt n+1 Q· Define an operator Lε δ by L
ε δ
1 1 df −f At + Bt + Qt f = dt ε δ
(2.5)
for any smooth vector-valued function f ·. Then Lεδ f = 0 if and only if f is a solution to the differential equation (2.2). We are now in a position to analyze the solution to (2.2). 3. ASYMPTOTIC EXPANSION Using singular perturbation techniques, to approximate the solution to (2.2), we seek outer–inner expansions of the form t t (3.1) + eε δ n t pε δ t = ϕε δ n t + ψε δ n ε δ
286
liu, yin, and zhang
where eε δ n t is the remainder, and the outer (regular part) and initial layer correction terms are given by ϕε δ n t =
n
εi δj ϕi j t
i+j=0
and
ψε δ n
t
n t t t εi δj ψi j = ε δ ε δ i+j=0
(3.2)
respectively, such that eε δ n t is small and the error bound holds uniformly in t. The main theorem is recorded below. Theorem 3.1. Assume (A1) and (A2). For small parameters ε > 0 and δ > 0, denote the unique solution to (2.2) by pε δ ·. Then for 0 ≤ i + j ≤ n, we can construct ϕi j · and ψi j · · such that (a)
ϕij · is twice differentiable on 0 T .
(b) For each i, there are κ1 > 0 and κ2 > 0 such that κ t κ t ij t ε 1 + K2 exp − 2 ψ ≤ K1 exp − ε δ ε δ (c) Suppose there exist constants h1 > 0 and h2 > 0 such that h1 ε ≤ δ ≤ h2 ε. Then the following estimate holds n n ε δ i j i j i j i j t t sup p t − ε δ ϕ t − εδψ ε δ t∈ 0 T i+j=0 i+j=0
≤ K εn+1 + δn+1 (3.3) When i = j = 0 in the above theorem, we obtain the limit of pεδ t for t > 0. Corollary 3.2. Suppose Qε δ · is continuously differentiable on 0 T , which satisfies (A1), and d/dtQ· is Lipschitz on 0 T . Then for all t > 0, limε δ→0 pε δ t = ϕ0 0 t. To obtain the desired result, we obtain the outer and inner expansion terms by direct construction in what follows. They involve solutions to algebraic-differential equations. 3.1. Outer Expansion We begin with the construction of ϕε δ n+1 · in the asymptotic expansion. Consider the differential equation Lε δ ϕε δ n+1 t = 0 where Lε δ is given by (2.5).
singularly perturbed markov chains
287
Equating coefficients of εi δj , we have for i + j = 0, ε−1 δ0 ε0 δ−1
ε 0 δ0
ε1 δ−1 ε−1 δ1 and for 1 ≤ i + j ≤ n, ε2 δ−1
ε 1 δ0
ε 0 δ1
0 = ϕ0 0 tAt 0 = ϕ0 0 tBt dϕ0 0 t = ϕ1 0 tAt + ϕ0 1 tBt dt + ϕ0 0 tQt 1 0 0=ϕ tBt 0 1 0=ϕ tAt 0 = ϕ2 0 tBt dϕ0 1 t = ϕ2 0 tAt + ϕ1 1 tBt dt + ϕ1 0 tQt dϕ0 1 t = ϕ1 1 tAt + ϕ0 2 tBt dt + ϕ0 1 tQt 0 2 0=ϕ tAt
(3.4)
(3.5)
ε−1 δ2 ··· · We construct the solutions to (3.4). For construction of solutions to systems with higher order, the approach is similar, only the notation is more involved. The main idea is as follows. We separate the equations in (3.4) into two groups. The first group consists of the first two equations. We show that they share a common solution. In fact, this common solution is the limit as ε → 0 and δ → 0. The second group includes the rest of the three equations. To solve the equations in the second group, we use the methods of undetermined coefficients. First, assume the solution to the last two equations is of particular form with certain functions (“multiplier”) to be determined. Then our task reduces to finding these functions by using the third equation. Such an idea is used in the construction of initial layer corrections as well as higher order terms. 3.1.1. Construction of ϕ0 0 t Based on (A1), Qt and Qt are irreducible. Let πt and λt be their quasi-stationary distributions, respectively. Denote πλ t =
λ1 tπt λl tπt ∈ m . According to assumption (A2), the vector-valued functions λt, πt and πλ t are (n + 1)-times continuously differentiable. We claim that the vector-valued function ϕ0 0 t = πλ t = 0, so solves the first two equations of the system (3.4). Note that πtQt
Qt ϕ0 0 tAt = λ1 tπt λl tπt diag Qt
= λ1 tπtQt λl tπtQt = 0
288
liu, yin, and zhang
where At = diagQt Qt. As for the second equation, λt
l Qt = 0; that is, i=1 λi tq¯ ij t = 0 for all 1 ≤ j ≤ l. Thus, q¯ 11 tIm0 q¯ 1l tIm0
ϕ0 0 tBt = λ1 tπt λl tπt q¯ l1 tIm0 q¯ ll tIm0 l l = λi tπtq¯ i1 tIm0 λi tπtq¯ il tIm0 i=1
=
l i=1
In addition, since
m 0 0 t = 1 i=1 ϕi
i=1
λi tq¯ i1 t πt
l
i=1
λi t = 1 and
m0
l i=1
i=1
λi tq¯ il t πt = 0
πi t = 1 ϕ0 0 t|m =
3.1.2. Construction of ϕ1 0 t To obtain the desired solutions, we use the methods of undetermined coefficients. We postulate that the fourth and fifth equations of (3.4) are given by ϕ1 0 t = λ1 txt λl txt and ϕ0 1 t = y1 tπt yl tπt where πt and λt are the quasi-stationary distributions of Qt and Qt, respectively, and where the vector-valued functions xt = x1 t xm0 t ∈ 1×m0 and yt = y1 t yl t ∈ 1×l are
1 0 to be determined. In addition, ϕ1 0 t|m = m t = 0, and i=1 ϕi
0 1 m 0 1 ϕ t|m = i=1 ϕi t = 0. The fourth and fifth equations of (3.4) are not enough to determine xt and yt. We also need to use the third equation of (3.4), which can be written as ϕ1 0 tAt + ϕ0 1 tBt =
dϕ0 0 t − ϕ0 0 tQt dt
The right-hand side of the above equation can be expressed as dϕ0 0 t = dπλ t − π tQt − ϕ0 0 tQt λ dt dt dπt dπt = λ1 t λl t dt dt
singularly perturbed markov chains 289 dλ1 t dλ t + πt l πt dt dt l l λi tqˆ i1 tπt λi tqˆ il tπt − i=1
i=1
Two equations may be derived from the third equation of (3.4). They are dπt dπt 1 0 ϕ λl t (3.6) tAt = λ1 t dt dt and dλ1 t dλ t ϕ0 1 tBt = πt l πt dt dt l l λi tqˆ i1 tπt λi tqˆ il tπt (3.7) − i=1
i=1
Consider (3.6). In view of the block diagonal form of At, (3.6) is equivalent to = λ t dπt λ1 txtQt 1 dt ··· = λ t dπt λl txtQt l dt It suffices to show that the following system has a solution = xtQt
dπt dt
and
m0 i=1
xi t = 0
(3.8)
by NQt. Denote the null space of Qt The weak irreducibility of Qt implies that rank Qt = m0 − 1 and hence dimNQt = 1 Note that NQt is spanned by |m0 . By virtue of the well-known Fredholm alternative, the first equation in (3.8) has a solution only if its right-hand side is orthogonal to NQt. Since NQt is spanned by |m0 , πt|m0 = 1 dπt|
m0 and dπt |m 0 = = 0, so the orthogonality is verified. Next we dt dt show that the system (3.8) has a unique solution. To this end, let us rewrite the system (3.8) as
q˜ 11 tx1 t + · · · + q˜ m0 1 txm0 t =
dπ1 t dt
q˜ 1 m0 tx1 t + · · · + q˜ m0 m0 txm0 t = x1 t + · · · + xm0 t = 0
dπm0 t dt
(3.9)
290
liu, yin, and zhang
The existence and uniqueness of the solution to (3.9) follows from the weak t = | Qt. irreducibility and Fredholm alternative. Denote Q Then c m0 the solution can be expressed as xt =
dπt tQ t−1 Q c t Q c c dt
Moreover xt is n-times continuously differentiable on 0 T . Consequently, (3.6) has a unique solution ϕ1 0 t = λ1 txt λl txt
m 1 0 t = 0. In addition, ϕ1 0 t is n-times with the condition i=1 ϕ continuously differentiable. 3.1.3. Construction of ϕ0 1 t Examine (3.7). Rewrite Eq. (3.7) in its component form l i=1
yi tq¯ ij tπt =
dλj t dt
πt −
l i=1
λi tqˆ ij tπt
for j = 1 l
To establish the existence of a solution, it suffices to show that = ytQt
dλt − λtQh t dt
(3.10)
l = 1, has a solution, where Qh t = qˆ ij ∈ l×l . Since i=1 λi t dλt h − d/dtλt| l = d/dtλt|l = 0. Owing to Q t|l = 0 dt
λtQh t |l = 0; i.e., the right-hand side of (3.10) is orthogonal to |l . By (A1), rankQt = l − 1. An argument similar to that of (3.8) confirms that = ytQt
dλt − λtQh t dt
l i=1
yi t = 0
(3.11)
has a unique solution yt = y1 t yl t. Hence ϕ0 1 t = y1 tπt yl tπt solve (3.7). It is easy to see that ϕ0 1 t is
0 1 n-times continuously differentiable. In addition, we have m t = i=1 ϕi
l
m 0
l j=1 yj t k=1 πk t = j=1 yj t = 0. 3.1.4. Construction of ϕi j t for 2 ≤ i + j ≤ n + 1 In essentially the same way, we can construct ϕij t for 2 ≤ i + j ≤ n + 1. The details are thus omitted. To summarize, we state the following theorem.
singularly perturbed markov chains
291
Theorem 3.3. Assume (A1) and (A2) hold. Then there exist n + 1 − i + j-times continuously differentiable solutions ϕi j t, 0 ≤ i + j ≤ n + 1, for (3.4) and (3.5). In addition, the following conditions are satisfied: m k=1
0 0
ϕk
t = 1
m k=1
i j
ϕk
t = 0
for 1 ≤ i + j ≤ n + 1
(3.12)
Remark 3.4. The conditions in (3.12) imply that, for 1 ≤ i + j ≤ n + 1, ϕi j 0|m = 0; i.e., ϕi j 0 is orthogonal to |m . This fact will help us to obtain the desired exponential decay property of the initial layer terms. 3.2. Inner Expansion In this section, we construct the initial layer terms ψi j · ·. It consists of two parts. First, we obtain the sequence ψi j · · by solving a system of partial differential equations. Then we present the exponential decay property of the solutions. 3.2.1. Construction of ψi j · · Following the idea in singular perturbation, define the stretched time variables as follows: t t (3.13) τ= µ= ε δ Note that τ → ∞ as ε → 0, and µ → ∞ as δ → 0 for any t > 0. Consider the differential equation Lε δ ψε δ n+1 τ µ = 0 where the operator Lε δ is defined in (2.5) and ψε δ n+1 τ µ is defined in (3.2). Thus the above equation can be written as 1 ∂ 1 ∂ (3.14) + ψε δ n+1 τ µ = ψε δ n+1 τ µQε δ τ µ ε ∂τ δ ∂µ Taking Taylor expansions of Qε δ t about t = 0, n+1
t k d k Q0 + Rn+1 t k k! dt k=0 n+1 k k −1 t k d k A0 t k d k Q0 −1 t d B0 = ε + Rn+1 t + δ + k k k k! dt k! dt k! dt k=0 n+1 k−1 t/εk d k A0 t/δk d k B0 = ε + δk−1 k k! dt k! dt k k=0 t/δk d k Q0 + Rn+1 t + δk k! dt k
Qε δ t =
292
liu, yin, and zhang
where Rn+1 t = Ot n+2 uniformly for t ∈ 0 T . Using τ and µ, we can write the above equation as Qεδ τ µ =
n+1
εk−1
k=0
τk d k A0 µk d k B0 µk d k Q0 + δk−1 + δk k k k! dt k! dt k! dt k
+ Rn+1 τ µ Dropping the term Rn+1 τ µ in Qεδ t, it follows from (3.14) ∂ ∂ ψε δ n+1 τµ ε−1 +δ−1 ∂τ ∂µ n+1 k k k k k−1 τk d k A0 k−1 µ d B0 k µ d Q0 ε +δ +δ = ψε δ n+1 τµ k! dt k k! dt k k! dt k k=0 That is,
n+1
ε
−1
i+j=0
=
n+1
∂ −1 ∂ +δ εi δj ψi j τ µ ∂τ ∂µ
εi δj ψi j τ µ
k=0
×
n+1
ε
i+j=0
k k k k d k A0 k−1 µ d B0 k µ d Q0 +δ +δ k! dt k k! dt k k! dt k
k−1 τ
k
Equating the coefficients of εi δj , we have for i + j = 0, ε−1 δ0
ε0 δ−1
ε 0 δ0
∂ψ0 0 τ µ = ψ0 0 τ µA0 ∂τ ∂ψ0 0 τ µ = ψ0 0 τ µB0 ∂µ ∂ψ1 0 τ µ ∂ψ0 1 τ µ + ∂τ ∂µ = ψ0 0 τ µDτ µ + ψ1 0 τ µA0 + ψ0 1 τ µB0
ε1 δ−1
∂ψ1 0 τ µ = ψ1 0 τ µB0 ∂µ
ε−1 δ1
∂ψ0 1 τ µ = ψ0 1 τ µA0 ∂τ
(3.15)
singularly perturbed markov chains
293
and for 1 ≤ i + j ≤ n, ε2 δ−1
∂ψ2 0 τ µ = ψ2 0 τ µB0 ∂µ
ε 1 δ0
∂ψ2 0 τ µ ∂ψ1 1 τ µ + ∂τ ∂µ = ψ0 0 τ µ
d 2 A0 2 τ + ψ1 0 τ µDτ µ dt 2
+ ψ2 0 τ µA0 + ψ1 1 τ µB0 ε 0 δ1
∂ψ1 1 τ µ ∂ψ0 2 τ µ + ∂τ ∂µ = ψ0 0 τ µ
(3.16)
d 2 B0 2 µ + ψ0 1 τ µDτ µ dt 2
+ ψ1 1 τ µA0 + ψ0 2 τ µB0 ε−1 δ2
∂ψ0 2 τ µ = ψ0 2 τ µA0 ∂τ
where
dA0 dB0 +µ + Q0 dt dt We will construct ψij · · by solving (3.15) and (3.16). Dτ µ = τ
Theorem 3.5. Assume (A1) and (A2). Then there exist continuously differentiable solutions ψi j t. 0 ≤ i + j ≤ n for the systems (3.15) and (3.16). The solution to (3.15) can be expressed as
ψ0 0 τ µ = p0 − ϕ0 0 0 expA0τ + B0µ ψ1 0 τ µ = H 1 0 τ expA0τ + B0µ
(3.17)
ψ0 1 τ µ = H 0 1 µ expA0τ + B0µ where the row vectors H 1 0 τ and H 0 1 µ are given by τ dA0 H 1 0 τ = p0 − ϕ0 0 0 expA0s dt 0 × exp−A0ss ds − ϕ1 0 0 H
0 1
dB0 1 µ = µ2 p0 − ϕ0 0 0 2 dt 0 0 0 − ϕ0 1 0 + µp − ϕ 0Q0
(3.18)
For 2 ≤ i + j ≤ n + 1, the functions ψij · · can be expressed similarly.
294
liu, yin, and zhang
Proof. Construction of ψij τ µ for 0 ≤ i + j ≤ 1: Let us determine τ µ from the first two equations of (3.15). First, ψ0 0 τ µ can be ψ taken as 0 0
ψ0 0 τ µ = C 0 0 expA0τ + B0µ
(3.19)
where C 0 0 is a constant vector to be determined by the initial value. From the first equation of (3.15), we have ψ0 0 τ µ = C1 µ expA0τ
(3.20)
where C1 µ is a vector-valued function. To determine C1 µ, substituting (3.20) into the second equation of (3.15), we obtain that dC1 µ expA0τ = C1 µ expA0τB0 dµ Since A0 and B0 commute, expA0τB0 = B0 expA0τ, and dC1 µ expA0τ = C1 µB0 expA0τ dµ
(3.21)
Note that expA0τ is invertible for any matrix A0 and its inverse is exp−A0τ. Postmultiplying exp−A0τ to both sides of (3.21), we ob1 µ tain dCdµ = C1 µB0 It follows that C1 µ = C 0 0 expB0µ, where C 0 0 is a constant vector. Substituting C1 µ into (3.20), ψ0 0 τ µ = C 0 0 expB0µ expA0τ. Note that expB0µ expA0τ = expA0τ + B0µ. It follows that ψ0 0 τ µ = C 0 0 expA0τ + B0µ. Thus (3.19) is obtained. To solve for ψ1 0 τ µ and ψ0 1 τ µ, start with the fourth and fifth equations in (3.15). Assume that the solutions can be written as ψ1 0 τ µ = H 1 0 τ expA0τ + B0µ ψ0 1 τ µ = H 0 1 µ expA0τ + B0µ where H 1 0 · and H 0 1 · are vector-valued functions to be determined later. As in the construction of the outer expansion terms, H 1 0 · and H 0 1 · need to be determined by the use of the third equation in (3.15). In fact, dH 1 0 τ dH 0 1 µ + expA0τ + B0µ dτ dµ = τC 0 0 expA0τ + B0µ + B0µ
dA0 + µC 0 0 expA0τ dt
dB0 + C 0 0 expA0τ + B0µQ0 dt
singularly perturbed markov chains
295
We separate the above equation into two equations dA0 dH 1 0 τ expA0τ + B0µ = τC 0 0 expA0τ + B0µ dτ dt dB0 dH 0 1 µ expA0τ + B0µ = µC 0 0 expA0τ + B0µ dµ dt + C 0 0 expA0τ + B0µQ0 Note that expA0τ + B0µ is invertible and its inverse is exp−A0τ − B0µ. By postmultiplying exp−A0τ − B0µ to both sides of the above two equations, we have dH 1 0 τ dA0 exp−A0τ − B0µ = τC 0 0 expA0τ + B0µ dt dτ dB0 dH 0 1 µ = µC 0 0 expA0τ + B0µ exp−A0τ − B0µ dµ dt exp−A0τ − B0µ + C 0 0 expA0τ + B0µQ0 In addition, the following communitivity holds dA0 dA0 expB0µ = expB0µ dt dt dB0 dB0 expA0τ + B0µ = expA0τ + B0µ dt dt expA0τ + B0µ = expA0τ + B0µQt Qt It follows that dH 1 0 τ dA0 = τC 0 0 expA0τ exp−A0τ dτ dt dB0 dH 0 1 µ = µC 0 0 + C 0 0 Q0 dµ dt The solutions to the above equations are τ dA0 H 1 0 τ = C 0 0 exp−A0ss ds + C 1 0 expA0s dt 0 (3.22) 1 2 0 0 dB0 0 1 0 0 0 1 Q0 + C + µC µ = µ C H 2 dt where C 1 0 and C 0 1 are constant row vectors to be determined by the initial data. Choosing matched (between outer and inner terms) initial conditions leads to ϕ0 0 0 + ψ0 0 0 0 = p0
ϕ1 0 0 + ψ1 0 0 0 = 0
ϕ0 1 0 + ψ0 1 0 0 = 0
296
liu, yin, and zhang
The first equation together with (3.19) gives C 0 0 = p0 − ϕ0 0 0 and the second and third equations together with (3.22) imply that C 1 0 = −ϕ1 0 0, and C 0 1 = −ϕ0 1 0. Substituting C 0 0 , C 1 0 , and C 0 1 into (3.22), we get
τ dA0 expA0s H 1 0 τ = p0 − ϕ0 0 0 dt 0 × exp−A0ss ds − ϕ1 0 0
dB0
1 H 0 1 µ = µ2 p0 − ϕ0 0 0 + µ p0 − ϕ0 0 0 Q0 2 dt − ϕ0 1 0 This completes the construction of ψij · · for 0 ≤ i + j ≤ 1. The construction of the solutions for 2 ≤ i + j ≤ n + 1 is similar. We omit the details. 3.2.2. Exponential Decay of ψi j · · We obtained the asymptotic series ψi j · · in Theorem 3.5. In this section, we will verify the exponential decay properties of the solutions. Similar to Lemma A.2 in [15, p.300], we have the following result. Lemma 3.6.
Suppose that assumptions (A1) and (A2) are satisfied.
(a)
Suppose Ps is the solution to the differential equation: dPs = PsA0 P0 = I ds A as s → ∞ and Then Ps → P expA0s − P A ≤ KA exp−κA s for some κA > 0
(3.23) (3.24)
A = diag|m ν0 |m ν0 and ν0 = ν1 0 νm 0 ∈ where P 0 0 0 1×m0 is the quasi-stationary distribution corresponding to the generator Q0. (b) Suppose Ps is the solution to the differential equation: dPs = PsB0 P0 = I (3.25) ds B as s → ∞ and Then Ps → P expB0s − P B ≤ KB exp−κB s for some κB > 0 (3.26) where
λ1 0Im0 λl 0Im0 B = (3.27) P λ1 0Im0 λl 0Im0 and λ0 = λ1 0 λl 0 is the quasi-stationary distribution of the gen erator Q0 = q¯ ij 0.
singularly perturbed markov chains
297
Proof. To prove (a), note that |m0 ν0 has identical rows ν1 0 νm0 0. In view of the block-diagonal structure of A0, we have Ps = expA0s = diagexpQ0s expQ0s By using Lemma A.2 in [15, p. 300], it is readily seen that there exist − |m0 ν0 ≤ constants KA > 0 and κA > 0 such that expQ0s A ≤ KA exp−κA s. Thus Ps → PA as s → ∞ and expA0s − P KA exp−κA s. The first result is proved. To prove (b), note that after some rows and columns are interchanged, B0 becomes the block diagonal form B∗ 0 = diagQ0 Q0. That m×m such that RB0R−1 = is, there is an invertible constant matrix R ∈ B∗ 0 Similar to the first part of this lemma, we have def B expB∗ 0s → diag|l λ0 |l λ0 ∈ m×m = P
as s → ∞
where diag|l λ0 |l λ0 is a block diagonal matrix with m0 blocks. Consequently, R−1 expB∗ 0sR → R−1 diag|l λ0 |1 λ0R
as s → ∞
Note that R−1 expB∗ 0sR = expR−1 B∗ 0Rs = expB0s B as B has the form given by (3.27). Thus we have expB0s → P and P s → ∞ with the desired convergence rate. This completes the proof. Lemma 3.7. Suppose that assumptions (A1) and (A2) are satisfied. Let PA and PB be defined as in Lemma 3.6. Then there exist constants KA > 0, B ≤ A P KB > 0, κA > 0, and κB > 0 such that expA0s + B0t − P KA exp−κA s + KB exp−κB t. Proof. Notice the fact that At and Bt commute. We start with the inequality: expA0s + B0t − P B A P B + expA0sP B B − P A P = expA0s + B0t − expA0sP B + expA0s − P A P B = expA0sexpB0t − P B + expA0s − P A ≤ expA0sexpB0t − P PB > 0, KB > 0, κA > 0, and κB > 0 By Lemma 3.6, there exist constants KA such that expA0s − P A ≤ KA exp−κA s and expB0t − P B ≤ KB exp−κB t
298
liu, yin, and zhang
B ≤ MB for some constant MB > 0 and expA0s ≤ MA for Since P some constant MA > 0, we have expA0s + B0t − P B ≤ MA KB exp−κB t + MB KA A P exp−κA s ≤ KB exp−κB t + KA exp−κA s This completes the proof. Now we are in a position to state the result about the exponential decay of ψij · ·. Theorem 3.8. Under the conditions of Theorem 3.1, for each i, j with 0 ≤ ij ij i + j ≤ n + 1, there exist polynomials cA τ µ and cB τ µ, and positive numbers κA and κB such that ij ψ τ µ ≤ c ij τ µ exp−κA τ + c ij τ µ exp−κB µ (3.28) B A
m 0 0
p0i = 1 and 0 = 1. It follows that Proof. Note that m i=1 ϕi
m 0 0
i=1
0 0 0 0 m m 0 0 0 = 0 = 0 That is, ψi 0 0 i=1 ψi i=1 pi − i=1 ϕi A and P B be defined in Lemma 3.6. is orthogonal to |m . Let P Recall the forms of PA and PB . Then λ1 0|m0 π0 λl 0|m0 π0 B = A P P λ1 0|m0 π0 λl 0|m0 π0 = |m λ1 0π0 λl 0π0 = |m πλ 0 B = ψ0 0 0 0|m πλ 0 = 0 i.e., A P As a Consequence, ψ0 0 0 0P ψ0 0 0 is orthogonal to πλ 0. By virtue of Lemma 3.7, 0 0 ψ τ µ = ψ0 0 0 0 expA0τ + B0µ A P A P B − ψ0 0 0 0P B = ψ0 0 0 0P + ψ0 0 0 0 expA0τ + B0µ B A P ≤ ψ0 0 0 0P A P B +ψ0 0 0 0expA0τ + B0µ − P B A P = ψ0 0 0 0expA0τ + B0µ − P ≤ ψ0 0 0 0KA exp−κA τ + KB exp−κB µ A exp−κA τ + K B exp−κB µ ≤K That is, ψ0 0 τ µ decays exponentially fast.
(3.29)
singularly perturbed markov chains
299
Next consider ψ0 1 τ µ. Recall that ψ0 1 τ µ = H 0 1 µ expA0 τ + B0µ, where
dB0
1 01 H 01 µ = µ2 p0 −ϕ00 0 0 +µ p0 −ϕ00 0 Q0−ϕ 2 dt
0 1 Note that ϕ0 1 0| = m 0 = 0 Since Q0 and B(0) are generi=0 ϕi dB0 dB0|m ators, Q0|m = 0, and dt |m = = 0. Thus we obtain dt 1 2 0 dB0 µ p − ϕ0 0 0 H 0 1 µ|m = 2 dt 0
0 0 +µ p − ϕ 0 Q0 − ϕ0 1 0|m =
dB0 1 2 0 µ p − ϕ0 0 0 |m 2 dt
0 1 + µ p0 − ϕ0 0 0 Q0| 0|m = 0 m−ϕ
i.e., H 0 1 µ is orthogonal to |m . Hence, 01 ψ 00 = H 01 µexpA0τ +B0µ
B +H 01 µ expA0τ +B0µ− P B A P A P = H 01 µP
A P B = H 01 µ expA0τ +B0µ− P A P B ≤ H 01 µ expA0τ +B0µ− P = H 01 µKA exp−κA τ+KB exp−κB µ 01
≤ cA 0 1 cA µ
01
µexp−κA τ+cB
µexp−κB µ
0 1 cB µ
and are polynomials of degree 2. Thus, ψ0 1 τ µ where also decays exponentially fast. Next we show the exponential decay of ψ1 0 τ µ. Recall that ψ1 0 τ µ = H 1 0 τ expA0τ + B0µ where
τ dA0 H 1 0 τ = p0 − ϕ0 0 0 expA0s exp−A0ss ds dt 0 − ϕ1 0 0
Since ϕ1 0 0 is orthogonal to |m , and ∞ dA0 −1n sn dA0 n exp−A0s| = A0 |m dt dt n! n=0 =
∞ −1n sn dA0 A0n |m dt n=0 n!
300
H
liu, yin, and zhang
1 0
=
s s2 dA0 Im |m − A0|m + A02 |m +··· dt 1! 2!
=
dA0 dA0 Im |m = |m = 0 dt dt
τ|m = p0 − ϕ0 0 0
τ
0
expA0s
dA0 exp−A0ss ds dt
− ϕ1 0 0 |m
τ dA0 exp−A0s|s ds = p0 − ϕ0 0 0 expA0s dt 0 − ϕ1 0 0|m = 0 So we have shown that H 1 0 τ is orthogonal to |m . Thus we have 1 0 ψ 0 0 = H 1 0 µ expA0τ + B0µ B A P = H 1 0 µexpA0τ + B0µ − P B A P ≤ H 1 0 µ expA0τ + B0µ − P = H 1 0 µKA exp−κA τ + KB exp−κB µ 1 0
≤ cA 1 0
1 0
µ exp−κA τ + cB
µ exp−κB µ
1 0
where cA µ and cB µ are polynomials of degree 2. The proof for the cases of 2 ≤ i + j ≤ n + 1 are similar to the above. Hence we omit the details. The theorem is proved. 1 0 1 0 Since the growth of cA µ and cB µ are much slower than exponential, we have the following two corollaries. Corollary 3.9. For 0 ≤ i + j ≤ n + 1, we have ψi j τ µ ≤ KA exp−κA + KB exp−κB Corollary 3.10. Suppose there exist constants h1 > 0 and h2 > 0 such that h1 ε ≤ δ ≤ h2 ε. Then for any integers 0 ≤ i + j ≤ n + 1, τψi j τ µ ≤ K1 , and µψi j τ µ ≤ K2 for some constants K1 > 0 and K2 > 0. 4. ERROR ANALYSIS In this section we will validate the asymptotic expansion. Recall that the operator Lε δ is defined by (2.5). We first prove a lemma.
singularly perturbed markov chains
301
ε δ ε δ Lemma 4.1. Suppose that for some 0 ≤ k ≤ n+ 1, sup t∈ 0 T kL kv k k ε δ ε δ t = Oε + δ and v 0 = 0. Then supt∈ 0T v t ≤ Oε + δ . Proof. Let ηε δ t be a vector-valued function satisfy supt∈ 0 T ηε δ tOεk + δk Consider the differential equation
Lε δ vε δ t = ηε δ t vε δ 0 = 0
(4.1) ε δ
the solution of the above equation is given by v t = Now t ε δ η sX ε δ t s ds, where X ε δ t s is a principal matrix solution. 0 Since X ε δ t s is a transition probability matrix, X ε δ t s ≤ K, for all t s ∈ 0 T . Therefore, we have the inequality t ηε δ s ds ≤ Kεk + δk sup vε δ t ≤ K sup t∈ 0T 0
t∈ 0T
This completes the proof. In view of (3.1), eε δ k t is defined as eε δ k t = pε δ t − ϕε δ k t − ψ t/ε t/δ where pε δ · is the solution to (2.2), and ϕε δ k · and ε δ k ψ · · are constructed in previous sections. We will estimate the error term eε δ n t. We have the following result. ε δ k
Proposition 4.2. Assume (A1) and (A2). Suppose there exist constants h1 > 0 and h2 > 0 such that h1 ε ≤ δ ≤ h2 ε. Then, for 0 ≤ i ≤ n, supt∈ 0T eε δ i t = Oεi+1 + δi+1 . Proof.
We first prove the result for 1
eε δ 1 t = pε δ t −
1
εi δj ϕi j t −
i+j=0
εi δj ψi j
t
i+j=0
t ε δ
ε δ 1
0 = 0, and hence the condition of Lemma 4.1 It is easy to see that e on the initial data holds. By the definition of Lε δ , Lε δ pε δ t = 0. Consequently, we have 1 i j i j Lε δ eε δ1 t = Lε δ pε δ t − Lε δ ε δ ϕ t i+j=0
1
− Lε δ
εi δj ψi j
i+j=0
= −L
=−
1
ε δ
i j
εδϕ
i j
1 i+j=0
i j
εδϕ
t − L
i+j=0
d dt
t t ε δ
i j
t −
ε δ
1 i+j=0
1 i+j=0
i j
εδψ
i j
t t ε δ
εi δj ϕi j tQε δ t
302
liu, yin, and zhang 1 d i j i j t t − εδψ dt i+j=0 ε δ −
1 i+j=0
i j
εδψ
i j
t t ε δ Q t ε δ
Based on the smoothness of ϕi j · on 0 T and the defining equation (3.4), 1 1 d i j i j ε δ ϕ t − εi δj ϕi j tQε δ t dt i+j=0 i+j=0 1 0 0 1 dϕ t t dϕ =ε − ϕ1 0 tQt − ϕ0 1 tQt +δ dt dt = Oε + δ Now let us estimate the terms containing ψi j · ·. Recall that 1 1 dA0 dB0 +µ + Q0 Qε δ t = A0 + B0 + τ ε δ dt dt + Oετ2 + δµ2 + δµ where τ = t/ε and µ = t/δ. According to the exponential decay property of ψi j · · and the defining equations (3.15) and (3.16), 1 1 t t d i j i j t t − Qε δ t εδψ εi δj ψi j dt i+j=0 ε δ ε δ i+j=0 1 1 d t t i j i j t t = εδψ εi δj ψi j − dt i+j=0 ε δ ε δ i+j=0 1 1 dA0 dB0 × A0 + B0 + τ +µ + Q0 + Oετ + δµ ε δ dt dt dB0 dA0 = −εψ1 0 τ µ + δψ0 1 τ µ τ +µ + Q0 dt dt −
1
εi δj ψi j τ µOετ2 + δµ2 + δµ
i+j=0
Based on the exponential decay of ψi j · · and Corollary 3.10, we obtain 1 1 t t d i j i j t t − Qε δ t = Oε + δ εδψ εi δj ψi j dt i+j=0 ε δ ε δ i+j=0
singularly perturbed markov chains
303
Thus we have proved that Lε δ eε δ 1 t = Oε + δ uniformly in 0 T . By Lemma 4.1, we have eε δ 1 t = Oε + δ uniformly in 0 T . Next we show the result holds for i = j = 0. Note that t t eε δ 1 t = eε δ 0 t − εϕ1 0 t − δϕ0 1 t − εψ1 0 ε δ t t − δψ0 1 ε δ By Corollary 3.10 we have εϕ
1 0
t + δϕ
0 1
t + εψ
1 0
t t ε δ
+ δψ
0 1
t t ε δ
= Oε + δ
uniformly in t ∈ 0 T . Thus eε δ 0 t = Oε + δ uniformly in t ∈ 0 T . Similarly, we can estimate eε δ n t. In fact, from n+1 n+1 i j i j i j i j t t eε δ n+1 t = pε δ t − ε δ ϕ t − εδψ ε δ i+j=0 i+j=0 we see that eεδn+1 0 = 0, and L
εδ εδn+1
e
t = L
εδ
p
εδ
εδ
= −L
t−
n+1
i j
εδϕ
i+j=0 n+1 i+j=0
i j
εδϕ
ij
ij
n+1
t−
εδψ
ij
i j
ij
i+j=0
t −L
i j
εδ
n+1
εδψ
i+j=0
t t ε δ
t t ε δ
n+1 i j ij i j ij d n+1 =− ε δ ϕ t − ε δ ϕ tQεδ t dt i+j=0 i+j=0 i j ij t t d n+1 − εδψ dt i+j=0 ε δ n+1 i j ij t t εδ − εδψ Q t ε δ i+j=0 It follows from the defining equations of ϕi j ·, n+1 n+1 i j i j i j i j d ε δ ϕ t − ε δ ϕ tQε δ t dt i+j=0 i+j=0 1 0 t dϕ = εn+1 − ϕ1 0 tQt dt
304
liu, yin, and zhang
dϕn 1 t − ϕn 1 tQt dt 0 n+1 t dϕ − ϕ0 n+1 tQt + · · · + δn+1 dt Based on the smoothness of ϕi j · on 0 T and the assumption h1 ε ≤ δ ≤ h2 ε, we have n+1 n+1 i j i j i j i j
d ε δ ϕ t − ε δ ϕ tQε δ t = O εn+1 + δn+1 dt i+j=0 i+j=0 + εn δ
uniformly in t. Now we estimate the terms containing ψi j · ·. According to the exponential decay property and the expansion of Qε δ · ·, n+1 n+1 i j i j t t i j i j t t d εδψ εδψ − Qεδ t dt i+j=0 ε δ ε δ i+j=0 n+1 n+1 i j i j t t i j i j t t d = − εδψ εδψ dt i+j=0 ε δ ε δ i+j=0 n+1
k k k k d k A0 k−1 µ d B0 k µ d Q0 × ε + δ + δ k! dt k k! dt k k! dt k k=0 n+1 i j i j t t − εδψ Oεn+1 τn+2 + δn+1 µn+2 ε δ i+j=0
= Oε
n+1
k−1 τ
+δ
k
n+1
+
n+1 i+j=0
i j
εδψ
i j
t t O εn+1 τn+2 + δn+1 µn+2 ε δ
= Oεn+1 + δn+1 so we have Lε δ eε δ n+1 t = Oεn+1 + δn+1 uniformly in 0 T . By Lemma 4.1, we have eε δ n+1 t = Oεn+1 + δn+1 uniformly in 0 T . Finally, from the expression of eε δ n+1 t, we have eε δ n+1 t = eε δ n t + Oεn+1 + δn+1 . This implies that eε δ n t = Oεn+1 + δn+1 uniformly in t. This completes the proof of the corollary and hence the proof of Theorem 3.1. 5. FURTHER REMARKS 5.1. Asymptotic Expansions for ε/δ = o1 In the previous sections we have constructed expansions for Markov chains with generators that have two small independent parameters such
singularly perturbed markov chains
305
that ε/δ is bounded. In this section, we consider the case ε/δ = o1; i.e., ε goes to zero much faster than δ. To be more specific, let ε = δ2 . Suppose that the generator is of the form (2.3) with ε = δ2 , where At, Bt, and Qt are as before. We seek asymptotic expansion of the form δ δ n p t = ϕ t + ψδ n t/δ2 + eδ n t, where the regular part the initial layer corrections are n n t t δi ϕi t ψδ n 2 = δi ψi 2 ϕδ n t = δ δ i=0 i=0 respectively, and eδ n t is the remainder. 5.1.1. Outer Expansion dϕδ n t dt
= ϕδ n tQδ t; that is, n n i 1 1 i dϕ t i i = δ δ ϕ t 2 At + Bt + Qt dt δ δ i=0 i=0
Consider the differential equation
Equating the coefficients of δi , we have the following system of equations δ−2
0 = ϕ0 tAt
δ−1
0 = ϕ1 tAt + ϕ0 tBt
δ0
δ1
dϕ0 t = ϕ2 tAt + ϕ1 tBt + ϕ0 tQt dt dϕ1 t = ϕ3 tAt + ϕ2 tBt + ϕ1 tQt dt
(5.1)
It is easy to see that ϕ0 t = πλ t = λ1 tπt λl tπt where λ· and π· are as defined previously. In addition, we have, for any t ∈
0 0 T , m i=1 ϕi t = 1. To find the solution ϕ1 t, we first notice that ϕ0 tBt = πλ tBt = 0. Now from the second equation of (5.1), we have ϕ1 tAt = 0 Hence the vector-valued function ϕ1 t = x1 tπt xl tπt is a solution to the second equation in (5.1), where xt ∈ 1×l is any vector-valued
function that satisfies the condition li=1 xi t = 0. To obtain ϕ2 t, we examine the third equation in (5.1). By substituting 0 ϕ t and ϕ1 t into the equation, we have dλ1 t dλl t dπt dπt πt πt + λ1 t λ1 t dt dt dt dt = ϕ2 tAt + x1 tπt xl tπtBt + λ1 tπt λl tπtQt
(5.2)
306
liu, yin, and zhang
To solve (5.2), it suffices to solve the following two equations derived from it, dπt dπt λ1 t λ1 t = ϕ2 tAt and (5.3) dt dt
dλ1 t dλ t πt l πt = x1 tπt xl tπtBt dt dt + λ1 tπt λl tπtQt (5.4)
We first look at (5.4). It can be reduced to
dλ1 t dλ t l dt dt
= x1 t xl tq¯ ij t + λ1 t λl ttqˆ ij t
Note that dλdt1 t dλdtl t |l = 0 and qˆ ij t|l = 0 By the weak irreducibility of q¯ ij t, we can uniquely solve the equation for xt with the
1 condition li=1 xi t = 0. The last equality implies m i=1 ϕi t = 0. Sim 2 ilarly, we can solve (5.3) for ϕ2 t so that m i=1 ϕi t = 0. For i ≥ 3, ϕi t can be obtained similarly. 5.1.2. Initial Layer Correction To construct the boundary layer terms, we consider another time scale τ = t/δ2 Consider the differential equation dψεn t/δ2 1 1 = ψεn t/δ2 2 At + Bt + Qt (5.5) dt δ δ Taking Taylor expansion of Qδ t at t = 0, we have n+1 k k k k τk d k A0 2k−1 τ d B0 2k τ d Q0 Qδ δ2 τ = δ2k−2 + δ + δ k! dt k k! dt k k! dt k k=0 + Rn+1 δ2 τ where Rn+1 t = Ot n+2 . Drop the term Rn+1 δ2 τ and substitute the rest of the terms into (5.5), n+1 n+1 i i i i d n+1 τk d k A0 δ2k−2 δ ψ τ = δ ψ τ dτ i=0 k! dt k i=0 k=0 k k µk d k B0 k µ d Q0 + δ2k−1 + δ k! dt k k! dt k
singularly perturbed markov chains
307
Equating the coefficients of δi , we obtain d 0 ψ τ = ψ0 τA0 dτ d 1 ψ τ = ψ1 τA0 + ψ0 τB0 dτ d 2 ψ τ = ψ2 τA0 + ψ1 τB0 + ψ0 τ dτ dA0 × τ + Q0 dt
(5.6)
The solutions to (5.6) are ψ0 τ = ψ0 0 expA0τ ψ1 t = ψ1 0 expA0τ τ + ψ0 sBs expA0τ − s ds 0
2
2
ψ t = ψ 0 expA0τ τ + ψ1 sBs + ψ0 sDs
(5.7)
0
× expA0τ − s ds + Q0. The initial values of ψi 0 satisfy ψ0 0 = where Ds = s dA0 dt p0 − ϕ0 0, ψi 0 = −ϕi 0, for i ≥ 1 A = diag|m ν0 |m ν0. Similar to the previous secDenote P 0 0 A ≤ KA exp−κA s for some κA > 0 In addition, tions, expA0s − P we have the following exponential decay properties of ψi ·. The proof of the following proposition is similar to that of Lemma 3.7 and is thus omitted. Proposition 5.1. For each i = 0 1 n + 1, there exist a polynomial c i τ and positive number κ such that ψi τ ≤ c i τ exp−κτ. By virtue of Proposition 5.1, we have for any i = 1 n + 1, τk ψi τ ≤ K for some K > 0 and i = 1 n + 1. 5.1.3. Asymptotic Validation Now we give the estimate for the error term eδ n t = pδ t −
n
n i i 2 i=0 ϕ t − i=0 ψ t/δ The next proposition is the asymptotic property of the expansion.
308
liu, yin, and zhang
Proposition 5.2. Assume (A1) and (A2). Then, for 0 ≤ i ≤ n, supt∈ 0 T eδ i t = Oδi+1 . The proof of the proposition follows from Proposition 3.7 and the proof of Proposition 5.1. Thus we omit the details here. As a corollary of the above proposition, we have limδ→0 pδ t = ϕ0 t = πλ t. 5.2. Concluding Remarks In this paper, we have developed asymptotic expansions of the probability vector. The results obtained will be useful for many optimal control problems that involve singularly perturbed Markovian models with multiple time scales. The choice of the generators At and Bt is largely motivated by the applications in control and optimization of manufacturing systems. If At and Bt are both weakly irreducible (i.e., consisting of one block), the asymptotic expansion similar to the development discussed in this paper can be obtained. Such a treatment can be used for certain reducible matrices having different forms than those of this paper. However, if At and Bt have different partitions in the most general form, it appears that the construction of the asymptotic expansion cannot be done as here since the algebraic and differential equations involved may not be consistent. Future study can be directed to the investigation of further probabilistic properties of the model. REFERENCES 1. A. Bensoussan, “Perturbation Methods in Optimal Control,” Wiley, Chichester, 1988. 2. M. H. A. Davis, “Markov Models and Optimization,” Chapman & Hall, London, 1993. 3. F. Delebecque and J. Quadrat, Optimal control for Markov chains admitting strong and weak interactions, Automatica 17 (1981), 281–296. 4. J. K. Hale, “Ordinary Differential Equations,” 2nd ed., Krieger, Malabar, 1980. 5. S. N. Ethier and T. G. Kurtz, “Markov Processes: Characterization and Convergence,” Wiley, New York, 1986. 6. R. Z. Khasminskii, G. Yin, and Q. Zhang, Asymptotic expansions of singularly perturbed systems involving rapidly fluctuating Markov chains, SIAM J. Appl. Math. 56 (1996), 277–293. 7. R. Z. Khasminskii, G. Yin, and Q. Zhang, Constructing asymptotic series for probability distribution of Markov chains with weak and strong interactions, Quart. Appl. Math. LV (1997), 177–200. 8. H. J. Kushner, “Approximation and Weak Convergence Methods for Random Processes, with Applications to Stochastic Systems Theory,” MIT Press, Cambridge, MA, 1984. 9. Z. G. Pan and T. Bas¸ar, H ∞ -control of Markovian jump linear systems and solutions to associated piecewise-deterministic differential games, in “New Trends in Dynamic Games and Applications” (G. J. Olsder, Ed.), pp. 61–94, Birkh¨auser, Boston, 1995. 10. A. A. Pervozvanskii and V. G. Gaitsgori, “Theory of Suboptimal Decisions: Decomposition and Aggregation,” Kluwer, Dordrecht, 1988.
singularly perturbed markov chains
309
11. R. G. Phillips and P. V. Kokotovic, A singular perturbation approach to modelling and control of Markov chains, IEEE Trans. Automat. Control 26 (1981), 1087–1094. 12. S. P. Sethi and Q. Zhang, “Hierarchical Decision Making in Stochastic Manufacturing Systems,” Birkh¨auser, Boston, 1994. 13. H. A. Simon and A. Ando, Aggregation of variables in dynamic systems, Econometrica 29 (1961), 111–138. 14. W. A. Thompson, Jr., “Point Process Models with Applications to Safety and Reliability,” Chapman & Hall, New York, 1988. 15. G. Yin and Q. Zhang, “Continuous-Time Markov Chains and Applications: A Singular Perturbation Approach,” Springer-Verlag, New York, 1998.
Singularly Perturbed Markov Chains with Two Small Parameters: A Matched Asymptotic Expansion Q. G. Liu and G. Yin1 Department of Mathematics, Wayne State University, Detroit, Michigan 48202
and Q. Zhang2 Department of Mathematics, University of Georgia, Athens, Georgia 30602 Submitted by S. P. Sethi Received March 31, 2000
This work is concerned with asymptotic properties of solutions to forward equations for singularly perturbed Markov chains with two small parameters. It is motivated by the model of a cost-minimizing firm involving production planning and capacity expansion and a two-level hierarchical decomposition. Our effort focuses on obtaining asymptotic expansions of the solutions to the forward equation. Different from previous work on singularly perturbed Markov chains, the inner expansion terms are constructed by solving certain partial differential equations. The methods of undetermined coefficients are used. The error bound is obtained. 2002 Elsevier Science (USA) Key Words: manufacturing system; singularly perturbed Markov chain; multiple scale; forward equation; asymptotic expansion.
1. INTRODUCTION Much of the current interest stems from the motivation of various applications in manufacturing systems. In such applications, one aims to 1 This research was supported in part by the National Science Foundation under Grant DMS-9877090. 2 The research of this author was supported in part by the Office of Naval Research Grant N00014-96-1-0263.
281 0022-247X/02 $35.00 2002 Elsevier Science (USA)
All rights reserved.
282
liu, yin, and zhang
find the optimal production-planning strategies and to use Markov jump structures in the modeling of uncertainty (e.g., machine capacity). In many applications, one often needs to deal with large-scale systems. Even for a seemingly not so complex system, the optimal control can be very difficult to obtain (see [12, Sect. 5.2]), not to mention the added difficulty of a very large dimensional state space involved. In fact, not only is it an impossible task to obtain analytic solutions, but also the direct numerical approximation can be overwhelming and the straightforward use of numerical methods may not be adequate. Naturally, one seeks to break the large job into small pieces with the hope that certain decompositions and aggregations will lead to a simplification of the intractable systems; see Simon and Ando [13]. A viable alternative calls for taking advantage of the high contrast rates (some states vary an order of magnitude faster than the rest) of changes in the physical systems and using a singular perturbation approach as a tool to reduce the complexity of the underlying systems. To have a thorough understanding of the problems, it is of the utmost importance to learn the intrinsic structures and asymptotic properties of the underlying Markov chains. Continuing effort has been devoted to studying singularly perturbed Markov chains; see [3, 9–12] among others. Recently, in [6], Khasminskii et al. used matched asymptotic expansion to establish the convergence of a sequence of the probability vectors. Singularly perturbed Markov chains with recurrent states, naturally divisible into a number of classes, are then treated in [7]. This line of work has been continued in [15], in which we have further derived asymptotic expansions for Markov chains with the inclusion of transient states and absorbing states, and asymptotic distributions for Markov chains having recurrent states. In this work, we examine a continuous-time model with two small parameters. The use of multiple small parameters stems from consideration of hierarchical structures of various stochastic systems in manufacturing as well as in communication networks. The motivation comes from a production-marketing system (see [Chap. 11]). The underlying problem is concerned with a cost-minimizing firm involving production planning and capacity expansion and a two-level hierarchical decomposition. In [12], Sethi and Zhang have found the asymptotic optimal strategies by using the limit distribution of the Markov chains. In this work, our main effort is devoted to obtaining asymptotic properties of the solutions to the forward equation satisfied by the probability vector. In previous work on singularly perturbed Markov chains, the asymptotic expansions are obtained by solving appropriate algebraic and ordinary differential equations, whereas in this paper, partial differential equations are also involved. The results are useful in the subsequent studies on Markov decision processes and controlled Markovian systems involving multiple small parameters. To proceed, let us begin with an example of the manufacturing system.
singularly perturbed markov chains
283
Example 1.1. Consider a production-marketing system that produces a single product type using a two-state production capacity αt ∈ 0 1. Let ut denote the production rate subject to the production constraints 0 ≤ ut ≤ αt. Let xt ∈ 1 denote the total surplus and let zt ∈ z1 z2 denote a two-state stochastic demand rate. The system is given by xt ˙ = ut − zt x0 = x 1
(1.1)
1
where x ∈ is the initial surplus. Let wt ∈ denote the rate of advertising satisfying 0 ≤ wt ≤ K, with K representing an upper bound on the advertising rate. The profit functional J· is defined by Jx α z u· w· ∞ =E e−ρt πzt − h1 xt + cut + wt dt 0
(1.2)
where ρ > 0 is the discount rate, π is the revenue per unit sale, h1 · is the surplus cost function, and c < π is the unit production cost. The problem is to find a control ut wt t ≥ 0, that maximizes Jx α z u· w·. Let = 0 z1 1 z1 0 z2 1 z2 denote the state space of the pair αt zt. Then as in [12], the process αt zt can be formulated as a Markov chain generated by 1 1 Au + Bw ε δ −µ1 µ1 0 0 1 λ u −λ1 u 0 0 = 1 0 0 −µ1 µ1 ε 0 0 λ1 u −λ1 u −µ2 w 0 µ2 w 0 1 0 µ2 w 0 −µ2 w + λ2 w 0 −λ2 w 0 δ 0 λ2 w 0 −λ2 w
Qεδ u w =
Here ε and δ are small parameters signifying the frequency of jumps of α and z. Note here µ1 represents the machine repair rate and hence is independent of production rate u. It is easily seen that
µ1 Q with Q = −µ1 Au = diag Q λ1 u −λ1 u −µ2 w µ2 w Bw = Qw ⊗ I2 with Qw = λ2 w −λ2 w where I2 denotes the 2 × 2 identity matrix, and “⊗” denotes the usual Kronecker product (i.e., with X = xij and Y = yij , the ijth entry of X ⊗ Y
284
liu, yin, and zhang
is defined to be X ⊗ Y ij = xij Y ). In what follows, we shall generalize this model to include many states and weak and strong interactions. The rest of the paper is arranged as follows. Section 2 gives the precise formulation of the problem. Section 3 proceeds with the construction of the asymptotic expansion. Section 4 concentrates on the error analysis. Finally, Section 5 concludes the paper with further remarks.
2. FORMULATION 2.1. Initial Formulation We work with a finite horizon 0 T for some T > 0. Recall that a generam×m is said to be weakly irreducible (see [15]), if f tQt = 0, tor Qt
m ∈ and i=1 fi t = 1 has a unique nonnegative solution. The unique solution is termed a quasi-stationary distribution. The structure of the generator Qε δ to be considered is motivated by Example 1.1. To proceed, first generalize the setup in the example to include many states. For ease of presentation, consider the stationary case. Let αε δ t be a Markov chain with αε δ t = αε1 t αδ2 t and state space = a1 b1 am0 b1 a1 bl am0 bl where l and m0 are some positive integers. Denote m = lm0 . Suppose the generator is given by Qε δ = 1/εA + 1/δB, where
Q ⊗ Im and B = Q (2.1) A = diag Q 0
= q˜ ∈ m0 ×m0 being a generator and Q = q¯ ij ∈ l×l being with Q ij m0 ×m0 a generator and Im0 ∈ is an m0 × m0 dimensional identity matrix. Henceforth, we use the symbol diag to denote a diagonal block matrix with and Q are both weakly irreducible. appropriate entries. Suppose that Q Then it is easy Let ν˜ denote the stationary distribution corresponding to Q. ˜ = Q/δ. is the to see that diag˜ν ν˜ Qε δ | This demonstrates that Q/δ generator of the Markov chain αδ2 t with state space b1 bl . Similarly, with state space we can show αε1 t is a Markov chain generated by Q/ε a1 am0 . We next generalize this idea further for nonstationary cases and to include weak and strong interactions. 2.2. More General Formulation Suppose ε > 0 and δ > 0 are small parameters, αε δ t is a finite state Markov chain with state space = 1 m generated by Qε δ t, and the row vector pε δ t = Pαε δ t = 1 Pαε δ t = m ∈ 1×m denotes the probability distribution of the Markov chain at time t. It is well
singularly perturbed markov chains
285
known that pε δ · is a solution to the forward equation dpε δ t = pε δ tQε δ t pε δ 0 = p0 (2.2) dt
where p0i ≥ 0 for each i and p0i = 1 is the initial probability distribution. Assume that the generator Qε δ t ∈ m×m has the form 1 1 At + Bt + Qt (2.3) ε δ Assume that At and Bt have the same partitioned form and the same and Q replaced by time form of Kronecker product as in (2.1) with Q dependent Qt and Qt, respectively. The slowly changing motion is described by the generator qˆ 11 tIm0 qˆ 1l tIm0 = (2.4) Qt qˆ l1 tIm0 qˆ ll tIm0 Qε δ t =
Throughout the rest of the paper, unless otherwise noted, we always work with (2.3) with time-varying generators. In addition, we often use the notion |j = 1 1 ∈ j×1 , for some integer j. We make the following assumptions: = q¯ ij t are weekly irre(A1) For each t ∈ 0 T , Qt and Qt ducible. (A2) For some n ≥ 0, A·, B·, and Q· are n + 1-time continuously differentiable on [0, T]. In addition, d n+1 /dt n+1 A·, are Lipschitz on [0,T]. d n+1 /dt n+1 B·, and d n+1 /dt n+1 Q· Define an operator Lε δ by L
ε δ
1 1 df −f At + Bt + Qt f = dt ε δ
(2.5)
for any smooth vector-valued function f ·. Then Lεδ f = 0 if and only if f is a solution to the differential equation (2.2). We are now in a position to analyze the solution to (2.2). 3. ASYMPTOTIC EXPANSION Using singular perturbation techniques, to approximate the solution to (2.2), we seek outer–inner expansions of the form t t (3.1) + eε δ n t pε δ t = ϕε δ n t + ψε δ n ε δ
286
liu, yin, and zhang
where eε δ n t is the remainder, and the outer (regular part) and initial layer correction terms are given by ϕε δ n t =
n
εi δj ϕi j t
i+j=0
and
ψε δ n
t
n t t t εi δj ψi j = ε δ ε δ i+j=0
(3.2)
respectively, such that eε δ n t is small and the error bound holds uniformly in t. The main theorem is recorded below. Theorem 3.1. Assume (A1) and (A2). For small parameters ε > 0 and δ > 0, denote the unique solution to (2.2) by pε δ ·. Then for 0 ≤ i + j ≤ n, we can construct ϕi j · and ψi j · · such that (a)
ϕij · is twice differentiable on 0 T .
(b) For each i, there are κ1 > 0 and κ2 > 0 such that κ t κ t ij t ε 1 + K2 exp − 2 ψ ≤ K1 exp − ε δ ε δ (c) Suppose there exist constants h1 > 0 and h2 > 0 such that h1 ε ≤ δ ≤ h2 ε. Then the following estimate holds n n ε δ i j i j i j i j t t sup p t − ε δ ϕ t − εδψ ε δ t∈ 0 T i+j=0 i+j=0
≤ K εn+1 + δn+1 (3.3) When i = j = 0 in the above theorem, we obtain the limit of pεδ t for t > 0. Corollary 3.2. Suppose Qε δ · is continuously differentiable on 0 T , which satisfies (A1), and d/dtQ· is Lipschitz on 0 T . Then for all t > 0, limε δ→0 pε δ t = ϕ0 0 t. To obtain the desired result, we obtain the outer and inner expansion terms by direct construction in what follows. They involve solutions to algebraic-differential equations. 3.1. Outer Expansion We begin with the construction of ϕε δ n+1 · in the asymptotic expansion. Consider the differential equation Lε δ ϕε δ n+1 t = 0 where Lε δ is given by (2.5).
singularly perturbed markov chains
287
Equating coefficients of εi δj , we have for i + j = 0, ε−1 δ0 ε0 δ−1
ε 0 δ0
ε1 δ−1 ε−1 δ1 and for 1 ≤ i + j ≤ n, ε2 δ−1
ε 1 δ0
ε 0 δ1
0 = ϕ0 0 tAt 0 = ϕ0 0 tBt dϕ0 0 t = ϕ1 0 tAt + ϕ0 1 tBt dt + ϕ0 0 tQt 1 0 0=ϕ tBt 0 1 0=ϕ tAt 0 = ϕ2 0 tBt dϕ0 1 t = ϕ2 0 tAt + ϕ1 1 tBt dt + ϕ1 0 tQt dϕ0 1 t = ϕ1 1 tAt + ϕ0 2 tBt dt + ϕ0 1 tQt 0 2 0=ϕ tAt
(3.4)
(3.5)
ε−1 δ2 ··· · We construct the solutions to (3.4). For construction of solutions to systems with higher order, the approach is similar, only the notation is more involved. The main idea is as follows. We separate the equations in (3.4) into two groups. The first group consists of the first two equations. We show that they share a common solution. In fact, this common solution is the limit as ε → 0 and δ → 0. The second group includes the rest of the three equations. To solve the equations in the second group, we use the methods of undetermined coefficients. First, assume the solution to the last two equations is of particular form with certain functions (“multiplier”) to be determined. Then our task reduces to finding these functions by using the third equation. Such an idea is used in the construction of initial layer corrections as well as higher order terms. 3.1.1. Construction of ϕ0 0 t Based on (A1), Qt and Qt are irreducible. Let πt and λt be their quasi-stationary distributions, respectively. Denote πλ t =
λ1 tπt λl tπt ∈ m . According to assumption (A2), the vector-valued functions λt, πt and πλ t are (n + 1)-times continuously differentiable. We claim that the vector-valued function ϕ0 0 t = πλ t = 0, so solves the first two equations of the system (3.4). Note that πtQt
Qt ϕ0 0 tAt = λ1 tπt λl tπt diag Qt
= λ1 tπtQt λl tπtQt = 0
288
liu, yin, and zhang
where At = diagQt Qt. As for the second equation, λt
l Qt = 0; that is, i=1 λi tq¯ ij t = 0 for all 1 ≤ j ≤ l. Thus, q¯ 11 tIm0 q¯ 1l tIm0
ϕ0 0 tBt = λ1 tπt λl tπt q¯ l1 tIm0 q¯ ll tIm0 l l = λi tπtq¯ i1 tIm0 λi tπtq¯ il tIm0 i=1
=
l i=1
In addition, since
m 0 0 t = 1 i=1 ϕi
i=1
λi tq¯ i1 t πt
l
i=1
λi t = 1 and
m0
l i=1
i=1
λi tq¯ il t πt = 0
πi t = 1 ϕ0 0 t|m =
3.1.2. Construction of ϕ1 0 t To obtain the desired solutions, we use the methods of undetermined coefficients. We postulate that the fourth and fifth equations of (3.4) are given by ϕ1 0 t = λ1 txt λl txt and ϕ0 1 t = y1 tπt yl tπt where πt and λt are the quasi-stationary distributions of Qt and Qt, respectively, and where the vector-valued functions xt = x1 t xm0 t ∈ 1×m0 and yt = y1 t yl t ∈ 1×l are
1 0 to be determined. In addition, ϕ1 0 t|m = m t = 0, and i=1 ϕi
0 1 m 0 1 ϕ t|m = i=1 ϕi t = 0. The fourth and fifth equations of (3.4) are not enough to determine xt and yt. We also need to use the third equation of (3.4), which can be written as ϕ1 0 tAt + ϕ0 1 tBt =
dϕ0 0 t − ϕ0 0 tQt dt
The right-hand side of the above equation can be expressed as dϕ0 0 t = dπλ t − π tQt − ϕ0 0 tQt λ dt dt dπt dπt = λ1 t λl t dt dt
singularly perturbed markov chains 289 dλ1 t dλ t + πt l πt dt dt l l λi tqˆ i1 tπt λi tqˆ il tπt − i=1
i=1
Two equations may be derived from the third equation of (3.4). They are dπt dπt 1 0 ϕ λl t (3.6) tAt = λ1 t dt dt and dλ1 t dλ t ϕ0 1 tBt = πt l πt dt dt l l λi tqˆ i1 tπt λi tqˆ il tπt (3.7) − i=1
i=1
Consider (3.6). In view of the block diagonal form of At, (3.6) is equivalent to = λ t dπt λ1 txtQt 1 dt ··· = λ t dπt λl txtQt l dt It suffices to show that the following system has a solution = xtQt
dπt dt
and
m0 i=1
xi t = 0
(3.8)
by NQt. Denote the null space of Qt The weak irreducibility of Qt implies that rank Qt = m0 − 1 and hence dimNQt = 1 Note that NQt is spanned by |m0 . By virtue of the well-known Fredholm alternative, the first equation in (3.8) has a solution only if its right-hand side is orthogonal to NQt. Since NQt is spanned by |m0 , πt|m0 = 1 dπt|
m0 and dπt |m 0 = = 0, so the orthogonality is verified. Next we dt dt show that the system (3.8) has a unique solution. To this end, let us rewrite the system (3.8) as
q˜ 11 tx1 t + · · · + q˜ m0 1 txm0 t =
dπ1 t dt
q˜ 1 m0 tx1 t + · · · + q˜ m0 m0 txm0 t = x1 t + · · · + xm0 t = 0
dπm0 t dt
(3.9)
290
liu, yin, and zhang
The existence and uniqueness of the solution to (3.9) follows from the weak t = | Qt. irreducibility and Fredholm alternative. Denote Q Then c m0 the solution can be expressed as xt =
dπt tQ t−1 Q c t Q c c dt
Moreover xt is n-times continuously differentiable on 0 T . Consequently, (3.6) has a unique solution ϕ1 0 t = λ1 txt λl txt
m 1 0 t = 0. In addition, ϕ1 0 t is n-times with the condition i=1 ϕ continuously differentiable. 3.1.3. Construction of ϕ0 1 t Examine (3.7). Rewrite Eq. (3.7) in its component form l i=1
yi tq¯ ij tπt =
dλj t dt
πt −
l i=1
λi tqˆ ij tπt
for j = 1 l
To establish the existence of a solution, it suffices to show that = ytQt
dλt − λtQh t dt
(3.10)
l = 1, has a solution, where Qh t = qˆ ij ∈ l×l . Since i=1 λi t dλt h − d/dtλt| l = d/dtλt|l = 0. Owing to Q t|l = 0 dt
λtQh t |l = 0; i.e., the right-hand side of (3.10) is orthogonal to |l . By (A1), rankQt = l − 1. An argument similar to that of (3.8) confirms that = ytQt
dλt − λtQh t dt
l i=1
yi t = 0
(3.11)
has a unique solution yt = y1 t yl t. Hence ϕ0 1 t = y1 tπt yl tπt solve (3.7). It is easy to see that ϕ0 1 t is
0 1 n-times continuously differentiable. In addition, we have m t = i=1 ϕi
l
m 0
l j=1 yj t k=1 πk t = j=1 yj t = 0. 3.1.4. Construction of ϕi j t for 2 ≤ i + j ≤ n + 1 In essentially the same way, we can construct ϕij t for 2 ≤ i + j ≤ n + 1. The details are thus omitted. To summarize, we state the following theorem.
singularly perturbed markov chains
291
Theorem 3.3. Assume (A1) and (A2) hold. Then there exist n + 1 − i + j-times continuously differentiable solutions ϕi j t, 0 ≤ i + j ≤ n + 1, for (3.4) and (3.5). In addition, the following conditions are satisfied: m k=1
0 0
ϕk
t = 1
m k=1
i j
ϕk
t = 0
for 1 ≤ i + j ≤ n + 1
(3.12)
Remark 3.4. The conditions in (3.12) imply that, for 1 ≤ i + j ≤ n + 1, ϕi j 0|m = 0; i.e., ϕi j 0 is orthogonal to |m . This fact will help us to obtain the desired exponential decay property of the initial layer terms. 3.2. Inner Expansion In this section, we construct the initial layer terms ψi j · ·. It consists of two parts. First, we obtain the sequence ψi j · · by solving a system of partial differential equations. Then we present the exponential decay property of the solutions. 3.2.1. Construction of ψi j · · Following the idea in singular perturbation, define the stretched time variables as follows: t t (3.13) τ= µ= ε δ Note that τ → ∞ as ε → 0, and µ → ∞ as δ → 0 for any t > 0. Consider the differential equation Lε δ ψε δ n+1 τ µ = 0 where the operator Lε δ is defined in (2.5) and ψε δ n+1 τ µ is defined in (3.2). Thus the above equation can be written as 1 ∂ 1 ∂ (3.14) + ψε δ n+1 τ µ = ψε δ n+1 τ µQε δ τ µ ε ∂τ δ ∂µ Taking Taylor expansions of Qε δ t about t = 0, n+1
t k d k Q0 + Rn+1 t k k! dt k=0 n+1 k k −1 t k d k A0 t k d k Q0 −1 t d B0 = ε + Rn+1 t + δ + k k k k! dt k! dt k! dt k=0 n+1 k−1 t/εk d k A0 t/δk d k B0 = ε + δk−1 k k! dt k! dt k k=0 t/δk d k Q0 + Rn+1 t + δk k! dt k
Qε δ t =
292
liu, yin, and zhang
where Rn+1 t = Ot n+2 uniformly for t ∈ 0 T . Using τ and µ, we can write the above equation as Qεδ τ µ =
n+1
εk−1
k=0
τk d k A0 µk d k B0 µk d k Q0 + δk−1 + δk k k k! dt k! dt k! dt k
+ Rn+1 τ µ Dropping the term Rn+1 τ µ in Qεδ t, it follows from (3.14) ∂ ∂ ψε δ n+1 τµ ε−1 +δ−1 ∂τ ∂µ n+1 k k k k k−1 τk d k A0 k−1 µ d B0 k µ d Q0 ε +δ +δ = ψε δ n+1 τµ k! dt k k! dt k k! dt k k=0 That is,
n+1
ε
−1
i+j=0
=
n+1
∂ −1 ∂ +δ εi δj ψi j τ µ ∂τ ∂µ
εi δj ψi j τ µ
k=0
×
n+1
ε
i+j=0
k k k k d k A0 k−1 µ d B0 k µ d Q0 +δ +δ k! dt k k! dt k k! dt k
k−1 τ
k
Equating the coefficients of εi δj , we have for i + j = 0, ε−1 δ0
ε0 δ−1
ε 0 δ0
∂ψ0 0 τ µ = ψ0 0 τ µA0 ∂τ ∂ψ0 0 τ µ = ψ0 0 τ µB0 ∂µ ∂ψ1 0 τ µ ∂ψ0 1 τ µ + ∂τ ∂µ = ψ0 0 τ µDτ µ + ψ1 0 τ µA0 + ψ0 1 τ µB0
ε1 δ−1
∂ψ1 0 τ µ = ψ1 0 τ µB0 ∂µ
ε−1 δ1
∂ψ0 1 τ µ = ψ0 1 τ µA0 ∂τ
(3.15)
singularly perturbed markov chains
293
and for 1 ≤ i + j ≤ n, ε2 δ−1
∂ψ2 0 τ µ = ψ2 0 τ µB0 ∂µ
ε 1 δ0
∂ψ2 0 τ µ ∂ψ1 1 τ µ + ∂τ ∂µ = ψ0 0 τ µ
d 2 A0 2 τ + ψ1 0 τ µDτ µ dt 2
+ ψ2 0 τ µA0 + ψ1 1 τ µB0 ε 0 δ1
∂ψ1 1 τ µ ∂ψ0 2 τ µ + ∂τ ∂µ = ψ0 0 τ µ
(3.16)
d 2 B0 2 µ + ψ0 1 τ µDτ µ dt 2
+ ψ1 1 τ µA0 + ψ0 2 τ µB0 ε−1 δ2
∂ψ0 2 τ µ = ψ0 2 τ µA0 ∂τ
where
dA0 dB0 +µ + Q0 dt dt We will construct ψij · · by solving (3.15) and (3.16). Dτ µ = τ
Theorem 3.5. Assume (A1) and (A2). Then there exist continuously differentiable solutions ψi j t. 0 ≤ i + j ≤ n for the systems (3.15) and (3.16). The solution to (3.15) can be expressed as
ψ0 0 τ µ = p0 − ϕ0 0 0 expA0τ + B0µ ψ1 0 τ µ = H 1 0 τ expA0τ + B0µ
(3.17)
ψ0 1 τ µ = H 0 1 µ expA0τ + B0µ where the row vectors H 1 0 τ and H 0 1 µ are given by τ dA0 H 1 0 τ = p0 − ϕ0 0 0 expA0s dt 0 × exp−A0ss ds − ϕ1 0 0 H
0 1
dB0 1 µ = µ2 p0 − ϕ0 0 0 2 dt 0 0 0 − ϕ0 1 0 + µp − ϕ 0Q0
(3.18)
For 2 ≤ i + j ≤ n + 1, the functions ψij · · can be expressed similarly.
294
liu, yin, and zhang
Proof. Construction of ψij τ µ for 0 ≤ i + j ≤ 1: Let us determine τ µ from the first two equations of (3.15). First, ψ0 0 τ µ can be ψ taken as 0 0
ψ0 0 τ µ = C 0 0 expA0τ + B0µ
(3.19)
where C 0 0 is a constant vector to be determined by the initial value. From the first equation of (3.15), we have ψ0 0 τ µ = C1 µ expA0τ
(3.20)
where C1 µ is a vector-valued function. To determine C1 µ, substituting (3.20) into the second equation of (3.15), we obtain that dC1 µ expA0τ = C1 µ expA0τB0 dµ Since A0 and B0 commute, expA0τB0 = B0 expA0τ, and dC1 µ expA0τ = C1 µB0 expA0τ dµ
(3.21)
Note that expA0τ is invertible for any matrix A0 and its inverse is exp−A0τ. Postmultiplying exp−A0τ to both sides of (3.21), we ob1 µ tain dCdµ = C1 µB0 It follows that C1 µ = C 0 0 expB0µ, where C 0 0 is a constant vector. Substituting C1 µ into (3.20), ψ0 0 τ µ = C 0 0 expB0µ expA0τ. Note that expB0µ expA0τ = expA0τ + B0µ. It follows that ψ0 0 τ µ = C 0 0 expA0τ + B0µ. Thus (3.19) is obtained. To solve for ψ1 0 τ µ and ψ0 1 τ µ, start with the fourth and fifth equations in (3.15). Assume that the solutions can be written as ψ1 0 τ µ = H 1 0 τ expA0τ + B0µ ψ0 1 τ µ = H 0 1 µ expA0τ + B0µ where H 1 0 · and H 0 1 · are vector-valued functions to be determined later. As in the construction of the outer expansion terms, H 1 0 · and H 0 1 · need to be determined by the use of the third equation in (3.15). In fact, dH 1 0 τ dH 0 1 µ + expA0τ + B0µ dτ dµ = τC 0 0 expA0τ + B0µ + B0µ
dA0 + µC 0 0 expA0τ dt
dB0 + C 0 0 expA0τ + B0µQ0 dt
singularly perturbed markov chains
295
We separate the above equation into two equations dA0 dH 1 0 τ expA0τ + B0µ = τC 0 0 expA0τ + B0µ dτ dt dB0 dH 0 1 µ expA0τ + B0µ = µC 0 0 expA0τ + B0µ dµ dt + C 0 0 expA0τ + B0µQ0 Note that expA0τ + B0µ is invertible and its inverse is exp−A0τ − B0µ. By postmultiplying exp−A0τ − B0µ to both sides of the above two equations, we have dH 1 0 τ dA0 exp−A0τ − B0µ = τC 0 0 expA0τ + B0µ dt dτ dB0 dH 0 1 µ = µC 0 0 expA0τ + B0µ exp−A0τ − B0µ dµ dt exp−A0τ − B0µ + C 0 0 expA0τ + B0µQ0 In addition, the following communitivity holds dA0 dA0 expB0µ = expB0µ dt dt dB0 dB0 expA0τ + B0µ = expA0τ + B0µ dt dt expA0τ + B0µ = expA0τ + B0µQt Qt It follows that dH 1 0 τ dA0 = τC 0 0 expA0τ exp−A0τ dτ dt dB0 dH 0 1 µ = µC 0 0 + C 0 0 Q0 dµ dt The solutions to the above equations are τ dA0 H 1 0 τ = C 0 0 exp−A0ss ds + C 1 0 expA0s dt 0 (3.22) 1 2 0 0 dB0 0 1 0 0 0 1 Q0 + C + µC µ = µ C H 2 dt where C 1 0 and C 0 1 are constant row vectors to be determined by the initial data. Choosing matched (between outer and inner terms) initial conditions leads to ϕ0 0 0 + ψ0 0 0 0 = p0
ϕ1 0 0 + ψ1 0 0 0 = 0
ϕ0 1 0 + ψ0 1 0 0 = 0
296
liu, yin, and zhang
The first equation together with (3.19) gives C 0 0 = p0 − ϕ0 0 0 and the second and third equations together with (3.22) imply that C 1 0 = −ϕ1 0 0, and C 0 1 = −ϕ0 1 0. Substituting C 0 0 , C 1 0 , and C 0 1 into (3.22), we get
τ dA0 expA0s H 1 0 τ = p0 − ϕ0 0 0 dt 0 × exp−A0ss ds − ϕ1 0 0
dB0
1 H 0 1 µ = µ2 p0 − ϕ0 0 0 + µ p0 − ϕ0 0 0 Q0 2 dt − ϕ0 1 0 This completes the construction of ψij · · for 0 ≤ i + j ≤ 1. The construction of the solutions for 2 ≤ i + j ≤ n + 1 is similar. We omit the details. 3.2.2. Exponential Decay of ψi j · · We obtained the asymptotic series ψi j · · in Theorem 3.5. In this section, we will verify the exponential decay properties of the solutions. Similar to Lemma A.2 in [15, p.300], we have the following result. Lemma 3.6.
Suppose that assumptions (A1) and (A2) are satisfied.
(a)
Suppose Ps is the solution to the differential equation: dPs = PsA0 P0 = I ds A as s → ∞ and Then Ps → P expA0s − P A ≤ KA exp−κA s for some κA > 0
(3.23) (3.24)
A = diag|m ν0 |m ν0 and ν0 = ν1 0 νm 0 ∈ where P 0 0 0 1×m0 is the quasi-stationary distribution corresponding to the generator Q0. (b) Suppose Ps is the solution to the differential equation: dPs = PsB0 P0 = I (3.25) ds B as s → ∞ and Then Ps → P expB0s − P B ≤ KB exp−κB s for some κB > 0 (3.26) where
λ1 0Im0 λl 0Im0 B = (3.27) P λ1 0Im0 λl 0Im0 and λ0 = λ1 0 λl 0 is the quasi-stationary distribution of the gen erator Q0 = q¯ ij 0.
singularly perturbed markov chains
297
Proof. To prove (a), note that |m0 ν0 has identical rows ν1 0 νm0 0. In view of the block-diagonal structure of A0, we have Ps = expA0s = diagexpQ0s expQ0s By using Lemma A.2 in [15, p. 300], it is readily seen that there exist − |m0 ν0 ≤ constants KA > 0 and κA > 0 such that expQ0s A ≤ KA exp−κA s. Thus Ps → PA as s → ∞ and expA0s − P KA exp−κA s. The first result is proved. To prove (b), note that after some rows and columns are interchanged, B0 becomes the block diagonal form B∗ 0 = diagQ0 Q0. That m×m such that RB0R−1 = is, there is an invertible constant matrix R ∈ B∗ 0 Similar to the first part of this lemma, we have def B expB∗ 0s → diag|l λ0 |l λ0 ∈ m×m = P
as s → ∞
where diag|l λ0 |l λ0 is a block diagonal matrix with m0 blocks. Consequently, R−1 expB∗ 0sR → R−1 diag|l λ0 |1 λ0R
as s → ∞
Note that R−1 expB∗ 0sR = expR−1 B∗ 0Rs = expB0s B as B has the form given by (3.27). Thus we have expB0s → P and P s → ∞ with the desired convergence rate. This completes the proof. Lemma 3.7. Suppose that assumptions (A1) and (A2) are satisfied. Let PA and PB be defined as in Lemma 3.6. Then there exist constants KA > 0, B ≤ A P KB > 0, κA > 0, and κB > 0 such that expA0s + B0t − P KA exp−κA s + KB exp−κB t. Proof. Notice the fact that At and Bt commute. We start with the inequality: expA0s + B0t − P B A P B + expA0sP B B − P A P = expA0s + B0t − expA0sP B + expA0s − P A P B = expA0sexpB0t − P B + expA0s − P A ≤ expA0sexpB0t − P PB > 0, KB > 0, κA > 0, and κB > 0 By Lemma 3.6, there exist constants KA such that expA0s − P A ≤ KA exp−κA s and expB0t − P B ≤ KB exp−κB t
298
liu, yin, and zhang
B ≤ MB for some constant MB > 0 and expA0s ≤ MA for Since P some constant MA > 0, we have expA0s + B0t − P B ≤ MA KB exp−κB t + MB KA A P exp−κA s ≤ KB exp−κB t + KA exp−κA s This completes the proof. Now we are in a position to state the result about the exponential decay of ψij · ·. Theorem 3.8. Under the conditions of Theorem 3.1, for each i, j with 0 ≤ ij ij i + j ≤ n + 1, there exist polynomials cA τ µ and cB τ µ, and positive numbers κA and κB such that ij ψ τ µ ≤ c ij τ µ exp−κA τ + c ij τ µ exp−κB µ (3.28) B A
m 0 0
p0i = 1 and 0 = 1. It follows that Proof. Note that m i=1 ϕi
m 0 0
i=1
0 0 0 0 m m 0 0 0 = 0 = 0 That is, ψi 0 0 i=1 ψi i=1 pi − i=1 ϕi A and P B be defined in Lemma 3.6. is orthogonal to |m . Let P Recall the forms of PA and PB . Then λ1 0|m0 π0 λl 0|m0 π0 B = A P P λ1 0|m0 π0 λl 0|m0 π0 = |m λ1 0π0 λl 0π0 = |m πλ 0 B = ψ0 0 0 0|m πλ 0 = 0 i.e., A P As a Consequence, ψ0 0 0 0P ψ0 0 0 is orthogonal to πλ 0. By virtue of Lemma 3.7, 0 0 ψ τ µ = ψ0 0 0 0 expA0τ + B0µ A P A P B − ψ0 0 0 0P B = ψ0 0 0 0P + ψ0 0 0 0 expA0τ + B0µ B A P ≤ ψ0 0 0 0P A P B +ψ0 0 0 0expA0τ + B0µ − P B A P = ψ0 0 0 0expA0τ + B0µ − P ≤ ψ0 0 0 0KA exp−κA τ + KB exp−κB µ A exp−κA τ + K B exp−κB µ ≤K That is, ψ0 0 τ µ decays exponentially fast.
(3.29)
singularly perturbed markov chains
299
Next consider ψ0 1 τ µ. Recall that ψ0 1 τ µ = H 0 1 µ expA0 τ + B0µ, where
dB0
1 01 H 01 µ = µ2 p0 −ϕ00 0 0 +µ p0 −ϕ00 0 Q0−ϕ 2 dt
0 1 Note that ϕ0 1 0| = m 0 = 0 Since Q0 and B(0) are generi=0 ϕi dB0 dB0|m ators, Q0|m = 0, and dt |m = = 0. Thus we obtain dt 1 2 0 dB0 µ p − ϕ0 0 0 H 0 1 µ|m = 2 dt 0
0 0 +µ p − ϕ 0 Q0 − ϕ0 1 0|m =
dB0 1 2 0 µ p − ϕ0 0 0 |m 2 dt
0 1 + µ p0 − ϕ0 0 0 Q0| 0|m = 0 m−ϕ
i.e., H 0 1 µ is orthogonal to |m . Hence, 01 ψ 00 = H 01 µexpA0τ +B0µ
B +H 01 µ expA0τ +B0µ− P B A P A P = H 01 µP
A P B = H 01 µ expA0τ +B0µ− P A P B ≤ H 01 µ expA0τ +B0µ− P = H 01 µKA exp−κA τ+KB exp−κB µ 01
≤ cA 0 1 cA µ
01
µexp−κA τ+cB
µexp−κB µ
0 1 cB µ
and are polynomials of degree 2. Thus, ψ0 1 τ µ where also decays exponentially fast. Next we show the exponential decay of ψ1 0 τ µ. Recall that ψ1 0 τ µ = H 1 0 τ expA0τ + B0µ where
τ dA0 H 1 0 τ = p0 − ϕ0 0 0 expA0s exp−A0ss ds dt 0 − ϕ1 0 0
Since ϕ1 0 0 is orthogonal to |m , and ∞ dA0 −1n sn dA0 n exp−A0s| = A0 |m dt dt n! n=0 =
∞ −1n sn dA0 A0n |m dt n=0 n!
300
H
liu, yin, and zhang
1 0
=
s s2 dA0 Im |m − A0|m + A02 |m +··· dt 1! 2!
=
dA0 dA0 Im |m = |m = 0 dt dt
τ|m = p0 − ϕ0 0 0
τ
0
expA0s
dA0 exp−A0ss ds dt
− ϕ1 0 0 |m
τ dA0 exp−A0s|s ds = p0 − ϕ0 0 0 expA0s dt 0 − ϕ1 0 0|m = 0 So we have shown that H 1 0 τ is orthogonal to |m . Thus we have 1 0 ψ 0 0 = H 1 0 µ expA0τ + B0µ B A P = H 1 0 µexpA0τ + B0µ − P B A P ≤ H 1 0 µ expA0τ + B0µ − P = H 1 0 µKA exp−κA τ + KB exp−κB µ 1 0
≤ cA 1 0
1 0
µ exp−κA τ + cB
µ exp−κB µ
1 0
where cA µ and cB µ are polynomials of degree 2. The proof for the cases of 2 ≤ i + j ≤ n + 1 are similar to the above. Hence we omit the details. The theorem is proved. 1 0 1 0 Since the growth of cA µ and cB µ are much slower than exponential, we have the following two corollaries. Corollary 3.9. For 0 ≤ i + j ≤ n + 1, we have ψi j τ µ ≤ KA exp−κA + KB exp−κB Corollary 3.10. Suppose there exist constants h1 > 0 and h2 > 0 such that h1 ε ≤ δ ≤ h2 ε. Then for any integers 0 ≤ i + j ≤ n + 1, τψi j τ µ ≤ K1 , and µψi j τ µ ≤ K2 for some constants K1 > 0 and K2 > 0. 4. ERROR ANALYSIS In this section we will validate the asymptotic expansion. Recall that the operator Lε δ is defined by (2.5). We first prove a lemma.
singularly perturbed markov chains
301
ε δ ε δ Lemma 4.1. Suppose that for some 0 ≤ k ≤ n+ 1, sup t∈ 0 T kL kv k k ε δ ε δ t = Oε + δ and v 0 = 0. Then supt∈ 0T v t ≤ Oε + δ . Proof. Let ηε δ t be a vector-valued function satisfy supt∈ 0 T ηε δ tOεk + δk Consider the differential equation
Lε δ vε δ t = ηε δ t vε δ 0 = 0
(4.1) ε δ
the solution of the above equation is given by v t = Now t ε δ η sX ε δ t s ds, where X ε δ t s is a principal matrix solution. 0 Since X ε δ t s is a transition probability matrix, X ε δ t s ≤ K, for all t s ∈ 0 T . Therefore, we have the inequality t ηε δ s ds ≤ Kεk + δk sup vε δ t ≤ K sup t∈ 0T 0
t∈ 0T
This completes the proof. In view of (3.1), eε δ k t is defined as eε δ k t = pε δ t − ϕε δ k t − ψ t/ε t/δ where pε δ · is the solution to (2.2), and ϕε δ k · and ε δ k ψ · · are constructed in previous sections. We will estimate the error term eε δ n t. We have the following result. ε δ k
Proposition 4.2. Assume (A1) and (A2). Suppose there exist constants h1 > 0 and h2 > 0 such that h1 ε ≤ δ ≤ h2 ε. Then, for 0 ≤ i ≤ n, supt∈ 0T eε δ i t = Oεi+1 + δi+1 . Proof.
We first prove the result for 1
eε δ 1 t = pε δ t −
1
εi δj ϕi j t −
i+j=0
εi δj ψi j
t
i+j=0
t ε δ
ε δ 1
0 = 0, and hence the condition of Lemma 4.1 It is easy to see that e on the initial data holds. By the definition of Lε δ , Lε δ pε δ t = 0. Consequently, we have 1 i j i j Lε δ eε δ1 t = Lε δ pε δ t − Lε δ ε δ ϕ t i+j=0
1
− Lε δ
εi δj ψi j
i+j=0
= −L
=−
1
ε δ
i j
εδϕ
i j
1 i+j=0
i j
εδϕ
t − L
i+j=0
d dt
t t ε δ
i j
t −
ε δ
1 i+j=0
1 i+j=0
i j
εδψ
i j
t t ε δ
εi δj ϕi j tQε δ t
302
liu, yin, and zhang 1 d i j i j t t − εδψ dt i+j=0 ε δ −
1 i+j=0
i j
εδψ
i j
t t ε δ Q t ε δ
Based on the smoothness of ϕi j · on 0 T and the defining equation (3.4), 1 1 d i j i j ε δ ϕ t − εi δj ϕi j tQε δ t dt i+j=0 i+j=0 1 0 0 1 dϕ t t dϕ =ε − ϕ1 0 tQt − ϕ0 1 tQt +δ dt dt = Oε + δ Now let us estimate the terms containing ψi j · ·. Recall that 1 1 dA0 dB0 +µ + Q0 Qε δ t = A0 + B0 + τ ε δ dt dt + Oετ2 + δµ2 + δµ where τ = t/ε and µ = t/δ. According to the exponential decay property of ψi j · · and the defining equations (3.15) and (3.16), 1 1 t t d i j i j t t − Qε δ t εδψ εi δj ψi j dt i+j=0 ε δ ε δ i+j=0 1 1 d t t i j i j t t = εδψ εi δj ψi j − dt i+j=0 ε δ ε δ i+j=0 1 1 dA0 dB0 × A0 + B0 + τ +µ + Q0 + Oετ + δµ ε δ dt dt dB0 dA0 = −εψ1 0 τ µ + δψ0 1 τ µ τ +µ + Q0 dt dt −
1
εi δj ψi j τ µOετ2 + δµ2 + δµ
i+j=0
Based on the exponential decay of ψi j · · and Corollary 3.10, we obtain 1 1 t t d i j i j t t − Qε δ t = Oε + δ εδψ εi δj ψi j dt i+j=0 ε δ ε δ i+j=0
singularly perturbed markov chains
303
Thus we have proved that Lε δ eε δ 1 t = Oε + δ uniformly in 0 T . By Lemma 4.1, we have eε δ 1 t = Oε + δ uniformly in 0 T . Next we show the result holds for i = j = 0. Note that t t eε δ 1 t = eε δ 0 t − εϕ1 0 t − δϕ0 1 t − εψ1 0 ε δ t t − δψ0 1 ε δ By Corollary 3.10 we have εϕ
1 0
t + δϕ
0 1
t + εψ
1 0
t t ε δ
+ δψ
0 1
t t ε δ
= Oε + δ
uniformly in t ∈ 0 T . Thus eε δ 0 t = Oε + δ uniformly in t ∈ 0 T . Similarly, we can estimate eε δ n t. In fact, from n+1 n+1 i j i j i j i j t t eε δ n+1 t = pε δ t − ε δ ϕ t − εδψ ε δ i+j=0 i+j=0 we see that eεδn+1 0 = 0, and L
εδ εδn+1
e
t = L
εδ
p
εδ
εδ
= −L
t−
n+1
i j
εδϕ
i+j=0 n+1 i+j=0
i j
εδϕ
ij
ij
n+1
t−
εδψ
ij
i j
ij
i+j=0
t −L
i j
εδ
n+1
εδψ
i+j=0
t t ε δ
t t ε δ
n+1 i j ij i j ij d n+1 =− ε δ ϕ t − ε δ ϕ tQεδ t dt i+j=0 i+j=0 i j ij t t d n+1 − εδψ dt i+j=0 ε δ n+1 i j ij t t εδ − εδψ Q t ε δ i+j=0 It follows from the defining equations of ϕi j ·, n+1 n+1 i j i j i j i j d ε δ ϕ t − ε δ ϕ tQε δ t dt i+j=0 i+j=0 1 0 t dϕ = εn+1 − ϕ1 0 tQt dt
304
liu, yin, and zhang
dϕn 1 t − ϕn 1 tQt dt 0 n+1 t dϕ − ϕ0 n+1 tQt + · · · + δn+1 dt Based on the smoothness of ϕi j · on 0 T and the assumption h1 ε ≤ δ ≤ h2 ε, we have n+1 n+1 i j i j i j i j
d ε δ ϕ t − ε δ ϕ tQε δ t = O εn+1 + δn+1 dt i+j=0 i+j=0 + εn δ
uniformly in t. Now we estimate the terms containing ψi j · ·. According to the exponential decay property and the expansion of Qε δ · ·, n+1 n+1 i j i j t t i j i j t t d εδψ εδψ − Qεδ t dt i+j=0 ε δ ε δ i+j=0 n+1 n+1 i j i j t t i j i j t t d = − εδψ εδψ dt i+j=0 ε δ ε δ i+j=0 n+1
k k k k d k A0 k−1 µ d B0 k µ d Q0 × ε + δ + δ k! dt k k! dt k k! dt k k=0 n+1 i j i j t t − εδψ Oεn+1 τn+2 + δn+1 µn+2 ε δ i+j=0
= Oε
n+1
k−1 τ
+δ
k
n+1
+
n+1 i+j=0
i j
εδψ
i j
t t O εn+1 τn+2 + δn+1 µn+2 ε δ
= Oεn+1 + δn+1 so we have Lε δ eε δ n+1 t = Oεn+1 + δn+1 uniformly in 0 T . By Lemma 4.1, we have eε δ n+1 t = Oεn+1 + δn+1 uniformly in 0 T . Finally, from the expression of eε δ n+1 t, we have eε δ n+1 t = eε δ n t + Oεn+1 + δn+1 . This implies that eε δ n t = Oεn+1 + δn+1 uniformly in t. This completes the proof of the corollary and hence the proof of Theorem 3.1. 5. FURTHER REMARKS 5.1. Asymptotic Expansions for ε/δ = o1 In the previous sections we have constructed expansions for Markov chains with generators that have two small independent parameters such
singularly perturbed markov chains
305
that ε/δ is bounded. In this section, we consider the case ε/δ = o1; i.e., ε goes to zero much faster than δ. To be more specific, let ε = δ2 . Suppose that the generator is of the form (2.3) with ε = δ2 , where At, Bt, and Qt are as before. We seek asymptotic expansion of the form δ δ n p t = ϕ t + ψδ n t/δ2 + eδ n t, where the regular part the initial layer corrections are n n t t δi ϕi t ψδ n 2 = δi ψi 2 ϕδ n t = δ δ i=0 i=0 respectively, and eδ n t is the remainder. 5.1.1. Outer Expansion dϕδ n t dt
= ϕδ n tQδ t; that is, n n i 1 1 i dϕ t i i = δ δ ϕ t 2 At + Bt + Qt dt δ δ i=0 i=0
Consider the differential equation
Equating the coefficients of δi , we have the following system of equations δ−2
0 = ϕ0 tAt
δ−1
0 = ϕ1 tAt + ϕ0 tBt
δ0
δ1
dϕ0 t = ϕ2 tAt + ϕ1 tBt + ϕ0 tQt dt dϕ1 t = ϕ3 tAt + ϕ2 tBt + ϕ1 tQt dt
(5.1)
It is easy to see that ϕ0 t = πλ t = λ1 tπt λl tπt where λ· and π· are as defined previously. In addition, we have, for any t ∈
0 0 T , m i=1 ϕi t = 1. To find the solution ϕ1 t, we first notice that ϕ0 tBt = πλ tBt = 0. Now from the second equation of (5.1), we have ϕ1 tAt = 0 Hence the vector-valued function ϕ1 t = x1 tπt xl tπt is a solution to the second equation in (5.1), where xt ∈ 1×l is any vector-valued
function that satisfies the condition li=1 xi t = 0. To obtain ϕ2 t, we examine the third equation in (5.1). By substituting 0 ϕ t and ϕ1 t into the equation, we have dλ1 t dλl t dπt dπt πt πt + λ1 t λ1 t dt dt dt dt = ϕ2 tAt + x1 tπt xl tπtBt + λ1 tπt λl tπtQt
(5.2)
306
liu, yin, and zhang
To solve (5.2), it suffices to solve the following two equations derived from it, dπt dπt λ1 t λ1 t = ϕ2 tAt and (5.3) dt dt
dλ1 t dλ t πt l πt = x1 tπt xl tπtBt dt dt + λ1 tπt λl tπtQt (5.4)
We first look at (5.4). It can be reduced to
dλ1 t dλ t l dt dt
= x1 t xl tq¯ ij t + λ1 t λl ttqˆ ij t
Note that dλdt1 t dλdtl t |l = 0 and qˆ ij t|l = 0 By the weak irreducibility of q¯ ij t, we can uniquely solve the equation for xt with the
1 condition li=1 xi t = 0. The last equality implies m i=1 ϕi t = 0. Sim 2 ilarly, we can solve (5.3) for ϕ2 t so that m i=1 ϕi t = 0. For i ≥ 3, ϕi t can be obtained similarly. 5.1.2. Initial Layer Correction To construct the boundary layer terms, we consider another time scale τ = t/δ2 Consider the differential equation dψεn t/δ2 1 1 = ψεn t/δ2 2 At + Bt + Qt (5.5) dt δ δ Taking Taylor expansion of Qδ t at t = 0, we have n+1 k k k k τk d k A0 2k−1 τ d B0 2k τ d Q0 Qδ δ2 τ = δ2k−2 + δ + δ k! dt k k! dt k k! dt k k=0 + Rn+1 δ2 τ where Rn+1 t = Ot n+2 . Drop the term Rn+1 δ2 τ and substitute the rest of the terms into (5.5), n+1 n+1 i i i i d n+1 τk d k A0 δ2k−2 δ ψ τ = δ ψ τ dτ i=0 k! dt k i=0 k=0 k k µk d k B0 k µ d Q0 + δ2k−1 + δ k! dt k k! dt k
singularly perturbed markov chains
307
Equating the coefficients of δi , we obtain d 0 ψ τ = ψ0 τA0 dτ d 1 ψ τ = ψ1 τA0 + ψ0 τB0 dτ d 2 ψ τ = ψ2 τA0 + ψ1 τB0 + ψ0 τ dτ dA0 × τ + Q0 dt
(5.6)
The solutions to (5.6) are ψ0 τ = ψ0 0 expA0τ ψ1 t = ψ1 0 expA0τ τ + ψ0 sBs expA0τ − s ds 0
2
2
ψ t = ψ 0 expA0τ τ + ψ1 sBs + ψ0 sDs
(5.7)
0
× expA0τ − s ds + Q0. The initial values of ψi 0 satisfy ψ0 0 = where Ds = s dA0 dt p0 − ϕ0 0, ψi 0 = −ϕi 0, for i ≥ 1 A = diag|m ν0 |m ν0. Similar to the previous secDenote P 0 0 A ≤ KA exp−κA s for some κA > 0 In addition, tions, expA0s − P we have the following exponential decay properties of ψi ·. The proof of the following proposition is similar to that of Lemma 3.7 and is thus omitted. Proposition 5.1. For each i = 0 1 n + 1, there exist a polynomial c i τ and positive number κ such that ψi τ ≤ c i τ exp−κτ. By virtue of Proposition 5.1, we have for any i = 1 n + 1, τk ψi τ ≤ K for some K > 0 and i = 1 n + 1. 5.1.3. Asymptotic Validation Now we give the estimate for the error term eδ n t = pδ t −
n
n i i 2 i=0 ϕ t − i=0 ψ t/δ The next proposition is the asymptotic property of the expansion.
308
liu, yin, and zhang
Proposition 5.2. Assume (A1) and (A2). Then, for 0 ≤ i ≤ n, supt∈ 0 T eδ i t = Oδi+1 . The proof of the proposition follows from Proposition 3.7 and the proof of Proposition 5.1. Thus we omit the details here. As a corollary of the above proposition, we have limδ→0 pδ t = ϕ0 t = πλ t. 5.2. Concluding Remarks In this paper, we have developed asymptotic expansions of the probability vector. The results obtained will be useful for many optimal control problems that involve singularly perturbed Markovian models with multiple time scales. The choice of the generators At and Bt is largely motivated by the applications in control and optimization of manufacturing systems. If At and Bt are both weakly irreducible (i.e., consisting of one block), the asymptotic expansion similar to the development discussed in this paper can be obtained. Such a treatment can be used for certain reducible matrices having different forms than those of this paper. However, if At and Bt have different partitions in the most general form, it appears that the construction of the asymptotic expansion cannot be done as here since the algebraic and differential equations involved may not be consistent. Future study can be directed to the investigation of further probabilistic properties of the model. REFERENCES 1. A. Bensoussan, “Perturbation Methods in Optimal Control,” Wiley, Chichester, 1988. 2. M. H. A. Davis, “Markov Models and Optimization,” Chapman & Hall, London, 1993. 3. F. Delebecque and J. Quadrat, Optimal control for Markov chains admitting strong and weak interactions, Automatica 17 (1981), 281–296. 4. J. K. Hale, “Ordinary Differential Equations,” 2nd ed., Krieger, Malabar, 1980. 5. S. N. Ethier and T. G. Kurtz, “Markov Processes: Characterization and Convergence,” Wiley, New York, 1986. 6. R. Z. Khasminskii, G. Yin, and Q. Zhang, Asymptotic expansions of singularly perturbed systems involving rapidly fluctuating Markov chains, SIAM J. Appl. Math. 56 (1996), 277–293. 7. R. Z. Khasminskii, G. Yin, and Q. Zhang, Constructing asymptotic series for probability distribution of Markov chains with weak and strong interactions, Quart. Appl. Math. LV (1997), 177–200. 8. H. J. Kushner, “Approximation and Weak Convergence Methods for Random Processes, with Applications to Stochastic Systems Theory,” MIT Press, Cambridge, MA, 1984. 9. Z. G. Pan and T. Bas¸ar, H ∞ -control of Markovian jump linear systems and solutions to associated piecewise-deterministic differential games, in “New Trends in Dynamic Games and Applications” (G. J. Olsder, Ed.), pp. 61–94, Birkh¨auser, Boston, 1995. 10. A. A. Pervozvanskii and V. G. Gaitsgori, “Theory of Suboptimal Decisions: Decomposition and Aggregation,” Kluwer, Dordrecht, 1988.
singularly perturbed markov chains
309
11. R. G. Phillips and P. V. Kokotovic, A singular perturbation approach to modelling and control of Markov chains, IEEE Trans. Automat. Control 26 (1981), 1087–1094. 12. S. P. Sethi and Q. Zhang, “Hierarchical Decision Making in Stochastic Manufacturing Systems,” Birkh¨auser, Boston, 1994. 13. H. A. Simon and A. Ando, Aggregation of variables in dynamic systems, Econometrica 29 (1961), 111–138. 14. W. A. Thompson, Jr., “Point Process Models with Applications to Safety and Reliability,” Chapman & Hall, New York, 1988. 15. G. Yin and Q. Zhang, “Continuous-Time Markov Chains and Applications: A Singular Perturbation Approach,” Springer-Verlag, New York, 1998.