Collaboration of discrete-time Markov chains: Tensor and product form

Performance Evaluation 67 (2010) 779–796

Contents lists available at ScienceDirect

Performance Evaluation journal homepage: www.elsevier.com/locate/peva

Collaboration of discrete-time Markov chains: Tensor and product form J.M. Fourneau INRIA project MESCAL and PRiSM, Université de Versailles Saint-Quentin, CNRS UMR 8144, UniverSud, Versailles, France

article

info

Article history: Received 2 March 2009 Received in revised form 28 December 2009 Accepted 23 January 2010 Available online 6 February 2010 Keywords: Discrete-time Markov chains Product form steady-state distribution Competing Markov chains Tensor algebra

abstract We consider first discrete-time Markov chains (DTMCs) in competition over a set of resources. We build a multidimensional Markov chain based on the Cartesian product of the state space and on competition rules between the chains. We then generalize this approach to DTMCs with collaboration between components. The competition and the collaboration between chains simply assume that when a resource is owned by a component (or when a component is in a specific subset of states) it affects the transition probabilities of the other components of the chain. We prove that under some competition rules the steady-state distribution of the multidimensional chain has a product form. This work extends Boucherie’s theory based on continuous-time chains. The proof relies on algebraic properties of the generalized tensor product defined by Plateau and Stewart. © 2010 Elsevier B.V. All rights reserved.

1. Introduction It is much more difficult in general to obtain closed form expressions for discrete-time Markov chains than for continuous-time models. For instance, there are only few results on discrete-time networks of queues with product form steady-state solutions (see however [1–3] for some results). The main difficulty arises from the large number of events which can occur simultaneously in discrete-time systems. Furthermore, the probability of a set of events is not in general easily related to the probability of individual events. Most of the results in the previous references are based on queues which are synchronized by customers’ movements. Here we study a completely different type of synchronized system. We assume that the transition probabilities of one of the chains depend, in general, on the states of the other chains in the model. In the Stochastic Automata Network (SAN in the following) terminology, this is denoted as a functional transition probability. An example of such a system in continuous time has been introduced by Boucherie in [4] as competing Markov chains. He considered two types of competitions and both types are proved to be modelled by a continuous-time Markov chain (CTMC) with product from steady-state solution. We briefly present here the first model. The second model which requires local balance will not be considered in this paper. First we need to introduce some concepts and notation from Boucherie’s presentation. He considered a collection of K CTMCs and defined a product process and some restrictions on the reachable state space due to the competition over resources. The competition is modelled as the exclusion of a part of the state space and the cancellation of some transitions. For instance, assume that two chains (say X1 and X2 ) compete over a resource; if chain X1 owns the resource, all the transitions of chain X2 are cancelled. The rule is symmetrical. If the resource is owned by X2 , X1 cannot move. Thus the process cannot enter states where several continuous-time chains own the resource. Let Ql be the transition rate matrix for CTMC Xl and let Sl be its state space. Boucherie introduced a set of resources (say I). In each state, a CTMC uses one and only one resource. So he can partition the state space Sl into mutually exclusive sets Zl,r . The states in Zl,r are the states where the resource r is owned by CTMC Xl . The reachable state space of the chains in competition is a subset of the Cartesian product. It does not contain the states in which the resources are used by several chains in competition. If CTMCs Xl and Xh compete over resource r, then the global process cannot reach the states E k where components kl ∈ Zl,r and kh ∈ Zh,r . Let Cl,r be the set of CTMCs which compete with CMTC Xl over resource r. E-mail address: [email protected]. 0166-5316/$ – see front matter © 2010 Elsevier B.V. All rights reserved. doi:10.1016/j.peva.2010.01.008

780

J.M. Fourneau / Performance Evaluation 67 (2010) 779–796

Fig. 1. Two continuous-time Markov chains in competition following Boucherie’s theory.

Boucherie assumed that the transition rate q(E k, E k0 ) of the process is : K X

" qj (kj , kj ) 0

K Y

i=1,i6=j

j =1

# 1(ki =k0i ) 1(∀ r ,ki ∈Zi,r ⇒j6∈Ci,r ) .

Fig. 1 presents two chains X 1 and X 2 respectively with states {0, 1, 2, 3} and {0, 1, 2} competing over one resource. The resource is owned by X 1 (resp. X 2) when it is in state 2 or 3 (resp. state 2). It is released in both chains when the chain jumps from state 2 or 3 to 1. When process X 1 is in state 2 or 3, process X 2 is stopped. If process X 1 is in state 0 or 1, process X 2 can move. We also have symmetrical rules when X 2 owns the resource. There is no priority scheduling in the continuous times model. Thus states in {2, 3} × {2} are forbidden and the reachable state space of the product process is {0, 1, 2, 3} × {0, 1, 2} \ {2, 3} × {2}. In Fig. 1, we have depicted the graphs of X 1 and X 2 on the top and the graph of the product process when the chains are competing over the resource on the bottom. Here we consider a discrete-time generalization of this model but the proof is purely algebraic and based on the generalized tensor product introduced by Plateau [5] and extensively studied by Fernandes, Plateau and Stewart [6]. For systems in discrete time, without synchronizations, with functional transition probabilities, the transition probability matrix of the global DTMC is the generalized tensor product of the functional matrices (denoted as ⊗g ) which will be defined formally in the next section. We consider the collaboration of DTMCs or their competition. The transition matrices of the chains in collaboration are functional (i.e. the entries are functions of the states of the other components of the model). We obtain a global Markov chain which models the chains and the collaboration/competition mechanism. It is possible to use Discrete-Time SANs [7] to represent the competition among the DTMCs. However, Discrete-Time SANs are much more difficult to use than their continuous-time analogs because the distinction between local events and synchronizations is not that clear in discrete time (see [8] for an analysis with Discrete-Time SANs). As the proofs only rely on the generalized tensor product, we specify our models with a set of functional stochastic matrices (formally defined in the next section). We also think that these results will be more easily used if they are independent of a high level modeling formalism. Note that in continuous-time models, the tensor representation of chains have been proved for many formalisms (of course SAN [9], Stochastic Petri Nets [10], and Stochastic Process Algebra [11]). Such a work remains to be done for discrete-time models. The approach adopted to prove product form for Performance Evaluation Process Algebra models (PEPA) is completely different as this process algebra relies on exponential delays for actions (see for instance in [12–14]). It is worthy to link tensor product (denoted as ⊗ in the following) and product form distribution: a distribution of probability π for a multicomponent chain with dimension n has a product form expression if there exist N marginal distributions πs such that π = ⊗Ns=1 πs . Thus, we have a strong relation between product form and tensor operations, and one may ask if it is possible to prove algebraically that:

⊗Ns=1 πs




  N ⊗g F (s) = ⊗Ns=1 πs , s =1

i.e. the steady-state solution of (⊗g Ns=1 F (s) ) is ⊗Ns=1 πs . A first answer is given in Sections 2 and 3.




J.M. Fourneau / Performance Evaluation 67 (2010) 779–796

781

An important drawback of this formulation for the transition probability matrix is that all the states of the Cartesian product of the component must be reachable. We only present two properties (see Propositions 18 and 17) which allow to consider subspaces of a Cartesian product and keep the product form solutions. Further research must be conducted to deal with models where all the states of the Cartesian product are not reachable. The paper is as follows. In the next section, we introduce the generalized tensor product and some of its properties related to eigenvalues and eigenvectors. Section 3 contains our main results about product form steady-state solutions for DTMCs described as a generalized product of functional matrices. We also study the irreducibility and the ergodicity problems. Finally, we consider a more complex model where the transition probability matrix is the summation of generalized tensor of functional matrices. This is a first step toward an analysis of arbitrary SAN. Section 4 is devoted first to a very simple model: a collection of Markov chains competing over one resource. We also study more complex systems with a larger set of resources and more complex competition rules. We also compare results for discrete-time models and continuous-time models as the sufficient conditions are more restrictive for DTMCs. 2. The generalized tensor product and the eigenvector problem We first define the usual tensor and we give some of its algebraic properties. Then we present the generalized tensor operations introduced by Plateau et al. in [5]. We give some algebraic properties of these operators and we prove a sufficient condition for product form. 2.1. Tensor representation Consider the case of a system that may be modelled by two completely independent discrete-time Markov chains. Let us assume that the first (resp. the second) chain has n1 (resp. n2 ) states and transition probability matrix given by Q (1) ∈ Rn1 ×n1 (resp. Q (2) ∈ Rn2 ×n2 ). The state of the overall (two-dimensional) system may be represented by the pair (i, j) where i ∈ {1, 2, . . . , n1 } and j ∈ {1, 2, . . . , n2 }, and the stochastic matrix of the two-dimensional system is given by Q (1) ⊗ Q (2) where ⊗ is the tensor (or Kronecker) product. In general, to define the tensor product of two matrices, A of dimensions (n1 × n1 ) and B of dimensions (n2 × n2 ), it is convenient to observe that the tensor product matrix has dimensions (n1 n2 × n1 n2 ) and may be considered as consisting of (n1 )2 blocks each having dimensions (n2 × n2 ), i.e., the dimensions of B. To specify a particular element, it suffices to specify the block in which the element occurs and the position within that block of the element under consideration. The tensor product C = A ⊗ B is defined by assigning the element of C that is in the (i2 , j2 ) position of block (i1 , j1 ), the value ai1 j1 bi2 j2 . We shall write this as c{(i1 ,j1 );(i2 ,j2 )} = ai1 j1 bi2 j2 . Now given N independent DTMCs, with associated transition probability matrices, Q (1) , Q (2) , . . . , Q (N ) , the stochastic matrix of the N-dimensional system, which we shall refer to as global, is given by: Q = ⊗Nk=1 Q (k) .

(1)

The steady-state probability distribution (if it exists) of the N-dimensional system, π , is given by the tensor product of the steady-state probability vectors of the individual components (i.e. a product form solution). Of course this product form solution is due to independence. But we consider more complex systems where the chains are not independent. 2.2. Functional interactions Now assume that the processes interact as follows. The probability at which a transition occurs may be a function of the states of the other components. Such transitions are called functional transitions. Transitions that are not functional are said to be constant. Definition 1 (Functional Stochastic Matrix). F isP a functional stochastic matrix if and only if the entries of F (denoted as fij ) are nonnegative functions and for each index i, j fij = 1. Functional transitions do not change the structure of the chain except when a probability becomes 0. Due to these functional entries in the stochastic matrices, the chains are not independent. Therefore in general the steady-state distribution does not have product form (see for instance Example 7). To deal with functional matrices, Plateau has introduced a generalization of the tensor products denoted as ⊗g in the following. Definition 2 (Generalized Tensor Product). Assume that matrix A is a function of the state of B and matrix B is a function of the state of A. We denote this dependence as A(B ) and B(A). The generalized tensor product C = A ⊗g B is defined by c{(i1 ,j1 );(i2 ,j2 )} = ai1 j1 (i2 )bi2 j2 (i1 ), because before the transition the state of A is i1 and the state of B is i2 (see [6] for more details).

782

J.M. Fourneau / Performance Evaluation 67 (2010) 779–796

Similarly we define the Generalized Tensor Sum (denoted as ⊕g ) as usual: D = A(B ) ⊕g B(A) ⇔ D = A(B ) ⊗g IdB + IdA ⊗g B(A), where IdM is the identity matrix the size of which is equal to the size of matrix M. This generalization of tensor product and tensor sum allows to build the transition rate matrix of continuous-time SANs [6,5], discrete-time SANs [15] or other components-based models with transition rates or probabilities may be functional (see for instance [11] for a new version of PEPA with functions and recent results [16]). Assumption 1. Here we assume that the stochastic matrix of the Markov chain is built as follows: N

Q = ⊗g F (s)

(2)

s=1

where F (s) is the functional stochastic matrix associated with component (or chain) s. We also assume that the state spaces are denumerable. Note that the generalized tensor product is still well defined and the tensor representation still valid if the matrices are infinite. The generalized tensor product and sums have many algebraic properties [6]. We give two of them in the following for the sake of completeness. Proposition 1 (Basic Properties of Generalized Tensor). Let A[B , C ], B[A, C ] and C [A, B ] be three arbitrary functional stochastic matrices, the following properties hold (see [6] for a proof):

• Associativity: A[B , C ] ⊗g B[A, C ] ⊗g C [A, B ] =
A[B , C ] ⊗g B[A, C ] ⊗g C [A, B ] .




• Distributivity over Addition: (A1 [B ] + A2 [B ]) ⊗g = + + +

(B1 [A] + B2 [A]) A1 [B ] ⊗g B1 [A] A1 [B ] ⊗g B2 [A] A2 [B ] ⊗g B1 [A] A1 [B ] ⊗g B2 [A].

A model is described by the set of functional matrices. For the sake of readability and to help the composition process, we describe a functional matrix as A(x) and we then relate x to the state of the other components. We now build the functional dependency graph (see Fig. 2). It is a directed graph the nodes of which are the components. A directed edge (x, y) in the functional dependency graph models that component x uses the state of component y to compute some of its transition probabilities. For the sake of simplicity we assume that the functional dependency graphs do not contain self-loops. The main result we obtain requires that the functional dependency graph is a Directed Acyclic Graph (DAG in the following). 2.3. A toy example We present two small examples of the composition of functional stochastic matrices. Example 1. Consider the functional matrix: 1 − x/2 0 x

" A(x) =

x/4 1 − x/2 0

x/4 x/2 . 1−x

#

If 0 < x < 1, A(x) is finite and irreducible. Thus it has a steady-state distribution. One can easily check that this distribution is (1/2, 1/4, 1/4) for any value of x. Note that it is not true in general. States of A are labelled 1, 2 and 3. Example 2 (Second Matrix). Consider the functional matrix: B(y) =



1 − y/2 1/4

y/2 . 3/4



If 0 < y < 2, B(y) is finite and irreducible. Thus it has a steady-state distribution for every value of y and this steady-state vector is (1/(2y + 1), 2y/(2y + 1)). States of B are labelled 1 and 2.

J.M. Fourneau / Performance Evaluation 67 (2010) 779–796

783

Fig. 2. Functional dependency graphs of the two models.

Example 3 (Composition of A(x) and B(y)). We just have to link x and y to the states of B and A. We assume that x is the state of B divided by 4 and y is the state of A divided by 2. The transition matrix P associated with the generalized product of the two functional matrices is:

0.65625 0.1875 0  0  0.0625 0.125

0.21875 0.5625 0 0 0.1875 0.375

0.046875 0.03125 0.4375 0.1875 0 0

0.015625 0.09375 0.4375 0.5625 0 0

0.046875 0.03125 0.0625 0.0625 0.1875 0.125

0.015625 0.09375  0.0625  . 0.1875   0.5625 0.375



Example 4 (Third Matrix). Consider now the constant (i.e. nonfunctional) matrix C = Its steady-state distribution is (1/3, 2/3). States of C are labelled 1 and 2.

h

3/4 1/8

1/4 7/8

i

. C is finite and irreducible.

Example 5 (Composition of A(x) and C ). We use the same composition rule for x: (i.e. x is the state of C divided by 4). And we obtain transition matrix P2 using the generalized tensor product:

0.65625 0.09375 0  0  0.1875 0.0625

0.21875 0.65625 0 0 0.0625 0.4375

0.046875 0.015625 0.65625 0.09375 0 0

0.015625 0.109375 0.21875 0.65625 0 0

0.046875 0.015625 0.09375 0.03125 0.5625 0.0625

0.015625 0.109375 0.03125  . 0.21875   0.1875 0.4375



The functional dependency graphs of both models are depicted in Fig. 2. Both have two nodes. For the first composition (graph on the left), we clearly obtain a complete directed graph with two nodes because both matrices are functional. The directed graph of the second model is a simple directed edge from node A to node C . 2.4. Eigenvector We begin with some lemmas and properties to show the difference between the usual tensor product (⊗) and the functional tensor product (⊗g ). Proposition 2 (Compatibility with Matrix Multiplication). For all matrices A and B and vectors πA and πB whose sizes are consistent:

(πA ⊗ πB ) × (A ⊗ B) = (πA × A) ⊗ (πB × B). where × is the ordinary product (see [6]). Proposition 3. Let πA be an eigenvector of A associated with eigenvalue λA and πB an eigenvector of B associated with eigenvalue λB , then πA ⊗ πB is an eigenvector of A ⊗ B with eigenvalue λA λB . Proof. Simple algebra.

(πA ⊗ πB ) × (A ⊗ B) = (πA × A) ⊗ (πB × B) = (λA πA ) ⊗ (λB πB ) = λA λB (πA ⊗ πB ).  Unfortunately these properties do not hold in general when we use the generalized tensor product for functions. However it is possible to prove some results with some assumptions on the dependency graph. Proposition 4. Let B be a positive matrix, let A(B ) be a matrix whose elements are functions of the index of B. Assume that w is an eigenvector of B with eigenvalue λ. Assume that for all states s of B, A(s) has an eigenvector v associated with eigenvalue µ. Then we have:


(v ⊗ w) × A(B ) ⊗g B = λµ (v ⊗ w) .

784

J.M. Fourneau / Performance Evaluation 67 (2010) 779–796

Proof. By algebraic manipulation to compute an arbitrary entry of vector h defined as (v ⊗ w) × (A(B ) ⊗g B). Let us define u = v ⊗ w and X = A(B ) ⊗g B. The elements of u and X are: u{(r );(s)} = vr ws and x{(r ,i);(s,k)} (i) = ari (s)bsk . Strictly speaking x is not a functional term here but we keep this notation to emphasise that x is obtained from the functional term ari (s). The resulting elements of vector h are: h{(i);(k)} =

XX r

u{(r );(s)} x{(r ,i);(s,k)} (s).

s

After substitution we get: h{(i);(k)} =

XX r

vr ws ari (s)bsk .

(3)

s

We can exchange the two summations as the matrices are positive. h{(i);(k)} =

X

ws bsk

X

s

vr ari (s).

(4)

r

Now we use the assumption that forP every value of s we have: µvi = r vr ari (s) for any index i. Thus after substitution and factorization, we get: h{(i);(k)} = µvi s ws bsk . Then we remember the first assumption about the eigenvalue and eigenvector of matrix B and we obtain to complete the proof:

P

h{(i);(k)} = λµvi wk . Remark 5. Note that the result does not apply if both matrices are functional. Indeed it is not possible to perform the factorization between Eqs. (3) and (4) if a() is a function of s and b() is a function of r. We have a similar property for continuous-time models. As the proof is similar to the proof of Proposition 4, it is omitted. Corollary 6. Let B be a transition rate matrix, let A(B ) be a transition rate matrix whose elements are functions of the index of B. Assume that w is in the kernel (or nullspace) of B (i.e. w B = 0). Assume that for all
states s of B, the kernel of A(s) contains a positive vector v which does not depend of s. Then we have: (v ⊗ w) × A(B ) ⊗g IdB = 0, and (v ⊗ w) × (IdA ⊗ B) = 0. We now generalize the property to a product of an arbitrary number of matrices. Proposition 7. Consider N stochastic functional matrices. Assume that the functional dependency graph is a DAG, and order the matrices with a topological ordering associated with the DAG. Thus, matrix AN does not use any function, matrix AN −1 may use functions whose argument is the index of matrix AN . In general matrix Al uses functions whose arguments may use index of matrices Al+1 to AN . Assume that for each i, there exist an eigenvector πi and an eigenvalue λi such that: πi Ai [x] = λi πi for every argument x. Then we have:

(π1 ⊗ π2 . . . ⊗ πN ) × ⊗g ⊗g ⊗g ⊗g

[A1 (A2 , . . . , AN ) A2 (A3 , . . . , AN )

...

AN −1 (AN ) AN ]

= (π1 ⊗ π2 ⊗ · · · ⊗ πN )

N Y

λi .

i =1

Proof. By induction on the number of matrices. Proposition 4 shows that the property is true for 2 matrices. Assume now that is it true for N − 1. That is:

(π2 ⊗ · · · ⊗ πN ) × ⊗g ⊗g ⊗g ⊗g

[A2 (A3 , . . . , AN ) A3 (A4 , . . . , AN )

...

AN −1 (AN ) AN ]

= (π2 ⊗ · · · ⊗ πN )

N Y

λi .

i =2

And let us define matrix B as follows: B = A2 (A3 , . . . , AN ) ⊗g · · · ⊗g AN −1 (AN ) ⊗g AN .

J.M. Fourneau / Performance Evaluation 67 (2010) 779–796

Clearly B is a constant matrix with eigenvalue ( product is associative:

QN

i=2

785

λi ) and eigenvector (π2 ⊗· · ·⊗πN ). Furthermore the generalized tensor

A1 (A2 , . . . , AN ) ⊗g A2 (A3 , . . . , AN )

⊗g ⊗g ⊗g =

...

AN −1 (AN ) AN (A1 (B ) ⊗g B).

The assumptions of Proposition 4 are still satisfied and we apply the proposition to complete the proof. Let us now turn back to models in continuous time and the associated generalized tensor sum. Proposition 8. Let A(B ) and B(A) be arbitrary functional transition rate matrices. Assume that w is in the kernel of B(y) for every y and that w is positive. Similarly assume that there exists a positive vector v which is in the kernel of A(x) for all x. Then we have:


(v ⊗ w) × A(B ) ⊕g B(A) = 0. Proof. First we apply the definition of the generalized tensor sum: A(B ) ⊕g B(A) = A(B ) ⊗g IdB + IdA ⊗g B(A) .







Because of distributivity we obtain:




(v ⊗ w) × A(B ) ⊕g B(A) = (v ⊗ w) × A(B ) ⊗g IdB + (v ⊗ w) × IdA ⊗g B(A) .
Now we can apply Corollary 6 to the first term (i.e. (v ⊗ w) × A(B ) ⊗g IdB . Clearly, the assumptions of that corollary are satisfied as the functional dependency graph for this term is a simple directed edge from A to IdB , w is an eigenvector of IdB with eigenvalue 1, v is an eigenvector of A(x) for every value of x and the associated eigenvalue is always 0. Thus:
(v ⊗ w) × A(B ) ⊗g IdB = 0.
Using similar arguments, one can prove that (v ⊗ w) × IdA ⊗g B(A) = 0 and the proof is completed. Thus for continuoustime models we do not need an assumption on the functional dependency graph. 3. Generalized tensor product and product form Following the idea developed in [17] we index the matrix of functions rather than factorize the function. Thus instead of F (s) we consider the set of matrices obtained when we vary the functions. We index the matrix with m and denote the matrices as Q (l,m) for the mth matrices in the set of matrices associated with chain numbered l. We have used in [17,18] the set of all matrices with a vector in their kernel. Let us first give some definitions and examples before we can state the main result. Definition 3. Let α be a probability distribution. We note by S (α) the set of transition rate matrices M such that α M = 0 (i.e. α is in the kernel of all matrices in S (α)). Example 6. Consider A(x) defined in Example 1 and the definition of x in the composition with B used in Example 3. Remember that x is the state of B divided by 4 and B has two states: 1 and 2. Thus we obtain two constant matrices for the functional matrix A(x): A(1) and A(2) such that: 7/8 0 1/4

" A

(1)

=

1/16 7/8 0

1/16 1/8 3/4

#

3/4 0 1/2

" (2)

A

=

1/8 3/4 0

1/8 1/4 . 1/2

#

Furthermore for every x such that A(x) is ergodic, vector ν = (1/2, 1/4, 1/4) is the stationary vector. Thus ν is in the kernel of matrix (A(x) − Id). For all x such that 0 < x < 1, matrix (A(x) − Id) is in S (ν). We keep the same notation and definition about S (α) to be consistent with reference [17] but for discrete-time models the following representation is easier to understand. Proposition 9. Let A and B be stochastic matrices. If (A − I ) and (B − I ) are both in set S (α), then α is an eigenvector of both matrices A and B associated with eigenvalue 1. Proposition 10. Let A be a stochastic matrix. If (A − I ) is in set S (α), then α is an eigenvector of: (1) matrix Ak for any integer k, (2) matrix pI + (1 − p)Ak for any integer k and probability p.

786

J.M. Fourneau / Performance Evaluation 67 (2010) 779–796

Definition 4 (Separable). A functional stochastic matrix A(s) is said to be separable iff we have: A(s) = Id + f (s)C where C is a constant matrix and f () is a real nonnegative function. The functional stochastic matrices presented in the former examples are separable. Note that it is easy to prove that C is a transition rate matrix. Furthermore: Proposition 11. Let A(s) = Id + f (s)C be a functional separable stochastic matrix. Assume that vector u is in the kernel of C ; Then, we have, for all s: uA(s) = u. Proof. Indeed uA(s) = uId + uf (s)C = u + f (s)uC = u. We do claim that it is easy to find stochastic matrices which share a common dominant eigenvector (or transition rate matrices with a common vector in their kernels). But we want to emphasise that this property has already been used in queueing theory and it may be worthy to explore again this idea. For instance (the proof may be found in many textbooks).  Proposition 12. Let A be an ergodic stochastic matrix with stationary vector α . Let us define as usual the reversed process and αA

its stochastic matrix by Mij = αj ji , then α M = α . i The second example is even more connected to the core of product form results already known. Definition 5 (Partial Balance Decomposition). A stochastic matrix A admits a partial balance decomposition ((Di )i=1..d ) iff it Pd exists a positive vector α such that A = Id + i=1 Di where for all i, Di is a transition rate matrix in S (α). Clearly α A = α . The decomposition of the matrix proposed in this last definition is the algebraic counterpart of the partial balance equations technique used in many proofs of product form steady-state solution. In the next section we will present two main application of this algebraic results. The product form for DTMCs in competition relies on separable DTMCs while the collaboration of DTMCs is based on Proposition 10. Let us now present the main theoretical result we need to prove that DTMCs in competition have a product form distribution. Note that this theorem can be used in a more general framework than DTMCs in competition or in collaboration as it addresses collection of functional stochastic matrices. We will show in the next section how we can use this result to analyze competition or collaboration of DTMCs. Theorem 1. Consider a collection of N functional stochastic matrices such that:

• The functional dependency graph is a DAG. • For every chain index l there exists a positive vector πl such that for every matrix index m, matrix (Q (l,m) − Id) is in S (πl ). • The Markov chain associated with the composition of these functional matrices is ergodic. Then the steady-state distribution has product form: ⊗Nl=1 πl . Proof. Due to Assumption 3 we know that the steady state exists (Assumption 3). Thus we just have to check that the tensor product of the πl is a stationary vector. We apply Proposition 7. As the functional dependency graph is a DAG, we can reorder the components such that the transition probability matrix is: F (1) (A2 , . . . , AN ) ⊗g · · · ⊗g F (N −1) (AN ) ⊗g F (N ) . We evaluate each of the F (l) to obtain the set of matrices Q (l,m) . Now consider the second assumption. If for every matrix index m, Q (l,m) − Id is in S (πl ), we have:

πl (Q (l,m) − Id) = 0. Therefore we get: πl Q (l,m) = πl for all m. Thus the assumptions of Proposition 7 are satisfied with λi = 1 for all i. It implies that the tensor product of the πl is the stationary vector. 3.1. Irreducibility Note that of matrices does not imply the irreducibility of their tensor product. For instance consider matrix h irreducibility i

M equal to

0 1

1 0

. One can easily check that M is irreducible but M ⊗ M is not (Fig. 3).

However it is quite simple to prove some sufficient conditions for irreducibly of the tensor product. Proposition 13. Consider stochastic matrices M1 , . . . , Mn . Assume that all matrices are irreducible and that every matrix contains a nonzero diagonal element, then matrix M1 ⊗ M2 ⊗ · · · ⊗ Mn is irreducible.

J.M. Fourneau / Performance Evaluation 67 (2010) 779–796

787

Fig. 3. Reducibility of M ⊗ M, Automata (on the right), resulting Markov chain (on the left).

Fig. 4. Irreducibility of M ⊗ M, Automata (on the right), resulting Markov chain (on the left).

Fig. 5. Irreducibility of M1 ⊗ M2 with only one loop in M1 , Automata (on the right), resulting Markov chain (on the left).

Proof. Without loss of generality we assume that for all matrices the nonzero diagonal element is in the first row (indexed by 0). By induction on the number of matrices, we can prove that matrix M2 ⊗ M2 ⊗ · · · ⊗ Mn is irreducible. Furthermore the first diagonal elements of matrix M2 ⊗ M2 ⊗ · · · ⊗ Mn is nonzero. Thus we only have to consider the case of two matrices to begin with the induction. Consider M1 ⊗ M2 . First let us prove that there is a path from an arbitrary node (x, y) to node (0, 0). As the second matrix is irreducible there is a path from y to 0. During these steps the first component moves (because of the tensor product use to describe the product process) from x to a some state z. Now, we continue with transitions of the first component from z to 0 and during these transitions the second component perform a loop in state 0. This is possible because entry (0, 0) of M2 is positive. Thus, there is a path from an arbitrary node (x, y) to node (0, 0) (see for instance Fig. 4). Let us now prove that we also have a path in the product process from state (0, 0) to an arbitrary state (x, y). We consider M1t and M2t (i.e. the transpositions of matrices M1 and M2 ). The directed graphs associated with M1t and M2t are obtained from the directed graphs associated M1 and M2 reversing the orientation of the arcs. Thus finding a path from (0, 0) to (x, y) in the product process defined by M1 and M2 is equivalent to finding a path from (x, y) to (0, 0) in the product process defined by M1t and M2t . Clearly the assumptions on M1 and M2 are also satisfied by M1t and M2t . Therefore, we can apply the previous part of the proof to find a path from (x, y) to (0, 0) and complete the proof.  Finally, note that the assumptions of Proposition 13 are only sufficient. The chain in Fig. 5 is irreducible while only the first diagonal entry of M1 is nonzero. These facts suggest that we can simplify the assumptions of Proposition 13 keeping the same proof and obtain the following claim. Indeed the fact that M1 has a nonzero diagonal entry is not used in the proof of Proposition 13. As one can reorder the automata we finally obtain the following proposition. Proposition 14. Consider stochastic matrices M1 , . . . , Mn . Assume that all matrices are irreducible and that every matrix except one contains a nonzero diagonal element, then matrix M1 ⊗ M2 ⊗ · · · ⊗ Mn is irreducible. 3.2. The DAG assumption Let us now try to remove this assumption on the functional dependency graph. We shall see now with the following example that if the DAG assumption is not satisfied we do not have a product form steady-state solution. First, let us define the matrices and verify the other assumptions of the theorem. Example 7. We consider functional matrix A(x) already defined in Example 1 and functional matrix D(y) =

1 − y/2 y/2 y/4 1 − y/4

h

i

.

If y is between 0 and 2, D(y) is finite and irreducible. It has a steady-state distribution: (1/3, 2/3) and it does not depend of the value of y. Now we build the SAN with the following two automata A1 and A2 .

• States of A1 are labelled 1, 2 and 3. • States of A2 are labelled 1 and 2.

788

J.M. Fourneau / Performance Evaluation 67 (2010) 779–796

• Transition matrix of A1 is functional matrix A(x) where x is the state of A2 divided by 4. • Transition matrix of A2 is functional matrix D(y) where y is the state of A1 divided by 2. Clearly the second and the third assumptions of Theorem 1 are satisfied. Let us now compute numerically the stochastic matrix and its dominant eigenvector. The transition matrix of the SAN is:

0.65625

0.21875 0.65625 0 0 0.1875 0.3125

0.09375 0  0  0.0625 0.1875

0.046875 0.015625 0.4375 0.1875 0 0

0.015625 0.109375 0.4375 0.5625 0 0

and its steady-state distribution is π = 0.1923 the two steady-state distribution is:



 πA ⊗ πD = 1/6

1/3

1/12

1/6

0.046875 0.015625 0.0625 0.0625 0.1875 0.1875

0.3169

1/12

0.015625 0.109375 0.0625   0.1875   0.5625 0.3125

0.0803



0.1664

0.0751

0.1690 , but the tensor product of



1/6 .



Clearly the two distributions differ. The SAN does not a have a product form distribution. The dependency graph is a complete directed graph on two states and this is not a DAG. The example proves that the last two assumptions of Theorem 1 are not sufficient. 3.3. Reversibility Let us begin with a simple property about reversibility whose proof is omitted. Proposition 15. Let A and B be two stochastic matrices associated with reversible ergodic discrete-time Markov chains, then if a Markov chain associated with matrix A ⊗ B is ergodic, it is reversible. Definition 6 (Global Separability). Let A(B ) and B(A) be functional matrices. We say that the model satisfies the global separability property iff: we can find two stochastic matrices C and D and a function f whose arguments are the initial and final states of the transition, such that: A(B ) ⊗g B(A) = (C ⊗ D)f .

(5)

This relation simply states that we can separate the functional part represented by f and the stochastic part represented by C and D. This means that:

(A(B ) ⊗g B(A)){(r ,i);(s,k)} = cri dsk f (r , s, i, k). Proposition 16. Let A(B ) and B(A) be functional matrices and assume that the model they define satisfies the global separability property. Furthermore assume that:

• matrices C and D are reversible, • v (resp. w ) is the dominant eigenvector for C (resp. D), • f (i, k, r , s) = f (r , s, i, k) for all i, r, k, s, then:

(v ⊗ w)(A(B ) ⊗g B(A)) = (v ⊗ w). Proof. Let h be equal to (v ⊗ w)(A(B ) ⊗g B(A)). With the usual notation we get: h{(i);(k)} =

XX r

vr ws cri dsk f (r , s, i, k).

(6)

s

Using the reversibility of matrix C (i.e. vr Cri = vi Cir ) and matrix D we get: h{(i);(k)} =

XX r

vi wk cir dks f (r , s, i, k).

(7)

s

Taking into account the assumption on function f , we obtain after substitution and factorization: h{(i);(k)} = vi wk

XX r

s

cir dks f (i, k, r , s).

(8)

J.M. Fourneau / Performance Evaluation 67 (2010) 779–796

789

As matrix (C ⊗ D)f must be stochastic, we must have:

XX r

cir dks f (i, k, r , s) = 1.

s

Thus h{(i);(k)} = vi wk and the proof is complete.



This property leads to a sufficient condition for product form. We have a more general graph of functional dependency but we now have constraints on the matrices and a symmetry in the functional dependency. We can even consider a weaker condition. Definition 7 (Partial Reversibility). Let A be a stochastic matrix associated with an ergodic Markov chain with state space S . Let π be its stationary vector. We consider a proper partition of S into S1 and S2 . We say that A is partially reversible according to that partition iff for all states i in S1 and j in S2 we have:

π (i)aij = π (j)aji . The usual reversibility is obtained if we consider a partition of S into singletons. We now see how the functional transitions preserve the product form. Proposition 17. Let M be a stochastic ergodic matrix which satisfies the partial reversibility property for partition (S1 , S2 ). Let π be its stationary vector. Assume that π has product form: π = K 1(⊗i πi ), where K 1 is a normalization constant. Consider the block decomposition of M according to the partition:

 M =

A C



B . D

Let P N be the stochastic matrix defined by N = A + E, where E is a diagonal matrix whose element E (i, i) is equal to 1 − j∈S1 A(i, j). Then if the stationary vector of N exists, it has product form (i.e. is it of the form K 2(⊗i πi ), where K 2 is a normalization constant). Proof. Consider an arbitrary state i in S1 and write the balance equation in i:

π (i) =

X

π (j)mji .

j∈S

Perform the decomposition according to the partition:

π (i) =

X

π (j)mji +

X

π (j)mji .

j∈S2

j∈S1

M is partially reversible according to the partition considered. Thus for every j in S2 , we have π (j)mji = π (i)mij . We substitute in the former equation and we factorize to obtain:

π (i) =

X

π (j)mji + π (i)

j∈S1

X

mij .

j∈S2

Taking into account the block decomposition we get:

! π (i) =

X

π (j)aji + π (i) 1 −

j∈S1

X

aij

.

j∈S1

After identification and taking into account the definition of matrix E we finally obtain:

π (i) =

X

π (j)aji + π (i)eii =

j∈S1

X

π (j)(aji + eji ).

j∈S1

Thus π is also a dominant eigenvector of A + E which is suitably normalized to obtain a stationary vector of probability. Therefore the solution still has product form after the truncation of the process. This proposition is a simple generalization of the well-known result proved by Kelly on the truncation of reversible Markov chain with product form [19]. The main idea is that the functional transitions allows to obtain N.  Example 8. Consider chain X1 (resp. X2 ), both with states {0, 1, 2} and transition matrix M1 (resp. M2 ): 0.5 2/3 0

" M1 =

0.5 0 1/3

0 1/3 2/3

#

" M2 =

0 0.5 0

1 0 0.5

0 0.5 . 0.5

#

790

J.M. Fourneau / Performance Evaluation 67 (2010) 779–796

Matrix M1 ⊗ M2 has a product form dominant eigenvector as the chains are independent. Now assume that the functional transitions make the following truncation: we keep in S1 the states such that X1 + X2 ≤ 3. As the chain is reversible it is also partially reversible according to the partition (S1 , {(2, 2)}). Matrix N is given by:

  1/4    N =  1/3   

1/2

1/4 1/4

1/4 2/3

1/4

1/3 1/3

1/3

1/2 1/4 1/6

1/6

1/3

 1/4 1/4 1/6 1/6

1/6 1/3

   1/3  .   1/6   2/3 1/3

The chain associated with N has a product form steady-state distribution according to Proposition 17 and the solution is obtained after truncation of the last element and normalization of (4/10, 3/10, 3/10) ⊗ (1/5, 2/5, 2/5). Finally we give an old result which is not related to reversibility or product form but which can help to find the solution of a truncated process. Let us first recall the definition of a censored Markov chain. Consider a DTMC {Xt : t = 1, 2, . . .} with finite state space S . Suppose that (E , E c ) is a proper partition of S . Suppose that the successive visits of Xt to E take place at time epochs 0 < t1 < t2 < · · · <. Then the chain {XuE = Xtu , u = 1, 2, . . .} is called the censored process (or chain) with censoring set E [20]. Let Q denote the transition matrix of chain Xt . The partition of the state space induces a block description of Q :

 Q =

QE QE c E



QEE c . QE c

(9)

The censored chain only watches the chain when it is in E. If matrix Q is ergodic, it can be proved [20] that the stochastic matrix of the censored chain is: ME = QE + QEE c

! ∞ X i (QE c ) QE c E .

(10)

i =0 i −1 c c Assume that (QE c ) does not contain any recurrent class, the fundamental matrix is . Censored i=0 (QE ) = (I − QE ) Markov chains have also been called restricted or watched Markov chains. When the chain is ergodic there are strong relations with the theory of stochastic complement [21]. When Q is irreducible, the transition probability matrix of the censored chains is the stochastic complement matrix defined by Meyer and we have the following results [21]:

P∞

Proposition 18. If Q is irreducible, with steady-state distribution πQ = (πE , πE c ) the steady-state distribution ΠME of the censored matrix ME is given by:

πME = P

πE . πE (i)

(11)

i∈E

Therefore if Q has a stationary vector with product form, ME satisfies the same property. We only have to change the normalization constant. Propositions 17 and 18 allow to find subspaces where we still have product form but we need more work to deal with a general state space which is not a Cartesian product. We also have to study more general models with a complete tensor representation or synchronizations (see for instance [22]). 3.4. A first step toward a complete tensor representation (s)

N We consider now that the stochastic matrix is equal to: Q = where τi is an arbitrary probability distrii=1 τi ⊗g s=1 Fi bution. We present a very simple generalization of Theorem 1. First we have to define more precisely the functional dependency graph as it is used in PEPS [23,5]. In Section 2, we built one functional dependency graph (FDG) for the product of the automata. In general we build one graph for each synchronization. Now we will follow this more detailed description of the model. Note that this new description was useless for Theorem 1 because we only have one line of tensor products while now we must consider several rows of them. PS (s) N Thus for matrix Q = i=1 τi ⊗g s=1 Fi , we design S FDG. Each one is associated with a value of the index in the summation. We require that each FDG is a DAG but the resulting directed graph obtained by superposition of these directed graphs is not necessarily a DAG.

PS

(s)

N Theorem 2. Consider a model described by Q = i=1 τi ⊗g s=1 Fi a collection of functional stochastic matrices such that:

PS

(s)

where τi is an arbitrary probability distribution and Fi

are

J.M. Fourneau / Performance Evaluation 67 (2010) 779–796

791

• The functional dependency graphs (one for each synchronization index i) are Directed Acyclic Graph. • For every chain index l there exists a positive vector πl such that for every matrix index m, matrix (Q (l,m) − Id) is in S (πl ). • The Markov chain associated with the composition of these functional matrices is ergodic. Then the steady-state distribution has product form. Proof. Consider the product: h = (⊗

N i =1

πi )

S X i=1

N

(s)

!

τi ⊗g Fi

.

s=1

Because of the distributivity property we get: h=

S X

  N τi (⊗Ni=1 πi ) ⊗g Fi(s) . s =1

i=1

(s)

We now have S expressions and all of them satisfy the assumptions of Theorem 1. Therefore we have (⊗Ni=1 πi )(⊗g Ns=1 Fi ) = ⊗Ni=1 πi and we obtain: h=

S X

τi (⊗Ni=1 πN ) = (⊗Ni=1 πN )

i=1

S X

τi .

i =1

N As i=1 τi = 1, we get: h = ⊗i=1 πN . We assume that the chain is ergodic, then it has a steady-state solution and this solution has product form. 

PS

4. Competition and collaboration Let us now give some examples based on the theory we have developed. We say that the chains are in competition or are collaborating because the number of transitions of a chain may depend of the state of another chain. When the number of transitions is zero, we have a clear competition: one chain is not allowed to move because the other one owns some resource. This is exactly the model described by Boucherie in [4]. When the number of transitions is not zero, we prefer to call that model a collaboration between the chains. 4.1. A simple competition over one resource We considered a collection of DTMCs and we introduce a product process which has a tensor representation and some restrictions on the chain due to the competition over resources. The competition over resources is modelled as the cancellation of some transitions because a chain is not allowed to move if a resource is owned by another chain. For instance, assume that chains X 1 and X 2 compete over one resource, if X 1 has the resource, all the transitions of chain 2 are cancelled except self-loops. Fig. 6 presents two chains X 1 and X 2 respectively with states {0, 1, 2, 3} and {0, 1, 2} competing over one resource. The resource is owned by X 1 (resp. X 2) when it is in state 2 or 3 (resp. state 2). It is released in both chains when the chain jumps from state 2 or 3 to 1. Note that this is exactly the same state space for chains X 1 and X 2 we have already used in Fig. 1 to illustrate the continuous-time models. We have just added some loops which are permitted in discrete-time models. When chain X 1 is in state 2 or 3, chain X 2 is stopped and the only transitions allowed are self-loops. If chain X 1 is in state 0 or 1, chain X 2 can move. We do not have the symmetrical rules when X 2 owns the resource. X 2 cannot block X 1. When X 2 owns the resource, X 1 can move and if it jumps to state 2 it takes the resource and X 2 is now blocked. In Fig. 6 (as in Fig. 1), we have depicted first the chains X 1 and X 2 on the top and the product process when the chains are competing over a resource. Let us turn now to the transition matrices involved in this model. When X 1 is in state 1 or 0 the transition probability matrix of X 2 is: a c 0

" B1 =

1−a 0 b

0

#

1−c , 1−b

while it is the identity when X 1 is in state 2 or 3. Let A be the transition matrix for chain X 1. Let A1 (resp. A2 ) be the 4 × 4 matrix with the first (resp. last) two rows of A. Clearly A = A1 + A2 . The transition matrix of the chain of the two DTMCs in competition is A1 ⊗ B1 + A2 ⊗ Id. Let u be the dominant eigenvector of A and v the dominant eigenvector of B1. Every eigenvector of B1 is also trivially an eigenvector of the identity. Clearly the transition probabilities in X 2 depend on the state of X 1 while the transitions in X 1 are not dependent on those of X 2. Thus, the functional dependency graph is a DAG and the Markov chain has a product form steady-state solution

792

J.M. Fourneau / Performance Evaluation 67 (2010) 779–796

Fig. 6. Two DTMCs in competition.

because we can apply Theorem 1. To illustrate the approach one can also verify that u ⊗ v is a stationary vector of the chains in competition. By distributivity, we get:

(u ⊗ v)(A1 ⊗ B1 + A2 ⊗ Id) = (u ⊗ v)(A1 ⊗ B1 ) + (u ⊗ v)(A2 ⊗ Id). Using the compatibility with the product, we have:

(u ⊗ v)(A1 ⊗ B1 + A2 ⊗ Id) = ((uA1 ) ⊗ (v B1 )) + ((uA2 ) ⊗ (v Id)). Because v is an eigenvector, we obtain:

(u ⊗ v)(A1 ⊗ B1 + A2 ⊗ Id) = ((uA1 ) ⊗ v) + ((uA2 ) ⊗ v). After factorization,

(u ⊗ v)(A1 ⊗ B1 + A2 ⊗ Id) = (u(A1 + A2 )) ⊗ v = (uA) ⊗ v = u ⊗ v. We can generalize this result to any number of components competing over one resource. The scheduling discipline must be based on a strict priority among the components. Here X 1 can block X 2 but cannot be blocked by X 2. Note that this is clearly different from the Competing Markov Chain model in continuous time proposed by Boucherie in [4] and recently extended in [17] which does not require any constraint on the functional dependency graph to obtain a product form steadystate distribution. Furthermore, competition of CTMCs are not based on preemption of the resource. We consider an arbitrary number of DTMCs X1 , . . . , Xn competing over one resource. We assume a strict priority among the chains in competition: Xi has priority over Xj iff i < j. When Xi owns the resource, all the chains Xj with j > i cannot move, while chain Xk with k < i can continue to move and may preempt the resource. If Xk takes the resource previously owned by Xi (with k < i), then Xi is now blocked even if Xi still has the resource. It must be clear that the resource ownership is not sufficient to move. Only the highest priority chain which owns the resource can move. Of course when the resource is not owned by anyone, the chains evolve independently. Let Mi be the stochastic matrix when the chain is allowed to move. We call Mi the stochastic matrix of chain Xi in isolation. Let Si be the state space of chain Xi . Assume that we can divide Si into a proper partition (A0i , A1i ) such that A0i is the set of states where chain Xi owns the resource. Markov chains in competition can clearly be modelled by functional stochastic matrices: when the chain can move, its transitions follow the usual stochastic matrix while when the chain is blocked, its transition matrix is the Identity (noted here IdMi to keep in mind it has the size of matrix Mi ). As chain Xi is only blocked when a chain with smaller index owns the resource, one can easily write the functional transition matrix of Xi as: i−1 Y j =1

! 1Xj ∈A1 Mi + 1 − j

i−1 Y j =1

! 1Xj ∈A1 IdMi , j

(12)

where 1() is the indicator function. Of course, the transition matrix of X1 is simply M1 as X1 has the highest priority. Clearly, the functional dependency graph is a DAG. Furthermore assume that vi is an eigenvector of Mi , it is also an eigenvector of IdMi . Thus, the assumptions of Theorem 1 (except ergodicity) are satisfied. Let us now state some sufficient conditions to obtain an ergodic chain.

J.M. Fourneau / Performance Evaluation 67 (2010) 779–796

793

Fig. 7. M ⊗ M and directed graph of the DTMCs in competition.

Assumption 2. Consider the following assumptions:

• • • • •

H1: We consider a proper partition of the state space into two sets A0i and A1i : i.e. for all i, A0i 6= ∅ and A1i 6= ∅, H2: for all i, matrix Mi is irreducible, H3: for all i, there exists a state xi in A1i and a state yi in A0i such that Mi (xi , xi ) > 0 and Mi (xi , yi ) > 0, H4: for all i, the restriction of Mi to the states of A0i is irreducible, H5: for all i, the restriction of Mi to the states of A1i is irreducible.

Remark 19. Remember that H2 is not sufficient to obtain an irreducible h i matrix for the tensor product. However the situation is even more complex for the chains in competition. Let M be

0 1

1 0

.We have depicted in Fig. 7 the directed graph of M ⊗ M

(on the left) and the directed graph of X 1 competing with X 2 assuming that both chains have M as transition matrix, that X 1 has priority over X 2, that the initial state is (0, 0), and that the resource is owned when a chain is in state 1. Clearly the graph of the chains in competition is strongly connected while the graph of the tensor product is not. Let us now prove that the chain of the DTMCs in competition is ergodic and has product form steady-state distribution. Theorem 3. Consider N finite DTMCs in competition over a resource such that assumptions H1 to H5 are satisfied. Without loss of generality we now assume that state xi defined in H3 is state 0. Then, the Markov chain associated with the competition is ergodic and has a product form steady-state distribution. Proof. • First, as all the chains are finite, the Cartesian product of their state spaces is finite and we only have to check that the chain is irreducible and aperiodic. • As all chains have a self-loop in state 0, we also have a transition from state (0, . . . , 0) to itself. Therefore the chain of the DTMCs in competition is also aperiodic. • Consider an arbitrary state (z1 , . . . , zN ). We prove that there exists a path in the directed graph of the chain from (z1 , . . . , zN ) to (0, . . . , 0) using the same argument as in Proposition 13. • We now prove that it is now possible to reach any state (z1 , . . . , zN ) from state (0, . . . , 0). Assume that zn 6= 0. Chains X1 to XN −1 perform loops at state 0 while chain XN follows a path to state zN . But during the last transition of this path, chain XN −1 jumps to a state in A0N −1 and chain XN is now blocked in state zn . We can now continue by induction on the chain index moving every component to zi and then blocking the component at that state because a higher priority chain takes the resource. If zn = 0, we begin by the chain with the largest index i such that zi 6= 0 and we proceed with the same induction. All the transitions in this proof have positive probability because of assumption H3. • As each Mi is finite and irreducible and contains a self-loop at 0, it is ergodic and has a steady-state distribution which is a eigenvector associated with eigenvalue 1. Let vi be this eigenvector. • The chain is finite, irreducible and aperiodic. Therefore it is ergodic and it has a steady-state distribution. • The functional dependency graph associated with the functional matrix in Eq. (12) is a DAG. vi is an eigenvector of Mi associated with eigenvalue 1 but we also have vi IdMi = vi . Therefore, all the assumptions of Theorem 1 are satisfied and the chain of the DTMCs in competition has product form solution.  Corollary 20. Theorem 3 remains true if the chains in competition are denumerable and ergodic. The proof is based on the last point of the proof of Theorem 3 which is not duplicated here for the sake of readability. Note that Theorem 1 supposes that the functional dependency graph is a DAG. Therefore it is not possible to consider competition with symmetrical rules. X2 cannot block chain X1 because we have a strict priority among the chains. Such a restriction does not exist in the continuous-time competition of chains studied by Boucherie. There are many differences between the discrete-time model and the continuous-time models. With DTMCs in competition, several transitions can occur at the same time and several chains may be in set A0i . In that case, only the chain with the smallest index will have the resource and the other chains are blocked. Competition according to Boucherie will always remove some states of the Cartesian product of state space but in some examples we have shown that all the states are reachable for DTMCs in competition. In [17,18] we have generalized Boucherie’s theory through algebraic arguments. The key property to explain why we need strong priority in discrete time while a race condition is sufficient in continuous time comes from Propositions 4 and 8 on the eigenvectors of generalized tensor product and generalized tensor sum of functional

794

J.M. Fourneau / Performance Evaluation 67 (2010) 779–796

matrices. The generalized tensor product (associated with discrete-time models) needs that the functional dependency graph is a DAG while the generalized tensor sum (associated with continuous-time models) does not add any constraint on the functional dependence. One can find in [24] how we can extend the continuous-time model to cope with a competition over several types of resources. 4.2. Collaboration and creation of resources We now study the case where the chains do not distinguish between the resources and may require several of them. However the semantics of the rules are now completely different. We first present the result for the collaboration two chains X1 and X2 to simplify the presentation. We still have a strict priority of X1 over X2 . The number of resources is not necessarily constrained and the chain with the highest priority (i.e. X1 ) allows X2 to perform k steps. k depends on the state of the highest priority chain. If k is zero, X2 is not allowed to move and we obtain again the competition described earlier. If k is always 1, we have independence. If k is larger than one and is state dependent, we obtain a number of steps performed by the other chains. Such models will be called collaboration of the DTMCs. SR More formally assume that the number of resources is R. We partition the state space of X1 into ( r =0 Bri ) where Bri is the set of nodes where X1 acts on r resources. When chain X1 acts upon r resources, the new number of resources is R + g1 (r ) where g1 is an arbitrary function on the integers. We just assume that g1 (r ) ≥ −R as the number of resources is not negative. In this model the functions gi represent the balance between the number of resources used by Xi and the amount of resources produced for the chains with a smaller priority. Let M2 be the stochastic matrix of chain X2 in isolation. Now X2 moves as follows due to the collaboration: if chain X1 leaves R + g1 (r ) resources available, the transitions of X2 in competition with X1 follow matrix (M2 )R+g1 (r ) . Note that this is consistent with the model of DTMCs in competition. Indeed if we have one resource (R = 1) used by X1 (i.e. g1 (1) = −1) when X1 uses 1 unit of resource, the transition matrix of X2 is (M2 )0 = IdM2 . As usual, the transitions of X1 are not dependent on the state of chain X2 . The functional stochastic matrix of X2 is: R X

1X1 ∈Br1 (M2 )R+g1 (r ) .

r =0

All the matrices (M2 )k have the same eigenvectors and the functional dependency graph is a two node graph with a directed edge from node associated with X1 to node associated with X2 . Therefore it is a DAG. Two of the assumptions of Theorem 4 are satisfied. If matrices M1 and M2 make the chain ergodic, the steady-state distribution has product form. We now give a more formal definition and state the theorem. We consider N chains and R equivalent resources. Assume SR that the state space of chain Xi is decomposed into a partition ( r =0 Bri ) where r is the number of resources used by chain Xi . Assumption 3. Assume that the chains obey the following rules for the competition:

• H1’: The resources are distributed among the Xi according to a priority based on the index. Chain Xi has a higher priority than Xj iff i < j. • H2’: At the beginning R resources are available. • H3’: The resources are created and used at every time slot. • H4’: If the initial state of Xi is in Bki and r ≥ k resources are available, the transition matrix of Xi is (Mi )k and r + gi (r , k) resources are available for Xi+1 . gi is an arbitrary function on the integers.

• H5’: If the initial state of Xi is in Bki and r < k resources are available, it is blocked. Its transition matrix is IdMi . The number of resources available for Xi+1 is r + gi (r , k). Theorem 4. Consider a collaboration between N DTMCs X1 , . . . , XN over a set of R equivalent resources. Suppose that assumptions H1 ’ to H5 ’ hold. Assume that the DTMC modeling the competition is ergodic, then the steady-state distribution has product form. Proof. As we assume that the chain is ergodic, it is sufficient to apply Theorem 1 once we have proved that the functional dependency graph is a DAG and that the eigenvectors of the functional matrices do not change when we make the functions vary. It is sufficient to write the functional matrix of chain Xi for an arbitrary i > 1. Remember that the transition probability matrix of X1 is matrix M1 which is not functional. For any i > 1, according to the assumptions, the functional matrix is: z1 X

1X1 ∈Br1

r1 =0

1

z2 X r2 =0

1X2 ∈Br2 . . . 2

zi−1 X ri−1 =0

1X

ri−1 i−1 ∈Bi−1

zi X

1X ∈Bri (Mi )ri , i

r i =0

i

where qi is defined by the following induction: z1 = R

and zi+1 = zi + gi (zi , ri ).

The functional dependency graph is a DAG because of Assumption H1’. And because of Assumptions H4’ and H5’, the functional matrices satisfy the assumptions of Theorem 1. The chain has product form steady-state distribution. 

J.M. Fourneau / Performance Evaluation 67 (2010) 779–796

795

Fig. 8. X1 , X2 and the DTMC of their collaboration.

Example 9. Let us consider the following collaboration of two DTMCs X 1 and X 2. They have both state space {0, 1, 2}. X1 has priority over X2 . The graph of the Markov chain is given in Fig. 8. When X1 is in state 0, X2 performs loops. If X1 is in state 1, X1 moves according to transition matrix M2 while it moves according to (M2 )2 when X1 is in state 2. X1 transitions have constant probabilities given by matrix M1 : 1/2 0 1

" M1 =

1/2 1/3 0

0 2/3 0

#

" M2 =

0 0 1/3

1 0 0

0 1 . 2/3

#

Clearly if v is an eigenvector of M2 it is also an eigenvector of Id and (M2 )2 . The functional dependency graph is a DAG. Therefore we can apply Theorem 4 to prove that the chain has product form steady-state distribution. To illustrate the theory, we can again prove it using very simple tensor arguments. First decompose M1 into A1 + A2 + A3 where Ak contains the row of M1 indexed by k. The transition matrix of the collaboration chains is: A1 ⊗ IdM2 + A2 ⊗ M2 + A3 ⊗ M22 . Let u be the dominant eigenvector of M1 and v be the dominant eigenvector of M2 . Using distributivity and compatibility, we get:

(u ⊗ v)(A1 ⊗ IdM2 + A2 ⊗ M2 + A3 ⊗ (M2 )2 ) = (uA1 ⊗ v) + (uA2 ⊗ v M2 ) + (uA3 ⊗ v(M2 )2 ). Taking into account that v is an eigenvector of M2 and (M2 )2 , we obtain:

(u ⊗ v)(A1 ⊗ IdM2 + A2 ⊗ M2 + A3 ⊗ (M2 )2 ) = (uA1 ⊗ v) + (uA2 ⊗ v) + (uA3 ⊗ v). After factorization:

(u ⊗ v)(A1 ⊗ IdM2 + A2 ⊗ M2 + A3 ⊗ (M2 )2 ) = (u(A1 + A2 + A3 ) ⊗ v). As A1 + A2 + A3 = M1 whose dominant eigenvector is u, we finally get u ⊗ v . Finally, note that this model can be generalized very easily. We only consider here an additive resource with arbitrary creation of resources (i.e. functions gi ). It may be possible to use more sophisticated composition of resource allocation schemes. 5. Conclusion This paper is mainly theoretical and based on some algebraic properties of the generalized tensor product which can be used to proved product form steady-state solutions. We have extended the preliminary results in [24] in many directions: we have obtained a new sufficient condition based on reversibility which does not require that the functional dependency graph is a DAG. We have also obtained a generalization of our previous result when the chain is described by a sum of tensor product.

796

J.M. Fourneau / Performance Evaluation 67 (2010) 779–796

Even if they are not considered in this paper, applications of these closed form solutions seem easy to find. Many systems evolve in discrete time because they use clocks or send fixed size packets, slots or TDMA frames. We hope that such results will allow to develop new techniques for performance modeling. It is also important to explore the links with well-known results for product forms (such as local balance) which also have an algebraic description. Acknowledgements This work is partially supported by ANR projects (ANR-05-BLAN-009-0. SMS) and (ANR-SETIN-CHECKBOUND). References [1] H. Daduna, Queueing Networks with Discrete Time Scale: Explicit Expressions for the Steady-State Behavior of Discrete Time Stochastic Networks, in: LNCS, vol. 2046, Springer Verlag, 2001. [2] W. Henderson, P.G. Taylor, Product form in networks of queues with batch arrivals and batch services, Queueing Syst. 6 (1) (1990) 71–87. [3] M.E. Woodward, Communication and Computer Networks Modelling with Discrete-Time Queues, Pentech Press, 1993. [4] R.J. Boucherie, A characterization of independence for competing Markov chains with applications to stochastic Petri nets, IEEE Trans. Softw. Eng. 20 (7) (1994) 536–544. Also available as INRIA Report 1880. [5] B. Plateau, J. Fourneau, K. Lee, PEPS: A package for solving complex Markov models of parallel systems, in: Proceedings of the 4th Int. Conf. on Modeling Techniques and Tools for Computer Performance Evaluation, Majorca, Spain, 1988. [6] P. Fernandes, B. Plateau, W.J. Stewart, Efficient descriptor-vector multiplications in Stochastic Automata Networks, J. ACM 45 (3) (1998) 381–414. [7] B. Plateau, K. Atif, Stochastic automata network for modeling parallel systems, IEEE Trans. Software Eng. 17 (10) (1991) 1093–1108. [8] J.-M. Fourneau, H. Maisonniaux, N. Pekergin, V. Vèque, Performance evaluation of buffer policies with Stochastic Automata Networks, in: H.G. Perros, G. Pujolle, Y. Takahashi (Eds.), Modelling and Evaluation of ATM Networks, in: IFIP Transactions, vol. C-15, North-Holland, 1993. [9] B. Plateau, On the stochastic structure of parallelism and synchronization models for distributed algorithms, in: Proc. of the SIGMETRICS Conference, Texas, 1985. [10] S. Donatelli, Superposed stochastic automata: A class of stochastic Petri nets with parallel solution and distributed state space, Perform. Eval. 18 (1) (1993) 21–36. [11] J. Hillston, L. Kloul, An efficient Kronecker representation for PEPA models, in: L. de Alfaro, S. Gilmore (Eds.), PAPM-PROBMIV, in: Lecture Notes in Comput. Sci., vol. 2165, Springer, 2001. [12] P.G. Harrison, Compositional reversed Markov processes, with applications to G-networks, Perform. Eval. 57 (3) (2004) 379–408. [13] G. Clark, J. Hillston, Product form solution for an insensitive stochastic process algebra structure, Perform. Eval. 50 (2/3) (2002) 129–151. [14] J. Hillston, N. Thomas, Product form solution for a class of PEPA models, Perform. Eval. 35 (3–4) (1999) 171–192. [15] A. Benoit, B. Plateau, W.J. Stewart, Réseaux d’automates stochastiques à temps discret, Tech. Sci. Inform. 24 (2–3) (2005) 229–248. [16] P.G. Harrison, Product-forms and functional rates, Perform. Eval. 66 (11) (2009) 660–663. [17] J.-M. Fourneau, B. Plateau, W. Stewart, Product form for Stochastic Automata Networks, in: P.W. Glynn (Ed.), Proceedings of the 2nd International Conference on Performance Evaluation Methodolgies and Tools, VALUETOOLS 2007, ICST, Nantes, France, 2007. [18] J.-M. Fourneau, B. Plateau, W. Stewart, An algebraic condition for product form in Stochastic Automata Networks without synchronizations, Perform. Eval. 85 (2008) 854–868. [19] F.P. Kelly, Reversibility and Stochastic Networks, J. Wiley, 1987. [20] Y. Zhao, D. Liu, The Censored Markov chain and the best augmentation, J. Appl. Probab. 33 (1996) 623–629. [21] C.D. Meyer, Stochastic complementation, uncoupling Markov chains, and the theory of nearly reducible systems, SIAM Rev. 31 (2) (1989) 240–272. [22] T.-H. Dao-Thi, J.-M. Fourneau, Stochastic automata networks with master/slave synchronization: Product form and tensor, in: K. Al-Begain, D. Fiems, G. Horváth (Eds.), ASMTA, in: Lecture Notes in Comput. Sci., vol. 5513, Springer, 2009. [23] L. Brenner, P. Fernandes, B. Plateau, I. Sbeity, PEPS2007 — Stochastic Automata Networks software tool, in: Fourth International Conference on the Quantitative Evaluaiton of Systems (QEST 2007), 17-19 September 2007, IEEE Computer Society, Edinburgh, Scotland, UK, 2007. [24] J.-M. Fourneau, Discrete time Markov chains competing over resources: product form steady-state distribution, in: Fith International Conference on the Quantitative Evaluation of Systems, QEST 2008, IEEE Computer Society, St Malo, France, 2008.

J.M. Fourneau is Professor of Computer Science at the University of Versailles St Quentin, France. He was formerly with Ecole Nationale des Telecommunications, Paris and University of Paris XI Orsay as an Assistant Professor. He graduated in Statistics and Economy from Ecole Nationale de la Statistique et de l’Administation Economique, Paris and he obtained is PHD and his habilitation in Computer Science at Paris XI Orsay in 1987 and 1991 respectively. He is the Head of the Performance Evaluation team within PRiSM laboratory at Versailles University and his recent research interests are algorithmic performance evaluation, Stochastic Automata Networks, G-networks, stochastic bounds, and application to high speed networks, and all optical networks.