A Matrix-Multiplicative Solution for Multi-Dimensional QBD Processes

Abstract

We consider an irreducible positive-recurrent discrete-time Markov process on the state space $X={\mathbb{Z}}_{+}^M\times J$, where ${\mathbb{Z}}_{+}$ is the set of non-negative integers and $J=\{1,2,\dots ,n\}$. The number of states in $J$ may be either finite or infinite. We assume that the process is a homogeneous quasi-birth-and-death process (QBD). It means that the one-step transition probability between non-boundary states $(k,i)$ and $(n,j)$ may depend on $i,j$, and $n-k$ but not on the specific values of $k$ and $n$. It is shown that the stationary probability vector of the process is expressed through square matrices of order $n$, which are the minimal non-negative solutions to nonlinear matrix equations.

1. Introduction

The processes of birth and death [1,2] represent a basic model that explains the temporal evolution of population size. In recent decades, various useful generalizations of this simple model have emerged.

The framework of interacting particle systems (IPS) introduced independently by Spitzer [3] and Dobrushin [4,5] makes it possible to describe not only the dynamics of population size but also spatio-temporal information about all individuals in the population. Members of the population are located on the set of vertices of a graph, typically the d-dimensional integer lattice. Each particle has a state selected from a finite set of possible states. The dynamics are described via local interactions indicating the rate at which a vertex changes its state based on the states of its neighbors. Lyapunov functions [6] are commonly employed to investigate IPS stability conditions. Contact processes [7], spatial birth and death processes [8], lattice birth and death processes [9], and spatial birth–death–move processes [10]—this is not a complete list of systems of interacting particles that generalize simple birth and death processes. For details, see, e.g., [11,12,13,14].

Other generalizations of the simple birth-and-death process are birth and death processes in random environments [15,16,17,18]. The birth and death process in a random environment is a two-component process, one component of which describes the dynamics of population size, and the other represents the state (phase) of the external environment. If such a process is a Markov chain, it is called a quasi-birth-and-death process (QBD).

The state space of a QBD can be partitioned into non-empty disjoint subsets called levels, such that one-step state transitions are restricted to states at the same level or two adjacent levels. The transition matrix of a QBD is block-tridiagonal. If all blocks along the main diagonal, except the boundary ones, are identical, then the QBD is said to be level-independent.

Let the stationary probability vector $p$ of an irreducible and positive recurrent level-independent QBD process be partitioned into subvectors $p=[{\pi }_0,{\pi }_1,{\pi }_2,\dots ]$, corresponding to the state space partition into levels. Then, the vector is such that.

$${\pi }_n={\pi }_{n-1}R, n\ge 1,$$

where the matrix $R$, called the rate matrix, is the minimal non-negative solution of a quadratic matrix equation [19,20] and Chapter 6 in [21]. In general, there is no explicit solution for the quadratic matrix equations. However, the iterative logarithmic reduction algorithm [22] has proven to be an efficient way to compute its minimal non-negative solution $R$ [23,24,25].

The matrix geometric method proposed in [19] led to the rapid development of new matrix–analytic methods for stochastic modeling. These methods initiated by Neuts [19,26] provide a powerful framework for the unified analysis of large classes of Markov processes and, more importantly, for their numerical solution. Matrix–analytic methods have found applications in a wide range of fields, including supply chains [27], retrial systems [28], cognitive radio [29], and more. Matrix analysis methods and their applications as it stands today are outlined in [22,30,31,32,33,34,35,36,37,38].

The multi-dimensional quasi-birth-and-death process (Md-QBD) is a Markov chain $x(t)=(\alpha (t),\beta (t))$ on the state space $X={\mathbb{Z}}_{+}^M\times J$, where ${\mathbb{Z}}_{+}$ is the set of non-negative integers and $J=\{1,2,\dots ,n\}$ is the environmental space. The first component of the process $x(t)$ is a multiclass birth-and-death process $\alpha (t)=({\alpha }_1(t),{\alpha }_2(t),\dots ,{\alpha }_M(t))$. The second component $\beta (t)$, called the phase process, takes values from the set $J=\{1,2,\dots n\}$. The number $n$ of phases may be either finite or infinite. One-step transitions of $x(t)$ from a state $(k,i)$ are restricted to states $(n,j)$ such that $k-n\in {\{-1,0,1\}}^M$.

By choosing any of the coordinates of the process $\alpha (t)$, say coordinate $k$, Md-QBD $x(t)=(({\alpha }_1(t),{\alpha }_2(t),\dots ,{\alpha }_M(t)),\beta (t))$ with the phase space $J$ can be transformed into one-dimensional level-dependent QBD (LD-QBD) processes $x_k(t)=({\alpha }_k(t),({\alpha }_1(t),\dots ,{\alpha }_{k-1}(t),$ ${\alpha }_{k+1}(t),\dots {\alpha }_M(t),\beta (t)))$ with the phase space ${\mathbb{Z}}_{+}^{M-1}\times J$ [39]. We obtain another one-dimensional LD-QBD process if each level $l$ is obtained by merging all state subsets $\{(k_1,\dots ,k_M)\}\times J$ with $\max k_i=l$ [40].

The explicit analytical representation for the stationary distribution of Md-QBDs is unknown, and most work has been devoted to deriving asymptotic formulas of the stationary distribution. Asymptotic properties of the stationary distribution of 2d-QBD processes were studied in [41,42,43,44]. The conditions ensuring a positive-recurrent or transient 2d-QBD process were analyzed in [45].

For general Md-QBD, several process components can change their values simultaneously. In simple Md-QBD, no more than one component of $\alpha (t)$ may be changed at a time. One-step transitions of simple Md-QBD from any state $((k_1,k_2,\dots ,k_M),i)$ are restricted to states $((n_1,n_2,\dots ,n_M),j)$ such as $|k_1-n_1|+|k_2-n_2|+\dots +|k_M-n_M|\le 1$. Simple Md-QBD may be analyzed using an iterative power series algorithm, as was introduced in [46]. The review article [47] and Chapter 4 in [48] describe the application of the PSA for the analysis of simple Md-QBD.

The state space of a Md-QBD can be partitioned into the levels $X_l=Z_l\times J$, $l\ge 0$, where $Z_0=\{n\in Z|n\cancel{\ge }1\},$ and $Z_l=\{n\in Z|n\ge 1, \min (n_i)=l\}$, $l\ge 1$. Due to the process’s homogeneity, it can be considered as the one-dimensional level-independent QBD with an infinite rate matrix $R$ in which the elements of the set $X_1$ index entries.

This work aims to show that the stationary distribution of a simple Md-QBD has a matrix-multiplicative solution expressed in terms of square matrices of order $n$ and propose iterative algorithms for computing these matrices. This may serve as a basis for creating efficient algorithms for computing stationary distributions of the Md-QBDs in the future. In Section 2, we study some systems of nonlinear matrix equations for substochastic matrices and propose several iterative algorithms for computing their minimal non-negative solutions. In Section 3, for simple irreducible and positive-recurrent discrete-time Md-QBD, we derive the matrix of the expected sojourn times in the states of the sets $X(w)=\{(k,i)\in X|k\ge w\}$, $w\ge 1$, before the first visit to any state outside $X(w)$. The matrix-multiplicative solution for the stationary distribution of the process in terms of square matrices of order $n$ is obtained in Section 4. We finally give some concluding remarks in Section 5.

We use bold capital letters to denote matrices and bold lowercase letters to denote vectors. For vectors $x=(x_1,x_2,\dots ,x_M)$ and $y=(y_1,y_2,\dots ,y_M)$, $x\ge y$ means that $x_j\ge y_j$ for all $j$, and $x\cancel{\ge }y$ means that $x_j<y_j$ for at least one value of $j$. Notations $x\le y$ and $x\cancel{\le }y$ are defined similarly. Vector 1 represents the vector of all 1 s, and the vector $e_m$ indicates the vector with zero entries except the mth entry, which equals one.

2. Preliminaries

In what follows, $Q_m$, $-M\le m\le M$, are non-negative square matrices of finite or infinite order such that $Q={\displaystyle \sum _{r=-M}^M{Q}_r}$ is a stochastic matrix, and monotonicity and convergence of a sequence of matrices mean their entry-wise monotonicity and convergence.

Theorem 1.

The sequence of $M$-tuples {$(G_m(k), \hspace{0.17em}1\le m\le M)$, $k\ge 0$}, recursively defined by

$$G_m(0)=O for 1\le m\le M, and$$

$$G_m(k+1)=Q_{-m}+(Q_0+{\displaystyle \sum _{s=1}^M{Q}_s{G}_s(k)})G_m(k) for 1\le m\le M and k\ge 0,$$

(1)

is monotonically increasing and converges to the $M$-tuple $(G_m,\hspace{0.17em} 1\le m\le M)$, which is the minimal solution of the system

$$Y_m=Q_{-m}+(Q_0+{\displaystyle \sum _{s=1}^M{Q}_s{Y}_s})Y_m for 1\le m\le M,$$

(2)

in the set of $M$-tuples $(Y_m,1\le m\le M)$ of non-negative matrices. Matrix $G={\displaystyle \sum _{r=1}^M{G}_r}$ is substochastic.

Proof of Theorem 1.

We first show that the sequence {$(G_m(k),\hspace{0.17em}1\le m\le M)$, $k\ge 0$} is monotonically increasing and satisfies ${\displaystyle \sum _{m=1}^M{G}_m(k)1}\le 1$ for all $k\ge 0$. We proceed using induction.

Since $G_m(0)=O$, we know that $G_m(1)=Q_{-m}\ge G_m(0)$ and ${\displaystyle \sum _{m=1}^M{G}_m(1)1}\le 1$. Let us assume that $G_m(k)\ge G_m(k-1)$ and ${\displaystyle \sum _{m=1}^M{G}_m(k)1}\le 1$ for some $k$ and all $1\le m\le M$. Then,

$$\begin{array}{ll}{G}_m(k+1)& =Q_{-m}+(Q_0+{\displaystyle \sum _{s=1}^M{Q}_s{G}_s(k)})G_m(k)\\ & \ge Q_{-m}+(Q_0+{\displaystyle \sum _{s=1}^M{Q}_s{G}_s(k-1)})G_m(k-1)=G_m(k),\end{array}$$

and

$$\begin{array}{ll}{\displaystyle \sum _{m=1}^M{G}_m(k+1)1}& ={\displaystyle \sum _{m=1}^M{Q}_{-m}1}+(Q_0+{\displaystyle \sum _{s=1}^M{Q}_s{G}_s(k)}){\displaystyle \sum _{m=1}^M{G}_m(k)1}\\ & \le {\displaystyle \sum _{m=1}^M{Q}_{-l}1}+(Q_0+{\displaystyle \sum _{s=1}^M{Q}_s{G}_s(k)})1\le {\displaystyle \sum _{m=-M}^M{Q}_{m}1}\le 1,\end{array}$$

which proves the induction step. Thus, for every $1\le m\le M$, the sequence $G_m(k)$, $k=0,1,2,\dots$ is bounded and monotonically increasing. This implies the existence of the limits $G_m=\underset{k\to \infty }{\lim }{G}_m(k)$, $1\le m\le M$, which satisfy the system (2) and the inequality $G1={\displaystyle \sum _{m=1}^M{G}_{m}1\le 1}.$

Assume that ($Y_m^{*}$, $1\le m\le M$) is another non-negative solution of (2). We show via induction that $G_m(k)\le Y_m^{*}$ for all $1\le m\le M$ and all $k$. Since $Y_m^{*}\ge O$, we know that $G_m(0)=O\le Y_m^{*}$ for all $1\le m\le M$. Now, let us assume that $G_m(k)\le Y_m^{*}$ for some $k$ and all $1\le m\le M$. Then,

$$G_m(k+1)=Q_{-m}+(Q_0+{\displaystyle \sum _{s=1}^M{Q}_s{G}_s(k)})G_m(k)\le Q_{-m}+(Q_0+{\displaystyle \sum _{s=1}^M{Q}_s{Y}_s^{*}})Y_m^{*}=Y_m^{*}$$

which proves the induction step and the minimal property of the $M-$tuple $(G_m,\hspace{0.17em}1\le m\le M)$. □

Substochastic matrices $G_m$ satisfy the system

$$G_m=Q_{-m}+(Q_0+{\displaystyle \sum _{s=1}^M{Q}_s{G}_s})G_m, 1\le m\le M.$$

(3)

Therefore, we have

$$(I-U)G_m=Q_{-m},1\le m\le M,$$

(4)

where the matrix $U$ is defined using

$$U=Q_0+{\displaystyle \sum _{s=1}^M{Q}_s{G}_s}.$$

(5)

The following theorem describes some properties of the matrix $U$.

Theorem 2.

Let the $M$-tuple $(G_m, \hspace{0.17em}1\le m\le M)$ be the minimal non-negative solution of the system (2). Then, the matrix $U$ defined in (5) is substochastic. It is the minimal solution of the equation

$$X=Q_0+{\displaystyle \sum _{n=0}^{\infty }{\displaystyle \sum _{m=1}^M{Q}_m{X}^n Q_{-m}}}$$

(6)

in the set of non-negative matrices $X$ such that the series converges.

The sequences $\{U^{\prime }(k),k\ge 1\}$ and $\{U^{″}(k),k\ge 1\}$ defined using

$$U^{\prime }(1)=Q_0, U^{\prime }(k+1)=Q_0+{\displaystyle \sum _{n=0}^k{\displaystyle \sum _{m=1}^M{Q}_m{U}^{\prime }{(k)}^n Q_{-m}}} for k\ge 1,$$

(7)

$$U^{″}(1)=Q_0, U^{″}(k+1)=Q_0+{\displaystyle \sum _{n=0}^{\infty }{\displaystyle \sum _{m=1}^M{Q}_m{U}^{″}{(k)}^n Q_{-m}}} for k\ge 1,$$

(8)

are monotonically increasing and converging to $U$.

Proof of Theorem 2.

The matrix is substochastic since the matrix $Q={\displaystyle \sum _{m=-M}^M{Q}_m}$ is stochastic and $G_{m}1\le 1$ for all $1\le m\le M$.

From (3) and (5) it follows that

$$\begin{array}{cc}U\hfill & =Q_0+{\displaystyle \sum _{m=1}^M{Q}_m{G}_m}=Q_0+{\displaystyle \sum _{m=1}^M{Q}_m(Q_{-m}+U{G}_m})=Q_0+{\displaystyle \sum _{m=1}^M{Q}_m{Q}_{-m}}+{\displaystyle \sum _{m=1}^M{Q}_{m}U{G}_m}=\dots \hfill \\ & =Q_0+{\displaystyle \sum _{m=1}^M{Q}_m{Q}_{-m}}+{\displaystyle \sum _{m=1}^M{Q}_{m}U{G}_m}+{\displaystyle \sum _{m=1}^M{Q}_m{U}^2{Q}_{-m}}+{\displaystyle \sum _{m=1}^M{Q}_m{U}^3{Q}_{-m}}+\dots \hfill \\ & =Q_0+{\displaystyle \sum _{n=0}^{\infty }{\displaystyle \sum _{m=1}^M{Q}_m{U}^n Q_{-m}}}.\hfill \end{array}$$

Therefore, the matrix $U$ satisfies (6).

Let us show via induction that the sequence $\{U^{″}(k),k\ge 1\}$ is monotonically increasing and satisfies $U^{″}(k)\le U$ for all $k\ge 1$. Since $U^{″}(1)=Q_0\le U$ and

$$U^{″}(2)=Q_0+{\displaystyle \sum _{n=0}^{\infty }{\displaystyle \sum _{m=1}^M{Q}_m{(Q_0)}^n Q_{-m}\le }} Q_0+{\displaystyle \sum _{n=0}^{\infty }{\displaystyle \sum _{m=1}^M{Q}_m{U}^n Q_{-m}}}=U,$$

we know that $U^{″}(2)\ge U^{″}(1)$ and $U^{″}(2)\le U$. Let us assume that $U^{″}(k)\ge U^{″}(k-1)$ and $U^{″}(k)\le U$ for some $k$. Then,

$$U^{″}(k+1)=Q_0+{\displaystyle \sum _{n=0}^{\infty }{\displaystyle \sum _{m=1}^M{Q}_m{U}^{″}{(k)}^n Q_{-m}}}\ge Q_0+{\displaystyle \sum _{n=0}^{\infty }{\displaystyle \sum _{m=1}^M{Q}_m{U}^{″}{(k-1)}^n Q_{-m}}}=U^{″}(k)$$

and

$$U^{″}(k+1)=Q_0+{\displaystyle \sum _{n=0}^{\infty }{\displaystyle \sum _{m=1}^M{Q}_m{U}^{″}{(k)}^n Q_{-m}}}\le Q_0+{\displaystyle \sum _{n=0}^{\infty }{\displaystyle \sum _{m=1}^M{Q}_m{U}^n Q_{-m}}}=U.$$

Thus, the sequence $U^{″}(k)$, $k=1,2,\dots$, is bounded and monotonically increasing. This implies the existence of the limit $U^{″}=\underset{k\to \infty }{\lim }{U}^{″}(k)$, which satisfies Equation (6) and the inequality $U^{″}\le U$. Similarly, it can be proven that the sequence $U^{\prime }(k)$, $k=1,2,\dots$, is monotonically increasing and bounded by $U$. The limit $U^{\prime }=\underset{k\to \infty }{\lim }{U}^{\prime }(k)\le U$ exists and satisfies Equation (6).

Consider the sequence $\{U(k),k\ge 1\}$ defined using

$$U(k)=Q_0+{\displaystyle \sum _{m=1}^M{Q}_m{G}_m(k-1)}$$

(9)

with $G_m(k)$ defined in Theorem 1. Since

$$G_m(k)=Q_{-m}+U(k)G_m(k-1),$$

(10)

we have

$$\begin{array}{cc}U(k+1)\hfill & =Q_0+{\displaystyle \sum _{m=1}^M{Q}_m{G}_m(k)}=Q_0+{\displaystyle \sum _{m=1}^M{Q}_m{Q}_{-m}+{\displaystyle \sum _{m=1}^M{Q}_{m}U(k)G_m(k-1)}})\hfill \\ & =Q_0+{\displaystyle \sum _{m=1}^M{Q}_m{Q}_{-m}+{\displaystyle \sum _{m=1}^M{Q}_{m}U(k)}}{Q}_{-m}+{\displaystyle \sum _{m=1}^M{Q}_{m}U(k)U(k-1)G_m(k-2)}=\dots \hfill \\ & =Q_0+{\displaystyle \sum _{n=0}^k{\displaystyle \sum _{m=1}^M{Q}_{m}U(n)U(n-1)\cdots U(1)Q_{-m}}}\hfill \end{array}.$$

(11)

It follows that

$$U(k+1)=U(k)+{\displaystyle \sum _{m=1}^M{Q}_{m}U(k)U(k-1)\cdots U(1)Q_{-m}},$$

which implies that the sequence $\{U(k),k\ge 1\}$ is increasing. From this and (9), we obtain the following inequality:

$$U(k+1)\le Q_0+{\displaystyle \sum _{n=0}^k{\displaystyle \sum _{m=1}^M{Q}_{m}U{(k)}^n{Q}_{-m}}}.$$

(12)

We use this to show that $U(k)\le U^{\prime }(k)\le U^{″}(k)$ for all $k\ge 1$. We then proceed using induction. We have $U(1)=U^{\prime }(1)=U^{″}(1)=Q_0$; therefore, $U(1)\le U^{\prime }(1)\le U^{″}(1)$. Now suppose that $U(k)\le U^{\prime }(k)\le U^{″}(k)$ for some $k$ and we can show that $U(k+1)\le U^{\prime }(k+1)\le U^{″}(k+1)$. It follows from (12) that

$$\begin{array}{cc}U(k+1)\hfill & \le Q_0+{\displaystyle \sum _{n=0}^k{\displaystyle \sum _{m=1}^M{Q}_{m}U{(k)}^n{Q}_{-m}}}\le Q_0+{\displaystyle \sum _{n=0}^k{\displaystyle \sum _{m=1}^M{Q}_m{U}^{\prime }{(k)}^n{Q}_{-m}}}=U^{\prime }(k+1)\\ & \le Q_0+{\displaystyle \sum _{n=0}^k{\displaystyle \sum _{m=1}^M{Q}_m{U}^{″}{(k)}^n{Q}_{-m}}}\le Q_0+{\displaystyle \sum _{n=0}^{\infty }{\displaystyle \sum _{m=1}^M{Q}_m{U}^{″}{(k)}^n{Q}_{-m}}}=U^{″}(k+1).\hfill \end{array}$$

Therefore, $U(k+1)\le U^{\prime }(k+1)\le U^{″}(k+1),$ which proves the induction step. This implies that $U(k)\le U^{\prime }(k)\le U^{″}(k)$ for all $k\ge 1$ and that $U\le U^{\prime }\le U^{″}$. Since we have already shown that $U^{″}\le U$, it means that the sequences $\{U(k),k\ge 1\}$, $\{U^{\prime }(k),k\ge 1\}$, and $\{U^{″}(k),k\ge 1\}$ converge to the same solution $U=U^{\prime }=U^{″}$ of Equation (6).

Assume that $X^{*}$ is another non-negative solution of (6). We show that $U^{″}(k)\le X^{*}$ for all $k$, such that $U=\underset{k}{\lim }{U}^{″}(k)\le X^{*}$. We prove this using induction. Since

$$X^{*}=Q_0+{\displaystyle \sum _{n=0}^{\infty }{\displaystyle \sum _{m=1}^M{Q}_m{(X^{*})}^n Q_{-m}}}\ge Q_0,$$

we have that $U^{″}(1)=Q_0\le X^{*}$. Now, suppose that $U^{″}(k)\le X^{*}$ for some $k$. Then,

$$U^{″}(k+1)=Q_0+{\displaystyle \sum _{n=0}^{\infty }{\displaystyle \sum _{m=1}^M{Q}_m{U}^{″}{(k)}^n Q_{-m}}}\le Q_0+{\displaystyle \sum _{n=0}^{\infty }{\displaystyle \sum _{m=1}^M{Q}_m{(X^{*})}^n Q_{-m}}}=X^{*},$$

which proves the induction step. Thus, $U=\underset{k}{\lim }U(k)\le X^{*}$, and $U$ is the minimal solution of (6). □

For any substochastic matrix $M$, we say that the series ${\displaystyle \sum _{k\ge 0}{M}^k}$ converges if it has finite entries. In this case, we say that the matrix $I-M$ is invertible and write ${\displaystyle \sum _{k\ge 0}{M}^k}={(I-M)}^{-1}$, even if the order of $M$ is infinite. If the series ${\displaystyle \sum _{k=0}^{\infty }{U}^k}$ converges, then the matrix $U$, defined using (5), satisfies

$$U=Q_0+{\displaystyle \sum _{m=1}^M{Q}_m{(I-U)}^{-1}{Q}_{-m}}.$$

(13)

We can use Equations (9) and (10) as the basis of the following algorithm for the calculation of matrices $G_m, \hspace{0.17em}1\le m\le M,$ and $U$:

$$G_m(0)=O,1\le m\le M;$$

(14)

$$U(k)=Q_0+{\displaystyle \sum _{m=1}^M{Q}_m{G}_m(k-1)},$$

(15)

$$G_m(k)=Q_{-m}+U(k)G_m(k-1),1\le m\le M, k=1,2,\dots$$

(16)

The convergence of this iterative scheme was proven during the proof of Theorem 2. Although the convergence of the sequence $U(k)$ is slower than that of the sequences in (7) and (8), the number of arithmetic operations per iteration is significantly less than in (7) and (8). The algorithm (14)–(16) involves $2M$ matrix multiplications and $2M$ matrix additions per iteration. For finite $n$, this requires $2M{n}^3$ multiplications of scalars and $2M(n^3-n^2)+2M{n}^2$ additions. Therefore, the worst-case computational complexity per iteration is $O(2M{n}^3)$.

3. Multi-Dimensional QBD Processes

Consider a discrete-time Md-QBD process $x(t)=(\alpha (t),\beta (t))$ on the state space $X={\mathbb{Z}}_{+}^M\times J$, and denote $P_{k,n}(i,j)$ the probability of a one-step transition from $(k,i)$ to $(n,j)$. We assume that the transition probability matrix $P=[P_{k,n}(i,j)]$ $(k,n\in {\mathbb{Z}}_{+}^M$, $i,j\in J)$ partitioned into blocks $P_{k,n}$ of order $n$, has the following form:

$$P_{k,n}=O \mathrm{if} {\displaystyle \sum _{m=1}^M|k_m-n_m|}>1, \mathrm{for} k,n\ge 0;$$

$$P_{n,n}=Q_0, P_{n-e_m,n}=Q_m, P_{n+e_m,n}=Q_{-m}, \mathrm{for} 1\le m\le M \mathrm{and} n\ge 1,$$

(17)

where $Q_m$, $-M\le m\le M$, are non-negative square matrices such that $Q={\displaystyle \sum _{r=-M}^M{Q}_r}$ is a stochastic matrix.

We denote using $P(w)$ the block matrix with blocks $P_{k,n}$, $k,n\ge w$. The following theorem demonstrates how the minimal non-negative solution of Equation (2) can be used for the decomposition of $P(w)$ as a product of the two block matrices.

Theorem 3.

Let the $M$-tuple $(G_m,1\le m\le M),$ be the minimal non-negative solution of Equation (2), let the matrix $U$ be defined using (5), and let a vector $w\in {\mathbb{Z}}_{+}^M$ satisfy $w\ge 1$. Then, the matrix $I-P(w)$ can be decomposed as a product

$$I-P(w)=\Theta (w)(I-\Psi (w)),$$

(18)

where matrices $\Theta (w)={[{\Theta }_{k,n}(w)]}_{k,n\ge w}$ and $\Psi (w)={[{\Psi }_{k,n}(w)]}_{k,n\ge w}$, partitioned into blocks of the order $n$, are defined as follows,

$${\Theta }_{k;k}(w)=I-U,{\Theta }_{k;k+e_m}(w)=-Q_m, 1\le m\le M;$$

(19)

$${\Theta }_{k,n}(w)=O if n\ne k and n\ne k+e_m for all 1\le m\le M;$$

(20)

$${\Psi }_{k;k-e_m}(w)=G_m, 1\le m\le M; {\Psi }_{k,n}(w)=O if n\ne k-e_m for all 1\le m\le M.$$

(21)

Proof of Theorem 3.

For the matrix$\Theta (w)\Psi (w)$ block located in a block row $k$ and a block column $n$, we have

$$\begin{array}{l}{\displaystyle \sum _{r\ge w}{\Theta }_{k,r}(w){\Psi }_{r,n}(w)}={\Theta }_{k,k}(w){\Psi }_{k,n}(w)+{\displaystyle \sum _{m=1}^M{\Theta }_{k,k+e_m}(w)}{\Psi }_{k+e_m,n}(w)\\ =(I-U){\Psi }_{k,n}(w)-{\displaystyle \sum _{m=1}^M{Q}_m{\Psi }_{k+e_m,n}}(w)=\left\{\begin{array}{l}-{\displaystyle \sum _{m=1}^M{Q}_m{G}_m} \mathrm{if} n=k,\hfill \\ \begin{array}{l}(I-U)G_r \mathrm{if} n=k-e_r \mathrm{for} \mathrm{some} 1\le r\le M,\hfill \\ O \mathrm{otherwise}.\hfill \end{array}\hfill \end{array}\right.\end{array}$$

From this and (19) it follows that

$${\Theta }_{k,n}(w)-{\displaystyle \sum _{r\ge w}{\Theta }_{k,r}(w){\Psi }_{r,n}(w)}=\left\{\begin{array}{c}(I-U)+{\displaystyle \sum _{m=1}^M{Q}_m{G}_m} \mathrm{if} n=k,\hfill \\ \begin{array}{l}-(I-U)G_r \mathrm{if} n=k-e_r \mathrm{for} \mathrm{some} 1\le r\le M,\hfill \\ -Q_r \mathrm{if} n=k+e_r \mathrm{for} \mathrm{some} 1\le r\le M,\hfill \\ O \mathrm{otherwise}.\hfill \end{array}\hfill \end{array}\right.$$

Finally, using (4) and (5), we obtain

$${\Theta }_{k,n}(w)-{\displaystyle \sum _{r\ge w}{\Theta }_{k,r}(w){\Psi }_{r,n}}(w)=\left\{\begin{array}{c}I-Q_0 \mathrm{if} n=k,\hfill \\ \begin{array}{l}-Q_{-r} \mathrm{if} n=k-e_r \mathrm{for} \mathrm{some} 1\le r\le M,\hfill \\ -Q_r \mathrm{if} n=k+e_r \mathrm{for} \mathrm{some} 1\le r\le M,\hfill \\ O \mathrm{otherwise},\hfill \end{array}\hfill \end{array}\right.$$

which proves Equality (22). □

The matrix $\Theta$ is the block upper triangular and $\Psi$ is the block low triangular in the following sense: ${\Theta }_{k,n}=O$ if $k\cancel{\le }n$; ${\Psi }_{k,n}=O$ if $n\cancel{\le }k.$ Thus, the decomposition in (18) may be considered as the block UL decomposition of the matrix $I-P(w)$.

Consider an irreducible and positively recurrent Markov chain with a state space $S$. Let $D$ be a proper subset of $S$, let $P_D$ be the submatrix of transition probabilities between the states of $D$, and let $N_D$ denote the matrix of the expected sojourn times in the states of $D$ before the first visit to any state outside $D$. Lemma 5.1.2 from [22] describes the relation between the matrices $P_D$ and $N_D$.

Lemma 1.

The matrix $N_D$ of the expected sojourn times in the subset $D$, before the first passage to the complementary subset $S\backslash D$, is given using $N_D={(I-P_D)}^{-1}$, where ${(I-P_D)}^{-1}={\displaystyle \sum _{k\ge 0}{P}_D^k}$ is the minimal non-negative solution of the systems $(I-P_D)N_D=I$ and $N_D(I-P_D)=I$.

We use this lemma to prove that, for the irreducible and positive-recurrent process $x(t)$, the matrix $\Theta (w)$ in the decomposition in (18) of the matrix $I-P(w)$ is invertible.

Theorem 4.

In addition to the conditions of Theorem 3, let the Md-QBD $x(t)$ be irreducible and positive-recurrent. Then, the matrix $I-U$ is invertible, and the matrix $I-P(w)$ can be decomposed as a product

$$I-P(w)=(I-\Phi (w))D(w)(I-\Psi (w)),$$

(22)

where matrices $D(w)={[D_{k,n}(w)]}_{k,n\ge w}$, $\Phi (w)={[{\Phi }_{k,n}(w)]}_{k,n\ge w}$ , and  $\Psi (w)={[{\Psi }_{k,n}(w)]}_{k,n\ge w}$, partitioned into blocks of the order $n$, are defined as follows,

$$D_{k;k}(w)=I-U, D_{k,n}(w)=O if n\ne k;$$

(23)

$${\Phi }_{k;k+e_m}(w)=R_m, 1\le m\le M;$$

(24)

$${\Phi }_{k,n}(w)=O if n\ne k+e_m for all 1\le m\le M;$$

(25)

$${\Psi }_{k;k-e_m}(w)=G_m, 1\le m\le M; {\Psi }_{k,n}(w)=O if n\ne k-e_m for all 1\le m\le M;$$

(26)

with the matrix $R_m$ given using

$$R_m=Q_m{(I-U)}^{-1}, 1\le m\le M.$$

(27)

Proof of Theorem 4.

Firstly note that $P(w)=P_D$, where the subset $D$ is given using $D=\{(k,j)\in X|k\ge w\}$, and from Lemma 1, it follows that the matrix $I-P(w)$ is invertible.

Consider the $r$th power $\Psi {(w)}^r={[{\Psi }_{k,n}^{(r)}(w)]}_{k,n\ge w}$ of the matrix $\Psi (w)$, partitioned into blocks of the order $n$. Diagonal blocks ${\Psi }_{k,k}^{(r)}(w)$ of the matrix $\Psi {(w)}^r$ are zero for all $r\ge 1$. If $k\ge n\ge w$ and ${\displaystyle \sum _{m=1}^M(k_i-w_i)}=l$, a non-diagonal block ${\Psi }_{k,n}^{(r)}(w)$ is only non-zero in the case where $r<l$ and ${\displaystyle \sum _{m=1}^M(n_i-w_i)}=l-r$. It follows that the series ${\displaystyle \sum _{r=0}^{\infty }{\Psi }_{k,n}^{(r)}}(w)$ is equal to ${\displaystyle \sum _{r=0}^l{\Psi }_{k,n}^{(r)}}(w)$. Therefore, the matrix $I-\Psi (w)$ in (22) is invertible.

Post-multiplying both sides of (22) using ${(I-\Psi (w))}^{-1}$ and pre-multiplying using ${(I-P(w))}^{-1}$, we find that

$${(I-\Psi (w))}^{-1}={(I-P(w))}^{-1}\Theta (w).$$

(28)

Let us partition the matrix ${(I-P(w))}^{-1}$ into blocks $N_{k,n}(w)$ ($k,n\ge w$) of order $n,$ and partition the matrix ${(I-\Psi (w))}^{-1}$ into blocks $W_{k,n}(w)$ ($k,n\ge w$) of order $n$. Then, Equation (28) may be written in block form as

$$W_{k,n}(w)={\displaystyle \sum _{r\ge w}{N}_{k,r}(w){\Theta }_{r,n}}(w).$$

From this, where $k=n=w$, we obtain

$$I={\displaystyle \sum _{r\ge w}{N}_{w,r}(w){\Theta }_{r,w}}(w)=N_{w,w}(w)(I-U),$$

which means that $N_{w,w}(w)={(I-U)}^{-1}$ and proves the invertibility of the matrix $I-U$. Now, the matrix $\Theta (w)$, defined in Theorem 3, can be decomposed as

$$\Theta (w)=(I-\Phi (w))D(w),$$

which implies (22), completing the proof. □

Note that the matrix $N_{w,w}(w)={(I-U)}^{-1}$ denotes the expected number of returns to $\left\{w\right\}\times J$ before the first passage to the subset $X_0(w)=X\backslash X(w)$, and the matrix $U$ records the return probabilities to $\left\{w\right\}\times J$ under the taboo of $X_0(w)$ (see Theorem 5.2.3 in [3]).

We define the function $\gamma (k,n)$ as $\gamma (k,n)={\displaystyle \sum _{m=1}^M(n_m-k_m)}$ if $k\le n$, and $\gamma (k,n)=0$ otherwise. We also denote using $\Theta (k,n)$ the set of sequences $\theta =({\theta }_1,{\theta }_2,\dots ,{\theta }_{\gamma })$ of the length $\gamma =\gamma (k,n)$ such that $k+e_{{\theta }_1}+e_{{\theta }_2}+\dots +e_{{\theta }_{\gamma }}=n$. Theorem 4 has the following consequence.

Corollary 1.

Let the matrix ${(I-P(w))}^{-1}$ be partitioned into blocks $N_{k,n}(w)$, $k,n\ge w,$ related to the division of the set $X(w)$ into subsets $\left\{k\right\}\times J$, $k\ge w$. Then, we have

$$N_{k,n}(w)={\displaystyle \sum _{\begin{array}{c}r:\\ w\le r\le k\\ w\le r\le n\end{array}}{\displaystyle \sum _{\sigma \in \Theta (r,k)}{\displaystyle \sum _{\theta \in \Theta (r,n)}{G}_{{\sigma }_1}{G}_{{\sigma }_2}\cdots G_{{\sigma }_{\gamma (r,k)}}}N{R}_{{\theta }_1}{R}_{{\theta }_2}\cdots R_{{\theta }_{\gamma (r,n)}}}},$$

(29)

where $N={(I-U)}^{-1}$, and it is assumed that the matrix products are equal to the identity matrix if $\gamma (r,k)=0$ or $\gamma (r,n)=0$.

Proof of Corollary 1.

It follows from Theorem 3 that the matrix ${(I-P(w))}^{-1}$ can be decomposed as

$${(I-P(w))}^{-1}=V(w)D{(w)}^{-1}W(w)$$

(30)

with the matrices $V(w)$ and $W(w)$ defined using

$$V(w)={(I-\Psi (w))}^{-1}={\displaystyle \sum _{k=0}^{\infty }\Psi {(w)}^k}, \mathrm{and} W(w)={(I-\Phi (w))}^{-1}={\displaystyle \sum _{k=0}^{\infty }\Phi {(w)}^k},$$

(31)

respectively. We partition these matrices as $V(w)={[V_{k,n}]}_{k,n\ge w}$ and $W(w)={[W_{k,n}]}_{k,n\ge w}$ following the division of the set $X(w)$ into subsets $\left\{k\right\}\times J$, $k\ge w.$ It follows from (30) that

$$V_{k,r}={\displaystyle \sum _{\sigma \in \Theta (r,k)}{G}_{{\sigma }_1}{G}_{{\sigma }_2}\dots G_{{\sigma }_{\gamma (r,k)}}}, \mathrm{for} r\le k,\mathrm{and}$$

(32)

$$W_{r,n}={\displaystyle \sum _{\theta \in \Theta (r,n)}{R}_{{\theta }_1}{R}_{{\theta }_2}\dots R_{{\theta }_{\gamma (r,n)}}}, \mathrm{for} r\le n.$$

(33)

Combining (31)–(33), we find that the matrices $N_{k,n}(w)$, $k,n\ge l1$, have the form of (30), which completes the proof. □

The submatrices $N_{k,n}(w)$, $k,n\ge w$, of the matrix ${(I-P(w))}^{-1}$ satisfy the following property:

$$N_{k,n}(w)=N_{k-w+1,n-w+1},$$

where $N_{k,n}=N_{k,n}(1)$, $k,n\ge 1$. Therefore, the matrices $N_{k,n}$, $k,n\ge 1$, uniquely define the matrices $N_{k,n}(w),$ $k,n\ge w,$ for all $w\ge 1$. In particular, we have that

$$N_{k,n}(l1)=N_{k-(l-1)1,n-(l-1)1} \mathrm{for} k,n\ge l1.$$

(34)

The matrices $R_m$, $1\le m\le M$, defined using (27), satisfy $R_m=Q_m+R_{m}U$, $1\le m\le M$, and the matrix $U$ may be written as

$$U=Q_0+{\displaystyle \sum _{m=1}^M{R}_m{Q}_{-m}}.$$

(35)

Therefore, the matrices $R_m$, $1\le m\le M$, satisfy the following system:

$$R_m=Q_m+R_m(Q_0+{\displaystyle \sum _{s=1}^M{R}_s}{Q}_{-s}), \mathrm{for} 1\le m\le M.$$

(36)

Note, if any of the matrix $U$, the $M$-tuple ($G_m$, $1\le m\le M$), or the $M$-tuple ($R_m$, $1\le m\le M)$ is known, then we may determine the other two by applying some of the following equations:

$$R_m=Q_m{(I-U)}^{-1}, G_m={(I-U)}^{-1}{Q}_{-m}, Q_m{G}_m=R_m{Q}_{-m},$$

(37)

$$U=Q_0+{\displaystyle \sum _{m=1}^M{R}_m{Q}_{-m}}, U=Q_0+{\displaystyle \sum _{m=1}^M{Q}_m{G}_m}.$$

(38)

4. Stationary Distribution

Let a vector $w\in {\mathbb{Z}}_{+}^M$ satisfy $w\ge 1$. We partition the stationary probability vector $p$ of $x(t)$ into subvectors ${\pi }_i(w)$, $(i\ge 0),$ related to the partition of the state set $X$ into the subsets $X_l(w)$, $l\ge 0$, and partition each vector ${\pi }_i(w)$ into subvectors $p_n$ $(n\in Z_i(w))$. The following theorem gives an expression for the vectors $p_n$, $n\in Z_1(w)$, in terms of the vectors $p_k$, $k\in Z_0(w)$.

Theorem 5.

Let Md-QBD $x(t)$ be irreducible and positive-recurrent, let the matrix $U$ be the minimal non-negative solution of (6), and let the matrices $R_m$, $1\le m\le M$, be defined using (27). If vectors $w,n\in {\mathbb{Z}}_{+}^M$ satisfy $w\ge 1$ and $n\in Z_1(w)$, then we have

$$p_n={\displaystyle \sum _{k\in Z_1(w)}{\displaystyle \sum _{\begin{array}{c}m=1\\ k_m=w_m\end{array}}^M{p}_{k-e_m}{Q}_m{N}_{k,n}(w)}}.$$

(39)

Proof of Theorem 5.

Let us partition the matrix ${(I-P(w))}^{-1}$ into blocks $H_{i,j}(w)$ $(i,j\ge 1),$ related to partitioning the set $X(w)$ into the subsets $X_l(w)$, $l\ge 0$. It is known (see Theorem 6.2.1 in [3]) that the stationary probability vector $p=[{\pi }_0(w),{\pi }_1(w),{\pi }_2(w),\dots ]$ satisfies the equation

$${\pi }_1(w)={\pi }_0(w)A(w)H_{1,1}(w),$$

(40)

where $A(w)$ is the matrix of the transition probabilities from states into states in $Z_1(w)\times J$.

Let us partition the matrices $A(w)={[P_{k,n}]}_{k\in Z_0(w),n\in Z_1(w)}$ and $H_{1,1}(w)={[N_{k,n}(w)]}_{k,n\in Z_1(w)}$ into blocks of the order $n$ and partition the vectors ${\pi }_0(w)={(p_k)}_{k\in Z_0(w)}$ and ${\pi }_1(w)={(p_k)}_{k\in Z_1(w)}$ into subvectors of length $n$.

For blocks $P_{k,n}$, $r\in Z_0(w)$, and $k\in Z_1(w)$ of the matrix $A(w)$ we have $P_{r,k}=Q_m$ if $r=k-e_m$ for some $1\le m\le M$; otherwise, we have $P_{r,k}=O$. Therefore, Equation (40) may be transformed as (39). Thus, the result is proven. □

Theorem 6.

Let Md-QBD $x(t)$ be irreducible and positive-recurrent, let the matrix $U$ be the minimal non-negative solution of (6), and let the matrices $R_m$, $1\le m\le M$, be defined using (27). If a vector $w\in {\mathbb{Z}}_{+}^M$ satisfies $w\ge 1$, then the vector of the stationary probabilities ${\pi }_0(w)={[p_n]}_{n\in Z_0(w)}$ is proportional to the unique solution $q(w)={[q_n]}_{n\in Z_0(w)}$ of the following linear system

$$q_n={\displaystyle \sum _{k\in Z_0(w)}{q}_k}{P}_{k,n}+{\displaystyle \sum _{r,s\in Z_1(w)}{\displaystyle \sum _{\begin{array}{c}m=1\\ k_m=w_m\end{array}}^M{q}_{r-e_m}{Q}_m{N}_{r,s}(w)P_{s,n}}},n\in Z_0(w);$$

(41)

$${\displaystyle \sum _{n\in Z_0(w)}{q}_n}1=1.$$

Proof of Theorem 6.

Consider the restricted process on $X_0(w)$. From Theorem 5.3.1 in [3], the transition matrix $P(X_0(w))$ of the restricted process is given using

$$P(X_0(w))=B(w)+A(w){(I-P(w))}^{-1}C(w).$$

(42)

The stationary probability vector $q(w)$ of the restricted process is proportional to ${\pi }_0(w)$ and is the unique solution of the system

$$q(w)=q(w)P(X_0(w)), q(w)1=1.$$

(43)

Let us partition the matrices $A(w)={[P_{k,n}]}_{k\in Z_0(w),n\in Z_1(w)}$, $B(w)={[P_{k,n}]}_{k,n\in Z_0(w)}$, and $C(w)={[P_{k,n}]}_{k\in Z_1(w),n\in Z_0(w)}$ into blocks of order $n$ and partition the vector $q$ into the subvectors $q_k$, $k\in Z_0(w)$. From (17) and (42), it follows that the system (43) may be written as (41), which completes the proof. □

5. Conclusions

This study presents several theoretical results that may be used to calculate the stationary distribution of simple Md-QBDs. It includes the following four stages.

(1)

First, there is a need to find the minimal non-negative solution $U$ of Equation (6) and calculate matrices $N={(I-U)}^{-1}$, $G_m=N{Q}_{-m}$, and $R_m=Q_{m}N$, $1\le m\le M$. This can be performed using the simple algorithms (14)–(16) or recursive procedures of Theorems 1 and 2;

(2)

Then, one must partition the set $Z={\mathbb{Z}}_{+}^M$ into levels $Z_l(1)$, $l\ge 0$, and determine the stationary probability vector $q={(q_n)}_{n\in Z_0(1)}$ of the restricted process on $X_0(1),$ which is the unique solution of the linear system in (41) with $w=1$;

(3)

After this, one must calculate a vector $x={(x_n)}_{n\in Z}$ proportional to the vector of the stationary probabilities. We set $x_n=q_n$ for $n\in Z_0(1)$ and then use Equation (39) to subsequently calculate subvectors $x_n$, $n\in Z_1(l1)$, for $l=1,2,\dots$;

(4)

Finally, the stationary probability vector $p$ may be obtained by normalizing the vector $x$ so that ${\displaystyle \sum _{n\in Z}{x}_{n}1}=1$.

For a practical implementation of this approach, it is necessary to overcome several difficulties similar to those encountered when calculating the stationary distribution of one-dimensional QBDs; see [22,24]. The normalization of $x$ might lead to problems of overflow if the value of $x1$ is much greater than l. To overcome such problems, one might renormalize the vector $x$ each time new subvectors $x_n$, $n\in Z_l(1)$, are computed during Stage 3. When $n$ is infinite, it is impossible to actually completely compute any infinite vectors and matrices, and it will be necessary to truncate this set of phases; see possible solutions in [49]. For Md-QBDs, it is also desirable to develop faster algorithms for calculating the matrices $U$ and $G_m$, similar to the logarithmic reduction algorithm for one-dimensional QBDs [21].