Non-Zero Sum Nash Game for Discrete-Time Infinite Markov Jump Stochastic Systems with Applications

Abstract

This paper is to study finite horizon linear quadratic (LQ) non-zero sum Nash game for discrete-time infinite Markov jump stochastic systems (IMJSSs). Based on the theory of stochastic analysis, a countably infinite set of coupled generalized algebraic Riccati equations are solved and a necessary and sufficient condition for the existence of Nash equilibrium points is obtained. From a new perspective, the finite horizon mixed robust $H_2/H_{\infty }$ control is investigated, and summarize the relationship between Nash game and $H_2/H_{\infty }$ control problem. Moreover, the feasibility and validity of the proposed method has been proved by applying it to a numerical example.

1. Introduction

Markov jump stochastic systems (MJSSs), as a typical class of stochastic hybrid dynamical systems, are widely used in practical engineering control systems. At the same time, a great deal of related research on the stability, ergodicity, robust control and filtering, and so on [1,2,3,4,5,6,7,8,9] has been undertaken. However, for the Markov chain, when the state space takes values in infinite countably sets, from the application point of view, it may be more extensive. Consequently, plentiful theory literature and research achievements have emerged for infinite Markov jump stochastic systems (IMJSSs). In recent years, various efforts have been made to cope with IMJSSs in a wide variety of systems. To be specific, via the operator spectrum method, exponential stabilities for discrete-time [10] and continuous-time [11] IMJSSs have been investigated respectively. Aiming at the practical time-delay factors, with the aid of Lyapunov stability theory, [12] discussed the stability analysis for IMJSSs with time-delay; further, stabilities for uncertain discrete-time IMJSSs with time-delay have been developed in [13], and on this basis, [14] addressed the finite horizon $H_2/H_{\infty }$ control problems. $H_2/H_{\infty }$ fuzzy filtering has been solved for nonlinear IMJSSs by the T-S fuzzy model approach in [15].

Dynamic game theory, which has come into wider use in many fields such as engineering, economics, and management science, has attracted great attention, and a large number of results have been obtained in the literature [16,17,18,19,20,21]. Furthermore, the Nash game problem has been studied for MJSSs, and further, a unified treatment approach for $H_2$, $H_{\infty }$ and $H_2/H_{\infty }$ control design problems was presented in [22]. In [23,24], the authors revealed the relationship between the Nash equilibrium points and $H_2/H_{\infty }$ control for continuous-time MJSSs. While many theoretical results have been established to study the stochastic system governed by a finite-state Markov chains, more research efforts are required to discuss IMJSSs. However, the causal and anticausal Lyapunov operators of IMJSSs are no more adjoint, which is the essential difference between finite and infinite MJSSs [10,25]. In most practical applications, IMJSSs have broader application prospects than MJSSs with finite jumps. As a consequence, the related research about IMJSSs is of critical importance. In the game field, [26] discussed an infinite horizon linear quadratic (LQ) Nash game for continuous-time IMJSSs. Unfortunately, to the best of our knowledge, there is almost no research about the Nash game problem for discrete-time IMJSSs.

In this article, we study a finite horizon LQ non-zero sum Nash game for discrete-time IMJSSs, which considers a more general system. From a new perspective, the finite horizon mixed $H_2/H_{\infty }$ control is further investigated. On the one hand, this is an extension of the previous study from MJSSs [22], to IMJSSs; on the other, it is a discrete-time case of [26]. Concretely, the main work and contribution of this thesis is as follows. First, the existence of Nash equilibrium points, which can be boiled down to the solvability of a countably infinite set of coupled generalized algebraic Riccati equations, is shown. For infinite Markov jump systems, the causal and anticausal Lyapunov operators are no more adjoint, which leads to the inequivalence between stochastic stability, asymptotic mean square stability, and exponential mean square stability. Further, this indicates the essential difference between the finite and infinite Markov jump systems. To this end, we introduce the infinite dimension Banach spaces, where the elements are countably infinite sequences of linear and bounded operators. Thus, the source of the problem resides in the way to solve this kind of equations, which are harder to solve than those in [22]. Second, the finite horizon mixed $H_2/H_{\infty }$ control is solved from the new view of a Nash game, and further, the relationship between the Nash game and $H_2/H_{\infty }$ control problem is summarized with some remarks. Finally, the typical example cited in the paper demonstrates the validity of the proposed method well.

The rest of this article is arranged as below. Some useful preliminary results are introduced in Section 2. The existence of Nash equilibrium points is solved by a necessary and sufficient condition in Section 3. In Section 4, some special cases with some remarks are given. In Section 5, a numerical example is illustrated, and a summary is provided in Section 6.

For convenience, the following notations are adopted. ${\mathcal{R}}^n$: n-dimensional real Euclidean space; ${\mathcal{R}}^{m\times n}$: the linear space of all m by n real matrices; $\parallel ·\parallel :$ the Euclidean norm of ${\mathcal{R}}^n$ or the operator norm of ${\mathcal{R}}^{m\times n}$; $I_n$: the $n\times n$ identity matrix; $N^{\prime }$: the transpose of a matrix (or vector) N; $N^{†}$: the pseudo-inverse of a matrix N; $N>0$$(\ge 0)$: N is positive (semi-positive) definite; $N_T:=\{0,1,\dots ,T\}$; $\mathcal{D}:=\{1,2,\dots \}$.

2. Preliminaries

On a complete probability space ($\mathsf{\Omega }$,$\mathcal{F}$,$\mathcal{P}$), we consider the following discrete-time IMJSS:

$$\left\{\begin{array}{cc}y(t+1)\hfill & =Q_0(t,{\xi }_t)y\left(t\right)+R_0(t,{\xi }_t)u\left(t\right)+U_0(t,{\xi }_t)v\left(t\right)\hfill \\ & +{\displaystyle \sum _{k=1}^r}[Q_k(t,{\xi }_t)y\left(t\right)+R_k(t,{\xi }_t)u\left(t\right)+U_k(t,{\xi }_t)v\left(t\right)]w_k\left(t\right),\hfill \\ z\left(t\right)=\hfill & \left[\begin{array}{c}L(t,{\xi }_t)y\left(t\right)\\ M(t,{\xi }_t)u\left(t\right)\end{array}\right],M{(t,{\xi }_t)}^{\prime }M(t,{\xi }_t)=I_{n_u},\hfill \\ y\left(0\right)=\hfill & y_0\in {\mathcal{R}}^n,\xi \left(0\right)={\xi }_0\in \mathcal{D},t\in N_T,\hfill \end{array}\right.$$

(1)

where $y\left(t\right)\in {\mathcal{R}}^n$ represents the system state, and $u\left(t\right)\in {\mathcal{R}}^{n_u}$ and $v\left(t\right)\in {\mathcal{R}}^{n_v}$ are the control processes of two different players, respectively. $z\left(t\right)\in {\mathcal{R}}^{n_z}$ stands for the controlled output. $w\left(t\right)=\left\{w\left(t\right)\right|w\left(t\right)=(w_1\left(t\right),w_2\left(t\right),\cdots ,w_r\left(t\right))\}$, $t\in N_T=\{0,1,\dots ,T\}$ is a standard r-dimensional Brownian motion. ${\left\{{\xi }_t\right\}}_{t\in N_T}$ denotes a infinite Markov jump process taking values in $\mathcal{D}=\{1,2,\dots \}$, and the transition probability matrix is $\mathbf{P}=[p(\varsigma ,\iota )]$ with $p(\varsigma ,\iota )=P({\xi }_{t+1}=\iota |{\xi }_t=\varsigma )$. In this paper, we assume $\mathbf{P}$ is nondegenerate, ${\pi }_0\left(\varsigma \right)=P({\xi }_0=\varsigma )>0$ for all $\varsigma \in \mathcal{D}$, and the stochastic processes ${\left\{w_t\right\}}_{t\in N_T}$ and ${\left\{{\xi }_t\right\}}_{t\in N_T}$ are mutually independent. Let ${\mathcal{F}}_t=\{{\xi }_k,w_s|0\le k\le t,0\le s\le t-1\},{\mathcal{F}}_0=\sigma \left({\xi }_0\right)$. $l^2(N_T;{\mathcal{R}}^m):=\{x\in {\mathcal{R}}^m|x$ is ${\mathcal{F}}_t$-measurable, and $({\sum }_{t=0}^T{E\parallel x\left(t\right)\parallel }^2{)}^{\frac{1}{2}}<\infty$}.

${\mathbb{H}}_{\infty }^{m\times n}$ is defined as the Banach space of the set $\left\{H\right|H=(H\left(1\right),H\left(2\right),\cdots ),H\left(\varsigma \right)\in {\mathcal{R}}^{m\times n}\}$ with the norm ${\parallel H\parallel }_{\infty }={sup}_{\varsigma \in \mathcal{D}}\parallel H\left(\varsigma \right)\parallel <\infty .$ In the sequel, we assume that all coefficients of considered systems have a finite norm ${\parallel ·\parallel }_{\infty }$. We make use of the notation of ${\mathbb{H}}_{\infty }^n$ for $m=n$. Further, ${\mathbb{H}}_{\infty }^{n+}$ denotes the subspace of ${\mathbb{H}}_{\infty }^n$ formed by all $H\left(\varsigma \right)\in S_n$ and $H\left(\varsigma \right)\ge 0,\varsigma \in \mathcal{D}$. For $\mathcal{L},\mathcal{M}\in {\mathbb{H}}_{\infty }^{n+}$, $\mathcal{L}\le \mathcal{M}$ is equivalent to $\mathcal{L}\left(\varsigma \right)\le \mathcal{M}\left(\varsigma \right),\varsigma \in \mathcal{D}$, which means ${\parallel \mathcal{L}\parallel }_{\infty }\le {\parallel \mathcal{M}\parallel }_{\infty }$. We set ${\mathbf{Q}}_k\left(t\right)={\left\{Q_k(t,\varsigma )\right\}}_{\varsigma \in \mathcal{D}}$, $0\le k\le r$, ${\mathbf{R}}_k\left(t\right)={\left\{R_k(t,\varsigma )\right\}}_{\varsigma \in \mathcal{D}},0\le k\le r$, ${\mathbf{U}}_k\left(t\right)={\left\{U_k(t,\varsigma )\right\}}_{\varsigma \in \mathcal{D}},0\le k\le r$, $\mathbf{L}\left(t\right)={\left\{L(t,\varsigma )\right\}}_{\varsigma \in \mathcal{D}}$, $\mathbf{M}\left(t\right)={\left\{M(t,\varsigma )\right\}}_{\varsigma \in \mathcal{D}}$, and assume ${\mathbf{Q}}_k\left(t\right)\in {\mathbb{H}}_{\infty }^n,$ ${\mathbf{R}}_k\left(t\right)\in {\mathbb{H}}_{\infty }^{n\times n_u},$ ${\mathbf{U}}_k\left(t\right)\in {\mathbb{H}}_{\infty }^{n\times n_v},$ $\mathbf{L}\left(t\right)\in {\mathbb{H}}_{\infty }^{n_z\times n},$ $\mathbf{M}\left(t\right)\in {\mathbb{H}}_{\infty }^{n_z\times n_u}$ for $t\in N_T$.

The two cost functions for the Nash game problem are given by

$$\begin{array}{ccc}\hfill & & J_1(x_0,{\xi }_0,u^{*}(·),v(·))=\sum _{t=0}^{T}E[{\gamma }^2{\parallel v\left(t\right)\parallel }^2-{\parallel z\left(t\right)\parallel }^2],\hfill \end{array}$$

(2)

$$\begin{array}{ccc}& & J_2(x_0,{\xi }_0,u(·),v^{*}(·))=\sum _{t=0}^T{E[\parallel z\left(t\right)\parallel }^2],\hfill \end{array}$$

(3)

where $\gamma >0$ is a given prescribed disturbance attenuation level. For simplicity, let the Nash game problem for cost Functions (2) and (3) with Equation (1) by the Nash game problem $\mathit{P}$. Hence, the Nash game problem $\mathit{P}$ is addressed, which is to find admissible control $(u^{*}(·),v^{*}(·))$ to minimize cost Functions (2) and (3) subject to Equation (1).

Next, we list some definitions and lemmas that are needed for the follow-up procedures.

Definition 1. 

A strategy pair $(u^{*}(·),v^{*}(·))\in l^2(N_T;{\mathcal{R}}^{n_u})\times l^2(N_T;{\mathcal{R}}^{n_v})$ is a Nash equilibrium point if

$$\begin{array}{ccc}& & J_1(x_0,{\xi }_0,u^{*}(·),v^{*}(·))\le J_1(x_0,{\xi }_0,u^{*}(·),v(·)),\hfill \end{array}$$

(4)

$$\begin{array}{ccc}& & J_2(x_0,{\xi }_0,u^{*}(·),v^{*}(·))\le J_2(x_0,{\xi }_0,u(·),v^{*}(·))\hfill \end{array}$$

(5)

for all $(u(·),v(·))\in l^2(N_T;{\mathcal{R}}^{n_u})\times l^2(N_T;{\mathcal{R}}^{n_v})$.

Lemma 1 

([27]). For a symmetric matrix S, we have

(i) 

$S^{†}={\left(S^{†}\right)}^{\prime }$;

(ii) 

$S{S}^{†}=S^{†}S$;

(iii) 

$S\ge 0$ if and only if $S^{†}\ge 0$.

Lemma 2 

([27]). Let matrices $F=F^{\prime }$, H, and $G=G^{\prime }$ with appropriate dimensions consider the following quadratic form:

$$\begin{array}{c}\hfill q(y,u)=E[y^{\prime }Fy+2y^{\prime }Hu+u^{\prime }Gu],\end{array}$$

where y and u are random variables defined on a probability space (Ω,$\mathcal{F}$,$\mathcal{P}$). Then, the following conditions are equivalent:

(i) 

${\displaystyle \underset{u}{inf}}q(y,u)>-\infty$ for any random variable y.

(ii) 

There exists a symmetric matrix $S=S^{\prime }$ such that ${\displaystyle \underset{u}{inf}}q(y,u)=E[y^{\prime }Sy]$ for any random variable y.

(iii) 

$G\ge 0$ and $H(I-G{G}^{†})=0$.

(iv) 

$G\ge 0$ and $Ker\left(G\right)\subseteq Ker\left(H\right)$.

(v) 

There exists a symmetric matrix $T=T^{\prime }$ such that $\left[\begin{array}{cc}F-T& H\\ H^{\prime }& G\end{array}\right]\ge 0$.

Moreover, if any of the above conditions hold, then (ii) is satisfied by $S=F-H{G}^{†}{H}^{\prime }$. In addition, $S\ge T$ for any T satisfying (v). Finally, for any random variable y, the random variable $u^{*}=-G^{†}{H}^{\prime }y$ is optimal and the optimal value is $q(y,u^{*})=E[y^{\prime }(F-H{G}^{†}{H}^{\prime })y]$.

The following LQ result for discrete-time IMJSSs can be directly yielded from Theorem 3.1 [14], which is the continuous-time case.

Lemma 3. 

For the following standard LQ optimal control problem with discrete-time IMJSSs,

$$\begin{array}{c}{\displaystyle \underset{u(·)\in l^2(N_T;{\mathcal{R}}^{n_u})}{min}}\{J_2(y_0,{\xi }_0,u(·))={\displaystyle \sum _{t=0}^T}{E[\parallel z\left(t\right)\parallel }^2]\},\hfill \end{array}$$

(6)

subject to

$$\left\{\begin{array}{c}y(t+1)=Q_0(t,{\xi }_t)y\left(t\right)+R_0(t,{\xi }_t)u\left(t\right)+{\displaystyle \sum _{k=1}^r}[Q_k(t,{\xi }_t)y\left(t\right)+R_k(t,{\xi }_t)u\left(t\right)]w_k\left(t\right),\hfill \\ z\left(t\right)=\left[\begin{array}{c}L(t,{\xi }_t)y\left(t\right)\\ M(t,{\xi }_t)u\left(t\right)\end{array}\right],M{(t,{\xi }_t)}^{\prime }M(t,{\xi }_t)=I_{n_u},\hfill \\ y\left(0\right)=y_0\in {\mathcal{R}}^n,\xi \left(0\right)={\xi }_0\in \mathcal{D},t\in N_T,\hfill \end{array}\right.$$

(7)

then, we find that $P(t,\varsigma ),(t,\varsigma )\in N_T\times \mathcal{D}$ solves the following coupled generalized algebraic Riccati equations:

$$\begin{array}{ccc}& & \left\{\begin{array}{c}P(t,\varsigma )={\displaystyle \sum _{k=0}^r}{Q}_k{(t,\varsigma )}^{\prime }\epsilon {}_{\varsigma }(t,P)Q_k(t,\varsigma )+L{(t,\varsigma )}^{\prime }L(t,\varsigma )\hfill \\ -{[{\displaystyle \sum _{k=0}^r}{Q}_k{(t,\varsigma )}^{\prime }\epsilon {}_{\varsigma }(t,P)R_k(t,\varsigma )]}^{\prime }\hfill \\ ·H{(t,\varsigma ,P)}^{-1}[{\displaystyle \sum _{k=0}^r}{Q}_k{(t,\varsigma )}^{\prime }\epsilon {}_{\varsigma }(t,P)R_k(t,\varsigma )],\hfill \\ P(T+1,\varsigma )=0,H(t,\varsigma ,P)>0,(t,\varsigma )\in N_T\times \mathcal{D},\hfill \end{array}\right.\hfill \end{array}$$

(8)

where

$$H(t,\varsigma ,P)=I_{n_u}+\sum _{k=0}^r{R}_k{(t,\varsigma )}^{\prime }\epsilon {}_{\varsigma }(t,P)R_k(t,\varsigma ),$$

$$\epsilon {}_{\varsigma }(t,P)=\sum _{\iota =1}^{\infty }p(\varsigma ,\iota )P(t+1,\iota ),$$

and

$$\underset{u(·)\in l^2(N_T;{\mathcal{R}}^{n_u})}{min}{J}_2(y_0,{\xi }_0,u(·))=J_2(y_0,{\xi }_0,u^{*}(·))=\sum _{\varsigma =1}^{\infty }{\pi }_0\left(\varsigma \right)y_0^{\prime }P(0,\varsigma )y_0,$$

$$u^{*}\left(t\right)=-H{(t,\varsigma ,P)}^{-1}[\sum _{k=0}^r{Q}_k{(t,\varsigma )}^{\prime }\epsilon {}_{\varsigma }(t,P)R_k(t,\varsigma )]y\left(t\right).$$

3. Nash Equilibrium Points

This section focuses on solving the Nash gameproblem $\mathit{P}$, and we assume that the linear, memoryless feedback strategy has the following form [28]:

$$\begin{array}{c}\hfill u\left(t\right)=G_2(t,{\xi }_t)y\left(t\right),v\left(t\right)=G_1(t,{\xi }_t)y\left(t\right),\end{array}$$

(9)

under this assumption, we obtain the feedback Nash equilibrium points by a countably infinite set of coupled generalized algebraic Riccati equations.

Theorem 1. 

The Nash game problem $\mathit{P}$ has unique Nash equilibrium points

$$(u^{*}\left(t\right)=G_2(t,{\xi }_t,X_2)y\left(t\right),v^{*}\left(t\right)=G_1(t,{\xi }_t,X_1)y\left(t\right))$$

iff the following coupled generalized algebraic Riccati equations,

$$\begin{array}{ccc}& & \left\{\begin{array}{c}{\mathsf{\Lambda }}_1(t,\varsigma )={\displaystyle \sum _{k=0}^r}{[Q_k(t,\varsigma )+R_k(t,\varsigma )G_2(t,\varsigma ,{\mathsf{\Lambda }}_2)]}^{\prime }\epsilon {}_{\varsigma }(t,{\mathsf{\Lambda }}_1)\hfill \\ ·[Q_k(t,\varsigma )+R_k(t,\varsigma )G_2(t,\varsigma ,{\mathsf{\Lambda }}_2)]-L{(t,\varsigma )}^{\prime }L(t,\varsigma )\hfill \\ -G_2{(t,\varsigma ,{\mathsf{\Lambda }}_2)}^{\prime }{G}_2(t,\varsigma ,{\mathsf{\Lambda }}_2)-G_3{(t,\varsigma ,{\mathsf{\Lambda }}_1)}^{\prime }{H}_1{(t,\varsigma ,{\mathsf{\Lambda }}_1)}^{†}{G}_3(t,\varsigma ,{\mathsf{\Lambda }}_1),\hfill \\ (I-H_1(t,\varsigma ,{\mathsf{\Lambda }}_1)H_1{(t,\varsigma ,{\mathsf{\Lambda }}_1)}^{†})G_3(t,\varsigma ,{\mathsf{\Lambda }}_1)=0,\hfill \\ {\mathsf{\Lambda }}_1(T+1,\varsigma )=0,H_1(t,\varsigma ,{\mathsf{\Lambda }}_1)\ge 0,(t,\varsigma )\in N_T\times \mathcal{D},\hfill \end{array}\right.\hfill \end{array}$$

(10)

$$\begin{array}{ccc}& & G_1(t,\varsigma ,{\mathsf{\Lambda }}_1)=-H_1{(t,\varsigma ,{\mathsf{\Lambda }}_1)}^{†}{G}_3(t,\varsigma ,{\mathsf{\Lambda }}_1),\hfill \end{array}$$

(11)

$$\begin{array}{ccc}& & \left\{\begin{array}{c}{\mathsf{\Lambda }}_2(t,\varsigma )={\displaystyle \sum _{k=0}^r}{[Q_k(t,\varsigma )+U_k(t,\varsigma )G_1(t,\varsigma ,{\mathsf{\Lambda }}_1)]}^{\prime }\epsilon {}_{\varsigma }(t,{\mathsf{\Lambda }}_2)\hfill \\ ·[Q_k(t,\varsigma )+U_k(t,\varsigma )G_1(t,\varsigma ,{\mathsf{\Lambda }}_1)]+L{(t,\varsigma )}^{\prime }L(t,\varsigma )\hfill \\ -G_4{(t,\varsigma ,{\mathsf{\Lambda }}_2)}^{\prime }{H}_2{(t,\varsigma ,{\mathsf{\Lambda }}_2)}^{-1}{G}_4(t,\varsigma ,{\mathsf{\Lambda }}_2),\hfill \\ {\mathsf{\Lambda }}_2(T+1,\varsigma )=0,H_2(t,\varsigma ,{\mathsf{\Lambda }}_2)>0,(t,\varsigma )\in N_T\times \mathcal{D},\hfill \end{array}\right.\hfill \end{array}$$

(12)

$$\begin{array}{ccc}& & G_2(t,\varsigma ,{\mathsf{\Lambda }}_2)=-H_2{(t,\varsigma ,{\mathsf{\Lambda }}_2)}^{-1}{G}_4(t,\varsigma ,{\mathsf{\Lambda }}_2)\hfill \end{array}$$

(13)

admit a group of solutions $({\mathsf{\Lambda }}_1(t,\varsigma ),G_1(t,\varsigma ,{\mathsf{\Lambda }}_1);{\mathsf{\Lambda }}_2(t,\varsigma ),G_2(t,\varsigma ,{\mathsf{\Lambda }}_2))$ with ${\mathsf{\Lambda }}_1(t,\varsigma )\le 0$, ${\mathsf{\Lambda }}_2(t,\varsigma )\ge 0$ for $(t,\varsigma )\in N_T\times \mathcal{D}$, where

$$\begin{array}{ccc}& & H_1(t,\varsigma ,{\mathsf{\Lambda }}_1)={\gamma }^2{I}_{n_v}+\sum _{k=0}^r{U}_k{(t,\varsigma )}^{\prime }\epsilon {}_{\varsigma }(t,{\mathsf{\Lambda }}_1)U_k(t,\varsigma ),\hfill \\ & & H_2(t,\varsigma ,{\mathsf{\Lambda }}_2)=I_{n_u}+\sum _{k=0}^r{R}_k{(t,\varsigma )}^{\prime }\epsilon {}_{\varsigma }(t,{\mathsf{\Lambda }}_2)R_k(t,\varsigma ),\hfill \\ & & G_3(t,\varsigma ,{\mathsf{\Lambda }}_1)=\sum _{k=0}^r{[Q_k(t,\varsigma )+R_k(t,\varsigma )G_2(t,\varsigma ,{\mathsf{\Lambda }}_2)]}^{\prime }\epsilon {}_{\varsigma }(t,{\mathsf{\Lambda }}_1)U_k(t,\varsigma ),\hfill \\ & & G_4(t,\varsigma ,{\mathsf{\Lambda }}_2)=\sum _{k=0}^r{[Q_k(t,\varsigma )+U_k(t,\varsigma )G_1(t,\varsigma ,{\mathsf{\Lambda }}_1)]}^{\prime }\epsilon {}_{\varsigma }(t,{\mathsf{\Lambda }}_2)R_k(t,\varsigma ).\hfill \end{array}$$

Proof. 

Sufficiency: Since Equations (10)–(13) have a group of solutions

$$({\mathsf{\Lambda }}_1(t,\varsigma ),G_1(t,\varsigma ,{\mathsf{\Lambda }}_1);{\mathsf{\Lambda }}_2(t,\varsigma ),G_2(t,\varsigma ,{\mathsf{\Lambda }}_2)),$$

by constructing $u^{*}\left(t\right)=G_2(t,{\xi }_t,{\mathsf{\Lambda }}_2)y\left(t\right)$, then, we substitute $u^{*}\left(t\right)$ into Equation (1) to get the equation as follows:

$$\left\{\begin{array}{c}y(t+1)=[Q_0(t,{\xi }_t)+R_0(t,{\xi }_t)G_2(t,{\xi }_t,{\mathsf{\Lambda }}_2)]y\left(t\right)+U_0(t,{\xi }_t)v\left(t\right)\hfill \\ +{\displaystyle \sum _{k=1}^r}\{[Q_k(t,{\xi }_t)+R_k(t,{\xi }_t)G_2(t,{\xi }_t,{\mathsf{\Lambda }}_2)]y\left(t\right)+U_k(t,{\xi }_t)v\left(t\right)\}{w}_k\left(t\right),\hfill \\ z\left(t\right)=\left[\begin{array}{c}L(t,{\xi }_t)y\left(t\right)\\ M(t,{\xi }_t)G_2(t,{\xi }_t,{\mathsf{\Lambda }}_2)y\left(t\right)\end{array}\right],M{(t,{\xi }_t)}^{\prime }M(t,{\xi }_t)=I_{n_u},\hfill \\ y\left(0\right)=y_0\in {\mathcal{R}}^n,\xi \left(0\right)={\xi }_0\in \mathcal{D},t\in N_T,\hfill \end{array}\right.$$

(14)

using similar processing, Lemma 1 in [29] can be generalized to the infinite Markov jump systems. In fact, via the assumption that ${\left\{w_t\right\}}_{t\in N_T}$ and ${\left\{{\xi }_t\right\}}_{t\in N_T}$ are mutually independent, and ${\left\{w_t\right\}}_{t\in N_T}$ is also irrelevant to $v\left(t\right)$, we have

$$\begin{array}{ccc}\hfill & & E[y{(t+1)}^{\prime }{\mathsf{\Lambda }}_1(t+1,{\xi }_{t+1})y(t+1)-y{\left(t\right)}^{\prime }{\mathsf{\Lambda }}_1(t,{\xi }_t)y\left(t\right)|{\mathcal{F}}_t,{\xi }_t=\varsigma ]\hfill \\ & & ={\left[\begin{array}{c}y\left(t\right)\\ v\left(t\right)\end{array}\right]}^{\prime }\left[\begin{array}{cc}C(t,\varsigma ,{\mathsf{\Lambda }}_1)-{\mathsf{\Lambda }}_1(t,\varsigma )& G_3(t,\varsigma ,{\mathsf{\Lambda }}_1)\\ G_3{(t,\varsigma ,{\mathsf{\Lambda }}_1)}^{\prime }& H_1(t,\varsigma ,{\mathsf{\Lambda }}_1)-{\gamma }^2{I}_{n_v}\end{array}\right]\left[\begin{array}{c}y\left(t\right)\\ v\left(t\right)\end{array}\right],\hfill \end{array}$$

(15)

it follows from taking summation from 0 to T on both sides of (15) that

$$\begin{array}{ccc}& & E[y_{T+1}^{\prime }{\mathsf{\Lambda }}_1(T+1,{\xi }_{T+1})y_{T+1}-y_0^{\prime }{\mathsf{\Lambda }}_1(0,{\xi }_0)y_0]\hfill \\ & & =\sum _{t=0}^{T}E{\left[\begin{array}{c}y\left(t\right)\\ v\left(t\right)\end{array}\right]}^{\prime }\left[\begin{array}{cc}C(t,\varsigma ,{\mathsf{\Lambda }}_1)-{\mathsf{\Lambda }}_1(t,\varsigma )& G_3(t,\varsigma ,{\mathsf{\Lambda }}_1)\\ G_3{(t,\varsigma ,{\mathsf{\Lambda }}_1)}^{\prime }& H_1(t,\varsigma ,{\mathsf{\Lambda }}_1)-{\gamma }^2{I}_{n_v}\end{array}\right]\left[\begin{array}{c}y\left(t\right)\\ v\left(t\right)\end{array}\right].\hfill \end{array}$$

Further, we can obtain the following result:

$$\begin{array}{ccc}\hfill & & J_1(y_0,{\xi }_0,u^{*}(·),v(·))=\sum _{t=0}^{T}E[{\gamma }^2{\parallel v\left(t\right)\parallel }^2-{\parallel z\left(t\right)\parallel }^2]\hfill \\ & & =E[y_0^{\prime }{\mathsf{\Lambda }}_1(0,{\xi }_0)y_0]-E[y_{T+1}^{\prime }{\mathsf{\Lambda }}_1(T+1,{\xi }_{T+1})y_{T+1}]\hfill \\ & & +\sum _{t=0}^{T}E{\left[\begin{array}{c}y\left(t\right)\\ v\left(t\right)\end{array}\right]}^{\prime }D(t,{\xi }_t,{\mathsf{\Lambda }}_1)\left[\begin{array}{c}y\left(t\right)\\ v\left(t\right)\end{array}\right],\hfill \end{array}$$

(16)

where

$$\begin{array}{ccc}& & D(t,{\xi }_t,{\mathsf{\Lambda }}_1)=D(t,\varsigma ,{\mathsf{\Lambda }}_1)=\left[\begin{array}{cc}C(t,\varsigma ,{\mathsf{\Lambda }}_1)-L{(t,\varsigma )}^{\prime }L(t,\varsigma )-{\mathsf{\Lambda }}_1(t,\varsigma )& G_3(t,\varsigma ,{\mathsf{\Lambda }}_1)\\ G_3{(t,\varsigma ,{\mathsf{\Lambda }}_1)}^{\prime }& H_1(t,\varsigma ,{\mathsf{\Lambda }}_1)\end{array}\right],\hfill \\ & & C(t,\varsigma ,{\mathsf{\Lambda }}_1)=\sum _{k=0}^r{[Q_k(t,\varsigma )+R_k(t,\varsigma )G_2(t,\varsigma ,{\mathsf{\Lambda }}_2)]}^{\prime }\epsilon {}_{\varsigma }(t,{\mathsf{\Lambda }}_1)\hfill \\ & & ·[Q_k(t,\varsigma )+R_k(t,\varsigma )G_2(t,\varsigma ,{\mathsf{\Lambda }}_2)],\hfill \end{array}$$

for ${\xi }_t=\varsigma .$ A combination of the method of completing square and (10) turns Equation (16) into

$$\begin{array}{ccc}\hfill & & J_1(y_0,{\xi }_0,u^{*}(·),v(·))=E[y_0^{\prime }{\mathsf{\Lambda }}_1(0,{\xi }_0)y_0]\hfill \\ & & +\sum _{t=0}^{T}E\left\{E\left\{{\left[\begin{array}{c}y\left(t\right)\\ v\left(t\right)\end{array}\right]}^{\prime }D(t,{\xi }_t,{\mathsf{\Lambda }}_1)\left[\begin{array}{c}y\left(t\right)\\ v\left(t\right)\end{array}\right]|{\mathcal{F}}_{t-1},{\xi }_t=\varsigma \right\}\right\}\hfill \\ & & =E[y_0^{\prime }{\mathsf{\Lambda }}_1(0,{\xi }_0)y_0]\hfill \\ & & +\sum _{t=0}^{T}E\left\{E\left\{{[v\left(t\right)-v^{*}\left(t\right)]}^{\prime }{H}_1(t,\varsigma ,{\mathsf{\Lambda }}_1)[v\left(t\right)-v^{*}\left(t\right)]|{\mathcal{F}}_{t-1},{\xi }_t=\varsigma \right\}\right\},\hfill \end{array}$$

(17)

where $v^{*}\left(t\right)=G_1(t,{\xi }_t,{\mathsf{\Lambda }}_1)y\left(t\right)$; therefore, the Nash equilibrium inequality (4) occurs naturally, that is to say

$$\begin{array}{c}\hfill J_1(y_0,{\xi }_0,u^{*}(·),v(·))\ge J_1({\mathsf{\Lambda }}_0,{\xi }_0,u^{*}(·),v^{*}(·))=E[y_0^{\prime }{\mathsf{\Lambda }}_1(0,{\xi }_0)y_0],\end{array}$$

additionally, by plugging $v^{*}\left(t\right)=G_1(t,{\xi }_t,{\mathsf{\Lambda }}_1)y\left(t\right)$ into Equation (1), Equation (1) can be converted into

$$\left\{\begin{array}{c}y(t+1)=[Q_0(t,{\xi }_t)+U_0(t,{\xi }_t)G_1(t,{\xi }_t,{\mathsf{\Lambda }}_1)]y\left(t\right)+R_0(t,{\xi }_t)u\left(t\right)\hfill \\ +{\displaystyle \sum _{k=1}^r}\{[Q_k(t,{\xi }_t)+U_k(t,{\xi }_t)G_1(t,{\xi }_t,{\mathsf{\Lambda }}_1)]y\left(t\right)+R_k(t,{\xi }_t)u\left(t\right)\}{w}_k\left(t\right),\hfill \\ z\left(t\right)=\left[\begin{array}{c}L(t,{\xi }_t)y\left(t\right)\\ M(t,{\xi }_t)u\left(t\right)\end{array}\right],M{(t,{\xi }_t)}^{\prime }M(t,{\xi }_t)=I_{n_u},\hfill \\ y\left(0\right)=y_0\in {\mathcal{R}}^n,\xi \left(0\right)={\xi }_0\in \mathcal{D},t\in N_T,\hfill \end{array}\right.$$

(18)

in the meantime, we also have

$$\begin{array}{c}\hfill J_2(y_0,{\xi }_0,u(·),v^{*}(·))=\sum _{t=0}^{T}E[y{\left(t\right)}^{\prime }L{(t,{\xi }_t)}^{\prime }L(t,{\xi }_t)y\left(t\right)+u{\left(t\right)}^{\prime }u\left(t\right)].\end{array}$$

(19)

Obviously, to illustrate that $u^{*}\left(t\right)$ minimizes (3), we only need to prove a kind of LQ optimal control problem of IMJSSs that causes (19) minimization under the condition of (18). Via Lemma 3, we can easily prove the Nash equilibrium inequality (5).

Necessity: Assume that $(u^{*}\left(t\right),v^{*}\left(t\right))=(G_2(t,{\xi }_t)y\left(t\right),G_1(t,{\xi }_t)y\left(t\right))$ is a linear feedback Nash equilibrium point for the Nash game (4) and (5). Putting $u^{*}\left(t\right)$ into Equation (1), we gain

$$\left\{\begin{array}{c}y(t+1)=[Q_0(t,{\xi }_t)+R_0(t,{\xi }_t)G_2(t,{\xi }_t)]y\left(t\right)+U_0(t,{\xi }_t)v\left(t\right)\hfill \\ +{\displaystyle \sum _{k=1}^r}\{[Q_k(t,{\xi }_t)+R_k(t,{\xi }_t)G_2(t,{\xi }_t)]y\left(t\right)+U_k(t,{\xi }_t)v\left(t\right)\}{w}_k\left(t\right),\hfill \\ z\left(t\right)=\left[\begin{array}{c}L(t,{\xi }_t)y\left(t\right)\\ M(t,{\xi }_t)G_2(t,{\xi }_t)y\left(t\right)\end{array}\right],M{(t,{\xi }_t)}^{\prime }M(t,{\xi }_t)=I_{n_u},\hfill \\ y\left(0\right)=y_0\in {\mathcal{R}}^n,\xi \left(0\right)={\xi }_0\in \mathcal{D},t\in N_T,\hfill \end{array}\right.$$

(20)

now discover that the above problem is transformed into the following indefinite LQ optimal control with optimal solution $v^{*}\left(t\right)=K_1(t,{\xi }_t)y\left(t\right)$:

$$\left\{\begin{array}{c}{\displaystyle \underset{v(·)\in l^2(N_T;{\mathcal{R}}^{n_v})}{min}}\{J_1(y_0,{\xi }_0,u^{*}(·),v(·))={\displaystyle \sum _{t=0}^T}E[{\gamma }^2{\parallel v\left(t\right)\parallel }^2-{\parallel z\left(t\right)\parallel }^2]\},\hfill \\ subjectto\left(20\right),\hfill \end{array}\right.$$

(21)

it is obvious that the above indefinite LQ problem (21) is well-posed. Next, we prove that the coupled generalized algebraic Riccati Equation (22) is solvable by mathematical induction:

$$\begin{array}{ccc}\hfill & & \left\{\begin{array}{c}{\tilde{\mathsf{\Lambda }}}_1(t,\varsigma )={\displaystyle \sum _{k=0}^r}{[Q_k(t,\varsigma )+R_k(t,\varsigma )G_2(t,\varsigma )]}^{\prime }\epsilon {}_{\varsigma }(t,{\tilde{\mathsf{\Lambda }}}_1)[Q_k(t,\varsigma )+R_k(t,\varsigma )G_2(t,\varsigma )]\hfill \\ -L{(t,\varsigma )}^{\prime }L(t,\varsigma )-G_2{(t,\varsigma )}^{\prime }{G}_2(t,\varsigma )\hfill \\ -G_3{(t,\varsigma ,{\tilde{\mathsf{\Lambda }}}_1)}^{\prime }{H}_1{(t,\varsigma ,{\tilde{\mathsf{\Lambda }}}_1)}^{†}{G}_3(t,\varsigma ,{\tilde{\mathsf{\Lambda }}}_1),\hfill \\ (I-H_1(t,\varsigma ,{\tilde{\mathsf{\Lambda }}}_1)H_1{(t,\varsigma ,{\tilde{\mathsf{\Lambda }}}_1)}^{†})G_3(t,\varsigma ,{\tilde{\mathsf{\Lambda }}}_1)=0,\hfill \\ {\tilde{\mathsf{\Lambda }}}_1(T+1,\varsigma )=0,H_1(t,\varsigma ,{\tilde{\mathsf{\Lambda }}}_1)\ge 0,(t,\varsigma )\in N_T\times \mathcal{D},\hfill \end{array}\right.\hfill \end{array}$$

(22)

to this end, the value function is introduced as follows:

$$\begin{array}{ccc}\hfill & & V(\tau ,y\left(\tau \right),{\xi }_{\tau })=\underset{v(·)\in l^2([\tau ,T];{\mathcal{R}}^{n_v})}{inf}{J}_1(y\left(\tau \right),{\xi }_{\tau },u^{*}(·),v(·))\hfill \\ & & =\underset{v(·)\in l^2([\tau ,T];{\mathcal{R}}^{n_v})}{inf}\sum _{t=\tau }^T{E\{y\left(t\right)}^{\prime }[-L{(t,{\xi }_t)}^{\prime }L(t,{\xi }_t)-G_2{(t,{\xi }_t)}^{\prime }{G}_2(t,{\xi }_t)]y\left(t\right)\hfill \\ & & +{\gamma }^{2}v{\left(t\right)}^{\prime }v\left(t\right)\},\hfill \end{array}$$

(23)

for $\tau =T$, we can obtain from ${\tilde{\mathsf{\Lambda }}}_1(T+1,i)=0$ that the existence of ${\tilde{\mathsf{\Lambda }}}_1(T,i)$ motivates

$$\begin{array}{ccc}\hfill & & V(T,y\left(T\right),{\xi }_T)=\underset{v\left(T\right)}{inf}{E\{y\left(T\right)}^{\prime }[-L{(T,{\xi }_T)}^{\prime }L(T,{\xi }_T)-G_2{(T,{\xi }_T)}^{\prime }{G}_2(T,{\xi }_T)]y\left(T\right)\hfill \\ & & +{\gamma }^{2}v{\left(T\right)}^{\prime }v\left(T\right)\}\hfill \\ & & =\underset{v\left(T\right)}{inf}E[y{\left(T\right)}^{\prime }{\tilde{\mathsf{\Lambda }}}_1(T,\varsigma )y\left(T\right)+{\gamma }^{2}v{\left(T\right)}^{\prime }v\left(T\right)],\hfill \end{array}$$

(24)

it is noteworthy that Equation (22) is available for $t=T$ with ${\tilde{\mathsf{\Lambda }}}_1(T,\varsigma )$, and the optimal value function is

$$\begin{array}{c}\hfill V(T,y\left(T\right),{\xi }_T)=E[y{\left(T\right)}^{\prime }{\tilde{\mathsf{\Lambda }}}_1(T,{\xi }_T)y\left(T\right)],\end{array}$$

(25)

in the process,

$$\begin{array}{c}\hfill v^{*}\left(T\right)=0=-H_1{(T,{\xi }_T,{\tilde{\mathsf{\Lambda }}}_1)}^{†}{G}_3(T,{\xi }_T,{\tilde{\mathsf{\Lambda }}}_1)y\left(T\right).\end{array}$$

(26)

then, for $t=\tau$, assume that the coupled generalized algebraic Riccati Equation (22) has a solution ${\tilde{\mathsf{\Lambda }}}_1(\tau ,\varsigma ),\varsigma \in \mathcal{D}$. Meanwhile, the optimal value function is

$$\begin{array}{c}\hfill V(\tau ,y\left(\tau \right),{\xi }_{\tau })=\sum _{i=1}^{\infty }{\pi }_{\tau }\left(i\right)y{\left(\tau \right)}^{\prime }{\tilde{\mathsf{\Lambda }}}_1(\tau ,{\xi }_{\tau })y\left(\tau \right),\end{array}$$

and the optimal control is

$$\begin{array}{c}\hfill v^{*}\left(\tau \right)=-H_1{(\tau ,{\xi }_{\tau },{\tilde{\mathsf{\Lambda }}}_1)}^{†}{G}_3(\tau ,{\xi }_{\tau },{\tilde{\mathsf{\Lambda }}}_1)y\left(\tau \right),\end{array}$$

now, our goal is to illustrate that for $t=\tau -1$, the existence of solution ${\tilde{\mathsf{\Lambda }}}_1(\tau -1,\varsigma ),\varsigma \in \mathcal{D}$ to Equation (22). By the aid of (20), the dynamic programming optimality principle applies to

$$\begin{array}{ccc}\hfill & & V(\tau -1,y(\tau -1),{\xi }_{\tau -1})=\underset{v(\tau -1)}{inf}E\{y{(\tau -1)}^{\prime }[-L{(\tau -1,{\xi }_{\tau -1})}^{\prime }L(\tau -1,{\xi }_{\tau -1})\hfill \\ & & -G_2{(\tau -1,{\xi }_{\tau -1})}^{\prime }{G}_2(\tau -1,{\xi }_{\tau -1})]y(\tau -1)\hfill \\ & & +{\gamma }^{2}v{(\tau -1)}^{\prime }v(\tau -1)+V(\tau ,y\left(\tau \right),{\xi }_{\tau })\},\hfill \\ & & =\underset{v(\tau -1)}{inf}{E[y(\tau -1)}^{\prime }\mathsf{\Gamma }(\tau -1,{\xi }_{\tau -1},{\tilde{\mathsf{\Lambda }}}_1)y(\tau -1)\hfill \\ & & +2v{(\tau -1)}^{\prime }{G}_3{(\tau -1,{\xi }_{\tau -1},{\tilde{\mathsf{\Lambda }}}_1)}^{\prime }y(\tau -1)\hfill \\ & & +v{(\tau -1)}^{\prime }{H}_1(\tau -1,{\xi }_{\tau -1},{\tilde{\mathsf{\Lambda }}}_1)v(\tau -1)],\hfill \end{array}$$

(27)

where

$$\begin{array}{ccc}& & \mathsf{\Gamma }(\tau -1,{\xi }_{\tau -1},{\tilde{\mathsf{\Lambda }}}_1)=\sum _{k=0}^r{[Q_k(\tau -1,{\xi }_{\tau -1})+R_k(\tau -1,{\xi }_{\tau -1})G_2(\tau -1,{\xi }_{\tau -1})]}^{\prime }\hfill \\ & & ·\epsilon {}_{{\xi }_{\tau -1}}(\tau -1,{\tilde{\mathsf{\Lambda }}}_1)[Q_k(\tau -1,{\xi }_{\tau -1})+R_k(\tau -1,{\xi }_{\tau -1})G_2(\tau -1,{\xi }_{\tau -1})]\hfill \\ & & -L{(\tau -1,{\xi }_{\tau -1})}^{\prime }L(\tau -1,{\xi }_{\tau -1})-G_2{(\tau -1,{\xi }_{\tau -1})}^{\prime }{G}_2(\tau -1,{\xi }_{\tau -1}),\hfill \end{array}$$

similar results to Lemma 2 apply to the infinite Markov jump case, it is demonstrated that ${\tilde{\mathsf{\Lambda }}}_1(\tau -1,\varsigma ),\varsigma \in \mathcal{D}$ satisfies the following equation:

$$\left\{\begin{array}{c}{\tilde{\mathsf{\Lambda }}}_1(\tau -1,\varsigma )=\Gamma (\tau -1,\varsigma ,{\tilde{\mathsf{\Lambda }}}_1)-G_3{(\tau -1,\varsigma ,{\tilde{\mathsf{\Lambda }}}_1)}^{\prime }{H}_1{(\tau -1,\varsigma ,{\tilde{\mathsf{\Lambda }}}_1)}^{†}{G}_3(\tau -1,\varsigma ,{\tilde{\mathsf{\Lambda }}}_1),\hfill \\ (I-H_1(\tau -1,\varsigma ,{\tilde{\mathsf{\Lambda }}}_1)H_1{(\tau -1,\varsigma ,{\tilde{\mathsf{\Lambda }}}_1)}^{†})G_3(\tau -1,\varsigma ,{\tilde{\mathsf{\Lambda }}}_1)=0,\hfill \\ {\tilde{\mathsf{\Lambda }}}_1(T+1,\varsigma )=0,H_1(\tau -1,\varsigma ,{\tilde{\mathsf{\Lambda }}}_1)\ge 0,\varsigma \in \mathcal{D},\hfill \end{array}\right.$$

(28)

the following conclusions can be handled in the same manner as (17), that is

$$\begin{array}{c}\hfill V(\tau -1,y(\tau -1),{\xi }_{\tau -1})=\sum _{\varsigma =1}^{\infty }{\pi }_{\tau -1}\left(i\right)y{(\tau -1)}^{\prime }P(\tau -1,\varsigma )y(\tau -1),\end{array}$$

(29)

and

$$\begin{array}{c}\hfill v^{*}(\tau -1)=-H_1{(\tau -1,{\xi }_{\tau -1},{\tilde{\mathsf{\Lambda }}}_1)}^{†}{G}_3(\tau -1,{\xi }_{\tau -1},{\tilde{\mathsf{\Lambda }}}_1)y(\tau -1),\end{array}$$

(30)

up to the present, one can infer that there exists ${\tilde{\mathsf{\Lambda }}}_1(t,\varsigma ),(t,\varsigma )\in N_T\times \mathcal{D}$ satisfying Equation (22). Furthermore, an optimal solution of the indefinite LQ problem (21) is $v^{*}\left(t\right)=-H_1{(t,{\xi }_t,{\tilde{\mathsf{\Lambda }}}_1)}^{†}{G}_3(t,{\xi }_t,{\tilde{\mathsf{\Lambda }}}_1)y\left(t\right)$ with $G_1(t,{\xi }_t)=-H_1{(t,{\xi }_t,{\tilde{\mathsf{\Lambda }}}_1)}^{†}{G}_3(t,{\xi }_t,{\tilde{\mathsf{\Lambda }}}_1)=G_1(t,{\xi }_t,{\tilde{\mathsf{\Lambda }}}_1)$. Amalgamating the above results into Equation (22), it is stated that ${\tilde{\mathsf{\Lambda }}}_1(t,\varsigma )={\mathsf{\Lambda }}_1(t,\varsigma ),(t,\varsigma )\in N_T\times \mathcal{D}$. It only remains to indicate ${\mathsf{\Lambda }}_1(t,\varsigma )\le 0,(t,\varsigma )\in N_T\times \mathcal{D}$. In practice,

$$\begin{array}{c}\hfill J_1(y_0,\varsigma ,u^{*}(·),v^{*}(·))=\sum _{\varsigma =1}^{\infty }{\pi }_0\left(\varsigma \right)y_0^{\prime }{\mathsf{\Lambda }}_1(0,\varsigma )y_0\le J_1(y_0,{\xi }_0,u^{*}(·),0)=\sum _{t=0}^T{E[-\parallel z\left(t\right)\parallel }^2]\}\le 0,\end{array}$$

in addition, if we plug $v^{*}\left(t\right)=G_1(t,{\xi }_t)y\left(t\right)$ into Equation (1), then (18) is gained. We can deduce that (5) is a standard LQ control subject to (18). Using Lemma 3, ${\mathsf{\Lambda }}_2(t,\varsigma )\ge 0,(t,\varsigma )\in N_T\times \mathcal{D}$ can be easily accessible. Moreover, $J_2(y_0,\varsigma ,u^{*}(·),v^{*}(·))={\displaystyle \sum _{\varsigma =1}^{\infty }}{\pi }_0\left(\varsigma \right)y_0^{\prime }{\mathsf{\Lambda }}_2(0,\varsigma )y_0$ with $u^{*}\left(t\right)=G_2(t,{\xi }_t)y\left(t\right)=G_2(t,{\xi }_t,{\mathsf{\Lambda }}_2)y\left(t\right)=-H_2{(t,{\xi }_t,{\mathsf{\Lambda }}_2)}^{-1}{G}_4(t,{\xi }_t,{\mathsf{\Lambda }}_2)y\left(t\right).$ The proof has been completed. □

Remark 1. 

Note that Theorem 1 can be considered as an extension of [22] to infinite jumps and multiplicative noise and a discrete-time version of [23].

Remark 2. 

If the infinite horizon cost function is concerned, it is much more challenging owing to the requirement of stabilization limitation for the closed-loop system. As discussed in [26], the infinite horizon LQ Nash game has been considered.

4. Application to Special Case

In the previous section, the Nash game problem for discrete-time IMJSSs is solved. It should be noted that when $\gamma$ takes different values or $v\left(t\right)$ is regarded as exogenous disturbance. As a special case, we discuss the finite horizon robust $H_2/H_{\infty }$ control problem from a new perspective and further explore the relationship between Nash equilibrium points and finite horizon $H_2/H_{\infty }$ control with some remarks.

4.1. Finite Horizon $H_2/H_{\infty }$ Control

$v\left(t\right)$ in system (1) is seen as one of the players in the Nash game problem; as a matter of fact, it is more a case of considering $v\left(t\right)$ as an exogenous disturbance in most control systems. Let $\gamma >0$ be a prescribed disturbance attenuation. In consequence, the original Nash game problem is turned into finding a controller $u^{*}(·)\in l^2(N_T;{\mathcal{R}}^{n_u})$ such that

(i)

$\parallel L_T\parallel <\gamma$ for the closed system of Equation (1) with $x_0=0$;

(ii)

if $v^{*}(·)$ exists, $u^{*}(·)$ minimizes the output energy:

$$\begin{array}{c}\hfill J_2(y_0,{\xi }_0,u(·),v^{*}(·))=\sum _{t=0}^T{E[\parallel z\left(t\right)\parallel }^2],\end{array}$$

(31)

where $v^{*}(·)$ is the worst case disturbance and

$$\begin{array}{cc}\hfill v^{*}(·)& =arg\underset{\begin{array}{c}v\end{array}}{max}{J}_1(y_0,{\xi }_0,u^{*}(·),v(·))\hfill \\ \hfill & =arg\underset{\begin{array}{c}v\end{array}}{max}\sum _{t=0}^{T}E[{\gamma }^2{\parallel v\left(t\right)\parallel }^2-{\parallel z\left(t\right)\parallel }^2],\hfill \end{array}$$

(32)

in other words, the above problem is called the finite horizon $H_2/H_{\infty }$ control problem. We need to pay attention to the definition of perturbed operator $\parallel L_T\parallel$ given in [29,30].

As an applications of Nash game problem, it can be obtained from Theorem 1 that

$$\begin{array}{c}\hfill \parallel L_T\parallel \le \gamma ,\end{array}$$

(33)

and

$$\begin{array}{c}\hfill J_2(y_0,{\xi }_0,u^{*}(·),v^{*}(·))\le J_2(y_0,{\xi }_0,u(·),v^{*}(·)),\end{array}$$

(34)

where $u^{*}\left(t\right)$ and $v^{*}\left(t\right)$ are defined in Theorem 1.

Remark 3. 

In accordance with the definition of $H_2/H_{\infty }$ control problem, it is crucial to note that $\parallel L_T\parallel \le \gamma$ does not come down to $\parallel L_T\parallel <\gamma$. If we can confirm that $\parallel L_T\parallel \le \gamma$ can be replaced by $\parallel L_T\parallel <\gamma$, then the following result would be given naturally.

Theorem 2. 

For system (1), assume the following coupled generalized algebraic Riccati equations:

$$\begin{array}{ccc}& & \left\{\begin{array}{c}{\mathsf{\Lambda }}_1(t,\varsigma )={\displaystyle \sum _{k=0}^r}{[Q_k(t,\varsigma )+R_k(t,\varsigma )G_2(t,\varsigma ,{\mathsf{\Lambda }}_2)]}^{\prime }\epsilon {}_{\varsigma }(t,{\mathsf{\Lambda }}_1)\hfill \\ ·[Q_k(t,\varsigma )+R_k(t,\varsigma )G_2(t,\varsigma ,{\mathsf{\Lambda }}_2)]-L{(t,\varsigma )}^{\prime }L(t,\varsigma )\hfill \\ -G_2{(t,\varsigma ,{\mathsf{\Lambda }}_2)}^{\prime }{G}_2(t,\varsigma ,{\mathsf{\Lambda }}_2)-G_3{(t,\varsigma ,{\mathsf{\Lambda }}_1)}^{\prime }{H}_1{(t,\varsigma ,{\mathsf{\Lambda }}_1)}^{-1}{G}_3(t,\varsigma ,{\mathsf{\Lambda }}_1),\hfill \\ {\mathsf{\Lambda }}_1(T+1,\varsigma )=0,H_1(t,\varsigma ,{\mathsf{\Lambda }}_1)>0,(t,\varsigma )\in N_T\times \mathcal{D},\hfill \end{array}\right.\hfill \end{array}$$

(35)

$$\begin{array}{ccc}& & G_1(t,\varsigma ,{\mathsf{\Lambda }}_1)=-H_1{(t,\varsigma ,{\mathsf{\Lambda }}_1)}^{-1}{G}_3(t,\varsigma ,{\mathsf{\Lambda }}_1),\hfill \end{array}$$

(36)

$$\begin{array}{ccc}& & \left\{\begin{array}{c}{\mathsf{\Lambda }}_2(t,\varsigma )={\displaystyle \sum _{k=0}^r}{[Q_k(t,\varsigma )+U_k(t,\varsigma )G_1(t,\varsigma ,{\mathsf{\Lambda }}_1)]}^{\prime }\epsilon {}_{\varsigma }(t,{\mathsf{\Lambda }}_2)\hfill \\ ·[Q_k(t,\varsigma )+U_k(t,\varsigma )G_1(t,\varsigma ,{\mathsf{\Lambda }}_1)]+L{(t,\varsigma )}^{\prime }L(t,\varsigma )\hfill \\ -G_4{(t,\varsigma ,{\mathsf{\Lambda }}_2)}^{\prime }{H}_2{(t,\varsigma ,{\mathsf{\Lambda }}_2)}^{-1}{G}_4(t,\varsigma ,{\mathsf{\Lambda }}_2),\hfill \\ {\mathsf{\Lambda }}_2(T+1,\varsigma )=0,H_2(t,\varsigma ,{\mathsf{\Lambda }}_2)>0,(t,\varsigma )\in N_T\times \mathcal{D},\hfill \end{array}\right.\hfill \end{array}$$

(37)

$$\begin{array}{ccc}& & G_2(t,\varsigma ,{\mathsf{\Lambda }}_2)=-H_2{(t,\varsigma ,{\mathsf{\Lambda }}_2)}^{-1}{G}_4(t,\varsigma ,{\mathsf{\Lambda }}_2)\hfill \end{array}$$

(38)

admit solutions $({\mathsf{\Lambda }}_1(t,\varsigma ),G_1(t,\varsigma ,{\mathsf{\Lambda }}_1);{\mathsf{\Lambda }}_2(t,\varsigma ),G_2(t,\varsigma ,{\mathsf{\Lambda }}_2))$ with ${\mathsf{\Lambda }}_1(t,\varsigma )\le 0$, ${\mathsf{\Lambda }}_2(t,\varsigma )\ge 0$ for $(t,\varsigma )\in N_T\times \mathcal{D}$, then the finite horizon $H_2/H_{\infty }$ control optimal controller is $u^{*}\left(t\right)=G_2(t,\varsigma ,{\mathsf{\Lambda }}_2)y\left(t\right)$, $v^{*}\left(t\right)=G_1(t,\varsigma ,{\mathsf{\Lambda }}_1)y\left(t\right)$. Conversely, if the finite horizon $H_2/H_{\infty }$ control problem has the solution $u^{*}\left(t\right)=G_2(t,\varsigma ,{\mathsf{\Lambda }}_2)y\left(t\right)$, $v^{*}\left(t\right)=G_1(t,\varsigma ,{\mathsf{\Lambda }}_1)y\left(t\right)$, and $H_1(t,\varsigma ,{\mathsf{\Lambda }}_1)>0$, then the coupled generalized algebraic Riccati Equations (35)–(38) admit a group of solutions $({\mathsf{\Lambda }}_1(t,\varsigma ),G_1(t,\varsigma ,{\mathsf{\Lambda }}_1);{\mathsf{\Lambda }}_2(t,\varsigma ),G_2(t,\varsigma ,{\mathsf{\Lambda }}_2))$ with ${\mathsf{\Lambda }}_1(t,\varsigma )\le 0$, ${\mathsf{\Lambda }}_2(t,\varsigma )\ge 0$ for $(t,\varsigma )\in N_T\times \mathcal{D}$.

Proof. 

Sufficiency: It is obvious from the sufficiency part in Theorem 1 that we need only to explain $\parallel L_T\parallel <\gamma$. In fact, in the light of the definition of perturbed operator $\parallel L_T\parallel$, when $x_0=0$, note that the condition $H_1(t,\varsigma ,{\mathsf{\Lambda }}_1)>0$ of (35), Equation (17) means $J_1(y_0,{\xi }_0,u^{*}(·),v(·))=0$ iff $v\left(t\right)=v^{*}\left(t\right)=G_1(t,\varsigma ,{\mathsf{\Lambda }}_1)y\left(t\right)$. Putting $v^{*}\left(t\right)$ into Equation (14), the obtained closed-loop system with initial state $y_0=0$ leads to the state response $y\left(t\right)\equiv 0,t\in N_T$. One step further, $v\left(t\right)=v^{*}\left(t\right)=0$ naturally occurs. Then, the inescapable conclusion is that $J_1(0,{\xi }_0,u^{*}(·),v(·))>0$ iff $v\left(t\right)=v^{*}\left(t\right)\ne 0$, which stands for $\parallel L_T\parallel <\gamma$.

Necessity: Assume that the finite horizon $H_2/H_{\infty }$ control problem has the solution $u^{*}\left(t\right)=G_2(t,\varsigma ,{\mathsf{\Lambda }}_2)y\left(t\right)$, $v^{*}\left(t\right)=G_1(t,\varsigma ,{\mathsf{\Lambda }}_1)y\left(t\right)$, a combination of (31) and (32) causes $J_1(y_0,{\xi }_0,u^{*}(·),v^{*}(·))\le J_1(y_0,{\xi }_0,u^{*}(·),v(·)),J_2(y_0,{\xi }_0,u^{*}(·),v^{*}(·))\le J_2(y_0,{\xi }_0,u(·),v^{*}(·))$ for all $(u(·),v(·))\in l^2(N_T;{\mathcal{R}}^{n_u})\times l^2(N_T;{\mathcal{R}}^{n_v})$. By now, it can be deduced from the necessity of Theorem 1 that Equations (10)–(13) have solutions. Keep in mind that $H_1(t,\varsigma ,{\mathsf{\Lambda }}_1)>0$, hence, Equations (35)–(38) admit a group of solutions $({\mathsf{\Lambda }}_1(t,\varsigma ),G_1(t,\varsigma ,{\mathsf{\Lambda }}_1);{\mathsf{\Lambda }}_2(t,\varsigma ),G_2(t,\varsigma ,{\mathsf{\Lambda }}_2))$ with ${\mathsf{\Lambda }}_1(t,\varsigma )\le 0$, ${\mathsf{\Lambda }}_2(t,\varsigma )\ge 0$ for $(t,\varsigma )\in N_T\times \mathcal{D}$. The proof has been completed. □

Remark 4. 

Although the finite horizon $H_2/H_{\infty }$ control problem has been solved in [30], we yield a similar result from the new perspective of a Nash game.

Remark 5. 

By comparing Theorem 1 with Theorem 2, surely the existence of Nash equilibrium points and the solvability of finite horizon $H_2/H_{\infty }$ control are not equivalent for system (1), which differs from the continuous-time case as described in [26]. The main cause of the inequivalence lies in that the condition $H_1(t,\varsigma ,{\mathsf{\Lambda }}_1)>0$ of Equation (35) cannot meet the Nash equilibrium problem, namely, $\parallel L_T\parallel <\gamma$ is not equivalent to $\parallel L_T\parallel \le \gamma$.

4.2. Some Remarks on Nash Equilibrium Points

As a point of fact, only when $H_1(t,\varsigma ,{\mathsf{\Lambda }}_1)>0$ is the equivalence between Nash equilibrium points and finite horizon $H_2/H_{\infty }$ control valid. Accordingly, the relationship between the Nash game and $H_2/H_{\infty }$ control problem can be discussed in the following theorem.

Theorem 3. 

For system (1), under the condition of

$$H_1(t,\varsigma ,{\mathsf{\Lambda }}_1)={\gamma }^2{I}_{n_v}+\sum _{k=0}^r{U}_k{(t,\varsigma )}^{\prime }\epsilon {}_{\varsigma }(t,{\mathsf{\Lambda }}_1)U_k(t,\varsigma )>0,$$

the following statements are equivalent:

(i) 

There exists a linear memoryless Nash equilibrium points $(u^{*}(·),v^{*}(·))$ with $(u^{*}\left(t\right)=G_2(t,\varsigma ,{\mathsf{\Lambda }}_2)y\left(t\right),v^{*}\left(t\right)=G_1(t,\varsigma ,{\mathsf{\Lambda }}_1)y\left(t\right));$

(ii) 

The finite horizon $H_2/H_{\infty }$ control is solvable with $u^{*}\left(t\right)=G_2(t,\varsigma ,{\mathsf{\Lambda }}_2)y\left(t\right),v^{*}\left(t\right)=G_1(t,\varsigma ,{\mathsf{\Lambda }}_1)y\left(t\right);$

(iii) 

The coupled generalized algebraic Riccati Equations (35)–(38) have a group of solutions $({\mathsf{\Lambda }}_1(t,\varsigma ),G_1(t,\varsigma ,{\mathsf{\Lambda }}_1);{\mathsf{\Lambda }}_2(t,\varsigma ),G_2(t,\varsigma ,{\mathsf{\Lambda }}_2))$ with ${\mathsf{\Lambda }}_1(t,\varsigma )\le 0,{\mathsf{\Lambda }}_2(t,\varsigma )\ge 0$ for $(t,\varsigma )\in N_T\times \mathcal{D}$.

Proof. 

Theorem 3 can be demonstrated through Theorem 1 and Theorem 2. □

Remark 6. 

Keep in mind that when $H_1(t,\varsigma ,{\mathsf{\Lambda }}_1)>0$, $H_1{(t,\varsigma ,{\mathsf{\Lambda }}_1)}^{†}=H_1{(t,\varsigma ,{\mathsf{\Lambda }}_1)}^{-1}$, at this point, Equations (10)–(13) are consistent with Equations (35)–(38). In other words, for system (1), the existence of Nash equilibrium problem, the solvable of finite horizon $H_2/H_{\infty }$ control and the solvability of Equations (35)–(38) are equivalent. In addition, under the limitation of $H_1(t,\varsigma ,{\mathsf{\Lambda }}_1)>0$, a unified treatment for $H_2$, $H_{\infty }$ and $H_2/H_{\infty }$ control can be investigated such as [22].

5. Numerical Example

In this section, to solve the coupled generalized algebraic Riccati Equations (35)–(38), we provide an iterative algorithm, which can be summarized as follows:

(i)

When $t=T$, the terminal condition ${\mathsf{\Lambda }}_1(T+1,\varsigma )=0$ and ${\mathsf{\Lambda }}_2(T+1,\varsigma )=0$ can obtain $H_1(T,\varsigma ,{\mathsf{\Lambda }}_1)$ and $H_2(T,\varsigma ,{\mathsf{\Lambda }}_2)$;

(ii)

Working out (36) and (38), $G_1(T,\varsigma ,{\mathsf{\Lambda }}_1)$ and $G_2(T,\varsigma ,{\mathsf{\Lambda }}_2)$ can be computed;

(iii)

To compute (35) and (37), it is found that ${\mathsf{\Lambda }}_1(T,\varsigma )\le 0$ and ${\mathsf{\Lambda }}_2(T,\varsigma )\ge 0$;

(iv)

Repeating the above procedures, for $t=T-1,T-2,\cdots ,0,$ we can compute that ${\mathsf{\Lambda }}_1(t,\varsigma )\le 0$, ${\mathsf{\Lambda }}_2(t,\varsigma )\ge 0$, $G_1(t,\varsigma ,{\mathsf{\Lambda }}_1)$ and $G_2(t,\varsigma ,{\mathsf{\Lambda }}_2)$.

Remark 7. 

It should be noted that for the coupled generalized algebraic Riccati Equations (35)–(38), $H_1(t,\varsigma ,{\mathsf{\Lambda }}_1)>0$ and $H_2(t,\varsigma ,{\mathsf{\Lambda }}_2)>0$ are prerequisites for the effectiveness of the above iterative algorithm. Similarly, we can easily derive the algorithm used to solve the coupled generalized algebraic Riccati Equations (10)–(13).

Next, a numerical example will be presented to show the validity of the proposed method.

Example 1. 

Consider a three-stage one-dimensional discrete-time IMJSS with coefficients in Table 1.

Table 1. Coefficients of considered system.

Coefficients	$\mathit{t}=0$	$\mathit{t}=1$	$\mathit{t}=2$
$Q_0(t,\varsigma )$	$\frac{1}{2}$	1	$\frac{1}{3(\varsigma +1)}$
$Q_1(t,\varsigma )$	1	1	$\frac{1}{\varsigma +1}$
$R_0(t,\varsigma )$	1	$-\frac{1}{\varsigma +1}$	$\frac{\varsigma }{\varsigma +1}$
$R_1(t,\varsigma )$	1	$\frac{1}{\varsigma +1}$	$\frac{1}{2{(\varsigma +1)}^2}$
$U_0(t,\varsigma )$	$-\frac{1}{\varsigma +1}$	1	$\frac{1}{\varsigma +1}$
$U_1(t,\varsigma )$	$\frac{1}{\varsigma +1}$	1	1
$L(t,\varsigma )$	$\sqrt{\frac{\varsigma }{\varsigma +1}}$	1	1
$M(t,\varsigma )$	1	1	1

For ${\left\{{\xi }_t\right\}}_{t\in N_T}$, the element of $\mathbf{P}=[p(\varsigma ,\iota )]$ is given by $p(\varsigma ,\varsigma )=\frac{1}{2},p(\varsigma ,\varsigma +1)=\frac{1}{2}$ and $p_{\varsigma \iota }=0$, $\varsigma \in \mathcal{D}$, $\iota \in \mathcal{D}/\{\varsigma ,\varsigma +1\}$. Set $\gamma =\frac{\sqrt{2}}{2}$, the coupled generalized algebraic Riccati Equations (10)–(13) are solved by the above iterative algorithm with

$$\begin{array}{ccc}& & G_1(0,\varsigma ,{\mathsf{\Lambda }}_1)=-\frac{3(\varsigma +1)}{3{(\varsigma +1)}^2-4},G_2(0,\varsigma ,{\mathsf{\Lambda }}_2)=-0.53,\hfill \\ & & {\mathsf{\Lambda }}_1(0,\varsigma )=-0.36-\frac{3}{6{(\varsigma +1)}^2-8}-\frac{\varsigma }{\varsigma +1}\le 0,\hfill \\ & & {\mathsf{\Lambda }}_2(0,\varsigma )=0.55+\frac{110-33{(\varsigma +1)}^2}{27{(\varsigma +1)}^4-72{(\varsigma +1)}^2+48}+\frac{\varsigma }{\varsigma +1}\ge 0.\hfill \end{array}$$

Therefore, the Nash equilibrium points and optimal $H_2/H_{\infty }$ controller of the considered system will be obtained naturally.

6. Conclusions

This paper mainly explores a finite horizon LQ non-zero sum Nash game for discrete-time IMJSSs, whose system is governed by a countable Markov chain. The Nash equilibrium points for the considered system are solved by a countably infinite set of coupled generalized algebraic Riccati equations. Then, some special cases are given via a new perspective of the Nash game, which is the finite horizon mixed $H_2/H_{\infty }$ control with some remarks, and we summarized the relationship between Nash game and $H_2/H_{\infty }$ control problem. The contents in this paper are an extension and improvement of the previous works [22,23] in the MJSSs case. In fact, to solve the difficulties caused by the countable Markov chain, we introduce the infinite dimension Banach spaces, where the elements are countably infinite sequences of linear and bounded operators. Besides, to overcome the difficulty of solving a countably infinite set of coupled generalized algebraic Riccati equations, we present an iterative algorithm. In the future, the infinite horizon Nash game for IMJSSs can be considered.