Study on Master Slave Interaction Model Based on Stackelberg Game in Distributed Environment

Abstract

In view of the problems such as low efficiency, difficulty in resolving local conflicts and lack of practical application scenarios, existing in the interaction model of multi-agent systems in a distributed environment, a multi-master multi-slave interaction model was designed based on the Stackelberg game, which is applied to the interaction game problem between the controller and the participant in the command and control process. Through optimizing the Stackelberg game model and multi-attribute decision-making, the multi-master, multi-slave, multi-agent system of the Stackelberg game was designed, and the closed loop problem under the Stackelberg game is solved for dimension reduction and optimal function value. Finally, through the numerical derivation simulation and the training results of related system data, the high efficiency and strong robustness of the model were verified from multiple perspectives, and this model algorithm was proved to be true and highly efficient.

1. Introduction

With the rapid development of network information technology, intelligent command and control is widely applied in military, network, biological and other fields to realize intelligent control, and it has a high degree of sensitivity and accuracy in information task management and command control [1]. In this paper, a multi-master and multi-slave Stackelberg game model was proposed. Command and control behavior modeling is an important part of military analysis simulation [2]. In the current dynamically changing battlefield environment, whether command and control can meet the requirements of rapid information collection and real-time decision-making is a crucial link [3]. A command and control software system enables commanders, staff officers, and other participants to exchange the information about mission information (commands) and situational awareness status (reports) [4]. In order to effectively manage emergencies, crises and disasters, different organizations and their command and control and sensing systems must continuously cooperate, exchange and share data information [5]. As the technologies including network physical system, pervasive computing, embedded systems, mobile ad-hoc network, wireless sensor networks, cellular networks, wearable computing, cloud computing, big data analysis and intelligent agent have created an environment with various heterogeneous functions and protocols, adaptive control is required to realize useful links and modification of users. In military applications, completely different IoT devices need to be integrated into a common platform that must interoperate with dedicated military protocols, data structures and systems [6]. The effectiveness of military organizations mainly depends on the command and control structure. Sun, Y [7] et al. studied the C2 structure design of distributed military organizations, established a multi-objective optimization model to balance and minimize the relative load of decision-makers, and proposed a multi-objective genetic algorithm. Evans J et al. [8] mainly studied how to meet the requirements of the rapid decision-making cycle of tactical command and control. We discussed the security requirements of security packages and how named data networks meet these requirements.

The Stackelberg game is a classic example of the double-level optimization problem, which is often encountered in game theory and economics. These are complex problems with a hierarchical structure, in which one optimization task is nested in another. In recent years, because of the advantages of multi-master multi-slave Stackelberg game, it has been studied deeply in many fields. Emerging mobile cloud computing technologies provide great potentials for mobile terminals to support highly complex applications. Wang Y et al. [9], based on the two-stage Stackelberg game model of multi-master and multi-slave, adopted an acceptable QoE to solve the resource management problem of MCC networks which maximized the utility function of the network. Modern communication networks are becoming highly virtualized, with a two-layer hierarchical structure. Zheng Z et al. [10] combined the optimization method with game theory and proposed an alternate direction multiplier based on multi-master and multi-slave games. This method excites multiple agents to execute the tasks of the controller, so as to meet the corresponding goals of the controller and the agent. Sinha A et al. [11] studied a special case of a multi-cycle, multi-master and multi-slave Stackelberg competition model with nonlinear cost and demand functions and discrete production variables. In literature [12], an asymmetric explicit model was proposed. The framework of Stackelberg game was used to capture the self-interested and hierarchical competitive properties of nodes, which proved the existence and uniqueness of Stackelberg equilibrium. Akbarid et al. proposed a new coordinated structure for a power generation system whereby power generation capacity was deregulated by transmission departments and decentralized capacity expansion planning of the transmission network. In this paper, a two-level Stackelberg game problem with multiple leaders and followers was solved by the diagonalization method [13]. Chen F et al. [14] studied the multi-master and multi-slave coordination problem based on neighborhood network topology. Scheme [15] mainly studied the problem of anti-interference transmission in a UAV (unmanned aerial vehicle) communication network, and a Bayes Stackelberg game method was proposed to describe the competitive relationship between a UAV (user) and jammer, namely, the jammer acts as a leader, and the user acts as a follower of the proposed game, and the jammer and user choose the optimal power control strategy according to their own utility function. Yuan Y et al. [16] studied elastic control under denial of service attack initiated by intelligent attackers. The elastic control system was modeled as a multi-stage hierarchical game with decision levels at the network and physical levels, respectively. Specifically, the interaction between different security agents in the network layer was modeled as a static infinite Stackelberg game.

Aiming at the above two game roles, the interaction model was constructed based on the game relationship, and the optimization model based on the Stackelberg game and the multi-attribute decisions discuss how the accuser and the participant interacted in the distributed collaborative environment. The current multi-agent model of leader-follower architecture still has a closed loop solution that cannot be explained. In fact, many studies have performed a preliminary investigation of it, which provided some insight for this paper. Thai C N et al. [17] studied the performance of closed-loop identification technology and chose an optimal closed-loop identification solution. The flexible transmission system used two recursive closed-loop identification methods and proposed an optimal closed-loop control scheme for the flexible transmission system. In reference [18], the dynamic mean-variance combinatorial optimization problem with deterministic coefficients was studied, and an inherent property of the closed-loop equilibrium solution was obtained for the first time, which proved that this optimization problem really had a unique equilibrium solution. Due to the limitations of the above studies, this paper will demonstrate the existence of the closed-loop solution from the perspective of mathematical derivation and solve the closed-loop solution, prove the reliability and rationality of the closed-loop solution with numerical simulation, and verify the model algorithm through the results of data training.

Based on the command and control model of multi-agent system under the distributed cooperative engagement, the game theories within the multi-agent system were discussed and studied, a multi-master and multi-slave Stackelberg game model was proposed [19]. It took two types of major decisions in the process of operation command as the research objects, one is accuser, namely leader, who is responsible for the command and control of the combat process; The other is the participant, namely, the follower, who is responsible for executing the decision scheme generated by the accuser. Aiming at the above two kinds of game roles, the interaction model was constructed based on the game relationship, and the optimization model based on Stackelberg game and the multi-attribute decisions discuss how the accuser and the participant interacted in the distributed collaborative environment.

Considering the problem presented by the limited individual capability of a single agent, the ability to deal with complex tasks can be enhanced through cooperation among individuals [20], so the multi-agent system has more advantages in dealing with complex tasks. First of all, the multi-agent system has a better battlefield situational awareness than the single-agent system. It can obtain global situational information and has the ability of parallel perception and parallel charge decision-making [21]. Secondly, the ability of a cooperative agent system is stronger than that of a single agent, so a multi-agent system has excellent expansibility and robustness. Finally, based on reasonable battlefield resource deployment and cooperative control between combat units, the multi-agent system with low resource loss was used to replace a single complex system with high resource loss, so as to achieve higher benefits.

2. Relevant Work

2.1. Multi-Agent Interaction Model for Multi-Master and Multi-Slave Stackelberg Game

Due to the low global coupling degree of distributed environments, the strategies are diversified, behavior planning is more complex, and solution space index level is high [22]. As the number of external access agents increases, the behavior of each agent will have a huge impact on the maximization of global benefits [23]. From the perspective of economics, both the accuser and the participant of the task tend to pursue the maximization of their own benefits in the process. Based on this principle, a balanced interaction model of multiple agents must be studied to make each agent meet its own goals on the premise of satisfying the maximum global benefits.

In this section, an agent interaction model based on multi-master and multi-slave Stackelberg game was proposed. In the process that the agent system dealt with complex tasks, the agent was divided into two roles: leader and follower. Based on these two roles, the game interaction model in the command and control system was studied in depth, and the validity and robustness of the game interaction model were demonstrated through numerical calculation and simulation [24]. In addition, the existence and uniqueness of the Stackelberg equilibrium solution in an interactive game were also deduced, and the closed-loop expression of the equilibrium solution was provided.

2.1.1. Optimization Model of Stackelberg Game

Game theory, as a commonly used model method, is used to deal with the issues of competition with multiple participants [25]. Game theory mainly studies the theory that various the decision-makers influence the behavior of each other under competition as well as decision-making equalization, and it is the mathematical theories and methods of the decision-making process to maximize their own profits.

Generally, according to whether there is a binding agreement between game participants, the game mode can be divided into a cooperative game and non-cooperative game. The two parties of a cooperative game [26] have a consistent income direction and a constraint agreement is reached. Otherwise, it is called a non-cooperative game. According to whether the sum of the cost functions of the two sides of the game has a loss, it can be divided into a zero-sum game and a non-zero-sum game. The game can be classified as either a complete information game or incomplete information game according to the players’ mastery of global situation information. According to the different status of each participating member in the Game process, the Game is divided into Nash Game, master slave Game (Stackelberg Game), etc. [27].

The Stackelberg game is a dynamic game, which is used where there is a level of decision behavior between two types of game players [28], one being the leader, and the other a follower. After leading the action, the followers make their own decision plans according to the action plan of the leader, so as to ensure the maximization of the decision benefits of the leader. However, in this paper, the original model was improved, and based on the relevant concept of distributed collaboration, a multi-master and multi-slave game model was established, wherein leader was redefined as the accuser, and the followers were defined as the participants so as to make the accuser adjust the follow-up action plan according to the participants’ decision-making scheme and maintain the benefits of participants to the greatest degree, thus obtaining the maximization of global income. Therefore, the optimal decision-making and response decision-making corresponding to participants constituted the equilibrium of the game [29].

Two types of game participants were defined. The accuser was $L_k\in U$, and its performance index function was $J_L\left(\mu ,\omega \right)$. The participant was $F_k\in U$, and its performance index function was $J_F\left(\mu ,\omega \right)$. The purpose of both parties was to minimize $J\left(\mu ,\omega \right)$, so as to maximize the benefits of both parties and maximize the global benefits. For any decision $\mu \prime \in Ud$ made by the accuser, the participants had a unique decision $\omega \prime \in W$ to minimize its performance index function $J_F\left(\mu \prime ,\omega \right)$. The mapping relationship between the two parties’ decision choices can be expressed as follows:

$$U\to W\Rightarrow J_L\left(\mu ,T_{\mu }\right)\le J_L\left(\mu ,\omega \right)\hspace{1em}\mu \in U;\omega \in W$$

(1)

At this time, considering the performance index of the accuser $J_L\left(\mu ,T_{\mu }\right)$, it was assumed that there was a unique solution $\mu *\times U$, i.e., $J_L\left(\mu *,T_{\mu }*\right)\le J_L\left(\mu *,T_{\mu }\right)\forall \mu \in U$. At this time, $\left(\mu *,\omega *\right)=\left(\mu *,T_{\mu }*\right)\in U\times W$ was the unique Stackelberg solution.

2.1.2. Multi-Attribute Decision-Making

The multi-attribute decision-making problem is a multi-criterion decision problem. Another multi-criterion decision problem is the multi-objective optimization or multi-objective decision-making problem [30]. The criterion of MADM is often invisible and indirect. Even in many cases, it cannot be accurately described in a quantitative way, while the constraint of MODM must be direct and accurate. MADM’s constraint on the target is often hidden in the attribute, while MODM does not have this feature due to the accuracy of the constraint. In addition, MADM alternatives are often limited, while MODMs can have an infinite number of alternatives [31].

$$optimize\left[u_1\left(s\right),u_2\left(s\right),\dots ,u_m\left(s\right)\right]\hspace{1em}Subject To:s\in S$$

(2)

where $S$ represents the set of alternatives $s\in S$ represents one of the alternatives in $S$, and $u_1,u_2,\dots ,u_m$ represents the attributes of the multi-attribute decision problem. Let $S=\left\{s_1,s_2,\dots ,s_n\right\}\left(n\in N^{+}\right)$ be the alternative scheme set, $S=\left\{s_1,s_2,\dots ,s_n\right\}\left(n\in N^{+}\right)$ the attribute set, and set $c_{ij}=u_j\left(s_i\right)\left(i\in \left[1,n\right];j\in \left[1,m\right]; i,j\in N^{+}\right)$, where $c_{ij}$ is the attribute value of $s_i$ for $u_j$. Then, a multi-attribute decision matrix can be constructed, i.e.,$C={\left[c_{ij}\right]}_{n\times m}$. $S,U,C$ are shown in the following

Table 1. Decision matrix.

	$u_1$	$u_2$	…	$u_m$
$s_1$	$c_{11}$	$c_{12}$	…	$c_{1m}$
$s_2$	$c_{21}$	$c_{22}$	…	$c_{2m}$
…	…	…	…	…
$s_n$	$c_{n1}$	$c_{n2}$	…	$c_{nm}$

:

At this time, the property value $c_{ij}$ is an estimate. The solving process of the multi-attribute decision-making problem is as follows. First of all, the decision matrix was listed. Ssecondly, according to whether the weight was known, the weight determination method was selected. Thirdly, based on the attribute values of decision matrix, the aggregation operator of the property matrix was determined, and according to the solving target and the form of decision matrix, the appropriate multi-attribute decision-making method was chosen to calculate. The calculation results were carried out with weight distribution and gathering, and eventually the score of each scheme was obtained, and the decisions were made according to the score situation [32].

In this paper, every member of the multi-master and multi-slave Stackelberg game model used multi-attribute decision making as the basic method to generate the decision-making scheme, optimized the weights of the method according to the roles of the decision-makers, and generated the optimal decision scheme according to the final score.

3. Multi-Master and Multi-Slave Stackelberg Game Model in a Distributed Environment

Game theory is widely used in the field of interactive multi-agent systems. The Stackelberg game [33] is a classic game model, and the strategy of various participants in the game is usually made on the basis of the maximization of self-interest, and there is cooperation or non-cooperation between each other. When all participants achieve maximum interests and global interests tend to be steady, this state is called equilibrium state.

The Stackelberg game is a game problem between upper and lower levels, in which each participant belongs to the relationship between upper and lower levels, namely leader and followers [34]. In the game model, the leader generates a decision scheme according to the state prediction of the task objective and the followers, and the followers respond according to the decision scheme of the leader. In the game process, information between leader and followers is not fully shared, that is, the leader’s information is partially shared with followers, but the followers’ information is fully shared with the leader. Considering such a design, the leader’s decision-making scheme constraint is based on the maximization of followers’ revenue, so in Stackelberg game [35], the leader doesn’t need to design a response function.

The main problem of the current multi-master and multi-slave Stackelberg game model is the difficulty in solving the closed loop solution, so the open-loop solution of Stackelberg game is usually the mainstream research object [36]. This section aims to obtain Stackelberg game equilibrium through reasoning, namely, the closed-loop solution of the Stackelberg game. Therefore, in this paper, the retrieval process of the open-loop solution won’t be demonstrated in detail. The closed-loop solution of multi-master and multi-slave Stackelberg game model under the accuser participants has the nonlinear characteristics, there is no explicit solution, so under certain conditions, this paper will use the regularity of a positive semidefinite matrix to give a closed-loop solution to the Stackelberg game, obtain the optimal decision through the accuser, optimize the participants’ decision-making, obtain the optimal decision scheme of participants, and eventually maximize the overall revenue.

3.1. A Closed-Loop Solution of Multi-Master and Multi-Slave Stackelberg Game Model

Under the distributed environment with a certain complexity, the optimal regularity of positive semidefinite quadratic performance index was used to give a closed-loop solution of multi-master and multi-slave Stackelberg game. In this section, a regular Riccati equation was introduced to solve positive semi-definite multi-master and multi-slave Stackelberg game problems, and the main contribution is that it gave a closed loop solution to the Stackelberg game. The general process is as follows: First of all, the optimal decision $\left(\mu ,\omega \right)$ was obtained in the decision optimization of the accuser. When the decision resulted in the maximum value $\omega$ of the accuser, the value of $\mu$ was the minimum. At this time, the weighted matrix of the control performance index was positive semi-definite, so there were arbitrary terms in the performance index function. Secondly, the participants’ decisions were optimized by continuously using arbitrary terms in $\mu$, and finally, combined with the solution of the regular Riccati equation, and according to the decision-making optimization process of the accuser and participants, the closed-loop solution of multi-master and multi-slave Stackelberg game problem.

3.2. Problem Description

The linear discrete time system is defined as follows:

$$x_{k+1}=A{x}_k+B_1{\mu }_k+B_2{\omega }_k$$

(3)

$$z_k=\left[\begin{array}{c}I{x}_k\\ K{\mu }_k\end{array}\right],$$

(4)

where $x_k\in R^n$ is the state variable of game participants and there is an initial value $x_0$ and it can be determined; ${\omega }_k\in R^l$ is the interference that can be quantified; $z_k$ is the output item; ${\mu }_k$ is the input item; and the interference attenuation factor is defined as $\gamma$, whose value range is $\gamma \ge 0$, then the following performance index function can be obtained:

$$J_L=x\prime \left(N+1\right)P\left(N+1\right)x\left(N+1\right)-x\prime \left(0\right){\displaystyle {\prod }_0^{-1}x}\left(0\right)+{\displaystyle \sum _{k=0}^N\left[{\Vert z_k\Vert }^2-{\gamma }^2{\Vert {\omega }_k\Vert }^2\right]},$$

(5)

$$J_F=x\prime \left(N+1\right)\tilde{P}\left(N+1\right)x\left(N+1\right)+{\displaystyle \sum _{k=0}^N{\left[\Vert z_k\Vert \right]}^2},$$

(6)

where $\gamma$ is the interference attenuation factor defined above and $\gamma \ge 0$; ${\displaystyle {\prod }_0^{-1}=}{\gamma }^2{\pi }_0{}^{-1}\left[111\right]$, where ${\pi }_0$ is the variance matrix of the initial state $x\left(0\right)$.

In this section, the accuser’s optimization and the participants’ decision-making optimization were carried out simultaneously. During the decision-making optimization of the accuser, under any initial environmental conditions $\left(k_0,x_0\right)$, $\left(\omega *,\mu *\right)$ meets $J_L^{*}\left(k_0,x_0;\omega *,\mu *\right)={\max }_{\omega }{\min }_{\mu }{J}_L\left(k_0,x_0;\omega ,\mu \right)\le 0$. During the decision-making optimization of the participants, for any initial environmental condition $\left(k_0,x_0\right)$, the optimal value $\mu *$ was sought to minimize $J$, that is, $J_F^{*}\left(k_0,x_0;\mu *\right)={\min }_{\mu }{J}_F\left(k_0,x_0;\mu \right)$.

3.3. Closed-Loop Solution of Semidefinite Control in Stackelberg Game

This section will describe the closed-loop solution process in the optimization of the multi-master multi-slave Stackelberg game, and the optimization process of this problem is mainly divided into two parts. The first part is the accuser’s decision-making optimization. By adjusting the value of $\mu$, the performance index function $J_L$ is minimized, and at the same time, by adjusting $\omega$, the performance index function is $J_L$ maximized. According to the positive definiteness of the weighted matrix, $\mu$ value contains arbitrary terms. In order to ensure the only decision-making of the accuser, arbitrary terms are used as new decision-making to be solved, and it is obtained from the participants’ decision-making optimization, so as to obtain the accuser’s optimal decision.

• Decision optimization of accuser

As can be seen from the above, the accuser’s decision-making optimization is carried out, that is $\max \min J_L\le 0 \mathrm{s}.\mathrm{t}$, where:

$$\begin{array}{ll}{J}_L& =x\prime \left(N+1\right)P\left(N+1\right)x\left(N+1\right)-x\prime \left(0\right){\displaystyle {\prod }_0^{-1}x}\left(0\right)+\\ & {\displaystyle \sum _{k=0}^N\left[\mu {\prime }_{k}R{\mu }_k+x{\prime }_{k}R{x}_k-{\gamma }^2\omega {\prime }_k{\omega }_k\right]}\end{array}$$

(7)

where $K\prime K=R\ge 0, I\prime I=Q\ge 0$. Firstly, the existence of the accuser’s optimal decision-making scheme is proved:

$$x_{k+1}=A{x}_k+B{\mu }_k+C{\omega }_k$$

(8)

$$p_{k-1}=A\prime p_k+Q{x}_k,$$

(9)

$$C\prime p_k-{\gamma }^2{\omega }_k=0,$$

(10)

$$R{\mu }_k+B\prime p_k=0$$

(11)

If the above equation is satisfied, it can be proven that the accuser has an optimal decision, that is, $\max \min J_L$ has a definite value, and the optimal solution satisfies Equations (10) and (11). The proof process can be seen in the literature [37].

According to Equations (3) and (9), it can be known that there is a non-secondary linear relationship between variable $x_k$ and $p_{k-1}$, so it is defined as follows:

$${\eta }_{k-1}=p_{k-1}-P_k{x}_k$$

(12)

where ${\eta }_{k-1}$ and $P_k$ will be defined below.

Substitute Equation (12) into Equation (11), then Equation (11) can be written as follows:

$$\begin{array}{ll}& B\prime \left[{\eta }_k+P_{k+1}{x}_{k+1}\right]+R{\mu }_k=0\\ \Rightarrow & \left[R+B\prime P_{k+1}B\right]{\mu }_k+B\prime P_{k+1}A{x}_k+B\prime P_{k+1}C{\omega }_k+B\prime {\eta }_k\end{array}$$

(13)

Substitute Equation (12) into Equation (10), and Equation (10) can be written as follows:

$$\begin{array}{ll}& C\prime \left[{\eta }_k+P_{k+1}{x}_{k+1}\right]-{\gamma }^2{\omega }_k=0\\ \Rightarrow & Z_k{\omega }_k+C\prime P_{k+1}A{x}_k+C\prime P_{k+1}B{\mu }_k+C\prime {\eta }_k\end{array}$$

(14)

where $Z_{k+1}=-{\gamma }^{2}I+C\prime P_{k+1}C$. Since there is an optimal solution for the accuser’s decision making, it can be seen that the weighting matrix $Z_{k+1}$ is determined, that is $Z_{k+1}\le 0$, then Equation (14) can be written as follows:

$${\omega }_k=-Z_{k+1}{}^{-1}\left[C\prime P_{k+1}A{x}_k+C\prime P_{k+1}B{\mu }_k+C\prime {\eta }_k\right]$$

(15)

Substitute Equation (15) into Equation (13), the following formula can be obtained:

$$\begin{array}{ll}B\prime & \left[{\eta }_k+P_{k+1}{x}_{k+1}\right]+R{\mu }_k=0\\ \Rightarrow & B\prime \left[{\eta }_k+P_{k+1}{x}_{k+1}\right]+R{\mu }_k\\ & =\left[B\prime P_{k+1}B+R\right]{\mu }_k+B\prime P_{k+1}A{x}_k+B\prime {\eta }_k-B\prime P_{k+1}\times C{Z}_k{}^{-1}\left[C\prime P_{k+1}A{x}_k+C\prime P_{k+1}B{\mu }_k+C\prime {\eta }_k\right]\\ & =\left[R+B\prime P_{k+1}B-B\prime P_{k+1}C{Z}_k^{-1}C\prime P_{k+1}A\right]x_k+\left[B\prime +B\prime P_{k+1}C{Z}_k^{-1}C\prime \right]{\eta }_k\\ & ={\psi }_{k+1}{\mu }_k+{\varsigma }_k{x}_k+\left[B\prime +B\prime P_{k+1}C{Z}_k^{-1}C\prime \right]{\eta }_k\end{array}$$

(16)

where in ${\psi }_{k+1}=R+B\prime {\overline{P}}_{k+1}B$, ${\varsigma }_{k+1}=B\prime {\overline{P}}_{k+1}A$, and ${\overline{P}}_{k+1}=P_{k+1}-Z_{k+1}^{-1}C{P}_{k+1}$. Assuming that when the weighted matrix is semi-positive definite, $\max \min J_L$ may have more than one optimal solution. At this time, it is necessary to introduce the regular Riccati equation to make $P_k$ satisfy the Riccati equation, which can be expressed as follows:

$$\begin{array}{ll}{P}_k=& Q+A\prime P_{k+1}A-A{P}_{k+1}B{\psi }_{k+1}^{*}B\prime {\overline{P}}_{k+1}A-A\prime P_{k+1}B{Z}_{k+1}^{-1}B\prime P_{k+1}A+\\ & A\prime P_{k+1}B\times Z_{k+1}^{-1}B\prime P_{k+1}B{\psi }_{k+1}^{*}B\prime {\overline{P}}_{k+1}A\end{array}$$

(17)

At this time, ${\eta }_k=0$ and $P_{N+1}=P_N{x}_{N+1}=H$, ${\psi }_{k+1}^{*}$ is the pseudo-inverse matrix of ${\psi }_{k+1}$. Then it can be obtained ${\eta }_N=0$, so Equation (12) is true at time $N$. Assuming Equation (12) is true at time k, then ${\eta }_k=p_k-P_{k+1}{x}_{k+1}$, when the range of the two accords with $Range\left[{\varsigma }_{k+1}\right]\subseteq Range\left[{\psi }_{k+1}\right]$, the following formula can be obtained from Equation (16):

$${\mu }_k=-{\psi }_{k+1}^{*}{\varsigma }_{k+1}{x}_k-{\psi }_{k+1}^{*}\left[B\prime -B\prime P_{k+1}C{Z}_{k+1}^{-1}\times C\prime \right]{\eta }_k+\left[I-{\psi }_{k+1}^{*}{\varsigma }_{k+1}\right]\phi$$

(18)

where $\phi$ is an arbitrary term. By substituting Equation (18) into Equation (15), the following formula can be obtained:

$$\begin{array}{ll}{\omega }_k=& -Z_{k+1}^{-1}C\prime P_{k+1}\left[A-B{\psi }_{k+1}^{*}{\varsigma }_{k+1}\right]x_k+Z_{k+1}^{-1}C\prime \\ & \left[P_{k+1}B{\psi }_{k+1}^{*}\left[B\prime -B\prime P_{k+1}C\times Z_{k+1}^{-1}C\prime \right]-I\right]\\ & {\eta }_k- Z_{k+1}^{-1}C\prime P_{k+1}B\times \left[I-{\psi }_{k+1}^{*}{\psi }_{k+1}\right]\phi \end{array}$$

(19)

By substituting Equations (18), (19), and (12) into (9), the following equation can be obtained:

$$\begin{array}{ll}{p}_{k-1}& =A\prime p_k+Q{x}_k\\ & =A\prime \left[{\eta }_k+P_{k+1}{x}_{k+1}\right]+Q{x}_k\\ & =A\prime \left[{\eta }_k+P_{k+1}\left[A{x}_k+B{\mu }_k+C{\omega }_k\right]\right]+Q{x}_k\\ & =P_k{x}_k+{\eta }_{k-1}\end{array}$$

(20)

As $P_k$ meets Equation (17), and ${\varsigma }_{k+1}\left[I-{\psi }_{k+1}\ast {\psi }_{k+1}\right]=0$, the following formula can be obtained:

$$\begin{array}{ll}{\eta }_{k-1}=& [A\prime -A\prime P_{k+1}B{\psi }_{k+1}^{*}\left[B\prime -B\prime P_{k+1}C{Z}_{k+1}^{-1}C\prime \right]\\ & +A\prime P_{k+1}C{Z}_{k+1}^{-1}C\prime P_{k+1}B{\psi }_{k+1}^{*}\left[B\prime -B\prime \times P_{k+1}C{Z}_{k+1}^{-1}C\prime \right]\\ & -A\prime P_{k+1}C{Z}_{k+1}^{-1}C\prime ]{\eta }_k\end{array}$$

(21)

It can be concluded from the above that ${\eta }_k=0,k=0,1,\dots ,N$. As can be seen from the above, when (1) $P\left(0\right)-{\displaystyle \prod _0^{-1}\le 0}$; (2) $Z_{k+1}\le 0$; (3) ${\psi }_{k+1}\ge 0$, the following formulas can be obtained:

$${\mu }_k*=-{\psi }_{k+1}{\varsigma }_{k+1}{x}_k+\left[I-{\psi }_{k+1}^{*}{\psi }_{k+1}\right]{\varphi }_k$$

(22)

$${\omega }_k*=-Z_{k+1}^{-1}{M}_{k+1}{x}_k-Z_{k+1}^{-1}C\prime P_{k+1}B\left[I-{\psi }_{k+1}^{*}{\psi }_{k+1}\right]{\varphi }_k$$

(23)

where ${\varphi }_k$ is arbitrary vector with appropriate dimension, ${\varphi }_k$. Then the optimal decision indicator function is as follows:

$$J_L^{*}=x_0^{\prime }\left[P_0-{\pi }_0^{-1}\right]x_0$$

(24)

The specific proof process can be seen in Appendix A.

• Accuser’s strategy optimization

It was found that there is an arbitrary term in the optimal decision scheme. In order to further optimize the arbitrary item in the optimal decision scheme, we first carry out matrix transformation to transform the arbitrary item into the participants to be solved. An elementary row transformation matrix $T_0\left(k\right)$ is introduced to make:

$$T_k^0\left[I-{\psi }^{\ast }{}_{k+1}{\psi }_{k+1}\right]=\left[\begin{array}{l}0\\ R{T}_k^0\end{array}\right]$$

(25)

$T_k^0\left[I-{\psi }^{\ast }{}_{k+1}{\psi }_{k+1}\right]{\phi }_k=[\begin{array}{l}0\\ {\mu }_k^1\end{array}]$ is defined, where ${\mu }^1{}_k={\psi }_{T^0{}_k}{\phi }_k$, and the following formula can be obtained:

$$\left[I-{\psi }^{\ast }{}_{k+1}{\psi }_{k+1}\right]{\phi }_k={\left(T_0^{-1}\right)}_k\left[\begin{array}{l}0\\ {\mu }_k^1\end{array}\right]=Z^0{}_{k+1}{\mu }_k^1$$

(26)

Based on the above analysis, $\mu \left(k\right)$ and $\omega \left(k\right)$ are rewritten as follows:

$${\mu }_k=-{\psi }_{k+1}^{*}{\tau }_{k+1}{x}_k+Z_{k+1}^0{\mu }_k^1$$

(27)

$$\omega \left(k\right)=-Z^{-1}{}_{k+1}{M}_{k+1}{x}_k-Z^{-1}{}_{k+1}{C}^{\prime }{P}_{k+1}B{Z}_{k+1}^0{\mu }_k^1$$

(28)

By substituting the above formula into the definition of $x_{k+1}$, the following equation can be obtained:

$$x_{k+1}={\overline{A}}_{k+1}{x}_k+{\overline{B}}_{k+1}{u}^1{}_k$$

(29)

where ${\overline{A}}_{k+1}=A-B{\psi }_{k+1}^{*}{\tau }_k-C{Z}_{k+1}^{-1}{M}_{k+1}$, ${\overline{B}}_{{}_{k+1}}=B{Z}^0{}_{k+1}-C{Z}^{-1}{}_{k+1}{C}^{\prime }{P}_{k+1}B{Z}^0{}_{k+1}$. By substituting ${\mu }_k$ and ${\omega }_k$ into $J_F$, the following equation can be obtained:

$$\begin{array}{lll}{{\displaystyle J}}_F& =& {x^{\prime }}_{N+1}{\overline{P}}_{N+1}{x}_{N+1}+{\displaystyle \sum _{k=0}^N\left[{{\mu }^{\prime }}_{k}R{\mu }_k+{x^{\prime }}_{k}Q{x}_k\right]}\\ & =& {x^{\prime }}_{N+1}{\overline{P}}_{N+1}{x}_{N+1}+{\displaystyle \sum _{k=0}^N[{x^{\prime }}_k{K}^1{{}^{\prime }}_{k+1}R{K}^1{}_{k+1}{x}_k}-2{x^{\prime }}_k{K}^1{{}^{\prime }}_{k+1}R{Z}^0{}_{k+1}{\mu }^1{}_k\\ & & +{\mu }^1{{}^{\prime }}_k{Z}^0{{}^{\prime }}_{k+1}R{Z}^0{}_{k+1}{\mu }^1{}_k]+{x^{\prime }}_{k}Q{x}_k]\end{array}$$

(30)

At this time, the decision-making scheme optimization problem of the participants is solved, namely:

$$\underset{u_1}{min}{{\displaystyle J}}_{2}s.t.$$

(31)

The participants’ decision scheme optimization is a standard linear quadratic optimization problem. First of all, the necessary conditions for a solution are given:

$$\begin{array}{ll}& {\theta }_{k-1}={\overline{A^{\prime }}}_{k+1}{\theta }_k+\left[Q+K^1{{}^{\prime }}_{k+1}R{K}_{k+1}^1\right]x_k-K_{k+1}^1{}^{\prime }R{Z}^0{}_{k+1}{\mu }_k^1\\ \Rightarrow & 0=-Z_{k+1}^0{}^{\prime }R{K}_{k+1}^0{}^{\prime }{x}_k+Z_{k+1}^0{}^{\prime }R{Z}_{k+1}^0{\mu }_k^1+{\overline{B^{\prime }}}_{k+1}{\theta }_k\end{array}$$

(32)

If the above equation is true, it indicates that there is an optimal solution to the participants’ decision-making scheme optimization problem. The specific proof process is as follows:

Firstly, the inhomogeneous relation between $x_k$ and ${\theta }_{k-1}$ is defined as follows:

$${\varsigma }_k={\theta }_k-{\tilde{P}}_{k+1}{x}_{k+1},$$

(33)

where ${\varsigma }_{k-1}$ and ${\overline{P}}_k$ are defined in the following lemma. By combining the definition of $J_F$ and ${\theta }_{k-1}$, the following equilibrium equation can be obtained:

$$\begin{array}{ll}0& =-Z_k^1{}^{\prime }{R}^{\prime }{K}_{k+1}^1{x}_k+Z_{k+1}^1{}^{\prime }R{Z}_{k+1}^0{\mu }_k^1+{\overline{B^{\prime }}}_{k+1}\left[{\varsigma }_k+{\tilde{P}}_{k+1}{x}_{k+1}\right]\\ & =-Z_{k+1}^0{}^{\prime }{R}^{\prime }{K}_{k+1}^1{x}_k+Z_{k+1}^0{}^{\prime }R{Z}_{k+1}^0{\mu }_k^1+{\overline{B^{\prime }}}_{k+1}\left[{\varsigma }_k+{\tilde{P}}_{k+1}\left[{\overline{A}}_{k+1}{x}_k+{\overline{B}}_{k+1}{\mu }_k^1\right]\right]\\ & =\left[-Z_{k+1}^0{}^{\prime }{R}^{\prime }{K}_{k+1}^1+{\overline{B^{\prime }}}_{k+1}{\tilde{P}}_{k+1}{\overline{A}}_{k+1}\right]x_k+\left[Z_{k+1}^0{}^{\prime }R{Z}_{k+1}^0+{\overline{B^{\prime }}}_{k+1}{\tilde{P}}_{k+1}{\overline{B}}_{k+1}\right]{\mu }_k^1\\ & ={\psi }^{{}_0}{}_{k+1}{\mu }_k^1+{\Gamma }_{k+1}{x}_k+{{\overline{B}}^{\prime }}_{k+1}{\mu }_k^1\end{array}$$

(34)

where

$$\begin{array}{l}{\phi }^{{}_0}{}_{k+1}=Z_{k+1}^0{}^{\prime }R{Z}_{k+1}^0+{\overline{B^{\prime }}}_{k+1}{\tilde{P}}_{k+1}{\overline{B}}_{k+1}\\ {\Gamma }_{k+1}=-Z_{k+1}^0{}^{\prime }{R}^{\prime }{K}_{k+1}^1+{\overline{B^{\prime }}}_{k+1}{\tilde{P}}_{k+1}{\overline{A}}_{k+1}\end{array}$$

(35)

In order to obtain the explicit solution of the optimal controller ${\mu }_k^1$, we consider the following situation, that is, the following hypothesis is obtained:

$$Range\left[{\Gamma }_{k+1}\right]\subseteq Range\left[{\psi }^{{}_0}{}_{k+1}\right]$$

(36)

When the control matrix in the performance index of the decision-making scheme of the participants is a semi-positive definite matrix, $\underset{u_1}{min}{{\displaystyle J}}_2$ has no unique solution. Here, the regular Riccati equation is introduced. Under the above regular hypothesis, this paper solves the solution of the regular Riccati equation and verifies the homogeneous relationship between the state variable $x\left(k\right)$ and the adjoint state variable $\theta \left(k-1\right)$. Under the above hypothesis, $\theta \left(k-1\right)=\tilde{P}\left(k\right)x\left(k\right)$, where:

$${\tilde{P}}_k=Q+K_{k+1}^1{}^{\prime }R{K}_{k+1}^1+{{\overline{A}}^{\prime }}_{k+1}{\tilde{P}}_{k+1}{\overline{A}}_{k+1}-{{\Gamma }^{\prime }}_{k+1}{\psi }^0{{}^{\prime }}_{k+1}{Z}_{k+1}$$

(37)

At this time, $\varsigma \left(k\right)=0$, and ${\theta }_N={\tilde{P}}_{N+1}{x}_{N+1}$ and ${\tilde{P}}_{N+1}=\tilde{H}$. Namely, ${\theta }_N={\tilde{P}}_{N+1}{x}_{N+1}$. If $\varsigma \left(k\right)$ is true at time k, so ${\varsigma }_k={\theta }_k-{\tilde{P}}_{k+1}{x}_{k+1}$, and the following formula is obtained:

$${\mu }_k^1=-{\psi }_{k+1}^0\ast Z_{k+1}{x}_k-{\psi }_{k+1}^0{\overline{B^{\prime }}}_{k+1}{\Gamma }_k+\left[I-{\psi }_{k+1}^0\ast {\psi }_{k+1}^0\right]{\phi }_k,$$

(38)

where $\phi \left(k\right)$ is the arbitrary terms with the appropriate dimension. By substituting this formula and ${\varsigma }_k={\theta }_k-{\tilde{P}}_{k+1}{x}_{k+1}$ into ${\theta }_{k-1}$, the following formula can be obtained:

$${\theta }_{k+1}={\tilde{P}}_k{x}_k+{\varsigma }_k$$

(39)

where ${{\Gamma }^{\prime }}_{k+1}\left[I-{\psi }_{k+1}^0\ast {\psi }_{k+1}^0\right]=0$ is applied to the derivation of the above formula, and the following formula can be obtained:

$${\varsigma }_{k-1}=[{\overline{A}}_{k+1}-{{\overline{A}}^{\prime }}_{k+1}{\tilde{P}}_{k+1}{\overline{B}}_{k+1}{\psi }_{k+1}^0\ast {\overline{B^{\prime }}}_{k+1}+K_{k+1}^1{}^{\prime }R{Z}_{k+1}^0{\psi }_{k+1}^0\ast {\overline{B^{\prime }}}_{k+1}]{\varsigma }_k$$

(40)

As ${\varsigma }_N=0$, it can be concluded that ${\varsigma }_k=0,k=0,1,\cdots ,N$.

Next, the state-space model and the indicator function of the decision scheme are considered. When $H_F$ meets the above assumptions, there is an optimal solution ${\mu }_k^1$ to the optimization problem, when the following conditions are satisfied: (1) ${\tilde{P}}_0\ge 0$; (2) ${\psi }_k\ge 0$. The problem is solvable when the optimal solution is met, and the optimal solution is as follows.

$${\mu }_k^1*=-{\psi }_{k+1}^1\ast {\Gamma }_{k+1}{x}_k+\left[I-{\psi }_{k+1}^1\ast {\psi }_{k+1}^1\right]{\phi }_k,$$

(41)

where $\phi \left(k\right)$ is the arbitrary vector with the appropriate dimension. Then the performance index of the corresponding optimal decision scheme is as follows:

$$J_F^{\ast }={x^{\prime }}_0{P}_0{x}_0$$

(42)

The specific proof process is shown in Appendix B.

At this time, ${\tilde{P}}_0\ge 0$ and ${\psi }^0{}_{k+1}\ge 0$. Therefore, the optimal decision-making scheme of participants exists, and the optimal accuser-participant decision-making scheme, i.e., the global optimal decision-making scheme, is as follows:

$${\mu }_k^{*}=-\left[{\psi }_{k+1}\ast {\tau }_{k+1}+Z_{k+1}^0{\psi }_{k+1}\ast {\Gamma }_{k+1}\right]x_k+Z_{k+1}^0\left[I-{\psi }_{k+1}^0\ast {\psi }_{k+1}^0\right]{\phi }_k$$

(43)

$$\begin{array}{ll}{\omega }_k^{*}=& -Z_{k+1}^{-1}\left[M_{k+1}-C^{\prime }{P}_{k+1}B{G}_k^0{\psi }_{k+1}^0\ast \times {\Gamma }_{k+1}\right]x_k\\ & -Z_{k+1}^{-1}{C}^{\prime }{P}_{k+1}B{G}_{k+1}^0\left[I-{\psi }_{k+1}^0\ast {\psi }_{k+1}^0\right]\end{array}$$

(44)

Thus, a closed-loop solution to the Stackelberg game is obtained, and the optimal decision scheme of the accuser, the participant is obtained, and global profit maximization is finally achieved.

3.4. Model Performance Comparison

In this paper, a multi-master, multi-slave Stackelberg game model is designed. Under the same resource deployment situation, the optimization effect and model complexity of the model designed in this paper, and the unimproved model under the same time limit are compared, as shown in the

Table 2. Model iteration efficiency comparison.

Model	Limited Time/s	Iterations	Average Objective Function Value
Model in this paper	1	20	7.458
	2	50	7.116
	5	100	7.054
Unimproved model	1	20	19.225
	2	50	10.636
	5	100	9.545

and

Table 3. Comparison of complexity between time and space of model.

Model	Time Complexity	Space Complexity	Sample Data Volume
Model in this paper	$O\left(n^2\right)$	$O\left(m^n\right)$	5000
	$O\left(1\right)$	$O\left(m\right)$	50,000
	$O\left(1\right)$	$O\left(m\right)$	1,000,000
Unimproved model	$O\left(n^2\right)$	$O\left(m^n\right)$	5000
	$O\left({\log }^{2n}\right)$	$O\left(m^n\right)$	50,000
	$O\left(n^2\right)$	$O\left(m\right)$	1,000,000
Reference 13 model	$O\left(n^2\right)$	$O\left(m^n\right)$	5000
	$O\left(n^2\right)$	$O\left(m\right)$	50,000
	$O\left(1\right)$	$O\left(m\right)$	1,000,000
Reference 9 model	$O\left({\log }^{2n}\right)$	$O\left(m\right)$	5000
	$O\left(n^2\right)$	$O\left(m\right)$	50,000
	$O\left(1\right)$	$O\left(m\right)$	1,000,000

.

According to the experimental data, after the learning rate is improved adaptively, the function value and the time-space complexity of the model are analyzed under the condition of the same number of iterations, and it is found that the model in this paper can achieve lower objective function values with better convergence and higher efficiency [38].

By comparing the performance of the model, this paper sets the same initial function value and iteration times between the literature model and the model. Finally, under the same iteration times, it can be seen that the model in this paper has better convergence from

Table 4. The comparison of model performance.

Model	Iterations	Initial Function Value	Objective Function Value
Reference [38]	27	2	1.4
Reference [39]	27	2	1.1
Model in this paper	27	2	1.0
Reference [38]	58	1.5	1.2
Reference [39]	58	1.5	0.3
Model in this paper	58	1.5	−1.0
Reference [38]	9	0.9959	0.9903
Reference [39]	9	0.9959	0.9950
Model in this paper	9	0.9959	0.9877

.

4. Numerical Simulation

To verify the validity of the above results, a numerical example is given in this section. Some parameters of the multi-agent system are assumed as follows:

$$\begin{array}{l}A=\left[\begin{array}{ll}1& 1\\ 1& 1\end{array}\right],B_1=\left[\begin{array}{ll}1& 0\\ 1& 0\end{array}\right],B_2=\left[\begin{array}{ll}1& 1\\ 1& 1\end{array}\right],\\ \\ C^{\prime }C=\left[\begin{array}{ll}1& 1\\ 1& 1\end{array}\right],D^{\prime }D=\left[\begin{array}{ll}1& 0\\ 1& 0\end{array}\right]\end{array}$$

(45)

${\psi }^2$ is equal to 4, ${\pi }_0^{-1}$ is equal to $\left[\begin{array}{ll}3& 2.98\\ 2.98& 3\end{array}\right]$. The initial state is $x_0=\left[\begin{array}{l}1\\ 2\end{array}\right]$. Let $N=2$, ${{\rm P}}_3=\left[\begin{array}{ll}1& 1\\ 1& 1\end{array}\right]$. First of all, the optimization process of $H_L$ is analyzed. Through algebraic iterative operation, the solution of the regular Riccati Equation (17) is obtained as follows:

$${\rm P}\left(2\right)=\left[\begin{array}{ll}2.33& 2.33\\ 2.33& 2.33\end{array}\right],{\rm P}\left(1\right)=\left[\begin{array}{ll}2.65& 2.65\\ 2.65& 2.65\end{array}\right],{\rm P}\left(0\right)=\left[\begin{array}{ll}2.68& 2.68\\ 2.68& 2.68\end{array}\right]$$

(46)

where $P\left(0\right)-{\pi }_0^{-1}<0$. By combining with the results obtained from the above operation, (20) and (21) are calculated, and the optimal controller described in theorem 1 can be obtained as follows:

$$\begin{array}{c}u\left(2\right)=\left[\begin{array}{c}-15.92\\ 0\end{array}\right]+\left[\begin{array}{cc}0& 0\\ 0& 1\end{array}\right]\mathsf{\phi }\left(2\right),\\ u\left(1\right)=\left[\begin{array}{c}-9.84\\ 0\end{array}\right]+\left[\begin{array}{cc}0& 0\\ 0& 1\end{array}\right]\mathsf{\phi }\left(1\right),\\ u\left(0\right)=\left[\begin{array}{c}-5.04\\ 0\end{array}\right]+\left[\begin{array}{cc}0& 0\\ 0& 1\end{array}\right]\mathsf{\phi }\left(0\right),\end{array}$$

(47)

$$\begin{array}{l}w\left(2\right)=\left[\begin{array}{l}7.84\\ 7.84\end{array}\right]+\left[\begin{array}{ll}0& 0.64\\ 0& 0.64\end{array}\right]\mathsf{\phi }\left(2\right),\\ w\left(1\right)=\left[\begin{array}{l}4.92\\ 4.92\end{array}\right]+\left[\begin{array}{ll}0& 0.62\\ 0& 0.62\end{array}\right]\mathsf{\phi }\left(1\right),\\ w\left(0\right)=\left[\begin{array}{l}2.52\\ 2.52\end{array}\right]+\left[\begin{array}{ll}0& 0.61\\ 0& 0.61\end{array}\right]\mathsf{\phi }\left(0\right),\end{array}$$

(48)

In the $H_F$ optimization, the solution of the regular Riccati Equation (27) is shown as follows:

$$\widetilde{{\rm P}}\left(2\right)=\left[\begin{array}{ll}6.70& 6.70\\ 6.70& 6.70\end{array}\right], \widetilde{{\rm P}}\left(1\right)=\left[\begin{array}{ll}20.16& 20.16\\ 20.16& 20.16\end{array}\right], \widetilde{{\rm P}}\left(0\right)=\left[\begin{array}{ll}12.65& 12.65\\ 12.65& 12.65\end{array}\right].$$

(49)

The arbitrary term ${\phi }_k$ in the decision-making scheme of the participants obtained in the previous optimization process can be transformed into ${\mu }^1{}_k$ through matrix transformation. The value of ${\mu }_k^1$ obtained by calculating Equation (32) is as follows:

$$\begin{array}{c}{\mu }_1\left(2\right)=\left[\begin{array}{c}-0.2019\\ 0\end{array}\right]+\left[\begin{array}{cc}0& 0\\ 0& 1\end{array}\right]\mathsf{\phi }\left(2\right),\\ {\mu }_1\left(1\right)=\left[\begin{array}{c}13.25\\ 0.3\end{array}\right]+\left[\begin{array}{cc}0& 0\\ 0& 1\end{array}\right]\mathsf{\phi }\left(1\right),\\ {\mu }_1\left(0\right)=\left[\begin{array}{c}173.65\\ 0\end{array}\right]+\left[\begin{array}{cc}0& 0\\ 0& 1\end{array}\right]\mathsf{\phi }\left(0\right).\end{array}$$

(50)

The arbitrary term $\phi \left(k\right)\left(k=2,1,0\right)$ in Equation (32) is as follows:

$$\mathsf{\phi }\left(2\right)=\left[\begin{array}{c}1\\ 0.1\end{array}\right],\mathsf{\phi }\left(1\right)=\left[\begin{array}{c}0.6\\ 0.3\end{array}\right],\mathsf{\phi }\left(0\right)=\left[\begin{array}{c}1\\ 0.7\end{array}\right]$$

(51)

Substituting ${\phi }_k$ into ${\mu }^1{}_k$, and then according to Equation (3), we can get the optimal controller (Equations (52) and (53), respectively):

$$\mathsf{\mu }\left(2\right)=\left[\begin{array}{c}-15.92\\ 0\end{array}\right],\mathsf{\mu }\left(1\right)=\left[\begin{array}{c}-9.84\\ 0\end{array}\right],\mathsf{\mu }\left(0\right)=\left[\begin{array}{c}-5.04\\ 0\end{array}\right],$$

(52)

$$\mathsf{\mu }\left(2\right)=\left[\begin{array}{c}-15.92\\ 0\end{array}\right],\mathsf{\mu }\left(1\right)=\left[\begin{array}{c}-9.84\\ 0\end{array}\right],\mathsf{\mu }\left(0\right)=\left[\begin{array}{c}-5.04\\ 0\end{array}\right],$$

(53)

After calculation, the optimal performance index of $H_L$ norm form is $J_L=-6.35$. The optimal performance index of $H_F$ norm form is $JF=79.65$.

5. The Experimental Simulation

In this section, the air defense of important places in the field of air defense and anti-missile defense is taken as the experimental scene, constraint conditions and objective functions are designed as the simulation environment, and a multi-agent system based on edge Laplace matrix is constructed to analyze the system complexity and verify the reliability and robustness of the system.

5.1. The Description of Simulation Scenario

This chapter will be based on the air defense of important place as the background. Considering the influence of mountains, the algorithm in this chapter will be used to model the multi-agent system, and the system will be used to deploy the battlefield fire unit to verify whether the key points can be defended and the target function value can be achieved under the specified incoming fire.

5.1.1. Parameters Definition

(1) Battlefield environment:

Terrain: mainly flat terrain, some areas for undulating mountains

Area: 300 km × 300 km

Object of protection: important place

(2) Strength formation:

Security object: important place

Security forces: 8 launching vehicles with a running speed of 40 km/h, which can stop shooting for a short time. Each vehicle is equipped with a number of medium range bombs with a range of 50 km and an average speed of 800 m/s; 20 short range bombs with a range of 20 km and an average speed of 600 m/s. There are five radar vehicles with a detection distance of 50 km for cruise missile, 120 km for fighter, 100 km for helicopter, and 80 km for UAV.

(3) Tactical purpose:

The coverage of detection area of one layer shall be more than 85%, that of two layers shall be more than 55%, and that of key areas shall be at least two layers; that of fire area of one layer shall be more than 80%, that of two layers shall be more than 70%, and that of key areas shall be at least three layers.

Resources are the operational objects that command and control the deployment behavior of resources. They are physical resource entities. Resources include fire units and detection units. The set of resources is $P=\left\{P_1,P_2,\cdots ,P_j\right\}$, and $j$ is the number of resources. In this chapter, the platform resources are mainly radar vehicle, launch vehicle, medium range missile and short-range missile.

Command entity: the command entity is the entity that receives situation information, makes decisions, and completes the deployment of fire units, task allocation, situation information interaction, command and decision-making, etc. The collection of charged entities is $D=\left\{D_1,D_2,\cdots ,D_n\right\}$, and $n$ is the number of charged entities. In this model, the command entity is radar vehicle, which belongs to scheme release agent. Each radar vehicle is the same level of command entity, but there is a logic center at some time, which is distributed deployment.

There are four capability parameters of the accusing entity:

(1)

How many launching vehicles can each radar vehicle $D_n$ control at the same time is called the control and management ability $C_{Control}^{\left(n\right)}$ of the command entity $D_n$.

(2)

How many identical tasks each radar vehicle $D_n$ performs with other radar vehicles is called the cooperation capability $C_{Cooperation}^{\left(n\right)}$ of the command entity $D_n$.

(3)

How many tasks can each radar vehicle $D_n$ handle at the same time is called the task processing capacity $C_{Task}^{\left(n\right)}$ of decision entity $D_n$.

(4)

How many identical tasks each radar vehicle $D_n$ performs with other radar vehicles is called the decision entity $D_n$’s cooperation capability $C_{Cooperation}^{\left(n\right)}$.

5.1.2. Element Definition and Load Design

The following elements need to be defined to describe the deployment of command and control resources:

(1)

task to resource assignment capability $n_{TP}^{\left(i,j\right)}(i=1,2,\cdots ,I;j=1,2,\cdots ,J)$ if task t $T_j$ is assigned to resource $P_j$, then $n_{TP}^{\left(i,j\right)}=1$, otherwise $n_{TP}^{\left(i,j\right)}=0$. In this model, the ability variable of task to resource allocation belongs to known quantity.

(2)

the control ability of decision-making entity over resources $n_{TP}^{\left(m,j\right)}(m=1,2,\cdots ,M;j=1,2,\cdots ,J)$, if resource $P_j$ belongs to decision-making entity $D_n$, then $n_{TP}^{\left(m,j\right)}=1$, otherwise $n_{TP}^{\left(m,j\right)}=0$.

(3)

assignment ability $n_{TP}^{\left(i,m\right)}(i=1,2,\cdots ,I;m=1,2,\cdots ,M)$ from task to decision entity. If task $T_j$ is assigned to decision entity $D_n$, then $n_{TP}^{\left(i,m\right)}=1$, otherwise $n_{TP}^{\left(i,m\right)}=1$.

Internal workload of decision entity.

The workload that decision entity $D_n$ controls its platform to perform tasks is called $D_n$’s internal workload, which is recorded as $W_{Task}^{\left(n\right)}$, and its calculation formula is $D_n$’s internal workload, which is recorded as $W_{Task}^{\left(n\right)}$, and its calculation formula is as follows:

$$\begin{array}{ll}{W}_{Task}^{\left(n\right)}=& Z_t\left[\frac{1}{t_{Complete}}{\displaystyle \sum _{j=1}^J{n}_{DP}^{\left(m,j\right)}\left({\displaystyle \sum _{i=1}^I{n}_{DP}^{\left(i,j\right)}{t}_{Task}^{\left(i\right)}}\right)}\right]\\ & +Z_d\left[{\displaystyle \sum _{j=1}^J{n}_{DP}^{\left(m,j\right)}\left({\displaystyle \sum _{i=1}^I{n}_{DP}^{\left(i,j\right)}{t}_{Task}^{\left(i\right)}}\right)}\right]\end{array}$$

(54)

where $t_{Task}^{\left(i\right)}$ is the execution time of task T_i; $t_{Complete}$ is the final completion time of all tasks; $d_{Task}^{\left(i\right)}$ is the difficulty of the task; $Z_t$ is the weighting coefficient of execution time factor; $Z_d$ is the weighting coefficient of task difficulty degree.

Collaborative load between decision entities.

The collaboration load of any two decision entities $D_n$ h and $D_n$ is the workload that $D_n$ and $D_n$ must cooperate to complete the task, which is recorded as $W_{Cooperation}^{\left(n,n\prime \right)}$, and its calculation formula is as follows:

$$W_{Cooperation}^{\left(n,n\prime \right)}=\frac{1}{t_{Complete}}{\displaystyle \sum _{i=1}^I{n}_{TD}^{\left(i,m\right)}{n}_{TD}^{\left(i,m\prime \right)}{t}_{Task}^{\left(i\right)}}$$

(55)

5.2. Constraint Analysis and Objective Function Design

5.2.1. Constraint Analysis

The constraints of the command and control resource deployment model are analyzed as follows:

The decision entity performs tasks through the platform it controls. Therefore, when at least one platform $P_i$ in the platform $D_m$ controls the variable $n_{TP}^{\left(i,j\right)}=1$ and $n_{DP}^{\left(i,j\right)}=1$, then $n_{TD}^{\left(i,m\right)}=1$, otherwise, $n_{TD}^{\left(i,m\right)}=0$, that is:

$$n_{TD}^{\left(i,m\right)}={\max }_{j=1}^J{n}_{DP}^{\left(m,j\right)}{n}_{TP}^{\left(i,j\right)},i=1,\cdots I;m=1\cdots M;j=1,\cdots J$$

(56)

There is an upper capacity limit for the control management ability of each decision-making entity, that is:

$$C_{control}^{\left(m\right)}={\displaystyle \sum _{j=1}^J{n}_{DP}^{\left(m,j\right)}}\le C_{comtrol},m=1,\cdots M;j=1,\cdots J$$

(57)

In the formula, $C_{comtrol}$ is the upper limit of control management capacity, and the constraint value of all decision entities is set to be the same in this paper. There is an upper capacity limit for the task processing capacity of each decision entity, namely:

$$C_{Task}^{\left(m\right)}={\displaystyle \sum _{i=1}^I{n}_{TD}^{\left(i,m\right)}}\le C_{Task},i=1,\cdots I;m=1\cdots M$$

(58)

In the formula, $C_{Task}$ is the upper limit of cooperative ability capacity, and the constraint value of all decision entities is set to be the same in this paper. There is an upper capacity limit for the task processing capacity of each decision entity, namely:

$$C_{Cooperation}^{\left(m\right)}={\displaystyle \sum _{m\prime =1,m\prime \ne m}^M{\displaystyle \sum _{i=1}^I{n}_{TD}^{\left(i,m\right)}{n}_{TD}^{\left(i,m\prime \right)}}\le C_{Cooperation}},i=1,\cdots I;m=1\cdots M$$

(59)

In the formula, $C_{Cooperation}$ is the upper limit of cooperation ability. In this paper, all decision entities are set with the same constraint value.

5.2.2. Objective Function Design

As can be seen from the above, the sum of workload of all agents remains the same during the task assignment, while the workload varies with the assignment. According to this characteristic, we want to achieve the optimal, then the total load should be minimized. Let the amount of tasks assigned to each agent be $w_1,w_2\dots w_n$, and the amount of tasks received by each agent be $f_1(w_1),f_2(w_2)\dots f_n(w_n)$. The workload for all agents is $g(w_1,w_2\dots w_n)=f_1(w_1)+f_2(w_2)+\dots +f_n(w_n)$. So if $g(w_1,w_2\dots w_n)$ is the smallest, the smaller the total cost, the more reasonable the assignment. Then the objective function is:

$$\min \left\{g(w_1,w_2\dots w_n)\right\}=\min \left\{f_1(w_1)+f_2(w_2)+\dots +f_n(w_n)\right\}$$

(60)

After using $\left\{\begin{array}{l}\frac{\partial g(w_1,w_2\dots w_n)}{\partial w_1}=0\\ ⋮\\ \frac{\partial g(w_1,w_2\dots w_n)}{\partial w_n}=0\end{array}\right\}$ and $\left\{\begin{array}{l}\frac{\partial g(w_1,w_2\dots w_n)}{\partial w_1}=0\\ ⋮\\ \frac{\partial g(w_1,w_2\dots w_n)}{\partial w_n}=0\end{array}\right\}$ in the boundary, the group $w_1,w_2\dots w_n$ which makes $g(w_1,w_2\dots w_n)$ the smallest is selected.

5.3. System Training Data Demonstration

This section will demonstrate and analyze the system from the perspective of a reinforcement learning network, global gain function and global loss function. When the number of iterations is between 2500 k and 3000 k, the maximum global gain is basically reached and becomes stable.

Experimental background is set to be the deployment of battlefield resources. Figure 1 is the iterative curve map of the stability of multi-agent system, where the horizontal axis is the number of iterations, and the vertical axis is the revenue function of multi-agent system. With the increase of the number of iterations, when the number of iterations reaches 2500 times, the revenue function tends to be stable, which proves that the multi-agent system tends to be stable and has excellent robustness. The details are shown in the following figure:

It can be concluded from Figure 1 that, as the training iteration proceeds, the global return gradually increases after a period of time from a low level, during which there is a small range of ups and downs, finally reaches the optimal global return, and tends to converge.

Figure 2 is the graphical display of reinforcement learning network. The 3D coordinate system of reinforcement learning network is established, and every point in the figure represents a performance index of the intelligent agent, and the location represents the index’s specific parameter value and the state function of the intelligent agent at that time. With the increase of the number of iterations, the state transfer of the intelligent agent is carried out, and the coordinate information also changes. There are more than 2000 parameters for the multi-agent system in the command and control system of this paper, and the specific details are shown in Figure 2:

As the number of iterations increases, the global gain increases, and as the corresponding global loss gradually decreases, when the algorithm approaches the global optimization, the loss function tends to be stable, with a small range of fluctuations. The change trajectory of the function corresponds to Figure 1, so as to ensure that the total battlefield situation remains unchanged. Figure 3 is the consistency analysis diagram of the multi-agent system. It can be seen from the figure that, with the increase of the number of iterations, when it reaches about 1000 k, the edge state of the intelligent agent tends to balance. Meanwhile, it can be seen that after a certain number of iterative training, the multi-agent system is locally Lipschitz continuous and has excellent consistency. The change trend of the specific loss value function is shown in the figure below:

It can be concluded from Figure 3 that, at the beginning of the iteration, that the loss value shows a rapid decline. When the number of iterations reaches 500–1000 k, the loss value starts to drop slightly, but the average value declines slowly. When the number of iterations reaches about 1500 k, the loss value starts to become stable and reaches the minimum value of the global loss function.

6. Conclusions

This paper analyzes and solves the game interaction model of the multi-agent system in distributed combat environment, and provides a solution to the accuser participant game problem in the multi-agent system with a semi-positive definite weighted matrix based on the performance index. In order to solve this problem, the optimal decision $\left(\mu ,\omega \right)$ is first obtained in the accuser’s optimization, in which the size of $\mu ,\omega$ is controlled to obtain the optimal value of $J_L$. Due to its semi-positive definite nature and the existence of arbitrary term item in $\mu$, the arbitrary term obtained before is transformed into the item to be solved in $J_F$ to further optimize $J_F$ after matrix transformation. Based on the above optimization process, the Riccati equation is solved and the closed-loop solution of the multi-master and multi-slave Stackelberg game model is given, which solves the problem of data interaction and local conflict resolution between various intelligent agents in the command and control system. The research of this paper not only broadens the depth of the academic theory of distributed command and control, but also satisfies the interactive game model mechanism in distributed multi-agent systems, which lays a theoretical foundation for intelligent warfare.