Reinforcement Learning-Based Decentralized Safety Control for Constrained Interconnected Nonlinear Safety-Critical Systems

Abstract

This paper addresses the problem of decentralized safety control (DSC) of constrained interconnected nonlinear safety-critical systems under reinforcement learning strategies, where asymmetric input constraints and security constraints are considered. To begin with, improved performance functions associated with the actuator estimates for each auxiliary subsystem are constructed. Then, the decentralized control problem with security constraints and asymmetric input constraints is transformed into an equivalent decentralized control problem with asymmetric input constraints using the barrier function. This approach ensures that safety-critical systems operate and learn optimal DSC policies within their safe global domains. Then, the optimal control strategy is shown to ensure that the entire system is uniformly ultimately bounded (UUB). In addition, all signals in the closed-loop auxiliary subsystem, based on Lyapunov theory, are uniformly ultimately bounded, and the effectiveness of the designed method is verified by practical simulation.

1. Introduction

Over the past few decades, safety has received increasing attention in autonomous driving [1], intelligent robots [2], robotic arms [3], adaptive cruise control [4], etc. The design of these systems and controllers require that the system state trajectories evolve within a set called the safe set, reflecting the inherent properties of the system [5]. In practice, many engineering systems must operate within a specific safety range, beyond which the controlled system may be at risk [6]. Safety-critical systems primarily refer to systems having control behaviors that prioritize safety. The designed control schemes aim to reduce the potential for severe consequences, such as personal injury and environmental pollution, which may arise due to system shutdown or operational errors [7]. To ensure the safety and reliability of the system, scholars developed many safety control schemes. The classical approach focused on extending and applying Naguma’s theorem to safe sets defined by continuously differentiable functions [8]. In particular, barrier functions have become an effective tool for verifying security and have been widely used in [9,10,11]. They were used to convert a system with security constraints into an equivalency system that satisfies security requirements and then a security controller was designed to protect the system. In [9,10], penalty functions and BF-based state transitions were employed to merge states into a reinforcement learning framework to solve optimal control problems with full-state constraints. In [11], a safe non-strategic reinforcement learning method to solve secure nonlinear systems with dynamic uncertainty was proposed. In [12,13], a new secure reinforcement learning method was proposed to solve secure nonlinear systems with symmetric input constraints. However, the results in [9,10,11,12,13], mentioned above, were mainly based on studying the optimal safety control in a single continuous-time/discrete-time nonlinear system. The security control of interconnected systems has not been fully resolved.

On the other hand, interconnected systems consist of multiple subsystems with interconnected characteristics, and designing controllers for them through a concept similar to that of a single-system approach is difficult [14]. To solve this problem [15,16,17], the decentralized control approach, based on local subsystem information, was proposed. This approach involved using multiple controllers to control the interconnected systems. In [18,19], the decentralized control approach differed by initially decomposing the entire system control problem into a series of subproblems that could be solved independently. The solutions to the subproblems (i.e., independent controllers) were then joined to form a decentralized controller to stabilize the entire system. In addition, implementing the decentralized control algorithm used only the local subsystem’s knowledge, not the complete system’s information. Recently, scholars have proposed many schemes or techniques for designing decentralized controllers, including quantization techniques [20], fuzzy techniques [21], and optimal control methods [22]. This paper develops decentralized control strategies from the optimal control perspective. Problems of optimal control are usually solved via the solution of the Hamilton–Jacobi–Bellman (HJB) partial differentiation equation [23,24]. However, the HJB equation is generally not solvable analytically due to its inherent nonlinearity [25,26]. Therefore, adaptive dynamic programming (ADP) and reinforcement learning (RL) algorithms were proposed to obtain numerical solutions to the HJB equation and were widely applied to nonlinear interconnected systems [27,28,29,30]. In [31,32], the two previously mentioned algorithms could be deemed closely related, as they exhibited similar characteristics in addressing optimal control problems. For example, in [27,28], the distributed optimal controller was designed using robust ADP for nonlinear interconnected systems with unknown dynamics and parameters. In [29], the optimal decentralized control problem for interconnected nonlinear systems subject to stochastic dynamics was solved by enhancing the performance function of the auxiliary subsystem and transforming the original control problem into a set of optimal control strategies sampled in periodic patterns. Furthermore, in [30], the identifier–critic network framework was used to solve the problem of decentralized event-triggered control based on sliding-mode surfaces, avoiding the need for knowledge of the system’s internal dynamics. It is worth noting that the control results provided in [27,28,29,30] did not consider input constraints.

Control constraints are commonly encountered in industrial processes, where they are widespread and have a detrimental impact on the performance of systems [33,34]. Therefore, the study of constrained nonlinear systems is of practical importance. In [35,36], the RL-based decentralized algorithm was developed for tracking control of constrained interconnected nonlinear systems. In [37], the problem of decentralized optimal control of a constrained interconnected nonlinear system was solved by introducing a nonquadratic performance function to overcome the symmetric input constraint. The results in [35,36,37], mentioned above, mainly addressed the symmetric input constraint. However, the problem of asymmetric input constraints was identified in several project cases [38,39]. In [40], the optimal decentralized control problem with asymmetric input constraints was solved by designing a new non-quadratic performance function. In [41], a new performance function was proposed for interconnected nonlinear systems to successfully overcome the asymmetric input constraint and to solve the decentralized fault-tolerant control problem. However, none of the above studies considered the safety of the system. The optimal decentralized safety control (DSC) for constrained interconnected nonlinear safety-critical systems has not been thoroughly investigated thus far, which inspired our current study.

Motivated by previous discussions, this paper proposes an RL-based decentralized DSC strategy for constrained interconnected nonlinear safety-critical systems. The primary achievements are concluded below:

• The reinforcement learning algorithm is used to solve the optimal DSC problem for restricted interconnected nonlinear safety-critical systems, and the asymmetric input constraint is successfully solved. The method optimizes the control strategy by minimizing the performance function, ensuring the safety of the system’s state, while considering the asymmetric input constraints.
• Nonlinear interconnected safety-critical systems with asymmetric input constraints and safety constraints are converted to equivalent systems that satisfy user-defined safety constraints using barrier functions. Unlike the nonlinear safety-critical systems [3,9,10,13], this paper solves the security constraint problem of the interconnection term through the potential barrier function, which ensures the interconnected nonlinear safety-critical system satisfies the security constraint.
• The asymmetric input constraints are solved by utilizing a single CNN architecture for online approximation of the performance function. Theoretical demonstrations show that the optimal DSC method can achieve uniformly ultimately bounded (UUB) system states and neural network weight estimation errors. In addition, a simulation example verified the feasibility and effectiveness of the developed DSC method.

The remainder of this article is structured as follows. In Section 2, the issue formulation and conversion are presented. In Section 3, the decentralized optimal safety DSC design scheme is presented. The design scheme for the critical neural network is presented in Section 4. In Section 5, the analyses of system stability are presented. In Section 6, the simulation sample demonstrates the effectiveness of the presented approach. Lastly, conclusions are given in Section 7.

2. Preliminaries

2.1. Problem Descriptions

Consider a constrained interconnected nonlinear safety-critical system composed of n subsystems and the formula below:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \left\{\begin{array}{c}\dot{x_i}\left(t\right)=f_i\left(x_i\left(t\right)\right)+g_i\left(x_i\left(t\right)\right)u_i\left(t\right)+▵h_i\left(x\left(t\right)\right),\hfill \\ x_i\left(0\right)=x_{i0},i=1,2,\dots ,n,\hfill \end{array}\right.\end{array}\end{array}$$

(1)

where $x_i\left(t\right)\in {\mathbb{R}}^{n_i}$ is the ith subsystem’s state vector and $x_i\left(0\right)$ represents the initial state, $x=\left[x_1^T,x_2^T,\dots ,x_n^T\right]\in {\mathbb{R}}^{{\sum }_{i=1}^n{n}_i}$ represents the overall state vector of the constrained interconnected nonlinear safety-critical system, $u_i={[u_{i,1},u_{i,2},\dots u_{i,\mathfrak{j}}]}^T\in {\mathfrak{k}}_i$ represents the control input, and the set of asymmetric constraints is represented as ${\mathfrak{k}}_i=\left\{u_{i,m_i}\in {\mathbb{R}}^{m_i},{\mathfrak{h}}_{imin}\le \left|u_{i,\mathfrak{j}}\right|\le {\mathfrak{h}}_{imax},\mathfrak{j}=1,2,\dots ,m_i\right\}$ with ${\mathfrak{h}}_{imin}$ and ${\mathfrak{h}}_{imax}$ being the asymmetric saturating minimum and maximum bounds, $f_i\left(·\right)\in {\mathbb{R}}^{n_i}$ and $g_i\left(·\right)\in {\mathbb{R}}^{n_i\times m_{{}_i}}$ represent the drift system dynamics and input dynamics of the ith subsystem, respectively, and are Lipschitz continuous, and $▵h_i\in {\mathbb{R}}^{n_i}$ represents the unknown interconnected term.

To simplify the design of the controller, let us introduce some assumptions. For $i=1,2,\dots ,n$, we suppose the equilibrium of the ith subsystem’s state is $x_i=0$.

Assumption 1.

For $i=1,2,\dots ,n$, the $▵h_i\left(x\right)$ satisfies the below unmatched condition:

$$△h_i={\eta }_i\left(x_i\right){\mathcal{P}}_i\left(x\right),$$

where ${\eta }_i\left(x_i\right)$ is a known function with ${\eta }_i\left(x_i\right)\in {\mathbb{R}}^{n_i\times q_{{}_i}}\ne g_i\left(x_i\right)$, and ${\mathcal{P}}_i\left(x\right)$ is a bounded vector function that satisfies

$$∥{\mathcal{P}}_i\left(x\right)∥\le \sum _{j=1}^n{b}_{i,j}{\beta }_{i,j}\left(x_j\right),$$

(2)

where $b_{i,j}>0$ is a constant, and ${\beta }_{i,j}\left(x_j\right)$ are normal definite functions. Furthermore, ${\beta }_{i,j}\left(0\right)=0$ and ${\mathcal{P}}_i\left(0\right)=0$. Then, assuming ${\beta }_j\left(x_j\right)=ma{x}_{1\le i\le n}\left\{{\beta }_{i,j}\left(x_j\right)\right\}$, the unequal Equation (2) is denoted as:

$$∥{\mathcal{P}}_i\left(x\right)∥\le \sum _{j=1}^n{C}_{i,j}{\beta }_j\left(x_j\right),$$

(3)

where $C_{i,j}\ge \left(b_{i,j}{\beta }_{i,j}\left(x_j\right)\right)/{\beta }_j\left(x_j\right)$ is a positive constant, and $j=1,2,\dots ,n$.

Remark 1.

It is noted that constraints (2) and (3) specified by Assumption 1 are strict restrictions on specific interrelated nonlinear systems. Nevertheless, when we consider the function ${\mathcal{P}}_i\left(x\right)$ that satisfies no constraints (2) and (3), we discover that the calculational costs to address the stability of the closed-loop system are high. In fact, in real-world applications, constraints like inequalities (2) and (3) impose on the mismatched interconnection terms of the system (1) [40,42].

Assumption 2.

For $i=1,2,\dots ,n$, the known function $g_i\left(x_i\right)$ is bounded as $∥g_i\left(x_i\right)∥\le g_{i,m}$, where $g_{i,m}$ is a known constant. Furthermore, $rank\left(g_i\left(x_i\right)\right)=m_i$ and $g_i^T\left(x_i\right){\eta }_i\left(x_i\right)=0$.

Based on the ith subsystem (1) described, the ith auxiliary subsystem is designed as:

$$\dot{x_i}=f_i\left(x_i\right)+g_i\left(x_i\right)u_i+\left(I_{n_i}-g_i\left(x_i\right)g_i^{+}\left(x_i\right)\right){\eta }_i\left(x_i\right)v_i,$$

(4)

where $v_i\in {\mathbb{R}}^{q_i}$ is used to compensate for mismatched interconnections and stands for auxiliary control, $g_i^{+}\left(x_i\right)\in {\mathbb{R}}^{m_i\times n_i}$ is Moore–Penrose pseudo-reverse. According to Assumption 2, it can be found that the matrix $g_i^{+}\left(x_i\right)={\left(g_i^T\left(x_i\right)g_i\left(x_i\right)\right)}^{-1}{g}_i^T\left(x_i\right)$ and $g_i^{+}\left(x_i\right){\eta }_i\left(x_i\right)={\left(g_i^T\left(x_i\right)g_i\left(x_i\right)\right)}^{-1}{g}_i^T\left(x_i\right){\eta }_i\left(x_i\right)=0$. Then, we rewrite the auxiliary subsystem (4) as:

$$\dot{x_i}=f_i\left(x_i\right)+g_i\left(x_i\right)u_i+{\eta }_i\left(x_i\right)v_i.$$

(5)

2.2. Security Conversion Issues

For the ith subsystem in the system (1), its state $x_i={\left[x_{i,1},x_{i,2},\dots ,x_{i,k}\right]}^T$ satisfies the following security constraints:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \left\{\begin{array}{c}{x}_{i,1}\in (a_{i,1},A_{i,1}),\hfill \\ x_{i,2}\in (a_{i,2},A_{i,2}),\hfill \\ .\hfill \\ .\hfill \\ .\hfill \\ x_{i,k}\in (a_{i,k},A_{i,k}).\hfill \end{array}\right.\end{array}\end{array}$$

(6)

For nonlinear interconnect safety-critical systems with asymmetric input constraints and security constraints, we need to define the performance function as:

$$J_i\left(x_i\right)={\int }_t^{\infty }{e}^{-{\alpha }_i\left(\tau -t\right)}\left({\iota }_i+\mathsf{\Theta }\left(x_i,u_i,v_i\right)\right)d\tau ,$$

(7)

where ${\alpha }_i$ is the discount factor, ${\iota }_i\left(x_i\right)=h_i{\beta }_j^2\left(x_i\right)$ and $\mathsf{\Theta }\left(x_i,u_i,v_i\right)=x_i^T{H}_i{x}_i+{\mathfrak{W}}_i\left(u_i\right)+{\xi }_i{v}_i^T{v}_i$ with $H_i$ and ${\mathfrak{W}}_i\left(u_i\right)$ are positive definite functions, where $h_i$ and ${\xi }_i$ are positive design parameters.

Remark 2.

Due to accounting for safety constraints and asymmetric input constraints in (7), the optimal control law does not converge to zero while the system state achieves the stable phase [43]. The discount factor ${\alpha }_i=0$, $J_i\left(x_i\right)$ may be unbounded, so it is necessary to consider the discount factor.

Problem 1.

(Decentralized control problems with security constraints and asymmetric input constraints) Consider the safety-critical system (1) and find the policy $u_i(.)$ and auxiliary control strategy $v_i(.):{\mathbb{R}}^{n_i}\to {\mathbb{R}}^{m_i}$ in the ith subsystem. The performance function is given by (7) with the ith subsystem state $x_i={[x_{i,1},\dots ,x_{i,k}]}^T$ and the control input $u_i$ satisfying the following conditions:

$$\begin{array}{cc}\hfill u_{i,min}& \le u_{i,\mathfrak{j}}\le u_{i,max},\left|u_{i,min}\right|\ne \left|u_{i,max}\right|,\hfill \end{array}$$

(8)

$$x_{i,k}\in (a_{i,k},A_{i,k}),\forall k=1,\dots ,n_i.$$

(9)

Ensure that the security-critical system state is consistently within the security constraints. Further, the definitions of some barrier functions are given.

Definition 1

(Barrier function [9,10]). The function $B\left(·\right):\mathbb{R}\to \mathbb{R}$ defined on interval (a, A) is referred to as the barrier function if

$$B\left(z;a,A\right)={log}_{}\frac{A\left(a-z\right)}{a\left(A-z\right)},\forall z\in \left(a,A\right),$$

(10)

where a and A are two constants satisfying $a<A$. Moreover, the potential function is invertible on the interval $(a,A)$, i.e.,

$$B^{-1}\left(y;a,A\right)=\frac{aA\left(e^{\frac{y}{2}}-e^{-\frac{y}{2}}\right)}{a{e}^{\frac{y}{2}}-A{e}^{-\frac{y}{2}}},\forall y\in \mathbb{R}.$$

(11)

Furthermore, the derivative of (11) is

$$\frac{\mathrm{d}{B}^{-1}\left(y;a,A\right)}{\mathrm{d}y}=\frac{A{a}^2-a{A}^2}{a^2{e}^y-2aA+A^2{e}^{-y}}.$$

(12)

Based on Definition 1, we consider the state transition based on the potential barrier function as follows:

$$\begin{array}{c}\hfill s_{i,k}=B\left(x_{i,k};a_{i,k},A_{i,k}\right),\end{array}$$

(13)

$$x_{i,k}=B^{-1}\left(s_{i,k};a_{i,k},A_{i,k}\right),$$

(14)

where $k=1,2,\dots ,n_i$. So, the $x_{i,k}$’s derivative concerning t is $\frac{\mathrm{d}{x}_{i,k}}{\mathrm{d}t}=\frac{\mathrm{d}{x}_{i,k}}{\mathrm{d}{s}_{i,k}}\frac{\mathrm{d}{s}_{i,k}}{\mathrm{d}t}$, and after using Definition 1, we obtain:

$$\begin{array}{cc}\hfill {\dot{s}}_{i,k}=& \frac{a_{i,k+1}{A}_{i,k+1}\left(e^{\frac{s_{i,k+1}}{2}}-e^{-\frac{s_{i,k+1}}{2}}\right)}{a_{i,k+1}e\frac{s_{i,k+1}}{2}-A_{i,k+1}e-\frac{s_{i,k+1}}{2}}\times \frac{A_{i,k}^2{e}^{-s_{i,k}}-2a_{i,k}{A}_{i,k}+a_{i,k}^2{e}^{s_{i,k}}}{A_{i,k}{a}_{i,k}^2-a_{i,k}{A}_{i,k}^2}\hfill \\ \hfill =& F_{ik}\left(s_{i,k},s_{i,k+1}\right),k=1,\dots ,n_i-1,\hfill \\ \hfill {\dot{s}}_{i,n_i}=& {\dot{x}}_i\times \frac{A_{i,k}^2{e}^{-s_{i,k}}-2a_{i,k}{A}_{i,k}+a_{i,k}^2{e}^{s_{i,k}}}{A_{i,k}{a}_{i,k}^2-a_{i,k}{A}_{i,k}^2}\hfill \\ \hfill =& F_{i,n_i}\left(s_{i,n_i}\right)+G_{i,n_i}\left(s_{i,n_i}\right)u_{i,n_i}+Y_{i,n_i}\left(s_{i,n_i}\right),\hfill \end{array}$$

where

$$\begin{array}{cc}\hfill F_{i,n_i}\left(s_i\right)& =\frac{a_{i,n_i}^2{e}_{i,n_i}^{s_i}-2a_{i,n_i}{A}_{i,n}+A_{i,n_i}{e}^{-s_{i,n_i}}}{A_{i,n_i}{a}_{i,n_i}^2-a_{i,n_i}{A}_{i,n_i}^2}\times f_i\left([B_{i,1}^{-1}\left(s_{i,1}\right)\dots B_{i,n_i}^{-1}\left(s_{i,n_i}\right)]\right),\hfill \\ \hfill G_{i,n_i}\left(s_i\right)& =\frac{a_{i,n_i}^2{e}_{i,n_i}^{s_i}-2a_{i,n_i}{A}_{i,n_i}+A_{i,n_i}{e}^{-s_{i,n_i}}}{A_{i,}{a}_{i,n_i}^2-a_{i,n_i}{A}_{i,n_i}^2}\times g_i\left([B_{i,1}^{-1}\left(s_{i,1}\right)\dots B_{i,n_i}^{-1}\left(s_{i,n_i}\right)]\right),\hfill \end{array}$$

and $Y_{i,n_i}\left(s_{i,n_i}\right)$ is the interconnection term of the $n_i$th term in the ith subsystem.

Then, the interconnected nonlinear safety-critical system (1) can be rewritten as:

$$\dot{s_i}=F_i\left(s_i\right)+G_i\left(s_i\right)u_i\left(t\right)+Y_i\left(s_i\right),$$

(15)

where $F_i\left(s_i\right)={[F_{i1}(s_{i,1},s_{i,2}),\dots ,F_{i,n_i}\left(s_i\right)]}^T$, $G_i\left(s_i\right)={[0,\dots ,G_{i,n_i}\left(s_i\right)]}^T$ and $Y_i\left(s_i\right)$ is the unknown interconnected term.

Based on Assumption 1, we define the unknown interconnection term after the system transformation as:

$$Y_i\left(s_i\right)={\wp }_i\left(s_i\right)U_i\left(s\right),$$

(16)

where ${\wp }_i\left(s_i\right)={[{\wp }_{1,n_1}\left(s_1\right),0,\dots ,0]}^T$, and

$${\wp }_{1,n_1}\left(s_1\right)=\frac{a_{i,2}^2{e}^{s_{i,2}}-2a_{i,2}{A}_{i,2}+A_{i,2}{e}^{-s_{i,2}}}{A_{i,2}{a}_{i,2}^2-a_2{A}_{i,2}^2}\times {\eta }_{1,n_1}\left(x_1\right),$$

and $U_i\left(s_i\right)$ is a bounded vector function that satisfies

$$\begin{array}{c}\hfill ∥U_i\left(s\right)∥\le \sum _{j=1}^n{b}_{i,j}{\vartheta }_{i,j}\left(s_j\right),\end{array}$$

(17)

where ${\vartheta }_{i,j}\left(s_j\right)$ is a positive definite function. Then, assuming ${\vartheta }_j\left(s_j\right)=ma{x}_{1\le i\le n}\left\{{\vartheta }_{i,j}\left(s_j\right)\right\}$ and ${\vartheta }_j\left(s_j\right)={[{\vartheta }_{j,1}(s_{j,1},s_{j,2}),\dots ,{\vartheta }_{j,n_i}\left(s_j\right)]}^T$, where

$${\vartheta }_{j,n_i}\left(s_j\right)=\frac{a_{j,n_i}^2{e}_{j,n_i}^{s_j}-2a_{j,n_i}{A}_{j,n_i}+A_{j,n_i}{e}^{-s_{j,n_i}}}{A_{j,n_i}{a}_{j,n_i}^2-a_{j,n_j}{A}_{j,n_i}^2}\times {\beta }_j\left([B_{j,1}^{-1}\left(x_1\right)\dots B_{j,n_i}^{-1}\left(x_j\right)]\right).$$

(18)

According to (3) and (18), the inequality (17) is expressed as:

$$∥U_i\left(s\right)∥\le \sum _{j=1}^n{S}_{i,j}{\vartheta }_j\left(s_j\right),$$

(19)

where $S_{i,j}\ge \left(b_{i,j}{\vartheta }_{i,j}\left(s_j\right)\right)/{\vartheta }_j\left(s_j\right)$ is a positive constant, and $i,j=1,2,\dots ,n$.

Assumption 3.

$F_i\left(s_i\right)$ is Lipschitz continuous with $F_i\left(0\right)=0$, ${\mathcal{P}}_i\left(0\right)=0$, $G_i\left(s_i\right)$ and ${\wp }_i\left(s_i\right)$ are upper-bounded, then $∥F_i\left(s_i\right)∥\le f_{i,m_i}∥s_i∥$, $∥G_i\left(s_i\right)∥\le g_{i,m_i}$, and $∥{\wp }_i\left(s_i\right)∥\le {\eta }_{i,m_i}$, $∥U_i\left(s_i\right)∥\le {\mathcal{P}}_{i,m_i}∥s_i∥$, where $f_{i,m_i}$, $g_{i,m_i}$, ${\eta }_{i,m_i}$, ${\mathcal{P}}_{i,m_i}$ are positive constants. $rank\left(G_i\left(s_i\right)\right)=m_i$ and $G_i^T\left(s_i\right){\wp }_i\left(s_i\right)=0$. Moreover, the modified system (15) is within the manageable range, and $s_i=0$ is the balance point for (15).

Lemma 1

([32]). $\forall \left(s_1,s_2\right)\in {\mathbb{R}}^2$, we have the following condition,

$$\begin{array}{c}\hfill s_1{s}_2\le \frac{{\epsilon }_1{}^{p_1}}{p_1}|s_1{|}^{p_1}+\frac{1}{p_2{\epsilon }_1{}^{p_{}2}}{\left|s_2\right|}^{p_2},\end{array}$$

where ${\epsilon }_1>0,(p_1-1)(p_2-1)=1$ and $p_1,p_2>1$.

Remark 3.

The barrier function in Definition 1, which has the following characteristics, ensures that the safety-critical system (15) always satisfies the safety constraints [9,10].

1. 

The state $s_i$ of the system is restricted to be bounded, so the system state $x_i$ satisfies constraints (8) and (9), i.e.,

$$\begin{array}{c}\hfill \left|B(z_i;a_i,A_i)\right|<+\infty ,\forall z_i\in (a_i,A_i).\end{array}$$

2. 

When the system’s state approaches the boundary of the safety area, the barrier function changes as follows:

$$\begin{array}{c}\hfill \underset{z_i\to a_i^{+}}{lim}B(z_i;a_i,A_i)=-\infty ,\\ \hfill \underset{z_i\to A_i^{-}}{lim}B(z_i;a_i,A_i)=+\infty .\end{array}$$

3. 

The barrier function fails to function when the system state reaches equilibrium, i.e.,

$$\begin{array}{c}\hfill B(0;a_i,A_i)=0,\forall a_i<A_i.\end{array}$$

3. Decentralized Optimal DSC Design

This section consists of two main subsections to establish the decentralized optimal DSC method. First, the security constraint problem is dealt with through the systematic transformation of the barrier function and the HJB equation for the ith auxiliary subsystem without security constraints is developed by introducing the improved performance function. Finally, the decentralized safety controller is constructed by solving the HJB equation for the auxiliary subsystem.

3.1. Barrier Function Conversion

According to the ith subsystem (15) described, the ith auxiliary subsystem is designed as:

$$\dot{s_i}=F_i\left(s_i\right)+G_i\left(s_i\right)u_i+\left(I_{n_i}-G_i\left(s_i\right)G_i^{+}\left(s_i\right)\right){\wp }_i\left(s_i\right)v_i,$$

(20)

where $G_i^{+}\left(s_i\right)\in {\mathbb{R}}^{m_i\times n_i}$ is Moore–Penrose pseudo-reverse. According to Assumptions 2 and 3, the matrix if found to be $G_i^{+}\left(s_i\right)={\left(G_i^T\left(s_i\right)G_i\left(s_i\right)\right)}^{-1}{G}_i^T\left(s_i\right)$ and $G_i^{+}\left(s_i\right){\wp }_i\left(s_i\right)={\left(G_i^T\left(s_i\right)G_i\left(s_i\right)\right)}^{-1}{G}_i^T\left(s_i\right){\wp }_i\left(s_i\right)=0$. Then, the auxiliary subsystem (20) is rewritten as:

$$\dot{s_i}=F_i\left(s_i\right)+G_i\left(s_i\right)u_i+{\wp }_i\left(s_i\right)v_i.$$

(21)

Regarding the converted system (15), analogous to (7), the performance function below is introduced:

$$V_i\left(s_i\right)={\int }_t^{\infty }{e}^{-{\alpha }_i\left(\tau -t\right)}\left({\pi }_i+\gamma \left(s_i,u_i,v_i\right)\right)d\tau ,$$

(22)

where ${\pi }_i\left(s_i\right)=h_i{\vartheta }_j^2\left(s_i\right)$ and $\gamma \left(s_i,u_i,v_i\right)=s_i^T{Q}_i{s}_i+{\mathfrak{W}}_i\left(u_i\right)+{\xi }_i{v}_i^T{v}_i$, $Q_i$ is the positive definition matrix. Furthermore, $s_{i0}=s_i\left(0\right)$ denotes the initial state, and ${\mathfrak{W}}_i\left(u_i\right)$ is a non-quadratic utility function that solves the asymmetric input constraint. Then, ${\mathfrak{W}}_i\left(u_i\right)$ is defined in the following form:

$${\mathfrak{W}}_i\left(u_i\right)=\sum _{j=1}^{m_i}2{\lambda }_i{\int }_{c_i}^{u_{i,j}}{\mathsf{\Psi }}^{-1}((v_i-c_i)/{\lambda }_i)\mathrm{d}{v}_i,$$

(23)

where ${\lambda }_i=({\mathfrak{h}}_{imax}-{\mathfrak{h}}_{imin})/2$ and $c_i=({\mathfrak{h}}_{imax}+{\mathfrak{h}}_{imin})/2$, and ${\mathsf{\Psi }}_i(.)$ represent the monotonic odd function, where ${\mathsf{\Psi }}_i\left(0\right)=0$. In this paper, without sacrificing generality, ${\mathsf{\Psi }}_i\left(s_i\right)=(e^{s_i}-e^{-s_i})/(e^{s_i}+e^{-s_i})$.

Remark 4.

Unlike the traditional form of symmetric input constraints [35], this article considered asymmetric constraints on the controlling inputs [44]. The revised hyperbolic tangent function presented in (22) effectively transforms the asymmetric constrained control problem into an unconstrained control problem by devising different maximum and minimum bounds.

Problem 2.

(Optimal decentralized control problems with asymmetric input constraints) Finding the control policy $u_i$ and auxiliary control strategy $v_i$ in the ith subsystem, the performance function becomes (22).

Based on the subsystem (21), as well as the performance function (22), the corresponding Hamiltonian is given by:

$$\begin{array}{cc}\hfill H(s_i,u_i,v_i,\nabla V_i\left(s_i\right))=& {(\nabla V_i\left(s_i\right))}^T(F_i\left(s_i\right)+G_i\left(s_i\right)u_i\left(t\right)+{\wp }_i\left(s_i\right)v_i)\hfill \\ \hfill & +{\pi }_i+\gamma \left(s_i,u_i,v_i\right)-{\alpha }_i{V}_i,\hfill \end{array}$$

(24)

with $\nabla V_i\left(s_i\right)=\frac{\partial V_i\left(s_i\right)}{\partial s_i}$.

The optimal performance function is

$$V_i^{*}\left(s_i\right)=\underset{u_i,v_i\in \mathsf{\Psi }\left({\mathsf{\Omega }}_i\right)}{min}{V}_i\left(s_i\right),$$

(25)

where $\mathsf{\Psi }\left({\mathsf{\Omega }}_i\right)$ is a collection of all acceptable control policies and auxiliary control strategies for ${\mathsf{\Omega }}_i$.

Based on Bellman’s optimality principle [31], $V_i^{*}\left(s_i\right)$ in (25) satisfies the HJB

$$\underset{u_i,v_i\in \mathsf{\Psi }\left({\mathsf{\Omega }}_i\right)}{min}H(s_i,u_i,v_i,\nabla V_i^{*}\left(s_i\right))=0,$$

(26)

where $\nabla V_i^{*}\left(s_i\right)=\frac{\partial V_i^{*}\left(s_i\right)}{\partial s_i}$. Then, the optimal control policy and the auxiliary control policy can be derived as follows:

$$u_i^{*}(s_i)=-{\lambda }_{i}tanh(\frac{1}{2{\lambda }_i}{G}_i^T(s_i)\nabla V_i^{*}(s_i))+c_i,$$

(27)

$$v_i^{*}\left(s_i\right)=-\frac{1}{2{\xi }_i}{\wp }_i^T\nabla V_i^{*}\left(s_i\right),$$

(28)

where $c_i=[c_1,\dots ,c_{m_i}]$.

Substituting $u_i^{*}\left(s_i\right)$ and $v_i^{*}\left(s_i\right)$ into (26), the HJB equation is rewritten as:

$$\begin{array}{cc}\hfill {(\nabla V_i^{*}\left(s_i\right))}^T{F}_i\left(s_i\right)+{(\nabla V_i^{*}\left(s_i\right))}^T{G}_i\left(s_i\right)u_i^{*}\left(s_i\right)-& {\xi }_i{∥v_i^{*}\left(s_i\right)∥}^2\hfill \\ \hfill -{\alpha }_i{V}_i^{*}+{\pi }_i\left(s_i\right)+s_i^T{Q}_i{s}_i+{\mathfrak{W}}_i\left(u_i^{*}\left(s_i\right)\right)& =0,\hfill \end{array}$$

(29)

with $V_i^{*}\left(0\right)=0$.

Through the BF-based system transformation, the decentralized control problem 1 with asymmetric input constraints and security constraints is transformed into an unconstrained optimization problem, i.e., the decentralized control problem 2. Next, the following lemma is discussed to ensure the equivalence between the decentralized control problems 1 and 2.

Lemma 2.

Assume that Assumptions 1 to 3 are met and that control policy $u_i(·)$ and auxiliary control strategy $v_i(·)$ solve the decentralized control problem 2 of (21). It follows, then, that the below holds:

1. 

If the initial state $x_0$ of the interconnected nonlinear safety-critical system (1) is in the range ($a_{i,k}$, $A_{i,k}$), $\forall k=1,2,\dots ,n_i$, then the closed-loop system satisfies (6).

2. 

If the functions $H_i\left(x\right)$ and $Q_i\left(x\right)$ satisfies the condition $H_i\left(x_i\right)=Q_i\left(B_i\left(x_i\right)\right)=Q_i\left(s_i\right)$, the performance described in (22) is equivalent to the one in (7).

Proof. 

Both the performance function and Assumption 3 satisfy the observability of zero states, guaranteeing the presence of the safety-optimal performance function $V_i^{*}\left(s_i\right)$. From (24), we obtain $\nabla V_i^{*}\left(t\right)\le 0$, which allows us to obtain $V_i^{*}\left(s_i\left(t\right)\right)\le V_i^{*}\left(s_i\left(0\right)\right)$ for all $t\ge 0$. Consequently, as stated in Remark 3, if the initial state $x_i\left(0\right)$ of the system (21) satisfies the security constraint (6), and $V_i^{*}\left(s_i\left(0\right)\right)$ is bounded, then the $V_i^{*}\left(s_i\left(t\right)\right)$ is also bounded. Finally, we obtain

$$x_{i,k}\left(t\right)\in (a_{i,k},A_{i,k}),k=1,2,\dots ,n_i.$$

(30)

Therefore, the given $u_i^{*}$ and $v_i^{*}$ satisfy the constraints of the decentralized control problem 1.

Now, consider the state transition based on the barrier function described in (13) and (14). Since $x_i$ satisfies the constraints given in (8), each element of the state $s_i={[B_{i,1}\left(x_{i,1}\right),\dots ,B_{i,k}\left(x_{i,k}\right)]}^T$ is finite. By comparing the performance functions (7) and (22), the equivalence relation $J_i\left(x_i\left(0\right)\right)=V_i\left(s_i\left(0\right)\right)$ is obtained, provided that $H_i\left(x_i\right)=Q_i\left(s_i\right)$. This completes the proof. □

3.2. Designing the Optimal DSC Strategy by Solving n HJB Equations

Throughout this section, we show that the optimal DSC strategies for interconnected nonlinear systems can be constructed by solving the n HJB equations.

Theorem 1.

Consider n subsystems under Assumptions 1 to 3 with DSC policies $u_i^{*}\left(s_i\right)$ and auxiliary control strategies $v_i^{*}\left(s_i\right)$, having the corresponding conditions as below:

$${∥v_i^{*}\left(s_i\right)∥}^2<s_i^T{Q}_i{s}_i,t\ge t_0.$$

(31)

Next, consider n positive constants $h_i^{*},i=1,2,\dots ,n$, so that for anything $h_i\ge h_i^{*}$, the optimal DSC policies $u_1^{*}\left(s_1\right)$, $u_2^{*}\left(s_2\right)$, …, $u_n^{*}\left(s_n\right)$ guarantee that the interconnected nonlinear system (15) with security constraints is UUB.

Proof. 

The Lyapunov candidacy function $L_{i,1}\left(s\right)$ below was selected:

$$L_{i,1}\left(s\right)=\sum _{i=1}^n{V}_i^{*}\left(s_i\right),$$

(32)

where the $V_i^{*}\left(s_i\right)$ is defined in the same way as (22), and we denote the time derivative along the trajectory $\dot{s_i}=F_i\left(s_i\right)+G_i\left(s_i\right)u_i\left(t\right)+Y_i\left(s_i\right)$ as:

$${\dot{L}}_{i,1}\left(s\right)=\sum _{i=1}^n{(\nabla V_i^{*})}^T(G_i\left(s_i\right)u_i^{*}+F_i\left(s_i\right)+Y_i\left(s\right)).$$

(33)

By using (27) and (28), we obtain:

$${(\nabla V_i^{*}(s_i))}^T{G}_i(s_i)=-2{\lambda }_i{tanh}^{-T}(\frac{u_i^{*}-c_i}{{\lambda }_i}),$$

(34)

$${(\nabla V_i^{*}\left(s_i\right))}^T{\wp }_i\left(s_i\right)=-2{\xi }_i{\left(v_i^{*}\left(s_i\right)\right)}^T.$$

(35)

Inserting (29), (34) and (35) into (33), we have

$$\begin{array}{c}\hfill {\dot{L}}_{i,1}\left(s\right)=\sum _{i=1}^n[{\alpha }_i{V}_i^{*}-{\pi }_i\left(s_i\right)-s_i^T{Q}_i{s}_i-{\mathfrak{W}}_i\left(u_i^{*}\right)+{\xi }_i{∥v_i^{*}\left(s_i\right)∥}^2-2{\xi }_i{\left(v_i^{*}\left(s_i\right)\right)}^T{U}_i\left(s\right)].\end{array}$$

(36)

According to the optimal DSC policy (27), the term ${\mathfrak{W}}_i\left(u_i^{*}\right)$ becomes

$${\mathfrak{W}}_i(u_i^{*}(s_i))=2{\lambda }_i\sum _{j=1}^{m_j}{\int }_0^{u_{i,j}^{*}-c_i}{tanh}^{-1}(\frac{u_i-c_i}{{\lambda }_i})d(u_i-c_i).$$

(37)

By appealing to the proof in [44], Equation (37) can be further reduced to

$$\begin{array}{cc}\hfill & {\mathfrak{W}}_i(u_i^{*}(s_i))=\underset{{\beta }_1}{\underbrace{{\lambda }_i^2\sum _{i=1}^{m_i}({tanh}^{-1}(\frac{u_{i,j}^{*}-c_i}{{\lambda }_i}))}}\hfill \\ \hfill & -\underset{{\beta }_2}{\underbrace{2{\lambda }_i^2\sum _{j=1}^{m_i}{\int }_0^{{tanh}^{-1}(\frac{u_{i,j}^{*}-c_i}{{\lambda }_i})}(u_i-c_i){tanh}^2(u_i-c_i)d(u_i-c_i)}},\hfill \end{array}$$

(38)

replacing (38) into (36), one has

$$\begin{array}{cc}\hfill {\dot{L}}_{i,1}\left(s\right)\le & -\sum _{i=1}^n\left(2{\xi }_i(s_i^T{Q}_i{s}_i-{∥v_i^{*}\left(s_i\right)∥}^2)\right)-\sum _{i=1}^n(1-2{\xi }_i)\left(s_i^T{Q}_i{s}_i\right)-\sum _{i=1}^n({\pi }_i\left(s_i\right)\hfill \\ \hfill & -2{\xi }_i\sum _{j=1}^{m_i}∥v_i^{*}\left(s_i\right)∥b_{i,j}{\vartheta }_{i,j}\left(s_j\right)+{\xi }^2{∥v_i^{*}\left(s_i\right)∥}^2)+{\alpha }_i{V}_i^{*}-{\beta }_1+{\beta }_2.\hfill \end{array}$$

(39)

It is known from [45] that there is a positive constant ${\delta }_{i,M}$ such that $0\le ∥\nabla V_i^{*}\left(s_i\right)∥\le {\delta }_{i,M}$. Therefore, using Lemma 1, Assumption 1, (17), (19), and (27), we obtain

$$\begin{array}{cc}\hfill 2{\beta }_1& \le 2{\lambda }_i^2{tanh}^{-T}(\frac{u_{i,j}^{*}-c_i}{{\lambda }_i}){tanh}^{-1}(\frac{u_{i,j}^{*}-c_i}{{\lambda }_i})\hfill \\ \hfill & =\frac{1}{2}{(\nabla V_i^{*}(s_i))}^T{G}_i(s_i)G_i^T(s_i)(\nabla V_i^{*}(s_i))\hfill \\ \hfill & \le \frac{1}{2}{G}_{i,m}^2{\delta }_{i,m}^2,\hfill \end{array}$$

(40)

Utilizing the integral median theorem [46] and the inequality (40), the ${\beta }_2$ (38) can be deduced as:

$$\begin{array}{cc}\hfill {\beta }_2& =2{\lambda }_i^2\sum _{j=1}^{m_i}{tanh}^{-1}(\frac{u_{i,j}^{*}-c_i}{{\lambda }_i}){\varpi }_i{tanh}^{-2}{\varpi }_i\hfill \\ \hfill & \le 2{\lambda }_i^2\sum _{j=1}^{m_i}{tanh}^{-1}(\frac{u_{i,j}^{*}-c_i}{{\lambda }_i}){\varpi }_i\hfill \\ \hfill & \le 2{\lambda }_i^2{tanh}^{-T}(\frac{u_{i,j}^{*}-c_i}{{\lambda }_i}){tanh}^{-1}(\frac{u_{i,j}^{*}-c_i}{{\lambda }_i})\hfill \\ & \le \frac{1}{2}{G}_{i,m}^2{\delta }_{i,m}^2,\hfill \end{array}$$

(41)

where ${\varpi }_i\in (0,{tanh}^{-1}(\frac{u_{i,j}^{*}-c_i}{{\lambda }_i}))$.

From [27], we conclude that $∥{\alpha }_i{V}_i^{*}(s_i)∥\le {\varrho }_{i,m}$, where ${\varrho }_{i,m}$ is a positive constant. Then, plugging (40) and (41) into (39), and taking into consideration the conclusion mentioned above, we can rephrase inequality (39) as follows:

$$\begin{array}{cc}\hfill & {\dot{L}}_{i,1}(s)\le -\sum _{i=1}^n(2{\xi }_i(s_i^T{Q}_i{s}_i-{∥v_i^{*}(s_i)∥}^2))-\sum _{i=1}^n(1-2{\xi }_i)(s_i^T{Q}_i{s}_i)\hfill \\ \hfill & -\sum _{i=1}^n(h_i{\vartheta }_i{(s_j)}^2-2{\xi }_i\sum _{j=1}^{m_i}∥v_i^{*}(s_i)∥b_{i,j}{\vartheta }_{i,j}(s_j)+{\xi }^2{∥v_i^{*}(s_i)∥}^2)+{\varrho }_i+\frac{1}{4}\sum _{i=1}^n{G}_{i,m}^2{\delta }_{i,m}^2,\hfill \end{array}$$

(42)

by denoting $\mathsf{\Lambda }=diag\left\{h_1,h_2,\dots ,h_n\right\}$ and $Z=[{\vartheta }_1(s_1),\dots ,{\vartheta }_n(s_n),{\xi }_1∥v_1^{*}(s_1)∥,\dots ,$ ${\xi }_n∥v_n^{*}(s_n)∥]$. Let the condition (31) be satisfied, so we have

$$\begin{array}{c}\hfill {\dot{L}}_{i,1}(s)\le -\sum _{i=1}^n(1-2{\xi }_i)(s_i^T{Q}_i{s}_i)-Z^{T}XZ+{\varrho }_i+\frac{1}{4}\sum _{i=1}^n{G}_{i,m}^2{\delta }_{i,m}^2,\end{array}$$

(43)

with $X=\left[\begin{array}{cc}\mathsf{\Lambda }& A^T\\ A& I_n\end{array}\right]$ and $A=\left[\begin{array}{ccc}{b}_{11}& \cdots & b_{1n}\\ ⋮& \ddots & ⋮\\ b_{n1}& \cdots & b_{nn}\end{array}\right]$.

From the matrix X expression, positive definiteness is maintained by choosing a sufficiently large $\mathsf{\Lambda }$. In other words, there is $h_i^{*}>0$, such that $h_i>h_i^{*}$, ensuring $Z^{T}XZ>0$. Thus, the inequality (43) is further deduced as:

$$\begin{array}{c}\hfill {\dot{L}}_{i,1}(s)\le -\sum _{i=1}^n(1-2{\xi }_i){\lambda }_{min}(Q_i){∥s_i∥}^2+{\varrho }_i+\frac{1}{4}\sum _{i=1}^n{G}_{i,m}^2{\delta }_{i,m}^2.\end{array}$$

(44)

The inequality (44) means that ${\dot{L}}_{i,1}\left(s\right)<0$ whenever $s_i\left(t\right)$ lies outside the following set $N_{s_i}$:

$$N_{s_i}=\left\{s_i:∥s_i∥\le \sqrt{\frac{\frac{1}{4}{G}_{i,M}^2{\delta }_{i,M}^2+{\varrho }_i}{{\lambda }_{min}\left(Q_i\right)(1-2{\xi }_i)}}\right\}.$$

(45)

Based on Lyapunov’s extension theorem [47], it is shown that the optimal performance functions $V_i^{*}\left(s_i\right)$ guarantee that the interconnected nonlinear system (15) with asymmetric input constraints is UUB. Since the performance function (7) and (22) yield the same results, it can be shown that the optimal performance function $J_i^{*}\left(x_i\right)$ guarantees that the interconnected nonlinear safety-critical system (1) with security constraints and asymmetric input constraints is UUB. □

4. Critic Network for Approximation

The critic neural network is introduced in this section, with the aim of approximating the optimal performance function. Then, the evaluation network of the auxiliary subsystem (21) is used to construct the estimated optimal control strategy. According to [48], $V_i^{*}\left(s_i\right)$ is expressed as:

$$V_i^{*}\left(s_i\right)=W_{{\mathfrak{c}}_i}^T{\sigma }_{{\mathfrak{c}}_i}\left(s_i\right)+{\epsilon }_{{\mathfrak{c}}_i}\left(s_i\right),$$

(46)

where ${\sigma }_{{\mathfrak{c}}_i}\left(s_i\right)=\left[{\sigma }_{{\mathfrak{c}}_i,1}\left(s_i\right),{\sigma }_{{\mathfrak{c}}_i,2}\left(s_i\right),\dots ,{\sigma }_{{\mathfrak{c}}_i,N_i}\left(s_i\right)\right]\in {\mathbb{R}}^{N_i}$ denotes the activation function, $W_{{\mathfrak{c}}_i}\in {\mathbb{R}}^{N_i}$ denotes the ideal weight vector, $N_i$ denotes the number of neurons, and ${\epsilon }_{{\mathfrak{c}}_i}\left(s_i\right)\in {\mathbb{R}}^{N_i}$ is the reconstruction error of NN. The vector activation function ${\sigma }_{{\mathfrak{c}}_i,p}\left(s_i\right)$ is denoted as a continuously differentiable function, where $p=1,2,\dots ,N_i$. For $s_i\ne 0$, ${\left\{{\sigma }_{{\mathfrak{c}}_i,p}\left(s_i\right)\right\}}_{p=1}^{N_i}$ is linearly independent. Then, the derivative of $V_i^{*}\left(s_i\right)$ can be expressed as:

$$\nabla V_i^{*}\left(s_i\right)=\nabla {\sigma }_{{\mathfrak{c}}_i}^T\left(s_i\right)W_{{\mathfrak{c}}_i}+\nabla {\epsilon }_{{\mathfrak{c}}_i}\left(s_i\right),$$

(47)

where $\nabla {\sigma }_{{\mathfrak{c}}_i}\left(s_i\right)=\frac{\partial {\sigma }_{{\mathfrak{c}}_i}\left(s_i\right)}{\partial s_i}$ and $\nabla {\epsilon }_{{\mathfrak{c}}_i}\left(s_i\right)=\frac{\partial {\epsilon }_{{\mathfrak{c}}_i}\left(s_i\right)}{\partial s_i}$.

From Equations (27), (28) and (47), the optimal safety control policy $u_i^{*}\left(s_i\right)$ and the auxiliary control strategy $v_i^{*}\left(s_i\right)$ are rephrased as:

$$u_i^{*}(s_i)=-{\lambda }_{i}tanh(\frac{1}{2{\lambda }_i}{G}_i^T(s_i)\nabla {\sigma }_{{\mathfrak{c}}_{\mathfrak{i}}}^T(s_i)W_{{\mathfrak{c}}_{\mathfrak{i}}})+c_{d_i}+{\epsilon }_{u_i}(s_i),$$

(48)

$$v_i^{*}(s_i)=-\frac{1}{2{\xi }_i}{\wp }_i^T(s_i)\nabla {\sigma }_{{\mathfrak{c}}_i}^T(s_i)W_{{\mathfrak{c}}_i}+{\epsilon }_{v_i}(s_i),$$

(49)

where

$$\begin{array}{cc}\hfill {\epsilon }_{u_i}(s_i)& =-\frac{1}{2}(I_{m_i}-{tanh}^2(\zeta ))G_i^T(s_i)\nabla {\epsilon }_{{\mathfrak{c}}_i}(s_i),\hfill \\ \hfill {\epsilon }_{v_i}(s_i)& =-\frac{1}{2{\xi }_i}{\wp }_i^T(s_i)\nabla {\epsilon }_{{\mathfrak{c}}_i}(s_i),\hfill \end{array}$$

with $I_{m_i}={[1,1,\dots ,1]}^T\in {\mathbb{R}}^{m_i}$. The seclected value of $\zeta$ is between $\frac{1}{2{\lambda }_i}{G}_i^T\left(s_i\right)\nabla {\sigma }_{{\mathfrak{c}}_{\mathfrak{i}}}^T\left(s_i\right)W_{{\mathfrak{c}}_{\mathfrak{i}}}$ and $\frac{1}{2{\lambda }_i}{G}_i^T\left(s_i\right)(\nabla {\sigma }_{{\mathfrak{c}}_{\mathfrak{i}}}^T\left(s_i\right)W_{{\mathfrak{c}}_{\mathfrak{i}}}+\nabla {\epsilon }_{{\mathfrak{c}}_i}\left(s_i\right))$.

The ideal weight vector $W_{{\mathfrak{c}}_{\mathfrak{i}}}$ is not available and the optimal control strategy $u_i^{*}\left(s_i\right)$ is not directly applicable. Therefore, the estimated weight vector ${\widehat{W}}_{{\mathfrak{c}}_i}$ is constructed to replace $W_{{\mathfrak{c}}_{\mathfrak{i}}}$ as:

$${\widehat{V}}_i^{*}\left(s_i\right)={\widehat{W}}_{{\mathfrak{c}}_i}^T{\sigma }_{{\mathfrak{c}}_i}\left(s_i\right).$$

(50)

The estimation error ${\tilde{W}}_{{\mathfrak{c}}_i}=W_{{\mathfrak{c}}_i}-{\widehat{W}}_{{\mathfrak{c}}_i}$ is defined. Similarly, according to (50), the (49) and (48) are further developed as:

$${\widehat{u}}_i\left(s_i\right)=-{\lambda }_{i}tanh(\frac{1}{2{\lambda }_i}{G}_i^T\left(s_i\right)\nabla {\sigma }_{{\mathfrak{c}}_{\mathfrak{i}}}^T\left(s_i\right){\widehat{W}}_{{\mathfrak{c}}_{\mathfrak{i}}})+c_{d_i},$$

(51)

$${\widehat{v}}_i\left(s_i\right)=-\frac{1}{2{\xi }_i}{\wp }_i^T\left(s_i\right)\nabla {\sigma }_{{\mathfrak{c}}_i}^T\left(s_i\right){\widehat{W}}_{{\mathfrak{c}}_i}.$$

(52)

Combining (50), (51) and (52), the Hamiltonian is re-expressed as:

$$\begin{array}{cc}\hfill H(s_i,{\widehat{u}}_i,{\widehat{v}}_i,\nabla {\widehat{V}}_i\left(s_i\right))=& {(\nabla {\widehat{V}}_i\left(s_i\right))}^T(G_i^T\left(s_i\right){\widehat{u}}_i+F_i\left(s_i\right)+{\wp }_i\left(s_i\right){\widehat{v}}_i)\hfill \\ \hfill & +{\pi }_i\left(s_i\right)+{\gamma }_i(s_i,{\widehat{u}}_i,{\widehat{v}}_i)-{\alpha }_i{\widehat{V}}_i.\hfill \end{array}$$

(53)

According to (53), the error of the Hamiltonian is given by:

$$\begin{array}{cc}\hfill e_i& =H(s_i,{\widehat{u}}_i,{\widehat{v}}_i,\nabla {\widehat{V}}_i\left(s_i\right))-H(s_i,u_i^{*},v_i^{*},\nabla V_i^{*}\left(s_i\right))\hfill \\ \hfill & ={\pi }_i\left(s_i\right)+s_i^T{Q}_i{s}_i+{\mathfrak{W}}_i\left({\widehat{u}}_i\right)+{\xi }_i{\widehat{v}}_i^T{\widehat{v}}_i+{\widehat{W}}_{{\mathfrak{c}}_i}^T{\varrho }_i,\hfill \end{array}$$

(54)

with ${\varrho }_i=\nabla {\sigma }_{{\mathfrak{c}}_{\mathfrak{i}}}\left(x_i\right)(G_i^T\left(s_i\right){\widehat{u}}_i+F_i\left(s_i\right)+{\wp }_i\left(s_i\right){\widehat{v}}_i)-{\alpha }_i{\sigma }_{{\mathfrak{c}}_{\mathfrak{i}}}\left(s_i\right)$. In order to make $u_i\left(s_i\right)\to u_i^{*}\left(s_i\right)$, the error $e_i$ should be guaranteed to be sufficiently small. To solve this issue, a critic weight adjustment law ${\widehat{W}}_{{\mathfrak{c}}_i}$ is proposed to minimize the objective function ${\varphi }_i=\frac{1}{2}{e}_i^T{e}_i$. Next, the critic updating law is developed as:

$${\widehat{W}}_{{\mathfrak{c}}_i}=-\frac{{\alpha }_{{\mathfrak{c}}_i}{\varrho }_i{e}_i}{{(1+{\varrho }_i^T{\varrho }_i)}^2},$$

(55)

where the constant ${\alpha }_{{\mathfrak{c}}_i}$ is the positive learning rate.

Remark 5.

To minimize the Hamiltonian error $e_i$, it is necessary to maintain the derivative of ${\varphi }_i$ as ${\dot{\varphi }}_i<0$. Therefore, the critic weight adjustment law is derived by employing the normalization term ${(1+{\varrho }_i^T{\varrho }_i)}^{-2}$ and applying the gradient descent method with respect to ${\widehat{W}}_{{\mathfrak{c}}_i}$ [49].

By considering the definition of ${\tilde{W}}_{{\mathfrak{c}}_i}$, we obtain

$${\dot{\tilde{W}}}_{{\mathfrak{c}}_i}=-{\alpha }_{{\mathfrak{c}}_i}{\ell }_i{\ell }_i^T{\tilde{W}}_{{\mathfrak{c}}_i}+\frac{{\alpha }_{{\mathfrak{c}}_i}{\ell }_i{e}_{H_i}}{{𝚤}_i},$$

(56)

where ${\ell }_i=\frac{{\varrho }_i}{1+{\varrho }_i^T{\varrho }_i}$ and ${𝚤}_i=1+{\varrho }_i^T{\varrho }_i$. $e_{H_i}$ denotes the residual error, defined as $e_{H_i}=\nabla {\sigma }_{{\mathfrak{c}}_{\mathfrak{i}}}\left(x_i\right)(G_i^T\left(s_i\right){\widehat{u}}_i+F_i\left(s_i\right)+{\wp }_i\left(s_i\right){\widehat{v}}_i)$.

The proposed decentralized DSC strategy for the ith subsystem with a single critic-NN is illustrated in Figure 1.

Figure 1. The block diagram of the developed optimal DSC scheme.

5. Stability Analysis

This section focuses on the stability of the n-auxiliary subsystem for the given control scheme. We need to make some Assumptions to satisfy the theorem.

Assumption 4.

For $s_i\in {\mathsf{\Omega }}_i,i=1,\dots ,n$, there exist some positive constants $D_{{\epsilon }_{u_i}},{\eta }_{i,M},D_{{\sigma }_{{\mathfrak{c}}_i}},D_{{\epsilon }_{v_i}}$ and $D_{e_{H_i}}$ satisfying $∥{\epsilon }_{u_i}\left(s_i\right)∥\le D_{{\epsilon }_{u_i}}$, $∥{\wp }_i\left(s_i\right)∥\le {\wp }_{i,M}$, $∥\nabla {\sigma }_{{\mathfrak{c}}_i}\left(s_i\right)∥\le D_{{\sigma }_{{\mathfrak{c}}_i}},∥{\epsilon }_{v_i}\left(s_i\right)∥\le D_{{\epsilon }_{v_i}}$ and $∥e_{H_i}∥\le D_{e_{H_i}}$.

Assumption 5.

Consider the time period $\left[t,t+t_{\mathfrak{k}}\right]$ and $t_{\mathfrak{k}}>0$. Then, the term ${\ell }_i{\ell }_i^T$ fulfills the following condition:

$${ϵ}_i{I}_{N_i}\le {\ell }_i{\ell }_i^T\le {\mathfrak{s}}_i{I}_{N_i},$$

(57)

where ${ϵ}_i$ and ${\mathfrak{s}}_i$ are positive constants.

Theorem 2.

For the nonlinear interconnected safety-critical system (15), we design the estimated optimal safety policies and auxiliary control strategies as (51) and (52), respectively. Assume that Assumptions 1–5 hold. If ${\widehat{W}}_{{\mathfrak{c}}_i}$ is updated by (55), then $s_i$ and ${\widehat{W}}_{{\mathfrak{c}}_i}$ are UUB if ${\alpha }_{{\mathfrak{c}}_i}$ in (55) satisfies

$${\alpha }_{{\mathfrak{c}}_i}>\frac{{\wp }_{i,M}^2{D}_{{\sigma }_{{\mathfrak{c}}_i}}^2}{{\xi }_i{\lambda }_{min}\left({\ell }_i{\ell }_i^T\right)}.$$

(58)

Proof. 

The candidate Lyapunov function is considered to be:

$$L_i(t)=\sum _{i=1}^n(V_i^{*}(s_i)+\frac{1}{2}{\tilde{W}}_{{\mathfrak{c}}_i}^T{\tilde{W}}_{{\mathfrak{c}}_i}).$$

(59)

Then, defining $L_{i,1}(t)=V_i^{*}(s_i)$ and $L_{i,2}(t)=\frac{1}{2}{\tilde{W}}_{{\mathfrak{c}}_i}^T{\tilde{W}}_{{\mathfrak{c}}_i}$, the time derivative by $L_{i,1}(t)$ is

$$\begin{array}{cc}\hfill {\dot{L}}_{i,1}(t)=& {(\nabla V_i^{*}(s_i))}^T(G_i^T(s_i){\widehat{u}}_i+F_i(s_i)+{\wp }_i(s_i){\widehat{v}}_i)\hfill \\ \hfill =& {(\nabla V_i^{*}(s_i))}^T(G_i^T(s_i)u_i^{*}+F_i(s_i)+{\wp }_i(s_i)v_i^{*})\hfill \\ \hfill & +\underset{{\beta }_3}{\underbrace{{(\nabla V_i^{*}(s_i))}^T{G}_i^T(s_i)({\widehat{u}}_i-u_i^{*})}}+\underset{{\beta }_4}{\underbrace{{(\nabla V_i^{*}(s_i))}^T{\wp }_i(s_i)({\widehat{v}}_i-v_i^{*})}}.\hfill \end{array}$$

(60)

Combining (29), (34) and (35). The (60) is further deduced as:

$$\begin{array}{cc}\hfill {\dot{L}}_{i,1}(t)=& {\alpha }_i{V}_i^{*}-{\pi }_i(s_i)-s_i^T{Q}_i{s}_i-{\mathfrak{W}}_i(u_i^{*})+{\xi }_i{∥v_i^{*}(s_i)∥}^2+{\beta }_3+{\beta }_4.\hfill \end{array}$$

(61)

According to Lemma 1, and taking into account (40), (48), (51), we observe that the ${\beta }_3$ term in (61) is satisfied by

$$\begin{array}{cc}\hfill {\beta }_3& \le {\lambda }_i^2∥{tanh}^{-1}(\frac{u_i^{*}(s_i)-c_{d_i}}{{\lambda }_i})∥+{∥{\widehat{u}}_i-u_i^{*}∥}^2\hfill \\ \hfill & \le {\beta }_1+\underset{{\beta }_5}{\underbrace{{∥{\lambda }_i(tanh({\mathfrak{Y}}_{i,1}(s_i))-tanh({\mathfrak{Y}}_{i,2}(s_i)))-{\epsilon }_{u_i}(s_i)∥}^2}}\hfill \\ \hfill & \le \frac{1}{4}{G}_{i,M}^2{\delta }_{i,M}^2+{\beta }_5,\hfill \end{array}$$

(62)

where ${\mathfrak{Y}}_i(s_i)=\frac{1}{2{\lambda }_i}{G}_i^T(s_i)\nabla V_i^{*}(s_i)$. Then, based on the fact $∥tanh({\mathfrak{Y}}_{i,\mathfrak{k}}(s_i))∥\le \sqrt{m_i},$ $\mathfrak{k}=1,2$ in [44], according to Assumption 5, ${\beta }_5$ is derived as:

$$\begin{array}{cc}\hfill {\beta }_5& \le 2{\lambda }_i^2{∥tanh({\mathfrak{Y}}_{i,1}(s_i))-tanh({\mathfrak{Y}}_{i,2}(s_i))∥}^2+2{∥{\epsilon }_{u_i}(s_i)∥}^2\hfill \\ \hfill & \le 4{\lambda }_i^2({∥tanh({\mathfrak{Y}}_{i,1}(s_i))∥}^2+{∥tanh({\mathfrak{Y}}_{i,2}(s_i))∥}^2)+2{∥{\epsilon }_{u_i}(s_i)∥}^2\hfill \\ \hfill & \le 8{\lambda }_i^2{m}_i+2D_{{\epsilon }_{u_i}}^2.\hfill \end{array}$$

(63)

Similarly, the last term of (61) is deduced from (35), (49) and (52) as:

$$\begin{array}{cc}\hfill {\beta }_4& \le -{\xi }_i{∥v_i^{*}∥}^2-{\xi }_i{∥{\widehat{v}}_i-v_i^{*}∥}^2\hfill \\ \hfill & \le -{\xi }_i{∥v_i^{*}∥}^2+2{\xi }_i({∥{\widehat{v}}_i∥}^2-{∥v_i^{*}∥}^2)+2{\xi }_i{∥{\epsilon }_{v_i}∥}^2\hfill \\ \hfill & \le -{\xi }_i{∥v_i^{*}∥}^2+\frac{1}{2{\xi }_i}{\wp }_{i,M}^2{D}_{{\sigma }_{{\mathfrak{c}}_i}}^2{∥{\tilde{W}}_{{\mathfrak{c}}_i}∥}^2+2{\xi }_i{D}_{{\epsilon }_{v_i}}^2.\hfill \end{array}$$

(64)

By using (38), (62)–(64) and the fact that $∥{\alpha }_i{V}_i^{*}\left(s_i\right)∥\le {\varrho }_{i,M}$ the following is derived:

$$\begin{array}{c}\hfill {\dot{L}}_{i,1}\left(t\right)\le -{\lambda }_{min}\left(Q_i\right){∥s_{{}_i}∥}^2+\frac{1}{2{\xi }_i}{\wp }_{i,M}^2{D}_{{\sigma }_{{\mathfrak{c}}_i}}^2{∥{\tilde{W}}_{{\mathfrak{c}}_i}∥}^2+{\mathsf{\Theta }}_i,\end{array}$$

(65)

with ${\mathsf{\Theta }}_i={\varrho }_{i,M}+\frac{1}{2}{G}_{i,M}^2{\delta }_{i,M}^2+8{\lambda }_i^2{m}_i+2D_{{\epsilon }_{u_i}}^2+2{\xi }_i{D}_{{\epsilon }_{v_i}}^2$.

The error weight update law ${\tilde{W}}_{{\mathfrak{c}}_i}$. $L_{i,1}\left(t\right)$ is considered with the time derivative

$${\dot{L}}_{i,2}\left(t\right)=-{\alpha }_{{\mathfrak{c}}_i}{\tilde{W}}_{{\mathfrak{c}}_i}^T{\ell }_i{\ell }_i^T{\tilde{W}}_{{\mathfrak{c}}_i}+{\alpha }_{{\mathfrak{c}}_i}\frac{{\tilde{W}}_{{\mathfrak{c}}_i}^T{\ell }_i}{{𝚤}_i}{e}_{H_i}.$$

(66)

Combining Lemma 1 and Assumption 4, the following conclusion is drawn:

$${\alpha }_{{\mathfrak{c}}_i}\frac{{\tilde{W}}_{{\mathfrak{c}}_i}^T{\ell }_i}{{𝚤}_i}{e}_{H_i}\le \frac{{\alpha }_{{\mathfrak{c}}_i}}{2}{\tilde{W}}_{{\mathfrak{c}}_i}^T{\ell }_i{\ell }_i^T{\tilde{W}}_{{\mathfrak{c}}_i}+\frac{{\alpha }_{{\mathfrak{c}}_i}}{2}{D}_{e_{H_i}}^2.$$

(67)

Combining inequalities (66) and (67), we derive the following inequalities:

$${\dot{L}}_{i,2}\left(t\right)\le -\frac{{\alpha }_{{\mathfrak{c}}_i}}{2}{\lambda }_{min}\left({\ell }_i{\ell }_i^T\right){∥{\tilde{W}}_{{\mathfrak{c}}_i}∥}^2+\frac{{\alpha }_{{\mathfrak{c}}_i}}{2}{D}_{e_{H_i}}^2.$$

(68)

Substituting (65) and (68) into (59), the following inequality is obtained:

$${\dot{L}}_i(t)\le \sum _{i=1}^n(-{\lambda }_{min}(Q_i){∥s_i∥}^2-{\mathfrak{x}}_i{∥{\tilde{W}}_{{\mathfrak{c}}_i}∥}^2+{\mathsf{\Theta }}_i+\frac{{\alpha }_{{\mathfrak{c}}_i}}{2}{D}_{e_{H_i}}^2),$$

(69)

where ${\mathfrak{x}}_i=\frac{{\alpha }_{{\mathfrak{c}}_i}}{2}{\lambda }_{min}\left({\ell }_i{\ell }_i^T\right)-\frac{1}{2{\xi }_i}{\wp }_{i,M}^2{D}_{{\sigma }_{{\mathfrak{c}}_i}}^2$, ${\lambda }_{min}\left({\ell }_i{\ell }_i^T\right)$ means the minimum eigenvalue of ${\ell }_i{\ell }_i^T$.

Therefore, Equations (58) and (69) mean ${\dot{L}}_i\left(t\right)<0$, provided that the parameters $s_i$ and ${\tilde{W}}_{{\mathfrak{c}}_i}$ are not in the set of

$$N_i\left\{s_i:∥s_i∥\le \sqrt{\frac{2{\mathsf{\Theta }}_i+D_{e_{H_i}}^2}{2{\lambda }_{min}\left(Q_i\right)}}\right\},$$

(70)

$$N_{{\tilde{W}}_{{\mathfrak{c}}_i}}\left\{{\tilde{W}}_{{\mathfrak{c}}_i}:∥{\tilde{W}}_{{\mathfrak{c}}_i}∥\le \sqrt{\frac{2{\mathsf{\Theta }}_i+D_{e_{H_i}}^2}{{\mathfrak{x}}_i}}\right\}.$$

(71)

Introducing Lyapunov’s extension theorem, ref. [47], ensures the stability of the closed-loop system. This proof ensures that the weight estimation error ${\tilde{W}}_{{\mathfrak{c}}_i}$ is UUB. At this point, this completes the proof process. □

Remark 6.

In contrast to techniques that aim to achieve input saturation [10,13], this article proposes an RL technique to solve the optimal DSC problem with safety constraints and asymmetric input constraints. This approach ensures not only the safety of the system but also minimizes the input constraints. Therefore, the developed reinforcement learning technique, based on security constraints and asymmetric input constraints, is better suited for some project applications, particularly for systems where the system state must be globally within the security settings.

6. Simulation Example

In this section, we provide a simulation example to verify the effectiveness of the proposed approach. The simulation involved a dual-linked robotic arm system [42]. The state space model of the system is defined by

$$\begin{array}{cc}\hfill {\dot{x}}_{1,1}& =x_{1,2},\hfill \\ \hfill {\dot{x}}_{1,2}& =-\frac{M_1}{{\tilde{G}}_1}{x}_{1,2}-\frac{m_1\tilde{g}{\tilde{l}}_1}{{\tilde{G}}_1}sin\left(x_{1,1}\right)+\frac{1}{{\tilde{G}}_1}{u}_1+△h_1,\hfill \\ \hfill {\dot{x}}_{2,1}& =x_{2,2},\hfill \\ \hfill {\dot{x}}_{2,2}& =-\frac{M_2}{{\tilde{G}}_2}{x}_{2,2}-\frac{m_2\tilde{g}{\tilde{l}}_2}{{\tilde{G}}_2}sin\left(x_{2,1}\right)+\frac{1}{{\tilde{G}}_2}{u}_2+△h_2,\hfill \end{array}$$

(72)

where $x_{i,1}$ and $x_{i,2}$$(i=1,2)$ indicate the angular location of the robot arm, $u_i$ stands for control input, and the $△h_i={\eta }_i{\mathcal{P}}_i$ represents the interconnection terms. The other parameters of the robotic arm system (72) are depicted in Table 1. The initial system state was selected as $x_0={[2,2,2,2]}^T$. We first defined the state variable $x_i={[x_{i,1},x_{i,2}]}^T$ and constructed the internal dynamics and input gain matrix as follows:

$$f_i\left(x_i\right)=\left[\begin{array}{c}{x}_{i,2}\\ -\frac{M_i}{{\tilde{G}}_i}{x}_{i,2}-\frac{m_i\tilde{g}{\tilde{l}}_i}{{\tilde{G}}_i}sin\left(x_{i,1}\right)\end{array}\right]+\left[\begin{array}{c}0\\ \frac{1}{{\tilde{G}}_i}\end{array}\right]u_i+\left[\begin{array}{c}1\\ 0\end{array}\right]{\mathcal{P}}_i\left(x_i\right),$$

where ${\mathcal{P}}_1\left(x_1\right)$, ${\mathcal{P}}_2\left(x_2\right)$ denote the uncertain interconnection terms of subsystems 1 and 2, i.e.,

$$\begin{array}{cc}\hfill {\mathcal{P}}_1\left(x_1\right)& =0.1x_{1,1}sin\left(x_{2,2}\right),\hfill \\ \hfill {\mathcal{P}}_2\left(x_2\right)& =(x_{1,2}-3sin\left(0.1x_{2,1}\right)).\hfill \end{array}$$

Table 1. Meanings and values of symbols used in robotic arm systems.

The ith Subsystem	Parameter	Meaning	Value
	$m_1$	Mass of payload	5 kg
	$M_1$	Viscous friction	2 N
The first subsystem	${\tilde{l}}_1$	Length of the arm	0.5 m
	${\tilde{G}}_1$	Moment of inertia	10 kg
	${\tilde{g}}_1$	Acceleration of gravity	9.81 m/s
	$m_2$	Mass of payload	10 kg
	$M_2$	Viscous friction	2 N
The second subsystem	${\tilde{l}}_2$	Length of the arm	1 m
	${\tilde{G}}_2$	Moment of inertia	10 kg
	${\tilde{g}}_2$	Acceleration of gravity	9.81 m/s

Furthermore, the two robotic arm subsystems were in a state that satisfied the below security constraints:

$$\begin{array}{cc}\hfill & x_{1,1}\in (-0.5,2.9),x_{1,2}\in (-1.5,2.5),\hfill \\ \hfill & x_{2,1}\in (-1,2.5),x_{2,2}\in (-3.5,3).\hfill \end{array}$$

(73)

Therefore, to deal with the security constraint, the following system of transformations without security constraint was obtained, using the BF-based system transformation (13):

$$s_i=F_i\left(s_i\right)+G_i\left(s_i\right)u_i+{\wp }_i\left(s_i\right)U_i,$$

(74)

where

$$\begin{array}{cc}\hfill F_i\left(s_i\right)& =\left[\begin{array}{c}\frac{a_{i,2}{A}_{i,2}(e^{\frac{s_{i,2}}{2}}-e^{-\frac{s_{i,2}}{2}})}{a_{i,2}{e}^{\frac{s_{i,2}}{2}}-A_{i,2}{e}^{-\frac{s_{i,2}}{2}}}\frac{a_{i,1}^2{e}^{s_{i,1}}-2a_{i,1}{A}_{i,1}+A_{i,1}{e}^{-s_{i,1}}}{A_{i,1}{a}_{i,1}^2-a_1{A}_{i,1}^2}\\ f_i\left(B^{-1}\left(s_i\right)\right)\frac{a_{i,2}^2{e}^{s_{i,2}}-2a_{i,2}{A}_{i,2}+A_{i,2}{e}^{-s_{i,2}}}{A_{i,2}{a}_{i,2}^2-a_{i,2}{A}_{i,2}^2}\end{array}\right],\hfill \\ \hfill G_i\left(s_i\right)& =\left[\begin{array}{c}0\\ \frac{1}{\tilde{G}}\frac{a_{i,2}^2{e}^{s_{i,2}}-2a_{i,2}{A}_{i,2}+A_{i,2}{e}^{-s_{i,2}}}{A_{i,2}{a}_{i,2}^2-a_2{A}_{i,2}^2}\end{array}\right],\hfill \\ \hfill {\wp }_i\left(s_i\right)& =\left[\begin{array}{c}\frac{a_{i,2}^2{e}^{s_{i,2}}-2a_{i,2}{A}_{i,2}+A_{i,2}{e}^{-s_{i,2}}}{A_{i,2}{a}_{i,2}^2-a_2{A}_{i,2}^2}\\ 0\end{array}\right].\hfill \end{array}$$

(75)

For the transformed dual-linked robotic arm system (74), the initial state was chosen by $s_{i,0}={[s_{i,0}\left(1\right),s_{i,0}\left(2\right)]}^T={[B(x_{i,0}\left(1\right);a_{i,1},A_{i,1}),B(x_{i,0}\left(2\right);a_{i,2},A_{i,2})]}^T$. The discount factors were chosen as ${\alpha }_1=1$ and ${\alpha }_2=0.1$. The matrices were designed as $Q_1=0.5I^2$ and $Q_2=I^2$, $R_1=1$ and $R_2=1$. The upper and lower limits were allocated as below: ${\mathfrak{h}}_{1max}=0.75$, ${\mathfrak{h}}_{1min}=-0.25$ and ${\mathfrak{h}}_{2max}=1.5$, ${\mathfrak{h}}_{2min}=-0.5$. Let ${\vartheta }_1=∥s_1∥$ and ${\vartheta }_2=∥s_2∥$. Additional design factors were setup as below: ${\xi }_1=8,{\xi }_2=4,a_{{\mathfrak{c}}_{\mathfrak{1}}}=2,a_{{\mathfrak{c}}_{\mathfrak{2}}}=2$. Choose the activation functions ${\sigma }_{{\mathfrak{c}}_{\mathfrak{i}}}\left(s_i\right)={[s_{1,1}^2,s_{1,1}{s}_{1,2},s_{1,2}^2]}^T$ and ${\sigma }_{{\mathfrak{c}}_{\mathfrak{i}}}\left(s_i\right)={[s_{2,1}^2,s_{2,1}{s}_{2,2},s_{2,2}^2]}^T$.

The simulation outcomes are presented in Figure 2, Figure 3, Figure 4, Figure 5, Figure 6, Figure 7, Figure 8, Figure 9, Figure 10, Figure 11, Figure 12 and Figure 13. The states of the system are depicted in Figure 2 and Figure 8, and it can be observed that the closed-loop system stabilized after 20 s and 35 s, respectively. However, the system failed to meet the specified security constraints. Figure 3 and Figure 9, shown in comparison with Figure 2 and Figure 8, not only assured that the system states converged to zero, but also satisfied the given safety constraints. The evolving states $s_1\left(t\right)$ and $s_2\left(t\right)$ are presented in Figure 4 and Figure 10, based on the safe control method with asymmetric input constraints. The optimal DSC policies are shown in Figure 5 and Figure 11. We found that the optimal DSC policies were restricted to the asymmetric set $[-0.25,0.75]$ and $[-0.5,1.5]$. Figure 6 and Figure 12 represent the optimal auxiliary control strategies for subsystems 1 and 2, respectively. Figure 7 and Figure 13 show the critic updated laws. It can be observed that the weights converged after 15 s. According to Theorem 3, we concluded that the proposed optimal safety control policy and the auxiliary control policy could stabilize the closed-loop nonlinear system and satisfy the safety constraints on the system state. Moreover, the optimal control policy eventually converged to a predefined set of constraints. Finally, the results of the simulation showed that the presented optimal DSC solution for constrained interconnected nonlinear safety-critical systems, affected by system state constraints, is effective.

Figure 2. Evolution of state $x_1\left(t\right)$ without using the DSC method.

Figure 3. Evolution of state $x_1\left(t\right)$ using the DSC method.

Figure 4. Evolution of state $s_1\left(t\right)$ using the DSC method.

Figure 5. Control evolution of input $u_1$.

Figure 6. Evolution of the auxiliary control input $v_1$ using the DSC method.

Figure 7. Evolution of the critic weight vector $W_{{\mathfrak{c}}_1}$ using the DSC method.

Figure 8. Evolution of state $x_2\left(t\right)$ without using the DSC method.

Figure 9. Evolution of state $x_2\left(t\right)$ using the DSC method.

Figure 10. Evolution of state $s_2\left(t\right)$ using the DSC method.

Figure 11. Control evolution of input $u_2$.

Figure 12. Evolution of the auxiliary control input $v_2$ using the DSC method.

Figure 13. Evolution of the critic weight vector $W_{{\mathfrak{c}}_2}$ using the DSC method.

7. Conclusions

This article presents an RL-based DSC scheme for interconnected nonlinear safety-critical systems with security constraints and asymmetric input constraints. The proposed method transformed an interconnected nonlinear safety-critical system with security and asymmetric input constraints into an interconnected nonlinear safety-critical system with only asymmetric input constraints by using the barrier function. The non-quadratic utility function was added to the performance function to address the asymmetric input constraint. The critic network was also used to approach the optimal performance function and to establish the best security policy. Our control scheme stabilizes the closed-loop system and minimizes the improved performance function. In addition, the simulation results demonstrated the efficacy of the proposed distributed security solution. Future work will explore the optimal safety control of stochastic interconnected nonlinear systems with event triggering.