An Algorithm for Solving Zero-Sum Differential Game Related to the Nonlinear ${\mathcal{H}}_{\infty }$ Control Problem

Abstract

This paper presents an approach for the solution of a zero-sum differential game associated with a nonlinear state-feedback ${\mathcal{H}}_{\infty }$ control problem. Instead of using the approximation methods for solving the corresponding Hamilton–Jacobi–Isaacs (HJI) partial differential equation, we propose an algorithm that calculates the explicit inputs to the dynamic system by directly performing minimization with simultaneous maximization of the same objective function. In order to achieve numerical robustness and stability, the proposed algorithm uses: quasi-Newton method, conjugate gradient method, line search method with Wolfe conditions, Adams approximation method for time discretization and complex-step calculation of derivatives. The algorithm is evaluated in computer simulations on examples of first- and second-order nonlinear systems with analytical solutions of ${\mathcal{H}}_{\infty }$ control problem.

1. Introduction

Although the theory of nonlinear ${\mathcal{H}}_{\infty }$ control [1,2] is well developed and can be considered standardized, developing algorithms for solving this problem that enable practical implementation is a very active area of research. Furthermore, it is well known that the ${\mathcal{H}}_{\infty }$ control problem can be formulated as a two-player zero-sum differential game [3,4] with the objective function including a parameter in such a way that the control vector represents the player that minimizes the objective function while the vector of uncertainty represents the player that maximizes the same objective function. All available scientific research is mainly based on two main approaches: the formulation of the problem in the form of linear matrix inequalities (LMI) [5,6,7,8] or on the determination of the approximate solution of the associated Hamilton–Jacobi–Isaacs (HJI) equation [9,10,11,12], which is in linear case equivalent to the generalized Riccati equation [13].

Furthermore, methods for solving the nonlinear ${\mathcal{H}}_{\infty }$ control problem of singular or descriptor systems have also been developed. The state-feedback scheme, impulse controllability and the well-known implicit function theorem for stability analysis for finite-time ${\mathcal{H}}_{\infty }$ control problem of descriptor systems subject to actuator saturation are adopted in [14], while in [15] impulse and hybrid controllers are combined, resulting in less conservative stability conditions than in state-feedback control strategy.

In nonlinear programming-based algorithms that have been proposed in the literature, the system dynamics is treated as an equality constraint and is included in the optimization process using the method of Lagrange multipliers. This results in a HJI equation that is very difficult or almost impossible to solve. For this reason, many approximation methods are developed in which actual computational complexity increases with the number of system states which need to be estimated. In [10,12], the reviews of reinforcement adaptive learning and adaptive dynamic programming techniques to solve multiplayer games related to ${\mathcal{H}}_{\infty }$ control are given. In [16], an improved iterative or successive approximation methods for solving the HJI is developed. A game theoretic differential dynamic programming based algorithm for calculating both minimizing and maximizing inputs based on Taylor series expansion of the associated HJI equation around a nominal trajectory is proposed in [11]. Using a critic neural network with time-varying activation functions, the HJI equation is approximately solved in [9]. An event-triggering function for two-player zero-sum games in the presence of control constraints is designed in [17]. Furthermore, in [13], an iterative procedure to find the stabilizing solution of a set of Riccati equations related to discrete-time ${\mathcal{H}}_{\infty }$ control problems for periodic systems is addressed. The randomized algorithm based on the Ray-Shooting Method for ${\mathcal{H}}_{\infty }$ optimal synthesis of proportional–integral–derivative (PID) controller for linear systems is proposed in [18].

In the case of the methods and algorithms mentioned above, the application of LMI and solving Riccati equation requires the linearization, and in this case the optimality of solution cannot be guaranteed in all operating states of nonlinear system dynamics. On the other hand, solving the HJI equation can be very complex and therefore difficult to apply in real control tasks.

In this paper, the nonlinear state-feedback ${\mathcal{H}}_{\infty }$ control problem is formulated as zero-sum differential game, and we approach its solution without the LMI formalism or the need to approximate the solution of the HJI equation. The main idea of the presented approach is in the application of the conjugate gradient method, where the first-order derivatives are calculated by matrix relations backwards in time, which gives a direct numerical calculation of the control and uncertainty variables that explicitly depend on the system states.

In contrast to the approaches proposed in [11,16], which in the case of including a more complex nonlinearity of the dynamic system in the objective function result in a HJI equation so complex that it is practically impossible to solve, in our approach the nonlinear system dynamics is not included in the objective function as an equality constraint, but the state variables, control law and uncertainty are coupled by recursive matrix relations and chain rule for ordered derivatives, which are used to calculate the objective function gradients that appear in conjugate gradient method. Furthermore, in contrast to approaches presented in [10,12], which can be computationally expensive since the tuning of neural network weights is based on a method of weighted residuals that includes calculation of Lebesgue integrals and estimation of state variables, in our approach, the computational complexity does not depend on the dimension of the state space since procedure proposed for gradient calculation has a backward in time structure similar to the back propagation through time algorithm.

Bedsides the conjugate gradient method, which is used for computation of saddle point of zero-sum differential game, the quasi-Newton method for ${\mathcal{L}}_2$-gain minimization, line search method with Wolfe conditions, Adams approximation method for time discretization and the complex-step method for calculation of derivatives are also systematically integrated into an efficient mathematical tool in order to achieve numerical robustness and stability.

The rest of the paper is organized as follows. In Section 2, the preliminaries of the nonlinear state-feedback ${\mathcal{H}}_{\infty }$ control from the concept of dissipativity point of view and also from the zero-sum differential games point of view are presented. Although theories of these concepts are well known, providing basic terms is necessary to understand the contributions contained in the following sections. A complete framework of the derivation of the algorithmic procedure that optimizes the ${\mathcal{L}}_2$-gain of nonlinear dynamic systems without solving the HJI partial differential equation is given in Section 3. In Section 4, the proposed algorithmic procedure is evaluated on examples of nonlinear systems for which the input variables can be determined exactly by solving the HJI equation. Finally, Section 5 concludes the paper.

Notation

The notation used is fairly standard. Matrices and vectors are represented in bold upper and bold lower case, respectively. Scalars are represented in italic lower case. $\mathbf{I}$ is an identity matrix and $\mathbf{0}$ is a null matrix. The dimensions of the matrices and vectors can generally be determined trivially by the context. The symbol ∇ stands for the gradient and is considered as a row vector. The symbol ${}^{\mathrm{T}}$ denotes transposition.

The $\mathrm{vec}(·)$ is a operator that stacks the columns of the matrix one underneath the other. The Kronecker product of the two matrices $\mathbf{A}(m\times n)$ and $\mathbf{B}(p\times q)$, denoted by $\mathbf{A}\otimes \mathbf{B}$ is a $mp\times nq$ matrix defined by $\mathbf{A}\otimes \mathbf{B}={\left(a_{ij}\mathbf{B}\right)}_{ij}$. The definitions of matrix differentials calculus and the algebras related to Kronecker products can be found in [19,20].

The Euclidean norm of the vector is defined as $\parallel \mathbf{x}\parallel =\sqrt{{\mathbf{x}}^{\mathrm{T}}\mathbf{x}}$. ${\mathcal{L}}_2\left(I,{\mathbb{R}}^n\right)$ stands for the standard Lebesgue spaces of vector valued square-integrable and essentially bounded functions mapping an interval $I\subset \mathbb{R}$ to ${\mathbb{R}}^n$. This space is equipped with an ${\mathcal{L}}_2$ norm defined by ${\parallel ·\parallel }_{{\mathcal{L}}_2}=\sqrt{{\int }_{t_0}^{t_f}{\parallel ·\parallel }^{2}dt}$. We avoid explicitly showing the dependence of the variables from the time when not needed.

2. Preliminaries and Problem Formulation

In this section, we give a review of the basic terms that include nonlinear state-feedback ${\mathcal{H}}_{\infty }$ control and related differential game theory, and this is mostly based on classic references from this field [1,2,3,4,21,22].

Consider the causal input-affine nonlinear system in the state space defined on a some manifold $\mathcal{X}\subseteq {\mathbb{R}}^n$ in the following form

$$\mathrm{\Sigma }:\left\{\begin{array}{c}\dot{\mathbf{x}}=\mathbf{f}\left(\mathbf{x}\right)+{\mathbf{G}}_1\left(\mathbf{x}\right)\mathbf{u}+{\mathbf{G}}_2\left(\mathbf{x}\right)\mathbf{d},\mathbf{x}\left(t_0\right)={\mathbf{x}}_0,\hfill \\ \mathbf{y}=\mathbf{x},\hfill \\ \mathbf{z}=\mathbf{h}\left(\mathbf{x}\right)+\mathbf{L}\left(\mathbf{x}\right)\mathbf{u},\hfill \end{array}\right.$$

(1)

where $\mathbf{x}\in \mathcal{X}$ is the state vector, $\mathbf{u}\in \mathcal{U}\subseteq {\mathbb{R}}^m$ is the control vector, $\mathbf{d}$ is the vector representing internal/external uncertainty belonging to the set $\mathcal{D}\subset {\mathcal{L}}_2\left(\left[t_0,t_f\right],{\mathbb{R}}^s\right)$. The output vector $\mathbf{y}\in {\mathbb{R}}^n$ contains all directly measured states of system $\mathrm{\Sigma }$. The vector $\mathbf{z}\in {\mathbb{R}}^q$ is the performance variable. Furthermore, the functions $\mathbf{f}:\mathcal{X}\to \mathcal{X}$, ${\mathbf{G}}_1:\mathcal{X}\to {\mathcal{M}}^{n\times m}\left(\mathcal{X}\right)$, ${\mathbf{G}}_2:\mathcal{X}\to {\mathcal{M}}^{n\times s}\left(\mathcal{X}\right)$, $\mathbf{h}:\mathcal{X}\to {\mathbb{R}}^q$, $\mathbf{L}:\mathcal{X}\to {\mathcal{M}}^{q\times m}\left(\mathcal{X}\right)$ are real $C^1$-functions of $\mathbf{x}$.

The following assumptions are employed:

Assumption 1.

$\mathbf{x}=\mathbf{0}$ is a unique equilibrium point, with $\mathbf{u}=\mathbf{0}$ and $\mathbf{d}=\mathbf{0}$, of system Σ and for simplicity $\mathbf{f}\left(\mathbf{0}\right)=\mathbf{0}$, $\mathbf{h}\left(\mathbf{0}\right)=\mathbf{0}$.

Assumption 2.

The vector $\mathbf{h}\left(\mathbf{x}\right)$ and matrix $\mathbf{L}\left(\mathbf{x}\right)$ are such that $\mathbf{h}{\left(\mathbf{x}\right)}^{\mathrm{T}}\mathbf{L}\left(\mathbf{x}\right)=\mathbf{0}$ i $\mathbf{L}{\left(\mathbf{x}\right)}^{\mathrm{T}}\mathbf{L}\left(\mathbf{x}\right)=\mathbf{I}$ form all $\mathbf{x}\in \mathcal{X}$, which implies

$$\mathbf{z}=\left[\begin{array}{c}\mathbf{h}\left(\mathbf{x}\right)\\ \mathbf{u}\end{array}\right]⟹{\parallel \mathbf{z}\parallel }^2={\parallel \mathbf{h}\left(\mathbf{x}\right)\parallel }^2+{\parallel \mathbf{u}\parallel }^2.$$

(2)

By introducing Assumption 2, the so-called singular problem is avoided. More details on solving the singular problem can be found in [23,24].

Assumption 3.

Initial state vector ${\mathbf{x}}_0$ is a priori known.

Furthermore, the following definition is introduced:

Definition 1

(${\mathcal{L}}_2$-gain). ${\mathcal{L}}_2$-gain from $\mathbf{d}$ to $\mathbf{z}$ of system Σ is the supremum of $\gamma >0$ satisfies

$${\parallel \mathbf{z}\parallel }_{{\mathcal{L}}_2}^2\le {\gamma }^2{\parallel \mathbf{d}\parallel }_{{\mathcal{L}}_2}^2+\beta \left({\mathbf{x}}_0\right),$$

(3)

for some bounded $C^0$-function $\beta :U\subset \mathcal{X}\to \mathbb{R}$ such that $\beta \left(\mathbf{0}\right)=0$.

In general, the problem of nonlinear ${\mathcal{H}}_{\infty }$ control of system $\mathrm{\Sigma }$ where all state variables are available (measurable or can be estimated) can be formulated as follows:

Problem 1.

The problem of optimal nonlinear state-feedback ${\mathcal{H}}_{\infty }$ control of system Σ is to determine the control law ${\mathbf{u}}^{*}=\mathbf{\mu }(\mathbf{x},t)$ and the “worst case” ${\mathbf{d}}^{*}=\mathbf{\nu }(\mathbf{x},t)$ such that $\gamma >0$ is minimized.

Assumption 4.

The functions $\mathbf{\mu }(\mathbf{x},t)$ and $\mathbf{\nu }(\mathbf{x},t)$ are $\mathbf{\mu }\in C^1\left(\mathcal{X}\right)$, $\mathbf{\nu }\in C^1\left(\mathcal{X}\right)$.

If for the system $\mathrm{\Sigma }$ there exists some $\gamma \ge {\gamma }^{*}$ that satisfies (3), then a zero-sum differential game can be defined. The optimal value of this game is given by the following expression:

$$J^{*}\left({\mathbf{x}}_0\right)=\underset{\mathbf{u}}{min}\underset{\mathbf{d}}{max}\underset{t_0}{\overset{t_f}{\int }}\left({\parallel \mathbf{h}\left(\mathbf{x}\right)\parallel }^2+{\parallel \mathbf{u}\parallel }^2-{\gamma }^2{\parallel \mathbf{d}\parallel }^2\right)dt,$$

(4)

with dynamic equality constraints (1) on a finite time horizon $t_f>t_0$.

The necessary conditions for the saddle point of the zero-sum differential game (4) follow from the minimum and maximum principles, and are of the form

$$\begin{array}{cc}\hfill {\mathbf{u}}^{*}(\mathbf{x},t)& =-\frac{1}{2}{\mathbf{G}}_1^{\mathrm{T}}\left(\mathbf{x}\right){\nabla }_{\mathbf{x}}^{\mathrm{T}}V(\mathbf{x},t),\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill {\mathbf{d}}^{*}(\mathbf{x},t)& =\frac{1}{2{\gamma }^2}{\mathbf{G}}_2^{\mathrm{T}}\left(\mathbf{x}\right){\nabla }_{\mathbf{x}}^{\mathrm{T}}V(\mathbf{x},t),\hfill \end{array}$$

(6)

where V is a smooth positive semidefinite solution of the HJI partial differential equation

$$\begin{array}{cc}\hfill & {\nabla }_{t}V(\mathbf{x},t)+{\nabla }_{\mathbf{x}}V(\mathbf{x},t)\mathbf{f}\left(\mathbf{x}\right)+\frac{1}{4}{\nabla }_{\mathbf{x}}V(\mathbf{x},t)\left[\frac{1}{{\gamma }^2}{\mathbf{G}}_2\left(\mathbf{x}\right){\mathbf{G}}_2^{\mathrm{T}}\left(\mathbf{x}\right)\right.\hfill \\ \hfill & -{\mathbf{G}}_1\left(\mathbf{x}\right){\mathbf{G}}_1^{\mathrm{T}}\left(\mathbf{x}\right)]{\nabla }_{\mathbf{x}}^{\mathrm{T}}V(\mathbf{x},t)+{\mathbf{h}}^{\mathrm{T}}\left(\mathbf{x}\right)\mathbf{h}\left(\mathbf{x}\right)=0,V(\mathbf{x}\left(t_f\right),t_f)=0.\hfill \end{array}$$

(7)

3. Synthesis of the Algorithm for the Solution of the Zero-Sum Differential Game

In this section, an approach of determining control and uncertainty variables for optimal ${\mathcal{H}}_{\infty }$ control (Problem 1) of nonlinear dynamic system $\mathrm{\Sigma }$ is proposed. The proposed approach does not require solving the HJI Equation (7), but the solution is reduced to the direct numerical determination of the saddle point of the related zero-sum differential game.

Control and uncertainty variables are approximated by functions with a linear dependence on a finite number of constant parameters. To calculate these parameters, an approach based on the integration of the quasi-Newton method, the conjugate gradient method, the Adams method and the complex-step derivative approximation into one algorithm is proposed. The goal is to obtain a control variable that explicitly depends on the state variables and in that form is simple for practical implementation. Additionally, the aim is to achieve a numerical solution that uniformly converges towards the optimal solution by increasing the order of complexity of the approximation, i.e., by increasing the number of parameters.

Since we introduced the Assumption 4, based on Weierstraß’s theorem ([25], p. 65) (which refers to polynomial approximation functions) and its generalizations [26,27,28,29] (which refer to non-polynomial forms of nonlinear approximation functions) on the uniform approximation of smooth functions, there are constants $p_j^i,r_j^i\in \mathbb{R}$ such that the i-th component of control and uncertainty vector can be written in the following form:

$$\begin{array}{cc}\hfill {\widehat{u}}_i\left(\mathbf{x}\right)& =\sum _{j=1}^{n_{\theta }}{p}_j^i{\theta }_j^i\left(\mathbf{x}\right),\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill {\widehat{d}}_i\left(\mathbf{x}\right)& =\sum _{j=1}^{n_{\psi }}{r}_j^i{\psi }_j^i\left(\mathbf{x}\right),\hfill \end{array}$$

(9)

where ${\theta }_j^i\left(\mathbf{x}\right)\in C^1\left(\mathcal{X}\right),{\psi }_j^i\left(\mathbf{x}\right)\in C^1\left(\mathcal{X}\right)$ such that ${\theta }_j^i\left(\mathbf{0}\right)=0$, ${\psi }_j^i\left(\mathbf{0}\right)=0$. Linear subspaces generated by sets $\left\{{\theta }_j^i\left(\mathbf{x}\right)\right\}$ i $\left\{{\psi }_j^i\left(\mathbf{x}\right)\right\}$ are dense in the Sobolev norm $W^{1,\infty }$ [30].

For well chosen functions ${\theta }_j^i\left(\mathbf{x}\right)$ and ${\psi }_j^i\left(\mathbf{x}\right)$, we have

$$\begin{array}{cc}\hfill |{\widehat{u}}_i\left(\mathbf{x}\right)-u_i\left(\mathbf{x}\right)|& <{\epsilon }_{u_i}\left(\mathbf{x}\right),\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill |{\widehat{d}}_i\left(\mathbf{x}\right)-d_i\left(\mathbf{x}\right)|& <{\epsilon }_{d_i}\left(\mathbf{x}\right),\hfill \end{array}$$

(11)

where ${\epsilon }_{u_i}\left(\mathbf{x}\right)$ i ${\epsilon }_{d_i}\left(\mathbf{x}\right)$ are the approximation errors. It follows that ${\epsilon }_{u_i}\left(\mathbf{x}\right)\to 0$, ${\epsilon }_{d_i}\left(\mathbf{x}\right)\to 0$ when $n_{\theta }\to \infty$, $n_{\psi }\to \infty$, respectively, while for fixed $n_{\theta }$ and $n_{\psi }$, approximation errors are bounded by constants on the compact set. It is well known from approximation theory (see for example [31,32]) that it is often possible to determine in advance how many terms of the basis functions expansion should be taken for the required accuracy.

Equations (8) and (9) can be written in the following matrix form:

$$\begin{array}{cc}\hfill \widehat{\mathbf{u}}\left(\mathbf{x}\right)& =\mathrm{\Theta }\left(\mathbf{x}\right)\mathbf{\pi },\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill \widehat{\mathbf{d}}\left(\mathbf{x}\right)& =\mathrm{\Psi }\left(\mathbf{x}\right)\mathbf{\rho },\hfill \end{array}$$

(13)

where

$$\begin{array}{cc}\hfill \mathrm{\Theta }\left(\mathbf{x}\right)& \equiv \left[\begin{array}{cccc}{\mathbf{\theta }}^1\left(\mathbf{x}\right)& \mathbf{0}& \cdots & \mathbf{0}\\ \mathbf{0}& {\mathbf{\theta }}^2\left(\mathbf{x}\right)& \cdots & \mathbf{0}\\ ⋮& ⋮& \ddots & ⋮\\ \mathbf{0}& \mathbf{0}& \cdots & {\mathbf{\theta }}^m\left(\mathbf{x}\right)\end{array}\right],\mathrm{\Psi }\left(\mathbf{x}\right)\equiv \left[\begin{array}{cccc}{\mathbf{\psi }}^1\left(\mathbf{x}\right)& \mathbf{0}& \cdots & \mathbf{0}\\ \mathbf{0}& {\mathbf{\psi }}^2\left(\mathbf{x}\right)& \cdots & \mathbf{0}\\ ⋮& ⋮& \ddots & ⋮\\ \mathbf{0}& \mathbf{0}& \cdots & {\mathbf{\psi }}^s\left(\mathbf{x}\right)\end{array}\right],\hfill \\ \hfill {\mathbf{\theta }}^i\left(\mathbf{x}\right)& \equiv \left[\begin{array}{cccc}{\theta }_1^i\left(\mathbf{x}\right)& {\theta }_2^i\left(\mathbf{x}\right)& \cdots & {\theta }_{n_{\theta }}^i\left(\mathbf{x}\right)\end{array}\right],{\mathbf{\psi }}^i\left(\mathbf{x}\right)\equiv \left[\begin{array}{cccc}{\psi }_1^i\left(\mathbf{x}\right)& {\psi }_2^i\left(\mathbf{x}\right)& \cdots & {\psi }_{n_{\psi }}^i\left(\mathbf{x}\right)\end{array}\right],\hfill \\ \hfill \mathbf{\pi }& \equiv \left[\begin{array}{c}{\mathbf{p}}^1\\ {\mathbf{p}}^2\\ ⋮\\ {\mathbf{p}}^m\end{array}\right],\mathbf{\rho }\equiv \left[\begin{array}{c}{\mathbf{r}}^1\\ {\mathbf{r}}^2\\ ⋮\\ {\mathbf{r}}^s\end{array}\right],{\mathbf{p}}^i\equiv \left[\begin{array}{c}{p}_1^i\\ p_2^i\\ ⋮\\ p_{n_{\theta }}^i\end{array}\right],{\mathbf{r}}^i\equiv \left[\begin{array}{c}{r}_1^i\\ r_2^i\\ ⋮\\ r_{n_{\psi }}^i\end{array}\right].\hfill \end{array}$$

(14)

In this paper, based on the previous considerations, the problem that needs to be solved can be formulated as follows:

Problem 2.

Determine the parameters $\mathbf{\pi }$ and $\mathbf{\rho }$ such that the ${\mathcal{L}}_2$-gain of the closed-loop

$$\dot{\mathbf{x}}=\mathbf{f}\left(\mathbf{x}\right)+{\mathbf{G}}_1\left(\mathbf{x}\right)\mathrm{\Theta }\left(\mathbf{x}\right)\mathbf{\pi }+{\mathbf{G}}_2\left(\mathbf{x}\right)\mathrm{\Psi }\left(\mathbf{x}\right)\mathbf{\rho },$$

(15)

is minimized. In other words, according to Definition 1, solve the following minimax optimization problem

$$J_{\mu }^{*}\left({\mathbf{x}}_0\right)=\underset{\mathbf{\pi }}{min}\underset{\mathbf{\rho }}{max}\left\{{\parallel \mathbf{h}\left(\mathbf{x}\right)\parallel }_{{\mathcal{L}}_2}^2+{\parallel \mathrm{\Theta }\left(\mathbf{x}\right)\mathbf{\pi }\parallel }_{{\mathcal{L}}_2}^2-\mu {\parallel \mathrm{\Psi }\left(\mathbf{x}\right)\mathbf{\rho }\parallel }_{{\mathcal{L}}_2}^2\right\},$$

(16)

which represents a zero-sum differential game, where the minimal ${\mathcal{L}}_2$-gain is ${\gamma }^{*}=\sqrt{\mu }$.

First, to propose an algorithm for minimization of the parameter $\mu$ in (16), the first-order derivative of the value of the differential game with respect to $\mu$ is needed. Since $J_{\mu }^{*}\left({\mathbf{x}}_0\right)$ is non-differentiable because it is defined with the min–max operator, the sub-differential is employed. It can easily be shown that sub-differential of $J_{\mu }^{*}\left({\mathbf{x}}_0\right)$ with respect to $\mu$ is

$$\partial J_{\mu }^{*}=-{\parallel \mathrm{\Psi }\left(\mathbf{x}\right)\mathbf{\rho }\parallel }_{{\mathcal{L}}_2}^2.$$

(17)

Next, due to the inaccurate calculation of the previously mentioned derivative, the quasi-Newton method is considered, for which the superlinear convergence and numerical robustness have been proven (see for example [33,34,35,36]). The quasi-Newton k-th iteration is defined by

$$\begin{array}{cc}\hfill {\mu }_{k+1}& ={\mu }_k-\frac{\parallel \mathbf{h}\left({\mathbf{x}}_k\right){\parallel }_{{\mathcal{L}}_2}^2+\parallel \mathrm{\Theta }\left({\mathbf{x}}_k\right){\mathbf{\pi }}_k{\parallel }_{{\mathcal{L}}_2}^2-{\mu }_k{\parallel \mathrm{\Psi }\left({\mathbf{x}}_k\right){\mathbf{\rho }}_k\parallel }_{{\mathcal{L}}_2}^2}{\partial J_{{\mu }_k}^{*}}\hfill \\ \hfill & =\frac{\parallel \mathbf{h}\left({\mathbf{x}}_k\right){\parallel }_{{\mathcal{L}}_2}^2+{\parallel \mathrm{\Theta }\left({\mathbf{x}}_k\right){\mathbf{\pi }}_k\parallel }_{{\mathcal{L}}_2}^2}{\parallel \mathrm{\Psi }\left({\mathbf{x}}_k\right){\mathbf{\rho }}_k{\parallel }_{{\mathcal{L}}_2}^2}.\hfill \end{array}$$

(18)

To minimize the ${\mathcal{L}}_2$-gain of the system (15), we propose the following quasi-Newton algorithm.

It is important to note that in the second step of Algorithm 1, unlike other known methods of nonlinear optimization, the dynamics of the system (15) is not included in the objective function.

Algorithm 1 quasi-Newton method for ${\mathcal{L}}_2$-gain minimization.

Require:  ${\mu }_0\in {\mathbb{R}}_{+}$, $\epsilon \in {\mathbb{R}}_{+}$. Ensure:  ${\mu }^{*}$. 1: Set $k\leftarrow 0$. 2: Solve zero-sum differential game $$J_{{\mu }_k}^{*}=\underset{\mathbf{\pi }}{min}\underset{\mathbf{\rho }}{max}\left\{\parallel \mathbf{h}\left({\mathbf{x}}_k\right){\parallel }_{{\mathcal{L}}_2}^2+\parallel \mathrm{\Theta }\left({\mathbf{x}}_k\right){\mathbf{\pi }}_k{\parallel }_{{\mathcal{L}}_2}^2-{\mu }_k{\parallel \mathrm{\Psi }\left({\mathbf{x}}_k\right){\mathbf{\rho }}_k\parallel }_{{\mathcal{L}}_2}^2\right\},$$ (19) where ${\mathbf{x}}_k$, ${\mathbf{\pi }}_k$ and ${\mathbf{\rho }}_k$ are coupled by (15). 3: Calculate $${\mu }_{k+1}=\frac{\parallel \mathbf{h}\left({\mathbf{x}}_k\right){\parallel }_{{\mathcal{L}}_2}^2+{\parallel \mathrm{\Theta }\left({\mathbf{x}}_k\right){\mathbf{\pi }}_k\parallel }_{{\mathcal{L}}_2}^2}{\parallel \mathrm{\Psi }\left({\mathbf{x}}_k\right){\mathbf{\rho }}_k{\parallel }_{{\mathcal{L}}_2}^2}.$$ (20) 4: If $|{\mu }_{k+1}-{\mu }_k|\le \epsilon$ then terminate. Otherwise, set $k\leftarrow k+1$ and return to step 2.

To solve the subproblem in the second step of Algorithm 1, we use the conjugate gradient method as described in the following algorithm.

Note that, in Algorithm 2, the maximization of the objective function is obtained by changing the sign in front of the gradient with respect to $\mathbf{\rho }$ in (23). Additionally, it should be noted that the strategy for solving Problem 2 proposed by Algorithms 1 and 2 requires appropriate initialization, i.e., appropriate selection of approximation functions and initial parameters ${\mathbf{\pi }}_0$, ${\mathbf{\rho }}_0$ and ${\mu }_0$. If they are inadequate, it cannot be guaranteed that the control law is for the “worst case” of uncertainty. In the general, it is very difficult to guarantee whether a solution for the “worst case” of uncertainty is obtained. However, the conditions that would give recommendations for the initialization of the parameters of the proposed algorithms could be derived by applying the concept of inverse min max optimal control. Solving the problem of inverse min–max optimal control goes beyond the scope of the research presented in this paper and will be part of future research.

Algorithm 2 Conjugate gradient method for saddle point computation.

Require:  ${\mathbf{x}}_0\in {\mathbb{R}}^n$, ${\mathbf{\xi }}_0\equiv {\left[{\mathbf{\pi }}_0^{\mathrm{T}}{\mathbf{\rho }}_0^{\mathrm{T}}\right]}^{\mathrm{T}}\in {\mathbb{R}}^{n_{\xi }}$, ${\mu }_k$, ${\beta }_0\in {\mathbb{R}}_{+}$, ${\eta }_0\in {\mathbb{R}}_{+}$, $ϵ\in {\mathbb{R}}_{+}$. Ensure:  ${\mathbf{\pi }}^{*}$, ${\mathbf{\rho }}^{*}$. 1: Set $j\leftarrow 1$. 2: Perform a conjugate gradient descent/ascent algorithm in the following form $${\mathbf{\xi }}_j={\mathbf{\xi }}_{j-1}+{\eta }_{j-1}{\mathbf{s}}_{j-1},$$ (21) $${\mathbf{s}}_j=-\mathcal{J}\left({\mathbf{\xi }}_j\right)+{\beta }_{j-1}{\mathbf{s}}_{j-1},$$ (22) where $\mathbf{s}$ is the search direction vector and $$\mathcal{J}\left({\mathbf{\xi }}_j\right)={\left[\begin{array}{c}{\nabla }_{\mathbf{\pi }}J-{\nabla }_{\mathbf{\rho }}J\end{array}\right]}^{\mathrm{T}},$$ (23) $$J=\parallel \mathbf{h}\left({\mathbf{x}}_j\right){\parallel }_{{\mathcal{L}}_2}^2+\parallel \mathrm{\Theta }\left({\mathbf{x}}_j\right){\mathbf{\pi }}_j{\parallel }_{{\mathcal{L}}_2}^2-{\mu }_k{\parallel \mathrm{\Psi }\left({\mathbf{x}}_j\right){\mathbf{\rho }}_j\parallel }_{{\mathcal{L}}_2}^2.$$ (24) 3: Determine ${\eta }_j>0$ by applying the line search strategy that satisfies Wolfe’s conditions (see Algorithm 3). 4: Determine ${\beta }_j>0$ by applying Dai-Yuan method [37]. 5: If $\parallel \mathcal{J}\left({\mathbf{\xi }}_j\right){\parallel }_{\infty }\le ϵ$ then terminate. Otherwise, set $j\leftarrow j+1$ and return to step 2.

Algorithm 3 Line search method with Wolfe conditions.

Require:  ${\mathbf{\xi }}_j$, ${\mathbf{s}}_j$, $J\left({\mathbf{\xi }}_j\right)$, $\mathcal{J}\left({\mathbf{\xi }}_j\right)$, $0<c_1<c_2<1$, $\nu \in (0,1)$. Ensure:  ${\eta }_j$. 1: Set $l\leftarrow 0$. 2: Choose initial ${\eta }_0$. 3: While ${\eta }_l$ satisfies Wolfe’s conditions $$J({\mathbf{\xi }}_j+{\eta }_l{\mathbf{s}}_j)\le J\left({\mathbf{\xi }}_j\right)+c_1{\eta }_l\mathcal{J}\left({\mathbf{\xi }}_j\right){\mathbf{s}}_j,$$ (25) $$\mathcal{J}({\mathbf{\xi }}_j+{\eta }_l{\mathbf{s}}_j){\mathbf{s}}_j\ge c_2\mathcal{J}\left({\mathbf{\xi }}_j\right){\mathbf{s}}_j,$$ (26) calculate (i) ${\eta }_{l+1}=\nu {\eta }_l$, (ii) $l\leftarrow l+1$. 4: Set ${\eta }_j={\eta }_l$.

Furthermore, the initialization of parameters $\beta$ and $\eta$ is dependent on a specific optimization problem. Through a series of numerical experiments in simulations, we determined that the Dai–Yuan conjugate gradient method reaches a similar level of numerical robustness and accuracy for various initial values of ${\beta }_0$, whereas for the standard gradient algorithm, choosing the initial parameters largely affects the convergence and can cause numerical instabilities.

To determine the convergence step ${\eta }_j>0$, we use a line search method that satisfies the Wolfe conditions as described in the following algorithm.

Details related to the line search method and Wolfe conditions can be found in [38].

Gradient Calculation

In this subsection, the relations for calculating the gradients that appear in (22) i.e., (23) is derived. In order to perform chain rule for derivatives backward in time, first the system (15) needs to be rewritten in discrete-time state-space form. For this purpose, we use the multistep Adams method. Compared to the most popular four stage Runge–Kutta method, which requires solving the approximation problem four times at each step, the Adams method requires only one calculation.

The explicit Adams approximation of system (15) can be formulated as follows. Let the time interval $\left[t_0,t_f\right]$ consists of $t_i=t_0+i\tau$ for $i=0,1,2,\dots ,N-1$, such that $\tau =(t_f-t_0)/N$ is the time step length, and let $\widehat{\mathbf{x}}$ be the extended $4n$-dimensional state vector (n is dimension of state vector $\mathbf{x}$ and 4 is order of Adams method). Then, the system can be written in the following discrete-time state-space form:

$$\widehat{\mathbf{x}}(i+1)=\widehat{\mathbf{f}}\left(\widehat{\mathbf{x}}\left(i\right)\right)+\mathbf{a}\otimes \mathbf{B}\left(\mathbf{x}\left(i\right)\right)\mathbf{\pi }+\mathbf{a}\otimes \mathbf{D}\left(\mathbf{x}\left(i\right)\right)\mathbf{\rho },\widehat{\mathbf{x}}\left(0\right)={\widehat{\mathbf{x}}}_0,$$

(27)

where the vector $\mathbf{a}$ contains a non-zero coefficients of Adams method (see ([39], page 358)), and where, for simplicity, we introduced

$$\mathbf{B}\left(\mathbf{x}\right)\equiv {\mathbf{G}}_1\left(\mathbf{x}\right)\mathrm{\Theta }\left(\mathbf{x}\right),\mathbf{D}\left(\mathbf{x}\right)\equiv {\mathbf{G}}_2\left(\mathbf{x}\right)\mathrm{\Psi }\left(\mathbf{x}\right).$$

(28)

General considerations regarding Adams method are given in [39], while several novel schemes for multistep Adams method are proposed in [40].

The discrete-time form of the objective function (24) is

$$J=\tau \sum _{i=0}^{N-1}\left({\mathbf{h}}^{\mathrm{T}}\left(\mathbf{x}\left(i\right)\right)\mathbf{h}\left(\mathbf{x}\left(i\right)\right)+{\mathbf{\pi }}^{\mathrm{T}}\mathbf{P}\left(\mathbf{x}\left(i\right)\right)\mathbf{\pi }-\mu {\mathbf{\rho }}^{\mathrm{T}}\mathbf{R}\left(\mathbf{x}\left(i\right)\right)\mathbf{\rho }\right),$$

(29)

where for simplicity we introduced

$$\mathbf{P}\left(\mathbf{x}\right)\equiv {\mathrm{\Theta }}^{\mathrm{T}}\left(\mathbf{x}\right)\mathrm{\Theta }\left(\mathbf{x}\right),\mathbf{R}\left(\mathbf{x}\right)\equiv {\mathrm{\Psi }}^{\mathrm{T}}\left(\mathbf{x}\right)\mathrm{\Psi }\left(\mathbf{x}\right).$$

(30)

The following is a derivation of recursive matrix relations for calculating ${\nabla }_{\mathbf{\pi }}J$ using the basic chain rule arithmetic and matrix differentials calculus as well as certain properties of vectorization of matrices and Kronecker algebra. The relations for calculating ${\nabla }_{\mathbf{\rho }}J$ are obtained in an analogous way.

The gradient of the objective function (29) with respect to the vector $\mathbf{\pi }$ is

$${\nabla }_{\mathbf{\pi }}J=\tau \sum _{i=0}^{N-1}\frac{\partial F\left(i\right)}{\partial \mathbf{\pi }},$$

(31)

where

$$F\left(i\right)={\mathbf{h}}^{\mathrm{T}}\left(\mathbf{x}\left(i\right)\right)\mathbf{h}\left(\mathbf{x}\left(i\right)\right)+{\mathbf{\pi }}^{\mathrm{T}}\mathbf{P}\left(\mathbf{x}\left(i\right)\right)\mathbf{\pi }-\mu {\mathbf{\rho }}^{\mathrm{T}}\mathbf{R}\left(\mathbf{x}\left(i\right)\right)\mathbf{\rho }.$$

(32)

Then, an operation of partial derivative with respect to vector $\mathbf{\pi }$ is performed over the expression (32), which gives:

$$\begin{array}{cc}\hfill & {\nabla }_{\mathbf{\pi }}F\left(i\right)=2{\mathbf{h}}^{\mathrm{T}}\left(\mathbf{x}\left(i\right)\right)·{\nabla }_{\widehat{\mathbf{x}}}\mathbf{h}\left(\mathbf{x}\left(i\right)\right)·{\nabla }_{\mathbf{\pi }}\widehat{\mathbf{x}}\left(i\right)-\mu {\left[\mathbf{\rho }\otimes \mathbf{\rho }\right]}^{\mathrm{T}}·{\nabla }_{\widehat{\mathbf{x}}}\mathrm{vec}\left(\mathbf{R}\left(\mathbf{x}\left(i\right)\right)\right)·{\nabla }_{\mathbf{\pi }}\widehat{\mathbf{x}}\left(i\right)\hfill \\ \hfill & +{\left[\mathrm{vec}\left(\mathbf{P}\right(\mathbf{x}\left(i\right)\left)\right)\right]}^{\mathrm{T}}·\left(\mathbf{\pi }\oplus \mathbf{\pi }\right)+{\left(\mathbf{\pi }\otimes \mathbf{\pi }\right)}^{\mathrm{T}}·{\nabla }_{\widehat{\mathbf{x}}}\mathrm{vec}\left(\mathbf{P}\left(\mathbf{x}\left(i\right)\right)\right)·{\nabla }_{\mathbf{\pi }}\widehat{\mathbf{x}}\left(i\right).\hfill \end{array}$$

(33)

In (33) it is necessary to calculate ${\nabla }_{\mathbf{\pi }}\widehat{\mathbf{x}}\left(i\right)$, which is obtained based on the expression (27) as follows:

$$\begin{array}{cc}\hfill & {\nabla }_{\mathbf{\pi }}\widehat{\mathbf{x}}\left(i\right)=\mathbf{a}\otimes \mathbf{B}\left(\mathbf{x}(i-1)\right)+{\nabla }_{\widehat{\mathbf{x}}}\left[\mathbf{a}\otimes \mathbf{B}\left(\mathbf{x}\right(i-1\left)\right)\mathbf{\pi }\right]·{\nabla }_{\mathbf{\pi }}\widehat{\mathbf{x}}(i-1)\hfill \\ \hfill & +{\nabla }_{\widehat{\mathbf{x}}}\widehat{\mathbf{f}}\left(\widehat{\mathbf{x}}(i-1)\right)·{\nabla }_{\mathbf{\pi }}\widehat{\mathbf{x}}(i-1)+{\nabla }_{\widehat{\mathbf{x}}}\left[\mathbf{a}\otimes \mathbf{D}\left(\mathbf{x}\right(i-1\left)\right)\mathbf{\rho }\right]·{\nabla }_{\mathbf{\pi }}\widehat{\mathbf{x}}(i-1),\hfill \end{array}$$

(34)

where

$$\begin{array}{cc}\hfill {\nabla }_{\widehat{\mathbf{x}}}\left[\mathbf{a}\otimes \mathbf{B}\left(\mathbf{x}\right(i-1\left)\right)\mathbf{\pi }\right]& =\mathbf{a}\otimes \left({\mathbf{\pi }}^{\mathrm{T}}\otimes {\mathbf{I}}_n\right)·{\nabla }_{\widehat{\mathbf{x}}}\mathrm{vec}\left(\mathbf{B}\left(\mathbf{x}\right(i-1\left)\right)\right),\hfill \end{array}$$

(35)

$$\begin{array}{cc}\hfill {\nabla }_{\widehat{\mathbf{x}}}\left[\mathbf{a}\otimes \mathbf{D}\left(\mathbf{x}\right(i-1\left)\right)\mathbf{\rho }\right]& =\mathbf{a}\otimes \left({\mathbf{\rho }}^{\mathrm{T}}\otimes {\mathbf{I}}_n\right)·{\nabla }_{\widehat{\mathbf{x}}}\mathrm{vec}\left(\mathbf{D}\left(\mathbf{x}\right(i-1\left)\right)\right).\hfill \end{array}$$

(36)

The expressions (34)–(36) represent recursive matrix relations for $i=1,2,\dots ,N-1$ with initial conditions

$${\nabla }_{\mathbf{\pi }}\widehat{\mathbf{x}}\left(0\right)=\mathbf{0}.$$

(37)

Note that the functions, $\widehat{\mathbf{f}}\left(\widehat{\mathbf{x}}\right)$, $\mathbf{h}\left(\mathbf{x}\right)$, $\mathbf{P}\left(\mathbf{x}\right)$, $\mathbf{R}\left(\mathbf{x}\right)$, $\mathbf{B}\left(\mathbf{x}\right)$ and $\mathbf{D}\left(\mathbf{x}\right)$, which appear in previous expressions, are known from system dynamics and predetermined approximation functions basis and their derivatives can be easily calculated to machine precision by applying a complex-step method.

The first-order derivative of $\widehat{\mathbf{f}}\left(\widehat{\mathbf{x}}\right)$ with respect to $\widehat{\mathbf{x}}$ using complex-step approximation is accomplished by approximating its component (let us say the k-th component) with a complex variable using a Taylor’s series expansion [41,42]

$${\widehat{f}}_k(\widehat{\mathbf{x}}+ih{\mathbf{e}}_j)={\widehat{f}}_k\left(\widehat{\mathbf{x}}\right)+ih{\nabla }_{\widehat{\mathbf{x}}}{\widehat{f}}_k\left(\widehat{\mathbf{x}}\right)·{\mathbf{e}}_j+\frac{{\left(ih\right)}^2}{2!}{\mathbf{e}}_j^{\mathrm{T}}·{\nabla }_{\widehat{\mathbf{x}}}^2{\widehat{f}}_k\left(\widehat{\mathbf{x}}\right)·{\mathbf{e}}_j+\dots .$$

(38)

where $\mathbf{e}$ is unit vector. Taking only the imaginary parts gives

$${\nabla }_{\widehat{\mathbf{x}}}{\widehat{f}}_k\left(\widehat{\mathbf{x}}\right)·{\mathbf{e}}_j=\frac{\mathrm{Im}\left\{{\widehat{f}}_k(\widehat{\mathbf{x}}+ih{\mathbf{e}}_j)\right\}}{h}+\mathcal{O}\left(h^2\right).$$

(39)

In the above expressions, i represents an imaginary unit $i^2=-1$, while previously with i we denoted the i-th time point. For this reason, the $\widehat{\mathbf{x}}\left(i\right)$ is omitted in these expressions, but this is implied. It can be seen that subtraction does not appear in the numerator of the expression (39), i.e., the subtractive cancellation problem has been removed. This means that the step h can be extremely small, so the derivative can be calculated to machine precision.

The implementation of a complex-step method can be very simply achieved by using a high-level programming language that does not require a prior definition of the variable types, but it is possible to define complex variables automatically by applying built-in functions.

The complex-step method for calculating ${\nabla }_{\widehat{\mathbf{x}}}\widehat{\mathbf{f}}\left(\widehat{\mathbf{x}}\right)$ is described by the pseudocode given in Algorithm 4. The calculation of derivatives of other known functions can be obtained in an analogous manner.

Algorithm 4 Calculation of derivatives at the i-th time point using the complex-step method.

Require:  n, h, $\widehat{\mathbf{x}}$, $\widehat{\mathbf{f}}\left(\widehat{\mathbf{x}}\right)$ Ensure:  ${\nabla }_{\widehat{\mathbf{x}}}\widehat{\mathbf{f}}\left(\widehat{\mathbf{x}}\right)$ 1: $\mathbf{x}1\leftarrow \widehat{\mathbf{x}}$ 2: for$j=1$ to n do 3:     $x{1}_j\leftarrow complex({\widehat{x}}_j,h)$ 4:     $\mathbf{f}1\leftarrow \widehat{\mathbf{f}}\left(\mathbf{x}1\right)$ 5:     for $k=1$ to n do 6:         ${\displaystyle \frac{\partial {\widehat{f}}_k\left({\widehat{x}}_j\right)}{\partial x_j}}\leftarrow {\displaystyle \frac{imag\left(f{1}_k\right)}{h}}$ 7:     end for 8:     $\mathbf{x}1\leftarrow \widehat{\mathbf{x}}$ 9: end for

4. Benchmark Examples with Analytic Solution

In this section, the algorithmic procedure described in the previous sections is tested on examples of nonlinear systems where the HJI equation can be solved analytically, thereby exactly determining the control and uncertainty vectors. In this way, it can be directly assessed whether the proposed algorithmic procedure calculates the required solutions with sufficient numerical efficiency in terms of convergence and accuracy. The algorithm is written and implemented in the MATLAB software package. When solving a particular problem, it is necessary to initialize the algorithm in a suitable way.

4.1. Example—First-Order System

Consider a scalar nonlinear system described by the following equation [21]:

$$\dot{x}=u+\mathrm{arctg}\left(x\right)·d,\mathbf{z}=\left[\begin{array}{c}x\\ u\end{array}\right].$$

(40)

If $\gamma >\frac{\pi }{2}$, then by solving the corresponding HJI equation (for details see [21]), the analytical feedbacks are

$$\begin{array}{cc}\hfill u^{*}& =-x{\left(1-\frac{1}{{\gamma }^2}{\mathrm{arctg}}^2\left(x\right)\right)}^{-\frac{1}{2}},\hfill \\ \hfill d^{*}& =\frac{1}{{\gamma }^2}x\mathrm{arctg}\left(x\right){\left(1-\frac{1}{{\gamma }^2}{\mathrm{arctg}}^2\left(x\right)\right)}^{-\frac{1}{2}}.\hfill \end{array}$$

(41)

Approximation functions of control and uncertainty variables are chosen in the following form

$$\begin{array}{cc}\hfill \widehat{u}\left(x\right)& =p_{1}x+p_2{x}^3+p_3{x}^5+p_4{x}^7+p_5{x}^9,\hfill \\ \hfill \widehat{d}\left(x\right)& =r_{1}x+r_2{x}^3+r_3{x}^5+r_4{x}^7+r_5{x}^9,\hfill \end{array}$$

(42)

i.e., written in the form (12) and (13) we have

$$\mathrm{\Theta }\left(x\right)=\mathrm{\Psi }\left(x\right)=\left[\begin{array}{ccccc}x& x^3& x^5& x^7& x^9\end{array}\right],\mathbf{\pi }=\left[\begin{array}{c}{p}_1\\ p_2\\ p_3\\ p_4\\ p_5\end{array}\right],\mathbf{\rho }=\left[\begin{array}{c}{r}_1\\ r_2\\ r_3\\ r_4\\ r_5\end{array}\right].$$

(43)

In this example, we set the following values of the algorithm parameters: we divided the time interval from $t_0=0$, to $t_f=10$ [s] into $N=$100,000 equal subintervals, i.e., the discretization step is $\tau ={10}^{-4}$ [s] and we applied the Adams method of the 4-th order; vector of initial conditions ${\mathbf{x}}_0=\left[1\right]$; initial value of the parameter ${\mu }_0={\gamma }_0^2=5$; vectors of initial parameters of approximation functions ${\mathbf{\pi }}_0={\mathbf{\rho }}_0=\mathbf{0}$; stopping criterion of the quasi-Newton method $\epsilon ={10}^{-3}$; stopping criterion of conjugate-gradient method $ϵ={10}^{-6}$, Dai–Yuan method is used by default, the initial numerical values of the conjugate gradient algorithm parameters are chosen as ${\eta }_0=0.1$ and ${\beta }_0=0.5$, coefficients $c_1={10}^{-3}$, $c_2=0.9$ and $\nu =0.8$.

We obtain the following parameter $\mu$ and parameters of approximation functions:

$${\mu }^{*}=2.4667,{\mathbf{\pi }}^{*}=\left[\begin{array}{c}-0.9999\\ -0.2045\\ 0.0781\\ -0.0391\\ 0.0112\end{array}\right],{\mathbf{\rho }}^{*}=\left[\begin{array}{c}0.0414\\ 0.9638\\ -1.7517\\ 1.8462\\ -0.7361\end{array}\right],$$

(44)

i.e., minimum ${\mathcal{L}}_2$-gain is ${\gamma }^{*}=\sqrt{{\mu }^{*}}=1.5706$.

Figure 1 shows the time dependence of the state variable, i.e., the response of the system (40) from the initial condition $x_0=1$, where the control variable and uncertainty variable are of the form (42) with parameters (44). The control and uncertainty variables from (42) obtained by the proposed algorithm with the parameters from (44) in comparison with the analytical solutions from the expression (41) depending on the state variable are shown in Figure 2. Figure 3 shows the solutions obtained by the derived algorithm in comparison with the analytical solutions (41) as a function of time. Based on everything shown, it can be concluded that the numerically obtained solutions approximate the analytical ones well with negligible error.

4.2. Example—Second-Order System

Consider the second order nonlinear system [43]

$$\begin{array}{cc}\hfill {\dot{x}}_1& =-\frac{1}{8}\left(29x_1+87x_1{x}_2^2\right)-\frac{1}{4}\left(2x_2+3x_2{x}_1^2\right)+u_1+\frac{1}{2}d,\hfill \\ \hfill {\dot{x}}_2& =-\frac{1}{4}\left(x_1+3x_1{x}_2^2\right)+3u_2+d,\hfill \\ \hfill \mathbf{z}& =\left[\begin{array}{c}\sqrt{2}\left(2x_1+6x_1{x}_2^2\right)\\ \sqrt{2}\left(4x_2+6x_1^2{x}_2\right)\\ u_1\\ u_2\end{array}\right].\hfill \end{array}$$

(45)

If ${\gamma }^{*}=1$ then by solving the corresponding HJI equation (for details see [43]), the analytical feedbacks are

$$\begin{array}{cc}\hfill u_1^{*}\left(\mathbf{x}\right)& =-x_1-3x_1{x}_2^2,\hfill \\ \hfill u_2^{*}\left(\mathbf{x}\right)& =-6x_2-9x_1^2{x}_2,\hfill \\ \hfill d^{*}\left(\mathbf{x}\right)& =\frac{1}{2}{x}_1+2x_2+3x_1^2{x}_2+\frac{3}{2}{x}_1{x}_2^2.\hfill \end{array}$$

(46)

Approximation functions of control and uncertainty variables are chosen in the following form:

$$\begin{array}{cc}\hfill {\widehat{u}}_1\left(\mathbf{x}\right)& =p_1^1{x}_1+p_2^1{x}_2+p_3^1{x}_1{x}_2+p_4^1{x}_1^2{x}_2+p_5^1{x}_1{x}_2^2+p_6^1{x}_1^2{x}_2^2+p_7^1{x}_1^2+p_8^1{x}_2^2,\hfill \\ \hfill {\widehat{u}}_2\left(\mathbf{x}\right)& =p_1^2{x}_1+p_2^2{x}_2+p_3^2{x}_1{x}_2+p_4^2{x}_1^2{x}_2+p_5^2{x}_1{x}_2^2+p_6^2{x}_1^2{x}_2^2+p_7^2{x}_1^2+p_8^2{x}_2^2,\hfill \\ \hfill \widehat{d}\left(\mathbf{x}\right)& =r_1{x}_1+r_2{x}_2+r_3{x}_1{x}_2+r_4{x}_1^2{x}_2+r_5{x}_1{x}_2^2+r_6{x}_1^2{x}_2^2+r_7{x}_1^2+r_8{x}_2^2,\hfill \end{array}$$

(47)

i.e., written in the form (12) and (13), we have

$$\begin{array}{cc}\hfill {\mathbf{\theta }}^1\left(\mathbf{x}\right)& ={\mathbf{\theta }}^2\left(\mathbf{x}\right)=\mathbf{\psi }\left(\mathbf{x}\right)=\left[\begin{array}{cccccccc}{x}_1& x_2& x_1{x}_2& x_1^2{x}_2& x_1{x}_2^2& x_1^2{x}_2^2& x_1^2& x_2^2\end{array}\right],\hfill \\ \hfill \mathrm{\Theta }\left(\mathbf{x}\right)& =\left[\begin{array}{cc}{\mathbf{\theta }}^1\left(\mathbf{x}\right)& \mathbf{0}\\ \mathbf{0}& {\mathbf{\theta }}^2\left(\mathbf{x}\right)\end{array}\right],\mathrm{\Psi }\left(\mathbf{x}\right)=\mathbf{\psi }\left(\mathbf{x}\right),\hfill \\ \hfill {\mathbf{p}}^1& =\left[\begin{array}{c}{p}_1^1\\ p_2^1\\ ⋮\\ p_8^1\end{array}\right],{\mathbf{p}}^2=\left[\begin{array}{c}{p}_1^2\\ p_2^2\\ ⋮\\ p_8^2\end{array}\right],\mathbf{\pi }=\left[\begin{array}{c}{\mathbf{p}}^1\\ {\mathbf{p}}^2\end{array}\right],\mathbf{\rho }=\left[\begin{array}{c}{r}_1\\ r_2\\ ⋮\\ r_8\end{array}\right].\hfill \end{array}$$

(48)

In this example, we set the following values of the algorithm parameters: we divided the time interval from $t_0=0$, to $t_f=3$ [s] into $N=5000$ equal subintervals, i.e., the discretization step is $\tau ={10}^{-4}$ [s] and we applied the Adams method of the 4-th order by default; vector of initial conditions ${\mathbf{x}}_0={\left[11\right]}^{\mathrm{T}}$; initial value of the parameter ${\mu }_0={\gamma }_0^2=5$; vectors of initial parameters of approximation functions ${\mathbf{\pi }}_0={\mathbf{\rho }}_0=\mathbf{1}$; stopping criterion of the quasi-Newton method $\epsilon ={10}^{-3}$; stopping criterion of conjugate-gradient method $ϵ={10}^{-3}$, Dai–Yuan method is used by default, the initial numerical values of the conjugate gradient algorithm parameters are chosen as ${\eta }_0=0.1$ and ${\beta }_0=0.5$, coefficients $c_1={10}^{-3}$, $c_2=0.9$ and $\nu =0.8$.

We obtain the following parameter $\mu$ and parameters of approximation functions:

$${\mu }^{*}=1.0021,{{\mathbf{p}}^1}^{*}=\left[\begin{array}{c}-0.9997\\ -0.0241\\ 0.0785\\ -0.0676\\ -3.0576\\ 0.0422\\ -0.0003\\ 0.0287\end{array}\right],{{\mathbf{p}}^2}^{*}=\left[\begin{array}{c}0.0004\\ -6.0319\\ 0.1224\\ -9.1208\\ -0.1533\\ 0.1016\\ -0.0009\\ 0.0825\end{array}\right],{\mathbf{\rho }}^{*}=\left[\begin{array}{c}0.4995\\ 2.0478\\ -0.1576\\ 3.1370\\ 1.6151\\ -0.0847\\ 0.0007\\ -0.0579\end{array}\right],$$

(49)

i.e., the minimum ${\mathcal{L}}_2$-gain is ${\gamma }^{*}=\sqrt{{\mu }^{*}}=1.0010$.

Figure 4 shows the time dependence of the state variables, i.e., the response of the system (45) from the initial conditions ${\mathbf{x}}_0={\left[11\right]}^{\mathrm{T}}$, where the control variables and uncertainty variable are of the form (47) with parameters (49). Figure 5 shows the solutions obtained by the derived algorithm in comparison with the analytical solutions (46). In this example, as in the previous, it is evident that the numerical solution approximates the analytical solution well.

5. Conclusions

In this paper, an approach for the solution of a nonlinear ${\mathcal{H}}_{\infty }$ control problem is presented. Instead of using the approximation methods for solving the corresponding HJI equation, the solution is obtained by direct numerical calculation of the control and uncertainty variables that explicitly depend on the system states. In order to achieve numerical efficiency, the proposed algorithmic procedure uses quasi-Newton method, conjugate gradient method, line search method with Wolfe conditions, Adams approximation method for time discretization, and complex-step calculation of derivatives. In spite of the fact that the methods used in this paper are known from the references cited, in our approach, they are integrated together to provide a suitable mathematical tool for numerical solution of the zero-sum differential game related to the nonlinear ${\mathcal{H}}_{\infty }$ control problem.

The extension of this approach can be continued in two research directions: (i) consider output measurement-feedback ${\mathcal{H}}_{\infty }$ optimal control problem, and (ii) the case where the initial state vector is unknown and treated as an uncertainty, i.e., the maximizing “player”. It can be assumed that the proposed strategy can be extended to these two cases without a significant increase in its complexity.