Stability Margin of Data-Driven LQR and Its Application to Consensus Problem

Abstract

In contrast with traditional control input design techniques based on mathematical models of the system, in data-driven control approaches, which have recently gained substantial attention, the controller is derived directly from the data that are collected from experiments or observations of the target system. In particular, several data-driven optimal control and model predictive control (MPC) techniques have been proposed. In this paper, it is shown that the recently proposed data-driven LQR (Linear Quadratic Regulator) has a stability margin that is the set of the uncertainties in the control input channels maintaining the closed-loop stability. As an application of the proposed stability margin of the data-driven LQR, the consensus problem is considered. Since the control design for the consensus of multi-agent systems can be reformulated into the robust stabilization of a linear system with uncertainty in the input channel, it is demonstrated that the derived stability margin can be used to design a controller for the consensus of multi-agent systems.

1. Introduction

Over the past few decades, research on model-based control design, a control input design technique based on mathematical models of the system under consideration, has dominated the control community. Using mathematical models of these systems, it has been possible to design control techniques that take into account various performance indicators of the system, such as optimality of the control performance expressed by a quadratic cost with respect to the control and state [1,2]. At the same time, it should be noted that the application of these control techniques requires the construction of an accurate mathematical model of the system.

However, constructing exact mathematical models is a difficult, expensive and time-consuming task [3]. At a first glance, the observed data from the running system are inseparable from noise. In the system identification field, researchers have attempted to design methods to address the influence of noise both on input and output data [4,5,6]. However, even though the system may be successfully modeled based on the data, there is no guarantee that the fitting model is the perfect mathematical description for the control task due to uncertainties [7]. After an exhaustive effort to optimally identify the underlying dynamics from the data, the actual plant still requires a robust controller to compensate for the above problems.

In contrast with model-based control design, data-driven control design aims to eliminate the need for system identification and directly tackle control problems when formulating the control input. For instance, one recent study [8] employs Petersen’s lemma through the linear matrix inequality formulation for data generated by linear and polynomial systems. An earlier study [9] constructs a quadratic matrix inequality (QMI) of a linear model with noise that is implied by another QMI that is composed of noisy data through S-Lemma [9]. In another recent study [10], the authors exploit prior knowledge of uncertainty or disturbance to design a non-conservative data-driven control design. These efforts are mainly motivated by the desire to remove the complexity and shorten the required time to build control systems while also dealing with problems that exist in the model-based control method.

The driving force of the transition to the data-driven control method is well outlined by Willems et al. [11]: a persistently excited input and output data pair can represent the behavior of a dynamic system. In light of this, two recent studies have presented a framework to design closed-loop data-driven control [12,13]. The notion of data informativity relaxes the requirement of persistently exciting data for some of the data-driven control problems [14]. The framework is then extended to directly formulate an optimal control method via the observed data such as in a LQR (Linear Quadratic Regulator) [15] and MPC (Model Predictive Control) [16,17].

In order to apply control inputs to real-world systems successfully, it is important to design control inputs that ensure system robustness, such as stability margins. It is well known that a continuous-time LQR provides stability margins such as the gain margin $[\frac{1}{2},\infty ]$ and the phase margin ($-\frac{\pi }{3},\frac{\pi }{3}$) [2,18,19]. Such a property is very useful in practice and quite special compared with other model-based control designs, which are usually weak against model uncertainties. In the case of a discrete-time LQR (DLQR), the stability margin is investigated By Kim et al. and Lee et al. [20,21]. Because of these robustness guarantees, LQR and DLQR have received a lot of attention over the years. A generalized stability margin, called the disc margin, has been investigated in the transfer function setting [22]. Furthermore, for the state space, a disc margin is proposed for a DLQR using Lyapunov stability theory [21]. To derive the disc margin of a DLQR, the closed-loop system including the uncertainty in the input channels is represented as $x(k+1)=Ax\left(k\right)+B\psi u\left(k\right)$, where $\psi$ is a complex number denoting the uncertainty and $u\left(k\right)$ is the DLQR. Then, the region for $\psi$ is derived using Lyapunov stability theory such that closed-loop stability is maintained. Since the region is a disc in the complex plane, it is also called a disc margin. Since $\psi$ is a complex number, the disc margin implies both the gain and phase margins simultaneously.

Recently [23], the idea of a data-driven LQR (DDLQR) has been presented using the closed-form solutions of the Riccati equation and Markov parameters estimation [24]. Given the fact that the LQR and DLQR have received attention for their robustness guarantees, a natural straightforward question is whether the DDLQR provides a stability margin or not. In this paper, as the first contribution, a disc margin (or stability margin) of a DDLQR is derived using Lyapunov stability theory and an explicit solution of the Riccati equation. In other words, when the control $u\left(t\right)$ is designed according to the DDLQR, a region for $\psi$ is derived in which the closed-loop stability of $x(k+1)=Ax\left(k\right)+B\psi u\left(k\right)$ is maintained. The result can be understood as a data-driven version of the result in Lee et al. [21]. In particular, it is shown that the disc margin of the DDLQR becomes almost the same as those of DLQR and LQR [2,21] if the system is asymptotically null controllable [25].

It is known that designing a controller for consensus of a MAS (Multi-Agent System) is equivalent to the robust control design for one agent in the MAS when the agent has uncertainties in its input channels [26,27,28]. Hence, a control design method guaranteeing the stability margin can be used for the controller design for consensus of the MAS [26,29]. Therefore, as the second contribution in this paper, it is shown that the derived stability margin of the DDLQR can be used to design a control for the consensus problem of a multi-agent system. Simulation results show that the DDLQR indeed provides a stability margin and DDLQR-based control leads to consensus of the MAS if the agent in the MAS is asymptotically null controllable.

Notation: $I_k$ denotes the $k\times k$ identity matrix and $x^{+}$ and x denote $x(k+1)$ and $x\left(k\right)$, respectively, where k is the discrete-time index. $\overline{\sigma }(·)$ and $\underline{\sigma }(·)$ represent the maximum and minimum singular values, respectively. System $x(k+1)=Ax\left(k\right)+Bu\left(k\right)$ is said to be stabilizable if there exists a matrix K such that $A+BK$ is Hurwitz stable. Furthermore, system $x(k+1)=Ax\left(k\right)+Bu\left(k\right),y\left(k\right)=Cx\left(k\right)$ is said to be detectable if there exists a matrix F such that $A+FC$ is Hurwitz stable.

2. Preliminaries

In this section, preliminary results required for deriving the main results are presented.

2.1. Guaranteed Stability Margin of Model-Based Discrete-Time LQR

In this subsection, the stability margin of DLQR (Discrete-Time Linear Quadratic Regulator) is briefly reviewed. To this end, consider the discrete-time LTI (Linear Time Invariant) system

$$\begin{array}{cc}\hfill x^{+}& =Ax+Bu,y=Cx,\hfill \end{array}$$

(1)

where $A\in {\mathbb{R}}^{n\times n}$ is the system matrix, $B\in {\mathbb{R}}^{n\times p}$ is the input matrix, $C\in {\mathbb{R}}^{q\times n}$ is the output matrix. Furthermore, $x\in {\mathbb{R}}^n$, $u\in {\mathbb{R}}^p$, and $y\in {\mathbb{R}}^q$ are the state, control input, and output, respectively. It is assumed that $(A,B)$ is stabilizable. Then the DLQR is derived by minimizing the following infinite horizon cost function

$$J=\sum _{k=0}^{\infty }\left[y^{\top }\left(k\right)Qy\left(k\right)+u^{\top }\left(k\right)Ru\left(k\right)\right],$$

(2)

where output weighting Q and control input weighting R are positive definite. It is assumed that the pair $(A,Q)$ is detectable. The minimizing DLQR is given by

$$u=-Kx,$$

(3)

where the optimal feedback gain K is given by

$$K={(B^{\top }PB+R)}^{-1}{B}^{\top }PA.$$

(4)

Note that the positive definite matrix P is the solution to the algebraic Riccati equation

$$A^{\top }PA-P-A^{\top }PB{(B^{\top }PB+R)}^{-1}{B}^{\top }PA+C^{T}QC=0.$$

(5)

In contrast with other model-based controls, it is well known that the DLQR provides a stability margin, which is important for practical applications [2,21]. To be specific, if there is input channel uncertainty in the real system such as $x^{+}=Ax+B\psi u$ where $\psi$ denotes the input channel uncertainty, the stability of the closed-loop system with a DLQR is maintained to some extent although the DLQR is designed $x^{+}=Ax+Bu$. This is formally presented in the following theorem.

Theorem 1 

([21]). Consider the closed-loop system consisting of (1) and (3) with the input channel uncertainty Ψ

$$x^{+}=Ax-B\Psi Kx,$$

(6)

where $\Psi =\mathrm{diag}\{{\psi }_1,\cdots ,{\psi }_p\}\in {\mathbb{C}}^{p\times p}.$ The closed-loop system (6) is asymptotically stable if ${\psi }_i$ belongs to the following set, called the disc margin, for all i

$${\Omega }_i=\left\{\sigma +j\omega \in \mathbb{C}:{\left[\sigma -\left(1+\frac{{\beta }_1}{{\alpha }_1-{\beta }_1}\right)\right]}^2+{\omega }^2<\frac{{\alpha }_1{\beta }_1+{\gamma }_1({\alpha }_1-{\beta }_1)}{{({\alpha }_1-{\beta }_1)}^2}\right\},$$

(7)

where j denotes the complex number, i.e., $j:=\sqrt{-1}$, ${\alpha }_1:=\overline{\sigma }(B^{\top }PB+R)$, ${\beta }_1:=\underline{\sigma }\left(R\right)$, and ${\gamma }_1:=\underline{\sigma }\left(Q\right)/\overline{\sigma }{\left(K\right)}^2$. In particular, if the matrix R is diagonal such that $R=\mathrm{diag}\{r_1,\cdots ,r_p\}$, the disc margin is given by

$${\Omega }_i=\left\{\sigma +j\omega :{\left[\sigma -\left(1+\frac{r_i}{\delta }\right)\right]}^2+{\omega }^2<\frac{r_i(r_i+\delta )+{\gamma }_1\delta }{{\delta }^2}\right\},$$

(8)

where $\delta :=\overline{\sigma }\left(B^{\top }PB\right)$.

Note that ${\psi }_i$ can be interpreted as the ith control input channel uncertainty and is used only for analysis. In other words, the LQR gain K is computed using the system model $(A,B)$ and the applied control input is $u=-Kx$. Then, in theory, the closed-loop system becomes $x^{+}=Ax-BKx$. However, in this paper, the stability of $x^{+}=Ax-B\Psi Kx$ is analyzed in order to investigate the stability margin. This method is again used later for the stability margin of the data-driven LQR. Since ${\Omega }_i$ can be viewed as the set of ${\psi }_i$ such that closed-loop stability of (6) is maintained, it can be interpreted as both the gain and phase margins simultaneously. For instance, the projection of the circle ${\Omega }_i$ in the complex plane onto the real axis corresponds to the gain margin. Theorem 1 means that the closed-loop stability is guaranteed with the DLQR if ${\psi }_i$ belongs to the disc margin (i.e., stability margin) in (7). Note that the DLQR is a model-based design, i.e., it needs the system information of $(A,B)$.

2.2. Data-Driven LQR (DDLQR)

This subsection summarizes the idea of the data-driven LQR [23], which is the main reference for the proposed result. In data-driven control design, the most important problem is how to obtain the feedback gain using measured data from the system rather than a system model ($A,B$). The approach [23] relies mainly on a model-based closed-form solution to the DARE (5) derived by Lewis and Furata and Wongsaisuwan [30,31], given by

$$P={\mathbf{O}}^{\top }{({\mathbf{Q}}_N^{-1}+\mathbf{S}{\mathbf{R}}_N^{-1}{\mathbf{S}}^{\top })}^{-1}\mathbf{O},$$

(9)

where N is sufficiently large and denotes the amount of measured data from the system, and

$$\begin{array}{cc}\hfill {\mathbf{R}}_N& \triangleq \mathrm{diag}(R,R,\cdots R)\in {\mathbb{R}}^{pN\times pN},\hfill \\ \hfill {\mathbf{Q}}_N& \triangleq \mathrm{diag}(Q,Q,\cdots Q)\in {\mathbb{R}}^{qN\times qN},\hfill \\ \hfill \mathbf{O}& \triangleq {\left[\begin{array}{cccc}{C}^{\top }& {(CA)}^{\top }& \cdots & {(C{A}^{N-1})}^{\top }\end{array}\right]}^{\top }\in {\mathbb{R}}^{qN\times n},\hfill \\ \hfill {\mathbf{O}}^{+}& \triangleq {\left[\begin{array}{cccc}{(CA)}^{\top }& {(C{A}^2)}^{\top }& \cdots & {(C{A}^N)}^{\top }\end{array}\right]}^{\top }\in {\mathbb{R}}^{qN\times n},\hfill \\ \hfill \mathbf{M}& \triangleq \left[\begin{array}{c}{M}_1\\ M_2\\ ⋮\\ M_N\end{array}\right]\in {\mathbb{R}}^{qN\times p},M_i=C{A}^{(i-1)}B\in {\mathbb{R}}^{q\times p},\hfill \\ \hfill \mathbf{S}& \triangleq \left[\begin{array}{ccccc}0& \cdots & \cdots & \cdots & 0\\ M_1& 0& \ddots & \ddots & ⋮\\ M_2& M_1& \ddots & \ddots & ⋮\\ ⋮& ⋮& \ddots & \ddots & ⋮\\ M_{N-1}& M_{N-2}& \cdots & M_1& 0\end{array}\right]\in {\mathbb{R}}^{qN\times pN}.\hfill \end{array}$$

Then, considering (4) and (9), the DLQR gain can be equivalently written as

$$K_{DLQR}={\left[R+{\mathbf{M}}^{\top }{\left({\mathbf{Q}}_N^{-1}+\mathbf{S}{\mathbf{R}}_N^{-1}{\mathbf{S}}^{\top }\right)}^{-1}\mathbf{M}\right]}^{-1}{\mathbf{M}}^{\top }{\left({\mathbf{Q}}_N^{-1}+\mathbf{S}{\mathbf{R}}_N^{-1}{\mathbf{S}}^{\top }\right)}^{-1}{\mathbf{O}}^{+}.$$

(10)

The main result of da Silva et al. [23] is to derive a data-driven version of (10), which is explained below. In view of the structure of the gain (10), since ${\mathbf{R}}_N$ and ${\mathbf{Q}}_N$ are weighting matrices, if $\mathbf{M}$, $\mathbf{O}$, and $\mathbf{S}$ can be estimated using the measured data from the system, it is possible to design a data-driven LQR. To this end, consider the Hankel matrix of a signal $z\left(k\right)$ defined by

$$H\left(z\left(k\right)\right)\triangleq \left[\begin{array}{cccc}z\left(k\right)& z(k+1)& \cdots & z(k+L_D-1)\\ z(k+1)& z(k+2)& \cdots & z(k+L_D)\\ ⋮& ⋮& \ddots & ⋮\\ z(k+N-1)& z(k+N)& \cdots & z(k+N+L_D-2)\end{array}\right]\in {\mathbb{R}}^{aN\times b{L}_D},z\left(k\right)\in {\mathbb{R}}^{a\times b},$$

where $L_D$ is a constant satisfying a rank condition $L_D\ge q(N+1)$ [12,32]. Furthermore, let us define matrices comprised of the set of historical input and output data

$$\begin{array}{cc}{U}_p=H\left(u\left(0\right)\right)\in {\mathbb{R}}^{pN\times L_D},\hfill & U_f=H\left(u\left(N\right)\right)\in {\mathbb{R}}^{pN\times L_D},\hfill \\ Y_p=H\left(y\left(0\right)\right)\in {\mathbb{R}}^{qN\times L_D},\hfill & Y_f=H\left(y\left(N\right)\right)\in {\mathbb{R}}^{qN\times L_D},\hfill \end{array}$$

where it is assumed that $(N+1)q\ge n$ is satisfied [23]. In order to construct matrices $U_p,U_f,Y_p,$ and $Y_f$, arbitrary input signals u are chosen and applied to the sytem and the corresponding outputs y are measured. From the main results in Overschee and de Moor and Lim et al. [33,34], a predictor of the system output can be written as

$$Y_f=\underset{\triangleq \mathbf{W}}{\underbrace{\left[\begin{array}{ccc}\mathbf{O}(\mathbf{D}+\mathbf{F}\mathbf{S})& -\mathbf{O}\mathbf{F}& \mathbf{S}\end{array}\right]}}\underset{\triangleq \Phi }{\underbrace{\left[\begin{array}{c}{U}_p\\ Y_p\\ U_f\end{array}\right]}}\triangleq \mathbf{W}\Phi ,$$

(11)

where

$$\mathbf{D}=\left[\begin{array}{cccc}{A}^{N}B& \cdots & AB& B\end{array}\right],$$

and $\mathbf{F}$ is a solution of $A^{N+1}+\mathbf{F}\mathbf{O}=0$. Then, for the data-driven design, $\widehat{\mathbf{S}}$ (i.e., the estimate of $\mathbf{S}$) is the last $pN$ columns of $\widehat{\mathbf{W}}$, which is the least squares solution of Equation (11). In other words,

$$\widehat{\mathbf{W}}=Y_f{\Phi }^{†},$$

(12)

where ${\Phi }^{†}$ stands for the pseudo inverse of $\Phi$. Using $\widehat{\mathbf{S}}$, the estimate of $\mathbf{O}$ can be obtained by

$$\widehat{\mathbf{O}}=(Y_p-\widehat{\mathbf{S}}{U}_p){\mathbf{X}}^{†},$$

(13)

where

$$\mathbf{X}\triangleq [x\left(0\right)x\left(1\right)\cdots x(L_D-1)].$$

(14)

Therefore, the data-driven LQR gain $K_D$ is given by

$$K_D={\left[R+{\widehat{\mathbf{M}}}^{\top }{\left({\mathbf{Q}}_N^{-1}+\widehat{\mathbf{S}}{\mathbf{R}}_N^{-1}{\widehat{\mathbf{S}}}^{\top }\right)}^{-1}\widehat{\mathbf{M}}\right]}^{-1}{\widehat{\mathbf{M}}}^{\top }{\left({\mathbf{Q}}_N^{-1}+\widehat{\mathbf{S}}{\mathbf{R}}_N^{-1}{\widehat{\mathbf{S}}}^{\top }\right)}^{-1}{\widehat{\mathbf{O}}}_{q-},$$

(15)

where $\widehat{\mathbf{M}}$ denotes the estimate of $\mathbf{M}$, and ${\widehat{\mathbf{O}}}_{q-}$ is the matrix obtained by removing the first q rows of $\widehat{\mathbf{O}}$. It should be noted that as more and more data is used, i.e., $N\to \infty$, the gain $K_D$ given in (15) converges to a gain $K_{DLQR}$ which is the optimal control gain for the DLQR problem [23,30]. Hence, strictly speaking, there is no guarantee that $K_D$ is optimal when N is not sufficiently large. In what follows, we assume that a sufficiently large amount of data are used to compute $K_D$ so that the computed gain, called data-driven DLQR gain, is optimal. Note that all variables in (15) are irrespective of the system model, i.e., $K_D$ is computed without using model information, which means that the gain $K_D$ is a data-driven LQR gain.

In contrast with the conventional LQR, which is based on the system model $(A,B)$, the DDLQR design computes the feedback gain $K_D$ relying only on the measured data from the system under the assumption that the amount of data N is sufficiently large.

3. Guaranteed Stability Margin of Data-Driven LQR

Considering the information provided in the previous section, it is a natural to question whether the DDLQR for (1) provides a stability margin or not.

To answer this question, consider the closed-loop system with the DDLQR gain $K_D$ in (15)

$$x^{+}=Ax-B\Psi K_{D}x,$$

(16)

where $\Psi =\mathrm{diag}\{{\psi }_1,\cdots ,{\psi }_p\}\in {\mathbb{C}}^{p\times p}$, and again $\Psi$ denotes the stability margin or the uncertainties in the control input channel. It turns out that the explained DDLQR guarantees a stability margin similarly to the model-based design of Lee et al. [21].

Theorem 2. 

The closed-loop system (16) with input channel uncertainty $\Psi =\mathit{diag}\left({\psi }_i\right)$ is asymptotically stable if ${\psi }_i$ belongs to the following set (i.e., disc margin) for all i

$${\overline{\Omega }}_i=\left\{\sigma +j\omega \in \mathbb{C}\left|{\left(\sigma -\frac{\alpha }{\beta }\right)}^2+{\omega }^2<\frac{{\alpha }^2+\beta (\gamma -\alpha )}{{\beta }^2}\right.\right\},$$

(17)

where

$$\begin{array}{cc}\hfill \alpha & \triangleq {\sigma }_{\max }(R+{\widehat{\mathbf{M}}}^{\top }{({\mathbf{Q}}_N^{-1}+\widehat{\mathbf{S}}{\mathbf{R}}_N^{-1}{\widehat{\mathbf{S}}}^{\top })}^{-1}\widehat{\mathbf{M}}),\hfill \\ \hfill \beta & \triangleq {\sigma }_{\max }\left({\widehat{\mathbf{M}}}^{\top }{({\mathbf{Q}}_N^{-1}+\widehat{\mathbf{S}}{\mathbf{R}}_N^{-1}{\widehat{\mathbf{S}}}^{\top })}^{-1}\widehat{\mathbf{M}}\right),\hfill \\ \hfill \gamma & \triangleq {\displaystyle \frac{{\sigma }_{\min }\left(Q\right)}{{\sigma }_{\max }\left(K_D^{\top }{K}_D\right)}}.\hfill \end{array}$$

In particular, if the matrix R is diagonal, i.e., $R=\mathrm{diag}\{r_1,\cdots ,r_p\}$, the disc margin is given by

$${\overline{\Omega }}_i=\left\{\sigma +j\omega \in \mathbb{C}\left|{\left(\sigma -\left(1+\frac{r_i}{\beta }\right)\right)}^2+{\omega }^2<\frac{r_i^2+\beta (r_i+\gamma )}{{\beta }^2}\right.\right\}.$$

(18)

Proof. 

See Appendix A. □

Since ${\psi }_i$ is a complex number, the disc margin in (17) (or (18)) represents both the gain and phase margins simultaneously. In view of Theorem 2, the DDLQR for system (1) also results in a stability margin.

It is well known that the gain and phase margins of a continuous-time LQR (CLQR) are given by $[\frac{1}{2},\infty )$ and $(-\frac{\pi }{3},\frac{\pi }{3})$, respectively. In the case of the discrete-time LQR, it is proved that the same gain and phase margins can be derived if the discrete-time system is asymptotically null controllable (ANCBC), which means that the system $(A,B)$ is controllable and the eigenvalues of A are inside the unit circle or on the unit circle [21]. The following theorem means that the guaranteed gain and phase margins of the DDLQR are also $[\frac{1}{2},\infty )$ and $(-\frac{\pi }{3},\frac{\pi }{3})$, respectively, if the system is ANCBC. This means that the stability margin of a DDLQR is the same as that of a DLQR or CLQR if the system is ANCBC.

Theorem 3. 

Suppose that ($A,B$) is ANCBC, and that the DDLQR gain denoted by $K_{D,ϵ}$ is obtained according to (15) with weighting matrices $Q=ϵI$ and $R=\mathit{diag}\{r_1,\cdots ,r_p\}$, where ϵ is a small positive constant. Consider a set

$${\overline{\Omega }}_{i,ϵ}=\left\{\sigma +j\omega \left|{\left(\sigma -\left(1+\frac{r_i}{\beta }\right)\right)}^2+{\omega }^2<\frac{r_i^2+\beta r_i}{{\beta }^2}\right.\right\},i=1,\cdots ,q,$$

(19)

which corresponds to the set ${\Omega }_i$ in (18) with $\gamma =0$. Then, as ϵ converges to zero, the set ${\overline{\Omega }}_{i,ϵ}$ approaches the set $H:=\{s\in \mathbb{C}:Re\left(s\right)>\frac{1}{2}\}$.

Since the proof of the theorem is quite similar to that of the corresponding theorem of Lee et al. [21], it is omitted here. However, it should be noted that the most important points in the proof are that ${lim}_{ϵ\to 0}{P}_{ϵ}=0$, where $P_{ϵ}$ is the solution of the discrete-time Riccati equation with $Q=ϵI$, that ${\delta }_{ϵ}=\overline{\sigma }\left(B^{\top }{P}_{ϵ}B\right)$ converges to zero as $ϵ\to 0$ if the system is ANCBC [25], and that the leftmost point of the disc margin is $\frac{{\delta }_{ϵ}+r_i-\sqrt{r_i^2+r_i{\delta }_{ϵ}}}{{\delta }_{ϵ}}$, which converges to $\frac{1}{2}$ as $ϵ\to 0$. ${lim}_{ϵ\to 0}{P}_{ϵ}=0$ holds in view of the structure of the explicit solution P in (9).

In view of the derivation in this section, the DDLQR also provides a stability margin. In the next section, how to use the stability margin for the control design for a consensus problem is presented.

4. Application of Disc Margin to a Consensus Problem

In this section, how to use the disc margin described in the previous section for controller design for a consensus problem is presented.

4.1. Consensus Problem

Consider a MAS (Multi-Agent System) consisting of $N_a$ identical discrete-time ANCBC systems written as

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill x_1^{+}& =A{x}_1+B{u}_1,\hfill \\ \hfill x_2^{+}& =A{x}_2+B{u}_2,\hfill \\ \hfill & ⋮\hfill \\ \hfill x_{N_a}^{+}& =A{x}_{N_a}+B{u}_{N_a},\hfill \end{array}\end{array}$$

(20)

where $x_i\in {\mathbb{R}}^n$, and $u_i\in {\mathbb{R}}^p$ stand for the state and input of the ith agent in the MAS, $A\in {\mathbb{R}}^{n\times n}$ and $B\in {\mathbb{R}}^{n\times p}$, the initial conditions of each agent are different, and the agents communicate with each other by exchanging state information between neighbor agents. It is assumed that the communication (i.e., the data exchange) is modeled by a graph Laplacian matrix $L\in {\mathbb{R}}^{N_a\times N_a}$ [21,27,28,35]. The Laplacian matrix is constructed by the degree of each agent and adjacency matrix, which indicates whether pairs of agents are adjacent or not in the graph. The diagonal element of the Laplacian matrix ${\ell }_{ii}=-{\Sigma }_{j\ne i}{\ell }_{ij}$ expresses the degree of agent i. ${\ell }_{ij}=-1$ for all $i\ne j$ if the agent i receives information from agent j. Otherwise, it is defined as ${\ell }_{ij}=0$.

The following facts about the Laplacian are instrumental in solving the consensus problem of the MAS:

• Fact 1. The eigenvalues of L are on the closed disc belonging to the complex plane, and its center and radius are $(N_a-1,0)$ and $N_a-1$, respectively, [36];
• Fact 2. At least one eigenvalue of the Laplacian matrix is located at the origin of the complex plane, and the associated eigenvector is $1_N$ [27];
• Fact 3. The zero eigenvalue at the origin is simple if the communication graph includes the directed spanning tree, which means that at least one node in the graph has a directed information path to all the other nodes in the graph [27,37].

For such a MAS, the consensus problem is to design a control achieving

$$\underset{k\to \infty }{lim}{x}_1\left(k\right)=x_2\left(k\right)=\cdots =x_{N_a}\left(k\right),$$

(21)

and the control of the ith agent is given by

$$u_i=K_c\sum _{j\in {\mathcal{N}}_i}{l}_{ij}(x_i-x_j),$$

where $K_c$ stands for the control gain for consensus, $l_{ij}$ is the ($i,j$)th element of the Laplacian matrix L, and ${\mathcal{N}}_i$ is the set of the neighbors of the ith agent. Note that the same control gain $K_c$ is used for each agent, and that the consensus problem requires only the design of the gain $K_c$ achieving (21).

Considering these properties of the algebraic graph theory and the definition of the consensus problem, the following assumption is inevitable in order to solve the consensus problem.

Assumption 1. 

A directed spanning tree is included in the communication graph modeling the information exchange of the MAS under consideration.

In designing $K_c$ for consensus of the MAS (20), Theorem 3 can be employed. Hence, only an ANCBC agent is considered in this paper.

Assumption 2. 

The system matrix A in (20) is ANCBC.

4.2. DDLQR-Based Consensus Algorithm

This subsection describes how the derived stability margin of the DDLQR can be effectively applied to the consensus problem for a multi-agent system (MAS).

The design of the control gain $K_c$ for consensus of the MAS relies on so-called simultaneous stabilization.

Theorem 4 

([21,26,29,38]). Suppose that Assumptions 1 and 2 hold. The MAS (20) reaches consensus if the control gain $K_c$ is designed such that

$$x^{+}=Ax+{\lambda }_{j}B{K}_{c}x,$$

(22)

is asymptotically stable for all $j=1,\cdots ,N_a-1$, where ${\lambda }_j$ denotes the nonzero eigenvalue of the graph Laplacian matrix L modeling the information exchange.

Note that the eigenvalue ${\lambda }_j$ of the Laplacian matrix is unknown in a distributed control setting. However, although ${\lambda }_j$ is unknown, if $K_c$ is designed according to the DDLQR and if it is possible to make all nonzero ${\lambda }_j$ belong to the stability margin of $K_c$, consensus is reached [21,26,29].

In order to design the control for the consensus, it is assumed that the real part of the smallest nonzero eigenvalue of the Laplacian matrix L is known. Note that there are existing methods to estimate the second-largest eigenvalue of the Laplacian matrix in the literature [39,40]. Let $\rho$ be the real part. Then, it is obvious that the real part of the smallest nonzero eigenvalue of $\frac{1}{\rho }L=:L^{\prime }$ is 1. As a result, it follows that the nonzero eigenvalues of $L^{\prime }$ belong to the area $\Theta$ in Figure 1 in light of Fact 1, Fact 3, and scaling by $\rho$.

In view of Theorem 3, the disc margin becomes larger as $ϵ$ gets smaller. Thanks to this tunable disc margin ${\overline{\Omega }}_{ϵ}$, there exists ${ϵ}^{\ast }$ such that $\Theta \subset {\overline{\Omega }}_{ϵ}$ for any $ϵ<{ϵ}^{\ast }$. Therefore, the control gain $K_c$ for consensus can be designed as $K_c=\frac{1}{\rho }{K}_{D,ϵ}$. In other words, the control for the consensus is given by

$$u_i=K_c\sum _{j\in {\mathcal{N}}_i}{l}_{ij}(x_i-x_j)=\frac{1}{\rho }{K}_{D,ϵ}\sum _{j\in {\mathcal{N}}_i}{l}_{ij}(x_i-x_j).$$

(23)

Then, the entire MAS can be written as

$${\xi }^{+}=[I_N\otimes A-L\otimes {\rho }^{-1}B{K}_{D,ϵ}]\xi =[I_N\otimes A-L^{\prime }B{K}_{D,ϵ}]\xi =:A_L\xi ,$$

(24)

where $\xi ={\left[\begin{array}{cccc}{x}_1^{\top }& x_2^{\top }& \cdots & x_N^{\top }\end{array}\right]}^{\top }$ and ⊗ denotes the Kronecker product. Using coordinate transformation $v:=(V^{-1}\otimes I_n)\xi$, where V is the matrix whose columns are the eigenvectors of L, it is possible to convert (24) into

$$v^{+}=\left[\begin{array}{ccccc}A& 0& 0& \cdots & 0\\ 0& A+{\lambda }_{1}B{K}_{D,ϵ}& 0& \cdots & 0\\ 0& 0& \ddots & & 0\\ 0& 0& \cdots & A+{\lambda }_{N-2}B{K}_{D,ϵ}& 0\\ 0& 0& \cdots & 0& A+{\lambda }_{N-1}B{K}_{D,ϵ}\end{array}\right]v.$$

For a detailed explanation, see Lee et al. [21]. Furthermore, it can be shown that all nonzero eigenvalues ${\lambda }_1,\cdots ,{\lambda }_{N-1}$ belong to $\Theta$ in Figure 1 due to the scaling constant $\rho$, and $\Theta \subset {\overline{\Omega }}_{ϵ}$ is satisfied due to sufficiently small $ϵ$. In other words, since all ${\lambda }_i,i=1,\cdots ,N-1$ belongs to the disc margin of the DDLQR, Theorem 3 holds. Hence, the consensus problems is solved owing to Theorem 3 and the disc margin of the DDLQR. The designed procedure for the DDLQR-based consensus controller can be summarized as follows.

1.

Estimate of the real part of the second smallest eigenvalue of the graph Laplacian L and set the value as $\rho$;

2.

Collect input, state, and output data;

3.

Choose sufficiently small $ϵ$ such that the imaginary part of $-{\omega }_{{\overline{\Omega }}_{ϵ}}$ is smaller than that of $-{\omega }_{\Theta }$, which means $\Theta \subset {\overline{\Omega }}_{ϵ}$;

4.

Calculate $K_c=\frac{1}{\rho }{K}_{D,ϵ}$, and each agent implements the controller (23).

Note that the proposed DDLQR-based consensus controller is a data-driven distributed controller in the sense that it does not use the agent’s mathematical model nor any global information of the graph.

5. Simulation Study

In this section, a numerical example of the stability margin of the DDLQR is shown and its application to the consensus problem is presented.

5.1. Numerical Example for the Disc Margin of Marginally Stable Systems with DDLQR

Since the DDLQR is heavily dependent on the amount of data, the effect of N on the accuracy of the DDLQR gain is presented first, viewing the DLQR gain as the ground truth. For this purpose, two cases with $N=3$ and $N=30$ are considered and $L_D$ is chosen as $L_D=3q(N+1)$. In addition, the gain margin of the DDLQR is presented by considering several uncertainty levels for an ANCBC system given by

$$A=\left[\begin{array}{cc}1.0& 0.8\\ 0.0& 0.2\end{array}\right],B=\left[\begin{array}{cc}0.2& 0\\ 0& 0.01\end{array}\right],C=I.$$

To obtain rich data with which the DDLQR gain can be computed, a PRBS (Pseudo Random Binary Sequence) is applied to the system for $(2N+L_D-1)$ steps, and the resulting state, input, and output data depicted in Figure 2 and Figure 3 are stored. Then, the weighting matrices of the DDLQR are set to $Q=0.8I_n$ and $R=1.3I_m$. The estimate $\widehat{\mathbf{S}}$ is obtained by solving (12) using the stored input and output data. In addition, the computed $\widehat{\mathbf{S}}$ is employed to calculate $\widehat{\mathbf{O}}$ and $\widehat{\mathbf{M}}$ together with the stored data. Finally, the DDLQR gain is obtained from (10) with $K_{D_3}=\left[\begin{array}{cc}0.2319& 0.2039\\ 0.0046& 0.0057\end{array}\right]$ and $K_{D_{30}}=\left[\begin{array}{cc}0.7247& 0.6993\\ 0.0299& 0.0311\end{array}\right]$, where $K_{D_i}$ denotes $K_D$ with $i=N$. As a comparison, we compute the model-based DLQR as $K_{DLQR}=\left[\begin{array}{cc}0.7247& 0.6993\\ 0.0299& 0.0311\end{array}\right]$. In view of $K_{D_3},K_{D_{30}}$, and $K_{DLQR}$, the DDLQR $K_{D_i}$ gain converges to the model-based DLQR gain $K_{DLQR}$ as i increases. In other words, the DDLQR gain converges to the DLQR gain as the amount of stored data increases. Hence, with sufficiently large N, the DDLQR gain $K_D$ becomes the DLQR gain $K_{DLQR}$.

Next, the derived stability margin is validated for various input channel uncertainties with $N=30$. The maximum and minimum real parts of the gain margin (19) for the system are $0.5194$ and $13.4183$, respectively. Five input channel uncertainty levels $p\in \{-5.0,0.6,7.0,13.0,18.0\}$ in Figure 4 are considered. In other words, input channel uncertainty p is applied to the system as follows

$$x^{+}=Ax+Bpu,$$

where $x_i\in {\mathbb{R}}^2$, $u\in \mathbb{R}$, $A\in {\mathbb{R}}^{2\times 2}$, and $B\in {\mathbb{R}}^{2\times 1}$. Figure 5 shows that the closed-loop system under the DDLQR is asymptotically stable when the input channel uncertainty p belongs to the disc margin depicted by the red circle in Figure 4. Conversely, it can be seen that the closed-loop system becomes unstable when the uncertainty p is not in the stability margin.

5.2. Application of the Disc Margin to the Consensus Problem

To present a DDLQR-based control design for consensus of a MAS consisting of ANCBC agents, consider an ANCBC agent

$${\Sigma }_i:x_i^{+}=\left[\begin{array}{cc}1.0& 0.8\\ 0.0& 0.2\end{array}\right]x_i+\left[\begin{array}{c}0.2\\ 0.1\end{array}\right]u_i,i=1,\cdots ,5,$$

where $x_i\in {\mathbb{R}}^2$ and $u_i\in \mathbb{R}$. It is assumed that they communicate with each other using the communication graph in Figure 6. The graph in Figure 6 can be modeled by the Laplacian matrix L. The Laplacian matrix $L=\left[{\ell }_{ij}\right]$ of a directed graph in Figure 6 is constructed as defined above. The following is the Laplacian matrix for the graph in Figure 6.

$$L=\left[\begin{array}{ccccc}3& -1& -1& 0& -1\\ -1& 2& 0& -1& 0\\ 0& -1& 2& -1& 0\\ -1& 0& -1& 3& -1\\ 0& -1& -1& -1& 3\end{array}\right].$$

To collect the data required for calculating the DDLQR gain, a PRBS is applied to an agent for $(2N+L_D-1)$ steps, then all input, output, and state data are saved. Thereafter, the procedure for calculating the DDLQR gain is similar to the previous example. One important difference is that an arbitrarily large disc margin ${\overline{\Omega }}_{ϵ}$ can be obtained by setting the weighting matrices as $Q=ϵI$ and $R=I$ with sufficiently small $ϵ$. As can be seen in Figure 7, the disc margin gets larger as $ϵ$ becomes smaller. To satisfy the requirement that the disc margin ${\overline{\Omega }}_{ϵ}$ has to include $\Theta$, this simulation set $ϵ=0.01$ and $\rho =0.5$ for scaling. The resulting set ${\overline{\Omega }}_{ϵ}$ and $\Theta$ are shown in Figure 1.

Figure 8a shows that the proposed consensus control (23) leads to consensus of the MAS consisting of the nominal ANCBC agents. Since the proposed data-driven consensus control is based on the stability margin, it is expected that robust consensus is achieved. To validate this, we consider the following MAS, in which each agent has different input channel uncertainties, i.e., the proposed consensus control is applied to the following MAS.

$${\Sigma }_i:x_i^{+}=\left[\begin{array}{cc}1.0& 0.8\\ 0.0& 0.2\end{array}\right]x_i+\left[\begin{array}{c}0.2\\ 0.1\end{array}\right]p_i{u}_i,i=1,\cdots ,5,$$

where $p_i=\left[\begin{array}{ccccc}15& 14.9& 14& 13.5& 12.5\end{array}\right]$. Figure 8b shows that the designed consensus control based on the stability margin of the DDLQR results in consensus of the MAS even when there are uncertainties in the input channels. In other words, although the agents have different uncertainties in their input channels, such as (6), the consensus is achieved by the proposed controller as long as the uncertainties are in the disc margin.

6. Conclusions

In this paper, it is shown that the data-driven LQR (Linear Quadratic Regulator) provides a stability margin. To achieve this, the closed-loop system is written such that the input channel is affected by uncertainties. Then, how much the uncertainty can change while closed-loop stability is maintained using Lyapunov stability is investigated. Since the uncertainty is expressed as a complex number, the set of possible uncertainty guaranteeing the closed-loop stability is given by a disc in the complex plane, which is the stability margin of the data-driven LQR. Furthermore, it is shown that the derived disc margin can be used to design a control for consensus of a MAS consisting of asymptotically null controllable linear agents.

Future work includes a similar analysis of data-driven MPC (Model Predictive Control). Since stability analysis of MPC is done posteriorly after computing the control input using the optimal cost in MPC, it might be possible to apply the proposed method if the prediction horizon is short. Furthermore, it would be beneficial to devise data-driven disturbance rejection methods.