Data-Driven Suboptimal Scheduling of Switched Systems

Abstract

In this paper, a data-driven optimal scheduling approach is investigated for continuous-time switched systems with unknown subsystems and infinite-horizon cost functions. Firstly, a policy iteration (PI) based algorithm is proposed to approximate the optimal switching policy online quickly for known switched systems. Secondly, a data-driven PI-based algorithm is proposed online solely from the system state data for switched systems with unknown subsystems. Approximation functions are brought in and their weight vectors can be achieved step by step through different data in the algorithm. Then the weight vectors are employed to approximate the switching policy and the cost function. The convergence and the performance are analyzed. Finally, the simulation results of two examples validate the effectiveness of the proposed approaches.

1. Introduction

Switched systems consisting of several subsystems and a switching policy ruling the switching among them [1,2] arise in certain application situations [3,4] such as a system which has to collect data sequentially from a number of sensory sources and switches its attention among the data sources [5,6]. The switching among subsystems complicates the control problems and many of the problems remain to be open such as the optimal control problems. Optimal control [7,8] problems of switched systems have attracted considerable attention over the past few years. Thereinto, the optimal scheduling problem of switched systems is investigated in this paper.

Multiple approaches have been introduced to solve the optimal control problems for switched systems. Gradient-based approaches are investigated to solve the optimal switching time problems [9] and optimal scheduling problems [10,11] directly, usually in a finite time horizon. By utilizing control inputs to represent the switching policy, embedding approaches transform the optimal control problems of switched systems to traditional optimal control problems to address [12,13,14]. Adaptive dynamic programming (ADP) [15] approaches are introduced to solve the optimal control problems for switched systems with different initial conditions directly [16,17,18,19,20,21,22]. For optimal scheduling of switched systems with infinite-horizon cost functions, ADP approaches perform well to provide approximate global optimal solutions directly.

Approximate global optimal solutions are derived in feedback forms through ADP for optimal scheduling of switched systems with discrete-time dynamics and finite-horizon cost functions [17] or infinite-horizon cost functions [18]. Then further research is conducted for problems with switching cost [19] or state jumps [20]. For optimal scheduling of switched systems with continuous-time dynamics, an approximate feedback solution is proposed based on policy iteration (PI) algorithm with its offline, online, and concurrent implementation [21]. Then, a PI algorithm with recursive least squares is proposed and modified into a single loop PI algorithm to reduce the computational burden [22].

The aforementioned optimal control approaches are deduced based on the a priori knowledge of system models. However, not all system models can be completed acquired so that approaches independent of system models [23,24,25] require investigated. For this purpose, some model-free optimal control approaches have been studied for switched systems. Adaptive dynamic programming approaches are presented respectively for a continuous-time switched system with an infinite-horizon cost function [26] and a discrete-time switched system with a finite-horizon cost function [27] under the assumption that dynamic equations can be evaluated at some sets. Gradient-decent approaches only employing state data are proposed to solve optimal switching problems for continuous-time switched systems with finite-horizon cost functions [28,29]. Data-driven research utilizes real-life data measured by sensory sources to achieve the intrinsic information of systems [30] and can solve the problems of switched systems with unknown subsystems, such as the data-driven framework for discovering cyber-physical systems directly from the data [31]. Thereinto, data-driven ADP approaches provide possible solutions for optimal control problems of switched systems [32,33,34] and based on that, the following approach is designed to adapt to the complexity of switched systems which is brought by switching. Then in this paper, the data-driven optimal scheduling approach is investigated for continuous-time switched systems with unknown subsystems. At first, an online PI-based algorithm inspired by the off-policy learning method [35,36] is proposed to approximate the optimal solution quickly for optimal scheduling problems with known system models first and based on that, a data-driven PI-based algorithm is formulated for optimal scheduling of continuous-time switched systems with infinite-horizon cost functions, which don’t require that dynamic equations can be evaluated or known at some sets. Moreover, common online algorithms usually keep collecting data with the updating switching policies applied to the system at each iteration while the online algorithms in this paper take advantages of the data produced only by the initial switching policy.

The contribution of this paper is stated as follows: (1) with the data produced by the initial switching policy, an online PI-based algorithm is proposed to approximate the optimal solution quickly for optimal scheduling of known switched systems. (2) A data-driven PI-based algorithm is designed to solve optimal scheduling problems for switched systems with infinite-horizon cost functions and unknown subsystems, solely from the data produced by the initial switching policy, which has not been achieved well in existing literature as far as we know. (3) The convergence is proved and the optimality is analyzed.

The remainder of the paper is organized as follows. In Section 2, the problem is stated and the classic PI approach is introduced for the optimal scheduling problem. In Section 3, online PI-based algorithms are proposed for switched systems with known subsystems and unknown subsystems. In Section 4, simulation results are shown to indicate the effectiveness of the algorithms. Finally, the conclusion is drawn in Section 5.

2. Preliminaries

Consider the switched system as follows:

$$\dot{x}(t)=f_v(x(t)),x(0)=x_0,$$

(1)

where $x(t)\in R^n$ is the system state, $v\in V$ represents the index of the active subsystem, $V=\{1,2,\dots ,N\}$ is the index set of all subsystems, N is the number of subsystems, and $f_v:R^n\to R^n$ denotes the unknown dynamics of subsystem v. $f_v(v\in V)$ is Lipschitz continuous in $\Omega$ where $\Omega \subset R^n$ including the origin is the region of interest and there exists $v\in V$ such that $f_v(0)=0$.

In this paper, the problem to be addressed consists in seeking out the optimal switching policy $v^{\ast }$ to minimize the following cost function:

$$J={\int }_0^{\infty }Q(x(\tau ))d\tau ,$$

(2)

where $Q:R^n\to R$ is a positive definite function. During the time interval, the cost-to-go from the time t to infinity with the state $x(t)$ at time t can be described as [37]:

$$V(x(t))={\int }_t^{\infty }Q(x(\tau ))d\tau$$

(3)

and then

$$V(x(t))={\int }_t^{t+\delta t}Q(x(\tau ))d\tau +V(x(t+\delta t)).$$

(4)

For the optimal problem, the admissible switching policy should be introduced and the relevant assumption is made as [22].

Definition 1.

For system (1), a switching policy is called admissible with respect to the cost (2), if it stabilizes the system in Ω and for all $x_0\in \Omega$, the cost $V(x_0)$ is finite.

Assumption 1.

There exists at least one admissible policy for the system.

According to the Bellman principle of optimality, the optimal cost-to-go can be represented by

$$V^{\ast }(x(t))=\underset{v}{min}({\int }_t^{t+\delta t}Q(x(\tau ))d\tau +V^{\ast }(x(t+\delta t))),$$

(5)

and when $\delta t\to 0$, the corresponding optimal switching policy $v^{\ast }$ can be represented in a state-feedback form:

$$v^{\ast }(x)=\underset{v}{argmin}({\int }_t^{t+\delta t}Q(x(\tau ))d\tau +V^{\ast }(x(t+\delta t))).$$

(6)

When $\delta t\to 0$, with first-order Taylor expansion of $V^{\ast }(x(t+\delta t))$ applied in (5) and (6), the HJB equation can be given as [21]:

$$\underset{v(x)}{min}(Q(x)+{(\frac{\partial V^{\ast }(x)}{\partial x})}^T{f}_{v(x)}(x))=Q(x)+{(\frac{\partial V^{\ast }(x)}{\partial x})}^T{f}_{v^{\ast }(x)}(x)=0,$$

where $x=x(t)$, $\frac{\partial (·)}{\partial x}={[\frac{\partial (·)}{\partial x_1},\frac{\partial (·)}{\partial x_2},\dots ,\frac{\partial (·)}{\partial x_n}]}^T$ when $(·)$ is a scalar and the corresponding optimal switching policy $v^{\ast }$ is given as [21]:

$$v^{\ast }(x)=\underset{i\in V}{argmin}({(\frac{\partial V(x)}{\partial x})}^T{f}_i(x)).$$

Then, a PI approach can be applied to solve the optimal scheduling problem. Given an admissible switching policy $v^0$, the PI approach [21,22] is stated as follows:

$$Q(x)+{(\frac{\partial V^k(x)}{\partial x})}^T{f}_{v^k(x)}(x)=0$$

(7)

$$v^{k+1}(x)=\underset{i\in V}{argmin}({(\frac{\partial V^k(x)}{\partial x})}^T{f}_i(x)),$$

(8)

where k is the iteration number. The cost-to-go is solved in policy evaluation (7) and the switching policy is updated in policy improvement (8) iteratively. The stability and convergence are stated in the following theorem which has been proved in [21,38].

Theorem 1.

For the system (1) with the cost (2), if the value function sequence ${\left\{V^k\right\}}_{k=0}^{\infty }$ and the switching policy sequence ${\left\{v^k\right\}}_{k=0}^{\infty }$ are generated through (7) and (8) initiating from a stabilizing initial policy $v^0$, the value functions ${\left\{V^k\right\}}_{k=0}^{\infty }$ converge to the optimal value function $V^{\ast }$, and the switching policies ${\left\{v^k\right\}}_{k=0}^{\infty }$ stabilize the system in Ω.

3. Main Results

3.1. PI-Based Algorithm for Known Switched Systems

The offline, online and concurrent implementation of the aforementioned PI approach for switched systems with known dynamics has been investigated in references [21,22,38]. In this subsection, a novel online PI-based algorithm is proposed to approximate the optimal switching policy quickly.

Common online algorithms keep collecting data from the systems to evaluate and improve the switching policy. To be specific, once a new switching policy is produced at each iteration, it is applied to the system and the newly produced data is collected to calculate a new cost and a new switching policy. It entails considerable time to apply the new switching policy, collect new data and calculate a new cost and a new switching policy sequentially at each iteration. Aiming at this, inspired by the off-policy ADP methods for ordinary systems [36,39,40], an online PI-based algorithm is proposed to approximate the optimal switching policy quickly with only an initial switching policy applied and only the data produced by the initial switching policy is required. Next, the algorithm starts from the initial admissible switching policy.

With the initial admissible switching policy $v^0$ applied only, large amounts of data can be produced in the process and the state trajectory $x(t)$ corresponding to the switching policy $v^0$ can be acquired which will be employed in the subsequent derivation. Along the acquired state trajectory $x(t)$, $V^k(x(t+\delta t))-V^k(x(t))$ can be represented as

$$V^k(x(t+\delta t))-V^k(x(t))={\int }_t^{t+\delta t}{(\frac{\partial V^k(x)}{\partial x})}^T{f}_{v^0(x)}(x)d\tau ,$$

(9)

where $\delta t>0$ is very small. To combine the policy iteration and the cost at the acquired state, integrating (7) along the acquired state trajectory $x(t)$ and adding the integration to (9) yield

$$V^k(x(t+\delta t))-V^k(x(t))={\int }_t^{t+\delta t}{(\frac{\partial V^k(x)}{\partial x})}^T(f_{v^0(x)}(x)-f_{v^k(x)}(x))d\tau -{\int }_t^{t+\delta t}Q(x)d\tau .$$

(10)

The cost $V^k(x)$ is unknown. However, for all $x\in \Omega$, it can be expressed by:

$$V^k(x)=W^k\Phi (x)+e_{\Phi }^k(x),$$

(11)

where $\Phi (x)={[{\Phi }_1(x),{\Phi }_2(x),\dots ,{\Phi }_{N_w}(x)]}^T$ is a vector concerning a set of linearly independent basis functions ${\Phi }_j(x):R^n\to R(j=1,2,\dots ,N_w)$, $W^k\in R^{1\times N_w}$ is the weight vector and $e_{\Phi }^k(x)$ is the approximation error. $N_w$ is the number of the basis functions. A set of basis functions can constitute a particular basis of a function space and can almost approximate any function in the function space. With the approximation function (11) applied, Equation (10) can be transformed into

$$W^k[\Phi (x(t+\delta t))-\Phi (x(t))+{\int }_t^{t+\delta t}\frac{\partial \Phi (x)}{\partial x}(f_{v^k(x)}(x)-f_{v^0(x)}(x))d\tau ]=-{\int }_t^{t+\delta t}Q(x)d\tau +e_1^k(t),$$

(12)

where $\frac{\partial (·)}{\partial x}={[\frac{\partial {(·)}_1}{\partial x},\frac{\partial {(·)}_2}{\partial x},\dots ,\frac{\partial {(·)}_m}{\partial x}]}^T\in R^{m\times n}$ when $(·)={[{(·)}_1,{(·)}_2,\dots ,{(·)}_m]}^T\in R^m$ and $e_1^k(t)=e_{\Phi }^k(x(t))-e_{\Phi }^k(x(t+\delta t))+{\int }_t^{t+\delta t}\frac{\partial e_{\Phi }^k(x)}{\partial x}(f_{v^0(x)}(x)-f_{v^k(x)}(x))d\tau$ is the approximation error. Then the estimation can be achieved:

$${\widehat{W}}^k[\Phi (x(t+\delta t))-\Phi (x(t))+{\int }_t^{t+\delta t}\frac{\partial \Phi (x)}{\partial x}(f_{{\widehat{v}}^k(x)}(x)-f_{v^0(x)}(x))d\tau ]=-{\int }_t^{t+\delta t}Q(x)d\tau ,$$

(13)

where the estimate ${\widehat{v}}^k(x)$ is achieved through substituting the estimate ${\widehat{W}}^k$ and the approximation (11) into (8) as follows:

$${\widehat{v}}^{k+1}(x)=\underset{i\in V}{argmin}({\widehat{W}}^k\frac{\partial \Phi (x)}{\partial x}{f}_i(x)),$$

(14)

with the initial switching policy estimate ${\widehat{v}}^0(x)=v^0(x)$.

To employ the acquired state data corresponding to some selected instants, some data matrices can be defined as follows:

$$\overline{\Phi }(t_r)=\Phi (x(t_r+\delta t))-\Phi (x(t_r))$$

$$b(t_r)={\int }_{t_r}^{t_r+\delta t}\frac{\partial \Phi (x)}{\partial x}(f_{{\widehat{v}}^k(x)}(x)-f_{v^0(x)}(x))d\tau$$

$$d(t_r)={\int }_{t_r}^{t_r+\delta t}Q(x)d\tau$$

where $t_1<t_2<\cdots <t_l$ are the selected instants, l is a positive integer, $r=1,2,\dots l$, $\overline{\Phi }(t_r)\in R^{N_w}$, $b(t_r)\in R^{N_w}$ and $d(t_r)\in R$. The following assumption concerning the data matrices is made as [40,41]:

Assumption 2.

There exist a positive integer $\overline{L}$ and a positive number α such that for all $L\ge \overline{L}$, the following equality holds:

$$\frac{1}{L}\sum _{r=1}^L(\overline{\Phi }(t_r)+b(t_r)){(\overline{\Phi }(t_r)+b(t_r))}^T\ge \alpha I_{N_w}.$$

In optimal control of ordinary systems, this kind of assumption can be satisfied by exerting an exploration noise in the input [39,40]. In the case of the switched systems, according to [21,22], it can be satisfied through random switching.

Then based on Assumption 2, ${\widehat{W}}^k$ can be achieved with the data matrices as the following formula:

$${\widehat{W}}^k=-[\sum _{r=1}^{L}d(t_r){(\overline{\Phi }(t_r)+b(t_r))}^T]·{[\sum _{r=1}^L(\overline{\Phi }(t_r)+b(t_r)){(\overline{\Phi }(t_r)+b(t_r))}^T]}^{-1}.$$

(15)

According to the analysis, the online PI-based algorithm for switched systems with known dynamics can be formulated in Algorithm 1.

Algorithm 1 Online policy iteration (PI)-based Algorithm.

Step 1. Start with an initial admissible switching policy $v^0(x)$ and set the iteration index $k=0$. Step 2. Apply $v^0(x)$ in the switched systems and acquire the state data. Set ${\widehat{v}}^0(x)=v^0(x)$. Calculate $\overline{\Phi }(t_r)$ and $d(t_r)$ for $r=1,2,\dots ,l$ according to their definition with the state data. Step 3. Calculate $b(t_r)$ for $r=1,2,\dots ,l$ according to its definition with the state data and then calculate ${\widehat{W}}^k$ from (15). Step 4. Update the switching policy ${\widehat{v}}^{k+1}(x)$ as (14). Step 5. If $∥{\widehat{W}}_{}^k-{\widehat{W}}_{}^{k-1}∥<\epsilon$, set ${\widehat{W}}_{}^{\ast }={\widehat{W}}_{}^k$ and exit. Otherwise, set $k=k+1$ and go back to Step 3.

In the algorithm, when the switching policy $v^0(x)$ is applied, the corresponding system state data can be obtained. With multiple samples, multiple data matrix $\overline{\Phi }(t_r)$ and $b(t_r)$ will be calculated. Sufficient samples should be employed to satisfy that $\frac{1}{L}{\displaystyle \sum _{r=1}^L}(\overline{\Phi }(t_r)+b(t_r)){(\overline{\Phi }(t_r)+b(t_r))}^T\ge \alpha I_{N_w}$ with a proper $\alpha$ so that the sampling stage must be long enough to ensure that. Moreover, since the weight vector ${\widehat{W}}^k$ to be solved in Equation (15) has $N_w$ components, at least $N_w$ instants $t_r$ should be employed to solve ${\widehat{W}}^k$ so that r is no less than $N_w$. Then through the algorithm, the approximate optimal cost function can be achieved as ${\widehat{V}}^{\ast }(x)={\widehat{W}}^{\ast }\Phi (x)$ with the corresponding approximate optimal switching policy ${\widehat{v}}^{\ast }(x)=\underset{i\in V}{argmin}({\widehat{W}}^{\ast }\frac{\partial \Phi (x)}{\partial x}{f}_i(x))$. It can be seen from Algorithm 1 that only the initial switching policy is applied and then Algorithm 1 can approximate the optimal switching policy quickly with the data produced by the initial switching policy.

3.2. Data-Driven PI-Based Algorithm for Unknown Switched Systems

In this subsection, based on the proposed PI approach dependent on system models, a data-driven PI-based algorithm is proposed for switched systems with unknown subsystems. The algorithm takes full advantage of data produced by an initial switching policy to approximate the optimal switching policy quickly.

From Section 3.1, with the initial admissible switching policy $v^0$ applied and along the acquired state trajectory $x(t)$, (10) can be achieved and the cost function can be represented by (11). In the process, $\frac{\partial V^k(x)}{\partial x}$ can be represented by ${(W^k\frac{\partial \Phi (x)}{\partial x})}^T$ and solved with known system models. However, due to the unknown subsystem models, $W^k\frac{\partial \Phi (x)}{\partial x}{f}_i(x)$ representing ${(\frac{\partial V^k(x)}{\partial x})}^T{f}_i(x)$ cannot work well as Section 3.1. So another approximation function is brought in to solve the problem. For all $x\in \Omega$, the unknown variable ${(\frac{\partial V^k(x)}{\partial x})}^T{f}_i(x),i\in V$ can be expressed by:

$${(\frac{\partial V^k(x)}{\partial x})}^T{f}_i(x)=C_i^k\Psi (x)+e_{\Psi ,i}^k(x),$$

(16)

where $\Psi (x)={\left[{\Psi }_1(x),{\Psi }_2(x),\dots ,{\Psi }_{N_c}(x)\right]}^T$ is a vector concerning a set of linearly independent basis functions ${\Psi }_j(x):R^n\to R(j=1,2,\dots ,N_c)$, $C_i^k,i\in V$ are the weight vectors and $e_{\Psi ,i}^k(x)$ is the approximation error. $N_c$ is the number of the basis functions.

With the approximation functions (11) and (16) applied, Equation (10) can be transformed into

$$W^k(\Phi (x(t+\delta t))-\Phi (x(t)))+{\int }_t^{t+\delta t}(C_{v^k(x)}^k-C_{v^0(x)}^k)\Psi (x)d\tau =-{\int }_t^{t+\delta t}Q(x)d\tau +e_2^k(t),$$

(17)

where $e_2^k(t)={\int }_t^{t+\delta t}(e_{\Psi ,v^0(x)}^k(x)-e_{\Psi ,v^k(x)}^k(x))d\tau +e_{\Psi }^k(x(t))-e_{\Psi }^k(x(t+\delta t))$ is the approximation error. Since $\delta t$ is very small, the values of $v^k(x)$ and $v^0(x)$ can be seen to be constant during the time interval $[t,t+\delta t)$ so that $C_{v^0(x)}^k$ and $C_{v^k(x)}^k$ are constant during the time interval $[t,t+\delta t)$. Therefore, the estimation can be made as (18):

$${\widehat{W}}^k(\Phi (x(t+\delta t))-\Phi (x(t)))+({\widehat{C}}_{{\widehat{v}}^k(x)}^k-{\widehat{C}}_{v^0(x)}^k){\int }_t^{t+\delta t}\Psi (x)d\tau =-{\int }_t^{t+\delta t}Q(x)d\tau .$$

(18)

For the subsequent algorithm, in addition to the data matrices defined in Section 3.1, some more matrices are required which are defined as follows:

$$g(t_r)={\int }_{t_r}^{t_r+\delta t}\Psi (x(\tau ))d\tau$$

where $t_1<t_2<\cdots <t_l$, l is a positive integer, $g(t_r)\in R^{N_c}$ and $r=1,2,\dots l$.

At first, the following assumption concerning the data matrices is made as Assumption 2:

Assumption 3.

There exist a positive integer ${\overline{L}}^{\prime }$ and a positive number α such that for all $L\ge {\overline{L}}^{\prime }$, the following equalities hold:

$$\frac{1}{L}\sum _{r=1}^L\overline{\Phi }(t_r)\overline{\Phi }{(t_r)}^T\ge \alpha I_{N_w}$$

where the time instants $t_r$ satisfies the condition that ${\widehat{v}}^k(x(t_r))=v^0(x(t_r))$;

$$\frac{1}{L}\sum _{r=1}^L\Psi (x(t_r))\Psi {(x(t_r))}^T\ge \alpha I_{N_c}$$

where the time instants $t_r$ satisfies the condition that $v^0(x(t_r))=i$ for $\forall i\in V$;

$$\frac{1}{L}\sum _1^{L}g(t_r)g{(t_r)}^T\ge \alpha I_{N_c}$$

where the time instants $t_r$ satisfies the conditions that ${\widehat{v}}^k(x(t_r))=i$ and $v^0(x(t_r))=j$ for $\forall i\ne j,(i,j\in V)$.

Remark 1.

Since the vector ${\widehat{C}}_{{\widehat{v}}^k(x)}^k-{\widehat{C}}_{v^0(x)}^k$ changes as x changes, the weight vector ${\widehat{W}}^k$ and ${\widehat{C}}_{{\widehat{v}}^k(x)}^k$ cannot be solved directly from (18) with the data matrices as Section 3.1. The difficulty in this problem is mainly calculating the weight vector ${\widehat{W}}^k$ and ${\widehat{C}}_{{\widehat{v}}^k(x)}^k$ or achieve enough useful knowledge about the weight vectors through the state data.

Based on the above analysis, the following approach is designed to acquire useful knowledge about the weight vector ${\widehat{W}}^k$ and ${\widehat{C}}_{{\widehat{v}}^k(x)}^k$ step by step through different state data. The weight vector $W^k$ is discussed firstly. The state data satisfying the condition that ${\widehat{v}}^k(x)=v^0(x)$ is selected from all the state data and then utilized in (18). It is obvious that when ${\widehat{v}}^k(x)=v^0(x)$, ${\widehat{C}}_{{\widehat{v}}^k(x)}^k={\widehat{C}}_{v^0(x)}^k$ and (18) can be simplified to:

$${\widehat{W}}^k(\Phi (x(t+\delta t))-\Phi (x(t)))=-{\int }_t^{t+\delta t}Q(x)d\tau .$$

(19)

Under Assumption 3, the estimate of the weight vector $W^k$ can be achieved from (19) with the data matrices defined in Section 3.1 concerning certain states as follows:

$${\widehat{W}}^k=-(\sum _{r=1}^{L}d(t_r)\overline{\Phi }{(t_r)}^T){(\sum _{r=1}^L\overline{\Phi }(t_r)\overline{\Phi }{(t_r)}^T)}^{-1},$$

(20)

where the time instants $t_r$ satisfies the condition that ${\widehat{v}}^k(x(t_r))=v^0(x(t_r))$.

Then, we concentrate on estimating the weight vector of $C_i^k$. Along the acquired state trajectory $x(t)$ produced by the switching policy $v^0$, it can be obtained from (16) and (7) that when $k=0$ the following formula holds:

$${\int }_t^{t+\delta t}{C}_{v^0(x)}^0\Psi (x)d\tau +{\int }_t^{t+\delta t}{e}_{\Psi ,v^0(x)}^0(x)d\tau +{\int }_t^{t+\delta t}Q(x)d\tau =0.$$

(21)

The estimation can be made as follows:

$${\widehat{C}}_{v^0(x)}^0{\int }_t^{t+\delta t}\Psi (x)d\tau +{\int }_t^{t+\delta t}Q(x)d\tau =0.$$

Under Assumption 3, the estimate of the weight vector $C_i^0$ can be easily achieved as follows:

$${\widehat{C}}_i^0=-[\sum _1^{L}d(t_r)g{(t_r)}^T]{(\sum _1^{L}g(t_r)g{(t_r)}^T)}^{-1},$$

(22)

where the time instants $t_r$ satisfies the condition that $v^0(x(t_r))=i(i\in V)$.

Though the weight vector $C_i^k$ is not easy to estimate, the estimate of $C_i^k-C_j^k$$(i\ne j)$ can be easily achieved from (18) with the data matrices concerning certain states under Assumption 3 as follows:

$${\widehat{C}}_i^k-{\widehat{C}}_j^k=-[\sum _1^L(d(t_r)+{\widehat{W}}^k\overline{\Phi }(t_r))g{(t_r)}^T]{(\sum _1^{L}g(t_r)g{(t_r)}^T)}^{-1},$$

(23)

where the time instants $t_r$ satisfies the conditions that ${\widehat{v}}^k(x(t_r))=i$ and $v^0(x(t_r))=j$$(i,j\in V$ and $i\ne j)$.

For a certain state x, ${(\frac{\partial V^k(x)}{\partial x})}^T{f}_{v^0(x)}(x)$ is constant and $\underset{i\in V}{argmin}({(\frac{\partial V^k(x)}{\partial x})}^T{f}_i(x))=\underset{i\in V}{argmin}({(\frac{\partial V^k(x)}{\partial x})}^T{f}_i(x)-{(\frac{\partial V^k(x)}{\partial x})}^T{f}_{v^0(x)}(x))$. Therefore, on the basis of (8), employing the estimate of (22) and (23), the switching policy $v^{k+1}(x)$ is estimated by

$${\widehat{v}}^{k+1}(x)=\left\{\begin{array}{cc}\underset{i\in V}{argmin}({\widehat{C}}_i^0\Psi (x))& k=0\\ \underset{i\in V}{argmin}({\widehat{C}}_i^k-{\widehat{C}}_{v^0(x)}^k)\Psi (x)& k>0\end{array}\right.$$

(24)

with the initial switching policy estimate ${\widehat{v}}^0(x)=v^0(x)$.

The approach utilizes different parts of the acquired state data to calculate the weight vector ${\widehat{W}}^k$, $C_i^0$ and $C_i^k-C_j^k$$(i\ne j)$ respectively. Then the weight vectors are employed to calculate the switching policy and the cost function.

According to the aforementioned analysis, the data-driven PI-based algorithm for switched systems can be formulated as follows:

Then, the approximate optimal cost function can be achieved as ${\widehat{V}}^{\ast }(x)={\widehat{W}}^{\ast }\Phi (x)$ with the corresponding approximate optimal switching policy ${\widehat{v}}^{\ast }(x)=\underset{i\in V}{argmin}(({\widehat{C}}_i^k-{\widehat{C}}_{v^0(x)}^k)\Psi (x))$.

Remark 2.

In Algorithm 2, only the initial switching policy $v^0(x)$ requires to be applied in the system and the produced state data is collected at the beginning. The data matrices $\overline{\Phi }(t_r)$, $d(t_r)$ and $g(t_r)$ are calculated once at the beginning and don’t require to be calculated repeatedly at each iteration. It is very convenient and timesaving to operate online according to Algorithm 2 and the calculation time is very short. Therefore, the optimal cost can be approximated rapidly in Algorithm 2. Moreover, Algorithm 2 is only based on data with no need for the knowledge of subsystems.

Algorithm 2 Data-driven PI-based algorithm.

Step 1. Start with an initial admissible switching policy $v^0(x)$ and set the iteration index $k=0$. Step 2. Apply $v^0(x)$ in the switched systems and acquire the state data. Set ${\widehat{v}}^0(x)=v^0(x)$. Calculate $\overline{\Phi }(t_r)$, $d(t_r)$ and $g(t_r)$ for $r=1,2,\dots ,l$ according to their definition with the state data. Step 3. Calculate ${\widehat{C}}_i^0$ for all $i\in V$ from (22) and then calculate ${\widehat{v}}^1(x)$ from (24). Set $k=1$. Step 4. Calculate ${\widehat{W}}^k$ from (20) and then calculate ${\widehat{C}}_i^k-{\widehat{C}}_j^k$ from (23) for all $i\ne j$$(i,j\in V)$. Step 5. Update the switching policy ${\widehat{v}}^{k+1}(x)$ as (24). Step 6. If $∥{\widehat{W}}_{}^k-{\widehat{W}}_{}^{k-1}∥<\epsilon$, set ${\widehat{W}}_{}^{\ast }={\widehat{W}}_{}^k$ and exit. Otherwise, set $k=k+1$ and go back to Step 4.

Next, the convergence of Algorithm 2 is analyzed. Based on Theorem 1, Theorem 2 is stated for the convergence analysis of Algorithm 2.

Theorem 2.

For system (1) with cost (2) under Assumptions 1 and 3, if the value function sequence ${\left\{V^k\right\}}_{k=0}^{\infty }$ and the switching policy sequence ${\left\{v^k\right\}}_{k=0}^{\infty }$ are generated through (7) and (8) initiating from an initial admissible policy $v^0$, for $\forall \epsilon >0$ and all $x\in \Omega$, there exists $\overline{N}>0$ such that for $\forall N>\overline{N}$, when $\delta t$ approaches to zero, the following inequalities hold:

$$\left|{\widehat{V}}^k(x)-V^k(x)\right|<\epsilon ,{\widehat{v}}^k(x)=v^k(x),$$

where $N=min\{N_w,N_c\}$, ${\widehat{V}}^k(x)={\widehat{W}}^k\Phi (x)$, ${\widehat{W}}^k$ and ${\widehat{v}}^k$ are generated through (20) and (24) in Algorithm 2, and $k=0,1,\dots$.

Proof of Theorem 2.

Mathematical induction is utilized to prove the convergence.

Firstly, we discuss the theorem when $k=0$. When $k=0$, ${\widehat{v}}^0(x)=v^0(x)$, ${\widehat{C}}_{{\widehat{v}}^0(x)}^k={\widehat{C}}_{v^0(x)}^k$ and it can be achieved from (17) and (18) that:

$$({\widehat{W}}^0-W^0)(\Phi (x(t+\delta t))-\Phi (x(t)))=-e_2^0(t)$$

and

$$\sum _{r=1}^L({\widehat{W}}^0-W^0)\overline{\Phi }(t_r){(\overline{\Phi }(t_r))}^T{({\widehat{W}}^0-W^0)}^T=\sum _{r=1}^L{e}_2^0{(t_r)}^2.$$

According to Assumption 3, it follows that

$$\frac{1}{L}\sum _{r=1}^L\overline{\Phi }(t_r){(\overline{\Phi }(t_r))}^T\ge \alpha I_{N_w}$$

and then ${∥{\widehat{W}}^0-W^0∥}^2\le \frac{1}{\alpha L}{\displaystyle \sum _{r=1}^L}{e}_2^0{(t_r)}^2$. The approximation theory [42] yields that for all $x\in \Omega$, $\underset{N_w\to \infty }{lim}{e}_{\Phi }^0(x)=0$, then $\underset{N_w\to \infty }{lim}{e}_2^0(t_r)=0$. Therefore, for $\forall \epsilon >0$ and $\forall x\in \Omega$, there exists ${\overline{N}}_w>0$ such that for $\forall N_w>{\overline{N}}_w$, ${∥{\widehat{W}}^0-W^0∥}^2\le \epsilon$ and then $\left|{\widehat{V}}^0(x)-V^0(x)\right|<\epsilon$. It follows that $\underset{N_w\to \infty }{lim}∥{\widehat{W}}^0-W^0∥=0$ and $\underset{N_w\to \infty }{lim}{\widehat{V}}^0(x)=V^0(x)$ for $\forall x\in \Omega$. Then, it can be achieved from (21) and (22) that:

$$({\widehat{C}}_{v^0(x)}^0-C_{v^0(x)}^0){\int }_t^{t+\delta t}\Psi (x)d\tau ={\int }_t^{t+\delta t}{e}_{\Psi ,v^0(x)}^0(x)d\tau +{\int }_t^{t+\delta t}(C_{v^0(x(\tau ))}^0-C_{v^0(x(t))}^0)\Psi (x)d\tau .$$

Similarly, the approximation theory [42] yields that for all $x\in \Omega$, $\underset{N_w\to \infty }{lim}{e}_{\Psi ,i}^0(x)=0(i\in V)$ and then $\underset{\delta t\to 0}{lim}{\int }_t^{t+\delta t}(C_{v^0(x(\tau ))}^0-C_{v^0(x(t))}^0)\Psi (x)d\tau =0$. Under Assumption 3, it can be deduced that for $\forall x\in \Omega$, $\underset{\genfrac{}{}{0pt}{}{N_c\to \infty }{\delta t\to 0}}{lim}(\frac{\partial V^0(x)}{\partial x}{\widehat{)}}^T{f}_i(x)={(\frac{\partial V^0(x)}{\partial x})}^T{f}_i(x)$. It implies that $\underset{\genfrac{}{}{0pt}{}{N_c\to \infty }{\delta t\to 0}}{lim}{\widehat{v}}^1(x)=v^1(x)$ for $\forall x\in \Omega$.

Secondly, we consider the theorem when $k=1$. When $k=1$, $\underset{N_c\to \infty }{lim}{\widehat{C}}_{{\widehat{v}}^1(x)}^1={\widehat{C}}_{v^1(x)}^1$. When ${\widehat{W}}^1$ is calculated, for $\forall x\in \Omega$ and it can be achieved from (17) to (19) that:

$$\begin{array}{c}({\widehat{W}}^1-W^1)(\Phi (x(t+\delta t))-\Phi (x(t)))+({\widehat{C}}_{{\widehat{v}}^1(x)}^1-{\widehat{C}}_{v^0(x)}^1){\int }_t^{t+\delta t}\Psi (x)d\tau -\hfill \\ {\int }_t^{t+\delta t}(C_{v^1(x)}^1-C_{v^0(x)}^1)\Psi (x)d\tau =-e_2^1(t),\hfill \end{array}$$

where ${\widehat{v}}^1(x)=v^0(x)$ and thus $\underset{N_c\to \infty }{lim}{v}^1(x)=v^0(x)$ for $\forall x\in \Omega$. Then it can be inferred that when $N_c\to \infty$

$$({\widehat{W}}^1-W^1)(\Phi (x(t+\delta t))-\Phi (x(t)))=-e_2^1(t)$$

Under Assumption 3, it can be deduced that for all $x\in \Omega$, $\underset{N_w\to \infty }{lim}{e}_2^1(t_r)=0$, $\underset{N_w,N_c\to \infty }{lim}∥{\widehat{W}}^1-W^1∥=0$ and $\underset{N_w,N_c\to \infty }{lim}{\widehat{V}}^1(x)=V^1(x)$. Then, it can be achieved from (17), (18) and (23) that:

$$\begin{array}{c}({\widehat{C}}_{{\widehat{v}}^1(x)}^1-{\widehat{C}}_{v^0(x)}^1){\int }_t^{t+\delta t}\Psi (x)d\tau -{\int }_t^{t+\delta t}(C_{v^1(x)}^1-C_{v^0(x)}^1)\Psi (x)d\tau \hfill \\ =-e_2^1(t)-({\widehat{W}}^1-W^1)(\Phi (x(t+\delta t))-\Phi (x(t))).\hfill \end{array}$$

Since it can be achieved that ${\int }_t^{t+\delta t}(C_{v^1(x)}^1-C_{v^0(x)}^1)\Psi (x)d\tau =(C_{v^1(x)}^1-C_{v^0(x)}^1){\int }_t^{t+\delta t}\Psi (x)d\tau +e_3^1(t)$ where $e_3^1(t)={\int }_t^{t+\delta t}((C_{v^1(x(\tau ))}^1-C_{v^1(x(t))}^1)-(C_{v^0(x(\tau ))}^1-C_{v^0(x(t))}^1)\Psi (x)d\tau$, it can be inferred that

$$\begin{array}{c}[({\widehat{C}}_{{\widehat{v}}^1(x)}^1-{\widehat{C}}_{v^0(x)}^1)-(C_{v^1(x)}^1-C_{v^0(x)}^1)]{\int }_t^{t+\delta t}\Psi (x)d\tau \hfill \\ =e_3^1(t)-e_2^1(t)-({\widehat{W}}^1-W^1)(\Phi (x(t+\delta t))-\Phi (x(t))).\hfill \end{array}$$

Since $\underset{\delta t\to 0}{lim}{e}_3^1(t)=0$, $\underset{N_w\to \infty }{lim}{e}_2^1(t)=0$ and $\underset{N_w,N_c\to \infty }{lim}∥{\widehat{W}}^1-W^1∥=0$, under Assumption 3, it can be deduced that $\underset{\genfrac{}{}{0pt}{}{N_w,N_c\to \infty }{\delta t\to 0}}{lim}∥({\widehat{C}}_{{\widehat{v}}^1(x)}^1-{\widehat{C}}_{v^0(x)}^1)-(C_{v^1(x)}^1-C_{v^0(x)}^1)∥=0$ and then $\underset{\genfrac{}{}{0pt}{}{N_w,N_c\to \infty }{\delta t\to 0}}{lim}({\widehat{C}}_{{\widehat{v}}^1(x)}^1-{\widehat{C}}_{v^0(x)}^1)\Psi (x)=(C_{v^1(x)}^1-C_{v^0(x)}^1)\Psi (x)$ for $\forall x\in \Omega$. It implies that $\underset{\genfrac{}{}{0pt}{}{N_w,N_c\to \infty }{\delta t\to 0}}{lim}{\widehat{v}}^2(x)=v^2(x)$ for $\forall x\in \Omega$.

Thirdly, we suppose the theorem holds when $k=k$. That is to say, $\underset{\genfrac{}{}{0pt}{}{N_w,N_c\to \infty }{\delta t\to 0}}{lim}{\widehat{V}}^k(x)=V^k(x)$ and $\underset{\genfrac{}{}{0pt}{}{N_w,N_c\to \infty }{\delta t\to 0}}{lim}{\widehat{v}}^{k+1}(x)=v^{k+1}(x)$ for $\forall x\in \Omega$. When ${\widehat{W}}^{k+1}$ is calculated, it can be achieved from (17) to (19) that:

$$\begin{array}{c}({\widehat{W}}^{k+1}-W^{k+1})(\Phi (x(t+\delta t))-\Phi (x(t)))+({\widehat{C}}_{{\widehat{v}}^{k+1}(x)}^{k+1}-{\widehat{C}}_{v^0(x)}^{k+1}){\int }_t^{t+\delta t}\Psi (x)d\tau -\hfill \\ {\int }_t^{t+\delta t}(C_{v^{k+1}(x)}^{k+1}-C_{v^0(x)}^{k+1})\Psi (x)d\tau =-e_2^{k+1}(t)\hfill \end{array}$$

where ${\widehat{v}}^{k+1}(x)=v^0(x)$ and $\underset{\genfrac{}{}{0pt}{}{N_w,N_c\to \infty }{\delta t\to 0}}{lim}{v}^{k+1}(x)=v^0(x)$ for $\forall x\in \Omega$. Then it can be inferred that

$$({\widehat{W}}^{k+1}-W^{k+1})(\Phi (x(t+\delta t))-\Phi (x(t)))=-e_2^{k+1}(t)$$

Under Assumption 3, it can be deduced that $\underset{\genfrac{}{}{0pt}{}{N_w,N_c\to \infty }{\delta t\to 0}}{lim}∥{\widehat{W}}^{k+1}-W^{k+1}∥=0$ and $\underset{\genfrac{}{}{0pt}{}{N_w,N_c\to \infty }{\delta t\to 0}}{lim}{\widehat{V}}^{k+1}(x)=V^{k+1}(x)$ for $\forall x\in \Omega$. Then, it can be achieved from (17), (18) and (23) that:

$$\begin{array}{c}[({\widehat{C}}_{{\widehat{v}}^{k+1}(x)}^{k+1}-{\widehat{C}}_{v^0(x)}^{k+1})-(C_{v^{k+1}(x)}^{k+1}-C_{v^0(x)}^{k+1})]{\int }_t^{t+\delta t}\Psi (x)d\tau \hfill \\ =-({\widehat{W}}^{k+1}-W^{k+1})(\Phi (x(t+\delta t))-\Phi (x(t)))-e_2^{k+1}(t)+e_3^{k+1}(t),\hfill \end{array}$$

where $e_3^{k+1}(t)={\int }_t^{t+\delta t}((C_{v^{k+1}(x(\tau ))}^{k+1}-C_{v^{k+1}(x(t))}^{k+1})-(C_{v^0(x(\tau ))}^{k+1}-C_{v^0(x(t))}^{k+1})\Psi (x)d\tau$. Since $\underset{\delta t\to 0}{lim}{e}_3^{k+1}(t)=0$, $\underset{N_w\to \infty }{lim}{e}_2^{k+1}(t)=0$ and $\underset{\genfrac{}{}{0pt}{}{N_w,N_c\to \infty }{\delta t\to 0}}{lim}∥{\widehat{W}}^{k+1}-W^{k+1}∥=0$, under Assumption 3, it can be deduced that $\underset{\genfrac{}{}{0pt}{}{N_w,N_c\to \infty }{\delta t\to 0}}{lim}∥({\widehat{C}}_{{\widehat{v}}^{k+1}(x)}^{k+1}-{\widehat{C}}_{v^0(x)}^{k+1})-(C_{v^{k+1}(x)}^{k+1}-C_{v^0(x)}^{k+1})∥=0$ and then $\underset{\genfrac{}{}{0pt}{}{N_w,N_c\to \infty }{\delta t\to 0}}{lim}({\widehat{C}}_{{\widehat{v}}^{k+1}(x)}^{k+1}-{\widehat{C}}_{v^0(x)}^{k+1})\Psi (x)=(C_{v^{k+1}(x)}^{k+1}-C_{v^0(x)}^{k+1})\Psi (x)$ for $\forall x\in \Omega$. It implies that $\underset{\genfrac{}{}{0pt}{}{N_w,N_c\to \infty }{\delta t\to 0}}{lim}{\widehat{v}}^{k+2}(x)=v^{k+2}(x)$ for $\forall x\in \Omega$. In brief, it can be deduced that the theorem holds when $k=k+1$ from the supposition that the theorem holds when $k=k$.

The proof is completed through mathematical induction. □

It can be achieved from Theorem 2 that the value function ${\widehat{V}}^k(x)$ generated through Algorithm 2 is an approximation of $V^k(x)$ and the corresponding switching policy is ${\widehat{v}}^k(x)=v^k(x)$ if the preconditions are satisfied. Theorem 2 combined with Theorem 1 indicates that the value function ${\widehat{V}}^{\ast }(x)={\widehat{W}}^{\ast }\Phi (x)$ is the approximate optimal cost function with the corresponding approximate optimal switching policy ${\widehat{v}}^{\ast }(x)=\underset{i\in V}{argmin}(({\widehat{C}}_i^k-{\widehat{C}}_{v^0(x)}^k)\Psi (x))$.

Remark 3.

In practice, the error of the approximation exists and the small parameter $\delta t$ can not get infinitely close to zero so that the solution calculated from Algorithm 2 is suboptimal.

4. Example

In this section, two examples are illustrated to validate the effectiveness of the suboptimal scheduling approach of Algorithm 1 and the data-driven suboptimal scheduling approach of Algorithm 2 in this paper.

Example 1.

Consider a switched system as [21,38] consisting of the following subsystems:

$$\dot{x}(t)=f_1(x(t))=-x(t),\dot{x}(t)=f_2(x(t))=-x^3(t)$$

with $x(0)=2$ and $Q(x(t))=x^2(t)$.

The optimal switching policy can be known from [17] as

$$v^{\ast }(x)=\left\{\begin{array}{cc}1& x\le 1\\ 2& x>1\end{array}\right.$$

Choose the initial switching policy $v^0(x)=1$ when $x\le 1.5$ and $v^0(x)=2$ when $x>1.5$. The basis functions are $\Phi (x)={[x^2,x^4,x^6,x^8,x^{10}]}^T$ selected as [21]. Set the sample period $\delta t=0.002$ s.

Apply Algorithm 1 and utilize the online state data from $t=0$ to $0.2$ s. Then after calculation in $0.02$ s, the approximate optimal cost is achieved with the corresponding approximate optimal switching policy through 3 iterations. The initial cost ${\widehat{V}}^0$, the approximate optimal cost ${\widehat{V}}^k$ and the optimal cost $V^{\ast }$ are demonstrated in Figure 1. The largest error between ${\widehat{V}}^k$ and $V^{\ast }$ is 0.1621 and it is obvious that the approximate optimal cost ${\widehat{V}}^k$ is very close to the optimal cost of $V^{\ast }$. The state trajectories with the initial switching policy $v^0$, the approximate optimal switching policy ${\widehat{v}}^k$ and the optimal switching policy $v^{\ast }$ applied in the system after $t=0.25$ s are demonstrated in Figure 2, where the state trajectory corresponding to ${\widehat{v}}^k$ coincides with the one corresponding to $v^{\ast }$ and their largest error is zero while the largest error between the state trajectory corresponding to $v^0$ and the one corresponding to $v^{\ast }$ is 0.0563. The online trajectories are too close to show the superiority of the proposed algorithm so that the switching policies $v^0$, ${\widehat{v}}^k$ and $v^{\ast }$ when $x\in [-2,2]$ are illustrated in Figure 3. Apparently, ${\widehat{v}}^k$ and $v^{\ast }$ are the same. The similarity rate of the schedules ${\widehat{v}}^k$ and $v^{\ast }$ is 100% while the similarity rate of the schedules $v^0$ and $v^{\ast }$ is 75.31%.

Apply Algorithm 2 and utilize the online state data from $t=0$ to $0.2$ s. Then after calculation in $0.05$ s, the approximate optimal cost is achieved with the corresponding approximate optimal switching policy through 6 iterations. The costs ${\widehat{V}}^0$, ${\widehat{V}}^k$ and $V^{\ast }$ are demonstrated in Figure 4. It can be obtained that compared with the initial cost of ${\widehat{V}}^0$, the approximate optimal cost of ${\widehat{V}}^k$ is close to the optimal cost of $V^{\ast }$. The state trajectories with $v^0$, ${\widehat{v}}^k$ and $v^{\ast }$ applied in the system after $t=0.25$ s are demonstrated in Figure 5, where the state trajectory corresponding to ${\widehat{v}}^k$ coincides with the one corresponding to $v^{\ast }$ and their largest error is 0.0155 while the largest error between the state trajectory corresponding to $v^0$ and the one corresponding to $v^{\ast }$ is 0.0563. The online trajectories also are too close to show the superiority of the proposed algorithm so that the switching policies $v^0$, ${\widehat{v}}^k$ and $v^{\ast }$ when $x\in [-2,2]$ are illustrated in Figure 6. It can be seen that ${\widehat{v}}^k$ is approximate to $v^{\ast }$. The similarity rate of the schedules ${\widehat{v}}^k$ and $v^{\ast }$ is 95.06% while the similarity rate of the schedules $v^0$ and $v^{\ast }$ is 75.31%.

Example 2.

Consider a mass-spring-damper system as [21,22]:

$$\begin{array}{c}{\dot{x}}_1(t)=x_2(t),\hfill \\ {\dot{x}}_2(t)=F_v-0.1x_1(t)-0.1x_2(t)\hfill \end{array}$$

with $v\in \{1,2,3\}$, $F_1=1$, $F_2=-1$ and $F_3=0$. Here, the state $x_1(t)$ is the displacement of the mass measured from the relaxed length of the spring. $F_v$ is the external force acting on the mass. The initial state is $x(0)=[2,2]$ and the function Q is

$$Q(x(t))=x{(t)}^T\left[\begin{array}{cc}1& 0\\ 0& 1.\end{array}\right]x(t)$$

Choose the initial switching policy $v^0(x)=3$ when $\left|x_1\right|\le 0.5$, $v^0(x)=2$ when $x_1>0.5$ and $v^0(x)=1$ when $x_1<-0.5$. The basis functions are polynomials with all possible combinations of the state variables up to the 4th degree without repetitions selected as [21,22]. Set the sample period $\delta t=0.02$ s.

Apply Algorithm 1 and utilize the online state data from $t=0$ to 23 s. Then after calculation in 7 s, the approximate optimal cost is achieved with the corresponding approximate optimal switching policy through 15 iterations. The initial cost ${\widehat{V}}^0$ and the approximate optimal cost ${\widehat{V}}^k$ are demonstrated in Figure 7. It can be seen that the approximate optimal cost ${\widehat{V}}^k$ is less than the initial cost ${\widehat{V}}^0$. The state trajectories with the initial switching policy $v^0$ and the approximate optimal switching policy ${\widehat{v}}^k$ applied in the system after $t=30$ s are demonstrated in Figure 8, where the state trajectory corresponding to ${\widehat{v}}^k$ converges to the origin quickly after $t=30$ s while the trajectory corresponding to $v^0$ converges slowly with decreasing oscillation amplitude. The corresponding switching policies $v^0$ and ${\widehat{v}}^k$ are illustrated in Figure 9. It can be seen that $v^0$ which switches among three subsystems and finally stays at subsystem 3 results in that the trajectory corresponding to $v^0$ converges slowly with decreasing oscillation amplitude, while $v^{\ast }$ which switches between subsystem 1 and 2 results in that the state trajectory corresponding to ${\widehat{v}}^k$ converges to the origin quickly after $t=30$ s.

Apply Algorithm 2 and utilize the online state data from $t=0$ to 23 s. Then after calculation in 7 s, the approximate optimal cost is achieved with the corresponding approximate optimal switching policy through 21 iterations. The costs ${\widehat{V}}^0$ and ${\widehat{V}}^k$ are demonstrated in Figure 10. It can be seen that the approximate optimal cost ${\widehat{V}}^k$ is less than the initial cost ${\widehat{V}}^0$. The state trajectories with $v^0$ and ${\widehat{v}}^k$ applied in the system after $t=30$ s are demonstrated in Figure 11, where the state trajectory corresponding to ${\widehat{v}}^k$ converges to the origin relatively quickly after $t=30$ s while the trajectory corresponding to $v^0$ converges slowly with decreasing oscillation amplitude. The corresponding switching policies $v^0$ and ${\widehat{v}}^k$ are illustrated in Figure 12. It can be seen that $v^0$ which switches among three subsystems and finally stays at subsystem three results in that the trajectory converges slowly with decreasing oscillation amplitude, while $v^{\ast }$ which switches between subsystem 1 and 2 results in that the state trajectory converges to the origin relatively quickly after $t=30$ s.

Remark 4.

In algorithm 2, the value of the initial switching policy $v^0(x)$ should include every element of V and there should exist enough data to calculate ${\widehat{C}}_i^0$ for every $i\in V$ so that the sampling stage must be long enough to ensure that.

Remark 5.

In the calculation, the fourth order Runge-Kutta algorithm is adopted to numerically evaluate the integrals which are necessary in certain terms such as $b(t_r)$ and $d(t_r)$.

Remark 6.

In Example 1, the proposed algorithms converge in $0.25$ s while the online algorithm investigated in [38] requires more time. In Example 2, the proposed algorithms converge in 30 s while the online algorithms investigated in [21,22] require more time.

In the two examples, the effectiveness of Algorithm 2 has been validated. The superiority of Algorithm 2 lies in that it can work for switched systems with unknown subsystems while Algorithm 1 can not work if the subsystems of switched systems are unknown. Practical examples of this kind of switched systems appear in a wide range of applications such as cyber-physical systems which embed software into the physical world and have proved resistant to modeling due to their intrinsic complexity arising from the combination of physical and cyber components and the interaction between them in [31]. Thereinto, data-driven research has been conducted for some specific examples such as complex electronics switching among low-voltage, middle-voltage and high-voltage models, and smart grid switching between base configuration and changed configuration. These examples require the data-driven approaches of Algorithm 2 where Algorithm 1 is inapplicable.

Simulation results show that Algorithms 1 and 2 can approximate the optimal solution effectively and efficiently. Algorithm 2 achieves the approximate optimal solution for unknown switched systems with infinite-horizon cost function, which has not been achieved well in existing literature as far as we know.

5. Conclusions

In this paper, an online PI-based algorithm inspired by the off-policy learning method and based on that, a data-driven PI-based algorithm, are proposed to approximate the optimal solution quickly for optimal scheduling problems. Only data produced by an initial switching policy is necessary and the approximation time is relatively short. The data-driven algorithm acquires useful knowledge of the weight vectors step by step through different data and solves the optimal scheduling problem for switched systems with unknown subsystems, only taking advantage of data. However, the dwell-time problems, which are important in practical applications, are not incorporated in this paper. So, optimal scheduling problems with dwell-time constraints will be investigated in the future.