An Improved Adaptive Dynamic Programming Algorithm Based on Fuzzy Extended State Observer for Dissolved Oxygen Concentration Control

Abstract

To solve the anti-disturbance control problem of dissolved oxygen concentration in the wastewater treatment plant (WWTP), an anti-disturbance control scheme based on reinforcement learning (RL) is proposed. An extended state observer (ESO) based on the Takagi–Sugeno (T-S) fuzzy model is first designed to estimate the the system state and total disturbance. The anti-disturbance controller compensates for the total disturbance based on the output of the observer in real time, online searches the optimal control policy using a neural-network-based adaptive dynamic programming (ADP) controller. For reducing the computational complexity and avoiding local optimal solutions, the echo state network (ESN) is used to approximate the optimal control policy and optimal value function in the ADP controller. Further analysis demonstrates the observer estimation errors for system state and total disturbance are bounded, and the weights of ESNs in the ADP controller are convergent. Finally, the effectiveness of the proposed ESO-based ADP control scheme is evaluated on a benchmark simulation model of the WWTP.

1. Introduction

The aim of a wastewater treatment plant (WWTP) is to degrade the organic matter in the wastewater to meet the various discharge performance requirements [1]. In municipal wastewater treatment plants, the activated sludge process (ASP) is widely used to remove pollutants. The dissolved oxygen (DO) concentration in the ASP is an important indicator for survival and growth of aerobic microorganisms [2]. If the DO concentration is too high, it will increase the aeration energy consumption, affect the ammonia nitrogen conversion efficiency and deteriorate the sludge quality. If the concentration is too low, it will affect the activity of microorganisms, thereby reducing the efficiency of organic matter degradation, resulting in sludge expansion [3]. Moreover, due to the complexity of the physical, chemical, and biological phenomena in the wastewater treatment process, as well as the variability of the influent flow, the WWTPs are difficult to be controlled [2,4].

In the past few decades, proportional-integral-derivative (PID) control has been the most widely used method in WWTPs [5]. However, it is often difficult to obtain satisfactory performance through PID controllers to control complex nonlinear systems. Model Predictive Control (MPC) uses predictive models of the systems to optimize future behavior and has received a lot of attention from WWTPs [6,7,8,9,10]. For example, the MPC was proposed for adjusting the DO concentration of aerobic ponds in WWTPs. It solved the control problem of the activated sludge process [6]. Vrečko et al. designed an MPC controller for the ammonia nitrogen and evaluated it in the actual activated sludge process [7]. In addition, neural networks and fuzzy models were also used in MPC to obtain more accurate predictive models, thus to improve the accuracy of MPC [8,9]. Zeng et al. used a neural-network-based model in MPC and obtained better prediction and control performance [10]. Nevertheless, it is difficult to establish a reasonable prediction model due to the nonlinear, time-varying and strong uncertainty of WWTPs, and MPC cannot satisfactorily control WWTPs in industrial practice [11].

In recent years, reinforcement learning (RL) has been widely used in control systems due to its excellent learning ability, which is closely related to traditional optimal control and adaptive control [12]. For the nonlinear systems, the optimal control policy is determined by the solution of Hamilton–Jacobi–Bellman (HJB) equation [13]. However, the traditional dynamic programming(DP) is difficult to obtain the optimal control policy because of the famous “curse of dimensionality” [14], and the RL-based controller can solve this problem well. The RL controller can obtain the approximate solution of the HJB equation through a learning approach. For instance, an important progress was reported in [15]. It investigated a data-driven iterative adaptive critic strategy for the WWTPs control. In [16], an online adaptive dynamic programming (ADP) scheme based on echo state networks (ESN) was proposed to solve the DO control problem. A direct heuristic dynamic programming (dHDP) controller was designed to solve the multivariate tracking control problem for dissolved oxygen and nitrate concentrations [17]. Notably, the disturbance in the WWTP control problem is rarely considered in these studies.

There are many large disturbances during the operation of WWTPs, such as perturbations in influent flow and pollutant loads, changes in kinetic parameter values under the influence of internal biochemical and external environmental factors, etc. [18]. Although ADP has been widely used in optimal control of WWTPs [15,16,17], to the best of our knowledge, very little literature has considered time-varying disturbances. In [19], Jiang et al. combined robust redesign, backpropagation techniques and nonlinear small gain theorems with ADP theory, to design robust optimal control for a class of uncertain nonlinear systems. Aside from robust control [20], another effective method to reduce the effects of disturbances is sliding mode control (SMC). Muñoz et al. [21] developed an SMC controller to adjust the DO concentration. Yang et al. [22] designed a nonlinear disturbance observer (DOB) to estimate disturbances, and proposed a novel SMC method to counteract the mismatch disturbances of a class of second-order systems. However, the SMC and the DOB require the complete nonlinear dynamics of the systems, which makes it very difficult to design the sliding surface [23], not to mention learning the optimal control policy online.

To address the DO concentration anti-disturbance control problem, the active disturbance rejection control (ADRC) proposed by Han [24] is an effective method, and some related works for WWTPs have been reported [25,26,27]. The core ideas of ADRC are (1) using the extended state observer (ESO), all uncertainties acting on the system (including internal unmodeled dynamics and external disturbances) are equated into a total disturbance, (2) then the system is compensated in real time according to the total disturbance [28]. The existing research on ADRC has mainly emphasized its industrial process applications and theoretical validation of different types of uncertainty systems, while the design of nominal controllers is rarely mentioned since the compensated systems are usually reduced to multiple integrators connected in series [29].

All the above mentioned works have made some progress in the DO control problem for WWTPs, but the anti-disturbance optimal control problem is still challenging. For nonlinear WWTPs subject to complex influent conditions, there are three major challenges to be solved:

(1)

How to ensure the stable tracking performance of DO concentration in the case of complex influent conditions and different operation conditions.

(2)

How to design a disturbance estimation strategy for the unknown nonlinear WWTPs.

(3)

How to design the stable reinforcement learning controller in the system with disturbance compensation.

Motivated by the above, this paper investigates the disturbance-rejection optimal tracking control problem for WWTPs with complex external disturbances. The main advantages of the proposed control scheme are highlighted as follows:

(1)

An RL-based anti-disturbance control framework is designed for WWTPs to obtain acceptable and stable DO tracking performance. Compared with the previous results of WWTPs [15,16,17,25,26,27], the adopted disturbance estimation and compensation techniques provide WWTPs with stronger adaptability to the environment disturbance and the adopted RL control technology provides convenience for unattended operation.

(2)

Since the unknown complex dynamics of WWTPs is not available, a T-S fuzzy model is designed to simulate the nonlinear properties, based on which an ESO with boundary constraints is designed, which not only facilitates the estimation of disturbances but also deals with the observer transient peaking problem [18,30]. The observer simultaneously provides estimates of the state vector, unknown parameter variations and unknown disturbances (total disturbance), which is then used by the compensation controller in order to provide appropriate compensation signals.

(3)

The designed neural-network-based ADP controller can accomplish the optimal tracking control of the compensated system. To simplify the training of the traditional artificial neural network (ANN) [15,17], ESN is considered to approximate the actor and the critic of ADP. The detailed proof shows the proposed ESO-based ADP controller can guarantee the stability of the considered system.

The remainder of this paper is organized as follows. Section 2 presents some preliminary ideas and formulation descriptions. Section 3 describes the design the fuzzy ESO. In Section 4, the details of ADP are presented. Section 5 presents the convergence analysis. The simulation results demonstrate the effectiveness of ESO-ADP in Section 6, and conclusions are given in Section 7.

2. Problem Formulation

Consider the following single-input-single-output unknown discrete-time nonlinear system with disturbance

$$\left\{\begin{array}{c}x(k+1)=f\left(x\right(k\left)\right)+g\left(x\right(k\left)\right)·u\left(k\right)+d\left(k\right)\hfill \\ y\left(k\right)=Cx\left(k\right)\hfill \end{array}\right.$$

(1)

where $x\left(k\right)\in R$ is the system state at time step k, $u\left(k\right)\in R$ is the control input, $d\left(k\right)\in R$ is the unknown time-varying disturbance, $f(·)$, $g(·)\in R$ are unknown and bounded nonlinear smooth functions, $C\in R$ is the system output factor. For system (1), the following assumption is given

Assumption 1.

The disturbance variable $d\left(k\right)$ and its first-order difference $d(k+1)-d\left(k\right)$ are bounded.

Inspired by ADRC [24], we consider the unknown total external disturbance in the system as an extended state, denoted as

$$x_2\left(k\right)\triangleq d\left(k\right)$$

(2)

under the assumption that $g\left(x\right(k\left)\right)$ is non-singular and the extended state of the system can be used for feedback, we can design the controller as follows

$$u\left(k\right)=u_0^{*}\left(x\left(k\right)\right)-\mu \frac{x_2\left(k\right)}{g\left(x\right(k\left)\right)}$$

(3)

where $0<\mu \le 1$ is the disturbance compensation factor. The second term is the compensation term for the system total disturbance, denoted as $u_d\left(k\right)=-\mu x_2\left(k\right)/g\left(x\left(k\right)\right)$, and $u_0^{*}\left(x\left(k\right)\right)$ is the optimal control strategy for the compensated system

$$\left\{\begin{array}{c}x(k+1)=f\left(x\right(k\left)\right)+g\left(x\right(k\left)\right)·u\left(k\right)\hfill \\ y\left(k\right)=Cx\left(k\right)\hfill \end{array}\right..$$

(4)

For calculating $u_0^{*}\left(x\left(k\right)\right)$, we consider the optimal control problem for system (4) in infinite-horizon and expected to find a feedback control law $u\in \Omega$ to minimize the cost function

$$\begin{array}{cc}\hfill & J(x\left(k\right),u\left(k\right))=\sum _{i=k}^{\infty }{\gamma }^{i-k}U(x\left(i\right),u\left(i\right))\hfill \\ \hfill & =U(x\left(k\right),u\left(k\right))+\sum _{i=k+1}^{\infty }{\gamma }^{i-k}U(x\left(i\right),u\left(i\right))\hfill \end{array}$$

(5)

where $0<\gamma \le 1$ is the discount factor and $U\left(x\right(k),u(k\left)\right)$ is the one-step cost function generated by the control at time step k, and $\Omega$ is the admissible control set.

According to the optimality principle, the optimal cost function defined as

$$J^{*}\left(x\left(k\right)\right)=\underset{u\in \Omega }{min}\sum _{i=k}^{\infty }{\gamma }^{i-k}U(x\left(i\right),u\left(i\right))$$

(6)

satisfies the discrete-time HJB equation

$$J^{*}\left(x\left(k\right)\right)=\underset{u\left(x\right(k\left)\right)}{min}\left\{U(x\left(k\right),u\left(x\left(k\right)\right))+J^{*}\left(x(k+1)\right)\right\}$$

(7)

for simplicity, denote $J\left(x\right(k),u(k\left)\right)$, $U\left(x\right(k),u(k\left)\right)$ as $J\left(k\right)$, $U\left(k\right)$, respectively. Then the HJB equation can be represented by

$$J^{*}\left(k\right)=\underset{u\left(x\right(k\left)\right)}{min}\left\{U\left(k\right)+J^{*}(k+1)\right\}.$$

(8)

The optimal control problem is to solve the control policy $u\left(k\right)$ from (7), and the optimal control policy $u^{*}\left(k\right)$ need to satisfy the Bellman optimality principle

$$u^{*}\left(k\right)=\underset{u\in \Omega }{argmin}[U\left(k\right)+\gamma J^{*}(k+1)].$$

(9)

In controller (3), the optimal control $u_0^{*}\left(x\left(k\right)\right)$ is usually not analytically feasible because the HJB equation is hard to be solved with the traditional manners and the extended state is not directly available. Therefore, this paper is dedicated to developing an ESO-based RL control scheme, where the ESO is used to estimate the total disturbance for the compensation term $u_d\left(k\right)$ and the ADP controller is designed to approximate the optimal control policy for the compensated system.

3. Observer Design

First, to estimate the system total disturbance, we design the nonlinear ESO for system (1). Considering the process dynamics are unknown and unavailable, the primary system without influent disturbance can be described as a T-S fuzzy form, the state space form of the T-S fuzzy system can be expressed as

$$\left\{\begin{array}{c}x\left(k+1\right)=\sum _{l=1}^r{h}_l\left(\xi \left(k\right)\right)\left(A_{l}x\left(k\right)+B_{l}u\left(k\right)\right)\hfill \\ y\left(k\right)=\sum _{l=1}^r{h}_l\left(\xi \left(k\right)\right)C_{l}x\left(k\right)\hfill \end{array}\right.$$

(10)

where $\xi \left(k\right)={\left[{\xi }_1\left(k\right),{\xi }_2\left(k\right),\dots ,{\xi }_s\left(k\right)\right]}^T$ is the premise variables to be selected, s is the number of premise variables, which can be determined according to the source of nonlinearity of the system, $h_l\left(\xi \left(k\right)\right)={\mu }_l\left(\xi \left(k\right)\right)/{\sum }_{i=1}^r{\mu }_i\left(\xi \left(k\right)\right)$ is the normalized weight of the sub-model, r denotes the number of sub-models, and ${\mu }_l\left(\xi \left(k\right)\right)={\prod }_{i=1}^s{\mu }_{M_i^{ł}}\left({\xi }_i\left(k\right)\right)$ is the affiliation function of the premise variables to each sub-model, that is, the weight of the sub-model. $A_i,B_i,C_i,i=1,2,\dots ,l$ are the parameters of each sub-model, and the sub-model coefficients are denoted as ${\sum }_{i=1}^r{h}_i\left(\xi \left(k\right)\right)A_i=A\left(h\right)$, ${\sum }_{i=1}^r{h}_i\left(\xi \left(k\right)\right)B_i=B\left(h\right)$, then the system (10) can be simplified as

$$\left\{\begin{array}{c}x\left(k+1\right)=A\left(h\right)x\left(k\right)+B\left(h\right)u\left(k\right)\hfill \\ y\left(k\right)=Cx\left(k\right)\hfill \end{array}\right..$$

(11)

Consider system (11) with unknown time-varying disturbance:

$$\left\{\begin{array}{c}x\left(k+1\right)=A\left(h\right)x\left(k\right)+B\left(h\right)u\left(k\right)+d\left(k\right)\hfill \\ y\left(k\right)=Cx\left(k\right)\hfill \end{array}\right.$$

(12)

then system (1) is described as system (12) and the observer of system (12) can be designed as

$$\left\{\begin{array}{c}e\left(k\right)=z_1\left(k\right)-x\left(k\right)\hfill \\ z_1(k+1)=z_2\left(k\right)-{\beta }_1\phi \left(e\left(k\right)\right)\hfill \\ +A\left(h\right)z_1\left(k\right)+B\left(h\right)u\left(k\right)\hfill \\ z_2(k+1)=z_2\left(k\right)-{\beta }_2\phi \left(e\left(k\right)\right)\hfill \end{array}\right.$$

(13)

where $z\left(k\right)={[z_1\left(k\right),z_2\left(k\right)]}^T\in R^2$ is the state vector of ESO representing the estimations of the system state and disturbance, $e\left(k\right)$ is the state estimation error, $\beta ={[{\beta }_1,{\beta }_2]}^T\in R^2$ is the appropriate observer gain vector and $\phi (·)$ is the nonlinear function to be designed and $\phi \left(0\right)=0$. Therefore, the control policy described by Equation (3) can be represented by

$$u\left(k\right)=u_0^{*}\left(k\right)-\mu \frac{z_2\left(k\right)}{B\left(h\right)}.$$

(14)

In order to counteract the peaking phenomenon generated during the transitions of ESO [28,31], we constrain the output of the observer within a compact set by using saturation techniques [32].

$${\overline{z}}_i=M_{i}sat\left(\frac{z_i}{M_i}\right),i=1,2,\dots ,n+1,$$

(15)

where $M_i,i=1,2,\dots ,n+1$ is the constraint boundary value to be selected, and the above constraint ensures that the peak value of the observer transition process will not be transmitted to the system. The constraint function $sat(·)$ is defined as follows

$$sat\left(x\right)=\left\{\begin{array}{cc}0,\hfill & x<0\hfill \\ x,\hfill & 0\le x\le 1\hfill \\ x+\frac{x-1}{\epsilon }-\frac{x^2-1}{2\epsilon },\hfill & 1\le x\le 1+\epsilon \hfill \\ 1+\frac{\epsilon }{2},\hfill & x\ge 1+\epsilon \hfill \end{array}\right.,$$

(16)

where $\epsilon$ is a small positive constant. This constraint function can achieve a smooth transition from the unsaturated state to the saturated state, and reduce the influence of the unstable factors of the observer on the system.

Remark 1.

The ESO is designed by rewriting the system as the T-S multi-model form of the state space, rather than the system dynamic equation(which is usually unknown in actual WWTPs). This means the T-S fuzzy model can be obtained by identification, which eliminates the reliance on the priori knowledge [18,33,34], while simplifying the structure of the control system and improving the versatility of the observer.

4. Online ADP Controller Design

When the observer (13) is capable of unbiased estimation of the total system disturbance and the system is real-time compensated, the optimal control policy of the compensated system (4) can be learned online by ADP.

4.1. Structure of ESO-ADP

The main idea of ADP is to obtain the optimal cost function and optimal control law by function approximation to satisfy the optimality principle and the HJB equation. To simplify the training of the traditional ANN, the ESN is used in the ADP controller. The principle of ADP with disturbance compensation control policy is shown in Figure 1, which contains three ESN modules, Actor ESN, Critic ESN and Model ESN. The Actor ESN represents the mapping between the system state variables to the control variables and is used to approximate the optimal control policy; Critic ESN takes the state variables as input and its output is the estimation of the optimal cost function; the Model ESN is used to describe the unknown nonlinear system to generate forward time difference for updating the Actor ESN. The Utility module serves to calculate the cost incurred by the one-step control.

The DO concentration in the WWTP is set as the state variable, which can be directly measured by the DO sensor. In other words, the system state is directly available and thus we use the measured state for feedback instead of the estimated value of the ESO. Assume the adjustable parameter of the critic network, actor network and model network are ${\theta }_c$, ${\theta }_a$ and ${\theta }_m$, respectively, then the workflow of the ESO-ADP controller with disturbance compensation is as follows

Step 1.

Let $k=0$, select simulation step N, randomly initialize ${\theta }_c\left(0\right)$, ${\theta }_a\left(0\right)$ and ${\theta }_m\left(0\right)$.

Step 2.

The actor ESN generates ${\widehat{u}}^{*}\left(k\right)$ based on $x\left(k\right)$ at time step k.

Step 3.

The Critic1 ESN evaluates the current state $x\left(k\right)$ and outputs $J\left(k\right)$.

Step 4.

The Utility module calculates $U\left(k\right)$.

Step 5.

The Model ESN predicts the state $\widehat{\mathit{x}}(k+1)$ at next time step $k+1$.

Step 6.

The Critic2 ESN evaluates $\widehat{\mathit{x}}(k+1)$ and outputs $J(k+1)$.

Step 7.

ESO estimates the system disturbance and outputs $z_2\left(k\right)$, and the compensation controller gives the compensation control policy $u_d\left(k\right)$.

Step 8.

Apply the control policy $u\left(k\right)={\widehat{u}}^{*}\left(k\right)+u_d\left(k\right)$ to the system, thus $x(k+1)$ is obtained.

Step 9.

Calculate the weight increment for each network, update the parameters as below

$${\theta }_c(k+1)\leftarrow {\theta }_c\left(k\right)+\Delta {\theta }_c\left(k\right)$$

$${\theta }_a(k+1)\leftarrow {\theta }_a\left(k\right)+\Delta {\theta }_a\left(k\right)$$

$${\theta }_m(k+1)\leftarrow {\theta }_m\left(k\right)+\Delta {\theta }_m\left(k\right)$$

Step 10.

When $k<N$, let $k=k+1$, go back to 2 and continue.

As the simulation time step k increases, the ADP controller will gradually approach the optimal control policy and the optimal cost function, while the total disturbance of the system is compensated.

4.2. Echo State Network Approximation

The principle of ESN used in ADP is shown in Figure 2. Proposed by Jaeger [35], the ESN consists of the input layer, the dynamic reservoir and the readout layer. The dynamic reservoir contains a large number of neurons, which are connected with a random sparse matrix, and the output of the hidden layer of the ESN can be expressed as

$$s\left(k\right)=\sigma \left(W_{IN}u\left(k\right)+W_{R}s(k-1)\right)$$

(17)

where $u\left(k\right)={[{\upsilon }_1\left(k\right),{\upsilon }_2\left(k\right),\dots ,{\upsilon }_K\left(k\right)]}^T$ is the input vector at time step k and $s\left(k\right)=[{\varsigma }_1\left(k\right),{\varsigma }_2\left(k\right),$$\dots ,{{\varsigma }_N\left(k\right)]}^T$ is the internal state at time step k, that is, the output of ESN dynamic reservoir neurons. $W_{\mathrm{IN}}\in R^{N\times K}$ is the weight matrix of input layer and $W_{\mathrm{R}}\in R^{N\times N}$ is the internal connection matrix. $\sigma (·)$ is the activation function of the hidden layer neurons, K denotes the input dimension, and N denotes the number of reservoir internal neurons. The ESN output can be described by

$$y\left(k\right)=s^{\mathrm{T}}\left(k\right)W_O$$

(18)

where $y\left(k\right)=[y_1\left(k\right),y_2\left(k\right),\dots ,{y_L\left(k\right)]}^T$ is the output vector at time step k, L denotes the dimensionality of the output vector, and $W_O\in R^{N\times L}$ is the readout layer weight. Only $W_O$ is trained, $W_{\mathrm{IN}}$ and $W_{\mathrm{R}}$ are randomly determined when the network is initialized and fixed during the training process [35].

4.3. Online Learning of the ADP Controller

The learning of the ADP controller is actually the parameter update of the ESNs. The weight-adjustment rules for each ESN are as follows.

4.3.1. Critic ESN

The aim of Critic ESN weight adjustment is to approximate the optimal cost function described by (6). The output weights of the Critic ESN are denoted as $W_C$, and the loss function is defined as

$$L_{\mathrm{c}}\left(k\right)=\frac{1}{2}{e}_{\mathrm{c}}^2\left(k\right)$$

(19)

where $e_{\mathrm{c}}\left(k\right)$ is defined as

$$e_{\mathrm{c}}\left(k\right)=J\left(k\right)-U\left(k\right)-\gamma J(k+1).$$

(20)

The weights of Critic ESN are updated as follows

$$W_{\mathrm{c}}(k+1)=W_{\mathrm{c}}\left(k\right)+\Delta W_{\mathrm{c}}\left(k\right)$$

(21)

the increment of the weights of the Critic ESN by gradient descent is

$$\begin{array}{cc}\hfill \Delta W_{\mathrm{c}}\left(k\right)& =-{\alpha }_{\mathrm{c}}\left(k\right){\left(\frac{\partial L_{\mathrm{c}}\left(k\right)}{\partial W_{\mathrm{c}}\left(k\right)}\right)}^{\mathrm{T}}\hfill \\ & =-{\alpha }_{\mathrm{c}}\left(k\right)e_{\mathrm{c}}\left(k\right){\left(\frac{\partial e_{\mathrm{c}}\left(k\right)}{\partial W_{\mathrm{c}}\left(k\right)}\right)}^{\mathrm{T}}\hfill \\ & =-{\alpha }_{\mathrm{c}}\left(k\right)e_{\mathrm{c}}\left(k\right){\left(\frac{\partial J\left(k\right)}{\partial W_{\mathrm{c}}\left(k\right)}\right)}^{\mathrm{T}}\hfill \end{array}$$

(22)

where ${\alpha }_{\mathrm{c}}\left(k\right)$ is the learning rate of the Critic ESN, (22) can be further simplified as

$$\Delta W_{\mathrm{c}}\left(k\right)=-{\alpha }_{\mathrm{c}}\left(k\right)e_{\mathrm{c}}\left(k\right)s_{\mathrm{c}}\left(k\right).$$

(23)

where $s_{\mathrm{c}}\left(k\right)$ is the internal state of Critic ESN.

4.3.2. Actor ESN

The aim of the Actor ESN weight adjustment is to generate control decisions to minimize the cost function, and the adjustable weight of actor ESN is denoted as $W_a\left(k\right)$, then the loss function for weight update is defined as [16,36]

$$\frac{\partial J\left(k\right)}{\partial W_a\left(k\right)}=\frac{\partial U\left(k\right)}{\partial W_a\left(k\right)}+\gamma \frac{\partial J(k+1)}{\partial W_a\left(k\right)}.$$

(24)

The necessary condition for the Actor ESN weight to be optimal is that (24) is equal to 0, so the increment of the actor ESN weight can be expressed as

$$\Delta W_{\mathrm{a}}\left(k\right)=-{\alpha }_{\mathrm{a}}\left(k\right)\frac{\partial J\left(k\right)}{\partial W_a\left(k\right)}$$

(25)

where ${\alpha }_{\mathrm{a}}\left(k\right)$ is the learning rate of the Actor ESN, based on (24) and the chain derivative rule, (25) can be written as

$$\begin{array}{cc}\hfill \Delta W_{\mathrm{a}}\left(k\right)& =-{\alpha }_{\mathrm{a}}\left(k\right)(\frac{\partial U\left(k\right)}{\partial u\left(k\right)}\hfill \\ & +\gamma \frac{\partial J(k+1)}{\partial u\left(k\right)}){(\frac{\partial u\left(k\right)}{\partial W_{\mathrm{a}}\left(k\right)})}^{\mathrm{T}}\hfill \end{array}$$

(26)

mark the middle part as

$$\Theta \left(k\right)=\frac{\partial U\left(k\right)}{\partial u\left(k\right)}+\gamma \frac{\partial J(k+1)}{\partial u\left(k\right)}$$

(27)

then (26) can be written as

$$\begin{array}{cc}\hfill \Delta W_a\left(k\right)& =-{\alpha }_a\left(k\right)\Theta \left(k\right){\left(\frac{\partial u\left(k\right)}{\partial W_a\left(k\right)}\right)}^T\hfill \\ & =-{\alpha }_a\left(k\right)\Theta \left(k\right)s_a\left(k\right)\hfill \end{array}$$

(28)

where $s_a\left(k\right)$ is the internal state of Actor ESN.

From (28), the increment of the weights of the Actor ESN can be calculated by simply calculating $\Theta \left(k\right)$ from (27). $U\left(k\right)$ is a one-step cost function defined according to the actual problem and $\partial U\left(k\right)/\partial u\left(k\right)$ is easy to be obtained, then the second term of (27) can be calculated by

$$\frac{\partial J(k+1)}{\partial u\left(k\right)}=\frac{\partial J(k+1)}{\partial x(k+1)}\frac{\partial x(k+1)}{\partial u\left(k\right)}.$$

(29)

According to (17), (18) and Figure 1, we can obtain

$$\frac{\partial J(k+1)}{\partial x(k+1)}=W_{in,c}^{\mathrm{T}}\left(k\right){\sigma }^{{}^{\prime }}\left({\theta }_c\left(k\right)\right)W_c\left(k\right)$$

(30)

$$\frac{\partial x(k+1)}{\partial u\left(k\right)}=W_{in,m}^{\mathrm{T}}\left(k\right){\sigma }^{{}^{\prime }}\left({\theta }_m\left(k\right)\right)W_m\left(k\right)$$

(31)

where ${\sigma }^{{}^{\prime }}$ is the derivative of the activation function of the dynamic reservoir neuron, ${\theta }_c\left(k\right)$ and ${\theta }_m\left(k\right)$ are the input of the dynamic reservoir neuron of Critic ESN and Model ESN, respectively. $W_c\left(k\right)$ and $W_m\left(k\right)$ are the output weight matrices of the Critic ESN and Model ESN, respectively. $W_{in,c}^{}\left(k\right)$ is the input weight vector of the Critic ESN2, and $W_{in,m}^{}\left(k\right)$ is the component of the input weight vector associated with u of the Model ESN.

4.3.3. Model ESN

For the Model ESN, the output weights are denoted as $W_m\left(k\right)$, and the loss function of the model network is defined as

$$L_m\left(k\right)=\frac{1}{2}{e}_m^2\left(k\right)$$

(32)

where $e_m\left(k\right)=y_{pre}\left(k\right)-y_m\left(k\right)$, $y_{pre}\left(k\right)$ is the predicted output of the Model ESN and $y_m\left(k\right)$ is the actual measured of the system output. $W_m\left(k\right)$ is updated by

$$\Delta W_m\left(k\right)=-{\alpha }_m\left(k\right)e_m\left(k\right)s_m\left(k\right)$$

(33)

where ${\alpha }_m\left(k\right)$ is the learning rate and $s_m\left(k\right)$ is the internal state of Model ESN.

5. Stability Analysis

In this section, the proof will be divided into two steps. In the first step the boundedness of the states and disturbance estimation errors of ESO is proved to ensure the availability of the observer. Then, in the second step, the convergence of the ESN weights in the ADP controller is given for the effectiveness of the neural network.

Step (1): The boundedness of the ESO estimation errors is shown as follows.

For the observer (13), we make the following assumption

Assumption 2.

The nonlinear function $\phi (·)$ to be selected is global Lipschitz, that is, for any $x_1$ and $x_2$, there exists a constant $l_p>0$, so that $\left|\right|\phi \left(x_1\right)-\phi \left(x_2\right)|{|}_2\le l_p\left|\right|x_1-x_2|{|}_2$ holds.

Defining the estimation errors of the system state and the total disturbance as $e_1\left(k\right)=z_1\left(k\right)-x\left(k\right)$, $e_2\left(k\right)=z_2\left(k\right)-d\left(k\right)$, respectively, the dynamics of $e_1$ and $e_2$ can be obtained as

$$\begin{array}{cc}\hfill e_1(k+1)=& A\left(h\right)z_1\left(k\right)+B\left(h\right)u\left(k\right)+z_2\left(k\right)\hfill \\ & -{\beta }_1\phi \left(e_1\left(k\right)\right)-A\left(h\right)x\left(k\right)\hfill \\ & -B\left(h\right)u\left(k\right)-d\left(k\right)\hfill \\ \hfill =& A\left(h\right)e_1\left(k\right)+e_2\left(k\right)-{\beta }_1\phi \left(e_1\left(k\right)\right)\hfill \end{array}$$

(34)

$$\begin{array}{cc}\hfill e_2(k+1)& =z_2\left(k\right)-{\beta }_2\phi \left(e_1\left(k\right)\right)-d(k+1)\hfill \\ & =e_2\left(k\right)-{\beta }_2\phi \left(e_1\left(k\right)\right)-[d(k+1)-d\left(k\right)].\hfill \end{array}$$

(35)

Denote $\tilde{e}\left(k\right)={[e_1\left(k\right),e_2\left(k\right)]}^T$, then the above two equations are combined to obtain

$$\begin{array}{cc}\hfill \tilde{e}(k+1)=& \left[\begin{array}{cc}A\left(h\right)& I\\ 0& I\end{array}\right]\tilde{e}\left(k\right)+\left[\begin{array}{c}-{\beta }_1\hfill \\ -{\beta }_2\hfill \end{array}\right]\psi \left(\tilde{e}\left(k\right)\right)\hfill \\ & +\left[\begin{array}{c}0\\ d(k+1)-d\left(k\right)\end{array}\right]\hfill \\ \hfill \triangleq & \tilde{A}\left(h\right)\tilde{e}\left(k\right)+\tilde{\beta }\psi \left(\tilde{e}\left(k\right)\right)+\tilde{D}\left(k\right)\hfill \end{array}$$

(36)

where $\psi \left(\tilde{e}\left(k\right)\right)=\phi \left(e_1\left(k\right)\right)$, $\tilde{D}\left(k\right)={[0,d(k+1)-d(k\left)\right]}^{\mathrm{T}}$.

Theorem 1.

Given a discrete T-S fuzzy system (12) with time-dependent non-repetitive disturbance, the following equation holds if there exists a positive definite symmetric matrix $P_1$ and a positive number $l_{p1}$:

$$\left[\begin{array}{cccc}-P_1+l_{p1}^{2}I& 0& P_1& {\tilde{A}}^T\left(h\right)P_1\\ \ast & -I& 0& {\tilde{\beta }}^T{P}_1\\ *& *& -P_1& 0\\ *& *& *& -P_1\end{array}\right]<0$$

(37)

where “*” denotes the transpose of the symmetric part of the matrix, then the estimation errors of the ESO is bounded.

Proof of Theorem 1.

Take the following Lyapunov function

$$V_1\left(k\right)={\left(\tilde{e}\left(k\right)-\tilde{D}(k-1)\right)}^{\mathrm{T}}{P}_1\left(\tilde{e}\left(k\right)-\tilde{D}(k-1)\right)$$

(38)

$$\begin{array}{cc}\hfill & \Delta V_1=V_1(k+1)-V_1\left(k\right)\hfill \\ \hfill & ={\tilde{e}}^{\mathrm{T}}\left(k\right)\left({\tilde{A}}^{\mathrm{T}}\left(h\right)P_1\tilde{A}\left(h\right)-P_1\right)\tilde{e}\left(k\right)\hfill \\ \hfill & +{\psi }^{\mathrm{T}}\left(\tilde{e}\left(k\right)\right){\tilde{\beta }}^{\mathrm{T}}{P}_1\tilde{\beta }\psi \left(\tilde{e}\left(k\right)\right)\hfill \\ \hfill & -{\tilde{D}}^{\mathrm{T}}(k-1)P_1\tilde{D}(k-1)\hfill \\ \hfill & +2{\tilde{e}}^{\mathrm{T}}\left(k\right)P_1\tilde{D}(k-1)\hfill \\ \hfill & +2{\tilde{e}}^{\mathrm{T}}\left(k\right){\tilde{A}}^{\mathrm{T}}\left(h\right)P_1\tilde{\beta }\psi \left(\tilde{e}\left(k\right)\right)\hfill \end{array}$$

(39)

with Assumption 2 and the condition $\phi \left(0\right)=0$, there is

$${∥\phi \left(e_1\left(k\right)\right)-\phi \left(0\right)∥}_2\le l_{\mathrm{p}1}{∥e_1\left(k\right)-0∥}_2$$

(40)

then, based on $\psi \left(\tilde{e}\left(k\right)\right)=\phi \left(e_1\left(k\right)\right)$, it is further deduced that

$$l_{\mathrm{p}1}^2{\tilde{e}}^{\mathrm{T}}\left(k\right)\tilde{e}\left(k\right)-{\psi }^{\mathrm{T}}\left(\tilde{e}\left(k\right)\right)\psi \left(\tilde{e}\left(k\right)\right)\ge 0$$

(41)

adding (41) to the right of (39) yields

$$\begin{array}{cc}\hfill & \Delta V_1\le {\tilde{e}}^{\mathrm{T}}\left(k\right)\left({\tilde{A}}^{\mathrm{T}}\left(h\right)P_1\tilde{A}\left(h\right)-P_1\right)\tilde{e}\left(k\right)\hfill \\ \hfill & +{\psi }^{\mathrm{T}}\left(\tilde{e}\left(k\right)\right)\left({\tilde{\beta }}^{\mathrm{T}}{P}_1\tilde{\beta }-I\right)\psi \left(\tilde{e}\left(k\right)\right)\hfill \\ \hfill & -{\tilde{D}}^{\mathrm{T}}(k-1)P_1\tilde{D}(k-1)\hfill \\ \hfill & +2{\tilde{e}}^{\mathrm{T}}\left(k\right)P_1\tilde{D}(k-1)\hfill \\ \hfill & +2{\tilde{e}}^{\mathrm{T}}\left(k\right){\tilde{A}}^{\mathrm{T}}\left(h\right)P_1\tilde{\beta }\psi \left(\tilde{e}\left(k\right)\right)\hfill \end{array}$$

(42)

the $\Delta V_1<0$ can be equated to

$$\left[\begin{array}{ccc}{\tilde{A}}^{\mathrm{T}}\left(h\right)P_1\tilde{A}\left(h\right)-P_1+l_{\mathrm{p}1}^{2}I& {\tilde{A}}^{\mathrm{T}}\left(h\right)P_1\tilde{\beta }& P_1\\ *& {\tilde{\beta }}^{\mathrm{T}}{P}_1\tilde{\beta }-I& 0\\ *& *& -P_1\end{array}\right]<0$$

(43)

according to Schur’s Compliment theorem, (43) can be equated as

$$\left[\begin{array}{cccc}-P_1+l_{\mathrm{p}1}^{2}I& 0& P_1& {\tilde{A}}^{\mathrm{T}}\left(h\right)\\ *& -I& 0& {\tilde{\beta }}^{\mathrm{T}}\\ *& *& -P_1& 0\\ *& *& *& -P_1^{-1}\end{array}\right]<0$$

(44)

multiplying left and right by $diag\left\{I,I,I,P\right\}$, there is

$$\left[\begin{array}{cccc}-P_1+l_{\mathrm{p}1}^{2}I& 0& P_1& {\tilde{A}}^{\mathrm{T}}\left(h\right)P_1\\ *& -I& 0& {\tilde{\beta }}^{\mathrm{T}}{P}_1\\ *& *& -P_1& 0\\ *& *& *& -P_1\end{array}\right]<0.$$

(45)

The above equation shows that the estimation errors $e_1\left(k\right)$ and $e_2\left(k\right)$ are bounded when $\Delta V_1<0$, that is, the observation error dynamic (36) is stable. □

Step (2): The convergence of the ESN weights is shown as follows.

Define the Lyapunov function as

$$V_2\left(k\right)=\frac{1}{2}{e}^2\left(k\right)$$

(46)

where $e\left(k\right)$ is the error function defined in the ESN learning process, let $\Delta e\left(k\right)=e(k+1)-e\left(k\right)$, then

$$\begin{array}{cc}\hfill \Delta V_2\left(k\right)& =\frac{1}{2}{e}^2(k+1)-\frac{1}{2}{e}^2\left(k\right)\hfill \\ & =\frac{1}{2}\Delta e\left(k\right)(e(k+1)+e\left(k\right))\hfill \\ & =\frac{1}{2}\Delta e\left(k\right)[\Delta e\left(k\right)+2e\left(k\right)].\hfill \end{array}$$

(47)

Theorem 2.

If ${\alpha }_c\left(k\right)$ satisfies the following condition, the weight of Critic ESN is convergent.

$${\alpha }_{\mathrm{c}}\left(k\right)<\frac{2}{{∥s_{\mathrm{c}}\left(k\right)∥}^2}$$

(48)

Proof of Theorem 2.

Definition $e\left(k\right)=e_{\mathrm{c}}\left(k\right)$, according to (20), there is

$$\frac{\partial e\left(k\right)}{\partial W_{\mathrm{c}}\left(k\right)}=\frac{\partial J\left(k\right)}{\partial W_{\mathrm{c}}\left(k\right)}=s_{\mathrm{c}}^{\mathrm{T}}\left(k\right)$$

(49)

according to the full differentiation theorem, one has

$$\Delta e\left(k\right)=\frac{\partial e\left(k\right)}{\partial W_{\mathrm{c}}\left(k\right)}\Delta W_{\mathrm{c}}\left(k\right)$$

(50)

substituting (23), (49) into (50) yields

$$\Delta e\left(k\right)=-{\alpha }_{\mathrm{c}}\left(k\right)e_{\mathrm{c}}\left(k\right)\left|\right|s_{\mathrm{c}}\left(k\right)|{|}^2$$

(51)

substituting (51) into (47) yields

$$\begin{array}{cc}\hfill & \Delta V\left(k\right)=\frac{1}{2}[-{\alpha }_{\mathrm{c}}\left(k\right)e_{\mathrm{c}}\left(k\right)\left|\right|s_{\mathrm{c}}\left(k\right)\left|{|}^2\right]·\hfill \\ \hfill & [-{\alpha }_{\mathrm{c}}\left(k\right)e_{\mathrm{c}}\left(k\right)|\left|s_{\mathrm{c}}\left(k\right)\right|{|}^2+2e_{\mathrm{c}}\left(k\right)]\hfill \\ \hfill & =-\frac{1}{2}[2-{\alpha }_{\mathrm{c}}\left(k\right)\left|\right|s_{\mathrm{c}}\left(k\right)\left|{|}^2\right]{e_{\mathrm{c}}}^2\left(k\right)·\hfill \\ \hfill & \left[{\alpha }_{\mathrm{c}}\left(k\right)\right|\left|s_{\mathrm{c}}\left(k\right)\right|{|}^2]\hfill \end{array}$$

(52)

therefore, as long as

$$\frac{1}{2}[2-{\alpha }_{\mathrm{c}}\left(k\right)\left|\right|s_{\mathrm{c}}\left(k\right)\left|{|}^2\right]>0$$

(53)

then $\Delta V_2\left(k\right)\le 0$, solving equation (53) yields equation (48), and according to the Lyapunov theory of discrete systems, the Critic ESN weight learning process is convergent when (48) holds. □

Theorem 3.

If ${\alpha }_a\left(k\right)$ satisfies the following condition, the weight of Actor ESN is convergent.

$${\alpha }_a\left(k\right)<\frac{2\left(U\right(k)+\gamma J(k+1\left)\right)}{{\Theta }^2\left(k\right){∥s_{\mathrm{a}}\left(k\right)∥}^2}$$

(54)

Proof of Theorem 3.

Let $e\left(k\right)=U\left(k\right)+\gamma J(k+1)$, then

$$\frac{\partial e\left(k\right)}{\partial W_a\left(k\right)}=\frac{\partial U\left(k\right)}{\partial W_a\left(k\right)}+\gamma \frac{\partial J(k+1)}{\partial W_a\left(k\right)}$$

(55)

according to (24)–(28) and (55), there is

$$\frac{\partial e\left(k\right)}{\partial W_a\left(k\right)}=\Theta \left(k\right)s_{\mathrm{a}}\left(k\right)$$

(56)

according to the full differentiation theorem, one has

$$\Delta e\left(k\right)=\frac{\partial e\left(k\right)}{\partial W_a\left(k\right)}\Delta W_a\left(k\right)$$

(57)

substituting (56) and (28) into (57) yields

$$\Delta e\left(k\right)=-{\alpha }_{\mathrm{a}}\left(k\right){\Theta }^2\left(k\right){∥s_{\mathrm{a}}\left(k\right)∥}^2$$

(58)

substituting (58) into (47) yields

$$\begin{array}{cc}\hfill & \Delta V_2\left(k\right)=\frac{1}{2}[-{\alpha }_{\mathrm{a}}\left(k\right){\Theta }^2\left(k\right){∥s_{\mathrm{a}}\left(k\right)∥}^2]·\hfill \\ \hfill & [-{\alpha }_{\mathrm{a}}\left(k\right){\Theta }^2\left(k\right){∥s_{\mathrm{a}}\left(k\right)∥}^2+2e\left(k\right)].\hfill \end{array}$$

(59)

When the following inequality holds

$$-{\alpha }_{\mathrm{a}}\left(k\right){\Theta }^2\left(k\right){∥s_{\mathrm{a}}\left(k\right)∥}^2+2e\left(k\right)>0$$

(60)

that is

$${\alpha }_{\mathrm{a}}\left(k\right)<\frac{2\left(U\right(k)+\gamma J(k+1\left)\right)}{{\Theta }^2\left(k\right){∥s_{\mathrm{a}}\left(k\right)∥}^2}$$

(61)

then $\Delta V_2\left(k\right)\le 0$, the Actor ESN weight is convergent. The proof is over. □

It is worth noting that the Critic ESN may not converge to zero, it is equivalent to $J\left(k\right)$. As the learning process proceeds, the Critic ESN will gradually approach the optimal cost function described by (6), and the total cost value decreases as the policy improvement proceeds, finally converging to an constant value.

Remark 2.

Based on the above discussion, unlike typical iterative adaptive critic control [15], the proposed ESO-ADP is learned online using on-policy, which retains the advantages of ESNs relative to single hidden layer feedforward neural networks. In general, this reinforcement learning method for neural network approximation can be widely used in practical applications. Meanwhile, based on the state estimation results of T-S fuzzy ESO, the proposed compensation controller can effectively suppress perturbations, so that the reinforcement learning method can take effective measures to make DO concentration recover quickly and keep it stable.

6. Simulation Studies

This section presents the simulation experiment to illustrate the effectiveness of ESO-ADP.

6.1. Parameter Initialization

The simulation experiments in this paper are conducted on the Benchmark Simulation Model No.1 (BSM1) developed by the International Water Quality Association and the European Union Scientific and Technical Cooperation [37], which aims to provide a standardized platform to compare different control strategies. The plant layout of BSM1 is a bioreactor and a secondary sedimentation tank, as shown in Figure 3.

BSM1 contains two main control loops, nitrate concentration $S_{NO}$ and DO concentration $S_{O,5}$, which are controlled by two single-loop controllers in this paper. For $S_{NO}$, PID control is used in all simulation experiments and the parameters are P = 10,000, $I=0.025$, $D=0$. The DO concentration is the mainly concerned variable in this paper, so the ESO-ADP controller is used in DO control loop. The manipulated variable of the $S_{O,5}$ is the oxygen transfer coefficient $K_L{a}_5$ of the fifth reactor, which operates in the range of 0–360/d; the manipulated variable of the nitrate control loop is the flow rate $Q_a$, which ranges from 0–92,230 m${}^3$/d. BSM1 provides the influent data under three different weather. They are dry weather, rain weather, and storm weather. This work evaluates the control performance of the proposed ESO-ADP controller under dry weather, then rain weather are also used to evaluate the anti-interference performance of ESO-ADP and the tracking ability.

$S_{O,5}$ is the state variable and output of the control system and $K_L{a}_5$ is the control input, while the second-order ESO for the DO control system described in (13) is designed as

$$\left\{\begin{array}{c}e\left(k\right)=z_1\left(k\right)-x\left(k\right)\hfill \\ z_1(k+1)=z_2\left(k\right)-{\beta }_{1}e\left(k\right)+A\left(h\right)z_1\left(k\right)\hfill \\ +B\left(h\right)u\left(k\right)\hfill \\ z_2(k+1)=z_2\left(k\right)-{\beta }_{2}e\left(k\right)\hfill \end{array}\right.$$

(62)

this means $\phi (·)$ in (13) is defined as $\phi \left(x\right)=x$. The T-S fuzzy model under constant influent conditions is described as

$$\left\{\begin{array}{c}x\left(k+1\right)=\sum _{l=1}^3{h}_l\left(\xi \left(k\right)\right)\left(A_{l}x\left(k\right)+B_{l}u\left(k\right)\right)\hfill \\ y\left(k\right)=\sum _{l=1}^3{h}_l\left(\xi \left(k\right)\right)C_{l}x\left(k\right)\hfill \end{array}\right.$$

(63)

where the premise variable $\xi \left(k\right)$ is the normalized $S_{0,5}\left(k\right)$, $h_l\left(\xi \left(k\right)\right)={\mu }_l\left(\xi \left(k\right)\right)/{\sum }_{i=1}^3{\mu }_i\left(\xi \left(k\right)\right)$ is the normalized sub-model weight, ${\mu }_i\left(\xi \left(k\right)\right),i=1,2,3$ is the affiliation function, and is expressed as

$${\mu }_1\left(\xi \left(k\right)\right)=\left\{\begin{array}{c}1,\xi \left(k\right)\le -1\hfill \\ \frac{\lambda -\xi \left(k\right)}{\lambda +1},-1<\xi \left(k\right)<\lambda \hfill \\ 0,\xi \left(k\right)\ge \lambda \hfill \end{array}\right.$$

(64)

$${\mu }_2\left(\xi \left(k\right)\right)=\left\{\begin{array}{c}0,\xi \left(k\right)\le -1\hfill \\ \frac{\xi \left(k\right)+1}{\lambda +1},-1<\xi \left(k\right)<\lambda \hfill \\ \frac{\mathsf{1}-\xi \left(k\right)}{1-\lambda },\lambda \le \xi \left(k\right)<1\hfill \\ 0,\xi \left(k\right)\ge \mathsf{1}\hfill \end{array}\right.$$

(65)

$${\mu }_3\left(\xi \left(k\right)\right)=\left\{\begin{array}{c}0,\xi \left(k\right)\le \lambda \hfill \\ \frac{\xi \left(k\right)-\lambda }{1-\lambda },\lambda <\xi \left(k\right)<\mathsf{1}\hfill \\ 1,\xi \left(k\right)\ge \mathsf{1}\hfill \end{array}\right.$$

(66)

where $\lambda =-0.5767$ is the center of the second affiliation function, which is determined by identification. In addition, the parameters of the three sub-models in (39) are shown in

Table 1. Sub-model parameters.

	${\mathit{A}}_{\mathit{l}}$	${\mathit{B}}_{\mathit{l}}$	${\mathit{C}}_{\mathit{l}}$
Model1	0.8225	0.1373	1
Model2	0.7245	0.4406	1
Model3	0.6573	0.3275	1

.

Remark 3.

The nonlinearity of the activated sludge system mainly comes from its structure, the growth characteristics of microorganisms and the settling law of sludge. Theoretically the premise variable should be selected as the state variable, but the unmeasurability of the premise variable is not conducive to parameter estimation. Considering the correlation between the state and the output, the premise variable is selected as DO concentration in this paper.

In the simulation, the observer parameter is set to ${\beta }_1=0.65$, ${\beta }_2=0.42$ and $\epsilon =0.01$, and the saturation bounds of the observer output ${\widehat{x}}_1$ and ${\widehat{x}}_2$ are set to $M_1=4$, $M_2=4$.

Table 2. ADP controller module inputs and outputs and parameters.

Name	Lr	Input	Output	n	SR	SD
Actor ESN	0.1	$R\left(k\right)-S_{O,5}\left(k\right)$	$\Delta K_L{a}_5(k+1)$	40	0.2	0.05
Critic ESN	0.2	$S_{O,5}\left(k\right)$	$J\left(k\right)$	40	0.2	0.05
Model ESN	0.01	${\left[S_{O,5}\left(k\right),K_L{a}_5\left(k\right)\right]}^{\mathrm{T}}$	$S_{O,5}(k+1)$	40	0.2	0.05

lists the inputs and outputs of each module of the ADP controller, where $Lr$ is ESN learning rate and $R\left(k\right)$ is the set value of DO. The parameter n denotes the number of neurons in the ESN dynamic reservoir, and $SD$ is the sparsity, which indicates the percentage of the number of not interconnected neurons. $SR$ is the spectral radius of the internal connection matrix, denoted by $\rho \left(W_R\right)$

$$\rho \left(W_R\right)=max\left(\right|eig\left(W_R\right)\left|\right).$$

(67)

Taking $tanh$ as the activation function of ESN, and the sampling time is set to T = 60 s. The compensation coefficient is set to $\mu =0.14$. The utility is defined as

$$U\left(k\right)=\frac{1}{2}{\left(R\left(k\right)-S_{O,5}\left(k\right)\right)}^2.$$

(68)

6.2. Results and Analysis

Three sub-sections will be given to illustrate the advantages of the proposed ESO-ADP. Furthermore, the proportional-integral-derivative controller (PID), the improved active disturbance rejection controller (ADRC) [27], and online adaptive dynamic programming controller (ADP) [16] applied in WWTPs are introduced to evaluate the proposed ESO-ADP performance, where the PID control parameters are $P=25$, $I=0.002$, $D=0$, and the other two controllers are configured with default values from the work [16,27].

6.2.1. Constant Value Control

The simulation results of DO regulation under dry and rainy weather conditions manipulated by the designed control algorithm are shown in Figure 4 and Figure 5. The first several days are the controller’s online learning period, so the results of 7–14 days are used for evaluation. The results in Figure 4 show that the proposed ESO-ADP ensures that the DO concentration can track the reference trajectory, and the control performance is better than PID, ADRC and ADP. The DO tracking error shown in Figure 5 demonstrates the accuracy of the ESO-ADP controller which is higher than other controllers. Figure 6 demonstrates the effectiveness of ESO. From Figure 6a, the system state fluctuates greatly in the initial moments and the estimation error of ESO is relatively large. The ESO estimation error can be kept near 0 when reaching stability (see Figure 6b), thus to provide reasonable estimation of the system state and total disturbance. In addition, the output constraint of the observer limits the estimation value to the target compact set.

Some performance indicators defined in BSM1 for controller assessment are given in

Table 3. Performance index of different control strategies in dry weather.

Controllers	IAE	ISE	DEVmax
ESO-ADP	0.0028	2.246 × 106	0.0022
PID	0.5119	0.0473	0.3309
ADRC	0.0529	0.0022	0.1116
ADP	0.0394	4.951 × 10−4	0.0394

, and the performance improvement of ESO-ADP relative to the other three controllers is shown in

Table 4. The improvement of ESO-ADP performance index.

Controllers	IAE	ISE	DEVmax
PID	99.45%	99.99%	96.70%
ADRC	96.59%	99.89%	98.02%
ADP	92.89%	99.54%	94.41%

. As can be seen, the ESO-ADP controller significantly reduces the values of three performance indicators compared with the three other controllers, which indicates the proposed ESO-ADP controller can obtain better stability performance.

To evaluate the anti-interference performance of the ESO-ADP controller, the rain-weather influent data are considered. Figure 4b and Figure 5b present the results of the ESO-ADP controller under rainy weather. The DO concentrations under PID control fluctuate greatly, with a peak value 2.12 mg/L and a minimum level 1.75 mg/L. The proposed ESO-ADP has the minimal fluctuation and tracking error even though ADRC and ADP controllers greatly improve performance compared to PID. The control curves of the ESO-ADP controller in the dry and rain weather are very smooth, and the DO concentration is stable around the set value.

Table 5. Control performance of ESO-ADP controller under different weather.

Weather	IAE	ISE	DEVmax
Dry	0.0028	2.246 × 10−6	0.0023
Rain	0.0026	1.973 × 10−6	0.0023

shows the ESO-ADP controller performance in dry and rain weather. The results show that the ESO-ADP controller has excellent control accuracy and adaptability to disturbances.

Remark 4.

It is worth noting that the three controllers (PID, ADRC, and ADP) without fuzzy ESO produced poor tracking performance and large error fluctuations in the above control results. In addition, Table 4 quantitatively shows the advantages of the proposed ESO-ADP controller. Therefore, the disturbance estimation technique introduced in this paper is beneficial to achieve active disturbance suppression and high tracking accuracy.

6.2.2. Online Learning of ESO-ADP Controller

The ESO-ADP controller is completely unknown to the system in the initial stage, while the weights are updated online by interacting with the environment. Similarly, the ESO’s estimation error requires a transition process to reach convergence. The DO concentration in the first four days is shown in Figure 7. The ESO-ADP controller reaches convergence at approximately 0.2 days. The process before 0.2 days is defined as the learning process of the controller. From Figure 7a, The deviation of DO concentration from the reference value is relatively large and fluctuates widely during learning process, and the control input under the ESO-ADP controller also fluctuates greatly as shown in Figure 7b, furthermore, when stabilization is reached the proposed ESO-ADP controller is able to generate more accurate control input. Figure 8 presents the weight-adjustment process of the ESNs in the ESO-ADP controller, obviously, the weights finally achieve convergence. Figure 8b,c indicate that the weights of Critic ESN and Model ESN are fine-tuned in a small range with the fluctuation of influent flow to continuously adapt to the influence of system disturbance and thus improve the anti-interference capability of the controller. Based on the above descriptions, it can be summarized that the proposed ESO-ADP has good convergence and stability.

6.2.3. Time-Varying Reference Trajectory Control

The performance of ESO-ADP controller in the case of time-varying reference trajectory is evaluated with the reference trajectory: 2 mg/L from day 7 to day 9, 1.9 mg/L from day 9 to day 11, 2.1 mg/L from day 11 to day 13, and 2 mg/L from day 13 to day 14. Figure 9 presents the results in dry and rain weather. The DO concentration under ESO-ADP controller tracked the reference trajectory quickly when the set value has a step change, but a large overshoot occurred. Whatever, the tracking performance of the ESO-ADP controller for the time-varying reference trajectory is still satisfactory, and the proposed ESO-ADP has better tracking performance than other controllers. The DO tracking error is shown in Figure 10. It can be seen that the system stability and control accuracy under ESO-ADP are better than PID, ADRC and ADP.

7. Conclusions

For DO concentration control in BSM1, a fuzzy ESO-based ADP controller and disturbance rejection control scheme are designed and investigated. The ESN is used to approximate the optimal control policy of the system online while compensating the total disturbance of the system. The fuzzy-model-based ESO is designed to estimate the disturbances that cannot be measured in the system using the pre-identified T-S fuzzy model, and the experiments show that the proposed method has high control accuracy and strong anti-interference ability.

The proposed fuzzy ESO is suitable for state and disturbance estimation in the BSM1 DO concentration control system, and can be combined with the existing ADP controller design methods to achieve disturbance rejection control. Theoretical analysis and experimental results show that the proposed reinforcement learning anti-disturbance control system is convergent and stable. The disturbance compensation controller enables the system to have a certain suppression capability for the system disturbance. Moreover, the online learning of the ESO-ADP controller makes it adaptive and able to adapt well to the changes in the operating environment. Therefore, the controller design method used in this paper and the disturbance suppression scheme are suitable for the control problem of unknown complex nonlinear systems with unknown disturbance. The univariate control of DO concentration is considered in this paper, and the future work will take DO concentration and nitrate level into consideration, design controllers for multiple-input-multiple-output (MIMO) systems to improve the policy search and disturbance rejection capability.