Machine Learning Algorithms That Emulate Controllers Based on Particle Swarm Optimization—An Application to a Photobioreactor for Algal Growth

Abstract

Particle Swarm Optimization (PSO) algorithms within control structures are a realistic approach; their task is often to predict the optimal control values working with a process model (PM). Owing to numerous numerical integrations of the PM, there is a big computational effort that leads to a large controller execution time. The main motivation of this work is to decrease the computational effort and, consequently, the controller execution time. This paper proposes to replace the PSO predictor with a machine learning model that has “learned” the quasi-optimal behavior of the couple (PSO and PM); the training data are obtained through closed-loop simulations over the control horizon. The new controller should preserve the process’s quasi-optimal control. In identical conditions, the process evolutions must also be quasi-optimal. The multiple linear regression and the regression neural networks were considered the predicting models. This paper first proposes algorithms for collecting and aggregating data sets for the learning process. Algorithms for constructing the machine learning models and implementing the controllers and closed-loop simulations are also proposed. The simulations prove that the two machine learning predictors have learned the PSO predictor’s behavior, such that the process evolves almost identically. The resulting controllers’ execution time have decreased hundreds of times while keeping their optimality; the performance index has even slightly increased.

1. Introduction

A common task in process engineering is to control processes whose quality is evaluated through a performance index. When the process model has certain mathematical properties, theoretical control laws can be adopted for implementation; on the contrary, when the process model is uncertain, incomplete, and imprecise or has profound nonlinearities, metaheuristic algorithms (MAs) like Evolutionary Algorithms, Particle Swarm Optimization, etc., within a suitable control structure could be successfully used [1,2,3]. Control engineering has afforded numerous examples where metaheuristics were used [4,5,6,7,8] owing to their robustness and capacity to deal with complex processes.

The role of an MA within a controller is usually to predict the optimal control values within each sampling period, but first, it searches for the optimal value. For example, the PSO algorithm follows its optimization mechanism using particles and the internal PM.

A control structure fitting this type of controller is Receding Horizon Control (RHC) [9,10,11]. This structure is suitable for implementing solutions to Optimal Control Problems (OCPs); it includes an internal process model (PM) [8,12,13].

Another facet of the prediction process also used within a control problem is described, for example, in [14]. The study proposes a physics-assisted transfer learning metamodeling framework to predict laser butt welding bead geometry and carbon emissions. This time, the learning process updates the process model.

A predilect research topic for the authors was implementing the prediction module within an RHC structure employing MAs. The results are partially reflected in previous works [8,15,16]. The robustness, efficiency, and usability of MAs inside a controller have a price to pay: the controller’s execution time. The optimization mechanism and the PM’s numerous numerical integrations take a relatively long time to find the optimal value after a convergence process. That is why this approach is mainly suitable for slow processes when the predictions’ computation time is smaller than the sampling period [17,18]. Decreasing the predictor’s execution time is a challenge [15,16] because it could extend the applicability of controllers using MAs. This work goes in the same direction but involves a new technique: using machine learning (ML) to emulate predictors based on MAs. Recently, we have proposed linear regression (LR) predictors that are “equivalent” in a certain sense to predictors based on Evolutionary Algorithms (Eas) [19]. This paper deals with OCPs having final costs and solutions involving PSO predictions.

For the reader, who is a newcomer equally in the fields of control systems, computational intelligence, and machine learning, we have to answer why our objective is to replace the PSO predictor with an ML predictor (inside the RHC structure) when solving an OCP.

• The PSO predictor predicts an optimal control value, but first, it searches for the optimal control sequence following its optimization mechanism using a swarm of particles and the PM. That is why it takes a relatively long time to find this value after a convergence process.
• The ML predictor (LR or RNN) predicts using an already-known regression function. Being an ML model, it reproduces what it has learned, the PSO predictor’s behavior. It does not search for anything. Moreover, it does not make numerical integrations of the PM. That is why it takes a much shorter time to calculate the predicted value.
• The ML predictor replaces the PSO predictor only in execution when the controller achieves the control action. Intrinsically, the solution is given by the PSO algorithm. The solutions are “learned” by the ML predictor; that is, the ML model emulates the PSO algorithm.

To continue the work presented in [19], we shall consider the equivalence mentioned above and implement Regression Neural Network (RNN) predictors besides the LR ones.

Throughout this paper, we have recourse to a specific OCP to make the explanations easy to follow for the reader. In [11], for the optimal control of a specific photobioreactor (PBR) lighted for algal growth, we have presented a solution in the same context, RHC and predictions based on PSO. We shall adopt the PSO predictor already constructed in [11]; by employing this one, we shall generate new ML predictors. The data generated by simulation modules, already developed previously, are stored or recorded. These data will be needed to train and test the ML models. Some results from [11] will be reported in this paper for comparison (Section 7.1).

Section 2 recalls the general approach developed in previous work [8,20,21] to solve such problems using PSO algorithms. Besides the recall of the PBR problem’s statement, Section 2 also introduces the notations and formulas that keep the presentation self-contained.

Section 3 answers the following two main questions:

• What data are needed to capture the optimal (quasi-optimal) behavior of the couple (PSO and PM)?
• How are the data sets for the ML algorithms generated?

The PSO prediction module, included in the controller, is available from our previous work, which has already solved the PBR problem. Section 3 presents an algorithm carrying out the closed-loop simulation over the control horizon using the PSO predictor. A mandatory hypothesis is that the real process and the internal PM are considered identical because the data recorded should capture only the behavior of the couple (PSO and PM).

At the end of the simulation, the sequence of optimal control values (optimal control profile) and the sequence of state variables (optimal trajectory) can be recorded. The optimal CP and trajectory can be seen as the “signature” data of the optimal solution. Considering together the two sequences, we obtain a sequence of couples (state; control value), one couple for each sampling period. The simulation program is executed M times (e.g., two hundred times); the two sequences are collected each time and aggregated into a data structure. This data structure expresses the PSO predictor’s experience as a decision maker; it will be used to obtain the ML models [22,23,24,25,26].

Section 4 presents the general approach to learning the optimal behavior of the couple (PSO and MP). The learning process is split at the level of each sampling period, and consequently, new data structures are derived for each of them. With each new data structure, which is a collection of couples (state; control value), a generic regression function [22,27,28] is associated. The latter is materialized through an ML regression model devoted to the sampling period at hand, which must be capable of giving accurate predictions.

An ML controller’s systematic design procedure is also proposed. We have to emphasize that the entire design procedure of the ML controller needs only simulations and offline program executions.

In this paper, we consider as regression models only two kinds of models: multiple linear regression [29,30,31] and Regression Neural Network [22,32]. Other regression models (trees, support vector machines, and Gaussian processes) were considered in our studies as well. Still, only the models LR and RNN are relevant to this presentation. Implementing an ML controller, in our context, involves determining a regression model for each sampling period.

Section 5 deals with constructing a set of linear regression (LR) models [29,31] that are trained with the data sets constructed in Section 4. A general construction algorithm using the stepwise regression [30] strategy is proposed. A table with the regressions’ coefficients is extracted from the models. The LR controller is implemented using the LR predictors; it is also integrated into a proposed closed-loop simulation program, allowing us to evaluate the entire approach. Some simulation results are given.

Section 6 proposes a general algorithm for constructing the models using Regression Neural Networks [32]. The training and testing data sets are already determined in Section 4, and they are saved in an external file. A specific closed-loop simulation program is also proposed; it includes the RNN controller using the RNN predictors. The simulation results are presented for further analysis.

The Discussion section first answers the following question: Did the two kinds of predictors “learn” the behavior of the couple (PSO and PM), such that the new process’s evolutions would also be quasi-optimal? To do this, we depict the new process’s evolutions and display some numerical information using the closed-loop simulation programs proposed in Section 5 and Section 6. The simulation results are compared with those already available concerning the PSO predictor. Owing to their generalization ability, both ML controllers make accurate predictions of the control value sent to the process, and the state evolutions are practically identical.

The second question that this section has to answer is as follows: Did the controller’s execution time decrease significantly?

The positive answer to both questions proves that the ML controllers are an effective way to avoid the large execution time of the controllers based on PSO while maintaining the optimality of the control. This result is important because it extends the possibility of using PSO (or other MA) controllers for processes with smaller time constants.

Special attention was paid to the implementation aspects such that the interested reader could find support to understand and, eventually, reproduce parts of this work or use it in their projects. With this aim in view, all algorithms used in this presentation are implemented, the associated scripts are attached in the Supplementary Materials, and all of the necessary details are given in the Appendix A, Appendix B, Appendix C, Appendix D and Appendix E.

2. Controllers with Predictions Based on PSO: Connection with Machine Learning Algorithms

Many controlled processes, such as biochemical processes, are repetitive, like those organized in batches. For efficiency reasons, they generate Optimal Control Problems involving three components:

• The process model can include nonlinearities, imprecise, incomplete, and uncertain knowledge, correspond to a distributed-parameter system, etc.
• There are constraints, such as initial conditions, bound constraints, final constraints, etc.
• The cost function, which should be optimized, leads to a performance index.

To solve such a control problem, we need an adequate control structure which will define the optimal controller. The latter includes a prediction module that calculates the optimal control sequence and the optimal trajectory over the prediction horizon or even until the end of the control horizon. For its work, the predictor uses the PM for a huge number of numerical integrations. In this context, the predictor has a very complex numerical task; that is why a metaheuristic algorithm is often a realistic solution to fulfil this task.

The Receding Horizon Control (RHC) [8,10,11] is a very simple control strategy that can easily integrate a metaheuristic algorithm as a predictor (Figure 1). The authors have studied and simulated the RHC in solving different OCPs in conjunction with an EA or PSO. The solutions are realistic, they can be used in real-time control, and several techniques can be used to decrease the numerical complexity of the predictor. Nevertheless, the inconvenience is that the control action takes up a big part of the sampling period.

An interesting and practical approach, in this context, is to replace the predictor with a machine learning algorithm inside the controller. The ML algorithm must emulate the predictor of the RHC structure following its training offline. To emphasize its role, we shall refer to this algorithm as the ML controller. In this work, we have to answer the following questions:

• What does it mean that the ML algorithm must emulate the predictor?
• What data sets are used in training the ML, and how are they obtained?
• What kind of ML model can be used to achieve an appropriate controller?

In [11], we have presented the optimal control of a continuously stirred flat-plate photobioreactor (PBR) lighted on a single side for algal growth in the same context: the RHC that uses predictions based on PSO.

In this presentation, all of the essential tasks concerning the ML do not need the PM; only the final simulations, which allow us to validate the entire approach, employ the PM. That is why the reader can find in Appendix A the equations modeling the PBR, the constraints, and the cost function of the OCP. The PBR is a distributed parameter system, but the PM is converted through discretization into a lumped parameter process. We have solved this problem in [11], adding the discretization constraint, which refers to the input variables:

$$U\left(t\right)=U(kT)\stackrel{\mathsf{\Delta }}{=}U(k), \mathrm{for} k\cdot T\le t<(k+1)\cdot T; k=0,\cdots ,H-1.$$

where T is the sampling period, and the final time of the batch equals H∙T. In our example, the input vector has a single component, i.e.,

$$U(k)=q(k), k=0,\cdots ,H-1.$$

(1)

The variable $q(k)$ is the intensity of incident light throughout the kth sampling period.

At every moment $0\le k<H-1$, the predictor calculates the optimal control sequence (1) using the usual version of the APSOA (adaptive PSO algorithm) and the PM, which is integrated a large number of times. The optimal control sequence minimizes the cost function $J(k,X(k))$ over the current prediction horizon $[k,H]$; in our example, the following holds:

$$\begin{gathered} J(k,X(k))=\underset{predicted sequence}{\min }\left\{w_1\cdot A\cdot C{\displaystyle \sum _{i=k}^{H-1}U(i)}+w_2\cdot \left[V\cdot x_1(H)-m_0\right]\right\}, \\ X(k)=[x_1(k) x_2(k)]. \end{gathered}$$

The vector $X(k)$ is the current state of the process. A predicted sequence is a control sequence with the following structure:

$$\overline{U}(k)=〈U(k),\dots ,U(H-1)〉,$$

(2)

Using the PM and Equation (2), the APSOA calculates the corresponding state sequence.

$$\overline{X}(k)=〈X(k),\dots ,X(H)〉.$$

The latter has $H-k+1$ elements.

When the APSOA converges, it supplies the best sequence $\overline{U}(k)$ for the current prediction horizon, denoted by $\overline{V}(k)$:

$$\overline{V}(k)\triangleq \arg \underset{\overline{U}(k)}{\min } J(\overline{U}(k),X(k))=〈V(k), \dots , V(H-1)〉$$

(3)

The controller’s best output, denoted by $U^{*}(k)$, is the first value of this sequence, i.e.,

$$U^{*}(k)\triangleq V(k).$$

(4)

Applying Equations (3) and (4) is, in fact, the control strategy “Receding Horizon Control”.

A sequence of H control vectors, $U(0), U(1), \cdots ,U(H-1)$, will be referred to as a “control profile” (CP). The latter will produce a state transition like in Figure 2.

Finally, the controller following the RHC strategy using predictions implemented by a PSO algorithm achieves the optimal CP for the given initial state and control horizon. The optimal CP denoted by $\mathsf{\Omega }\left(X_0\right)$, which represents our problem’s solution, is the concatenation of the optimal controls $U^{*}(k), k=0,\cdots ,H-1$:

$$\mathsf{\Omega }\left(X_0\right)\triangleq 〈U^{*}(0), U^{*}(1),\dots , U^{*}(H-1)〉$$

(5)

Forced by this CP, the process will follow an “optimal trajectory” $\mathsf{\Gamma }(X_0)$:

$$\mathsf{\Gamma }(X_0)\triangleq 〈X_0,X^{*}(1),\dots ,X^{*}(H)〉.$$

(6)

A closed-loop simulation is the simulation of the controller, which includes the APSOA and PM, connected to the (real) process (see Figure 1). Our study requires only the situation when the real process and the PM are identical.

Remark 1.

The two sequences (5) and (6) can fully characterize the process’s optimal behavior in the context of closed-loop simulation over the control horizon when the process and its model are identical.

Supposing the convergence of the APSOA, the value $J_0=J(0,X(0))$ theoretically equals the optimal cost function. Practically, at the end of a closed-loop simulation, the two values will be very close to each other, so Ω(X₀) would be a quasi-optimal solution for the problem at hand.

The predictor’s behavior depends on two factors: the metaheuristic algorithm (APSOA) and the PM. In this work, the main objective is to capture the predictor’s behavior through an ML algorithm. Hence, the latter has to “learn” the optimal behavior of the couple (APSOA and PM).

Remark 2.

Our purpose is to capture the predictor’s behavior using an ML algorithm, that is, to “learn” the optimal behavior of the couple (APSOA and PM). The final objective is to replace the predictor with the new ML algorithm, such that the process’s state evolution and the performance index would be maintained. In this situation, we can state that the ML algorithm emulates the predictor.

The two sequences ($\mathsf{\Omega }\left(X_0\right)$ and $\mathsf{\Gamma }(X_0)$) are the data results of a closed-loop simulation and can be considered the “signature” data of the couple (APSOA and PM); there is a correspondence between the values $X^{*}(k)$ and $U^{*}(k)$ signifying that “when the process is in the state $X^{*}(k)$ at the moment $k$, the APSOA will predict the best control value $U^{*}(k)$”.

So, the source of the data used in a potential learning process can be a closed-loop simulation, considering the (real) process and the PM identical. Of course, the data produced by a single simulation over the entire control horizon cannot be sufficient for the learning process.

3. Data Generation Using Closed-Loop Simulation over Control Horizon

For any OCP like the PBR problem, the designed controller must be validated by closed-loop simulation, considering the (real) process and the PM identical. This validation must be carried out before using the implemented controller in real time, connected to the (real) process. Hence, we must have a simulation program that fulfils this task of closed-loop simulating over the control horizon, with a given initial state, and considering the process and the PM identical.

A simulation can be carried out in more realistic situations, for example, when the process takes into consideration, besides the PM, unmodeled dynamics and noises. But we do not need such simulations.

Remark 3.

The fact that the process and the PM are identical is not a simplification to render our study’s conclusion more favorable but is a necessity. The ML algorithm has to learn the behavior of the couple (APSOA and PM); otherwise, it will “learn”, besides the APSOA and PM, the influence of other perturbating factors.

Figure 3 shows the closed-loop simulation program’s flowchart in the conditions mentioned above. This program is generically called “ContrlLoop_PSO”. The function “Predictor_PSO” returns the predicted sequence $\overline{V}(k)$, whose first element will give the optimal control value $U^{*}(k)$. Sending the latter value to the PM and integrating the process over a sampling time, the function “ProcessStep” will determine the process’s next state, that is, at the next moment, k + 1.

When the controller designer decides to use an ML algorithm to replace the couple (APSOA and PM), the functions “Predictor_PSO” and “ProcessStep” are already written as a part of the PSO controller’s construction. This is also the case with the PBR problem; we have already accomplished the entire design procedure (for more details, see [11]).

The reader can understand and execute the “ContrlLoop_PSO” program using the script ControlLoop_PSO_RHC.m. Details are also given in Appendix B, concerning the generic functions “Predictor_PSO” and “ProcessStep”.

As we already mentioned, after the closed-loop simulation, the data generated are a couple of sequences ($\mathsf{\Omega }\left(X_0\right)$ and $\mathsf{\Gamma }(X_0)$), which can be renamed (control profile—trajectory). To prepare the data for the ML process, we shall repeat M times (e.g., M = 200) the closed-loop simulation and produce M different quasi-optimal couples (CP—trajectory). There are two reasons why data couples are different:

• The PSO has a stochastic character, and the convergence process is imperfect. So, the optimal control values are different (and so are the state vector’s values), even if the initial state is strictly the same.
• The initial state values are not the same. A standard initial state (of the standard batch) could be perturbed to simulate different initial conditions (the standard ones are imprecisely achieved).

The optimal control value and the optimal states are stored in the matrices uRHC (H × m) and state (H + 1 × n), respectively, with their structure presented in Figure 4.

Hence, the optimal CP and trajectory are described by the matrices uRHC and state, respectively, which are the images of $\mathsf{\Omega }\left(X_0\right)$ and $\mathsf{\Gamma }(X_0)$ sequences (see (5) and (6)). For each of the M simulations, the two matrices are saved in the cell array STATE and the matrix UstarRHC (M × H), as suggested in Figure A1 (Appendix B) for our case study.

The script LOOP_M_ControlLoop_PSO.m collects the data from M executions of the closed-loop simulation. The data structures presented in Figure A1 are created and loaded. A concrete example of the data collected in the first simulation is given in Appendix B.

4. The ML Controller: The Design Procedure and the General Algorithm

The M simulations can be collectively presented in Figure 5, where the state variables and control output are regrouped by sampling periods.

At each step $k, 0\le k\le H-1$, of the control horizon, the controller predicts the optimal control output relying on the couple (APSOA and PM). The state vectors considered at the same step have some common characteristics:

• The same PSO algorithm generates the M state inside a group.
• The M simulations work with the same PM.
• Each state ${X_i}^{*}(k), 1\le i\le M$, is transferred as the initial state to the predictor.
• The prediction horizon has $H-k$ sampling periods.

The APSOA calculates the prediction $\overline{V}(k)$, and the controller extracts only the optimal control values ${U_i}^{*}(k), 1\le i\le M$.

The simulation results for step k can be organized as a data set, and a table can be constructed, as shown below.

We have considered, as usual, that the state and control vector are column vectors. In our case study, the state vector (n = 2) is generated like a line vector to avoid transposition.

When M has a big enough value, the data set from

Table 1. Data set for step k.

XT	UT
${\left({X_1}^{*}(k)\right)}^T$	${\left({U_1}^{*}(k)\right)}^T$
……	……
${\left({X_M}^{*}(k)\right)}^T$	${\left({U_M}^{*}(k)\right)}^T$

represents to some extent the ability of the couple (APSOA and PM) to predict optimal control values at step k. Our desideratum is to generalize this ability to predict the optimal control when the process accesses states other than those from Table 1; this can be achieved using a machine learning algorithm.

Remark 4.

The four characteristics enumerated before are the reasons making us adopt the hypothesis that the examples (data points) of Table 1 belong to the same data-generating process; that is, they are independently identically distributed.

With each group of states presented in Figure 5, equivalent to a table like Table 1, a regression function $f_k(X)$ can be associated:

$$f_k:R^n\to R^m, k=0, 1,\dots , H-1.$$

When these functions exist, they can be used successively within the controller to replace the predictor at each control step.

Remark 5.

The regression function $f_k$ models how the APSOA determines the optimal prediction at step k. The entire set of functions $\mathsf{\Phi }=\left\{f_k\left|k=0, 1,\dots , H-1\right.\right\}$ is the couple (APSOA–PM) machine learning model. The behavior of the PSO algorithm, which, in turn, depends on the PM, is captured by the set of functions $\mathsf{\Phi }$.

To be systematic, at this point of our presentation, we propose a design procedure for the ML controller that the interested reader could use in their implementation.

Design Procedure

• Write the “ControlLoop_PSO” program simulating the closed-loop functioning of the controller based on the PSO algorithm over the control horizon. The output data are the quasi-optimal trajectory and its associated control profile ($\mathsf{\Omega }\left(X_0\right)$ and $\mathsf{\Gamma }(X_0)$).
• Repeat M times the “ControlLoop_PSO” program’s execution to produce the sequences ($\mathsf{\Omega }\left(X_0\right)$ and $\mathsf{\Gamma }(X_0)$) and save them in data structures similar to those in Figure A1 (Appendix B).
• For each sampling period k, derive data sets similar to those in Table 1 from the data saved in step 2.
• Determine the set of functions $\mathsf{\Phi }$ using the data sets derived in step 3 and an ML model; a function $f_k$ is associated with each sampling period k.
• Implement the new controller based on the ML model, i.e., the set of functions $\mathsf{\Phi }$ determined in step 4.
• Write the “CONTROL_loop” program to simulate the closed-loop functioning equipped with the ML controller. The proposed method’s feasibility, performance index, solution quality, and execution time will be evaluated.

Remark 6.

The entire design procedure of the ML controller needs only simulations and offline program executions. The ML models for each sampling period are determined offline ahead of using the ML controller in real time.

Steps 1–2 are already covered by the details given in the anterior section.

Step 3 Implementation

This step yields the data sets that the ML model would use for training and testing.

Remark 7.

The controller’s optimal behavior is specific to each sampling period, whose prediction horizon is specific to  $H-k$. So, optimal behavioral learning will be performed for each sampling period.

For each k, $k=0,\cdots ,H-1$, we construct a matrix SOCSK $\left(\mathrm{M} \times (n+m)\right)$ (SOCSK stands for “States and Optimal Control values concerning Step K”), which is Table 1’s image. Line i, $1\le i\le \mathrm{M}$, is devoted to experience i as follows:

$$SOCS{K}_i\leftarrow \left[{\left({X_i}^{*}(k)\right)}^T{\left({U_i}^{*}(k)\right)}^T\right].$$

Using the data structures proposed before, the following holds:

$$SOCS{K}_i\leftarrow \left[STAT{E}_i\left(k,1:n\right) UstarRHC(i,k)\right].$$

STATE_i is the ith element of the STATE cell array. In the PBR case (n = 2; m = 1), the data set for the current step will be as follows:

$$SOCSK=\left[\begin{array}{lll}{x}_1{\left(k\right)}^1\hfill & x_2{\left(k\right)}^1\hfill & u{(k)}^1\hfill \\ \dots \hfill & \dots \hfill & \dots \hfill \\ x_1{\left(k\right)}^i\hfill & x_2{\left(k\right)}^i\hfill & u{(k)}^i\hfill \\ \dots \hfill & \dots \hfill & \dots \hfill \\ x_1{\left(k\right)}^M\hfill & x_2{\left(k\right)}^M\hfill & u{(k)}^M\hfill \end{array}\right].$$

A fragment of the SOCSK matrix produced by a MATLAB script is presented in Appendix C for step k = 1. Only when k = 0 does the value of x₂(0) equal 0 for any observation.

Owing to Remark 7, step 3 should establish for each k the data sets for training and testing; these sets are stored in the cell arrays DATAKTest and DATAKTrain.

Table 2. The pseudocode preparing the data sets for the ML models’ training and testing.

/*This pseudocode describes the construction of the data sets needed by the ML models at the level of each sampling period*/ Inputs: cell array STATE, matrix UstarRHC; Outputs: matrix SOCSK, table datak, cell arrays DATAKTest, DATAKTrain
1.	#Load the file containing the data structure STATE and UstarRHC (Figure A1)
2.	$\mathrm{k}\leftarrow 0$
3.	while  $\mathtt{k}\mathtt{\le }\mathtt{H}\mathtt{-}\mathtt{1}$
4.	for i = 1, ∙∙∙, M
5.	${\mathtt{SOCSK}}_{\mathtt{i}}\mathtt{\leftarrow }\left[{\mathtt{STATE}}_{\mathtt{i}}\left(\mathtt{k}\mathtt{,}\mathtt{1}\mathtt{:}\mathtt{n}\right) \mathtt{UstarRHC}\mathtt{(}\mathtt{i}\mathtt{,}\mathtt{k}\mathtt{)}\right]$
6.	end
7.	#Convert the matrix SOCSK into the table datak.
8.	datakTest ← lines #1—60 of datak
9.	datakTrain ← lines #61—120 of datak
10.	DATAKTest{k} ← datakTest
11.	DATAKTrain{k} ← datakTrain
12.	$\mathrm{k}\leftarrow \mathrm{k}+1$
13.	end
14.	#Save the cell array DATAKTrain and DATAKTest in a file.

presents the pseudocode of the script preparing all of the sets needed by the learning algorithm.

Step 4’s implementation will determine the set of ML models $\mathsf{\Phi }=\left\{f_k\left|k=0, 1,\dots , H-1\right.\right\}$ and will be addressed in the next section.

Step 5 aims to implement the ML controller. Once the set of regression models is determined in step 4, the controller can be written following the algorithm in

Table 3. The structure of the ML controller’s algorithm.

The General Algorithm of the ML Controller
	/*The controller program is called at each sampling period, k */
1	Get the current value of the state vector, X(k); /* Initialize $k \mathrm{and} X(k)$*/
2	Predict the optimal control value $U*(k)$ using the regression model $f_k(X(k))$ /* whatever is the regression model’s type */
3	Send the optimal control value $U*(k)$ towards the process.
4	Wait for the next sampling period.

.

Notice that the cumulative effect of calling the controller at each sampling period is to achieve the following sequence of predictions using the regression models and the current states that the process accesses:

$${U_1}^{*}=f_0(X_0); {U_2}^{*}=f_1(X_1); \dots {U_{H-1}}^{*}=(X_{H-1}).$$

In the sequel, the controllers based on ML models will be called LR controller (from linear regression) or RNN controller (from Regression Neural Network).

5. Linear Regression Controller

5.1. General Algorithm

The first approach that the authors considered was to use multiple linear regression for the function set $\mathsf{\Phi }$. Such a model contains an intercept, linear terms for each state variable, squared terms, products of features (interactions), etc. Hence, as functions of state variables, the regression functions could be nonlinear.

For our example, the stepwise regression strategy [30], which adds or removes terms starting from a constant model, was also applied. We consider in this presentation only models with an intercept, linear terms, and an interaction:

$$f_k(X(k))={C^k}_0+{C^k}_1\cdot x_1(k)+{C^k}_2\cdot x_2(k)+{C^k}_{12}\cdot x_1(k)\cdot x_2(k).$$

(7)

Remark 8.

Our goal is not to find the best sequence of linear regression models but to validate our approach, i.e., the ML model can successfully replace the couple (APSOA and PM) inside a new controller.

As we shall see, the model (7) is largely sufficient for our goal.

Table 4. The pseudocode of the linear regressions’ construction.

Construction of the linear regression models Input:  cell arrays DATAKTrain, DATAKTest Output: matrix KOEF ($H\times (n+1)$), /* the regression coefficients for each                   sampling period */     cell array MODELSW {$H\times 1$} /* cell array storing objects that are the                   linear models $f_k$ */
1	for k = 0…H-1.
2	datakTrain$\leftarrow$ DATAKTrain{k}; /* Recover the data set for training */
3	datakTest$\leftarrow$ DATAKTest{k}; /* Recover the data set for testing */
4	mdlsw$\leftarrow$fitting_to_data(datakTrain); /* Training the linear regression */
5	#display mdlsw; /* mdlsw is the linear regression model */
6	coef(:)$\leftarrow$get_the_coefficients(mdl)
7	KOEF(k,:) $\leftarrow$coef(:); /* The kth line of KOEF receives the coefficients */
8	MODELSWP{k,1}$\leftarrow$mdlsw;
9	uPred $\leftarrow$fpredict(mdlsw, datakTest) /* The vector uPred stores the predicted control values */
10	# Make the comparison between uPred and the real control values;
11	end.

presents the construction of the H models representing linear regressions in a general manner, that is, not only for our example. This pseudocode describes the linear models’ training and testing using the sets generated in step 3.

The script in Table 4 uses the generic functions fitting_to_data, get_the_coefficients, and fpredict, which make actions suggested by the comments.

The implementation of this algorithm is included in the script GENERATE_ModelSW; some details are given in Appendix D.

5.2. Simulation Results

Although we have determined the usual linear regressions that contain the two linear terms (for $x_1$ and $x_2$) and an intercept, we present hereafter the stepwise version as it is implemented in the MATLAB system.

Table 5. The actions and results of the stepwise linear regression for the 14th sampling period.

&&&&kp1 = 14
1. Adding x1, FStat = 12.7755, p Value = 0.000484491
Linear regression model:   u ~ 1 + x1
Estimated Coefficients:
	Estimate	SE	tStat	p Value
(Intercept)	−985.91	437.95	−2.2512	0.025952
x1	2147.7	600.87	3.5743	0.00048449
Number of observations: 140; Error degrees of freedom: 138 Root mean square error: 117 R-squared: 0.0847; Adjusted R-squared: 0.0781 F-statistic vs. constant model: 12.8; p-value = 0.000484

displays a listing’s fragment obtained during the script GENERATE_ModelSW’s execution; this one presents the model for a single sampling period.

The procedure begins with only an intercept, and after that, it tries and succeeds in adding the term corresponding to $x_1$. Statistical parameters do not allow us to add another term. So, the prediction (control value) will be $f_{13}([x_1,x_2])=-985.91+2147.7\cdot x_1$.

We notice that the training time for all 120 linear regressions is 6.166654 s.

Following the algorithm presented before, the resulting coefficients of the H regression are given in

Table 6. The coefficients of the H linear regressions determined with a stepwise strategy.

k	C0	C1	C2
0	564.18		0
1	585.53	0	0
2	−20.368	1441.5	0
:	:	:	:
9	591.85	0	0
10	119.74	0	620.84
11	591.48	0	0
12	590.95	0	0
13	−985.91	2147.7	0
14	−328.16	1205.6	0
:	:	:	:
111	4055.1	−1476.6	0
112	5482.6	−2067.6	0
113	597.8	0	0
114	3300.2	0	−308.67
115	587.66	0	0
116	6446.3	−2451.2	0
117	7144	−2732.8	0
118	4410.7	0	−419.32
119	565.2	0	0

.

There are sampling periods for which the regression model has only the intercept C₀; this situation will be discussed in Section 7. These coefficients will be used directly by the controller as a control law.

The comparison achieved in lines #9-10 of the construction algorithm is summarized in Figure 6.

We have to mention that the predicted values were calculated directly using the formulas, not using the generic function fpredict. The table datakTest supplied the 60 examples, states—control value, for testing the linear regressions.

The quality of the predictions will be evaluated at the time of using the regression models within the controller, that is, inside the closed-loop simulation. The ultimate evaluation of predictions would be the optimality of the process evolution.

To prepare this evaluation, we need the simulation program for the control loop working with the LR controller. The flowchart in Figure 7 describes this program, CONTROL_loopLINREG, which is step #6 of the design procedure.

Although there are similarities with Figure 3, there actually are big differences in execution:

• The state variable has two elements.
• The coefficients’ matrix must be loaded from an existing file.
• The gray instructions make the predictions, avoiding any numerical integration.
• The green instruction updates the next state, which has two components. The amount of light irradiated in the current sampling period is added to x₂(k).

Only the orange column of the flowchart has big similarities because it is about the simulation results needed to depict the process evolution and the performance index.

The script CONTROL_loopLINREG.m included in the attached folder implements the presented algorithm.

The closed-loop simulation program produces a listing, a fragment of which is presented in Figure A3 (Appendix D), and two drawings reproduced in Section 7.

We notice the very short time period used to control the process over the entire control horizon, 0.6401 s (the simulation processor is Intel(R) Core(TM) i7-6700HQ CPU @ 2.60 GHz).

6. Controller Based on Regression Neural Networks

6.1. General Approach

Because the linear regression could seem much too simple, we have studied other types of models (trees, support vector machines, and Gaussian processes) trying to improve capturing the optimality of the couple (APSOA and PM), the final target being that the designed ML controller would better approach the optimal behavior. The obtained models have performances weaker than those of the LR and RNN models.

Better predictions than those obtained with other types of ML models are produced by Regression Neural Networks (RNNs), of course, with a possible penalty concerning the model’s size. The decision to choose between these models and the linear regressions in implementing the controller will be analyzed in Section 7.

As in the case of linear regressions, the RNN models must be obtained offline, and their construction must be organized in a loop because the number of sampling periods could be large, like in our case. The pseudocode of the RNNs’ construction is presented in

Table 7. The pseudocode of the regression NNs’ construction.

Construction of the RNN models Input: cell arrays DATAKTrain, DATAKTest Output: cell array MODELNN {$H\times 1$} /* cell array storing objects that are RNN */
1	for k = 0…H-1.
2	datakTrain$\leftarrow$ DATAKTrain{k, 1}; /* Recover the data set for training */
3	datakTest$\leftarrow$ DATAKTest{k, 1}; /* Recover the data set for testing */
4	mdlNN$\leftarrow$trainRegNN(datakTrain);  /* Training the RNN */
5	MODELNN{k,1}$\leftarrow$mdlNN /* Store the object mdlNN into the cell array MODELNN */
6	predictionNN$\leftarrow$mdlNN.predictFcn(datakTest) /* Make predictions and store them into the table predictionNN */
7	# Comparison between predictionNN and datakTest
8	end

.

In our case study, for k = 0, i.e., the first sampling period, we have a special situation because $x_2\equiv 0$ (the light amount equals 0 through initialization) for all examples. Hence, this variable cannot be a prediction variable. For this situation, the tables datakTrain and datakTest have different structures, and the RNN model has a single predictor variable $x_1$. To maintain the general structure of the algorithm in Table 7, we did not treat the first sampling period distinctly.

Most data structures were presented before, except for the cell array MODELNN that collects the model for each sampling period, called “mdlNN”. The function trainRegNN trains the current RNN using its specific data set [32].

In line #6, the predictions made by the method “mdlNN.predictFcn” are stored in the local table predictionNN, which can be compared to datakTest or saved for further utilization.

The implementation of this algorithm is achieved by the script GENERATE_ModelNN; some details are given in Appendix E.

6.2. Simulation Results

The execution of the script GENERATE_ModelNN gives us an indication of the RNNs’ construction complexity (training and testing). A fragment of its listing is given in

Table 8. The execution of RNN training (a fragment of the listing).

>> GENERATE_ModelNN

&&& vrmse = 108.4187 RMSEValid = 111.9601 kplus1 = 1 uNN = 564.1789 valreal = 487.0276 &&& vrmse = 104.5341 RMSEValid = 130.6362 kplus1 = 2 uNN = 574.4657 valreal = 672.5530 &&& vrmse = 106.0061 RMSEValid = 127.3596 kplus1 = 3 uNN = 559.1601 valreal = 676.6160 ---------------------- &&& vrmse = 112.0696 RMSEValid = 91.4825 kplus1 = 119 uNN = 588.0487 valreal = 634.9497 &&& vrmse=130.1694 RMSEValid = 125.2698 kplus1 = 120 uNN = 576.2782 valreal = 714.3906 Elapsed time is 232.829571 s.

.

The RMSEValid is the root mean square error (RMSE) between the predictions and datakTest vectors. The vrmse value is the RMSE calculated in the training process (the phase of validation). The program displays the k value, the prediction uNNn, and the control value (valreal) for the states included in record #10 of the data set (as an example).

We notice that all 120 RNNs are trained in 233 s (offline, as we mentioned before).

The comparison mentioned in line #7 of the algorithm can be achieved by calculating the root mean square error (RMSE) between the predicted and observed values. For graphical analysis, Figure 8 plots the predicted values yielded by the RNN model versus the real control values from the datakTest table.

As in Section 5.2, to evaluate the controller’s performances, we need the simulation program for the closed-loop functioning with the RNN models. Its algorithm’s flowchart would be very similar to that of Figure 7, except for two instructions. That is why we do not redraw the flowchart, as we are content with indicating only the changes.

The gray instruction

This block means that the neural network model mdlNN is selected as the current RNN from the cell array of objects MODELNN. Its method predictFcn will calculate the predicted control value as a function of the current state.

The second change is inside the block “Initializations”. Instead of loading the coefficients’ table, C, it will load the cell array MODELNN; the latter is already saved in a file created by the script constructing the RNN models (see GENERATE_ModelNN.m).

The script CONTROL_loopNN.m included in the attached folder implements the above algorithm.

The simulation program CONTROL_loopNN produces a listing, a fragment of which is presented in Figure A4 (Appendix E), and two drawings, presented in Section 7.

We notice, as in the case of the linear regression controller, the very short time period used to control the process over the entire control horizon, 1.35 s.

7. Discussion

7.1. Comparison between PSO and ML Predictors

In this section, we have to answer the following questions:

• Did the ML predictors succeed in “learning” the behavior of the couple (APSOA and PM), such that the process’s evolution would be quasi-optimal?
• Did the controller’s execution time decrease significantly?

As we mentioned, we have already solved the PBR problem using RHC and a predictor based on PSO. For the sake of simplicity, we shall refer to its controller as the PSO Controller. Using the ControlLoop_PSO script, the closed-loop simulation produces the typical evolution depicted in Figure 9. This simulation is one among the M = 200 evolutions that contributed to our big data set with CPs and trajectories. The final lines of the simulation’s listing summarize its performances, as given in

Table 9. Execution of closed-loop simulation program (ControlLoop_PSO).

>>ControlLoop_PSO_RHC

x00 = 0.3660

Yield mass = 3.0000

Light = 9.2474

Perf index = 9.2474

Elapsed time is 447.281879 s.

.

The simulation programs CONTROL_loopLINREG and CONTROL_loopNN produce, besides the data in Figure A3 and Figure A4, the evolutions depicted in Figure 10 and Figure 11, respectively.

Remark 9.

The state evolutions ($x_1$ and $x_2$) and the evolution of the biomass, which can be considered the output of the process, are practically identical. Hence, both ML controllers emulate the PSO controller.

The LR controller and the RNN controller are facing situations when the current state is totally new (states unobserved in the training or testing data sets). In this situation, the generalization ability of the ML model is exploited but also verified. That is why the simulation programs are adequate tests for the predictions’ testing.

Remark 10.

Using the controller inside a closed-loop simulation program over the control horizon will be a test in which the predictor experiences new process states unobserved in the training and testing phase of the ML model’s construction.

The fact that Figure 9a, Figure 10a and Figure 11a, describing the control value’s evolution, are very different has no relevance to the matter at hand; the following aspects uphold this:

• The similarity at this level would imply the same sequences of states, but we just stated that the ML controllers could experience new unobserved states. So, the three processes do not pass through the same set of states (the sets of accessible states are different).
• The learning is made at the level of each sampling period and the model “learns” couples (state; control value), not globally, but at the control profile level; our method is not based on learning CPs. On the other hand, the PSO predictor is very “noisy” due to its stochastic character and produces outliers among the 200 control values from time to time.

To make a quantitative comparison among the three controllers,

Table 10. Quantitative comparison among the three controllers.

	PSO Controller	LR Controller	RNN Controller
x00	0.360	0.360	0.360
Yield mass	3.0000	3.0282	3.0249
Light	9.2474	9.3166	9.3115
Perf index	9.2474	9.5981	9.5604
Control time [s]	447.28	0.64	1.35
Training time [s]	-	5.4	232.83
Model size	-	3 kB	17 kB
Root mean square error (RMSE for k = 10)	-	113.73	116.63

displays some pieces of information from Figure A3 and Figure A4, and Table 9.

The first four lines of Table 10 are evidence that the three controllers can be considered equally performant, although the ML controllers have slightly better parameters. However, the time devoted to controlling the process over the control horizon (all of the H sampling period) has values much inferior to that of the PSO controller (hundreds of times smaller). This analysis and Remark 9 allow us to state the following conclusions:

• The two ML controllers have predictors that have learned the behavior of the couple (APSOA and PM), such that in closed-loop functioning, the process evolves almost identically.
• The resulting controllers have execution times hundreds of times smaller than that of the PSO Controller.

7.2. Comparison between the LR and RNN Controllers

In Table 10, all of the parameters in the second column are superior to those in the third column. The differences are not relevant for some of them, but the control time, training time, and model size make the LR controller preferable to the RNN controller. However, we have trained all its RNNs using hyperparameter optimization.

The usual comparison between the predicted and real (observed) values is illustrated in Figure 6 and Figure 8 for the LR and RNN predictors, respectively. This comparison is made for k = 10 (as an example) and its testing table datakTest. For the other values of k, the situation is the same.

At first sight, both predictors seem to be similar, but the values of RMSE from Table 10 show that the LR predictor is slightly better than the RNN predictor. This remark goes in the same direction as the superiority of the LR controller. However, because the tatakTest has 60 data points, we can consider the difference between RMSEs irrelevant and that they are similar from an accuracy point of view.

Considering Remark 6, the time to train offline 120 RNNs (for the RNN controller) of 4 min is absolutely acceptable. So, even this controller can be considered a good solution for the PBR problem or another OCP.

For the reader, who is a newcomer equally in the fields of control systems, computational intelligence, and machine learning, we must compare the role of the PSO predictor versus the role of the ML predictor when solving an OCP.

• The ML predictor replaces the PSO predictor only in execution when the controller achieves the control action. So, the controller’s execution time is hundreds of times smaller compared to initially. That was our desideratum.
• When we solve a new OCP, sometimes we need a metaheuristic (PSO, EA, etc.) that searches for the optimal solution inside of a control structure. If the controller’s execution time is not acceptable, we can use the approach presented in this paper to create an ML controller. However, initially, we need the MA to search for the optimal solution.

8. Conclusions

In this paper, we have proposed two ML controllers (LR controller and RNN controller), including the linear regression and Regression Neural Network predictors, that can replace the controller using a PSO algorithm; the optimal control structure works with an internal process model.

The main conclusions are given as follows:

• The machine learning models succeed in “learning” the quasi-optimal behavior of the couple (PSO and PM) using data capturing the PSO predictor’s behavior. The training data are the optimal control profiles and trajectories recorded during M offline simulations of the closed-loop over the control horizon.
• The current paper proposes algorithms for collecting data and aggregating data sets for the learning process. The learning process is split according to the level of each sampling period so that a predictor model is trained for each one. The multiple linear regression and the Regression Neural Networks are considered the predicting models.
• For each case, we propose algorithms for constructing the set of ML models and the controller (LR or RNN controller). Algorithms for the closed-loop simulations using the two controllers are also proposed; they allow us to compare the process evolutions involved by the three controllers, the PSO, LR, and RNN controllers.
• The final simulations show that the new controllers preserve the quasi-optimality of the process evolution. In the same conditions, the process evolutions are almost identical.
• An advantage of our approach refers to data collection, data set preparation for the training process, and the construction of ML models; all of these activities need only simulations (using the PSO controller) and offline program executions (Remark 6). The ML models for each sampling period are determined offline ahead of using the ML controller in real time.
• We emphasize that during the final closed-loop simulations, the ML controller encounters new process states unobserved in the training and testing of its predictor (Remark 10). Owing to its generalization ability, the controller makes accurate predictions of the control value sent to the process.

The PSO predictor first searches for the optimal control value, following its optimization mechanism using particles and the PM, before predicting it. This search sometimes means a big computational effort and a large controller execution time. The ML controller (LR or RNN) predicts using an already-known regression function, which can emulate the PSO predictor. In other words, the ML predictor replaces the PSO predictor only in execution when the controller achieves the control action. That is why the controller’s execution time decreases drastically. However, the solution belongs intrinsically to the PSO predictor.

When we solve a new OCP, sometimes, for different reasons, we shall need a metaheuristic (PSO, EA, etc.) searching for the optimal solution inside of a control structure. Finally, the implemented controller integrated into the control structure can be one of the two ML controllers.

In our opinion, this work goes beyond the controller’s execution time decrease and opens a perspective to emulate and replace in a general manner optimization structures.