Online Tuning of a PID Controller with a Fuzzy Reinforcement Learning MAS for Flow Rate Control of a Desalination Unit

Abstract

This paper proposes a hybrid Zeigler-Nichols (Z-N) fuzzy reinforcement learning MAS (Multi-Agent System) approach for online tuning of a Proportional Integral Derivative (PID) controller in order to control the flow rate of a desalination unit. The PID gains are set by the Z-N method and then are adapted online through the fuzzy Q-learning MAS. The fuzzy Q-learning is introduced in each agent in order to confront with the continuous state-action space. The global state of the MAS is defined by the value of the error and the derivative of error. The MAS consists of three agents and the output signal of each agent defines the percentage change of each gain. The increment or the reduction of each gain can be in the range of 0% to 100% of its initial value. The simulation results highlight the performance of the suggested hybrid control strategy through comparison with the conventional PID controller tuned by Z-N.

1. Introduction

The lack of potable water in many areas worldwide and especially in rural (lack of potable water) areas has led to the development of different desalination processes. Such methods are the desalination from distillation, electro dialysis, and reverse osmosis [1]. The desalination process needs an amount of energy in order to produce potable water. From the aforementioned processes, the more energy efficient process is the reverse osmosis. Due the fact that a large number of the desalination plants have been installed in rural areas where the production of energy is limited, due to local generation, makes the reverse osmosis desalination process an ideal choice. In these areas, due to lack of energy generation, energy management amongst the generators, the consumptions, and the storage units must be applied. Thus, the desalination plant, as a consumption unit, can be controlled in order to consume less or more energy according to the power balance and the surplus of the potable water. The control of the desalination plant can be achieved by controlling the flow rate of the potable water by changing the feeding voltage. The most well-known method in order to perform control processes in industrial applications is the proportional integral derivative (PID) control, due to its robustness and the ease of implementation. The PID controller needs the adjustment of its three gains. There are many methods used to set these gains to a value that gives an adequate control, with the most well-known being the Zeigler-Nichols (Z-N) [2]. In this method the gains of the controller are set to their values according to the control process. The advantage of this method is the ease of implementation, on the other hand this is an empirical method that generally delivers very large overshoot. In order to overcome this limitation, intelligent PID controllers have been introduced by embedding intelligent techniques. These techniques are mainly offline techniques which are based on genetic algorithms [3,4,5,6], ant colony [6], particle swarm optimization [7,8], multi-objective extremal optimization [9], extended classifier system algorithm [10], etc. These methods calculate the gains of the PID controller regarding a fitness function. These methods lead to better control performance, compared to the Z-N methods, but the drawback of these methods lies in keeping the gains constant through the entire control process. Keeping the gains constant can many times lead to poor controlling results due to the dynamic nature of the processes. In order to overcome this limitation, many techniques have been developed for online adaptation of the gains. These methods are mainly based on fuzzy logic [11,12,13] and neural networks [14,15,16,17,18,19,20]. These techniques have the drawback that, in many cases, they require the knowledge of the expert regarding the process. Furthermore, they use complex structures, which means increment of development complexity and computational resources.

Specifically, in terms of controlling desalination units, the use of intelligent PID controllers is limited and lies in only offline approaches. Genetic algorithms have been used in order to optimize the performance of the PID by keeping the desalination plant operation close to optimal [21]. Additionally, particle swarm optimization has been used to tune the parameters of the PID [22,23]. The main disadvantage of these methods is that they can only optimize the controller offline and cannot confront the changes of the dynamic behavior of the process.

This paper proposes a hybrid Z-N fuzzy Q-learning [24] multi agent system (MAS) approach for online adjustment of the gains of the PID controller. By deploying an online adjustment of the PID gains, the overshoot of the Z-N method is limited and, simultaneously, problems regarding the dynamic nature of the process are confronted. The three gains of the PID controller are initially set by the Z-N method and then through the control process are adjusted online via the MAS. The MAS consists of three agents, each agent is dedicated to the adjustment of only one gain. This allows the online adjustment of the gains by interaction with the environment without the need of the expert’s knowledge. Additionally, decomposing the problem to simpler ones keeps the implementation complexity and the required computational resources at a low level. Fuzzy Q-learning has been used in our previous work in order to supervise a PID controller for a motor speed application [25]. The results in this work were very encouraging even if only one agent was used. The main drawback of this approach was the usage of one input of the agent (error) in order to not exponentially increase the state-action space. Additionally, fuzzy Q-learning in MAS has been successfully applied in energy management application for direct control of the power exchange amongst the units of a microgrid [26].

The main contributions of this paper are as follows:

• We arrange a Q-learning MAS for adapting online the gains of a PID controller. The initial values of the gains have been set by the Z–N method.
• The PID controller is used to control the flow rate of a desalination plant. The proposed control strategy is not only independent of prior knowledge, but it is also not based on the expert’s knowledge.
• The computational resources and the implementation complexity remain low in respect to other online adaptation methods. This makes the proposed approach easy to implement.
• In order to deal with the continuous state-action space, a fuzzy logic system (FLS) is used as a fuzzy function approximator in a distributed approach.

The structure of this paper is as follows: Section 2 provides preliminaries about PID controllers, fuzzy logic systems, reinforcement learning, and MAS. Section 3 presents the model of the desalination plant and the proposed control strategy. Section 4 presents the experimental results based on the simulated process. Section 5 discusses the experimental results and outlines future work.

2. Preliminaries

2.1. PID Controller

The block diagram of the PID controller is depicted in Figure 1. The input of the PID controller is the error $e\left(t\right)$, the error arises by the subtraction of the set point $r\left(t\right),$ with the true value of $y\left(t\right)$. The aim of the PID controller is to produce the appropriate control signal $u\left(t\right)$ in order to eliminate the error.

In order to increase the system’s response speed and to decrease steady-state error, the proportional gain ($Kp$) was used in order the control signal u(t) to respond to the error immediately. The integral gain ($ki$) was used for eliminating the offset error but caused overshoot. In order to reduce this overshoot, the derivative gain ($kd$) was used [27]. The transfer function of the PID controller was:

$$G_{PID}\left(s\right)=kp+ki·\frac{1}{s}+kd·s$$

(1)

2.2. FLS

The main advantage of fuzzy logic systems is that they can express complex processes by using fuzzy if/then rules. These rules contain knowledge of the expert and can express relationships among fuzzy variables using linguistic terms [28]. The generic form of a fuzzy rule is:

$$Ru:\mathrm{if}(x_1\mathrm{is}{D}_1)\mathrm{and}/\mathrm{or}(x_2\mathrm{is}{D}_2)\dots \mathrm{and}/\mathrm{or}(x_m\mathrm{is}{D}_m\left)\mathrm{then}\right(c\mathrm{is}E),$$

where $D_j,j=1,\dots ,m$ is a fuzzy set of $j$th input, $\mathit{x}=\left(x_1,x_2,\dots ,x_m\right)$ is the crisp input vector, c is the output variable, and $E$ is a fuzzy set defined by the expert.

The Sugeno [29] type FLS is the most commonly used due to its computational efficiency, as it does not need a defuzzifier. In the simplest case, where a zero-order type Sugeno (the consequents of the rules are constant numbers) is used, the global output of the FLS can be calculated by the Wang-Mendel model [30].

$$a\left(\mathit{x}\right)=\frac{{{\displaystyle \sum }}_{i=1}^N{w}_i\left(\mathit{x}\right)a_i}{{{\displaystyle \sum }}_{i=1}^N{w}_i\left(\mathit{x}\right)}$$

(2)

where $a_i$ is the consequent of rule $i$ and $w_i$ is the firing strength of the rule $i$.

2.3. Reinforcement Learning

The term reinforcement learning refers to a family of algorithms inspired by human and animal learning. The aim of the reinforcement learning is to find a policy that maximizes the expected discounted reward [31]. The maximization of the expected discounted reward is performed via exploration/exploitation in the space of possible state-action pairs. Actions that give good performance when performed in a given state are rewarded with a positive reinforcement signal (reward), and actions that give poor performance are punished with a negative reinforcement signal (punishment). This signal provides feedback to the system in order to learn the “value” of actions in different states.

2.3.1. Q-Learning

The Q-learning is a reinforcement learning method. In this method the agent calculates the Q-function which evaluates the future discounted rewards of candidate actions performed in states [32]. The Q-function’s output for a given state $\mathit{x}$ and an action $a$ is denoted as $Q\left(\mathit{x},a\right)$. The Q value of each action $a$ when performed in a state $\mathit{x}$ is updated as follows:

$$Q{}^{\prime }\left(\mathit{x},a\right):=Q\left(\mathit{x},a\right)+\eta \left(R\left(\mathit{x},a,{\mathit{x}}^{\prime }\right)+\gamma {}_a{}^{max}Q\left(\mathit{x}{}^{\prime },a\right)-Q\left(\mathit{x},a\right)\right)$$

(3)

where $Q{}^{\prime }\left(\mathit{x},a\right)$ is the new value of the state-action combination after the updating from the reward $R\left(\mathit{x},a,{\mathit{x}}^{\prime }\right)$ which acquired by performing the action $a$ in the state $\mathit{x}$. The algorithm of Q learning assumes that the agent continues from state $\mathit{x}$ by performing the optimal policy, thus ${}_a{}^{max}Q\left({\mathit{x}}^{\prime },a\right)$ is the maximum value when the best action is applied in state $\mathit{x}{}^{\prime }$ (the next state which arises after executing the action $a$ in state $\mathit{x}$). The learning rate $\eta$ defines the degree that the new information overrides the old one [33], and the discount factor $\gamma$ defines the importance of the future rewards [34]. Generally, the Q-learning agent selects the action $a$ to be performed in state $\mathit{x}$ and identifies the consequent state $\mathit{x}{}^{\prime }$. It obtains the reward from the transition $R\left(\mathit{x},a,{\mathit{x}}^{\prime }\right)$, and updates the value for the combination $\left(\mathit{x},a\right),$ supposing that the optimal policy is applied from state $\mathit{x}$ and onwards. A Q-learning agent uses an exploration/exploitation scheme in order to figure a policy that maximizes payoff.

The main characteristic that makes Q-learning suitable to our study is its model-free approach. On the contrary, the main disadvantage of the Q-learning is that it cannot be applied to continuous action-state spaces that our problem involves. In order to confront this problem, fuzzy function approximation can be used.

2.3.2. Fuzzy Q-Learning

One main advantage of FLS is that they can achieve good approximations [35] in the Q-function making possible the usage of the Q-learning in continuous state-space problems (fuzzy Q-learning) [36]. In fuzzy Q-learning, $\mathit{x}$ is the crisp set of the inputs that defines the state of the agent. These are converted into fuzzy values and each fuzzy rule corresponds to a state. Thus, the firing strength of each rule determines the degree to which the agent is in a state. Additionally, the rules do not have standard consequents; meaning there are no predefined state-action pairs, and the consequents of each rule arises via the exploration/exploitation algorithm. So, the FLS has competing actions for each rule and the rules have the form:

$$\begin{array}{llll}\mathrm{if}& \mathit{x}is{S}_i& \mathrm{then}& \alpha \left[i,1\right]\\ & & \mathrm{or}& \alpha \left[i,2\right]\\ & & ⋮\\ & & \mathrm{or}& \alpha \left[i,k\right]\end{array}$$

where $\alpha \left[i,k\right]$ is the $k^{th}$ potential action in fuzzy rule $i$. For each candidate action in rule $i$ there is a corresponding value $q\left[i,k\right]$. This value defines how “good” the action is performed, when the rule $i$ is fired. The state $S_i$ is defined by $\left(x_{1}is{S}_{i,1}and{x}_{2}is{S}_{i,2}\dots and{x}_{n}is{S}_{i,n}\right)$, where $S_{i,j}$, $j=1,\dots ,n$ are fuzzy sets.

The fuzzy Q-learning algorithm has to follow:

• Observe state $\mathrm{x}$
• Select an action for each fired rule according to the exploration/exploitation algorithm.
• Calculate the global output $a\left(x\right)$ from the following equation:

  $$a\left(\mathit{x}\right)=\frac{{{\displaystyle \sum }}_{i=1}^N{w}_i\left(\mathit{x}\right)a_i}{{{\displaystyle \sum }}_{i=1}^N{w}_i\left(\mathit{x}\right)}$$

  (4)

  where $a_i$ corresponds to the chosen action of rule $i$.
• Calculate the corresponding value $Q\left(x,a\right)$ as follows:

  $$Q\left(\mathit{x},a\right)=\frac{{{\displaystyle \sum }}_{i=1}^N{w}_i\left(\mathit{x}\right)q\left[i,i^{†}\right]}{{{\displaystyle \sum }}_{i=1}^N{w}_i\left(\mathit{x}\right)}$$

  (5)

  where $q\left[i,i^{†}\right]$ is the corresponding q-value of the fired rule $i$ for the choice of the action $i^{†}$ by the exploration/exploitation strategy.
• Apply the action $a\left(x\right)$ and observe the new state $\mathrm{x}{}^{\prime }$.
• Calculate the reward $R\left(x,a,x{}^{\prime }\right).$
• Update the q values as follows:

  $$\Delta q\left[i,i^{†}\right]=\eta \Delta Q\frac{w_i\left(\mathit{x}\right)}{{{\displaystyle \sum }}_{i=1}^N{w}_i\left(\mathit{x}\right)}$$

  (6)

  where $\Delta Q=R\left(\mathit{x},a,\mathit{x}{}^{\prime }\right)+\gamma ·Q\left(\mathit{x}{}^{\prime },a^{*}\right)-Q(\mathit{x},a,Q\left({\mathit{x}}^{\prime },a^{*}\right)=\frac{{{\displaystyle \sum }}_{i=1}^N{w}_i\left({\mathit{x}}^{\prime }\right)q\left[i,i^{*}\right]}{{{\displaystyle \sum }}_{i=1}^N{w}_i\left({\mathit{x}}^{\prime }\right)}$ and $q\left[i,i^{*}\right]$ is the choice of the action $i^{*}$ that has the maximum Q value for the fired rule $i$. The block diagram of a fuzzy Q-learning agent is depicted in Figure 2. Thus, the q values are updated as:

  $$q{}^{\prime }\left[i,i^{†}\right]:=q\left[i,i^{†}\right]+\eta \left(R\left(\mathit{x},a,{\mathit{x}}^{\prime }\right)+\gamma Q\left({\mathit{x}}^{\prime },a^{*}\right)-Q\left(\mathit{x},a\right)\right)\frac{w_i\left(\mathit{x}\right)}{{{\displaystyle \sum }}_{i=1}^N{w}_i\left(\mathit{x}\right)}$$

  (7)

2.4. MAS and Q-Learning

A system that consists of a set of agents that are able to each interact with the environment and with the other agents is called MAS [37]. The MAS are considered to be the basic field of the distributed artificial intelligence. A MAS is developed in order to solve a complex problem that a single agent cannot solve (or it is very difficult to solve), or to solve a problem which is distributed by its nature.

A MAS consists of a set of agents that act together in order to solve a problem. In problems which are distributed, the agents must cooperate together so as to maximize the total discounted reward [38]. This type of problem can be expressed as an extension of a Markov Decision Process (MDP) for a single agent [39]. Solving this MDP results in the extraction of the policy. According to Q-learning, a MAS can be expressed as a single agent with multiple states and an action vector, which can apply the conventional Q-learning algorithm [40]. Following this algorithm has the disadvantage that the state-action space of the agent grows exponentially according the number of the state variables and the action vector. In order to overcome this disadvantage, a variety of approaches have been developed, with the most well-known being coordinated reinforcement learning, distributed value function, and independent learners. The simplest approach of these three approaches is the independent learners because, according to this method, each agent acts in an individual manner. The agents act independently and each agent learns its own policy. By this way, an agent does not know either the Q-values or the policies of the other agents. Each agent stores and updates its own Q-table (Equation (8)) noting that each $Q_i$ is based on the global state $\mathit{x}$ [41].

$$Q_i\left(\mathit{x},a_i\right)\leftarrow Q_i\left(\mathit{x},a_i\right)+\eta \left[R_i\left(\mathit{x},\mathit{a},\mathit{x}{}^{\prime }\right)+\gamma {}_{{a^{\prime }}_i}{}^{ma\mathit{x}}Q{}_i\left({\mathit{x}}^{\prime },{a^{\prime }}_i\right)-Q_i\left({\mathit{x}}^{\prime },{a^{\prime }}_i\right)\right]$$

(8)

The main disadvantage of this method is that the environment is no longer stationary [42] and the convergence cannot be guaranteed. This happens because an agent ignores the presence of other agents. An agent can sense possible actions of other agents indirectly as changes of the environment occur. Despite this disadvantage, this method has been previously successfully applied in many control problems [40].

3. Desalination Model, Control Strategy and Implementation

3.1. Desalination Model

In this work an adaptive neuro fuzzy inference system (ANFIS) model of a reverse osmosis desalination plant was used. The model was data-driven and had one input and one output. The input was the DC voltage, from 0 to 32 V, and the output was the flow rate of the produced water, from 0 to 105 L/h. The data came from a pilot desalination unit installed in the Agricultural University of Athens. The desalination unit consisted of two 25- to 40-inch spiral-wound seawater Filmtec membrane modules. A rotary vane pump, of positive displacement, pressurized the water to one of the two cylinders of the Clark pump. The Clark pump was used as an energy recovery unit, instead of the high-pressure pumps that can be found in the conventional desalination units. The high-pressure brine was introduced into the second cylinder and exchanged the hydraulic energy with the feed water pressure of 13 bar. By this exchange, the pressure of the feed water was risen to 50 bar, which is the appropriate pressure of the membrane. More information about the desalination unit can be found on [43,44,45] and more information about the development of the ANFIS model can be found on [46].

3.2. Control Strategy

The control strategy of the desalination plant is depicted in Figure 3. The error arose from the subtraction of the set point (desire water flow rate) and the true value of the flow rate. The error signal was the input of the PID controller. The output of the PID controller was the reference voltage of the Pulse Width Modulation (PWM) generator. The PWM generator [47] produced pulses, which drove the switch of the converter. The converter received voltage in the input and, accordingly, the duty cycle of the pulses changed its output voltage. The output voltage of the converter was the input voltage of the desalination plant. The signal error and the derivative of error were the inputs of the MAS. The action of each agent adjusted each gain of the PID controller.

The converter of the system was a buck converter [48]. The converter changed the output voltage according the duty cycle of the pulses:

$$V_{out}=D·V_{in}$$

(9)

where $V_{out}$ is the output voltage of the converter, $D$ is the duty cycle of the pulses, and $V_{in}$ is the input voltage of the converter.

The MAS had two input signals, which were the error and the derivative of error. These signals defined the global state of the MAS. The MAS consisted of three agents, each one for one gain of the PID controller. There were the $A{G}_1$ agent, the $A{G}_2$ agent, and the $A{G}_3\mathrm{agent},$ which were dedicated to adjust the gains $Kp$, $Ki,$ and $Kd,$ respectively. The input signals were conditioning in the range of {−1, 1} and, through the fuzzy approximator, defined the current states of the MAS. The membership functions for each state variable (for both inputs) were seven (Figure 4). The quantization of the inputs in the seven areas, with 50% overlapping, provided enough details in order to cover the range of each input. In addition to that, the state space remained small (two inputs with seven MFs each results in a total of 49 states) and was easy to explore. These states were represented by an equal number of fuzzy rules and, consequently, the MAS could simultaneously be from one state to four (each state had each firing strength). The NB, NM, NS, Z, PS, PM, and PB denote Negative Big, Negative Medium, Negative Small, Zero, Positive Small, Positive Medium, and Positive Big, respectively.

The output of each agent defined the percentage change of each gain according to the initial values of the gains tuned by the Z-N method. Each agent’s output had a group of five fuzzy actions (fuzzy singletons). The set of the actions for all the agents was the same: $A_i=\left\{\frac{1}{-0.5}+\frac{1}{-0.25}+\frac{1}{0}+\frac{1}{0.25}+\frac{1}{0.5}\right\}$. where “+” is the operator of union and “–” represents a membership degree to a value of the membership function domain. The state-action space of each agent contained 245 combinations.

According to Q-learning, in order for each agent to acquire knowledge it interacts with the environment through an exploration/exploitation algorithm. This algorithm defines whether the agent performs exploration or exploitation. When the MAS becomes a new state, the agent $A{G}_1$ explores for a certain number of rounds per state (500 rounds/state), and then checks and applies the action that has not been applied at all (if there are any), while the other two agents are keeping each value constant (zero). After the MAS performs the first 500 rounds, the second agent $A{G}_2$ explores for 500 rounds/state, and then after 1000 rounds, the third agent $A{G}_3$ performs exploration for another 500 rounds/state. By this way the extensive exploration phase for each agent has little overlapping with the other two, and allows the algorithm to converge (the performing of all the agent’s extensive exploration simultaneously may lead to oscillations). After the extensive exploration phase, each agent performs exploitation for 99% and exploration for 1% for a given state [26].

The reward ($R$) is defined as follows:

$$R=\frac{1}{1+\left|e^{\prime }\right|}-\frac{1}{1+\left|e\right|}$$

(10)

where $e^{\prime }$ is the value of the error in the resulting state. The fraction $\frac{1}{1+\left|e\right|}$ takes values in the range {0, 1}. If the value of error is zero, the value of this quantity becomes “1”, otherwise, high positive or negative values of error reduce this quantity. The outcome of subtracting this quantity with the quantity that has the error in the succeeding state, gives a reward which can take positives and negative values. Positive values show that the error in the succeeding state is reduced while negative values show that the error in the succeeding state is raised. Greater values of reward indicate greater reduction of the error and vice versa [25].

3.3. Implementation

The whole system implemented in Matlab/Simulink (Mathworks, R2015a, Massachusetts, U.S.A) can be seen in Figure 5. There were three agents; each one had two inputs (error and derivative of error). The output of each agent adapted each gain of the PID controller. The output of the PID controller was the input of the PWM generator. The PWM generator had a frequency of $5\mathrm{kHz}$. The pulses produced by the PWM generator drove the switch of the buck converter. The converter had a capacitor of $0.001\mathrm{F}$ and an inductor of $0.001\mathrm{H}$. The input voltage of the converter was a $32V$ direct current voltage source. The output of the converter was the voltage fed into the desalination plant. The simulation time was set to $80\mathrm{s}$ with a simulation time step of $5·{10}^{-5}\mathrm{s}$. During the simulation, different values of the set point (flow rate of the produced water) were set. The set point was changing every 1 s. The range of the set point was from 0 to $105\mathrm{L}/\mathrm{h}$. The aim of the simulation was to compare a conventional PID controller tuned by Z-N with the proposed control strategy. The values of the gains of the PID controller which arose from the Z-N method were $\mathrm{kp}=0.01$, $\mathrm{ki}=0.025,$ and $\mathrm{kd}=0.000625$. The learning rate was set to $\mathsf{\eta }=0.1$ and the discount factor set to $\mathsf{\gamma }=0.9$. The time round for the MAS was set to 0.01 s.

4. Simulation Results

In Figure 6, the set point of the water flow rate, the water flow rate of the desalination unit controlled by the PID controller tuned by the Z-N, and the water flow rate of the proposed method are presented. Figure 7 presents the control signals (reference voltage of the PWM generator) of both methods. In the beginning of the simulation, where the MAS performed extensive exploration, we could see that the output of the system of the proposed method had a higher overshoot and a higher settling time compared with the conventional PID controller. On the contrary, during the exploitation phase, we could see that the proposed method performed better. The settling time was much less compared to the PID controller tuned by Z-N and with a small overshoot. These two points (Figure 6 and Figure 7) were at 2 s (exploration phase) and at 65 s (exploitation phase).

The improvement of the performance of the proposed method was highlighted in Figure 8, where the values of the Integral Absolute Error (IAE), the Integral Time Absolute Error (ITAE), and the Integral Square Error (ISE) through time are presented. The IAE, the ITAE, and the ISE are defined as below:

$$IAE={\displaystyle \int }\left|e\right|dt$$

(11)

$$ITAE={\displaystyle \int }t\left|e\right|dt$$

(12)

$$ISE={\displaystyle \int }{e}^{2}dt$$

(13)

We can see that the values of the IAE, ITAE, and ISE were the same for both methods in the beginning (exploration phase). As the MAS turned from the exploration phase to exploitation phase, the values of the IAE, ITAE, and ISE turned so that they were lower for the proposed method than the conventional PID controller.

Figure 9 presents the percentage change of the three gains of the controller (actions of the three agents) of the proposed control method through time, in respect to the various changes of the set point.

The proposed approach was tested under another two different scenarios with different random set points. In Figure 10 the set point of the water flow rate, the water flow rate of the desalination unit controlled by the PID controller tuned by the Z-N, and the water flow rate of the proposed method for the first scenario are presented while Figure 11 presents the values of the IAE, ITAE, and ISE through time for the same scenario. Figure 12 and Figure 13 present the same quantities with Figure 10 and Figure 11 respectively for the second scenario. We can see that the values of the IAE, ITAE, and ISE were the same for both methods in the beginning (exploration phase) for both scenarios. As the MAS turned from the exploration phase to the exploitation phase, the values of the IAE, ITAE, and ISE turned so that they were lower for the proposed method, than the conventional PID controller, for both scenarios. The performance of the proposed method for these scenarios highlights its superiority compared to the Z-N method and confirms its effectiveness and robustness.

5. Discussion

This paper presents a fuzzy Q-learning MAS for adjusting online the three gains of a PID controller. The controller was used to control the flow rate of a desalination unit. The gains of the controller were initialized by the Z-N method. The input variables of the MAS were the error and the derivative of error. The output of each agent was the percentage change of the gain that each agent was dedicated to adapt. Fuzzy logic was used in combination with Q-learning in each agent in order to confront the continuous state-action space. The simulation results indicated the better performance of the proposed method and especially the lower values of the IAE, ITAE, and ISE. Using a single agent with multiple input variables and an action vector resulted in the exponential growth of the state-action space. By decomposing the problem in many simple problems using the independent learners’ approach, the state-action space was reduced. On the other hand, the drawback of this approach was that it could not guarantee convergence. The agents were acting and updating their tables without taking into account the actions of the other agents, In future, our aim is to focus on the implementation of other reinforcement learning approaches for MAS in order to overcome this limitation. Additionally, the combination between online learning algorithms and offline optimization methods for tuning the membership function of both the inputs and the outputs of the agents can be tested in order to improve the overall performance of the system.