Improvement of Refrigeration Efficiency by Combining Reinforcement Learning with a Coarse Model

Abstract

It is paramount to improve operational conversion efficiency in air-conditioning refrigeration. It is noticed that control efficiency for model-based methods highly relies on the accuracy of the mechanism model, and data-driven methods would face challenges using the limited collected data to identify the information beyond. In this study, a hybrid novel approach is presented, which is to integrate a data-driven method with a coarse model. Specifically, reinforcement learning is used to exploit/explore the conversion efficiency of the refrigeration, and a coarse model is utilized to evaluate the reward, by which the requirement of the model accuracy is reduced and the model information is better used. The proposed approach is implemented based on a hierarchical control strategy which is divided into a process level and a loop level. The simulation of a test bed shows the proposed approach can achieve better conversion efficiency of refrigeration than the conventional methods.

1. Background and Motivation

Along with the comfortable environment, the air-conditioning brings a problem of energy consumption. According to the U.S. Energy Information Administration, the energy consumption of buildings for residential and commercial users accounts for 20.1% of the global energy consumption worldwide. Of this amount, the buildings’ heating, ventilation, and air conditioning (HVAC) systems can account for up to 50% of total building energy demand [1,2]. To minimize overall system energy input or operating cost while still maintaining the satisfying indoor thermal comfort and healthy environment all kinds of control algorithms are employed on HVAC. In [3], model-based monitoring of the occupants’ thermal state by predictive controlling was proposed. In [4], the supervisory and optimal controls were summarized and classified into the model-based methods which used physical models, gray-box models, and black-box models, and the model-free category which used expert systems and pure learning approaches. In [5] the existed methods were grouped into the hard control (HC), such as basic controls involving PID control, optimal control, nonlinear control, robust or H∞ control, and adaptive control, and the soft control (SC) involving neural networks (NNs), fuzzy logic (FL), genetic algorithms (GAs), and other evolutionary methods. Moreover, hybrid control resulting from the fusion of SC and HC may achieve better performance.

Most of today’s air-conditioning systems work based on the principle of compression refrigeration in which refrigeration is the core of the air-conditioning system [6]. The theoretical optimal conversion efficiency is used in the design of air-conditioning based on the specified loads, according to refrigeration theory. However, it is inevitable in practice for performance degradation due to the inconsistencies between design requirements and operational conditions. For example, a public place will provide a little refrigerating output to respond to small load at night, meanwhile it will need a large number of refrigerating output for the varying flow of people. The conventional control algorithm will result in the low operational conversion efficiency in this time-varying and uncertain condition because as we know the efficiency of compression refrigeration is confined to the working point and affected by the loads.

With the development of modern electronic and measurement technologies, such as supervisory control and data acquisition (SCADA), and smart sensors, the refrigeration system is prone to obtain abundant data which provide the information about consistency in the current refrigeration states and the operational environment. It is promising for the data-driven methods to improve the conversion efficiency by directly using these data. The reinforcement learning (RL) is a famous data-driven method which is regarded as a model-free supervisory control method in [5] and a soft control (SC) in [6], is about learning from interaction and how to behave in order to achieve a goal. The basic idea of reinforcement learning is simply to capture the most important aspects of an agent, which includes the sensation, the action, and the goal [7]. A reward rule is then employed to encourage the agent to seek the goal by adjusting its action with exploration and exploitation. Finally, the agent owns the optimal action to adapt to the surroundings by trials and errors. In this way, the RL can achieve optimal action based on the current states and environment. Along with incredible success in games [8], the RL has attracted great interest in various industries, such as robot [9,10], fault detection [11], and fault-tolerant control [12,13].

The RL has been introduced to HVA and some excellent results have been published recently. Baeksuk Chu et al. proposed an actor-critic architecture and natural gradient method with an improvement of the efficiency by the recursive least-squares (RLS) method aiming at two objectives: Maintaining an adequate level of pollutants and minimizing power consumption [14]. Pedro Fazenda et al. followed the idea of a multi-objective optimal supervisory control and studied the application of a discrete and continuous reinforcement-learning-based supervisory control approach, which actively learned how to appropriately schedule thermostat temperature set-points [15]. In [16], a multi-grid method of Q-learning was addressed to handle the problem that online reinforcement learning often suffered from slow convergence and faults in early stages. In [17], a suitable learning architecture was proposed for an intelligent thermostat that comprised a number of different learning methods, each of which contributed to creating a complete autonomous thermostat capable of controlling an HVAC system and proposed a novel state action space formalism to enable RL successfully. In [18], a ventilated facade with PCM was controlled using a reinforcement learning algorithm. In [19], a reinforcement learning control (RLC) approach was presented for optimal control of low exergy buildings. An improved reinforcement learning controller was designed in [20] to obtain an optimal control strategy of blinds and lights, which provided a personalized service via introducing subject perceptions of surroundings gathered by a novel interface as the feedback signal. In [21], a model-free actor-critic reinforcement learning (RL) controller was addressed using a variant of artificial recurrent neural networks called long-short-term memory (LSTM) networks. A model-free Q-learning was addressed in [22] that made optimal control decisions for HVAC and window systems to minimize both energy consumption and thermal discomfort. Recently, the latest methods on RL, for example, deep reinforcement learning control [23,24], regularized fitted Q-iteration approach [25], proximal actor critic [26], and auto-encoder [27,28] were appeared in air-conditioning to minimize both energy consumption and thermal discomfort. However, all the studies do not involve the time-varying and uncertain loads.

The RL is classified as online learning and offline learning, according to the resource of training data [29]. The online learning will make sense in the refrigeration system for the reason of overcoming the loads time-varying and uncertainty. There are two challenges: (1) The new states should be obtained only after the actions have acted on the system. It is a risk for the system because no one knows the results of these actions. (2) The conventional RL uses an errors and trials method to train, which is a forced way to the unknown environment but with low efficiency due to little information between states and actions. In this study, a coarse model is proposed to help solve both problems. We consider a coarse model based on a fact that the human has grasped the principles of refrigeration conversion. It is possible to build a coarse model to discover the relations of state variables according to the principles based on the figures and charts through a large number of experiences. On the one hand, it will bring more information between states and actions to speed up the learning process. On the other hand, it will forecast the effects of actions as a simulation. We give up the accurate model for the reasons of the system complexity and surrounding disturbances. In addition, the RL only needs the reward as a basis of action adjustment in the process of seeking for the goal. The reward can be obtained from the tendency of performance indictor which reduces to the requirement of model accuracy. As a result, it is enough for a coarse model to be used for RL. Based on a coarse model, the computer simulation can be approximated to estimate the next states instead of the real system which reduces the risk of unknown states after a trial action.

Motivated by the above discussions, a novel approach combines the reinforcement learning with a coarse model in which we take the conversation efficiency as a goal of reinforcement learning, the measurement data of refrigeration as the sensation of agent, and the control valve of refrigeration as the action of agent, and learn by Q-algorithm, is proposed to improve the conversion efficiency of refrigeration and the advantages are summarized as follows:

• This is an online control strategy with data-driven control and learning simultaneously which reduces the requirement of model accuracy.
• It has the ability to get the optimal action by exploration and exploitation to achieve a better conversion efficiency which fits the current operating conditions.
• The risk on the system in the learning process, which is complex and unknown in advance is prevented mostly by a hierarchical control of the process level, which makes use of the existing knowledge of refrigeration and the loop level, which implements the tracking control.

The remaining parts of this paper are organized as follows: The aim of the study and the preliminaries on a compression refrigerating system are introduced in Section 2. The fundamentals of reinforcement learning algorithms are reviewed in Section 3. Based on Section 2 and Section 3, a reinforcement learning control algorithm for compression refrigerating systems is proposed in Section 4, and the case studies are addressed in Section 5. The paper is ended by conclusions in Section 6.

2. Preliminaries

2.1. Aim of the Research

For a compression refrigerating system, the compressor drives refrigerants to a circulatory flow to transfer the heating of cryogenic medium to the high-temperature medium via a phase change of refrigerant. In this process, the temperature is the key indicator of reflecting efficiency which is a comprehensive result of the interaction of mass, pressure, process, and loads, so the coefficient of performance (COP) at time $\mathrm{k}$ is defined as

$$\mathrm{COP}\left(\mathrm{k}\right)=\frac{{\mathrm{T}}_{\mathrm{c}}\left(\mathrm{k}\right)}{{\mathrm{T}}_{\mathrm{c}}\left(\mathrm{k}\right)-{\mathrm{T}}_{\mathrm{e}}\left(\mathrm{k}\right)}$$

(1)

where ${\mathrm{T}}_{\mathrm{c}}$ and ${\mathrm{T}}_{\mathrm{e}}$ are the evaporation temperature and the condensation temperature at sampling k, respectively. Our aim is to maximize the coefficient of performance of the refrigeration cycle during a period of time $l$

$$\mathrm{COP}=\max \left\{{\displaystyle \sum }_{\mathrm{k}=1}^l\frac{{\mathrm{T}}_{\mathrm{c}}\left(\mathrm{k}\right)}{{\mathrm{T}}_{\mathrm{c}}\left(\mathrm{k}\right)-{\mathrm{T}}_{\mathrm{e}}\left(\mathrm{k}\right)}\right\}$$

(2)

2.2. Principle of Compression Refrigerating System

A compression refrigerating system is always made up of four elements: The compressor, the evaporator, the condenser, and the throttle mechanic. The compressor provides power for the circulation of the refrigerant. The evaporator is employed to implement the heating change process between the refrigeration and the chilled water. The condenser is used to complete the heating change process between the refrigeration and the cooling water or air cooling system. The throttle mechanic (mainly electronic expansion valve) is applied to adjust the mass flow rate of the refrigerant in the circulation. Here, the principle of each element is reviewed for the integrity according to [30,31,32,33] and more details can be found in [30,31,32,33] and the references therein.

2.2.1. Principle of the Evaporator

The scheme of the evaporator process is dictated in Figure 1.

The inner region of the evaporator is divided into a two-phase zone where the refrigerant lives in the combination of liquid and gas and a superheated zone where the liquid refrigerant evaporates into a gas with absorbing the heating and finally becomes the superheated steam. The refrigerant is moving from the two-phase zone to the superheated zone. An average void fraction correlation is used to express the proportion of gas in the two-phase zone whose definition is an integrating of void-age along the pipeline direction

$$\overline{\mathsf{\gamma }}=\frac{1}{{\mathrm{x}}_2-{\mathrm{x}}_1}{{\displaystyle \int }}_{{\mathrm{x}}_1}^{{\mathrm{x}}_2}\mathsf{\gamma }\left(\mathrm{x}\right)\mathrm{d}\left(\mathrm{x}\right)$$

(3)

where $\mathsf{\gamma }$ is a void fraction correlation with a form of formula (4)

$$\mathsf{\gamma }=\frac{1}{1+(\frac{1-\mathrm{x}}{\mathrm{x}})(\frac{{\mathsf{\rho }}_{\mathrm{g}}}{{\mathsf{\rho }}_{\mathrm{l}}})\mathrm{S}}$$

(4)

in which $\mathrm{x}$ is the vapor quality, ${\mathsf{\rho }}_{\mathrm{g}}$ and ${\mathsf{\rho }}_{\mathrm{l}}$ are the density of saturated steam and the density of saturated liquid, respectively, $\mathrm{kg}/{\mathrm{m}}^3$, $\mathrm{S}$ is the slip ratio.

According to the law of mass conservation, the law of momentum conservation and the law of energy conservation, one will obtain the state equation of evaporator as formula (5)

$$\mathrm{E}\left({\mathrm{x}}_{\mathrm{e}}\right){\dot{\mathrm{x}}}_{\mathrm{e}}=\mathrm{f}\left({\mathrm{x}}_{\mathrm{e}},{\mathrm{u}}_{\mathrm{e}}\right)$$

(5)

$$\left[\begin{array}{ccccc}{\mathrm{E}}_{11}& {\mathrm{E}}_{12}& 0& 0& 0\\ {\mathrm{E}}_{21}& {\mathrm{E}}_{22}& {\mathrm{E}}_{23}& 0& 0\\ {\mathrm{E}}_{31}& {\mathrm{E}}_{32}& {\mathrm{E}}_{33}& 0& 0\\ 0& 0& 0& {\mathrm{E}}_{44}& 0\\ {\mathrm{E}}_{51}& 0& 0& 0& {\mathrm{E}}_{55}\end{array}\right]\left[\begin{array}{c}{\dot{\mathrm{L}}}_{\mathrm{e}1}\\ {\dot{\mathrm{P}}}_{\mathrm{e}}\\ \begin{array}{c}{\dot{\mathrm{h}}}_{\mathrm{eo}}\\ {\dot{\mathrm{T}}}_{\mathrm{ew}1}\\ {\dot{\mathrm{T}}}_{\mathrm{ew}2}\end{array}\end{array}\right]=\left[\begin{array}{c}{\dot{\mathrm{m}}}_{\mathrm{ei}}({\mathrm{h}}_{\mathrm{ei}} - {\mathrm{h}}_{\mathrm{g}})+{\mathsf{\alpha }}_{\mathrm{ei}1}{\mathrm{A}}_{\mathrm{ei}}(\frac{{\mathrm{L}}_{\mathrm{ei}1}}{{\mathrm{L}}_{\mathrm{etotal}}})({\mathrm{T}}_{\mathrm{ew}1} - {\mathrm{T}}_{\mathrm{er}1})\\ {\dot{\mathrm{m}}}_{\mathrm{eo}}({\mathrm{h}}_{\mathrm{g}} - {\mathrm{h}}_{\mathrm{eo}})+{\mathsf{\alpha }}_{\mathrm{ei}2}{\mathrm{A}}_{\mathrm{ei}}(\frac{{\mathrm{L}}_{\mathrm{ei}2}}{{\mathrm{L}}_{\mathrm{etotal}}})({\mathrm{T}}_{\mathrm{ew}2} - {\mathrm{T}}_{\mathrm{er}2})\\ {\dot{\mathrm{m}}}_{\mathrm{ei}} - {\dot{\mathrm{m}}}_{\mathrm{eo}}\\ {\mathsf{\alpha }}_{\mathrm{ei}1}{\mathrm{A}}_{\mathrm{ei}}({\mathrm{T}}_{\mathrm{er}1} - {\mathrm{T}}_{\mathrm{ew}1})+{\mathsf{\alpha }}_{\mathrm{eo}}{\mathrm{A}}_{\mathrm{eo}}({\mathrm{T}}_{\mathrm{chw}} - {\mathrm{T}}_{\mathrm{ew}1})\\ {\mathsf{\alpha }}_{\mathrm{ei}2}{\mathrm{A}}_{\mathrm{ei}}({\mathrm{T}}_{\mathrm{er}2} - {\mathrm{T}}_{\mathrm{ew}2})+{\mathsf{\alpha }}_{\mathrm{eo}}{\mathrm{A}}_{\mathrm{eo}}({\mathrm{T}}_{\mathrm{chw}} - {\mathrm{T}}_{\mathrm{ew}2})\end{array}\right]$$

$${\mathrm{E}}_{11}={\mathsf{\rho }}_1\left({\mathrm{h}}_1-{\mathrm{h}}_{\mathrm{g}}\right)\left(1-\overline{\mathsf{\gamma }}\right){\mathrm{A}}_{\mathrm{ecs}}$$

where

$${\mathrm{E}}_{12}=\left[\left(\frac{{\mathrm{d}\mathsf{\rho }}_1{\mathrm{h}}_1}{{\mathrm{dP}}_{\mathrm{e}}}-\frac{{\mathrm{d}\mathsf{\rho }}_1}{{\mathrm{dP}}_{\mathrm{e}}}{\mathrm{h}}_{\mathrm{g}}\right)\left(1-\overline{\mathsf{\gamma }}\right)+\left(\frac{{\mathrm{d}\mathsf{\rho }}_{\mathrm{g}}{\mathrm{h}}_{\mathrm{g}}}{{\mathrm{dP}}_{\mathrm{e}}}-\frac{{\mathrm{d}\mathsf{\rho }}_{\mathrm{g}}}{{\mathrm{dP}}_{\mathrm{e}}}{\mathrm{h}}_{\mathrm{g}}\right)\left(\overline{\mathsf{\gamma }}-1\right)\right]{\mathrm{A}}_{\mathrm{ecs}}{\mathrm{L}}_{\mathrm{e}1}$$

$${\mathrm{E}}_{21}={\mathsf{\rho }}_{\mathrm{e}2}\left({\mathrm{h}}_{\mathrm{g}}-{\mathrm{h}}_{\mathrm{e}2}\right){\mathrm{A}}_{\mathrm{ecs}}$$

$${\mathrm{E}}_{22}=\left[\left(\left({\frac{\partial {\mathsf{\rho }}_{\mathrm{e}2}}{\partial {\mathrm{P}}_{\mathrm{e}}}|}_{{\mathrm{h}}_{\mathrm{e}2}}\right)-\frac{1}{2}\left({\frac{\partial {\mathsf{\rho }}_{\mathrm{e}2}}{\partial {\mathrm{h}}_{\mathrm{e}2}}|}_{{\mathrm{p}}_{\mathrm{e}}}\right)\left(\frac{{\mathrm{dh}}_{\mathrm{g}}}{{\mathrm{dP}}_{\mathrm{e}}}\right)\right)\left({\mathrm{h}}_{\mathrm{e}2}-{\mathrm{h}}_{\mathrm{g}}\right)+\frac{1}{2}\left(\frac{{\mathrm{dh}}_{\mathrm{g}}}{{\mathrm{dP}}_{\mathrm{e}}}\right){\mathsf{\rho }}_{\mathrm{e}2}-1\right]A_{\mathrm{ecs}}{\mathrm{L}}_{\mathrm{e}2}$$

$${\mathrm{E}}_{23}=\left[\frac{1}{2}\left({\frac{\partial {\mathsf{\rho }}_{\mathrm{e}2}}{\partial {\mathrm{h}}_{\mathrm{e}2}}|}_{{\mathrm{p}}_{\mathrm{e}}}\right)\left({\mathrm{h}}_{\mathrm{e}2}-{\mathrm{h}}_{\mathrm{g}}\right)+\frac{{\mathsf{\rho }}_{\mathrm{e}2}}{2}\right]{\mathrm{A}}_{\mathrm{ecs}}{\mathrm{L}}_{\mathrm{e}2}$$

$${\mathrm{E}}_{31}=\left[\left({\mathsf{\rho }}_{\mathrm{g}}-{\mathsf{\rho }}_{\mathrm{e}2}\right)+\left({\mathsf{\rho }}_1-{\mathsf{\rho }}_{\mathrm{g}}\right)\left(1-\overline{\mathsf{\gamma }}\right)\right]{\mathrm{A}}_{\mathrm{ecs}}$$

$${\mathrm{E}}_{32}=\left[\left(\left({\frac{\partial {\mathsf{\rho }}_{\mathrm{e}2}}{\partial {\mathrm{P}}_{\mathrm{e}}}|}_{{\mathrm{h}}_{\mathrm{e}2}}\right)+\frac{1}{2}\left({\frac{\partial {\mathsf{\rho }}_{\mathrm{e}2}}{\partial {\mathrm{h}}_{\mathrm{e}2}}|}_{{\mathrm{p}}_{\mathrm{e}}}\right)\left(\frac{{\mathrm{dh}}_{\mathrm{g}}}{{\mathrm{dP}}_{\mathrm{e}}}\right)\right){\mathrm{L}}_{\mathrm{e}2}+\left(\frac{{\mathrm{d}\mathsf{\rho }}_1}{{\mathrm{dP}}_{\mathrm{e}}}\left(1-\overline{\mathsf{\gamma }}\right)+\left(\frac{{\mathrm{d}\mathsf{\rho }}_{\mathrm{g}}}{{\mathrm{dP}}_{\mathrm{e}}}\right)\overline{\mathsf{\gamma }}\right){\mathrm{L}}_{\mathrm{e}1}\right]{\mathrm{A}}_{\mathrm{ecs}}$$

$${\mathrm{E}}_{33}=\frac{1}{2}\left({\frac{\partial {\mathsf{\rho }}_{\mathrm{e}2}}{\partial {\mathrm{h}}_{\mathrm{e}2}}|}_{{\mathrm{p}}_{\mathrm{e}}}\right){\mathrm{A}}_{\mathrm{ecs}}{\mathrm{L}}_{\mathrm{e}2},{\mathrm{E}}_{34}={\left({\mathrm{C}}_{\mathrm{p}}\mathsf{\rho }\mathrm{V}\right)}_{\mathrm{ew}},$$

$${\mathrm{E}}_{51}=-{\left({\mathrm{C}}_{\mathrm{p}}\mathsf{\rho }\mathrm{V}\right)}_{\mathrm{ew}}\frac{{\mathrm{T}}_{\mathrm{ew}2}-{\mathrm{T}}_{\mathrm{ew}1}}{{\mathrm{L}}_{\mathrm{e}2}},{\mathrm{E}}_{55}={\left({\mathrm{C}}_{\mathrm{p}}\mathsf{\rho }\mathrm{V}\right)}_{\mathrm{ew}}$$

The meanings of states and parameters are seen in

Table 1. Description of model symbols.

Letter	Name	Subscript	Name	Subscript	Name
T	Temperature, °C	k	Compressor	ei2	Inner of superheated zone of evaporator
M	Mass, Kg	c	Condenser	er2	Refrigerant in superheated zone of evaporator
ρ	Density, Kg/m3	e	evaporator	ew2	Tube wall of superheated zone of evaporator
N	Power, W	val	Expansion valve	ei1	Inner of two-phase zone of evaporator
η	Efficiency	r	refrigerant	er1	Refrigerant in two-phase zone of evaporator
K	Heat transfer coefficient, W/m2·°C	w	Tube wall, water	ew1	Tube wall of two-phase zone of evaporator
α	Heat transfer coefficient, W/m2·°C	i	Input/inner	ci2	Inner of superheated zone of condenser
μ	Dynamic viscosity, Pa·s	o	Output/outer	cr2	Refrigerant in superheated zone of condenser
A	Area, m2	sc	supercool	cw2	Tube wall of superheated zone of condenser
m	Mass flow, Kg·s	sh	superhot	ci1	Inner of two-phase zone of condenser
C	Specific heat capacity, KJ/Kg·°C	th	Theoretical value	cr1	Refrigerant in two-phase zone of condenser
ν	Specific volume, m3/Kg	g	Gas	cw1	Tube wall of two-phase zone of condenser
h	Specific enthalphy, KJ/Kg	l	Liquid	ci3	Tube wall of supercool zone of condenser

.

2.2.2. Principle of the Condenser

The condenser inner region is divided into a two-phase zone, a superheated zone, and a supercool zone. For each zone, the partial differential equations have been built according to the law of conservation of mass, the law of conservation of momentum and the law of conservation of energy based on the properties of refrigerants and the experience equations. Therefore, the state equation of the condenser, obtained by integrating along the pipeline direction, is given as follows:

$$\mathrm{C}\left({\mathrm{x}}_{\mathrm{c}}\right){\dot{\mathrm{x}}}_{\mathrm{c}}=\mathrm{f}\left({\mathrm{x}}_{\mathrm{c}},{\mathrm{u}}_{\mathrm{c}}\right)$$

(6)

$$\left[\begin{array}{ccccccc}{\mathrm{C}}_{11}& 0& {\mathrm{C}}_{13}& 0& 0& 0& 0\\ {\mathrm{C}}_{21}& {\mathrm{C}}_{22}& {\mathrm{C}}_{23}& {\mathrm{C}}_{24}& 0& 0& 0\\ {\mathrm{C}}_{31}& {\mathrm{C}}_{32}& {\mathrm{C}}_{33}& {\mathrm{C}}_{34}& 0& 0& 0\\ {\mathrm{C}}_{41}& {\mathrm{C}}_{42}& {\mathrm{C}}_{43}& {\mathrm{C}}_{44}& 0& 0& 0\\ {\mathrm{C}}_{51}& 0& 0& 0& {\mathrm{C}}_{55}& 0& 0\\ 0& 0& 0& 0& 0& {\mathrm{C}}_{66}& 0\\ {\mathrm{C}}_{71}& 0& 0& 0& 0& 0& {\mathrm{C}}_{77}\end{array}\right]\left[\begin{array}{c}{\dot{\mathrm{L}}}_{\mathrm{c}1}\\ {\dot{\mathrm{L}}}_{\mathrm{c}2}\\ \begin{array}{c}{\dot{\mathrm{P}}}_{\mathrm{c}}\\ {\dot{\mathrm{h}}}_{\mathrm{co}}\\ \begin{array}{c}{\dot{\mathrm{T}}}_{\mathrm{cw}1}\\ {\dot{\mathrm{T}}}_{\mathrm{cw}2}\\ {\dot{\mathrm{T}}}_{\mathrm{cw}3}\end{array}\end{array}\end{array}\right]=\left[\begin{array}{c}{\dot{\mathrm{m}}}_{\mathrm{ci}}\left({\mathrm{h}}_{\mathrm{ci}}-{\mathrm{h}}_{\mathrm{g}}\right)+{\mathsf{\alpha }}_{\mathrm{ci}1}{\mathrm{A}}_{\mathrm{ci}}\left(\frac{{\mathrm{L}}_{\mathrm{c}1}}{{\mathrm{L}}_{\mathrm{ctotal}}}\right)\left({\mathrm{T}}_{\mathrm{cw}1}-{\mathrm{T}}_{\mathrm{cr}1}\right)\\ {\dot{\mathrm{m}}}_{\mathrm{ci}}{\mathrm{h}}_{\mathrm{g}}-{\dot{\mathrm{m}}}_{\mathrm{co}}{\mathrm{h}}_{\mathrm{l}}+{\mathsf{\alpha }}_{\mathrm{ci}2}{\mathrm{A}}_{\mathrm{ci}}\left(\frac{{\mathrm{L}}_{\mathrm{c}2}}{{\mathrm{L}}_{\mathrm{ctotal}}}\right)\left({\mathrm{T}}_{\mathrm{cw}2}-{\mathrm{T}}_{\mathrm{cr}2}\right)\\ {\dot{\mathrm{m}}}_{\mathrm{co}}\left({\mathrm{h}}_{\mathrm{l}}-{\mathrm{h}}_{\mathrm{co}}\right)+{\mathsf{\alpha }}_{\mathrm{ci}3}{\mathrm{A}}_{\mathrm{ci}}\left(\frac{{\mathrm{L}}_{\mathrm{c}3}}{{\mathrm{L}}_{\mathrm{ctotal}}}\right)\left({\mathrm{T}}_{\mathrm{cw}3}-{\mathrm{T}}_{\mathrm{cr}3}\right)\\ {\dot{\mathrm{m}}}_{\mathrm{ci}}-{\dot{\mathrm{m}}}_{\mathrm{co}}\\ {\mathsf{\alpha }}_{\mathrm{ci}1}{\mathrm{A}}_{\mathrm{ci}}\left({\mathrm{T}}_{\mathrm{cr}1}-{\mathrm{T}}_{\mathrm{cw}1}\right)+{\mathsf{\alpha }}_{\mathrm{co}}{\mathrm{A}}_{\mathrm{co}}\left({\mathrm{T}}_{\mathrm{a}}-{\mathrm{T}}_{\mathrm{cw}1}\right)\\ {\mathsf{\alpha }}_{\mathrm{ci}2}{\mathrm{A}}_{\mathrm{ci}}\left({\mathrm{T}}_{\mathrm{cr}2}-{\mathrm{T}}_{\mathrm{cw}2}\right)+{\mathsf{\alpha }}_{\mathrm{co}}{\mathrm{A}}_{\mathrm{co}}\left({\mathrm{T}}_{\mathrm{a}}-{\mathrm{T}}_{\mathrm{cw}2}\right)\\ {\mathsf{\alpha }}_{\mathrm{ci}3}{\mathrm{A}}_{\mathrm{ci}}\left({\mathrm{T}}_{\mathrm{cr}3}-{\mathrm{T}}_{\mathrm{cw}3}\right)+{\mathsf{\alpha }}_{\mathrm{co}}{\mathrm{A}}_{\mathrm{co}}\left({\mathrm{T}}_{\mathrm{a}}-{\mathrm{T}}_{\mathrm{cw}3}\right)\end{array}\right]$$

where

$${\mathrm{C}}_{11}={\mathsf{\rho }}_{\mathrm{c}1}\left({\mathrm{h}}_{\mathrm{c}1}-{\mathrm{h}}_{\mathrm{g}}\right){\mathrm{A}}_{\mathrm{ecs}}$$

$${\mathrm{C}}_{13}=\left[\left({\left.\frac{\partial {\mathsf{\rho }}_{\mathrm{e}1}}{\partial {\mathrm{P}}_{\mathrm{c}}}\right|}_{{\mathrm{h}}_{\mathrm{c}1}}\right){\mathrm{h}}_{\mathrm{c}1}+\frac{1}{2}\frac{{\mathrm{dh}}_{\mathrm{g}}}{{\mathrm{dP}}_{\mathrm{c}}}\left(\left({\left.\frac{\partial {\mathsf{\rho }}_{\mathrm{c}1}}{\partial {\mathrm{P}}_{\mathrm{c}1}}\right|}_{{\mathrm{P}}_{\mathrm{c}}}\right)\left({\mathrm{h}}_{\mathrm{c}1}-{\mathrm{h}}_{\mathrm{g}}\right)+{\mathsf{\rho }}_{\mathrm{c}1}\right)-1\right]{\mathrm{A}}_{\mathrm{ccs}}{\mathrm{L}}_{\mathrm{c}1}$$

$${\mathrm{C}}_{21}=\left({\mathsf{\rho }}_{\mathrm{c}1}{\mathrm{h}}_{\mathrm{g}}-{\mathsf{\rho }}_{\mathrm{c}3}{\mathrm{h}}_1\right){\mathrm{A}}_{\mathrm{ecs}},{\mathrm{C}}_{22}=\left[\left({\mathsf{\rho }}_{\mathrm{g}}{\mathrm{h}}_{\mathrm{g}}-{\mathsf{\rho }}_1{\mathrm{h}}_1\right)\overline{\mathsf{\gamma }}+\left({\mathsf{\rho }}_1-{\mathsf{\rho }}_{\mathrm{c}3}\right){\mathrm{h}}_1\right]{\mathrm{A}}_{\mathrm{ccs}}$$

$$\begin{array}{ll}{\mathrm{C}}_{23}=[\left({\frac{\partial {\mathsf{\rho }}_{\mathrm{c}1}}{\partial {\mathrm{P}}_{\mathrm{c}}}|}_{{\mathrm{h}}_{\mathrm{c}1}}\right)& +\frac{1}{2}\left({\frac{\partial {\mathsf{\rho }}_{\mathrm{c}1}}{\partial {\mathrm{P}}_{\mathrm{c}1}}|}_{{\mathrm{P}}_{\mathrm{c}}}\right)\frac{{\mathrm{dh}}_{\mathrm{g}}}{{\mathrm{dP}}_{\mathrm{c}}}]{\mathrm{h}}_{\mathrm{g}}{\mathrm{A}}_{\mathrm{ccs}}{\mathrm{L}}_{\mathrm{c}1}\\ & + \left(\frac{{\mathrm{d}\mathsf{\rho }}_1{\mathrm{h}}_1}{{\mathrm{dP}}_{\mathrm{c}}}\left(1-\overline{\mathsf{\gamma }}\right)+\frac{{\mathrm{d}\mathsf{\rho }}_{\mathrm{g}}{\mathrm{h}}_{\mathrm{g}}}{{\mathrm{dP}}_{\mathrm{e}}}\left(\overline{\mathsf{\gamma }}-1\right)\right){\mathrm{A}}_{\mathrm{ccs}}{\mathrm{L}}_{\mathrm{c}2} \\ & + \left[\left({\frac{\partial {\mathsf{\rho }}_{\mathrm{c}3}}{\partial {\mathrm{P}}_{\mathrm{c}}}|}_{{\mathrm{h}}_{\mathrm{c}3}}\right)+\frac{1}{2}\left({\frac{\partial {\mathsf{\rho }}_{\mathrm{c}3}}{\partial {\mathrm{P}}_{\mathrm{c}3}}|}_{{\mathrm{P}}_{\mathrm{c}}}\right)\frac{{\mathrm{dh}}_1}{{\mathrm{dP}}_{\mathrm{c}}}\right]{\mathrm{h}}_1{\mathrm{A}}_{\mathrm{ccs}}{\mathrm{L}}_{\mathrm{c}3}\end{array}$$

$${\mathrm{C}}_{24}=\frac{1}{2}\left({\frac{\partial {\mathsf{\rho }}_{\mathrm{c}3}}{\partial {\mathrm{h}}_{\mathrm{c}3}}|}_{{\mathrm{P}}_{\mathrm{c}}}\right){\mathrm{h}}_1{\mathrm{A}}_{\mathrm{ccs}}{\mathrm{L}}_{\mathrm{c}3},{\mathrm{C}}_{31}={\mathsf{\rho }}_{\mathrm{c}3}\left({\mathrm{h}}_1-{\mathrm{h}}_{\mathrm{c}3}\right){\mathrm{A}}_{\mathrm{ccs}},{\mathrm{C}}_{32}={\mathsf{\rho }}_{\mathrm{c}3}\left({\mathrm{h}}_1-{\mathrm{h}}_{\mathrm{c}3}\right){\mathrm{A}}_{\mathrm{ccs}}$$

$${\mathrm{C}}_{33}=\left[\left(\left({\frac{\partial {\mathsf{\rho }}_{\mathrm{c}3}}{\partial {\mathrm{P}}_{\mathrm{c}}}|}_{{\mathrm{h}}_{\mathrm{c}3}}\right)+\frac{1}{2}\frac{{\mathrm{dh}}_1}{{\mathrm{dP}}_{\mathrm{c}}}\left({\frac{\partial {\mathsf{\rho }}_{\mathrm{c}3}}{\partial {\mathrm{P}}_{\mathrm{c}3}}|}_{{\mathrm{P}}_{\mathrm{c}}}\right)\right)\left({\mathrm{h}}_{\mathrm{c}3}-{\mathrm{h}}_1\right)+\frac{1}{2}\frac{{\mathrm{dh}}_1}{{\mathrm{dP}}_{\mathrm{c}}}{\mathsf{\rho }}_{\mathrm{c}3}-1\right]{\mathrm{A}}_{\mathrm{ccs}}{\mathrm{L}}_{\mathrm{c}3}$$

$${\mathrm{C}}_{34}=\left[\frac{1}{2}\left({\frac{\partial {\mathsf{\rho }}_{\mathrm{c}3}}{\partial {\mathrm{h}}_{\mathrm{c}3}}|}_{{\mathrm{P}}_{\mathrm{c}}}\right)\left({\mathrm{h}}_{\mathrm{c}3}-{\mathrm{h}}_1\right)+\frac{1}{2}{\mathsf{\rho }}_{\mathrm{c}3}\right]{\mathrm{A}}_{\mathrm{ccs}}{\mathrm{L}}_{\mathrm{c}3}$$

$${\mathrm{C}}_{41}=\left({\mathsf{\rho }}_{\mathrm{c}1}-{\mathsf{\rho }}_{\mathrm{c}3}\right){\mathrm{A}}_{\mathrm{ccs}},{\mathrm{C}}_{42}=\left[\left({\mathsf{\rho }}_{\mathrm{g}}-{\mathsf{\rho }}_1\right)\overline{\mathsf{\gamma }}+\left({\mathsf{\rho }}_1-{\mathsf{\rho }}_{\mathrm{c}3}\right)\right]{\mathrm{A}}_{\mathrm{ccs}}$$

$$\begin{array}{c}{\mathrm{C}}_{43}=\left[\left({\left.\frac{\partial {\mathsf{\rho }}_{\mathrm{c}1}}{\partial {\mathrm{P}}_{\mathrm{c}}}\right|}_{{\mathrm{h}}_{\mathrm{c}1}}\right)+\frac{1}{2}\left({\left.\frac{\partial {\mathsf{\rho }}_{\mathrm{c}1}}{\partial {\mathrm{h}}_{\mathrm{c}1}}\right|}_{{\mathrm{P}}_{\mathrm{c}}}\right)\frac{{\mathrm{dh}}_{\mathrm{g}}}{{\mathrm{dP}}_{\mathrm{c}}}\right]{\mathrm{A}}_{\mathrm{ccs}}{\mathrm{L}}_{\mathrm{c}1}+\left(\frac{{\mathrm{d}\mathsf{\rho }}_1}{{\mathrm{dP}}_{\mathrm{c}}}\left(1-\overline{\mathsf{\gamma }}\right)+\frac{{\mathrm{d}\mathsf{\rho }}_{\mathrm{g}}}{{\mathrm{dP}}_{\mathrm{e}}}\overline{\mathsf{\gamma }}\right){\mathrm{A}}_{\mathrm{ccs}}{\mathrm{L}}_{\mathrm{c}2}\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+\left[\left({\left.\frac{\partial {\mathsf{\rho }}_{\mathrm{c}3}}{\partial {\mathrm{P}}_{\mathrm{c}}}\right|}_{{\mathrm{h}}_{\mathrm{c}3}}\right)+\frac{1}{2}\left({\left.\frac{\partial {\mathsf{\rho }}_{\mathrm{c}3}}{\partial {\mathrm{h}}_{\mathrm{c}3}}\right|}_{{\mathrm{P}}_{\mathrm{c}}}\right)\frac{{\mathrm{dh}}_1}{{\mathrm{dP}}_{\mathrm{c}}}\right]{\mathrm{A}}_{\mathrm{ccs}}{\mathrm{L}}_{\mathrm{c}3}\end{array}$$

$${\mathrm{C}}_{44}=\frac{1}{2}\left({\frac{\partial {\mathsf{\rho }}_{\mathrm{c}3}}{\partial {\mathrm{h}}_{\mathrm{c}3}}|}_{{\mathrm{P}}_{\mathrm{c}}}\right){\mathrm{A}}_{\mathrm{ccs}}{\mathrm{L}}_{\mathrm{c}3},{\mathrm{C}}_{51}={\left({\mathrm{C}}_{\mathrm{p}}\mathsf{\rho }\mathrm{V}\right)}_{\mathrm{cw}}\frac{{\mathrm{T}}_{\mathrm{cw}2}-{\mathrm{T}}_{\mathrm{cw}1}}{{\mathrm{L}}_{\mathrm{c}1}},{\mathrm{C}}_{55}= {\left({\mathrm{C}}_{\mathrm{p}}\mathsf{\rho }\mathrm{V}\right)}_{\mathrm{cw}}$$

$${\mathrm{C}}_{66}={\left({\mathrm{C}}_{\mathrm{p}}\mathsf{\rho }\mathrm{V}\right)}_{\mathrm{cw}},{\mathrm{C}}_{71}={\left({\mathrm{C}}_{\mathrm{p}}\mathsf{\rho }\mathrm{V}\right)}_{\mathrm{cw}}\frac{{\mathrm{T}}_{\mathrm{cw}2}-{\mathrm{T}}_{\mathrm{cw}3}}{{\mathrm{L}}_{\mathrm{c}3}},{\mathrm{C}}_{72}= {\left({\mathrm{C}}_{\mathrm{p}}\mathsf{\rho }\mathrm{V}\right)}_{\mathrm{cw}}\frac{{\mathrm{T}}_{\mathrm{cw}2}-{\mathrm{T}}_{\mathrm{cw}3}}{{\mathrm{L}}_{\mathrm{c}3}},{\mathrm{C}}_{77}={\left({\mathrm{C}}_{\mathrm{p}}\mathsf{\rho }\mathrm{V}\right)}_{\mathrm{cw}}$$

The meanings of the states and parameters are provided in Table 1.

2.2.3. Principle of the Compressor

The refrigerant mass flow rate of the compressor outlet is

$${\dot{\mathrm{m}}}_{\mathrm{com}}={\mathrm{f}\mathsf{\eta }}_{\mathrm{vol}}{\mathrm{V}}_{\mathrm{com}}{\mathsf{\rho }}_{\mathrm{g}}$$

(7)

where ${\dot{\mathrm{m}}}_{\mathrm{com}}$ is the refrigerant mass flow rate of the compressor outlet, $\mathrm{kg}/\mathrm{s}$, $\mathrm{f}$ is the working frequency, $\mathrm{Hz}$, ${\mathrm{V}}_{\mathrm{com}}$ is the theoretical gas delivery determined by the producer, ${\mathrm{m}}^3/\mathrm{s}$, ${\mathsf{\rho }}_{\mathrm{g}}$ is the density of refrigerant at the entrance of compressor, $\mathrm{kg}/{\mathrm{m}}^3$, ${\mathsf{\eta }}_{\mathrm{vol}}$ is the volume efficiency of the compressor which follows the formula (8)

$${\mathsf{\eta }}_{\mathrm{vol}}=0.98-0.085[{\left(\frac{{\mathrm{P}}_{\mathrm{c}}}{{\mathrm{P}}_{\mathrm{e}}}\right)}^{\frac{1}{\mathrm{k}}}-1]$$

(8)

where ${\mathrm{P}}_{\mathrm{c}}$ and ${\mathrm{P}}_{\mathrm{e}}$ are the pressure of condensation and evaporation, respectively, $\mathrm{k}$ is the polytrophic exponent.

2.2.4. Principle of the Electronic Expansion Valve

The mass flow of the electronic expansion valve is

$${\dot{\mathrm{m}}}_{\mathrm{e}}={\mathrm{C}}_{\mathrm{v}}\mathrm{k}\sqrt{{\mathsf{\rho }}_{\mathrm{l}}\left({\mathrm{P}}_{\mathrm{c}}-{\mathrm{P}}_{\mathrm{e}}\right)}$$

(9)

where ${\mathrm{C}}_{\mathrm{v}}$ and $\mathrm{k}$ is the flow coefficient and opening of the expansion valve, ${\mathsf{\rho }}_{\mathrm{l}}$ is the density of refrigerant at the entrance of expansion valve, $\mathrm{kg}/{\mathrm{m}}^3$.

3. Reinforcement Learning

3.1. Reinforcement Learning

Reinforcement learning has been paid much attention during the past decades [34,35], whose basics are reviewed as follows. Consider the Markov decision process $MDP(\mathcal{X},\mathcal{A},\mathcal{P},\mathcal{R})$, where $\mathcal{X}$ is a set of states and $\mathcal{A}$ is a set of actions or controls. The transition probabilities $\mathcal{P}:\times \mathcal{X}\times \mathcal{A}\times \mathcal{X}\to [0,1]$ represent each state $\mathrm{x}\mathsf{ϵ}\mathcal{X}$ and action $\mathrm{a}\mathsf{ϵ}\mathcal{A}$, the conditional probability $\mathrm{P}\left(\mathrm{x}\left(\mathrm{k}+1\right),\mathrm{x}\left(\mathrm{k}\right),\mathrm{a}\left(\mathrm{k}\right)\right)=\mathrm{Pr}\left\{\mathrm{x}\left(\mathrm{k}+1\right)\right|\mathrm{x}\left(\mathrm{k}+1\right),\mathrm{a}\left(\mathrm{k}\right)\}$ of transitioning to state $\mathrm{x}\left(\mathrm{k}+1\right)\mathsf{ϵ}\mathcal{X}$ where the MDP is in state $\mathrm{x}\left(\mathrm{k}\right)$ and takes action $\mathrm{a}\left(\mathrm{k}\right)$. The cost function $\mathcal{R}:\mathcal{X}\times \mathcal{A}\times \mathcal{X}\to \mathcal{R}$ is the expected immediate cost ${\mathrm{R}}_{\mathrm{k}}\left(\mathrm{x}\left(\mathrm{k}+1\right),\mathrm{x}\left(\mathrm{k}\right),\mathrm{a}\left(\mathrm{k}\right)\right)$ paid after transition to state $\mathrm{x}\left(\mathrm{k}+1\right)\in \mathcal{X}$, given that the MDP starts in state $\mathrm{x}\left(\mathrm{k}\right)\in \mathcal{X}$ and takes action $\mathrm{a}\left(\mathrm{k}\right)\in \mathcal{U}$. The value of a policy ${\mathrm{V}}_{\mathrm{k}}^{\mathsf{\pi }}\left(\mathrm{x}\left(\mathrm{k}\right)\right)$ is defined as the conditional expected value of the future cost ${\mathrm{E}}_{\mathsf{\pi }}\left\{{{\displaystyle \sum }}_{\mathrm{i}=\mathrm{k}}^{\mathrm{k}+\mathrm{T}}{\mathsf{\gamma }}^{\mathrm{i}-\mathrm{k}}{\mathrm{R}}_{\mathrm{i}}\right\},{\mathcal{R}}_{\mathrm{i}}\in \mathrm{R}$ when starting in state $\mathrm{x}\left(\mathrm{k}\right)$ at time $\mathrm{k}$ and following policy $\mathsf{\pi }\left(\mathrm{x},\mathrm{a}\right)$, The target is to find the optimal actions that maximize the value of the future cost ${\mathrm{V}}_{\mathrm{k}}^{\mathsf{\pi }}\left(\mathrm{x}\right)$

$$\begin{array}{l}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{\mathrm{V}}_{\mathrm{k}}^{\mathsf{\pi }}\left(\mathrm{x}\right)={\mathrm{E}}_{\mathsf{\pi }}\left\{{\displaystyle {\displaystyle \sum }_{\mathrm{i}=\mathrm{k}}^{\mathrm{k}+\mathrm{T}}}{\mathsf{\gamma }}^{\mathrm{i}-\mathrm{k}}{\mathrm{R}}_{\mathrm{i}}\right\}\\ ={\displaystyle {\displaystyle \sum }_{\mathrm{a}}}\mathsf{\pi }\left(\mathrm{x},\mathrm{a}\right){\displaystyle {\displaystyle \sum }_{\mathrm{x}\left(\mathrm{k}+1\right)}}\mathrm{P}\left(\mathrm{x}\left(\mathrm{k}+1\right),\mathrm{x}\left(\mathrm{k}\right),\mathrm{a}\left(\mathrm{k}\right)\right)[{\mathrm{R}}_{\mathrm{k}}\left(\mathrm{x}\left(\mathrm{k}+1\right),\mathrm{x}\left(\mathrm{k}\right),\mathrm{a}\left(\mathrm{k}\right)\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+{\mathsf{\gamma }\mathrm{E}}_{\mathsf{\pi }}\left\{{\displaystyle {\displaystyle \sum }_{\mathrm{i}=\mathrm{k}+1}^{\mathrm{k}+\mathrm{T}}}{\mathsf{\gamma }}^{\mathrm{i}-\left(\mathrm{k}+1\right)}{\mathrm{R}}_{\mathrm{i}}\right\}]\\ \hspace{1em}={\displaystyle {\displaystyle \sum }_{\mathrm{a}}}\mathsf{\pi }\left(\mathrm{x},\mathrm{a}\right){\displaystyle {\displaystyle \sum }_{\mathrm{x}\left(\mathrm{k}+1\right)}}\mathrm{P}\left(\mathrm{x}\left(\mathrm{k}+1\right),\mathrm{x}\left(\mathrm{k}\right),\mathrm{a}\left(\mathrm{k}\right)\right)\left[{\mathrm{R}}_{\mathrm{k}}\left(\mathrm{x}\left(\mathrm{k}+1\right),\mathrm{x}\left(\mathrm{k}\right),\mathrm{a}\left(\mathrm{k}\right)\right)+{\mathsf{\gamma }\mathrm{V}}_{\mathrm{k}+1}^{\mathsf{\pi }}\left(\mathrm{x}\left(\mathrm{k}+1\right)\right)\right]\end{array}$$

(10)

with $\mathrm{T}=\infty$, the value function ${\mathrm{V}}_{\mathrm{k}}^{\mathsf{\pi }}\left(\mathrm{x}\right)$ for the policy $\mathsf{\pi }\left(\mathrm{x},\mathrm{a}\right)$ satisfies the Bellman equation:

$$\begin{array}{c}{\mathrm{V}}_{\mathrm{k}}^{\mathsf{\pi }}\left(\mathrm{x}\right)={\displaystyle {\displaystyle \sum }_{\mathrm{a}}}\mathsf{\pi }\left(\mathrm{x},\mathrm{a}\right){\displaystyle {\displaystyle \sum }_{\mathrm{x}\left(\mathrm{k}+1\right)}}\mathrm{P}\left(\mathrm{x}\left(\mathrm{k}+1\right),\mathrm{x}\left(\mathrm{k}\right),\mathrm{a}\left(\mathrm{k}\right)\right)[{\mathrm{R}}_{\mathrm{k}}\left(\mathrm{x}\left(\mathrm{k}+1\right),\mathrm{x}\left(\mathrm{k}\right),\mathrm{a}\left(\mathrm{k}\right)\right)\\ +{\mathsf{\gamma }\mathrm{V}}_{\mathrm{k}+1}^{\mathsf{\pi }}\left(\mathrm{x}\left(\mathrm{k}+1\right)\right)]\end{array}$$

(11)

To a deterministic system ${{\displaystyle \sum }}_{\mathrm{a}}{\mathsf{\pi }}_{\mathrm{k}}\left(\mathrm{x},\mathrm{a}\right){{\displaystyle \sum }}_{\mathrm{x}\left(\mathrm{k}+1\right)}\mathrm{P}\left(\mathrm{x}\left(\mathrm{k}+1\right),\mathrm{x}\left(\mathrm{k}\right),\mathrm{a}\left(\mathrm{k}\right)\right)=1$, so the optimal actions can be gained by alternating the policy evaluation (12) and policy improvement (13) according to the following two equations:

$${\mathrm{V}}_{\mathrm{k}}\left(\mathrm{x}\right)={\mathrm{R}}_{\mathrm{k}}\left(\mathrm{x}\left(\mathrm{k}+1\right),\mathrm{x}\left(\mathrm{k}\right),\mathrm{a}\left(\mathrm{k}\right)\right)+{\mathsf{\gamma }\mathrm{V}}_{\mathrm{k}}\left(\mathrm{x}\left(\mathrm{k}+1\right)\right)$$

(12)

$${\mathsf{\pi }}_{\mathrm{k}}\left(\mathrm{x},\mathrm{a}\right)= \underset{\mathsf{\pi }}{\mathrm{argmax}}{\mathrm{R}}_{\mathrm{k}}\left(\mathrm{x}\left(\mathrm{k}+1\right),\mathrm{x}\left(\mathrm{k}\right),\mathrm{a}\left(\mathrm{k}\right)\right)+{\mathsf{\gamma }\mathrm{V}}_{\mathrm{k}}\left(\mathrm{x}\left(\mathrm{k}+1\right)\right)$$

(13)

where $\mathsf{\gamma }$ is a discount factor,$0\le \mathsf{\gamma }<1$ in order to be convergent.

3.2. Q-Algorithm

In fact, formulas (12) and (13) cannot be solved directly because the need the information of ${\mathrm{V}}_{\mathrm{k}}\left(\mathrm{x}\left(\mathrm{k}+1\right)\right)$ that no one knows. A Q-algorithm proposed by Watkins [34] provides an effective solution by substituting function Q. The Q-algorithm defines the evaluation function $\mathrm{Q}\left(\mathrm{x}\left(\mathrm{k}\right),\mathrm{a}\left(\mathrm{k}\right)\right)$ as the maximum discounted cumulative reward that can be achieved from state $\mathrm{x}\left(\mathrm{k}\right)$ and $\mathrm{a}\left(\mathrm{k}\right)$ as the first action:

$$\mathrm{Q}\left(\mathrm{x}\left(\mathrm{k}\right),\mathrm{a}\left(\mathrm{k}\right)\right)\stackrel{\mathrm{def}}{=}{\mathrm{R}}_{\mathrm{k}}\left(\mathrm{x}\left(\mathrm{k}\right),\mathrm{a}\left(\mathrm{k}\right)\right)+{\mathrm{V}}^{*}\left(\mathrm{x}\left(\mathrm{k}+1\right)\right)$$

(14)

where the state $\mathrm{x}\left(\mathrm{k}+1\right)$ comes from $\mathrm{x}\left(\mathrm{k}\right)$ and $\mathrm{a}\left(\mathrm{k}\right)$, ${\mathrm{V}}^{*}\left(\mathrm{x}\left(\mathrm{k}+1\right)\right)$ is the optimization of ${\mathrm{V}}_{\mathrm{k}}\left(\mathrm{x}\left(\mathrm{k}+1\right)\right)$ beginning with $\mathrm{x}\left(\mathrm{k}+1\right)$ and following the optimal actions.

Denote the optimum of $\mathrm{Q}$ as ${\mathrm{Q}}^{*}$, therefore one has

$$\begin{array}{l}{\mathrm{Q}}^{*}\left(\mathrm{x}\left(\mathrm{k}\right),\mathrm{a}(\mathrm{k}\right))\\ \hspace{1em}\hspace{1em}=\underset{\mathrm{a}\left(\mathrm{k}\right)}{\max }\left[{\mathrm{R}}_{\mathrm{k}}\left(\mathrm{x}\left(\mathrm{k}\right),\mathrm{a}\left(\mathrm{k}\right)\right)+{\mathrm{V}}^{*}\left(\mathrm{x}\left(\mathrm{k}+1\right)\right)\right]={\mathrm{R}}^{*}\left(\mathrm{x}\left(\mathrm{k}\right),\mathrm{a}\left(\mathrm{k}\right)\right)+{\mathrm{V}}^{*}\left(\mathrm{x}\left(\mathrm{k}+1\right)\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}={\mathrm{V}}^{*}\left(\mathrm{x}\left(\mathrm{k}\right),\mathrm{a}\left(\mathrm{k}\right)\right)\end{array}$$

(15)

where the superscript $*$ expresses the optimal values.

It is seen from formula (15) that ${\mathrm{Q}}^{*}\left(\left(\mathrm{x}\left(\mathrm{k}\right),\mathrm{a}(\mathrm{k}\right)\right)$ is equivalent to ${\mathrm{V}}^{*}\left(\mathrm{x}\left(\mathrm{k}\right),\mathrm{a}\left(\mathrm{k}\right)\right)$ with the same action. Therefore, the optimal actions ${\mathrm{a}}^{*}\left(\mathrm{k}\right)$ can be obtained by the value iteration following the formula:

$$\begin{array}{l}\mathrm{Q}\left(\mathrm{x}\left(\mathrm{k}\right),\mathrm{a}\left(\mathrm{k}\right)\right)\leftarrow \mathrm{Q}\left(\mathrm{x}\left(\mathrm{k}\right),\mathrm{a}\left(\mathrm{k}\right)\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+\mathsf{\alpha }\left[\mathrm{R}\left(\mathrm{x}\left(\mathrm{k}\right),\mathrm{a}\left(\mathrm{k}\right)\right)+\mathsf{\gamma }\mathrm{maxQ}\left(\mathrm{x}\left(\mathrm{k}+1\right),\mathrm{a}\left(\mathrm{k}\right)\right)-\mathrm{Q}\left(\mathrm{x}\left(\mathrm{k}\right),\mathrm{a}\left(\mathrm{k}\right)\right)\right]\end{array}$$

(16)

where $\mathsf{\alpha }$ is a learning rate,$0\le \mathsf{\alpha }<1$, and the states $\mathrm{x}\left(\mathrm{k}+1\right)$ will be got at the next sampling time $\mathrm{k}+1$. By using this iteration formula, it will finally converge to steady state by continuously adjusting the action $\mathrm{a}\left(\mathrm{k}+1\right)$ and one obtains the responding control ${\mathrm{a}}^{*}\left(\mathrm{k}\right)$.

$${\mathrm{a}}^{*}\left(\mathrm{k}\right)=\arg \underset{\mathrm{a}}{\max }\mathrm{Q}\left(\mathrm{x}\left(\mathrm{k}\right),\mathrm{a}\left(\mathrm{k}\right)\right)$$

(17)

A proof on strict convergence of Q-valued function (16) was given by Watskin [35], whose theorem is rewritten here as Lemma 1.

Define ${\mathrm{n}}^{\mathrm{i}}\left(\mathrm{x},\mathrm{a}\right)$ as the index of the $\mathrm{i}$th time that action $\mathrm{a}$ is tried in state x.

Lemma 1 [35]:

Given bounded rewards$\left|R_n\right|\le C$, learning rates$0\le {\alpha }_n<1$, and${{\displaystyle \sum }}_{i = 1}^{\infty }{\alpha }_{n^i\left(x,a\right)} = \infty$,${{\displaystyle \sum }}_{i = 1}^{\infty }{\left[{\alpha }_{n^i\left(x,a\right)}\right]}^2<\infty ,$$\forall x,a$, then$Q_n\left(x,a\right)\to Q^{*}\left(x,a\right)$as$n\to \infty$,$\forall x,a$with probability 1.

4. The Proposed Approach

The scheme of the proposed approach is indicted as Figure 2. It is a hierarchical control of an integer of the process level and the loop level. The process level targets the conversion efficiency of the refrigerating system by adjusting action variables whose optimization ${\mathrm{a}}^{*}$ is the reference of the loop level. The loop level as a basic control circuit implements the reference control by adjusting the control variables $\mathrm{u}$ whose elements are the compressor frequency and the electronic expansion valve opening, respectively.

4.1. Process Level

4.1.1. States Vector, Action Vector, and Cost Function

As discussed in Section 2.2, there are seven states in evaporation and five states in condensation with a complex description and high order nonlinearity. Some hypotheses are considered for the sake of simplicity. (1) The evaporator and the condenser are regarded as a two-phase zone with the extent of superheat in evaporator being estimated by the local energy conservation because the most area in heating change belongs to the two-phase zone, according to experiences. (2) Only the temperature of heating exchange is changing in the thermodynamic equation and only the length of the two-phase zone is changing in the mass conversation equation. (3) The void fraction is considered as a constant based on the fact of a small change in the working condition. (4) The heat transfer coefficient and the refrigerant physical parameter, which can be obtained by the identification technique are supposed to be constant though they are different depending on conditions. Based on the above hypotheses the states vector is reduced as

$$\mathrm{x}={\left[\begin{array}{ccc}{\mathrm{x}}_1& {\mathrm{x}}_2& \begin{array}{ccc}{\mathrm{x}}_3& {\mathrm{x}}_4& {\mathrm{x}}_5\end{array}\end{array}\right]}^{\mathrm{T}}={\left[\begin{array}{ccc}{\mathrm{T}}_{\mathrm{e}}& {\mathrm{l}}_{\mathrm{e}}& \begin{array}{ccc}{\mathrm{T}}_{\mathrm{c}}& {\mathrm{l}}_{\mathrm{c}}& {\mathrm{T}}_{\mathrm{wa}}\end{array}\end{array}\right]}^{\mathrm{T}}$$

(18)

where ${\mathrm{l}}_{\mathrm{e}}$ is the length of the two-phase zone in evaporator, m, ${\mathrm{T}}_{\mathrm{wa}}$ is the tube wall temperature of the condenser. It is important to point out that the temperature states can be measured directly by sensors and the length states can be obtained by soft-sensor with the degree of superheat.

The action vector of the process is

$$\mathrm{a}={\left[{\mathrm{m}}_{\mathrm{e}},{\mathrm{m}}_{\mathrm{com}}\right]}^{\mathrm{T}}$$

(19)

where ${\mathrm{m}}_{\mathrm{e}}$ and ${\mathrm{m}}_{\mathrm{com}}$ are the mass flow of the expansion valve refrigerant and of the compressor refrigerant which are controlled by the opening of the expansion valve $\mathrm{K}$ and the frequency of the compressor $\mathrm{f}$, according to formula (7) and formula (9), respectively.

The cost function is selected as the goal under an action $\mathrm{a}\left(\mathrm{k}\right)\in \mathcal{U}$

$${\mathrm{R}}_{\mathrm{k}}\left(\mathrm{x}\left(\mathrm{k}\right),\mathrm{a}\left(\mathrm{k}\right)\right)={\frac{T_c\left(k\right)}{T_c\left(k\right)-T_e\left(k\right)}|}_{a\left(k\right)}$$

(20)

As a result, the optimal actions ${\mathrm{a}}^{*}\left(\mathrm{k}\right)$ can be obtained by Q-algorithm following the formulas (16) and (17).

4.1.2. Exploitation and Exploration

The RL gets the optimal action according to the reward from the interaction with the environment. There are two ways called the exploitation, which seeks the optimal action according to the best rewards of Q-value and the exploration, which seeks the optimal action by dispatching randomly at a certain probability in order to escape the local optimization of exploitation or find a new unknown optimization. The $\mathsf{\epsilon }$-greedy method is proposed to carry out the exploitation and the exploration with the form of formula (21)

$$\mathrm{a}=\{\begin{array}{ll}\arg \max \mathrm{Q}\left(\mathrm{s},\mathrm{a}\right),& \mathrm{p}<\epsilon \\ \hspace{1em}\hspace{1em} \mathrm{rand}\left(\mathrm{U}\right),& \mathrm{p}\ge \epsilon \end{array}$$

(21)

where $\mathsf{\epsilon }$ is a pre-set probability (usually as a large probability event), $\mathrm{p}$ is the probability of action selection, and $\mathrm{U}$ is the action space.

4.1.3. Procedure of the Proposed Algorithm

Based on the selection of action vector, state vector, and the cost function, the procedure of the proposed algorithm that begins with state vector $\mathrm{x}\left(\mathrm{k}\right)$ is summarized as procedure 1.

Procedure 1.

Step 1:

Select an action $\mathrm{u}\left(\mathrm{k}\right)$ randomly.

Step 2:

Receive immediate reward $\mathrm{R}\left(\mathrm{x}\left(\mathrm{k}\right),\mathrm{a}\left(\mathrm{k}\right)\right)$ according to formula (20).

Step 3:

Get the new state $\mathrm{x}\left(\mathrm{k}+1\right)$ according to the coarse model and compute value function according to formula (10).

Step 4:

Step 4: Renew the reward value $\mathrm{Q}$ based on current state $\mathrm{x}\left(\mathrm{k}\right)$

$$\mathrm{Q}\left(\mathrm{x}\left(\mathrm{k}\right),\mathrm{a}\left(\mathrm{k}\right)\right)=\mathrm{R}\left(\mathrm{x}\left(\mathrm{k}\right),\mathrm{a}\left(\mathrm{k}\right)\right)+\mathsf{\gamma }\underset{\mathrm{a}\left(\mathrm{k}+1\right)}{\min }\mathrm{Q}\left(\mathrm{x}\left(\mathrm{k}+1\right),\mathrm{a}\left(\mathrm{k}+1\right)\right)$$

(22)

Step 5:

Adjust a new control vector $\mathrm{a}\left(\mathrm{k}+1\right)$ with $\mathsf{\epsilon }$-greedy method according to (21).

Step 6:

Repeat step 1 to step 5 until it is convergent.

Step 7:

Get the optimal ${\mathrm{a}}^{*}\left(\mathrm{k}\right)$ according to the best reward

$${\mathrm{a}}^{*}\left(\mathrm{k}\right)=\underset{\mathrm{a}\left(\mathrm{k}\right)}{\mathrm{argmax}}\mathrm{Q}\left(\mathrm{x}\left(\mathrm{k}\right),\mathrm{a}\left(\mathrm{k}\right)\right)$$

(23)

Step 8:

Apply ${\mathrm{a}}^{*}\left(\mathrm{k}\right)$ to the refrigeration system as the reference of the loop level and then get $\mathrm{x}\left(\mathrm{k}+1\right)$ under the control $\mathrm{u}$.

Step 9:

Replace $\mathrm{x}\left(\mathrm{k}\right)$ by $\mathrm{x}\left(\mathrm{k}+1\right)$ and go to step 1.

4.2. Loop Level

The technology of loop control has been very mature and a simple PID is proposed to achieve this function for each loop whose framework is in Figure 3 and Figure 4. Here ${\mathrm{m}}_{\mathrm{com}}$ and ${\mathrm{m}}_{\mathrm{e}}$ are the reference variables which come from the process level by RL. The $\mathrm{u}1$ and $\mathrm{u}2$ are the control variables which exert the effect on the system by the associated actuators that are parts of the compression refrigeration system. The $\mathrm{y}1$ and $\mathrm{y}2$ are the system outputs which are obtained by the sensors.

5. Case Studies

Our test-bed of a compression refrigeration system is seen in Figure 5, and its structure is dedicated in Figure 6 [36,37]. The liquid refrigerant stored in the tank goes into the heating interexchange through the desiccator, where they gasify by absorbing the heat. The gas comes back to the condenser driven by a compressor and becomes liquid in the stored tank. The exchanged cool air goes into the compound air conditioning plant to adjust the environment. The main devices and their specifications are listed in

Table 2. The devices and specifications.

No.	Name of Devices	Specifications
1	Compressor	Semi-closed piston compressor, rated speed 1450 rpm, 7.5 Kw/10 HP, rated discharge 38.25 m3/h, rated refrigeration capacity 30 KW
2	Frequency Converter for Compressor	Capacity 11 KVA, rated current 17 A, rated voltage 380 V, frequency 0–50 Hz, V/F signal 0–10 V
3	Condenser	Borehole heat exchangers, heat exchange 35 Kw, heat transfer area 3.4 m2, heat transfer coefficient K = 970 W/m2·°C
4	Evaporator	Borehole heat exchangers, heat exchange 30 kW, heat transfer area 3.02 m2, heat transfer coefficient K = 1455 W/m2·°C
5	Electronic Expansion Valve	Flow coefficient Kvs = 0.63 m3/h, refrigerating capacity 74 kW, caliber DN15, AC24 V, control signal 0–10 V
6	Cooling Water Pump	Multistage centrifugal pump: Rated power 2.2 kW, rated speed 2840 rpm, lift 32 m, flow 8.4 m3/h
7	Frequency Converter for Cooling Water Pump	Rated voltage 380 V, frequency 0–50 Hz, V/F signal 0–10 V
8	Chilled Water Pump	Single-stage centrifugal pump, rated power 1.5 kW, rated speed 2840 rpm, lift 27.4 m, flow 7.8 m3/h
9	Frequency Converter for Refrigerated Water Pump	Rated voltage 380 V, frequency 0-50 Hz, V/F signal 0-10 V
10	The Cooling Tank	Volume 1 m3
11	The Chilled Tank	Volume 1 m3
12	Reservoir	Volume 20 L

.

The temperature was obtained by the temperature transmitter of Pt100 with the range of 0–30 °C, 0–60 °C, and 0–100 °C under the degree of accuracy ±0.1 °C (class A). The temperature of the evaporator and condenser was limited within the range of −5–15 °C and 30–50 °C, respectively. The pressure was obtained by the pressure transmitter with the range of 0–2.5 MPa, 0–1.6 MPa under the degree of accuracy 0.25%. The electromagnetic flowmeters with the degree of accuracy 0.5% were added to measure the ${\mathrm{m}}_{\mathrm{com}}$ and ${\mathrm{m}}_{\mathrm{e}}$, which were the action vectors of the process level. The traditional coefficient of performance ($\mathrm{COP}$) was obtained by the stable state of the compression refrigeration system. However, the loads are often uncertain which makes the compression refrigeration system stay in the transient process. As a result, a 30 kW tunable stainless steel electric heater was applied to simulate a thermal load with the range from 22.5 to 30 kW and the sampling time was selected as 1 min. In the experiment, we adjusted the electric heater by changing the resistance value through rotating an adjustment button. In this way, we simulated the load climb by running the adjustment button in one direction and the load uncertainty by random changing the direction.

5.1. Effect of Model Accuracy on Performance

A coarse model was built based on the principle of compression refrigeration in the process level with the following form:

$$\{\begin{array}{c}{\dot{\mathrm{T}}}_{\mathrm{e}} = {\mathrm{c}}_0{\mathrm{m}}_{\mathrm{com}}+{\mathrm{c}}_1{\mathrm{l}}_{\mathrm{e}}\left({\mathrm{T}}_{\mathrm{w}}-{\mathrm{T}}_{\mathrm{e}}\right)\\ {\dot{\mathrm{l}}}_{\mathrm{e}} = {\mathrm{c}}_2\left({\mathrm{m}}_{\mathrm{e}}-{\mathrm{m}}_{\mathrm{com}}\right)\\ {\dot{\mathrm{T}}}_{\mathrm{c}} = {\mathrm{c}}_3{\mathrm{m}}_{\mathrm{com}}+{\mathrm{c}}_4{\mathrm{l}}_{\mathrm{c}}\left({\mathrm{T}}_{\mathrm{wa}}-{\mathrm{T}}_{\mathrm{c}}\right)\\ {\dot{\mathrm{l}}}_{\mathrm{c}} ={\mathrm{c}}_5\left({\mathrm{m}}_{\mathrm{com}}-{\mathrm{m}}_{\mathrm{e}}\right)\\ {\dot{\mathrm{T}}}_{\mathrm{wa}} = {\mathrm{c}}_6\left({\mathrm{T}}_{\mathrm{c}}-{\mathrm{T}}_{\mathrm{wa}}\right)+{\mathrm{c}}_7\left({\mathrm{T}}_{\mathrm{wa}}-{\mathrm{T}}_{\mathrm{a}}\right)\end{array}$$

(24)

where the variables of ${\mathrm{T}}_{\mathrm{e}},{\mathrm{T}}_{\mathrm{w}}$,${\mathrm{T}}_{\mathrm{c}}$, ${\mathrm{l}}_{\mathrm{e}}$,${\mathrm{l}}_{\mathrm{c}}$,${\mathrm{T}}_{\mathrm{wa}}$,${\mathrm{T}}_{\mathrm{a}}$,${\mathrm{m}}_{\mathrm{e}}$ and ${\mathrm{m}}_{\mathrm{com}}$ are the same meaning in Table 1. The coefficients ${\mathrm{c}}_0$-${\mathrm{c}}_7$ are obtained based on collected data by the technology of the system identification [33]. The values are seen in

Table 3. The value of parameters in the coarse model [33].

C0	C1	C2	C3	C4	C5	C6	C7
−1672	1	5	1	456	0.3	−1	1

.

Taking the values from Table 3 as the criterion (called the first coarse model) we built the second coarse model by changing the value of coefficient c0 from −1672 to −1500, the values with the rate 4.58%, and the value of coefficient c4 from 456 to 450. The control algorithm follows the procedure 1. The evolution of states, the instantaneous efficiency and the action variables are seen in Figure 7, Figure 8 and Figure 9. The refrigerant circulated between gas and liquid with endothermic and exothermic reactions. The evaporation temperature was the external temperature of the refrigerant during the evaporation of the evaporator which usually is constantly pre-set according to the environmental requirement. It is seen from Figure 7 that the evaporation temperature Te (either of the first coarse model (blue curve) or of the second coarse model) fluctuated around 7 °C. The condensing temperature is prone to be affected by the working pressure. More energy changing will produce higher pressure which will increase the condensing temperature. It is seen from Figure 7 that the condensing temperature Tc (either of the first coarse model (blue curve) or of the second coarse model) was rising slowly to meet the requirement of the loads from 22.5 to 30 kW. Different from the conventional coefficient of performance (COP) which is defined in the steady state, our COP is extended to the whole process which includes both the transient and steady. The transient is often prevailing due to the resistance change. Therefore, the mean of the COP during a certain period of time is meaningful according to our definition. The curve of instantaneous efficiency in Figure 8 shows the differences between coarse models on the system. There was an instantaneous error between the first coarse model (blue curve) and the second coarse model (green curve). However, the average efficiency of both coarse models during 200 min was 1.2681 and 1.2685, respectively, with the error of 0.03%. We calculated the residual to evaluate the robustness of both coarse models. First, we made a polynomial fitting by the least square method with the same style of cubic curves (no significant error change over this order) according to the standard procedure [36] and obtained the estimation $\widehat{COP}$.

$$\widehat{COP} = p_1·CO{P}^3+p_2·CO{P}^2+p_3·COP+p_4$$

(25)

where the coefficients are shown in

Table 4. Coefficients of the polynomial fitting.

	${\mathit{p}}_1$	${\mathit{p}}_2$	${\mathit{p}}_3$	${\mathit{p}}_4$
The first coarse model	7.9673 × 10−8	3.1662 × 10−5	0.0034666	1.1841
The second coarse model	1.0584 × 10−7	3.9588 × 10−5	0.0043562	0.69583

.

Therefore, the norm of residual ($\sigma$) is obtained according to the formula (26):

$$\sigma = \sqrt{\frac{{{\displaystyle \sum }}_{k=1}^l{\left(COP\left(k\right)-\widehat{COP}\left(k\right)\right)}^2}{l-1}}$$

(26)

where $l$ is the number of examples. The responding norms of residual are ${\sigma }_1 = 0.60575$ and ${\sigma }_2 =$0.67583, respectively, which means the coarse model has a good robustness due to the approximation norm of residual. As a result, we can evaluate the effect of model accuracy on the performance of efficiency by comparing the result of the COP with changing the value of parameters. The mean change of 0.03% compared with the parameter’s change of 4.58% indicates only a small amount of the impact among the coarse models.

Figure 9 gives the action variables of ${\mathrm{m}}_{\mathrm{e}}$ and ${\mathrm{m}}_{\mathrm{com}}$. It can be seen that there were noticeable fluctuations in ${\mathrm{m}}_{\mathrm{e}}$ and ${\mathrm{m}}_{\mathrm{com}}$. It is due to the RL implementation and load variations. We adopted the classical table mode to map the relation between the states and the actions. However, limited by our computer’s capability (Intel(R) Core(TM)2 Duo CPU E7300 @2.66GHz @2.67GHz RAM 4.00GB) the actions were divided into 1K pieces which causes some errors between the acquired actions and the ideal actions. The errors were somewhat enlarged in the process of evaluating the action value because we took each episode to last 200 steps instead of $\mathrm{T}=\infty$. Another important factor is the influence of the load variations. The final actions of the RL algorithm were determined by the initial states and the current loads. The load variation was simulated by sequentially rotating the adjustment button which lead to the continuous transition.

5.2. Comparison with the Conventional Approach

There is no method to deal with the coefficient of performance (COP) under the temperature changing by load variation. The predominant approach in HVAC is to keep the outlet temperature matching the loads by automatically adjusting the flow with an electronic expansion valve and maintaining the constant speed of the compressor. We compared the proposed method with this conventional approach. Figure 10, Figure 11 and Figure 12 show the evolution of states Te and Tc, the instantaneous efficiency, and the control variables. The blue curves and the red curves represent the results of the proposed approach and the conventional approach, respectively. It is seen from Figure 11 that the Tc and Te with the conventional approach and the proposed approach are a similar tendency to meet the requirement of loads during the test period. Te with the proposed approach shows large fluctuations due to the coarse model leading to a rough estimation of states. While the Te with the conventional approach shows a good smooth curve because it is under classical control. The Tc has small fluctuations because the accuracy of Tc is better.

There is a small fluctuation of instantaneous efficiency for the conventional approach because it tries to keep the temperature bias of inlet and outlet by taking control. Our proposed method releases the control of temperature within the range of limitation, so the instantaneous efficiency formulated from the ${\mathrm{T}}_{\mathrm{e}}$ and Tc shows bigger fluctuations, even sometimes low efficiency and sometimes high efficiency. We tried to reduce this fluctuation by adjusting the parameters. First, we changed the pre-set probability $\mathsf{\epsilon }$ from 0.7 to 0.9, which is the general value for errors and trials method. However, there was no improvement of fluctuation and efficiency. Considering the requirement of real time, we did not decrease under 0.7, which means that 30% more searches were used to explore, but in fact, the new states can be computed according to the coarse model, which means it does not need a high probability to explore. Instead, we tried to increase the value of $\mathsf{\epsilon }$ and get a good COP (1.34% better than the conventional method) even by setting $\mathsf{\epsilon }$ as 0.99, though there was no improvement in the fluctuation. We also tried to adjust the learning rate $\mathsf{\alpha }$ that is a compromise between the speed and the accuracy of training. If it is too small, the training will take more time. If it is too big. the training will reduce accuracy. There is no theoretical basis so far in reinforcement learning but with an empirical value of 0.95–0.98. It was noticed that the simulation shows the fluctuations were still present. A representative is shown in Figure 11. It is vague for instantaneous efficiency in Figure 11. However, as with an energy conversion process, we are more concerned with refrigeration efficiency over a period of time. The average efficiency was proposed as the metrics. The average efficiency with the conventional approach (red curve) and with the proposed approach (blue curve) was 1.2513 and 1.2681, respectively. The proposed approach will raise the efficiency by 1.34% more than the conventional approach on average during the period of 200 min.

The control variables are shown in Figure 12. As we have pointed out in 4.2, the controller parameters of PID1 and PID2 should be predesigned. The noticeable fluctuations in our proposed method cause difficulties in regulating the parameters of PID. It is fortunate that the refrigeration is a slow second-order system and the constant parameters of PID have a wide feasible scope around the working point. The values of parameters are prepared group by group in the table based on the previous experiments by considering the system stability. Therefore, the controller will directly call the group of parameter values from this preparatory table to track the references of actions provided by the process level which are fluctuating due to the partition error of actions and the load variation.

6. Conclusions

In this paper, we have discussed an online data-driven approach to improve the conversion efficiency of refrigeration system under the condition of load variation. A reinforcement learning approach that has an ability of seeking the unknown environment is proposed to find the optimal actions based on the online data in the process level and its risk on the system because the training process is complex and unknown in advance due to the warning from the computer simulation. A coarse model is developed to evaluate the action value which reduces the dependence of traditional control methods on model accuracy. Finally, the actions are achieved as the pre-set variables by implementing the single loop control. The simulation shows the proposed algorithm is better than the conventional methods in the conversion efficiency of the refrigeration system from the viewpoint of the average, although larger fluctuations are noticeable. How to reduce this fluctuation is our further study.