Digitalization for Sustainable Heat Pump Operation: Review on Smart Control and Optimization Strategies

Abstract

This review provides a comprehensive analysis of advanced control strategies and operational optimization of energy systems, focusing on heat pumps, with an emphasis on their role in enhancing energy efficiency and operational flexibility. The study concentrates on methods supported by artificial intelligence algorithms, highlighting Model Predictive Control (MPC), Reinforcement Learning (RL), and hybrid approaches that combine the advantages of both. These methods aim to optimize both the operation of heat pumps and their interaction with thermal energy storage (TES) systems, renewable energy sources, and power grids, thereby enhancing the flexibility and adaptability of the systems under real operating conditions. Through a systematic analysis of the existing literature, 95 studies published after 2019 were examined to identify research trends, key challenges such as computational requirements and algorithm interpretability, and future opportunities. Furthermore, significant benefits of applying advanced control compared to conventional practices were highlighted, such as reduced operational costs and lower CO₂ emissions, emphasizing the importance of heat pumps in the energy transition. Thus, the analysis highlights the need for digital solutions, robust and adaptive control frameworks, and holistic techno-economic evaluation methods in order to fully exploit the potential of heat pumps and accelerate the transition to sustainable and flexible energy systems.

1. Introduction

The impacts of climate change are already affecting humanity through more frequent extreme weather events, rising global temperatures, and disruptions to ecosystems, agriculture, and public health. These pressing challenges necessitate the transformation of energy systems, specifically by reducing reliance on fossil fuels and promoting sustainable energy solutions to secure a resilient global future. In response to these urgent needs, the European Commission announced the European Green Deal in December 2019, aiming for climate neutrality by 2050 in line with the commitments of the Paris Agreement [1].

1.1. Heat Pumps as a Key Technology for Decarbonization

The demand for thermal energy is a critical area of this transition, as heating and cooling account for half of global energy consumption [2] and are expected to increase further in the coming years due to climate change. In this framework, heat pumps powered by low-emission electricity emerge as one of the most critical technologies for achieving climate targets and facilitating the transition to a sustainable energy future [3]. By harnessing low-temperature renewable sources, heat pumps offer high energy efficiency and can significantly reduce dependence on fossil fuels. Their ability to be integrated with renewable energy systems, thermal storage, and smart grids makes them a strategic element for the decarbonization of both the building and industrial sectors. A main challenge in the energy transition concerns the electrification of systems and their smooth integration into the grids, resulting from the variability of generation and the imbalance between supply and demand. In this context, digitalization emerges as a key enabler for transforming energy systems, as it can address these issues by facilitating energy management, improving energy efficiency, and reducing CO₂ emissions [3]. In this framework of digitalization and artificial intelligence development, a growing research focus is emerging on the control and optimization of heat pump systems, aiming to enhance energy efficiency and enable a more sustainable energy future.

1.2. Literature Analysis Approach

This review offers a comprehensive and critical examination of the current state of optimal control for heat pump systems within the context of advanced digitalization of energy systems. The objective is to identify the capabilities, challenges, and future directions that can enhance energy efficiency, flexibility, and the contribution of heat pumps to the transition towards a sustainable energy system. To enhance the organization of the article collection and the overall research, three research questions were identified, serving as a roadmap and aiding the entire process:

• RQ1: What are the main advanced control strategies for heat pump systems?
• RQ2: What are the advantages and limitations of each strategy?
• RQ3: What are the reported outcomes of implementing advanced control strategies in heat pump systems, and what are the main challenges for their widespread adoption?

Guided by these three research questions, the methodology followed for the literature review is illustrated in Figure 1.

Initially, a search was conducted to identify various scientific studies in the literature database Scopus and the academic search engine Google Scholar, which were selected for their broad representation of topics in science and engineering. A search was conducted for various papers using a range of carefully selected keywords. These phrases included “model-free control heat pump systems,” “model predictive control heat pump systems,” “digitalization heat pump systems,” as well as terms related to flexibility, smart control, and other aspects of heat pump system optimization and digitalization. This approach ensured a comprehensive coverage of the scientific literature on the topic. After a rapid screening of the abstracts, the most relevant studies were selected for detailed analysis. Subsequently, the selected articles were further refined using specific filters: review papers were excluded, only peer-reviewed articles written in English were retained, and the selection was limited to articles published from 2019 and later. Furthermore, each chosen article was assessed for quality, considering the authors’ scientific contributions and the research methodologies applied, with priority given to papers published in reputable journals and conferences (e.g., Elsevier, MDPI, IEEE). Finally, 95 papers were selected for further analysis. The results were structured into categories based on the control strategy applied in each system and the application domain. This structure enabled the identification of the advantages of each method and technology, as well as the highlighting of future directions and the main obstacles to their implementation.

Additionally, a bibliometric analysis was conducted using VOSviewer (v.1.6.20) on the 95 selected articles, as shown in Figure 2. VOSviewer was used to extract and visualize the keywords of the articles, displaying their frequency and co-occurrence over time. The temporal mapping highlighted emerging trends, shifts in research areas, and the evolution of key topics related to heat pump control, optimization, digitalization, and related fields.

This visualization enabled a clearer understanding of the concepts that have gained prominence in recent years, thus facilitating the interpretation and categorization of control strategies and application domains. As observed, model-free control, such as deep reinforcement learning, has attracted significant research interest in recent years, indicating that it still holds considerable potential for further investigation. Moreover, additional studies are needed to verify and evaluate the performance of new control strategies for heat pump systems. For this reason, the hardware-in-the-loop approach is gaining increasing importance in the current literature, while district energy systems have also emerged as a recent area of interest within the framework of decarbonizing energy systems. All the findings from the VOSviewer analysis are described, analyzed, and discussed in the following sections.

1.3. Paper Structure

The structure of the article is illustrated in Figure 3. Section 1 served as the introduction, presenting the motivation behind this review and the literature analysis approach that was adopted. Subsequently, Section 2 provides a concise overview of the fundamental principles of heat pump systems, as well as the system characteristics and parameters that influence their performance and enable adaptable operation, which are topics increasingly gaining importance in contemporary research. In Section 3, the results of the extensive literature review on the control and optimization of various heat pump systems are presented and discussed, alongside a structured categorization and analysis of the selected articles. Section 4 presents a critical analysis of the reviewed control strategies, highlighting the relationships between control methods, application fields, and performance indicators, and identifying key insights, challenges, and research gaps. Finally, Section 5 presents the conclusions of this review, summarizing the main findings, highlighting the key contributions, and outlining future research directions for heat pump systems.

2. Technologies and Application of Heat Pumps

This chapter provides a comprehensive overview of heat pump systems, focusing on their classification, main technologies, and practical applications. It begins with an examination of the classification of heat pumps, differentiating them according to operating principles, energy sources, and system configurations. Next, some of the key applications of heat pumps with significant market presence are highlighted. Finally, the chapter discusses the potential of heat pumps to participate in demand response strategies, illustrating how these systems can support grid balancing, facilitate the integration of variable renewable energy sources, and optimize overall energy consumption. This overview lays the essential technical foundation for the understanding of subsequent chapters on control strategies, system optimization, and digitalization of heat pump technologies.

2.1. Classification of Heat Pump Systems

A heat pump is a device that transfers heat from a lower temperature to a higher temperature, contrary to the natural flow of heat, using energy. Heat pumps can be classified into electrically driven heat pumps and thermally driven heat pumps [4]. Electrically driven heat pumps use electricity to operate the compression cycle, while thermally driven heat pumps use heat or an engine to drive the sorption or compression cycle. Gas sorption heat pump, thermal compression heat pump, and gas engine heat pump are the three main types of thermally driven heat pumps. Both electrically driven and thermally driven heat pumps can operate with various cold sources and heat sinks, such as air, water, and ground [5]. Furthermore, heat pumps can also be classified according to their supply temperature. Low-temperature supply heat pumps typically provide around 35 °C, while intermediate-temperature systems operate at approximately 45 °C. Medium-temperature heat pumps deliver heat at about 55 °C, and high-temperature units can reach up to 200 °C [4], mainly used in industrial applications.

2.2. Key Developments and Applications of Heat Pumps

This section presents certain heat pump technologies and applications that play a significant role in both the market and current research activity, without being considered the only or dominant ones. The most notable among these include:

• Heat pumps combined with energy storage
• Solar-assisted heat pumps
• Heat pumps in district heating and cooling networks

2.2.1. Heat Pumps Combined with Energy Storage

Regarding heat pump systems combined with energy storage, this is divided into thermal energy storage (TES) or electrical energy storage (EES). Integrating a heat pump system with TES can provide multiple benefits, such as increased energy efficiency, reduced energy costs, enhanced energy security, and improved flexibility. The incorporation of TES with heat pump systems decouples heat production from its use, allowing high-power heat generation to be shifted to off-peak hours [6]. TES systems can be classified according to various criteria, such as the temperature level of the stored energy (hot or cold storage), the storage duration (short-term or long-term), and the mechanism applied for energy storage (sensible heat storage, latent heat storage, chemical reaction heat storage) [7]. Beyond thermal energy storage, electrical energy storage also provides numerous benefits in reducing energy costs and enhancing the flexibility of a heat pump system. A common form of electrical energy storage is the use of batteries to store electricity generated from photovoltaic (PV) systems. In this way, a Battery Energy Storage System (BESS) can supply power not only to the heat pump but also to other loads, such as lighting, in a residential building. Since PV systems play a significant role in the operation and efficiency of heat pumps, this study briefly refers to solar-assisted heat pump systems.

2.2.2. Solar Assisted Heat Pump Systems

Solar-assisted heat pump systems lead to a significant reduction in their energy consumption and achieve a satisfactory level of energy independence from the national electricity grid, thereby contributing to the overall reduction in primary energy consumption and CO₂ emissions [8]. In solar-assisted heat pump systems, the solar assistance usually comes from PV panels, solar thermal collectors, or PV/T collectors. In recent years, the installation of photovoltaic systems integrated with heat pumps for heating and cooling has gained significant popularity, not only in residential buildings but also in commercial and other applications. The efficiency of these systems increases when combined with a BESS; however, the initial installation cost also rises sharply due to the high cost of purchasing batteries. Regarding heat pump systems combined with solar thermal collectors or PV/T, they can be classified into two main types: Direct Expansion (DX) and Indirect Expansion (IDX) systems [9]. In DX systems, the solar collector functions as the evaporator of the heat pump, with the working fluid in the collector acting as the refrigerant. Finally, the combination of different solar collectors with thermally driven cooling technologies also attracts significant research interest, as these systems can efficiently utilize low-temperature sources such as solar energy, while exhibiting lower power consumption compared to conventional vapor compression cooling systems, since they do not require a mechanical compressor [10].

2.2.3. Heat Pumps in District Heating and Cooling Networks

District energy can significantly contribute to lowering carbon emissions in heating and cooling by maximizing energy efficiency, integrating renewable energy sources, and facilitating sector coupling, according to the Heat Roadmap Europe [11]. In recent years, sustained research efforts have been directed toward developing new technologies for district network systems. The evolution of district heating and cooling systems is commonly characterized by a series of “generations,” where each successive generation attains improved efficiency through the incorporation of advanced technologies, as shown in Figure 4.

In the past, heat in district heating networks was produced from industrial processes or from combined heat and power (CHP) plants, where waste heat from the combustion of fossil fuels was utilized by the district heating system. Nowadays, as the supply temperatures to end users are lower, the use of heat pumps plays a crucial role. Overall, heat pumps can contribute to the more sustainable development of district heating systems while increasing their operational flexibility, enabling them to better adapt to fluctuating demand and the integration of renewable energy sources. The level of flexibility in district networks is largely determined by the control strategies applied within the system [13]. The exploration of various potential strategies is a major focus of both the research and academic community, as will be illustrated in the following chapters of this paper.

2.3. Flexibility and Demand Response Capabilities of Heat Pumps

In order to achieve the decarbonization targets of the building stock and beyond, the penetration of heat pumps is considered essential. However, electrifying all buildings could significantly increase electricity demand and introduce mismatches between supply and demand, complicating grid management. Therefore, investigating how such systems can provide flexibility to the grid, facilitating the integration of renewable energy sources, represents an important research issue. An effective measure to address this issue is the demand response (DR) mechanism [14]. The DR mechanism falls under demand-side management and allows the adjustment of energy consumption according to the requirements of the grid, utilizing the variability of power and capacity of energy systems. The integration of DR technologies can help stabilize the grid by providing services such as load shedding and demand redistribution. Heat pump systems are capable of providing demand response (DR) services to the power system since their electricity consumption is inherently flexible [15]. For residential heat pumps in particular, the deployment of suitable control schemes and communication links between the heat pump, the building energy management system, and the power grid is essential. The main factors affecting the flexibility of heat pumps are the thermal demand, the size of the heat pump, the storage capacity, and the dynamic properties of the system [16].

An important role in reducing real-time imbalances in the electricity grid is expected to be played by advanced control strategies for heat pump systems [17]. The development of artificial intelligence algorithms for the control and optimization of these systems has become a key area of current research. The following chapters focus on modern control and optimization methods for heat pump systems within the context of digitalization and AI.

3. Control Approaches for Heat Pump Systems

For the optimization of heat pump systems, optimal control is essential in combination with reliable system modeling and characterization of the parameters affecting it, utilizing appropriate optimization techniques. System control strategies, whether conventional or advanced, aim to ensure system stability and improve energy efficiency, increasingly leveraging the benefits of artificial intelligence. This chapter presents control methods for various energy systems with a focus on heat pumps and provides an extensive literature review of these approaches. Emphasis is placed on modern and highly promising methods that integrate artificial intelligence to achieve greater energy efficiency and flexibility in heat pump systems. Initially, the essential concepts for each control system are introduced.

The control of a system consists of two subsystems: a controlling system and a controlled system. They are also referred to as the controller and the plant, respectively. Furthermore, control is categorized as closed-loop control (CLC) when the controlling system receives feedback from the controlled system. Conversely, it is termed open-loop control (OLC) if no such feedback is received. This feedback is generated by the input control signal sent to the controlled system and typically reflects the result of the optimization performed [18]. Various control strategies for heat pump systems can be grouped into three main categories [19], as illustrated in Figure 5.

3.1. Classical Control

The main strategy applied in the classical control approach is Rule-Based Control (RBC). The RBC strategy is a heuristic method that has been extensively used for the control of heat pump systems. This strategy is based on a set of if–then rules, such as, for example, turning on the heat pump when the temperature of a heated space falls below a predetermined setpoint. The RBC strategy is widely used in building applications, often in combination with PID controllers. RBC techniques typically operate at the supervisory level to manage overall operational strategies, while PID controllers handle local HVAC components and internal building dynamics [20]. In such systems, significant time response delays are observed in PID control compared to strategies with adaptive and predictive control capabilities. RBC techniques also lack continuous adaptation to changing conditions [21]. Consequently, these control systems are unable to provide optimal real-time control for HVAC systems, as their operation is influenced by numerous factors, such as external weather conditions, building occupancy, and equipment variations, which are often uncertain or difficult to predict based on predefined rules [22]. These limitations highlight the need for extensive research into the development of advanced control strategies, such as model-based and model-free approaches, particularly with the assistance of artificial intelligence. However, under the RBC framework, focus should be given to the development of predictive RBC (PRBC), which can potentially strike a balance between the higher computational demands of advanced control strategies and the advantages offered in terms of energy savings for heat pump systems. In this context, studies have been conducted highlighting the usefulness of PRBC in enhancing the flexibility of residential heat pump systems and demonstrating the significant economic benefits achieved by reducing electricity consumption by heat pumps [23,24]. Notably, the benefits of using PRBC were also observed in an office building, both from an environmental perspective, with a consequent reduction in CO₂ emissions, and from an economic standpoint, with a recorded decrease in heating costs [25]. However, studies that combine the classical RBC strategy with artificial intelligence algorithms are of particular interest, such as the study by Hu et al. [26], where an ANN is employed to predict the building’s load and energy usage, and is combined with a rule-based control optimization model to improve the performance of the HVAC system. Similarly, the study by Meng et al. employed the RBC strategy combined with LSTM (Long Short-Term Memory) for energy load forecasting and the implementation of a demand response peak regulation strategy for the winter [27]. Finally, Betzold et al. [28] conducted a comparative study of a PRBC and an MPC for space heating and DHW provision in a residential building, with results showing that MPC offers a comparative advantage in terms of cost savings. Thus,

Table 1. Summary of reviewed studies on Predictive Rule-Based Control (PRBC) in heat pump systems.

Ref.	Control Approach	HP Use	Application Field				Objective
			R	NR	DHC	I
[23]	PRBC	SH and DHW	✓				Reduce electricity costs and greenhouse gas emissions from the heat pump’s electricity consumption
[24]	PRBC	SH, SC	✓				Reduce electricity costs
[25]	PRBC	SH		✓			Reduce heat energy costs and CO2eq emissions
[26]	PRBC	SH, SC		✓			Minimize operating costs
[27]	PRBC	SH, SC	✓				Peak shaving
[28]	PRBC-MPC	SH and DHW	✓				Optimize HP-TES operation for PV self-consumption and cost savings

Abbreviation—R: Residential, NR: Non-Residential, DHC: District Heating/Cooling Network, I: Industrial.

summarizes the aforementioned studies.

However, the limitations described earlier and the rapid development of artificial intelligence have led to the emergence of advanced control strategies, which have received increasing attention from the scientific community in recent years. The following section presents the key elements that constitute a model-based control approach, as well as a number of relevant studies on its application in heat pump systems.

3.2. Model-Based Control

In a model-based approach, controller design relies on the dynamic model of the system being controlled. The most widely used model-based control technique is Model Predictive Control (MPC), which incorporates system constraints and carries out optimization using predictive system behavior. As shown in Figure 6, a typical structure of a Model Predictive Controller. An MPC consists of a prediction model and an optimizer. At each time step k, the prediction model uses disturbance forecasts (e.g., internal loads, dynamic prices, etc.) to estimate the future system states at every step of the prediction horizon. The optimizer, taking into account the objective function, constraints, and the model predictions, determines the sequence of optimal control actions, of which only the first is applied to the real system. These control actions are implemented in the real system, which is also subject to external disturbances. The system returns the measured outputs to the MPC, allowing the control process to be continuously updated.

The optimization problem that the MPC optimizer is required to solve at each time step depends on the nature of the controlled system and the defined cost function. A common approach for classifying optimization methods is to distinguish between mathematical programming methods and metaheuristic methods [29], as illustrated in Figure 7.

Mathematical programming employs various numerical techniques to solve optimization problems, and the choice of the appropriate method depends on the functions and dynamics of the system being modeled. The processes and components of a system may be linear in theory; however, in practice, complex systems such as heat pumps are typically nonlinear. Linear algorithms are generally simpler and faster to implement, but their formulation requires careful consideration, as approximating nonlinear behavior can affect the accuracy. Consequently, nonlinear systems are often represented using linear algorithms to obtain practical and effective solutions. In general, the main techniques used for solving optimization problems in energy control systems include Linear Programming (LP) [30], Mixed-Integer Linear Programming (MILP) [31], Mixed-Integer Quadratic Programming (MIQP) [32], Mixed-Integer Nonlinear Programming (MINLP) [33], and Dynamic Programming (DP) [34]. Moreover, nonlinear optimization methods are also required when the objective function includes nonlinear terms, such as the nonlinear formulation of the heat pump COP. For this purpose, metaheuristic methods, such as Particle Swarm Optimization (PSO) [35] and Genetic Algorithms (GA) [36], are often employed. In general, the metaheuristic approach is a higher-level process that seeks the optimal solution through trial and error. These methods provide relatively fast solutions with low computational effort. However, they are often approximate solutions [29].

Generally, MPC has been an especially popular control strategy in recent years, as it offers the capabilities to model and counteract disturbances, control varying operating points under time-dependent system dynamics, and consider multiple optimization variables with different priorities. As evidenced by the conducted literature review, MPC has been applied in a wide range of heat pump system applications, from building applications to industrial systems, as well as district heating and cooling systems.

The results from the application of MPC in building-related applications, such as office buildings, are particularly encouraging [37,38,39], as they demonstrated that significant energy cost savings can be achieved in offices [37], a notable reduction in CO₂ emissions while maintaining thermal comfort conditions within the spaces [38], and also substantial benefits in heat pump flexibility with the grid, as the approach by Tang et al. [39] showed that the proposed MPC not only provides demand response services by exploiting the energy flexibility of the heat pump and thermal energy storage system but can also operate safely and efficiently to deliver adequate domestic hot water.

Thereafter, the implementation of MPC has shown particularly promising results in enhancing the energy flexibility of residential buildings [40,41,42], thereby enabling effective peak load shaving during peak periods. Building upon this concept, studies [43,44,45] applied MPC algorithms to increase the self-consumption of PV-generated energy, consequently reducing the cost associated with heating, cooling, and domestic hot water (DHW) provision. However, with the increasing penetration of heat pumps and other temperature-controlled loads, the frequency regulation capability from the generation side is reduced, making it necessary to develop frequency regulation strategies from the load side. Model Predictive Control (MPC) offers a highly promising approach to addressing these challenges, as demonstrated by studies [46,47].

Especially noteworthy are studies on the integration of MPC in the control of ground source heat pump (GSHP) systems. Since GSHP systems require a high initial investment, the effective optimization of their design and control is fundamental for enhancing their energy performance and reducing the payback period. Various studies have focused on the use of model predictive control [48,49,50,51]. Notably, study [49] utilizes aquifer thermal energy storage (ATES), which is distinguished by its high storage capacity, exploiting groundwater-saturated aquifers. However, long-term operation may reduce the geothermal potential of the soil and cause thermal imbalance. In this context, study [50] focuses on the development of MPC, where the model emphasizes the spatial distribution of temperature variations in the subsurface, while study [51] applies MPC by defining the objective function in a way that also accounts for the long-term effects on the subsurface caused by the boreholes used for the system installation.

Another significant field with strong research interest is the study of district heating and cooling networks. To achieve the targets set for climate neutrality in buildings, heat pumps play a central role in such systems. Consequently, in the optimal control of such systems, MPC plays a critical role, as it can provide significant benefits, such as minimizing energy use by exploiting system flexibility and optimizing district heating and cooling (DHC) network operation while reducing electricity consumption through the flexibility of thermal energy storage (TES), as demonstrated by studies [52,53]. Moreover, the integration of heat pumps with thermal energy storage systems can serve as a key strategy for decarbonizing industrial energy supply, particularly when processes do not operate continuously. In this context, the study by Agner et al. [54] employed MPC to ensure the reliable operation of non-continuous industrial processes under real-world disturbances. Fuhrmann et al. [55] also employed predictive control to prevent potential bottlenecks in energy supply and to ensure the smooth operation of batch processes, thereby demonstrating in turn the value of MPC in maintaining production reliability in industrial batch-process applications.

Furthermore, regarding large-scale heat pumps, it is important to note that in many industrial activities, 30–70% of the input energy is lost as heat. Therefore, utilizing this waste heat can significantly reduce energy consumption and serve as a powerful decarbonization tool. Heat recovery directly reduces fuel consumption, CO₂ emissions, and operating costs, but at the same time further increases the complexity of the process. Also, residual heat usually refers to lower-temperature heat (<100–150 °C) that is harder to reuse but still valuable. Exploiting the benefits of MPC, studies [56,57] have shown not only an improvement in the operational performance of district heating systems utilizing industrial waste heat, but also the ability of MPC to satisfactorily ensure the operational constraints of the systems. However, with the integration of heat pumps into industrial processes, it is also necessary to ensure system stability and correct power distribution [58]. In this context, the study by Rasmussen et al. [59] proposes an MPC-coordinated primary frequency control for both small- and large-scale heat pumps (HPs). Finally, regarding large-scale heat pump systems, notable are the results of a study on a high-power air-to-water transcritical CO₂ air-source heat pump, which demonstrated that MPC can regulate the heat rejection pressure close to its optimal value, with a very small relative error (0.3–0.8%), thereby ensuring optimal heat pump performance [60].

Depending on the specific problem, the targeted objective function, the imposed constraints, and the modeling complexity, MPC can be classified into different types, such as Adaptive MPC, Robust MPC, and Stochastic MPC. Adaptive MPC offers the capability to dynamically adjust model parameters, such as the COP function, ensuring more efficient and flexible real-time control [61,62]. Robust MPC is used when stability and performance criteria must be satisfied despite all possible model variations and noise signals, provided that the uncertainty remains within set limits [63]. This approach often achieves reductions in operating costs and improvements in system reliability, particularly when combined with machine learning techniques for demand or load prediction [64,65]. Finally, Stochastic MPC, which employs a stochastic dynamic model of the process to predict its potential future evolution, also deals with uncertainties—but through probabilistic distributions—and can be applied both to residential systems and to larger-scale energy networks [66,67].

Beyond the various studies highlighting the interest in using MPC in heat pump systems, significant results have also emerged from field studies and different experimental setups. Field studies have managed to achieve reductions in operational costs [68,69] and significant improvements in efficiency [70], thereby generally supporting the further application of predictive control and advancing the technology toward greater emission reductions [71,72]. Similarly, laboratory and experimental studies, including hardware-in-the-loop tests [73,74], achieved significant energy savings, demonstrated the ability of the MPC controller to perform load shifting and exploit system flexibility [75], and were also applied to pioneering systems, such as thermal energy storage integrated into a ceiling with PCM, embedded in an air-source integrated heat pump (ASIHP) system [76].

The challenges faced by all modern energy systems concern both the impacts of climate change and operational parameters that require optimization, such as heat pump noise. For this reason, the present review also presents studies that are expected to play a central role in research over the coming years. In the context of climate change, the design and operation of heat pump systems must take into account that we are moving towards conditions where cooling demand clearly dominates [77]. Within this framework, both simulation studies [36,78,79] and experimental studies [80] examine the application of an MPC controller exclusively for addressing cooling loads. Finally, an area of growing research interest concerns the reduction in noise generated by heat pumps. Both [81] and [82] demonstrate that Model Predictive Control (MPC) can effectively reduce noise emissions from heat pumps while maintaining or improving energy efficiency. These studies highlight the potential of MPC to manage compressor and fan speeds, leveraging system flexibility to minimize acoustic discomfort in building applications.

Table 2. Summary of reviewed studies on Model Predictive Control (MPC) in heat pump systems.

Ref.	Control Approach	HP Use	Application Field				Objective
			R	NR	DHC	I
[37]	MPC	SH, SC		✓			Cost reduction, grid support enhancement via load shifting
[38]	MPC	SH		✓			Minimize the operational costs and minimize the marginal CO2 emissions
[39]	MPC	DHW		✓			Flexibility assessment and flexibility exploitation
[40]	MPC	SH, SC, and DHW	✓				Minimize energy cost with demand flexibility and thermal comfort
[41]	MPC	SH	✓				Heating load shifting and heating costs reduction
[42]	MPC	SH, SC, and DHW	✓				Minimization of thermal energy, electricity costs, and marginal CO2 emissions
[43]	MPC	SH, SC, and DHW	✓				Minimize electricity costs, increase PV self-consumption, and enhance indoor comfort
[44]	MPC	SH and DHW	✓				PV self-consumption maximization, CO2 emissions minimization, economic cost minimization, and final energy use minimization
[45]	MPC	SH and DHW	✓				Reduction in operational costs and increase in photovoltaic self-consumption through the heat pump
[46]	MPC	SH	✓				Minimization of energy costs and CO2 emissions
[47]	MPC	SH	✓				Low-frequency load shedding control
[48]	MPC	SH		✓			Εnsuring thermal comfort while minimizing energy consumption
[49]	MPC	SH		✓			Optimization of the operation of a geothermal heat pump with an ATES system.
[50]	MPC	SH		✓			Sustainable and cost-efficient operation of ATES systems
[51]	MPC	SH	✓				Optimization of a GSHP system taking into account the long-term effects occurring in the borefield.
[52]	MPC	SH, SC			✓		Minimization of energy use by exploiting system flexibility
[53]	MPC	SH, SC			✓		Optimize DHC network operation and reduce electricity consumption via TES flexibility
[54]	MPC	Provision of heating and cooling to process streams				✓	Ensure reliable operation under real-world disturbances
[55]	MPC	Provision of heating				✓	Minimization of energy cost in thermal batch processes, while simultaneously ensuring production reliability
[56]	MPC	Heating, Data Center Cooling			✓		Minimization of the total energy cost
[57]	MPC	SH			✓		Optimization of the heat pump’s operational efficiency
[59]	MPC	SH			✓		Minimize the frequency deviation from the nominal frequency
[60]	MPC	Provision of hot water				✓	Optimize the energy performance and operation of a transcritical CO2 air-source heat pump
[62]	Adaptive MPC	DHW	✓				Ensure effective and adaptive system control
[64]	Robust MPC	SH			✓		Frequency regulation and flexible heating operation
[65]	Robust MPC	SH, SC		✓			Reducing energy cost while ensuring thermal comfort
[66]	Stochastic MPC	SH and DHW	✓				Enhance power-to-heat flexibility under uncertainty in electricity prices, weather, and occupancy patterns
[67]	Stochastic MPC	SH, SC, and DHW			✓		Reduce energy costs
[68]	ΜPC	SH, SC, and DHW	✓				Minimize electricity costs through thermal storage flexibility
[69]	MPC	SH, SC		✓			Energy savings and improvement of thermal comfort conditions
[70]	MPC	Heating, Cooling				✓	Optimal scheduling of heat pumps and thermal storage units
[71]	MPC	SH	✓				Reduce the house’s total annual heating costs
[72]	MPC	SH, SC		✓			Reduce HVAC energy consumption
[73]	MPC	SH	✓				Minimization of heat pump power consumption within comfort constraints
[74]	MPC	SH, SC	✓				Utilize the technical flexibility of a building polygeneration system
[75]	MPC	SH, SC, and DHW	✓				Minimize operational costs and marginal CO2 emissions
[76]	MPC	SH, SC	✓				Load shifting and minimizing energy costs
[78]	MPC	SC	✓				Trade-off between thermal comfort and energy costs
[79]	MPC	SC			✓		Minimize electricity consumption from the grid by exploiting the building’s energy flexibility
[36]	MPC	SC		✓			Minimize backup energy consumption while satisfying the cooling demand
[80]	MPC	SC			✓		Reduce the operating costs of the district cooling network
[81]	MPC	SH	✓				Minimizing the noise nuisance generated by heat pumps
[82]	MPC	SH	✓				Improve energy efficiency and reduce noise emissions

Abbreviation—R: Residential, NR: Non-Residential, DHC: District Heating/Cooling Network, I: Industrial.

summarizes all the aforementioned studies.

Despite the extensive research conducted on the use of MPC in heat pump systems, its implementation remains limited. First, the development of such models is a laborious process, as these approaches rely heavily on the mathematical models of the system under consideration. The development of accurate models is demanding due to their complexity and nonlinear characteristics [19]. Data-driven MPC attempts to address this issue; nevertheless, it remains a challenging task for MPC-based operation. Moreover, MPC-based control is a computationally demanding process, as it requires solving the optimization problem at each control step. This often prevents its real-time implementation, especially in more complex systems. Thus, given the continuously growing research activity in the field of artificial intelligence, model-free control approaches for heat pump systems have gained significant attention because of their flexibility and their ability to adapt to complex and dynamic systems without relying on explicit mathematical models.

3.3. Model-Free Control

The model-free control approach can implement control strategies in a system without the need to develop its mathematical model, as it interacts with the environment and adapts to changes in the system’s dynamics. The main categories observed in the literature for controlling heat pump systems are Neural Networks, Reinforcement Learning, Fuzzy Logic, and Extremum Seeking Control.

3.3.1. Neural Network Control

Neural networks represent an attractive option for controlling complex and nonlinear systems, such as heat pump systems. Control via neural networks is considered a model-free approach, as they learn directly from data and dynamically adapt to capture the underlying system dynamics without requiring explicit knowledge of its internal workings [19]. The general structure of a neural network consists of interconnected neurons, organized into three layers: the input layer, the hidden layer, and the output layer. The outputs (control commands) are calculated based on the system inputs, the network weights, and the network transfer function [83]. The training of neural networks involves adjusting the internal parameters in order to minimize a function that quantifies the difference between the predicted actions and the truly optimal control actions. A typical structure of a neural network controller is shown in Figure 8, which illustrates the control of the optimal mass flow rate in a solar system for domestic hot water supply in a residential building.

The accuracy of the actions performed by a neural network depends on its structure, the number of neurons, and its training algorithm. Thus, there are various neural network structures, with the most used architectures for controlling heat pump systems being recurrent neural networks (RNNs) such as LSTMs and feedforward neural networks (FNNs). One of the main challenges faced by neural networks and other model-free approaches is that the learning process requires a significant volume of reliable data, which is frequently difficult to collect in real-world settings.

Nevertheless, neural network controllers can significantly contribute to reducing the operational energy costs of heat pump systems and enhance their flexibility in managing electricity consumption. Particularly in residential buildings, they allow the adaptation of heat pump operation to changing external conditions, ensuring that energy consumption dynamically responds to actual demand, resulting in significant improvements in the performance of a designed heat pump system [85,86,87].

3.3.2. Reinforcement Learning Control

Research on the application of Reinforcement Learning for heat pump system control has been increasingly active in recent years. Reinforcement Learning (RL) is a subfield of Machine Learning (ML) and represents a learning method in which an agent learns to make decisions based on the current state of the environment by trying different actions within it. The agent receives feedback in the form of rewards for each action it performs. Its goal is to learn which actions, in each state, lead to the best outcomes in order to maximize the overall reward in the long term. An illustrative example of applying RL to a heat pump system is shown in Figure 9, where a residential heat pump system with a hot water storage tank (environment) is controlled by a central controller (agent). The agent’s actions relate to the decisions taken by the agent to control the environment, such as the supply temperature to the heating radiators. The states feed the agent and are retrieved from the environment. Typical examples include the outdoor air temperature and the COP of the heat pump. Finally, the reward represents the evaluation of the environment by the agent based on the action taken and may include a penalty for failing to achieve thermal comfort conditions.

Other key components of RL include the value function, the policy, and the model of the environment [88]. The value function of a state estimates the total expected reward that can be accumulated in the future starting from that state, indicating how “beneficial” or desirable the state is in the long term. The policy determines the agent’s decision-making process, defining the chances of selecting each action in a given state. Finally, the environment model is designed to replicate the behavior of the environment. However, the existence of a model is optional and is used only in Model-based RL approaches. In the Model-free RL approach, the agent learns through trial and error by interacting with the environment, without requiring a model of the environment. It operates by having the agent learn the best actions for each state, without establishing probabilities for transitions between states [88]. The Reinforcement Learning (RL) problem is typically formulated based on Markov Decision Processes (MDPs), which are mathematical tools used to describe the interaction between the agent and the environment, as well as how the agent’s actions are rewarded [61]. Furthermore, there are three primary approaches for executing RL: value-based, policy-based, and actor-critic. In value-based reinforcement learning, the key aim is to determine the value function. In the policy-based approach, the main objective is to optimize the policy that the agent uses to determine its actions, while the actor–critic approach combines both methods: the actor learns to select actions that maximize rewards, and the critic evaluates the chosen actions using the value function. Finally, it should be noted that many applications in studies concerning the control and optimization of heat pump systems implement Deep Reinforcement Learning (DRL). DRL is a modern Machine Learning method that combines the ability of Deep Learning to handle nonlinear states with the reliable decision-making capability of Reinforcement Learning [88].

In the literature, Reinforcement Learning is the most widely used model-free control approach. Numerous studies, including simulations as well as experimental and laboratory setups, have been carried out to highlight the effectiveness of this approach. The majority of studies focus on small-scale heat pump systems, primarily in residential buildings. RL-based control plays a key role in the intelligent management of such systems, offering energy savings and enhancing their flexibility, as it can shift heating loads to periods where the heat pump can operate most efficiently, as demonstrated in the study by Rohrer et al. [89]. Additionally, it contributes to the prevention and protection against decreased efficiency or even faults, as demonstrated by the study of Klingebiel et al. [90], who developed a self-optimizing defrost start controller using DRL. Significant energy savings have been reported in previous studies [91,92] on residential heating. Furthermore, the application of reinforcement learning (RL) control has demonstrated notable advantages, including enhanced self-consumption, increased energy flexibility, and improved energy efficiency while maintaining thermal comfort, in the context of residential heat pump systems integrated with photovoltaic (PV) [93,94,95,96,97] or even photovoltaic-thermal (PVT) systems [98]. Significant research has also been conducted in non-residential buildings, such as university campuses [99,100,101], as well as in larger-scale energy systems, including district heating networks [102,103] and the industrial sector [104]. Heat pump technologies that utilize waste heat [105,106,107] are of particular interest for improving system performance, promoting their faster adoption, and replacing energy generation units based on fossil fuels. The control of these systems using DRL significantly contributes to accelerating their deployment. Also, a notable study in the international literature is the study by Diaz et al. [108], which applied an RL-based control approach to optimize the operation of an absorption solar cooling system.

Finally, experimental studies also have an important role in demonstrating the efficiency and operational optimization of systems through Reinforcement Learning (RL)-based control. In the present review, several experimental studies published in recent years are highlighted, such as the study by Klingebiel et al. [109], which focused on the development of a self-optimizing defrost controller for air-to-water heat pumps (ASHPs). The study was conducted on a hardware-in-the-loop laboratory setup, where the RL agent demonstrated high control performance in this experimental environment. In the same context, studies [110,111,112] also confirmed the applicability and effectiveness of RL or DRL controllers in heat pump systems. However, the study by Boutahri et al. [110] is of particular interest, as the results obtained from the real-world case study showed smaller performance gains compared to those achieved in the simulations.

3.3.3. Fuzzy Logic Controller

Another control approach that has attracted considerable research interest is Fuzzy Logic Control. Fuzzy logic control is a model-free approach, as it does not require a mathematical model of the controlled system but relies on human reasoning and a linguistic model, applying simple mathematics in cases of nonlinear and complex systems [83]. Fuzzy Logic Controllers (FLCs) use fuzzy sets to handle imprecise and uncertain data. Fuzzy sets are defined through membership functions, which map input values to a degree of membership within the range [0, 1]. An FLC is based on a predefined rule base, usually constructed from empirical knowledge or historical data. Such rule bases may include, for example, rules like “If the room is cold, then activate heating”. In general, the typical structure of an FLC is illustrated in Figure 10 and consists of fuzzification, rule base, inference engine, and defuzzification [19].

Specifically, during fuzzification, crisp input data are transformed into fuzzy sets, while the rule base contains a set of “if–then” rules that define the controller’s behavior. Next, the inference engine, taking the crisp inputs and the rule base as input, calculates the fuzzy output set. Fuzzy inference systems are generally divided into two types: Mamdani and Sugeno. The Mamdani type is typically used for all kinds of systems, whereas the Sugeno approach is applied mainly to dynamic, nonlinear systems. Finally, the last step is defuzzification, where the fuzzy output sets are converted into crisp control commands. As conditions change, the system continually evaluates the rules, ensuring efficient operation. However, the frequent need for expert knowledge to define rules and membership functions in Fuzzy Logic Control (FLC) highlights the importance of integrating fuzzy logic with other methods. A relatively common and effective solution is the Adaptive Neuro-Fuzzy Inference System (ANFIS), which merges the learning abilities of artificial neural networks with the interpretability and flexibility of fuzzy logic, enabling enhanced system performance and smoother adaptation to optimal operating conditions.

Encouraging findings regarding the use of FLC for heat pump systems emerge from the present literature review. An increase in the energy efficiency of such systems is observed both in heat pump systems providing heating and cooling to buildings [114,115] and in other applications, such as the study by Rodríguez-Valderrama et al. [116] on a domestic refrigerator, the study by Chuensiri et al. [117] on a Hybrid Ground Source Heat Pump (HGSHP) system, and the study by Kim et al. [118] on a solar geothermal hybrid heat pump system. Finally, experimental studies on the application of FLC control in heat pump systems appear to be limited. However, the results from the experimental work of Langner et al. [119] indicate that cost reduction and the maintenance of thermal comfort can be effectively achieved through the use of such a model-free control approach.

3.3.4. Extremum Seeking Control

Finally, another commonly used control approach is Extremum Seeking Control. Extremum Seeking Control (ESC) is a real-time, adaptive, model-free control strategy capable of adapting to a system’s unknown dynamics [2]. ESC is a dynamic gradient-search method. Using pairs of dither-demodulation signals along with proper filtering, the gradient estimation is performed in real time. As shown in Figure 11, which illustrates a simple form of an Extremum Seeking Control, the basic idea is to introduce a signal to the system. Then, by observing and acting on the output of the performance function, it becomes possible to estimate the gradient, which can be used by a state regulator (e.g., an integrator) to iteratively search for an extremum point.

An important advantage of using ESC is that the optimum search process is linked to the dither frequencies, making it nearly independent of process variations, external disturbances, and measurement noise at other frequencies. However, the convergence to the optimal operating point using the classical perturbation-based ESC scheme may occur at a rate where the time scales are slower than the process dynamics. A common approach to address this slow convergence is the development of a proportional–integral ESC (PI-ESC). The final category of model-free control discussed in the paper is Extremum Seeking Control (ESC).

From the literature review, it was found that most studies investigating ESC in heat pump systems primarily focus on high-temperature heat pumps [121,122,123]. Gong et al. [121] use ESC to achieve a reduction in power consumption of a high-temperature vapor-injection heat pump system. Liu et al. [122] implemented a multivariable ESC for a high-temperature heat pump system with double refrigerant injection technology, while Wang et al. [123] applied ESC to a Cascade Heat Pump system (CASHP) aiming to minimize power consumption and maximize COP, achieving satisfactory convergence under both steady-state and transient conditions, thereby confirming the feasibility of the proposed control strategy. Finally, beyond the application of ESC in high-temperature heat pump systems, there is significant interest in its use for residential heat pumps, as shown by Vering et al. [124], aiming to optimize both energy performance and acoustic behavior. Simultaneous energy and acoustic optimization is expected to attract considerable research attention, as, according to a study [124], the acoustically optimal configuration does not coincide with the energetically optimal one.

Table 3. Summary of reviewed studies on model-free control in heat pump systems.

Ref.	Control Approach	HP Use	Application Field				Objective
			R	NR	DHC	I
[85]	ANN	SH, SC		✓			Increase the system performance
[86]	ANN	SH	✓				Minimize the electricity cost with a time-variable electricity tariff
[87]	ANN	Cooling	-	-	-	-	Performance optimization
[89]	DRL	SH	✓				Shift the heating to low price periods
[90]	DRL	SH	✓				Maximize heat pump performance through autonomous defrost optimization.
[91]	DRL	SH	✓				Minimize energy cost while considering thermal comfort
[92]	RL	SH	✓				Minimize electricity costs
[93]	RL	SH and DHW	✓				Save energy while maintaining the health and comfort of occupants.
[94]	RL	SH and DHW	✓				Μaintain the required temperatures in the thermal storage and enhance PV self-consumption
[95]	DRL	SH, SC, and DHW	✓				Minimize energy costs and maximize PV self-consumption in a home energy management system
[96]	DRL	SH and DHW	✓				Minimize energy consumption and shift loads while maintaining thermal comfort
[97]	DRL	SH and DHW	✓				Optimize the trade-off between energy consumption cost and comfort of living
[98]	RL	SH and DHW	✓				Minimize the operating costs
[99]	RL	SC		✓			Increase self-consumption, energy flexibility, and energy savings while maintaining thermal comfort
[100]	DRL	SH, SC		✓			Reduce heat pump energy consumption
[101]	DRL	SH, SC		✓			Control optimization
[102]	DRL	SH and DHW			✓		Enhance energy efficiency and minimize unnecessary heating
[103]	DRL	SH			✓		Reduce the electrical energy consumption while maintaining thermal comfort
[104]	Rainbow DRL	Greenhouse Heating				✓	Minimize electricity costs
[105]	RL	SH, SC		✓			Minimize energy consumption and power fluctuations while maintaining thermal comfort in multi-zone building HVAC systems
[106]	DRL	Heating		✓			Optimization of energy use
[107]	DRL	Heating, Cooling			✓		Reduction in energy consumption and maintenance of the annual thermal balance of the Aquifer TES
[108]	RL	Cooling				✓	Optimize the operation of solar thermal cooling systems (STCS)
[109]	DRL	SH	-	-	-	-	Heat pump defrost control
[110]	RL	SH	✓				Improves indoor comfort while optimizing energy use
[111]	RL	SH, SC		✓			Optimize heat pump RL control
[112]	DRL	SH, SC			✓		Minimizing energy costs while maintaining occupant comfort
[114]	FLC	Heating/Cooling	-	-	-	-	Enhance energy efficiency and temperature control
[115]	FLC	SH		✓			Βalance between thermal comfort and electricity load reduction
[116]	FLC	Domestic refrigerator	-	-	-	-	Reduce energy consumption
[117]	ANFIS	Heating/Cooling	-	-	-	-	Control the water flow rate with a Variable Speed Drive (VSD) to improve overall system performance
[118]	FLC	SH, SC				✓	Minimize energy costs
[119]	FLC	SH	✓				Minimize energy costs while maintaining thermal comfort
[121]	ESC	Heating				✓	Reduce power consumption
[122]	ESC	Heating				✓	Reduce power consumption and optimize system operating point
[123]	ESC	Heating		✓			Minimize power consumption
[124]	ESC	Heating	-	-	-	-	Optimize heat pump operation for energy efficiency and noise reduction

Abbreviation—R: Residential, NR: Non-Residential, DHC: District Heating/Cooling Network, I: Industrial.

summarizes all the aforementioned studies based on model-free control approaches. It should be noted that in some studies, the application field of the investigated system is not specified.

In summary, model-free control approaches rely solely on data and do not require an accurate mathematical model of the system. However, they require a large volume of data to train their algorithms. With the increasing research activity around sensors and IoT devices, the amount of available data for training model-free control approaches is expected to grow significantly. However, two major challenges that remain in the operation of these approaches are: the difficulty of enforcing constraints during system operation and the limited interpretability of the decisions they sometimes produce. The next section presents recent trends and studies that aim to combine the advantages of model-based and model-free approaches in order to improve the performance, safety, and flexibility of control systems.

3.4. Recent Advances in Predictive and Learning-Based Control

As shown in the literature review of model-free control approaches, the dominant technique, both in simulations and experimental studies, was Reinforcement Learning (RL). Among the main advantages of model-free RL were its ability to handle complex systems as well as its high adaptability. However, a challenge lies in the extensive data training required by RL and its dependence on sample efficiency. An alternative approach for accelerating learning and promoting the application of RL in heat pump systems is model-based RL. With the incorporation of a model into RL, the dependence on real-world interactions is reduced, and the convergence speed is improved [125]. In this context, an important comparative study of model-free and model-based RL control was presented by Gao et al. [126], where they compared the performance of model-free and model-based RL for heat supply from a heat pump in a residential building. The comparison was also made against a baseline control strategy, and it was shown that both RL control approaches outperformed it. Notably, model-based RL achieved almost the same performance as model-free RL but with a shorter training time due to its higher sample efficiency. Additionally, field tests for the application of model-based RL in a residential heat pump system were presented by Soares et al. [127], where the model-based RL control approach contributes to the maximization of instantaneous self-consumption of local photovoltaic generation. Nevertheless, model-based RL also has limitations due to the need for accurate system modeling.

Model Predictive Control (MPC) is also a well-established approach, with numerous studies conducted both at the simulation level and through experimental investigations. Despite several experimental studies on heat pump systems, MPC has not been widely adopted in practice, mainly due to the high requirements for data infrastructures as well as the lack of know-how among practitioners. A solution that addresses these issues is approximate MPC. The goal of approximate MPC is to find explicit control laws by approximating the input–output relations of the optimal controller (teacher). The logic of approximate MPC is based on the process of imitating the optimal controller, which is usually carried out by applying machine learning algorithms. These models substitute the MPC optimization and replicate its outputs for the same inputs. In this way, a new controller is obtained, consisting of the trained model, known as the approximator, and consequently, no optimization during closed-loop operation is required [3]. As shown in Figure 12, the development and evaluation of an approximate MPC starts with the creation of the MPC controller to control the system. Next, the controller is applied to the system, and the system’s operational data are collected as input/output data. These input/output data are then used to train the approximator. The most popular approximators in the literature are decision tree-based algorithms and artificial neural networks (ANN). Finally, the approximate MPC controller is implemented in the system, and its performance is evaluated [128].

In recent years, there has been growing research interest in the development of approximate MPC (approximator MPC) for heat pump systems. For example, Maier et al. [129] investigated the use of random forest (RF) algorithms to extract control rules from optimal MPC strategies in heat pump systems with storage, successfully mimicking the MPC performance while also improving the comprehensibility and practical applicability of the rules. Löhr et al. [130] applied an approximate MPC using neural networks for a heat pump system with thermal energy storage to provide heating and domestic hot water (DHW) in a building, while Maier et al. [131] implemented an approximate MPC for a hydronic heat pump system based on the BOPTEST framework, performing a comparison of various advanced approximators, specifically between artificial neural networks (ANNs), random forest (RF), linear, and logistic regression. The study demonstrates that the feature selection of the approximator significantly affects performance, while also showing the superiority of ANNs and RF.

Although approximate MPC requires less sophisticated hardware and software since the control laws are explicitly computed through model inferences, it nevertheless presents a significant drawback, which is its reliance on datasets collected from closed-loop experiments. A new control method combines machine learning techniques with MPC, aiming to reveal system behaviors that cannot be extracted from collected data, while also eliminating the need for a complete redesign of the system, which MPCs often face. A particularly promising combination is that of RL with MPC, where MPC can function as a function approximator, in which the actions are explicitly imposed on the model during the prediction horizons. This new control method is called differentiable MPC [20]. The differentiable MPC has the capability to optimize the model parameters end-to-end. This specific method was also adopted by Chen et al. [132] for the planning and control of HVAC systems. Another possible combination of RL with MPC focuses mainly on the explicit imposition of the action in the first control step, using the controller model, and employs the RL value function for the remaining prediction horizon. Such a study was presented by Arroyo et al. [133], where they investigated an RL-MPC control combination for providing heating in a residential building. Furthermore, a notable study on the combination of MPC and RL for advanced control of residential heat pumps was presented by Lee et al. [134]. In particular, they employed a Reinforcement Learning-based method, Behavioral Cloning, to model an MPC policy using a neural network. Unlike approximate MPC, this approach allows the controller to generalize across different buildings and electricity tariffs, whereas approximate MPC requires separate training for each building. Finally, also interesting is the approach of Cai et al. [135], who proposed an MPC-based RL approach for home energy management systems, specifically the parameterization of the MPC model, the cost function, and the constraints, and the training of these parameters through RL, with the aim of obtaining an optimal policy that satisfies both economy and comfort. All the aforementioned studies are summarized in

Table 4. Summary of reviewed studies on recent advances in predictive and learning-based control in heat pump systems.

Ref.	Control Approach	HP Use	Application Field				Objective
			R	NR	DHC	I
[126]	MBRL	SH	✓				Enhance energy efficiency and maintain indoor comfort in the HVAC system
[127]	MBRL	DHW	✓				Maximize the instantaneous self-consumption of the local photovoltaic production
[129]	Approximate MPC	SH, SC		✓			Optimize the operation schedule for minimizing annual energy consumption and operational costs
[130]	Approximate MPC	SH, DHW		✓			Satisfy thermal comfort
[131]	Approximate MPC	SH	✓				Evaluate the performance of Approximate MPC for a hydronic heat pump system
[132]	Differentiable MPC	SH, SC		✓			Minimize energy consumption while maintaining occupant comfort
[133]	Differentiable MPC	SH	✓				Minimize operational costs while ensuring thermal comfort
[134]	Behavioral cloning RL	SH, SC	✓				Minimize the time-varying electricity cost while maintaining thermal comfort
[135]	MPC-RL	SH	✓				Minimize operational costs while ensuring thermal comfort

Abbreviation—R: Residential, NR: Non-Residential, DHC: District Heating/Cooling Network, I: Industrial.

.

The potential for applying these methods to different heat pump systems appears particularly high, especially within the ongoing transition toward the digitalization of energy systems. In this context, research in energy systems is increasingly focusing on the development of Digital Twins (DTs). A Digital Twin represents a modern approach for system validation and experimental assessment, bridging the gap between conventional simulations and real-world implementation. Specifically, a Digital Twin consists of three components. The first component refers to the physical system (Physical Twin), the second to the digital representation of the physical system and its continuous operations (Digital Twin), and the last to a bidirectional exchange of data and information, which allows the Digital Twin to be updated and provides feedback to the Physical System [136]. The integration of these advanced control strategies with the capabilities of Digital Twins enables real-time monitoring and simulation of systems, making predictive optimization possible, supporting performance evaluation under different operational and environmental conditions, and facilitating informed decision-making for energy-efficient operation [137].

4. Discussion

This article examines advanced control approaches for heat pump systems, taking into account the demands of the current energy transition. This section summarizes the main general trends identified in the study, aiming to highlight future research directions on the advanced control of heat pump systems.

Specifically, a critical analysis of the findings from this review is conducted, offering a better-grounded overview of prevailing trends, recurring design patterns, and underexplored areas. This critical analysis is carried out through the visualization of the relationships between control strategies, application domains, and corresponding benefits, as illustrated in Figure 13 and Figure 14. In Figure 13, a diagram is presented that directly maps each control type to the dominant performance indicators optimized in each study, allowing for a clear comparison of their research focus.

It should be noted, however, that the results of a study may simultaneously involve multiple performance indicators in the case of multi-objective optimization. From the diagram in Figure 13, it is evident that the primary objective of all problems is the minimization of energy consumption and, consequently, operational costs. Furthermore, there is a greater preference for using MPC to enhance system flexibility, primarily due to its predictive nature, while hybrid MPC-RL approaches have not yet been investigated in a sufficient number of studies to allow for conclusive findings. However, it should be noted that, in contrast to indicators related to cost, energy, and comfort, environmental metrics such as CO₂ emissions do not demonstrate a clear research preference, as energy and cost savings are often directly equated with environmental benefits, without accounting for variations in grid emissions over time, which conflicts with carbon reduction goals. Thus, a pathway opens for studies that will take into account both this variation and the parameter of heat pump noise, which remains relatively unexplored, mainly of course for residential applications, as is also evident in Figure 14.

The second diagram (Figure 14) provides a more detailed representation, incorporating the intermediate level of application domains and revealing how different control approaches are distributed across various building and system environments. As illustrated, the advanced control methods explored to date have been mainly applied in residential settings and have yet to be extended to larger-scale systems, including district heating and cooling networks and industrial applications. It is noted that the N/A category mainly refers to experimental studies, in which the reason for meeting the required load is not sufficiently clarified. Furthermore, from this diagram, it is interesting to observe that applications with model-based control are more widespread in residential and non-residential contexts. The figure also confirms that industrial and district systems are more suited to model-free approaches due to the computational complexity required to model these systems. Additionally, the literature analysis highlights the research opportunities emerging in the field of intelligent control for large-scale and industrial applications, such as solar cooling systems, as well as in district heating systems, particularly with an emphasis on meeting the required thermal comfort conditions. Notably, according to the International Energy Agency (IEA) [138], heat pumps are increasingly emerging as a key technology for utilizing waste heat from data centers in district heating networks, opening new opportunities for energy-efficient and sustainable applications, as also demonstrated by the relevant studies in the review [56,107].

Also, ensuring system frequency (frequency regulation) is a key challenge for a resilient and efficient energy network, particularly as the electrification of energy systems expands. The increased pressure on existing infrastructure and the growing need for flexibility to balance fluctuations in electricity generation are expected to intensify. In this context, energy storage technology (TES) plays a central role in managing the mismatch between supply and demand. According to the review, many studies have examined the use of TES in their research, as illustrated in Figure 15. It is noteworthy that TES usage is particularly prevalent in industrial and DHC network applications, mainly due to the flexibility it provides, especially under batch process conditions. It is important to note, however, that underground thermal energy storage (UTES) differs from conventional thermal energy storage in terms of its dynamics, a factor that must be taken into account when designing an advanced control strategy for each system under study. In UTES, the thermal response of the ground is slow and depends on long-term heat accumulation or extraction, necessitating models with extended horizons and the prediction of thermal imbalance. In contrast, conventional thermal energy storage exhibits more predictable thermal behavior, enabling immediate corrective actions and more accurate short-term optimization. In this context, study [50] is particularly relevant, as the controller is designed so that the model focuses on the spatial distribution of temperature changes in the ground.

However, although the aforementioned control strategies for heat pump systems show highly promising results, they still lack experimental validation, as the collected studies indicate that only 14% are based on experimental data, while the remaining 86% rely on simulation-based studies. The implementation of advanced control methods through IoT devices is accompanied by several technical challenges. The diversity in the characteristics of IoT devices and the effort to integrate them with traditional building management systems lead to significant technical obstacles [139]. Moreover, the effective management of large amounts of data collected from various sensors and ensuring the reliability of the data against loss or tampering are critical requirements [140]. The decisions of AI algorithms for controlling heat pump systems rely on data from IoT networks, which may contain errors due to sensor battery depletion, device malfunctions, or communication loss. Failure to ensure the quality of this data can lead to incorrect decisions and reduced performance of the controlled system [141]. Therefore, for the successful integration of advanced control methods in heat pump systems, the accuracy of the sensors used must be particularly high, which, besides being rarely achieved, also increases implementation costs [69,71]. Specifically, regarding MPC, regular and computationally intensive optimization is required, particularly in real-time building applications where fast response times are crucial [139], while they also face unexpected transient phenomena that are usually not predicted by the model, negatively affecting their efficiency [75]. On the other hand, model-free control strategies require large amounts of data, making extensive and complete datasets necessary to achieve effective learning. Furthermore, the validation of a model in a real environment is critical for evaluating its performance, as it accounts for the real dynamics and complexity of the system during the learning process. However, implementing this validation faces significant challenges, such as limitations in computational infrastructure, security issues, and the need for access to real data [142]. Research efforts are directed towards addressing all the aforementioned issues to enable the successful implementation of advanced control in heat pump systems.

Finally, in this context, as also highlighted by the present review, the metrics related to computation and implementation, such as execution time performance and control loop frequency [143], remain significantly limited, serving as a factor that continues to hinder the widespread application of these control strategies in real-world systems, while it is also worth noting that most research efforts focus on optimizing heat pump systems during their operational phase, overlooking the impact that the initial system sizing and primary design have on their overall performance.

5. Conclusions

This review article provides a comprehensive analysis of the current state of optimal control of heat pump systems within the context of the ongoing energy transition. The findings emphasize the crucial role of heat pumps as a key technology for decarbonizing the heating and cooling sectors, supported by recent technological advancements. Compared to existing reviews, the present study provides a holistic and comparative presentation of advanced control strategies across all energy systems that focus on heat pumps, and it focuses not only on energy performance but also on operational flexibility, renewable energy integration, and demand management challenges. Its goal is to identify significant imbalances, limitations, and research gaps, thereby providing clear directions for the future development and optimization of these systems.

Thus, the study revealed that in the field of control strategies and operational optimization of energy systems with heat pumps, recent research increasingly focuses on methods supported by artificial intelligence algorithms. Model Predictive Control (MPC) is extensively examined; however, its application is often limited by high computational complexity. At the same time, Reinforcement Learning (RL) has emerged in the literature as the dominant model-free approach for advanced heat pump control, despite challenges related to processing large data volumes and enforcing constraints. Recent trends aim to combine the advantages of MPC and RL (e.g., Differentiable MPC, Approximate MPC) with the goal of developing more flexible, effective, and less computationally demanding controllers. A key direction for future research is the integration of advanced control strategies into the simultaneous optimization of the design of each energy system. This approach emphasizes that control and operational strategies should not be considered independently, but should be taken into account from the early stages of system design. In this way, the design can incorporate variable operating conditions as well as the interactions between components from the initial design phase.

The review indicates that significant barriers, such as high computational complexity and the requirements for data and model validation, greatly limit the practical implementation of experimental testing of advanced control methods in heat pump systems. Despite the existing challenges in the practical implementation of advanced control methods, developments in digitalization and artificial intelligence are expected to further enhance the efficiency and sustainability of heat pumps, accelerating their role as a key technology for the decarbonization of energy systems.

Summarizing all the above, and taking into account the points discussed in Section 4 of the discussion, future research directions could focus on the following:

• Integrated Design and Operation Optimization using multi-objective optimization, taking into account the entire lifecycle of the system. This approach considers both energetic and economic aspects, aiming to develop sustainable and efficient energy systems.
• Experimental evaluation of control algorithms in real-world engineering applications, combined with an analysis of their economic viability.
• Further research is needed on advanced control methods for thermally driven heat pump systems and large-scale applications, such as solar cooling systems and district heating/cooling networks.
• In 5th generation district heating and cooling (5GDHC) networks, heat pumps are installed at each consumer to adjust the supply temperature according to specific requirements, whether for heating or cooling purposes. This setup creates significant opportunities for further research, as the energy flexibility and resilience provided through advanced control represent additional benefits that become increasingly important in scenarios with high penetration of distributed renewable energy sources.
• An emerging research direction is the investigation of heat pumps from an acoustic perspective. Evaluating the acoustic impact of heat pumps is crucial, especially in residential settings, where noise emissions can greatly influence the quality of life. Therefore, future research on the control of heat pump systems to optimize their acoustic performance is recommended.

These future research directions can be further supported and validated through experimental studies and practical demonstrations, which provide insights into real-world system behavior and performance. In conclusion, the integration of predictive, adaptive, and learning-based strategies, combined with digitalization and real-time data collection, can significantly enhance the performance, flexibility, and sustainability of heat pump systems.