Stochastic Optimal Control Problem and Sensitivity Analysis for a Residential Heating System

Abstract

We consider a network of a residential heating system (RHS) composed of two types of agents: a prosumer and a consumer. Both are connected to a community heating system (CHS), which supplies non-intermittent thermal energy for space heating and domestic hot water. The prosumer utilizes a combination of solar thermal collectors and CHS heat, whereas the consumer depends entirely on the CHS. Any excess heat generated by the prosumer can either be stored on-site or fed back into the CHS. Weather conditions, modeled as a common noise term, affect both agents simultaneously. The prosumer’s objective is to minimize the expected discounted total cost, taking into account storage charging and discharging losses as well as uncertainties in future heat production and demand. This leads to a stochastic optimal control problem addressed through dynamic programming techniques. Scenario-based analyses are then performed to examine how different parameters influence both the value function and the resulting optimal control strategies. For a common noise coefficient ${\sigma }_0=0.4$, the prosumer incurs an approximate $16.08\%$ increase in the aggregated discounted cost from the case of no common noise. For a discharging efficiency ${\eta }_E={\displaystyle \frac{1}{0.9}}$, the maximum aggregated discounted cost increases by approximately $1.85\%$ as compared to the perfect discharging efficiency. Similarly, for a charging efficiency ${\eta }_E=0.9$, we observe an approximate $1.94\%$ increase in the aggregated discounted cost as compared to a perfect charging efficiency. Furthermore, we derive insights into the maximum expected discounted investment that a consumer would need to make in renewable technologies in order to transition into a prosumer.

1. Introduction

In recent decades, global weather patterns have shifted markedly, largely due to human-driven factors such as extensive fossil fuel use and deforestation, both of which have significantly increased the global carbon footprint. Projections from the United Nations Department of Economic and Social Affairs indicate that, by 2050, urbanization will reach $68\%$, with more than 40 metropolitan areas exceeding 10 million inhabitants [1]. As noted in [1], cities currently account for over two-thirds of global ${\mathsf{CO}}_{\mathsf{2}}$ emissions and energy consumption. These combined pressures—accelerating urbanization, climate change, and rising energy demand—underscore the urgent need to transition away from a business-as-usual economic model toward one that prioritizes sustainability and energy efficiency across all sectors. According to the International Energy Agency (IEA), buildings contribute roughly $30\%$ of global final energy use and $26\%$ of energy-related emissions. Consistent with [2], in the United States and many European countries, 18–30% of total energy consumption is associated with the thermal energy needs of buildings. In this study, we consider a residential heating system (RHS) composed of two agents—a prosumer and a pure consumer (hereafter simply “consumer”)—as illustrated in Figure 1. In this setting, a prosumer is an entity capable of producing and consuming thermal energy, whereas a consumer only uses energy supplied to it. The RHS provides heat for space heating and domestic hot water demands. The prosumer is equipped with a solar thermal collector for local heat generation, an internal storage (IS), and an external storage (ES), both modeled as water-based storage tanks for simplicity [2]. The IS serves short-term storage needs, while the ES accommodates longer-term storage. Because solar production and residual thermal demand are highly intermittent and weather-dependent, additional support is required to ensure a stable and sustainable thermal supply. To address this, both buildings are connected to a community heating system (CHS). In this work, we consider a low-temperature bidirectional CHS (LTB-CHS) [1,3,4,5] and assume, for simplicity, that the temperature in the distribution pipes remains constant. The LTB-CHS is a relatively recent concept in district energy networks, offering benefits such as improved integration of renewable heat sources, simplified coupling with distributed technologies, and reduced carbon emissions. Its bidirectional capability allows buildings to inject surplus heat into the network, lowering peak central production requirements and enabling economic incentives whereby building owners can sell excess heat. Since the supply temperature in an LTB-CHS is lower than that in conventional systems, a heat pump (HP) is required to upgrade the delivered heat to the desired level.

In our modeling approach, daily temperature variations influence both the prosumer and consumer and are represented by a common noise process. Following [2,6,7], we define residual demand as the imbalance between thermal supply and demand. We assume that the prosumer can meet this residual demand either via the CHS or by using the ES, whereas the consumer relies solely on the CHS. When production falls short, the prosumer either purchases heat from the CHS or releases it from the ES. Conversely, during periods of surplus production, the prosumer may store the excess in the ES (subject to capacity limits) or sell it to the CHS. The consumer, lacking local production and storage, always faces unmet demand that must be satisfied exclusively through CHS purchases. Consistent with [2], we assume that the CHS is responsible for fulfilling the residual thermal demands of both agents, even though this demand fluctuates with respect to the common noise. Moreover, energy transfer between the IS and ES, as well as energy exports from the prosumer to the CHS, requires an ordinary pump (OP). The prosumer aims to minimize the expected total discounted cost associated with meeting its thermal demand, while accounting for uncertain weather conditions and variability in solar output. In contrast, the consumer—lacking local production or storage—continuously faces unmet thermal demand that must be supplied entirely from the CHS.

Including both agents in our framework enables us to determine the maximum expected investment that would make it economically reasonable for a consumer to adopt distributed renewable heating technologies and storage infrastructure, thereby transitioning into a prosumer.

Previous work, such as [2], has formulated an RHS model and solved a stochastic optimal control problem using semi-Lagrangian and dynamic programming approaches. In Ref.  [7], the authors addressed a similar problem for a stand-alone system with geothermal storage. However, these studies did not explicitly model the dependence of residual demand on daily temperature variations via a common noise process, nor did they include ES charging and discharging inefficiencies. Our model fills this gap by incorporating these features, which increase realism. Models with common noise are advantageous because they reflect uncertainties that affect all agents simultaneously, potentially altering both qualitative and quantitative system behavior. Applications of common noise models in the energy sector include [6,8,9,10,11,12,13].

Unlike in [2,7], we incorporate ES charging and discharging efficiencies, which account for storage losses when heat is stored or released. For simplicity, we assume that these efficiency factors depend solely on the residual demand, as in [14].

Our contributions include first modeling a consumer connected to the CHS. We also motivate a common noise in the dynamics of the residual demands. This allows us to capture the impact of daily temperature variations on the residual demand. In addition, we include charging/discharging efficiencies in the dynamics of the ES temperature level. In reality, losses are observed during charging or discharging processes of the ES. Therefore, modeling these efficiencies is required. Furthermore, we perform a sensitivity analysis on the common noise, charging and discharging parameters to evaluate their impact on the expected total discounted cost. We also show that the value function is a viscosity solution of the associated partial differential equation (PDE). We derive the discrete-time state-dependent control set from the state constraint without imposing charging and discharging thresholds. In this work, the state-dependent control intervals depend on the charging/discharging efficiencies. We compare the costs of satisfying residual demands for both the consumer and the prosumer, and we estimate the expected investment that a consumer would need to undertake to transition into a prosumer. Finally, we perform extensive numerical experiments and provide economic interpretations.

The remainder of this paper is structured as follows: Section 2 introduces the model, including state variables, control variables, constraints, and the heat price specification. Section 3 presents the optimal control problems for both agents. Section 4 and Section 5 describe the semi-Lagrangian discretization and numerical results, including sensitivity analysis. Section 6 concludes and outlines directions for future research.

2. Model Formulation

In what follows, we study a model that includes both a prosumer and a consumer connected to a CHS. The prosumer is able to generate thermal energy locally while also drawing from, or supplying heat to, the CHS. In contrast, the consumer has neither local production capabilities nor on-site storage units. We present a mathematical framework and corresponding optimization problems for both agents. Throughout this paper, we work on a fixed probability space $(\mathsf{\Omega },\mathcal{F},\mathbb{P})$ endowed with a filtration $\mathbb{F}={\left({\mathcal{F}}_t\right)}_{t\in [0,T]}$ that satisfies the usual conditions and supports two independent standard Brownian motions $W_R$ and $W^0$. The parameter $T>0$ denotes the finite planning horizon.

An RHS is designed to supply thermal energy for both space heating and domestic hot water in a building. The prosumer is equipped with a solar collector, two storage units, a heat pump, and ordinary pumps. Thermal energy generated by the solar collector is first directed to the internal storage (IS) to meet immediate demand, while the external storage (ES) serves as a long-term buffer to handle periods of insufficient or excess production. We describe the imbalance between thermal supply and demand, resulting from stochastic fluctuations in solar collector output, by a stochastic process $R={\left(R\left(t\right)\right)}_{t\in [0,T]}$. This process is decomposed as $R\left(t\right)={\mu }^R\left(t\right)+Z^R\left(t\right)$, where ${\mu }^R\left(t\right)$ denotes the seasonal average and $Z^R\left(t\right)$ represents random deviations from this mean.

The IS is modeled as a cylindrical tank whose water temperature must remain within prescribed bounds, $i_{min}$ and $i_{max}$. In this study, we treat the IS as a black box and do not model its internal thermodynamic behavior explicitly. The ES is also represented as a cylindrical water tank with a fixed mass of stored water, which can be heated up to a maximum temperature $\overline{q}$ through transfers of thermal energy from the IS. Conversely, it can be cooled to a minimum temperature $\underline{q}$ when heat is withdrawn to satisfy the residual demand. For simplicity, we assume that charging and discharging the ES involves only ordinary pumps.

At any time $t\in [0,T]$, the prosumer’s state variables consist of the exogenous, deseasonalized residual demand $Z^R$ and the endogenous ES temperature level Q. We therefore define the prosumer’s state process as $X={(Z^R,Q)}^{⊺}$.

For the consumer, the state is given solely by the deseasonalized residual demand $\tilde{Z}$, which is entirely exogenous.

Residual Demand: The deseasonalized residual demand processes $Z^R$ and $\tilde{Z}$, associated with the prosumer and the consumer, respectively, evolve according to

$$\begin{array}{cc}\hfill \mathrm{d}{Z}^R\left(t\right)& =-{\kappa }_R{Z}^R\left(t\right)\mathrm{d}t+{\sigma }_R\left(t\right)\mathrm{d}{W}_R\left(t\right)+{\sigma }_0\mathrm{d}{W}^0\left(t\right),\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill \mathrm{d}\tilde{Z}\left(t\right)& =-{\kappa }_0\tilde{Z}\left(t\right)\mathrm{d}t+{\sigma }_0\mathrm{d}{W}^0\left(t\right),\hfill \end{array}$$

(2)

where $W_R$ represents the idiosyncratic noise capturing uncertainties specific to the prosumer’s heat demand, and $W^0$ is the common noise accounting for shared weather-driven fluctuations.

The seasonal behavior of the residual demands is modeled by bounded and deterministic seasonality functions ${\mu }^R,\tilde{\mu }:[0,T]\to \mathbb{R}$ for the prosumer and the consumer, respectively. Typical choices of $\widehat{\mu }\left(t\right)={\mu }^R,\tilde{\mu }$ include

$$\widehat{\mu }\left(t\right)=c_0+ct+\sum _{j=1}^{m_1}\left(c_{j}cos\left({\displaystyle \frac{2\pi }{{\rho }_j}}(t-t_j)\right)+d_{j}sin\left({\displaystyle \frac{2\pi }{{\rho }_j}}(t-t_j)\right)\right),$$

(3)

where $c_0$ is the long-term mean residual demand; $c_j,d_j$ represent the amplitudes of seasonality component j of the cosine and sine terms, respectively; ${\rho }_j,t_j$ represent the length and a reference time of seasonality component j, respectively; c is the coefficient of the linear term; and $m_1$ is the number of seasonality components.

Since the prosumer is equipped with a local production unit, which can lead to overproduction, the expression of $R\left(t\right)$ agrees with our modeling perspectives. Similarly, we model $\tilde{R}\left(t\right)=\tilde{\mu }\left(t\right)+\tilde{Z}\left(t\right)$ and choose parameters of $\tilde{\mu }\left(t\right)$ and $\tilde{Z}\left(t\right)$ for which there exists $\nu \ll 1$ such that $\mathbb{P}(\tilde{R}\left(t\right)<0)<\nu$. Below, $R>0$ is referred to as unsatisfied demand and $R<0$ as overproduction.

The control $\alpha \in [0,1]$ denotes the proportion of residual demand satisfied by the CHS. As in Ref. [2], we assume the decomposition $R\left(t\right)=\alpha \left(t\right)R\left(t\right)+(1-\alpha (t\left)\right)R\left(t\right)$, where $\alpha \left(t\right)R\left(t\right)$ is the residual demand satisfied by the ES, and $(1-\alpha (t\left)\right)R\left(t\right)$ is the residual demand satisfied by the CHS. Satisfying the residual demand via the CHS suggests either buying (for $R>0$) or selling (for $R<0$). Likewise, satisfying the residual demand via the ES implies either discharging (for $R>0$) or charging (for $R<0$).

External Storage: In the following, we consider as storage a water tank where the water mass and specific heat capacity are given by $m_Q\left[kg\right]$ and $c_P[KWh/kg K]$, respectively. The surface area of the tank is denoted as $A\left[m^2\right]$. From the heat fluxes and the Newton cooling law with the heat transfer coefficient $\gamma$ [$kW/m^{2}K$], the change in the thermal energy of the water inside the ES is given by

$$\mathrm{d}Q\left(t\right)={\displaystyle \frac{-1}{m_Q{c}_P}}({\eta }_E\left(t\right)(1-\alpha \left(t\right))R\left(t\right)+A\gamma (Q\left(t\right)-\underline{q}))\mathrm{d}t, Q\left(0\right)=Q_0.$$

(4)

The first term ${\eta }_E\left(t\right)(1-\alpha \left(t\right))R\left(t\right)$ represents the ES’s charging or discharging rate. Indeed, for an unsatisfied demand ($R\left(t\right)>0$) and for a non-empty ES ($Q\left(t\right)>\underline{q}$), the prosumer discharges the storage at a rate $\alpha \left(t\right)>0$. For an overproduction ($R\left(t\right)<0$) and for a non-full ES ($Q\left(t\right)<\overline{q}$), the prosumer can charge the storage at the rate $\alpha \left(t\right)<0$. The second term $A\gamma (Q\left(t\right)-\underline{q})$ is the rate of heat transfer with the environment. ${\eta }_E$ represents the charging or discharging efficiency of the ES. Letting ${\eta }_E^C,{\eta }_E^D\in (0,1)$, we model ${\eta }_E$ for simplicity as in [14] by

$${\eta }_E\left(t\right):={\eta }_E\left(R\left(t\right)\right)=\left\{\begin{array}{cc}{\eta }_E^C, \hfill & \mathrm{if} R\left(t\right)<0,\hfill \\ 1/{\eta }_E^D, \hfill & \mathrm{if} R\left(t\right)\ge 0.\hfill \end{array}\right.$$

(5)

Remark 1. 

• A large γ implies large losses per unit of time, prohibiting long-term storage. Similarly, a small γ implies small losses per unit of time, allowing for long-term storage.
• In the case $\alpha =0$, the prosumer does not satisfy its residual demand via the CHS. In the event of an unsatisfied demand ($R\left(t\right)>0$), $\alpha =1$ implies that the prosumer satisfies all of the residual demand via the CHS. However, for an overproduction ($R\left(t\right)<0$), $\alpha =1$ implies that the prosumer sells all the residual demand to the CHS.

State and Control Constraints: In order to reflect physical limitations of our external storage, we suggest that $Q\left(t\right)\in [\underline{q},\overline{q}]$, where $\underline{q}$ and $\overline{q}$ are the minimum and maximum storage levels, respectively. $Q\left(t\right)=\underline{q}$ is referred to as an empty storage, while $Q\left(t\right)=\overline{q}$ is a full storage.

The operational limitations experienced by the prosumer give rise to state-dependent control constraints:

1.

If $Q\left(t\right)=\underline{q}$, only charging is feasible. Therefore,

• If $R\left(t\right)>0$, all residual demand is satisfied by the CHS, implying $\alpha \left(t\right)=1$.
• If $R\left(t\right)<0$, there are no control constraints other than the requirement $\alpha \left(t\right)\in [0,1]$.

2.

If $Q\left(t\right)=\overline{q}$, we examine the following cases:

• $R\left(t\right)>0$, no control constraints other than the requirement $\alpha \left(t\right)\in [0,1]$.
• $R\left(t\right)<0$ and (4) suggest that ${\eta }_E(1-\alpha \left(t\right))R\left(t\right)+A\gamma (Q\left(t\right)-\underline{q})\ge 0$, hence implying that $\alpha \left(t\right)\ge 1+{\displaystyle \frac{A\gamma (\overline{q}-\underline{q})}{R\left(t\right){\eta }_E^C}}=:\widehat{\chi }(t,R\left(t\right))$. In the following, we define $\chi (t,R\left(t\right))=\widehat{\chi }(t,R\left(t\right))\vee 0$ and choose $\alpha \left(t\right)=\chi (t,R(t\left)\right)=\chi \left(t\right)$.

Hence, the state-dependent set of feasible controls, $\mathcal{U}(t,z^R,q)$, is defined by

$$\mathcal{U}(t,z^R,q)=\left\{\begin{array}{c}\begin{array}{cc}[0,1],\hfill & q>\underline{q}, z^R\ge -{\mu }^R\left(t\right),\hfill \\ \left\{1\right\},\hfill & q=\underline{q}, z^R\ge -{\mu }^R\left(t\right),\hfill \\ [0,1],\hfill & q<\overline{q}, z^R<-{\mu }^R\left(t\right),\hfill \\ \left[\chi \right(t),1],\hfill & q=\overline{q}, z^R<-{\mu }^R\left(t\right).\hfill \end{array}\hfill \end{array}\right.$$

Heat Price Formulation: We introduce time-varying heat buying and selling prices, denoted by $P_{buy}\left(t\right)$ and $P_{sell}\left(t\right)$, respectively, for all $t\in [0,T]$. We assume a contractual agreement between the prosumer and the CHS, binding the latter to always satisfy the residual demand of the former. To incentivize the CHS, it retains the right to determine the values of $P_{buy}\left(t\right)$ and $P_{sell}\left(t\right)$. We model $P_{buy}$ as a deterministic function composed of a constant baseline and multiple seasonal fluctuations. $P_{sell}$ is a fixed markdown from the buying price through the spread parameter $\xi >0$. Therefore, we obtain

$$P_{buy}\left(t\right)={𝓁}_0+\sum _{j=1}^{m_2}{𝓁}_{j}cos\left({\displaystyle \frac{2\pi }{{\rho }_j^S}}(t-t_j^S)\right), \mathrm{and} P_{sell}\left(t\right)=P_{buy}\left(t\right)-\xi .$$

(6)

The constant term ${𝓁}_0$$\left[{\displaystyle \frac{€}{kWh}}\right]$ corresponds to the long-term average heat price, while each cosine term introduces a seasonal component with amplitude ${𝓁}_j$$\left[{\displaystyle \frac{€}{kWh}}\right]$, period ${\rho }_j^S$$\left[h\right]$, and phase shift $t_j^S$$\left[h\right]$; $m_2$ represents the number of seasonalities. The sum in the heat price model means that it is not driven by a simple seasonal pattern but by the combined effects of several independent and interpretable cycles, each corresponding to a different physical, operational, or economic rhythm of the heating system. The deterministic seasonal structure makes prices transparent and predictable for the prosumer, while the positive spread guarantees economic viability for the CHS. The bid–ask spread reflects transaction costs, operational constraints, or the market power of the CHS. It ensures that the CHS is compensated for providing balancing and backup services.

Assumption 1. 

Let $P_c$ and $i_d$ denote the heat pump inlet and outlet temperatures, respectively. Similar to [2], we assume that the condition $\underline{q}>i_d>i_{min}>P_c$ holds.

Assumption 1 suggests that no heat pumps are necessary when satisfying the residual demand via the ES. This justifies the use of ordinary pumps in our model.

3. Problem Formulation

In this section, we formulate both the prosumer’s and consumer’s problems. The prosumer’s objective is to satisfy its heating and hot water demand while minimizing the cost of running the pumps and payments to the CHS.

3.1. Prosumer’s Problem

In the following, we present a mathematical framework to determine both the value and the optimal control strategy for the prosumer’s problem. This leads to the formulation of a stochastic optimal control problem for which the state process is defined as $X=(Z^R,Q)\in \mathcal{X}=\mathbb{R}\times [\underline{q},\overline{q}]$, representing the relevant system variables. From the dynamics described in the first equation in (1) and (4), the state process evolves as follows:

$$\begin{array}{ccc}\hfill \mathrm{d}{Z}^R\left(t\right)& =f\left(Z^R\left(t\right)\right)\mathrm{d}t+{\sigma }_R\mathrm{d}{W}_R\left(t\right)+{\sigma }_0\mathrm{d}{W}_0\left(t\right),\hfill & Z^R\left(0\right)=Z_0^R\in \mathbb{R},\\ \hfill \mathrm{d}Q\left(t\right)& =h(Z^R\left(t\right),Q\left(t\right),\alpha \left(t\right))\mathrm{d}t,\hfill & Q\left(0\right)=Q_0\in [\underline{q},\overline{q}],\end{array}$$

(7)

where $f\left(z^R\right)=-{\kappa }_R{z}^R$ and $h(z^R,q,\upsilon )={\displaystyle \frac{-1}{m_Q{c}_P}}({\eta }_E\left(t\right)(1-\upsilon )({\mu }^R\left(t\right)+z^R)+A\gamma (q-\underline{q}))$.

Assumption 2. 

Let $(z^R,q)\in \mathbb{R}\times [\underline{q},\overline{q}]$ and $q_1,q_2\in [\underline{q},\overline{q}]$.

1.

The function $h(z^R,q,\upsilon )$ is $\mathcal{F}$-measurable.

2.

The function $h(z^R,q,\upsilon )$ is continuous for almost all $z^R$.

3.

For almost all $z^R$, there exists $C\in \mathbb{R}$ such that $|h(z^R,q_1,\upsilon )-h(z^R,q_2,\upsilon )|\le C|q_1-q_2|$.

As in [15], this assumption ensures that the random ordinary differential equation (RODE) in (7) is well defined.

Cost Formulation

The costs incurred for operating the prosumer’s system are divided into a running cost and a terminal cost. Together, they define the total cost that needs to be minimized.

Running Cost: This consists of the cost (revenue) of satisfying the residual demand via the CHS, along with the cost of electricity consumption for operating the pumps when interacting with both the CHS and the ES. At $t\in [0,T]$, the prosumer can either purchase or sell thermal energy from or to the CHS at prices $P_{buy}\left(t\right)$ and $P_{sell}\left(t\right)$, respectively, to satisfy its residual demand. Let ${\phi }_0$ denote the cost (revenue) per unit of time of buying (selling) thermal energy from (to) the CHS, defined by

$${\phi }_0(t,z^R,\upsilon )=\left\{\begin{array}{cc}\upsilon ({\mu }^R\left(t\right)+z^R)P_{buy}\left(t\right), \hfill & z^R\ge -{\mu }^R\left(t\right),\hfill \\ \upsilon ({\mu }^R\left(t\right)+z^R)P_{sell}\left(t\right), \hfill & z^R<-{\mu }^R\left(t\right).\hfill \end{array}\right.$$

Since we assume that the prosumer is connected to an LTB-CHS, we incur an additional cost for the electricity consumption due to the use of a heat pump when satisfying the residual demand via the CHS. Let $b_1$ and $b_2\left[K^{-1}\right]$ denote positive constants penalizing a high water flow rate and a high temperature difference between $P_c$ and $i_d$, respectively. Letting S denote the electricity price, the  cost of electricity consumption ${\phi }_1$ incurred when interacting with the CHS is modeled by

$${\phi }_1(t,z^R,\upsilon )=\left\{\begin{array}{cc}\upsilon ({\mu }^R\left(t\right)+z^R)(b_1+b_2(i_d-P_c))S,\hfill & z^R\ge -{\mu }^R\left(t\right),\hfill \\ -\upsilon ({\mu }^R\left(t\right)+z^R)b_{1}S,\hfill & z^R<-{\mu }^R\left(t\right),\hfill \end{array}\right.$$

For $z^R\ge -{\mu }^R\left(t\right)$, the first equation in ${\phi }_1$ is the cost per unit of time of electricity consumption to run the heat pump to increase the temperature from $P_c$ to $i_d$. On the other hand, for $z^R<-{\mu }^R\left(t\right)$, the last equation in ${\phi }_1$ is the cost per unit of time of electricity consumption for operating the ordinary pump when selling thermal energy to the CHS.

The cost of satisfying the residual demand via the ES comes from the cost of electricity consumption to operate the ordinary pumps and is given by

$${\phi }_2(t,z^R,\upsilon )=(1-\upsilon )|{\mu }^R\left(t\right)+z^R|b_{1}S.$$

Therefore, the running cost $\mathsf{\Gamma }(t,z^R,\upsilon )$ is given by $\mathsf{\Gamma }(t,z^R,\upsilon )={\phi }_0(t,z^R,\upsilon )+{\phi }_1(t,z^R,\upsilon )+{\phi }_2(t,z^R,\upsilon )$.

Terminal Cost: In the context of finite-time horizon models with terminal time $T>0$, it is customary to incorporate a terminal cost. In our context, this depends on the ES storage level at time T and is denoted by $\mathsf{\Phi }\left(Q\right(T\left)\right)$. This terminal cost can reflect a contractual agreement for the ES temperature to be at an agreed level at T. Let $P_{liq},P_{pen}$ denote the liquidation price and penalty price, respectively. Indeed, at terminal time, the prosumer can sell the leftover thermal energy at $P_{liq}$ per unit of thermal energy. Similarly, the prosumer can be penalized for failing to keep the ES at a certain level at T, where $P_{pen}$ is the penalty price per unit of thermal energy. The liquidation price $P_{liq}$ and penalty price $P_{pen}$ satisfy the conditions $P_{liq}<P_{sell}\left(T\right)$ and $P_{pen}>P_{buy}\left(T\right)$, respectively. In the following, we let $q_{ref}$ denote a pre-agreed temperature level in the ES, such that

$$\mathsf{\Phi }\left(q\right)=\left\{\begin{array}{cc}{\displaystyle \frac{P_{pen}{m}_Q{c}_P(q_{ref}-q)}{{\eta }_E^C}},\hfill & q<q_{ref},\hfill \\ -{\eta }_E^D{P}_{liq}{m}_Q{c}_P(q-q_{ref}),\hfill & q\ge q_{ref}.\hfill \end{array}\right.$$

(8)

In this model, the terminal cost $\mathsf{\Phi }\left(q\right)$ now depends on the charging and discharging efficiency term ${\eta }_E\left(R\left(t\right)\right)$.

The performance criterion $J_P$: $[0,T]\times \mathcal{X}\times \mathcal{U}\to \mathbb{R}$ denotes the expected aggregated total discounted cost over the time interval $[0,T]$ and is defined as follows:

$$J_P(t,x,\alpha )={\mathbb{E}}_{t,x}\left[{\int }_t^T{e}^{-\delta (s-t)}\mathsf{\Gamma }(s,Z^R\left(s\right),\alpha \left(s\right))\mathrm{d}s+e^{-\delta (T-t)}\mathsf{\Phi }\left(Q\left(T\right)\right)\right],$$

(9)

where $x=(z^R,q)$, $\delta \ge 0$ is a discount rate and ${\mathbb{E}}_{t,x}[·]$ is the conditional expectation given that at initial time t, $X\left(t\right)=x$.

Assumption 3. 

For any admissible control α, the functions Γ and Φ satisfy the condition

$$\mathbb{E}\left[{\int }_t^T\left|\mathsf{\Gamma }\right(s,Z^R\left(s\right),\alpha \left(s\right)\left)\right|\mathrm{d}s+\left|\mathsf{\Phi }\right(Q\left(s\right)\left)\right|\right]<\infty .$$

(10)

Given that we want to use dynamic programming techniques, we restrict to Markov controls defined by $\alpha \left(t\right)=\theta (t,X(t\left)\right)$, for all $t\in [0,T]$ with a measurable function $\theta$: $[0,T]\times \mathcal{X}\to \mathcal{U}$, which is called a decision rule and is adapted to the filtration $\mathbb{F}$. We denote by $\mathcal{A}$ the class of admissible controls, defined as follows:

$$\begin{array}{cc}\hfill \mathcal{A}& =\{{\left(\alpha \left(t\right)\right)}_{t\in [0,T]} | \alpha \mathrm{is} \mathbb{F}-\mathrm{progressively} \mathrm{measurable},\alpha \left(t\right)=\theta (t,X\left(t\right)),\hfill \\ & \forall t\in [0,T],\theta (t,x)\in \mathcal{U}(t,x) \mathrm{and} \mathrm{Equations} \left(7\right) \mathrm{and} \left(9\right) \mathrm{are} \mathrm{well} \mathrm{defined}\}.\hfill \end{array}$$

(11)

The prosumer seeks to minimize $J_P$ over all admissible controls. The value of the prosumer’s problem for all $(t,x)\in [0,T]\times \mathcal{X}$ is given by

$$V(t,x)=\underset{\alpha \in \mathcal{A}}{inf}{J}_P(t,x,\alpha ).$$

(12)

3.2. Consumer’s Problem

We discuss the mathematical framework to determine the value of the consumer’s problem. We recall that the consumer is not equipped with local production or storage units and is always subject to a positive residual demand. As a result, it always satisfies all residual demand via the CHS. The state process of the consumer is given by

$$\mathrm{d}\tilde{Z}\left(t\right)=f_2\left(\tilde{Z}\left(t\right)\right)\mathrm{d}t+{\sigma }_0\mathrm{d}{W}_0\left(t\right), \tilde{Z}\left(0\right)={\tilde{Z}}_0\in \mathbb{R},$$

(13)

where $f_2\left(\tilde{z}\right)=-{\kappa }_0\tilde{z}$.

Similar to the prosumer’s case, the consumer’s running cost consists of the cost of purchasing thermal energy from the CHS and the cost of operating the heat pump to raise the temperature from $P_c$ to ${\tilde{i}}_d$. Let ${\tilde{\phi }}_0$ and ${\tilde{\phi }}_1$ denote the cost per unit of time of purchasing thermal energy from the CHS and that of electricity consumption to run the heat pump, respectively, which are defined as follows:

$$\begin{array}{cc}\hfill {\tilde{\phi }}_0(t,\tilde{z})& =\tilde{R}\left(t\right)P_{buy}\left(t\right),\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill {\tilde{\phi }}_1(t,\tilde{z})& =\tilde{R}\left(t\right)(b_1+b_2({\tilde{i}}_d-P_c))S,\hfill \end{array}$$

(15)

where $\tilde{R}\left(t\right)=\tilde{\mu }\left(t\right)+\tilde{Z}\left(t\right)$. Therefore, $\tilde{\mathsf{\Gamma }}(t,\tilde{z})={\tilde{\phi }}_0(t,\tilde{z})+{\tilde{\phi }}_1(t,\tilde{z})$.

Since the consumer is not equipped with a local storage unit, there is no terminal cost. The performance criterion $J_C:[0,T]\times \mathbb{R}\to {\mathbb{R}}^{+}$ denotes the expected aggregated total discounted cost over the time interval $[0,T]$ and is defined as follows:

$$J_C(t,\tilde{z})={\mathbb{E}}_{t,\tilde{z}}\left[{\int }_t^T{e}^{-\delta (s-t)}\tilde{\mathsf{\Gamma }}(s,\tilde{Z}\left(s\right))\mathrm{d}s\right],$$

(16)

where $\delta \ge 0$ is a discount rate and ${\mathbb{E}}_{t,\tilde{z}}[·]$ is the conditional expectation given that at initial time t, $\tilde{Z}\left(t\right)=\tilde{z}$. The value of the consumer’s problem for all $(t,\tilde{z})\in [0,T]\times \mathbb{R}$ is given by

$$\tilde{V}(t,\tilde{z})=J_C(t,\tilde{z}).$$

(17)

4. Semi-Lagrangian Discretization

In this section, we recall the discrete-time numerical scheme in [2]. We focus on the prosumer’s problem, since the consumer’s problem can be solved directly. We start by discussing the state discretization.

State Discretization: Let $N_t,N_z$, and $N_q$ denote the number of grid points in the t-, $z^R$-, and q-directions, respectively. For computational reasons, we truncate the domain $\mathcal{X}=\mathbb{R}\times [\underline{q},\overline{q}]$ to $\widehat{\mathcal{X}}=[{\underline{z}}^R,{\overline{z}}^R]\times [\underline{q},\overline{q}]$, such that for a tolerance $ϵ\ll 1,\mathbb{P}(Z^R\left(t\right)\in [{\underline{z}}^R,{\overline{z}}^R])\ge 1-ϵ$, for all $t\in [0,T]$. ${\underline{z}}^R$ and ${\overline{z}}^R$ represent the minimum and maximum values of the deseasonalized residual demand $Z^R$, respectively. Given the asymptotic standard deviation $s_0=\sqrt{{\displaystyle \frac{{\sigma }_R^2+{\sigma }_0^2}{2{\kappa }_R}}}$ of $Z^R$, the 3-$\sigma$ rule motivates ${\underline{z}}^R=-3s_0$ and ${\overline{z}}^R=3s_0$.

Let $t_0<t_1<\dots <t_{N_t}$, $z_0^R<z_1^R<\dots <z_{N_z}^R$, and $q_0<q_1<\dots <q_{N_q}$ be a finite number of grid points in the $z^R$ and q-directions. We define

$$\begin{array}{cc}\hfill \widehat{\mathcal{G}}& ={\widehat{\mathcal{G}}}_t\times {\widehat{\mathcal{G}}}_z\times {\widehat{\mathcal{G}}}_q=[t_0,\dots ,t_{N_t}]\times [z_0^R,\dots ,z_{N_z}^R]\times [q_0,\dots ,q_{N_q}],\hfill \end{array}$$

where $t_n=t_0+n\Delta t$, $z_i^R={\underline{z}}^R+i\Delta z^R$, $q_k=\underline{q}+k\Delta q$, for $n\in {\widehat{\mathcal{G}}}_n=\{0,1,\dots ,N_t\}$, $i=0,\dots ,N_z$ and $k=0,\dots ,N_q$, as a 3-dimensional equidistant grid on $\widehat{\mathcal{X}}$ with the temporal and spatial step sizes

$$t_{n+1}-t_n=:\Delta t={\displaystyle \frac{T-t_0}{N_t}}, \Delta z^R={\displaystyle \frac{{\overline{z}}^R-{\underline{z}}^R}{N_z}}, \Delta q={\displaystyle \frac{\overline{q}-\underline{q}}{N_q}}.$$

We now recall the discrete-time numerical scheme discussed in [2]. This is an alternative to the semi-Lagrangian approach introduced in [16] and extended in [17,18]. It is a finite difference scheme based on the theory presented in [19].

The control problem in (12) can be solved through dynamic programming techniques that rely on the following dynamic programming principle (DPP). The subsequent result is useful for solutions to the control problem in (12).

Theorem 1 

(DPP). For all $(t,x)\in [0,T)\times \mathcal{X},h>0$ and $t+h\le T$, the value function $V(t,x)$, satisfies the DPP

$$\begin{array}{cc}\hfill V(t,x)& =\underset{\alpha \in \mathcal{A}}{inf}{\mathbb{E}}_{t,x}\left[{\int }_t^{t+h}{e}^{-\delta (s-t)}\mathsf{\Gamma }(s,Z^R\left(s\right),\alpha \left(s\right))\mathrm{d}s+e^{-\delta h}V(t+h,X(t+h))\right].\hfill \end{array}$$

(18)

Proof. 

For a proof, we refer to [20].    □

The following result shows that the value function V, defined in (12), is a viscosity solution of an associated partial differential equation (PDE).

Proposition 1. 

Let $\Theta (t,x,\upsilon )=f\left(z^R\right), h(z^R,q,\upsilon ), \mathsf{\Gamma }(t,z^R,\upsilon ), \mathsf{\Phi }\left(q\right)$. Thus, there exists $L>0$ such that

$$\begin{array}{cc}\hfill |\Theta (t,x,\upsilon )-\Theta (t,\widehat{x},\upsilon )|& \le L|x-\widehat{x}|, \forall t\in [0,T],x,\widehat{x}\in \mathcal{X},\upsilon \in \mathcal{U},\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill |\Theta (t,0,\upsilon \left)\right|& \le L, \forall (t,\upsilon )\in [0,T]\times \mathcal{U}.\hfill \end{array}$$

(20)

Therefore, the value function V is a viscosity solution of the PDE

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial V}{\partial t}}+\mathcal{L}V+\underset{\upsilon \in \mathcal{U}(t,z^R,q)}{inf}\{{\mathcal{L}}_{q}V+\mathsf{\Gamma }(t,z^R,\upsilon )\}& =0,\hfill \end{array}$$

(21)

with the terminal condition $V(T,q)=\mathsf{\Phi }\left(q\right)$ and $\mathcal{L}V=f\left(z^R\right){\displaystyle \frac{\partial V}{\partial z^R}}+{\displaystyle \frac{1}{2}}({\sigma }_R^2\left(t\right)+{\sigma }_0^2){\displaystyle \frac{{\partial }^{2}V}{\partial {\left(z^R\right)}^2}}-\delta V$, ${\mathcal{L}}_q=h(z^R,q,\upsilon ){\displaystyle \frac{\partial V}{\partial q}}$.

Proof. 

First, we note that the functions $f\left(z^R\right), h(z^R,q,\upsilon ), \mathsf{\Gamma }(t,z^R,\upsilon ), \mathsf{\Phi }\left(q\right)$ are linear in the states $z^R$ and q. Then, there exits $L>0$, such that

$$\begin{array}{cc}\hfill |\Theta (t,x,\upsilon )-\Theta (t,\widehat{x},\upsilon )|& \le L|x-\widehat{x}|, \forall t\in [0,T],x,\widehat{x}\in \mathcal{X},\upsilon \in \mathcal{U}.\hfill \end{array}$$

In addition, we have $|\Theta (t,0,\upsilon \left)\right|\le L, \forall (t,\upsilon )\in [0,T]\times \mathcal{U}$. Finally, the result follows from Theorem 2.3 in [21].    □

Idea of the Scheme: We begin by fixing the triple $(t_n,z_i^R,q_k)$ and assume that, for $t\in [t_n,t_{n+1})$, the deseasonalized residual demand remains constant, i.e., $Z^R\left(t\right)=z_i^R$, where $z_i^R$ is fixed and known. Next, we assume that the control is constant over this interval, i.e., $\alpha \left(t\right)={\alpha }_{i,k}^n=:\upsilon$, which is fixed but unknown. Under these assumptions, and using the explicit solution, we compute the arrival point ${\mathcal{Q}}_{k(i,n)}^{\upsilon ,n+1}$ of the ES temperature level, given that at time $t_n$, the storage level was $q_k$, the residual demand was ${\mu }^R\left(t_n\right)+z_i^R$, and constant action $\upsilon$ was taken. We then apply the DPP in two steps: In the first step, using the above discretization, we obtain an approximation of the optimal control ${\alpha }_{i,k}^{*n}=:{\upsilon }^{*}$. In the second step, we substitute this approximate optimal control in the DPP and “release” the deseasonalized residual demand $Z^R$. Finally, by applying the Feynman–Kac formula to the second step, we derive a one-period PDE, which we discretize using finite difference methods to obtain a discrete-time scheme. This scheme is implemented in Algorithm 1.

From the above discussion, we formulate the following assumptions:

Assumption 4 

(Piecewise Constant Control). For $n=0,1,\dots ,N_t-1$ and $t\in [t_n,t_{n+1})$, the control α and the associated decision rule θ are kept constant between two consecutive grid points of the time discretization. That is,

$$\alpha \left(t\right)=\alpha \left(t_n\right) \mathrm{and} \theta (t,X\left(t\right))=\theta (t_n,X\left(t_n\right)).$$

Algorithm 1: Backward Recursion Algorithm

Assumption 5 

(Piecewise Constant Model Parameters). For $n=0,1,\dots ,N_t-1$ and $t\in [t_n,t_{n+1})$, the time-dependent seasonality ${\mu }^R$ and the efficiency parameter ${\eta }_E$ are kept constant between two consecutive grid points of the time discretization. That is,

$$\begin{array}{cc}\hfill {\mu }^R\left(t\right)& ={\mu }^R\left(t_n\right)={\mu }_n^R, {\mu }^R\left(T\right)={\mu }_{N_t}^R,\hfill \\ \hfill {\eta }_E\left(t\right)& ={\eta }_E\left(t_n\right)={\eta }_{E,n}, {\eta }_E\left(T\right)={\eta }_{E,N_t},\hfill \end{array}$$

Assumptions 4 and 5 suggest that the system’s time-dependent parameters are adjusted only at discrete-time points and remain constant within two consecutive discrete-time points. This situation is also consistent with reality.

Let $Q^{\upsilon }\left(t\right)$ denote the solution of the ODE (7) for the ES temperature level on the time interval $[t_n,t_{n+1})$ for a fixed but unknown control $\upsilon$, satisfying

$$\begin{array}{cc}\hfill \mathrm{d}{Q}^{\upsilon }\left(t\right)& =h(Z^R\left(t\right),Q^{\upsilon }\left(t\right),\upsilon )\mathrm{d}t, Q^{\upsilon }\left(t_n\right)=q_k, t\in [t_n,t_{n+1}).\hfill \end{array}$$

(22)

Lemma 1. 

Let Assumptions 4 and 5 hold and $\lambda ={\displaystyle \frac{A\gamma }{m_Q{c}_P}}$. Then, for $n=0,1,\dots ,N_t,i=0,1,\dots ,N_z$, $k=0,1,\dots ,N_q$, $t\in [t_n,t_{n+1})$ and $Q^{\upsilon }\left(t_n\right)=q_k$, the closed-from solution of $Q^{\upsilon }$ is given by

$$Q^{\upsilon }\left(t\right)=q_k{e}^{-\lambda (t-t_n)}+\left({\displaystyle \frac{{\eta }_{E,n}(\upsilon -1)({\mu }_n^R+z_i^R)}{A\gamma }}+\underline{q}\right)(1-e^{-\lambda (t-t_n)}).$$

(23)

Moreover, letting $t=t_{n+1}$ and ${\theta }_{i,k}^n={\displaystyle \frac{A\gamma q_k-[{\eta }_{E,n}(\upsilon -1)({\mu }_n^R+z_i^R)-A\gamma \underline{q}]}{\Delta q}}{\displaystyle \frac{1-e^{-\lambda \Delta t}}{A\gamma }}$, (23) becomes

$${\mathcal{Q}}_{k(i,n)}^{\upsilon ,n+1}=q_k-{\theta }_{i,k}^n\Delta q.$$

(24)

Proof. 

Let $\lambda ={\displaystyle \frac{A\gamma }{m_Q{c}_P}}$. For $n=0,1,\dots ,N_t,i=0,1,\dots ,N_z$ and $k=0,1,\dots ,N_q$, assume that $Q^{\upsilon }\left(t_n\right)=q_k$. Under Assumptions 4 and 5, (22) is a linear first-order ODE with constant coefficients and source term, which is solved to obtain the desired result. Now, substituting $t=t_{n+1}$ in (23) yields

$$\begin{array}{c}\hfill {\mathcal{Q}}_{k(i,n)}^{\upsilon ,n+1}=q_k{e}^{-\lambda \Delta t}+\left({\displaystyle \frac{{\eta }_{E,n}(\upsilon -1)({\mu }_n^R+z_i^R)}{A\gamma }}+\underline{q}\right)(1-e^{-\lambda \Delta t})\end{array}$$

(25)

Rearranging the terms in (25) yields (24). □

${\mathcal{Q}}_{k(i,n)}^{\upsilon ,n+1}$ denotes the ES temperature level at time $t_{n+1}$ knowing that, at time $t_n$, the ES was at the level $q_k$ with a residual demand ${\mu }_n^R+z_i^R$ and an action $\upsilon$ was taken.

Discrete-Time State-Dependent Control Constraints: In order for ${\mathcal{Q}}_{k(i,n)}^{\upsilon ,n+1}$ to always satisfy the condition ${\mathcal{Q}}_{k(i,n)}^{\upsilon ,n+1}\in [\underline{q},\overline{q}]$, we reformulate $\mathcal{U}$ to adapt to the discrete-time setting, where the control can only be adjusted at the end of the time interval $[t_n,t_{n+1})$. We obtain

$${\mathcal{U}}_d^n(z_i^R,q_k)=\{\upsilon \in \mathcal{U}(t_n,z_i^R,q_k) | {\mathcal{Q}}_{k(i,n)}^{\upsilon ,n+1}\in [\underline{q},\overline{q}]\}.$$

(26)

In the following lemma, we give the full expression of ${\mathcal{U}}_d^n(z_i^R,q_k)$.

Lemma 2. 

Under Assumptions 4 and 5, the set of discrete-time state-dependent constraints is

$${\mathcal{U}}_d^n(z_i^R,q_k)=\left\{\begin{array}{c}\begin{array}{cc}[max(0,{\upsilon }_{min}^d),min(1,{\upsilon }_{max}^d)],\hfill & q_k>\underline{q}, {\mu }_n^R+z_i^R\ge 0,\hfill \\ \left\{1\right\},\hfill & q_k=\underline{q}, {\mu }_n^R+z_i^R\ge 0,\hfill \\ max(0,{\upsilon }_{min}^c),min(1,{\upsilon }_{max}^c)],\hfill & q_k<\overline{q}, {\mu }_n^R+z_i^R<0,\hfill \\ max(\chi \left(t_n\right),{\upsilon }_{min}^c),min(1,{\upsilon }_{max}^c)],\hfill & q_k=\overline{q}, {\mu }_n^R+z_i^R<0,\hfill \end{array}\hfill \end{array}\right.$$

where

$$\begin{array}{cc}\hfill {\upsilon }_{min}^d& =1+{\displaystyle \frac{{\eta }_E^D[A\gamma (\underline{q}-q_k{e}^{-\lambda \Delta t})-\underline{q}(1-e^{-\lambda \Delta t})]}{(1-e^{-\lambda \Delta t})({\mu }_n^R+z_i^R)}},\hfill \\ \hfill {\upsilon }_{max}^d& =1+{\displaystyle \frac{{\eta }_E^D[A\gamma (\overline{q}-q_k{e}^{-\lambda \Delta t})-\underline{q}(1-e^{-\lambda \Delta t})]}{(1-e^{-\lambda \Delta t})({\mu }_n^R+z_i^R)}},\hfill \\ \hfill {\upsilon }_{min}^c& =1+{\displaystyle \frac{A\gamma (\overline{q}-q_k{e}^{-\lambda \Delta t})-\underline{q}(1-e^{-\lambda \Delta t})}{{\eta }_E^C(1-e^{-\lambda \Delta t})({\mu }_n^R+z_i^R)}},\hfill \\ \hfill {\upsilon }_{max}^c& =1+{\displaystyle \frac{A\gamma (\underline{q}-q_k{e}^{-\lambda \Delta t})-\underline{q}(1-e^{-\lambda \Delta t})}{{\eta }_E^C(1-e^{-\lambda \Delta t})({\mu }_n^R+z_i^R)}}.\hfill \end{array}$$

Proof. 

From the condition ${\mathcal{Q}}_{k(i,n)}^{\upsilon ,n+1}\in [\underline{q},\overline{q}]$, we obtain

$$\begin{array}{cc}\hfill {\displaystyle \frac{A\gamma (\underline{q}-q_k{e}^{-\lambda \Delta t})}{1-e^{-\lambda \Delta t}}}-\underline{q}& \le {\eta }_E(\upsilon -1)({\mu }_n^R+z_i^R)\hfill \\ \hfill & \le {\displaystyle \frac{A\gamma (\overline{q}-q_k{e}^{-\lambda \Delta t})}{1-e^{-\lambda \Delta t}}}-\underline{q}.\hfill \end{array}$$

If ${\mu }_n^R+z_i^R\ge 0$, ${\eta }_E={\displaystyle \frac{1}{{\eta }_E^D}}$, and we obtain $\upsilon \in [{\upsilon }_{min}^d,{\upsilon }_{max}^d]$. Similarly, if ${\mu }_n^R+z_i^R<0$, ${\eta }_E={\eta }_E^C$, and we have $\upsilon \in [{\upsilon }_{min}^c,{\upsilon }_{max}^c]$. Therefore, for ${\mu }_n^R+z_i^R\ge 0$, ${\mathcal{U}}_d^n(z_i^R,q_k)=\mathcal{U}(t_n,z_i^R,q_k)\cap [{\upsilon }_{min}^d,{\upsilon }_{max}^d]$, and for ${\mu }_n^R+z_i^R<0$, ${\mathcal{U}}_d^n(z_i^R,q_k)=\mathcal{U}(t_n,z_i^R,q_k)\cap [{\upsilon }_{min}^c,{\upsilon }_{max}^c]$. □

One-Step Terminal Value Problem: Starting from the DPP as in [22], we derive the following proposition, which is analogous to Theorem 4.3 in [2]. This result provides the one-step approximate optimal control together with the associated terminal value problem. The latter is then solved numerically to determine the value and optimal strategy of the prosumer’s optimization problem.

Proposition 2. 

Let $(t_n,z_i^R,q_k)$ be fixed, and suppose that Assumptions 4 and 5 hold. Then, starting from the DPP, the one-step approximate optimal control ${\upsilon }^{*}$ is given by

$${\upsilon }^{*}=\underset{\upsilon \in {\mathcal{U}}_d^n(z_i^R,q_k)}{arg\; min}\left\{{\int }_{t_n}^{t_{n+1}}{e}^{-\delta (t-t_n)}\mathsf{\Gamma }(t,z_i^R,\upsilon )\mathrm{d}t+e^{-\delta \Delta t}V(t_{n+1},z_i^R,{\mathcal{Q}}_{k(i,n)}^{\upsilon ,n+1})\right\}.$$

(27)

In addition, setting $H(t,z^R)=V(t,z^R,{\mathcal{Q}}^{{\upsilon }^{*}})$, we obtain the one-step terminal value problem

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial H}{\partial t}}(t,z^R)+\mathcal{L}H(t,z^R)+\mathsf{\Gamma }(t,z^R,{\upsilon }^{*})& =0, \mathrm{on} [t_n,t_{n+1})\times \mathcal{X}\hfill \\ \hfill H(t_{n+1},z^R)& =V(t_{n+1},z^R,{\mathcal{Q}}_{k(i,n)}^{{\upsilon }^{*},n+1}),\hfill \end{array}$$

(28)

where $\mathcal{L}H=f\left(z^R\right){\displaystyle \frac{\partial H}{\partial z^R}}+{\displaystyle \frac{1}{2}}({\sigma }_R^2\left(t\right)+{\sigma }_0^2){\displaystyle \frac{{\partial }^{2}H}{\partial {\left(z^R\right)}^2}}-\delta H$.

Proof. 

The proof is similar to that of Theorem 4.3 in [2], taking ${\sigma }^2={\sigma }^2\left(t\right)+{\sigma }_0^2$. □

Positivity Condition: In order to solve the one-step terminal value problem (28) using numerical techniques, we first discretize the differential operator $\mathcal{L}$. We let $\mathsf{\Lambda }$ denote the discretization operator of $\mathcal{L}$ and denote ${\vartheta }_i=-{\kappa }_R{z}_i^R$. Given that the sign of ${\vartheta }_i$ changes according to $z_i^R$ and cannot easily be determined, we apply the upwind discretization for the convection term ${\displaystyle \frac{\partial V}{\partial z^R}}$. Subsequently, we apply the central second-order finite difference for the diffusion term ${\displaystyle \frac{{\partial }^{2}V}{\partial {\left(z^R\right)}^2}}$. Letting ${\sigma }_R\left(t_n\right)={\sigma }_{R,n}$, we obtain

$$\mathsf{\Lambda }{V}_{i,k}^n={\displaystyle \frac{1}{2}}({\sigma }_{R,n}^2+{\sigma }_0^2){\displaystyle \frac{V_{i-1,k}^n-2V_{i,k}^n+V_{i+1,k}^n}{{(\Delta z^R)}^2}}-\delta V_{i,k}^n+\left\{\begin{array}{cc}{\vartheta }_i{\displaystyle \frac{V_{i,k}^n-V_{i-1,k}^n}{\Delta z^R}},\hfill & {\vartheta }_i\ge 0,\hfill \\ {\vartheta }_i{\displaystyle \frac{V_{i+1,k}^n-V_{i,k}^n}{\Delta z^R}},\hfill & {\vartheta }_i<0,\hfill \end{array}\right.$$

$$=A_i{V}_{i+1,k}^n-B_i{V}_{i,k}^n+C_i{V}_{i-1,k}^n,$$

(29)

where the expressions of $A_i$, $B_i$, and $C_i$ change depending on the sign of ${\vartheta }_i$. The following result gives us the upper bound for $\Delta z^R$ and defines the positivity condition for the coefficients $A_i,B_i$, and $C_i$.

Lemma 3 

(Positivity Condition). For $i=1,\dots ,N_z-1$ and $n=0,\dots ,N_t-1$, the coefficients $A_i,B_i$, and $C_i$ remain positive provided

$$\Delta z^R\le {\displaystyle \frac{\sqrt{2{\kappa }_R({\sigma }_{R,n}^2+{\sigma }_0^2)}}{6{\kappa }_R}}.$$

(30)

Proof. 

Similar to the proof of Proposition 4.4 in [2]. □

CFL Condition: In the following, we formulate the Courant–Friedrichs–Lewy (CFL) condition, as introduced in [23]. It relates the spatial step size $\Delta q$ to the time step $\Delta t$ and ensures that the arrival point ${\mathcal{Q}}_{k(i,n)}^{\upsilon ,n+1}$ consistently lies within the interval defined by the neighboring grid points $q_{k-1}=q_k-\Delta q$ and $q_{k+1}=q_k+\Delta q$ adjacent to $q_k$. From this assumption, the following inequalities can be derived

$$\begin{array}{ccc}\hfill \underline{q}:=q_0& \le \underline{q}-{\theta }_{i,0}^n\Delta q\le q_1:=\underline{q}+\Delta q,\hfill & {\theta }_{i,0}^n\le 0,\\ \hfill q_{k-1}& \le q_k-{\theta }_{i,k}^n\Delta q \le q_{k+1},\hfill & k=1,\dots ,N_q-1,\\ \hfill \overline{q}-\Delta q:=q_{N_q-1}& \le \overline{q}-{\theta }_{i,N_q}^n\Delta q\le q_{N_q}:=\overline{q},\hfill & {\theta }_{i,N_q}^n\ge 0,\end{array}$$

(31)

In the following, we obtain the Courant–Friedrichs–Lewy (CFL) condition ([23]) relating the time step size $\Delta t$ to the ES step size $\Delta q$. This ensures the stability of the derived numerical scheme. Let ${\displaystyle {\underline{\mu }}^R=\underset{n\in \{0,1,\dots ,N_t\}}{min}{\mu }_n^R}$, ${\displaystyle {\overline{\mu }}^R=\underset{n\in \{0,1,\dots ,N_t\}}{max}{\mu }_n^R}$. Therefore, from the above inequalities and the state-dependent set of feasible controls, the CFL condition is given by

$$\Delta q\ge {\displaystyle \frac{\Delta t}{m_Q{c}_P}}max\left(-{\eta }_E^C({\underline{\mu }}^R+{\underline{z}}^R),{\displaystyle \frac{{\overline{\mu }}^R+{\overline{z}}^R}{{\eta }_E^D}}+A\gamma (\overline{q}-\underline{q})\right).$$

(32)

Interpolation: As noted in [2], the arrival point ${\mathcal{Q}}_{k(i,n)}^{\upsilon ,n+1}$ does not necessarily coincide with a grid point $q_k\in {\widehat{\mathcal{G}}}_q$. This prompts an interpolation of $V(t_{n+1},z_i^R,{\mathcal{Q}}_{k(i,n)}^{\upsilon ,n+1})$ based on the function values $V_{i,k}^{n+1}$ at the grid points of $\widehat{\mathcal{G}}$. Following [17], a linear interpolation is sufficient to construct a monotone difference scheme. In what follows, we denote by $V_{k(i,n)}^{n+1}$, the interpolated values of $V(t_{n+1},z_i^R,{\mathcal{Q}}_{k(i,n)}^{\upsilon ,n+1})$ defined in the result below.

Proposition 3. 

Let ${\theta }_{i,k}^n={\displaystyle \frac{A\gamma q_k-[{\eta }_{E,n}(\upsilon -1)({\mu }_n^R+z_i^R)-A\gamma \underline{q}]}{\Delta q}}{\displaystyle \frac{1-e^{-\lambda \Delta t}}{A\gamma }}$, and assume that the CFL condition holds. Then, the interpolated value $V_{k(i,n)}^{n+1}$ is given by

$$\begin{array}{cc}\hfill V_{k(i,n)}^{n+1}& ={\mathcal{D}}_{i,k}^{(q,n)}{V}_{i,k}^{n+1}+{\mathcal{F}}_{i,k}^{(q,n)}{V}_{i,k-1}^{n+1}+{\mathcal{H}}_{i,k}^{(q,n)}{V}_{i,k+1}^{n+1},\hfill \end{array}$$

(33)

for $i=0,\dots ,N_{z^R}, k=1,\dots ,N_q-1, n=0,\dots ,N_t-1$ and where

${\mathcal{D}}_{i,k}^{(q,n)}=1-\left|{\theta }_{i,k}^n\right|, {\mathcal{F}}_{i,k}^{(q,n)}={\displaystyle \frac{{\theta }_{i,k}^n+\left|{\theta }_{i,k}^n\right|}{2}} and {\mathcal{H}}_{i,k}^{(q,n)}=-{\displaystyle \frac{{\theta }_{i,k}^n-\left|{\theta }_{i,k}^n\right|}{2}}$.

Furthermore, we obtain that for $k=0$ and $k=N_q$,

$$\begin{array}{cc}\hfill V_{0(i,n)}^{n+1}& ={\mathcal{D}}_{i,0}^{(q,n)}{V}_{i,0}^{n+1}+{\mathcal{H}}_{i,0}^{(q,n)}{V}_{i,1}^{n+1},\hfill \\ \hfill V_{N_q(i,n)}^{n+1}& ={\mathcal{D}}_{i,N_q}^{(q,n)}{V}_{i,N_q}^{n+1}+{\mathcal{F}}_{i,N_q}^{(q,n)}{V}_{i,N_q-1}^{n+1}.\hfill \end{array}$$

Proof. 

See Appendix C.3 in [2]. □

Now, we discretize the one-step terminal value problem (28) to obtain

$${\displaystyle \frac{V_{i,k}^{n+1}-V_{i,k}^n}{\Delta t}}+\mathsf{\Lambda }{V}_{i,k}^n+\mathsf{\Gamma }(t_n,z_i^R,{\upsilon }^{*})=0.$$

(34)

Finally, we obtain the fully implicit scheme given by

$$\begin{array}{c}{V}_{i,k}^n-\Delta t\left\{A_i{V}_{i+1,k}^n-B_i{V}_{i,k}^n+C_i{V}_{i-1,k}^n\right\}=V_{k(i,n)}^{n+1}+\Delta t\mathsf{\Gamma }(t_n,z_i^R,{\upsilon }^{*}),\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} {\mathcal{Q}}_{k(i,n)}^{\upsilon ,n+1}=q_k-{\theta }_{i,k}^n\Delta q,\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{\upsilon }_{i,k}^{*n}=\underset{\upsilon \in {\mathcal{U}}_d^n(z_i^R,q_k)}{arg\; min}\{{\int }_{t_n}^{t_{n+1}}{e}^{-\delta (t-t_n)}\mathsf{\Gamma }(t,z_i^R,\upsilon )\mathrm{d}t\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+e^{-\delta \Delta t}{V}_{k(i,n)}^{n+1}\}\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{\mathcal{U}}_d^n(z_i^R,q_k)=\{\upsilon \in \mathcal{U}(t_n,z_i^R,q_k) | {\mathcal{Q}}_{k(i,n)}^{\upsilon ,n+1}\in [\underline{q},\overline{q}]\}\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{V}_{i,k}^{N_t}=\mathsf{\Phi }\left(q_k\right).\hfill \end{array}$$

(35)

From the first equation in (35), we want to form a system of linear algebraic equations that are useful to obtain the values of $V_{i,k}^n$. In that regard, we set ${\mathsf{\Gamma }}_{i,k}^{n+1}=\mathsf{\Gamma }(t_n,z_i^R,{\upsilon }^{*})$ and assume that, for fixed n and k, the optimal strategy ${\upsilon }^{*}={\upsilon }_{i,k}^{*n}$, the CFL condition, and the terms in (33) for all i are known. We denote by ${\mathsf{\Psi }}_{i,k}^{n+1}$ the known right-hand side of the difference equation (35) for fixed $t_n,q_k$ and for all i. Hence, for $i=1,\dots ,N_z-1,$ $k=1,\dots ,N_q-1,n=0,1,\dots ,N_t-1$, we obtain

$$(1+\Delta t{B}_i)V_{i,k}^n-\Delta t{C}_i{V}_{i-1,k}^n-\Delta t{A}_i{V}_{i+1,k}^n={\mathsf{\Psi }}_{i,k}^{n+1},$$

(36)

At the boundary $q=\underline{q}$, i.e., $k=0$, we recall that ${\theta }_{i,0}^n\le 0$. Similarly, at the boundary $q=\overline{q}$, i.e., $k=N_q$, we have ${\theta }_{i,N_q}^n\ge 0$. Therefore, we have the following

$$V_{k(i,n)}^{n+1}=\left\{\begin{array}{cc}\left(1-{\theta }_{i,k}^n\right)V_{i,k}^{n+1}+{\theta }_{i,k}^n{V}_{i,k-1}^{n+1}\hfill & {\theta }_{i,k}^n\ge 0, k=1,\dots ,N_q,\hfill \\ \left(1+{\theta }_{i,k}^n\right)V_{i,k}^{n+1}-{\theta }_{i,k}^n{V}_{i,k+1}^{n+1}\hfill & {\theta }_{i,k}^n<0, k=0,\dots ,N_q-1.\hfill \end{array}\right.$$

(37)

Boundary Conditions: To ensure that the one-step terminal value problem, formulated as a PDE, is well posed, we specify boundary conditions at $z^R={\underline{z}}^R,{\overline{z}}^R$ and $q=\underline{q},\overline{q}$. These conditions arise from truncating the computational domain of the PDE from $\mathcal{X}$ to a bounded domain $\widehat{\mathcal{X}}$. Accordingly, we require that

$${\displaystyle \frac{{\partial }^{2}V}{\partial {\left(z^R\right)}^2}}(t,{\underline{z}}^R,q)=0, {\displaystyle \frac{{\partial }^{2}V}{\partial {\left(z^R\right)}^2}}(t,{\overline{z}}^R,q)=0, (t,q)\in (0,T)\times [\underline{q},\overline{q}].$$

(38)

$${\displaystyle \frac{{\partial }^{2}V}{\partial q^2}}(t,z^R,\underline{q})=0, {\displaystyle \frac{{\partial }^{2}V}{\partial q^2}}(t,z^R,\overline{q})=0, (t,z^R)\in (0,T)\times [{\underline{z}}^R,{\overline{z}}^R].$$

(39)

We note that the corner values $V_{0,0}^n,V_{0,N_q}^n,V_{N_z,0}^n$, and $V_{N_z,N_q}^n$ are computed using previously obtained values $V_{i,k}^n$ for $i=0,\dots ,N_z$, $k=0,\dots ,N_q$, so that

$$\begin{array}{cc}\hfill V_{0,0}^n& =2V_{1,0}^n-V_{2,0}^n,\hfill \\ \hfill V_{0,N_q}^n& =2V_{0,N_q-1}^n-V_{0,N_q-2}^n,\hfill \\ \hfill V_{N_z,0}^n& =2V_{N_z,1}^n-V_{N_z,2}^n,\hfill \\ \hfill V_{N_z,N_q}^n& =2V_{N_z,N_q-1}^n-V_{N_z,N_q-2}^n.\hfill \end{array}$$

(40)

Combining (40), the boundary conditions, and the difference equation (35), we proceed to the matrix formulation below.

Matrix Formulation: Simultaneously varying the indices $i,k$, for n fixed in the difference equation (35), and using (40), we obtain the following system of linear equations:

$$G{\mathit{V}}_k^n={\mathsf{𝚿}}_k^{n+1}, \mathrm{for} n=0,\dots ,N_t-1, k=1,\dots ,N_q-1,$$

(41)

where ${\mathit{V}}_k^n={(V_{1,k}^n,\dots ,V_{N_z-1,k}^n)}^T$, ${\mathsf{𝚿}}_k^{n+1}={({\mathsf{\Psi }}_{1,k}^{n+1},\dots ,{\mathsf{\Psi }}_{N_z-1,k}^{n+1})}^T$, with G an $(N_z-1)\times (N_z-1)$ tridiagonal matrix given as

$$G=\left[\begin{array}{ccccccc}{e}_1& p_1& 0& 0& \dots & 0& 0\\ h_2& e_2& p_2& 0& \dots & 0& 0\\ 0& h_3& e_3& p_3& \dots & 0& 0\\ \dots & \dots & \dots & \dots & \dots & \dots & \dots \\ 0& 0& 0& \dots & h_{N_z-2}& e_{N_z-2}& p_{N_z-2}\\ 0& 0& 0& \dots & 0& h_{N_z-1}& e_{N_z-1}\end{array}\right]$$

(42)

where

$$\begin{array}{ccc}\hfill e_1& =1+\Delta t{B}_1-2\Delta t{C}_1,\hfill & p_1=\Delta t(C_1-A_1),\\ \hfill e_k& =1+\Delta t{B}_k, h_k=-\Delta t{C}_k,\hfill & p_k=-\Delta t{A}_k, k=2,\dots ,N_z-2\\ \hfill e_{N_z-1}& =1+\Delta t{B}_{N_z-1}-2\Delta t{A}_{N_z-1}\hfill & h_{N_z-1}=-\Delta t(C_{N_z-1}-A_{N_z-1}).\end{array}$$

It now remains to obtain the values $V_{0,k}^n,V_{N_z,k}^n,V_{i,0}^n$, and $V_{i,N_q}^n$. To do so, we substitute the boundary conditions (38) in the difference equation (35) to obtain

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial V}{\partial t}}(t,z^R,q)+{\mathcal{L}}_{1}V(t,z^R,q)+\mathsf{\Gamma }(t,z^R,{\upsilon }^{*})& =0,\hfill \end{array}$$

(43)

where for $i=0,{\mathcal{L}}_{1}V=-{\kappa }_R{\underline{z}}^R{\displaystyle \frac{\partial V}{\partial z^R}}-\delta V$, while for $i=N_z,{\mathcal{L}}_{1}V=-{\kappa }_R{\overline{z}}^R{\displaystyle \frac{\partial V}{\partial z^R}}-\delta V$. Following similar steps as above, we obtain the following for $z^R={\underline{z}}^R$:

$$K{\mathit{V}}_0^n={\mathsf{𝚿}}_0^{n+1}+\Delta t{A}_0{\mathit{V}}_1^n, \mathrm{for} n=0,\dots ,N_t-1,$$

(44)

${\mathit{V}}_0^n={(V_{0,1}^n,\dots ,V_{0,N_q-1}^n)}^T$, ${\mathsf{𝚿}}_0^{n+1}={({\mathsf{\Psi }}_{0,1}^{n+1},\dots ,{\mathsf{\Psi }}_{0,N_q-1}^{n+1})}^T$, ${\mathit{V}}_1^n={(V_{1,1}^n,\dots ,V_{1,N_q-1}^n)}^T$.

Similarly, for $z^R={\overline{z}}^R$, we have

$$M{\mathit{V}}_{N_z}^n={\mathsf{𝚿}}_{N_z}+\Delta t{C}_{N_z}{\mathit{V}}_{N_z-1}^n, \mathrm{for} n=0,\dots ,N_t-1,$$

(45)

${\mathit{V}}_{N_z}^n ={(V_{N_z,1}^n,\dots ,V_{N_z,N_q-1}^n)}^T$, ${\mathsf{𝚿}}_{N_z}^{n+1}={({\mathsf{\Psi }}_{N_z,1}^{n+1},\dots ,{\mathsf{\Psi }}_{N_z,N_q-1}^{n+1})}^T$,

${\mathit{V}}_{N_z-1}^n={(V_{N_z-1,1}^n,\dots ,V_{N_z-1,N_q-1}^n)}^T$. Letting $I_{N_q-1}$ denote the $(N_q-1)\times (N_q-1)$ identity matrix, K and M are $(N_q-1)\times (N_q-1)$ diagonal matrices given by

$$K=(1+\Delta t{B}_0)I_{N_q-1}, M=(1+\Delta t{C}_{N_z})I_{N_q-1}.$$

(46)

Substituting $k=0$ and $k=N_q$, respectively, in the difference equation (35), we obtain the following for $n=0,\dots ,N_t-1$:

$$\begin{array}{cc}\hfill G{\mathit{V}}_0^n& ={\mathsf{𝚿}}_0^{n+1}, \mathrm{for} i=1,\dots ,N_z-1,\hfill \\ \hfill G{\mathit{V}}_{N_q}^n& ={\mathsf{𝚿}}_{N_q}^{n+1}, \mathrm{for} i=1,\dots ,N_z-1,\hfill \end{array}$$

(47)

${\mathit{V}}_0^n={(V_{1,0}^n,\dots ,V_{N_z-1,0}^n)}^T$, ${\mathsf{𝚿}}_0^{n+1}={({\mathsf{\Psi }}_{1,0}^{n+1},\dots ,{\mathsf{\Psi }}_{N_z-1,0}^{n+1})}^T$, ${\mathit{V}}_{N_q}^n={(V_{1,N_q}^n,\dots ,V_{N_z-1,N_q}^n)}^T$, ${\mathsf{𝚿}}_{N_q}^{n+1}={({\mathsf{\Psi }}_{1,N_q}^{n+1},\dots ,{\mathsf{\Psi }}_{N_z-1,N_q}^{n+1})}^T$. Further details can be obtained in [2].

Backward Recursion Algorithm: The approximate optimal control presented in (27), along with the value functions in (41), (44), (45), and (47), can be computed using Algorithm 1, proceeding backward in time from the terminal time step $N_t$. To determine ${\upsilon }^{*}={\upsilon }_{i,k}^{*n}$, we take into account the storage level and distinguish between scenarios of unsatisfied demand and overproduction. This classification indicates which subinterval of ${\mathcal{U}}_d^n$ should be evaluated. Subsequently, an optimization procedure is carried out to identify the control value that minimizes the objective function. The resulting control is then interpreted as the optimal decision for the given state variables at grid points $z_i^R$ and $q_k$. This procedure is applied iteratively across all combinations of grid points to recover the complete set of optimal controls.

5. Numerical Results

In this section, we discuss the numerical solution for the prosumer’s problem. The results are based on the implementation of Algorithm 1 to find the optimal strategies and the value functions as well as studying the properties of the obtained value functions. As the terminal condition, we consider a penalization problem modeled as

$$\mathsf{\Phi }\left(q\right)=\left\{\begin{array}{cc}{\displaystyle \frac{P_{pen}{m}_Q{c}_P(q_{pen}-q)}{{\eta }_E^C}},\hfill & q<q_{pen},\hfill \\ 0,\hfill & q\ge q_{pen}.\hfill \end{array}\right.$$

(48)

For the purpose of the numerical simulations, the seasonality function and heat price are modeled, respectively, by

$$\begin{array}{cc}\hfill {\mu }^R\left(t\right)& =c_0+ccos\left({\displaystyle \frac{2\pi t}{\rho }}\right), P_{buy}\left(t\right)={𝓁}_0+𝓁cos\left({\displaystyle \frac{2\pi t}{\rho }}\right), P_{sell}\left(t\right)=P_{buy}\left(t\right)-\xi .\hfill \end{array}$$

(49)

The full description of the model parameter values is given in the

Table 1. Model and discretization parameters.

Parameters	Values	Units	Parameters	Values	Units
${\kappa }_R$	0.025	$h^{-1}$	S	0.335	${\displaystyle \frac{€}{kWh}}$
${\sigma }_R,{\sigma }_0$	0.005, 0.4	${\displaystyle \frac{kW}{\sqrt{h}}}$	$P_{pen}$	0.325	${\displaystyle \frac{€}{kWh}}$
$c_0,c$	0.37, 1.00	$kW$	$\xi$	0.02	${\displaystyle \frac{€}{kWh}}$
			${𝓁}_0,𝓁$	0.17, 0.15	${\displaystyle \frac{€}{kWh}}$
$\rho$	8760	h	${\underline{z}}^R,{\overline{z}}^R$	−5.37, 5.37	$kW$
$b_1$	0.01		$T,\Delta t$	8760, 1	h
$b_2$	0.012	$K^{-1}$	$N_t$	8760
$m_Q$	7854	$kg$	${\eta }_E^D,{\eta }_E^C$	0.95, 0.95
$c_P$	$0.0012$	${\displaystyle \frac{kWh}{kg K}}$	$N_z,N_q$	85, 60
A	21.99	$m^2$	$P_c,i_d$	20, 25	°C
$\gamma$	$2.34\times {10}^{-4}$	${\displaystyle \frac{kW}{m^2 K}}$	$\underline{q},q_{pen},\overline{q}$	25, 40, 85	°C

below.

Parameters $\gamma ,c_0,c,b_1$, and $b_2$ are calibrated to the model, and the idea for the calibration is provided in [2].

Terminal Cost:Figure 2 shows the terminal cost $\mathsf{\Phi }\left(q\right)$ as a function of the residual demand r and the ES storage level q. Here, the terminal cost is formulated as a penalization problem. By construction as in (48), we observe that the value function is constant with respect to the residual demand at the terminal time. However, if the storage level falls below the reference temperature $q_{ref}$ = 40 °C, the prosumer incurs a cost proportional to the deviation from $q_{ref}$ and the charging efficiency ${\eta }_E^C$. This cost increases the further the storage temperature is below the reference. For temperatures above the reference, no additional cost is incurred, and the storage is effectively considered to have no residual value.

Value Function and Optimal Strategy for Time-Dependent Heat Buying and Selling Prices:Figure 3 depicts the value functions and optimal strategies of the prosumer at the initial time $t=0$ and on day $t=362$. From the top- and bottom-left panels, we can observe that as the ES temperature increases, the prosumer incurs lower costs. Conversely, as the residual demand increases, the prosumer incurs higher costs. The top-right panel shows that, for an unsatisfied demand ($r\ge 0$), the prosumer satisfies all residual demand via the CHS if the ES is empty; otherwise, it discharges the ES. In the case of overproduction ($r<0$), all excess production is sold to the CHS. From the bottom-right panel, we can note that, for an unsatisfied demand, the prosumer satisfies all residual demand via the CHS when $q<40$ °C. For $q\ge 40$ °C, all residual demand is satisfied by the ES. In the case of overproduction, the prosumer first stores some energy in the ES and subsequently sells any remaining excess to the CHS. Since the horizon is one year (365 days), with a penalization cost at the terminal time, we can observe that on day 362, the prosumer begins filling the storage close to the penalty temperature $q_{ref}=40$ °C to avoid higher terminal costs.

Figure 4 shows the prosumer’s value functions and optimal strategies as functions of the residual demand r and the ES temperature level q on day $t=230$. In the top-left panel, we can observe that, for a given residual demand, the prosumer incurs lower costs as the ES temperature increases. The expected aggregated discounted cost is higher for the strongest unsatisfied residual demand and lower for the strongest overproduction. On the other hand, the bottom-left panel shows that the cost increases with respect to the residual demand, with the highest cost incurred for an empty storage. The smallest cost is incurred for a full storage. In the top-right panel, we can observe that, for both the strongest and smallest unsatisfied demand, the prosumer first purchases thermal energy from the CHS. Afterwards, for a sufficiently filled ES, it discharges the storage to meet the unsatisfied demand. For the strongest overproduction, the prosumer stores all excess production in the ES and only sells to the CHS if the ES is full. In the bottom-right panel, for both an empty and $50\%$-full ES, the prosumer stores the excess production in the ES for an overproduction and satisfies all unsatisfied demand via the CHS for an unsatisfied residual demand. Now, for a full ES, the prosumer compensates the loss to the environment of the ES in the case of overproduction. For an unsatisfied demand, all residual demand is satisfied by the ES.

Value Function and Optimal Strategy for Constant Heat Buying and Selling Prices: For the case of constant heat buying and selling prices, we focus on the value function and optimal strategies as functions of the residual demand r and the ES temperature level q. Let $P_{buy}^C$ and $P_{sell}^C$ denote the constant heat buying and selling prices, respectively. In the following, we model $P_{buy}^C$ and $P_{sell}^C$ as

$$P_{buy}^C=\underset{t\in [0,T]}{max}{P}_{buy}\left(t\right) P_{sell}^C=P_{buy}^C-\xi ,$$

(50)

where $P_{buy},P_{sell}$, and $\xi$ are the same as in (49). In Figure 5, we focus on the top- and bottom-right panels since the top- and bottom-left panels have the same interpretation as in Figure 4. In the top-right panel of Figure 5, for both the strongest and the smallest unsatisfied demand, the prosumer discharges the ES if it is not empty; otherwise, all residual demand is satisfied by the CHS, at a higher cost. For the strongest overproduction, all excess thermal energy is sold to the CHS for revenue. As shown in the bottom-right panel, for an empty ES and overproduction, the prosumer sells all residual demand to the CHS. However, for an unsatisfied demand, all residual demand is satisfied by the CHS. For a half-full or full ES, the prosumer sells all residual demand to the CHS in the event of overproduction. For an unsatisfied demand, it discharges the ES.

Let $V^{max}$ denote the maximum value function of the prosumer for time-dependent heat buying and selling prices. $V_{Constant}^{max}$ denotes the maximum value function of the prosumer for constant heat buying and selling prices, and ${\tilde{V}}^{max}$ is the maximum value function of the consumer for time-dependent heat buying and selling prices.

In

Table 2. Maximum value function for an imperfect efficiency with time-dependent and constant heat buying and selling prices, in EUR, for a prosumer and a consumer.

Value Function	Numerical Value
$V^{max}$	1562.9
$V_{Constant}^{max}$	1465.1
${\tilde{V}}^{max}$	40,474.0

, we note that the prosumer incurs a higher expected aggregated discounted cost under the time-dependent heat prices model than with its constant counterpart. This is due to the fact that the time-dependent heat prices model reflects the changes in heat prices during cold and warm seasons, which is not observed in the constant pricing case. Since the consumer is not equipped with heat production and storage units, it relies solely on the CHS for its unsatisfied residual demand, hence leading to a much higher expected aggregated discounted cost. From the values of ${\tilde{V}}^{max}$ and $V^{max}$, we can compute the expected consumer investment cost into heat production and storage units in order to reduce the cost of satisfying its heating and hot water demands. This investment cost is denoted by ${\tilde{C}}_{invest}={\tilde{V}}^{max}-V^{max}$.

5.1. Sensitivity Analysis

In this section, we perform a sensitivity analysis on ${\sigma }_0,{\eta }_E^C$, and ${\eta }_C^D$ to study their impact on the expected aggregate discounted cost. The results presented correspond to the case of time-dependent heat buying and selling prices. All parameters are fixed, with values given in Table 1, except for ${\sigma }_0,{\eta }_E^C$ and ${\eta }_C^D$, which are varied. Throughout this section, we let $V^{max}$ denote the maximum value function, as in Table 2.

In

Table 3. Maximum value function from varying common noise volatility coefficient ${\sigma }_0$ under time-dependent heat buying and selling prices, in EUR.

${\sigma }_0$	0	0.1	0.15	0.25	0.35	0.4
$V^{max}$	1346.4	1377.3	1400.8	1459.4	1527.1	1562.9

, we can observe that as the impact of the weather conditions increases, the aggregated discounted cost becomes higher. This is the case because higher values of ${\sigma }_0$ increase the residual demand, which, in turn, increases the prosumer’s cost. Thus, the more pronounced the impact of weather conditions is on the heating system, the higher the aggregated discounted cost the prosumer will incur. For example, for ${\sigma }_0=0$, the prosumer incurs a maximum cost of $V^{max}=1346.4$ EUR whereas for ${\sigma }_0=0.4$, $V^{max}=1562.9$ EUR, which is approximately a $16.08\%$ increase from the case of no common noise.

In

Table 4. Maximum value function for varying ${\eta }_E^D$ under time-dependent heat buying and selling prices, in EUR.

${\eta }_E^D$	0.90	0.92	0.94	0.96	1
$V^{max}$	1574.3	1570.2	1565.5	1559.9	1545.7

, we can observe that the better the discharging efficiency, the smaller the cost incurred by the prosumer. This is due to the fact that, for a weak discharging efficiency, the prosumer, when satisfying residual demand via the ES, discharges more thermal energy to compensate for the loss during the process. Hence, a better discharging efficiency reduces losses during the discharge, hence contributing to the reduction in the prosumer’s aggregated discounted cost. For a discharging efficiency ${\eta }_E={\displaystyle \frac{1}{0.9}}$, we can observe an increase of approximately $1.85\%$ in the aggregated discounted cost as compared to the case of a perfect discharging efficiency ${\eta }_E=1$.

As the charging efficiency improves in

Table 5. Maximum value function for varying ${\eta }_E^C$ under time-dependent heat buying and selling prices, in EUR.

${\eta }_E^C$	0.90	0.92	0.94	0.96	1
$V^{max}$	1575.2	1570.7	1565.7	1559.7	1545.2

, we can observe that the prosumer incurs a smaller aggregated discounted cost. For a weak charging efficiency, not all overproduction can be stored in the ES, due to the losses to the environment during charging. Therefore, as the charging efficiency improves, more thermal energy can be stored in the ES for further use. Hence, for a charging efficiency of 0.9, we observe an approximate $1.94\%$ increase in the aggregated discounted cost as compared to a perfect charging efficiency.

5.2. Optimal Path of ES Level

In Figure 6, we show the optimal path of the ES temperature level Q together with the residual demand R, the seasonality function ${\mu }^R$, and the absolute optimal control ${\alpha }^{*}R$ for a time-dependent heat price. At the initial time, we assume that the prosumer starts with a full storage. Hence, for an unsatisfied residual demand, the prosumer discharges the ES until it becomes empty. For overproduction, the prosumer slowly charges the ES and drives it almost full in late summer. We further observe that for $R>0$, the prosumer discharges the ES to satisfy the unsatisfied demand and charges the ES for $R<0$.

6. Summary and Outlook

In this work, we investigated a stochastic optimal control problem involving a prosumer and a consumer, both connected to a CHS. The prosumer is equipped with a local renewable heat production source (solar collector) and a local storage unit. The ES allows the prosumer to store excess thermal energy for future use. However, we assume that it is subject to charging and discharging efficiencies. In contrast, the consumer has no local production or storage unit and is therefore constantly subject to an unsatisfied demand. We focused primarily on the prosumer problem, as the consumer always satisfies all residual demand directly via the CHS. The prosumer’s problem was formulated as a mathematical optimization problem, which we solved using semi-Lagrangian techniques. We presented the numerical results for both a time-dependent and a constant heat price formulation and observed that, under the former price formulation, the prosumer incurred a higher maximum expected aggregated discounted cost than in the latter price formulation. We further performed a sensitivity analysis on the common noise and charging/discharging efficiency parameters. For the case of the common noise, we noted that for ${\sigma }_0=0.4$, the maximum expected aggregated discounted cost increases by approximately $16.08\%$ as compared to no common noise. Meanwhile, for the discharging efficiency, we observed that for ${\eta }_E={\displaystyle \frac{1}{0.9}}$, the prosumer incurs a cost about $1.85\%$ higher than in the case of a perfect discharging efficiency. Also, for the charging efficiency, we noted that for ${\eta }_E=0.90$, the maximum expected aggregated discounted cost increases by approximately $1.94\%$ as compared to the case of a perfect charging efficiency. Our numerical results also highlight the value function and the optimal strategies of the prosumer. Finally, for the consumer, we evaluated the maximum expected investment required in local heat production and storage to transform into a prosumer.

Looking ahead, it would be interesting to incorporate additional weather factors, such as ambient temperature, to better model the residual demand. Furthermore, modeling electricity prices as a stochastic differential equation (SDE) could provide more realistic dynamics. These extensions are currently under investigation in ongoing research.