Optimal Power Procurement for Green Cellular Wireless Networks Under Uncertainty and Chance Constraints

Abstract

Given the increasing global emphasis on sustainable energy usage and the rising energy demands of cellular wireless networks, this work seeks an optimal short-term, continuous-time power-procurement schedule to minimize operating expenditure and the carbon footprint of cellular wireless networks equipped with energy-storage capacity, and hybrid energy systems comprising uncertain renewable energy sources. Despite the stochastic nature of wireless fading channels, the network operator must ensure a certain quality-of-service (QoS) constraint with high probability. This probabilistic constraint prevents using the dynamic programming principle to solve the stochastic optimal control problem. This work introduces a novel time-continuous Lagrangian relaxation approach tailored for real-time, near-optimal energy procurement in cellular networks, overcoming tractability problems associated with the probabilistic QoS constraint. The numerical solution procedure includes an efficient upwind finite-difference solver for the Hamilton–Jacobi–Bellman equation corresponding to the relaxed problem, and an effective combination of the limited memory bundle method (LMBM) for handling nonsmooth optimization and the stochastic subgradient method (SSM) to navigate the stochasticity of the dual problem. Numerical results, based on the German power system and daily cellular traffic data, demonstrate the computational efficiency of the proposed numerical approach, providing a near-optimal policy in a practical timeframe.

1. Introduction

Since 2021, worldwide mobile broadband traffic has increased by an average of 19.6% annually, reaching 1 zettabyte (ZB) in 2023 [1]. The 4G network coverage increased from 41% of the world population in 2015 to 92% of the world population in 2024 [1]. The estimated transmission network electricity consumption in 2022 was 240 to 340 TWh, an increase of 64% from 2015 [2]. Next-generation transmission technology (e.g., 5G) requires even more power to ensure sufficient mobile network coverage [3]. A consequence of the high energy demand is increased greenhouse gas emissions. Data-transmission networks accounted for around 330 Mt ${\mathrm{CO}}_2$-equivalent in 2020, around 0.9% of all energy-related emissions [2]. About 83% of the energy consumed by cellular networks came from fossil fuels in 2021, whereas only 9% came from renewables [4]. This problem motivates the coupling of cellular wireless networks with renewable energy to reduce the costs and carbon footprint of the telecommunications industry. Hybrid energy systems provide a consistent power supply to wireless networks by combining renewable energy sources (e.g., solar and wind) [5]. However, modeling these systems and deriving an optimal power-procurement policy can be challenging, due to the stochasticity of solar/wind power and wireless channels [6].

Previous studies have considered the optimal power-procurement problem in a discrete-time setting for isolated energy systems under an uncertain energy demand and incoming renewable energy [7,8,9]. The corresponding optimization problem for wireless networks has additional difficulties due to the uncertainty in wireless channels and probabilistic constraints due to the network quality of service (QoS) [10,11,12,13]. The solution method for this problem in the current literature can be generally categorized into robust optimization and stochastic optimization. The robust optimization approach constructs a bounded uncertainty set encompassing all realizations of the stochastic variables and optimizes the worst-case objective function value [14,15,16,17]. Robust optimization, while ensuring feasibility under the worst-case scenario, often leads to overly conservative policies that could underutilize renewable energy resources. The stochastic optimization approach assumes a known probability distribution for the random variables and optimizes the expected value of the objective function over the probability distribution of the uncertain parameters [18,19].

Chance-constrained programming is a subclass of stochastic optimization methods, first introduced in [20,21]. This approach allows constraints to be violated with a certain prescribed probability or risk level. The standard approach to solving discrete chance-constrained problems is to replace the chance constraint with a conservative but tractable approximation using the Bernstein inequality [22,23]. This approach is well-suited for wireless networks, since the associated QoS constraint is commonly defined probabilistically, allowing the operator to balance between conservatism and optimality. However, the computational complexity of the method in the discrete-time setting increases exponentially for large datasets and complex problems. For example, the authors in [24,25,26] solved the resulting discrete optimization problem using heuristic techniques, yielding suboptimal solutions.

This work proposes a novel mathematical modeling and numerical framework for the optimal management of hybrid energy systems to power cellular wireless networks under uncertainty and chance constraints in continuous time for a short-term planning horizon, where the system comprises locally installed renewable energy sources, fossil fuel power stations, and a storage capacity modeled by a single battery. We employ stochastic differential equations (SDEs) to model the incoming instantaneous renewable power and wireless fading channels. Data-driven SDEs have been employed to model instantaneous wind [27,28,29,30] and solar power [31,32,33]. These SDE models follow a mean reversion to deterministic wind speed/solar irradiance forecasts. The parameters of the SDE characterizing its uncertainty are calibrated from historic discrepancies (data) between the actual and forecasted quantities. Moreover, we employ SDEs to model stationary wireless fading channels. The authors in [34,35,36,37] constructed and configured SDEs to reproduce characteristics of various fading channel models, primarily concerning their stationary distribution.

This work formulates and solves a time-continuous stochastic optimal control problem to derive a near-optimal policy minimizing the expected operating expenditure and carbon footprint of cellular base stations. We optimize for both cost minimization and environmental impact, introducing a trade-off parameter $w\in [0,1]$ to balance these objectives. The optimal control is subject to demand constraints, a probabilistic and time-pointwise QoS constraint, and the dynamics and capacity limits of energy systems and wireless channels. The probabilistic constraint prevents the direct application of the dynamic programming principle because the constraint controls the joint distribution of the random variables. The optimal policy must be a function of the current state and time for the standard dynamic programming lemma to hold, not a function of its distribution [38]. To overcome this problem, we pose the corresponding dual problem by introducing a time-continuous Lagrangian relaxation to penalize violations of the probabilistic constraint [39]. This dual formulation yields a standard time-continuous stochastic optimization problem solved iteratively by optimizing the dual function. Each iteration involves solving the Hamilton–Jacobi–Bellman (HJB) partial differential equation (PDE) [40,41] using an upwind finite-difference scheme [42] to compute the dual objective value and subgradient. With access to only noisy subgradients, we solve the convex, nonsmooth dual problem using a novel combination of the limited memory bundle method (LMBM) [43,44,45] and the stochastic subgradient method [46,47]. Moreover, we design a refinement strategy for the Lagrangian multipliers to control the pointwise violation in the chance constraint. The numerical experiments and results based on the German power system and daily cellular traffic profiles validate the efficiency of the proposed models and approach.

The contributions can be summarized as follows:

• We propose a novel time-continuous optimization framework for optimal power procurement for green wireless cellular networks, subject to SDE dynamics and chance constraints. Compared to the discrete-time formulation in previous studies [24,25,26], the proposed approach decouples the model development from the numerical approximation, enhancing the model fidelity (see Remark 3). This formulation also yields a continuous control curve over time, allowing its application for any time discretization scheme and eliminating the need for ad hoc interpolations [48].
• We calibrate the data-driven SDE model developed for instantaneous wind power in [30] using German wind power data from the year 2023. The calibrated SDE is a driving dynamic for the stochastic optimal control problem.
• We derive an SDE to model the instantaneous Nakagami wireless fading channel [49] with a shifted-gamma invariant distribution, using the Pearson class of diffusions [50]. This SDE is a driving dynamic for the stochastic optimal control problem.
• We apply Lagrangian relaxation to the probabilistic QoS constraint, transforming the problem into a standard time-continuous stochastic optimization problem. Studies have explored numerical methods for time-continuous stochastic optimization with final-time chance constraints using Lagrangian relaxation [51,52,53] or reformulation as a stochastic target problem [54,55]. However, the proposed approach is novel in addressing a chance constraint that must be satisfied at every time point. Moreover, we implement this within the context of cellular wireless networks.
• We develop an iterative algorithm to optimize the dual function within a finite-dimensional function class numerically. Each iteration involves solving the HJB PDE to compute the dual function value and its noisy subgradient. The proposed approach extends the work in [48] on the time-continuous deterministic optimization of coupled hydrothermal power systems to the stochastic setting.
• With access only to noisy realizations of the subgradient, we combine the LMBM method [56] with the stochastic subgradient method [47] to solve the dual optimization problem. We iteratively refine the Lagrange multiplier to ensure compliance with the chance constraint at every time point.

The outline of the paper is as follows. Section 2 introduces the complete system model, detailing the costs, power generation, wireless network, and state dynamics. Next, Section 3 formulates the primal stochastic optimal control problem and the associated dual problem by Lagrangian relaxation of the chance constraint. Then, Section 4 introduces the numerical approach for solving the dual problem and presents all components of the final algorithm in detail. This section discusses the numerical solution of the HJB PDE, subgradient estimation, dual-problem optimization, and numerical errors arising from the proposed approach. Finally, Section 5 demonstrates the numerical results of applying the above approach when solving the stochastic optimization problem for a cellular base station based on the German power system and daily cellular traffic profiles.

2. System Model

This work considers a cellular network of multiple noninteracting base stations, each powered by renewable energy sources (solar panels or wind turbines) and battery storage. Figure 1 in [25] depicts the schematic network. Figure 1 illustrates the electrical power flow in each base station, each with a battery that stores the power generated from renewables ($P_R$). The power transmitted by the base station to serve cellular users consists of power procured from the battery ($P_A$) and bought from the grid ($P_F$). Any extra energy stored in the battery could be sold back to the grid ($P_S$) for revenue. We aim to optimize the energy procurement at each base station for a one-day operation cycle.

2.1. Base Station Model

The instantaneous power procured by the mobile operator from the traditional grid to power the base station is denoted by $\{P_F\left(t\right)\in {\mathbb{R}}^{+}:t\in [0,T]\}$, where 0 and T denote the time at the beginning and end of the day, respectively. The corresponding unit price is denoted by $\{K_b\left(t\right)\in {\mathbb{R}}^{+}:t\in [0,T]\}$. This price can vary during the day depending on strategies followed by the stakeholders in the energy market. Moreover, the base station has its own internal source of power (i.e., renewable power generator) whose generated power is denoted by $\{P_R\left(t\right)\in {\mathbb{R}}^{+}:t\in [0,T]\}$. This power is assumed to be free of charge. The internal power source is assumed to have a maximum power output capacity denoted by ${\overline{P}}_R\in {\mathbb{R}}^{+}$. The base station has a battery, allowing the mobile operator to store the incoming renewable power. As the base station is interconnected with the traditional grid, the mobile operator chooses to use the stored renewable power to run the base station or sell the extra energy back to the grid for revenue. The instantaneous power drawn from the battery to run the base station is denoted by $\{P_A\left(t\right)\in {\mathbb{R}}^{+}:t\in [0,T]\}$. The instantaneous power from the battery sold back to the grid is denoted by $\{P_S\left(t\right)\in {\mathbb{R}}^{+}:t\in [0,T]\}$, and the corresponding unit price is denoted by $\{K_s\left(t\right)\in {\mathbb{R}}^{+}:t\in [0,T]\}$. We reasonably assume $K_s\left(t\right)\le K_b\left(t\right)$ for all $t\in [0,T]$. The instantaneous power transmitted by the base station to serve its users is denoted by $\{P_{\mathrm{tx}}^{\mathrm{tot}}\left(t\right)\in {\mathbb{R}}^{+}:t\in [0,T]\}$. We assume the total transmitted power is equally divided among each average mobile user connected to the network. The instantaneous power transmitted by the base station per average user is denoted by $\{P_{\mathrm{tx}}\left(t\right)\in {\mathbb{R}}^{+}:t\in [0,T]\}$ and $P_{\mathrm{tx}}^{\mathrm{tot}}\left(t\right)=N_u\left(t\right)P_{\mathrm{tx}}\left(t\right)$ for all $t\in [0,T]$, where $\{N_u\left(t\right)\in {\mathbb{R}}^{+}:t\in [0,T]\}$ is the number of connected mobile users at time t. The instantaneous power balance equation for the base station is

$$P_A\left(t\right)+P_F\left(t\right)=C_{\mathrm{scal}}{N}_u\left(t\right)P_{\mathrm{tx}}\left(t\right)+C_{\mathrm{offset}},0\le t\le T,$$

(1)

where $C_{\mathrm{scal}}\in {\mathbb{R}}^{+}$ represents a factor that scales with the transmitted power due to the amplifier and feeder losses in the network, and $C_{\mathrm{offset}}\in {\mathbb{R}}^{+}$ models the offset power required to operate the base station irrespective of the transmitted power [25]. We assume that the base station can transmit a maximum power of ${\overline{P}}_{\mathrm{tx}}\in {\mathbb{R}}^{+}$.

$$P_{\mathrm{tx}}^{\mathrm{tot}}\left(t\right)\le {\overline{P}}_{\mathrm{tx}},0\le t\le T·$$

(2)

The term $N_u\left(t\right)$ in (1) is determined by the choice of the daily mobile user traffic model, heavily influencing the power demand in the base station. One such model is introduced in Section 5.1.

2.2. Cellular Network Model

According to the power law, the power transmitted by the base station decays with increased distance from the base station. Moreover, the transmitted power is uncertain due to atmospheric conditions, natural and human-made environmental obstacles, and interference from other signals [11]. Hence, the instantaneous power received by a user u at position ${\mathbf{x}}_u\left(t\right)=(x_u\left(t\right),y_u\left(t\right))\in {\mathbb{R}}^2$ from a base station at ${\mathbf{x}}_{\mathrm{BS}}=(x_{\mathrm{BS}},y_{\mathrm{BS}})\in {\mathbb{R}}^2$ is given as $P_{\mathrm{rx}}^u\left(t\right)=P_{\mathrm{tx}}\left(t\right)\xi \left(t\right)\kappa \left|\right|{{\mathbf{x}}_u\left(t\right)-{\mathbf{x}}_{\mathrm{BS}}\left|\right|}^{-\eta }$, where $\kappa ,\eta \in {\mathbb{R}}^{+}$ denotes the path loss constant and exponent, respectively. Moreover, $\left|\right|·\left|\right|$ denotes the Euclidean norm. $\xi :[0,T]\times \mathit{\Omega }\to [\underline{\xi },\infty )$ represents a stochastic process, that is greater than $\underline{\xi }\in {\mathbb{R}}^{+}$ almost surely (a.s.), modeling the wireless fading channel.

This work assumes the channel follows Nakagami fading [49]. The Nakagami fading model is a good data-fit model for cellular systems in urban and suburban areas [57]. In this model, the instantaneous power is gamma distributed, a Pearson type III distribution [58]. Hence, Pearson diffusion [50] is employed to model the wireless fading channel. Thus, this work constructs an SDE for the process $\xi$ (3) with a mean-reverting linear drift and a squared diffusion, which is a second-order polynomial of the state, and whose invariant distribution is the shifted-gamma distribution [50] with shape parameter $\mu >1$ and scale parameter $\theta \in {\mathbb{R}}^{+}$. We consider the probability space $\{\mathit{\Omega },\mathcal{F},{\left\{{\mathcal{F}}_t\right\}}_{t\ge 0},P\}$, where ${\mathcal{F}}_t$ is the filtration of the standard Wiener process $W:[0,T]\times \mathit{\Omega }\to \mathbb{R}$:

$$\{\begin{array}{cc}\hfill \mathrm{d}\xi \left(t\right)& =-\theta \left(\xi \left(t\right)-\underline{\xi }-\mu \right)\mathrm{d}t+\sqrt{2\theta (\xi \left(t\right)-\underline{\xi })}\mathrm{d}W\left(t\right),t>0\hfill \\ \hfill \xi \left(0\right)& \sim {\mu }_0^{\xi },\hfill \end{array}.$$

(3)

where ${\mu }_0^{\xi }$ is the invariant distribution of the process $\xi$.

Remark 1 (Computational efficiency of Pearson diffusions). 

Pearson diffusion naturally models the continuous-time wireless fading channel, where process $\xi$ lies in $\left[\underline{\xi },\infty \right)$ for all $t\in [0,T]$. This model also has the advantage of analytical tractability, providing closed-form solutions for the invariant distribution of the process $\xi$ and offering better insight into its behavior. This approach makes the model choice computationally more efficient than other Markov-based models (e.g., discrete-state Markov chains), whose complexity increases quadratically as the number of possible states increases.

In this work, the network performance is characterized by the offered QoS, determined by the signal-to-noise ratio (SNR) threshold ${\mathrm{SNR}}_{\mathrm{th}}$. The SNR, in decibels (dB), at user u at ${\mathbf{x}}_u\left(t\right)$ served by the base station can be expressed as follows:

$${\mathrm{SNR}}^u\left(t\right)=10{log}_{10}\left({\displaystyle \frac{P_{\mathrm{rx}}^u\left(t\right)}{{\sigma }_0}}\right)=10{log}_{10}\left({\displaystyle \frac{P_{\mathrm{tx}}\left(t\right)\xi \left(t\right)\kappa \left|\right|{{\mathbf{x}}_u\left(t\right)-{\mathbf{x}}_{\mathrm{BS}}\left|\right|}^{-\eta }}{{\sigma }_0}}\right)·$$

(4)

The ambient noise level is denoted by ${\sigma }_0\in {\mathbb{R}}^{+}$. The QoS of the base station is determined by the proportion of users whose SNR is above ${\mathrm{SNR}}_{\mathrm{th}}$. A user u is defined to be in outage at time t if ${\mathrm{SNR}}^u\left(t\right)<{\mathrm{SNR}}_{\mathrm{th}}$. The proportion of users in outage is denoted by $\{{\varphi }_{\mathrm{out}}\left(t\right)\in [0,1]:t\in [0,T]\}$ and can be expressed as follows:

$${\varphi }_{\mathrm{out}}\left(t\right)={\displaystyle \frac{1}{N_u\left(t\right)}}\sum _{u=1}^{N_u\left(t\right)}{\mathbb{1}}_{\{{\mathrm{SNR}}^u\left(t\right)<{\mathrm{SNR}}_{\mathrm{th}}\}},$$

(5)

where ${\varphi }_{\mathrm{out}}\left(t\right)$ can be interpreted as an empirical average over a user distribution. The following approximation assumes that each user is identical and that sufficient number of mobile users exist at all times:

$$\begin{array}{cc}\hfill {\varphi }_{\mathrm{out}}\left(t\right)& ={\displaystyle \frac{1}{N_u\left(t\right)}}\sum _{u=1}^{N_u\left(t\right)}{\mathbb{1}}_{\{{\mathrm{SNR}}^u\left(t\right)<{\mathrm{SNR}}_{\mathrm{th}}\}}\hfill \\ \hfill & ={\displaystyle \frac{1}{N_u\left(t\right)}}\sum _{u=1}^{N_u\left(t\right)}{\mathbb{1}}_{\left\{10{log}_{10}\left({\displaystyle \frac{P_{\mathrm{tx}}\left(t\right)\xi \left(t\right)\kappa \left|\right|{{\mathbf{x}}_u\left(t\right)-{\mathbf{x}}_{\mathrm{BS}}\left|\right|}^{-\eta }}{{\sigma }_0}}\right)<{\mathrm{SNR}}_{\mathrm{th}}\right\}}\hfill \\ \hfill & \approx {\int }_{{\mathbb{R}}^2}{\mathbb{1}}_{\left\{\left|\right|z-{\mathbf{x}}_{\mathrm{BS}}\left|\right|>{\left({\displaystyle \frac{P_{\mathrm{tx}}\left(t\right)\xi \left(t\right)\kappa }{{\sigma }_0{10}^{\frac{{\mathrm{SNR}}_{\mathrm{th}}}{10}}}}\right)}^{{\displaystyle \frac{1}{\eta }}}\right\}}{\rho }_z\left(t\right)z\mathrm{d}z,\hfill \end{array}$$

(6)

where ${\rho }_z\left(t\right)$ denotes the distribution of mobile users at time t. In (6), ${\varphi }_{\mathrm{out}}\left(t\right)$ is the complementary cumulative distribution function (ccdf) of the user distribution. This quantity can be computed (even analytically) for many well-known distributions.

The objective of the network operator is to ensure ${\varphi }_{\mathrm{out}}\left(t\right)<{\varphi }_{\mathrm{th}}$ for all $t\in [0,T]$, where $0<{\varphi }_{\mathrm{th}}\ll 1$ represents a threshold ratio. Due to the uncertainty associated with channel fading, achieving the objective in an a.s. sense is infeasible. Instead, we introduce the following chance constraint for the network QoS.

$$\mathbb{P}\left[{\varphi }_{\mathrm{out}}\left(t\right)\ge {\varphi }_{\mathrm{th}}\right]\le ϵ,0\le t\le T,$$

(7)

where $1-ϵ$, for $0<ϵ\ll 1$, is the confidence level of satisfying the constraint in (7). The network operator also generates revenue from servicing the users throughout the day:

$${\mathcal{R}}_1=\mathbb{E}\left[{\int }_0^T{K}_{\mathrm{net}}\left(t\right)N_u\left(t\right)(1-{\varphi }_{\mathrm{out}}\left(t\right))\mathrm{d}t\right],$$

(8)

where $\{K_{\mathrm{net}}\left(t\right)\in {\mathbb{R}}^{+}:t\in [0,T]\}$ denotes the unit price paid by users to connect to the network, and $N_u\left(t\right)(1-{\varphi }_{\mathrm{out}}\left(t\right))$ represents the number of users that are connected to the network (not in outage).

2.3. Renewable Power Model

This work also incorporates the uncertainty in renewable power generation. We apply a data-driven parametric SDE, whose solution defines a stochastic process that models the error in a given forecast for renewable power [30]. Let the process $R:[0,T]\times \mathit{\Omega }\to [0,1]$ denote the normalized generated renewable power.

$$\{\begin{array}{cc}\hfill \mathrm{d}R\left(t\right)& =\left(\dot{p}\left(t\right)-\theta \left(t\right)(R\left(t\right)-p\left(t\right))\right)\mathrm{d}t+\sqrt{2\alpha {\theta }_{0}R\left(t\right)(1-R\left(t\right))}\mathrm{d}W\left(t\right),t>0\hfill \\ \hfill R\left(0\right)& \sim {\mu }_0^R,\hfill \end{array}.$$

(9)

where $\left\{p\right(t)\in [0,1]:t\in [0,T\left]\right\}$ is a deterministic forecast for the normalized power provided by an official source and $\{\dot{p}\left(t\right):t\in [0,T]\}$ is its time derivative. In addition, $\left\{\theta \right(t):t\in [0,T\left]\right\}$ denotes the mean-reversion time-varying parameter ensuring that the process R is the unique strong solution to (9) for all $t\in [0,T]$ with range $[0,1]$, a.s. Moreover, $\alpha ,{\theta }_0\in {\mathbb{R}}^{+}$ are parameters inferred from discrepancies between historical real-world observations and their forecasts. In addition, $W:[0,T]\times \mathit{\Omega }\to \mathbb{R}$ is a standard Wiener process. The drift function in (9) is designed such that the process reverts to its mean $p\left(t\right)$, with a time-varying speed $\theta \left(t\right)$ proportional to the deviation of the process $R\left(t\right)$ from its mean, and it tracks the time derivative $\dot{p}\left(t\right)$. The diffusion function in (9) is designed so that the process $R\left(t\right)$ avoids exiting from the range $[0,1]$. The net renewable power generated is then given by $P_R\left(t\right)={\overline{P}}_{R}R\left(t\right)$ for all $t\in [0,T]$.

2.4. Battery Model

This work uses a simple model for the battery based on a circuit model of the state of the charge of a lithium-ion battery [59]. We assume that it has no energy loss when used and that no associated operational costs or aging effects exist. Process $A:[0,T]\to [0,1]$ denotes the normalized charge in the battery:

$$\{\begin{array}{cc}\hfill \mathrm{d}A\left(t\right)& ={\displaystyle \frac{\left(P_R\left(t\right)-P_S\left(t\right)-P_A\left(t\right)\right)}{\overline{A}}}\mathrm{d}t\hfill \\ \hfill A\left(0\right)& =A_0\hfill \\ \hfill -{\underline{P}}_A\left(A\left(t\right)\right)& \le P_A+P_S-P_R\le {\overline{P}}_A\left(A\left(t\right)\right),\hfill \end{array}.$$

(10)

where $\overline{A}$ represents the maximum battery charge capacity, ${\underline{P}}_A$ denotes the maximum power the battery can absorb, and ${\overline{P}}_A$ indicates the maximum power that the battery can supply. These quantities are characteristic of the battery. Generally, the battery can supply more power than it can absorb; hence, ${\underline{P}}_A<{\overline{P}}_A$ for all $t\in [0,T]$. In addition, $A_0\in [0,1]$ represents the normalized charge in the battery at the start of the day. The requirement that $0\le A\left(t\right)\le 1$ for all $t\in [0,T]$ is enforced via constraints on charging and discharging at the extremes:

$$\left\{\begin{array}{cc}{P}_R\left(t\right)-P_S\left(t\right)-P_A\left(t\right)\hfill & \ge 0,\mathrm{if}A\left(t\right)=0\hfill \\ P_R\left(t\right)-P_S\left(t\right)-P_A\left(t\right)\hfill & \le 0,\mathrm{if}A\left(t\right)=1·\hfill \end{array}\right.$$

(11)

The network operator generates revenue by selling some energy stored in the battery throughout the day:

$${\mathcal{R}}_2=\mathbb{E}\left[{\int }_0^T{K}_s\left(t\right)P_S\left(t\right)\mathrm{d}t\right]·$$

(12)

Remark 2 (Battery model improvements). 

Although the battery model (10) is straightforward and idealistic, realistic battery efficiency losses in the model can be incorporated by adding simple energy loss terms in the objective function of the optimal control problem. For example, this adjustment could include Ohmic efficiency losses [60] as a polynomial function of the battery discharge ($P_A+P_S-P_R$), and state-of-charge losses [61] as an exponential function of the normalized battery charge A, which is outside the current scope of the work.

2.5. Grid Power Model

The network operator incurs the following cost of buying power from the traditional grid throughout the day:

$${\mathcal{C}}_1=\mathbb{E}\left[{\int }_0^T{K}_b\left(t\right)P_F\left(t\right)\mathrm{d}t\right]·$$

(13)

The power from the traditional grid is primarily generated by fossil fuel stations, and has a highly negative environmental effect. This work does not impose an upper limit on how much power could be bought from the grid. To restrict the green cellular network from buying substantial amounts of grid power, we also impose an additional "environmental" cost of buying fossil fuel-based power [62].

$${\mathcal{C}}_2=\mathbb{E}\left[{\int }_0^T\left(C_1{P}_F\left(t\right)+C_2{P}_F{\left(t\right)}^2\right)\mathrm{d}t\right],$$

(14)

where $C_1\in {\mathbb{R}}^{+}$ and $C_2\in {\mathbb{R}}^{+}$ denote the pollutant emission coefficients of grid power.

2.6. Running Horizon Framework

Although this work aims to formulate an optimal power-procurement problem for a one-day operation cycle, such a short-term approach could lead to spurious results at the end of the day. For example, the resulting optimal policy might have operators selling all stored energy at the end of the day to generate revenue. This scenario is unrealistic because the operator would prefer to keep some charge in the battery for the next day. This work devises a running horizon framework to avoid producing such impractical optimal policies and still work in a short-term finite horizon. The framework consists of two facets. First, we add fictitious revenue from storing battery charge at the end of the day. More specifically, we introduce a third revenue source, as follows:

$${\mathcal{R}}_3=\mathbb{E}\left[P_K\overline{A}A\left(T\right)\right],$$

(15)

where $P_K\in {\mathbb{R}}^{+}$ represents the fictitious cost per unit of stored battery charge. For example, publicly available deterministic forecasts of normalized renewable power and day-ahead energy price forecasts can be applied to determine a suitable value of $P_K$.

Next, we formulate the optimal control problem for a two-day operation cycle (instead of one) but apply the optimal power-procurement policy only for one day. This approach requires two-day-ahead forecasts of renewable power, energy prices, and cellular user demand. The optimal power-procurement policy is updated daily, as and when the operators receive the two-day-ahead forecasts of these quantities. Regarding the problem formulation, the only change is that the time horizon is denoted by $[0,T]$, where 0 denotes beginning of the day, and T denotes the end of the second day (instead of at the end of the current day). Henceforth, this notation is applied while formulating the optimization problem.

2.7. Model Summary

The state and control vectors are denoted by $\mathbf{X}\left(t\right)$ and $\mathit{\varphi }\left(t\right)$, respectively:

$$\begin{array}{cc}\hfill \mathbf{X}\left(t\right)& :=\left[A\left(t\right),R\left(t\right),\xi \left(t\right)\right],\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill \mathit{\varphi }\left(t\right)& :=\left[P_A\left(t\right),P_F\left(t\right),P_{\mathrm{tx}}\left(t\right),P_S\left(t\right)\right]·\hfill \end{array}$$

(17)

The state vector $\mathbf{X}\left(t\right)$ is defined on a probability space $(\mathit{\Omega },\mathcal{F},P)$ and each of its components satisfy the controlled dynamics in (3), (9), and (10), respectively, in $[0,T]$. The initial conditions are also random. The control $\mathit{\varphi }\left(t\right):=\mathit{\varphi }(t,\mathbf{X}(t\left)\right)$ is an ${\mathcal{F}}_t$-adapted Markovian control that is measurable, where $\{{\mathcal{F}}_t,t\ge 0\}$ represents the filtration on the probability space.

Constraints 1 (A.s. instantaneous control constraints). 

The controls must satisfy the following constraints for all $t\in [0,T]$:

$$\begin{array}{cc}\hfill 0& \le P_A\left(t\right),\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill 0& \le P_F\left(t\right),\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill 0& \le P_{\mathrm{tx}}\left(t\right)\le {\displaystyle \frac{{\overline{P}}_{\mathrm{tx}}}{N_u\left(t\right)}},\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill 0& \le P_S\left(t\right)·\hfill \end{array}$$

(21)

The controls must also follow the following state-dependent constraint:

$$-{\underline{P}}_A\left(A\left(t\right)\right)\le P_A\left(t\right)+P_S\left(t\right)-{\overline{P}}_{R}R\left(t\right)\le {\overline{P}}_A\left(A\left(t\right)\right)·$$

(22)

Furthermore, for a given deterministic forecast of the number of cellular users connected to the network, $N_u\left(t\right)$, the controls are constrained to satisfy the following:

$$P_A\left(t\right)+P_F\left(t\right)=C_{\mathrm{scal}}{N}_u\left(t\right)P_{\mathrm{tx}}\left(t\right)+C_{\mathrm{offset}},0\le t\le T·$$

(23)

Constraints 2 (Probabilistic instantaneous control constraints). 

The controls must satisfy the following probabilistic constraint for all $t\in [0,T]$:

$$\mathbb{P}\left[{\varphi }_{\mathit{out}}\left(t\right)\ge {\varphi }_{\mathrm{th}}\right]\le ϵ·$$

(24)

The shorthand notation ${\varphi }_{\mathrm{out}}\left(t\right):={\varphi }_{\mathrm{out}}(t,\mathbf{X}\left(t\right),\mathit{\varphi }\left(t\right))$ is used here. The constraints in (18) to (24) define the set of admissible Markov controls $\mathcal{A}(t,\mathbf{X}(t\left)\right)$ at each time $t\in [0,T]$. Then, we define the set of admissible policies as follows:

$$\overline{\mathcal{A}}=\left\{\mathit{\varphi }:\mathit{\varphi }\mathrm{is}{\mathcal{F}}_t-\mathrm{adapted},\mathit{\varphi }(t,\omega )\in \mathcal{A}(t,\mathbf{X}(t,\omega )),\forall \omega \in \mathit{\Omega },\forall t\in [0,T]\right\}·$$

(25)

3. Stochastic Optimal Control Formulation

Problem 1 (Primal problem with the probabilistic constraint). 

Given the initial data $\mathbf{X}\left(0\right)=[A_0,R_0\sim {\mu }_0^R,{\xi }_0\sim {\mu }_0^{\xi }]$, deterministic forecasts for daily cellular user traffic profile $N_u\left(t\right)$, energy spot prices $K_b\left(t\right)$ and $K_s\left(t\right)$, renewable energy forecast $p\left(t\right)$, and mobile network usage price $K_{\mathit{net}}\left(t\right)$ for $0\le t\le T$, we solve the following:

$${\mathit{\varphi }}^{*}=\underset{\{\mathit{\varphi }\in \overline{\mathcal{A}}\}}{argmin}\mathcal{U}\left(\mathit{\varphi }\right)=w\left({\mathcal{C}}_1-{\mathcal{R}}_1-{\mathcal{R}}_2-{\mathcal{R}}_3\right)+(1-w){\mathcal{C}}_2,$$

(26)

where the minimization is done over all controls satisfying Constraints 1 and 2 with dynamics (3), (9), and (10). We define $\mathcal{U}\left(\mathit{\varphi }\right)$ as the net utility of the base station. In addition, $w\in [0,1]$ is a Pareto parameter to weigh the two objectives: financial cost $({\mathcal{C}}_1-{\mathcal{R}}_1-{\mathcal{R}}_2)$ and environmental cost (${\mathcal{C}}_2$), where w can be chosen by the base station operators, depending on how environmentally conscious they are. The dependence of the costs on the controls $\mathit{\varphi }$ in (26) is suppressed for ease of notation.

Remark 3 (Discrete vs. continuous-time formulation). 

The optimal solution to a discrete-time formulation leads to a discrete set of control decisions at each time step, dependent on the selected discretization. Applying the dynamic programming principle to a discrete-time formulation leads to a stochastic optimization problem at each time step. Finer grids substantially increase the computational work, making the problem computationally infeasible without guaranteeing that the optimal policies would converge. In a continuous-time formulation, the limit of the dynamic programming principle exists as the time step becomes infinitesimally small, leading to the derivation of the associated HJB PDE. Numerical discretization methods are applied just once to solve this PDE and obtain a solution at all time points. This approach facilitates discretization error control, ensuring convergence to the true optimal solution, and enables the flexible application of adaptive and higher-order numerical approximation schemes.

The probabilistic constraint in 2 does not allow classical dynamic programming to solve Problem 1 because the constraint controls the joint distribution of the Markovian dynamics, rather than each realization. The standard dynamic programming lemma [38] cannot be applied if the Markovian control function depended additionally on the state’s distribution. We overcome this problem using Langrangian relaxation of the probabilistic constraint for all $t\in [0,T]$, yielding a continuous-time Lagrangian relaxation. For this approach, we define the time-continuous deterministic Lagrange multiplier function $\lambda :[0,T]\to {\mathbb{R}}^{+}$, associated with the relaxed constraint in 2. We let $\mathit{\Lambda }$ denote the admissible space of Lagrange multiplier functions. The new set of admissible controls is only defined by the constraints in (18)–(23) and is denoted by ${\mathcal{A}}^{\mathrm{rel}}(t,\mathbf{X}\left(t\right))$ at each time $t\in [0,T]$. Then, we define the new set of admissible policies as follows:

$${\overline{\mathcal{A}}}^{\mathrm{rel}}=\left\{\mathit{\varphi }:\mathit{\varphi }\mathrm{is}{\mathcal{F}}_t-\mathrm{adapted},\mathit{\varphi }(t,\omega )\in {\mathcal{A}}^{\mathrm{rel}}(t,\mathbf{X}(t,\omega )),\forall \omega \in \mathit{\Omega },\forall t\in [0,T]\right\}·$$

(27)

We define the Lagrangian using terminology from the classical Lagrangian relaxation technique for constrained optimization problems [63] as follows:

$$\begin{array}{cc}\hfill \mathcal{L}(\mathit{\varphi },\lambda )& =\mathcal{U}\left(\mathit{\varphi }\right)+{\int }_0^T\lambda \left(t\right)\left(\mathbb{P}\left[{\varphi }_{\mathrm{out}}(t,\mathbf{X}\left(t\right),\mathit{\varphi }\left(t\right))\ge {\varphi }_{\mathrm{th}}\right]-ϵ\right)\mathrm{d}t\hfill \\ \hfill & =\mathcal{U}\left(\mathit{\varphi }\right)+{\int }_0^T\mathbb{E}\left[\lambda \left(t\right)\left({\mathbb{1}}_{\{{\varphi }_{\mathrm{out}}(t,\mathbf{X}\left(t\right),\mathit{\varphi }\left(t\right))\ge {\varphi }_{\mathrm{th}}\}}-ϵ\right)\right]\mathrm{d}t·\hfill \end{array}$$

(28)

Problem 2 (Relaxed problem). 

Given the initial data $\mathbf{X}\left(0\right)=[A_0,R_0\sim {\mu }_0^R,{\xi }_0\sim {\mu }_0^{\xi }]$, Lagrange multiplier function $\lambda$, deterministic forecasts for daily cellular user traffic profile $N_u\left(t\right)$, energy spot prices $K_b\left(t\right)$ and $K_s\left(t\right)$, renewable energy forecast $p\left(t\right)$, and mobile network usage price $K_{\mathit{net}}\left(t\right)$ for $0\le t\le T$, we solve the following:

$${\mathit{\varphi }}^{*}\left(\lambda \right)=\underset{\{\mathit{\varphi }\in {\overline{\mathcal{A}}}^{\mathrm{rel}}\}}{argmin}\mathcal{L}(\mathit{\varphi },\lambda ),$$

(29)

where the minimization is done over all controls satisfying the constraint in 1 with dynamics (3), (9), and (10).

With the relaxation of the probabilistic constraint in 2, the dynamic programming principle can be applied to solve Problem 2. The dual problem associated with the primal Problem 1 is formulated below.

Problem 3 (Dual problem). 

We determine

$${\lambda }^{*}=\underset{\lambda \in \mathit{\Lambda }}{argmax}\mathit{\Theta }\left(\lambda \right),$$

(30)

where the dual function $\mathit{\Theta }\left(\lambda \right)$ is given by

$$\mathit{\Theta }\left(\lambda \right)=\mathcal{L}({\mathit{\varphi }}^{*}\left(\lambda \right),\lambda ),$$

(31)

and ${\mathit{\varphi }}^{*}\left(\lambda \right)$ solves Problem 2 for a given $\lambda$.

The admissible set $\mathit{\Lambda }$ of Lagrange multiplier functions depends on the problem structure. For the considered optimization problem, $\lambda$ denotes a positive real-valued function of time t due to the deterministic nature of the relaxed constraint (24).

3.1. HJB Equation Related to Problem 2

As a first step to solving the dual problem (Problem 3), Problem 2 is solved to obtain $\mathit{\Theta }\left(\lambda \right)$ for a given $\lambda$. Using terminology from standard continuous-time stochastic optimal control [38,40], we first define the cost-to-go function $u:[0,T]\times \mathit{\Gamma }\to \mathbb{R}$ associated with Problem 2 as follows:

$$\begin{array}{cc}\hfill u(t,\mathbf{x})& =\underset{\mathit{\varphi }\in {\overline{\mathcal{A}}}^{\mathrm{rel}}}{min}{\mathcal{U}}_{t,\mathbf{x}}\left(\mathit{\varphi }\right)\hfill \\ \hfill & =\underset{\mathit{\varphi }\in {\overline{\mathcal{A}}}^{\mathrm{rel}}}{min}\mathbb{E}\left[{\int }_t^T\right(w\left(K_b\left(s\right)P_F\left(s\right)-K_s\left(s\right)P_S\left(s\right)-K_{\mathrm{net}}\left(s\right)N_u\left(s\right)(1-{\varphi }_{\mathrm{out}}(s,\xi \left(s\right),P_{\mathrm{tx}}\left(s\right)))\right)\hfill \\ \hfill & +(1-w)\left(C_1{P}_F\left(s\right)+C_2{P}_F{\left(s\right)}^2\right)+\lambda \left(s\right)\left({\mathbb{1}}_{\{{\varphi }_{\mathrm{out}}(s,\xi \left(s\right),P_{\mathrm{tx}}\left(s\right))\ge {\varphi }_{\mathrm{th}}\}}-ϵ\right))\mathrm{d}s\hfill \\ \hfill & -P_K\overline{A}A\left(T\right)|\mathbf{X}\left(t\right)=\mathbf{x}]·\hfill \end{array}$$

(32)

We define the components of $\mathbf{x}=[a,r,\chi ]$, where a, r, and $\chi$ denote the variables corresponding to components of the state vector $A\left(t\right)$, $R\left(t\right)$, and $\xi \left(t\right)$, respectively. In addition, $\mathit{\Gamma }$ denotes the domain of $\mathbf{x}$, in this case $[0,1]\times [0,1]\times [\underline{\xi },\infty )$, because $A\left(t\right)$ and $R\left(t\right)$ are $[0,1]$-valued random variables and $\xi \left(t\right)$ is a $[\underline{\xi },\infty )$-valued random variable for all $t\in [0,T]$ (see Equations (3), (9) and (10)). Using (32), the dual function is defined as $\mathit{\Theta }\left(\lambda \right)=u(0,\mathbf{X}(0\left)\right)$, where u, for given $\lambda$, solves the following second-order nonlinear HJB final value PDE.

$$\{\begin{array}{c}{\displaystyle \frac{\partial u}{\partial t}}+\mathcal{H}(t,\mathbf{x},\nabla u,{\nabla }^{2}u;\lambda )=0\hfill \\ u(T,\mathbf{x})=-P_K\overline{A}a,\forall \mathbf{x}\in \mathit{\Gamma },\hfill \end{array}.$$

(33)

where $\nabla ·$ denotes the gradient vector, and ${\nabla }^2·$ represents the Hessian matrix of a scalar-valued function. The Hamiltonian $\mathcal{H}$ associated with Problem 2 in (33) is defined as follows:

$$\begin{array}{cc}\hfill \mathcal{H}(t,\mathbf{x},\nabla u,{\nabla }^{2}u;\lambda )& =\underset{\mathit{\varphi }\in {\mathcal{A}}^{\mathrm{rel}}(t,\mathbf{x})}{min}[{\displaystyle \frac{\left({\overline{P}}_{R}r-P_S-P_A\right)}{\overline{A}}}{\displaystyle \frac{\partial u}{\partial a}}+\left(\dot{p}\left(t\right)-\theta \left(t\right)(r-p\left(t\right))\right){\displaystyle \frac{\partial u}{\partial r}}\hfill \\ \hfill & +\alpha {\theta }_{0}r(1-r){\displaystyle \frac{{\partial }^{2}u}{\partial r^2}}-\theta \left(\chi -\underline{\xi }-\mu \right){\displaystyle \frac{\partial u}{\partial \chi }}+\theta (\chi -\underline{\xi }){\displaystyle \frac{{\partial }^{2}u}{\partial {\chi }^2}}+w(K_b\left(t\right)P_F\hfill \\ \hfill & -K_s\left(t\right)P_S-K_{\mathrm{net}}\left(t\right)N_u\left(t\right)(1-{\varphi }_{\mathrm{out}}(t,\chi ,P_{\mathrm{tx}})))+(1-w)\left(C_1{P}_F+C_2{P}_F^2\right)\hfill \\ \hfill & +\lambda \left(t\right)\left({\mathbb{1}}_{\{{\varphi }_{\mathrm{out}}(t,\chi ,P_{\mathrm{tx}})>{\varphi }_{\mathrm{th}}\}}-ϵ\right)]·\hfill \end{array}$$

(34)

We obtain an approximation $\overline{\mathit{\Theta }}\left(\lambda \right)$ of the dual function $\mathit{\Theta }\left(\lambda \right)$ by numerically approximating the solution to (33). Section 4 discusses this approach in detail.

3.2. Finite-Dimensional Approximation of Problem 3

Problem 3 is an infinite-dimensional optimization problem to satisfy the constraint in 2 pointwise in time. We discretize $\lambda \left(t\right)$ into a piecewise constant function to approximate the infinite-dimensional dual problem, allowing for practical numerical optimization. This approach implies satisfying the constraint for a finite number of subintervals in $[0,T]$. That is, the integral of the pointwise violation of the constraint in 2 over each subinterval must be 0. We consider the time discretization $0=t_0<t_1<\dots <t_{\mathit{\ell }-1}<t_{\mathit{\ell }}=T$ of the time domain $[0,T]$. The Lagrange multiplier is approximated as follows:

$$\lambda \left(t\right)\approx {\lambda }^{\mathit{\ell }}\left(t\right)=\sum _{i=1}^{\mathit{\ell }}{\mathit{{\rm Y}}}_i^{\mathit{\ell }}{\mathbb{1}}_{\{t\in [t_{i-1},t_i]\}},\forall t\in [0,T],$$

(35)

where ${\left\{{\mathit{{\rm Y}}}_i^{\mathit{\ell }}\right\}}_{i=1}^{\mathit{\ell }}$ denotes components of the vector of Lagrange multipliers ${\mathit{{\rm Y}}}^{\mathit{\ell }}$. For the given time discretization, we define

$${\overline{\mathit{\Lambda }}}^{\mathit{\ell }}=\{{\lambda }^{\mathit{\ell }}:\mathrm{with}{\lambda }^{\mathit{\ell }}\left(t\right)\mathrm{defined}\mathrm{in}(35)\mathrm{and}{\mathit{{\rm Y}}}_i^{\mathit{\ell }}\in {\mathbb{R}}^{+}\forall i\in [1,\dots ,\mathit{\ell }]\}·$$

(36)

With this approximation, the finite-dimensional approximation of Problem 3 is formulated.

Problem 4 (Finite-dimensional dual problem). 

For a given ℓ, we determine the following:

$${\overline{\lambda }}^{\mathit{\ell }}=\underset{{\lambda }^{\mathit{\ell }}\in {\overline{\mathit{\Lambda }}}^{\mathit{\ell }}}{argmax}\mathit{\Theta }\left({\lambda }^{\mathit{\ell }}\right)·$$

(37)

The following Lagrangian is written for a given ${\lambda }^{\mathit{\ell }}\in {\overline{\mathit{\Lambda }}}^{\mathit{\ell }}$:

$$\mathcal{L}(\mathit{\varphi },{\lambda }^{\mathit{\ell }})=\mathcal{U}\left(\mathit{\varphi }\right)+\sum _{i=1}^{\mathit{\ell }}{\mathit{{\rm Y}}}_i^{\mathit{\ell }}{\left(\mathfrak{D}\mathit{\Theta }\left({\lambda }^{\mathit{\ell }}\right)\right)}_i$$

(38)

with

$${\left(\mathfrak{D}\mathit{\Theta }\left({\lambda }^{\mathit{\ell }}\right)\right)}_i=\mathbb{E}\left[{\int }_{t_{i-1}}^{t_i}\left({\mathbb{1}}_{\{{\varphi }_{\mathrm{out}}(t,\mathbf{X}\left(t\right),\mathit{\varphi }\left(t\right))\ge {\varphi }_{\mathrm{th}}\}}-ϵ\right)\mathrm{d}t\right],\forall i\in [1,\dots ,\mathit{\ell }],$$

(39)

where ${\left(\mathfrak{D}\mathit{\Theta }\left({\lambda }^{\mathit{\ell }}\right)\right)}_i$ denotes the $i^{\mathrm{th}}$ component of an ℓ-dimensional subgradient vector $\mathfrak{D}\mathit{\Theta }$ of the dual function $\mathit{\Theta }$ evaluated at ${\lambda }^{\mathit{\ell }}\in {\overline{\mathit{\Lambda }}}^{\mathit{\ell }}$. To solve Problem 4, the optimal ℓ-dimensional vector ${\mathit{{\rm Y}}}^{\mathit{\ell }}\in {\left({\mathbb{R}}^{+}\right)}^{\mathit{\ell }}$ that maximizes the dual function $\mathit{\Theta }$ is selected. Hence, Problem 4 is a convex, ℓ-dimensional optimization problem but is not generally smooth. Therefore, we apply subgradient methods to solve Problem 4. Moreover, the choice of ℓ and the corresponding time discretization grid is set to control the pointwise violation in Constraint 2. Section 4 details this approach.

4. Numerical Approach

4.1. Numerically Solving the HJB PDE

We employ the explicit Euler upwind finite-difference scheme to numerically approximate the solution to the HJB equation in (33). This method provides a convergent numerical scheme under certain conditions that can be proven for the considered problem [42].

We consider the following time and space discretization of $(t,\mathbf{x})\in [0,T]\times [0,1]\times [0,1]\times [\underline{\xi },\infty )$, where $\mathbf{x}=(a,r,\chi )$, and the following finite-difference grid $\tau ={\tau }_t\times {\tau }_a\times {\tau }_r\times {\tau }_{\chi }$:

$$\begin{array}{cc}\hfill & {\tau }_t:0=t_0<t_1<\dots <t_{N_t-1}<t_{N_t}=T,\Delta t={\displaystyle \frac{T}{N_t}}\hfill \\ \hfill & {\tau }_a:0=a_0<a_1<\dots <a_{N_1-1}<a_{N_1}=1,\Delta a={\displaystyle \frac{1}{N_1}}\hfill \\ \hfill & {\tau }_r:0=r_0<r_1<\dots <r_{N_2-1}<r_{N_2}=1,\Delta r={\displaystyle \frac{1}{N_2}}\hfill \\ \hfill & {\tau }_{\chi }:0={\chi }_0<{\chi }_1<\dots <{\chi }_{N_3-1}<{\chi }_{N_3}=1,\Delta \chi ={\displaystyle \frac{1}{N_3}}·\hfill \end{array}$$

(40)

For the dimension $\chi$, we approximate the domain $[\underline{\xi },\infty )$ by $[\underline{\xi },\overline{\chi }]$ for an appropriately chosen $\overline{\chi }$, which is scaled to $[0,1]$. We employ uniform grids in all dimensions; that is,

$$\begin{array}{cc}\hfill t_n& =n\Delta t,n=0,\dots ,N_t,\hfill \\ \hfill a_i& =i\Delta a,i=0,\dots ,N_1,\hfill \\ \hfill r_j& =j\Delta r,j=0,\dots ,N_2,\hfill \\ \hfill {\chi }_k& =k\Delta \chi ,k=0,\dots ,N_3·\hfill \end{array}$$

We combine the scaled drift and diffusion terms of dynamics (3), (9), and (10) into vector-valued functions F and G, respectively.

$$F(t,\mathbf{x},\mathit{\varphi })=\left[\begin{array}{c}{\displaystyle \frac{{\overline{P}}_{R}r-P_S-P_A}{\overline{A}}}\\ \dot{p}\left(t\right)-\theta \left(t\right)\left(r-p\left(t\right)\right)\\ -\theta \left(\chi -{\displaystyle \frac{\mu }{\overline{\chi }-\underline{\xi }}}\right)\end{array}\right],G(t,\mathbf{x},\mathit{\varphi })=\left[\begin{array}{c}0\\ \alpha {\theta }_{0}r(1-r)\\ {\displaystyle \frac{\theta \chi }{\overline{\chi }-\underline{\xi }}}\end{array}\right]·$$

For convenience, we define the following function representing the running cost in (34).

$$\begin{array}{cc}\hfill H(t,\mathbf{x},\mathit{\varphi })& =w\left(K_b\left(t\right)P_F-K_s\left(t\right)P_S-K_{\mathrm{net}}\left(t\right)N_u\left(t\right)(1-{\varphi }_{\mathrm{out}}(t,\chi ,P_{\mathrm{tx}}))\right)\hfill \\ \hfill & +(1-w)\left(C_1{P}_F+C_2{P}_F^2\right)+\lambda \left(t\right)\left({\mathbb{1}}_{\{{\varphi }_{\mathrm{out}}(t,\chi ,P_{\mathrm{tx}})\ge {\varphi }_{\mathrm{th}}\}}-ϵ\right)·\hfill \end{array}$$

For a function f, we define $f^{+}(t,\mathbf{x},\mathit{\varphi }):=max\left(f(t,\mathbf{x},\mathit{\varphi }),0\right)$ and $f^{-}(t,\mathbf{x},\mathit{\varphi }):=max\left(-f(t,\mathbf{x},\mathit{\varphi }),0\right)$. For a function f, we define $f_{(i,j,k)}^n:=f(t_n,[a_i,r_j,{\chi }_k],\mathit{\varphi }(t_n,[a_i,r_j,{\chi }_k]))$. Let ${\overline{u}}_{(i,j,k)}^n$ denote the numerical approximation of the cost-to-go function $u(t,\mathbf{x})$ at the point $(t_n,a_i,r_j,{\chi }_k)$ on the grid $\tau$. Then, the explicit update rule for the upwind finite-difference scheme is given as follows:

$$\begin{array}{cc}\hfill {\overline{u}}_{(i,j,k)}^{n-1}& ={\overline{u}}_{(i,j,k)}^n(1-2{\left(G_1\right)}_{(i,j,k)}^n{\displaystyle \frac{\Delta t}{\Delta r^2}}-2{\left(G_2\right)}_{(i,j,k)}^n{\displaystyle \frac{\Delta t}{\Delta {\chi }^2}}\hfill \\ & -{\displaystyle \frac{\Delta t}{\Delta a}}|{\left(F_1\right)}_{(i,j,k)}^n|-{\displaystyle \frac{\Delta t}{\Delta r}}|{\left(F_2\right)}_{(i,j,k)}^n|-{\displaystyle \frac{\Delta t}{\Delta \chi }}\left|{\left(F_3\right)}_{(i,j,k)}^n\right|)\hfill \\ & +{\overline{u}}_{(i+1,j,k)}^n{\displaystyle \frac{\Delta t}{\Delta a}}{\left(F_1^{+}\right)}_{(i,j,k)}^n+u_{(i-1,j,k)}^n{\displaystyle \frac{\Delta t}{\Delta a}}{\left(F_1^{-}\right)}_{(i,j,k)}^n\hfill \\ & +{\overline{u}}_{(i,j+1,k)}^n\left({\left(G_2\right)}_{(i,j,k)}^n{\displaystyle \frac{\Delta t}{\Delta r^2}}+{\left(F_2^{+}\right)}_{(i,j,k)}^n{\displaystyle \frac{\Delta t}{\Delta r}}\right)\hfill \\ & +{\overline{u}}_{(i,j-1,k)}^n\left({\left(G_2\right)}_{(i,j,k)}^n{\displaystyle \frac{\Delta t}{\Delta r^2}}+{\left(F_2^{-}\right)}_{(i,j,k)}^n{\displaystyle \frac{\Delta t}{\Delta r}}\right)\hfill \\ & +{\overline{u}}_{(i,j,k+1)}^n\left({\left(G_3\right)}_{(i,j,k)}^n{\displaystyle \frac{\Delta t}{\Delta {\chi }^2}}+{\left(F_3^{+}\right)}_{(i,j,k)}^n{\displaystyle \frac{\Delta t}{\Delta \chi }}\right)\hfill \\ & +{\overline{u}}_{(i,j,k+1)}^n\left({\left(G_3\right)}_{(i,j,k)}^n{\displaystyle \frac{\Delta t}{\Delta {\chi }^2}}+{\left(F_3^{-}\right)}_{(i,j,k)}^n{\displaystyle \frac{\Delta t}{\Delta \chi }}\right)+\Delta t{H}_{(i,j,k)}^n·\hfill \end{array}$$

(41)

$$\begin{array}{cc}\hfill {\overline{u}}_{(i,j,k)}^{N_t}& =-P_K\overline{A}{a}_i,\forall i,j,k·\hfill \end{array}$$

(42)

In (41), we apply the notation ${\left\{F_i\right\}}_{i=1}^3$ and ${\left\{G_i\right\}}_{i=1}^3$ to denote the $i^{\mathrm{th}}$ component of the vector-valued functions F and G, respectively. Optimal controls at grid point $(t_n,a_i,r_j,{\chi }_k)$, denoted by ${\left({\mathit{\varphi }}^{*}\right)}_{(i,j,k)}^n$, are computed using the known derivatives (up to the second order) of the numerical cost-to-go-function as follows:

$$\begin{array}{cc}\hfill {\left({\mathit{\varphi }}^{*}\right)}_{(i,j,k)}^n& =\underset{\mathit{\varphi }\in {\mathcal{A}}^{\mathrm{rel}}(t_n,[a_i,r_j,{\chi }_k])}{argmin}[F_1^{+}(t_n,[a_i,r_j,{\chi }_k],\mathit{\varphi }){\left({\Delta }_a^{+}\overline{u}\right)}_{(i,j,k)}^n+F_1^{-}(t_n,[a_i,r_j,{\chi }_k],\mathit{\varphi }){\left({\Delta }_a^{-}\overline{u}\right)}_{(i,j,k)}^n\hfill \\ \hfill & +H(t_n,[a_i,r_j,{\chi }_k],\mathit{\varphi })],\hfill \end{array}$$

(43)

where ${\left({\Delta }_a^{+}\overline{u}\right)}_{(i,j,k)}^n$ and ${\left({\Delta }_a^{-}\overline{u}\right)}_{(i,j,k)}^n$ represent the upwind derivatives given by

$${\left({\Delta }_a^{+}\overline{u}\right)}_{(i,j,k)}^n={\displaystyle \frac{{\overline{u}}_{(i+1,j,k)}^n-{\overline{u}}_{(i,j,k)}^n}{\Delta a}},{\left({\Delta }_a^{+}\overline{u}\right)}_{(i,j,k)}^n={\displaystyle \frac{{\overline{u}}_{(i,j,k)}^n-{\overline{u}}_{(i-1,j,k)}^n}{\Delta a}}·$$

(44)

In general, boundary conditions must be imposed on the HJB PDE (33) to obtain a unique numerical solution. However, boundary conditions at the boundaries $a=0,1$, $r=0,1$, and $\chi =0$ are unnecessary in this setting because the drift terms ${\left\{F_i\right\}}_{i=1}^3$ naturally have a sign at the boundaries ensuring the use of an interior point to approximate first-order derivatives. Similarly, the diffusion terms ${\left\{G_i\right\}}_{i=1}^3$ are naturally 0 at these boundaries, ensuring no exterior points are necessary to approximate second-order derivatives. At the boundary $\chi =\overline{\chi }$, we impose a nonreflective boundary condition to indicate that the actual domain is not bounded at $\chi =\overline{\chi }$. We impose the Courant–Friedrichs–Lewy (CFL) condition at each grid point in $\tau$ to ensure the stability of the numerical scheme

$$\Delta t\left({\displaystyle \frac{|{\left(F_1\right)}_{(i,j,k)}^n|}{\Delta a}}+{\displaystyle \frac{|{\left(F_2\right)}_{(i,j,k)}^n|}{\Delta r}}+{\displaystyle \frac{|{\left(F_3\right)}_{(i,j,k)}^n|}{\Delta \chi }}+{\displaystyle \frac{2{\left(G_2\right)}_{(i,j,k)}^n}{\Delta r^2}}+{\displaystyle \frac{2{\left(G_3\right)}_{(i,j,k)}^n}{\Delta {\chi }^2}}\right)\le 1·$$

(45)

Although the HJB PDE (33) is nonlinear, the numerical scheme given by Equations (41)–(43) is explicit because the cost-to-go function at time $t_{n-1}$ is completely determined by its value and derivatives at time $t_n$. However, this relationship is nonlinear and cannot be vectorized. Moreover, the four-dimensional (4D) constrained optimization problem in (43) must be solved at every grid point in $\tau$. Depending on the structure of ${\varphi }_{\mathrm{out}}(t,\chi ,P_{\mathrm{tx}})$, (43) can be solved analytically or numerically. Linear interpolation is applied to extend the numerical solution $\overline{u}$ from the grid $\tau$ to the entire domain $[0,T]\times \mathit{\Gamma }$. Next, the dual function $\mathit{\Theta }\left(\lambda \right)$ is approximated by $\overline{\mathit{\Theta }}\left(\lambda \right)=\overline{u}(0,\mathbf{X}\left(0\right))$ for a given $\lambda$. Algorithm A1 in Appendix A presents a pseudo-algorithm of the numerical solver.

Remark 4 (Discretization error of the PDE solver). 

The upwind finite-difference scheme in Section 4.1 is a first-order explicit scheme with a discretization error of $𝒪(\Delta t+\Delta a+\Delta r+\Delta \chi )$. When solving the overall optimization problem up to a relative tolerance of $\mathrm{TOL}$ with respect to the dual function value $\mathit{\Theta }\left(\lambda \right)$, the PDE discretization error must be $𝒪\left(\right|\mathit{\Theta }\left(\lambda \right)\left|\mathrm{TOL}\right)$. Thus, $\Delta t,\Delta a,\Delta r,\Delta \chi$ must be $𝒪\left(\right|\mathit{\Theta }\left(\lambda \right)\left|\mathrm{TOL}\right)$, implying the computational work of $𝒪\left({\left|\mathit{\Theta }\left(\lambda \right)\right|}^{-4}{\mathrm{TOL}}^{-4}\right)$ to solve the HJB PDE once.

4.2. Estimating Subgradient $\mathfrak{D}\mathit{\Theta }$

Upon solving the HJB PDE (33) numerically and obtaining an approximate cost-to-go function in the domain $[0,T]\times \mathit{\Gamma }$, we can numerically simulate optimally controlled paths of the state variable $\mathbf{X}\left(t\right)$ forward in time, from the initial condition $\mathbf{X}\left(0\right)$ using dynamics (3), (9), and (10). We use the Euler–Maruyama time discretization of the SDEs in (3), (9), and (10). We consider the discretization $0={\overline{t}}_0<{\overline{t}}_1<\dots <{\overline{t}}_{{\overline{N}}_t}=T$ of the time domain $[0,T]$ with ${\overline{N}}_t$ uniform time steps. It follows that ${\overline{t}}_n=n\times \Delta \overline{t},n=0,1,\dots ,{\overline{N}}_t$, and $\Delta \overline{t}={\displaystyle \frac{T}{{\overline{N}}_t}}$. If ${\overline{\mathbf{X}}}^{{\overline{N}}_t}$ is the time-discretized version of the stochastic process $\mathbf{X}$, then the Euler–Maruyama time discretization of the SDEs is expressed as follows:

$$\{\begin{array}{cc}\hfill {\overline{\mathbf{X}}}^{{\overline{N}}_t}\left({\overline{t}}_{n+1}\right)& ={\overline{\mathbf{X}}}^{{\overline{N}}_t}\left({\overline{t}}_n\right)+F({\overline{t}}_n,{\overline{\mathbf{X}}}^{{\overline{N}}_t}\left({\overline{t}}_n\right),{\mathit{\varphi }}^{*}({\overline{t}}_n,{\overline{\mathbf{X}}}^{{\overline{N}}_t}\left({\overline{t}}_n\right)))\Delta \overline{t}\hfill \\ & +G({\overline{t}}_n,{\overline{\mathbf{X}}}^{{\overline{N}}_t}\left({\overline{t}}_n\right),{\mathit{\varphi }}^{*}({\overline{t}}_n,{\overline{\mathbf{X}}}^{{\overline{N}}_t}\left({\overline{t}}_n\right))):\sqrt{\Delta \overline{t}}{\epsilon }^n,n=0,\dots ,{\overline{N}}_t-1\hfill \\ \hfill {\overline{\mathbf{X}}}^{{\overline{N}}_t}\left({\overline{t}}_0\right)& =\mathbf{X}\left(0\right),\hfill \end{array}.$$

(46)

where ${\epsilon }^n\sim \mathcal{N}(\mathbf{0},{\mathbb{I}}_3)$ for $n=0,\dots ,{\overline{N}}_t-1$, denotes independent and identically distributed (i.i.d.) three-dimensional standard normal random variables, and $·:·$ denotes elementwise multiplication of two vectors. The optimal controls ${\mathit{\varphi }}^{*}({\overline{t}}_n,{\overline{\mathbf{X}}}^{{\overline{N}}_t}\left({\overline{t}}_n\right))$ in (46) are obtained by minimizing (43) at the current point of the state ${\overline{\mathbf{X}}}^{{\overline{N}}_t}\left({\overline{t}}_n\right)$, which may be a point outside the grid $\tau$. The values of the numerical derivatives ${\Delta }_a^{\pm }\overline{u}$ of the cost-to-go function at those points are linearly interpolated from the computed derivative values at the nearest points in grid $\tau$. The subgradient of $\mathit{\Theta }$ at ${\lambda }^{\mathit{\ell }}\in {\overline{\mathit{\Lambda }}}^{\mathit{\ell }}$ is computed according to (39), approximating the expected value using Monte Carlo sampling with $M_{\mathrm{SG}}$ i.i.d. sample paths of the optimally controlled process ${\overline{\mathbf{X}}}^{{\overline{N}}_t}$. Moreover, the integral in (39) is approximated using a forward Euler approximation. For each $i\in [1,2,\dots ,\mathit{\ell }]$, we consider the discretization $t_{i-1}={\tilde{t}}_0<{\tilde{t}}_1<\dots <{\tilde{t}}_{{\tilde{N}}_t}=t_i$ of the time domain $[t_{i-1},t_i]$, corresponding to the $i^{\mathrm{th}}$ component of the subgradient, with ${\tilde{N}}_t$ uniform time steps. It follows that ${\tilde{t}}_n=n\times \Delta \tilde{t},n=0,1,\dots ,{\tilde{N}}_t$, and $\Delta \tilde{t}={\displaystyle \frac{t_i-t_{i-1}}{{\tilde{N}}_t}}$. The numerical approximation of the $i^{\mathrm{th}}$ component of the ℓ-dimensional subgradient is denoted by ${(\overline{\mathfrak{D}}\mathit{\Theta }\left({\lambda }^{\mathit{\ell }}\right))}_i$ and given by

$${\left(\overline{\mathfrak{D}}\mathit{\Theta }\left({\lambda }^{\mathit{\ell }}\right)\right)}_i={\displaystyle \frac{1}{M_{\mathrm{SG}}}}\sum _{m=1}^{M_{\mathrm{SG}}}\sum _{n=0}^{{\tilde{N}}_t}\left({\mathbb{1}}_{\{{\varphi }_{\mathrm{out}}({\tilde{t}}_n,{\overline{\mathbf{X}}}^{{\overline{N}}_t}({\tilde{t}}_n,{\omega }^{\left(m\right)}),\varphi ({\tilde{t}}_n,{\overline{\mathbf{X}}}^{{\overline{N}}_t}({\tilde{t}}_n,{\omega }^{\left(m\right)})))\ge {\varphi }_{\mathrm{th}}\}}-ϵ\right)\Delta \tilde{t},\forall i\in [1,\dots ,\mathit{\ell }],·$$

(47)

where ${\omega }^{\left(m\right)}$ denotes the $m^{\mathrm{th}}$ i.i.d. realization of the random variables required to generate sample paths of ${\overline{\mathbf{X}}}^{{\overline{N}}_t}$. The time discretization for subgradient computation and the Lagrange multiplier function ${\lambda }^{\mathit{\ell }}$ generally need not coincide. In this case, Brownian bridge interpolation is necessary to evaluate the optimally controlled paths ${\overline{\mathbf{X}}}^{{\overline{N}}_t}$ at the time discretization points ${\left\{{\tilde{t}}_n\right\}}_{n=0}^{{\tilde{N}}_t}$ for each $i\in [1,\dots ,\mathit{\ell }]$. The same $M_{\mathrm{SG}}$ realisations of the optimally controlled path ${\overline{\mathbf{X}}}^{{\overline{N}}_t}$ are used to estimate all ℓ components of $\overline{\mathfrak{D}}\mathit{\Theta }\left({\lambda }^{\mathit{\ell }}\right)$. Algorithm A2 in Appendix A presents the corresponding pseudo-algorithm.

Moreover, (47) is a Monte Carlo approximation of (39). It is subject to noisy statistical error controlled in a probabilistic sense by the choice of $M_{\mathrm{SG}}$. Thus, we only have access to noisy subgradients of the dual function. Hence, we apply the stochastic subgradient method (SSM) to solve the convex, nonsmooth optimization Problem 4.

Remark 5 (Subgradient estimation error). 

The error in the subgradient estimation consists of two parts: a statistical Monte Carlo error from approximating the expected value in (39) using a sample average with $M_{\mathrm{SG}}$ samples and a discretization error from approximating the integral in the expectation in (39) using a forward Euler summation with ${\tilde{N}}_t$ steps. From the central limit theorem [64], the Monte Carlo approximation error is $𝒪(M_{\mathrm{SG}}^{-{\displaystyle \frac{1}{2}}})$. The discretization error for the first-order forward Euler summation is $𝒪\left({\tilde{N}}_t^{-1}\right)$, yielding a total error in the subgradient estimation of $𝒪(M_{\mathrm{SG}}^{-{\displaystyle \frac{1}{2}}}+{\tilde{N}}_t^{-1})$. When solving the overall optimization problem up to a relative tolerance of $\mathrm{TOL}$ with respect to the subgradient, then we must estimate the subgradients up to an absolute tolerance of $ϵ\mathrm{TOL}$. This requirement implies that $M_{\mathrm{SG}}=𝒪\left({ϵ}^{-2}{\mathrm{TOL}}^{-2}\right)$ and ${\tilde{N}}_t=𝒪\left({ϵ}^{-1}{\mathrm{TOL}}^{-1}\right)$, yielding a total computational work of $𝒪\left({ϵ}^{-3}{\mathrm{TOL}}^{-3}\right)$ for each subgradient estimation.

4.3. Dual Problem Optimization

After approximating the dual function $\mathit{\Theta }\left({\lambda }^{\mathit{\ell }}\right)$ and its subgradient $\mathfrak{D}\mathit{\Theta }\left({\lambda }^{\mathit{\ell }}\right)$ for a given Lagrange multiplier function ${\lambda }^{\mathit{\ell }}\in {\overline{\mathit{\Lambda }}}^{\mathit{\ell }}$, we must solve the nonsmooth convex dual optimization problem in Problem 4. We solve this in two stages: (i) by constructing an approximation ${\lambda }^{\mathit{\ell }}$, of $\lambda$ (as in (35)), and (ii) by numerically optimizing the amplitudes ${\mathit{{\rm Y}}}^{\mathit{\ell }}$ of the approximated function ${\lambda }^{\mathit{\ell }}$. Algorithm 1 illustrates the overall procedure. Figure A1 depicts Algorithm 1 visually for further clarity.

Algorithm 1: Numerical dual optimization procedure

Input: $\mathrm{TOL}$, ${\mathrm{TOL}}_{\mathrm{init}}$, max-iter, ${\overline{N}}_{\mathrm{iter}}$, $N_{\mathrm{iter}}$, ${\mathit{{\rm Y}}}_1^1=1$,            ${\beta }_F,M_{\mathrm{SG}},N_t,N_1,N_2,N_3,{\overline{N}}_t,{\tilde{N}}_t,\mathbf{X}\left(0\right)$ Output: Optimal controls ${\mathit{\varphi }}^{*}$, optimal Lagrange multiplier function ${\lambda }^{\mathit{\ell }}\left(t\right)$ Construct ${\lambda }^1\left(t\right)$ with ${\mathit{{\rm Y}}}_1^1$ using (35); Obtain ${\widetilde{\mathit{{\rm Y}}}}_1^1$ using initialization Algorithm A3 with inputs ${\mathrm{TOL}}_{\mathrm{init}}$,${\mathit{{\rm Y}}}_1^1=1$,   ${\beta }_F,M_{\mathrm{SG}},N_t,N_1,N_2,N_3,{\overline{N}}_t,\mathbf{X}\left(0\right)$; Construct ${\lambda }^1\left(t\right)$ with ${\widetilde{\mathit{{\rm Y}}}}_1^1$ using (35); Obtain ${\widehat{\mathit{{\rm Y}}}}_1^1$ with the LMBM routine [65] with starting point ${\widetilde{\mathit{{\rm Y}}}}_1^1$, number of iterations   $N_{\mathrm{iter}}$ and parameters specified in Table A3. Parameters to run the LMBM routine [65] in Algorithm 1 to produce numerical results described in Section 5. See [65] for more details. Parameter Description Value RPAR(1) Tolerance for changes in the function value  $0.1$ RPAR(2) Second tolerance for changes in the function value $-1$ (ignored) RPAR(3) Minimum acceptable function value 0 RPAR(4) Tolerance for the first termination parameter  ${10}^{-2}$ RPAR(5) Tolerance for the second termination parameter  ${10}^{-2}$ RPAR(6) Distance measure parameter  $0.5$ RPAR(7) Line search parameter  $0.2$ RPAR(8) Maximum step size 10 IPAR(1) Exponent for distance measure 2 IPAR(2) Maximum iterations 50 IPAR(3) Maximum function evaluations 100 IPAR(4) Maximum iterations with changes of function values smaller than RPAR(1) 5 IPAR(5) Printout specification  $-1$ IPAR(6) Selection of method 0 (LMBM) IPAR(7) Selection of scaling strategy 0 ; Construct ${\lambda }^1\left(t\right)$ with ${\widehat{\mathit{{\rm Y}}}}_1^1$ using (35);

4.3.1. Lagrange Multiplier Refinement

This work constructs a finite-dimensional piecewise constant approximation, ${\lambda }^{\mathit{\ell }}$, of the Lagrange multiplier function, $\lambda$, as detailed in Section 3.2. Increasing the refinement level ℓ enhances the approximation quality of $\lambda$ by ${\lambda }^{\mathit{\ell }}$, but also increases the optimization dimension (also ℓ). This work determines a sufficient ℓ that controls the violation of the relaxed constraint. We start with $\mathit{\ell }=1$ and keep uniformly doubling it, until the violation of Constraint 2 is sufficiently controlled pointwise in time.

4.3.2. Numerical Optimization

This work solves the optimization of $\mathit{\Theta }\left({\lambda }^{\mathit{\ell }}\right)$ for ${\lambda }^{\mathit{\ell }}\in {\overline{\mathit{\Lambda }}}^{\mathit{\ell }}$ with respect to the amplitudes ${\mathit{{\rm Y}}}^{\mathit{\ell }}\in {\left({\mathbb{R}}^{+}\right)}^{\mathit{\ell }}$ as discussed in Section 3.2. For a given evaluation point ${\lambda }^{\mathit{\ell }}$, we have access to the dual function value (approximated in Section 4.1) and a noisy subgradient (approximated in (47)). Hence, the SSM [47] is employed to solve the nonsmooth optimization problem. The SSM is a subgradient-based search algorithm using a dual function evaluation and noisy evaluation of an arbitrary subgradient at each evaluation point. At each refinement level ℓ, the SSM runs until the subgradient norm is below a prescribed relative tolerance $\mathrm{TOL}$ or the number of iterations exceeds a threshold max-iter. The algorithm output at each ℓ is the optimal amplitude vector ${\mathit{{\rm Y}}}^{\mathit{\ell }}$. However, this optimal value is not necessarily reached at the final iteration of the SSM because the objective function to be maximized does not increase at every step of the subgradient method [47]. Hence, we track the highest value of the dual function and store its corresponding ${\mathit{{\rm Y}}}^{\mathit{\ell }}$ as the optimal value.

The choices for the starting point and step-size are crucial for quick convergence of the SSM, especially in higher dimensions. This work uses a nonsummable diminishing step-size of $𝒪\left({\displaystyle \frac{1}{k+1}}\right)$, where k denotes the iteration number. The convergence of the SSM with step-size is proven in [47]. The associated constant, denoted by $C_{\mathrm{SSM}}$ in Algorithm 1, is tuned once such that the step-size taken is $𝒪\left({\mathit{{\rm Y}}}^{\mathit{\ell }}\right)$. $C_{\mathrm{SSM}}$ need not be tuned again for a different set of parameters with the current problem structure. For enhanced performance, the step-size at each iteration is adjusted by dividing the current subgradient by its norm. Furthermore, the SSM runs for the first ${\overline{N}}_{\mathrm{iter}}$ iterations using a constant step-size before reverting to a diminishing step size. The Lagrange multiplier function ${\lambda }^{\mathit{\ell }-1}\left(t\right)$ constructed using the computed optimal amplitudes ${\mathit{{\rm Y}}}^{\mathit{\ell }-1}$ is used as the starting point of the SSM at level ℓ. However, we must still choose a good starting point at level $\mathit{\ell }=1$. This work devises an initialization algorithm and complements it with a nonsmooth deterministic optimizer.

Remark 6 (SSM convergence rate). 

A convergence rate of $𝒪(k^{-{\displaystyle \frac{1}{2}}})$ for the SSM has been proven for a class of convex functions with suitable step-size choice [66]. When solving the optimization problem up to a relative tolerance of $\mathrm{TOL}$ with respect to the dual function value $\mathit{\Theta }\left(\lambda \right)$, the SSM requires an expected number of $𝒪\left({\left|\mathit{\Theta }\left(\lambda \right)\right|}^{-2}{\mathrm{TOL}}^{-2}\right)$ iterations to converge under the assumptions in [66].

Remark 7 (Primal feasible solution). 

Note that Algorithm 1 uses the subgradient as a stopping criterion, instead of a duality gap. This is because the output of Algorithm 1 is already a primal feasible solution up to given relative tolerance $\mathrm{TOL}$. That is, the optimal controls produce a solution that minimizes the primal cost while only violating Constraint 2 lesser than a small value of $ϵ\mathrm{TOL}$.

4.3.3. LMBM-Boosted Initialization

Level $\mathit{\ell }=1$ implies that the Lagrange multiplier function ${\lambda }^1\left(t\right)$ is constant in time, with the constant denoted by ${\mathit{{\rm Y}}}_1^1$ (see (35)). First, this work devises an initialization algorithm starting with the arbitrary point ${\mathit{{\rm Y}}}_1^1=1$ and estimating the dual function value and a subgradient at that point. Then, if the subgradient at ${\mathit{{\rm Y}}}_1^1=1$ is positive, ${\mathit{{\rm Y}}}_1^1$ is continually increased by a factor ${\beta }_F$ until the subgradient becomes negative or its norm reaches a prescribed relative tolerance ${\mathrm{TOL}}_{\mathrm{init}}$. Conversely, if the subgradient at ${\mathit{{\rm Y}}}_1^1=1$ is negative, ${\mathit{{\rm Y}}}_1^1$ is continually decreased by a factor ${\beta }_F$, until the subgradient becomes positive or its norm reaches relative tolerance ${\mathrm{TOL}}_{\mathrm{init}}$. The final obtained point is stored as ${\widetilde{\mathit{{\rm Y}}}}_1^1$. Algorithm A3 in Appendix A presents the pseudo-algorithm of the devised initialization procedure. Figure A2 depicts Algorithm A3 visually for further clarity.

To obtain a better starting point, ${\widetilde{\mathit{{\rm Y}}}}_1^1$ is used as a starting point for a deterministic nonsmooth optimization routine called the LMBM [45,56]. Bundle methods are more robust than a simple subgradient-based method because they approximate the entire subdifferential of the objective function, enhancing its convergence speed [44]. However, the LMBM requires access to deterministic estimates of the objective function value and its subgradient at each evaluation point. Although this work only has access to noisy Monte Carlo estimates of subgradients (47), we can still employ the LMBM routine to reach a good starting point for the SSM, especially in the regime where changes in the objective function value considerably outweigh the noise in subgradient estimates. This work runs the LMBM for a fixed number of iterations $N_{\mathrm{iter}}$. Table A3 in Appendix B.4 specifies the parameters required to run the LMBM routine [65]. The output of this routine ${\widehat{\mathit{{\rm Y}}}}_1^1$ is applied as a starting point for the SSM algorithm described in Section 4.3.2.

Remark 8 

(Scale of $\lambda$). With respect to the parameter $ϵ$ that determines the confidence level of satisfying Constraint 2, the optimal primal cost $\mathcal{U}\left(\mathit{\varphi }\right)$ associated with Problem 4 is an $𝒪\left(1\right)$ term, while the optimal subgradient would be $𝒪\left(ϵ\right)$ (ideally 0). Equation (38) then implies that the optimal Lagrange multiplier $\lambda$ is $𝒪\left({\displaystyle \frac{1}{ϵ}}\right)$. This yields a good guess for an initial point ${\mathit{{\rm Y}}}_1^1$ for the LMBM-boosted initialization procedure.

5. Numerical Experiments and Results

This section presents a model example of a cellular base station powered by the German power grid. This section describes the system and all dynamics driving the operation of the base station, and provides the results of applying the proposed numerical approach to solve the optimal power-procurement problem for the system.

5.1. Description of Model Cellular Base Station System

Figure 1 schematically illustrates the considered base station model.

Table A1. Parameters and coefficients for modeling the cellular base station.

Parameter	Unit	Description	Value
$C_{\mathrm{scal}}$		Base station power-loss scaling factor	$7.84$
$C_{\mathrm{offset}}$	Watt (W)	Base station offset power	$71.5$
${\overline{P}}_{\mathrm{tx}}$	Watt (W)	Maximum base station transmission limit	$5\times {10}^3$
${\mathbf{x}}_{\mathrm{BS}}$		Base station location	$[0,0]$
$\kappa$		Path loss constant	1
$\eta$		Path loss exponent	2
${\mathrm{SNR}}_{\mathrm{th}}$	Decibel (dB)	Signal-to-noise ratio threshold	15
${\sigma }_0$	Watt (W)	Ambient transmission noise	$3.1623\times {10}^{-8}$
${\overline{P}}_R$	Watt (W)	Maximum renewable power-production capacity	${10}^4$
$\overline{A}$	Watt-hour (Wh)	Maximum battery charge capacity	${10}^4$
${\underline{P}}_A$	Watt (W)	Maximum battery charge capacity	$7.5\times {10}^3$
${\overline{P}}_A$	Watt (W)	Maximum battery discharge capacity	$3\times {10}^4$
$C_1$	EUR/Wh	Pollutant emission Coefficient 1	$4\times {10}^{-4}$
$C_2$	EUR/${\mathrm{W}}^2$h	Pollutant emission Coefficient 2	${10}^{-4}$
$P_K$	EUR/Wh	Fictitious cost per unit battery charge	$0.0064$
w		Pareto parameter	$0.5$

provides the descriptions and numerical values of all coefficients used to describe the model.

The daily mobile user traffic profile primarily drives the power demand of a cellular base station. The description of the daily traffic profile consists of two facets: (i) the number of people connected to the network $N_u\left(t\right)$, and (ii) the physical distribution of users around the base station ${\rho }_z\left(t\right)$. The numerical experiment applies the following sinusoidal profile to model $N_u\left(t\right)$:

$$N_u\left(t\right)=max\left[{\underline{N}}_u,{\overline{N}}_u{\displaystyle \frac{1}{2^{\varrho }}}{\left(1+sin\left({\displaystyle \frac{\pi t}{6}}+\pi \right)\right)}^{\varrho }\right]·$$

(48)

This model and the stochastic versions of it have been empirically demonstrated to approximate practical user patterns closely for calibrated values of the smoothness parameter $\varrho$ [67,68]. The parameter $\varrho$ determines the rate of increase or decrease in $N_u\left(t\right)$ during the day. A higher value of $\varrho$ indicates a steeper increase or decrease rate, and $\varrho =3$ in the numerical experiment. This model also considers two peak hours (about 09.00 to 10.00 and 20.00 to 21.00) and few connected users at night (00.00 to 05.00). Moreover, ${\underline{N}}_u$ and ${\overline{N}}_u$ denote the minimum and maximum number of users, respectively, that are connected to the base station at any time. This work sets ${\underline{N}}_u=100$ and ${\overline{N}}_u=2000$. These values were selected using the data on the average population densities of German cities [69] and the geographical area served by an average cellular base station in Germany [70]. Figure 2 depicts the model for $N_u\left(t\right)$.

The user distribution ${\rho }_z\left(t\right)$ crucially determines the estimation of the outage proportion ${\varphi }_{\mathrm{out}}\left(t\right)$ at given time t (Section 2.1). Some simple 2D distributions have been employed to model the mobile user distribution in urban zones efficiently. For example, the uniform distribution can be employed to model the mobile user distribution in suburban areas very well, and the Gaussian distribution is a good model for mobile user distributions in industrial zones [71]. Figure 3 visualizes this model. Analytical expressions can be derived for the corresponding ${\varphi }_{\mathrm{out}}\left(t\right)$ for these distributions. Appendix B.2 provides the closed-form expressions. For the numerical experiments, we apply the symmetric 2D Gaussian distribution centered on the site of the base station ${\mathbf{x}}_{\mathrm{BS}}$ with a diagonal covariance matrix with variance ${\sigma }_u^2={300}^2{\mathrm{m}}^2$ along both directions.

We apply the SDE in (3) to model the instantaneous Nakagami fading channel, whose invariant distribution is a shifted-gamma distribution with the shape parameter $\mu$ and scale parameter $\theta$. We set $\mu =3$ and $\theta =1$ so that a quick, smooth convergence occurs to the invariant distribution and $\mathbb{E}\left[\xi {\left(t\right)}^{-2}\right]$ is bounded for all $t\in [0,T]$. We set the shift $\underline{\xi }$, corresponding to the value of $\xi$ that can still achieve an outage proportion of ${\displaystyle \frac{{\varphi }_{\mathrm{th}}}{2}}$ in the worst-case scenario when $N_u\left(t\right)={\overline{N}}_u$. Moreover, we set the initial distribution ${\mu }_0^{\xi }$ the same as the invariant distribution of process $\xi$. Figure 4 visualizes the samples paths and the invariant distribution of the SDE in (3).

We set $K_{\mathrm{net}}\left(t\right)=0.01$ EUR/h per person for $t\in [0,T]$. The price was taken from the Vodafone 5G prices in Germany [72]. The constant price assumption is generally valid because regulators set these prices and do not move within a day.

We apply the SDE in (9) to model the instantaneous normalized generated renewable power. This work uses the 2023 German wind power forecast and production data with a 15-min frequency from the operator 50Hertz [73] to calibrate the SDE in (9). The forecast data are employed to construct $p\left(t\right)$ and $\dot{p}\left(t\right)$, and the discrepancy between the forecast and true production data is applied to calibrate the parameters $\alpha$ and ${\theta }_0$ in (9). Apart from these parameters, we also calibrate an additional parameter $\varsigma$. The discrepancy between the forecast and production at time $t=0$ is not typically zero in real-world applications. $\varsigma$ is defined as the time before $t=0$, for which the forecast error can be set to zero. This approach ensures that uncertainty also exists in renewable power production at $t=0$. We follow the calibration procedure detailed in [30] (see [30] for details). The mean-reversion parameter $\theta \left(t\right)$ in (9) is defined as follows:

$$\theta \left(t\right)=max\left({\theta }_0,{\displaystyle \frac{\alpha {\theta }_0+\left|\dot{p}\left(t\right)\right|}{min\left(p\right(t),1-p(t\left)\right)}}\right)·$$

The calibrated values for the above data are $\alpha =0.34$, ${\theta }_0=2.3948$, and $\varsigma =0.054$. Given the deterministic forecasts of wind power from the same operator in 2024, we can generate sample paths of the SDE in (9) with parameters $\alpha ,{\theta }_0,\varsigma$ calibrated from 2023 data. Figure 5 illustrates the results for two sample days in 2024.

This work applies (10) to model the instantaneous normalized battery charge. As given in (10), the maximum charging capacity ${\underline{P}}_A$ and the maximum discharge capacity ${\overline{P}}_A$ are defined as functions of the stored battery charge. This function is typically called the characteristic curve of the battery. The numerical experiment employs a simple characteristic curve illustrated in Figure 6. The characteristic curve imposes (11), implying that the battery cannot discharge power when it has zero charge and can no longer charge itself when fully charged.

Forecasts of buying and selling prices, $K_b\left(t\right)$ and $K_s\left(t\right)$, are constructed from publicly available day-ahead spot prices of grid power. We set $K_b\left(t\right)=K_s\left(t\right)$, implying that the buying and selling prices of power are the same. This assumption is valid because the market commonly ensures that this holds in real life. The numerical experiment employs the German day-ahead spot-price data from 2024 available in [74]. However, these data are generally available only at a 1-h frequency. Hence, we smoothed and extended the data to domain $[0,T]$ by fitting a polynomial curve to them and using the polynomial function as the input $K_b\left(t\right)$. Figure 7 illustrates an example of the spot-price data.

5.2. Numerical Results

We present the results of solving the stochastic optimal control problem for the model cellular base station (Section 5.1) using the numerical approach in Section 4.

5.2.1. Cost-to-Go Function of the Dual Problem

The numerical solution of the HJB PDE in (33) for a given Lagrange multiplier function ${\lambda }^{\mathit{\ell }}\left(t\right)$ (35) is obtained using the scheme in Section 4.1. We apply a uniform rectangular spatial grid with $N_1=N_2=N_3=N_x=10$, where $N_t$ is set as the lowest value that satisfies the CFL condition (45). In this case, we obtain $N_t=800$. The presence of second-order derivatives in the HJB PDE in (33) yields $N_t=𝒪\left(N_x^2\right)$ for a numerically stable solution, explaining the high value obtained for $N_t$. These discretization parameter choices ensure that the relative numerical PDE discretization error is below $1\%$. We set $\overline{\chi }$ as the upper $95\%$ quantile of the invariant distribution of process $\xi \left(t\right)$.

The optimization problem in (43) must be solved at each point in the grid $\tau$. For the simple cases of uniform or Gaussian cellular user distribution, we obtain closed-form expressions for ${\varphi }_{\mathrm{out}}$ (Appendix B.2). The resulting ${\varphi }_{\mathrm{out}}$ ensures that (43) is a convex optimization problem and can be solved analytically. Appendix B.3 presents additional details. Figure 8 presents three slices of the numerical cost-to-go function $\overline{u}$ with the optimal Lagrange multiplier function ${\lambda }^{\mathit{\ell }}\left(t\right)$, each with two fixed state variables. The numerical dual function approximation $\overline{\mathit{\Theta }}\left({\lambda }^{\mathit{\ell }}\right)$ is obtained by evaluating $\overline{u}(0,\mathbf{X}\left(0\right))$.

5.2.2. Obtaining Reference Costs for Comparison

This work compares the values for the dual cost above with reference values obtained from two related, easy-to-solve stochastic optimal control problems. This comparison was conducted to ensure that the output of the numerical algorithm makes sense with respect to these reference values. First, we solve the same optimization problem as in Problem 1, but with the probabilistic constraint replaced with a deterministic a.s. constraint, solving the following problem.

Problem 5 (Primal problem with the a.s. QoS constraint). 

Given the initial data $\mathbf{X}\left(0\right)=[A_0,R_0\sim {\mu }_0^R,{\xi }_0\sim {\mu }_0^{\xi }]$, and deterministic forecasts for the daily cellular user traffic profile $N_u\left(t\right)$, energy spot prices $K_b\left(t\right),K_s\left(t\right)$, renewable energy forecast $p\left(t\right)$, and mobile network usage price $K_{\mathit{net}}\left(t\right)$ for $0\le t\le T$, we solve the following:

$${\mathit{\varphi }}^{*}=\underset{\{\mathit{\varphi }\in {\overline{\mathcal{A}}}_1\}}{argmin}\mathcal{U}\left(\mathit{\varphi }\right)=w\left({\mathcal{C}}_1-{\mathcal{R}}_1-{\mathcal{R}}_2-{\mathcal{R}}_3\right)+(1-w){\mathcal{C}}_2,$$

(49)

where the minimization is subject to dynamics (3), (9), and (10), and is done over all controls that satisfy the constraint in 1 and the following a.s. constraint (instead of the constraint in 2).,

$${\varphi }_{\mathrm{out}}(t,\xi \left(t\right),P_{\mathrm{tx}}\left(t\right))={\varphi }_{\mathrm{th}},\forall t\in [0,T],$$

(50)

where (50) provides a solution for the control $P_{\mathrm{tx}}\left(t\right)$ for all $t\in [0,T]$.

Hence, the control vector for this problem has only three components, $\mathit{\varphi }\left(t\right)=[P_A\left(t\right),P_F\left(t\right),P_S\left(t\right)]$. Problem 5 is a standard continuous-time stochastic optimal control problem; hence the dynamic programming principle can be used to derive and numerically solve the associated HJB PDE, analogous to Section 3.1. The cost-to-go function for this problem is expected to be higher than the dual cost of Problem 4 because the network operator would need to buy more power to keep ${\varphi }_{\mathrm{out}}$ constant at all times, whereas the operator is allowed to relax the constraint probabilistically in Problem 4 as described in Constraint 2. Next, we solve the same optimization problem as Problem 1, but without Constraint 2.

Problem 6 (Primal problem with no QoS constraint). 

Given the initial data $\mathbf{X}\left(0\right)=[A_0,R_0\sim {\mu }_0^R,{\xi }_0\sim {\mu }_0^{\xi }]$, and deterministic forecasts for the daily cellular user traffic profile $N_u\left(t\right)$, energy spot prices $K_b\left(t\right),K_s\left(t\right)$, renewable energy forecast $p\left(t\right)$, and mobile network usage price $K_{\mathit{net}}\left(t\right)$ for $0\le t\le T$, we solve the following:

$${\mathit{\varphi }}^{*}=\underset{\{\mathit{\varphi }\in {\overline{\mathcal{A}}}^{\mathit{rel}}\}}{argmin}\mathcal{U}\left(\mathit{\varphi }\right)=w\left({\mathcal{C}}_1-{\mathcal{R}}_1-{\mathcal{R}}_2-{\mathcal{R}}_3\right)+(1-w){\mathcal{C}}_2,$$

(51)

where the minimization is taken over all controls satisfying the constraint in 1 with dynamics (3), (9), and (10).

Problem 6 is also a standard continuous-time stochastic optimal control problem; hence the dynamic programming principle can be applied to derive and numerically solve the associated HJB PDE. Without the Lagrangian relaxation, the cost-to-go function associated with this problem is the same as in (32). Thus, this problem can be solved using the numerical scheme in Section 4.1, setting $\lambda \left(t\right)=0$. The cost-to-go function for this problem is expected to be less than the dual cost of Problem 4 because the network operator only maximizes revenue without ensuring that QoS is achieved. The HJB PDEs corresponding to Problems 5 and 6 must be numerically solved only once because no associated dual problem needs to be solved.

5.2.3. Dual Optimization Results

This work runs Algorithm 1 with parameters set to values in Appendix B.4. Notably, we set ${\varphi }_{\mathrm{th}}={10}^{-3}$ and $ϵ=0.1$, implying that the network operator must ensure that the proportion of users in outage ${\varphi }_{\mathrm{out}}$ should be less than ${10}^{-3}$ with a probability of $90\%$ for all $t\in [0,T]$. Figure 9 presents the results of the initialization Algorithm A3 and the LMBM routine. Figure 9a reveals the dual cost versus ${\mathit{{\rm Y}}}_1^1$ at each iteration of Algorithm A3. The dual cost increases with the number of iterations, and it is within the two reference values attained by solving Problems 5 and 6. Figure 9b plots the corresponding estimated subgradient values. The initialization algorithm reaches the prescribed tolerance in three iterations. Figure 9 reveals that the LMBM offers a more optimal point than the output of Algorithm A3, providing an excellent starting point for the dual optimization in Algorithm 1.

Figure 10 illustrates the evolution of the dual function values and its corresponding subgradient at each iteration of the SSM for a fixed ℓ. Figure 10a verifies that not every step of the SSM is taken towards maximization. This behavior illustrates the difficulty of solving the nonsmooth optimization in Problem 4. Due to the satisfactory choice of starting values and step sizes (Section 4.3), Algorithm 1 converges for each ℓ within max-iter iterations with respect to the subgradient.

Figure 11 plots the evolving optimal dual function $\overline{\mathit{\Theta }}\left({\lambda }^{\mathit{\ell }}\right)$ and its corresponding subgradient $\overline{\mathfrak{D}}\mathit{\Theta }\left({\lambda }^{\mathit{\ell }}\right)$ constructed using the optimal amplitudes ${\mathit{{\rm Y}}}^{\mathit{\ell }}$ at each level ℓ of Algorithm 1.

Table 1. Comparing the optimal dual costs achieved in various stages of Algorithm 1 with those of Problems 5 and 6.

Description	ℓ	Optimal Cost (EUR)
Problem 5		$-173.57$
Problem 6		$-667.83$
Initialization Algorithm A3	1	$-661.52$
LMBM	1	$-663.62$
Dual optimization Algorithm 1	2	$-661.8$
Dual optimization Algorithm 1	4	$-661.59$
Dual optimization Algorithm 1	8	$-660.51$

lists the optimal dual cost at each stage of Algorithm 1. Table 1 presents minor increases in the dual function evaluations as the refinement level ℓ increases, implying that we successively obtain more optimal solutions as the Lagrange multiplier function ${\lambda }^{\mathit{\ell }}\left(t\right)$ becomes finer. We normalize the subgradient vector norm by the number of dimensions for a fair comparison between the norm of vectors of varying dimensions.

Table 1 also highlights the advantage of applying a probabilistic constraint to achieve the QoS in cellular networks, instead of doing it a.s. for all $t\in [0,T]$ (as in Problem 5). The optimal dual cost achieved by Algorithm 1 is much closer to the expected optimal cost of the unconstrained Problem 6 and around four times less expensive than satisfying (50), a.s. (Problem 5). In rare occasions when an unreasonable surge in demand ($N_u\left(t\right)$) occurs, or a very low wireless fading channel ($\xi \left(t\right)$) exists due to environmental conditions or low availability of renewable power ($P_R\left(t\right)$), the numerical approach allows the network operator to relax the stringent QoS constraint to save substantial expenditure while still serving as many customers as possible. This approach contrasts the solution to Problem 5, which enforces (50) even in such periods, significantly increasing operating expenditure.

Figure 12 presents the evolution of the optimal Lagrange multiplier function as the refinement level ℓ increases in Algorithm 1. For the given example, Algorithm 1 reaches prescribed $\mathrm{TOL}$ at $\mathit{\ell }=8$, and does not refine the Lagrange multiplier function further. High-demand periods (Figure 2) correspond to high Lagrange multiplier function values. This behaviour is expected because a high ${\lambda }^{\mathit{\ell }}$ forces the mobile operator to satisfy Constraint 2 in times of high demand, although it significantly increases costs. The oscillation frequency of the base station power demand dictates the oscillations in the Lagrange multiplier function.

5.2.4. Optimal Power-Procurement Policy

With the optimal Lagrange multiplier function ${\lambda }^{\mathit{\ell }}$ from Algorithm 1, the dual cost is EUR $-660.51$ (Table 1). Figure 13 illustrates the optimal power-procurement policy for the model cellular base station for the next 24 h using the optimal solution, including the (a) net power consumption of the base station $C_{\mathrm{scal}}{N}_u\left(t\right)P_{\mathrm{tx}}\left(t\right)+C_{\mathrm{offset}}$ (refer (1)), (b) power drawn from the battery to run the base station $P_A\left(t\right)$, (c) power bought from the grid $P_F\left(t\right)$, (d) power sold back to the grid $P_S\left(t\right)$, and (e) optimal charge stored in battery $A\left(t\right)$. Figure 13a indicates that the base station consumes high power during periods of high demand (refer Figure 2). Figure 13b–d reveal that the battery provides most of the power to run the base station throughout the day. The clean, renewable energy stored in the battery powers a large proportion of base station operations so that the mobile operator does not need to buy expensive grid power. This approach reduces costs and the carbon footprint of the base station. The operator must only buy some power from the grid to satisfy high demand during peak hours (Figure 13c). The base station sells a negligible amount of power back to the grid (Figure 13d). The optimal policy also advises the operator to store incoming renewable power and charge the battery during low-demand hours before using that power to serve customers during high-demand hours (Figure 13e). Figure 13e also indicates that the battery has some stored energy at the end of the day that could be used the next day. This observation establishes the effect of implementing the running horizon framework (see Section 2.6), although we solve the problem for a short-term horizon.

We estimate the time-pointwise violation of Constraint 2 using the Monte Carlo method and compare it with the final subgradient vector $\overline{\mathfrak{D}}\mathit{\Theta }\left({\lambda }^{\mathit{\ell }}\right)$ from Algorithm 1 to validate the optimality of the solution further. We run approximate sample numerical paths of the optimally controlled dynamics, according to (46), with the optimal policy above. Figure 14b plots this information for a discrete set of points in $t\in [0,T]$. We also compare this with the optimal policy attained from solving the unconstrained problem in Problem 6 in Figure 14a. Figure 14a reveals that the user outage proportion ${\varphi }_{\mathrm{out}}$ for the optimal policy with no constraints on the QoS is almost consistently greater than the threshold ${\varphi }_{\mathrm{th}}$. In contrast, the proposed optimal solution controls the probabilistic constraint in an integral (averaged) sense (refer (39)), even though there are some oscillations around zero in the time-pointwise violation of Constraint 2. Figure 14b illustrates this behavior, with the time-averaged violation in the probabilistic Constraint 2 close to zero up to the prescribed relative tolerance $\mathrm{TOL}$.

5.2.5. Sensitivity Analysis

This work investigates the performance of the proposed algorithm under certain scenarios that could be of interest to network operators, which are listed below.

• Scenario A: No incoming renewable power for a day ($P_R\left(t\right)=0$ for all $t\in [0,T]$).
• Scenario B: The wireless fading channel is substantially low due to extreme weather ($\underline{\xi }$ is halved).
• Scenario C: High weight is assigned to minimizing operating expenditure ($w=0.99$).
• Scenario D: The price users pay to connect to the network substantially reduces ($K_{\mathrm{net}}=0.001$ EUR/h per person).
• Scenario E: The probabilistic QoS constraint 2 must be satisfied with low confidence ($ϵ=0.2$).

All parameters and dynamics are set as in Table A1 and Section 5.1, except for the prescribed parameter change to simulate each scenario. For each scenario, we solve three problems: (i) dual optimization (Problem 4), (ii) Problem 5 with a.s. QoS constraint, and (iii) Problem 6 with no QoS constraint.

Table 2. Comparing the optimal dual costs (in EUR) achieved with the dual optimization Algorithm 1 for each scenario with the optimal primal costs of the corresponding unconstrained and a.s. QoS constrained problems.

	Dual Algorithm 1	Problem 6	Problem 5
Scenario A	$-441.8$	$-564.9$	$1941.9$
Scenario B	$-649.28$	$-660.27$	$4160.4$
Scenario C	$-1334.7$	$-1338.8$	$1980.5$
Scenario D	$-80.52$	$-92.69$	$1209.5$
Scenario E	$-664.03$	$-667.83$	$-173.57$

reports the optimal costs in each case for all five scenarios.

In all scenarios, the optimal dual cost lies between the primal costs of the corresponding unconstrained and a.s. QoS constrained problems. Moreover, the optimal dual cost is closer to the primal cost of the unconstrained problem in all cases because it is costly to satisfy the QoS constraint a.s. at all times. The optimal dual cost in Scenario A is 33% higher than the optimal dual cost in Table 1 because there is no “free” incoming renewable energy for the operator to run the base station, increasing the expenditure. The optimal dual cost of Scenario C is 102% higher than the optimal dual cost in Table 1, directly implying the high weight assigned to financial profits, with no regard for its carbon footprint. The optimal cost of Scenario E confirms the intuition that it is less expensive to satisfy the QoS with low confidence (80%), compared to 90% confidence. To quantify differences in the optimal procurement policy for each scenario, we produced plots analogous to Figure 13, and approximated the area below the mean curve for each plot to obtain the expected energy balance.

Table 3. Comparison of the expected energy balance (in Wh) balance using the optimal power-procurement policy from the optimal $\overline{u}$ obtained from the dual optimization Algorithm 1 for each scenario.

Expected Energy	Consumed	Battery	Bought	Sold
Base scenario	$4199.6$	$4079.9$	$119.63$	$49.64$
Scenario A	$4040$	$3295.7$	$744.36$	$29.94$
Scenario B	$4584.3$	$4440.6$	$143.68$	$43.5$
Scenario C	$4402.8$	$2347.5$	$2055.2$	$5363.6$
Scenario D	$3883.3$	$3811.6$	$71.64$	$50.99$
Scenario E	$3870.8$	$3763.3$	$107.48$	$52.35$

lists these values. The base scenario in Table 3 refers to the original problem described in Section 5.1.

In Scenario A, the operator buys six times more energy than the base scenario. The lack of incoming renewable power forces the network operator to buy more power from the grid to satisfy the QoS constraint. In Scenario C, the operator buys almost half the energy consumed by the base station from the grid and sells a considerable amount of energy stored in the battery. The optimal policy is to make as much financial profit as possible because a high weight is assigned to minimizing financial costs. The base station consumes 9% more energy in Scenario B than the base scenario because the energy demand of satisfying QoS during periods of low fading channel (extreme weather) is higher. An 8% decrease occurs in the energy consumed by the base station in Scenario E compared with the base scenario. Less energy is required to satisfy the QoS constraint with lower confidence.

This work also performs scenario simulations over probability distributions of both model and algorithmic parameters in Table A1 and

Table A2. Simulation parameters to run Algorithm 1 and produce numerical results as described in Section 5.

Parameter	Description	Value
${\varphi }_{\mathrm{th}}$	Mobile user outage proportion threshold	${10}^{-3}$
$ϵ$	Confidence level of violating the constraint in 2	$0.1$
$A_0$	Normalized battery charge level at $t=0$	$0.5$
${\mu }_0^{\xi }$	Initial distribution of the wireless fading channel	$\gamma (3,1)$
${\mu }_0^r$	Initial distribution of the normalized wind power	$\mathcal{N}\left(p(-\varsigma )+\dot{p}(-\varsigma )\varsigma ,2\alpha {\theta }_{0}p(-\varsigma )(1-p(-\varsigma ))\varsigma \right)$
$N_1$	Discretization of (33) in the a domain	10
$N_2$	Discretization of (33) in the r domain	10
$N_3$	Discretization of (33) in the $\chi$ domain	10
$N_t$	Discretization of (33) in the t domain	800
$\mathrm{TOL}$	Prescribed relative tolerance for Algorithm 1	$0.1$
${\mathrm{TOL}}_{\mathrm{init}}$	Prescribed relative tolerance for Algorithm A3	1
max-iter	Prescribed maximum iterations in Algorithm 1	50
${\overline{N}}_{\mathrm{iter}}$	Initial number of SSM iterations with a constant step-size	10
$N_{\mathrm{iter}}$	Prescribed number of LMBM [65] iterations	50
${\beta }_F$	Factor of increase/decrease in Algorithm A3	5
$M_{\mathrm{SG}}$	Number of sample paths in Algorithm A2	${10}^3$
${\overline{N}}_t$	Time discretization parameter in Algorithm A2	64
${\tilde{N}}_t$	Time discretization parameter in Algorithm A2	${\displaystyle \frac{{\overline{N}}_t}{\mathit{\ell }}}$

, respectively.

Table A4. Distribution of model and algorithmic parameters used to run randomized multi-scenario simulations of Algorithm 1.

Parameter	Distribution
${\overline{P}}_{\mathrm{tx}}$	${log}_{10}\left({\overline{P}}_{\mathrm{tx}}\right)\sim \mathfrak{U}[3,4]$
${\sigma }_0$	${\sigma }_0\sim \mathcal{N}\left(3.1623\times {10}^{-8},{10}^{-16}\right)$
${\overline{P}}_R$	${log}_{10}\left({\overline{P}}_R\right)\sim \mathfrak{U}[2,4]$
$\overline{A}$	${log}_{10}\left(\overline{A}\right)\sim \mathfrak{U}[3,4]$
w	$w\sim \mathfrak{U}[0,1]$
${\varphi }_{\mathrm{th}}$	${log}_{10}\left({\varphi }_{\mathrm{th}}\right)\sim \mathfrak{U}[-2,-4]$
$A_0$	$A_0\sim \mathfrak{U}[0,1]$
$K_{\mathrm{net}}$	${log}_{10}\left(K_{\mathrm{net}}\right)\sim \mathfrak{U}[-1,1]$

lists the various probability distributions used to randomize the simulations. Algorithm 1 is run independently for 50 i.i.d. samples of the various parameters with no additional tuning between runs. Figure illustrates the optimal dual cost and subgradient achieved in each case. Figure 15b reveals that Algorithm 1 achieves the prescribed subgradient tolerance in all 50 cases, demonstrating the robustness of the proposed numerical procedure. These results emphasize the practicality of applying the proposed numerical approach as a decision-making tool for stakeholders in the wireless communications industry.

6. Conclusions

This work formulates a short-term, continuous-time stochastic optimal control problem that minimizes the operating expenditure and carbon footprint of a cellular wireless network powered by a hybrid energy system and equipped with an energy-storage capacity under uncertainty in renewable energy and wireless fading channels. We constructed an SDE with the shifted-gamma invariant distribution to model the stochastic Nakagami wireless fading channel. We employed an SDE model [30] to model the uncertain renewable power, with parameters calibrated using publicly available German energy market data from 2023. The QoS of the wireless network was posed as a probabilistic constraint to address uncertainties efficiently in the system. We developed a novel Lagrangian relaxation procedure in continuous time to relax the above probabilistic constraint. We developed a robust numerical procedure to solve the corresponding dual problem based on a finite-dimensional approximation of the Lagrange multiplier function, using an SSM algorithm driven by the numerical solution of the associated HJB PDE. An LMBM-boosted initialization procedure and an adaptive refinement of the Lagrange multiplier function enhanced algorithm performance. A running horizon framework resolved the spurious results at the end of the day due to the short-term horizon problem formulation. The practical application of the numerical procedure was demonstrated on a model cellular base station driven by the German power system and daily cellular traffic demand. The results illustrate the viability and efficiency of the proposed approach for real-world problems.

A potential future research direction is incorporating stochasticity in other dynamics, such as energy prices $K_b\left(t\right),K_s\left(t\right)$, and the cellular user traffic profile $N_u\left(t\right)$ [68]. This research would build a more realistic algorithm that is robust to random fluctuations in these dynamics but would increase the dimension of the associated HJB PDE. Another limitation of the current work is that small values of $ϵ$ ($\ll {10}^{-3}$) cannot be achieved, which modern-day cellular networks aim to achieve, because the naive Monte Carlo method is employed to estimate the subgradient. For a small $ϵ$, the Fokker–Planck PDE for the joint distribution of the optimally controlled dynamics could be solved to get a deterministic estimate of the subgradient. Alternatively, importance sampling could be applied to reduce the relative statistical error of the Monte Carlo subgradient estimates. A further improvement is devising a better strategy for refining the Lagrange multiplier function $\lambda \left(t\right)$. The computed time-pointwise violation in constraint values could be employed to construct a clever strategy to refine $\lambda$ instead of equally spaced refinements.