Data-Driven Dynamic Optimization for Hosting Capacity Forecasting in Low-Voltage Grids

Abstract

The sustainable integration of Distributed Energy Resources (DER) with the next-generation distribution networks requires robust, adaptive, and accurate hosting capacity (HC) forecasting. Dynamic Operating Envelopes (DOE) provide real-time constraints for power import/export to the grid, ensuring dynamic DER integration and efficient network operation. However, conventional HC analysis and forecasting approaches struggle to capture temporal dependencies, the impact of DOE constraints on network operation, and uncertainty in DER output. This study introduces a dynamic optimization framework that leverages the benefits of the sensitivity gate of the Sensitivity-Enhanced Recurrent Neural Network (SERNN) forecasting model, Particle Swarm Optimization (PSO), and Bayesian Optimization (BO) for HC forecasting. The PSO determines the optimal weights and biases, and BO fine-tunes hyperparameters of the SERNN forecasting model to minimize the prediction error. This approach dynamically adjusts the import/export of the DER output to the grid by integrating the DOE constraints into the SG-PSO-BO architecture. Performance evaluation on the IEEE-123 test network and a real Australian distribution network demonstrates superior HC forecasting accuracy, with an $R^2$ score of 0.97 and 0.98, Mean Absolute Error (MAE) of 0.21 and 0.16, and Root Mean Square Error (RMSE) of 0.38 and 0.31, respectively. The study shows that the model effectively captures the non-linear and time-sensitive interactions between network parameters, DER variables, and weather information. This study offers valuable insights into advancing dynamic HC forecasting under real-time DOE constraints in sustainable DER integration, contributing to the global transition towards net-zero emissions.

1. Introduction

The increasing penetration of Distributed Energy Resources (DER) in modern power systems triggers operational issues like voltage instability, thermal violation, and equipment overloading [1]. These challenges necessitate innovative approaches for accurate and adaptive Hosting Capacity (HC) forecasting under dynamic network conditions. While large Renewable Energy Resources (RESs) are connected with Medium Voltage (MV) or High Voltage (HV) levels, such as a high proliferation of DER—such as in Rooftop Photovoltaics (PVs), residential Battery Energy Storage Systems (BESSs), and Electric Vehicles (EVs)—they create operational challenges in Low Voltage (LV) distribution networks. This study focuses on various network and DER integration challenges, including voltage rise, line congestion, and reverse power flow due to limited capacity and infrastructure constraints of the network. The intermittent characteristics of DER output [2], limited data availability, and dynamic network behavior pose further challenges for forecasting network HC. Different studies address these challenges by deploying different forecasting and optimization frameworks.

Various strategies were investigated for hyperparameter tuning and optimization in forecasting models such as Particle Swarm Optimization (PSO), Bayesian Optimization (BO), Genetic Algorithm (GA), and other hybrid approaches. The probabilistic optimization framework was studied using the PSO algorithm in [3] for PV HC maximization. The PSO demonstrated potential improvement in convergence and accuracy in load forecasting models [4]. It also showed excellent performance for efficient parameter tuning [5] and balancing long/short dependencies [6] in load forecasting. The BO algorithm was employed to optimize the ensemble weights [7], hyperparameter tuning [8], and model stochastic nature of PV generation [9]. The GA was used for high-dimensional parameter optimization [10] and feature selection with dynamic interactions [11]. In [12], the authors employed the RBF-based surrogate optimization for reducing evaluation costs in optimization. Gradient descent optimization for the LSTM-based wind power prediction model was explored in [13]. The authors in [14] studied the ensemble deep learning model with quantile regression to achieve high generalization and reliability. Random and grid searches were studied in [15] to achieve computational efficiency in big data and hyperparameter optimization [16]. The hybrid optimization model also showed promising performance in handling various aspects like uncertainties in renewable resources [17], long-term dependencies in grids [18], and reliability and autocorrelation issues [19]. The authors in [20] investigated the impact of fixed, single-axis, and dual-axis tracking mechanisms on PV capacity enhancement. Their study demonstrated that dual-axis tracking systems can significantly increase PV contributions while maintaining operational stability in isolated Low-Voltage (LV) networks, such as the PV system on Ikaria Island, Greece. However, the study did not address the dynamic hosting capacity of the network or explore real-time and short-term forecasting of hosting capacity using data-driven models, which are critical for operational flexibility under high DER variability. The day-ahead optimization algorithm for energy communities was studied in [21], focusing on minimizing operational costs under net billing. By considering PV generation, battery energy storage systems (BESS), and flexible loads, the study demonstrated that, in a hypothetical Greek energy community, costs could be reduced by up to 25% through BESS integration and by up to 24.5% through cooperative energy sharing. However, the study relied on static day-ahead scheduling and did not address real-time hosting capacity forecasting or adaptive operational strategies.

DOE has gained much attention among researchers. In [22], the authors highlighted real-time tracking and DER integration scalability through the Alternating Direction Method of Multipliers (ADMM). They proposed a two-stage DOE-based demand control strategy, ensuring compliance with network statutory limits. DOE allocation in MV-LV distribution networks was investigated in [23], ensuring fair allocation of network HC while maintaining voltage stability. In this study, the authors introduced a top-down two-stage DOE allocation, using real-world Australian data. DER integration through Flexibility Envelopes (FEs) for economic dispatch of power and flexibility planning in power system operations was discussed in [24]. The authors claimed that the proposed model can improve renewable energy integration and system-wide flexibility. DOE-based power system operation through the Active Network Management (ANM) and DOE allocation framework in [25] provided a principled approach for DOE allocation to integrate with the market mechanism. In [26], the authors proposed a deterministic method to calculate the Robust DOE (RDOE) to ensure network compliance considering the voltage and thermal constraints and load/generation variability. Residential DER integration was studied in [27] utilizing the available HC of the distribution network. In the study, the authors focused on the scalability of DOE frameworks in real-world systems and implemented time-varying DOE for household DER participation. The fair allocation of dynamic export limits in advance through the day-ahead DOE estimation for low-voltage networks was investigated in [28]. The authors emphasized handling uncertainty in load and solar irradiance in their study.

Despite extensive research on optimization strategies, a substantial gap remains in developing a unified, adaptive, and resilient optimization framework for HC analysis and forecasting in distribution networks with integrated DER scenarios. Existing studies emphasized specific algorithms like PSO, BO, and GA to focus on selected domains. Moreover, the DOE presents a relatively undiscovered domain for the HC analysis and forecasting of next-generation distribution networks. These indicate the lack of a hybrid approach that can address the real-time adaptability, scalability, and sensitivity to dynamic temporal and spatial dependencies. However, the existing models are unsuccessful in accommodating the dynamic nature of the network and DER variables, missing explicit integration of time-sensitive variables and limited adaptability to highly dynamic and large-scale datasets typical in DER-integrated networks. The Sensitivity-Enhanced Recurrent Neural Network (SERNN) forecasting model addresses temporal dependencies and DER integration attributes with its dynamic sensitivity gate [29]. Therefore, an advanced optimization framework is required to fine-tune its hyperparameters and feature selection dynamically to achieve optimal performance and scalability, ensuring real-time adaptability to distribution networks’ stochastic and dynamic nature.

Table 1. Comparison of the proposed approach with previous methods.

Study/Approach	Optimization Technique	Network Level	DOE Constraints	Real-Time Adaptation	Temporal Dependency Handling
PSO for PV HC	PSO	LV	Not included	No	Limited
GA for DER optimization	GA	MV	Not included	No	Limited
BO for PV stochastic modeling	Bayesian Optimization (BO)	LV	Not included	No	Some
DOE with deterministic constraints	None (Deterministic method)	LV/MV	Included	No	Limited
Proposed (SG-PSO-BO)	SG-PSO + Bayesian Optimization	LV	Included (DOE)	Real-time HC adaptation	Explicit temporal sensitivity via SERNN

compares the proposed SG-PSO-BO hybrid approach with the existing ones.

The proposed SG-PSO-BO optimization framework is adaptable across voltage levels. This study focuses on low-voltage distribution networks due to their unique operational challenges. The low-voltage distribution network experiences frequent network constraint violations, such as voltage fluctuations, reverse power flow, thermal violations, and equipment overloading issues due to high penetration of DER. These issues necessitate dynamic constraint formulation and objective functions within the hosting capacity analysis and optimization process, making LV network hosting capacity forecasting both technically critical and distinct from MV or HV contexts.

This research proposes a hybrid optimization framework that integrates Sensitivity-Gated-Particle Swarm Optimization (SG-PSO) and Bayesian Optimization (BO) to optimize the SERNN model for real-time HC forecasting, complying with the real-time DOE constraints. The framework is designed to capture dynamic relationships among distribution network variables, DER, and weather features, while explicitly modeling the time sensitivity of the variables using the sensitivity gate. The optimization framework minimizes the forecasting error by refining the weights, biases, and hyperparameters of the SERNN model to capture the dynamic relationships between network variables, DER outputs, and weather conditions. The contribution of the study can be summarized as follows:

(a)

Hybrid SG-PSO-BO optimization for adaptive hyperparameter and weight tuning in hosting capacity forecasting model:

The sequential hybrid optimization framework integrates SG-PSO for parameter optimization, integrating sensitivity dynamics of the SERNN and BO for hyperparameter tuning. The SG-PSO calculates the optimal weights and biases of the SERNN model by capturing complex, non-linear relationships among network, DER, and weather variables. In contrast, the BO stage optimizes model hyperparameters to ensure global performance, balancing accuracy, and computational efficiency. This hybrid approach supports scalable, real-time forecasting, efficiently handling large, diverse datasets, and improving the model’s adaptability to real-world scenarios.

(b)

Integration of the sensitivity gate of the SERNN model with the SG-PSO-BO framework under DOE constraints:

The optimization framework uniquely integrates the sensitivity gate functionality of the SERNN forecasting model to capture the temporal dynamics under DOE constraints. The SG-PSO optimization explicitly modulates the network, DER, and weather variables’ temporal dependencies as dynamic input features. The fitness function balances forecasting accuracy and sensitivity gate stability in the proposed hybrid optimization model. On the other hand, the regularization of the sensitivity gate output prevents temporal overfitting and ensures model robustness.

(c)

Real-time hosting capacity forecasting framework with DOE adaptation:

The study develops a real-time DOE-aware optimization framework that integrates voltage stability, thermal constraints, and DER import/export limits into the forecasting model’s decision-making process. This framework enhances grid stability, improves network visibility, and supports sustainable DER integration by ensuring safe operational limits and avoiding transformer overloading, voltage violations, and excessive DER curtailment.

The rest of the paper is organized as follows: Section 2 presents the sensitivity-aware optimization framework. Section 3 discusses the results, and Section 4 highlights the study outcomes and practical implications. Finally, Section 5 summarizes the key findings, implications of the study, and future research direction.

2. Sensitivity-Aware Optimization Framework for DOE-Constrained Hosting Capacity Forecasting

The core of the proposed model is the SERNN, designed to capture the temporal dependencies and dynamic characteristics of the input variables, like network and DER parameters, and weather variables [29]. The unique sensitivity gate, processing time-sensitive input (e.g., hour), adjusts the dynamic response of the model to the periodic nature of the network and DER variables. To enhance performance, a hybrid optimization framework is deployed, comprising modified PSO equipped with dynamic inertia weights and BO for hyperparameter tuning. This hybrid approach ensures effective learning of the model parameters under dynamic network conditions, including the learning rate, number of neurons, batch size, number of hidden layers, and activation units.

Figure 1 illustrates overall process flow of the sensitivity-aware optimization framework. After cleaning and removing outliers, the input data are normalized to ensure data integrity. The feature engineering process uses cyclic and binary encoding for time-dependent variables, as well as DER variables and weather parameters. The modified PSO and BO provide optimized hyperparameters for training, validation, and testing of the forecasting model. The performance indices like Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), and Coefficient of Determination ($R^2$) compare the performance with the baseline models. The following outlines the overall process flow of the proposed optimization framework.

2.1. Dynamic Operating Envelope (DOE) Constraints

The DOE defines the limits of time-varying DER import/export, ensuring network constraints for real-time control while optimizing the HC of the distribution network [25]. Unlike static constraints, DOE enables real-time control of DER injections, ensuring grid stability under varying load and generation conditions [30]. The proposed HC optimization framework incorporates the voltage and thermal limits, transformer overloading, and power import/export limits as the key DOE constraints as stated below.

• Voltage constraint: The voltage at node i at time t must be within permissible limits. Due to power export/import from DER, this voltage may violate the limits. To prevent such violations, the DOE dynamically adjusts the power export/import using a sensitivity-based voltage response function by modifying the allowable DER injection at each node. Applying the voltage response factor $S_{ij}$ at base voltage $V_i\left(t\right)$ of node i, the node voltage $V_i\left(t\right)$ at time t can be represented by,

  $$V_i\left(t\right)=V_{\mathrm{base}}+\sum _{j\in N}{S}_{ij}{P}_j\left(t\right)$$

  (1)
  Thus, the DOE ensures the node voltage remains within limits:

  $$V_{lower}\le V_i\left(t\right)\le V_{upper},\forall i\in N,\forall t$$
• Thermal limit: The current passing through the i conductor $I_i\left(t\right)$ at time t must be within the rated capacity $I_{i,rated}$ such that $I_i\le I_{i,rated}$.
• Transformer overloading: The transformer must ensure that its rated capacity will not be violated such that $P_{xformer}\le P_{rated}$.
• Export/Import constraints: The network must ensure that the power export/import through node i at time t always remains within permissible limits following the export/import limits as $P_{export,i}\left(t\right)\le P_{export,max}$ and $P_{import,i}\left(t\right)\le P_{import,max}$.

These constraints enable dynamic adjustment of DOE constraints for stable network operation and real-time HC forecasting of the distribution network.

2.2. Hosting Capacity (HC) Formulation

HC measures the distribution network capacity to integrate DER without violating the network operating constraints [31]. The HC ($H{C}_t$) of the distribution network with the integrated DER at time t can be expressed by Equation (2) [32],

$$H{C}_t=max\sum _{i\epsilon J}DE{R}_{i,t}subjectto\left\{\begin{array}{c}{v}_{lower}\le v_i\le v_{upper}\hfill \\ I_i\le I_{rated}\hfill \\ P_{xformer}\le P_{rated}\hfill \end{array}\right.$$

(2)

where $DE{R}_{i,t}$ is the power injected from the integrated DER at time t.

Conventional approaches fail to capture real-time variations in network topology, demand fluctuations, and DER intermittency, resulting in sub-optimal HC limits. To overcome these limitations, the DOE-enhanced HC estimation approach introduces time-dependent constraints and dynamically regulates DER injections, improving responsiveness to real-time conditions. This adaptive approach is modeled in Equation (3), which incorporates the time-dependent limits and penalty factors to ensure the dynamic constraint compliance subject to the DOE limits. Equation (3) dynamically calculates the lower DOE considering the allowable under-voltage limits of the network based on the minimum power intake from the integrated DER, maintaining the network operation constraints. It considers real-time load conditions and network topology, ensuring that the lower bound adapts to changing demand and system conditions to prevent network instability.

$$H{C}_t=max\sum _{i\in J}{P}_{DER,i}\left(t\right)-{\lambda }_{DOE}\sum _{i\in J}max{(0,P_{DER,i}\left(t\right)-P_{DER,DOE,i}\left(t\right))}^2$$

(3)

The penalty function in Equation (3) enforces strict adherence to DOE constraints. The quadratic function ensures that small violations incur mild penalties, while larger violations are penalized more aggressively, discouraging unsafe operating conditions. The voltage sensitivity factor ($S_{ij}$) captures the DER-induced voltage variations. The penalty term ${\lambda }_{DOE}$ ensures that DOE violations are actively penalized, preventing excessive DER injections. Thus, the HC reflects dynamic, real-world operating conditions more accurately.

2.3. Data Preparation

The structured data preparation process is essential for accurate and robust HC forecasting, supporting sustainable and efficient DER integration. The data preparation process includes power flow analysis, HC estimation, feature engineering, and data processing. The following outlines the time-series data preparation process for the forecasting model.

2.3.1. Power Flow Analysis

The power flow analysis was conducted to determine the network HC with integrated DER. Based on the real-time DOE constraints, the simulation used the network parameters, DER characteristics, and weather data. The simulation employed the OpenDSS, interfaced with Python 3.11 using the py-DSS-interface 2.1.1 package, to simulate the dynamic interactions between network and DER variables under varying operational scenarios.

Starting from the baseline capacity, DER integration was incrementally increased until technical constraints were violated. Key constraints included voltage compliance (0.95  per unit ≤ voltage ≤ 1.05 per unit), thermal limits of the conductors, transformer loading capacities, and power export/import limits. The analysis generated hourly time-series data of critical feature variables: total circuit power, maximum and minimum node voltages, maximum current passing through the conductors, maximum power passing through the transformers measured at the secondary terminals, PV output, irradiance, temperature, and network HC, as defined in Equation (3).

2.3.2. Feature Engineering

To enhance the forecasting model performance, the temporal features were encoded using sine–cosine encoding to capture the cyclical patterns [33]. The hour of the day (h) was encoded as,

$$h_{sine}=sine(2\pi \ast h/24)$$

(4)

$$h_{cos}=cos(2\pi \ast h/24)$$

(5)

Additionally, one-hot encoding technique [34] modified PV output and irradiance to represent their intermittency,

$$f\left(x_t\right)=\left\{\begin{array}{cc}1& if{x}_t>0;\\ 0& otherwise\end{array}\right.$$

(6)

where $x_t$ denotes the value of the feature variable at time t, separating active from inactive DER/weather conditions.

2.3.3. Data Processing

To ensure quality and consistency, the dataset was cleaned to remove duplicates, outliers, and missing values. All variables aligned at hourly intervals to synchronize network and DER dynamics at uniform temporal resolution. The Min–Max normalization [35] was applied to scale variables and mitigate disproportionate influences. The dataset was split into training $(80\%)$ and testing $(20\%)$ subsets, with additional splitting of the testing data into validation sets to reduce overfitting and enhance generalization.

This comprehensive data preparation process ensures the model receives high-quality, temporally coherent input, capturing the realistic network and DER integration, forming the basis for accurate and reliable HC forecasting.

2.4. Sensitivity-Enhanced Recurrent Neural Network (SERNN) Forecasting Model

The Sensitivity-Enhanced Recurrent Neural Network (SERNN) is designed to capture time-sensitive dynamics of distribution networks, DER, and weather inputs [29]. Figure 2 illustrates the data flow and internal gating mechanism of the SERNN model.

• Input Gate: Regulates the flow of new information from the current input, previous hidden state, and cell state using a sigmoid activation function:

  $$i_t=\sigma (W_i[x_t;h_{t-1};C_{t-1}]+b_i)$$

  (7)
• Forget Gate: Controls flow of information from the previous memory by applying a sigmoid function that determines information retention and discards conditionalities:

  $$f_t=\sigma (W_f[x_t;h_{t-1};C_{t-1}]+b_f)$$

  (8)
• Sensitivity Gate: Enables adaptive weighting of time-dependent external variables (e.g., hour of the day, irradiance, PV output) to influence state updates dynamically.

  $$s_t=\sigma (W_s{x}_s+b_s)$$

  (9)
• Cell State Update: The candidate update $i_{(t,s)}$ is modulated by the sensitivity gate and the input gate, shaping the information flow by the $tanh$ function.

  $$i_{(t,s)}=tanh[\left(\sigma (W_s{x}_s+b_s)(W_i[x_t;h_{t-1};C_{t-1}]+b_i)\right]$$

  (10)
  The new cell state is computed as: $C_t=f_t{C}_{t-1}+i_{t,s}$
• Hidden State and Output: The hidden state is derived from the updated cell state through the $tanh$ activation function.

  $$i{h}_t=tanh\left(C_1\right)$$

  (11)
  The linear transformation of the current hidden state combined with an output bias produces the final output of the model as:

  $$y_t=W_o{h}_t+b_o$$

where,

• $\sigma$: activation function;
• $W_i$: weight factor for the input gate;
• $x_t$: input matrix at time t;
• $h_{t-1}$: previous hidden state;
• $C_{t-1}$: previous cell state;
• $b_s$: input bias/sensitivity bias;
• $W_f$: weight factor for the forget gate;
• $b_f$: forget gate bias;
• $W_s$: weight matrix for sensitivity gate;
• $x_s$: sensitivity gate input;
• $f_t$: forget gate input;
• $i_{(t,s)}$: cell state update with sensitivity input;
• $W_o$: output weight matrix;
• $b_o$: output bias.

The modified architecture enables the SERNN to better capture temporal dependencies and context-aware features that influence HC forecasting under realistic and dynamic operating conditions.

The gate value b is not a fixed threshold. Within the proposed sensitivity gate mechanism, b is determined through a data-driven learning process. The model adaptively learns b during training via backpropagation, enabling it to dynamically capture the temporal sensitivity of input features without manual threshold selection.

2.5. Objective Function

The objective of the proposed optimization framework is to minimize forecasting error in HC prediction using the SERNN satisfying network and DOE constraints. The optimization problem is formulated as a multi-objective function that balances forecasting accuracy, model stability, constraint compliance, and hyperparameter optimization. The optimization is conducted using the SG-PSO-BO framework, which tunes model weights, biases, and hyperparameters using the objective function defined in Equation (12).

$$\begin{array}{cc}\hfill J_{SG-PSO-BO}^{\prime }& =\sum _{i=1}^N{\beta }_i\ell (y_i,{\widehat{y}}_i)+{\lambda }_s\left(t\right)\sum _{i=1}^N{\parallel g_s(s_i;w_s,b_s)\parallel }^2\hfill \\ \hfill & +\gamma \sum _{i=1}^N{\parallel \nabla \ell (y_i,{\widehat{y}}_i)\parallel }^2+J_{constraints}+J_{DOE}+J_{BO}\hfill \end{array}$$

(12)

where,

• $\ell \left(y_i,{\widehat{y}}_i\right)$: loss function of the forecasting model;
• ${\beta }_i$: feature importance factor;
• $g_s(s_i;w_s,b_s)$: sensitivity factor controlling time-sensitive variables;
• ${\lambda }_s\left(t\right)$: time-dependent regularization coefficient for the sensitivity gate function $g_s(s_i;w_s,b_s)$;
• $J_{constraints}$, $J_{DOE}$: DOE-based penalty factors ensuring HC constraints compliance;
• $J_{BO}$: optimization function for hyperparameter tuning;
• $\gamma {\displaystyle \sum _{i=1}^N}{\parallel \nabla \ell (y_i,{\widehat{y}}_i)\parallel }^2$: gradient penalty for enhancing model stability.

The loss function $\ell \left(y_i,{\widehat{y}}_i)\right)$ measures the difference between the actual HC $\left(y_i\right)$ and model prediction $\left({\widehat{y}}_i\right)$. This function helps to achieve robustness and sensitivity to outliers:

$$\ell (y_i,{\widehat{y}}_i)=\alpha ·{\displaystyle \frac{1}{n}}\sum _{i=1}^n{(y_i-{\widehat{y}}_i)}^2+(1-\alpha )·{\displaystyle \frac{1}{n}}\sum _{i=1}^n|y_i-{\widehat{y}}_i|$$

where the weighting factor $\alpha$ adjusts the balance between MAE and MSE.

The regularization mechanism incorporates the sensitivity gate function $\left\{g_s(s_i,w_s,b_s)\right\}$ to modulate the time-sensitive feature variables. The regularization parameter $\left({\lambda }_s\right)$ serves as a control mechanism to balance the contribution of the sensitivity gate in enhancing the temporal sensitivity of the model without introducing instability or overfitting. This mechanism improves the model generalization and temporal adaptability.

The gradient-based penalty discourages abrupt parameter changes by penalizing large gradients in the loss function, thus stabilizing the model’s learning process.

The constraint penalty $J_{constraints}$ ensures model safety and operational stability by enforcing voltage and thermal limits:

$$J_{constraints}=J_{voltage}+J_{thermal}$$

The voltage constraint penalty $J_{voltage}$ maintains the voltage within permissible limits whereas the thermal constraint penalty $J_{thermal}$ ensures the permissible loading conditions:

$$J_{voltage}={\lambda }_1\sum _{i=1}^{N}max{(0,V_i\left(t\right))}^2$$

$$J_{thermal}={\lambda }_2\sum _{i=1}^{N}max{(0,P_i\left(t\right)-P_{rated})}^2$$

where ${\lambda }_1$ and ${\lambda }_2$ denote penalty coefficients for voltage and thermal violations, respectively. These terms penalize instances where voltage or power flow exceed permissible limits.

The DOE-based constraint penalty $J_{DOE}$ prevents the predicted HC from exceeding the allowable DER integration limit defined by real-time DOE regulations:

$$J_{DOE}={\lambda }_3\sum _{i=1}^{N}max{(0,P_{DER,i}\left(t\right)-P_{DER,DOE,i}\left(t\right))}^2$$

where $P_{DER,DOE,i}\left(t\right)$ is the real-time network HC limit imposed by DOE and ${\lambda }_3$ controls the severity of the penalty.

The objective function in Equation (12) integrates prediction accuracy, temporal regularization, and constraint adherence into a unified optimization framework. Thus, the proposed hybrid optimization framework achieves robust and scalable performance for time-sensitive HC forecasting in distribution networks.

2.6. Stages of Optimization Process

The proposed hybrid optimization framework leverages the benefits of Sensitivity Gate, PSO, and BO optimization algorithms to optimize the weights, biases, and hyperparameters. The framework addresses the temporal dependencies, stabilizes the impact of sensitivity input by modulating sensitivity gate output, and enhances forecasting performance through an efficient and scalable optimization process.

2.6.1. Stage 1: Sensitivity-Gated Particle Swarm Optimization (SG-PSO)

The Sensitivity-Gated Particle Swarm Optimization (SG-PSO) optimizes weights $\left(w\right)$ and biases $\left(b\right)$ of the SERNN model, including the parameters controlling the sensitivity gate. It minimizes the forecasting error while the model dynamically adjusts time-sensitive DOE constraints. The algorithm dynamically updates the particle velocity by incorporating the DOE penalties and sensitivity adjustment, as shown in Equation (13):

$$V_i^{t+1}=\omega \left(t\right)V_i^t+c_1{r}_1(P_{best,i}-P_i^t)+c_2{r}_2(G_{best}-P_i^t)+S_i\left(t\right).\Delta P_{DOE,i}$$

(13)

where,

• $w\left(t\right)$: Inertia weight that controls exploration–exploitation;
• $c_1,c_2$: Acceleration coefficients to find the local and global best;
• $r_1,r_2$: Random factors that ensure the stochastic movement of particles;
• $S_i\left(t\right)$: Sensitivity gate function to dynamically adjust DOE constraints;
• $\Delta P_{DOE,i}$: DOE violations correction term.

The sensitivity function $S_i\left(t\right)$ in Equation (13) dynamically adjusts the exploration–exploitation balance such that $S_i\left(t\right)=exp\left(-{\lambda }_s\left(t\right){\sum }_{j=1}^{N}max(0,P_{DER,j}\left(t\right)-P_{DER,DOE},j\left(t\right))\right)$. The exponential term reduces exploration to ensure faster convergence during higher DOE violations. The position of each particle is updated by: $P_i^{t+1}=P_i^t+V_i^{t+1}$.

The fitness function evaluates the forecasting accuracy and compliance with DOE constraints by minimizing the objective function:

$$\begin{array}{c}\hfill J_{SG-PSO}^{\prime }=\sum _{i=1}^N\ell (y_i,{\widehat{y}}_i)+{\lambda }_s\left(t\right)\sum _{i=1}^N{∥g_s(s_i;w_s,b_s)∥}^2+J_{voltage}+J_{thermal}+J_{DOE}\end{array}$$

(14)

The SG-PSO terminates when the objective function converges or the algorithm reaches the maximum number of iterations, ensuring efficient convergence.

2.6.2. Stage 2: Hyperparameter Optimization Using the Bayesian Optimization (BO)

BO effectively solves computationally expensive optimization problems when the objective function lacks a closed-form expression and derivatives are challenging [36]. After optimizing model weights and biases using the SG-PSO algorithm, the BO is employed to fine-tune the hyperparameters $\left(\theta \right)$ such as learning rate ($\alpha$), batch size (B), dropout rate (d), and hidden layers size (h). The acquisition function minimizes the forecasting error and regularization parameter defined as:

$$J_{BO}\left(\theta \right)=min\left(J_{SG-PSO}^{\prime }(w^{*},b^{*},\theta )+J_{DOE}\right)$$

(15)

where $\theta$ denotes the hyperparameter set to be optimized, $w^{*}$ and $b^{*}$ represent the optimized weights and biases obtained from SG-PSO, and $J_{DOE}$ ensures DOE-compliant constraints during optimization.

The Gaussian Process Regression (GPR) model [37] is used to maintain the probabilistic estimate of the objective function in the BO process. At each iteration, the next candidate set of hyperparameters is selected by maximizing the Expected Improvement (EI) function [38]:

$$EI\left(\theta \right)=(\mu \left(\theta \right)-J_{best})\Phi \left(Z\right)+\sigma \left(\theta \right)\phi \left(Z\right)$$

(16)

where $\mu \left(\theta \right)$ and $\sigma \left(\theta \right)$ represent the mean and variance of the GPR model, $J_{best}$ is the current best observed value of the objective function, $\Phi \left(Z\right) \mathrm{and} \varphi \left(Z\right)$ denote the Cumulative Distribution Function (CDF) and Probability Density Function (PDF) of the standard normal distribution, respectively.

The optimization process terminates either when the maximum number of iterations is reached or the improvement of $J_{BO}$ falls below a specified threshold. The BO enhances model stability, avoids overfitting, and maintains a balance between computational efficiency and forecasting accuracy. Furthermore, the model incorporates DOE constraints dynamically into hyperparameter selection. The combined SG-PSO and BO framework provides a robust, scalable, and accurate optimization strategy for the SERNN forecasting model. The following outlines the overall solution approach.

2.7. Overall Solution Approach

The solution approach aims to optimize forecasting accuracy and network stability while ensuring compliance with HC and DOE constraints. It integrates data processing and feature engineering, SG-PSO, for updating the SERNN model weights and biases, and BO for hyperparameter tuning. This comprehensive approach supports real-time adaptability and regulatory compliance in next-generation distribution networks.

2.7.1. Data Processing and Feature Engineering

The necessary data for the forecasting model was generated through HC analysis using power flow analysis, feature engineering, and data processing steps, as discussed in Section 2.3. This process prepares the dataset to capture the temporal dependencies and feature variability, enabling the model to reflect the real-world distribution network scenarios with integrated DER.

(1)

Power Flow Analysis: The network and DER variables were collected through the power flow analysis using OpenDSS. The py-DSS-interface application library was used to interface with OpenDSS for data extraction and further analysis.

(2)

Feature Engineer: Temporal dependency and cyclical behavior of the time-dependent variables were captured using sine–cosine encoding. Binary representation was applied to weather data and DER output to reflect their intermittency and non-linear characteristics.

(3)

Sensitivity Input Selection: Time-dependent input, specifically the hour of the day, was encoded as a sensitivity factor to enhance the temporal adaptability of the model in dynamic operational scenarios.

(4)

Data Preparation for Model Training: Input features $X_t$ and target variables $y_t$ were structured to incorporate temporal dependency and operational constraints across a look-back window $\left(t\right)$ and forecasting horizon $\left(T\right)$ for real-time HC forecasting. The $X_t$ vector includes historical network, DER, and weather variables over the look-back window, while $y_t$ forecasts the network HC at a future time $t+T$. This formulation integrates DOE constraints by aligning prediction with real-time constraints such as voltage, thermal limits, and the overloading threshold. Therefore, the model supports sustainable DER integration and efficient network operation.

2.7.2. Hybrid Model Optimization Using the SG-PSO and Bayesian Optimization

The proposed hybrid optimization framework integrates the SG-PSO for tuning weights and biases and Bayesian Optimization (BO) for hyperparameter tuning, thereby enhancing the accuracy and robustness of the SERNN in HC forecasting.

Stage 1: SG-PSO for Weights and Biases Optimization

SG-PSO is employed to minimize the SERNN forecasting error and stabilize the sensitivity gate output by optimizing the weights $\left(w\right)$ and biases $\left(b\right)$ through the following steps:

(1)

Particle initialization: Each particle represents a candidate set for the SERNN model parameters. Each particle encodes a candidate solution comprising the weights $\left(w\right)$ and biases $\left(b\right)$ of the SERNN model.

(2)

Fitness function evaluation: Minimize the forecasting error $(\ell (y_i,{\widehat{y}}_i))$ and regularize sensitivity gate output $g_s$ for stability by evaluating the objective function, as defined in Equation (12).

(3)

Velocity and position updates: Particles iteratively update their position convergence to the optimal weights $\left(w^{*}\right)$ and biases $\left(b^{*}\right)$. The algorithm continues until minimized forecasting error is achieved or a predefined iteration limit is exhausted.

(4)

Output: Optimized weights $\left(w^{*}\right)$ and biases $\left(b^{*}\right)$ of the SERNN model to minimize error and stabilize sensitivity gate outputs.

Stage 2: Bayesian Optimization (BO) for Hyperparameter Tuning

The Bayesian Optimization (BO) is deployed to fine-tune the hyperparameter set of the SERNN model using the following steps:

(1)

Define the search space: The key hyperparameters like learning rate, batch size, and number of LSTM units are defined with upper and lower boundaries based on the network design and operational constraints.

(2)

Optimize the acquisition function: The EI acquisition function is employed to balance between exploration and exploitation:

$$EI\left(\theta \right)=(\mu \left(\theta \right)-J_{best})\Phi \left(Z\right)+\sigma \left(\theta \right)\phi \left(Z\right)$$

(3)

Iterative updates: The algorithm continues until it achieves the desired hyperparameter convergence.

(4)

Output: The model provides the optimal hyperparameter set that enhances model generalization and forecasting accuracy.

(5)

Termination: The process terminates when the desired improvement becomes negligible over successive iterations.

2.8. Algorithm of the SG-PSO-BO Hybrid Optimization Framework

The Algorithm 1 outlines a stepwise optimization process for dynamic HC forecasting, ensuring compliance with real-time operational constraints.

Model Training and Evaluation

The SERNN forecasting model is trained with processed data using the optimal parameters derived from the SG-PSO-BO. The model dynamically follows the network and DOE constraints during the training process. The forecasted HC is compared with performance evaluation metrics and compared with standard optimization models.

(1)

Model training: The SERNN forecasting model is trained using the processed time-series data derived from the test networks and the optimal parameters gathered from the SG-PSO-BO. The model incorporates the network and DOE constraints to prevent any possible violations.

(2)

Model Evaluation: The model is evaluated using performance metrics such as MAE, RMSE, and Coefficient of Determination ($R^2$):

(a)

Mean Absolute Error (MAE): $\mathrm{MAE}={\displaystyle \frac{1}{n}}{\sum }_{i=1}^n|y_i-{\widehat{y}}_i|$;

(b)

Root Mean Square Error (RMSE): $\mathrm{RMSE}=\sqrt{{\displaystyle \frac{1}{n}}{\sum }_{i=1}^n{(y_i-{\widehat{y}}_i)}^2}$;

(c)

Coefficient of Determination ($R^2$): ${\mathcal{R}}^{\mathcal{2}}=1-{\displaystyle \frac{\sum {(y_i-{\widehat{y}}_i)}^2}{\sum {(y_i-\overline{y})}^2}}$.

(3)

Model Validation: The performance of the trained model is compared against the baseline models to validate its performance improvements in forecasting efficiency.

By incorporating network and DOE constraints at every stage of the optimization framework, the final forecasting model ensures robust and scalable HC forecasting, suitable for real-world applications.

Algorithm 1 SG-PSO-BO optimization for HC forecasting.

1: Input: Feature variables X, sensitivity input S, target variable y (HC) 2:    Network and DOE constraints, Hyperparameter search space $\Theta$ 3: Output: Optimized weights $w^{*}$, biases $b^{*}$, and hyperparameters ${\theta }^{*}$ 4:    Final trained SERNN model   Step 1: Data Processing and Feature Engineering 5: Load power system data 6: Apply feature engineering techniques and look-back window transformation   Step 2: SG-PSO for Weights and Biases Optimization 7: Initialize particle swarm with randomly generated SERNN weights and biases. 8: for each particle do 9:     Evaluate DOE-aware fitness function 10:     Update velocity and position using sensitivity input s. 11:     Apply dynamic HC and DOE constraints 12: end for 13: Store optimized weights $w^{*}$, biases $b^{*}$.   Step 3: Bayesian Optimization for Hyperparameter Tuning 14: Initialize Bayesian Optimization using Gaussian Process (GP) surrogate model 15: Define the hyperparameter objective function: 16: for each iteration do 17:     Select hyperparameters using the EI acquisition function 18:     Evaluate the objective function and update GP model. 19: end for 20: Store optimal hyperparameter set ${\theta }^{*}$.   Step 4: Final Model Training and Evaluation 21: Train the SERNN model with optimized parameters $w^{*},b^{*},{\theta }^{*}$. 22: Validate the model on the test dataset. 23: Compute evaluation metrics and compare with baseline models   Step 5: Real-time HC forecasting 24: Deploy the optimized SERNN model for HC forecasting 25: Dynamically adapt to operational based on real-time network conditions 26: Continuously update model parameters and hyperparameters as required

3. Results

This section evaluates the performance of the proposed SG-PSO-BO hybrid optimization framework, emphasizing its capability to improve HC accuracy, computational efficiency, and adaptability under real-time conditions. Model effectiveness was assessed using key performance metrics like MAE, RMSE, and Coefficient of Determination ($R^2$). The framework was also benchmarked against several baseline models, including PSO, GA, Random Search, Grid Search, and TLBO. Furthermore, convergence behavior, computational efficiency, and scalability were analyzed to validate the model’s performance for real-time operational conditions.

3.1. Simulation Setup

This study evaluates the performance of the proposed hybrid SG-PSO-BO framework by implementing it on the SERNN model for the HC forecasting in the next-generation distribution networks. The evaluation was conducted using time-series data from both the IEEE-123 test network and a real Australian Distribution Network. The DE, including PV, EVs, and BESS were modeled according to the OpenDSS simulation architecture [39]. Weather information like temperature (Figure 3a) and irradiance (Figure 3b) data, required for PV operation, were obtained from the Bureau of Meteorology (BoM), Australia. Wind speed was not included as a primary input feature in this study, as the focus is on low-voltage distribution networks integrated with DERs such as PV, BESS, and EVs, which are not directly influenced by wind conditions.

Time-series datasets were generated through the power flow simulation using OpenDSS [40], iteratively increasing PV, EVs, and BESS penetration while monitoring for HC constraint violations. Hourly data of network, DER, and Weather variables were recorded over a one-year period to reflect realistic variability. Load profiles were derived from real Australian distribution network data. PV systems were initially set at 10 kVA with a unity power factor, and their capacity was incremented by 10% in each iteration. EVs and BESS were operated in ‘Schedule’ mode with the following parameters:

(a)

Rated Capacity: 120 kW and 360 kWh;

(b)

Initial Stored Capacity: 50% of rated capacity;

(c)

Reserve Capacity: 20% of rated capacity;

(d)

Charging Rate: 25% of rated capacity;

(e)

Discharging Rate: 50% of rated capacity.

In this study, different charging and discharging rates were assigned to replicate the real-world scenarios. The lower charging rate was set to utilize the PV power and increase. On the other hand, the higher discharging was set to support the network through rapid response during peak load period. This operational setup aligns with real-world DER usage patterns and grid-support strategies. A 20% reserve capacity threshold was applied in this study to define the minimum allowable State of Charge (SoC) for both BESS and EV systems. This value represents a technical lower limit to prevent full battery depletion and maintain basic operational flexibility. While industry practice often favors higher reserve margins to account for battery aging, safety margins, and operational contingencies, the 20% level was chosen to explore maximum discharge flexibility within safe operating conditions. To maximize the available storage capacity of BESS and EVs, the upper SoC limit was set to 100% in this study, allowing for simplified simulation under ideal storage availability conditions. This approach enabled the assessment of maximum DER utilization by evaluating system performance at the theoretical upper limit of storage capacity. However, in practical applications, the maximum SoC is typically capped at 90–95% in accordance with industry best practices to mitigate battery degradation and extend battery lifespan.

The Australian Distribution Network considered in this study comprises 17 distribution transformers configured in a Delta-Wye topology. The network integrates 8 PV systems, among which five are three-phase and three are single-phase installations, reflecting a typical mix of residential and small commercial rooftop PV deployments. Additionally, the network includes eight energy storage systems, each of which is capacity-matched (in KWh rating) to its corresponding PV installation. This configuration enables the analysis of coordinated PV–battery interactions and their impact on network hosting capacity under varying load and generation conditions. Load data were recorded at one-hour intervals over a 356-day period, providing detailed insight into daily and seasonal demand patterns. Network and DER parameters—including PV output, state of charge (SoC) of storage systems, node voltages, line currents, and transformer loading—were obtained using time-series power flow analysis conducted at hourly resolution over 365 consecutive days. This dataset enables a comprehensive evaluation of the dynamic behavior of both demand and DER across the entire network, capturing their combined impact on hosting capacity in different operating scenarios.

In the IEEE-123 test network, DERs were randomly allocated on the eligible nodes, with total DER penetration levels incrementally increased during the simulation. The sizing of individual PV and BESS systems was assigned proportionally based on average residential/commercial load demands. BESS follows a ratio of 1:2 with PV capacity depending on the scenario. EVs were distributed evenly across residential loads following a realistic adoption rate (30% penetration scenarios).

Forecasting performance was evaluated using MAE, RMSE, and Coefficient of Determination ($R^2$). Both MAE and RMSE were measured in megawatts (MW), consistent with the HC of the test networks. The Coefficient of Determination ($R^2$) is unitless and ranges between 0 and 1, indicating the proportion of variance explained by the model.

All simulations were executed on a system equipped with an 11th Gen Intel(R) Core (TM) i7-1185G7 CPU @ 3.00GHz and 16 GB RAM. Network simulation was carried out using OpenDSS version 9.6.1.1 (64-bit build) interfaced with Python 3.11 via the Py-DSS-Interface 2.0.4 within the PyCharm 2024.3.6 Community Edition. The test networks considered a voltage constraint of 0.95–1.05 per unit, thermal limits based on conductor capacity, and the transformer loading limit defined at the secondary terminal.

3.2. Feature Correlation

Feature correlation analysis identifies the implications of feature variables on target variables by removing irrelevant information. This process enhances model performance by reducing computational complexity, improving learning efficiency, and increasing comprehension of the learning model [41].

Figure 4 shows the correlation analysis of the feature variables using the Spearman, Pearson, and Kendall [42] correlation analysis techniques. The Pearson analysis assessed the linear relationship, Spearman captured the non-linear relationship, and Kendall measured the ranking and dependency between the feature variables and network HC with integrated DER. The analysis revealed that circuit power () and renewable output exhibited strong positive correlations with HC. Strong non-linear dependency (0.74) of time-based cyclic effects was observed for the cosine component of the time of the day (‘Hour’) variable, reflecting significant time-based cyclic effects. On the other hand, irradiance (Irrad) and renewable output (PV_kW) showed strong negative correlation with network HC. Capturing the positive, negative, and non-linear correlation among the feature variables, the findings highlight the effectiveness of the SERNN with SG-PSO-BO hybrid optimization model for accurate and robust HC forecasting of next-generation distribution networks.

3.3. Hosting Capacity Forecasting

Table 2. Optimization model performance for HC forecasting.

Metrices	IEEE-123 Test Network	Australian Distribution Network
MAE	0.21	0.16
RMSE	0.38	0.31
R2	0.97	0.98
LSTM Layers	55	55
Dense Layers	28	30
Learning Rate	0.001	0.01
Execution Time (s)	145	147

summarizes the performance of the SERNN forecasting model optimized using the SG-PSO-BO across two distribution networks with integrated DER: the IEEE-123 Test Network and a real Australian Distribution Network. The table compares key performance metrics, network-specific model architecture, hyperparameter, and execution time.

The results illustrate that the optimized forecasting model demonstrates high forecasting accuracy across both networks, achieving an $R^2$ of 0.97 and 0.98 for the IEEE-123 Australian networks, respectively, indicating a strong correlation between predicted and actual HC. The Australian distribution network achieves lower MAE (0.16) and RMSE (0.31), suggesting better model generalization in real-world operational settings.

The SG-PSO-BO-optimized SERNN forecasting model uses slightly different optimized architectures for both networks. Although the LSTM architecture remains consistent across both cases, slight variations in dense layer count and learning rate were observed. The higher learning rate (0.01) for the Australian distribution network facilitated faster convergence while maintaining comparable execution time (147 s). These results validate the effectiveness and adaptability of the SG-PSO-BO optimization framework in dynamic HC forecasting of next-generation with real-time DOE requirements.

3.4. Forecasting Results (Actual vs. Predicted Hosting Capacity)

The SG-PSO-BO optimization framework was deployed to enhance the efficiency and reliability for optimization of the SERNN model in forecasting HC under DOE conditionalities. The HC forecasting process used the following hyperparameter boundaries:

The parameter ranges in

Table 3. Hyperparameter boundaries of the optimization framework.

Optimizer	Hyperparameter	Parameter Boundary
Adam	Number of Neurons	16, 64
	Learning Rate	$1\times {10}^{-4}$, $1\times {10}^{-2}$
	Batch Size	16, 32
	Epochs	10, 50
	Dropout Rate	$1\times {10}^{-1}$, $5\times {10}^{-1}$
	Number of Hidden Layers	1, 3
	Number of Activation Units	16, 64

were selected based on a combination of best practices in energy time series forecasting and preliminary sensitivity testing. This approach ensured that the model captured both effective configurations and computational efficiency, while providing sufficient flexibility for the optimization algorithm to explore high-performance parameter combinations.

Figure 5a and Figure 5b present a comparison of actual vs. forecasted HC of the IEEE-123 test network and a real Australian Distribution Network, respectively. The results exhibit minimal deviation between measured and predicted HC, demonstrating strong model generalization and reliability across different networks.

3.5. Training Convergence Analysis (Loss (MSE) vs. Epochs)

Figure 6 presents the learning behavior of the SG-PSO-BO optimization framework for both test networks. The steady decreasing trend of the learning curve signifies effective model training, with stable loss after several epochs, ensuring robust training. Lower loss in the Australian distribution network compared to the IEEE-123 test network indicates smoother convergence and better generalization of the framework in real-world scenarios. These results demonstrate that the proposed framework effectively minimizes forecasting errors while maintaining stable learning dynamics.

Figure 6 showed rapid convergence due to the limited number of training epochs (10 epochs per iteration) used during hyperparameter optimization to ensure computational feasibility during the simulation. The $adam$ optimizer with adaptive learning rates and the batch normalization effect accelerates convergence during each training cycle. In practical deployments, more training epochs can be employed to further fine-tune performance after optimization.

3.6. Error Convergence Analysis

The model error minimization process is illustrated in Figure 7, which compares the MAE loss over training epochs for both test networks. The Australian distribution network demonstrates faster convergence than the IEEE-123 test network, indicating model adaptability to real-world network conditions. The consistent decline in MAE throughout training demonstrates the model’s capability for dynamic adjustments of weights and biases to minimize forecasting errors. This capability makes the model more suitable to accommodate dynamic DOE constraints.

3.7. Residual Error Analysis

The residual error analysis in Figure 8 illustrates the deviation between the actual and predicted HC values for both the IEEE-123 test network and the Australian distribution network. The residuals for both networks are closely clustered around zero, indicating high predictive accuracy and minimal systemic bias. Moreover, the Australian distribution network exhibits lower residual variation compared to the IEEE-123 test network, suggesting that the SG-PSO-BO model demonstrates superior generalization capability and robustness in real-world operating conditions.

3.8. Long-Term Prediction Error Analysis

The close alignment between the actual and predicted HC of the SG-PSO-BO optimization framework, as shown in Figure 9, demonstrates robust generalization capability under varying network conditions. The error margin is lower in the Australian distribution network, suggesting that the model effectively incorporates real-time learning adjustments over extended operational periods.

3.9. Comparison with the Conventional Optimization Models

This study evaluated the proposed SG-PSO-BO optimization framework against several stand-alone optimization models such as PSO, BO, GA, Grid Search, Random Search, and TLBO. The results demonstrate that the SG-PSO-BO consistently outperforms these benchmark models, achieving superior forecasting accuracy and computational efficiency.

Table 4. Comparison of the proposed framework with the conventional models.

Models	LSTM Units	Dense Units	Learning Rate	MAE	RMSE	R2	Execution Time (s)
IEEE-123 Test Network
SG-PSO-BO	55	28	0.001	0.21	0.38	0.97	145
PSO	60	40	0.001	3.25	5.67	0.92	163
BO	50	35	0.002	3.1	5.45	0.93	150
GA	45	32	0.004	3.2	5.5	0.92	160
Grid Search	40	30	0.005	3.4	5.8	0.91	180
Random Search	55	38	0.003	3.3	5.7	0.91	140
TLBO	58	39	0.003	3.15	5.48	0.92	130
Australian Distribution Network
SG-PSO-BO	55	30	0.001	0.16	0.31	0.98	147
PSO	60	40	0.001	3.25	5.67	0.92	163
BO	50	35	0.002	3.1	5.45	0.93	150
GA	45	32	0.004	3.2	5.5	0.92	160
Grid Search	40	30	0.005	3.4	5.8	0.91	180
Random Search	55	38	0.003	3.3	5.7	0.91	140
TLBO	58	39	0.003	3.15	5.48	0.92	130

demonstrates that the Australian distribution network achieved the lowest prediction errors, with MAE (0.16) and RMSE (0.31), along with the highest accuracy ($R^2$: 0.98). These results indicate superior generalization capability in real-world scenarios. The PSO and BO exhibited significantly higher error rates and lower forecasting accuracy, rendering them less suitable for SERNN optimization in forecasting HC accurately in real-world scenarios. GA, TLBO, and Random Search showed moderate improvements and lacked the necessary adaptability and precision for time-sensitive HC forecasting. The proposed SG-PSO-BO optimization framework, integrated with SERNN, effectively captures the dynamic response of DOE constraints, enhancing its ability to model temporal variations in network and DER variables. This capability makes the framework particularly robust for accurate HC forecasting in dynamic environments.

3.10. Comparison with the Hybrid Models

Various hybrid optimization techniques were compared with the SG-PSO-BO to optimize the SERNN model for HC forecasting in next-generation distribution networks under DOE conditionalities.

As shown in

Table 5. Comparison of the proposed framework with hybrid optimization models.

Hybrid Models	LSTM Units	Dense Units	Learning Rate	MAE	RMSE	R2	Execution Time (s)
IEEE-123 Test Network
SG-PSO-BO	55	28	0.001	0.21	0.38	0.97	145
PSO-BO	72	28	0.001	0.3	0.53	0.97	763
TLBO-PSO	81	20	0.001	0.26	0.37	0.98	3407
TLBO-BO	54	27	0.003	1.11	1.98	0.82	181
GA-BO	98	45	0.001	0.26	0.47	0.97	198
SERNN-BO	26	42	0.002	0.5	0.99	0.9	151
Australian Distribution Network
SG-PSO-BO	55	30	0.001	0.16	0.31	0.98	147
PSO-BO	72	28	0.001	0.3	0.53	0.94	763
TLBO-PSO	81	20	0.001	0.26	0.37	0.95	3401
TLBO-BO	54	27	0.003	1.11	1.98	0.82	171
GA-BO	98	45	0.001	0.26	0.47	0.93	185
SERNN-BO	26	42	0.002	0.5	0.99	0.94	153

, the SG-PSO-BO framework demonstrated superior performance compared to competing models across all performance metrics. Specifically, the SG-PSO-BO achieved the highest forecasting accuracy, with an $R^2$ value of 0.97 for the IEEE-123 test network and 0.98 for the real Australian distribution network. In addition, it attained the lowest errors and execution time for both networks, demonstrating reduced computational overhead. Its ability to dynamically adjust model parameters and enhance forecasting precision highlights its practical applicability in large-scale distribution networks with dynamic DER integration. The framework’s robustness and efficiency position it as a practical solution for real-world grid management applications.

3.11. Adaptive DER Power Import/Export Management

The proposed optimization framework integrates the SERNN forecasting model with SG-PSO-BO algorithm under DOE conditions to dynamically manage power import/export, BESS operation, and EV charging/discharging schedules. By maximizing PV utilization and enabling a forecast-driven decision-making process, the framework ensures efficient DER operations. Real-time BESS control and EV charging is achieved through DOE-based parameter tuning, leveraging HC forecasts to determine optimal charging and discharging.

The adaptive management of the DER power depends on multiple influencing factors such as PV output, BESS capacity, EV travel patterns, load demand, and grid constraints. To achieve an adaptive DER management framework, the SERNN forecasting layer, DOE-driven optimization layer, and real-time execution layer should be synchronized. The first layer provides HC availability and DER constraints, the second layer refines operational parameters of the network, PV, BESS, and EVs for optimal performance, and the third layer dynamically implements optimized DER schedules dynamically based on real-time HC and grid conditions.

Algorithm 2 illustrates the steps for adaptive DER management procedures. It comprises four stages: (i) HC forecasting, (ii) scheduling optimization, (iii) real-time execution, and (iv) performance evaluation. The process begins with forecasting HC using the SERNN model, DOE constraints, and DER parameters, including PV output, EVs, and BESS capacity. The forecasting model provides HC prediction for the next time-step $t+1$ to estimate the available HC of the network to integrate the DER. PV output, BESS capacity, charging/discharging rate, and EV trip distances and charging modes are used to select the schedules for optimum DER power utilization. Considering the HC, DER, and DOE constraints, the DER scheduling optimizes real-time power export/import decisions.

Algorithm 2 Real-Time DER scheduling with HC forecasting and optimization.

1: Input: Real-time PV Output, Load Demand, Historical DER Data 2: Scheduling Parameters: PV Levels, BESS Discharge Rates, EV Trip Distances, Charging Modes 3: Output: Optimized DER Power Export/Import and BESS/EV Charging/Discharging Schedules 4: Step 1: HC Forecasting using SERNN Model 5: Train SERNN using historical HC data, network constraints, and DER variables 6: Predict HC availability for the next time step 7: Step 2: DOE-Enabled Optimization using SG-PSO-BO 8: for each DOE setting (PV Levels, BESS rates, EV trip distances) do 9:     Apply SG-PSO-BO to optimize scheduling parameters 10: end for 11: Step 3: Real-Time DER Scheduling Execution 12: Initialize $BES{S}_{SOC}=20\%$, $E{V}_{SOC}=20\%$ 13: for each time-step t in real-time operation do 14:     Load real-time PV Output, HC forecast, and Load Demand 15:     if BESS SOC ≤ 20% and PV Output ≥ BESS Capacity then 16:         Start BESS charging until SOC reaches 100% 17:     end if 18:     if 5PM ≤ Time ≤ 23:00 and BESS SOC > 20% then 19:         Discharge BESS at DOE-specified rate to support grid demand 20:     end if 21:     if EV is in travel window $[07:00-09:00]$ or $[16:00-21:00]$ then 22:         Discharge EV based on DOE-specified trip distance 23:     end if 24:     if EV charging demand exists and PV ≥ EV requirement and $E{V}_{SOC}<90\%$ then 25:         Start EV charging with available PV power 26:     end if 27:     if DER Output ≥ BESS and EV requirement then 28:         Export excess DER power to the grid 29:     end if 30:     Update $BES{S}_{SOC}$ and $E{V}_{SOC}$ after each transaction 31: end for 32: Store results and visualize performances

The real-time DER scheduling algorithm dynamically manages the availability of PV output, peak demand support through BESS, and EV usage patterns. To maintain operational continuity, the minimum allowable State of Charge (SoC) for both the BESS and EV state is maintained at 20% of their respective capacities.

BESS initiates charging when $SoC\le 100\%$ and the PV output meets or exceeds the BESS charging requirement. Also, BESS supports peak demand by discharging power at a predefined rate until the SoC reaches 20% of the rated capacity, ensuring support for grid stability during peak demand. EV discharging is determined by the total distance traveled, with energy discharged at a predetermined rate based on EV specification. EV charging is activated when a charging request is detected, sufficient PV output is available, and the EV SoC falls below the specified threshold value. The DER power export decision is determined based on the availability of surplus energy after fulfilling BESS and EV charging requirements. Excess DER power is exported to the grid following HC constraints and DOE operational conditions.

Figure 10 illustrates the dynamic variations in the SoC of the BESS and EVs and the DER power export to the grid over a simulation period. The x-axis represents the time step (in hours), while the y-axis indicates the SoC for BESS and EV and the exported DER power to the grid. The figure demonstrates that the BESS follows a periodic charging and discharging cycle, charging when PV output is available, reaching 100% SoC at peak charging times. It discharges to support grid demand, reducing to 20% SoC, which is the lower operational threshold. The EV follows a distinct charging/discharging pattern based on PV power availability and scheduling constraints. EV discharging occurs in specific time slots corresponding to predefined driving schedules. EVs recharge when PV output is sufficient, following a stepped charging profile. DER exports power to the network when BESS and EV demands are met and excess power is available. Export patterns align with high PV generation periods, suggesting optimal utilization of surplus power. This dynamic interaction among BESS, EVs, and DER export highlights the effectiveness of real-time energy management and ensures efficient DER integration.

The adaptive DER management strategy optimizes real-time energy dispatch by scheduling PV output, BESS, and EVs based on HC forecasting. This approach ensures efficient energy storage operation by preventing overcharging/discharging and facilitates optimal export decisions under the DOE constraints. Integrating HC forecasting, DOE-driven optimization, and real-time execution layers improves the network responsiveness to dynamic conditions, making it a robust strategy for DER integration.

4. Discussion and Limitations

The SG-PSO-BO framework efficiently optimizes the SERNN forecasting model for robust, accurate, and adaptive HC forecasting of the next-generation distribution networks complying with the DOE constraints. This hybrid approach leverages PSO to determine the optimum weights and biases of the model, capturing the temporal dependency of network variables through its sensitivity gate architecture. Additionally, the BO fine-tunes hyperparameters to ensure reliability and accuracy for real-time HC forecasting.

Table 4 demonstrates the feature impact on SERNN model performance. The model incorporates network parameters like circuit power, voltage profile, and current passing through the conductors to observe operational conditionality while dynamically adjusting DER import/export limits based on the weather conditions and DOE requirements. Temporal dynamics, captured through the time of the day (‘Hour’) as the sensitivity gate input, alongside sine and cosine encoding, enhance the model’s capability to address cyclic behaviors of the network parameters, DER variables, and weather conditions.

The SG-PSO-BO-optimized SERNN achieves impressive HC forecasting performance, achieving $R^2$ values of 0.97 and 0.98, and significantly reduces MAE (0.21 and 0.16) and RMSE (0.38 and 0.31) for the IEEE-123 test network and the real Australian distribution network, respectively. These results demonstrate the superior capability of the proposed architecture to capture the non-linear and time-sensitive relationships between network and DER variables, as well as weather conditions. The proposed framework outperforms conventional standalone optimization models such as PSO, BO, and TLBO, and hybrid approaches, highlighting scalability, computational efficiency, and adaptability in real-time HC forecasting. The SERNN model’s sensitivity gate effectively addresses the time-sensitive impact of input features like PV output, temperature, and circuit power. The study shows that the Australian distribution network demonstrates superior performance due to actual parameters and real-world scenarios of the network operations. This emphasizes the real-world adaptation of the proposed model.

The proposed optimization framework is designed with modular flexibility, allowing the incorporation of additional physical constraints beyond voltage profile, equipment overloading, and thermal capacity of the conductors. Constraints such as power factor, phase angle, voltage imbalance thresholds, and battery operational restrictions can be integrated within the optimization process. This capability enhances the practicality of the approach, enabling customization to reflect specific network configurations and regulatory requirements.

In low voltage distribution networks, phase unbalances caused by DER locations and uncertain power injection, along with uneven load distribution, can influence the network parameter and HC limits. This study focused on the aggregated power flow analysis at the distribution transformer level, where the total active and reactive power flows are considered without explicitly modeling phase-specific imbalances. Economic aspects, including net metering, net billing, or dynamic pricing schemes, were not considered in this analysis. Other important technical requirements stipulated in distribution codes, such as power quality standards, fault ride-through capability, protection coordination, and anti-islanding regulations, are not explicitly modeled in this framework. These factors are essential for the practical implementation of DERs in real distribution networks. Future research will aim to incorporate these distribution code requirements to enhance the practical applicability and regulatory compliance of the proposed framework.

The proposed SG-PSO-BO framework is based on established methods, and its novelty lies in the synergistic integration tailored for real-time hosting capacity forecasting under DOE constraints in low voltage networks. Besides numerical accuracy improvements, the method also enhances forecasting consistency, dynamic adaptability, and network stability, addressing real-world operational needs. Even moderate improvements in error metrics can lead to significant reductions in unnecessary DER curtailment, better voltage compliance, and greater DER hosting without infrastructure upgrades. These advantages make the proposed framework a practical and scalable solution for distribution network operators facing increasing DER penetration.

5. Conclusions

This research introduces a sensitivity-aware hybrid optimization framework, integrating the SG-PSO and BO for robust, reliable, and accurate HC forecasting of the distribution networks with integrated DER. This approach optimizes the SERNN forecasting model, incorporating the DOE-constrained model to capture the interactions among the network, DER, and weather variables. The PSO determines the optimized weights and biases and fine-tunes the hyperparameters of the SERNN forecasting model. Thus, the model dynamically adjusts the DER power import/export to the network, ensuring adaptable DER integration and reliable grid operations. The model achieves high forecasting accuracy in terms of $R^2$ value of 0.97 for the IEEE-123 test network and 0.98 for the real Australian distribution network. It also demonstrates significantly low MAE (0.21 and 0.16) and RMSE (0.38 and 0.31) for the IEEE-123 test network and Australian distribution network, respectively. Additionally, it achieves stable convergence, low variation of residuals, and a lower error margin between the actual and predicted HC. The optimization framework is validated on real-world networks, demonstrating practical applicability through sustainable DER integration. This framework advances the dynamic network HC forecasting by addressing the limitations of existing models, providing a scalable, real-time solution for the next-generation distribution networks. The high forecasting accuracy and real-time adaptability validate the effectiveness of the proposed SG-PSO-BO hybrid optimization framework in enhancing SERNN model performance for dynamic HC forecasting. Validation on real-world networks highlights its practical applicability in facilitating sustainable DER integration and grid stability. This study will provide necessary network visibility to the network operators for real-time DER integration under DOE constraints.

Future research could include other DER types, such as wind power and other renewable resources, and explore pricing and tariff structures under diverse network scenarios to enhance the robustness and practical applicability of the proposed optimization framework. These advancements would offer grid operators a generalized forecasting tool to effectively manage next-generation distribution networks. However, the model modification may require the incorporation of high-resolution wind forecasting data, probabilistic power output modeling, and additional temporal–spatial parameters to capture the intermittency of wind. Furthermore, enhancement in the existing optimization algorithm may be needed to coordinate between wind and existing DERs to ensure system stability and optimal hosting capacity under diverse generation scenarios.