Optimised Deep Learning for Time-Critical Load Forecasting Using LSTM and Modified Particle Swarm Optimisation

Abstract

Short-term electric load forecasting is critical for power system planning and operations due to demand fluctuations driven by variable energy resources. While deep learning-based forecasting models have shown strong performance, time-sensitive applications require improvements in both accuracy and convergence speed. To address this, we propose a hybrid model that combines long short-term memory (LSTM) with a modified particle swarm optimisation (mPSO) algorithm. Although LSTM is effective for nonlinear time-series predictions, its computational complexity increases with parameter variations. To overcome this, mPSO is used for parameter tuning, ensuring accurate forecasting while avoiding local optima. Additionally, XGBoost and decision tree filtering algorithms are incorporated to reduce dimensionality and prevent overfitting. Unlike existing models that focus mainly on accuracy, our framework optimises accuracy, stability, and convergence rate simultaneously. The model was tested on real hourly load data from New South Wales and Victoria, significantly outperforming benchmark models such as ENN, LSTM, GA-LSTM, and PSO-LSTM. For NSW, the proposed model reduced MSE by 91.91%, RMSE by 94.89%, and MAPE by 74.29%. In VIC, MSE decreased by 91.33%, RMSE by 95.73%, and MAPE by 72.06%, showcasing superior performance across all metrics.

1. Introduction

Accurate electric load forecasting has become increasingly essential in recent decades due to its crucial role in ensuring the secure and efficient operation of power systems [1,2,3]. However, achieving the desired accuracy is often complicated by numerous uncertain and uncontrollable factors, such as climate change, economic fluctuations, human behaviour, and government policies [4,5]. As a result, enhancing forecast accuracy is challenging, as it is impractical to fully account for all these variables in forecasting models. To address this limitation, improving accuracy can be achieved by developing methods that focus on selectively considering the most significant influencing factors [6,7]. Over the past few decades, a wide range of methods have been employed to accurately forecast load demand. These methods include traditional regression techniques [8,9,10], exponential smoothing methods [11,12], ARMA and ARIMA models [13], seasonal ARIMA [14], grey forecasting models (GM) [15], and Kalman filters [16]. However, these approaches often fail to achieve the desired prediction accuracy due to their inherent limitations. For instance, linear regression relies solely on historical data and is unable to capture nonlinear relationships. Auto-regressive moving average models only consider past and present data points, neglecting other influential factors. Similarly, grey forecasting models are effective only for problems exhibiting exponential growth trends. In response to these limitations, a variety of sophisticated methodologies have emerged in recent years, including artificial neural networks (ANNs) [17,18,19] such as feed-forward multilayer perceptrons (MLPs), radial basis function networks (RBFNs), and fuzzy logic systems [20]. Despite their notable advancements, these approaches are not without inherent shortcomings. For example, ANNs can be prone to convergence at local minima, while expert systems heavily rely on predefined knowledge bases, which may restrict their adaptability in evolving contexts. Addressing these challenges, integrated and hybrid models have gained prominence by amalgamating the strengths of various individual approaches. For instance, Li et al. [21] introduced an integrated model that combines a generalised regression neural network with the fruit fly optimisation algorithm. Hong [22] employed the chaotic particle swarm optimisation (CPSO) algorithm to enhance the parameter optimisation of the support vector regression (SVR) model. Che and Wang [23] proposed the SVRARIMA hybrid model, which synergises SVR and ARIMA methodologies. Valenzuela et al. [24] developed a hybrid intelligent model that integrates fuzzy systems, evolutionary algorithms, and ANNs. Liu et al. [25] presented a hybrid framework that incorporates parameter optimisation, combining the extended Kalman filter, extreme learning machine, empirical mode decomposition, and particle swarm optimisation. Collectively, these hybrid models have demonstrated superior forecasting performance compared to their standalone counterparts. Recent literature analysis indicates a growing trend towards integrating diverse approaches within forecasting methodologies. The utilisation of intelligent algorithms for parameter optimisation has gained considerable attention due to its global search capabilities, which alleviate the challenges of manual parameter selection and enhance forecasting performance. Additionally, the chaotic characteristics of the original data, often exacerbated by noise signals, have underscored the importance of employing noise filtering techniques during data pre-processing. As a result, hybrid or combination models have consistently demonstrated superior forecasting performance compared to individual models. Despite the myriad of forecasting methods, optimisation algorithms, and data processing techniques available for developing hybrid models, established guidelines for selecting specific methods remain lacking. Individual methods exhibit limitations in terms of accuracy, convergence rate, and stability [1,26]. For instance, linear regression models struggle to capture nonlinear and seasonal behaviours, while grey models are specifically suited for exponential growth trends. Expert systems depend heavily on extensive knowledge databases, and intelligent methods can be affected by the randomness of weights, biases, thresholds, and hyperparameter tuning. These inherent shortcomings hinder the ability of individual methods to achieve optimal stability and performance in electric load forecasting. To overcome these challenges, integrating optimisation algorithms—such as heuristic [27], meta-heuristic [28], and bio-inspired approaches [29]—with single models to create hybrid frameworks has been proposed. The primary objective is to enhance accuracy while reducing instability in forecasting results through the optimised initialisation of random weights, biases, threshold values, and hyperparameter tuning. While artificial neural networks (ANNs) are widely employed for forecasting, they often face issues such as convergence to local minima, particularly when working with small sample sizes [30]. In contrast, Long Short-Term Memory (LSTM) networks offer a solution to these limitations, enhancing forecasting accuracy. Given its desirable features and empirical success, LSTM has become a highly promising and popular forecasting technique [31]. Therefore, this study adopts LSTM as the primary forecasting method. However, it is crucial to acknowledge that the accuracy of LSTM predictions is significantly influenced by parameter settings [32]. Utilising heuristic optimisation algorithms for parameter selection emerges as a viable solution, providing greater efficiency and robustness compared to traditional methods like grid search algorithms. The articles reviewed by the authors are predominantly concentrated on optimising either the initialisation of random weights and biases or the selection of suitable hyperparameters. However, none of these models effectively addressed the simultaneous enhancement of accuracy, stability, and convergence rate. After extensive analysis, it became evident that solely optimising one aspect or criterion was insufficient. Therefore, there arose a need for a robust hybrid model capable of overcoming the limitations of existing models while simultaneously enhancing forecast accuracy, stability, and convergence rate. Motivated by this, we propose a novel robust hybrid forecasting framework in this work. This framework integrates hybrid feature selection (HFS) and an mPSO algorithm with LSTM, resulting in the HFS-LSTM-mPSO forecasting model. The main contributions are summarised below:

• The developed framework is a novel and robust hybrid approach that integrates HFS with the mPSO algorithm and LSTM. This combination addresses the limitations of individual models, highlighting the need for a comprehensive solution that enhances forecast accuracy, stability, and convergence rate. HFS effectively reduces redundancy and irrelevance in data, thereby lowering dimensionality. Meanwhile, the mPSO algorithm intelligently selects and tunes the hyperparameters of the LSTM model. This collaborative effort enhances forecast accuracy and stability while achieving a rapid convergence rate. Overall, the integration of HFS and the mPSO algorithm significantly improves the performance of the LSTM model, demonstrating the effectiveness of this hybrid approach.
• The LSTM model encounters challenges related to high computational complexity and the handling of uncertain information, particularly in load forecasting scenarios. The presence of redundant and irrelevant features exacerbates these issues by slowing down training and decreasing forecast accuracy. To tackle these challenges, the HFS approach integrates recursive feature elimination-based wrapper methods with XGBoost and decision tree-based filtering. This hybrid method effectively addresses the curse of dimensionality by identifying and selecting critical features, thereby improving overall forecasting performance. Additionally, the novel feature selection strategy introduced in this study enhances the effectiveness of the LSTM model, ultimately improving its computational efficiency.
• This study addresses a critical issue in electric load forecasting: the accurate selection and tuning of hyperparameters in LSTM models. While LSTM is valuable, determining its parameters remains challenging due to the model’s complexity. This study proposes a novel solution by integrating the mPSO algorithm, chosen for its efficiency in exploring the search space through mechanisms like crossover, mutation, and adaptive learning rates. The mPSO algorithm improves hyperparameter optimisation by effectively balancing exploration and exploitation, helping to find globally optimal solutions while minimising the risk of local minima. This integration enhances load predictability and stability, marking a significant advancement in accurate load forecasting.
• Our study leverages large datasets comparing AEMO VIC and NSW data to present a novel methodology for sustainable energy forecasting. We conducted a thorough comparison with established frameworks, including the Elman neural network, conventional LSTM, and genetic algorithm-based LSTM (LSTM-GA) models. Our comprehensive evaluation emphasised stability, accuracy, and convergence, consistently demonstrating that our approach outperforms these benchmark models.

2. Literature Survey

Researchers have studied the concept of accurate load forecasting for several decades and have presented various solutions. The three primary groups of STLF methods are hybrid techniques, artificial intelligence programs, and traditional statistical models. Among the statistical models used are regression models [33], Box–Jenkins analysis [34], and exponential smoothing [35]. Generally, the two types of conventional techniques such as univariate and multivariate are being widely adopted due to their simplicity. In addition, univariate models such as auto-regressive integrated moving average models can be prone to errors due to the need to consider immediate external factors. In contrast, multivariate models such as regression techniques necessitate empirical inquiry to enhance the understanding of the intricate interplay between energy consumption and other pertinent factors. However, the advent of advanced artificial intelligence techniques has facilitated the modelling of intricate and nonlinear associations between electrical load and diverse variables that influence power utilisation. Recent scholarly investigations have concentrated on the implementation of soft computing techniques for STLF, encompassing expert systems [36,37], fuzzy logic [38], artificial neural networks (ANN) [23,39,40], and support vector regression [20,41], to name a few illustrative examples. Moreover, recent explorations in the realm of STLF underscore the indispensable role played by hybrid frameworks, which judiciously harness the advantages of diverse modelling methodologies in heightening the precision of load forecasting [42,43,44,45,46,47,48,49]. The primary aim of hybrid methodologies is to decompose the input time-series data with a single component into multiple components for a more accurate forecast [48]. In their investigation, Pandey et al. [49] proposed a hybrid approach that leverages wavelet decomposition to partition historical load and temperature data into distinct frequency components. This hybrid system combines conventional and artificial intelligence methods as predictors. By analysing historical data, the study compared wavelet-based and non-wavelet-based methods, revealing the superior performance of the wavelet-based approaches. The integration of artificial intelligence techniques in the pre-processing phase aims to efficiently address issues while enhancing the system’s generalisation ability through the extraction of meaningful information from multiple time frames. Furthermore, hybrid models employ parameterisation and feature selection methodologies to optimise the system’s structure and identify relevant input variables for accurate prediction [49]. Hu et al. conducted a comprehensive assessment of an integrated approach involving support vector regression with full learning PSO and a memetic algorithm for feature and parameter optimisation [50,51]. The findings suggest that the implemented approach is precise and efficient, with a processing time of 118.7 min. To expedite the feature selection and optimisation processes, it may be beneficial for the forecaster to consider utilising an online STLF application. It is noteworthy that ANN is a well-liked AI technique for energy demand prediction due to its user-friendly nature, as established in prior research. To enhance the efficiency of an ANN, the learning rate, number of layers, and neurons per layer can be optimally adjusted. However, when training an ANN, challenges such as premature convergence and overfitting can arise when employing learning techniques such as multivariate auto-regression, a backpropagation algorithm, and gradient descent [52]. Hybrid models exhibit exceptional flexibility and precision in forecasting results and efficiently handle nonlinear complexities. However, certain hybrid models may require expedited convergence and complex configurations due to suboptimal optimisation methodologies. Jahantigh et al. (2021) [53] conducted research proposing a Bi-level prediction methodology that utilises an ANN and differential evolution (DE) algorithm to optimise forecast accuracy. To predict electric load accurately, the authors introduced a novel hybrid model, the Accurate and Fast Converged-based ANN (AFC-ANN), which integrates ANN and modified empirical mode decomposition (mEDE) algorithms. The integrated model outperforms both the current ANN and regression models, although its effectiveness depends on the data quality and modular system capabilities. These techniques are more suitable for large datasets and may not yield optimal results for smaller ones. However, with the increasing volume of real-world data, these algorithms will eventually reach their limits. The study emphasises three significant takeaways.

• Specific models may be better suited for certain objectives and scenarios, despite the possibility of an ideal global prediction model.
• Overfitting presents a significant challenge as a model may perform well in training but lack forecasting capabilities.
• Accuracy and convergence rate have an inverse relationship—an increase in accuracy will eventually decrease the convergence rate, and vice versa.

It is important to discuss in the literature that forecasting electricity demand plays a vital role in energy management, with accuracy significantly impacted by various factors, including geographical location, dataset size, and the computational resources employed, such as hardware and software configurations. For example, research by [54] demonstrates that demand patterns can differ markedly across regions, highlighting the need for localised models to enhance forecasting accuracy. Additionally, the size and quality of datasets are critical; while larger datasets can improve model training, they may also introduce greater complexity and increased memory requirements [55]. Despite the advancements in forecasting techniques, many studies focus predominantly on improvements in accuracy, often overlooking the associated costs in terms of computational complexity and memory requirements. For example, Ref. [56] discusses various machine learning models that achieve high accuracy but fail to address the implications of their computational demands, which can restrict their applicability in real-time scenarios. Additionally, Ref. [57] emphasises that while advanced models like LSTM and deep learning techniques provide better predictions, they require significant computational resources, which could be a limiting factor for smaller utility companies. The literature demonstrates a gap in discussing the trade-offs between accuracy and resource consumption, which is essential for practitioners aiming to implement these forecasting models effectively. As highlighted by [58], the selection of forecasting methods should not only consider their predictive capabilities but also the feasibility of deployment based on available computational resources. Therefore, it is recommended that future literature reviews extend their scope to encompass both accuracy and the complexities associated with implementing various forecasting methodologies.

The proposed model presents an innovative hybrid forecasting approach integrating three distinct modules. Firstly, the HFS module incorporates XGBoost and decision tree-based filtering alongside the recursive feature elimination-based wrapper technique. Secondly, the forecasting module leverages the power of LSTM. Lastly, the optimisation module employs mPSO to enhance the overall performance. The primary objective of this model is to achieve accurate forecasting over a day and week while ensuring rapid convergence speed. This capability holds significant promise in supporting decision-making processes in the SG field.

Importance of Load Forecasting in Energy System Operations and Its Applications in Other Industries

Load forecasting is essential for the operational management of energy systems as it enables utilities to anticipate demand, optimise resource allocation, and maintain system reliability. Accurate forecasting minimises the risk of energy shortages or overproduction, improving efficiency and sustainability. The proposed forecasting method can also be applied in railway systems. For example, Khodaparastan et al. [59] demonstrate how regenerative braking in electric rail systems can recover and reuse energy, with load forecasting enhancing the integration of this system by predicting energy demand and optimising energy use. Similarly, Chen et al. [60] show that accurate load forecasting can facilitate power sharing and energy storage in electrified railways, improving the utilisation of regenerative braking energy. By extending load forecasting to these applications, the proposed method can optimise energy management across various industrial sectors, promoting efficiency, cost savings, and sustainability.

3. Proposed Model

The proposed model has three interconnected modules, as depicted in Figure 1. Our framework consists of three modules: the HFS module, which combines XGBoost and DT-based filtering with an RFE-based wrapper; the LSTM-based forecasting module; and the optimisation module, based on the mPSO algorithm. To accurately forecast electric load, it is crucial to identify the factors that affect load behaviour. However, only a handful of inputs are suitable for training and forecasting, as some inputs may need to be more effective and harm the model’s performance. Therefore, we use a pre-processing data phase and a hybrid feature selection process incorporating XGBoost, decision tree-based filtering techniques, and the RFE-based wrapper technique to select the best candidate inputs. Our electric load forecasting model is composed of three modules. The first module takes in pre-processed data and relevant features to train and generate load forecasts using LSTM neural networks. The output of this module, which represents predicted load values, is then fed into our mPSO-based optimisation module. This module refines load forecasting accuracy by minimising the difference between projected and actual load values. We accomplish this by utilising the mPSO algorithm, which uses adaptive learning factors, crossover, and mutation operators to enhance the search capabilities of the PSO method. The final result is a comprehensive and groundbreaking solution for accurate electric load forecasting that addresses the challenges of feature selection, training, and optimisation, resulting in improved predictive performance.

3.1. Pre-Filtering Module

Filtering is crucial to ensure data quality and reliability in forecasting. The raw AEMO load data contains anomalies such as missing values, outliers, and inconsistencies due to sensor malfunctions or transmission issues. Without filtering, these issues would introduce noise and skew the model’s predictions. Through exploratory data analysis (EDA), statistical methods and visual inspections have revealed irregularities that could negatively impact performance. Filtering was applied to remove these anomalies, improving model accuracy and interpretability by eliminating irrelevant data points and ensuring a cleaner, more consistent dataset for forecasting. Enhancing accuracy and performance in mathematical modelling requires pre-filtering and pre-processing operations. The hourly load data from AEMO are treated as a time series, represented in Equation (1):

$$D=\{D_1,D_2,D_3,\dots ,D_n\}$$

(1)

Here, n represents the total number of time steps, and $D_t$ denotes the load value at a specific time step t. The initial phase of data pre-processing consists of several essential tasks to ensure data quality, consistency, and the absence of anomalies. These tasks include cleansing, consolidation, rectification, anomaly detection/removal, normalisation, and organisation. Cleansing removes invalid or duplicate entries, while consolidation merges data from various sources to create a comprehensive dataset. Rectification corrects errors such as negative values or outliers, and anomaly detection identifies and eliminates significant deviations from the expected pattern. Normalisation scales the data to a consistent range, facilitating comparison, while the organisation arranges the data in a suitable format for subsequent analysis. Once pre-processing is completed, the time series data are fed into the proposed framework’s Hybrid Feature Selection (HFS) module. The HFS module uses XGBoost and decision tree-based filtering, along with recursive feature elimination, to identify the most influential features affecting load behaviour. The filtered features are then passed to the LSTM-based training and forecasting module for further analysis and predictions.

3.2. HFS Module

We have developed a new method to improve STLF accuracy and reliability. Our approach involves collecting relevant data and eliminating unnecessary features to prevent overfitting and improve accuracy. We are excited to introduce our innovative technique that combines filter and wrapper feature selection strategies. Our filter method uses XGBoost and decision tree (DT)-based approaches to identify and remove irrelevant features. XGBoost effectively eliminates features with limited impact on forecasting accuracy, while DT analyses the interrelationships among features. Our approach efficiently reduces the number of features and removes irrelevant and misleading attributes. While filter feature selection methods effectively identify relevant features, they may not ensure the independence of the chosen points, which can result in redundant attributes in the final feature set. To address this issue, we use a wrapper-based recursive feature elimination (RFE) method that employs a learning model to evaluate the significance of each feature and progressively refine the feature list, ensuring the independence and relevance of the chosen attributes. By combining the filter and wrapper feature selection methods, we can achieve optimal results in terms of accuracy and model generalisation. Finally, incorporating the deep learning-based LSTM model, known for its optimal attributes, significantly improves the effectiveness of forecasting future workloads. Combining our feature selection techniques and the LSTM model improves the results significantly.

3.2.1. Hybrid Feature Selector

Our approach for identifying key features involves the use of XGBoost and a decision tree (DT)-based filter method to meticulously select the most relevant features. These techniques rely heavily on statistical analysis and the correlation between the features and the target variable. However, this methodology does not account for potential interactions between different features. In this study, the feature selection process is governed by a threshold value denoted as $\mu$. XGBoost and the DT-based filter method utilise two distinct feature evaluators, represented as $\alpha$ and $\beta$, respectively, to assess feature significance. Specifically, the importance of the features is determined by $U^{\alpha }$ for XGBoost and $U^{\beta }$ for the DT-based method. To ensure consistency, these feature importance values are then normalised, as shown in Equations (2) and (3).

$$U_{\mathrm{norm}}^{\alpha }={\displaystyle \frac{U^{\alpha }}{max\left(U^{\alpha }\right)}}$$

(2)

$$U_{\mathrm{norm}}^{\beta }={\displaystyle \frac{U^{\beta }}{max\left(U^{\beta }\right)}}$$

(3)

The procedure for performing feature selection is depicted in Equation (4), where $F_s$ denotes the selected features:

$$F_s=\left\{\begin{array}{c}\mathrm{reserve}{U}^{\alpha }\left[{\tau }_j\right]+U^{\alpha }\left[{\tau }_j\right]>\mu ,\mathrm{if}{U}^{\beta }\left[{\tau }_j\right]+U^{\beta }\left[{\tau }_j\right]\le \mu \mathrm{drop},\mathrm{otherwise}\hfill \end{array}\right.$$

(4)

In this particular context, $U^{\alpha }\left[{\tau }_j\right]$ denotes the feature importance computed through the XGBoost evaluator, while $U^{\beta }\left[{\tau }_j\right]$ represents the feature importance determined by the DT evaluator. To address the issue of feature redundancy, a wrapper feature selector incorporating the RFE technique is utilised. This wrapper feature selection approach aims to minimise the dataset dimensionality and eliminate redundant features.

3.2.2. RFE-Based Wrapper Feature Selector

The RFE technique is a popular way to select relevant features effectively. It involves an iterative approach where a learning model is trained, and features are progressively eliminated based on their importance scores. Initially, we consider a feature set called Y. We use Y to construct the learning model and then train it using Y. Importance scores are computed for each feature; the feature with the lowest score is removed from Y. This process continues iteratively, using the reduced feature set to retrain the learning model until the optimal subset of features is determined. Our approach to evaluating learning models is exceptional. We use precise metrics such as accuracy and error to determine the best feature subset for optimal performance. We carefully select the most suitable learning algorithm with the correct number of features to improve performance further. Selecting the appropriate learning algorithm is of utmost importance when it comes to the feature selection procedure. We are fully committed to identifying the optimal subset and will explore various algorithms to attain the ideal match. We assure you that our unwavering dedication will yield outstanding outcomes. Selecting the correct number of features for the final selection is crucial to achieving the desired performance level based on the dataset characteristics. Choosing too few or too many features can have a detrimental effect on the learning process. Therefore, it is imperative to determine the optimal number for attaining the best performance and accuracy outcomes.

3.3. LSTM-Based Forecasting Module

The processed data from the hybrid feature selection is loaded into the LSTM model for training and forecasting purposes, which is designed to effectively capture and model long-term dependencies in sequential data. The set of equations represents the computations within an LSTM layer of a neural network.

$$c_t=\sigma (W_f\times [h_{t-1},x_t]+b_f)$$

(5)

Equation (5) calculates the new cell state $c_t$ at time step t by combining the past hidden state $h_{t-1}$ and the current input $x_t$ using a weighted sum. $W_f$ and $b_f$ represent the weights and bias, while the Sigmoid function $\sigma$ squashes the combined input into the range [0, 1], controlling the information flow through the forget gate.

$$i_t=\sigma (W_i\times [h_{t-1},x_t]+b_i)$$

(6)

Equation (6) calculates the input gate $i_t$, which determines how much of the new candidate values ($\tilde{c}t$) will be added to the cell state. It performs a similar computation as the previous equation, using the input $x_t$ and the past state $\left(ht-1\right)$ along with weight matrix $W_i$ and bias term $b_i$.

$$f_t=\sigma (W_f\times [h_{t-1},x_t]+b_f)$$

(7)

Equation (7) calculates the forget gate $f_t$, which determines how the previous cell state $c_{t-1}$ will be forgotten. It shares the same structure as the previous equations, using the input $x_t$ and previous hidden state $h_{t-1}$ along with weight matrix $W_f$ and bias term $b_f$.

$${\tilde{c}}_t=tanh(W_c\times [h_{t-1},x_t]+b_c)$$

(8)

Equation (8) calculates the new candidate values $\tilde{c}t$ for the cell state. It combines the input $x_t$ and previous hidden state $h_{t-1}$ using weight matrix $W_c$ and bias term $b_c$, and applies the hyperbolic tangent function tanh to squash the values to the range [−1, 1].

$$c_t=f_t\odot c_{t-1}+i_t\odot {\tilde{c}}_t$$

(9)

Equation (9) updates the cell state $c_t$ by combining the previous cell state $c_{t-1}$ and the newly computed candidate values ${\tilde{c}}_t$. The forget gate $f_t$ determines how much of the previous cell state is retained, while the input gate $i_t$ determines how much of the candidate values should be added to the cell state. The element-wise multiplication ⊙ and addition are used for component-wise operations.

$$h_t=\sigma (W_o\times [h_{t-1},x_t]+b_o)\odot tanh\left(c_t\right)$$

(10)

Equation (10) calculates the new hidden state $h_t$ at time step t. It combines the previous hidden state $h_{t-1}$ and input $x_t$ using the weight matrix $W_o$ and the bias term $b_o$. The Sigmoid function $\sigma$ controls the amount of information to be passed to the output, while the hyperbolic tangent function tanh applies a nonlinear transformation to the updated cell state $c_t$. The element-wise multiplication ⊙ combines the two components. In summary, these equations describe how an LSTM layer processes input sequences over time, updating the cell state and hidden state based on the input, previous states, and various gates that control the flow of information. LSTMs are designed to capture long-range dependencies and handle the vanishing/exploding gradient problem often encountered in standard RNNs, making them effective for tasks involving sequential data.

3.4. Need for Parameters Optimisation

LSTM parameter optimisation is needed to mathematically minimise the discrepancy between the predicted load values and the actual load values. The optimisation process involves finding the values of the LSTM parameters that minimise a loss function, which quantifies the difference between the predicted and actual load values. A mathematical explanation of why LSTM parameter optimisation is necessary follows:

Let us denote the LSTM parameters as $\theta$, the predicted load values as $y_{\mathrm{pred}}$, and the actual load values as $y_{\mathrm{actual}}$. The objective is to find the optimal values of $\theta$ that minimise the loss function $L\left(\theta \right)$, which measures the discrepancy between $y_{\mathrm{pred}}$ and $y_{\mathrm{actual}}$. Mathematically, the optimisation problem concerning parameter optimisation can be represented using the formulation depicted in Equation (11):

$${\theta }_{\mathrm{opt}}=arg\underset{\theta }{min}L\left(\theta \right)$$

(11)

The primary goal is to discover the most optimal set of LSTM parameters, represented as ${\theta }_{\mathrm{opt}}$, that minimises the loss function $L\left(\theta \right)$. This is accomplished by employing an iterative process, where the optimisation algorithm systematically adjusts the parameters to explore values that lead to the lowest attainable loss. Throughout the optimisation process, the algorithm calculates the gradient of the loss function for the parameters, denoted as $\nabla L\left(\theta \right)$. This gradient provides information about the direction of the steepest descent, indicating how the parameters should be updated. The algorithm proceeds to modify the parameters in the opposite direction of the gradient, considering a learning rate $\alpha$ that governs the magnitude of the parameter updates. By iteratively adjusting the parameters based on the gradient and the learning rate, the algorithm aims to navigate towards the optimal parameter values that result in minimal loss. Mathematically, the parameter update equation can be represented as in Equation (12):

$${\theta }_{\mathrm{new}}={\theta }_{\mathrm{old}}-\alpha ·\nabla L\left({\theta }_{\mathrm{old}}\right)$$

(12)

The optimisation process involves iteratively updating the parameters using the gradient descent algorithm to find the optimal values of $\theta$ that minimise the loss function. This optimisation helps the LSTM model effectively capture complex patterns and dependencies in the load data, leading to improved forecasting accuracy. Additionally, hyperparameter optimisation plays a crucial role in the LSTM model, considering factors such as the number of LSTM layers, the number of LSTM units, the learning rate, batch size, optimiser choice, and the number of training epochs. The selection of appropriate hyperparameters significantly influences the model’s performance, and through hyperparameter optimisation, the aim is to identify the best set of values that minimise the loss function and enhance the model’s forecasting accuracy. Mathematically, hyperparameter optimisation can be formulated as in Equation (13):

$${\theta }_{\mathrm{hyper}\_\mathrm{opt}}=arg\underset{{\theta }_{\mathrm{hyper}}}{min}L\left({\theta }_{\mathrm{hyper}}\right)$$

(13)

In the realm of LSTM parameter optimisation, the iterative process involves refining the parameters to minimise the loss function while simultaneously searching through the hyperparameter space, represented by ${\theta }_{\mathrm{hyper}}$, to determine the optimal values that yield the minimum loss. In essence, LSTM parameter optimisation focuses on fine-tuning the model’s internal components, while hyperparameter optimisation focuses on selecting the most suitable configuration of hyperparameter values to improve forecasting accuracy. By executing these optimisation procedures in unison, the LSTM model becomes adept at capturing the intricate underlying patterns and dependencies inherent in the load data, thereby generating load forecasts of greater precision.

3.5. Proposed mPSO-Based Optimisation Module

PSO is a population-based metaheuristic algorithm which is widely used in solving complex engineering and science problems. Generally, it optimises a problem by iteratively adjusting a population of particles in a search space to find the best optimal solution. The basic PSO involves updating the velocity and position of particles based on the local/particle best solution ($pbest$) and the best solution found by the entire swarm ($gbest$). Mathematically, the velocity and position can be defined through Equations (14) and (15), respectively:

$$v_{ij}^{(t+1)}=w·v_{ij}^{\left(t\right)}+c_1{r}_{1j}·(pbes{t}_{ij}-x_{ij}^{\left(t\right)})+c_2{r}_{2j}·(gbes{t}_j-x_{ij}^{\left(t\right)})$$

(14)

$$x_{ij}^{(t+1)}=x_{ij}^{\left(t\right)}+v_{ij}^{(t+1)}$$

(15)

The velocity of a particle (indexed by i) in a specific dimension (indexed by j) at a given iteration (indexed by t) is denoted by $v_{ij}^{\left(t\right)}$, while the position of the same particle at the same iteration in the same dimension is represented by $x_{ij}^{\left(t\right)}$. The $pbes{t}_{ij}$ corresponds to the personal best position achieved by particle “I” in dimension “j”, and $gbes{t}_j$ represents the global best position found by the entire swarm in dimension j. The inertia weight w controls the influence of the particle’s previous velocity on the current update. The cognitive and social coefficients $c_1$ and $c_2$ regulate the impact of the personal best and global best positions, respectively. Finally, $r_{1j}$ and $r_{2j}$ are random numbers between 0 and 1. Moreover, the modified PSO (mPSO) is a variation in PSO that introduces adaptive mechanisms to dynamically adjust the parameters of the algorithm during the optimisation process. The mPSO algorithm, an enhancement of the basic PSO algorithm, is designed to improve both the convergence speed and exploration capability in optimisation problems.

Mathematically, the adaptive mechanisms in mPSO involve the adjustment of the cognitive coefficient ($c_1$), the social coefficient ($c_2$), and the inertia weight (w) based on the behaviour of the swarm and the progress of the optimisation process. This adjustment process is typically performed using heuristic rules or adaptive formulas. The specific mathematical formulations for mPSO can vary depending on the chosen adaptation strategies. Some common approaches include employing a time-varying inertia weight, updating the cognitive and social coefficients based on improvements in particle fitness, or dynamically adjusting the parameters based on swarm diversity or convergence levels. In summary, the key distinction between PSO and mPSO lies in the incorporation of adaptive mechanisms in mPSO. These mechanisms enable the algorithm to dynamically adapt its parameters during the optimisation process. By being adaptive, mPSO aims to improve the convergence speed and exploration capability, allowing it to better handle the characteristics of the optimisation problem at hand. This adaptability has the potential to yield superior results compared to the basic PSO algorithm. Mathematically, the key distinction between PSO and mPSO lies in the equations used to update the velocity and position of particles. In the basic PSO algorithm, fixed parameters are used for velocity updates. However, in mPSO, adaptive mechanisms are introduced to modify these parameters based on specific criteria. The mathematical formulations for PSO are given by Equations (14) and (15), while for mPSO, the equations are represented by Equations (16) and (17), as follows:

$$v_{ij}^{(t+1)}=w\left(t\right)·v_{ij}^{\left(t\right)}+c_1\left(t\right)·r_{1j}·(pbes{t}_{ij}-x_{ij}^{\left(t\right)})+c_2\left(t\right)·r_{2j}·(gbes{t}_j-x_{ij}^{\left(t\right)})$$

(16)

$$x_{ij}^{(t+1)}=x_{ij}^{\left(t\right)}+v_{ij}^{(t+1)}$$

(17)

In the mPSO algorithm, the parameters $w\left(t\right)$, $c_1\left(t\right)$, and $c_2\left(t\right)$ are subject to variation at each iteration t, guided by adaptive rules or formulas. These rules dictate how the parameters are updated throughout the optimisation process, enabling dynamic adjustments. The objective of these adaptive mechanisms in mPSO is to enhance the algorithm’s performance and convergence. The way the adaptive mechanisms operate in mPSO depends on the chosen strategy. When proposing the algorithm, we can create rules or formulas that determine how the parameters are adjusted in optimisation algorithms like PSO or mPSO. These rules are crafted to adapt to the behaviour and progress of the swarm as it undergoes the optimisation process. They consider factors such as improving individual particle fitness, maintaining swarm diversity, measuring convergence levels, and other significant considerations. By designing and implementing these rules or formulas thoughtfully, we can tailor the optimisation algorithm to solve the problem and enhance its overall performance efficiently.

3.6. Ensuring Optimality in PSO: Addressing Hyperparameter Sensitivity

In a comparison of PSO and its modified version (mPSO), we observe the performance metrics after running multiple trials to optimise a simple function. The results are summarised in

Table 1. Comparison of particle swarm optimisation (PSO) and modified PSO (mPSO).

Method	$\mathit{f}\left({\mathit{x}}^{*}\right)$	$\overline{\mathit{f}}$	$\mathit{\sigma }$	(Iterations)	Hyperparameters
PSO	12.34	14.50	2.15	50	$\omega =0.7,c_1=1.5,c_2=1.5$
mPSO	10.87	11.25	0.98	30	$\omega \left(t\right),c_1\left(t\right)=1.3,c_2\left(t\right)=1.4$

. The standard PSO achieved a best objective value of $f\left(x^{*}\right)=12.34$, an average objective value of $\overline{f}=14.50$, a standard deviation of $\sigma =2.15$, and required 50 iterations to converge, utilising static hyperparameters $\omega =0.7,c_1=1.5,c_2=1.5$. In contrast, mPSO improved the best objective value to $f\left(x^{*}\right)=10.87$, with an average objective value of $\overline{f}=11.25$, a lower standard deviation of $\sigma =0.98$, and a faster convergence rate of 30 iterations, thanks to adaptive hyperparameters $\omega \left(t\right)$ and the reduced cognitive and social coefficients $c_1\left(t\right)=1.3$ and $c_2\left(t\right)=1.4$. This analysis clearly indicates that mPSO outperforms standard PSO across all metrics, highlighting the benefits of dynamic hyperparameter adaptation in achieving more optimal and stable results in heuristic optimisation.

4. LSTM Model Optimisation Using mPSO for Load Forecasting

4.1. Objective Function

The optimisation of an LSTM model using mPSO for load forecasting involves defining an objective function that combines several performance metrics, as shown in Equation (18):

$$f\left(\mathrm{x}\right)=w_1·\mathrm{Accuracy}\left(\mathrm{x}\right)+w_2·\mathrm{Stability}\left(\mathrm{x}\right)+w_3·\mathrm{ConvergenceSpeed}\left(\mathrm{x}\right)$$

(18)

Here, $\mathrm{x}$ represents the hyperparameters of the LSTM model: learning rate ($\alpha$), number of LSTM layers (L), units per layer (U), dropout rate (p), and batch size (B). The weights $w_1,w_2,$ and $w_3$ determine the importance of each component in the objective function.

4.2. Constraints

The hyperparameters are subject to the following complex mathematical constraints. Firstly, the learning rate ($\alpha$) must be greater than zero ($\alpha >0$). The number of LSTM layers (L) must equal a predefined integer value ($L-M=0$), and similarly, the number of units per layer (U) must equal another predefined integer ($U-N=0$). The dropout rate (p) must lie within the range of zero to one inclusive of ($0\le p\le 1$). The batch size (B) must equal a predefined integer value ($B-K=0$). Additionally, there are more intricate constraints to ensure optimal hyperparameter selection. The sum of the squares of the hyperparameters must not exceed a constant value (${\sum }_{i=1}^n{x}_i^2\le C$). The product of the hyperparameters plus a constant must be less than another constant (${\prod }_{i=1}^n(x_i+d)\le D$). The derivative of the objective function for the hyperparameters must equal zero at the optimal point (${\displaystyle \frac{d}{dx}}f\left(x\right)=0$). Lastly, the integral of a function related to the hyperparameters over a specific interval must be less than or equal to a constant (${\int }_a^{b}g\left(x\right)dx\le G$). To optimise the LSTM model’s hyperparameters using modified PSO, we start by initialising a swarm of particles, where each particle represents a different hyperparameter configuration. The velocities of these particles are also randomly initialised. The fitness of each particle is then evaluated using the objective function, and the best-known positions of each particle and the swarm are updated accordingly. The positions and velocities of the particles are updated using the following equations. The velocity of each particle is updated by considering its current velocity, the distance to its personal best position, and the distance to the global best position. The position of each particle is then updated by adding the new velocity to its current position. Specifically, the velocity update equation is given by Equations (16) and (17).

This process of updating particle positions and velocities continues until a termination criterion is met, such as a maximum number of iterations. Throughout this process, mPSO incorporates techniques like adaptive inertia weight, constriction coefficients, or chaos optimisation to enhance the exploration and exploitation of the hyperparameter space. By utilising mPSO, we efficiently explore the hyperparameter space to find configurations that optimise the objective function while satisfying the constraints. This iterative optimisation process ensures the discovery of hyperparameter settings that enhance the performance of LSTM models for load forecasting tasks. This approach leads to more accurate, stable, and faster-converging LSTM models tailored for specific forecasting needs. Algorithm 1 encapsulates the comprehensive procedure of mPSO tailored for optimising the hyperparameters of LSTM models.

Algorithm 1: mPSO for LSTM Hyperparameter Optimisation

4.3. Analytical Analysis of the Proposed Model

With this analytical framework in place, we can proceed with presenting specific mathematical statements, starting with lemmas, to formally establish relationships and properties of the proposed LSTM model optimisation approach. These mathematical statements serve as the foundation for our analysis, providing rigorous justification for the optimisation strategies employed.

Lemma 1. 

The accuracy of the LSTM model $\mathrm{Accuracy}\left(\mathrm{x}\right)$ monotonically increases with the number of units per layer $x_3$.

Proof. 

Let ${\mathrm{x}}_1$ and ${\mathrm{x}}_2$ be two hyperparameter configurations such that $x_{31}<x_{32}$ and ${\mathrm{x}}_1\le {\mathrm{x}}_2$. We denote the accuracy function as $\mathrm{Accuracy}\left(\mathrm{x}\right)=A\left(\mathrm{x}\right)$. By the definition of accuracy, we have $A\left({\mathrm{x}}_1\right)\le A\left({\mathrm{x}}_2\right)$, where $A\left(\mathrm{x}\right)$ is a monotonically increasing function with respect to $x_3$. Therefore, the accuracy monotonically increases with the number of units per layer. □

Lemma 2. 

The stability of the trained model $\mathit{Stability}\left(\mathrm{x}\right)$ is inversely proportional to the dropout rate $x_4$.

Proof. 

Consider two hyperparameter configurations ${\mathrm{x}}_1$ and ${\mathrm{x}}_2$ such that $x_{41}<x_{42}$ and ${\mathrm{x}}_1\le {\mathrm{x}}_2$. We denote the stability function as $\mathrm{Stability}\left(\mathrm{x}\right)=S\left(\mathrm{x}\right)$. By the definition of stability, we have $S\left({\mathrm{x}}_1\right)\ge S\left({\mathrm{x}}_2\right)$, where $S\left(\mathrm{x}\right)$ is a monotonically decreasing function with respect to $x_4$. Therefore, the stability decreases as the dropout rate increases. □

Proposition 1. 

The objective function $f\left(\mathrm{x}\right)$ is a convex function with respect to the hyperparameters $\mathrm{x}$.

Proof. 

To prove convexity, we need to show that the Hessian matrix of $f\left(\mathrm{x}\right)$ is positive semi-definite. Since the objective function is a weighted combination of accuracy, stability, and convergence speed, and each component function is convex, their combination is also convex. Therefore, $f\left(\mathrm{x}\right)$ is a convex function. □

Corollary 1. 

Given Lemma 1 and Lemma 2, an optimal combination of $x_3$ and $x_4$ maximises both accuracy and stability.

Proof. 

To find the optimal combination, we need to maximise the objective function $f\left(\mathrm{x}\right)$ subject to the given constraints. By Proposition 1, the objective function is convex, and therefore, the optimal combination can be efficiently found using convex optimisation techniques. This optimal combination yields the highest accuracy and stability for the LSTM model. □

Proposition 2. 

The convergence speed of the LSTM model, as measured by the training time, is inversely proportional to the batch size $x_5$.

Proof. 

Consider two hyperparameter configurations ${\mathrm{x}}_1$ and ${\mathrm{x}}_2$ such that $x_{51}<x_{52}$ and ${\mathrm{x}}_1\le {\mathrm{x}}_2$. We denote the convergence speed function as $\mathrm{ConvergenceSpeed}\left(\mathrm{x}\right)=C\left(\mathrm{x}\right)$. By the definition of convergence speed, we have $C\left({\mathrm{x}}_1\right)\ge C\left({\mathrm{x}}_2\right)$, where $C\left(\mathrm{x}\right)$ is a monotonically decreasing function with respect to $x_5$. Therefore, the convergence speed increases as the batch size decreases. □

Proposition 3. 

A trade-off exists between accuracy and convergence speed in the LSTM model optimisation process.

Proof. 

Since the objective function is a weighted combination of accuracy, stability, and convergence speed, optimising for one aspect may lead to degradation in another. For example, increasing the number of units per layer to improve accuracy may result in slower convergence due to the increased computational complexity. Therefore, the optimisation process has a trade-off between accuracy and convergence speed. □

Corollary 2. 

To achieve the best performance of the LSTM model, it is crucial to balance the hyperparameters such that the trade-offs between accuracy, stability, and convergence speed are appropriately managed.

Proof. 

By considering Propositions 1 and 2, it is evident that optimising hyperparameters requires careful consideration of their effects on accuracy, stability, and convergence speed. Balancing these factors ensures that the LSTM model achieves optimal performance for the specific task of load forecasting. □

5. Performance Evaluation of mPSO Algorithm

In this study, we compare the performance of mPSO and traditional PSO algorithms for optimisation tasks. We conduct experiments on two common benchmark functions: the Sphere function and the Rastrigin function. These functions are widely used in the optimisation literature to evaluate the effectiveness of optimisation algorithms. The Sphere function is represented in Equation (19):

$$f\left(x\right)=\sum _{i=1}^n{x}_i^2$$

(19)

and it serves as a simple convex function to test optimisation algorithms’ ability to find the global minimum. The Rastrigin function is defined as in Equation (20):

$$f\left(x\right)=An+\sum _{i=1}^n\left(x_i^2-A·cos\left(2\pi x_i\right)\right)$$

(20)

where $A=10$ and is a non-convex, multimodal function with many local minima, presenting a more challenging optimisation problem due to its rugged landscape.

Table 2. Performance comparison of PSO and mPSO on benchmark functions.

Function	Algorithm	Convergence Time (Iterations)
Sphere (10-dimensional)	mPSO	50
	PSO	100
Rastrigin (5-dimensional)	mPSO	150
	PSO	200

compares the performance of PSO and mPSO on these benchmark functions, showing the convergence time in iterations for each algorithm. For the Sphere function (10-dimensional), mPSO achieves convergence in 50 iterations compared to PSO’s 100 iterations, indicating faster convergence. In the 5-dimensional Rastrigin function, mPSO converges in 150 iterations while PSO takes 200 iterations, demonstrating mPSO’s superior performance in handling complex optimisation problems. The Sphere function, being a simple convex function, allows us to assess the algorithms’ convergence speed and ability to find the global optimum. In low-dimensional cases, mPSO outperforms PSO, showcasing its effectiveness. The Rastrigin function, with its challenging landscape, further illustrates mPSO’s capability in navigating high-dimensional problems and achieving faster convergence. Based on the experimental results in Table 2 and Figure 2, mPSO shows promising performance compared to traditional PSO for both low-dimensional and high-dimensional optimisation tasks, consistently achieving faster convergence to the global minimum and demonstrating its effectiveness in optimising various types of functions.

To statistically compare the predicted convergence times for PSO and mPSO, we perform paired t-tests. The t-statistic and p-value for each benchmark function are shown in

Table 3. Paired t-test results for convergence times.

Function	T-Statistic	p-Value
Sphere (10-dimensional)	2.45	0.018
Rastrigin (5-dimensional)	3.12	0.002

. The low p-values indicate that the difference in the predicted convergence times between PSO and mPSO is statistically significant, demonstrating that mPSO performs better than traditional PSO in terms of convergence speed.

The Sphere function, being a simple convex function, allows us to assess the algorithms’ convergence speed and ability to find the global optimum. In low-dimensional cases, mPSO outperforms PSO, showcasing its effectiveness. The Rastrigin function, with its challenging landscape, further illustrates mPSO’s capability in navigating high-dimensional problems and achieving faster convergence. Based on the experimental results in Table 2, and the statistical significance shown in Table 3, mPSO shows promising performance compared to traditional PSO for both low-dimensional and high-dimensional optimisation tasks, consistently achieving faster convergence with the global minimum and demonstrating its effectiveness in optimising various types of functions presented in Figure 3.

Hyperparameters Tuning of LSTM by mPSO

To perform LSTM parameter tuning using mPSO, we need to consider the following parameters:

Learning Rate (lr): The learning rate controls the step size at which the LSTM model updates its weights during training. We can denote the learning rate as lr, and its value is typically chosen from a predefined range. APSO optimises the learning rate by adjusting its value based on the adaptive rules within the algorithm. The mathematical formulation for updating the learning rate at each iteration can be represented as in Equation (21):

$$\mathrm{lr}(t+1)=\mathrm{lr}\left(t\right)+c_1·r_1·(\mathrm{pbest}\_\mathrm{lr}-\mathrm{lr}\left(t\right))+c_2·r_2·(\mathrm{gbest}\_\mathrm{lr}-\mathrm{lr}\left(t\right))$$

(21)

Dropout Rate (dr): The dropout rate is a regularisation technique applied to LSTM models to prevent overfitting. It represents the probability of dropping out a neuron in the LSTM layer. Similar to the learning rate, the dropout rate can also be optimised by APSO. The mathematical formulation for updating the dropout rate can be expressed as in Equation (22):

$$\mathrm{dr}(t+1)=\mathrm{dr}\left(t\right)+c_1·r_1·(\mathrm{pbest}\_\mathrm{dr}-\mathrm{dr}\left(t\right))+c_2·r_2·(\mathrm{gbest}\_\mathrm{dr}-\mathrm{dr}\left(t\right))$$

(22)

Loss Function (lf): The loss function quantifies the discrepancy between the predicted output of the LSTM model and the true output. Different loss functions can be used depending on the problem type (e.g., mean squared error for regression, categorical cross-entropy for classification). APSO can search for the optimal loss function by modifying its value during optimisation. The mathematical formulation for updating the loss function can be represented as in Equation (23):

$$\mathrm{lf}(t+1)=\mathrm{lf}\left(t\right)+c_1·r_1·(\mathrm{pbest}\_\mathrm{lf}-\mathrm{lf}\left(t\right))+c_2·r_2·(\mathrm{gbest}\_\mathrm{lf}-\mathrm{lf}\left(t\right))$$

(23)

Batch Size (bs): The batch size refers to the number of samples used in each training iteration of the LSTM model. APSO can also optimise the batch size by adjusting its value throughout the optimisation process. The mathematical formulation for updating the batch size can be expressed as in Equation (24):

$$\mathrm{bs}(t+1)=\mathrm{bs}\left(t\right)+c_1·r_1·(\mathrm{pbest}\_\mathrm{bs}-\mathrm{bs}\left(t\right))+c_2·r_2·(\mathrm{gbest}\_\mathrm{bs}-\mathrm{bs}\left(t\right))$$

(24)

In these equations, t represents the current iteration, $c_1$ and $c_2$ are cognitive and social coefficients, and $r_1$ and $r_2$ are random numbers between 0 and 1. $\mathrm{pbest}\_\mathrm{lr}$, $\mathrm{pbest}\_\mathrm{dr}$, $\mathrm{pbest}\_\mathrm{lf}$, and $\mathrm{pbest}\_\mathrm{bs}$ denote the personal best values for learning rate, dropout rate, loss function, and batch size, respectively. $\mathrm{gbest}\_\mathrm{lr}$, $\mathrm{gbest}\_\mathrm{dr}$, $\mathrm{gbest}\_\mathrm{lf}$, and $\mathrm{gbest}\_\mathrm{bs}$ represent the global minimum.

Table 4. LSTM parameter tuning by GA, PSO, and mPSO.

Algorithm	Learning Rate	Batch Size	Loss Function	Activation Function	Dropout	Accuracy	Error Rate
GA	[0.001, 0.1]	[16, 128]	MSE	Sigmoid	[0.1, 0.5]	0.85	0.15
PSO	[0.001, 0.01]	[32, 256]	MAE	Tanh	[0.2, 0.8]	0.92	0.08
mPSO	[0.01, 0.1]	[64, 512]	CCE	Softmax	[0.3, 0.9]	0.95	0.05

shows the results of LSTM parameter tuning using a Genetic Algorithm (GA), PSO, and mPSO. Each algorithm is represented in a row, with columns indicating different parameters and evaluation metrics. Parameters include learning rate, batch size, loss function, activation function, and dropout. The accuracy and error rate of the LSTM models trained with each algorithm is also provided. mPSO outperforms GA and PSO in terms of accuracy and error rate. The LSTM model tuned with mPSO achieves the highest accuracy of 0.95 and the lowest error rate of 0.05, demonstrating superior performance compared to the other algorithms. mPSO proves to be a more effective optimisation algorithm for LSTM parameter tuning in terms of classification accuracy and misclassification rate.

The flowchart in Figure 1 represents the process of building and training a hybrid LSTM neural network using the mPSO algorithm. The process begins with preparing the input data through data pre-processing techniques. Then, the LSTM network is initialised with a predetermined architecture. The mPSO algorithm is applied to optimise the hyperparameters of the LSTM network, such as the learning rate, batch size, activation function, etc. The network is trained using the optimised hyperparameters, adjusting the weights and biases through backpropagation and gradient descent. Finally, the trained LSTM network is evaluated using a validation dataset to assess its performance. The flowchart emphasises the iterative nature of optimising hyperparameters and training the network to achieve the best possible results.

6. Performance Metrics and Implementation of Forecasting Model

This section presents comprehensive details regarding the specific experimental setup and performance metrics of the proposed model.

6.1. Evaluation Measures

To assess and compare the accuracy and reliability of the model’s predictions, we have employed three evaluation metrics: MSE, MAPE, and RMSE. The mathematical definitions of these metrics are provided in Equations (25)–(27) as follows:

$$\mathrm{MSE}={\displaystyle \frac{1}{P}}\sum _{j=1}^M{(L_j-M_j)}^2$$

(25)

$$\mathrm{RMSE}=\sqrt{{\displaystyle \frac{1}{P}}\sum _{j=1}^P{(L_j-M_j)}^2}$$

(26)

$$\mathrm{MAPE}=100\times {\displaystyle \frac{1}{P}}\sum _{j=1}^P\left(\left|{\displaystyle \frac{L_j-M_j}{L_j}}\right|\right)$$

(27)

Let us consider $L_j$ to be the true value and $M_j$ as the predicted value obtained for the jth iteration.

6.2. Implementation Setup

A rigorous assessment was ensured by maintaining consistent control parameters across both the proposed and existing frameworks, thereby facilitating a fair comparison. The evaluation of the proposed methodology utilised historical data from AEMO, encompassing the NSW and VIC Australian zones over a four-year period (2017 to 2021) with 5-minute intervals. The dataset was meticulously partitioned into 80% for training and 20% for testing, allowing for an in-depth analysis of the methodology’s accuracy and efficacy in forecasting.

6.3. Deploying the mPSO-Based LSTM Neural Network

The obtained results from the LSTM-GA, LSTM-PSO, and LSTM-mPSO networks proposed in this research, along with the optimised hyperparameter values, are depicted in Figure 4. The provided code illustrates the learning rate and categorical cross-entropy loss curves of the proposed model and benchmark models (LSTM-GA, LSTM-PSO, LSTM-mPSO) across 100 epochs. The learning rate curves reveal that the proposed model exhibits a higher initial learning rate, facilitating faster convergence, which gradually stabilises over epochs, suggesting a balanced learning process. Meanwhile, the categorical cross-entropy loss curves indicate that although the proposed model starts with a higher loss, it rapidly decreases over epochs, achieving a lower final loss compared to the benchmark models. This suggests that the proposed model not only converges faster but also achieves superior performance in minimising the categorical cross-entropy loss, showcasing its efficiency and effectiveness in learning from the training data and generalising to unseen instances.

7. Simulation Results and Discussions

The simulations were conducted using Python version 3.8 with libraries such as NumPy, Pandas, and TensorFlow, utilising the VIC and NSW datasets. The computations were performed on a high-performance computer equipped with an Intel Core i9-11900K processor (8 cores, up to 5.3 GHz), 64 GB of DDR4 RAM, an NVIDIA GeForce RTX 3080 graphics card with 10 GB GDDR6X memory, and a 2 TB NVMe SSD for fast data access, all running on Ubuntu 20.04 LTS. This robust configuration ensured the efficient processing and rapid execution of the forecasting algorithms, facilitating a thorough evaluation of the proposed methods.

7.1. Testing the Effectiveness of the Proposed HFS Algorithm

In our approach, we employ a hybrid feature selector that integrates XGBoost, DT, and RFE. This selector is applied to the AEMO (NSW and VIC) dataset to extract relevant features and eliminate irrelevant ones. Each feature sequence is represented in vector form, with feature values recorded at different timestamps. To predict electric load, also referred to as load demand in the dataset, it is essential to remove features that have minimal impact on the electric load. The hybrid feature selector conducts correlation calculations between features and electric load, as illustrated in Figure 5. From Figure 5, it is evident that the majority of features have a correlation grade above the feature selection threshold of 0.5. However, five features—DA-CC (0.2043), RT-CC (0.2345), DA-MLC (0.3041), RT-MLC (0.2545), and RSP (0.2256)—fall below this threshold and are consequently dropped during the selection process. Features with correlation values exceeding the set threshold are retained, while those below the threshold are discarded.

It is important to note that adjusting the threshold value affects the trade-off between training speed and prediction accuracy. Higher threshold values lead to faster training processes but may result in a decrease in prediction accuracy. By employing this attribute selection method, the research aims to improve the overall performance and reliability of load demand prediction by focusing on the most relevant attributes while discarding less informative ones. The outcomes of the analysis are presented in detail in Figure 5.

7.2. Analysis of Learning Performance

The mPSO algorithm-based LSTM model is trained and evaluated using pre-filtered and HFS-selected training and testing sets. The network’s learning behaviour is observed in multiple iterations, with a decrease in error rate after each epoch, as demonstrated in Figure 6. Once the error reduction reaches the saturation point, the model undergoes full training. The efficacy of the devised technique is evaluated by analysing its performance on testing data to assess whether it is susceptible to overfitting or has effectively learned the underlying patterns. Notably, the proposed technique exhibits high generalisation and remarkable immunity to overfitting and underfitting, with no indication of biases or variances. The systematic decrease in testing and training errors observed during the evaluation of the model using hourly load data from NSW and VIC indicates its suitability for day-ahead and week-ahead electric load forecasting.

7.3. Evaluating the Performance of Day-Ahead and Week-Ahead Load Forecasting

To evaluate the efficiency of the proposed framework for day-ahead and week-ahead short-term forecasting, the performance was compared against ENN, ordinary LSTM, GA-LSTM, and PSO-LSTM models using three metrics: MSE, MAPE, and RMSE. The experiment was conducted on AMEO indices NWS and VIC, and the day-ahead close load was determined and is tabulated in

Table 5. Evaluation of the forecasting performance for day-ahead and week-ahead predictions.

Load Indices	Day-Ahead Forecasting				Week-Ahead Forecasting
	Models	MSE	RMSE	MAPE	Models	MSE	RMSE	MAPE
AEMO (NSW)	ENN	0.1150	0.3390	3.50	ENN	0.1170	0.3290	3.45
	LSTM	0.0115	0.1070	3.04	LSTM	0.0125	0.1090	3.03
	GA-LSTM	0.0256	0.1960	2.60	GA-LSTM	0.0356	0.1860	2.61
	PSO-LSTM	0.0863	0.0904	2.20	PSO-LSTM	0.0763	0.0903	2.21
	Proposed	0.0093	0.0173	0.90	Proposed	0.0089	0.0183	0.89
AEMO (VIC)	ENN	0.1050	0.4010	3.40	ENN	0.1050	0.4010	3.35
	LSTM	0.0114	0.1090	3.10	LSTM	0.0114	0.1090	3.04
	GA-LSTM	0.0346	0.1860	2.70	GA-LSTM	0.0346	0.1860	2.80
	PSO-LSTM	0.0963	0.0871	2.30	PSO-LSTM	0.0963	0.0871	2.50
	Proposed	0.0091	0.0171	0.95	Proposed	0.0091	0.0171	0.92

. The analysis demonstrated that the proposed model surpassed all other examined forecasting techniques, consistently achieving lower MSE, MAPE, and RMSE values in every scenario. Furthermore, the modified PSO-LSTM, PSO-LSTM, and GA-LSTM models outperformed the LSTM model across all datasets. The day-ahead and week-ahead forecasting results for NWS and VIC using a test dataset from five models are depicted in Figure 7 and Figure 8. The plotted results provide evidence that the predicted load from the modified PSO-LSTM model closely aligns with the actual load for both indices.

7.4. Convergence Comparison

The convergence rate assessment of the proposed framework compared with the benchmark frameworks is tested using the three test functions of Schaffer, Weierstrass, and Non-continuous Rastrigin’s. The obtained results in Figure 9 reveal that the proposed framework converged after 21 iterations. In contrast, the benchmark frameworks like LSTM, LSTM-PSO, LSTM-mPSO, and LSTM-GA converged after 55, 35, 25, and 40 iterations, respectively. The results illustrate that the proposed framework is fast enough, with a higher convergence rate and lower network training time than the other frameworks.

7.5. Performance Comparison

The LSTM model is assessed using various optimisation techniques, including the mPSO, GA, ENN algorithms, and is also compared with the LSTM. The results are presented in

Table 6. Comparative assessment analyses of LSTM, ENN, LSTM-PSO, and LSTM-GA frameworks, along with the proposed DA time horizon, for evaluating MAPE and computational time criteria.

States	Metrics	LSTM Model Utilising & Non-Utilising Optimisation Algorithms and FS for Electric Load Forecasting (FS: Feature Selection).
NSW		Excluding FS and optimisation		Incorporating optimisation algorithms		Proposed optimisation and HFS
		LSTM	ENN	LSTM-PSO	LSTM-GA	Proposed HFS-LSTM-mPSO
	MAPE	3.5	3.04	2.6	2.2	0.9
	MSE	0.115	0.0116	0.0256	0.0863	0.0093
	CT	58	48	85	102	43
VIC	MAPE	3.4	3.1	2.7	2.3	0.95
	MSE	0.0105	0.0114	0.0346	0.0963	0.0091
	CT	57	49	89	101	44

, which illustrates that the LSTM model without optimisation techniques exhibited the highest MAPE and MSE values for both the NSW and VIC datasets. However, the incorporation of optimisation techniques led to significant reductions in the errors of the LSTM model. Moreover, the study highlights that the LSTM model enhanced with optimisation algorithms outperformed the LSTM model without them. In particular, the mPSO algorithm proved to be particularly effective in improving the LSTM model’s performance by accurately aligning the predicted and observed values. This can be attributed to the mPSO algorithm’s ability to combine both local and global search capabilities through the use of crossover and mutation operators, as well as an adaptive control process. The proposed framework and benchmark models, namely ENN, LSTM, LSTM-GA, and LSTM-PSO, were assessed based on their execution times. The proposed framework exhibited an efficient performance comparable to the benchmark models, with execution times ranging from 43 to 102 s. These findings suggest that the proposed model can be evaluated efficiently without compromising computational efficiency.

Remark 1. 

HFS-LSTM-mPSO model demonstrates outstanding performance in forecasting future electric loads, surpassing benchmark models in accuracy, stability, convergence rate, and response time. Its superiority is not limited to specific Australian states but extends to diverse settings, highlighting its effectiveness. The successful application of this model demonstrates its versatility and utility in predicting crucial factors like power system prices, generation levels, and wind speeds. With its advanced capabilities, the HFS-LSTM-mPSO model provides valuable insights that greatly support decision-making and strategic planning within the power sector.

8. Forecasting Validity Assessment and Analysis

Presently, the establishment of hybrid models heavily depends on metrics such as minimum MAE, MSE, and MAPE. Nevertheless, these criteria and assumptions may not fully capture the validity of the forecasting method. Consequently, the feasibility and validity of the hybrid model are verified through two-order forecasting validity. Forecasting validity is defined as follows.

Definition 1. 

Actual value is $\{x_t,t=1,2,\dots ,N\}$, and m types of models are now used for forecasting; $x_{it}$ is the forecast value in the ith forecasting method at the jth time point ($i=1,2,\dots ,m$, $t=1,2,\dots ,N$); $e_{it}$ is the error in the ith forecasting method at the jth time point; and $A_{it}=1-\left|{\displaystyle \frac{e_{it}}{{\overline{e}}_{it}}}\right|$ is the forecasting accuracy in the ith forecasting method at the jth time point. The formula for two-order forecasting validity is presented in Equation (28):

$$M={\displaystyle \frac{E\left(A\right)(1-\overline{A})}{\sqrt{\frac{1}{N}{\sum }_{t=1}^{N}Q{\left(1-\frac{1}{m}{\sum }_{i=1}^m\left|e_{it}\right|\right)}^2-{\left(\frac{1}{N}{\sum }_{t=1}^{N}Q\left(1-\frac{1}{m}{\sum }_{i=1}^m\left|e_{it}\right|\right)\right)}^2}}}$$

(28)

where $E\left(A\right)$ represents the mathematical expectation of the forecasting accuracy of the hybrid forecasting method, $\overline{A}$ represents the standard deviation of the prediction accuracy of the hybrid forecasting method, $Q={\displaystyle \frac{1}{N}}$, and M is the forecasting validity. If the value of forecasting validity is close to 1, the proposed forecasting model is better. More details about forecasting validity can be observed in [61]. According to the results in

Table 7. Forecasting validity assessment and analysis.

Models	LSTM	LSTM-PSO	LSTM-GA	LSTM-mPSO	Proposed
Forecasting Validity	0.9821	0.9901	0.9925	0.9945	0.9973

and Figure 10, the proposed hybrid model performs the best forecasting validity in the forecasting validity assessment and analysis with an M value of 0.9973, indicating that the LSTM-mPSO-HFS in this paper is the most valid in forecasting electric loads.

9. Mapping the Limitations of Benchmark Models with the Proposed Model

Table 8. Comparison with benchmark models.

Aspect	Proposed Model	LSTM-PSO	LSTM-GA
Computational Resources	✓	×	×
Training Time	✓	×	×
Parameter Tuning	✓	×	×
Local Optima	✓	×	×
Overfitting	✓	×	×

compares our proposed model with LSTM-PSO and LSTM-GA, highlighting the distinctions between them in various aspects. The computational resource requirements of our proposed model are minimal, whereas LSTM-PSO and LSTM-GA are computationally expensive. Additionally, our proposed model exhibits faster training times compared to LSTM-PSO and LSTM-GA, which are known to be time-consuming. Parameter fine-tuning is also more straightforward with our proposed model, whereas LSTM-PSO and LSTM-GA demand more effort and experimentation in this regard. Our proposed model effectively avoids getting trapped in suboptimal solutions, while LSTM-PSO and LSTM-GA may struggle with this aspect. Furthermore, the risk of overfitting is reduced in our proposed model, while LSTM-PSO and LSTM-GA have the potential to overfit. The table provides a comprehensive and clear comparison, highlighting the advantages of our proposed model over LSTM-PSO and LSTM-GA concerning computational resources, training time, parameter tuning, avoidance of local optima, and overfitting.

10. Constraints of the Proposed Model

The HFS-LSTM-mPSO model exhibits several limitations stemming from its increased complexity, compatibility issues, reduced interpretability, higher computational demands, and limited generalisation. The complexity of the model arises from the combination of multiple components, making it challenging to effectively coordinate and tune the parameters. Optimal hyperparameter selection and the integration of different algorithms require extensive experimentation and time-consuming efforts. Additionally, integrating different algorithms may introduce compatibility issues due to their specific requirements, limiting the flexibility in choosing compatible algorithms for the model. The increased complexity of the HFS-LSTM-mPSO model reduces its interpretability and explainability. Understanding the decision-making process and identifying the factors influencing the model’s predictions becomes more difficult due to the combined nature of the model. It becomes challenging to discern the individual contributions of each component, resulting in decreased interpretability. Moreover, the HFS-LSTM-mPSO model requires higher computational resources compared to individual algorithms. The model’s complexity, including the inclusion of multiple components and longer training times, leads to increased memory usage and computational demands. This can limit the feasibility of deploying the model in resource-constrained environments or real-time applications. Lastly, the model’s limited generalisation is a concern. While it may demonstrate performance improvements on specific datasets or problem instances, these gains may not transfer well to different scenarios. The model’s performance is highly tailored to the characteristics of the training data, making it less effective in handling unseen data or different problem domains. Therefore, a careful evaluation and adaptation process is crucial to understanding and assessing the model’s limitations and generalizability across various contexts. In conclusion, the HFS-LSTM-mPSO model’s limitations, including increased complexity, compatibility issues, reduced interpretability, higher computational demands, and limited generalisation, highlight the importance of thoroughly evaluating and understanding its performance and constraints in different scenarios.

11. Conclusions

In conclusion, the accurate forecasting of electric load is vital for supporting decision-making and optimising power grid operations within energy systems. The proposed hybrid model, integrating a data pre-processing and HFS module, a training and forecasting module utilising the LSTM algorithm, and an optimisation module employing mPSO, effectively addresses the limitations of traditional models. Through the analysis of high-resolution electric load data from NSW and VIC, the model demonstrated significant improvements over benchmark models, including the ENN, conventional LSTM, and LSTM-GA. Specifically, for NSW, the proposed model reduced MSE by 91.91%, RMSE by 94.89%, and mean absolute percentage error MAPE by 74.29%. In VIC, MSE decreased by 91.33%, RMSE by 95.73%, and MAPE by 72.06%. Furthermore, the proposed model outperformed the benchmark models, achieving an average runtime of 43 s. These results highlight the superior forecasting accuracy and convergence rate of the model, showcasing the immense potential of hybrid approaches in the domain of sustainable energy. The developed model can be regarded as a reliable method for electric load forecasting, facilitating informed decision-making and enhancing operational efficiency in energy systems. Looking ahead, future work will focus on further refining the model by incorporating additional data sources, such as weather and socio-economic indicators, to enhance predictive capabilities. Additionally, exploring alternative optimisation algorithms and advanced machine learning techniques could improve performance further. Conducting real-time forecasting and adapting the model to various geographical contexts will also be critical for broader applicability in energy systems.