Ultra-Short-Term Wind Power Prediction Based on LSTM with Loss Shrinkage Adam

Abstract

With the rapid increase in wind power, its strong randomness has brought great challenges to power system operation. Accurate and timely ultra-short-term wind power prediction is essential for the stable operation of power systems. In this paper, an LsAdam–LSTM model is proposed for ultra-short-term wind power prediction, which is obtained by accelerating the long short-term memory (LSTM) network using an improved Adam optimizer with loss shrinkage (LsAdam). For a specific network topology, training progress heavily depends on the learning rate. To make the training loss of LSTM shrink faster with standard Adam, we use the past training loss-changing information to finely tune the next learning rate. Therefore, we design a gain coefficient according to the loss change to adjust the global learning rate in every epoch. In this way, the loss change in the training process can be incorporated into the learning progress and a closed-loop adaptive learning rate tuning mechanism can be constructed. Drastic changes in network parameters will deteriorate learning progress and even make the model non-converging, so the gain coefficient is designed based on the arctangent function with self-limiting properties. Because the learning rate is iteratively tuned with past loss-changing information, the trained model will have better performance. The test results on a wind turbine show that the LsAdam–LSTM model can obtain higher prediction accuracy with much fewer training epochs compared with Adam–LSTM, and the prediction accuracy has significant improvements compared with BP and SVR models.

1. Introduction

In recent years, wind energy has gained significant attention as an environmentally friendly, clean renewable energy. Wind power generation has the potential to alleviate the shortage of conventional energy and mitigate increasing environmental pollution [1,2]. However, due to the strong volatility and randomness of wind power, its integration into the grid on a large scale can adversely impact the stability of the power system [3]. Accurate prediction of wind power can aid power companies, power plants and grid management departments in planning and dispatching wind power generation systems, improving the reliability and stability of power systems and reducing dependence on traditional energy sources [4,5].

Wind power prediction methods mainly include physical methods, statistical methods and combination methods [6]. The physical methods need to establish a model based on the complex physical relationship between various physical quantities such as meteorological information and geomorphological information, which has a high computational cost of prediction and is not suitable for ultra-short-term prediction [7]. Statistical methods establish a model that can express the mapping relationship between input and output by mining the inherent laws between a large number of historical data [8]. Commonly used statistical methods include time series analysis [9,10,11], the BP neural network (BPNN) [12,13], support vector regression (SVR) [14,15,16,17] and the deep learning network [18,19]. Wind power has the characteristics of a time series and has strong non-linearity. Models such as the time series analysis method, BPNN and SVR have difficulty in extracting the deeper features in a wind power series and cannot cope with their complex change trends, so it is difficult to predict wind power accurately using these methods. In contrast, deep learning models have stronger non-linear mapping abilities, self-learning abilities and feature extraction abilities [20]. In particular, the long short-term memory (LSTM) network is suitable for wind power prediction due to its unique network structure [21]. Several prediction models of wind power based on the LSTM have been proposed in [22,23,24,25,26]. These models use the LSTM network to learn the time series characteristics in wind power data and have higher prediction accuracy than linear models, traditional machine learning models and artificial neural networks.

The performance of deep learning networks heavily depends on the learning progress, so the optimization algorithm of the network training is crucial [27]. For the stochastic gradient descent method, the learning rate in the next round of training is adjusted according to the loss change in the training process and the current learning rate, which can improve the training speed and accuracy of the model [28]. However, this method could not adaptively update every parameter in the network, and the prediction accuracy of the model was also limited. The adaptive moment estimation (Adam) algorithm is improved from the root mean square propagation (RMSProp) algorithm [29]. It can adaptively update each parameter in deep learning networks and is also one of the commonly used optimization algorithms in deep learning networks. However, the global learning rate of Adam is a fixed value, and to avoid network non-converging caused by a learning rate that is too large in the later stage, the value of the global learning rate is often very small; therefore, the optimization effect of the Adam algorithm on the network is greatly limited.

To further improve the prediction accuracy and modeling efficiency of the model, we propose a loss shrinkage Adam (LsAdam) optimization algorithm by adjusting the global learning rate of the Adam according to the loss change during the model training process and construct an LsAdam–LSTM wind power prediction model.

2. Principle of LSTM Ultra-Short-Term Wind Power Prediction

2.1. LSTM Model Structure

The recurrent neural network (RNN) is suitable for processing time series problems with its special recurrent structure, while the LSTM network is a deep learning network improved on the basis of the RNN. LSTM is different from the RNN because it uses gating structures, known as “forget gate”, “input gate” and “output gate”, to control information flow [30]. The basic structure of the LSTM cell is shown in Figure 1.

In Figure 1, f_t is the “forget gate”, i_t is the “input gate”, o_t is the “output gate” and ${\widehat{c}}_t$ is the temporary state. The “forget gate” can determine how much historical information is abandoned. The “input gate” determines what is retained from the input. The role of the “output gate” is to control the information that the current unit outputs to the next unit. Current information becomes the temporary state of the unit when it enters the current unit; then, the historical information and the current temporary state are combined to obtain the current state of the unit, and, finally, the output of the current unit is obtained by passing through the output gate. These gated structures can effectively solve gradient explosion and gradient disappearance problems from the RNN, which means that the LSTM network can better learn the long-term dependence between wind power data. The calculation method of the LSTM cell is shown in Algorithm 1.

Algorithm 1 LSTM algorithm

$\mathbf{Require}: \mathrm{Random} \mathrm{initialization} U_f$$, U_i$$, U_o$$, U_c$$, b_f$$, b_i$$, b_o$$\mathrm{and} b_c$$, h_0=0$$, c_0=0$

$\mathbf{For} t\leftarrow 1$ to $n$ do

Calculate “forget gate”

$f_t=\sigma (h_{t-1}\cdot W_f+x_t\cdot U_f+b_f)$

Calculate “input gate”

$i=\sigma (h_{t-1}\cdot W_i+x_t\cdot U_i+b_i)$

Calculate “output gate”

$o_t=\sigma (h_{t-1}\cdot W_o+x_t\cdot U_o+b_o)$

Calculate temporary state

${\widehat{c}}_t=\tanh (h_{t-1}\cdot W_c+x_t\cdot U_c+b_c)$

Calculate current state

$c_t=f_t\circ c_{t-1}+i_t\circ {\widehat{c}}_t$

Calculate output

$h_t=o_t\circ \tanh (c_t)$

end

$\mathbf{return} h_1,h_2,\dots ,h_n$

where $c_{t-1}$ is the state of the cell at the previous time, $h_{t-1}$ is the output of the cell at the previous time, $x_t$ is the input of the cell at the current time, $W_f$, $W_i$, $W_o$ and $W_c$ are the weights of $h_{t-1}$, $U_f$, $U_i$, $U_o$ and $U_c$ are the weights of $x_t$, $b_f$, $b_i$, $b_o$ and $b_c$ are the biases, $n$ is the length of the time series, $\sigma$ is the Sigmoid function, $f_t$ is the output of the forget gate, $i_t$ is the output of the input gate, $o_t$ is the output of the output gate, ${\widehat{c}}_t$ is the temporary state in the cell, $c_t$ is the current state in the cell, and $h_t$ is the output of the cell at the current time.

2.2. Learning Rate Update Strategy Based on Adam

As one of the commonly used optimization algorithms, Adam can adaptively update each parameter in the deep learning network. The optimization process of Adam is shown in Algorithm 2.

In Algorithm 2, $t$ is the counter of iteration number, ${\beta }_1$ and ${\beta }_2$ are the exponential decay rate of moment estimation, and the range of ${\beta }_1$ and ${\beta }_2$ is [0, 1); usually, ${\beta }_1=0.9$, ${\beta }_2=0.999$. $\mu$ is a tiny number to prevent the denominator from being 0; usually, $\mu =1\times {10}^{-8}$. ${\theta }_0$ is the initial parameter vector, $g_t$ is the current gradient of the parameter, $m_t$ is the biased first moment estimate, $v_t$ is the biased second moment estimate, ${\widehat{m}}_t$ is the corrected first moment estimate, ${\widehat{v}}_t$ is the corrected second moment estimate, $lr$ is the global learning rate, $\Delta \theta$ is the update of the parameter, and ${\theta }_t$ is the updated parameter

Algorithm 2 Adam optimization algorithm

Require: ${\beta }_1,{\beta }_2\in [0,1)$$, \mu =1\times {10}^{-8}$,$m_0=0$$, v_0=0$$, t=0$$, {\theta }_0$: initial parameter ve$\mathrm{ctor}, lr$: global learning rate

$\mathbf{while} {\theta }_t$ not converged

$t\leftarrow t+1$

Calculate the gradient at step t

$g_t={\nabla }_{\theta }{f}_t({\theta }_{t-1})$

Update biased first moment estimate and biased second moment estimate

$m_t={\beta }_1\cdot m_{t-1}+(1-{\beta }_1)\cdot g_t$

$v_t={\beta }_2\cdot v_{t-1}+(1-{\beta }_2)\cdot (g_t\odot g_t)$

Update unbiased first moment estimates and unbiased second moment estimates

${\widehat{m}}_t=m_t/(1-{\beta }_1^t)$

${\widehat{v}}_t=v_t/(1-{\beta }_2^t)$

Update change of the parameters

$\Delta \theta =lr\cdot {\widehat{m}}_t/(\mu +\sqrt{{\widehat{v}}_t})$

Updated parameters

${\theta }_t={\theta }_{t-1}-\Delta \theta$

end while

$\mathbf{return} {\theta }_t$

When the Adam algorithm optimizes the LSTM network, although each parameter in the network can be adaptively updated, its global learning rate is fixed. In addition, to ensure that the LSTM network can converge in the later training stage, the global learning rate can only be set to a small value, and the convergence speed and prediction accuracy of the network will be limited.

A lot of information will be fed back in the process of network training, and some of the information can reflect the pros and cons of the network learning process. We can use this feedback information during the training process to adjust the global learning rate and further optimize the training effect of the network.

3. LsAdam–LSTM Wind Power Prediction Model

Loss change is very important in the training process because the direction and magnitude of loss change can reflect the quality of the network learning process. Therefore, to improve the training speed of the wind power prediction model based on LSTM, further improve the prediction accuracy and overcome the limitations of the fixed global learning rate, it is necessary to adaptively adjust the global learning rate through loss change during network training. If the loss decreases after each training epoch, it is reasonable to think that parameter adjustment can gradually make the network converge, and the global learning rate can be appropriately increased to accelerate the training process. In contrast, if the loss increases after each training epoch, it means that the current parameter adjustment is not conducive to the training process, and the global learning rate can be appropriately reduced to stabilize the training process. To ensure faster loss shrinkage after each epoch of training, a gain coefficient based on loss change is used to dynamically adjust the global learning rate, as shown in Equation (1).

$$\Delta \theta =l{r}_T\cdot {\widehat{m}}_t/(\mu +\sqrt{{\widehat{v}}_t})$$

(1)

where $l{r}_T$ is the global learning rate at the $T_{th}$ training epoch, $l{r}_T=G_T\cdot l{r}_{T-1}$, and $G_T$ is the gain coefficient at the $T_{th}$ training epoch.

Unlike the Adam algorithm in which the global learning rate is a fixed value, we consider the loss change during training to adjust the global learning rate in real time and construct the LsAdam algorithm based on Equation (1) to determine the global learning rate at each training epoch. Based on the Adam algorithm, the proposed algorithm adaptively adjusted the global learning rate through the gain coefficient with the information of loss change, which could effectively alleviate the limitations of the model caused by the small fixed value of the global learning rate during the training process.

To effectively determine the gain coefficient, the loss change before and after each epoch of training needs to be evaluated. To better reflect the magnitude of the loss change during training, we use the relative loss change to measure its change, as shown in Equation (2).

$$\Delta l_T=(l_{T-1}-l_T)/l_{T-1}$$

(2)

where $\Delta l_T$ is the relative change of the loss after the $T_{th}$ training epoch compared with the previous epoch, $l_{T-1}$ is the loss value of the network after the ${(T-1)}_{th}$ training epoch, and $l_T$ is the loss value of the network after the $T_{th}$ training epoch.

To prevent the learning rate from being too large or too small due to the introduction of the gain coefficient, this coefficient should be negatively correlated with the current global learning rate. To reduce the influence of random factors such as data noise, an insensitive range $[-\epsilon ,\epsilon ]$ is set, i.e., the global learning rate is kept constant when $\Delta l_T\in [-\epsilon ,\epsilon ]$, i.e., the gain coefficient is 1.

Therefore, the rules for the value of the gain coefficient must satisfy the following conditions:

• $G_{T+1}$ should be bounded and positive.
• When $-\epsilon \le \Delta l_T\le \epsilon$, $G_{T+1}=1$.
• When $\Delta l_T>\epsilon$, $G_{T+1}>1$ and $G_{T+1}$ is positively correlated with $\Delta l_T$.
• When $\Delta l_T<-\epsilon$, $G_{T+1}<1$ and $G_{T+1}$ is negatively correlated with $\left|\Delta l_T\right|$.
• When $\Delta l_T<-\epsilon$ or $\Delta l_T>\epsilon$, $G_{T+1}$ is negatively correlated with $[-\epsilon ,\epsilon ]$.

To this end, the gain coefficient is determined based on the basis of the amount of loss change and the current global learning rate. The inverse tangent function is a monotonically increasing function with limiting characteristics, and the value domain of the positive semi-axis of the independent variable is $(0,\pi /2)$. Therefore, the arctangent function is used as the basis for constructing the value rules for the gain coefficients. We construct the gain coefficients as shown in Equation (3); then, the update rule of the global learning rate is shown in Equation (4).

$$G_{T+1}=\{\begin{array}{l}1+\arctan (k_2\sqrt{\Delta l_T}+l{r}_1/l{r}_T)/k_1, \Delta l_T>\epsilon \\ 1-\arctan (k_2\sqrt{\left|\Delta l_T\right|}+l{r}_T/l{r}_1)/k_1, \Delta l_T<-\epsilon \\ 1, -\epsilon \le \Delta l_T\le \epsilon \end{array}$$

(3)

$$l{r}_{T+1}=\{\begin{array}{l}l{r}_T(1+\arctan (k_2\sqrt{\Delta l_T}+l{r}_1/l{r}_T)/k_1), \Delta l_T>\epsilon \\ l{r}_T(1-\arctan (k_2\sqrt{\left|\Delta l_T\right|}+l{r}_T/l{r}_1)/k_1), \Delta l_T<-\epsilon \\ l{r}_T, -\epsilon \le \Delta l_T\le \epsilon \end{array}$$

(4)

where $l{r}_1$ is the initial global learning rate, which is generally taken as the global learning rate commonly used in the Adam optimization algorithm, such as $l{r}_1=0.01$; $\epsilon$ is the gain coefficient insensitivity threshold, which is taken as $\epsilon =0.001$ in this paper. $k_1$ is a constant, which plays the role of controlling the range of values of $G_{T+1}$ and then controlling the change in the global learning rate. To satisfy $G_{T+1}>0$, it is necessary to make $k_1>\pi /2$. If $k_1$ is smaller, the change in the global learning rate is larger, and the algorithm has a more obvious optimization effect on the model; if $k_1$ is larger, the change in the global learning rate is smaller, and the risk of model non-convergence is lower. So, $k_1$ is taken as $k_1=5\pi$ in this paper. To effectively prevent excessive effects on the gain coefficient due to a wide range of $\left|\Delta l_T\right|$ values, $\sqrt{\left|\Delta l_T\right|}$ is chosen. $k_2$ is a constant, which is used to regulate the sensitivity of the gain coefficient to $\sqrt{\left|\Delta l_T\right|}$, and $k_2>0$. A smaller $k_2$ will make it easier for the global learning rate of the model to reach a smaller value in the later stages of training, which is beneficial to the convergence of the model; a larger $k_2$ allows the global learning rate to change more significantly according to $\sqrt{\left|\Delta l_T\right|}$, which makes the algorithm have a more obvious acceleration effect. So, $k_2$ is taken as $k_2=10$ in this paper.

In addition, when $l{r}_T>l{r}_1$, $l{r}_T$ will have a suppressive effect on the increase in the gain coefficient and a facilitating effect on the decrease in the gain coefficient; when $l{r}_T<l{r}_1$, $l{r}_T$ will have a facilitating effect on the increase in the gain coefficient and a suppressive effect on the decrease in the gain coefficient. As the network model converges, the magnitude of loss change will tend to decrease and the degree of determination of $l{r}_T$ on $l{r}_{T+1}$ will gradually become larger, which makes the global learning rate eventually close to $l{r}_1$ and is conducive to model convergence.

In the LsAdam algorithm, the global learning rate is kept constant when the loss change is within the threshold range, and when the loss change is beyond the threshold range, the loss change and the current global learning rate will jointly determine the value of the global learning rate in the next training epoch. The process of LsAdam–LSTM is shown in Algorithm 3.

Algorithm 3 LsAdam–LSTM algorithm

Require: ${\beta }_1,{\beta }_2\in [0,1), \mu =1\times {10}^{-8}, k_1>\pi /2, k_2>0, \epsilon =0.001, m_0=0, v_0=0, l{r}_1$: initial global learning rate, ${\theta }_0$: initial parameter vector, $T_{\max }: \max \mathrm{epochs},$$los{s}_{aim}$: accuracy requirement, $I$: number of iterations per epoch

$\hspace{1em}T\leftarrow 0$$, t\leftarrow 0$

LSTM model initialization

$\mathrm{Calculate} \mathrm{model} \mathrm{loss} L_T$

$\hspace{1em}\mathbf{while} T<T_{\max }$ $\mathrm{or} L_T>los{s}_{aim}$

$\hspace{1em}\hspace{1em}T\leftarrow T+1$

$\hspace{1em}\hspace{1em}i\leftarrow 0$

$\hspace{1em}\hspace{1em}\mathbf{while} i<I$

$\hspace{1em}\hspace{1em}\hspace{1em}i\leftarrow i+1$

$\hspace{1em}\hspace{1em}\hspace{1em}t\leftarrow t+1$

Calculate the gradient at step t

$\hspace{1em}\hspace{1em}\hspace{1em}{g}_t={\nabla }_{\theta }{f}_t({\theta }_{t-1})$

Update biased first moment estimate

$\hspace{1em}\hspace{1em}\hspace{1em}{m}_t={\beta }_1\cdot m_{t-1}+(1-{\beta }_1)\cdot g_t$

Update biased second moment estimate

$\hspace{1em}\hspace{1em}\hspace{1em}{v}_t={\beta }_2\cdot v_{t-1}+(1-{\beta }_2)\cdot (g_t\odot g_t)$

Update unbiased first moment estimates

$\hspace{1em}\hspace{1em}\hspace{1em}{\widehat{m}}_t=m_t/(1-{\beta }_1^t)$

Update unbiased second moment estimates

$\hspace{1em}\hspace{1em}\hspace{1em}{\widehat{v}}_t=v_t/(1-{\beta }_2^t)$

Update change of the LSTM model parameters

$\hspace{1em}\hspace{1em}\hspace{1em}\Delta \theta =l{r}_T\cdot {\widehat{m}}_t/(\mu +\sqrt{{\widehat{v}}_t})$

Updated LSTM model parameters

$\hspace{1em}\hspace{1em}\hspace{1em}{\theta }_t={\theta }_{t-1}-\Delta \theta$

end while

$\mathrm{Calculate} \mathrm{model} \mathrm{loss} L_T$

$\mathrm{Calculate} \Delta L_T$ by Equation (2)

$\mathrm{Calculate} l{r}_{T+1}$ by Equation (4)

end while

end

The overall change trend in the global learning rate of LsAdam–LSTM has the following characteristics during the training process. In the early stage of training, the general loss will decrease rapidly, and the global learning rate will increase with the decrease in loss, which can improve the convergence speed of the model. In the middle and late stages of training, the reduction speed of loss becomes slower, and the global learning rate gradually decreases. In the later stage of training, the global learning rate will gradually stabilize near the initial global learning rate, which is conducive to the convergence of the model. Based on this characteristic, the proposed model has higher prediction accuracy and a faster convergence speed.

4. Experimental Verification

4.1. Data Preparation

In this work, data from the SCADA systems are used to test the proposed LsAdam–LSTM, including wind speed, wind direction, generated power, etc. The data were sampled with a 10 min interval from the SCADA system of a wind turbine that works and generates power in Turkey. We only used a past generated power series to predict the next power output (10 min in the future). In addition, one month of observations from April 2018 were used to construct the sample set, with a total of 4320 pieces of data as shown in Figure 2. When the wind speed is between the cut-in wind speed and the rated wind speed, the output power of the wind turbine is higher with a higher wind speed. When the wind speed is between the rated wind speed and the cut-out wind speed, the wind power is approximately the rated power. However, when the wind speed is lower than the cut-in wind speed or higher than the cut-out wind speed, the wind power is zero. Wind speed fluctuates over a wide range, so we can see that wind power has strong randomness and intermittency.

To evaluate the prediction performance more objectively, the max–min normalization method was used to scale the data to the interval [0, 1], as shown in Equation (5).

$$x^{\prime }=\frac{x-x_{\min }}{x_{\max }-x_{\min }}$$

(5)

where $x$ is the original data, $x_{\min }$ is the minimum value in the original data, $x_{\max }$ is the maximum value in the original data, and $x^{\prime }$ is the scaled data.

For ultra-short-term power prediction, we used the actual power generated from the past 100 min to predict the power output of the next 10 min. To implement the prediction task with deep learning methods, the samples were constructed using the window receding approach. The width of the time window was chosen to be 10 intervals, so every sample included 11 pieces of data. As shown in

Table 1. Experimental samples.

Sample Index	Input (Past 10 Observations)					Label (Future Power)
1	3603.64	3603.22	…	3603.00	3602.82	3602.66
2	3603.22	3603.36	…	3602.82	3602.66	3602.86
…	…	…	…	…	…	…
4308	232.15	176.85	…	59.71	57.46	3.15
4309	176.85	166.88	…	57.46	3.15	74.91
4310	166.88	138.42	…	3.15	74.91	97.37

, the 1st to 10th data of the sample denote the past actual power, and the last piece of data is used as the future power generated. For one month of power generated records, a sample set with 4310 samples was obtained. To evaluate the generalization performances of the prediction models, the first 80% of the samples (a total of 3448 samples) were used as the training set, and the remaining 20% (a total of 862 samples) were used as the test set.

4.2. Evaluation Index of Ultra-Short-Term Wind Power Prediction Model

According to the Chinese national standard “Technical requirements for dispatching side forecasting system of wind or photovoltaic power” (GB/T 40607-2021), the mean square error (MSE), mean absolute error (MAE) and correlation coefficient (R) of the predicted power and actual power were used as evaluation indexes. Among them, MSE and MAE can reflect the difference between the predicted power and the actual power, and R can reflect the closeness of the correlation between the predicted power and the actual power. These three evaluation indexes were used to measure prediction performance, and the calculation of each index is shown in Equations (6)–(8).

$$MSE=\frac{1}{n}{\displaystyle \sum _{k=1}^n{(P_{M,k}-P_{P,k})}^2}$$

(6)

$$MAE=\frac{1}{n}{\displaystyle \sum _{k=1}^n\left|P_{M,k}-P_{P,k}\right|}$$

(7)

$$R=\frac{{\displaystyle \sum _{k=1}^n[(P_{M,k}-{\overline{P}}_M)(P_{P,k}-{\overline{P}}_P)]}}{\sqrt{{\displaystyle \sum _{k=1}^n{(P_{M,k}-{\overline{P}}_M)}^2{\displaystyle \sum _{k=1}^n{(P_{P,k}-{\overline{P}}_P)}^2}}}}$$

(8)

where n is the number of samples, $P_{M,k}$ is the $k_{th}$ true power value, $P_{P,k}$ is the $k_{th}$ predicted power value, ${\overline{P}}_M$ is the average value of the true power, and ${\overline{P}}_P$ is the average value of the predicted power value.

4.3. Experiment and Discussion

To verify the ultra-short-term wind power prediction performances, several models were trained and tested on the same wind turbine, including traditional shallow network BP, classical machine learning method SVR, Adam–LSTM and the proposed LsAdam–LSTM.

For the deep learning method LSTM, the optimization algorithm used in the training had a heavy effect on the final model. To evaluate the improved LsAdam–LSTM algorithm, the experiments were carried out using LsAdam and standard Adam as the optimization methods for LSTM training, respectively; then, the ultra-short-term wind power prediction models were obtained; namely, LsAdam–LSTM and Adam–LSTM. To further evaluate the prediction accuracy of the proposed method, comparison experiments were implemented with the BP neural network and SVR algorithm to build the ultra-short-term wind power prediction models.

The parameters of all the algorithms used in the experiments were fine-tuned. The main parameters were described as follows. The constructed LSTM network model contained two hidden layers, and each hidden layer contained 64 cells. The activation function was the Tanhyperbolic function, and the parameters in the LsAdam optimization algorithm were chosen as $k_1=5\pi$, $k_2=10$ and $\epsilon =0.001$. When training the LsAdam–LSTM model and the Adam–LSTM model, the maximum epoch of training was set to 500, the initial global learning rate was set to 0.01, and full batch training was used. MSE was used as the error (loss) in the back-propagation process. To further check the adaptability and stability of the LsAdam–LSTM model, the LsAdam–LSTM model and Adam–LSTM model were tested under several commonly used global learning rates of 0.005, 0.02 and 0.03. The structure of the BP neural network model was 10-32-1, the learning rate was 0.1, the training epoch was 300, and the activation function was the Sigmoid function. The SVR model used the radial basis function as the kernel function and set the error penalty factor to 1.

The experiments were implemented with initial global learning rates of 0.005, 0.01, 0.02 and 0.03. The loss-changing trends were similar. As shown in Figure 3, the training progress of LsAdam–LSTM was significantly superior to that of Adam–LSTM, where the initial global learning rate was 0.01. Although the loss change of both methods can decrease with the training epoch globally, the loss of LsAdam–LSTM decreased more drastically in the early stage of training and reached a lower final value. Section (a) in Figure 3 shows the detail of the loss changing in the early training stage. The training process of LsAdam–LSTM was more stable and the decline in loss was faster. In contrast, the Adam–LSTM had several large fluctuations in the early stages of training, and the loss after the 4th training epoch was even greater than the loss at the initial time. Section (b) in Figure 3 shows the detail of the loss changing in the middle and late training stages. The loss of LsAdam–LSTM can converge to lower values more quickly, and the training process loss was lower overall than that of the Adam–LSTM. In addition, the Adam–LSTM underwent relatively drastic fluctuations after about the 270th epoch of training, which had a bad impact on the convergence of the Adam–LSTM.

The global learning rate change during the training of the two LSTM models is shown in Figure 4. The global learning of the LsAdam–LSTM increased first and then decreased and, eventually, settled around the initial value, while the global learning rate of the Adam–LSTM was a fixed value. Compared with Adam–LSTM, LsAdam–LSTM introduced gain coefficients with loss change information to regulate the global learning rate during the training process. When the loss decreases, the global learning rate should be appropriately increased, which, in turn, increases the convergence speed of the model, so the loss of LsAdam–LSTM decreased faster. The opposite occurs when the loss increases. The global learning rate should be appropriately reduced, which, in turn, prevents larger oscillations during the convergence of the model; thus, the LsAdam–LSTM had a much smoother loss descent process. In this way, the LsAdam–LSTM model formed a closed-loop feedback mechanism between the training loss and the learning rate and then accelerated the shrinkage of the model’s loss.

The performance metrics of the LsAdam–LSTM model and the Adam–LSTM model with the commonly used initial global learning rates are shown in

Table 2. Performance of LsAdam–LSTM and Adam–LSTM with different initial global learning rates.

Initial Global Learning Rate		0.005	0.01	0.02	0.03
Trained Epoch	LsAdam–LSTM	319	241	257	282
	Adam–LSTM	413	313	325	398
MSEtrain (×10−3)	LsAdam–LSTM	3.5798	3.5961	3.5872	3.5667
	Adam–LSTM	3.7459	3.7490	3.7469	3.7484
MAEtrain (×10−2)	LsAdam–LSTM	2.6517	2.6637	2.6797	2.7085
	Adam–LSTM	2.7446	2.7493	2.7335	2.7585
Rtrain	LsAdam–LSTM	0.9831	0.9830	0.9830	0.9833
	Adam–LSTM	0.9823	0.9823	0.9823	0.9823
MSEtest (×10−3)	LsAdam–LSTM	2.1560	2.1628	2.1639	2.1483
	Adam–LSTM	2.2752	2.2374	2.1990	2.2461
MAEtest (×10−2)	LsAdam–LSTM	2.6069	2.6016	2.6202	2.6323
	Adam–LSTM	2.6992	2.6215	2.6405	2.6108
Rtest	LsAdam–LSTM	0.9903	0.9903	0.9903	0.9903
	Adam–LSTM	0.9898	0.9900	0.9901	0.9898

. For every initial global learning rate, the prediction accuracy of the LsAdam–LSTM model was improved compared to the Adam–LSTM model, and the number of training epochs could be reduced by at least 68. It is shown that the LsAdam–LSTM model could converge faster while maintaining higher prediction accuracy for all the initial global learning rates. Moreover, the variation of the prediction results of the LsAdam–LSTM model on the test set was smoother under different initial global learning rates. This indicates that the proposed model is less sensitive to the initial global learning rate and has strong adaptability and stability. Meanwhile, the results on the test and training sets show that the proposed model has good generalization ability.

To evaluate the prediction accuracy, we obtained the ultra-short-term wind power prediction models using BP, SVR, Adam–LSTM and LsAdam–LSTM, respectively. The prediction results and local details of the four models on the test set are shown in Figure 5, where the prediction results of both LSTM models are those at an initial global learning rate of 0.01. It can be seen that the prediction results of BP deviate significantly from the true value in almost the entire power range, and the prediction error of SVR is largest near both ends of the power range, while the prediction results of the Adam–LSTM model are improved significantly. The results are improved further with LsAdam–LSTM.

Table 3. Comparison of the prediction performance of each model.

Dataset	Model	MSE (×10−3)	MAE (×10−2)	R
Train set	BP	9.6654	4.9652	0.9537
	SVR	7.0193	7.0999	0.9813
	Adam–LSTM	3.7490	2.7493	0.9823
	LsAdam–LSTM	3.5961	2.6637	0.9830
Test set	BP	6.5487	4.9301	0.9702
	SVR	4.4337	5.3506	0.9867
	Adam–LSTM	2.2374	2.6215	0.9900
	LsAdam–LSTM	2.1628	2.6016	0.9903

shows the prediction performance of all the above models. All of the parameters are the same as those in Figure 5. As for the MSE between the predicted values and true values, the prediction results of LsAdam–LSTM in the training set improved by 62.79%, 48.77% and 4.08% over BP, SVR and Adam–LSTM, respectively. The prediction results in the test set improved by 66.97%, 51.22% and 3.33%, respectively. The prediction accuracy of the LsAdam–LSTM also improved in terms of other evaluation metrics. It can be seen that the prediction results of the LsAdam–LSTM model outperformed the other models on both the training and test sets, which means that the model has higher prediction accuracy and a better generalization ability.

In summary, the proposed LsAdam–LSTM can predict ultra-short-term wind power more accurately and efficiently. By integrating the loss shrinkage information, the training of LSTM can actually be accelerated with better generalization performance. Compared with Adam, the global learning rate of every epoch can be tuned adaptively, which allows the learning progress to improve continuously. Compared with the traditional machine learning methods BP and SVR, the LsAdam–LSTM model has the nature of deep learning architecture, a stronger learning ability, and results that are significantly improved.

5. Conclusions

Ultra-short-term wind power prediction can predict power generation for a future period, which is conducive to the formulation of more reasonable power dispatching plans and makes the supply and demand balance of the power system more stable. Traditional machine learning methods, such as SVR and the BPNN, have difficulty in learning the deeper features of wind power sequences. In contrast, LSTM can extract time series features in a wind power series and has great potential advantages in wind power prediction.

To improve the training efficiency and prediction accuracy of the ultra-short-term wind power prediction model, a novel method named LsAdam–LSTM is proposed. By introducing a gain coefficient with loss change information during the training process, the global learning rate can be continuously tuned adaptively in every epoch according to the loss change. By taking into account the non-linear relationship between the loss change and the learning rate, the learning progress can be improved by the novel learning rate updating strategy. Thus, LsAdam–LSTM can alleviate the limitation caused by the small fixed global learning rate and effectively improve convergence speed and prediction accuracy. The experimental results on a wind turbine showed that the prediction accuracy of the LsAdam–LSTM is higher than that of Adam–LSTM, BP and SVR. In addition, fewer training epochs means that the proposed method has better efficiency, which provides an efficient method for the ultra-short-term prediction of wind power.