#
Short-Term Load Forecasting of Natural Gas with Deep Neural Network Regression ^{†}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Overview of Natural Gas Forecasting

_{ref}is the reference temperature [3]. Reference temperature is indicated by concatenating it to HDD, i.e., HDD65 indicates a reference temperature of 65 °F.

_{ref}as seen in Figure 1.

## 3. Prior Work

_{0}is the natural gas load not dependent on temperature. The natural gas load dependent on temperature is captured by the sum of β

_{1}and β

_{2}. The two reference temperatures better model the smooth transition from heating to non-heating days. β

_{3}accounts for recency effects [5,6]. Finally, β

_{4}models small, but not insignificant, temperature effects during non-heating days.

## 4. Artificial and Deep Neural Networks

_{i}represent the ith input to the node of a neural network, w

_{i}the weight of the ith input, b the bias term, n the number of inputs, and o the output of the node. Then

_{i}represent the ith visible node, w

_{i}the weight of the ith visible node, c the bias term, n the number of visible nodes, and h the hidden node.

_{j}represent the jth hidden node, w

_{j}the weight of the jth hidden node, b the bias term, m the number of hidden nodes, and v the visible node. Then

**W**

^{T}is the transpose of

**W**.

**v**

_{0}and a training rate ε. Algorithm 1 is performed on iterations (epochs) of all input vectors.

Algorithm 1: Training restricted Boltzmann machines using contrastive divergence | |

1 | //Positive Phase |

2 | h_{0} = σ (Wv_{0} + c) |

3 | for each hidden unit h_{0i}: |

4 | if h_{0i} > rand(0,1)//rand(0,1) represents a sample drawn from the uniform distribution |

5 | h_{0i} = 1 |

6 | else |

7 | h_{0i} = 0 |

8 | //Negative Phase |

9 | v_{1} = σ (W^{T}h_{0} + b) |

10 | for each visible units v_{1j}: |

11 | if v_{1j} > rand(0,1) |

12 | v_{1j} = 1 |

13 | else |

14 | v_{1j} = 0 |

15 | //Update Phase |

16 | h_{1} = σ (Wv_{1} + c) |

17 | W = ε (h_{0}v_{0}^{T} − h_{1}v_{1}^{T}) |

18 | b = ε (h_{0} − h_{1}) |

19 | c = ε (v_{0} − v_{1}) |

## 5. Data

## 6. Methods

## 7. Results

^{−7}and 6.4 × 10

^{−4}, respectively, meaning that the DNN performed better, in general, than the ANN or LR, and that the difference in performance is statistically significant in both cases.

^{−5}. This means that the Large DNN offers a statistically significant better performance over the 62 areas than the small DNN. However, much like in the comparison between the DNN and other models, the small DNN performs better in some areas, which supports the earlier claim that complex models do not necessarily outperform simpler ones.

## 8. Conclusions

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## Nomenclature

b | bias term of a neural network node |

c | the bias term of a restricted Boltzmann machine (RBM) |

CDD | cooling degree days |

DPT | dew point |

Dth | dekatherm |

h | vector of hidden nodes of a RBM |

HDD | heating degree days |

h_{j} | jth hidden node of a RBM |

MAPE | mean absolute error |

o | output of a neural network node |

RMSE | root mean square error |

s | natural gas demand |

T | temperature in degrees Fahrenheit |

T_{ref} | reference temperature for HDD and CDD |

v | vector of visible nodes of a RBM |

v_{i} | ith visible node of a RBM |

W | weight matrix of a neural network |

w_{i} | weight of the ith input of a neural network node |

WMAPE | weighted mean absolute percentage error |

x_{i} | ith input to the node of a neural network |

## References

- Natural Gas Explained. Available online: https://www.eia.gov/energyexplained/index.php?page=natural_gas_home (accessed on 24 July 2018).
- Merkel, G.D.; Povinelli, R.J.; Brown, R.H. Deep neural network regression for short-term load forecasting of natural gas. In Proceedings of the International Symposium on Forecasting, Cairns, Australia, 25–28 June 2015; p. 90. [Google Scholar]
- Asbury, J.G.; Maslowski, C.; Mueller, R.O. Solar Availability for Winter Space Heating: An Analysis of the Calendar Period, 1953–1975; Argonne National Laboratory: Argonne, IL, USA, 1979.
- Dahl, M.; Brun, A.; Kirsebom, O.; Andresen, G. Improving short-term heat load forecasts with calendar and holiday data. Energies
**2018**, 11, 1678. [Google Scholar] [CrossRef] - Vitullo, S.R.; Brown, R.H.; Corliss, G.F.; Marx, B.M. Mathematical models for natural gas forecasting. Can. Appl. Math. Q.
**2009**, 17, 807–827. [Google Scholar] - Ishola, B. Improving Gas Demand Forecast during Extreme Cold Events. Master’s Thesis, Marquette University, Milwaukee, WI, USA, 2016. [Google Scholar]
- Haida, T.; Muto, S. Regression based peak load forecasting using a transformation technique. IEEE Trans. Power Syst.
**1994**, 9, 1788–1794. [Google Scholar] [CrossRef] - Hornik, K.; Stinchcombe, M.; White, H. Multilayer feedforward networks are universal approximators. Neural Netw.
**1989**, 2, 359–366. [Google Scholar] [CrossRef] - Park, D.C.; El-Sharkawi, M.A.; Marks, R.J.; Atlas, L.E.; Damborg, M.J. Electric load forecasting using an artificial neural network. IEEE Trans. Power Syst.
**1991**, 6, 442–449. [Google Scholar] [CrossRef] [Green Version] - Längkvist, M.; Karlsson, L.; Loutfi, A. A review of unsupervised feature learning for time series modeling. Pattern Recognit. Lett.
**2014**, 42, 11–24. [Google Scholar] [CrossRef] - Szegedy, C.; Liu, W.; Jia, Y.; Sermanet, P.; Reed, S.; Anguelov, D.; Erhan, D.; Vanhoucke, V.; Rabinovich, A. Going deeper with convolutions. In Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 7–12 June 2015; pp. 1–9. [Google Scholar]
- Hinton, G.E.; Hinton, G.E.; Osindero, S.; Osindero, S.; Teh, Y.W.; Teh, Y.W. A fast learning algorithm for deep belief nets. Neural Comput.
**2006**, 18, 1527–1554. [Google Scholar] [CrossRef] [PubMed] - Qiu, X.; Zhang, L.; Ren, Y.; Suganthan, P.N.; Amaratunga, G. Ensemble deep learning for regression and time series forecasting. In Proceedings of the 2014 IEEE Symposium on Computational Intelligence in Ensemble Learning, Orlando, FL, USA, 9–12 December 2014; pp. 1–6. [Google Scholar] [CrossRef]
- Busseti, E.; Osband, I.; Wong, S. Deep Learning for Time Series Modeling; Stanford University: Stanford, CA, USA, 2012. [Google Scholar]
- Dalto, M.; Matusko, J.; Vasak, M. Deep neural networks for time series prediction with applications in ultra-short-term wind forecasting. In Proceedings of the IEEE International Conference on Industrial Technology (ICIT), Seville, Spain, 17–19 March 2015; pp. 1657–1663. [Google Scholar]
- Kuremoto, T.; Kimura, S.; Kobayashi, K.; Obayashi, M. Time series forecasting using a deep belief network with restricted Boltzmann machines. Neurocomputing
**2014**, 137, 47–56. [Google Scholar] [CrossRef] - Glorot, X.; Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Sardinia, Italy, 13–15 May 2010; Teh, Y.W., Titterington, M., Eds.; PMLR: London, UK, 2010; pp. 249–256. [Google Scholar]
- Kennedy, J.; Eberhart, R. Particle swarm optimization. In Proceedings of the IEEE International Conference on Neural Networks, Perth, Australia, 27 November–1 December 1995; Volume 4, pp. 1942–1948. [Google Scholar]
- Ryu, S.; Noh, J.; Kim, H. Deep neural network based demand side short term load forecasting. Energies
**2017**, 10, 3. [Google Scholar] [CrossRef] - Kuo, P.-H.; Huang, C.-J. A high precision artificial neural networks model for short-term energy load forecasting. Energies
**2018**, 11, 213. [Google Scholar] [CrossRef] - Li, C.; Ding, Z.; Yi, J.; Lv, Y.; Zhang, G. Deep belief network based hybrid model for building energy consumption prediction. Energies
**2018**, 11, 242. [Google Scholar] [CrossRef] - Chen, K.; Chen, K.; Wang, Q.; He, Z.; Hu, J.; He, J. Short-term load forecasting with deep residual networks. IEEE Trans. Smart Grid
**2018**. [Google Scholar] [CrossRef] - Hosein, S.; Hosein, P. Load forecasting using deep neural networks. In Proceedings of the IEEE Power & Energy Society Innovative Smart Grid Technologies Conference (ISGT), Washington, DC, USA, 23–26 April 2017; pp. 1–5. [Google Scholar]
- Lin, C.-T.; Lee, C.S.G. Neural Fuzzy Systems—A Neuro-Fuzzy Synergism to Intelligent Systems; Prentice-Hall: Upper Saddle River, NJ, USA, 1996. [Google Scholar]
- Bengio, Y. Learning deep architectures for AI. Found. Trends Mach. Learn.
**2009**, 2, 1–127. [Google Scholar] [CrossRef] - Hinton, G. A practical guide to training restricted Boltzmann machines. In Neural Networks: Tricks of the Trade; Springer: Berlin/Heidelberg, Germany, 2012. [Google Scholar]
- Hansen, L.K.; Salamon, P. Neural network ensembles. IEEE Trans. Pattern Anal. Mach. Intell.
**1990**, 12, 993–1001. [Google Scholar] [CrossRef] [Green Version] - Brown, R.H.; Vitullo, S.R.; Corliss, G.F.; Adya, M.; Kaefer, P.E.; Povinelli, R.J. Detrending daily natural gas consumption series to improve short-term forecasts. In Proceedings of the IEEE Power and Energy Society General Meeting, Denver, CO, USA, 26–30 July 2015. [Google Scholar]
- Ruchti, T.L.; Brown, R.H.; Garside, J.J. Kalman based artificial neural network training algorithms for nonlinear system identification. In Proceedings of the IEEE International Symposium on Intelligent Control, Chicago, IL, USA, 25–27 August 1993; pp. 582–587. [Google Scholar]

**Figure 1.**Weighted combination of several midwestern U.S. operating areas, including Illinois, Michigan, and Wisconsin. Authors obtained data directly from local distribution companies. The data is from 1 January 2003 to 19 March 2018.

**Figure 2.**The same data as in Figure 1 colored by day of the week.

**Figure 3.**A feedforward ANN with four visible nodes, three nodes in the first hidden layer, two nodes in the second hidden layer, and a single node in the output layer.

**Figure 4.**A restricted Boltzmann machine with four visible units and three hidden units. Note the similarity with a single layer of a neural network.

**Figure 6.**This figure shows two histograms: (

**a**) A comparison of the performance of all 62 models between the DNN and the LR. Instances to the left of the center line are those for which the LR performed better, while those on the right are areas where the DNN performs better. The distance from the center line is the difference in WMAPE. (

**b**) The same as (

**a**) but comparing the ANN to the DNN. One instance (at 10.1) in (

**b**) is cut off to maintain consistent axes.

**Figure 7.**A comparison of the performance of all 62 models between the DNN and the Large DNN. Instances to the left of the center line are those for which the Large DNN performed better, while those on the right are areas where the DNN performs better. The distance from the center line is the difference in WMAPE.

LR WMAPE | ANN WMAPE | DNN WMAPE | Large DNN WMAPE | |
---|---|---|---|---|

Mean | 6.41 | 6.41 | 5.78 | 5.58 |

Standard Deviation | 2.49 | 2.83 | 2.11 | 2.09 |

© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Merkel, G.D.; Povinelli, R.J.; Brown, R.H.
Short-Term Load Forecasting of Natural Gas with Deep Neural Network Regression ^{†}. *Energies* **2018**, *11*, 2008.
https://doi.org/10.3390/en11082008

**AMA Style**

Merkel GD, Povinelli RJ, Brown RH.
Short-Term Load Forecasting of Natural Gas with Deep Neural Network Regression ^{†}. *Energies*. 2018; 11(8):2008.
https://doi.org/10.3390/en11082008

**Chicago/Turabian Style**

Merkel, Gregory D., Richard J. Povinelli, and Ronald H. Brown.
2018. "Short-Term Load Forecasting of Natural Gas with Deep Neural Network Regression ^{†}" *Energies* 11, no. 8: 2008.
https://doi.org/10.3390/en11082008