# Using Different ML Algorithms and Hyperparameter Optimization to Predict Heat Meters’ Failures

^{1}

^{2}

^{*}

## Abstract

**:**

## Featured Application

**Design of advanced heating systems for smart buildings and optimization of stocks and maintenance processes in existing heat meter networks.**

## Abstract

## 1. Introduction

_{2}emissions in the European Union. Around 40% of the consumption of final energy is spent on heating and cooling, from which most is used in buildings [1]. This sector is treated as pivotal in accelerating the reduction of emissions of the energy system. It is also a strategic industry in the context of energy safety, since according to the forecasts, till 2030, heating and cooling will account for about 40% of the consumption of renewable energy sources.

## 2. Materials and Methods

#### 2.1. Heat Meters

#### 2.1.1. Maintenance of Heat Meters

#### 2.1.2. Review of Typical Faults

- Failure of the flow transducer (in a mechanical meter) caused by the accumulation of deposits on mechanical elements. The repair involves exchanging the rotor or the whole transducer.
- Failure of temperature meter: Most frequently, this occurs in non-mechanical heat meters and is usually caused by rodents (rats) damaging the conductors. The repair is based on the replacement of the temperature meter.
- Exhaustion of the battery: Despite the theoretical calculations that mounted batteries should work for 5–7 years (exceeding the required verification period), it frequently happens that the batteries run out already after 2–3 years. The repair of such equipment requires only the replacement of the battery.

#### 2.2. Data

- bus: meters regularly send their updates to the central panel installed in the same building, which also collects data from other meters by cable connection;
- funk: similar to the bus, but the connection of the meter with the control panel does not require the additional wiring system;
- walk by: on specific days (programmed), the meter sends data, which have to be collected by the technician sent to the neighborhood and equipped with the receiving device;
- without a module: the reading has to be done manually directly on the meter.

#### 2.3. Data Analysis

- Python 3.6.3;
- Keras 2.2.0: open source library for creating neural networks;
- TensorFlow 1.8.0: open source library written by the Google Brain Team for linear algebra and neural networks;
- Scikit-learn 0.19.1: open source library implementing many different methods of machine learning.

#### 2.4. Machine Learning Algorithms

## 3. Results and Discussion

#### 3.1. Statistical Analysis

#### 3.1.1. Communication Type

#### 3.1.2. Producer

#### 3.2. Reliability Analysis

#### 3.3. Machine Learning

#### 3.3.1. Metrics

#### 3.3.2. Hyperparameter Optimization

#### 3.4. Ensemble Model

## 4. Conclusions

- The intensity of failures of heat meters was almost independent of the operating time.
- Due to the high reliability of the meters and their considerable cost, condition-based or predictive maintenance of heat meters was justified and possible. It was shown in Section 3.2 that the mandatory five-year verification period could be extended.
- Collecting different parameters about devices in use made sense even if they are not required at the moment. This allowed building more general and better ML models in the future.
- Heat meter data allowed building a ranking list of their producers and optimize deliveries.
- Heat meters with remote communication were twice as reliable as the ones with manual reading.

## Supplementary Materials

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## Abbreviations

ANN | Artificial Neural Network |

AMI | Advanced Metering Infrastructure |

AUC | Area Under the ROC Curve |

BDT | Bagging Decision Tree |

EI | Expected Improvement |

GP | Gaussian Process |

IoT | Internet of Things |

LDA | Linear Discriminant Analysis |

MCC | Matthews Correlation Coefficient |

MDPI | Multidisciplinary Digital Publishing Institute |

ML | Machine Learning |

NILM | Non-Intrusive Load Monitoring |

PCA | Principal Component Analysis |

RBF | radial basis function |

RFR | Random Forest Regressions |

ROC | Receiver Operating Characteristic |

SEM | Smart Electricity Meters |

SMBO | Sequential Model-Based Optimization |

SVM | Support Vector Machine |

TPE | Tree-structured Parzen Estimator |

## References

- European Commission. Energy Efficiency. 2019. Available online: https://ec.europa.eu/energy/en/topics/energy-efficiency/energy-performance-of-buildings (accessed on 2 May 2019).
- Gungor, V.C.; Lu, B.; Hancke, G.P. Opportunities and challenges of wireless sensor networks in smart grid. IEEE Trans. Ind. Electron.
**2010**, 57, 3557–3564. [Google Scholar] [CrossRef] - Kabalci, Y. A survey on smart metering and smart grid communication. Renew. Sustain. Energy Rev.
**2016**, 57, 302–318. [Google Scholar] [CrossRef] - Khairalla, M.; Abdrabou, A.; Dahir, A.; Gaouda, A. Fault location based on smart meters time synchronized measurements. In Proceedings of the 2015 First Workshop on Smart Grid and Renewable Energy (SGRE), Doha, Qatar, 22–23 March 2015; pp. 1–6. [Google Scholar] [CrossRef]
- Shivakumar, A.; Pye, S.; Anjo, J.; Miller, M.; Rouelle, P.B.; Densing, M.; Kober, T. Smart energy solutions in the EU: State of play and measuring progress. Energy Strategy Rev.
**2018**, 20, 133–149. [Google Scholar] [CrossRef] - The OMS-Group e.V. Open Metering System. Available online: https://oms-group.org/en/ (accessed on 2 May 2019).
- European Commission. Directive 2012/27/EU of the European Parliament and of the Council. 2012. Available online: https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32012L0027 (accessed on 5 April 2019).
- European Commission. Directive (EU) 2018/2002 of the European Parliament and of the Council. 2018. Available online: https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32018L2002 (accessed on 2 June 2019).
- Zhang, G.; Wang, G.G.; Farhangi, H.; Palizban, A. Data mining of smart meters for load category based disaggregation of residential power consumption. Sustain. Energy Grids Netw.
**2017**, 10, 92–103. [Google Scholar] [CrossRef] - Makonin, S.; Wang, Z.J.; Tumpach, C. RAE: The rainforest automation energy dataset for smart grid meter data analysis. Data
**2018**, 3, 8. [Google Scholar] [CrossRef] - Yang, Z.; Chen, Y.; Li, Y.F.; Zio, E.; Kang, R. Smart electricity meter reliability prediction based on accelerated degradation testing and modeling. Int. J. Electri. Power Energy Syst.
**2014**, 56, 209–219. [Google Scholar] [CrossRef] - Alahakoon, D.; Yu, X. Advanced analytics for harnessing the power of smart meter big data. In Proceedings of the 2013 IEEE International Workshop on Inteligent Energy Systems (IWIES), Vienna, Austria, 14 November 2013; pp. 40–45. [Google Scholar]
- Alahakoon, D.; Yu, X. Smart electricity meter data intelligence for future energy systems: A survey. IEEE Trans. Ind. Inform.
**2015**, 12, 425–436. [Google Scholar] [CrossRef] - Lee, J.Y.; Yeo, M.S.; Kim, K.W. Predictive control of the radiant floor heating system in apartment buildings. J. Asian Arch. Build. Eng.
**2002**, 1, 105–112. [Google Scholar] [CrossRef] - Javed, A.; Larijani, H.; Ahmadinia, A.; Emmanuel, R. Comparison of the robustness of RNN, MPC and ANN controller for residential heating system. In Proceedings of the 2014 IEEE Fourth International Conference on Big Data and Cloud Computing, Sydney, NSW, Australia, 3–5 December 2014; pp. 604–611. [Google Scholar]
- Yang, I.H.; Yeo, M.S.; Kim, K.W. Application of artificial neural network to predict the optimal start time for heating system in building. Energy Convers. Manag.
**2003**, 44, 2791–2809. [Google Scholar] [CrossRef] - Keçebaş, A.; Alkan, M.A.; Yabanova, İ.; Yumurtacı, M. Energetic and economic evaluations of geothermal district heating systems by using ANN. Energy Policy
**2013**, 56, 558–567. [Google Scholar] [CrossRef] - Sajjadi, S.; Shamshirband, S.; Alizamir, M.; Yee, L.; Mansor, Z.; Manaf, A.A.; Altameem, T.A.; Mostafaeipour, A. Extreme learning machine for prediction of heat load in district heating systems. Energy Build.
**2016**, 122, 222–227. [Google Scholar] [CrossRef] - Bergstra, J.; Yamins, D.; Cox, D.D. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In Proceedings of the 30th International Conference on Machine Learning, Atlanta, GA, USA, 16–21 June 2013. [Google Scholar]
- Bergstra, J.S.; Bardenet, R.; Bengio, Y.; Kégl, B. Algorithms for hyper-parameter optimization. In Proceedings of the NIPS 2011, Granada, Spain, 12–17 December 2011; pp. 2546–2554. [Google Scholar]
- Polikar, R. Ensemble learning. In Ensemble Machine Learning; Springer: Berlin/Heidelberg, Germany, 2012; pp. 1–34. [Google Scholar]
- Pałasz, P.; Żurek, J.; Przysowa, R. Markov reliability model for heat meters. J. KONBiN
**2018**, 45, 83–96. [Google Scholar] [CrossRef] - Witten, I.H.; Frank, E.; Hall, M.A.; Pal, C.J. Data Mining: Practical Machine Learning Tools and Techniques; Morgan Kaufmann: Burlington, MA, USA, 2016; Chapter 4. [Google Scholar]
- Gudivada, V.; Apon, A.; Ding, J. Data quality considerations for big data and machine learning: Going beyond data cleaning and transformations. Int. J. Adv. Softw.
**2017**, 10, 1–20. [Google Scholar] - Hall, M.A. Correlation-based feature selection of discrete and numeric class machine learning. In Proceedings of the Seventeenth International Conference on Machine Learning, Stanford, CA, USA, 29 June–2 July 2000; pp. 359–366. [Google Scholar]
- Goodfellow, I.; Bengio, Y.; Courville, A.; Bengio, Y. Deep Learning; MIT Press: Cambridge, MA, USA, 2016; Volume 1. [Google Scholar]
- Pérez, R.; Vásquez, C. Fault location in distribution systems with distributed generation using support vector machines and smart meters. In Proceedings of the 2016 IEEE Ecuador Technical Chapters Meeting (ETCM), Guayaquil, Ecuador, 12–14 October 2016; pp. 1–6. [Google Scholar] [CrossRef]
- Quinlan, J.R. Bagging, boosting, and C4.5. In AAAI/IAAI; American Association for Artificial Intelligence: Menlo Park, CA, USA, 1996; Volume 1, pp. 725–730. [Google Scholar]
- Sapna, S.; Tamilarasi, A.; Kumar, M.P. Backpropagation learning algorithm based on Levenberg Marquardt Algorithm. Comp. Sci. Inform. Technol.
**2012**, 2, 393–398. [Google Scholar] - Zhang, D.; Zhou, X.; Leung, S.C.; Zheng, J. Vertical bagging decision trees model for credit scoring. Expert Syst. Appl.
**2010**, 37, 7838–7843. [Google Scholar] [CrossRef] - Johnson, R.A.; Wichern, D.W. Applied Multivariate Statistical Analysis, 6th ed.; Prentice Hall: Upper Saddle River, NJ, USA, 2007. [Google Scholar]
- Biały, J.; Rurak, A.; Tomaszewska, J.; Toruń, A.; Woch, M.; Zieja, M.; Żurek, J. Statistical Analysis of the Volume of Passenger Air Traffic in Europe in years 2004–2015. Transp. Res. Procedia
**2018**, 35, 72–79. [Google Scholar] [CrossRef] - Martinez, W.L.; Martinez, A.R.; Solka, J. Exploratory Data Analysis with MATLAB; CRC Press: Boca Raton, FL, USA, 2017. [Google Scholar]
- Bradley, E. Reliability Engineering A Life Cycle Approach; CRC Press: Boca Raton, FL, USA, 2017. [Google Scholar]
- He Ren, X.C.; Chen, Y. Reliability Based Aircraft Maintenance Optimization and Application; Aerospace Engineering; Elsevier: Amsterdam, The Netherlands, 2017. [Google Scholar]
- Rykov, V. Reliability of Engineering Systems and Technological Risks; John Wiley & Sons, Inc.: Hoboken, NJ, USA, 2016. [Google Scholar] [CrossRef]
- Murphy, K.P. Machine Learning: A Probabilistic Perspective; Number 12 in Adaptive Computation and Machine Learning Series; MIT Press: London, UK, 2013. [Google Scholar]
- Domingos, P. The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World; Basic Books: New York, NY, USA, 2015; p. 354. [Google Scholar]
- Davis, J.; Goadrich, M. The relationship between Precision-Recall and ROC curves. In Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, PA, USA, 25–29 June 2006; pp. 233–240. [Google Scholar] [CrossRef]
- Bergstra, J.; Bengio, Y. Random search for hyper-parameter optimization. J. Mach. Learn. Res.
**2012**, 13, 281–305. [Google Scholar] - Snoek, J.; Larochelle, H.; Adams, R.P. Practical bayesian optimization of machine learning algorithms. In Proceedings of the NIPS 2012, Stateline, NV, USA, 3–8 December 2012; pp. 2951–2959. [Google Scholar]
- Zhang, C.; Ma, Y. Ensemble Machine Learning: Methods and Applications; Springer: Berlin/Heidelberg, Germany, 2012. [Google Scholar]
- Ferrero Bermejo, J.; Gómez Fernández, J.F.; Olivencia Polo, F.; Crespo Márquez, A. A Review of the Use of Artificial Neural Network Models for Energy and Reliability Prediction. A Study of the Solar PV, Hydraulic and Wind Energy Sources. Appl. Sci.
**2019**, 9, 1844. [Google Scholar] [CrossRef] - Shamshirband, S.; Petković, D.; Enayatifar, R.; Abdullah, A.H.; Marković, D.; Lee, M.; Ahmad, R. Heat load prediction in district heating systems with adaptive neuro-fuzzy method. Renew. Sustain. Energy Rev.
**2015**, 48, 760–767. [Google Scholar] [CrossRef]

Name | Description | Type |
---|---|---|

Age | Life of the meter in months from the moment of installation to the last readout or failure | $\mathbb{N}$ number |

ZIP | Postal code of flat/office, in which the meter is installed | $\mathbb{N}$ number |

Floor | Floor where the meter is installed | $\mathbb{N}$ number |

FlatType | Type of usable area | String of letters |

RoomType | Type of area | String of letters |

AccConsumption | Consumption from the moment of registration in the system | $\mathbb{R}$ Number |

CurrentConsumption | Consumption in the last settlement period | $\mathbb{R}$ Number |

CommType | Type of communication with the meter | String of letters |

Producer | Name of the manufacturer of the meter | String of letters |

RatingFactor | Equalization factor | $\mathbb{R}$ number |

BillingNo | Next number of the settlement period | $\mathbb{N}$ number |

CurrentValue | Last recorded value of the meter | $\mathbb{R}$ number |

AvgConsumption | Average consumption in all settlement periods | $\mathbb{R}$ Number |

MaxConsumption | Maximum consumption in the settlement period | $\mathbb{R}$ Number |

MinConsumption | Minimum consumption in the settlement period | $\mathbb{R}$ Number |

CalculatedAge | Calculated age of the meter in months: CurrentValue/AvgConsumption | $\mathbb{R}$ number |

Communication Type | Meter # | Failure # | % of Failures |
---|---|---|---|

Walk by | 1045 | 18 | 1.72% |

Funk | 10,996 | 1716 | 15.61% |

Bus | 12,631 | 1778 | 14.08% |

Without communication | 27,577 | 8273 | 30.00% |

Producer | Meter # | Failure # | % of Failure |
---|---|---|---|

1 | 2555 | 290 | 11.35% |

2 | 118 | 114 | 96.61% |

3 | 1674 | 281 | 16.79% |

4 | 671 | 303 | 45.16% |

5 | 713 | 187 | 26.23% |

6 | 823 | 345 | 41.92% |

7 | 1708 | 438 | 25.64% |

8 | 1494 | 603 | 40.36% |

9 | 1832 | 443 | 24.18% |

10 | 2026 | 332 | 16.39% |

11 | 1853 | 600 | 32.38% |

12 | 2440 | 234 | 9.59% |

13 | 3348 | 1859 | 55.53% |

14 | 3506 | 655 | 18.68% |

15 | 3977 | 2059 | 51.77% |

16 | 23,511 | 3042 | 12.94% |

Metric | Result |
---|---|

MCC-S | 97.34% |

AUC | 96.45% |

$f{1}_{avg}$ | 98.22% |

$f{1}_{1}$ | 95.81% |

$f{1}_{0}$ | 98.82% |

$Recal{l}_{avg}$ | 98.32% |

$Recal{l}_{1}$ | 93.34% |

$Recal{l}_{0}$ | 99.56% |

$Precisio{n}_{avg}$ | 98.15% |

$Precisio{n}_{1}$ | 98.42% |

$Precisio{n}_{0}$ | 98.08% |

Accuracy | 98.15% |

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Pałasz, P.; Przysowa, R.
Using Different ML Algorithms and Hyperparameter Optimization to Predict Heat Meters’ Failures. *Appl. Sci.* **2019**, *9*, 3719.
https://doi.org/10.3390/app9183719

**AMA Style**

Pałasz P, Przysowa R.
Using Different ML Algorithms and Hyperparameter Optimization to Predict Heat Meters’ Failures. *Applied Sciences*. 2019; 9(18):3719.
https://doi.org/10.3390/app9183719

**Chicago/Turabian Style**

Pałasz, Przemysław, and Radosław Przysowa.
2019. "Using Different ML Algorithms and Hyperparameter Optimization to Predict Heat Meters’ Failures" *Applied Sciences* 9, no. 18: 3719.
https://doi.org/10.3390/app9183719