Deep Reinforcement Learning for the Detection of Abnormal Data in Smart Meters
Abstract
:1. Introduction
- (1)
- In view of the problem of massive, diverse, and complex factors influencing electricity consumption data from electricity meters, this paper proposes an abnormal data detection model for smart meters based on Deep Reinforcement Learning.
- (2)
- In this paper, the FCM algorithm is used to realize semi-supervised learning in the DQN network, to predict the sample state in the next moment through the FCM algorithm, and then to predict its Q value through the target network.
- (3)
- The method proposed in this paper is analyzed and tested on a real user power consumption dataset. The proposed method can significantly improve detection accuracy and speed, and the model shows strong generalization ability and applicability.
2. Related Work
2.1. Smart Grid Security Technologies
2.2. Abnormal Detection Method of Smart Meter Data
2.3. Deep Reinforcement Learning
3. Method
3.1. Overview of the Deep Q-Network Model
Algorithm 1 Q-learning algorithm. |
|
3.2. Model Design
4. Experiments
4.1. Dataset
4.2. Experimental Environment and Configuration
4.3. Evaluation Metrics
4.4. Experimental Results and Analysis
4.4.1. Comparison with Existing Methods
4.4.2. Parametric Analysis
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Suthaharan, S. Machine Learning Models and Algorithms for Big Data Classification; Integrated Series in Information Systems; Springer: New York, NY, USA, 2016; Volume 36, pp. 1–12. [Google Scholar]
- Wiering, M.A.; Van Otterlo, M. Reinforcement learning. Adapt. Learn. Optim. 2012, 12, 729. [Google Scholar]
- Mousavi, S.S.; Schukat, M.; Howley, E. Deep reinforcement learning: An overview. In Proceedings of the SAI Intelligent Systems Conference, London, UK, 21–22 September 2016; pp. 426–440. [Google Scholar]
- Arulkumaran, K.; Deisenroth, M.P.; Brundage, M.; Bharath, A.A. Deep reinforcement learning: A brief survey. IEEE Signal Process. Mag. 2017, 34, 26–38. [Google Scholar] [CrossRef] [Green Version]
- François-Lavet, V.; Henderson, P.; Islam, R.; Bellemare, M.G.; Pineau, J. An introduction to deep reinforcement learning. Found. Trends® Mach. Learn. 2018, 11, 219–354. [Google Scholar] [CrossRef] [Green Version]
- Chen, L. Deep Learning and Practice with MindSpore; Springer Nature: Berlin, Germany, 2021. [Google Scholar]
- Tan, S.; De, D.; Song, W.Z.; Yang, J.; Das, S.K. Survey of Security Advances in Smart Grid: A Data Driven Approach. IEEE Commun. Surv. Tutor. 2017, 19, 397–422. [Google Scholar]
- Li, F.; Bo, L.; Peng, L. Secure Information Aggregation for Smart Grids Using Homomorphic Encryption. In Proceedings of the 2010 First IEEE International Conference on Smart Grid Communications (SmartGridComm), Gaithersburg, MD, USA, 4–6 October 2010. [Google Scholar]
- Zhang, Y.; Wang, L.; Sun, W.; Robert, I.I.; Alam, M. Distributed Intrusion Detection System in a Multi-Layer Network Architecture of Smart Grids. IEEE Trans. Smart Grid 2012, 2, 796–808. [Google Scholar] [CrossRef] [Green Version]
- Jokar, P.; Arianpoo, N.; Leung, V.C. Electricity theft detection in AMI using customers’ consumption patterns. IEEE Trans. Smart Grid 2015, 7, 216–226. [Google Scholar] [CrossRef]
- Costa, B.C.; Alberto, B.L.; Portela, A.M.; Maduro, W.; Eler, E.O. Fraud detection in electric power distribution networks using an ann-based knowledge-discovery process. Int. J. Artif. Intell. Appl. 2013, 4, 17. [Google Scholar] [CrossRef]
- Jamil, F. On the electricity shortage, price and electricity theft nexus. Energy Policy 2013, 54, 267–272. [Google Scholar] [CrossRef]
- León, C.; Biscarri, F.; Monedero, I.; Guerrero, J.I.; Biscarri, J.; Millán, R. Variability and trend-based generalized rule induction model to NTL detection in power companies. IEEE Trans. Power Syst. 2011, 26, 1798–1807. [Google Scholar] [CrossRef]
- Fontugne, R.; Tremblay, N.; Borgnat, P.; Flandrin, P.; Esaki, H. Mining anomalous electricity consumption using ensemble empirical mode decomposition. In Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, Canada, 26–31 May 2013; pp. 5238–5242. [Google Scholar]
- Nagi, J.; Yap, K.S.; Tiong, S.K.; Ahmed, S.K.; Nagi, F. Improving SVM-based nontechnical loss detection in power utility using the fuzzy inference system. IEEE Trans. Power Deliv. 2011, 26, 1284–1285. [Google Scholar]
- Cheng, C.; Zhang, H.; Jing, Z.; Chen, M.; Jiao, L.; Yang, L. Study on the anti-electricity stealing based on outlier algorithm and the electricity information acquisition system. Power Syst. Prot. Control 2015, 43, 69–74. [Google Scholar]
- Mnih, V.; Kavukcuoglu, K.; Silver, D.; Graves, A.; Antonoglou, I.; Wierstra, D.; Riedmiller, M. Playing atari with deep reinforcement learning. arXiv 2013, arXiv:1312.5602. [Google Scholar]
- Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A.A.; Veness, J.; Bellemare, M.G.; Graves, A.; Riedmiller, M.; Fidjeland, A.K.; Ostrovski, G.; et al. Human-level control through deep reinforcement learning. Nature 2015, 518, 529–533. [Google Scholar] [CrossRef] [PubMed]
- Liu, Q.; Zhai, J.W.; Zhang, Z.Z.; Zhong, S.; Zhou, Q.; Zhang, P.; Xu, J. A survey on deep reinforcement learning. Chin. J. Comput. 2018, 41, 1–27. [Google Scholar]
- Silver, D.; Lever, G.; Heess, N.; Degris, T.; Wierstra, D.; Riedmiller, M. Deterministic policy gradient algorithms. In Proceedings of the International Conference on Machine Learning, Beijing, China, 21–26 June 2014; pp. 387–395. [Google Scholar]
- Lillicrap, T.P.; Hunt, J.J.; Pritzel, A.; Heess, N.; Erez, T.; Tassa, Y.; Silver, D.; Wierstra, D. Continuous control with deep reinforcement learning. arXiv 2015, arXiv:1509.02971. [Google Scholar]
- Silver, D.; Huang, A.; Maddison, C.J.; Guez, A.; Sifre, L.; Van Den Driessche, G.; Schrittwieser, J.; Antonoglou, I.; Panneershelvam, V.; Lanctot, M.; et al. Mastering the game of Go with deep neural networks and tree search. Nature 2016, 529, 484–489. [Google Scholar] [CrossRef]
- Browne, C.B.; Powley, E.; Whitehouse, D.; Lucas, S.M.; Cowling, P.I.; Rohlfshagen, P.; Tavener, S.; Perez, D.; Samothrakis, S.; Colton, S. A survey of monte carlo tree search methods. IEEE Trans. Comput. Intell. AI Games 2012, 4, 1–43. [Google Scholar] [CrossRef]
- Clifton, J.; Laber, E. Q-learning: Theory and applications. Ann. Rev. Stat. Appl. 2020, 7, 279–301. [Google Scholar] [CrossRef] [Green Version]
- Naseer, S.; Saleem, Y.; Khalid, S.; Bashir, M.K.; Han, J.; Iqbal, M.M.; Han, K. Enhanced network anomaly detection based on deep neural networks. IEEE Access 2018, 6, 48231–48246. [Google Scholar] [CrossRef]
- Sethi, K.; Sai Rupesh, E.; Kumar, R.; Bera, P.; Venu Madhav, Y. A context-aware robust intrusion detection system: A reinforcement learning-based approach. Int. J. Inf. Secur. 2020, 19, 657–678. [Google Scholar] [CrossRef]
- Mahfouz, A.M.; Venugopal, D.; Shiva, S.G. Comparative analysis of ML classifiers for network intrusion detection. In Fourth International Congress on Information and Communication Technology; Springer: Singapore, 2020; pp. 193–207. [Google Scholar]
- Jaiswal, R.; Chakravorty, A.; Rong, C. Distributed fog computing architecture for real-time anomaly detection in smart meter data. In Proceedings of the 2020 IEEE Sixth International Conference on Big Data Computing Service and Applications (BigDataService), Oxford, UK, 3–6 August 2020; pp. 1–8. [Google Scholar]
- Zhang, Y.; Ai, Q.; Wang, H.; Li, Z.; Zhou, X. Energy theft detection in an edge data center using threshold-based abnormality detector. Int. J. Electr. Power Energy Syst. 2020, 121, 106162. [Google Scholar] [CrossRef]
- Gu, J.; Wang, Z.; Kuen, J.; Ma, L.; Shahroudy, A.; Shuai, B.; Liu, T.; Wang, X.; Wang, G.; Cai, J.; et al. Recent advances in convolutional neural networks. Pattern Recognit. 2018, 77, 354–377. [Google Scholar] [CrossRef]
- Graves, A. Long short-term memory. In Supervised Sequence Labelling with Recurrent Neural Networks; Springer: Berlin/Heidelberg, Germany, 2012; pp. 37–45. [Google Scholar]
- Lea, C.; Flynn, M.D.; Vidal, R.; Reiter, A.; Hager, G.D. Temporal convolutional networks for action segmentation and detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21– 26 July 2017; pp. 156–165. [Google Scholar]
User Information | Value |
---|---|
Time range | 1 January 2014–7 February 2017 |
Total number of samples | 149,186 |
Number of normal samples | 140,434 |
Number of abnormal samples | 8752 |
Number of training samples | 89,500 |
Number of validation samples | 29,843 |
Number of test samples | 29,843 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Sun, S.; Liu, C.; Zhu, Y.; He, H.; Xiao, S.; Wen, J. Deep Reinforcement Learning for the Detection of Abnormal Data in Smart Meters. Sensors 2022, 22, 8543. https://doi.org/10.3390/s22218543
Sun S, Liu C, Zhu Y, He H, Xiao S, Wen J. Deep Reinforcement Learning for the Detection of Abnormal Data in Smart Meters. Sensors. 2022; 22(21):8543. https://doi.org/10.3390/s22218543
Chicago/Turabian StyleSun, Shuxian, Chunyu Liu, Yiqun Zhu, Haihang He, Shuai Xiao, and Jiabao Wen. 2022. "Deep Reinforcement Learning for the Detection of Abnormal Data in Smart Meters" Sensors 22, no. 21: 8543. https://doi.org/10.3390/s22218543
APA StyleSun, S., Liu, C., Zhu, Y., He, H., Xiao, S., & Wen, J. (2022). Deep Reinforcement Learning for the Detection of Abnormal Data in Smart Meters. Sensors, 22(21), 8543. https://doi.org/10.3390/s22218543