# On the Use of Concentrated Time–Frequency Representations as Input to a Deep Convolutional Neural Network: Application to Non Intrusive Load Monitoring

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. STFT-Based Time–Frequency Analysis

#### 2.1. Short-Time Fourier Transform (STFT)

#### 2.2. Reassignment and Synchrosqueezing

#### 2.3. Reassignment

#### 2.4. Synchrosqueezing

#### 2.5. Time-Reassigned Synchrosqueezing

#### 2.6. Discretization

## 3. Non-Intrusive Load Monitoring

#### 3.1. Problem Formulation

#### 3.2. Electrical Features Computed from Current and Voltage Measurements

#### 3.2.1. Electrical Features Based on Fourier Coefficients

#### 3.2.2. New Proposed STFT-Based Electrical Features

#### 3.3. Proposed CNN Architectures

- We compute the TF representation of the instantaneous power signal defined as:$$s\left[n\right]=v\left[n\right]i\left[n\right].$$The spectrogram of this signal looses information about the active and reactive powers. However, it has the advantage of producing a single real-valued matrix that can easily be processed by a classical single input CNN architecture.
- We compute the product between the voltage TF representation and the complex conjugate of the current TF representation according to Equation (27) which produces a complex matrix $X=P+jQ$. The resulting two-dimensional tensor that contains the real and the imaginary parts, can be processed with the proposed CNN architectures. Our first CNN architecture uses a two-channel model and the second one separately process the real and the imaginary part through two distinct CNNs for which their outputs are concatenated at the last layer.

#### 3.3.1. Single-Input CNN Architecture

#### 3.3.2. Two-Channel Input CNN Architecture

#### 3.3.3. Concatenated CNN Architecture

## 4. Numerical Results

#### 4.1. Materials

#### 4.2. HEA Recognition Results

#### 4.3. Relevance Analysis of the Learned Features

#### 4.3.1. Layer-Wise Relevance Propagation

#### 4.3.2. Relevance Maps

## 5. Discussion and Future Works

## Author Contributions

## Funding

## Conflicts of Interest

## Abbreviations

CNN | Convolutional Neural Network |

HEA | Home Electrical Appliance |

LRP | Layer-wise Relevance Propagation |

NILM | Non Intrusive Load Monitoring |

PLAID | Plug Load Appliance Identification Dataset |

ReLU | REctified Linear Unit |

STFT | Short-Time Fourier Transform |

TF | Time-Frequency |

## References

- Flandrin, P. Explorations in Time-Frequency Analysis; Cambridge University Press: Cambridge, UK, 2018. [Google Scholar]
- Flandrin, P. Time-Frequency/Time-Scale Analysis; Academic Press Inc.: Cambridge, MA, USA, 1998. [Google Scholar]
- Daubechies, I.; Maes, S. A nonlinear squeezing of the continuous wavelet transform. Wavelets Med. Biol.
**1996**, 527–546. [Google Scholar] [CrossRef] - Auger, F.; Flandrin, P.; Lin, Y.; McLaughlin, S.; Meignen, S.; Oberlin, T.; Wu, H. TF Reassignment and Synchrosqueezing: An Overview. IEEE Signal Process. Mag.
**2013**, 30, 32–41. [Google Scholar] [CrossRef][Green Version] - Fourer, D.; Auger, F.; Czarnecki, K.; Meignen, S.; Flandrin, P. Chirp rate and instantaneous frequency estimation: Application to recursive vertical synchrosqueezing. IEEE Signal Process. Lett.
**2017**, 24, 1724–1728. [Google Scholar] [CrossRef][Green Version] - Souriau, R.; Fourer, D.; Chen, H.; Lerbet, J.; Maaref, H.; Vigneron, V. High-Voltage Spindles detection from EEG signals using recursive synchrosqueezing transform. In Proceedings of the GRETSI’19, Lille, France, 26–29 August 2019. [Google Scholar]
- Goodfellow, I.; Bengio, Y.; Courville, A. Deep Learning; MIT Press: Cambridge, MA, USA, 2016. [Google Scholar]
- Bengio, Y.; Courville, A.; Vincent, P. Representation learning: A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell.
**2013**, 35, 1798–1828. [Google Scholar] [CrossRef] - Zoha, A.; Gluhak, A.; Imram, M.; Rajasegarar, S. Non intrusive load monitoring approaches for disaggregated energy sensing: A survey. Sensors
**2012**, 12, 16838–16866. [Google Scholar] [CrossRef][Green Version] - Hart, G.W. Non-intrusive appliance load monitoring. Proc. IEEE
**1992**, 80, 1870–1891. [Google Scholar] [CrossRef] - Faustine, A.; Mvungi, N.H.; Kaijage, S.; Michael, K. A survey on non-intrusive load monitoring methodies and techniques for energy disaggregation problem. arXiv
**2017**, arXiv:1703.00785. [Google Scholar] - Kong, W.; Dong, Z.; Ma, J.; Hill, D.J.; Zhao, J.; Luo, F. An extensible approach for non-intrusive load disaggregation with smart meter data. IEEE Trans. Smart Grid
**2018**, 9, 3362–3372. [Google Scholar] [CrossRef] - Kato, T.; Cho, H.S.; Lee, D.; Toyomura, T.; Yamazaki, T. Appliance Recognition from Electric Current Signals for Information-Energy Integrated Network in Home Environments. In Proceedings of the International Conference on Smart Homes and Health Telematics, ICOST 2009, Tours, France, 1–3 July 2009; pp. 150–157. [Google Scholar]
- Sadeghianpourhamami, N.; Ruyssinck, J.; Deschrijver, D.; Dhaene, T.; Develder, C. Comprehensive feature selection for appliance classification in NILM. Energy Build.
**2017**, 151, 98–106. [Google Scholar] [CrossRef][Green Version] - Houidi, S.; Auger, F.; Ben Attia Sethom, H.; Fourer, D.; Miègeville, L. Relevant feature selection for home appliance recognition. In Proceedings of the Electrimacs, Toulouse, France, 4–6 July 2017. [Google Scholar]
- De Baets, L.; Ruyssinck, J.; Develder, C.; Dhaene, T.; Deschrijver, D. Appliance classification using VI trajectories and convolutional neural networks. Energy Build.
**2018**, 158, 32–36. [Google Scholar] [CrossRef][Green Version] - Gao, J.; Giri, S.; Kara, E.C.; Bergés, M. PLAID: A Public Dataset of High-resolution Electrical Appliance Measurements for Load Identification Research: Demo Abstract. In Proceedings of the ACM Conference on Embedded Systems for Energy-Efficient Buildings, Memphis, TN, USA, 5–6 November 2014; pp. 198–199. [Google Scholar]
- Houidi, S.; Auger, F.; Frétaud, P.; Fourer, D.; Miègeville, L.; Sethom, H.B.A. Design of an electricity consumption measurement system for Non Intrusive Load Monitoring. In Proceedings of the 2019 10th International Renewable Energy Congress (IREC), Sousse, Tunisia, 26–28 March 2019. [Google Scholar]
- Houidi, S.; Fourer, D.; Auger, F.; Sethom, H.B.A.; Miègeville, L. Home Electrical Appliances Recognition using Relevant Features, Deep learning and Transfer Learning: A Comparative Study. Energy Build.
**2020**, submitted. [Google Scholar] - Fourer, D.; Auger, F.; Flandrin, P. Recursive versions of the Levenberg-Marquardt reassigned spectrogram and of the synchrosqueezed STFT. In Proceedings of the 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Shanghai, China, 20–25 March 2016; pp. 4880–4884. [Google Scholar]
- Auger, F.; Flandrin, P. Improving the readability of time-frequency and time-scale representations by the reassignment method. IEEE Trans. Signal Process.
**1995**, 43, 1068–1089. [Google Scholar] [CrossRef][Green Version] - Fourer, D.; Harmouche, J.; Schmitt, J.; Oberlin, T.; Meignen, S.; Auger, F.; Flandrin, P. The ASTRES Toolbox for Mode Extraction of Non-Stationary Multicomponent Signals. In Proceedings of the 2017 25th European Signal Processing Conference (EUSIPCO), Kos, Greece, 28 August–2 September 2017; pp. 1170–1174. [Google Scholar]
- Meignen, S.; Oberlin, T.; McLaughlin, S. A New Algorithm for Multicomponent Signals Analysis Based on SynchroSqueezing: With an Application to Signal Sampling and Denoising. IEEE Trans. Signal Process.
**2012**, 60, 5787–5798. [Google Scholar] [CrossRef] - He, D.; Cao, H.; Wang, S.; Chen, X. Time-reassigned synchrosqueezing transform: The algorithm and its applications in mechanical signal processing. Mech. Syst. Signal Process.
**2019**, 117, 255–279. [Google Scholar] [CrossRef] - Fourer, D.; Auger, F. Second-Order Time-Reassigned Synchrosqueezing Transform: Application to Draupner Wave Analysis. In Proceedings of the 2019 27th European Signal Processing Conference (EUSIPCO), A Coruna, Spain, 2–6 September 2019. [Google Scholar]
- Langella, R.; Testa, A. IEEE standard definitions for the measurement of electric power quantities under sinusoidal, nonsinusoidal, balanced, or unbalanced conditions. In Revision of IEEE Std. 1459–2000; IEEE: Piscataway, NJ, USA, 2010; pp. 1–40. [Google Scholar]
- Eigeles, E.A. On the Assessment of Harmonic Pollution. IEEE Trans. Power Deliv.
**1995**, 10, 693–698. [Google Scholar] - Liang, J.; Ng, S.K.K.; Kendall, G.; Cheng, J.W.M. Load Signature Study Part I: Basic Concept, Structure, and Methodology. IEEE Trans. Power Deliv.
**2010**, 25, 551–560. [Google Scholar] [CrossRef] - Badshah, A.M.; Ahmad, J.; Rahim, N.; Baik, S.W. Speech Emotion Recognition from Spectrograms with Deep Convolutional Neural Network. In Proceedings of the 2017 International Conference on Platform Technology and Service (PlatCon), Busan, Korea, 13–15 February 2017; pp. 1–5. [Google Scholar]
- Solanki, A.; Pandey, S. Music instrument recognition using deep convolutional neural networks. Int. J. Inf. Technol. (IJITEE)
**2019**, 8, 1076–1079. [Google Scholar] [CrossRef] - Ruzzelli, A.; Nicolas, C.; Schoofs, A.; O’Hare, G. Real-time recognition and profiling of appliances through a single electricity sensor. In Proceedings of the 7th Annual IEEE Communications Society Conference on Sensor, Mesh and Ad Hoc Communications and Networks (SECON), Boston, MA, USA, 21–25 June 2010; pp. 1–9. [Google Scholar]
- Bouhouras, A.; Gkaidatzis, P.; Paschalis, A.; Chatzisavvas, K.; Panagiotou, E.; Poulakis, N.; Christoforidis, G. Load Signature Formulation for Non-Intrusive Load Monitoring Based on Current Measurements. Energies
**2017**, 10, 538. [Google Scholar] [CrossRef] - Caracalla, H.; Roebel, A. Sound texture synthesis using RI spectrograms. In Proceedings of the ICASSP 2020—45th International Conference on Acoustics, Speech, and Signal Processing, Barcelona, Spain, 4–8 May 2020; pp. 416–420. [Google Scholar]
- Costa, Y.M.G.; de Oliveira, L.E.S.; Silla, C.N. An evaluation of Convolutional Neural Networks for music classification using spectrograms. Appl. Soft Comput.
**2017**, 52, 28–38. [Google Scholar] [CrossRef] - Phaye, S.S.R.; Benetos, E.; Wang, Y. SubSpectralNet—Using Sub-spectrogram Based Convolutional Neural Networks for Acoustic Scene Classification. In Proceedings of the ICASSP 2019—2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK, 12–17 May 2019; pp. 825–829. [Google Scholar]
- Tensorflow-Guide to the Keras Functional API. Available online: https://www.tensorflow.org/overview (accessed on 18 August 2020).
- Powers, D.M. Evaluation: From precision, recall and F-measure to ROC, informedness, markedness and correlation. J. Mach. Learn. Technol.
**2011**, 2, 37–63. [Google Scholar] - Bach, S.; Binder, A.; Montavon, G.; Klauschen, F.; Müller, K.R.; Samek, W. On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation. PLoS ONE
**2015**, 10, e0130140. [Google Scholar] [CrossRef] [PubMed][Green Version] - Samek, W.; Montavon, G.; Lapuschkin, S.; Anders, C.J.; Müller, K.R. Toward Interpretable Machine Learning: Transparent Deep Neural Networks and Beyond. arXiv
**2020**, arXiv:2003.07631. [Google Scholar] - Yang, Y.; Tresp, V.; Wunderle, M.; Fasching, P.A. Explaining Therapy Predictions with Layer-Wise Relevance Propagation in Neural Networks. In Proceedings of the IEEE International Conference on Healthcare Informatics (ICHI), New York, NY, USA, 4–7 June 2018; pp. 152–162. [Google Scholar]
- Lapuschkin, S.; Wäldchen, S.; Binder, A.; Montavon, G.; Samek, W.; Müller, K.R. Unmasking Clever Hans predictors and assessing what machines really learn. Nat. Commun.
**2019**, 10, 1096. [Google Scholar] [CrossRef] [PubMed][Green Version] - Montavon, G.; Binder, A.; Lapuschkin, S.; Samek, W.; Müller, K. Layer-Wise Relevance Propagation: An Overview. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning; Springer: Berlin, Germany, 2019. [Google Scholar]
- Kohlbrenner, M.; Bauer, A.; Nakajima, S.; Binder, A.; Samek, W.; Lapuschkin, S. Towards best practice in explaining neural network decisions with LRP. arXiv
**2019**, arXiv:1910.09840. [Google Scholar] - Alber, M.; Lapuschkin, S.; Seegerer, P.; Hägele, M.; Schütt, K.T.; Montavon, G.; Samek, W.; Müller, K.R.; Dähne, S.; Kindermans, P.J. iNNvestigate neural networks! J. Mach. Learn. Res.
**2019**, 20, 1–8. [Google Scholar] - Binder, A.; Montavon, G.; Lapuschkin, S.; Müller, K.R.; Samek, W. Layer-wise relevance propagation for neural networks with local renormalization layers. In Proceedings of the International Conference on Artificial Neural Networks (ICANN), Barcelona, Spain, 6–9 September 2016; Springer: Berlin, Germany, 2016; pp. 63–71. [Google Scholar]

**Figure 1.**Comparison of the different Time–Frequency Representations (TFRs) respectively provided by the Short-time Fourier Transform (STFT): (

**a**) the reassigned spectrogram, (

**b**) the synchrosqueezing, (

**c**) and the time-reassigned synchrosqueezing (

**d**) for the instantaneous power signal of a measured appliance from the PLAID dataset [17].

**Figure 2.**Illustration of the general process for supervised Home Electrical Appliance (HEA) recognition from voltage and current measurements.

**Figure 3.**Description of hand-crafted electrical features proposed in [15].

**Figure 4.**Proposed Convolutional Neural Network (CNN) architecture used for predicting the label of a HEA from the time-frequency representation of its instantaneous power signal.

**Figure 5.**Two-channel input CNN architecture used for predicting the label of a HEA from the time-frequency representations of its active power (P) and reactive power (Q) signals.

**Figure 6.**Concatenated CNN architecture used in this research work. The input corresponds to one channel of active power (P) TF representation and one channel of reactive power (Q) TF representation.

**Figure 7.**Spectrograms and synchrosqueezed STFT with their corresponding overlayed relevance maps of a correctly predicted individual from class 1 (General Electric incandescent light bulb).

**Figure 8.**Spectrograms and synchrosqueezed STFT with their corresponding overlayed relevance maps of a correctly predicted individual from class 32 (Samsung microwave).

**Figure 9.**TFRs with overlayed relevance maps of misclassified individuals. (

**a**) corresponds to an individual of class 1 (General Electric incandescent light bulb) predicted as an individual of class 32 (Samsung microwave) and (

**b**) corresponds to an individual of class 2 (Frigidaire fridge) predicted as an individual of class 16 (Whirlpool washing machine).

**Table 1.**Comparative results (in percentage) of the different HEA recognition methods applied to the PLAID dataset. The window parameter L is empirically chosen to provide the best results.

Acc | ${\mathbf{F}}_{\mathbf{M}}$ | Rec | Pre | |
---|---|---|---|---|

P, Q + Random Forest [15,19] | 97.8 | 97.7 | 97.6 | 97.9 |

STFT (L = 60, single-input CNN) | 87.1 | 87.2 | 87.3 | 88.4 |

STFT (L = 600, CNN with two channels) | 97.7 | 97.5 | 97.5 | 97.9 |

STFT (L = 600, CNN concatenated) | 95.6 | 95.7 | 95.5 | 96.1 |

Synchrosqueezing (L = 600, single-input CNN) | 91.9 | 92.1 | 92.4 | 93.1 |

Synchrosqueezing (L = 60, CNN with two channels) | 85.4 | 85.0 | 85.4 | 86.1 |

Synchrosqueezing (L = 60, CNN concatenated) | 87.2 | 87.3 | 87.4 | 87.9 |

Time-reassigned synchrosqueezing (L = 60, single-input CNN) | 85.8 | 86.1 | 86.4 | 85.9 |

Time-reassigned synchrosqueezing (L = 60, CNN with two channels) | 91.4 | 91.2 | 90.9 | 92.1 |

Time-reassigned synchrosqueezing (L = 60, CNN concatenated) | 92.3 | 92.3 | 92.4 | 91.9 |

Reassigned spectrogram (L = 600, single-input CNN) | 74.4 | 75.0 | 74.1 | 77.3 |

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Houidi, S.; Fourer, D.; Auger, F. On the Use of Concentrated Time–Frequency Representations as Input to a Deep Convolutional Neural Network: Application to Non Intrusive Load Monitoring. *Entropy* **2020**, *22*, 911.
https://doi.org/10.3390/e22090911

**AMA Style**

Houidi S, Fourer D, Auger F. On the Use of Concentrated Time–Frequency Representations as Input to a Deep Convolutional Neural Network: Application to Non Intrusive Load Monitoring. *Entropy*. 2020; 22(9):911.
https://doi.org/10.3390/e22090911

**Chicago/Turabian Style**

Houidi, Sarra, Dominique Fourer, and François Auger. 2020. "On the Use of Concentrated Time–Frequency Representations as Input to a Deep Convolutional Neural Network: Application to Non Intrusive Load Monitoring" *Entropy* 22, no. 9: 911.
https://doi.org/10.3390/e22090911