As part of the development phase, the proposed algorithm was implemented and tested to evaluate its performance with real data.
4.1. The Proposed System Setup
As explained in
Section 2, the SFRA technique was performed by plugging the instrument into a standard household socket. As previously discussed, the input signal is a variable frequency sinusoidal signal applied between the phase conductor terminal and ground, while the output signal is the measured signal between the neutral conductor terminal and ground. Both signals are acquired and processed.
Figure 9 shows the measurement system used.
The measurement system must be connected to the test system by means of cables with suitable bandwidth and the same characteristic impedance of the generator to avoid reflection and signal mismatch and to improve the sensitivity, repeatability, and reliability of the measurement.
The input signal and related acquisition for the SFRA were performed using the Digilent Analog Discovery 2 NI Edition card with a BNC adapter.
The control system was developed using LabVIEW and run on a PC; this software automatically programs the Discovery FPGA at startup, with a configuration file designed to implement the measurement application. Once programmed, the integrated FPGA communicates with the PC via a USB 2.0 connection. The PC enables the creation of the user interface to access the data and process them in the experimental phase. A final NILM system can bypass the PC by integrating post-processing directly into the system.
The Discovery FPGA has a ±25 V input range, a 14-bit resolution, a 100 MS/s sampling frequency, and a 30 MHz bandwidth. It is equipped with an arbitrary function generator with an output range of ±5 V, a bandwidth of 20 MHz, and a sampling rate of 100 MS/s.
For appropriate interfacing with the network, the instrument is equipped with a coupling circuit for each of the three channels (one for generation and two for acquisition), as shown in
Figure 9. The coupling circuit includes a third-order Butterworth filter with a flat passband and high attenuation outside the desired frequency range. The generation section and acquisition section coupling circuits both involve a 50 Ω resistor in series and parallel, respectively, to allow impedance adaptation. In addition, all coupling circuits are provided with a high-voltage ac blocking capacitor, connected in series with a 1:1 pulse transformer. The features of the filters developed for the SFRA apparatus are shown in
Figure 10 and
Figure 11.
In order to avoid unwanted over-voltages due to resonance phenomena at high frequencies, the amplitude of the applied signal must not exceed a few volts (5 Vpp in the present case). The accuracy of the adopted measurement system, as discussed in a previous paper [
57], has been evaluated using a reference parallel LCR circuit. This circuit consists of a 50 Ω resistive adapter, a fixed inductance, and a variable capacitance. The referenced values of the circuit impedance were measured with a Keysight E4980AL precision LCR meter. The estimated accuracy of the Vout/Vin ratio was better than ±0.2 dB in the interval from +5 to −25 dB and in the frequency range of 5 kHz to 1.5 MHz.
The SVM was implemented on a desktop computer (based on the Windows 10 × 64-bit operating system) using the open-source Python 3.7 from Anaconda [
58]; the machine-learning algorithm was developed using the Scikit-learn library. Python is the programming language mostly used in artificial intelligence (AI) applications due to the availability of numerous libraries for continuous data acquisition and processing.
4.2. The Achieved Results
The proposed measurement technique is innovative and does not appear to have been tested by other authors. Due to the specificity of the acquired data (frequency response), there are no public datasets used by other authors against which to compare the performance of the proposed algorithm [
59].
The measurement system was installed on a test facility, which was designed to generate electrical loads created by domestic users as part of the “non-intrusive infrastructure for monitoring loads in residential users” research project. The facility, located in the Electrical Engineering Laboratory of the University of L’Aquila (I), allows for the generation of electrical loads in a single or simultaneous way.
During the test phase, various parameters were evaluated in order to define the most significant sub-bands, the number of measurement points to be acquired, and the number of training examples needed to obtain a satisfactory performance. To this end, the precision, recall, and F1-Score during classification were evaluated [
60]. These parameters were obtained using the numbers of true positive (TP), false positive (FP), true negative (TN), and false negative (FN) as follows:
The concept of positive has been attributed to the ON state of household appliances and that of negative to the OFF state. Precision indicates all of the times the system has provided an indication of the ON state of an appliance and how many times the prediction has been correct. Precision does not take FNs into account. On the other hand, Recall indicates how many times the system has provided a correct indication about the ON state of the appliance compared to all of the samples in which the appliance was actually in the ON state. Recall does not take FPs into account. To have a metric capable of taking into account both FPs and FNs, the F1-Score is used, which is a harmonic mean of Precision and Recall.
Since, as already explained above, each appliance is associated with a SVM algorithm that reveals its presence, or not, the performance of each SVM was evaluated individually.
We started by acquiring 20 samples for each of the 24 scenarios, for a total of 480 training samples. Each sample consisted of an SFRA trace in which 200 points were acquired for each of the 3 sub-bands. Performance was evaluated on a test set consisting of 50 samples for each scenario, for a total of 1200 test samples. The obtained results, shown in
Table 2, are already excellent, as 480 training samples is a relatively low number considering that acquiring a single sample takes about 40 s. The system does not make mistakes for five of the eight appliances analyzed and also shows high performance regarding the other three appliances. To define which of the three sub-bands made the most significant contribution to the identification of household appliances, the system’s performance was evaluated by providing the three sub-bands separately as input to the machine-learning system. The results are reported in
Table 3 and a graphical comparison is provided in
Figure 12.
In light of these results, it was decided that we would consider only the sub-bands of 10–100 kHz and 100 kHz–1 MHz in order to reduce the time required for the measurement. In fact, it is evident from
Figure 12 that the 1–1.5 MHz band never allows for appliance discrimination that outperforms the previous bands. This reduces the time it takes to acquire a single trace to 22.56 s.
Table 4 reports the performance evaluation using only the first two sub-bands as input.
Comparing the results with those of
Table 2, it can be seen that the system’s performance has remained roughly unchanged. However, there is a significant improvement in the detection of the drill, highlighting that the 1–1.5 MHz sub-band introduced useless randomness for identification purposes. In this way, 400 points are acquired in the 10 kHz–1 MHz frequency band.
The possibility of decreasing the number of acquired points has been evaluated. Therefore, in
Table 5, the performances obtained for 200, 134, and 100 points are reported. Furthermore,
Figure 13 shows a graphical comparison of the impact of the number of acquired points on the F1-Score.
The performance proved to be very good, even when only using 100 measurement points as a system input. In these conditions, in fact, the system made errors only for three of the eight appliances analyzed while maintaining a minimum F1-Score of 0.94. This reduction allowed a decrease in the execution time of the measurement system from 22.56 s to 6.09 s. The performances shown so far always foresaw 480 training samples (20 for each of the 24 scenarios). As a final analysis, the impact of the number of training samples on performance was evaluated as shown in
Figure 14.
Table 6 reports the results obtained using an SFRA trace consisting of 100 points acquired in the 10 kHz–1 MHz frequency band, reducing the number of samples used in the training phase.
The system maintains interesting performances even when trained with only one training sample for each scenario (therefore with 24 total training samples). This is mainly because the SVM natively suffers more from the quality of the training samples rather than the quantity, which is precisely because it builds a model based only on the most difficult samples to discriminate.
Lower performance was found in the detection of the Lamp, Laptop, and Drill. In the case of the Lamp, this is due to the insignificance of its related load compared to the overall network, while in the case of the Laptop and Drill, it is due to the extreme variability of their working conditions. However, F1-Score values of 0.78, 0.87, and 0.94, respectively, can be considered largely satisfactory for a trained system with such a small number of samples.
In order to provide an overall assessment of the system’s performance, metrics widely used for multi-label classification systems were used, including micro-average and macro-average. As reported in (11)–(13), in the micro-average, all TPs, TNs, FPs, and FNs are summed for all of the labels and subsequently averaged:
On the other hand, the macro-average, as reported in (14)–(16), is simply the average of the Precision and Recall for each label:
The difference between the two lies is the fact that the micro-average reflects any imbalances in the dataset. Unbalance means there are test samples in a greater number of one or more classes than the others. In other words, having more samples for a given scenario, the macro-average, by creating a simple average of Precision, Recall, and F1-Score, does not consider this imbalance. On the contrary, the micro-average takes these situations into account.
In the case in question, the dataset is balanced; therefore, both averages are functional and adequate for verifying the performance of this system.
Table 7 reports the micro-averages and macro-averages calculated based on the values reported in
Table 6.
An additional consideration needs to be made to integrate the proposed system into an electrical system. As explained above, there is no interference with the normal operation of the devices during system operation. Furthermore, the system poses no problems to the EMI filters, which are the input stage of the monitored devices, as the powers involved—which can be associated with the test signal—are extremely low.
To analyze the operating conditions of the measurement system in detail, it was simulated in a SPICE environment.
Specifically, the simulation was oriented to analyze the effects produced by the test signal on commercial EMI filters that could be connected (to other devices) in proximity to the system being tested. The analysis was extended to the entire range of frequencies involved; as a reference, a commercial EMI filter family was considered [
61] for standard use in commercial and residential apparatuses for AC currents up to 16 A
rms in single-phase systems.
The analysis was extended to the entire range of frequencies involved.
Figure 13 summarizes the scheme considered for the simulation. The resistance R
Load equal to 50 Ω was chosen in order to simulate the load of a generic household appliance (230 V
rms/50 Ω = 4.6 A
rms).
The system’s response was evaluated by varying the frequency in the range in which the proposed system operates in the final configuration (10 kHz–1 MHz). The frequency response of the current entering the EMI filter was evaluated. Several simulations were carried out by varying the RLC parameters of the EMI filter. The current was found to be harmless across the entire spectrum. As an example,
Figure 15 shows the input current response obtained with the RLC parameters reported in
Figure 16. The spectrum shows two resonance peaks and a maximum current draw of 4.64 mA.
The reduced value of this peak current does not lead to overheating of the filter components since the associated dissipated power is reduced. Furthermore, such verification is pejorative for the following reasons:
- (1)
The proposed system adopts a Digilent Analog Discovery 2 board, which has a limitation on the maximum output current that can be supplied by the DAC channels at 4 mA.
- (2)
In our simulation, the measurement system is only connected to the device being tested. In the real case, the generator is connected to a generic socket of the electrical system; therefore, the current that can be supplied (4 mA) is distributed in the various parallel branches of the other connected devices, greatly reducing the intensity of the portion that could affect the EMI filters.