Studies on 1D Electronic Noise Filtering Using an Autoencoder

Abstract

Autoencoders are neural networks that have applications in denoising processes. Their use is widely reported in imaging (2D), though 1D series can also benefit from this function. Here, three canonical waveforms are used to train a neural network and achieve a signal-to-noise reduction with curves whose noise energy is above that of the signals. A real-world test is carried out with the same autoencoder subjected to a set of time series corrupted by noise generated by a Zener diode, biased on the avalanche region. Results showed that, observing some guidelines, the autoencoder can indeed denoise 1D waveforms usually observed in electronics, particularly square waves found in digital circuits. Results showed an average of 2.8 dB in the signal-to-noise ratio for square and triangular waveforms.

1. Introduction

Autoencoders are neural networks operated using unsupervised learning. The input data are compressed to a smaller-scale representation, called a latent variable, which contains the most relevant characteristics of the raw data. This compression function is carried out by the encoder, and the decoder, which comes in sequence, decompresses the data back into their original dimension. The latent variable is a minimal representation of the features contained in the data, and it operates as a kind of bottleneck in the convolutional neural network (CNN) architecture. Their main role can be seen as creating a rough copy of the input data, which resembles the training data [1]. Layers named MaxPooling1D concentrate the inputs from the preceding layer into single neurons, thereby reducing the data dimensionality [2]. When the input data contain complex values, the real and imaginary parts need to be treated separately since CNNs deal primarily with real numbers [3].

The goal of a denoising autoencoder is the creation of a model that is more robust against noise energy. Recurrent neural networks, where there is feedback from the output to previous layers, is sometimes used for arbitrarily long sequences [4]. Although usually applied for images, some cases report improvements in the quality of 1D waveforms. Signals from ultrasonic non-destructive testing have been subjected to denoising using an autoencoder and compared with singular value decomposition (SVD), principal component analysis (PCA) and wavelet, with the CNN providing better performance, particularly regarding the robustness [5]. Signals from seismic sensors had activation functions added to the neurons to introduce non-linearity in the process, improving the overall performance [6]. Another 1D application reported the use of denoising network followed by feature extraction on the diagnosis of a mechanical gearbox [7]. Real known defect conditions were used in the training phase. In a similar approach, denoising followed a 1D CNN, and was used to diagnose fault conditions in rotating machinery [8]. The preliminary denoising procedure is of paramount importance given the fact that real-world acquired signals are contaminated with noise, which otherwise might demand extra effort in the classification network.

When noisy conditions to be used in the training are not easy to achieve using experimental tests, the corrupted data can be created using several different alternatives, such as using a Gaussian noise distribution and back-and-forth Fourier Transforms for the case of EEG [2]. EKG signals, in turn, had high-frequency artifacts caused by electrode movements cleaned using Discrete Cosine and wavelet transform pre-processing, followed by an autoencoder, which helped improve the signal-to-noise ratio (SNR) by approximately 25 dB [9].

Concerning the field of communications, cognitive radio networks are meant to operate in a crowded electromagnetic environment using the currently most efficient modulation scheme and frequency range; therefore, they need to to learn about the available electromagnetic resources continuously. CNNs were reported to identify eight kinds of communication signals (Costas and Barker codes, binary phase shift, frequency modulation, etc.) to be applied in cognitive radio networks [10].

A convolutional autoencoder was integrated into an automated optical inspection (AOI) system, which used images from a charged-coupled device (CCD) camera to identify defects in printed circuit boards (PCBs) [11]. Similarly, vibration signals from permanent-magnet synchronous motors were processed by an autoencoder to decrease the noise levels before being classified in terms of types of fault by a support vector machine (SVM) [12]. Taking advantage of the prediction capabilities an autoencoder provides, a supervised learning network was trained to predict the reliability of digital circuits, whose results were then compared to a Monte-Carlo-based analysis [13].

Radars, in particular, operate with very-low-amplitude signals, contaminated with noise and clutter, and often have more energy than the signals of interest. Hyper-parameters of a CNN network have been analyzed to provide radar waveform recognition and classification, in real time [14]. Another similar study focused on low-probability-of-intercept (LPI) radar waveforms, identified by a dense CNN, with the original signals converted into time–frequency waveforms using Choi–Williams (CWD) distribution [15], achieving a success rate of 93.4% for an $SNR$ of −8 dB. The intra-pulse modulation in radar signals was recognized using a convolutional denoising autoencoder and a deep convolutional neural network, with the raw data converted to images using the Cohen time–frequency distribution [16].

Real-world electronic signals are usually contaminated with noise, resulting from disturbances such as unwanted electromagnetic coupling among the circuit components, shot noise observed in semiconductors and human-made electromagnetic interference (EMI). Band-pass filtering is one alternative to reduce the noise power within the signal bandwidth, at the expense of the tradeoff between the filter time and frequency domains. Steep roll-offs in frequency imply large group delays that eventually distort the time response. More sophisticated alternatives to 1D denoising are wavelets [17], empirical mode decomposition [18] and curvelets [19]. In line with this, this paper presents an autoencoder used for denoising signals commonly found in electronics, i.e., square, triangular and sine waves. In particular, square waves are usually employed in digital electronics, and preserving their original waveform shape is directly correlated with the data throughput [20]. Disturbed digital waveforms imply false readings since the wave amplitude is used for decision-making between the binary logic levels. An ensemble of clean and noisy waveforms is generated by Matlab 2022/Octave 3.6.4. and later used for the training and testing of a CNN implemented in Python 3.8.2. For the sake of comparison with real-world signals, a noise generator circuit was built and added to a signal source, outputting the same three wave shapes, and the denoising process was applied using the previously trained neural networks. Its fast response enables its use as an option after analog signals are acquired and digitized. The use of a Zener-based noise generator innovatively proved that the CNN was operating against real-world conditions, instead of signals generated in a purely computational fashion.

2. Materials and Methods

2.1. Waveform Generation

The data to be used in the training were prepared in Matlab. Each time series was set to contain 1000 samples of real numbers, distributed in the square, triangular and sine types, each case with 2000 different waveforms to be inserted into the CNN. The parameter to be evaluated, the signal-to-noise ratio $SNR$, in dB, was computed according to

$$SNR=10{log}_{10}{\displaystyle \frac{{\sum }_{i=1}^{2000}{s}_i^2}{{\sum }_{i=1}^{2000}s{n}_i^2}}$$

(1)

where s and $sn$ are the clean and noisy waveforms, respectively. Each waveform has the following characteristics:

2.2. Square Wave

Its duty cycle was set to be 50%, and the on and off parts were set to 20 points. Amplitude was set to vary between 0 and 1, and the different samples had the phase randomly varied. Noise was numerically added using the rand function in Matlab. A total of 2000 waveforms were stored without noise and 2000 were contaminated with noise. The complete waveform generation and file storage took 4.1 s, using Matlab 2018a.

2.3. Triangular Wave

Similar to the square wave, there were 2000 series, each with 1000 points. The rising and falling times were set to comprise 20 points, and their amplitude varied between 0 and 1. Generation and storage took about 4.6 s.

2.4. Sine Wave

Sine waves were generated, in contrast to the former two cases, with both frequency and phase allowed to vary within the samples. Generation of data, storage and plotting figures took about 3.9 s.

Three different samples of the waveforms, clean and noisy, are shown in Figure 1. The sine waves are shown with the 2$\pi$ scale on the x-axis.

The respective $SNR$ histograms of the created input data are shown in Figure 2. They show that the noise energy is larger than that of the signal, across all the 2000 samples.

2.5. CNN

An autoencoder is a neural network that takes an input and reproduces the same input as the output, using unsupervised learning [21]. The deployed network representation is shown in Figure 3, based on Python Tensorflow and Keras, the latter an application programming interface (API) that runs on top of the former [22]. The aforementioned vectors of length 1000 (dimension 1000 × 1) are supplied as input and sequential convolutional filters are used, followed by MaxPooling and upsampling blocks, which downsample and upsample the incoming data by a factor of 2, respectively. Convolutional filters extract the features from the input data, whereas MaxPooling performs an averaging operation on the former convolutional filter data and reduces the number of parameters. This first half-part of the CNN is named the encoder network. In other words, convolutional layers detect patterns in the incoming data, and MaxPooling reduces their dimensionality. The process is then reversed, in the so-called decoder network, to retrieve the same original input length, and instead of MaxPooling the dimensionality is progressively increased using upsampling layers. Overall, 20% of the total number of samples was allocated to testing, and the rest was used for training. Throughout the CNN, the activation function was the Rectified Linear Unit (RELU).

A total of 50 epochs were used for the training.

Table 1. Elapsed time.

Waveform	Time [s]
Rectangular	82.7
Triangular	82.2
Sine	81.5

shows the elapsed time for the three cases; it can be seen that no relevant difference was observed for the waveshapes. The loss parameter’s evolution across the epochs is shown in Figure 4. The square wave required more effort to be trained, in contrast to the other two waveforms.

Samples of the clean and noisy signals are shown in Figure 5, collected after the training was finished. The output waveforms are visually clearer, showing the success of the denoising network in cleaning severely contaminated signals, independent of the waveshape type.

An $SNR$ histogram analysis is applied to the input and output signals (Figure 6). It can be seen that the sine case had the worst performance, given the smaller distance between the noisy and denoised samples. This difference can be ascribed to the larger variation in the seed signals, i.e., either the frequency and phase were made to vary in contrast to the other two cases.

3. Real-World Noisy Signal

To evaluate the performance of the denoising autoencoder against real-world signals, a test was performed with a circuit subjected to a controlled level of noise. For the noise source, a Zener diode was employed, biased on its avalanche area [23]. The test was carried out using the circuit displayed in Figure 7.

An oscilloscope (input impedance 100 M$\Omega$, DC-coupled and with a bandwidth of 200 MHz) stored the noisy waveforms, in ASCII format, which were afterward read and processed by the CNN, trained using the Matlab-generated signals, i.e., there was not another training with the Zener-contaminated signals. The instrument’s maximum time series length is by default 2000 points, so the acquired data vector was later truncated to 1000, to be consistent with the trained input signals. Individual circuit components were manually optimized, based on the visual noise amplitude seen on the oscilloscope. For instance, the 6.6 V diode was chosen among others due to its observed higher noise levels. The signal generator couples to the Zener using a 1 k$\Omega$ resistor, and the capacitor blocks DC levels from entering into the signal generator and oscilloscope inputs. Differently from the computer-generated square waveforms, the generator had finite rise and fall times (set to its minimum default value of 17 ns). The generator was set to output square, triangular and sine waves, as displayed in Figure 8, together with the cleaned signals after the autoencoder. The generator outputs zero-offset waveforms, and the capacitor’s presence cuts eventual DC offsets. However, the CNN was trained with positive amplitudes, so to the acquired signals had their amplitude numerically shifted to positive-valued amplitudes.

Alongside the time domain results, their power spectra are shown, for either raw or processed waveforms. It is possible to see that the CNN operates as a low-pass filter, attenuating frequencies above the range where the signals concentrate most of their energies. For the specific case of the square wave (Figure 8a), it started to attenuate from approximately 4 KHz. The triangular case (Figure 8b), in turn, was found to be damped after 5 kHz, and for the sine wave (Figure 8c) this was after 7 kHz. The fundamental for all these waves was 500 Hz. It is interesting to stress that this effect was automatically found by the neural network operation after the training step. In the rectangular signal case, in particular, one notices that harmonics above the third are cut, probably due to CNN training that was performed with a perfect shape (i.e., null rise and fall times). Among the whole ensemble encompassing the three signals, the triangular wave had the worst behavior, with its shape slightly rounded in the time domain, and the effect visualized on its power spectrum with its second harmonic was completely attenuated from the denoised version.

It was found that the acquired signals did not need to be compressed into the original normalized range (0 to 1), but they were very sensitive to the baseline, i.e., a DC offset had to be added otherwise the lower part of the signals were not correctly denoised. This points out that the training signals should comprise a more diverse ensemble, particularly with randomly chosen DC offsets. Figure 9 shows an example, where the offset is not added to the acquired square wave. It is the reason that explains both the initial and final discontinuities observed in the denoised waveforms seen in Figure 8. For the present case, this DC level was added during the test period, after the CNN was trained, and its amplitude was found after trial and error.

4. Discussion

The CNN was set up using a total of seven layers, and both input and output operated with a length of 1000, real positive numbers, representing a periodic time-varying waveform, whose spectral content was preserved according to the Nyquist criterion. The initial ensemble of curves had all the elements with negative $SNR$, meaning the noise energy was larger than that of the signal. Nevertheless, the observed results showed that the autoencoder indeed reduced the noise levels in typical waveforms usually found in electronic circuits. An initial set of curves with average $SNR$ (−2.8, −2.5 and −0.8) were improved to (−0.2, 0 and 0) for the square, triangular and sine waves, respectively, in dB.

To address real-world signals, the denoising autoencoder was tested using a Zener diode noise generator, which proved that the training phase has, however, to fulfill a broader set of parameters to become more general. Namely, phase, frequency and DC offsets have to cover the range expected to be found in the noisy signals to be later cleaned by the CNN. Its operation on the signals was found to be similar to low-pass filtering, for instance, the 500 Hz rectangular wave had attenuation taking place around 4000 Hz. Since the rectangular wave has odd harmonics, the CNN converged to attenuate signals up to its fourth harmonic, which corresponds to 1/8 of the fundamental amplitude. Waveform amplitudes were not found to have a relevant impact on the denoising, so the training could be performed with the usual normalized values. That, however, was a nuisance when the laboratory test was performed since the signals were swinging towards negative amplitudes, which required posterior processing, adding a DC-offset level to make the time series positive-valued.

Also, though more time-consuming and less convenient, a more robust option would have been performing the training with real-world instruments, so that the signals better resembled the future noisy waveforms. It is not easy to predict which parameter found in the real-world case might impact the network response.

Table 2. Comparison with other studies.

Ref.	Vector Size	Application	Improvement	Real-World Test
[2]	1 × 7680	EEG	6 dB	Yes
[7]	1 × 8192	Gearboxes	96% defective recogn.	Yes
[9]	1 × 1024	EKG	25 dB	No
[16]	1 × 1024	radar	95% recogn. result	No
[24]	1 × 1024	Manufacturing	95% recogn. result	Yes
This study	1 × 1000	Electronic waveforms	3 dB	Yes

shows a comparison of similar 1D denoising studies, in different application fields.

5. Conclusions

A denoising CNN was tested against numerically synthesized waveforms, contaminated by random noise. The designed CNN was able to provide cleaner signals, where $SNR$ measurements on both inputs and outputs proved its effectiveness. An initial set of curves with average $SNR$ (−2.8, −2.5 and −0.8) was improved (to −0.2, 0 and 0) for the square, triangular and sine waves, respectively, in dB. To innovate the way the signals are tested, jumping from shared databases or pure numerical time series, a circuit with a noise generator based on a Zener device was used to inject noise levels into a generator, which later was tested using the previously trained CNN. The results showed good performance, with the laboratory tests pointing out that concrete applications might require a broader set of parameters to be contemplated during the training phase.