A Novel Combination Method of a Convolutional Neural Network and Energy Operators for the Detection of Change-Points in Electromyographic Signals

Abstract

Currently, neural network algorithms based on time-domain features are used for change-point detection problems, and they have proven to be effective. However, due to the instability of human biosignals, establishing a training dataset with labels is difficult. For supervised learning methods, wherein parameters are updated on a small sample set through a feed-forward mechanism, it is difficult to ascertain the degree to which the performance of the trained neural network corresponds to the overfitting of the dataset upon which the network was trained. To this end, this paper attempted to directly replace the parameters in the convolutional neural network that need to be updated by training. A method based on the combination of the Teager–Kaiser energy operator (TKEO) and the convolutional neural network is proposed. We tested the proposed method on simulated EMG data with different signal-to-noise ratios and real data with labels, respectively. Compared with multiple detection methods, the proposed method had significant advantages in terms of reliability, accuracy, and computational speed. Furthermore, the proposed method does not require any prior knowledge about the signal, lending itself to be flexible and adaptable to any application. It may be a promising alternative to solving change-point detection problems.

1. Introduction

Electromyogram (EMG) signals are superimposed motor unit action potentials (MUAPs) generated by motor neuron firing, reflecting a high level of neural control signals from the brain, and can be decoded by change-point detection algorithms to provide very specific information on the timing of muscle action [1]. The temporal information of muscle action can be used as a reliable neural control signal for mechanical exoskeletons [2,3], as a source of information for clinical evaluation of neuromuscular injuries [4,5], as a reference indicator for sports training [6,7], and as an interface signal for human–computer interaction [8,9,10]. It has important research significance and application value.

The most common method for EMG change-point detection (CPD) is the threshold method based on the time domain, which is very simple and intuitive [11]. There is a consensus among researchers that the amplitude variation of the EMG signal can reflect the activation intensity of MUAPs and the duration of muscle activity [12]. Therefore, comparing the EMG signal with a fixed threshold, the time point recorded when the amplitude of the EMG signal is above or below this threshold reflects the temporal characteristics of muscle activation. However, in situations where the threshold level is derived from the EMG amplitude, any change in the electrical activity amplitude will proportionally affect the threshold level [13]. Therefore, a fixed-constant threshold is often unsatisfactory for the CPD of noisy EMG signals. To overcome this problem, a double-threshold method was proposed. Double-threshold detectors allow the user to adopt the link between false alarms and detection probability with a higher degree of freedom than single-threshold methods. The user can tune the detector according to different optimal criteria, adapting its performance to the characteristics of each specific signal and application. However, double-threshold detectors still rely on complex signal preprocessing techniques, and they require prior acquisition of the features of the baseline segment (noise only) of the EMG signals. Consequently, frequency domain-based methods such as short-time Fourier transform (STFT) [14,15] and wavelet transform [16,17,18,19] are used to deal with the CPD issue of EMG and exhibit strong anti-noise ability. However, the time-to-frequency conversion of the frequency-domain-based methods requires a long window length to ensure a certain frequency-domain resolution, so the detector has poor real-time performance and is prone to distortion at high frequencies. With the development of neural network technology, dealing with CPD issues has refocused on the time-domain of the signals. The neural network detectors learn the time-domain features of the signal, which not only solves the real-time problem of the frequency-domain approach but also achieves strong anti-noise ability compared with the threshold-based method. However, such methods require a certain amount of labelled training data to train the parameters for optimal performance, and they also need to avoid problems such as overfitting or underfitting. It is difficult to obtain data with accurate labels due to the instability of human biosignals. Therefore, although the neural network detectors have good CPD performance in experimental scenarios [20,21], they are still not suitable for practical applications without solving the robustness of few-shot learning. Recently, published statistical methods, including the Bayesian change-point analysis methodology [22], profile likelihood maximization (PLM) [23,24], and CUSUM [25], are attractive alternatives. However, prior knowledge of the distribution assumption, probability parameter estimation, and density function selection can seriously affect the performance of CPD. Although the execution time of the statistics-based method can be improved by assuming that the underlying data distribution is Laplace (e.g., PLM-Lap) and by replacing the exhaustive search method with a discrete Fibonacci search to seek the maximum of profile likelihood function [24], ensuring the accuracy of the distribution fit is unreliable for a small number of sampling points. In other words, statistical-based detectors, when dealing with different types of EMG signals (e.g., fast, slow, or maximum voluntary contractions), are likely to fail if a false prior assumption is selected [23]; moreover, the number of sampling points for data distribution estimation should not be too small. In summary, while many new CPD techniques offer relatively good detection performance, the algorithms themselves are becoming increasingly complex, which leads to more demanding prerequisites for their implementation and deployment.

Because of this, it has been shown that the use of a simple but well-configured detector can match or even surpass the performance of many complex models when dealing with time-series tasks, such as outlier detection, anomaly detection, or change-point detection [26]. Due to the similarity between biosignals and speech signals, many techniques used to process speech signals can be used to process biosignals, such as Teager–Kaiser energy operator (TKEO) [27,28]. An important feature of the TKEO is that the time–frequency information of signals can be extracted through an extremely narrow window containing only three sampling points [9]. Therefore, the TKEO is a very effective pre-processing technique with the advantages of low computational complexity and high real-time performance, and it has been proven to be effective in improving the performance of EMG signal CPD [22,23,29,30,31,32,33].

Most previous studies in this area have employed a combination strategy of energy operators and threshold-based detectors [34,35,36], because threshold-based detectors can retain the real-time advantage of energy operators. Trained neural network detectors have the same advantage of a fast detection speed, and, at the same time, they are more robust [20,21]. Furthermore, extended versions of TKEO have been demonstrated to have improved detection performance for multiple detectors [37,38]. In this paper, a novel EMG activity detector (EONND, energy operator neural metwork detector) is proposed, which uses the energy operators to define the structure and set the parameters of a convolutional neural network, replacing the training process of the neural network, thus solving the problems that the update of neural network parameters requires a large amount of data with labels and that the design of the network structure is too complicated. Compared to several existing detectors, EONND has higher validity and reliability, and it has a definite advantage in terms of running time.

The remainder of this paper is organized as follows: Section 2 introduces the fundamental theory of EONND. We simulated EMG signals with a variety of signal-to-noise ratios (SNRs) in Section 3 and compared several existing methods using simulated and real data to verify the validity of EONND. In Section 4, the experimental results are discussed. Finally, the conclusions are drawn in Section 5.

2. Materials and Methods

2.1. Background of Energy Operators

The Teager–Kaiser energy operator (TKEO) is a signal processing technique that was proposed by Kaiser to calculate the instantaneous “energy” of nonlinear signals based on a conversation with Teager, and it has important applications in the processing of various nonlinear signals, such as speech signals [39,40], physiological signals [23,30,41], and mechanical vibration and noise signals [42,43,44,45]. The most commonly used discrete domain TKEO formula is obtained by converting the continuous domain to the discrete domain through the forward difference method:

$${\Psi }_d\left[x\left(n\right)\right]=x^2\left(n\right)-x\left(n+1\right)x\left(n-1\right)$$

(1)

where $x\left(n\right)$ denotes the data value at the $n$th sampling point, and $n+1$ and $n-1$ represent the pre- and post-sampling points adjacent to the $n$th sample, respectively. Equation (1) shows that the TKEO can monitor the “energy” of the signal in a very narrow window of three sampling points, with simple operations and no division, making it particularly suitable for processing zero-crossing signals. The TKEO is actually a special case under a broader operator paradigm. Kumaresan et al. generalized the TKEO into a matrix framework and interpreted the general framework through the determinant of a Toplitz matrix containing the signal and its derivatives. Using this, we can obtain the determinant of Equation (1), as follows [46]:

$${\Psi }_d\left[x\left(n\right)\right]=\left(\begin{array}{cc}x\left[n\right]& x\left[n-1\right]\\ x\left[n+1\right]& x\left[n\right]\end{array}\right)$$

(2)

The determinant is time-invariant for a signal with constant frequency. If such a matrix is generalized to an $M\times M$ Toeplitz matrix by adding delayed $x\left[n\right]$ up to $x\left[n\pm \left(\mathrm{M}-1\right)\right]$, the determinant is also time-invariant but for signals with multi frequency [46,47]:

$${\Psi }_M\left[x\left(n\right)\right]=\left(\begin{array}{cc}x\left[n\right]& x\left[n-\left(M-1\right)\right]\\ x\left[n+\left(M-1\right)\right]& x\left[n\right]\end{array}\right)$$

(3)

Different $M$ values can enhance different frequency components [47], which not only makes the algorithm ‘observe’ more signal information to enhance the robustness but also improves the signal conditioning performance of the energy operators.

In fact, converting a discrete signal into a determinant form can be understood as slicing the signal:

$$X_n=\left[x\left[n\right]\hspace{1em}x\right[n-m\left]\hspace{1em}\dots \hspace{1em}x\right[n-\left(d-1\right)m]{]}^T$$

(4)

where $m$ and $d$ are integer time-lag values, and ${\left(·\right)}^T$ denotes the transpose operator. The sliced signal of Equation (4) can still be matrixed by the Toeplitz matrix [38,46]:

$$X_n=\left(\begin{array}{llll}x\left[n\right]\hfill & x\left[n+s\right]\hfill & \dots \hfill & x\left[n+\left(d-1\right)s\right]\hfill \\ x\left[n-m\right]\hfill & x\left[n-m+s\right]\hfill & \dots \hfill & x\left[n-m+\left(d-1\right)s\right]\hfill \\ ⋮\hfill & ⋮\hfill & \ddots \hfill & ⋮\hfill \\ x\left[n-\left(d-1\right)m\right]\hfill & x\left[n-\left(d-1\right)m+s\right]\hfill & \dots \hfill & x\left[n-\left(d-1\right)\left(m-s\right)\right]\hfill \end{array}\right)$$

(5)

where $s\ne 0$ is a time-lag value, as are $m$ and $d$. Important factors for maintaining the efficiency of energy operators are the length of their corresponding index windows and the symmetry of their time alignment [44]. Therefore, we set $s=m=1$, $d-1=\mathrm{k}$ and converted Equation (5) into a more intuitive format of temporal symmetry, as follows:

$$X_n=\left(\begin{array}{llll}x\left[n\right]\hfill & x\left[n+1\right]\hfill & \dots \hfill & x\left[n+\mathrm{k}\right]\hfill \\ x\left[n-1\right]\hfill & x\left[n\right]\hfill & \dots \hfill & x\left[n+\mathrm{k}-1\right]\hfill \\ ⋮\hfill & ⋮\hfill & \ddots \hfill & ⋮\hfill \\ x\left[n-\mathrm{k}\right]\hfill & x\left[n-\mathrm{k}+1\right]\hfill & \dots \hfill & x\left[n\right]\hfill \end{array}\right)$$

(6)

Equations (3) and (6) address two key issues of EONND. Firstly, Equation (6) converts 1-D discrete signal data into the form of 2-D Toeplitz matrices, providing the mathematical basis for the combination of energy operator methods with neural networks. Secondly, Equation (3) provides a computational method for obtaining energy operators from 2-D matrices data. It is worth mentioning that Equation (6) can be regarded as a matrix form of a multi-resolution Teager energy operator (MTEO) [48,49].

2.2. Relational Mapping of Convolutional Neural Networks to Change-Point Detectors

The main types of neural networks are feedforward networks (FFN), recurrent neural networks (RNN), long short-term memory (LSTM), auto encoder (AE), generative adversarial network (GAN), and convolutional neural networks (CNN) [50]. By analyzing the structure of neural networks, we found that the unique architecture of CNN is relatively suitable to integrate with energy operator methods to construct change-point detectors.

The architecture of the simplest typical CNN generally consists of an input layer, a convolutional layer (CONV), a ReLU layer, a pooling layer, a fully connected layer (FC), and an output layer [51,52], and the CONV layer and ReLU layer are often collectively referred to as the convolutional layer. In more detail, CONV is a filter that extracts signal features and filters the signal. The filter is equivalent to a convolution kernel for data in 2-D matrix form, defining the “vision” of the detector and assigning weights to each data point. The pooling layer pools the signal features from the CONV output, extracts the main features, and implements feature compression. After CONV and pooling, the data are abstracted into features with higher information content, and FC is subsequently used to classify these features.

The architecture of a typical change-point detector is almost identical to that of CNN. First, a filter (Butterworth, Baseline detrending, TKEO, etc.) is needed for the noise reduction or feature enhancement of the raw signal; then, the signal is framed by a sliding window to extract features such as the mean or variance of each frame. Finally, the features are classified by a classifier. Due to the one-to-one correspondence between the functional components of CNN and the traditional change-point detector, the mapping can be established intuitively, as shown in Figure 1.

In the CNN algorithm, the parameters (such as weights and biases) of neurons in the CONV, pooling layer, and FC need to be updated through training with a large amount of labelled data. The training process generates most of the computation demand in CNN and is the main source of algorithmic complexity. Furthermore, although CNN is nonlinear, functional components such as the CONV, pooling layer, and FC are linear, and the nonlinear properties of neural network algorithms mainly come from activation functions.

By introducing the functional components of CNN, it can be observed that, if the weights and biases of CNN are pre-set in advance, and the activation function part is retained, not only can the complexity of the algorithm be reduced, but also the non-linear property of the detector can be preserved.

2.3. EONND: Energy Operator Detector with Neural Network Architecture

CNN can process signals in multiple dimensions, including 1-D for time series, such as speech or biosignals; 2-D for images or audio spectrograms; and 3-D for video or volumetric images [51]. The CONV and pooling layers can be stacked in any number, and the more layers a CNN model has, the higher-level features it will extract [52]. In contrast, the processing of 1-D biosignals is relatively simple, and it can be expected that a simplified CNN may still have a good detection performance. In this paper, the EMG signal was converted into a $k\times k$ Toplitz matrix by a multi-resolution energy operator, as shown in Equation (6). Equation (6) describes the range of data observed by the detector at each detection, and Equation (3), similar to the convolution kernel function, describes how the sampling points are calculated.

As shown in Figure 2, the construction of EONND is very simple and has only the most basic architecture of a typical CNN. Each layer will be described in detail in the following section. It is important to note that, although there is no real convolution operation in EONND, the algorithmic framework of EONND is consistent with CNN. Therefore, the labels of each layer are also consistent with CNN for clarity of description and ease of understanding.

2.3.1. Convolutional Layer

As shown in Figure 2, in the first step, the signal is converted into the form of a $k\times k$ Toplitz matrix by a multi-resolution energy operator. The Toplitz matrix is determined by the elements of the first row and the first column. In this paper, the first row was $\left[x\left(n\right) x\left(n+1\right) x\left(n+\cdots \right)\dots x\left(n+k\right)\right]$, and the first column was $\left[x\left(n\right) x\left(n-1\right) x\left(n-\cdots \right)\dots x\left(n-k\right)\right]$, where $x\left(n\right)$ represents the $n$th sampling point. Based on the sampling points calculation relationship established by Equation (3), the convolution layer of EONND is shown in Figure 3.

Figure 3 illustrates the logic of EONND for sampling-points processing in the form of the most common neurons in neural network algorithms. This part of the algorithm has the function of signal filtering and enhancing the active segment features, since the computational relationship between the sampling points is a multi-resolution energy operator. More specifically, $f\left(x\right)=x^2$ is used to replace ReLU as the activation function, and a local connection is used to establish the computational relationship for the data in the range $k$ before and after sampling point $n$. Compared with the traditional TKEO approach of observing only three sampling points to obtain the “energy” of the signal, Equation (7) allows the energy operator to observe more contextual content and thus obtain more accurate features of sampling point $n$.

$$C\left(k\right)=W_1{\left(w_{11}x\left(n+k\right)\right)}^2+W_2{\left(w_{22}x\left(n\right)\right)}^2+W_3{\left(w_{33}x\left(n-k\right)\right)}^2+W_4{\left(w_{14}x\left(n+k\right)+w_{34}x\left(n-k\right)\right)}^2$$

(7)

The weights parameter $W_{1-4}$ in Equation (7) is updated by training with the traditional neural network algorithms, while in this paper, the parameter $W$ was determined by establishing the relationship of Equation (8) with the multi-resolution energy operators.

$$C\left(k\right)={\Psi }_k\left[x\left(n\right)\right]=x^2\left(n\right)-x\left(n+k\right)x\left(n-k\right)$$

(8)

Then, we were able to obtain: $W_1=W_2=W_3=1$, $W_4=-1$, $w_{11}=w_{14}=2$, $w_{22}=1$, and $w_{33}=w_{34}=1/4$.

In fact, after determining the neuron parameters by Equation (8), the time-lag value $k$ becomes the only hyperparameter in CONV that affects the performance of EONND. According to previous studies, the multi-resolution energy operator with odd values in the range of the scale factor $k=15$ can cover the signal “energy” up to 450 Hz [36,39,47,48,49]. This paper set $k=15$, so that for each sampling point $n$, we could obtain odd values within $15$ [$C\left(1\right)$, $C\left(3\right)$, $C\left(5\right)$, …, $C\left(15\right)$], for a total of $8$ possible “energy” features.

As shown in Figure 3, CONV processes $2k+1$ sampling points at a time and outputs possible “energy” features as input to the pooling layer through a local connected layer and a global connected layer. In addition, all parameters within CONV were fixed by the relational Equation (8).

2.3.2. Pooling Layer

The pooling layer is responsible for fusing the multiple possible “energy” features estimated by CONV for sampling point $n$ into an “optimal” estimate, completing the sample-level filtering. Usually, the pooling functions are max pooling, linear softmax, exp. softmax, and average pooling. If they are interpreted as the weighted average, the four pooling functions exist in the relationship shown in Figure 4 [53].

As shown in Figure 4, max pooling and average pooling are the two extremes. When pooling the “energy” features, the max pooling function gives the maximum “energy” all the weight, while the average pooling function gives all “energy” features equal weight. In between, the weight distribution of the linear softmax pooling function and exp. softmax pooling function are assigned in a relatively soft way. The performance differences of the four pooling functions are discussed in Section 3.

2.3.3. Fully Connected Layer

The fully connected layer in a typical CNN is actually a deep neural network (DNN). DNN updates the linear coefficients $w$ and bias $b$ of each neuron using a backpropagation algorithm through the training-set data. The training process of neural networks not only tests the ability of engineers to tune the parameters but also requires a large amount of training data with labels. In this paper, instead of the fully connected layer, a sampling layer and a sigmoid layer were used to classify the signal point by point at the sample-level. The sampling layer actually normalizes the output of the pooling layer:

$$y_i=\frac{x_i-\underset{\left(i-2k\right)<i}{\min }\left(x_i\right)}{\underset{\left(i-2k\right)<i}{\max }\left(x_i\right)-\underset{\left(i-2k\right)<i}{\min }\left(x_i\right)},$$

(9)

where $i$ denotes the current sampling point, and the extreme values are obtained on signal segments in the range [$i-2k,i$]. The sigmoid layer, which is connected to the sampling layer, is responsible for classifying the output of the sampling layer. The sigmoid activation function is given in Equation (10):

$$f\left(x\right)=\frac{1}{1+e^{-x}}.$$

(10)

The sigmoid activation function requires a pre-set threshold T, and, when the output of the neuron exceeds T, the output value is fixed to 1 by a hard limiter, indicating that the muscle is activated, while the output in other cases is 0, indicating that the muscle is not activated.

The complete framework of EONND is described above. Using the raw EMG signal as input, EONND can return a binary output, 0 or 1, for each sampling point. However, after experimental testing, the direct output classification results contained a large number of false alarms. A post-processing step was applied to the output of EONND to reject the erroneous transitions due to the stochastic nature of the EMG signal [20]. Since the shortest interval between muscle contractions is generally considered to be $>30$ ms [11], all the muscle activations lasting less than 30 ms were discarded [34].

2.4. Performance Evaluation

The performance of the detectors was assessed and compared in terms of (1) precision, (2) recall, (3) F1 score, (4) onset bias, and (5) offset bias:

$$precision=\frac{TP}{TP+FP},$$

(11)

$$recall=\frac{TP}{TP+FN},$$

(12)

$$F1 score=\frac{2\left(recall\times precision\right)}{\left(recall+precision\right)},$$

(13)

$$onset/offset bias=\frac{1}{N}{{\displaystyle \sum }}_{i=1}^N(|\widehat{t_i}-t|).$$

(14)

In this paper, simulated and real EMG data with labels were used. Therefore, the algorithm performance could be examined by quantitatively comparing the detected onset and offset time points with the ground truth. $\widehat{t_i}$ in Equation (14) represents the ground truth, and $t_i$ is the estimate of the onset/offset time points. We can express the algorithm accuracy by the absolute difference between the detected time point and the true time point, as well as describing the robustness of the detector by the precision, recall, and F1 score. The confusion matrix of the three indices of the precision, recall, and F1 score was obtained by classifying the sampling points one by one using the following rules:

• True positives (TP): the number of EMG time points correctly classified as 1 by the detectors in activity segments;
• True negatives (TN): the number of EMG time points correctly classified as 0 by the detectors in baseline segments;
• False positives (FP): the number of EMG time-points incorrectly classified as 1 by the detectors in baseline segments;
• False negatives (FN): the number of EMG time-points incorrectly classified as 0 by the detectors in activity segments.

In addition, the running time of the program on the same PC platform was recorded as a way to verify the computational resource usage of the algorithm. Detailed comparison results are discussed in Section 3.

2.5. Experimental Data

2.5.1. Real Data

Similar to the method proposed by Chopra et al. to synchronously label change-points with the help of a high-performance gaming mouse to obtain EMG signals with labels [9], this paper used a high-precision gaming steering wheel to locate the change-points. Specifically, fifteen healthy participants (10 males and 5 females, 24–32 years old) with right-handedness participated in the experiment, completing one signal acquisition per day for five days. During each signal acquisition, the electrodes were fixed on the extensor carpi radialis longus (ECRL) and flexor carpi radialis (FCR) of the right limb of participants after skin treatment. Participants operated the game steering wheel to the left and right 30 times, respectively, in their most natural state without limiting speed or strength. The control signal collected by the game steering wheel can be regarded as the human motion data to segment the active and baseline segments of the EMG signals (including the onset and offset time points). The raw real EMG signals are illustrated in Figure 5b.

It should be noted that, due to the antagonistic characteristics of the muscles, the active muscles (spontaneous muscles) of the right forearm are different when the steering wheel is operated to the left or right, which results in half of the real EMG signals acquired in this paper not being accurately detected at the change-points of the active segments and needing to be discarded.

2.5.2. Simulated Data

In this paper, the performance of EONND was tested on real and simulated EMG signals. The simulated data was able to compensate for the lack of quantity of real data and the deviation of data labels. More importantly, the parameters of the simulated data could be adjusted to suit the experimental purpose, such as SNR. However, simulating EMG signals is very difficult because it is not possible to accurately describe all the physiological, anatomical, and physical properties of EMG signals [54]. Since the EMG signal model is generally simplified as the superposition of a noisy signal (baseline segment) and EMG signal (activity segment), the EMG signal change-point detection can be described as a source separation issue [55].

Based on the objectives of the experiment, a simple EMG signal model was simulated using NeuroKit2 (a Python toolbox for signal processing in neurophysiology) [56], which was as follows:

$$Simulated EMG=S_{activity}+S_{noise},$$

(15)

where ${\mathrm{S}}_{activity}$ represents the electrical activity generated by each muscle during contraction, and $S_{noise}$ denotes the background noise generated mainly by the neighboring muscles, the friction between electrodes and skin, and the ambient noise [31,57,58]. As illustrated in Equation (15), the simulated EMG signals with different SNR values could be obtained by changing $S_{noise}$ while keeping ${\mathrm{S}}_{activity}$ constant. Ten different values of SNR were simulated (SNR = 14.41, 14.12, 12.51, 10.59, 8.91, 7.52, 6.38, 5.45, 4.69, and 4.05 dB). For each SNR value, 90 active segments were simulated at one time, for a total of 10 simulations. The simulated EMG signal is shown in Figure 5a.

Therefore, a dataset consisting of 4500 real and 9000 simulated activity segments was created. It should be noted that the labels of the real EMG signals in Figure 5b are described as ‘rough’ labels, as the responsiveness of the experimental device and the steering wheel (ignoring the physiological electro-mechanical delay) determines the temporal bias of the real data labels.

3. Results

3.1. Comparison of the Pooling Functions

Four pooling functions are tested. The performance of the linear softmax pooling function in EONND was very unstable, and the effective threshold selection range was narrow. This meant that the parameter was difficult to tune, so it was excluded from the comparison. In addition, since the F1 score reflects the overall performance of the algorithms, this section takes the F1 score as an example to show the performance trends of different pooling functions in the threshold range of 0.5 to 0.6 for processing different SNR values. Additional details on the performance evaluation can be found in the Supplementary Materials. In particular, data (onset bias, offset bias, precision, recall, and F1 score) are reported separately, and bar diagrams comparing the three pooling functions are also provided.

Figure 6a–c shows the 3D surface plots of F1 score for the three pooling functions with the SNR on the X-axis, the threshold values on the Y-axis, and the F1 score statistics for the detector at the corresponding parameter settings on the Z-axis. Figure 6d shows that the histogram of the detection performance of the three pooling functions varied with SNR under the optimal threshold conditions (max T = 0.53, exp. T = 0.54, ave. T = 0.54). It can be seen that the three pooling functions showed a consistent trend with condition changes, and the exp. and ave. pooling functions had a similar performance, while max pooling was stronger. When EONND obtained the optimal threshold parameter, as shown in Figure 6d, although the peak performance of the three pooling functions was very close at a higher SNR, only with the max pooling function did EONND still maintain a higher F1 score as SNR decreased, indicating that the use of the max pooling function in the EONND framework allows the detector to be more robust to noise.

The same conclusion was drawn from the results tested on real EMG signals. As shown in Figure 7, the performance of EONND with the three pooling functions on the real EMG signals was still the best with the max pooling function. EONND with the max pooling function achieved all three reliability metrics at higher values than the other pooling functions, with smaller deviations.

In terms of detection accuracy, max pooling is still the best choice.

Table 1. Onset/offset bias averaged over the simulated EMG data for each value of SNR and for each pooling function.

SNR (dB)	Onset Bias (ms)			Offset Bias (ms)
	Max T = 0.53	Exp. T = 0.54	Ave. T = 0.54	Max T = 0.53	Exp. T = 0.54	Ave. T = 0.54
14.41	16.59	25.55	25.55	0.16	0.32	0.33
14.12	17.00	25.14	25.14	0.20	0.37	0.37
12.51	17.38	25.53	25.53	0.31	0.49	0.54
10.59	17.10	26.17	26.17	0.36	0.51	0.51
8.91	17.39	27.44	27.44	0.43	0.60	0.60
7.52	17.90	28.16	28.00	0.76	0.84	0.80
6.38	16.69	27.95	27.95	0.62	1.32	1.27
5.45	15.07	25.24	25.24	0.52	2.17	2.17
4.69	14.05	23.94	23.61	1.29	2.63	2.76
4.05	12.53	22.73	22.73	1.27	2.83	2.83
Real EMG	102.37 ± 19.01	117.25 ± 21.95	120.73 ± 23.43	204.54 ± 71.65	218.28 ± 80.71	231.30 ± 85.87

T represents the threshold parameter.

shows the average values of the detection bias of the three pooling functions for the onset and offset under the optimal threshold parameters. Three phenomena can be observed from Table 1: (1) The exp. and ave. pooling functions had similar accuracy, while max pooling showed a better performance; (2) For the simulated signals, the detection of the offset points was more accurate than the onset points; (3) For real EMG signals, the accuracy of the onset points was higher than the offset points.

In summary, by testing four pooling functions, max pooling should be used as the best matching choice for EONND. The following section will compare the performance of EONND with the max pooling function against several popular EMG activity detectors to highlight the effectiveness of the proposed method in this paper.

3.2. Comparison of EONND with Popular Detectors

EONND is an energy-operator-based method, and, as a reference, the traditional TKEO method and a variant version, MTEO, are used for comparison [34,36]. Other detectors used for comparison were the Envelope method, the Pelt method, the Biosppy method [59], the Silva method [60], and the traditional amplitude-based method. Among them, Pelt detector and Silva detector were excluded first for their poor performance on the test data.

The TKEO and MTEO use a double-threshold detector after full-wave rectification; for details of the detector parameters, refer to [34,36]. The Envelope method first requires noise reduction of the signal through a fourth-order 100 Hz high-pass Butterworth filter followed by a constant detrending, then the signal is conditioned by the TKEO, and finally the envelope of the signal is obtained by second-order Butterworth filter with zero lag to complete the complex signal pre-processing [56]. Based on the above reduction of signal noise, Biosppy also needs to perform a two-step operation of full-wave rectification and sliding average. The amplitude-based method is based on noise reduction by a fourth-order 100 Hz high-pass Butterworth filter and detrending, followed by a sliding average to smooth the signal and the square of the filtered sample points as signal features, and, finally, a double-threshold detector for classification. For convenience, the above detectors are named abs_TKEO, abs_MTEO, Biosppy, ENV, and AMP, respectively. In addition, all algorithms require threshold parameters. For the experimental data in this paper, the method of automatic threshold update based on SNR [32,61] was not very effective, so before proposing an effective mutual mapping relationship between the threshold and signal features, this paper determined the threshold by the traversal method. Figure 8 and Figure 9 show the schematic diagrams of the detection results of the six detectors for the simulated and real EMG signals, respectively. As shown in Figure 8 and Figure 9, each detector had different requirements for signal pre-processing, resulting in significant changes in signal features, with Biosppy being the closest to the raw EMG signal.

Figure 10 illustrates the performance of the six detectors on simulated and real data. As shown in Figure 10a, for Precision, EONND achieved the highest Precision when the SNR was high, but as the SNR decreased, the Precision values of EONND decreased slightly and were surpassed by Biosppy, indicating that Biosppy has a low false detection rate for the change-point detection of noise-containing signals. The Precision values of the TKEO and MTEO were between those of EONND and Biosppy, and MTEO performed better. Figure 10b shows that, although the ENV method had the highest Recall for signals with a high SNR, the performance of the ENV method decreased significantly as the SNR decreased, and it was gradually surpassed by other detectors. As the SNR continued to decrease, the Recall values of Biosppy also decreased significantly, while the performance of the energy-operator-based detectors was relatively stable, with EONDD being the best, which indicated that EONND had fewer missed detection problems due to the decrease in SNR. Figure 10c illustrates that the overall performance of EONND was significantly better than the other detectors, especially in scenarios wherein the SNR was below 7 dB. Figure 10d shows the performance of the detectors under real EMG signal conditions. Although EONND was still the best performing detector, the statistical metric, F1 score, only achieved 77.46%, but with 98.75% Precision. The reason for this phenomenon is analyzed in detail in the Discussion section.

Table 2. Onset/offset bias averaged over the simulated EMG data for each value of SNR and for each EMG activity detector.

	SNR (dB)	EONND	abs_TKEO	abs_MTEO	AMP	Biosppy	ENV
Onset bias (ms)	14.41	16.59	30.62	8.46	31.36	26.43	1.73
	14.12	17.00	30.34	10.87	31.33	27.06	1.78
	12.51	17.38	29.74	14	29.63	28.16	1.16
	10.59	17.10	27.47	19.54	26.94	29.39	17.66
	8.91	17.39	26.49	25.21	26.72	30.77	[‘Nan’]
	7.52	17.90	27.42	27.57	27.1	32.37	[‘Nan’]
	6.38	16.69	28.73	29.96	28.53	34.21	[‘Nan’]
	5.45	15.07	29.88	34.20	30.72	35.22	[‘Nan’]
	4.69	14.05	31.89	37.05	32.75	38.55	[‘Nan’]
	4.05	12.53	34.87	40.26	33.78	40.44	[‘Nan’]
	Real EMG	102.37 ± 19.01	144.22 ± 25.77	126.17 ± 21.96	187.54 ± 33.65	177.32 ± 35.65	166.54 ± 52.35
Offset bias (ms)	14.41	0.16	0.00	0.00	0.03	0.00	0.00
	14.12	0.20	0.01	0.00	0.03	0.00	0.00
	12.51	0.31	0.03	0.00	0.03	0.00	0.00
	10.59	0.36	0.09	0.08	0.09	0.00	0.00
	8.91	0.43	0.13	0.09	0.25	0.04	[‘Nan’]
	7.52	0.76	0.21	0.26	0.23	0.28	[‘Nan’]
	6.38	0.62	0.24	0.36	0.26	1.85	[‘Nan’]
	5.45	0.52	0.4	1.07	0.52	1.81	[‘Nan’]
	4.69	1.29	2.22	2.89	1.03	6.17	[‘Nan’]
	4.05	1.27	4.38	8.06	4.37	13.31	[‘Nan’]
	Real EMG	204.54 ± 71.65	246.83 ± 85.22	221.09 ± 77.47	264.23 ± 81.28	257.31 ± 77.85	252.34 ± 78.43

[‘Nan’] represents undetected.

gives the comparison results of the accuracy of all detectors. For onset points, the “energy”-based methods did not change significantly as the SNR decreased and were relatively stable, with EONND being obviously more accurate. The bias of Biosppy gradually increased, and the performance of the ENV method was very extreme; when the SNR was high (SNR > 12 dB), the accuracy of ENV was substantially better than the other methods, but after the SNR fell below 10 dB, almost no accurate change points were detected. In terms of offset points, the performance of ENV showed a similar trend to that of onset detection; Biosppy had a good detection accuracy for signals with SNR > 5 dB, but as the SNR continued to decrease, the performance of Biosppy also decreased significantly. EONND maintained a low detection bias, and, as the SNR decreased, the detection accuracy of EONND for offset points only showed a small increase in bias. For the detection of real EMG signals, EONND had the smallest detection bias, indicating the highest accuracy.

3.3. Comparison of Running Time

The runtime is an important consideration for EMG change-point-detection algorithms, especially in real-time applications or when they are deployed on low computing power devices. In this experiment, we deployed all algorithms via Python 3.9.12 on a PC platform with a CPU of i5-5200U @ 2.20 GHz and 8 GB (1600 MHz) of RAM. Ninety EMG signal activity detections were executed on the same simulated signal to obtain the total running time, as well as the average running time. The statistical results are shown in

Table 3. Running time comparisons.

Methods	EONND	abs_MTEO	abs_TKEO	AMP	Biosppy	ENV
Total	165.6 ± 14.31 ms	374.8 ± 19.19 ms	162.6 ± 13.92 ms	500 ± 57.22 ms	16.56 ± 0.65 s	19.54 ± 0.62 s
Mean	1.84 ms	4.16 ms	1.8 ms	5.56 ms	183 ms	217.11 ms

.

As shown in Table 3, the running time of the “energy”-based methods was much lower than that of Biosppy and ENV, and EONND was close to the traditional TKEO and lower than MTEO, indicating that the proposed combination of multi-resolution energy operators and neural networks has lower computational complexity. It should be noted that the running time in Table 3 is only the time taken by the algorithm to detect the change-points and does not include the time to perform any pre-processing of the signal. From the perspective of calculation complexity, the threshold method is the simplest one to implement, since the algorithm only needs to compare instant signal amplitude to a predefined threshold [11]. Therefore, the running time of the “energy”-based methods was much shorter than that of ENV, which was sufficient to demonstrate the advantage of the low computational complexity of the energy operator methods.

4. Discussion

Based on our previous analyses, EONND showed better reliability and accuracy than other detectors on simulated and real EMG. According to the results, five additional issues need to be discussed in detail.

• The reason for the good robustness of EONND.

Although the detector proposed in this paper was not specifically designed with an adaptive mechanism, the special neural network structure of EONND can indirectly give EONND some adaptiveness by pooling multiple energy operator features from the output of the “convolutional” layer (this is not a real convolutional layer but shows the position relationship). Therefore, EONND with the max pooling function still maintained a high change-point-detection performance for signals with very low SNR. In addition, the baseline segments of low SNR signals contain a lot of noise, which seriously affects the accuracy of the traditional threshold-based methods to determine the threshold values by obtaining the baseline segment features. EONND does not require any prior knowledge of the signal for change-point detection and is not disturbed by noise in the baseline segments. This is because EONND is essentially equivalent to a trained neural network. Therefore, EONND has a strong advantage in the processing of low SNR signals.

2.

The issue of setting EONND hyperparameters.

In the EONND method proposed in this paper, the parameters of the “convolution” layer were fixed by a multi-resolution energy operator; therefore, updating the parameters through training was not required. However, in practice, EONND still has two hyperparameters that need to be set or tuned in advance; the energy operator scale factor $k$ and the threshold $\mathrm{T}$. The scale factor $k$ of the energy operators determines how many sampling points can be observed by EONND at a time. Since this is a preliminary exploration, this paper did not discuss the issue of how to obtain the optimal value of $k$. Instead, we directly set the value recommended by relevant studies, i.e., $k=15$, because such a setting can cover the frequency of EMG signals [47,48]. For the problem of setting the threshold T, we used the traversal method to predict the value. The traversal method requires a certain amount of data with labels, but not as much data as required by the neural network algorithms. As the SNR of the simulated signal decreased, the detection performance of EONND did not show significant degradation, indicating that EONDD also has good performance under a fixed threshold value. It should be noted that the optimal threshold values for real and simulated signals are different (${\mathrm{T}}_{\mathrm{real}}=0.506$, ${\mathrm{T}}_{\mathrm{simulated}}=0.53$). Therefore, the development of an automatic mapping mechanism between the threshold and signal will be the focus of future research.

3.

The problem of abnormal accuracy performance statistics.

When the detection accuracy metrics of the simulated EMG were counted, an abnormal scenario with bias values of 0 occurred, indicating that the detection values did not deviate from the ground truth. It is important to note that this result only represents a performance trend, as such accuracy is almost impossible to achieve when dealing with real-world signals. In addition, all detectors had a higher detection accuracy for the offset points than for the onset points, but according to previous studies [62] and the detection of real EMG signals in this paper, the normal phenomenon for the detection of real EMG signals should be higher accuracy for onset than for offset, which is determined by MUAPs discharge characteristics. However, the simulated signal simplified the signal model and therefore produced the opposite result. Although there was a deviation between the study approach using simulated signals and that using real EMG signals, it is this deviation that precisely validated the generalization performance of EONND in processing different types of signals, providing the possibility to expand EONND to other signal areas (such as speech, ECG, EEG, etc.). Therefore, the study approach using simulated signals can still reflect the performance differences between detectors.

4.

Performance metrics show significant degradation on real data.

The detection results of the simulated data were significantly better than the real data, both in terms of reliability and accuracy metrics. There are two reasons for this phenomenon: (1) the device used to synchronize the onset and offset time labels and (2) the way the confusion matrix used to calculate the statistical metrics was constructed. As mentioned above, the real-world data for this paper were collected during an experiment based on steering wheel rotation. Since the temporal bias of the algorithm was heavily dependent on the responsiveness of the steering wheel (ignoring the physiological electro-mechanical delay) and the rotational torque of the steering wheel affected the features of the EMG signals, low values of the statistical metrics resulted.

Figure 11a illustrates a raw EMG signal and the onset and offset points labelled by the motion data acquired through the game steering wheel. With Figure 11a as a reference, Figure 11b–g shows the detection results of the six detectors. The vertical solid lines are the ground truth values, while the vertical dashed lines are the change points detected by the detectors. Since the ground truth values were automatically labelled by the experimental equipment, they were more objective than the manual labelling carried out by human experts. However, it is obvious that there was a period of EMG silence during the muscle voluntary contraction, called the silent period (SP) [63]. All sampling points in the SP were classified as FN by the confusion matrix. Therefore, for the real data, the performance metric Recall was significantly lower than the simulated data, as was the F1 score for the same reason. Nevertheless, the Precision of EONND was 98.75% (as shown in Figure 10d), indicating that the false detection rate of EONND was quite low.

5.

EONND applicability issues.

With the current proliferation of wearable devices, it is valuable to build an algorithmic framework with low computational complexity for low-computing-power devices. The efficiency of EONND has been demonstrated by comparison with simple methods. EONND operates almost as fast as the scheme combining the TKEO with a threshold detector and about 56% faster than the scheme combining the MTEO with a threshold detector. In addition, EONND does not require any prior knowledge about the noise segment of the signal, which is advantageous for the application of EONND to other signals with change-point detection tasks (e.g., speech signals, ECG, EEG, etc.).

5. Conclusions

This paper presents a novel EMG change-point detector, EONND. Unlike the previous approaches that use only energy operators as a pre-processing technique, EONND maps the computational relationships of energy-operator-processing sampling points to the computational operations between neurons of neural networks. As a result, EONND is able to perform the change-points detection task in a form similar to that of a trained neural network. The method proposed in this paper not only has the ability to condition the signal using a multi-resolution energy operator but also inherits the nonlinear property of neural networks while improving the detection efficiency. The performance of EONND was tested under multiple signal conditions (simulated and real signals, signals with multiple SNRs). Compared with multiple detectors, EONND showed significant advantages in terms of reliability, accuracy, and computational speed. In particular, EONND operated almost as fast as the conventional TKEO, indicating that the detector proposed in this paper had a lower computational complexity than the conventional threshold method. In future work, we need to further investigate the optimization mechanism of hyperparameters to further enhance the generalization performance of the algorithm. We expect that the proposed method will be helpful for EMG change-point detection and will provide ideas for studies related to the extraction of temporal information on muscle activity.