Next Article in Journal
Optimal Coronavirus Optimization Algorithm Based PID Controller for High Performance Brushless DC Motor
Previous Article in Journal
Optimization of the Weighted Multi-Facility Location Problem Using MS Excel
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Convolutional Neural Network with an Elastic Matching Mechanism for Time Series Classification

College of Electronic Science and Technology, National University of Defense Technology, Changsha 410073, China
*
Author to whom correspondence should be addressed.
Algorithms 2021, 14(7), 192; https://doi.org/10.3390/a14070192
Submission received: 25 May 2021 / Revised: 16 June 2021 / Accepted: 24 June 2021 / Published: 25 June 2021

Abstract

:
Recently, some researchers adopted the convolutional neural network (CNN) for time series classification (TSC) and have achieved better performance than most hand-crafted methods in the University of California, Riverside (UCR) archive. The secret to the success of the CNN is weight sharing, which is robust to the global translation of the time series. However, global translation invariance is not the only case considered for TSC. Temporal distortion is another common phenomenon besides global translation in time series. The scale and phase changes due to temporal distortion bring significant challenges to TSC, which is out of the scope of conventional CNNs. In this paper, a CNN architecture with an elastic matching mechanism, which is named Elastic Matching CNN (short for EM-CNN), is proposed to address this challenge. Compared with the conventional CNN, EM-CNN allows local time shifting between the time series and convolutional kernels, and a matching matrix is exploited to learn the nonlinear alignment between time series and convolutional kernels of the CNN. Several EM-CNN models are proposed in this paper based on diverse CNN models. The results for 85 UCR datasets demonstrate that the elastic matching mechanism effectively improves CNN performance.

1. Introduction

Time series classification (TSC) is an important research topic in data mining communities [1]. It has a wide range of applications in human activity recognition [2], speech analysis [3], electrocardiogram (ECG) monitoring [4], and biological research [5].
Deep learning is a subfield of machine learning concerned with deep structures with adjustable parameters. Many deep learning architectures exist for TSC. Compared with other classical architectures such as the multilayer perceptron and recurrent neural network (RNN), the convolutional neural network (CNN) has become one of the most prevalent architectures for TSC in recent years [6]. However, the CNN architecture is sensitive to temporal distortion [7], such as differences in rates and local translation within a pattern [8].
Many studies have been conducted on temporal distortion for TSC. One of the most representative studies is on dynamic time warping (DTW). In conjunction with a one-nearest-neighbor (1NN) classifier, DTW achieves great success in TSC. Compared with the lock-step matching in Euclidean distance (ED) [9], elastic matching is exploited in DTW to achieve invariance in temporal distortion. However, DTW is a global distance measure that discards the matching information [8]. In addition, DTW could match two series that have dissimilar local structures [10].
Inspired by the elastic matching in DTW, an elastic matching mechanism combined with CNN called Elastic Matching CNN (EM-CNN) is proposed in this paper. Instead of lock-step alignments between the time series and convolutional kernels as CNN, a matching matrix is used to adaptively learn the alignments between these in the EM-CNN. The EM-CNN is an architecture that learns the matching relationship and convolutional kernel simultaneously. The primary contributions of this paper are concluded as follows:
  • An elastic matching mechanism is proposed to measure the similarity between the time series and convolutional kernels. This mechanism can be extended to different architectures based on the CNN.
  • The experiments performed on 85 University of California, Riverside (UCR) time series datasets [11] demonstrate that the proposed mechanism improves the performance of CNN on classification tasks.
The remainder of this paper is organized as follows. This paper briefly reviews the related work in Section 2. In Section 3, an elastic matching mechanism is proposed to learn the matching relationship between the time series and convolutional kernels. Next, the experiments are performed on 85 UCR datasets, and the results are analyzed in Section 4. Additional discussion is presented in Section 5. Finally, a conclusion is provided in Section 6.

2. Related Work

2.1. Dynamic Time Warping

Dynamic time warping is a point-to-point matching method to measure the similarity between two different time series. In general, DTW allows a time series to be “stretched” or “compressed” to provide a better match with another time series [12]. Finding a better match in DTW is equivalent to finding an optimal path in the warping matrix with certain restrictions and rules. A dynamic programming algorithm is used to obtain the cumulative distance of the optimal path. A smaller cumulative distance results in a higher similarity between two time series.
The point-to-point matching in DTW is dependent on the value differences between two points. A point of a series could map a further point or multiple points of other series, leading to misclassification, especially in such applications as image retrieval [13]. Constraint techniques, such as Sakoe–Chuba [14] Band and Itakura Parallelogram [15] are introduced to DTW to reduce the matching space. Weighted DTW [12] considers the phase differences besides value differences to penalize the further points which are probably outliers. Derivative DTW [16] and shapeDTW [10] encode the local neighborhood information rather than the values at a point to measure the similarity between two points.

2.2. Dynamic Time Warping with the Convolutional Neural Network

The artificial neural network (ANN) is famous for its powerful feature extraction capability in the last decades. Recently, ANNs such as the RNN and CNN, have been used to learn supervised [17] or unsupervised representation [18] for time series analysis. The RNN is well-known for time series forecasting [17] with the advantage of sequential learning. Some improvements are proposed to reduce inference time [19] and predict sudden time-series changes [20]. Although, the RNN is also exploited in the TSC, the CNN achieves better performance in supervised learning on the UCR archive [21]. The CNN, such as the fully convolutional network (FCN) and residual network (ResNet) [22], have achieved strong baselines for TSC. Some attempts have been made to combine DTW and CNN to overcome the brittleness to temporal distortions in the conventional CNN. These attempts are roughly categorized into two categories. The first category, DTW, is a preprocessing method to transform the raw time series. Then, the transformed series are used as inputs to the CNN. In [8], a multimodal fusion CNN (MMF-CNN) is employed to predict a label for the multidimensional time series. The multidimensional time series are composed of the coordinate features and local distance features which are extracted by measuring the DTW similarity between the original time series and prototypes. The second category directly incorporates the DTW into the CNN and training an end-to-end classification framework. In [23], DTW is used to determine a more optimal alignment between convolutional kernels and time series. The DTWNet [7] replaces the inner product kernel with the DTW kernel against the Doppler effect and improves the capability to do feature extraction.

3. Proposed Method

3.1. Elastic Matching in Dynamic Time Warping

Elastic matching in DTW is first reviewed to better demonstrate the proposed mechanism. Considering two different time series X = ( x 1 , x 2 , x i , x n ) T and W = ( w 1 , w 2 , w j , w m ) T , a dynamic programming algorithm composed of Equations (1) and (2) is used to decide which points should be matched. The second point in X could match the third and fourth points in W (red rhombuses in Figure 1) using DTW. Compared with lock-step matching (blue circles in Figure 1), used in ED, the matching relationship in DTW is data-dependent and elastic:
D T W ( X , W ) = c ( i , j ) ,
where c ( i , j ) is the cumulative distance:
c ( i , j ) = | x i w j | 2 + m i n { c ( i 1 , j 1 ) , c ( i 1 , j ) , c ( i , j 1 ) }

3.2. Elastic Matching in the Convolutional Neural Network

The CNN extracts features from the time series by measuring the local similarity between the time series X = ( x 1 , x 2 , x i , x n ) T and convolutional kernel W = ( w 1 , w 2 , w j , w m ) T . In general, the similarity measure adopted in the CNN is the inner product. Considering the definition of the inner product, the matching mechanism of the inner product is similar to the ED. A point of one series only matches the point of another series in the same position. Hence, the inner product is inappropriate to measure similarity for temporal distortion. An elastic matching mechanism is incorporated into the inner product to better model the matching relationship between the time series and convolutional kernels. The elastic matching mechanism allows the kernel points to construct relationships with points in different positions of the time series. The similarity of the ith location is defined by Equation (3). The convolutional layer combined with the elastic matching mechanism is called the matching convolutional (MConv) layer. The structure of the MConv layer is presented in Figure 2. A fully-connected (FC) layer is used to learn the matching relationship between the series and kernels. The weights of the FC layer in Figure 2 correspond to the matching matrix M in Equation (3).
S i m i l a r i t y i = W T M X i : i + m ,
where m is the length of a convolutional kernel, and M is an m × m matching matrix.
When M is an identity matrix, Equation (3) degenerates to the inner product, and Equation (3) can be considered an extension of DTW. The proof is as follows.
Without loss of generality, this proof is based on the example in Figure 1. The time series and convolutional kernel have the same length in a sliding window for the CNN. Red rhombuses represent the optimal path in Figure 1. Hence, series X = ( x 1 , x 2 , x i , x 7 ) T and kernel W = ( w 1 , w 2 , w j , w 7 ) T are transformed to X and W as shown in Equation (4), respectively:
X = ( x 1 , x 1 , x 2 , x 2 , x 3 , x 4 , x 4 , x 5 , x 5 , x 6 , x 7 ) T W = ( w 1 , w 2 , w 3 , w 4 , w 4 , w 5 , w 6 , w 6 , w 7 , w 7 , w 7 ) T .
If a dot product is used to measure the similarity between two points, the DTW similarity between X and W is equivalent to the inner product between X and W as presented in Equation (5):
S i m i l a r i t y D T W ( X , W ) = X · W .
Equation (5) can be further expressed as a matrix multiplication as indicated in Equation (6):
S i m i l a r i t y D T W ( X , W ) = W T M X ,
where M is a binary matrix and satisfies the conditions as shown in Equation (7):
M i , j = 1 , x i , j X a n d w i , j W 0 , o t h e r w i s e .
Comparing Equations (3) and (7), DTW is a special case of the proposed matching mechanism.

3.3. EM-CNN

In this section, three EM-CNN architectures including elastic matching FCN (EM-FCN), elastic matching ResNet (EM-ResNet) and elastic matching Inception (EM-Inception) are proposed. The architectures in Figure 3 and Figure 4 are EM-FCN and EM-ResNet. The backbone of the EM-FCN and EM-ResNet are FCN and ResNet which are strong baselines for TSC [22]. The EM-FCN is similar to FCN; the difference between FCN and EM-FCN is the convolutional layers in FCN are replaced by the MConv layers in EM-FCN. The EM-FCN comprises three basic modules, one global average pooling (GAP) layer, and one FC layer. Each basic module contains one MConv layer, one batch normalization (BN) layer, and one Rectified Linear Unit (ReLU) layer. The kernel sizes and numbers of kernels corresponding to the three MConv layers are 8, 5, and 3 and 128, 256, and 128, respectively.
Compared with the EM-FCN, the EM-ResNet is deeper. It has three bottlenecks, which consist of three basic modules like EM-FCN. The number of kernels corresponding to three bottlenecks is 64, 128, and 128, respectively. Convolutional layers are only replaced by MConv layers in the residual branches for stable training.
Compared with the EM-FCN and EM-ResNet, EM-Inception (Figure 5) is based on Inception [24] which extracts features in a multiscale manner. Inception is composed of two bottlenecks, one GAP layer, and one FC layer.
Each bottleneck has three basic Inception modules. Multiple paralleled convolutional operators of different kernel sizes in conjunction with a max-pooling operator are performed in each module in Figure 6. Like EM-ResNet, a shortcut connection is used between the consecutive bottlenecks, and the MConv layers only take the place of the convolutional layers in the residual branches.
The MConv layer is the core of the EM-CNN. The matching matrix M in the MConv layer is learned by backpropagation. Similar to the derivation in [25,26], the details for calculating the gradients needed for the backpropagation algorithm are as follows. Figure 2 illustrates that, assuming the response in each location for the MConv layer is y, and the optimized objective function J ( W , M ) is as follows:
J ( W , M ) = m i n W , M { 1 2 y W T M X 2 } .
Equation (8) becomes the objective function of CNN if M is an identity matrix. The gradient descent for the CNN is easy to calculate:
d W t d t = X ( y W t T X ) T W t .
Similar to the derivation of Equation (9), we let W ^ = W T M using the chain rule, and the gradient descent for the EM-CNN can be calculated as shown in Equation (10):
d W t ^ d t = M t T M t X T ( y W t T X ) + X T ( y W t T X ) W t T W t ,
where W 0 is initialized using the Xavier method, and M 0 is initialized with an identity matrix.

4. Experiments

In this section, experiments are performed on the UCR archive to validate the effectiveness of the elastic matching mechanism.

4.1. Hyperparameter Settings

The EM-FCN, EM-ResNet, and EM-Inception were tested on 85 ‘bake-off’ datasets on the UCR archive. Default train/test split was used as [22,24] to train the model and evaluate the performance. The matching matrix M was changed per layer and initialized using an identity matrix. The Adam optimizer was used to train the EM-FCN (2000 epochs), EM-ResNet (1500 epochs) and EM-Inception (1500 epochs) with the initial learning rate of 0.001, β 1 = 0.9 , β 2 = 0.999 and ϵ = 1 × 10−8. The best model corresponding to the minimum training loss as [21] is used to evaluate the architecture generalization over the testing sets.

4.2. Metrics

The evaluation metrics to compare the performance of different methods are the accuracy ratios on each dataset, number of Win, average arithmetic ranking (AVG-AR), average geometric ranking (AVG-GR), and mean per-class error (MPCE). The definition of the MPCE is presented in Equation (11):
P C E k = e k c k M P C E i = 1 K P C E k
where k refers to each dataset and i represents each method, K is the number of datasets, c k and e k are the number of categories and error rates for the k-th dataset, respectively.
The critical difference defined by Equation (12) is also tested to compare different methods statistically over multiple datasets [27]. A critical difference diagram was proposed to visualize this comparison where a cluster of methods (a clique) connected by a thick horizontal line are not-significantly different in terms of accuracy [24].
C r i t i c a l D i f f e r e n c e = q α N c ( N c + 1 ) 6 K ,
where the critical value q α is the Studentized range statistic divided by 2 , N c is the number of methods. The value of α is set to 0.05 in the experiments.

4.3. Evaluation on the UCR Archive

The first experiment compares the EM-FCN, EM-ResNet, and EM-Inception with FCN, ResNet, and Inception to demonstrate the effectiveness of the elastic matching mechanism for the CNN. Table 1 and Figure 7 indicate that CNN architectures with the elastic matching mechanism exhibit better performance than lock-step matching. Compared with other methods, EM-Inception obtains the best rank in all the metrics.
The second experiment is to validate that the elastic matching mechanism is suitable to address the temporal distortion. The compared methods surveyed in this experiment consist of the following: DTW [28], shapeDTW [10] and DTW feature (DTW-F) [29], Edit distance (Subsequence (LCSS) distance [30], Edit Distance with Real Penalty (ERP) [31], Time warp edit (TWE) distance [32] and Move–Split–Merge (MSM) [33]), Ensembles of elastic distance measures (EE) [34], Hierarchical Vote Collective of Transformation-based Ensembles (HIVE-COTE) [35], Time warping invariant Echo State Networks (TWIESN) [36], MMF-CNN [8] and EM-Inception. The results in Table 2 and Figure 8 indicate that EM-Inception achieves a comparable performance with HIVE-COTE (the state-of-the-art method on the UCR archive). Moreover, HIVE-COTE is an ensemble method based on 35 different classifiers, including DTW-1NN, MSM-1NN, and others. It has a robust ability to address temporal distortion. Hence, the experimental results also reflect the effectiveness of the proposed mechanism. Compared with other methods, such as MMF-CNN and shapeDTW, the superiority of EM-Inception demonstrates that an end-to-end learning architecture with an elastic matching mechanism is preferred.

4.4. Effects of the Different Numbers of Layers

The models EM-FCN(2) and EM-FCN(1) are generated by EM-FCN to analyze the effects of the number of layers. In addition, EM-FCN(2) removes the third basic module of EM-FCN, and EM-FCN(1) removes the second and third modules of EM-FCN, simultaneously. The same technique is used to generate FCN(2) and FCN(1) from FCN. As illustrated in Figure 9, regardless of the number of layers, architectures based on EM-FCN are superior to the corresponding architectures based on the FCN. The performance difference between the EM-FCN(2) and FCN is small. It indicates that a deep CNN architecture could be replaced by a shallow CNN architecture based on the elastic matching mechanism to mitigate overfitting to small datasets.

4.5. Effects of the Different Kernel Sizes

The kernel size of EM-FCN(1) is 8, which is relatively small for large-scale patterns. In this experiment, the kernels are enlarged from 8 to 20 and 40 to generate EM-FCN(1,20) and EM-FCN(1,40), respectively, which have large receptive fields. Moreover, FCN(1,20) and FCN(1,40) are generated from FCN(1) in the same way. As depicted in Figure 10, even when the kernel size is 40, EM-FCN(1,40) still improves the performance of FCN(1,40). The results demonstrate that the elastic matching mechanism strengthens the feature extraction capability of CNN architectures for multiple scales.

4.6. Effects of the Different Kernel Initialization

In this section, an experiment is performed to compare the EM-CNN and kernel-varying EM-CNN (KEM-CNN). In the KEM-CNN, the matching matrix M is the independent initialization for each kernel. In addition, KEM-FCN, KEM-ResNet, and KEM-Inception are the EM-FCN, EM-ResNet, and EM-Inception models with varying matching matrices M for each kernel, respectively. The comparison between the EM-CNN and KEM-CNN on 85 UCR datasets is as follows.
Intuitively, KEM-CNN should be better than EM-CNN due to the larger modeling capacity. However, Figure 11a–c indicates that no matter what backbone is used, the EM-CNN wins on more datasets than KEM-CNN on the UCR archive. Furthermore, another comparison based on the MPCE (a lower value indicates better performance) is made, and the same result is observed in Figure 11d. These results prove that KEM-CNN is more prone to overfitting on the UCR archive.

4.7. Computational Complexity

Compared to the conventional CNN, the extra parameters added in the EM-CNN come from the matching matrix M. The parameters learned in matrix M are proportional to the corresponding kernel size S l . Moreover, the number of matrices M is proportional to the number of kernels N l used in each layer and in layers L in the network. Hence, the overall parameter N p added in EM-CNN is as presented in Equation (13):
N p = l = 1 L S l 2 × N l .
Compared to the parameter learned in the convolutional layers, as in Equation (14), the parameter added by the matching matrix M is at least min l { S l } times N c o n v . Thus, the EM-CNN can overfit on the UCR archive. Therefore, the matching matrix M of the EM-CNN is fixed on each layer in the experiment. An experimental comparison between the EM-CNN and KEM-CNN is conducted in the next section to confirm the necessity of this:
N c o n v = l = 1 L S l × N l .

5. Discussion

From the results shown in Table 2 and Figure 7, the performance of the EM-FCN, EM-ResNet, and EM-Inception are better than the FCN, ResNet, and Inception, respectively. Nevertheless, it should be noted that the EM-CNN is not better than the corresponding CNN in all the datasets and the base architecture is important to the performance. It is more helpful to combine the elastic matching mechanism with the CNN in the “motion” datasets such as “InlineSkate”, “UwaveGestureLibraryAll” because it is common for different people to perform the same movement for different durations. Moreover, as shown in Figure 9, the improvement from the elastic matching mechanism decreases as the number of layers increases. The reason is that temporal distortion is adjusted layer by layer. In the limiting case, if the temporal distortion disappears at some layer, it is expected that the EM-CNN degenerates to the CNN, and performance improvement also disappears.
Besides, EM-CNN is a static model because the matching matrix is fixed after the training is completed. Hence, it is not an optimal solution in theory. The probable solution is to train another auxiliary network to adjust the matching matrix according to the different inputs. Furthermore, despite the results shown in Figure 11, KEM-CNN still has a larger capacity to model the nonlinear relationship between the time series and convolutional kernels, in theory, it is meaningful to apply the KEM-CNN to the large-scale datasets.

6. Conclusions

In this paper, an elastic matching mechanism was proposed to learn the matching relationship between the time series and convolutional kernels. Experiments on the EM-FCN, EM-ResNet and EM-Inception show that this elastic matching mechanism is appropriate to assist CNN to model the nonliear alignment between the time series and convolutional kernels. As presented in the discussion, this elastic matching mechanism is also beneficial to CNN with a different number of layers and convolutional kernel sizes. Compared with the conventional CNN, the extra computational complexity from this elastic matching mechanism is small which ensures this elastic matching mechanism is flexible. In future work, we will consider combining the dynamic filter with the elastic matching mechanism to more complex applications such as multivariable time series classification and clustering.

Author Contributions

Methodology, K.O.; supervision, Y.H. and S.Z.; writing—original draft, K.O.; writing—review and editing, Y.H. and Y.Z. All authors have read and agreed to the published version of the manuscript.

Funding

This work was funded by the National Natural Science Foundation of China under Grant No. 61903373.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

The data used in this study is from the UCR archive which can be found here: http://www.timeseriesclassification.com/.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Liu, C.L.; Hsaio, W.H.; Tu, Y.C. Time series classification with multivariate convolutional neural network. IEEE Trans. Ind. Electron. 2018, 66, 4788–4797. [Google Scholar] [CrossRef]
  2. Ordóñez, F.J.; Roggen, D. Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition. Sensors 2016, 16, 115. [Google Scholar] [CrossRef] [Green Version]
  3. Graves, A.; Jaitly, N.; Mohamed, A. Hybrid speech recognition with deep bidirectional LSTM. In Proceedings of the 2013 IEEE Workshop on Automatic Speech Recognition and Understanding, Olomouc, Czech Republic, 8–13 December 2013; pp. 273–278. [Google Scholar]
  4. Übeyli, E.D. Wavelet/mixture of experts network structure for EEG signals classification. Expert Syst. Appl. 2008, 34, 1954–1962. [Google Scholar] [CrossRef]
  5. Lee, D.J.; Schoenberger, R.B.; Shiozawa, D.; Xu, X.; Zhan, P. Contour matching for a fish recognition and migration-monitoring system. In Two-and Three-Dimensional Vision Systems for Inspection, Control, and Metrology II; International Society for Optics and Photonics: Bellingham, WA, USA, 2004; Volume 5606, pp. 37–48. [Google Scholar]
  6. Fawaz, H.I.; Forestier, G.; Weber, J.; Idoumghar, L.; Muller, P.A. Deep learning for time series classification: A review. Data Min. Knowl. Discov. 2019, 33, 917–963. [Google Scholar] [CrossRef] [Green Version]
  7. Cai, X.; Xu, T.; Yi, J.; Huang, J.; Rajasekaran, S. DTWNet: A dynamic time warping network. In Advances in Neural Information Processing Systems; The MIT Press: Cambridge, MA, USA, 2019; pp. 11640–11650. [Google Scholar]
  8. Iwana, B.K.; Uchida, S. Time series classification using local distance-based features in multi-modal fusion networks. Pattern Recognit. 2020, 97, 107024. [Google Scholar] [CrossRef]
  9. Serra, J.; Arcos, J.L. An empirical evaluation of similarity measures for time series classification. Knowl.-Based Syst. 2014, 67, 305–314. [Google Scholar] [CrossRef] [Green Version]
  10. Zhao, J.; Itti, L. shapedtw: Shape dynamic time warping. Pattern Recognit. 2018, 74, 171–184. [Google Scholar] [CrossRef] [Green Version]
  11. Dau, H.A.; Bagnall, A.; Kamgar, K.; Yeh, C.C.M.; Zhu, Y.; Gharghabi, S.; Ratanamahatana, C.A.; Keogh, E. The UCR time series archive. IEEE/CAA J. Autom. Sin. 2019, 6, 1293–1305. [Google Scholar] [CrossRef]
  12. Ding, H.; Trajcevski, G.; Scheuermann, P.; Wang, X.; Keogh, E. Querying and mining of time series data: Experimental comparison of representations and distance measures. Proc. VLDB Endow. 2008, 1, 1542–1552. [Google Scholar] [CrossRef] [Green Version]
  13. Jeong, Y.S.; Jeong, M.K.; Omitaomu, O.A. Weighted dynamic time warping for time series classification. Pattern Recognit. 2011, 44, 2231–2240. [Google Scholar] [CrossRef]
  14. Sakoe, H.; Chiba, S. Dynamic programming algorithm optimization for spoken word recognition. IEEE Trans. Acoust. Speech Signal Process. 1978, 26, 43–49. [Google Scholar] [CrossRef] [Green Version]
  15. Itakura, F. Minimum prediction residual principle applied to speech recognition. IEEE Trans. Acoust. Speech Signal Process. 1975, 23, 67–72. [Google Scholar] [CrossRef]
  16. Górecki, T.; Łuczak, M. Using derivatives in time series classification. Data Min. Knowl. Discov. 2013, 26, 310–331. [Google Scholar] [CrossRef] [Green Version]
  17. Passalis, N.; Tefas, A.; Kanniainen, J.; Gabbouj, M.; Iosifidis, A. Deep Adaptive Input Normalization for Time Series Forecasting. IEEE Trans. Neural Netw. Learn. Syst. 2020, 31, 3760–3765. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  18. Franceschi, J.Y.; Dieuleveut, A.; Jaggi, M. Unsupervised scalable representation learning for multivariate time series. In Advances in Neural Information Processing Systems; The MIT Press: Cambridge, MA, USA, 2019; pp. 4650–4661. [Google Scholar]
  19. Dennis, D.K.; Acar, D.; Mandikal, V.; Sadasivan, V.; Simhadri, H.; Saligrama, V.; Jain, P. Shallow RNNs: A method for accurate time-series classification on tiny devices. In Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, BC, Canada, 8–14 December 2019. [Google Scholar]
  20. Vincent, L.; Thome, N. Shape and time distortion loss for training deep time series forecasting models. In Advances in Neural Information Processing Systems; The MIT Press: Cambridge, MA, USA, 2019; pp. 4189–4201. [Google Scholar]
  21. Wang, J.; Wang, Z.; Li, J.; Wu, J. Multilevel wavelet decomposition network for interpretable time series analysis. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, London, UK, 19–23 August 2018; pp. 2437–2446. [Google Scholar]
  22. Wang, Z.; Yan, W.; Oates, T. Time series classification from scratch with deep neural networks: A strong baseline. In Proceedings of the 2017 International Joint Conference on Neural Networks (IJCNN), Anchorage, AK, USA, 14–19 May 2017; pp. 1578–1585. [Google Scholar]
  23. Iwana, B.K.; Uchida, S. Dynamic Weight Alignment for Convolutional Neural Networks. arXiv 2017, arXiv:1712.06530. [Google Scholar]
  24. Fawaz, H.I.; Lucas, B.; Forestier, G.; Pelletier, C.; Schmidt, D.F.; Weber, J.; Webb, G.I.; Idoumghar, L.; Muller, P.A.; Petitjean, F. Inceptiontime: Finding alexnet for time series classification. Data Min. Knowl. Discov. 2020, 34, 1936–1962. [Google Scholar] [CrossRef]
  25. Gunasekar, S.; Woodworth, B.; Bhojanapalli, S.; Neyshabur, B.; Srebro, N. Implicit regularization in matrix factorization. In Proceedings of the 2018 Information Theory and Applications Workshop (ITA), San Diego, CA, USA, 11–16 February 2018; pp. 1–10. [Google Scholar]
  26. Liu, W.; Liu, Z.; Rehg, J.M.; Song, L. Neural similarity learning. In Advances in Neural Information Processing Systems; The MIT Press: Cambridge, MA, USA, 2019; pp. 5025–5036. [Google Scholar]
  27. Demšar, J. Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res. 2006, 7, 1–30. [Google Scholar]
  28. Bagnall, A.; Lines, J.; Bostrom, A.; Large, J.; Keogh, E. The great time series classification bake off: A review and experimental evaluation of recent algorithmic advances. Data Min. Knowl. Discov. 2017, 31, 606–660. [Google Scholar] [CrossRef] [Green Version]
  29. Kate, R.J. Using dynamic time warping distances as features for improved time series classification. Data Min. Knowl. Discov. 2016, 30, 283–312. [Google Scholar] [CrossRef]
  30. Vlachos, M.; Kollios, G.; Gunopulos, D. Discovering similar multidimensional trajectories. In Proceedings of the 18th International Conference on Data Engineering, San Jose, CA, USA, 26 February–1 March 2002; pp. 673–684. [Google Scholar]
  31. Chen, L.; Özsu, M.T.; Oria, V. Robust and fast similarity search for moving object trajectories. In Proceedings of the 2005 ACM SIGMOD International Conference on Management of Data, Baltimore, MD, USA, 14–16 June 2005; pp. 491–502. [Google Scholar]
  32. Marteau, P.F. Time warp edit distance with stiffness adjustment for time series matching. IEEE Trans. Pattern Anal. Mach. Intell. 2008, 31, 306–318. [Google Scholar] [CrossRef] [Green Version]
  33. Stefan, A.; Athitsos, V.; Das, G. The move-split-merge metric for time series. IEEE Trans. Knowl. Data Eng. 2012, 25, 1425–1438. [Google Scholar] [CrossRef] [Green Version]
  34. Lines, J.; Bagnall, A. Time series classification with ensembles of elastic distance measures. Data Min. Knowl. Discov. 2015, 29, 565–592. [Google Scholar] [CrossRef]
  35. Lines, J.; Taylor, S.; Bagnall, A. Time series classification with HIVE-COTE: The hierarchical vote collective of transformation-based ensembles. ACM Trans. Knowl. Discov. Data 2018, 12. [Google Scholar] [CrossRef] [Green Version]
  36. Tanisaro, P.; Heidemann, G. Time series classification using time warping invariant echo state networks. In Proceedings of the 2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA), Anaheim, CA, USA, 18–20 December 2016; pp. 831–836. [Google Scholar]
Figure 1. Example of elastic matching in dynamic time warping (DTW); the yellow rectangles are the beginning and end of the two paths; blue circles and red rhombuses represent the optimal paths in Euclidean distance and DTW, respectively.
Figure 1. Example of elastic matching in dynamic time warping (DTW); the yellow rectangles are the beginning and end of the two paths; blue circles and red rhombuses represent the optimal paths in Euclidean distance and DTW, respectively.
Algorithms 14 00192 g001
Figure 2. Structure of the matching convolutional (MConv) layer.
Figure 2. Structure of the matching convolutional (MConv) layer.
Algorithms 14 00192 g002
Figure 3. Architecture of EM-FCN, MConv1D denotes the one-dimensional MConv layer.
Figure 3. Architecture of EM-FCN, MConv1D denotes the one-dimensional MConv layer.
Algorithms 14 00192 g003
Figure 4. Architecture of EM-ResNet, MConv1D denotes the one-dimensional MConv layer.
Figure 4. Architecture of EM-ResNet, MConv1D denotes the one-dimensional MConv layer.
Algorithms 14 00192 g004
Figure 5. Architecture of EM-Inception.
Figure 5. Architecture of EM-Inception.
Algorithms 14 00192 g005
Figure 6. Architecture of the EM-Inception module, MConv1D denotes the one-dimensional MConv layer. The input of the EM-Inception module is filtered by three Mconv1D with 32 kernels which sizes are 10, 20, and 40, respectively, and one max-pooling layer (stride = 3) followed by a 1 × 1 convolutional layer. The output of the EM-Inception module is a feature map with 128 channels.
Figure 6. Architecture of the EM-Inception module, MConv1D denotes the one-dimensional MConv layer. The input of the EM-Inception module is filtered by three Mconv1D with 32 kernels which sizes are 10, 20, and 40, respectively, and one max-pooling layer (stride = 3) followed by a 1 × 1 convolutional layer. The output of the EM-Inception module is a feature map with 128 channels.
Algorithms 14 00192 g006
Figure 7. Critical difference diagram of a pairwise statistical difference comparison of FCN, ResNet, Inception, EM-FCN, EM-ResNet and EM-Inception on the UCR archive.
Figure 7. Critical difference diagram of a pairwise statistical difference comparison of FCN, ResNet, Inception, EM-FCN, EM-ResNet and EM-Inception on the UCR archive.
Algorithms 14 00192 g007
Figure 8. Critical difference diagram of a pairwise statistical difference comparison of 12 methods on the UCR archive.
Figure 8. Critical difference diagram of a pairwise statistical difference comparison of 12 methods on the UCR archive.
Algorithms 14 00192 g008
Figure 9. Critical difference diagram of a pairwise statistical difference comparison of EM-FCN and FCN with different numbers of layers.
Figure 9. Critical difference diagram of a pairwise statistical difference comparison of EM-FCN and FCN with different numbers of layers.
Algorithms 14 00192 g009
Figure 10. The critical difference diagram of a pairwise statistical difference comparison of EM-FCN and FCN with different kernel sizes.
Figure 10. The critical difference diagram of a pairwise statistical difference comparison of EM-FCN and FCN with different kernel sizes.
Algorithms 14 00192 g010
Figure 11. Comparison between the EM-CNN and KEM-CNN. (a) shows the comparison between EM-FCN and KEM-FCN, (b) shows the comparison between the EM-ResNet and KEM-ResNet, (c) shows the comparison between the EM-Inception and KEM-Inception, (d) shows the MPCE based on different backbone.
Figure 11. Comparison between the EM-CNN and KEM-CNN. (a) shows the comparison between EM-FCN and KEM-FCN, (b) shows the comparison between the EM-ResNet and KEM-ResNet, (c) shows the comparison between the EM-Inception and KEM-Inception, (d) shows the MPCE based on different backbone.
Algorithms 14 00192 g011
Table 1. Evaluation metrics involving 85 time series datasets on the UCR archive. The values before the last four rows represent the accuracy ratios comparison on each dataset and the last four rows represent the Number of Win, AVG-AR, AVG-GR and MPCE comparison between different methods.
Table 1. Evaluation metrics involving 85 time series datasets on the UCR archive. The values before the last four rows represent the accuracy ratios comparison on each dataset and the last four rows represent the Number of Win, AVG-AR, AVG-GR and MPCE comparison between different methods.
DatasetFCNEM-FCNResNetEM-ResNetInceptionEM-Inception
Adiac0.84140.85170.83320.81590.83120.8261
ArrowHead0.84340.87430.83770.81600.82290.8457
Beef0.68000.86670.75330.85330.66670.8667
BeetleFly0.91000.85000.85000.87000.75000.8500
BirdChicken0.94001.00000.88000.90000.95000.9500
Car0.91330.93330.91670.92660.86670.9333
CBF0.99380.99110.99580.99890.99441.0000
ChlorineConcentration0.81650.82370.85280.84110.85960.8898
CinCECGTorso0.82880.90870.83780.80430.86450.8159
Coffee1.00001.00001.00001.00001.00001.0000
Computers0.81920.80000.80560.80800.78000.7560
CricketX0.79440.76410.79900.79740.82820.8436
CricketY0.79280.76670.81030.83590.84100.8513
CricketZ0.80970.75380.80870.82050.83330.8692
DiatomSizeReduction0.34640.50980.95100.96410.93140.9575
DistalPhalanxOutlineAgeGroup0.71800.71220.71800.74100.74820.7410
DistalPhalanxOutlineCorrect0.76010.74640.77030.73910.77900.7645
DistalPhalanxTW0.69500.66910.66330.64030.66910.6403
Earthquakes0.72520.74100.71220.71940.72660.6906
ECG2000.88800.88000.87400.84000.92000.9100
ECG50000.94000.93870.93510.94180.93690.9438
ECGFiveDays0.98540.97790.96630.97331.00001.0000
ElectricDevices0.70650.72310.72790.72830.70210.7081
FaceAll0.93750.93310.86670.94970.79640.8231
FaceFour0.92950.86360.95450.93180.95450.9659
FacesUCR0.94340.93900.95420.94780.96340.9654
FiftyWords0.64570.68130.74020.74950.80440.8462
Fish0.96110.97710.98060.99430.98290.9714
FordA0.91410.97050.93700.93560.95530.9545
FordB0.77230.79140.81310.80740.86790.8630
GunPoint1.00001.00000.99071.00001.00001.0000
Ham0.70670.72380.75810.75000.72380.7810
HandOutlines0.79890.64860.91350.92970.94590.9351
Haptics0.48960.53250.50970.55840.56490.5325
Herring0.64380.59380.60000.62500.67190.5781
InlineSkate0.33160.50550.37710.39820.46550.4855
InsectWingbeatSound0.39190.38590.49930.54550.63280.6409
ItalyPowerDemand0.96290.96020.96150.96020.95530.9689
LargeKitchenAppliance0.90290.89870.90130.90130.90400.9067
Lightning20.73440.72130.78030.73770.80330.8689
Lightning70.82470.69860.82740.83560.80820.8082
Mallat0.96710.95740.97360.97530.94290.9710
Meat0.80330.93330.99000.98330.91670.9667
MedicalImages0.77840.77240.76970.77240.79080.8000
MiddlePhalanxOutlineAgeGroup0.53510.48700.54550.53250.54550.5260
MiddlePhalanxOutlineCorrect0.79450.79040.82610.80760.81440.7938
MiddlePhalanxTW0.50130.48700.49480.54550.52600.4740
MoteStrain0.93580.94490.92400.93130.88260.8962
NonInvasiveFetalECGThorax10.95830.95780.94140.94810.96180.9496
NonInvasiveFetalECGThorax20.95310.95730.94360.94350.95880.9542
OliveOil0.72000.86670.84670.88000.83330.9000
OSULeaf0.97850.94210.98020.99170.92560.9463
PhalangesOutlinesCorrect0.81770.80300.84520.81930.83800.8310
Phoneme0.32800.33600.33340.36230.32490.3191
Plane1.00001.00001.00001.00001.00001.0000
ProximalPhalanxOutlineAgeGroup0.82540.85850.84680.87320.85370.8390
ProximalPhalanxOutlineCorrect0.90650.91070.91960.91410.93470.9244
ProximalPhalanxTW0.76100.76590.77270.79020.78540.7854
RefrigerationDevices0.49650.51470.53010.53600.54130.5440
ScreenType0.62190.60270.61550.56800.57070.5680
ShapeletSim0.70560.86670.78220.91440.98330.8833
ShapesAll0.89400.89500.92630.91830.91500.9367
SmallKitchenAppliances0.77710.77870.78130.79200.76800.7653
SonyAIBORobotSurface10.95840.95840.96070.92710.85020.9534
SonyAIBORobotSurface20.98030.96430.97540.96640.94540.9423
StarLightCurves0.96500.97450.97230.97450.97890.9492
Strawberry0.97510.97570.98000.97030.98110.9568
SwedishLeaf0.96740.97760.96260.96480.94720.9760
Symbols0.95540.95480.89310.97590.98290.9769
SyntheticControl0.98870.99330.99671.00000.99331.0000
ToeSegmentation10.96140.95610.95700.96490.95610.9737
ToeSegmentation20.88920.88460.89380.89230.94620.9462
Trace1.00001.00001.00001.00001.00001.0000
TwoLeadECG0.99951.00001.00001.00000.99560.9991
TwoPatterns0.87050.87581.00001.00001.00001.0000
UWaveGestureLibraryX0.75380.78310.78120.79290.81300.8275
UWaveGestureLibraryY0.64250.68010.66580.67780.75010.7493
UWaveGestureLibraryZ0.72670.75150.74860.76070.74820.7510
UWaveGestureLibraryAll0.81790.82100.86080.87830.94220.9764
Wafer0.99720.99820.99810.99890.99820.9977
Wine0.61110.79630.72220.73700.75930.7963
WordSynonyms0.56110.56900.61660.63950.73200.7508
Worms0.78180.80520.76100.72730.75320.8182
WormsTwoClass0.74290.75320.74810.71430.79220.6883
Yoga0.83720.87600.86670.87200.90530.9237
Number of Win91710212435
AVG-AR4.15293.62353.45883.05882.89412.7177
AVG-GR3.67153.04603.09362.58622.34122.1272
MPCE0.05150.04800.04530.04430.04280.0417
Table 2. Performance of 12 different methods on the UCR archive.
Table 2. Performance of 12 different methods on the UCR archive.
DTW-1NNERP-1NNLCSS-1NNMSM-1NNTWE-1NNDTW-F
Number of Win232224
AVG-AR7.94127.94127.55297.03537.75296.0706
AVG-GR7.36667.20106.86276.31747.18565.2437
MPCE0.06920.06720.06950.06600.06860.0592
EEHIVE-COTETWIESNMMF-CNNshapeDTWEM-Inception
Number of Win630126533
AVG-AR5.12942.56479.84714.07066.81182.5647
AVG-GR4.53572.06729.13622.79905.52981.9884
MPCE0.05980.04110.08210.04260.05960.0417
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Ouyang, K.; Hou, Y.; Zhou, S.; Zhang, Y. Convolutional Neural Network with an Elastic Matching Mechanism for Time Series Classification. Algorithms 2021, 14, 192. https://doi.org/10.3390/a14070192

AMA Style

Ouyang K, Hou Y, Zhou S, Zhang Y. Convolutional Neural Network with an Elastic Matching Mechanism for Time Series Classification. Algorithms. 2021; 14(7):192. https://doi.org/10.3390/a14070192

Chicago/Turabian Style

Ouyang, Kewei, Yi Hou, Shilin Zhou, and Ye Zhang. 2021. "Convolutional Neural Network with an Elastic Matching Mechanism for Time Series Classification" Algorithms 14, no. 7: 192. https://doi.org/10.3390/a14070192

APA Style

Ouyang, K., Hou, Y., Zhou, S., & Zhang, Y. (2021). Convolutional Neural Network with an Elastic Matching Mechanism for Time Series Classification. Algorithms, 14(7), 192. https://doi.org/10.3390/a14070192

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop