Next Article in Journal
The Roadmap of Graphene-Based Sensors: Electrochemical Methods for Bioanalytical Applications
Previous Article in Journal
Recent Development of Fluorescent Nanodiamonds for Optical Biosensing and Disease Diagnosis
 
 
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Feature-Based Information Retrieval of Multimodal Biosignals with a Self-Similarity Matrix: Focus on Automatic Segmentation

by 1,2,*,†, 1,2,*,†, 1,3,†, 3,†, 2 and 1,*
1
Laboratory for Instrumentation, Biomedical Engineering and Radiation Physics, NOVA School of Science and Technology, Campus da Caparica, 2829-516 Caparica, Portugal
2
Cognitive Systems Lab, University of Bremen, Bibliothekstraße 1, 28359 Bremen, Germany
3
Associação Fraunhofer Portugal Research, Rua Alfredo Allen 455/461, 4200-135, Porto, Portugal
*
Authors to whom correspondence should be addressed.
These authors contributed equally to this work.
Biosensors 2022, 12(12), 1182; https://doi.org/10.3390/bios12121182
Received: 1 October 2022 / Revised: 14 December 2022 / Accepted: 15 December 2022 / Published: 19 December 2022
(This article belongs to the Special Issue Advances in Biometrics and Biosensors Technologies and Applications)

Abstract

:
Biosignal-based technology has been increasingly available in our daily life, being a critical information source. Wearable biosensors have been widely applied in, among others, biometrics, sports, health care, rehabilitation assistance, and edutainment. Continuous data collection from biodevices provides a valuable volume of information, which needs to be curated and prepared before serving machine learning applications. One of the universal preparation steps is data segmentation and labelling/annotation. This work proposes a practical and manageable way to automatically segment and label single-channel or multimodal biosignal data using a self-similarity matrix (SSM) computed with signals’ feature-based representation. Applied to public biosignal datasets and a benchmark for change point detection, the proposed approach delivered lucid visual support in interpreting the biosignals with the SSM while performing accurate automatic segmentation of biosignals with the help of the novelty function and associating the segments grounded on their similarity measures with the similarity profiles. The proposed method performed superior to other algorithms in most cases of a series of automatic biosignal segmentation tasks; of equal appeal is that it provides an intuitive visualization for information retrieval of multimodal biosignals.

1. Introduction

In recent years, the continuous increase in accessible wearable technology has contributed to a significant amount of data available. Continuous data collection from wearable devices through mobile phones, smartwatches, hearables, wristbands and other non-invasive wearable sensors has provided a valuable volume of information. As reported in Tankovska et al., wearable devices’ usage has more than doubled between 2016 and 2019, reaching 722 million, each of which relies on high-quality data acquisition and processing [1].
The data recorded by wearable devices carry information in the form of time series, which typically has an ordered structure. The displayed mechanics may be the expected or consequence of the nature of the acquisition environment, e.g., controlled laboratory experiments or natural scenarios. Researchers, such as data scientists, are interested in understanding the structure of the recorded signals, the meaning behind them, and the influences of the context. For instance, in the context of audio signals from musical pieces, it is helpful to acknowledge the different structural components for distinguishing the intro from the chorus and the bridge. Shifting to biosignals-related domains, such as Human Activity Recognition (HAR), the melody can change and evolve from walking to jogging ( Biosensors 12 01182 i001)—this Accelerometer (ACC) signal contains two main periodic regimes, which could be segmented as WW…W and JJ…J. Another example comes from the Electrocardiogram (ECG), a typical physiological signal. The standard cyclic nature of the ECG, interesting to most users, may be affected by several sources, such as motion artefacts, muscular contractions or even symptomatic events. For instance, the signal piece Biosensors 12 01182 i002 has two cycles of an ECG disturbed by noise for which we can interpret that it has three segments, and the first segment is very similar to the third one, i.e., ABA.
The examples mentioned above manifest the relevance and importance of the following approaches:
  • Novelty segmentation: to identify significant changes in the signal’s behaviour.
  • Periodic segmentation: to detect the presence of repeating cyclic patterns.
  • Labelling: to measure how similar the segments are between each other.
This study explored and proposed the solution to these three problems mentioned above, inspired by a method used for audio signal analysis and thumbnail generation [2,3,4,5]. A moderately similar operation was introduced for speech recognition [6]. Surprisingly, such a method has not yet been extended to other types of time series domains that could greatly benefit from it [7]. The method uses a feature-based Self-Similarity Matrix (SSM) of (multidimensional) time series, from which visual and analytical information is rendered to perform the segmentation process and associate subsequences of the time series with each other.
Figure 1 shows a teaser example of how the SSM works. The time series has seven segments (A, B, C, D, E, F and G), divided into three different groups of sub-sequences (1: [A, C, E, G], 2: [B, D] and 3: [F]), among which groups 1 and 3 are periodic.
As conceptualized in Figure 1, by applying the proposed solution, the following tasks were handled intuitively:
  • Novelty search: the signal is segmented into homogeneous groups by the novelty function.
  • Periodic search: the periods in the signal are segmented by the similarity function.
  • Similarity measurement: the similarity between segments can be reasonably explained by the colour of the corresponding sub-sequence pair on the matrix, and the values can be used to cluster the segments by similarity, as illustrated on the dendrogram of Figure 1 (right).
In this article, the effectiveness of the proposed method as a powerful tool for unsupervised signal analysis will be explored in a set of examples in different domains and levels of information (novelty, periodicity and similarity). The results of the novelty segmentation will be validated and compared to public benchmarks.

1.1. Essentials

The segmentation of time series has a subset of extensive and relevant applications, transverse to all domains. The task of dividing a time series into segments is context- and purpose-oriented, depending on different interest levels of instants or events for correctly selecting breaking points on the time series. Before delving into more details, the main concepts frequently used in this study must be clarified:
  • Time Series: A time series is a sequence of real values ordered in time with length n N : T = ( t 1 , t 2 , , t n ) . A biosignal is a category of time series. Several data domains rely on the multidimensional time series acquisition from one sensor’s multiple axes, such as an ACC’s three directions, or multiple sources, such as an Inertial Motion Unit (IMU) that fuses three different sensors.
  • Multidimensional Time Series: A multidimensional time series is a set of k N time series belonging to the same acquisition: { T 1 , T 2 , , T k } . Segments of interest, called subsequences, are often searched inside a time series.
  • Subsequence: A subsequence is a segment of a (multidimensional) time series with size w N , starting from a given position i and ending at position i + w. Therefore, two instants defined as events, delimit a subsequence in time.
  • Event: An event is an instant in time e that indicates the presence of a relevant occurrence in the time series. Multiple events segment the time series into several subsequences of different lengths. Hence, event detection is often considered time series segmentation or change point detection [8]. To be clear, we will use the terms event detection and segmentation when discussing our methods, but we can eventually use the term change point detection when comparing with other methods.
  • Moving Window: A moving window is a process of sliding along a time series T to apply a specific method on each subsequence it hovers, a common strategy used in time series data mining to find relevant subsequences and events. The window has, similar to the subsequence, a predefined size w N , which starts at a given position i and ends at position i + w. The process of moving windows is iterative, and windows can overlap each other. The following window will start at i + o, where o  [ 1 , w ] is the overlapping size ( o = 1 for a total overlap and o = w for no overlap). On each moving window from each subsequence of the (multidimensional) time series, features can be extracted to form a feature series.
  • Feature Series: A feature series is a feature representation of a time series with size m = n w o that depends on the overlap size o N of the moving window. In the case of a multidimensional time series, the feature series stack a multifeature series with size f k , m . Multiple features extracted from one dimension or various dimensions are grouped into a feature matrix.
  • Feature Matrix ( F M ): A feature matrix with size r × ( k × m ) , represents that each of the k dimensions produces r features. This feature matrix, which characterizes the (multidimensional) time series in statistical, temporal or spectral domains, is used to compute the self-similarity matrix.
  • Self-Similarity Matrix (SSM): A self-similarity matrix is a pairwise distance matrix between each subsequence of the time series. In this study, it is calculated by the dot product between the F M and its transpose. The SSM reveals several meaningful structures that indicate the presence of events and measures how similar segmented subsequences are. Applying the SSM, we highlight functions for the novelty segmentation and periodic segmentation process, and also introduce the similarity profiles derived from the similarity values of the SSM.
  • Novelty Segmentation: A change point event or segmentation point indicates a significant instant in time between subsequence i and subsequence i + 1, segmenting the time series. The novelty function computes such events, detailed in Section 4.3.1.
  • Periodic Segmentation: A periodic event segments the periods of a cyclic time series into subsequences. The similarity function computes these events, detailed in Section 4.3.2.
  • Similarity Profiles: A similarity profile is a time series that exhibits how similar one segmented subsequence is to all other subsequences in the time series, which helps organize the subsegments by similarity level.

1.2. Search Ranks

Figure 2 illustrates the search ranks of the problem formed by three layers:
  • Dimensionality: The search can be applied to one or multiple time series. In multidimensional space, some events can coincide in several time series, while others are specified on a particular dimension. For example, some gestures produce noticeable signals on only one dimension of the three-axis ACC.
  • Timescale: Events’ occurrence can vary from different timescales. For example, when the signal being analysed is zoomed in from hour to minute scales, some events may disappear while new events may be detected.
  • Representation: The searchable objects can be straightforwardly the temporal nature of time series or other representations, such as frequency or other extracted features.
Besides the ranks mentioned above, the search procedure can be customized by context or target, which is highly related to the relevance given to an event or a subsequence. Types of events that are considered significant include:
  • Property change: The change of a property, such as a change in mean or frequency, or a set of properties is greater than a threshold, e.g., Biosensors 12 01182 i003.
  • Peak/Valley: Peaks and valleys can typically be associated with significant physical changes, e.g., ECG peaks such as Biosensors 12 01182 i004.
  • Periodicity: The starting points of each period in a periodic signal are considered relevant, e.g., Arterial Blood Pressure (ABP) periods such as Biosensors 12 01182 i005.
  • Recurrent pattern: Re-occurrences of similar subsequences with specific patterns should be of interest. Unlike periodicity, recurrent patterns do not have a temporal regularity, e.g., the arrhythmias found in an ECG signal such as Biosensors 12 01182 i006.
  • Anomaly: Highly dissimilar subsequences with particular patterns are of the reference value, e.g., noise in a clean signal such as Biosensors 12 01182 i007.

1.3. Proposed Method

In order to fill as many research gaps as possible, this study started by defining the search space, considering that if the time series is transformed in the feature space, any feature’s change would be relevant. For instance, changes in the mean, standard deviation, frequency or other properties are all options worth searching for. By characterizing the signal in the feature space, we can explore changes in all feature representations. Additionally, an event should separate two different behaviours. The notion of difference in time series can be associated with distance/similarity, enabling finding segmentation points, recurrent patterns, anomalies, and periodic shapes.
Therefore, we propose an unsupervised methodology that searches for events (1) in uni- and multidimensional space, (2) with a fixed timescale and potential multi-timescale application opportunities, and (3) on an SSM computed by a feature space representation of the time series. The events to be searched are any changes in the SSM related to a segmentation point and/or a periodic event.
The proposed method’s reliability for event detection will be evidenced by considerable experiments in various type-agnostic databanks of multiple time series domains and comparisons to state-of-the-art methods. It should be highlighted that events in different datasets are extracted from the same information source, i.e., SSM.

2. Related Literature

2.1. Applications

We live in an “era of big data” [9]. As mentioned in Section 1, wearable sensors are currently available on a large scale, promoting the acquisition of massive amounts of data. Datasets of this size can no longer be handled by trivial means and call for engineers and data scientists with expertise in data mining, machine learning, and data analysis [10]. This increase in wearable usage has also been seen in industrial environments, which is motivated by the current trend of Industry 4.0 [11], promoting the use of sensors to monitor in real-time their machines for damage prevention, and their workers for occupational-related disorder prevention and productivity improvements [12,13]. Research areas such as intelligent rehabilitation [14,15,16,17,18,19], advanced sensing technologies [20], orthotics [21,22,23,24], sports science [25,26,27,28,29,30,31,32,33,34,35], activity modelling [36,37,38,39,40], exoskeletons [41,42], psychological stress classification [43], machine learning edutainment [44], surgical index monitoring [45,46], and fall detection [47,48,49], have also leveraged the power of biosignals from wearables.
In research aspects of time series analysis, biosignals produced by various types of sensors require the data science community to develop tools to extract meaningful information for the acquisition, including reporting, pattern recognition, event detection, periodic signal segmentation, and classification, among other data mining tasks [50,51]. The availability of more reliable data and practical information is more beneficial, primarily as machine learning is increasingly applied. Numerous fields could benefit from our proposed methods, including physiological event detection for healthcare (e.g., noise, sleep problems, and epilepsy), biomedical signal analysis of ECG/Electroencephalogram (EEG)/Electromyogram (EMG), climate change detection, audio-based automatic speech segmentation and recognition, motion sequence segmentation, behaviour transition detection, human activity research, feature space study, and manufacturing industries, among others.
Ultimately, data preparation is essential for data analysis and machine learning application development. After data acquisition, the effort in data processing and preparing implies challenges, an active research subject. One of the critical issues in data preparation is the lack of labelled data. Labelling data is a sensitive and time-consuming process, of which the complexity rises with the data quantity. Nevertheless, accurately labelled data are essential for data analysis and model training: In [52], the authors stated that data scientists rely solely on a small portion of the available datasets because it is too expensive to label all the data. Such a thought reflects how paramount it is to have solutions that can improve the existing data labelling strategies to reduce labour, time costs, and ground truth quality.

2.2. General Segmentation Approaches

Prior works in event detection focus on change point detection or segmentation, where the strategies are categorized as online versus offline, univariate versus multivariate, model-based versus non-parametric, and unsupervised versus supervised [8,53,54].

2.2.1. Supervised Methods

Supervised methods include multi-class, binary and virtual classifiers optimized to detect change points [53], where the nature of the change can be provided as an additional advantage. However, supervised methods rely upon brittle training sets and class imbalance since there are more in-state sequences than change point sequences [53]. An additional problem reported by [8] is that most algorithms’ performance was validated in synthetic data, where the given nature of the application was not optimal. In response, a benchmark is available for change point detection [8], where methods can be compared on real data. This study applies this benchmark as a reference of state-of-the-art methods.

2.2.2. Unsupervised Methods

Existing classic non-supervised methods in change point detection, such as the Bayesian Online Change Point Detection (BOCPD) [55,56], Binary Segmentation (BINSEG) [57], and Segmentation Neighbourhood (SegNeigh) [58], are witnessed to be able to perform state-of-the-art applications in various domains [8]. BOCPD is a non-supervised model-based method for change point detection that was simultaneously introduced by Fearnhead and Liu [56] and Adams and MacKay [55]. The method infers a change point based on the fact that the model parameters before and after the change point are independent. It relies on learning a joint probability distribution since the time of the most recent change point (run length) by means of a recursive message-passing algorithm. The calculated recursive probabilities will be tested to evaluate if the run length will be zero. If so, a change point is detected; if not, the run length continues to increase [8,55]. This method needs hyperparameter tuning for sound performance [8]. The BINSEG method is a greedy sequential algorithm, recursively partitioning the signal into smaller segments. The position where the signal is segmented is typically selected where the cost function is minimized. BINSEG has not been reported to cope with a multi-timescale change [8,57]. The available repository [8] collecting the implementation of some offline methods [54] above lacks a visual output that can provide users with the location of the change points. In this work, we used the benchmark dataset available in this repository to compare the performance of the proposed solution with the mentioned methods.
Window-based segmentation, typically relying on a sliding window that is divided into two smaller windows based on the comparison using a cost function, can apply to real data domains [54]. Another approach, called Fast Low-cost Online Semantic Segmentation (FLOSS) [59,60], searches regime changes based on the nearest neighbours of subsequences, which allows the similarity comparison between segments for the segmentation and summarization of long-term time series.

2.3. Biosignals’ Segmentation Approaches

Some general approaches mentioned in Section 2.2.1 and Section 2.2.2 do not necessarily work well for biosignal segmentation tasks. Emerging works focus specifically on biosignal segmentation, e.g., applying neural networks (NN) to ECG signals. In [61], an NN with transfer learning was used for the segmentation of periodic biosignals (motion and ECG). Convolutional NN has also been found for ECG segmentation. In [62], a convolutional NN on a binary classification task (heartbeat or not a heartbeat) was proposed, while Aman Malali et al. put forward a convolutional long short-term memory (LSTM) NN for the same task [63]. Last but not least, Viktor Moskalenko et al. used a UNet-like full convolutional NN for the ECG signal segmentation of P and T waves, as well as the QRS complex [64]. Without model training, the ECG segmentation can also be solved through a subsequence search in the context of a carefully selected query pattern [65].
Wearable technology has also improved the field of gait analysis, increasing interest in gait event detection [66]. Recently, machine learning approaches have been found for gait segmentation related to Parkinson’s disease [66], and hidden Markov models (HMM) were also used for the same purposes [67]. Gait event detection methods based on rapid positive changes in Gyroscope (GYR) data were employed for rehabilitation research [68,69]. The work from Matteo et al. shows the ability of deep learning techniques to improve gait segmentation [66]. Traditional signal processing methods, such as the integral of the signal envelope, can be applied to EMG signal segmentation for gait analysis [70].
Biosignals’ segmentation also facilitates medical research. For the study of sleep staging segmentation, Mathias Perslev et al. introduced a fully convolutional network [71]. A square-root velocity function to segment periodic data for posterior alignment and the statistical analysis helped disease classification based on the Karcher mean [72].

2.4. Segmentation with the Self-Similarity Matrix (SSM)

The SSM has been used for segmentation in the audio domain, based on a feature representation of the audio signal [73]. The advantage of the SSM is that it provides a considerable amount of information for a specific timescale. This study promotes SSM concepts and applications from the audio domain to other time series domains. The proposed method can detect events with context, associating the estimated events with patterns, (dis)similarities, periodicity and novelty, and a possible extension is the task of summarization. The search mechanism is primitively based on a specific timescale and can evolve recursively to perform multi-timescale searches.

3. Datasets

In order to test and validate the proposed method, we applied public datasets with segmentation requirements. As a multimodal, complex patterned, and versatile type of time series signals, sensor-based biosignals are the experimental target of this study: inertial signals for motion in the domain of HAR, EMG for onset/offset detection and ECG for noise detection. The datasets and their sources are described below.

3.1. Dataset 1—HAR

Each participant of the 30-subject dataset [74,75] was wearing a Samsung Galaxy A2 smartphone on his/her waist while performing the following activities: (1) Walking, (2) Walking Upstairs, (3) Walking Downstairs, (4) Sitting, (5) Standing and (6) Laying. Each activity was performed for approximately 60 s and labelled. The device records the internal ACC and GYR data at a constant rate of 50 Hz.
The ACC of the dataset was used to search for segmentation points on the signal. The usage of both sensors was not possible because the ACC and GYR signals are not synchronized with each other. The absence of average variation makes the switch between the static poses (4)–(6) listed above not evident for GYR. Therefore, we only used the ACC channels in this study.
Generally, ACC is recognized as one of the most helpful wearable sensors in multimodal biosignal-based HAR. For instance, [76] demonstrates that ACC’s HAR performance outperforms other kinds of sensors. Each activity label uses the same timestamp as the corresponding signals. We defined the ground truth for segmentation borders as the switch of labels.

3.2. Dataset 2—ECG1

The dataset [77,78] comprehends 12 half-hour ECG acquisitions and 3 half-hour collections of noise typical in ambulatory ECG recordings. The noise recordings were collected from physically active volunteers using standard ECG trackers, leads, and electrodes. The three noise records were assembled from the recordings by selecting intervals containing predominantly baseline wander (in record “bw”), muscle (EMG) artefact (in record “ma”), and electrode motion artefact (in record “em”). Two selected clean ECG signals were noised with different Signal-to-Noise-Ratio (SNR).
This dataset was used in the context of change point detection to validate the proposed method for estimating transitions to and from signal sections with added noise. The subsequences with standardized noise, annotated by an expert in time series data mining, were applied as our experimental ground truth.

3.3. Dataset 3—ECG2

The dataset [78,79] for studying ECG’s motion artefacts and sparsity encloses short-duration ECG signals at 500 Hz recorded from a healthy 25-year-old male performing the physical activities of standing, walking, and single jumping.
This dataset was used in the context of change point detection to validate the proposed method for estimating transitions to and from sections with noise added due to a jump.

3.4. Dataset 4—EMG

A Myo Thalmic bracelet worn on the user’s forearm was applied for the 36-subject dataset [80]. The bracelet is equipped with eight sensors equally spaced around the forearm that simultaneously acquire EMG signals at 200 Hz. Each participant performed two recording series, each consisting of six three-second basic gestures with a three-second pause between each gesture pair.
In the context of change point detection, this dataset helped validate the proposed method for estimating transitions between the activation and relaxation of the muscular activity. Each activity label uses the same timestamp as the corresponding signals. We defined the ground truth for segmentation borders as the switch of labels.

3.5. Dataset 5—CPDBenchmark

For an objective evaluation, we also compared the proposed method with existing approaches on a change point detection benchmark [8], comprising several time series from real-domain contexts. The repository was built by the Alan Turing Institute for the evaluation of change point detection algorithms.
This dataset has ground truth events for each time series. In addition, the available performance of several existing approaches was compared to the proposed method’s results on the same time series. Our proposed method is also foreseen to be competent in other sorts of unidimensional or multidimensional time series, and related studies are on the agenda.

3.6. Dataset 6—BVP (for Illustrative Instances)

In the dataset [81], ten subjects’ slow tilt, rapid tilt and standing-up activities were monitored and recorded with ECG and Blood Volume Pressure (BVP) to investigate how the two physiological signals respond to the angular changes during the activities [78,81].
The dataset’s BVP channel was used as an example to demonstrate the proposed algorithm’s capability in detecting pattern-based physiological changes in a distinctive research instance. The ground truth of the changes is marked by the angular signal, suggesting the moments a tilt or standing-up activity occurred.

3.7. Dataset 7—ECG Pulsus Paradoxus (for Illustrative Instances)

The signal used was extracted from the dataset available in the UCR Semantic Segmentation Benchmark [60]. The signal represents an ECG recorded from a patient who had an onset of pulsus paradoxus [82,83].
The signal, with a regime change at the 10,000 th sample, was used as an illustrative application scenario for the proposed method.

4. Method

The extraction of relevant events from time series starts by computing the SSM. As explained in Section 1.1, the SSM has relevant structural information to retrieve events, namely blocks, paths and similarity profiles. Figure 3 summarizes the calculation steps for the SSM.

4.1. Feature Extraction

The structural information on the SSM reflects how informative the feature set can translate the signal’s changes and disruptions. Behavioural changes may be related to a variate set of features. As a feature can be sensitive to a particular type of change, the set of features should be diverse to identify a multivariate set of events and be agnostic to various signal types. We turned to the available features from the Time Series Feature Extraction Library (TSFEL) [84] for new, which has been proven effective and efficient in previous work on multimodal biosignal processing [36,85,86] and other research fields [87,88,89,90,91]. We selected over 50% of all TSFEL features in the statistical, temporal, and frequency domains with relatively lower computational costs, as listed in Table A1 in Appendix A, regarding our proposed method’s high calculation resource consumption.
The features are extracted with a moving window with size w, specified by the user, with an overlap percentage o. The selection of the two sizes significantly influences the results: w defines the timescale at which features are extracted so that the wider the window, the more zoomed-out the search will be. The second parameter defines the pixel resolution of the resulting feature series, increasing the amount of information with a larger overlap.
The extracted features are grouped into a feature matrix ( F M ), where the rows represent a feature series and the columns correspond to all subsequences. In the multidimensional case, r features extracted from each of the k dimensions are ordered in the F M as rows, forming r × k elements in each row, as illustrated in Figure 3.
Each feature series (rows of the F M ) is z-normalized for a more balanced contribution to characterizing the signal. A further normalization is applied to the feature vector (columns of the F M ), which optimizes the cosine similarity computation between feature vectors by simply adding the dot product to calculate the SSM.

4.2. Feature-Based SSM

After grouping all the features extracted, the next stage is to apply a similarity measure to the feature space and compute the SSM. This process consists in comparing each subsequence with all the other subsequences. Since each column of the F M is each subsequence’s feature characterization in the entire feature set, the SSM, i.e., the comparison between segments, is obtained by calculating the dot product between the z-normalized transposed F M and itself:
S S M = F M T · F M .
The dot product scores the similarity based on the subsequence’s feature values. Cells of the SSM with higher similarity scores indicate that the corresponding subsequences have similar feature values [3,4]. As a result, the SSM provides rich visual information, highlighting structures that describe the signal’s morphological behaviour over time and structure, such as blocks and paths.
In Figure 4, the main structures are illustrated and highlighted in an example of an SSM [3] computed from an ABP signal, where the main structures are blocks and paths. Our proposed method utilizes the resulting main structures to extract the desired information.
Paths show recurrence of patterns, which indicates the morphological matching between corresponding subsequences. Circles in the sf layer exhibit when the paths start. The cross-pattern in block C means that the subsequences are periodic and symmetric.
Differently, blocks are square-shaped structures of homogeneous areas in the SSM, translated as constant behaviour in the time series. The change between block structures along the main diagonal displays a relevant change in morphology and behaviour in the time series. In Figure 4, the SSM is segmented into several blocks on layer nf, for which the Δs mark the change points that separate blocks A, B and C. Besides paths and blocks, the SSM provides similarity measures between subsequences, which can be used to spotlight (dis)similar segments, such as anomalies, motifs or cycles. Several strategies were applied to the SSM to extract the mentioned information.

4.3. Information Retrieval

The SSM is a powerful visual tool per se, exposing relevant information that a raw observation could miss. Automatic discovery of information of interest will increase the SSM’s practicability and versatility, for which three approaches for information retrieval on the SSM are put forward: (1) novelty search of block transitions, (2) periodic pattern search of paths, and (3) similarity profiles of subsequences.

4.3.1. Novelty Search

The search for novelty is inspired by a method used in musical structure analysis [92], which is computed with the help of the libfmp Python package [93]. The process involves searching for transitions between blocks using a moving chequerboard square matrix, resulting in a one-dimensional function: the novelty function.
As shown in Figure 5, block transitions along the diagonal are represented by a chequerboard pattern. Such patterns can be detected by correlating a standard chequerboard matrix with the diagonal of the SSM, for which a sliding squared matrix, designated kernel, is used. The kernel incorporates a Gaussian function with a smoothing factor. The kernel K N combines two different square matrices: K H and K C . K H is responsible for identifying the homogeneity of the SSM on each side of the centre – the more homogeneous the pattern is, the higher the corresponding values will be. K C measures the cross-similarity level. Therefore, when sliding the kernel K N along the diagonal, a higher correlation value will be returned when it reaches a segment of the SSM with a similar chequerboard pattern. The result is the mentioned novelty function [2,73,94].
In position A of Figure 5 (right), due to the high homogeneity, the kernel returns a value approaching 0 when summing the product between it and the section of the SSM it overlaps. In contrast, the kernel in position B reaches a segment with low cross-similarity and high diagonal similarity, which results in high correlation values with a chequerboard pattern. Therefore, high novelty function values are witnessed in these transition segments [2,73,94].
Each section of the kernel has the same size, L N , and D = 2 × L + 1 configures the total kernel size. The kernel has an odd size to adapt zero values in centred points, and a total size of D × D . K N is defined by [2,73]:
K N ( i , j ) = s i g n ( a i ) · s i g n ( b j ) ,
where a , b [ L : L ] and s i g n represent the sign function (1, 0 or −1). A radially symmetric Gaussian function is used to smooth the Kernel [2,73]:
ϕ ( p , u ) = exp ( 1 2 L σ 2 ( p 2 + u 2 ) ) ,
where σ is the standard deviation, equal for both x and y dimensions of the matrix, L the size of each kernel’s section, and p and u the position in the x and y dimensions, respectively. The kernel K G is computed by point-wise multiplication with the Gaussian function:
K G = ϕ · K N .
The novelty function  n f is calculated by correlating the kernel with the diagonal of the matrix:
n f ( m ) = i , j = 0 2 L + 1 K G ( a i , b j ) S S M ( m + a i , m + b j ) ,
being the sample of the novelty function m [ 0 N ] and a , b [ L : L ] . The change point events are represented by local maxima (peaks) in the novelty function, which can be detected by standard peak-finding strategies.

4.3.2. Periodic Search

As aforementioned, paths indicate the presence of similarity and reoccurring patterns can be visualized on the SSM. The path’s start point punches where the period of the pattern begins. In order to find the periodicity, we compute the similarity function s f by summing the values of the symmetric SSM column-wise or, equally, row-wise. Each element of the s f is calculated by
s f ( x ) = i = 0 m S S M i x ,
where i is the column position for the sum, s f j is the sample of the function at position j and m is the feature-series size. As segments with similar morphology will be similarly described by the extracted features, the columns will have a similar representation, resulting in similar values on the sf. The similarity function will enhance such behaviour when facing periodic series. The identification of events related to the periodicity of a time series is then feasible by searching for local minima (valleys) on the similarity function.
Although not validated in this work, an additional application of the similarity function should be outlooked. Considering that each sample of the s f is an average similarity of a subsequence to all other subsequences, it is possible to find anomalies. Regarding an anomaly as a subsequence highly unique and different from all the rest of the time series, its average similarity to all the other subsequences should have a low value.

4.3.3. Similarity Profiles

The principal elements, blocks and paths, are the information basis for segmenting the time series. Besides, SSM also provides pairwise similarity values between all subsequences of the time series, an important measure that can be used for clustering and motifs/discords discovery. The similarity profiles exploit the similarity values of the SSM to facilitate subsequences comparison. A similarity profile charts the similarity values of a subsequence (one column/row of the SSM) to all the other subsequences. The higher the values, the more similar the subsequences are. In addition to the subsequences’ comparison, the similarity profile can also compare between entire segments of the signal. For instance, all three A-segments highlighted in Figure 4, whose profiles are highly similar despite the different sizes.
Although the segment comparison could be directly based on the region of the SSM delimited by two subsequences, we propose a more effective measure of two segments’ similarity/difference according to their similarities/differences to all the other subsequences. A similarity profile  P s ( c ) of a segment is computed as the column(row)-wise average similarity values of the region delimited by the segment being profiled (size l), and all the other subsequences of the time series (size m):
P s ( c ) = Σ i = 0 l S S M ( i , c ) l .
The similarity profile is computed column(row)-wise. Each column(row) c ( r ) is the average similarity value between the reference segment and the segment corresponding to c. The reasoning is that similar segments should have closer similarity profiles. Since the profiles have the same size, they can be compared by certain distance measures, such as the Euclidean Distance (ED), to form clusters, an automatic clustering solution based on the segments generated by the novelty and similarity functions.

5. Experimental Analysis, Validation, and Discussion

5.1. Illustrative Evaluation in Various Application Scenarios

Experiments from multiple domains were carried on to validate the practicability and universality of the process to represent the time series into a feature-based SSM and the method of retrieving information from the SSM.

5.1.1. Acceleration Signals in Human Activity Domain

Accelerometers are usually considered one of the most effective sensors for wearable-based HAR [76,95]. Figure 6 (top) exemplifies the SSM’s usage on a record of Dataset 1 (HAR, see Section 3.1), where the data of all three ACC axes are applied. The SSM was computed using a 250-sample window size and a 95% overlap. Along the diagonal, the novelty function generates block-wise references for estimating activity transition using a 45-sample kernel.
We can identify in Figure 6 (top) that the detected segmentation points match the activity transitions. Although all transitions are visible on the novelty function, the transitions between similar activity patterns in the walking category (straightforward, upstairs and downstairs) are more challenging to differentiate, as block A suggests, which is plausible since the properties of these segments are morphologically similar.
The proposed unsupervised method automatically and sensitively detects any significant change in properties. As can be found in the yellow-marked part in Figure 6 (top), the period in which the subject was performing the Upstairs activity is affected by other changes in the time series. These are significant and also correspond to block transitions, which are also evident in the novelty function.
When zooming the SSM into segment A in Figure 6 (top), the three activities in the walking category can be effortlessly segmented based on the change points revealed by chequerboard patterns, as the two most prominent peaks in the corresponding novelty function pinpoint (see Figure 6 (bottom left)). In addition, it is noticeable that the matrix segments related to Upstairs/Downstairs are also segmentable into smaller blocks. As the information is not available in the dataset description, we believe they are a flight of stairs.
Questions may arise at this point. Why is the signal periodicity of the three walking activities not evident in Figure 6 (bottom left)? The reason is that the window size used to compute the SSM is relatively large. If features are extracted with a smaller window size closer to the walking period, the paths delineating the pattern recurrence will be visible. Figure 6 (bottom right) shows the SSM built from segment B of the original time series, with a window size of 10 samples and an overlap of 95%. The paths in the matrix enable the periodicity detection with the similarity function s f B .

5.1.2. Arterial Blood Pressure (ABP) Signals in Posture Recognition Domain

Many biomedical signals, such as ECG, ABP, and Respiratory Inductance Pletismography (RESP), contain retrievable structural information such as periodicities. Meanwhile, unexpected changes may occur during the acquisition due to physiological responses, medical disorders, or sensing problems such as noise, interferences, artefacts, and electrode detachment. We visualize two examples of physiological changes in different types of periodic signals.
The ABP signal can vary due to postural changes, as an available experiment at Physionet confirms [78,81]. Figure 7 (top) shows the process of segmenting the ABP signal based on postural changes, where the trapezoidal and the square wave tag the ground truth of slow and fast postural transitions. The proposed strategy well perceives the change points. Observably, the shape of the raw ABP signal in each regime is undistinguishable through the naked eye. Therefore, it is constructive to rely solely on the signal itself to implement postural change detection. It is important to point out that the periodicity of the signal is not visible on the matrix because the features were extracted with a window size of 5000 samples, which is much larger than the period size. A smaller window size of 250 samples in the current scenario allows periodic segmentation, as Figure 7 (bottom) illustrates, where the SSM is computed on the first 10,000 samples of Dataset 7 (ECG Pulsus Paradoxus, see Section 3.7). The resulting similarity function gives prominence to the periodic nature of the ABP signal.
The SSM of Figure 7 (top) also shows which segments are similar to each other. The blue-coloured parts in the matrix indicate high similarity, presenting that segments from the same posture are more similar than between different postures. For further illustrative purposes, we computed the similarity profiles of each segment as if segmented by the novelty function, which evidences that the corresponding sections could be well clustered based on the similarity profiles ( P A = P C and P B = P D ). In the same way, the similarity profiles in Figure 7 (bottom) examine the similarity between segmented subsequences. Profiles with a similar shape can be grouped together ( P A = P C = P E = P G and P B = P D ), which can be applied to automatic clustering, as exhibited in Figure 1.

5.1.3. Electrocardiography (ECG) Signals in Biomedical Domain

Another widely used biomedical signal, ECG, also testifies to the feasibility of our proposed method. The ECG signal in Figure 8 (left) displays the presence of the condition called pulsus paradoxus, an exaggerated fall (>10 mmHg) in the subject’s blood pressure during inspiration [83], which can also occur when the patient changes sleeping posture after heart surgery [60], as the following example relates. Similar to the ABP signal elucidated in Section 3.6, the human eye hardly perceives the change points in ECG signals. Once again, our proposed strategy shows strength.
In addition to the novelty detection, segment A previous to the pulsus paradoxus occurrence can be partially detailed again to reveal minor changes due to additional noise, verifying SSM’s sensitivity to structural changes, as Figure 8 (right) imparts.

5.1.4. Single Channel versus Multidimensionality Application in Multi-Sensor Scenarios

The proposed method accepts both single- and multidimensional records. The difference regards the number of features extracted. As compared in Figure 9, the same set of features is extracted from each time series to build the F M . Using a single or several time series of a multidimensional record is an option, depending on the purpose. In some cases, the use of non-complete dimensions may miss relevant events, as Figure 9 instances the record “Occupancy” from Dataset 5 (CPDBenchmark, see Section 3.5).
The record is a multidimensional time series that measures room occupancy based on temperature, humidity, light, and C O 2 . By comparing the signals and formations in the left and the right parts of Figure 9, it can be understood that some events can be detected using the C O 2 series exclusively, but some are missed.

5.2. Statistical Performance Evaluation

In order to evaluate the performance of our proposed method with biosignals as well as in general scenarios, we applied the algorithm to all datasets introduced in Section 3. The evaluation was divided into biosignals-related applications and the general change point detection benchmark (Dataset 5). The biosignals experiments are associated with public datasets 1–4 from Physionet, the UCI Machine Learning Repository and the UCR Semantic Segmentation Benchmark, involving different contexts (HAR, hand posture, and noise detection) and sensor types (ACC, EMG and ECG).

5.2.1. Metrics for Quantitative Evaluation

The quantitative evaluation on biosignals’ public datasets was made by accumulating true positive ( T P ), false positive ( F P ), and false negative ( F N ) values with a tolerance zone around the ground truth events. The applied reasonable tolerance was the ground truth wrapped by a window size of the SSM computation, inside which a detected event was counted as a T P . The case that no estimated event was found inside the tolerance band was considered an F N . An estimated event outside the tolerance or duplicating an already counted T P was regarded as an F P . The F1-score, based on the precision and recall values, was calculated from T P , F N , and F P values, following Equations (8)–(10):
P r e c i s i o n ( P ) = T P T P + F P
R e c a l l ( R ) = T P T P + F N
F 1 - s c o r e ( F 1 ) = 2 1 P + 1 R = 2 · P · R P + R
In Section 5.2.2 and Section 5.2.3, we present two evaluation layers: (1) evaluation of biosignals’ segmentation and (2) benchmark evaluation. On (1), the performance of our proposed method was compared with existing approaches available on the Python library ruptures, namely the window-based segmentation ( W S ) and the binary segmentation ( B S ) [54] based on the F1-score. The benchmark evaluation referred to the best score obtained from the state-of-the-art methods available on the repository [8]. The evaluation procedure to detect T P , F P , and F N was the one followed on [8]. In addition, we also compare the F1-scores of all methods with a critical distance plot in Figure 10. The plot associates statistical tests over the F1-scores of each method. The test evaluates whether the performance difference is significant (critical difference) or not. In this work, we borrow an implemented critical difference method of [96] that uses the Wilcoxon–Holm test [97], which counteracts the problem of multiple comparisons and calculates pairwise significance between all methods evaluated.
The method has been computed in the same conditions and followed the same procedure for all datasets’ records. The features used were the same for each record (see Appendix A), varying the timescale parameter, the overlap size of the sliding window, and the kernel size. The peak detection strategy based on a threshold mechanism is the same for all records, while the threshold value varies from record to record. Results for publicly available datasets are listed in Table 1 and Table 2, and Table 3 expands the performance by F1-scores in detecting the change point events.

5.2.2. Biosignals’ Segmentation

The illustrative examples provided in Section 5.1 corroborate the proposed method’s capability in segmenting real, complex, and multimodal biosignals datasets. As Table 1 conveys, an overall macro-averaged 0.94 F1-score is achieved, while the competitors’ overall F1-scores are 0.84 ( W S ), and 0.69 ( B S ), respectively. Table 2 broadcasts the F1-scores of our method on the matches against the other two methods in the form of “Win/Draw/Lose”, announcing that it has, in most cases of Datasets 3 (ECG2, see Section 3.3) and 4 (EMG, see Section 3.4), higher F1-score, while loses in Dataset 1 (HAR, see Section 3.1) and draws in most cases for Dataset 2 (ECG1, see Section 3.2). Overall, it has more wins in three of the four tested datasets. Table A2, Table A3, Table A4 and Table A5 in Appendix A detail the parameters of window sizes, kernel sizes, and thresholds applied to the signals in each dataset, as well as the obtained F1-scores. An intuitive graphical comparison can be found in Figure A1, which plots the distribution of F1-scores in each dataset.
For Datasets 1, 3 and 4 (HAR, ECG2, and EMG, see Section 3.1 and Section 3.4 respectively), the window-based methods, novelty and W S , performed much better than the B S method, mainly because the sliding window algorithm with a full set of features comprehensively characterizes changes in the signal. The standard W S uses cost functions searching for mean/variance value changes in the signal, which achieves a high F1-score in Dataset 1 (HAR, see Section 3.1), even identifying transitions between dynamic activities, such as Walking/Upstairs. Our proposed method had a similar performance with a worse count in F P values, where the added features did not improve the segmentation performance. In contrast, our method, complemented by additional features, had a much better performance than the W S method in Dataset 4 (EMG, see Section 3.4). Adding features enabled a more robust and sensitive detection of pattern changes, although it missed some changes between similar patterns, such as Walking and Upstairs/Downstairs in Dataset 1 (HAR, see Section 3.1), which are the primary source of the F N value. Similar to the F N values, the F P values of our proposed method are mostly superior to other methods. It nonetheless leaves room for discussion. Some events are not marked as changes in specific activities, but the signal pattern actually changes. For example, Figure 6 (bottom) exposes pattern changes during an Upstairs/Downstairs activity unlabelled in the ground truth, possibly due to a flight of stairs. The novelty function is sensitive to such pattern changes, which inevitably contributes to the F P values during the comparison with the ground truth. Considering the good performance of both methods, further research should be made in other HAR domains to understand the differences in performance between them better.
Specifically for ECG signals, our proposed method shows its capability on both Datasets 2 (ECG1, see Section 3.2) and 3 (ECG2, see Section 3.3). Although ECG-based jump artefact detection is fundamental, the W S method could not find the segmentation borders, while the B S method worked better. In Dataset 3 (ECG2, see Section 3.3), the same ECG signal was noised with different SNR levels to form a new set of resultant signals. Overall, our proposed method was able to detect the changes between noisy segments and clean segments with noise down to 12 dB. At 6 dB, the proposed method achieved an F1-score of 0.67 with references of 0.64 ( W S ), and 0.34 ( B S ).

5.2.3. Segmentation Benchmark

In order to compare the proposed method with other state-of-the-art approaches, we used a benchmark provided by the Alan Turing Institute [8] (Dataset 5—CPDBenchmark, see Section 3.5). The performance was evaluated by change point event detection in each time series available, summarized in Figure 10, where each referenced method applied its best score in the benchmark (see Table 3 and Table A6).
As unfolded in Figure 10, the critical distance diagram ranks the proposed method second, suggesting no significant difference in performance among methods that only work in uni-dimensional datasets. The global average F1 measure of the proposed method is 0.87 for both uni and multidimensional datasets. Overall, the proposed method had a total of 16 higher F1-scores than the rest, 6 draws, and 12 losses. The two null scores are because no change point was supposed to be found in the corresponding time series.
The results obtained in this benchmark restate that our proposed method is promising, having a performance that competes with several state-of-the-art methods in the problem of novelty segmentation. It should be stressed that the proposed method applies to multidimensional time series, while two of the best-ranked methods in Figure 10 do not. In addition, the proposed method retrieves not solely segmentation points but also higher-level information as periodic changes and cross-segment similarity measures, which is an advantage over the BOCPD.

5.3. Time Complexity

In terms of computation time, the algorithm performs (1) a sliding window to extract features, which is O(n) complexity and (2) performs the dot product between matrices, which is traditionally O( m 2 n ) (recall that r is the number of features and n is the size of the inputted signal). Finally, the correlation of a kernel on the SSM’s diagonal has a complexity of O ( n M 2 ) (recall that M = 2 L + 1 , being the size of the sliding kernel).
The sliding window to extract features has an O ( n ) time complexity. The dot product between matrices has a conventional O ( m 2 n ) time complexity. Expressly, m and n represent the number of features and the inputted signal’s size, respectively, in our proposed method. The correlation computation of a kernel on the SSM’s diagonal has a complexity of O ( n M 2 ) , where the sliding kernel’s size M equals 2 L + 1 .

5.4. Overall Discussion

Several parameters affect the detection results of desired patterns, especially the window size, the overlap percentage, and the kernel size, which influence visual outputs and the novelty function. These parameters can be explained with the analogy of a camera:
  • The window size works like the zoom function, defining the scale of interest in the time series. Larger windows, corresponding to lower zoom values, allow the similarity calculation of longer subsequences, while smaller windows, serving like a zoom-in function, search for local details and unobtrusive changes.
  • The overlap percentage, working as a down-sampler of the time series, is the camera sensor, which determines the image’s pixel resolution. A full resolution of the SSM is only achieved with total overlap, and the lower the overlap percentage, the less accurate are the highlighted changes.
  • The sliding kernel’s size concerns the novelty function’s sharpness of the detected changes. The larger it is, the smoother the output function will be. Potentially, the kernel size can be scaled to the window size, even with a slight accuracy decrease.
With enough computational resources available, the overlap percentage can be maximized so that the SSM can mirror the full details. Admittedly, such an operation is not necessary for many real applications but reduces the variables to facilitate other parameters’ tuning experiments, which is one of our subsequent research topics. The computational resource, i.e., the memory bandwidth and the calculation time (see Section 5.3) is a limitation in this current stage since the SSM increases exponentially with the increase in the time series size. We ascertained that downsampling the time series with a lower overlap percentage is a valid option, advanced by a hierarchical search strategy for addressing the memory limitation, as exemplified in the walking-series instance in Section 5.1.1. Another potential efficiency-enhancing solution is only computing the SSM’s central diagonal with the kernel size corresponding to the interest areas of segmentation borders, which obtains efficiency gains in exchange for the sacrifice of periodicity and similarity measures between subsequences.
A reasonable intuition based on the understanding of signal characteristics should help configure the parameters mentioned above that are fundamental for computing the SSM and the novelty function, as Figure 11 demonstrates a starter example for segmentation purposes. The upper part of Figure 11 draws different SSMs on the same ECG record (A) from Dataset 2 (ECG1, see Section 3.2) computed with sequentially larger window lengths from 0.01 to 2 s. The appropriate window length depends on the purpose of the search:
  • If a small window length, e.g., 0.05 s, is chosen, the novelty function will mostly detail changes within a heartbeat.
  • If opting for a window length approximately equal to the ECG’s PQRS complex, each transition between complexes will be projected.
  • If even larger windows are applied, e.g., 1 or 2 s, the jump artefact will be more significant on the SSM and be spotlighted on the novelty function. Hence, in such a case, the window length of 1 second should be appropriate for segmenting clean versus noisy ECG signals.
When using the same window length on all the other records of the same dataset, the SSM is expected to highlight the same regions of interest. Figure 11 (bottom) characterizes that parameters can be identical when working on the same data type and purpose, but the peak selection on the novelty function is not a matter of convention, which depends on the preset threshold.
The threshold used to determine which peaks are considered points of interest is not relevant to the SSM calculations but is closely related to event detection and automatic segmentation. If the data are a black box, the choice of threshold is a matter of observation and speculation. With informed knowledge of the data, the threshold can be predetermined and experimented with rules based on ranking the detected peaks from highest to lowest:
  • Set the total number (or quantity range) of points of interest as the threshold. e.g., an ECG series with x heartbeats; an ACC series with y recorded gaits of walking activity.
  • Count the total number of peaks and specify a percentage as the threshold. This work takes such an approach because of the diverse datasets and signals involved.
  • Add sliding windows to the novelty function based on the known periodicity information of the time series, and define the number of points of interest expected to be present in each window as the threshold.
Typical signals in specialized fields should have approximately uniform methods and metrics for setting thresholds, and related follow-up studies are on the agenda.

6. Conclusions and Future Work

This article put forward a method based on the self-similarity matrix (SSM) for information retrieval of multimodal time series, with more interest in segmentation and further applications in automatic labelling. The proposed method uses a feature representation of the time series, arguing that a change in the signal can be detected by searching for differences in all the feature dimensions. The presented strategy requires three main parameters that can potentially be reduced to only the sliding window size, which is domain-agnostic and works with multidimensional time series, providing transparent visual intuition for the dynamics of the data. Furthermore, the SSM contains information for the posterior analysis of the subsequences segmented according to the detected events, thus enabling additional advanced applications, such as clustering/labelling based on the distance measures available on the SSM for each subsequence and summarizing the time series profiles of segmentability, periodicity and similarity. The SSM’s computational resource consumption can be reduced by setting variables based on available knowledge of the study objects and fields of interest.
Various application scenarios and types of signals involved in this work validated our proposed method’s high feasibility and usability by witnessing its capacity for novelty segmentation with remarkable performance that stands out among state-of-the-art methods. It is foreseeable that traditional video-based time-series segmentation tools could be upgraded to be more efficient and accessible with the aid of our proposed method.
The future work is widely branching out. A range of studies targeting the effective use of features, such as feature selection, feature stacking, feature space reduction, and high-level feature design, are crucial domain-related research topics. For example, when orienting the research field of HAR, the previous findings of [38,85,98] can be drawn upon in further experiments. Particular pattern changes in certain types of time series may be detected efficaciously by specific features or combinations thereof. Further studies also include, among others, similarity profiles-based automatic segmentation via the extensive use of our promising method, hierarchical segmentation for saving computational resources and better structural analysis, automatic clustering and information summarisation.
One of the most critical efforts should be put into investigating the association between parameters used for the detection of events, namely the window size, the kernel size and the overlap percentage, to reduce the parameter numbers, limit the ranges of attempts, or even provide reference parameter values for different domain-specific signals utilizing fixed-variable and greedy approaches. As we mentioned in Section 5.4, while the overlap percentage can be maximized to save one parameter in the experiments, there is potentially a relationship between the window size and the kernel size, which should be further explored and revealed with domain specificity.
Interfaceisation and softwareisation will enable the proposed method to be widely and practically accessible to help data users from all walks of life. For instance, using a scrollbar to expediently adjust various parameters for the intuitive observation of the SSM pattern transition and the event detection results will bring a qualitative leap forward in various research works, such as data mining, information retrieval, temporal structural analysis, and automatic segmentation and labelling.

Author Contributions

Conceptualization, J.R., H.L., D.F. and D.B.; methodology, J.R.; data curation, J.R. and H.L.; implementation, J.R.; experiment design, J.R., H.L., D.F. and D.B.; evaluation and validation, J.R., H.L., D.F. and D.B.; writing—original draft preparation, J.R.; writing—review and editing, J.R., H.L., D.F. and D.B.; visualization, J.R.; supervision, T.S. and H.G.; funding acquisition, J.R., H.L. and H.G. All authors have read and agreed to the published version of the manuscript.

Funding

The APC was funded by the Open Access Initiative of the University of Bremen and the DFG via SuUB Bremen. This work was also partially funded by Fundação para a Ciência e Tecnologia: PD/BDE/142816/2018; Volkswagen Autoeuropa: PD/BDE/142816/2018; Hanse Wissenschaftskolleg - Institute for Advanced Study: BRAIN Program.

Institutional Review Board Statement

Not applicable for this study considering it used published, peer-reviewed and publicly available datasets.

Informed Consent Statement

Not applicable for this study considering it used published, peer-reviewed and publicly available datasets.

Data Availability Statement

All datasets used in this work are publicly available. Please refer to the link to access each of the used datasets (all accessed on 30 September 2022):

Conflicts of Interest

The authors declare no conflict of interest.

Abbreviations

The following abbreviations are used in this manuscript:
ABPArterial Blood Pressure
ACCAccelerometer
BINSEGBinary Segmentation
BOCPDBayesian Online Change Point Detection
BSBinary Segmentation
BVPBlood Volume Pressure
ECGElectrocardiogram
EDEuclidean Distance
EEGElectroencephalogram
EMGElectromyogram
EOGElectrocularogram
FLOSSFast Low-cost Online Semantic Segmentation
FNFalse Negative
FPFalse Positive
GYRGyroscope
HARHuman Activity Recognition
HMMHidden Markov Model
IMUInertial Motion Unit
LSTMLong Short-Term Memory
NNNeural Networks
RESPRespiratory Inductance Pletismography
SegNeighSegment Neighbourhood
SNRSignal-to-Noise-Ratio
SSMSelf-Similarity Matrix
TNTrue Negative
TPTrue Positive
TSFELTime Series Feature Extraction Library
TSSEARCHTime Series Subsequence Search Library
WSWindow-Based Segmentation

Appendix A. Feature List, Parameter Configurations, and Statistical Results of Segmentation Experiments

Figure A1. F1-scores’ distribution for three segmentation methods on Datasets 1–4. WS: window-based segmentation; BS: binary segmentation; Novelty: the proposed novelty function-based segmentation.
Figure A1. F1-scores’ distribution for three segmentation methods on Datasets 1–4. WS: window-based segmentation; BS: binary segmentation; Novelty: the proposed novelty function-based segmentation.
Biosensors 12 01182 g0a1
Table A1. Features applied in this work for creating the SSMs. The Time Series Feature Extraction Library (TSFEL) is utilized for feature extraction.
Table A1. Features applied in this work for creating the SSMs. The Time Series Feature Extraction Library (TSFEL) is utilized for feature extraction.
Temporal DomainStatistical DomainFrequency Domain
Absolute energyInterquartile RangeEntropy
Area under the curveKurtosisFundamental frequency
CentroidMaximumMax frequency
Cumulative centroidMeanRoll off
DistanceMean absolute deviationRoll on
Maximum peakMedianSpectral distance
Mean absolute differenceMinimumSpectral kurtosis
Mean differenceRoot mean squareSpectral skewness
Median absolute differenceSkewnessSpectral spread
Total energyStandard deviation
Variance
Table A2. Parameter configuration and experimental results of each signal in Dataset 1 (HAR, see Section 3.1). W s i z e : window size; K % : kernel size in percentage of the window size; O % : overlap percentage of the window size. T % : amplitude threshold for event detection; WS: window-based segmentation; BS: binary segmentation. The tolerance used to calculate the F1-score ( F 1 ) equals the window size for novelty function computation.
Table A2. Parameter configuration and experimental results of each signal in Dataset 1 (HAR, see Section 3.1). W s i z e : window size; K % : kernel size in percentage of the window size; O % : overlap percentage of the window size. T % : amplitude threshold for event detection; WS: window-based segmentation; BS: binary segmentation. The tolerance used to calculate the F1-score ( F 1 ) equals the window size for novelty function computation.
Novelty Function-BasedWSBS
Signal W size K % O % T % F 1 W size F 1 F 1
115001.50.900.010.9115000.950.68
215001.50.900.010.9415000.950.72
Table A3. Parameter configuration and experimental results of each signal in Dataset 2 (ECG1, see Section 3.2). W s i z e : window size; K % : kernel size in percentage of the window size; O % : overlap percentage of the window size. T % : amplitude threshold for event detection; WS: window-based segmentation; BS: binary segmentation. The tolerance used to calculate the F1-score ( F 1 ) equals the window size for novelty function computation.
Table A3. Parameter configuration and experimental results of each signal in Dataset 2 (ECG1, see Section 3.2). W s i z e : window size; K % : kernel size in percentage of the window size; O % : overlap percentage of the window size. T % : amplitude threshold for event detection; WS: window-based segmentation; BS: binary segmentation. The tolerance used to calculate the F1-score ( F 1 ) equals the window size for novelty function computation.
Novelty Function-BasedWSBS
Signal W size K % O % T % F 1 W size F 1 F 1
04003.740.550.951.05000.50.5
15002.590.950.951.04001.01.0
21504.000.820.950.82000.50.5
31502.660.320.951.03001.01.0
43002.660.320.951.03001.01.0
53002.660.590.951.01500.51.0
61502.660.640.951.04000.51.0
71502.660.280.951.05001.01.0
81502.660.550.951.02000.51.0
Table A4. Parameter configuration and experimental results of each signal in Dataset 3 (ECG2, see Section 3.3). W s i z e : window size; K % : kernel size in percentage of the window size; O % : overlap percentage of the window size. T % : amplitude threshold for event detection; WS: window-based segmentation; BS: binary segmentation. The tolerance used to calculate the F1-score ( F 1 ) equals the window size for novelty function computation.
Table A4. Parameter configuration and experimental results of each signal in Dataset 3 (ECG2, see Section 3.3). W s i z e : window size; K % : kernel size in percentage of the window size; O % : overlap percentage of the window size. T % : amplitude threshold for event detection; WS: window-based segmentation; BS: binary segmentation. The tolerance used to calculate the F1-score ( F 1 ) equals the window size for novelty function computation.
Novelty Function-BasedWSBS
Signal W size K % O % T % F 1 W size F 1 F 1
112,5000.630.140.951.0050,0000.960.97
212,5001.250.190.950.9750,0000.960.97
325,0002.500.100.950.9250,0000.920.83
450,0000.620.280.950.9350,0000.850.48
550,0000.620.140.950.6750,0000.640.34
612,5000.630.140.951.0050,0000.960.90
712,5001.250.190.951.0050,0000.960.90
812,5001.250.230.951.0025,0000.930.90
912,5001.250.280.951.0025,0000.860.55
1012,5001.250.320.950.9625,0000.710.34
1125,0002.500.410.950.7550,0000.670.34
1212,5001.250.100.951.0050,0000.960.90
Table A5. Parameter configuration and experimental results of each signal in Dataset 4 (EMG, see Section 3.4). W s i z e : window size; K % : kernel size in percentage of the window size; O % : overlap percentage of the window size. T % : amplitude threshold for event detection; WS: window-based segmentation; BS: binary segmentation. The tolerance used to calculate the F1-score ( F 1 ) equals the window size for novelty function computation.
Table A5. Parameter configuration and experimental results of each signal in Dataset 4 (EMG, see Section 3.4). W s i z e : window size; K % : kernel size in percentage of the window size; O % : overlap percentage of the window size. T % : amplitude threshold for event detection; WS: window-based segmentation; BS: binary segmentation. The tolerance used to calculate the F1-score ( F 1 ) equals the window size for novelty function computation.
Novelty Function-BasedWSBS
Signal W size K ( % ) O ( % ) T ( % ) F 1 W size F 1 F 1
120000.620.950.190.9020000.720.29
220001.250.950.280.8615000.750.39
315001.270.950.101.0020000.950.78
415000.630.950.141.0020000.890.49
520001.250.950.280.9215000.850.20
615000.630.950.280.8815000.740.29
720001.250.950.190.9520000.950.49
815001.270.950.100.9320000.850.63
920000.620.950.281.0020000.950.44
1020001.250.950.280.9715000.900.54
1110001.250.950.101.0020000.960.82
1220001.250.950.140.8320000.850.34
1310000.620.950.371.0020000.750.78
1410001.250.950.320.9515000.650.29
1520000.620.950.140.9315000.850.54
1620000.620.950.280.9020000.810.39
1715002.530.950.100.9820000.830.29
1820000.620.950.500.9715000.900.49
1920000.620.950.230.9315000.750.44
2015000.630.950.100.9520000.950.59
2120001.250.950.190.9520000.950.44
2220001.250.950.190.8420000.900.29
2315001.270.950.280.9015000.850.54
2420000.620.950.190.8620000.760.39
2515001.270.950.370.9520000.850.68
2615000.630.950.190.9820000.900.54
2715001.270.950.190.8315000.750.59
2815001.270.950.370.9015000.800.29
2915001.270.950.231.0020000.760.34
3020002.500.950.100.9220000.830.20
3110001.250.950.191.0015000.900.73
3215001.270.950.410.8920000.770.29
3320000.620.950.230.8315000.800.63
3420000.620.950.280.9310000.700.34
3515000.630.950.140.9520000.840.83
3610002.500.950.141.0015000.800.29
Table A6. Parameter configuration and novelty function-based experimental results of each signal in Dataset 5 (CPDBenchmark, see Section 3.5). W s i z e : window size; K % : kernel size in percentage of the window size; T % : amplitude threshold for event detection; The tolerance corresponded to five samples, specified by the benchmark specifications [8].
Table A6. Parameter configuration and novelty function-based experimental results of each signal in Dataset 5 (CPDBenchmark, see Section 3.5). W s i z e : window size; K % : kernel size in percentage of the window size; T % : amplitude threshold for event detection; The tolerance corresponded to five samples, specified by the benchmark specifications [8].
Signal F 1 W size K % T %
apple0.9510500.65
bank0.671001000.90
bee_waggle_60.661002500.80
bitcoin0.6910650.15
brent_spot0.8610300.75
businv0.9320150.70
centralia0.98620.70
children_per_woman0.8810100.70
co2_canada0.8510200.35
construction0.9320500.70
debt_ireland0.97620.70
gdp_argentina0.9720100.70
gdp_croatia1.0020100.70
gdp_iran0.9210100.70
gdp_japan1.0010100.70
global_co20.62100500.70
homeruns0.9315250.50
iceland_tourism0.651501500.99
jfk_passengers0.9820300.80
lga_passengers0.8920300.80
measles0.1720300.80
nile1.0020300.80
occupancy0.9510200.40
ozone0.86640.60
quality_control_11.006200.60
quality_control_21.006200.60
quality_control_31.0020300.80
quality_control_40.9710500.75
rail_lines0.91620.75
ratner_stock0.936500.75
robocalls0.986100.50
scanline_1260070.896100.30
scanline_420490.986200.50
seatbelts0.666200.30
shanghai_license0.9820100.80
unemployment_nl0.82460.27
usd_isk0.916250.30
us_population0.93460.27
well_log0.8110150.40

References

  1. Tankovska, H. Global Connected Wearable Devices 2016–2022. 2020. Available online: https://www.statista.com/statistics/487291/global-connected-wearable-devices/ (accessed on 30 September 2022).
  2. Müller, M. Fundamentals of Music Processing. Springer International Publishing: Cham, Switzerland, 2015. [Google Scholar] [CrossRef]
  3. Paulus, J.; Müller, M.; Klapuri, A. Audio-based Music Structure Analysis. In Proceedings of the International Conference on Music Information Retrieval (ISMIR), Utrecht, The Netherlands, 9–13 August 2010; pp. 625–636. [Google Scholar]
  4. Bello, J.P.; Grosche, P.; Müller, M.; Weiss, R.J. Content-based Methods for Knowledge Discovery in Music. In Springer Handbook on Systematic Musicology; Bader, R., Ed.; Springer: Berlin/Heidelberg, Germany, 2018; pp. 823–840. [Google Scholar] [CrossRef]
  5. Bhandari, G.M.; Kawitkar, R.S.; Borawake, M.P. Audio Segmentation for Speech Recognition Using Segment Features. In ICT and Critical Infrastructure: Proceedings of the 48th Annual Convention of Computer Society of India–Vol II; Satapathy, S.C., Avadhani, P.S., Udgata, S.K., Lakshminarayana, S., Eds.; Springer International Publishing: Cham, Switzerland, 2014; pp. 209–217. [Google Scholar] [CrossRef]
  6. Ney, H. The use of a one-stage dynamic programming algorithm for connected word recognition. IEEE Trans. Acoust. Speech Signal Process. 1984, 32, 263–271. [Google Scholar] [CrossRef]
  7. Agres, K.R.; Schaefer, R.S.; Volk, A.; van Hooren, S.; Holzapfel, A.; Bella, S.D.; Müller, M.; de Witte, M.; Herremans, D.; Melendez, R.R.; et al. Music, Computing, and Health: A roadmap for the current and future roles of music technology for health care and well-being. Music. Sci. 2021, 4, 2059204321997709. [Google Scholar] [CrossRef]
  8. van den Burg, G.J.; Williams, C.K. An evaluation of change point detection algorithms. arXiv 2020, arXiv:2003.06222. [Google Scholar] [CrossRef]
  9. Chen, M.; Mao, S.; Liu, Y. Big data: A survey. Mob. Netw. Appl. 2014, 19, 171–209. [Google Scholar] [CrossRef]
  10. Weiner, J.; Diener, L.; Stelter, S.; Externest, E.; Kühl, S.; Herff, C.; Putze, F.; Schulze, T.; Salous, M.; Liu, H.; et al. Bremen Big Data Challenge 2017: Predicting University Cafeteria Load. In Joint German/Austrian Conference on Artificial Intelligence (Künstliche Intelligenz); Springer: Dortmund, Germany, 2017; pp. 380–386. [Google Scholar] [CrossRef]
  11. Xu, L.D.; Xu, E.L.; Li, L. Industry 4.0: State of the art and future trends. Int. J. Prod. Res. 2018, 56, 2941–2962. [Google Scholar] [CrossRef]
  12. Varandas, R.; Folgado, D.; Gamboa, H. Evaluation of Spatial-Temporal Anomalies in the Analysis of Human Movement. In Proceedings of the 12th International Joint Conference on Biomedical Engineering Systems and Technologies—BIOSIGNALS, Prague, Czech Republic, 22–24 February 2019; pp. 163–170. [Google Scholar] [CrossRef]
  13. Santos, S.; Folgado, D.; Rodrigues, J.; Mollaei, N.; Fujão, C.; Gamboa, H. Explaining the Ergonomic Assessment of Human Movement in Industrial Contexts. In Proceedings of the 13th International Joint Conference on Biomedical Engineering Systems and Technologies—BIOSIGNALS, Valletta, Malta, 24–26 February 2020; INSTICC. SciTePress: Setubal, Portugal, 2020; pp. 79–88. [Google Scholar] [CrossRef]
  14. Liu, H.; Schultz, T. A Wearable Real-time Human Activity Recognition System using Biosensors Integrated into a Knee Bandage. In Proceedings of the 12th International Joint Conference on Biomedical Engineering Systems and Technologies—BIODEVICES, Prague, Czech Republic, 22–24 February 2019; INSTICC. SciTePress: Setubal, Portugal, 2019; pp. 47–55. [Google Scholar] [CrossRef]
  15. Patel, S.; Park, H.; Bonato, P.; Chan, L.; Rodgers, M. A review of wearable sensors and systems with application in rehabilitation. J. Neuroeng. Rehabil. 2012, 9, 21. [Google Scholar] [CrossRef] [PubMed]
  16. Bonato, P. Advances in wearable technology and applications in physical medicine and rehabilitation. J. Neuroeng. Rehabil. 2005, 2, 2. [Google Scholar] [CrossRef]
  17. Sung, M.; Marci, C.; Pentland, A. Wearable feedback systems for rehabilitation. J. Neuroeng. Rehabil. 2005, 2, 17. [Google Scholar] [CrossRef]
  18. Chen, K.H.; Chen, P.C.; Liu, K.C.; Chan, C.T. Wearable sensor-based rehabilitation exercise assessment for knee osteoarthritis. Sensors 2015, 15, 4193–4211. [Google Scholar] [CrossRef]
  19. Jakob, I.; Kollreider, A.; Germanotta, M.; Benetti, F.; Cruciani, A.; Padua, L.; Aprile, I. Robotic and sensor technology for upper limb rehabilitation. PM&R 2018, 10, S189–S197. [Google Scholar] [CrossRef]
  20. Liu, H.; Gamboa, H.; Schultz, T. Sensor-Based Human Activity and Behavior Research: Where Advanced Sensing and Recognition Technologies Meet. Sensors 2022, in press.
  21. Zhou, C.; Yang, Z.; Li, K.; Ye, X. Research and Development of Ankle-Foot Orthoses: A Review. Sensors 2022, 22, 6596. [Google Scholar] [CrossRef] [PubMed]
  22. Menz, H.B.; Bonanno, D.R. Objective measurement of adherence to wearing foot orthoses using an embedded temperature sensor. Med Eng. Phys. 2021, 88, 19–24. [Google Scholar] [CrossRef]
  23. Zhou, C.; Yang, L.; Liao, H.; Liang, B.; Ye, X. Ankle foot motion recognition based on wireless wearable sEMG and acceleration sensors for smart AFO. Sens. Actuators A Phys. 2021, 331, 113025. [Google Scholar] [CrossRef]
  24. Mangukiya, Y.; Purohit, B.; George, K. Electromyography (EMG) sensor controlled assistive orthotic robotic arm for forearm movement. In Proceedings of the 2017 IEEE Sensors Applications Symposium (SAS), Glassboro, NJ, USA, 13–15 March 2017; pp. 1–4. [Google Scholar] [CrossRef]
  25. Li, K.; Zhou, C. Estimation of Gait Parameters based on Motion Sensor Data. In Proceedings of the 13th International Joint Conference on Biomedical Engineering Systems and Technologies—BIODEVICES, Valletta, Malta, 24–26 February 2020; INSTICC. SciTePress: Setubal, Portugal, 2020; pp. 129–135. [Google Scholar] [CrossRef]
  26. Liu, H.; Schultz, T. How Long Are Various Types of Daily Activities? Statistical Analysis of a Multimodal Wearable Sensor-Based Human Activity Dataset. In Proceedings of the 15th International Joint Conference on Biomedical Engineering Systems and Technologies—HEALTHINF, Online, 9–11 February 2022; INSTICC. SciTePress: Setubal, Portugal, 2022; pp. 680–688. [Google Scholar] [CrossRef]
  27. Mendes, J.J.A., Jr.; Vieira, M.E.M.; Pires, M.B.; Stevan, S.L., Jr. Sensor fusion and smart sensor in sports and biomedical applications. Sensors 2016, 16, 1569. [Google Scholar] [CrossRef]
  28. Ji, Q.; Yang, L.; Li, W.; Zhou, C.; Ye, X. Real-time gait event detection in a real-world environment using a laser-ranging sensor and gyroscope fusion method. Physiol. Meas. 2018, 39, 125003. [Google Scholar] [CrossRef]
  29. Howard, R.M.; Conway, R.; Harrison, A.J. A survey of sensor devices: Use in sports biomechanics. Sport. Biomech. 2016, 15, 450–461. [Google Scholar] [CrossRef]
  30. McNab, T.; James, D.A.; Rowlands, D. iPhone sensor platforms: Applications to sports monitoring. Procedia Eng. 2011, 13, 507–512. [Google Scholar] [CrossRef]
  31. Liu, H.; Hartmann, Y.; Schultz, T. CSL-SHARE: A Multimodal Wearable Sensor-Based Human Activity Dataset. Front. Comput. Sci. 2021, 3, 90. [Google Scholar] [CrossRef]
  32. Howard, R. Wireless sensor devices in sports performance. IEEE Potentials 2016, 35, 40–42. [Google Scholar] [CrossRef]
  33. Yuji, O. Mems sensor application for the motion analysis in sports science. Memory 2005, 32, 128Mbit. [Google Scholar]
  34. Espinosa, H.G.; Lee, J.; James, D.A. The Inertial Sensor: A Base Platform for Wider Adoption in Sports Science Applications. J. Fit. Res. 2015, 4, 1. [Google Scholar]
  35. Ohgi, Y. Microcomputer-based acceleration sensor device for sports biomechanics-stroke evaluation by using swimmer’s wrist acceleration. In Proceedings of the SENSORS, 2002 IEEE, Orlando, FL, USA, 12–14 June 2002; Volume 1, pp. 699–704. [Google Scholar] [CrossRef]
  36. Liu, H. Biosignal Processing and Activity Modeling for Multimodal Human Activity Recognition. Ph.D. Thesis, University of Bremen, Bremen, Germany, 2021. [Google Scholar] [CrossRef]
  37. Chen, L.; Hoey, J.; Nugent, C.D.; Cook, D.J.; Yu, Z. Sensor-based activity recognition. IEEE Trans. Syst. Man Cybern. Part C 2012, 42, 790–808. [Google Scholar] [CrossRef]
  38. Hartmann, Y.; Liu, H.; Schultz, T. Feature Space Reduction for Multimodal Human Activity Recognition. In Proceedings of the 13th International Joint Conference on Biomedical Engineering Systems and Technologies—BIOSIGNALS, Valletta, Malta, 24–26 February 2020; INSTICC. SciTePress: Setubal, Portugal, 2020; pp. 135–140. [Google Scholar] [CrossRef]
  39. Chen, L.; Nugent, C.; Okeyo, G. An ontology-based hybrid approach to activity modeling for smart homes. IEEE Trans. Hum. Mach. Syst. 2013, 44, 92–105. [Google Scholar] [CrossRef]
  40. Liu, H.; Hartmann, Y.; Schultz, T. Motion Units: Generalized Sequence Modeling of Human Activities for Sensor-Based Activity Recognition. In Proceedings of the 29th European Signal Processing Conference (EUSIPCO 2021), Dublin, Ireland, 23–27 August 2021. [Google Scholar] [CrossRef]
  41. Wege, A.; Zimmermann, A. Electromyography sensor based control for a hand exoskeleton. In Proceedings of the 2007 IEEE International Conference on Robotics and Biomimetics (ROBIO), Sanya, China, 15–28 December 2007; pp. 1470–1475. [Google Scholar] [CrossRef]
  42. Ganesan, Y.; Gobee, S.; Durairajah, V. Development of an upper limb exoskeleton for rehabilitation with feedback from EMG and IMU sensor. Procedia Comput. Sci. 2015, 76, 53–59. [Google Scholar] [CrossRef]
  43. Zhong, J.; Liu, Y.; Cheng, X.; Cai, L.; Cui, W.; Hai, D. Gated Recurrent Unit Network for Psychological Stress Classification Using Electrocardiograms from Wearable Devices. Sensors 2022, 22, 8664. [Google Scholar] [CrossRef]
  44. Hartmann, Y.; Liu, H.; Schultz, T. Interactive and Interpretable Online Human Activity Recognition. In Proceedings of the 2022 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops), Pisa, Italy, 21–25 March 2022; pp. 109–111. [Google Scholar] [CrossRef]
  45. Zhang, Y.; Ren, X.; Liang, X.; Ye, X.; Zhou, C. A Refined Blood Pressure Estimation Model Based on Single Channel Photoplethysmography. IEEE J. Biomed. Health Inform. 2022, 26, 5907–5917. [Google Scholar] [CrossRef]
  46. Zhou, C.; Wang, H.; Zhang, Y.; Ye, X. Study of a ring-type surgical pleth index monitoring system based on flexible PPG sensor. IEEE Sens. J. 2020, 21, 14360–14368. [Google Scholar] [CrossRef]
  47. Chen, J.; Kwong, K.; Chang, D.; Luk, J.; Bajcsy, R. Wearable sensors for reliable fall detection. In Proceedings of the 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference, Shanghai, China, 17–18 January 2006; pp. 3551–3554. [Google Scholar] [CrossRef]
  48. Nyan, M.; Tay, F.E.; Murugasu, E. A wearable system for pre-impact fall detection. J. Biomech. 2008, 41, 3475–3481. [Google Scholar] [CrossRef]
  49. Xue, T.; Liu, H. Hidden Markov Model and Its Application in Human Activity Recognition and Fall Detection: A Review. In Communications, Signal Processing, and Systems; Springer: Singapore, 2022; pp. 863–869. [Google Scholar] [CrossRef]
  50. Rodrigues, J.; Belo, D.; Gamboa, H. Noise detection on ECG based on agglomerative clustering of morphological features. Comput. Biol. Med. 2017, 87, 322–334. [Google Scholar] [CrossRef]
  51. Belo, D. Learning Biosignals Using Deep Learning. Ph.D. Dissertation, Nova University of Lisbon, Lisbon, Portugal, 2020. [Google Scholar]
  52. Roh, Y.; Heo, G.; Whang, S.E. A survey on data collection for machine learning: A big data-ai integration perspective. IEEE Trans. Knowl. Data Eng. 2019, 33, 1328–1347. [Google Scholar] [CrossRef]
  53. Aminikhanghahi, S.; Cook, D.J. A Survey of Methods for Time Series Change Point Detection. Knowl. Inf. Syst. 2017, 51, 339–367. [Google Scholar] [CrossRef] [PubMed]
  54. Truong, C.; Oudre, L.; Vayatis, N. Selective review of offline change point detection methods. Signal Process. 2020, 167, 107299. [Google Scholar] [CrossRef]
  55. Adams, R.P.; Mackay, D.J.C. Bayesian Online Changepoint Detection. arXiv 2007, arXiv:0710.3742. [Google Scholar] [CrossRef]
  56. Fearnhead, P.; Liu, Z. On-line inference for multiple changepoint problems. J. R. Stat. Soc. Ser. B Stat. Methodol. 2007, 69, 589–605. [Google Scholar] [CrossRef]
  57. Bai, J. Estimating Multiple Breaks One at a Time. Econom. Theory 1997, 13, 315–352. [Google Scholar] [CrossRef]
  58. Auger, I. Algorithms for the optimal identification of segment neighborhoods. Bull. Math. Biol. 1989, 51, 39–54. [Google Scholar] [CrossRef]
  59. Gharghabi, S.; Ding, Y.; Yeh, C.C.M.; Kamgar, K.; Ulanova, L.; Keogh, E. Matrix Profile VIII: Domain Agnostic Online Semantic Segmentation at Superhuman Performance Levels. In Proceedings of the 2017 IEEE International Conference on Data Mining (ICDM), 2017, New Orleans, LA, USA, 18–21 November 2017; pp. 117–126. [Google Scholar] [CrossRef]
  60. Gharghabi, S.; Yeh, C.; Y, D.; Wei, D.; Hibbing, P.; LaMunion, S.; Crouter, S.E.; Kaplan, C.S.; Eamonn, K. Domain agnostic online semantic segmentation for multi-dimensional time series. Data Min. Knowl. Discov. 2019, 33, 96–130. [Google Scholar] [CrossRef]
  61. Matias, P.; Folgado, D.; Gamboa, H.; Carreiro, A. Time Series Segmentation Using Neural Networks with Cross-Domain Transfer Learning. Electronics 2021, 10, 1805. [Google Scholar] [CrossRef]
  62. Silva, P.; Luz, E.; Silva, G.; Moreira, G.; Wanner, E.; Vidal, F.; Menotti, D. Towards better heartbeat segmentation with deep learning classification. Sci. Rep. 2020, 10, 20701. [Google Scholar] [CrossRef]
  63. Malali, A.; Hiriyannaiah, S.; Siddesh, G.M.; Srinivasa, K.G.; Sanjay, N.T. Supervised ECG wave segmentation using convolutional LSTM. ICT Express 2020, 6, 166–169. [Google Scholar] [CrossRef]
  64. Moskalenko, V.; Zolotykh, N.; Osipov, G. Deep Learning for ECG Segmentation. In Advances in Neural Computation, Machine Learning, and Cognitive Research III; Kryzhanovsky, B., Dunin-Barkowski, W., Redko, V., Tiumentsev, Y., Eds.; Springer International Publishing: Cham, Switzerland, 2020; pp. 246–254. [Google Scholar] [CrossRef]
  65. Folgado, D.; Fernandes Barandas, M.; Antunes, M.; Nunes, M.L.; Liu, H.; Hartmann, Y.; Schultz, T.; Gamboa, H. TSSEARCH: Time Series Subsequence Search Library. SoftwareX 2022, 18, 101049. [Google Scholar] [CrossRef]
  66. Gadaleta, M.; Cisotto, G.; Rossi, M.; Rehman, R.Z.U.; Rochester, L.; Din, S.D. Deep Learning Techniques for Improving Digital Gait Segmentation. In Proceedings of the 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Berlin, Germany, 23–27 July 2019. [Google Scholar] [CrossRef]
  67. Roth, N.; Küderle, A.; Ullrich, M.; Gladow, T.; Marxreiter, F.; Klucken, J.; Eskofier, B.; Kluge, F. Hidden Markov Model based Stride Segmentation on Unsupervised Free-living Gait Data in Parkinson’s Disease Patients. J. Neuroeng. Rehabil. 2021, 18, 93. [Google Scholar] [CrossRef] [PubMed]
  68. Maqbool, H.F.; Husman, M.A.B.; Awad, M.I.; Abouhossein, A.; Iqbal, N.; Dehghani-Sanij, A.A. A Real-Time Gait Event Detection for Lower Limb Prosthesis Control and Evaluation. IEEE Trans. Neural Syst. Rehabil. Eng. 2017, 25, 1500–1509. [Google Scholar] [CrossRef] [PubMed]
  69. Catalfamo, P.; Ghoussayni, S.; Ewins, D. Gait Event Detection on Level Ground and Incline Walking Using a Rate Gyroscope. Sensors 2010, 10, 5683–5702. [Google Scholar] [CrossRef] [PubMed]
  70. Wang, M.; Wang, X.; Peng, C.; Zhang, S.; Fan, Z.; Liu, Z. Research on EMG segmentation algorithm and walking analysis based on signal envelope and integral electrical signal. Photonic Netw. Commun. 2019, 37, 195–203. [Google Scholar] [CrossRef]
  71. Perslev, M.; Jensen, M.H.; Darkner, S.; Jennum, P.J.; Igel, C. U-TIME: A FULLY CONVOLUTIONAL NETWORK FOR TIME SERIES SEGMENTATION APPLIED TO SLEEP STAGING. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, BC, Canada, 8–14 December 2019; Curran Associates Inc.: Red Hook, NY, USA, 2019. [Google Scholar] [CrossRef]
  72. Kurtek, S.; Wu, W.; Christensen, G.E.; Srivastava, A. Segmentation, alignment and statistical analysis of biosignals with application to disease classification. J. Appl. Stat. 2013, 40, 1270–1288. [Google Scholar] [CrossRef]
  73. Müller, M.; Zalkow, F. FMP Notebooks: Educational Material for Teaching and Learning Fundamentals of Music Processing. In Proceedings of the International Conference on Music Information Retrieval (ISMIR), Delft, The Netherlands, 4–8 November 2019. [Google Scholar]
  74. Anguita, D.; Ghio, A.; Oneto, L.; Parra, X.; Reyes-Ortiz, J.L. Human Activity Recognition on Smartphones Using a Multiclass Hardware-Friendly Support Vector Machine. In Ambient Assisted Living and Home Care; Bravo, J., Hervás, R., Rodríguez, M., Eds.; Springer: Berlin/Heidelberg, Germany, 2012; pp. 216–223. [Google Scholar] [CrossRef]
  75. Anguita, D.; Ghio, A.; Oneto, L.; Parra Perez, X.; Reyes Ortiz, J.L. A public domain dataset for human activity recognition using smartphones. In Proceedings of the 21th International European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, Bruges, Belgium, 24–26 April 2013; pp. 437–442. [Google Scholar]
  76. Liu, H.; Schultz, T. ASK: A Framework for Data Acquisition and Activity Recognition. In Proceedings of the 11th International Joint Conference on Biomedical Engineering Systems and Technologies—BIOSIGNALS, Funchal, Portugal, 19–21 January 2018; INSTICC. SciTePress: Setubal, Portugal, 2018; pp. 262–268. [Google Scholar] [CrossRef]
  77. Moody, G.B.; Muldrow, W.K.; Mark, R.G. A Noise Stress for Arrythmia Detectors. Comput. Cardiol. 1984, 11, 381–384. [Google Scholar]
  78. Goldberger, A.L.; Amaral, L.A.N.; Glass, L.; Hausdorff, J.M.; Ivanov, P.C.; Mark, R.G.; Mietus, J.E.; Moody, G.B.; Peng, C.K.; Stanley, H.E. PhysioBank, PhysioToolkit, and PhysioNet. Circulation 2000, 101, e215–e220. [Google Scholar] [CrossRef]
  79. Behravan, V.; Glover, N.E.; Farry, R.; Chiang, P.Y.; Shoaib, M. Rate-adaptive compressed-sensing and sparsity variance of biomedical signals. In Proceedings of the 2015 IEEE 12th International Conference on Wearable and Implantable Body Sensor Networks (BSN), Cambridge, UK, 9–12 June 2015; pp. 1–6. [Google Scholar] [CrossRef]
  80. Lobov, S.; Krilova, N.; Kastalskiy, I.; Kazantsev, V.; Makarov, V. Latent Factors Limiting the Performance of sEMG-Interfaces. Sensors 2018, 18, 1122. [Google Scholar] [CrossRef]
  81. Heldt, T.; Oefinger, M.; Hoshiyama, M.; Mark, R. Circulatory response to passive and active changes in posture. Comput. Cardiol. 2003, 2003, 263–266. [Google Scholar] [CrossRef]
  82. Chuttani, K.; Tischler, M.D.; Pandian, N.G.; Lee, R.T.; Mohanty, P.K. Diagnosis of cardiac tamponade after cardiac surgery: Relative value of clinical, echocardiographic, and hemodynamic signs. Am. Heart J. 1994, 127, 913–918. [Google Scholar] [CrossRef]
  83. Van Dam, M.N.; Fitzgerald, B.M. Pulsus Paradoxus. Eur. Respir. J. 2022, 42, 1696–1705. [Google Scholar]
  84. Barandas, M.; Folgado, D.; Fernandes, L.; Santos, S.; Abreu, M.; Bota, P.; Liu, H.; Schultz, T.; Gamboa, H. TSFEL: Time Series Feature Extraction Library. SoftwareX 2020, 11, 100456. [Google Scholar] [CrossRef]
  85. Hartmann, Y.; Liu, H.; Schultz, T. Feature Space Reduction for Human Activity Recognition based on Multi-channel Biosignals. In Proceedings of the 14th International Joint Conference on Biomedical Engineering Systems and Technologies—BIOSIGNALS, Online, 11–13 February 2021; INSTICC. SciTePress: Setubal, Portugal, 2021; pp. 215–222. [Google Scholar] [CrossRef]
  86. Naseeb, C.; Saeedi, B.A. Activity recognition for locomotion and transportation dataset using deep learning. In Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers, Online, 12–17 September 2020; pp. 329–334. [Google Scholar] [CrossRef]
  87. Demrozi, F.; Turetta, C.; Chiarani, F.; Kindt, P.H.; Pravadelli, G. Estimating indoor occupancy through low-cost BLE devices. IEEE Sens. J. 2021, 21, 17053–17063. [Google Scholar] [CrossRef]
  88. Liu, H.; Jiang, K.; Gamboa, H.; Xue, T.; Schultz, T. Bell Shape Embodying Zhongyong: The Pitch Histogram of Traditional Chinese Anhemitonic Pentatonic Folk Songs. Appl. Sci. 2022, 12, 8343. [Google Scholar] [CrossRef]
  89. Konar, S.; Auluck, N.; Ganesan, R.; Goyal, A.K.; Kaur, T.; Sahi, M.; Samra, T.; Thingnam, S.K.S.; Puri, G.D. A non-linear time series based artificial intelligence model to predict outcome in cardiac surgery. Health Technol. 2022, 12, 1169–1181. [Google Scholar] [CrossRef]
  90. Pasquali, D.; Gonzalez-Billandon, J.; Aroyo, A.M.; Sandini, G.; Sciutti, A.; Rea, F. Detecting Lies is a Child (Robot)’s Play: Gaze-Based Lie Detection in HRI. Int. J. Soc. Robot. 2021, 1–16. [Google Scholar] [CrossRef]
  91. Liu, H.; Xue, T.; Schultz, T. Merged Pitch Histograms and Pitch-Duration Histograms. In Proceedings of the 19th International Conference on Signal Processing and Multimedia Applications—SIGMAP, Lisbon, Portugal, 14–16 July 2022; INSTICC. SciTePress: Setubal, Portugal, 2022; pp. 32–39. [Google Scholar] [CrossRef]
  92. Foote, J. Automatic audio segmentation using a measure of audio novelty. In Proceedings of the 2000 IEEE International Conference on Multimedia and Expo. ICME2000. Proceedings. Latest Advances in the Fast Changing World of Multimedia (Cat. No.00TH8532), New York, NY, USA, 30 July–2 August 2000; Volume 1, pp. 452–455. [Google Scholar] [CrossRef]
  93. Müller, M.; Zalkow, F. libfmp: A Python Package for Fundamentals of Music Processing. J. Open Source Softw. 2021, 6, 3326. [Google Scholar] [CrossRef]
  94. Dannenberg, R.B.; Goto, M. Music Structure Analysis from Acoustic Signals. In Handbook of Signal Processing in Acoustics; Springer: New York, NY, USA, 2008; pp. 305–331. [Google Scholar] [CrossRef]
  95. Liu, H.; Hartmann, Y.; Schultz, T. A Practical Wearable Sensor-Based Human Activity Recognition Research Pipeline. In Proceedings of the 15th International Joint Conference on Biomedical Engineering Systems and Technologies—HEALTHINF, Online, 9–11 February 2022; INSTICC. SciTePress: Setubal, Portugal, 2022; pp. 847–856. [Google Scholar] [CrossRef]
  96. Ismail Fawaz, H.; Forestier, G.; Weber, J.; Idoumghar, L.; Muller, P.A. Deep learning for time series classification: A review. Data Min. Knowl. Discov. 2019, 33, 917–963. [Google Scholar] [CrossRef]
  97. Aickin, M.; Gensler, H. Adjusting for multiple testing when reporting research results: The Bonferroni vs Holm methods. Am. J. Public Health 1996, 86, 726–728. [Google Scholar] [CrossRef] [PubMed]
  98. Hartmann, Y.; Liu, H.; Lahrberg, S.; Schultz, T. Interpretable High-Level Features for Human Activity Recognition. In Proceedings of the 15th International Joint Conference on Biomedical Engineering Systems and Technologies—BIOSIGNALS, Online, 9–11 February 2022; INSTICC. SciTePress: Setubal, Portugal, 2022; pp. 40–49. [Google Scholar] [CrossRef]
Figure 1. A visual structural description of functions on a time series for retrieving relevant events, segmenting, and associating the previously segmented subsequences based on the feature-based SSM. (Left): an Arterial Blood Pressure (ABP) signal’s SSM representing the pairwise similarity between subsequences, where the “novelty search” signal in green below the matrix demonstrates the novelty function and the “periodic search” signal in orange depicts the similarity function; (right): the clustering procedure of the novelty function-based segmented subsequences according to their similarity values in the SSM.
Figure 1. A visual structural description of functions on a time series for retrieving relevant events, segmenting, and associating the previously segmented subsequences based on the feature-based SSM. (Left): an Arterial Blood Pressure (ABP) signal’s SSM representing the pairwise similarity between subsequences, where the “novelty search” signal in green below the matrix demonstrates the novelty function and the “periodic search” signal in orange depicts the similarity function; (right): the clustering procedure of the novelty function-based segmented subsequences according to their similarity values in the SSM.
Biosensors 12 01182 g001
Figure 2. Event search in different ranks of dimensionality, timescales, and representation.
Figure 2. Event search in different ranks of dimensionality, timescales, and representation.
Biosensors 12 01182 g002
Figure 3. A step-by-step flowchart for calculating and analysing the SSM. The signal-based calculation requires input parameters of the window size w and the overlapping percentage o to fulfil the first-stage feature extraction. Features are extracted on each subsequence ( s T 1 , s T 2 , , s T N ) , where N is the total number of windows. K features are extracted from window i ( s T i : f i 1 , f i 2 , , f i K ). Different features are associated with different shapes ( , , and Δ) in the figures. The features can be extracted on an M-variable record and each feature is positioned as a row on the F M for the SSM computation.
Figure 3. A step-by-step flowchart for calculating and analysing the SSM. The signal-based calculation requires input parameters of the window size w and the overlapping percentage o to fulfil the first-stage feature extraction. Features are extracted on each subsequence ( s T 1 , s T 2 , , s T N ) , where N is the total number of windows. K features are extracted from window i ( s T i : f i 1 , f i 2 , , f i K ). Different features are associated with different shapes ( , , and Δ) in the figures. The features can be extracted on an M-variable record and each feature is positioned as a row on the F M for the SSM computation.
Biosensors 12 01182 g003
Figure 4. The informative structures of an ABP signal’s SSM. The three main structures are highlighted in the simplified illustration: A—the homogeneous segments corresponding to periods in the ABP signal; B—the homogeneous segment representing missing data; C—the homogeneous segment cueing sensor detachment. The “blocks” in the figure accentuate homogeneous behaviour, while the paths in the figure depict periodicity in the segment. Segment C has a cross pattern, which symbolizes periodicity and symmetry. n f : novelty function; s f : similarity function; Δ: change points separating blocks A, B and C.
Figure 4. The informative structures of an ABP signal’s SSM. The three main structures are highlighted in the simplified illustration: A—the homogeneous segments corresponding to periods in the ABP signal; B—the homogeneous segment representing missing data; C—the homogeneous segment cueing sensor detachment. The “blocks” in the figure accentuate homogeneous behaviour, while the paths in the figure depict periodicity in the segment. Segment C has a cross pattern, which symbolizes periodicity and symmetry. n f : novelty function; s f : similarity function; Δ: change points separating blocks A, B and C.
Biosensors 12 01182 g004
Figure 5. (Left): description of the matrix (kernel) used to compute the novelty function, based on the works of Mueller et al. [2,73]. The chequerboard pattern of the kernel K N is achieved by combining the kernel K H (homogeneity measure) and K C (cross-similarity measure). Combined with a Gaussian function, the K G is obtained; (right): the process to compute the novelty function based on the works of [2,73,94]. Kernel K G slides along the diagonal of the SSM to compute the novelty function presented as the bottom sub-plot. Positions A and B point to the effect of block transitions on the novelty function.
Figure 5. (Left): description of the matrix (kernel) used to compute the novelty function, based on the works of Mueller et al. [2,73]. The chequerboard pattern of the kernel K N is achieved by combining the kernel K H (homogeneity measure) and K C (cross-similarity measure). Combined with a Gaussian function, the K G is obtained; (right): the process to compute the novelty function based on the works of [2,73,94]. Kernel K G slides along the diagonal of the SSM to compute the novelty function presented as the bottom sub-plot. Positions A and B point to the effect of block transitions on the novelty function.
Biosensors 12 01182 g005
Figure 6. An SSM-based novelty search strategy to detect segmentation events on a signal piece from Dataset 1 (HAR, see Section 3.1). (Top): w i n d o w s i z e = 250 samples, k e r n e l s i z e = 45 samples, and overlap = 95% on the activity sequence S t a n d i n g L a y i n g W a l k i n g U p s t a i r s D o w n s t a i r s S i t t i n g S t a n d i n g L a y i n g W a l k i n g U p s t a i r s D o w n s t a i r s S i t t i n g S t a n d i n g L a y i n g . The novelty function is presented and peaks are aligned with ground truth events, represented as the dashed line and circles; (bottom left): signal change point detection on segment A with a size of 5000 samples, an overlap of 75%, and a kernel size of 25 samples. The novelty function is displayed and peaks are aligned with ground truth events, represented as the dashed line and circles; (bottom right): further zooming in with a window size of 10 samples and an overlap of 95%, to reveal more periodic details of segment B. The similarity function is presented and valleys are aligned with ground truth events, represented as the dashed line and circles.
Figure 6. An SSM-based novelty search strategy to detect segmentation events on a signal piece from Dataset 1 (HAR, see Section 3.1). (Top): w i n d o w s i z e = 250 samples, k e r n e l s i z e = 45 samples, and overlap = 95% on the activity sequence S t a n d i n g L a y i n g W a l k i n g U p s t a i r s D o w n s t a i r s S i t t i n g S t a n d i n g L a y i n g W a l k i n g U p s t a i r s D o w n s t a i r s S i t t i n g S t a n d i n g L a y i n g . The novelty function is presented and peaks are aligned with ground truth events, represented as the dashed line and circles; (bottom left): signal change point detection on segment A with a size of 5000 samples, an overlap of 75%, and a kernel size of 25 samples. The novelty function is displayed and peaks are aligned with ground truth events, represented as the dashed line and circles; (bottom right): further zooming in with a window size of 10 samples and an overlap of 95%, to reveal more periodic details of segment B. The similarity function is presented and valleys are aligned with ground truth events, represented as the dashed line and circles.
Biosensors 12 01182 g006
Figure 7. Novelty and similarity search on an ABP signal from Dataset 6 (BVP, see Section 3.6). (Top): a window size of 5000 samples, an overlap of 95%, and a kernel size of 200 samples. The trapezoidal and the square wave mark the ground truth of slow and fast postural transitions. Similarity profiles P A P D show how similar each segment ( A D ) are. For instance, P D is more similar to P B . (Bottom): the first 10,000 samples, with a window size of 250 samples, an overlap of 95% and a kernel size of 200 samples. The right parts of the top and bottom subfigures plot the corresponding similarity profiles for each subsequence segmented by the novelty function. In both figures, the novelty function is displayed and peaks are aligned with ground truth events, represented as dashed lines and circles. The bottom plot also shows the similarity function (sf) with circles representing the ground truth of periods. In addition, similarity profiles P A P G , show how similar are each segment resulting from the novelty function. For instance, segment B is more similar to segment D, and this is verified by P D being more similar to P B .
Figure 7. Novelty and similarity search on an ABP signal from Dataset 6 (BVP, see Section 3.6). (Top): a window size of 5000 samples, an overlap of 95%, and a kernel size of 200 samples. The trapezoidal and the square wave mark the ground truth of slow and fast postural transitions. Similarity profiles P A P D show how similar each segment ( A D ) are. For instance, P D is more similar to P B . (Bottom): the first 10,000 samples, with a window size of 250 samples, an overlap of 95% and a kernel size of 200 samples. The right parts of the top and bottom subfigures plot the corresponding similarity profiles for each subsequence segmented by the novelty function. In both figures, the novelty function is displayed and peaks are aligned with ground truth events, represented as dashed lines and circles. The bottom plot also shows the similarity function (sf) with circles representing the ground truth of periods. In addition, similarity profiles P A P G , show how similar are each segment resulting from the novelty function. For instance, segment B is more similar to segment D, and this is verified by P D being more similar to P B .
Biosensors 12 01182 g007aBiosensors 12 01182 g007b
Figure 8. An ECG signal with a pulsus paradoxus condition starting at the 10,000th sample from Dataset 7 (ECG Pulsus Paradoxus, see Section 3.7). (Left): the SSM diagnoses two modes in the signal, whose patterns are zoomed in the circle thumbnails, respectively; (right): zooming parts of the original signal can verify SSM’s ability of automatic ECG pattern change detection and contribution to segmentation. The novelty function is presented, and the peak is aligned with the ground truth event, represented as a circle.
Figure 8. An ECG signal with a pulsus paradoxus condition starting at the 10,000th sample from Dataset 7 (ECG Pulsus Paradoxus, see Section 3.7). (Left): the SSM diagnoses two modes in the signal, whose patterns are zoomed in the circle thumbnails, respectively; (right): zooming parts of the original signal can verify SSM’s ability of automatic ECG pattern change detection and contribution to segmentation. The novelty function is presented, and the peak is aligned with the ground truth event, represented as a circle.
Biosensors 12 01182 g008
Figure 9. The proposed method was applied to the Occupancy record of Dataset 5 (CPDBenchmark, see Section 3.5). (Left): calculations on the separate C O 2 time series only; (right): calculations performed by extracting features on the complete four time series. The novelty function is presented and peaks are aligned with ground truth events, represented as the dashed line and circles.
Figure 9. The proposed method was applied to the Occupancy record of Dataset 5 (CPDBenchmark, see Section 3.5). (Left): calculations on the separate C O 2 time series only; (right): calculations performed by extracting features on the complete four time series. The novelty function is presented and peaks are aligned with ground truth events, represented as the dashed line and circles.
Biosensors 12 01182 g009
Figure 10. Critical distance diagram comparing the methods used in [8] (except RBOCPDMS) and the novelty function on Dataset 5 (CPDBenchmark, see Section 3.5). The performance measure corresponds to the F1-score for all single-dimension datasets of the benchmark, except for the ones identified in Table 3 with a grey background. A thick horizontal line groups a set of classifiers that are not significantly different in the statistical test [96].
Figure 10. Critical distance diagram comparing the methods used in [8] (except RBOCPDMS) and the novelty function on Dataset 5 (CPDBenchmark, see Section 3.5). The performance measure corresponds to the F1-score for all single-dimension datasets of the benchmark, except for the ones identified in Table 3 with a grey background. A thick horizontal line groups a set of classifiers that are not significantly different in the statistical test [96].
Biosensors 12 01182 g010
Figure 11. An illustrative example of window length intuition on records of Dataset 2 (ECG1, see Section 3.2). Top: different SSMs on the same ECG record A computed with sequentially larger window lengths from 0.01 to 2 s. The novelty functions are calculated with a kernel size equal to the window size and an overlap of 95%. Bottom: The 1-second window length is further applied as an example to indicate that parameters turned in the representation experiments can be generalized to all other records of the same dataset (B-I) to compute their corresponding SSM representations and novelty functions.
Figure 11. An illustrative example of window length intuition on records of Dataset 2 (ECG1, see Section 3.2). Top: different SSMs on the same ECG record A computed with sequentially larger window lengths from 0.01 to 2 s. The novelty functions are calculated with a kernel size equal to the window size and an overlap of 95%. Bottom: The 1-second window length is further applied as an example to indicate that parameters turned in the representation experiments can be generalized to all other records of the same dataset (B-I) to compute their corresponding SSM representations and novelty functions.
Biosensors 12 01182 g011
Table 1. Results obtained by experiments with the novelty function-based, window-based, and binary segmentation approaches on different datasets, including true positives (TP), false positives (FP) and false negatives (FN), precision (P), recall (R) and F1-score (F1) values. The last row provides the macro averaged F1-scores (M.A. F1) of the four datasets.
Table 1. Results obtained by experiments with the novelty function-based, window-based, and binary segmentation approaches on different datasets, including true positives (TP), false positives (FP) and false negatives (FN), precision (P), recall (R) and F1-score (F1) values. The last row provides the macro averaged F1-scores (M.A. F1) of the four datasets.
Novelty Function-BasedWindow-BasedBinary
DatasetTPFPFNPRF1TPFPFNPRF1TPFPFNPRF1
(1) HAR16616130.910.930.921699100.950.940.9512558540.680.700.69
(2) ECG1181.000.951.000.97135.050.720.720.72162.020.890.890.89
(3) ECG21559.0130.950.920.9313913.0290.910.830.8712258.0460.680.730.70
(4) EMG69568330.910.950.936081231200.830.840.833514133770.460.480.47
M.A. F1-----0.94-----0.84-----0.69
Table 2. Points table recording segmentation match between the novelty-based (novelty), window-based (WL), and binary (BS) segmentation approaches on different datasets in the format of “Win/Draw/Lose” based on F1-scores. The last row reports the points for all datasets.
Table 2. Points table recording segmentation match between the novelty-based (novelty), window-based (WL), and binary (BS) segmentation approaches on different datasets in the format of “Win/Draw/Lose” based on F1-scores. The last row reports the points for all datasets.
Novelty Function-BasedWindow-BasedBinary
DatasetWinsDrawsLossesWinsDrawsLossesWinsDrawsLosses
(1) HAR1002200002
(2) ECG1270045063
(3) ECG2101100121110
(4) EMG312332310036
Overall301103004
Table 3. Comparison of the F1-scores between our proposed method (novelty) and other algorithms’ benchmarks in Datasets 7 (ECG Pulsus Paradoxus, see Section 3.5). The calculation of all one-dimensional signals’ average performance and all signals’ average performance does not include columns with a grey background where no change point should be detected, or a signal error was present. Bold values represent the best F1-score for this specific dataset. T: timed out; F: failed compiling.
Table 3. Comparison of the F1-scores between our proposed method (novelty) and other algorithms’ benchmarks in Datasets 7 (ECG Pulsus Paradoxus, see Section 3.5). The calculation of all one-dimensional signals’ average performance and all signals’ average performance does not include columns with a grey background where no change point should be detected, or a signal error was present. Bold values represent the best F1-score for this specific dataset. T: timed out; F: failed compiling.
Datasetnoveltyamocbinsegbocpdbocpdmscpnpecpkcpapeltprophetrbocpdmsrfpopsegneighwbszero
One-dimensional
bank01.0001.0001.0000.5000.0540.2000.3330.4001.000T0.0151.0000.0431.000
bitcoin0.6940.5070.6900.7330.5330.6110.6250.6650.7350.446T0.2840.7350.6900.450
brent_spot0.8610.4650.6700.6090.2390.6070.6360.5530.5860.249T0.5210.5860.5640.315
businv0.9270.5880.5880.5880.4550.3860.3700.2940.4900.2750.3700.2610.5880.2890.588
centralia0.9840.9091.0001.0001.0001.0000.9091.0001.0000.7630.8461.0001.0000.5560.763
children_per_woman0.8790.6780.6630.7120.4050.3440.5510.5250.6370.3100.5040.2460.6370.5000.507
co2_canada0.8510.5440.8560.9240.4790.6420.8750.8670.6700.4820.5420.5690.8720.6810.361
construction0.9330.6960.7090.7090.4100.6020.7090.6340.7090.3240.3400.1850.7090.5230.696
debt_ireland0.9740.7601.0001.0000.8920.9580.9801.0001.0000.4690.7480.8241.0000.5380.469
gdp_argentina0.9680.8890.9470.9470.5830.8180.8890.8000.9470.6150.4520.6150.9470.4210.824
gdp_croatia1.0001.0000.8241.0000.5831.0000.8240.5830.8240.8240.8240.4000.8240.1670.824
gdp_iran0.9210.6960.6520.8620.4920.6200.8240.7340.8080.6520.7370.6360.8080.5760.652
gdp_japan1.0001.0000.8891.0000.6150.6671.0000.5000.8890.8890.8890.2220.8890.2220.889
global_co20.6250.9290.9290.8890.4580.6670.9290.6670.9290.4630.5470.2930.9290.2500.846
homeruns0.9330.8120.8290.8290.6500.6500.8290.8290.8120.7230.3970.6610.8120.6640.659
iceland_tourism0.6520.9470.9470.9470.4860.3911.0000.4860.6430.2200.6670.2000.9470.2000.947
jfk_passengers0.9780.7760.7760.7760.6500.6020.6510.4370.7760.354T0.4910.7760.4370.723
lga_passengers0.8850.5610.6200.7040.5630.6060.8920.5260.5370.366T0.5920.5370.6740.535
measles00.9470.9470.9470.4860.1180.0800.2810.1530.391F/T0.0300.9470.0410.947
nile1.0001.0001.0001.0000.8001.0001.0000.8241.0000.8240.6671.0001.0001.0000.824
ozone0.8570.7760.7230.8570.7780.7501.0000.6671.0000.7230.6510.4291.0000.2860.723
quality_control_11.0001.0001.0001.0000.6670.6671.0000.6671.0000.5000.2860.6671.0000.6670.667
quality_control_21.0001.0001.0001.0000.6671.0001.0001.0001.0000.750.4291.0001.0001.0000.750
quality_control_31.0001.0001.0001.0000.7660.5711.0001.0001.0000.667T0.8001.0001.0000.667
quality_control_40.9740.8100.8730.7870.5610.6580.7260.6580.7800.780T0.2410.7800.6080.780
quality_control_501.0001.0001.0000.5001.0001.0001.0001.0001.0000.5001.0001.0001.0001.000
rail_lines0.9090.8460.8460.9660.8890.9660.9660.8000.8460.5370.7300.6150.8890.2050.537
ratner_stock0.9330.7760.8240.8680.5590.3960.7760.7540.8240.280T0.2030.8240.3780.571
robocalls0.9790.8000.9660.9660.7500.8620.9660.9660.9660.6360.8460.7140.9660.7140.636
scanline_1260070.8870.7100.9200.9210.8290.9060.8700.8380.8890.644T0.6490.8890.8180.644
scanline_420490.9770.4850.8790.9620.8890.7130.9100.9080.9100.269T0.4600.9100.6500.276
seatbelts0.6590.8240.8380.6830.5830.7350.6830.6210.6830.4520.3830.5630.7350.5830.621
shanghai_license0.9790.9660.8680.8680.6050.6000.8680.4650.8680.5320.3890.3570.8680.3850.636
uk_coal_employmentFFFF0.617F0.5130.513F0.639FFFF0.513
unemployment_nl0.8200.7420.8890.8760.5920.7470.7550.7440.7880.566F/T0.6280.7880.8010.566
us_population0.6361.0000.8891.0000.6150.2320.4710.2760.5000.159T0.8890.8890.1130.889
usd_isk0.9140.7850.7040.7850.6780.6740.7850.6010.6570.4890.5100.4620.6780.6360.489
well_log0.8140.3360.9140.8320.7430.8220.9280.7760.8730.149T0.9230.8730.8320.237
Average F1-measure (1D)0.8450.7390.7980.8220.5960.6510.7840.6570.7660.4820.3540.5170.7970.5170.599
Multidimensional
apple0.949 0.9160.445 0.7450.634 F/T 0.594
bee_waggle_60.657 0.9290.481 0.2330.634 0.245 0.929
occupancy0.953 0.9190.735 0.9320.812 F/T 0.341
run_log0.994 1.0000.469 0.9900.909 0.380 0.446
Average F1-measure (ALL)0.871n.a.n.a.0.8550.604n.a.0.7970.683n.a.n.a.0.343n.a.n.a.n.a.0.61
WINS (ALL)1602200300000000
DRAWS (ALL)6981116849103820
LOSES (ALL)122524213328233025333431263234
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Rodrigues, J.; Liu, H.; Folgado, D.; Belo, D.; Schultz, T.; Gamboa, H. Feature-Based Information Retrieval of Multimodal Biosignals with a Self-Similarity Matrix: Focus on Automatic Segmentation. Biosensors 2022, 12, 1182. https://doi.org/10.3390/bios12121182

AMA Style

Rodrigues J, Liu H, Folgado D, Belo D, Schultz T, Gamboa H. Feature-Based Information Retrieval of Multimodal Biosignals with a Self-Similarity Matrix: Focus on Automatic Segmentation. Biosensors. 2022; 12(12):1182. https://doi.org/10.3390/bios12121182

Chicago/Turabian Style

Rodrigues, João, Hui Liu, Duarte Folgado, David Belo, Tanja Schultz, and Hugo Gamboa. 2022. "Feature-Based Information Retrieval of Multimodal Biosignals with a Self-Similarity Matrix: Focus on Automatic Segmentation" Biosensors 12, no. 12: 1182. https://doi.org/10.3390/bios12121182

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop