Anomaly Detection in Multichannel Data Using Sparse Representation in RADWT Frames

Abstract

We introduce a new methodology for anomaly detection (AD) in multichannel fast oscillating signals based on nonparametric penalized regression. Assuming the signals share similar shapes and characteristics, the estimation procedures are based on the use of the Rational-Dilation Wavelet Transform (RADWT), equipped with a tunable Q-factor able to provide sparse representations of functions with different oscillations persistence. Under the standard hypothesis of Gaussian additive noise, we model the signals by the RADWT and the anomalies as additive in each signal. Then we perform AD imposing a double penalty on the multiple regression model we obtained, promoting group sparsity both on the regression coefficients and on the anomalies. The first constraint preserves a common structure on the underlying signal components; the second one aims to identify the presence/absence of anomalies. Numerical experiments show the performance of the proposed method in different synthetic scenarios as well as in a real case.

1. Introduction

Anomaly detection (AD) is an important problem that has received much attention in recent years. It consists of establishing whether a given data deviates from nominal shape or form. It is not possible to establish a single mathematical framework to deal with AD because it depends on the type of application. There are a lot of surveys in the literature, some of them specific to certain applications and some others are broader and application-free. See, for example, the review by [1] regarding the Intrusion Detection Systems, or the paper by [2] providing an overview on AD, analysis, and prediction in Internet of Things (IoT) systems or the paper by [3] considering the application of AD techniques to aviation and how they improve the safety and service of flight operations, just to cite some. However, a milestone of the literature is the paper by Chandola et al. [4], where the authors offer a very good understanding of the subject, defining anomalies, describing its challenges, depicting a relevant taxonomy of the different techniques, and illustrating the various domains of applications. As well explained by the authors, the AD problem depends on the nature of input data (points, sequence, functions, graphs, objects of different nature), on the type of anomaly (point anomalies, contextual or behavioral anomalies, or their combination), on the availability of labeled data for training/validation of the AD techniques (leading to unsupervised AD and supervised AD), and on the type of output of AD (scores or label). Two more recent reviews are the ones by Pimentel et al. [5] and by Ahmed et al. [6], where the authors display a huge and comprehensive number of references on the topic. In particular, reference [5] is devoted to novelty detection, where the distinction with the word anomaly is that the former considers, as normal, the data after their detection. Another important and very recent paper is the one by Thudumu et al. [7], which reviews AD in the context of big data where a curse of dimensionality occurs, bringing the failure of different methodologies.

We can broadly divide the AD techniques into four categories as very well represented by the taxonomy of Figure 3 in [6]: classification techniques, statistical techniques, clustering techniques, and information theory techniques.

In this paper, we focus on statistical techniques, whose recipe is to fit a statistical model to the given data and then check if some of them do not fit this model. We think that a good survey on statistical methods is given by Rousseeuw and Hubert in [8], where they consider robust regression techniques, signal processing techniques, and robust principal component analysis (PCA) techniques.

Our paper is placed in the context of statistical signal processing techniques, where the data are indeed time series, signals, or functions. It is worth observing that signal processing is synonymous with functional data analysis in more classical statistics nomenclature, as well as outliers and anomalies. In particular, we can distinguish between isolated and persistent outliers, see [9]. The isolated outliers insist on a small part of the signal, while the persistent outliers insist on most of their domain. It is important to stress that the literature on AD in functional data is quite recent and is mainly devoted to univariate functions.

In this paper, we deal with the AD problem for multichannel (i.e., multivariate) data affected by persistent outliers and approach it from the perspective of penalized multiple regression framework. We model the nominal shape as K multiple signals from K channels sharing a joint sparse representation into a special dictionary, and then we add a possible anomaly term of any form and shape to each signal component. This framework represents situations where we expect similar shapes for the components of the signal, but some components can undergo anomaly behavior for some unpredictable reason. In particular, we consider nominal signals with a fast oscillating characteristic that can be well represented only by using an appropriate dictionary, namely a RADWT, as done in the context of multiple regression in [10]. A RADWT is a modern and fast computational tool for analyzing signals which are a mixture of oscillatory and nonoscillatory transient behaviors. Such kinds of signals are not periodic and may describe many physiological and physical signals (for example, speech, stock-market, biomedical EEG, etc.) which could not be represented sparsely in the classic orthogonal bases such as Fourier, wavelet, etc. The request of joint sparse representation formalizes the hypothesis that the signal components are expected to have the same spectral characteristics while preserving numerical differences. The idea we develop in our work is the following: a linear model is applied to simultaneously estimate the signal by using its sparse representation in this special dictionary and detect potential anomalies modeled as residuals in this sparse representation.

A similar idea has been already applied in the literature under different model assumptions, for example in [11,12,13]. Specifically, in [11] the authors propose to learn the dictionary by using a set of nominal signals through PCA, as well as in [12] the authors learn the dictionary by using a training set of “normal” nominal signals, i.e., not affected by anomalies, under the specific hypothesis that they are multivariate mixtures of discrete and continuous signals and applying their sparse coding algorithm to select the training signals having the highest residuals. Subsequently, in both papers, the authors detect anomalies for a new signal occurrence by evaluating the residual of its sparse representation in this dictionary. On the other hand, in [13], the authors face the univariate AD problem considering a linear regression model where the univariate signal is expressed as a linear combination of columns of a given design matrix without the sparseness hypothesis. The authors propose detecting anomalies by simultaneously estimating both the nominal signal and its residual using nonconvex penalized regression with the constraint on the outliers vector.

Our paper stands in between these recent proposals [11,12,13] for two reasons. First, we use a dictionary for sparsely representing functions (as in [11,12]) but without requiring a training set of “normal” nominal signals for dictionary learning because we rely on an innovative instrument such as the RADWT transform to deal with fast oscillating signals. Second, we look for anomalies by simultaneously estimating both the nominal signal and its residual (as in [13]) but imposing a double constraint on the regression coefficients and the anomalies, taking into account that few RADWT coefficients are necessary to represent the signals. To ensure that the residuals can preserve information about the outliers/anomalies, we must use robust regression techniques. As we will clarify later on, a good AD undergoes a good robust regression analysis, because the procedure aims simultaneously to detect the anomalies and to estimate the signal’s components, the success of the first goal ensuring the success of the second and vice versa. Hence, our paper can be also considered as part of the literature on robust regression in the high-dimensional setting, see [14,15]. In this perspective, we also refer to [16], from which we were inspired to use robust losses, such as the Huber loss and the skipped mean loss.

The remainder of the paper is organized as follows. Section 2 describes the data model with the working hypothesis. Section 3 presents and discusses the proposed AD procedure within the paradigm of group-lasso regression, enlightening the connections with other existing procedures. Section 4 describes some important implementation issues. Section 5 shows numerical experiments on synthetic and real data and finally, the last section discusses conclusions.

2. The Data Model and Problem Setting

We will consider hereafter a functional dataset consisting of K curves $f^{\left(k\right)}\left(t\right)$, $k=1, \dots ,K$ observed on a set of equispaced grid points $t_1, \dots ,t_n$ and resulting in observed K dimensional response vectors ${\mathbf{y}}^{\left(1\right)}$, ${\mathbf{y}}^{\left(2\right)}$, …,${\mathbf{y}}^{\left(K\right)}$. The observed curves are composed by nominal signals ${\mathbf{f}}^{\left(1\right)}$, ${\mathbf{f}}^{\left(2\right)}$, …${\mathbf{f}}^{\left(K\right)}$ which may have some functional properties to which are eventually added anomaly signals ${\mathbf{a}}^{\left(1\right)}$, ${\mathbf{a}}^{\left(2\right)}$, …, ${\mathbf{a}}^{\left(K\right)}$ which may appear as functional outliers $a^k\left(t\right)$, $k=1,\dots ,K$. Once the data is discretized our data model becomes

$$\left\{\begin{array}{ccc}{\mathbf{y}}^{\left(1\right)}& =& {\mathbf{f}}^{\left(1\right)}+{\mathbf{a}}^{\left(1\right)}+{\epsilon }^{\left(1\right)}\\ {\mathbf{y}}^{\left(2\right)}& =& {\mathbf{f}}^{\left(2\right)}+{\mathbf{a}}^{\left(2\right)}+{\epsilon }^{\left(2\right)}\\ & ⋮& \\ {\mathbf{y}}^{\left(K\right)}& =& {\mathbf{f}}^{\left(K\right)}+{\mathbf{a}}^{\left(K\right)}+{\epsilon }^{\left(K\right)}\end{array}\right.$$

(1)

where, for each $k=1, \dots ,K$, ${\mathbf{f}}^{\left(k\right)}\in {\mathbb{R}}^{n\times 1}$ is the underlying unknown nominal signal, ${\mathbf{a}}^{\left(k\right)}\in {\mathbb{R}}^{n\times 1}$ is the potential anomaly, and ${\epsilon }^{\left(k\right)}\sim N(\mathbf{0},{\sigma }^2{I}_n)$ is the additive white noise. Note that if the Gaussian noise in some channel is correlated with known covariance structure, we can easily manage this situation reparametrizing the data.

The mathematical hypothesis is that, while the K components have a nominal behavior which results in a joint sparse representation into a RADWT dictionary, the anomalies can be of any shape, so they have no sparse representation in the same dictionary, and they are independent of each other, so an anomaly can be detected in any, some, or all components.

We do not hypothesize functions $f^{\left(k\right)}\left(t\right)$ belong to some functional Sobolev space $H_{p,q}^s[a,b]$ as it is usually done in functional nonparametric regression setting, instead we let these functions be much more general and we restrict our attention to their finite-dimensional representation. Since many physiological and physical signals exhibit a mixture of oscillatory and nonoscillatory behaviors (for example, speech, stock-market, biomedical, EEG, etc.), we suppose that each component is a “high resonance” signal or a “low resonance” signal or a sum of both types of signal. By a high-resonance component, we mean a signal consisting of multiple simultaneous sustained oscillations; in contrast, by a low-resonance component, we mean a signal consisting of nonoscillatory transients of unspecified shape and duration. We stress that the high and low resonance components of a signal can not be extracted from its high and low frequencies components in a time-scale decomposition, but they can be well represented by a high-Q factor RADWT and a low-Q factor RADWT respectively, see [17]. The RADWT is a normalized tight dictionary of $L_2\left(R\right)$ defined as ${\left\{{\left(\frac{q}{p}\right)}^{k/2}\psi \left({\left(\frac{q}{p}\right)}^{k}t+\frac{sp}{q}l\right)\right\}}_{k,l\in Z}$ where $\psi$ is a wavelet function and ($p,q,s$) is a triplet of parameters that gives the time-scale characteristic of the dictionary. In particular, the ratio $q/p>1$ is closely related to the scale (or frequency) dilatation factor, the parameter s is closely related to the time dilatation factor, and $\frac{p}{s(q-p)}$ is the redundant factor. The Q-factor depends on these parameters although there is not an explicit formula, in particular setting the dilatation factor $q/p$ between 1 and 2 and $s>1$ gives a RADWT with high Q-factor, while setting $s=1$ we obtain a low Q-factor RADWT with time-scale characteristic similar to the dyadic wavelet transform. In particular, when $q=2$, $p=1$ and $s=1$, the dictionary reduces to the classical wavelet basis. Given a finite energy signal $\mathbf{x}$ of length n and $J\in \mathbb{N}$ levels of decomposition, the RADWT transform is obtained by a sequence of proper downsampling operations and fast Fourier transforms; it ends up with $\lceil \frac{n{p}^J}{q^J}\rceil$ scaling coefficients (low-pass filtering) and $\lceil \frac{n{p}^j}{q^{j}s}\rceil$ wavelet coefficients (high-pass filtering) at each level $j=1,\dots J$. See [18] for details on fast analysis and synthesis schemes.

In the following, we will use these signal processing results in order to formulate our working setup. Let $\mathbf{\Psi }\in {\mathbb{R}}^{n\times d1}$ be the finite matrix representation of the low Q-factor analysis filter, where $d1$ indicates the cardinality of the low Q-factor RADWT, and let $\mathbf{\Phi }\in {\mathbb{R}}^{n\times d2}$ be the finite matrix representation of the high Q-factor analysis filter (the synthesis operators being just the transpose matrices), where $d2$ indicates the cardinality of the high Q-factor RADWT. Let us define the dictionary $\mathit{W}=[\mathbf{\Psi } \mathbf{\Phi }]\in {\mathbb{R}}^{n\times d}$, with $d=d1+d2$, then our working hypothesis is the following:

(H1)

signals ${\mathbf{f}}^{\left(k\right)}$ have a jointly sparse representation in $\mathit{W}$, i.e., setting ${\mathbf{f}}^{\left(k\right)}=\mathit{W}{\mathit{\beta }}^{\left(k\right)}$, ${\mathit{\beta }}^{\left(k\right)}\in {\mathbb{R}}^{d\times 1}$, the coefficients matrix

$$\begin{array}{c}\hfill \mathbf{B}=\left[\begin{array}{cccc}{\mathit{\beta }}^{\left(1\right)},& {\mathit{\beta }}^{\left(2\right)},& \dots ,& {\mathit{\beta }}^{\left(K\right)}\end{array}\right]\in {\mathbb{R}}^{d\times K}\end{array}$$

has a minimal number of rows different from zero. This means considering the following constraint

$$\begin{array}{c}\hfill {∥\mathbf{B}∥}_{0,2}={\displaystyle \sum _{i=1}^d}\mathcal{I}\left({∥{\mathbf{b}}_{i,.}∥}_2\right)={\displaystyle \sum _{i=1}^d}\mathcal{I}\left(\sqrt{{\displaystyle \sum _{k=1}^K}{\left({\beta }_i^{\left(k\right)}\right)}^2}\right)\le T\end{array}$$

(2)

where ${∥\mathbf{B}∥}_{2,0}$ counts the number of non zero rows, T is the a priori parameter indicating the expected degree of sparsity level, and $\mathcal{I}\left(a\right)$ is the indicator function:

$$\begin{array}{c}\hfill \mathcal{I}\left(a\right)=\left\{\begin{array}{cc}1\hfill & \mathrm{if} x\ne 0\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.\end{array}$$

(3)

It is worth observing that the role of dictionary matrix $\mathit{W}$ could be played by any other dictionary in which the nominal signals have a sparse joint representation. What we will expose in the following is not related to the nature of the dictionary $\mathit{W}$ and can therefore be considered a valid method for other types of signals as long as there is a dictionary that sparsely represents them. For example, $\mathit{W}$ could be made up of only $\mathbf{\Phi }$ if it is known that the f signals have only a high resonance component, or it could be made up of the union of other types of basis, frames, or dictionaries. However, when $\mathbf{\Psi }$ and $\mathbf{\Phi }$ are the finite representation of RADWT, they constitute thigh frames, i.e., $\mathbf{\Psi } {\mathbf{\Psi }}^{\prime }={\mathit{I}}_n$ and $\mathbf{\Phi } {\mathbf{\Phi }}^{\prime }={\mathit{I}}_n$, so $\mathit{W} {\mathit{W}}^{\prime }=2{\mathit{I}}_n$ and then, once we obtain the estimate of the coefficients it is straightforward to obtain the estimate of the signal. When the choice of the dictionary change, $\mathbf{\Psi } {\mathbf{\Psi }}^{\prime }={\mathit{I}}_n$ and $\mathbf{\Phi } {\mathbf{\Phi }}^{\prime }={\mathit{I}}_n$ is not necessarily satisfied, and hence signal reconstruction is not obvious.

3. Proposed Procedures

The linear model in (1) can be rewritten in terms of dictionary coefficients as follows

$$\begin{array}{c}\hfill \left\{\begin{array}{ccc}{\mathbf{y}}^{\left(1\right)}& =& \mathit{W}{\beta }^{\left(1\right)}+{\mathbf{a}}^{\left(1\right)}+{\mathit{\epsilon }}^{\left(1\right)}\\ {\mathbf{y}}^{\left(2\right)}& =& \mathit{W}{\mathit{\beta }}^{\left(2\right)}+{\mathbf{a}}^{\left(2\right)}+{\mathit{\epsilon }}^{\left(2\right)}\\ & ⋮& \\ {\mathbf{y}}^{\left(K\right)}& =& \mathit{W}{\mathit{\beta }}^{\left(K\right)}+{\mathbf{a}}^{\left(K\right)}+{\epsilon }^{\left(K\right)}\end{array}\right.\end{array}$$

(4)

which turns out to be a linear multiple regression model with a special common design matrix. A first and somewhat naive approach, but sub-optimal, would consist of treating each channel separately, ignoring the underlying common structure. This is the reason why such kind of problem is reformulated in terms of a unique regression problem in the following form:

$$\begin{array}{ccc}\left[\begin{array}{c}{\mathbf{y}}^{\left(1\right)}\\ {\mathbf{y}}^{\left(2\right)}\\ ⋮\\ {\mathbf{y}}^{\left(K\right)}\end{array}\right]& =& \left[\begin{array}{cccc}\mathit{W}& \mathbf{0}& \dots & \mathbf{0}\\ \mathbf{0}& \mathit{W}& \dots & \mathbf{0}\\ \dots & & & \dots \\ \mathbf{0}& \mathbf{0}& \dots & \mathit{W}\end{array}\right]\left[\begin{array}{c}{\mathit{\beta }}^{\left(1\right)}\\ {\mathit{\beta }}^{\left(2\right)}\\ ⋮\\ {\mathit{\beta }}^{\left(K\right)}\end{array}\right]+\left[\begin{array}{c}{\mathbf{a}}^{\left(1\right)}\\ {\mathbf{a}}^{\left(2\right)}\\ ⋮\\ {\mathbf{a}}^{\left(K\right)}\end{array}\right]+\left[\begin{array}{c}{\mathit{\epsilon }}^{\left(1\right)}\\ {\mathit{\epsilon }}^{\left(2\right)}\\ ⋮\\ {\mathit{\epsilon }}^{\left(K\right)}\end{array}\right]\hfill \\ \mathbf{y}& =& \mathit{X} \mathit{\beta }+\mathbf{a}+\mathit{\epsilon }\hfill \end{array}$$

(5)

with obvious correspondence between elements of the two expressions. So, $\mathbf{y}$ is a column vector of $n·K$ response variables, $\mathit{X}$ a design matrix of dimension $n·K\times K·d$, $\mathit{\beta }$ an unknown regression coefficients column vector of length $K·d$, $\mathbf{a}$ is an unknown anomalies column vector of length $n·K$, and, finally without loss of generality, we let $\mathit{\epsilon }$ be a $n·K$-variate Gaussian random column vector with zero mean and covariance matrix ${\sigma }^2{\mathit{I}}_{n·K}$. Our regression problem (5) has $d·K+n·K$ regression parameters and only $n·K$ data point, so it clearly falls into the class of high-dimensional regression problems. Under the working hypothesis (H1), we expect $\mathit{\beta }$ to be group sparse, i.e., for many $j=1,\dots ,d$ we expect that ${\beta }_j^{\left(k\right)}=0$, for all $k=1,\dots ,K$. This provides the following nonoverlapping group structure for the whole vector

$$\{1,2\dots ,K·d\}=G_1\cup \dots \cup G_d,$$

(6)

with

$$G_j=\{j,j+d,j+2d\dots ,j+(K-1)d\}, j=1,\dots ,d$$

group of size K. Let us define the following $l_{2,1}$ norm

$${\parallel \mathit{\beta }\parallel }_{2,1}={\displaystyle \sum _{j=1}^d}{∥{\mathit{\beta }}_{G_j}∥}_2\hspace{1em}\mathrm{and}\hspace{1em}{\parallel \mathbf{a}\parallel }_{2,1}={\displaystyle \sum _{j=1}^K}{∥{\mathbf{a}}^{\left(k\right)}∥}_2$$

with ${\mathit{\beta }}_{G_j}$ denoting the reduction of vector $\mathit{\beta }$ to the subset of index $G_j$. The problem we are dealing with is not a classic regression problem, because our interest is to simultaneously estimate the vector $\mathbf{a}$, more specifically the ${\mathbf{a}}^{\left(1\right)}$, …, ${\mathbf{a}}^{\left(K\right)}$ anomalies that compose it, and the $\mathit{\beta }$ vector. For example, if an anomaly is present in the first channel, the vector ${\mathbf{a}}^{\left(1\right)}$, representing the residual of the linear regression of ${\mathbf{y}}^{\left(1\right)}$ on $\mathit{W}$ will have a $l_2$-norm much higher than the residual in another channel with no anomaly. While the anomalies vectors are not connected, because the anomalies can appear in any, some, or all channels, the underlying signals are not independent since by hypothesis they share the same spectral characteristic. For these reasons, we propose solving the following convex minimization problem

$$(\widehat{\mathit{\beta }},\widehat{\mathbf{a}})=\underset{\mathit{\beta }\in {\mathbb{R}}^{K·d\times 1}, \mathbf{a}\in {\mathbb{R}}^{K·n\times 1}}{argmin}\hspace{1em}\underset{F(\mathit{\beta },\mathbf{a})}{\underbrace{\frac{1}{2}{\parallel \mathbf{y}-\mathit{X}\mathit{\beta }-\mathbf{a}\parallel }_2^2+{\lambda }_1{\parallel \mathit{\beta }\parallel }_{2,1}+{\lambda }_2{\parallel \mathbf{a}\parallel }_{2,1}}}$$

(7)

where ${\lambda }_1>0$ is a regularization parameter that controls the group sparsity of vector $\mathit{\beta }$, while ${\lambda }_2>0$ is a regularization parameter that controls the number of expected anomalies. Both the regularization parameters represent a trade-off between the fit and the complexity of the model. Being ${\lambda }_2$ fixed, ${\lambda }_1\to 0$ implies a non sparse $\mathit{\beta }$, generalizing the model proposed in [13] to the multichannel setting, on the contrary ${\lambda }_1\to \infty$ implies a sparser $\mathit{\beta }$ vector, increasing the energy of the anomalies. On the other hand, being ${\lambda }_1$ fixed, ${\lambda }_2\to 0$ detects anomalies in each channel, while ${\lambda }_2\to \infty$ brings to a model without anomalies. Their choice is a delicate point and we will discuss it in the Implementation, Section 4. Finally, for each $k\in \{1,2,\dots ,K\}$ we detect an anomaly if the obtained estimate ${∥{\widehat{\mathbf{a}}}^{\left(k\right)}∥}_2\ne 0$, where $\widehat{\mathbf{a}}=\left({\widehat{\mathbf{a}}}^{\left(1\right)},\dots ,{\widehat{\mathbf{a}}}^{\left(K\right)}\right)$ is the solution of the optimization problem (7). Moreover, the intensity of the detected anomaly, measured as $\parallel {\widehat{\mathbf{a}}}^{\left(k\right)}{\parallel }_2$, can be considered as a score for the anomaly and treated as an incipient level of fault according to the specific application which defines an alarm threshold.

From a statistical perspective, the problem given in Equation (7) falls into the class of group lasso regression problem applied to the augmented parameter vector $\tilde{\mathit{\beta }}={({\mathit{\beta }}^t;{\mathbf{a}}^t)}^t$ and augmented design matrix $\tilde{\mathit{X}}=[\mathit{X} I_{n·K}]$. Indeed, the problem in Equation (7) can be rewritten as

$$\widehat{\tilde{\mathit{\beta }}}=\underset{\tilde{\mathit{\beta }}\in {\mathbb{R}}^{K·d+K·n\times 1}}{argmin}\hspace{1em}\frac{1}{2}{\parallel \mathbf{y}-\tilde{\mathit{X}}\tilde{\mathit{\beta }}\parallel }_2^2+{\lambda }_1\mathcal{P}\left(\tilde{\mathit{\beta }}\right).$$

with group Lasso penalty $\mathcal{P}(\widetilde{\mathit{\beta })}={\sum }_{j=1}^d{∥{\mathit{\beta }}_{G_j}∥}_2+\frac{{\lambda }_2}{{\lambda }_1}{\sum }_{j=1}^K{∥{\mathbf{a}}^{\left(k\right)}∥}_2$. If the unknown vector $\tilde{\mathit{\beta }}$ is sparse, under appropriate hypothesis on the design matrix $\tilde{\mathit{X}}$, consistency of the proposed estimator is guaranteed, see [19,20,21].

From a numerical perspective, problem (7) is jointly convex in $\mathit{\beta }$ and $\mathbf{a}$, and its simple form suggests that we can apply an alternating optimization given initial estimates for $\mathit{\beta }$ and $\mathbf{a}$. Of course, we need an estimate for $\mathit{\beta }$ that takes into account the potential presence of the anomaly, i.e., robust.

Given an initial robust estimate for $\mathit{\beta }$, i.e., ${\widehat{\mathit{\beta }}}^0$, we define ${\widehat{\mathbf{a}}}^0=\mathbf{y}-\mathit{X}{\widehat{\mathit{\beta }}}^0$, then at each iteration the procedure solves the following two sub-problems,

• given $\mathbf{a}$, estimate $\mathit{\beta }$

  $$\widehat{\mathit{\beta }}=\underset{\mathit{\beta }\in {\mathbb{R}}^{Kd}}{argmin}\underset{F_2\left(\mathit{\beta }\right)}{\underbrace{\left\{\frac{1}{2}\left|\right|\mathbf{y}-\mathit{X}\mathit{\beta }-{\mathbf{a}\left|\right|}_2^2+{\lambda }_1{\left|\right|\mathit{\beta }\left|\right|}_{2,1}+{\lambda }_2{\parallel \mathbf{a}\parallel }_{2,1}\right\}}}$$

  (8)
• given $\mathit{\beta }$, estimate $\mathbf{a}$

  $$\widehat{\mathbf{a}}=\underset{\mathbf{a}\in {\mathbb{R}}^{Kn}}{argmin}\underset{F_1\left(\mathbf{a}\right)}{\underbrace{\left\{\frac{1}{2}\left|\right|\mathbf{y}-\mathit{X}\mathit{\beta }-{\mathbf{a}\left|\right|}_2^2+{\lambda }_1{\left|\right|\mathit{\beta }\left|\right|}_{2,1}+{\lambda }_2{\parallel \mathbf{a}\parallel }_{2,1}\right\}}}.$$

  (9)

Iteration stops when a given number of maximum iteration is reached or when the relative error on the anomaly vector $\mathbf{a}$ is below a fixed threshold. Let us give a closer look at the two sub-problems. Function $F_1\left(\mathbf{a}\right)=F_1\left({\mathbf{a}}^{\left(1\right)},\dots ,{\mathbf{a}}^{\left(K\right)}\right)$ is separable in each channel, i.e.,

$$F_1\left(\mathbf{a}\right)={\displaystyle \sum _{k=1}^K}\frac{1}{2}{∥{\mathbf{r}}^{\left(k\right)}-{\mathbf{a}}^{\left(k\right)}∥}_2^2+{\lambda }_2{∥{\mathbf{a}}^{\left(k\right)}∥}_2+const,$$

where ${\mathbf{r}}^{\left(k\right)}={\mathbf{y}}^{\left(k\right)}-\mathit{W}{\mathit{\beta }}^{\left(k\right)}$ is the current residual of the regression of ${\mathbf{y}}^{\left(k\right)}$ on $\mathit{W}$; so the solution of sub-problem (9) is obtained by the multivariate-Soft thresholding rule

$${\widehat{\mathbf{a}}}^{\left(k\right)}=\frac{{\mathbf{r}}^{\left(k\right)}}{{∥{\mathbf{r}}^{\left(k\right)}∥}_2}{\left({∥{\mathbf{r}}^{\left(k\right)}∥}_2-{\lambda }_2\right)}_{+},$$

(10)

where the directional vector $v/\left|\right|v\left|\right|$ plays the rule of $sign\left(\right)$ function in classical one dimensional setting, see [22]. In Equation (10), the Soft thresholding operator ${(.)}_{+}$ acts on the vector ${\mathbf{r}}^{\left(k\right)}$ by shortening it towards 0 by an amount ${\lambda }_2$ if its norm is greater than ${\lambda }_2$, by setting it to zero if its norm is less than ${\lambda }_2$. For the seek of completeness, we sketch below how to derive the above statement. Vector in Equation (10) is the solution of the following convex problem

$${\widehat{\mathbf{a}}}^{\left(k\right)}=\underset{\mathbf{a}\in {\mathbb{R}}^{n\times 1}}{argmin}\left\{{\displaystyle \sum _{k=1}^k}\frac{1}{2}{∥{\mathbf{r}}^{\left(k\right)}-{\mathbf{a}}^{\left(k\right)}∥}_2^2+{\lambda }_2{∥{\mathbf{a}}^{\left(k\right)}∥}_2\right\}.$$

Considering the case ${\mathbf{a}}^{\left(k\right)}\ne 0$ and setting the derivative with respect to ${\mathbf{a}}^{\left(k\right)}$ to zero, we obtain

$$-\left({\mathbf{r}}^{\left(k\right)}-{\mathbf{a}}^{\left(k\right)}\right)+{\lambda }_2\frac{{\mathbf{a}}^{\left(k\right)}}{{∥{\mathbf{a}}^{\left(k\right)}∥}_2}=0,$$

from which the following two equations derive:

$${\mathbf{a}}^{\left(k\right)}={\left(1+\frac{{\lambda }_2}{{∥{\mathbf{a}}^{\left(k\right)}∥}_2}\right)}^{-1}{\mathbf{r}}^{\left(k\right)}\hspace{1em}\mathrm{and}\hspace{1em}{∥{\mathbf{a}}^{\left(k\right)}∥}_2={\left(1+\frac{{\lambda }_2}{{∥{\mathbf{a}}^{\left(k\right)}∥}_2}\right)}^{-1}{∥{\mathbf{r}}^{\left(k\right)}∥}_2,$$

by solving the second equation in $∥{\mathbf{a}}^{\left(k\right)}∥$ and substituting into the first we get,

$${\widehat{\mathbf{a}}}^{\left(k\right)}=\frac{{\mathbf{r}}^{\left(k\right)}}{{∥{\mathbf{r}}^{\left(k\right)}∥}_2}\left({∥{\mathbf{r}}^{\left(k\right)}∥}_2-{\lambda }_2\right).$$

(11)

Now, consider the case ${\mathbf{a}}^{\left(k\right)}=0$. We need to impose that the $\mathbf{0}$ vector belongs to the pseudogradient evaluated at ${\mathbf{a}}^{\left(k\right)}$, i.e., there exists a vector $\mathit{v}\in {\mathbb{R}}^{n\times 1}$, $\mathit{v}\ne 0$ and ${\parallel \mathit{v}\parallel }_2\le 1$ such that $-{\mathbf{r}}^{\left(k\right)}+{\lambda }_2\mathit{v}=0$. This is possible only if ${∥{\mathbf{r}}^{\left(k\right)}∥}_2\le {\lambda }_2$. Finally, combining both cases into a single equation, we get exactly the expression in Equation (10).

Let us now consider sub-problem (8). It is possible to prove that minimization of $F_2\left(\mathit{\beta }\right)=F(\mathit{\beta },\widehat{\mathbf{a}})$, where $\widehat{\mathbf{a}}$ is the solution of Equation (9), i.e., vectors ${\widehat{\mathbf{a}}}^{\left(k\right)}$ given by Equation (10), is equivalent to solve the following Huber’s cost functional with parameter ${\lambda }_2$ plus a group lasso penalty. i.e.,

$$\widehat{\mathit{\beta }}=\underset{\mathit{\beta }\in {\mathbb{R}}^{K·d\times 1}}{argmin}\left\{{\displaystyle \sum _{k=1}^K}{\rho }_{{\lambda }_2}^H\left({\mathbf{y}}^{\left(k\right)}-\mathit{W}{\mathit{\beta }}^{\left(k\right)}\right)+{\lambda }_1{\left|\right|\mathit{\beta }\left|\right|}_{2,1}\right\}.$$

Here the mutivariate Huber cost function is given by the following rule:

$${\rho }_{{\lambda }_2}^H\left(v\right)=\left\{\begin{array}{ccc}{∥v∥}_2^2/2\hfill & if& {∥v∥}_2\le {\lambda }_2\\ {\lambda }_2{∥v∥}_2-{\lambda }_2^2/2\hfill & if& {∥v∥}_2>{\lambda }_2,\end{array}\right.$$

(12)

for any $v\in {\mathbb{R}}^{n\times 1}$.

For the sake of completeness, we sketch below how to derive the above statement. Let $I=\left\{k:{\widehat{\mathbf{a}}}^{\left(k\right)}=0\right\}=\left\{k:{∥{\mathbf{r}}^{\left(k\right)}∥}_2<{\lambda }_2\right\}$ and $I^c$ be its complement.

$$\begin{array}{ccc}\hfill F_1\left(\mathit{\beta }\right)& =& F(\mathit{\beta },\widehat{\mathbf{a}})\hfill \\ & =& \frac{1}{2}{\displaystyle \sum _{k=1}^K}{∥{\mathbf{y}}^{\left(k\right)}-\mathit{W}{\mathit{\beta }}^{\left(k\right)}-{\widehat{\mathbf{a}}}^{\left(k\right)}∥}_2^2+{\lambda }_1{\parallel \mathit{\beta }\parallel }_{2,1}+{\lambda }_2{\displaystyle \sum _{k=1}^K}{∥{\widehat{\mathbf{a}}}^{\left(k\right)}∥}_2\hfill \\ & =& \frac{1}{2}\sum _{k\in I}{∥{\mathbf{y}}^{\left(k\right)}-\mathit{W}{\mathit{\beta }}^{\left(k\right)}∥}_2^2+\frac{1}{2}\sum _{k\in I^c}{\lambda }_2^2+{\lambda }_2\sum _{k\in I^c}\left({∥{\mathbf{r}}^{\left(k\right)}∥}_2-{\lambda }_2\right)+{\lambda }_1{∥\mathit{\beta }∥}_{2,1}\hfill \\ & =& \frac{1}{2}\sum _{k\in I}{∥{\mathbf{y}}^{\left(k\right)}-\mathit{W}{\mathit{\beta }}^{\left(k\right)}∥}_2^2+{\lambda }_2\sum _{k\in I^c}{∥{\mathbf{r}}^{\left(k\right)}∥}_2-\frac{1}{2}\sum _{k\in I^c}{\lambda }_2^2+{\lambda }_1{∥\mathit{\beta }∥}_{2,1}.\hfill \end{array}$$

In robust regression, i.e., regression in presence of outliers, it is well known that Huber loss is less sensitive to outliers than quadratic loss because it combines the squared loss and the absolute loss in an adaptive way. Intuitively, when the data points are not too big, Huber’s loss function penalizes like the squared loss; otherwise, it penalizes like the absolute loss. However, it is not resistant to high leverage outliers (outliers in the predictors) because it never rejects gross outliers that have moderate or high leverage. Its breakdown point is $1/n$. Indeed, according to [23], a convex criterion is inherently incompatible with robustness.

The proposed procedure is summarized in Algorithm 1.

Algorithm 1 Calculate $\left(\widehat{\mathit{\beta }},\widehat{\mathbf{a}}\right)$ with the Soft Thresholding operator

Input data: K curves ${\mathbf{y}}^{\left(1\right)},\dots ,{\mathbf{y}}^{\left(K\right)}$; parameters for the calculation of the low Q-factor and high Q-factor analysis filters $p,q,s,J;$ maximum number of iterations max_iter; $ϵ>0$ (1) Build matrices $\mathbf{\Phi }$ and $\mathbf{\Psi }\to \mathit{X}$, see Equation (5) (2) Set initial estimate for $j=0$, ${\widehat{\mathit{\beta }}}^0$ and ${\widehat{\mathbf{a}}}^0=\mathbf{y}-\mathit{X}{\widehat{\mathit{\beta }}}^0$ while no convergence do $j=j+1$ i. solve ${\widehat{\mathit{\beta }}}^j=\underset{\mathit{\beta }\in {\mathbb{R}}^{Kd}}{argmin}\left\{\frac{1}{2}\left|\right|\mathbf{y}-\mathit{X}\mathit{\beta }-{\widehat{\mathbf{a}}}^{j-1}{\left|\right|}_2^2+{\lambda }_1{\left|\right|\mathit{\beta }\left|\right|}_{2,1}+{\lambda }_2{\parallel {\widehat{\mathbf{a}}}^{j-1}\parallel }_{2,1}\right\}$ ii. evaluate residuals ${\widehat{\mathbf{r}}}^j=\mathbf{y}-\mathit{X}{\widehat{\mathit{\beta }}}^j=\left[\begin{array}{c}{\widehat{\mathbf{r}}}^{\left(1\right),j}\\ {\widehat{\mathbf{r}}}^{\left(2\right),j}\\ ⋮\\ {\widehat{\mathbf{r}}}^{\left(K\right),j}\end{array}\right]$ iii. Soft threshold the residuals ${\widehat{\mathbf{a}}}^{\left(k\right),j}=\frac{{\widehat{\mathbf{r}}}^{\left(k\right),j}}{{∥{\widehat{\mathbf{r}}}^{\left(k\right),j}∥}_2}{\left({∥{\widehat{\mathbf{r}}}^{\left(k\right),j}∥}_2-{\lambda }_2\right)}_{+}$ iv. check convergence $\frac{{∥{\widehat{\mathbf{a}}}^j-{\widehat{\mathbf{a}}}^{j-1}∥}_2}{{∥{\widehat{\mathbf{a}}}^{j-1}∥}_2}<ϵ$ or $j$ = max_iter end while Output:$\left(\widehat{\mathit{\beta }},\widehat{\mathbf{a}}\right)=\left({\widehat{\mathit{\beta }}}^j,{\widehat{\mathbf{a}}}^j\right)$

Possible Improvements

In this section we explore the possibility to improve our estimator, adopting a different type of threshold operator. We are inspired by [13], where the authors deal with the outliers detection problem in the case of one channel. In the paper they propose a two step procedure: after a proper initialization for $\mathit{\beta }$, in the first step, they apply a Soft thresholding operator to the residual, in the second step they use Ordinary Least Squares (OLS) to update vector $\mathit{\beta }$. The main difference with our procedure is in the second step, where we perform a group lasso regression instead of an OLS to face the multitask regression. Although in [13] the authors deal with the outliers detection, looking for pointwise anomalies, the philosophy of their estimator is comparable to ours. Moreover, they note that in the presence of multiple outliers, the Soft threshold is not able to deal with masking and swamping effects, and propose to replace Soft-thresholding with Hard-thresholding obtaining a great improvement. Then they generalize the idea and introduce an entire class of methods, namely $\mathsf{\Theta }-IPOD$ which uses a general $\mathsf{\Theta }$ thresholding operator instead of the Soft thresholding operator, discovering important results and outstanding performance. In this direction of research, we explore the possibility to improve our estimator, substituting the first step of our procedure, i.e., the multivariate Soft thresholding operator expressed in Equation (10), by an appropriate $\mathsf{\Theta }$ operator as done by the authors of [13]. Specifically, we substitute Equation (10), by the following Hard-thresholding operator

$${\widehat{\mathbf{a}}}^{\left(k\right)}={\mathbf{r}}^{\left(k\right)}\mathbb{I}\left({∥{\mathbf{r}}^{\left(k\right)}∥}_2<{\lambda }_2\right),$$

(13)

where $\mathbb{I}(·)$ is the indicator function defined in Equation (3). We stress that substituting Equation (13) into the first step of our procedure, the second step becomes equivalent to a skipped-mean loss penalized with a group lasso penalty. We sketch below how to derive it.

$$\begin{array}{ccc}\hfill F_2\left(\mathit{\beta }\right)& =& F(\mathit{\beta },\widehat{\mathbf{a}})\hfill \\ & =& \frac{1}{2}{\displaystyle \sum _{k=1}^K}{∥{\mathbf{y}}^{\left(k\right)}-\mathit{W}{\mathit{\beta }}^{\left(k\right)}-{\widehat{\mathbf{a}}}^{\left(k\right)}∥}_2^2+{\lambda }_1{∥\mathit{\beta }∥}_{2,1}+\frac{{\lambda }_2^2}{2}{\displaystyle \sum _{k=1}^K}\mathbb{I}\left({\widehat{\mathbf{a}}}^{\left(k\right)}\ne 0\right)\hfill \\ & =& \frac{1}{2}\sum _{k\in I}{∥{\mathbf{y}}^{\left(k\right)}-\mathit{W}{\mathit{\beta }}^{\left(k\right)}∥}_2^2+\frac{{\lambda }_2^2}{2}card\left(I^c\right)+{\lambda }_1{∥\mathit{\beta }∥}_{2,1}\hfill \end{array}$$

(14)

where $I=\left\{k:{\widehat{\mathbf{a}}}^{\left(k\right)}=0\right\}=\left\{k:{∥{\mathbf{r}}^{\left(k\right)}∥}_2\le {\lambda }_2\right\}$ and $I^c$ its complement.

Hence, substituting Soft threshold with Hard threshold automatically makes the second step equivalent to the following penalized skipped-mean loss

$$\widehat{\mathit{\beta }}=\underset{\mathit{\beta }\in {\mathbb{R}}^{K·d\times 1}}{argmin}\left\{{\displaystyle \sum _{k=1}^K}{\rho }_{{\lambda }_2}^{SM}\left({\mathbf{y}}^{\left(k\right)}-\mathit{W}{\mathit{\beta }}^{\left(k\right)}\right)+{\lambda }_1{\left|\right|\mathit{\beta }\left|\right|}_{2,1}\right\}.$$

with loss defined as:

$${\rho }_{{\lambda }_2}^{SM}\left(v\right)=\left\{\begin{array}{ccc}{∥v∥}_2^2/2\hfill & if& {∥v∥}_2\le {\lambda }_2\\ {\lambda }_2^2/2\hfill & if& {∥v∥}_2>{\lambda }_2,\end{array}\right.$$

(15)

for any $v\in {\mathbb{R}}^{n\times 1}$. Note that skipped-mean loss is not-convex and is a special case of Hampel loss, that belongs to the family of redescending M-estimators, see [24], with high breakdown point, close to 0.5. This means that is robust to outliers. For a summary, see Appendix 1 in [16].

Figure 1 shows a comparison between quadratic loss, Huber loss ${\rho }_{{\lambda }_2}^H\left(v\right)$, and skipped-mean loss ${\rho }_{{\lambda }_2}^{SM}\left(v\right)$.

We can conclude that, when using Hard threshold instead of Soft threshold, we solve the following problem instead of the problem in Equation (7)

$$(\widehat{\mathit{\beta }},\widehat{\mathbf{a}})\in \underset{\mathit{\beta }\in {\mathbb{R}}^{Kd}, \mathbf{a}\in {\mathbb{R}}^{Kn}}{argmin}\left\{\frac{1}{2}{\parallel \mathbf{y}-\mathit{X}\mathit{\beta }-\mathbf{a}\parallel }_2^2+{\lambda }_1{\parallel \mathit{\beta }\parallel }_{2,1}+\frac{{\lambda }_2^2}{2}{\parallel \mathbf{a}\parallel }_{2,0}\right\},$$

(16)

with ${\parallel \mathbf{a}\parallel }_{2,0}={\sum }_{k=1}^K\mathbb{I}\left({\mathbf{a}}^{\left(k\right)}\ne 0\right)$, the $l_0$ grouped norm.

Of course, the problem in Equation (16) is not convex, because the complexity penalty is not convex; therefore, we can only aspire to find one of the possible local minima of the objective function. Nonconvex functions may possess local optima that are not global optima, and our iterative method may terminate undesirably in one of these local optima. From a statistical perspective, although theoretical results for nonconvex penalties have been studied in [14,15] proving that all local optima are essentially as good as a global optimum, we cannot apply those results because our penalty does not satisfy their hypothesis. Therefore, our finding for this second procedure is purely empirical and we include the side condition $\parallel \widehat{\mathit{\beta }}{\parallel }_1+{\parallel \mathbf{a}\parallel }_1<R$ to guarantee the existence of at least one local/global optima. In addition, we will require $R\ge \parallel {\mathit{\beta }}_0{\parallel }_1+{\parallel {\mathbf{a}}_0\parallel }_1$ so that the true regression vector ${\mathit{\beta }}_0$ and the true anomaly vector ${\mathbf{a}}_0$, that satisfy Equation (4), are feasible.

The proposed procedure is summarized in Algorithm 2.

Algorithm 2 Calculate $\left(\widehat{\mathit{\beta }},\widehat{\mathbf{a}}\right)$ with the Hard Thresholding operator

Input data: K curves ${\mathbf{y}}^{\left(1\right)},\dots ,{\mathbf{y}}^{\left(K\right)}$; parameters for the calculation of the low Q-factor and high Q-factor analysis filters $p,q,s,J;$ maximum number of iterations max_iter; $ϵ>0$ (1) Build matrices $\mathbf{\Phi }$ and $\mathbf{\Psi }\to \mathit{X}$, see Equation (5) (2) Set initial estimate for $j=0$, ${\widehat{\mathit{\beta }}}^0$ and ${\widehat{\mathbf{a}}}^0=\mathbf{y}-\mathit{X}{\widehat{\mathit{\beta }}}^0$ while no convergence do $j=j+1$ i. solve ${\widehat{\mathit{\beta }}}^j=\underset{\mathit{\beta }\in {\mathbb{R}}^{Kd}}{argmin}\left\{\frac{1}{2}\left|\right|\mathbf{y}-\mathit{X}\mathit{\beta }-{\widehat{\mathbf{a}}}^{j-1}{\left|\right|}_2^2+{\lambda }_1{\left|\right|\mathit{\beta }\left|\right|}_{2,1}+{\lambda }_2{\parallel {\widehat{\mathbf{a}}}^{j-1}\parallel }_{2,0}\right\}$ ii. evaluate residuals ${\widehat{\mathbf{r}}}^j=\mathbf{y}-\mathit{X}{\widehat{\mathit{\beta }}}^j=\left[\begin{array}{c}{\widehat{\mathbf{r}}}^{\left(1\right),j}\\ {\widehat{\mathbf{r}}}^{\left(2\right),j}\\ ⋮\\ {\widehat{\mathbf{r}}}^{\left(K\right),j}\end{array}\right]$ iii. Hard threshold the residuals ${\widehat{\mathbf{a}}}^{\left(k\right),j}={\mathbf{r}}^{\left(k\right),j}\mathbb{I}\left({∥{\mathbf{r}}^{\left(k\right),j}∥}_2<{\lambda }_2\right),$ iv. check convergence $\frac{{∥{\widehat{\mathbf{a}}}^j-{\widehat{\mathbf{a}}}^{j-1}∥}_2}{{∥{\widehat{\mathbf{a}}}^{j-1}∥}_2}<ϵ$ or $j$= max_iter end while Output:$\left(\widehat{\mathit{\beta }},\widehat{\mathbf{a}}\right)=\left({\widehat{\mathit{\beta }}}^j,{\widehat{\mathbf{a}}}^j\right)$

4. Implementation

In this section, we describe all the issues that make the proposed procedure effective: namely the initialization step, the numerical algorithm, and the choice of the regularization parameters ${\lambda }_1$ and ${\lambda }_2$.

As described in Section 3, we need to initialize the two step procedure by a robust estimator of parameters $\mathit{\beta }$ which gives a robust estimator of $\mathit{X}\mathit{\beta }$ to plug into expression of Equation (9). First, we note that vector ${\left(\mathit{X}\mathit{\beta }\right)}^t=\left({\left(\mathit{W}{\mathit{\beta }}^{\left(1\right)}\right)}^t,\dots ,{\left(\mathit{W}{\mathit{\beta }}^{\left(K\right)}\right)}^t\right)$, hence to initialize our procedure is enough to establish a robust estimator for each of the signal component $f^{\left(k\right)}=\mathit{W}{\mathit{\beta }}^{\left(k\right)}$. In our procedure, under the prior knowledge that the number of channels with anomaly are less then the half number of channels, we initialize $f^{\left(k\right)}$ by the median of the data channels, namely $median\left(y^{\left(1\right)},\dots ,y^{\left(K\right)}\right)$

As regards the numerical algorithm we note that the first step described in Equation (9) has a closed-form solution given in Equation (10) and Equation (13) in the case of Soft and Hard threshold respectively; the second step in Equation (8) requires the solution of a linear regression problem with a group Lasso penalty. Hence for the solution of the second step we employed the grpreg R package that implements the efficient Group Descendent Algorithm presented in [25,26]. This algorithm works groupwise by using the separability of the model (5) in terms of a group of variables, i.e., it updates each group of variables freezing the other groups to their current value until convergence. The updating of each group of variables is performed through a multivariate soft-thresholding operator, under the assumption of “orthonormal groups”. We stress that the “orthonormal group” property refers to the condition ${\mathit{X}}_{G_j}^t{\mathit{X}}_{G_j}=I$, with ${\mathit{X}}_{G_j}$ denoting the reduction of design matrix $\mathit{X}$ to columns of the subset of index $G_j$. When this condition is not satisfied the grpreg automatically orthonormalizes the design matrix, but this practice leads to a slight modification of the $l_1/l_2$-norm contained in the penalty, as pointed out in [27,28]. However, this is not our case, because the design matrix defined in Equation (5) satisfies the “orthonormal groups” property by construction, and hence, we can take complete advantage of the Group Descendent Algorithm implemented in the grpreg package.

As in any penalized regression approach, the choice of the regularization parameter is really strategic. In particular, in the proposed procedure the regularization parameter has two components, namely ${\lambda }_1$ and ${\lambda }_2$. The latter controlling the degree of regularization in the regression model on parameter $\mathit{\beta }$, the last controlling the threshold over which the residual of the previous regression is considered not acceptable. Under the data model described in Equation (1), the expected size of the residual ${\mathbf{r}}^{\left(k\right)}$, in absence of anomaly in channel k, is given by $\sigma \sqrt{n}$, hence it is natural to fix ${\lambda }_2=\widehat{\sigma }\sqrt{n}$ where $\widehat{\sigma }$ is an estimator of the noise variance. In particular, we adopt $\widehat{\sigma }=median\left({∥{\mathbf{r}}^{\left(1\right)}{)\parallel }_2,\dots ,\parallel {\mathbf{r}}^{\left(K\right)}∥}_2\right)/\sqrt{n}$ where ${\mathbf{r}}^{\left(k\right)}$ is the residual after having initialized $f^{\left(k\right)}$ as described above. The choice of the regularization ${\lambda }_1$ is more demanding; we avoided CV criterion since it is quite computationally heavy and in principle when applied on the $(\mathbf{y},\mathit{X})$ data, a large prediction error can be due to both a suboptimal $\mathit{\beta }$ as well as to the presence of an anomaly. Hence we suggest tuning parameter ${\lambda }_1$ by the BIC criterion, which is much less computationally demanding and easily modifiable to our setting. In particular, we adopted the following formula

$$BIC\left({\lambda }_1\right)=nR log(RSS\left({\lambda }_1\right)/nR)+df\left({\lambda }_1\right) log\left(nR\right)$$

where $df\left({\lambda }_1\right)$ is evaluated according to formula (22) in [26] and $RSS\left({\lambda }_1\right)={\parallel \mathbf{y}-\mathit{X}\widehat{\mathit{\beta }}-\widehat{\mathbf{a}}\parallel }_2^2$, with $\widehat{\mathit{\beta }}$ and $\widehat{\mathbf{a}}$ estimates obtained for ${\lambda }_1$. The grid of ${\lambda }_1$ values has been obtained as in Section 3.5 of [26].

5. Simulations and Real Examples

To show the performance of the proposed methodology, we performed numerical experiments both in a simulation setting and in a real dataset case.

5.1. Synthetic Data

We generated data according to the model in Equation (1) with some known functions $f^k\left(t\right)$ and some known anomalies and evaluated the performance of the proposed techniques in terms of signals reconstruction and anomalies detection.

Since the proposed method deals with nominal signals with fast oscillating characteristics, we choose the HiSine and the TwoChirp signals (represented in Figure 2 and Figure 3) among the classical signals employed in the literature. Both can be well represented by using an appropriate RADWT. In particular, a RADWT with Q-factor almost 5 ($p=8$, $q=9$, $s=3$ and $j=10$) is appropriate for their sparse representation.

As regards the anomalies, there are no assumptions on their shape, i.e., they can be of any duration and shape and they can appear in any of the signal components, so we decided to model them by the following formula

$$a\left(t\right)=M·I(t_1\le t\le t_2)$$

(17)

to represent situations where the mean of the signal undergoes to a shift during a specific interval of time $(t_1,t_2)$; where the shift $M=h·{max}_{1\le i\le n,k}\left({\mathbf{y}}^{\left(k\right)}\left(t_i\right)\right)$ with $h=0.5 \mathrm{or} 2$, to mimic situation where the anomaly is weak or strong with respect to the underlying signals. We generated data according to model (1), building K signals, ${\mathbf{f}}^{\left(1\right)},\dots ,{\mathbf{f}}^{\left(K\right)}$ of length $n=256$ and for such a choice the W matrix has dimension $n\times d=256\times 695$. Each signal has been generated randomizing the original signal $\mathbf{f}$ ($HiSine$ or $Twochirp$) by the formula ${\mathbf{f}}^{\left(k\right)}=\mathbf{f}·u$, where u is a vector of i.i.d. variables with distribution $\mathrm{Unif}(0,1)$. These generations guarantee that the true underlying signals ${\mathbf{f}}^{\left(1\right)},\dots ,{\mathbf{f}}^{\left(K\right)}$ have similar shape and the same sparsity patterns. According to the model in Equation (1), we added noise to each channel as ${\mathit{\epsilon }}^{\left(k\right)}\sim N(\mathbf{0},{\sigma }^{2}I)$ with ${\sigma }^2$ related to the variance of the true underlying signal realizing three different type of $SNR=(0.5,1,6)$ to mimic situations of different severity of noise. For completeness, here is the $SNR$ expression

$$SNR=\frac{\frac{1}{K}{\sum }_{i=1}^K\mathrm{Var}\left({\mathbf{f}}^{\left(k\right)}\right)}{{\sigma }^2}.$$

Finally, we added anomalies ${\mathbf{a}}^{\left(k\right)}$ to some channels as in Formula (17).

To sketch, we have considered two settings:

• $\mathbf{f}=HiSine$, $K=3$, ${\mathbf{a}}^{\left(1\right)}$ as in Formula (17) with $(t_1,t_2)=(78,177)$;
• $\mathbf{f}=TwoChirp$, $K=5$, ${\mathbf{a}}^{\left(1\right)}$ as in Formula (17) with $(t_1,t_2)=(78,177)$ and ${\mathbf{a}}^{\left(4\right)}$ as in Formula (17) with $(t_1,t_2)=(157,256).$

To summarize, for each of these two settings ($\mathbf{f}=HiSine$ and $\mathbf{f}=TwoChirp$), we analyzed two different levels of anomalies ($h=0.5$ and $h=2$ in Formula (17)) and three different levels of SNR ($SNR=0.5,1$ and 6). These choices are arbitrary but they were dictated by the idea of understanding a sort of breakdown point of our procedure. Naturally, we expect that the weaker the anomalies and the stronger the noise, the worse the problem. Finally, we evaluated performance by computing the MSE for each signal’s component as

$${\mathrm{MSE}}_k=\frac{1}{n}{\displaystyle \sum _{i=1}^n}{\left({\widehat{f}}^{\left(k\right)}\left(t_i\right)-{\mathbf{f}}^{\left(k\right)}\left(t_i\right)\right)}^2, k=1,\dots ,K;$$

with ${\widehat{\mathbf{f}}}^{\left(k\right)}$ estimate of ${\mathbf{f}}^{\left(k\right)}$, and by computing the relative number of (falsely) detected anomalies (FPR) as well as the relative number of not detected anomalies (FNR), namely

$$\begin{array}{ccc}\hfill \mathrm{FPR}& =& \frac{1}{K-A}{\displaystyle \sum _{i=1}^K}\mathbb{I}\left({∥{\widehat{\mathbf{a}}}^{\left(k\right)}∥}_2\ne 0\wedge {∥{\mathbf{a}}^{\left(k\right)}∥}_2=0\right)\hfill \\ \hfill \mathrm{FNR}& =& \frac{1}{A}{\displaystyle \sum _{i=1}^K}\mathbb{I}\left({∥{\widehat{\mathbf{a}}}^{\left(k\right)}∥}_2=0\wedge {∥{\mathbf{a}}^{\left(k\right)}∥}_2\ne 0\right),\hfill \end{array}$$

A being the true number of channel with anomalies. In addition, we evaluated the intensity of the estimated anomalies to compare with the intensity of the simulated ones and to analyze the detection capabilities of both our proposals as

$$\frac{{∥{\widehat{\mathbf{a}}}^{\left(k\right)}∥}_2^2}{n},\hspace{1em}k=1,\dots ,K.$$

To be robust to the particular realization in generating synthetic data (and corresponding noise), each experiment was run several times. In particular, we ran 10 instances for each of the 12 considered cases (2 (function settings) × 2 (levels of anomaly) × 3 (SNRs)). For the sake of exposition, in the following we show separately results obtained for each function’s settings. Specifically, Figure 4, Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9 and

Table 1. False positives rates FPR (%) and false negatives rates FNR (%) for the simulations carried out on HiSine signals, $h=0.5$. We report average values of the indicators over 10 simulations.

HiSine	$\mathit{SNR}=0.5$	$\mathit{SNR}=1$	$\mathit{SNR}=6$
${\mathrm{FPR}}_{\mathrm{hard}}$	0.05	0.05	0
${\mathrm{FNR}}_{\mathrm{hard}}$	0	0	0
${\mathrm{FPR}}_{\mathrm{soft}}$	0	0	0
${\mathrm{FNR}}_{\mathrm{soft}}$	1	1	1

and

Table 2. False positives rates FPR (%) and false negatives rates FNR (%) for the simulations carried out on HiSine signals, $h=2$. We report average values of the indicators over 10 simulations.

HiSine	$\mathit{SNR}=0.5$	$\mathit{SNR}=1$	$\mathit{SNR}=6$
${\mathrm{FPR}}_{\mathrm{hard}}$	0	0	0
${\mathrm{FNR}}_{\mathrm{hard}}$	0	0	0
${\mathrm{FPR}}_{\mathrm{soft}}$	0	0	0
${\mathrm{FNR}}_{\mathrm{soft}}$	0.1	0	0

show results of the 6 different combinations of levels of anomaly and SNRs obtained when the function is $\mathbf{f}=HiSine$ and Figure 10, Figure 11, Figure 12, Figure 13, Figure 14 and Figure 15 and

Table 3. False positives rates FPR (%) and false negatives rates FNR (%) for the simulations carried out on TwoChirp signals, $h=0.5$. We report average values of the indicators over 10 simulations.

TwoChirp	$\mathit{SNR}=0.5$	$\mathit{SNR}=1$	$\mathit{SNR}=6$
${\mathrm{FPR}}_{\mathrm{hard}}$	0	0	0
${\mathrm{FNR}}_{\mathrm{hard}}$	0	0	0
${\mathrm{FPR}}_{\mathrm{soft}}$	0	0	0
${\mathrm{FNR}}_{\mathrm{soft}}$	1	1	0.8

and

Table 4. False positives rates FPR (%) and false negatives rates FNR (%) for the simulations carried out on TwoChirp signals, $h=2$. We report average values of the indicators over 10 simulations.

TwoChirp	$\mathit{SNR}=0.5$	$\mathit{SNR}=1$	$\mathit{SNR}=6$
${\mathrm{FPR}}_{\mathrm{hard}}$	0	0	0
${\mathrm{FNR}}_{\mathrm{hard}}$	0	0	0
${\mathrm{FPR}}_{\mathrm{soft}}$	0	0	0
${\mathrm{FNR}}_{\mathrm{soft}}$	0	0	0

show analogous results when the function is $\mathbf{f}=TwoChirp$. Moreover, we show in Figure 5 and Figure 6 the plots of the shape of the unknown signals and the goodness of reconstructions by the Hard thresholding procedure in the two extreme cases, weak anomaly ($h=0.5$) with highest noise ($SNR=0.5$) and strong anomaly ($h=2$) with lowest noise level ($SNR=6$) for the first setting $\mathbf{f}=HiSine$; while in Figure 7 and Figure 8 we show the same results obtained by the Soft thresholding procedure. For the second setting $\mathbf{f}=TwoChirp$, four analogous plots are given in Figure 11 and Figure 12 and Figure 13 and Figure 14 for the Hard and Soft thresholding procedures respectively.

Comment on the results. Concerning anomaly detection performance, we observe that for both procedures, Hard and Soft, and both settings, $\mathbf{f}=HiSine$ and $\mathbf{f}=TwoChirp$, it is worst in the case of weak anomaly ($h=0.5$, Table 1 and Table 3) with respect to the case of strong anomaly ($h=2$, Table 2 and Table 4). This is a consequence of the masking effect: the weaker the anomaly the easier the noise, and the anomalies can mask each other. However, there is an interesting difference between the two procedures. In particular, for the Hard thresholding, we observe that $FNR$ is always zero, while there can be some $FPR>0$ in the more difficult setting, i.e., when the anomalies are weaker and the level of noise is higher. This means that the Hard thresholding procedure always detects the true anomalies, being prone to some false positives detection when stressing the problem setting. This observation is relevant from the practical point of view because in such extreme cases one can always adopt a post-processing rule to classify the detected anomalies, being confident enough not to miss any anomaly. On the contrary, the Soft thresholding is much more conservative, hence when stressing the setting, i.e., when the anomalies are weaker ($h=0.5$) and the level of noise higher, the Soft thresholding procedure is more prone to false negatives detection. Indeed, for the Soft we always get $FPR=0$, while stressing the setting we get some $FNR>0$ (the opposite of Hard). This is not a drawback, rather a different way of reacting to the difficulty of the problem. However, from a practical point of view, when using Soft instead of Hard it can be a disadvantage to have false-negative detection because when the result undergoes post-processing the anomalies not detected are not further analyzed. However, although our two procedures are not the definitive answer to the AD problem, it is important from a practical point of view to be aware that the Hard thresholding tends to select more anomalies (and therefore to have some false positives) while the Soft threshold tends to select fewer anomalies (and therefore to have some false negatives). In addition, Figure 9 and Figure 15 show the boxplots of the intensity of the simulated anomalies and of the estimated anomalies by both Hard and Soft thresholding. We can note the good performance of the Hard thresholding procedure with respect to the Soft thresholding one, both in the case of weak anomalies and strong anomalies, with a better intensities estimate in the last case.

As regards the estimation performance, we observe that for both procedures, Hard and Soft, and both settings, $\mathbf{f}=HiSine$, and $\mathbf{f}=TwoChirp$, it decreases when increasing the level of noise (i.e., decreasing the SNR). In fact, the $MSE$ significantly decreases when moving from the left to the right in all box-plot shown in Figure 4 for the first setting $\mathbf{f}=HiSine$ and in Figure 10 for the second setting $\mathbf{f}=TwoChirp$; panels (a)–(c) refer to Hard thresholding for $h=0.5$ and $h=2$, respectively, and panels (b) and (d) refer to Soft thresholding for $h=0.5$ and $h=2$, respectively. Interestingly, in line with what we observed on the detection performance results, we have significant differences in terms of MSE performance on the anomalous channels between the Hard and the Soft. See box-plots in positions 1, 4, 7, corresponding to the first channel for the $HiSine$ signal and box-plots in positions 1, 4, 6, 9, 11, 14, corresponding to the first and the fourth channels for the $TwoChirp$ signal. The anomalous channels always have a better MSE for the Hard than for the Soft. This is a direct consequence of our double-step procedure: when we correctly detect the anomaly (in the first step), we correctly adjust the residual (in the second step) and hence we obtain a better estimation. On the channels without anomaly, detection performance is comparable.

Finally, we stress that what we have observed on average (looking at box-plots) about estimation performance, can be inspected visually for a randomly chosen realization looking at the first channel in Figure 5 (Hard thresholding, $h=0.5$, $SNR=0.5$), Figure 6 (Hard thresholding, $h=2$, $SNR=6$), Figure 7 (Soft thresholding, $h=0.5$, $SNR=0.5$) and Figure 8 (Soft thresholding, $h=2$, $SNR=6$), which refer to $\mathbf{f}=HiSine$ and looking at the first and fourth channels in Figure 11 (Hard thresholding, $h=0.5$, $SNR=0.5$), Figure 12, (Hard thresholding, $h=2$, $SNR=6$), Figure 13 (Soft thresholding, $h=0.5$, $SNR=0.5$) and Figure 14 (Soft thresholding, $h=2$, $SNR=6$), which refer to $\mathbf{f}=TwoChirp$.

5.2. Real Data

To illustrate the performance of our procedures on a real example, we considered the problem of detecting anomalies for signals recording measurements on a water pump. The signals represent the behavior of the pump in both normal and abnormal conditions and a complete description of it is given in [29]. The dataset is part of a repository managed by the Delft University of Technology and is freely available at http://homepage.tudelft.nl/n9d04/occ/index.html (accessed on the 18 February 2021) (dataset 541: Delft pump AR app). Each signal has a length $n=160$ and has been obtained by first fitting an AutoRegressive model of order 32 (AR(32)) to each of 5 vibration measurements and then by combining the coefficients in a single vector.

This dataset has been previously analyzed in [30]. In that paper, the authors proposed a data domain description, named the Support Vector Data Description (SVDD) to find the finest representation of the data such that the normal target signals are optimally clusterized and can be distinguished as best as possible from the abnormal ones. Their approach has a good performance and in some sense, it is another perspective for looking at the detection of anomalies. However, the data representation proposed in [30] does not take into account the longitudinal shape of the data, which are ordered coefficients; hence the method proposed in [30] is expected to be invariant under permutation of the data. On the other hand, our analysis is truly functional, since we treat the data as functions/signals; moreover, the dataset satisfies the hypothesis of our model. Since the underlying signals have a fast oscillating characteristic, they share a similar shape which presupposes the same sparsity pattern into the RADWT dictionary and the signals measured under abnormal conditions can be modeled as the signals measured under normal conditions plus some shift in the first part of the interval.

We considered 20 signals, 12 nominal and 8 with the anomaly, randomly chosen from the whole dataset, see Figure 16. We applied both procedures, Hard and Soft thresholding, obtaining reconstruction shown in Figure 17, panels (a) and (b). In the same figure, panels (c) and (d), we also display the retrieved residuals which represent anomalies. While the Soft thresholding has no false positives neither false negatives, the Hard thresholding has 2 false positives (channel 10 and 11), with the norm of the order of the estimated variance 0.0035, see

Table 5. ${∥\widehat{\mathbf{a}}∥}^2/n$ for the Hard estimator.

Pump Data
${∥{\widehat{\mathbf{a}}}^{\left(\mathbf{1}\right)}∥}_2^2/n,\dots ,{∥{\widehat{\mathbf{a}}}^{\left(\mathbf{9}\right)}∥}_2^2/n$	0
${∥{\widehat{\mathbf{a}}}^{\left(\mathbf{10}\right)}∥}_2^2/n$	0.0037
${∥{\widehat{\mathbf{a}}}^{\left(\mathbf{11}\right)}∥}_2^2/n$	0.0036
${∥{\widehat{\mathbf{a}}}^{\left(\mathbf{12}\right)}∥}_2^2/n$	0
${∥{\widehat{\mathbf{a}}}^{\left(\mathbf{13}\right)}∥}_2^2/n$	0.0205
${∥{\widehat{\mathbf{a}}}^{\left(\mathbf{14}\right)}∥}_2^2/n$	0.0225
${∥{\widehat{\mathbf{a}}}^{\left(\mathbf{15}\right)}∥}_2^2/n$	0.0207
${∥{\widehat{\mathbf{a}}}^{\left(\mathbf{16}\right)}∥}_2^2/n$	0.0202
${∥{\widehat{\mathbf{a}}}^{\left(\mathbf{17}\right)}∥}_2^2/n$	0.0192
${∥{\widehat{\mathbf{a}}}^{\left(\mathbf{18}\right)}∥}_2^2/n$	0.0190
${∥{\widehat{\mathbf{a}}}^{\left(\mathbf{19}\right)}∥}_2^2/n$	0.0203
${∥{\widehat{\mathbf{a}}}^{\left(\mathbf{20}\right)}∥}_2^2/n$	0.0201

. However, the Hard thresholding estimates better the intensity of the anomalies (which represent residuals for the regression on $\mathit{\beta }$), permitting a better signals reconstruction. This does not happen for the Soft thresholding, which results in over-smoothed signal reconstruction. Finally, we can conclude that the Hard thresholding procedure has a better performance although it has two false positives, which could be easily detected by standard post-processing.

6. Conclusions

In this paper, we presented two procedures for anomaly detection in multichannel signals under a structural hypothesis on the underlying signals covering some specific real-life situations. In particular, we dealt with fast oscillating signals under the assumption that they share a common specific sparsity pattern in a given dictionary. The construction of the dictionary leverages on a complete filter bank (RADWT) of $L_2\left(R\right)$ which guarantees a perfect reconstruction property and a tunable Q-factor. We based the two procedures on group penalized regression methods and we implemented them by a two-step iterative algorithm. The two methods differ in the second step iteration, the first applying Soft thresholding to the residual vectors, the second one applying Hard thresholding. Moreover, we made connections between them and the robust regression literature in a high dimensional setting obtaining interesting interpretations. From the computational point of view, we observed that the Hard thresholding procedure reveals some advantages to the Soft thresholding procedure, confirming some previous results on nonconvex robust regression.