Aiding Dictionary Learning Through Multi-Parametric Sparse Representation

Abstract

The ${\ell }_1$ relaxations of the sparse and cosparse representation problems which appear in the dictionary learning procedure are usually solved repeatedly (varying only the parameter vector), thus making them well-suited to a multi-parametric interpretation. The associated constrained optimization problems differ only through an affine term from one iteration to the next (i.e., the problem’s structure remains the same while only the current vector, which is to be (co)sparsely represented, changes). We exploit this fact by providing an explicit, piecewise affine with a polyhedral support, representation of the solution. Consequently, at runtime, the optimal solution (the (co)sparse representation) is obtained through a simple enumeration throughout the non-overlapping regions of the polyhedral partition and the application of an affine law. We show that, for a suitably large number of parameter instances, the explicit approach outperforms the classical implementation.

1. Introduction

We commonly represent m-dimensional data, or signals, through a collection of m linearly independent vectors that together form a base. For certain m-dimensional signals, or classes of signals, we have prior information or insight regarding that they are in fact perturbed versions of a much simpler, original, true signal that lies in an s-dimensional sub-space. In order to be able to retrieve the original signal, we need to have access to a proper base, containing the right s vectors that were used when the true signal was created. With such limited information, it is hard to pick such a base, especially for a large class of signals, which is why it is common to extend it with more vectors, making it a redundant base, also called a frame or, as will be used through out this paper, a dictionary. The dictionary has $n>m$ columns that are also called atoms. The idea is to have more options from which to choose s vectors.

Assume we are given a good dictionary $\mathit{D}\in {\mathbb{R}}^{m\times n}$ that we want to use to represent a signal $\mathit{y}\in {\mathbb{R}}^m$ with just s atoms. This leads to $\left(\genfrac{}{}{0pt}{}{s}{n}\right)$ possible choices. As the number of atoms increases, so does the complexity of finding the best choice. Even when we decide on a set of atoms, there is still the question of their associated coefficients in the direction they are pointing towards.

The task of finding the atoms and their coefficients is called the sparse representation problem [1]. Existing algorithms pursue this task for each signal $\mathit{y}$ that they receive, by solving a particular optimization problem based on $\mathit{y}$ that provides the s-sparse representation $\mathit{x}$.

Solving the sparse representation problem for multiple vectors $\mathit{y}$ is in fact the repeated resolution of a constrained optimization problem which differs only through the m-dimensional vector $\mathit{y}$ which is to be represented. This has led us to consider a multi-parametric interpretation where the vector $\mathit{y}$ becomes a parameter which governs the optimization’s problem solution and its domain of validity.

While the multi-parametric approach is well-known in the optimization field [2], to the best of our knowledge, it appears not to have been employed by the dictionary learning (DL) community.

Conceptually, the idea is simple: instead of solving repeatedly very similar optimization problems (with the same structure and differing only in the current value of a parameter), we instead break the problem into two distinct stages:

(i)

The offline stage: find the way in which the optimal solution depends on parameter $\mathit{y}$ and store this information for later use in the online stage;

(ii)

The online stage: for the current value of $\mathit{y}$, retrieve and apply the a priori computed solution.

Further assume that the cost is quadratic and the constraints linear leads, in the offline stage, to a piecewise formulation with a polyhedral support [3,4]. That is, the optimal solution is an affine law in parameter $\mathit{y}$ which is active on a so-called polyhedral critical region. Enumerating [5] all these disjoint critical regions covers completely the parameter space (or the restriction over which $\mathit{y}$ is defined). Thus, at the online stage, it remains only to find the region in which the current $\mathit{y}$ lies [6,7,8], retrieve the associated affine law and apply it. For further details with a practical bent, see the survey [9] which deals with the subtleties of multi-parametric problem in the context of explicit MPC.

As a first step, we rewrite the sparse representation problem in its dual form (Karash–Kuhn–Tucker) in order to enumerate sets of active constraints from which the piecewise affine law and its polyhedral support are deduced. A couple of issues differentiate this approach from other formulations encountered in the literature (e.g., [8,10] which solve a generic complementarity problem):

(i)

The cost from the constrained optimization problem is rank-deficient (the dictionary is over-determined which means that the quadratic cost is defined by a semi-definite matrix);

(ii)

The ${\ell }_1$ norm used here (as a relaxation from the sparse restriction induced by the ${\ell }_0$ norm) leads to a very-particular set of linear constraints (they define a cross-polytope, whose structure influences the optimization problem formulation [11]).

Tackling these issues leads to a particular multi-parametric formulation and represents the main contribution of this paper (beyond the usage of the multi-parametric formalism in the DL context):

(i)

The first issue requires a careful decomposition of the matrices (in order to avoid degeneracy in the formulations [12]); the upshot is that the particular KKT representation appears in a simple form (which allows simple matrix decompositions).

(ii)

The second requires to consider a vertex-representation of the cross-polytope (as this is a more compact representation than its equivalent half-space representation [13]). In general, the opposite is true: to a reasonable number of linear inequalities corresponds a significantly larger number of vertices. Thus, most if not all of space partitioning induced by the multi-parametric representation exploit the “half-space” description of the feasible domain [9].

(iii)

We highlight a compact storage and retrieval procedure (which exploits the symmetry of the cross-polytope domain) with the potential to significantly reduce the numerical issues.

For a better understanding of the issues, we also consider some of the specific shortcomings of multi-parametric representations: the over-approximation of quadratic constraints (as appear whenever the ${\ell }_2$ norm is used as a constraint; the enumeration problem when dealing with large-size parameters and some ideas for alleviating the issue [14].

The rest of the paper is organized as follows. Section 2 presents the sparse and cosparse representations. Section 3 gives a multi-parametric solution for a generic optimization problem which is then particularized for the (co)sparse representation case in Section 4. The multi-parametric approach is tested over a standard example and comparison with the standard case is shown in Section 5. The conclusions are drawn in Section 6.

2. The Dictionary Learning Problem

Given the signal $\mathit{y}\in {\mathbb{R}}^m$ and the fixed dictionary $\mathit{D}\in {\mathbb{R}}^{m\times n}$, the sparse representation problem imposes an error bound $\epsilon$ and seeks the sparsest representation

$$\begin{array}{cccc}\underset{\mathit{x}}{min}\hfill & {\parallel \mathit{x}\parallel }_0\hfill & \hfill \mathrm{s}.\mathrm{t}.& \hfill \parallel \mathit{y}-\mathit{D}\mathit{x}\parallel \le \epsilon ,\end{array}$$

(1)

where ${∥.∥}_0$ is the ${\ell }_0$ pseudo-norm counting the non-zero entries of $\mathit{x}$. The process is depicted in Figure 1a, where the full signal $\mathit{y}$ uses $s=3$ dictionary atoms for its sparse representation $\mathit{x}$.

Due to the non-convex nature of the ${\ell }_0$-norm, we prefer its convex ${\ell }_1$ relaxation

$$\begin{array}{cccc}\underset{\mathit{x}}{min}\hfill & {\parallel \mathit{y}-\mathit{D}\mathit{x}\parallel }^2\hfill & \hfill \mathrm{s}.\mathrm{t}.& \hfill {\parallel \mathit{x}\parallel }_1\le \theta ,\end{array}$$

(2)

which is known as the lasso problem [16]. Here, $\theta$ is a sparsity promoting bound for the 1-norm of the solution. The focus in (2) is on the approximation, such that the objective is minimized up to a point where $\mathit{x}$ has too many large non-zero elements. If instead we desire the sparsest ${\ell }_1$ solution satisfying a given approximation quality $\epsilon$, the optimization problem becomes

$$\begin{array}{cccc}\underset{\mathit{x}}{min}\hfill & {\parallel \mathit{x}\parallel }_1\hfill & \hfill \mathrm{s}.\mathrm{t}.& \hfill {\parallel \mathit{y}-\mathit{D}\mathit{x}\parallel }^2\le \epsilon .\end{array}$$

(3)

Given a set of $\mathit{Y}\in {\mathbb{R}}^{m\times N}$ training signals, the dictionary learning (DL) problem [15] is

$$\begin{array}{cccc}\underset{\mathit{D},\mathit{X}}{min}\hfill & {∥\mathit{Y}-\mathit{D}\mathit{X}∥}_F^2\hfill & \hfill \mathrm{s}.\mathrm{t}.& \hfill {∥{\mathit{x}}_{\ell }∥}_1\le \theta ,\ell =1:N,∥{\mathit{d}}_j∥=1,j=1:n,\end{array}$$

(4)

where $\mathit{D}\in {\mathbb{R}}^{m\times n}$ is the dictionary, whose columns are also called atoms, and $\mathit{X}\in {\mathbb{R}}^{n\times N}$ are the resulting sparse representations. Note that (4), in contrast to (1), does not only find the sparse representation $\mathit{X}$ but also learns the dictionary $\mathit{D}$.

As depicted in Algorithm 1, problem (4) is usually solved by alternative optimization: fixing $\mathit{D}$ we enter the sparse representation stage (step 2) where we compute the representations $\mathit{X}$ by solving either of (1)–(3) for each signal $\mathit{y}\in \mathit{Y}$, and then we enter the dictionary update stage (step 3) where we fix $\mathit{X}$ and update $\mathit{D}$ by taking each atom $\mathit{d}\in \mathit{D}$ and refining it based on the signals that use it in their sparse representation. The process is repeated K times (step 1) until the objective in (4) converges or the error becomes satisfactory.

Algorithm 1: Alternate optimization dictionary learning.

The Cosparse Model

While the sparse representation is interested in finding the non-zero coefficients corresponding to a low-dimensional subspace where signal $\mathit{x}$ lies, the cosparse representation is interested in finding its complementary subspace, the subspace orthogonal to $\mathit{x}$. The sparse model can be seen as a synthesis process, focused on constructing $\mathit{x}$ based on the dictionary, and the cosparse model as its analysis counter-part, where the focus is on the original signal $\mathit{y}$ and its null subspace.

As depicted in Figure 1b, in the cosparse model [17], we apply the analysis dictionary $\mathrm{\Omega }\in {\mathbb{R}}^{n\times m}$ to the signal $\mathit{y}$ and obtain the sparse vector

$$\mathbf{\gamma }=\mathrm{\Omega }\mathit{y}.$$

(5)

The atoms are now the rows of dictionary $\mathrm{\Omega }$, of which only a few are used in the multiplication with $\mathit{y}$. Unlike the sparse model, here we are interested in the atoms that are not used by $\mathit{y}$, the atoms that are orthogonal to it and create the zero entries in $\mathit{\gamma }$.

Assume that signal $\mathit{y}$ is the noisy version of the original, unknown, signal $\mathit{z}\in {\mathbb{R}}^m$ that lies in a lower-dimensional subspace

$$\mathit{y}=\mathit{z}+\mathit{v},$$

(6)

where $\mathit{v}$ is white additive noise. Let $\mathrm{\Omega }\in {\mathbb{R}}^{n\times m}$ be the given cosparse dictionary. Then, recovering $\mathit{z}$ means finding the low-dimensional subspace to which it belongs and implies finding the set of atoms $\mathrm{\Lambda }$ from $\mathrm{\Omega }$ orthogonal to $\mathit{z}$. In the literature, $\mathrm{\Lambda }$ is sometimes called the cosupport of $\mathit{z}$.

The ${\ell }_0$ optimization problem becomes

$$\begin{array}{cccc}\underset{\mathit{z},\mathrm{\Lambda }}{min}\hfill & {\parallel \mathit{y}-\mathit{z}\parallel }^2\hfill & \hfill \mathrm{s}.\mathrm{t}.& \hfill {\mathrm{\Omega }}_{\mathrm{\Lambda }}\mathit{z}=0,\mathrm{rank}\left({\mathrm{\Omega }}_{\mathrm{\Lambda }}\right)=m-s,\end{array}$$

(7)

where we are interested in minimizing the distance between $\mathit{y}$ and $\mathit{z}$ such that the cosupport defines an orthogonal $m-s$ subspace in which $\mathit{z}$ lies. Thus, applying the cosparse dictionary leads to an s-sparse signal as in Figure 1b.

For our purposes, we relax the equality constraints of (7) and move to its convex form

$$\begin{array}{cccc}\underset{\mathit{z},\mathrm{\Lambda }}{min}\hfill & {\parallel \mathit{y}-\mathit{z}\parallel }^2\hfill & \hfill \mathrm{s}.\mathrm{t}.& \hfill {∥{\mathrm{\Omega }}_{\mathrm{\Lambda }}\mathit{z}∥}_1\le \theta ,\mathrm{rank}\left({\mathrm{\Omega }}_{\mathrm{\Lambda }}\right)\le m-s,\end{array}$$

(8)

where we induce sparsity using the ${\ell }_1$-norm just like we did in (2). Focusing on sparsity instead of approximation, we can rewrite (8) as

$$\begin{array}{cccc}\underset{\mathit{z},\mathrm{\Lambda }}{min}\hfill & {∥{\mathrm{\Omega }}_{\mathrm{\Lambda }}\mathit{z}∥}_1\hfill & \hfill \mathrm{s}.\mathrm{t}.& \hfill {\parallel \mathit{y}-\mathit{z}\parallel }^2\le \epsilon ,\mathrm{rank}\left({\mathrm{\Omega }}_{\mathrm{\Lambda }}\right)\le m-s.\end{array}$$

(9)

3. Analysis of the Multi-Parametric Formulation Induced by the DL Problem

Let us consider the constrained optimization problem

$$\begin{array}{cccc}\underset{\mathit{\xi }}{min}\hfill & {\parallel \mathit{T}\mathit{\xi }-\mathit{y}\parallel }_2^2\hfill & \hfill \mathrm{s}.\mathrm{t}.& \hfill {\parallel \mathbf{\Delta }\mathit{\xi }\parallel }_1\le \delta ,\end{array}$$

(10)

with $\mathit{\xi }\in {\mathbb{R}}^n$, $\mathit{y}\in {\mathbb{R}}^m$, $\mathit{T}\in {\mathbb{R}}^{m\times n}$, $\mathbf{\Delta }\in {\mathbb{R}}^{p\times n}$. $\delta \in {\mathbb{R}}_{+}$ is a positive scalar.

Note that we make no assumption on the relation between space dimensions m, n, p. In fact, we will be mostly interested in the degenerate case $n>m$, which implies that mapping $\mathit{T}\mathit{\xi }$ is a projection from a higher dimensional space to a lower one (from ${\mathbb{R}}^n$ to ${\mathbb{R}}^m$).

The goal of (10) is to provide the constrained optimal solution ${\mathit{\xi }}^{\star }$ minimizing the cost while simultaneously respecting the constraints. While multiple [2] tools and theoretical algorithms can be employed to solve (10), in here we propose to exploit two of its particularities:

(i)

The ${\ell }_1$ norm constraint characterizes a scaled and/or projected cross-polytope;

(ii)

The cost is affinely parametrized after parameter $\mathit{y}$.

3.1. Geometrical Interpretation of the ${\ell }_1$ Norm

The first item stems from a well-known result from polyhedral sets’ state of the art [18]: the norm inequality ${\parallel \mathit{\xi }\parallel }_1\le 1$ describes the n-th order cross-polytope (the dual of a hypercube):

$$C_n=conv\{\pm {\mathit{e}}_i\}=\{\mathit{\xi }\in {\mathbb{R}}^n{:\parallel \mathit{\xi }\parallel }_1\le 1\},$$

(11)

where ${\mathit{e}}_i$ are the column vectors that have ‘1’ on the i-th position and ‘0’ anywhere else. Consequently, it follows that ${\parallel \mathbf{\Delta }\mathit{\xi }\parallel }_1\le \delta$ is equivalent with

$$\{\mathit{\xi }\in {\mathbb{R}}^n{:\parallel \mathbf{\Delta }\mathit{\xi }\parallel }_1\le \delta \}=conv\{\pm {\mathbf{\Delta }}_i^{+}\}·\delta ,$$

(12)

with ${\mathbf{\Delta }}_i^{+}\in {\mathbb{R}}^n$, the column vectors of matrix ${\mathbf{\Delta }}^{+}$, the pseudo-inverse of matrix $\mathbf{\Delta }$. Through (11), we have that $\mathbf{\Delta }\mathit{\xi }=conv\{\pm {\mathit{e}}_i\}$. Taking ${\mathbf{\Delta }}^{+}$ which verifies ${\mathbf{\Delta }}^{+}\mathbf{\Delta }=\mathit{I}$ allows for writing relation (12). Writing explicitly the convex sum from (12), we arrive at

$$\{\mathit{\xi }\in {\mathbb{R}}^n{:\parallel \mathbf{\Delta }\mathit{\xi }\parallel }_1\le \delta \}=\{\mathbf{\xi }:\mathit{\xi }=\mathit{F}\mathit{\alpha },\mathit{\alpha }\ge 0,{\mathbf{1}}^{\top }\mathit{\alpha }=\delta \}.$$

(13)

Note that we made the notation (for compactness reasons) $\mathit{F}=\left[\begin{array}{cc}{\mathbf{\Delta }}^{+}& -{\mathbf{\Delta }}^{+}\end{array}\right]$ and that $\mathit{\alpha }\in {\mathbb{R}}^{2n}$. Any permutation of column vectors $\pm {\mathbf{\Delta }}_i^{+}$ in $\mathit{F}$ is equally valid. We have simply chosen this form as it permits for compactly writing $\mathit{F}$ with respect to $\mathbf{\Delta }$. With these notations, we rewrite (10) into the form:

$$\begin{array}{cc}\underset{\mathit{\alpha }}{min}\hfill & {\parallel \mathit{T}\mathit{F}\mathit{\alpha }-\mathit{y}\parallel }_2^2,\hfill \\ \mathrm{s}.\mathrm{t}.\hfill & \mathit{\alpha }\ge 0,{\mathbf{1}}^{\top }\mathit{\alpha }=\delta .\hfill \end{array}$$

(14)

Note that the optimal solution of (14) is ${\mathit{\alpha }}^{\star }$ and, thus, ${\mathit{\xi }}^{\star }={\mathit{F}\mathit{\alpha }}^{\star }$.

Remark 1.

Any polyhedral set has a dual representation: half-space (as a finite intersection of linear inequalities) and convex (as a convex sum of extreme points and linear combination of rays—in which case it is not bounded). Most often, the half-space representation is preferred since it is more compact (there are usually many fewer constraints than extreme points). On the other hand, the cross-polytope is the go-to counter-example: there are $2n$ extreme points in (11) but to these correspond $2^n$ inequalities in the half-space representation. It is thus significantly more efficient to exploit the convex representation in this particular case.

3.2. Karush–Kuhn–Tucker Form

The second particularity of (10) (or, equivalently, of (14)) is that the optimization depends on parameter $\mathit{y}$. Inspired by the approach followed in explicit MPC constructions [9] (and which, to the best of our knowledge, has no equivalence in the dictionary learning state of the art), we propose to provide an explicit description of the optimal solution in terms of parameter $\mathit{y}$.

The first step is to rewrite (14) in its dual form (i.e., the Karash–Kuhn–Tucker representation—see [2] or any standard optimization book for a more in-depth description):

$$\begin{array}{cc}\hfill \mathrm{stationarity}:& \nabla \left({\parallel \mathit{y}-\mathit{T}\mathit{F}\mathit{\alpha }\parallel }_2^2\right)+\nabla \left(-\mathit{\alpha }\right)·\mathit{\lambda }+\nabla \left({\mathbf{1}}^{\top }\mathit{\alpha }\right)·\mu =0,\hfill \end{array}$$

(15a)

$$\begin{array}{cc}\hfill \mathrm{primal}\mathrm{feasibility}:& -\mathit{\alpha }\le 0,{\mathbf{1}}^{\top }\mathit{\alpha }=\delta ,\hfill \end{array}$$

(15b)

$$\begin{array}{cc}\hfill \mathrm{dual}\mathrm{feasibility}:& \mathit{\lambda }\ge 0,\hfill \end{array}$$

(15c)

$$\begin{array}{cc}\hfill \mathrm{complementarity}:& \mathit{\lambda }\times (-\mathit{\alpha })=0.\hfill \end{array}$$

(15d)

The ∇ operator denotes the gradient operation applied for: the cost, the inequalities and the equality appearing in the primal problem (14). $\mathbf{\lambda }\in {\mathbb{R}}^{2n}$ and $\mu \in \mathbb{R}$ are the Lagrangian multipliers (also called dual variables) associated with the inequalities, respectively the equality constraint of (14). ‘×’ denotes the complementarity condition which requires that either the i-th Lagrangian multiplier ${\lambda }_i$ is zero or the i-th inequality is active ($-{\alpha }_i\le 0\mapsto -{\alpha }_i=0$).

Applying the gradient, the KKT form (15) becomes

$$\begin{array}{cc}\hfill 2{\left(\mathit{TF}\right)}^{\top }·\left(\mathit{T}\mathit{F}\mathit{\alpha }-\mathit{y}\right)-\mathit{\lambda }+\mathbf{1}\mu & =0,\hfill \end{array}$$

(16a)

$$\begin{array}{cc}\hfill \mathit{\lambda }\ge 0,-\mathit{\alpha }& \le 0,{\mathbf{1}}^{\top }\mathit{\alpha }=\delta ,\hfill \end{array}$$

(16b)

$$\mathit{\lambda }\times (-\mathit{\alpha })=0.$$

(16c)

Under Slater’s conditions of optimality [2], the primal variable $\mathbf{\alpha }$ and dual variables $\mathit{\lambda },\mu$ verifying (16) describe the optimum (primal and dual).

Using the well-known result that $rank\left(AB\right)\le min(rank(A)rank(B\left)\right)$ and that for a matrix $A\in {\mathbb{R}}^{r\times q}$, $rank\left(A\right)\le min(r,q),$ we have that $rank\left(\mathit{T}\mathit{F}\right)\le min\left(min(m,n),min(n,2n)\right)=min(m,n)$. Thus, the semi-definite matrix ${\left(\mathit{T},\mathit{F}\right)}^{\top }\mathit{T}\mathit{F}$ appearing in (16a) is rank-deficient (is a square matrix of dimension $2n$ but its rank is at most $min(m,n)$). Hereinafter, we consider (the opposite case would unfold in the same manner and is not followed here) that $m\le n$ and proceed accordingly (i.e., $min(m,n)=m$).

As a next step, we consider the SVD decomposition $\mathit{T}\mathbf{\Delta }=\mathit{U}\mathbf{\Sigma }{\mathit{W}}^{\top }$ with $\mathit{U}\in {\mathbb{R}}^{m\times m}$ and $\mathit{V}\in {\mathbb{R}}^{n\times n}$ orthogonal matrices and $\mathbf{\Sigma }\in {\mathbb{R}}^{m\times n}$ the matrix containing the singular values of $\mathit{T}\mathbf{\Delta }$. This decomposition allows for writing $\mathit{T}\mathit{F}=\mathit{U}\mathbf{\Sigma }{\mathit{V}}^{\top }$, where ${\mathit{V}}^{\top }=\left[\begin{array}{cc}{\mathit{W}}^{\top }& -{\mathit{W}}^{\top }\end{array}\right]\in {\mathbb{R}}^{n\times 2n}$. For further use, we consider the decompositions $\mathbf{\Sigma }=\left[\begin{array}{cc}{\mathbf{\Sigma }}_{\mathbf{1}}& \mathbf{0}\end{array}\right]$ and ${\mathit{V}}^{\top }={\left[\begin{array}{cc}{\mathit{V}}_1& {\mathit{V}}_2\end{array}\right]}^{\top }$, where ${\mathbf{\Sigma }}_{\mathbf{1}}\in {\mathbb{R}}^{m\times m}$ and ${\mathit{V}}_1^{\top }\in {\mathbb{R}}^{m\times 2n}$, ${\mathit{V}}_2^{\top }\in {\mathbb{R}}^{(n-m)\times 2n}$. Introducing these notations in (16) allows for rewriting (16a) as follows:

$$\begin{array}{cc}\hfill {\mathit{V}}_1^{\top }\mathit{\alpha }-{\left({\mathbf{\Sigma }}_{\mathbf{1}}^{-1}\right)}^{\top }{\mathit{U}}^{\top }\mathit{y}-\frac{1}{2}{\left({\mathbf{\Sigma }}_{\mathbf{1}}^{-2}\right)}^{\top }{\mathit{V}}_1^{\top }\left(\mathit{\lambda }-\mathbf{1}\mu \right)& =0,\hfill \end{array}$$

(17a)

$$\begin{array}{cc}\hfill {\mathit{V}}_2^{\top }\left(\mathit{\lambda }-\mathbf{1}\mu \right)& =0.\hfill \end{array}$$

(17b)

Remark 2.

Note that we assume that ${\mathbf{\Sigma }}_{\mathbf{1}}$ is full-rank, which, coupled with its square form, implies that it is non-singular (and consequently admits an inverse). Particular cases where ${\mathbf{\Sigma }}_{\mathbf{1}}$ is singular can be treated similarly, by extracting the non-singular part and resizing accordingly ${\mathit{V}}_1^{\top },{\mathit{V}}_2^{\top }$.

3.3. Multi-Parametric Interpretation

The difficulty of solving (16), even when making use of (17), lies in the complementarity condition (16c). There are multiple formulations in the literature [4,8,10], each proposing a particular ‘flavor’. In what follows, we make use of one of the most common approaches: the active set method [6]. This approach enumerates (the selection of sets of active constraints and the stop criteria for the search of feasible combinations are subject to significant research and still an open topic [14]) combinations of active (those for which ${\lambda }_i>0$) and inactive constraints (those for which ${\lambda }_i=0$). Subsequently, problem (16) is solved for each of the particular combinations of active constraints.

As a first step, we form submatrices associated with the active/inactive set of constraints ($\widetilde{(·)}$ for those corresponding to active constraints and $\widehat{(·)}$ for those corresponding to inactive constraints (for example, ${\tilde{\mathit{V}}}_2^{\top }$ is the matrix constructed from the columns of ${\mathit{V}}_2^{\top }$ which correspond to active constraints—those for which ${\tilde{\alpha }}_i=0$) and reformulate (16) correspondingly:

$$\begin{array}{cc}\hfill \left[\begin{array}{c}{\widehat{\mathit{V}}}_1^{\top }\\ {\mathbf{1}}^{\top }\end{array}\right]\widehat{\mathit{\alpha }}-\left[\begin{array}{c}{\left({\mathbf{\Sigma }}_1^{-1}\right)}^{\top }{\mathit{U}}^{\top }\mathit{y}+\frac{1}{2}{\left({\mathbf{\Sigma }}_1^{-2}\right)}^{\top }\left({\tilde{\mathit{V}}}_1^{\top }\tilde{\mathit{\lambda }}-{\mathit{V}}_1^{\top }\mathbf{1}\mu \right)\\ \delta \end{array}\right]& =0,\hfill \end{array}$$

(18a)

$$\begin{array}{cc}\hfill {\tilde{\mathit{V}}}_2^{\top }\tilde{\mathit{\lambda }}-{\mathit{V}}_2^{\top }\mathbf{1}\mu & =0,\hfill \end{array}$$

(18b)

$$\begin{array}{cc}\hfill \tilde{\mathit{\lambda }}>0,\tilde{\mathit{\alpha }}& =0,\hfill \end{array}$$

(18c)

$$\begin{array}{cc}\hfill \widehat{\mathit{\lambda }}=0,\widehat{\mathit{\alpha }}& >0.\hfill \end{array}$$

(18d)

We note with $\tilde{m},\widehat{m}$ the number of active, respectively inactive, constraints (hence $\tilde{m}+\widehat{m}=n$).

At this stage, most multi-parametric formulations employ various heuristics for the selection of the set of active constraints [5,8] and solve (18). We proceed similarly with the observation that the cardinality of the set of active constraints has to lead to a well-defined problem (18). In other words, we expect that $\widehat{m}\le m+1$ (i.e., the number of rows in the matrix left-multiplying $\widehat{\mathit{\alpha }}$ in (18a) is at least equal with the number of columns, which means that the matrix admits a full-rank pseudo-inverse).

Having to accommodate the case $\widehat{m}<m+1$ requires a further partitioning of the matrices appearing in (18), this time in row-blocks: we add the subscript ‘1’ for the sub-matrices formed from the first $\widehat{m}-1$ rows and subscript ‘2’ for the sub-matrices formed from the remaining $m-\widehat{m}+1$ rows. Note that, as before, the partitioning is not unique and any permutation which does not change the number of rows is admissible. For example, ${\widehat{\mathit{V}}}_1^{\top }\in {\mathbb{R}}^{m\times \widehat{m}}$ is partitioned into ${\widehat{\mathit{V}}}_1^{\top }={\left[\begin{array}{cc}{\widehat{\mathit{V}}}_{11}& {\widehat{\mathit{V}}}_{12}\end{array}\right]}^{\top }$ where ${\widehat{\mathit{V}}}_{11}^{\top }\in {\mathbb{R}}^{(\widehat{m}-1)\times \widehat{m}}$ and ${\widehat{\mathit{V}}}_{12}^{\top }\in {\mathbb{R}}^{(m-\widehat{m}+1)\times \widehat{m}}$. With these notations, (18a)–(18b) become:

$$\begin{array}{cc}\hfill \left[\begin{array}{c}{\widehat{\mathit{V}}}_{11}^{\top }\\ {\mathbf{1}}^{\top }\end{array}\right]\widehat{\mathit{\alpha }}-\left[\begin{array}{c}{\left({\mathbf{\Sigma }}_{11}^{-1}\right)}^{\top }{\mathit{U}}^{\top }\mathit{y}+\frac{1}{2}{\left({\mathbf{\Sigma }}_{11}^{-2}\right)}^{\top }\left({\tilde{\mathit{V}}}_1^{\top }\tilde{\mathit{\lambda }}-{\mathit{V}}_1^{\top }\mathbf{1}\mu \right)\\ \delta \end{array}\right]& =0,\hfill \end{array}$$

(19a)

$$\begin{array}{cc}\hfill {\widehat{\mathit{V}}}_{12}^{\top }\widehat{\mathit{\alpha }}-\left[{\left({\mathbf{\Sigma }}_{12}^{-1}\right)}^{\top }{\mathit{U}}^{\top }\mathit{y}+\frac{1}{2}{\left({\mathbf{\Sigma }}_{12}^{-2}\right)}^{\top }\left({\tilde{\mathit{V}}}_1^{\top }\tilde{\mathit{\lambda }}-{\mathit{V}}_1^{\top }\mathbf{1}\mu \right)\right]& =0,\hfill \end{array}$$

(19b)

$$\begin{array}{cc}\hfill {\tilde{\mathit{V}}}_2^{\top }\tilde{\mathit{\lambda }}-{\mathit{V}}_2^{\top }\mathbf{1}\mu & =0.\hfill \end{array}$$

(19c)

Rearranging (19) to highlight the unknown terms ($\widehat{\mathit{\alpha }}$, $\tilde{\mathit{\lambda }}$ and $\mu$) leads to the following compact form:

$$\begin{array}{c}\left[\begin{array}{c}{\widehat{\mathit{V}}}_{11}^{\top }\\ {\mathbf{1}}^{\top }\end{array}\right]\widehat{\mathit{\alpha }}=\left[\begin{array}{c}{\left({\mathbf{\Sigma }}_{11}^{-1}\right)}^{\top }{\mathit{U}}^{\top }\\ 0\end{array}\right]\mathit{y}+\left[\begin{array}{c}0\\ \delta \end{array}\right]+\left[\begin{array}{c}{\left(\frac{1}{2}{\mathbf{\Sigma }}_{11}^{-2}\right)}^{\top }\\ 0\end{array}\right]\left[\begin{array}{cc}{\tilde{\mathit{V}}}_1^{\top }& -{\mathit{V}}_1^{\top }\mathbf{1}\end{array}\right]\left[\begin{array}{c}\tilde{\mathit{\lambda }}\\ \mu \end{array}\right],\hfill \end{array}$$

(20a)

$$\begin{array}{c}\left[\begin{array}{cc}\frac{1}{2}{\left({\mathbf{\Sigma }}_{12}^{-2}\right)}^{\top }{\tilde{\mathit{V}}}_1^{\top }& -\frac{1}{2}{\left({\mathbf{\Sigma }}_{12}^{-2}\right)}^{\top }{\mathit{V}}_1^{\top }\mathbf{1}\\ {\tilde{\mathit{V}}}_2^{\top }& -{\mathit{V}}_2^{\top }\mathbf{1}\end{array}\right]\left[\begin{array}{c}\tilde{\mathit{\lambda }}\\ \mu \end{array}\right]=\left[\begin{array}{c}{\widehat{\mathit{V}}}_{12}^{\top }\\ 0\end{array}\right]\widehat{\mathit{\alpha }}-\left[\begin{array}{c}{\left({\mathbf{\Sigma }}_{12}^{-1}\right)}^{\top }{\mathit{U}}^{\top }\\ 0\end{array}\right]\mathit{y}.\hfill \end{array}$$

(20b)

Note that the matrices appearing to the left of the equal sign in both (20a) and (20b) are full-row rank: the first is in ${\mathbb{R}}^{\widehat{m}\times \widehat{m}}$ and the second is in ${\mathbb{R}}^{(n-\widehat{m}+1)\times (2n-\widehat{m}+1)}$. Thus, we can proceed as follows:

(i)

We obtain ${\left[\begin{array}{cc}{\tilde{\mathit{\lambda }}}^{\top }& \mu \end{array}\right]}^{\top }$ from (20b) as a function of $\widehat{\mathbf{\alpha }}$ and $\mathit{y}$;

(ii)

We introduce it in (20a) and obtain $\widehat{\mathbf{\alpha }}$ as an affine form of $\mathit{y}$;

(iii)

We go back in (20b), replace the now-known $\widehat{\mathit{\alpha }}$ and obtain ${\left[\begin{array}{cc}{\tilde{\mathit{\lambda }}}^{\top }& \mu \end{array}\right]}^{\top }$ as an affine form of $\mathit{y}$.

For ease of notation, we assume that the steps mentioned above lead to

$$\widehat{\mathit{\alpha }}\left(\mathit{y}\right)={\widehat{\mathit{M}}}_{\alpha }\mathit{y}+{\widehat{\mathit{N}}}_{\alpha },\tilde{\mathit{\lambda }}\left(\mathit{y}\right)={\tilde{\mathit{M}}}_{\lambda }\mathit{y}+{\tilde{\mathit{N}}}_{\lambda },\mu \left(\mathit{y}\right)={\mathit{M}}_{\mu }\mathit{y}+{\mathit{N}}_{\mu }.$$

(21)

Combining with the fact that $\tilde{\mathit{\alpha }}=\mathbf{0},\widehat{\mathit{\lambda }}=\mathbf{0}$, as required in (18c)–(18d), allows for writing (by interlacing rows of zeroes in matrices ${\widehat{\mathit{M}}}_{\alpha }$, ${\widehat{\mathit{N}}}_{\alpha }$, ${\tilde{\mathit{M}}}_{\lambda }$, ${\tilde{\mathit{N}}}_{\lambda }$) the explicit solutions for $\mathit{\alpha },\mathit{\lambda },\mu$:

$$\mathit{\alpha }\left(\mathit{y}\right)={\mathit{M}}_{\alpha }\mathit{y}+{\mathit{N}}_{\alpha },\mathit{\lambda }\left(\mathit{y}\right)={\mathit{M}}_{\lambda }\mathit{y}+{\mathit{N}}_{\lambda },\mu \left(\mathit{y}\right)={\mathit{M}}_{\mu }\mathit{y}+{\mathit{N}}_{\mu }.$$

(22)

Recalling that $\tilde{\mathit{\lambda }}>0$ and $\widehat{\mathit{\alpha }}>0$, as required in (18c)–(18d), and using the notation from (21) gives the feasible domain over which the affine laws (21) describe the optimal solution:

$$\mathcal{R}=\{\mathit{y}:{\widehat{\mathit{M}}}_{\alpha }\mathit{y}+{\widehat{\mathit{N}}}_{\alpha }>0,{\tilde{\mathit{M}}}_{\lambda }\mathit{y}+{\tilde{\mathit{N}}}_{\lambda }>0\}=\left\{\mathit{y}:\left[\begin{array}{c}{\widehat{\mathit{M}}}_{\alpha }\\ {\tilde{\mathit{M}}}_{\lambda }\end{array}\right]\mathit{y}+\left[\begin{array}{c}{\widehat{\mathit{N}}}_{\alpha }\\ {\tilde{\mathit{N}}}_{\lambda }\end{array}\right]>0\right\}.$$

(23)

To recapitulate: starting from an initially chosen collection of active constraints, we have arrived at explicit formulations, parametrized affinely in $\mathit{y}$, for the primal solution $\mathit{\alpha }$, the dual solutions $\mathit{\lambda },\mu$ as in (22) and the region (23) for which these solutions are optimal. Note that (23) may turn to be infeasible (the empty set). In this case, it simply means that the associated collection of active constraints is not a feasible one and it should be discarded from the enumeration.

Enumerating all feasible combinations of active constraints will [9], thus, partition the parameter domain (the space ${\mathbb{R}}^m$ or some restriction of it in which $\mathit{y}$ lies) into a non-overlapping collection of regions (23), which is the support of a piecewise law affine in $\mathit{y}$, given by (22):

$$\mathit{\alpha }\left(\mathit{y}\right)={\mathit{M}}_{\alpha }^i\mathit{y}+{\mathit{N}}_{\alpha }^i,\forall \mathit{y}\in {\mathcal{R}}^i=\{\mathit{y}:{\widehat{\mathit{M}}}_{\alpha }^i\mathit{y}+{\widehat{\mathit{N}}}_{\alpha }^i>0,{\tilde{\mathit{M}}}_{\lambda }^i\mathit{y}+{\tilde{\mathit{N}}}_{\lambda }^i>0\}.$$

(24)

Index i appearing in (24) augments the shorthand ‘hat’ and ‘tilde’ notation used earlier for compactness reasons (that is, each time we had, e.g., ${\tilde{\mathit{V}}}_1$ we should had have written ${\tilde{\mathit{V}}}_1^i$ to highlight that we were considering the i-th combination of active constraints).

Remark 3.

With (24) computed offline, at runtime, we only need to identify the index i for which $\mathit{y}\in {\mathcal{R}}^i$ holds and apply the corresponding law $\mathit{\alpha }\left(\mathit{y}\right)$. For large dimensions, this may create storing and retrieval problems (as the number of regions ${\mathcal{R}}^i$ increases exponentially). Various solutions, with limited success, are proposed in the literature [4,7,14].

Remark 4.

Due to the continuity of the piecewise law $\mathit{\alpha }\left(y\right)$ and the fact that partition ${\cup }_i{\mathcal{R}}^i$ covers the entire domain, we do not require the analysis of cases where more than $m+1$ constraints are inactive [12].

3.4. Illustrative Example

For illustration purposes, we depict a proof-of-concept example. We take $\mathit{T}=\left[\begin{array}{ccc}1& 0& \sqrt{2}\\ 0& 1& \sqrt{2}\end{array}\right]$, $\mathbf{\Delta }={\mathit{I}}_3$, $\delta =1$, thus the the space dimensions are $m=2$, $n=3$, $p=3$. Having taken $(\mathsf{\Delta }={\mathit{I}}_3,\delta =1)$ means that ${\parallel \mathbf{\Delta }x\parallel }_1\le \delta$ describes the three-dimensional, cross-polytope $C_3=conv\left(\pm {\mathit{e}}_1,\pm {\mathit{e}}_2,\pm {\mathit{e}}_3\right)$, as shown (semi-transparent red object) in Figure 2.

The geometrical interpretation of solving (10) is that the unconstrained optimum (the one for which $\mathit{T}\mathit{\xi }-\mathit{y}=0$) is projected on the feasible domain, thus leading to the constrained optimum (i.e., the solution ${\mathit{\xi }}^{\star }$). This is done by inflating the sub-level cost sets (those for which the cost remains constant on their boundary) until there is an intersection with the feasible domain. Since, in the particular case of (10), the quadratic cost depends on a semi-definite matrix (${\mathit{T}}^{\top }\mathit{T}$ is not full-rank); this means that the optimum cost lies in the null subspace of matrix ${\mathit{T}}^{\top }T$ and the sub-level sets correspond to a degenerate ellipsoid (the blue-and-dotted cylinder shown in Figure 2). Note that, in (14) and its subsequent variations, we made a change of variable that increases the search space from ${\mathbb{R}}^3$ to ${\mathbb{R}}^6$, which is, of course, not representable in 3D.

Continuing with the geometric interpretation: the sub-level surface is inflated and touches the feasible domain at some point on its boundary. As long as this point is described by the same combination of active constraints (this means in Figure 2 that it is described by the same combination of vertices), we have that the solution $\alpha \left(\mathit{y}\right)$ is given as in (24) for a fixed index i. Moreover, we identify the region ${\mathcal{R}}^i$ in which $\mathbf{y}$ lies. For this numerical example, the 16 regions ${\mathcal{R}}^i$ obtained through the active set enumeration procedure are shown in Figure 3a and the associated cost value in Figure 3b.

We have not illustrated the actual solution ${\alpha }^{\star }\left(\mathit{y}\right)$ (due to its dimension). We can however note that, as expected, no more than $m=2$ vertices (corresponding to an edge in Figure 2) are active for any of the critical regions.

4. Integration of the Multi-Parametric Formulation in the Representation Problem

The sparse representation stage of (4) is a repeated computation of (2), parametrized after signals $\mathit{y}\in \mathit{Y}$. Thus, we may re-interpret (2) as a multi-parametric problem. This means that an explicit representation of the optimal solution can be obtained as a piecewise function, linear in parameter $\mathit{y}$: ${\mathit{x}}^{\star }\left(\mathit{y}\right)$. Thus, the offline part consists in computing the critical regions and their associated laws, whereas the online part is simply a lookup procedure which identifies and subsequently applies the law associated with the current value of parameter $\mathit{y}$.

In what follows, we proceed to show in Section 4.1 how Section 3 applies to the formulations of Section 2 and to highlight some of the remaining difficulties in Section 4.2 and Section 4.3.

4.1. Multi-Parametric Formulations for the Sparse and Cosparse Representations

In what follows, we particularize the results of Section 3 for the sparse and cosparse dictionary representations from Section 2.

For compactness, in Section 3, we have dealt with a somewhat generic constrained optimization problem (particularized in the fact that we start with cross-polytopic constraints and with a rank-deficient quadratic cost). This is justified by

Table 1. Equivalences of the sparse and cosparse representations w.r.t. form (14).

Sparse Representation (2)	Form (14)	Cosparse Representation (8)
$\mathit{x}\in {\mathbb{R}}^n$	$\mathit{\xi }$	$\mathit{z}\in {\mathbb{R}}^m$
$\mathit{y}\in {\mathbb{R}}^m$	$\mathit{y}$	$\mathit{y}\in {\mathbb{R}}^m$
$\mathit{D}\in {\mathbb{R}}^{m\times n}$	$\mathit{T}$	$\mathit{I}\in {\mathbb{R}}^m$
$\mathit{I}\in {\mathbb{R}}^{n\times n},\mathbf{1}\in {\mathbb{R}}^n$	$\mathbf{\Delta },\mathit{\delta }$	$\mathrm{\Omega }\in {\mathbb{R}}^{n\times m},\mathbf{1}\in {\mathbb{R}}^n$
$\left[\begin{array}{cc}\mathit{D}& -\mathit{D}\end{array}\right]\in {\mathbb{R}}^{m\times 2n}$	$\mathit{T}\mathit{F}$	$\left[\begin{array}{cc}\mathrm{\Omega }& -\mathrm{\Omega }\end{array}\right]\in {\mathbb{R}}^{n\times 2m}$

, which shows that both the sparse (2) and the cosparse representation (8) are particular instances of (14). A few details are in order.

The sparse representation (2) meshes easily onto (14) with the observation that the constraints describe a canonical cross-polytope (not scaled or projected in a lower-space). Thus, the application of the multi-parametric proceeds directly and requires no further clarification.

The cosparse representation can as well be written in form (14) with some caveats. Foremost, since it is natural to take $m\ll n$, we may consequently assume that $m\le 2n$. In this case, $\mathit{T}\mathit{F}$ is full column-rank and thus ${\left(\mathit{T}\mathit{F}\right)}^{\top }\left(\mathit{T}\mathit{F}\right)\in {\mathbb{R}}^{2m\times 2m}$ is full-rank, which simplifies the reasoning around (19)–(20). On the other hand, the constraint representation is more challenging. In (8), there appears ${\mathrm{\Omega }}_{\mathrm{\Lambda }}$, which is the restriction of $\mathrm{\Omega }$ by the elimination of $m-s$ columns. This is done in (8) through the addition of a rank condition. Within the framework of Section 3, we can instead restrict the enumeration to those sets of active constraints which verify the rank condition.

4.2. The Enumeration Roadblock

The “dirty” secret in multi-parametric programming is that the number of critical regions explodes exponentially with the parameter space size and with the complexity of the feasible domain [9]. Moreover, the number of feasible (non-empty) regions is not usually known a priori [6].

In this particular case, the cross-polytope’s specific structure means that we can bound a priori the number of critical regions and enumerate them constructively. To do this, we recall [11] that, for each pair of non-opposite vertices, there is an edge joining them. More generally, each set of $k+1$ orthogonal vertices corresponds to a distinct k-dimensional face that contains them (particular, more familiar instances of the ‘k-dimensional face’ are the $0,1,n-1$-dimensional faces—the vertices, edges and, respectively, facets of a polyhedron). The number of k-dimensional faces in an n-dimensional cross-polytope is therefore

$$2^{k+1}\left(\genfrac{}{}{0pt}{}{n}{k+1}\right).$$

(25)

As highlighted in Section 3, the constrained optimum ${\mathit{\xi }}_i^{\star }=\mathit{F}{\mathit{\alpha }}_i^{\star }$ is found by fixing $\widehat{m}$ of the cross-polytope’s vertices (i.e., those for which $\widehat{\mathit{\alpha }}>0$). Due to the cross-polytope’s properties, we know that these vertices describe an $\widehat{m}-1$-dimensional face. With the full-rank condition employed in (18), $\widehat{m}\le m+1$, and applying the bound from (25), we have that the total number of faces with dimension m or lower is

$$\sum _{\widehat{m}=1}^{m+1}{2}^{\widehat{m}}\left(\genfrac{}{}{0pt}{}{n}{\widehat{m}}\right).$$

(26)

Each $\widehat{m}$-dimensional face corresponds to (a possibly empty) region (23). Thus, the total number of feasible combinations of active constraints is (26).

Unfortunately, depending on the values of $m,n$, the number of regions may prove to be prohibitively large (both from a computational and from a memory footprint/retrieval viewpoint).

While there are multiple approaches that attempt to alleviate the issue [14], there is no panacea. In what follows, we consider a simple approach: regions ${\mathcal{R}}^i$ are added to the partition only on an “as-needed” basis. More precisely, $\mathit{y}$ is searched in the list of already-computed regions; if found, the affine law (24) is applied; if not, (20) is solved for the current parameter $\mathit{y}$ in order to identify the optimal solution and the critical region over which it is valid. Once found, these are stored for further use.

A sketch of the approach is shown in Algorithm 2 (for simplicity’s sake, only for the sparse representation case is the cosparse case similar and can be deduced easily).

Algorithm 2: Multi-parametric implementation of the sparse problem.

Steps 3 and 6 of the algorithm deserve some additional thought. Recall that ${\mathit{V}}^{\top }=\left[\begin{array}{cc}{\mathit{W}}^{\top }& -{\mathit{W}}^{\top }\end{array}\right]$ has a symmetrical structure. Coupling this with the observations around (25)–(26) shows that, for each combination of $\widehat{m}$ active constraints (i.e., those for which $\widehat{\mathit{\alpha }}>0$), there are $2^{\widehat{m}}$ permutations in the column selection from ${\mathit{V}}^{\top }$ such that the resulted sub-matrices differ only through the signs of their columns. That is, there exist suitably chosen diagonal matrices $\widehat{\mathit{P}},\tilde{\mathit{P}}$ with $\pm 1$ entries such that, e.g., ${\tilde{\mathit{V}}}_1^{\top }\mapsto {\tilde{\mathit{V}}}_1^{\top }\tilde{\mathit{P}}$, ${\widehat{\mathit{V}}}_{12}^{\top }\mapsto {\widehat{\mathit{V}}}_1^{\top }\widehat{\mathit{P}}$. Recalling that, for an active set selection $\widehat{(·)},\widetilde{(·)}$ there exist solutions $\mathit{\alpha }\left(\mathit{y}\right),\mathit{\lambda }\left(\mathit{y}\right)$, introducing the sign permutations characterized by $\widehat{\mathit{P}},\tilde{\mathit{P}}$ into (20) leads to a similar sign permutation of the solution:

$$\left\{\begin{array}{cc}\widehat{\mathit{\alpha }}\left(\mathit{y}\right)\hfill & ={\widehat{\mathit{M}}}_{\alpha }\mathit{y}+{\widehat{\mathit{N}}}_{\alpha },\hfill \\ \tilde{\mathit{\lambda }}\left(\mathit{y}\right)\hfill & ={\tilde{\mathit{M}}}_{\lambda }\mathit{y}+{\tilde{\mathit{N}}}_{\lambda },\hfill \end{array}\right.\stackrel{\widehat{\mathit{P}},\tilde{\mathit{P}}}{\to }\left\{\begin{array}{cc}\widehat{\mathit{P}}\widehat{\mathit{\alpha }}\left(\mathit{y}\right)\hfill & ={\widehat{\mathit{M}}}_{\alpha }\mathit{y}+{\widehat{\mathit{N}}}_{\alpha },\hfill \\ \tilde{\mathit{P}}\tilde{\mathit{\lambda }}\left(\mathit{y}\right)\hfill & ={\tilde{\mathit{M}}}_{\lambda }\mathit{y}+{\tilde{\mathit{N}}}_{\lambda },\hfill \end{array}\right.\stackrel{{\widehat{\mathit{P}}}^2=\mathit{I},{\tilde{\mathit{P}}}^2=\mathit{I}}{\to }\left\{\begin{array}{cc}\widehat{\mathit{\alpha }}\left(\mathit{y}\right)\hfill & =\widehat{\mathit{P}}{\widehat{\mathit{M}}}_{\alpha }\mathit{y}+\widehat{\mathit{P}}{\widehat{\mathit{N}}}_{\alpha },\hfill \\ \tilde{\mathit{\lambda }}\left(\mathit{y}\right)\hfill & =\tilde{\mathit{P}}{\tilde{\mathit{M}}}_{\lambda }\mathit{y}+\tilde{\mathit{P}}{\tilde{\mathit{N}}}_{\lambda }.\hfill \end{array}\right.$$

(27)

The last implication exploits the structure of $\widehat{\mathit{P}},\tilde{\mathit{P}}$ by noting that their squared forms are the identity matrices. The meaning of (27) is twofold:

(i)

Only a single representative of the same family of active constraints has to be stored (Step 6 of Algorithm 2); the remaining $2^{\widehat{m}}-1$ variations may be deduced by multiplying with suitable $\widehat{\mathit{P}},\tilde{\mathit{P}}$;

(ii)

The inclusion test (Step 3 of Algorithm 2) has to account for the sign permutations; this can be done, e.g., by checking whether $\left[{\widehat{\mathit{M}}}_{\alpha }\mathit{y}+{\widehat{\mathit{N}}}_{\alpha }\right]\odot {\left[{\tilde{\mathit{M}}}_{\lambda }\mathit{y}+{\tilde{\mathit{N}}}_{\lambda }\right]}_{\mathit{j}}\ge \mathbf{0}$ holds. We denoted with ‘⊙’ the elementwise product and with ${[·]}_{\mathit{j}}$ the selection of $\widehat{m}$ (out of $\tilde{m}$) rows which correspond to the ones appearing in $\widehat{\mathit{\alpha }}$. Recall that weights $\mathit{\alpha }$ come in pairs (since they are attached to the unit vectors $\pm {\mathit{e}}_i$). This means that in a pair of indices $(i,i+n),\forall i\in \{1\cdots n\}$ there can be at most an active constraint (either the i-th or the $i+n$-th). Thus, whenever we permute the signs we, in fact, switch these indices between the active and inactive sets.

Some remarks are in order.

Remark 5.

Algorithm 2 only provides an alternative for step 2 of Algorithm 1. Still, the same reasoning can be extended to the dictionary learning as it is, a relatively similar optimization problem.

Remark 6.

Completing online the partition of critical regions (as is done in step 6 of Algorithm 1) may still lead to an exorbitant number of regions (as it may happen that solving for each $\mathit{y}$ requires describing a significant part of the entire partitioning). We expect this to happen when the set of signals $\mathbf{Y}$ is chosen randomly. Fortunately, as stated earlier, $\mathit{Y}$ is a collection of corrupted signals coming from a smaller sub-space. We expect thus that the number of regions actually described is much less than the total number of feasible ones.

Remark 7.

Another approach, not described here, is to merge the explicit and implicit approaches [14]. Instead of enumerating all sets of active constraints for a relatively large parameter dimension, we may choose to consider a reduced number of constraints. Thus, in the offline stage, we compute and store a reasonable number of regions and in the online stage we solve a reduced-size optimization problem (based on information retrieved from the stored lookup tables).

Remark 8.

While here, we store and search sequentially through the collection of regions (23), and we note that more efficient storing options are possible. A generic solution is to use the Least Recently Used (LRU) algorithm to keep a fixed number of regions while minimizing the chance of a miss during search [19]. Of possible use is the adjacency graph structure in which the cross-polytope’s faces are (since the critical regions (23) are in a one-to-one relation with the cross-polytopes’s faces).

Remark 9.

The computational costs of Algorithm 2 can be divided between $t_r$ (retrieval time when searching through the stored critical regions), $t_s$ (time spent solving an online optimization procedure) and $t_k$ (time spent constructing a critical region and its associated affine law). Among these, $t_k$ is negligible as it requires only matrix operations. As long as, on average, enough signals $\mathit{y}$ are found in the stored regions, the procedure is useful: it suffices that a signal $\mathit{y}$ is found more often than $⌊{\displaystyle \frac{t_s}{t_r}}⌋$.

4.3. On the Multi-Parametric Interpretation of the ‘Given Approximation Quality’ Case

For simplicity, we discuss only the sparse representation case shown in (7). The cosparse case (9) can be treated similarly and is not detailed here.

Recall that, as stated in Section 2, the “true” DL formulations are those shown in (1) and (7) where the ${\ell }_0$ norm appears (either explicitly in the first case or implicitly in the second case). Thus, all of the other formulations appearing in Section 2 are relaxations (exploiting the sparsity induced by the ${\ell }_1$ norm, appearing either in the cost or as a constraint) which aim to simplify the computational effort.

Until here, we have concentrated on these forms where the ${\ell }_1$ norm appears in the constraint part of the optimization problem. Now, we mention some of the drawbacks and possible solutions for the other class of relaxations (those for which the ${\ell }_1$ norm appears in the cost). First, we consider an over-approximation of the quadratic constraint. Any ellipsoid (and therefore the ball ${\parallel \mathit{y}-\mathit{D}\mathit{x}\parallel }_2^2\le ϵ$) is a convex set, which by definition means that it can be approximated arbitrarily well by a polyhedral set [18], say, a construction like

$$\{\mathit{x}:\mathit{F}(\mathit{D}\mathit{x}-\mathit{y})\le \theta \}\subset \{\mathit{x}:\parallel \mathit{y}-\mathit{D}\mathit{x}{\parallel }_2^2\le ϵ\}\subset \{\mathit{x}:\mathit{F}(\mathit{D}\mathit{x}-\mathit{y})\le (1+ϵ)\theta \},$$

(28)

with $F,\theta$ appropriately chosen.

Proceeding as in Section 3, we arrive at the dual (KKT) form

$$\begin{array}{cc}\hfill \mathrm{stationarity}:& \mathbf{0}\in \partial \left({\left|\right|\mathit{x}\left|\right|}_1\right)+\nabla (\mathit{F}\mathit{D}\mathit{x}-\mathit{F}\mathit{y}-\theta )·\lambda ,\hfill \end{array}$$

(29a)

$$\begin{array}{cc}\hfill \mathrm{primal}\mathrm{feasibility}:& \mathit{F}\mathit{D}\mathit{x}-\mathit{F}\mathit{y}-\theta \le 0,\hfill \end{array}$$

(29b)

$$\begin{array}{cc}\hfill \mathrm{dual}\mathrm{feasibility}:& \mathit{\lambda }\ge 0,\hfill \end{array}$$

(29c)

$$\begin{array}{cc}\hfill \mathrm{complementarity}:& \mathit{\lambda }\times (\mathit{F}\mathit{D}\mathit{x}-\mathit{F}\mathit{y}-\theta )=0.\hfill \end{array}$$

(29d)

There are several issues that make (29) challenging to solve as in Section 3:

(i)

The gradient is not well defined in inflexion points (where at least one of the vector’s components is zero); this requires the use of the ‘sub-gradient’ notion which is set-valued (29a) becomes ‘$\mathbf{0}\in sign\left(\mathit{x}\right)+{\left(\mathit{F}\mathit{D}\right)}^{\top }·\mathit{\lambda }$’, where

$$sign\left(x_i\right)=\left\{\begin{array}{cc}1,\hfill & x_i>0,\hfill \\ -1,\hfill & x_i<0,\hfill \\ ,\hfill & x_i=0.\hfill \end{array}\right.$$

(ii)

The problem may be relatively large (depending on the number of inequalities used to over-approximate the initial quadratic constraint).

Remark 10.

An alternative to item 4.3 is to consider a zonotopic representation of the quadratic constraints. The ellipsoid they describe is in fact a zonoid (a set which can be approximated arbitrarily well by a zonotope [20]). The advantage of this method is that, in generator representation, a zonotope has a much more compact form (which directly leads to a reduced KKT form).

5. Results

In this section, we present a few numerical simulations on different types of data. We start with synthetic data generated from a random Gaussian distribution such that the signals evenly populate the space in which they lie. Then, we investigate graph signals produced by networks of water distribution, thus studying the impact on data with embedded structure. Finally, we present simulations on images where data items represent small pixel patches. The scripts have been written in Matlab 2018b (MathWorks, Natick, MA, USA), using the MPT3 toolbox [21] and Yalmip [22]. See Supplementary Materials for access to the source code implementation.

5.1. Synthetic Data

In our first experiment, we use synthetic signals. We generate a random dictionary of $m\times n$ elements drawn from a zero mean and unit variance Gaussian distribution. After normalizing the dictionary, we randomly choose s atoms to generate r sparse signals whose coefficients are also picked at random. Our goal is to generate N total signals that are perturbed, dense versions of r clean, sparse signals. We call the r signals seeds and we proceed to perturb each seed $N/r$ times with different additive white Gaussian noise:

$$\begin{array}{cc}{\mathit{y}}_i={\mathit{D}}_0\mathit{x}+{\mathit{v}}_i,\hfill & i=1:N/r,\hfill \end{array}$$

(30)

where ${\mathit{D}}_0$ is the random dictionary, $\mathit{x}$ the sparse seed signal and ${\mathit{v}}_i$ the additive noise. Signal ${\mathit{y}}_i$ is thus a dense signal; all its elements are non-zero. We repeat (30) r times to generate N synthetic signals that we store in $\mathit{Y}$, which is thus known to have a sparse structure.

In the following we generate $n=1500$ signals, of $m=20$ elements each, from $r=\{15,150,1500\}$ seeds built as the linear combination of $s=\{3,4,5\}$ vectors from a dictionary with $n=50$ atoms.

We proceed with the construction shown in Algorithm 2 where the critical regions and associated optimal laws are computed as-needed. As stated earlier, this is justified by the excessive number of potentially feasible sets of active constraints (26).

Note that we use the real dictionary ${\mathit{D}}_0$. For a full analysis (not the object of this paper), we may repeat the procedure for all the intermediate dictionaries obtained along the dictionary learning procedure (step 4 in Algorithm 1).

In Figure 4 and Figure 5, we depict the plots in red, blue, green to denote sparsity (3, 4 and 5 respectively) and add diamond, triangle and square markers to denote the seed size (15, 150 and 150, respectively). Thus, a blue with diamond-shaped marker denotes the case $s=4,r=15$.

Figure 4 shows that the number of regions (23) is significantly lower than the number of iterations of the parameter vector $\mathit{y}$ for all combinations of parameters s (sparsity) and r (seed).

As expected, the most relevant factor in the synthetic examples is the “randomness” of the data: the best cases are those for which $n=1500$ vectors $\mathit{y}$ are the result of only $r=15$ seeds; still, even in the worst case, where each $\mathit{y}$ is associated with a single seed ($n=r=1500$), we have a significant difference between the number of regions and of iterations.

Figure 5 details the distribution of the inclusion occurrences (i.e., how many vector instances $\mathit{y}$ stay in a certain critical region). Beyond the expected fact that to the cases with lower randomness correspond more inclusions, we also note that, in general, the plots have a steep descent.

Table 2. Number of regions as a percentage of the total number of signals ([%]).

Sparsity/Seed Size	r = 15	r = 150	r = 1500
s = 3	33.67	43.33	84.40
s = 4	21.33	43.87	86.67
s = 5	5.33	41.13	89.87

shows the number of regions as a percentage of the total number of signals.

Out of the many stored regions (23), only relatively few contain multiple vectors $\mathit{y}$. This is of interest in cases of limited storage (see Remark 8).

The computation times show empirically the effect of having a large number of stored regions. While the signal shown in Figure 6 is noisy, a clear ascending trend is visible (at each iteration step, there are more regions to be searched—step 3 of Algorithm 2).

Disregarding memory considerations, the continued storage of critical regions makes sense until the point where searching through the existing regions takes longer than solving the online optimization problem.

5.2. Water Networks

Dictionary learning has only recently been applied for modelling water networks, showing promising results at fault detection and isolation (FDI) tasks [23]. Represented as a graph, where the edges are pipes and the nodes’ distribution junctions, the water network main characteristic for FDI is the recorded pressure at each of its nodes. Given a known nominal pressure vector ${\mathit{p}}_0$, whose elements represent the nominal pressure in each junction, we compare it to the current measurement $\mathit{p}$ leading to the pressure residue

$$\mathit{r}=\mathit{p}-{\mathit{p}}_0.$$

(31)

Thus, faults are detected by the network nodes by investigating the residue vectors over time.

In our experiments, we choose the benchmark Hanoi water network that consists of 31 junctions. We iterate through each of the nodes and produce faults of various magnitudes. For each scenario, we record the pressure residue (31) and store all of them as columns in the matrix $\mathit{R}$. A single fault is expected to have impact on the measured pressured of its node and its first and second level neighbours. See [23] for a thorough description of the water network and the data generation process.

We generate 16 faults for each node leading to $n=496$ training signals of $m=31$ entries each. We perform DL on a dictionary of $n=124$ atoms with sparsity $s=\{3,5\}$ for 50 iterations and depict in Figure 7 the last sparse representation stage during training. In Figure 7a, we can see that the number of regions is around 36% of the total number of signals, which is confirmed by the number of region re-utilization depicted in Figure 7b.

5.3. Images

For the last experiment, we train our dictionary on a small set of patches extracted from a given image. We use square patches of $3\times 3$ pixels that we vectorize leading to signals $\mathit{y}$ of $m=9$ elements. It is good practice to use overlapping patches for both training and representation with the learned dictionary afterwards. For example, using overlapping patches when performing image denoising provides better results and eliminates the blocking effect.

In our simulations, we use overlapping patches with a sliding step of one pixel that we extract from the image of Lena [24]. Using $n=3000$ patches, we train a dictionary of $n=36$ atoms with sparsity $s=3$ for 50 iterations. With the resulting dictionary, we proceed to represent $N_t=\mathrm{50,000}$ test patches from Lena that were not used in the DL process.

The results are depicted in Figure 8. We again observe that there are critical regions re-used (the number of regions is approximately 74% of the total number of signals) but we also note that the problem’s size leads to relatively large storage requirements. The large problem size (at least when compared with the previous examples) hinders the critical region generation (due badly conditioned matrices) and highlights that a large number of regions need to be stored until a significant number of inclusions occur.

6. Conclusions and Future Directions

In this article, we proposed a novel sparse representation method for both the spare and cosparse models by looking at the problem as a multi-parametric formulation and solving it through the Karush–Kuhn–Tucker conditions. Thus, we solve a large optimization problem once in the beginning, and then, for each signal, we simply enumerate through a set of possible regions where the solution might lie. The advantage of this approach is the significant reduction in sparse representation execution times: if the trained dictionary is distributed together with the associated collection of regions, representation reduces to performing a few matrix-vector computations.

Our model is a good fit for online representation scenarios which is why we plan on investigating in the future its adaptation to online dictionary learning [25], where the dictionary is also updated with each new signal that is sparsely represented. Existing methods have shown that the dictionary update problem can be reduced to a simple rank-1 update $\mathit{D}\leftarrow \mathit{D}+\mathbf{\Gamma }$ ([15], Chapter 5) and we are currently working on changing $\mathbf{\Gamma }$ in order to exploit the existing multi-parametric model of the existing dictionary.

Further work will consider efficient storage and retrieval of the active critical region, e.g., by exploiting the symmetry inherent in the cross-polytope face lattice. This will alleviate issues pertaining to memory limitations and search through the list of stored regions (as shown in (27), a single representative suffices in characterizing an entire family of critical regions and affine laws).