Adaptive-Size Dictionary Learning Using Information Theoretic Criteria

Abstract

Finding the size of the dictionary is an open issue in dictionary learning (DL). We propose an algorithm that adapts the size during the learning process by using Information Theoretic Criteria (ITC) specialized to the DL problem. The algorithm is built on top of Approximate K-SVD (AK-SVD) and periodically removes the less used atoms or adds new random atoms, based on ITC evaluations for a small number of candidate sub-dictionaries. Numerical experiments on synthetic data show that our algorithm not only finds the true size with very good accuracy, but is also able to improve the representation error in comparison with AK-SVD knowing the true size.

1. Introduction

Dictionary learning (DL) is now a mature field [1,2,3], with several efficient algorithms for solving the basic problem or its variants and with numerous applications in image processing (denoising and inpainting), classification, compressed sensing and others. The basic DL problem is: given N training signals gathered as the columns of the matrix $\mathit{Y}\in {\mathbb{R}}^{m\times N}$ and the sparsity level s, find the dictionary $\mathit{D}\in {\mathbb{R}}^{m\times n}$ by solving

$$\begin{array}{cc}\underset{\mathit{D},\mathit{X}}{min}\hfill & {\parallel \mathit{Y}-\mathit{D}\mathit{X}\parallel }_F^2\hfill \\ \mathrm{s}.\mathrm{t}.\hfill & \parallel {\mathit{x}}_{\ell }{\parallel }_0\le s,\ell =1:N\hfill \\ & \parallel {\mathit{d}}_j{\parallel }_2=1,j=1:n\hfill \end{array}$$

(1)

Here, ${\parallel ·\parallel }_F$ is the Frobenius norm; and ${\mathit{x}}_{\ell }$ and ${\mathit{d}}_j$ are columns of $\mathit{X}$ and $\mathit{D}$, respectively. The first constraint says that the matrix $\mathit{X}$ has at most s nonzeros on each column. The second constraint is the normalization of the atoms (columns of the dictionary).

An alternative view is to relate Equation (1) to the sparse matrix factorization—or dictionary recovery (DR)—problem. The different assumption is that there indeed exist a dictionary $\mathit{D}$ and a sparse matrix $\mathit{X}$ such that $\mathit{Y}\approx \mathit{DX}$; the purpose is to recover them from the data $\mathit{Y}$. Such a ground truth is usually not available in practical applications. However, investigating DR is useful for theoretical and practical developments. Significant recent work on this matter can be found in [4,5,6,7]. Our contribution belongs more to the DR line of thought.

In most DR algorithms, the dictionary size n (the number of atoms) is assumed to be known. In DL applications, n is chosen with a rather informal trial-and-error procedure: a few sizes are used and that giving the best performance in the application at hand is selected. A dictionary with more atoms typically gives a smaller error: if ${\mathit{D}}_1\in {\mathbb{R}}^{m\times n_1}$ and ${\mathit{D}}_2\in {\mathbb{R}}^{m\times n_2}$, with $n_1<n_2$, we expect that $\parallel \mathit{Y}-{\mathit{D}}_1{\mathit{X}}_1{\parallel }_F^2>{\parallel \mathit{Y}-{\mathit{D}}_2{\mathit{X}}_2\parallel }_F^2$. However, this does not mean that ${\mathit{D}}_2$ is necessarily better. We aim here at an automatic choice of the size that is the most appropriate to the data, based on Information Theoretic Criteria (ITC).

The only previous work based on ITC [8] uses Minimum Description Length for the choice of the size and the sparsity level. Their algorithm implements a virtual coder and searches exhaustively over all considered dictionary sizes. A recent method [9] reporting promising results has a similar purpose, but is based on geometric properties of the DR problem.

All other methods are based on heuristics that essentially aim to obtain a dictionary having the same representation error as that obtained with a standard method, but with smaller size. We list a few of the successful approaches. In [10,11], a small dictionary is grown by adding representative atoms from time to time in the iterative DL process. Clustering ideas are used in [12,13,14] to reduce the size of a large dictionary. A direct attempt of optimizing the size is employed in [15] by introducing in the objective a proxy for the size as penalization. An Indian Buffet Process is the tool in [16]. Other techniques based on Bayesian learning are [17,18].

Unlike the method in [8] but similarly to that in [9], our approach tries to adapt the size during the DL process. It also has a much simpler implementation, using standard ITC that do not intervene in the learning itself, but only in the selection of the atoms. Hence, the complexity is not much higher than that of the underlying DL algorithm.

Section 2 presents the ITC, specialized to the DL problem, and describes the pool of candidate dictionaries during the DL algorithm. Section 3 gives the details of our algorithm for adapting the dictionary size. Section 4 is dedicated to experimental results on synthetic data that show that our algorithm is able to recover the size of the true dictionary, in various noise and sparsity level conditions and with various initializations. Our algorithm also gives, almost always, recovery errors on test data that are better than those given by the underlying DL algorithm in possession of the true dictionary size. Comparison with the method in [9] is also favorable.

2. Ingredients

2.1. Information Theoretic Criteria

ITC serve for assessing the adequacy of a model to a process described by experimental data, by combining its goodness of fit (approximation error) with its complexity. The underlying model in Equation (1) is $\mathit{Y}=\mathit{DX}+\mathit{U}$, where $\mathit{U}$ is a matrix with entries that follow a Gaussian distribution with zero mean and an unknown variance (the same for all entries) [19]. Denoting $T=mN$, the goodness of fit is expressed via the Root Mean Square Error

$$\mathrm{RMSE}=\frac{1}{\sqrt{T}}{\parallel \mathit{Y}-\mathit{D}\mathit{X}\parallel }_F.$$

(2)

The complexity depends primarily on the number of parameters; in the DL or DR case, this is

$$\mathrm{NoP}=sN+(m-1)n.$$

(3)

The first term corresponds to the number of nonzeros in $\mathit{X}$. The second term is the number of independent elements of the dictionary; we subtract n from the total of $mn$ elements to account for the atom normalization constraints.

After preliminary investigation with several ITC, we kept only two for our DL approach. The first is Bayesian Information Criterion (BIC) [20], extended to the form

$$\mathrm{EBIC}=2log\mathrm{RMSE}+\frac{logT}{T}\mathrm{NoP}+\frac{2N}{T}log\left(\genfrac{}{}{0pt}{}{n}{s}\right).$$

(4)

The first two terms are the standard ones; we have added a third term to account for all possible positions of the nonzero entries in the matrix $\mathit{X}$, inspired by [21].

The second ITC is Renormalized Maximum Likelihood (RNML) [22,23], adapted as in [24] to our context, to which we have added the same combinatorial term as above:

$$\begin{array}{c}{\mathrm{ERNML}}_1=(T-\mathrm{NoP})log\frac{{\mathrm{RMSE}}^2}{T-\mathrm{NoP}}+\mathrm{NoP}log\frac{{\parallel \mathit{DX}\parallel }_F^2}{T·\mathrm{NoP}}\hfill \\ +log\left[\mathrm{NoP}(T-\mathrm{NoP})\right]+2Nlog\left(\genfrac{}{}{0pt}{}{n}{s}\right).\hfill \end{array}$$

(5)

We note that a different version, ERNML${}_2$, derived in [25], further analyzed in [26] and adapted as Equation (5), gave results similar to those of ERNML${}_1$, thus we do not report it here. The ITC dismissed after preliminary (and thus possibly insufficient) investigation are those from [27,28].

2.2. Candidate Dictionaries

Application of ITC needs several candidate dictionaries from which selection is made with the minimum ITC value. Since we aim to run a single instance of the learning algorithm and, as such, we have a single dictionary, the only possibility is to compare smaller dictionaries made of a subset of the atoms. Even so, there are too many possible combinations. To reduce their number, we order the atoms based on their importance in the representations. Since

$$\mathit{DX}=\sum _{j=1}^n{\mathit{d}}_j{\mathit{x}}_j^T,$$

where ${\mathit{x}}_j^T$ is the j-th row of $\mathit{X}$, we sort the atoms in decreasing order of their “power”

$$P\left({\mathit{d}}_j\right)={\parallel {\mathit{x}}_j^T\parallel }_2.$$

(6)

We still name $\mathit{D}$ the sorted dictionary. For selection, we consider dictionaries ${\mathit{D}}_{\nu }\in {\mathbb{R}}^{m\times \nu }$ that are made of the first $\nu \le n$ atoms of $\mathit{D}$. Thus, there are at most n candidates. However, since small dictionaries are certainly not useful, we can also impose a lower bound $n_{min}$ and take $\nu \ge n_{min}$. One can choose $n_{min}=m$ or even larger values.

2.3. DL Algorithm

Many DL algorithms are suited to the framework that we propose. We confine the discussion to standard algorithms, which aim to iteratively improve the dictionary and whose iterations have two stages: (i) sparse coding, in which the representation matrix $\mathit{X}$ is computed for fixed dictionary $\mathit{D}$; and (ii) dictionary update, in which $\mathit{D}$ is improved, possibly together with the nonzero elements of $\mathit{X}$, but without changing the nonzero positions. Such algorithms are impervious to dictionary size changes; atoms can be removed or added between iterations without any change in the algorithm. Since they give good results and are fast, we adopt some of the simplest algorithms: Orthogonal Matching Pursuit (OMP) [29] for sparse coding and Approximate K-SVD (AK-SVD) [30] for dictionary update.

3. Algorithm

We assume first that the sparsity level is known. Our strategy is implemented by Algorithm 1, named ITC-ADL. Starting with an initial dictionary of size $n_{\mathrm{init}}$, we run the DL algorithm. We change the size only every c iterations, to allow the current set of atoms to be sufficiently trained together. Thus, the selection based on ITC described in Section 2.2 can be meaningful.

Although we could compute ITC values for all dictionary sizes between $n_{min}$ and the current n, it is more efficient to consider only a smaller number of candidates $n_{\mathrm{cand}}$; note that the representations have to be recomputed for each sub-dictionary; although this can be done economically (see below the discussion on complexity), it may become a significant burden. Let us denote $n_{\mathrm{ITC}}$ the size with minimum ITC value among the $n_{\mathrm{cand}}$ largest possible dictionaries (those with sizes n, $n-1$, …, $n-n_{\mathrm{cand}}+1$).

The main question is now how to use this value. If $n_{\mathrm{ITC}}<n$, we might be tempted to continue the learning process with only $n_{\mathrm{ITC}}$ atoms. However, this seems to be (assertion confirmed by numerical experiments) a too drastic decision, that can easily lead to premature shrinking of the dictionary. Instead, we take $n_{\mathrm{ITC}}$ as an indicator of the direction where n must evolve. If $n_{\mathrm{ITC}}$ is much smaller than the current n, we decrease the size by ${\delta }_{-}$ (this number is 5 in our experiments); if $n_{\mathrm{ITC}}$ is only slightly smaller, than we decrease the size by one; finally, we interpret $n_{\mathrm{ITC}}=n$ as a sign that the size needs to be increased and add ${\delta }_{+}$ atoms to the dictionary (we take ${\delta }_{+}=5$). There are several methods for generating new atoms [3] (Section 3.9); we choose the simplest: random atoms.

The whole algorithm is run, as typical in DL, for a preset number K of iterations. Only at the end of these iterations we take $n_{\mathrm{ITC}}$ as a true size information. With this size, we run c more DL iterations, as a final refinement.

Algorithm 1: ITC-ADL: DL with ITC-adapted dictionary size.

It is relatively hard to estimate the complexity of ITC-ADL, due to the dictionary size variations. We describe only the extra operations with respect to a standard DL algorithm, disregarding the size. There are two main categories of operations that increase the complexity. The first is the total number of iterations, that has to be larger than for standard DL, in order to let the dictionary size converge. The ITC give reliable information if the dictionary is well trained, hence neither K nor c can be small. In the tests reported below, we took $K\ge 200$ and $c=5$.

The second extra operation is the computation of ITC, which involves the recomputation of the representations for each dictionary. Since we already have the representations $\mathit{X}$ for the full dictionary of size n, we can progressively recompute only the representations that change as the size decreases. For example, the dictionary of size $n-1$ lacks atom ${\mathit{d}}_n$, but is otherwise identical with $\mathit{D}$. We need to recompute only the representations that contain ${\mathit{d}}_n$. Their number should be less than $\left(Ns\right)/n$, since atom ${\mathit{d}}_n$, which is the least used, should appear in less representations than the average. Thus, overall, the number of recomputed representations is likely bounded by $(n_{\mathrm{cand}}s/n)N$, which is comparable with N (the number of signals represented at each iteration); since this happens only every cth iteration, the extra complexity is relatively small.

In the experimental conditions described in the next section, ITC-ADL is about 3–5 times slower than the underlying AK-SVD, which is one of the fastest DL algorithms. This is not an excessive computational burden.

If the sparsity level s is not known, we simply run ITC-ADL for several candidate values; the best ITC value decides the best $(n,s)$ pair. Adapting also s during the algorithm may be possible, but seems more difficult than adapting only the size and was left for future work.

4. Numerical Results

We tested our algorithm on synthetic data, obtained with “true” dictionaries ${\mathit{D}}_{\mathrm{true}}$ whose unit norm atoms are generated randomly following a Gaussian distribution. Given the sparsity level s, the representations $\mathit{X}$ are also generated randomly, with nonzeros on random positions. The data are $\mathit{Y}={\mathit{D}}_{\mathrm{true}}\mathit{X}+\mathit{U}$, where the entries of $\mathit{U}$ are statistically independent, Gaussian distributed, with zero mean. The variance of the additive noise is chosen to have four different values for the signal-to-noise ratio (SNR): 10 dB, 20 dB, 30 dB and 40 dB. The dictionary sizes are $n\in \{128,192,256\}$; the overcompleteness factor $n/m$ has moderate values, as in most applications.

Some of the input data for Algorithm 1 are constant throughout all the experiments. The number of signals is $N=4000$ and their size is $m=64$. The number of iterations is $K\in \{200,300,400\}$, increasing with n. The dictionary size changes are made every $c=5$ iterations. The size steps are ${\delta }_{+}={\delta }_{-}=5$. The number of size candidates is $n_{\mathrm{cand}}=20$. All the results are obtained with 50 runs for the same data, but with different realizations of ${\mathit{D}}_{\mathrm{true}}$ and $\mathit{U}$.

We report in this section only a few representative results. More results can be found in Appendix A. Representative Matlab sources are given at www.schur.pub.ro/download/itc-adl.zip.

4.1. Experiments with Known Sparsity Level

In all the experiments reported in this section, ITC-ADL is in possession of the true value of s, which takes even values from 4 to 12. In the first round of experiments, the dictionary ${\mathit{D}}_{\mathrm{true}}$ has $n_{\mathrm{true}}=128$ atoms. The size $n_{\mathrm{init}}$ of the initial dictionary takes random values between 80 and 180, in order to test the robustness of the algorithm to initialization.

Table 1. Minimum (top), average (middle) and maximum (bottom) sizes computed when $n_{\mathrm{true}}=128$. Left: ERNML${}_1$. Right: EBIC.

SNR (dB)	$\mathit{s}=4$	6	8	10	12	SNR (dB)	$\mathit{s}=4$	6	8	10	12
	128	128	128	128	128		128	128	128	128	124
10	129.02	128.22	128.04	128.12	129.74	10	128.82	128.24	128.00	128.06	127.86
	143	132	130	130	143		143	135	128	129	130
	128	128	128	128	128		128	128	128	128	128
20	128.20	128.06	128.06	128.18	128.34	20	128.22	128.14	128.12	128.06	128.30
	130	129	129	131	137		130	131	132	129	131
	128	128	128	128	128		128	128	128	128	128
30	128.18	128.14	128.26	128.24	128.94	30	128.18	128.10	128.24	128.76	129.18
	129	133	130	131	138		130	129	132	133	135
	128	128	128	128	128		128	128	128	128	128
40	128.18	128.08	128.70	128.78	130.00	40	128.12	128.36	128.88	129.04	129.08
	129	129	143	132	140		129	132	133	137	132

reports the average, minimum and maximum size n computed by ITC-ADL over the 50 runs. Our algorithm is able to find very good estimates of the size, for all considered noise and sparsity level values. An important conclusion is that the size of the initial dictionary has no impact on the results.

We evaluate the performance of the dictionaries given by ITC-ADL on 1000 test signals generated similar to the training ones. For comparison, we compute the RMSE for two dictionaries: (i) ${\mathit{D}}_{\mathrm{true}}$, for which the representations are computed with OMP; and (ii) the dictionary computed by AK-SVD with the true size $n=n_{\mathrm{true}}$ and the same number of iterations ($K=200$, in this case). We denote RMSEt and RMSEn the RMSE obtained with these dictionaries. Both approaches have an advantage over ours: the first has the true dictionary, so it is in fact an oracle; the second uses the same DL algorithm that we use, but knows the size.

Table 2. Ratios RMSE/RMSE$\mathrm{t}$ and RMSE/RMSE$\mathrm{n}$ on test data, when $n_{\mathrm{true}}=128$. Left: ERNML${}_1$. Right: EBIC.

SNR (dB)	$\mathit{s}=4$	6	8	10	12	SNR (dB)	$\mathit{s}=4$	6	8	10	12
10	1.0194	1.0224	1.0258	1.0313	1.0402	10	1.0194	1.0224	1.0257	1.0327	1.0553
	0.9482	0.9741	0.9809	0.9883	0.9820		0.9482	0.9741	0.9808	0.9896	0.9964
20	1.0181	1.0203	1.0235	1.0256	1.0319	20	1.0190	1.0192	1.0227	1.0296	1.0417
	0.7326	0.8360	0.8792	0.9153	0.9231		0.7332	0.8352	0.8786	0.9187	0.9318
30	1.0178	1.0310	1.0432	1.0572	1.1006	30	1.0380	1.0228	1.0445	1.0776	1.1427
	0.3257	0.4835	0.6283	0.7133	0.7467		0.3323	0.4809	0.6310	0.7242	0.7722
40	1.0177	1.0181	1.2080	1.1955	1.3554	40	1.0177	1.0536	1.1560	1.3266	1.5319
	0.1516	0.2402	0.5606	0.5114	0.5721		0.1516	0.2459	0.5399	0.5857	0.6495

shows the average value of the ratios RMSE/RMSEt and RMSE/RMSEn. Values below 1 mean that our algorithm is better. Our algorithm always gives worse results than the oracle, which is expected, but the difference is often very small. Compared with AK-SVD, our algorithm is superior in all cases, the advantage growing with the SNR. Another conclusion that could be drawn is that the problem becomes harder as s and the SNR grow. The first part is natural; for the second part, an explanation can be that, as the SNR grows, there are fewer local minima with values close to the global one; our algorithm appears to be able to find them, while AK-SVD may be trapped in poor local minima; running it with several initializations can improve the results.

The results for $n_{\mathrm{true}}=192$ are only slightly worse, thus we jump to those for $n_{\mathrm{true}}=256$, shown in

Table 3. Minimum (top), average (middle) and maximum (bottom) sizes computed when $n_{\mathrm{true}}=256$. Left: ERNML${}_1$. Right: EBIC.

SNR (dB)	$\mathit{s}=4$	6	8	10	12	SNR (dB)	$\mathit{s}=4$	6	8	10	12
	256	256	256	254	330		256	256	222	206	225
10	258.74	256.88	256.64	346.60	444.28	10	258.46	256.32	253.40	246.30	308.10
	268	270	261	566	616		277	260	257	257	386
	255	256	256	256	256		256	256	256	256	239
20	259.80	256.54	256.50	256.62	267.36	20	259.30	256.84	256.36	256.42	256.04
	288	261	260	260	444		275	261	260	260	273
	256	256	256	256	256		256	256	256	256	236
30	258.90	261.98	260.32	269.66	325.76	30	258.84	261.02	258.12	261.86	263.68
	312	335	357	374	477		312	329	267	287	291
	256	256	256	256	256		256	256	256	256	256
40	267.02	261.50	271.12	304.62	324.72	40	265.12	260.12	260.80	275.56	268.28
	343	343	357	422	489		343	336	283	320	322

. The size $n_{\mathrm{init}}$ takes random values between 160 and 360. Now, some of the harder problems with large sparsity level ($s\ge 10$) are no longer well solved: some size estimations are wrong. However, the good behavior of the RMSE persists: most results are near-oracle and clearly better than AK-SVD.

We can see now some differences between the two ITC: ERNML${}_1$ tends to overestimate n, but hardly ever underestimates it, while EBIC is more prone to underestimation. However, in most setups, both ITC give sizes that are near from the true one. Regarding the RMSE (see

Table 4. Ratios RMSE/RMSE$\mathrm{t}$ and RMSE/RMSE$\mathrm{n}$ on test data, when $n_{\mathrm{true}}=256$. Left: ERNML${}_1$. Right: EBIC.

SNR (dB)	$\mathit{s}=4$	6	8	10	12	SNR (dB)	$\mathit{s}=4$	6	8	10	12
10	1.0427	1.0500	1.0617	1.0859	1.4794	10	1.0426	1.0504	1.0905	1.1843	1.5705
	0.9430	0.9670	0.9717	0.9835	0.9540		0.9429	0.9674	0.9981	1.0723	1.0123
20	1.0400	1.0408	1.0448	1.0508	1.0646	20	1.0385	1.0414	1.0466	1.0563	1.1320
	0.7105	0.8184	0.8342	0.8434	0.8792		0.7095	0.8189	0.8356	0.8479	0.9375
30	1.0568	1.0482	1.0488	1.0616	1.1677	30	1.0535	1.0466	1.0610	1.0882	1.2282
	0.3024	0.4160	0.4996	0.5599	0.7156		0.3017	0.4152	0.5066	0.5718	0.7482
40	1.1130	1.0600	1.1090	1.2461	1.3096	40	1.1304	1.0574	1.0790	1.2541	1.2866
	0.1147	0.1647	0.2471	0.4346	0.5666		0.1164	0.1627	0.2410	0.4363	0.5613

), the situation is somewhat reversed: ERNML${}_1$ is slightly better than EBIC.

4.2. Execution Times and Discussion of Parameter Values

We present here some characteristics of ITC-ADL based on experimental evidence.

The average running times of ITC-ADL, in the configurations described above, are shown in

Table 5. Average execution times (in seconds) of ITC-ADL (with ERNML${}_1$) and AK-SVD.

n	Algorithm	$\mathit{s}=4$	6	8	10	12
128	ITC-ADL	39.2	52.1	66.6	83.7	104.6
	AK-SVD	10.3	14.2	17.9	22.0	26.4
192	ITC-ADL	75.0	101.8	133.2	165.9	212.6
	AK-SVD	20.3	28.5	36.3	44.2	52.6
256	ITC-ADL	121.9	162.2	212.1	267.0	374.2
	AK-SVD	36.0	48.5	63.0	75.9	90.7

, together with those of AK-SVD. Over the considered n and s values, the ratio between the execution times of ITC-ADL and AK-SVD varies between $3.34$ and $4.12$.

Without reporting any actual times, we note that the ITC and the SNR have almost no influence. In addition, the execution time is roughly proportional with the number of iterations K and the number of signals N. The other parameters have obvious influences: more frequent dictionary size changes (smaller c) increase the time; same effect has a larger number of candidates $n_{\mathrm{cand}}$; the size steps ${\delta }_{+}$ and ${\delta }_{-}$ only slightly affect the time. We note that our implementation is not fully optimized, but it is based on a very efficient implementation of AK-SVD [30].

We chose the parameter values for the experiments reported in the previous section with the aim to show that a single set of values ensures good results for all considered dictionary sizes and sparsity levels. In fact, ITC-ADL is quite robust to the parameter values. Nevertheless, fine tuning is possible when n and s have a more limited range of values. We present below a few results that show the effect of the parameters on the outcome of ITC-ADL. We considered only the cases $n\in \{128,256\}$, $s\in \{6,10\}$, SNR$=30$ dB; the ITC is ERNML${}_1$. Since we have run again the algorithm, some results may be different from those from the previous section, due to the random factors involved.

We gave the size steps ${\delta }_{+}$ and ${\delta }_{-}$ values between 2 and 8. For $n=128$, the results are very similar. For $n=256$, a trend is (barely) visible in

Table 6. Average dictionary sizes given by ITC-ADL, for $n=256$, $s=6$, when ${\delta }_{+}={\delta }_{-}$ vary.

${\mathit{\delta }}_{+},{\mathit{\delta }}_{-}$	2	3	4	5	6	7	8
size	257.90	257.88	257.88	258.16	259.86	258.16	259.24

: larger size steps lead to poorer results. This is natural, since a large dictionary increase is a perturbing factor when the algorithm is near convergence. On the other hand, a small size step is not useful in the first iterations of the algorithm; if the initial size is far from the true one, the convergence can be very slow. Thus, although we have obtained good results with constant ${\delta }_{+}={\delta }_{-}=5$, some refinements are certainly possible. It makes sense to decrease ${\delta }_{+}$, ${\delta }_{-}$ as the algorithm evolves. This allows fine tuning of the size when the algorithm approaches convergence.

The number of candidates $n_{\mathrm{cand}}$ has more influence on the results but again this is visible especially for $n=256$.

Table 7. Average dictionary sizes given by ITC-ADL, for $n=256$, for several values of $n_{\mathrm{cand}}$.

	${\mathit{n}}_{\mathbf{cand}}=10$	15	20	25
$s=6$	263.86	258.34	257.30	256.80
$s=10$	288.18	286.68	280.66	274.84

shows the average dictionary sizes given by ITC-ADL for $n_{\mathrm{cand}}\in \{10,15,20,25\}$. It is clear that a larger number of candidates is beneficial; since this leads to a larger execution time, a compromise is necessary. This was our reason for taking $n_{\mathrm{cand}}=20$. However, for $n=128$, even $n_{\mathrm{cand}}=10$ seems sufficient, as the sizes (not shown here) are virtually the same for all considered $n_{\mathrm{cand}}$ values.

We turn now to the parameter c, the number of iterations between size changes. Intuitively, small values of c lead to better results, but with more computational effort.

Table 8. Average dictionary sizes given by ITC-ADL, for $n=256$, for several values of c.

	$\mathit{c}=3$	4	5	6	8	10
$s=6$	257.16	258.04	258.32	256.92	257.28	256.96
$s=10$	265.36	273.00	280.36	285.46	282.64	284.02

shows the results for $n=256$, for $c\in \{3,4,5,6,8,10\}$. The effect of c is visible for the more difficult problems. For small n and s, a larger c can give faster good results.

Finally, we partially justify our choices for the number of iterations. Generally, the rule is simple: more iterations lead to better results. However, there are many factors that influence the convergence speed and there are many random elements in the algorithm (the initial dictionary, the added atoms, the initial size and, of course, the noise). We illustrate in Figure 1 the evolution of the main variables, the current dictionary size (as given by the ITC) and RMSE (on the training data), for $n=128$, $s=10$ and 10 runs of ITC-ADL with random initial sizes. The values are shown for iterations that are multiple of c; the size is constant between them and the RMSE typically decreases. It is visible that, after 100 iterations, the size is close to the true one and the RMSE is at about its final (and almost optimal, as we have seen) value. Typically, the size becomes larger than the true one when the RMSE reaches the lowest value. Then, ITC-ADL gradually trims the dictionary almost without changing the RMSE. Thus, if our purpose is dictionary learning (and perfect recovery is not sought because there is actually no true dictionary), the number of iterations can be smaller than for dictionary recovery.

4.3. Experiments with Unknown Sparsity Level

Using the same setup as in Section 4.1, we run ITC-ADL with several values of s, not only the true one, and choose the sparsity level that gives the best ITC value. Otherwise, the algorithm is unchanged and estimates the dictionary size as usual.

Table 9. Average values of s (top) and n (bottom), when $n_{\mathrm{true}}=128$. Left: ERNML${}_1$. Right: EBIC.

SNR (dB)	$\mathit{s}=4$	6	8	10	12	SNR (dB)	$\mathit{s}=4$	6	8	10	12
10	4.00	5.10	6.90	7.88	8.24	10	3.98	4.64	5.00	6.00	8.00
	128.38	128.14	128.04	128.02	128.04		128.78	128.10	128.10	128.12	127.88
20	4.00	6.00	8.00	10.00	11.86	20	4.00	6.00	8.00	9.76	11.10
	128.14	128.02	128.12	128.16	128.00		128.28	128.02	128.10	128.14	128.16
30	4.02	6.00	8.06	10.18	12.16	30	4.04	6.00	8.08	10.12	12.30
	128.04	128.14	128.52	128.46	129.16		128.14	128.04	128.56	128.78	129.56
40	4.04	6.10	8.16	10.42	12.40	40	4.00	6.04	8.14	10.48	12.60
	128.22	128.72	129.36	132.66	132.52		128.12	128.66	130.18	130.34	132.76

presents the average values of the sparsity level and of the dictionary size given by the above procedure for $n_{\mathrm{true}}=128$.

The variance of the estimated s is very low: typically, only two values are obtained. For example, when $s=8$, for SNR = 10 dB, only values of 6 and 7 are obtained; when SNR=40 dB, only values of 8 and 9 are obtained; etc. One can see that the sparsity level estimations are rather accurate; the only deviation is at low SNR, when s is underestimated. The dictionary size is well estimated, in accordance with the previous results.

A competing algorithm is Adaptive ITKrM [9] (Matlab sources available at https://www.uibk.ac.at/mathematik/personal/schnass/code/adl.zip), reported to give very good results in DR problems. However, it seems that this algorithm needs more signals than ours. In our setup with $N=4000$, A-ITKrM gives very poor results.

Table 10. Results with A-ITKrM: average values of s (top), n (middle) and RMSE/RMSE$\mathrm{i}$ (bottom).

SNR (dB)	$\mathit{s}=4$	6	8	10	12
	3.00	3.00	3.00	3.00	3.00
10	128.00	127.85	126.40	123.55	123.20
	0.9906	0.9856	0.9545	0.9045	0.8995
	3.00	4.00	4.00	4.00	4.00
20	128.00	128.00	128.00	128.00	128.50
	0.9454	0.9113	0.8821	0.8544	0.8217
	3.00	4.00	4.00	4.00	4.00
30	128.00	128.00	128.00	128.00	128.55
	0.6906	0.5830	0.5186	0.4830	0.4564
	3.00	4.00	4.00	4.00	4.00
40	128.00	128.00	128.00	128.05	128.45
	0.2868	0.2247	0.2234	0.2462	0.3249

gives results for N = 20,000, where A-ITKrM becomes competitive. Due to time constraints, the results are averaged over only 20 runs. The table shows the obtained average sparsity level, dictionary size, and ratio RMSE/RMSEi, where RMSE is the error of our algorithm (using ERNML${}_1$) and RMSEi is the error of A-ITKrM. Since A-ITKrM severely underestimates s, we have computed RMSEi (and RMSE) using OMP with the true s. Even so, our algorithm finds a more accurate version of the dictionary. While ITKrM gives very good estimates of n and a good approximation of the dictionary, it seems unable to refine the dictionary as well as our algorithm.

In the current unoptimized implementations, the execution times of ITC-ADL and A-ITKrM are comparable, our algorithm being faster for small s and slower for large s. Limited trials with N = 50,000 suggest that the above remarks continue to stand true.

5. Conclusions

We present a dictionary learning algorithm that works for unknown dictionary size. Using specialized Information Theoretic Criteria, based on BIC and RNML, the size is adapted during the evolution of a standard DL algorithm (AK-SVD in our case). Experimental results show that the algorithm is able to discover the true size of the dictionary used to generate data and, somewhat surprisingly, can give better results than AK-SVD run with the true size.

There are several possible directions for future research. Testing ITC-ADL in various applications is a first aim; they can range from direct applications, such as denoising or missing data estimation (inpainting), to more complex ones, e.g. classification. We also plan to combine the ITC idea with other DL algorithms, not only AK-SVD, especially with algorithms for which the sparsity level is not the same for all signals. Another direction is to find ITC that are suited for other types of noise, not Gaussian as here, with the final purpose of obtaining a single tool that analyzes data, finds the appropriate sparse representation model and designs the optimal dictionary.