Mitigating the Drawbacks of the L₀ Norm and the Total Variation Norm

Abstract

In compressed sensing, it is believed that the L₀ norm minimization is the best way to enforce a sparse solution. However, the L₀ norm is difficult to implement in a gradient-based iterative image reconstruction algorithm. The total variation (TV) norm minimization is considered a proper substitute for the L₀ norm minimization. This paper points out that the TV norm is not powerful enough to enforce a piecewise-constant image. This paper uses the limited-angle tomography to illustrate the possibility of using the L₀ norm to encourage a piecewise-constant image. However, one of the drawbacks of the L₀ norm is that its derivative is zero almost everywhere, making a gradient-based algorithm useless. Our novel idea is to replace the zero value of the L₀ norm derivative with a zero-mean random variable. Computer simulations show that the proposed L₀ norm minimization outperforms the TV minimization. The novelty of this paper is the introduction of some randomness in the gradient of the objective function when the gradient is zero. The quantitative evaluations indicate the improvements of the proposed method in terms of the structural similarity (SSIM) and the peak signal-to-noise ratio (PSNR).

1. Introduction

The focus of this paper is on the L₀ norm minimization. If $x$ is a scalar, the L₀ norm of $x$ is defined as [1,2,3].

$${‖x‖}_{L_0}=\left\{\begin{array}{ccc}0& if& x=0,\\ 1& if& x\ne 0.\end{array}\right.$$

(1)

If $X$ is a vector or a matrix, the L₀ norm of $X$ is defined as

$${‖X‖}_{L_0}=\mathrm{The} \mathrm{total} \mathrm{count} \mathrm{of} \mathrm{the} \mathrm{nonzero} \mathrm{elements} \mathrm{in} X$$

(2)

Here we use the term ‘norm’ even though the L₀ norm is not a true norm in the mathematical sense. It is better to refer to the L₀ norm as the cardinality function or sparsity measure. A norm must have the positive homogeneity property

$$‖kx‖=k‖x‖,$$

(3)

for all $x$ in its domain and $k>0.$ Clearly, the L₀ norm does not satisfy the positive homogeneity property (3). In fact, according to (1), we have

$${‖kx‖}_{L_0}={‖x‖}_{L_0},$$

(4)

for all $x$ in its domain and $k>0.$

One important application of the L₀ norm is in compressed sensing, which was introduced in the first decade of this century [4,5,6,7,8,9,10]. The principle of compressed sensing is to exactly reconstruct a signal using a small number of samples if the signal is sparse. The problem can be described as a system of linear equations:

$$AX=P,$$

(5)

where $A$ is the system matrix, $P$ is the measurement vector, and $X$ is the unknown vector. In a compressed sensing problem, the number of measurements in $P$ is much smaller than the number of unknowns in $X$. In other words, the system matrix $A$ has more columns than rows. We call the solution X sparse if most of the elements in X are zero.

Another situation in compressed sensing is that the solution X of the system (5) is not sparse; however, a transformed version of X is sparse. If such a sparsification transformation operator is denoted by $\psi$, and the elements in vector Y

$$Y=\psi \left(X\right)$$

(6)

are dominated by zeros; then Y is sparse. The total count of non-zero elements in $Y$ is the L₀ norm of $Y$. In theory, the solution $X$ can be obtained by minimizing the following objective function F

$$F={‖AX-P‖}_{L_2}^2+\alpha \left|\right| \psi \left(X\right){\left|\right|}_{L_0},$$

(7)

where $\alpha$ is a tuning parameter.

Let us explain (5) and (7) further. Many real-word problems can be modeled as a system of linear equations (5), where the measurements are represented as a vector P and the object being measured is represented by a vector X. The measurements P usually contain noise. The vector X is a discretized version of the real-world object to be estimated. The system matrix A describes the first-order approximation of measurement physics. The matrix A is assumed to be known. In compressed sensing, the object is under-sampled. Thus, the size of P is smaller than the size of X, and the matrix A has more columns than rows.

The first term in (7), ${‖AX-P‖}_{L_2}^2$, is the data fidelity term. Minimizing this term is equivalent to solving (5). When the system (5) is under-determined, the solutions in (5) is not unique. The second term in (7), $\alpha \left|\right| \psi \left(X\right){\left|\right|}_{L_0}$, is the Bayesian term. Minimizing this term enforces $\psi \left(X\right)$ to be sparse or enforces X to be piecewise-constant.

Since the L₀ norm counts the total number of non-zero elements, it is difficult to use a gradient-based algorithm to minimize the objective function F defined in (7). In fact, minimizing the L₀ norm is an NP-hard problem [11,12,13,14,15], making it computationally infeasible for large-scale problems.

Many researchers attempted to minimize L₀ norm and found it to be difficult to deal with the L₀ norm directly [2,3,16,17,18,19,20,21,22,23,24,25]. One way is to approximate the L₀ norm by other norms [2,3,16,17,18,19]. Another way is to convert an L₀-norm minimization problem to an integer programming problem [20,21]. Still another way is to approximate the L₀-norm by a smooth function [22,23,24,25]. The L₀-norm minimization problem can also be converted to some sub problems by using some thresholds [26,27]. Robitzsch compared an L_p-norm method with an approximation of the L₀-norm method and showed that the L₀-norm method is slightly better [28].

The most popular method to replace the L₀-norm is the use of the total variation (TV) norm [29,30,31]. In the TV norm methods, the total variation (TV) norm of the unknown $X$ is used to replace the second term in (7). The TV norm minimization method is to use finite difference as the sparsification operator $\psi$ and use the L₁ norm to approximate the L₀ norm. If $X$ is piecewise constant, the finite difference version is sparse. The L₁ norm for a vector $X$ of n elements is given by

$${‖X‖}_{L_1}=\sum _{i=1}^n\left|x_i\right|.$$

(8)

Thus, the TV norm of a vector X of n elements is

$${‖X‖}_{TV}=\sum _{i=1}^{n-1}\left|x_i-x_{i-1}\right|.$$

(9)

Using the TV norm optimization, the objective function (7) for the vector case becomes

$$F={‖AX-P‖}_{L_2}^2+\alpha {‖X‖}_{TV}={‖AX-P‖}_{L_2}^2+\alpha \sum _{i=1}^{n-1}\left|x_i-x_{i-1}\right|.$$

(10)

The TV norm of a matrix is not a direct extension of (9) to a higher dimension. We have two different definitions of the TV norm for a two-dimensional matrix (that is, a two-dimensional image) based on the two different ways to define the finite difference [1]. One TV norm is referred to as the isotropic TV norm and the other TV norm is referred to as the anisotropic TV norm. We use double indices for each element of the matrix $X$.

The isotropic TV norm is defined as [1]

$$T{V}_{iso}\left(X\right)=\sum _{i,j}\sqrt{{\left(x_{i+1,j}-x_{i,j}\right)}^2+{(x_{i,j+1}-x_{i,j})}^2},$$

(11)

and the anisotropic TV norm is defined by [1]

$$T{V}_{aniso}\left(X\right)=\sum _{i,j}\left(\left|x_{i+1,j}-x_{i,j}\right|+\left|x_{i,j+1}-x_{i,j}\right|\right).$$

(12)

For the two-dimensional image case, the objective function (10) can then be written as

$$F={‖AX-P‖}_{L_2}^2+\alpha T{V}_{iso}\left(X\right)={‖AX-P‖}_{L_2}^2+\alpha \sum _{i,j}\sqrt{{\left(x_{i+1,j}-x_{i,j}\right)}^2+{(x_{i,j+1}-x_{i,j})}^2}$$

(13)

and

$$F={‖AX-P‖}_{L_2}^2+\alpha T{V}_{aniso}\left(X\right)={‖AX-P‖}_{L_2}^2+\alpha \sum _{i,j}\left(\left|x_{i+1,j}-x_{i,j}\right|+\left|x_{i,j+1}-x_{i,j}\right|\right),$$

(14)

respectively.

The justification of using L₁ norm to approximate the L₀ norm is that they both prefer a solution with more zeros. Figure 1 shows an exemplary solution line $x_2=m{x}_1+b$; any point $\left(x_1, x_2\right)$ on this line is a solution to (5). This solution line may intersect the coordinate axes at two points: $p_1$ and $p_2$, with $p_1=\left(-b/m,0\right)$ and $p_2=\left(0,b\right)$, respectively. The L₀ minimization method will select either $p_1$ or $p_2$ to be the solution, because $|\left|p_1\right|{|}_{L_0}=|\left|p_2\right|{|}_{L_0}= 1$ and other points on the line have a larger L₀ norm of $|\left|\left(x_1, x_2\right)\right|{|}_{L_0}=2$. As for the L₁ norm, $|\left|p_1\right|{|}_{L_1}=|b/m|$, and $|\left|p_2\right|{|}_{L_1}=|b|$. The L₁ minimization method will select $p_1$ if $\left|m\right|>1$ or $p_2$ if $\left|m\right|<1$ to be the solution. The L₁ minimization method will select either $p_1$ or $p_2$ to be the solution if $\left|m\right|=1$. Therefore, L₀ minimization and L₁ minimization may select the same solutions. The justification illustrated in our toy example is unrealistic because we do not have the luxury to obtain this solution line to search along.

In the objective function (10), the first term involving the L₂ norm is dominating. Thus, when the parameter α is small, the optimal solution of (10) may not make $\psi \left(X\right)$ sparse.

A drawback of the TV norm is that it cannot tell the difference between a smooth monotonic transition and a sharp monotonic transition (See Figure 2).

Therefore, using the L₀ norm and objective functions (7) may work better than objective functions (13) and (14), in the sense that the L₀ norm can tell the difference for the two curves shown in Figure 2. A drawback of the finite difference using the L₀ method is that we do not have an effective and efficient way to minimize the L₀ norm. The main goal of the paper is to directly minimize the objective function (7) with the use of the L₀ norm and find an innovative way to handle the L₀ norm.

2. Methods

In Section 1, we analyzed the drawbacks of the TV norm and the L₀ norms. In this section, we will develop a method to replace the TV norm by the L₀ norm for a practical iterative algorithm.

We notice from the traditional definition of the L₀ norm (see Figure 3 Top)

$${‖x‖}_{L_0}=\left\{\begin{array}{ccc}0& if& x=0\\ 1& if& x\ne 0.\end{array}\right.$$

that the “if $x=0$” statement is hardly true in a practical computer algorithm, because in practice a very small value (for example, an image pixel value of 0.000000000001) can be treated as zero. For the scalar case, it is reasonable to replace the definition (1) by (15) below

$$f_0(x)=\left\{\begin{array}{ccc}{\displaystyle \frac{\left|x\right|}{c}}& if& \left|x\right|\le c\\ 1& if& \left|x\right|>c.\end{array}\right.$$

(15)

for a chosen $c>0$. In (15), c is a tuning parameter, determined by trial-and-error. This modification is illustrated in Figure 3 (Middle). The piecewise linear function $f_0\left(x\right)$ defined in (5) is continuous and is a good approximation of ${‖x‖}_{L_0}$ when $c\to 0$.

In fact, we can have many versions of the L₀ norm. For example, the version shown in Figure 3 (Bottom) is a smooth function; the derivative of the curve exists everywhere. One such smooth function is

$$g_0\left(x\right)=1-e^{-x^2/c}$$

for a small $c>0$.

The requirement to be differentiable everywhere is not needed. We believe that a gradient-based optimization algorithm only requires the existence of the left and right derivatives everywhere. Let us consider a toy example of

$$f\left(x\right)=\left|x\right|$$

(16)

We want to find the minimum of the function defined in (16). It is obvious that the solution is

$$x=0$$

(17)

We notice that the function $f\left(x\right)=\left|x\right|$ is not differentiable at $x=0$. At $x=0$, the left derivative and right derivative of $f\left(x\right)$ are $f_{left}^{\prime }\left(0\right)=-1$ and $f_{right}^{\prime }\left(0\right)=1$, respectively. A gradient-based optimization algorithm can be crafted as

$$x^{next}=x^{current}-\lambda {\displaystyle \frac{f_{left}^{\prime }\left(x^{current}\right)+f_{right}^{\prime }\left(x^{current}\right)}{2}}$$

(18)

where the parameter $\lambda >0$ controls the step size of the iterative optimization algorithm. Algorithm (18) is nothing but the commonly used gradient descent algorithm, except that the gradient is replaced by the average of the left and right gradients. Therefore, our definition of the L₀ norm is user-friendly in gradient-based iterative optimization algorithms. Using a smooth function $g_0\left(x\right)$ shown in Figure 3 (Bottom), to approximate the L₀ norm is not necessary.

To extend the revised L₀ definition from the scalar case (15) to a matrix X, whose elements are denoted as $x_{i,j}$, we have

$$f_0(X)=\sum _{i,j}{f}_0\left(x_{i,j}\right)$$

(19)

where $f_0\left(x_{i,j}\right)$ is defined in (15). The definition of (19) is still not effective in a gradient-based iterative optimization algorithm. The partial derivative of $f_0\left(X\right)$ with respect to $x_{i,j}$ is calculated as

$${\displaystyle \frac{\mathsf{\partial }{f}_0\left(X\right)}{\mathsf{\partial }{x}_{i,j}}}={\displaystyle \frac{d{f}_0\left(X\right)}{d{x}_{i,j}}}=\left\{\begin{array}{ccc}{\displaystyle \frac{ 1}{c}}sgn\left(x_{i,j}\right)& if& \left|x_{i,j}\right|\le c\\ 0& if& \left|x_{i,j}\right|>c.\end{array}\right.$$

(20)

Let

$$\mathrm{s}\mathrm{t}\mathrm{e}\mathrm{p}(x)=\left\{\begin{array}{ccc}1& if& x\ge 0\\ 0& if& x<0.\end{array}\right.$$

(21)

Then (20) can be expressed as

$${\displaystyle \frac{\mathsf{\partial }{f}_0\left(X\right)}{\mathsf{\partial }{x}_{i,j}}}={\displaystyle \frac{d{f}_0\left(X\right)}{d{x}_{i,j}}}={\displaystyle \frac{ 1}{c}}\times sgn\left(x_{i,j}\right)\times step(c-\left|x_{i,j}\right|).$$

(22)

The plot of (22) is shown in Figure 4. It is observed that when $\left|x\right|>c$ the derivative is 0. In other words, there will be no update action in the iterative algorithm for most of the image pixels. This makes the optimization algorithm almost inactive and ineffective.

In order to obtain more actions in the iterative algorithm, our next innovation is to replace the zero values in $d{f}_0\left(x\right)/dx$ by small zero-mean random variables. Thus, Figure 4 becomes Figure 5. Since the optimization algorithms need the expression of $d{f}_0\left(x\right)/dx$ and do not care if there is a corresponding expression of $f_0\left(x\right)$, we do not bother to investigate what the definition of $f_0\left(x\right)$ is corresponding to the $d{f}_0\left(x\right)/dx$ shown in Figure 5.

The mathematical expression for the revised derivative shown in Figure 5 is given as

$${\displaystyle \frac{\mathsf{\partial }{f}_0\left(X\right)}{\mathsf{\partial }{x}_{i,j}}}={\displaystyle \frac{d{f}_0\left(X\right)}{d{x}_{i,j}}}={\displaystyle \frac{ 1}{c}}\times sgn(x_{i,j})\times step(c-\left|x_{i,j}\right|)+rand\times step(\left|x_{i,j}\right|-c),$$

(23)

where $rand$ is a small zero-mean random variable. The tuning parameter c is selected by trail-and-error, depending on the application.

We remind the reader that our images are not sparse, but piecewise constant. We need a sparsification transformation operator $\psi$ to convert a piecewise-constant image to a sparse image. We chose the finite difference operator as the sparsification transformation operator $\psi$. Just like cases of isotropic TV (11) and anisotropic (12) TV definitions, we can have two definitions of ${\mathsf{\partial }‖X‖}_{L_0}/\mathsf{\partial }{x}_{i,j}$, one being the isotropic derivative version and the other being the anisotropic version.

The isotropic version is defined as

$$\begin{array}{c}{\displaystyle \frac{{\mathsf{\partial }‖X‖}_{L_0}}{\mathsf{\partial }{x}_{i,j}}}={\displaystyle \frac{{\mathsf{\partial }{\sum }_{n,m}‖\sqrt{{\left(x_{n+1,m}-x_{n,m}\right)}^2+{(x_{n,m+1}-x_{n,m})}^2}‖}_{L_0}}{\mathsf{\partial }{x}_{i,j}}}\\ ={\displaystyle \sum _{n,m}}{\displaystyle \frac{\mathsf{\partial }{‖\sqrt{{\left(x_{n+1,m}-x_{n,m}\right)}^2+{(x_{n,m+1}-x_{n,m})}^2}‖}_{L_0}}{\mathsf{\partial }\sqrt{{\left(x_{n+1,m}-x_{n,m}\right)}^2+{(x_{n,m+1}-x_{n,m})}^2}}}{\displaystyle \frac{\mathsf{\partial }\sqrt{{\left(x_{n+1,m}-x_{n,m}\right)}^2+{(x_{n,m+1}-x_{n,m})}^2}}{\mathsf{\partial }{x}_{i,j}}}\\ ={\displaystyle \sum _{n,m}}g(n,m)\times {\displaystyle \frac{\mathsf{\partial }\sqrt{{\left(x_{n+1,m}-x_{n,m}\right)}^2+{(x_{n,m+1}-x_{n,m})}^2}}{\mathsf{\partial }{x}_{i,j}}}\\ =g(i-1,j) {\displaystyle \frac{\mathsf{\partial }\sqrt{{\left(x_{i,j}-x_{i-1,j}\right)}^2+{(x_{i-1,j+1}-x_{i-1,j})}^2}}{\mathsf{\partial }{x}_{i,j}}}\\ +g(i,j) {\displaystyle \frac{\mathsf{\partial }\sqrt{{\left(x_{i+1,j}-x_{i,j}\right)}^2+{(x_{i,j+1}-x_{i,j})}^2}}{\mathsf{\partial }{x}_{i,j}}}\\ +g(i,j-1) {\displaystyle \frac{\mathsf{\partial }\sqrt{{\left(x_{i+1,j-1}-x_{i,j-1}\right)}^2+{(x_{i,j}-x_{i,j-1})}^2}}{\mathsf{\partial }{x}_{i,j}}}\\ =g(i-1,j){\displaystyle \frac{x_{i,j}-x_{i-1,j}}{\sqrt{{\left(x_{i,j}-x_{i-1,j}\right)}^2+{(x_{i-1,j+1}-x_{i-1,j})}^2}}}\\ +g(i,j-1){\displaystyle \frac{x_{i,j}-x_{i,j-1}}{\sqrt{{\left(x_{i+1,j-1}-x_{i,j-1}\right)}^2+{(x_{i,j}-x_{i,j-1})}^2}}}\\ -g(i,j){\displaystyle \frac{x_{i+1,j}+x_{i,j+1}-2x_{i,j}}{\sqrt{{\left(x_{i+1,j}-x_{i,j}\right)}^2+{(x_{i,j+1}-x_{i,j})}^2}}}\\ =g(i-1,j){\displaystyle \frac{x_{i,j}-x_{i-1,j}}{u(i-1,j)}}+g(i,j-1){\displaystyle \frac{x_{i,j}-x_{i,j-1}}{u(i,j-1)}}-g(i,j){\displaystyle \frac{x_{i+1,j}+x_{i,j+1}-2x_{i,j}}{u(i,j)}}\end{array}$$

(24)

with

$$u(n,m)=\sqrt{{\left(x_{n+1,m}-x_{n,m}\right)}^2+{(x_{n,m+1}-x_{n,m})}^2}$$

(25)

and

$$\begin{array}{c}g(n,m)={\displaystyle \frac{\mathsf{\partial }{‖\sqrt{{\left(x_{n+1,m}-x_{n,m}\right)}^2+{(x_{n,m+1}-x_{n,m})}^2}‖}_{L_0}}{\mathsf{\partial }\sqrt{{\left(x_{n+1,m}-x_{n,m}\right)}^2+{(x_{n,m+1}-x_{n,m})}^2}}}\\ ={\displaystyle \frac{ 1}{c}}\times step(c-\sqrt{{\left(x_{n+1,m}-x_{n,m}\right)}^2+{(x_{n,m+1}-x_{n,m})}^2})\\ +rand\times step(\sqrt{{\left(x_{n+1,m}-x_{n,m}\right)}^2+{(x_{n,m+1}-x_{n,m})}^2}-c)\\ ={\displaystyle \frac{ 1}{c}}\times step(c-u(n,m))+rand\times step(u(n,m)-c).\end{array}$$

(26)

The anisotropic version is defined by

$${\displaystyle \frac{{\mathsf{\partial }‖X‖}_{L_0}}{\mathsf{\partial }{x}_{i,j}}}={\displaystyle \frac{\mathsf{\partial }{\sum }_{n,m}\left({‖x_{n+1,m}-x_{n,m}‖}_{L_0}+{‖x_{n,m+1}-x_{n,m}‖}_{L_0}\right)}{\mathsf{\partial }{x}_{i,j}}}$$

$$={\displaystyle \frac{\mathsf{\partial }\left({‖x_{i,j}-x_{i-1,j}‖}_{L_0}+{‖x_{i-1,j+1}-x_{i-1,j}‖}_{L_0}\right)}{\mathsf{\partial }{x}_{i,j}}}$$

$$+{\displaystyle \frac{\mathsf{\partial }\left({‖x_{i+1,j}-x_{i,j}‖}_{L_0}+{‖x_{i,j+1}-x_{i,j}‖}_{L_0}\right)}{\mathsf{\partial }{x}_{i,j}}}$$

$$+{\displaystyle \frac{\mathsf{\partial }\left({‖x_{i+1,j-1}-x_{i,j-1}‖}_{L_0}+{‖x_{i,j}-x_{i,j-1}‖}_{L_0}\right)}{\mathsf{\partial }{x}_{i,j}}}$$

$$={\displaystyle \frac{\mathsf{\partial }\left({‖x_{i,j}-x_{i-1,j}‖}_{L_0}\right)}{\mathsf{\partial }{x}_{i,j}}}+{\displaystyle \frac{\mathsf{\partial }\left({‖x_{i+1,j}-x_{i,j}‖}_{L_0}\right)}{\mathsf{\partial }{x}_{i,j}}}+{\displaystyle \frac{\mathsf{\partial }\left({‖x_{i,j+1}-x_{i,j}‖}_{L_0}\right)}{\mathsf{\partial }{x}_{i,j}}}+{\displaystyle \frac{\mathsf{\partial }\left({‖x_{i,j}-x_{i,j-1}‖}_{L_0}\right)}{\mathsf{\partial }{x}_{i,j}}}$$

(27)

$$={\displaystyle \frac{ 1}{c}}\times sgn(x_{i,j}-x_{i-1,j})\times step(c-\left|x_{i,j}-x_{i-1,j}\right|)+ran{d}_1\times step(\left|x_{i,j}-x_{i-1,j}\right|-c)$$

$$+{\displaystyle \frac{ 1}{c}}\times sgn(x_{i+1,j}-x_{i,j})\times step(c-\left|x_{i+1,j}-x_{i,j}\right|)+ran{d}_2\times step(\left|x_{i+1,j}-x_{i,j}\right|-c)$$

$$+{\displaystyle \frac{ 1}{c}}\times sgn(x_{i,j+1}-x_{i,j})\times step(c-\left|x_{i,j+1}-x_{i,j}\right|)+ran{d}_3\times step(\left|x_{i,j+1}-x_{i,j}\right|-c)$$

$$+{\displaystyle \frac{ 1}{c}}\times sgn\left(x_{i,j}-x_{i,j-1}\right)\times step\left(c-\left|x_{i,j}-x_{i,j-1}\right|\right)+ran{d}_4\times step\left(\left|x_{i,j}-x_{i,j-1}\right|-c\right)$$

If a gradient based iterative algorithm is used to minimize an objective function, the algorithm does not update the image pixel $x_{i,j}$ when the partial derivative of the objective function with respect to $x_{i,j}$ is zero. More often than not, the L₀ Bayesian term is ‘silent’ and has little contribution to find a piecewise constant solution. If we replace zero with zero-mean random variables, we actually disturb the algorithm so that it is not ‘silent.’

This strategy is similar to the simulated annealing algorithm, which generates random solutions and gradually rejects less optimal solutions [32]. However, there is no theoretical guarantee that random solutions are better solutions.

By replacing zero by zero-mean random variables does not change the fact that the L₀ defined norm is not convex. To obtain the global minimum, one must do exhaustive search for the entire solution space. In other words, L₀ minimization even with the current modification is still NP-hard. The proposed algorithm is gradient based, does not do exhaustive search, and usually does not reach the global minimum.

A drawback of the objective function (7) is the difficulty in selecting the control parameter $\alpha$. Instead of minimizing the objective function (7) directly, we propose to alternatively minimize the data fidelity term ${‖AX-P‖}_{L_2}^2$ and the Bayesian term $\left|\right| \psi \left(X\right){\left|\right|}_{L_0}$. In this way, the value of tuning parameter α is no longer important. In other words, we use an iterative Projection onto Convex Sets (POCS) algorithm.

There are many algorithms to minimize the first term (i.e., the data fidelity term). We chose the maximum likelihood expectation maximization (MLEM) algorithm in our implementation [33,34,35,36,37]. On the other hand, the Bayesian term is minimized by a gradient descent algorithm. For an image reconstruction task, at the pixel $x_{i,j}$, the gradient is given by (27).

Now, we use a different way to explain the algorithm to enforce the L₀ norm of the gradient image. For an image reconstruction task, we want to update the pixel $x_{i,j}$, as shown in Figure 6. The horizontal and vertical gradients of pixel $x_{i,j}$ are $x_{i,j}-x_{i-1,j}$, $x_{i,j}-x_{i+1,j}$, and $x_{i,j}-x_{i,j-1}$, $x_{i,j}-x_{i,j+1}$, correspondingly. We want to minimize the L₀ norms of these four differences. The derivative of these four L₀ norms have this common expression:

$$D(x_{i,j}, x_{neighbor}) = Derivative=a\times sgn\left(x_{i,j}-x_{neighbor}\right)$$

(28)

where

$$a=\left\{\begin{array}{ccc}1/c,& when& \left|x_{i,j}-x_{neighbor}\right|<c,\\ rand,& when& \left|x_{i,j}-x_{neighbor}\right|\ge c.\end{array}\right.$$

(29)

A gradient descent algorithm to update $x_{i,j}$ is given as

$$x_{i,j}^{next}=x_{i,j}-\mu \left[D\right(x_{i,j},x_{i,j-1})+D(x_{i,j},x_{i,j+1})+D(x_{i,j},x_{i,j-1})+D(x_{i,j},x_{i,j+1}\left)\right]$$

(30)

Here, the zero-mean random noise, rand, is uniformly distributed in $[-1, 1]$. The tuning parameter $c$ was chosen as 0.1 by trial-and-error in our implementation.

In our computer simulations, we used 100,000 iterations of the POCS algorithm. At each POCS iteration, we first ran 10 iterations of the MLEM algorithm to minimize the data fidelity term and then we ran 10 iterations of the gradient descent algorithm to minimize the L₀ term with a step size of 0.1.

We summarize the main steps in the development of the proposed algorithm as follows. We started with the original L₀ definition (1). Then, this expression was replaced by a piecewise-linear continuous approximation (15). Next, the partial derivative (22) was replaced by (23). The proposed POCS algorithm is summarized as a flowchart in Figure 7.

3. Results

We applied the proposed POCS algorithm to a limited-angle two-dimensional image reconstruction problem. The computer-generated phantoms were considered. The first phantom had a large, uniform disk as the background and 13 smaller, uniform squares and disks. The sizes, locations, and intensities for each square and disk are listed in

Table 1. Parameters for the first phantom.

Type	Center x	Center y	Diameter or Side Length	Rotation Angle	Density
Circle	0	0	230.40	0	0.5
Square	89.60	0	25.60	0	1.0
Square	0	89.60	23.04	0	0.5
Square	−89.60	0	20.48	0	1.0
Square	0	−89.60	17.92	0	1.0
Square	64.00	64.00	15.36	0	0.5
Square	−64.00	−64.00	12.80	0	1.0
Square	−64.00	64.00	10.24	0	1.0
Square	64.00	−64.00	7.68	0	0.5
Square	0	0	5.12	0	1.0
Circle	44.80	0	23.04	0	1.0
Circle	0	44.80	17.92	0	0.5
Circle	−44.80	0	12.80	0	1.0
Circle	0	−44.80	7.68	0	0.5

. The second phantom was the famous Shepp–Logan head phantom [38]. No noise was added to the phantom projection data.

In the computer simulations, the parallel-beam imaging geometry was considered. The scanning angle for the first phantom was 40°, and for the second phantom was 90°. The image size was 256 × 256. For these two phantom studies, three image reconstruction algorithms were compared in Figure 8 and Figure 9, respectively: The well-known MLEM algorithm [34], the MLEM-TV algorithm [33], and the proposed POCS revised L₀-norm minimization algorithm.

The iteration number was 400 in the POCS algorithm. Within each iteration, there were 10 iterations of the MLEM algorithm for data fidelity enforcement and 10 iterations of the gradient decent algorithm for piecewise-constant enforcement.

The only tuning parameter in the MLEM algorithm is the number of iterations. However, the gradient descent algorithm has two tuning parameters: the number of iterations and the step size. The step size was chosen as 0.0001 in the first phantom reconstruction and was 0.00001 in the second phantom reconstruction.

The MLEM reconstruction has the most limited-angle artifacts, and the shapes of the small objects are not well defined. The TV reconstruction slightly improves the boundaries of the small objects in the image. The most significant improvement is achieved by the proposed POCS L₀ minimization algorithm.

Table 2. Quantitative evaluation results for the first phantom.

Method	PSNR	SSIM
MLEM (B)	9.3822	0.4001
TV (C)	13.4989	0.6341
Proposed 1 (D1)	14.6675	0.6761
Proposed 2 (D2)	17.0831	0.7144

and

Table 3. Quantitative evaluation results for the second phantom.

Method	PSNR	SSIM
MLEM (B)	17.2455	0.7067
TV (C)	17.2679	0.7300
Proposed (D)	22.0501	0.8547

show the quantitative evaluation results with the structural similarity (SSIM) [39] and the peak signal-to-noise ratio (PSNR) [40] for the two phantom studies, respectively. An SSIM value closer to 1 indicates better image quality. A greater PSNR value indicates better image quality.

We observe that the reconstruction artifacts depend heavily on the size and the orientation of an object. A larger object tends to have more severe artifacts. If there are many smaller objects inside a larger object, the artifacts from each smaller object interact. Therefore, the distance between objects affects the overall artifacts. In the first phantom, the small objects are isolated from each other. In the second phantom, the small objects are close to each other. The second phantom is more difficult to reconstruct than the first phantom.

4. Conclusions

The TV norm has a drawback that it cannot distinguish a smooth function and a piecewise-constant function. The TV Bayesian objective function may not be effective in promoting a piecewise constant solution. The L₀ norm, on the other hand, is difficult to implement in a gradient-based optimization algorithm. This paper aims to address these drawbacks.

The difficulty of using the L₀ norm in an optimization algorithm is well known. Remedies have been proposed the tested by many researchers. The efforts can be classified into two categories: approximating the L₀ norm by a different norm and approximating the L₀ norm itself by a continuous functional. The well-known TV-norm optimization belongs to the first category. Our paper belongs to the second category. A unique feature of our method is in the region where the signal is not zero. In this region, the traditional L₀ norm has ${\displaystyle \frac{{\mathsf{\partial }‖X‖}_{L_0}}{\mathsf{\partial }{x}_{i,j}}}=0$. We replace this 0 with a zero-mean random variable. There are many methods that combine L₀, L_1, and TV models. The unique feature of our method is the introduction of randomness in the derivative.

The L₀ norm is not convex. The gradient-based algorithm usually does not converge to the global minimum. We introduce a zero-mean random perturbation to the algorithm; this random perturbation gives the algorithm a chance to ‘jump out’ from a local minimum to another local minimum with a smaller objective function value.

In this paper, we replace the Bayesian algorithm with a POCS algorithm and revise the derivative of the L₀ norm so that it does not have a constant zero in most cases. As an application to limited-angle tomography, the proposed algorithm outperforms the MLEM-TV algorithm when the scanning angular range is small.

It is difficult to compare any two algorithms in general, because the performance of the algorithms is application-dependent. As shown in our two phantom studies, different applications may require different minimal scanning angular ranges.