Local Contrast-Based Pixel Ordering for Exact Histogram Specification

Abstract

Histogram equalization is one of the basic image processing tasks for contrast enhancement, and its generalized version is histogram specification, which accepts arbitrary shapes of target histograms including uniform distributions for histogram equalization. It is well known that strictly ordered pixels in an image can be voted to any target histogram to achieve exact histogram specification. This paper proposes a method for ordering pixels in an image on the basis of the local contrast of each pixel, where a Gaussian filter without approximation is used to avoid the duplication of pixel values that disturbs the strict pixel ordering. The main idea of the proposed method is that the problem of pixel ordering is divided into small subproblems which can be solved separately, and then the results are merged into one sequence of all ordered pixels. Moreover, the proposed method is extended from grayscale images to color ones in a consistent manner. Experimental results show that the state-of-the-art histogram specification method occasionally produces false patterns, which are alleviated by the proposed method. Those results demonstrate the effectiveness of the proposed method for exact histogram specification.

1. Introduction

Histograms represent the distribution of pixel values in images, and can offer various image statistics [1] to us, i.e., histograms have useful information for image processing applications such as image enhancement, compression, and segmentation [2,3,4]. Therefore, histograms are the basis for numerous spatial domain processing techniques [5], which include intensity transformation as a special case where the smallest neighborhood of size $1\times 1$ is used. Histogram equalization, which theoretically transforms an input image into the corresponding output image having a uniform histogram, is one of the intensity transformation techniques for contrast enhancement [5]. More generally, the method for generating an output image that has a specified target histogram is called histogram specification or histogram matching [5], in which histogram equalization is included as a special case [6]. It is pointed out that histogram equalization may cause over enhancement [7]. In such situations, histogram specification will be a good candidate for substitute methods because it accepts any shape of histograms. In this paper, although we would like to use a Gaussian distribution constantly for the target histogram in histogram specification to simplify our explanation, other choices such as bimodal or multimodal distribution may work well.

For digital images that are composed of pixels with quantized values, it is usual that the number of pixels in an image is greater than the number of possible quantized pixel values. Therefore, there are a large number of pixels that share the same quantized pixel value in a digital image, which means that exact histogram specification for digital images is an ill-posed problem [8]. For the effective utilization of all possible quantized pixel values and the suppression of the occurrence of false contours, a number of research works have been conducted for more than four decades since Hall published his work on discrete distribution transformation [9]. In addition to these technical needs, it is our genuine curiosity to make and see exactly histogram-equalized or specified images because the solution of the ill-posed problem is not unique. A promising approach to exact histogram specification is to obtain a meaningful strict ordering of all pixels in an image. Once we obtain the ordering of pixels, we can assign a limited number of pixel values to the ordered pixels so that the resultant histogram coincides with a specified target histogram.

The simplest way of pixel ordering is random ordering [10]. However, the random ordering is not a meaningful ordering. Coltuc et al. [11] described the process of exact histogram specification when a strict ordering is given, and proposed an ordering method, which is called the local mean method (LM), based on a filter bank of multiple moving average filters. Wan and Shi [12] proposed a wavelet-based exact pixel ordering algorithm (WA) for both exact histogram specification and image enhancement, which takes into account not only local mean intensity values, but also local edge information. Nikolova and Steidl [13] proposed a fast ordering algorithm for exact histogram specification based on a variational approach, which is equivalent to an iterative nonlinear filtering, and demonstrated that their variational ordering method outperforms both LM [11] and WA [12]. They also applied their algorithm to hue and range preserving enhancement of color images [14], where the histogram of an intensity image is exactly equalized or specified, and then an affine transform is applied to each color in a hue and range preserving manner.

In this paper, we propose a local contrast-based pixel ordering method for exact histogram specification, which computes the local contrast of each pixel by using a Gaussian filter [15] without approximation, and compare the proposed method with the state-of-the-art method by Nikolova and Steidl [13]. Furthermore, the proposed method for grayscale images is extended to that for color images in a consistent manner. Experimental results demonstrate that Nikolova and Steidl’s method can produce false patterns in flat areas of images, which are alleviated by the proposed method. For color images, it is experimentally shown that the proposed method outputs exactly histogram-equalized or specified images with closer hue to original images than that obtained by a separable method that utilizes Nikolova and Steidl’s method for exact histogram equalization and specification of each color channel. These results demonstrate the effectiveness of the proposed method compared with the state-of-the-art method.

The rest of this paper is organized as follows: Section 2 provides a brief review of some related works. Section 3 briefly summarizes conventional histogram equalization. Section 4 summarizes conventional histogram specification and Nikolova and Steidl’s exact histogram specification [13]. Section 5 proposes a local contrast-based exact histogram specification method, which is extended to that for color images. Section 6 shows experimental results, where the proposed method is compared with the state-of-the-art methods by Nikolova and Steidl [13] and Ramos et al. [16]. Finally, Section 7 concludes this paper.

2. Related Work

In this section, we review recent related works on histogram equalization and specification. Recently, Pallavi et al. [17] reviewed state-of-the-art techniques for image enhancement, in which the techniques of histogram equalization are also included such as contrast limited adaptive histogram equalization (CLAHE) [18] and brightness preserving bi-histogram equalization (BBHE) [19]. Trongtirakul and Agaian [20] proposed a weighted histogram equalization using entropy of probability density function, which outperformed related methods including a low-rank regularized retinex model (LR3M) [21]. More recently, Zhang et al. [22] proposed an unsupervised low-light image enhancement method via a histogram equalization prior (HEP) based on the observation that the feature maps of histogram equalization enhanced image and the ground truth are similar.

Hussain et al. [23] proposed a locally transformed histogram-based technique for dark image enhancement, which does not get affected from the over-enhancement problem. Balado [24] examined an optimum exact histogram specification problem and the inverse problem, and presented their closed-form performance analyses. Ramos et al. [16] proposed two algorithms for histogram specification and quantile transformation of data without local information. As one of the state-of-the-art algorithms for histogram specification, we implemented the first algorithm of Ramos et al. in this paper as described below. However, their algorithm does not produce an output image having the histogram being exactly identical to the target one.

The above review of recent related works draws a conclusion that Nikolova and Steidl’s algorithm [13] is still one of the state-of-the-art algorithms for exact histogram specification.

3. Histogram Equalization

Let $F=\left[f_{ij}\right]$ be a grayscale image for $i=1,2,\dots ,m$ and $j=1,2,\dots ,n$, where $f_{ij}\in \{0,1,\dots ,L\}$ denotes the pixel value at the position $(i,j)$ on the image plane with m rows and n columns. The maximum pixel value L is typically given by $L=2^8-1=255$ in an 8-bit grayscale. Then, the histogram of the pixel values in F is given by $\mathit{h}=[h_0,h_1,\dots ,h_L]$ whose kth element is computed by

$$\begin{array}{c}\hfill h_k=\sum _{i=1}^m\sum _{j=1}^n{\delta }_{k,f_{ij}}\end{array}$$

(1)

for $k=0,1,\dots ,L,$ where ${\delta }_{k,f_{ij}}$ denotes the Kronecker delta function defined by ${\delta }_{k,f_{ij}}=0$ for $k\ne f_{ij}$, and 1 for $k=f_{ij}$. Note that the histogram $\mathit{h}$ is not normalized, i.e., ${\sum }_{k=0}^L{h}_k=mn\ne 1$. The corresponding cumulative histogram is given by $\mathit{H}=[H_0,H_1,\dots ,H_L]$ whose lth element is defined by

$$\begin{array}{c}\hfill H_l=\sum _{k=0}^l{h}_k\end{array}$$

(2)

for $l=0,1,\dots ,L,$ and recursively computed by $H_l=H_{l-1}+h_l$ for $l=1,2,\dots ,L$ with $H_0=h_0$. Histogram equalization converts a pixel value $f_{ij}$ into

$$\begin{array}{c}\hfill f_{ij}^{HE}={\varphi }_{\mathit{h}}^{HE}\left(f_{ij}\right):=\mathrm{round}\left(L\frac{H_{f_{ij}}-H_{f_{min}}}{H_{f_{max}}-H_{f_{min}}}\right),\end{array}$$

(3)

where $f_{min}={min}_{i,j}\left\{f_{ij}\right\},f_{max}={max}_{i,j}\left\{f_{ij}\right\}$, and the ‘round’ operator rounds a given argument toward the nearest integer, and ${\varphi }_{\mathit{h}}^{HE}$ denotes the function of converting pixel values in histogram equalization.

Figure 1 shows an example of histogram equalization, where an original image in Figure 1a is converted into the image in Figure 1d, and their histograms and cumulative ones are shown in Figure 1b,c and Figure 1e,f, respectively, where the horizontal axis of each graphic denotes the pixel value ranging from 0 to 255, and the vertical axis denotes the (cumulative) number of pixels for each (cumulative) histogram. As shown in Figure 1e, the resultant histogram is not equalized actually, but the cumulative one is linearized as shown in Figure 1f. That is, to be more precise, the conventional histogram equalization is a cumulative histogram linearization.

4. Histogram Specification

In this section, we summarize conventional histogram specification, which is not an exact method, and then briefly summarize the state-of-the-art method for exact histogram specification by Nikolova and Steidl [13]. After that, we propose a local contrast-based method for exact histogram specification, whose objects to be processed are extended from grayscale images to color images.

4.1. Conventional Histogram Specification

Let $\widehat{\mathit{h}}=[{\widehat{h}}_0,{\widehat{h}}_1,\dots ,{\widehat{h}}_L]$ be a target histogram into which we want to convert the histogram $\mathit{h}$ of the original image F, where we assume that ${\sum }_{k=0}^L{\widehat{h}}_k={\sum }_{k=0}^L{h}_k$ for convenience. Then, we can compute the cumulative histogram $\widehat{\mathit{H}}=[{\widehat{H}}_0,{\widehat{H}}_1,\dots ,{\widehat{H}}_L]$ from $\widehat{\mathit{h}}$ in the same manner as above. Theoretically, we can describe the relationship between the target histogram $\widehat{\mathit{h}}$ and the original one $\mathit{h}$ as

$$\begin{array}{c}\hfill f_{ij}^{HS}={\left({\varphi }_{\widehat{\mathit{h}}}^{HE}\right)}^{-1}\left(f_{ij}^{HE}\right)={\left({\varphi }_{\widehat{\mathit{h}}}^{HE}\right)}^{-1}\left({\varphi }_{\mathit{h}}^{HE}\left(f_{ij}\right)\right),\end{array}$$

(4)

where $f_{ij}^{HS}$ denotes the output pixel value of histogram specification corresponding to an input pixel value $f_{ij}$, and ${\left({\varphi }_{\widehat{\mathit{h}}}^{HE}\right)}^{-1}$ denotes the inverse function of ${\varphi }_{\widehat{\mathit{h}}}^{HE}$ which equalizes the target histogram $\widehat{\mathit{h}}$. However, in the digital condition, the functions ${\varphi }_{\mathit{h}}^{HE}$ and ${\varphi }_{\widehat{h}}^{HE}$ become staircase functions, which are not bijective. Therefore, they have no inverse functions.

A prescription for this situation may be the use of linear interpolation as follows: Assume that ${\varphi }_{\widehat{\mathit{h}}}^{HE}(l-1)<{\varphi }_{\mathit{h}}^{HE}\left(f_{ij}\right)\le {\varphi }_{\widehat{\mathit{h}}}^{HE}\left(l\right)$ for $l\in \{0,1,\dots ,L\}$, where we define that ${\varphi }_{\widehat{\mathit{h}}}^{HE}(-1)=0$. Then, we can determine the output pixel value of histogram specification by

$$\begin{array}{c}\hfill f_{ij}^{HS}\approx \mathrm{round}\left(\frac{\left[{\varphi }_{\widehat{\mathit{h}}}^{HE}\left(l\right)-{\varphi }_{\mathit{h}}^{HE}\left(f_{ij}\right)\right]max\{l-1,0\}+\left[{\varphi }_{\mathit{h}}^{HE}\left(f_{ij}\right)-{\varphi }_{\widehat{\mathit{h}}}^{HE}(l-1)\right]l}{{\varphi }_{\widehat{\mathit{h}}}^{HE}\left(l\right)-{\varphi }_{\widehat{\mathit{h}}}^{HE}(l-1)}\right)\end{array}$$

(5)

corresponding to an input pixel value $f_{ij}$.

Figure 2 shows an example of histogram specification, where the resultant image in Figure 2a is obtained with a Gaussian target histogram with mean $127.5$ and standard deviation 50 from the original image in Figure 1a. Figure 2b shows the obtained histogram, which is not Gaussian, but the cumulative one fits into the target cumulative histogram as shown in Figure 2c, where a blue solid line denotes the target one, and the yellow dashed line denotes the obtained one. That is, to be more precise, the conventional histogram specification is a cumulative histogram specification.

4.2. Nikolova and Steidl’s Method

In this subsection, we briefly summarize Nikolova and Steidl’s fast ordering algorithm for exact histogram specification [13]. They proposed a fixed point algorithm that attains the minimizer of fully smoothed $l_1$-TV (total variation) functional as follows:

$$\begin{array}{c}\hfill J(\mathit{u},\mathit{f}):=\sum _{\iota =1}^{mn}\theta (u_{\iota }-f_{\iota })+\beta \sum _{\kappa =1}^r\theta \left({\left(G\mathit{u}\right)}_{\kappa }\right),\end{array}$$

(6)

where $\beta >0$, $f={[f_1,f_2,\dots ,f_{mn}]}^T=\mathrm{vec}\left(F\right)={[f_{11},f_{21},\dots ,f_{m-1,n},f_{mn}]}^T$ is the vectorization [25] of F where the superscript T denotes the matrix transpose [26], ${\left(G\mathit{u}\right)}_{\kappa }$ denotes the $\kappa$th component of $G\mathit{u}\in {\mathbb{R}}^r$ with $r=2mn-m-n$, where $\mathbb{R}$ denotes the set of real numbers, and G is a forward difference operator given by

$$\begin{array}{c}\hfill G:=\left(\begin{array}{c}{I}_n\otimes D_m\\ D_n\otimes I_m\end{array}\right)\in {\mathbb{R}}^{r,mn},\end{array}$$

(7)

where ⊗ denotes the Kronecker product, $I_n$ is the $n\times n$ identity matrix, $D_n$ denotes the forward difference matrix defined by

$$\begin{array}{c}\hfill D_n:=\left(\begin{array}{cccccc}-1& 1& & & & \\ & -1& 1& & & \\ & & & \ddots & & \\ & & & & -1& 1\end{array}\right)\in {\mathbb{R}}^{n-1,n},\end{array}$$

(8)

and the function $\theta$ is defined by

$$\begin{array}{c}\hfill \theta \left(t\right):=\left|t\right|-\alpha log\left(1+\frac{\left|t\right|}{\alpha }\right)\end{array}$$

(9)

with a constant $\alpha >0$, which is a smooth approximation of $l_1$ norm, and has the derivative $z={\theta }^{\prime }\left(t\right)=\frac{t}{\alpha +\left|t\right|}$. The inverse function of ${\theta }^{\prime }$ is defined by $\xi \left(z\right):={\left({\theta }^{\prime }\right)}^{-1}\left(z\right)=\frac{\alpha z}{1-\left|z\right|}$, whose derivative is given by ${\xi }^{\prime }\left(z\right)=\frac{\alpha }{{(1-|z\left|\right)}^2}$. These functions are illustrated in Figure 3, where blue lines denote the functions for $l_1$ norm, and orange lines denote their smooth approximations.

From $\frac{\partial J}{\partial u_{\iota }}={\theta }^{\prime }(u_{\iota }-f_{\iota })+\beta {\sum }_{\kappa =1}^r{\theta }^{\prime }\left({\left(Gu\right)}_{\kappa }\right)G_{\kappa ,\iota }=0$ where $G_{\kappa ,\iota }$ means the $(\kappa ,\iota )$th element of G, we have $u_{\iota }=f_{\iota }-\xi \left(\beta {\sum }_{\kappa =1}^r{\theta }^{\prime }\left({\left(G\mathit{u}\right)}_{\kappa }\right)G_{\kappa ,\iota }\right)$ for $\iota =1,2,\dots ,mn$, which are combined into a vector equation as $\mathit{u}=\mathit{f}-\xi \left(\beta G^T{\theta }^{\prime }\left(G\mathit{u}\right)\right)$, where the functions ${\theta }^{\prime }$ and $\xi$ are applied to each element of their arguments. This is a fixed point equation for $\mathit{u}$ which gives rise to a fixed point algorithm for minimizing $J(\mathit{u},\mathit{f})$ as described in Algorithm 1.

Algorithm 1 Fully smoothed $l_1$-TV minimization algorithm [13]

Require:  a vectorized image $\mathit{f}={[f_1,f_2,\dots ,f_{mn}]}^T$, parameters $\beta >0$ and $\alpha >0$ used in functions $\xi$ and ${\theta }^{\prime }$, the number of iterations $T>0$ Ensure:  the minimizer ${\mathit{u}}^{\left(T\right)}$ of a fully smoothed $l_1$-TV functional $J(\mathit{u},\mathit{f})$ in (6)   1: Initialize the variable $\mathit{u}$ as ${\mathit{u}}^{\left(0\right)}\leftarrow \mathit{f}$   2: for$t\leftarrow 1,2,\dots ,T$do   3:     ${\mathit{u}}^{\left(t\right)}\leftarrow \mathit{f}-\xi \left(\beta G^T{\theta }^{\prime }\left(G{\mathit{u}}^{(t-1)}\right)\right)$   4: end for

In this algorithm, we set $\alpha =0.05,\beta =0.1$ and $T=5$ according to Nikolova and Steidl [13]. The obtained image from ${\mathit{u}}^{\left(T\right)}$ with the input image $\mathit{f}$ in Figure 1a is shown in Figure 4a. This procedure for computing ${\mathit{u}}^{\left(T\right)}$ from $\mathit{f}$ can be viewed as a nonlinear filter which reduces the quantization noise in $\mathit{f}$ slightly. Therefore, the image in Figure 4a is very close to that in Figure 1a. The pixel values in ${\mathit{u}}^{\left(T\right)}$ can be ordered in a strict way with a high probability [8].

Assume that the pixel values in ${\mathit{u}}^{\left(T\right)}={[u_1^{\left(T\right)},u_2^{\left(T\right)},\dots ,u_{mn}^{\left(T\right)}]}^T$ are ordered in ascending order: $u_{{\lambda }_1}^{\left(T\right)}<u_{{\lambda }_2}^{\left(T\right)}<\cdots <u_{{\lambda }_{mn}}^{\left(T\right)}$ where ${\lambda }_{\iota }$ for $\iota =1,2,\dots ,mn$ denote the ordered indices, and the cumulative histogram $\widehat{\mathit{H}}=[{\widehat{H}}_0,{\widehat{H}}_1,\dots ,{\widehat{H}}_L]$ is computed from a given target histogram $\widehat{\mathit{h}}$. Then, if ${\widehat{H}}_{l-1}<\iota \le {\widehat{H}}_l$ with ${\widehat{H}}_{-1}=0$ for an index $\iota$ and a pixel value l, then the corresponding output pixel value is given by $f_{i_{\iota },j_{\iota }}^{NS}=l$, where $(i_{\iota },j_{\iota })$ denotes the pixel position corresponding to the ordered index ${\lambda }_{\iota }$, i.e., $({\lambda }_{\iota }-1)=(j_{\iota }-1)m+(i_{\iota }-1)$ with $0\le (i_{\iota }-1)<m$ is satisfied. This procedure is repeated for $\iota =1,2,\dots ,mn$ to obtain Nikolova and Steidl’s output image $F^{NS}=\left[f_{ij}^{NS}\right]$. Figure 4b shows the output image with the same Gaussian target histogram as Figure 2, and Figure 4c shows the target and obtained histograms, where we can see that they have the same Gaussian shape, which demonstrates the exactness of Nikolova and Steidl’s method.

5. Proposed Method

In this section, we propose a local contrast-based pixel ordering method for exact histogram specification. We first describe the method for grayscale images, and then it is extended to that for color images. We also give a complexity analysis of the proposed method. The main idea of our approach is that the problem of pixel ordering can be divided into 256 subproblems for an 8-bit grayscale image. We solve the downsized subproblems separately to have 256 groups of ordered pixels, and then concatenate them to have a sequence of all ordered pixels.

5.1. Local Contrast-Based Exact Histogram Specification for Grayscale Images

As demonstrated above, we can transform the histogram of a digital image into the specified one, when all pixels in the image are ordered in a strict and faithful way [13]. In most cases, pixels in a digital image take a limited number of discrete values, e.g., 256 values are available for 8-bit grayscale images, which is smaller than the number of pixels in the image, e.g., $m\times n=256\times 256=65,536\gg 256$ for the above image in Figure 1a. Therefore, many pixels share the same pixel value with each other in the image, and the ordering of the pixels with the same values becomes an ill-posed problem [8]. However, it is fortunate for us that we have a rough ordering of 256 groups of pixels in the image, which means that the entire pixel ordering problem can be divided into 256 subproblems that can be solved separately. In this subsection, we propose a method for ordering pixels in each group on the basis of the local contrast of each pixel, and then the separately ordered subgroups are finally merged into an entire pixel ordering.

The first step of the proposed method is Gaussian filtering without approximation, which means that the weighted mean of all pixels is outputted at each pixel, where the weights are given by a Gaussian function as follows:

$$\begin{array}{c}\hfill f_{ij}^{GF}=\frac{{\displaystyle \sum _{k=1}^m\sum _{l=1}^{n}exp\left(-\frac{{(i-k)}^2+{(j-l)}^2}{2{\sigma }^2}\right)f_{kl}}}{{\displaystyle \sum _{k=1}^m\sum _{l=1}^{n}exp\left(-\frac{{(i-k)}^2+{(j-l)}^2}{2{\sigma }^2}\right)}}=\frac{{\displaystyle \sum _{k=1}^{m}exp\left(-\frac{{(i-k)}^2}{2{\sigma }^2}\right)\sum _{l=1}^{n}exp\left(-\frac{{(j-l)}^2}{2{\sigma }^2}\right)f_{kl}}}{{\displaystyle \sum _{k=1}^{m}exp\left(-\frac{{(i-k)}^2}{2{\sigma }^2}\right)\sum _{l=1}^{n}exp\left(-\frac{{(j-l)}^2}{2{\sigma }^2}\right)}}\end{array}$$

(10)

for $i=1,2,\dots ,m$ and $j=1,2,\dots ,n$, where $\sigma$ denotes the standard deviation positive constant. Let $F^{GF}=\left[f_{ij}^{GF}\right]$ be an $m\times n$ matrix which expresses the Gaussian-filtered image of F. Then, we can compute $F^{GF}$ by the following matrix operations. The numerator of (10) is computed by

$$\begin{array}{c}\hfill N_u=G_{L}F{G}_R,\end{array}$$

(11)

where the left matrix $G_L$ is given by $G_L=exp(-D_L\odot D_L/2{\sigma }^2)$ with

$$\begin{array}{c}\hfill D_L={\mathbf{1}}_m\left[\begin{array}{cccc}1& 2& \cdots & m\end{array}\right]+{\left[\begin{array}{cccc}1& 2& \cdots & m\end{array}\right]}^T{\mathbf{1}}_m^T-2{\left[\begin{array}{cccc}1& 2& \cdots & m\end{array}\right]}^T\left[\begin{array}{cccc}1& 2& \cdots & m\end{array}\right],\end{array}$$

(12)

where ⊙ denotes the Hadamard product or elementwise product of matrices, and ${\mathbf{1}}_m$ denotes the m-dimensional column vector of ones. Similarly, the right matrix $G_R$ in (11) is given by $G_R=exp(-D_R\odot D_R/2{\sigma }^2)$ with

$$\begin{array}{c}\hfill D_R={\mathbf{1}}_n\left[\begin{array}{cccc}1& 2& \cdots & n\end{array}\right]+{\left[\begin{array}{cccc}1& 2& \cdots & n\end{array}\right]}^T{\mathbf{1}}_n^T-2{\left[\begin{array}{cccc}1& 2& \cdots & n\end{array}\right]}^T\left[\begin{array}{cccc}1& 2& \cdots & n\end{array}\right].\end{array}$$

(13)

On the other hand, the denominator of (10) is computed by

$$\begin{array}{c}\hfill D_e=G_{L}E{G}_R,\end{array}$$

(14)

where E denotes the $m\times n$ matrix of ones. Then, we have the Gaussian-filtered image by $F^{GF}=N_u\oslash D_e$, where ⊘ denotes the Hadamard division or elementwise division of matrices.

Next, we define the local contrast (LC) of each pixel by $d_{ij}:=f_{ij}-f_{ij}^{GF}$ which is added to the corresponding pixel as an attribute, and divide the $mn$ pixels in F into $(L+1)$ groups as $S_k:=\left\{(f_{ij},d_{ij})\right|f_{ij}=k\}$, where we have that $h_k=\left|S_k\right|$ for $k=0,1,\dots ,L$, i.e., $h_k$ is the cardinality of $S_k$. Then, we sort the elements of $S_k$ in the ascending order of $d_{ij}$ for $k=0,1,\dots ,L$, and concatenate them to have a sequence. As a result, we expect to have the entire pixel ordering ${\mu }_{\iota }\in \{1,2,\dots ,mn\}$ for $\iota =1,2,\dots ,mn$, from which the corresponding pixel position $(i_{\iota },j_{\iota })$ is given as the integer part $(j_{\iota }-1)$ and remainder $(i_{\iota }-1)$ of the division $({\mu }_{\iota }-1)/m$, i.e., $({\mu }_{\iota }-1)=(j_{\iota }-1)m+(i_{\iota }-1)$, and $\left\{d_{ij}\right\}$ is ordered as follows:

$$\begin{array}{c}\hfill d_{i_1,j_1}<\cdots <d_{i_{\iota },j_{\iota }}<\cdots <d_{i_{mn},j_{mn}}.\end{array}$$

(15)

For exact histogram specification, we would like to have a strict ordering as described in (15). That is, for different pixels with the same pixel value as $f_{ij}=f_{i^{\prime }{j}^{\prime }}$ with $i\ne i^{\prime }$ or $j\ne j^{\prime }$, it is expected that $d_{ij}\ne d_{i^{\prime }{j}^{\prime }}$ or $f_{ij}^{GF}\ne f_{i^{\prime }{j}^{\prime }}^{GF}$. For conventional Gaussian filters with finite kernel sizes such as $3\times 3,5\times 5$ and $7\times 7$ pixels, the lower bounds of the probability of $f_{ij}^{GF}=f_{i^{\prime }{j}^{\prime }}^{GF}$ are ${\left(\frac{1}{256}\right)}^{7\times 7}<{\left(\frac{1}{256}\right)}^{5\times 5}<{\left(\frac{1}{256}\right)}^{3\times 3}\approx 0$. However, some symmetric patterns may result in the case of $f_{ij}^{GF}=f_{i^{\prime }{j}^{\prime }}^{GF}$ accidentally. To reduce the probability of such cases, we adopt the Gaussian filter with sufficiently large kernel size as described in (10).

For an input image F with a target histogram $\widehat{\mathit{h}}$ and its cumulative version $\widehat{H}=[{\widehat{H}}_0,{\widehat{H}}_1,\dots ,{\widehat{H}}_L]$, the proposed method outputs a histogram-specified image $F^{LC}=\left[f_{ij}^{LC}\right]$, where each pixel value is given by $f_{i_{\iota },j_{\iota }}^{LC}=l$ for the index ${\mu }_{\iota }$ satisfying ${\widehat{H}}_{l-1}<\iota \le {\widehat{H}}_l$ with ${\widehat{H}}_{-1}=0$ as well as Nikolova and Steidl’s method summarized in Section 4.2. The proposed method is summarized in Algorithm 2.

Algorithm 2 Local contrast-based exact histogram specification

Require:  a grayscale image $F=\left[f_{ij}\right]$, a positive constant $\sigma$ and a target histogram $\widehat{\mathit{h}}$ Ensure:  a histogram-specified image $F^{LC}=\left[f_{ij}^{LC}\right]$   1: Compute the Gaussian-filtered image $F^{GF}$ of F with a positive constant $\sigma$   2: Compute the local contrast $D=\left[d_{ij}\right]$ by $D=F-F^{GF}$   3: Make $(L+1)$ groups $S_k=\left\{(f_{ij},d_{ij})\right|f_{ij}=k\}$ for $k=0,1,\dots ,L$   4: $\iota \leftarrow 1$   5: for$k\leftarrow 0,1,\dots ,L$do   6:       Sort the elements of $S_k$ in the ascending order of $d_{ij}$ as $d_{i_1,j_1}<\cdots <d_{i_u,j_u}<\cdots <d_{i_{|S_k|},j_{|S_k|}}$   7:       for $u\leftarrow 1,2,\dots ,|S_k|$ do   8:             ${\mu }_{\iota }\leftarrow m(j_u-1)+i_u$   9:             $\iota \leftarrow \iota +1$ 10:        end for 11: end for 12: Compute the target cumulative histogram $\widehat{H}=[{\widehat{H}}_0,{\widehat{H}}_1,\dots ,{\widehat{H}}_L]$ from $\widehat{\mathit{h}}$ 13: for$\iota \leftarrow 1,2,\dots ,mn$do 14:       Search for l satisfying ${\widehat{H}}_{l-1}<\iota \le {\widehat{H}}_l$ with ${\widehat{H}}_{-1}=0$ 15:       $f_{i_{\iota },j_{\iota }}^{LC}\leftarrow l$, where $(j_{\iota }-1)$ and $(i_{\iota }-1)$ are the integer part and remainder of $({\mu }_{\iota }-1)/m$ 16: end for

Additionally, we can make an image whose pixel values indicate ordered numbers as $F^I=\left[f_{ij}^I\right]$, where each pixel value is given by $f_{i_{\iota },j_{\iota }}^I=\iota$ for $\iota =1,2,\dots ,mn$. For histogram equalization, the number of pixels to which the same pixel value is assigned is given by $\overline{h}=\lfloor mn/(L+1)\rfloor$, where $\lfloor x\rfloor$ denotes the floor function that gives the largest integer less than or equal to x [27]. That is, we make a target histogram $\widehat{\mathit{h}}=\left[{\widehat{h}}_k\right]$ as ${\widehat{h}}_k=\overline{h}$ for $k=0,1,\dots ,L$. However, the truncation by the floor function may cause the shortage of total amount in the histogram: ${\sum }_{k=0}^L{\widehat{h}}_k<mn$. To cover the shortage, if $Q=mn-{\sum }_{k=0}^L{\widehat{h}}_k>0$, then we add 1 to ${\widehat{h}}_0,{\widehat{h}}_1,\dots ,{\widehat{h}}_{Q-1}$ to have ${\sum }_{k=0}^L{\widehat{h}}_k=mn$. This modified histogram can be inputted to Algorithm 2 as a target histogram for exact histogram equalization.

Figure 5 shows the results of the proposed method, where Figure 5a,b show the results of the histogram equalization and specification with $\sigma =50$ for the original image in Figure 1a, respectively, and Figure 5c shows their histograms, where we can see that both exact histogram equalization and specification are achieved.

5.2. Extension to Color Images

In this subsection, we extend the above proposed method for grayscale images to color images. Let $C=\left[{\mathbf{c}}_{ij}\right]$ be an RGB color image, where ${\mathbf{c}}_{ij}=[c_{ij1},c_{ij2},c_{ij3}]$ denotes the RGB color vector of the $(i,j)$th pixel in C, and $c_{ijk}\in \{0,1,\dots ,L\}$ for $k=1,2$ and 3. Then, we make a histogram into which all RGB values in C are united as follows:

$$\begin{array}{c}\hfill {\mathbf{h}}^C=[h_0^C,\dots ,h_l^C,\dots ,h_L^C],h_l^C=\sum _{i=1}^m\sum _{j=1}^n\sum _{k=1}^3{\delta }_{l,c_{ijk}}\mathrm{for}l=0,1,\dots ,L.\end{array}$$

(16)

For example, Figure 6 shows an RGB color image and its histogram defined in (16).

The first step of our exact histogram specification is Gaussian filtering applied to each color channel in C. Let $C^{GF}=\left[{\mathbf{c}}_{ij}^{GF}\right]$ be the resultant image, where ${\mathit{c}}_{ij}^{GF}=[c_{ij1}^{GF},c_{ij2}^{GF},c_{ij3}^{GF}]$ denotes the RGB color vector of each pixel in $C^{GF}$. Then, we compute the LC as $D^C=\left[{\mathit{d}}_{ij}^C\right]=C-C^{GF}$ where ${\mathit{d}}_{ij}^C=[d_{ij1}^C,d_{ij2}^C,d_{ij3}^C]={\mathit{c}}_{ij}-{\mathit{c}}_{ij}^{GF}$, each element of which is added to the corresponding element of the corresponding pixel as an attribute: $(c_{ijk},d_{ijk}^C)$ for $i=1,2,\dots ,m;j=1,2,\dots ,n$ and $k=1,2,3$, and divide them into $(L+1)$ groups as $S_l:=\left\{(c_{ijk},d_{ijk}^C)\right|c_{ijk}=l\}$, where we have that $h_l^C=\left|S_l\right|$ for $l=0,1,\dots ,L$. Then, we sort the elements of $S_l$ in the ascending order of $d_{ijk}^C$ for $l=0,1,\dots ,L$, and concatenate them to have a sequence. As a result, we expect to have the entire element ordering ${\eta }_{\iota }\in \{1,2,\dots ,3mn\}$ for $\iota =1,2,\dots ,3mn$, each of which is connected with the corresponding pixel position $(i_{\iota },j_{\iota })$ and channel number $k_{\iota }$ by satisfying

$$\begin{array}{c}\hfill d_{i_1,j_1,k_1}^C<\cdots <d_{i_{\iota },j_{\iota },k_{\iota }}^C<\cdots <d_{i_{3mn},j_{3mn},k_{3mn}}^C.\end{array}$$

(17)

Let ${\widehat{\mathit{h}}}^C$ be a target histogram with the cumulative one ${\widehat{\mathit{H}}}^C=[{\widehat{H}}_0^C,{\widehat{H}}_1^C,\dots ,{\widehat{H}}_L^C]$. Then, the histogram-specified image $C^{LC}=\left[{\mathbf{c}}_{ij}^{LC}\right]$ is given by ${\mathbf{c}}_{ij}^{LC}=[c_{ij1}^{LC},c_{ij2}^{LC},c_{ij3}^{LC}]$ with $c_{i_{\iota },j_{\iota },k_{\iota }}^{LC}=l$ for the index ${\eta }_{\iota }$ satisfying ${\widehat{H}}_{l-1}^C<\iota \le {\widehat{H}}_l^C$ with ${\widehat{H}}_{-1}^C=0$. This procedure for color images is summarized in Algorithm 3.

Algorithm 3 Local contrast-based exact histogram specification for color images

Require:  a color image $C=\left[{\mathit{c}}_{ij}\right]$, a positive constant $\sigma$ and a target histogram ${\widehat{\mathit{h}}}^C$ Ensure:  a histogram-specified image $C^{LC}=\left[{\mathit{c}}_{ij}^{LC}\right]$   1: Compute the Gaussian-filtered image $C^{GF}$ of C with a positive constant $\sigma$   2: Compute the local contrast $D^C=\left[{\mathit{d}}_{ij}^C\right]$ by $D^C=C-C^{GF}$ where ${\mathit{d}}_{ij}^C=[d_{ij1}^C,d_{ij2}^C,d_{ij3}^C]$   3: Make $(L+1)$ groups $S_l=\left\{(c_{ijk},d_{ijk}^C)\right|c_{ijk}=l\}$ for $l=0,1,\dots ,L$   4: $\iota \leftarrow 1$   5: for$l\leftarrow 0,1,\dots ,L$do   6:       Sort the elements of $S_l$ in the ascending order of $d_{ijk}^C$ as $d_{i_1,j_1,k_1}^C<\cdots <d_{i_u,j_u,k_u}^C<\cdots <d_{i_{|S_l|},j_{|S_l|},k_{|S_l|}}^C$   7:        for $u\leftarrow 1,2,\dots ,|S_l|$ do   8:              ${\eta }_{\iota }\leftarrow mn(k_u-1)+m(j_u-1)+i_u$   9:              $\iota \leftarrow \iota +1$ 10:         end for 11: end for 12: Compute the target cumulative histogram $\widehat{H}=[{\widehat{H}}_0,{\widehat{H}}_1,\dots ,{\widehat{H}}_L]$ from $\widehat{\mathit{h}}$ 13: for$\iota \leftarrow 1,2,\dots ,3mn$do 14:       Search for l satisfying ${\widehat{H}}_{l-1}<\iota \le {\widehat{H}}_l$ with ${\widehat{H}}_{-1}=0$ 15:       $c_{i_{\iota },j_{\iota },k_{\iota }}^{LC}\leftarrow l$, where $i_{\iota },j_{\iota }$ and $k_{\iota }$ are related to ${\eta }_{\iota }$ by ${\eta }_{\iota }=mn(k_{\iota }-1)+m(j_{\iota }-1)+i_{\iota }$ 16: end for

Figure 7 shows the results of the proposed method for color images, where Figure 7a,b show the results of the histogram equalization and specification with $\sigma =50$ for the original color image in Figure 6a, respectively, and Figure 7c shows their histograms, where we can also see that both exact histogram equalization and specification are achieved.

5.3. Complexity Analysis

In this subsection, we analyze the complexity of the proposed method in Algorithm 2. Let $N=mn$ be the input size. The computation of Gaussian filtering in line 1 is described as matrix calculations. The time complexity of the numerator (11) is estimated as $m{n}^2+m^{2}n=mn(m+n)=N(m+n)\approx N^{3/2}$, which is the same as that of the denominator (14). The Hadamard division requires N elementwise divisions. Then, the order of time complexity of Gaussian filtering is $O\left(N^{3/2}\right)$. The next main procedure is sort in line 6, which is repeated $L+1$ times. Assume that the cardinality $|S_l|$ is $\frac{N}{L+1}$ on average. Then, the time complexity of the sorting procedure is estimated as $(L+1)\times \frac{N}{L+1}log\frac{N}{L+1}=N[logN-log(L+1)]$ from which we have $O(NlogN)$. The time complexity of the remaining procedures is bounded by $O\left(N\right)$. Consequently, the time complexity of the proposed method is given by $O\left(N^{3/2}\right)$.

The space complexities of images $F,F^{GF},D$, and $F^{LC}$ are $O\left(N\right)$, and that of $(L+1)$ groups $S_k$ for $k=0,1,\dots ,L$ in line 3 is also $O\left(N\right)$. Histogram arrays $\widehat{\mathit{h}}$ and $\widehat{\mathit{H}}$ require $O(L+1)$. Selecting the largest one, we can see that the space complexity of the proposed method is $O\left(N\right)$.

6. Experimental Results

In this section, we show experimental results in comparison with Nikolova and Steidl’s method [13]. First, we show the results with a synthetic input image shown in Figure 8a. Nikolova and Steidl’s method summarized in Section 4.2 outputs the image in Figure 8b, where we can see that vertical stripes appear in the flat areas in the original image. As a result, Nikolova and Steidl’s method achieves the equalized histogram shown in Figure 8c. Figure 8d shows the horizontal profile of the output image in Figure 8b, where the vertical and horizontal axes denote the pixel value and the column index j, respectively. Although the pixel values fluctuate with small steps, the pixel values on the left side are larger than that on the right side as well as the original image.

Figure 9 shows the results of the proposed method described in Section 5, where Figure 9a,b show the output images of exact histogram equalization and specification with the Gaussian target histogram used in the previous section. Their histograms are shown in Figure 9c. The proposed method enhances the edge in the center of the input image in Figure 8a, and does not produce the pseudo-stripes observed in Figure 8b that are graphically confirmed by the horizontal profiles in Figure 9d.

Next, we compare the ability in strict ordering by Nikolova and Steidl’s and the proposed methods. Figure 10 compares the filter outputs used in the two methods with the input image in Figure 1a. Nikolova and Steidl’s filter in Algorithm 1 outputs ${\mathit{u}}^{\left(T\right)}$, the values in which are sorted in ascending order in Figure 10a where the vertical and horizontal axes denote the pixel value and index, respectively. The differences between the neighboring pixel values in Figure 10a are very small, and therefore, we take the logarithm of them as shown in Figure 10b. Similarly, we sort the values of the output of the Gaussian filter in (10) as shown in Figure 10c, and take the logarithm of the difference values in Figure 10c as shown in Figure 10d, where we can see that the minimum value is greater than that in Figure 10c, which is preferable for strict ordering because the larger the difference is, the clearer the ordering is. The minimum values of the differences without taking logarithm are $4.23\times {10}^{-12}$ and $1.66\times {10}^{-8}$ for Figure 10b,d, respectively.

The proposed method has a parameter $\sigma$ in (10), the effect of which is investigated in Figure 11a, where the vertical and horizontal axes denote the minimum difference in the sorted output values of the Gaussian filter and the parameter $\sigma$, respectively, for the input image in Figure 1a. In this figure, although we cannot find clear tendency of the minimum difference as a function of $\sigma$, the obtained difference values are larger than that of ${\mathit{u}}^{\left(T\right)}$ given by Algorithm 1 that exemplifies the insensitivity of the proposed method to $\sigma$. Figure 11b,c show the results of histogram equalization with $\sigma =1$ and 70, respectively. These images demonstrate that the proposed method is insensitive to the values of parameter $\sigma$.

We investigate the limitation of the proposed method on the value of $\sigma$. Figure 12 shows the results of histogram equalization for the input image in Figure 8a, where Figure 12a,b correspond to $\sigma ={10}^8$ and $5\times {10}^8$, respectively. Figure 12a is similar to Figure 9a. However, in Figure 12b, we observe an artifact. As a result, we estimate that the limitation of the value of $\sigma$ exists between ${10}^8$ and $5\times {10}^8$, and recommend using the value of $\sigma$ smaller than ${10}^8$.

Figure 13 shows twelve grayscale images in the Standard Image Data-BAse (SIDBA) [28], which are enhanced by Nikolova and Steidl’s exact histogram equalization as shown in Figure 14, where the contrast of all images is enhanced well because the histogram of each image is exactly equalized.

Similarly, Figure 15 shows the results by the proposed method, which also equalizes the histogram of each image exactly. Therefore, the output images of both methods are similar to each other.

Figure 16a shows the root mean squared error (RMSE) between Nikolova and Steidl’s output image and the corresponding output image by the proposed method, where “Text” image shown in (b) get the largest RMSE value, but the difference is small as visualized in (c), where error-free pixels have the neutral gray value 127.

We also compare the results of exact histogram specification with the Gaussian target histogram, which are shown in Figure 17 and Figure 18 for Nikolova and Steidl’s and the proposed methods, respectively. In Figure 17, we observe that the obtained contrast is not so high as that of Figure 14 because the Gaussian target histogram suppresses the numbers of extremely dark and bright pixels, and assigns many pixel values to middle-range gray pixels. As a result, the dynamic range of gray pixels is boosted relatively. The output images in Figure 18 have a similar tendency to that in Figure 17 because the proposed method also obtains an exact Gaussian target histogram as well as Nikolova and Steidl’s method.

Figure 19a shows the RMSE between the corresponding output images in Figure 17 and Figure 18, where three images “Bui.”, “Lig.” and “Tex.” have relatively large RMSE values. We can find the differences at the bottom of those images as shown in Figure 19b, where the bottom areas of “Bui.”, “Lig.” and “Tex.” are arranged from top to bottom. The corresponding results by Nikolova and Steidl’s and the proposed methods are shown in Figure 19c and Figure 19d, respectively, where we can see artifacts in (c) similar to that in Figure 8b. On the other hand, Figure 19d has no such artifacts.

Figure 20 shows twelve color images in the SIDBA image dataset [28], which are enhanced by a separable histogram equalization that applies Nikolova and Steidl’s exact histogram equalization algorithm to each color channel separately as shown in Figure 21, where we can see that the contrast of every image is enhanced, but the hue has changed from the original images in Figure 20.

Figure 22 shows the results from the proposed histogram equalization method described in Section 5.2, where we can see that the hue of each image is closer to the original one than that in Figure 21.

Figure 23 shows the RMSE of hue of each output image from the corresponding original image, where the vertical and horizontal axes denote the RMSE value and the images, respectively, and the blue and orange bars denote the separable and proposed methods, from which we observe that the proposed method preserves the original hue better than the separable method.

Figure 24 shows the results of a separable histogram specification that applies Nikolova and Steidl’s exact histogram specification algorithm to each color channel separately. Similarly to the above results of histogram equalization in Figure 21, here we can see the hue change from the original images.

On the other hand, Figure 25 shows the results by the proposed histogram specification method, where we can also see that the hue of each image is closer to the original one than that in Figure 24.

Figure 26 shows the RMSE of hue of each output image from the corresponding original image, from which we also observe that the proposed method preserves the original hue better than the separable method.

The above experimental results demonstrate the effectiveness of the proposed local contrast-based pixel ordering method for exact histogram equalization and specification compared with the state-of-the-art method by Nikolova and Steidl [13]. A drawback of Nikolova and Steidl’s algorithm observed in Figure 8b, where a stripe pattern occurred on flat areas, may be a newfound phenomenon. The proposed method can avoid the occurrence of such false patterns.

In the above experiments, although we used a Gaussian target histogram for histogram specification constantly, we can choose other candidates such as multimodal distributions without any problems.

For the implementation of Nikolova and Steidl’s algorithm described in Algorithm 1, we need to use sparse matrices even if the input image is not so large, e.g., the original image in Figure 1a with $256\times 256$ pixels because the difference operator G in (7) becomes a very large matrix with $r\times mn\approx 8.56\times {10}^9$ elements. On the other hand, the proposed method can be implemented without sparse matrices.

Additionally, we implemented Ramos’s recent algorithm for histogram specification [16]. Figure 27 shows the result of Ramos’s histogram specification, where Figure 27a shows the output image computed from the input image in Figure 1a with the Gaussian target histogram with mean 127.5 and standard deviation 50. Figure 27b shows the obtained histogram of the output image in Figure 27a in a blue line with the target histogram in the orange line. This figure shows that Ramos’s algorithm is not an exact histogram specification method, and the obtained histogram is similar to that of the conventional histogram specification shown in Figure 2b. On the other hand, the proposed method gives exactly the same histogram as a prescribed target histogram as shown in Figure 2c. In Ramos’s formulation of $l^p$ norm minimization, we set $p=\infty$. For details, please refer to Ramos’s original paper [16]. For the implementation of Ramos’s algorithm, we need to use sparse matrices with about $mn\times L$ elements as well as Nikolova and Steidl’s one.

A limitation of the proposed method is that users need to prepare a target histogram for histogram specification. However, it may be difficult to know the optimal shape of the histogram for a given image in advance. The estimation of suitable target histograms for given images will be a subject to be considered in the future.

7. Conclusions

In this paper, we proposed a method for ordering pixels in an image for exact histogram specification, and compared it with the state-of-the-art algorithm by Nikolova and Steidl [13]. We divided the problem of pixel ordering into small subproblems which can be solved memory-efficiently. This idea makes the problem of exact histogram specification more tractable than ever. The proposed method uses the Gaussian filter without approximation instead of the nonlinear filter used in Nikolova and Steidl’s algorithm. We use the local contrast defined with the Gaussian-filtered image as a key to ordering pixels. As a result, it was experimentally demonstrated that we could avoid the occurrence of false patterns that were observed in the results obtained by Nikolova and Steidl’s algorithm.

We also extended the proposed method for grayscale images to that for color images. Experimental results showed that the proposed method keeps the hue of the original images better than conventional separable methods of histogram equalization and specification for color images.

Our future work will include the development of a method for making a preferable target histogram for histogram specification depending on a given input image.