Color Inverse Halftoning Method with the Correlation of Multi-Color Components Based on Extreme Learning Machine

Abstract

Look-up table (LUT) based method is a popular and effective way for inverse halftoning. However, it still has very large development space to improve the reconstructed color image quality for color halftone images, because most of the existing color inverse halftoning methods are the simple extension of LUT methods to each color components separately. To this end, this paper presents a novel color inverse halftoning method by exploiting the correlation of multi-color components. Through considering all existent contone values with the same halftone pattern in three color component tables, we firstly propose a concept of common pattern. Then the extreme learning machine (ELM) is employed to estimate the contone values for nonexistent patterns according to common patterns in color LUT, which can not only improve the fitting precision of nonexistent values but also has fast transformation speed. Experimental results show that the proposed method achieves a better image quality when compared to previously published methods.

1. Introduction

Digital halftoning is a special technique of converting a continuous image into a binary image, which is widely used in many fields, such as printing industry, textile industry and the electronic display field [1,2]. The binary image, called halftone image, can simulate the original continuous tone image in bi-level output devices. However, when halftone images need to be further processed by applying general image methods, such as image zooming, rotation, compression, and feature extraction, they are typically difficult to operate. Thus, another technique named inverse halftoning comes into being for this purpose. Inverse halftoning is a technique which is used to reconstruct a continuous tone image from its corresponding halftone image. Due to a certain amount of information loss in the process of digital halftoning, inverse halftoning is an ill-posed problem. Thus, reconstructing a high-quality continuous tone image from its halftone version is a challenging but promising research topic.

Over the past several decades, many inverse halftoning methods have been presented. The proposed methods can be classified into five categories: filtering, deconvolution, optimization estimation, vector quantization, and machine learning [1,2]. The filtering-based method includes Gaussian low-pass filtering method [3], adaptive filtering [4], linear or non-linear filtering [5], and transform-domain filtering [6]. The low-pass filtering is a simple and fast inverse halftoning method, it can remove most of the noise in the reconstructed continuous tone image. Nevertheless, it also causes edge blurring. Besides low-pass filtering, other filtering-based methods, such as adaptive filtering, linear or non-linear filtering, and transform-domain filtering, are designed for obtaining a sharp image with a low perceived noise level. Although these methods can produce good results on image edges, they also create artifacts in the image smooth areas. Inverse halftoning can be considered as a deconvolution problem, which was first proposed by Kite et al. [7] as a linear approximation model. This method can work well for images generated by error diffusion halftoning method but does not apply to other types of halftone images. Lai et al. [8] proposed a novel inverse halftoning by applying classified vector quantization, it is also effective for error diffusion halftoning. The machine-learning-based inverse halftoning method is to train a model by using continuous tone images and their halftone versions, and then the continuous tone value can be estimated by this model. The model includes neural network [9,10], sparsity-based [1,11,12], and look-up table [13,14]. Compared to other two machine-learning-based methods, the method based on look-up table is a promising method because of its fast speed and universality. Thus, it has attracted the attention of many researchers.

The method based on look-up table was first proposed by Mese et al. [13] for inverse halftoning. Since then, many methods were proposed to improve the performance of this method [14,15,16,17,18,19,20,21,22]. In summary, the improvements include three aspects. The first is the template selection and optimization [14,15], where the correlation method and the elitist genetic algorithm were used to obtain the best template. The second is to reduce the size of the table and improve the speed of searching look-up table [16,17,18,19], while the third is to get the accurate contone value from each template pattern [20,21,22]. Although these improvements have produced relatively satisfactory results for grey images, very little research has addressed color inverse halftoning. Except for references [13] and [14], most of the existing color inverse halftoning methods are a simple extension of the look-up table (LUT) method to each color planes separately. Since these color inversing methods cannot consider the correlation between the color planes, they lead to a poor quality of color continuous-tone images. In references [13] and [14], the template of each color plane, as shown in Tables XIII–XV in Reference [14], is constructed not only by its neighborhood pixels, but also considering the two pixels at the same position in the other two-color components, thus the quality of the reconstructed color image is improved to a certain extent. Since the correlation between the color components proposed in reference [13] and [14] is a weak correlation, there is still room to improve the reconstructed color image quality for color halftone images.

Armed with all the neighborhoods of each color plane, we propose a novel color inverse halftoning method by exploiting the correlation of multi-color components. The proposed method uses all existent values in the same patterns of three-color component tables to estimate the nonexistent values in color table. In order to increase the estimation speed and accuracy, the extreme learning machine (ELM) is employed to learn the relation between color template patterns and their corresponding color contone values, which can not only improve the fitting precision of nonexistent values but also has fast transformation speed. Experimental results show that the proposed method achieves a better image quality when compared to the previously published method.

The remaining part of this paper is organized as follows. Section 2 gives an overview of our proposed method. In Section 3, we detail our proposed method. The experiment is demonstrated in Section 4 by comparing it with other methods. Finally, conclusions are addressed in Section 5.

2. Overview of the Proposed Method

As the relation between the halftone pattern of color plane and the contone value of a specific color plane is a complex nonlinear, a lot of nonexistent values could not be estimated accurately in the building color LUT. In view of the same color halftone patterns corresponding to the similar values of the R, G, B planes as shown in Figure 1, we propose a novel color inverse halftoning method, which uses all common patterns to estimate the nonexistent values in the color LUT. The common pattern is defined as those halftone patterns which have the same halftone patterns in each color table, while their corresponding contone values are not empty. Figure 1 shows the overall framework of the proposed method.

As shown in Figure 1a, the first step of the color inverse halftoning is to separate a color halftone image into different color component images, such as R, G, B images, while the second step is to get the contone value from the pre-computed LUT. In order to remove the artifacts in the smooth areas of the obtained continuous image, a low-pass filter is introduced for this purpose. Figure 1b gives the generation of color LUT. In a similar way to Reference [14], we first use sample images to build an initial lookup table for each color component, and then estimate the nonexistent values using ELM.

Compared to the method in reference [14], there are two different aspects of the proposed method as shown in Figure 1b. The first is that the halftone values of all color components are used to estimate nonexistent values. For nonexistent values, we only use those items which have the same halftone patterns in each color component table where their contone values are not empty. The other is that ELM is employed to establish the relation between halftone patterns and their corresponding contone values. A discriminative feature of the proposed method is the introduction of a common pattern, because common patterns can describe the strong relation of color components.

3. Methodology

In the LUT method, the distribution of pixels in the neighborhood defined by the template is called a halftone pattern. The core task of designing LUT is to seek a contone value for each halftone pattern by using sample images. Due to a limited number of sample images, only a part of halftone patterns can obtain contone values directly, we will discuss how to acquire these patterns’ contone values in Section 3.1. For nonexistent patterns which have no contone values in any of the sample images, we will give our solution in Section 3.2.

3.1. Design of the Initial LUT

In the training phase for designing an initial LUT, the template of 19-pixels [14] is employed in this paper. According to the selected template, the halftone pattern and its contone value at any pixel can be obtained from sample images. If a halftone pattern has many different contone values, its corresponding contone value will be the mean of these contone values.

Table 1. A case of designed LUT.

	R_LUT	G_LUT	B_LUT
Number of existing pattern	316,579	353,271	330,563
Number of nonexistent pattern	207,709	156,413	173,886
Percent of nonexistent pattern (%)	39.62	32.62	36.95

is a case of our designed LUT using sample classic images, such as lena, baboon, parrot, etc., in which we use RGB color space. From Table 1, we can see the numbers of the existing patterns are 316579, 353271 and 330563 for R, G, B color table respectively, while about 30% of items in each color component LUT are nonexistent patterns. In addition, our statistical results show that about 45% of existing patterns in each color table have similar values. Therefore, we call these patterns common patterns which have a strong correlation between color components.

3.2. Estimation of Nonexistent Values Based on Common Patterns and ELM

In order to estimate the contone values of nonexistent patterns from the existent patterns, some methods have been proposed for this problem, such as low-pass filtering, hamming distance, and best linear estimator [14]. However, the precision of the estimated contone values will decrease with the increase of the percentage of nonexistent patterns when using these methods. Fortunately, ELM has a strong nonlinear fitting capability, which was put forward by Professor Huang in 2004 [23]. The ELM has been proved to be superior to BP (Back Propagation) and SVM (Support Vector Machine) in solving the problem of complex systems modeling with advantages of simple structure, fast learning speed, and good generalization performance, thus it is employed as a fitting model for estimating the contone values of nonexistent patterns in this paper. Besides fitting model, the correlation between color components is another problem worth considering for color inverse halftoning.

As described in Section 3.1, the common patterns have a strong correlation between color components because they have the same halftone patterns in each color table. By using these common patterns and their corresponding contone color values in the initial LUT, the ELM model is firstly trained. Then, the contone values of nonexistent patterns can be estimated based on the trained ELM. Figure 2 gives the scheme of estimating nonexistent values based on ELM.

In the process of training ELM model, common patterns with the same halftone pattern in each color plane are used as the input of ELM. The output of ELM is the continuous values of each color component corresponding to the halftone patterns in RGB plane. Since the weight between the input layer and the hidden layer can be generated randomly, the optimum solutions of ELM focus on the weights between the hidden layer and the output layer. We determine the number of hidden layers by the following experiment:

Let $\mathcal{T}=\{({\mathbf{x}}_i,c_i)|{\mathbf{x}}_i\in {\Re }^{1\times N},c_i\in {\Re }^{1\times C},i=1,\dots ,\mathrm{M}\}$ denote the set of common patterns in the initial LUT, where ${\mathbf{x}}_i$ is a common pattern, $c_i$ is the contone value of a color component. Define the weight matrix between the input layer and the hidden layer as $\mathsf{\alpha }={({\alpha }_{i,j})}_{L\times N}$, the bias of hidden nodes as $b_i,i=1,2,\cdots ,L$, where $L$ is the number of hidden layer nodes. The mapping function can be recorded as $h(\mathbf{x})=\{G(a_1,b_1,\mathbf{x}),\dots ,G(a_L,b_L,\mathbf{x})\}$, where $G(a_i,b_i,\mathbf{x})$ is the activation function. There are many activation functions, such as sigmoid, sin, tanh and hardlim. Among them, the sigmoid function is a classic activation function which is expressed as Equation (1)

$$G(a_i,b_i,\mathbf{x})=g(a_i\cdot \mathbf{x}+b_i)=\frac{1}{1+e^{-(a_i\cdot \mathbf{x}+b_i)}}$$

(1)

According to Reference [23], the solution to the weight matrix $\mathsf{\beta }$ between the hidden layer and the output layer can be considered as the least square solution $\widehat{\mathsf{\beta }}$ of the linear system $\mathbf{H}\mathsf{\beta }=\mathbf{T}$, which can be formulated as follows:

$$\Vert \mathbf{H}\widehat{\mathsf{\beta }}-\mathbf{T}\Vert =\underset{\mathsf{\beta }}{\min }\Vert \mathbf{H}\mathsf{\beta }-\mathbf{T}\Vert$$

(2)

The solution of Equation (2) for the weight matrix $\mathsf{\beta }$ is defined in Equation (3).

$$\widehat{\mathsf{\beta }}={\mathbf{H}}^T{\left(\frac{\mathbf{I}}{\lambda }+\mathbf{H}{\mathbf{H}}^T\right)}^{-1}\mathbf{T}$$

(3)

4. Experimental Results

Due to scarce research on color inverse halftoning in published literature, we mainly compare the proposed method with the one proposed by Mese et al. in reference [14]. The main aim of this paper is to improve the performance of the method based on LUT. To make the result reasonable, the dataset used in reference [14] is employed to build look-up tables which include thirty real-life images. In addition, we use the template of 19-pixels in reference [14] and RGB color space in the experiment. Furthermore, ten classical color images and the method based on wavelet [24] are used to evaluate the performance.

4.1. Parameters Selection for ELM

For ELM, there are two parameters that should be determined. The first one is the number of hidden neurons which is denoted by $L$, while the other is the activation function. To make a qualitative comparison, the mean absolute error of the values of nonexistent pixels in an image, named $MAD$, is defined as Equation (4).

$$MAD=\frac{1}{n}\left|N\_im{g}_c-N\_im{g}_p\right|$$

(4)

where $n$ is the number of nonexistent pixels in a color component, $N\_im{g}_c$ is the actual contone value of a nonexistent pixel in a continuous image, $N\_im{g}_p$ is the corresponding estimated value of a nonexistent pixel.

Figure 3 illustrates the variety of $MAD$ with $L$ for image Lena and Baboon. From Figure 3, we can see the fitting precision of nonexistent pixels has a considerable improvement when $L$ goes from 10 to 100, and then it increases gradually with $L$ from 100 to 300. In the end, it starts to decrease when $L$ is greater than 300. Thus, the number of hidden neurons is determined as 300. For the activation function, Figure 4 shows the experimental results of three common activation functions, which are sigmoid, sin, and hardlim. It is obvious that the sigmoid activation function is the best choice.

4.2. Analysis of the Estimated Values

For color LUT, the key problem is to estimate the nonexistent values. To better demonstrate the performance of the ELM regression model, the two cases are studied in the experiment. The first is to estimate the nonexistent patterns of each color table separately by using ELM model based on the information of a single look-up table, named as S_LUT_ELM. The second is to use the model of the best linear estimator which was proposed by Mese et al. in Reference [14], named as B_LIN_EST.

For convenience, our proposed method is named M_LUT_ELM. The M_LUT_ELM method not only considers the correlation of multi-color components but also employs the ELM model to estimate the nonexistent patterns. The performance is evaluated by comparing the estimated values and their actual values.

Table 2. Comparison of estimated values of null-value in R-LUT.

Number of Null-Value	Actual Value	B_LIN_EST		S_LUT_ELM		M_LUT_ELM
		Estimated Value	Relative Error (%)	Estimated Value	Relative Error (%)	Estimated Value	Relative Error (%)
1	76	62.26	18.07	75.53	0.62	54.09	28.83
2	103	81.70	20.68	91.68	10.99	84.16	18.29
3	94	113.60	20.85	123.34	31.21	109.22	16.19
4	206	188.26	8.61	190.86	7.35	187.32	9.07
5	98	103.59	5.70	101.27	3.34	103.83	5.95
6	131	131.99	0.75	133.90	2.21	132.78	1.36
7	144	100.26	30.38	111.10	22.84	98.92	31.30
8	131	127.97	2.31	136.58	4.26	132.80	1.37
9	107	104.15	2.66	105.38	1.52	110.15	2.95
10	122	129.06	5.79	127.16	4.23	123.22	1.00
11	175	195.43	11.67	188.31	7.60	187.59	7.20
12	89	103.07	15.81	109.61	23.15	113.90	27.98
13	134	149.58	11.62	156.08	16.48	146.98	9.68
14	107	104.15	2.66	105.38	1.52	110.15	2.95
15	100	91.29	8.71	98.52	1.48	105.36	5.36
16	194	199.83	3.00	191.48	1.30	199.50	2.83
17	161	164.86	2.40	159.37	1.01	162.47	0.91
18	175	183.54	4.88	172.47	1.44	180.01	2.86
19	78	90.65	16.22	91.92	17.85	99.72	27.84
20	165	188.58	14.29	176.90	7.21	183.72	11.34
21	134	132.44	1.17	132.48	1.13	131.08	2.18
22	170	173.26	1.92	162.64	4.33	172.79	1.64
23	159	161.57	1.62	154.45	2.86	157.64	0.86
24	129	157.42	22.03	155.17	20.29	148.34	14.99
25	105	97.60	7.05	104.94	0.06	102.37	2.50
26	118	124.98	5.92	121.10	2.63	125.65	6.48
27	190	175.83	7.46	168.21	11.47	175.92	7.41
28	92	74.38	19.15	74.94	18.54	80.84	12.13
29	189	190.85	0.98	188.28	0.38	190.92	1.02
30	211	206.24	2.26	200.93	4.77	206.43	2.17

shows the experimental results, in which we only illustrate the values of R channel by randomly selecting 30 pixels with null-values. Since the results of G and B channels are similar to R channel, they are not given in Table 2.

As shown in Table 2, only 3 of the 30 pixels, whose relative error of the estimated value, are minimal by using B_LIN_EST. In contrast, there are 15 pixels with minimal relative error for the method S_LUT_ELM, and 12 pixels for the method M_LUT_ELM. These results mean the method based on ELM is more powerful than the best linear estimator. Although the performance of S_LUT_ELM is relatively higher than M_LUT_ELM for R Channel, that does not mean S_LUT_ELM is better than M_LUT_ELM. Since the reconstructed contone color values depend on three-color components, we use the average relative error of three-color components to evaluate the performance of S_LUT_ELM and M_LUT_ELM. The results are shown in Figure 5, in which the average relative error of 17 of the 30 pixels is minimal for the method M_LUT_ELM. From Figure 5, we can conclude that using multi-color channels correlation can improve the precision of the estimated values.

4.3. Comparison with State-of-the-Art Methods

To the best of our knowledge, only the method proposed by Mese et al. in References [13,14] considers the problem of multi-color components correlation. Thus, we mainly compare our proposed method with this method, named Mese’s C_LUT. For Mese’s C_LUT, the correlation of multi-color components is considered, and the best linear estimator is employed. Moreover, the method based on wavelet in reference [24] is also compared with the proposed method. The method in reference [24] is an improved method based on the method proposed by Luo [6]. In the experiment, the PSNR (Peak Signal to Noise Ratio) and SSIM (Structural Similarity Index Measurement) are used to evaluate the performance of restored continuous color images. The experimental results are listed in

Table 3. Performance comparison of the proposed method with other methods (PSNR/SSIM).

Method	Wavelet Method [24]	Mese’s C_LUT	M_LUT_ELM	M_LUT_ELM+Filter
Airplane	25.47/0.68	26.16/0.71	26.40/0.72	27.75/0.84
Anhinga	23.15/0.62	25.21/0.65	25.38/0.65	25.57/0.72
Baboon	21.30/0.59	22.04/0.62	22.26/0.72	21.93/0.57
Bardowl	24.35/0.65	25.61/0.69	25.76/0.70	26.16/0.71
Bluheron	26.12/0.64	28.23/0.65	28.28/0.65	29.67/0.76
Bobcat	26.23/0.62	27.35/0.67	27.45/0.67	28.29/0.74
Lena	25.87/0.60	27.32/0.61	27.92/0.64	28.75/0.76
Parrot	25.64/0.52	27.32/0.46	27.98/0.50	28.63/0.60
Peppers	24.38/0.51	25.18/0.54	25.82/0.59	27.28/0.70
Sailboat	23.89/0.56	25.22/0.65	25.44/0.66	26.11/0.72
Average	24.64/0.60	25.96/0.63	26.27/0.64	27.01/0.71

, which shows the PSNR and SSIM of 10 classical images restored by the proposed method and other methods. In Table 3, our method M_LUT_ELM achieves the highest value for all of the 10 images. To further remove the noise in the smooth areas of the restored image, the average filter with a radius of 3 is used. Compared to the method of M_LUT_ELM in Table 3, the performance of all but one of the 10 images has improved on PSNR and SSIM by the method of M_LUT_ELM + filter. Especially, the average PSNR and SSIM obtain the best performance compared with the other methods.

Besides the objective evaluation, we applied high-definition color images to do subjective evaluation and comparison. Figure 6 gives some reconstructed continuous-tone images by using the proposed method and the method Mese’s C_LUT. From Figure 6, we can see that the visual effect of the recovered continuous-tone images by our method is higher than that of the method Mese’s C_LUT, in which more fine details are presented, such as Baboon’s beard, Parror’s feather on the head, and Lena’s brim et al. In addition, the enlarged view of Lena’s brim is given in Figure 6c, we can see that more details are recovered, and the number of noise points is reduced in flat areas by our method.

5. Conclusions

Inverse halftoning is an image recovery problem from degraded halftone images to corresponding continuous-tone images. Due to the differences in image data value and range between continuous-tone images and their halftone versions, inverse halftoning is an ill-posed problem. To better use look-up table-based method for color inverse halftoning, the correlation between color components is exploited in this paper by using ELM. The proposed method uses all the three-color component’s information that has the same halftone pattern to estimate the nonexistent value, which can improve the fitting precision of the estimated values. Moreover, ELM is introduced as the fitting model in this paper, which has not only the ability to represent the complex nonlinear relation between a halftone pattern and its contone value but also has the fast computation speed. The experimental results show the performance of our proposed method is superior to the state-of-the-art methods. In the future work, we plan to investigate more advanced machine learning technologies, such as deep learning, for promoting the practicality and the popularization of color inverse halftoning technique.