A Color Restoration Algorithm for Diffractive Optical Images of Membrane Camera

In order to verify the technology of the membrane diffractive imaging system for Chinese next generation geo-stationary earth orbit (GEO) satellite, a series of ground experiments have been carried out using a membrane optical camera with 80 mm aperture (Φ80) lens. The inherent chromatic aberration due to diffractive imaging appears in the obtained data. To address the issue, an effective color restoration algorithm framework by matching, tailoring, and non-linearly stretching the image histograms is proposed in this letter. Experimental results show the proposed approach has good performances in color restoration of the diffractive optical images than previous methods. The effectiveness and robustness of the algorithm are also quantitatively assessed using various color deviation indexes. The results indicate that the chromatic aberration of diffractive images can be effectively removed by about 85%. Also, the proposed method presents reasonable computational efficiency.


Introduction
The geo-stationary earth orbit (GEO) satellite can provide remote sensing image data with wide observation range and high temporal resolution [1,2]. However, because of the high orbital altitude, achieving high spatial resolution is still a challenge for the GEO optical system. Theoretically, to achieve the meter-level resolution at the GEO height, the aperture of camera can be up to 20 m [3,4]. Considering the support and control systems, the size and weight of satellite would be extremely large. This is prohibitive for the engineering fabrication and deployment of traditional optical camera.
To alleviate the technique contradictions of GEO remote sensing system, a novel membrane optical system has been proposed and seen increasing interests in relevant studies [5][6][7][8][9]. Unlike the traditional reflective optics, the membrane camera adopts the transmissive diffractive imaging mechanism. The diffractive primary is manufactured by the macromolecule polymer material which has the merits of light weight, low cost, and high flexibility. Thus, large aperture but light weight of the optical imaging system can be achieved. Also, easy deployment (light and packable) of the membrane optical system virtually eliminates the tight surface shape tolerances and significantly reduces the complexity of the control architecture faced by conventional large reflecting apertures [9,10]. Several ongoing missions equipped with the diffractive membrane elements include the "Eyeglass" telescope mission, the Membrane Optical Imager Real-time Exploitation (MOIRE) mission and the FalconSat-7 mission [9,[11][12][13]. However, the inherent spectral dispersion and wavefront distortion of diffractive primary lead to prominent degradations of the diffractive images, e.g., image blurring, image hazing, and color distortion [14,15]. In addition, m to infinity. In lab measurements, the DMC has diffraction efficiency of about 25%. Before the ground experiments, the sensor of the DMC was strictly calibrated in the laboratory to make sure the systematic biases are completely eliminated. The DMC and imaging targets were loaded on a folding-jib overhead working truck or a meteorological observation tower for imaging at various distances and imaging conditions. Figure 1b shows the bar target loaded on the overhead working truck. Figure 1c shows the work scene of using the overhead working truck during the ground experiments at the Huailai remote sensing comprehensive experimental site in Hebei province of China. Figure 1d presents the meteorological tower of the Institute of Atmospheric Physics, Chinese Academy of Sciences, used in the third ground experiment for loading the DMC. Synchronously, a traditional lens with the focal length of 647 mm and the Φ80 mm aperture was also employed for reference imaging. The reference camera used the same back-end image processing system as the DMC. Particularly, in the third ground experiments, to simulate the imaging condition on the GEO orbit, the DMC was loaded on the meteorological tower, which is more than 300 m high. The targets were placed on the ground. The meteorological tower can only support one researcher to work onboard. Considering the weight of camera systems and manual operations at high elevation, it was not safe to frequently switch operations between the DMC and the reference camera. As a result, very few reference images were synchronously obtained in the third ground experiments.  During the ground experiments, more than 20,000 diffractive images were obtained with various imaging parameter settings. Basically, the obtained diffractive images can be divided into two categories: The images of artificial targets (including various bar and resolution targets, toy models, and printed images) and the images of natural scenes (random scenes around the experimental site). Based on these image data, the imaging quality of the DMC was assessed using the spatial resolution, the modulation transfer function (MTF), and the signal to noise ratio (SNR), etc. Numerical results indicate that the image degradations appear in the single-band DMC images, which are expected and similar to the MOIRE mission [9]. Previous studies have suggested various empirical and physical-based restoration methods for diffractive images [14,15]. After implementing certain restoration approaches to the degraded single-band images, we found the qualities of DMC images basically meet the mission requirements in terms of the spatial resolution, the MTF and the SNR. However, due to the missing of chromatic aberration correction system of the DMC, serious color distortions are observed in the color diffractive images. As interpreted previously, this is resulted from the spectral distortion of diffractive primary and the wavelength correlation of atmospheric scattering. The chromatic aberration can be up to larger than 60%. Particularly, in the third ground experiment, because of the misuse of optical filter, all produced images present greenish color. Figure 2 shows two color distorted images obtained in ground experiments and the corresponding histograms. It is seen that the RGB histograms remarkably disperse in the grayscale. Notably, in the third ground experiment, because of the misuse of optical filter, all produced images present greenish color. Therefore, such color deviations can be seen as the superimposition of artificial distortion and inherent chromatic aberration of diffractive imaging. The greenish distortion has dependency of imaging circumstance which cannot be systematically removed. In addition, the atmospheric scattering and relatively low diffraction efficiency also cause the hazing effect of images so that the image visual quality would be diminished. Therefore, it is significant to perform color restoration to the diffractive images before remote sensing applications. During the ground experiments, more than 20,000 diffractive images were obtained with various imaging parameter settings. Basically, the obtained diffractive images can be divided into two categories: The images of artificial targets (including various bar and resolution targets, toy models, and printed images) and the images of natural scenes (random scenes around the experimental site). Based on these image data, the imaging quality of the DMC was assessed using the spatial resolution, the modulation transfer function (MTF), and the signal to noise ratio (SNR), etc. Numerical results indicate that the image degradations appear in the single-band DMC images, which are expected and similar to the MOIRE mission [9]. Previous studies have suggested various empirical and physicalbased restoration methods for diffractive images [14,15]. After implementing certain restoration approaches to the degraded single-band images, we found the qualities of DMC images basically meet the mission requirements in terms of the spatial resolution, the MTF and the SNR. However, due to the missing of chromatic aberration correction system of the DMC, serious color distortions are observed in the color diffractive images. As interpreted previously, this is resulted from the spectral distortion of diffractive primary and the wavelength correlation of atmospheric scattering. The chromatic aberration can be up to larger than 60%. Particularly, in the third ground experiment, because of the misuse of optical filter, all produced images present greenish color. Figure 2 shows two color distorted images obtained in ground experiments and the corresponding histograms. It is seen that the RGB histograms remarkably disperse in the grayscale. Notably, in the third ground experiment, because of the misuse of optical filter, all produced images present greenish color. Therefore, such color deviations can be seen as the superimposition of artificial distortion and inherent chromatic aberration of diffractive imaging. The greenish distortion has dependency of imaging circumstance which cannot be systematically removed. In addition, the atmospheric scattering and relatively low diffraction efficiency also cause the hazing effect of images so that the image visual quality would be diminished. Therefore, it is significant to perform color restoration to the diffractive images before remote sensing applications.

Methodology
In this section, we are presenting a color restoration algorithm for the diffractive images with color distortion. The aim of the color restoration is to make the images in conformity with the human visual system, so as to improve their interpretation value. Here we firstly introduce several color deviation indexes as the theoretical basis and evaluation criteria of the image color restoration.

Color Deviation Indexes
The classic gray world assumption argues that the averages of digital number (DN) values over the entire image of R, G, and B channels are nearly equivalent. Thus, based on the GWA hypothesis, we propose a mean dispersion index (MDI) to evaluate the image color deviation. For an image with no color deviation, the GWA considers the differences among the mean values of DN for various channels are much smaller than the mean value of all DNs of the image [19]. Thus, we define the MDI as where I represents the DN value at each pixel. I R , I G and I B represent the mean DN values of each channel in the RGB color space. I S represents the summation of the mean DN values of all three channels. They are defined as where M denotes the number of image pixels. From the definition, one can see the MDI is normalized between 0 and 1. For a color image, the closer the MDI is to 0, the less color deviation the image has.

Histogram Overlap Area
Histogram overlap area (HOA) is a commonly used index for evaluating the color deviation of image [32]. It describes the consistency of three channels in RGB color space and has the definition as where h k and H k represent the probability density histogram and histogram of the corresponding channel, respectively. number() represents the statistic function for counting times. Unlike the MDI, an image with the HOA close to 1 has less color deviation.

CIEDE2000 Chromatic Aberration Coefficient
The CIEDE2000 is the newest standard chromatic aberration coefficient proposed by the international commission on illumination (CIE). It can quantitatively evaluate the difference between two colors with high accuracy. Moreover, the evaluation result is closer to the human visual system than previous coefficients. Thus, we employ the CIEDE2000 to assess the color restoration effect of the proposed algorithm. The computation of the CIEDE2000 is summarized in [33].

The HMTS Algorithm
To address the chromatic aberration issue of the DMC images as shown in Figure 2, we propose a color restoration algorithm framework in this letter. It incorporates the operations of matching, tailoring, and non-linearly stretching the image histograms. Thus, we name it the HMTS algorithm. The main idea of this algorithm is to exploit the spectral and statistic information of diffractive images to achieve large HOA and small MDI of the restored images. Specifically, the histogram matching (HM) is to improve the channel consistency that complies with the gray world assumption and human visual sense. Tailoring and stretching (TS) of channel histograms can suppress the noises induced by the HM and improve the image contrast. The implementing steps of the HMTS algorithm are listed as follows.

1.
Input the diffractive image with color deviation; and 2.
Compute the mean DN values of each channel (i.e., I R , I G , and I B ) using (2). 3.
Perform histogram matching, with selecting the channel with medium average as a reference, to the other two channels. Histogram matching is a commonly used method for image enhancement by matching the image histogram to a reference image. In this work, the inter-channel HM can significantly eliminate the luminance deviations among channels of diffractive images. Since the HM is a well-developed algorithm of image processing, one can refer to [18] for the details of HM. For the sake of simplicity, here we show the results of inter-channel HM for one of the test images shown in Figure 3. From the figure, one can see that the brightness of the original blue channel is significantly increased after the HM to the red channel. The histogram of blue channel is reformulated as that of the red one. However, as shown in Figure  3, it is also noted that the image noise of blue channel is also enlarged due to the HM operation. This phenomenon was also reported in [18]. These noises generally have very large or very small DN values. In other words, they concentrate at two ends of the grayscale histogram. A typical way to remove the image noises is to perform filtering. However, image filtering would be time consuming, particularly for the remote sensing images. Considering our motivation of supporting the real-time chromatic aberration correction system for the membrane diffractive camera, we utilize the histogram tailoring at two histogram ends to efficiently suppress the noise. Then, for the tailored histogram, we implement a non-linear histogram stretching procedure to improve the image contrast induced by the hazing effect of diffractive imaging. The details of histogram tailoring and stretching are illustrated in the following step.

4.
Tailor and non-linearly stretch the histogram of each channel using where f k (x, y) and g k (x, y) denote the DN of pixel at (x, y) of original and restored images for the corresponding channel. I k,bm and I k,tp are the bottom and top DN cutoffs which are generally determined by the cutting percentage between 0.01% and 2%. Here the cutting percentage is defined as the number of pixels with DN smaller than I k,bm and larger than I k,tp divided by the total pixel numbers. Thus, the DN cutoffs can be given by the gray cumulative histogram for a specific cutting percentage. For the non-linear histogram stretching of the tailored histogram between DN cutoffs, we follow [34] and use the quadratic transformation function to stretch the histogram, which is described as follows (I k,bm ) = αI 2 k,bm + βI k,bm + γ = 0 I k,str. (I k,mn ) = αI 2 k,mn + βI k,mn + γ = η 255 2 + (1 − η)I k,mn I k,str. I k,tp = αI 2 k,tp + βI k,tp + γ = 255 where I k,mn is the mean DN value of corresponding channel. η ∈ [0, 1] is the brightness weight which is selected at 0.4 in this study. There are two reasons to account for the process of histogram tailoring and stretching. Firstly, in the previous step of histogram matching, the image noise of the matching channel could be enlarged by matching to the reference channel. Those noise signals typically have extreme values. Thus, histogram tailoring at the two ends of grayscale histogram would be conducive to suppress the noise. Secondly, implementing the histogram tailoring also aims to improve the image contrast and definition to remove the hazing effect. From (7), it is noted that both histogram tailoring and stretching depend on the bottom and top DN cutoffs of histogram, i.e., I k,bm and I k,tp . The selection of the DN cutoffs determines the final color restoration effects. Theoretically, removing the hazing effect would increase the image depth of filed so that the image information would be increased. However, on the other hand, too much histogram tailoring would reduce the image information. Thus, the basic principle of selecting the histogram cutoffs is to reduce image noise and enhance the contrast to largest extent with losing less image information. As a result, we repeatedly perform histogram tailoring and non-linear stretching with various cutoffs to obtain the image with maximum information. The initial cutting percentage is given as 0.01%. Particularly, the information entropy (IE) is employed to evaluate the image information and be a stopping criterion. The IE is defined as [35] where p i denotes the probability of grayscale i, which can be calculated from the image histogram (see (5) and (6)) as follows By repeatedly performing this procedure of histogram tailoring and non-linear stretching using various cutting percentages in a grid one by one, the image with largest IE would be obtained. The corresponding cutting percentage can be used to remove noises but retain the useful information. The detailed algorithm of histogram tailoring and stretching is described as follows (Algorithm 1).

5.
Output the restored image.
stretching using various cutting percentages in a grid one by one, the image with largest IE would be obtained. The corresponding cutting percentage can be used to remove noises but retain the useful information. The detailed algorithm of histogram tailoring and stretching is described as follows (Algorithm 1).

Results and Evaluations
In Figure 4, the proposed HMTS algorithm is implemented to the color distorted DMC images. The results are also compared to those using the GWA, the PRM, the WPR, the RAWB, the SCB, the IWB, and the DWBE. In particular, the auto white-balance correction module in the IWB was practically utilized. The DWBE was trained on the dataset which is available on http://cvil.eecs.yorku.ca/projects/public_html/sRGB_WB_correction/ dataset.html (accessed on 1 February 2021). To comprehensively assess the performances of various methods in restoring the chromatic aberration of diffractive images, total 8 test diffractive images are used which involve all three ground experiments. For a fair and objective evaluation, we also present the corresponding reference images. However, for the test image 1 (warning board in the field) obtained in the first experiment, the reference camera unfortunately failed to follow up and capture image. In addition, as introduced in Section 2, due to the limitation of experimental environment, very few reference images were synchronously obtained in the third ground experiments. Thus, for those diffractive images with no reference images obtained, we used other images to show the original color of targets. Notably, the reference image is the digital image for printing, in which the colors are slightly different from those in the printed target for imaging. Table 1 presents the MDI and HOA indexes of original images and color restored images using various algorithms, which are presented in blue and red, respectively. Particularly, for the convenience of evaluation and discussion, the smallest two MDIs and the largest two HOAs are emphasized in black bold in Table 1.
From Figure 4, one could see that the proposed HMTS method in general has the best color restoration performance for the test diffractive images. The image visual effects are close to the reference images. This is also illustrated in Table 1 that the results of HMTS basically have smallest MDIs and largest HOAs. In addition, with considering the hazing effect due to low diffraction efficiency and atmospheric scattering, the restored images of HMTS present large image contrast and depth of field. For the test images 2 and 3, considering the cloudy and haze weather of the imaging circumstance (note this in the reference images), the diffractive images after color restoration are slightly overexposed due to the sky background. For the comparison methods, the performances of the GWA method are barely satisfactory. The chromatic aberrations in test images 1, 6, 7, and 8 are mostly removed by the GWA, which are also indicated by the color deviation indexes shown in Table 1. Notably, since the definition of MDI index obeys the basic assumption of the GWA, all the restored images using the GWA have very small MDI. Based on the visual evaluation and the quantitative indexes, the PRM and WPR also have relatively good performances for test images 1 and 7 and test images 4, 7, and 8, respectively. The RAWB method yields acceptable results for test images 1 and 4. However, it almost gives no improvements to the diffractive images with greenish color deviation obtained in the third ground experiment. For the SCB method, since it stretches the histograms of all channels as much as it can be, all the restored images have very large contrasts. In terms of the MDI, the SCB algorithm has relatively good performance for test images 6, 7 and 8. The IWB and DWBE also have different performances on various test images. It is seen that the IWB has better color restoration effects on test image 6 and 8, while the DWBE scores on test image 2 and 3. In addition, for the test image 5 (the white tower), it should be note that almost all comparison methods fail to give satisfying results. According to the HOA index, the restored result of test image 5 using the HMTS has much higher HOA than those of other comparison approaches.
In Table 2, we present the CPU time consumption of various algorithms to compare their computational efficiencies. The color restoration experiments are carried out on the device equipped with the Intel Core i7-8700 CPU at 3.20 GHz and 32 GB of memory capacity. According to the average time consumptions, one could see that the WPR is the fastest algorithm among the methods, followed closely by the PRM and our proposed HMTS. The DWBE and the RAWB have lowest computational efficiency. Considering the motivations of good color restoration performance and real-time processing capability for the CAC system development, the proposed HMTS algorithm has reasonable computational efficiency.    To further verify the robustness of the HMTS algorithm, more experiments are conducted on the diffractive images obtained from the third ground experiments with serious color deviation. In Figure 5, the HMTS algorithm is implemented to various resolution targets and image targets of ocean scenes and land scenes. The corresponding MDI and HOA of images are computed as well. By comparing the image before and after processing, one can see that the color of the diffractive images are well restored and the image visual effects are significantly improved. It is also noted that the HOAs increase and the MDIs decrease for all test images after implementing the HMTS algorithm. Considering all experimental images shown in Figures 4 and 5 and taking the statistical averages of the color deviation indexes, it is found that the average MDI decreases from 0.349 to 0.037, and the average HOA increases from 0.154 to 0.776, after implementing the color restoration with HMTS. These indicate that the HMTS algorithm has good performance and strong robustness of color restoration for diffractive images of various scenes and targets. To further verify the robustness of the HMTS algorithm, more experiments are conducted on the diffractive images obtained from the third ground experiments with serious color deviation. In Figure 5, the HMTS algorithm is implemented to various resolution targets and image targets of ocean scenes and land scenes. The corresponding MDI and HOA of images are computed as well. By comparing the image before and after processing, one can see that the color of the diffractive images are well restored and the image visual effects are significantly improved. It is also noted that the HOAs increase and the MDIs decrease for all test images after implementing the HMTS algorithm. Considering all experimental images shown in Figures 4 and 5 and taking the statistical averages of the color deviation indexes, it is found that the average MDI decreases from 0.349 to 0.037, and the average HOA increases from 0.154 to 0.776, after implementing the color restoration with HMTS. These indicate that the HMTS algorithm has good performance and strong robustness of color restoration for diffractive images of various scenes and targets. Moreover, we utilize a standard color-checker (SCC) chart produced by the X-Rite to further assess the effects of the HMTS in color restoration. The CIEDE2000 coefficients are computed referring to the SSC image taken by the reference camera. Moreover, several comparison methods, e.g., the GWA, the WPR, the SCB and the IWB, which have relatively good color restoration performances for the images obtained in the third ground experiment, are also taken into comparisons. In Table 3, comparisons of the SCC images and the corresponding CIEDE2000 coefficients before and after implementing color restorations are presented. As shown in the table, comparing to the other four methods, the SCC images after implementing the HMTS have closer visual effects to those taken by reference camera. The mean value of the CIEDE 2000 coefficients of SCC images after color restoration using the HMTS is only about one seventh of that before color restoration, which is the smallest one among the comparison methods. This indicates that about 85% of the color deviations in diffractive images have been removed. Moreover, we utilize a standard color-checker (SCC) chart produced by the X-Rite to further assess the effects of the HMTS in color restoration. The CIEDE2000 coefficients are computed referring to the SSC image taken by the reference camera. Moreover, several comparison methods, e.g., the GWA, the WPR, the SCB and the IWB, which have relatively good color restoration performances for the images obtained in the third ground experiment, are also taken into comparisons. In Table 3, comparisons of the SCC images and the corresponding CIEDE2000 coefficients before and after implementing color restorations are presented. As shown in the table, comparing to the other four methods, the SCC images after implementing the HMTS have closer visual effects to those taken by reference camera. The mean value of the CIEDE 2000 coefficients of SCC images after color restoration using the HMTS is only about one seventh of that before color restoration, which is the smallest one among the comparison methods. This indicates that about 85% of the color deviations in diffractive images have been removed.