This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/).
In this paper, we present a blind image restoration algorithm to reconstruct a high resolution (HR) color image from multiple, low resolution (LR), degraded and noisy images captured by thin (< 1
TOMBO (Thin Observation Module by Bound Optics) imaging systems are a new generation of paperthin imaging products, which integrate microoptics, photosensing elements and processingcircuitry all on one single silicon chip (
We have previously proposed a novel spectralbased image restoration algorithm that require neither prior information about the imaging system nor the original scene [
The proposed spectralbased color image restoration method averages out all LR captured images, making the color channels globally independent of each other. Compared to previously reported color restoration techniques [
In this section, we extend the grayscale image restoration algorithm reported in [
Consider a TOMBO color system with (
“* *” represents the 2D convolution operator w.r.t
↓
Our main goal is to develop a restoration method that is able to reconstruct a HR, noiseless, color image of the original scene using only the (
The general model of the color TOMBO imaging system is described by
The system model in
In the above terms, the constant
By multiplying both sides of
Since the frequency bandwidth of the blurring PSF function
To minimize the effects due to interchannel crosscorrelation terms (crosstalks)
From
The above equation is valid under some constraints [
Under some constraints [
In this section, we employ the work formulated above to develop a color image restoration algorithm. Using
For restoring the image
For estimating the PSFs
where,
Practical considerations for the iterative algorithm implementation can be summarized as follows:
Pixel amplitudes that reach values greater than 255 are scaled using the following histogram normalization,
The mean value of the input image(s) and the output image is to be maintained (note that there are twice as many green pixels as red/blue pixel for the Bayer filter)
To resolve the problem of having zeros or nulls in the spectra, the following equation for the interpolated
For initialization, one of the images is used as an initial estimate of the HR image. The upsampling and interpolation process is done by zeropadding in the spatial domain between the image samples. Afterwards the FFT is applied. In the Fourier domain, a single spectrum is then taken out of the repetitive spectra using a low pass filter with cutoff frequency
We use the 2D fast fourier transform (FFT) to estimate spectra and cross spectra needed for the algorithm
The proposed HR color image restoration algorithm is detailed in
From
For a value of 0
Based on spectral estimation principals [
Thus
From the above analysis, it can be clearly seen that for any value of 0
To evaluate the performance of the proposed color image restoration algorithm, we consider the following tasks:
Restore a HR image from multiple blurred, LR and noisy “
Restore a HR image from multiple blurred, LR and noisy images and compare the results with the previous method [
Restore a HR image from multiple blurred, noisy “
In this section, we test the performance of our proposed algorithm in restoring a HR image from simulated TOMBO images of “Lena” [
In this section, we compare our algorithm with the advanced restoration methods developed by Sina in [
At high SNRs, our algorithm does not perform as well as Sina's, which enforces more constraints including (i) a penalty term to enforce similarities between the raw data and the HR estimate (data fidelity penalty term), (ii) a penalty term to encourage sharp edges in the luminance component of the HR image (spatial luminance penalty term), (iii) a penalty term to encourage smoothness in the chrominance component of the HR image (spatial chrominance penalty term), and (iv) A penalty term to encourage homogeneity of the edge location and orientation in different color bands (intercolor dependencies penalty term). In Sroubek's method, the following constraints are enforced: (i) a fidelity term, (ii) a smoothing term using variational integrals, (iii) a consistency term that binds the different volatile PSFs, (iv) a smoothing term to overcome the higher nullity of integerfactors, and (v) an anisotropic term for edge preservation. Although our algorithm only imposes two fundamental constraints, its performance is visually satisfactory, as seen in
Furthermore, we have observed that Sina's method experience some limitations when applied to images blurred using semigaussian PSFs. This could be due to the gaussian kernel used in Sina's approach. It is also important to point out that Sina's method uses multiple frames of an image with different displacements (i.e. diversity is guaranteed, see section VI in [
In this section, we investigate the performance of our proposed algorithm with real captured TOMBO color images. In this example, the captured images are that of a “teddy bear” picture, with a plan object located at 200 mm from the sensor module. Each unit imager has 60 × 60 pixels and each pixel is 6.25
A blind color image restoration method is proposed for the reconstruction of HR color images using multiple LR, degraded and noisy color images captured by TOMBO imagers. The proposed spectralbased method only imposes two fundamental image restoration constraints (positivity and support region) to (i) correct the blur that affects the captured LR images, (ii) minimize the interchannel crosscorrelations between RGB color components, (iii) significantly reduce the impact of additive noise, and (iv) reconstruct a HR color image. The computation complexity of the algorithm is low compared with existing techniques because it uses FFT for spectral estimation.
The proposed restoration algorithm has a rapid convergence rate of 10 to 20 iterations. Results show that the proposed algorithm is capable of restoring a HR image from the degraded LR color TOMBO images even when the SNER is as low as 5 dB. The proposed algorithm uses FFT and only two fundamental image restoration constraints, which makes it suitable for silicon integration with the TOMBO imager.
This work is supported by the Australian Research Council's Discovery Project DP0664909. The authors would like to thank Prof. Tanida and Dr. Ryoichi Horisaki for providing us with images of their experimental TOMBO imager. Thanks also go to Prof. Peyman Milanfar and Prof. Sina Farsiu for providing us with the lighthouse images and the MDSP resolution enhancement software; Dr Sroubek Filip and Dr Barbara Zitova for providing us with the BSR superresolution and blind deconvolution GUI. The authors would also like to thank Ms. Sabine Betts from Microelectronics Research Group (MRG) for her support.
The architecture of a color TOMBO imaging system.
Point operations categories.
System model for the color TOMBO system.
Simulation results, 6 × 6 images,
Simulation results, 6 × 6 images,
Simulation results for 12 blurred LR images of the lighthouse, in noiseless, and noisy, SNER=10.4616 dB,
Experimental results, 4 × 4 unit images,
Proposed color image restoration algorithm.




then estimate the original image by updating the estimates using 


Input parameters for simulated TOMBO images.
SNER  α_{2}  # Iterations  

6 × 6  60 × 60    7  240 × 240  0.1  10  0.1  20  
6 × 6  60 × 60  4.968 dB  7  240 × 240  0.1  10  0.1  20 
PSNR values (in dB) for the restoration methods.
Sina [ 
Sroubek [ 
This Work  

Noiseless  21.986  18.250  16.348 
Noisy  14.13  13.78  10.89 
Input parameters for real captured TOMBO images,
μ × μ  SNER  α_{1}1  α_{2}  # of Iterations  

4 ×4  60 × 0  15.774 dB  5  4  240  40  0.001  0.001  0.9  25 