Next Article in Journal
Real-Time Traffic Risk Detection Model Using Smart Mobile Device
Next Article in Special Issue
Polarized Light Field Imaging for Single-Shot Reflectance Separation
Previous Article in Journal
Target Localization and Tracking by Fusing Doppler Differentials from Cellular Emanations with a Multi-Spectral Video Tracker
Previous Article in Special Issue
Optimized Multi-Spectral Filter Array Based Imaging of Natural Scenes
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:

Survey of Demosaicking Methods for Polarization Filter Array Images

Centre de Recherche en Informatique, Signal et Automatique de Lille, Université de Lille—CRIStAL, UMR 9189, 59655 Villeneuve d’Ascq CEDEX, France
Institut de Recherche en Informatique, Mathématiques, Automatique et Signal, Université de Haute-Alsace—IRIMAS, EA 7499, 68093 Mulhouse CEDEX, France
Author to whom correspondence should be addressed.
Sensors 2018, 18(11), 3688;
Submission received: 19 September 2018 / Revised: 23 October 2018 / Accepted: 26 October 2018 / Published: 30 October 2018
(This article belongs to the Special Issue Snapshot Multi-Band Spectral and Polarization Imaging Systems)


Snapshot polarization imaging has gained interest in the last few decades. Recent research and technology achievements defined the polarization Filter Array (PFA). It is dedicated to division-of-focal plane polarimeters, which permits to analyze the direction of light electric field oscillation. Its filters form a mosaicked pattern, in which each pixel only senses a fraction of the total polarization states, so the other missing polarization states have to be interpolated. As for Color or Spectral Filter Arrays (CFA or SFA), several dedicated demosaicking methods exist in the PFA literature. Such methods are mainly based on spatial correlation disregarding inter-channel correlation. We show that polarization channels are strongly correlated in images. We therefore propose to extend some demosaicking methods from CFA/SFA to PFA, and compare them with those that are PFA-oriented. Objective and subjective analysis show that the pseudo panchromatic image difference method provides the best results and can be used as benchmark for PFA demosaicking.

1. Introduction

Polarization imaging is a way to analyze the particular direction of oscillation of the electric field described by the light. In opposition with conventional color or multispectral imaging that sample the spectral information, polarization imaging considers the electric field as a vector. Such a vector field is contained in a plane perpendicular to the direction of propagation. As the wave travels, it can oscillate in one particular direction (linear polarization), or in an ellipse (elliptic or circular polarization). Values of polarization images depend on the polarization properties of both the light source and the objects that compose the observed scene. The light can be partially polarized or unpolarized, resulting from either a rapidly changing state of polarization, or an interference effect of polarization.
Several polarization imaging systems, called polarimeters, have been developed in the last past few decades for recovering the polarization state of a lighted scene from few acquisitions. Such systems combine a standard panchromatic imaging device with polarizing optics, e.g., polarization filter, liquid crystal modulator, or prism. Reviews of recent polarimeters have been achieved in the literature [1,2]. The most simple optical setup consists in the rotation of a linear polarization filter at several polarization angles in front of a camera. After a preliminary calibration step (radiometric and polarimetric), the polarization states of the incoming irradiance that reaches the sensor can be estimated. However, this setup is sequential and slow, since several image acquisitions at different filter orientations are required to recover the polarization states of a single scene. To overcome this limitation, Polarization Filter Array (PFA) imaging provides a way for snapshot acquisition that could be useful for many imaging applications. It is an extension of the so-called Color Filter Array (CFA) and Spectral Filter Array (SFA) technologies that previously came on the market. We will briefly review the CFA and SFA technologies and concepts, before to introduce the specificities of PFA.
The CFA technology [3] has quickly become the standard for one-shot color imaging. The technology is lightweight, cheap, robust, and small enough to be embedded in imaging systems. It is composed by a single silicon sensor fitted with a CFA, so that each sensor site senses only one spectral band according to the CFA. A demosaicking procedure is therefore required to recover the incomplete color samples per site. Such procedure uses reflectance properties of acquired images in order to recover the missing color components at each pixel position. Properties of reflectance consist in high spatial correlation in homogeneous areas that constitute an object, and spectral correlation between different channels. The widely-used Bayer CFA for instance samples the green band at half of sites, which makes it a prominent candidate to begin the demosaicking process using spatial correlation. Spectral correlation is then generally assumed in order to estimate red and blue channels using the well estimated green channel. The demosaicking algorithm has to be carefully selected since color reconstruction quality is highly affected by its artifacts, such as blur, zipper effect, etc.
The last past few decades have seen the emergence of an extension of CFA with more than three channels: The SFA technology [4,5,6]. Supplementary channels are generally required for applications that need good color reproduction [7], illuminant estimation and spectral adaptation [8], reflectance reconstruction [9], etc. SFA design considers a trade-off between spatial resolution for spatial reconstruction in the demosaicking process, and spectral resolution for spectrum reconstruction. Thus, some SFA demosaicking algorithms privilege spatial resolution by sampling a dominant channel that represents half of pixels [10] (as for the Bayer CFA), while other privileges spectrum reconstruction by maximizing the number of channels [11].
Polarization Filter Array (PFA) technology has been patented in 1995 [12], but most of the practical implementations and technology advances were made from 2009 to nowadays. Manufacturing processes are various and done by designing metal wire grid micro-structures [13,14,15], liquid crystals [16,17], waveplate array of silica glass [18], or intrinsically polarization-sensitive detectors [19,20]. Some of the most evolved PFA structures are presented as being bio-inspired, and implement additional features, e.g., mixing spectral and polarization feature recovery [21], high dynamic range [22], or motion detection [23].
The PFA is composed of pixel-size linear polarizers oriented at four different angles ( 0 , 45 , 90 , and 135 are the polarization orientations employed in most of the PFA cameras), superimposed on a camera sensor chip, as shown in Figure 1. In front of the sensor, the PFA samples the polarization direction by filtering the incoming irradiance according to polarizer angles. Therefore, each pixel measures the intensity coming from only 1 of the 4 different polarizers. Some PFA cameras appear on the market, like the Polarcam device from 4D Technology [24], and more recently, the IMX250MZR polarization-dedicated sensor from SONY (Tokyo, Japan). Both PFA use the same filter arrangement that is described in Figure 1. But the SONY sensor that comes in 2018 is particularly cheap, and holds the polarization matrix bellow the lens array, which limits the cross-talk effect in adjacent pixels [6]. Moreover, as it was previously done for other computational imaging [25,26] and computer vision [27] algorithms, Lapray et al. [2] have recently proposed an implementation of a real-time polarization imaging pipeline using an FPGA.
Demosaicking PFA images aims to retrieve the full resolution images that represent the four polarization channels. Stokes imaging is a tool that uses these channels to represent in an efficient way the linear and circular state of polarization of the incoming light. Thus, the final goal of demosaicking is to minimize errors and artifacts in the reconstructed Stokes parameters and the derived descriptors. The Degree Of Linear Polarization ( D O L P ) and the Angle Of Linear Polarization ( A O L P ) descriptors are computed from the first three Stokes parameters of the Stokes vector S . In this work, we limit ourself to the linear case, as the most of existing PFA are based only on linear polarizers (but some existing tentatives add plasmonic quarter-wave retarders to retrieve the circular polarization component [28]). Let us consider the intensities of light measured I 0 , I 45 , I 90 , I 135 after the light is filtered by linear polarization filters (oriented at 0°, 45°, 90°, and 135°). In the literature, the choice of these 4 angles forms an optimal system for polarization imaging in the presence of noise, as described in [29]. The mathematical formulations for Stokes parameters and descriptors are as follows:
S = S 0 S 1 S 2 0 = I 0 + I 90 I 0 I 90 I 45 I 135 0 ,
D O L P = S 1 2 + S 2 2 S 0 ,
A O L P = 1 2 arctan S 2 S 1 ,
The total incoming irradiance is represented by S 0 , S 1 is the horizontal/vertical polarization difference, whereas S 2 is the +45/−45° polarization difference. If we consider that channels I k , k { 0 , 45 , 90 , 135 } are normalized intensity values comprised between 0 and 1, S 1 and S 2 have values between 1 and + 1 . A O L P values are scaled in the range [ 0 , 180 ] , whereas D O L P values are scaled in the range [ 0 , 1 ] , and are often expressed in percentage of polarized light.
It is useful to note that a radiometric calibration is very important in case of polarimetric imaging, even more than for color imaging, as the different channel errors are coupled, and thus it can invalidate greatly the parameter estimation [1]. An example of a complete 2D Stokes processing starting from a PFA image is given in Figure 2.
The purpose of this paper is to first study the correlation properties of polarization channels and their similarities with those of spectral channels. Then to review some existing interpolation strategies dedicated to filter array imaging, i.e., CFA, SFA, and PFA. Finally, we propose to evaluate objectively the methods and those we have adapted to the PFA case, in the special context of PFA. A diagram of the proposed analysis is shown in Figure 3. We organize the paper as follows. First, a data correlation study across the polarization channels is presented in Section 2. Next, different CFA, SFA, and PFA interpolation techniques are presented in Section 3. Results and discussion of the surveyed methods is proposed in Section 4. The paper ends with several conclusions in Section 5.

2. Polarimetric Channel Correlation Study

All demosaicking methods estimate missing values using spatial (intra-channel) (i) and/or inter-channel (ii) correlation assumptions. (i) The spatial correlation assumes that; if a pixel p and its neighborhood belong to the same homogeneous area, the value of p is strongly correlated with the values in its neighborhood. Thus, assuming that a channel is composed of homogeneous areas separated by edges, the value of a pixel can be estimated by using its neighbors within the same homogeneous area. Spatial gradients are often used as indicators to determine whether two pixels belong to the same homogeneous area. Indeed, gradient considers the difference between values of two spatially close pixels. We can therefore assume that these pixels belong to the same homogeneous area if the gradient is low, and that they belong to different homogeneous area otherwise. (ii) The inter-channel correlation (also called spectral correlation in CFA and SFA imaging) assumes that the high frequencies (textures or edges) of the different channels are strongly correlated. If the filter array contains a spatially dominant band, demosaicking generally estimates the associated channel whose high frequencies can be faithfully reconstructed, then uses it as a guide to estimate other channels. The faithfully reconstructed image can be used to guide the high frequency estimation within the different channels [30].
Usual PFA demosaicking methods assume only spatial correlation, thus disregarding correlation among polarization channels. In order to extend CFA and SFA demosaicking methods that also use the inter-channel correlation to PFA images, we propose to compare the spatial and inter-channel correlations in multispectral images with those of polarization images. For this purpose, we use the database proposed in [31]. Images were acquired by the single-band near-infrared sensor from the JAI AD080 GE camera, coupled with a linear polarizer from Thorlabs (LPNIRE100-B). A precision motorized rotation stages (Agilis™ Piezo Motor Driven Rotation Stages) allowed to take the four images at four orientation angles ( [ I 0 , I 45 , I 90 , I 135 ] T ). A registration procedure aligned the images [32] pixel-to-pixel. The images were also calibrated with respect to the spatial deviation of the illuminant and the non-linearities. There are ten multispectral images, each one being provided with four different polarization angles k { 0 , 45 , 90 , 135 } . Scenes imply different objects with materials like fabrics, plastics, food, color checkers, glass, and metal. Conditions of acquisition are constant for all scenes, i.e., constant illuminant source (tungsten halogen source) and location, constant field of view and constant lens aperture. Multispectral recoverred images are composed of six spectral channels: Five channels are associated with the visible domain, whereas one channel is associated with the Near-InfraRed domain (NIR). The six spectral channels C u are arranged so that their associated spectral band wavelengths increase with respect to u { 1 , , 6 } .
Let us first study the properties of multispectral images with respect to the polarization angle of analysis. For this purpose we assess the spatial correlation within a given channel C u using the Pearson correlation coefficient (PCC) between the value C p u of each pixel p and that of its right neighbor C q u at spatial distance 2. This coefficient is defined as [33]
P C C [ C u ] = p ( C p u μ u ) ( C q u μ u ) p ( C p u μ u ) 2 p ( C q u μ u ) 2 ,
where μ u is the mean value of channel C u . We also assess the inter-channel correlation using the PCC between any pair of spectral channels C u and C v , ( u , v ) { 1 , , 6 } 2 as
P C C C u , C v = p ( C p u μ u ) ( C p v μ v ) p ( C p u μ u ) 2 p ( C p v μ v ) 2 .
Note that in Equations (4) and (5), we select a centered area excluding the 16 pixels on the image borders to avoid border effects, that are induced by the registration step used on raw images (described in [31]). Moreover the choice of 16 border pixels is done to match with the experimental assessment (see Section 4) of demosaicking methods presented in Section 3.
Table 1 is the spatial correlation within each spectral channel and the inter-channel correlation between the six spectral channels according to each of the four polarization angles. Table 1 shows that the spatial correlation is relatively high (0.9504 on average over all channels and polarization angles), which validates the use of the spatial correlation assumption for both SFA and PFA demosaicking. According to Table 1a,d, the spatial correlation has the same behavior for the four polarization angles. It also highlights that the channel C 4 has low spatial correlation. We believe that it is due to the database acquisition setup, which uses the dual-RGB method leading to unbalanced channel sensitivities. In this configuration, the spectral sensitivity function associated with the channel C 4 is lower than other channels over the spectrum. Thus, all channels don’t share the same noise level, and poor information recovery (especially for C 4 ) could lead to low correlation values.
Regarding the spectral inter-channel correlation, the usual behavior is that close spectral channels in term of wavelength band are more correlated than distant ones, and channels in the visible are weakly correlated with the near-infrared channel [11]. Except the channel C 4 that exhibit low correlation values, this behavior is observed in Table 1. Indeed, P C C ( C 1 , C 2 ) > P C C ( C 1 , C 3 ) > P C C ( C 1 , C 5 ) > P C C ( C 1 , C 6 ) for instance. Moreover the correlation between the NIR channel C 6 and other channels is low (ranges on average between 0.7953 and 0.8787), while the correlation between channels in the visible domain reaches up to 0.9596 (correlation between C 2 and C 3 ). Table 1a,d show that the inter-channel correlation depends on the polarization angle. Indeed, Table 1a has values close to Table 1d, whereas Table 1b has values close to Table 1c. We can therefore expect that the polarization channels at 0° are more correlated with those at 135° than those at 45° or 90°.
Now, let us consider the polarization images composed of four polarization angles for a given spectral band. The spatial and inter-channel correlations are assessed using the PCC applied respectively to channels I k , k { 0 , 45 , 90 , 135 } (see Equation (4)), and to any pair of polarization channels I k and I l , ( k , l ) { 0 , 45 , 90 , 135 } 2 (see Equation (5)).
Table 2 is the average polarization correlation between the four channels of polarization images, according to each of the six spectral bands. Results highlight that the spatial correlation is high and does not depend on the considered spectral band (except for channel C 4 ). Results also confirm that channel I 0 is highly correlated with channel I 135 and channel I 45 is highly correlated with channel I 90 . In general terms, inter-channel correlation between polarization channels is higher than inter-channel correlation between spectral channels (see Table 1). Indeed, if the incoming irradiance is not polarized, the associated pixel has only the information of the total intensity divided by two, that is the same from one channel to another.
Since the inter-channel correlation is high in polarization images, we propose to apply SFA demosaicking schemes based on inter-channel correlation assumption on PFA images. For this purpose, we can choose the four polarization channels associated to any spectral band but not the one associated to C 4 . Since dual-RGB method is not applied for the channel C 6 , we selected it for the experimental assessment in Section 4.

3. State-of-the-Art

3.1. Demosaicking Problem and Properties

A PFA camera provides a raw image I r a w with X × Y pixels, in which a single polarization angle k { 0 , 45 , 90 , 135 } is available at each pixel p according to the PFA arrangement. Let S be the set of all pixels (with a cardinal of | S | = X × Y ) and S k be the pixel subset where the PFA samples the angle k, such that S = k { 0 , 45 , 90 , 135 } S k . A PFA can be defined as a function P F A : S { 0 , 45 , 90 , 135 } that associates to each pixel p its polarization angle. Therefore the pixel subset where the PFA samples the polarization angle k can be defined as S k = p S , P F A ( p ) = k . The raw image I r a w can then be seen as a sampled version of the fully-defined reference image I = { I k } k { 0 , 45 , 90 , 135 } (that is unavailable in practice) according to the PFA:
p S , I p r a w = I p P F A ( p ) .
The raw image can also be seen as the direct sum of four sparse channels I ˜ k , k { 0 , 45 , 90 , 135 } that contains the available values at pixel positions in S k and zero elsewhere:
I ˜ k = I r a w m k ,
where ⊙ denotes the element-wise product and m k is a binary mask defined at each pixel p as:
m p k = 1 if P F A ( p ) = k , i . e . , p S k , 0 otherwise .
Demosaicking is performed on each sparse channel I ˜ k to obtain an estimated image I ^ = { I ^ k } k = 1 K with four fully-defined channels, among which three are estimated at each pixel p. For illustration purpose, Figure 4 shows the demosaicking problem formulation for a PFA raw image.
In the following, we review the demosaicking methods dedicated to PFA. We also review those dedicated to CFA/SFA that can be used or adapted to our considered PFA. All these methods were either re-coded, adapted to PFA, or kindly provided by authors (online or in private). See Table 3 for an overview of all methods.

3.2. PFA Demosaicking

Among PFA demosaicking methods, we exclude the learning-based methods [46], since they require well-adapted dictionaries, and methods that exploit multiple sampling of the raw data [47]. We also exclude the gradient-based method [48], since a SFA method has a very close behavior (BTES).

3.2.1. Bilinear with 5 Different Kernels (B)

Bilinear interpolation dedicated to PFA was firstly investigated by Ratliff et al. [34]. They employ three different bilinear and two weighted bilinear kernels (see Figure 5). Bilinear interpolation is one of the most commonly used technique due to its low computational complexity. It is based on space-invariant linear filtering. Two kinds of bilinear interpolations exist. One uses a linear combination of neighboring pixel values using equal weights. Another employs non-equal weights in accordance to the Euclidean distance between the interpolated pixel location and centers of neighboring pixels. The subtractive nature of the Stokes vector processing results in strong edge artifacts in the reconstructed images. Based on this assumption, authors define the term of Instantaneous Field-Of-View (IFOV), which is the local deviation between an ideal full-resolution polarimeter and the interpolated PFA pixel responses at each position. They evaluate the interpolation performance of the methods using synthetic data, in the frequency domain of the reconstructed Stokes images. As evaluation metrics, they computed the modulation and intermodulation transfer functions in the descriptor images, along with the DOLP Mean Squared Error (MSE). It is found that the larger the size of the kernels becomes, the more DOLP artifacts are reduced, at the cost of loosing the high spatial frequency features. They found that the 12-pixel neighborhood kernel ( B 4 in the Figure 5) gives the best performance in term of IFOV removal. For algorithm implementations, we used the same weights as the original paper for the two weighted bilinear kernels.

3.2.2. Linear System (LS)

Tyo et al. [35], in 2009, elaborates a new method to reconstruct the first three Stokes parameters directly from the mosaicked image, without estimating I ^ . The four polarization images I ^ 0 , I ^ 45 , I ^ 90 , and I ^ 135 are thus not available with this method. The philosophy starts from the analysis of a raw PFA image in the frequency domain. By doing the discrete 2D Fourier transform, they define the spatial low-pass and high-pass filters. They assume that S 0 , S 1 , and S 2 are spatially band limited in the frequency domain. The centering baseband of the Fourier transform represents S 0 , whereas the horizontal and vertical sidebands represent S 1 + S 2 and S 1 S 2 respectively. They could then reconstruct S 1 and S 2 after applying the filters in the Fourier domain, and by doing the inverse Fourier transform of images.

3.2.3. Adaptive (A)

An extension of the bilinear interpolation was proposed by Ratliff et al. [36]. In this work, the principle is inspired by Ramanath et al. [49]. The loss of high frequency features in bilinear interpolation techniques is compensated by local computation using a 3 × 3 filtering. First, S 0 is approximated using not only I 0 and I 90 , but the four available neighboring intensities, as it is suggested in the literature [50]. A 2 × 2 low-pass filtering of the raw PFA image is performed with the kernel as follows:
h S 0 = 1 2 · 1 1 1 1 .
Then, intensity similarity masks and Euclidian distance masks are computed, in such a way that the weights are higher for pixels that have similar intensity values within a close neighborhood. These local interpolation weights are computed at each position in the image, and avoid interpolation across edges, and thus preserve high frequency contents. Results show that IFOV artifacts and false edges are minimized in the DOLP image, while high spatial frequencies are preserved. Only a subjective evaluation of their algorithm is performed in the article. The parameter ρ i in the paper was selected to be equal to 0.3 in our implementation.

3.2.4. CuBic (CB) and Cubic-SPline (CBSP)

An article was published by Gao et al. [37] to compare bilinear, weighted bilinear, cubic, and cubic-spline. In our work, the cubic and bicubic interpolation methods have been implemented using built-in functions from Matlab software (The MathWorks, Inc., Natick, MA, USA). Cubic interpolation uses the third order polynomial fitting to interpolate an area delimited by four corners, and uses three directional derivatives (horizontally, vertically and diagonally) as input. The cubic-spline method is a sequence of an horizontal interpolation and a vertical interpolation. Polynomial fitting (third order) is also used to reconstruct missing values from adjacent pixels, but with the additional constraint that the first and second derivative at the interpolation points are continuous. A modulation transfer function study is done to investigate on how the high spatial frequencies are preserved. A visual and objective evaluation (using MSE) are done on real data. Main results show that the cubic-spline methods performed the best in terms of visual artifacts removal and MSE. It appears that bilinear and weighted bilinear give the worst results.

3.2.5. Intensity Correlation among Polarization Channels (ICPC)

Another method by Zhang et al. [38] takes advantage of the correlations in PFA to enhance the spatial resolution of images. Spatial and polarization correlations between channels are investigated in a close pixel neighborhood, directly in the raw PFA image. Edges can not be accurately distinguished if the incoming light is polarized at some degree. Thus, in their work, edge orientations are estimated using the intensity correlation. They start by computing correlation errors from the assumption that edges have poor correlation within the pixel neighborhood. The correlation error magnitude reflects the presence of a homogeneous zone, or of a horizontal, vertical, or diagonal edge. For the interpolation in itself, a diagonal interpolation is firstly done by applying a bicubic-spline interpolation. Then, horizontal and vertical interpolation are performed by bicubic-spline interpolations, according to the correlation errors previously computed. Evaluation of their method is done by constructing a set of four ground truth polarization images using a linear polarizer rotated at four different angles. They found that their method performs better visual results, and has better RMSE compared to bilinear, bicubic, bicubic-spline, and gradient-based methods.

3.3. CFA Demosaicking

Bayer CFA has a dominant green band that represents half of pixels, and is used as a guide containing the high spatial frequencies. Therefore, CFA demosaicking methods generally estimate the green channel first in order to use the spectral correlation by considering that green channel is strongly correlated with blue and red channels. Here, we extend residual interpolation methods [39,40] from the CFA to the PFA pattern by considering the intensity image S 0 as a guide instead of the estimated green channel.

3.3.1. Residual Interpolation (RI)

Kiku et al. [39] propose a demosaicking scheme based on the residual interpolation. Their method requires a well estimated guide image, i.e., the estimated green channel that is dominant in the Bayer CFA raw image. Since there is no dominant band in our considered PFA, we adapt their method by using the intensity image S 0 as a guide. It is well estimated from a simple 2 × 2 bilinear kernel (see Equation (9)). Each channel I ^ k is then recovered by following these successive steps:
  • It computes a tentative estimated channel I ˇ k by applying the guided filter [51] and the guide image to the sparse channel I ˜ k . Note that such process modifies the raw values in the tentative estimated channel I ˇ k .
  • It computes the residuals defined by a difference between I ˜ k and tentatively estimated channel I ˇ k at pixels in S k .
  • It performs a bilinear interpolation of the residuals by using B 3 filter.
  • The finally estimated channel I ^ k is given by the summation of the tentative estimated channel I ˇ k and the interpolated residuals.

3.3.2. Adaptive Residual Interpolation (ARI)

Monno et al. [40] improve the RI by applying a Laplacian filter on I ˜ k and the guide before using the guided filter. The parameters for RI and ARI implementations are h = 5 , v = 5 , and ϵ = 0 .

3.4. Spectral Demosaicking Methods for a 2 × 2 Pattern

Among SFA demosaicking methods, we exclude learning-based methods since they require fully-defined images [30,52,53], and methods that assume sparsity of the raw data [54,55,56].

3.4.1. Binary-Three Edge Sensing (BTES)

In our knowledge, BTES interpolation [41] is the first SFA demosaicking method applicable on PFA raw images. This method improves the bilinear interpolation by considering weights inversely proportional to the directional gradient. It follows two steps, in which only a subset of pixels are estimated as shown in Figure 6. In a first step, a quarter of pixels are estimated using their four closest neighbors weighted with respect to the diagonal gradients. In a second step, the rest of the pixels ( c a r d ( S ) 2 ) are estimated using their four closest neighbors weighted with respect to horizontal (for an horizontal neighbor) or vertical (for a vertical neighbor) gradients.
As bilinear interpolation, this method is only based on spatial correlation since there is no dominant channel. Other SFA demosaicking methods also consider the inter-channel correlation to estimate the missing channels.

3.4.2. Spectral Difference (SD)

Brauers and Aach [42] estimate missing values of a channel using the inter-channel correlation. They consider the available value in the raw image at the missing position, i.e., a pixel p of a channel I k is estimated using the information of channel I P F A ( p ) as follows:
  • It computes the sparse difference channel Δ ˜ k , P F A ( p ) between channel I k and the channel I ^ B 3 P F A ( p ) estimated by bilinear interpolation (using filter B 3 ) at pixels in S k .
  • It estimates the fully-defined difference channel Δ ^ B 3 k , P F A ( p ) by bilinear interpolation.
  • The value of I ^ p k is given by the sum between the difference channel Δ ^ B 3 k , P F A ( p ) and the available value at p in the raw image.
Mizutani et al. [57] further improve this method using an additional assumption: Spectrally close channels are more correlated than distant ones. Since this assumption is not validated for polarization images, we cannot use it in this context.

3.4.3. Vector Median (VM)

Wang et al. [43] consider that each pixel of an image as a vector with four dimensions. For each pixel p, the method defines many pseudo-pixels by column vectors ( [ I p 0 , I p 45 , I p 90 , I p 135 ] T in our case) according to the mosaic, and it affects the median pseudo-pixel to p. The pseudo-pixels at p represents all the possible combinations of the four channels in a 5 × 5 neighborhood around p. The four values of a pseudo-pixel are taken from spatially connected pixels. To preserve value discontinuities and color artifacts, authors also propose a post-processing in a 3D-spheric space.

3.4.4. Discrete Wavelet Transform (DWT)

Wang et al. [44] extend the DWT-based CFA demosaicking to SFA demosaicking. By considering an image as low-frequency (homogeneous areas) and high-frequency contents (edges), This approach assumes two things: The low-frequency content is well estimated by bilinear interpolation, and the high-frequency contents have to be determined more accurately and have to be the same in different channels. The algorithm first estimates a fully-defined multispectral image I ^ B 3 by bilinear interpolation, then applies five successive steps to each channel I ^ B 3 k as follows:
  • It decomposes I ^ B 3 k into K Down-Sampled (DS) images, so that each DS image is composed of pixels in S l , l { 0 , 45 , 90 , 135 } . Note that one DS image is only composed of raw values.
  • It decomposes each DS image into spatial frequency sub-bands by DWT using Haar wavelet D2.
  • It replaces the spatial high-frequency sub-bands of all estimated DS images by those of the corresponding DS images of the mid-spectrum channel, assuming this is the sharpest one. In PFA images, there is no mid-spectrum channel, we therefore propose to use arbitrarily the I ^ b 3 90 channel.
  • DS images are transformed by inverse DWT.
  • It recomposes the full-resolution channel I ^ k from the four transformed DS images.

3.4.5. Multi-Local Directional Interpolation (MLDI)

Shinoda et al. [45] combine BTES and SD approaches into the MLDI method that follows the two steps of BTES. Each pixel is estimated using the difference planes, as in SD scheme. Moreover, instead of simply estimating the fully-defined difference planes by bilinear interpolation, the authors use directional gradients (following the step in BTES scheme), which improves the estimation. Shinoda et al. [45] also propose a post-processing that updates each estimated channel by taking into account the previous estimated values.

3.4.6. Pseudo-Panchromatic Image Difference (PPID)

The Pseudo-Panchromatic Image (PPI) is defined in each pixel as the average of all channels. By assuming that PPI values of neighboring pixels are strongly correlated, Mihoubi et al. [11] estimate the PPI from the PFA image by applying an averaging filter M proposed in [58]. Such filter estimates the PPI as the average value of all channels in a given neighborhood of each pixel. For this purpose, it takes all channels into account, while being as small as possible to avoid estimation errors. For our considered PFA arrangement, the filter M is adapted as:
M = 1 16 · 1 2 1 2 4 2 1 2 1 .
In the case of strong spectral correlation ( 0.9 ), authors propose to restore the edges of the estimated PPI using directional gradients. However, the condition is not validated for PFA images. The estimated PPI is thereafter used in a PPI difference scheme that estimates each channel k as follows:
  • It computes the sparse difference channel Δ ˜ k , P P I between channel I k and the PPI at pixels in S k .
  • It estimates the fully-defined difference channel Δ ^ k , P P I by weighted bilinear interpolation in which the weights are inversely proportional to the gradients computed from the raw image.
  • The finally estimated channel I ^ k is the sum between the estimated PPI and the difference plane.

3.4.7. Pseudo-Panchromatic Image based Discrete Wavelet Transform (PPIDWT)

The PPI has similar information to the mid-spectrum channel, and it is better estimated. Mihoubi et al. [11] therefore propose to replace the spatial high-frequency sub-bands by those of the PPI instead of I b 3 90 channel in the DWT scheme.

4. Performance Evaluation of Demosaicking Algorithms

4.1. Experimental Setup

PFA image simulation is employed to assess the interpolation strategies. As for the correlation study in Section 2, the polarimetric images from the database of Lapray et al. [31] was used as references.
All methods of Table 3 were either re-coded (R), adapted to PFA (A), or provided by authors in Matlab/ImageJ language software (P). They are further integrated into the framework presented in Figure 4 in order to assess and compare the performances of demosaicking. Stokes descriptors are then computed for both reference and estimated images, according to Equations (1)–(3). To avoid precision errors during image conversions, all considered images and processing are using 32-bit float data representation.
We consider the Peak Signal-to-Noise Ratio (PSNR) as quality metric. Between each couple of reference (R) and estimated (E) channel/descriptor, the PSNR is computed as follows:
P S N R ( R , E ) = 10 log 10 max p R 2 M S E ( R , E ) ,
where M S E ( R , E ) denotes the mean squared error between R and E. Because max p R can differ from a channel (or a descriptor) to another, Equation (11) takes into account this actual maximal level rather than the theoretical one to avoid misleading PSNR values. In PSNR computation, as for the previous correlation study, we exclude the 16 pixels in each of the four borders of the image to avoid inherent border effect related to either registration or demosaicking processing.

4.2. Results and Discussion

Table 4 displays the PSNR values provided by the demosaicking methods on average over the ten database scenes. Results show that among bilinear filters, B 3 provides the best results for I 0 , I 45 , I 90 , I 135 , S 0 , S 1 , and D O L P , while B 4 slightly exceeds it for S 2 and A O L P . Among PFA-oriented methods, CB and CBSP generally provide the best results. Our proposition to adapt RI and ARI CFA demosaicking methods to the PFA case (with S 0 as guide) provides better results than classical PFA-oriented methods. We remark that RI and ARI are very close together in the PSNR results. RI also provides the best results among all tested methods regarding S 2 parameter and D O L P descriptor.
For PFA methods, it is important to note that the output interpolated pixel is shifted by half pixel when using bilinear kernels B 1 , B 4 , and B 5 , compared to other bilinear kernels B 2 , B 3 . The output pixel is either aligned to the original interpolated pixel position center, or in the pixel boundaries. We did not correct for this misalignment because applying an image translation by half pixel needs an additional cubic or linear interpolation. So such a registration process cannot be used as a pre-processing for an acceptable comparison methodology over the bilinear demosaicking methods. Thus, the results for B 1 , B 4 , and B 5 should be interpreted with care.
For tested SFA-oriented methods, the use of spectral correlation generally provides better performance than simple bilinear interpolations. Moreover, methods based on gradient computation (BTES, MLDI, and PPID) exhibit the best demosaicking performances. By considering the PPI as a guide for demosaicking, PPID shows the best demosaicking performances among all methods for all polarization channels, also for S 0 , S 1 parameters and A O L P descriptors.
To visually compare the results provided by demosaicking methods on S 0 , A O L P , and D O L P descriptors, we select a zoomed area from the “macbeth_enhancement” scene of the database. Among demosaicking methods, we show the results of the most intuitive method (bilinear interpolation using B 3 filter), and the pseudo panchromatic image difference (PPID) that globally provides the best PSNR results. Figure 7 shows that there is no significant difference regarding the S 0 parameter, except that the two highlight dots are more apparent in PPID demosaicked image. Computing A O L P and D O L P parameter from a bilinearly interpolated image generates many artifacts that are fairly reduced using PPID demosaicking method.
Generally speaking, we found that demosaicking that are dedicated to PFA don’t necessary give better PSNR result. Thus, it was not obvious that considering color and spectral demosaicking techniques applied to PFA arrangement could be beneficial. The results highlights that this can benefit the pre-processing of PFA.
However, we can express some reservations about the results obtained. First, we limited our study on a relatively small database. Other polarization database in the literature [59] furbish only the Stokes parameters and polarization descriptors, but no fully-defined reference image I = { I k } k { 0 , 45 , 90 , 135 } are available. Natural scene samples could also be beneficial for a complementary algorithm classification. Secondly, the database used in this work was made with the same experimental conditions, i.e., constant angle/plan of incidence and a unique non-polarized illuminant. We think that supplementary tests of the best identified algorithms in an extended database containing a better statistical distribution of data could be valuable. Thirdly, the noise associated with reference images is not quantified, and is slightly different from a PFA acquisition system. We thus disregarded recent denoising-demosaicking algorithms that estimate sensor noise to improve the accuracy of demosaicking [60,61,62,63].
The arrangement of the filter array investigated consists in a 2 × 2 square periodic basic pattern with no dominant band. Our goal was to stay general and to apply the evaluation on a well-used pattern. But some other tentatives of designing extended pattern have been proposed in the literature [64], for a better spatial distribution of linear polarization filters. An extensive evaluation of best demosaicking algorithms on different arrangements would be considered in a future work.
We found that the acquisition setup may induces correlation between some polarized channels that could be exploited for demosaicking. Since these properties are data-dependent, we have chosen to not use them in our study, despite that they are used in few SFA demosaicking methods.
We remarked that some algorithms need more computation time than others, without necessary giving better results. No computational complexity consideration has been reported in this work. We think that there is a lake of information about these aspects in the original articles. Moreover, Matlab or ImageJ can not provide a consistent evaluation of the complexity of the selected algorithms, e.g., for their potential ability to be parallelized in a hardware acceleration for real-time computing.

5. Conclusions

By considering the inter-channel correlation, CFA and SFA schemes aim to improve the spatial reconstruction of channels from the information of other channels. Experiments on the only available polarization image database have shown that such methods provides better results in term of PSNR than PFA-oriented methods. More particularly, we proposed to adapt two CFA demosaicking algorithms based on residual interpolation to the PFA case, and showed that they provide better results than classical PFA-oriented methods. Moreover, the SFA PPID method provides the overall best results, and largely reduces visual artifacts in the reconstructed polarization descriptors in comparison with bilinear method.
Correlation study has shown that the spectral band considered in the acquisition of polarization channels has no influence on the correlation between polarization channels. The correlation results from this study could be an input and provide assumptions for the design of new demosaicking algorithms applied on cameras that mix spectral and polarization filters.
All algorithms were tested on a small database of images. As future work, we hope that an extensive database of polarization and spectral images will be available soon in the research community. Thus, further evaluations on more various materials and image statistics would validate more deeply our conclusions. Furthermore, we believe that a real-time pipelined implementation of the PPID method using GPU or FPGA needs to be investigated, that would be a valuable tool for machine vision applications.

Author Contributions

Conceptualization, methodology, software, analysis, writing—original draft preparation, writing—review and editing, S.M. and P.-J.L.; project administration, supervision, funding acquisition, L.B.


This research received no external funding.


We want to thanks Jonathan Ledy and Michel Basset for their technical support to acquire images of Figure 2.

Conflicts of Interest

The authors declare no conflict of interest.


  1. Tyo, J.S.; Goldstein, D.L.; Chenault, D.B.; Shaw, J.A. Review of passive imaging polarimetry for remote sensing applications. Appl. Opt. 2006, 45, 5453–5469. [Google Scholar] [CrossRef] [PubMed]
  2. Lapray, P.J.; Gendre, L.; Foulonneau, A.; Bigué, L. An FPGA-based pipeline for micropolarizer array imaging. Int. J. Circuit Theory Appl. 2018, 46, 1675–1689. [Google Scholar] [CrossRef]
  3. Bayer, B.E. Color Imaging Array. U.S. Patent 3,971,065, 20 July 1976. [Google Scholar]
  4. Lapray, P.J.; Thomas, J.B.; Gouton, P. A Multispectral Acquisition System Using MSFAs. In Proceedings of the 22nd Color and Imaging Conference, Vancouver, BC, Canada, 12–16 November 2018; Society for Imaging Science and Technology: Springfield, VA, USA, 2014; Volume 2014, pp. 97–102. [Google Scholar]
  5. Lapray, P.J.; Wang, X.; Thomas, J.B.; Gouton, P. Multispectral Filter Arrays: Recent Advances and Practical Implementation. Sensors 2014, 14, 21626–21659. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  6. Thomas, J.B.; Lapray, P.J.; Gouton, P.; Clerc, C. Spectral Characterization of a Prototype SFA Camera for Joint Visible and NIR Acquisition. Sensors 2016, 16, 993. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  7. Brill, M.H. Acquisition and reproduction of color images: Colorimetric and multispectral approaches. Color Res. Appl. 2016, 27, 304–305. [Google Scholar] [CrossRef]
  8. Khan, H.A.; Thomas, J.B.; Hardeberg, J.Y.; Laligant, O. Spectral Adaptation Transform for Multispectral Constancy. J. Imaging Sci. Technol. 2018, 62, 20504-1–20504-12. [Google Scholar] [CrossRef]
  9. Zhao, Y.; Berns, R.S. Image-based spectral reflectance reconstruction using the matrix R method. Color Res. Appl. 2007, 32, 343–351. [Google Scholar] [CrossRef]
  10. Monno, Y.; Tanaka, M.; Okutomi, M. Multispectral Demosaicking Using guided Filter. In Proceedings of the SPIE Digital Photography VIII, Burlingame, CA, USA, 24 January 2012; Volume 8299. [Google Scholar]
  11. Mihoubi, S.; Losson, O.; Mathon, B.; Macaire, L. Multispectral demosaicing using pseudo-panchromatic image. IEEE Trans. Comput. Imaging 2017, 3, 982–995. [Google Scholar] [CrossRef]
  12. Rust, D.M. Integrated Dual Imaging Detector. U.S. Patent 5,438,414, 1 August 1995. [Google Scholar]
  13. Tokuda, T.; Sato, S.; Yamada, H.; Sasagawa, K.; Ohta, J. Polarisation-analysing CMOS photosensor with monolithically embedded wire grid polariser. Electron. Lett. 2009, 45, 1. [Google Scholar] [CrossRef]
  14. Sasagawa, K.; Shishido, S.; Ando, K.; Matsuoka, H.; Noda, T.; Tokuda, T.; Kakiuchi, K.; Ohta, J. Image sensor pixel with on-chip high extinction ratio polarizer based on 65-nm standard CMOS technology. Opt. Express 2013, 21, 11132–11140. [Google Scholar] [CrossRef] [PubMed]
  15. Zhang, M.; Wu, X.; Cui, N.; Engheta, N.; der Spiegel, J.V. Bioinspired Focal-Plane Polarization Image Sensor Design: From Application to Implementation. Proc. IEEE 2014, 102, 1435–1449. [Google Scholar] [CrossRef]
  16. Zhao, X.; Boussaid, F.; Bermak, A.; Chigrinov, V.G. High-resolution thin “guest-host” micropolarizer arrays for visible imaging polarimetry. Opt. Express 2011, 19, 5565–5573. [Google Scholar] [CrossRef] [PubMed]
  17. Myhre, G.; Hsu, W.L.; Peinado, A.; LaCasse, C.; Brock, N.; Chipman, R.A.; Pau, S. Liquid crystal polymer full-stokes division of focal plane polarimeter. Opt. Express 2012, 20, 27393–27409. [Google Scholar] [CrossRef] [PubMed]
  18. Ohfuchi, T.; Sakakura, M.; Yamada, Y.; Fukuda, N.; Takiya, T.; Shimotsuma, Y.; Miura, K. Polarization imaging camera with a waveplate array fabricated with a femtosecond laser inside silica glass. Opt. Express 2017, 25, 23738–23754. [Google Scholar] [CrossRef] [PubMed]
  19. Park, H.; Crozier, K.B. Elliptical silicon nanowire photodetectors for polarization-resolved imaging. Opt. Express 2015, 23, 7209–7216. [Google Scholar] [CrossRef] [PubMed]
  20. Maruyama, Y.; Terada, T.; Yamazaki, T.; Uesaka, Y.; Nakamura, M.; Matoba, Y.; Komori, K.; Ohba, Y.; Arakawa, S.; Hirasawa, Y.; et al. 3.2-MP Back-Illuminated Polarization Image Sensor With Four-Directional Air-Gap Wire Grid and 2.5-μm Pixels. IEEE Trans. Electron Devices 2018, 65, 2544–2551. [Google Scholar] [CrossRef]
  21. Garcia, M.; Edmiston, C.; Marinov, R.; Vail, A.; Gruev, V. Bio-inspired color-polarization imager for real-time in situ imaging. Optica 2017, 4, 1263–1271. [Google Scholar] [CrossRef]
  22. Garcia, M.; Davis, T.; Blair, S.; Cui, N.; Gruev, V. Bioinspired polarization imager with high dynamic range. Optica 2018, 5, 1240–1246. [Google Scholar] [CrossRef]
  23. Sarkar, M.; Bello, D.S.S.; van Hoof, C.; Theuwissen, A.J.P. Biologically Inspired CMOS Image Sensor for Fast Motion and Polarization Detection. IEEE Sens. J. 2013, 13, 1065–1073. [Google Scholar] [CrossRef]
  24. Brock, N.J.; Crandall, C.; Millerd, J.E. Snap-shot Imaging Polarimeter: Performance and Applications. In Proceedings of the Polarization: Measurement, Analysis, and Remote Sensing XI, Baltimore, MD, USA, 5–9 May 2014; Volume 9099. [Google Scholar]
  25. Lapray, P.J.; Heyrman, B.; Rossé, M.; Ginhac, D. A 1.3 Megapixel FPGA-Based Smart Camera for High Dynamic Range Real Time Video. In Proceedings of the 2013 Seventh International Conference on Distributed Smart Cameras (ICDSC), Palm Springs, CA, USA, 29 October–1 November 2013; pp. 1–6. [Google Scholar]
  26. Lapray, P.J.; Heyrman, B.; Ginhac, D. HDR-ARtiSt: An adaptive real-time smart camera for high dynamic range imaging. J. Real-Time Image Proc. 2016, 12, 747–762. [Google Scholar] [CrossRef]
  27. Singh, S.; Shekhar, C.; Vohra, A. FPGA-Based Real-Time Motion Detection for Automated Video Surveillance Systems. Electronics 2016, 5, 10. [Google Scholar] [CrossRef]
  28. Bachman, K.A.; Peltzer, J.J.; Flammer, P.D.; Furtak, T.E.; Collins, R.T.; Hollingsworth, R.E. Spiral plasmonic nanoantennas as circular polarization transmission filters. Opt. Express 2012, 20, 1308–1319. [Google Scholar] [CrossRef] [PubMed]
  29. Tyo, J.S. Design of optimal polarimeters: Maximization of signal-to-noise ratio and minimization of systematic error. Appl. Opt. 2002, 41, 619–630. [Google Scholar] [CrossRef] [PubMed]
  30. Jaiswal, S.; Fang, L.; Jakhetiya, V.; Pang, J.; Mueller, K.; Au, O.C. Adaptive Multispectral Demosaicking Based on Frequency Domain Analysis of Spectral Correlation. IEEE Trans. Image Proc. 2017, 26, 953–968. [Google Scholar] [CrossRef] [PubMed]
  31. Lapray, P.J.; Gendre, L.; Bigué, L.; Foulonneau, A. A Database of Polarimetric and Multispectral Images in the Visible and NIR Regions. In Proceedings of the SPIE Unconventional Optical Imaging, Strasbourg, France, 22–26 April 2018; Volume 10677. [Google Scholar]
  32. Evangelidis, G. IAT: A Matlab Toolbox for Image Alignment. 2013. Available online: (accessed on 29 October 2018).
  33. Li, X.; Gunturk, B.; Zhang, L. Image Demosaicing: A Systematic Survey. In Proceedings of the SPIE Visual Communications and Image Processing 2008, San Jose, CA, USA, 27–31 January 2008; Volume 6822. [Google Scholar]
  34. Ratliff, B.M.; LaCasse, C.F.; Tyo, J.S. Interpolation strategies for reducing IFOV artifacts in microgrid polarimeter imagery. Opt. Express 2009, 17, 9112–9125. [Google Scholar] [CrossRef] [PubMed]
  35. Tyo, J.S.; LaCasse, C.F.; Ratliff, B.M. Total elimination of sampling errors in polarization imagery obtained with integrated microgrid polarimeters. Opt. Lett. 2009, 34, 3187–3189. [Google Scholar] [CrossRef] [PubMed]
  36. Ratliff, B.M.; LaCasse, C.F.; Tyo, J.S. Adaptive Strategy for Demosaicing Microgrid Polarimeter Imagery. In Proceedings of the 2011 Aerospace Conference, Big Sky, MT, USA, 5–12 March 2011; pp. 1–9. [Google Scholar]
  37. Gao, S.; Gruev, V. Bilinear and bicubic interpolation methods for division of focal plane polarimeters. Opt. Express 2011, 19, 26161–26173. [Google Scholar] [CrossRef] [PubMed]
  38. Zhang, J.; Luo, H.; Hui, B.; Chang, Z. Image interpolation for division of focal plane polarimeters with intensity correlation. Opt. Express 2016, 24, 20799–20807. [Google Scholar] [CrossRef] [PubMed]
  39. Kiku, D.; Monno, Y.; Tanaka, M.; Okutomi, M. Residual Interpolation for Color Image Demosaicking. In Proceedings of the 2013 IEEE International Conference on Image Processing, Melbourne, Australia, 15–18 September 2013; pp. 2304–2308. [Google Scholar]
  40. Monno, Y.; Kiku, D.; Tanaka, M.; Okutomi, M. Adaptive Residual Interpolation for Color Image Demosaicking. In Proceedings of the 2015 IEEE International Conference on Image Processing (ICIP), Quebec City, QC, Canada, 27–30 September 2015; pp. 3861–3865. [Google Scholar]
  41. Miao, L.; Qi, H.; Ramanath, R.; Snyder, W.E. Binary Tree-based Generic Demosaicking Algorithm for Multispectral Filter Arrays. IEEE Trans. Image Proc. 2006, 15, 3550–3558. [Google Scholar] [CrossRef]
  42. Brauers, J.; Aach, T. A Color Filter Array Based Multispectral Camera; RWTH Aachen University: Aachen, Germany, 2006. [Google Scholar]
  43. Wang, X.; Thomas, J.B.; Hardeberg, J.Y.; Gouton, P. Median Filtering in Multispectral Filter Array Demosaicking. In Proceedings of the SPIE Digital Photography IX, Burlingame, CA, USA, 3–7 February 2013; Volume 8660. [Google Scholar]
  44. Wang, X.; Thomas, J.B.; Hardeberg, J.Y.; Gouton, P. Discrete Wavelet Transform Based Multispectral Filter Array Demosaicking. In Proceedings of the 2013 Colour and Visual Computing Symposium (CVCS 2013), Gjøvik, Norway, 5–6 September 2013; pp. 1–6. [Google Scholar]
  45. Shinoda, K.; Ogawa, S.; Yanagi, Y.; Hasegawa, M.; Kato, S.; Ishikaway, M.; Komagatay, H.; Kobayashi, N. Multispectral Filter Array and Demosaicking for Pathological Images. In Proceedings of the APSIPA Annual Summit and Conference 2015, Hong Kong, China, 16–19 December 2015. [Google Scholar]
  46. Zhang, J.; Luo, H.; Liang, R.; Ahmed, A.; Zhang, X.; Hui, B.; Chang, Z. Sparse representation-based demosaicing method for microgrid polarimeter imagery. Opt. Lett. 2018, 43, 3265–3268. [Google Scholar] [CrossRef] [PubMed]
  47. Ratliff, B.M.; Tyo, J.S.; Black, W.T.; LaCasse, C.F. Exploiting Motion-Based Redundancy to Enhance Microgrid Polarimeter Imagery. In Proceedings of the SPIE Polarization Science and Remote Sensing IV, San Diego, CA, USA, 11 August 2009; Volume 7461. [Google Scholar]
  48. Gao, S.; Gruev, V. Gradient-based interpolation method for division-of-focal-plane polarimeters. Opt. Express 2013, 21, 1137–1151. [Google Scholar] [CrossRef] [PubMed]
  49. Ramanath, R.; Snyder, W.E. Adaptive demosaicking. J. Electron. Imaging 2003, 12, 633–643. [Google Scholar] [CrossRef]
  50. Perkins, R.; Gruev, V. Signal-to-noise analysis of Stokes parameters in division of focal plane polarimeters. Opt. Express 2010, 18, 25815–25824. [Google Scholar] [CrossRef] [PubMed]
  51. He, K.; Sun, J.; Tang, X. Guided Image Filtering. In Proceedings of the 11th European Conference on Computer Vision (ECCV 2010), Crete, Greece, 5–11 September 2010; Volume 6311, pp. 1–14. [Google Scholar]
  52. Amba, P.; Thomas, J.B.; Alleysson, D. N-LMMSE Demosaicing for Spectral Filter Arrays. J. Imaging Sci. Technol. 2017, 61, 40407-1–40407-11. [Google Scholar] [CrossRef]
  53. Aggarwal, H.K.; Majumdar, A. Single-Sensor Multi-Spectral Image Demosaicing Algorithm Using Learned Interpolation Weights. In Proceedings of the 2014 International Geoscience and Remote Sensing Symposium (IGARSS 2014), Quebec City, QC, Canada, 13–18 July 2014; pp. 2011–2014. [Google Scholar]
  54. Zeiler, M.D.; Krishnan, D.; Taylor, G.W.; Fergus, R. Deconvolutional Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’10), San Francisco, CA, USA, 13–18 June 2010; pp. 2528–2535. [Google Scholar]
  55. Shinoda, K.; Hamasaki, T.; Kawase, M.; Hasegawa, M.; Kato, S. Demosaicking for multispectral images based on vectorial total variation. Opt. Rev. 2016, 23, 559–570. [Google Scholar] [CrossRef]
  56. Aggarwal, H.K.; Majumdar, A. Compressive Sensing Multi-Spectral Demosaicing from Single Sensor Architecture. In Proceedings of the IEEE China Summit International Conference on Signal and Information Processing (ChinaSIP’2014), Xi’an, China, 9–13 July 2014; pp. 334–338. [Google Scholar]
  57. Mizutani, J.; Ogawa, S.; Shinoda, K.; Hasegawa, M.; Kato, S. Multispectral Demosaicking Algorithm Based on Inter-Channel Correlation. In Proceedings of the IEEE Visual Communications and Image Processing Conference (VCIP 2014), Valletta, Malta, 7–10 December 2014; pp. 474–477. [Google Scholar]
  58. Mihoubi, S.; Losson, O.; Mathon, B.; Macaire, L. Multispectral Demosaicing Using Intensity-Based Spectral Correlation. In Proceedings of the 5th International Conference on Image Processing Theory, Tools and Applications (IPTA’15), Orleans, France, 10–13 November 2015; pp. 461–466. [Google Scholar]
  59. Hu, S.; Short, N.J.; Riggan, B.S.; Gordon, C.; Gurton, K.P.; Thielke, M.; Gurram, P.; Chan, A.L. A Polarimetric Thermal Database for Face Recognition Research. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Las Vegas, NV, USA, 26 June–1 July 2016. [Google Scholar]
  60. Gilboa, E.; Cunningham, J.P.; Nehorai, A.; Gruev, V. Image interpolation and denoising for division of focal plane sensors using Gaussian processes. Opt. Express 2014, 22, 15277–15291. [Google Scholar] [CrossRef] [PubMed]
  61. Zhang, J.; Luo, H.; Liang, R.; Zhou, W.; Hui, B.; Chang, Z. PCA-based denoising method for division of focal plane polarimeters. Opt. Express 2017, 25, 2391–2400. [Google Scholar] [CrossRef] [PubMed]
  62. Li, S.; Ye, W.; Liang, H.; Pan, X.; Lou, X.; Zhao, X. K-SVD Based Denoising Algorithm for DoFP Polarization Image Sensors. In Proceedings of the 2018 IEEE International Symposium on Circuits and Systems (ISCAS), Florence, Italy, 27–30 May 2018; pp. 1–5. [Google Scholar]
  63. Abubakar, A.; Zhao, X.; Li, S.; Takruri, M.; Bastaki, E.; Bermak, A. A Block-Matching and 3-D Filtering Algorithm for Gaussian Noise in DoFP Polarization Images. IEEE Sens. J. 2018, 18, 7429–7435. [Google Scholar] [CrossRef]
  64. LeMaster, D.A.; Hirakawa, K. Improved microgrid arrangement for integrated imaging polarimeters. Opt. Lett. 2014, 39, 1811–1814. [Google Scholar] [CrossRef] [PubMed]
Figure 1. Polarization Filter Array principle. A polarization filter array covers the pixel matrix of a radiometric sensor. The array of polarimetric filters are located, either directly above the matrix of pixels, or over the micro-lens array.
Figure 1. Polarization Filter Array principle. A polarization filter array covers the pixel matrix of a radiometric sensor. The array of polarimetric filters are located, either directly above the matrix of pixels, or over the micro-lens array.
Sensors 18 03688 g001
Figure 2. Polarization Filter Array (PFA) imaging overview. (a) Raw output image from the 4D Technology camera. (be) Downscaled polarization images (without spatial interpolation). (fh) Polarization descriptor images associated to downsampled images.
Figure 2. Polarization Filter Array (PFA) imaging overview. (a) Raw output image from the 4D Technology camera. (be) Downscaled polarization images (without spatial interpolation). (fh) Polarization descriptor images associated to downsampled images.
Sensors 18 03688 g002
Figure 3. Overview of the analysis which is done in this paper.
Figure 3. Overview of the analysis which is done in this paper.
Sensors 18 03688 g003
Figure 4. General mosaicking/demosaicking testing framework used in this work.
Figure 4. General mosaicking/demosaicking testing framework used in this work.
Sensors 18 03688 g004
Figure 5. The five demosaicking kernels of the five bilinear methods B 1 5 from the work by Ratliff et al. [34]. It refers to the neighborhood used for interpolation. (ac) Are simple bilinear kernels, whereas (d,e) are weighted bilinear kernels.
Figure 5. The five demosaicking kernels of the five bilinear methods B 1 5 from the work by Ratliff et al. [34]. It refers to the neighborhood used for interpolation. (ac) Are simple bilinear kernels, whereas (d,e) are weighted bilinear kernels.
Sensors 18 03688 g005
Figure 6. Neighborhood used for weight computation in any channel according to the step of binary-three edge sensing (BTES) algorithm. Pixels in black are known or previously estimated, whereas pixels in gray are the estimated pixels. Pixels in white are unknown and not estimated at the current step.
Figure 6. Neighborhood used for weight computation in any channel according to the step of binary-three edge sensing (BTES) algorithm. Pixels in black are known or previously estimated, whereas pixels in gray are the estimated pixels. Pixels in white are unknown and not estimated at the current step.
Sensors 18 03688 g006
Figure 7. Zoom in of the “macbeth_enhancement” scene from the database of Lapray et al. [31]. (ac) Images resulting S 0 , angle of linear polarization (AOLP), and degree of linear polarization (DOLP) processed using the full-resolution reference. (df) The bilinearely interpolation [34] ( B 3 ). (gi) The pseudo-panchromatic image difference (PPID) interpolation [11].
Figure 7. Zoom in of the “macbeth_enhancement” scene from the database of Lapray et al. [31]. (ac) Images resulting S 0 , angle of linear polarization (AOLP), and degree of linear polarization (DOLP) processed using the full-resolution reference. (df) The bilinearely interpolation [34] ( B 3 ). (gi) The pseudo-panchromatic image difference (PPID) interpolation [11].
Sensors 18 03688 g007
Table 1. Inter-channel correlation between the six spectral channels relatively to the polarization angle of analysis. Last line of each subtable is the spatial correlation within each channel. Values are averaged over the ten multispectral images from [31]. Last subtable (e) is the average over the four polarization channels.
(a) 0°
(a) 0°
C 1 C 2 C 3 C 4 C 5 C 6
C 1 1.00000.88480.91000.76180.85710.8411
C 2 0.88481.00000.95610.89560.85840.8495
C 3 0.91000.95611.00000.90020.94540.8846
C 4 0.76180.89560.90021.00000.83520.7962
C 5 0.85710.85840.94540.83521.00000.8927
C 6 0.84110.84950.88460.79620.89271.0000
(b) 45°
(b) 45°
C 1 C 2 C 3 C 4 C 5 C 6
C 1 1.00000.93330.91270.79490.83920.8181
C 2 0.93331.00000.96200.89120.86790.8245
C 3 0.91270.96201.00000.91500.94170.8528
C 4 0.79490.89120.91501.00000.87340.7812
C 5 0.83920.86790.94170.87341.00000.8664
C 6 0.81810.82450.85280.78120.86641.0000
(c) 90°
(c) 90°
C 1 C 2 C 3 C 4 C 5 C 6
C 1 1.00000.94900.91330.82250.83270.8102
C 2 0.94901.00000.96300.89600.86870.8229
C 3 0.91330.96301.00000.93350.94130.8481
C 4 0.82250.89600.93351.00000.90440.8024
C 5 0.83270.86870.94130.90441.00000.8622
C 6 0.81020.82290.84810.80240.86221.0000
(d) 135°
(d) 135°
C 1 C 2 C 3 C 4 C 5 C 6
C 1 1.00000.89610.91270.77880.85450.8422
C 2 0.89611.00000.95730.89740.86290.8473
C 3 0.91270.95731.00000.90770.94640.8827
C 4 0.77880.89740.90771.00000.85010.8014
C 5 0.85450.86290.94640.85011.00000.8936
C 6 0.84220.84730.88270.80140.89361.0000
(e) Average
(e) Average
C 1 C 2 C 3 C 4 C 5 C 6
C 1 1.00000.91580.91220.78950.84590.8279
C 2 0.91581.00000.95960.89500.86450.8360
C 3 0.91220.95961.00000.91410.94370.8670
C 4 0.78950.89500.91411.00000.86580.7953
C 5 0.84590.86450.94370.86581.00000.8787
C 6 0.82790.83600.86700.79530.87871.0000
Table 2. Inter-channel correlation between the four polarization channels according to the spectral band. Last line of each subtable is the spatial correlation within each channel. Values are averaged over the ten multispectral images from [31]. Last subtable is the average over the six spectral channels.
(a) C1
(a) C1
I 0 I 45 I 90 I 135
I 0 1.00000.92270.89800.9763
I 45 0.92271.00000.96990.9250
I 90 0.89800.96991.00000.9126
I 135 0.97630.92500.91261.0000
(b) C2
(b) C2
I 0 I 45 I 90 I 135
I 0 1.00000.92620.87870.9372
I 45 0.92621.00000.94700.8969
I 90 0.87870.94701.00000.8974
I 135 0.93720.89690.89741.0000
(c) C3
(c) C3
I 0 I 45 I 90 I 135
I 0 1.00000.90770.89600.9486
I 45 0.90771.00000.94860.8970
I 90 0.89600.94861.00000.9024
I 135 0.94860.89700.90241.0000
(d) C4
(d) C4
I 0 I 45 I 90 I 135
I 0 1.00000.87870.84440.9317
I 45 0.87871.00000.92860.8816
I 90 0.84440.92861.00000.8688
I 135 0.93170.88160.86881.0000
(e) C5
(e) C5
I 0 I 45 I 90 I 135
I 0 1.00000.90740.89200.9444
I 45 0.90741.00000.95240.8986
I 90 0.89200.95241.00000.8955
I 135 0.94440.89860.89551.0000
(f) C6
(f) C6
I 0 I 45 I 90 I 135
I 0 1.00000.90490.81550.9107
I 45 0.90491.00000.89650.8823
I 90 0.81550.89651.00000.8674
I 135 0.91070.88230.86741.0000
(g) Average
(g) Average
I 0 I 45 I 90 I 135
I 0 1.00000.90790.87080.9415
I 45 0.90791.00000.94050.8969
I 90 0.87080.94051.00000.8907
I 135 0.94150.89690.89071.0000
Table 3. Summary of the color, spectral, and polarization filter array (CFA/SFA/PFA) interpolation methods. R, A and P abbreviations mean that the algorithms were Re-coded, Adapted, or Provided by the authors of the original work.
Table 3. Summary of the color, spectral, and polarization filter array (CFA/SFA/PFA) interpolation methods. R, A and P abbreviations mean that the algorithms were Re-coded, Adapted, or Provided by the authors of the original work.
Bilinear with 5 different kernels [34] B 1 5 2009R
Linear system [35]LS2009R
Adaptive [36]A2011R
Cubic [37]CB2011R
Cubic-Spline [37]CBSP2011R
Intensity Correlation among Polarization Channels [38]ICPC2016P
Residual Interpolation [39]RI2013A
Adaptive Residual Interpolation [40]ARI2015A
Binary-Three Edge Sensing [41]BTES2006R
Spectral Difference [42]SD2006R
Vector median [43]VM2013P
Discrete Wavelet Transform [44]DWT2013P
Multi Local Directional Interpolation [45]MLDI2015R
Pseudo-Panchromatic Image Difference [11]PPID2017A
Pseudo-Panchromatic Image based Discrete Wavelet Transform [11]PPIDWT2017A
Table 4. Demosaicking peak signal-to-noise ratio (PSNR) results, which are averaged over all scenes of the testing database. The descriptors are computed using Equations (1)–(3). The best result for each channel or descriptor is highlighted as bold.
Table 4. Demosaicking peak signal-to-noise ratio (PSNR) results, which are averaged over all scenes of the testing database. The descriptors are computed using Equations (1)–(3). The best result for each channel or descriptor is highlighted as bold.
I 0 I 45 I 90 I 135 S 0 S 1 S 2 DOLPAOLP
B 1 34.8836.7436.8835.8738.5936.8634.9924.6317.15
B 2 37.1839.4439.5438.5640.7739.5337.9627.1118.35
B 3 40.8144.8144.9943.9744.8243.8743.5831.7220.66
B 4 36.8739.0839.2538.1938.5143.6543.8031.4120.88
B 5 36.3638.4038.5637.5138.2741.6240.6729.5119.45
CFA-oriented (Adapted)
R I 42.1647.0647.5647.1146.1745.8146.9433.7721.69
A R I 41.7746.9347.3946.9345.7845.5246.7433.5121.60

Share and Cite

MDPI and ACS Style

Mihoubi, S.; Lapray, P.-J.; Bigué, L. Survey of Demosaicking Methods for Polarization Filter Array Images. Sensors 2018, 18, 3688.

AMA Style

Mihoubi S, Lapray P-J, Bigué L. Survey of Demosaicking Methods for Polarization Filter Array Images. Sensors. 2018; 18(11):3688.

Chicago/Turabian Style

Mihoubi, Sofiane, Pierre-Jean Lapray, and Laurent Bigué. 2018. "Survey of Demosaicking Methods for Polarization Filter Array Images" Sensors 18, no. 11: 3688.

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop