Optimized Multi-Spectral Filter Array Based Imaging of Natural Scenes

Abstract

Multi-spectral imaging using a camera with more than three channels is an efficient method to acquire and reconstruct spectral data and is used extensively in tasks like object recognition, relighted rendering, and color constancy. Recently developed methods are used to only guide content-dependent filter selection where the set of spectral reflectances to be recovered are known a priori. We present the first content-independent spectral imaging pipeline that allows optimal selection of multiple channels. We also present algorithms for optimal placement of the channels in the color filter array yielding an efficient demosaicing order resulting in accurate spectral recovery of natural reflectance functions. These reflectance functions have the property that their power spectrum statistically exhibits a power-law behavior. Using this property, we propose power-law based error descriptors that are minimized to optimize the imaging pipeline. We extensively verify our models and optimizations using large sets of commercially available wide-band filters to demonstrate the greater accuracy and efficiency of our multi-spectral imaging pipeline over existing methods.

1. Introduction

Multi-channel spectral imaging using a single-camera is an efficient way to acquire spectral data. Compared with multi-camera systems that use multiple cameras with modified camera sensitivity, it is more compact and practical. Typically, the multiple different channels of the spectrum are captured (a) over multiple shots using a different color filter in front of the camera sensor for each shot; or (b) in a single shot using a mosaic of the multiple filters on the sensor forming a multiple spectral filter array (MSFA).

Traditionally, narrow band filters are used to provide a large number of channels, thereby providing higher spectral resolution assuring an accurate spectral recovery. However, this severely compromises the acquisition rate in multi-shot capture and the spatial resolution in single shot capture. Since spatial resolution is very important for most scenarios, commercially available multi-spectral cameras are multi-shot. Narrow band filters also significantly compromise light efficiency and therefore almost all multi-spectral cameras use longer exposures, thereby limiting its capture to only static scenes. Recently, use of wide-band filters has been explored to alleviate these problems by reducing the number of channels used [1]. However, the channels are usually chosen in an ad hoc manner, thereby not assuring an accurate spectral recovery. Recently, a work showed that, with channel selection, the spectral reconstruction accuracy could be improved by over $33\%$ [2].

Previous works have shown that both man-made and natural objects or phenomena (like illumination) have smooth spectral power distribution. Statistically, their power spectrum follows a strong power law [3,4]. We exploit this statistical property to carefully select a small number of optimal wide band channels (usually 5–6) for multi-spectral imaging that can ensure an accurate spectral reconstruction when used in an MSFA for a single shot capture. The main contributions of this work are as follows.

• First, we design and develop a novel power-law prior based channel optimization method that models the various errors associated with spectral reconstruction—namely error due to estimation (reconstruction error), noise (imaging error) and demosaicing (demosaicing error). These errors depend only on the camera parameters (e.g., spectral sensitivities of channels, the MSFA pattern, the demosaicing order, and variance of the sensor noise) and not on the content. To the best of our knowledge, this is the first model for defining all the different errors in a content-independent multi-spectral imaging pipeline.
• Second, we construct an objective function that quantifies the total error using a combination of the three above-mentioned errors. Next, we use a discrete particle swarm optimization method to optimize the imaging pipeline by (1) selecting a few channels from a large set of candidate channels; (2) constructing a conducive mosaic pattern with the chosen channels on the MSFA; and (3) selecting a channel ordering during demosaicing that minimizes the objective function and hence the total error in spectral reconstruction.

2. Related Works

Multispectral Imaging and Reconstruction: a large number of the existing imaging techniques today are either bulky, or expensive and require professional calibration [5,6]. Our focus in this paper are the alternate, more compact spectral imaging techniques that are popularly used for commodity applications. There are two existing spectral imaging techniques that fit this bill—(a) compressive spectral imaging and (b) multi-spectral filter array (MSFA) based spectral imaging.

Compressive computational imaging systems randomly encode captured spectra [7], and then reconstruct multi-spectral images using compressive sensing reconstruction techniques [8,9,10]. Taking full advantages of the sparseness of spectral image data, these reconstruction techniques can recover multi-spectral images using fewer observations than conventionally needed. Furthermore, due to the high light throughput (≈0.5 [9]) of these systems, the quality of reconstructed images is robust to imaging noise. However, the sparse recovery process is time-consuming. For example, it can take minutes or even hours to reconstruct a single $512\times 512\times 31$ multi-spectral frame. Therefore, it is not suitable for real-time applications, such as rendering relighted scenes, or object detection.

Contrary to compressive computational imaging, MSFA offers a very time efficient option for spectral reconstruction. Recently, it has been shown that, when using wide-band filters, MSFA based methods can be more accurate in the common case where the “photon to read noise ratios” (PRR) is not very low [11]. Therefore, various multi-channel spectral imaging system with MSFA have been designed recently. For example, Yasuma et al. [12] proposed a sub-micron pixel image sensor design with seven colors and two exposures; Sajadi et al. [13] combined a red-green-blue Bayer color filter array (CFA) with a cyan-magenta-yellow CFA in a layered CFA design to capture images with high color fidelity; Monno et al. [14] presented a high-performance multi-spectral demosaicing algorithm and utilized a single sensor with 5-channel MSFA to capture multi-spectral images. However, none of these single-shot methods focus on optimizing image pipeline parameters (like the channels selected, spatial arrangement of channels on the MSFA, demosaicing order) in a unified manner and hence cannot provide the desired spatial and spectral precision with accuracy of reconstruction.

Channel Selection: references [15,16] optimize spectral sensitivity of channels with a regular three-channel RGB Bayer color filter array (CFA) , while references [17,18] optimize channels to minimize the error in spectral recovery, but only for spectral data that have strong priors like known distribution of the captured spectral functions (e.g., daylight [19]). Such domain-specific priors yield content-dependent methods. More importantly, theoretically conducive spectral sensitivity of the channels are assumed (e.g., radial basis function [20,21], Fourier basis function [22]). This yields poor results in practice due to a large deviation of the sensitivities of commercially available filters from such well-behaved functions [23] (Figure 1).

In contrast, we select filters from a set of commercially available wide-band filters to create a practically feasible spectral imaging pipeline. A handful of earlier works take this route. Yasuma et al. provided a cost function to describe spectral recovery error that is optimized to select filters for the generalized assorted pixel (GAP) camera [12]. However, the design and spatial arrangement of the GAP camera lacks freedom, while impacting the quality of spectral recovery significantly. Chi et al. [1] heuristically minimize the condition number of spectral sensitivity of channels to make the spectral recovery specifically robust to noise. Other work [24] optimizing spectral reflectance targets can also be used to select channels. All these methods cannot guarantee spectral error minimization and therefore optimal spectral reconstruction. Recently, Arad et al. [2] used different three-channel combinations to reconstruct multispectral images and choose the best combination. However, it can not be extended to multi-channel selection directly since it is too time-consuming and may depend on specific spectral reconstruction method.

Demosaicing: demosaicing methods have been devised independent of the other elements of a spectral imaging pipeline. Interestingly, it provides a diverse view of a conducive filter array and demosaicing methods. While some prior works [25] emphasize that the spectral sensitivities of neighboring pixels should be highly correlated to minimize inter-channel demosaicing error, other works (e.g., [26]) demonstrate the benefits of completely independent demosaicing fostered by low correlation between narrow band channels. Many works [21,27] treat the channel with highest sampling rate and largest throughput as the dominant channel and complete missing values of other channels that are considered dependent on the guide channel. In addition, a recent work uses a channel-dependent demosaicing strategy for a non-channel-dominant narrow-band MSFA [28].

Comparison with Proposed Work: the aforementioned literature survey reveals that mutually correlated problems of channel selection, filter array design, demosaicing methods and imaging noise have always been addressed in exclusion and therefore failed to yield solutions that will be optimal in terms of accuracy and efficiency for the final output of the entire spectral imaging pipeline. Therefore, we attempt to address all these correlated issues in a unified manner and optimize them together.

3. Modeling Error in Spectral Recovery

Let the spectral power distribution (SPD) function of the spectrum at spatial coordinate $(x,y)$ be $s(x,y,\lambda )$. Consider a multi-spectral camera with N channels ($N>3$). Let channel i ($1\le i\le N$) have the spectral sensitivity function $m_i(\lambda )$ that can be obtained from specification sheets or measured using a broadband light source using some low-cost methods [29]. Let the response of channel i of the camera to $s(x,y,\lambda )$ be $x_i$.

In a practical system, $x_i$ is comprised of ground truth, ${\tilde{x}}_i$, and errors, ${ϵ}_i$. If the channel i at $(x,y)$ is interpolated during demosaicing, then ${ϵ}_i={\zeta }_i+{\delta }_i$ where ${\zeta }_i$ denotes error due to imaging noise and ${\delta }_i$ denotes demosaicing error. For pixels where channel i is captured directly, ${\delta }_i=0$. Therefore,

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill x_i& ={\tilde{x}}_i+{ϵ}_i={\int }_{{\lambda }_1}^{{\lambda }_2}s(\lambda )m_i(\lambda )d\lambda +{ϵ}_i\hfill \\ & ={\int }_{{\lambda }_1}^{{\lambda }_2}s(\lambda )m_i(\lambda )d\lambda +{\zeta }_i+{\delta }_i.\hfill \end{array}\end{array}$$

(1)

We can write the above equation using matrices considering all channels as

$$X=Ms+ϵ=Ms+\zeta +\delta ,$$

(2)

where $[{\lambda }_1,{\lambda }_2]$ is the range of wavelength of visible light (e.g., 380–780 nm), M is a known matrix of dimension $N\times K$ where each row of M denotes one of the channels at a spectral resolution of K, s is the $K\times 1$ SPD function to be recovered and X denotes the $N\times 1$ known response vector and $ϵ$, $\zeta$ and $\delta$ are all $N\times 1$ error vectors.

Let us assume that the estimated spectrum $\widehat{s}$ is given by multiplying X with reconstruction matrix W. Using a pseudo-inverse matrix $W^{+}=M^T{(M{M}^T)}^{-1}$ or a Wiener pseudo inversion $W^{+}=Cor{r}_s{M}^T{(MCor{r}_s{M}^T)}^{-1}$ [30] with an approximated autocorrelation matrix $Cor{r}_s$ to represent the reconstruction matrix, the recovered spectrum $\widehat{s}$, is given by

$$\widehat{s}=WX=W(Ms+ϵ)=WMs+Wϵ.$$

(3)

We use the mean squared error ($MSE$) between the original and the recovered spectrum to describe the average error in the estimation as

$$\begin{array}{c}MSE=\mathrm{Tr}\left\{\mathrm{E}[(s-\widehat{s}){(s-\widehat{s})}^T]\right\}.\hfill \end{array}$$

(4)

Now, replacing Equation (3) in Equation (4), we get

$$\begin{array}{cc}\hfill MSE& =\mathrm{Tr}\left\{E[(s-WMs-Wϵ){(s-WMs-Wϵ)}^T]\right\}\hfill \\ & =\mathrm{Tr}\left\{\mathrm{E}[(Hs-Wϵ){(Hs-Wϵ)}^T]\right\},\hfill \end{array}$$

(5)

where $\mathrm{E}[.]$ is the expected values of a variable, $H=(I-WM)$, and $\mathrm{Tr}\left\{.\right\}$ denotes the trace of a square matrix defined as the sum of the elements on the main diagonal. Please note that pseudo-inverse reconstruction method only provides a first approximation. Therefore, the $MSE$ is not very accurate and only gives an upper-bound of total errors. Since the errors $ϵ$ are independent from the spectrum s, $\mathrm{E}[s{ϵ}^T]=0$ and $\mathrm{E}[ϵs^T]=0$. Therefore, $MSE$ can be rewritten as

$$\begin{array}{c}MSE=\mathrm{Tr}\left\{HCor{r}_s{H}^T\right\}+\mathrm{Tr}\left\{WCor{r}_{ϵ}{W}^T\right\},\hfill \end{array}$$

(6)

where $Cor{r}_s$, $Cor{r}_{ϵ}$ are the autocorrelation matrices $\mathrm{E}[s{s}^T]$, $\mathrm{E}[ϵ{ϵ}^T],$ respectively. Interestingly, in the above equation, the first term describes the error due to spectral recovery while the latter accounts for the error due to imaging noise and demosaicing. Next, we analyze the statistical properties of the natural multi-spectral images in order to model and approximate the correlation matrix $Cor{r}_s$ and $Cor{r}_{ϵ}$. Finally, we also describe how MSFA patterns and demosaicing interpolation affect demosaicing error and noise, respectively.

3.1. Spectral Characteristics of Natural Images

In order to study this, we use three hyperspectral datasets—CAVE dataset, Harvard dataset, and one that we ourselves capture using a Surface Optics SOC-730 (Surface Optics Corporation, San Diego, CA, USA) camera with a $1024\times 1024$ spatial resolution and a 2 nm (400–1000 nm) spectral resolution—this dataset contains 60 images of a sample, which is shown in Figure 2. Prior work has shown that multi-spectral images can be decomposed into Cartesian products of spectral and spatial components [31]. Therefore, we discuss them separately.

Spectral components—prior works observe the following: (a) illumination and reflectance spectra of natural or man-made objects and phenomena are relatively smooth [32] and therefore frequency-limit functions can be used to approximate them [33]; (b) in addition to being smooth, these illumination and reflectance spectra also show remarkably similar energy distribution in the frequency domain (Figure 3). In fact, the surfaces formed by these distribution functions have a shape close to an exponential function with an exponent of (−2). Therefore, they follow Steven’s power law of human perception. Steven’s Law is the fundamental empirical law of human perception that most human responses to input environmental stimuli is related by a power function [34].

We illustrate the aforementioned properties in Figure 3. To eliminate the effect of varying illumination and brightness of the images, we first compute the difference spectrum $\Delta s=s-\mu ,$ where $\mu$ is the mean value in spectral direction of multispectral images. For each $\Delta s$, we compute

$$E[|\Delta s_{\mathcal{F}}(k){|}^2]\propto 1/k^{2-\beta },$$

(7)

where $\Delta s_{\mathcal{F}}(k)$ is the Fourier transform of difference spectrum $\Delta s(\lambda )$, k is the frequency of spectrum, $2-\beta$ is the frequency exponent, and $\beta$ clusters around 0 for natural spectra.

Spatial components—it is well known that the RGB real-world images, including both natural landscapes and man-made environments, also follow the Power Law [35,36] and tend to be scale invariant. This property has already been widely used in many computer graphics applications [37], e.g., exposing forgeries, imaging denoising. Interestingly, we also observe this property across different bands of real-world multispectral images (Figure 3). As in spectral components, we can show for spatial components as well that the difference band images, $\Delta I=I-\mu ,$ where $\mu$ is the mean of the spectral band across space, exhibits the same property as:

$$E[|\Delta I_{\mathcal{F}}(f,\theta ){|}^2]\propto A(\theta )/f^{2-\alpha (\theta ),}$$

(8)

where $\Delta I_{\mathcal{F}}$ is the 2D Fourier transform of difference band-images $\Delta I$, $(f,\theta )$ is the polar coordinates of 2D frequency of band-images, $A(\theta )$ and $2-\alpha (\theta )$ are the scaling function and the frequency exponent function respectively for each orientation $\theta$. Here, $\alpha (\theta )$ clusters around 0 for natural images. Furthermore, for natural objects, the scaling function $A(\theta )$ clusters around 1; while, for man-made objects, the value of $A(\theta )$ for oblique orientation, ${\theta }_o$, is much smaller than its value for horizontal and vertical orientations, ${\theta }_h$ and ${\theta }_v,$ respectively. This is due to the fact that man-made objects usually contain horizontal and vertical edges [35].

3.2. Modeling Recovery Error

The statistical fact that spatial and spectral components of multi-spectral images follow a power law is not dependent on any particular data set. Therefore, this provides a prior that we use to approximate recovery error $Tr\left\{HCor{r}_s{H}^T\right\}$ in Equation (6). In order to achieve this, we approximate $Cor{r}_s$ by exploiting the content independent power law and the Wiener–Khinchin theorem. Here, every spectral wavelength sample point $s_i$ is treated the same. Mathematically, we assume $s_i=\Delta s_i+\mu$ ($1\le i\le K$) have the same distribution with the mean value $\mu$ and the standard deviation $\sigma$; therefore, $E[s_i^2]$ = $E[s_j^2]$ = ${\mu }^2+{\sigma }^2$ ($1\le i,j\le K$).

We approximate $Cor{r}_s(i,j)$ in matrix $Cor{r}_s$ using the autocorrelation value $Cor{r}_s(i,j)={\mu }^2+{\eta }_1{C}_s(\Delta \lambda )$, where $\Delta \lambda$ is the distance between spectral band i and j, and ${\eta }_1$ is a scale factor. According to the Wiener–Khinchin theorem, the autocorrelation of a signal could be represented by using the inverse Fourier transform of power spectral function. Thus, a general form of autocorrelation value $C_s(\Delta \lambda )=\int \Delta s(\lambda )\Delta s(\lambda \pm \Delta \lambda )d\lambda$ can be described as:

$$C_s={\mathcal{F}}^{-1}\{E[|\Delta s_{\mathcal{F}}{(k)|}^2]\}.$$

(9)

Finally, our approximation of $Cor{r}_s$ can be expressed as

$$Cor{r}_s={\mu }^2+{\eta }_1\left[\begin{array}{cccc}{C}_s(0)& C_s(1)& \cdots & C_s(n-1)\\ C_s(1)& \ddots & & C_s(n-2)\\ ⋮& & \ddots & ⋮\\ C_s(n-1)& C_s(n-2)& \cdots & C_s(0)\end{array}\right].$$

(10)

Therefore, the final recovery error is estimated by $Tr\left\{HCor{r}_s{H}^T\right\}$. Compared with the methods constructing the autocorrelation matrix derived from some datasets, we use this error model because it only depends on the average intensity $\mu$ and the scale factor ${\eta }_1$. The effectiveness of the model will be verified in Section 5.1.

3.3. Modeling Demosaicing Error and Imaging Noise

In addition to spectral recovery error, the demosaicing error and imaging noise considerably affect the quality of the reconstruction. However, unlike the spectral recovery error, these errors are heavily dependent on (1) the design of MSFA (multi-spectral filter array), which involves the distribution and placement of the different channels in the filter array; and (2) the demosaicing technique that decides the order of the demosaicing process based on the dependency between channels or lack thereof.

Design of MSFA: prior works [39,40] have shown that a good MSFA pattern should have the following properties: (i) it should be spatially uniform to ensure robustness in the face of image sensor imperfections; (ii) it should be periodic to ensure efficiency of image reconstruction; and (iii) it should be neighbor consistent (each channel has the same neighbor channels) to ensure immunity to optical/electrical cross talk between neighboring pixels. Reference [40] presents a binary tree based MSFA design method that assures all the above properties and is elegant in design and implementation. Therefore, we adapt this design and customize it to suit our needs.

In the binary tree based MSFA design [40], the generation of the MSFA is an iterative process of binary splitting as illustrated in Figure 4a. A parent channel evenly divides into two children channels in each splitting, increasing the number of channels by one while scaling down the sampling rate of the children created by half. Reference [40] shows that, since the MSFA patterns satisfy the three properties above, the sibling channels are exchangeable. We can use binary trees to represent such MSFA patterns (Figure 4b) where a leaf node at level l has a sampling rate of $2^{-l}$. Figure 4c,d shows that the patterns generated may not be unique. However, the binary tree for a particular sampling rate combination is unique.

Demosaicing strategy: channel-independent demosaicing completes an undemosaiced channel at a subsampled resolution without dependence of other channels. On the contrary, channel-dependent demosaicing completes an undemosaiced channel at a subsampled resolution using the content from one of more different channels. The channel dependent methods are categorized as fully dependent or partially dependent based on if they use all the channels or a subset thereof for demosaicing, respectively.

Our observation reveals that neither “one-to-many channel-dependent demosaicing” [21] nor “all channel- independent demosaicing” [26] strategy are optimal (see Figure 5). Initially, we start with full resolution mosaic of the most important channel (e.g., green), we use dashed arrows from channel a to channel b to denote the demosaicing of the subsampled resolution channel b with the guidance of the full resolution channel a: self loops indicate channel-independent demosaicing of channel b without guidance while other arrows indicate channel-dependent demosaicing (Figure 5). It is important to note that channel selection, channel spatial arrangement, demosaicing strategy and imaging noise levels are mutually correlated. Therefore, we devise an adaptive demosaicing based on channels and imaging noise levels. In the following sections, we will model and quantify channel-independent demosaicing and channel-dependent demosaicing errors and use it for the design of the MSFA and the demosaicing method.

3.4. Channel-Independent Demosaicing Error

First, we consider channel-independent demosaicing for a sensor with resolution R. Let the set of pixels’ measuring channel i (at level l in the binary tree) directly on the sensor be $I_i$. Let us consider a pixel p and $x_i(p)$ denote the response of channel i $(1\le i\le m)$ at the pixel p. If pixel $p\in I_i$, the response $x_i(p)$ is the addition of noise-free ground truth response ${\tilde{x}}_i(p)$ and imaging noise ${\zeta }_i(p)$. If $p\notin I_i$, i.e., $x_i(p)$ is interpolated from other directly measured pixels during demosaicing, then $x_i(p)$ is the sum of ${\tilde{x}}_i(p)$, ${\zeta }_i(p)$ and demosaicing error ${\delta }_i(p)$.

Therefore, the response $x_i$ at pixel p can be written as:

$$x_i(p)=B_i(p)\left({\tilde{x}}_i(p)+{\zeta }_i(p)\right)+\overline{B_i}(p)\sum _{p^{\prime }\in P}{w}_{l_i}(p,p^{\prime })\left({\tilde{x}}_i(p^{\prime })+{\zeta }_i(p^{\prime })\right),$$

(11)

where $B_i(p)$ is a binary flag that 1 for $p\in I_i$ and 0, otherwise, and $w_{l_i}(p,p^{\prime })$ are corresponding interpolation weights, determined by both the level $l_i$ and the Gaussian filter used. Combining Equations (2) and (11), we find the demosaicing error and imaging noise of demosaiced pixel p as:

$$\left\{\begin{array}{c}{\zeta }_i(p)=B_i(p){\zeta }_i(p)+\overline{B_i}(p)\sum _{p^{\prime }\in P}{w}_{l_i}(p,p^{\prime }){\zeta }_i(p^{\prime }),\hfill \\ {\delta }_i(p)=\overline{B_i}(p)\left({\tilde{x}}_i(p)-\sum _{p^{\prime }\in P}{w}_{l_i}(p,p^{\prime }){\tilde{x}}_i(p^{\prime })\right).\hfill \end{array}\right.$$

(12)

The noise of directly observed pixels come from measurement errors and quantization errors resulting from analog to digital conversion and therefore spatially independent [41]. We use ${\overline{\zeta }}_i^2$ to denote the average directly observed noise variance in $I_i$. Noise variance of channel i, $E[{\zeta }_i^2]$ is a constant that can be estimated using previous method by Shimano [42] as:

$$\begin{array}{cc}\hfill E[{\zeta }_i^2]& =\frac{1}{R}\sum _p\left(B_i(p){\overline{\zeta }}_i^2+\overline{B_i}(p)\sum _{p^{\prime }\in P}{w}_{l_i}^2(p,p^{\prime }){\overline{\zeta }}_i^2\right)\hfill \\ & =\left(2^{-l_i}+(1-2^{-l_i}){\phi }_{l_i}\right){\overline{\zeta }}_i^2,\hfill \end{array}$$

(13)

where ${\phi }_{l_i}$ denotes ${\displaystyle \sum _p}{\sum }_{p^{\prime }\in P}{w}_{l_i}^2(p,p^{\prime })$ for abbreviation. Demosaicing error of channel i at pixel p is given by the square of difference between ground truth response $x_i(p)$ and the interpolated value obtained from the ground truth responses of the neighboring pixels. Let us now assume that, for each pixel p, the square of ground truth response has the same distribution. Thus, for two pixels, p and q, $p\ne q$, we have $E[{\tilde{x}}_i^2(p)]=E[{\tilde{x}}_i^2(q)]$. Therefore, demosaicing error variance $E[{\delta }_i^2]$ of channel i is given by the average of the variance $mean({\delta }_i{(p)}^2)$ of all the pixels in the image and is derived from Equation (14) as:

$$\begin{array}{cc}\hfill E[{\delta }_i^2]& =\frac{1}{R}\sum _p\overline{B_i}(p){\left({\tilde{x}}_i(p)-\sum _{p^{\prime }\in P}{w}_l(p,p^{\prime }){\tilde{x}}_i(p^{\prime })\right)}^2\hfill \\ & ={\alpha }_{l}E[{\tilde{x}}_i^2]+\sum _{p,q}^{p\ne q}{\beta }_l(p,q)E[{\tilde{x}}_i(p){\tilde{x}}_i(q)],\hfill \end{array}$$

(14)

where ${\alpha }_l$ and ${\beta }_l(p,q)$ are coefficients of $E[{\tilde{x}}_i^2(p)]$ and $E[{\tilde{x}}_i(p){\tilde{x}}_i(q)],$ respectively, in the expansion of Equation (14). We model $E[{\tilde{x}}_i(p){\tilde{x}}_i(q)]$ as $M_{i}E[{\tilde{I}}_i(p){\tilde{I}}_i{(q)}^T]M_i^T$, where $M_i$ is the known spectral sensitivity of i th channel. We assume a spatially independent autocorrelation matrix $E[{\tilde{I}}_i(p){\tilde{I}}_i{(p)}^T]$ that can be simplified to $E[{\tilde{I}}_i{\tilde{I}}_i^T]$. Similar to spectral variable $s_i$, we assume the distribution of ${\tilde{I}}_i$ has the mean value $\mu$ and the the standard deviation $\sigma ,$ where ${\tilde{I}}_i=\mu +\Delta {\tilde{I}}_i$. We use $C_I(\Delta u,\Delta v)=\int \int \Delta \tilde{I}(u,v)\Delta \tilde{I}(u\pm \Delta u,v\pm \Delta v)dudv$ to represent the ratio between correlation matrix $E[{\tilde{I}}_i(p){\tilde{I}}_i{(q)}^T]$ and $E[{\tilde{I}}_i{\tilde{I}}_i^T]$:

$$E[{\tilde{I}}_i(p){\tilde{I}}_i{(q)}^T]=\frac{{\mu }^2+{\eta }_2{C}_I(\Delta u,\Delta v)}{{\mu }^2+{\eta }_2{C}_I(0,0)}E[{\tilde{I}}_i{\tilde{I}}_i^T],$$

(15)

where $(\Delta u,\Delta v)$ is given by the absolute value of vector difference between locations of pixel p and q, and ${\eta }_2$ is a scale factor. Similar to the approximation of spectral autocorrelation matrix, $C_I(\Delta u,\Delta v)$ can also be approximated using Wiener–Khinchin theorem as:

$$C_I={\mathcal{F}}^{-1}\{E[|\Delta I_{\mathcal{F}}{(f,\theta )|}^2]\}.$$

(16)

Since demosaicing error is independent from imaging noise, the variance $E[{ϵ}_j^2]$ in correlation matrix $Cor{r}_e$ is the sum of variance of demosaicing error and imaging noise: $E[{ϵ}_i^2]=E[{\delta }_i^2]+E[{\zeta }_i^2]$.

3.5. Channel-Dependent Demosaicing Error

In channel dependent demosaicing, we denote the guidance channel and the demosaiced channel by g and r from an upper and lower level $l_g$ and $l_r$ respectively. Prior works show that color-difference invariance in local regions is a reasonable assumption for chrominance (red/blue) channel demosaicing [43] yielding low computational complexity [44]. This color-difference invariance results in a linear demosaicing operation that is convenient for representation and computation. Therefore, we construct a demosaicing error model using the same color-difference invariance and express the response $x_r(p)$ of demosaiced channel r at position p as:

$$\begin{array}{cc}\hfill x_r(p)=& {\tilde{x}}_g(p)+{ϵ}_g(p)+B_r(p)\left({\tilde{x}}_r(p)+{\zeta }_r(p)-{\tilde{x}}_g(p)-{ϵ}_g(p)\right)\hfill \\ & +{\overline{B}}_r(p)\sum _{p^{\prime }\in P}{w}_{l_r}(p,p^{\prime })\left({\tilde{x}}_r(p^{\prime })+{\zeta }_r(p^{\prime })-{\tilde{x}}_g(p^{\prime })-{ϵ}_g(p^{\prime })\right).\hfill \end{array}$$

(17)

From the above equation, demosaicing error variance can be transformed into a form like Equation (14) as:

$$E[{\delta }_r^2]={\alpha }_{l_r}E[{({\tilde{x}}_{r\backslash g})}^2]+\sum _{p,q}^{p\ne q}{\beta }_{l_r}(p,q)E[({\tilde{x}}_{r\backslash g}(p))({\tilde{x}}_{r\backslash g}(q))],$$

(18)

where ${\tilde{x}}_{r\backslash g}$ denotes ${\tilde{x}}_r-{\tilde{x}}_g$. Here, the demosaicing error just contains the interpolation error caused in this demosaicing. The imaging noise from demosaiced channel and accumulative errors from guide channel are induced into the noise variance as:

$$\begin{array}{cc}\hfill E[{\zeta }_r^2]& =\frac{1}{R}\sum _p\left[B_r(p){\overline{\zeta }}_r^2+\overline{B_r}(p)\sum _{p^{\prime }\in P}{w}_{l_r}^2(p,p^{\prime }){\overline{\zeta }}_r^2+\overline{B_r}(p)(1+\sum _{p^{\prime }\in P}{w}_{l_r}^2(p,p^{\prime }))\frac{E[{ϵ}_g^2]-2^{-l_g}{\overline{\zeta }}_g^2}{1-2^{-l_g}}\right]\hfill \\ & =\left(2^{-l_r}+(1-2^{-l_r}){\phi }_{l_r}\right){\overline{\zeta }}_r^2+\frac{(1-2^{-l_r})(1+{\phi }_{l_r})(E[{ϵ}_g^2]-2^{-l_g}{\overline{\zeta }}_g^2)}{1-2^{-l_g}},\hfill \end{array}$$

(19)

where $E[{ϵ}_g^2]$ denote the error variance of guide channel g. Note that, in Equation (19), the noise variance of channel r consists of two components: (1) the imaging noise variance ${\zeta }_r^2$ has the same form of channel-independent demosacing (as in Equation (13)); (2) the error variance $E[{ϵ}_g^2]$ of guidance channel with the coefficient decided by the level of the channel r and the filter used in the demosaicing algorithm. Similar to the case of channel-independent demosaicing, the error variance of demosaiced channel r can be calculated by $E[{ϵ}_r^2]=E[{\delta }_r^2]+E[{\zeta }_r^2]$.

The relationship between channels in demosaicing can be represented using a Demosaicing Forest as illustrated in Figure 6. In this forest, the channel demosaiced without guidance of other channels is the root of a tree in the forest, and a pair of parent and child nodes in a tree represents a pair of guidance and demosaiced channels in channel-dependent demosaicing. Hence, for each channel i, there is a unique demosaicing chain containing all reachable nodes from the root to channel i, denoted by $DC(i)$. Intuitively, if two channels i and j come from different trees, i.e., $DC(i)\cap DC(j)=\varnothing$, then the two channels are independent; otherwise, the two channels would have a lowest common chain elementk, where the level of k, $l_k=max(level(DC(i)\cap DC(j)))$. The errors of these two channels are then partially correlated due to sharing of the same component from the error of channel k. Thus, for the off-diagonal elements $E[{ϵ}_i{ϵ}_j]$ (the correlation between the errors of channel i and j) in correlation $Cor{r}_e$, we have:

$$\begin{array}{c}E[{ϵ}_i{ϵ}_j]=\left\{\begin{array}{c}0,\mathrm{if}DC(i)\cap DC(j)=\varnothing ,\hfill \\ \frac{(1-2^{-l_i}){}^{\frac{3}{2}}(1-2^{-l_j}){}^{\frac{3}{2}}(E[{ϵ}_k^2]-2^{-l_k}{\overline{n}}_k{}^2)}{\prod _p(1+{\phi }_{l_p}){}^{-\frac{1}{2}}\prod _q(1+{\phi }_{l_q}){}^{-\frac{1}{2}}(1-2^{-l_k})},\mathrm{otherwise},\hfill \end{array}\right.\hfill \\ s.t.i\ne j,p\in DC(i)-DC(k),q\in DC(j)-DC(k).\hfill \end{array}$$

(20)

Therefore, the objective function $MSE$ is built with a few parameters: the Gaussian filter we used in demosaicing, scale factors $\frac{{\eta }_1}{{\mu }^2}$ and $\frac{{\eta }_2}{{\mu }^2}$, and imaging noise ${\overline{\zeta }}_i^2$ of each channel.

4. Imaging Optimization Method

In this section, to optimize reconstruction accuracy, we will seek possible combinations of channels, MSFA patterns, and demosaicing patterns that minimize the objective function (Equation (6)) with specific parameters. We propose an optimization method that selects n channels from m candidate channels ($n\ll m$) and determines the MSFA patterns and demosaicing orders to facilitate accurate recovery of spectral reflectance of most natural and man-made objects.

Considering manufacturing complexity, we assume the number of channels, n, on MSFA to be relatively small ($n<7$). Using the binary-tree-based constraint, we know that the number of possible patterns $\mathcal{P}$ with n channels are also small. For example, for $n<7$, num ($\mathcal{P})=2,3,5$ when $n=4,5,6$. Therefore, we can enumerate every possible pattern for a specific number of channels, and calculate the optimal channel selection, the mosaic pattern, and demosaicing order for each pattern.

However, searching for an ordered subset of n channels from m candidates that would minimize the objective function is NP hard. Therefore, an exhaustive search is time-consuming. For example, with $n=6$ and $m=100$, we will need to evaluate $P(100,6)\approx {10}^{12}$ permutations. Therefore, we apply a discrete particle swarm optimization (DPSO) method, detailed in Algorithm 1 to search for the optimal imaging parameters.

Particle swarm optimization [45] is a global search method that operates on a population of particles where each particle represents a candidate solution. The method moves these particles around in the search-space and updates the position and velocity of each particle iteratively.

Algorithm 1 The Proposed DPSO Method

1: Input: Set of candidate channels: $\left\{M_j(\lambda )\right|j=1,\dots ,m\}$; Number of output channels: n; Pattern $\mathcal{P}$. 2: Output: Selected ordered set of n channels; Decided demosaicing order $\mathcal{O}$. 3: Initialization: 4: Generate population $X_i$ and velocity $V_i(1\le i\le P)$ 5: $[X^{gb},{\mathcal{O}}^{gb}]=findbest(X_i^{pb},\mathcal{P})$ 6: for j = 1 … G do 7:  for i = 1 … P do 8:   $X_i^{pb}=f(M_{X_i},\mathcal{P})<f(M_{X_i^{pb}},\mathcal{P})?X_i:X_i^{pb}$ 9:   $V_i=w(V_{i-1}+c_{1}r\ast (X_i^{pb}\ominus X_i)+c_{2}r\ast (X^{gb}\ominus X_i))$ 10:   ${\overline{V}}_i=reducemat(V_i,p)$ 11:   $X_i=X_i\oplus {\overline{V}}_i$ 12:  end for 13:  $[X^{gb},{\mathcal{O}}^{gb}]=findbest(X_i^{pb},\mathcal{P})(1\le i\le P)$ 14: end for 15: return n ordered channels $X^{gb}$, demosaicing order ${\mathcal{O}}^{gb}$;

Let the number of particles be P and the number of iteration be G. Note that ith particle $X_i$ denotes ith candidate solution where each solution consists of n ordered channel selected from m candidate channels, and therefore each particle $X_i$ is a $m\times 1$ integer vector. In vector $X_i$, the element with positive value k indicates the channel is the k th selected channel in pattern $\mathcal{P}$, while those with 0 value indicate the channels that are not selected. The function $findbest()$ finds the global optimal tuple of ordered channels $X^{gb}$ and the optimal demosaicing order ${\mathcal{O}}^{gb}$ for the specific pattern $\mathcal{P}$. The behavior of the algorithm is adjusted using four parameters: $c_1$, $c_2$, w, and p. For each iteration j, the algorithm maintains the global best solution $X^{gb}$ and the local best solution $X_i^{pb}$ of each particle. The objective function $f(M_{X_i},\mathcal{P})$ calculates the $MSE$ of the imaging with selected channels $X_i$ and specific pattern $\mathcal{P}$ for every possible demosaicing orders and then return the minimum $MSE$.

Each iteration of particle i uses standard procedures in a basic particle swarm optimization but with modified operations. We first calculate the velocity from $X_i$ to $X_i^{pb}$ and the velocity from $X_i$ to $X^{gb}$ using the mutation operation ⊖. The velocity from vector A to B is defined as a differential matrix of swapping pairs from A to B. Then, the algorithm updates the velocity $V_i$ using weighted sum of previous velocity $V_{i-1}$, $X_i^{pb}\ominus X_i$, and $X^{gb}\ominus X_i$; here, the weighted values indicate the possibility to swapping pairs. Then, the algorithm uses the function $reducemat()$ to retain the principal component of velocity $V_i$ by clipping the velocity with a threshold probability p; finally, it updates the solution by adding clipped velocity ${\overline{V}}_i$ to the current solution $X_i$. The adding operation $X_i\oplus {\overline{V}}_i$ is defined as swapping the status(selected with index or unselected) of the xth channel and the yth channel in set $X_i$, where $(x,y)$ is a swapping pair in matrix ${\overline{V}}_i$ [46].

5. Evaluation and Comparison

In this section, we evaluate our error and noise models in simulation using popular multi-spectral image datasets. Next, we compare our optimization method with GAP camera design and Chi’s filter selection method. Finally, we explore the optimal number of channels and how imaging noise level affects MSFA design and demosaicing strategy. The candidate set of channels used in simulation is constructed with spectral transmission data of 30 off-the-shelf coating color filters from http://www.rosco.com/filters (see Figure 7).

5.1. Error Models

We evaluate the estimated spectral recovery from our method and compare it with Chi’s [1] and Shen’s method [18]. We randomly choose 100 different channel combinations from the candidate set, each channel combination consisting of six different channels. Next, we calculate average estimated spectral recovery error only (ignoring the error due to noise and demosaicing) for each channel combination using different spectral dataset in [47], which contains natural spectral reflectances in abundance.

Figure 8 shows the relationship between objective function values for different channel combinations and the corresponding spectral recovery error on two different datasets for Chi’s, Shen’s and our method. Note Chi’s method is content-independent, but Shen’s method uses a prior based on a large generic spectral dataset [32]. Unlike other methods, our method shows a strong linear relationship between evaluated objective function and the estimated spectral recovery error, confirming the higher accuracy of our objective function as a predictor for spectral recovery error.

To evaluate our overall models, we consider three different random combination of channels, MSFA patterns, and demosaicing orders as shown in Figure 9. We adopt the bilinear interpolation method and the binary tree-based generic(BTG) demosaicing method [26] as the channel-independent demosaicing, and adopt state-of-the-art guided filter (GF) method [48] and residual interpolation (RI) method [49] as the channel-dependent demosaicing in reconstruction. In error estimation, we used scale factor ${\eta }_1=0.025$ and ${\eta }_2=0.15$, and random Gaussian noise with $SNR\approx 50$ db to the response of camera. The values of the scale factors are directly obtained from the statistics of the multispectral images in the used datasets.

The comparison between estimated errors and actual reconstruction errors using different demosaicing methods is shown in Figure 10. The actual errors are acquired by calculating the difference between ground truth and reconstructed results. It is worth noting that impact of different combination is much more than that of different demosaicing methods. Therefore, the combination is the primary factor while the methods are secondary. Our estimated errors are close to the average actual errors and illustrate the suitability of our model as a good descriptor for overall reconstruction error.

5.2. Comparison with Previous Methods

To verify the effectiveness of our optimization method, we select channels from candidate channel set using our method, decide channel arrangement on MSFA pattern and demosaicing order using three methods—GAP camera [12], Chi and Monno’s method [1,14], and our method—as shown in Figure 11, and compare spectral reconstruction accuracy of our method with the other two methods on three different datasets—’CAVE’ spectral dataset [38], Harvard’s spectral dataset [31], and our dataset. Chi and Monno’s method is a combination of Chi’s channel selection method and Monno’s MSFA design and demosaicing order.

We used the demosaicing method in [26] and added random Gaussian noise to responses of camera with $SNR\approx 50$ db. To quantify the reconstruction accuracy of multispectral images, especially on edges, we use relative error $|\Delta s(\lambda )|$ defined as:

$$|\Delta s(\lambda )|=\sqrt{\frac{\int {(s(\lambda )-\widehat{s}(\lambda ))}^{2}d\lambda }{\int s^2(\lambda )d\lambda }}.$$

(21)

Table 1. Spectral reconstruction error of three optimized channels (see Figure 11) of GAP, Chi and Monno’s method, and our method.

Methods	CAVE’s Dataset			Harvard’s Dataset			Our Dataset
	Max	Mean	Std	Max	Mean	Std	Max	Mean	Std
GAP	0.4518	0.3231	0.0880	0.2849	0.0794	0.0854	0.2602	0.1964	0.0421
Chi and Monno’s	0.4381	0.2852	0.0867	0.2498	0.0744	0.0753	0.2231	0.1880	0.0428
Ours	0.4115	0.2775	0.0814	0.2196	0.0629	0.0679	0.1999	0.1586	0.0342

shows reconstruction error of the three optimized channel combinations in Figure 11 along with the error statistics (mean, maxinum, standard deviation). The optimized channels of our method show superior results to GAP camera’s and Chi and Monno’s results irrespective of the dataset used.

Figure 12 shows comparison of the three methods on two multi-spectral images. Since our method takes into account the demosaicing error, our estimated spectral error near sharp edges in images is significantly smaller than others.

6. Discussion

Optimal Number of Channels. In order to explore the best number of mosaiced channels on MSFAs, in Figure 13, we plot the estimated error using our error model and optimization methods against the different number of channels. Note that the recovery error decreases with an increase in number of channels, while the errors caused by demosaicing and imaging noise increase with the increasing number of channels. This is expected since more channels would result in more accurate recovery while sacrificing spatial resolution for each channel. Therefore, the sweet spot is where the sum of these two curves has a minima. In Figure 13, we see that, using our candidate channels, this is around 5–6 channels.

Behavior of Imaging Noises. Adopting our model and optimization method, we can also explore how imaging noise affects optimal channel selection, design of MSFA, and demosaicing strategy via simulation. Figure 14 shows the optimized results under different imaging noise levels. It can be seen that, under higher noise levels, the optimized channel combination would have higher light throughput to preserve SNR of cameras (the observation is similar to previous works [13,20]). We also found that, in the presence of higher noise, the MSFA pattern tends to be more uniform, that is, the binary tree is more balanced. Furthermore, the demosaicing method tends to be more channel-independent. This is expected since, under high noise levels, the magnitude of noise is much larger than the demosaicing error, while channel-dependent demosaicing propagates noise between channels, although it can reduce demosaicing error. These conclusions provide guidelines to effectively reduce the search space in our optimization method.

7. Conclusions

In summary, we propose new error models for multi-spectral imaging and utilize the models to select optimal channel combination, the pattern of MSFA, and demosaicing order for multi-spectral imaging. We verified the effectiveness of our method and compared it with previous methods.

Our method can also be applied to other similar problems. For example, projection display is a dual imaging system, and, therefore, selecting few efficient primaries for a spectrally accurate display is a dual problem of our channel selection. In other areas, such as wireless sensor networking, where sensor or observation selection [50] is critical, our optimization might provide an effective solution in acceptable time.

In the future, we plan to apply our method to such varied domains, our immediate interest being in multi-spectral display. Moreover, we plan to explore the relationship between spectral sensitivity of tunable filter camera channels and accuracy of multi-spectral image registration.