Designing Quadcolor Cameras with Conventional RGB Channels to Improve the Accuracy of Spectral Reflectance and Chromaticity Estimation

Abstract

Quadcolor cameras with conventional RGB channels were studied. The fourth channel was designed to improve the estimation of the spectral reflectance and chromaticity from the camera signals. The RGB channels of the quadcolor cameras considered were assumed to be the same as those of the Nikon D5100 camera. The fourth channel was assumed to be a silicon sensor with an optical filter (band-pass filter or notch filter). The optical filter was optimized to minimize a cost function consisting of the spectral reflectance error and the weighted chromaticity error, where the weighting factor controls the contribution of the chromaticity error. The study found that using a notch filter is more effective than a band-pass filter in reducing both the mean reflectance error and the chromaticity error. The reason is that the notch filter (1) improves the fit of the quadcolor camera sensitivities to the color matching functions and (2) provides sensitivity in the wavelength region where the sensitivities of RGB channels are small. Munsell color chips under illuminant D65 were used as samples. Compared with the case without the filter, the mean spectral reflectance rms error and the mean color difference (ΔE₀₀) using the quadcolor camera with the optimized notch filter reduced from 0.00928 and 0.3062 to 0.0078 and 0.2085, respectively; compared with the case of using the D5100 camera, these two mean metrics reduced by 56.3%.

1. Introduction

Traditional tricolor cameras are equipped with red, green, and blue channels, which are used to capture long-wavelength, medium-wavelength, and short-wavelength image signals, respectively. This RGB camera can not only capture color images but also recover surface spectral reflectance pixel by pixel. Spectral reflectance images have broad application prospects in the fields of color reproduction, medical diagnosis, agricultural detection, and remote sensing [1,2,3,4,5,6,7,8,9,10]. Using cameras to measure spectral reflectance has the advantages of fast measurement speed, high pixel resolution, low cost, and a small form factor [11,12,13,14], compared with imaging spectrometers [15,16]. Therefore, the application potential of cameras is high.

However, the disadvantage of using a camera is its measurement accuracy, since the spectral reflectance is estimated from the camera signal. The sensitivity spectrum of conventional cameras ranges from 400 nm to 700 nm due to the presence of an infrared-cut (IR-cut) filter. Conventional cameras usually use an IR-cut filter to filter out long-wave light. Since infrared is not visible, filtering can significantly improve the accuracy of camera color calibration. Using an RGB camera, the estimation maps the 3D signal to a 31D spectrum with a sampling wavelength interval of 10 nm. This estimation method requires a set of training/reference samples to calculate the model parameters. The sample set can be prepared according to a specific application. In addition, using image recognition technology to identify the target of the field application, an estimation method with a suitable sample set can be selected to improve accuracy.

Studies have shown that the accuracy of spectral reflectance estimation can be improved by using a quadcolor camera that maps the 4D signal to a 31D spectrum [17,18]. Figure 1a shows the Bayer color filter array (CFA) for a conventional RGB camera, in which a unit cell contains one red, two green, and one blue square filter. This arrangement has the advantage of high spatial resolution. A quadcolor camera uses a modified CFA without sacrificing spatial resolution, as shown in Figure 1b, where a fourth color filter (labeled “Q”) replaces one of the green filters.

References [17,18] studied a quadcolor camera whose 4th channel has no color filter and whose spectral sensitivities of RGB channels are the same as those of Nikon D5100. The spectral sensitivity of the 4th channel is the product of the sensitivity of the silicon image sensor and the transmittance of the IR-cut filter. This quadcolor camera was called RGBF camera, where the 4th channel was called the F-channel because it was color-filter-free. In [18], the mean root-mean-square (RMS) error of spectral reflectance (E_Ref) and the color difference (ΔE₀₀) of the test samples reduced from 0.01785 and 0.4778 using D5100 to 0.00928 and 0.3062 using the RGBF camera, respectively. The objective of this article is to investigate the filter design for the 4th channel to further improve the accuracy of spectral reflectance and chromaticity estimation.

Many estimation methods have been proposed for spectral reflectance recovery, which are based on basis spectra [19,20,21,22], Wiener estimation [12,23,24], regression [25,26,27,28,29,30,31,32], and interpolation [17,18,33,34,35,36,37]. Except for the basis-spectrum method, these methods have the problem of irradiance dependence, i.e., the spectral shape of the estimated reflectance depends on the irradiance of the object surface. As mentioned above, it requires a sample set that is prepared under illuminants. However, the irradiance on the object surface may change in field applications.

Reference [18] proposed an irradiance-independent look-up table (II-LUT) method based on interpolation to estimate spectral reflectance. The results showed that the estimation accuracy of the II-LUT method is better than that of the basis-spectrum method. When the interpolation method estimates a sample, it refers to the reference samples adjacent to it in the signal space, while the base spectrum method refers to all training samples even if it uses the weighting technique [20]. Therefore, the II-LUT method has higher accuracy. This article adopts the II-LUT method to recover spectral reflectance.

The 4th channel using a short-pass or long-pass filter was studied in [17], where the traditional irradiance-dependent spectrum reconstruction methods were used. The study showed that the use of short-pass and long-pass filters is beneficial for reducing chromaticity error and reflectance error, respectively. In addition, when either the chromaticity error or the reflectance error decreases, the other error increases. In this article, a cost function consisting of the mean reflectance error and the weighted mean chromaticity error was used for optimizing the filter, where band-pass filters and notch filters were considered. The results show that the use of a notch filter can effectively reduce reflectance and chromaticity errors, but if ultra-low chromaticity errors are desired, using a band-pass filter is better.

This article is organized as follows. Section 2 and Section 3 describe the materials, assessment metrics, and methods, where the assumed color filter spectral transmittance, the optimization cost function, and the II-LUT method are presented in Section 3.1, Section 3.2, and Section 3.3, respectively. Section 4.1 and Section 4.2 present and discuss the results using band-pass optical filters and notch optical filters, respectively. Section 4.3 shows some examples of the recovered spectra. Section 5 provides the conclusions.

2. Materials and Assessment Metrics

2.1. Camera Spectral Sensitivities

By convention, we sampled a spectrum from 400 nm to 700 nm in steps of 10 nm, where the spectrum vector S = [S(400 nm), S(410 nm), …, S(700 nm) ]^T and S(λ) is the spectral amplitude at wavelength λ. There are a total of M_w =31 sampling wavelengths.

The RGB channels of the quadcolor camera considered are those of the Nikon D5100 camera [38]. The spectral sensitivity vectors of the three channels and the 4th channel were designated as S_CamR, S_CamG, S_CamB, and S_CamQ, respectively. Without the filter specified in Section 3.1, the 4th channel is the F-channel of the RGBF camera considered in [17,18], and its spectral sensitivity vector was designated as S_CamF.

2.2. Color Samples

The 1268 matt Munsell color chips under the illuminant D65 were taken as color samples [39]. The same 202 reference/training samples and 1066 test samples in [17,18,37] were adopted.

The reflection spectrum vector from a color chip is

$${\mathit{S}}_{\mathrm{Reflection}}={\mathit{S}}_{\mathrm{Ref}}\circ {\mathit{S}}_{\mathrm{D}65},$$

(1)

where S_Ref and S_D65 are the spectral reflectance vector of a color chip and the spectrum vector of the illuminant D65, respectively, and the operator $\circ$ is the element-wise product. Reference [37] showed the reference/training samples and test samples in the CIELAB color space, where the CIE 1931 color matching functions (CMFs) were adopted.

The spectral sensitivity vector D_CamU and the signal value U of the U channel of the quadcolor camera, where U = R, G, B, and Q, under the white balance condition are

$${\mathit{D}}_{\mathrm{Cam}U}={\mathit{S}}_{\mathrm{Cam}U}/{({\mathit{S}}_{\mathrm{White}}\circ {\mathit{S}}_{\mathrm{D}65})}^{\mathrm{T}}{\mathit{S}}_{\mathrm{Cam}U},$$

(2)

U = S_Reflection^TD_CamU

(3)

In Equation (2), S_White is the spectral reflectance vector of a white card. The same white card in [17,18,37] was taken. The signal vector representing a quadcolor camera was designated as C = [R, G, B, Q]^T. Reference [17] showed the reference/training samples and test samples in the signal space of the RGBF camera.

2.3. Assessment Metrics

When the reflection spectrum vector S_Rec was reconstructed using the II-LUT method, the spectral reflectance vector S_RefRec is the vector S_Rec divided by the vector S_D65 element by element. This article used the same metrics as [18] to assess the reconstructed reflection spectrum and the recovered reflectance spectrum. They are the RMS error E_Ref = ||S_RefRec − S_Ref||₂/M_w^1/2, where the operation ||·||₂ stands for 2-norm; the goodness-of-fit coefficient GFC = ||S_RefRec^TS_Ref||₂/||S_RefRec||₂||S_Ref||₂; the CIEDE2000 ΔE₀₀, which measures the color difference between S_Rec and S_Reflection; and the spectral comparison index (SCI), which is an index of metamerism [40].

From reconstructed test samples, we calculated the mean μ, standard deviation σ, 50th percentile PC50, 98th percentile PC98, and maximum MAX of the metrics E_Ref, ΔE₀₀, and SCI, for which smaller values are better; we also calculated the mean μ, standard deviation σ, 50th percentile PC50, RGF99, and minimum MIN of the metric GFC, for which larger values are better. RGF99 is the ratio of good-fit samples with a GFC > 0.99 [37]. The spectral curve shape is well fitted for GFC > 0.99 [36,41].

3. Methods

3.1. Color Filter Spectral Transmittance

Both band-pass optical filters and notch optical filters were considered.

The band-pass filter is a super-Gaussian filter (SGF), whose spectral transmittance was assumed to be

$$T_{SGF}\left(\lambda \right)=T_{S0}\exp \left[-{\left|{\displaystyle \frac{\lambda -{\lambda }_{SGF}}{{\sigma }_W}}\right|}^{\alpha }\right],$$

(4)

where T_S0 is the maximum transmittance; λ_SGF is the filter peak wavelength; and σ_W and α are parameters determined by the filter width Δλ_F and edge width Δλ_E. The filter width is the full width at half maximum transmittance, i.e., T_SGF(λ_SGF + 0.5Δλ_F) = 0.5T_S0. The edge width Δλ_E was defined as the wavelength interval from 0.1T_S0 to 0.9T_S0. From Equation (4) and the Δλ_F and Δλ_E definitions, we have

$${\displaystyle \frac{\Delta {\lambda }_{\mathrm{E}}}{\Delta {\lambda }_{\mathrm{F}}}}={\displaystyle \frac{1}{2}}{\left(\ln 2\right)}^{-{\displaystyle \frac{1}{\alpha }}}\left[{\left(\ln 10\right)}^{{\displaystyle \frac{1}{\alpha }}}-{\left(-\ln 0.9\right)}^{{\displaystyle \frac{1}{\alpha }}}\right],$$

(5)

Given the values of Δλ_F and Δλ_E, the α in Equation (4) is solved from Equation (5), and the parameter σ_W is

$${\sigma }_W={\displaystyle \frac{1}{2}}{\left(\ln 2\right)}^{-{\displaystyle \frac{1}{\alpha }}}\Delta {\lambda }_F.$$

(6)

The notch filter is an inverted super-Gaussian filter (ISGF), whose spectral transmittance was assumed to be

$$T_{ISGF}\left(\lambda \right)=T_{I0}\left(1-\exp \left[-{\left|{\displaystyle \frac{\lambda -{\lambda }_{ISGF}}{{\sigma }_W}}\right|}^{\alpha }\right]\right),$$

(7)

where T_I0 is the maximum transmittance; λ_ISGF is the central wavelength; and σ_W and α are parameters determined by the wavelength separation Δλ_FS and edge width Δλ_E. The wavelength separation Δλ_FS is twice the wavelength interval from zero to the half maximum transmittance, i.e., T_ISGF(λ_ISGF + 0.5Δλ_FS) = 0.5T_I0. Given the values of Δλ_FS and Δλ_E, the values of σ_W and α also can be solved from Equations (5) and (6), except that Δλ_F is replaced by Δλ_FS. This article assume T_S0 = T_I0 = 0.96, which will not affect the results.

The spectral sensitivity vector S_CamQ = T $\circ$ S_CamF, where T is the spectral transmittance vector calculated from either Equation (4) or Equation (7).

3.2. Optimization Cost Function

If only spectral reflectance recovery is desired, the optimization cost function can be set to the mean E_Ref. However, color reproduction applications require low color differences. Therefore, we consider a cost function defined as follows:

θ = E_Ref,μ + γ ΔE_00,μ,

(8)

where E_Ref,μ and ΔE_00,μ are the mean E_Ref and mean ΔE₀₀, respectively; the Lagrange multiplier γ is a weighting factor that controls the contribution of color difference. Since the values of E_Ref,μ and ΔE_00,μ may differ greatly in magnitude, the Lagrange multiplier was chosen logarithmically. γ = 10^−p, where the power p was selected from −3 to 0 in steps of 0.05, i.e., 61 Lagrange multipliers were considered.

For a given value of γ, the case of using an SGF with wavelength λ_SGF from 400 nm to 700 nm in steps of 10 nm was considered. For each SGF wavelength, the full width Δλ_F and the edge width Δλ_E were optimized for the minimum cost function. It was found that the cost function exhibits a highly nonlinear relationship with the full width Δλ_F and the edge width Δλ_E. The optimization used a brute-force grid search method to ensure that the solution was the global minimum. For practical considerations, the ranges of Δλ_F and Δλ_E were set to 50 nm to 400 nm and 20 nm to 200 nm, respectively. The optimization process for the case of using ISGF is the same as that for using SGF, and the ranges of the wavelength λ_ISGF, the full wavelength separation Δλ_FS, and the edge width Δλ_E are also the same.

3.3. II-LUT Method

The II-LUT method interpolates a test sample in the normalized signal space, defined as [18]

r = R/(R + G + B + Q),

(9)

g = G/(R + G + B + Q),

(10)

b = B/(R + G + B + Q).

(11)

We designated the normalized signal vector as c = [r, g, b]^T. Reference [18] showed the reference/training samples and test samples in the normalized signal space of the RGBF camera. A tetrahedron mesh can be generated from the normalized reference signal vectors.

If the tetrahedron enclosing a normalized test signal vector c is located, the interpolation method assumes that

$$\mathit{c}={\displaystyle \sum _{k=1}^4{\beta }_k{\mathit{c}}_k},$$

(12)

where c_k is the k-th normalized signal vector at the vertices of the tetrahedron, β_k is the barycentric coordinate, and

$$1={\displaystyle \sum _{k=1}^4{\beta }_k}.$$

(13)

The coefficients β_k, k = 1, 2, 3, and 4, are solved from Equations (12) and (13). From [18], we have

$${\mathit{S}}_{\mathrm{Rec}}={\displaystyle \sum _{k=1}^4\hspace{1em}{\eta }_k{\beta }_k{\mathit{S}}_k},$$

(14)

where S_k is the k-th reference spectrum for k = 1, 2, 3, and 4; the coefficient

η_k = (R + G + B + F)/(R_k + G_k + B_k + F_k).

(15)

In Equation (15), R_k, G_k, B_k, and F_k are the signal values of the k-th reference spectrum.

Since the vectors c and c_k in Equation (12) are independent of the irradiance on the object surface, the spectral shape of the reconstructed reflection spectrum vector S_Rec is independent of the irradiance. The signal values R, G, B, and F increase linearly with the irradiance, and the magnitude of the reconstructed reflection spectrum vector S_Rec also increases linearly with the irradiance.

In summary, there are five steps to interpolate the test sample using the II-LUT method, where the flow chart is shown in Figure 2.

• STEP 1: Convert the signal vector C of the test sample into the normalized signal vector c.
• STEP 2: Locate the tetrahedron enclosing vector c in the normalized signal space.
• STEP 3: Solve the coefficients β_k, k = 1, 2, 3, and 4, from Equations (12) and (13).
• STEP 4: Calculate the coefficients η_k, k = 1, 2, 3, and 4, according to Equation (15).
• STEP 5: Calculate the reconstruction spectrum S_Rec according to Equation (14).

Due to high saturation, 131 test samples were outside the convex hull of the tetrahedral mesh for the RGBF camera [18]. Since they cannot be interpolated, we extrapolated them using auxiliary reference samples, which are highly saturated samples measured using appropriately selected color filters and reference color chips. This article adopted the same filters and color chips as in [18] so that all test samples could be interpolated/extrapolated using the considered quadcolor cameras.

4. Results and Discussion

4.1. The Use of Band-Pass Optical Filter

Figure 3a shows the mean color difference ΔE₀₀ versus the mean RMS error E_Ref using the quadcolor cameras with several SGF wavelength λ_SGF, where the black diamond symbol labeled RGBF indicates that the 4th channel has no color filter. Using the RGBF camera, mean E_Ref = 0.00928 and mean ΔE₀₀ = 0.3062. As expected, for a given SGF wavelength λ_SGF, the mean ΔE₀₀ decreases with increasing Lagrange multiplier γ. Although 61 Lagrange multipliers ranging from 10⁻³ to 1 are selected for each SGF wavelength, the optimized full width Δλ_F and edge width Δλ_E may be the same or nearly the same for multiple Lagrange multipliers. For example, for λ_SGF = 550 nm, there are three solution groups, as shown by the green dots in Figure 3a. When γ < 10⁻², the performance of the first group is about the same, all close to the RGBF data point. When 10⁻² ≤ γ < 10^−1.55, the performance of the second group is also about the same, where the mean E_Ref ≈ 0.01072 and mean ΔE₀₀ ≈ 0.1392. When γ ≥ 10^−1.55, the 3rd group has smaller mean ΔE₀₀, but larger mean E_Ref compared with the 2nd group.

In Figure 3a, the pink dots show the results for λ_SGF = 400 nm, where the minimum mean ΔE₀₀ is with γ = 1. Since the filter peak wavelength is 400 nm, the SGF is equivalently a short-pass filter in the visible wavelength region. Figure 3b shows the spectral transmittance for λ_SGF = 400 nm, where γ = 10⁻³ and 1. The difference between the two cases is not significant.

Table 1. Full width Δλ_F and edge width Δλ_E of several SGFs. The CMFMisF, mean ΔE₀₀, and mean E_Ref using D5100, RGBF, and the quadcolor cameras with the filters are shown. The best values of CMFMisF and ΔE₀₀ are shown in bold red. The best value of E_Ref is shown in bold green.

λSGF (nm)	D5100	RGBF	400		480		680		700
γ	NA	NA	10−3	1	10−3	1	10−3	1	10−3	1
ΔλF (nm)	NA	NA	327.8	301.6	179.2	155.8	56.8	173	50	223.8
ΔλE (nm)	NA	NA	99.9	98.0	78.5	83.6	50.0	20.0	35.8	20.0
CMFMisF	0.1498	0.1495	0.1277	0.0973	0.1336	0.0986	0.1496	0.1491	0.1498	0.1490
mean ΔE00	0.4778	0.3062	0.1655	0.1096	0.1733	0.1107	0.3022	0.2898	0.3012	0.2898
mean ERef	0.01785	0.00928	0.00962	0.01047	0.00959	0.01057	0.00875	0.00894	0.00867	0.00893

shows the full width Δλ_F and edge width Δλ_E of the two SGFs. The spectral sensitivities using the two SGFs are also shown in Figure 3b, where their peak sensitivities are around 525 nm.

The blue “+” symbols in Figure 3a indicate the results for λ_SGF = 500 nm, where the minimum mean ΔE₀₀ is slightly larger than that for λ_SGF = 400 nm and γ = 1 for both. Such solutions are band-pass filters in the visible wavelength region. Figure 4a is the same as Figure 3a except for λ_SGF shorter than 500 nm, and the case of λ_SGF = 400 nm is also shown for comparison. Note that their minimum mean ΔE₀₀ are about the same. In fact, their spectral sensitivities are also about the same. Figure 4b shows the spectral transmittance and spectral sensitivities for λ_SGF = 480 nm, where γ = 10⁻³ and 1. Comparing Figure 4b with Figure 3b, although their spectral transmittances are quite different, their spectral sensitivities are similar for the same γ value. The full width Δλ_F and edge width Δλ_E for λ_SGF = 480 nm are shown in Table 1. From the spectral sensitivities of γ = 1 shown in the two figures, it can be seen that the peak wavelengths are 520 nm and 525 nm for λ_SGF = 400 nm and 480 nm, respectively; the full width at half-maximum is about 100 nm in both cases.

The mean ΔE₀₀ is related to the CMF mismatch factor (CMFMisF), defined as [17]

CMFMisF = (σ_x² + σ_y² + σ_z²)^1/2,

(16)

σ_m = ||u_Fit − u_m||₂/|| u_m||₁,

(17)

for m = x, y, and z. In Equation (17), the operation ||∙||₁ is 1-norm, so the relative RMS error σ_m has the sense of the ratio of standard deviation to the mean; u_m is the CMF vector, u_x = [$\overline{x}$(λ₁), $\overline{x}$(λ₂), …, $\overline{x}$(λ_Mw)]^T, u_y = [$\overline{y}$(λ₁), $\overline{y}$(λ₂), …, $\overline{y}$(λ_Mw)]^T, u_z = [$\overline{z}$(λ₁), $\overline{z}$(λ₂), …, $\overline{z}$(λ_Mw)]^T, and λ_j = 400 + (j – 1) × 10 nm for j = 1, 2, …, M_w; u_Fit is the least squares fit of a CMF vector:

u_Fit = β_RS_CamR + β_GS_CamG + β_BS_CamB + β_QS_CamQ,

(18)

and β_R, β_G, β_B and β_Q are the fitting coefficients. The equation for calculating σ_m in [17] has been corrected to Equation (17).

Figure 5a shows the spectral sensitivities of red, green, blue, and F-channels of the RGBF camera under the white balance condition. Figure 5b shows the least squares fits of the spectral sensitivity vectors of the RGBF camera to the CMF vectors. Figure 6a,b are the same as Figure 5a,b, respectively, except that the filter for λ_SGF = 480 nm and γ = 1 is applied to the 4th channel. The fits of spectral sensitivities to CMFs shown in Figure 6b are better than those shown in Figure 5b. Table 1 shows the CMFMisF and mean ΔE₀₀ for the cases shown in Figure 3b and Figure 4b, where the results using the D5100 and RGBF cameras are also shown. For the case of λ_SGF = 400 nm and γ = 1, both the CMFMisF and mean ΔE₀₀ are minimum in the table.

Table 2. Assessment metric statistics for the spectrum reconstruction of the 1066 test samples using quadcolor cameras. The filter of the 4th camera channel is indicated. The RGBF camera is without a filter. The best values are shown in bold red.

Metric	Filter	w/o	SGF			ISGF
	λSGF/λISGF (nm)	NA	480	480	400	600	600	600	590
	γ	NA	10−3	1	1	10−3	10−1.65	1	10−3
ERef	mean μ	0.00928	0.00959	0.01057	0.01047	0.00787	0.00792	0.01055	0.00780
	std σ	0.00961	0.00934	0.01238	0.01198	0.00763	0.00788	0.01155	0.00756
	PC50	0.00603	0.00661	0.00667	0.00677	0.00538	0.00534	0.00714	0.00520
	PC98	0.04290	0.03872	0.04569	0.04498	0.03248	0.03257	0.04563	0.02872
	MAX	0.07328	0.07939	0.12590	0.14719	0.06071	0.07282	0.10320	0.06686
GFC	mean μ	0.99880	0.99891	0.99859	0.99862	0.99918	0.99917	0.99873	0.99923
	std σ	0.00342	0.00243	0.00458	0.00450	0.00231	0.00243	0.00316	0.00218
	PC50	0.99962	0.99959	0.99961	0.99960	0.99972	0.99972	0.99957	0.99973
	MIN	0.93870	0.96364	0.89529	0.89440	0.94917	0.95098	0.93413	0.95152
	RGF99	0.98311	0.99062	0.98124	0.98311	0.99156	0.98874	0.98874	0.99156
ΔE00	mean μ	0.30618	0.17333	0.11066	0.10960	0.18919	0.18143	0.13121	0.20859
	std σ	0.36686	0.14776	0.09288	0.09193	0.19811	0.19134	0.11139	0.22091
	PC50	0.18992	0.12470	0.08618	0.08561	0.13567	0.13044	0.10220	0.14965
	PC98	1.53876	0.60145	0.37249	0.37480	0.67183	0.62592	0.40659	0.83240
	MAX	3.38551	0.99401	0.76597	0.76625	2.24135	2.24168	1.32258	2.24832
SCI	mean μ	3.03986	2.26793	2.19007	2.18351	2.41083	2.37555	2.26130	2.47910
	std σ	3.35378	1.79260	1.81088	1.79095	2.23688	2.24981	1.99434	2.35225
	PC50	1.99793	1.75189	1.63791	1.63828	1.72560	1.70528	1.64829	1.75844
	PC98	16.37406	8.80518	7.84793	7.83212	8.24166	8.12446	8.06511	9.14487
	MAX	27.56020	13.56906	19.45063	18.72469	28.13400	29.57345	20.46814	25.54052

shows the assessment metric statistics for the three cases using SGF: (1) λ_SGF = 480 nm and γ = 10⁻³, (2) λ_SGF = 480 nm and γ = 1, and (3) λ_SGF = 400 nm and γ = 1. We can see that the statistics of the latter two cases are about the same due to the similar spectral sensitivities shown in Figure 3b and Figure 4b. Figure 7a–d show the E_Ref, GFC, ΔE₀₀, and SCI histograms for the case of λ_SGF = 480 nm and γ = 1, where the enlarged sub-figures show the distributions of the test samples with worse metric values. The counts of the vertical axis of the figures are the number of samples within the 1066 test samples. The color difference ΔE₀₀ of all test samples using the SGF is less than 1.0, but at the expense of increased spectral reflectance error E_Ref.

From Figure 3a, it can be seen that the mean E_Ref can be reduced using the filter of long SGF wavelength, e.g., the black dots for 700 nm. The case using the SGF of λ_SGF = 700 nm and γ = 10⁻³ has the minimum mean E_Ref. Figure 8a is the same as Figure 3a, except that SGF wavelengths range from 660 nm to 700 nm in steps of 10 nm. Figure 8b shows the spectral transmittance and spectral sensitivities for the cases of λ_SGF = 700 nm, where γ = 10⁻³ and 1. The SGF with λ_SGF = 700 nm is equivalently a long-pass filter in the visible wavelength region. If a band-pass filter is preferred, Figure 8b also shows the spectral transmittance and spectral sensitivities for λ_SGF = 680 nm and γ = 10⁻³. Table 1 gives the full width Δλ_F and edge width Δλ_E for the cases shown in Figure 8b.

The spectral sensitivities of a commercial tricolor camera, such as D5100, were designed to fit CMFs as closely as possible for color calibration. Due to the characteristics of CMFs, they have low spectral sensitivity in the long-wavelength region. Since the main sensitivity of the 4th channel shown in Figure 8b ranges from 650 nm to 700 nm, the spectral sensitivities of the quadcolor camera cannot fit the CMFs well, and the CMFMisF is larger than that shown in Figure 3b and Figure 4b. Although mean ΔE₀₀ increases, the case of using the long-wavelength SGF has the advantage of a lower mean E_Ref. The 4th-channel signal can provide a clue of the spectral characteristics of the reflection spectrum in the long-wavelength region for better spectrum reconstruction.

4.2. The Use of Notch Optical Filter

From Section 4.1, we know that although CMFMisF and mean ΔE₀₀ increase, a sensitivity lobe around 685 nm is required to effectively reduce the mean E_Ref. As will be shown below, in addition to the long-wavelength sensitivity lobe, ISGF can also provide another sensitivity lobe around 500 nm to reduce CMFMisF, thereby reducing both the mean E_Ref and mean ΔE₀₀.

Figure 9a is the same as Figure 3a, except that ISGF is used. For the visible wavelength region, optical filters with λ_ISGF = 400 nm and 700 nm are equivalently long-pass and short-pass, respectively. The effective notch filter wavelength lies between the two extremes. When λ_ISGF = 550 nm, there is only one solution group with the mean E_Ref ≈ 0.00832 and mean ΔE₀₀ ≈ 0.2367, which is worse than when using λ_ISGF = 600 nm. The red “x” symbols indicate that using the ISGF with wavelength λ_ISGF = 600 nm can effectively reduce both the mean E_Ref and mean ΔE₀₀. Figure 9b is the same as Figure 9a, except that λ_ISGF is from 580 nm to 620 nm in steps of 10 nm, where the optimal case is λ_ISGF = 600 nm in green dots for both low mean E_Ref and mean ΔE₀₀. Figure 10a shows the enlarged plot of the mean ΔE₀₀ versus the mean E_Ref for λ_ISGF = 600 nm, where the data points of three Lagrange multipliers γ are indicated with the “+” sign. Note that the case of γ = 10^−1.65 is the optimal value that represents a trade-off between the mean E_Ref and the mean ΔE₀₀.

Figure 10b shows the spectral transmittance and spectral sensitivity for λ_ISGF = 600 nm and γ = 10⁻³, 10^−1.65, and 1. There are two sensitivity lobes in each case. The peak sensitivity of the long-wavelength lobe is around 685 nm for all three cases. The peak sensitivity of the short-wavelength lobe is at 500 nm, 510 nm, and 440 nm when γ = 10⁻³, 10^−1.65, and 1, respectively. Note that CMF values are lower around 500 nm. The short-wavelength lobe shifts to around 500 nm for γ = 10⁻³ and 10^−1.65, so it may also lead to a decrease in the mean E_Ref in addition to the mean ΔE₀₀. When γ = 1, the short-wavelength lobe shifts to 440 nm, improving the fit to CMFs in the short-wavelength region and making the long-wavelength lobe very small, thus reducing the CMFMisF and the mean ΔE₀₀.

Table 3. The same as Table 1, except that ISGF is used and full width Δλ_F is replaced by full wavelength separation Δλ_FS. The best values of CMFMisF and ΔE₀₀ are shown in bold red. The best values of E_Ref are shown in bold green.

λISGF (nm)	D5100	RGBF	600				590
γ	NA	NA	10−3	10−1.65	10−1.2	1	10−3	10−1	10−0.5	1
ΔλFS (nm)	NA	NA	171.6	158.6	164.8	399.2	165.4	336	380.2	361
ΔλE (nm)	NA	NA	53.1	24.7	26.0	68.9	20.7	46.0	51.0	47.5
CMFMisF	0.1498	0.1495	0.1355	0.1363	0.1333	0.0979	0.1455	0.1077	0.1111	0.1112
mean ΔE00	0.4778	0.3062	0.1892	0.1814	0.1796	0.1312	0.2085	0.1351	0.1347	0.1346
mean ERef	0.01785	0.00928	0.00787	0.00792	0.00808	0.01055	0.00780	0.00997	0.01011	0.01013

is the same as Table 1, except that ISGF is used and the full width Δλ_F is replaced by the full wavelength separation Δλ_FS, where the cases using the ISGF with λ_ISGF = 600 nm and four values of γ are shown. We can see that the CMFMisF and mean ΔE₀₀ decrease as the γ value increases, except for the case of γ = 10^−1.65; mean E_Ref increases with γ. The exception is worth noting, where γ = 10^−1.65. Compared with the case of γ = 10⁻³, its CMFMisF is slightly larger, but the mean ΔE₀₀ is slightly smaller. This shows that the mean ΔE₀₀ does not decrease monotonically with CMFMisF. However, if the difference in CMFMisF values is large, the mean ΔE₀₀ does decrease with CMFMisF.

Figure 11a,b are the same as Figure 6a,b respectively, except that the ISGF of λ_ISGF = 600 nm and γ = 10^−1.65 is applied to the 4th channel. Comparing Figure 11b with Figure 5b and Figure 6b, the spectral sensitivity of this case is better fitted to CMFs than the RGBF camera due to the sensitivity lobe between 450 nm and 550 nm in Figure 11a, but worse than the camera using the SGF of λ_SGF = 480 nm and γ = 1. The sensitivity lobe between 650 nm and 700 nm in Figure 11a deteriorates the fit, as shown in Figure 11b, but its CMFMisF is still smaller than that of the RGBF camera, and it can provide a “sharper” clue of long-wavelength spectral characteristics for spectrum reconstruction than the RGBF camera. Therefore, both the mean ΔE₀₀ and the mean E_Ref are lower than those of the RGBF camera.

Figure 9b shows that the mean E_Ref is the smallest when λ_ISGF = 590 nm and γ is small. This figure shows four data points represented by blue “+” symbols for λ_ISGF = 590 nm. Figure 12a shows the full wavelength separation Δλ_FS, edge width Δλ_E, and cost function versus the Lagrange multiplier γ. The cost function increases smoothly with γ, but there are only four solutions. The four solutions are shown in Table 3, where the γ values shown are just for reference in that using multiple γ values may result in the same solution. Figure 12a also shows the mean ΔE₀₀, mean E_Ref, and weighted chromaticity to reflectance error ratio wCR, where wCR is the ratio of the second term to the first term of the cost function defined in Equation (8), i.e.,

wCR = γ ΔE_00,μ/E_Ref,μ.

(19)

From Figure 9b and Figure 12a, the wCR discontinuity at γ = 10^−1.5 shows the transition from the smallest mean E_Ref to the second-smallest mean E_Ref when λ_ISGF = 590 nm. Figure 12b shows the filter spectral transmittance and spectral sensitivity using λ_ISGF = 590 nm and with γ = 10⁻³, 10^−1.5, 10^−0.5, and 1. The spectral transmittances of the latter three cases are apparently different, but their spectral sensitivities are nearly the same. Due to their small long-wavelength sensitivity lobes, their mean E_Ref and mean ΔE₀₀ increase and decrease, respectively, compared with the case of γ = 10⁻³.

Table 2 shows the assessment metric statistics for three cases using the ISGF with λ_ISGF = 600 nm and γ = 10⁻³, 10^−1.65, and 1. The case of λ_ISGF = 590 nm and γ = 10⁻³ is also shown. The statistics for the cases with smaller values of γ are similar. The best values in all cases shown in Table 2 are shown in bold red.

Compared with the case without a filter, using the ISGF of 590 nm in Table 2, the mean and PC98 of E_Ref decrease from 0.00928 and 0.0429 to 0.0078 and 0.02872, respectively; the percentage of test samples with a GFC > 0.99 (RFG99) increases from 98.311% to 99.156%; the mean and PC98 of ΔE₀₀ decrease from 0.30618 and 1.53876 to 0.20859 and 0.8324, respectively. The improvements achieved by using the ISGF of 590 nm are significant.

Compared with the case without a filter, using the SGF of 400 nm in Table 2, the mean and PC98 of ΔE₀₀ decrease to 0.1096 and 0.3748, respectively. Such reductions are significant, but the spectral reflectance error is larger.

Figure 7a–d also show the E_Ref, GFC, ΔE₀₀, and SCI histograms for the cases of λ_ISGF = 590 nm (γ = 10⁻³) and 600 nm (γ = 10^−1.65). Since the mean values of E_Ref, ΔE₀₀, and SCI are low for all cameras in Table 2, and the mean values of GFC are high, the counts in the low-value slots of E_Ref, ΔE₀₀, and SCI and the counts in the high-value slot of GFC are very high. It can be seen from the enlarged sub-figures in Figure 7a,b that the counts in the worse slots using ISGF are apparently less than those using SGF and no filter (RGBF). From the enlarged sub-figures in Figure 7c,d, it can be seen that the counts in the worse slots without using the filter (RGBF) are larger than those using SGF and ISGF. The improvement in the four metrics using the ISGF is evident.

4.3. Recovered Spectrum Examples

Figure 13 shows the reconstructed spectra of the light reflected from the (a) 5Y 8.5/8, (b) 10P 7/8, (c) 2.5R 4/12, (d) 2.5G 7/6, (e) 10BG 4/8, and (f) 5PB 4/12 color chips, respectively, using the cameras considered in Figure 7, where the target spectra are also shown for comparison. The vertical axis of the figures is the spectral power density of the light reflected from the color chips before reaching the camera. The spectral reflectance recovered from the light reflected from the color chips in Figure 13a–f are shown in Figure 14a–f, respectively. These eight color chips were also taken as examples in [17,18], where the cases of (a) 5Y 8.5/8 and (b) 10P 7/8 color chips are interpolation examples and the other four are extrapolation examples. Regardless of using the II-LUT method or other methods in the literature, spectral recovery of extrapolation samples is challenging due to the lack of highly saturated training/reference samples. We can see that the recovered reflection spectra are accurate except for the (c) 2.5R 4/12 and (f) 5PB 4/12 color chips using the SGF with λ_SGF = 480 nm. The camera signal from the channel using the SGF with λ_SGF = 480 nm cannot provide a clue of the spectral characteristic from 650 nm to 700 nm. Thus, using the ISGFs shown in the figures is more reliable for spectrum reconstruction.

Table 4. Color difference ΔE₀₀ and RMS error E_Ref for the cases shown in Figure 13a–f and Figure 14a–f, respectively. The case with the worst E_Ref value is shown in bold blue.

Metrics	ΔE00				ERef
Camera	RGBF	SGF 480 nm	ISGF 590 nm	ISGF 600 nm	RGBF	SGF 480 nm	ISGF 590 nm	ISGF 600 nm
(a)	0.43872	0.09394	0.06724	0.04982	0.01841	0.01161	0.01025	0.00969
(b)	0.14554	0.06427	0.15826	0.17569	0.01926	0.01794	0.0204	0.02247
(c)	0.06335	0.07416	0.04467	0.03945	0.00561	0.06686	0.0060	0.00647
(d)	0.09678	0.09912	0.18562	0.17538	0.00304	0.00362	0.00505	0.00485
(e)	1.33057	0.26433	0.64571	0.59835	0.01300	0.00645	0.00778	0.00843
(f)	0.46338	0.17233	0.32332	0.16304	0.02361	0.02504	0.01378	0.01546

shows the color difference ΔE₀₀ and RMS error E_Ref for the cases shown in Figure 13a–f and Figure 14a–f, respectively. For the worst case, recovering the (c) 2.5R 4/12 chip using the SGF with λ_SGF = 480 nm, the E_Ref value is poor, but the ΔE₀₀ value is small. For this case, although the reconstructed reflection spectrum is poor, the spectrum with wavelengths between 650 nm and 700 nm contributes little to the tristimulus values.

5. Conclusions

The design of quadcolor cameras for spectral reflectance recovery was studied using the irradiance-independent look-up table method to reconstruct the reflection spectra from color samples. The RGB channels of the quadcolor camera were assumed to be the same as those of Nikon D5100. The 4th channel using a band-pass optical filter or notch optical filter was considered. The super-Gaussian function and the inverted super-Gaussian function were assumed to represent the spectral transmittance of the band-pass filter and the notch filter, respectively. The parameters of the filter were optimized to minimize the cost function consisting of the spectral reflectance error and the weighted color difference, where the weighting factor controls the contribution of the color difference.

The results of using the band-pass filters are summarized below.

(1)

The spectral sensitivity of the 4th channel with a peak wavelength between 500 nm and 550 nm reduces color difference owing to the improved fit of the camera spectral sensitivities to CMFs. Compared with the case without a filter, the mean color difference ΔE₀₀ can be reduced from 0.3062 to 0.1096 when a filter was used, but the mean spectral reflectance error E_Ref increases from 0.00928 to 0.01047.

(2)

The spectral sensitivity of the 4th channel with a peak wavelength of around 685 nm reduces the spectral reflectance error because the sensitivities of RGB channels are small in this wavelength region, but it impairs the CMF fit. Compared with the case without a filter, the mean E_Ref and mean ΔE₀₀ can be reduced to 0.00867 and 0.3012, respectively, when a filter was used.

When using the notch filter, the spectral sensitivity of the 4th channel can have two lobes with peak wavelengths at about 500 nm and 685 nm. Compared with the case without a filter, the mean E_Ref and mean ΔE₀₀ can be reduced to 0.0078 and 0.2085, respectively, after using the optimized notch filter centered at 590 nm. The reduced spectral reflectance error is smaller than that obtained using the band-pass filter optimized for minimum E_Ref. Therefore, both spectral reflectance error and color difference can be effectively reduced.

Compared with D5100, where the mean E_Ref and mean ΔE₀₀ are 0.01785 and 0.4778, respectively, using the quadcolor camera with the optimized notch filter can result in both mean metrics being 56.3% lower. Similarly, compared with the RGBF camera, using the quadcolor camera with the optimized notch filter can reduce the mean E_Ref and mean ΔE₀₀ by 15.9% and 31.9%, respectively.

Since the RGB channels were not changed in this study, the optimized filters shown are for the Nikon D5100 camera. However, the spectral sensitivity design of a tricolor camera is usually as close to the CMFs as possible for color calibration, and its RGB signals cannot provide an effective clue of the spectral characteristics of a reflection spectrum in the long-wavelength region (650 nm to 700 nm) for spectrum reconstruction. Therefore, the technique of utilizing notch filters to improve the accuracy of spectral reflectance and chromaticity estimation can be applied to other quadcolor cameras modified from commercially available tricolor cameras.

Based on the above results, the design guidelines for the filters of a quadcolor camera are summarized as follows: Chromaticity accuracy can be increased by improving the fit of the camera spectral sensitivities to the CMFs. Spectral reflectance accuracy can be increased by adding a spectral sensitivity lobe in the long-wavelength region, where the CMF values are lower, but at the expense of reduced chromaticity accuracy. The optimal design is a trade-off between reflectance accuracy and chromaticity accuracy, depending on specific application requirements.