Auxiliary Reference Samples for Extrapolating Spectral Reflectance from Camera RGB Signals

Surface spectral reflectance is useful for color reproduction. In this study, the reconstruction of spectral reflectance using a conventional camera was investigated. The spectrum reconstruction error could be reduced by interpolating camera RGB signals, in contrast to methods based on basis spectra, such as principal component analysis (PCA). The disadvantage of the interpolation method is that it cannot interpolate samples outside the convex hull of reference samples in the RGB signal space. An interpolation method utilizing auxiliary reference samples (ARSs) to extrapolate the outside samples is proposed in this paper. The ARSs were created using reference samples and color filters. The convex hull of the reference samples and ARSs was expanded to enclose outside samples for extrapolation. A commercially available camera was taken as an example. The results show that with the proposed method, the extrapolation error was smaller than that of the computationally time-consuming weighted PCA method. A low cost and fast detection speed for spectral reflectance recovery can be achieved using a conventional camera.


Introduction
Surface spectral reflectance is useful for the color reproduction of industrial products and artwork [1][2][3]. It can be measured directly with an imaging spectrometer [4,5]. However, direct spectral measurements are expensive. Indirect measurements using the spectrum reconstruction technique are of interest [6][7][8][9][10][11][12][13][14]. The spectrum of the image pixel is reconstructed from the channel outputs of the image acquisition device. Since no diffractive optical imaging system is required, the indirect method has the advantages of a low cost and fast detection speed. Therefore, using a conventional camera makes more field applications possible, e.g., smartphone cameras used as sensors to measure surface spectral reflectance.
Orthogonal projection [6], principal component analysis (PCA) [7,8], Gaussian mixture [9], non-negative matrix transformation (NMT) [10,11] and interpolation [11][12][13][14] have been proposed for spectrum reconstruction. Indirect methods that require training spectra are also known as learning-based methods, such as orthogonal projection, PCA and NMT. The training spectra are used to derive basis spectra. The reconstructed spectrum is a linear combination of basis spectra. The coefficients of basis spectra can be solved from simultaneous equations describing the channel outputs of the imaging device. The accuracy of the reconstructed spectrum increases with the number of channels. For cases with a conventional tricolor camera, where only three channels are available, the accuracy of the reconstructed spectrum might not be high enough.
The interpolation method uses reference spectra to reconstruct a spectrum interpolated from input values, e.g., XYZ tristimulus values [11][12][13] and RGB signal values [14]. Due to the use of a look-up table (LUT) to store the reference spectra, this method is often referred to as the LUT method. The authors of [11][12][13][14] showed that the LUT method has the advantage of being more accurate than the PCA method, where the reference spectra for interpolation are the same as the training spectra for the PCA method. A spectrum is interpolated from its neighboring reference samples using the LUT method, whereas the basis spectra of the PCA method are derived from all training samples. The weighted PCA (wPCA) method was proposed to enhance the contribution of neighboring training samples in the CIELAB color space to the basis spectra [8,14]. Since basis spectra depend on the sample to be reconstructed, the computation time of the wPCA method is significantly increased compared to the conventional PCA method and the LUT method.
Learning-based methods require camera spectral sensitivities to formulate simultaneous equations describing the channel outputs. Camera spectral sensitivities can be directly measured using a monochromator [15], but accurate measurement is expensive. Without the use of a monochromator, the camera spectral sensitivities can be estimated by solving a quadric minimization problem [15,16]. An alternative approach is to estimate the spectral sensitivities including the camera and light source so that the reflectance spectrum can be calculated from the camera signals [17][18][19][20][21]. The estimation errors of spectral sensitivities cause additional errors in the reconstructed spectrum.
The LUT method does not require the spectral sensitivity functions because the reconstructed spectrum is interpolated from the measured spectra of the reference samples. However, if the sample lies outside the convex hull of the reference samples in the RGB signal space, it cannot be interpolated. Such a sample can be called an outside sample to distinguish it from the samples inside the convex hull. In the literature, modified PCA and NMT methods have been used to extrapolate outside samples [11][12][13][14], although spectral sensitivity functions are required. The authors of [11][12][13] considered interpolation in the XYZ color space, where spectral sensitivity functions were equivalently assumed to be the CIE color matching functions (CMFs). The authors of [14] considered interpolation in the RGB signal space, where the camera was assumed to follow the sRGB standard so that RGB signal values and XYZ tristimulus values can be converted to each other via the well-known sRGB matrix. This hypothetical camera is called the sRGB camera, and its spectral sensitivities are presented in [22].
The authors of [11,13] used 2D interpolation and 3D interpolation to extrapolate outside samples from reference samples, respectively. It is not guaranteed that the 2D interpolation method will extrapolate all outside samples [11]. The authors of [14] extrapolated outside samples from reference samples and additional reference samples; the latter are called model-based metameric spectra of extreme points (MMSEPs , respectively, where the maximum values of the signals are normalized to 1.0; the subscript T denotes the transpose operation. The metameric spectra are the reflection spectra from eight surfaces under D65 illumination. The spectral reflectance of the eight surfaces was constructed using the sRGB camera. The MMSEPs were equivalently constructed using the spectral sensitivities of the sRGB camera. Inspired by [14], we propose the use of auxiliary reference samples (ARSs) for extrapolating outside samples using the LUT method. ARSs are high-saturation samples. They are created using appropriately chosen color filters and color chips. Color filters are in turn mounted on the spectroradiometer to measure the spectrum of filtered reflection light from a color chip. The RGB signal values corresponding to the filtered reflection light are recorded by a camera mounted with the same color filter. Color filters and color chips are chosen so that outside samples can be enclosed by reference samples and ARSs in the RGB signal space for extrapolation. Numerical studies of the proposed method were carried out. A comparison of the LUT method utilizing ARSs, the LUT method utilizing MMSEP samples, the wPCA method and other methods is presented. For ease of reference, Table 1 lists the abbreviations defined herein in alphabetical order.

Materials and Assessment Metrics
A Nikon D5100 camera was taken as an example. Its spectral sensitivities of the red, green and blue signal channels measured by a monochromator are shown in Figure 1a [16]. The average wavelengths of the spectral sensitivities of the red, green and blue channels are denoted as λ CamR , λ CamG and λ CamB , respectively, which are called the channel wavelengths for simplicity. The full width at half maximum (FWHM) of the spectral sensitivities of the red, green and blue channels is denoted as ∆λ CamR , ∆λ CamG and ∆λ CamB , respectively. The spectral specifications of the camera are shown in Table 2.  The reflectance spectra of matt Munsell color chips measured by a Perkin-Elmer lambda 9 spectroradiometer were adopted for preparing reference samples and test samples [23]. The available measurement data in [23] comprise 1269 records, but 2 of them are duplicates, namely, record 1242 (annotation 10RP 7/2) and record 1249 (annotation 10RP 7/4). Therefore, 1268 reflectance spectra were used in this paper. The light source was assumed to be illuminant D65. A total of 202 color chips were selected for the preparation of the reference samples.
A spectrum can be represented by the vector S = [S(λ 1 ), S(λ 2 ), . . . , S(λ Mw ) ] T , where S(λ j ) is the spectral amplitude at wavelength λ j , λ j = λ 1 + (j − 1)∆λ is the j-th sampling wavelength, j = 1, 2, . . . , M w , and ∆λ is the wavelength sampling interval; M w is the number of sampling wavelengths. In this paper, spectra were sampled from 400 nm to 700 nm in steps of 10 nm, i.e., λ 1 = 400 nm, ∆λ = 10 nm and M w = 31. The spectrum vector of the light reflected from a color chip is S Reflection = S Ref • S D65 , where S Ref and S D65 are the spectral reflectance vector of the color chip and the spectrum vector of the illuminant D65, respectively; the operator • is the Hadamard product, also known as the element-wise product. Figure 2a shows the color points of the reflection light from the 1268 Munsell color chips in the CIELAB color space, where the 202 reference samples and 1066 test samples are shown as red and blue dots, respectively. Figure 2b-d are the same as Figure 2a, but with different viewing angles. The CIE 1931 CMFs were adopted in this paper. The measured signal of a color channel is U Meas = S Reflection T S U , where U = R, G and B for the red, green and blue channels, respectively; S Reflection is the reflection spectrum vector; S U is the spectral sensitivity of the channel. For the white balance condition, the channel signals are normalized to U = U Meas /U MeasD65 , where U = R, G and B; U MeasD65 is the measured signal when S Reflection = S White • S D65 ; S White is the spectral reflectance of a white card. The white side of a Kodak gray card was taken as the white card, where its spectral reflectance is approximately 0.9 in the visible wavelength range. The vector representing the camera signals is designated as C = [R, G, B] T . Figure 3a shows the color points of the reflection spectra from the Munsell color chips in the RGB signal space using the Nikon D5100, where the 202 reference samples and 1066 test samples are shown as red and blue dots, respectively. There are 62 samples in the convex hull of the 202 reference samples. Figure 3b shows the convex hull. For a given test signal vector, the LUT method to reconstruct the reflection light spectrum is shown in Section 3.1. The reconstructed spectrum vector is designated as S Rec . The reconstructed spectral reflectance vector S RefRec was calculated as the reflection spectrum vector S Rec divided by the D65 spectrum vector S D65 element by element.
The reconstructed spectral reflectance vector S RefRec was assessed by the root mean where |·| stands for the norm operation. The color difference between S Rec and S Reflection was assessed using CIEDE2000 ∆E 00 . The spectral comparison index (SCI) was also used to assess the reconstructed results [24,25]. The parameter k in the formula for calculating SCI shown in [24] was set to 1.0. For the values of E Ref , ∆E 00 and SCI, the smaller, the better. The statistics of the three metrics were calculated, which are the mean µ, standard deviation σ, 50th percentile PC50, 98th percentile PC98 and maximum MAX. For the value of GFC, the larger, the better. The statistics of GFC were calculated, which are the mean µ, standard deviation σ, 50th percentile PC50 and minimum MIN. The fit of the spectral curve shape is good if GFC > 0.99 [14,26]. The ratio of samples with GFC > 0.99 was calculated, which is called the ratio of good fit and designated as RGF99.

Reflection Spectrum Reconstruction
This subsection describes the LUT method to reconstruct the reflection spectrum vector S Rec from the test signal vector C [12]. The color points of the reference samples in the RGB signal space were not regularly distributed as shown in Figure 3a. Therefore, the use of the scattered data interpolation method was required. Several interpolation methods were surveyed in [27]. Among these methods, linear tetrahedral interpolation was adopted due to its simplicity and computational time savings [11][12][13]. A tetrahedral mesh in the RGB signal space was generated from the reference signal vectors. Note that the tetrahedrization is not unique, and the interpolation result depends on the tetrahedrization [12,27]. All programs in this paper were implemented in MATLAB (version R2021a, MathWorks). The tetrahedral mesh used for interpolation was generated by the MATLAB function "delaunay" [11,13,14]. There were three steps to interpolate the test sample. The tetrahedron that encloses the color point Q of the vector C in the RGB signal space was located. Figure 4 shows the tetrahedron, with vertices Q 1 , Q 2 , Q 3 and Q 4 enclosing the color point Q. A database or look-up table storing the tetrahedral mesh can be used to save processing time in locating the tetrahedron [13]. This paper used the MATLAB function "pointLocation" to locate the tetrahedron, which is a related function of "delaunay". The reference signal vectors of Q 1 , Q 2 , Q 3 and Q 4 were assumed to be C 1 , C 2 , C 3 and C 4 , respectively. It is required that C is the linear combination of the reference signal vectors, and where the coefficients α 1 , α 2 , α 3 and α 4 are weighting factors. Equation (1a) comprises three scalar equations because tristimulus vectors are 3D. Equation (1b) guarantees that Q is inside the tetrahedron if 0 < α 1 , α 2 , α 3 , α 4 < 1. The four coefficients in Equation (1a,b) were solved. STEP 3: Calculate the reconstructed reflection spectrum.
The reconstructed reflection spectrum vector is where S j is the reference spectrum vector corresponding to the vertex Q j for j = 1, 2, 3 and 4. If the reconstructed spectrum has negative values, the value is set to zero. If both sides of Equation (2) are multiplied by the spectral sensitivity function of a signal channel and integrated over the wavelength, we obtain Equation (1a) corresponding to the signal channel. However, the interpolation is an inverse problem. The reconstructed spectrum vector S Rec is one of numerous metameric spectrum vectors corresponding to the test signal vector C. Equation (1a,b) are the four constraints for finding a metameric spectrum vector. The difference between the target spectral reflectance vector S Ref and the reconstructed spectral reflectance vector S RefRec calculated from the metameric spectrum vector S Rec was assessed using the metrics defined in Section 2. Figure 5 shows a flow chart for reconstructing the spectral reflectance vector S RefRec from the test signal vector C. The convex hull of the tetrahedral mesh of the reference signal vectors is denoted as H R . An example of H R is shown in Figure 3b. The convex hull of the tetrahedral mesh of the reference signal vectors and ARS vectors is denoted as H RA . The method for creating ARSs is shown in Section 4. If the test signal vector is inside H R , its reflection spectrum vector S Rec is interpolated from the reference samples using the three-step procedure in Section 3.1. If the test signal vector is outside H R and inside H RA , its reflection spectrum vector S Rec is extrapolated from the expanded reference sample set including the reference samples and ARSs using the three-step procedure in Section 3.1. If the test signal vector is outside H RA , its reflection spectrum vector must be extrapolated using the other method. Therefore, ARSs must be chosen to guarantee that the test signal vectors of interest are inside H RA .

ARS Creation
The ARSs were described in the last paragraph of Section 1. Figure 6 shows a five-step flow chart for creating a set of ARSs. The description below uses the Nikon D5100 as an example. The example color filters were optimized for the Nikon D5100. Appropriate cyan, yellow and magenta filters were selected. They were used to filter reflection light to increase color saturation. Figure 7a shows the spectral transmittance of an example filter set. Given a reference sample of a signal vector [R, G, B] T , its signal vector becomes [R f , G f , B f ] T after filtering. If the filter is cyan, the ratios B f /R f and G f /R f will be larger than B/R and G/R, respectively, and a more saturated sample is created. If the reference sample is magenta (G << B, R), a highly saturated blue sample is created. The issue of color filter selection is discussed further in Section 4.2. The color filters selected in STEP 2 were sequentially mounted on the camera and the spectroradiometer for measurement. For each color filter, the RGB signal values and spectrum of the reflection light from the reflective surfaces selected in STEP 1 were measured. There were 3N + 3 = 189 measured samples using the color filters, called the raw ARSs. In this paper, the RGB signal values and spectra were calculated according to Section 2 for numerical study. Figure 8a shows an example of the raw ARSs in the RGB signal space, where the color filters in Figure 7a are used. In Figure 8a, the raw ARSs from the color chips and the white card are shown as red dots and crosses, respectively. From Figures 3a and 8a, we can see that the 3N raw ARSs from the color chips cannot properly enclose the test samples due to attenuation of the reflection light passing through the filter. The spectra and RGB signal vectors of these raw ARSs were multiplied by an amplification factor γ greater than 1.0. If an RGB signal vector is multiplied by an amplification factor, its color point in the RGB signal space moves away from the black point in the direction from the black point to its original color point. The value of γ could be sample-dependent to reduce extrapolation error. For simplicity, γ = 1.5 was empirically set for all these samples except for the samples corresponding to the color chips of L* = 90 in Figure 2a-d, which have a value of 9 in the Munsell annotation. The exception samples were multiplied by a smaller amplification factor so that they remained within the RGB signal cube, where γ = 1.185. All amplified raw ARSs are shown as blue dots in Figure 8a. The ARSs are the samples in the convex hull of the amplified raw ARSs, the three raw ARSs from the white card, the white ARS and the black ARS. The latter two are shown as green crosses in Figure 8a. The convex hull is denoted as H A and is shown as a blue mesh in Figure 8b. The total number of ARSs in H A is 53. The convex hull of the reference samples H R defined in Section 3.2 is also shown as a red mesh in Figure 8b for comparison. From Figure 8b, H R is completely inside H A . In this case, H A and H RA are the same because the color points of the reference samples are well enclosed by H A . It seems unnecessary to measure 189 samples in STEP 3 as there are only 53 samples in H A . However, before building H A , we do not know which samples will be in H A .

Color Filter Design Method
Only cyan, yellow and magenta filters were selected in STEP 2 and used in STEP 3. Extrapolation error can be further reduced by using more color filters, e.g., using additional red, green and blue filters. This paper limited the number of color filters to three because (1) the use of cyan, yellow and magenta color filters enables all outside samples to be extrapolated for the cases under consideration, and (2) the cost of creating ARSs increases with the number of color filters. The spectral transmittance functions of the considered cyan, yellow and magenta filters are based on a super-Gaussian function. Such color filters can be absorption filters or interference filters [28]. There are stock color filters in various specifications on the market.
The cyan filter is a short-pass optical filter whose spectral transmittance is assumed to be where f C0 is the maximum transmittance; λ S = 400 nm; σ C and a C are parameters determined by the filter edge wavelength λ C and edge width ∆λ C . Figure 9a shows the definitions of λ C and ∆λ C . The edge wavelength λ C is the wavelength of the half-maximum transmittance, i.e., f C (λ C ) = 0.5 f C0 . The edge width ∆λ C is the wavelength interval from 0.1 f Y0 to 0.9 f Y0 . From Equation (3) and the definitions of λ C and ∆λ C , the following equation can be derived.
Given the values of λ C and ∆λ C , the value of a C can be solved from Equation (4) using the MATLAB function "fzero". After the value of a C is solved, the parameter σ C can be easily calculated by The yellow filter is a long-pass color filter whose spectral transmittance is assumed to be where f Y0 is the maximum transmittance; λ L = 700 nm; σ Y and a Y are parameters determined by the filter edge wavelength λ Y and edge width ∆λ Y . Figure 9a also shows the definitions of λ Y and ∆λ Y , which are similar to those of λ C and ∆λ C . The parameters σ Y and a Y can be solved from the same equations as Equations (4) and (5), except that a C , (λ C − λ S ) and ∆λ C in Equation (4) are replaced by a Y , (λ L − λ Y ) and ∆λ Y , respectively, and σ C , a C and (λ C − λ S ) in Equation (5) are replaced by σ Y , a Y and (λ L − λ Y ), respectively. The magenta filter is a notch optical filter whose spectral transmittance is assumed to be where f M0 is the maximum transmittance; λ M is the central wavelength; σ M and a M are parameters determined by the wavelength separation ∆λ Sep and edge width ∆λ M . Figure 9b shows Given the values of ∆λ Sep and ∆λ M , the value of a M can be solved from Equation (8). After the value of a M is solved, the parameter σ M can be easily calculated by The wavelengths λ MS and λ ML were taken as the specifications of the magenta filter, where λ MS = λ M − ∆λ Sep /2 and λ ML = λ M + ∆λ Sep /2.
For simplicity, the maximum transmittance of the filters was set to 0.96, i.e., f Y0 = f C0 = f M0 = 0.96; all edge widths for the three filters were set to 30 nm, i.e., ∆λ Y = ∆λ C = ∆λ M = 30 nm. The four edge wavelengths λ C , λ Y , λ MS and λ ML were optimized for the minimum mean E Ref of the outside samples under the constraints The constraints are empirical, but reasonable. For example, from Equation (10a), the edge wavelength λ C of the cyan filter should lie between the wavelengths of the green and red camera channels so that the spectrum amplitudes at short and medium wavelengths are less attenuated. Half the bandwidth of the spectral sensitivity was used as the tolerance for the upper bound in Equation (10a,b,d).
The optimization process consisted of two steps. The first step was to optimize the four edge wavelengths using the Bayesian optimization function "bayesopt" implemented in MATLAB. The objective function is the mean E Ref of outside samples. Since Bayesian optimization does not use the derivative of the objective function to find the minimum objective value [29], the second step used the MATLAB optimization function "lsqnonlin" to further optimize the four edge wavelengths, where the result of the first step was taken as the initial trial solution. The function "lsqnonlin" was used because the optimization problem is nonlinear. The optimized edge wavelengths λ C , λ Y , λ MS and λ ML are denoted as λ Copt , λ Yopt , λ MSopt and λ MLopt , respectively.
The edge wavelengths of the optimized filters for the Nikon D5100 are shown in Table 3. The spectral transmittance of the optimized color filters is shown in Figure 7a. The convex hulls H A and H RA using the optimized filters are shown as blue meshes in Figures 8b and 10, respectively, though they are the same for the case considered.

Results and Discussion
In this section, in addition to the Nikon D5100, an artificial camera is used as a second camera for comparison. The spectral sensitivities of the artificial camera were assumed to be the CIE 1931 CMFs as shown in Figure 1b. It is called the CMF camera, whose spectral specifications are shown in Table 2. The camera used in the numerical results below is the Nikon D5100 unless otherwise specified. Table 4 shows the assessment metric statistics for the test samples using the LUT method, where the Nikon D5100 and CMF cameras were used. As can be seen from Table 4, out of the 1066 test samples, about 860 samples were inside samples that can be interpolated. The table also shows the extrapolation results for about 200 outside samples, which are discussed in the next subsection. The assessment metric statistics for the inside samples using the two cameras were about the same except for the color difference ∆E 00 . If the spectral sensitivities of a camera are different from the CMFs, the color difference ∆E 00 will be a non-zero value from Equations (1a) and (2). While not zero, most of the inside samples using the Nikon D5100 showed little color difference. The spectrum reconstructions of the test samples using the PCA and wPCA methods are considered for comparison. In the wPCA method, the i-th training sample was multiplied by a weighting factor 1/(∆E i + s), where ∆E i is the color difference between the test sample and the i-th training sample in CIELAB; s is a small-valued constant to avoid division by zero [8]. Weighted training samples were used to derive basis spectra. A camera device model was used to convert RGB signal values into tristimulus values for calculating ∆E i . A third-order root polynomial regression model (RPRM) was employed and trained using the reference samples [30]. The accuracy of the RPRM was slightly higher than that of the polynomial regression model in this case.

Interpolation
The PCA and wPCA methods were used to reconstruct all test samples using the spectral sensitivities of the Nikon D5100 in Figure 1a. Table 5 shows the assessment metric statistics for the test samples using the PCA and wPCA methods, where the inside samples and outside samples were the same as those using the LUT method. The spectrum reconstruction error using the wPCA method was apparently smaller than that using the PCA method, as expected. Comparing Table 4 with Table 5, we can see that the LUT method outperformed the wPCA method except for GFC. Figure 11a,

b show the E Ref and
GFC histograms for the 864 inside samples, respectively, where the cases using the LUT, PCA and wPCA methods are shown.  The computation time required for the LUT method is at least two orders of magnitude faster than that required for reconstruction methods using basis spectra that emphasize the relationship between the test and training samples [13]. In this work, the ratio of the computation time required to use the LUT method and wPCA method was 1:80.2, where samples were reconstructed from their signal vector C to the spectral reflectance vector S RefRec using MATLAB on the Windows 10 platform.

Extrapolation Using the LUT Method Utilizing Optimized ARSs
The color filters optimized as in Section 4.2 were used to create the ARSs in this subsection. The edge wavelengths of the optimized color filters for the Nikon D5100 and CMF cameras are shown in Table 3. The filter spectral transmittance for the Nikon D5100 and CMF cameras is shown in Figure 7a,b, respectively. From Table 4, there were 202 and 203 outside samples for the Nikon D5100 and CMF cameras, respectively. The assessment metric statistics for the outside samples are shown in Table 4. As expected, the mean assessment metrics for the outside samples were worse than those for the inside samples. The assessment metric statistics for the two cameras were about the same except for the color difference ∆E 00 . The assessment metric statistics for all samples are also shown in Table 4.
For the Nikon D5100, there were 98, 79, 22 and 3 outside samples that referenced 1, 2, 3 and 4 ARSs, respectively. Figure 12a-f show the reconstructed spectra S Rec using the LUT method for the 2.5G 7/6, 10P 7/8, 2.5R 4/12, 2.5Y 9/4, 10BG 4/8 and 5PB 4/12 color chips, respectively, where their target spectrum S Reflection and neighboring reference spectra are also shown. The case in Figure 12a is an interpolation example for comparison. The cases in Figure 12b Table 5 shows the assessment metric statistics for the 202 outside samples of the Nikon D5100 using the PCA and wPCA methods. As expected, the spectrum reconstruction error using the wPCA method was apparently smaller than that using the PCA method. Comparing Table 4 with Table 5, we can see that the extrapolation using the LUT method utilizing optimized ARSs outperformed the wPCA method. Note that the ratio of good fit RGF99 was reduced from 0.9803 for the inside samples to 0.7772 for the outside samples when using the wPCA method, i.e., 22.28% of the outside samples had a GFC of less than 0.99. When using the LUT method utilizing optimized ARSs, RGF99 was slightly reduced from 0.9375 for the inside samples to 0.9257 for the outside samples. It was found that when ARSs were included in the training samples of the wPCA method, the extrapolation error did not decrease, but increased further.

PCA and wPCA Methods
Figure 13a-f also show the reconstructed spectral reflectance using the PCA and wPCA methods. The RMS errors E Ref = 0.0135, 0.0246, 0.1142, 0.0352, 0.0393 and 0.0366 for the cases using the PCA method in Figure 13a-f, respectively. The RMS errors E Ref = 0.0095, 0.0221, 0.0794, 0.03, 0.0297 and 0.0392 for the cases using the wPCA method in Figure 13a-f, respectively.

Nearest Tetrahedron 3D Extrapolation Method
By definition, an outside sample cannot be enclosed by any tetrahedron in the tetrahedral mesh of reference samples in the RGB signal space. However, it can be extrapolated from the nearest tetrahedron [13]. The reference samples of the tetrahedron vertices are used to extrapolate the outside sample, using the same method as interpolation, except that the coefficients in Equation (1a,b) are not restricted to be between 0 and 1. The nearest tetrahedron can be located according to its circumcenter, in-center or centroid. For example, if the locating rule is based on the circumcenter, the nearest tetrahedron is the tetrahedron with the shortest Euclidian distance between its circumcenter and the outside sample. Table 6 shows the assessment metric statistics for the 202 outside samples of the Nikon D5100 using the nearest tetrahedron 3D extrapolation method. The methods in the table using the locating rules based on the circumcenter, in-center and centroid are designated as NTCC, NTIC and NTCE, respectively. The results using the LUT method utilizing optimized ARSs and the wPCA method shown in Tables 4 and 5 are also shown in Table 6 for comparison. As can be seen from Table 6, the mean extrapolation error using the NTCC method was much less than that of the NTIC and NTCE methods but was larger than that of the LUT method utilizing optimized ARSs. Table 6. Assessment metric statistics for the 202 outside samples of the Nikon D5100 using the NTCC method, NTIC method, NTCE method, LUT method utilizing MMSEP samples and LUT method utilizing ARSs in the BSLB case (ARS(BSLB)). Also shown are the cases using the LUT method utilizing optimized ARSs (ARS(Opt)) and the wPCA method for comparison.

LUT Method Utilizing MMSEP Samples
The extrapolation using the LUT method utilizing MMSEP samples is considered. As described in Section 1, eight spectral reflectance functions were constructed so that their color points under D65 illumination were as close as possible to the corners of the RGB signal cube. The eight MMSEP samples were included in the reference sample dataset for extrapolation. The white MMSEP sample and black MMSEP sample are the same as the white ARS and black ARS, respectively. The spectral reflectance functions of the other six MMSEPs are based on the sigmoid function with parameters optimized for the minimum objective function defined in [14]. Table 7 shows the optimized RGB signal values of the MMSEP samples for the Nikon D5100. The RGB signal values were not close to their target values, except for the white and black MMSEP samples. Taking the green MMSEP sample as an example, if the value of its G signal is close to 1.0, its R and B signals will not be small in value because the spectral sensitivities overlap, as shown in Figure 1a. The convex hull of the eight MMSEP samples is shown as a red mesh in Figure 10a,b. The convex hull H AR is smaller in size than the MMSEP convex hull but extends more in the yellow and purple regions. The convex hull H AR can be expanded further if red, green and blue filters are used. The LUT method utilizing MMSEP samples is equivalent to the LUT method utilizing only eight ARSs, where six color filters are used and the white card is the only reflective surface. The assessment metric statistics for the 202 outside samples of the Nikon D5100 using the LUT method utilizing MMSEP samples are shown in Table 6. There were 148, 46 and 8 outside samples that referenced 1, 2 and 3 MMSEP samples, respectively. As can be seen from Table 6, using the optimized ARSs improved the assessment metrics compared to using MMSEP samples.

Effect of Filter Edge Wavelengths
The spectral sensitivities of a camera can be measured or estimated as described in Section 1. If the measurements or estimates are accurate, the color filters can be optimized using the same method as in Section 4.2. On the other hand, estimates of sensitivity spectral shapes may not be very accurate, but estimates of channel wavelengths may be accurate enough for color filter design. It is found that the specifications of optimized color filters are related to the channel wavelengths. For the Nikon D5100, from Tables 2 and 3, the optimized edge wavelengths λ Copt , λ Yopt , λ MSopt and λ MLopt are about λ CamR +∆λ CamR /4, (λ CamC + λ CamG )/2, λ CamC and λ CamR , respectively, where λ CamC is an average wavelength and λ CamC = (λ CamB + λ CamG )/2, These approximate relationships are roughly valid for the CMF camera. Since the channel wavelength is the average wavelength of the spectral sensitivity, the specifications of appropriate color filters could be estimated from less accurate estimates of the spectral sensitivities.
In this subsection, the use of color filters that are not optimized is studied. Deviations of filter specifications from their optimized values are expressed as It is found that the appropriate range of edge wavelengths can be roughly estimated as where λ CamY is an average wavelength and For the Nikon D5100, λ CamC = 498.7 nm and λ CamY = 567 nm. The empirical estimates shown in Equation (12a-d) are based on the comparison of Figure 7a,b with Figure 1a,b, respectively, and the tolerance analysis shown below.
The deviation of the edge wavelength from the optimized value results in a change in the convex hull H RA . Since λ Copt > λ CamR , increasing positive δλ C will result in an increase in the R signal with little change in the B and G signals, which will cause the ratios B/R and G/R to decrease. The decrease in the signal saturation results in a smaller convex hull H RA in the cyan region, and some outside samples may not be extrapolated. Therefore, the upper bound in Equation (12a) is set to the optimized edge wavelength of the cyan filter. The upper bounds in Equation (12b-d) are changed for similar reasons. Since λ CamC < λ Yopt < λ CamG , increasing positive δλ Y will result in a greater reduction in the G signal, which will cause the ratio G/B to decrease. Since λ MSopt ≈ λ CamC and λ MLopt ≈ λ CamR , increasing positive δλ MS and δλ ML will result in a greater increase in the G signal and a greater reduction in the R signal, respectively, which will cause the ratios B/G and R/G to decrease.  Tables 8 and 9, respectively, for the cases in Figure 15a-i. In the two tables, the corresponding filter edge wavelength deviations and the ratio RGF99 are also shown.   At the origin in Figure 15a-i, all outside samples can be extrapolated. Away from the origin, the red dotted line represents the boundary where at least one outside sample cannot be extrapolated. Beyond the boundary, the mean E Ref of outside samples that can be extrapolated is shown. The white dotted line in the figure represents the contour of the mean E Ref = 0.0213, which is the value obtained using the wPCA method. Using color filters that meet the specifications within both the red and white dotted lines, all outside samples can be extrapolated while keeping the mean E Ref < 0.0213. For the cases with δλ C = 0 in Figure 15c,f,i, the area enclosed by the red and white dotted lines is small because some cyan outside samples cannot be extrapolated. They can be extrapolated by using a blue-shift cyan filter, i.e., δλ C < 0, but will increase the extrapolation error. When δλ C = −9.0 nm, the red dotted line is about the same as in the cases of δλ C = −15.5 nm in Figure 15b,e,h. Therefore, if a larger edge wavelength tolerance is required, a blue-shift cyan filter is preferred. For such a requirement, the upper bound λ Copt in Equation (12a) can be replaced by λ CamR .
The point of (δλ ML , δλ MS ) = (−41.1 nm, −33 nm) in Figure 15g is the lower bound case in Equation (12a-d), where λ C = λ CamY , λ Y = λ CamC , λ MS = λ CamB and λ ML = λ CamY . Since all filter edge wavelengths are blue-shifted, this case is called the blue-shift lower bound (BSLB) case. In contrast, the upper bound case in Equation (12a-d) is the optimized case at the origin in Figure 15c. The assessment metric statistics of the BSLB case are shown in Table 6, where the mean E Ref = 0.019 and RGF99 = 0.8416. As can be seen from Table 6, the assessment metric statistics of the BSLB case were worse than those of the optimized case, but better than those of the wPCA and NTCC methods. Figure 13b-f also show the reconstructed spectral reflectance of the BSLB case, where the RMS errors E Ref = 0.0187, 0.0469, 0.0203, 0.0274 and 0.0482, respectively. Figure 16a,b show the E Ref and GFC histograms, respectively, for the 202 outside samples of the BSLB case. Also shown are the results of the cases using the LUT method utilizing optimized ARSs and the LUT method utilizing MMSEP samples for comparison. As can be seen from the two figures and Table 6, for the RMS error E Ref , the unoptimized BSLB case was slightly better than the case including MMSEP samples, but for the goodness-of-fit coefficient GFC, the case including MMSEP samples was slightly better. The extrapolation performance of the two cases is comparable. Note that it is easy to design better color filters than the BSLB case for extrapolation.

Conclusions
The reconstruction of spectral reflectance using the LUT method to interpolate camera RGB signals was investigated. Using the LUT method has the advantages of a high accuracy, saving computation time and eliminating the need to use the camera spectral sensitivity functions. The disadvantage of this method is that it cannot interpolate samples outside the convex hull of the reference samples in the RGB signal space. The outside samples can be extrapolated by using the method based on basis spectra, but it has two disadvantages: (1) accurate camera spectral sensitivity functions are required; (2) the calculation of the algorithm with a low spectrum reconstruction error is time-consuming. This paper proposed the LUT method utilizing auxiliary reference samples for extrapolating outside samples. The auxiliary reference samples were created by using reference samples and color filters so that the convex hull of the reference samples and auxiliary reference samples can enclose the outside samples in the RGB signal space. Therefore, outside samples can be extrapolated by using the LUT method utilizing auxiliary reference samples.
The proposed method was tested with Munsell color chips as examples of reflective surfaces. The Nikon D5100 camera was taken as an example camera. The method to create auxiliary reference samples was described. Cyan, yellow and magenta filters were used in this study. The optimized design of the three filters was presented. The results show that the mean extrapolation error using the proposed method was smaller than that of the weighted PCA method. The specifications for the optimized color filters mainly depend on the average wavelengths of the camera spectral sensitivities. The appropriate range of color filter edge wavelengths was shown. The design of color filters may not require accurate measurement or estimation of the camera spectral sensitivities. Therefore, the proposed method is feasible for overcoming the extrapolation problem. It was also shown that the ratio of computation time required to use the LUT method and the wPCA method was 1:80.2.
Since only one commercially available camera was considered, further studies are required for cameras with other sensitivity characteristics. Further research should use more than three color filters to expand the convex hull of the reference samples and auxiliary reference samples, and further reduce the extrapolation error where possible. Studies implementing the proposed method for field application will be published in the future.