Mars-On-Orbit Color Image Spectrum Model and Color Restoration

Abstract

Deep space Color Remote Sensing Images (DCRSIs) are of great significance in reconstructing the three-dimensional appearance of celestial bodies. Among them, deep space color restoration, as a means to ensure the authenticity of deep space image colors, has significant research value. The existing deep space color restoration methods have gradually evolved into a joint restoration mode that integrates color images and spectrometers to overcome the limitations of on-orbit calibration plates; however, there is limited research on theoretical models for this type of method. Therefore, this article begins with the physical process of deep space color imaging, gradually establishes a color imaging spectral model, and proposes a new color restoration method for the color restoration of Mars remote sensing images. The experiment verifies that our proposed method can significantly reduce color deviation, achieving an average of 8.43 CIE DE 2000 color deviation units, a decrease of 2.63 (23.78%) compared to the least squares method. The color deviation decreased by 21.47 (71.81%) compared to before restoration. Hence, our method can improve the accuracy of color restoration of DCRSIs in space orbit.

1. Introduction

Deep space color remote imaging technology is a crucial technological path for humans to explore the geometric structure, surface characteristics, and components beyond the solar system and Earth’s gravitational field [1,2,3]. The color information of a celestial body’s surface, obtained through deep space exploration color imaging, is an important source for studying its geometric structure, surface characteristics, and composition [4,5,6,7]. However, deep space imaging has slight color deviations due to the unique nature of deep space environments, and inconsistencies between the color response characteristics of the camera. Color restoration is the most scientific and practical technical way to solve the color deviation problem [8,9,10].

Different from ground color restoration [11,12,13], deep space color restoration needs to face two special challenges: (1) the lack of atmosphere in the deep space environment makes the ground calibration parameter not suitable, resulting in color distortion, such as the exploration of the moon and Mars; (2) the lack of prior information in the face of unknown environment makes the existing methods based on learning and dark primaries prior not applicable. Therefore, it is a natural idea to directly collect data on-orbit and perceive the characteristics of the deep space environment to recover the actual color.

The earliest color restoration suitable for deep space environments is the image-based method [14,15,16]. To overcome the imaging difference between the deep space environment and the ground, using a color standard plate is a common and reliable choice. The Mars surface “Curiosity” (2011) rover is equipped with three on-orbit color standard plates, and The Mars “Perseverance” (2020) rover also takes more refined on-orbit color standard plates, as shown in Figure 1a,b. In 2020, China’s “Zhurong” Mars rover also carried similar color standard plates, as shown in Figure 1c.

Based on the above equipment, numerous image-based studies have proposed solutions. For example, Zhao Rujin et al. [20,21] designed a two-step color correction method that combines color restoration and white balance, achieving satisfactory results through ground verification, as shown in Figure 2a,b. Ron L. Levin and Gilbert V. Levin utilized an active light source to calibrate the color intensity and applied this technique to the Viking and Pathfinder missions [22], as shown in Figure 2c,d. Shi Z et al. proposed a Color restoration subnetwork and an enhancer subnetwork to correct the image color deviation [23,24]. Then, Liu W et al. proposed that the problem of image color transfer and restoration should be attributed to the problem of reversible information hiding in the image. The internal correlation architecture is proposed [25]. The advantages of this image-based method lie in its ease of implementation and good robustness. However, its shortcomings lie in its dependence on the color fidelity of the color standard board. However, long-term spatial radiation leads to color degradation and distortion, and the board consumes a significant amount of quality resources.

In the absence of a standard color chart, accurately and consistently restoring color is a key issue. Therefore, it is necessary to introduce new information and establish a color restoration model suitable for deep space environments [26,27]. The reference spectroscopy method for achieving color correction involves supplementing the influence of the difference between the illumination spectrum of the extraterrestrial environment and the ground by utilizing spectral information obtained from multispectral cameras or spectrometers. In 2016, Park, Chulhee and Kang, Moon Gi proposed a color restoration method for RGBN multispectral filter array sensor images based on spectral decomposition [28]. In 2020, Wang H et al. proposed a novel color restoration method based on a normalized RGB color model using the Mars Exploration Multi-spectral Camera [29]. Similar methods are applied to full-waveform multispectral LiDAR [30]. The latest research comes from Ren Xin [31], who combined a mineral spectrometer with an RGB camera and applied it to “Tianwen-1” to achieve color correction and balance across Mars [32,33,34], as shown in Figure 3.

This method has significant advantages in color difference evaluation compared to purely image-based methods. However, its drawback is that it often requires a large number of empirical formulas and parameters, and its theoretical research is relatively limited. There is relatively little interaction between deep space color imaging and deep space spectroscopy in the physical model.

In summary, the existing on-orbit color restoration methods are primarily based on on-orbit color calibration plates, with a few methods combining camera and spectral information to achieve color restoration. However, the existing methods for combining spectral information lack an accurate model for the spectral effects during the imaging process. The impact of spectral information on imaging processes is often only qualitatively described, lacking quantitative models. The color restoration technology for deep space exploration in orbit still has difficulties in aerospace technology.

To address the issues of lacking prior color information and image spectrum models, this paper proposes the color image spectrum model to quantitatively describe the color imaging process and establish a mathematical model between deep space environmental reflectance spectra and imaging colors. The beneficial effects can be mainly concluded as follows:

• Unlike models based on three stimulus functions, the color imaging spectral model starts from the radiation energy spectrum of ambient light, A quantitative spectral imaging method has been developed, considering factors such as optical lenses, Bayer filter arrays, and the quantum characteristics of image sensors.
• Unlike subjective indicators based on the three stimulus functions, the color spectrum model proposed in this paper relies only on measurable physical quantities, making it more objective.
• In terms of solving CCM, the least squares method is first used to estimate CCM. Then, the CIE DE 2000 color deviation weight matrix is designed to optimize CCM, effectively reducing color deviation and improving restoration accuracy.

2. Related Works

In the Introduction Section, we have introduced the development process of in-orbit color restoration technology, from methods based on in-orbit calibration plates to methods based on image information and gradually to methods that fuse spectral information. This section mainly discusses the in-orbit color restoration technology that integrates spectral information.

Hong Wang et al. [29] proposed a new color calibration method in brightness-independent chromaticity space to eliminate the influence of light source brightness and relative spectral distribution on color calibration. The core lies in converting complex imaging mechanisms into the product of brightness components and spectral components. Thus, by normalizing the RGB color space to eliminate the differences in brightness components, color calibration can be achieved by simply calculating the 4 × 4 matrix in the normalized RGB space. This method still relies on the color calibration board to obtain the true color value but eliminates the influence of brightness during the experimental process.

Chulhee Park et al. [28] proposed a color restoration method for an imaging system based on the MSFA image sensor with RGBN filters. The proposed color restoration method estimates the spectral intensity in the NIR band and recovers hue and color saturation by decomposing the visible band component and the NIR band component in each RGB color channel. However, this method still requires imaging of the standard color palette to obtain a color reference.

In summary, although existing in-orbit color restoration methods have many innovations in processing color information, they all rely on imaging data of standard color plates, and this dependence is one of the key factors restricting the development of in-orbit color restoration technology.

3. Materials and Methods

3.1. Color Image Spectrum Model

The physical process of color optical imaging can be described as follows: the sunlight reflected by the target object is received by the optical lens of the deep space camera and then transmitted through the Bayer color filter to illuminate the imaging sensor. Finally, the imaging sensor converts light energy into electrical signals through the photoelectric effect, thereby obtaining the target image. The details of the above processes are shown in Figure 4.

Three key physical processes are involved in Figure 4, including optical transmission, photoelectric conversion, and the Bayer color filter process.

3.1.1. Optical Transmission Model

The optical transmission model describes the process by which reflected light from the target passes through the optical lens and illuminates the image sensor.

In the color imaging spectrum model, we are not concerned with the specific location of the target in the image but rather with its grayscale value, which is the output of the color image sensor. Hence, from the perspective of energy transmission, we naturally simplify the complex effect of optical lenses on reflected light and convert it into an optical transmission model, as shown in (1).

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\mathsf{\Phi }}_{lens}\left(\lambda \right)={\displaystyle \frac{{\Omega }_{lens}}{{\Omega }_0}}{\mathsf{\Phi }}_0\left(\lambda \right)m\left(\lambda \right)\end{array}\end{array}$$

(1)

where ${\mathsf{\Phi }}_0\left(\lambda \right)$ is the emission radiance spectrum of the target, ${\mathsf{\Phi }}_{lens}\left(\lambda \right)$ is the incident radiance spectrum of the image sensor, $m\left(\lambda \right)$ is the lens transmittance, and ${\Omega }_{lens}$ and ${\Omega }_0$ are, respectively, the lens solid angle and the target emission solid angle.

In the optical transmission model, we utilize the radiance spectrum (such as ${\mathsf{\Phi }}_0\left(\lambda \right)$) to represent the energy distribution of light and we utilize ${\displaystyle \frac{{\Omega }_{lens}}{{\Omega }_0}}$ to model the lens’ ability to focus and diverge light rays. $m\left(\lambda \right)$ describes the degree of light transmission through the lens material itself. We have overlooked the impact of the lens field of view on energy, which is relatively small.

3.1.2. Photoelectric Conversion Model

The photoelectric conversion model describes the process of converting light irradiated on an image sensor into electrical signals. This model primarily establishes the relationship between the energy distribution of light, image sensor parameters, and image gray-scale values, as shown in (2).

$$\begin{array}{c}\hfill \begin{array}{c}\hfill N_0\left(\lambda \right)={\displaystyle \frac{{\mathsf{\Phi }}_{lens}\left(\lambda \right)t{S}_0}{E_{\lambda }}}\end{array}\end{array}$$

(2)

where $N_0\left(\lambda \right)$ is the number of photons with wavelength $\lambda$, $S_0$ is the target luminous area, t is the integration time, and $E_{\lambda }=hc/\lambda$ is the energy of the photons with wavelength $\lambda$(h is Planck constant, c is the speed of light).

In (2), we convert the distribution of light energy into the distribution of photon numbers because the parameter describing the photoelectric conversion efficiency in image sensor parameters is based on the number of photons. Therefore, the relationship between the gray-scale value $DN$ of the image and the number of photons is shown in (3).

$$\begin{array}{c}\hfill \begin{array}{c}\hfill DN={\int }_{{\lambda }_1}^{{\lambda }_2}{N}_0\left(\lambda \right)\ast QE\left(\lambda \right)\ast G\end{array}\end{array}$$

(3)

where $DN$ is the gray-scale value of the image, and $QE\left(\lambda \right)$ is the quantum efficiency of the image sensor, and G is the conversion factor.

In (3), $QE\left(\lambda \right)$ represents the conversion efficiency of image sensors for light of different wavelengths, while G represents the ability to convert photons into electrons.

Substituting (2) and (3), we can obtain the relationship between the image gray-scale value $DN$ and the incident radiance spectrum of image sensor ${\mathsf{\Phi }}_{lens}\left(\lambda \right)$ (also referred to as light energy distribution), as shown in (4).

$$\begin{array}{c}\hfill \begin{array}{c}\hfill DN={\int }_{{\lambda }_1}^{{\lambda }_2}{\displaystyle \frac{{\mathsf{\Phi }}_{lens}\left(\lambda \right)t{S}_0}{E_{\lambda }}}\ast QE\left(\lambda \right)\ast G\end{array}\end{array}$$

(4)

however, (4) can only depict the process of panchromatic imaging, ignoring the spectral selection effect of the Bayer color filter; therefore, the model still needs further adjustment. ${\lambda }_1$ and ${\lambda }_2$ cover the entire visible light spectrum (380 nm 780 nm), generally.

3.1.3. Bayer Filter Spectrum Model

The Bayer filter model is established to model the spectral selection effect of Bayer color filter arrays. The basic mechanism of the Bayer color filter array is to utilize the filter to have high transmittance for specific wavelengths of light while reflecting or absorbing light of other wavelengths. So, we naturally establish a simple transmittance model (5) to describe this characteristic.

$$\begin{array}{c}\hfill \begin{array}{c}\hfill m_{bayer}\left(\lambda \right)=\{\begin{array}{cc}m\left(\lambda \right)& {\lambda }_{low}<\lambda <{\lambda }_{upper}\\ 0& other\end{array}\end{array}\end{array}$$

(5)

where $m\left(\lambda \right)$ is the lens transmittance (same as (1)), and ${\lambda }_{low}$ and ${\lambda }_{upper}$ are the lower and upper wavelength bounds of the red, yellow, and green ternary colors, respectively.

3.1.4. Brief Summary

Combining (5), (4), and (1), the color image spectrum model fused with the Bayer color filter effect can be obtained, as shown in (6).

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \begin{array}{c}D{N}_r={\int }_{{\lambda }_{r1}}^{{\lambda }_{r2}}{\displaystyle \frac{{\mathsf{\Phi }}_{lens}\left(\lambda \right)t{S}_0}{E_{\lambda }}}\ast QE\left(\lambda \right)\ast G\\ D{N}_g={\int }_{{\lambda }_{g1}}^{{\lambda }_{g2}}{\displaystyle \frac{{\mathsf{\Phi }}_{lens}\left(\lambda \right)t{S}_0}{E_{\lambda }}}\ast QE\left(\lambda \right)\ast G\\ D{N}_b={\int }_{{\lambda }_{b1}}^{{\lambda }_{b2}}{\displaystyle \frac{{\mathsf{\Phi }}_{lens}\left(\lambda \right)t{S}_0}{E_{\lambda }}}\ast QE\left(\lambda \right)\ast G\end{array}\end{array}\end{array}$$

(6)

where $(D{N}_r,D{N}_g,D{N}_b)$ are the color-values output by the image sensor, $({\lambda }_{r1},{\lambda }_{r2})$ is the wavelength range of red light, $({\lambda }_{g1},{\lambda }_{g2})$ is the wavelength range of green light, $({\lambda }_{b1},{\lambda }_{b2})$ is the wavelength range of blue light. For specific image detector manuals, these values can be found. For CMV12000, $({\lambda }_{r1},{\lambda }_{r2})=\left(580\mathrm{nm}800\mathrm{nm}\right),({\lambda }_{g1},{\lambda }_{g2})=\left(450\mathrm{nm}610\mathrm{nm}\right),({\lambda }_{b1},{\lambda }_{b2})=\left(380\mathrm{nm}550\mathrm{nm}\right)$.

3.2. Deep Space Color Restoration Combined with On-Orbit Spectrum

The basic principle of color restoration is to directly establish a mapping relationship between the standard color value and the output of the image sensor. Since the basic elements of color are the three primary colors, this mapping relationship can be modeled as a 3 × 3 matrix (namely the color correction matrix, CCM).

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \begin{array}{c}\hfill \left[\begin{array}{c}{r}_c\\ g_c\\ b_c\end{array}\right]=\left[\begin{array}{ccc}{C}_{11}& C_{12}& C_{13}\\ C_{21}& C_{22}& C_{23}\\ C_{31}& C_{32}& C_{33}\end{array}\right]\left[\begin{array}{c}D{N}_r\\ D{N}_g\\ D{N}_b\end{array}\right]=CCM\left[\begin{array}{c}D{N}_r\\ D{N}_g\\ D{N}_b\end{array}\right]\end{array}\end{array}\end{array}$$

(7)

where $(r_c,g_c,b_c)$ are standard colors, and $(D{N}_r,D{N}_g,D{N}_b)$ are the color values output by the image sensor.

On the ground, the core of color restoration is only the accurate calculation of the CCM method by the known standard color and the color value outputs of the image sensor. After all, standard colors and their output in image sensors are readily available. Specifically, the required data can be obtained by imaging the standard color palettes.

However, in deep space environments and the absence of standard color palettes, color restoration will face a new serious problem: how to obtain the true color value in a deep space environment. Therefore, we propose a deep space color restoration method combined with on-orbit spectral data. The core innovation lies in using on-orbit spectrum and image sensor characteristics to calculate the true color imaging values of deep space environments.

The overall process of the proposed method is shown in Figure 5.

In Figure 5, our proposed method consists of two key parts: (1) deep space color true value calculation module is used to calculate the deep space color true value $(r_c,g_c,b_c)$ based on the environmental spectrum collected by the mineral spectrometer; (2) color deviation iterative optimization algorithm is designed to obtain the optimal CCM though $(r_c,g_c,b_c)$ and the outputs of the image sensor. The details are shown as follows.

3.2.1. Deep Space True Color Calculation Module

Based on the details of the color image spectrum model in Section 3.1, we can see that the key factors determining color differences are quantum efficiency $QE\left(\lambda \right)$, emission radiance spectrum of target ${\mathsf{\Phi }}_0\left(\lambda \right)$, and lens transmittance $m\left(\lambda \right)$. They represent the selective effects of image sensors, deep space environments, and optical systems on spectra.

Considering that in actual camera design, the lens transmittance curve can be measured and adjusted, the model simplifies $m\left(\lambda \right)$ to a fixed value (usually 0.7) at the transparent wavelength $\lambda$. By merging the wavelength-independent terms in (6), the final (8) can be obtained as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \begin{array}{c}D{N}_r={\int }_{600\mathrm{nm}}^{800\mathrm{nm}}{\mathsf{\Phi }}_{lens}\left(\lambda \right)\ast QE\left(\lambda \right)\ast K\\ \\ D{N}_g={\int }_{450\mathrm{nm}}^{650\mathrm{nm}}{\mathsf{\Phi }}_{lens}\left(\lambda \right)\ast QE\left(\lambda \right)\ast K\\ \\ D{N}_b={\int }_{340\mathrm{nm}}^{540\mathrm{nm}}{\mathsf{\Phi }}_{lens}\left(\lambda \right)\ast QE\left(\lambda \right)\ast K\end{array}\end{array}\end{array}$$

(8)

where the red $D{N}_r$ corresponds to $600\mathrm{nm}\sim 800\mathrm{nm}$, the green $D{N}_g$ corresponds to $450\mathrm{nm}\sim 650\mathrm{nm}$, and the blue $D{N}_b$ corresponds to $340\mathrm{nm}\sim 540\mathrm{nm}$. The specific wavelength range can be adjusted appropriately according to the quantum efficiency curve of the image sensor.

For the quantum efficiency $QE\left(\lambda \right)$, we usually obtain it by experiment. Specifically, we scan the spectral response of the four channels $(R,G_1,G_2)$, and B of the color image sensor through the integrating sphere and record the integrating sphere irradiance, sensor output $DN$, and dark field gray-scale value for each channel at each wavelength. Taking the “Tianwen-1” medium-field camera as an example, the quantum efficiency is shown in Section 4.2.

For the emission radiance spectrum of target ${\mathsf{\Phi }}_0\left(\lambda \right)$, we can directly collect it on-orbit through the mineral spectrum analyzer. The details are described in detail in Section 4.2.

3.2.2. Color Deviation Iterative Optimization Algorithm

Due to the lack of atmospheric scattering in the deep space exploration environment, there are no light areas and deep space backgrounds in the imaging area (i.e., the imaging sensor output only contains background noise and no effective signal). Therefore, we set a weight matrix W to remove these special pixels. Then, we designed a color deviation iterative optimization algorithm to improve the restoration accuracy further and obtain the optimal CCM, as shown in Figure 6.

The true color of the image should be infinitely close to the response color of the “standard human eye”. Therefore, the obtained N sets of corrected chromaticity value data $C_m$ are fitted with the corresponding standard chromaticity value data $C_c$. According to the traditional ground color restoration method, the least squares method is used to obtain the CCM preliminarily, and the color correction matrix CCM is calculated in the sRGB color space:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill CCM={\left(C_m^T{C}_m\right)}^{-1}{C}_m^T{C}_C\end{array}\end{array}$$

(9)

To ensure consistency between the color deviation calculation results and visual evaluation, we chose the CIE DE 2000 color deviation formula [35] as the color evaluation model. Apply the preliminarily obtained CCM to the sample to be calibrated and use the CIE DE 2000 color deviation formula to calculate the calibrated color deviation.

To further optimize CCM, we normalize the obtained color deviation as a weight matrix and use the following formula to re-optimize CCM until satisfactory color deviation results are obtained, obtaining the optimal CCM:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill CCM={(C_m^T\ast \omega \ast C_m)}^{-1}{C}_m^T\ast \omega \ast C_C\end{array}\end{array}$$

(10)

Finally, apply the obtained optimal CCM to the entire image I as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill I^{\prime }=I\ast CCM\end{array}\end{array}$$

(11)

This reverses the true color of the image, $I^{\prime }$ refers to the image that meets the standard color response of the human eye after correction.

4. Results

4.1. Evaluation Index

We used the CIE DE 2000 color deviation formula as an evaluation indicator. The formula for color deviation is an effective computational method for quantitatively describing color deviation, ensuring the visual consistency of color deviation evaluation.

It corrects for the non-uniformity of the LAB color space and is the latest color deviation calculation method proposed by CIE. This formula has accurate prediction performance in all current international visual experimental data tests, both new and old.

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \Delta E_{00}=\sqrt{{\left({\displaystyle \frac{\Delta L^{\prime }}{K_L{S}_L}}\right)}^2+{\left({\displaystyle \frac{\Delta C_{ab}^{\prime }}{K_C{S}_C}}\right)}^2+\left({\displaystyle \frac{\Delta H_{ab}^{\prime }}{K_H{S}_H}}\right)+R_T\left({\displaystyle \frac{\Delta C^{\prime }}{K_C{S}_C}}\right)\left({\displaystyle \frac{\Delta H_{ab}^{\prime }}{K_H{S}_H}}\right)}\end{array}\end{array}$$

(12)

where $\Delta L^{\prime }$ is the difference in luminosity, and $\Delta C_{ab}^{\prime }$ is the difference in chroma, $\Delta H_{ab}^{\prime }$ represents hue difference, and $K_C,K_L,K_H$ parameter weighting factors, and $S_L,S_C,S_H$ weight coefficients. The specific calculation method is described in [36].

4.2. Principal Verification

The principle validation is to verify the correctness of the color imaging spectral model. Specifically, the imaging color of the target is calculated based on the known quantum efficiency curve and target reflection spectrum, and the color theory value is compared with the model output to determine the correctness of the principle. The theoretical value of the target reflection spectrum is based on the reflection spectrum data provided by the standard color plate, while the quantum efficiency curve requires experimental measurement. The measurement method and results are given below.

The quantum efficiency of the deep space exploration camera test platform is shown in Figure 7a. In the quantum efficiency test, we ran the experiment with four steps as follows:

• Placing the camera in front of the integrating sphere.
• Covering the camera lens with a file.
• Setting the appropriate exposure time.
• Scanning the fixed visible light band with the integrating sphere and filter to test the output response of the four channels.

The test wavelength range is 350 nm to 850 nm, with 11 wavelength points at an interval of 50 nm. The integral sphere irradiation brightness, the optical response grayscale value of each pixel of the camera, and the dark field grayscale value in each wavelength band should be recorded.

The quantum efficiency values of the R, G1, G2, and B channels of the camera’s visible light band obtained through the test are shown in

Table 1. Quantum efficiency of each color channel.

$\mathit{\lambda }$	Quantum Efficiency R%	Quantum Efficiency ${\mathit{G}}_1$%	Quantum Efficiency ${\mathit{G}}_2$%	Quantum Efficiency B%
350	0.56	1.5	1.5	1.41
400	0.38	0.61	0.62	1.34
450	2.03	12.38	12.55	61.34
500	4.9	63.38	63.56	50.82
550	9.27	91.61	91.15	14.81
600	68.87	28.56	27.96	7.97
650	81.96	15.76	15.47	9.65
700	62.82	28.93	28.37	13.95
750	62.09	38.16	37.44	14.24
800	0.54	0.47	0.46	0.39
850	0.02	0.01	0.01	0.01

. The range is from 0 to 100%, and G1, G2 represent two different green channels in the Bayer color filter array.

Figure 7b shows the quantum efficiency curve obtained from the experiment, the curve function of quantum efficiency of each RGB channel R($\lambda$), B($\lambda$), G($\lambda$). Among them, G($\lambda$) is calculated from the average of G1($\lambda$) and G2($\lambda$). The curves of the three channels show significant differences from the “standard human eye” CIE 1931 color matching function, which can lead to significant color deviation in images from deep space environments, further underscoring the necessity of on-orbit color restoration.

To verify the accuracy of the theoretical calculation model, we selected three color blocks that closely match the color of Mars. We compared the theoretical calculation of color display with the results of camera shooting to validate the rationality of our proposed color imaging spectral model in ground experiments.

In Figure 8, the chromaticity values of each color block are as follows: (a1) [88,62,27]; (b1) [88,60,21]; (a2) [124,83,39]; (b2) [123,82,28]; (a3) [189,135,61]; (b3) [190,128,49]. Among them, (a1), (a2), and (a3) are the theoretical color values of the color block itself. And (b1), (b2), and (b3) are color values calculated based on the quantum efficiency curve in Table 1 and color block reflection radiation spectrum [35]. The chromaticity values calculated by the model are very close to those captured, demonstrating the model’s accuracy and enabling it to verify the correspondence between pixels and spectra.

4.3. Ground Verification Test

4.3.1. Experimental Design

To verify the correctness of the proposed method, we conducted simulation experiments and ground tests. The experimental platform is shown in Figure 9. The experimental setup consists of a 24-color standard color scale, “Color-Checker CLASSIC” [37] a D65 standard light source lightbox [38], a camera, and a spectro-radiometer. The above devices were used for ground verification of on-orbit joint color restoration.

Place the Color-Checker under a standard D65 light source as the standard, acquiring image data of each color code block and obtaining chromaticity values in the camera system’s sRGB color space.

The manufacturer determines the standard values of color patches in traditional ground restoration methods and may not be applicable in different scenarios. This paper utilizes a spectro-radiometer to obtain the reflection spectra of color patches when irradiated at a 45-degree angle with a standard D65 light source, within 2-degrees tolerance of each color patch.

4.3.2. Test Results

Through ground verification tests, we obtained the color response of each color block. Figure 10 presents a comparison of results obtained before and after restoration using the Color-Checker method. The sub-figures a, b, c, and d correspond to the different scenarios: the “standard human eye”, pre-restoration [20], and post-restoration using the least squares method [21], and the result after obtaining the optimal CCM effect through this method, respectively. Before restoration, the response of each color block differed significantly from that of the human eye, exhibiting a yellowish and brighter overall color tone.

The comparison methods include the “standard human eye” method, the pre-restoration method, and the post-restoration, which uses the least squares method. The improvement achieved by traditional ground restoration methods is not significant; however, the overall effect obtained through the CCM method in this paper has been dramatically improved, especially for the 1–18 color patches, namely the color blocks in the first three rows of Figure 11. The results are already very close to human visual perception.

Table 2. Comparison of CIE DE 2000 color deviation results.

	Pre-Restoration	East Square Method	First, Iteration	Our Method
Swatch 1	6.46	5.78	8.23	2.73
Swatch 2	4.76	2.93	6.54	1.68
Swatch 3	5.18	3.15	2.77	2.96
Swatch 4	9.54	9.37	7.67	4.74
Swatch 5	16.12	5.55	2.03	1.86
Swatch 6	10.83	7.21	6.85	6.73
Swatch 7	2.36	3.05	7.7	5.09
Swatch 8	14.47	6.69	5.91	2.34
Swatch 9	11.27	7.03	4.67	2.85
Swatch 10	38.35	37.93	36.21	23.56
Swatch 11	2.91	2.3	3.35	3.74
Swatch 12	7.88	8.79	7.18	7.25
Swatch 13	14.25	11.14	9.87	7.71
Swatch 14	17.79	13.44	13.22	12.45
Swatch 15	10.77	9.26	8.3	7.01
Swatch 16	10.64	8.94	6.76	5.13
Swatch 17	11.27	8.89	6.48	3.55
Swatch 18	7.62	1.15	1.96	6.02
Swatch 19	39.88	40.14	33.53	33.23
Swatch 20	38.34	38.69	32.76	32.54
Swatch 21	38.4	38.73	29.93	25.32
Swatch 22	34.74	35.11	22.15	30.35
Swatch 23	28.47	28.97	10.1	21.38
Swatch 24	28.28	28.96	20.2	22.36
Average	17.11	15.13	12.28	11.36

presents the results of the color deviation calculations. Figure 11 provides a comparison chart of the color deviation results for the colorcheker. After restoration, the color deviation of most color patches has been reduced. Using the color deviation iteration method, after one iteration, the color deviation can be improved by 2 CIE DE 2000 color deviation units. After applying the optimal CCM, the color deviation was reduced by 11.32 units, with a decrease of four units (26.14%) compared to the CIE DE 2000 color deviation unit using the least squares method. The system’s resolution increased by six units of CIE DE 2000 (35.61%) compared to the original system.

4.4. On-Orbit Verification and Results

According to our proposed method flow described in Section 3, we used measured data on the Mars exploration “Tianwen 1” for on-orbit color restoration, and compared it with traditional ground color restoration methods. Figure 12 shows a set of spectra obtained by the mineral spectrometer on-orbit, corresponding to the on-orbit Mars remote sensing image shown in Figure 13.

In Figure 14, each row of images is the result of using the same correction method for different images, and each column represents the results of using different correction methods for the same image. Figure 14(a1–d1) are the pre-restoration results; Figure 14(a2–d2) are the traditional ground restoration results; Figure 14(a3–d3) are our proposed results. Intuitively, our proposed method is more balanced overall, without any yellowing or redness.

Next, we quantitatively analyzed and compared the performance of the comparison methods.

Table 3. Comparison of the color restoration effects in different regions.

Mars Area	Pre-Restoration	Traditional Ground Method	Our Proposed Method
South Crater	30.17	11.08	8.41
Malea Planum	29.91	11.41	8.57
Huygens Crater	28.61	10.45	8.22
Arabia Terra	30.91	11.3	8.54
Average color deviation	29.9	11.06	8.43

shows the comparison results of the on-orbit restoration color deviation.

Due to the complexity of on-orbit environmental lighting, the recovered images taken on-orbit exhibit a larger color deviation than those on the ground. However, after being corrected by our proposed method, the average color deviation can reach 8.43. Compared to the least squares method, our method has improved by 2.63 (23.78%). Compared to the pre-restoration method, our method has an improvement of 21.47 (71.81%). It has been confirmed that the calculation method based on real-time data on-orbit in this paper exhibits good adaptability on-orbit. Compared to ground restoration methods, our proposed method can more effectively reduce color deviations.

5. Discussion

The objective of our research article is to tackle the challenge of obtaining accurate color values in deep space environments without the use of in-orbit calibration plates. Our goal is to achieve effective color recovery for color cameras operating in deep space environments. We started with the principal analysis, then developed a color imaging spectrum model, and ultimately demonstrate the validity and accuracy of our proposed method through principal verification, ground validation, and in-orbit testing.

1. Principal analysis

From the physical mechanism of color imaging, the spectrum perceived by the image sensor determines the color of the image. Therefore, it is sensible to recover the color information of targets in deep space environments using spectral data. The spectral information perceived by image sensors is determined by two main factors: the target’s reflection spectrum and the camera’s quantum efficiency. The reflection spectrum can be measured using a spectrometer, while the quantum efficiency can be assessed with ground testing equipment. Therefore, it is feasible to restore the true color values in deep space environments through spectral information.

2. Modeling

The process of model building is the physical process of color imaging. We first list the transmission paths of light in order, starting from the light source and ending at the point where the image sensor receives it. Then, we analyze each stage step by step, identify the stages that change the spectral information, and model them in detail. For links unrelated to spectral information, simplification was carried out, resulting in an optical transmission model, a photoelectric conversion model, and a Bayer filter model.

3. Verification

To verify the correctness of our proposed model, we conducted a comprehensive discussion in three stages: principle verification, ground verification, and in-orbit verification.

(3.1) The principal verification part is shown in Section 4.2. The purpose of this verification is to validate the principle correctness of the model by measuring the camera quantum efficiency curve and the theoretical reflection spectrum of color. The first row in Figure 8 displays the theoretical imaging values of three color blocks, while the second row shows the calculated values obtained from the model. Both numerical and subjective differences are small. The model proposed in this article is theoretically correct.

(3.2) Ground verification and in-orbit verification are described in Section 4.3 and Section 4.4, respectively. Two experiments were conducted to verify the applicability of our method in both ground and deep space environments. The average color reduction achieved by our method in the ground validation was 11.36. During in-orbit verification, the average color difference was reduced to 8.43, which is superior to existing methods.

6. Conclusions

Our contribution can be summarized in two parts:

• Proposed the color image spectrum model to quantitatively describe the color imaging process in deep space environments, solving the problem of how to obtain true color values with the lacking of a color palettes, and providing a theoretical basis for color recovery of color cameras by fusing spectral information.
• Proposed a method for on-orbit joint color restoration of color cameras for deep space exploration, which aims to solve the problem of color bias in remote sensing images caused by the complex operating environment, poor lighting conditions, and the unavailability of color restoration boards in on-orbit conditions.

This method uses spectral data provided by the on-orbit payload spectrometer to calibrate on-orbit colors. In terms of data fitting, the least squares method and the color deviation iteration method are used to solve the optimal CCM. After the ground and on-orbit testing, this method can significantly reduce color deviations, with an average color deviation of 8.43, which is 2.63 (23.78%) lower than that achieved by the least square method; which makes the camera observation effect more consistent with the visual perception effect of the “standard human eye.” It improved the accuracy and on-orbit adaptability of the color restoration of DCRSIs.