1. Introduction
Light detection and ranging (LiDAR) system has been known as one of the most effective survey tools to characterize vegetation structure for the past decades because of its excellent ranging accuracy and canopy penetration capability [
1,
2]. The emerging full-waveform LiDAR (FWL) system can record the entire backscatter waveform echo, providing an opportunity for researchers to explore the waveform data further, thereby obtaining the physical properties of the target with increased accuracy and reliability [
2,
3]. The shape features (e.g., central location, pulse width and amplitude) extracted from waveform data [
1,
4] are greatly helpful in various applications, such as land cover classification [
5], building extraction [
6,
7], canopy height retrieval [
8] and biomass estimation [
9].
Reflective and texture features of the target surface are also required to meet the application needs as mentioned above. However, the current full-waveform data at a single wavelength can mainly retrieve the geometric attributes of targets along the light path by detecting waveform components, resulting in the lack of spectral or color attribute information. The existing methods mainly collect the texture and spectral data by the fusion of passive image data such as multispectral image [
10]. Since the passive imaging sensors rely on solar illumination, certain inevitable errors are caused by geometric registration between the active and passive datasets. Although many studies have focused on solving the geometric registration problems and achieved high registration accuracy [
11,
12,
13], considerable time and labor are still needed to achieve a satisfactory registration effect.
In recent years, the development of multispectral LiDAR (MSL) or even hyperspectral LiDAR (HSL) systems [
14,
15,
16,
17,
18] have effectively increased the receiving channels of full waveform data for interest wavelengths. These new MSL/HSL systems often utilize a supercontinuum laser source which produces broadband light almost covering the whole visible light band, including red (R), green (G) and blue (B) bands [
19]. Thereby, the simultaneous acquisition of texture and spatial information becomes possible and makes passive imaging data no longer necessary. In terms of obtaining color with LiDAR data directly, Bretagne et al. [
20] attempted to interpret intensity as a green level in RGB or sRGB color spaces under standard illuminant of International Commission on Illumination (CIE) by using a commercial single wavelength LiDAR device at 532 nm. Although the statue faces appear clear and homogeneous with the G level, the color information is still incomplete to visualize the real scene. Conversely, since the multispectral full-waveform data could cover the whole RGB bands, it has great potential in restoring the complete color information directly without fusion of any passive images.
For the multispectral full-waveform data, it reflects the interaction between all targets along the light path at multiple receiving channels, mixed with a variety of texture and spatial information in the waveforms. To date, such types of data have not been tested to extract color information. Consequently, the overall goal of this paper is to develop a color restoration method for the full-waveform data at RGB bands in order to extract the color information directly and efficiently, thus avoiding the adverse effects of registration and solar illumination. The objectives of this study are to: (1) propose a theoretical model for color restoration for the multispectral full-waveform data at RGB bands; (2) perform the color restoration method with measured multispectral waveform data to verify the feasibility and efficiency of the new method.
2. FWMSL System
In the past few years, we have been committed to the development of MSL. We used four solid-state lasers at 556, 670, 700 and 780 nm with full-waveform recording of the receiving channels for monitoring the growth and nutritional status of vegetation. The previous MSL was effective in capturing abundant biochemical information of vegetation [
15] and had increased waveform decomposition accuracy [
14]. Given its significant improvement in vegetation detection, we found that directly synthesizing the true color of targets based on these selected laser bands remains difficult due to the lack of a B band laser source, which results in low visualization and limits the capability to render the true surface color of the target. Although the B band is insensitive to vegetation, it is necessary for restoring the B value in the RGB color space. Therefore, we further updated our MSL with a supercontinuum laser (NKT Photonics, SuperK) source, which covers a wide range of wavelengths, including almost the entire visible band (400–700 nm), instead of the synthesis of multiple laser sources. The laser has a repetition rate of 20 kHz with average power of 100 mW.
We have analyzed the results of true color synthesis through different wavelength combinations for an HSL system and found that the most suitable RGB wavelength combination is 466, 546 and 626 nm [
21]. Meanwhile, on the basis of the principle of supercontinuum laser, the broad-spectrum laser is always pumped with the seed laser at a fundamental frequency of 1064 nm, which is far from the visible band, resulting in a relatively low output energy in the short band but the energy increases sharply from around 450 nm (
Figure 1a). Therefore, we have selected three wide bands near the corresponding wavelengths, especially for the blue band which is relatively wide, in order to increase the energy of the RGB channels to obtain a higher signal-to-noise ratio (SNR). Considering the spectral energy distribution of the supercontinuum laser source and CIE 1931 color space chromaticity (
Figure 1b), the RGB bands, namely, 434.5–474.5, 517–537 and 612–644 nm, in the visible spectral portion were selected for the receiving channels of color information. In addition, the photo multiplier tube (PMT) selected as the detector should be more suitable for detecting laser echoes in visible bands than the avalanche photodiode especially in the relatively low-energy B band since PMT has higher quantum efficiency and gain. Moreover, both the transmitted pulse and receiving echo at RGB bands are full-waveform recorded so that the receiving echoes can be calibrated with the transmitted pulses to eliminate errors caused by laser output energy instability. The waveforms of the three receiving channels were digitized in a 12-bit analog to digital converter (SP Devices, ADQ412), with a 1.8 GHz sampling rate. The length of each active signal is set to 20 samples based on the laser pulse width of 2 ns and sampling rate, thus greatly reducing the amount of data.
4. Results
To investigate the feasibility and accuracy of the color restoration that extracts color information from multispectral full-waveform data, the experimental datasets were utilized as described in
Section 3.2. Since the waveform data measured by the FWMSL system is slightly asymmetrical with trailing, the measured data was fitted using both the proposed multispectral lognormal function and the Gaussian function [
14] in order to verify whether the former can improve the color restoration accuracy compared to the latter. In addition, the intensity can be expressed as amplitude or integral area as mentioned in
Section 3.1.1. This study aims to find the most suitable fitting method and intensity expression form to achieve the best effect of color restoration by comparing the four types of intensity, Gaussian amplitude, lognormal amplitude, Gaussian area and lognormal area. At the same time, this study also explores the optimal number of pulse accumulation in order to attenuate random errors and increase SNR of waveform data, thus improving the accuracy of color restoration finally.
We first test the FWMSL system under five different ambient light conditions including no light environment and four levels of light (Light 1, Light 2, Light 3 and Light 4) from weak to strong provided by the flashlight in order to validate the robustness of the color restoration method. 1000 echoes per channel per color square are recorded under the five lighting conditions. The probability density distributions of the standard deviation of background noise in RGB channels under different lighting conditions are shown in
Figure 6.
This histogram shows that the probability density distributions of the background noise of the G and B channels are more significantly affected by lighting conditions than the R channel. Regardless of whether ambient light is provided or not, the standard deviations of the background noise of the G and B channels are concentrated in the range of 2 to 2.5, which indicates that the influence of lighting conditions on the background noise is acceptable.
After analyzing the effects of ambient light, we further use the fixed point experimental data to determine the best type of intensity as well as the optimal number of pulse accumulation in order to avoid the impact of scanning on color restoration. The color restoration results in terms of RSD (relative standard deviation) at RGB channels by using the four types of intensity are displayed in
Figure 7, which represents the stability of the retrieved color. Each sub-figure consists of color restoration results that vary with the number of accumulated pulses shown as different colored lines, which represent 24 colored squares of the color checker. In general,
Figure 7 shows that the RSDs of color restoration results decrease with the number of pulse accumulation and gradually become stable.
Comparing the subgraphs of each column in
Figure 7, we find that the retrieved R values exhibit a higher stability (RSD = 0 ~ 0.1) (
Figure 7a1–a4) than the retrieved B values (RSD = 0 ~ 0.5) (
Figure 7c1–c4). This is attributed to the uneven distribution of the laser emission power (
Figure 1a) as mentioned in
Section 2, which results in a relatively low SNR for the B channel. Meanwhile, the RSD at R channel is reduced by 20% (
Figure 7a3,a4) and the RSD at B channel is decreased from 0.2 to 0.16 (
Figure 7c3,c4) using the lognormal function when number of pulse accumulation is 5. This result indicates that lognormal function is more suitable than Gaussian for the measured waveform data, offering higher color restoration stability and reliability.
Furthermore, in order to evaluate the color restoration accuracy, the correlation coefficient R
2 between the retrieved RGB values and the corresponding true values are presented in
Figure 8. This figure depicts a comparison among the color restoration results using the four types of intensity at each channel. Generally, R
2 increases with the number of pulse accumulates and tends to stabilize until the accumulation number reaches 5.
Figure 8 exhibits that R
2 is also affected by the SNR of each channel, so that the R channel performs higher accuracy than the B channel. Moreover, the results in RGB channels all evidence that the lognormal area has greater performance than the other three types of intensity according to the relatively high R
2 about 0.9. Particularly,
Figure 8 c displays that intensity expressed as integral area is significantly better than in amplitude, improving R
2 from 0.6 to 0.9. Although the advantage of using the lognormal function is not obvious due to the small tail of the measured waveform,
Figure 8c still shows that it improves the accuracy of B channel compared to Gaussian. Both
Figure 7 and
Figure 8 illustrate that the color restoration accuracy and the scanning efficiency can reach an optimal balance when the number of the pulse accumulation is 5. Additionally, in order to better compare the difference of the results using lognormal area with using the other intensity types, the corresponding results in
Figure 7 and
Figure 8 with 5 as the number of accumulated pulses are listed in
Table 1 to display the color restoration performance in more detail.
Table 1 displays color restoration results in terms of RSD and R
2 at RGB channels with 5 as the number of accumulated pulses. The RSD values indicate that the lognormal area (R channel: 0.0304; B channel: 0.1596) has greater performance than the other three types of intensity (R channel: 0.0428, 0.0387, 0.0379; B channel: 0.2505, 0.1686, 0.1936), especially in R and B channels. Additionally, the R
2 values shows that the color restoration accuracy in B channel can be greatly improved by using lognormal area (0.8865) compared with the others (0.5663, 0.5830 and 0.8706).
After evaluating the RGB values separately, the accuracy of the composite color is then assessed, which must meet the requirements of human eye recognition. Therefore, the term
that represents the color difference is introduced to compare the retrieved color with the corresponding true color. The
of 1.0 is the smallest color difference the human eye can distinguish. The
of color restoration results using the four types of intensity with the pulse accumulation number of 5 are depicted in
Figure 9.
Figure 9 illustrates that the difference between the retrieved and true color of most squares are imperceptible when using lognormal area, while the results of other intensity types exceed the tolerance range of the human eyes. In order to visually observe the color difference, the retrieved colors using lognormal area with 5-pulse accumulation are presented in the CIE Luv space, as shown in
Figure 10. The figure has 24 sub-figures in total. In each of them, the position of the center circle and the black scatters represent the true and retrieved colors, respectively. The sub-figures’ sizes are the same at 0.2 × 0.2. Thus, the color difference can be directly seen from the distance between the center and scatters as well as the background color.
Figure 10 displays that most of the color squares resemble their true colors based on the background color of the scatters and center point. Particularly, in the 8th, 10th and 15th sub-figures, despite the relatively far distance between the center and scatters, the background color looks similar. The previous results evidence the effectiveness of the color restoration method to extract color information from multispectral full-waveform data by using lognormal area with the pulse accumulation number of 5.
To further verify the color restoration performance of the proposed method on the scanned scene, the results using Gaussian and lognormal area with 5-pulse accumulation are both presented in
Figure 11. Comparing with the point cloud colored by Gaussian area (
Figure 11b), the color obtained with lognormal area (
Figure 11c) visually overall looks much more close to the color checker (
Figure 11a). Especially the color displayed in 2th, 4th, 7th, 9th, 12th and 16th squares have obvious improvement with lognormal area. Additionally, we can find that the 7th, 9th which have similar color display in
Figure 11b, can be distinguished clearly in
Figure 11c. The scanning results indicate the advantage in retrieving abundant colors for the asymmetric waveform.
After previous qualitative discussion, the colored point clouds are quantitatively compared with the actual color checker according to the RGB values and RSD. A total of 1000 points are randomly taken from each square to calculate the mean and RSD of the retrieved color values using Gaussian area (GA) and lognormal area (LA), as listed in
Table 2.
Table 2 illustrates the improvement in color information obtained by using lognormal area. Specifically, the retrieved RGB values of the 1th, 3th and 7th squares using lognormal area (R/G/B: 1th 137/106/88, 3th 129/148/166, 7th 214/113/65) are closer to the true values (R/G/B: 1th 115/82/68, 3th 98/122/157, 7th 214/126/44) compared with that using Gaussian area (R/G/B: 1th 68/88/110, 3th 98/122/157, 7th 180/54/21).
Table 2 also exhibits that the measured data has considerable higher accuracy in R channel than the other two channels according to RSD, which is related to the stability of color restoration. The RSD of the retrieved R values ranges from 0.83% to 3.61% while that of G and B values are over 10% in some squares. This metric also shows higher stability of the proposed method using lognormal area than Gaussian in color restoration. Additionally, the accuracy seems to be also related to the RGB values themselves, especially for the B channel. The accuracy of B channel is significant improved in the 19th square (B value: 242; RSD: GA 0.89, LA 0.75) compared with the 24th square ((B value: 52; RSD: GA 12.89, LA 11.37).
5. Discussion
This study proposed a theoretical model for color restoration using FWMSL measurements and verified the feasibility of the proposed method by utilizing the multispectral full-waveform data at RGB bands. This study is the first to establish a transformation model from high-sampling-rate multispectral waveform data to RGB values, thereby achieving the goal of obtaining color information only through active remote sensing without a camera. Previous studies have obtained the color values (R, G and B) by means of passive images [
11,
12,
13]. Although these studies have achieved relatively high accuracy, geometric registration errors during active and passive fusion cannot be avoided, which needs considerable time and labor to compensate. Moreover, the image quality is affected by ambient light to a great extent, which will directly reduce the accuracy of the final color restoration results.
In this study, three main factors may limit the retrieval accuracy of the color information for FWMSL. First, the shape of the echo waveform is usually regarded as Gaussian, while data recorded by the FWMSL system is slightly asymmetric. Given that the measured waveform has a slight tailing, we assumed that the echo waveform conforms to the lognormal distribution and further proposed a multispectral lognormal function considering the spatial consistency between bands. In addition, for other multispectral LiDAR systems, if the shape of the measured waveform is different from that in this study, another suitable waveform fitting model should be selected accordingly to obtain accurate position and intensity information.
Secondly, the supercontinuum laser has extremely low output energy before 450 nm, near the blue band, which will cause color distortion to a certain extent. If the spectral information in the blue light region can be obtained through spectral simulation [
21], the accuracy of color restoration will be improved.
Thirdly, this study analyzed the accuracy of color restoration under different pulse accumulations and provided the optimal number of pulse accumulation for color restoration. However, in practice, considering the scanning efficiency, the accuracy of color restoration is sometimes not optimal.
Further experiments involving extensive types of targets should be conducted with the FWMSL system to validate the performance of the system in acquiring the color information of different materials. The proposed theoretical model for color restoration should also be extended to other FWMSL systems and try to employ different waveform fitting models, such as Gaussian, generalized Gaussian or other custom models. The FWMSL system, with detectors covering a large wavelength range and full waveform recording of each channel may reveal more details about targets.
6. Conclusions
This study proposes a color restoration method for the FWMSL system, to extract color information from multispectral full-waveform data. An experimental dataset of 24-color standard color card measured by the FWMSL system. Results show that the current FWMSL system with three channels, which covers RGB bands, can restore the color information accurately.
The validation effort provided insights into several potential concerns of using multispectral full-waveform Lidar data to extract color information. First, lognormal function is more suitable to fit asymmetrical waveforms with trailing compared with Gaussian function, which helps to obtain waveform parameters more accurately, thus contributing to the accuracy of color restoration finally. The integral area is more stable than the amplitude and its intensity can significantly improve the accuracy of the color restoration. Based on the characteristics of the high-speed scanning of the LiDAR system, the SNR can be improved by accumulating pulses, thereby further improving the accuracy.
With the evaluation presented in this study, the color restoration method was proven feasible for retrieving color information from multispectral full-waveform data. The FWMSL application is not limited to the geometrical and physiological features of targets but it also extends to surface texture of targets. The FWMSL system provides great convenience for extracting target surface color information, which is beneficial to ecological, environmental and urban applications.