A Cloud Vertical Structure Optimization Algorithm Combining FY-4A and DSCOVR Satellite Data

Abstract

Clouds are important for Earth’s energy budget and water cycles, and precisely characterizing their vertical structure is essential for understanding their impact. Although passive remote sensing offers broad coverage and high temporal resolution, sensor and algorithmic limitations impede the accurate depiction of cloud vertical profiles. To improve estimates of their key structural parameters, e.g., cloud top height (CTH) and cloud vertical extent (CVE), we propose a multi-source collaborative optimization algorithm. The algorithm synergizes the wide-coverage FY-4A (FengYun-4A) and DSCOVR (Deep Space Climate Observatory) cloud products with high-precision CloudSat vertical profile data and establishes LightGBM-based CTH/CVE optimization models. The models effectively reduce systematic errors in the FY-4A and DSCOVR cloud products, lowering the CTH Mean Absolute Error (MAE) to 1.8 km for multi-layer clouds, an improvement of 4–8 km over the original. The CVE MAEs for single- and multi-layer clouds are ~2.5 km. Some bias remains in complex cases, e.g., multi-layer thin clouds at low altitudes, and error tracing analysis suggests this may be related to cloud layer number misclassification. The proposed algorithm facilitates daytime near-hourly cloud retrievals over China and neighboring regions.

1. Introduction

Clouds are important to the Earth’s energy budget and atmospheric water cycle, with their spatiotemporal distribution directly affecting the radiation balance and precipitation patterns [1,2]. The cloud vertical structure (CVS), particularly the cloud top height (CTH), is a key parameter for monitoring storm systems. Within a TC, an increase in the CTH of deep convective clouds typically reflects enhanced convective activity and a greater magnitude of latent heat release, which in turn promotes the TC’s development and intensification [3]. Meanwhile, the CVS modulates the precipitation process, as within deep convective systems, greater vertical development, which results in a higher CTH, is capable of producing more intense rainfall [4]. Therefore, precise large-scale detection of CTH/CBH is of great value in understanding water cycle dynamics and predicting extreme weather.

The CTH and cloud vertical extent (CVE) are typically derived from active or passive satellite observations. Active sensors (e.g., CloudSat) offer high precision but have limited coverage [5,6], while passive sensors (e.g., MODIS) provide broad coverage but greater uncertainty regarding the CVS [7]. Consequently, the dominant approach is to integrate both active and passive observations, often using machine learning (ML) to fuse the datasets and enhance retrieval accuracy. For example, recent studies have successfully improved CTH retrievals by using ML to relate high-precision data from CALIOP (on CALIPSO) with passive observations from MODIS [8] and Himawari-8 [9], yielding promising results.

Although ML has improved the accuracy of CTH retrievals, the models above based on a single satellite data source still face notable limitations. One major challenge is the inability to effectively correct systematic errors due to the difficulty for passive remote sensing to obtain the signals of the lower part of clouds. For example, Tan et al. [10] developed a multi-layer cloud detection algorithm using Himawari-8 AHI data, but the algorithm exhibited a misclassification rate of up to 30%, primarily due to limitations in radiative transfer modeling for complex multi-layer cloud structures. Studies have shown that integrating heterogeneous satellite observations with varying spectral response functions and spatial resolutions can establish an error compensation mechanism through machine learning [11]. This is physically grounded in the fact that sensor-specific errors are typically independent [12], enabling the more effective decoupling of instrumental biases from true atmospheric signals. As a result, multi-source data fusion not only enhances model generalization but also improves the depth and robustness of error correction.

This study develops a fusion algorithm based on the Light Gradient Boosting Machine (LightGBM) and using the FY-4A (Feng-Yun-4A) and DSCOVR (Deep Space Climate Observatory) cloud products. The principle is that the CTH exhibits the existing regular error patterns in the different remote sensing products. Therefore, based on a data-driven approach, the actual vertical structure of the cloud can be inferred from these patterns. The technical approach includes data matching and model architecture. Data matching focuses on selecting spatiotemporally matched samples from the FY-4A, DSCOVR, and CloudSat. Model architecture involves establishing three models for cloud layer classification and CTH/CVE regression. By combining measurements from FY-4A and DSCOVR and regarding CloudSat’s vertical profile data as the ground truth, the new models deliver continuous daytime near-hourly CTH/CVE estimations in the Asia-Pacific region.

2. Data

This study investigates cloud profiles where the uppermost layer is classified as either in a liquid or ice crystal phase, while explicitly excluding profiles with a mixed-phase top layer (the detailed justification for this decision is provided in Appendix A). Our analysis is based on a dataset from January to May 2019, a period selected for the continuous and concurrent availability of all necessary cloud products from FY-4A, DSCOVR, and CloudSat/CALIPSO.

2.1. FY-4A Products

Feng-Yun-4A (FY-4A) is China’s second-generation geostationary meteorological satellite located at 104.7°E in a geostationary orbit. It carries the advanced Geostationary Earth Radiation Imager (AGRI) with 14 spectral channels, including the thermal infrared bands of channels 11–14, which cover 10.3–13.5 μm. Its view covers one-third of the Earth’s surface with a spatial resolution of 4 km every 15 min, spanning latitudes from 80.56°N to 80.56°S and longitudes from 174.72°E to 24.12°W.

In this study, we use Level 2 cloud products provided by the National Satellite Meteorological Center (NSMC, http://www.nsmc.org.cn (accessed on 3 September 2024)), which include the cloud top height (CTH, $h_{\mathrm{F}}$), cloud top temperature (CTT, $t_{\mathrm{F}}$), cloud top pressure (CTP, $p_{\mathrm{F}}$), cloud layer phase (CLP, ${\phi }_{\mathrm{F}}$), and cloud layer type (CLT, ${\kappa }_{\mathrm{F}}$). The FY-4A CTH product adopts the CO₂/Split-Window algorithm [6,9,13]. The Split-Window method applies observations from two infrared window bands (with central wavelengths of 10.8 μm and 12.0 μm), and the CO₂ slicing method uses observations from the CO₂ absorption band (with a central wavelength of 13.5 μm). These two techniques are combined to retrieve the cloud top height [14].

2.2. DSCOVR Products

The Deep Space Climate Observatory (DSCOVR) satellite, positioned in orbit at the Sun–Earth first Lagrange point, continually observes the entire Earth disk illuminated by the Sun, offering enhanced spatial coverage in high-latitude regions [15,16]. Its Earth Polychromatic Imaging Camera (EPIC) features 10 narrowband spectral channels, including 764 nm (Oxygen A band) and 687.75 nm (Oxygen B band). It aims to retrieve cloud parameters over the Sun-illuminated surface at a spatial resolution of approximately 8 km [7,17,18]. The observation frequency fluctuates during the year. The interval is approximately 66 min during the period from late April to early September, and extends to 108 min during other months.

In this study, we utilize the EPIC Version 03 cloud products released by NASA Earthdata (https://search.earthdata.nasa.gov (accessed on 5 September 2024)), which include cloud effective height (CEH, $h_A\mathrm{a}\mathrm{n}\mathrm{d}{h}_B$), pressure (CEP, $p_A\mathrm{a}\mathrm{n}\mathrm{d}{p}_B$), and temperature (CET, $t_A\mathrm{a}\mathrm{n}\mathrm{d}{t}_B$) based on the oxygen A/B bands, as well as cloud phase (${\mathsf{\phi }}_{\mathrm{D}}$), optical thickness of a liquid and ice cloud (${\tau }_L$ and ${\tau }_I$), and view geometry such as the sensor zenith angle (${\vartheta }_D$) and solar zenith angle (${\theta }_D$). The satellite retrieves cloud properties by measuring solar radiation absorption characteristics in the oxygen A-band (760 nm) and B-band (687 nm) and processes these measurements with the Mixed Lambertian-Equivalent Reflectivity (MLER) model. The fundamental principle assumes the cloud top acts as a reflecting surface at a certain pressure level. Given that atmospheric oxygen concentration and absorption cross-sections are known, measuring oxygen absorption intensity enables the estimation of photon path lengths above clouds, from which cloud top height can be derived [19,20]. By measuring the oxygen absorption in the path above this surface, an effective cloud pressure and height can be derived. However, the operational algorithm simplifies the retrieval by ignoring photon penetration and scattering within the cloud, yielding what are defined as the CEP and CEH, rather than the true geometric top values. In this study, we utilize these effective quantities (CEH, CEP, and the associated CET), for ease of comparison, as the DSCOVR CTH, CTP, and CTT products for our analysis.

2.3. CloudSat/CALIPSO Products

CloudSat and CALIPSO (Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observations) operate in a 705 km Sun-synchronous orbit, crossing the equator at local time 13:30, and forming a collaborative observation system. CloudSat is equipped with a 94 GHz active cloud profiling radar (CPR), offering high sensitivity to cloud water content and the strong penetration of thick clouds [2]. CALIPSO carries the Cloud-Aerosol Lidar with Orthogonal Polarization (CALIOP), which operates at wavelengths of 532/1064 nm and uses reflectivity depolarization techniques to retrieve the vertical distribution and phase of clouds [21,22]. The lidar is sensitive to thin clouds and aerosols, making it highly capable of detecting thin clouds.

We utilize the CPR and the CALIOP 2B-CLDCLASS-LIDAR product, which has a vertical resolution of 480 m with an orbital period of 98.3 min, covering the 82°N-S latitude range [22,23]. A primary limitation of CALIOP is that its lidar signal is attenuated by optically thick clouds, often preventing the detection of underlying cloud structures. Conversely, the CPR has a relatively coarse vertical resolution and may miss the optically thin clouds such as cirrus, and may instead retrieve the height of an underlying, warmer cloud as the cloud system’s true cloud top height [21]. Furthermore, recent validation studies indicate that the synergistic use of these sensors does not necessarily improve cloud phase classification; the 2B-CLDCLASS-LIDAR product, for instance, exhibits a systematic bias by overestimating the frequency of mixed-phase clouds, likely through the misclassification of pure ice or liquid layers [24].

Despite known uncertainties in determining the precise phase and boundaries of clouds, the synergy between CPR and CALIOP greatly enhances the characterization of these properties, making it a standard validation benchmark in numerous studies [8,9,13]. We use the 2B-CLDCLASS-LIDAR product as the ground truth in this study, including cloud layer top height ($h_C$), cloud layer base height ($b_C$), number of layers ($n_C$), and cloud phase (${\phi }_C$). Additionally, to describe the vertical dimension of the clouds, we use the general term cloud vertical extent (CVE). The specific calculation of CVE depends on the cloud layering. For single-layer clouds, CVE is defined as cloud geometrical thickness (CGT), calculated as the difference between the cloud top and base height. For multi-layer systems, CVE is defined as the cloud system depth (CSD), which is the distance from the top of the highest layer to the base of the lowest layer, a parameter that better represents the total vertical range perceived by passive sensors.

Adjustment of Cloud Layer Number

While the 2B-CLDCLASS-LIDAR product provides a precise layer count ($n_C$), this fine-grained classification can create feature redundancy from the perspective of a passive sensor. Columns with varying layer counts might exhibit similar characteristics from the view of passive remote sensing; for example, two distinct, optically thin cirrus layers may be indistinguishable from a single, thicker one.

To create a training label more representative of such passive observations, we propose an adjustment based on physical similarity. This adjustment is defined by three criteria. First, the layers can be merged and must share the same cloud phase, as this indicates similar microphysical and radiative properties. Second, the vertical distance between them must be less than 1.5 km; radiative transfer within the intervening clear-sky gap cannot be considered negligible when the distance is large. Third, the thickness of any individual layer must be less than 3 km, which prevents the incorrect merging of separate, well-developed cloud systems. The resulting “adjusted cloud layer number” (CLN, $n_C^{\prime }$) is used as the ground-truth label in this study.

It is acknowledged that the CloudSat/CALIPSO product, while being the best available ground truth for global CVS studies, has its own uncertainties. A primary concern is the ~500 m physical vertical resolution of the CPR, which might lead to ambiguities in identifying closely spaced, thin cloud layers. However, this limitation must be evaluated in the context of the problem we aim to solve. Passive sensors, which this study seeks to optimize, are themselves insensitive to fine vertical structures; their effective vertical resolution for the underlying layers is significantly coarser than that of the active sensors. Our layer-merging approach is therefore a deliberate design choice, coarsening the “truth” classification to better match what a passive sensor can realistically discern. Judged against the observational capabilities of passive remote sensing, the ~500 m vertical resolution of the CloudSat/CALIPSO product is thus considered more than adequate for the purpose of training our optimization model.

2.4. Problems in Passive Remote Sensing

Passive remote sensing has its own systematic errors, stemming from the various principles by which sensors acquire information. Therefore, we utilize active remote sensing cloud products to assess the accuracy of passive remote sensing cloud products. Below are the systematic errors present in the passive remote sensing CTH products used in this study.

2.4.1. FY-4A CTH Systematic Error

The CTH retrieval for FY-4A is based on thermal infrared (TIR) radiative transfer theory. The core principle involves retrieving cloud top temperatures from the radiation difference between the 10.8 μm and 12.0 μm infrared window channels, and then calculating the CTH using the atmospheric temperature profile [14]. Its performance heavily relies on the characteristics of the cloud layer structure, as thermal emission from lower portions of the cloud or underlying layers can also contribute to the satellite-observed radiance.

Figure 1a presents the cloud top height (CTH) retrieval accuracy for single-layer cloud systems, demonstrating an RMSE of 3.9 km and MAE of 2.6 km. The data points collectively exhibit a systematic underestimation pattern, while most of them align reasonably well with the 1:1 reference line.

In contrast, the multi-layer cloud systems (Figure 1d) show significantly degraded retrieval accuracy, with RMSE increasing to 6.6 km and MAE to 5.4 km. This performance reduction stems from the known limitations of TIR-based retrievals when observing high, semi-transparent clouds. The underlying physical mechanism is that the passive sensor does not sense the true geometric top but effectively “sees into” the cloud to a certain optical depth [6]. This results in the received radiance being contaminated by warmer thermal emissions from within or below the cloud, leading to a retrieved CTH that is systematically biased to be low. This effect is particularly pronounced in multi-layer cloud systems, where studies consistently find that the retrieved CTH is often placed somewhere between the upper and lower cloud layers [13,25].

Figure 2 reveals a clear height-dependent bias in the CTH retrievals from all instruments. For low clouds (CTH < 4 km), the FY-4A retrievals show low bias, with the mean error (ME) consistently between −1 km and +1 km for both single- and multi-layer systems (yellow lines). However, as the cloud top height increases, a systematic underestimation bias becomes prominent. For clouds with a CTH above 4 km, the ME becomes progressively more negative, indicating that the retrieved height is increasingly lower than the true height. This underestimation is more severe for multi-layer clouds (solid yellow line) than for single-layer clouds (dashed yellow line). For instance, at 16 km, the underestimation for multi-layer clouds (ME ≈ −8 km) is significantly greater than that for single-layer clouds (ME ≈ −5 km).

2.4.2. DSCOVR CEH Discrepancy

Figure 1b,c compare the DSCOVR cloud effective height (CEH) against the CTH from CloudSat/CALIPSO for single-layer cloud systems (RMSE = 5.5/6.1 km, MAE = 4.2/4.7 km), showing a cluster in line as Figure 1a does, but with a lower slope, which indicates a larger discrepancy.

For the multi-layer cloud system shown in Figure 1e,f, the discrepancy becomes even more pronounced (RMSE = 9.4/10.1 km, MAE = 8.9/9.5 km), with points densely clustering in the lower right corner, indicating a greatly negative bias in the high-altitude interval.

Figure 2 reveals the height-dependent discrepancy between DSCOVR’s CEH and the reference CTH. The mean difference (ME) becomes increasingly negative with altitude, and this discrepancy is consistently larger for the B-band (red lines) than the A-band (blue lines), and is more pronounced for multi-layer (solid lines) versus single-layer (dashed lines) clouds. For high-altitude multi-layer systems, this culminates in a large ME, with values more negative than -10 km for both the A-band and B-band.

Key explanations for this phenomenon are provided by theoretical studies from Yang et al. [7], Davis et al. [17,18], and Marshak et al. [15]. The operational Mixed Lambertian-Equivalent Reflectivity (MLER) algorithm intentionally simplifies the radiative transfer problem by ignoring photon penetration and scattering within the cloud. This model deficiency results in a systematic negative bias in DSCOVR CEH retrievals for single- and multi-layer cloud systems. Notably, Yang et al. [26] further discovered that in multi-layer cloud systems, the complexity of the cloud structure and interactions between layers amplify this systematic bias. In Figure 1, the RMSE of the CEH retrievals for multi-layer clouds (9.4 km) is 71% higher than that of single-layer clouds (5.5 km), confirming the negative impact of vertical cloud structure complexity on retrieval accuracy.

The CTH products from FY-4A and DSCOVR satellites both exhibit distinctly linear point distributions for single-layer clouds, with their primary difference manifested in slope variations. In contrast, under multi-layer cloud conditions, their distribution characteristics show marked differences. FY-4A data points are predominantly clustered in the upper right corner near the 1:1 line, while DSCOVR data points concentrate densely in the lower right corner. The two products demonstrate fundamentally distinct discrepancy patterns between single-layer and multi-layer clouds. These layer-dependent patterns suggest that combining both satellite products may effectively identify the number of layers.

3. Methodology

3.1. Spatiotemporal Collocation

3.1.1. Matching for Model Training

To optimized the cloud heights from FY-4A and DSCOVR using the 2B-CLDCLASS-LIDAR CTH as ground truth, we established a multi-sensor dataset that was collocated in both space and time through the following strict screening criteria: (1) temporal collocation, which requires near-simultaneous satellite transits, and (2) spatial consistency, which requires co-located valid data from all three sensors within a defined geographic domain. The filtered pixels (FY-4A and DSCOVR) and soundings (CloudSat/CALIPSO) are recognized as temporally concurrent and spatially adjacent. Figure 3 illustrates the overlap of FY-4A, DSCOVR, and CloudSat/CALIPSO at several consecutive times.

A key challenge in fusing these datasets is the significant mismatch in their observational characteristics. The instruments operate on different schedules and spatial resolutions: FY-4A provides imagery with a spatial resolution of 4 km every 15 min; DSCOVR provides imagery with a spatial resolution of 8 km every 66 to 108 min, with the observations for the 10 wavelengths taken over a time span of 7 min; while CloudSat/CALIPSO offers fine-resolution (1.4 km) along-track soundings approximately every 98 min. Due to this lack of spatiotemporal synchronization, exact co-locations are rare. Consequently, this study suggests a spatiotemporal proximate matching strategy, outlined as follows:

1.

Temporal Pre-Screening: FY-4A and DSCOVR images are selected based on their imaging times and CloudSat/CALIPSO’s flight plan. One CloudSat/CALIPSO orbit may correspond to multiple FY-4A images. This is because, when CloudSat/CALIPSO passes through the sunlit side of the Earth, FY-4A captures images many times with a much higher frequency than that of CloudSat/CALIPSO.

2.

Spatial Matching: Given the large number of pixels in the images, global brute-force matching is time-consuming. Therefore, we employ a two-step method to improve the matching efficiency. First, for each CloudSat/CALIPSO sounding, we define a 3° × 3° bounding box (approximately 1.5° in each direction) and discard all FY-4A and DSCOVR pixels falling outside this area. Second, we calculate the geodesic distance from each of these candidate pixels in the box to the sounding using the Haversine formula [27]. The closest pixel is then identified, and to ensure high spatial consistency, the geodesic distance from the matched FY-4A and DSCOVR pixels to the sounding should be less than 5 km.

3.

Temporal Matching: A temporal matching is carried out at the pixel level. If the time difference between the matched FY-4A and DSCOVR pixels and the CloudSat/CALIPSO sounding is less than 15 min, they are considered to have been captured within the adjacent time frame. The 15 min threshold is chosen as a compromise between dataset size and physical consistency. A stricter window (< 5 min) was found to discard over 80% of potential matches. Conversely, a longer window is physically questionable, as the cloud properties from a fixed-point perspective significantly decorrelate on a timescale of about 15.5 min [28]. Furthermore, the 15 min window is designed to accommodate occasional 30 min gaps in the FY-4A data stream, ensuring better data matching.

3.1.2. Matching for Model Application

In the model application phase, we have to perform spatiotemporal matching of the FY-4A and DSCOVR cloud products and apply the model to the matched data to optimize the CTH/CVE in the overlapping areas of the FY-4A and DSCOVR cloud products. The matching method is similar to that in Section 3.1.1. Since the DSCOVR cloud product has a lower spatial resolution, we center the matching process on DSCOVR pixels and find the spatiotemporally closest FY-4A pixels.

3.2. CTH/CVE Optimization

The different error distributions of FY-4A and DSCOVR in Figure 1, along with the strong correlation between their errors and cloud height in Figure 2, suggest that the differences between these cloud products can reflect different types of cloud systems and even indicate their deviations from the actual cloud top height. Following this line of thought, we present an algorithm made up of three models: the cloud layer number (CLN) model, designed to identify single-layer and multi-layer cloud systems; the cloud top height (CTH) model, which directly optimizes the cloud top height; and the cloud vertical extent (CVE) model, which is for indirectly calculating the cloud base height.

3.2.1. Dataset Splitting

A total of 332,194 cloudy pixels were collected and divided into two groups with a 1:4 ratio. The first group was used for the CLN model, while the second was for the CTH/CVE model. For each dataset, we split it into training, validation, and test sets in an 8:1:1 ratio again and trained the three models mentioned individually.

3.2.2. CLN Estimation Model

In this study, to address the challenges of CTH retrievals in multi-layer cloud systems (refer to Section 2.4), we plan to use the CLN as the input for the CTH/CVE model. However, since FY-4A and DSCOVR cloud products do not supply the number of cloud layers, it is crucial first to establish a CLN estimation model based on the differing sensitivities in CLN between FY-4A and DSCOVR, and to train it using the adjusted cloud layer number ($n_C^{\prime }$) determined with CloudSat/CALIPSO. This model can then provide the estimated cloud layer number ($n_{\mathrm{M}}$) over large areas and assist in estimating CTH and CVE. The data list for model training is displayed in

Table 1. Inputs for training the CLN, CTH, and CVE estimation models.

Model	Input	Label
$\mathrm{CLN}(n_M$)	$\mathrm{FY}-4\mathrm{A}:h_F$$,t_F$$,p_F$ $\mathrm{DSCOVR}:h_A$$,t_A$$,p_A$$,h_B$$,t_B$$,p_B$$,{\tau }_L$$,{\tau }_I$$,{\vartheta }_D$$,{\theta }_D$	$\mathrm{CloudSat}/\mathrm{CALIPSO}:n_C^{\prime }$
CTH/CVE	$\mathrm{FY}-4\mathrm{A}:h_F$$,t_F$$,p_F$$,{\phi }_F$$,{\kappa }_F$ $\mathrm{DSCOVR}:h_A$$,t_A$$,p_A$$,h_B$$,t_B$$,p_B$$,{\phi }_D$$,{\tau }_L$$,{\tau }_I$$,{\vartheta }_D$$,{\theta }_D$    $\mathrm{CLN}:n_M$	$\mathrm{CloudSat}/\mathrm{CALIPSO}:h_C$$,l_C$

.

3.2.3. CTH/CVE Optimization Model

The cloud top height (CTH) model and the cloud vertical extent (CVE) model are trained separately, using the same input. The input variables for the CTH/CVE model are set as follows:

1.

Cloud Parameters: These include the cloud top height ($h_{\mathrm{F}}$), temperature ($t_{\mathrm{F}}$), and pressure ($p_{\mathrm{F}}$) from the AGRI/FY-4A products, along with those retrieved from the oxygen A/B absorption bands of EPIC/DSCOVR ($h_A$, $t_A$, $p_A$, $h_B$, $t_B$, $p_B$). Additionally, the cloud layer phase (${\phi }_{\mathrm{F}}$) and type (${\kappa }_{\mathrm{F}}$) from FY-4A, as well as the liquid and ice cloud optical thickness (${\tau }_L$, ${\tau }_I$) from DSCOVR are selected as input variables.

2.

View Geometry: Changes in the solar zenith angle (${\theta }_D$) can affect the depth of photon penetration into the cloud layers and the length of the transmission path [3,29]. This geometric effect directly leads to systematic biases in the retrieval of CTH based on the O2-A/B absorption bands (as discussed in Section 2.4.2). Additionally, an increase in the sensor zenith angle (${\vartheta }_{\mathrm{D}}$) can alter the cloud top radiation characteristics [13]. The model incorporates both angle parameters as key radiation transfer constraint variables.

3.

Cloud Layer Number: The number of cloud layers distinguishes between single- and multi-layer clouds. We include the estimation of cloud layer number, as discussed in Section 3.2.2, as an input variable.

3.2.4. Model Training

The model development, data analysis, and visualization for this study were performed using Python (version 3.12) and MATLAB (version R2023a). The core optimization models were developed using the LightGBM algorithm (version 4.6.0) [30] based on the Gradient Boosting Decision Tree (GBDT) framework. LightGBM’s leaf-wise growth strategy prioritizes splitting the leaf node that yields the great reduction in loss, demonstrating superior computational efficiency and predictive accuracy in regression tasks [31,32]. This feature makes LightGBM particularly well-suited for managing high-dimensional remote sensing datasets derived from multi-source satellites like FY-4A and DSCOVR.

For the CLN classification model, we used the multi-class cross-entropy loss (MCEL) as the evaluation metric, while the Root Mean Squared Error (RMSE) was used for the CTH/CVE regression models. Through a series of tuning experiments, an optimal learning rate of 0.005 was identified for all models, which provided a good balance between convergence speed and final accuracy. The complete, final hyperparameter configurations for each model, which were determined through multiple tests, are summarized in

Table 2. Hyperparameter configurations for the CLN, CTH, and CVE estimation models.

Hyperparameter	CLN Model	CTH/CVE Model	Description
boosting_type	gbdt	gbdt	Gradient Boosting Decision Tree
n_estimators	10,000	10,000	Number of boosting iterations
learning_rate	0.005	0.005	Step size shrinkage
max_depth	10	10	Maximum tree depth
num_leaves	380	300	Maximum number of leaves per tree
bagging_fraction	0.75	0.75	Fraction of data to be used for each iteration
feature_fraction	0.75	0.75	Fraction of features to be used for each iteration
objective	multiclass	regression	Specifies the learning task and objective

.

To further enhance the algorithm’s robustness, a dynamic early stopping strategy and a double random sampling mechanism are integrated for the CLN, CTH, and CVE models. Specifically, the model training automatically terminates if the evaluation metric does not improve for 50 consecutive rounds. This helps balance the model’s convergence efficiency and the risk of overfitting. Additionally, in the feature dimension, only 75% of the features are randomly selected for node splitting in each iteration, reducing the model’s sensitivity to noisy features by suppressing feature collinearity [33]. In the sample dimension, Bootstrap sampling is performed every four iterations, randomly selecting 75% of the training samples to update the data distribution. This forces the model to optimize on differentiated data subsets continuously, enhancing the diversity of the gradient update directions. Experiments showed that, compared to traditional GBDT strategies, the double random sampling strategy reduced the RMSE on the validation set by 5.5%.

3.3. Evaluation Methods

This study uses the RMSE (Root Mean Squared Error), MAE (Mean Absolute Error), and R² (coefficient of determination) as evaluation metrics for the CTH and CVE optimization, and the accuracy (α) and recall rate (${\rho }_i$) for the CLN estimation. In Equations (1) and (2), $n_{i,j}$ represents the number of samples that actually have i layers and are determined to have j layers. RMSE measures the model’s estimation ability. MAE is robust to outliers and is more stable than RMSE. The smaller the RMSE or MAE, the more accurate the model’s predictions. R² represents the proportion of the total variance that the model explains. It measures the degree to which the model fits the data. The value ranges from 0 to 1, and the closer the value is to 1, the stronger the model’s explanatory power.

$$\alpha ={\displaystyle \frac{\sum _i{n}_{i,i}}{\sum _i\sum _j{n}_{i,j}}}$$

(1)

$${\rho }_i={\displaystyle \frac{n_{i,i}}{\sum _j{n}_{i,j}}}$$

(2)

4. Results

4.1. CLN Model Performance

In the CLN model training, we utilized the adjusted number of clouds ($n_C^{\prime }$) rather than the original ($n_C$) as the training label. Figure 4a,b illustrate the accuracy of CLN models trained using the cloud layer number before and after adjustment.

The adjustment shows the benefits we expected. Firstly, the adjusted model achieves an overall accuracy of 74%, representing a 20% improvement over the original model. Secondly, the adjusted model has enhanced multi-layer cloud identification. For clouds with two and three layers, the recall rates reach 69% and 56%, respectively, marking improvements of 18% and 24% compared to the original. Finally, the adjustment improves consistency in identifying multi-layer cloud systems. The original 2B-CLDCLASS-LIDAR dataset contains numerous high-complexity cloud samples with more than three layers, accounting for about 7% of the total samples. The recall rates for the samples with four and five layers in Figure 4a are as low as 22% and 17%, respectively. After the adjustment, the proportion of multi-layer cloud samples with more than three layers is reduced to 2%, minimizing redundant features.

Figure 4b also shows that a few single-layer clouds are still incorrectly identified as multi-layer clouds and vice versa. For instance, 19% of double-layer clouds are misclassified as single-layer clouds, while 15% of single-layer clouds are misclassified as double-layer clouds. These misclassified samples would introduce systematic biases in the subsequent CTH/CVE model training.

4.2. CTH/CVE Model Performance

The CTH/CVE optimization model proposed in this study demonstrates good performance on the test set. Figure 5a demonstrates that the CTH model achieves superior performance with an RMSE of 2.8 km, an MAE of 1.9 km, and an R² of 0.70. After excluding outliers, the RMSE and MAE decrease to 2.0 km and 1.5 km, respectively, and the R² increases to 0.84. In the lower part (CTH < 6 km) of Figure 5a, the slope of the fitted line is close to the 1:1 line but the intercept is about 2 km. For the higher part (CTH ≥ 8 km), the fitted line is closer to the 1:1 line, revealing better accuracy for high clouds.

Outliers were identified using the Interquartile Range (IQR) method, which classifies a data point as an outlier if its prediction error—calculated as the difference from the CloudSat/CALIPSO ground truth—falls by more than 1.5 times the IQR below the first quartile (Q1) or above the third quartile (Q3) [34].

Figure 5b presents the accuracy of the CVE (cloud geometrical thickness (CGT) for single-layer clouds, cloud system depth (CSD) for multi-layer clouds) model, with an RMSE of 3.5 km, MAE of 2.6 km, and R² of 0.50. After excluding outliers, the RMSE and MAE decrease to 3.0 km and 2.3 km, respectively, and the R² increases to 0.62. In the thinner part (CVE < 6 km), points cluster at 1–4 km above the 1:1 line. Meanwhile, for the thicker part (CVE ≥ 6 km), the fitted line slope decreases significantly, revealing a systematic underestimation and suggesting the model’s limitations in capturing thick clouds.

The evaluation metrics for removing outliers are only used in the above two paragraphs. The evaluation metrics containing outliers are used in all subsequent paragraphs and are summarized in

Table 3. The evaluation metrics of the original and optimized cloud top height.

	All Clouds			Single-Layer Clouds			Multi-Layer Clouds
	R2	RMSE (km)	MAE (km)	R2	RMSE (km)	MAE (km)	R2	RMSE (km)	MAE (km)
FY-4A	−0.09	5.5	4.0	0.45	3.9	2.6	−2.85	6.6	5.4
DSCOVR A-band	−1.15	7.7	6.5	−0.06	5.5	4.2	−6.75	9.4	8.9
DSCOVR B-band	−1.52	8.3	7.1	−0.30	6.1	4.7	−7.90	10.1	9.5
New Model	0.70	2.8	1.9	0.69	3.0	2.0	0.60	2.7	1.8

.

(A)

Performance in Single-Layer Clouds

Based on the estimated CLN, we categorize the samples used in Figure 5 into single- and multi-layer cloud systems. Figure 6a demonstrates the CTH’s accuracy in single-layer cloud systems, achieving an R² of 0.69, an RMSE of 3.0 km, and an MAE of 2.0 km, which is improved beyond the FY-4A (RMSE = 3.9 km, MAE = 2.6 km) and the DSCOVR (RMSE = 5.5 km, MAE = 4.2 km). The single-layer cloud test set contains 8.09% outliers, and their removal leads to 30% and 25% improvements in RMSE and MAE, respectively. Specifically referring to the part below 6 km in Figure 6a, the points cluster above the 1:1 line, showing a systematic overestimation of 1–2 km. Within the height ranges of 9–12 km and 13–17 km, the points closely align with the 1:1 line, demonstrating the model’s good accuracy in single-layer clouds with high cloud tops.

Figure 7a shows the MAE of CTH calculated in each 1 km height bin for single- and multi-layer clouds. For single-layer clouds (the dark blue line), the MAEs are generally under 2 km and the calculation performs particularly well in the densely sampled 1–4 km range, where the MAE is consistently around 1 km. Slightly higher MAEs (>2 km) are observed in the 0–1 km, 4–7 km, and 16–18 km ranges, which correspond to regions with relatively fewer samples.

Figure 6c shows the CVE model’s performance under single-layer cloud conditions, yielding an R² of 0.43, an RMSE of 3.3 km, and an MAE of 2.3 km. Notably, although the MAE of single-layer clouds in Figure 7b rises intensely along with the increasing vertical extent (the red line), thick cloud samples account for a relatively small proportion in the dataset (the red histogram in Figure 7b), thus the impact on operational applications remains relatively minor.

(B)

Performance in Multi-Layer Clouds

Figure 6b shows CTH optimization in multi-layer systems, achieving an R² of 0.60, an RMSE of 2.7 km, and an MAE of 1.8 km, which represent a substantial improvement over FY-4A (RMSE = 6.6 km, MAE = 5.4 km) and DSCOVR (RMSE = 9.4 km, MAE = 8.9 km). The clouds with high cloud tops (CTH > 8 km) cluster tightly along the 1:1 line. The light blue line in Figure 7a reveals that the worst MAE is at the 0–1 km range, but is a minor fraction of the dataset (the light blue histogram in Figure 7a). The peak at 5–6 km accounts for a minor part as well. Most clouds, such as those with cloud tops between 8 and 17 km, have the best MAE under 2 km.

Figure 6d shows the CVE model’s performance under multi-layer cloud conditions, yielding an R² of 0.43, an RMSE of 3.6 km, and an MAE of 2.8 km. Clouds with a vertical extent less than 6 km tend to be overestimated, while those exceeding 10 km are generally underestimated. Multi-layer clouds (the orange line in Figure 7b) present a similar MAE–height variation trend as single-layer clouds (the red line) do, indicating that the CSD optimization performs well in thin clouds (CSD < 10 km), but degrades with increasing cloud vertical extent.

A seemingly counter-intuitive result is observed for clouds at high altitudes and large thicknesses, where the model for multi-layer clouds appears to perform better (i.e., has a lower MAE) than for single-layer clouds. This phenomenon is likely attributable to a significant imbalance in the validation dataset’s sample distribution. For instance, considering CTH, the number of multi-layer samples above 12 km is nearly double that of single-layer samples (79,630 versus 40,464), yielding a more statistically stable error metric for multi-layer cases.

Whether in single- or multi-layer clouds, the MAE of the optimized CTH remains below 2 km for the clouds with cloud tops between 8 and 17 km, which constitute the main part of the total dataset. For CVE optimization, the MAE stays under 2 km for the clouds with a vertical extent less than 10 km, which constitute the main part of the total dataset as well. These results represent significant improvements over FY-4A and DSCOVR products.

5. Discussion

5.1. Error Source Analysis

Section 4.1 points out that the CLN model misclassifies some single-layer clouds as multi-layer clouds and vice versa. Therefore, it is necessary to assess how these misclassifications impact the performance of the CTH/CVE model. For clarity, we define Cloud 1 as all single-layer cloud samples (where $n_C^{\prime }$ = 1), Cloud 21 as double-layer clouds misclassified as single-layer clouds (where $n_C^{\prime }$ = 2, but $n_M$ = 1), Cloud 2 as all double-layer cloud samples (where $n_C^{\prime }$ = 2), and Cloud 12 as single-layer clouds misclassified as double-layer clouds (where $n_C^{\prime }$ = 1, but $n_M$ = 2).

Figure 8a–f compare the distribution probabilities of the above four cloud categories on CTH and CTT, and the data used are from the FY-4A and DSCOVR products. Cloud 1 (blue histogram) and Cloud 21 (blue line) exhibit similar distributions for both CTH and CTT, as do Cloud 2 (red histogram) and Cloud 12 (red line). However, Figure 8g,h demonstrate Cloud 1 and Cloud 21 have different vertical structures based on the true CTH and CVE from the CloudSat/CALIPSO products. This suggests that cloud misclassification stems from their cloud top properties appearing more similar to those of clouds with different layer numbers in passive remote sensing observations. For example, cases where thin cirrus overlies cumulus may be misclassified as single-layer clouds in passive remote sensing. The training process of the CLN model is susceptible to interference from clouds with different layer numbers but similar cloud top characteristics, resulting in cloud layer misclassification.

The samples with erroneous layer number labels may subsequently compromise the accuracy of both the CTH and CVE models. To validate this hypothesis, we designed a comparative experiment of using the true cloud layer number (${n\mathrm{\prime }}_{\mathrm{C}}$) instead of the estimated values (obtained via the CLN model, $n_{\mathrm{M}}$) as the input and retrained the CTH and CVE models. Figure 9 is the validation on the test set, and all the data used are the same as those in Figure 6.

Figure 9 shows the improvements in the CTH and CVE optimizations compared to Figure 6. Regarding the CTH, the R² for single- and multi-layer clouds increases by 11% (from 0.69 to 0.80) and 10% (from 0.60 to 0.70), respectively. Meanwhile, the RMSE decreases from 3.0 km to 2.4 km for single-layer clouds and 2.7 km to 2.3 km for multi-layer clouds (achieving relative improvements of 20% and 14%). The MAE also reduces by 0.4 km for single-layer clouds and 0.2 km for multi-layer clouds.

The CVE optimization shows a better improvement than the CTH. R² for single- and multi-layer clouds increases by 36% (from 0.43 to 0.79) and 32% (from 0.43 to 0.75), respectively. The RMSE shows remarkable optimization from 3.3 km to 2.0 km for single-layer clouds and from 3.6 km to 2.4 km for multi-layer clouds (achieving relative improvements of 40% and 33%). The MAE decreases by about 1.0 km for single- and multi-layer clouds.

Based on the above analysis, we infer that the reason for cloud layer misclassification stems from the similar appearance in passive remote sensing, and accurate cloud layer numbers can greatly improve the performance of the CTH and CVE optimization models.

5.2. Importance of Input Variables

We conducted an ablation experiment to evaluate the importance of input variables in the CTH and CVE optimizations. As shown in Figure 10, the input variables are divided into six groups: FY-4A cloud products (i.e., CTH ($h_{\mathrm{F}}$), CTT ($t_{\mathrm{F}}$), CTP ($p_{\mathrm{F}}$), CLT (${\kappa }_{\mathrm{F}}$), CLP (${\phi }_{\mathrm{F}}$)), DSCOVR A- and B-band cloud products (i.e., CTH ($h_A\mathrm{a}\mathrm{n}\mathrm{d}{h}_B$), CTT ($t_A\mathrm{a}\mathrm{n}\mathrm{d}{t}_B$), CTP ($p_A\mathrm{a}\mathrm{n}\mathrm{d}{p}_B$)), cloud optical thickness from DSCOVR (i.e., liquid and ice (${\tau }_L$ and ${\tau }_I$)), geometric observation angles from DSCOVR (i.e., sensor and solar zenith angle (${\vartheta }_D$ and ${\theta }_D$)), and number of layers ($n_M$). The experiment excludes each group to obtain corresponding MAE–height trends. Additionally, the importance is quantified using LightGBM’s information gain metric [35], which measures the total information gain from feature splits, robustly capturing contributions in high-dimensional, non-linear systems.

The ablation experiments reveal significant MAE differences between the control group (without exclusion) and experimental groups, especially for low clouds (CTH < 5 km), thin clouds (CVE < 4 km), and thick clouds (CVE > 8 km), validating our feature selection.

Figure 10a shows the MAE for CTH optimizations. For instance, groups excluding FY-4A parameters or observation angles exhibit 1–2 km higher MAEs at 2–6 km altitudes, and excluding the cloud layer number also increases the MAE in this range, highlighting their importance. Furthermore, Figure 11a confirms that the FY-4A CTH is the most important input variable, followed by the cloud layer number.

Figure 10b shows the MAE for CVE optimizations. For clouds with a 4–8 km vertical extent, the MAE is consistent between experimental and control groups. Nevertheless, in other ranges, experimental groups’ MAEs differ by approximately 0.8 km from the control group. All experimental groups display similar trends across vertical extent ranges. Figure 11b identifies cloud layer number as being paramount for CVE optimizations, with FY-4A’s CTH/CTP and observation angles as being secondary important input variables.

Notably, Figure 11a,b reveal a different variable importance between CTH and CVE optimization. CTH depends more on cloud top parameters (height/pressure), while CVE relies on vertical structure information (e.g., layer number). The reliance on the cloud layer number corresponds to the discussion in Section 5.1 that using the true cloud layer number ($n_C^{\prime }$) reduces the MAE by 16% for CTH and 38% for CVE, and it has the potential to improve the CTH/CVE optimization. In addition, FY-4A’s CTH/CTT ranks second in contributing to the CVE optimization.

It is worth noting that, although LightGBM’s gain metric highlights the cloud layer number as an important feature for both CTH and CVE models (Figure 11), its removal alone causes an MAE change comparable in magnitude to the removal of other entire feature groups (Figure 10). This seeming inconsistency may stem from the following:

1.

Other ablation groups removed multiple variables, whose combined effects amplified performance drops, making them comparable to the cloud layer number removal case;

2.

The cloud layer number provides classification information only and lacks direct height or vertical extent information, requiring interaction with other variables for a full effect.

The cloud layer number acts as a crucial prior constraint. It identifies multi-layer structures and separates error sources. Multi-layer clouds often indicate phase layering or greater vertical extent, thus adding the cloud layer number constrains the combinations of cloud top parameters and reduces ambiguity. As a structural classifier, the cloud layer number complements cloud top parameters, together enabling a more complete representation of cloud 3D structures.

5.3. Underlying Principles of the ML-Based Method

The models developed in this study estimate cloud vertical structure using only passive remote sensing observations from FY-4A and DSCOVR. This approach presents a fundamental challenge, as passive sensors inherently lack the ability to probe into clouds and directly measure their vertical profiles. Clarifying the feasibility of this approach is therefore essential. The following discussion elaborates on the underlying physical principles that enable this retrieval by leveraging the synergy between the two different sensor systems.

For CTH and CVE estimation, a key consideration is that passive retrievals yield ‘effective’ rather than true geometric values, as their signals are contaminated by radiation from within and below the cloud. Since FY-4A (TIR-based) and DSCOVR (O₂-band-based) operate on different physics, the systematic patterns of this ‘contamination’ differ for each sensor when viewing the same cloud system. For instance, for a high, semi-transparent cirrus over a lower, warmer water cloud, the FY-4A TIR sensor will be biased low due to contamination from the warm emission below. The DSCOVR O₂-band retrieval will also be biased as the lower cloud alters the effective photon path length, but this bias pattern is physically distinct from that of the TIR method. Our model learns to decode the discrepancy between these two different, biased measurements. By simultaneously analyzing both sets of products, the model can better constrain the true vertical location (CTH) and extent (CVE) of the cloud system, a task that is ill-posed for a single passive instrument.

For CLN estimation, the mechanism is likely more complex. We hypothesize that the model learns to distinguish between single- and multi-layer clouds by recognizing the different signatures they leave on the full suite of input variables. It is plausible that the complex inter-layer radiative interactions unique to multi-layer systems (e.g., multiple reflections, thermal trapping) create subtle, systematic patterns in the relationships between the various cloud top parameters. The CLN model may then learn to identify these multi-variate ‘fingerprints’ to perform its classification.

5.4. Post-Processing Bias Correction

A potential limitation of our results is the presence of systematic, magnitude-dependent biases, where CTH is overestimated for low clouds and underestimated for high clouds, with CVE exhibiting a similar pattern for thin and thick clouds, respectively. Given the clear pattern of systematic bias in our initial results, we investigated two distinct post-processing correction approaches to mitigate it: a piecewise linear regression, and a more complex data-driven calibration using a second LightGBM model.

While both methods were effective at reducing systematic bias, neither led to a significant improvement in overall accuracy metrics (R², RMSE, MAE), which perhaps suggests that a more sophisticated correction technique is required than those we investigated. A detailed description of these experiments, including the methodology and comparative figures, is provided in Appendix B. We acknowledge, however, that exploring more advanced bias-correction techniques remains a valuable direction for future research.

6. Conclusions

This study presents a LightGBM-based algorithm for the optimization of cloud top height (CTH) and cloud vertical extent (CVE) by integrating the FY-4A and DSCOVR passive sensing cloud products. The algorithm enables high-frequency (hourly), high-accuracy, wide-area cloud vertical structure measurements. Validation based on CloudSat/CALIPSO shows MAEs of 2.0 km for the single-layer CTH, 2.3 km for single-layer cloud geometrical thickness (CGT), 2.7 km for multi-layer cloud system depth (CSD), and 1.8 km for multi-layer CTH, improving accuracy by 4–8 km over the FY-4A and DSCOVR products. The key findings are as follows:

The complementarity between cloud layer number and cloud top parameters (CTH/CTT/CTP) reveals a novel synergistic paradigm. The cloud layer number acts as a “structural classifier” providing vertical structure prior constraints through discrete labels (e.g., single- and multi-layer clouds), while cloud top parameters serve as “physical quantifiers” conveying radiative characteristics. Their combination effectively decouples systematic errors between FY-4A and DSCOVR.

The model demonstrates good performance in typical cloud systems: it achieves relatively small errors for the CTH/CVE of high-top thick clouds, but still faces challenges in characterizing vertically complex clouds due to the radiative signal aliasing effect, particularly in distinguishing single-layer clouds from multi-layer systems with similar cloud top properties. The experiment of using the true cloud layer number achieves the best accuracy and highlights the importance of reliable cloud layer classification for CTH and CVE estimations.

The optimization algorithm offers a new approach for the real-time monitoring of cloud vertical structure evolution. This method utilizes existing satellites to enhance the ability to observe the Earth, which is conducive to global-scale cloud water cycle monitoring and to improving the ability to track cloud vertical structural changes during extreme weather events. Moreover, the fundamental framework of this algorithm is not sensor-specific and could be readily adapted for use with other geostationary satellites, such as the GOES or Meteosat series, to potentially achieve near-global, high-frequency CVS monitoring.