A Novel Method for Boundary Value Determination in the Fernald Inversion for Horizontal Lidar Measurements

Abstract

In the conventional Fernald inversion, the boundary value at the calibration point is commonly estimated using a slope-based method. This procedure increases algorithmic complexity and can introduce retrieval errors. Here, we propose an alternative boundary-value determination scheme that exploits the tendency of the Fernald forward-integration equation to diverge. Simulation experiments show that the proposed scheme is more stable than the slope method under atmospheric inhomogeneity and measurement noise. We further applied the method to horizontal lidar scans acquired in Lankao (Henan Province, China), capturing a regional pollution transport and dispersion episode. Together, these results suggest that the method enables real-time monitoring of the horizontal distribution of regional pollutants.

1. Introduction

Over the past five decades, atmospheric pollution in China has intensified alongside rapid industrialization and urbanization. Among these pollutants, particulate matter (aerosols) is a major concern [1,2]. Elevated aerosol loadings have well-documented impacts on public health (e.g., respiratory and cardiovascular disease), visibility and transportation safety, and climate and weather through radiative effects. Aerosols also modulate the budgets of trace gases such as O₃, NO₂, and SO₂ via heterogeneous surface reactions [3,4,5,6].

Lidar, an active optical remote-sensing instrument, provides long-range measurements with high spatiotemporal resolution and has been widely used for aerosol monitoring for decades [7,8,9,10]. Most lidar aerosol observations have focused on the vertical distribution of aerosols. For elastic (Mie) lidar, calibration is typically performed in an upper, relatively clean layer where molecular (Rayleigh) scattering dominates the return. The boundary condition—often expressed as the backscatter ratio (total-to-molecular backscatter)—is then set close to unity (e.g., 1.01), and the Fernald method is used to retrieve aerosol extinction profiles [11].

More recently, environmental agencies have deployed scanning lidar to map the horizontal distribution of particulate matter over urban areas [12,13,14,15]. Horizontal aerosol extinction is commonly retrieved using either segmented Collis-type slope methods or Fernald-type inversions [13,16,17]. However, Collis-type slope methods assume horizontal homogeneity along the optical path. Consequently, the range profile must be pre-segmented into quasi-uniform sections, and sharp enhancements in the return from aerosol layers or fog can violate the assumption and degrade the robustness of the slope estimate. By contrast, for horizontal-path Fernald retrievals, an upper “clean layer” analogous to that used in vertical profiling is generally unavailable. A calibration point must therefore be selected, typically near the far end of the range where the atmosphere is assumed to be relatively uniform. This step often relies on accurate segmentation of the lidar signal to identify quasi-uniform intervals [18,19,20]. The aerosol extinction at the calibration point is then estimated with a slope method and used to initialize the boundary condition for the Fernald inversion. When the signal-to-noise ratio at the calibration point is low, the logarithm of the range-corrected signal (RCS) becomes noise-dominated, biasing the slope estimate and propagating errors into the boundary condition and the retrieved extinction [21,22]. To mitigate this sensitivity, several iterative schemes have been proposed to estimate the boundary condition more reliably [23,24,25]. In summary, horizontal lidar retrievals are constrained by the need to select a calibration point and to specify a reliable boundary condition.

Here we propose a simple boundary-condition determination method for the Fernald inversion. The method exploits anomalous peaks that emerge during forward integration to infer the boundary condition, without requiring prior segmentation or strict homogeneity at the calibration location.

2. Principle and Methods

In this paper, the Fernald method is employed to retrieve the aerosol extinction coefficient. The backward integral formula of Fernald method is as follows:

$${\alpha }_a\left(r\right)={\displaystyle \frac{P\left(r\right)r^2\cdot \exp \left[2\left(S_a-S_m\right){\displaystyle {\int }_r^{r_c}{\beta }_m\left(r^{\prime }\right)d{r}^{\prime }}\right]}{\frac{P\left(r_c\right){r_c}^2}{S_{a}Rb\left(r_c\right){\beta }_m\left(r_c\right)}+2{\displaystyle {\int }_r^{r_c}P\left(r^{\prime }\right){r^{\prime }}^2\exp \left[2\left(S_a-S_m\right){\displaystyle {\int }_r^{r_c}{\beta }_m\left(r^{″}\right)d{r}^{″}}\right]d{r}^{\prime }}}}-S_a{\beta }_m\left(r\right)$$

(1)

The forward integral formula is:

$${\alpha }_a\left(r\right)={\displaystyle \frac{P\left(r\right)r^2\cdot \exp \left[2\left(S_a-S_m\right){\displaystyle {\int }_{r_c}^r{\beta }_m\left(r^{\prime }\right)d{r}^{\prime }}\right]}{\frac{P\left(r_c\right){r_c}^2}{S_{a}Rb\left(r_c\right){\beta }_m\left(r_c\right)}-2{\displaystyle {\int }_{r_c}^{r}P\left(r^{\prime }\right){r^{\prime }}^2\exp \left[-2\left(S_a-S_m\right){\displaystyle {\int }_{r_c}^r{\beta }_m\left(r^{″}\right)d{r}^{″}}\right]d{r}^{\prime }}}}-S_a{\beta }_m\left(r\right)$$

(2)

where ${\alpha }_a\left(r\right)$ is the aerosol extinction coefficient at range r; P(r) is the received signal; $S_a$ is the ratio of the aerosol extinction coefficient to the aerosol backscattering coefficient, i.e., the lidar ratio of aerosol, which is determined to be 50 Sr for the urban aerosol in this paper [26,27]; S_m is the lidar ratio of atmospheric molecules, taking a constant of 8π/3; β_m(r) is the atmospheric molecular backscattering coefficient, which is obtained from the atmospheric model; r_c is the range of calibration point; Rb(r_c) is the ratio of the total backscattering coefficient to the molecular backscattering coefficient at the calibration point, namely is the boundary value.

Typically, the calibration point is set at a height almost free of aerosols, with a boundary value of 1.01 for vertical lidar observations. Unfortunately, air near the ground is not clean, and the atmosphere at the calibration point is not always uniform enough to enable the calculation of boundary value using the Collis slope method. Therefore, for most lidar inversion algorithms, it is inevitably necessary to initially identify a signal segment characterized by uniform atmosphere, subsequently followed by the computation of boundary values [13,14].

We propose a straightforward approach based on the Fernald forward inversion result to determine the boundary value. This approach does not require atmospheric uniformity at the calibration point or the segmentation of the signal profiles. This method is based on the phenomenon that when the boundary value is slightly overestimated, the inverted aerosol extinction coefficient profile will exhibit a significant peak beyond the calibration point. Figure 1a illustrates the logarithmic profiles of the simulated range corrected lidar signal (RCS) under two cases: with noise and without noise. The calibration point is selected at 7.5 km, where the true boundary value of the backscattering ratio (Rb) is 1.9597. The boundary values for the inversion are set to 1.01 times the true values. Consequently, the resulting aerosol extinction coefficients yield a oscillatory peak amplitude exceeding 200 at 23.4 km, as shown by the green line (with noise) and the red line (without noise) in Figure 1b. The true aerosol extinction coefficients are also indicated by the black dashed line in Figure 1b. The reason for the occurrence of the oscillatory peak is that the denominator of the Fernald forward integration equation passes through zero at this distance. To better explain this phenomenon, we simplify Equation (2) as follows:

$${\alpha }_a\left(r\right)+S_a{\beta }_m\left(r\right)={\displaystyle \frac{X\left(r\right)}{C-2I\left(r_c,r\right)}}$$

(3)

where

$$X\left(r\right)=P\left(r\right)r^2\cdot \exp \left[2\left(S_a-S_m\right){\displaystyle {\int }_{r_c}^r{\beta }_m\left(r^{\prime }\right)d{r}^{\prime }}\right]$$

(4)

$$C={\displaystyle \frac{P\left(r_c\right){r_c}^2}{S_{a}Rb\left(r_c\right){\beta }_m\left(r_c\right)}}$$

(5)

$$I\left(r_c,r\right)={\displaystyle {\int }_{r_c}^{r}P\left(r^{\prime }\right){r^{\prime }}^2\exp \left[-2\left(S_a-S_m\right){\displaystyle {\int }_{r_c}^r{\beta }_m\left(r^{″}\right)d{r}^{″}}\right]d{r}^{\prime }}$$

(6)

Figure 2a,b illustrate the variations in terms C and 2I(r_c,r) as a function of distance for the noise-free signal and the noisy signal, respectively. It can be observed that when the boundary values are properly configured, C (solid red line in the figures) is consistently larger than 2I(r_c,r) (blue line in the figures), ensuring the denominator in Equation (3) remains positive at all ranges. However, when the boundary values are 1% larger than the true values, the C value becomes relatively low, as indicated by the dashed red line in the figures, and intersect with the 2I(r_c,r) curve. This causes the denominator in Equation (3) to pass through zero, thereby resulting in the oscillatory peak in the inversion results.

The impacts of boundary value variations on the zero cross bins of denominator in Equation (3) and the resulting inversion error are investigated. As the relative error of Rb at the calibration point (i.e., the boundary value) starts to increase from zero, the zero cross bin gradually shifts forward following a logarithmic law, as depicted in Figure 3a. Simultaneously, the inversion error increases accordingly. The path integrated aerosol extinction coefficient from the lidar to the calibration point is defined as AOD. Figure 3b shows that the relative error of AOD increases with that of Rb. Since the x-axis of Figure 3 adopts a logarithmic scale, the relative errors of AOD and Rb exhibit a near-linear relationship. Figure 3c,d are analogous to Figure 3a,b, illustrating the impacts of boundary value variations on inversion results derived from noisy signals. Affected by noise, the zero cross bin may become unstable when the error of boundary value is extremely small. Nevertheless, for typical lidar signals, once the relative error of boundary value exceeds 0.001, the oscillatory peak in the inversion results can be reliably identified, with the corresponding relative error of AOD is approximately 0.13%.

As can be inferred from the above discussion, when the oscillatory peak initially emerges, the selected boundary value is extremely close to the true value, thereby yielding minimal inversion error in front of the calibration point. The effective detection range of a typical aerosol lidar for horizontal measurements is only a few kilometers, thus it is not affected by distant oscillatory peaks. Furthermore, the valid measurement results are generally calculated from the backward integration formula of the Fernald equation (Equation (1)), which is convergent, meaning that the closer the detection distance from the lidar, the smaller the retrieval error induced by boundary value errors. Therefore, within the allowable signal-to-noise ratio (SNR) range, the calibration point should be positioned as far as possible to minimize near-field detection errors.

Figure 4 depicts the method to determine the boundary value at the calibration point via the oscillatory peak, as well as the corresponding inversion error induced by this method resulting from it. The maximum value extracted from the inverted aerosol extinction coefficient profile is defined as the “peak value of α_a”. Figure 4a illustrates that this peak value surges abnormally once the relative error of Rb slightly exceeds zero. Since the actual aerosol extinction coefficient does not exceed 10 under the vast majority of atmospheric conditions, the last point before the peak value exceeds 10 is defined as the boundary where the oscillatory peak initially emerges, corresponding to an Rb relative error of 2 × 10⁻⁴ in this case, and is marked with a red dashed line in the figure. Figure 4b shows the corresponding relative error of the AOD, where the relative error of AOD at the red dashed line is 8.5 × 10⁻⁵. With regard to the signal noise level, the relative error magnitude introduced by this method exhibits a certain degree of randomness; nevertheless, it is generally sufficient to meet the detection accuracy requirements. The data processing scheme is as shown in Figure 5, which includes four key steps that can be described as:

(1)

Using a signal-to-noise ratio (SNR) threshold of 3, determine the calibration point at the far end of the lidar signal [28,29].

(2)

Scan the boundary value starting from 1.0 with a large step size (such as 0.1), with the aerosol extinction coefficient computed via the Fernald method. The scanning stops when the maximum of the aerosol extinction coefficient exceeds the predefined threshold (set to 10 in this case). At this juncture, the scanned boundary value is assumed to be the i-th iteration, denoted as Rb(i).

(3)

Adopt Rb(i − 1) and Rb(i) as the new starting and ending points, reduce the step size by a factor of 10, then perform a boundary value scan, and repeat the procedure outlined in Step 2. This iterative process can be repeated multiple times until the accuracy requirement for the boundary value is met.

(4)

The last Rb for which the extinction coefficient does not exceed the threshold is defined as the final boundary value, which is then substituted into Equations (1) and (2) to retrieve the horizontal distribution of the aerosol extinction coefficient.

3. Results and Discussion

3.1. Comparison with Slope Method (Simulation Study)

Based on the noisy simulated lidar signal, the previously proposed method and the Collis slope method are employed respectively to derive the boundary values at the calibration points, thereby retrieving the aerosol extinction coefficients for comparison purpose. Figure 6a presents the logarithmic profile of the simulated RCS. Taking SNR > 3 as the threshold, the calibration point should be selected within 10.7 km. As a special case, the 981th bin (7.36 km) is chosen here as the calibration point, marked by a red circle in Figure 6a. For the Collis slope method, 100 data bins (corresponding to 750 m) centered at the calibration point are selected to perform linear fitting on the ln(RCS) signal, as shown by the red line in Figure 6a. The boundary value calculated from the slope of the fitted line is 2.152, which is overestimated. This leads to an overestimation of the retrieved aerosol extinction coefficient, as illustrated in Figure 6b, where the black line denotes the true input aerosol extinction coefficient. In contrast, the boundary value derived by the proposed new method is 1.411. The resulting retrieved aerosol extinction coefficient, depicted by the red line in Figure 6b, exhibits excellent agreement with the input true values. Figure 6c demonstrates that with increasing detection range, the retrieval error in the aerosol extinction coefficient derived from the slope method rises progressively and reaches 0.2 km⁻¹ at the calibration point. In contrast, the corresponding error for the proposed method remains within 0.08 km⁻¹, which is primarily attributed to random instrumental errors.

For the same signal, the 912th bin (6.84 km) is selected as the calibration point, as indicated by the red circle in Figure 7a. When processed using the Collis slope method (consistent with the previous approach), the slope of the fitted line is significantly reduced, as shown by the red line in Figure 7a. Under this condition, the calculated boundary value is 1.059, which results in an underestimation of the aerosol extinction coefficient, depicted by the blue line in Figure 7b. In contrast, the proposed new method yields a boundary value of 2.291. The resulting retrieved aerosol extinction coefficient which depicted by the red line in Figure 7b, still exhibits excellent agreement with the true values. Retrieval errors in the aerosol extinction coefficient derived from the slope method and the new method are presented in Figure 7c, with maximum values of −0.13 km⁻¹ and −0.06 km⁻¹, respectively, ahead of the calibration point.

Indeed, the theoretical boundary value—directly derived from the input aerosol extinction coefficient—is 1.960. The discrepancy between this theoretical value and the boundary values yielded by the proposed new method is primarily attributed to signal noise. Notably, this comparative experiment is somewhat biased against the slope method: data bins with signal jitter that deviates from the fitting line were intentionally selected as calibration points, which in turn results in relatively large errors in the resulting boundary values. Nevertheless, the comparative results demonstrate that the new method outperforms the slope method in suppressing the impacts of signal fluctuations and noise at the calibration points.

3.2. Scanning Observation Case Study

A series of experiments were performed in Lankao city, Henan Province (34°50′49.8″ N, 114°49′44.5″ E) in August 2025. Figure 8 presents a map of the lidar scanning area, with the lidar system deployed on the rooftop of a high-rise apartment building. The scanning azimuth ranged from 0° to 360° (where 0° corresponds to north, 90° corresponds to east, 180° corresponds to south, and 270° corresponds to west) with a angular resolution of 2°. The dwell time per angular step was 10 s, so the average scanning time of a cycle was about 32 min. Ultimately, a horizontal lidar map of aerosol distribution with a diameter of 10 km was acquired.

We applied the proposed method to process data throughout the entire scanning measurement campaign. In the data processing, we use the 680th data bin (5.1 km) as the fixed calibration point for retrieval, regardless of the atmospheric homogeneity at this point. Figure 9 presents the first six lidar-derived aerosol extinction coefficient maps obtained over the observation period spanning from the night of 4 August to the early morning of 5 August. The corresponding observation times for Figure 9a–f are 23:15, 23:46, 00:17, 00:49, 01:20, and 01:52 (UTC+8), respectively. Three distinct gaps can be seen at azimuth angles of 108°, 306°,and 318° in the lidar maps, resulting from obstruction of the laser beam by tall buildings. As shown in the figures, a distinct pollution source can be seen at approximately 3.8 km southeast of the lidar, where several furniture factories and building materials factories are located nearby. Figure 9a indicates that the pollutants were initially transported northwestward. Figure 9b–e illustrate that the transport direction gradually shifted northward, with the pollution intensity progressively increasing along the transport path. In Figure 9e, the pollutants had spread to the entire urban area and western suburbs, while the eastern suburbs remained relatively clean. Figure 9f shows that the pollution levels in the southwestern and southern suburbs were temporarily alleviated, as pollutants converged toward the urban area.

Figure 10 presents the temporal variation in aerosol extinction coefficients at three locations (points A, B and C, denoted in Figure 10b). Figure 10a shows that around 00:00 (UTC+8), the first aerosol extinction coefficient peak emerged at point A, followed by the peak at point B at approximately 01:30, and a peak at point C at roughly 03:30. The sequential occurrence of peaks at these three spatially separated points indicates that the pollution plume (the red area in Figure 10b) propagated slowly eastward. In terms of pollution severity, point B—located in the vicinity of the pollution plume—experienced the most severe pollution, exhibiting multiple distinct pollution peaks. In contrast, the eastern suburbs where point C is located remained relatively clean throughout the observation period.

4. Conclusions

Traditionally, applying the Fernald method to horizontal lidar data begins by identifying a segment that is approximately homogeneous and selecting its midpoint as the calibration point. The extinction coefficient at this point is then estimated using a slope method. The corresponding boundary condition is subsequently derived and used to initialize the Fernald inversion. Under complex atmospheric conditions, however, such a segment may be difficult to identify from the profile. Moreover, the slope-derived boundary condition effectively reflects only an average optical state near the calibration point and is sensitive to local atmospheric fluctuations and measurement noise, which can propagate into retrieval errors. Exploiting the fact that forward integration can diverge rapidly when the boundary condition slightly exceeds its optimal value, we propose a new algorithm to determine the boundary condition at the calibration point. Although the inferred boundary condition may deviate slightly from the optimum, the backward integration in the Fernald formulation is convergent. As a result, the near-range retrievals upstream of the calibration point are only weakly affected. Simulations show that, relative to the slope method, the proposed approach is more robust to atmospheric variability and random noise near the calibration point. We applied the algorithm to horizontal scanning observations and captured a pollutant transport and dispersion episode, supporting its use for real-time monitoring of urban-scale spatiotemporal pollution patterns.