Hybrid Atmospheric Modeling of Refractive Index Gradients in Long-Range TLS-Based Deformation Monitoring

Highlights

What are the main findings?

• Advanced physical modeling of refractive index gradients along the laser path and hybrid physical modeling with neural networks significantly improve TLS 3D point cloud accuracy compared to conventional physical modelling, from centimeter- to millimeter-level.
• Field experiments with two long-range scanners (Leica ScanStation P50 and Maptek I-Site 8820) at mine sites (ranges exceeding 800 m) and dam sites (steep vertical angles larger than 80° confirm consistent and reliable millimeter-level accuracy in 3D point coordinates across diverse atmospheric environments attached to the scanner setups.

What is the implication of the main finding?

• This is the first time that pathway-based atmospheric refraction modeling and hybrid physical–data-driven atmospheric refraction modeling have been introduced for long-range TLS-based deformation monitoring, enabling reliable detection of true geometric displacements.
• The improved millimeter-level accuracy strengthens TLS-based deformation monitoring and further supports critical applications requiring high-precision monitoring, such as in mining, dam safety, and structural health assessment.

Abstract

Terrestrial laser scanners (TLS) are widely used for deformation monitoring due to their ability to rapidly generate 3D point clouds. However, high-precision deliverables are increasingly required in TLS-based remote sensing applications to distinguish between measurement accuracies and actual geometric displacements. This study addresses the impact of atmospheric refraction, a primary source of systematic error in long-range terrestrial laser scanning, which causes laser beams to deviate from their theoretical path and intersect different object points on the target surface. A comprehensive study of two physical refractive index models (Ciddor and Closed Formula) is presented here, along with further developments on 3D spatial gradients of the refractive index. Field experiments were conducted using two long-range terrestrial laser scanners (Leica ScanStation P50 (Leica Geosystems, Heerbrugg, Switzerland) and Maptek I-Site 8820 (Maptek, Adelaide, Australia)) with reference back to a control network at two monitoring sites: a mine site for long-range measurements and a dam site for vertical angle measurements. The results demonstrate that, while conventional physical atmospheric models provide moderate improvement in accuracy, typically at the centimeter- or millimeter-level, the proposed advanced physical model—incorporating refractive index gradients—and the hybrid physical model—combining validated field results from the advanced model with a neural network algorithm—consistently achieve reliable millimeter-level accuracy in 3D point coordinates, by explicitly accounting for refractive index variations along the laser path. The robustness of these findings was further confirmed across different scanners and scanning environments.

1. Introduction

1.1. Problem Description

Terrestrial laser scanners (TLS) are active sensors capturing millions of points per second by evaluating the reflected signal from target surfaces at the TLS. To guarantee the high quality of TLS deliverables (3D point clouds), acquired observations must be assessed in terms of four systematic error sources: instrumental imperfections, atmospheric effects (refraction), target and surface-related parameters, and scanning geometry (reflectivity). It is reasonable that an understanding of the optical effects of the laser, refraction, and reflection, is necessary for designing a more rigorous calibration setup.

Since the emergence of terrestrial laser scanners, they have become crucial in performing engineering geodesy and deformation analysis tasks. For instance, due to safety reasons, the preferred technique of observation for unapproachable areas and high walls, especially in mining and structural sites, is the long-range terrestrial laser scanning method, typically within the 500 m–1000 m range or more. Long-range scanning demands detailed knowledge of the optical influences on the geometry of the line of sight to genuinely differentiate between the actual geometric displacements and TLS observations accuraciesaccuracies. Given the optical influences along the line of sight, geodetic refraction is the dominating effect. Other optical occurrences, especially reflectivity or scattering, are not addressed in this research, but those are future research topics into the TLS performance.

Geodetic refraction is a deviation of the signal from its direct line of travel due to the varying velocity of the wave propagating through different media of the atmosphere. Therefore, the signal follows the quickest path through the medium to reach its destination (the surface of the targets). A change in refraction might take place numerous times during the period of observations due to variations in atmospheric conditions. The atmospheric variables, such as air temperature, atmospheric pressure, relative humidity, etc., are predominantly contributing elements for geodetic refraction [1].

1.2. Significance and Purpose

The fundamental restriction in the determination of refraction is the non-uniformity of atmospheric conditions (i.e., its turbulence) over the optical path. In addition, no current technological advancement is capable of monitoring the variation in refractivevariation in refractive index with respect to the corresponding atmospheric conditions. Previously, observations of atmospheric conditions at both terminals of the sightline were acquired, and the mean calculation of the refractive index at the highest precision of [1 − 5] × 10⁻⁸ was employed to correct the measured range, to support millimeter- or sub-millimeter-accuracy of distance measurements over long baselines. For example, range corrections for the refractive index effects ∆r over 1000 m will be of the order of sub-millimeter if the refractive index is precise at the level of 1 × 10⁻⁸ (∆r = 1 × 10⁻⁸ × 1000 m = 0.01 mm. Therefore, ideally, this level of precision or better for refractive index estimation must be satisfied. This principle refers to the second velocity correction of the range measurements [1,2,3].

The current study, for the first time, introduces a generic atmospheric error model for long-range terrestrial laser scanning based on physical refractive index parameterization. Two physical refractive index models—the Ciddor and Closed Formula models—are presented, and further theoretical developments are revised to account for spatial variations in the refractive index-vertically and horizontally. This development results in nonlinear refractive index modeling along the sightline by incorporating varying spatial refractive index gradients within the assumed vertically stratified atmospheric layers, rather than relying on the conventional physical model that applies an average of refractive indices from both endpoints. The robustness of the methodologies is subsequently compared and validated using two geodetic monitoring datasets acquired with two long-range scanners: the Leica ScanStation P50 and the Maptek I-Site 8820. The first dataset, collected at a mine site, enabled long-range scanning exceeding 800 m, while the second, collected at a dam site, allowed extreme vertical viewing angles approaching the zenith (80°).

To achieve high-precision calibration results, both test fields were established within a calibrated network using post-processed GPS control points and onsite terrestrial surveys, delivering ±1 mm accuracy for the range and 1″ for the vertical angle observations. Additionally, in situ atmospheric observation methods were implemented in conjunction with each scan setup. A high precision for refractive index estimation 1 × 10⁻⁹ is sought before assessing the accuracy of 3D point coordinates. Accuracy assessments are completed by comparing the residuals between calibrated results and the control points. The results show the improvements in range accuracy—34% (from 3.6 mm to 2.4 mm) for the Leica ScanStation P50, and 16% (from 9.2 mm to 7.7 mm) for the Maptek I-Site 8820—as well as the improvements in vertical angle accuracy—44% (from 18″ to 10″) for the Leica ScanStation P50 and 20% (from 24″ to 19″) for the Maptek I-Site 8820. These are accomplished through the implementation of the advanced physical refractive index model, which addresses the nonlinearity of refractive index modeling along the line of sight by incorporating varying gradients of refractive index. Compared to the insignificant improvements obtained by the conventional physical models, these findings are substantial. Consequently, the 3D point coordinates exhibit maximum accuracy improvements of 41% and 18% for the Leica ScanStation P50 and Maptek I-Site 8820, respectively. The associated accuracies are reduced from the centimeter level to the millimeter level for 3D point coordinates through the implementation of the advanced approach. In addition, it is recognized that points at longer ranges (greater than ≈200 m) and/or at steep vertical angles (greater than ≈60°) benefit more from atmospheric corrections than points located at shorter ranges and/or close to the same horizontal plane as the scanner station.

The hybrid refractive index model, a data-driven approach that integrates neural network techniques with the advanced physical model, ensures further improvements in measurement accuracy—1.8 mm and 9″ for the Leica ScanStation P50, and 2.3 mm and 15″ for the Maptek I-Site 8820—consistently reducing 3D point coordinate errors to a reliable millimeter level. This enhancement is achieved by maximizing the precision of refractive index estimation while minimizing sensitivity to 3D spatial gradients, particularly vertical gradient, thus capturing real-world refractivity patterns along the propagation path. To conclude, the robustness of these findings—based on a comparison of three methods: conventional and advanced physical refractive index models versus the hybrid refractive index model—was further validated across different scanners and varying scanning environments, consistently achieving millimeter-level accuracy in 3D point coordinates.

2. Literature Review

2.1. TLS Principle

TLS is a very high-speed and movable total station that is able to capture millions of points in a second as a consequence of measuring three spherical coordinates, range r, vertical angle v and horizontal angle h from a returned signal reflected from a single point. The mathematical conversion from 3D spherical coordinates [r_p v_p h_p] into Cartesian coordinates [x_p y_p z_p] is applied as below (i.e., p is the number of measured points from 1 to n) [4] as follows

$${\left[\begin{array}{c}{x}_p\\ y_p\\ z_p\end{array}\right]}_{p=1\dots n}={\left[\begin{array}{c}{r}_p\cos v_p\cos h_p\\ r_p\cos v_p\sin h_p\\ r_p\sin v_p\end{array}\right]}_{p=1\dots n},$$

(1)

Afterwards, the projection is required from the scanner coordinate system j to object coordinate system into space [X_P Y_P Z_P] using the rotation matrix M and translation parameters of [X_S Y_S Z_S] as follows

$$\left[\begin{array}{c}{x}_p^j\\ y_p^j\\ z_p^j\end{array}\right]=M·\left(\left[\begin{array}{c}{X}_P\\ Y_P\\ Z_P\end{array}\right]-\left[\begin{array}{c}{X}_S\\ Y_S\\ Z_S\end{array}\right]\right),$$

(2)

There are three types of range measurement techniques used by TLS: time of flight (TOF), phase-based, and waveform digitizer (WFD) (Figure 1). The TOF method is a technique in which time plays an integral role in capturing the range (the time is recorded between emitting the signal pulse from the instrument and receiving the same pulse at the instrument). TOF uses either pulsed modulation or continuous wave modulation (CW). Whereas the phase-based method conveys the data by modulation of the phase of the signal (i.e., the range is determined by the shift in phase between the emitted and received signal, considering the number of full wavelengths). The main distinction between TOF and the phase-based principle is that the phase-based technique provides more information to evaluate the entire signal, including signal shape, channel amplification, etc., which enables more accurate distance determination compared to TOF. However, TOF is appropriate for long-range measurement with less accurate deliverables due to the expansion of the laser spot size within a range. WFD is the advanced combination of both techniques, in which the time between a start and stop pulse is calculated, and the entire received signal is digitized [5].

Additionally, for digital angle recordings of laser scanners, various technologies have been introduced: camera, hybrid, and panoramic. Those are classified based on varying horizontal and vertical fields-of-view (Figure 2). Then, to determine vertical and horizontal angles for panoramic scanners, the following equations must be used to convert the Cartesian coordinates to spherical coordinates: $v=\pi -{\mathit{tan}}^{-1}({\displaystyle \frac{z}{\sqrt{x^2+y^2}}})$ and $h={\mathit{tan}}^{-1}({\displaystyle \frac{y}{x}})-\pi$ [6,7].

Figure 2. Different techniques for the angle measurements ((a) camera, (b) hybrid, and (c) panoramic scanner) [8].

Figure 2. Different techniques for the angle measurements ((a) camera, (b) hybrid, and (c) panoramic scanner) [8].

In principle, the electromagnetic wave (EM) is the main source of illumination for the TLS measurements (i.e., the employed domain for wavelength is typically from visible to far infrared (400 nm to 2000 nm)). Therefore, the accuracy of observations highly depends on the accuracy of the propagating wavelength and its velocity through the traveling time period. Electromagnetic waves are generally described by wavelength λ in m, frequency f in Hz or s⁻¹ and the propagation velocity c in [1]. The relationship is expressed based on the following:

$$\lambda ={\displaystyle \frac{c}{f}},$$

(3)

EM in air is influenced by several atmospheric conditions, with the most prominent ones being the variation in air temperature, atmospheric pressure of air, water vapor in air (e.g., humidity), the effects of carbon dioxide content, etc. Those environmental influences change the direction of the line of sight from the chord (the corrected line of the straight sightline) to the actual ray path, resulting in a range and vertical angle deviation.

Figure 3 shows that the deflection from the chord to the actual ray results in different observations on the surface of scanned objects for the range and potentially for the angle observations from the perspective of the apparent ray (i.e., dv is the potential deviated vertical angle). In optics, this phenomenon is defined by Fermat’s principle and Snell’s law as the refraction of a wave. Then, the relationship of the propagating velocity of EM in air c_air compared to the velocity of the identical wave in a vacuum is indicated as the refractive index n (the index is dimensionless).

$$n={\displaystyle \frac{c_{vacuum}}{c_{air}}},$$

(4)

Figure 3. Geodetic refraction over the line of sight [9].

Figure 3. Geodetic refraction over the line of sight [9].

And refractivity N (without the metric unit) can be explained as follows [1,3]:

$$N=(n-1)\times {10}^6,$$

(5)

The type of refraction shown in Figure 3 is a convex condition. When the refractive index from the first to the second medium drops (n₂ < n₁), it refers to a convex condition; otherwise, if the refractive index rises from the first to the second medium (n₂ > n₁), the refraction condition is concave [10].

2.2. Review and Results on Mathematical Developments of Refraction

The difficulty of mitigating the relevant impact of refractive index is the non-uniformity of atmospheric conditions (i.e., the turbulence of refractivity) over the optical path. To deal with this limitation, two methodologies were presented in the literature: the direct method and the indirect method. The direct method of refractive index observations is implemented using an interference refractometer [11]. This technique brings some disadvantages despite the direct measurement of the refractive index. One of the disadvantages is that the refractive index is determined independently of atmospheric variations in the air (i.e., it does not physically reflect the real condition of the atmosphere). Moreover, the internal calibration is required to align the initial crude measurement relative to the length of the refractometer cell, which introduces more computational efforts [11]. The alternative method is the indirect method. In this technique, the refractive index is derived from the measurement of atmospheric parameters. Thus, to achieve the optimum performance of the index, it is recommended that all atmospheric conditions are precisely obtained at both terminals of the line, and the mean calculation for refractive index is determined at least at the precision level of [1 − 5] × 10⁻⁸ to guarantee millimeter- or sub-millimeter-precision range [1].

Mathematical developments of refractive index with respect to the atmospheric variables have been updated regularly since the 19th century. The index has been repeatedly studied and investigated by different scientists to achieve better results (i.e., the studies are listed by years [11,12,13,14,15,16,17,18,19,20,21,22,23], the model adopted by the International Association of Geodesy is called the Closed Formula in 1999 [24,25,26]). Under each notion, at least three elements of atmospheric conditions have been considered as follows: air temperature, atmospheric pressure, and relative humidity.

M. Sabzali et al. [27] conducted extensive research on the proposed models of refractivity and investigated the impacts of refraction on the simulated and real datasets for the range observations. The aim of the work was to update atmospheric modeling for the range measurements. Their findings were validated over existing range measurement techniques: TOF and phase shift principles. After improving the atmospheric error model, the variation in the reference index dn over the entire line of path is verified. In Equation (6), dn is the comparison is between two actual atmospheric conditions (obtained from two varying media):

$$dn\times {10}^6=-0.93dT+0.28dP-0.039de,$$

(6)

where each change in air temperature either dT (K) or dt (°C), atmospheric pressure dP (hPa) and partial water vapor pressure (humidity) de (hPa) respectively introduces −0.93 ppm, +0.28 ppm and −0.039 ppm refraction over the measured range r_o [24].

The impact of refractivity on the angle measurements—vertical and horizontal angle—was also separately studied (Figure 4).

Figure 4. Spatial gradients of refractive index (a) vertical refraction on z plane and (b) horizontal refraction on xy plane (convex condition) [10,28].

Figure 4. Spatial gradients of refractive index (a) vertical refraction on z plane and (b) horizontal refraction on xy plane (convex condition) [10,28].

The improved error models are modified for the refracted vertical angle dv and the refracted horizontal angle dh based on the experimental analysis in the following [29,30]:

$$dv=\left[0.00026dT-0.00004dP\right]·r_o,$$

(7)

$$dh=\left[0.001\times {10}^{-6}dT+1.7\times {10}^{-6}dP\right]·r_o,$$

(8)

Here, a single unit rise in temperature and pressure between the mediums, respectively, leads to 0.26″ and −0.04″ refraction in measured vertical angles within the assumed range, observation of 1000 m. However, the variations for the horizontal angles are considerably smaller (i.e., given the insignificant effect of humidity on angle measurements). Further, M. Sabzali. et al. [30] acknowledged that the effects of refraction on vertical angles for points located close to nadir and zenith are maximum due to the sensitivity between vertical and horizontal gradients of refractions.

3. Methods

The quantification of refractive index n along the line of sight must be precisely addressed when the beam ray experiences different refraction conditions with respect to the atmospheric variations in each medium. This might occur several times during the time of the measurement. Therefore, the error model for optimization of the systematic error caused by atmospheric effects, from Equation (1), can be rewritten as follows:

$${\left[\begin{array}{c}{x}_p\\ y_p\\ z_p\end{array}\right]}_{p=1\dots n}={\left[\begin{array}{c}({r+dr)}_p\cos ({v+dv)}_p\cos {(h+dh)}_p\\ {(r+dr)}_p\cos ({v+dv)}_p\sin ({h+dh)}_p\\ {(r+\mathit{dr})}_p\sin ({v+dv)}_p\end{array}\right]}_{p=1\dots n},$$

(9)

where [dr dv dh] refers to the refracted range, refracted vertical angle, and refracted horizontal angle, respectively. To reduce the complexity of the problem, it was suggested that accurate observations of atmospheric conditions at both terminals of the sight line be measured, and then the mean calculation of the refractive index be applied as the real refractive index [1]. These correction factors are referred to as the first velocity correction, and their comparison with the reference refractive index n_ref is the second velocity correction for range measurements. The second velocity correction is typically implemented in order to ensure the required precision for estimation of refractive index (at least better than [1 – 8] × 10⁻⁸). Note, in most cases, for either end, the defaulted value within the instrument is assumed as the reference refractive index [3,31]. Since TOF is the scanning mechanism behind long-range scanners for range measurements, the first and second velocity correction factors must be simultaneously substantiated after precise observations of atmospheric variables, through a physical refractive index model. In addition, the corresponding infrared wavelength is particularly larger than 900~1000 nm which would be ideal for long-range detection due to the larger spot size with the increase in range.

As discussed earlier, there are a number of physical refractive index models. Among all sets of calculations, Ciddor’s parameterizations provide more robust results than the previous versions [22,23]. Under this formulation, it was noticed that this setup is appropriate over a broader range of wavelengths (within 300 nm–1690 nm). Additionally, Ciddor’s physical model supports more flexibility under extreme environmental conditions [26,27]. To implement Ciddor’s refraction model, the following three steps must be implemented:

1.

The first step is to differentiate the phase refractive indices (To differentiate the group refractive index n_g and the phase refractive index n_ph, the group refractive index determines the speed at which energy or information travels through a medium, while phase refractive index governs the propagation of individual wavefronts. Those can be simply converted using the following equation [26]:)

$$n_g=n_{ph}+\sigma \left({\displaystyle \frac{d{n}_{ph}}{d\sigma }}\right),$$

(10)

of standard air (The standard air condition was defined at t = 15 °C, P = 1007 hPa, and e = 13 hPa by Reuger in 1990 for analytical tasks [1] (i.e., corresponding refractivity for standard air condition is N_st = 304.5).) n_st and water vapor n_wv as the function of the wavelength and irrespective of atmospheric variables as follows:

$${N_{st}=\left(n_{st}-1\right)\times 10}^8={\displaystyle \frac{a_1}{a_2-{\sigma }^2}}+{\displaystyle \frac{a_3}{a_4-{\sigma }^2}},$$

(11)

Here, the wave number σ in μm⁻¹ is the reciprocal of wavelength λ (μm).

The empirical coefficients are listed in

Table A1. Empirical coefficients for physical refractive index models [22,23,24].

Empirical Coefficients	Constant Values
a1	5,792,105
a2	238.0185
a3	167,917
a4	57.362
a5	295.235
a6	2.6422
a7	−0.03238
a8	0.004028
a9	7.9266
a10	0.1619
a11	0.028196
a12	1.58123 × 10−6
a13	−2.9331 × 10−8
a14	1.1043 × 10−10
a15	5.707 × 10−6
a16	−2.051 × 10−8
a17	1.9898 × 10−4
a18	−2.376 × 10−6
a19	1.83 × 10−11
a20	−0.765 × 10−8
b1	287.6155
b2	4.8866
b3	0.068
b4	273.15
b5	1013.25
b6	11.27

in Appendix A.

$$N_{wv}={\left(n_{wv}-1\right)\times 10}^8=cf\left(a_5+a_6{\sigma }^2-a_7{\sigma }^4+a_8{\sigma }^6\right),$$

(12)

The correction factor cf = 1.022 is considered (dimensionless).

The group refractive indices are also computed as follows:

$${N_{st}=\left(n_{st}-1\right)\times 10}^8={\displaystyle \frac{a_1 \left(a_2+{\sigma }^2\right)}{{\left(a_2-{\sigma }^2\right)}^2}}+{\displaystyle \frac{ a_3(a_4+{\sigma }^2)}{{(a_4-{\sigma }^2)}^2}},$$

(13)

$$N_{wv}={\left(n_{wv}-1\right)\times 10}^8=cf\left(a_5+a_9{\sigma }^2-a_{10}{\sigma }^4+a_{11}{\sigma }^6\right),$$

(14)

2.

Next step is to compute the refractive index based on the atmospheric conditions, density components of the dry air p_axs and p_a, and the moist air p_wv and p_w with corresponding values of compressibility of air COM as follows:

$${\rho }_{axs}={\rho }_{wv}={\displaystyle \frac{P{M}_a}{\left(COM\right)R T}}\left(1-x_w\left(1-{\displaystyle \frac{M_w}{M_a}}\right)\right),$$

(15)

where T is temperature K, P is the pressure (hPa), x_w is the water vapor pressure component of the air, depending on the humidity (hPa) (as three major atmospheric components for this contribution), Ma is the molar mass of water vapor containing x_c ppm of CO₂ kg/mol, Mw is the molar mass of water vapor (=0.018015 kg/mol), and R is the gas constant (=8.314651 Jmol⁻¹K⁻¹). Then, compressibility COM based on each air condition—either standard dry air or pure water vapor—are computed:

$$COM=1-\left({\displaystyle \frac{P}{T}}\right)\left[a_{12}+a_{13}t+a_{14}{t}^2+\left(a_{15}+a_{16}t\right)x_w+\left(a_{17}+a_{18}t\right)x_w^2+{\left({\displaystyle \frac{P}{T}}\right)}^2\left(a_{19}+a_{20}{x}_w\right)\right],$$

(16)

$${\rho }_{w }={\displaystyle \frac{P{M}_w{x}_w}{\left(COM\right)RT}},$$

(17)

$${\rho }_{a }={\displaystyle \frac{P{M}_a{(1-x}_w)}{\left(COM\right)RT}},$$

(18)

Here, t is the temperature in °C (t = T − 273.15).

3.

Ultimately, the combined evaluation of both refractive indices, under dry air and water vapor components, is determined by the following [22,23]:

$$N_{ciddor}=(n_{ciddor}-1) {\times 10}^8=\left({\displaystyle \frac{{\rho }_{a }}{{\rho }_{axs}}}\right)\left(n_{st}-1\right)+\left({\displaystyle \frac{{\rho }_{w }}{{\rho }_{wv}}}\right)\left(n_{wv}-1\right),$$

(19)

Ciddor’s parameterizations have been later adopted by the International Association of Geodesy (IAG) in 1999, as the standard equation for calculating the index of refraction for geodetic instruments operating within the visible and near-infrared waves [24]. The principle is referred to as the Closed Formula model. IAG’s proposal provides more accurate results under more extreme temperature, pressure and humidity conditions through simplification of Ciddor’s principles and less computational skills. Thus, the methods to achieve the group refractive index n_g as a function of λ in μm is straightforward as follows:

$$N_g=\left(n_g-1\right)\times {10}^6=b_1+{\displaystyle \frac{b_2}{{\lambda }^2}}+{\displaystyle \frac{b_3}{{\lambda }^4}},$$

(20)

Afterwards, the group refractive index n_iag under either standard air condition or water vapor is computed as follows:

$$N_{iag}=(n_{iag}-1)\times {10}^6={\displaystyle \frac{b_4 N_{g}P}{b_{5}T}}-{\displaystyle \frac{b_6 e}{T}},$$

(21)

All empirical coefficients are listed in Table A1 in Appendix A.

Given each model, either the Ciddor or Closed Formula, three spatial variations in the refractive index n(x, y, z) in 3D Cartesian coordinates can be parameterised as the gradient of the refractive index ∇n as follows:

$$\nabla n={\displaystyle \frac{\partial n}{\partial x}}\overrightarrow{i}+{\displaystyle \frac{\partial n}{\partial y}}\overrightarrow{j}+{\displaystyle \frac{\partial n}{\partial z}}\overrightarrow{k},$$

(22)

The elements on the horizontal plane [∂n⁄∂x ∂n⁄∂y] affecting the horizontal directions are called horizontal gradients of refraction, while ∂n⁄∂z refers to the vertical gradient of refractive index impacting the vertical directions. Therefore, the gradient of refractive index is rewritten as a function of the following atmospheric variables:

$$\nabla n=\left({\displaystyle \frac{\partial n}{\partial T}}\nabla T+{\displaystyle \frac{\partial n}{\partial P}}\nabla P+{\displaystyle \frac{\partial n}{\partial e}}\nabla e\right),$$

(23)

where, $\nabla T=\left[\begin{array}{ccc}{\displaystyle \frac{\partial T}{\partial x}}& {\displaystyle \frac{\partial T}{\partial y}}& {\displaystyle \frac{\partial T}{\partial z}}\end{array}\right] (\mathrm{K}/\mathrm{m}$or$℃/\mathrm{m}),\nabla P=\left[\begin{array}{ccc}{\displaystyle \frac{\partial P}{\partial x}}& {\displaystyle \frac{\partial P}{\partial y}}& {\displaystyle \frac{\partial P}{\partial z}}\end{array}\right] (\mathrm{h}\mathrm{P}\mathrm{a}/\mathrm{m})$ and $\nabla e=\left[\begin{array}{ccc}{\displaystyle \frac{\partial e}{\partial x}}& {\displaystyle \frac{\partial e}{\partial y}}& {\displaystyle \frac{\partial e}{\partial z}}\end{array}\right] (\mathrm{h}\mathrm{P}\mathrm{a}/\mathrm{m})$ are horizontal and vertical gradients of air temperature, atmospheric pressure and the humidity of the air, respectively [32].

To investigate different gradient components of the refractive index, the stable stratification of the atmosphere-based on air temperature—is sometimes supposed in analytical studies. The stable stratification condition is defined when the air temperature decreases gradually with height. In contrast, unstable stratification occurs when the air temperature decreases rapidly with height, or changes irregularly with height. The vertical temperature gradient is generally described as the variation in temperature vertically under stable stratification conditions. There have been several measurement techniques presented by different scientists throughout the years to estimate the vertical temperature gradient under controlled laboratory conditions. More methods and equipment deployments were presented in [29]. However, one of the important principles that can be employed under real atmospheric conditions is the distribution of the temperature sensors on the rod. Normally, the height of the rod is 3 m, where it was assumed that within this height range of ground level, the maximum variation in atmospheric conditions are expected. The measurement technique is completed by using three temperature sensors t₁, t₂ and t₃ at different heights arranged on the rod above the ground level at the corresponding h₁, h₂ and h₃.

In the following studies [29,33], different layers of the atmosphere—from 0 m (directly above the ground) to over 100 m—were categorized. The classifications were arranged with respect to the height from ground level and the corresponding practices for computing the vertical temperature gradient accordingly. In summary, vertical temperature gradients are substantially intense in the layers close to the ground surface within a range of 0 to 3 m (between −47 K/m and +20 K/m) and drop to the small value −0.006 K/m at the highest assumed level [29]. The results indicate that the lower atmospheric layers contribute most significantly to refractive index variations, impacting the importance of accurately capturing the vertical temperature gradients near the ground surface.

Alternatively, the variation in temperature horizontally or laterally, impacting the horizontal direction of refractive index, is the horizontal temperature gradient. There have been several methodologies to determine the horizontal temperature gradient under real-world atmospheric conditions. For example, B. G. Bomford [2] assumed that for approximate distances of 1000 m horizontally and 3000 m vertically, with 5 K temperature rise between the terminals, the difference in the average horizontal temperature gradient is nearly 0.005 K/m (i.e., the increase of 5 K per kilometer in lateral difference). Another theory in the lowest layer of the atmosphere—where a line of sight is two meters above the ground level, with the vertical temperature gradient of 0.3 K/m—the horizontal temperature gradient is checked, and it is assumed to be negligible.

Comparing these two gradients, the effect of atmospheric variables horizontally is trivial. Therefore, Equation (24) depicts the relationship between the vertical gradient of refractive index ${\displaystyle \frac{\partial n}{\partial z}}$and the vertical temperature gradient ${\displaystyle \frac{\partial T}{\partial z}}$ as well as the vertical pressure gradient ${\displaystyle \frac{\partial P}{\partial z}}$and vertical humidity gradient ${\displaystyle \frac{\partial e}{\partial z}}$ [27] as follows:

$${\displaystyle \frac{\partial n}{\partial z}}=\left[{\displaystyle \frac{11.27e}{T}}-{\displaystyle \frac{76 P}{T^2}}\right]{\displaystyle \frac{\partial T}{\partial z}}+{\displaystyle \frac{76}{T}}{\displaystyle \frac{\partial P}{\partial z}}+-{\displaystyle \frac{11.27}{T}}{\displaystyle \frac{\partial e}{\partial z}}$$

(24)

As discussed earlier, the primary contributor to the vertical gradient of refraction is the vertical temperature gradient, with minimal influence from other vertical gradients. Further developments regarding the pressure and humidity vertical gradients are presented in Appendix A.2.

Consequently, under these assumptions, the corrected range r_c and corrected vertical angle v_c can be expressed in terms of the refracted range dr and the refracted vertical angle dv, respectively. These are derived by integrating of the refractive index effects over the entire length of actual ray (observed range r_o) (Figure 1) as follows:

$$r_c={\int }_o^{r_o}{\displaystyle \frac{dr}{n\left(z\right)}},$$

(25)

$$v_c=dv+v_o={\int }_o^{r_o}{\displaystyle \frac{1}{n}}{\displaystyle \frac{dn}{dz}}dr+v_o,$$

(26)

Also, a refracted horizontal angle dh is reparametrized in terms of the horizontal gradient of refractive index ${\displaystyle \frac{dn}{dy}}$ as follows:

$$dh= {\int }_o^{r_o}{\displaystyle \frac{1}{n}}{\displaystyle \frac{dn}{dy}}dr,$$

(27)

Accordingly, to achieve an optimal performance of the refractive index correction through a physical model, the refractive index is approximated by the averaged values obtained at both terminals of the line of sight z₀ and z₁ [1]. This means that refractive index is expressed as follows (Section 5.1):

$$n\left(z\right)\approx {\displaystyle \frac{1}{2}}\left[n\left(z_0\right)+n\left(z_1\right)\right],$$

(28)

However, the imposed simplification underestimates the potential nonlinear vertical variations in the atmosphere along the actual ray path, as the laser beam experiences multiple refraction abnormalities when traversing different atmospheric layers. Therefore, in this research, a more accurate physical model is proposed to account for the varying vertical gradients of the refractive index along the propagation path—referred to as the advanced physical refractive index model. According to Equation (25), it can be rewritten in the following expression:

$$r_c={\int }_o^{r_o}{\displaystyle \frac{dr}{n\left(z\left(r\right)\right)}},$$

(29)

where z(r) represents the height profile along the laser path, with r being the counter that moves between 0 and r_o. The refractive index is approximated as a nonlinear function of the range rather than applying the mean value between two refractive indices as follows:

$$n\left(z\left(r\right)\right)\approx n\left[z_0+{\displaystyle \frac{z_1-z_0}{r_o}} r\right],$$

(30)

Therefore, from Equation (26), the refracted vertical angle dv can be as follows (Section 5.2):

$$dv= {\int }_o^{r_o}{\displaystyle \frac{1}{n\left(z\left(r\right)\right)}}{\displaystyle \frac{dn}{dz}}dr,$$

(31)

Note, in either case, to guarantee milimetre- or sub-milimetre-accuracy for the observations, a precision of at least [1 − 5] × 10⁻⁸ must be accomplished for the estimation of the refractive index [1]. Then, it enables reducing sensitivity to the spatial variations in the refractive index and enhancing the overall accuracy of 3D point coordinates.

To represent refractivity along the entire line of sight, the illustrated techniques on a physical model establish a reasonable relationship between environmental parameters (e.g., in situ atmospheric recordings) and wave number (i.e., scanner wavelength) as the inputs, and the real-world refractivity along the beam path as the output. However, to achieve rigorous precision and consistency in TLS-based physical refractive index modeling, a hybrid physical-data-driven model is proposed as a follow-up. This hybrid model integrates results from the advanced physical model with a neural network approach, and its function is justified through field-validated outcomes from previously established physical models. It guarantees optimal millimeter-level accuracy in 3D point coordinates and enables consistency checks across two long-range scanners under varying atmospheric conditions (Section 5.3).

In short, a neural network is a machine learning algorithm simulated from the structure of the human brain. It generally consists of a variety of layers of interconnected neurons, where each one collects the inputs and interacts with the result of the next consecutive layers to generate rigorous outputs. These neurons utilize certain mathematical algorithms and adjust internal weights during each training interval to minimize the prediction errors for both datasets, according to the received residuals [34]. The minimization of residuals of 3D spherical coordinates signifies the lowest ultimate accuracy for the 3D point coordinates (on the order of milimetre or sub-milimetre relative precision). It ultimately ensures the robust prediction by the most accurate output-refractive index and its spatial gradients along the laser path.

In summary, three different methods are tested to improve calibration accuracy (

Table 1. Proposed methods for refractive index modeling in TLS-based applications.

Methods	Refractive Index Models	Approaches
Conventional physical model (Section 5.1)	Ciddor and Closed Formula	Average of refractive indices from both terminals (linear)
Advanced physical model (Section 5.2)	Developed Ciddor	Incorporating varying vertical refractive indices (nonlinear)
Hybrid physical model (Section 5.3)	Developed Ciddor and Neural Network	Combination of the results from an advanced model with a neural network (data-driven)

). To achieve high-precision calibration setups, two geodetic test fields, a mine site and dam site, were established within a calibrated network using post-processed GPS control points and onsite terrestrial surveys, delivering ±1 mm accuracy in range and 1″ in vertical angle observations. In addition, in situ atmospheric observations were collected at each scan station to improve refractive index modeling. Using the proposed approaches, refractive index estimation with high precision is implemented before 3D point coordinate accuracy assessments.

4. Data Experiments

The above theoretical developments were tested on real case studies acquired from a mine site and a dam site (Figure 5). The mine site experimental test field examines long-range scanning with a maximum range of 846.304 m, while the dam site experimental test field provides the flexibility for investigation of a steep vertical angle from the bottom of the dam to the dam crest (maximum vertical angle captured on-site is 80°4′22″). At the mine site, the data field capture was set up within a calibrated network using eight GPS control points distributed at different elevations across the site (red points shown in Figure 5). The reason for distributing the control points at varying heights is to investigate the varying vertical gradient of refractive indices across different horizontal stratifications of the atmosphere for the advanced hybrid model (e.g., 74.936 m for Station 1, 84.803 m for Station 2, and 128.531 m for Station 3) (

Table A2. GPS control points after the post processing (dataset 1: mine site).

Control Point Number	X	Y	Z
1 (station)	4925.017	2129.698	74.936
2 (station)	5149.091	1760.136	84.803
3 (station)	4818.470	2537.959	128.531
4 (base)	4933.628	2187.957	76.004
5 (base)	4878.079	2353.563	78.715
6 (base)	4881.232	2337.204	80.758
7 (base)	4910.580	2022.894	86.111
8 (base)	4982.600	1986.988	77.782

in Appendix A.3). At the dam site, a Leica Nova MS60 MultiStation (Leica Geosystems, Heerbrugg, Switzerland) was used to measure 14 black and white targets established on the semi-vertical dam walls (i.e., range and angular accuracies of the Leica MS60 are 1 mm +1.5 ppm and 1″, respectively (https://leica-geosystems.com/products/total-stations/multistation/leica-nova-ms60, accessed on 20 May 2025)).

Furthermore, GPS control points for the mine site were collected on-site using static mode and have been post-processed after the field collection at the office to achieve the highest accuracy within 1 to 5 mm. GPS control coordinates and survey control marks for both datasets are listed in Table A2 and

Table A3. Average onsite survey control points (dataset 2: dam site).

Control Point Number	X	Y	Z
1	−16.357	13.571	121.442
2	−142.955	−42.874	143.190
3	−142.975	−42.874	143.190
4	−96.051	70.568	155.164
5	−96.071	70.568	155.164
6	−96.091	70.568	155.164
7	−145.998	−33.878	161.536
8	−145.938	−33.878	161.536
9	−145.958	−33.878	161.536
10	−145.978	−33.878	161.536
11	−145.998	−33.878	161.536
12	−145.918	−33.878	161.536
13	−149.511	−35.227	164.691
14	−149.531	−35.227	164.691

in Appendix A.3, respectively. For scanning, two long-range scanners-Leica ScanStation P50 (measurement range 1000 m) and Maptek I-Site 8820 (measurement range 2000 m) were employed (Figure 6), and

Table 2. Scanner specifications and undertaken scanning characteristics.

Specifications (Scanner and Scanning)		Leica Scanstation P50 1	Maptek I-Site 8820 2
Accuracy	Range	3 mm +10 ppm (over full range 570 m/>1 km) 1.2 mm+10 ppm (over full range 120 m)	6 mm
	Angle	8″	12″
Maximum possible range of scanning		1000 m	2000 m
Wavelength *		1550 nm	1550 nm
Measurement techniques	Range *	TOF	TOF
	Angle	Panoramic	Hybrid
Field-of-view	Vertical	290°	160°
	Horizontal	360°	360°
Instrumental resolution		6.3 mm at 10 m	fine resolution
Time per scan		25 min	35 min

* Long-range terrestrial laser scanners typically operate at longer wavelengths using a time-of-flight (TOF) measurement mechanism. It is possible to capture the distances over extended ranges due to a larger beam footprint, despite limited range accuracy. ¹ https://leica-geosystems.com/products/laser%20scanners/scanners/leica-scanstation-p50 (accessed on 20 May 2025). ² https://www.maptek.com/featured-news/introducing-maptek-sentry-and-i-site_8820/ (accessed on 20 May 2025)

indicates the technical specifications of the scanners, reported by the manufacturers, and contains the scanning characteristics used in this research.

The datasets from three nominal scanner stations were captured under identical field instructions on 10 December 2024 (mine site) and 15 February 2025 (dam site), during working hours from 8:00 to 17:00. During scanning, long-range mode within the scanners was activated, and all default correction factors, including instrumental atmospheric refraction, were switched off. At least two scans from each station with the maximum possible instrumental resolution were acquired (Table 2) (i.e., each with a different horizontal orientation).

The environmental conditions of the sites were precisely recorded during scanning time using a Kerstal 2500 weather meter sensor (Kestrel Instruments, Boothwyn, PA, USA). The reported precisions for the temperature and pressure are 0.5 °C and 1.5 hPa, respectively. The link regarding the technical provisions of the thermometer was provided (https://kestrelinstruments.com/kestrel-2500-pocket-weather-meter?srsltid=AfmBOoplhrdnrU13HBFQOwqExIAqWxdPGwV9IYr1ByrsDTOy6oNU04jE, accessed on 20 May 2025). Figure 7 shows two variant temperature recordings across two geodetic sites. The reason for this difference is to validate the robustness of the proposed refractive index methods under varying atmospheric conditions for further reproducible generalization.

To optimally comprehend the level of atmospheric variations across the field, the atmospheric recordings were achieved by at least two weather meter sensors, and the measurements were initiated from the surroundings of each scanner station and continued onsite close to every target location—from the nearest to furthest target location—during scanning time. For example, at 9 am when the scanner was set up at the first station, the observed temperature close to the station was 43 °C at the mine site. Across the entire test field, the temperature varied by ±2 °C relative to the station’s record (i.e., the atmospheric data attached to each scanner station). By 3 pm, however, the temperature had increased to 46 °C across the whole site. In addition to temperature recordings, the atmospheric pressure of both sites was observed using the same sensor. Compared to temperature, the atmospheric pressure across the areas was considerably stable during the observation time (1009 hPa and 1012 hPa for the mine and dam site, respectively).

5. Results and Analysis

The datasets from both scanners were collected in the calibrated test field under uncontrolled environmental conditions. For the mine site dataset, the range consistency calibration method is implemented, while for the dam site dataset, on-site calibrated angle measurements using terrestrial surveying are undertaken. In short, the range consistency method provides the geometric accuracy as the result of verifying that the ranges between each pair of corresponding control points remain invariant across multiple scanner stations [35,36,37]. The existence of eight control points for the mine site dataset delivers 28 Euclidean ranges for each station, and 14 targets for the dam site dataset offer the same number of angles per scanner setup. Total redundancy for each dataset is six times those numbers (depending on the number of scans). Then, the following step is the accuracy assessment of different proposed physical models in long-range terrestrial laser scanning.

After importing scanned data for each instrument separately using Maptek PointStudio 2024.1.1 (https://www.maptek.com/products/pointstudio/, accessed on 20 May 2025) and the Leica Register 360 software (https://leica-geosystems.com/products/laser-scanners/software/leica-cyclone/leica-cyclone-register-360, accessed on 20 May 2025), each of the corresponding control points was manually selected, and spherical coordinates using the developed MATLAB codes were determined. There was no software registration implemented for the calibration arrangement. The reason is that either software registration (automatic point-to-point or cloud-to-cloud) or manual registration imposes an additional root mean square error (RMSE), originating from the software comparison between the clouds. Importantly, the in situ atmospheric recordings were attached to each scanner station (Figure 7).

Primarily, the reliability of the datasets is examined using hypothesis tests. The objective is to eliminate the measurements containing outliers due to the manual target selection and reduce the potential propagation of noise, which might otherwise degrade the precision of refractive index estimation, leading to lower accuracy of 3D point coordinates. The hypothesis test compares the weighted W sum of the squares of the residuals V^tWV against the chi-square distribution ${\chi }_{a,r}^2$ , with redundancy numbers r, and the significance level α. Given the assumed significance level at least 2% for both datasets and corresponding redundancy, the test fails if V^tWV is greater than the critical value of the distribution (outliers exist in the measurements), or if this is smaller than the critical value, the test passes (better precision than prescribed) [38] (where v and σ are residuals and standard deviation, respectively).

$$V^{t}WV={\displaystyle \frac{v_i^2}{{\sigma }_i^2}}+\dots .+{\displaystyle \frac{v_n^2}{{\sigma }_n^2}},$$

(32)

The underlying assumption of the hypothesis test is that outliers can be detected in a reasonable manner (internal reliability), and that the impacts of other undetected outliers are insignificant (external reliability). Then, given either physical refractive index model (the Ciddor or the Closed Formula model), the refractive index is determined at the maximum level of precision, optimally reflecting the real-world atmospheric conditions along the sightline. Subsequently, using Equations either 25, 26, or 28, the estimated precision of the refractive index directly affects the accuracy of the 3D spherical coordinates. The accuracy of the 3D Cartesian coordinates [σ_x σ_y σ_z] is evaluated using the principle of propagation error, representing the a posteriori accuracy of the advanced model—whether physically or hybrid as follows:

$$\left[\begin{array}{c}{{\sigma }_x}^2\\ {{\sigma }_y}^2\\ {{\sigma }_z}^2\end{array}\right]=\left[\begin{array}{c}{(\cos v\cos h{\sigma }_r)}^2+{\left(-\mathrm{r}sinv\cos h{\sigma }_v\right)}^2+{(-\mathrm{r}\mathrm{c}\mathrm{o}\mathrm{s}v\sin h{\sigma }_h)}^2\\ {\left(\cos v\sin h{\sigma }_r\right)}^2+{\left(-r\sin v\sin h{\sigma }_v\right)}^2+{\left(r\cos v\cos h{\sigma }_h\right)}^2\\ {(\sin v{\sigma }_r)}^2+{(r\cos v{\sigma }_v)}^2\end{array}\right],$$

(33)

Here, [σ_r σ_v σ_h] represent the a posteriori accuracy of the 3D spherical coordinates. However, the accuracy assessment is interpreted as relative precision, given the accuracy of the control network (±1σ = 1 mm and 1″). Note that identical data analysis procedures are followed for both datasets. Figure 8 highlights a broad summary of the proposed calibration methodologies, considering the mentioned criteria.

5.1. Physical Refractive Index Model: Conventional Approach

The mine site dataset examines long-range scanning within a calibrated network using a distribution of eight control points. To initiate the data analysis, the computed inter-target ranges from selected targets in each station’s point cloud are determined and validated against the control points through a range consistency method (i.e., a total of 168 ranges for the entire network). The range consistency calibration method ensures that the ranges between each pair of corresponding control points in each scan data remain consistent across different scanner stations [35,36,37].

For pre-processing of the physical model, a reliability test on selected targets is conducted (using the hypothesis test, Equation (32)), resulting in 18 measurements for the Leica ScanStation P50 and 12 measurements for the Maptek I-Site 8820 being detected as outliers and eliminated from the dataset. Figure 9 shows the results of comparing each pair of control ranges with the average residuals, considering their height differences (i.e., the average ranges are derived from observations made at multiple scanner stations).

The comparison above highlights that, after applying the lowest significance level for the hypothesis test (2%), the residuals exhibit noticeable variations. However, the obtained standard deviations of the measured ranges are 3.6 mm for the Leica ScanStation P50 and 9.2 mm for the Maptek I-Site 8820. Note, the larger standard deviation for the Maptek device was anticipated, as the scanner has been operated in open-pit mining sites and has not recently been under accuracy assessment. Additionally, the nonlinear relationship between the ranges and the residuals for both scanners underscores the complexity of atmospheric modeling for long-range terrestrial laser scanning. So far, the conventional refractive index correction models for range measurements have typically made the assumption of a linear relationship, according to the relative consistency of the refractive index along a horizontal path for terrestrial survey measurements. Then, using the mean value of refractive indices from both ends of the sightline is recommended (Equations (25) and (28)) [1,27].

The following points elaborate on the limitations of the conventional approach for the physical refractive index model. Firstly, vertical stratifications of atmospheric conditions are significant and play a vital role in atmospheric correction modeling for terrestrial laser scanners [29,33]. M. Sabzali et al. [29] introduced different horizontal layers for the atmosphere depending on the height from ground level, according to the vertically stratified atmosphere. These layers commence from the layer closest to the ground surface, 0–3 m (lowest layer, with the most influential atmospheric variations), 3–20 m (intermediate layer), 20–100 m, and above 100 m (highest layer, with the least influence in terms of atmospheric variations) (Figure 10). The vertical gradients of the atmospheric parameters, particularly temperature, are substantially influential in the layers close to the ground surface.

It is acknowledged that the vertical variation in the atmosphere does not remain stable along the laser path passing through these layers, particularly in terms of vertical temperature gradients. This vertical stratification is a key consideration for improving current TLS atmospheric correction models. Due to the extended vertical field of view of terrestrial laser scanners (Table 1), a large number of points are observed close to or at the zenith and nadir when the vertical field of view is maximized, dissimilar to terrestrial surveying, where more points are captured near the horizon. Therefore, the observed ranges from terrestrial laser scanners experience varying atmospheric gradients along the path = the varying vertical gradient of the refractive index—impacting not only the range but also the vertical angle at different horizontal layers of the atmosphere.

Thus, points at different height levels receive variable residuals, regardless of their measured ranges. For instance, three peak residuals correspond to different height differences and range observations (i.e., −2.846 m for r₁₈ = 153.915 m, 9.866 m for r₁₂ = 432.298 m, and 53.595 m for r₁₃ = 425.325 m). The first two are located within short- and mid-range observations, but their height differences indicate that both terminals of these lengths lie within the lowest atmospheric layer (0–3 m) and between the lowest and intermediate atmospheric layers (3–20 m), respectively. In contrast, for the latter pair of points, the path extends from the lowest to the highest atmospheric layer. Hence, the sightline travels from layers close to the ground, with substantial atmospheric variations, to the highest layer with minimal atmospheric influence (Figure 10, condition (a)).

In contrast, each pair of points in the highest layer or above (Figure 10, conditions (b) and (c)), regardless of their height and range, is less affected by atmospheric components, such as the ranges 200.091 m, 354.867 m, 577.662 m, 636.217 m, 652.411 m, and 846.304 m encompassing varying height differences (

Table A4. Range, height differences, and range residuals (m) for the control points employed for range consistency method (dataset 1: mine site).

Range		Height Differences	Leica ScanStation P50	Maptek I-Site 8820
			Residuals	Residuals
r56	16.785	−2.043	−0.0010	0.0006
r14	58.902	1.068	−0.0045	−0.0093
r78	81.687	14.021	0.0003	−0.0003
r17	108.353	11.175	0.0045	0.0200
r18	153.916	−2.846	0.0078	0.0211
r46	158.249	−4.754	0.0004	−0.0004
r74	166.970	10.107	−0.0002	−0.0005
r45	174.695	−2.711	−0.0005	0.0003
r35	200.092	−49.816	0.0004	0.0028
r48	206.887	3.914	−0.0004	0.0003
r16	212.155	5.822	−0.0042	−0.0216
r36	215.694	−47.773	−0.0005	0.0026
r15	228.764	3.779	−0.0072	−0.0171
r28	281.678	12.713	0.0003	−0.0001
r67	315.723	5.353	0.0003	−0.0001
r57	332.345	7.396	−0.0007	−0.0009
r27	354.867	1.308	0.0004	0.0004
r68	364.694	8.668	0.0000	−0.0002
r34	372.185	−52.527	−0.0001	−0.0098
r58	381.243	6.625	−0.0009	0.0005
r13	425.326	53.595	−0.0121	−0.0190
r12	432.299	9.867	0.0064	0.0132
r24	479.095	−8.799	0.0000	0.0003
r37	524.953	−42.420	−0.0005	0.0002
r38	577.662	−56.441	−0.0005	0.0009
r26	636.217	−4.045	0.0003	−0.0001
r25	652.411	−6.088	−0.0006	0.0007
r23	846.304	−43.728	0.0007	0.0001

in Appendix A.3). The minimum, maximum, and negligible cases, as shown in Figure 10, represent the bounds of refractive index correction that must be applied in the advanced model.

Given the arguments above, Z-coordinates of points observed at greater vertical viewing angles relative to the station require more significant atmospheric corrections than X- and Y-coordinates at similar angles (e.g., Figure 10, maximum cases). Nevertheless, points observed at smaller vertical angles, such as those lying near the same horizontal atmospheric layer as the scanner station, often require negligible correction (e.g., Figure 10, negligible cases). This simplification leads to an underestimation of atmospheric effects in conventional surveying tasks.

To investigate the introduced sensitivity in the vertical direction, the second dataset from the dam site provides significant vertical angle variations from three nominated scanner stations—from the bottom of the dam wall (the lowest layer of atmosphere) to the survey targets on the dam wall and dam crest (the highest layer of atmosphere). In this experiment, 14 survey targets were positioned within a shorter scanning range but at higher vertical viewing angles (from 45° to 80°). The spherical coordinates were determined and validated against on-site survey data (i.e., a total of 84 vertical angles were derived for the entire network). After identifying seven and eleven outliers for the Leica ScanStation P50 and Maptek I-Site 8820, respectively, the results are depicted to compare the on-site surveyed vertical angles with the average Z-coordinate residuals (extracted from the average of measured coordinates), considering their measured ranges (Figure 11).

The large vertical angles are associated with increased Z residuals (i.e., the standard deviations of the measured vertical angles are 18″ for the Leica ScanStation P50 and 24″ for the Maptek I-Site 8820). This outlines that observations at vertical angles beyond ±50° exhibit greater sensitivity to atmospheric refraction effects than those within ±50° (Dataset 1: mine site). Figure 10 illustrates the maximum cases of vertical gradient of refractive indices under condition (a), highlighting variations from the lowest to either the intermediate or highest atmospheric layers.

In summary, when both ends of the sightline are located within the intermediate or higher atmospheric layers, refraction effects remain stable in the relatively same horizontal layer (negligible cases; Figure 10, conditions (b) and (c)). However, when either end of the sightline is placed within intermediate or lower atmospheric layers, the impact of atmospheric refraction becomes more significant and varies with respect to the vertical gradient of refractive index (maximum cases; Figure 10, conditions (a) and (c)). Since terrestrial laser scanners typically transmit signals from one end of the sightline, the systematic error in relation to refractivity increases with the rise in the vertical angle from the horizontal plane in a nonlinear manner. This comparative discussion underscores the requirement for a physical model to account for the variability of the temperature gradient along the laser path. The advanced physical model proposed here addresses this effect by incorporating varying vertical temperature gradients (Equations (29) and (31)) for points observed with a long-range baseline (greater than 200 m and a steep vertical angle (larger than ±50°) [27]. Consequently, the interdependence of spatial gradients of refractive index can be better resolved through high-precision determination of the refractive index along the traveling path [28].

5.2. Physical Refractive Index Model: Advanced Approach

To overcome the limitations of the conventional physical model, the advanced physical model was introduced. The conventional approach of a physical model, which assumes a uniform refractive index based on the average of two terminals, underestimates refraction impacts along the path with varying vertical gradients of refractive index, particularly at long ranges and steep vertical angles. The advanced model aims to incorporate vertical variations in refractive index, thus accounting for the nonlinear influence of refractivity along the propagation path (Equations (29) and (31)). In practice, the model divides different segmentations of a straight-line sight path into multiple segments passing though different atmospheric layers, with each layer characterized by its corresponding vertical temperature gradient and refractive index gradient. This approach depicts the vertical stratification of the atmosphere more realistically, especially for the TLS observations near the zenith and nadir, where sensitivity to vertical temperature gradients is highest. The required vertical temperature gradients for each layer are summarized in Table 3. Using Equation (24), the corresponding vertical gradients of refractive index are then computed. These provide the foundation for the improved TLS atmospheric correction–referred to as the advanced refractive index model.

Table 3. Vertical temperature gradient (K/m) for each stratified atmospheric layer [29].

Atmospheric Layers	Vertical Temperature Gradient ${\displaystyle \frac{\partial T}{\partial z}}$
Lowest	Variant (between −0.4 and +0.6)
Intermediate	≈+0.5
Highest	Variant (between −0.01 and −0.006)

Table 3. Vertical temperature gradient (K/m) for each stratified atmospheric layer [29].

Atmospheric Layers	Vertical Temperature Gradient ${\displaystyle \frac{\partial T}{\partial z}}$
Lowest	Variant (between −0.4 and +0.6)
Intermediate	≈+0.5
Highest	Variant (between −0.01 and −0.006)

By integrating these vertical gradients along the propagation path, the advanced model provides corrected ranges and vertical angles that more accurately represent real-world atmospheric conditions, bridging the limitations of the conventional approach. The results from the advanced model are then compared with those obtained from the conventional physical model (i.e., a priori residuals represent the computed residuals after applying the conventional Ciddor refractive index model, while a posteriori residuals correspond to the computed residuals after implementing the advanced Ciddor refractive index model) (Figure 12).

Table 4. Range accuracy obtained from measured data and physical refractive index models (conventional (a priori) and advanced (a posteriori)) using the range consistency calibration method (Dataset 1: mine site).

Accuracy (mm)
TLSs		Leica ScanStation P50	Maptek I-Site 8820
Measured		3.6	9.1
A priori	Ciddor	3.6	9.1
	Closed Formula	3.6	9.1
A posteriori *		2.4	7.6
Improvement		34%	16%

* The a posteriori accuracies and their improvements are based on the Ciddor developments.

also compares the a priori and a posteriori accuracies of the range observations in the range consistency method, expressed as root mean square error (RMSE) values. These comparisons highlight the improvements achieved by the advanced model over the physical model.

Results indicate that the conventional physical refractive index model—based on the average refractive indices at both endpoints of the range. This ensures consistency of the refractive index correction along the path, primarily represents the horizontal layer of the atmosphere and is unable to significantly enhance range accuracy over the long baselines. As discussed earlier, this consistency remains important and cannot be overlooked due to the sensitivity of range observations to 3D spatial gradients of the refractive index (impacting the derived 3D spherical coordinates). However, a more advanced refractive index model, that accounts for the vertical height profile within each stratified atmospheric layer, is less sensitive to the spatial gradients and improves accuracy by approximately 34% and 16% for the Leica ScanStation P50 and Maptek I-Site 8820, respectively. Note, both physical models yield identical RMSE values (a priori accuracies), supporting their suitability for TLS atmospheric correction.

Moreover, moderate improvements in the range consistency method do not necessarily guarantee an identical level of improvement in 3D point coordinates accuracy. Due to the existence of large baselines and restricted vertical viewing angles (i.e., the maximum vertical angle is approximately 14° at the mine site), the sensitivity between vertical gradient of refractive index ${\displaystyle \frac{dn}{dz}}$ and range refractive index n cannot be detected. Supporting this fact, the accuracy of the vertical angle was checked before and after applying the advanced model and found to remain unchanged (18″ for the Leica ScanStation P50 and 24″ Maptek I-Site 8820, with only a marginal sub-arcsecond improvement). However, the sensitivity between the two other horizontal gradients of refractive index $\left[\begin{array}{cc}{\displaystyle \frac{dn}{dx}}& {\displaystyle \frac{dn}{dy}}\end{array}\right]$and range refractive index n for points within these limited fields-of-view are partially resolved (2% and 6% accuracy improvement in X- and Y-coordinates, respectively) (

Table 5. Three-dimensional point coordinates accuracy obtained from measured data and physical refractive index models (conventional (a priori) and advanced (a posteriori)) using the range consistency calibration method (dataset 1: mine site).

TLSs	3D Point Coordinates	Accuracy (mm)
		A Priori	A Posteriori
Leica ScanStation P50	X	10.5	10.3
	Y	7.8	7.3
	Z	27.8	27.8
Maptek I-Site 8820	X	16.4	16.1
	Y	13.5	12.7
	Z	37.1	37.1

).

Generally, points observed at shallow vertical viewing angles exhibit reduced sensitivity to refraction effects. However, the absence of noticeable improvement in the Z-coordinates offers a stronger sensitivity between the vertical refractive index gradient ${\displaystyle \frac{dn}{dz}}$and the range refractive index n, which becomes more prominent at steeper vertical viewing angles (

Table 6. Vertical angle accuracy obtained from measured data and physical refractive index models (conventional (a priori) and advanced (a posteriori)) using onsite terrestrial survey under an average daytime temperature of 20 °C with a variation of ±1–2 °C at a 2% significance level for the observations (dataset 2: dam site).

Accuracy (″)
TLSs	Leica ScanStation P50	Maptek I-Site 8820
A priori	18″	24″
A posteriori	10″	19″
Improvement	44%	20%

). This behavior demonstrates the directional dependence of 3D point coordinate accuracy on the vertical gradients of the refractive index, as a function of the observed vertical angle. Conventionally, the influence of atmospheric refraction on the X- and Y-coordinates is minimal for the points near the zenith, whereas Z-coordinates are more affected by changes in refraction (i.e., sin v in Equation (1)).

Results underscore that shorter baselines combined with larger vertical angles (Dataset 2: dam site) are more susceptible to atmospheric distortions than longer baselines with smaller vertical angles (Dataset 1: mine site). This indicates that the improved correction model not only enhances the accuracy of the vertical angle (Table 6), but it also improves the range accuracy (due to the diminished sensitivities) (

Table 7. Three-dimensional point coordinates accuracy obtained from measured data and physical refractive index models (conventional (a priori) and advanced (a posteriori)) using onsite terrestrial survey (Dataset 2: dam site).

TLSs	3D Point Coordinates	Accuracy (mm)		Improvement
		A Priori	A Posteriori
Leica ScanStation P50	X	12.7	7.5	41%
	Y	7.1	5.9	17%
	Z	12.4	7.3	41%
Maptek I-Site 8820	X	17.8	14.6	18%
	Y	10.2	9.5	7%
	Z	17.6	14.6	17%

). These combined effects extend to the overall 3D point coordinate accuracy. Accordingly, the advanced physical model consistently propagates corrections across 3D point coordinates, where a higher percentage of improvement is expected for X- and Z-directions than for the Y-direction. This model also acknowledges that the impact of the horizontal gradients of the refractive index on the overall accuracy of 3D point coordinates is minimal, in comparison with the maximal impact of the vertical gradient of the refractive index on 3D point coordinates.

In summary, the findings indicate a moderate level of improvement with the advanced physical refractive index model, Ciddor’s model (Equations (19) and (24)), where 3D point coordinate accuracies were enhanced from the centimeter to the millimeter level for the Leica ScanStation P50, and predominantly in the Y-coordinate for the Maptek I-Site 8820. However, two major concerns remain and are worth further investigation: (1) the limited parameterization of physical refractive index models for practical long-range terrestrial laser scanning (whether the Ciddor or the Closed Formula), and (2) the potential unreliability of 3D spatial gradients of atmospheric parameters, particularly vertical temperature gradients across stratified atmospheric layers, when applied to estimate refractive index gradients along the entire path in the advanced physical refractive index model. These limitations have likely contributed to the modest improvements reported in Table 7. To address these issues, hybrid physical–data-driven neural network models are proposed, supported by validated outcomes from physical modeling and cross-referenced with the control network.

5.3. Hybrid Refractive Index Model

The insights gained from the physical refractive index indicate that a moderate reduction in the accuracy of 3D point coordinates can be optimized, provided that the maximum possible precision for estimating the refractive index is assigned along the propagating path, with the least sensitivity to its spatial gradients. Following this principle, a hybrid refractive index model is proposed. This approach allows the model to assign variable weights to the 3D spatial gradients of the refractive index, performing as a comprehensive solution to compensate for the inherent limitations of physical algorithms and accurately evaluate atmospheric refraction along the line of sight. Consequently, the results aid in validating the field results obtained from advanced physical models and improve the overall 3D point coordinate accuracy (i.e., consistent millimeter-level relative precision) through the nonlinear treatment of refractivity along the path across different scanning environments and using various scanners.

To support accurate prediction through this data-driven approach, it is recommended that any remaining systematic errors be removed from both field datasets. However, these are expected to be quite negligible compared to the a priori standard deviation of the observations (Table 1). In general, the following steps must be taken:

• Generate vectors of in situ atmospheric recordings (e.g., air temperature, atmospheric pressure, and/or relative humidity, including their spatial gradients) together with intrinsic scanner characteristics (wavelength number, range, and angular accuracy) as the input data, to predict the refractive index as the output.
• Implement the training of a neural network to model the nonlinear relationship between the input parameters and the refractive index along the path. The network consisted of 2 hidden layers with 10 neurons each, through using the hyperbolic tangent sigmoid (tansig) activation function in the hidden layers and a linear (purelin) function in the output layer. The network was trained by the Levenberg–Marquardt backpropagation algorithm (trainlm), with 80% of the dataset used for training and 20% for testing. The random seed was fixed, and the training window was disabled to ensure reproducible results.
• Perform symbolic regression on the neural network outputs using a physical interpretation of basis functions, such as the Ciddor formulation (Equations (13)–(19), (24), and (30)) [22,23].
• Derive closed-form symbolic expressions for refractive index, applicable to new atmospheric input conditions (optional). For illustration, the MATLAB implementation of these steps is presented in Appendix A.4.

Through symbolic expressions derived from the neural network, the increased precision in the estimation of refractive index and its gradients further reduces the sensitivity to the spatial gradients of the refractive index. The a posteriori accuracies in range from the mine site dataset and vertical angle from the dam site dataset were reduced to 1.8 mm and 9″ for the Leica ScanStation P50, and 2.3 mm and 15″ for the Maptek I-Site 8820. Additionally, these consistently diminish the accuracy of 3D point coordinates to a reliable millimeter level (Figure 13).

Figure 13 illustrates the improvement in 3D point coordinate accuracy obtained through hybrid atmospheric correction models based on Ciddor developments, which elevate the refractive index estimation precision along the laser path and reduces accuracy from the centimeter to the reliable millimeter level for the 3D point coordinates. Worth emphasizing, both models, the advanced physical model and the hybrid model, consistently enhance 3D point coordinates accuracy, with particularly notable improvements in the X- and Z-coordinates. These results confirm the effectiveness of the proposed models in mitigating atmospheric errors in long-range terrestrial laser scanning compared with the conventional physical refractive index modeling.

6. Discussion

Previously, the impact of atmospheric refraction on the 3D point coordinates was investigated in terms of range and vertical angle refraction at two monitoring locations under varying atmospheric conditions. Given Table 5 and Table 7, it was shown that the improved atmospheric correction model primarily affects the point coordinates, with the maximum impact on the Z-coordinates close to the zenith, but the minimum impact on the X- and Y-coordinates close to the horizon, and negligible changes near the zenith. In the discussion section, the comparison of all observed points in each dataset with their corresponding ranges, vertical angles, and Z-residuals—obtained through the three proposed refractive index models—is initially presented (Figure 14 and Figure 15). This further enables visualization of the behavior of the entire 3D point cloud after applying the atmospheric correction models for the dam site dataset, acquired by the Leica ScanStation P50 with the real sand extended simulated scanning ranges.

The analysis of the second dataset—under cooler daytime temperatures recorded at a different time of the year (i.e., 20 °C with a variation of ±1–2 °C (Figure 7))—also validates the rigorous implementation of the advanced and hybrid refractive index models for the steep vertical angle datasets. The results indicate that the observations at larger vertical angles (particularly greater than ≈60° have a considerably higher impact than the observations at shallower vertical angles on the overall accuracy of the 3D point coordinates. This outcome arises due to the increasing sensitivity between range refraction and vertical gradient refraction, representing that the vertical gradient of refractive index dominates long-range TLS atmospheric error modeling, compared to the range refractive index investigated in the mine site dataset (Figure 15). Importantly, the requirement to account for non-uniform atmospheric stratification and the anisotropic distribution of noise at larger vertical viewing angles is essential. Therefore, based on the successful results accomplished from the advanced refractive index model, a combination of optimized weightings using the vertical gradient of refractive index into the advanced physical model provides a robust and intuitive solution to mitigate systematic atmospheric errors, allowing reliable millimeter- to sub-millimeter-level precision in 3D point cloud reconstruction for TLS-based deformation monitoring applications.

To enhance the intuitiveness of the results, the overall distribution of 3D point cloud corrections obtained from the three refractive index modeling methods is compared in Figure 16. The residuals in the advanced and hybrid models with respect to the observed range show a noticeable improvement compared to the conventional method.

Moreover, Figure 16 also depicts that, since atmospheric corrections are more significant at larger vertical angles, the high proportion of range corrections occurs from the Z-direction rather than the X- or Y-directions (i.e., the vertical angle observations are more sensitive to refractive index variations along the laser path). Consequently, this makes vertical angle corrections critical for ensuring the geometric accuracy of 3D point coordinates at long ranges (Figure 17).

To further illustrate the practical significance of atmospheric effects on the 3D point cloud, the following simulations are conducted across three representative scanning ranges (200 m, 400 m, and 1000 m (the maximum reported scanning range for the Leica ScanStation P50)). The results of these simulations in Figure 18 support the residual patterns identified in Figure 14 and Figure 15. At higher vertical angles (between 60° and 80°) and ranges greater than 200 m, the atmospheric correction increases by more than a factor of two—from 6.7 mm to 24 mm. At the simulated maximum range of 1000 m, this correction reaches 50.1 nm.

In summary, the influence of temperature variations above zero plays a dominant role in real-world surveying refraction [27]. To comprehensively validate both the advanced and hybrid atmospheric correction models, two distinct atmospheric conditions, reflecting different seasonal and temporal variations across Australia, were incorporated for the entire acquired range and vertical angle datasets. This approach highlights the major implication of accurately capturing real atmospheric conditions and their spatial variations when assessing a robust and generic atmospheric correction model for long-range terrestrial laser scanning. Note, these variations predominantly affect the corrections in the Z-direction more than X- and Y-coordinates at range longer than 200 m and steep vertical angle of 60°.

7. Conclusions

The current research aimed to investigate the effect of atmospheric variations along the line of sight when long-range terrestrial laser scanning is required. The traveling laser beam through the atmosphere is predominantly affected by several optical phenomena. One of the most significant occurrences is the refraction of the optical path. Refraction causes deviations from the theoretical optical path and introduces variations in the intersection between the laser beam and the surface of the targets. This is identified as one of the systematic error sources in long-range scanning. To address the refractivity patterns along the optical path, the detailed mathematical developments of two physical refractive index models (the Ciddor and the Closed Formula) are elaborated here.

The physical model establishes the relationship between atmospheric conditions (air temperature, atmospheric pressure, and relative humidity) and the refractive index using the given wavelength. In the conventional approach, the average of refractive indices, based on the first and last terminals of the sightline, is obtained to address the linear condition of refractivity along the path. Furthermore, a higher precision on the order of [1 − 5] × 10⁻⁸ is recommended for the estimation of refractive index to ensure millimeter or sub-millimeter accuracy for the observations [1]. For advancements of the physical models, 3D spatial gradients of the refractive index are developed in the current work, in relation to the nonlinear physical modeling, where varying refractive indices along the path are demanded. These requirements are applied when the laser pulses experience different media (vertically stratified atmospheric layers).

In the field experiments for the atmospheric modeling, two long-range terrestrial laser scanners (Leica ScanStation P50 and Maptek I-Site 8820) were employed for quality testing under varying atmospheric conditions at two deformation monitoring sites: a mine site—enabling the long-range scanning over 800 m—and the dam site—providing the flexibility of a steep vertical viewing angle close to zenith 80°. Both sites were controlled under a calibrated network arrangement, utilizing an on-site range consistency calibration method supported by GPS control points, along with field survey observations to verify vertical angle accuracy. During each scanning setup, the atmospheric conditions of air with respect to each scanner station (in situ atmospheric recordings attached to each station) were tracked across the entire sites.

The recognized ranges from the mine site dataset were evaluated for network reliability, and after maximizing the precision of refractive index estimation through the advanced approach of the physical refractive index model, the improvements of 34% for the Leica ScanStation P50 and 16% for the Maptek I-Site 8820 in range accuracy were observed. Since several points were located at the shallow vertical angle in the mine site, the sensitivity to 3D spatial gradients of the refractive index—specifically, the vertical gradient of refractive index across stratified atmospheric layers—was not detected, and this led to limited improvement in 3D point coordinate accuracies. The dam site dataset demonstrates mitigation strategies for this sensitivity, particularly for points located at higher elevations (with vertical viewing angles larger than 60°). With the advanced physical model, improvements occurred not only in the a posteriori accuracy of the vertical gradient of the refractive index (44% (from 18″ to 10″) for the Leica ScanStation P50 and 20% (from 24″ to 19″) for the Maptek I-Site 8820), but also in the accuracies of the 3D point coordinates (reliable centimeter- to millimeter-level accuracy). The reason is that the points, located on approximately the same horizontal plane as the scanner station, are more uniformly affected by atmospheric corrections (Dataset 1: mine site), while the points, located at different horizontal planes, require larger X- and Z-coordinate corrections (Dataset 2: dam site). The hybrid model, referencing back to the advanced physical refractive index model, is advised to achieve higher millimeter-level relative precision in 3D point coordinates consistently. Millimeter-level accuracy is ultimately attained by reducing the range and vertical angle accuracies to 1.8 mm and 9″ for the Leica ScanStation P50, and 2.3 mm and 15″ for the Maptek I-Site 8820 (Table 6). The advantage of the proposed approaches—the advanced and hybrid physical models—is their ability to represent real-world refractivity conditions along the laser path by maximizing the precision of refractive index estimation and minimizing sensitivity to spatial gradients, achieved through a stochastic weighting of the vertical refractive index gradient within different atmospheric layers.

For future work, several pathways exist toward achieving millimeter- or sub-millimeter-level accuracy in the field calibration of long-range terrestrial laser scanning. First, additional optical effects of the laser line, such as scattering and reflection, play an integral role in determining robust radiometric and spatial calibration results. Second, the algorithmic steps presented here for the physical refractive index model are recommended to be further modified based on more accurate in situ atmospheric observations (e.g., temperature accuracy better than 0.5 °C. This is particularly suggested for highly sensitive and high-risk deformation projects. Preferably, attached thermometer arrangements should enable recording of the epoch-wise atmospheric conditions of the laser. Alternatively, for advanced physical refractive index modeling, it is strongly recommended to create a temperature heat map sensitive to height profiles, as the dominating factor, to better indicate temperature variations across the monitoring test sites. Third, the use of an appropriate stochastic model for 3D spherical observations is highly advised for comprehensive system calibration of the 3D point cloud, addressing geometric error models. Finally, the relevant atmospheric correction factors should be ideally applied within the instrument, enhanced by a robust atmospheric measurement technique along the entire path, which can be supported by the manufacturing principal assembly.