Derived Effective (${\mathit{K}}_{\mathbf{e}\mathbf{f}\mathbf{f}}$) Versus Scalar (${\mathit{K}}_{\mathbf{0}}$) Attenuation in the Baltic Sea: Characterising Spectral Divergence and Physical Drivers

Abstract

The optical complexity of shallow Case 2 waters challenges remote sensing accuracy due to the non-linear behaviour of optically active constituents. This study evaluates the spectral divergence between the target-derived effective attenuation ($K_{\mathrm{e}\mathrm{f}\mathrm{f}}$) and the ambient scalar attenuation coefficient ($K_0$) across 12 Baltic Sea locations. Using hyperspectral radiometry and K-Means clustering, three optical water types (OWTs) were identified. We demonstrate that the historical static approximation based on the diffuse attenuation coefficient ($K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ ≈ 2$K_{\mathrm{d}}$) is systematically biased in scattering-dominated environments. Our empirical results yielded a regional relationship of $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ = 2.33$K_0$ ($R^2$ = 0.65); however, residual analysis reveals that linear multipliers fail to capture non-linear light decay. Random Forest regression identified total suspended matter (TSM) as the primary driver of $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ variance (28.0%), confirming that “geometric rejection” of scattered photons artificially inflates signal loss in turbid waters. This divergence is most pronounced in the 500–650 nm range, where low absorption facilitates multiple scattering events. We conclude that active remote sensing requires a sensor-fusion approach, utilising passive OWT classification to dynamically parameterise active attenuation models.

1. Introduction

The optical complexity of shallow coastal waters, commonly classified as Case 2 waters, poses a persistent challenge for remote sensing and underwater visibility modelling [1,2]. Unlike open ocean (Case 1) environments, where optical properties are primarily covariant with phytoplankton biomass [3], coastal zones are shaped by independent variations in optically active constituents (OACs), including terrestrial runoff, anthropogenic inputs, and sediment resuspension [4,5,6]. These factors create dynamic optical regimes where scattering often outweighs absorption, complicating the analysis of radiative transfer processes that govern light penetration [1,7,8].

Effective monitoring of these Case 2 water bodies requires accurate vertical profiling of the water column. While passive satellite radiometry provides synoptic coverage [8,9], it is frequently confounded by bottom reflectance artefacts [10,11]. Furthermore, the impact of terrestrial runoff, high sediment loads, and complex geomorphology severely limits the accuracy of satellite-derived bathymetry (SDB) in these shallow waters, restricting remote observations to the first optical depth [8,12]. Active remote sensing, particularly LiDAR (Light Detection and Ranging), offers a potential solution for depth-resolved measurements [13]. However, the accuracy of such systems depends heavily on a robust assessment of the effective attenuation coefficient ($K_{\mathrm{e}\mathrm{f}\mathrm{f}}$) [14,15] and its relationship to the ambient light field.

Historically, bio-optical models have relied on a geometric simplification, assuming that the effective active signal attenuates at approximately twice the rate of passive diffuse signal ($K_{\mathrm{e}\mathrm{f}\mathrm{f}}\approx 2K_{\mathrm{d}}$) to account for the two-way photon path [12,13]. While valid in clear, absorption-dominated waters [16,17], this linear assumption frequently breaks down in turbid environments [18,19]. In such waters, the specific angular acceptance of narrow-field sensors results in the “geometric rejection” (or exclusion) of scattered photons—photons that are lost to the active signal but continue to contribute to the diffuse irradiance field measured by broad-field passive sensors [12,17,20,21].

To accurately evaluate this divergence in highly diffuse coastal environments, it is essential to compare $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ against the true total ambient light field rather than a geometrically limited proxy. Consequently, the ambient scalar attenuation coefficient ($K_0$) provides a more physically comprehensive baseline than the planar diffuse coefficient ($K_{\mathrm{d}}$). Because a spherical sensor gives equal weight to photons arriving from all directions, it successfully integrates the highly scattered and bottom-reflected light that a planar cosine collector geometrically excludes [22]. By evaluating $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ directly against $K_0$, the specific scattering penalty (the artificial loss of signal due to the narrow-field active sensor, even as scattered photons continue to propagate) can be accurately quantified against the baseline of total surviving ambient light.

Although validation studies have examined the relationship between systems and diffuse attenuation for LiDAR ($K_{\mathrm{L}\mathrm{i}\mathrm{D}\mathrm{A}\mathrm{R}}$) applications [13,23,24], these are predominantly restricted to monochromatic wavelengths (typically 532 nm). Furthermore, while the partitioning of diffuse attenuation into constituent drivers, i.e., coloured dissolved organic matter (CDOM), total suspended matter (TSM), and phytoplankton, is well-established for passive-only optical models in Baltic Sea waters [25,26], these studies do not account for the geometric complexities of the active LiDAR signal. The contribution of this study, therefore, is not the discovery that the standard geometric ratio is highly inconsistent in coastal waters, but rather the spectral characterisation of this divergence across contrasting optical water types and the explicit identification of the constituent-specific drivers controlling it.

This study addresses these uncertainties by analysing in situ optical data collected from the southern Baltic Sea, encompassing a diverse range of optical conditions from turbid estuarine to clearer open coastal waters. We aim to: (1) classify the regional optical variability into distinct optical water types (OWTs) using K-Means clustering; (2) quantify the spectral deviation of the observed $K_{\mathrm{e}\mathrm{f}\mathrm{f}}/K_0$; and (3) determine the specific drivers of attenuation for derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ and scalar $K_0$ using Random Forest (RF) regression. By identifying the influence of specific OACs, this study demonstrates how constituent-driven scattering effects dictate this spectral divergence, proposing an empirical correction for retrieving attenuation properties in optically complex, shallow waters.

2. Materials and Methods

2.1. Study Area

The study was conducted along the southern coast of the Baltic Sea, which is a shallow, micro-tidal, brackish environment where optical complexity is driven by a combination of seasonal freshwater discharge, high dissolved organic loads and wind-wave-induced sediment resuspension [27,28]. The 16 individual samples across 12 distinct geographic stations along the Mecklenburg-Vorpommern coastline (spanning from Wismar Marina to Dierhagen) were selected to capture a comprehensive range of water types in varied coastal morphologies and optical regimes (Figure 1).

The sampling stations are representative of: (1) sheltered, semi-enclosed embayments of marinas (e.g., Wismar Marina, Rerik) characterised by limited water exchange and localised turbidity; (2) estuarine environments (e.g., Schnatermann, Warnemünde), which are heavily influenced by the Warnow River plume, introducing high concentrations of terrestrial CDOM and nutrients; and (3) exposed open coastal stretches (e.g., Kühlungsborn, Graal Müritz, Dierhagen), where the optical water type is predominantly shaped by wave action and the resuspension of inorganic sandy-muddy bottom sediments.

Field campaigns were conducted over two consecutive years: June–July 2022 and July–August 2023. The water depth at the sampling stations ranged from shallow nearshore waters (1.2 m) to deeper coastal stations (4.0 m). A map of the study area and specific station locations is provided in Figure 1. A small hand-operated grab and an underwater camera mounted on a tripod were used to qualitatively determine bottom features. In parallel, Secchi-depth was determined by means of lowering a white disk of 50 cm diameter to determine water depth and transparency.

2.2. Datasets (Data Acquisition)

2.2.1. Water Sample Analyses

Surface water samples were collected at mid-depths using a water sampler to quantify key OACs. This is assumed representative of the water column due to the well-mixed and shallow nature of all stations. The water was sampled in two replicates to be averaged. The OACs are quantified for (1) pigments: chlorophyll-a and phaeopigments as proxy to phytoplankton biomass (Chl-a and Phaeo, respectively) in mg m⁻³; (2) TSM in g m⁻³; and (3) absorption coefficient of coloured dissolved organic matter at 440 nm ($a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}}\left(440\right)$) in m⁻¹.

Samples were analysed for Chl-a, Phaeo, and CDOM using a Perkin Elmer Lambda 2 double-beam spectrophotometer (Perkin-Elmer, Überlingen, Germany). To determine pigment concentrations, particulate matter was collected on 45 mm GF/F filters (0.7 µm pore size) and extracted using ethanol. Following the initial Chl-a measurement, the extract was acidified with hydrochloric acid to quantify Phaeo, consistent with the protocol described in [29].

Prior to analysis, both the samples and ultrapure water reference were passed through 0.2 µm Millipore filters. The absorbance was measured and converted to absorption coefficients following the protocols in [30]; the coefficient at 440 nm was used to represent CDOM concentration. To adjust for baseline offsets caused by scattering or salinity differences, the average absorption between 700 and 750 nm was subtracted from each spectrum [31].

TSM was quantified gravimetrically following the protocols in [32,33]. A known volume of water was filtered through pre-weighed 45 mm GF/F filters (0.7 µm pore size). To determine the mass of the retained solids, the filters were oven-dried for a minimum of 24 h and re-weighed.

2.2.2. In Situ Irradiance and Radiometry

In-water spectral scalar irradiance (${\mathit{E}}_0\left(\mathit{\lambda }\right)$) was acquired using a Macam Photometrics SR9910 spectroradiometer (Macam Photometrics Ltd., Livingston, UK). Fitted with an integrating sphere light guide, the instrument provided a quasi-spherical field of view (FOV) to capture photons arriving from all directions. Measurements were recorded across a spectral range of 240 to 800 nm with a bandwidth of 1 nm, expressed in units of W m⁻² nm⁻¹. Underwater profiles were conducted at discrete depths ranging from 0.5 m to 3 m, with each depth measurement representing the average of two replicates. All profiles were measured from the sun-exposed side of the sampling platform to minimise shading artefacts. Because simultaneous above-water reference measurements were not recorded, profiles were conducted in rapid succession under assumed stable illumination conditions to ensure that variations in ambient sunlight did not introduce artefacts into the derivations.

Complementary above-water hyperspectral radiometry was conducted using a UDS-1100 spectroradiometer (Spectral Evolution, Haverhill, MA, USA), covering a spectral range of 320 to 1100 nm with a 1 nm sampling bandwidth. This system employed a dual-sensor configuration to acquire simultaneous radiometric readings: downwelling solar irradiance ($E_{\mathrm{d}}\left(\lambda \right)$) was measured via a cosine-corrected optical diffuser positioned in the air ($0^{+}$), while the corresponding upwelling radiance ($L_{\mathrm{u}}\left(\lambda \right)$) was measured using a fibre-optic probe with a fixed 4° FOV.

Prior to each sampling session, the instrument was calibrated against a white Spectralon reference panel (Labsphere Inc., North Sutton, NH, USA) to ensure radiometric accuracy. To quantify the depth-dependent attenuation of light, a white reference target was lowered into the water column at incremental depths of 0.5 m. A minimum of five replicate scans were acquired and averaged for each depth step to minimise random noise. Sampling was restricted to stable illumination conditions between 11:00 and 15:00 local time. To mitigate environmental artefacts, measurements were taken on the sun-exposed side of the sampling area to avoid shading, and areas with glint or surface agitation were avoided. At stations influenced by tidal current, sampling coincided with high water slack to minimise turbidity caused by sediment resuspension.

2.3. Methodology (Data Analysis)

2.3.1. Derivation of Attenuation Coefficients ($K_0$ and Derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$)

To derive the spectrally resolved ambient scalar attenuation coefficient $K_0\left(\mathit{\lambda }\right)$, the raw data were first cleaned by removing negative signals, which represent non-physical instrumental noise. Additionally, values outside the visible range (400–750 nm) were masked to exclude spectral instability and low signal-to-noise ratios caused by the naturally high absorption of pure water in the near-infrared and ultraviolet regions.

Following spectral filtering, the derivation of $K_0\left(\mathrm{z},\mathit{\lambda }\right)$ was based on the exponential decay of light through the water column, governed fundamentally by the Beer-Bouguer-Lambert (BBL) law. $E_0\left(\mathit{\lambda }\right)$ was recorded at discrete 0.5 m or 1 m intervals. Depending on the maximum depth of each sampled station, vertical profiles consisted of either two depths or three discrete depths. For each wavelength across the measured spectrum, $K_0\left(\mathit{\lambda }\right)$ was computed for each station by applying a least-squares linear regression to the natural logarithm of the irradiance versus depth ($z$), defined in Equation (1):

$$\ln \left(E_0(z,\mathit{\lambda })\right)= - K_0\left(\mathit{\lambda }\right)z +C$$

(1)

where $z$ is the depth in metres and $C$ represents the y-intercept.

Consequently, $K_0\left(\mathit{\lambda }\right)$ corresponds to the negative slope of this regression line. For shallow stations restricted to two-depth measurements, this linear derivation is mathematically equivalent to a direct, two-point algebraic calculation.

To derive the spectrally resolved, target-derived effective attenuation coefficient (proxy for active signal attenuation) $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$, the computational workflow was divided into three distinct steps: (1) isolating the target signal, (2) applying the physical decay model, and (3) extracting the coefficient via linear regression.

First, the remote sensing reflectance ($R_{\mathrm{r}\mathrm{s}}$) was calculated as the ratio of upwelling radiance to downwelling irradiance. To isolate the reflectance decay of the target itself, the background signal from the water column (measured without the target) was subtracted from the total signal measured with the submerged target. The isolated target reflectance at a specific depth, $z$ is denoted as $R_{\mathrm{r}\mathrm{s},\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}\left(z,\mathit{\lambda }\right)$ in Equations (2) and (3):

$$R_{\mathrm{r}\mathrm{s}, \mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}\left(z,\lambda \right)=R_{\mathrm{r}\mathrm{s},\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}\left(z,\lambda \right)-R_{\mathrm{r}\mathrm{s},\mathrm{w}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{r}}\left(\lambda \right)$$

(2)

$$R_{\mathrm{r}\mathrm{s},\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}\left(z,\lambda \right)=\left({\displaystyle \frac{L_{\mathrm{u},\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}(z,\lambda )}{E_{\mathrm{d}}\left(\lambda \right)}}\right)-\left({\displaystyle \frac{L_{\mathrm{u},\mathrm{w}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{r}}\left(\lambda \right)}{E_{\mathrm{d}}\left(\lambda \right)}}\right)$$

(3)

where $R_{\mathrm{r}\mathrm{s},\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}\left(z,\lambda \right)$ is the total reflectance measured with the white target at depth $z$; $R_{\mathrm{r}\mathrm{s},\mathrm{w}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{r}}\left(\lambda \right)$ is the background reflectance of the water column measured without the target; $L_{\mathrm{u},\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}$ and $L_{\mathrm{u},\mathrm{w}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{r}}$ are the corresponding radiance for surface water with target, and without target, respectively; $E_{\mathrm{d}}\left(\lambda \right)$ is the concurrent surface downwelling irradiance. Because this experimental set up utilises passive radiometry to simulate active attenuation principles, $E_{\mathrm{d}}\left(\lambda \right)$ was required to standardise the upwelling radiance into $R_{\mathrm{r}\mathrm{s}}$. This normalisation effectively isolates the inherent decay of the target’s signal from natural variations in ambient sunlight during vertical profiling.

It must be noted that applying Equation (2) assumes the background water column signal remains constant. The placement of a submerged target physically blocks upwelling radiance from the water beneath it, potentially leading to a slight over-subtraction of the background signal ($R_{\mathrm{r}\mathrm{s},\mathrm{w}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{r}}\left(\lambda \right)$), particularly at shallow target depths. This ‘target blocking effect’ may be minimised in highly turbid Case 2 waters, as 90% of the upwelling signal originates from the first optical depth [34]. Because $K_{\mathrm{d}}$ is high in these turbid environments, the first optical depth is shallow. Consequently, a target lowered beyond this primary scattering layer intercepts only the residual deep-water signal, which accounts for a negligible fraction of the total ambient remote sensing reflectance [35,36]. However, it remains a recognised methodological constraint of this study.

As part of the initial quality control, samples PL2 and NHB were excluded from final analysis due to non-physical behaviour (i.e., increasing reflectance with depth), which could be caused by passing turbidity, a highly absorbing or scattering layer, or boundary effect at sites where the bottom was visible [12].

With the target signal successfully isolated, the depth-dependent decay of this target reflectance was then modelled using the BBL extinction law [17,37]. While this fundamental principle describes exponential decay in idealised media, the multiple scattering characteristic of turbid Case 2 waters necessitates an empirical approximation for the return signal [38]. Applying this law to the depth-dependent decay of the measured target signal yields the physical model defined in Equation (4):

$$R_{\mathrm{r}\mathrm{s},\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}\left(z,\mathit{\lambda }\right)=R_{\mathrm{r}\mathrm{s},\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}\left(\mathit{\lambda },0\right)·e^{-K_{\mathrm{e}\mathrm{f}\mathrm{f}}\left(\mathit{\lambda }\right)·z}$$

(4)

where $R_{\mathrm{r}\mathrm{s},\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}\left(z,\mathit{\lambda }\right)$ is the measured target reflectance at depth $z$; $R_{\mathrm{r}\mathrm{s},\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}\left(\mathit{\lambda },0\right)$ represents the extrapolated theoretical reflectance just beneath the surface.

To computationally extract the derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}\left(\lambda \right)$ from this exponential decay model, the measured reflectance values across all depths were linearised using a natural logarithmic transformation. This converts the exponential decay curve into a linear relationship, defined by Equation (5):

$$\mathit{ln}\left(R_{\mathrm{r}\mathrm{s},\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}\left(z,\lambda \right)\right)=\mathit{ln}(R_{\mathrm{r}\mathrm{s},\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}\left(0,\lambda \right))-K_{\mathrm{e}\mathrm{f}\mathrm{f}}(\lambda )·z$$

(5)

For each station, a log-linear least squares regression was applied to this transformed data. Within this model, derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}\left(\lambda \right)$ corresponds to the negative slope of the resulting regression line. By iteratively fitting this relationship and calculating the gradient for every wavelength, a continuous derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ spectrum was generated.

To ensure the physical validity of the calculated attenuation coefficients, the goodness-of-fit was rigorously assessed for each spectral band. The reliability of the regression was quantified using the coefficient of determination ($R^2$), the Root Mean Square Error (RMSE) and standard error for each station (

Table A1. Statistical summary of the log-linear model performance for the derivation of $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ and $K_0$ across all sampling stations. Metrics include the coefficient of determination (R²), root mean square error (RMSE), and standard error (SE). Data are presented as mean and extreme values (min/max) to demonstrate model robustness across varying environmental conditions.

Sample ID	${\mathit{K}}_{\mathbf{e}\mathbf{f}\mathbf{f}}$		${\mathit{K}}_0$		${\mathit{K}}_{\mathbf{e}\mathbf{f}\mathbf{f}}$		${\mathit{K}}_0$		${\mathit{K}}_{\mathbf{e}\mathbf{f}\mathbf{f}}$		${\mathit{K}}_0$
	${\mathit{R}}^2$				RMSE				Standard Error
	Mean	Min	Mean	Min	Mean	Max	Mean	Max	Mean	Max	Mean	Max
MS1	0.96	0.86	0.98	0.93	0.39	1.01	0.07	0.14	0.42	1.57	0.05	0.10
MS2	0.99	0.91	0.96	0.76	0.10	1.13	0.11	0.27	0.55	2.76	0.08	0.19
SH1	0.99	0.85	NA	Inf	0.17	0.94	NA	Inf	0.26	1.19	NA	Inf
SH2	0.95	0.85	NA	Inf	0.37	0.69	NA	Inf	0.53	0.99	NA	Inf
RR1	0.99	0.87	NA	Inf	0.13	0.83	NA	Inf	0.18	1.56	NA	Inf
RR2	0.99	0.86	NA	Inf	0.15	0.69	NA	Inf	0.24	1.59	NA	Inf
KB2	0.92	0.86	NA	Inf	0.26	0.62	NA	Inf	0.34	0.84	NA	Inf
WS2	0.95	0.85	0.93	0.60	0.17	1.14	0.18	0.73	0.41	2.79	0.12	0.50
PL1	1.00	1.00	0.63	0.00	0.00	0.00	0.14	0.57	NA	Inf	0.17	0.69
HLB	0.97	0.85	0.96	0.78	0.17	0.85	0.09	0.36	0.28	1.13	0.11	0.44
KBB	0.99	0.92	0.83	0.37	0.06	0.35	0.47	1.37	0.08	0.44	0.58	1.68
HMB	0.97	0.89	0.85	0.52	0.20	0.97	0.18	0.95	0.19	1.20	0.22	1.17
GMB	0.97	0.85	0.57	0.00	0.20	0.99	0.41	0.82	0.42	2.43	0.51	1.00
DHB	0.98	0.90	0.99	0.93	0.13	0.74	0.13	0.60	0.36	1.80	0.16	0.73

in Appendix A).

When deriving $K_0$, the vertical profiles for several shallower stations (SH1, SH2, RR1, RR2 and KB2) were restricted to two discrete depths. In these instances, the linear regression reduces to a deterministic two-point derivation of the BBL. Because a linear model with two data points has zero degrees of freedom, the calculation of goodness-of-fit statistics—specifically $R^2$, RMSE and Standard Error—is mathematically precluded. These parameters are consequently reported as NA or Inf in Table A1. However, for deeper stations with three or more profiling depths, the regression yielded highly consistent fits (median Mean $R^2$ > 0.93, excluding PL1), indicating a well-mixed water column and stable ambient illumination. The strong statistical performance of these multi-depth profiles provides high confidence in the underlying sampling methodology, thereby validating the use of the deterministic two-point calculations for the shallower stations where additional depths were not physically attainable.

For the derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$, it should be noted that PL1 was restricted to two valid depths, reducing the calculation to a deterministic two-point derivation (yielding a mathematical $R^2$ of 1.00 and precluding standard error; Table A1). For all other multi-depth stations, wavelengths were systematically excluded from the analysis if the regression yielded an $R^2$ < 0.85. This strict linearity criterion confirmed the robustness of the derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$, ensuring that the signal was not compromised by the noise floor at greater depths. Furthermore, the maintenance of high linearity ($R^2$ ≥ 0.85) suggests that bottom reflectance did not significantly contaminate the slope calculation for the retained stations, as such boundary effects would typically induce non-linear curvature in the log-transformed depth profiles.

For visualisation purposes, high-frequency noise inherent in the raw measurements was smoothed using a 10-point moving average filter. A 10 nm smoothing window was specifically selected as it effectively minimises random instrumental noise and environmental artefacts, whilst safely preserving the broader, fundamental spectral absorption features of the OACs. Raw, unsmoothed data were used for further quantitative analyses. Data in the spectral extremes were also masked to exclude instability and low signal-to-noise ratios resulting from high water absorption and atmospheric interference, considering only the wavelength range from 400 to 750 nm.

2.3.2. Empirical Relationship Between $K_0$ and Derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$

To systematically evaluate the relationship between $K_0$ and $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$, the analysis was conducted in four sequential steps. First, the two attenuation coefficients were plotted for each station to visually assess baseline differences in their spectral shape and magnitude. Second, to quantify the divergence between the coefficients, spectral attenuation ratio ($K_{\mathrm{e}\mathrm{f}\mathrm{f}}/K_0$) was calculated across all measured wavelengths. Third, the historical theoretical approximation ($K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ ≈ 2$K_{\mathrm{d}}$) was tested using the $K_0$. To determine if a comparable $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ ≈ 2$K_0$ relationship holds true within these highly scattering Case 2 waters, the empirical dataset was evaluated against this theoretical 1:2 line. A zero-intercept linear regression was fitted to the data to derive a region-specific empirical equation, and the goodness-of-fit was quantified using $R^2$.

Finally, to assess the validity of applying a static linear multiplier, a residual analysis was conducted. The residuals from the empirical regression were compared against those generated by the 1:2 assumption to evaluate relative predictive performance. A locally estimated scatterplot smoothing (LOESS) regression was subsequently applied to the empirical residuals to detect any underlying non-linear, wavelength-specific bias that a simple linear geometric might fail to capture across varying optical regimes.

2.3.3. Statistical and Machine Learning Model (Random Forest)

Statistical analyses and RF regression modelling were performed using R version 4.2.2 (R Core Team, Vienna, Austria, 2022) within the RStudio integrated development environment (version 2026.04.0, Build 526; Posit Software, PBC, Boston, MA, USA). To classify regional variability, K-Means clustering was applied to the OACs. Prior to clustering, all constituent concentrations were standardised (Z-score normalised) to account for their differing magnitudes and units, ensuring no single variable disproportionately influenced the distance metric. The optimal number of clusters (K = 3) was robustly determined using the elbow method, establishing three distinct OWTs. To guarantee convergence on the global optimum, the algorithm was configured with 25 random initialisations (nstart = 25).

To quantify the multivariate and non-linear influence of water constituents and wavelengths on attenuation, a RF regression model was implemented using the randomForest package in R [39]. This algorithm, based on the method by Breiman [40], was specifically selected for its robustness in handling noisy in situ measurements and its capability to resolve complex, non-linear interactions between biogeochemical variables without assuming parametric distributions.

Prior to model training, the integrated hyperspectral dataset (comprising n = 4364 paired derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ and $K_0$) was compiled, designating $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ and $K_0$ as the target variables, and Chl-a, Phaeo, TSM, $a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}}\left(440\right)$, and Wavelength as predictive features. Wavelength was specifically integrated as a predictor to allow the RF to account for the inherent spectral dependencies of light absorption and scattering processes across the visible domain. The dataset is scaled prior to running the model to minimise features’ range bias.

To ensure reproducibility, a random seed was initialised. Given the high spatial heterogeneity of the limited coastal stations, traditional data splitting risks excluding entire optical regimes from the training process. Therefore, the models were trained on the full dataset using an ensemble of 1000 decision trees (ntree = 1000), with the number of variables tried at each split ($m_{\mathrm{t}\mathrm{r}\mathrm{y}}$) set to 1. Model performance and predictive accuracy were evaluated using the algorithm’s internal Out-of-Bag (OOB) samples to provide an unbiased, cross-validated assessment of model accuracy and variance explained.

Finally, the relative influence of each OACsand Wavelength on derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ and $K_0$ was assessed via variable importance analysis. Importance was evaluated using the Percent Increase in Mean Squared Error (%IncMSE), which quantifies the loss in OOB predictive accuracy when a variable is randomly permuted, providing a more robust and physically meaningful metric than Node Purity. For comparison across models, these %IncMSE values were subsequently normalised to 100%.

3. Results

3.1. Optical Characterisation and Classification of Water Types

Data were collected from 12 geographical locations along the southern Baltic Sea coast, encompassing a diverse range of optical conditions, reflecting the transition from estuarine to open coastal water. The observed water depths ranged from 1.2 m to 4.0 m, with Secchi depths varying between 1.2 m and 3.8 m. In half of the sampled stations, the Secchi depth extended to the seafloor, indicating full water column transparency to the bottom (

Table A2. Summary of sampling stations, showing geographical location (Latitude and Longitude in Decimal Degrees, DD), water Depth (m) and concentrations of key optically active constituents, OACs. Constituents include chlorophyll-a (Chl-a; mg m⁻³), phaeopigments (Phaeo; mg m⁻³), total suspended matter (TSM; g m⁻³), coloured dissolved organic matter absorption at 440 nm ($a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}}\left(440\right)$; m⁻¹). Sample ID abbreviates the stations. Secchi depth represents a minimum transparency (bottom visible) rather than the true optical extinction depth for many stations where it equals to or very close to bottom. Note: While this table summarises the 16 physical samples, the subsequent statistical modelling and empirical regressions are performed on the hyperspectral data (comprising matched pairs of $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ and $K_0$ at a specific 1 nm wavelength band for a given sample) based on 14 valid samples.

Date	Station	Sample ID	Latitude (DD)	Longitude (DD)	Water Depth (m)	Secchi Depth (m)	Chl-a (mg m−3)	Phaeo (mg m−3)	TSM (g m−3)	${\mathit{a}}_{\mathbf{C}\mathbf{D}\mathbf{O}\mathbf{M}}\left(440\right)$ (m−1)	Optical Classification
16 June 2022	Warnemünde	MS1	54.1845	12.0953	4.0	3.0	6.49	3.89	6.20	0.03	Mesotrophic/Coastal
21 June 2022		MS2			4.0	3.0	6.63	4.12	3.50	0.08
23 June 2022	Schnatermann	SH1	54.1738	12.1415	3.0	2.0	21.9	9.82	25.3	0.24	Estuarine/ High Turbidity
28 June 2022		SH2			3.0	2.0	31.3	15.1	5.50	0.15
12 July 2022	Rerik	RR1	54.1021	11.6121	3.0	2.0	5.67	3.25	2.00	0.07	Mesotrophic/Coastal
14 July 2022		RR2			3.0	2.0	4.59	2.35	0.33	0.08
7 July 2023	Kühlungsborn Marina	KB2	54.1536	11.7722	1.5	1.5	9.43	4.19	16.0	0.12	Clear/Low Turbidity (TSM-dominated)
	Wismar Marina	WS2	53.9099	11.4379	3.2	3.2	8.09	3.56	11.5	0.05
11 July 2023	Timmendorf Marina	PL1	53.9919	11.3735	1.2	1.2	2.21	1.53	11.7	0.04
12 July 2023		PL2 *			1.5	1.5	2.61	1.53	11.7	0.04
1 August 2023	Nienhagen	NHB *	54.1662	11.9442	3.1	3.1	3.41	1.71	14.5	0.01
	Heiligendamm	HLB	54.1467	11.8438	3.8	3.8	4.68	1.76	11.9	0.04
	Kühlungsborn	KBB	54.1563	11.7514	3.2	3.2	5.26	2.57	12.3	0.01
2 August 2023	Hüttelmoor	HMB	54.2205	12.1646	3.2	3.0	4.22	2.39	11.8	0.02
	Graal Müritz	GMB	54.2612	12.2367	2.8	2.8	1.21	0.81	13.5	0.04
	Dierhagen	DHB	54.3004	12.3371	3.8	2.0	1.95	1.23	14.0	0.03

* Sample excluded from final analysis due to non-physical behaviour (i.e., increasing reflectance or irradiance with depth). Optical classification of water types based on K-Means clustering of Chl-a, TSM and CDOM ($a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}}\left(440\right)$). Cluster mean concentrations are: Mesotrophic/Coastal (Chl-a 5.85 mg m⁻³, TSM 3.00 g m⁻³ and $a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}}\left(440\right)$ 0.07 m⁻¹); Estuarine/High Turbidity (Chl-a 26.6 mg m⁻³, TSM 15.4 g m⁻³ and $a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}}\left(440\right)$ 0.20 m⁻¹), and Clear/Low Turbidity (TSM-dominated) (Chl-a 4.31 mg m⁻³, TSM 12.9 g m⁻³, and $a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}}\left(440\right)$ 0.04 m⁻¹).

in Appendix A). Prior to final attenuation analyses, two samples(PL2 and NHB) were excluded due to non-physical radiative transfer behaviour (i.e., increasing reflectance with depth), resulting in a final dataset of 14 samples.

The concentrations of OACs exhibited significant spatial variability across the study area (Figure 2). Chl-a concentrations ranged from a minimum of 1.21 mg m⁻³ at Graal Müritz (GMB) to a maximum of 31.3 mg m⁻³ at Schnatermann (SH2). TSM concentrations followed a similarly wide pattern of variability, with the lowest concentration observed at Rerik (RR2, 0.33 g m⁻³) and the highest at Schnatermann (SH1, 25.3 g m⁻³). Furthermore, $a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}}\left(440\right)$ ranged from 0.01 m⁻¹ in the clearer coastal waters at Nienhagen (NHB) and Kühlungsborn (KBB) to 0.24 m⁻¹ in the estuarine-influenced waters (SH1).

Based on K-Means clustering of these core constituents, the 14 samples were categorised into three distinct OWTs to systematically partition the varying optical environments (Figure 3):

• Clear/Low Turbidity (TSM-dominated): Characterised by high particulate scattering relative to phytoplankton biomass. This cluster encompasses the majority of the open coastal stations, including Kühlungsborn Marina (KB2), Wismar Marina (WS2), Timmendorf Marina (PL1), Heiligendamm (HLB), Kühlungsborn (KBB), Hüttelmoor (HMB), Graal Müritz (GMB), and Dierhagen (DHB), with cluster means: Chl-a 4.31 mg m⁻³, TSM 12.9 g m⁻³, and $a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}}\left(440\right)$ 0.04 m⁻¹.
• Mesotrophic/Coastal: Characterised by moderate biological activity and lower overall turbidity. This cluster groups the intermediate coastal stations at Warnemünde (MS1, MS2) and Rerik (RR1, RR2), with cluster mean: Chl-a 5.85 mg m⁻³, TSM 3.00 g m⁻³, and $a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}}\left(440\right)$ 0.07 m⁻¹).
• Estuarine/High Turbidity: Characterised by elevated organic loads and strong CDOM influence. This cluster is entirely driven by the highly turbid Schnatermann stations (SH1, SH2), with cluster means: Chl-a 26.6 mg m⁻³, TSM 15.4 g m⁻³, and $a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}}\left(440\right)$ 0.20 m⁻¹.

3.2. Vertical Decay of Spectral Reflectance

The vertical attenuation of the isolated target signal (serving as a proxy for active beam attenuation) was examined through spectral reflectance decay profiles. Figure 4 presents these vertically stacked profiles for three representative stations corresponding to the defined OWTs: (a) SH1, Estuarine/High turbidity, (b) RR1, Mesotrophic/Coastal, and (c) KBB, Clear/Low Turbidity. In all cases, the apparent spectral reflectance decreased with depth, although the magnitude and spectral shape of this decay varied spatially.

The spectral shape of the reflectance reveals distinct regional variations, particularly in the shorter wavelengths (400 to 500 nm). For the estuarine and mesotrophic stations (SH1 and RR1; Figure 4a,b), the reflectance magnitude rises gradually across the blue-to-green spectrum. In contrast, the clearer coastal baseline station (KBB; Figure 4c) exhibits a distinctly steeper initial slope, rising rapidly from 400 nm before peaking in the green spectrum.

Furthermore, the absolute magnitude of reflectance varied considerably between the regimes. Station RR1 recorded the highest shallow-target (0.5 m) reflectance, peaking at approximately 18% around 580 nm, which was significantly higher than the peak magnitudes observed at SH1 (approximately 9.5%) and KBB (approximately 8%). The profiles also demonstrate variations in the depth limit of target detection. In the highly turbid SH1 waters, the target signal decays rapidly, merging with the ambient background (“No Target” baseline) between 1.5 m and 2.0 m. Conversely, the less turbid waters at KBB maintain a distinct target signal separation from the background up to the maximum measured depth of 2.5 m.

3.3. Spectral Comparison of Attenuation Coefficients

Figure 5 compares the spectral distribution of $K_0$ and derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$. In most cases except within the clearer coastal waters, the derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ consistently exceeded $K_0$, although the magnitude and spectral shape of this divergence varied spatially.

The spectral gap between the two coefficients reveals distinct regional variations, particularly across the shorter wavelengths (400 to 550 nm). For the estuarine and mesotrophic stations (SH1 and RR1; Figure 5a,b), the divergence remains notably wide across the entire blue-to-green spectrum. In contrast, for the clearer coastal baseline station (KBB; Figure 5c), this divergence narrows considerably, particularly within the green (approximately 500 to 550 nm) and red (>650 nm) spectral regions.

Furthermore, the absolute magnitude of both attenuation coefficients varied considerably between the regimes. Station SH1 recorded highest attenuation values, with $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ exceeding 8 m⁻¹ and $K_0$ approaching 3 m⁻¹ in the shorter wavelengths (near 400 nm). These peak magnitudes systematically decreased moving towards the clearer coastal waters: at station KBB, both coefficients generally remained below 2 m⁻¹ across the measured spectrum. The profiles demonstrate that as overall water column attenuation decreases, the absolute spectral gap between $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ and $K_0$ correspondingly diminishes.

Figure 6 shows the spectral ratio of $K_0$ and derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$, with a theoretical baseline of 2, representing the common assumption for two-way diffuse attenuation. The magnitude and spectral shape of this ratio exhibited extreme spatial variability across the contrasting water types.

In the optically complex estuarine and mesotrophic regimes (SH1 and RR1; Figure 6a,b), the spectral ratio predominantly exceeded the theoretical baseline of 2. For the highly turbid SH1 station, the ratio remained elevated across the entire spectrum, fluctuating between approximately 2.5 and 4.8, with the maximum divergence observed in the green-to-yellow wavelengths (550–600 nm). The mesotrophic RR1 station exhibited an even more extreme mid-spectrum divergence: while the ratio started near 2 in the blue spectrum (400 nm), it spiked sharply to nearly 10 around 550 nm before steeply declining towards the red spectrum.

Conversely, the clearer coastal baseline station (KBB; Figure 6c) demonstrated a fundamentally different attenuation relationship. Across the entirety of the measured spectrum, the ratio remained strictly below the 2 threshold, generally fluctuating between 0.8 and 1.5. This structural shift highlights that the relative scattering penalty applied to $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ is highly wavelength-dependent and shifts drastically between different optical environments.

3.4. Empirical Relationship Between Derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ and $K_0$

Figure 7 shows the relationship between derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ and $K_0$ when compared to the linear regression model based on the historical geometric assumption that $K_{\mathrm{e}\mathrm{f}\mathrm{f}}\approx 2K_{\mathrm{d}}$. The derived empirical relationship is given by Equation (6):

$$K_{\mathrm{e}\mathrm{f}\mathrm{f}}={2.33K}_0$$

(6)

A positive overall relationship with $R^2$ = 0.65 has exceeded the theoretical geometric multiplier. Furthermore, significant data scatter surrounds the bulk regression line. This wide dispersion confirms that a singular, fixed mathematical coefficient is insufficient to accurately predict active signal attenuation across diverse OWTs, highlighting the need to examine constituent-specific optical drivers.

Figure 8 further differentiates the performance of the empirically derived fit against the theoretical geometric assumption. The residuals for the empirical zero-intercept regression (Figure 8a) are centred around the zero-error baseline across the measured spectrum. This demonstrates an unbiased fit, although heteroscedasticity is evident, with the variance of the residuals increasing substantially at higher $K_0$. Conversely, the theoretical model (Figure 8b) exhibits a distinct, systemic positive bias. As $K_0$ increases, the theoretical assumption consistently underestimated the true magnitude of the $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$, leaving the majority of residuals falling well above zero.

3.5. Drivers of Attenuation Variability

The RF models showed high predictive accuracy, accounting for 80.3% of the observed variance for $K_0$ and 78.7% for $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ (Figure 9). The importance distribution for $K_0$ is relatively balanced across the features, with $a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}}\left(440\right)$ (19.5%) and pigments (Chl-a at 18.4%, Phaeopigments at 18.2%) acting as significant drivers alongside TSM (17.4%) and Wavelength (17.3%). In contrast, the derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ exhibits a distinct shift in hierarchy, dominated by TSM (28.0%) and Wavelength (24.3%). While pigments and CDOM remain influential, their relative contribution to the derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ variance was lower than that observed for $K_0.$

4. Discussion

4.1. The Physical Basis of Divergence

A central finding of this study is the spectral characterisation of the divergence between derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ and ambient $K_0$ across highly contrasting Case 2 optical water types. While historical deep-water models frequently rely on a static geometric assumption based on diffuse attenuation ($K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ ≈ 2$K_{\mathrm{d}}$), our findings demonstrate that this relationship is fundamentally dynamic, wavelength-dependent, and governed by the prevailing particulate assemblage, when compared with ambient $K_0$. As shown by the spectral profiles across representative OWTs (Figure 5), the magnitude of the derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ does not simply scale with the ambient scalar $K_0$. Instead, this divergence shifts dramatically, narrowing considerably in clearer coastal waters while expanding massively across the entire visible spectrum in estuarine and mesotrophic environments.

This pronounced spatial and spectral variability is directly tied to constituent-specific drivers, specifically the severe angular impact of high particulate scattering in conditions where the scattering coefficient (b) often significantly exceeds absorption (a) [11,12]. The machine learning importance analysis (Figure 9) provides explicit empirical evidence for this divergence. By identifying TSM as the dominant driver of derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ variability (28.0%), compared to its lesser role in ambient $K_0$ (17.4%), the models highlight a condition of ‘scattering penalty’ inherent to active LiDAR systems. In the optical literature, this is also recognised as FOV loss or spatial rejection, where the narrow angular acceptance of the active receiver physically excludes enhanced scattered photons from the returning signal [14,41,42]. In the ambient light field, $K_0$ acts as a scalar measurement, integrating scattered photons from all angular trajectories [16,23]. Conversely, the narrow FOV of the active receiver dictates that even minor scattering deflections effectively remove photons from the detectable return path [13,43,44]. Consequently, the derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ scales much more closely with total beam attenuation (c = a + b) due to this geometric rejection (scattered light is excluded by the sensor and counted as “loss”) [10,13] driving the optical divergence, while $K_0$ remains constrained by absorption (a) [16].

This spectral divergence is most pronounced in the 500–650 nm range (Figure 6a,b), a region that typically corresponds to the “minimum absorption window” in coastal waters [45,46]. Paradoxically, the lower absorption in this window enhances the survival of photons, allowing them to travel further and undergo more scattering events, a mechanism particularly relevant to the highly scattering conditions characterising our sampling sites [47,48]. For the ambient $K_0$, these scattered photons are retained within the wide angular acceptance of the deployed spherical collector (oriented to measure the downwelling field). However, for the active, collimated LiDAR signal (and so the derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$), these same photons are scattered out of the receiver’s geometric acceptance cone [13,49,50]. This ‘scattering penalty’ effectively removes photons from the active signal in the specific spectral bands where transmission is theoretically highest [51], leading to the distinct peak observed in the $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$/$K_0$ spectral ratio for turbid waters. However, this penalty is highly dependent on the prevailing optical regime [42]: as evidenced in the clearer coastal baseline water (KBB; Figure 6c), where the significantly lower TSM concentration caused the K_eff/K₀ spectral ratio to remain strictly below the theoretical threshold of 2. This structural shift confirms that the extreme divergence observed in coastal waters is not a universal constant, but a direct consequence of particulate scattering.

To quantify the bulk magnitude of this divergence across all sampled environments, the zero-intercept linear regression (Figure 7) establishes an empirical, region-specific multiplier of 2.33 ($K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ = 2.33$K_0$). This slope mathematically formalises the effect of scattering penalty in the context of Case 2 waters. However, the considerable data dispersion ($R^2$= 0.65) and subsequent residual analysis (Figure 8) show that applying a static linear multiplier cannot fully capture the optical complexity of the studied system, and that the accuracy of the assessed static multipliers fundamentally depends on overall water turbidity. Both the empirical 2.33 fit (Figure 8a) and the historical assumption factor of 2 (Figure 8b) exhibit a distinct, non-linear failure pattern: they consistently underestimate the active attenuation in relatively clear waters ($K_0$< 1.0 m⁻¹) and subsequently overestimate it as conditions become highly turbid. The threshold at which these models cross into overestimation is quantitatively distinct (Residual Error = 0): the 2.33 multiplier begins to systematically overestimate derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ at an ambient attenuation of $K_0$ ≈ 1.1 m⁻¹, while the historical assumption factor of 2 delays this crossover until $K_0$ ≈ 1.5 m⁻¹, resulting in a much broader underestimation across the bulk of the dataset. Because this study focuses predominantly on complex, turbid environments ($K_0$ > 1.5 m⁻¹), the application of any static linear multiplier inherently results in a systematic overestimation of the active signal loss. This overestimation is driven by the non-linear decay of light in extreme Case 2 waters; once turbidity reaches a critical threshold where the scattering penalty maximises, the physical signal decay flattens out, rendering linear predictions fundamentally incapable of capturing reality [42,48].

4.2. Vertical Decay and the Empirical Limits of $K_{eff}$

The theoretical divergence between derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ and $K_0$ directly manifests as a serious operational constraint on target detection depth in Case 2 waters. The vertical spectral reflectance profiles (Figure 4) visually capture this constraint, showing how the ‘scattering penalty’ established in Section 4.1 effectively truncates the depth limit of active sensing systems based on the prevailing OWTs.

In the estuarine, highly turbid regime (SH1; Figure 4a), the $R_{\mathrm{r}\mathrm{s}}$ experiences significant decay. Despite the theoretical existence of a “minimum absorption window” in the green spectrum, the highTSM load (up to 25.3 g m⁻³) and $a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}}\left(440\right)$ concentration (0.24 m⁻¹) drive extreme geometric scattering exclusion and absorption. Consequently, the target signal becomes indistinguishable from the ambient water column background (‘No Target’ baseline) between 1.5 m and 2.0 m. This rapid signal extinction proves that in environments where $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ is both heavily scattered and absorbed, penetration depth is inherently limited not by the lack of photons, but by the inability of those photons to maintain a coherent return path to the receiver [46,48,52].

Conversely, the clear coastal regime (KBB; Figure 4c) demonstrates the system’s performance at significantly lower TSM (12.3 g m⁻³) and $a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}}\left(440\right)$ (0.01 m⁻¹) concentrations. The derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ remains closely aligned with the ambient $K_0$ (as seen in Figure 6c), because the geometric exclusion of scattered light is minimised, and absorption reduced. The physical decay of light thereby maintains structural integrity and clear separation from the background up to the maximum measured depth of 2.5 m.

Interestingly, the mesotrophic regime (RR1; Figure 4b) presents a unique optical middle-ground. This station recorded the highest absolute shallow-target reflectance (approximately 18% at 0.5 m, compared to <10% at SH1 and KBB). This anomaly can be attributed to the specific constituent balance of the mesotrophic water column: it contains sufficient suspended particulate matter to generate strong initial backscatter (TSM = 2.0 g m⁻³), but lacks the extreme $a_{\mathrm{C}\mathrm{D}\mathrm{O}\mathrm{M}}\left(440\right)$ absorption (0.07 m⁻¹) found in the estuarine stations that would immediately quench the signal. However, because $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ remains highly sensitive to that scattering, the signal still decays much faster than in the clearer coastal waters, creating a “bright but shallow” operational window.

These vertical profiles confirm the necessity of differentiated bio-optical modelling for Case 2 waters. Algorithms predicting active signal attenuation cannot rely on scalar ambient light proxies: instead, they must prioritise bulk particulate and dissolved contributions [13,52,53]. Specifically, operational models should explicitly integrate the region’s OACs to account for both the geometric scattering driven by suspended particulates and strong absorption imposed by dissolved matter. Furthermore, as established in Section 4.1, this constituent-driven decay is fundamentally wavelength-dependent. Penetration models must therefore be spectrally resolved to accurately predict how different OWTs shift the signal loss across the visible spectrum. This complex, non-linear interplay between specific OACs and spectral scattering-absorption directly motivates the predictive machine learning approach explored in the following section.

4.3. Machine Learning for Optical Monitoring in Case 2 Waters

To overcome the limitations of static geometric scaling established previously, optical monitoring strategies for Case 2 waters must move beyond fixed Apparent Optical Properties (AOPs) to Inherent Optical Properties (IOPs) scaling. Predicting active signal decay in highly scattering coastal environments requires non-linear frameworks capable of resolving the complex interactions between specific OACs and the collimated light field. As the identification of distinct OWTs in this study demonstrates, applying a singular, “one-size-fits-all” regression inherently fails across spatially heterogeneous coastal zones. Future active sensors could utilise the spectral shape of the return signal to classify the water type (e.g., Mesotrophic vs. TSM-dominated) and dynamically switch to a class-specific attenuation model. This would significantly improve the accuracy of subsurface retrievals in spatially heterogeneous coastal zones.

Consequently, operational penetration models must transition toward adaptive algorithms. To address this, a RF machine learning approach was applied to predict the derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ directly from the in situ constituent matrix (Figure 9). Rather than relying on a fixed geometric factor, this algorithm dynamically weights the prevailing OWTs, successfully capturing the non-linear divergence that confounds traditional linear scaling. By directly integrating the measured concentrations of key OACs, the RF regression explained 78.7% of the variance in the derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$. This high predictive accuracy confirms that signal loss in Case 2 waters is driven by a complex matrix of particulate scattering and dissolved absorption, rather than a simple geometric response to ambient light. However, while the RF framework demonstrates superior predictive accuracy, it represents a regional optimisation based on the 12 distinct geographic sites in the Baltic Sea estuarine environments. Its current operational application serves as a structural proof-of-concept for constituent-driven modelling, rather than yielding a universal predictive constant, and must be retrained before deployment in structurally different Case 2 waters.

Relying on a fixed geometric factor systematically under-corrects for attenuation, leading to depth overestimation [11,12] driven by the high, variable scattering of suspended sediments [54,55]. While dynamic parameterisation of a geometric factor is theoretically possible, the challenge of accurately inverting $R_{\mathrm{r}\mathrm{s}}$ for scattering properties in Case 2 waters makes this difficult. A more robust approach in the context of LiDAR is to circumvent scaling entirely by retrieving the beam attenuation coefficient (c) via active LiDAR ($K_{\mathrm{L}\mathrm{i}\mathrm{d}\mathrm{a}\mathrm{r}}$). Techniques that explicitly solve for c, such as LiDAR Radiative Transfer Equation (RTE) inversion [56], single-photon elastic-Raman normalisation [21], and sediment-adaptive bathymetry models [11,53], demonstrate superior robustness by eliminating the uncertainties of approximate geometric factors.

4.4. Limitations and Future Perspectives

Restricting the study to shallow waters (<4.0 m) prevents asymptotic instability and bottom reflectance artefacts [11]. Because these boundary effects invalidate standard reflectance models, we rigorously excluded all data exhibiting increasing reflectance with depth to ensure signal integrity. However, this protocol highlights a critical limitation: avoiding bottom-contaminated data restricts spatial coverage in the shallowest nearshore fringes. To advance optical monitoring in these optically shallow environments, future research must transition from avoiding bottom contamination to explicitly resolving it. Field campaigns should prioritise in situ characterisation of diverse bottom albedos, such as varying sand compositions, macroalgae beds, and mud [57,58]. Recent research demonstrates that integrating these explicitly constrained bottom-boundary conditions into semi-analytical radiative transfer models is essential to overcome the retrieval errors commonly associated with unparameterised benthic reflectance in optically complex coastal zones [59].

Methodologically, the target subtraction technique utilised to isolate derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ is an approximation of the true two-way photon path, and the inherent ‘target blocking effect’ may introduce minor over-subtraction biases [13]. Future research should incorporate in situ black target calibrations. Lowering a highly absorbing black target to identical depths allows for the precise isolation of the intervening water column’s upwelling radiance, preventing the over-subtraction of deep-water signals. Analytically, the study also lacks direct measurements for non-algal particle absorption ($a_{\mathrm{N}\mathrm{A}\mathrm{P}}$) and backscattering (${bb}_{\mathrm{N}\mathrm{A}\mathrm{P}}$). While TSM served as a robust proxy due to its strong covariance with NAP in turbid systems [12,19], this precludes the resolution of mass-specific sediment coefficients [19,60,61]. Future campaigns must incorporate comprehensive IOP budgets to precisely decouple the scattering and absorbing roles of the mineral fraction.

Despite these methodological boundaries, focusing exclusively on the shallow domain contributes important ground-truth data in regions often masked in passive satellite products due to bottom contamination. This study successfully characterises these ‘blind spots’, where standard sensing fails. A primary strength of this approach is the robust, in situ hyperspectral dataset (n = 4364). However, it must be acknowledged that because these observations derive from 14 samples, the dataset exhibits inherent spectral autocorrelation. Consequently, the effective independent sample size is smaller than the nominal n, which suggests that the OOB variance estimates yielded by the RF regression may be slightly optimistic. Despite this, these findings capture real-world optical complexities across a diverse gradient of highly scattering Case 2 waters and provides a valuable empirical baseline that inherently accounts for the complex, overlapping physical interactions that theoretical models often struggle to replicate [62,63].

Ultimately, the distinct behavioural divergence between active and passive signals advocates for a dedicated sensor fusion approach. By synergising passive radiometry for OWT classification and active LiDAR for vertical profiling, future systems can overcome the fixed geometric scaling limits of Case 2 waters. This combination leverages the strengths of spectral discrimination and depth resolution simultaneously, defining the next critical frontier in coastal optical monitoring [64,65,66].

5. Conclusions

This study provides a critical physical and spectral re-evaluation of radiative transfer assumptions in shallow, highly scattering Case 2 waters. By decoupling the drivers of derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ and ambient $K_0$ optical signals along the southern Baltic Sea coast, we demonstrate that traditional theoretical geometric multipliers fundamentally fail due to the non-linear angular rejection (exclusion) of scattered light. Ultimately, this work advances coastal bio-optics through four core contributions:

• Spectral characterisation of the Scattering Penalty: We established that the divergence between derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ and ambient $K_0$ is highly wavelength-dependent and dictated by the prevailing optical regime. While derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ closely tracks ambient attenuation in clearer waters, a severe ‘scattering penalty’ drives massive divergence in turbid estuarine environments. This divergence peaks paradoxically within the 500–650 nm minimum absorption window, where high photon survival leads to increased scattering events that are systematically excluded by narrow-field active sensors.
• The Failure of Static Geometric Scaling: To quantify regional bulk divergence, we derived an empirical relationship of derived $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ = 2.33$K_0$ ($R^2$ = 0.65). However, comprehensive residual analysis confirms that static linear multipliers are fundamentally incapable of capturing complex radiative transfer. Fixed geometric factors systematically underestimate active attenuation in clearer waters ($K_0$ < 1.1 m⁻¹) and vastly overestimate it as environments cross into high turbidity, highlighting the non-linear decay of light in Case 2 waters.
• Constituent-Driven Attenuation: Standardised Random Forest regression (explaining 78.7% of $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$ variance) revealed a fundamental divergence in how optical constituents influence signal loss. The algorithm explicitly identified total suspended matter (TSM) as the dominant driver of $K_{\mathrm{e}\mathrm{f}\mathrm{f}}$, demonstrating the active signal’s acute vulnerability to particulate scattering, whereas ambient $K_0$ remains largely constrained by absorption. This proves that predictive attenuation models cannot rely on scalar ambient light proxies but must prioritise explicit bulk particulate and dissolved contributions.
• Applied Value for Coastal Sensor Fusion: The identification of distinct optical water types (OWTs) underscores that a singular, “one-size-fits-all” regression inherently fails across spatially heterogeneous coastal zones. Consequently, operational penetration algorithms must transition toward adaptive modelling. We advocate for a dedicated sensor-fusion strategy: deploying passive optical sensors to classify prevailing OWTs and subsequently dynamically applying water-type-specific attenuation models to active LiDAR returns. This framework circumvents the limits of geometric scaling, substantially improving the accuracy of subsurface depth and property retrievals in optically complex coastal environments.