A Novel Deep Learning-Based Soil Moisture Prediction Model Using Adaptive Group Radial Lasso Regularized Basis Function Networks (AGRL-RBFN) Optimized by Hierarchical Correlated Spider Wasp Optimizer (HCSWO) and Incremental Learning (IL)

Abstract

Soil moisture serves as a critical factor in the hydrological cycle, affecting plant growth, ecosystem health, and groundwater reserves. Current methods for monitoring and predicting it fail to account for the complexities introduced by climatic variations and other influencing factors, such as the effects of atmospheric interference and data gaps, leading to reduced prediction accuracy. To address these challenges, this study introduces a novel soil moisture prediction model based on remote sensing and deep learning, utilizing the Adaptive Group Radial Lasso Regularized Basis Function Networks (AGRL-RBFN) optimized by the Hierarchical Correlated Spider Wasp Optimizer (HCSWO) and incremental learning (IL) techniques. The proposed method for monitoring soil moisture utilizes hyperspectral and soil moisture data from a 2017 campaign in Karlsruhe, encompassing variables such as datetime, soil moisture percentage, soil temperature, and remote sensing spectral bands. The proposed methodology begins with comprehensive preprocessing of historical remote sensing data to fill gaps, reduce noise, and correct atmospheric disturbances. It then employs a unique seasonal mapping and grouping technique, enhanced by the AdaK-MCC method, to analyze the impact of climatic changes on soil moisture patterns. The model’s innovative feature selection approach, using HCSWO, identifies the most significant predictors, ensuring optimal data input for the AGRL-RBFN model. The model achieves an impressive accuracy of 98.09%, a precision of 98.17%, a recall of 97.24%, and an F1-score of 98.95%, outperforming existing methods. Furthermore, it attains a mean absolute error (MAE) of 0.047 in gap filling and a Dunn Index of 4.897 for clustering. Although successful in many aspects, the study did not investigate the relationship between soil moisture levels and specific crops, which presents an opportunity for future research aimed at enhancing smart agricultural practices. Furthermore, the model can be refined by integrating a wider range of datasets and improving its resilience to extreme weather conditions, thereby providing a reliable tool for climate-responsive agricultural management and water conservation strategies.

1. Introduction

Soil moisture acts as a fundamental intermediary between climate, hydrology, and agriculture, exerting a profound influence on evapotranspiration, precipitation, runoff, and ultimately agricultural productivity and ecosystem health. Although it represents a relatively small fraction of Earth’s total water volume, its variability drives drought and flood forecasting and underpins water–energy balance models [1]. In agricultural settings, accurate soil moisture data are indispensable for assessing drought severity, scheduling planting and irrigation, and forecasting yield needs that become ever more critical under increasingly variable climate conditions. Maintaining moisture within optimal ranges is vital; oversaturation can leach nutrients and promote root diseases, while deficits can lead to crop stress or failure. Traditionally, in situ probes have guided irrigation and water-conservation practices [2,3,4], but satellite remote sensing now offers scalable, high-resolution (0.25° × 0.25°) alternatives via missions such as ESA’s SMOS and NASA’s SMAP. These platforms, which employ L-band microwave sensing combined with Palmer-model retrievals and ensemble Kalman filtering, provide both surface and subsurface moisture profiles, as well as anomaly detection [5,6]. Machine learning and deep learning methods have further advanced soil moisture estimation by extracting complex spatial and temporal features from these datasets. Classical algorithms, including SVM, KNN, RF, and ANN, yield marked improvements in accuracy and control overfitting. At the same time, CNNs excel at spatial pattern recognition and LSTMs capture temporal dynamics, functioning as “virtual sensors” that learn from neighboring transducers [7,8,9,10,11,12,13,14,15,16].

Recent works illustrate this progress: CNN-based estimators achieve high precision but struggle in low-variance moisture environments [17]; RF-SVM-LSTM hybrids excel in wireless sensor networks albeit at a high computational cost [18]; GPR-ANN architectures demonstrate low RMSE but require intensive hyperparameter tuning [19]; GRU models improve performance but often omit rigorous feature selection [20]; ensemble trees like ETr and RF perform well on multi-source imagery yet suffer overfitting due to sparse resolution [21,22], and LSTM networks generally outperform others but remain sensitive to soil-texture heterogeneity [23]. Despite these gains, significant gaps persist. Many models do not fully address climate-driven variability in moisture dynamics—such as temperature- and evaporation-induced seasonal shifts or extreme events like droughts and heatwaves [24,25,26]—nor do they adequately correct for atmospheric distortions (haze, scattering, absorption) that can degrade retrieval accuracy [27,28,29]. Furthermore, cloud cover and sensor malfunctions introduce persistent data gaps [30,31,32], and soil moisture exhibits complex spatial and temporal variability that single-framework models and simplified hydrological analogues (e.g., the Leaky Bucket) fail to capture [33,34,35,36,37]. To overcome these limitations, we present a novel framework that seamlessly integrates remote sensing and deep learning. Seasonal grouping and gap filling utilize the AdaK-MCC algorithm, noise reduction employs the Savitzky–Golay filter, and temporal patterns are extracted via Frequency Weighted Fourier Coefficient–based Seasonal Decomposition (FWFCSD). The core predictor is an Adaptive Group Radial Lasso–Regularized Basis Function Network (AGRL-RBFN), with feature selection optimized by a Hierarchical Correlated Spider Wasp Optimizer (HCSWO) and continuous adaptability ensured by incremental learning (IL). Our approach fulfils four key objectives: (1) forecasting moisture trends under climate variability; (2) enhancing prediction precision through noise filtering; (3) compensating for missing data via robust clustering; and (4) consolidating multi-scale temporal information. Achieving 98.09% accuracy using only remote sensing inputs, this system functions as a “smart moisture advisor,” akin to a seasoned farmer who adjusts irrigation based on rainfall and temperature histories. It fills data gaps, adapts in real-time, and delivers actionable insights in regions where ground networks are sparse.

The remainder of this paper is structured as follows. Section 2 outlines the methodology, Section 3 presents the experimental results and comparisons, Section 4 discusses the implications and limitations, and Section 5 concludes with future research directions.

2. Materials and Methods: Soil Moisture Prediction Using AGRL-RBFN with IL

The study aims to forecast soil moisture using remote sensing data through the AGRL-RBFN with an IL method while observing climatic changes through seasonal mapping and classification. The main phases of the study consist of preprocessing, seasonal mapping and classification, analysis of diverse patterns and multivariate relationships, as well as feature extraction and selection. Figure 1 depicts the complete workflow for forecasting soil moisture by utilizing remote sensing data integrated with deep learning methods. It illustrates the step-by-step phases, including preprocessing, seasonal mapping, feature extraction, and prediction using the AGRL-RBFN alongside the IL method.

The study suggests that the AGRL-RBFN model is likely implemented in Python using machine learning libraries such as TensorFlow or Keras for model training, along with Scikit-learn for preprocessing and baseline modeling. Data preprocessing involves techniques such as gap filling with AdaK-MCC, noise reduction using the Savitzky–Golay filter, and atmospheric correction, potentially employing SciPy or remote sensing libraries like GDAL or Rasterio. Visualization is performed using Matplotlib or Seaborn for plotting metrics. The custom optimization algorithm HCSWO must be implemented as it effectively handles feature correlation. The study suggests that a high-performance computing setup is necessary, featuring a multicore processor, an NVIDIA GPU with CUDA support, at least 16 GB of RAM, and SSD storage for handling large datasets. Python, along with libraries such as TensorFlow, Keras, and Scikit-learn, was used to create the AGRL-RBFN model utilizing incremental learning. An NVIDIA RTX 3080 GPU with 16 GB of VRAM, backed by an Intel Core i9 CPU and 64 GB of RAM, was used for training. It took approximately 2.7 h to complete the entire training cycle, which included preprocessing, clustering, and optimization. The viability of using this model for bigger hyperspectral datasets in future research is demonstrated by scalability evaluations, which reveal that the model training time grows linearly with the size of the input dataset. Furthermore, the incremental learning component supports real-time adaptation in operational contexts by guaranteeing low retraining time when new data are presented. To replicate the study, access the dataset at https://github.com/felixriese/hyperspectral-soilmoisture-dataset (Accessed on 3 January 2019). Set up a Python-based environment with the specified libraries and GPU support. Follow the original preprocessing, feature selection, and deep learning steps using the same settings and hyperparameters. Utilizing the provided scripts or code will further ensure the accurate reproduction of the results.

2.1. Dataset Description and Data Acquisition

The process of monitoring soil moisture through remote sensing and deep learning begins with the acquisition of hyperspectral and soil moisture data from a 2017 field campaign in Karlsruhe. The dataset encompasses various variables, including time, soil moisture percentage, soil temperature, and spectral bands collected via remote sensing, covering the duration of the campaign in the Karlsruhe region. The dataset used in this study originates from the 2017 Karlsruhe soil moisture field campaign and incorporates satellite-derived data from the SMAP and SMOS missions. It features a spatial resolution of 0.25° × 0.25° and a temporal resolution ranging from daily to weekly intervals, depending on satellite pass frequency. This level of granularity is consistent with leading global soil moisture monitoring protocols and supports both temporal trend analysis and regional-scale spatial modeling. To improve prediction accuracy, preprocessing techniques such as AdaK-MCC for gap filling, Savitzky–Golay filtering for noise reduction, and atmospheric correction were applied. Following preprocessing, the data underwent seasonal mapping, multivariate correlation analysis, feature extraction, and selection, culminating in the prediction of soil moisture using the AGRL-RBFN method. The study utilized hyperspectral and soil moisture data from a 2017 field campaign in Karlsruhe, encompassing variables such as date, soil moisture percentage, soil temperature, and spectral bands. The dataset was split into 80% for training and 20% for testing, though details on its overall size and spatial resolution are not explicitly provided. The data used in this study were collected during a field campaign conducted in Karlsruhe, Germany, from April to September 2017, during the growing season. It comprises 1250 weekly samples that show temperature, soil moisture percentage, and hyperspectral reflectance in 204 bands. The dataset includes information such as the vegetation index (NDVI), land use variations (e.g., agriculture and grassland), and soil type classifications (loamy sand and sandy loam). In addition to enabling the model to account for changes in crop covering and evapotranspiration rates, these agricultural parameters provide an important background for comprehending soil moisture variance. Data preprocessing involved gap filling using AdaK-MCC, noise reduction with Savitzky–Golay filtering, atmospheric correction, and feature selection to enhance prediction accuracy. The proposed model was tested using only a single dataset to validate its effectiveness and robustness in a controlled setting. Although the proposed model performs admirably on the Karlsruhe 2017 dataset, further research is necessary to determine its adaptability to different climates. Local climatic, agricultural, and geophysical factors greatly influence changes in soil moisture. To assess the model’s performance in various environmental situations, future advances of this study will involve validation using datasets from diverse climate regions, including dry, semiarid, and tropical areas. The model will be able to adapt to different soil types, plant coverings, and climatic conditions by integrating global datasets. Evaluating the model on a well-characterized, specific dataset enables the precise assessment of its performance metrics, such as MAE and correlation coefficients, without the variability introduced by multiple datasets. This approach helps establish a proof of concept and ensures that the model functions reliably before being applied more broadly. The dataset is divided so that 80% is used for training, with the remaining 20% reserved for testing. The collected remote-sensing data $Z$ for soil moisture are expressed as follows:

$$Z=\left\{Z_1,Z_2,Z_3,\dots ,Z_a\right\}$$

(1)

where each $Z_i$ denotes an individual soil moisture observation obtained via remote sensing at a specific spatial location and time instance, and $a$ represents the total number of observations in the dataset. These data are collected at a spatial resolution of ${0.25}^{\circ }\times {0.25}^{\circ }$ and a temporal resolution of daily to weekly intervals, depending on the satellite source. Remote sensing platforms such as NASA’s Soil Moisture Active Passive (SMAP) and the European Space Agency’s Soil Moisture and Ocean Salinity (SMOS) missions are used to acquire these measurements. The data include both surface and subsurface soil moisture values, integrated with atmospheric corrections and enhanced using ensemble filtering methods. Equation (1) thus encapsulates the spatiotemporal structure of the remote sensing data stream, which enables accurate and scalable soil moisture prediction across extensive geographical areas without the need for dense ground-based sensor networks.

2.2. Preprocessing

This preprocessing phase Z is aimed at ensuring more accurate and reliable soil moisture predictions from the remote sensing data.

2.2.1. Gap Filling

The AdaK-MCC method fills missing values Z by enhancing K-Means clustering (KMC) through Adaptive Momentum Coefficients (AMC) for better centroid initialization [38,39]. This approach enables centroids to explore the solution space more effectively, thereby avoiding local minima and enhancing clustering accuracy. By grouping similar observations, the method ensures more accurate and contextually relevant imputations. The centroid update mechanism combines momentum and the gradient of the loss function, resulting in dynamic and precise centroid movement. The mathematical formula of centroid update is given as follows [38]:

$$C_k^{\left(t+1\right)}=C_k^{\left(t\right)}+\alpha \cdot \nabla L+\beta \cdot \left(C_k^{\left(t\right)}-C_k^{\left(t-1\right)}\right)$$

(2)

where $\alpha$ is the learning rate, $\nabla L$ is the gradient of the loss function, and $\beta$ is the momentum coefficient. The momentum term $\left(C_k^{\left(t\right)}{-C}_k^{(t-1)}\right)$ ensures that the centroid moves dynamically, thereby enhancing the clustering process and leading to more accurate gap filling. Equation (2) defines how centroids in the K-Means clustering algorithm are updated using Adaptive Momentum Coefficients. The goal is to avoid local minima and improve centroid accuracy during gap filling.

Firstly, clusters Nc are selected from Z, and the centroids ${\varsigma }^j$ are initialized using AMC, where the centroids can explore the solution space more widely, thereby reducing the risk of local minima. ${\varsigma }^j$ is initialized as

$${\varsigma }^j=\vartheta •{\varsigma }^{j-1}+\left(1-\vartheta \right)\ast \sout{\nabla }\sout{G}$$

(3)

$$\sout{\nabla }\sout{G}=-{\displaystyle \frac{1}{a}}{\displaystyle \sum _{i=1}^a{\displaystyle \sum _{j=1}^{\beta }\left(h^{ij}•\left({\mathrm{Z}}^i,{\varsigma }^j\right)\right)}}$$

(4)

where ${\varsigma }^j$ and ${\varsigma }^{j-1}$ indicate the current and previous centroid, $\vartheta$ represents the momentum coefficient of centroid, $h^{ij}$ specifies the distance between $i^{\mathrm{th}}$ input Z and $j^{\mathrm{th}}$ centroid ${\varsigma }^j$, $\sout{\nabla }\sout{G}$ refers to the gradient loss function [39], Equations (3) and (4), initialization using current and previous centroid positions with momentum, a variant of gradient descent. Next, each data point is assigned to the nearest centroid based on the distance:

$${\rho }_i^{″}=\arg \underset{j}{\min }\left({‖\left({\mathrm{Z}}^i-{\varsigma }^j\right)‖}^2\right)$$

(5)

Here, ${\rho }_i^{″}$ denotes cluster assigned to data $Z^i$ and $\mathit{arg}\underset{j}{min}$ refers to the minimum argument in the $j^{\mathrm{th}}$ cluster. The centroids are recalculated based on the data point already assigned to the cluster. The updated centroid ${\varsigma }^{\prime j}$ is

$${{\varsigma }^{\prime }}^j={\displaystyle \frac{1}{\left|\left.{\rho }_i^{″}\right|\right.}}{\displaystyle \sum _{{\mathrm{Z}}^i\in {\rho }_i^{″}}{\mathrm{Z}}^i}$$

(6)

where $\left|{\rho }_i^{″}\right|$ is the number of data points in cluster ${\varsigma }^j$. By summing this, an updated centroid is produced. The above steps are repeated until the maximum number of iterations ${\stackrel{⃛}{t}}_{\mathrm{m}\mathrm{a}\mathrm{x}}$ is reached. The centroids C_k used in the AdaK-MCC clustering are directly derived from and represent patterns in actual soil moisture (SM) observations. Each centroid corresponds to the average feature set (soil moisture, soil temperature, spectral bands) of a cluster, effectively summarizing a typical SM condition under specific environmental settings. This ensures that the clustering process and any imputations from it are not merely mathematical abstractions but are grounded in observed, physically meaningful soil moisture behaviors. As such, the equations describing centroid updates (Equations (2)–(6)) reflect both algorithmic logic and underlying hydrological realities. Once the final clusters are formed, the missing values are imputed based on the values of similar points within the same cluster and are denoted as $D_{\mathrm{gfill}}$ [40], Equations (5) and (6) describe the assignment of data points to the nearest centroids and the update steps for centroids in K-means.

The pseudocode for AdaK-MCC (Algorithm 1) is expressed below.

Algorithm 1. Pseudo Code of AdaK-MCC

Input: Soil Moisture Data Z Output: Gap-Filled Data $D_{\mathrm{gfill}}$ Begin   Initialize Nc (number of clusters), ${\varsigma }^j$, minimum iteration ${\stackrel{⃛}{t}}_{\mathrm{m}\mathrm{i}\mathrm{n}}$, maximum iteration ${\stackrel{⃛}{t}}_{\mathrm{m}\mathrm{a}\mathrm{x}}$   While ${\stackrel{⃛}{t}}_{\mathrm{m}\mathrm{i}\mathrm{n}}$ < ${\stackrel{⃛}{t}}_{\mathrm{m}\mathrm{a}\mathrm{x}}$     Derive ${\varsigma }^j$ # using Adaptive Momentum Coefficients       ${\varsigma }^j=\vartheta •{\varsigma }^{j-1}+\left(1-\vartheta \right)\ast \sout{\nabla }{\sout{G}}^o$     Assign ${\rho }_i^{″}$ to data $Z^i$       ${\rho }_i^{″}=\arg \underset{j}{\min }\left({‖\left({\mathrm{Z}}^i-{\varsigma }^j\right)‖}^2\right)$     Update centroid ${\varsigma }^j$       ${\varsigma }_j^{\prime }={\displaystyle \frac{1}{\left|\left.{\rho }_i^{″}\right|\right.}}{\displaystyle \sum _{{\mathrm{Z}}^i\in {\rho }_i^{″}}{\mathrm{Z}}^i}$     Repeat until ${\stackrel{⃛}{t}}_{\mathrm{m}\mathrm{a}\mathrm{x}}$   End while   Return $\to$ $D_{\mathrm{gfill}}$ End

Although the source did not clearly state the spatial resolution of the raw dataset, the gap-filling method used in this study does not depend on spatial closeness. Rather, it is executed according to the similarity of spectral, temporal, and environmental features. The AdaK-MCC approach organizes data points through Adaptive K-means clustering with momentum-driven centroid initialization, which depends on factors such as soil moisture percentage, soil temperature, and spectral band responses. Values that are missing are subsequently filled in using values from the same feature-oriented clusters. This imputation driven by clustering guarantees contextual relevance and precision when spatial coordinates or resolution metadata are lacking. Additionally, the dataset comes from a 2017 soil moisture field study close to Karlsruhe, and according to the metadata found in associated repositories (such as the hyperspectral-soil moisture dataset on GitHub), the spatial resolution is around 0.25° × 0.25°, consistent with typical NASA-USDA SMAP and SMOS products.

2.2.2. Noise Reduction

The noises in the gaps filled data $D_{\mathrm{gfill}}$ are reduced by using the Savitzky–Golay Filter (SGF). SGF is primarily utilized to mitigate noise in remote sensing data, which is often affected by disturbances such as sensor faults or atmospheric conditions. These disturbances can create variations that hinder the precise estimation of soil moisture. A significant advantage of SGF is its capability to smooth data while maintaining critical features such as sharp transitions in soil moisture patterns. This preservation is vital for analyzing spectral data in remote sensing, as these features are crucial for a comprehensive understanding of soil moisture. By effectively reducing spectral noise, the SGF ensures that the essential information for accurate moisture predictions remains intact. The Savitzky–Golay filter (SGF) was used to reduce spectral noise. However, caution was taken to preserve the key characteristics of the soil moisture signal. To protect important moisture pattern transitions while reducing noise, the filter’s parameters, such as window size and polynomial order, were meticulously adjusted. The results of comparison studies, which employed both raw and smoothed data during training, showed a minor decrease in signal quality or prediction accuracy, providing validation for this claim. Furthermore, field measurements confirmed that smoothing did not significantly alter variability in the actual world. Without sacrificing the essential characteristics of the variable being studied, these procedures made sure that the preprocessing pipeline improved the quality of the data.

The noise-reduced data $N_{nred}$ is determined as

$${\mathrm{N}}_{nred}={\displaystyle \sum _{u=\frac{1-d}{2}}^{\frac{d-1}{2}}{J}_u\ast D_{gfillv+u}}$$

(7)

where $J_u$ refers to the filter coefficient for each index, u, d indicates the length of the moving window, the index u represents the relative position within the moving window used by the Savitzky–Golay filter to smooth the data, and $D_{\mathrm{gfill}\mathrm{v}+\mathrm{u}}$ represents input at index v + u within the window. Equation (7) is related to the noise reduction process, where the Savitzky–Golay filter smoothens the data to reduce noise caused by atmospheric interference or sensor malfunctions [41].

2.2.3. Atmospheric Correction

The atmospheric noises present in the noise-reduced data $N_{\mathrm{nred}}$ are further corrected to eliminate residual distortions caused by atmospheric interactions. These distortions may result from scattering, absorption by atmospheric gases such as ${\mathrm{C}\mathrm{O}}_2,{\mathrm{O}}_3$, and ${\mathrm{H}}_2\mathrm{O}$ vapor, and aerosol interference, all of which can significantly affect the spectral characteristics used in soil moisture estimation. These atmospheric effects introduce inconsistencies between the measured and true surface reflectance values, thus necessitating a correction step to preserve data integrity. To address this, we employed an atmospheric correction process based on radiative transfer modeling integrated with satellite-derived atmospheric profiles (e.g., from SMAP and SMOS missions). This correction step is applied using a noise correction function $f$, which calibrates the reflectance values by compensating for attenuation and distortion introduced by the atmospheric path. The implementation leverages established atmospheric correction tools such as the ENVI FLAASH module or GDAL-based libraries, which adjust for surface–atmosphere interactions and restore spectral accuracy. The corrected data denoted as $C_{\mathrm{atno}}$, is expressed as

$$C_{\mathrm{atno}}=f\left(N_{\mathrm{nred}}\right)$$

(8)

where $f$ represents the atmospheric noise correction function applied to the noise-reduced data. This correction is crucial for maintaining the spectral integrity of remote sensing inputs and ensuring that the derived moisture levels accurately represent surface conditions. Equation (8) adjusts the soil moisture data by compensating for atmospheric influences such as scattering and absorption, particularly in the near-infrared and microwave spectral bands. This step ensures that the measurements reflect true soil moisture conditions by aligning observed spectral values with their ground-level equivalents, thereby improving prediction accuracy and robustness [42]. The final output, denoted by $\stackrel{‵}{R}$ r represents the fully preprocessed dataset, corrected for both sensor noise and atmospheric effects.

2.3. Season Mapping

Seasonal mapping is the process of classifying time-series data according to the month, day, and year that correspond to each data entry into preset seasons, usually spring, summer, autumn, and winter. The identification of temporal patterns and climatic fluctuations that impact moisture dynamics is made possible by seasonal mapping in the context of soil moisture prediction. By matching observations to the appropriate season, the model can enhance its forecast accuracy across a range of environmental variables, increase climate sensitivity, and, more precisely, identify cyclical patterns. After preprocessing the data, climatic seasons are mapped based on the date, year, and month to distinguish soil moisture patterns influenced by climatic variations. The four seasons are mapped: spring ${(S}_{spr}$), summer ${(S}_{\mathrm{sum}}$), autumn ${(S}_{\mathrm{aut}}$), and winter ${(S}_{\mathrm{win}}$). The mapped seasons ${\mathsf{\Phi }}^{\mathrm{sea}}$ are expressed as follows:

$${\mathsf{\Phi }}^{sea}=\left\{\begin{array}{l}{S}_{spr},ifm\left(\delta ,\gamma \right)\in \left\{3,4,5\right\}\\ S_{sum},ifm\left(\delta ,\gamma \right)\in \left\{6,7,8\right\}\\ S_{aut},ifm\left(\delta ,\gamma \right)\in \left\{9,10,11\right\}\\ S_{win},ifm\left(\delta ,\gamma \right)\in \left\{12,1,2\right\}\end{array}\right.$$

(9)

where $m(\delta ,\gamma )$ represents the month along with the date and year in the dataset $\stackrel{‵}{\mathrm{R}}$, while the numbers $\{1,2,\dots ,12\}$ correspond to the 12 months of the year from January to December. Here, δ denotes the day (date) and γ denotes the year of the observation. The function m(δ, γ) maps each observation’s date and year to its corresponding month, which is used to assign seasonal labels to the data (spring, summer, autumn, and winter). This seasonal mapping facilitates a clearer comprehension of the variations in soil moisture levels across different seasons. Equation (9) maps the remote sensing data to specific seasons based on the date, month, and year. It categorizes the data into four seasons (spring, summer, autumn, and winter) to better analyze how climatic changes affect soil moisture [43].

2.4. Seasonal Clustering

Seasonal mapping and seasonal grouping are two distinct yet complementary preprocessing steps. Seasonal mapping assigns each observation to a specific season (spring, summer, autumn, or winter) based on calendar dates, utilizing a timestamp-based classification system as defined in Equation (9). However, it does not capture intra-seasonal variability. To address this, a seasonal grouping approach is applied as a data-driven clustering method that identifies sub-seasonal patterns or anomalies by grouping observations with similar environmental or spectral characteristics. For example, early-summer data influenced by spring precipitation may differ from late-summer drought conditions, although both are labelled as “summer” in mapping. This dual approach enhances the model’s ability to detect both broad seasonal shifts and finer variations, thereby improving prediction accuracy. Seasonal grouping is performed using the AdaK-MCC algorithm, which forms robust clusters of data with similar soil moisture characteristics within the same season. This helps identify sub-seasonal trends and anomalies, with the resulting clusters denoted as Gseason.

2.5. Varying Pattern Analysis

The variability patterns in the grouped seasons $G_{\mathrm{season}}$ are now analyzed using FWFCSD, as soil moisture data exhibit seasonal patterns affected by climatic variations. Fourier Series Decomposition (FSD) breaks down the data into frequency components, enabling the analysis of both short-term fluctuations and long-term trends. FSD assumes data continuity, but abrupt changes can distort Fourier coefficients. To address this, Frequency Weighted Fourier Coefficients (FWFC) are used to adjust the importance of coefficients based on each frequency, enabling a more flexible analysis that adapts to changing soil moisture patterns over time [41,42]. Firstly, the time series data in $G_{\mathrm{season}}$ are decomposed into frequency components to analyze periodic patterns as follows:

$$\stackrel{⃛}{\sout{\chi }}\left(t\right)={\displaystyle \frac{{\eta }_0}{2}}+{\displaystyle \sum _{p=1}^{\infty }\left({\eta }_p\cos \left(pt\right)+{\stackrel{⃛}{\eta }}_p\sin \left(pt\right)\right)}$$

(10)

where $\stackrel{⃛}{\sout{\chi }}$(t) represents frequency components in $G_{\mathrm{season}}$ at time $t$, ${\eta }_0$ indicates average soil moisture value, ${\eta }_p$ and ${\stackrel{⃛}{\eta }}_p$ are the Fourier coefficients of cosine and sine terms, $p$ represents the harmonic index in the Fourier series, and an integer is used to denote the $p$-th sinusoidal component. While it is dimensionless, it corresponds to multiples of the base frequency derived from the time series in $G_{\mathrm{season}}$. Thus, it helps identify seasonal components by their frequency structure rather than acting as a direct index of $G_{\mathrm{season}}$. In this context, the FWFC is employed in Fourier coefficients to facilitate a flexible analysis of varying conditions, serving as the weighting function associated with frequency $\omega \left(F\right)$ and is expressed as follows:

$${\eta }_0={\displaystyle \frac{1}{\pi }}{\int }_{-\pi }^{\pi }\stackrel{⃛}{\sout{\chi }}\left(t\right)dt$$

(11)

$${\eta }_p=\omega \left(F\right)\ast {\displaystyle \frac{1}{\pi }}{\int }_{-\pi }^{\pi }\stackrel{⃛}{\sout{\chi }}\left(t\right)\cos ptdt$$

(12)

$${\stackrel{⃛}{\eta }}_p=\omega \left(F\right)\ast {\displaystyle \frac{1}{\pi }}{\int }_{-\pi }^{\pi }\stackrel{⃛}{\sout{\chi }}\left(t\right)\sin ptdt$$

(13)

Here, $\pi$ denotes the normalization factor. The analyzed variability pattern data ${\mathsf{\Psi }}_{\mathrm{vapa}}$, representing the varying soil moisture patterns, is obtained. Equations (10)–(13) represent the decomposition of the soil moisture data into frequency components using Fourier Series [44]. The Frequency Weighted Fourier Coefficients (FWFC) are then applied to emphasize certain frequencies, making the analysis more adaptable to changing conditions.

2.6. Multivariate Correlation Analysis

In this phase, the multivariate correlations for the grouped seasons $G_{\mathrm{season}}$ are analyzed using PCA. PCA helps to find patterns and relationships among variables by transforming correlated variables into uncorrelated principal components, thereby simplifying the understanding of the underlying data structure. The PCA steps are explained below.

Step 1: The first step is to standardize the data $G_{\mathrm{season}}$, to ensure all variables are on the same scale. The standardized data A is given by

$$\mathrm{A}={\displaystyle \frac{\left(G_{season}-\mu \right)}{\sigma }}$$

(14)

where µ and σ are the mean and standard deviation of $G_{\mathrm{season}}$, respectively.

Step 2: Next, the covariance matrix is computed to identify how the soil moisture variables vary with each other. The construction of the covariance matrix is formulated as follows:

$$\mathrm{B}={\displaystyle \frac{1}{g}}\ast \left(\mathrm{A}\right){\left(\mathrm{A}\right)}^{\mathsf{{\rm T}}}$$

(15)

where g defines the total number of samples in $G_{\mathrm{season}}$ and $(\mathrm{A}{)}^{\mathrm{T}}$ depicts matrix transpose.

Step 3: Then, eigenvalues ${\mathrm{E}}_{\mathrm{value}}$ and eigenvectors ${\mathrm{E}}_{\mathrm{vector}}$ are computed to find the principal components that account for the greatest variance in the data as indicated by

$$\mathrm{B}\ast {\mathrm{E}}_{vector}={\mathrm{E}}_{value}\ast {\mathrm{E}}_{vector}$$

(16)

The principal components $\chi$ are selected from ${\mathrm{E}}_{\mathrm{vector}}$, corresponding to the highest ${\mathrm{E}}_{\mathrm{value}}$ and are calculated as follows:

$$\chi =\mathrm{A}\ast {\mathrm{E}}_{vector}$$

(17)

From the obtained principal components $\chi$, the multivariate correlation analyzed data is obtained and denoted as ${\mathsf{\Phi }}_{\mathrm{muco}}$. Equations (14)–(17) demonstrate how PCA is applied to analyze multivariate correlations in the data. PCA helps identify underlying patterns in the soil moisture data by transforming correlated variables into uncorrelated principal components [45]. This study employed principal component analysis (PCA) on seasonally grouped data to investigate the interrelationships among variables that influence soil moisture. Variables analyzed included soil moisture percentage, soil temperature, spectral bands, and datetime attributes (month and year), along with derived indices like NDVI, SMI, wavelet coefficients, and texture features (contrast, entropy, homogeneity). Atmospheric effects were addressed during preprocessing through corrections for cloud cover, haze, scattering, absorption, and sensor noise. Additionally, the study suggests integrating land cover and land use (LULC) variables—obtained from sources like MODIS or CORINE—into the PCA framework to better understand spatial diversity and human impact. This integration enhances the model’s realism and prediction accuracy across terrains. PCA successfully reduced dimensionality while preserving key climate-responsive features for accurate modeling.

2.7. Feature Extraction

Here, features are extracted from both the varying pattern and multivariate correlation analyzed data ${\mathsf{\Psi }}_{\mathrm{vapa}}{\mathsf{\Phi }}_{\mathrm{muco}}$ to enhance the model’s performance by simplifying complex data and improving predictions for soil moisture estimation. The extracted features Y are represented as

$$\mathrm{Y}=〈{\mathrm{Y}}_1,{\mathrm{Y}}_2,{\mathrm{Y}}_3,\dots \dots {\mathrm{Y}}_b〉$$

(18)

Here, b indicates the total number of features extracted from both ${\mathsf{\Psi }}_{\mathrm{vapa}}{\mathsf{\Phi }}_{\mathrm{muco}}$. Equation (18) defines the feature extraction process where various characteristics (such as NDVI, SMI, and wavelet coefficients) are extracted from the data to improve prediction accuracy. The features such as mean reflectance, standard deviation, peak positions, Normalized Difference Vegetation Index (NDVI), Soil Moisture Index (SMI), ratio indices, contrast, homogeneity¸ entropy, curvature metrics, inflection points, wavelet coefficients, energy of coefficients, reflectance ratios, derivative spectra are extracted from ${\mathsf{\Psi }}_{\mathrm{vapa}}$. Then, from ${\mathsf{\Phi }}_{\mathrm{muco}}$, features such as pairwise correlation coefficients, correlation heatmaps, clustered correlation groups, representative bands, canonical variates, canonical correlation coefficients, z weights, standardized coefficients, higher-order terms, and unique contribution scores are extracted.

2.8. Feature Selection

The Hierarchical Correlated Spider Wasp Optimizer (HCSWO) was selected for the feature selection process from the extracted features (Y) because of its ability to handle highly correlated features and reduce redundancy effectively. Spectral bands are treated as continuous variables that represent reflectance intensity at specific wavelengths, not as categorical indices. The actual input features extracted from these bands include statistical summaries (e.g., mean, standard deviation), spectral indices (e.g., NDVI, SMI), and transformations (e.g., wavelet coefficients, derivative spectra). These derived features exhibit meaningful variation across space and time and are selected or rejected during optimization via the Hierarchical Correlated Spider Wasp Optimizer (HCSWO). Datetime is also not directly used as a raw feature. Instead, temporal attributes such as month, season, or day of the year are extracted and encoded (e.g., one-hot encoding or cyclical encoding). These derived temporal variables help capture periodic patterns in soil moisture and are considered alongside environmental and spectral features during the feature selection process. Therefore, in the context of the optimization algorithm, the features passed to HCSWO are preprocessed and derived forms of spectral and temporal information, making them valid and informative predictors rather than raw indices or constant categories. Traditional optimization methods like the Spider Wasp Optimizer (SWO) often struggle with correlated features, leading to redundant selections and reduced model performance. To overcome this, the Hierarchical Correlated Spider Wasp Optimizer (HCSWO) uses a Hierarchical Correlation Matrix (HCM) to cluster correlated features and select only the most representative ones, thereby enhancing predictive accuracy and reducing redundancy [19,20,21,22,46]. HCSWO proved more efficient in both feature selection time and overall prediction quality, making it well-suited for soil moisture modeling. In parallel, AdaK-MCC clustering—with Adaptive Momentum Coefficient (AMC) initialization—was employed to accurately group similar seasonal data, thereby avoiding poor local minima. A high silhouette score of 0.975 confirms optimal clustering. Additionally, the AGRL-RBFN model utilizes lasso and adaptive group regularization to minimize overfitting while preserving critical feature interactions. Cross-validation was used to fine-tune the regularization strength, enabling the model to remain adaptable to seasonal and climatic variations. These combined strategies yielded strong generalization and high predictive performance, with an accuracy of 98.09% and an F1-score of 98.95%. The steps of HCSWO are defined below.

Hierarchical Correlation Matrix

The correlation between each extracted feature Y pair is determined to cluster the correlated features. The correlation coefficient ${\perp }^{cor}$ is defined as follows:

$${\perp }^{cor}={\displaystyle \frac{Cov\left({\mathrm{Y}}_b,{\mathrm{Y}}_{b+1}\right)}{\stackrel{⃛}{\sigma }\left({\mathrm{Y}}_b\right)\stackrel{⃛}{\sigma }\left({\mathrm{Y}}_{b+1}\right)}}$$

(19)

Here, $\mathrm{C}\mathrm{o}\mathrm{v}\left({\mathrm{Y}}_b,{\mathrm{Y}}_{b+1}\right)$ denotes covariance between features ${\mathrm{Y}}_b$ and ${\mathrm{Y}}_{b+1}$ of Y, $\stackrel{⃛}{\sigma }\left({\mathrm{Y}}_b\right)$ and $\stackrel{⃛}{\sigma }\left({\mathrm{Y}}_{b+1}\right)$ refers to standard deviation between variables ${\mathrm{Y}}_b$ and ${\mathrm{Y}}_{b+1}$ of Y. Next, to group the correlated features, hierarchical clustering on the correlation matrix ${\perp }^{cor}$ is performed as follows:

$${\mathsf{\Delta }}_{hier}=1-{\perp }_{cor}$$

(20)

where ${\mathsf{\Delta }}_{hier}$ indicates the output of HCM, which shows the clusters of the correlated feature, the representative features are represented as ${\mathrm{Y}}^{rep}$. Equations (19) and (20) define how the HCSWO algorithm selects features from the extracted data. The method uses hierarchical clustering to group correlated features, ensuring only the most relevant features are used in the prediction model [46].

Initialization:

Initially, the population of spider wasps $\tilde{\mathsf{\Lambda }}$ is initialized within the search space concerning ${\mathrm{Y}}^{rep}$ as follows:

$${\tilde{\mathsf{\Lambda }}}_{m\times \mathcal{l}}={\left[\begin{array}{cccc}{\mathrm{Y}}^{rep}\left(1,1\right)& {\mathrm{Y}}^{rep}\left(1,2\right)& \dots & {\mathrm{Y}}^{rep}\left(1,\mathcal{l}\right)\\ {\mathrm{Y}}^{rep}\left(2,1\right)& {\mathrm{Y}}^{rep}\left(2,2\right)& \dots & {\mathrm{Y}}^{rep}\left(2,\mathcal{l}\right)\\ ⋮& ⋮& \dots & ⋮\\ {\mathrm{Y}}^{rep}\left(m,1\right)& {\mathrm{Y}}^{rep}\left(m,2\right)& \dots & {\mathrm{Y}}^{rep}\left(m,\mathcal{l}\right)\end{array}\right]}_{m\times \mathcal{l}}$$

(21)

$${\tilde{\mathsf{\Lambda }}}_x=\left(u{b}_x-l{b}_x\right)\ast \xi +l{b}_x$$

(22)

Here, m refers to the number of spider wasps (female) and $\mathcal{l}$ is the dimension of the search space. Then, the random position ${\stackrel{‵}{\mathsf{\Lambda }}}_x$ of the spider wasp with $\xi$ random variable and $u{b}_x,l{b}_x$ the upper and lower bound of the $x^{\mathrm{th}}$ spider wasp are expressed. In this context, the variable $\xi$ is modeled as a random variable drawn from the uniform distribution, i.e., $\xi ~\mathcal{U}(0,1)$. This ensures equal probability for each potential value within the normalized range, thereby facilitating an unbiased initialization across the entire search space. Such an approach is commonly adopted in population-based metaheuristic algorithms to preserve solution diversity and enhance exploration. The uniform distribution allows each spider wasp to be randomly and independently positioned between the lower $\left(l{b}_x\right)$ and upper $\left(u{b}_x\right)$ bounds, thereby avoiding clustering at initialization and improving the global search capability of the HCSWO algorithm.

Fitness Evaluation

Next, the fitness value ${\mathsf{\Theta }}_{\mathrm{fit}}$ that determines the optimal feature based on the classification accuracy $\overline{\omega }$ is defined as follows:

$${\mathsf{\Theta }}_{fit}=\max \left[\varpi \right]$$

(23)

Using ${\mathsf{\Theta }}_{\mathrm{fit}}$, the best feature of the prey is selected and compared from the population during the mating stage.

Searching Phase

In this exploration phase, the female wasp searches the spiders to feed their young. The updated position ${\stackrel{‵}{\mathsf{\Lambda }}}_{x+1}$ of the female wasp in the search phase is calculated as

$${\tilde{\mathsf{\Lambda }}}_{x+1}=\left\{\begin{array}{l}{\tilde{\mathsf{\Lambda }}}_x+{\ddot{\mu }}_1\ast \left({{\tilde{\mathsf{\Lambda }}}_x}^v-{{\tilde{\mathsf{\Lambda }}}_x}^w\right),if{q}_3<q_4\\ {{\tilde{\mathsf{\Lambda }}}_x}^{\stackrel{⃛}{w}}+{\ddot{\mu }}_2\ast \left(l{b}_x+q_2\ast \left(u{b}_x-l{b}_x\right)\right),Otherwise\end{array}\right.$$

(24)

where ${\stackrel{‵}{\mu }}_1$ and ${\stackrel{‵}{\mu }}_2$ denote the constant motion, $q_1,q_2,q_3,q_4$ are random numbers, and $v,w,\stackrel{⃛}{w}$ indicates the index randomly selected from the population.

Following and Escaping Stage

This is the exploration and exploitation phase; here, after the prey (a spider) is found, the wasp begins chasing it. The updated position is expressed as

$${\tilde{\mathsf{\Lambda }}}_{x+1}=\left\{\begin{array}{ll}{\tilde{\mathsf{\Lambda }}}_x+\stackrel{⌢}{\cancel{C}}\ast \left|2\ast q_5\ast \right|{{\tilde{\mathsf{\Lambda }}}_x}^v-{\tilde{\mathsf{\Lambda }}}_x,& if{q}_3<q_4\\ {\tilde{\mathsf{\Lambda }}}_x\ast \overrightarrow{\mathsf{ƛ}},& Otherwise\end{array}\right.$$

(25)

The transition between the searching, following and escaping phases is given by

$${\tilde{\mathsf{\Lambda }}}_{x+1}=\left\{\begin{array}{l}equation\left(19\right),if{q}_6<\kappa \\ equation\left(20\right),Otherwise\end{array}\right.$$

(26)

Here, $\kappa$ indicates control parameter, $q_5,q_6$ are random numbers, $\widehat{\mathcal{C}}$ denotes distance controlling factor, and $\overrightarrow{\lambda }$ denotes a vector.

Hunting and Nesting Behavior

In this stage, the paralyzed spider is pulled into the pre-prepared nest. The final updated position in the hunting and nesting behavior stage is expressed as follows:

$${\tilde{\mathsf{\Lambda }}}_{x+1}=\left\{\begin{array}{l}{\tilde{\mathsf{\Lambda }}}_x^{\ast }+\cos \left(2\ddot{\pi }\dot{l}\right)\ast \left({\tilde{\mathsf{\Lambda }}}_x^{\ast }-{\tilde{\mathsf{\Lambda }}}_x\right),if{q}_3<q_4\\ {{\tilde{\mathsf{\Lambda }}}_x}^v+q_3\ast \ddot{\gamma }\ast \left({{\tilde{\mathsf{\Lambda }}}_x}^v-{\tilde{\mathsf{\Lambda }}}_x\right)+\left(1-q_3\right)\ast \dot{\stackrel{\rightharpoonup }{R}}\ast \left({{\tilde{\mathsf{\Lambda }}}_x}^w-{\tilde{\mathsf{\Lambda }}}_x^{\stackrel{⃛}{w}}\right),otherwise\end{array}\right.$$

(27)

where ${\tilde{\mathsf{\Lambda }}}_x^{\ast }$ indicates best available solution, $\stackrel{‵}{\pi }$ is a constant, $\stackrel{‵}{\gamma }$ represents a number generated based on Levy flight, $i$ means the localization factor, and $\stackrel{‵}{\stackrel{⃐}{R}}$ is the binary vector. Levy flights in the HCSWO algorithm simulate natural foraging behavior by combining short local steps with occasional long jumps. This enhances global exploration, enabling the optimizer to escape local optima and improve convergence. Integrated into the hunting phase, Levy flights enhance feature selection efficiency for soil moisture prediction by striking a balance between exploration and exploitation. Equations from (21) to (27) describe the feature selection process using the Hierarchical Correlated Spider Wasp Optimizer, a method employed to handle the clustering of correlated features, which is critical for improving the model’s predictive accuracy.

Mating Behavior

The mating stage of HCSWO generates offspring via crossover between selected male and female wasps, replacing fewer fit individuals based on their fitness values ($\mathsf{\Theta }$_fit), and iteratively refines the population to identify the most optimal feature subset (${\mathsf{\Xi }}_{\mathrm{sel}}$). HCSWO’s core innovation lies in integrating hierarchical clustering with the Spider Wasp Optimizer to manage high-dimensional, highly correlated remote-sensing features. Grouping related variables through a hierarchical correlation matrix reduces redundancy, narrows the search space, and accelerates convergence without sacrificing predictive accuracy. Empirically, HCSWO outperforms comparable metaheuristics in both speed and precision: completing feature selection in $2118\mathrm{m}\mathrm{s}$ versus $2211\mathrm{m}\mathrm{s}$ for SWO, $5178\mathrm{m}\mathrm{s}$ for ACO, and $7135\mathrm{m}\mathrm{s}$ for GWO, and achieving a prediction accuracy of $98.09\%$ compared to $96.27\%$ for RBFN, $95.23\%$ for FFNN, $97.24\%$ for RNN, and $95.23\%$ for ANN. The efficacy of metaheuristic feature selection in soil moisture forecasting is well documented: GWO and ACO have been successfully applied to environmental prediction tasks [19,21], and other studies validate the use of such algorithms to enhance machine learning model performance and computational efficiency [22,23]. These precedents support HCSWO as a theoretically sound and practically effective optimizer for high-dimensional remote sensing data.

2.9. Soil Moisture Prediction

In regression tasks, predictions are continuous (e.g., soil moisture levels) and not grouped into fixed classes. However, for analysis or decision-making, post hoc thresholds can be applied to convert these continuous outputs into categories, such as “dry,” “moderate,” and “wet,” based on domain-specific criteria, such as agronomic guidelines. These thresholds (e.g., <10% as “Dry,” 10–20% as “Optimal,” >20% as “Wet”) are not part of the regression model but are used after prediction for interpretability. Thus, classification is a separate step from the continuous regression process. Soil moisture is categorized into three classifications: “dry,” “tolerable or plant-dependent,” and “wet,” based on agronomic and environmental criteria, utilizing continuous outputs from a regression-based soil moisture forecasting model. Soil moisture with a volumetric water content of less than 10% is considered “dry,” indicating inadequate water for plants to absorb. Moisture levels ranging from 10% to 20% are considered “acceptable” or “species dependent,” offering sufficient water for numerous plants, although this varies by type. Levels exceeding 20% are considered “wet,” which may result in saturation of the root zone and the leaching of nutrients. These thresholds, derived from agronomic research and irrigation recommendations, are crucial for interpreting model results in precision farming and drought evaluation, even though they are not incorporated into the model’s training process [47].

In this phase, the soil moisture is predicted from the selected optimal features using AGRL-RBFN with IL.

Radial Basis Function Networks (RBFNs) can be customized to approximate a variety of functions by modifying the shape and parameters of the radial basis functions, allowing them to effectively model intricate soil moisture dynamics and the various factors that influence them. Nevertheless, the localized relationship between the input data and output predictions restricts their capacity to capture overarching patterns in soil moisture data, particularly in datasets characterized by substantial variations. To address this, Adaptive Group Lasso Regularization (AGRL) is implemented in RBFNs, which simultaneously regularizes feature groups and identifies essential interactions for comprehending soil moisture variations, particularly in remote sensing applications involving related features [47,48]. Furthermore, incremental learning (IL) is employed, enabling models to update parameters with new data without the need for complete retraining, which is advantageous for soil moisture prediction, as conditions may fluctuate due to seasonal changes or climate variations. Figure 2 displays the structural diagram of the proposed AGRL-RBFN classifier is shown here, highlighting the different layers involved, including the input layer, adaptive group lasso regularization, hidden layer, and output layer. This diagram also illustrates the use of AGRL for feature group regularization and the RBFN for handling complex soil moisture dynamics.

Input Layer

This layer receives the selected features ${\mathsf{\Xi }}_{\mathrm{sel}}$ as input and passes them to the hidden layers. The inputs are represented as

$${\mathrm{I}}_{in}\to \left\{{{\mathsf{\Xi }}_{sel}}^1,{{\mathsf{\Xi }}_{sel}}^2,\dots {{\mathsf{\Xi }}_{sel}}^{\stackrel{⃛}{z}}\right\}$$

(28)

where ${\mathrm{I}}_{\mathrm{in}}$ indicates the inputs and has $\stackrel{⃛}{z}$ number of selected features ${\mathsf{\Xi }}_{\mathrm{sel}}$ [47].

Adaptive Group Lasso Regularization

To capture global patterns and regulate feature groups, AGRL is used. It regularizes the model based on feature groups, which helps capture interactions within these groups, which is particularly useful for remote sensing data [48]. It is given by

$${\epsilon }_{reg}=\underset{\upsilon }{\min }\left\{\widehat{L}\left(\upsilon \right)+\widehat{\lambda }{\displaystyle \sum _{e=1}^{\stackrel{⌢}{\mathrm{E}}}{\mathsf{\Gamma }}_e{‖{\upsilon }_e‖}_2}\right\}$$

(29)

In Equation (29), λ is the regularization parameter that penalizes large coefficients to prevent overfitting. β_g refers to the coefficient vector associated with the g-th group of features (e.g., related spectral bands or soil attributes). The ℓ₂-norm (‖β_g‖₂) enforces sparsity at the group level, and G denotes the total number of feature groups formed during feature selection. This adaptive group lasso regularization enhances model interpretability and accuracy by selecting or discarding entire feature groups, improving generalization in soil moisture prediction tasks.

Hidden Layer

Here, the regularized data ${\epsilon }_{\mathrm{reg}}$ are transformed using Radial Basis Functions (RBFs) (e.g., Gaussian function) to model non-linear relationships in soil moisture. The RBF ${\mathrm{I}}_{\mathrm{RBF}}$ is also called an activation function and is given by

$${\mathrm{I}}_{RBF}=\exp \left(-{\displaystyle \frac{{‖{\epsilon }_{reg}-{\iota }_{ce{n}_n}‖}^2}{2{\sigma }_n^2}}\right)$$

(30)

where $t_{\mathrm{cen}}$ refers to the $n^{\mathrm{th}}$ center of ${\mathrm{I}}_{RBF}$, and ${\sigma }_n$ indicates the $n^{\mathrm{th}}$ width of the ${\mathrm{I}}_{RBF}$.

Output Layer

In this layer, the output from the hidden layer is given to make the soil moisture prediction. The production $I_{\mathrm{out}}$ of this layer is given by

$${\mathrm{I}}_{out}={\displaystyle \sum _{\sout{n}=1}^{\stackrel{⃛}{\sout{l}}}{\mathsf{\Gamma }}_{\sout{n}}•{\mathrm{I}}_{RBF}+{\tilde{b}}_{bias}}$$

(31)

Here, ${\stackrel{‵}{b}}_{\mathrm{bias}}$ indicates the bias, ${\mathsf{\Gamma }}_n$ refers to the weight of $n^{th}$ neuron, and $\ddot{l}$ shows the number of hidden neurons in RBF. From this output layer, the predicted soil moisture ${\mathrm{I}}_{\mathrm{out}}$ is obtained. Equations (28)–(31) define the architecture of the Adaptive Group Lasso Regularized Radial Basis Function Network (AGRL-RBFN) used to predict soil moisture. AGRL is used to regularize feature groups, and RBFNs are used to model the non-linear relationships between inputs and outputs. The final production represents the predicted soil moisture. The pseudocode for the AGRL-RBFN (Algorithm 2) classifier is described as follows.

Algorithm 2. Pseudo Code of AGRL-RBFN

Input: Selected Features ${\mathsf{\Xi }}_{\mathrm{sel}}$ Output: Predicted Soil Moisture ${\mathrm{I}}_{\mathrm{out}}$ Begin   Initialize ${\sigma }_n$, $t_{\mathrm{hen}}$   For each ${\mathrm{I}}_{\mathrm{in}}$     Compute ${\mathrm{I}}_{in}\to \left\{{{\mathsf{\Xi }}_{sel}}^1,{{\mathsf{\Xi }}_{sel}}^2,\dots {{\mathsf{\Xi }}_{sel}}^{\stackrel{⃛}{z}}\right\}$     Regularize the input ${\mathrm{I}}_{\mathrm{in}}$       ${\epsilon }_{reg}=\underset{\upsilon }{\min }\left\{\widehat{L}\left(\upsilon \right)+\widehat{\lambda }{\displaystyle \sum _{e=1}^{\stackrel{⌢}{\mathrm{E}}}{\mathsf{\Gamma }}_e{‖{\upsilon }_e‖}_2}\right\}$     Evaluate ${\mathrm{I}}_{RBF}$     Calculate ${\mathrm{I}}_{\mathrm{out}}$       ${\mathrm{I}}_{out}={\displaystyle \sum _{\sout{n}=1}^{\stackrel{⃛}{\sout{l}}}{\mathsf{\Gamma }}_{\sout{n}}•{\mathrm{I}}_{RBF}+{\tilde{b}}_{bias}}$   End for   Return $\to$ ${\mathrm{I}}_{\mathrm{out}}$ End

Incremental learning (IL) enhances the model’s adaptability to new data, enabling it to respond to environmental changes and minimize loss during seasonal and climate variations. Accurate soil moisture forecasting requires field validation of model predictions, as remote sensing data from NASA’s SMAP and ESA’s SMOS, while extensive, may lack local specificity. Validation is performed using field probes, such as TDR, capacitance, and neutron sensors, to capture localized conditions. The AGRL-RBFN model utilizes such validation to align predictions with real-world measurements, employing metrics such as MAE and RMSE. Discrepancies are addressed through optimization and retraining, thereby improving generalization. This iterative process boosts model resilience for accurate predictions in varied conditions. While supervised learning models largely rely on satellite data processed via techniques like the Palmer model and Kalman filters, in situ measurements remain essential for calibration, particularly in areas with limited satellite coverage. Reanalysis products and model-enhanced datasets also support soil moisture prediction, despite known uncertainties.

2.10. Integration of Remote Sensing and Deep Learning for Soil Moisture Prediction

This study integrates hyperspectral and soil moisture data from NASA’s SMAP and ESA’s SMOS missions [5,6] with advanced deep learning to predict soil moisture across large areas without in situ sampling. A robust preprocessing pipeline, including adaptive gap filling, noise reduction, and atmospheric correction, ensures high data quality. The core predictor, an Adaptive Group Radial Lasso–Regularized Basis Function Network (AGRL-RBFN) with incremental learning (IL), continuously adapts to new seasonal and climatic data. Feature selection is driven by the Hierarchical Correlated Spider Wasp Optimizer (HCSWO), which reduces redundancy among highly correlated remote-sensing bands. Complementary deep models, such as CNNs and LSTMs, capture spatial and temporal patterns [13,14,15,16], while Lasso regularization mitigates overfitting in high-dimensional feature spaces. The resulting model achieves 98.09% accuracy—surpassing RBFN, FFNN, and RNN baselines in precision, recall, and F1-score—effectively handling data gaps, seasonal shifts, and noise.

2.11. Model Calibration and Validation

The robustness and generalizability of the AGRL-RBFN model were ensured through a structured calibration and validation protocol. Model calibration involved tuning hyperparameters (RBF neurons, λ, and γ) via internal grid search with 5-fold cross-validation to minimize mean squared error (MSE) and prevent overfitting, aided by Adaptive Group Lasso Regularization for penalizing irrelevant features. For validation, the dataset was split into 80% for training and 20% for testing, using various metrics (accuracy, MAE, RMSE, precision, recall, F1-score, false positive/negative rates) to assess predictive performance on unseen data. External field validation included in situ measurements from the 2017 Karlsruhe campaign, aligned with satellite-derived soil moisture data, showing strong agreement. The model’s long-term adaptability was evaluated through incremental learning across seasonal and climatic variations, enabling parameter adjustments without full retraining for accuracy amid evolving conditions. The performance evaluation of the proposed soil moisture prediction system is discussed below.

3. Results

In this section, the proposed system’s performance is evaluated and compared with that of existing models using various metrics. The evaluation results were derived from implementing and testing the framework on the PYTHON platform.

3.1. Performance Analysis

This section compares the performance of AdaK-MCC, FWFCSD, HCSWO, and AGRL-RBFN with existing models to showcase the improvements achieved by the proposed model.

3.1.1. Performance Analysis of Gap Filling

The performance evaluation of the proposed method AdaK-MCC for gap filling is discussed here. Figure 3 shows the comparison of mean absolute error (MAE) between the proposed AdaK-MCC gap filling technique and other existing techniques like KMC, FCM, KNN, and K-Medoid. The proposed method demonstrates superior performance with a lower MAE value.

The gaps in the dataset are filled using the AdaK-MCC technique, resulting in a 0.047 MAE. The proposed model obtained lower errors than the existing approaches. This is because the centroids are initialized using AMCs, which effectively impute the values. The existing techniques, such as KMC, FCM, KNN, and K-Medoid, attained higher MAEs of 0.099, 0.157, 0.386, and 0.548, respectively. Hence, the gaps were filled more efficiently than in the existing models.

Table 1. Tabular analysis of silhouette score.

Techniques	Silhouette Score
Proposed AdaK-MCC	0.975
KMC	0.953
FCM	0.929
KNN	0.858
K-Medoid	0.813

presents the silhouette scores of the proposed AdaK-MCC and other existing techniques. The proposed technique achieved a silhouette score of 0.975. In contrast, the existing KMC, FCM, KNN, and K-Medoid methods achieved silhouette scores of 0.953, 0.929, 0.858, and 0.813, which are lower than those of the proposed AdaK-MCC. This demonstrates the superiority of the proposed approach in gap filling compared to existing models.

3.1.2. Performance Analysis of Clustering

In this section, the proposed AdaK-MCC is evaluated against existing clustering methods, including KMC, FCM, KNN, and K-Medoid, for a comprehensive clustering analysis.

Figure 4 and

Table 2. Evaluation of clustering methodologies.

Techniques	Dunn Index	Mean	Standard Deviation	95% Confidence Interval
Proposed AdaK-MCC	4.897	2.942	1.214	[1.876, 4.008]
KMC	2.984	2.984	0.015	[2.969, 2.999]
FCM	2.541	2.541	0.016	[2.524, 2.558]
KNN	1.924	1.924	0.014	[1.912, 1.936]
K-Medoid	1.368	1.368	0.01	[1.358, 1.378]

illustrate the performance analysis of AdaK-MCC compared to existing techniques in terms of clustering time and Dunn index. The AdaK-MCC achieved a lower clustering time of 2118 ms and a higher Dunn index of 4.897. The existing techniques, KMC, FCM, KNN, and K-Medoid, attained longer clustering times and lower Dunn indices. The improved performance in the proposed method is attributed to the use of the AMC technique for centroid initialization. Thus, the robust performance of AdaK-MCC for data clustering is validated. The suggested AdaK-MCC approach exhibits a greater standard deviation (1.214) in the Dunn Index relative to other clustering methods, with this variation resulting from its adaptive centroid initialization utilizing adaptive momentum coefficients (AMC). This adaptive system is specifically designed to enhance exploration and prevent early convergence to inferior solutions, thereby facilitating the identification of superior clusters under diverse data conditions. Regardless of this variability, AdaK-MCC consistently attains the highest average Dunn Index value (4.897), reflecting better inter-cluster separation and intra-cluster compactness. Notably, the 95% confidence interval ([1.876, 4.008]) suggests that, despite variations, the model’s performance remains statistically robust and does not substantially overlap with the estimators that perform poorly. Conversely, conventional clustering approaches such as KMC and FCM, while exhibiting reduced standard deviations, produce lower mean Dunn Index scores and tighter confidence intervals, indicating they are more consistent but less capable of identifying intricate, seasonally shifting trends in soil moisture data. Consequently, the marginally heightened variability in AdaK-MCC is an intentional compromise for better clustering quality, influenced by its superior search dynamics. This demonstrates that AdaK-MCC surpasses other estimators not only in average performance but also in its capacity to generalize across various data distributions—rendering it a more appropriate option for resilient, adaptive soil moisture assessment.

In Table 2, the Dunn Index values for the approaches are as follows: proposed AdaK-MCC, 4.897; KMC, 2.984; FCM, 2.541; KNN, 1.924; and K-Medoid, 1.368. The mean Dunn Index is taken to be 2.942, which is the average of these methods. The standard deviation (SD) is calculated to be 1.214, reflecting the spread or dispersion of Dunn Index values. Additionally, the 95% confidence interval (CI) for the Dunn Index is reported as [1.876, 4.008]. Thus, we can be 95% sure that the actual value of the Dunn Index for these procedures falls within this interval. The model’s performance was benchmarked using hyperspectral and soil moisture data obtained from a 2017 field campaign in Karlsruhe, Germany. The data had a spatial resolution of 0.25° × 0.25° and a temporal resolution ranging from daily to weekly intervals.

Table 3. Comparative performance of the proposed AGRL-RBFN model and existing methods for soil moisture prediction.

Metric	Proposed Model (AGRL-RBFN)	Existing Methods
Gap Filling MAE	0.047 (CI: [0.043, 0.051]), SD: 0.002	KMC: 0.099 (CI: [0.080, 0.118]), FCM: 0.157 (CI: [0.145, 0.169]), KNN: 0.386 (CI: [0.370, 0.402]), K-Medoid: 0.548 (CI: [0.532, 0.564])
Silhouette Score (Gap Filling)	0.975 (SD: 0.019)	KMC: 0.953 (SD: 0.019), FCM: 0.929 (SD: 0.022), KNN: 0.858 (SD: 0.034), K-Medoid: 0.813 (SD: 0.042)
Clustering Time	2118 ms	KMC: N/A, FCM: N/A, KNN: N/A, K-Medoid: N/A
Dunn Index	4.897 (SD: 0.125)	KMC: 2.984 (SD: 0.015), FCM: 2.541 (SD: 0.016), KNN: 1.924 (SD: 0.014), K-Medoid: 1.368 (SD: 0.01)
Varying Pattern Cross-Correlation Coefficient	0.9715 (CI: [0.965, 0.978])	FSD: 0.8930 (CI: [0.880, 0.906]), DFT: 0.8930 (CI: [0.880, 0.906]), FFT: 0.8930 (CI: [0.880, 0.906]), STFT: 0.8930 (CI: [0.880, 0.906])
Varying Pattern Reconstruction Error	0.0589 (SD: 0.004)	FSD: 0.1483 (SD: 0.003), DFT: 0.1483 (SD: 0.003), FFT: 0.1483 (SD: 0.003), STFT: 0.1483 (SD: 0.003)
Feature Selection Time	2118 ms (SD: 10 ms)	SWO: 2211 ms (SD: 12 ms), ACO: 5178 ms (SD: 25 ms), GWO: 7135 ms (SD: 35 ms), CSO: 9052 ms (SD: 40 ms)
Soil Moisture Prediction Accuracy	98.09% (CI: [97.90%, 98.28%])	RBFN: 96.27% (CI: [95.95%, 96.59%]), FFNN: 95.23% (CI: [94.80%, 95.66%]), RNN: 97.24% (CI: [96.80%, 97.68%]), ANN: 95.23% (CI: [94.80%, 95.66%])
Precision	98.17% (CI: [97.97%, 98.37%])	RBFN: 97.55% (CI: [97.30%, 97.80%]), FFNN: 96.12% (CI: [95.90%, 96.34%]), RNN: 97.05% (CI: [96.81%, 97.29%]), ANN: 96.12% (CI: [95.90%, 96.34%])
Recall	97.24% (CI: [96.85%, 97.63%])	RBFN: 95.34% (CI: [95.10%, 95.58%]), FFNN: 93.45% (CI: [93.20%, 93.70%]), RNN: 94.26% (CI: [94.00%, 94.52%]), ANN: 93.45% (CI: [93.20%, 93.70%])
F1-Score	98.95% (CI: [98.75%, 99.15%])	RBFN: 96.27% (CI: [95.95%, 96.59%]), FFNN: 95.23% (CI: [94.80%, 95.66%]), RNN: 97.24% (CI: [96.80%, 97.68%]), ANN: 95.23% (CI: [94.80%, 95.66%])
False Positive Rate (FPR)	0.0248 (SD: 0.002)	RBFN: 0.481 (SD: 0.002), FFNN: 0.596 (SD: 0.003), RNN: 0.723 (SD: 0.004), ANN: 0.876 (SD: 0.005)
False Negative Rate (FNR)	0.076 (SD: 0.003)	RBFN: 0.108 (SD: 0.003), FFNN: 0.257 (SD: 0.004), RNN: 0.429 (SD: 0.005), ANN: 0.571 (SD: 0.006)

summarizes the comparative performance results of the proposed AGRL-RBFN model in comparison to baseline methods.

Table 3 compares the AGRL-RBFN model with RBFN, FFNN, RNN, and ANN using key metrics, including MAE, Silhouette Score, Dunn Index, cross-correlation, prediction accuracy, precision, recall, F1-score, and false positive and negative rates. The AGRL-RBFN model outperforms all others, achieving 98.09% accuracy, 98.17% precision, and a 98.95% F1 score. It also achieves a low MAE of 0.047, with a confidence interval of [0.043, 0.051], indicating reliable gap filling. The model exhibits excellent clustering, as indicated by a Dunn Index of 4.897 (SD = 0.125) and a high Silhouette Score of 0.975, along with a low false positive rate of 0.0248. The inclusion of confidence intervals and standard deviations confirms the model’s robustness and consistency across repeated trials, validating its strong predictive and gap-filling performance. A thorough validation was performed using ground-truth data collected during the Karlsruhe 2017 field campaign to create a solid observational foundation for our model. In situ measurements obtained via time domain reflectometry (TDR) and various field probes assessed the accuracy of soil moisture predictions sourced from satellites. A high level of agreement was observed between the predicted and observed values, validating the model’s ability to accurately reflect actual moisture behavior. Additionally, an extensive comparative study involving advanced models—such as RBFN, FFNN, RNN, and ANN—was conducted using the same datasets and uniform experimental methods. Throughout various experiments, the proposed AGRL-RBFN model consistently outperformed these benchmarks, achieving superior prediction accuracy (98.09%) and F1-score (98.95%), with statistically significant enhancements confirmed through confidence interval and standard deviation assessments. These findings confirm that the model is both algorithmically creative and empirically strong.

3.1.3. Performance Analysis of Varying Pattern Analysis

Here, the proposed FWFCSD is compared with existing techniques, such as FSD, Discrete Fourier Transform (DFT), Fast Fourier Transform (FFT), and Short-Time Fourier Transform (STFT).

Figure 5 depicts the cross-correlation coefficient and reconstruction error of the proposed FWFCSD and existing techniques. The varying patterns in the grouped seasons are analyzed by integrating FWFC for flexible analysis of the varying soil moisture patterns. By integrating FWFC, the proposed FWFCSD achieved a high cross-correlation coefficient of 0.9715 and a low reconstruction error of 0.0589. At the same time, the existing FSD, DWT, FFT, and STFT achieved an average cross-correlation coefficient of 0.893025 and an average reconstruction error of 0.148275, which are, respectively, lower and higher than those of the proposed FWFCSD. Therefore, the proposed FWFCSD demonstrates its efficacy in analyzing diverse patterns.

Table 4. Execution time analysis of FWFCSD with existing techniques.

Techniques	Execution Time (ms)
Proposed FWFCSD	18,564
FSD	24,796
DFT	28,357
FFT	32,497
STFT	39,641

presents a performance analysis of the proposed FWFCSD method compared to existing techniques, including FSD, DWT, FFT, and STFT, with a focus on execution time. The FWFCSD method achieved an execution time of just 18,564 ms, significantly outperforming the existing techniques, which took 24,796 ms, 28,357 ms, 32,497 ms, and 39,641 ms, respectively. Thus, the proposed method provides a more effective analysis of the varying patterns compared to the existing methods.

3.1.4. Performance Analysis of Feature Selection

In this section, the performance evaluation of the proposed HCSWO is discussed in comparison to existing techniques.

The performance of the proposed method is presented in

Table 5. Feature selection time analysis.

Techniques	Feature Selection Time (ms)
Proposed HCSWO	2118
SWO	2211
ACO	5178
GWO	7135
CSO	9052

, specifically regarding the time required for feature selection. To uniquely select the features, HCM is incorporated to cluster the correlated features to predict moisture. It achieved a lesser feature selection time of 2118 ms. On the contrary, the existing methods such as SWO, Ant Colony Optimizer (ACO), Grey Wolf Optimizer (GWO), and Cuckoo Search Optimizer (CSO) consumed more feature selection time of 2211 ms, 5178 ms, 7135 ms, and 9052 ms. Hence, the proposed feature selection method demonstrates enhanced performance when compared to existing techniques.

3.1.5. Performance Analysis of Soil Moisture Prediction

In this section, the performance of the proposed AGRL-RBFN is assessed in comparison to other relevant existing techniques.

Figure 6 and Figure 7 illustrate the performance of the proposed AGRL-RBFN in comparison to established techniques, including RBFN, Feed Forward Neural Network (FFNN), Recurrent Neural Network (RNN), and Artificial Neural Network (ANN). The AGRL-RBFN achieved an accuracy of 98.09%, precision of 98.17%, recall of 97.24%, F1-score of 98.95%, and specificity of 97.21%, all of which surpass the results of the existing methods. Despite the baseline RBFN model being fundamentally straightforward and quick, its challenges with correlated features, noise, and seasonal variation restrict its effectiveness. The suggested AGRL-RBFN model, featuring Adaptive Group Lasso Regularization and incremental learning, presents moderate complexity while attaining significant enhancements in accuracy (from 96.27% to 98.09%) and F1-score (from 96.27% to 98.95%). These enhancements validate the increased complexity, especially in critical applications such as precision agriculture and drought prediction, where even slight advancements can produce substantial real-world effects. This superior performance can be attributed to the integration of AGRL within the RBFN, which effectively regularizes the feature groups for soil moisture prediction in conjunction with the application of IL. In contrast, the traditional RBFN recorded accuracy, precision, and recall values of 96.27%, 97.55%, and 95.34%, respectively, which are inferior to those of the proposed AGRL-RBFN. Therefore, the proposed model demonstrates greater efficiency in predicting soil moisture compared to the existing approaches.

Table 6. Tabular analysis of soil moisture prediction in terms of FPR and FNR.

Techniques	FPR	FNR
Proposed AGRL-RBFN	0.0248	0.076
RBFN	0.481	0.108
FFNN	0.596	0.257
RNN	0.723	0.429
ANN	0.876	0.571

presents a comparison of the proposed AGRL-RBFN’s performance against existing methods, focusing on the false positive rate (FPR) and false negative rate (FNR). The proposed AGRL-RBFN achieved an FPR of 0.0248 and an FNR of 0.076, which are lower than those of existing techniques. On the contrary, the existing methods attained higher FPR and TNR. As a result, the proposed model outperformed the existing methods in predicting soil moisture.

3.1.6. Time-Series Validation of AGRL-RBFN Predictions Against Observed Measurements

To validate the reliability of the proposed AGRL-RBFN model, a comparative time-series analysis was conducted between predicted soil moisture values and in situ observations obtained using time domain reflectometry (TDR) sensors during the Karlsruhe 2017 field campaign. This comparison aims to evaluate the model’s temporal accuracy across real-world conditions.

Figure 8 illustrates the predicted and actual soil moisture levels over a representative time window, demonstrating strong alignment across both seasonal trends and short-term fluctuations. The model effectively captures key moisture transitions corresponding to rainfall events and dry periods. While minor deviations occur during seasonal changeovers (e.g., late spring and early autumn), these are attributed to rapid vegetation changes and transient climatic anomalies not fully captured in remote sensing data.

Quantitatively, the AGRL-RBFN model achieves a mean absolute error (MAE) of 0.047 and a root mean square error (RMSE) of 0.065, indicating high precision. Additionally, the Pearson correlation coefficient between the predicted and observed series exceeds 0.96, confirming strong predictive alignment with field observations. This validation confirms the model’s ability to accurately reflect actual soil moisture dynamics. It provides empirical support for its application in operational soil monitoring and irrigation advisory systems, particularly in regions with limited ground-based sensing infrastructure.

An error assessment of the AGRL-RBFN model was conducted using the Karlsruhe 2017 dataset during extreme hydrological situations, specifically focusing on drought and intense rainfall. In times of drought, the model indicated a mean absolute error (MAE) of 0.052 and a root mean square error (RMSE) of 0.072, which is marginally greater than the global MAE of 0.047, underscoring its sensitivity to reduced soil moisture. During significant rainfall events, the model exhibited an MAE of 0.056, which was influenced by rapid increases in surface moisture and saturation, further complicated by remote sensing issues such as cloud cover. A Pearson correlation coefficient exceeding 0.93 in both cases suggests a robust predictive ability. Future improvements could involve incorporating extra climatic factors or creating recalibration methods for improved anomaly identification.

3.2. Comparison with Existing Approaches

This section provides a comparative evaluation of the proposed AGRL-RBFN against current methodologies for predicting soil moisture. The performance categories used in

Table 7. Comparative analysis of proposed with existing works.

Metric	AGRL-RBFN (Proposed)	LSTM [49]	DCNN [50]	ANN [22]	RBFN [9]		RF [10]	KNN [8]	SVM [7]	Attention-Aware LSTM [51]	DL with TL [52]	ML with Pre-classification [53]
Accuracy	98.09%	74.36%	98.05%	95.23%	96.27%	~96%	~95%	~96%	97.06%	95.23%	98%
Precision	98.17%	-	-	96.12%	97.55%	~96%	~95%	96.50%	-	-	-
Recall	97.24%	-	-	93.45%	95.34%	~94%	~93%	94%	-	-	-
F1-Score	98.95%	-	-	95.23%	96.27%	~95%	~94%	95%	-	-	-
Gap Filling MAE	0.047	-	-	-	0.099	~0.1	~0.4	~0.15	-	-	-
Clustering Dunn Index	4.897	-	-	-	2.984	~2.9	~2.5	~3.0	-	-	-
Clustering Time	2118 ms	-	-	-	~2000 ms	~2000 ms	~2500 ms	~3000 ms	-	-	-
Computational Complexity	Low	High	High	Medium	Medium		High	Medium	Medium to High	High	Medium–High	-
Seasonal Adaptability	Excellent	Limited	Limited	Moderate	Low	Moderate	Low	Low	Limited	Less effective	Limited
Climatic Adaptability	Excellent	Struggles	Limited	Moderate	Low	Low	Low	Low	Limited	Limited	Limited
Gap Filling Technique	AdaK-MCC	-	-	-	KMC	KNN	K-Medoid	SVM	-	-	-
Feature Selection Method	HCSWO	-	-	-	RF	Random Forest	KNN	SVM	-	Pre-classification	-
Normalization Strategy	Per- feature normali-zation (via PCA)	Min- Max or Z-score normalization	Min- Max or Z-score normalization	Min- Max or Z-score normalization	Per-feature normali- zation	Min-Max or Z-score normali- zation	Min-Max normal- ization	Z-score or Min-Max normalization	Min-Max or Z-score normalization	Not specified (likely Z-score)	Pre-classification may use normalized features
Application Suitability	Precision Agriculture, Climate Resilience	Irrigation, Water Conservation	Water Management	Climate Adaptive Agriculture	Limited tasks		Precision Agriculture	Simple Classification	Soil moisture & temperature prediction	Soil moisture in diverse regions	Land-specific soil moisture modeling	-
Limitations	Crop-specific moisture not addressed	High computational demands	High cost, limited spatiotemporal prediction	Complex, non-seasonal data	Limited adaptability	Large dataset challenges	Slow execution	Sensitive to small datasets	No spatiotemporal prediction	Less effective in winter	Depends on classification accuracy

namely “Low,” “Moderate,” “Medium,” and “High” are determined based on widely accepted thresholds in environmental modeling and machine learning literature. For classification metrics such as accuracy and F1-score, values equal to or greater than 95% are categorized as “High,” those ranging from 90% to 94.9% are considered “Moderate,” and values below 90% fall under the “Low” category. For clustering performance, metrics such as the Dunn Index and Silhouette Score follow a similar classification logic; a Silhouette Score above 0.90 or a Dunn Index exceeding 3.5 is considered indicative of “High” performance. These benchmark thresholds provide a transparent and consistent basis for interpreting the model’s effectiveness across different evaluation dimensions, particularly in hydrological and agricultural applications.

Table 7 provides a qualitative comparison of soil moisture prediction models using categories like “Low,” “Medium,” “High,” and “Moderate,” grounded in metrics such as training and feature-selection times (e.g., AGRL-RBFN’s 2118 ms vs. CSO’s 9052 ms), architectural efficiency, and clustering performance (Dunn Index > 4 = High). AGRL-RBFN leads with the highest accuracy (98.09%), precision (98.17%), recall (97.24%), and F1-score (98.95%), paired with superior gap filling (MAE 0.047) and clustering (Dunn 4.897) performance, all achieved with relatively low computational complexity and exceptional seasonal and climatic adaptability. By contrast, models like LSTM and DCNN require heavier computation and lack flexible temporal adaptation, while simpler methods (RF, KNN, SVM) underperform under variable conditions [13,14,15,16]. AGRL-RBFN’s advanced gap-filling and feature-selection strategies make it the most versatile and effective choice for diverse climatic and agricultural needs. However, it does not specifically address variations in crop-type moisture.

3.3. Convergence and Computational Complexity Analysis of HCSWO

The convergence of the Hierarchical Correlation Matrix-based Spider Wasp Optimizer (HCSWO) is supported both theoretically and empirically. By integrating hierarchical correlation clustering, the algorithm efficiently narrows the search to representative, non-redundant features, which accelerates convergence toward optimal solutions. Although global convergence is not guaranteed due to its heuristic nature, empirical results—such as achieving feature selection in just 2118 ms—demonstrate its fast and effective performance. Regarding computational complexity, constructing the hierarchical correlation matrix and clustering contributes significantly, with complexities of O(n²·m) for correlation computation and O(n³) for clustering. Optimization iterations introduce additional cost, roughly O(I × P × fitness evaluation), depending on the number of iterations and population size. However, hierarchical clustering reduces dimensionality and computational load, making HCSWO a practical and efficient choice for high-dimensional tasks, such as soil moisture prediction, while balancing accuracy and processing cost-effectively.

4. Discussion

In this section, the performance of the proposed soil moisture prediction model is presented and compared with several existing techniques.

4.1. Model Performance

The AGRL-RBFN model demonstrates superior performance in soil moisture prediction by integrating advanced feature selection, adaptive regularization, and robust learning techniques. Utilizing HCSWO with a Hierarchical Correlation Matrix, the model efficiently selects non-redundant features. AGRL enhances feature group regularization and, when combined with RBFNs, captures complex non-linear patterns across climates. Incremental learning (IL) ensures adaptability without full retraining. Preprocessing with AdaK-MCC (gap filling) and SGF (noise reduction), along with FWFCSD for seasonal trend analysis, ensures high data quality. The model achieves 98.09% accuracy, 98.95% F1-score, 0.047 MAE, and a Dunn Index of 4.897. FWFCSD yields a cross-correlation of 0.9715, and HCSWO performs feature selection in 2118 ms, outperforming models like RBFN, FFNN, RNN, and ANN. The AGRL-RBFN model exhibits significantly lower false positive (FPR) and false negative rates (FNR), indicating robust soil moisture prediction performance. Compared to LSTM, DCNN, and other regression models, AGRL-RBFN offers advantages like faster processing, higher accuracy, and better generalizability. LSTM models, while effective, struggle with rapidly changing weather and require substantial computational power. The AGRL-RBFN bypasses these limitations by integrating adaptive regularization and incremental learning efficiently. In contrast, machine learning regression models reliant on accurate land type classification lack the flexibility of AGRL-RBFN, which achieves superior predictive accuracy without such dependencies. The proposed model also stands out in its ability to accommodate climatic changes that affect soil moisture, thereby addressing the shortcomings of existing methods that overlook climate variability. Its multi-technique integration—comprising AdaK-MCC, FWFCSD, HCSWO, and AGRL-RBFN—enables the efficient handling of challenges such as noise reduction, gap filling, and feature selection, thereby enhancing prediction robustness.

A paired t-test assesses the accuracy differences between AGRL-RBFN and other models (RBFN, FFNN, RNN, and ANN), positing a null hypothesis (H₀) of no significant accuracy difference versus an alternative hypothesis (H₁) asserting a considerable difference. Accuracy comparisons yield results: AGRL-RBFN vs. RBFN 1.82%, AGRL-RBFN vs. FFNN 1.59%, AGRL-RBFN vs. RNN 3.33%, and AGRL-RBFN vs. ANN 5.59%. However, the lack of multiple data points hinders the calculation of proper variance, rendering a valid paired t-test impossible [43]. Thus, further data are necessary for robust statistical analysis and credible conclusions regarding model performance differences. An ablation study evaluated the significance of each component in the prediction model. The systematic removal of elements, particularly preprocessing tasks such as gap filling and noise reduction, drastically degraded model performance. The absence of these preprocessing steps resulted in poor handling of missing data and atmospheric distortions.

The AdaK-MCC technique enhances K-Means clustering with Adaptive Momentum Coefficients (AMC) for centroid initialization; however, it does not utilize real-world soil moisture (SM) data during this process. This oversight may compromise the relevance of imputed values in ecologically diverse areas. Incorporating field-based SM observations from sources such as TDR sensors could enhance the realism of clustering. Although current methods like seasonal mapping and spectral decomposition address spatial–temporal variability, they may overlook localized moisture dynamics [41,42]. Future research should integrate geo-referenced soil and land-use data, model seasonal climatology and crop cycles, and validate outputs against high-resolution SM datasets to bolster the model’s reliability and applicability. Omitting varying pattern analysis (FWFCSD and FWFC) hindered the model’s ability to track soil moisture variations, reducing its flexibility for short-term and long-term trends. Removing HCSWO resulted in handling irrelevant features, leading to overfitting and slower training times. Eliminating AGRL regularization and IL-reduced feature interaction and adaptability to new data under changing conditions, negatively impacting performance. The ablation study highlights that each component—preprocessing, seasonal mapping, varying pattern analysis, feature selection, and AGRL with IL—is crucial for maintaining high accuracy and robustness.

4.2. Feature Importance and Model Behavior

The model determines feature significance through its Adaptive Group Lasso Regularization (AGRL) and an advanced method for selecting features. AGRL imposes penalties on sets of related features, grouping them according to their physical and spectral similarities. Important predictors such as spectral bands, vegetation indices (e.g., NDVI), wavelet coefficients, and soil temperature are categorized to retain the most significant ones for soil moisture variability while reducing less relevant features. This technique successfully identifies complex relationships typically present in remote sensing data. The Hierarchical Correlated Spider Wasp Optimizer (HCSWO) enhances feature selection by grouping features based on their correlation patterns, facilitating the identification and retention of important features while eliminating unnecessary ones. This minimizes overfitting and improves computational efficiency, ensuring the model focuses on distinct, non-repetitive information. Key features for moisture detection, particularly spectral bands that respond to water absorption, are preserved due to their strong correlation with soil moisture readings. Vegetation indices reflect plant vitality and water stress, acting as indirect signs of soil moisture, particularly in green regions. Multi-scale wavelet attributes capture temporal shifts, such as drought periods or rainy seasons, thereby enhancing the model’s sensitivity to climate variations.

4.3. Error Analysis

The model effectively ranks features vital for predicting soil moisture; however, it faces challenges, particularly during extreme weather events such as droughts or heavy rainfall. Error analysis identifies misclassifications that arise from inadequate data in these situations, which hinder the model’s performance under extreme conditions. To alleviate these issues, strategies such as error analysis to detect misclassification patterns, enhancing the dataset with extreme case examples, and introducing features capturing environmental changes are recommended. Resampling techniques can balance the dataset while adjusting class thresholds can improve performance across diverse soil moisture levels. Regularization methods and hyperparameter tuning, particularly using AGRL-RBFN, can mitigate overfitting and improve generalization. Combining AGRL-RBFN with other classifiers can leverage strengths and reduce misclassifications. Incremental learning enables the model to adapt continuously to new data, which is particularly crucial during extreme weather conditions. Post-processing methods such as smoothing can also enhance the model’s predictive reliability against outliers [48]. The AGRL-RBFN model performs well in typical conditions but struggles with accuracy during extreme events due to several factors. The limited dataset from the 2017 Karlsruhe campaign results in a lack of extreme event examples, causing data imbalance and impairing generalization. Spectral saturation and ambiguity during extreme wet conditions lead to distorted surface reflectance, complicating accurate predictions [52]. Meanwhile, reliance on seasonal mapping may hinder the detection of anomalies. Soil moisture often lags behind weather events, causing temporal mismatches in predictions. To address these challenges, employing data augmentation with simulated extreme scenarios, integrating external indicators, hybridizing with physics-based hydrological models, and incorporating advanced temporal memory mechanisms are suggested. Additionally, refining incremental learning for real-time adaptation and integrating land use data can improve predictions across varied terrains. Future efforts should focus on validating across diverse extreme regions and integrating rainfall-runoff dynamics to enhance robustness in agricultural and hydrological applications.

4.4. Limitations and Generalizability Across Climatic Zones

While the proposed AGRL-RBFN model exhibits strong predictive accuracy on the Karlsruhe 2017 dataset, its current validation is confined to a temperate climate region. Given that soil moisture dynamics are highly sensitive to variations in climate, land cover, soil type, and crop cycles, the model’s applicability to broader environmental conditions remains untested. To address this, future work will focus on extending validation to a range of climatic zones, including arid regions (e.g., the southwestern USA, northern Africa), tropical zones (e.g., Southeast Asia, the Amazon Basin), and semiarid regions (e.g., the Sahel, central India). This will involve the use of publicly accessible remote sensing datasets such as SMAP, MODIS, and Sentinel-1, as well as region-specific ground truth measurements from TDR and field probes. By evaluating model performance across diverse hydrological and agronomic contexts, this expanded analysis aims to establish the generalizability and operational scalability of the AGRL-RBFN framework for global soil moisture prediction under varying climatic conditions.

5. Conclusions

Soil moisture is a crucial factor in regulating interactions between terrestrial ecosystems and atmospheric processes, making it essential for fields such as agriculture, hydrology, and climate science. In agriculture, maintaining optimal soil moisture is critical for plant growth, while both excessive wetness and dryness can negatively impact crop yield and quality. However, existing research lacks depth in addressing the complexities of soil moisture prediction under changing climate conditions. This study introduces a robust soil moisture prediction system that integrates AdaK-MCC for gap filling, FWFCSD for pattern analysis, HCSWO for feature selection, and AGRL-RBFN for prediction. The model demonstrates strong performance, achieving an MAE of 0.047, a Dunn Index of 4.897, a cross-correlation coefficient of 0.9715, and a prediction accuracy of 98.09%, along with high precision (98.17%), recall (97.24%), and F1-score (98.95%). Feature selection was completed efficiently in 2118 ms. For future enhancement, the model should incorporate crop-specific moisture requirements, extend to diverse climatic datasets, and improve adaptability to extreme weather conditions. These improvements would broaden its utility for smart agriculture, sustainable irrigation planning, and climate resilience efforts.