NDMI-Derived Field-Scale Soil Moisture Prediction Using ERA5 and LSTM for Precision Agriculture

Abstract

Accurate soil moisture prediction is fundamental to precision agriculture, facilitating optimal irrigation scheduling, efficient water resource allocation, and enhanced crop productivity. This study employs a Long Short-Term Memory (LSTM) deep learning model, integrated with high-resolution ERA5 remote sensing data, to improve soil moisture estimation at the field scale. Soil moisture dynamics were analyzed across six commercial potato production sites in Quebec—Goulet, DBolduc, PBolduc, BNiquet, Lalancette, and Gou-new—over a five-year period. The model exhibited high predictive accuracy, with correlation coefficients (R) ranging from 0.991 to 0.998 and Nash–Sutcliffe efficiency (NSE) values reaching 0.996, indicating strong agreement between observed and predicted soil moisture variability. The Willmott index (WI) exceeded 0.995, reinforcing the model’s reliability. The integration of NDMI assessments further validated the predictions, demonstrating a strong correlation between NDMI values and LSTM-based soil moisture estimates. These findings confirm the effectiveness of deep learning in capturing spatiotemporal variations in soil moisture, underscoring the potential of AI-driven models for real-time soil moisture monitoring and irrigation optimization. This research study provides a scientifically robust framework for enhancing data-driven agricultural water management, promoting sustainable irrigation practices, and improving resilience to soil moisture variability in agricultural systems.

1. Introduction

Recent advancements in remote sensing (RS) and deep learning (DL) have transformed agricultural water management by enabling more accurate soil moisture predictions, which are essential to precision irrigation [1,2]. Despite the increasing availability of high-resolution remote sensing datasets, estimating soil moisture with high accuracy remains challenging due to spatiotemporal variability and the complex interactions among soil properties, vegetation, and atmospheric conditions [3].

Irrigation in Quebec, Canada, began in the late 1940s as a strategy to stabilize soil moisture for tobacco crops (L.) during dry spells [4]. This practice was later expanded to other crops, such as corn, soybean, vegetables, potato (L.), and horticultural crops [5]. The agricultural regions of Quebec experience ample annual rainfall, averaging 940 mm, with nearly two-thirds falling during the growing season [6]. While this is generally sufficient for crop growth, short drought events from May through August often disrupt soil moisture levels, particularly in the shallow root zones of crops such as vegetables, resulting in significant yield reductions [7]. Based on research by Gallichand et al. [7], irrigation requirements for potatoes on loamy sand for the months May to August are (5–15), (15–30), (25–35), and (10–25) mm/week. These fluctuations highlight the need for advanced predictive models that can accurately capture soil moisture dynamics and optimize irrigation management strategies. Researchers can enhance soil moisture prediction by leveraging deep learning techniques and remote sensing data, ultimately improving water use efficiency and supporting sustainable agricultural practices. These variations in soil moisture due to irrigation highlight the need for advanced predictive models that can accurately capture such fluctuations. Researchers can enhance soil moisture prediction by leveraging deep learning techniques and remote sensing data, ultimately leading to more efficient irrigation management strategies.

Numerous studies have explored deep learning models for soil moisture prediction, including CAE-DLSTM, SoMo.ml, DNNR, and PSO-LSTM [8,9,10,11]. While these models have demonstrated promising results, they often struggle in regions with irregular soil moisture variations, highlighting the need for hybrid modeling approaches or localized fine tuning to improve predictive accuracy. Additionally, their high computational demands make them less suitable for real-time monitoring in resource-constrained environments [12]. For instance, Huang et al. [13] developed a Conv-LSTM model for spatiotemporal soil moisture prediction. However, this approach did not fully exploit spatiotemporal dependencies, and it overlooked irrigation as a key human-induced factor affecting soil moisture dynamics.

Integrating deep learning models with remote sensing data has emerged as a promising strategy to enhance the ability of predictive models to handle spatiotemporal variability. Pan et al. [14] demonstrated that combining LSTM-based models with remote sensing datasets, such as SMAP and ERA5, improved spatial heterogeneity representation and enhanced irrigation precision. Additionally, Ling et al. [15] found that ERA5 reanalysis data effectively captured soil moisture variations across different temporal and spatial scales, making it a reliable dataset for soil moisture prediction.

In remote sensing, among spectral bands, shortwave infrared (SWIR) reflectance is particularly sensitive to changes in soil moisture, as drier conditions increase the fraction of absorbed radiative energy [16]. Similarly, near-infrared (NIR) reflectance is responsive to variations in soil water content [17].

The Normalized Difference Moisture Index (NDMI) is one of the lesser-studied vegetation indices in remote sensing for soil moisture estimation [18]. Recent studies have demonstrated its utility in predicting soil moisture content [19]. For example, Safi et al. [20] used NDMI values to assess water stress levels in crops such as irrigated wheat, potatoes, and grapes, integrating it into broader methodologies for evaluating plant moisture deficits.

Despite these advancements, existing studies have primarily used the NDMI or deep learning models separately for soil moisture estimation, without adequately considering the role of irrigation in influencing soil moisture dynamics. Given the reliance on irrigation to stabilize soil moisture levels in Quebec’s agricultural regions, integrating these factors into predictive models is crucial to improving estimation accuracy and informing better water management strategies. This research study aims to bridge this gap by integrating NDMI-derived moisture assessments with an LSTM-based framework, offering an improved approach to soil moisture prediction in precision agriculture. Specifically, this study seeks to address the challenges of localizing deep learning models and reducing computational demands for soil moisture prediction. To achieve this, we develop a Long Short-Term Memory (LSTM)-based framework that integrates ERA5 remote sensing data to enhance model scalability and adaptability across diverse agricultural landscapes. This approach leverages ERA5 reanalysis data to capture temporal dynamics and soil moisture variability better, ensuring robust, high-resolution predictions.

The specific objectives of this research are as follows:

• Develop and implement an LSTM-based model for high-resolution soil moisture prediction, incorporating both temporal and spatial variability.
• Integrate ERA5 remote sensing data to enhance the accuracy and generalizability of soil moisture predictions across heterogeneous agricultural environments.
• Demonstrate the applicability of the model for precision irrigation, offering insights into optimal water allocation strategies to improve sustainable agricultural water management.

This study hypothesizes that integrating NDMI-derived soil moisture assessments with an LSTM-based framework will enhance soil moisture prediction accuracy by addressing the current gap in research, where the NDMI and deep learning models have been predominantly used in isolation. By leveraging ERA5 remote sensing data, this study further hypothesizes that the generalizability and spatial accuracy of deep learning-based soil moisture predictions will be significantly improved across heterogeneous agricultural environments. Given the ability of LSTM to capture sequential dependencies in time-series data, it is expected that the LSTM model will outperform traditional modeling approaches to capturing spatiotemporal soil moisture dynamics. However, the model’s performance is hypothesized to be influenced by NDMI values and local environmental conditions, where higher NDMI values (indicating sufficient soil moisture) are expected to yield better predictive performance, while drought-prone areas may exhibit higher errors. Lastly, this study posits that LSTM-based soil moisture prediction can support precision irrigation strategies by providing actionable, data-driven insights into optimal water allocation, ultimately reducing water waste and improving crop yields. By testing these hypotheses, this research study aims to contribute to advancing AI-driven soil moisture monitoring and its applications in sustainable agricultural water management.

By integrating deep learning techniques with high-resolution remote sensing data, this study contributes to ongoing research in precision irrigation by addressing the specific challenges of soil moisture variability in Quebec’s agricultural regions. Given the historical reliance on irrigation to mitigate drought effects on crops such as potatoes, this research study provides a data-driven approach to optimizing water use and improving irrigation scheduling in response to local conditions. The proposed LSTM-based framework overcomes limitations of conventional modeling approaches, providing a scalable and adaptable solution for soil moisture estimation. Beyond its scientific contributions, this research study has practical implications, including optimized irrigation scheduling, reduced water wastage, and enhanced agricultural resilience to drought. The findings can inform data-driven decision making for farmers, policymakers, and agricultural stakeholders, fostering the adoption of intelligent irrigation strategies that enhance crop productivity while ensuring efficient water use.

2. Materials and Methods

2.1. Materials

2.1.1. Study Area

In this study, agricultural parcels in Quebec, Canada, are analyzed in relation to Quebec’s agricultural drought monitoring system. The study area includes six agricultural parcels dedicated to potato cultivation, located in the cities of Dolbeau and Peribonka: Goulet, Daniel Bolduc (DB), Patrice Bolduc (PB), Bergeron et Niquet (BN), Marc-André Lalancette (LAL), and a newly identified parcel, Gou-new (Figure 1). The geographical boundaries of the study area are defined as follows: west = −72.23, east = −71.99, north = 48.82, and south = 48.74. These parcels serve as key locations for monitoring soil moisture conditions and assessing drought impact on agricultural productivity.

2.1.2. Normalized Difference Moisture Index (NDMI)

In this section, static data are utilized to assess drought conditions across the studied parcels by monitoring soil moisture levels. This analysis underscores the importance of agricultural drought monitoring within parcels. The Normalized Difference Moisture Index (NDMI) is computed via an image from the Sentinel-2 L2A dataset. Python-based scripts for importing images and indices were executed within a Google Collaboratory Notebook environment. The images were subsequently visualized via ArcGIS Pro (version 3.3.0).

The NDMI is a key indicator for assessing the water content of vegetation and monitoring drought conditions. NDMI values are derived from reflectance values, which are themselves dimensionless and do not have a physical unit (ranging from −1 to 1 or expressed as a percentage). Values approaching −1 indicate dry soil, values near zero (−0.2 to 0.4) typically reflect water stress, and high positive values (approximately 0.4 to 1) are associated with dense, healthy vegetation with no water stress.

The NDMI combines information from the NIR and SWIR bands to assess vegetation moisture. While NIR signals reflect leaf structure and dry matter, SWIR signals are sensitive to water content and mesophyll structure. This combination minimizes interference, improving the accuracy of vegetation water content estimation. SWIR reflectance, which is negatively correlated with leaf water content, is a key indicator for estimating the moisture levels of vegetation [21] (Equation (1)).

$$NDMI={\displaystyle \frac{NIR-SWIR}{NIR+SWIR}},$$

(1)

where NIR represents the near-infrared band value from aerial images and SWIR denotes the shortwave infrared band value from the same image [22].

The classification range of the NDMI is delineated into the following categories: very dry soil, dry soil, slightly dry soil, moderate soil moisture, adequate soil moisture, moist soil, and high soil moisture. The categories are defined specifically for the studies parcels in the current research study inspired by Copernicus Browser and research by [18]. Parcel filtering according to the NDMI, as depicted in Figure 2, reveals that most of the areas are susceptible to soil dryness and low moisture levels at the pixel scale.

2.1.3. Remote Sensing-Based Datasets

Remote sensing moisture indices can be used for irrigation scheduling to monitor water stress spatiotemporally. Given the strengths and weaknesses of each dataset source, merging satellites and modeled products can reduce uncertainty and improve irrigation planning and scheduling. All datasets used in the current study were obtained from model-based products from the land component of the Fifth Generation European Reanalysis (ERA5-Land) dataset. ERA5-Land serves as an alternative way to monitor moisture status over broad spatial extents by providing volumetric soil moisture (VSM) estimates at various depths and time scales [23].

ERA5 and its land-specific component, ERA5-Land, are advanced global reanalysis datasets developed by the ECMWF [24]. While ERA5-Land mainly provides land data at a 9 km spatial scale (ERA5 provides hourly SM measurements at a spatial resolution of 0.08°), the dataset also includes 50 variables related to global land water and energy cycles at the hourly scale. This model-based product encompasses the VSM extracted at different soil depths and a schematic of the soil depths (Figure 2).

The soil depth divisions in ERA5 are based on the Hydrology Tiled ECMWF Scheme for Surface Exchanges over Land (HTESSEL), which parametrizes land surface processes, including soil moisture and temperature dynamics. The model defines three standard layers: 0–7 cm (surface layer), 7–28 cm (subsurface layer), and 28–100 cm (root-zone layer). These divisions are established to represent vertical soil moisture and thermal gradients, considering infiltration, evapotranspiration, and root water uptake [25].

The surface layer (0–7 cm) accounts for rapid interactions with atmospheric fluxes, responding dynamically to precipitation and evaporation, thus playing a critical role in short-term hydrological processes and surface energy balance. The subsurface layer (7–28 cm) acts as an intermediate zone with moderated sensitivity to meteorological forcing, influencing plant water availability and root uptake processes. The root-zone layer (28–100 cm) represents a deeper storage reservoir, characterized by a slower response to atmospheric inputs and controlling longer-term soil moisture trends. Additionally, ERA5-Land extends this framework with a fourth layer (100–289 cm), improving the representation of deep soil moisture storage.

These depth classifications are constrained by numerical modeling requirements while ensuring compatibility with land surface hydrological processes. The layered structure enables the partitioning of water fluxes and energy exchanges, optimizing soil moisture assimilation and retrieval in Earth system models [26].

The ERA5-Land hourly dataset was used to assess hourly trends over the past 5 years, from 2019 to 2023, and in June, July, August, and September (6, 7, 8, and 9, respectively; June and September = 01–30, July and August = 01–31), over 1 h intervals (00:00–23:00). The data were downloaded in NetCDF-3 (Network Common Data Form) format, an experimental application for storing and processing climate and geospatial datasets. The geographical data were clipped via subregion extraction (west = −72.23, east = −71.99, north = 48.82, and south = 48.74). As we focus on soil moisture, the only variable extracted from this dataset was VSM (v, cubic meters per cubic meter; hereafter m³ m⁻³) in layer 1 (0–7 cm).

The aggregated hourly VSM data captured from ERA5 were analyzed over five years, from 2019 to 2023, across the different parcels (Figure 3). The pair plot reveals strong linear correlations between most pairs of parcels, and the range of changes in the VSM content for each parcel was calculated based on the correlation coefficient, which can reflect small changes in the maximum and minimum VSM values.

The trends in the VSM data from ERA5 were closely monitored across the different parcels, and these data were analyzed via histogram plots (Figure 4). The analysis revealed that the data for the Goulet, PBolduc, Lalancette, and DBolduc plots exhibit approximately normal distributions with slight skewness. Notably, the Goulet and DBolduc plots display the highest variability, evident from their distributions’ broader basis. On the other hand, the histograms for the BNiquet and Gou-new parcels exhibit sharp peaks and are skewed; specifically, the BNiquet distribution displays a left skew, whereas the Gou-new distribution exhibits a right skew. These patterns suggest variations in water retention or drainage characteristics among these regions.

Interestingly, the peak VSM values ranged from approximately 0.25 to 0.3 m³/m³ in most parcels, a range that could indicate optimal soil moisture levels for potato cultivation under typical conditions. However, the skewness observed in some plots indicates periods with notably higher or lower soil moisture contents, which must be managed through careful irrigation or drainage practices. The variability and skewness in the soil moisture data suggest the potential benefit of real-time soil moisture monitoring, allowing for dynamic adjustments to irrigation schedules. This is particularly important for parcels such as BNiquet and Gou-new, where the skewness indicates occasional deviations from the norm, highlighting the necessity for more precise water management strategies in these areas.

2.2. Methods

2.2.1. Methodology

In this study, the ERA5 reanalysis model was employed to monitor the soil moisture status based on VSM to support irrigation scheduling. ERA5-Land provides global reanalysis data with hourly meteorological estimates at a 9 × 9 km spatial resolution [24]. The dataset includes soil moisture values (m³ m⁻³) for three soil layers: the upper soil layer (0–7 cm, referred to as “layer 1” or L1), the 7–28 cm depth (L2), and the 28–100 cm depth (L3). The ERA5-Land dataset integrates satellite observations with atmospheric forcing data [23,27].

In this study, VSM (m³ m⁻³) values were extracted from the topsoil layer (0–7 cm, or L1). To ensure consistent spatial coverage and facilitate product comparisons, all datasets were reprojected to the UTM coordinate system by using the WGS84 datum via ArcGIS Pro software (version 3.3.2) [28]. The images were further cropped to precisely align with the boundaries of the study area, maintaining the scope of the analysis to the designated region.

Image processing and analysis were conducted on the Google Colab platform [29], executed in Python (3.11.11 version) [30]. The VSM data were spatially confined to the study area, as defined by geographical boundaries: west = −72.23, east = −71.99, north = 48.82, and south = 48.74. Temporally, the analysis spanned five years (2019–2023), with a focus on June to September. Data were extracted at daily intervals for June (1–30), July and August (1–31), and September, with an hourly interval (00:00–23:00).

This research study involves several key steps: First, the input data are defined both temporally (2019–2023) and spatially, using shapefiles representing all parcels. Satellite data from the ERA5 product are then imported into the environment by installing and utilizing the “geemap” package in Google Colab. Soil moisture values are extracted from the NetCDF-3 file by using the “Zonal Statistics” tool (0.20.0 version), a spatial analysis technique implemented in Python. This method computes summary statistics for raster data within specified zones defined by a vector dataset [31]. Specifically, the median statistic is applied at a 30 m scale, following the recommendation by Huang, Xia [32], for the six stations depicted in Figure 5. Finally, the ERA5-derived data are prepared for machine learning by splitting it into training (70%), validation (15%), and testing (remaining points) subsets, ready for application in the LSTM model.

2.2.2. Long Short-Term Memory (LSTM) Networks

LSTM [33] is a specialized RNN designed to handle long-term dependencies in sequential data. Unlike traditional feedforward networks, it uses feedback connections to retain over long sequences [34]. While standard RNN suffers from vanishing/exploding gradient [35,36,37], LSTM overcomes this issue by incorporating a memory cell and a set of gating mechanisms to control information, retain it, and forget it [38]. A typical LSTM unit consists of memory cells with three main gates: forget, input, and output gates (Figure 5). These gates regulate information flow, selectively retaining or discarding data to maintain efficient memory over time. Sigmoid and tanh activation functions help scale values and decide which piece of information to keep or discard.

LSTM’s effectiveness comes from its cell state and hidden state [38]. The cell state (C_t) acts as memory, retaining and selectively forgetting information over time [39], while the hidden state (h_t) serves as the output, reflecting both historical and current inputs [38]. Together, they enable efficient sequence processing.

The forget gate controls how much of the previous cell state, C_t−1, is retained or discarded. It calculates the forget vector f_t by using the hidden state from the previous time step, h_t−1, and the current input, x_t (Equation (2)):

$$f_t=\sigma \left(W_f\cdot \left[h_{t-1},x_t\right]+b_f\right)$$

(2)

where σ is the sigmoid activation function, which normalizes the output to between 0 and 1, and W_f is the weight matrix for the forget gate. b_f is the bias for the forget gate, and h_t−1 and x_t are concatenated and transformed, enabling the model to focus on specific parts of the input and the previous hidden state.

The input gate in LSTM updates the cell state by first selecting which values should be modified (Equation (3)), ensuring relevant new information is added:

$$i_t=\sigma \left(W_i\cdot \left[h_{t-1},x_t\right]+b_i\right)$$

(3)

They then generate new candidate values, denoted by ${\tilde{C}}_t$, which are added to the cell state (Equation (4)).

$${\tilde{C}}_t=\tanh \left(W_C\cdot \left[h_{t-1},x_t\right]+b_C\right)$$

(4)

The output of the input gate, i_t, and the candidate cell state, ${\tilde{C}}_t$, are combined to update the cell state. The new cell state, C_t, is established by combining forget and input gate information. The formula is as follows (Equation (5)):

$$C_t=f_t·C_{t-1}+i_t·{\tilde{C}}_t$$

(5)

This equation allows the cell to retain information from the previous state C_t−1 and update it with new information from ${\tilde{C}}_t$, guided by the forget and input gates.

The output gate that determines the hidden state, h_t, which serves as the output for the current time step and influences the information passed to the next cell (Equation (6)):

$$o_t=\sigma \left(W_o\cdot \left[h_{t-1},x_t\right]+b_o\right)$$

(6)

The final hidden state h_t is computed by applying the output gate o_t to the updated cell state C_t (Equation (7)):

$$h_t=o_t·tanh\left(C_t\right)$$

(7)

The hidden state h_t is then output and used as the input for the next cell in the sequence.

In an LSTM network, the sigmoid and tanh functions enable information flow. The sigmoid function (Equation (8)) used in the forget, input, and output gates controls how much information passes through by scaling values between 0 and 1, enabling the selective forgetting, updating, and retention of relevant data. The tanh function (Equation (9)) scales state values between −1 and 1, allowing LSTM to handle a broad range of information. This balanced scaling helps capture complex data patterns and relationships, enhancing the network’s performance in modeling sequential data.

$$\sigma (x)={\displaystyle \frac{1}{1+e^{-x}}}$$

(8)

$$\tanh (x)={\displaystyle \frac{e^x-e^{-x}}{e^x+e^{-x}}}$$

(9)

2.2.3. LSTM-Based Modeling

Figure 6 shows a sequential neural network model with LSTM layers for regression tasks. Designed for time-series and sequential data, the model begins with a “sequence input layer”, which preserves the time-dependent nature of the data, ensuring effective sequence processing.

Following the input layer, the first “LSTM layer” captures initial temporal dependencies by using memory cells with gating mechanisms to retain relevant information from previous steps and discard irrelevant details. This helps the model learn both short- and long-term patterns in time-series data [37]. After the first LSTM layer, a ReLU (rectified linear unit) activation layer is applied, introducing nonlinearity by converting negative values to zero, preventing vanishing gradient, and enhancing the model’s ability to capture complex patterns [40].

A second LSTM layer is added to enhance the model’s ability to capture complex sequential dependencies. While the first LSTM layer captures basic temporal and patterns, the second layer extracts more nuanced temporal relationships, enabling hierarchical learning and improving predictive accuracy for complex tasks. A second ReLU activation layer follows the second LSTM layer, adding nonlinearity and improving the model’s ability to fit complex patterns without gradient issues. Placing a ReLU after each LSTM layer enhances flexibility and performance in regression tasks. The output from the second ReLU layer is passed to a fully connected layer, transforming the learned features into suitable output for regression tasks. This layer aggregates information from previous layers, ensuring dense connections that enhance prediction quality.

A dropout layer follows the fully connected layer to prevent overfitting [41] by randomly deactivating neurons during training. This forces the network to rely on various pathways, enhancing generalization and robustness for real-world data variations. The architecture ends with a regression output layer, designed for continuous target variables. It enables precise predictions for tasks like time-series forecasting, ensuring the model is optimized for regression applications.

The LSTM model is trained by using input and output pairs. Notably, the overall dataset is divided into three subsets, 60%, 20%, and 20%, of the data for training, validation, and testing, respectively. The training data are utilized during the training phase to adjust the network parameters. The validation data are used to monitor the training process and assess the occurrence of overfitting.

An LSTM experiment was conducted to determine the optimal parameters for the LSTM model, such as the number of blocks, the number of layers, and the dropout rate. All modeling was performed over 1000 epochs by using the entire dataset, TensorFlow, and the Keras Deep Learning Library on the Python programming platform. We experimented with different optimizers to expedite training and ensure sufficient model performance and stability, thereby eliminating the need for typical regularization methods such as dropout. Ultimately, the Adam optimizer [42,43] was chosen for its superior performance, with a learning rate of 0.0001. The Adam optimization algorithm enhances the computational efficiency of models and improves performance when working with large datasets and numerous parameters [44,45]. The mean square error was used as the basis of the loss function, and the root mean square error was selected as the metric for evaluating model accuracy. The Adam optimization algorithm was used as the descent algorithm in this study. Adam, which stands for Adaptive Moment Estimation, combines the advantages of two other popular optimization methods: the AdaGrad algorithm (which works well with sparse gradients) and RMSProp (which performs well in online and non-stationary settings). This algorithm was selected because it enhances computational efficiency and performance when working with large datasets and models with numerous parameters. By adaptively adjusting the learning rate for each parameter, Adam helps stabilize the optimization process and reduces the need for additional regularization methods such as dropout.

2.2.4. Validation Criteria and Metrics

Given the random variability inherent in hydrological factors, relying solely on one measure to assess a statistical model’s effectiveness is inadequate [46,47]. Therefore, in this study, both graphical representations and standardized statistical indices are applied to assess the reliability of the proposed LSTM model. To gauge its predictive ability, the root mean square error (RMSE), the mean relative absolute error (MRAE), and the correlation coefficient (r) were selected as the primary statistical indicators.

$$RMSE=\sqrt{{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N{\left(S{M}_{i,o}-S{M}_{i,f}\right)}^2}}$$

(10)

$$MARE={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N\left|\raisebox{1ex}{$\left(S{M}_{i,o}-S{M}_{i,f}\right)$}\!\left/ \!\raisebox{-1ex}{$S{M}_{i,o}$}\right.\right|}$$

(11)

$$r=\left({\displaystyle \frac{{\displaystyle \sum _{i=1}^N\left(S{M}_{i,o}-\stackrel{\_\_}{S{M}_o}\right)} \left(S{M}_{i,f}-\stackrel{\_\_}{S{M}_f}\right)}{\sqrt{{{\displaystyle \sum _{i=1}^N\left(S{M}_{i,o}-\stackrel{\_\_}{S{M}_o}\right)}}^2\sqrt{{\displaystyle \sum _{i=1}^{N}S{M}_{i,f}-\stackrel{\_\_}{S{M}_f}}}}}}\right) , -1\le r\le 1$$

(12)

Here, N represents the total number of data points within the testing phase (independent dataset), and SM_i,o and SM_i,f denote the ith values of the observed and forecasted time series, respectively.

The RMSE and the MARE are used to assess the fit of the model without considering the difference in direction between the forecasted and observed values. However, the RMSE is ideally suited for cases in which the forecasting error is normally distributed [48]. In practice, not all models meet this criterion; therefore, in this study, the MARE was employed, as it considers all relative deviations between the forecasted and observed data equally, regardless of the sign [49]. The correlation coefficient r, generally constrained within [−1, 1], quantifies the proportion of variance in the observed values explained by the model, although it requires linearity assumptions. Additionally, r, RMSE, and MARE may lack sensitivity to outliers and additive or proportional discrepancies between model predictions and actual observations [50]. Relying solely on r could lead to misleading results, especially if there is a systematic bias in the forecasted values.

Additional insights into the accuracy of the LSTM model were obtained by calculating Nash–Sutcliffe efficiency (NSE), the refined Willmott index (WI), and the Legates and McCabe index (LMI) as standardized measures of model fit [51].

$$NSE=1-\left[{\displaystyle \frac{{\displaystyle \sum _{i=1}^N{\left(S{M}_{o,i}-S{M}_{f,i}\right)}^2}}{{\displaystyle \sum _{i=1}^N{\left(S{M}_{o,i}-{\stackrel{\_\_}{SM}}_{o,i}\right)}^2}}}\right], -\infty \le NSE\le 1$$

(13)

$$WI=\left\{\begin{array}{c}1-\left[{\displaystyle \raisebox{1ex}{${\displaystyle \sum _{i=1}^N\left(S{M}_{i,f}-S{M}_{i,o}\right)}$}\!\left/ \!\raisebox{-1ex}{$C {\displaystyle \sum _{i=1}^N\left|S{M}_{i,o}-{\stackrel{\_\_}{SM}}_o\right|}$}\right.}\right], when \\ {\displaystyle \sum _{i=1}^N\left(S{M}_{i,f}-S{M}_{i,o}\right)}\le C {\displaystyle \sum _{i=1}^N\left|S{M}_{i,o}-{\stackrel{\_\_}{SM}}_o\right|}\\ \\ \left[{\displaystyle \raisebox{1ex}{$C {\displaystyle \sum _{i=1}^N\left|S{M}_{i,o}-{\stackrel{\_\_}{SM}}_o\right|}$}\!\left/ \!\raisebox{-1ex}{${\displaystyle \sum _{i=1}^N\left(S{M}_{i,f}-S{M}_{i,o}\right)}$}\right.}\right]-1, when \\ {\displaystyle \sum _{i=1}^N\left(S{M}_{i,f}-S{M}_{i,o}\right)}>C {\displaystyle \sum _{i=1}^N\left|S{M}_{i,o}-{\stackrel{\_\_}{SM}}_o\right|}\end{array}\right., 0\le WI\le 1$$

(14)

$$LMI=1-\left[{\displaystyle \frac{{\displaystyle \sum _{i=1}^N\left|S{M}_{o,i}-S{M}_{f,i}\right|}}{{\displaystyle \sum _{i=1}^N\left|S{M}_{o,i}-{\stackrel{\_\_}{SM}}_{o,i}\right|}}}\right], 0\le LMI\le 1$$

(15)

In Equation (14), the parameter C is generally set to a value of 2 [52]. NSE is used to assess a model’s ability to predict data that diverge from the mean [51]; however, this measure is sensitive to discrepancies between observed and predicted values [50]. To further evaluate the predictive capacity of the model, the WI was utilized as an improvement over the RMSE, r, and NSE. The WI quantifies the ratio of the mean square error to the potential error, multiplied by the number of observations, and is then subtracted from 1. This metric, which focuses on squared differences, addresses NSE’s lack of sensitivity [53], as it encompasses the ratio of model errors rather than their squared deviations [54]. The Legates and McCabe index (0 ≤ LMI ≤ 1.0) was also used because of its advantage over the WI, especially when high forecast values are expected, even if the model fit is poor [54]. Unlike the WI, which may yield elevated values due to the squared differences between the observed and predicted data, the LMI provides a more refined approach by applying a balanced weighting to model errors. Specifically, the LMI reduces the influence of extreme values or outliers, which can disproportionately affect the WI due to its reliance on squared differences. This characteristic makes the LMI particularly effective in scenarios where high forecast values or significant discrepancies are expected, ensuring a more robust evaluation of model performance [53]. Thus, the LMI [50] was chosen as an improvement over the WI, as it applies balanced weighting to model errors and differences without inflating them through squaring.

3. Results

3.1. LSTM Analysis Across Six Parcels

The performance of the LSTM model was evaluated across six agricultural parcels, Goulet, Gou-new, BNiquet, DBolduc, PBolduc, and Lalancette, by using scatter plots (Figure 7) and multiple statistical indices (Figure 8). Figure 7 demonstrates strong agreement between observed and predicted soil moisture values, with most data points closely clustered around the 1:1 line and falling within the ±5% deviation range. This pattern indicates that the model effectively captures soil moisture dynamics across all locations, with minimal prediction errors.

Quantitative metrics in Figure 8 further validate the model’s accuracy. The coefficient of determination (R²) ranged from 0.991 to 0.998, with the highest values observed for PBolduc (0.997) and BNiquet (0.998). Similarly, the Nash–Sutcliffe efficiency (NSE) values were highest for BNiquet (0.996), PBolduc (0.994), and DBolduc (0.994), highlighting robust performance in these parcels. The Willmott index (WI) values exceeded 0.995 for all parcels, peaking at 0.999 for DBolduc, PBolduc, and BNiquet, indicating near-perfect agreement between predictions and observations.

Despite the overall high performance, some variability was observed. Lalancette and BNiquet exhibited the highest mean absolute relative error (MARE) and root mean square error (RMSE), reflecting relatively higher prediction errors in these parcels. The lower Legates and McCabe index (LMI) values for Lalancette (0.927) and Goulet (0.932) suggest that unique environmental or soil characteristics could influence model accuracy in these areas.

Figure 9 further explores the model’s error dynamics, showing that more than 90% of predictions for all parcels have a relative absolute error (RAE) of less than 1%. Goulet, Gou-new, and PBolduc demonstrated the steepest cumulative frequency curves, indicating the highest predictive accuracy, while Lalancette and BNiquet exhibited slower curves, reflecting larger error margins.

Finally, Figure 10 compares the discrepancy ratios across parcels. Goulet, Gou-new, and BNiquet showed the closest agreement to the ideal ratio of 1.00, while Lalancette displayed greater variability, particularly at higher soil moisture levels. These findings highlight the need for targeted improvements, such as incorporating localized soil and meteorological data, to enhance model accuracy in parcels with higher variability.

In summary, the LSTM model demonstrates robust performance across all parcels, with Goulet, Gou-new, and PBolduc showing the best results based on high accuracy and low error rates. Conversely, Lalancette and BNiquet exhibit slightly higher errors, suggesting that localized soil and environmental factors influence model performance. These results indicate that while LSTM effectively captures soil moisture dynamics across the study area, its predictive accuracy varies depending on site-specific conditions.

3.2. Correlation Between LSTM Model Performance and NDMI-Based VSM Assessment

The performance of the LSTM model aligns closely with the VSM conditions revealed by the NDMI, providing a comprehensive understanding of the dynamics across the six parcels (Figure 11). PBolduc and Goulet demonstrated the best results in both analyses. Their high LSTM predictive accuracy is supported by the NDMI maps, which show extensive regions of higher NDMI values (green tones), indicating adequate soil moisture and healthy vegetation. The histograms for these parcels further confirm this, with values concentrated near the higher end of the water stress range (−0.1 to 0.4). These findings suggest that effective irrigation or favorable environmental conditions have contributed to consistent moisture levels, enabling the LSTM model to perform reliably.

In contrast, Gou-new and DBolduc displayed moderate performance in both LSTM- and NDMI-based evaluations. The NDMI maps (Figure 11) and histograms (Figure 12) for these parcels reveal a mix of dry and moist areas, reflecting spatial variability in soil moisture, likely caused by differences in soil drainage or retention capacity. This variability presents challenges for maintaining consistent soil moisture conditions and impacts the LSTM model’s predictive reliability. Targeted interventions, such as localized irrigation strategies, could help address these issues and enhance both soil moisture balance and model performance.

The most drought-prone parcels, Lalancette and BNiquet, exhibited the lowest NDMI values and the highest LSTM prediction errors. The NDMI maps show extensive areas of low values (orange and yellow tones), while histograms indicate concentrations near the lower range (−0.2 to −0.1), reflecting severe water stress. The LSTM model’s higher errors for these parcels, as evidenced by elevated MARE and RMSE values, highlight the difficulty of accurately predicting soil moisture under such extreme drought conditions. Lalancette shows significant variability in both NDMI values and LSTM performance, suggesting that unique environmental factors, such as poor soil retention or inconsistent irrigation, exacerbate these challenges.

The correlation between the NDMI and LSTM results underscores the utility of the NDMI in diagnosing soil moisture variability and guiding model refinements. Higher NDMI values correspond to better LSTM performance, while parcels with a low NDMI require targeted water management strategies to mitigate drought stress and improve model reliability. These findings suggest that integrating NDMI-based insights with localized soil and meteorological data can enhance both moisture assessments and predictive modeling, ensuring sustainable water management and precision agriculture practices.

4. Discussion

4.1. Predictive Performance of LSTM Model

The results of this study demonstrate the effectiveness of LSTM in predicting soil moisture dynamics, particularly in well-irrigated agricultural parcels, where its predictive accuracy closely aligns with the soil moisture variability captured by the NDMI. The model achieved strong performance in PBolduc and Goulet, where higher NDMI values indicated adequate moisture conditions and healthy vegetation. These findings are consistent with those obtained by Han et al. [8], and Qi et al. [55], who reported that deep learning models, particularly CAEDLSTM, enhance soil moisture predictions by capturing temporal dependencies. While CAEDLSTM achieved a 5.01% increase in R² and a 12.89% reduction in RMSE compared with standard LSTM, its computational complexity limits real-time applications. In contrast, the NDMI-integrated LSTM approach presented in this study provides a computationally efficient alternative, demonstrating high predictive accuracy under stable moisture conditions.

Despite these promising results, moderate prediction variability was observed in Gou-new and DBolduc, where the NDMI maps showed a mix of dry and moist areas, suggesting that soil drainage capacity and retention influence soil moisture variability. These discrepancies are similar to the findings obtained by Zheng et al. [10], who noted that deep learning models such as DNNR exhibited higher errors in regions with irregular soil moisture patterns. Their study emphasized the importance of multi-feature processing and regional calibration, which aligns with the need to refine LSTM predictions through additional environmental parameters, such as precipitation and evapotranspiration.

The largest prediction errors were observed in Lalancette and BNiquet, the most drought-prone parcels, where the NDMI values were the lowest. The LSTM model showed higher MARE and RMSE in these areas, likely due to extreme soil moisture variability and limited representation of such conditions in the training data. These results are consistent with those obtained by Pan et al. [14], who developed a Graph Convolutional LSTM (GCCL) model to address soil moisture heterogeneity. Their graph-based framework improved prediction accuracy in complex topographical regions but required high-resolution data and increased computational power, a limitation that LSTM mitigates by offering a more practical alternative for precision agriculture.

4.2. Role of NDMI in Improving Soil Moisture Prediction

The observed correlation between NDMI values and LSTM performance reinforces the importance of integrating remote sensing indices with deep learning for soil moisture estimation. Higher NDMI values consistently corresponded to lower prediction errors, while lower NDMI values were associated with greater discrepancies. These findings align with those obtained by Wu et al. [11], who found that PSO-LSTM, optimized with Sentinel-1 VV and VH polarization data, improved soil moisture prediction accuracy (R² = 0.96). However, their study was limited to citrus orchards, whereas this research study applies NDMI-derived soil moisture estimates across multiple agricultural parcels, broadening its applicability.

Several studies have suggested that LSTM models struggle with extreme soil moisture conditions, particularly under drought stress. Ling et al. [15] and Ojha et al. [56] emphasized that model-based soil moisture products, such as ERA5 and GLDAS, fail to capture irrigation effects due to the absence of direct irrigation signals. In contrast, our study incorporates the NDMI as an indirect measure of soil moisture, enhancing prediction accuracy in precision irrigation applications. Future improvements could integrate multi-sensor data fusion techniques—a recommendation also proposed by Ling et al. [15], who found that ESA CCI SM provided better soil moisture estimates but suffered from temporal variability issues.

4.3. Comparing LSTM with Hybrid and Physics-Based Models

Although the LSTM model demonstrated strong predictive capabilities, it has inherent limitations in capturing extreme moisture variability, like other deep learning models. Several studies have explored hybrid approaches to addressing these challenges. The authors in [57] reported that RNN-LSTM models were highly effective in predicting soil moisture at shallow depths (10–20 cm), but performance declined at 30 cm, where internal soil properties rather than meteorological factors influenced soil moisture retention. Our findings mirror these results, as the predicted accuracy was higher in surface layers, where soil moisture is more dynamic.

Similarly, Huang et al. [13] demonstrated that Conv-LSTM models incorporating static and dynamic variables (land cover, soil type, elevation, etc.) improved prediction accuracy (R² = 0.92 compared with R² = 0.84 without static variables). While our study did not incorporate land surface characteristics as explicit input variables, the NDMI-based assessment provides indirect information on vegetation health and moisture levels, supporting the argument that integrating physical and remote sensing data enhances soil moisture prediction accuracy.

Another promising direction is the fusion of machine learning with physics-based hydrological models. Li et al. [58] highlighted that hybrid models (e.g., LSTM + CNN and LSTM + Random Forest) outperform standard deep learning models by improving spatial generalization. The observed discrepancies in this study suggest that integrating ensemble learning or graph-based models could refine predictions in regions with complex moisture dynamics.

4.4. Implications for Precision Agriculture and Future Research

While the NDMI-integrated LSTM model presents a promising tool for soil moisture prediction and irrigation management, several limitations must be addressed to enhance its applicability. One of the key challenges is the reliance on remote sensing data, such as ERA5-derived soil moisture estimates, which may have limited spatial resolution. Moreover, LSTM models require large datasets for training, making them less feasible for regions with sparse ground observations. Similar concerns were raised by Ling et al. [15], who suggested improving bias correction techniques for soil moisture estimation. Future research should focus on integrating higher-resolution meteorological and soil data to reduce uncertainty and develop adaptive models that can be dynamically fine-tuned for specific locations.

Incorporating multi-source remote sensing products (e.g., Sentinel-1, Sentinel-2, and MODIS) could further enhance spatial and temporal resolution, as suggested by Wu et al. [11] and Pan et al. [14].

Additionally, hybrid modeling techniques such as combining LSTM with physics-based hydrological models could improve interpretability and predictive robustness, ensuring better alignment with ground-truth soil moisture measurements.

Overall, this study confirms the potential of LSTM models for precision irrigation, particularly when integrated with NDMI-based assessments. By refining model calibration with additional environmental variables and improving training datasets, LSTM models can become a valuable tool for optimizing irrigation strategies and supporting sustainable agricultural practices under increasing climate variability.

5. Conclusions

This study confirms the effectiveness of LSTM models in predicting soil moisture across six agricultural parcels, with a strong alignment between LSTM predictions and NDMI-based soil moisture assessments. The model achieved high accuracy in PBolduc, Goulet, and Gou-new, where R² values exceeded 0.995, NSE values were above 0.994, and WI values peaked at 0.999, indicating near-perfect agreement with observed soil moisture.

Conversely, Lalancette and BNiquet exhibited the highest errors, with the lowest Legates and McCabe index (LMI) values of 0.927 and 0.932, respectively, highlighting challenges in predicting soil moisture under high variability. The highest mean absolute relative error (MARE) and root mean square error (RMSE) were also observed in these parcels, indicating greater prediction uncertainty.

Additionally, the correlation between NDMI values and LSTM performance revealed that higher NDMI values corresponded to better model accuracy, while lower NDMI values aligned with increased prediction errors, underscoring the importance of integrating remote sensing data in soil moisture assessment.

These findings suggest that while LSTM models are highly effective in capturing soil moisture dynamics, their accuracy is influenced by spatial heterogeneity and extreme drought conditions. The model’s predictive capability makes it a valuable tool for smart irrigation applications, as it can provide real-time soil moisture predictions that help optimize irrigation scheduling. By incorporating NDMI-based assessments and additional environmental variables (e.g., precipitation, evapotranspiration, and soil texture), the model can support data-driven decision making for precision irrigation, ensuring efficient water use and improved crop health.