A Two-Step Reconstruction Approach for High-Resolution Soil Moisture Estimates from Multi-Source Data

Abstract

Accurate soil moisture (SM) estimates with high spatial resolution are highly desirable for agricultural, hydrological, and environmental applications. This study developed a two-step reconstruction approach to obtain a high-quality and high-spatial-resolution (0.05°) SM dataset from microwave and model-based SM products, combining Bayesian three-cornered hat (BTCH) merging and machine/deep learning downscaling algorithms. Firstly, a three-cornered hat (TCH) method was used to analyze the uncertainty of seven SM products on four main land cover types in the Pearl River Basin (PRB). On this basis, the SM products with low uncertainty were merged using the BTCH method. Secondly, two machine/deep learning algorithms (random forest, RF, and long short-term memory, LSTM) were applied to downscale the merged SM data from 0.25° to 0.05° based on the relationship between SM and auxiliary variables. The overall performance of RF and LSTM downscaling models with/without antecedent precipitation were compared. The merged and downscaled SM results were validated against in situ observations and the China Meteorological Administration (CMA) Land Data Assimilation System (CLDAS) SM data. The results indicated the following: (1) The BTCH-based SM estimate outperformed the parent products and the AVE-based SM estimate (the arithmetic average), indicating that BTCH is a fusion approach that can effectively reduce data uncertainties and optimize weights. (2) The optimal time scale for the cumulative effect of precipitation on SM was 35 days during 2015–2020 in the PRB. SM estimations using RF and LSTM downscaling algorithms both had substantial improvement by considering the antecedent precipitation variable, both at the 0.25° and 0.05° spatial scales. Feature importance assessment also revealed the most important role of antecedent precipitation (30.01%). Moreover, the LSTM model with antecedent precipitation performed slightly better than the RF model with antecedent precipitation. (3) The downscaled SM results all mitigated the overestimation inherent in the original SM data, though they were inevitably limited by the performance of the original SM data and difficult to surpass. The developed two-step reconstruction approach was effective in generating an accurate SM dataset at a finer spatial scale for wide regional applications.

1. Introduction

Soil moisture (SM) is a key state variable in the Earth’s system that controls the exchange of water and energy fluxes between the land surface and atmosphere [1,2], and it has been widely utilized in numerous disciplines and applications. Accurate and detailed SM information is highly desirable for many regional applications, such as droughts and floods [3,4], agricultural irrigation [5], and water resources management [6]. However, due to the common impact of meteorological, topographic, vegetative, and pedological factors [7,8], SM generally presents large spatiotemporal variability, and it is still challenging to obtain accurate SM data with high spatial resolution.

SM can be obtained through ground-based observations, remotely sensed estimation, and land surface modeling, each with their own strengths and limitations. Specifically, ground observations can provide precise SM estimation at multi-layer depths but are costly and labor-intensive, and they suffer a lot from the poor spatial representativeness of discrete information at particular points [9]. Satellite-based SM products can estimate spatially continuous SM of the surface soil layer (0–5 cm) at a large scale, but they contain considerable uncertainties caused by underlying conditions, instrumental deficiencies, and retrieval algorithms [10]. Land surface models can generate spatial completeness and temporal continuity SM simulations of different soil layers globally; however, their accuracy has unignorable errors from the model parameterization, model structure, and quality of forcing data [11]. Data merging is currently an effective solution to improve SM quality, which could overcome the drawbacks and combine the superiorities of each individual dataset [12]. Moreover, microwave and model-based SM products usually have relatively coarse spatial resolution (around tens of kilometers), greatly hindering their applications in many regional hydrological and agricultural studies. Hence, it was of great significance to acquire accurate SM data merging from multi-source datasets at a fine spatial resolution, particularly at the basin scale.

Data fusion can integrate multi-source datasets and knowledge into a better data product [13]. Linear weight averaging is the most common and preferable way for data fusion due to its simplicity, interpretability, applicability to any number of datasets, and visible improvements in practice [14]. The key to linear weight averaging is to calculate the weight values derived from the correlations or errors of the individual components to be fused. At present, triple collocation (TC) [15] and three-cornered hat (TCH) [16] are two commonly used methods for estimating the error variance of soil moisture retrievals without any a priori knowledge. The difference was that the TC method can only quantify the error variance of three mutually independent datasets, while the TCH method can be applied to more than three time series and considered the cross correlation among datasets [9]. Linear weighted data fusion based on TCH-derived error has been adopted in numerous data fusion studies focusing on various geophysical variables such as precipitation [17], gross primary productivity [18], and evapotranspiration [19]. Nevertheless, for SM, most research has been conducted on uncertainty analysis [20], with few focusing on the multi-source data fusion based on TCH uncertainty. Combining the satellite-based retrievals and model simulations can significantly reduce the uncertainties of SM data on a global scale [21]. Therefore, a TCH-based linear fusion of SM estimates from active and passive satellites and models is necessary to be investigated.

To obtain SM estimates with high spatial resolution, various spatial downscaling techniques have been proposed by accounting for the impact of numerous auxiliary variables, such as the regression fitting [22], the Disaggregation based on Physical And Theoretical scale CHange (DISPATCH) [23], the triangular/trapezoidal feature space [24], and machine/deep learning approaches [8]. The idea behind these methods is to establish either a statistical correlation or a physically based model between coarse-scale SM and fine-scale auxiliary variables [25]. There are two main differences among these methods, including different characteristics of downscaling models and different types of auxiliary input data. For downscaling model construction, machine/deep learning algorithms, such as random forest (RF) [2,26], long short-term memory networks (LSTM) [10], and convolutional neural networks (CNNs) [27], have received the most attention in recent years owing to their advantages in dealing with massive remote-sensed data and non-linear problems [28]. For auxiliary variables closely related to SM, land surface temperature (LST), vegetation index, surface albedo, brightness temperature, contemporary precipitation, SM memory, topographic factors, and soil texture are all widely applied in various SM downscaling models [2,8,26,29,30,31]. Based on the principle of water balance, the current soil water content is primarily determined by the effective precipitation that has occurred in the preceding period if no irrigation [32,33]. However, few studies have viewed antecedent precipitation as an important spatial downscaling feature. Thus, adding an antecedent precipitation variable to the downscaling model and considering the nonlinear relationship between SM and corresponding auxiliary variables may theoretically improve the estimation accuracy of SM.

Within this context, this study aimed to develop a two-step reconstruction approach based on the TCH-based linear fusion and machine/deep learning downscaling algorithms, which was designed to generate a high-quality pentad (5-day) SM dataset with a spatial resolution of 0.05° from microwave and model-based SM products. Specifically, the objectives of this study were (1) to identify the effectiveness of the TCH-based fusion method in improving SM data accuracy, (2) to evaluate the impacts of antecedent precipitation on downscaled SM quality in the spatial downscaling schemes, and (3) to validate the performance of the merged and downscaled SM results against in situ observations and the China Meteorological Administration (CMA) Land Data Assimilation System (CLDAS) SM data. The results of this study would be valuable in providing more detailed and accurate information on SM for regional hydrometeorological applications.

2. Study Area and Data

2.1. Study Area

The Pearl River Basin (PRB) is located in southern China (Figure 1), with a mainstem length of 2214 km and an area of 453,700 km². This study area can generally be divided into three geographical zones from west to east, including the Yunnan–Guizhou Plateau, the Guangdong–Guangxi Hills, and the Pearl River Delta (PRD) [34], with elevations ranging from −55 to ~2933 m. Located in the subtropical monsoon climate zone, the PRB has an average annual temperature of 14 to 22 °C and average precipitation of 1200 to 2200 mm, with 80% of the total annual precipitation falling between April and September each year [26]. Influenced by the complex topography and climate, extreme hydrological events such as floods and droughts occur frequently in the basin, requiring detailed and accurate SM information. The PRD was not considered in this paper because the high imperviousness makes it difficult for the soil to store water through precipitation infiltration.

2.2. Data

2.2.1. Soil Moisture Data

SM products used in this study included four microwave products (one active and three passive), three model-based products, and two validation datasets, with their main characteristics shown in

Table 1. Overview of SM data used in this study.

Datasets	Spatial Resolution	Temporal Resolution	Spatial Projection	Unit	Depth
CCI-active	0.25°	Daily	GCS_WGS_1984	%	0~3 cm
AMSR2	0.25°	Des: 01:30 Asc: 13:30	GCS_WGS_1984	m3/m3	0~2 cm
FY-3C	25 km	Des: 10:00 Asc: 22:00	EASE-Grid	m3/m3	0~2 cm
SMAP	36 km	Des: 06:00 Asc: 18:00	EASE-Grid 2.0 Global	m3/m3	0~5 cm
ERA5-Land	0.1°	Hourly	GCS_WGS_1984	m3/m3	0~7 cm
GLDAS	0.25°	3-h	GCS_WGS_1984	m3/m3	0~10 cm
GLEAM	0.25°	Daily	GCS_WGS_1984	m3/m3	0~10 cm
CLDAS	0.0625°	Hourly	GCS_WGS_1984	m3/m3	0~5 cm
In situ	Point scale	Hourly	--	m3/m3	0~10 cm

.

The active microwave product was derived from the European Space Agency’s (ESA) Soil Moisture Climate Change Initiative (CCI) project [35], hereafter referred to as CCI-active (version 7.1). The relative SM (%) was converted into volumetric unit (m³/m³) using soil porosity information from the ESA CCI.

Three passive microwave products included SM datasets from the Advanced Microwave Scanning Radiometer 2 (AMSR2) [36], the Fengyun-3C (FY-3C) [37], and the Soil Moisture Active Passive (SMAP) [38]. To reduce radio frequency interference influence [39], the Land Parameter Retrieval Model (LPRM) AMSR2 L3 X-band SM product was used (hereafter AMSR2). FY-3C is China’s second-generation polar-orbiting meteorological satellite that was launched in September 2013. For this study, the FY-3C L2 SM observations from April 2015 to February 2020 were used. The SMAP product used in this study was the level-3 daily SM product (version 8) posted on a 36 km grid using EASE-Grid 2.0. Daily SM for each of the three products was obtained by averaging their ascending and descending datasets.

Three model-based products included SM datasets from the land component of the 5th generation of European ReAnalysis (ERA5), referred to as ERA5-Land [40], NASA’s Global Land Data Assimilation System (GLDAS) [41], and the Global Land Evaporation Amsterdam Model (GLEAM) [42]. ERA5-Land hourly data provide SM estimates at four soil layers (0–7, 7–28, 28–100, and 100–289 cm), in which the top layer SM data were used. The GLDAS product used in this study was the GLDAS-2.1 Noah L4 3-hourly 0.25° data (hereafter GLDAS), which describes SM in four layers (0–10, 10–40, 40–100, and 100–200 cm), and the top 0–10 cm SM data were used. For GLEAM, the daily surface SM (0–10 cm) of version 3.6a was collected. The ERA5-Land hourly and GLDAS 3-hourly SM estimates were averaged to yield the daily SM.

In situ SM measurements and the CLDAS SM data (version 2.0) [43] were used as validation datasets. The CLDAS provides hourly SM in five layers (0–5, 0–10, 10–40, 40–100, and 100–200 cm), in which the top 0–5 cm SM data from January 2017 to December 2019 were applied. In situ observations were collected from China’s national meteorological stations, providing hourly SM from 10 to 100 cm depth with an interval of 10 cm, and the top 0–10 cm SM data were used. Hourly SM estimates from the CLDAS and in situ observation were averaged to obtain the daily value.

2.2.2. Auxiliary Data

Land cover type data were from GlobeLand30 [44] in the year 2020, with an overall classification accuracy of 85.72%. There are a total of six land cover types in the PRB, with cropland, woodland, grassland, and shrubland being the four main land cover types after water bodies and artificial surfaces are removed. Precipitation data were collected from the Climate Hazards Group Infrared Precipitation with Stations (CHIRPS) daily product. All-weather LST data were from the Global daily 0.05° spatiotemporal continuous land surface temperature dataset (2002–2022) hosted at the National Tibetan Plateau Data Center [45]. The datasets from four overpass times were averaged to obtain the daily LST. Enhanced Vegetation Index (EVI) data were collected from the MOD13C1 (v6.1). Quality control, linear interpolation, and Savitzky–Golay (S–G) filter [46] methods were applied to obtain a high-quality daily EVI dataset. Albedo data were obtained from the MCD43C3 (v6.1), in which shortwave black-sky albedo and white-sky albedo layers were quality controlled and used to calculate the real surface albedo [47]. Then, linear interpolation and S–G filter methods were also applied to generate a gap-filled and high-quality real surface albedo dataset. Digital elevation model (DEM) data were obtained from the NASA Shuttle Radar Topography Mission [48]. Elevation, slope, and aspect were extracted as topographical factors. Soil texture data (proportion of sand, silt, and clay content) were obtained from the Harmonized World Soil Database version 1.2 [49].

Table 2. Overview of auxiliary data used in this study.

Datasets	Variables	Spatial Resolution	Temporal Resolution
GlobeLand30	Land cover types	30 m	Static
CHIRPS v2.0	Precipitation	0.05°	Daily
MOD11C1/MYD11C1	LST	0.05°	Four times a day
MOD13C1	EVI	0.05°	16-day
MCD43C3	Albedo	0.05°	Daily
SRTM	Elevation, slope, and aspect	90 m	Static
HWSD v1.2	Soil texture	1 km	Static

lists the basic information of the above datasets.

2.2.3. Data Preprocessing

To ensure that all datasets had a uniform specification, data preprocessing was performed as follows. The time range of all SM and auxiliary datasets was unified from April 2015 to December 2020, except for the FY-3C and CLDAS. The spatial resolution of FY-3C, SMAP, ERA5-Land, and GlobeLand30 datasets was resampled to 0.25°, consistent with other SM products. The spatial resolution of DEM and soil texture data was resampled to 0.05°, consistent with other auxiliary products. The temporal resolution was unified as pentad (5-day average, and sum for precipitation), and the geospatial reference was set to GCS_WGS_1984.

3. Methods

3.1. SM Reconstruction Approach

Figure 2 shows the two-step SM reconstruction approach (first merging then downscaling) developed in this study. Note that the spatial downscaling was conducted based on an assumption that models established by auxiliary variables over an appropriate range of scales are applicable to SM reproduction [50]. The specific steps are described as follows:

• The TCH method was firstly used to analyze uncertainties of multiple SM products on different land cover types at a spatial resolution of 0.25°. The products with low uncertainties were then combined to produce a merged SM product using the BTCH merging method.
• All auxiliary variables including real albedo, precipitation, EVI, LST, soil texture (sand, clay, and silt), and terrain (elevation, slope, and aspect) were resampled to 0.25° spatial resolution, the same as the merged SM product.
• RF and LSTM models were constructed between the merged SM and auxiliary variables at a coarse spatial resolution (0.25°), and the corresponding downscaled SM was generated by inputting auxiliary variables at a high spatial resolution (0.05°). To explore if antecedent precipitation can improve the downscaling results, we trained the RF and LSTM models under two conditions each: with and without the antecedent precipitation, for comparison.
• The performances of each downscaled SM and the original (merged) SM product were evaluated using in situ observation and the CLDAS SM data, and they were inter-compared.

Figure 2. The SM reconstruction approach developed in this study.

Figure 2. The SM reconstruction approach developed in this study.

3.2. Bayesian Three-Cornered Hat (BTCH) Merging

Suppose there are N sets of SM products ${\left\{{SM}_i\right\}}_{i=1, 2,\cdots ,N}$, $i$ corresponds to different products. According to the Bayesian theory, the probability density function (PDF) for the $i$th SM product (${SM}_i$) can be expressed as

$$p\left({SM}_i|{SM}_{true}\right)={\displaystyle \frac{1}{{\sigma }_i\sqrt{2\pi }}}exp\left[-{\displaystyle \frac{{\epsilon }_i^2}{2{\sigma }_i^2}}\right]=L\left({SM}_{true}|{SM}_i\right)$$

(1)

$${\epsilon }_i={SM}_i-{SM}_{true}$$

(2)

where ${SM}_{true}$ is the true value of SM (actually, ${SM}_{true}$ is not available and is a hypothetical true value, also representing the SM value after merging), and ${\epsilon }_i$ and ${\sigma }_i$ are the zero-mean white noise and error variance of ${SM}_i$, respectively. L(•) is the likelihood function.

In the same way, the PDF for $j$th SM product (${SM}_j$) can be expressed as

$$p\left({SM}_j|{SM}_{true}\right)={\displaystyle \frac{1}{{\sigma }_j\sqrt{2\pi }}}exp\left[-{\displaystyle \frac{{\epsilon }_j^2}{2{\sigma }_j^2}}\right]=L\left({SM}_{true}|{SM}_j\right)$$

(3)

$${\epsilon }_j={SM}_j-{SM}_{true}$$

(4)

where ${\epsilon }_j$ and ${\sigma }_j$ are the zero-mean white noise and error variance of ${SM}_j$, respectively.

The maximum likelihood of true SM (${SM}_{true}$) is the maximum value of its joint probability distribution:

$$\begin{gathered} \mathrm{m}\mathrm{a}\mathrm{x}L\left({SM}_{true}|{SM}_i{,SM}_j\right)=p\left({SM}_i|{SM}_{true}\right)p\left({SM}_j|{SM}_{true}\right) \\ ={\displaystyle \frac{1}{2\pi {\sigma }_i{\sigma }_j}}exp\left[-{\displaystyle \frac{{\epsilon }_i^2}{2{\sigma }_i^2}}-{\displaystyle \frac{{\epsilon }_j^2}{2{\sigma }_j^2}}\right] \end{gathered}$$

(5)

To obtain the maximum likelihood value of ${SM}_{true}$, the cost function $J$ is defined as

$$J\left({SM}_{true}\right)={\displaystyle \frac{{\epsilon }_i^2}{2{\sigma }_i^2}}+{\displaystyle \frac{{\epsilon }_j^2}{2{\sigma }_j^2}}={\displaystyle \frac{1}{2}}\left[{\displaystyle \frac{{\left({SM}_i-{SM}_{true}\right)}^2}{{\sigma }_i^2}}+{\displaystyle \frac{{\left({SM}_j-{SM}_{true}\right)}^2}{{\sigma }_j^2}}\right]$$

(6)

Letting the first variation of $J\left({SM}_{true}\right)$ be zero, $J_0\left({SM}_{true}\right)=0$, then ${SM}_{true}$ could be obtained as

$${SM}_{true}={\displaystyle \frac{{\sigma }_i^2}{{\sigma }_i^2+{\sigma }_j^2}}{SM}_i+{\displaystyle \frac{{\sigma }_j^2}{{\sigma }_i^2+{\sigma }_j^2}}{SM}_j$$

(7)

If we define ${SM}_{true}=w_i{SM}_i+w_j{SM}_j$, then

$$w_i={\displaystyle \frac{{\sigma }_i^2}{{\sigma }_i^2+{\sigma }_j^2}}, w_j={\displaystyle \frac{{\sigma }_j^2}{{\sigma }_i^2+{\sigma }_j^2}}$$

(8)

where $w_i$ and $w_j$ are weight values of ${SM}_i$ and ${SM}_j$, respectively.

Similarly, for N sets of SM products, Equation (7) can be expressed as

$${SM}_{true}=w_1{SM}_1+w_2{SM}_2+\cdots +w_N{SM}_N$$

(9)

The weight values of each SM product (e.g.,$w_k$) are defined as

$$w_k={\displaystyle \frac{{\prod }_{i=1,i\ne k}^N{\sigma }_i^2}{{\sum }_{k=1}^N\left({\prod }_{i=1,i\ne k}^N{\sigma }_i^2\right)}}$$

(10)

The error covariance (${\sigma }_i$) of each SM product is derived by the TCH method [16]. Since ${SM}_{true}$ is not available, the difference series between (N − 1) SM products and a reference SM product (${SM}_{ref}$) (chosen arbitrarily from N SM products) can be expressed as

$$y_i={SM}_i-{SM}_{ref}={\epsilon }_i-{\epsilon }_{ref}, i=1, 2, \cdots , N-1$$

(11)

Note that the error covariance results of the TCH method are independent of the selected reference dataset [51]. Store the (N − 1) difference sequence in the following matrix:

$$Y=\left[\begin{array}{cccc}{y}_{11}& y_{12}& \cdots & y_{1\left(N-1\right)}\\ y_{21}& y_{22}& \cdots & y_{2\left(N-1\right)}\\ ⋮& ⋮& \ddots & ⋮\\ y_{M1}& y_{M2}& \cdots & y_{M\left(N-1\right)}\end{array}\right]$$

(12)

where $Y$ is the difference matrix with M rows and (N − 1) columns (M is the number of each SM product). The covariance matrix of $Y$ is given as

$$S=\mathrm{c}\mathrm{o}\mathrm{v}\left(Y\right)=\left[\begin{array}{cccc}{s}_{11}& s_{12}& \cdots & s_{1\left(N-1\right)}\\ s_{21}& s_{22}& \cdots & s_{2\left(N-1\right)}\\ ⋮& ⋮& \ddots & ⋮\\ s_{\left(N-1\right)1}& s_{\left(N-1\right)2}& \cdots & s_{\left(N-1\right)\left(N-1\right)}\end{array}\right]$$

(13)

where $cov\left(•\right)$ is the covariance operator. The unknown $N\times N$ covariance matrix of the individual noises $R$ ($R$ is a symmetric matrix) is related to $S$ by

$$S=J·R·J^T$$

(14)

where $J$ is the identity matrix and can be defined as

$$J_{N-1, N}=\left[\begin{array}{ccccc}1& 0& \cdots & 0& -1\\ 0& 1& \cdots & 0& -1\\ ⋮& ⋮& \ddots & ⋮& ⋮\\ 0& 0& 0& \cdots & -1\end{array}\right]$$

(15)

and matrix $R$ is defined as

$$R=\left[\begin{array}{cccc}{\sigma }_{11}& {\sigma }_{12}& \cdots & {\sigma }_{1N}\\ {\sigma }_{12}& {\sigma }_{22}& \cdots & {\sigma }_{2N}\\ ⋮& ⋮& \ddots & ⋮\\ {\sigma }_{1N}& {\sigma }_{2N}& \cdots & {\sigma }_{NN}\end{array}\right]$$

(16)

where ${\sigma }_{ij}=cov\left({\epsilon }_i,{\epsilon }_j\right), \left(i,j=1, 2,\cdots ,N\right)$. The question now is that $N\times \left(N+1\right)/2$ unknowns cannot be solved for only $N\times \left(N-1\right)/2$ equations. There remain N “free” parameters that must be reasonably determined to obtain a unique solution. To determine the N free parameters, an objective function is minimized based on the Kuhn–Tucker theorem [52], and it always fulfills the positive definiteness of $R$. The objective function F is given by

$$F_{\left({\sigma }_{1N},\cdots ,{\sigma }_{NN}\right)}={\displaystyle \frac{1}{K^2}}·\sum _{i<j}^N{\left({\sigma }_{ij}\right)}^2$$

(17)

with a constraint function H

$$H_{\left({\sigma }_{1N},\cdots ,{\sigma }_{NN}\right)}=-{\displaystyle \frac{\left|Q\right|}{\left|S\right|·K}}<0$$

(18)

where $K=\sqrt[N-1]{det\left(S\right)}$, $Q$ is a diagonal matrix with elements ${\sigma }_{11},{\sigma }_{22},\cdots ,{\sigma }_{NN}$ on its diagonal, which can be obtained by minimizing Equation (17) through initial condition iterations [19]. ${\sigma }_{11},{\sigma }_{22},\cdots ,{\sigma }_{NN}$ are the error covariances that equal to ${\sigma }_{i(i=1,\hspace{0.17em}2,\cdots ,N)}$, and their square roots represent the relative uncertainty of each SM product.

3.3. Machine/Deep Learning Algorithms

The details and parameters of RF and LSTM algorithms are as follows. Note that we randomly sampled 80% of the data to train the model, and the remaining data (20%) were used as a test dataset to verify the model performance [53]. A total of 80% of the data were mainly used to determine model parameters, i.e., grid search and cross-validation in the RF model and empirical parameters in the LSTM model. If the model performs comparably on both the training and test datasets, it indicates that this model is suitable for our dataset.

RF is a popular machine learning algorithm widely used in regression, in which a number of decision trees are first constructed during training phase, and then the mean prediction from these trees is generated as the prediction of the whole model [54]. To accurately estimate SM, a grid-search approach and 10-fold cross-validation were employed to automatically optimize and determine the key hyperparameters. The key hyperparameter values were as follows: the number of trees in the forest was 1000, the maximum features was 2, the minimum sample leaf was 1, the minimum samples split was 2, the mean squared error was used to measure the quality of a split, and bootstrap samples were used when building the trees.

LSTM is a special kind of recurrent neural network, and it can handle both long-term dependency problems and reduce the possibility of exponential explosion [55]. In this study, a two-layer LSTM structure was used, with a cell/hidden state length of 80 and 100. The output from the last LSTM layer at the last step was connected to a single output neuron through a traditional dense layer. Between the layers, a dropout technique (20% in our dataset) was added to prevent the model from overfitting [56]. Some other hyperparameters were experientially set as follows: the optimizer was Adam, the initial learning rate was 0.001, the loss function was ‘mean_squared_error’, the batch size was 64, and the maximum number of iterations was 50.

3.4. Evaluation Method

Taylor diagram and statistical metrics were adopted to provide quantitative analysis in this study. The metrics include coefficient of determination (R²), Pearson correlation coefficient (R), root mean square error (RMSE), unbiased RMSE (ubRMSE), mean absolute error (MAE), and Bias, with their equations shown in

Table 3. The statistical metrics used in this study.

Metric	Equation	Range	Best Value
R2	$R^2=1-{\displaystyle \frac{{\sum }_{i=1}^n{\left(Y_i-X_i\right)}^2}{{\sum }_{i=1}^n{\left(Y_i-\overline{Y}\right)}^2}}$	[0, 1]	1
R	$R={\displaystyle \frac{{\sum }_{i=1}^n\left(Y_i-\overline{Y}\right)\left(X_i-\overline{X}\right)}{\sqrt{{\sum }_{i=1}^n{\left(Y_i-\overline{Y}\right)}^2}\sqrt{{\sum }_{i=1}^n{\left(X_i-\overline{X}\right)}^2}}}$	[0, 1]	1
RMSE	$RMSE=\sqrt{{\displaystyle \frac{{\sum }_{i=1}^n{\left(Y_i-X_i\right)}^2}{n}}}$	$[\left.0,+\mathrm{\infty }\right)$	0
ubRMSE	$ubRMSE=\sqrt{{\displaystyle \frac{{\sum }_{i=1}^n{\left[\left(X_i-\overline{X}\right)-\left(Y_i-\overline{Y}\right)\right]}^2}{n}}}$	$[\left.0,+\mathrm{\infty }\right)$	0
MAE	$MAE={\displaystyle \frac{{\sum }_{i=1}^n\left|Y_i-X_i\right|}{n}}$	$[\left.0,+\mathrm{\infty }\right)$	0
Bias	$Bias={\displaystyle \frac{{\sum }_{i=1}^n{X}_i-{\sum }_{i=1}^n{Y}_i}{n}}$	($-\mathrm{\infty },+\mathrm{\infty }$)	0

Note(s): $X$ and $Y$ denote the estimated SM and the reference SM, respectively, and n is the sample size.

.

4. Results

4.1. TCH Uncertainty Assessment

Figure 3 shows the TCH-based uncertainty on four main land cover types in the PRB for seven SM products. Obviously, the uncertainty ranking results for the seven SM products were consistent across the four land cover types, which are (GLDAS and GLEAM) < ERA5-Land < SMAP < CCI-active < FY-3C < AMSR2.

The high uncertainty of the AMSR2 product could be attributed to its poor retrieval precision globally, as revealed in previous studies [39,57]. For the FY-3C product, the median SM uncertainty was higher in woodland and shrubland than in cropland and grassland. This implies that the FY-3C SM is significantly influenced by vegetation, consistent with existing studies [58]. For the CCI-active product, a higher median uncertainty was observed primarily in grassland and shrubland. The median uncertainty of the SMAP product was higher in woodland than in other land cover types. SMAP is known to underestimate surface SM over densely vegetated areas due to biased surface temperature and other potential factors [59,60]. For the ERA5-Land product, there was no obvious difference in the median values of uncertainty across the four land cover types. The uncertainty of GLDAS and GLEAM products showed the best performance over all four land cover types, with both medians below 0.023 m³/m³. Moreover, the small amplitude in box variation suggests their stable performances over different land cover types. Considering the poor performances of AMSR2 and FY-3C, only the other five SM products were used in subsequent BTCH merging.

4.2. BTCH Merging

Figure 4 displays the Taylor diagram comparing BTCH-based, AVE-based, and five-parent SM products with in situ observations. As shown in this figure, the BTCH-based SM estimate outperformed these five-parent SM products with a significantly closer distance to in situ observations. This suggests that the BTCH merging method can effectively integrate the desirable characteristics of the original parent products and reduce unwanted random retrieval errors. Comparatively, the BTCH-based SM estimate also showed slightly better performance than the AVE-based SM estimate, providing full proof of the significance of weight optimization. In contrast to arithmetic average, the BTCH merging method assigns relatively high weights to pixels with low uncertainty, while high uncertainty indicates low weights. The arithmetic average method is only a special case that assumes the weights of multiple datasets to be equal; however, it rarely does so in practical applications.

The relative weight (%) of each SM dataset has a key role in the BTCH merging method, as shown in Figure 5. In terms of weight values, the GLDAS product had the highest spatial-averaged weight (0.352), followed by GLEAM (0.284), then ERA5-Land (0.163), SMAP (0.139), and CCI-active (0.062). This is consistent with the uncertainty results in Figure 3, where products with low uncertainty are assigned high weight values and vice versa. Moreover, the merging weights varied among the parent products and showed large spatial variability. Specifically, the GLDAS SM product had relatively higher weights in the central and southeastern basin, while the GLEAM assigned high weights mainly in the northern and eastern basin. In contrast, the ERA5-Land and SMAP had high weights distributed in the southern and western basin, respectively. And the CCI-active displayed relatively lower weight over the entire basin. The regions with higher weight in each parent SM product mean that the merged SM over these parts will be more heavily weighted towards a particular parent SM product than the other products [61].

Furthermore, we evaluated the effect of SM uncertainty on the merged result, taking the 10 stations in the PRB on 1–5 April 2015 as an example. The SM uncertainty and the contribution of each SM product to the merged result were extracted at the station location, and their relevant relationship was examined, as shown in Figure 6. The contribution of each SM product to the merged result was quantified using ${\left\{{\displaystyle \frac{w_i{SM}_i}{{SM}_{merged}}}\times 100\%\right\}}_{i=1,\hspace{0.17em}2,\hspace{0.17em}3,\hspace{0.17em}4,\hspace{0.17em}5}$, where ${SM}_{merged}$ is the merged SM. As shown in Figure 6, the SM uncertainty was obviously negatively correlated with its contribution to the merged result, i.e., the larger the SM uncertainty, the smaller its contribution to the merged result.

4.3. Spatial Downscaling of the Merged SM

4.3.1. Selection of Antecedent Precipitation Days

There is a delayed effect of meteorological factors on SM [62]. For precipitation, the SM condition on a specific day is not solely related to precipitation on that day; it may have a stronger correlation with the sliding cumulative value of precipitation from previous days, or even tens of days. Figure 7 illustrates the relationship between the sliding cumulative values of precipitation (antecedent precipitation ranging from 5 to 90 days) and SM from 2015 to 2020 in the PRB. It was observed that the antecedent precipitations ranging from 5 to 90 days were all significantly and positively related to SM, with the R curve first increasing rapidly and then decreasing slowly. The highest R value (0.530) was found between the 35-day sliding cumulative precipitation and SM. Therefore, an optimal time scale for the cumulative effect of precipitation on SM was set at 35 days in this paper.

4.3.2. The Performance of Downscaling Models on the Test Dataset

Figure 8 shows scatterplots and statistical metrics of the RF-based and LSTM-based downscaling models on the 20% test dataset. In this paper, the RF and LSTM models without and with antecedent precipitation are referred to as RFpre, RFpre35, LSTMpre, and LSTMpre35, respectively. As indicated, the models with antecedent precipitation always outperformed the models without antecedent precipitation. That is, the scatterplot distributions for the RFpre35 and LSTMpre35 models (Figure 8b,d) were more concentrated than those for the RFpre and LSTMpre models (Figure 8a,c), also with large improvements in statistical metrics. For example, R² increased by 10% and 12%, RMSE decreased by 14% and 17%, and MAE decreased by 14% and 18%, respectively. In addition, both the RFpre35 and LSTMpre35 models showed high consistency against the test dataset, with the LSTMpre35 model performing slightly better than the RFpre35 model. This suggested that the LSTMpre35 model can better learn the complex relationship between environmental variables and the target SM variable at a 0.25° spatial scale.

4.3.3. Spatial Distribution of the Downscaled SM

Figure 9 shows the spatial distributions of the original (merged) and downscaled SM (RFpre, RFpre35, LSTMpre, and LSTMpre35) on 1–5 April 2015. Compared to the original SM product (0.25°), the downscaled results at 0.05° resolution showed a significant improvement in spatial detail. Visually, the RFpre35 and LSTMpre35 downscaled SM maps better preserved the spatial patterns observed in the original SM data than the RFpre and LSTMpre maps. High SM values were generally found in the northeastern basin, while low values were observed in the central-western and some parts of the eastern basin. This indicates that antecedent precipitation plays a key role in influencing SM spatial distribution. Moreover, it is apparent that the RFpre35 downscaled SM map shows a smoothing effect, leading to underestimation of high values and overestimation of low values. In contrast, the LSTMpre35 downscaled SM map effectively reproduces the dynamic range of the original SM product, representing a large progress over other machine learning downscaling algorithms [30]. Thus, the LSTMpre35 downscaled result achieved the highest spatial consistency with the original SM product in the PRB.

4.4. Validations of the Downscaled SM

4.4.1. Comparison with Ground Observations

Table 4. Validation results of the original and downscaled SM against in situ observations. N represents the sample size.

		56,697	56,785	56,788	56,875	56,880	56,881	56,883	56,889	56,985	59,205	Ave
Original (0.25°)	R	0.828	0.882	0.639	0.492	0.681	0.537	0.840	0.744	0.669	0.802	0.711
	Bias	0.028	0.008	0.037	−0.029	0.114	0.106	−0.029	0.035	0.044	0.030	0.034
	RMSE	0.064	0.033	0.076	0.060	0.122	0.118	0.042	0.079	0.059	0.036	0.069
	MAE	0.048	0.026	0.058	0.049	0.114	0.107	0.034	0.062	0.049	0.031	0.058
	ubRMSE	0.058	0.032	0.066	0.053	0.044	0.053	0.031	0.070	0.039	0.019	0.047
N		384	384	384	383	383	384	371	382	383	384
RFpre	R	0.653	0.688	0.425	0.383	0.603	0.466	0.693	0.601	0.452	0.510	0.547
	Bias	−0.006	−0.006	0.014	−0.054	0.115	0.092	−0.041	0.026	0.048	0.053	0.024
	RMSE	0.072	0.032	0.079	0.064	0.120	0.100	0.056	0.087	0.062	0.060	0.073
	MAE	0.058	0.026	0.065	0.056	0.115	0.092	0.046	0.073	0.055	0.053	0.064
	ubRMSE	0.072	0.031	0.077	0.033	0.035	0.038	0.038	0.084	0.039	0.026	0.047
RF pre35	R	0.707	0.832	0.551	0.482	0.628	0.500	0.785	0.689	0.580	0.648	0.640
	Bias	0.001	0.007	0.027	−0.034	0.117	0.097	−0.031	0.030	0.047	0.055	0.032
	RMSE	0.067	0.027	0.076	0.051	0.124	0.106	0.045	0.084	0.059	0.060	0.070
	MAE	0.053	0.022	0.061	0.042	0.117	0.097	0.037	0.067	0.053	0.055	0.060
	ubRMSE	0.067	0.026	0.071	0.038	0.039	0.042	0.033	0.078	0.036	0.023	0.045
LSTM pre	R	0.633	0.604	0.328	0.336	0.610	0.443	0.684	0.578	0.466	0.555	0.524
	Bias	−0.008	−0.017	0.009	−0.074	0.097	0.085	−0.054	0.012	0.001	0.039	0.009
	RMSE	0.073	0.042	0.083	0.084	0.105	0.096	0.066	0.081	0.044	0.047	0.072
	MAE	0.059	0.033	0.069	0.076	0.097	0.085	0.056	0.068	0.033	0.041	0.062
	ubRMSE	0.072	0.039	0.082	0.041	0.041	0.045	0.039	0.081	0.044	0.026	0.051
LSTM pre35	R	0.732	0.786	0.514	0.468	0.627	0.518	0.786	0.696	0.633	0.646	0.641
	Bias	−0.005	−0.006	0.009	−0.042	0.119	0.096	−0.025	0.032	0.028	0.038	0.024
	RMSE	0.067	0.034	0.075	0.062	0.127	0.106	0.041	0.081	0.046	0.045	0.068
	MAE	0.053	0.027	0.062	0.049	0.119	0.096	0.033	0.064	0.039	0.039	0.058
	ubRMSE	0.067	0.034	0.074	0.046	0.044	0.046	0.033	0.074	0.037	0.023	0.048
N		391	391	391	390	390	391	378	389	390	391

presents the statistical metrics of the original and downscaled SM against in situ observations. Compared to the original SM product, the downscaled results did not achieve favorable performance, with no significant improvement in the error statistics except for the average bias value. As shown, the original SM product had an average bias value of 0.034 m³/m³, while the downscaled SM had average bias values of 0.009~0.032 m³/m³.

Generally, the RFpre35 and LSTMpre35 downscaled results outperformed their respective counterparts, RFpre and LSTMpre. The R values at each station were higher, and the RMSE, MAE, and ubRMSE values were also smaller at most stations, with the same situation for the average metric values. In addition, the overestimation (average bias) of the LSTMpre and LSTMpre35 downscaling results was lower than that of the RFpre and RFpre35 results, respectively. Despite comparable accuracy performance, the LSTMpre35 downscaled result slightly outperformed the RFpre35 result, except for the average ubRMSE value with a difference of 0.002 m³/m³.

4.4.2. Comparison with CLDAS SM Data

The CLDAS SM data were employed as a reference to more fully evaluate the accuracy of the original and downscaled SM, as shown in Figure 10. To match the CLDAS SM data, the spatial resolution of both original and downscaled SM was resampled to 0.0625°. Considering the available CLDAS SM data, the time range for this section covers the period from 2017 to 2019.

Compared to the original SM product, the downscaled results did not show improved performance; however, they slightly reduced the overestimation. The original SM product had a median bias of 0.020 m³/m³, while the downscaled results had median biases of 0.017~0.019 m³/m³. Clearly, the downscaled SM results of RFpre35 and LSTMpre35 outperformed those of RFpre and LSTMpre, respectively. Moreover, the LSTMpre35 downscaled SM result (R = 0.719, Bias = 0.019 m³/m³, RMSE = 0.039 m³/m³, MAE = 0.030 m³/m³ and ubRMSE = 0.031 m³/m³) was superior to the RFpre35 result (R = 0.697, Bias = 0.017 m³/m³, RMSE = 0.040 m³/m³, MAE = 0.030 m³/m³ and ubRMSE = 0.032 m³/m³) in terms of the median metrics, although the difference was small. The above validation results were totally consistent with those against in situ observations.

4.5. Feature Importance Assessment

Figure 11 shows the importance of each explanatory feature from the RFpre35 downscaling model. It was found that pre35 predominated SM retrieval with the highest relative importance (30.01%), suggesting that antecedent precipitation played a significant role in reducing errors during model training. As a key variable directly causing SM changes, contemporary precipitation (pre) also showed high importance (15.76%), second only to pre35. The significance of pre35 and pre highlights that SM estimation is mainly influenced by precipitation, with antecedent precipitation having a greater impact than contemporary precipitation.

LST was also identified as a critical variable (10.62%) due to the controlling effect of surface SM on surface energy exchange and partitioning [2]. Vegetation index and albedo are commonly used to describe SM variations in triangular/trapezoidal methods because of their ability to reflect vegetation status and surface energy fluxes [63]. In this study, EVI was used to avoid the problem of vegetation saturation. As expected, EVI and real surface albedo showed relatively high importance with 10.38% and 8.45%, respectively. Among three topographic factors, elevation was found to be the most important (7.58%), followed by slope (5.99%) and aspect (4.88%). This indicates a greater influence of elevation on SM over regions with large height differences (about 3000 m in the PRB). In addition, soil texture presented 1–3% importance for sand, silt, and clay, respectively, due to the strong influence of these properties on water infiltration rates, permeability, and soil water storage capacity [8]. It is noted that a lower ranking does not necessarily mean that a feature is unimportant, as the SM estimation of the downscaling model is based on a combined judgement of all input variables.

5. Discussion

5.1. Advantages and Limitations in the BTCH Method

Although the TCH-based uncertainty may underestimate the absolute uncertainty as compared to gauge-based validation, the relative uncertainty among individual products can be well retained [17]. Based on Figure 3 and Figure 4, five SM products were merged using the BTCH method, and the BTCH-based SM estimate demonstrated superior performance to the five parent products and the AVE-based SM estimate. The advantages of the BTCH merging in uncertainty reduction and weight optimization were also reported in previous studies regarding other geographic elements, such as gross primary productivity [18] and terrestrial evapotranspiration [19].

It is known that SM contains spatio-temporal non-stationary random errors that may be stationary in one dimension but non-stationary in another [64]. The TCH method assumed that the errors of SM data were static and time-invariant, failing to fully characterize the error information of SM, especially in time domain [9]. There are some studies that consider the spatio-temporal non-stationary errors in TC-based merging [64,65,66]; however, there are fewer in TCH-based merging. It would make sense to fill and enrich this gap in further studies.

5.2. Importance of Antecedent Precipitation

The choice of the length of the antecedent precipitation depends on study area characteristics. The period of antecedent precipitation deemed significant in previous studies varies considerably, ranging from 10 to 90 days [32,33,67,68]. For example, Schoener et al. [67] proposed a new approach for estimating catchment-scale SM using an antecedent precipitation index (API)-based model, in which an antecedent period of 13 days was determined through a sensitivity analysis. Liu et al. [33] developed a linear regression model that links antecedent effective precipitation and SM, using a 90-day antecedent period. The model achieved an accuracy rate of over 93% when applied to monitor autumn and winter drought in certain areas of Hubei in 2007. These studies showed that considering antecedent precipitation on an appropriate timescale is essential for accurate estimation and effective application of SM.

In this paper, the optimal time scale for the cumulative effect of precipitation on SM was set at 35 days. The comparison of downscaling models (Figure 8, Figure 9 and Figure 10 and Table 4) revealed that the models with antecedent precipitation obtained significantly better performance, demonstrating their strong stabilities in estimating SM at both temporal and spatial scales. This result was reasonably explained by the feature importance ranking in Figure 11, which showed the highest contribution (30.01%) of antecedent precipitation in the study region.

Moreover, this study determined the optimal time scale (35 days) for the cumulative effect of precipitation on SM in the entire PRB throughout the whole study period. However, the effect of precipitation on SM is highly variable both in temporal and spatial domains. In terms of temporal scale, other temporal dynamics that may affect SM (e.g., seasonal variability and wet/dry seasons, or lag effects beyond the 35-day window) should also be explored in future research. In terms of spatial scale, different areas should be delineated to better characterize the relationship between precipitation and SM, especially in a large area encompassing several climatic zones and different topographies.

5.3. Uncertainty in the Validation of Downscaled SM

According to the validation results in Table 4 and Figure 10, the downscaled SM did not show better performances than the original SM product. Other studies have reported similar decreases in accuracy after downscaling [26,28], even when the downscaled SM was compared with sufficient ground measurements. It is challenging for the regressed SM (the downscaled SM at 0.05°) to transcend the original SM product (the merged SM at 0.25°) in data accuracy because the original SM data (80% of the data) were employed as a training sample for model building [28,69,70]. It should be noted that the predictive range of SM is restricted to those covered by the training data. Thus, the capability of downscaling models to reproduce SM at a finer scale is closely related to the moisture contrasts in the training data. As the bottleneck factor of precision restriction, the quality of the original SM product needed to be improved, such as the BTCH merging used in this study. There are also other ways to improve the quality of the original merged SM product, e.g., improving the quality of SM products involved in the merging process, or using alternative merging methods that take into account the inherent error of SM products. For satellite SM products, the improvement of retrieval algorithms and sufficient in situ measurement-based correction would be effective ways. Model-based SM products could be further refined by improved quality of meteorological forcing data and the soil property database. Moreover, other merging methods that considered the inherent error of SM products, such as Bayesian model averaging [71], data assimilation based on Kalman filtering [72], and optimal interpolation [73], should be applied to compare with the BTCH approach in future studies.

Moreover, the statistical metrics calculated in Table 4 are inevitably influenced by many factors, such as the limited number of in situ stations; the depth difference between the merged and downscaled SM (0–5 cm) and in situ SM (10 cm); and the spatial scale difference between the merged SM (0.25°), in situ SM (point scale), and the downscaled SM (0.05°). There was always unreasonable spatial matching when verifying remote sensing or model-based SM products with ground measurements. To address this issue, alternative methods were required, such as the CLDAS SM data used in this study or other SM-related physical elements that may be employed in future studies. Moreover, more robust methods, e.g., geostatistical approaches, regional-scale validation [57], and higher-density ground networks that are spatially representative, could also mitigate these issues. For example, Wang et al. [74] proposed a model-based geostatistical approach to upscale ground-based SM observations of unequal precision, and they successfully verified this approach in the Heihe Watershed Allied Telemetry Experimental Research experiment.

5.4. Limitations and Future Directions of Machine/Deep Learning Model

Machine/deep-learning-based models with transfer learning capabilities should be developed in the future for SM estimation. Note that transferability without additional training over a new location’s data has remained a significant challenge for all machine/deep learning models. This would involve applying the model to a diverse range of environments, from arid to tropical regions, and from flat plains to mountainous terrains. Yang et al. [75] proposed a Climate-Adaptive Transfer Learning (CATL) framework to improve SM estimation for a data scarce region, the Qinghai–Tibet Plateau (QTP). Specifically, regarding the QTP as the target region, selecting the areas with similar climate types with QTP as the source region, they train the machine/deep learning model in the source region and then transfer it to the target region via fine-tuning strategy. Results indicate that the CATL framework significantly improved the SM estimation in the QTP, achieving better precision.

Machine/deep-learning-based models with physical interpretation should be developed in the future for SM estimation. While machine/deep-learning-based models have shown superior efficiency and accuracy in our study and others, they are often criticized due to problems of overfitting and lack of physical understanding and explainability. Recently, Singh and Gaurav [76] proposed a learning bias physics-informed machine learning (PIML) model to estimate surface SM, incorporating the physics in the loss function of a fully connected feed-forward neural network (FFNN). Similarly, Chavoshi et al. [77] proposed a physics-informed neural networks (PINN) model that estimates the vadose zone soil water content (VZSWC) profile by incorporating Richardson’s equation as its primary physical constraint, and it achieved satisfactory performances. Both PIML-SM and PINN-SM models demonstrated that models that can effectively bridge the gap between data-driven models, and physical understanding will be the future development trend in environmental modelling applications.

6. Conclusions

This paper developed a two-step SM reconstruction approach, combining the BTCH merging and machine/deep-learning downscaling algorithms to generate a high-quality and high-spatial-resolution (0.05°) pentad SM dataset from microwave and model-based SM products. The reconstruction approach was tested over the PRB, and the performance of the obtained merged and downscaled SM results was fully assessed, with the main conclusions as below:

(1)

During 2015–2020, the TCH-based uncertainty ranking of seven SM products on four main land cover types in the PRB was consistent: (GLDAS and GLEAM) < ERA5-Land < SMAP < CCI-active < FY-3C < AMSR2. Moreover, the BTCH-based SM estimate outperformed five parent products and the AVE-based SM estimate, indicating that BTCH is a fusion approach that can effectively reduce data uncertainties and optimize weights.

(2)

The optimal time scale for the cumulative effect of precipitation on SM is 35 days for the period from 2015 to 2020 in the PRB. Validation on the 20% test dataset, visual comparison on spatial distributions, and validation against in situ observations and the CLDAS SM data all showed that adding an antecedent precipitation variable as a predictor can greatly improve the performance of SM downscaling models, both at the 0.25° and 0.05° spatial scales. The feature importance assessment also revealed that precipitation is a key variable to our SM downscaling models, with a greater influence of antecedent precipitation (30.01%) than contemporary precipitation (15.76%). Moreover, the LSTMpre35 model performed slightly better than the RFpre35 model.

(3)

Validation against in situ observations and the CLDAS SM data also indicated that the downscaled SM results were mainly limited by the original SM data in terms of data accuracy and were difficult to surpass. However, they did alleviate the overestimation inherent in the original SM data.

In conclusion, the two-step SM reconstruction approach has great potential in generating a high-quality and high-spatial-resolution SM dataset and could act as a reference for SM studies in other areas or for other earth science variables. Future works will investigate the potential of the obtained SM dataset in drought monitoring or other hydrological applications.