Evaluating Satellite Soil Moisture Datasets for Drought Monitoring in Australia and the South-West Pacific

Abstract

Soil moisture (SM) is critical in monitoring the time-lagged impacts of agrometeorological drought. In Australia and several south-west Pacific Small Island Developing States (SIDS), there are a limited number of in situ SM stations that can adequately assess soil-water availability in a near-real-time context. Satellite SM datasets provide a viable alternative for SM monitoring and agrometeorological drought provision in these regions. In this study, we investigated the performance of Soil Moisture Active Passive (SMAP), Soil Moisture and Ocean Salinity (SMOS), Soil Moisture Operational Products System (SMOPS), SM from the Advanced Microwave Scanning Radiometer 2 (AMSR-2) and SM from the Advanced Scatterometer (ASCAT) over Australia and south-west Pacific SIDS. Products were first evaluated in Australia, given the presence of several in-situ SM monitoring stations and a state-of-the-art hydrological model—the Australian Water Resources Assessment Landscape modelling system (AWRA-L). We further investigated the accuracy of SM satellite datasets in Australia and the south-west Pacific through Triple Collocation analysis with two other SM reference datasets—ERA5 reanalysis SM data and model data from the Global Land Data Assimilation System (GLDAS) dataset. All datasets have differing observation periods ranging from 1911-now, with a common period of observations between 2015–2021. Results demonstrated that ASCAT and SMOS were consistently superior in their performance. Analysis in the six south-west Pacific SIDS indicated reduced performance for all products, with ASCAT and SMOS still performing better than others for most SIDS with median R values ranging between 0.3–0.9. We conducted a case study of the 2015 El Niño and Positive Indian Ocean Dipole-induced drought in Papua New Guinea. It was shown that ASCAT is a valuable dataset indicative of agrometeorological drought for the nation, highlighting the value of using satellite SM products to provide early warning of drought in data-sparse regions in the south-west Pacific.

1. Introduction

Soil moisture (SM) is a key state variable that parametrises several hydrological processes such as infiltration, evaporation, and runoff. SM provides insight into the water availability of terrestrial ecosystems and modulates the primary productivity of several agricultural processes [1].

Information on the spatiotemporal variability of SM is of critical importance for monitoring meteorological, climatic, and hydrologic extremes such as drought. Defining a multi-faceted and slow-onset hazard like drought is a complex and sometimes arbitrary endeavour. For this reason, drought definitions tend to be categorised by propagation and impact, where an initial rainfall deficiency (meteorological drought) may be accompanied by time-lagged abiotic stress (agricultural drought), water supply stress (hydrological drought), and reduced financial productivity for drought-vulnerable communities (socioeconomic drought). SM is a key antecedent drought variable since it provides insight into surface and root zone water availability that can be used to track, monitor, and understand agricultural drought impacts before, during and after a drought event.

SM is monitored through in-situ observations or remote sensing retrieval and modelled using water balance land models. Generally, in-situ measurements of SM are considered the most reliable account of soil water profile [2,3]. Common SM instruments include neutron probes, electromagnetic sensors, heat pulse probes, and gamma radiation sensors [2]. However, given that such sensors are relatively discrete and point-based, they have limited ability to represent catchment-scale SM dynamics [4]. SM instruments can also be susceptible to errors from elevated salinity; reduced soil-sensor contact from soil shrinkage and ecological activity; limited performance at shallow depths; temperature extremes impacting performance; and limited soil-specific calibration [2]. Additionally, implementation of a high-density SM sensor network is a complex and financially infeasible process that is often of low priority in regions that are impacted by other hazards [5].

In gauge-sparse regions, like the south-west Pacific, where in-situ SM monitoring infrastructure may be limited in its availability, remote sensing of SM offers a promising solution. Microwave remote sensing is the prevailing technique deployed on various satellite missions owing to the longer than optical wavelengths of microwave radiation (1 mm–1 m wavelengths). This substantially eliminates the effects of atmospheric scattering from clouds, haze, dust, and rainfall that typically impact the accuracy of optical remote sensing methods [2,3]. Microwave remote sensing of SM can be conducted using passive or active sensors (or some combination of both) [2]. Active sensors (radars) emit pulses of microwave radiation and measure the backscattered fraction returned to the sensor. On the other hand, passive sensors (radiometers) do not actively emit microwave radiation. Instead, they measure the natural microwave emission of an emitting body through its brightness temperature [2,4]. Both passive and active retrieval methods rely on the naturally occurring contrast between the dielectric constants of dry and wet soil to estimate soil water capacity where wetter soils emit cooler brightness temperatures (passive) and reflect higher backscatter values (active) [6]. Passive remote sensing is more sensitive to surface SM and is commonly used to map global SM in X, C, and L band frequencies [3,7]. Passive SM satellite sensors require large antennas to allow for adequate detection of weak thermal emissions, which can result in coarser resolutions [6]. Active sensors, on the other hand, may have finer spatial resolutions but can be impacted by surface backscattering from vegetation and surface roughness. Both retrieval methods have an extensive array of advantages and disadvantages, which is why some SM space-missions deploy hybrid combinations that utilise attributes of both. This makes SM retrieval and algorithm generation an evolving area of SM research [2,3].

In the south-west Pacific region, monitoring soil moisture (and even rainfall) through a dense and uniform station network is a challenging task [8,9], owing in most part to the region’s remoteness and limited financial and infrastructural capacity for meteorological monitoring [10]. Additionally, given geographic distributions of orography, atolls, and low-lying islands, several regions in the Pacific are often vast distances away from capital islands where primary rain and soil moisture gauges are commonly placed [11]. Limited monitoring and forecasting abilities exacerbate the region’s vulnerability and exposure to drought, making it difficult to monitor and warn communities most at risk in the period leading up to drought. This means that historically, the region has experienced widespread devastation and loss of livelihood from drought episodes, with the notable El Niño periods of 1997–1998 and 2015–2016 leading to extensive crop losses, famine, and socioeconomic damage [11,12]. The El Niño-Southern Oscillation (ENSO) has varied drought impacts throughout the region as complex oceanic and atmospheric dynamics combined with other climate drivers that modulate drought [13]. For example, in Papua New Guinea (PNG), drought risk can be high for certain regions during both El Niño and La Niña periods. Regardless of the active ENSO phase influencing drought on the seasonal scale, impacts are expected to intensify on the multidecadal scale as anthropogenic climate change accelerates. The Intergovernmental Panel on Climate Change’s sixth assessment report notes Pacific Islands polewards of ±20° latitude have experienced a drying trend that is predicted to continue in the coming decades [14].

To aid, prepare and warn communities of drought, the operational implementation of satellite monitoring rainfall products is becoming more commonplace in the Pacific region as remote-sensing research progresses [15,16,17,18]. Yet, satellite SM evaluation is a relatively under-explored research space, owing to the minimal-to-none in situ SM stations or water balance models that can be used to evaluate satellite SM products in the Pacific region.

The objective of this study is to examine the performance of satellite SM products in Australia and the south-west Pacific region. Australia is chosen as an initial SM study area due to its range of climate zones; topographies; soil types; well-maintained SM station networks, and the availability of a state-of-the-art gridded water balance model used for operational SM provision by the Australian Bureau of Meteorology (BoM). The in-situ and gridded analysis is then accompanied by a Triple Collocation (TC) analysis over the South-West Pacific. All datasets have a common study period of 2015–2021. The TC analysis is a useful extension of the in-situ and gridded analysis that has not been performed for SM evaluation in this region before.

2. Data and Methods

In this study, we investigated the performance of Soil Moisture Active Passive (SMAP), Soil Moisture and Ocean Salinity (SMOS), Soil Moisture Operational Products System (SMOPS), SM from the Advanced Microwave Scanning Radiometer 2 (AMSR-2) and SM from the Advanced Scatterometer (ASCAT) over Australia and the south-west Pacific using a suite of statistical methods. A summary of dataset resolution, temporal coverage, latency, measurement unit, and sampling depth is provided in

Table 1. Summary of the datasets used in this study. Bold text indicates the resolution and sampling depths used in this study.

Dataset [Reference]		Sensor/Model/Probe Type (Frequency, GHz)	Sponsor	Sampling Depth (cm)	Resolution/ Nstations	Temporal Coverage	Unit
Satellite	ASCAT [19]	Active, C band (5.3)	EUMETSAT	0.5–2	12.5/25/50 km	2007-now	% saturation
	SMOS [20]	Passive, L band (1.4)	ESA	0–5	15/25 km	2010-now	m3/m3
	SMOPS [21]	merged product	NASA	1–5	0.25 deg	2015-now	m3/m3
	ASMR-2 [22]	Passive, C (6.9/7.3) X (10.7)	NASA/JAXA	0–0.5	0.1/0.25 deg	2012-now	m3/m3
	SMAP [23]	Active/Passive, L band (1.26/1.41)	NASA	0–5	3/9/36 km	2015-now	m3/m3
Modelled	AWRA-L [24]	Open loop (no data assimilation)	BoM	0–10	0.05 deg	1911-now	m3/m3
	GLDAS [25]	Global Water Balance Model	NASA	0–10	0.25 deg	1948–2020	kg/m2
	ERA-5 Land [26]	ECMWF Reanalysis	ECMWF/Copernicus	0–7	0.1 deg	1950-now	m3/m3
In-Situ	OzNet [27]	Reflectometers/Hydraprobes	Monash/Melbourne	0–5, 0–8, 0–10, 0–30, 30–60, 60–90	62	2001-now	m3/m3
	CosmOz [28]	Cosmic Neutron Ray	CSIRO/TERN		14	2010-now	m3/m3
	OzFlux [29]	Reflectometers/Hydraprobes	Monash		23	2002-now	m3/m3

(BoM = Bureau of Meteorology, CSIRO = Commonwealth Scientific and Industrial Research Organisation, ECMWF = European Centre for Medium-Range Weather Forecasts, ESA = European Space Agency, EUMETSAT = European Organization for the Exploitation of Meteorological Satellites, NASA = National Aeronautics and Space Administration, TERN = Terrestrial Ecosystem Research Network).

and are discussed in detail in the following sections.

First, we validated satellite SM products against in-situ, point-based soil moisture data. Then a gridded analysis was performed—using monthly SM grids from a water balance model. Both the gridded and point-based analyses were performed over Australia. To expand our SM analysis to the south-west Pacific region, a TC analysis was conducted using global water balance models. The best-performing dataset from the TC analysis is then applied to a case study analysing the impacts of an extreme drought period in Papua New Guinea. For our analysis, all satellite, model, and in-situ data were either obtained in monthly pre-processed formats or were processed into monthly grids and values. This was done not only because drought impacts tend to occur on a multi-month timescale but also due to satellite swathes and sensor readings having sporadic daily data availability depending on passes, calibration algorithms, and SM instruments used [30]. The common period between each station, AWRA-L and TC triplets was used, 2015–2021. Additionally, all gridded datasets were resampled to a resolution of 0.25° using bilinear interpolation as a trade-off between higher resolution land-surface model data (0.05°) and some of the coarser resolution satellite data (0.10–0.5°).

2.1. Satellite Datasets

2.1.1. ASCAT

The Advanced Scatterometer (ASCAT) is a C-band radar onboard the MetOp-A and MetOp-B satellites. The ASCAT algorithm derives surface SM through a linear change detection retrieval algorithm developed by Wagner et al. (1999) [19]. ASCAT has a revisit frequency of 1–3 days with 9:30/21:30 LST overpass times. The algorithm detects SM in the degree of saturation (%) which we convert into volumetric SM (m³/m³) by multiplying ASCAT degree of saturation SM with soil porosity data obtained from the Harmonized World Soil Database (included within the ASCAT datasets). High resolution and monthly processed ASCAT SM was obtained from the Integrated Climate Data Centre (ICDC) [31], where we used descending overpasses (in the morning, i.e., AM) similar to previous studies that cite plant and soil temperatures, vertical profiles, and soil dielectric properties to be more consistent for AM overpasses [32,33].

2.1.2. SMOPS

The Soil Moisture Operational Products System (SMOPS) is a merged satellite SM product developed by the National Oceanic and Atmospheric Administration (NOAA) [21]. SMOPS was designed to increase the spatial and temporal coverage of existing satellite SM datasets [34] and has been an operational product at NOAA’s National Environmental Satellite Data and Information Service since 2013. SMOPS ingests SM data from ASCAT, SMOS, AMSR-2, and SMAP datasets and generates daily and 6-hourly global surface SM grids via the algorithm detailed in Zhan et al. (2016) [34]. For our study, we use monthly SMOPS grids at a 0.25° resolution provided by the World Meteorological Organization’s Space-based Weather and Climate Extremes Monitoring (SWCEM) [8].

2.1.3. SMOS

The Soil Moisture Ocean Salinity (SMOS) satellite mission was launched in November 2009 by the European Space Agency (ESA) with the goal of surveying surface SM and sea surface salinity globally. SMOS carries an L-band radiometer with 69 uniformly spaced dual polarization antennas that detect brightness temperatures emitted from Earth’s surface with an average spatial resolution of 43 km and a revisit time of 2–3 days [20]. The brightness temperatures are related to surface SM via a retrieval algorithm detailed by Kerr et al. (2010) [20]. Monthly processed data was obtained from the Barcelona Expert Centre (BEC), where raw point-based ESA SMOS L2 data is gridded to 0.25°, filtered, and binned through the algorithm described by Pablos et al. (2020) [35]. For consistency, we continue to use descending overpasses (AM).

2.1.4. AMSR-2

The Advanced Microwave Scanning Radiometer 2 (AMSR-2) is onboard the Japan Aerospace Exploration Agency’s (JAXA) GCOM-W satellite. AMSR-2 is the successor of AMSR-E, which retired in October 2011 and was onboard the National Aeronautics and Space Administration’s (NASA) Earth Observing System Aqua satellite. AMSR-2 detects emissions at several frequencies; for SM content specifically, AMSR-2 measures brightness temperatures in the C and X bands with a revisit time of 1–2 days [22,36]. Several studies have found that C-band SM retrievals have the potential to suffer from radio frequency interference which is why AMSR-2 is also equipped with X-band sensors [2,36]. For this study, we use C- and X-band L3 AMSR2 (LPRM algorithm) descending daily passes sourced from NASA’s Earth Data Center, which is processed into monthly grids [22]. To process monthly grids from daily swath passes, we use an approach similar to Wang et al. (2018) [37], where the monthly averaging conversion is only applied to grid cells with valid data of more than 15 days in a given month.

2.1.5. SMAP

The Soil Moisture Active Passive (SMAP) satellite used combined measurements from an L-band radar and an L-band radiometer to retrieve surface SM in the top 5 cm of soil. SMAP was launched in January 2015. However, a hardware mishap in July of the same year rendered the satellite’s radar instrument unworkable. SMAP’s working L-band radiometer provides observations with a revisit time of 2–3 days and a latency of 12–24 h [38]. In this study, we used dual-channel level 3 descending daily composite data sourced from NASA’s Earth Data Center and processed the swathes into monthly grids similarly to AMSR-2, described earlier [23]. We also used the 36 km resolution as 9 km datasets were too computationally demanding to process. Given that all our analysis uses a common resolution of 0.25°, we do not deem this to be a limiting factor in our evaluation.

2.2. Validation Metrics

Satellite products are commonly evaluated in their performance to some ‘truth’ dataset using conventional metrics such as Mean Bias Error (MBE), Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), unbiased RMSE (ubRMSE), and Pearson correlation coefficient (R) [2,4,6]. A summary of the equations used in this study are provided in

Table 2. Summary of the validation metrics used in the point-based and gridded SM evaluations.

Metric and Analytical Purpose	Equation	Range	Perfect Value	Unit
MBE Mean systematic difference between estimated SM and ‘true’ SM.	$\frac{1}{N}{\displaystyle {\displaystyle \sum }_{i=1}^N}\left(E_i-T_i\right)$	(−∞, ∞)	0	m3/m3
MAE Mean absolute difference between estimated SM and ‘true’ SM.	$\frac{1}{N}{\displaystyle {\displaystyle \sum }_{i=1}^N}\left|E_i-T_i\right|$	[0, ∞)	0	m3/m3
RMSE Spread or standard deviation of errors for estimated SM.	$\sqrt{\frac{1}{N}{\displaystyle {\displaystyle \sum }_{i=1}^N}{\left(E_i-T_i\right)}^2}$	[0, ∞)	0	m3/m3
ubRMSE Spread or standard deviation of random error for estimated SM after systematic differences are removed.	$\sqrt{RMS{E}^2-MB{E}^2}$	[0, ∞)	0	m3/m3
R A statistical metric that measures the strength and direction of correlation between estimated SM and ‘true’ SM.	$\frac{{\sum }_{i=1}^N\left[\left(E_i-\overline{E}\right)\left(T_i-\overline{T}\right)\right]}{{\sigma }_{E_i} {\sigma }_{T_i}}, {\sigma }_{X_i}=\sqrt{{\displaystyle {\displaystyle \sum }_{i=1}^N}{\left(X_i-\overline{X}\right)}^2}$	[−1, 1]	1	unitless

.

Here E_i is the estimated SM value at grid cell i, and T_i is the ‘true’ SM value at grid cell i.These metrics are used to evaluate satellite SM products in our point-based (O_i is station data) and gridded analysis (O_i is gridded water balance model data). For our TC analysis, a range of TC-specific mathematical metrics is used. We note that MBE, MAE, and RMSE are similar in exploring error characteristics. We retain their inclusion similar to other holistic satellite SM studies [32,38] that use MBE, MAE, and RMSE for bias and error spread insight and ubRMSE and correlations for insight into the systematic difference between datum pairs.

2.3. Point-Based Analysis Datasets

For the point-based analysis, in-situ SM data from three separate SM networks was processed. These datasets, OzFlux, OzNet, and CosmOz, are comprised of a combination of probes, reflectometers, and sensors with differing time periods (all ranging somewhere between 2001–the present) and with different spatial coverage throughout Australia. The distribution of these SM networks across different Australian climate zones is depicted in Figure 1. A detailed summary of each station’s metadata, including latitude, longitude, elevation, and sensor type, is provided in Appendix A. In cases with more than one station within a grid cell, the same grid cell is verified twice against the relevant station/s within it. Looking at Figure 1, this would only be the case for some OzNet stations.

SM is measured differently depending on the type of instrument and between networks [2]. This introduces certain systematic variations and biases between networks that Holgate et al. [40] note can be a point of interest when comparing to satellite datasets that also have systematic biases in their detection algorithms. Additionally, depth mismatches between different satellite datasets and instruments introduce other uncertainties into our analysis—pairing this in situ analysis with the varying SM depths of the gridded and TC analysis will help us scrutinize the extent to which depth mismatches can influence performance. We consulted network metadata to only include stations with active and calibrated, high-quality data. Raw station data ranged from hourly, tri-hourly, and daily values which were resampled into monthly averages to match satellite and water balance data temporal resolutions.

2.3.1. OzNet

The OzNet hydrological monitoring network is an SM sensor network located in the Murrumbidgee River Catchment in southern New South Wales [27]. OzNet data measures SM in several soil layers: 0–5, 0–8, 0–30, 30–60, and 60–90 cm using reflectometers and hydraprobes. We used 0–5/0–8 cm measurements to correspond with the soil surface measurements made by satellite products. Raw OzNet data is clustered in seasonal Excel workbooks that were cleaned and compiled into monthly data frames using the Pandas library in Python. In our study, we use SM data from sixty-two OzNet station sites (all clustered around the Murrumbidgee Catchment) with data ranging from 2001 to the present.

2.3.2. CosmOz

CosmOz is an Australian SM probe network that provides SM estimates using Cosmic Ray Probes (CRPs). These probes measure fast neutrons produced by cosmic rays (high-energy atomic nuclei produced from different astrophysical processes that travel close to the speed of light) moving through Earth’s atmosphere. Fast neutrons interact strongly with hydrogen atoms (mostly contained in water molecules) [2]. Thus, the intensity of scattered fast neutrons is inversely related to the abundance of hydrogen atoms in a certain region [28]. Unlike other SM reflectometers and sensors that measure volumetric SM in small quantities of soil, CRPs are recognized for their ability to deduce average SM on larger spatial scales up to 30 hectares [28]. In our study, we use Level 4 SM data from fourteen CosmOz station sites across Australia that have calibrated and real-time data available, all ranging from 2010–the present.

2.3.3. OzFlux

OzFlux is an SM flux tower network that is part of the international SM network, FluxNet [29]. Similar to OzNet data, SM is measured using a range of reflectometers and hydraprobes. For our study, we use Level 6 OzFlux grids for twenty-three sites available from the TERN/OzFlux THREDDS Data server. These pre-processed monthly NetCDF grids provide data for a range of terrestrial and climate variables, with data ranging between 2002–present.

2.4. Gridded Analysis—AWRA-L

The Australian Water Resource Assessment (AWRA-L) is the BoM’s operational gridded SM product. AWRA-L is a model that evolves the physical conditions of the atmosphere, soil, and ground water [24] to produce daily running water balances on a 0.05° × 0.05° Australian grid. AWRA-L outputs data for several hydrological state variables, with soil moisture outputs divided into the categories of upper (0–0.1 m), lower (0.1–1 m), root zone (0–1 m), and deep soil moisture (1–6 m) [24]. Since satellite SM products operate in the skin layer of soil (top ~5 cm), we used upper SM outputted by this model. For our analysis, we used version 6 of the product, which is open loop and has no data assimilation.

2.5. Triple Collocation Analysis

TC analysis defines error variances through the concept that the unknown truth can be derived through an additive or multiplicative combination of three independent, collocated datasets [41,42]. TC was first proposed by Stoffelen (1998) [42] to calibrate wind estimates and has now been broadly used to estimate error variance for satellite-based atmospheric and climatic state variables. TC has three underlying assumptions, (i) linearity and (ii) stationarity of datasets, as well as (iii) error independence of datasets. In this study, we use the additive TC model similar to other satellite SM TC studies [32,38], such that three SM datasets satisfy the linear relationship described in Equation (1).

$$\begin{array}{c}{\mathsf{\theta }}_1={\mathsf{\beta }}_1+{\mathsf{\alpha }}_1\mathsf{\theta }+{\mathsf{\epsilon }}_1\\ {\mathsf{\theta }}_2={\mathsf{\beta }}_2+{\mathsf{\alpha }}_2\mathsf{\theta }+{\mathsf{\epsilon }}_2\\ {\mathsf{\theta }}_3={\mathsf{\beta }}_3+{\mathsf{\alpha }}_3\mathsf{\theta }+{\mathsf{\epsilon }}_3\end{array}$$

(1)

where θ is the unknown truth, θ_i (i = 1,2,3) are values from the three independent datasets, and α_i, β_i, and ε_i are the intercepts, scale factor, and zero-mean random error, respectively, for each dataset. Then using TC assumptions, the unknown truth, θ, can be eliminated, and the linear relationship can be rearranged to provide the error variance (σ_ε,i) term for each dataset (Equation (2)) in terms of the covariance, Q_i,j, between datasets. This error variance term is analogous to conventional RMSE in that it measures the spread of errors in a particular dataset.

$$\begin{array}{c}{\mathsf{\sigma }}_{{\mathsf{\epsilon }}_1}=\sqrt{{\mathrm{Q}}_{1,1}-\frac{{\mathrm{Q}}_{1,2}{\mathrm{Q}}_{1,3}}{{\mathrm{Q}}_{2,3}}}\\ {\mathsf{\sigma }}_{{\mathsf{\epsilon }}_2}=\sqrt{{\mathrm{Q}}_{2,2}-\frac{{\mathrm{Q}}_{1,2}{\mathrm{Q}}_{2,3}}{{\mathrm{Q}}_{1,3}}}\\ {\mathsf{\sigma }}_{{\mathsf{\epsilon }}_3}=\sqrt{{\mathrm{Q}}_{2,2}-\frac{{\mathrm{Q}}_{1,3}{\mathrm{Q}}_{2,3}}{{\mathrm{Q}}_{1,2}}}\end{array}$$

(2)

Draper et al. (2014) [43] proposed dividing this absolute error variance term by the standard deviation of each dataset to derive a relative error variance term, denoted by fRMSE, which we also use in our analysis.

Similarly, McColl et al. (2013) [44] solved the linear relationships outlined in Equation (1) for the Pearson correlation coefficient (R) between each dataset and the unknown truth that lies between all three (Equation (3)) in an approach they termed ‘Extended Triple Collocation’.

$$\begin{array}{c}{\mathrm{R}}_1=\sqrt{\frac{{\mathrm{Q}}_{1,2}{\mathrm{Q}}_{1,3}}{{\mathrm{Q}}_{1,1}{\mathrm{Q}}_{2,3}}}\\ {\mathrm{R}}_2=\mathrm{sign}\left({\mathrm{Q}}_{1,3}{\mathrm{Q}}_{2,3}\right)\sqrt{\frac{{\mathrm{Q}}_{1,2}{\mathrm{Q}}_{2,3}}{{\mathrm{Q}}_{2,2}{\mathrm{Q}}_{1,3}}}\\ {\mathrm{R}}_3=\mathrm{sign}\left({\mathrm{Q}}_{1,2}{\mathrm{Q}}_{2,3}\right)\sqrt{\frac{{\mathrm{Q}}_{1,3}{\mathrm{Q}}_{2,3}}{{\mathrm{Q}}_{3,3}{\mathrm{Q}}_{1,2}}}\end{array}$$

(3)

In our study, we create triplets between each satellite SM product; SM derived from a global water balance model (GLDAS) and SM from a reanalysis product (ERA5-Land), similar to Ming et al. (2022) [45]. Additionally, TC is most robust for anomaly error characterization because the errors of the seasonal cycle of different products can be cross-correlated, which would violate the TC assumption [12]. For this reason, all datasets used in our TC analysis had their seasonality removed before TC functions were performed. The south-west Pacific study domain for our TC analysis is depicted in Figure 2. This map was created in QGIS using topography data from ETOPO2v2 [46].

The two datasets used to form triplets with the satellite SM products are described in the following sections. In using GLDAS and ERA5-Land (both modelled products) for our TC analysis, we acknowledge the risk of error dependence between the two triplet datasets underlying the application of TC. Analysis of GLDAS and ERA5 metadata indicates fundamental algorithmic differences between the products and their various input parameters. We reference the work of Ming et al. (2022) [45], who also used GLDAS and ERA5-Land in TC triplets with satellite SM data and found the TC analysis to be “reasonable for the comparison and evaluation of satellite and model-based SM products” [45].

Figure 2. TC Analysis domain with the six-country sub-domains of Australia, Timor-Leste, Papua New Guinea, Solomon Islands, New Caledonia, Fiji, and Vanuatu pictured.

Figure 2. TC Analysis domain with the six-country sub-domains of Australia, Timor-Leste, Papua New Guinea, Solomon Islands, New Caledonia, Fiji, and Vanuatu pictured.

2.5.1. GLDAS

The Global Land Data Assimilation System (GLDAS) ingests satellite data for several state variables (such as precipitation, brightness temperature, elevation, and leaf area index). It uses complex land surface modelling and assimilation methods to generate land surface states [25]. We obtain 0–10 cm monthly L4 SM GLDAS data from NASA’s Earth Data Center. We use GLDAS V2.1 data which are modelled using an “open-loop” approach (i.e. no data assimilation). GLDAS SM data has a resolution of 0.25° and is in units (kg/m²) which are converted to satellite SM units (m³/m³) using the method described in NASA’s Land Data Assimilation System Documentation, wherein kg/m² SM values are divided by the mm depth of the SM layer (100 in this case), the assumption here being that the density of water in the soil is 1000 kg/m³ [47].

2.5.2. ERA5-Land

ERA5-Land is a reanalysis dataset produced by evolving the land component of the ERA5 global atmospheric reanalysis produced by the European Centre for Medium-Range Weather Forecasts (ECMWF). Unlike ERA5 global atmospheric reanalysis, ERA5-Land is evolved without coupling to the atmospheric module of the ECMWF forecasting system. ERA5-Land produces SM estimates for four soil layers, 0–7, 7–28, 28–100, and 100–289 cm, with data available from 1950–now [26]. For our analysis, we use the 0–7 cm layer and rescale it from its raw resolution of 0.10° to the study resolution of 0.25°.

2.6. Agrometeorological Drought Case Study

From the TC analysis, the best performing satellite SM dataset over PNG is used to analyze a critical drought period in the nation. We apply the Standardized Precipitation Index (SPI) and Vegetation Health Index (VHI) at 1,3- and 6-month accumulations to quantify the severity of the 2015 El Niño and Positive Indian Ocean Dipole (IOD) drought event in PNG. The SPI is commonly used to quantify precipitation deficit and involves fitting a region’s precipitation to a gamma distribution. SPI values then represent the number of standard deviations observed precipitation differs from the long-term median with −1 to −1.5; −1.5 to −2 and SPI < −2 indicative of moderate, severe, and extreme drought, respectively [48]. PNG’s rain gauge density is non-uniform and of limited capacity (less than ten rain gauges to capture over 460 000 km² of land). Thus, we compute SPI using satellite-derived rainfall from JAXA’s Global Satellite Mapping of Precipitation (GSMaP) satellite which has been shown to have the promising performance in capturing precipitation over PNG [15]. For VHI, we use NASA’s MODIS-derived VHI dataset, which was also effective at capturing the vegetative stress associated with agricultural drought in PNG [15]. VHI indicates plant health; it is composed of the Vegetation Condition Index (calculated using the Normalized Difference Vegetation Index) and the Temperature Condition Index (calculated using land surface temperatures) [49]. VHI values of 20–30, 10–20, and 0–10 indicate moderately, severely, and extremely stressed vegetation, respectively.

3. Results

This section is presented in order of investigation—point-based analysis using in situ Australian SM data, followed by AWRA-L gridded analysis, and finally TC analysis conducted over a broader south-west Pacific domain.

3.1. Point-Based Analysis

The performance of the five SM satellite products is compared to station data in the box-whisker format in Figure 3. While SMOPS and AMSR-2 have marginally better performance across these metrics, there is no dominating satellite dataset that has significantly superior performance across SM stations.

Figure 4 depicts both the spatial correlational performance of stations with significant correlations as well as the station averaged box plot. Unlike our earlier results, where SMOPS had only marginally better performance with MBE, MAE, RMSE, and ubRMSE metrics, in the correlational analysis, ASCAT has markedly superior performance with SM station time series. ASCAT not only had a higher positive correlation on average but also had a greater number of significant correlations than the other satellite SM products (this is why the ASCAT scatterplot of significant station correlations is most populated).

An accompanying box-whisker plot demonstrates that ASCAT, SMOS, and AMSR-2 have the smallest positive spread of correlation coefficients across SM stations. The spatial analysis also highlights an outlier station along the northern QLD coast that has a negative correlation (orange dot) with SMOPS, SMOS, AMSR-2, and SMAP. This station is station 10 from the CosmOz network. It is not immediately clear why Station 10 had a negative correlation for all datasets aside from ASCAT. We investigated time series for this station and satellite datasets to verify the result. CosmOz data providers note that the ‘counting rates’ of all probes within the network are scaled to this station [28]. Intuitively this should yield a significant positive correlation similar to the other CosmOz stations in our analysis; however, the opposite was observed.

Figure 5 plots the satellite SM with the highest significant correlation for each station, we note that SMOS (teal) and ASCAT (purple) have superior time series performance for several stations, with ASCAT, SMOPS, SMOS, AMSR-2, and SMAP being the best performing for 12, 13, 29, 2 and 3 stations, respectively.

Referring back to ASCAT’s performance for MBE, MAE, RMSE, and ubRMSE, we note acceptable metrics of −0.019, 0.092, 0.108, and 0.067, respectively. Similarly, for our second-best correlative dataset, SMOS, we note MBE, MAE, RMSE, and ubRMSE performance as −0.029, 0.079, 0.092, and 0.054, respectively. Both SMOS and ASCAT have slight negative biases compared to station observed SM and similarly small RMSE values with a similar variation in their random error when systematic biases are removed (ubRMSE). Thus, from the point-based analysis, we conclude that ASCAT and SMOS have promising performance when using in situ SM station data.

3.2. Gridded Analysis

Figure 6 depicts the spatial MBE across the five satellite SM datasets spatially and as box-whisker plots. Unlike in situ results, we note a positive bias between AWRA-L upper SM grids and satellite SM products except for a small area in western Tasmania, which has slight negative biases for ASCAT and SMOS. Here, SMOPS, AMSR-2, and SMAP also have large positive biases along the southeast coast, and SMOPS has the highest mean bias on average.

Similarly, for MAE (Figure 7) and RMSE (Figure 8), we find that AWRA-L average errors are similar in magnitude to average errors from the point-based analysis. Similar to MBE, SMOPS has the largest mean MAE and RMSE over Australia, and for AMSR-2, we note large MAE values over the south-eastern coast and southern Tasmania. For SMOS and ASCAT, there are also bullseye regions with high MAE values close to the Lake Eyre region of South Australia.

Figure 9 depicts the spatial ubRMSE plots and the accompanying box-whisker plots. Despite ASCAT having comparatively lower RMSE values, we note its above-average ubRMSE values. This is also consistent with the point-based analysis where Figure 3d depicts ASCAT as having the highest average ubRMSE. This indicates a larger spread of ‘random’ errors that remain when systematic bias errors are removed for ASCAT.

Figure 10 depicts the results of the correlation between AWRA-L and the satellite SM datasets. Here only spatial plots of ASCAT, SMOPS, and SMOS are depicted as time series correlations between AMSR-2/SMAP and AWRA were insignificant. Furthermore, any grid cells that have insignificant correlations are masked. Despite SMOS having the higher correlation coefficient across Australia on average, we note that its performance along Australia’s southeast coast is considerably lower than ASCAT and even insignificant (masked) in several regions.

To visualize the best performing satellite SM datasets at each grid cell in our domain, we plot the dataset with the highest and second highest significant correlation for each grid cell in Figure 11; the percentage of each satellite dataset in both analyses is provided in Table 3.

Table 3 depicts the percent totals of the ranked datasets from Figure 11. We see that ASCAT has the highest percentage of grid cells with the highest significant correlation, closely followed by SMOS. The third best-performing SM product is SMOPS. As expected in regions where ASCAT has insignificant correlations, SMOS is the best performing dataset, with the converse also being true. This is particularly observable along the south-eastern coast, where SMOS correlations are insignificant, leaving ASCAT as the better performing dataset in this region.

Table 3. Percent totals of each satellite SM dataset in for highest significant correlations and second highest significant correlation. Best and second-best performing datasets are indicated in bold.

	ASCAT	SMOPS	SMOS	AMSR-2	SMAP
% Highest Significant Correlation	38.36	24.79	35.18	0.02	1.65
% Second Highest Significant Correlation	32.07	29.44	32.17	1.08	5.24

Table 3. Percent totals of each satellite SM dataset in for highest significant correlations and second highest significant correlation. Best and second-best performing datasets are indicated in bold.

	ASCAT	SMOPS	SMOS	AMSR-2	SMAP
% Highest Significant Correlation	38.36	24.79	35.18	0.02	1.65
% Second Highest Significant Correlation	32.07	29.44	32.17	1.08	5.24

To expand on this correlation analysis, correlation coefficients were averaged based on climate zone (Figure 12). Here we use the same climate zones depicted in Figure 1. These climate zones are classified using the same data used to create Figure 1 (BoM derived Köppen classification data derived from a 30-year climatology [39]).

Overall, performance in the Equatorial and Desert climate zones is considerably lower across all satellite datasets. When considering significant correlations only, most satellite SM datasets have average correlation coefficients close to 0 except for ASCAT. The lack of significant correlations for the Equatorial region is also likely due to only comprising 7.8% of the study area. In contrast, the Temperate and Grassland regions comprise 24.7% and 32.5% of the study region, respectively. Nonetheless, we again see that ASCAT and SMOS perform best across the six major climate zones, with SMOS performing best in the Grassland region and ASCAT performing best in the others.

In Figure 5, we noted that SMOS and SMAP have superior performance when compared with Western Australian stations and also several stations in the interior of the country that would largely be based in Desert and Grassland climate zones. Figure 12 indicates that in our gridded analysis ASCAT (μ_desert = 0.55, μ_grassland = 0.63) and SMOS (μ_desert = 0.49, μ_grassland = 0.58) have similar performance in Desert and Grassland regions, with Desert climate zones being the poorest area of performance for ASCAT and Equatorial climate zones being the poorest area of performance for SMOS.

3.3. Triple Collocation Analysis

Figure 13 depicts the spatial results for each metric included in the TC analysis. For further details on TC results, consult the table in Appendix B. Spatial plots indicate generally superior performance of ASCAT and SMOS (which is consistent with AWRA-L results). However, it is difficult to consider the best-performing datasets in smaller SIDS. Thus Figure 14 depicts the results of the TC analysis as box-whisker plots categorized by country.

Figure 14 indicates somewhat varying best-performing satellite SM candidates from country to country. Overall, we note that compared to Australia, satellite SM performance is generally lower for south-west Pacific countries. In Australia, PNG, Solomon Islands, Timor-Leste, and Fiji, ASCAT has stronger median correlational performance (and generally strong RMSE performance). However, unlike Australia, where ASCAT and SMOS generally have similar performance in PNG, Solomon Islands, and Fiji, ASCAT is the sole best-performing candidate. In New Caledonia, SMOPS has the superior median correlational performance. In Vanuatu, results vary, with ASCAT and AMSR-2 having the highest median correlational performance. Of all the SIDS considered here, Vanuatu has many islands with small land sizes. It is also comprised of several atolls and archipelagos, which would only be represented by a small plurality of pixels, if any. This may be contributing to the spread of correlational performance for Vanuatu. N_gridcells for each national sub-domain are included in brackets. We caution readers that our results have the most uncertainty over the smallest islands like Vanuatu. While these results are novel in their evaluation of satellite SM products over this region, we acknowledge the limiting effects of sea water on microwave SM retrieval over land. Even though such small islands may not have the capacity for SM sensor network implementation, even small-scale in situ research activities would contribute greatly to our understanding of microwave SM retrieval in these complex archipelagic regions.

Figure 14. TC RMSE, fRMSE, and R categorized by country in the study domain—whiskers extend out to show the interquartile range, and the black bar indicates the median value of the distribution. N_gridcells for each sub-domain are indicated in brackets—we caution the impact of sea water on coarse microwave SM land observations for these regions.

Figure 14. TC RMSE, fRMSE, and R categorized by country in the study domain—whiskers extend out to show the interquartile range, and the black bar indicates the median value of the distribution. N_gridcells for each sub-domain are indicated in brackets—we caution the impact of sea water on coarse microwave SM land observations for these regions.

3.4. Case Study: Agrometeorological Drought in PNG

In 2015 a compound El Niño and Positive IOD led to severe and extreme drought conditions in PNG. This event had widespread impacts on the nation’s food supply, with drought-related frost prompting crop failure and famine conditions throughout PNG. It was estimated that up to 2.4 million people were affected, and many perished from famine conditions [12,50].

From Figure 14, we note ASCAT to have superior median correlational performance over PNG. Thus we analyze ASCAT SM time series and spatial plots for the onset, duration, and cessation of the 2015 drought period over PNG. Figure 15 plots the time series values of SPI-1-3-6, VHI-1-3-6, and ASCAT SM averaged over all of Papua New Guinea and plotted for the common period between all datasets (2014–2021).

Correlation results between the time series and ASCAT SM are also provided in the legend within Figure 15. ASCAT has significant positive correlations for all SPI and VHI accumulations. For SPI, when averaged over time and over all of PNG, ASCAT SM correlates strongest with the 3-month accumulation, which is consistent with our understanding of the agrometeorological impacts associated with 3-month rainfall deficits. For VHI, ASCAT SM correlates strongest with the 1-month accumulation.

4. Discussion

4.1. Varying Dataset Performance

Our analysis found ASCAT and SMOS to perform consistently well across all analyses and AMSR-2 to be the consistently poorer candidate. SMOS is an L-band radiometer, and several studies note L-band radiometers to be more sensitive to SM retrieval than other bands (e.g., C and X) [2,51,52]. The increased L-band SM sensitivity varies from site to site but is believed to be strongest for vegetated sites as L-band radiometers are less sensitive to surface roughness, soil texture, topography, soil bunk density impacts, and may even have higher canopy penetration capability [2,52]. As a result, several studies acknowledge the high correlational performance of SMOS across the Australian landmass [40,51,53,54,55]. In evaluating SMAP, AMSR-2, and ASCAT over three global in-situ networks, Link et al. (2020) [51] similarly found SMOS to have overall better performance and AMSR-2 to have poor correlational performance over the Yanco stations in the OzNet network with R values 0.6–0.8 and 0.3–0.5 for SMOS and AMSR-2 respectively. They also found that despite ASCAT having high raw correlational performance, it had reduced anomaly correlation performance. However, it is important to note that their analysis was conducted for the Australian Yanco site, which is clustered predominantly in a Grasslands region and could yield varying anomaly results. Anomaly correlation results are not described in detail in our work as high-level anomaly analysis using AWRA-L as truth yielded very similar results. Thus, we maintained raw correlational results for our analysis (Aside from the TC analysis where maintaining raw correlations has potential to violate TC assumptions.), but further investigation could benefit from an in-depth seasonal anomaly investigation of all datasets considered to further validate the trends observed.

While the L-band radiometer, SMOS, performed highly in our analysis, ASCAT had comparable, if not superior, performance at times. Of all five satellite SM datasets analyzed, ASCAT is the only radar-based product. Unlike passive microwave sensors that measure brightness temperatures and derive SM using a radiative transfer model, ASCAT measures backscatter coefficients to derive relative SM content. ASCAT’s slightly superior performance over Australia may be attributed to its noted high sensitivity as an active sensor for detecting SM over sparsely vegetated areas [56]. Additionally, in our gridded analysis, we used AWRA version 6 and note that the BoM’s version 7 of AWRA-L will use ASCAT satellite data assimilation approaches for model output and calibration [57]. While the reason for switching from AMSR-derived calibration to ASCAT-derived calibration is not explicitly clear in AWRA documentation, our analysis, coupled with model evaluation reports, finds ASCAT to be an outperforming satellite SM dataset over Australia (except for desert regions where performance is markedly reduced).

Montzka et al. (2017) [53] evaluated AMSR-2, ASCAT, SMAP, SMOS, and GLDAS over a range of CRP sites in Germany, the USA, Kenya, India, and Australia. For Australia, they considered one station in NSW from the CosmOz network and found the following R values 0.5454 (AMSR-2), 0.7876 (ASCAT), 0.8818 (SMAP), 0.8674 (SMOS), and 0.7514 (GLDAS). They also found ASCAT, SMAP, and SMOS to have a negative bias (~0.05 m³/m³). Our point-based analysis detected the same for ASCAT and SMOS. For ASCAT, they note this underestimation bias may be related to uncertainties in the soil porosity information provided within the ASCAT dataset that is used to convert the degree of saturation SM (%) to volumetric SM (m³/m³) [53].

From our results, we also note that AMSR-2 and SMAP had a competitive performance with point-based data, which could indicate these two datasets may still have the potential to be effective SM detection datasets in Australia but that they have fundamentally different fingerprints than water balance models and reanalysis products which also have their own systematic biases [40]. Additionally, verifying different modelled, in situ, and satellite SM, all with differing depth coverage, also introduces uncertainties in our analysis. AMSR-2 detects SM in the top 10 mm of soil which is the thinnest depth coverage out of all the datasets included in our analysis—it is possible that SM in this skin layer correlates poorly with modelled SM in a larger surface layer.

Holgate et al. (2016) [40] investigated satellite SM and water balance datasets against a range of in situ Australian SM stations. They also found (i) SMAP and AMSR-2 to have moderate-high performance against station data and (ii) that at certain sites, station data correlate only moderately with AWRA-L and other water balance data (R values of 0.58–0.88 for AWRA-L). This is a further indicator of the complexities involved with using water balance and reanalysis SM data as ‘truth’.

4.2. Performance along the Southeast Coast

Our analysis also found decreased bias, RMSE, and R performance along Australia’s mountainous southeast coast. Satellite retrievals have notoriously inconsistent performance over topography [2,58]. In our results, SMOS has insignificant time series correlations, AMSR-2 has unfavorably high RMSE values, and all satellite SM datasets display some biases over this topographical southeast coastal region. Aside from this region’s topographically complex nature, parts of the Victorian and NSW eastern coast are also home to forested land cover types. Satellite SM retrieval algorithms have systematic variation between different climate zones and different land cover types [3]. In our gridded analysis against AWRA-L, satellite SM performance was poorest in Equatorial and Desert regions. Similarly, a comprehensive satellite SM study across different terrain, climate zone, and vegetation cover by Beck et al. (2021) found performance to be poorest in regions of dense vegetation and for cold and arid climates [58].

4.3. Drought Monitoring and Early Warning Implications

We found that across both Australia and the south-west Pacific, more broadly, SMOS and ASCAT had generally superior performance making both datasets a valuable candidate for drought detection in these regions. Raw daily satellite swath passes of ASCAT and SMOS have a latency of 12–24 h and are available from open-source platforms. This increases their appeal for inclusion in drought application in regions where in-situ or modelled SM provision is limited. In our work, building capacity for a drought Early Warning System (EWS) in PNG, we pair satellite monitoring datasets with seasonal forecasts to provide staged categorical warnings for drought at a provincial scale [59]. This drought EWS considers meteorological drought (through the SPI) and agricultural drought (through the VHI). We note that its agrometeorological nature could be made more robust with soil moisture information. Our results from the 2015 case study showed that ASCAT-derived SM could detect the severe drought period with a strong correlation with SPI-3 and VHI-1. This highlights the value of using satellite SM for drought monitoring in this study region. The ability to monitor SM conditions provides a valuable layer of complexity that can be used to accompany drought EWS warnings or even integrated into the EWS thresholds and decision rules. In other SIDS, similar to PNG, where in-situ SM monitoring is limited, drought management and response mechanisms could incorporate satellite SM data (amongst a suite of other inputs) to assist communities in managing drought proactively. Satellite SM data also has significant potential to be coupled with traditional knowledge indicators for local staple and cash crop management [10,60,61]. This is an evolving area of research with minimal exploration yet with significant potential for impact in Pacific communities most at risk from famine, socioeconomic instability, and ecosystem degradation resultant of drought.

5. Conclusions

In regions where in situ SM monitoring capacity is limited, satellite SM products provide a viable alternative to monitoring SM variation. Our study focused on SM use for drought monitoring and evaluated five monthly satellite SM datasets. ASCAT and SMOS performed consistently throughout the point-based, gridded, and TC analysis. On the other hand, AMSR-2 and SMAP were outliers in their gridded and TC performance. Their relatively promising performance compared to point-based data may indicate their systematically different fingerprint to modelled or reanalysis datasets with potential for further research for clarification. The gridded analysis found ASCAT to have superior performance over the Equatorial region of Australia with an average R-value of 0.77 (despite this region being spatially small in Australia). The TC analysis in the south-west Pacific found performance to be reduced across all datasets, albeit with smaller reductions in performance for ASCAT and SMOS, indicating their potential for use in this region. Our case study of a severe drought period in Papua New Guinea found ASCAT SM to be promising in its ability to capture the time-lagged impacts of precipitation deficit, with correlations strongest at the 3-month accumulation for SPI and 1-month accumulation for VHI. The results of this study pose interesting questions about the inclusion of satellite SM for proactive drought monitoring and early warning approaches in Australia and the south-west Pacific, both of which are expected to experience more frequent, intense, and severe droughts resultant of anthropogenic climate change in the future.