Multi-Scale Evaluation of the SMAP Product Using Sparse In-Situ Network over a High Mountainous Watershed , Northwest China

As the latest L-band mission to date, evaluation of the Soil Moisture Active Passive (SMAP) products is one of its post-launch objectives. However, almost all previous studies have been conducted at the core validation sites (CVS) of the SMAP mission. This paper presents an evaluation of the SMAP soil moisture Level 3 (L3) and Level 4 (L4) products under different vegetation types at multiple tempo-spatial scales over the upper reach of the Heihe River Watershed, a topographically complex mountainous area in Northwest China. This was done through comparisons of the L3 and L4 products with ground-based observations from a sparse in situ network of permanent and temporary stations from 1 April 2015 to 22 June 2017. Results show that, compared with in situ observations at point scale, both the L3 and L4 products represent the temporal trends of the in situ observations in the study area well, with R values of 0.601 and 0.538 for the L3 ascending and descending products, respectively, and ranging from 0.353 to 0.410 for the L4 product at eight overpassing moments. However, because of the uncertainties of brightness temperature TBp and effective temperature Teff as well as their propagations in the inversion algorithm, both products did not achieve the accuracy of 0.04 m3/m3 in mountainous area. These uncertainties also result in the “dry bias” of the SMAP products in almost all the evaluations to date. Compared with areal average values at the watershed scale, the L3 product is far beyond the accuracy of 0.04 m3/m3 and the L4 product basically achieves the accuracy. In vegetation-covered land, the suitability and the variability of the coefficient bp result in both products performing best in cropland, then coniferous forest, sparse grassland, dense grassland, and alpine meadow, and worst in shrub. In barren land, the errors in estimating surface roughness h caused by the complex topography lead to poor performance of the SMAP products. With the relative errors of the SMAP brightness temperature observations and the corresponding land model forecast in the assimilation; the L3 and L4 products show different performance at both temporal and spatial scales; and the L3 product provides more reliable soil moisture estimates in the study area. Based on the results of this study, we propose: quantifying the uncertainties in estimating brightness temperature TBp and effective temperature Teff; determine coefficient bp and surface roughness h factor under various conditions; improving Goddard Earth Observing Model System Version 5 (GEOS-5) model; and deriving the SMAP-only climatology to improve the SMAP soil moisture estimates in the future.


Introduction
Soil moisture is a key variable in the global energy and water cycle, and is also important in many land surface disciplines such as agriculture, hydrology and environmental sciences [1,2].In recent years, it has been considered as an Essential Climate Variable (ECV) by the World Meteorological Organization [3].Although in situ measurements have been considered as the most accurate methods for obtaining soil moisture at point scale, it is impossible to represent the spatial distribution of highly variant soil moisture at large scale from in situ measurement networks [4,5].In recent decades, satellite products of soil moisture at large scale have become available via different active and passive microwave remote sensing techniques [6][7][8][9][10].Compared with C-band and X-band products, L-band products are much more suitable for the estimation of soil moisture because of their higher soil penetration depth, higher frequency and better vegetation penetration [11], especially for regions where soil moisture is a controlling factor on land-atmosphere exchanges [9].
Employing both an L-band radar and an L-band radiometer, the National Aeronautics and Space Administrati (NASA) Soil Moisture Active Passive (SMAP) satellite mission was launched on 31 January 2015.Both radar and radiometer instruments shared a rotating 6-m mesh reflector antenna on a platform in a 685-km sunsynchronous near-polar orbit, viewing the Earth's surface at a constant incidence angle of 40 • with swath width of 1000 km [12].However, the radar instrument encountered an irrecoverable hardware failure on 7 July 2015, and the production of soil moisture data products continues using the radiometer data alone [12,13].The SMAP mission delivers data products of four levels, including the instrument measurements (Level 1), geophysical retrievals (swath based, Level 2, and daily composite, Level 3), and land surface models assimilating SMAP measurements (Level 4) [12,14].The SMAP mission aims to provide soil moisture estimates in the top 5 cm of soil over the global land area with an unbiased root mean square error (ubRMSE) less than 0.04 m 3 /m 3 [14,15].To meet the science requirements, one of its post-launch objectives is to validate the accuracy of the science data products [14].However, so far, only a few validations of the SMAP products have been conducted, and almost all of them have been carried out at the core validation sites (CVS) [12,[16][17][18][19][20][21][22].Because observations from the CVS have been used to refine and validate retrieval algorithms, it is important to validate the SMAP products over other regions beyond the CVS [12,14,21,23].
The objectives of this work are first to validate the SMAP soil moisture Level 3 and Level 4 products under different vegetation types at multiple tempo-spatial scales over the upper reach of the Heihe River Watershed, a topographically complex, high mountainous area in Northwest China, then to quantify the differences between both products, and subsequently determine their suitability in mountainous areas.These were done through comparisons with ground-based measurements from a sparse in situ network of permanent and temporary stations from 1 April 2015 to 22 June 2017.Finally, the suitability of both products is analyzed and the future improvements are suggested.

Study Area
The Heihe River, with a drainage area of 128,000 km 2 , is the second largest inland river (or terminal lake) in China [24].The upper reach of the Heihe River Watershed is located in the Qilian Mountain ranges (Figure 1), with a length of 313 km and a drainage area of 10,009 km 2 , producing the majority of runoff for the entire Heihe River Watershed [25].The local climate is influenced by both the continental climate and the Qinghai-Tibet plateau climate [26], the mean annual temperature ranges from −3.1 to 3.6 • C with the minimum temperature of −28 • C. Mean annual precipitation ranges from 250 mm below 1900 m asl (meters above sea level) to 700 mm above 3600 m asl, and most precipitation (over 60%) falls between June and September.The elevation in the region ranges from 1674 to 5584 m asl [25,27].In the upper reach of the Heihe River Watershed, the major vegetation types include coniferous forest (Picea crassifolia), shrub (Potentilla fruticosa), steppe (Stipa purpurea Griseb), alpine meadow (Kobresia pygmaea Clarke), alpine sparse vegetation (Saussurea medusa Maxim), and desert (Sympegma regelii Bunge) [28].There are various soil types in the study area, e.g., aeolian sandy soil, cold desert soil, alpine meadow soil, mountain swamp chestnut soil and mountain marshy soil [26].Because of the complex terrain surface, it is difficult to accurately estimate soil moisture at watershed scale in the study area [29].

The SMAP Product
The NASA's SMAP satellite mission, was designed to provide global mapping of soil moisture and landscape freeze/thaw state with high resolution [9].It aims to provide soil moisture estimates in the top 5 cm of soil over the global land area, excluding surfaces with permanent ice and snow, urban areas, wetlands, and vegetated areas with vegetation water content greater than 5 kg/m 2 [14].In the upper reach of the Heihe River Watershed, the major vegetation types include coniferous forest (Picea crassifolia), shrub (Potentilla fruticosa), steppe (Stipa purpurea Griseb), alpine meadow (Kobresia pygmaea Clarke), alpine sparse vegetation (Saussurea medusa Maxim), and desert (Sympegma regelii Bunge) [28].There are various soil types in the study area, e.g., aeolian sandy soil, cold desert soil, alpine meadow soil, mountain swamp chestnut soil and mountain marshy soil [26].Because of the complex terrain surface, it is difficult to accurately estimate soil moisture at watershed scale in the study area [29].

The SMAP Product
The NASA's SMAP satellite mission, was designed to provide global mapping of soil moisture and landscape freeze/thaw state with high resolution [9].It aims to provide soil moisture estimates in the top 5 cm of soil over the global land area, excluding surfaces with permanent ice and snow, urban areas, wetlands, and vegetated areas with vegetation water content greater than 5 kg/m 2 [14].The mission requirement is ubRMSE less than 0.04 m 3 /m 3 for the surface soil moisture for the 36 km and 9 km gridded products within the retrieval domain.
The SMAP L3_SM_P_E and L4_SM_gph products from 1 April 2015 to 22 July 2017 over the study area are selected for validation in this research.Given the failure of the SMAP radar in July 2015 and the short measurement period, the L3_SM_P_E product (hereinafter referred to as "the L3 product") is daily geophysical retrieval at 9 km by 9 km of half orbits based on the downscaling results of the observed brightness temperature with a resolution of 36 km by 36 km.The L4_SM_gph product (hereinafter referred to as "the L4 product") is modeling value-added surface soil moisture product at a 3-h interval based on the SMAP retrievals and simulations by Goddard Earth Observing Model System Version 5 (GEOS-5) [14].Both the SMAP L3 and L4 products are available with equally spaced grid at ~9 km by 9 km, known as the Discrete Global Grid (DGG) [30].The entire study area is covered by roughly 410 DGGs of the SMAP products.Both products have been distributed by NASA National Snow and Ice Data Center Distributed Active Archive Center (NSIDC DAAC) and the NASA Alaska Satellite Facility Distributed Active Archive Center (ASF DAAC).Data and further details are available at http://nsidc.org/.

Sparse In situ Network
Sparse network measurements can be used as additional resources to expand the spatial and temporal scopes of the validation [12,23].In this study, the sparse in situ soil moisture network was established in June 2013 by authors' team over the upper reach of the Heihe River Watershed.It includes 35 nodes, which were established based on the combinations of vegetation and soil types in the study area (Figure 1, Table 1).Although with only 35 nodes, the sparse in situ network provides observations on the main characteristics of vegetation-soil types, making it representative to compare with the dense in-situ networks [31].
Through experimental analysis, soil textures of the in situ nodes are silt loam, silt, and sandy loam according to the United States Department of Agriculture (USDA) soil classification system, with the silt loam being the dominant texture (Table 1).For each node, soil moisture at depths of 5, 15, 25, 40 and 60 cm was measured at 30-min intervals by soil moisture sensor (Decagon's 5TE), and the data at depths of 5 cm were used as the surface soil moisture data in this study.The measurements by the soil moisture sensor were calibrated using field collected soil samples [32,33].Soil samples were first weighed in the field and then weighed again after oven dried for 24 h at 105 • C. The details of the field sampling and analysis can be found in [33,34].According to the manual provided by Decagon Company [35], the soil moisture at each in situ point was subsequently calculated based on the weight difference between the wet and dry soil and volume of the container.

Evaluation Metrics
Both the SMAP L3 and L4 products, as well as the in situ observations are in units of m 3 /m 3 .In order to facilitate comparisons with other evaluation studies, we used the following three metrics for the validation in this study.The formula of three statistical indicators, correlation coefficient (R), root-mean-square errors (RMSE) and the mean bias (bias) are as follows [5,9,36,37].
where, SM obs t and SM sat t are the soil moisture values of ground sites and satellite-based products on tth day, SM obs and SM sat are the average values of ground sites and satellite-based products during the evaluation period, n is the number of days.As the target accuracy of the SMAP product is expressed by ubRMSE (unbiased RMSE) [14,15], ubRMSE has also been analyzed in this study.It is related to

Temporal Stability Analysis
The temporal stability analysis has been conducted to compare the similarity of the spatial distributions of soil moisture data between the L3 and L4 products, over the same period from 1 April 2015 to 22 June 2017 [11,[38][39][40].For each grid x, the stability of soil moisture estimates from one time step to another is evaluated by mean relative difference (MRD) δ x and standard deviation σ(δ x ).
where, n is the number of days during the evaluation period, δ xt is the relative difference of grid x at tth day, and, SM sat t is the areal mean value for all m grids at tth day and calculated as Formula ( 7), and, SM sat xt is the soil moisture estimates of the SMAP product for grid x at tth day.The MRD and standard deviation were calculated for each grid to illustrate tempo-spatial distributions of both the L3 and L4 products in the study area.Generally, higher standard deviations of MRDs indicate a lower persistence of soil moisture distribution in time, and drier (wetter) areas will get a low (high) MRD and with a low (high) rank [11].Therefore, the similarity of MRD ranks of the L3 and L4 products is a measure of the similarity of their spatial distribution.

Evaluation of the SMAP L3 and L4 Products at Each Overpassing Moment
The differences between the retrievals of the SMAP products at different overpassing moments remain unclear [14].Therefore, the SMAP L3 and L4 products, at each overpassing moment from 1 April 2015 to 22 June 2017, were compared with the corresponding observations at all the in situ sites.As shown in Table 2, the L3 product of both the ascending and descending orbits, catches the trend of the observations well with R values of 0.601 and 0.538, respectively, with significance of α = 0.001.The bias values are negative, indicating that the L3 product underestimated soil moisture, so called "dry bias" [11].The RMSE values are 0.062 and 0.065 m 3 /m 3 , while the ubRMSE values are 0.054 and 0.053 m 3 /m 3 for the ascending and descending products, respectively.Thus, compared with in situ observations at point scale, the SMAP L3 product did not achieve the accuracy of 0.04 m 3 /m 3 in the study area.With larger R values, smaller RMSE and absolute bias values, and almost the same ubRMSE values, the L3 ascending product performed better than the L3 descending product.The evaluation indices of the L4 product at all the eight overpassing moments are shown in Table 2.With R values ranging from 0.353 to 0.410 (at significant level of α = 0.001), the L4 product at all the eight overpassing moments catches the trend of the observations well.The bias values are positive ranging from 0.024 to 0.028, thus the L4 product overestimated soil moisture data in the study area.As the RMSE values range from 0.074 to 0.077 m 3 /m 3 , and ubRMSE values range from 0.070 to 0.072 m 3 /m 3 , the SMAP L4 product did not achieve the accuracy of 0.04 m 3 /m 3 in the study area compared with in situ observations at point scale.Although R values slightly vary with overpassing moments, both RMSE and bias values are nearly the same with overpassing moments, indicating that the L4 product has systemic errors in the study area.Furthermore, the L3 product fits the observation better than the L4 product with larger R values, and smaller RMSE and ubRMSE values (Table 2).

Evaluation of the SMAP L3 and L4 Products under Different Vegetation Types
Soil heterogeneity contributes less than 0.7% volumetric soil-moisture error for L-band product [41].Moreover, soil textures of the in situ nodes show little difference in this study (Table 1).Thus, clustered in situ soil moisture time series by vegetation type were compared with the SMAP estimates to investigate the differences of soil moisture estimates under different vegetation types.As shown in Table 3, the L3 product catches the trend of the observations well with R values ranging from 0.160 to 0.752 with all at α = 0.001 significant level.The bias values are negative, indicating that the L3 product also shows "dry bias" for all the vegetation types.As the RMSE values range from 0.047 to 0.208 m 3 /m 3 , while the ubRMSE values range from 0.025 to 0.079 m 3 /m 3 , the L3 product did not achieve the accuracy of 0.04 m 3 /m 3 for all the vegetation types except cropland and coniferous forest.
Regarding to the L4 product, with R values ranging from 0.218 to 0.546 (all of them pass significance test of 0.001), it catches the trend of the observations well at all the eight overpassing moments, but also exhibits a substantial bias (Table 3).The L4 product underestimated soil moisture in alpine meadow and shrub with negative bias values, overestimated soil moisture in sparse and dense grasslands, coniferous forest and cropland with positive bias values, while showing negligible bias in barren land with values near zero.As the RMSE values range from 0.048 to 0.163 m 3 /m 3 , and ubRMSE values range from 0.046 to 0.117 m 3 /m 3 , the SMAP L4 product did not achieve the accuracy of 0.04 m 3 /m 3 for all the vegetation types in the study area.Because R values are strongly affected by the data amount, and RMSE values are strongly affected by the value range, ubRMSE is used as the main index to evaluate the differences of the soil moisture estimates between vegetation types when R values pass significance test of 0.001.As shown in Table 3, both the L3 and L4 products match the observations best in cropland, then coniferous forest, sparse grassland, dense grassland, alpine meadow, barren land and shrub.With larger R values and smaller ubRMSE values, the L3 product shows better performance than the L4 product for all the vegetation types except shrub.In shrub, the L4 product shows larger R value and larger ubRMSE value.

Comparison of the SMAP L3 and L4 Products at Watershed Scale
The SMAP L3 and L4 products, as well as in situ observations, were processed into daily areal average values for fair comparison at the watershed scale.The comparison of time series variance is shown in Figure 2. At the watershed scale, the R values are 0.810 and 0.411 for the L3 and L4 products, respectively, indicating that both products capture soil moisture dynamics very well and the L3 product performed better.The ubRMSE values are 0.027 and 0.042 m 3 /m 3 for the L3 and L4 products, thus the L3 product is far beyond the accuracy of 0.040 m 3 /m 3 at the watershed scale, while the L4 product basically achieves the accuracy.The bias values are −0.024 and 0.019 m 3 /m 3 , indicating that the L3 product underestimates soil moisture data and the L4 product overestimates soil moisture data in the study area.It is worth mentioning that, in winter (from December to March), the L3 product lacked data and the L4 product provided unreliable estimates with a linear trend showing significantly overestimations.The details of seasonal differences are shown in Table 4.Both the L3 and L4 products performed best in autumn, then followed by summer and spring.Evaluated by the combination of R, RMSE, absolute bias and ubRMSE values, the L3 product fits the observation better than the L4 product in spring, summer and autumn.

Temporal Stability Analysis
The temporal stability analysis is applied in this study because it is less influenced by sampling size and not so much dependent on the absolute values of soil moisture, but rather accounts for the tempo-spatial distributions of soil moisture [11].The MRDs and their standard deviations of both the L3 and L4 products are shown in Figure 3.The L3 product has MRDs between −0.489 and 0.902, and for the L4 product the range of MRDs is from −0.438 to 0.945.The standard deviations of the L3 MRDs are quite high, ranging from 0.099 to 0.371 with mean value over all pixels of 0.184.For the L4 product, the standard deviations of MRDs range from 0.027 to 0.137 with an areal mean value of 0.072.Compared to the L4 product, the higher ranges of MRDs and higher standard deviations of the L3 product imply more dynamic in its tempo-spatial distributions.
For the L3 product, the MRDs are presented as smaller in the northwest and larger in the southeast, indicating that the estimates are drier in the northwest and wetter in the southeast.For the L4 product, the MRDs are smaller in the west and slightly larger in the east, which means that the estimates are drier in the west and wetter in the east.The precipitation in the study area shows increasing trends from northwest to southeast [42,43], leading to same spatial pattern of soil moisture.Thus, the L3 product better catches the spatial pattern of soil moisture in the study area.For both products, the standard deviations are smaller in central areas and larger in outlying areas.This implies lower persistence of soil moisture distributions over time in outlying areas than in central areas for both products.Nevertheless, the standard deviations of the L4 product show significantly smaller range, indicating that the L4 product had greater persistence of soil moisture distributions over time than the L3 product (Figure 3).It is worth mentioning that, in winter (from December to March), the L3 product lacked data and the L4 product provided unreliable estimates with a linear trend showing significantly overestimations.The details of seasonal differences are shown in Table 4.Both the L3 and L4 products performed best in autumn, then followed by summer and spring.Evaluated by the combination of R, RMSE, absolute bias and ubRMSE values, the L3 product fits the observation better than the L4 product in spring, summer and autumn.

Temporal Stability Analysis
The temporal stability analysis is applied in this study because it is less influenced by sampling size and not so much dependent on the absolute values of soil moisture, but rather accounts for the tempo-spatial distributions of soil moisture [11].The MRDs and their standard deviations of both the L3 and L4 products are shown in Figure 3.The L3 product has MRDs between −0.489 and 0.902, and for the L4 product the range of MRDs is from −0.438 to 0.945.The standard deviations of the L3 MRDs are quite high, ranging from 0.099 to 0.371 with mean value over all pixels of 0.184.For the L4 product, the standard deviations of MRDs range from 0.027 to 0.137 with an areal mean value of 0.072.Compared to the L4 product, the higher ranges of MRDs and higher standard deviations of the L3 product imply more dynamic in its tempo-spatial distributions.
For the L3 product, the MRDs are presented as smaller in the northwest and larger in the southeast, indicating that the estimates are drier in the northwest and wetter in the southeast.For the L4 product, the MRDs are smaller in the west and slightly larger in the east, which means that the estimates are drier in the west and wetter in the east.The precipitation in the study area shows increasing trends from northwest to southeast [42,43], leading to same spatial pattern of soil moisture.Thus, the L3 product better catches the spatial pattern of soil moisture in the study area.For both products, the standard deviations are smaller in central areas and larger in outlying areas.This implies lower persistence of soil moisture distributions over time in outlying areas than in central areas for both products.Nevertheless, the standard deviations of the L4 product show significantly smaller range, indicating that the L4 product had greater persistence of soil moisture distributions over time than the L3 product (Figure 3).

Impacts of Uncertainties in Retrieval Algorithms
In the SMAP mission, the tau-omega model has been applied to describe the components from the soil, and the vegetation canopy contributes to the L-band brightness temperature[9] [14].If the air, vegetation, and near surface soil can be assumed to be in thermal equilibrium, then the vegetation temperature (TC) is approximately equal to soil effective temperature (Ts), and both temperatures TC and Ts can be replaced by the single effective temperature for the scene (Teff) in the radiative transfer equation.To retrieve soil moisture data, it is necessary to isolate the soil surface emissivity p e in Formula (8) by inversion of the tau-omega model.There are uncertainties in estimating both TBp and Teff in the SMAP mission.In mountainous areas, angles of incidence in the target area could not be derived solely from the radiometer

Impacts of Uncertainties in Retrieval Algorithms
In the SMAP mission, the tau-omega model has been applied to describe the components from the soil, and the vegetation canopy contributes to the L-band brightness temperature [9,14].If the air, vegetation, and near surface soil can be assumed to be in thermal equilibrium, then the vegetation temperature (T C ) is approximately equal to soil effective temperature (Ts), and both temperatures T C and Ts can be replaced by the single effective temperature for the scene (T eff ) in the radiative transfer equation.To retrieve soil moisture data, it is necessary to isolate the soil surface emissivity e p in Formula (8) by inversion of the tau-omega model. where and γ = exp(−τ p sec θ) (10) and T Bp is the brightness temperature of each SMAP grid cell, τ p is the nadir vegetation opacity, ω p is the vegetation effective scattering albedo, θ is the surface incidence angle of 40 • .Based on the derived value of e p , the smooth surface soil emissivity is determined by removing the roughness effects, and then soil moisture can be retrieved based on Fresnel equation and dielectric mixing model.Therefore, the accuracy of T Bp and T eff , as well as the suitability of τ p and ω p are crucial for the retrieval of soil moisture data [9,14].There are uncertainties in estimating both T Bp and T eff in the SMAP mission.In mountainous areas, angles of incidence in the target area could not be derived solely from the radiometer observation angle, and the surrounding reflection significantly affects brightness temperature simulation [44].Nevertheless, both of them have been ignored in the SMAP mission [14].Meanwhile, the L3_SM_P_E product with a resolution of 9 km by 9 km, has been retrieved from the downscaled brightness temperatures in the L2_SM_P product [45].The downscaling algorithms would cause uncertainties in T Bp estimates, especially in areas with complex land surface conditions [45,46].In the SMAP mission, the effective temperature T eff has been provided by the GMAO (GSFC Global Modeling and Assimilation Office) model, also with unclear uncertainty.Moreover, the uncertainties of both T Bp and T eff would propagate through the soil moisture inversion algorithm [45], which is still unclear and requires further investigations in the future.Therefore, although well catching the temporal trend of in situ observations, both the L3 and L4 products did not achieve the accuracy of 0.04 m 3 /m 3 at each overpassing moment in the study area.
The L3 product shows "dry bias" for all the vegetation types at each overpassing moment in this study, which has also been revealed at both footprint scale and global scale in previous studies, especially in mountainous areas (Table 5).Colliander et al. [12] attributed the "dry bias" to growth effects of vegetation.However, as shown in Table 6, both the L3 ascending and descending products underestimated soil moisture for all the vegetation types in both growing seasons (summer and autumn) and non-growing season (spring), with even larger underestimation in non-growing season.It implies that the "dry bias" is more related to the system structure of retrieval algorithms rather than the growth effects.Both τ p and ω p are the factors describing vegetation effects, and the retrieval parameters of them have been trained or validated by observations from the CVS [12], thus they can be assumed approximately precise at the CVS.However, the evaluations of the SMAP product at the CVS also show the "dry bias" [12].Therefore, the "dry bias" of the SMAP L3 product is mainly caused by the uncertainties of the T Bp and T eff estimates rather than other factors.Furthermore, because the assumptions of thermal equilibrium are more likely to be true at the 6 a.m.SMAP overpass [9,14], the L3 ascending product fits the observation better than the L3 descending product.

Impacts of Parameters b p and h under Different Vegetation Types
According to Formula (8), τ p and ω p are two important factors related to vegetation effects [9,14].In the SMAP mission, the nadir vegetation opacity τ p is related to the total columnar vegetation water content W (kg/m 2 ) by τ p = b p W with the coefficient b p dependent on vegetation type [9,14].Both b p and ω p have been determined before launch [12,14].Meanwhile, surface soil reflectivity r p , related to the soil emissivity e p by e p = 1−r p , has been smoothed by r smooth p = r rough p / exp(−h) in the SMAP retrieval.As a linear function of the root mean square of surface heights, surface roughness h is also an important factor for soil moisture estimation in mountainous area [51,53].
As the parameters ω p and b p are functions of vegetation geometry and vegetation water content, both of them are important factors affecting the performance under vegetation conditions.Better characterizing parameter ω p is important in soil moisture retrievals from space-borne observations [54-57].However, as shown in Table 7, differing significantly in the values of ω p , both the SMAP L3 and L4 products show same performance trends under different vegetation types in the study area.Thus, the effects of ω p on the performance differences of both products under different vegetation types are not discussed in this study.The b p factor is more sensitive than surface roughness h in vegetated condition, thus the b p factor is the most important parameter in soil moisture retrieval of the SMAP product for vegetation [58].In the study area, the main crop is corn in cropland, which has been considered in the SMAP mission [14,59], thus leading to best performance in cropland in the study area far beyond the accuracy of 0.04 m 3 /m 3 .For coniferous forest, the parameters of evergreen needle leaf have been calibrated and validated by the CanExSM10 (The Canadian Experiment for Soil Moisture in 2010) measurements [14,60,61].The main tree types are pine and spruce over the BERMS (Boreal Ecosystem Research and Monitoring Sites) forested sites in CanExSM10 [61], both of them are coniferae as well as picea crassifolia in our study area.Thus, the parameters of forest are suitable in the study area because of the similarity of tree species, resulting in better performance of the SMAP products in coniferous forest far beyond the accuracy of 0.04 m 3 /m 3 .
The parameters of grasslands have also been considered in the SMAP mission [14,62], but performed not as well as in coniferous forest.This is because the b p factor varies depending on different soil and vegetation conditions [58], but only average values for each type have been applied in the SMAP mission [14,62].Compared with the coniferous forest, the invariable b p factor results in worse performance because of stronger intra-annual variation in grasslands (alpine meadow, sparse and dense grasslands).Since the brightness temperature shows higher sensitivity to vegetation under wet soil conditions [9,63], the misevaluation impacts of the b p factor are much more significant in wet conditions [61,64].Because of the wetness conditions of three grasslands (Table 8 and Figure 4), the L3 product shows best performance in sparse grassland under the driest condition, then dense grassland, and alpine meadow under the wettest condition.The SMAP L3 product shows poor performance in estimating soil moisture in shrub land.This is because the dominant species is potentilla fruticosa in the shrub land in the upper reach of the Heihe River Watershed, which belongs to the vegetation class of open shrublands in the SMAP mission.However, in the study area, the shrubland is often mixed with grassland, which leads to complexity in soil moisture retrieval.Moreover, potentilla fruticosa shows strong intra-annual variations because of growth effects.Thus, the invariable b p factor results in errors of soil moisture retrievals in the shrubland.Meanwhile, the brightness temperature is much more sensitive to vegetation under wet soil conditions [9,63].Under the wettest condition with higher soil moisture data in the shrub (Table 8 and Figure 4), the misevaluation impacts of the b p factor is most significant and thus leading to worst performance in the shrub in the study area.
The L4 product is the assimilation results of the SMAP retrievals and the GEOS-5 model simulations, thus it shows similar performance with the L3 product under different vegetation types.Overall, the suitability as well as the variability of the b p factor result in the performance differences of both the L3 and L4 products under different vegetation types in the study area.The impacts of the b p factor is more significant under wet conditions, resulting in performance differences in the three grasslands with a declining order of sparse grassland, dense grassland and alpine meadow.
For barren land, compared to surface roughness and soil moisture, the backscatter weakly depends on soil type [65].The uncertainties of surface roughness h dominate the error budget of T Bp modeling over barren soil surface [53].Meanwhile, although depending on surface soil moisture, the parameterization of h performs better when soil surface is relatively smooth than when the soil surface gets rougher [53].Because of the complex topography in the study area, the errors in parameterization of h lead to poor performance of the SMAP products in barren land in the study area.

Impacts of Seasonal Frozen Soil
In the SMAP mission, the soil moisture data are retrieved by the relationship between soil moisture and dielectric constant.As soil moisture increases, the soil dielectric constant increases, which leads to an increase in soil reflectivity or a decrease in soil emissivity [14].However, besides dry soil, low dielectric constant can also be associated with frozen soil which has a similar dielectric constant to dry soil independent of water content [14].Thus, landscape freeze/thaw state is important in soil moisture data retrieval in the SMAP mission [66].
In the upper reach of the Heihe River Watershed, seasonal frozen soils account for about 65% of the study area [67], which strongly affects the soil moisture retrieval in the SMAP mission.In winter, the freeze_thaw_fraction values of all grids are near 1.0, leading to missing estimates, thus there is no data from the L3 product in the study area.For those frozen grids with estimates in the study area, the SMAP mission estimates unfrozen soil water, and the freeze_thaw_fraction values range from 0.008 to 0.892, the R values are 0.464 and 0.334 for the ascending and descending products, respectively, while the ubRMSE values are 0.051 and 0.053 m 3 /m 3 , respectively.Because the in situ soil moisture sensor 5TE can measure free water in frozen soil [68], the L3 product well catches the trend of observations under the frozen states.Under the unfrozen state, the R values are 0.624 and 0.555 for the ascending and descending products, respectively, and the corresponding ubRMSE values are 0.054 m 3 /m 3 for both products.With slight differences in bias, the L3 product better catches the temporal trend of the in situ observations under the unfrozen state than the frozen state.Meanwhile, because of the soil frozen/thaw processes in spring in the study area, both the L3 and L4 products showed better performance in summer and autumn than in spring.However, the better performance of both products in autumn than in summer is caused by growth effects of the vegetation during the growing season.

Impacts of Assimilation System
The L4 product is soil moisture estimate by assimilating the SMAP observation (downscaled brightness temperature from L2 product) with simulation by a catchment land surface model, GEOS-5 [14].The assimilation causes the performance differences in the L3 and L4 products.
Because the L-band brightness temperatures generated by the GEOS-5 model and its associated microwave radiative transfer model have been calibrated to match the climatology of satellite observations, the GEOS-5 model provides unbiased modeled brightness temperatures in long-term mean ignoring seasonal variations in bias [17].Meanwhile, the brightness temperature by the SMAP observations is converted to anomalies for assimilation by removing the climatology of satellite observations [17].With lacking of SMAP-only climatology, the aforementioned climatology of satellite observations has been derived by the Soil Moisture and Ocean Salinity (SMOS) product, thus leading to the systemic errors of soil moisture estimates by the SMAP L4 product.
The calibration of the GEOS-5 model makes its simulation results more capable in reflecting the climatological trends with smaller variation.In winter, as there is no SMAP observed brightness temperature because of frozen states as for the L3 product, the estimates of the L4 product are all the simulation results by the GEOS-5 model, thus leading to linear temporal trends of the L4 product in winter.Also because of the better reflection in the climatological trends by the GEOS-5 model, the L4 product shows smaller variations with significantly smaller coefficient of variation (CV) values than the L3 product (Figure 4), thus the L3 product is more dynamic than the L4 product in catching the tempo-spatial distributions of soil moisture at the watershed scale.
In winter, the larger areal average values of soil moisture indicate that the GEOS-5 model overestimates soil moisture in the study area.This is because the GEOS-5 model has been calibrated over unfrozen land to simulate all the water in the soils [69].However, the observed soil moisture is free water in the frozen soils, thus the values would be much smaller than the model simulations.Meanwhile, as shown in Table 4 and Figure 4, compared to the L3 product, the L4 product shows larger soil moisture estimates in all the seasons and under all the vegetation types.In the SMAP mission, the L3 product is the retrieval of downscaled (9-km) brightness temperatures, and the L4 product is assimilation result by retrieval of downscaled (9-km) brightness temperatures and GEOS-5 simulation.Because the same algorithm has been applied to retrieve soil moisture from brightness temperatures in both the L3 and L4 products, the larger estimates of the L4 product are mainly caused by the GEOS-5 model simulations.Moreover, the L3 product underestimates soil moisture in the study area, the overestimation of the L4 product indicates that the GEOS-5 model overestimates soil moisture in the study area.In summary, the assimilation makes the L4 product show similar temporal trends with the L3 product, but significantly differ in tempo-spatial distributions.Because of the relative errors of the SMAP brightness temperature observations and the corresponding land model forecast in the assimilation [14], the L3 product shows better performance than the L4 product in the study area under all the evaluation cases.

Conclusions
This paper presents evaluation of the SMAP L3 and L4 products under different vegetation types at multiple tempo-spatial scales over the upper reach of the Heihe River Watershed, Northwest China.The results are expected to increase the understanding of the suitability and the future improvements of the SMAP soil moisture products.Both products were compared with ground-based observations from a sparse in situ network from 1 April 2015 to 22 June 2017.Results show that both the L3 and L4 products well catch the temporal trend of the in situ observations in the study area.However, compared with in situ observations at point scale, both of them did not achieve the accuracy of 0.04 m 3 /m 3 because of the uncertainties of brightness temperature T Bp and effective temperature T eff as well as their propagations in the inversion algorithm.Compared with areal average values at the watershed scale, the L3 product is far beyond the accuracy of 0.04 m 3 /m 3 and the L4 product basically achieves the accuracy.The performance differences of both products at point scale and at watershed scale are caused by the scale mismatch in the evaluations, which is not discussed here because this study focuses on the impacts of vegetation types on the soil moisture estimates.
Almost all the evaluations show the "dry bias" of the SMAP product, it is more related to the aforementioned uncertainties of T Bp and T eff rather than the growth effects stated by Colliander et al. (2017a).Under different vegetation types, although both the L3 and L4 products catch the temporal trend of the observations well, the suitability and the variability of the b p factor result in that both products performed best in cropland, then coniferous forest, sparse grassland, dense grassland, alpine meadow and worst in shrub.In barren land, the errors in estimating surface roughness h caused by the complex topography lead to poor performance of the SMAP product.
Although with similar temporal trends, the L3 and L4 products show different performance at both temporal and spatial scales, which are mainly caused by the assimilation system.Because the climatology of satellite observations used in the assimilation system has been derived by the SMOS product, the SMAP L4 product has systemic errors in soil moisture estimates.The calibration of the GEOS-5 model makes its simulation results more capable in reflecting the climatological trends with smaller variations, leading to larger variations of the L3 product than the L4 product in both temporal variations and spatial distributions.Moreover, the overestimations of soil moisture by the GEOS-5 model cause the overestimations of the L4 product.Although lacking data in winter, the L3 product well estimated the free soil water under both unfrozen state and frozen state.However, the L4 product provides unreliable estimates with a linear trend showing significantly overestimations in winter.Overall, because of the relative errors of the SMAP brightness temperature observations and the corresponding land model forecast in the assimilation, the L3 product provides more reliable soil moisture estimates with more dynamic spatial patterns than the L4 product in the study area.
Based on the results of this study, the following suggestions are proposed for the future improvements of the SMAP products.First, it is necessary to quantify the uncertainties of the estimates of brightness temperature T Bp and effective temperature T eff to retrieve more reliable soil moisture data.Second, the suitable values of coefficient b p and surface roughness h are crucial in soil moisture retrievals, it is essential to determine b p factor and h factor under various conditions in the future, especially for vegetation types with strong intra-annual variations.Finally, both the SMAP-only climatology and improvements of the GEOS-5 model are needed to improve the SMAP soil moisture estimates.

Figure 1 .
Figure 1.The Study area (a) The Heihe River Watershed in Northwest China; (b) The upper reach of the Hehe River Watershed.

Figure 1 .
Figure 1.The Study area (a) The Heihe River Watershed in Northwest China; (b) The upper reach of the Hehe River Watershed.

Figure 2 .
Figure 2. Time series of daily areal average values of the SMAP L3 and L4 products and in situ observation in the study area.

Figure 2 .
Figure 2. Time series of daily areal average values of the SMAP L3 and L4 products and in situ observation in the study area.

Figure 3 .
Figure 3. Temporal stability analysis of the SMAP L3 and L4 products.(a) mean relative differences (MRDs) of the L3 product; (b) standard deviation of the L3 product; (c) MRDs of the L4 product; (d) standard deviation of the L4 product.

and
TBp is the brightness temperature of each SMAP grid cell, p τ is the nadir vegetation opacity, p ω is the vegetation effective scattering albedo, θ is the surface incidence angle of 40°.Based on the derived value of p e , the smooth surface soil emissivity is determined by removing the roughness effects, and then soil moisture can be retrieved based on Fresnel equation and dielectric mixing model.Therefore, the accuracy of TBp and Teff, as well as the suitability of p τ and p ω are crucial for the retrieval of soil moisture data [9,14].

Figure 3 .
Figure 3. Temporal stability analysis of the SMAP L3 and L4 products.(a) mean relative differences (MRDs) of the L3 product; (b) standard deviation of the L3 product; (c) MRDs of the L4 product; (d) standard deviation of the L4 product.

Table 1 .
The in-situ observation points in the study area.

Table 2 .
Evaluation of the Soil Moisture Active Passive (SMAP) L3 and L4 products in the study area.
Note: * refers to passing significance test of 0.001.

Table 3 .
Evaluation of the SMAP L3 and L4 products under different vegetation types.
Note: * refers to passing significance test of 0.001.The indices are average values of all the overpassing moments for each product.

Table 4 .
Evaluation of daily average values from the SMAP L3 and L4 products at seasonal scale.

Table 4 .
Evaluation of daily average values from the SMAP L3 and L4 products at seasonal scale.

Table 5 .
Evaluations of the SMAP products with in situ observations in previous studies.

Table 6 .
Evaluation of the SMAP L3 products of both ascending and descending orbits at seasonal scale.
Note: No data of the SMAP product in winter.* refers to passing significance test of 0.001.

Table 7 .
Values of ω p used in both the SMAP L3 and L4 products under different vegetation types in the study area.

Table 8 .
Statistics of in-situ observation under different vegetation types.