Assessment of Satellite-Derived Precipitation Products for the Beijing Region

: Performance of four satellite precipitation products, namely, the China Meteorological Forcing Dataset (CMFD), Climate Prediction Center morphing technique (CMORPH), as well as 3B42 calibrated and 3B42 - RT dataset, which are derived from the Tropical Rainfall Measuring Mission (TRMM) and Multi-satellite Precipitation Analysis (TMPA), were evaluated from daily to annual temporal scales over Beijing, using observations from 36 ground meteorological stations. Five statistical properties and three categorical metrics were used to test the results. The assessment showed that all four satellite precipitation products captured the temporal variability of precipitation. Although four satellite precipitation products captured the trend of more precipitation in the northeastern regions, all four products showed different distribution from the observations for 2001–2015 over Beijing. All precipitation products tended to overestimate moderate precipitation events and underestimate heavy precipitation events over Beijing, except for 3B42RT, which tended to overestimate most precipitation events. By comparison, the CMFD performed better than the CMORPH, 3B42 calibrated, and 3B42-RT datasets, having the higher correlation coefficient and low root mean squared difference, and mean absolute difference at all temporal scales. The average correlation coefficient of the CMFD, CMORPH, 3B42 calibrated, and 3B42-RT products for all 36 stations were 0.70, 0.60, 0.59, and 0.54 for daily precipitation and 0.78, 0.32, 0.74, and 0.44 for monthly precipitation. Overall, the CMFD was the most reliable for the Beijing region. The observations showed that station elevation is slightly negatively correlated with annual precipitation. The same slightly negative correlation was present between the station elevation and the CMFD dataset, while the CMORPH and TMPA datasets (both 3B42 calibrated and 3B42-RT) presented a slightly positive correlation with the station elevation.


Introduction
Precipitation plays an important role in global water cycles, linking the atmosphere and the land surface, and affecting meteorology, climatology, and hydrology [1][2][3]. The Fifth Assessment Report of the Intergovernmental Panel on Climate Change (IPCC AR5) concluded that climate change has affected extreme events such as extreme temperatures, extreme precipitation, and droughts, etc., and some of the changes in weather and climate extremes observed in the late 20th century are projected to continue into the future. It has been demonstrated that the current global surface temperature warmed by 0.85 °C from 1880 to 2010 [4,5], lakes have been warming with the lake surface water temperature in some regions exceeding nearby surface air temperature changes during the 20th Zhang et al. [37], who evaluated three satellite precipitation datasets in the Tianshan mountain area in China, concluded that global precipitation measurement performed better than 3B42 calibrated and CMORPH in daily precipitation. Although some of the satellite precipitation products show good correlation with ground observations in particular regions, no evidence supports the use of one dataset for all applications. To date, previous studies evaluating satellite precipitation data were focused on the global, national, and basin scales. This reveals that a comprehensive evaluation of satellite precipitation products at the scale of a megacity is lacking.
In this study, the performance of four satellite-based precipitation products, namely, the China Meteorological Forcing Dataset (CMFD), CMORPH, as well as 3B42 calibrated and 3B42-RT (version 7) was comprehensively evaluated using ground observations of precipitation from 36 meteorological stations at multiple temporal scales over Beijing. Our objective was to evaluate these satellite precipitation products for Beijing to provide more reliable data for water resource management.

Study Area
Beijing, the capital of China, is located in northern China, with a latitudinal range of 39°28′ N-41°05′ N and a longitudinal range of 115°25′ E-117°30′ E, as shown in Figure 1. Its total area is 16.8 × 10 3 km 2 , with mountain and plain areas of 10.4 × 10 3 km 2 and 6.4 × 10 3 km 2 , respectively. The topography of Beijing changes from mountains to plains, from west to east and from north to south. The climate of Beijing is typical of a semi-humid continental monsoon climate of the warm temperate zone. The average annual temperature is between 11-13 °C, although the highest temperatures in summer can reach 42 °C and the lowest temperatures in winter can be as low as −27 °C [38][39][40]. The average annual precipitation for Beijing is 508.8 mm/year, according to observed precipitation data from 2001 to 2015. Regional differences in annual precipitation are considerable. The maximum average annual precipitation can reach 682.9 mm/year at the Zaoshulin station, which is located in the northeast areas of Beijing, while the minimum average annual precipitation is only 372.1 mm/year at the Yanhecheng station, which is located in the southwest mountain areas. The seasonal distribution of precipitation is uneven. The amount of summer (June-August) precipitation is large, accounting for above 70% of the total annual precipitation, while precipitation in winter is only around 2% of the total annual precipitation [41].

Datasets
The period from 1 January 2001 to 31 December 2015 was chosen as the study period since this period is the overlapping period for four satellite precipitation products. The CMFD, CMORPH, 3B42 calibrated, and 3B42-RT (version 7) precipitation data derived from Tropical Rainfall Measuring Mission (TRMM) and Multi-satellite Precipitation Analysis (TMPA) were selected for study because: (1) previous research has indicated that the TMPA (version 7) products and CMORPH data are suitable for mainland China, and TMPA (version 7) shows an improved performance compared with TMPA (version 6) products [36]; and (2) the CMFD is a set of reanalysis data that incorporates meteorological data from the China Meteorological Administration. All four satellite precipitation datasets were interpolated to each station using bilinear interpolation to match the scale of ground observations.

Ground Observations
Daily precipitation data measured at 36 meteorological stations over Beijing, as shown in Figure  1, were used in this study. The average annual precipitation for each station was calculated by daily precipitation data from 2001 to 2015. The dataset was provided by the Beijing Hydrology Bureau and the National Climate Centre of China. Detailed information about 36 ground stations are provided in Table 1. The observed daily precipitation data required some preprocessing for the missing data, which were replaced by multi-year daily means for the given time point.

Satellite-Derived Precipitation Datasets
The CMFD was developed by the Institute of Tibetan Plateau Research of the Chinese Academy of Sciences (http://westdc.westgis.ac.cn/data/7a35329c-c53f-4267-aa07-e0037d913a21) [42]. It includes near surface meteorological and environmental factors. Using the Princeton reanalysis data, Global Land Data Assimilation System (GLDAS), Global Energy and Water Cycle Experiment-Surface Radiation Budget (GEWEX-SRB) radiation data, and the TMPA 3B42 calibrated products as the background field, the CMFD was integrated with conventional meteorological observations from the China Meteorological Administration, which are used to correct systematic departures in the background data. The CMFD includes seven variables, namely, near surface temperature, near surface pressure, near surface air specific humidity, near surface wind speed, downward short-wave radiation data, downward long-wave radiation data, and precipitation rate [42,43]. The spatial and temporal resolutions of the CMFD are 0.1° × 0.1° and 3 h. Daily data were obtained by adding eight sets of consecutive 3-h data.
The CMORPH dataset was developed by the Climate Prediction Center of the National Oceanic and Atmospheric Administration. It is observed and produced using multiple platforms and interpolated in both temporal and spatial scales [44]. The infrared (IR) observations of the geostationary satellite platform and the passive microwave (PMW) information of the low-orbit satellite are combined within the CMORPH dataset. In this case, the IR observations have the advantage of high temporal resolution and the microwave data have good inversion properties. Two steps were included in the fusion of the CMORPH dataset. In the first step, the moving vector of the IR cloud system was calculated every 30 min, while the instantaneous precipitation distribution was obtained from the inversed PMW information. Both were extrapolated to the target analysis time along a moving vector to produce a continuous spatial distribution of precipitation. The IR information was used to acquire the moving vector for extrapolating the precipitation distribution from the PMW inversion, rather than by evaluating the precipitation data [15,[45][46][47]. The spatial and temporal resolutions of the CMORPH data used in this study are 0.25° × 0.25° and 24 h.
The TRMM was launched by the National Aeronautics and Space Administration and the Japan Aerospace Exploration Agency in 1997. It is a joint remote sensing precipitation observation mission. The TMPA (version 7) consists of 3B42 calibrated and 3B42-RT precipitation datasets, which are intended to supply the "best" global quasi-precipitation data [14,[48][49][50]. The TMPA products are constructed using four steps: (1) the Goddard profiling algorithm (version 2010) is used to estimate the optimal value of the PMW measurements; (2) this optimal value is used to create IR precipitation estimates; (3) combining the value of the PMW data and the IR estimates, the IR data are used to fill any gaps in the PMW data; and (4) the combined value is re-scaled and calibrated using the ground radar, disdrometer data, and ground rain observation data [20,34]. The post-real-time product (3B42 calibrated) provides data that is corrected using the monthly ground precipitation data of the Global Precipitation Climatology Centre. These 3B42 calibrated data can be obtained 10 to 15 days after the end of each month, with spatial and temporal resolutions between 50° N and 50° S of 0.25° × 0.25° and 3 h. The 3B42-RT are real time precipitation data, calibrated using the TRMM microwave imager. The spatial and temporal resolutions of the 3B42-RT data is 0.25° × 0.25° between 60° S and 60° N and 3 h. Daily data were obtained by adding eight sets of consecutive 3-h data over a given day from 00:00 UTC. The main difference between 3B42 calibrated datasets is that 3B42-RT precipitation data are not adjusted using ground observations [32]. The spatial and temporal resolutions of the TMPA data used in this study are 0.25° × 0.25° and 24 h. Four satellite precipitation products are demonstrated in Table 2. Table 2. Summary information for the four satellite precipitation products used in this study.

Statistical Metrics
Five statistical properties were calculated to evaluate the performance of the satellite-based precipitation products' estimation of ground observed precipitation, including the correlation coefficient (CC), root mean squared difference (RMSD), mean absolute difference (MAD), relative bias (RBS), and the standard deviation (SD). Compared to the observed reference data, CC represents the similarity of investigated data, RMSD and MAD draws the mean difference between the investigated data and reference data, RBS measures the underestimation or overestimation of the investigated data, and SD reflects the dispersion of the statistical metrics among the stations. These statistical properties are defined, as follows [51,52]: where Yi,obs represents observed precipitation at station i; Yi,sat is the satellite precipitation data at station i; obs Y represents the mean value of observed precipitation at all stations; and sat Y represents the mean value of the satellite-derived precipitation data of all stations; Xi represents the value of the statistical metrics at station i, X is the mean value of the statistical metrics, N is the total number of stations.

Categorical Metrics
Three categorical metrics were used to quantify the capacity of the four satellite precipitation products to detect precipitation, i.e., the probability of detection (POD), a false alarm ratio (FAR), and the Critical Success Index (CSI). In this study, 1.0 mm/day was used as the threshold for precipitation events [2,15,32,50]. The POD calculates the proportion of the precipitation events correctly detected by the satellite-based products. The FAR represents the ratio of the precipitation events falsely detected by the satellite-based products when the observed data have no precipitation. The CSI is a function of both the POD and FAR. The closer that POD and CSI are to 1, and the closer the FAR is to 0, the more accurate the satellite precipitation products are at detecting precipitation [53,54]. These categorical metrics were computed, as follows: where H is the number of observed precipitation events correctly detected by the satellite precipitation data, F is the number of the precipitation events detected by satellite precipitation data, but not observed at stations, and M is the number of the precipitation events detected at stations but not by satellite-derived precipitation data.

Daily Variation
The performance of the four satellite precipitation products was compared at daily intervals. The average, minimum, and maximum values of the statistical metrics for all four satellite precipitation products indicate how they compared with ground observations on a daily scale, as shown in Table 3. The CMFD had the highest value of CC and the lowest values of RMSD and MAD, with average CC, RMSD, and MAD values for all 36 stations of 0.70, 4.64 mm, and 1.19 mm, respectively, followed by the CMORPH and 3B42 calibrated datasets. Meanwhile, the 3B42-RT dataset presented the lowest value of CC and the highest values of RMSD, MAD, and RBS. Moreover, the CMFD had the smallest SD values of the CC, MAD, and RBS compared with the other three satellite precipitation products. The variation in statistical metrics for each product are shown in Figure 2. The spatial distributions of CC and RMSD (as examples) of all four satellite-derived precipitation datasets are shown in Figure 3. Clearly, the performance of the CMFD was better than the other three datasets, meanwhile, the CMFD was more stable than the other three satellite precipitation products.
The detection ability of all four satellite precipitation products, evaluated using average, minimum, and maximum values of the categorical metrics at 36 stations, is shown in Table 4. The CMFD had the largest average values of POD and CSI, and the lowest average value of FAR, with average POD, CSI, and FAR values of 0.97, 0.64, and 0.34, followed by the CMORPH data with average values of 0.87, 0.55, and 0.40. TMPA datasets (both 3B42 calibrated and 3B42-RT) had the lowest detection success. The distribution of POD, FAR, and CSI for all four satellite precipitation products are shown in Figure 4. The detection success of the CMFD was also more stable than all other datasets. Note: minimum (min); maximum (max); correlation coefficient (CC); root mean squared difference (RMSD); mean absolute difference (MAD); relative bias (RBS).

Monthly Variation
Evaluation of satellite detection of precipitation on a monthly scale was examined for individual months, and two hydrological periods, defined as the flood season (June-September) and non-flood season (October-May) of the monsoon climate. The average statistical metrics for monthly precipitation for individual months and for both hydrological periods during the study period (2001)(2002)(2003)(2004)(2005)(2006)(2007)(2008)(2009)(2010)(2011)(2012)(2013)(2014)(2015) are shown in Table 5. The variation in average monthly precipitation values between ground observations and the four satellite precipitation products is shown in Figure 5. Table 5. Average monthly or seasonal statistical metrics for detection of precipitation at 36 stations in Beijing using four satellite-derived precipitation products.  29.80 mm, and 0.12%, and 0.69, 37.78 mm, 29.28 mm, and −2.79%, respectively. For the non-flood season, the CMFD was more accurate than the other precipitation datasets, with the CC, RMSD, MAD, and RBS values of 0.84, 7.76 mm, 5.47 mm, and −15.62%. Moreover, the CMORPH dataset presented the lowest value of CC at monthly scales, despite its reasonable performance at daily scales. As shown in Figure 5, the CMFD and the 3B42 calibrated dataset showed slight overestimation during the non-flood season but underestimation during the flood season, while the CMORPH data produced large overestimation in June and underestimation from September to December, and the 3B42-RT dataset produced overestimation in all months.

CMFD
Generally, the four satellite products captured the temporal trend in monthly precipitation reasonably well. The CMFD had the best performance, especially in non-flood seasons. The 3B42 calibrated dataset also exhibited reasonable accuracy, compared with the CMORPH and 3B42-RT datasets.

Annual Variation
The values of average annual precipitation calculated from ground observations varied from 372.1 to 682.9 mm/year at different stations, with an average annual precipitation at these 36 stations of 508.8 mm/year. The average annual precipitation at these 36 stations for the period 2001-2015, based on (a) ground observations, (b) CMFD, (c) CMORPH, (d) 3B42 calibrated, and (e) 3B42-RT are given in Figure 6. The annual differences between ground observations and each satellite precipitation dataset from 2001 to 2015 are shown in Figure 7.
All satellite precipitation data showed an upward trend from 2001 to 2015, which is consistent with the trend in ground observations. Both CMFD and the 3B42 calibrated dataset had almost the same values as ground observations. In contrast, CMORPH data overestimated values by up to 7.8% of the total precipitation amount. The 3B42-RT dataset also showed significant overestimation by values up to 35.2% of the total precipitation amount, causing this dataset to have the worst performance over the study period.
The spatial distributions of the average annual precipitation over Beijing for ground observations and the four satellite precipitation products, obtained using the ordinary Kriging interpolation method, are shown in Figure 8. The spatial distribution of annual precipitation based on ground observations showed that more precipitation occurred in the northeastern areas, with lower annual precipitation in western areas and southern suburban regions. It reveals that precipitation in the plain areas is greater than in the mountain areas over Beijing, which was concluded by Song et al. and Zhai et al. [55]. This is partially because the topography of Beijing is surrounded by mountains from the southwest to the northeast, and the land-ocean boundary is located to its east [56]. All satellite precipitation products captured higher annual precipitation in the northeast regions, the CMFD, CMORPH, and 3B42 calibrated products showed lower precipitation in southern regions, and the CMFD and TMPA datasets (both 3B42 calibrated and 3B42-RT) presented lower precipitation in the western mountain areas. In contrast, the CMORPH product showed more precipitation in western regions, which were quite different from the observations. Generally, the CMFD and the 3B42 calibrated dataset showed almost the same value as the observations, while the CMORPH and 3B42-RT dataset showed an overestimation of the annual precipitation from 2001-2015. For the spatial distribution of the four satellite precipitation datasets, although four satellite precipitation products captured the trend of more precipitation in the northeastern regions, all satellite precipitation products showed a different distribution from the observations.

Precipitation Intensity
Percentages of days with slight precipitation (P < 10 mm/day), moderate precipitation (10 mm/day < P ≤ 25 mm/day), heavy precipitation (25 mm/day < P ≤ 50 mm/day), and extreme heavy precipitation (P > 50 mm/day) accounted for 13%, 2.7%, 1.0%, and 0.3% of the study period, based on the ground observations, as shown in Figure 9. All satellite products capture more precipitating events than the reference dataset for slight and moderate precipitation. However, with the exception of 3B42-RT, all the satellite precipitation products detected less heavy and extreme precipitation events than the ground measurements in the range of P ≥ 50 mm. The CMFD dataset gave the highest number of days with slight precipitation. The number of days with moderate precipitation derived from CMFD, CMORPH, and 3B42 calibrated data were 11.13%, 23.17%, and 18.94% higher than the observed data. Likewise, the number of days with heavy precipitation derived from CMFD, CMORPH, and 3B42 calibrated data were 15.59%, 6.52%, and 5.97% less than the observed data, while days with extreme heavy precipitation derived from CMFD, CMORPH, and 3B42 calibrated data were 62.13%, 29.08%, and 53.9% less than observations. In contrast, the 3B42-RT dataset showed overestimation of all precipitation events. Figure 9. Probability density function (PDF) of daily precipitation during the study period derived from ground observations, and satellite precipitation products.

Discussion
Our assessment suggests that the CMFD dataset performed better than the CMORPH, 3B42 calibrated, and 3B42-RT datasets at all temporal scales. The performance of the CMFD and 3B42 calibrated datasets were better than the CMORPH and 3B42-RT datasets at monthly and yearly scales. The 3B42 calibrated dataset was integrated into the CMFD dataset as the background field for the precipitation analysis, which means the occurrence of a precipitation event in the CMFD product was determined by the 3B42 calibrated dataset, however, the 3B42 calibrated dataset provides relative few data north of 40° N. The GLDAS precipitation dataset was used as the background field in this region [57,58]. The performance of CMFD was better than the 3B42 calibrated data, which may be attributed to the performance of the GLDAS dataset and the integration of the conventional meteorological observations from the China Meteorological Administration into the CMFD dataset. For the TMPA products, including 3B42 calibrated and 3B42-RT data, the performance of the 3B42 calibrated data was better than the 3B42-RT product, which may be attributed to the 3B42-RT product not having been adjusted using ground observations.
The climate conditions, topography, and urbanization may have a great influence on the spatial distribution of precipitation in Beijing [39]. Previous studies reveal that the topographic elevation influences precipitation, although there is a unique relationship between elevation and precipitation in each place [59,60]. The relationship between the station elevation and annual precipitation in Beijing is shown in Figure 10, with the regression line shown for observations and four satellite precipitation datasets. The observations showed that station elevation is slightly negatively correlated with annual precipitation. The same slightly negative correlation was present between the station elevation and the CMFD dataset, while the CMORPH and TMPA datasets (both 3B42 calibrated and 3B42-RT) presented a slightly positive correlation with the station elevation. Four satellite precipitation datasets showed overestimation of daily precipitation, with average RBS values of −2.06%, −10.55%, −1.41%, and −38.69%. These results may be explained by the precipitation intensity assessment, although CMFD, CMORPH, and 3B42 calibrated data underestimate the heavy and extreme heavy precipitation events, the slight overestimation of CMFD, CMORPH, and 3B42 calibrated data are related to overestimation in light to moderate precipitation events, while the 3B42-RT dataset showed overestimation of all precipitation events, which is also concluded by Qin et al. and Shen et al. [32,35]. The CMFD and the 3B42 calibrated datasets underestimated precipitation during flood seasons and overestimated it during non-flood seasons. This indicates that heavy precipitation volumes are too low in flood seasons, but moderate precipitation volumes are too high in non-flood season, when estimated using CMFD and 3B42 calibrated data, which can cause great differences for the various applications, such as runoff simulation, flood forecasting, etc. Generally, the satellite precipitation data were not sensitive to extreme precipitation events, which is consistent with the findings of Ebrahimi et al. [61]. Urban floods, landslides, and extreme drought disasters caused by the extreme precipitation are posing a threat to the safety of regional society and economy [62,63]. Accurate prediction of extreme precipitation has great significance to human society and ecological environment. Assessment of the satellite precipitation products reveals that there is a large space for improvement in prediction of extreme precipitation for all four satellite precipitation products evaluated in this study.
There are some uncertainties and shortcomings in this study due to the different scales between the satellite precipitation products and reference observations. The satellite precipitation products are the gridded precipitation data, and the observations are the station precipitation data. The satellite precipitation datasets were interpolated to each station using bilinear interpolation in order to match the scale of the ground observations in this study. Although fine resolution satellite precipitation products can be produced through spatial interpolation techniques which are widely used to produce continuous precipitation surfaces, for complex regions the precipitation pattern is greatly affected by altitude. Therefore, using downscaling techniques with emphasis on topography will increase the accuracy of satellite precipitation in small scales. Moreover, it is needed to evaluate the performance of CMFD with other latest satellite precipitation products, such as IMERG.

Conclusions
In this study, we comprehensively assessed four satellite precipitation products, i.e., the CMFD, CMORPH, as well as 3B42 calibrated and 3B42-RT, evaluating their estimates against ground observations from 36 meteorological stations measuring precipitation in Beijing. We compared daily, monthly, and annual precipitation data for the Beijing region.
We found that all four satellite precipitation products captured temporal patterns of precipitation for the period 2001-2015 over Beijing. For the spatial distribution of the four satellite precipitation datasets, although four satellite precipitation products captured the trend of more precipitation in the northeastern regions, all four satellite precipitation products presented different distributions from the observations. Although all satellite precipitation products captured the distribution of precipitation events during the study period, all four products overestimated moderate precipitation events but underestimated extreme precipitation events, except for the 3B42-RT dataset, which overestimated all precipitation events. Specifically, the CMFD performed better than the CMORPH, 3B42 calibrated, and 3B42-RT datasets, having the higher correlation coefficient and lower values of RMSD and MAD at all temporal scales. The average CC of the CMFD, CMORPH, 3B42 calibrated, and 3B42-RT products for all 36 stations were 0.70, 0.60, 0.59, and 0.54 for daily precipitation, and 0.78, 0.32, 0.74, and 0.44 for monthly precipitation. By contrast, the 3B42-RT product showed consistent overestimation of precipitation. Overall, the CMFD data proved to be most suitable for Beijing city.
Author Contributions: M.R. and J.L. designed the technical routes of the study; M.R. analyzed the data and wrote the manuscript; Z.X., B.P., and W.L. proposed suggestions to improve the quality of the paper; L.D. and R.W. provided the observation data.
Funding: This work was financially supported by the National Key R&D Program of China (2017YFC1502701) and the Key Research Projects "Sponge city construction and urban flooding/waterlogging disaster in the subcenter of Beijing City" (Z171100002217080).