Assessment of Ground-Reference Data and Validation of the H-SAF Precipitation Products in Brazil

Abstract

The uncertainties associated with rainfall estimates comprise various measurement scales: from rain gauges and ground-based radars to the satellite rainfall retrievals. The quality of satellite rainfall products has improved significantly in recent decades; however, such algorithms require validation studies using observational rainfall data. For this reason, this study aims to apply the H-SAF consolidated radar data processing to the X-band radar used in the CHUVA campaigns and apply the well established H-SAF validation procedure to these data and verify the quality of EUMETSAT H-SAF operational passive microwave precipitation products in two regions of Brazil (Vale do Paraíba and Manaus). These products are based on two rainfall retrieval algorithms: the physically based Bayesian Cloud Dynamics and Radiation Database (CDRD algorithm) for SSMI/S sensors and the Passive microwave Neural network Precipitation Retrieval algorithm (PNPR) for cross-track scanning radiometers (AMSU-A/AMSU-B/MHS sensors) and for the ATMS sensor. These algorithms, optimized for Europe, Africa and the Southern Atlantic region, provide estimates for the MSG full disk area. Firstly, the radar data was treated with an overall quality index which includes corrections for different error sources like ground clutter, range distance, rain-induced attenuation, among others. Different polarimetric and non-polarimetric QPE algorithms have been tested and the Vulpiani algorithm (hereafter, $R_{q2Vu15}$) presents the best precipitation retrievals when compared with independent rain gauges. Regarding the results from satellite-based algorithms, generally, all rainfall retrievals tend to detect a larger precipitation area than the ground-based radar and overestimate intense rain rates for the Manaus region. Such behavior is related to the fact that the environmental and meteorological conditions of the Amazon region are not well represented in the algorithms. Differently, for the Vale do Paraíba region, the precipitation patterns were well detected and the estimates are in accordance with the reference as indicated by the low mean bias values.

1. Introduction

The knowledge about the distribution of water around the globe is an aspect of extreme relevance for the management of natural resources. The precipitation is, within the hydrological cycle, unanimously recognized as a central component, regulating the energy balance through the interactions of water vapor and clouds, where redistribution of latent heat occurs in the atmosphere. A detailed characterization about precipitation, its formation processes and its life cycle is essential to improve the quality of weather and climate forecasts and also to help decision-makers in their resolutions to be taken in areas affected by the rain.

Due to its inherent complexity, rainfall presents high spatial and temporal variability resulting in different regimes, all these factors combined make the tasks of observation, comprehension and prediction more challenging. The quantification of precipitation, in terms of frequency and intensity, is performed by rain gauge and meteorological radars [single and double polarization] at ground. However, these, in turn, have a sparse and uneven distribution, especially in mountainous regions, forest and over the ocean. Thus, satellite-based precipitation estimates fill these gaps and complement the rainfall observation system.

The quality of satellite rainfall products has improved significantly in recent decades, especially with the advent of satellites/missions such as the Tropical Rainfall Measuring Mission (TRMM) [1] and the Global Precipitation Measurement (GPM) [2]. In addition to the greater number of channels and radiometers available to clouds and precipitation exploitation, the improvements to computational methods and precipitation clouds modeling provided the possibility of development of several rainfall retrieval techniques (i.e, [3,4,5]).

In this context, the EUMETSAT Satellite Application Facility on Support to Operational Hydrology and Water Management (H-SAF) provides rainfall estimations based on infrared and microwave satellite sensors aboard polar and geostationary satellites. On each new Continuous Development Phase (CDOP), new products are released and validation processes are necessary to verify the algorithms performance both in the H-SAF area, as in extra-European regions. The validation of these satellite retrievals is performed by the H-SAF Precipitation Product Validation Group (PPVG) where a common validation methodology has been defined inside the PPVG in order to make those validation results from several institutes comparable and understandable [6,7].

From the validation perspective, the rain gauge networks can provide accurate rainfall information; however they are point measurements and are unevenly distributed around the globe. On the other hand, meteorological radars can provide far better coverage in space and time, especially considering dual polarization radars that provide information about the hydrometeors such as size, shape and variety, which allows for better understanding cloud and precipitation microphysics. Nonetheless, an important issue related to radar systems is the error sources associated with it, which makes challenging the quantitative precipitation estimation (QPE)-based solely on radar, unless these error sources are properly treated. Typically, the error sources that can be identified are: radar calibration, ground-clutter, wet-radome attenuation, rain-induced attenuation, vertical profile of reflectivity (VPR) and non-uniform beam filling. Despite this, polarimetric radars offer new variables that allow greater insight in precipitation and new ways to deal with these error sources [8].

Regarding the quantification of the characteristics of the uncertainties (random components and systematic errors) associated with satellite precipitation estimates, several efforts have been carried out around the world. One example is the CHUVA Project (Cloud processes of the main precipitation systems in Brazil: A contribution to cloud resolving modeling and to the GPM [Global Precipitation Measurement]) that aims to study the clouds and precipitation processes, through six field experiments under different precipitation regimes in Brazil. One of the major goals of the CHUVA project is to evaluate and improve the quality of satellite-based precipitation estimates from the GPM constellation [9]. More about the CHUVA project and its field campaigns can be found online in the CHUVA Project website: http://chuvaproject.cptec.inpe.br.

This study has two main goals: (i) to apply the H-SAF consolidated radar data processing to the X-band radar used in the CHUVA campaigns and; (ii) apply the consolidated H-SAF validation procedure to these data and verify the quality of H-SAF products over specific regions in Brazil. In Section 2, the study areas, radar characteristic and rain gauge distribution are described. In the same section, the methodology for radar data treatment (quality index) and radar-based rainfall algorithms are described. A brief description of satellite algorithms is also provided. In Section 3 the results of the evaluation of rainfall radar estimates with respect to the rain gauges are presented, as well the results from the satellite verification process, where statistical and pixel by pixel evaluations are performed. Finally, a summary of the main results is presented in Section 5.

2. Materials and Methods

2.1. Study Area and Data Sources (Radar and Rain Gauges)

This study was developed based on two CHUVA field campaigns. The first experiment occurred in the Vale do Paraíba region located in the southeastern part of Brazil and the second campaign took place in Manaus city in the northern region of Brazil (centrally located in the Amazon basin). The satellite-based evaluation process considers the precipitating events with largest rain rates for both campaigns. Specifically, the case studies analyzed in Manaus took place on 15, 21, 23, 24, 25, 26 February and 2 and 8 March 2014 (8 days) while in the Vale do Paraíba campaign 6 days were considered: 11, 13 November and 1, 8, 14 and 20 December 2011 for a total of 14 precipitating events. For both campaigns the X-band polarimetric radar, manufactured by Gematronik (Germany) has been employed with the main characteristics: Magnetron with 35 Kw per channel, simultaneous horizontal and vertical polarization, pulse width of 0.5 $\mathsf{\mu }$s, operative pulse repetition frequency (PRF) of 1500 Hz, 1.8 m antenna diameter, 1.3${}^{\circ }$ beam width, operation frequency of 9.375 GHz, 150 m of range resolution and maximum distance of 100 km.

The Vale do Paraíba campaign occurred in São Paulo State in an elevated valley between the Serra da Mantiqueira and Serra do Mar mountain ranges. This field campaign had the longest duration for the CHUVA experiment, with an Intensive Observation Period (IOP) starting from 1 November to 22 December 2011, followed by a second period with less intensive measurements through 31 March 2012. The site strategy is indicated in Figure 1, where the X-band polarimetric radar was installed near São José dos Campos (Lat. 23${}^{\circ }$12${}^{\prime }$31.33${}^{″}$S, Lon. 45${}^{\circ }$57${}^{\prime }$7.87${}^{″}$W, 650 m ASL) above the roof of the UNIVAP building (Vale do Paraíba University) being approximately 82 km inland from the ocean. The radar scanning strategy produced a volume scan with 13 elevations (varying from 1 to 25 degrees) with a scan time of 6 min. Seven measurement sites (called UNIVAP, CTA, IEAV, Jambeiro, CESP, Pousada and Caragua) were established and equipped with rain gauges located at 9, 11, 22, 43, 51 and 75 km from the radar site, respectively, along a perpendicular line towards the ocean (Figure 1).

The main rainfall systems that were observed during the campaign were caused by the penetration of cold fronts, local convection and organized mesoscale systems. The presence of a 500 hPa trough to the east (first half of the period) and west (second part) was responsible for the atmospheric moisture flow over the region and, consequently, the precipitation regime. Several thunderstorm events, some associated with hail, were reported during the campaign [10].

During the Manaus campaign, the X-band polarimetric radar was installed in the Amazon rainforest about 60 km from Manaus, between the Negro and Amazon rivers (Lat. 3${}^{\circ }$12${}^{\prime }$46.86${}^{″}$S, Lon. 60${}^{\circ }$35${}^{\prime }$53.92${}^{″}$W, 69 m ASL). The X-band radar scanning strategy produced one volume scan with 15 elevations (varying from 0.5 to 30 degrees) every 10 min. Two measurement sites (called T3 and Manacapuru) were established and equipped with rain gauges. T3 is located in the same position of the radar site while Manacapuru is approximately 10 km from the radar site (Figure 2). The field campaign occurred in two IOPs, the first happened from 13 February to 31 March 2014 during the wet season, and the second one between 1 and 30 September 2014 at the end of the dry season.

The austral winter corresponds to the dry season in most of the Amazon region, although it represents a rainy maximum for the far northwest of the basin [11]. Well-defined wet and dry seasons are associated with the so-called South American Monsoon System (SAMS) [12].

2.2. Radar Data Quality Index

To perform a reliable validation procedure, the common validation methodology developed by the H-SAF Precipitation Product Validation Group (PPVG) was applied to the CHUVA radar data. The proposed scheme aims to compensate or, at least, minimize or eliminate those uncertainties. Moreover, a quality indicator for each source of error was introduced through appropriate tests. These quality matrices are composed by partial indexes that will be part of an overall data quality indicator as shown in Figure 3. In this figure, the main error sources taken into account are shown: ground clutter, partial beam blocking (PBB), range distance, non-uniform vertical profiles of reflectivity (VPR), differential phase processing and rain induced attenuation [8].

Any contamination of the radar signal by non-precipitation echoes, including returns from ground is considered in the $q_{clutter}$ index. In order to eliminate these effects, the raw volumetric data is treated trough a filtering technique based on a fuzzy logic approach. More details about this methodology can be accessed at [13]. The Figure 4 shows an example of clutter correction applied for an event occurred on 1 December 2011 in the Vale do Paraíba campaign.

Another type of contamination is when the radar beam intercepts an obstruction. In this case, two situations are possible: (1) only part of the beam cross section illuminates the intercepted topography (partial blockage), or (2) the radar beam is completely blocked (total blockage). The beam blockage filtering approach $q_{PBB}$ applied in this study is described in [14]. An example is shown in Figure 5 of the visibility map obtained at the lowest elevation angle.

The radar data quality decreases with the beam broadening with the distance r from the radar. This effect cannot be corrected, however the data range-related deterioration can be determined quantitatively and taken into account in the $q_{distance}$. Following the approach proposed by [15] we adapted the equation from [6] considering q = 0.5 for r ≥ $r_{max}$ to avoid an abrupt decrease in the quality index. The value 134 (km) is set so that for r = 100 (the radar range) we obtain q = 0.5 for r ≥ $r_{max}$. It was also introduced a square root operation and the $q_{distance}$ index can be calculated using a non-linear function, as in following Equation (1):

$$q_{\mathrm{distance}}=\left\{\begin{array}{cc}0.5\hfill & \mathrm{for}r\ge r_{max}\hfill \\ 1\hfill & \mathrm{for}r\le r_{min}\hfill \\ \sqrt{{\displaystyle \frac{134-r}{134-r_{min}}}}\hfill & \mathrm{for}{r}_{min}<r<r_{max}\hfill \end{array}\right.$$

(1)

where $r_{max}$ can be set to 100 km and $r_{min}=\Delta r/2$ ($\Delta r$ is the radar range resolution). The $r_{max}$ is the radius of radar domain, the maximum range is 100 km. Range resolution is the ability of a radar system to distinguish between two or more targets on the same beam but at different ranges. It depends mainly on the width of the transmitted pulse and for this radar it is 150 m ($\Delta r$ = 150 m). The square root is introduced in order to ensure that the quality does not drop too fast as the range distance increases.

Reflectivity values within the layer close to the 0${}^{\circ }$ isotherm (referred to as melting layer or bright band) are overdetermined since it consists mainly of water-coated non-Rayleigh scatterers. Because of that, the most important parameter that defines the non-uniform vertical profiles of reflectivity (VPR) is the height of the freezing level (FL). The $q_{VPR}$ is estimated following the Friedrich approach and taking into account the freezing level height, the layer between 200 m above the FL and 500 m below it, the beam width and the antenna elevation [15].

A polarimetric radar system provides measurements of the total differential phase ${\mathsf{\Phi }}_{DP}$ that is the sum of the differential propagation ${\mathsf{\Phi }}_{DP}$ and the backscatter phase $\delta$hv. Only the propagation component is considered for attenuation correction and for rainfall estimation purposes, with $K_{DP}$ being related to the range derivative of ${\mathsf{\Phi }}_{DP}$, which is affected by system noise, offset, and potential aliasing problems. In order to handle this issue ($q_{noise}$), an Iterative moving-window range Finite Derivative scheme (IFD) approach is used and can be summarized through flow chart of Figure 6 and Equation (2). More information about this scheme can be accessed at [13].

$$q_{\mathrm{noise}}=\left\{\begin{array}{cc}1\hfill & \mathrm{if}{K}_{DP}\ge 0.5\hfill \\ 2K_{DP}\hfill & \mathrm{if}0<K_{DP}\le 0.5\hfill \\ 0\hfill & \mathrm{if}{K}_{DP}<0\hfill \end{array}\right.$$

(2)

Usually, there are a variety of possible solutions for attenuation correction in dual-polarization radars. All of them are based on the use of differential phase shift [16]. In the common polarimetric approach, for compensating rain path attenuation, the specific attenuation (AH) assumes a linear relationship with the specific differential phase ($K_{DP}$) [17]. This approach is applied for frequencies ranging from 5 GHz to at least 19 GHz, which includes both the C (5.6 GHz) and X-band (9.4 GHz) radars. The quality index $q_{Att}$ associated with rain path attenuation can be defined as Equation (3):

$$q_{\mathrm{att}}=\left\{\begin{array}{cc}1\hfill & \mathrm{for}PIA<PI{A}_{min}\hfill \\ 0.5\hfill & \mathrm{for}PIA>PI{A}_{max}\hfill \\ {\displaystyle \frac{2PI{A}_{max}-PI{A}_{min}-PIA}{2(PI{A}_{max}-PI{A}_{min})}}\hfill & \mathrm{for}PI{A}_{min}\le PIA\le PI{A}_{max}\hfill \end{array}\right.$$

(3)

where $PI{A}_{min}$ = 3 dB and $PI{A}_{max}$ = 15 dB.

Finally, the overall radar data quality Q can be retrieved by combining all the considered quality indicators. All the partial quality matrices are used in a multiplicative combination. Figure 7 shows an example of the attenuation correction map in the Vale do Paraíba and Manaus campaigns. Additional examples of the overall quality index will be shown in Section 3.

2.3. Rainfall Estimation from Radar Data

Beyond the quality index procedure described in the previous section, another important aspect to be taken into account is that there are different algorithms in use for radar-based rainfall retrievals. Traditionally, the simplest rainfall relation is the Z-R relation, where rainfall (R) is estimated from reflectivity (Z). With the availability of polarimetric variables, it is possible to exploit several combinations (e.g., Z and $Z_{DR}$, among others) towards an optimal rainfall rate estimation. From the reflectivity (Z), the rainfall can be calculated by means of a power-law type relationship in Equation (4):

$$R\left(Z_H\right)=a{Z}_H^b$$

(4)

where R is rainfall rate in mm ${\mathrm{h}}^{-1}$ and $Z_H$ in ${\mathrm{mm}}^6{\mathrm{m}}^{-3}$ units, while a and b are two coefficients whose value depends on the type of precipitation. The use of polarimetric quantities clearly provides better results in terms of precipitation estimation, being able to employ more than the reflectivity $Z_H$ also the specific phase $K_{DP}$ and the differential reflectivity $Z_{DR}$ in various combinations. For the precipitation estimation, the R-Z and the R-$K_{DP}$ relationships were adopted. Indeed, the algorithms that estimate rainfall from $K_{DP}$ are particularly attractive at wavelengths such as X-band because they are derived from phase measurements and they are unaffected by absolute calibration error and attenuation caused by precipitation along the propagation path [8]. The R-$K_{DP}$ relationship can be written as Equation (5):

$$R\left(K_{DP}\right)=c·\left(\right|K_{DP}{\left|\right)}^d·sign\left(K_{DP}\right)$$

(5)

For the parameterization of $R\left(Z_H\right)$ and $R\left(K_{DP}\right)$ (Equations (4) and (5)), the approach from [18] (MP48) and from CHUVA campaign in Fortaleza [19] (SC12) were used respectively. Another pair of coefficients is adopted for R-$K_{DP}$ model, using [8] (BC01). Additionally, the parameters for $R\left(Z_H\right)$ and $R\left(K_{DP}\right)$ are summarized in

Table 1. Parameters of R($Z_H$) and R($K_{DP}$) power law relations for the CHUVA X-band Radar.

	R(${\mathit{Z}}_{\mathit{H}}$)			R(${\mathit{K}}_{\mathit{DP}}$)
	a	b		c	d
MP48	0.0208	0.680	SC12	15.813	0.774
			BC01	19.193	0.850

.

The use of radar reflectivity for radar-based rainfall retrievals is frequently subject of underestimation [6], especially when the lowest beam map (LBM) is used. In order to reduce this effect, an additional polarimetric rainfall algorithm was proposed by Vulpiani [20]. The technique is based on the combination of reflectivity factor and specific differential phase and considers two types of weighting. The first consideration uses the $K_{DP}$ gradually with increasing rainfall intensity, the combined algorithm takes the form of a weighted sum as Equation (6):

$$R_q=q_{\mathrm{noise}}·R_K+(1-q_{\mathrm{noise}})·R_Z$$

(6)

where $R_Z$ and $R_K$ are the rainfall estimates obtained by applying specific power laws (Equations (4) and (5)) to the lowest non-shielded radar bin of $Z_H$ and $K_{DP}$, respectively. The coefficients that were adopted to derive $R_K$ and $R_Z$ are shown in Table 1, while the weight is the quality index $q_{noise}$ (Equation (2)). The second consideration uses a combined polarimetric rainfall algorithm with another type of weighting between the contributions of $Z_H$ and $K_{DP}$, as Equation (7):

$$R_q=\frac{q_{\mathrm{loss}}·R_Z+q_{\mathrm{noise}}·R_K}{q_{\mathrm{loss}}+q_{\mathrm{noise}}}$$

(7)

where $R_K$ and $R_Z$ parameters are the same of Equation (6), while the weights are the quality indexes $q_{loss}$ ($q_{loss}$ = $q_{pbbx}\times q_{att}$) and $q_{noise}$ indicated in Equation (2). Therefore, the different algorithms used with the various coefficient combinations are summarized as follow:

• $R_Z$: (4) with MP48 coefficients.
• $R_{K1}$: (5) with SC12 coefficients.
• $R_{K2}$: (5)with BC01 coefficients.
• $R_{q1}$: (7) with MP48 and SC12 coefficients.
• $R_{q2}$: (7) with MP48 and BC01 coefficients.
• $R_{q1Vu15}$: (6) with MP48 and SC12 coefficients.
• $R_{q2Vu15}$: (6) with MP48 and BC01 coefficients.

The results related with the different algorithms will be shown in Section 3.

2.4. Satellite Products and Dataset Generation

The passive microwave (PMW) precipitation products within the EUMETSAT H-SAF ([21]) are based on the development and refinement of retrieval techniques exploiting all available radiometers in the GPM constellation. In this context, operational PMW precipitation products for the different radiometers are being released within H-SAF. They are based on two approaches ([22]): the physically based Bayesian Cloud Dynamics and Radiation Database (CDRD) algorithm ([23,24]) for conically scanning radiometers and the Passive microwave Neural network Precipitation Retrieval algorithm (PNPR) for cross-track scanning radiometers ([25,26]). Three PMW H-SAF products were considered in this study: H01 (CDRD approach applied to SSMIS), H02 (PNPR developed for AMSU/MHS), and H18 (PNPR adapted to ATMS).

The algorithms are based on the use of a cloud-radiation database made up of thousands of microphysical-meteorological profiles derived from cloud-resolving model simulations of different precipitation events including 60 simulations over the European/Mediterranean area ([23]) and 34 simulations over Africa and Southern Atlantic. The main features of each product can be accessed in detail in the respective references cited above.

To perform the validation over Brazil, we had to acquire all the input data (brightness temperatures in TB) for the the CHUVA experiments timeframe, and then, to process the H-SAF products to extend the coverage of the retrievals in order to include the whole country (extended to 75${}^{\circ }$N–60${}^{\circ }$S and 80${}^{\circ }$W–80${}^{\circ }$E). The inputs for the H01 were the SSMI/S orbits of the DMSP F16, F17 and F18 satellites. For the H02 algorithm, the input files were the AMSU-A, AMSU-B and MHS orbits from NOAA-18, NOAA-19, MetOp-A and MetOp-B. In addition, lastly, we used the ATMS orbits from the Suomi-NPP satellite as input for the H18 algorithm. Though we have analyzed the three products: H01, H02 and H18 for the Manaus campaign, we only analyzed the H01 and H02 products for the Vale do Paraíba campaign because the ATMS data was not available in 2011.

2.5. Application of the Common Validation Code (CVC)

The common validation code (CVC) developed by the Precipitation Product Validation Group (PPVG) [7] enables implementation of a common validation procedure to make the validation results comparable. The products to be validated differ in terms of retrieval technique, spatial and temporal resolutions. Therefore, each product requires a specific validation procedure. The methodology can be divided on the following general steps:

• Ground data error analysis;
• Upscaling of radar data to match the satellite product nominal resolution;
• Temporal matching of precipitation products (satellite and ground);
• Application of evaluation statistical methods (continuous and multi-categorical) to all available overpasses, for each pixel pair (satellite-ground).

In order to use the CVC on the Brazilian radar data, the code had to be adapted by including the new radar coordinates (Vale do Paraíba and Manaus) and changing the reading routine to the new radar data. The CVC was configured to match a maximum temporal difference between satellite and radar of 16 min. The radar data was upscaled to the satellite product nominal resolution, considering the antenna pattern (Gaussian function), viewing geometry, and scanning strategy (conical and cross-track) of the MW radiometers. As the radar data was filtered beforehand (quality control), the pixels with low quality were eliminated in the upscale processing. To investigate the performance of the precipitation products, the statistical scores commonly used in the pixel-based validation by the H-SAF PPVG were considered.

3. Results

In this section, we present the results of the evaluation of rain gauge measurements and the performance radar-based rainfall algorithms for one case study of Vale do Paraíba and Manaus, respectively. Sequentially, we performed the validation of the H01, H02 and H18 algorithms, breaking it down into two phases, the statistical evaluation and pixel-by-pixel analysis. In order to summarize the results for all case studies, we chose to exhibit one representative event for each campaign; however, the results of the remaining cases will also be discussed further ahead.

3.1. Vale do Paraíba Campaign: 13 November 2011

This precipitating event refers to the occurrence of extensive convection followed by stratiform rainfall. Figure 8 shows the 24-h accumulation (from 00:00 to 24:00 UTC) derived from the distinct rainfall algorithms considered in this study. Each map corresponds to a specific rainfall technique and includes the cumulative values recorded by rain gauges on the left of each map. In comparison to rain gauges, the radar QPE based on $R_Z$ estimator (Figure 8a), shows clear signs of underestimation, especially regarding the convective cores. On the other hand, the $K_{DP}$-based algorithms have a similar pattern to those recorded by rain gauges.

3.2. Manaus Campaign: 8 March 2014

This event was characterized by extensive and intense stratiform precipitation in Manaus. Figure 9 presents the different algorithms with the rain gauge records indicated at the bottom-left of each map. Similarly of the Vale do Paraíba event the algorithm just based on reflectivity ($R_Z$) also presented a general underestimation of 24-h accumulation (Figure 9a) and the $R_{q2Vu15}$ presented the best performance (Figure 9g).

A performance verification of the radar-based rainfall retrievals was made regarding the rain gauges (figures and tables were not shown here). The comparison considering the nearest value shows better results. In general, the radar-based rainfall $R_Z$ (just based on reflectivity) presented underestimation, the $K_{DP}$-based algorithms had good performance when compared with rain gauges measurements. The best estimator was the $R_{q2Vu15}$, because it presented the perfect correlation coefficient score, which is one. And bias, mean absolute error and root mean square not higher than 0.11, 0.13 and 0.41, respectively.

3.3. Validation of H-SAF Precipitation Products

The second part, which is the main focus of this study, is to identify the performance of the H-SAF products in order to provide the algorithm developers with information on the limitations and issues of the retrievals over specific regions in Brazil. The case studies under analysis are the same 14 cases (for both campaigns) that were considered previously on the radar data quality analysis. As mentioned before, three satellite rainfall products were analyzed: H01 (CDRD approach applied to SSMIS), H02b (PNPR v1 developed for AMSU/MHS), and H18 (PNPR v2 adapted to ATMS). We present a statistical analysis based on continuous and dichotomous statistical scores computed within the CVC, along with an in-depth analysis of the selected cases, followed by a pixel by pixel analysis. The considerations concerning the acquisition of the matching pairs (radar × satellite) are exposed in Section 2.5.

3.3.1. Statistical Evaluation

Because of its best estimation performance, the $R_{q2Vu15}$ algorithm was chosen for the statistical evaluation. In order to investigate the impact of different quality indexes in the radar retrievals, we performed a sensitivity analysis (not shown here). According to this analysis, we found that a quality index value equal or greater than 0.7 could be considered a good compromise between the desired performance of the statistical scores and the size of the dataset (that guarantees the reliability of the results), i.e., the number of pixels in the sample. The statistical scores used in the validation procedure are presented in

Table 2. Continuous statistical scores.

Score	Perfect Score	Calculation
Mean error or bias (ME)	0	${\displaystyle ME=\frac{1}{N}\sum _{k=1}^n(sa{t}_k-ob{s}_k)}$
Standard deviation (SD)	0	${\displaystyle SD=\sqrt{\frac{1}{N}\sum _{k=1}^n{(sa{t}_k-ob{s}_k-ME)}^2}}$
Root mean square error (RMSE)	0	${\displaystyle RMSE=\sqrt{\frac{1}{N}\sum _{k=1}^n{(sa{t}_k-ob{s}_k)}^2}}$
Fractional standard error (FSE)	0	${\displaystyle FSE=\frac{\sqrt{\frac{1}{N}{\sum }_{k=1}^n{(sa{t}_k-ob{s}_k)}^2}}{\frac{1}{N}{\sum }_1^{n}ob{s}_k}=\frac{RMSE}{\frac{1}{N}{\sum }_1^{n}ob{s}_k}}$
Correlation coefficient (CC)	1	${\displaystyle CC=\frac{{\sum }_{k=1}^n(sat-\overline{sat})(obs-\overline{obs}))}{\sqrt{{\sum }_{k=1}^n{(sat-\overline{sat})}^2}\sqrt{{\sum }_{k=1}^n{(obs-\overline{obs})}^2}}}$

and

Table 3. Multicategory scores.

Score	Perfect Score	Calculation
Probability of Detection (POD)	1	${\displaystyle POD=\frac{hits}{hits+misses}=\frac{hits}{observedyes}}$
False alarm rate (FAR)	0	${\displaystyle FAR=\frac{falsealarms}{hits+falsealarms}=\frac{falsealarms}{forecastyes}}$
Critical success index (CSI)	1	${\displaystyle CSI=\frac{hits}{hits+misses+falsealarms}}$

. The continuous statistical scores were computed for the pixels in which both radar and satellite give rainfall estimates larger than 0.25 mm/h (hits only). On the other hand, all pixels are considered for the multicategory scores. The number of pixels for each algorithm is different because it depends on the number of available satellite overpasses.

For Manaus, the events under analysis counted a total of 33 overpass matchings (which means correspondence in time and space for both the satellite and radar measurements) for H01, 49 matchings for H02 and 13 matchings for H18. Since ATMS is aboard a single satellite, the number of overpasses over the region of interest is less than the other sensors (consequently, lower number of matched pixel pairs).

Table 4. Statistical continuous scores for Manaus for H01, H02 and H18. The number inside parenthesis represents the number of matched pixel pairs for each algorithm.

Algorithm	Sat. Mean	Rad. Mean	ME	RMSE	SD	FSE	CORR
H01 (406)	4.60	0.93	3.67	5.20	3.69	5.59	0.40
H02 (631)	3.78	1.18	2.59	5.11	4.40	4.32	0.32
H18 (140)	4.53	1.46	3.07	5.10	4.07	3.48	0.34

presents continuous scores for the algorithms in analysis, the number inside the brackets refers to the number of matched pixel pairs for each algorithm. All algorithms tend to overestimate the X-band radar estimates. The H01 presented the largest values for ME, RMSE and FSE. In general H02 had slightly better scores than H01, such as lower mean rainfall rate values, ME, RMSE, SD and FSE. Finally, the H18 presents intermediate scores in relation to the other algorithms, having just the smallest FSE. Regarding multicategory scores (

Table 5. Multicategory scores for Manaus for H01, H02 and H18. The number inside parenthesis concerns the number of matched pixel pairs for each algorithm.

Algorithm	POD	FAR	CSI
H01 (3149)	0.75	0.54	0.39
H02 (1889)	0.96	0.47	0.51
H18 (916)	0.81	0.39	0.53

), it is worth noticing that both algorithms based on the neural network approach presented better detection skills than H01, the highest POD (0.96 and 0.81), the lowest FAR (0.47 and 0.39) and highest critical CSI (0.51 and 0.53), respectively.

The Vale do Paraíba campaign counted a total of 27 overpass matchings for H01 and 31 matchings for H02. The respective statistical scores are presented in

Table 6. Continuous scores for Vale do Paraíba for H01 and H02. The number inside parenthesis concerns the number of matched pixel pairs for each algorithm.

Algorithm	Sat. Mean	Rad. Mean	ME	RMSE	SD	FSE	CORR
H01 (181)	2.53	2.89	−0.35	4.00	3.98	1.38	0.44
H02 (124)	2.45	2.76	−0.30	3.56	3.55	1.29	0.37

and

Table 7. Multi-category scores for Vale do Paraíba for H01 and H02. The number inside parenthesis concerns the number of matched pixel pairs for each algorithm.

Algorithm	POD	FAR	CSI
H01 (2420)	0.49	0.49	0.33
H02 (991)	0.43	0.21	0.39

. We can see from Table 6 that H01 and H02 algorithms presented similar mean rain rates, slightly lower than the radar estimates, thus determining small and negative values for ME. These results can be attributed to either the better performance of the satellite products for this region (at least in terms of bias), or the better quality of the ground-radar estimates over this region. Compared to Manaus region, the multicategory scores for Vale do Paraíba presented lower FAR values (0.39 for H01 and 0.21 for H02), lower POD values (0.49 for H01 and 0.43 for H02) and similar CSI values

3.3.2. Pixel by Pixel Analysis

In order to further analyze the algorithms performance, a case study analysis for each product and each campaign, with pixel by pixel comparison, was carried out.

A case study for Manaus, which occurred on 21 December 2014 was presented, with the overall quality index shown in Figure 10a and the radar rainfall field in its original resolution in Figure 10b. On this day, a well defined squall line approached the X-band radar region. The systems persisted on the region from the morning around 08:20 UTC to the afternoon 15:40 UTC. It is possible to see the presence of convective cores with rainfall rate upwards to 35 mm/h in the regions between the north-west and the south-west quadrants in Figure 10b. The remaining regions are dominated by light and stratiform precipitation with rainfall rates going up to 6 mm/h. In Figure 11a it is possible to see the filtered radar data by applying the quality index threshold at 0.7 and upscaled to the satellite native grid, and in Figure 11b the respective H01 rainfall retrieval is shown. The light rain rate provided by the radar (light blue), is associated with moderate precipitation values by H01 (shades of green). Additionally, the moderate rain rates from radar (green and dark blue) is strongly overestimated by the H01 (orange to red) algorithm. We can say that H01 has a general tendency to overestimate all rainfall classes.

For H02, one overpass for the case which occurred on 8 March 2014 (at 05:30 UTC) is shown. Convective cells are visible along the border of the Rio Negro River, followed by smaller convective cells over the Southeast quadrant (Figure 12). Figure 13a,b show that the H02 algorithm provides precipitations from light to moderate values (shades of green) in pixels where the radar detects absence of precipitation (gray pixels). This tendency to produce a large area of precipitating pixels is related to the precipitation screening, which is the algorithm module where potential precipitating pixels are selected. This feature is related to the relatively high FAR scores (shown in Table 7).

The example for H18 shown in Figure 14 and Figure 15 is for the same event presented in H02, on 8 March 2014 but at 05:10 UTC. Despite the fact that the satellite overpasses (H02 and H18) were very close in time (20 min of difference) it was enough for modifications in the precipitation field to appear. The H18 precipitation pattern is similar to H02 because they are based on the same precipitation screening method. It is worth pointing out that the screening of precipitation is equal for all products in analysis in this study and it is applied over all background surfaces, except over desert. It is very likely that the screening procedure is not well tuned for the atmospheric conditions of the Amazon region, characterized by the high water vapor content. On the other hand, in the area with the most intense convective cores, the H18 tends to produce a lower overestimation rate when compared to H02. This aspect is related to the difference in the two retrieval algorithms (PNPR for AMSU/MHS and PNPR for ATMS) (as pointed out by [26], where H18 is based on just one neural network for all surface types, trained with a unique database with the additional channels in the water vapor absorption band at $T{B}_{183\pm 3}$ ([26]).

It is noticeable that the three algorithms have a tendency to overestimate the larger rain rates. The study in reference [27] evaluated the ability of two GPM rainfall algorithms (GPROF2014 and IMERG) in reproducing the main characteristics and the diurnal cycle of precipitation as observed by the S-band SIPAM radar, in the Manaus region during the CHUVA campaign. The authors have found similar results where GPROF2014 (GMI) presents large overestimation of the rain rate volume and occurrence greater than 10 mm ${\mathrm{h}}^{-1}$ during IOP1.

The first example for the Vale do Paraíba campaign refers to the presence of convective clusters associated with the SACZ (South Atlantic Convergence Zone). The coincident overpass between the satellite and the radar occurred at 21:18 UTC and shows the presence of a few intense convective cells along the Rio Negro River (Figure 16b). The overall quality index (Figure 16a) in this region is most affected by the blocking effects caused by two mountain ranges, the Serra da Mantiqueira (western) and Serra do Mar (eastern). Analyzing the upscaled maps (Figure 17) it is clear that, differently from Manaus, the precipitation pattern is well detected by the H01. Concerning the rainfall estimation, in this case, the tendency of the algorithm is to underestimate the highest precipitation rates seen by the radar. Despite this feature, the estimates are in better agreement with the radar than the estimates in the Manaus campaign (as shown by the lower ME for this region in Table 6).

The second example for the Vale do Paraíba campaign (Figure 18 and Figure 19) consists of the occurrence of local convection with sparse intense convective cells occurring predominantly in the afternoon (Figure 18b). During this event, the precipitation pattern is also quite close to the ground-based reference, which is also reflected on the lowest FAR values (Table 7). Concerning the rainfall estimation, in this case, the algorithm tends to underestimate the highest precipitation rates observed by the radar. However, the PMW estimates are in better agreement with the radar observations than the estimates generated in the Manaus campaign (as indicated by the lower ME for this region in Table 6).

4. Discussion

We will discuss the main features of the H01 algorithm that mostly influence the results in Brazilian areas: the precipitation screening process, the surface classification and the representativeness of the cloud model simulations in the a priori database.

The first aspect, the precipitation screening process, selects the potentially precipitating pixels before the retrieval process. The screening used in the algorithms is based on the method described by [28], which uses the comparison of the TBs, which are the water vapor absorption band at $T{B}_{183\pm 3}$ and $T{B}_{183\pm 7}$ GHz and the $T{B}_{53}$ GHz channel in the oxygen absorption band around 50 GHz. A detailed description of the implementation of the H-SAF products screening process can be accessed in [21,25].

Figure 20 present the radiative signatures from the $T{B}_{183\pm 3}$, $T{B}_{183\pm 7}$ and $T{B}_{53}$ GHz channels for Manaus for the case study on 21 February 2014. Analyzing the radar area, we can notice that in Amazon region (Figure 20a,b) both water vapor absorption channels present a strong TB decay (minimal values around 160–180 K). This pattern seems to be associated with a change in the weight function peak of these channels in the presence of high water content, typical of the Amazon rainforest. As a consequence, an extensive area of precipitation can be seen in the screening map and in the final rainfall rates (Figure 20d,e). This can affect the efficiency of the screening in identifying potential precipitating pixels, especially when the clouds are not very deep as they appear to be during the wet season in the Amazon region.

On the other hand, for the Vale do Paraíba region (Figure 21), the channels 183 ± 7 and 183 ± 3 GHz do not present strong decay in TB inside the radar area and the screening map (Figure 21d) follows the same pattern that was observed in 183 ± 7 GHz channel ((Figure 21b). Differently from Manaus, we can see that the retrieval does not generate precipitation (gray pixels in Figure 21e) in the entire area provided by the screening as potentially precipitating (indicated by red in Figure 21d).

In the Amazon region we have noticed that the surface characteristics were not properly classified. This region is dominated by large inland water bodies (Amazonas and Rio Negro rivers) and surrounded by vegetated land. Figure 22a shows that, in the radar area, the algorithm classifies the surface as coastal pixels, which is the least populated database, therefore, the least representative and usually affected by larger uncertainties. In the Vale do Paraíba region (Figure 22b) the algorithm considers large part of the radar area as vegetated land and the Bayesian approach look for land profiles, which leads to the choice of more appropriate rain rate profiles.

In the Bayesian approach, the surface classification impacts the choice of channels used in retrieval processing, which influences the selection of hydrometeor profiles in the a priori database and, ultimately, the rainfall rate retrieval.

Finally, the representativeness of the cloud model simulations in the a priori database, which is currently optimized for Europe and Africa, does not properly represent the typical microphysical and rainfall profiles of brazilian regions. Improving the database including simulations that represent the brazilian rainfall regimes could solve the issue.

5. Conclusions

This work analyzed the data collected by mobile X-band polarimetric radars during two campaigns of the CHUVA project, both in the Vale do Paraíba and Manaus regions. First, in order to ensure that the verification of the H-SAF precipitation products is based on well-treated data, the ground radar data was submitted to a quality control procedure and the rainfall estimates were tested with different algorithms. The current polarimetric processing chain attempted to correct the error contributions of the different sources of uncertainty and provide the estimation of the reflectivity ($Z_H$) and the specific differential phase ($K_{DP}$), which contain the microphysical information required to perform attenuation correction and quantify the precipitation rate. Different polarimetric QPE algorithms have been coded and evaluated at a hourly time-step using independent rain gauges. The algorithms that were tested are: simple Z-R relationship (Marshall-Palmer, 1948 [18]) indicated as $R_Z$, algorithms based on $K_{DP}$ solely ($R_{K1}$ and $R_{K2}$), algorithms based on combination of $K_{DP}$ and $Z_H$, whose weights are the quality indexes q${}_{loss}$ and q${}_{noise}$ ($R_{q1}$ and $R_{q2}$), and algorithms based on combination of $K_{DP}$ at medium-high $K_{DP}$ (above 0.5${}^{\circ }{\mathrm{km}}^{-1}$) and attenuation-corrected $Z_H$ at low $K_{DP}$ (below 0.5${}^{\circ }{\mathrm{km}}^{-1}$) ($R_{q1Vu15}$ and $R_{q2Vu15}$). The results confirmed the benefits brought by polarimetry to quantify radar rainfall retrievals and seems to indicate (considering the problems of a possible radome attenuation, which are not considered here), that the algorithm which best estimates the precipitation intensity was $R_{q2Vu15}$. Even though $R_{K2}$ also provided good results, $R_{q2Vu15}$ had better performance for both low and for high precipitation rates.

Regarding the results from the satellite algorithms validation, for the Manaus region, the CDRD algorithm (H01) tends to overestimate all rain rates classes (light to heavy). The PNPR algorithm for AMSU-A/AMSU-B/MHS sensors (H02) presents better POD than H01, but it also presents high FAR values. The PNPR for the ATMS sensor (H18) presents lower overestimation of heavy rain rates when compared to H02, probably due to the different neural network used on H18. It is worth considering that the sample size for this verification study (14 cases) was quite small, and a more extensive validation with a larger ground-based dataset would be suggested to perform a more comprehensive quality assessment.

All analyzed rainfall retrieval algorithms for the Amazon region showed high FAR values and larger precipitation patterns which are deeply related to the precipitation screening scheme. The screening seems to be substantially affected by the high water vapor content in this region. The H01 (CDRD for SSMIS—Bayesian approach), in specific, it was affected by highly variable surface emissivities, impacting the selection of hydrometeor profiles in the a priori database and, ultimately, the rainfall rate retrieval. Moreover, we must take into account the uncertainties on precipitation estimates, which in the Bayesian approach are represented by the coefficients of error covariance matrices (for TBs, and ancillary and meteorological parameters) as analyzed in [24] ) and in [29]. For the Vale do Paraíba region, both algorithms, H01 and H02, produced ME values that were quite close to zero (or negative) and lower FAR values (from 0.21 to 0.49) than Manaus. Unlike the Manaus case, the precipitation patterns were well detected and the estimations were in good agreement with the reference as indicated by the low ME values.

As for future plans, we intend to perform new verifications in a denser and more comprehensive sample for other Brazilian regions. Since we identified limitations in the precipitation screening scheme over the Amazon region, we plan on developing a new scheme that is able to handle the water vapour content in this region. That would require, furthermore, work on the development of a new surface classification capable of characterizing the surface diversity properly. The next steps must be made using the pluviometric precipitation dataset provided by the National Institute for Space Research (CPTEC/INPE) and also the dataset from new radars acquired by the National Center for Natural Disaster Monitoring and Alerts (CEMADEN). Lastly, for future work we intend to extend the cloud-radiation database used as a priori information in the algorithms in order to obtain radiative and microphysical profiles representative of brazilian rainfall regimes.