Validation of NOAA CyGNSS Wind Speed Product with the CCMP Data

The National Aeronautics and Space Administration (NASA) Cyclone Global Navigation Satellite System (CyGNSS) mission was launched in December 2016, which can remotely sense sea surface wind with a relatively high spatio-temporal resolution for tracking tropical cyclones. In recent years, with the gradual development of the geophysical model function (GMF) for CyGNSS wind retrieval, different versions of CyGNSS Level 2 products have been released and their performance has gradually improved. This paper presents a comprehensive evaluation of CyGNSS wind product v1.1 produced by the National Oceanic and Atmospheric Administration (NOAA). The Cross-Calibrated Multi-Platform (CCMP) analysis wind (v02.0 and v02.1 near real time) products produced by Remote Sensing Systems (RSS) were used as the reference. Data pairs between the NOAA CyGNSS and RSS CCMP products were processed and evaluated by the bias and standard deviation SD. The CyGNSS dataset covers the period between May 2017 and December 2020. The statistical comparisons show that the bias and SD of CyGNSS relative to CCMP-nonzero collocations when the flag of CCMP winds is nonzero are –0.05 m/s and 1.19 m/s, respectively. The probability density function (PDF) of the CyGNSS winds coincides with that of CCMP-nonzero. Furthermore, the average monthly bias and SD show that CyGNSS wind is consistent and reliable generally. We found that negative deviation mainly appears at high latitudes in both hemispheres. Positive deviation appears in the China Sea, the Arabian Sea, and the west of Africa and South America. Spatial–temporal analysis demonstrates the geographical anomalies in the bias and SD of the CyGNSS winds, confirming that the wind speed bias shows a temporal dependency. The verification and comparison show that the remotely sensed wind speed measurements from NOAA CyGNSS wind product v1.1 are in good agreement with CCMP winds.


Introduction
Sea surface wind is an important physical parameter in oceanography, as it regulates the spatial distribution of air-sea fluxes of heat, moisture, gases, momentum, and other physical parameters, thus determining and maintaining the air-sea interactions and atmospheric dynamics [1][2][3][4]. In the process of ocean dynamics, sea surface wind is the most influential factor in the generation of ocean hydroynamic phenomena such as wind waves. It not only plays a role in global and regional ocean circulation, but also in weather forecasting and other atmospheric and oceanographic science studies [5][6][7][8]. Consequently, enhancement in the spaceborne GNSS-R sea wind retrieval algorithms, the performance of CyGNSS product has been improved [30,31,[58][59][60]. Ref. [61] stated that the accuracy of the NASA CyGNSS Level 2 baseline surface ocean winds product by buoys is less than 2 m/s over the tropical and subtropical oceans [61]. Some evaluations of CyGNSS v1.0 wind produced by the NOAA have been compared against HWRF and ECMWF model winds, and remote sensing sea surface wind of scatterometers and radiometers, which demonstrated that the NOAA CyGNSS wind product has achieved its goals of consistency, reliability, and repeatability [62].
The validation of the NOAA CyGNSS wind product is important for its further applications in different areas. This paper reports a probe into the performance of the NOAA CyGNSS wind product against the CCMP one. A quantitative temporal and spatial comparison was implemented for the period between May 2017 and December 2020. The structure of this paper is as follows: The next Section provides a detailed description of the dataset and the processing method. In Section 2, the statistical comparison and validation results between the CyGNSS and CCMP wind on global scale are presented. A preliminary discussion is presented in Section 4. Section 5 draws the conclusions.

Data Set Description and Data Processing
In this study, the data sets of the RSS CCMP analysis wind product and NOAA CyGNSS v1.1 wind products were used. Brief descriptions of the data and method are provided below.

RSS CCMP Wind Product
The CCMP wind product was used to evaluate the performance of NOAA CyGNSS wind product. As an analysis product, the ECMWF ERA-Interim reanalysis winds are taken as the first-guess wind field. CCMP simultaneously integrates multisatellite sea surface winds of scatterometers from QuikSCAT and METOP-A/ASCAT; radiometers from SSM/I, SSMIS, AMSR, TMI, WindSat, and GMI; and in situ observations from NDBC, TAO, TRITON, RAMA, PIRATA, and Canadian [26,63]; and has its own unique characteristics. The spatial and temporal resolutions of CCMP are 0.25 • and 6 h (00:00, 06:00, 12:00, and 18:00 UTC), respectively. Considering the continuity of wind field in space, multiple wind speed observations are usually sufficient to determine an accurate wind vector [64]. For each cell of CCMP, if there is no measured data (the nobs flag is zero), the background wind field is adopted. Such data from CCMP are named CCMP-zero. Otherwise, they are named CCMP-nonzero in this paper (Equation (1)). CCMP wind analyses data including U-and V-components were downloaded from Remote Sensing Systems (Available online: www.remss.com (accessed on 31 January 2021)), covering the period 2017-2020. Note that the CCMP v02.0 product during 2017-2018 and the near real-time Version-02.1 (v02.1.NRT) product during 2019-2020 were acquired for this study. Previous studies showed that the CCMP wind speed is closer to conventional in situ measurements from ships than ECMWF and NCEP-CFSR products [65]. CCMP winds and Tropical Atmosphere Ocean (TAO) mooring observations were compared, which showed good agreement with an root mean square error (RMSE) of 1 m/s and a correlation coefficient of 0.95 [66]. The CCMP wind product is a newly released global ocean wind dataset and suitable for scientific study at various temporal and spatial resolutions. We used the CCMP sea surface wind speeds to assess the performance of the NOAA CyGNSS product on the global scale.

NOAA CyGNSS Wind Product
The Cyclone GNSS (CyGNSS) mission was designed to observe ocean winds by a constellation of eight microsatellites with high spatial and temporal resolutions for quickly tracking tropical cyclones [56,67]. In this study, the quality of CyGNSS sea surface wind speed operational product reproduced by NOAA was assessed. The NOAA's dataset is available at (https://manati.star.nesdis.noaa.gov/, (accessed on 31 January 2021)). The NOAA CyGNSS wind v1.1 product was used for comparison with the CCMP gridded surface wind analysis data product. Early studies of CyGNSS v1.0 and v1.1 retrievals and validation against ECMWF and ASCAT scatterometry observations were accomplished [62,68]. The standard deviation (SD) of overall CyGNSS v1.1 decreased from 1.33 to 1.19 m/s against ECMWF. Table 1 lists the wind speed requirements for the CyGNSS mission. The designed accuracy of CyGNSS mission is <2 m/s or <10% in RMSE for wind retrievals. Prior to launch, the mission design of the satellites and science payloads referred to these requirements. Once in orbit, they can be used to evaluate whether the mission is successful [69].

Requirement Performance
Retrieval uncertainty for winds < 20 m/s 2 m/s Retrieval uncertainty for winds > 20 m/s 10%

Along Track Retrieval Algorithm for NOAA CyGNSS
Considering the sensitivity of CyGNSS observation to the information of waves that are not generated by local winds, NOAA's scientific team improved the performance of CyGNSS wind speed retrieval using a priori knowledge of the wave height as the input parameter [70,71]. In addition, NOAA's Along Track Retrieval Algorithm (ATRA) is based on wind speed (U 10 ), angle of incidence (θ), the significant wave height (Hs), and CyGNSS σ 0 , as shown in Equation (2). ATRA also uses track-wise processing to reduce the systematic errors due to the uncertainties of the transmitted power and receiver instrumental effects. A wind-wave GMF is presented for retrieving the sea surface wind speed along the track of CyGNSS. A detailed description of this algorithm is provided in [70].

Spatial and Temporal Collocation
Both CyGNSS and CCMP winds are provided at the height of 10 m above the sea surface in neutral conditions. Before matching, quality control was applied to remove poor-quality samples of NOAA CyGNSS according to quality flags (Equation (3) [72]). If quality_ f lags = 0, then the wind speed samples were considered of good quality.
where sample_ f lags is the variable in the NOAA Level 2 CyGNSS NetCDF file, MOD(sample_ f lags, 2) is the function that returns the modulus after the division of sample_ f lags by 2, and quality_ f lags is the final quality of the flags used in this paper. As sea surface states differ over time [5], it is necessary to ensure that the acquisition time of the CyGNSS measurement is close to that of CCMP wind speed. The spatial and temporal windows are the collocation criteria for ensuring the consistency of ocean winds observed by different sources. The CCMP winds were collocated with CyGNSS specular point using the closest cell in space and time. The following describes the detailed procedure of collocating CyGNSS and CCMP. First, considering the consistency of wind in time, the CyGNSS specular points were selected by a temporal window within ±5 min relative to the time of CCMP. Then, the nearest grid was located according to the center of a CyGNSS specular point's longitude and latitude. Finally, the CyGNSS/CCMP collocations were obtained from the CCMP product. To compare and analyze the CyGNSS wind product, we extracted the sea surface wind speed that had the flag with nonzero observation numbers (nobs > 0) from CCMP. To compare the spatial distribution, CyGNSS winds were divided on a regular geographical grid of 1 • by 1 • , and the metrics (bias and SD) were calculated against by the CCMP wind speed in every grid cell ( Figure 1). In this study, the deviation (Bias) and SD were the metrics used to quantitatively describe and evaluate the CyGNSS/CCMP collocations. Their definitions are as follows: where x is NOAA CyGNSS v1.1 wind speed, y is CCMP wind speed, z is the difference between CyGNSS and CCMP, z is the mean z, and n is the number of CyGNSS/CCMP collocations.

Comparisons between CyGNSS and CCMP Winds
In this study, we used CCMP v02.0 (2017-2018) and v02.1.NRT (2019-2020) winds to assess the accuracy of the NOAA CyGNSS v1.1 product in global oceans. Here, we present the comparison of the CyGNSS/CCMP collocations during the period between May 2017 and December 2020. For illustrative purposes, Figure 2 shows the distribution of CyGNSS/CCMP collocations in global oceans. It can be seen that the area from 38 • S to 38 • N can be observed by CyGNSS due to the limitation of orbit [73]. The detectable area depends on the specular point of the associated global positioning system (GPS) transmitting satellite and CyGNSS receiving satellite [74]. The CCMP wind was regarded as the groundtruth observation, and was used to evaluate the performance of the NOAA CyGNSS wind product. CCMP wind was collocated with each CyGNSS specular point on the basis of the spatial and temporal collocation criteria. After the CyGNSS/CCMP collocating procedure, the total number of matchups was ∼7.45 million for May 2017-December 2020. According to the flag named nobs from CCMP as mentioned above, CyGNSS/CCMP collocations were divided into CCMP-zero and CCMP-nonzero as shown in Figure 1. Figure 3 shows the volume of collocation data over the global oceans during the period between May 2017 and December 2020. The percentages per calendar month are displayed in each bar; the number of collocations is relatively lower in 2018 overall. The average percentage was 70.2% from 2017 to 2020. We found that the percentage of CCMP collocations with observation data flags was more than 60% in each month. This provide valuable collocations for us to assess the accuracy of the NOAA CyGNSS v1.1 product. The overall performance of NOAA CyGNSS v1.1 compared with CCMP was evaluated in this study. Scatterplots of the wind speeds between CyGNSS and CCMP are presented in Figure 4, which show the global statistics for CyGNSS winds against the reference CCMP winds in the form of bias and SD. The average SDs of CyGNSS/CCMP and CyGNSS/CCMP-nonzero were 1.32 m/s and 1.19 m/s during the period between May 2017 and December 2020, respectively. This result is consistent with the analyses in [62], which reported that NOAA CyGNSS v1.1 against ECMWF achieved an SD of 1.19 m/s for wind speed. Figure 4b shows that the highest density in the wind speed scatterplot (2-10 m/s) is very close to y = x (magenta line), which is in agreement with the CCMP-nonzero wind speed. However, compared with CCMP winds, as shown in Figure 4a, the outliers still remained in the red circle, as shown in Figure 4b. The first reason may be the stability of the NOAA retrieval model, although a large amount of data was used for fitting. It could be possible that the outliers highlighted in Figure 4b are associated to low signal-to-noise ratio (SNR) levels (e.g., below 3 dB). The quality control may not be so strict, leading to some poor-quality data being used for inversion.  Figure 5 illustrates that the averaged metrics for wind speed ranging from 2 to 25 m/s were calculated according to the CCMP-nonzero wind speeds. The biases are close to zero, associated with small oscillations, and SDs are within 2 m/s in the range of 2 to 12 m/s, which shows that CyGNSS wind is very consistent with the CCMP-nonzero wind. However, average bias decreases with the increase in the CCMP-nonzero wind speed ranging from 12 to 25 m/s. We noted that the CyGNSS retrievals showed a large error for high wind speeds, which may be GNSS-R observations become relatively insensitive to high wind. Moreover, high wind is a very small proportion of the total, which produces instability when modeling the empirical model. To analyze the features of the error distribution, the whole collocated set was divided into three subsets according to the magnitude of CCMP-nonzero wind speed. In the low wind speed regime, the wind speed is lower than 4 m/s; in the medium wind speed regime, the wind speed is between 4 and 20 m/s; wind speed greater than 20 m/s was classified as high wind speed. A total of ∼15.4% of the whole collocated set was classified as low wind speeds. The average bias was calculated by collocations between 0 and 4 m/s according to the magnitude of the CCMP-nonzero wind speed, which was −0.03 m/s. Correspondingly, the average SD was 0.92 m/s. In the high speed regime (which included only 0.03% of the whole collocated set), Figure 5 shows a large negative mean deviation when the CCMP-nonzero wind speed is greater than 20 m/s. We clearly see that it decreases linearly from 0 to −5 m/s with the increase in CCMP-nonzero wind speed. Lastly, in the medium wind speed range, the bias and SD on average were −0.06 m/s and 1.23 m/s, respectively, showing agreement with CCMP-nonzero and good performance. According to the overall accuracy (bias = −0.05 m/s and SD = 1.19 m/s), the sea surface wind speeds can be effectively provided by the NOAA CyGNSS product.
The performance of the NOAA CyGNSS product was also analyzed over the period between May 2017 and December 2020. Figure 6 provides the metrics of CyGNSS/CCMP collocations for each individual month. Compared with CCMP-zero winds, the CyGNSS winds had a persistent positive deviation of 0.10 m/s. The overall bias of CyGNSS/CCMPnonzero was −0.05 m/s. However, the SD of CyGNSS/CCMP-zero was large at 1.57 m/s. The comparison shows that NOAA CyGNSS winds are in good agreement with CCMPnonzero winds. In addition, from Figure 6, the monthly SDs of CyGNSS/CCMP-zero were all bigger than those of CyGNSS/CCMP-nonzero from 2017 to 2020. The monthly bias of CyGNSS/CCMP-zero in 2018 was higher than in other years. On the contrary, the monthly bias of CyGNSS/CCMP-nonzero was lower. Subsequently, both monthly biases appear to have stabilized in 2019 and 2020. The time series of bias shows a discrepancy in the earlier period. There may be two reasons for this finding: One is that the block IIR-M and most IIF GPS experienced power flex events in 2018. As a result, the CyGNSS observations estimated from these are questionable. So, the performance was degraded. Another reason is that both CCMP v02.0 (2017-2018) and CCMP v02.1.NRT (2019-2020) products were used to evaluate the accuracy of the CyGNSS sea surface wind speed in global oceans. There will be a difference between the two products.   Figure 7, the CCMP-zero winds are shown to be slightly higher than NOAA CyGNSS winds at moderate wind speeds. Figure 7b indicates the PDF of CyGNSS winds coincides with that of CCMP-nonzero winds. The performance of CyGNSS winds relative to CCMP-nonzero ones is significantly better, with a SD of 1.19 m/s on average. In general, the CyGNSS winds are in agreement with CCMP-nonzero winds. The NOAA CyGNSS v1.1 product has approximately the same wind speeds with CCMP-nonzero wind: the PDF curves almost coincide, consistent with the result in Figure 7b.

Global Statistics of CyGNSS/CCMP-Nonzero
The geographical distribution characteristics of CyGNSS in terms of bias and SD were investigated. CyGNSS/CCMP-nonzero collocations were divided on a regular geographical grid of 1 • by 1 • . The relevant information of the re-gridded map method is described in Section 2.4. A global map with spatial resolution of 1 • gridded SD was constructed as shown in Figure 8, which illustrates the SD of NOAA CyGNSS values compared with CCMP-nonzero values in the global oceans. A large part of the global 1 • gridded map in SD is blue, which indicates that NOAA CyGNSS winds well-agree with CCMP-nonzero ones. Islands have little influence on retrieval accuracy, e.g., near Malaysia and the Philippines. However, we found that large SDs occur in the Mediterranean Sea, the southern coast of Africa, and the east coast of China. These errors should be further investigated. Figure 9 shows the distribution of the wind speed bias between CyGNSS and CCMPnonzero. We found that negative deviation mainly appears at high latitudes in both hemispheres, and especially in the southern hemisphere. The bias is obvious when the latitude exceeds 30 • S. The negative deviation of the northern hemisphere mainly occurs in the Northern Pacific and the north center of the Atlantic Ocean, especially along the east coast of the United States and Australia. However, positive deviations appear in the China Sea, Arabian Sea, and the west of Africa and South America. The red density points reveal where CyGNSS winds overestimate CCMP-nonzero winds for every ocean between 20 • S and 20 • N, as shown in Figure 9. In particular, the quality of CyGNSS winds in the China Sea is not ideal, with a large SD and a positive bias. In addition, as shown in Figure 8, the map of the wind speed SD between CyGNSS and CCMP-nonzero shows the high accuracy, and the wind speed bias has a small deviation in the adjacent sea region to Malaysia and the Philippines.  Sampling points with poor inversion accuracy were mostly distributed along the coast, as shown in Figures 8 and 9. The wind speed retrieval performance of NOAA CyGNSS in coastal regions was also analyzed. Coastal winds (within 50 km from the coastline) of CyGNSS/CCMP-nonzero were determined using the global coastline dataset (http://www.soest.hawaii.edu/pwessel/gshhg/, (accessed on 31 January 2021)) [75]. The accuracy of CyGNSS winds in coastal regions was evaluated: a bias of −0.19 m/s and an SD of 1.45 m/s were obtained. This performance is not as good as the overall performance. The main reason for this degradation is that GMFs are established by global winds, so these are inapplicable of retrieving coastal winds. The wind field from the coastal regions is strongly affected by shoreline and shallow water, which increase the difficulty of retrieval of winds by GMFs.
The counts, SD, and bias of NOAA CyGNSS values with respect to CCMP-nonzero ones over the tropical Pacific, Atlantic, and Indian Oceans were investigated and analyzed. The number of collocations in each ocean depends on its size. The accuracy evaluation of the Pacific, Atlantic,and Indian Oceans is shown in Figure 10. The results showed that the difference in the Pacific, Atlantic, and Indian Oceans was non-significant in bias. The mean deviation of the Indian Ocean was −0.05 m/s with an SD of 1.22 m/s; the mean deviation of the Pacific Ocean was −0.04 m/s with an SD of 1.17 m/s; the mean deviation of the Atlantic Ocean was −0.09 m/s with an SD of 1.19 m/s. Figure 10 also shows the histogram of NOAA CyGNSS winds relative to CCMP-nonzero ones over the tropical Pacific, Atlantic, and Indian Oceans. We see this clearly in Figure 10, where the biases of the Pacific, Atlantic, and Indian Oceans are concentrated between −3 and 3 m/s. The mean deviations for each ocean are negative, which demonstrates the overall underestimation in CyGNSS wind speed over the tropical Pacific, Atlantic ,and Indian Oceans. The performance in the Atlantic Ocean is the worst. The number of collocations can also affect the accuracy. Figure 10 provides the counts of the tropical Pacific, Atlantic, and Indian Oceans. The more likely reason for this finding is that this difference in the three oceans may be determined by their own wind field characteristics, such as the occurrence and geographical distribution of high wind events. Certainly, these differences could also be due to the performance of the CCMP product in different ocean basins. The geographical dependence of the wind speed retrieval error should be further investigated.

Discussion
The performance verification of wind speed retrieval is a key criterion for operational products. At present, according to our comparison result, the NOAA CyGNSS v1.1 product meets the design performance requirements for wind speed accuracy across the wind speed range from 2 to 20 m/s at a 25 km resolution. The results described in this paper are consistent with those described in [68], whereas the performance in the high wind regime troubles. CyGNSS always underestimates high wind speeds. Maybe GNSS-R observations are relatively insensitive to high wind. Moreover, high wind represents a very small proportion of the total, which poses difficulties for modeling empirical models. So, the retrieval performance at high wind speeds should be further investigated.
As a result of the aforementioned comparison and analysis, the overall performance consistency and reliability were determined. However, abnormal geographical anomalies were found. From Figure 9, we found that negative deviation mainly appears in high latitude areas in both hemispheres. Positive deviations appear in the China Sea, Arabian Sea, and the west of Africa and South America. In particular, a large error (both SD and bias) was found in the coastal areas of China. The aforementioned phenomenon occurred in the previous version as well. Both NASA v2.1 and NOAA v1.0 wind products show large errors relative to ECMWF reanalysis data in the coastal areas of China, which was presented in a technical report [76]. A recent study showed that the overall performance of CyGNSS v1.1 against ECMWF also displays this phenomenon. Said et al. discussed and explained this phenomenon, which may be caused by radio frequency interference [68] .
To further determine whether the geographical anomalies are temporally temporary, the interannual variations in the SD and wind speed bias between CyGNSS and CCMPnonzero were investigated, as shown in Figures 11 and 12. The SD of CyGNSS along the China coast has been improved over time, with the SD (red color) gradually becoming small (white color). Unfortunately, the accuracy remains poor (red color) in the Mediterranean region and the middle-upper region of the Pacific Ocean. The distribution of SD between CyGNSS and CCMP-nonzero in other regions is similar to the average SD in 2018-2020. Figure 12 shows that there was no significant difference in the spatial distribution of the wind speed bias from 2018 to 2020. Note that the global distribution of the wind speed bias in 2017 showed larger errors occurred in the China Sea and Arabian Sea. This is consistent with the result shown in Figure 9. The improvement in the wind speed bias is not as obvious as that in SD over time. Negative deviations mainly appeared in high latitudes in 2017-2020, as shown in Figure 12, especially in the southern hemisphere. This basically confirms that the geographical anomaly is not temporary. The specific reasons may be related to the distribution of wind speed and the retrieval method, which will not be discussed in detail here.

Conclusions
This paper presented a preliminary assessment of the quality of remotely sensed wind speed measurements produced by NOAA CyGNSS. The performance of CyGNSS v1.1 winds between 38 • S and 38 • N was investigated. Compared with CCMP-zero and nonzero collocations, we found that NOAA CyGNSS winds agree well with CCMP-nonzero winds that combine in situ sea surface wind speed observations. A bias of nearly −0.05 m/s and an SD of 1.19 m/s were obtained against CCMP-nonzero winds during the period from May 2017 to December 2020. Further analysis indicated that the wind retrievals substantially degrade for wind speeds greater than 20 m/s and are most accurate in the 2-12 m/s wind speed regime. Furthermore, we systematically compared the NOAA CyGNSS and CCMPnonzero winds on spatial and temporal scales. The results of time series analysis illustrated that the CyGNSS wind is consistent and reliable generally. Additionally, the geographical distribution characteristics of the CyGNSS winds were investigated using the interannual variations in the SD and wind speed bias. We found that the geographical anomaly in bias was not temporary. Moreover, evaluation of coastal CyGNSS winds (within 50 km from the coastline) showed that its performance was not as good as the overall performance.
The metrics of CyGNSS/CCMP-nonzero were calculated in the Pacific, Atlantic, and Indian Oceans. We found the NOAA CyGNSS wind in the Pacific Ocean performed better than in the Atlantic and Indian Oceans. It can be seen from the metrics of three oceans that the difference in the three oceans may be determined by their own wind field characteristics. Overall, the quantitative comparison in this study indicates that the CyGNSS wind v1.1 product provides good and stable performance over time. The sea surface wind product acquired from the CyGNSS mission in 2017-2020 is valuable for practical use.