One of the requirements for ensuring food security is a timely inventory of agricultural areas and the regional proportion of different crop types [1
]. Next to the public sector, the private sector, including the agro- and insurance industries, benefit as well from early season crop inventories as an important component of crop production estimation and agricultural statistics [2
]. In addition to regional estimates, early crop type information at the parcel level is also an essential prerequisite for any crop monitoring activity that aims at early anomaly detection. Satellite remote sensing has proven to be an invaluable tool for accurate crop mapping in regions across the world [4
The most conventional way of performing satellite-based crop classifications is the use of optical imagery. This has started with the Landsat missions [8
]. Gradually, the spatial, temporal, and spectral resolutions have increased, constantly improving the classification results [11
In June 2015, Sentinel-2 was launched as part of the Copernicus Sentinel-2 mission [15
]. This mission consists of a constellation of two identical satellites, Sentinel-2A and Sentinel-2B. The revisit frequency of one satellite is 10 days, resulting in a revisit frequency of 5 days for the constellation. Considering overlap between adjacent orbits, the coverage frequency in Europe is often even below 5 days. Sentinel-2 carries a MultiSpectral Instrument (MSI) with 13 bands in the visible, near infrared, and shortwave infrared part of the spectrum. These state-of-the-art specifications have been used in recent research on crop classification [16
However, clouds remain a major drawback for optical sensors, since they cannot be penetrated by optical radiation. Clouds and cloud shadows, therefore, lead to gaps in optical imagery and missing data in optical time series [19
]. For classification and monitoring purposes, this drawback significantly affects performance [6
Among possible strategies to overcome the disadvantages of clouds and shadows are multisensor combinations, especially strategies that exploit different parts of the electromagnetic spectrum [21
]. Microwave radiation in the C-band, for example, has wavelengths well above the typical size of cloud particles and can therefore penetrate clouds mostly unaltered [22
]. Satellite sensors exploiting this part of the spectrum need to send out their own energy pulse and subsequently measure the reflected energy. A synthetic aperture radar (SAR) is such a system that uses the motion of the instrument to attain a satisfactory ground resolution [22
Although the potential of SAR observations from space has been demonstrated extensively, their availability in the past was restricted to specific campaigns [23
]. However, with the advent of the Sentinel-1 mission, SAR applications have become more widespread, with almost global data availability at no charge and regular time intervals [25
]. The revisit frequency is 6 days now that both the Sentinel-1A as well as the Sentinel-1B satellites are operational. Considering overlap and combining ascending and descending orbits, this leads to a Sentinel-1 observation every ~2 days in Europe.
Contrary to optical imagery that provides insights into biophysical processes of vegetation, SAR imagery reflects the structural components and moisture content of vegetation and the underlying surface, resulting in a complementary data source that can be used for classification purposes [27
]. Figure 1
shows Sentinel-1 radar backscatter profiles in addition to Sentinel-2 normalized difference vegetation index (NDVI) observations for a winter wheat field in Belgium. The radar response to the structural development of the wheat plants is clear, with for example a pronounced decrease in VV backscatter during the vertical stem elongation phase.
The rich information on plant structure that is present in SAR backscatter amplitudes has been used for classification purposes in previous studies, in particular for rice and forest mapping [28
], but more recently also for broader crop type mapping [31
The complementarity of optical and radar information allows the development of multisensor classification procedures that exploit both sources simultaneously [21
]. This has led to numerous successes in integrated SAR/optical land use classifications [32
]. In addition to these broad land use classes, previous studies have also shown how radar and optical imagery can be jointly used for crop type identification. Blaes et al. [37
] reported at least 5% accuracy gain for crop type discrimination when adding ERS and RADARSAT radar imagery on top of optical SPOT and Landsat images. Mcnairn et al. [38
] successfully used several RADARSAT-1, Envisat ASAR, SPOT-4/5, and Landsat-5 images for annual crop inventories, while Soria-Ruiz et al. [39
] used RADARSAT-1 and Landsat and reported higher accuracies for land use classifications in cloudy regions in Mexico, considering a few major crop classes. Based on combined Sentinel-1 and Landsat-8 data, Inglada et al. [40
] specifically quantified the added value of joint radar/optical imagery for early crop type identification, pointing towards future potential for integration of Sentinel-2 imagery. Zhou et al. [41
] also combined Sentinel-1 and Landsat-8 but focused on the detection of winter wheat in China. Recently, Sentinel-1 was also shown to improve the detection of irrigated crops in India, especially during the monsoon season [42
]. For the agricultural landscape in the United States, recent work by Torbick et al. [43
] has demonstrated within-season crop inventories based on near real-time (NRT) Sentinel-1, Sentinel-2, and Landsat-8 imagery.
However, despite the existing efforts to combine optical and radar imagery for crop type identifications, few studies exist to date that (i) combine dense time series of Sentinel-1 and Sentinel-2, (ii) classify a broad range of crop types on country scale, and (iii) provide insights into classification confidence and performance along the growing season.
The aim of this study was therefore to address these shortcomings and to provide further evidence of the added value of joint radar and optical imagery for crop type identification. We used dense time series of Sentinel-1 backscatter intensity and Sentinel-2 NDVI during the growing season of 2017 over Belgium to address the following research questions:
What is the pixel-level classification accuracy in function of the input source (optical, radar, or a combination of both)?
What is the evolution of classification accuracy during the growing season and how can this be framed in terms of crop phenology?
What is the individual importance of each input predictor for the classification task?
What is the classification confidence and how can this information be used to assess the accuracy?
We first describe the relevant data sources used and the preprocessing routines for Sentinel-1 and Sentinel-2 imagery. Next, a hierarchical classification methodology is discussed in view of different classification schemes. Finally, the results are discussed in light of crop type differences across the growing season.
Crop classification requires uniform inputs across the whole region. For Sentinel-1, this was accomplished by creating 12-day backscatter mosaics in which special care was taken to reduce incidence angle effects on the results. An enhanced Lee filter was used to clean the radar images, although radar speckle unavoidably remained part of the signal. However, by considering 12-day backscatter mosaics and using time series as input features instead of single-date images, the impact of radar speckle could be minimized. The use of 12-day backscatter mosaics could somehow limit the methodology for NRT applications, but this is not an issue in classification tasks that are not time critical such as in this study. For NRT applications, a multitemporal filtering approach such as that proposed by Quegan et al. [68
] could be considered as well, although this was not part of the present study.
For Sentinel-2, this was realized by smoothing all cloud-free NDVI observations at the pixel level and creating 10-daily cloud-free NDVI mosaics. These time series inputs were used in a hierarchical random forest classification to yield a crop map for the whole of Belgium. The hierarchical two-step procedure resulted in a 1.5% OA gain over a nonhierarchical approach in which all classes were predicted in one step. This indicates that performing a first classification step to distinguish among a broad crop class and the other classes was helpful in the classification procedure, albeit with a small accuracy gain. Indeed, OA of this first classification step at the end of August was 98%, owing to the distinct radar and optical signatures of the three noncrop classes (forest, built-up, and water) against one major crop class consisting of more similar radar and optical signatures. The second classification step then allowed a specific training of the random forest classifier to detect differences between the crop types.
OA and kappa clearly increased along the growing season. However, typical summer crops such as potato, maize, and sugar beet well emerged only by the end of May. Winter cereals, in turn, were already well developed by then. This led to classification accuracies of particular crop types deviating considerably from the OA. For example, winter cereals were mapped against the other crops by the end of May with a user’s accuracy of 96% and a producer’s accuracy of 71%, while potato was mapped against the other crops with a user’s accuracy of 86% and a producer’s accuracy of only 40%. This demonstrates how the accuracy within the growing season depends strongly on the evolution of the crop’s phenological stages. These insights agree with previous research showing that using optical time series which include phenological differences in classification tasks has a positive impact on the performance [14
]. The ability of discriminating winter cereals earlier in the season compared to summer crops could in principle be used in future work to discriminate winter cereals early in the growing season, as this might be helpful for some applications. However, further identification amongst these crops (e.g., winter wheat vs. winter barley) is at this moment in the growing season hardly possible due to their strong similarities. A discrimination of winter cereals early in the season might also lead to error propagation of this early discrimination layer later on in the season. It is therefore generally better to perform the winter cereals classification simultaneously with the other crops.
From the confusion matrix in Table 3
and Table 4
, it became apparent that specific confusion occurred between winter cereals and grassland on the one hand and among the different summer crops on the other hand. The former could be explained by the fact that both winter cereals and grassland appear green early in the season and both start growing in April–May, causing confusion amongst these classes. The latter could be explained by the fact that all summer crops (maize, potato, beet) appear very similar until the end of April due to their comparable bare soil conditions or presence of indistinguishable seedlings. These are physical causes for confusion that cannot be easily solved in a satellite-based classification task.
The increasing OA and kappa during the season lead to a tradeoff: where the quality of the classification results improved with time, the timeliness of the results decreased. Our analysis provided insight into this tradeoff and may help choose the appropriate time in the growing season for a specific application where crop classification is needed. We found that early in the season, the OA of a classification based on Sentinel-1 was higher than a similar classification based on Sentinel-2 (at the end of March 47% and 39%, respectively). However, after only one month in the season, the amount of predictor variables varied considerably between Sentinel-1 and Sentinel-2, with 12 predictors for the former (both VV and VH 12-daily backscatter mosaics in ascending and descending passes) and only 3 predictors for the latter (three 10-daily NDVI mosaics). The higher accuracy for radar-only classification could therefore also be due to the amount of input features. Indeed, when limiting this analysis only to VH backscatter for Sentinel-1, OA dropped to 36%, which is in fact even lower than the Sentinel-2 NDVI classification. Based on our analyses, no clear conclusion could therefore be made with regard to the performance differences of exclusive radar vs. exclusive optical classification early in the season.
Limited performance of the optical-only classifications could be attributed to the use of NDVI as the only predictor. NDVI is not a full descriptor of Sentinel-2 optical imagery, although it has been used successfully in the past [69
]. It is therefore often beneficial to use more bands, especially when exclusively using optical imagery [70
]. However, the least-squares smoothing procedure as outlined in Section 2.1.3
was not designed to work on the individual spectral bands of optical sensors, but on vegetation indices such as NDVI. Hence, the 10-daily smoothed data set cannot be realized using individual spectral bands without introducing new challenges with regard to country-wide mosaicking of individual acquisitions, and thereby jeopardizing the ability to closely monitor crops during the growing season. Therefore we focused on 10 m resolution NDVI as a well-known vegetation index, while the addition of other optical metrics could be subject to future research. This is also the case for radar-only classification which we solely based on backscatter intensity mosaics. It has already been shown in the past that other derived radar products can increase classification accuracies, such as stacks of interferometric coherences [71
]. It should therefore be subject to future research whether an optimized classification based on radar only (including more radar predictors) performs better early in the season than an optimized classification based on optical only (including more optical predictors), which would provide answers to our hypothesis that radar images might be more helpful early in the season due to their sensitivity to early crop stages during an in general cloudy part of the growing season.
Considering the feature importance in the final classification task at the end of August (Figure 8
), we found that the NDVI predictors show two distinct periods of increased importance. In April and May, the winter cereals showed rapid stem elongation and some of the summer crops started emerging, enhancing their separability compared to bare soil or seedling conditions. In addition, this period is in general less cloudy than the preceding months, leading to higher-quality optical images. July and August were found to be important months for optical acquisitions as well, since the period of winter cereal harvesting and peak productivity of the summer crops also led to increased separability of the different crop types. Regarding SAR predictors, we found no clear difference in importance between the two polarization channels (VV and VH), which is somewhat contradictory to Inglada et al. [40
], who found VV to be more important for their classification than VH. Considering joint Sentinel-1 and Sentinel-2 predictors, it became apparent that the most important features were based on key NDVI acquisitions, while the other important features were a mixture of optical and radar predictors, stressing the added value of a combined optical/radar classification.
The classification confidence was shown to provide helpful insights at the pixel level with regard to the quality of classification. Parcel borders were typically characterized by lower classification confidence than the center parts of the parcels. Figure 9
demonstrated the strong correlation between estimated classification confidence and final classification accuracy, suggesting that this information can be qualitatively used to assess specific classification performances. However, it was also mentioned that this confidence is not equal to a statistical probability and should therefore not be used to infer any statistical conclusion on classification accuracy.
Our results clearly showed the advantage of using in addition to optical Sentinel-2 observations complementary Sentinel-1 SAR observations that give insight into the plant structural components [27
]. An advantage of the approach implemented here is the application of dense image time series over single-date images avoiding the need for manual date selections which would hamper its use in an operational context [73
]. Several recent studies that used combined radar and optical imagery reported higher accuracies compared to this work. One of the reasons is that, while some of these studies were segment or field-level classifications [74
] or implemented some kind of postclassification filtering routine [38
], our approach was an unfiltered pixel-wise classification, which is generally characterized by lower OA values. In addition, classification of 11 different classes, of which 8 are crop types, is a challenging task for a classifier and might explain lower accuracies compared to other studies that focus on a few specific classes, such as very accurate winter wheat [41
], paddy rice [77
], and maize and grassland mapping [39
]. Notwithstanding these differences with previous studies, the general conclusions of this work are largely in agreement with other research [21
Further steps could include working on the feature instead of the pixel level, where combined Sentinel-1 and Sentinel-2 observations could serve as input to an automatic parcel detection tool. Classification can subsequently be performed at the parcel level with the strong advantage that spectral signatures can be averaged for a field, significantly increasing the signal-to-noise ratio [79
]. This is especially beneficial for analyzing radar signatures, where reduced radar speckle at the parcel level will decrease intraclass variability and increase interclass variability, thereby improving classification accuracies. Future work could also focus on a comparison between radar-only and optical-only classification early in the season with a larger set of predictors, including for example interferometric coherence for radar acquisitions and SWIR bands for optical imagery, allowing a fairer comparison between both data sources. Lastly, it should be the subject of future research whether some crop types, such as winter wheat and winter barley, could be classified separately and earlier in the season than the other crop types based on their specific phenological differences.
Growing food demands require timely estimates of crop acreages at a large scale. Identification of different crop types forms the basis for such tasks. Also, at the parcel level, early crop type information is essential for crop monitoring purposes that aim at early anomaly detection. State-of-the-art earth observation technologies allow such identification from space. With the unprecedented potential of sensor synergies in the Copernicus program, multisensor classification procedures that go beyond traditional optical-only procedures can now be developed operationally.
Previous research clearly demonstrated the complementary nature of optical and radar signals to improve classification accuracies, in addition to alleviating cloud obstruction, and therefore challenging optical-only monitoring of vegetation.
We created 12-day Sentinel-1 backscatter mosaics and 10-daily Sentinel-2 smoothed NDVI mosaics over Belgium to develop a multisensor crop mapping approach tested on the 2017 growing season. Based on an optimized random forest classifier, we predicted eight different crop types with a maximum accuracy of 82% and a kappa coefficient of 0.77. The combination of radar and optical data always outperformed a single-sensor classification procedure, with a clearly increasing accuracy throughout the growing season up till July, when the maximum mapping accuracy was reached. While the most important predictors in the final classification were found to be related to optical acquisitions during key periods of the growing season, other important predictors consisted of a combination of optical and radar predictors, stressing the added value of a multisensor perspective. The concept of classification confidence was shown to provide additional information going beyond a hard classification result, indicating regions where confidence was lower, such as near parcel boundaries.
These results highlight the need for multisensor crop classification and monitoring efforts to fully exploit the rich potential of existing and future complementary satellite sensors (The final 2017 cropmap produced in this study is available upon request).