A Convolutional Neural Network Method for Rice Mapping Using Time-Series of Sentinel-1 and Sentinel-2 Imagery

Abstract

Rice is one of the most essential and strategic food sources globally. Accordingly, policymakers and planners often consider a special place in the agricultural economy and economic development for this essential commodity. Typically, a sample survey is carried out through field observations and farmers’ consultations to estimate annual rice yield. Studies show that these methods lead to many errors and are time-consuming and costly. Satellite remote sensing imagery is widely used in agriculture to provide timely, high-resolution data and analytical capabilities. Earth observations with high spatial and temporal resolution have provided an excellent opportunity for monitoring and mapping crop fields. This study used the time series of dual-pol synthetic aperture radar (SAR) images of Sentinel-1 and multispectral Sentinel-2 images from Sentinel-1 and Sentinel-2 ESA’s Copernicus program to extract rice cultivation areas in Mazandaran province in Iran. A novel multi-channel streams deep feature extraction method was proposed to simultaneously take advantage of SAR and optical imagery. The proposed framework extracts deep features from the time series of NDVI and original SAR images by first and second streams. In contrast, the third stream integrates them into multi-levels (shallow to deep high-level features); it extracts deep features from the channel attention module (CAM), and group dilated convolution. The efficiency of the proposed method was assessed on approximately 129,000 in-situ samples and compared to other state-of-the-art methods. The results showed that combining NDVI time series and SAR data can significantly improve rice-type mapping. Moreover, the proposed methods had high efficiency compared with other methods, with more than 97% overall accuracy. The performance of rice-type mapping based on only time-series SAR images was better than only time-series NDVI datasets. Moreover, the classification performance of the proposed framework in mapping the Shirodi rice type was better than that of the Tarom type.

1. Introduction

According to the Food and Agriculture Organization (FAO), rice is one of the essential commodities in the world [1,2,3]. More than half of the world’s population depends on rice as the primary food [1]. Rice paddies account for approximately 12% of the global croplands of the planet [4]. According to FAO’s 2016 statistics, global rice cultivation is around 741 million hectares, and about 50% of the world’s rice is cultivated in India and China. Historical records show rice has been cultivated in Iran since the first century BC. Based on statistics, Iran is a medium rice producer [5].

Given the importance of rice in the household food basket, creating jobs and income for large-scale agricultural producers and the government’s willingness to replace high-yielding cultivars with other varieties are crucial challenges for regional and local authorities [6,7,8]. According to the latest reports from governmental institutions in Iran, statistics about the area under cultivation and the annual rate of rice harvesting were carried out through field sampling and census data collection (www.maj.ir, accessed on 23 October 2022). The reliability of this approach is very low, and consequently, these results cannot be used in critical decision-making due to uncertainties. One of the most critical aspects of this lack of information and accuracy is the spatial distribution of rice paddies in this region [9].

Identifying and classifying the area under the cultivation of various agricultural products is vital for decision-making, improving management and policymaking, and economic planning for agricultural development [10,11]. Monitoring and mapping rice paddies continuously and efficiently play an essential role in agriculture and a sustainable environment, water and food security, damage assessment, and policy and decision-making [12,13,14]. Remote sensing techniques can provide timely and valuable information on crop distribution and the area under cultivation and yield potential for decision-makers [15,16].

For decades, optical remote sensing data have been used in different agricultural applications, including crop types classification [17], biomass and leaf area estimation [18], yield modeling [19], and disease identifications [20]. These applications used data from various spectral bands and sensors, such as Moderate Resolution Imaging Spectroradiometer (MODIS), SPOT, Landsat, etc. [21].

Due to the increased availability of multispectral remote sensing images, rice mapping has become a hot research topic. To this end, many studies have been done for mapping rice based on optical multispectral satellite imagery. For instance, Zhang et al. [22] used deep learning convolutional neural network (CNN) for paddy-rice mapping from multi-temporal Landsat 8, phenology data, and thermal datasets. In the first step, the spatial-temporal adaptive fusion model was employed to fuse MODIS and Landsat data to simulate multi-temporal Landsat-like data. Then, a thresholding procedure was conducted to derive the phenological variables from the Landsat-like NDVI time series. A generalized single-channel algorithm was used to derive LST from Landsat 8. Finally, multi-temporal Landsat 8 images, combined with phenology and LST data, were used to map rice paddies using a CNN framework.

Moreover, Jiang et al. [23] have recently used the differenced NDVI (dNDVI) and a thresholding approach based on Landsat’s NDVI to monitor changes in rice-cultivated areas in Southern China. They used two Landsat images to identify double-cropping rice (DCRs) using dNDVI and one Landsat image to identify single-cropping rice (SCRs). Most studies using optical data to identify rice-cultivated areas have relatively decent accuracies and results. The greatest challenge related to these studies is obtaining a significant number of images (i.e., high temporal resolution) to correctly identify the rice cultivation pattern. However, due to the small size of agricultural lands, the low spatial resolution causes spectral mixing issues, which highly affects the performance of small rice paddy detection and mapping.

According to the previous studies, methods based on the rice phenology cycle employ data from the time-series images of Landsat and MODIS. Due to the high temporal resolution of the MODIS data, it has a high potential to identify the area under rice cultivation. However, due to the low spatial resolution of this data and the small size of the paddies, MODIS pixels may include several different classes. As a result, this spectral mixture may lead to inaccurate identification of the areas under rice cultivation and increases uncertainty. On the other hand, Landsat images have a higher spatial resolution, which can resolve the problem of the mixed pixels and can much better extract the areas under rice cultivation. Due to the high cloud coverage in the study area and the low temporal resolution of Landsat imagery, it is not easy to obtain a significant amount of Landsat data. On the other side, using a high temporal and spatial resolution of the Sentinel-2 images, identifying the area under rice cultivation with the phenological-based rice approach is possible. Due to adverse weather conditions, e.g., clouds and rain, in the northern region of Iran, most of the time, optical data may not be beneficial for rice monitoring.

SAR data has shown attractive characteristics for agricultural applications thanks to their independence from weather and illumination conditions. These advantages of SAR satellite imagery caused some studies to focus on rice mapping based on SAR imagery. For instance, Nguyen, et al. [24] have studied the potential of time series of C-band SAR data for seasonal mapping of rice-cultivated areas. They showed that Sentinel-1 high-resolution images could increase the accuracy of the classification of rice paddies. Besides, Clauss, et al. [25] also used the Sentinel-1 time series and super-pixel segmentation for mapping rice fields in Poyang Lake in China. They applied image segmentation using the super-pixel method (i.e., SLIC). They also used a phenology-based decision tree for the extracted time series to classify each segment or object as either rice or no rice. Additionally, Bazzi, et al. [26] have analyzed the temporal behavior of SAR backscattering in many plots covering different crops. This analysis identified the rice paddies using several metrics derived from the Gaussian profile, the VV/VH time-series variance, and the slope of the linear regression of the VH time series.

Despite all these results, the biggest challenge with radar images and phenology-based methods for identifying rice paddies is the existence of similar classes with rice phenology cycles, such as wetlands, which slightly increases the error rate. Several studies have overlooked this challenge by integrating optical imagery, potentially improving discrimination among similar classes. The fusion of SAR and optical datasets can be considered an essential source for rice mapping. In this regard, Torbick et al. [27] have produced a land cover map through a random forest (RF) algorithm, including crop, water, forest, shrub, and urban areas, jointly using Sentinel-1, Landsat-8 OLI, and PALSAR-2 satellites in Myanmar. They used only crop pixels and masked non-crop pixels and analyzed the behavior of time-series pixels of crops for Sentinel-1 images. Finally, they used these analyses to extract the rice crop using the decision tree method. Their results showed that the mapped cultivated rice areas were close to the census statistics (e.g., R² = 78%).

Moreover, Onojeghuo et al. [28] employed RF and SVM algorithms to extract rice paddies from multi-temporal Sentinel-1A SAR data and Landsat-derived NDVI data. When the NDVI time-series data were fused with the various combinations of multi-temporal polarization channels (VH, VV, and VH/VV), overall classification improved significantly. In another study, Park et al. [29] classified paddy rice using RF and SVM machine learning algorithms based on satellite sensors Landsat, RadarSAT-1, and ALOS PALSAR. They first used Landsat images to identify rice paddies using RF and SVM algorithms. Then, SAR images were used to classify and extract rice paddies. Finally, they combined Landsat, SAR, and 30-m DEM to classify paddy rice. The results have shown that the fusion of optical and SAR sensor time-series data has the highest accuracy.

On the other hand, SAR data can be an attractive alternative to optical imagery for studying the dynamics of rice. However, optical data can still help identify rice paddies better, thanks to the absence of speckles in these images. Because of this region’s relatively small rice paddies and the necessity of providing reliable information for operational applications, high-spatial-resolution imageries from both SAR and optical sensors are primordial for rice mapping and monitoring. In addition, multi-temporal satellite imagery is essential for crop monitoring applications in general and rice in particular; this is because, according to the rice cultivation pattern, earth observations with a few days or a week of revisit time are needed to accurately study the crop conditions during the growing season.

The performance of deep learning-based frameworks has been assessed in many agricultural fields, and rice-type mapping is one of the most critical applications. Recently, some deep learning-based rice mapping frameworks have been proposed [30,31]. Unlike standard supervised methods, deep learning-based frameworks can automatically generate high-level instances and spatial features from the input dataset [32]. To this end, this research proposes a novel deep-learning framework for rice-type mapping. The proposed framework uses three-channel streams for deep feature extraction. The first and second channels investigate the original SAR and optical datasets, while the third channel considers the fusion of both. This research has made the following key contributions:

(1)

Presenting a novel deep learning procedure for mapping rice types based on multi-stream CNN.

(2)

Introducing an informative channel attention module for rice mapping.

(3)

Utilizing point-wise and group dilated convolution layers for improving the deep feature extractor.

(4)

Assessing the performance of the advanced rice-type methods based on deep learning frameworks and machine learning-based algorithms.

2. Study Area and Datasets

2.1. Study Area

Historical records show that rice has been cultivated in Iran since the first century BC, and today Iran is known as a mid-producer of rice (FAOSTAT, 2020). A small percentage of agricultural lands cultivated annually in Iran is under rice cultivation. About 70% of the rice cultivated areas are located in the two northern provinces, Gilan and Mazandaran, located south of the Caspian Sea. The position and original location of Mazandaran province represent two major areas, consisting of the Alborz coastal line and alluvial plains. The weather of the study area is moderate, humid in the summer, and relatively cold and dry in the winter, and the average temperature required for rice cultivation in this region is 23.5 °C. The total annual precipitation in this province is 631 mm (www.irimo.ir, accessed on 23 October 2022). The study area of this research is located in the eastern part of Mazandaran province, with an area of 73 km² (Figure 1).

2.2. Sentinel Imagery

The rice growth stages are from 85 to 100 days from planting to harvesting in the study area. Rice is transplanted in this area from early May to mid-June and harvested from the beginning of August through mid-September using mechanical or traditional methods (Figure 2).

According to the crop calendar of the study area, multi-temporal imagery from Sentinel-1 and Sentinel-2 in 2019 was used to determine the rice cultivated areas. Fifteen Sentinel-1 images were collected every 12 days and used as SAR data. However, only twelve cloud-free Sentinel-2 images were helpful over the study area during the growing season (Figure 2).

Sentinel-1 provides dual-pol SAR data in Interferometric Wide Swath (IW) mode Level-1. The ground-range detected high-resolution (GRDH) products were employed. These image data were collected during the spring and summer of 2019. The pre-processing steps include radiometric calibration, speckle filtering, and terrain correction. To remove the speckle from SAR data in this study, the Enhanced Lee Filter (3 × 3) has been applied. The multi-temporal imagery from Sentinel-2 was also used as the optical/multispectral data. The Sen2Cor processor was then used for Sentinel-2 data correction [33,34]. All Sentinel-1 and Sentinel-2 datasets were resampled to a 10 m grid.

2.3. Reference Samples

In the study area, various surveys were conducted to collect samples of rice cultivation systems according to the growing season. The distribution of the in-situ samples over the study area is shown in Figure 3. A total of 354 samples, including 259 rice fields of Tarom-Hashemi and Shirodi varieties and 167 non-rice fields and other land covers, including wheat, canola, gardens, roads, wetlands, and rural areas, were collected (

Table 1. A description of the statistical details of the reference samples.

ID	Crop Type	All Samples	Training (4.1%)	Validation (0.9%)	Test (95%)
1	Non-Rice	93,991	3854	846	89,291
2	Tarom-Hashemi	15,568	638	140	14,790
3	Shirodi	26,009	1066	235	24,708
Total		135,568	5558	1221	128,789

). These samples were randomly divided into three groups for training, validation, and testing of classification results, respectively.

In addition, during the field study, local farmers were consulted to collect some information regarding cropping information. Moreover, statistics from the ministry of agriculture were obtained about two significant rice varieties cultivated in the study area. Both rice varieties in the study area are cultivated simultaneously, and the time interval between planting and harvesting is similar. Nevertheless, the only thing distinguishing these two rice varieties is that one has a higher vegetation density and more weight than the other.

We have tried to select training samples from these two varieties equally. Another critical factor in selecting training samples in the study area is the soil type throughout the region. In some areas, farmers add clay soils or sea sands to their paddies each year to have more fertile paddies and grow more rice products, but in some other areas, paddies have rough soil, which causes low cropping per year. We have considered this factor in collecting training samples and identified areas with different soil types and high quality through the ministry of agriculture statistics and the local farmers. We also collected training samples in several areas with more diverse soils.

3. Rice Type Classification

The overview of the rice types classification proposed framework is illustrated in Figure 4. Based on the presented flowchart, the rice-type mapping framework is applied in three main steps.

3.1. Data Preparation

This research used the pre-processed Sentinel-2 L1-C dataset available through the Sentinel Hub by ESA Copernicus program. However, the atmospheric correction was applied to these data using the SNAP’s Sen2Cor module. Next, the NDVI time series was generated. Feature extraction can be carried out in various ways (i.e., statistical methods, spectral indices), but one of the most common analyses is combining spectral bands using simple mathematical operations. Since NDVI is simple and highly applicable for crop mapping, it was chosen from different spectral indices. The NIR and red bands were used to calculate NDVI (see Equation (1)).

$$NDVI=\frac{\left(NIR-Red\right)}{\left(NIR+Red\right)}$$

(1)

Furthermore, Sentinel-1 SAR Level-1 Ground Range Detected (GRD) images were used. The main pre-processing of Sentinel-1 SAR imagery includes despeckling (refined Lee filter) and the geometric correction conducted in SNAP software.

3.2. Proposed Multi-Streams Framework

Deep feature extraction is the most crucial part of rice-type mapping. The multi-stream deep feature extract framework is widely used for analyzing a bi-temporal dataset. The Siamese architectures are the most common such architecture that has two streams. Our proposed framework inspired the Siamese network architecture but with more modification. Thus, this research proposed a novel multi-streams deep feature extraction framework. Figure 5 shows the proposed architecture for rice-type mapping in this research. The proposed architecture has three streams that extract the deep features from the input dataset based on this figure. The first channel considers the task of deep feature extraction from the optical time-series NDVI dataset. The third stream investigates the time series SAR images for deep feature generation. The second stream extracts the deep feature by fusion of optical and SAR deep features at different abstracts. Next, the feature maps are fed to the global average-pooling layer to reduce feature map size. The latest part is the classification, which uses the fully connected and soft-max layers. This part extracts deep features by three streams concatenated by concatenating layers. Then, the deep features are fed to a fully connected layer for more consideration. Finally, the soft-max layer is employed to make decisions on the input datasets and map rice-type. In comparison with other CNN frameworks, the proposed architecture has the following differences:

(1)

Utilizing a multi-streams framework that can consider high-level meaningful features from the original dataset and fuse them in different feature levels.

(2)

Introducing a channel attention module for informative feature extraction.

(3)

Using residual, point-wise, and group dilated convolution layers for deep feature extraction.

(4)

Employing the global average-pooling layer instead of a flattening layer reduces the model parameters.

3.2.1. Channel Attention Mechanism (CAM)

Each high-level channel mapping of a feature can be considered a class-specific response, with different semantic responses being interrelated [35]. By leveraging the dependencies between channel maps, you can emphasize interdependent functional maps and improve the functional representation of specific semantics [17]. The result is a CAM that explicitly models channel interdependence [36]. Thus, this research used a CAM for rice mapping to improve the generalization capabilities of the proposed model. According to Figure 6, the channel attention module has the following general structure.

The out-of-the-CAM module (Ω) for the input feature map ($f$) can be formulated as follows:

$${\Omega }_{i,j}=\frac{e^{(f_i{f}_j)}}{{\sum }_{i=1}^C{e}^{(f_i{f}_j)}}$$

(2)

where C is the number of channels, and ${\Omega }_{i,j}$ calculates the impact of the ith channel on the jth channel. Finally, the output of the CAM module ($\mathscr{H}$) can be calculated by multiplying a scale parameter $\left(\mathsf{\eta }\right)$ and applying an element-wise sum operation with ($f$).

$${\mathscr{H}}_j=\mathsf{\eta }{\sum }_{i=1}^C\left({\Omega }_{i,j}{f}_i\right)+f_j$$

(3)

where $\mathsf{\eta }$ gradually learns a weight from 0.

3.2.2. Convolution Layer

Convolutional layers are the fundamental basis of the CNN framework, and their primary function is to generate high-level depth features from the feature data ($f$) [32]. At the convolution layer, both spatial and spectral features are considered. The basic calculation for the lth convolution layer with a activation function (χ) is given below [37].

$$f^l=\mathsf{\chi }\left(w^l{f}^{l-1}\right)+b^l$$

(4)

where w and b denote weight and bias, respectively. Equation (1) for a 2D-convolution at position ($\varphi$, $\lambda$) with a dilation rate $\left(\zeta \right)$ can be expanded as follows [38]:

$$f_l^{\varphi \lambda }=\mathsf{\chi }(b_l+{\displaystyle \sum }_m^{ }{\displaystyle \sum }_x{\displaystyle \sum }_y{W}_{l,m}^{x,y}{f}_{l-1,m}^{\left(\varphi +x\times \zeta \right)\left(\lambda +y\times \zeta \right)})$$

(5)

where m denotes the feature map associated with the current feature map in the ($\alpha -1$)th layer. The point-wise convolution layer uses the kernel convolution filter with size (1 × 1). Moreover, the group dilated convolution layers use the convolution layers with different dilation rates (i.e., 1, 2) and kernel sizes. In this layer, the number of filters for each layer ${\theta }_n$ are determined by multiplying a ratio of parameters ${\tau }_n$ with $\theta$. The mathematical number of filters can be described as follows:

$${\theta }_n={\tau }_n\times \theta s.t:{\sum }_{n=1}^N{\tau }_n=1$$

(6)

where $n$ is the number of layers in the group convolution layer, which this study set at 3, and $\tau$ is the coefficient that determines the number of filters for the convolution layer. The residual block allows gradients to be directly back-propagated to earlier layers, which is particularly useful for preventing vanishing gradients or explosions.

3.2.3. Comparison with Other Classification Methods

The result of rice mapping is compared with the most common machine learning and deep learning-based methods. To this end, we implemented four methods: RF, XGBoost, 2D-CNN, and 3D-CNN. The 2D-CNN proposed by Zhao et al. [24] includes two 2D-convolution and 2D-pooling layers and soft-max layers for making input decisions. Fernandez-Beltran proposed the 3D-CNN framework and utilized 3D convolution layers. This method uses six 3D-convolution and 3D pooling layers for feature map reduction.

Finally, a fully connected layer and soft-max layer are used for making a decision. The input data of these methods are stacking time-series Sentinel-1 and NDVI derived from Sentinel-2 images. The results of rice-type mapping are assessed by quality measurement indices that include overall accuracy (OA), kappa coefficient (KC), user’s accuracy (UA), producer’s accuracy (PA), omission error (OE), and commission error (CE).

4. Experiment and Discussion

4.1. Parameter Setting

Some parameters need to be set for rice mapping classifier algorithms. The best value of classifier parameters is obtained by trial and error and evaluated on the validation dataset.

Table 2. Optimal values for the classifier parameters.

Data	Description
RF	estimators = 105, features to split each node = 3
XGBoost	rounds = 500, subsample = 1, min-child-weight = 1, max-depth = 5.
Deep Learning Models	Dropout-Rate = 0.1, Mini-Batch-Size = 150, Iterations = 500, Initial Learning = 10−4

provides the optimum value of machine learning methods and deep learning algorithms.

In deep learning-based methods, patch size is of utmost importance. We evaluated the sensitivity of patch size in the performance of the proposed method (Figure 7).

4.2. Classification Results

Optical and Radar earth observations from Sentinel-1 and Sentinel-2 sensors have advantages and disadvantages for rice crop mapping. For example, the pixel-based rice map would be noisy due to speckle noise in Sentinel-1 images. On the other hand, it is necessary to have a regular SAR image every few days due to the rice cultivation pattern. Moreover, SAR data would be a real advantage for the cloudy conditions of the study area. In contrast, the Sentinel-2 data are affected by clouds and shadows but are free of noise.

Consequently, we integrated these two data to improve the classification performances for efficient and reliable rice crop mapping. As shown in Figure 8, the rice maps classified by RF and XGBoost methods provide many noisy labels. In contrast, the DL-based methods performed better thanks to their internal use of spatial and spectral features for classification.

In this area, rice paddies are mainly located near wetlands to benefit from water for irrigation. According to the field observations from the study area, in the center of the study area, more Tarom rice is cultivated, and in the south, more Shirodi rice is cultivated. The rice map obtained from our proposed method corresponds more with field observations.

Nevertheless, as shown in Figure 9, the second factor is the large rice volume in some areas, with a few pixels of another type of rice among them. This may be because the rice-type seeds are not completely separated, which means that one type of rice does not grow uniformly in the rice fields.

The performance of the rice-type mapping was assessed based on the field samples collected. Figure 10 shows the proposed and implemented classification methods’ confusion matrices. As can be seen, many pixels are incorrectly classified in RF and XGBOOST methods, and the error rate in Deep Learning methods is much lower. In deep learning methods, the wrong pixels classified in the 3D-CNN algorithm are gradually less than in the 2D-CNN algorithm, which may be because the 3D-CNN algorithm analyses the data in more detail. Furthermore, in our proposed method, the percentage of wrong pixels is much lower than other algorithms because it has a more complex architecture than other methods that perform much more accurate analysis.

Table 3. Comparison of different classification algorithm accuracies. The bold values show the highest accuracies.

Method	Class	UA	PA	CE	OE	OA	KC
RF	Non-Rice	96.5	92.5	3.5	7.5	80.8	0.60
	Tarom-Hashemi	34.1	54.6	65.9	45.4
	Shirodi	68.4	53.9	31.5	46.1
XGBOOST	Non-Rice	94.2	96.6	5.8	3.4	84.0	0.65
	Tarom-Hashemi	47.2	32.2	52.7	67.8
	Shirodi	63.6	69.7	36.4	30.3
2D-CNN	Non-Rice	98.9	98.9	1.0	1.0	93.4	0.86
	Tarom-Hashemi	73.1	79.6	26.8	20.4
	Shirodi	86.1	81.5	13.8	18.4
3D-CNN	Non-Rice	99.6	99.2	0.3	0.7	95.8	0.91
	Tarom-Hashemi	81.4	86.1	18.5	13.8
	Shirodi	90.8	89.2	9.1	10.7
Proposed	Non-Rice	99.4	99.7	0.6	0.3	97.1	0.94
	Tarom-Hashemi	89.1	89.4	10.8	10.6
	Shirodi	93.7	92.5	6.2	7.5

summarizes the statistical accuracy assessment for the rice-type mapping utilizing different accuracy measures. A non-deep learning algorithm, the RF algorithm, had the lowest performance (OA = 81% and KC = 0.60), while the XGBoost classifier provided satisfactory results (OA = 84% and KC = 0.65). In contrast, all deep learning algorithms except the 2D-CNN algorithm achieved over 90% by OA. In particular, the proposed method presented the highest accuracy for rice-type mapping with an OA and KC of 97% and 0.94, respectively. In addition, 3D-CNN provided UA more than the proposed method (0.2%) in the Non-Rice class, while it missed its performance in the other classes.

4.3. Ablation Analysis

Various aspects of artificial intelligence methods can be evaluated using ablation analysis. The primary purpose of this analysis is to obtain an insight into the effects of removing a part of the system, e.g., an attention layer, on the general model performance. Using three scenarios, we examined the effectiveness of our proposed method after ablation analysis (S1) without attention block, (S2) without a pooling layer, and (S3) without a Fully Connected Layer.

Table 4. Ablation analysis of proposed method in rice variety mapping. The bold values show the highest accuracies.

Method	Class	UA	PA	CE	OE	OA	KC
S1	Non-Rice	99.2	99.4	0.7	0.5	95.5	0.90
	Tarom-Hashemi	81.3	85.1	18.6	14.8
	Shirodi	90.7	87.4	9.2	12.5
S2	Non-Rice	99.5	99.1	0.4	0.8	96.1	0.92
	Tarom-Hashemi	86.0	85.4	13.9	14.6
	Shirodi	90.0	91.6	9.9	8.3
S3	Non-Rice	99.5	99.6	0.4	0.3	96.3	0.92
	Tarom-Hashemi	86.1	85.0	13.8	15.0
	Shirodi	90.6	91.2	9.4	8.8
All Components	Non-Rice	99.4	99.7	0.6	0.3	97.1	0.94
	Tarom-Hashemi	89.1	89.4	10.8	10.6
	Shirodi	93.7	92.5	6.2	7.5

shows the result of ablation analysis in the different scenarios. Based on numerical results, the first scenario (without attention block) has the highest effect on the result of rice mapping. Moreover, the third scenario, i.e., without a Fully Connected Layer, has the lowest effect on the proposed model’s performance.

4.4. Discussion

4.4.1. Accuracy Assessment and Comparison with Other Models

The result of rice mapping shows that the proposed method outperformed other methods. The OA index provided the proposed method with more than 97% accuracy. Furthermore, the proposed method has high efficiency in detecting rice type compared with other methods, as the UA and PA are more than 89%. Based on numerical analysis, the proposed method’s performance in classifying Shirodi is better than Tarom-Hashemi. This issue derived from the used sample data for Shirodi is more than Tarom-Hashemi.

Figure 9 shows the classified map of the proposed method in different areas using VHR imagery. As can be seen, our proposed method has been very good at separating roads and even the boundaries between rice paddies. It has correctly identified residential areas, water areas, gardens, etc., as a non-rice class. The accuracy of the classified map using our proposed method is 97.12, which is approximately a three percent error, which can be due to two reasons. Wetlands are the first cause of the change process in this crop calendar, somewhat similar to rice cultivation. In the first stage, wetlands are dry areas filled with water to supply rice paddies. Then water gradually decreases, and the vegetation grows until the water is completely drained, and the vegetation will be the size of a rice plant.

Several studies have been done for rice mapping based on various remote sensing datasets in recent years.

Table 5. The comparison of numerical results from rice-type mapping is based on multisource remote sensing imagers.

Reference	Method	Dataset	OA (%)
Zhai et al. [39]	RF classifier	Sentinel-2, Radarsat-2	94
Xu et al. [40]	SVM classifier	Landsat-8	88
Cao et al. [41]	Decision tree classifier	Landsat-5/Landsat-8	85
Zhang and Lin [42]	Object-based	MODIS/Landsat	92
Liu et al. [43]	RF	Sentinel-2	91
Lasko et al. [44]	RF classifier	Sentinel-1	93
Zhang et al. [45]	Object-based	Sentinel-1, Sentinel-2	90
Nguyen, Gruber and Wagner [24]	Decision tree classifier	Sentinel-1	87
Zhao, Liu, Ding, Liu, Wu and Wu [31]	Deep Learning	HJ-1 A/B	93
Xu et al. [46]	Deep Learning	Sentinel-1	91
Wei et al. [16]	Deep Learning	Sentinel-1	91
Proposed Method	Deep Learning	Sentinel-1, Sentinel-2	97

shows some recent and advanced rice mapping methods based on remote sensing satellite images. The proposed method has provided promising results compared with other machine and deep learning methods.

Rice cultivation in Iran mainly covers a small region in the north. The size of rice paddies in this region is typically small. Therefore, spatial resolution can be essential for rice mapping in such areas. There are many studies for rice-type mapping based on medium-resolution remote sensing datasets (i.e., MODIS, VIIRIS) [47,48]. These studies reported promising accuracy for rice-type mapping. For example, Gumma et al. [49] reported an OA under 90 for rice classes based on MODIS imagery.

Moreover, the medium-resolution sensors have a high temporal resolution. They can provide rich information for rice-type mapping but cannot map the rice fields with small regions. This issue is more evident in northern Iran, where more rice fields cover small areas. Thus, mapping small rice fields is more complex, and utilizing medium-resolution datasets is unsuitable for such areas. High-resolution satellite imagery (i.e., Sentinel-1, Sentinel-2, and Landsat) can be a valuable data source for this application.

The sample data collection is the most crucial factor in rice mapping by supervised learning methods. The collection of relevant and reliable sample data is time-consuming and challenging. It is worth noticing that the proposed model is trained only by about 6000 pixels. Moreover, other semantic segmentation-based methods (i.e., U-Net, Deeplab) [50] demand more sample datasets with all label pixels for a patch. Thus, the proposed method can be applied to rice-type mapping with sample data lower than used by other semantic segmentation-based methods.

4.4.2. Variables Assessment

The fusion of remote sensing datasets has improved mapping results in many applications. This study takes advantage of time-series Sentinel-1 and Sentinel-2 images for rice-type mapping. To evaluate the effectiveness of these datasets, we assessed the performance of the proposed method by three scenarios: S1 for only time-series Sentinel-1, S2 for only Sentinel-2 NDVI time-series, and S3 for a combination of Sentinel-1 and Sentinel-2 NDVI time-series.

Table 6. Comparison time-series Sentinel-1 and Sentinel-2 based NDVI datasets in rice variety mapping. The bold values show the highest accuracies.

Method	Class	UA	PA	CE	OE	OA	KC
S1	Non-Rice	99.0	97.4	1.0		90.7	0.81
	Tarom-Hashemi	75.0	51.5	25.0
	Shirodi	72.5	90.3	27.5
S2	Non-Rice	99.3	97.4	0.7	2.6	89.5	0.78
	Tarom-Hashemi	61.6	56.3	38.4	43.7
	Shirodi	72.3	80.9	72.3	19.1
S3	Non-Rice	99.4	99.7	0.6	0.3	97.1	0.94
	Tarom-Hashemi	89.1	89.4	10.8	10.6
	Shirodi	93.7	92.5	6.2	7.5

represents the results of the evaluation of rice-type mapping by the proposed method. The rice map from Sentinel-1 data is more accurate than the time series of Sentinel-2 NDVI datasets. Furthermore, the combination of both datasets improved the result of rice-type mapping significantly compared with the other only one dataset.

4.4.3. Feature Fusion Strategies

Because the study area always has rainy and cloudy conditions for more of the time during the cultivation season, the SAR data cloud is used to improve the result of crop mapping. Integrating multisource data is the most common strategy to enhance the mapping results. The integration of multisource datasets can be applied on different levels. The feature-level fusion is widely used in many applications of remote sensing. The deep learning methods extract deep features at different levels from the input dataset. The deep features show spatial features (i.e., the object’s edge) in the shallow levels, while deep high-level features present semantic spectral information. This research proposed a multi-streams deep learning-based method that fuses deep features at low and high levels, improving the mapping results.

The deep feature level fusion can be applied in the first, intermediate, and latest layers. We evaluated the performance of feature-level fusion in the different layers that are included: (1) first-layers (concatenating optical and SAR dataset), (2) latest-layer (concatenating extracted deep features from optical and SAR datasets), and (3) all layers (concatenating all deep features of all layers).

Table 7. Comparison of different strategies for deep feature fusion. The bold values show the highest accuracies.

Method	Class	UA	PA	CE	OE	OA	KC
First-layer	Non-Rice	98.9	96.9	1.1	3.1	87.3	0.73
	Tarom-Hashemi	49.4	72.8	50.6	27.2
	Shirodi	77.6	61.3	22.4	38.7
Latest-layer	Non-Rice	98.9	98.6	1.1	1.4	92.3	0.84
	Tarom-Hashemi	68.9	76.1	31.1	23.9
	Shirodi	83.7	79.3	16.3	20.7
All-layer	Non-Rice	99.28	99.4	0.7	0.6	95.9	0.91
	Tarom-Hashemi	84.6	84.6	15.4	15.4
	Shirodi	90.4	89.8	9.6	10.2

shows the influence of feature-level fusion in the different layers. As seen, the fusion of deep features in the latest layers is better than the first layers as it improves the accuracy of the OA by 5%. Furthermore, the fusion of deep features from different layers provided the best performance. They improved the accuracy of rice-type mapping by more than 8% and 3% compared with the first and latest layers, respectively.

5. Conclusions

This study aimed to develop an operational framework for mapping and inventorying rice using remote sensing technology in Iran, as there has been a lack of information for statistics and planning needs in the agricultural sector. This study analyzed two-time series of Sentinel-1 and Sentinel-2 datasets from the spring and summer of 2019 to identify the areas under rice cultivation. This research proposes a novel rice-mapping framework that can detect rice areas. Furthermore, the proposed framework can identify the rice type into two main classes that can inform decisions to support food security. The proposed framework fuses time-series SAR and optical images for rice-type mapping. The model is built based on a multi-stream deep feature fusion and extraction manner that extract deep features from the original dataset and fuse them with another stream. The proposed model has some advantages over the currently available models: (1) high accuracy in rice-type mapping, (2) extracting deep features at the same time from SAR and optical datasets and fusing them in different levels, (3) ability to detect rice type in the two main classes, and (4) ability to take advantage of the CAM, residual block, and multi-scale kernel convolutions.