CerealNet: A Hybrid Deep Learning Architecture for Cereal Crop Mapping Using Sentinel-2 Time-Series

Abstract

Remote sensing-based crop mapping has continued to grow in economic importance over the last two decades. Given the ever-increasing rate of population growth and the implications of multiplying global food production, the necessity for timely, accurate, and reliable agricultural data is of the utmost importance. When it comes to ensuring high accuracy in crop maps, spectral similarities between crops represent serious limiting factors. Crops that display similar spectral responses are notorious for being nearly impossible to discriminate using classical multi-spectral imagery analysis. Chief among these crops are soft wheat, durum wheat, oats, and barley. In this paper, we propose a unique multi-input deep learning approach for cereal crop mapping, called “CerealNet”. Two time-series used as input, from the Sentinel-2 bands and NDVI (Normalized Difference Vegetation Index), were fed into separate branches of the LSTM-Conv1D (Long Short-Term Memory Convolutional Neural Networks) model to extract the temporal and spectral features necessary for the pixel-based crop mapping. The approach was evaluated using ground-truth data collected in the Gharb region (northwest of Morocco). We noted a categorical accuracy and an F1-score of 95% and 94%, respectively, with minimal confusion between the four cereal classes. CerealNet proved insensitive to sample size, as the least-represented crop, oats, had the highest F1-score. This model was compared with several state-of-the-art crop mapping classifiers and was found to outperform them. The modularity of CerealNet could possibly allow for injecting additional data such as Synthetic Aperture Radar (SAR) bands, especially when optical imagery is not available.

1. Introduction

The world population is increasing at an alarming rate. It has almost quadrupled to 6.2 billion during the last century [1], and this number is expected to reach 10 billion by 2050 [2]. To be able to keep up with the ever-increasing global food demand, crop production needs to approximately double by 2050 [3]. To do so, the public and private sectors need timely, efficient, and, most importantly, reliable agricultural data. Such data would help to better monitor food demand, reduce costs, and target agricultural policies [4]. Agricultural data in Morocco can be obtained using one of three methods: (I) direct communication with farmers, (II) photo-interpretation of high-resolution digital imagery, and (III) spatially limited land surveys. The first two methods are liable to produce subjective, biased, and limited information. The third method, however, is the most accurate when it comes to crop mapping. However, it is not without its drawbacks, for it is time-consuming and expensive, which limits its use as a practical periodic crop-monitoring tool [5].

Identifying vegetation canopies in agricultural landscapes using earth observation techniques such as satellite imagery, drone imagery, and smart ground sensors has continued to grow in economic importance [6]. This is mainly because traditional data-collection techniques such as survey missions are now outdated, costly, and inefficient. Traditional Machine Learning (ML) classifiers have proven their effectiveness in crop mapping using multispectral aerial and satellite imagery [7,8,9,10]. Support Vector Machine (SVM) and Random Forest (RF) in particular are frequently used in the literature for crop identification based on multi-temporal satellite imagery [11,12,13]. Although used as a baseline, these models have achieved competitive overall accuracy on multiple occasions. The performance of these classifiers is especially high when the classes belong to different crop types whose spectral signatures are very different (cereals, pulses, vegetables, …etc.) [14]. ML models have been known to struggle when presented with crops that belong to the same family; this issue presents itself time and time again in the literature [15,16,17,18,19,20,21,22,23,24]. For this reason, it is common to merge crops that have similar spectral responses such as small-grain cereal crops (soft wheat and durum wheat) in a single class (wheat) so as to minimize class confusion [25]. This solution does not allow for estimating the real area and production of each crop. Deep Learning (DL) approaches have been recommended as a potential route to distinguish between crops with similar biological characteristics [26]. Unlike traditional ML classifiers, where each input feature is an independent variable, the Long Short-Term Memory unit (LSTM) processes time-series as sequences where the order of the time-series has a significant effect on the results. This quality could help in mining small differences that might exist in the phenological cycle of problematic crops. While DL has been used extensively for crop mapping [12,13,27,28,29,30,31], we could not find a study dealing with crops from the same family.

Ground-based hyperspectral data has the capability of extracting discriminant features based on fine variations in the full-wave spectral profile [32]. These differences are thought to be indiscernible with multispectral aerial or satellite imagery [25]. Small-grain cereal crops (soft wheat, durum wheat, oat, and barley) are notorious for being difficult to classify accurately. This issue stems from the fact that these crops have a near identical spectral profile throughout their entire phenological cycle. Cereal crops are considered to be the most prolific food source on the planet [33]. They are grown on more land than any other food crop [34], amounting to 86% of the total area sown in Morocco in 2021. It is therefore imperative that they are accurately mapped. Our overall goal in this paper is to contribute to food security and balance in food imports and exports, as well as the currency regulations that comes with it. This is why decision-makers and managers of the agricultural sector need fast and precise satellite-based solutions.

In this study, we propose a unique deep learning-based approach for mapping four cereal crops: soft wheat, durum wheat, oats, and barley. The proposed model architecture, dubbed “CerealNet”, uses unidimensional convolutional neutral networks (Conv1D) to extract spectral features from Sentinel-2 satellite imagery, and employs Long a Short-Term Memory unit (LSTM) to derive phenological features from the Normalized Difference Vegetation Index (NDVI). The NDVI itself is calculated from the Sentinel-2 time-series. The rest of this document is structured as follows: In Section 2, we present the study area and the data used in the study. Section 3 details the methodology and the motivation behind the adopted approach. In Section 4, the results of the crop mapping are showcased and comparisons with state-of-the-art models are made, evaluated and discussed. Finally, in Section 5, we conclude by listing the main findings of the study, as well as some recommendations going forward.

2. Study Area and Data

2.1. Study Area

The study area is located in the Gharb plain, northwest of Morocco. It covers an area of more than 50,000 ha centered around the city of Sidi Slimane (Figure 1). It is characterized by a hot Mediterranean climate with a dry summer, an average temperature of 19 °C, and an annual precipitation of 451 mm. These climatic conditions are sufficient to maintaining traditional agriculture. Intensive agriculture has been promoted in the Gharb plain, owing to good alluvium soils and an efficient irrigation system [7]. The plain has approximately 114,000 ha of land that is equipped and suitable for large-scale hydraulic development.

2.2. Data

2.2.1. Ground Truth

Ground truth data were collected through a field survey on June 2021 in the study area. During this period, most of cereal plots were at the maturity stage. Small-grain cereal crops—apart from oats—are notorious for being difficult to classify due to their biological similarities (Figure 2). Soft wheat, durum wheat, and barley all have spikelets (elementary inflorescence characteristic of grasses) that are grouped into spikes. These crops were identified by an agricultural expert on site. Then, the limits of the plots were recorded digitally using SW-Maps in the shapefile format [35]. Along with the geographical coordinates, other information such as crop type, previous campaign’s crop, and irrigation system were collected.

In the study area, cereals are generally rain-fed crops, and are sown between mid-November and early December. The harvest occurs between late May and mid-June. Concerning the details of the collected data, we recorded 660 cereal plots: 426 of which were of soft wheat, 86 barley, 82 oats, and 66 durum wheat parcels. For each field, Sentinel-2 ground pixel values were sampled. The fields were then randomly split into a training set and a testing set. This split was performed to ensure that training and testing pixels are not taken from the same field.

Table 1. Summary of the number of plots and pixels per cereal crop in the study area.

Crop	Plots			Pixels
	Train	Test	Total	Train	Test	Total
Barley	63	23	86	3117	979	4217
Soft wheat	359	67	426	17,200	5744	23,006
Durum wheat	50	16	66	3004	1040	4157
Oats	61	21	82	1727	540	2291
Total	533	127	660	25,048	8303	33,671

shows the number of samples generated for each cereal crop type.

2.2.2. Remote Sensing Data

In this study, we needed an imagery dataset with a high revisit frequency, as well as high spatial and spectral resolutions. On top of being a public dataset, level 2-A bottom-of-atmosphere Sentinel-2 satellite imagery has a high revisit frequency, with the constellation of Sentinel-2A and 2B [36]. More importantly, this dataset offers both a high spectral and spatial resolution. The bands used in this study are listed in

Table 2. Sentinel-2 bands used in this study. VNIR: Visible and Near-Infrared; SWIR: Short Wave Infrared.

Band	Resolution (m)	Central Wavelength (nm)	Description
B2	10	490	Blue
B3	10	560	Green
B4	10	665	Red
B5	20	705	VNIR
B6	20	740	VNIR
B7	20	783	VNIR
B8	10	842	VNIR
B8A	20	865	VNIR
B11	20	1619	SWIR
B12	20	2190	SWIR

.

A number of 14 cloud-free images were selected throughout the phenological cycle of the four cereal crops.

Table 3. Satellite imagery used in this study.

Observation Date	Tiles	Cloud Cover %
2020-12-22	30STC - 30STD	0
2020-12-27	30STC - 30STD	1
2021-01-03	30STC - 30STD	1
2021-01-18	30STC - 30STD	1
2021-01-26	30STC - 30STD	0
2021-02-15	30STC - 30STD	0
2021-03-14	30STC - 30STD	2
2021-03-22	30STC - 30STD	0
2021-03-24	30STC - 30STD	0
2021-04-13	30STC - 30STD	4
2021-04-18	30STC - 30STD	1
2021-05-06	30STC - 30STD	3
2021-05-18	30STC - 30STD	1
2021-05-21	30STC - 30STD	0

provides an overview of the different images used. The length of the time series was limited by the cloud presence in the study area, as only imagery with less than 5% cloud cover was used. The study area is covered by two Sentinel-2 tiles—30STC and 30STD; the tiles were therefore merged and co-registered to correct for pixel shift. Since the spectral bands have different resolutions, we re-sampled the Visible and Near Infra-Red bands (VNIR), as well as the Short Wave Infra-Red bands (SWIR) to 10 $\mathrm{m}$. The re-sampling was performed by splitting each 20 $\mathrm{m}$ pixel into four 10 $\mathrm{m}$ pixels and keeping the original pixel value. After this operation, the study area was extracted.

3. Methodology

In this study, we aimed to develop an efficient and accurate approach for cereal crop mapping (Figure 3). The proposed methodology uses a pixel-based image analysis (PBIA) approach for multi-spectral satellite imagery time-series. First, a field survey was performed to collect ground truth data. Then, Sentinel-2 satellite imagery was downloaded and preprocessed. Feature engineering was conducted to select eliminate unnecessary bands and to generate the vegetation index. Pixel values from multispectral satellite imagery and the vegetation index were then combined with the ground truth data in order to build the dataset. This dataset was then used to train the CerealNet model. Additionally, four different crop mapping classifiers were compared to the proposed method to evaluate its performance.

3.1. Traditional Machine Learning

3.1.1. Support Vector Machine

An SVM is a robust supervised non-parametric classification method that is capable of handling noisy and complex data [37,38,39]. Based on the statistical learning theorem, the SVM first transforms the inputs into a higher-dimensional space using a specified kernel, and then searches for a linear decision boundary in this space [37,38]. Taking as input a set of points belonging to two classes, a linear SVM finds a hyper-plane that leaves the maximum possible number of points of the same class on the same side while maximizing the distance between the hyper-plane and the two classes. According to [39], this hyper-plane minimizes the risk of having misclassified points.

The SVM model used in this paper was trained using the Scikit-learn library [40] in Python [41]. The optimal hyperparameters of the SVM model were identified using the GridSearch function in Scikit. The best weights were generated using the polynomial kernel. The optimal value of C, which defines the regularization of the error, was 1, whereas gamma, a hyperparameter that indicates how loosely the model will fit the training, was scaled based on the input features (Equation (1)).

$$\gamma =\frac{1}{n\times v}$$

(1)

where $\gamma$ represents the value of gamma, n is the number of features, and v is the variance.

3.1.2. Random Forest

The random forest classifier is an extension of the standard Decision Tree (DT) algorithm [42]. RF is robust and resistant to overfitting. The majority vote of all decision trees in an RF is used to assign a target class of each input. Its capability of handling huge amounts of data, along with its few user-defined inputs and fast execution time [43], make RF one of the most popular classifiers for crop mapping.

Similar to the SVM, the RF classifier used in this study was trained using Sickit-learn, and the hyperparameters were identified using GridSearch. The optimal number of trees was found to be 200, with a maximum depth of 8. The criterion for the quality of the split was entropy, and the maximum number of features to consider when looking for the best split was set to sqrt, the square root of the number of features.

3.2. Deep Learning

3.2.1. LSTM

Traditional Recurrent Neural Networks (RNNs) are a type of network where feedback connections are used to store representations of recent input events in the form of activation [44]. This feature makes RNN appropriate for solving tasks that rely on processing sequential data. However, because of the vanishing gradient problem, RNNs are unable to learn in the presence of time lags greater than 5–10 discrete time-steps between relevant input events and target signals [45]. This issue has led to the development of LSTM, which can prevent older signals from vanishing during processing. Even though LSTM has been developed since the late 1990s, it has only recently (2016) begun attracting attention in the fields of agriculture and remote sensing [46,47,48,49].

The hyperparameters for the LSTM classifier were identified empirically. We found that the model performs best with only one LSTM layer containing eight units. The activation function of choice was tanh, and the best optimizer was found to be the Stochastic Gradient Descent (SGD). Additionally, a dropout of 0.5 was used to reduce overfitting.

3.2.2. CNN

Convolutional Neural Networks (CNNs) are a type of neural network designed to process data that come in the form of multiple arrays [50]. In the case of multispectral satellite imagery, each pixel can be represented by a 1D array. In a time-series, the shape of the array becomes (s, t, b), where “s” is the number of samples, “t” refers to the number of dates in the time-series, and “b” indicates the number of multispectral bands used.

What makes CNNs powerful is their ability to recognize patterns that they learn regardless of their location. Because the patterns they learn are translation invariant, they need less training samples than regular dense layers to make generalizations [51]. The input to the CNN was 14 sets of 10 (14, 10). 14 dates, and 10 Sentinel-2 bands. Much like LSTM, the hyperparameters of the CNN were identified empirically. The best weights were generated using one unidimensional CNN layer with 16 filters, a kernel size of 1, and rectified linear unit (relu) activation function. A dropout of 0.5 was used to correct for overfitting. Prior to the output layer, a fully connected layer of eight units was used. The best activation function for this layer was also relu, and SGD was the best choice for the optimizer.

3.2.3. CerealNet

For the purposes of this study, we developed a unique deep neural network model, specifically designed to accurately identify and discriminate between four cereal crop types, hence the name CerealNet. This is a hybrid architecture with two branches and two separate inputs. The flowchart gives an overview of the architecture used (Figure 4). The LSTM branch is made of two layers with the same number of units (eight). The transfer function for each layer is tanh. This branch takes the Normalized Difference Vegetation Index (NDVI) as input (Equation (2)). This input is a handcrafted time-series feature derived from Sentinel-2 bands. In this model, the NDVI time series of size 14 (14 dates) is used to simulate the phenological cycle of the four cereal crops. The second branch is made from one unidimensional CNN layer with the goal of extracting spectral features from the Sentinel-2 reflectance time series. The optimal kernel size was found to be 1 and the number of filters used was set to 8, matching the size from the last LSTM layer. Additionally, a unidimensional MaxPooling layer of size 3 was used. The temporal and spectral features extracted were then flattened and passed through three fully connected (dense) layers with decreasing width (512, 128, 32), with the same activation function as the 1D-CNN (relu). After the first dense layer (512 units), a dropout of 0.3 was applied to prevent overfitting. The softmax transfer function was used for the output layer. This dense layer has five nodes, one for each crop type and an additional node for the class “other”. We found that the model performed best with the SGD optimizer. As CerealNet is a multi-class classifier, the loss was calculated using the categorical cross-entropy function. Categorical accuracy, F-measure, and Cohen Kappa were calculated, as they provide useful information on the performance of the model. After the compilation of the model, the network had 144,605 trainable parameters.

$$NDVI=\frac{NIR-RED}{NIR+RED}$$

(2)

where $NIR$ represents the NIR band (B8) and $RED$ corresponds to the red band (B4).

4. Results and Discussion

4.1. Spectro-Temporal Analysis

In order to study the evolution of cereal crops throughout their vegetative cycle, the spectral profiles of the four cereal crops were extracted (Figure 5). We discovered that all four crops exhibit the same spectro-temporal patterns throughout the season, with the profiles distinctly tracing the different growing stages. The amplitude difference between the reflectance in the green and NIR bands is minimal during the tillering stage; after that, it rises gradually, reaching its maximum value in mid-March. After this date, the reflectance values in the NIR bands drop in a uniform way as the crops reach the maturity stage. Owing to a difference in its physiology, oats have a higher reflectance in NIR throughout April, effectively giving this crop a distinct characteristic that allows for its identification. This feature is further reflected in Figure 6, where the NDVI of oats stays saturated for much longer than that of the other three crops, only dropping by May. For soft wheat, durum wheat, and barley, the difference in reflectance was not as prominent. We found that both wheat classes had a lower reflectance in the green band in 22–24 March 2022 (Figure 7). Additionally, soft wheat tends to have a lower reflectance in comparison with durum wheat across the NIR bands at the start of the season. As for barley, it had a higher reflectance value in the Short Wave Infrared (SWIR) bands at the end of the agricultural season.

4.2. CerealNet Results

After training the CerealNet model on the Gharb dataset, it was applied to the multi-temporal Sentinel-2 imagery of the study area with the goal of illustrating the output of the model. Figure 8 shows the output cereal crops map. The output of the model shows that cereal crops cover the vast majority of the study area. Except for the city of Sidi Slimane located to the north and the steep hills in the south-west, cereal plants seem to cover the extent of the whole zone. In fact, cereal crops are sown on more land than all the other crops combined, with an area of 23,484 ha, making over 60% of the study area. The rest of the pixels in the output map represented built-up areas and non-cereal crops, which amounted to 15,420 ha.

The cereal crop map shows a clear dominance of soft wheat over the other cereal classes, covering over 75% of the cereal area. Barley ranks second, with 11.5% area of coverage. Oats are not too far behind, with 9% area sown. Durum wheat, however, has the smallest contribution to the total area sown by cereal with only 4%. The statistics derived from the predicted cereal crop map were in line with the ratio of soft wheat, durum wheat, and barley reported by the Haut Commisariat au Plan (HCP) of Morocco for the Rabat-Sale-Kenitra region where the study area is located in 2021 [52]. The spatial distribution of the four cereal crops is not random. After analyzing the geography of the region, it is clear that soft wheat, durum wheat, and oats favor the plain with its small slope angles, whereas barley has a more frequent abundance in the southwest, where the altitudes are much higher and slopes steeper. This distribution is due to the high adaptability of barley to harsh terrain. Even though we used a pixel-based approach, parcel limits are clearly distinguishable in the map. This is especially true for oats, which could be found adjacent to or surrounded by soft wheat parcels.

When it comes to parcel size, all crop classes—apart from barley—are sown in groups of large parcels that can reach up to more than 200 $\mathrm{h}\mathrm{a}$. The size of barley parcels is relatively smaller, not surpassing 80 $\mathrm{h}\mathrm{a}$. Oats, on the other hand, have very small parcels on average, not surpassing 5 ha, usually in groups. Unlike soft wheat, durum wheat, and barley, this fodder crop’s main purpose is feeding horses, which explains the small parcel size in comparison with other cereal crops. In fact, there are some cases where oats were grown in the limits of other cereal parcels as a measure of protection against cattle. Sheep would eat the oats at the border and not intrude on the soft wheat inside the parcel.

Out-of-sample data were used to evaluate the performance of the model. We noted an overall categorical accuracy of 95%, which indicates that the model was successful in the generalization process. A confusion matrix was used to provide a detailed view of the performance of CerealNet in classifying the four cereal classes (

Table 4. Confusion matrix of the CerealNet classification.

	Barley	Soft Wheat	Durum Wheat	Oats	Other	UA	PA
Barley	788	134	10	0	24	0.84	0.82
Soft wheat	155	5355	4	0	90	0.96	0.96
Durum wheat	0	99	916	0	0	0.98	0.90
Oats	0	0	0	508	10	1.00	0.97
Other	0	3	0	0	3765	0.97	1.00

). The rows represent the ground truth target values, whereas the columns represent the model’s predictions. The maximum number of pixels is located along the diagonal of the matrix. This indicates that the model was able to extract meaningful features from the data, which was useful in discriminating between the four cereal crops. Due to the distinctive characteristics of oats, which were discussed previously, our model was able to correctly identify it even though it is a minority class. In fact, oats had the highest user and producer accuracy (UA and PA) of the four cereal crops. Barley, on the other hand, had a moderate level of confusion with all classes and therefore had the lowest UA and PA (<85%), as it closely resembles soft wheat and durum wheat.

4.3. Comparison with State-Of-The-Art Classifiers

The proposed methodology was compared with two traditional ML classifiers (SVM and RF) (

Table 5. Accuracy comparison of the traditional ML classifiers.

Crop	SVM			RF
	UA	PA	F1-Score	UA	PA	F1-Score
Barley	0.73	0.67	0.70	0.79	0.83	0.81
Soft wheat	0.91	0.93	0.92	0.96	0.91	0.94
Durum wheat	0.95	0.63	0.76	0.93	0.93	0.93
Oats	0.78	0.96	0.86	1.00	0.90	0.95
Other	0.96	1.00	0.98	0.93	1.00	0.96
Total	0.86	0.84	0.84	0.92	0.91	0.92

) and two DL models (LSTM and CNN) (

Table 6. Accuracy comparison of the DL classifiers.

Crop	CNN			LSTM			CerealNet
	UA	PA	F1-Score	UA	PA	F1-Score	UA	PA	F1-Score
Barley	0.85	0.82	0.86	0.80	0.81	0.81	0.84	0.82	0.83
Soft wheat	0.96	0.91	0.94	0.94	0.87	0.90	0.96	0.96	0.96
Durum wheat	1.00	0.93	0.95	0.85	0.90	0.87	0.98	0.90	0.94
Oats	1.00	0.99	0.99	0.96	0.76	0.85	1.00	0.97	0.98
Other	0.89	1.00	0.91	0.89	1.00	0.94	0.97	1.00	0.98
Total	0.94	0.92	0.93	0.89	0.87	0.87	0.95	0.93	0.94

). We found that CerealNet outperformed the competition with an F1-score of 0.94, and high class accuracy. Among the classifiers used in this study, the SVM had the lowest performance metrics, with an F1-score of 0.84. RF, on the other hand, reached an F1-score of 0.92, 8% higher than that of the SVM. Out of the three DL classifiers, LSTM performed the worst. In fact, LSTM is the only classifier (other than the SVM) that had an F1-score lower than 0.90. By contrast, the CNN had a much higher ranking, only 1% lower than the F1-score of CerealNet.

5. Conclusions

In this study, we successfully developed a novel deep learning architecture for identifying four spectrally similar cereal crops using multi-temporal Sentinel-2 imagery. The hybrid LSTM-Conv1D model, dubbed CerealNet, was evaluated using ground truth data collected from the Gharb region in Morocco. We noted an out-of-sample categorical accuracy of 95% and an F1-score of 94%. These values were higher than those of several state-of-the-art deep learning crop classifiers with which we compared our model. Despite the fact that the dataset utilized had a significant imbalance in the number of samples, our model was able to generalize such that the class accuracy was unaffected by the sample size. As a matter of fact, oats, while having the lowest number of ground truth samples, had the highest accuracy and F1-score of the four cereal classes. This is due to the specific temporal characteristics of oats, which makes it simpler to distinguish them from other cereal crops. Barley had the lowest F1-score (0.83) out of all the classes, due to the fact that it is too similar to both common and durum wheat. We suggest gathering more ground truth samples to address this problem, as well as using a longer time-series. During this study, only cloud-free Sentinel-2 level 2-A imagery was used to build the dataset. The presence of clouds in the study area reduced the amount of usable data. It is well known that the length of the time series has a direct effect on the performance of crop mapping models [1]. This is especially true if the missing imagery coincides with key phenological stages. Increasing the length of the time series by adding SAR (Synthetic Aperture Radar) imagery could be beneficial in increasing the model’s performance [53,54]. In fact, thanks to the modularity of CerealNet, SAR could be used to estimate the NDVI when optical imagery is not available [55]. Additionally, SAR polarimetric bands could be used as a separate feature set to identify crops [30].