Deep Learning in the Mapping of Agricultural Land Use Using Sentinel-2 Satellite Data

Abstract

Continuous observation and management of agriculture are essential to estimate crop yield and crop failure. Remote sensing is cost-effective, as well as being an efficient solution to monitor agriculture on a larger scale. With high-resolution satellite datasets, the monitoring and mapping of agricultural land are easier and more effective. Nowadays, the applicability of deep learning is continuously increasing in numerous scientific domains due to the availability of high-end computing facilities. In this study, deep learning (U-Net) has been implemented in the mapping of different agricultural land use types over a part of Punjab, India, using the Sentinel-2 data. As a comparative analysis, a well-known machine learning random forest (RF) has been tested. To assess the agricultural land, the major winter season crop types, i.e., wheat, berseem, mustard, and other vegetation have been considered. In the experimental outcomes, the U-Net deep learning and RF classifiers achieved 97.8% (kappa value: 0.9691) and 96.2% (Kappa value: 0.9469), respectively. Since little information exists on the vegetation cultivated by smallholders in the region, this study is particularly helpful in the assessment of the mustard (Brassica nigra), and berseem (Trifolium alexandrinum) acreage in the region. Deep learning on remote sensing data allows the object-level detection of the earth’s surface imagery.

1. Introduction

In agricultural land, several factors affect the mapping of smallholder farm plots, such as the multiple types of land management techniques, the scale of plots, vegetation variation, and lack of accurate field boundaries among a majority of the land management system [1,2]. Thus, the mapping of smallholder farmers’ farms, their operation, and the vegetation area spatial distribution remain the key components in monitoring [3,4,5]. In India, small farmers account for approximately 86% of operational landholdings [6,7], which significantly contributes to the gross domestic product (GDP), as well as achieving sustainable development goals (SDG) [8,9,10]. Therefore, it is necessary to monitor the agricultural land on a regular basis, as crop insurance is also an integral component of agriculture planning, and estimating crop yields, acreages, and crops at the block/tehsil level [11,12]. However, there are many challenges involved in conducting field surveys, such as cost constraints, weather conditions, and topography of the land [13,14].

Currently, remote sensing via space-borne satellite sensors plays an active role in decision-making support via the delivery of accurate data with temporal and spatial resolution [15,16]. In contrast to earlier studies, it was difficult to distinguish the smaller plots from low-or-medium resolution images [16,17,18,19]. However, with the integration of deep learning in moderate-resolution datasets, detailed and accurate information can be extracted in a more accurate and detailed way. To classify or categorize the vegetation types, numerous remote sensing datasets, e.g., Sentinel-1 (microwave-based), Sentinel-2 (optical-based) [20], Scatterometer [21,22,23], Landsat-8 [24], and many more were utilized to provide the cloud-free information [25,26]. Among the various satellite datasets, Sentinel-2 attracts the interest of many researchers due to its finer spatial resolution, free availability of datasets, and applicability in vegetation type/area classification and yield forecasting [27,28,29].

The processing of satellite datasets with advanced models, i.e., object-based classification or deep learning, allows the estimation yield data for highly commercial vegetation types, such as wheat berseem and mustard [30]. Previous literature has shown the effectiveness of different machine learning or deep learning algorithms, such as autoencoder (AE), stacked AE (SAE), convolutional neural network (CNN), recurrent CNN (R-CNN), fully convolutional network (FCN), recurrent NN (RNN), graph NN (GNN), and restricted Boltzmann machine (RBM) [30,31,32,33].

Table 1. A brief summary of different classification algorithms.

References	Classes	Accuracy	Satellite Dataset	Classifiers
[40]	10	77%	WV-2 1	ANN 2
[41]	22	86.2%	Colour Plant Images	DCNN 3
[42]	11	85%	Sentinel-1	CNN 4
[25]	11	96%	Sentinel-1	RNN 5, K-NN 6, RF 7 and VSM 8
[43]	5	87.3%	Landsat-8, Sentinel-2	ML 9
[44]	14	92%	Sentinel-2	ML 9 and DL 10
[45]	8	94.94%	Sentinel-1, Sentinel-2	TWINNS 11
[46]	3	97.53%	Digital images	R-CNN 12
[47]	4	94.85%	Landsat-8, Sentinel-2	FCNs 13
[48]	15	96.5%	Sentinel-2	R-CNN 12
[49]	10	98.7%	Sentinel-2	CNN 4 and R-CNN 12
[50]	2	91%	Landsat-8, Sentinel-2	DL 10
[27]	5	88%	Sentinel-2	DL 10
[17]	21	77.6%	Sentinel-2	RF 7, SVM 14
[15]	8	90.33%	Sentinel-1, Sentinel-2	RF 7
[6]	5	97.2%,	Sentinel-2	DLCD 15, RF 7, CNN 4, SVM 14
[8]	6	96.7%	Landsat-5, Landsat-8	CA-ANN 16, SVM 14
[3]	9	97.22%	Sentinel-2	KNN 17, RF 7
[1]	6	95.48%	Landsat-8, Sentinel-2	RF 7, SVM 14

¹ WV-2: World View-2, ² ANN: Artificial Neural Network, ³ DCNN: Deep Convolutional NN, ⁴ CNN: Convolutional NN, ⁵ RNN: Recurrent NN, ⁶ K-NN: K-Nearest Neighbors, ⁷ RF: Random Forest, ⁸ VSM: Vector Support Machines, ⁹ ML: Machine Learning, ¹⁰ DL: Deep Learning, ¹¹ TWINS: TWIn NN for Sentinel Dataset, ¹² R-CNN: Recurrent-Convolutional NN, ¹³ FCN: Fully Convolutional Network, ¹⁴ SVM: Support Vector Machines, ¹⁵ DLCD: Deep Learning Change Detection, ¹⁶ CA-ANN: Cellular Automata-ANN, ¹⁷ KNN: K-Nearest Neighbour.

shows the summary of the applicability of different satellite datasets on classification algorithms [33]. In addition, various authors investigated the effectiveness of using deep learning with different satellite datasets, such as hyperspectral [34], Synthetic Aperture Radar (SAR) [35], POLSAR (Polarimetric SAR) data [36,37,38,39] and many more [33].

Deep learning is dependent on the trained model for reducing user intervention and making the process semiautonomous or autonomous [51,52]. On the other hand, deep learning models require more computational power compared to machine learning algorithms [21,53,54]. Furthermore, the extraction of different land-use types from a medium-resolution satellite dataset is a challenging task in deep learning. This may be due to the impact of the atmospheric/radiometric influence on the medium-resolution satellite dataset. Thus, analysis of the impact of deep learning in the identification of land-use types from medium-resolution satellite datasets is essential.

In this study, deep learning (U-Net) has been implemented to map the different agricultural land use types using Sentinel-2. The objectives of the study included: (a) Implementation of U-Net-based deep learning over a region of Punjab, India; (b) detection of major agriculture land use types, i.e., wheat, berseem, mustard, other vegetation, water, and buildup; (c) validation and comparative analysis with RF classifier. This area was selected because of its agriculture-intensive zone, where vegetation lands have been divided into small holdings [55]. In addition, it contains a healthy mix of various types of land use and contains significant areas under both edible vegetation, most of which is wheat, and cash crops, which are mostly mustard. Furthermore, the area also contains several large species of fish as a result of the conversion of former fields into aquaculture ponds.

2. Study Area and Dataset

The region-of-interest (ROI) has been acquired from Fatehgarh Sahib (Punjab) India at geographic coordinates of 30°33′ N to 30°35′ N and 76°29′ E to 76°32′ E, with a coverage of 443 hectares, as shown in Figure 1. The study area includes various class categories, such as wheat, berseem, mustard, other vegetation (unable to separate using medium resolution imagery due to very small land holding), water, and buildup. The Punjab region (India) is an important food grain state of India that contributes the greatest portion of the wheat and rice stock. The ROI was acquired during the winter season on a cloud-free day, i.e., 19 February 2018, from Sentinel-2 (https://earthexplorer.usgs.gov/, accessed date: 15 September 2022). During this period, the different crops, i.e., wheat, berseem, and mustard are in the budding/flowering stage and the atmosphere is nearly clear, which helps in the acquisition of imagery with the least impact of clouds on remote sensing images. The spatial resolution of the Sentinel-2 data varies from 10 m to 60 m with respect to thirteen spectral wavelengths (0.443–2.190 µm). Moreover, the high-resolution dataset has been utilized from the Pléiades constellation for accuracy assessment and training purposes.

3. Methodology

As shown in Figure 2, the proposed methodology involves the acquisition of Sentinel-2 satellite data of ROI, preprocessing of the data to overcome atmospheric and radiometric errors, implementation of deep learning-based U-Net and machine learning based RF classifier to detect the different land use types from ROI, and validation or accuracy assessment procedures. This framework solves geographic problems in detection of agricultural land (the average size of the landholdings in India was 1.08 hectares as per the Agriculture Census) with the help of deep learning-based U-Net [22,56]. It requires training the model using input label rasters that indicate known feature samples [36,57].

3.1. Preprocessing of the Input Data

The Sentinel-2 data needs to be preprocessed to overcome atmospheric and radiometric errors. The Sen2Cor v2.9 module (for Sentinel-2 Level 2A, Sentinel application platform (SNAP) version) has been utilized to perform the atmospheric and radiometric corrections. In addition, it effectively corrects sun angle variations, daytime haze effects, and smaller haze effects, but it does not remove clouds. Hence, cloud-free images are recommended when processing the Sentinel-2 data via Sen2Cor. Once the data is preprocessed, deep learning or machine learning can be implemented to generate classified maps.

3.2. Deep Learning and Machine Learning

As an essential part of machine learning, deep learning models work on the principle of neural networks to learn and recognize data [58]. It can be defined as (a) supervised, (b) unsupervised, and (c) transfer-based [8,59,60]. A supervised learning process associates a target subject with a class affiliation explicitly; in an unsupervised learning process, data structures are used to associate class affiliation with the target concept [15]. In the third category, information is utilized across different activities in a coordinated manner to increase efficiency [17]. The object-level detection accuracy of the model generally depends on the depth of the neural network, as in CNN and GNN [31,52]. The U-Net, which evolved from the CNN, was utilized in many other applications, such as glacier research [58], biomedical imaging [61], sea-ice or ice shelf mapping [57], land boundaries [56], and big data remote sensing for next-generation mapping [62].

The U-Net as a supervised semantic segmentation network has been implemented in the present work to map agricultural land use types [61]. To implement the U-Net, deep learning (Ver. 1.1) in ENVI’s software v5.6 has been utilized. This architecture is encoder-decoder and mask-based to classify the satellite data into different land use types. To perform the classification, an open-source software library, named TensorFlow model, is utilized as a main part of the deep learning procedure. The TensorFlow model provides flexibility, portability, and optimization of performance [63]. As shown in Figure 2, there are two major parts of the U-Net architecture, i.e., downscaling (used to increase the robustness with respect to the distortion in the imagery) and upscaling (used for restoration and decoding of the object features with respect to the input imagery).

In ENVI’s deep learning model, there are two options, i.e., ENVINet5 (single class classification) and ENVI Net-Multi (multiple classes categorization), to classify input data [64]. To train the TensorFlow model, multiple samples have been selected for each class category, i.e., wheat, berseem, mustard, other vegetation, water, and buildup. In the training process, the TensorFlow model converts the spatial/spectral information in input imagery into a class activation or thematic map. Once the model is trained, the georeferenced activation and classified maps are generated. The activation maps are the fractional maps generated for each class category and they help improve the classified maps with the selection of manual or automatic threshold methods [6]. For the comparative analysis, a machine learning-based RF classifier has also been implemented. The RF classifier as a random decision forest allows the construction of multiple decision trees at training time to classify the datasets.

4. Results and Discussion

Initially, the Sentinel-2 data was acquired and preprocessed through the Sen2Cor v2.8 module to generate the reflectance imagery, as shown in Figure 1 [65,66]. To implement the deep learning classifier, the model needs to be trained for different class categories. A total of 30–35 polygons are repeated throughout the image as inputs for this training for classifying rabi season crops with deep learning and random forest classifiers [20,67]. The 100–150 pixels were considered in each polygon segment during the training process [7,68,69]. Afterwards, the activation (fractional) maps (i.e., wheat, berseem, mustard, other vegetation, water, and buildup area) are generated via ENVI’s U-Net-based deep learning, as shown in Figure 3. The intensity level of the activation maps varies from the maximum (represented by pure white) to the minimum (represented by pure black). Activation maps allow the reproduction of classified maps with improved accuracy while dealing with medium-resolution data. Furthermore, the different threshold methods could also be tested to generate the accuracy of classified maps in ENVI’s U-Net-based deep learning. Reference [6] highlighted the significance of activation maps in the detection of seasonal agricultural variations.

Figure 4 represents the visual interpretation of classified outputs generated from U-Net-based deep learning and RF-based machine learning. To perform the statistical analysis, the various parameters have been computed, i.e., producer’s accuracy (PA), user’s accuracy (UA), overall accuracy (OA), kappa coefficient (k), Reference Total (RT); Classified Total (CT); and Number of correct (NC).

Table 2. Accuracy assessment computed from U-Net and RF classification algorithms.

	Total No. of Class Categories	Parameters of Accuracy Assessment
		RT (%)	CT (%)	NC (%)	PA (%)	UA (%)	k
U-Net	Wheat	38.6	39.2	39.4	98.9	97.4	0.95
	Berseem	12.6	12.4	12.4	96.8	98.3	0.98
	Mustard	5	5.2	5.1	100	96.1	0.95
	Other vegetation	33.6	34.2	34.3	100	98.2	0.97
	Water	5	4.6	4.4	88	95.6	0.95
	Buildup	5.2	4.4	4.4	84.6	100	1.00
	OA = 97.8%; k = 0.9691
RF	Wheat	38.2	39.2	39	98.9	95.9	0.93
	Berseem	12.4	12.4	12.4	95.1	95.1	0.94
	Mustard	5.8	5.2	5.6	89.6	100	1.00
	Other vegetation	33.6	34.2	33.8	98.7	95.3	0.93
	Water	5.2	4.6	4.7	88.4	100	1.00
	Buildup	4.8	4.4	4.5	91.6	100	1.00
	OA = 96.2%; k = 0.9469

Note: For accuracy assessment, 500 samples have been selected from each class category.

represents the accuracy assessment computed from U-Net-based deep learning and RF-based machine learning. It was observed that U-Net-based deep learning performed better (OA = 97.8% and k = 0.9691), as compared to RF-based machine learning (OA = 96.8% and k = 0.9691).

In the previous studies [6], the authors attempted to assess ENVI’s U-Net architecture in the detection of seasonal variation with limited class categories. As an objective of the present work, the performance of ENVI’s U-Net architecture was evaluated over agricultural land in the mapping of different vegetation class categories, i.e., wheat, berseem, mustard, and other vegetation using medium-resolution Sentinel-2 satellite imagery. From the results, it is depicted that ENVI’s U-Net architecture has the potential for the extraction of different vegetation class categories. As compared to many other versions of the U-Net model architecture, ENVI’s U-Net is easy to implement without any explicit programming [56,58]. It also included a greater number of features for different applications, such as boundary mapping, threshold algorithms, and an easy user interface [64].

As ENVI’s U-Net (deep learning v1.1, ENVI v5.6) is in the initial stage of development, the major challenges involved in its use included the lack of full customization of various model parameters, inability to classify the entire input raster when least values of parameters were selected, and lack of accuracy in object-level detection. However, the accuracy of classification could be improved by applying the in-built manual or automatic threshold algorithms (e.g., Otsu, empirical, etc.) on the activation maps in ENVI’s U-Net. This procedure allows the regeneration of thematic maps without the re-computation of the entire deep learning model. Furthermore, the outcomes of ENVI’s U-Net could also be improved in terms of more customization and integration with other models. As far as the cloud problem is concerned [70], microwave data can be utilized or infused with optical datasets to get accurate outcomes.

5. Conclusions

In this work, ENVI’s deep learning (U-Net) has been evaluated in the estimation and mapping of different vegetation class categories, i.e., wheat, berseem, mustard, and other vegetation, using medium-resolution Sentinel-2 satellite data. As part of the comparative analysis, a well-known and commonly used RF classifier has also been implemented and it was concluded that U-Net-based deep learning performed marginally better compared to RF-based machine learning. Although it performed well in the automatic identification of land-use features, many challenges are still involved in ENVI’s deep learning, which need to be addressed with more flexibility in the selection of parameter values and integration with self-designed programming models. It also needs to be improved by the addition of features with respect to different scientific domains. As far as agriculture is concerned, it is necessary to correctly identify different seasonal crops on land plots to successfully update images to provide an up-to-date inventory of crops across seasons, making agriculture policy and mapping more accurate. Further studies may comprise the deep learning of the multisource data fusion in the cloud-free monitoring of agricultural land types.