Monitoring of Soybean Maturity Using UAV Remote Sensing and Deep Learning

Abstract

Soybean breeders must develop early-maturing, standard, and late-maturing varieties for planting at different latitudes to ensure that soybean plants fully utilize solar radiation. Therefore, timely monitoring of soybean breeding line maturity is crucial for soybean harvesting management and yield measurement. Currently, the widely used deep learning models focus more on extracting deep image features, whereas shallow image feature information is ignored. In this study, we designed a new convolutional neural network (CNN) architecture, called DS-SoybeanNet, to improve the performance of unmanned aerial vehicle (UAV)-based soybean maturity information monitoring. DS-SoybeanNet can extract and utilize both shallow and deep image features. We used a high-definition digital camera on board a UAV to collect high-definition soybean canopy digital images. A total of 2662 soybean canopy digital images were obtained from two soybean breeding fields (fields F1 and F2). We compared the soybean maturity classification accuracies of (i) conventional machine learning methods (support vector machine (SVM) and random forest (RF)), (ii) current deep learning methods (InceptionResNetV2, MobileNetV2, and ResNet50), and (iii) our proposed DS-SoybeanNet method. Our results show the following: (1) The conventional machine learning methods (SVM and RF) had faster calculation times than the deep learning methods (InceptionResNetV2, MobileNetV2, and ResNet50) and our proposed DS-SoybeanNet method. For example, the computation speed of RF was 0.03 s per 1000 images. However, the conventional machine learning methods had lower overall accuracies (field F2: 63.37–65.38%) than the proposed DS-SoybeanNet (Field F2: 86.26%). (2) The performances of the current deep learning and conventional machine learning methods notably decreased when tested on a new dataset. For example, the overall accuracies of MobileNetV2 for fields F1 and F2 were 97.52% and 52.75%, respectively. (3) The proposed DS-SoybeanNet model can provide high-performance soybean maturity classification results. It showed a computation speed of 11.770 s per 1000 images and overall accuracies for fields F1 and F2 of 99.19% and 86.26%, respectively.

1. Introduction

Soybeans are a high-quality source of plant protein and raw materials for the production of hundreds of chemical products [1,2]. China’s soybean-growing areas include the Northeast China Plain [3] and the North China Plain [4] (ranging from the north latitude of 30° to 48°). Soybean breeders must develop early-maturing, standard, and late-maturing varieties for planting at different latitudes to ensure that soybean plants fully utilize solar radiation. Therefore, timely and accurate monitoring of soybean breeding line maturity can facilitate soybean breeding decision-making and agricultural management [5,6,7,8].

Traditional methods for measuring field breeding line maturity are time-consuming and labor-intensive [7]. Meanwhile, the expertise and bias of the investigators can affect the accuracy of field surveys. Breeding fields have thousands of breeding lines with different maturation times. Manual surveys cannot quickly provide high-frequency breeding line maturity information to meet harvesting and yield measurement scheduling requirements. Unmanned aerial vehicle (UAV) remote sensing technology can be used to collect high-resolution crop canopy images and has thus been widely used in precision agricultural crop trait monitoring [9,10,11,12]. Compared with satellite and airborne remote sensing technologies, UAV remote sensing technology is relatively inexpensive and flexible in its operation, and it requires less space for landing and takeoff [13]. More importantly, the digital images obtained by low-altitude UAVs have a high ground spatial resolution (centimeter-scale or higher); thus, they contain rich crop-canopy surface information for crop phenotypic research [14,15]. In recent years, UAV remote sensing technology has been widely used to collect crop trait information [9,10,11,12,16,17]. UAVs equipped with high-definition digital cameras can acquire soybean canopy ultrahigh ground spatial resolution digital images over a field scale [14,15]. Many UAV-based methods have been proposed for monitoring various types of crop trait information, including the leaf area index (LAI) [18], leaf chlorophyll content [18,19,20,21], biomass [15,22], and crop height [23].

Machine learning has been successfully applied in several areas, such as image classification, target recognition, and language translation [24,25,26]. In recent years, machine learning techniques have been widely used to recognize various crop traits based on remote sensing images [27]. Gniewko et al. [28] used an artificial neural network (ANN), growing degree days, and total precipitation to estimate soybean yields. Letícia et al. [29] conducted a study to identify nematode damage to soybeans through the use of UAV remote sensing and a random forest (RF) model. The results obtained by Eugenio et al. [30] and Paulo et al. [31] indicated that machine learning techniques are efficient and flexible for remote sensing monitoring of soybean yields. Abdelbaki et al. [32] conducted a study to predict the soybean LAI and fractional vegetation cover (FVC) based on the RF model and UAV remote sensing. Compared with traditional machine learning methods (e.g., SVM and RF), deep learning methods such as long short-term memory (LSTM) [33,34], deep convolutional neural networks (CNNs) [26,35], and transformers [14] have been applied to image recognition, medical image analysis, climate change, and Weiqi game analysis, where they can provide results with similar or even higher precision than human experts. Deep learning uses multiple layers to extract higher-level features from the raw input. In recent years, deep learning techniques have been widely used to recognize various crop traits in remote sensing images, e.g., in leaf disease identification, weed identification, and crop trait recognition [1,26,33,34,35,36,37]. Wang et al. [34] developed an LSTM model by integrating MODIS LAI data to predict crop yields in China. Khan et al. [37] used a YOLOv4 model to identify apple leaf diseases in digital images captured by mobile phones. Zhang et al. [26] used a YOLOv4 model to identify weeds in digital photos of a peanut field. Khalied et al. [38] proposed a model based on MobileNetV2 for fruit identification and classification. Yonis et al. [39] proposed a CNN model adopting the VGG16 architecture for seed identification and classification. Notably, most of these widely used networks (e.g., YOLOv4 [40], ResNet50 [41], MobileNet [42], VGG16 [39], and InceptionResNetV2 [43]) did not take full advantage of shallow features. Shallow features derived from the shallow layers of CNNs are rich in image details, which are generally used in areas such as fine texture detection or small target detection [44,45]. Fusing the deep and shallow features of CNNs may improve performance in soybean maturity classification [44,45,46].

The objective of this work was to monitor soybean maturity using UAV remote sensing and deep learning. We designed a new convolutional neural network architecture (DS-SoybeanNet) to extract and utilize both shallow and deep image features to improve the performance of UAV-based soybean maturity information monitoring. We used a high-definition digital camera on board a UAV to collect high-definition soybean canopy digital images from two soybean breeding fields. We compared the UAV-based soybean maturity information monitoring performances of conventional machine learning methods (support vector machine (SVM) and random forest (RF)), current deep learning methods (InceptionResNetV2, MobileNetV2, and ResNet50), and our proposed DS-SoybeanNet method. Our results indicate that the proposed DS-SoybeanNet method can extract both shallow and deep image feature information and can realize high-performance soybean maturity classification.

2. Materials

2.1. Study Area

The study area was located at the Shengfeng Experimental Station (E: 116°22′10″–116°22′20″, N: 35°25′50″–35°26′20″, Figure 1) of the National Center for Soybean Improvement, Jiaxiang County, Jining City, Shandong Province, China. Jiaxiang County is situated on the North China Plain, with a warm continental monsoon climate, concentrated precipitation, and an average annual sunshine duration of 2405.2 h. The average annual temperature is 13.9 °C.

2.2. UAV Flights and Soybean Canopy Image Collection

We used a high-definition digital camera on board an eight-rotor electric UAV to collect high-resolution soybean canopy remote sensing images (

Table 1. Parameters of the UAV and digital camera used in this study.

UAV	Parameter	Camera	Parameter
UAV name	DJI S1000	Camera name	SONY DSC-QX100
Flight height	Approximately 50 m	Image size	5472 × 3648
Flight speed	Approximately 8 m/s	Image dpi	350
Flight time	>20 min	Aperture	f/11
		Exposure	1/1250 s
		ISO	ISO-1600
		Focal length	10 mm
		Channels	Red, green, blue
		Ground spatial resolution	0.016 m

). In the soybean breeding experimental field, the size of each planting area was approximately 2.5 m × 5 m. As shown in Figure 1, we selected two independent soybean planting fields (fields F1 and F2) in the study area to obtain soybean canopy digital images and maturity information.

For field F1, we conducted five UAV flights (29 July, 13 August, 31 August, 17 September, and 28 September 2015). A total of 2116 soybean canopy digital images and their maturity information were obtained, which were used to calibrate the SoybeanNet model. For field F2, we made only one observation on 30 September, 2015. There were immature, near-mature, mature, and harvested soybean breeding lines in field F2 on 30 September. A total of 546 planting areas were set up in field F2 for the mapping and independent evaluation of the DS-SoybeanNet model.

The soybean image collection and image stitching process mainly included the following three steps:

(1)

Before the UAV took off, we set the flight route information according to the field size; the heading and lateral overlap were set to 80%. Table 1 shows the digital camera exposure parameters.

(2)

During the UAV flight, the soybean canopy images and corresponding position and orientation system (POS) information were collected using the digital camera, inertial measurement unit, and global positioning system device on board the UAV.

(3)

After the UAV flight, we imported the digital images and POS information into PhotoScan software to stitch together the high-definition digital images collected by the UAV. After the image stitching process, five soybean canopy digital orthophoto maps (ground spatial resolution (GSD): 0.016 m) for field F1 and one soybean canopy digital orthophoto map (GSD: 0.016 m) for field F2 were acquired.

2.3. Soybean Canopy Image Labeling

In this study, soybean maturity information was manually labeled. The labeling method was based on the standards of soybean harvesting. The labeling method is described in

Table 2. Standards used for labeling the soybean plots.

Label	Priority		Description
L0	Immature	Low	All upper canopy leaves are green or there are a few yellow leaves.
L1	Near-mature	High	Approximately half of the upper canopy leaves are yellow.
L2	Mature	Highest	The upper leaves of the canopy are yellow but have yet to be harvested.
L3	Harvested	Low	The soybean planting area has been harvested.

. For workers to customize schedules for harvesting soybean planting plots, four categories were used: immature (L0), near-mature (L1), mature (L2), and harvested (L3). L2 plots have the highest harvesting priority and need to be harvested as soon as possible, L1 plots have a high priority because the soybean will mature in less than a week, L0 and L3 plots have a lower priority because L0 plots generally take longer to grow, and no outdoor work is required for L3 plots.

Since different soybean breeding lines have different maturation times, the numbers of images corresponding to the four labels varied between the two fields. Sixty percent of the images of each type in the dataset were randomly chosen to train the model, and the remaining 40% were used to evaluate the model’s accuracy.

Table 3. Numbers of soybean images for model calibration and validation.

Label	Training Dataset (Field F1)	Validation Dataset (Field F1)	Independent Validation Dataset (Field F2)
L0	542	318	64
L1	257	163	219
L2	70	52	198
L3	400	314	65
Total	1269	847	546
Enhancement	25,380	16,940	-

shows the numbers of samples used to train and validate the DS-SoybeanNet model. Figure 2 shows the soybean images used for model calibration and validation.

2.4. Data Enhancement

In this study, we produced a DOM for the entire area by mosaicking together the digital images collected during each UAV flight. Since an orthoimage has a uniform scale, the ground spatial resolutions and solar angles were the main differences between the five DOMs. We used image rotation (four rotation angles: 0° (i.e., the original image), 90°, 180°, and 270°) and scaling (four scaling factors: 1.0 (i.e., the original image), 1.2, 1.5, 1.8, and 2.0) to enhance the soybean canopy image dataset collected from field F1. Image rotation and magnification helped us to obtain soybean canopy images with different resolutions and angles; in addition, they helped prevent overfitting of the model due to the small number of samples collected in the field.

After data enhancement, the number of original soybean canopy images obtained from field F1 was increased by 20 times. The number of independent validation datasets obtained from field F2 was not increased. In this study we used the Python open-cv and NumPy libraries to extract, rotate, and magnify the soybean canopy images.

3. Methods

3.1. Proposed DS-SoybeanNet

CNNs were originally proposed based on the receptive field mechanism in biology and they are a widely used deep learning technology [47]. CNNs are designed to process images with a lattice-like structure. The multilayer convolution, weight sharing, and rotational-shift invariance of CNNs make them effective in image classification and feature recognition. The deep and complex features extracted by CNNs are often used to effectively describe differences between different image categories and can be used to quickly and accurately complete classification tasks. Currently, widely used networks (e.g., ResNet50 and MobileNetV2) ignore shallow image feature information. We designed a network structure (Figure 3) that considers both shallow and deep image features to enhance the model’s generalization ability. The advantage of DS-SoybeanNet is that the shallow and deep features are linked together by means of a concatenation module. Consequently, DS-SoybeanNet can extract and utilize both shallow and deep image features to improve the accuracy of soybean maturity information classifications. Figure 3 shows the architecture of DS-SoybeanNet. DS-SoybeanNet contains five convolutional layers, five flattening modules, one concatenation module, and four fully connected layers. The layers are described as follows:

(1) Input layer

The input data were collected via UAV remote sensing technology in the form of soybean canopy orthophotos and were then manually labeled and cropped to produce sample data. The sample size was 108 × 108 × 3, and the sample data were divided into four types: immature, near-mature, mature, and harvested.

(2) Convolutional and pooling layers

The purpose of the convolution operation was to extract the different features of the input images. DS-SoybeanNet was designed with five convolutional layers; each convolutional layer was combined with the ReLU activation function to achieve delinearization. The pooling layers can reduce the dimensions of the feature maps by summarizing the presence of features in patches of the feature map.

(3) Flattening and concatenation layers

A flattening layer can reshape the feature maps into the dimensions required for the subsequent layers. A concatenation layer concatenates inputs along a specified dimension.

(4) Fully connected layers and output layer

Four fully connected layers were designed, and dropout layers were attached to the first three layers to prevent overfitting and improve model generalization. The output of the model was soybean maturity information derived from the input images.

3.2. Transfer Learning Based on InceptionResNetV2, MobileNetV2, and ResNet50

Transfer learning is a strategy for solving similar or related tasks using existing methods and data. Many deep learning networks show effective performance in image classification and target recognition from natural images (e.g., InceptionResNetV2 [43], ResNet50 [41], and MobileNetV2 [42]). Using a pretrained model to extract the features of remote sensing images can solve, to a certain extent, the problems involved with training a network for remote sensing image scene classification when there is a lack of training data. In this study, we used InceptionResNetV2, MobileNetV2, and ResNet50 as the pretrained deep learning models for transfer learning and performance comparisons with the proposed DS-SoybeanNet model.

(1)

ResNet50: The ResNet50 network contains 49 convolutional layers and a fully connected layer. The core CNN components are the convolutional filter and the pooling layer. ResNet50 is a CNN derivative with a core component skip-connection to circumvent the gradient disappearance problem. The ResNet structure can accelerate training and improve performance (preventing gradient dispersion).

(2)

InceptionResNetV2: The Inception module can obtain sparse or nonsparse features in the same layer. InceptionResNetV2 performs very well, but compared with ResNet, InceptionResNetV2 has a more complex network structure.

(3)

MobileNetV2: MobileNetV2 is a lightweight CNN model proposed by Google for embedded devices, such as mobile phones, with a focus on optimizing latency while considering the model’s size. MobileNetV2 can effectively balance latency and accuracy.

Transfer learning requires a low learning rate for retraining because the feature extraction module of the model already has some ability to extract image feature information after pretraining. An ideal learning rate can promote model convergence, whereas an unsuitable rate can cause training oscillations or even directly lead to the “explosion” of the loss value of the objective function. In addition to transfer learning methods based on InceptionResNetV2, MobileNetV2, and ResNet50, we also tested the performance of the AlexNet [48] and VGG16 [38] models to monitor soybean maturity.

3.3. SVM and RF

We also compared the soybean maturity information classification accuracy of our proposed DS-SoybeanNet with those of conventional machine learning models (SVM and RF). SVM is a generalized linear classifier that performs binary data classification in supervised learning [49]. Its decision boundary is the maximum marginal hyperplane solved for the learned samples, which reduces the classification problem to a convex quadratic programming problem. SVM has a low composition risk, its training is challenging to implement on large samples, and it is not ideal for solving multiclassification problems. RF is based on an integrated learning strategy, which combines multiple decision trees [50]. These decision trees are independent and unrelated to each other. Random forest uses the bagging strategy and repeated sampling to generate multiple trees. Under the bagging and bootstrap aggregation strategy, a subset of the samples are randomly selected from the dataset for training, and voting is conducted to obtain the average value as the resulting output. This strategy significantly avoids incorrect sample data, and thus shows improved accuracy.

3.4. Accuracy Evaluation

Figure 4 shows the experimental methodology used in this work. The canopy images of field F1 were used to calibrate and validate the models, whereas all canopy images of field F2 were used to validate the models.

The confusion matrix is a widely used tool for model accuracy evaluations.

Table 4. Confusion matrix.

Type	Predicted condition
Actual condition	Label	Positive (P)	Negative (N)
	Positive (T)	True Positive (TP)	False Negative (FN)
	Negative (N)	False Positive (FP)	True Negative (TN)

shows the confusion matrix for the binary classification problem. Accuracy and recall can be obtained based on the confusion matrix. Generally, a higher accuracy and recall indicate a higher classification accuracy.

$$\mathrm{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN}$$

(1)

$$Recall=\frac{TP}{TP+FN}$$

(2)

TP, TN, FP, and FN represent the true-positive, true-negative, false-positive, and false-negative categories, respectively, in the confusion matrix (Table 4). Confusion matrices are not limited to binary classification but can also be used for multiclass classification. In this study, we used the confusion matrix, accuracy, and recall to evaluate the soybean maturity classification accuracy of the proposed DS-SoybeanNet model.

4. Results and Discussion

4.1. Model Calibration and Validation Based on Field F1

We used the calibration dataset of field F1 to train the proposed DS-SoybeanNet, AlexNet, VGG16, InceptionResNetV2, MobileNetV2, ResNet50, SVM, and RF models. Each model was trained and validated three times, and the model with the highest performance was saved. The learning rates were set to 0.0005, 0.0001, and 0.00001 for the transfer learning models (InceptionResNetV2, MobileNetV2, and ResNet50), and the number of epochs was set to 100. For DS-SoybeanNet, we analyzed the model accuracy with different convolution window sizes.

4.1.1. Validation of AlexNet, VGG16, SVM, and RF

We tested the SVM and RF models for monitoring soybean breeding line maturity (

Table 5. Classification results of AlexNet, VGG16, SVM, and RF.

Label	SVM	RF	AlexNet	VGG16
L0	99.69%	99.06%	99.69%	98.74%
L1	100%	100%	99.39%	100%
L2	90.38%	90.38%	98.08%	84.62%
L3	99.04%	99.36%	99.36%	98.41%
Accuracy	92.31%	94.23%	99.44% *	97.99%

Note: * indicates the highest accuracy.

) based on the validation dataset from field F1. The L0, L1, and L3 classification recall values were higher than 99% for the traditional machine learning models (SVM and RF). The classification accuracies of SVM and RF were 92.31% and 94.23%, respectively. We also tested the performance of the AlexNet and VGG16 models (Table 5). The performances of AlexNet (99.44%) and VGG16 (97.99%) were higher than those of SVM (92.31%) and RF (94.23%).

4.1.2. Validation of Transfer Learning Based on InceptionResNetV2, MobileNetV2, and ResNet50

We also tested the performance of the three deep learning models using three learning rates.

Table 6. Classification results of transfer learning based on InceptionResNetV2, MobileNetV2, and ResNet50.

Label	InceptionResNetV2			MobileNetV2			ResNet50
	Rate 1	Rate 2	Rate 3	Rate 1	Rate 2	Rate 3	Rate 1	Rate 2	Rate 3
L0	98.09%	100%	99.69%	100%	100%	99.69%	99.69%	100%	99.69%
L1	96.93%	100%	98.16%	95.09%	96.32%	92.02%	100%	96.93%	98.16%
L2	82.69%	98.08%	98.08%	69.23%	84.62%	82.69%	88.46%	96.15%	94.23%
L3	99.36%	98.73%	99.04%	99.36%	97.77%	97.77%	99.04%	98.73%	99.36%
Accuracy	97.41%	99.49%	99.09%	96.93%	97.52%	96.46%	98.93%	98.77%	98.97%

Note: Rate 1 = 0.0005; Rate 2 = 0.0001; Rate 3 = 0.00001.

shows the accuracies of the models using different learning rates. The performances of the three deep learning models (InceptionResNetV2, MobileNetV2, and ResNet50) were similar when using different learning rates. Our results indicate that the soybean maturity classification accuracy of traditional machine learning models (RF: 94.23%; SVM: 92.31%) was lower than that of InceptionResNetV2, MobileNetV2, and ResNet50.

There were notable differences in recall among the four labels. For example, the L2 classification recall of InceptionResNetV2 was much lower than those of L0, L1, and L3 when the learning rate was 0.0005. The same was observed for MobileNetV2 and ResNet50, which had L2 classification recalls of 69.23% and 88.46%, respectively.

4.1.3. Validation of the Proposed DS-SoybeanNet Model

We tested the proposed DS-SoybeanNet model in the monitoring of soybean breeding line maturity.

Table 7. Classification results of the proposed DS-SoybeanNet.

Label		DS-SoybeanNet
		3 × 3	5 × 5	7 × 7	9 × 9	11 × 11	16 × 16	21 × 21
Recall	L0	100%	100%	100%	100%	100%	100%	100%
	L1	96.93%	100%	100%	100%	99.39%	99.39%	99.39%
	L2	92.31%	90.38%	90.38%	78.85%	88.46%	80.77%	75.00%
	L3	99.36%	99.36%	99.68%	99.36%	99.68%	96.50%	97.77%
Accuracy		98.70%	99.17% *	99.19% *	98.47%	99.06%	97.40%	97.52%

Note: * indicates the highest accuracy.

shows the classification results of the DS-SoybeanNet model with the convolution kernel size set to 3 × 3, 5 × 5, 7 × 7, 9 × 9, 11 × 11, 16 × 16, and 21 × 21. The results indicate that there was little difference in performance among the seven convolution kernel sizes (with classification accuracies ranging from 97.52% to 99.19%). The results suggest that the model had the best soybean maturity classification accuracy when the convolution kernel size was set to 5 × 5 (99.17%) or 7 × 7 (99.19%). Figure 5 shows the training accuracy and loss curves of the DS-SoybeanNet with kernel sizes of 5 × 5 and 7 × 7. These results indicate that the model reached convergence at about 40 epochs. Training the DS-SoybeanNet (5 × 5) for about 100 epochs could take about 40 min and 5 s.

Table A1. Details of the proposed DS-SoybeanNet with 5 × 5 kernels.

Layer (Type)	Output Shape	Param	Connected to
input_1 (Input Layer)	[(None,108,108,3)	0
conv2d (Conv2D)	(None,108,108,32)	2432	input_1 [0][0]
conv2d_1 (Conv2D)	(None,108,108,16)	12816	conv2d [0][0]
max_pooling2d_1 (MaxPooling2D)	(None,27,27,16)	0	conv2d_1 [0][0]
conv2d_2 (Conv2D)	(None,27,27,32)	12832	max_pooling2d_1 [0][0]
conv2d_3 (Conv2D)	(None,27,27,16)	12816	conv2d_2 [0][0]
max_pooling2d (MaxPooling2D)	(None,27,27,32)	0	conv2d [0][0]
max_pooling2d_2 (MaxPooling2D)	(None,13,13,32)	0	conv2d_2 [0][0]
max_pooling2d_3 (MaxPooling2D)	(None, 13,13,16)	0	conv2d_3 [0][0]
conv2d_4 (Conv2D)	(None,27,27,16)	6416	conv2d_3 [0][0]
flatten (Flatten)	(None,23328)	0	max_pooling2d [0][0]
flatten_1 (Flatten)	(None,11664)	0	max_pooling2d_1 [0][0]
flatten_2 (Flatten)	(None,5408)	0	max_pooling2d_2 [0][0]
flatten_3 (Flatten)	(None,2704)	0	max_pooling2d_3 [0][0]
flatten_4 (Flatten)	(None,11664)	0	conv2d_4 [0][0]
concatenate (Concatenate)	(None,54768)	0	flatten [0][0]
			flatten_1 [0][0]
			flatten_2 [0][0]
			flatten _3 [0][0]
			flatten _4 [0][0]
dropout (Dropout)	(None,54768)	0	concatenate [0][0]
dense (Dense)	(None,4096)	224333824	dropout [0][0]
dropout_1 (Dropout)	(None,4096)	0	dense [0][0]
dense_1 (Dense)	(None,512)	4195328	dropout_1 [0][0]
dropout_2 (Dropout)	(None,512)	0	dense_1 [0][0]
dense_2 (Dense)	(None,4)	4100	dropout_2 [0][0]
Total params: 228,580,564
Trainable params: 228,580,564
Non-trainable params: 0

and

Table A2. Details of the proposed DS-SoybeanNet with 7 × 7 kernels.

Layer (Type)	Output Shape	Param	Connected to
input_1 (Input Layer)	[(None,108,108,3)	0
conv2d (Conv2D)	(None,108,108,32)	4736	input_1 [0][0]
conv2d_1 (Conv2D)	(None,108,108,16)	25104	conv2d [0][0]
max_pooling2d_1 (MaxPooling2D)	(None,27,27,16)	0	conv2d_1 [0][0]
conv2d_2 (Conv2D)	(None,27,27,32)	25120	max_pooling2d_1 [0][0]
conv2d_3 (Conv2D)	(None,27,27,16)	25104	conv2d_2 [0][0]
max_pooling2d (MaxPooling2D)	(None,27,27,32)	0	conv2d [0][0]
max_pooling2d_2 (MaxPooling2D)	(None,13,13,32)	0	conv2d_2 [0][0]
max_pooling2d_3 (MaxPooling2D)	(None, 13,13,16)	0	conv2d_3 [0][0]
conv2d_4 (Conv2D)	(None,27,27,16)	12560	conv2d_3 [0][0]
flatten (Flatten)	(None,23328)	0	max_pooling2d [0][0]
flatten_1 (Flatten)	(None,11664)	0	max_pooling2d_1 [0][0]
flatten_2 (Flatten)	(None,5408)	0	max_pooling2d_2 [0][0]
flatten_3 (Flatten)	(None,2704)	0	max_pooling2d_3 [0][0]
flatten_4 (Flatten)	(None,11664)	0	conv2d_4 [0][0]
concatenate (Concatenate)	(None,54768)	0	flatten [0][0]
			flatten_1 [0][0]
			flatten_2 [0][0]
			flatten _3 [0][0]
			flatten _4 [0][0]
dropout (Dropout)	(None,54768)	0	concatenate [0][0]
dense (Dense)	(None,4096)	224333824	dropout [0][0]
dropout_1 (Dropout)	(None,4096)	0	dense [0][0]
dense_1 (Dense)	(None,512)	4195328	dropout_1 [0][0]
dropout_2 (Dropout)	(None,512)	0	dense_1 [0][0]
dense_2 (Dense)	(None,4)	4100	dropout_2 [0][0]
Total params: 228,625,876
Trainable params: 228,625,876
Non-trainable params: 0

show the model architecture and parameter information of DS-SoybeanNet with 5 × 5 and 7 × 7 kernels.

4.2. Performance Comparison Based on Field F2

We used the 546 images from field F2 to test the performance of MobileNetV2, InceptionResNetV2, ResNet50, SVM, RF, and the proposed DS-SoybeanNet model in monitoring soybean maturity.

Table 8. Confusion matrices of MobileNetV2 (a), InceptionResNetV2 (b), ResNet50 (c), SVM (d), RF (e), DS-SoybeanNet with kernel sizes of 5 × 5 (f) and 7 × 7 (g), AlexNet (h), and VGG16 (i).

(a)	Predicted Condition					(b)	Predicted Condition					(c)	Predicted Condition
Actual condition	Label	L0	L1	L2	L3	Actual condition	Label	L0	L1	L2	L3	Actual condition	Label	L0	L1	L2	L3
	L0	52	0	4	8		L0	39	0	17	8		L0	46	16	1	1
	L1	20	0	168	31		L1	5	18	184	12		L1	9	97	109	4
	L2	1	0	17	25		L2	0	1	193	4		L2	0	3	191	4
	L3	0	0	1	64		L3	0	0	10	55		L3	0	0	5	60
(d)	Predicted Condition					(e)	Predicted Condition					(f)	Predicted Condition
Actual condition	Label	L0	L1	L2	L3	Actual condition	Label	L0	L1	L2	L3	Actual condition	Label	L0	L1	L2	L3
	L0	64	0	0	0		L0	64	0	0	0		L0	59	5	0	0
	L1	98	119	1	1		L1	89	94	33	3		L1	16	185	18	0
	L2	2	85	102	9		L2	1	40	137	20		L2	0	27	171	0
	L3	2	0	2	61		L3	0	0	3	62		L3	0	0	9	56
(g)	Predicted Condition (5 × 5)					(h)	Predicted Condition					(i)	Predicted Condition
Actual condition	Label	L0	L1	L2	L3	Actual condition	Label	L0	L1	L2	L3	Actual condition	Label	L0	L1	L2	L3
	L0	59	5	0	0		L0	51	11	0	2		L0	59	5	0	0
	L1	13	179	25	2		L1	5	97	115	2		L1	25	169	19	6
	L2	0	22	169	7		L2	0	3	192	3		L2	0	28	163	7
	L3	0	0	12	53		L3	0	0	5	60		L3	0	0	7	58

shows the confusion matrices of the soybean maturity classifications of the eight models.

Table 9. Classification results of eight models from field F2.

Model	Rank	Precision				Accuracy
		L0	L1	L2	L3
DS-SoybeanNet (5 × 5)	1	92.19%	84.47%	86.36%	86.15%	86.26%
DS-SoybeanNet (7 × 7)		92.19%	81.74%	85.35%	81.54%	84.25%
VGG16	2	92.19%	77.17%	82.32%	89.23%	82.23%
AlexNet	3	79.37%	43.89%	96.95%	92.31%	72.89%
ResNet50	4	71.87%	44.29%	96.46%	92.31%	72.16%
RF	5	100%	42.92%	69.19%	95.38%	65.38%
SVM	6	100%	54.34%	51.52%	93.85%	63.37%
InceptionResNetV2	7	60.93%	8.22%	97.47%	84.62%	55.86%
MobileNetV2	8	81.25%	0%	39.53%	98.46%	52.75%

shows the classification results of the eight models using the data from field F2. Our results (Table 8 and Table 9) indicated that the proposed DS-SoybeanNet model exhibited a higher classification accuracy than the other machine learning models.

The conventional machine learning models (SVM and RF) exhibited the highest classification recall (100%) in the classification of immature soybeans (L0) (see Table 9). AlexNet (96.95%) showed the highest classification recall for mature soybeans (L2). As shown in Table 8 and Table 9, the conventional machine learning models (SVM and RF) and deep learning models (MobileNetV2, InceptionResNetV2, and ResNet50) showed lower recalls for near-mature soybeans (L1), which led to lower overall classification accuracies for these models. DS-SoybeanNet (84.47%) had the highest classification recall for near-mature soybeans (L1) (see Table 9).

As shown in Table 9, the ResNet50 model exhibited a high classification accuracy of 72.16%. The RF (65.38%) and SVM (63.37%) models had similar classification accuracies. The soybean classification accuracies of InceptionResNetV2 (55.86%) and MobileNetV2 (52.75%) were lower than those of the other five models. The accuracies of DS-SoybeanNet based on 5 × 5 and 7 × 7 convolution kernels, namely, 86.26% and 84.25%, respectively, were notably higher than those of the other models.

Note that the eight models’ performance decreased when using the field F2 dataset to test the models (Table 5, Table 6, Table 7 and Table 9). As shown in Table 9, the top 3 models were DS-SoybeanNet, AlexNet, and VGG16 when monitoring soybean maturity using the field F2 dataset. Recently, the AlexNet [48] and VGG16 [39] models have been used to detect crop maturity by many researchers. Our results show that the new DS-SoybeanNet model performed better than the AlexNet and VGG16 models in the classification of immature (L0) and near-mature soybeans (L1). For the field F1 dataset, the recall of L0 for DS-SoybeanNet was 100%, which is higher than that of AlexNet (99.69%) and VGG16 (98.74%). For the field F2 dataset, the recall of L0 and L1 for DS-SoybeanNet was 92.19% and 84.47%, which was notably higher than that of the AlexNet (L0: 79.37%, L1: 43.89%) model.

To further evaluate the fusion of deep and shallow CNN features and to explore the efficiency of the proposed DS-SoybeanNet model, we set up three ablation experiments for DS-SoybeanNet, as described below. Figure 6 shows the architectures of the CNNs used for experiments 2 and 3. Each model was trained and validated three times, and the model with the highest performance was saved.

• Experiment 1. DS-SoybeanNet (Figure 3);
• Experiment 2. DS-SoybeanNet with only shallow image features (Figure 6a); and
• Experiment 3. DS-SoybeanNet with only deep image features (Figure 6b);

Our results (

Table 10. Classification results of three experiments with 5 × 5 and 7 × 7 kernels.

Label		Experiment 1		Experiment 2		Experiment 3
		Validation Dataset (Field F1)	Independent Validation Dataset (Field F2)	Validation Dataset (Field F1)	Independent Validation Dataset (Field F2)	Validation Dataset (Field F1)	Independent Validation Dataset (Field F2)
Recall	L0	100%	92.19%	100%	98.44%	100%	96.88%
	L1	100%	84.47%	100%	74.89%	99.39%	83.11%
	L2	90.38%	86.36%	84.62%	87.37%	69.23%	71.21%
	L3	99.36%	86.15%	98.09%	75.38%	98.09%	87.69%
Accuracy		99.17% *	86.26% *	98.35%	82.23%	97.28%	80.95%
Recall	L0	100%	92.19%	100%	75.00%	100%	89.06%
	L1	100%	81.74%	99.39%	78.08%	99.39%	81.74%
	L2	90.38%	85.35%	86.54%	82.83%	78.85%	75.25%
	L3	99.68%	81.54%	98.73%	92.31%	98.41%	81.54%
Accuracy		99.19% *	84.25% *	98.58%	81.14%	97.99%	80.22%

Note: Bold and * indicate the highest accuracy.

) indicate that the soybean maturity classification accuracy in experiment 2 (only shallow image features) and experiment 3 (only deep image features) was lower than that in experiment 1. This further proved that fusing deep and shallow CNN features [44,45,46] may improve the performance of the model in image classification tasks.

4.3. Soybean Maturity Mapping

For soybean maturity mapping, the following three steps were carried out:

(a)

A soybean canopy DOM of field F2 was obtained after the UAV flight and the image stitching process. Then, all soybean breeding line plots (26 rows and 21 columns) were manually labeled, and the soybean plot image coordinates (plot center) were recorded.

(b)

The soybean canopy images (108 × 108 × 3) were extracted automatically using the image coordinates and soybean canopy DOM using a Python script. Then, we used DS-SoybeanNet to classify these soybean canopy images.

(c)

We then mapped the soybean maturity based on the soybean maturity information and soybean plot image coordinates.

Figure 7 shows a true-color RGB image and the maturity maps calculated for field F2 using DS-SoybeanNet with 5 × 5 and 7 × 7 convolution kernels. Our results indicate that the estimated soybean maturity information for field F2 had a high accuracy. The soybean maturity information obtained from the DS-SoybeanNet model with 5 × 5 and 7 × 7 convolution kernels was similar.

4.4. Advantages and Disadvantages of UAV + DS-SoybeanNet

As soybeans mature, the leaf chlorophyll level gradually decreases, contributing to a slow change in the leaves’ color from green to yellow [51,52]. Crop leaf chlorophyll variation is asynchronous among layers of leaves [52]. For example, leaves in the top layer of a soybean canopy tend to have a younger leaf age and thus turn yellow later than the leaves in the bottom layer. Consequently, green and yellow leaves appear in the soybean canopy when the soybeans are nearly mature (Figure 2). Breeding fields commonly have thousands of breeding lines with different maturation times. Thus, timely monitoring of soybean breeding line maturity is crucial for soybean harvesting management and yield measurements [5,6,7,8]. UAV remote sensing technology can be utilized to collect high-resolution crop canopy images and has been widely used in precision agricultural crop trait monitoring [14,15]. Many studies have evaluated the crop parameter monitoring performance of digital cameras and multispectral sensors on board lightweight UAVs [17,18,19]. In our study, we attempted to evaluate the potential of using UAV remote sensing to monitor soybean breeding line maturity. We developed DS-SoybeanNet, which can extract and utilize both shallow and deep image features, and which thus helps to provide soybean breeding line maturity monitoring that is more robust than that offered by conventional machine learning methods. DS-SoybeanNet achieved the best accuracy of 86.26% (Table A1), which was notably higher than those of the conventional machine learning models (SVM and RF). However, DS-SoybeanNet has various disadvantages compared with conventional machine learning methods, such as its long elapsed time and large size (

Table 11. Models’ elapsed times and sizes.

Model	Time (s)/1000 Samples	Size
RF	0.003	24.1 KB
SVM	0.007	7.70 KB
MobileNetV2	6.607	53.3 MB
DS-SoybeanNet (5 × 5)	11.770	2616 MB
AlexNet	19.011	151 MB
DS-SoybeanNet (7 × 7)	22.955	2616 MB
ResNet50	36.099	306 MB
InceptionResNetV2	44.328	653 MB
VGG16	67.080	623 MB

). In machine learning, CNNs have a more complex network structure and higher computational complexity than conventional machine learning models with larger model sizes.

Table 11 shows the time required to process 1000 samples using each model and the models’ sizes. The computation times of the CNN models (ranging from 6.607 s to 67.080 s) were notably higher than those of the conventional machine learning models, SVM and RF (0.003 s and 0.007 s). In addition, a high-performance device is required to calibrate CNN models. As shown in Table 11, the model sizes of DS-SoybeanNet, ResNet50, and InceptionResNetV2 were more than 300 MB. The proposed DS-SoybeanNet model had the largest size (2616 MB) compared to the other models. The DS-SoybeanNet model’s large size may mean that it requires large storage when deployed on lightweight platforms (e.g., Raspberry Pi) for stationary observations. Nevertheless, DS-SoybeanNet (5 × 5) had approximately the same calculation speed as MobileNetV2 and a much higher monitoring accuracy than the other deep learning models. Therefore, we consider DS-SoybeanNet a fast and high-performance deep-learning tool for monitoring soybean maturity.

Many previous studies have used AlexNet, VGG16, Inception-V3, and VGG19 in crop maturity classifications. Faisal et al. [53] compared the performance of pre-trained VGG-19 (99.4%), Inception-V3 (99.4%), and NASNet (99.7%) in detecting fruit maturity. Atif et al. [54] used AlexNet and VGG16 to classify the maturity levels of jujube fruits (best: VGG16 = 99.17%). Sahil et al. [55] developed a method that used YOLOv3 to pinpoint the locations of tomatoes (94.67%) and used an AlexNet-like CNN model to classify their maturity levels (90.67%). In this work, we compared the results of conventional machine learning models (SVM (92.31%) and RF (94.23%)) and six CNN machine learning models (DS-SoybeanNet (99.19%), VGG16 (97.99%), AlexNet (99.44%), ResNet50 (98.97%), InceptionResNetV2 (99.49%), and MobileNetV2 (97.52%)) in soybean maturity information monitoring based on UAV remote sensing. The accuracy results reported in this study were close to those of previous studies based on AlexNet, VGG16, Inception-V3, and VGG16. Thus, our results further proved that deep learning is a good tool for crop maturity information monitoring [48,53,54,55,56]. The combination of UAV remote sensing and deep learning can be used for high-performance soybean maturity information monitoring. However, our results indicate that selected machine learning models’ performance decreased when using the field F2 dataset to test the models (Table 5, Table 6, Table 7 and Table 9). We suspect that changes in the UAV’s working environment—for example, varying sunlight intensity over time—led to a direct decline in the models’ performance. This is perhaps not surprising because the farmland environment is affected by varying cropland conditions (e.g., irrigation, wind). Thus, future research should be focused on the factors influencing cropland images.

In this study, the performance obtained when using soybean canopy images captured by the UAV’s remote sensing digital camera may have been limited by the varying sunlight intensity over time. Since DS-SoybeanNet did not normalize the image differences due to sunlight, a normalization module may improve its performance in soybean maturity classification. Therefore, future studies need to develop a normalization module to weaken the effect of the sun. Thus, more experiments with different varieties and regions of soybeans are needed to improve the generalizability of the DS-SoybeanNet model. In this study, the proposed DS-SoybeanNet was validated using only two breeding fields from a single site; thus, further validation is required from additional fields and study sites.

5. Conclusions

In this study, we designed a network, namely, DS-SoybeanNet, to extract and utilize both shallow and deep image features to improve the performance of UAV-based soybean maturity information monitoring. We compared conventional machine learning methods (SVM and RF), current deep learning methods (AlexNet, VGG16, InceptionResNetV2, MobileNetV2, and ResNet50), and our proposed DS-SoybeanNet model in terms of their soybean maturity classification accuracy. The results were as follows.

(1)

The conventional machine learning methods (SVM and RF) had lower calculation times than the deep learning methods (AlexNet, VGG16, InceptionResNetV2, MobileNetV2, and ResNet50) and our proposed DS-SoybeanNet model. For example, the computation speed of RF was 0.03 s per 1000 images. However, the overall accuracies of the conventional machine learning methods were notably lower than those of the deep learning methods and the proposed DS-SoybeanNet model.

(2)

The current deep learning methods were outperformed in terms of universality by the DS-SoybeanNet model in the monitoring of soybean maturity. The overall accuracies of MobileNetV2 for fields F1 and F2 were 97.52% and 52.75%, respectively.

(3)

The proposed DS-SoybeanNet model was able to provide high-performance soybean maturity classification results. Its computation speed was 11.770 s per 1000 images and its overall accuracies for fields F1 and F2 were 99.19% and 86.26%, respectively.

(4)

Furthermore, future studies are needed in order to develop a normalization module to weaken the effect of the sun. Moreover, further validation is required using additional fields and study sites.