Detection of Fundamental Quality Traits of Winter Jujube Based on Computer Vision and Deep Learning

Abstract

Winter jujube (Ziziphus jujuba Mill. cv. Dongzao) has been cultivated in China for a long time and has a richly abundant history, whose maturity grade determined different postharvest qualities. Traditional methods for identifying the fundamental quality of winter jujube are known to be time-consuming and labor-intensive, resulting in significant difficulties for winter jujube resource management. The applications of deep learning in this regard will help manufacturers and orchard workers quickly identify fundamental quality information. In our study, the best fundamental quality of winter jujube from the correlation between maturity and fundamental quality was determined by testing three simple physicochemical indexes: total soluble solids (TSS), total acid (TA) and puncture force of fruit at five maturity stages which classified by the color and appearance. The results showed that the fully red fruits (the 4th grade) had the optimal eating quality parameter. Additionally, five different maturity grades of winter jujube were photographed as datasets and used the ResNet-50 model and the iResNet-50 model for training. And the iResNet-50 model was improved to overlap double residuals in the first Main Stage, with an accuracy of 98.35%, a precision of 98.40%, a recall of 98.35%, and a F1 score of 98.36%, which provided an important basis for automatic fundamental quality detection of winter jujube. This study provided ideas for fundamental quality classification of winter jujube during harvesting, fundamental quality screening of winter jujube in assembly line production, and real-time monitoring of winter jujube during transportation and storage.

1. Introduction

There is a precise correlation between the maturity and fundamental quality of the fruit. The maturity of the fruit at harvest has an important impact on the softening and texture change of the fruit [1]. For the appearance of the fruit, height, weight and color changed with the progress of maturity [2]. For the content of compounds contained in the fruit, the maturity of the fruit will affect the content of total soluble solids and total acid [3]. For the above reasons, the fundamental quality of fruit can be effectively maintained by choosing the optimal harvest maturity [4]. The researchers selected different indicators as evaluation criteria for different fruit maturity to grade the fundamental quality. Zhu et al. proposed following three maturity levels for grading Camellia oleifera fruit samples: unripe, ripe, and overripe, and the results indicate that there were significant differences in seed oil content, seed soluble protein content, seed soluble sugar content, seed starch content, dry seed weight, and moisture content among the maturity stages graded [5]. Iqbal and Hakim utilized appearance features such as shape, texture, color, and size to grade eight different cultivars of harvested mangoes for extra class, class I and class II [6]. Ma et al. used the color and sweetness values of the three different parts of banana to classify the maturity of the fruit for six stages [7].

Deep learning has brought about a significant transformation in conventional internet-based industries, including web search and advertising. In addition, it has enabled the development of new products and businesses by assisting people in various domains such as healthcare, education, agriculture, and even autonomous driving [8]. For instance, deep learning has significantly improved the accuracy of predicting subcellular protein localization [9]. Notably, deep learning has been instrumental in advancing computer vision, which has witnessed rapid progress in recent times. He et al. introduced the influential Residual Networks, which have remained the gold-standard since the inception of the ResNet architecture [10]. Residual Networks are considered easy to optimize and can alleviate the problem of gradient disappearance that is caused by increasing depth in deep neural networks. As a result, ResNet is often used as the default architecture in studies or as a baseline for comparison when new architectures are proposed [11]. Further, Duta et al. proposed an improved version of ResNet. The proposed enhancements tackle the three key elements of a ResNet—namely, the flow of information across network layers, the residual building block, and the projection shortcut. These enhancements can ensure improved network accuracy and learning astringency [12].

The convolutional neural network is commonly utilized in the field of agriculture [13]. Mamat et al. utilized YOLO net for automated annotation of fruit images. The mean Average Precision achieved for oil palm fruit was 98.7%, while the accuracy for fruit variety classification reached 99.5% [14]. Gulzar implemented TL-MobileNetV2 for the classification of forty different types of fruits. The model achieved an accuracy of 99%, a recall of 99%, and a f1 score of 99% [15]. Osako et al. used a pre-trained VGG16 model to develop a cultivar classification system for litchi fruit. The model achieved an accuracy of 98.33% [16]. Chen et al. photographed images of apricot fruits in both outdoor and indoor settings, and created a dataset that enables accurate fruit classification using a U-net model. The model achieved an impressive F-score of 99%. Furthermore, the researchers collected four datasets containing seeds that are challenging to identify, and trained a VGG16 model to classify them, achieving an accuracy of 97% [17]. Suzuki et al. proposed the use of a deep neural network to predict the occurrence of a severe fruit disorder in persimmon, namely rapid over-softening, using simple RGB images. Their research revealed that all the CNN models examined were successful in binary classification of the rapidly over-softened fruits and controls, reaching an accuracy of over 80% across various criteria [18]. Unal and Aktas used Inception V3 model and EfficientNet model to accurately classify hazelnut kernels, with an accuracy of 97.85% and an accuracy of 99.28% respectively [19].

Winter jujube (Ziziphus jujuba Mill. cv. Dongzao) is an excellent late maturing fresh-eating variety in China [20]. Tree-ripened winter jujube is highly appreciated by consumers for its taste, flavor and sweetness [21]. Regrettably, such fruits have a brief postharvest lifespan, primarily due to rapid softening, resulting in increased vulnerability to mechanical damage and the development of decay [22]. The traditional maturity classification methods such as mechanical testing can easily cause damage to the surface of winter jujube [23]. Hence, the classification of winter jujube based on deep learning is an efficient and non-destructive way to solve this problem. Lu et al. used YOLOv3 algorithm to train models, with an accuracy of 97.28%. Additionally, they designed an automatic winter jujube classification robot based on computer vision [24]. Al-Saif et al. devised a technique for distinguishing between various cultivars of Indian jujube fruits by utilizing a single fruit’s color and morphological characteristics and training an artificial neural network classifier. Their approach achieved an accuracy rate of 97.56% [25]. Feng et al. used hyperspectral imaging with pixel-wise deep learning method to detect subtle bruises on winter jujube. Their study found that the CNN model based on all geographical origins performed the best, with most accuracy surpassing 85% [26]. Despite some progress in research in this field, there are still challenges to overcome, including insufficient accuracy and a limited number of categories. In this study, we classified the maturity of winter jujube into five levels [27,28]. The puncture force, TSS, and TA content of winter jujube was tested at each maturity level to determine the optimal consumption period of winter jujube [29]. We used the automatic recognition and classification of each maturity of winter jujube images by deep learning model. We took image data to train ResNet-50, iResNet-50, and designed the improved network model base on iResNet-50 to increase the accuracy of winter jujube classification. By implementing a more refined categorization approach, our model has acquired the capability to automatically classify the maturity levels of fruits and identify preliminary defects.

2. Materials and Methods

2.1. Plant Materials

Winter jujubes were collected manually in winter jujubes production base from Gucheng Town, Zhanhua District, Binzhou City. Fresh samples are immediately measured upon arrival at the laboratory. The maturities of winter jujubes were divided into five levels according to their appearance and color [30,31,32] (Figure 1): green fruits (almost completely green, with 0 to 1/8 of the skin red; 1st grade), white ripe fruits (1/8 to 1/4 of the skin red; 2nd grade), semi-red fruits (1/4 to 3/4 of the skin red; 3rd grade), fully red fruits (3/4 to 1 of the skin red; 4th grade), and softened fruits (fruits that have lost their commercial value due to crumpling, softening, and decay; 5th grade). The fruits (front and back side) were placed on a black-background cloth and photographed with a smart phone (iPhone12 pro max) (Figure 2). The images were captured under the lighting conditions of fluorescent lamps in a laboratory setting. 5000 images of the original data were taken. 3500 images were used as the training set, 1000 images as the validation set, and 500 images as the test set. Each classification of images is randomly and equally distributed in the dataset (

Table 1. Allocation of the dataset.

	Green Fruits	White Ripe Fruits	Semi-Red Fruits	Fully Red Fruits	Softened Fruits
training set	700	700	700	700	700
validation set	200	200	200	200	200
test set	100	100	100	100	100

).

2.2. Image Processing

The image data was processed into 224 × 224 pixel images and the captured photos were manually labeled. The dataset was expanded by using image enhancement to increase the amount of information and improve the effectiveness of image interpretation and recognition [33].

2.3. Physical and Chemical Index Detection

2.3.1. Total Soluble Solids and Titratable Acidity

A pocket refractometer (ATAGO, PAL-BX/ACID, Tokyo, Japan) was used to test four types of winter jujubes which have not lost their commercial value [34]. For each group, a random sample of ten winter jujube fruits was chosen to measure the TSS and TA. The fruits were sliced into small pieces and squeezed to extract the juice, filtered through four layers of gauze. The meshes size of the gauze used is 425 micrometers. 200 μL of the juice supernatant was taken to measure the TSS. Then 200 μL of the juice supernatant was diluted with 9.8 mL of deionized water, and TA was measured by a Brix-Acidity Meter [35]. Each group was tested three times.

2.3.2. Puncture Force

The puncture force of each unpeeled winter jujube fruit was measured at two equatorial sites, evenly distributed around the fruit, using a Texture Analyzer (Stable Micro Systems, TA.XT plus, Surrey, UK) [36]. The results were expressed in grams (g). P/2n (diameter = 2 mm) needle probe was chosen. The testing parameters used were as

Table 2. Setting of texture analyzer parameters.

Pre-Test Speed	Test Speed	Post-Test Speed	Distance	Time	Trigger Force
1.00 mm/s	5.00 mm/s	5.00 mm/s	5.000 mm	5.00 s	5.0 g

.

2.4. Classification Training

ResNet-50, iResNet-50 and improved iResNet-50 was prepared for training and compared their classification performance. The improved model is built based on iResNet-50 (Figure 3). In the Start Stage, the features are initially extracted by a convolutional layer, and then the extracted features are compressed by a maximum pooling layer to remove redundant information and improve the training efficiency. The extracted features are then sent to four Main Stages. The internal composition of a Main Stage include one Start ResBlock, several Middle ResBlocks (the first Main Stage has one Middle ResBlock, the second Main Stage has two Middle ResBlocks, the third Main Stage has four Middle ResBlocks, and the fourth Main Stage has one Middle ResBlock) and an Ending ResBlock. Each ResBlock has three convolutional layers (two 1 × 1 convolutional layers and one 3 × 3 convolutional layer), for a total of 16 ResBlocks, which serves as a 50-layer network. For the ResBlock in the first Main Stage, double residuals are added to the output part. Since the Start ResBlock performs batch normalization of the signal at the end, it is not necessary to use the batch normalization layer at the beginning of the first Middle ResBlock, and it is also guaranteed that there is a batch normalization operation after each convolutional layer. In the output phase of the model, the five maturity classifications of winter jujube fruits are corresponded to the outputs from 0 to 4 (

Table 3. Maturity grades and network outputs.

Grade	Green Fruits	White Ripe Fruits	Semi-Red Fruits	Fully Red Fruits	Softened Fruits
Output	0	1	2	3	4

).

2.5. Model Training Configuration

In this study, we used an Ali Cloud server to train the models and configured Tensorflow and other dependency libraries within an Anaconda virtual environment to create the training environment. All models presented in this paper were built using the Tensorflow framework. Parameters used are as

Table 4. Setting of training parameters.

Epoch	Initial Learning Rate	Batch Size	Optimizer
80	1 × 10−3	16	Adam

.

2.6. Model Evaluation Criterion

Accuracy, precision, recall and F1-score were used to evaluate the three classification models results in our study [37]. The accuracy rate is a measure of the overall prediction accuracy of a model, while the precision rate represents the accuracy of positive predictions made by the model. Recall is a metric specific to the original sample and represents the probability of a positive sample being correctly identified as positive by the model. A high recall means that the model is able to identify most of the positive samples, but may also result in more false detections. The F1-Score is a composite metric that takes both precision and recall into account, providing a more comprehensive evaluation of a model’s performance. It is used to balance the effects of precision and recall, and is a commonly used metric for evaluating classification models. Figure 4 shows the relationship between F1-score, precision and recall.

$$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}=\frac{\sum _{i=0}^{\mathrm{n}=4}{\mathrm{T}\mathrm{P}}_i}{\mathrm{A}\mathrm{l}\mathrm{l}\mathrm{s}\mathrm{a}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e}\mathrm{s}}$$

$${\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}}_i=\frac{{\mathrm{T}\mathrm{P}}_i}{{\mathrm{T}\mathrm{P}}_i+{\mathrm{F}\mathrm{P}}_i}$$

$${\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}_i=\frac{{\mathrm{T}\mathrm{P}}_i}{{\mathrm{T}\mathrm{P}}_i+{\mathrm{F}\mathrm{N}}_i}$$

$${\mathrm{F}1-\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}}_i=\frac{2\times {\mathrm{P}}_i\times {\mathrm{R}}_i}{{\mathrm{P}}_i+{\mathrm{R}}_i}$$

Various indicators for all five classifications were calculated, where TP represents the number of true positives, FP represents the number of false positives, and FN represents the number of false negatives. “i” is a classification tag.

3. Results

3.1. Total Soluble Solids, Titratable Acidity and Puncture Force

The test results including TSS, TA, TSS/TA and puncture force for the four grades are shown in the

Table 5. Total soluble solids (TSS), total acid (TA), TSS/TA and Puncture force of different grade jujube.

Grade	TSS/% *	TA/%	TSS/TA	Puncture Force/g
green fruits (1st classification)	16.97 ± 0.61 d **	0.75 ± 0.03 b	22.63 ± 0.39 c	292.50 ± 22.75 b
white ripe fruits (2nd classification)	20.13 ± 0.82 c	0.79 ± 0.02 ab	25.48 ± 1.54 b	315.24 ± 43.63 b
semi-red fruits (3rd classification)	21.47 ± 0.12 b	0.83 ± 0.01 a	25.87 ± 0.35 b	318.87 ± 50.70 b
fully red fruits (4th classification)	24.23 ± 0.05 a	0.83 ± 0.03 a	29.19 ± 1.09 a	352.93 ± 42.43 a
softened fruits (5th classification)	NA	NA	NA	NA

* The results were expressed with means ± standard deviation (SD). ** The different letters represent the significant (p < 0.05) difference in the same column.

TSS and TA are important fundamental quality indexes that represent the fruit sugar and acid contents, which contribute to the unique flavor of winter jujube. The concentration of sugars and organic acids, particularly high sugar and high acid, are important factors in determining the fundamental quality of winter jujube. [38,39]. As shown in Table 5, the TSS improved with maturity increased, and the change in TA stabilized. Meanwhile TSS/TA was gradually improving. The fully red fruits had the highest TSS/TA. The puncture force increased continuously with maturity levels until the winter jujube fruit rots and loses water.

3.2. Data Augmentation

After the images were cropped and resized to fit network requirements, the dataset was extended by rotating the images. This was an effective way to increase the diversity and number of datasets, thus improving the performance and accuracy of the model [40]. By data augmentation, we were able to obtain different angles, orientations, and perspectives, making the model more robust and better able to handle images in different scenes. In addition, data augmentation prevented overfitting problems. The total number of images was extended to 20,000 by adjusting the rotation according to the method shown in Figure 5.

3.3. Classification Models

3.3.1. Model Training

The model underwent training for a total of 80 epochs, and it achieved its optimal performance after the 27th training epoch. The model parameters were then saved for future testing purposes (Figure 6).

3.3.2. Feature Extraction

The performance of three different convolutional neural network models for image classification was investigated: ResNet-50, iResNet-50, and an improved version of iResNet-50. The improved iResNet-50 was based on the original iResNet-50 architecture, but modified to improve its feature extraction capabilities. To evaluate the effectiveness of the improved model, we visualized and presented the features that were extracted by the intermediate process of the network in Figure 7. As observed in Figure 7, the improved iResNet-50 was able to extract a richer set of features from the input image data, which contributed to its superior performance in classification tasks. This was achieved by carefully selecting and processing these features at various stages of the network. The image data was first subjected to a preliminary feature extraction step, which involved passing it through a layer of convolutional layers. The resulting data was then dimensionally reduced and processed to [109 × 109 × 64], to reduce the number of parameters and computational resources required. A subsequent layer of maximum pooling was applied for feature selection, further reducing the number of features and dimensional size of the data to [55 × 55 × 64]. Next, the data was processed through four Main Stages, each consisting of a Start ResBlock, several Middle ResBlocks (with different numbers depending on the Main Stage), and an Ending ResBlock. These ResBlocks each contained two 1 × 1 convolutional layers and one 3 × 3 convolutional layer, and served to progressively extract more complex and abstract features from the input data. As shown in Figure 8, after the completion of the first Main Stage, the extracted features are processed for visualization. The network used for feature extraction was primarily based on color and texture. The improved network retains more significant features for subsequent convolutional layers to learn. The output features were then extracted into a [7 × 7 × 2048] tensor. To obtain a one-dimensional feature vector, the output tensor of the final convolutional layer was passed through a global average pooling operation, which computes the average of each feature map. This pooling operation is less prone to overfitting and improves computational efficiency [41]. Subsequently, a fully connected layer with Softmax activation was applied to the resulting vector to obtain the probability of each classification.

3.3.3. Classification Results and Comparison

The performance comparison of five different winter jujube maturity classification models was presented in

Table 6. Performance comparison for winter jujube maturity classification models.

Method	Classification	Precision/%	Recall/%	F1-score/%	Accuracy/%
VGG16	Avg	67.40	77.30	72.01	77.30
Inception net	Avg	89.66	89.65	89.65	89.65
ResNet-50	0	100	92.00	95.83	90.20
	1	91.44	85.50	88.37
	2	84.63	89.50	87.00
	3	82.01	98.00	89.29
	4	96.35	85.75	90.74
	Avg	90.89	90.15	90.25
iResNet-50	0	100	96.00	97.96	96.75
	1	95.93	94.50	95.21
	2	90.09	100	94.79
	3	99.73	93.50	96.51
	4	99.01	99.75	99.83
	Avg	96.95	96.75	96.77
Improved iResNet-50	0	100	99.75	99.87	98.35
	1	99.74	96.25	97.96
	2	94.51	99.00	96.70
	3	97.76	98.00	97.88
	4	100	98.75	99.37
	Avg	98.40	98.35	98.36

based on four different metrics: precision, recall, F1-score, and accuracy. The accuracy of VGG16 and Inception Net is relatively low. The ablation study includes ResNet-50, iResNet-50, and an improved version of iResNet-50. The results show that the improved iResNet-50 model outperformed both ResNet-50 and iResNet-50 in all metrics. The improved model achieved a precision of 98.40%, recall of 98.35%, an F1-score of 98.36%, and an accuracy of 98.35%. These results demonstrate superior performance on the classification task. One of the major reasons for the improved performance of the improved iResNet-50 model is the introduction of the double residual connection in the first Main Stage which improved the model’s ability to distinguish between similar objects. Additionally, the use of grouped building blocks helped to mitigate the vanishing gradient problem and allowed the model to learn more effectively from the image data. Our results demonstrate that the improved iResNet-50 model can effectively reduce the spatial dimensions of the feature maps, while preserving the most salient features for downstream tasks such as image classification. By carefully selecting and processing these features, the model achieved high accuracy and robustness in classification tasks, while also minimizing the number of parameters and computational resources required.

4. Discussion

4.1. Postharvest Fruit Fundamental Quality Analysis

According to our study, the maturity grade of winter jujube fruit is crucial in determining the appropriate picking period and post-harvest management techniques to ensure high fundamental quality fruit. Our results show that the fully red fruits have the best fundamental quality, characterized by its superior sweetness, flavor and texture. Moreover, the green fruits, white ripe fruits and semi-red fruits exhibit a longer storage time compared to the fully red fruits. Zhang et al. experimental results also showed that, the contents of TSS and TA in jujube fruit increase as it grows and develops, ultimately enhancing the fruit’s flavor [35]. By doing puncture tests on apples, Chang et al. found that for freshly harvested fruit, there is a positive correlation between puncture force and fruit crispness [42]. Fruit crispness increases with maturity. Therefore, farmers and other stakeholders in the industry should consider these factors when determining the optimal time to harvest winter jujube fruit and implement appropriate post-harvest management techniques to maintain quality and extend shelf life.

4.2. Model Detection Five Classifications Analysis

Based on the matrices presented in Figure 9, although the success in discrimination between five classifications is quite average, the highest errors were between classification 1, classification 2 and classification 3. The model misclassifies more instances from other classifications as belonging to these three classifications. The reason for this is that the input image data for this experiment exhibits subtle differences in characteristics, and often includes many data points that are critically graded. Furthermore, the three classifications in this dataset have images that are more difficult to define. It can be further seen from Figure 10 that the three classification models work better for classification 0 and classification 4 recognition, with a higher F1-score. This technique of superimposing double residuals helps to enhance the network’s ability to capture intricate details in the input data, especially near the data input part. By minimizing misclassification from adjacent classification, the model can further improve its accuracy rate. This approach helps to improve the network’s expressiveness and make it more effective in handling datasets for this study.

The results obtained in this study were compared with those reported in the literature for the use of deep learning models in grading the maturity of winter jujube. Mahmood et al. classified the maturity of jujubes into three levels. The augmentation process resulted in a total of 4398 images in the newly formed augmented dataset. The highest accuracy achieved for accurately classifying maturity levels using the AlexNet and VGG16 deep learning models, for both actual and augmented images, were as follows: 94.17% and 97.65%, and 98.26% and 99.17%, respectively [43]. Guo et al. developed a composite CNN method that utilized residual networks to detect defects in jujube. The method achieved an impressive accuracy rate of 99.2% [44]. Although deep learning models have become increasingly popular as a highly effective method for image classification in the agriculture and food industry, there are not many studies on winter jujube maturity grading based on deep learning. The results obtained in our study are considered to be suitable for the winter jujube fruits picking or post-harvest sorting scenes since the overall discrimination accuracy of classification models is over 98%.

5. Conclusions

Our study confirms the importance of proper maturity grading in the production and marketing of winter jujube fruit. By selecting the optimal picking period and implementing effective post-harvest management techniques, farmers and other stakeholders in the industry can ensure that winter jujube fruit meets consumer expectations for flavor, texture, and shelf life. Five automatic maturity detection models were developed for winter jujube using three different algorithms. After thorough comparison, we found that the model based on the improved iResNet-50 algorithm with double residuals superimposed in the first Main Stage achieved the best performance. Specifically, the model achieved an accuracy of 98.35%, an average precision of 98.40%, and an average recall of 98.35% on the test set of this experiment. Moreover, the average F1 score of the model was 98.36%, which demonstrated the feasibility and effectiveness of the improved iResNet-50 based algorithm for automatic grading of winter jujube maturity. Our model facilitates the automated classification of winter jujube after harvesting. By incorporating the classification of “softened fruits” into the model, it expands beyond maturity classification and enables preliminary identification of damaged fruits. This enhancement enhances the model’s capabilities and improves its utility in post-harvest fruit classification. Overall, the findings of this study offer important insights into the use of deep learning techniques for automatic grading of agricultural products, and can provide valuable guidance for the development of similar applications in the future. However, it should be noted that the performance of the models may vary depending on the dataset and the specific task at hand, and further optimization and validation may be necessary to achieve optimal results in practical scenarios.