1. Introduction
Tea is an important economic crop in China, where the tea cultivation area ranks first in the world and the total value of production is more than CNY 600 billion (USD 86.6 billion). To improve the utilization of tea resources, the machine harvesting of fresh tea leaves has begun in several important tea growing areas. The fresh leaves of machine-picked tea have different specifications, including single leaf (SL), one bud and one leaf (OBOL), one bud and two leaves (OBTL), and one bud and three leaves (OBRL), as well as low uniformity [
1]. These four types of tea fresh leaves not only differ in the number of leaves they contain in appearance, but they also differ in the content of internal substances such as L-theanine and catechins. Furthermore, the production of Chinese famous tea has very high requirements for fresh leaf raw materials, for example, Liu’an Gua Pian made only from SL as raw material.
Therefore, it is impossible to use machine harvesting in the processing of famous tea products, resulting in a significant decline in economic benefits. There are two main ways to solve the above problem: One is to develop an intelligent harvesting machine for famous tea [
2] to realize the intelligent identification of the fresh tea leaf grade of the tea. The other, proposed in this study, is applying the idea of picking and fusion to develop an intelligent sorting system for fresh tea leaf sorting according to the requirements of different types of famous tea. The fresh tea leaves are categorized into SL, OBOL, OBTL, and OBRL, as shown in
Figure 1, and correspond to the grading system used in the processing of famous tea.
Owing to the complexity and variability of the physical properties of fresh tea leaves, especially the fact that the fresh leaves will continue to lose moisture for some time after harvesting, which leads to significant changes in the physical surface properties of the fresh leaves after harvesting up until the sorting process [
3], it is challenging to develop a fast and highly accurate identification system suitable for the fresh tea leaf grades. Computer vision is nondestructive and highly automated, and it has been used in strawberry grading systems [
4], fermentation degree monitoring of black tea [
5,
6], apple sorting systems [
7], and in other fields. In recent years, conventional machine learning has been applied to the development of fresh tea leaf grade recognition systems and has achieved some success [
8,
9]. Gan et al. investigated feature classification methods using an improved genetic algorithm to screen features with the best combination of three different classifiers with the best number of dimensions, and finally a support vector machine (SVM) achieved a recognition accuracy of 97% in the 28-dimensional feature set [
10]. Chen et al. extracted the features of tea leaves by also determining the topology of the tea leaf and used SVM to identify the fresh tea leaves based on the sorting; consequently, the recognition accuracy reached 94% [
11]. However, it is difficult for machine learning to obtain effective feature values for some complex problems, and there are differences in recognition results when different sorting scenarios are present. Recently, deep learning algorithms have been proven effective in tea pest detection [
12,
13,
14], famous tea picker recognition [
15], and fresh tea leaf sorting [
16]. They have also been successfully applied in product sorting research [
17] for date fruit classification [
18], grading and sorting of garlics [
19], and seed sorting [
20,
21]. However, given the large number of materials in the fresh tea leaf grading link and the problems of fresh leaf collusion and entanglement, its detection speed and accuracy do not achieve the desired effect. The proposed You Only Look Once (YOLO) model provides unlimited opportunities to solve this problem and has been applied to the development of various agricultural grading systems, such as lotus seed sorting [
22], fruit classification [
23], and pear fruit sorting [
24]. For example, Gui et al. used YOLOv5-based lightweight tea bud detection, the average precision of which was 9.66% higher than YOLOv5 [
25]. Bao et al. utilized a combination of the multiscale RFB module and YOLOv5-based dual-dimensional mixed attention (DDMA) to reduce missed detection and false detection caused by dense blade distribution, thus resulting in a 6.5% recall increase [
14].
In addition, it is not easy to develop high-precision, fast, and intelligent sorting equipment suitable for the actual production of fresh tea leaf grades. Currently, most common tea sorting devices use mechanized classification, mainly tumbling sieve sorting, round sieve sorting, and wind sorting [
26,
27,
28]. These methods can categorize the machine-picked fresh leaf raw materials by size to a certain extent and effectively separate normal buds and leaves from broken buds or coarse stems and other things, but there is still a certain discrepancy between the grading effect and the requirements of famous fresh tea leaves. Zhen-Yu Gao et al. developed a fresh tea leaf intelligent sorting system and built a convolutional neural network recognition model based on a seven-layer structure to recognize and sort fresh tea leaves automatically with a recognition rate of 90% [
29]. Chen et al. developed an automatic sorting machine based on the recognition method of vision and applied SVM to the recognition and sorting of four grades of fresh tea leaves using nozzles, with a success rate of 80% and an efficiency of 15 kg/h [
11]. Although the aforementioned studies have achieved some success in the intelligent classification of fresh tea leaves, there is still room for improvement in terms of sorting accuracy and speed, especially given the large number of fresh tea leaf sorting scenarios.
To perform the sorting of tea leaves quickly and with high precision, a database of fresh tea leaf samples with different staking states was first created in this study. The YOLOv5 model was trained using databases at different spreading degrees, and the optimal training database composition was determined (contains as many fresh leaves as possible at different spreading stages), resulting in a fresh tea leaf grade recognition system with high generalization ability for the variation in physical features on the fresh tea leaf surface. We equipped the identification system on a three-stage conveyor belt with a multi-channel air-blowing device with high-speed execution capability and determined the optimal air-blowing angle and material collection position of the fresh tea leaf sorting equipment by positioning and analyzing the whole process from identification to the sorting link. Finally, a fresh tea leaf intelligent sorting system with a fast sorting speed and high success rate was developed for sorting the four grades—SL, OBOL, OBTL, and OBRL.
The following points summarize the contributions of this study:
- (1)
A deep learning recognition model with high generalization ability for fresh tea leaf samples is developed; it provides a better solution to the degradation of recognition accuracy caused by the presence of mixed fresh tea leaf grades; a mixture of old, mature, and tender leaves; and collusion and entanglement among individual fresh leaves during sorting.
- (2)
The potential relationship between the recognition accuracy of recognition models and water content of fresh tea leaves is investigated.
- (3)
Based on the theory and experiment, the whole process of positioning fresh tea leaves, from identification to sorting, is achieved.
The deep learning recognition model is mounted on a sorting platform with a high-speed air-blowing device to achieve high-precision and fast sorting of fresh tea leaf grades.
2. Materials and Methods
The development process of the fresh tea leaf sorting equipment is based on a combination of theoretical calculations, simulations, and design trials.
2.1. Program Design
Figure 2 shows the structure of the fresh tea leaf sorting platform, which includes three levels. The first level is sorting platform SL, the second level is sorting platform OBOL, the third level is sorting platform OBRL, and OBTL is the last platform level for the free fall of material, which finally realizes the four levels of fresh tea leaf sorting. The structure of each level of the sorting platform mainly includes a vibrating screen feeding system, a transfer device, an image acquisition system, a central control system, and a high-speed air-blowing sorting execution system.
The vibrating screen feeding system makes the fresh tea leaves fall evenly on the conveyor belt by adjusting the vibration frequency of the V-shaped vibrating screen (50 Hz), and the conveyor belt serves as a transfer device to transfer the fresh tea leaves to the image acquisition system. The image acquisition system consists of a Basler industrial camera (Basler Inc., Ahrensburg, Germany), two LED strips (14.2 mA, 12.2 W, 20 Klux), and a light shield. The industrial camera is attached to the top light shield to capture continuous images of the material and transmit them to the PC via a USB data cable. The PC acts as a central control system to process the image patterns transmitted from the camera and sends the processing results and position information to the high-speed air-blow sorting execution system through the serial port. The high-speed air-blow sorting actuator system includes a 32-bit STM32F407VET6 microcontroller based on the ARM® Cortex® core, an air pump, and a high-speed solenoid valve.
To achieve fast and high-precision sorting, the material is first shaken by the V-shaped vibrating screen and falls evenly onto the conveyor belt, which transports the material to the image acquisition system. In the image acquisition system, the Balser industrial camera captures the images and transmits them to the computer, which imports the images into the trained fresh tea leaf grade recognition model for recognition processing and outputs the label results with coordinate positioning. If the label result corresponds to the target material label of the sorting platform at this level, the PC sends the extracted fresh tea leaf coordinates to the STM32, which executes the air blow command to blow the target material into the collection frame after receiving the command. If the label result does not correspond to the target material label of the sorting platform at this level, the air blow command will not be executed, and the material will automatically enter the conveyor belt of the next level to wait for sorting again, as shown in
Figure 3.
2.2. Image Acquisition and Dataset Creation
In this study, fresh tea leaves, picked from the tea tree variety
C. sinensis cv. Nongkangzao were used as the test material. The grades of fresh tea leaves were classified into four categories, namely, SL, OBOL, OBTL, and OBRL, and the corresponding labels were 0, 1, 2, and 3, as shown in
Figure 1. In this study, to solve the problem of significant differences in the surface characteristics of fresh tea leaves with different moisture content values and according to the degree of spreading, the tea fresh leaves were categorized into four groups (a, b, c, and d). For each group were collected 400 images each of SL, OBOL, OBTL, and OBRL.
Then the actual sorting scenario was simulated to ensure that the fresh tea leaves were vibrated down from the vibrating screen onto the conveyor belt. Then, they were passed through the image acquisition area at a speed of 4 cm/s, where the camera captured the different spreading patterns. The fresh leaf samples were captured at a rate of 0.8 s per frame. A total of 6400 raw images in jpg format with a pixel resolution of 1430 × 1030 was obtained using the method described above (see
Table 1).
In this study, the Labellmg annotation tool was used to annotate the original images. To increase the size of the dataset and the diversity of the samples and mitigate overfitting during the model training, the dataset was expanded five-fold by mapping, brightening, changing contrast Normalization, and sharpening of the samples to randomly divide the dataset into a training set and a test set at a ratio of 8:1:1 for model training, optimization, and testing, respectively.
2.3. Model Selection
Advanced deep learning models are increasingly being proposed with the advancement in technology. Popular deep learning methods include YOLO, Fast RCNN, SSD, and Alex net; each has its advantages. Therefore, it is extremely important to select a model suitable for a particular dataset for the development of recognition systems.
In this study, we used all 6400 datasets to train and compare models (YOLOv5, YOLOv3, Fast RCNN, SSD). The comparison results are shown in
Table 2.
From the table, it can be observed that the recognition accuracy and speed of the Fast RCNN and SSD models are lower compared to the YOLO series models. It can be established that although the recognition accuracy of the YOLOv5s recognition model is about 0.8% lower than that of YOLOv3, its image recognition speed is about 4 times higher. In addition to this, YOLOv5 is far ahead of the other three models in terms of weights size. Therefore, considering all aspects, we chose the YOLOv5s model to apply the development of the tea fresh leaf recognition system.
2.4. Model Construction
YOLOv5 is one of the most advanced models in the field of deep learning. It features high detection accuracy and fast detection speed, and it meets the requirements of real-time monitoring. YOLOv5 is divided into four versions: s, m, l, and x. Considering the computer performance, training time, and other factors, the recognition model YOLOv5s was built in this experiment.
The YOLOv5s network model architecture built in this study is summarized in
Figure 4. It mainly consists of a Backbone network, Neck network, and Prediction network. Image samples are input from the input side, feature extraction is performed in the Backbone network, and then features are combined in the Neck layer and passed to the Prediction layer for the final prediction output.
2.5. Identification of Indicators to Evaluate Model Test Results
The accuracy (Precision, P), recall (Recall, R), mAP_0.5:0.95 (mean Average Precision), and F1 Score were used as evaluation metrics to objectively evaluate the criteria for evaluating the overall recognition performance of the trained recognition models, and the calculation formulas are as follows:
where TP is the number of correctly matched prediction frames with labeled frames; FP is the number of incorrectly predicted prediction frames; and FN is the number of mislabeled frames.
2.6. Localization of Fresh Tea Leaves and Tracking Modeling
To achieve continuous sorting of fresh tea leaves and improve sorting efficiency, it is necessary to determine the exact position of fresh tea leaves at different links and moments. Since the quality distribution and shape of tea leaves are irregular, and there are differences between different grades and even between different leaves of the same grade, the uncertainties in leaving the conveyor belt and making a throwing motion to reach the actual sorting point have a certain influence on the actual air-blowing effect. In the process of air blowing, the closer the air-blowing point is to the center of the fresh tea leaves, the better the sorting effect. If the air-blowing point is at the edge of the leaf, it is easy for the trajectory to shift or the air-blowing process to fail and so forth. Therefore, in this study, the locations of the morphological centroids of the fresh tea leaves (as shown in
Figure 5) were considered as the coordinates of the mass of the fresh tea leaves by determining them.
The target fresh leaves move in the direction of the conveyor belt with speed V. The image captured by the camera on the conveyor belt has a size of 640 × 640 pixels, which is used as the reference plane for defining the coordinate system for real-time positioning of fresh tea leaves. As shown in
Figure 6, point A is the image acquisition point of the fresh tea leaves. The camera captures the target material, and the image is transmitted to the PC. The PC identifies the material based on the smallest rectangular box and determines the coordinates of the four corners of the rectangular box (
,
), (
,
), (
,
), and (
,
), and finally obtains the morphology of the fresh tea leaves by calculating the coordinates of the center point A. Point O is the middle point of the coordinate system, whose coordinates are known to be (320, 320). The distance from point O to the end of the conveyor belt at point C is
. The conveyor belt runs at a speed
Point B is the location of the target after the fresh tea leaves are collected by the camera and processed at
time, and the coordinates of the corresponding fresh tea leaves are (
,
); then the fresh tea leaves are transported to the end of the conveyor belt at point C, and the transport time is set as
; then the fresh tea leaves make a flat throwing motion to reach the actual sorting point D, and the time of this process is set as
.
The fresh tea fresh leaf picture after the processor calculation process shows tea leaf coordinate positions when moving from point A to point B. Let the PC send to the air-blow sorting execution system through the serial port after the waiting time
, for the execution of the air blow command. Then,
Here,
yield
where
is the height of the blowing nozzle from the conveyor belt, and
is the height of the actual air-blowing point from the conveyor belt.
In this process, the fresh tea leaves make a parabolic movement after they leave the conveyor belt. In this process, the fresh tea leaves rotate by themselves, which leads to the uncertainty of the contact area between the airflow from the air-blowing mouth and the fresh tea leaves, which has a certain influence on the sorting success rate. Therefore, in this study, the force area of the fresh tea leaves was analyzed when the airflow was blown to the morphological center of the tea leaves at each angle, using the plane of air-blowing nozzle as the first viewpoint, and the optimal air-blowing angle was determined based on the trajectory analysis.
As shown in
Figure 7, fresh tea leaves undergo rotational motion due to body movement. Set and airflow are directly opposite the deflection angle β; α is the angle between the gas blowing mouth and the ground; the gas blowing mouth has a diameter of 2R; the gas blowing contact surface long axis is
; the short axis is
; and the area is
. The height of the air-blowing sorting point from the ground is
, and the height of the material collecting port is
. Fresh tea leaves continue to undergo parabolic motion due to the airflow generated by the force
; thus,
From the momentum theorem, it follows that:
where P = 1 MPa,
is the action time of the force
on the fresh tea leaves, which is set as 0.01 s, and m is the mass of fresh tea leaves, which is set to 0.1 g.
Fresh tea leaves perform a parabolic motion after being subjected to air-blowing force, and it can be seen from the above figure that the fresh tea leaves have different trajectories when the angle of the air-blowing mouth is different. In this section, to develop the optimal device for collecting fresh tea leaves considering the relevant factors, the motion trajectory under different air-blowing angles is analyzed, and the time
of the air-blowing force to the movement to the highest point can be expressed as follows:
The displacement X of the motion in the vertical direction can be expressed as follows:
The displacement Y of the motion in the vertical direction can be expressed as follows:
Finally, we have
where t is the movement time of the fresh tea leaves in the air after receiving the air-blowing force.
According to the analysis, under the strong air-blowing action at an air-blowing pressure of 1 MPa, the influence of different air-blowing action areas caused by the self-rotation of fresh tea leaves in free fall can be ignored. Once the correlation between the air-blowing trajectory and the air-blowing angle is known, theoretical analysis of the air-blowing angle and the size and position of the material collection port can be performed. In this study, the material collection port is first set at a position with a horizontal distance of 0.8 m from the air-blowing port. When the air-blowing port is horizontal, that is, α is equal to 0°, the material dropping point at X = 0.8 m is below the air-blowing nozzle. However, considering that the trajectory deviation is caused by the position of the blade edge blown by the air flow in the actual process, it is necessary to extend the height of the collection port downward to increase the collection range. However, in the structure of the machine, the downward extension of the collection port easily coincides with the V-shaped vibrating screen, which leads to the failure of material collection. When the angle between the air-blowing angle and the horizontal plane is 30°, the ideal height of the material collection point is 0.46 m above the air-blowing port, and the downward extension range of the collection port height is large at this height, which theoretically completely improves the collection. When the air-blowing angle continues to increase, both the collection height and the movement time of fresh tea leaves in the air increase, in addition to the uncertainty factors, which affect the success rate of sorting. Therefore, the air-blowing angle was set at 30° in this test.
4. Conclusions
A YOLOv5-based tea fresh leaf intelligent fresh tea leaf sorting system was developed to provide high precision and fast online sorting for fresh tea leaf grade identification.
In this study, tea leaf samples with different gradients of moisture content were collected and combined into a dataset with different gradients of moisture content to investigate the effect of the moisture content of tea leaves on the recognition accuracy of the model. The results show that the recognition effect of the tea recognition model on fresh leaf samples mixed with different staking times is positively correlated with the coverage rate and moisture content gradient of the fresh leaves in the model training data set, with the recognition model accuracy of the four moisture content gradient groups as high as 88.8%. At the same time, online grade recognition with the YOLOv5 model cannot only realize the fast and high-precision recognition of 4.5 ms/frame but also cope well with the recognition problem of a small amount of tea leaf collusion and entanglement caused by uneven vibration in the actual sorting process.
After determining the optimal structural parameters of the system in terms of sorting success rate through a combination of theory and experiment, the system performance was statically evaluated, and the sorting success rate reached 85.4%, and the sorting efficiency was 3 kg/h, which can cope well with the fast and high-precision sorting of fresh tea leaf grades. However, there is room for improvement. For future work, we intend to start with the modification of the model architecture and the optimization of the sorting equipment structure to further improve the accuracy and speed of sorting. This should increase the possibility of using machine-picked tea for the processing of fine tea.