A Multi-Fruit Recognition Method for a Fruit-Harvesting Robot Using MSA-Net and Hough Transform Elliptical Detection Compensation

Abstract

In the context of agricultural modernization and intelligentization, automated fruit recognition is of significance for improving harvest efficiency and reducing labor costs. The variety of fruits commonly planted in orchards and the fluctuations in market prices require farmers to adjust the types of crops they plant flexibly. However, the differences in size, shape, and color among different types of fruits make fruit recognition quite challenging. If each type of fruit requires a separate visual model, it becomes time-consuming and labor intensive to train and deploy these models, as well as increasing system complexity and maintenance costs. Therefore, developing a general visual model capable of recognizing multiple types of fruits has great application potential. Existing multi-fruit recognition methods mainly include traditional image processing techniques and deep learning models. Traditional methods perform poorly in dealing with complex backgrounds and diverse fruit morphologies, while current deep learning models may struggle to effectively capture and recognize targets of different scales. To address these challenges, this paper proposes a general fruit recognition model based on the Multi-Scale Attention Network (MSA-Net) and a Hough Transform localization compensation mechanism. By generating multi-scale feature maps through a multi-scale attention mechanism, the model enhances feature learning for fruits of different sizes. In addition, the Hough Transform ellipse detection compensation mechanism uses the shape features of fruits and combines them with MSA-Net recognition results to correct the initial positioning of spherical fruits and improve positioning accuracy. Experimental results show that the MSA-Net model achieves a precision of 97.56, a recall of 92.21, and an mAP@0.5 of 94.81 on a comprehensive dataset containing blueberries, lychees, strawberries, and tomatoes, demonstrating the ability to accurately recognize multiple types of fruits. Moreover, the introduction of the Hough Transform mechanism reduces the average localization error by 8.8 pixels and 3.5 pixels for fruit images at different distances, effectively improving the accuracy of fruit localization.

1. Introduction

Fruits constitute one of the largest segments in the global agricultural production market, contributing significantly to the development of the agricultural economy [1]. With a wide variety of types, they offer high nutritional value and excellent taste [2]. In recent years, with the increasing demand for fruits, more farmers have entered the fruit cultivation industry, and orchard coverage has gradually increased [3]. Due to the shortage of agricultural labor, the planting, maintenance, and harvesting of fruits are trending toward mechanization and automation [4]. Accurate and rapid fruit recognition is the crucial foundation for these operations.

Currently, research on visual models capable of detecting single types of fruits is mature. Wang et al. used various improved visual models to accurately recognize strawberries at different distances, effectively guiding orchard yield estimation and harvesting work [5,6]. However, orchards typically grow multiple types of fruits simultaneously. As fruit market prices fluctuate, farmers need to adjust the types of crops they plant to maximize economic benefits [7]. Replacing the visual model used in the detection production process will cause interruptions in the picking process and result in economic losses. Xiao et al. believe that using a visual framework that can detect multiple fruits is a potential cost-saving solution [8]. Therefore, developing a visual model that can simultaneously recognize multiple fruits has practical value.

Sensors are widely used in precision agriculture and can effectively obtain various data for use as inputs in deep learning models for tasks such as fruit variety and fruit disease classification [9,10]. Current methods for identifying multiple types of fruits mainly include deep learning models and image processing algorithms. Deep-learning-based methods extract features of the fruits to be recognized by performing multiple convolution operations on fruit images. Duong et al. efficiently classified various fruits based on fruit recognition using EfficientNet and MixNet [11]. The DCNN model, due to the depth of its convolutional processing, has been applied to fruit recognition, effectively distinguishing multiple fruits with similar characteristics [12]. Rachmawati used the PCoST model, which defines a semantic color for different fruits, effectively identifying various types of fruits [13]. Nguyen explored the advantages and disadvantages of deep learning algorithms, effectively eliminating the impact of external lighting variations to classify different types of fruits [14]. Given the numerous types of fruits to be detected, the similar shapes of different types of fruits, and the significant color differences at different maturity stages of the same fruit, it is currently challenging to detect multiple types of fruits with high precision using deep learning models.

Image processing methods can distinguish different fruits using their color, texture, and size, achieving precise fruit recognition. Wang accurately identified citrus fruits based on texture features and blueberry fruits based on color features using the improved LBP algorithm and the MSRCR algorithm, respectively [15,16]. Image processing methods can achieve high-precision detection of various target fruit types by adjusting algorithm steps. However, when fruit types change, these adjustments lack flexibility. He et al. used a gray-level co-occurrence matrix and a color histogram to extract the color and texture features of different fruits as the basis for identifying different fruits [17].

This paper aimed to develop a general fruit recognition model based on the Multi-Scale Attention Network (MSA-Net) and a Hough Transform localization compensation mechanism to address the limitations of existing methods. Specifically, MSA-Net was applied first, using a multi-scale attention mechanism to generate multi-scale feature maps for different types of targets, effectively learning the features of fruits of different sizes (including blueberries, citrus, lychees, and tomatoes). Secondly, this paper introduced a Hough Transform elliptical detection compensation mechanism. The model’s initial recognition results further corrected the initial localization of spherical fruits based on the position of the ellipse center. This study conducted a series of experiments and evaluated the detection performance of the MSA-Net model based on metrics such as mAP@0.5, precision, recall, and the F1 score. Additionally, it analyzed the performance of the Hough Transform compensation algorithm based on localization pixel errors.

2. Materials and Methods

2.1. Data Acquisition

There were citrus orchards, lychee orchards, tomato orchards, and blueberry orchards located in Zengcheng District, Guangzhou City, Guangdong Province, China (23°16′ N, 113°51′ E) for dataset collection. A Canon 200D II DSLR camera (Canon, Tokyo, Japan) was used to capture images. A total of 3331 raw images were collected and saved in .jpeg format, with a resolution of 4032 × 3024 pixels. The collected data were preliminarily screened. The variance of the initial image after Laplace transform was calculated. If the value was less than 100, it was considered that the image had defocusing and blurring problems, and out-of-focus or blurry images were removed. Overall, 20% of the images were randomly selected from the initial dataset and enhanced through random flipping, tilting, and other operations. The dataset was categorized into several classes, as shown in Figure 1, with the specific composition detailed in

Table 1. Dataset composition table.

Image Type	Number of Blueberry Images	Number of Strawberry Images	Number of Lychee Images	Number of Tomato Images	Total
Distant fruit image	158	214	174	131
Close-range-exposure fruit image	179	194	216	184
Close-range natural-light fruit image	314	337	402	367
Close-range backlit fruit image	144	95	138	84
Total	795	840	930	766	3331

.

2.2. MSA-Net

The backbone structure of MSA-Net is shown in Figure 2.

When multiple manually labeled fruit images are input into the backbone of MSA-Net, several subnetworks of the backbone layer, namely low-level, mid-level, and high-level subnetworks, capture and integrate scale-specific features of different fruits. The fruit features captured by different subnetworks are further complemented across scales to achieve network feature fusion. The features fused from the subnetworks are output after multi-scale aggregation, resulting in the extracted intermediate features.

The overall structure of MSA-Net is shown in Figure 3.

MSA-Net uses an asymmetric encoder–subnetwork–decoder architecture. The encoder extracts features at different scales through four residual blocks. In the subnetwork, AFeB and AMB modules are alternately stacked to integrate contextual information. In the decoder, the first three AFuB blocks adaptively sample and transfer fine-grained detail features to coarse-grained contextual features. The final AFuB blocks samples and transfers image details from noisy inputs, achieving the final output of MSA-Net, which detects multiple fruits in different locations. The feature maps of different types of fruits have different sizes, and MSA-Net has multiple subnetworks for processing feature maps of different sizes, enabling effective feature extraction of different fruits and reducing feature omissions.

2.3. Hough Transform Compensation Mechanism

To refine the recognition results of the MSA-Net model, this study proposed an ellipse compensation matrix based on Hough Transform, grounded in the shape characteristics of the fruits to be detected. The method’s workflow is illustrated in Figure 4.

The images detected by MSA-Net served as the input for the algorithm. In step 1, the Canny algorithm was applied to the input images to extract the edge lines of the fruits and branches. In step 2, three points were randomly selected and all points within neighborhoods of the same size centered on these three points were fitted into an ellipse using the least squares method. In step 3, a fourth point was randomly selected from the edge points, and it was determined whether this point lies on the fitted ellipse. If so, an ellipse was constructed using these four points. If the overlap area between the bounding box and the constructed ellipse exceeded 70%, a line was drawn between the center point of the bounding box and the center point of the ellipse, and the midpoint of this line was taken as the corrected fruit center coordinate.

2.4. Experimental Design

To validate the effectiveness of the MSA-Net model and the Hough Transform algorithm in this study, three sets of experiments were conducted sequentially.

In the first set of experiments, the MSA-Net model proposed in this study, along with several common deep learning network models, such as YOLOv4, YOLOv5s, YOLOv8s, and faster-RCNN models, was trained using the same dataset. A comparative performance analysis of the models was conducted.

The second set of experiments was used to test the performance of the proposed MSA-Net model to identify fruits of different categories in natural environments.

In the third set of experiments, 100 images of each type of fruit detected by MSA-Net were selected. The Hough Transform compensation mechanism was used to correct the fruit localization in each image. The actual center positions of the detected targets in the images were precisely determined manually using Photoshop 2018 software. The errors between the detection center positions before and after algorithm correction and the manually marked positions were recorded and analyzed.

The experimental hardware setup primarily involved a computer system featuring an Intel i5-14600kf processor, 32 GB RAM, and a GeForce GTX 4070 GPU (12G GDDR6X). The computer was configured with CUDA 12.0 parallel computing architecture and used the NVIDIA cuDNN 8.9.3 GPU acceleration library. The software simulation environment was built on the PyTorch deep learning framework (Python version 3.9). The data pre-processing involved the use of Matlab 2022 b. For configuring and managing the virtual environment, Anaconda was used, and program compilation and execution were carried out using Pycharm 2020. Model performance metrics mainly included P (precision), R (recall), the F1 score (harmonic average), the average precision (AP), and the mean average precision (mAP), as shown in Formulas (1)–(3).

$$\left\{\begin{array}{c}\begin{array}{c}\mathrm{precision}={\displaystyle \frac{{\mathrm{T}}_{\mathrm{p}}}{{\mathrm{T}}_{\mathrm{p}}{ + \mathrm{F}}_{\mathrm{p}}}}\\ \mathrm{recall}={\displaystyle \frac{{\mathrm{T}}_{\mathrm{p}}}{{\mathrm{T}}_{\mathrm{p}}{ + \mathrm{F}}_{\mathrm{N}}}}\end{array}\hspace{1em}|\\ |\\ {\mathrm{F}}_1={\displaystyle \frac{2\times \mathrm{precision}\times \mathrm{recall}}{\mathrm{precision}+\mathrm{recall}}}\end{array}\right.$$

(1)

$$\mathrm{AP}={\displaystyle \frac{\sum \mathrm{precision}}{\mathrm{N}}}$$

(2)

$$\mathrm{mAP}@0.5={\displaystyle \frac{{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{K}}{\mathrm{AP}}_{\mathrm{i}}}{{\mathrm{N}}_{\mathrm{C}}}}$$

(3)

where ${\mathrm{T}}_{\mathrm{p}}$ represents the number of fruits correctly identified, ${\mathrm{F}}_{\mathrm{p}}$ represents the number of fruits incorrectly identified, ${\mathrm{F}}_{\mathrm{N}}$ represents the number of missed fruits, $\mathrm{N}$ represents the total number of images, and ${\mathrm{N}}_{\mathrm{C}}$ represents the number of categories of each fruit. $\mathrm{AP}$, representing the integral of accuracy rate to recall rate, is equal to the area under the P–R curve, and $\mathrm{mAP}@0.5$ is the average of the average precision of all categories.

3. Results and Discussion

3.1. Detection Performance of the MSA-Net Model

In the first set of experiments, different models underwent training with an epoch of 300. The performance results of different models are shown in

Table 2. Performance of different models, including the MSA-Net model.

Model	Precision	Recall	F1	mAP@0.5	mAP@0.5:0.95
MSA-Net	97.56	92.21	94.81	92.72	61.85
YOLOV8s	97.84	83.57	90.14	86.77	56.14
YOLOV5s	96.51	84.97	90.37	88.69	58.79
FasterRCNN	93.19	90.48	91.82	76.88	50.77
YOLOV4tiny	90.86	87.74	89.27	86.31	55.98

.

The precision of the MSA-Net model was 97.56, which is 0.28 lower than that of YOLOv8s but higher than that of YOLOv5s, Faster R-CNN, and YOLOv4-tiny by 1.05, 4.37, and 6.7, respectively. MSA-Net achieved the highest recall value of 92.21, which is 8.64, 7.24, 1.73, and 4.47 higher than the recall values of YOLOv8s, YOLOv5s, Faster R-CNN, and YOLOv4-tiny, respectively. MSA-Net had the highest F1 score of 94.81, which is 4.67, 4.44, 2.99, and 5.54 higher than the F1 scores of YOLOv8s, YOLOv5s, Faster R-CNN, and YOLOv4-tiny, respectively. MSA-Net also achieved the highest mAP@0.5 and mAP@0.5:0.95 values, which were 92.72 and 61.85, respectively.

While the precision value of MSA-Net is slightly higher than that of other models, its recall value is significantly better. This is due to the model’s use of a multi-scale network for detection, where multiple subnetworks are configured to process the feature maps of fruits of different sizes based on the type of fruit being detected. This comprehensive approach to recognizing various types of fruits reduces missed detections, resulting in a higher recall value. The detection performance of MSA-Net is similar to that of conventional models for detecting single fruits, and it has the accuracy to complete actual detection [18]. MSA-Net’s highest mAP@0.5 and mAP@0.5:0.95 values indicate that the model has high recognition accuracy and can detect multiple types of fruits with high confidence.

The changes in mAP@0.5 during the training process for different models are shown in Figure 5.

From Figure 5, it can be seen that the mAP@0.5 curve of the MSA-Net model rose rapidly with the increase in epochs and gradually stabilized, ultimately converging to a value superior to that of other models. MSA-Net uses multiple subnetworks to extract features, enabling the model to quickly and effectively learn the characteristics of various fruits and achieve a high mAP value.

3.2. Analysis of MSA-Net Recognition Results

The detection results of MSA-Net for fruits under different environmental conditions are shown in Figure 6.

From the detection results of MSA-Net, it can be seen that for strawberries under various environmental conditions, such as occlusion and backlighting, the model performs comprehensive detection with a confidence level above 0.82. For tomatoes in environments with close-ups, distant views, and dense occlusion, the model detects tomatoes with a confidence level above 0.78. For lychees and blueberries, under various conditions, such as distance, close-up, backlighting, and exposure, the model can comprehensively detect lychees in images with a confidence level above 0.70. The robustness of MSA-Net is reflected in its recognition accuracy under various lighting and distance conditions. The multi-scale detection layers of MSA-Net effectively adapt to different fruits in various environments, achieving accurate detection.

3.3. Analysis of Hough Transform Compensation Results

The Hough Transform compensation mechanism was used to correct the detection results of MSA-Net. The errors before and after correction are shown in

Table 3. Hough Transform compensation results.

Image Type	Initial Positioning Average Error (Close Range)/Pixel	Initial Positioning Average Error (Close Range)/Pixel	Positioning Accuracy Improvement (Close Range)/Percentage	Initial Positioning Average Error (Distant Range)/Pixel	Average Error after Compensation (Distant Range)/Pixel	Positioning Accuracy Improvement (Distant Range)/Percentage
Blueberry	29	18	37.93%	18	17	5.55%
Lychee	34	30	11.76%	25	23	8.00%
Tomato	41	22	46.34%	30	22	26.67%
Strawberry	26	25	3.84%	19	16	15.79%
Average	33	24	24.97%	23	20	14.01%

.

As shown in the table, the error reduction was related to different distances and fruit types. For close-up fruit images, the average localization error for different fruits decreased by 8 pixels. Among these, the localization error for tomatoes decreased the most, by 19 pixels. The localization error for blueberries decreased by an average of 11 pixels, for lychees by an average of 4 pixels, and for strawberries by only 1 pixel. For distant fruit images, the average localization error for different fruits decreased by 3 pixels. The localization error for tomatoes decreased the most, by 8 pixels, while the errors for blueberries, lychees, and strawberries decreased by 1, 2, and 3 pixels, respectively. The positioning error of fruits at close range reduced by an average of 24.97%, while the positioning error of fruits at long distances reduced by an average of 14.01%.

From these results, it can be seen that the Hough Transform algorithm significantly reduces the localization error for tomatoes at different distances because the shape of tomatoes closely matches an ellipse, making the Hough Transform results more accurate and effectively correcting the localization error. In contrast, the shape of strawberries differs more from a circle, resulting in a smaller change in the localization error after the algorithm is applied. Lychees and blueberries, having shapes closer to a circle, also show good error correction results.

The correction effect of the Hough Transform compensation mechanism is shown in Figure 7.

From the aforementioned results, it can be seen that the proposed Hough Transform compensation mechanism can further improve the localization accuracy for different types of spherical fruits. More accurate coordinates can be used in subsequent operations, such as picking, thereby increasing the success rate of these tasks. However, the effectiveness of the Hough Transform compensation mechanism is mainly reflected in approximately circular fruits, which can further improve positioning accuracy. For irregularly shaped fruits, there are errors in the detection process of Hough Transform, which affects the effectiveness of the compensation mechanism but still has a certain reduction effect on positioning errors (such as strawberries that are approximately diamond shaped).

The method proposed in this article can be easily integrated into automation systems to reduce labor and improve efficiency [19]. For example, combining this technology with other innovations, such as drones, soil sensors, or weather models, can create comprehensive smart agriculture solutions that improve fruit recognition and overall farm management [20]. The proposed method demonstrates high accuracy in correcting the localization of spherical fruits, but it still has some limitations. The method uses Hough Transform for elliptical detection, which works well when dealing with regularly shaped or symmetrical fruits. However, for irregularly shaped or asymmetrical fruits, such as strawberries, the model’s performance is noticeably constrained. This drawback primarily arises from the limitations of elliptical detection in recognizing non-spherical objects [21]. The current model also lacks robustness when handling various environmental conditions, such as changes in lighting, occlusion, and fruit overlap, leading to blurred fruit boundaries and consequently affecting the detection accuracy of Hough Transform [22]. Future research should consider incorporating new morphological feature analysis methods and developing localization compensation algorithms adaptable to various fruit shapes to enhance the model’s applicability across different fruit morphologies and improve its robustness under complex environmental conditions [23].

4. Conclusions

This study aimed to develop a method capable of recognizing various fruits. First, an MSA-Net model was proposed for identifying multiple fruits. Based on the shape characteristics of different fruits, a Hough Transform compensation mechanism was introduced to further refine the model’s recognition results. The specific conclusions are as follows:

• MSA-Net can effectively learn the features of different types of fruits. For a comprehensive dataset including blueberries, lychees, strawberries, and tomatoes, the model achieved a precision of 97.56, a recall of 92.21, and an mAP@0.5 of 94.81, accurately identifying various fruits in the environment.
• The introduction of the Hough Transform ellipse detection compensation mechanism further refines the initial localization of spherical fruits. For close-up fruit images, the average localization error for different fruits decreased by 8.8 pixels. For distant fruit images, the average localization error for different fruits decreased by 3.5 pixels, further improving the accuracy of fruit localization.

The proposed model for recognizing multiple fruits has a certain degree of scalability. When new fruit varieties need to be recognized, the existing model can be appropriately extended and fine-tuned, eliminating the need to develop a new model from scratch. This demonstrates considerable application value.