Intelligent Fruit Localization and Grasping Method Based on YOLO VX Model and 3D Vision

Abstract

Recent years have seen significant interest among agricultural researchers in using robotics and machine vision to enhance intelligent orchard harvesting efficiency. This study proposes an improved hybrid framework integrating YOLO VX deep learning, 3D object recognition, and SLAM-based navigation for harvesting ripe fruits in greenhouse environments, achieving servo control of robotic arms with flexible end-effectors. The method comprises three key components: First, a fruit sample database containing varying maturity levels and morphological features is established, interfaced with an optimized YOLO VX model for target fruit identification. Second, a 3D camera acquires the target fruit’s spatial position and orientation data in real time, and these data are stored in the collaborative robot’s microcontroller. Finally, employing binocular calibration and triangulation, the SLAM navigation module guides the robotic arm to the designated picking location via unobstructed target positioning. Comprehensive comparative experiments between the improved YOLO v12n model and earlier versions were conducted to validate its performance. The results demonstrate that the optimized model surpasses traditional recognition and harvesting methods, offering superior target fruit identification response (minimum 30.9ms) and significantly higher accuracy (91.14%).

1. Introduction

In recent years, the application of intelligent robots + agriculture has attracted people’s attention; in particular, automation and intelligent target positioning pickup technology under indoor constant temperature conditions are more favored by users. However, the current research methods have many shortcomings, especially in real-time control, occlusion robustness, and lighting conditions; in particular, in the aspect of target detection in dynamic environment, it is not enough to only consider YOLO VX, 3D vision or Arduino Control alone, as this cannot improve the efficiency and comprehensive performance of fruit recognition. It is necessary to effectively integrate the three. In order to improve the efficiency of fruit picking in smart orchards, determining how to use automatic recognition and positioning, mobile manipulator navigation, and flexible grasping technology to realize real-time intelligent picking of fruit is the main technical problem to be solved in smart agriculture. To this end, it is necessary to find efficient deep learning models, collect image sets to train the positive sample library, design a reasonable mature fruit matching algorithm, use 3D vision technology to obtain the spatial pose and appearance size information of the center point of the target entity, transmit it to the mechanical arm, and control the flexible end to achieve accurate grasping.

Scholars have carried out related research on target detection, 3D positioning, dynamic environment adaptation, grasping planning, and system integration. Firstly, in terms of target detection and fruit recognition, YOLO-series models are central to fruit localization. References achieved an average precision (AP) of 99.5% in kiwifruit localization using YOLOv5x combined with calyx detection, significantly reducing positioning errors (X/Y/Z-axis deviations ≤7.9 mm) [1,2,3]. Reference [4] improved YOLOv8s by introducing a dynamic sparse attention mechanism (BiFormer) and a small-target detection layer, increasing the mAP for citrus detection to 88.45% and optimizing depth estimation accuracy (MAE = 0.53). Additionally, Reference [5] proposed the FasterNet-YOLOv7 model, which enhances feature extraction and attention mechanisms (CoTAttention) for underwater complex environments, achieving an mAP of 91.8% and real-time detection performance at 83.21 FPS. However, existing studies still face challenges such as insufficient accuracy in small-target detection (e.g., the fragility of berry skins in Reference [3]) and dynamic occlusions (e.g., dynamic interference in SLAM systems in Reference [6]). Secondly, in terms of 3D positioning and hand–eye calibration, 3D positioning technologies primarily include monocular, binocular, and depth-camera-based methods. Reference [1] optimized 3D coordinates by matching calyx feature points in binocular images combined with depth maps (DIDM), reducing localization errors by 53.8% compared to traditional methods. Reference [7] proposed a low-cost monocular vision solution that fuses multi-resolution depth estimation, achieving a depth estimation MAE of 0.49 within a 30–60 cm range and a grasping success rate exceeding 80%. For hand–eye calibration, Reference [8] established an “I = AXB” transformation matrix using robot base point clouds and learning algorithms, achieving a translation deviation of 0.93 mm and a rotation deviation of 0.265, with calibration completed in 1 s, significantly outperforming traditional marker-dependent methods. Thirdly, in terms of dynamic environment adaptation and SLAM technology, robustness in dynamic environments is critical for fruit grasping systems. References [9] and [6] proposed dynamic SLAM systems based on YOLOv5s and Darknet19-YOLOv3, respectively, which remove dynamic feature points through semantic segmentation and geometric constraints (RANSAC), reducing trajectory errors (RMSE) by 51.28–98.13% on the TUM dataset. Reference [10] further introduced double polyline constraints and Gaussian models to segment dynamic targets, enhancing localization stability. However, challenges remain, such as complex lighting (Reference [4]), disordered scenes (Reference [11]), and high-speed motion (Reference [12]). Fourthly, in terms of grasp planning and system integration, grasp planning requires integrating target pose estimation and robotic arm control. Reference [12] proposed a YOLO-GGCNN cascaded model that constructs environmental maps through multi-robot collaborative SLAM, achieving a grasping accuracy of 86%. Reference [2] emphasized visual servo control and continuous behavior coordination, proposing an integrated framework for orchard-wide motion and grasping to enhance commercial feasibility. Additionally, Reference [13] reviewed three key tasks in vision-based grasping (target localization, pose estimation, and grasp estimation), highlighting the potential of end-to-end learning methods (e.g., 6D force estimation using the 9DTact tactile sensor in Reference [14]) in complex contact scenarios. In a word, current research has made significant progress in fruit detection accuracy, 3D positioning efficiency, and robustness in dynamic environments.

(1)

The above studies provide some methods for fruit picking, but there are still deficiencies in fruit maturity matching, random location capture, and picking methods. In terms of application scenarios, this study is limited to greenhouse or strongly light-controlled environments, because the picking path of the robot is about 40 klux~80 klux, and the indoor light intensity is less than this value, so direct sunlight should be avoided as much as possible. To this end, this method focuses on using the latest YOLO VX deep learning model under the indoor constant-temperature condition, and synchronously calculates the spatial appearance and appearance characteristics sending the position information to the robotic arm in real time through visual calibration and servo control and then relying on the three-finger flexible claw to complete the picking. The principal contributions of this study are summarized as follows:

(2)

We developed a hybrid multimodal autonomous fruit ripeness recognition algorithm based on C3K2 and SPPF, reducing real-time control latency to 30.9 ms; this is more efficient than earlier models. By implementing cascaded pooling to cover larger image regions, the system achieves enhanced robustness in detecting large-scale targets. Compared to conventional SPP methods, the serial design preserves richer edge information, resulting in finer feature extraction through small-kernel pooling.

(3)

We implemented a spatial coordination framework that integrates 3D-vision-derived centroid coordinates and dimensional parameters into a collaborative robotic control system, enforcing synchronized locomotion between the mobile chassis and manipulator via triaxial coordinate system alignment.

(4)

A hybrid communication architecture combining TX/RX serial protocol and Ethernet connection is designed to establish a unified address scheme for 3D spatial data and Arduino-based motion control, which optimizes the path planning of autonomous mobile robot (AMR) operation compared with traditional methods.

(5)

Experimental validation demonstrated that the proposed machine vision–Arduino integrated framework achieves 91.14% target recognition accuracy with ±1.5 mm positioning precision in agricultural harvesting scenarios.

2. System Scheme Design

The system design comprises three primary components: a mobile aerial delivery platform, an image processing unit, and a target harvesting unit, which collectively integrate a mobile robot, a collaborative robotic arm, a 3D camera system, and fruit tree detection modules. To optimize computational efficiency and data transmission, each subsystem processes and transmits collected data through dedicated microcontroller-based systems [9,15,16].

Figure 1 presents the system layout and structural configuration: (1) mobile chassis, (2) collection platform, (3) feeding mechanism, (4) flexible gripper assembly, (5) vision support frame, (6) 3D vision sensor, (7) adaptive lighting system, (8) target fruit tree, and (9) SLAM (Simultaneous Localization and Mapping) module. Correspondingly, Figure 2 illustrates the visual servo control architecture and SLAM implementation scheme specifically developed for this methodology. The end of the robot arm adopts a flexible three-finger gripper, which is controlled by pneumatic contraction and expansion. The maximum working frequency is 300 times/min, the accuracy range is ±0.05 mm, the service life is 3 million times, and the safe working pressure is 120 KPa.

According to the system design flow chart in Figure 2, this study aims to develop an intelligent fruit positioning and grasping system based on the YOLO VX model and three-dimensional vision technology, so as to realize efficient automatic operation through the collaboration of a mobile robot, a cooperative mechanical arm, and multimodal sensing technology [17]. The technical process is divided into a perception layer and a control layer: the perception layer first obtains the fruit information through the image acquisition module and uses the YOLO VX model to accurately identify the spatial coordinates, size, and attitude of the fruit. Relying on the Arduino system, the control layer regulates the motor through TB6612ENG and A498.8 drivers, realizes the multi-freedom motion planning of the robotic arm end actuator, and synchronously coordinates the positioning of the wheel mobile platform and the precise operation of the screw capture mechanism. Through the closed-loop architecture of “image recognition–location–control”, the system integrates the deep learning algorithm with three-dimensional visual depth, solves the problems of a complex environment and target occlusion in traditional picking, and combines high-precision recognition and dynamic adaptability [18,19,20,21]. This study can provide a technical paradigm for automated picking in agriculture and promote the intelligent application of robot technology in unstructured scenarios.

3. Establishing a YOLO X Network Model for Fruit Target Recognition

The proposed framework was implemented through a Python V3.10.0-based virtual environment under a Linux environment, where the network model was trained and validated with stratified dataset partitioning [12,22]. Our core innovation lies in the Improved YOLOX Target-Following Algorithm (IYTFA), specifically designed for cooperative robotic applications. As depicted in Figure 3, the system architecture comprises three functionally integrated modules: (a) multimodal target detection, (b) adaptive tracking, and (c) dynamic following control. The technical workflow unfolded as follows: First, we architecturally enhanced YOLOX by substituting its Darknet53 backbone with MobileNet V2X, achieving a 37% reduction in computational load while maintaining 92% feature extraction fidelity. RGB video streams are processed through this optimized network to generate hierarchical feature maps [23,24,25]. Subsequently, a dual-branch learning paradigm was implemented, where dedicated re-identification and detection loss functions simultaneously optimize the recognition and localization branches, respectively. For motion tracking, we developed an Enhanced Kalman Filter (EKF) incorporating (i) trajectory affinity-based target association, (ii) occlusion-aware discrimination through depth probability constraints, and (iii) adaptive state prediction during visual occlusion scenarios. Furthermore, a vision-guided active pursuit strategy was devised, enabling real-time pose adjustment of the mobile robot via visual servo control mechanisms.

The algorithm network is divided into three parts, namely the main trunk network, the network layer, and the prediction layer. Its backbone network uses a Darknet53 feature extraction network, the network layer adopts a feature pyramid network, and the prediction layer uses three decoupling heads. The input pictures of the fruit set were shallow feature extraction in the backbone network part, three feature layers for deep feature extraction, and three decoupling heads for target detection. However, YOLOX backbone networks usually use Darknet53 networks, which have problems such as a large model size and slow inference speed. Therefore, to realize mobile robot real-time target detection, this method proposed the YOLOX M2X network, a YOLOX backbone network using the lightweight feature extraction network C3K2; the convolution core layer of the network is a deep separable convolution layer, the output feature diagram channel is reduced to half and then merged with the original convolution layer extraction. Compared with the early YOLO model, the C3K2 module has faster detection speed and further reduces the calculation cost by removing redundant convolution paths. The parameter tuning process comprises five distinct stages: Stage 1 (Rounds 0–100): Perform basic training while analyzing real-time mAP@50-95 responses. Stage 2 (Rounds 101–150): Optimize the learning rate to mitigate potential oscillations and continue training. Stage 3 (Rounds 151–200): Further reduce the learning rate. Stage 4 (Rounds 201–250): Analyze the training and validation loss curves. If significant oscillations are detected, implement corrective measures: further reduce the learning rate, decrease the bounding box loss weight, and increase the segmentation loss weight. Stage 5 (Rounds 251–300): Fine-tune and refine model parameters to identify optimal hyperparameters.

The proposed system integrates the AT-S1000-06C-S 3D binocular structured-light depth camera, a multi-sensor fusion device equipped with four synergistic sensing elements: (i) dual infrared stereo vision modules, (ii) an active IR pattern projector, and (iii) an RGB chromatic imaging sensor [26,27,28]. As demonstrated in Figure 4 and Figure 5, this configuration enables millimeter-level depth resolution through synchronized operation of its optical subsystems—the IR projector emits encoded light patterns while paired IR cameras perform phase-shift analysis, with the RGB sensor concurrently capturing 1280 × 720 texture information at 30 FPS.

To ensure the accuracy of fruit picking, the equal error method was used to disperse the contour curves. According to the diameter of the fruit captured by the camera, the coordinates of the grasping points at both ends were determined, which were defined as $F_0$ and $F_N$, respectively. Then the actual grasping points of the robot are n + 1, and the next node can be calculated from Equation (1):

$$t_{i+1}=t_i+\mathsf{\Delta }{t}_i=t_i+{\displaystyle \frac{{\left(8\epsilon R_i\right)}^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.}}{\left|F^{’}\left(t_i\right)\right|}}$$

(1)

In formula, $F\left(t_i\right)$ indicates the current node, $F\left(t_{i+1}\right)$ represents the next discrete node as determined by the equal error method, $t_i$ indicates the $F\left(t_i\right)$ parameter value at the node, ε represents discrete accuracy, and $R_i$ indicates the radius of curvature at $F\left(t_i\right)$.

4. Visual Recognition System Design

4.1. Collaborative Robotic Arm and Binocular Camera Calibration

The stereo calibration process serves to establish a precise geometric mapping between pixel coordinates and real-world spatial dimensions. This paper adopts the calibration method of eye-to-hand, as shown in Figure 6. The calibration process is performed to obtain the internal and external parameters of the camera, so as to calculate the relationship between the pixel coordinates and the actual coordinates through the transformation of the formula [29,30]. In the figure, A represents the coordinate transformation relationship between the camera and the robot base, and B represents the coordinate transformation relationship between the camera and the earth. The mathematical expression of actual coordinates $({\mathrm{X}}_{\mathrm{W}},{\mathrm{Y}}_{\mathrm{W}},{\mathrm{Z}}_{\mathrm{W}})$ and pixel coordinates (u, v) is shown in Equation (2):

$${\mathrm{Z}}_{\mathrm{c}}\left[\begin{array}{c}\mathrm{u}\\ \mathrm{v}\\ 1\end{array}\right]=\left[\begin{array}{cccc}{\displaystyle \frac{\mathrm{f}}{{\mathrm{d}}_{\mathrm{x}}}}& 0& {\mathrm{u}}_0& 0\\ 0& {\displaystyle \frac{\mathrm{f}}{{\mathrm{d}}_{\mathrm{y}}}}& {\mathrm{v}}_0& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{cc}\mathrm{R}& \mathrm{T}\\ 0^{\mathrm{T}}& 1\end{array}\right]\left[\begin{array}{c}{\mathrm{X}}_{\mathrm{w}}\\ {\mathrm{Y}}_{\mathrm{w}}\\ {\mathrm{z}}_{\mathrm{w}}\\ 1\end{array}\right]={\mathrm{M}}_1{\mathrm{M}}_2\mathrm{X}$$

(2)

In the formula, M refers to camera internal parameters, including the focal length f, the width of a single pixel element ${\mathrm{d}}_{\mathrm{y}}$, the height of a single pixel element ${\mathrm{d}}_{\mathrm{x}}$, the pixel value of the image length ${\mathrm{u}}_0$, and the pixel value of the image width ${\mathrm{v}}_0$. In addition to these five parameters, ${\mathrm{M}}_2$ refers to external parameters of the camera, including the rotation matrix {R} and the translation matrix {T}.

In this method, the calibration board data file and the calibration board image file can be obtained by using the “gen caltab” operator, as shown in Figure 7, where the first and second parameters of the operator represent the number of origin markers in each row and each column, the third parameter represents the distance between each two MARK points (in meters), the fourth parameter represents the diameter of the MARK point, and the fifth and sixth parameters represent the storage location of the calibration board file and the storage location of the image, respectively.

The calibration process begins with preparing a calibration board and capturing its images. To ensure accuracy, the images must meet strict criteria: high clarity, the absence of occlusions or stains on the calibration plate, and clear visibility of all marker points. Since image quality directly impacts calibration results, multiple captures are required, involving systematic rotation and translation of the calibration plate to cover diverse spatial orientations. This iterative approach enhances the robustness and precision of the calibration outcome [31,32]. Figure 7 shows part of the pictures collected by the camera and calibration plate in this study.

4.2. Obtaining Fruit Features

The external dimensions of the target fruit were quantified using binocular camera measurements, as summarized in

Table 1. Measurement indicators of binocular cameras.

I/O Parameter	Measurement Index
Input	Real-time image and video resolution	Live format (RGB/MP4)	Camera center position and distortion coefficient	Camera initialization parameters
Output	Image and video resolution	Image video format	Depth information after image/video matching	The distance of the target object from the camera

. The workflow proceeded as follows: (1) Image Preprocessing: The acquired images were filtered using median and Gaussian filters to reduce noise. (2) Contour Extraction: Feature matching, adaptive binarization, area thresholding, and morphological operations were applied to isolate the fruit region. (3) Dimensional Analysis: Subpixel-accurate contours were obtained through the eXtended Line Descriptor (XLD) method, from which the pixel-based length and width measurements were derived. The process is shown in Figure 8.

To ensure the accuracy of the fruit shape measurements, the fruit edge images were next acquired for the XLD contour (subpixel edge, extended line descriptions) obtained from the box edge image. The purpose of contour extraction is to extract the spatial direction of the gray value in the image by calculating the reciprocal of the gray value change. For the processing of edge and line features with large gradients, for a continuous image function f(x, y), the gradient is expressed in Equation (3):

$$\partial \mathrm{f} (\mathrm{x}, \mathrm{y})= \mathrm{G}\left(\mathrm{x}, \mathrm{y}\right)={\left[{\displaystyle \frac{\partial \mathrm{f}}{\partial \mathrm{x}}} {\displaystyle \frac{\partial \mathrm{f}}{\partial \mathrm{y}}}\right]}^{\mathrm{T}}$$

(3)

The basic idea of obtaining the image edge is as follows: after the image is smoothed and filtered by a Gaussian filter, it is processed by non-extremum suppression technology to obtain the image edge. The steps are as follows:

First, use the smooth image f(x, y) with a Gaussian filter G(x, y) to produce a smoothed image as shown in Equations (4) and (5):

$$\mathrm{G}\left(\mathrm{x}, \mathrm{y}\right)={\mathrm{e}}^{{\displaystyle \frac{{\mathrm{x}}^2+{\mathrm{y}}^2}{{2\mathsf{\sigma } }^2}}}$$

(4)

$${\mathrm{f}}_{\mathrm{s}}(\mathrm{x}, \mathrm{y})=\mathrm{G} (\mathrm{x}, \mathrm{y}) \mathrm{\ast } \mathrm{f} (\mathrm{x}, \mathrm{y})$$

(5)

Second, calculate the magnitude M(x, y) and direction α(x, y) of each pixel of the above image as shown in Equations (6) and (7):

$$\mathrm{M}=\sqrt{{\mathrm{G}(\mathrm{x}, \mathrm{y})}^2+\mathrm{f}{\left(\mathrm{x}, \mathrm{y}\right)}^2}$$

(6)

$$\mathsf{\alpha }\left(\mathrm{x}, \mathrm{y}\right)=\arctan \left({\displaystyle \frac{\mathrm{G}(\mathrm{x}, \mathrm{y})}{\mathrm{f} (\mathrm{x}, \mathrm{y})}}\right)$$

(7)

Third, set the local maximum point to 0 to refine the edge.

Fourth, set high and low thresholds “${\mathrm{T}}_1$” and “${\mathrm{T}}_2$” to detect and connect edges.

After the pixel values of the fruit length and width were obtained, the previous calibration data was introduced below the program, and the world coordinates were solved using the pixel coordinate conversion method. The first parameter of the operator is the camera parameter, the second parameter is the camera pose, and the fifth parameter is the unit of measurement, with mm generally being chosen, that is, the actual length of 100 pixels is X mm, and then Equation (8) is used to the formula to convert pixel coordinates to actual coordinates:

$$S{J}_c:=-{XS}_c \ast \mathrm{X}/50+0.36$$

(8)

Here, $S{J}_c$ is the actual coordinate, and ${XS}_c$ is the pixel coordinate.

4.3. Fruit Grasping Position and Force Mixing Control

To achieve precise force regulation during fruit grasping, an intelligent control algorithm was developed to enhance the mechanical arm’s force control. This algorithm compensates for PID control limitations by incorporating a neural network (NN), leveraging its nonlinear approximation capability, adaptive learning, and self-optimization to improve control stability, speed, and robustness.

Position–force mixing control transforms the combined system and computational model of the actual operating arm into a series of independent, decoupled unit mass systems by using the dynamic model of Cartesian space. The basic block diagram of the position–force mixing control is shown in Figure 9. The force control part consists of two channels, neural network PI and force feed-forward. Neural network PI channels take the expected Cartesian space generalized force F_d at the end of the robot as a given condition, and the force feedback is obtained from the force sensor measurements. The control system is introduced with the matrices S and $\overline{S}$ to determine the control task that different degrees of freedom should take according to the task. S is a diagonal array; the elements on the diagonal array are 0 or 1, that is, $S\triangleq \left[0,I\right],\overline{S}\triangleq \left[I,0\right]$. S and $\overline{S}$ are equivalent to an interlock switch and are used to select the control mode of each degree of freedom. The traditional PD control algorithm is selected as the position control strategy because the position control requires a fast response speed. The force control strategy uses the neural network PI control algorithm, and the force control requires less error. $\overline{S}$= 1 − S. Then, the control moment of the robotic arm under the Cartesian coordinate system is calculated as shown in Equation (9):

$$\tau =K_p\overline{S}∆x+K_v\overline{S}∆x+K_{Fp}S∆F+K_{FI}S{\int }_0^t∆Fdt+F_d$$

(9)

According to the Jacobian matrix of the arm, the driving moment corresponding to each joint of the arm can be calculated as shown in Equation (10):

$${\tau }_q={J^T\tau =K}_p{J}^{-1}\overline{S}J∆q+K_v{J}^{-1}\overline{S}J∆q+{J^T(K}_{Fp}S∆F+K_{FI}S{\int }_0^t∆Fdt+F_d)$$

(10)

4.4. Motion Control System Design

The developed fruit-picking mobile manipulator integrates an Arduino-based control system that communicates with the machine vision module and coordinates structural actuation [33,34,35], as illustrated in Figure 10. To ensure operational safety and stability, the system implements an incremental discrete PID closed-loop control strategy for motor speed regulation [18,19,36,37].

According to the principle of positional PID control, the control amount of the robot at time n − 1 is given by Equations (11)–(13):

$$\mathrm{u}\left(\mathrm{n}-1\right)=K_p\{e\left(n-1\right)+{\displaystyle \frac{T}{T_i}}{\sum }_{i=0}^{n}e\left(i\right)+{\displaystyle \frac{T_d}{T}}[e\left(n-1\right)-e(n-2)]\}$$

(11)

With the hypothesis

$$∆\mathrm{u} \left(\mathrm{n}\right)=\mathrm{u}(\mathrm{n})-\mathrm{u} (\mathrm{n}-1)$$

(12)

get

$$∆\mathrm{u}\left(\mathrm{n}\right)=K_p\left[e\left(n\right)-e\left(n-1\right)\right]+{\displaystyle \frac{{K_{p}T}_d}{T_i}}e\left(n\right)+{\displaystyle \frac{{K_{p}T}_d}{T}}[e\left(n\right)-2e\left(n-1\right)+e(n-2)]$$

(13)

Make K $=K_p$ the integral coefficient and $K_d=K_p{\displaystyle \frac{T_d}{T}}$ the differential coefficient; the above formula can be simplified as Equation (14):

$$∆\mathrm{u}\left(\mathrm{n}\right)=K_p\left[e\left(n\right)-e\left(n-1\right)\right]+K_{i}e\left(n\right)+K_d[e\left(n\right)-2e(n-1)+e(n-2\left)\right]$$

(14)

Here, incremental discrete PID control is used to form a speed closed loop, that is, to calculate the incremental output Pwm, as shown in Equation (15):

$$\mathrm{P}\mathrm{w}\mathrm{m} =K_p\left[e\left(n\right)-e\left(n-1\right)\right]+K_{i}e\left(n\right)+K_d[e\left(n\right)-2e(n-1)+e(n-2\left)\right]$$

(15)

Only PI control is used in the speed closed-loop control system, so the PID formula can be simplified as Equation (16):

$$\mathrm{P}\mathrm{w}\mathrm{m} =K_p\left[e\left(n\right)-e\left(n-1\right)\right]+K_{i}e\left(n\right)$$

(16)

where e(n) is the current deviation, e(n − 1) is the deviation of the neighbor, e(n − 2) is the deviation of the interval, Pwm is the incremental output, T is the sampling period, and n is the discrete variable.

Based on the Arduino microcontroller, the speed of the motor is controlled and programmed, and the closed-loop speed is used for feedback adjustment so that the speed of each motor is consistent, thereby ensuring the safety and stability of the robot during the movement process [20,21,38,39,40,41,42].

5. Test

5.1. Dimensions and Location

Firstly, the calibration test of the 3D camera was conducted to solve the calibration values of the internal and external parameters. The internal parameters obtained are shown in

Table 2. Internal parameters of camera calibration.

Camera Internal Parameter	Calibration Value	Camera Internal Parameter	Calibration Value
Focal length f/m	0.0120	Single pixel height/m	4.400 × 10−6
$\mathrm{Distortion} \mathrm{coefficient}/{\mathrm{m}}^{-2}$	−397.275	Center point X coordinate/pixel	813.183
Single pixel width/m	4.399 × 10−6	Center point Y coordinate/pixel	618.186

, and the external parameters are shown in

Table 3. Camera calibration external parameters.

Camera External Parameter	Calibration Value	Camera External Parameter	Calibration Value
Translation matrix T		Rotation matrix R
Δx/m	−0.0301154	α/(°)	1.19178
Δy/m	−0.0267293	θ/(°)	359.161
Δz/m	0.599653	γ/(°)	0.231355

:

By calibrating the internal and external parameters of the camera, it can be seen that the internal parameters of the camera are position indicators, mainly including the focal length, distortion coefficient, pixel shape, and center point coordinates, while the external parameter is the pose index, which mainly consists of three translation matrices and three rotation matrices with six degrees of freedom. The combination of internal and external parameters can solve the position and posture of the target object in space and facilitate the accurate grasp of the terminal mechanical arm.

Next, 2505 training sets, 850 verification sets, and 523 test sets were selected for the experiment, as shown in

Table 4. Statistics of various types of apple datasets.

Dataset	Training Set	Validation Set	Test Set	Total Quantity
Blade shielding type	652	205	106	963
Fruit overlapping type	643	211	105	959
Surface glare	608	226	105	939
Missing characteristics	602	208	107	917
Total quantity	2505	850	523	3778

. The positive sample set was formed by a high-resolution camera used to shoot videos of fruits with different maturities and different forms under the same conditions. Then the videos were imported into labelimg plugin, and each image was read and formed. After these images were imported into the software, they were annotated by the feature extraction method. As shown in Figure 11, the ripe and unripe fruits were distinguished by RGB color (an R value greater than 128 was valid), and the occluded and non-occluded fruits were determined by roundness and fit degree (greater than 50% was valid). To reduce the error, the edge of the fruit was marked as closely as possible. In this process, the UPSAMPLE data enhancement module was used for the core operation of increasing the spatial resolution of the feature map. We used transposed convolution to insert blanks (strides) between the input elements and then filled in the convolution.

As shown in

Table 5. Training parameters.

Training Parameter	Value
Input image size/pixel	640 × 640 × 3
Init learning rate	0.01
Batch_size	16
Epoch	300

, the training set serves as the learning material for the model, used to update the weights and learn the patterns. The validation set is used to evaluate the model’s performance during training, fine-tune hyperparameters, and prevent overfitting. It provides data not directly used for training the model’s weights, which helps assess the model’s generalization ability—its capability to make accurate predictions on unseen data. Objectively comparing the model’s performance under different combinations of hyperparameters makes it evident that the model performs well with the hyperparameter settings shown in Table 5 (learning rate 0.01, training batch size 16, epoch 300). The test set is used to further evaluate the model’s generalization ability. After the problem was solved, three kinds of loss functions appeared, namely a. a box loss curve, b. a class loss curve, and c. a confidence loss curve, and 300 training cycles were carried out.

5.2. Outline Dimension Test

After the vision system of this design was calibrated, the pixel values obtained by the preprocessing process of the acquired images and the converted actual values were obtained, as shown in

Table 6. Outline dimension test.

Size Type	Measured Value/pix	Actual Value/pix	Differential Rate/%
Length value of the identification box	487.889	488.600	0.15
Wide value of the identification box	238.109	240.100	0.83
Calculated length/mm	8.990	9.770	8
Calculated width/mm	7.315	7.710	5

. And the position identification and characteristic parameter output are shown in Figure 11.

To comprehensively assess the performance of the optimized YOLO model in comparison to its predecessors, we conducted extensive testing across multiple YOLO series models using datasets with robust generalization capabilities [43,44,45,46,47,48]. Our evaluation framework incorporated four key performance metrics: accuracy, recall rate, F1 score (excluding HALCON due to unavailability), and mean precision, as detailed in

Table 7. Comprehensive performance comparison of YOLO v12n and early models/%.

Model Category	Accuracy	Recall	F1 Score	Mean Accuracy
YOLO v8n	90.66	84.57	87.36	93.32
YOLO v10n	90.98	84.05	87.38	90.06
YOLO v12n	91.14	84.32	87.60	91.76
HALCON	66.67	41.67	/	58.54

. The comparative analysis clearly demonstrates that the YOLOv12n model achieves superior overall performance [49,50,51,52,53].

As shown in Table 7, the model performance evaluation presents comprehensive testing results on the benchmark dataset through rigorous offline evaluation, demonstrating that the YOLOv12n model achieves the most significant accuracy improvement among all evaluated versions. The assessment incorporates four key metrics: precision (the ratio of correctly predicted positives to all predicted positives), recall (the ratio of correctly predicted positives to actual positives), F1 score (harmonic mean of precision and recall), and mean accuracy (area under the precision–recall curve). This analysis reveals critical performance trade-offs—high-precision models tend to generate false negatives, while high-recall models produce false positives, with the F1 score ensuring balanced performance. These results confirm the model’s strong generalization capability, further validated by

Table 8. Comparison of detection time (ms) between YOLO v12n and early models.

Detection Link	HALCON	YOLO v8n	YOLO v10n	YOLO v12n
Image preprocessing	4.2	3.1 ms	2.8 ms	2.9 ms
image segmentation	372.9	/	/	/
Morphological processing	708.3	/	/	/
Feature extraction	270.2	/	/	/
Target screening/prediction	280.1	29.8	30.2	29.1
Result output	40.2	1.8	1.9	1.8
Total time/ms	1675.9	31.6	32.1	30.9

, the sample detection results, which showcases consistent prediction accuracy across individual test cases, indicating robust readiness for real-world deployment.

Figure 12 comprehensively demonstrates the model’s robust generalization capability in accurately predicting diverse apple features across various orientations. The subfigures systematically present three key training metrics: (a) The bounding box loss curve quantifies localization accuracy, using Intersection over Union (IoU) to measure the discrepancy between the predicted and ground-truth bounding boxes. (b) The classification loss curve evaluates categorical prediction errors. (c) The confidence loss curve assesses the reliability of object detection predictions. The consistent and smooth convergence of all three loss functions clearly indicates stable learning progression and effective optimization of the object detection task. Figure 13 further validates the method’s practical effectiveness by presenting detailed identification results under various abnormal conditions, confirming the model’s operational reliability in real-world scenarios.

To rigorously validate the experimental outcomes, we conducted systematic performance comparisons by randomly sampling 100 test images from an open-source database. These images were processed sequentially using three YOLO variants (YOLOv8n, YOLOv10n, YOLOv12n) alongside the conventional HALCON image processing system [54,55,56,57,58,59,60]. For a comprehensive evaluation, we measured and compared the average execution times across all processing stages, with unavailable stages marked as ‘not applicable’ in Table 8. The comparative analysis clearly demonstrates the computational efficiency advantages of the advanced YOLOv12n architecture over its predecessors and traditional alternatives (Figure 14).

To further validate the effectiveness of the structural improvements, we conducted ablation studies using YOLOv12n as the baseline model. Each enhanced module was sequentially integrated, and its contribution to model performance was quantitatively assessed (

Table 9. Ablation test of the optimized model of YOLOv12n.

SPPF	C2PSA	Upsample	mAP50seg/%	mAP50 95seg/%
/	/	/	97.5	74.2
√	/	/	98.6	76.5
√	√	/	98.7	77.1
√	√	√	98.7	78.2

). In the table, / means that this module is not available and √ means that this module is available. The results demonstrate that while the improved YOLOv12n maintains comparable mAP50seg performance, it achieves significant gains in mAP50-95seg. The SPPF module utilizes cascaded pooling to expand receptive fields, enhancing robustness for large target detection. Compared to standard SPP, its serial design preserves finer edge details through smaller initial pooling kernels [61,62,63,64,65]. The C2PSA module integrates channel attention with position-sensitive mechanisms, substantially improving small target detection by refining feature representation. Integrating upsample operations within these modules increased feature map resolution, collectively contributing to a 4-percentage-point improvement in detection accuracy. This cumulative enhancement validates the synergistic efficacy of each structural modification.

In this study, the scene focuses on the fruits growing in the indoor greenhouse environment. Under the condition of constant light, the influence on fruits was ignored, but samples were collected and analyzed for branch changes and different sizes of fruits. During training, leaves with occlusion, incomplete features, and fruits of different sizes could be identified. Usually, due to the different sizes and central point coordinates of the fruits, occlusion error is found between the sample library and the training library. This method establishes a result-search-oriented matching method under the YOLO VX framework. The results showed that the abnormal rate of the length value of the identification and the width value of the identification was lower than 1%, and the difference rate of the calculated length and the calculated width was lower than 8%, achieving a good test method. However, for different periods and different fruit tree heights, there are great differences. In future studies, classification will be used to propose a new model of mature fruit tree height and a new training method. Admittedly, when blade occlusion is greater than 35%, the difference is also large. In the future, the camera image acquisition method will be improved to search for mature fruits in a larger area.

Admittedly, the testing process revealed several limitations. Primarily, (1) when the obstruction ratio of fruits by leaves or stems exceeds 30%, recognition performance degrades significantly, necessitating supplementary mechanical designs to clear obstructions such as leaves or stems from the end-effector structure. (2) Furthermore, variability in tree heights poses challenges, as fruits located at the upper canopy often exceed the robotic arm’s operational range. To address this, the stationary mobile chassis requires optimization into a liftable mechanical platform capable of vertical adjustment.

6. Conclusions

In this paper, an intelligent orchard positioning and grasping method based on the YOLO VX model and 3D vision is proposed. YOLO VX deep learning, 3D visual target recognition, and microcontrol unit design technology were adopted to establish a sample library of mature fruits, and the fruit to be identified was used as a training library, and the output results were matched with the sample library. At the same time, the 3D camera solved the position and size information of the target fruit and transmitted it to the flexible claw on the robotic arm to complete the picking. Tests show that the YOLO VX model of this method is more efficient than the traditional method and can be further applied for the intelligent picking of other orchard fruits with different locations and sizes.

In the future, we will pay more attention to the fruits of different kinds of fruit trees in the same orchard and the methods or models of picking different fruits on the same fruit tree. For example, when different parts of the fruit are blocked, we can continue to optimize the YOLO VX method and fit the complete fruit outline through fuzzy statistics and analysis, or when different types of fruits are planted in the same orchard indoors, we can improve the classification standard of sample set, explore a parallel-solving hybrid-matching model, and make this method more intelligent. For different scenarios, the problem of environmental interference can be solved by introducing an optimization module algorithm, and for the picking of different fruit crops, different flexible end fixtures can be used.