A Progressive Hybrid Automatic Switching Visual Servoing Method for Apple-Picking Robots

Abstract

Position-Based Visual Servoing (PBVS) and Image-Based Visual Servoing (IBVS) struggle to balance end effector pose accuracy and robustness in apple picking. They are also prone to target loss and control singularities. A progressive Hybrid Automatic Switching Visual Servoing (HAVS) method is proposed and applied to an apple-picking robotic system. HAVS integrates PBVS and IBVS to coordinate control of the manipulator end effector pose. A depth-based switching function is designed. When target depth is below an optimal threshold, the controller switches to PBVS for precise final positioning. This reduces target loss and control singularities. An adaptive proportional-derivative (PD) controller with fuzzy gain scheduling updates the control gains online to enhance responsiveness and stability. The hardware consists of a six-axis manipulator, a depth camera, and a mobile base. You Only Look Once version 5 (YOLOv5) performs apple detection and generates control commands. Indoors, success rate was 96%, which was 4 and 10 percentage points higher than PBVS only and IBVS only. Average picking time was 12.5 s, 0.3 s, and 1.1 s shorter. Outdoors, success rate was 87.5%, average time was 13.2 s, and damage rate was 4.2%. This method provides a reference implementation for visual servo control in agricultural picking robots.

1. Introduction

Apples are among the most widely consumed fruits worldwide and are eaten fresh or used in processed products such as juices and sauces [1]. Apple consumption has been associated with health benefits, including lower risks of cancer and cardiovascular disease [2]. As orchard cultivation expands globally, harvesting under adverse weather and rising labor costs have become major constraints [3,4]. Advances in artificial intelligence and automated control have accelerated the adoption of robotic apple harvesters as alternatives to manual picking [5,6].

Recent years have seen substantial progress in mechanized apple harvesting [7,8,9,10,11]. Vision systems provide key information for environmental perception and target localization in harvesting robots. Vision-based servo control remains an active research topic in robotics [12,13,14]. In complex operating environments, vision sensors capture rich external information and can improve robotic stability and precision [15]. Orchard environments are unstructured, and tree growth patterns and fruit locations are highly uncertain. During harvesting, the manipulator must grasp target fruits accurately and efficiently [16]. A key challenge is to locate fruits rapidly and precisely in these conditions. Visual servo control must then guide the manipulator to the target for reliable harvesting [17].

The concept of visual servoing was first proposed by Hill et al. [18]. Unlike conventional robot control, visual servoing forms a closed-loop system that uses visual features as feedback for real-time motion control. Hutchinson et al. [19] further formalized visual servoing and categorized it into Position-Based Visual Servoing (PBVS) and Image-Based Visual Servoing (IBVS).

PBVS converts image measurements into a three-dimensional target pose for control [20]. The controller extracts image features. Using camera-to-hand calibration parameters, it estimates the target-to-manipulator relative pose in real time. Control laws are then formulated in Cartesian space to drive the manipulator and reduce pose errors [21]. Mehta et al. [22] developed a vision-based estimation and control system for robotic fruit harvesting. The system employed rotational and hybrid translation controllers for manipulation. A stability analysis was conducted to assess closed-loop performance, indicating suitability for medium-to-large citrus harvesting. IBVS defines and regulates the control error directly in the two-dimensional image plane [23]. The controller selects image features, such as points or lines, and specifies their desired locations. It computes the pixel error between current and desired feature positions in real time. Based on this error, it generates camera velocity commands to minimize the image error [24]. Li et al. [25] proposed a novel uncalibrated visual servoing method that integrates a hybrid visual configuration with an adaptive tracking controller. Asymptotic convergence was established using Lyapunov stability analysis, and experiments validated the method.

These studies indicate that PBVS emphasizes accurate positioning and orientation control. It is therefore suitable for tasks that require high end-effector accuracy. However, PBVS is sensitive to camera calibration, hand-eye calibration, and target model errors [26]. During motion, the target can move out of the field of view, which can cause control failure. IBVS is generally more robust to calibration and model errors, and it tends to keep the target within the field of view [27]. However, IBVS can induce complex and hard-to-predict three-dimensional trajectories. Image singularities may also occur.

To address the limitations of IBVS and PBVS, Aabdul et al. [28] proposed a hybrid visual servoing method that uses a switching function to transition between the two schemes. When joint variables approach their limits, the controller switches to PBVS to avoid excessive distortion in the commanded camera motion. When feature points move close to the image boundary, the controller switches back to IBVS to maintain target tracking and stable control. Lei et al. [29] proposed a hybrid visual servoing method for a coal-mine tunnel bolting and drilling robot. The method achieved high-precision alignment and improved efficiency. However, the stability and reliability of the vision system still require further validation. Li et al. [30] proposed a hybrid visual servo control (HVSC) method for cherry-tomato harvesting robots. The method switches between PBVS for coarse positioning and IBVS for fine alignment, enabling smooth and accurate localization tasks. However, the system relies on prior assumptions about color and shape. This may reduce recognition robustness. In addition, it has not been validated in real harvesting environments.

To address these limitations, this study proposes a progressive Hybrid Automatic Switching Visual Servoing (HAVS) method tailored for unstructured agricultural environments. The main contributions are:

(1)

Development of a HAVS method that employs hybrid IBVS–PBVS control during the coarse alignment phase for rapid target approach and guaranteed target retention within the Field-of-View (FOV). In addition, when the depth of the target falls below the optimal threshold, PBVS is employed for fine alignment.

(2)

Proposal of an adaptive PD controller with fuzzy gain scheduling that updates control gains online to improve response speed and dynamic stability.

(3)

Construction of an apple-picking robot system with its overall performance verified through indoor simulated and field picking experiments.

2. Design of the Apple-Picking Robot System

2.1. Hardware and Software Design of the Apple-Picking Robot

Figure 1 shows the apple-picking robot system. It comprises a uFactory (Shenzhen, China) xArm6 six-degree-of-freedom (6-DoF) collaborative manipulator, an Intel (Santa Clara, CA, USA) RealSense D435i depth camera, an electrically actuated three-finger flexible gripper, a control terminal, and a mobile platform. The xArm6 comprises a six-joint serial structure with independently actuated revolute joints. It provides 6-DoF motion. This configuration enables the end-effector to reach the desired position and orientation within its reachable workspace [31]. The end-effector uses an electrically actuated three-finger flexible gripper. The gripper supports compliant grasping and stall protection. This helps reduce apple damage during picking. The three-finger design accommodates apples with diameters of 50–150 mm.

In robotic vision systems, depth-camera mounting is typically categorized as eye-in-hand or eye-to-hand. In an eye-to-hand setup, the camera is fixed on the mobile base and does not move with the manipulator. This setup provides a wider field of view and stronger global perception. However, target localization accuracy is lower [32]. In an eye-in-hand setup, the camera is mounted on the end-effector and moves with the manipulator. The camera can approach the target, which reduces localization error. Apple picking requires high target localization accuracy. Therefore, the eye-in-hand configuration was adopted.

The software stack is built on Ubuntu 22.04 and the Robot Operating System (ROS). C++ programs are developed in Microsoft (Redmond, WA, USA) Visual Studio 2022. LibTorch 1.13.0, the PyTorch C++ application programming interface (API), is used for graphics processing unit (GPU)-accelerated inference for apple detection and localization. Image processing uses the depth-camera software development kit (SDK) and the Open-Source Computer Vision Library (OpenCV) 4.8. Motion planning and visual servo control are implemented with the xArm SDK. ROS supports mobile platform navigation and locomotion control. It also enables inter-module communication.

2.2. Operational Workflow of the Apple-Picking Robot

Figure 2 illustrates the overall workflow of the apple-picking robot. The workflow includes three phases: target detection, vision-based servo control, and the picking action. First, the red–green–blue and depth (RGB-D) camera captures synchronized color and depth images of the target. After preprocessing, a target-detection network identifies the target fruit. The target center is then computed in both the base frame (3D) and the image plane (2D). An adaptive PD controller with fuzzy gain scheduling is used. PBVS drives the manipulator towards the target apple. IBVS regulates the end-effector orientation to keep the target on the camera optical axis. When the target depth falls below a preset threshold, the controller switches to PBVS for final fine positioning. After the desired pose is reached, the gripper closes. The manipulator then retracts and releases the fruit to complete the picking operation.

3. HAVS Control Method

The proposed progressive Hybrid Automatic Switching Visual Servoing (HAVS) method adopts an eye-in-hand vision configuration. It enables real-time detection and three-dimensional localization of target apples. It also supports visual servoing of the manipulator. The software uses multi-threaded architecture. The vision, control, and display threads run concurrently. The vision thread performs apple detection and localization. The control thread runs the hybrid visual servoing algorithm and issues motion commands. The display thread supports visualization and human–machine interaction. This design improves system stability and meets real-time constraints.

3.1. Object Detection

Object detection networks are commonly divided into one-stage methods, such as the You Only Look Once (YOLO) family, and two-stage methods, such as Faster Region-based Convolutional Neural Networks (Faster R-CNN) [33,34]. Apple orchard environments are highly unstructured, characterized by dense fruit and severe occlusion. Though anchor-free detectors (e.g., YOLOv8) perform well on general datasets, YOLOv5’s anchor-based approach offers a more robust geometric prior. It generates detection anchors via K-means clustering, enabling it to more effectively separate adjacent fruits in dense clusters. In addition, YOLOv5 offers superior deployment suitability and well-established algorithmic maturity. Therefore, this study uses YOLOv5 for target detection.

YOLOv5 has fewer parameters and lower computational cost than Faster R-CNN, Focal Loss for Dense Object Detection (RetinaNet), and Single Shot MultiBox Detector (SSD). It therefore achieves faster inference. This supports real-time detection on resource-constrained platforms while maintaining high accuracy [35,36,37]. Therefore, the network provides timely localization information for the picking task. YOLOv5 provides four model scales: s, m, l, and x. They share the same architecture but differ mainly in depth and feature-map width [38]. YOLOv5s is the smallest variant and uses a shallower network with narrower feature maps. Larger variants increase depth and width to improve detection performance. YOLOv5s is usually less accurate than larger variants, but it provides faster inference. Given the real-time requirement, YOLOv5s is selected for target detection.

The MinneApple dataset [39,40,41] was used for training and evaluation. It is designed for apple detection and segmentation in orchard scenes and is generally more challenging than the indoor and fielded picking setup considered in this study. It contains 1000 high-resolution orchard images covering diverse lighting, occlusion, and background conditions. It includes polygon mask annotations for more than 41,000 apple instances. This supports both detection and segmentation tasks. The dataset was randomly split into training (80%), validation (10%), and test (10%) sets. This split supports model training, hyperparameter tuning, and generalization evaluation. Performance was evaluated using mean average precision at an intersection-over-union (IoU) threshold of 0.5 (mAP@0.5).

Transfer learning was employed by initializing the detection network using pre-trained weights obtained from the COCO dataset. We set the image size to 640 × 640 pixels, with 300 training epochs, a learning rate of 0.01, momentum of 0.937, weight decay of 0.0005, and adopt the Stochastic Gradient Descent (SGD) optimizer. We adopted standard YOLOv5 data augmentation techniques, including Mosaic augmentation and random HSV color jittering. Model training was conducted on a laptop with Ubuntu 22.04, a 13th Gen Intel (R) Core (TM) i7-13620H (2.40 GHz) CPU and an NVIDIA GeForce RTX 4060 Laptop GPU. Typical detection results are shown in Figure 3.

3.2. Basic Modules and Control-Law Design

3.2.1. IBVS Control Module

IBVS defines the control error on a two-dimensional image plane and forms a closed-loop system. IBVS has low computational cost and is relatively insensitive to camera calibration errors. This study uses the detected apple center as the image feature. The image center is set as the desired feature location. IBVS aims to align the apple center with the image center.

With an image resolution of 640 × 480, the image center is located at pixel (320, 240). The IBVS error is defined as the pixel offset between the apple center and the image center.

$${\mathit{e}}_{IBVS}=[u-u_c,v-v_c{]}^{\mathrm{T}}$$

(1)

A desired depth of d₀ (150 mm) is specified to indicate that the apple is close to the gripper. The depth error is defined as the difference between the desired and measured depths. This constraint limits distance variation along the line-of-sight direction.

$$e_d=d-d_0$$

(2)

Proportional control maps the pixel and depth errors to correction commands for the end-effector orientation. After empirical tuning, the proportional gains K_pα and K_pβ are set to −0.1 and 0.1, respectively.

$$\mathsf{\Delta }\alpha =K_{p\alpha }(u-u_c),\mathsf{\Delta }\beta =K_{p\beta }(v-v_c)$$

(3)

Here, Δα and Δβ denote the control increments for the end-effector roll and pitch angles. For the IBVS baseline, control commands computed from depth error were used to drive the manipulator toward the target.

$$\mathsf{\Delta }z=K_{p2}{e}_d$$

(4)

3.2.2. PBVS Control Module

PBVS maps image observations and depth measurements to a target position in three-dimensional space. Control is performed in this spatial domain. The objective is to drive end-effector motion and minimize the position error between the gripper working point and the target apple center. The error is reduced to near zero. This requires coordinating transformations among the camera, the manipulator, and the target. The coordinate frames are defined in Figure 4.

The Intel RealSense D435i captures RGB images and depth maps. YOLOv5s estimates the target apple center pixel coordinates (u, v) from the RGB image. The detection results are aligned with the depth map. The depth value at (u, v) is then read. A moving-average filter smooths the depth value to suppress fluctuations from illumination changes and sensor noise. The depth map measures the distance from the apple surface to the camera. It does not directly provide the depth of the apple’s geometric center. The apple is approximated as a sphere. Its equivalent image radius r is estimated from the YOLOv5s bounding box. The center depth is corrected by adding r to the measured depth d.

The depth-camera intrinsics include focal lengths (f_x, f_y), and principal point coordinates (c_x, c_y). Using the pinhole camera model, (u, v) and the corrected depth are back-projected to obtain the three-dimensional target center in the camera frame.

$$X_c={\displaystyle \frac{(u-c_x)\cdot d}{f_x}},Y_c={\displaystyle \frac{(v-c_y)\cdot d}{f_y}},Z_c=d+r$$

(5)

The target-center position in the camera frame is denoted by p_cam, written in homogeneous form as [Xc, Yc, Zc, 1]^T. A hand-eye calibration transform maps this point from the camera frame to the end-effector frame.

$${\mathbf{T}}_{\mathit{cam}\to \mathit{end}}=\left[\begin{array}{cccc}-0.0080& -0.9990& 0.0428& 85.0887\\ 0.9998& -0.0074& 0.0149& -30.0342\\ -0.0146& 0.0430& 0.9989& 5.0004\\ 0.0& 0.0& 0.0& 1.0\end{array}\right]$$

(6)

$${\mathit{p}}_{end}={\mathbf{T}}_{\mathit{cam}\to \mathit{end}}\cdot {\mathit{p}}_{cam}$$

(7)

This transform is denoted as T_cam→end, which is a homogeneous transformation matrix obtained from hand-eye calibration. For control, the point is further expressed in the base frame. The xArm SDK provides forward kinematics for the xArm6. It outputs the end-effector position x_end and orientation (α, β, γ) in the base frame. These kinematics define the base-to-end-effector transformation. The target point is then mapped into the base frame.

$${\mathbf{R}}_{end\to base}=\left[\begin{array}{ccc}\cos \gamma \cos \beta & \cos \gamma \sin \beta \sin \alpha -\sin \gamma \cos \alpha & \cos \gamma \sin \beta \cos \alpha +\sin \gamma \sin \alpha \\ \sin \gamma \cos \beta & \sin \gamma \sin \beta \sin \alpha +\cos \gamma \cos \alpha & \sin \gamma \sin \beta \cos \alpha -\cos \gamma \sin \alpha \\ -\sin \beta & \cos \beta \sin \alpha & \cos \beta \cos \alpha \end{array}\right]$$

(8)

$${\mathbf{T}}_{end\to base}=\left[\begin{array}{cc}{\mathbf{R}}_{end\to base}& {\mathit{x}}_{end}\\ 0& 1\end{array}\right]$$

(9)

$${\mathit{p}}_{base}={\mathbf{T}}_{end\to base}\cdot {\mathit{p}}_{end}$$

(10)

The mapped point in the base frame is denoted as p_base. Its first three components give the target position p_target. A fixed offset l_offset of 185 mm is measured between the gripper working point and the end-effector frame along the Z-axis. This offset is used to compute the gripper working-point position in the base frame.

$${\mathit{x}}_{gripper}={\mathit{x}}_{end}+{\mathbf{R}}_{end\to base}\cdot {[\mathrm{0,0},l_{\mathrm{offset}}]}^{\mathrm{T}}$$

(11)

The PBVS position error is the difference between the target position and the gripper working-point position. It is used as the input to end-effector position control.

$${\mathit{e}}_{PBVS}={\mathit{x}}_{target}-{\mathit{x}}_{gripper}$$

(12)

3.2.3. Adaptive PD Control Module with Fuzzy Gain Scheduling

A PD controller is used to regulate the manipulator’s end-effector position and balance response speed and stability. The error derivative is approximated using a discrete-time difference. The control period T is about 50 ms.

$${\dot{\mathit{e}}}_{PBVS}(t)={\displaystyle \frac{{\mathit{e}}_{PBVS}\left(t\right)-{\mathit{e}}_{PBVS}(t-T)}{T}}$$

(13)

The PD update computes the end-effector position increment at each control cycle.

$$\mathsf{\Delta }\mathit{x}\left(t\right)=K_{p1}{\mathit{e}}_{PBVS}\left(t\right)+K_d{\dot{\mathit{e}}}_{PBVS}\left(t\right)$$

(14)

In unstructured orchard environments, fixed-gain PD control may not achieve both fast convergence and stable behavior over the full operating range. To address these trade-offs, a Mamdani fuzzy inference-based gain scheduler is designed [42,43,44]. The scheduler updates K_p1 and K_d online based on the current error and its rate of change. Compared with advanced control strategies such as model reference adaptive control (MRAC) and active disturbance rejection control (ADRC), the fuzzy PD controller exhibits unique advantages in unstructured orchard environments. It relies solely on error feedback and does not require an accurate system model, thereby mitigating complications caused by model mismatch. Furthermore, its low computational complexity facilitates future deployment of the algorithm on embedded systems.

The scheduler uses a dual-input dual-output structure. It aims to transition smoothly from proportional (P) control to a higher-damping PD regime. The control flow is shown in Figure 5. Two inputs are used. The position error (PE) is the Euclidean distance between the end-effector and the target point. The error change rate (EC) describes the rate of change of this distance. For PE, the fuzzy subsets are {ZO, PS, PM, PB}. For EC, the fuzzy subsets are {NB, NM, NS, ZO, PS, PM, PB}. In these labels, P and N denote positive and negative, ZO denotes zero, and S, M, and B denote small, medium, and big.

The design of fuzzy rules strictly adheres to the physical principles underlying error dynamics. For large PE, the proportional gain K_p1 is increased to accelerate convergence; for small PE, K_p1 is reduced to prevent overshoot. Similarly, the derivative gain K_d is dynamically adjusted according to the rate of EC to provide effective damping. Based on the fundamental principles, the membership functions are further optimized via experimental tuning, as shown in Figure 6. The fuzzy rules are listed in

Table 1. Fuzzy control rules for K_p1 and K_d.

	EC	NB	NM	NS	ZO	PS	PM	PB
PE
PB		B, S	B, ZO	B, ZO	B, ZO	B, ZO	B, ZO	B, ZO
PM		M, M	M, S	B, ZO	B, ZO	B, ZO	B, S	B, M
PS		S, B	S, M	M, S	M, S	B, S	B, S	B, M
ZO		ZO, B	ZO, B	S, M	S, M	M, M	B, M	B, M

.

Fuzzy inference produces a fuzzy output set. Defuzzification is required to obtain crisp control gains [45]. Weighted averaging is used for defuzzification to compute K_p1 and K_d.

$$K_{out}={\displaystyle \frac{{\sum }_{i=1}^n{\mu }_i\cdot y_i}{{\sum }_{i=1}^n{\mu }_i}}$$

(15)

Here, μ_i denotes the firing strength of the rule i. The corresponding single-point output is denoted by y_i. Its quantization parameters are determined experimentally, as shown in

Table 2. Quantization values of the output variables.

Fuzzy Linguistic Level	Quantized Value of Kp1	Quantized Value of Kd	Physical Interpretation
B	0.65	0.08	Aggressive correction with strong damping
M	0.35	0.04	Standard correction with medium damping
S	0.12	0.01	Fine adjustment with weak damping
ZO	0.00	0.00	Stop command with no damping

. Finally, the online-updated gains are applied in the PD update to compute the end-effector position increment Δx(t) at each control cycle. This method improves adaptability and operational stability in complex orchard environments while retaining the simplicity of the PD structure.

3.3. HAVS Switching Method and Coordinated Control

PBVS defines the error in three-dimensional space and drives the end-effector towards the target through coordinate transformations. PBVS often provides good global convergence and avoids image-plane singularities. It also controls the end-effector position directly in three-dimensional space, which supports high positioning accuracy. However, PBVS requires higher computation and is sensitive to calibration and depth errors. When the camera is far from the target, large motion can move the target out of the field of view. This can degrade subsequent control. In contrast, IBVS defines the error directly in the image plane and adjusts the manipulator to align target features with their desired locations. IBVS typically has lower computational cost and faster response. It is also more robust to calibration and depth errors. At close range, IBVS can suffer from image singularities and oscillations. It also provides limited direct constraints on global motion in three-dimensional space. Therefore, a progressive HAVS fusion method is proposed. It combines PBVS and IBVS in a complementary manner to balance field-of-view maintenance and end-effector positioning accuracy.

During long-range motion, HAVS runs PBVS and IBVS in parallel to control the manipulator. Target detection and coordinate transformation provide a three-dimensional position error and a two-dimensional image-plane error. PBVS uses the three-dimensional position error and an adaptive PD controller with fuzzy gain scheduling to drive the end-effector towards the target. In parallel, IBVS uses the image-plane error to regulate end-effector orientation. This keeps the target near the image center and reduces the risk of leaving the field of view. A geometric offset exists between the gripper working point and the camera optical axis. IBVS also provides limited direct constraints on three-dimensional positions. Therefore, when the target depth falls below a preset threshold, the controller switches to PBVS for final fine positioning. When the target position error drops below 2.5 mm, the gripper closes to complete the picking action. This threshold corresponds to about 5% of a typical apple dimension. It is consistent with the required picking accuracy. Figure 7 illustrates the laboratory simulated picking process. Under indoor conditions, the picking success rate reached 96%. Video S1 in the Supplementary Materials shows the indoor picking process of the robot.

4. Results and Discussion

After training, the YOLOv5s model achieved 78.1% mAP@0.5 on the MinneApple test set. Despite the dataset’s challenging lighting and occlusion conditions, deployment on the prototype achieved 89% detection accuracy in 100 indoor tests with randomly selected target positions conducted under a simpler and more controlled setting. Unlike static dataset evaluation, both detection confidence and success rate improve as the manipulator approaches the target. Regarding the latency, the object detection module is implemented using LibTorch (C++) with FP32 precision on an NVIDIA RTX 4060 Laptop GPU. The target detection module achieves an inference speed exceeding 60 FPS (10–15 ms), ensuring the real-time performance and stability of visual detection.

To evaluate the performance of the HAVS-based apple-picking system, a series of comparative experiments were conducted in an indoor simulated picking environment. Field picking trials were also conducted at X Farm in Beijing, China, to assess performance in a real orchard.

4.1. Indoor Simulated Picking Experiments

4.1.1. Determination of Switching Thresholds

Seven candidate depth thresholds for switching to PBVS were set from 0.20 m to 0.50 m in 0.05 m increments. In the indoor simulated picking environment, 50 different target positions were randomly selected, with varied lighting and occlusion conditions across these positions. Seven depth thresholds were used for picking experiments at each target position, and the system performance was evaluated by average picking time and picking success rate. The picking time is defined as the duration from the initialization of the robotic system to the moment the manipulator releases the fruit into the collection basket, and the average value is calculated across 50 independent experiments. In each trial, a picking attempt is deemed successful only if the target is successfully detected, clamped and released into the collection basket.

The results are shown in Figure 8. Error bars indicate the standard deviation across 50 trials. Switching to PBVS when the target depth fell below 0.40 m yielded the best overall performance. If the threshold is too high, the larger long-range displacement increases the chance that the target leaves the camera field of view. This can interrupt control continuity. If the threshold is too low, the switch occurs at close range, and the system becomes more prone to oscillations. This increases the trajectory length and reduces picking efficiency.

4.1.2. Comparative Experiments

After introducing fuzzy gain scheduling, 50 randomized HAVS picking trials were conducted in the indoor environment. Compared with fixed-gain PD control, the average picking time decreased by 1.3 s, and the success rate increased by 4 percentage points.

HAVS was further compared with PBVS-only and IBVS-only baselines to evaluate performance. A total of 100 target positions with varied lighting and occlusion conditions were randomly selected in the indoor environment. At each position, picking tests were conducted separately using three control methods. All three methods were tested under identical experimental conditions. The fuzzy rules and fixed-gain settings followed the specifications described above and are employed in the baseline methods. For a representative target position, time-varying position-error curves were recorded for each method, as shown in Figure 9. From target detection to reaching the desired pose, HAVS converged in about 5.8 s. This was 0.7 s and 1.7 s faster than PBVS and IBVS, respectively.

Figure 10 plots the end-effector trajectories of the three methods when picking the same target position. The target apple position in the base frame is (488.0, 170.4, 656.0). The end-effector starts at (249.5, 0.0, 633.7). In three-dimensional space, the HAVS trajectory is smoother, with no noticeable jitter or redundant backtracking. The IBVS trajectory shows local oscillations. The PBVS trajectory contains abrupt direction changes. Trajectory-length analysis shows that HAVS produces the shortest path among the three methods. The HAVS path length is 298.9 mm. This is 10.0 mm shorter than PBVS and 12.2 mm shorter than IBVS. Overall, HAVS combines PBVS and IBVS with automatic switching. This yields smoother end-effector trajectories and shorter paths, improving motion efficiency.

Results from the 100 trials are summarized in Figure 11. The lower success rate of IBVS is primarily attributed to insufficient 3D positional constraints during the final alignment phase. Furthermore, IBVS frequently failed due to Jacobian singularities at close range, which resulted in unavoidable trajectory jitter. PBVS primarily failed due to FOV loss. As the controller imposes no constraints in the image plane, extensive manipulator movements can cause the target to exit the FOV, which ultimately leads to control instability and failure.

In the indoor simulated picking task, HAVS achieved a 96% success rate. This is 4 and 10 percentage points higher than PBVS and IBVS, respectively. The average picking time of HAVS was 12.5 s. It was 0.3 s and 1.1 s shorter than PBVS and IBVS, respectively. The picking-time distribution shows that HAVS has a standard deviation of 1.1 s, which is lower than 1.4 s for IBVS and 1.2 s for PBVS. This suggests that HAVS suppresses visual-deviation-induced corrections more quickly across scenarios, which reduces ineffective manipulator motion time. As a result, picking-time fluctuations decrease and system robustness improves.

4.2. Field Picking Experiments

The field experimental setup is shown in Figure 12. Apple trees were planted in rows. The access path was about 2.5 m wide, and the trees were about 3 m tall. Most apples were within the manipulator workspace and were treated as picking targets. Four experimental groups were conducted on the same day at different times. Each group included 30 picking attempts, for a total of 120 trials. Four groups of experimental apple trees were randomly selected. The first group was tested from 9:00 to 10:00 under intense light; the second group from 10:00 to 11:00 under weak light with partial cloud cover; the third group from 12:00 to 13:00 under direct sunlight with the strongest light intensity; and the fourth group from 15:00 to 16:00 under the weakest light intensity, with thick clouds completely blocking sunlight. Strong sunlight slightly reduced detection accuracy and decreased the picking success rate, but both remained within an acceptable range. The field picking results are summarized in

Table 3. Field results of the apple-picking robotic system.

Experiment Group	Picking Attempts	Average Picking Time (s)	Successful Picks	Success Rate (%)	Damaged Apples	Damage Rate (%)
1	30	12.5	26	86.7	1	3.3
2	30	13.2	27	90.0	1	3.3
3	30	14.1	24	80.0	2	3.3
4	30	13.0	28	93.3	1	6.7
Total	120	13.2	105	87.5	5	4.2

. In the orchard field trials, 105 of 120 attempts were successful, corresponding to a success rate of 87.5%. The average picking time was 13.2 s. Five apples were damaged, corresponding to a damage rate of 4.2%. Video S2 in the Supplementary Materials shows the outdoor orchard picking process of the robot.

4.3. Discussion of Experimental Results

Among the 120 field picking trials, 15 failures were observed. As shown in Figure 13, the failures fall into two categories. In the first failure mode, the gripper inadvertently grasped nearby branches while closing. This prevented a secure grasp, and the fruit slipped from the gripper during manipulator retraction. Branch detection could be added to the vision pipeline to guide the approach and avoid branches. In the second failure mode, foliage occluded the camera field of view during motion, causing target detection to stop. Active obstacle avoidance could mitigate this issue. The manipulator trajectory can be adjusted using obstacle point clouds to avoid dense foliage.

Table 4. Comparison of picking performance with representative picking robots.

Picking Robots	Fruit Type	Average Picking Time (s)	Success Rate (%)	Damage Rate (%)
This work	Apple	13.20	87.50	4.2
Xu et al. [11]	Apple	25.00	95.00	/
Wang et al. [46]	Apple	14.69	70.77	4.55
Shi et al. [47]	Apple	/	84.70	0.88
Chen et al. [48]	Dragon fruit	15.19	76.90	/
Park et al. [49]	Cucumber	56.00	56.60	4.7
Choi et al. [50]	Citrus	21.33	83.33	/

compares the proposed system with representative picking robots reported in the literature. Unreported metrics are denoted by “/”. Table 4 presents a performance comparison between the proposed system and other visual servoing systems. It is important to note that direct quantitative comparison is limited by different experimental conditions, including hardware setups, fruit categories and test environments. Nevertheless, these results demonstrate that the proposed method achieves comparable performance on standard computing hardware, which validates the feasibility of its deployment on mobile robotic platforms.

Existing fruit harvesting robots demonstrate substantial differences in system architecture and visual sensor selection. In terms of overall design, some studies prioritize end-effector development: for example, Wang et al. [46] proposed a soft gripper and Shi et al. [47] developed a composite pneumatic manipulator. In contrast, Chen et al. [48], Park et al. [49], and Choi et al. [50] implemented more sophisticated visual schemes. Specifically, Chen et al. [48] employed binocular stereo vision cameras supporting both global and local localization, rendering them suitable for continuous operation in large-scale orchards.

For apple-picking tasks, the average picking time was 13.20 s. This was shorter than the 25.00 s reported by Xu et al. [11] and the 14.69 s reported by Wang et al. [46]. The picking success rate was 87.50%. It was higher than the 70.77% reported by Wang et al. [46] and the 84.70% reported by Shi et al. [47] but lower than the 95.00% reported by Xu et al. [11]. Xu et al. [11] evaluated their system on indoor potted apple trees under relatively controlled conditions. Their results were therefore not directly comparable with orchard field trials. The fruit damage rate was 4.2%. This was like the 4.55% reported by Wang et al. [46] but higher than the 0.88% reported by Shi et al. [47]. Shi et al. [47] used a flexible end-effector that integrates suction adhesion with finger gripping, which can reduce fruit damage.

Table 4 also includes results from Chen et al. [48] for dragon fruit, Park et al. [49] for cucumber, and Choi et al. [50] for citrus. Because the target fruits and experimental conditions differ, cross-fruit comparisons are provided for reference only. For reference, the average picking time in this study was 13.20 s. This was shorter than 15.19 s (Chen et al. [48]) and 21.33 s (Choi et al. [50]), and much shorter than 56.00 s (Park et al. [49]). The picking success rate was 87.50%. This was higher than 76.90% (Chen et al. [48]), 83.33% (Choi et al. [50]), and 56.60% (Park et al. [49]).

Considering average picking time, success rate, and damage rate together, the proposed system shows competitive efficiency for apple-picking tasks and suggests potential for further engineering deployment.

Although field trials have validated the system for picking, further optimization is possible. First, outdoor robustness can be improved by refining the YOLOv5 detection method. Multimodal fusion can also be explored to improve accuracy under strong illumination and occlusion. In addition, branch detection and active obstacle-avoidance planning will be introduced to improve the robustness of visual servo control in complex orchard environments. Second, the current hardware limits the workspace and chassis maneuverability. Potential upgrades include improving mobile-based travers ability and adding an elevatable manipulator base to expand the workspace. In addition, adjustable compliant end-effectors for different fruit sizes and shapes could improve adaptability and reduce damage risk.

5. Conclusions

This study proposes a progressive Hybrid Automatic Switching Visual Servoing (HAVS) method and develops an apple-picking robotic system. Through the integration of PBVS and IBVS in the coarse alignment phase, together with a switch to PBVS-only control for fine alignment, the critical problems of target loss in PBVS and control singularities in IBVS are effectively overcome. Additionally, the adoption of a PD controller with fuzzy gain scheduling improves the system’s control efficiency and overall operational stability. Compared with fixed-gain PD control, the average picking time decreased by 1.3 s, and the success rate increased by 4 percentage points.

In indoor simulations, HAVS achieved a 96% success rate. This was 4 and 10 percentage points higher than PBVS and IBVS, respectively. The average picking time was 12.5 s, which was 0.3 s and 1.1 s shorter than PBVS and IBVS, respectively. In orchard field trials, four sessions were conducted with 120 picking attempts. The overall success rate was 87.5%, the average picking time was 13.2 s, and the apple damage rate was 4.2%. These results provide a reference implementation for visual servo control in agricultural picking robots.

Future work will focus on enhancing apple recognition performance under strong light and occlusion conditions through multimodal fusion as well as integrating active obstacle avoidance planning to further improve the robustness of the robotic system in complex unstructured orchard environments.