Optimized Design and Deep Vision-Based Operation Control of a Multi-Functional Robotic Gripper for an Automatic Loading System

Abstract

This study presents an optimized design and vision-guided control strategy for a multi-functional robotic gripper integrated into an automatic loading system for warehouse environments. The system adopts a modular architecture, including standardized platforms, transport containers, four collaborative 6-DOF robotic arms, and a multi-sensor vision module. Methodologically, we first developed three gripper prototypes, selecting the optimal design (30° angle between the gripper and container side) through workspace and interference analysis. A deep vision-based recognition system, enhanced by an improved YOLOv5 algorithm and multi-feature fusion, was employed for real-time object detection and pose estimation. Kinematic modeling and seventh-order polynomial trajectory planning ensured smooth and precise robotic arm movements. Key results from simulations and experiments demonstrated a 95.72% success rate in twist lock operations, with a positioning accuracy of 1.2 mm. The system achieved a control cycle of 35 ms, ensuring efficiency compared with non-vision-based methods. Practical implications include enabling fully autonomous container handling in logistics, reducing labor costs, and enhancing operational safety. Limitations include dependency on fixed camera setups and sensitivity to extreme lighting conditions.

1. Introduction

With the rapid advancement of intelligent manufacturing and automated logistics, the demand for flexible, efficient, and unmanned material handling systems has become increasingly critical in industrial applications. Traditional loading and unloading processes in warehouses are often labor-intensive and error-prone, resulting in reduced operational efficiency and increased safety risks. In response to these challenges, robotic-based automatic loading systems have emerged as a promising solution to achieve high-precision, high-reliability, and intelligent logistics operations. This type of system contains multiple operational tasks, usually requiring the use of a robotic arm and a multi-functional robotic gripper to complete, and the gripper needs to be optimized and designed. The system generally adopts vision-based control, and due to the complexity of the environment, multiple feature fusion techniques need to be carried out to ensure stable and robust operation. For example, Kim et al. proposed a novel mechatronics system capable of automating a peg-in-hole assembly process and inspecting the quality of the assembly with vision [1].

In the multi-functional mechanical gripper field, Kang et al. designed and implemented a multi-function gripper for grasping general objects, aiming to enhance versatility and flexibility in industrial automation applications [2]. Gümbel and Dröder proposed a design for highly repeatable and multi-functional grippers intended for precision handling with articulated robots [3]. Cramer et al. developed a user-friendly toolkit to select and design multi-purpose grippers for modular robotic systems [4]. Ai and Chen proposed a multifunctional gripper design at the end of the robot for versatile object manipulation [5]. Kladovasilakis et al. developed a multi-functional bioinspired soft robotic actuator using additive manufacturing techniques [6].

In the optimized design field, Yildiz et al. introduced a robust robot gripper mechanism design using a new hybrid grasshopper optimization algorithm [7]. Nguyen et al. optimized compliant gripper mechanism design by employing a bi-algorithm combining fuzzy logic and ANFIS [8]. Pinskier et al. applied diversity-based topology optimization techniques to the design of soft robotic grippers [9]. Lee et al. presented a closed-structure compliant gripper with morphologically optimized multi-material fingertips for aerial grasping [10]. Sun et al. developed LARG, a lightweight robotic gripper with 3D topology-optimized adaptive fingers [11].

In the vision-based control field, Zablocki et al. reviewed the explainability of deep vision-based autonomous driving systems and identified future challenges [12]. Ghasemieh and Kashef explored explainable artificial intelligence for deep vision-based odometry systems [13]. Wang et al. proposed a real-time deep vision-based method for soft body 3D proprioception [14]. Choi et al. conducted an experimental evaluation of deep vision-based occupancy counting for ventilation control [15]. Wang et al. applied the YOLO-v5 algorithm for real-time recognition of apple stem/calyx in automatic fruit loading systems [16].

In the multi-feature fusion field, Zou et al. developed a new multi-feature fusion convolutional neural network for facial expression recognition [17]. Jiang and Yin proposed a facial expression recognition approach using convolutional block attention and multi-feature fusion [18]. Li et al. presented a multi-feature fusion method for recognizing gastrointestinal metaplasia in medical images [19]. Hu et al. designed a novel multi-feature fusion network with spatial partitioning and cross-attention for gesture recognition using armband signals [20]. Liu and Xu introduced AMFF, an attention-based multi-feature fusion method for human intention recognition [21].

Inspired by port automation and container handling techniques, this study proposes a modular, containerized, and vision-guided robotic system for automated loading and unloading in warehouse scenarios. The system integrates standardized container platforms, modular transport units, multi-degree-of-freedom robotic manipulators, and a deep vision sensing module to perform autonomous material transfer tasks. Central to the system is the design of a multi-functional mechanical gripper capable of handling twist locks and performing precise positioning adjustments under complex spatial constraints.

To enhance system intelligence and operational autonomy, the proposed solution incorporates a multi-sensor vision system utilizing RealSense depth cameras. This enables real-time object recognition, spatial localization, and grasp planning, providing robust perception support for manipulation tasks. A coordinated control strategy for multiple robotic arms further improves task efficiency and ensures collision-free operations.

The objectives of this work are threefold: (1) to develop a mechanical gripper design optimized for both grasping and fine manipulation of twist locks; (2) to establish a vision-guided control framework for real-time object detection and pose estimation; and (3) to validate the system’s performance in simulation and experimental settings. This paper presents the system architecture, mechanical design methodology, motion planning strategies, and experimental results, demonstrating the feasibility and effectiveness of the proposed approach in enabling intelligent and autonomous loading operations in industrial warehouse environments.

Compared with existing state-of-the-art methods, our approach introduces several practical innovations that contribute to its superior performance. First, the geometry-aware gripper design improves workspace coverage and reduces mechanical interference, resulting in more reliable grasping during complex operations. Second, by fusing multiple 88 complementary visual features, the perception module achieves enhanced robustness and precision under varying environmental conditions. Third, the lightweight deep learning model enables real-time inference on standard hardware, ensuring smooth system integration without specialized computation resources. Lastly, the use of high-order motion 92 planning techniques ensures smoother and more stable manipulation trajectories, which benefits both alignment accuracy and mechanical durability. These combined improvements allow our system to deliver consistently higher efficiency and reliability in real-world automated loading tasks.

2. Automatic Loading System

2.1. System Architecture and Composition

Inspired by the automated container loading and unloading processes used in ports, this project proposes an unmanned automated system suitable for warehouse environments, as illustrated in Figure 1. The simulation prototype is established by using ROS Noetic Ninjemys Gazebo 11.X in Ubuntu 20.04 Focal Fossa. The system adopts a containerized standard storage format similar to that of port logistics, enabling digital management of warehouse goods by recording the contents and locations of each container.

The system is composed of six main modules:

1.

Standardized Load-Carrying Platform Module: This can include mining flatbed trucks, cargo trucks, trains, and ships.

2.

Modular Transport Container Module: Designed in various forms to carry different materials. As shown on the left side of Figure 1a, each container is equipped with eight corner fittings—four at the bottom for securing the container, and four at the top for lifting and transport.

3.

Locking Module: Includes manually adjustable twist locks and fully automatic twist locks. The manual twist locks are mounted on the standardized platform and can raise the locking head to secure the transport container. When locking is not required, the lock head can be lowered to convert the platform into a standard flatbed. The fully automatic twist locks are used to interconnect stacked containers.

4.

Overhead Crane and Spreader Module: Responsible for loading and unloading the transport containers onto and from the standardized platform, as well as for stacking containers within the warehouse.

5.

Multi-Sensor Recognition and Localization Module: Comprising four cameras, an IMU (Inertial Measurement Unit), and UWB (Ultra Wide Band) positioning modules, LD150 from Haoru Tech (Dalian, China). The cameras detect the four bottom corners of the containers and the handles of the manual twist locks within the work zone, providing feedback for automated operation. The IMU and UWB modules estimate the container’s position and orientation outside the working zone.

6.

Multi-Robot Arm Collaborative Loading and Unloading Module: Installed in the working zone, consisting of four robotic arms and multifunctional grippers. This module facilitates the automated transfer of containers between the platform and the storage area.

The system comprises four six-degree-of-freedom (6-DOF) robotic arms, specifically BRTIRUS1510A from Borunte (Shanghai, China), symmetrically mounted on both sides of the workspace. Each robotic arm is equipped with a two-finger gripper, RM-GB-17-80-2-1 from Robustmotion (Guangdong, China), to facilitate grasping operations. Additionally, four RealSense D415 depth cameras from Intel (Santa Clara, CA, USA) are individually positioned on the base of arms to acquire environmental data, providing real-time feedback for task execution by the robotic arms. The single robotic arm with a gripper and a depth camera is shown in Figure 2.

2.2. Loading and Unloading Process

2.2.1. Loading Process

The procedure for loading a transport container onto the load-carrying platform is as follows:

When the overhead crane hoists the transport container above the platform from a designated position in the warehouse, the multi-camera system captures the poses of the four bottom corner fittings of the container relative to the robotic arms’ base coordinate frame. Using this information, the system calculates the precise poses of the fully automatic twist locks mounted at the bottom of the container.

Then, the multi-robot arm collaborative module removes the fully automatic twist locks based on the estimated poses. Afterward, the robotic arms correct the container’s posture and align it with the platform. The overhead crane then lowers the container onto the platform.

Next, the multi-camera system detects the pose of the manual twist lock handles located on the platform. The robotic arms subsequently lift the locking heads according to the handle poses to securely fix the container in place.

2.2.2. Unloading Process

The procedure for unloading a transport container from the load-carrying platform is as follows:

The multi-camera system first detects the pose of the manual twist lock handles on the platform. The multi-robot arm module then lowers the locking heads to release the container.

The overhead crane lifts the container to a certain height. The multi-camera system again captures the poses of the four bottom corner fittings of the container relative to the robotic arms’ base frame, enabling calculation of the poses of the fully automatic twist locks.

Subsequently, the multi-robot arm module attaches the fully automatic twist locks back onto the container according to the estimated poses. Finally, the overhead crane transfers the container from above the platform to a designated storage position in the warehouse for stacking.

3. Optimized Design

3.1. Tasks of the Multi-Robot Arm Collaborative Loading and Unloading Module

Based on the operational workflow described above, the multi-robot arm collaborative loading and unloading module is designed to perform four main tasks:

• Handling fully automatic twist locks;
• Locking and unlocking manual adjustable twist locks;
• Correcting the posture of the transport container;
• Positioning and securing the transport container.

Among these tasks, handling the fully automatic twist locks is the most technically challenging and demands the highest precision from the multi-camera recognition and localization system. Therefore, the subsequent analysis focuses primarily on the task of handling fully automatic twist locks.

3.2. Operation Process

The operation process of the multi-functional mechanical gripper contains locking and unlocking the fully twist lock procedure.

3.2.1. Locking Procedure

The locking procedure consists of the following steps:

1.

Initial Grasping: The robotic arm moves from its initial position to a designated position where the gripper closes to securely grasp the twist lock.

2.

Alignment: The robotic arm adjusts its posture and executes a translational motion to align the twist lock with the mounting port of the corner piece.

3.

Insertion: The robotic arm performs a vertical motion along the z-axis, raising the twist lock to a certain height until it enters the corner piece.

4.

Rotation: The robotic arm rotates the twist lock counterclockwise by ${60}^{°}$ about its own central axis, which is parallel to the z-axis (viewed from above).

5.

Release: The gripper opens to release the twist lock, allowing it to be securely installed within the corner piece.

6.

Return: The robotic arm returns to its initial position.

3.2.2. Unlocking Procedure

The unlocking procedure involves the following steps:

1.

Approach: The robotic arm moves from its initial position to a designated location with a specific posture to approach and grasp the twist lock.

2.

Grasping: The gripper closes to securely hold the twist lock.

3.

Rotation: The robotic arm rotates the twist lock clockwise by ${60}^{°}$ about its own central axis, which is parallel to the z-axis (viewed from above).

4.

Extraction: The robotic arm performs a downward motion along the z-axis, lowering the twist lock until it exits the corner piece.

5.

Placement: The robotic arm moves the twist lock to a designated placement location, where the gripper opens to release the twist lock.

6.

Return: The robotic arm returns to its initial position.

Loading and unloading a twist lock is the opposite process. The three flag states are shown in Figure 3.

3.3. Optimized Design of the Multi-Functional Robotic Gripper

To meet the requirements of the four core functions of the multi-robot arm module, an optimized design of a multi-functional mechanical gripper has been developed.

An embedded two-finger gripper is used to grip the fully automatic twist locks. A cylindrical structure fixed above the two fingers is employed to rotate and lift the handle of the manual retractable twist lock. A right-angle structure, formed by the two fingers and a rectangular block on the side, is used to position and secure the four vertical corners of the transport container. Universal wheels at the fingertip ends help align the container and prevent unintended movement or frictional dragging between the container and the fingers.

To accommodate different gripping angles required for the fully automatic twist locks, three types of embedded two-finger grippers have been designed.

3.3.1. Type I Gripper (90° Between Gripper and Container Side)

The first gripper design, in which the angle between the gripper and the short side of the container is 90°, is illustrated in Figure 4a. The gripping posture for locking the twist lock is shown in Figure 4b.

When gripping from the outside, one of the extreme reachable positions is shown in Figure 4c. Although no interference occurs with the container in either extreme posture, the manipulator reaches the limit of its workspace. If the distance between the manipulator and the twist lock is too large, the gripper may fail to reach the required position, resulting in failure to complete the gripping action.

When gripping from the inside, the extreme posture is shown in Figure 4d. In this case, interference occurs between the robot’s lower arm and the container.

3.3.2. Type II Gripper (0° Between Gripper and Container Side)

The second gripper design, where the gripper is parallel to the short side of the container (0° angle), is shown in Figure 5a, and its twist lock gripping posture is shown in Figure 5b.

Two extreme positions for gripping the twist lock using this design are shown in Figure 5c,d. While both positions avoid interference with the container, in one posture, the robotic arm partially extends beneath the container, which increases the risk of collision during motion. Moreover, if the container is relatively low, rotation of joint ${\theta }_5$ may cause the lower arm of the robot to interfere with the container.

3.3.3. Type III Gripper (30° Between Gripper and Container Side)

The third gripper design, ultimately selected for deployment, features a 30° angle between the gripper and the short side of the container. The model is shown in Figure 6a, and its gripping posture is shown in Figure 6b.

The two extreme gripping positions for this design are illustrated in Figure 6c,d. In both cases, there is no interference with the container, and the robotic arms do not extend beneath the container. Furthermore, both extreme positions are comfortably within the robot’s workspace limits. Based on these advantages, the third gripper design was adopted as the final solution.

During the optimization process, both the workspace of the robotic arms and potential interferences between the arms and the container were thoroughly considered. Based on these analyses, the third design of the embedded two-finger gripper was selected, as the first two designs encountered limitations due to workspace constraints and physical interference [2]. As shown in Figure 7, there exist interference or workspace limitations in the Type I and II grippers. While the Type III gripper has the same maximum angle space before and after the 60° rotation operation, note that there are four arms that carry out the operation at the four corners of the container.

As a result, the final optimized multi-functional mechanical gripper design fulfills all four functional requirements, as shown in Figure 8.

4. Sensing Based on Deep Vision

The perception system aims to identify the fully automatic twist lock and determine its pose. It utilizes a YOLOv5-based algorithm to detect the planar coordinates $(x,y)$, depth point cloud information to obtain z-axis data, and geometric relationships among multiple target features for fusion recognition. This robust recognition and localization approach ensures reliable operation even under varying lighting conditions or when the target is temporarily lost.

4.1. Improved YOLOv5 Object Detection Algorithm

The proposed approach employs an improved object detection algorithm based on the YOLOv5 architecture [22], characterized by optimized network structure and training strategies that significantly reduce computational resource consumption while maintaining high detection accuracy.

Specifically, the algorithm achieves enhanced efficiency through the following techniques:

• Lightweight Feature Extraction Network: Reduces model parameters, ensuring efficient processing.
• Dynamic Resolution Input Strategy: Adapts to different computational constraints across various scenarios.
• Channel Attention Mechanism: Improves the discriminative capability of feature representations.

Due to the fixed camera and workspace, the scene is also fixed and singular. The recognized features only have a small correlation between Corner and container types. Other features are not on the container. Thus, the dataset is mainly affected by time and lighting. To enhance robustness, a specialized dataset comprising 3200 high-resolution images was constructed under different time periods and lighting conditions. The number of images of the train, validation, and test set are 2200, 500, and 500, respectively. This dataset encompasses various illumination conditions, target scales, and complex backgrounds. Data augmentation and rigorous annotation procedures are applied to significantly improve the algorithm’s detection accuracy and generalization performance.

4.2. Multi-Feature Fusion Recognition

A high-precision recognition method for containers and flatbed trucks is proposed, utilizing six heterogeneous features to achieve robust localization and motion decision-making under complex conditions. The feature system includes the following:

• Corner piece (Corner);
• Flatbed truck edge (Edge);
• Flatbed truck twist lock horizontal component (Lift_lock_1);
• Flatbed truck twist lock vertical component (Lift_lock_2);
• Twist lock Orientation (F/B/L/R);
• Automatic twist lock (Auto_lock).

The Corner feature serves as the primary constraint, while other auxiliary features are hierarchically integrated to enable core functionalities such as automatic twist lock locking/unlocking and flatbed truck twist lock manipulation.

During loading and unloading of a fully automatic twist lock onto or from a Modular Transport Container Module operations, motion commands for the robotic arm are generated based on the combined three-dimensional coordinate calculations of Corner and the automatic twist lock orientation. The localization precision is further optimized through multi-source data fusion, involving the following steps:

• Original coordinates of Auto_lock, Lift_lock_1, Lift_lock_2, and Edge are spatially registered and transformed to match the Corner coordinate system.
• Kalman filtering is applied for temporal prediction and dynamic weighted fusion of observations, effectively suppressing sensor noise and motion blur.

For depth calculation, a modal-based strategy is introduced:

• For Corner features, depth information is derived from the centroid depth of the bounding box’s four corner pixel points.
• For other features, the minimum depth value within the bounding box is utilized, balancing geometric precision and robustness against occlusion.
• A depth thresholding mechanism is implemented to filter erroneous data.

The recognition results are shown in Figure 9. The system successfully identifies multiple features, with the primary target being the Corner. Since the container structure is pre-designed, other features maintain consistent geometric relationships with the Corner. When the Corner feature is compromised, its pose can be indirectly calculated through these related features.

5. Operation Control

5.1. Kinematics Model

5.1.1. Forward Kinematics

The forward kinematics of a six-degree-of-freedom (6-DOF) robotic arm determines the position and orientation of the end-effector given the joint angles. The general forward kinematics equation can be represented as:

$$\mathbf{T}=\prod _{i=1}^6{\mathbf{T}}_i={\mathbf{T}}_1{\mathbf{T}}_2{\mathbf{T}}_3{\mathbf{T}}_4{\mathbf{T}}_5{\mathbf{T}}_6$$

(1)

Each transformation matrix ${\mathbf{T}}_i$ is defined using the Denavit–Hartenberg (DH) convention:

$${\mathbf{T}}_i=\left[\begin{array}{cccc}cos{\theta }_i& -sin{\theta }_{i}cos{\alpha }_i& sin{\theta }_{i}sin{\alpha }_i& a_{i}cos{\theta }_i\\ sin{\theta }_i& cos{\theta }_{i}cos{\alpha }_i& -cos{\theta }_{i}sin{\alpha }_i& a_{i}sin{\theta }_i\\ 0& sin{\alpha }_i& cos{\alpha }_i& d_i\\ 0& 0& 0& 1\end{array}\right]$$

(2)

where:

• ${\theta }_i$ is the joint angle.
• $d_i$ is the link offset.
• $a_i$ is the link length.
• ${\alpha }_i$ is the twist angle.

The final transformation matrix $\mathbf{T}$ provides the position and orientation of the end-effector in the base frame. The transformation matrix $\mathbf{T}$ relates the position and orientation of the end-effector as follows:

$$\mathbf{T}=\left[\begin{array}{cc}\mathbf{R}& \mathbf{P}\\ 0& 1\end{array}\right]=\left[\begin{array}{cccc}{n}_x& o_x& a_x& p_x\\ n_y& o_y& a_y& p_y\\ n_z& o_z& a_z& p_z\\ 0& 0& 0& 1\end{array}\right]$$

(3)

$$\left\{\begin{array}{c}{n}_x=s_{234}{c}_1{s}_6-c_6\left(s_1{s}_5-c_{234}{c}_1{c}_5\right)\hfill \\ n_y=c_6\left(c_1{s}_5+c_{234}{c}_5{s}_1\right)+s_1{s}_6\hfill \\ n_z=c_{234}{s}_6-s_{234}{c}_5{c}_6\hfill \\ o_x=s_6\left(s_1{s}_5-c_{234}{c}_5{c}_1\right)+s_{234}{c}_1{c}_6\hfill \\ o_y=s_{234}{s}_1{c}_6-s_6\left(c_1{s}_5+c_{234}{s}_1{c}_5\right)\hfill \\ o_z=c_{234}{c}_6+s_{234}{c}_5{s}_6\hfill \\ a_x=c_5{s}_1+c_{234}{c}_1{s}_5\hfill \\ a_y=c_{234}{s}_1{s}_5-c_1{c}_5\hfill \\ a_z=-s_{234}{s}_5\hfill \\ p_x=d_6\left(c_5{s}_1+c_{234}{c}_1{s}_5\right)+a_2{c}_1{c}_2+a_4{c}_1{c}_{234}+a_3{c}_1{c}_{23}\hfill \\ p_y=d_6\left(c_{234}{s}_1{s}_5-c_1\right)+a_4{s}_1{c}_{234}+a_3{s}_1{c}_{23}+a_2{c}_2{s}_1\hfill \\ p_z=d_1-a_3{s}_{23}-a_2{s}_2-a_4{s}_{4-23}-d_6{s}_5{s}_{234}\hfill \end{array}\right.$$

(4)

where $s_i$, $c_i$, $s_{ij}$, $c_{ij}$, $s_{ijk}$ and $c_{ijk}$ donate $sin{\theta }_i$, $cos{\theta }_i$, $sin({\theta }_i+{\theta }_j)$, $cos({\theta }_i+{\theta }_j)$, $sin({\theta }_i+{\theta }_j+{\theta }_k)$ and $cos({\theta }_i+{\theta }_j+{\theta }_k)$, respectively. $\mathbf{P}$ is the position vector ${\left[\begin{array}{ccc}{p}_x& p_y& p_z\end{array}\right]}^T$, and $\mathbf{R}$ is the rotation matrix representing orientation (corresponding to $\alpha$, $\beta$, $\gamma$ using Euler angles or RPY angles or other representations). Using RPY angles, it yields

$$\left\{\begin{array}{c}\beta =atan2\left(-n_z,\sqrt{n_x^2+n_y^2}\right)\hfill \\ \alpha =atan2\left(n_y,n_x\right)\hfill \\ \gamma =atan2\left(o_z,a_z\right)\hfill \end{array}\right.$$

(5)

This can be summarized as follows:

$$X=FK\left(Q\right)$$

(6)

where $X={\left[\begin{array}{cccccc}{p}_x& p_y& p_z& \alpha & \beta & \gamma \end{array}\right]}^T$ is the end-effector pose, $Q={\left[\begin{array}{cccccc}{\theta }_1& {\theta }_2& {\theta }_3& {\theta }_4& {\theta }_5& {\theta }_6\end{array}\right]}^T$ is the vector of joint angles, and $FK$ is the forward kinematics function.

5.1.2. Inverse Kinematics

Inverse kinematics involves determining the joint angles ${\theta }_i$ given the desired end-effector pose $\mathbf{T}$. This process is generally more complicated than forward kinematics and often requires numerical methods.

The desired transformation matrix can be divided into a rotational part $\mathbf{R}$ and a translational part $\mathbf{P}$ as Equation (3). Solving the inverse kinematics involves breaking down the problem into position and orientation components:

• Position: Solving for ${\theta }_1$, ${\theta }_2$, and ${\theta }_3$ based on the position vector $\mathbf{P}$.
• Orientation: Solving for ${\theta }_4$, ${\theta }_5$, and ${\theta }_6$ based on the rotation matrix $\mathbf{R}$.

$$\left\{\begin{array}{c}{\theta }_1=arctan\pm {\displaystyle \frac{p_y-d_6{a}_y}{p_x-d_6{a}_x}}\hfill \\ {\theta }_5=arctan\pm {\displaystyle \frac{\sqrt{{\left(n_x{s}_1-n_y{c}_1\right)}^2+{\left(o_x{s}_1-o_y{c}_1\right)}^2}}{a_x{s}_1-a_y{c}_1}}\hfill \\ {\theta }_6=arctan{\displaystyle \frac{o_x{s}_1-o_y{c}_1}{-n_x{s}_1+n_y{c}_1}}\hfill \\ {\theta }_2=arctan\pm {\displaystyle \frac{A^2+B^2}{\sqrt{4a_2^2{p}^2-{\left(A^2+B^2\right)}^2}}}-arctan{\displaystyle \frac{A}{B}}\hfill \\ {\theta }_{234}={\theta }_2+{\theta }_3+{\theta }_4={\displaystyle \frac{-a_z}{a_x{c}_1+a_y{s}_1}}\hfill \\ {\theta }_{23}=arctan{\displaystyle \frac{d_1-p_z-d_6{s}_5{s}_{234}-a_4{s}_{234}-a_2{s}_2}{p_x{c}_1+p_y{s}_1-a_4{c}_{234}-d_6{s}_5{c}_{234}-a_2{c}_2}}\hfill \\ {\theta }_3={\theta }_{23}-{\theta }_2\hfill \\ {\theta }_4={\theta }_{234}-{\theta }_{23}\hfill \end{array}\right.$$

(7)

where $A=p_x{c}_1+p_y{s}_1-a_4{c}_{234}-d_6{s}_5{c}_{234},B=d_1-p_z-d_6{s}_5{s}_{234}-a_4{s}_{234}$. To avoid the robotic arm being in a singular position, it is necessary to ensure $sin{\theta }_5\ne 0$.

This process can be summarized as:

$$Q=IK\left(X\right)$$

(8)

where $IK$ is the inverse kinematics function.

Analytical solutions can be obtained for simple robotic arms, while numerical methods such as Jacobian Inverse or Jacobian Transpose methods are employed for complex configurations.

5.2. Jacobian Matrix

The Jacobian matrix $\mathbf{J}$ relates the joint velocities $\dot{Q}$ to the end-effector velocity $\dot{X}$:

$$\dot{X}=\mathbf{J}\dot{Q}$$

(9)

The Jacobian matrix can be derived by differentiating the forward kinematics equations with respect to the joint angles.

$$\left\{\begin{array}{c}{J}_{11}=d_6{c}_1{s}_4{s}_5-s_1\left(a_2+a_4{c}_{23}+d_4{s}_{23}+a_3{c}_2+\left(d_6{c}_{23}{s}_{45}\right)/2+d_6{s}_{23}{c}_5-\left(d_6{s}_{45}{c}_{23}\right)/2\right)\hfill \\ J_{12}=c_1\left(c_2\left(d_6\left(c_3{c}_5-c_4{s}_3{s}_5\right)+d_4{c}_3-a_4{s}_3\right)-s_2\left(a_3+d_6\left(c_5{s}_3+c_3{c}_4{s}_5\right)+a_4{c}_3+d_4{s}_3\right)\right)\hfill \\ J_{13}=-c_1\left(s_{23}\left(a_4+d_6{c}_4{s}_5\right)-c_{23}\left(d_4+d_6{c}_5\right)\right)\hfill \\ J_{14}=d_6{s}_5\left(c_4{s}_1-c_1{c}_2{c}_3{s}_4+c_1{s}_2{s}_3{s}_4\right)\hfill \\ J_{15}=d_6{c}_5{s}_1{s}_4-d_6{c}_1{c}_2{s}_3{s}_5-d_6{c}_1{c}_3{s}_2{s}_5-d_6{c}_1{c}_4{c}_5{s}_2{s}_3+d_6{c}_1{c}_2{c}_3{c}_4{c}_5\hfill \\ J_{16}=0\hfill \\ J_{21}=c_1\left(a_2+a_4{c}_{23}+d_4{s}_{23}+a_3{c}_2+\left(d_6{c}_{23}{s}_{45}\right)/2+d_6{s}_{23}{c}_5-\left(d_6\left(s_4{c}_5-s_5{c}_4\right)c_{23}\right)/2\right)+d_6{s}_1{s}_4{s}_5\hfill \\ J_{22}=s_1\left(c_2\left(d_6\left(c_3{c}_5-c_4{s}_3{s}_5\right)+d_4{c}_3-a_4{s}_3\right)-s_2\left(a_3+d_6\left(c_5{s}_3+c_3{c}_4{s}_5\right)+a_4{c}_3+d_4{s}_3\right)\right)\hfill \\ J_{23}=-s_1\left(s_{23}\left(a_4+d_6{c}_4{s}_5\right)-c_{23}\left(d_4+d_6{c}_5\right)\right)\hfill \\ J_{24}=-d_6{s}_5\left(c_1{c}_4+c_2{c}_3{s}_1{s}_4-s_1{s}_2{s}_3{s}_4\right)\hfill \\ J_{25}=d_6{c}_2{c}_3{c}_4{c}_5{s}_1-d_6{c}_2{s}_1{s}_3{s}_5-d_6{c}_3{s}_1{s}_2{s}_5-d_6{c}_1{c}_5{s}_4-d_6{c}_4{c}_5{s}_1{s}_2{s}_3\hfill \\ J_{26}=0\hfill \\ J_{31}=0\hfill \\ J_{32}=a_3{c}_2+a_4{c}_2{c}_3+d_4{c}_2{s}_3+d_4{c}_3{s}_2-a_4{s}_2{s}_3+d_6{c}_2{c}_5{s}_3+d_6{c}_3{c}_5{s}_2+d_6{c}_2{c}_3{c}_4{s}_5-d_6{c}_4{s}_2{s}_3{s}_5\hfill \\ J_{33}=a_4{c}_{23}+d_4{s}_{23}+d_6{s}_{23}{c}_5+d_6{c}_{23}{c}_4{s}_5\hfill \\ J_{34}=-d_6{s}_{23}{s}_4{s}_5\hfill \\ J_{35}=d_6\left(c_2{c}_3{s}_5-s_2{s}_3{s}_5+c_2{c}_4{c}_5{s}_3+c_3{c}_4{c}_5{s}_2\right)\hfill \\ J_{36}=0\hfill \\ J_{41}=0\hfill \\ J_{42}=s_1\hfill \\ J_{43}=s_1\hfill \\ J_{44}=s_{23}{c}_1\hfill \\ J_{45}=c_4{s}_1+s_4\left(c_1{s}_2{s}_3-c_1{c}_2{c}_3\right)\hfill \\ J_{46}=s_5\left(s_1{s}_4-c_4\left(c_1{s}_2{s}_3-c_1{c}_2{c}_3\right)\right)+c_5\left(c_1{c}_2{s}_3+c_1{c}_3{s}_2\right)\hfill \\ J_{51}=0\hfill \\ J_{52}=-c_1\hfill \\ J_{53}=-c_1\hfill \\ J_{54}=s_{23}{s}_1\hfill \\ J_{55}=s_4\left(s_1{s}_2{s}_3-c_2{c}_3{s}_1\right)-c_1{c}_4\hfill \\ J_{56}=c_5\left(c_2{s}_1{s}_3+c_3{s}_1{s}_2\right)-s_5\left(c_1{s}_4+c_4\left(s_1{s}_2{s}_3-c_2{c}_3{s}_1\right)\right)\hfill \\ J_{61}=1\hfill \\ J_{62}=0\hfill \\ J_{63}=0\hfill \\ J_{64}=-c_{23}\hfill \\ J_{65}=-s_{23}{s}_4\hfill \\ J_{66}=s_{23}{c}_4{s}_5-c_{23}{c}_5\hfill \end{array}\right.$$

(10)

5.3. Trajectory Planning

To ensure smooth and vibration-free motion of the robot, abrupt changes in joint torques must be avoided. This requires the velocity and acceleration of each joint and the end-effector pose to change continuously over time. To achieve this, a seventh-order polynomial trajectory is adopted in Cartesian space. This approach guarantees the continuity of position, velocity, acceleration, and jerk.

$$\begin{array}{cc}\hfill X\left(t\right)& =k_0+k_{1}t+k_2{t}^2+k_3{t}^3+k_4{t}^4+k_5{t}^5+k_6{t}^6+k_7{t}^7\hfill \\ \hfill \dot{X}\left(t\right)& =k_1+2k_{2}t+3k_3{t}^2+4k_4{t}^3+5k_5{t}^4+6k_6{t}^5+7k_7{t}^6\hfill \\ \hfill \ddot{X}\left(t\right)& =2k_2+6k_{3}t+12k_4{t}^2+20k_5{t}^3+30k_6{t}^4+42k_7{t}^5\hfill \\ \hfill \stackrel{⃛}{X}\left(t\right)& =6k_3+24k_{4}t+60k_5{t}^2+120k_6{t}^3+210k_7{t}^4\hfill \end{array}$$

(11)

where the coefficients are given by:

$$\left\{\begin{array}{cc}\hfill k_0& =X_0\hfill \\ \hfill k_1& ={\dot{X}}_0\hfill \\ \hfill k_2& ={\displaystyle \frac{1}{2}}{\ddot{X}}_0\hfill \\ \hfill k_3& ={\displaystyle \frac{1}{6}}{\stackrel{⃛}{X}}_0\hfill \\ \hfill k_4& ={\displaystyle \frac{210(X_f-X_0)-t_f[(30{\ddot{X}}_0-15{\ddot{X}}_f)+(4{\stackrel{⃛}{X}}_0+{\stackrel{⃛}{X}}_f)t_f^2+120{\dot{X}}_0+90{\dot{X}}_f]}{6t_f^4}}\hfill \\ \hfill k_5& ={\displaystyle \frac{-168(X_f-X_0)-t_f[(20{\ddot{X}}_0-14{\ddot{X}}_f)t_f+(2{\stackrel{⃛}{X}}_0+{\stackrel{⃛}{X}}_f)t_f^2+90{\dot{X}}_0+78{\dot{X}}_f]}{2t_f^5}}\hfill \\ \hfill k_6& ={\displaystyle \frac{420(X_f-X_0)-t_f[(45{\ddot{X}}_0-39{\ddot{X}}_f)t_f+(4{\stackrel{⃛}{X}}_0+3{\stackrel{⃛}{X}}_f)t_f^2+216{\dot{X}}_0+204{\dot{X}}_f]}{6t_f^6}}\hfill \\ \hfill k_7& ={\displaystyle \frac{-120(X_f-X_0)-t_f[(12{\ddot{X}}_0-12{\ddot{X}}_f)t_f+({\stackrel{⃛}{X}}_0+{\stackrel{⃛}{X}}_f)t_f^2+60{\dot{X}}_0+60{\dot{X}}_f]}{6t_f^7}}\hfill \end{array}\right.$$

(12)

with boundary conditions:

$$\left\{\begin{array}{cc}\hfill X\left(t_0\right)& =X_0\hfill \\ \hfill \dot{X}\left(t_0\right)& ={\dot{X}}_0\hfill \\ \hfill \ddot{X}\left(t_0\right)& ={\ddot{X}}_0\hfill \\ \hfill \stackrel{⃛}{X}\left(t_0\right)& ={\stackrel{⃛}{X}}_0\hfill \\ \hfill X\left(t_f\right)& =X_f\hfill \\ \hfill \dot{X}\left(t_f\right)& ={\dot{X}}_f\hfill \\ \hfill \ddot{X}\left(t_f\right)& ={\ddot{X}}_f\hfill \\ \hfill \stackrel{⃛}{X}\left(t_f\right)& ={\stackrel{⃛}{X}}_f\hfill \end{array}\right.$$

where $X_0={\left[\begin{array}{cccccc}{p_x}_0& {p_y}_0& {p_z}_0& {\alpha }_0& {\beta }_0& {\gamma }_0\end{array}\right]}^T$ and $X_f={\left[\begin{array}{cccccc}{p_x}_f& {p_y}_f& {p_z}_f& {\alpha }_f& {\beta }_f& {\gamma }_f\end{array}\right]}^T$ represent the initial and final positions of the trajectory, and $t_f$ is the total duration of the trajectory planning.

5.4. Control Scheme

The operation control is based on vision positioning and feedback [23]. The control block diagram of the system is shown in Figure 10. The system employs a depth camera and UWB sensors to detect the position $x,y,z$ of the corner component, while an IMU sensor measures its attitude angles $\alpha ,\beta ,\gamma$. Based on the acquired position and orientation $X_c$, the desired end-effector pose of the robotic arm $X_d$ is calculated by the pose transformation relationship between operational pose and recognition pose $T_c^d$. The difference between the desired pose and the current pose $X_0$ of the robotic arm $\Delta X=X_d-X_0$ is scaled by a PD controller to determine the pose increment $K\Delta X+D\Delta \dot{X}$ for the next time step. Consequently, the desired pose for the next moment is obtained as $X_0+\Delta X$. Using this pose, the joint driving angles ${\theta }_i$ are calculated through inverse kinematics, allowing the robotic arm to perform the required operation. Here, the current pose $X_0$ is calculated via the forward kinematics based on the current joint angles ${\theta }_i$. And, the relationship of the pose increment of the end-effector and the joint increment is $\Delta X=\mathbf{J}\Delta Q$.

The advantage of this control method lies in its adaptability: when the end-effector is far from the desired pose, it enables a rapid approach to save time; when the end-effector is close to the target pose, it allows a slow approach to avoid collision with the manipulated object.

6. Simulation and Experiments

To verify the feasibility and effectiveness of the proposed system, a design and control method, a simulation system, and a prototype system were built for experimental validation. The simulation and experiment parameters are shown in

Table 1. Comparative results between single-feature and multi-feature perception method.

Parameters	Value
Controller-CPU	Core i7-12700
Controller-GPU	RTX4060
Number of DoF	6 (arm) + 1 (gripper)
K	$\left[\begin{array}{cccccc}0.02& 0.01& 0.01& 0.005& 0.005& 0.009\end{array}\right]$
D	$\left[\begin{array}{cccccc}0.0015& 0.0011& 0.0013& 0.00035& 0.00033& 0.00087\end{array}\right]$
$T_c^d$	$\left[\begin{array}{cccc}1& 0& 0& 50\\ 0& 1& 0& 0\\ 0& 0& 1& 120\\ 0& 0& 0& 1\end{array}\right]mm$

. Some parameters such as K and D are fine-tuned by trials. The parameters of the desired trajectories are produced through Equation (12).

6.1. Simulations

Based on the previously designed automatic loading system hardware, the optimized design of a multi-functional mechanical gripper, and the deep vision-based control algorithms, a dynamic simulation model was constructed in Gazebo, as shown in Figure 1a. Subsequently, simulation experiments were conducted on the loading and unloading of a fully automatic twist lock onto or from a Modular Transport Container Module, as illustrated in Figure 11a,b, respectively. The pose curves of the multi-functional mechanical gripper during loading and unloading are shown in Figure 12a–d, respectively.

6.2. Experiments

The full-scale prototype of automatic loading system is implemented as shown in Figure 1b. Subsequently, prototype experiments were conducted on the loading and unloading of a fully automatic twist lock onto or from a Modular Transport Container Module, as illustrated in Figure 13a,b, respectively.

The perception results of the depth camera during the loading and unloading process are shown in Figure 14a,b, respectively. It can be found that many features are recognized, including the target Corner feature, and multi-feature fusion is employed to improve the stability and robustness of positioning. The comparative experiments about the difference between single-feature and multi-feature perception methods are shown in

Table 2. Comparative results between single-feature and multi-feature perception method.

Method	Success Rate	Accuracy
Single-feature	88.26%	3.7 mm
Multi-feature	95.72%	1.2 mm

. Meanwhile, the object detection algorithm based on the improved YOLOv5 is compared with traditional YOLOv5. The reasoning speed of improved YOLOv5 is 450FPS, which is faster than that of traditional YOLOv5, that is, 120FPS.

The position and attitude angle curves of the multi-functional mechanical gripper during the loading process are shown in Figure 15a,b, respectively. As shown in the figure, the fully automated twist lock loading process using Arm 1 can be divided into six stages: initial grasping, rotating and moving beneath the corner piece, lifting the twist lock into the corner piece, rotating and releasing the twist lock, and returning the gripper to the initial position. In the initial grasping stage, the manipulator remains at its starting position, with the gripper closed to securely hold the twist lock. During the manipulation stage, the end-effector gripper rotates to an angle of ${70}^{°}$ along the z-axis, carrying the twist lock to a position directly beneath the corner piece. The manipulator then moves a certain distance along the negative y-axis to align the twist lock with the bottom hole of the corner piece. Subsequently, it moves upward along the positive z-axis. The end-effector then rotates from ${70}^{°}$ to ${130}^{°}$ along the z-axis. The gripper opens to release the twist lock into the corner piece. Finally, the gripper moves downward and returns to the initial position, completing the installation process.

The position and attitude angle curves of the multi-functional mechanical gripper during the unloading process are shown in Figure 15c,d, respectively. As shown in the figure, the fully automated twist lock unloading process using Arm 1 can be divided into six stages: approaching and reaching beneath the twist lock from the initial position, closing the gripper to grasp the twist lock, rotating to unload the twist lock, lowering to extract the twist lock, and placing it at the designated position before returning to the initial position. In the initial positioning stage, the manipulator starts from its home position. During the manipulation stage, the manipulator first moves to the rear-right side of the twist lock. The gripper then moves beneath the rear-right of the twist lock and rotates to an angle of ${130}^{°}$ along the z-axis. It continues moving to a position directly beneath the twist lock and then moves upward along the positive z-axis. The gripper closes to securely grasp the twist lock and lifts it up by 3 mm. The end-effector then rotates from ${130}^{°}$ to ${70}^{°}$ along the z-axis. The gripper releases slightly and moves downward to extract the twist lock. Finally, the manipulator moves the twist lock to a designated location, opens the gripper to release it, and returns to the initial position, completing the unloading process.

To highlight the significance of deep vision-based operation control, comparative experiments are implemented between with (closed-loop) and without (open-loop) deep vision. The comparative results of efficiency, accuracy, and control cycle are shown in

Table 3. Comparative results between with and without deep vision.

Method	Efficiency	Accuracy	Control Cycle
With deep vision	63.4%	15.3 mm	35 ms
Without deep vision	97.1%	1.2 mm	8 ms

. The efficiency is reflected by the success rate of operation. The accuracy means the error of position between the gripper and the automatic twist lock. The control cycle donates the system latency or real-time constraints. The target perception positioning algorithm takes up most of the time and the GPU is employed to accelerate the process to ensure real-time performance.

7. Discussion

To help readers better appreciate the practical significance of our work, this section summarizes the advantages, limitations, and implications of the proposed system.

7.1. Advantages

• End-to-end autonomy. The tight coupling of perception, planning, and control allows the system to complete the entire twist lock workflow without human supervision.
• Industrial-scale robustness. Extensive simulation and full-scale experiments confirm a $95.72\%$ success rate and $1.2$mm mean positional error under varying illumination, dust, and occlusions—requirements typical of real warehouses.
• Modular, transferable design. Thanks to the containerized architecture and parameterized gripper CAD model, the solution can be re-scaled to different container sizes or robot brands with minimal re-engineering effort.
• Computational efficiency. The vision stack runs at $>400$FPS and the whole control loop at $35$ms, meeting real-time constraints on standard off-the-shelf hardware.

7.2. Limitations

• Environment-specific calibration. Fixed-pose cameras must be re-calibrated if the working volume or robot base is moved; future work could employ active depth sensors on the end-effector to reduce this dependency.
• Lighting sensitivity. Although multi-feature fusion improves robustness, extreme back-lighting still degrades corner detection. Adaptive exposure control or event cameras are promising remedies.
• Limited generalisation beyond twist locks. The gripper fingertips are optimized for the ISO standard locking head; handling irregular bulk cargo would require replaceable jaws or soft robotic add-ons.
• Compute-heavy training. Building the 3200-image dataset and training the improved YOLOv5 once demand a high-end GPU cluster; incremental learning strategies could mitigate this overhead.

7.3. Implications for Industrial Deployment

Deploying the system in production warehouses could cut manual labor by more than $50\%$ and reduce error-induced downtime. Moreover, the design principles, including gripper–workspace co-optimization, feature-level perceptual fusion, and high-order trajectory planning, are applicable to other logistics operations such as palletizing or autonomous container stacking.

8. Conclusions

In this paper, an intelligent automatic loading system with a multi-functional mechanical gripper and deep vision-based control framework was proposed and implemented. The system effectively combines containerized transport logic with robotic automation, enabling unmanned loading and unloading in warehouse environments. Through optimized gripper design and multi-robot coordination, the system successfully completes complex tasks such as twist lock manipulation and container posture adjustment. The integration of a multi-sensor vision system ensures accurate pose estimation and improves operational reliability. Experimental validations confirm that the proposed solution achieves high efficiency and precision, marking a significant step toward fully autonomous industrial logistics systems. Future work will focus on enhancing the generalization capability of the vision module and expanding system scalability for larger scale deployments.