6-DoF Grasp Detection Method Based on Vision Language Guidance

Abstract

The interactive grasp of robots can grasp the corresponding objects according to the user’s choice. Most interactive grasp methods based on deep learning comprise visual language and grasp detection models. However, in existing methods, the trainability and generalization ability of the visual language model is weak, and the robot cannot cope well with grasping small target objects. Therefore, this paper proposes a 6-DoF grasp detection method guided by visual language, which converts text instructions and RGBD images of the scene to be grasped into inputs and outputs for the 6-DoF grasp posture of the object corresponding to the text instructions. In order to improve the trainability and feature extraction ability of the visual language model, a multi-head attention mechanism combined with hybrid normalization is designed. At the same time, a local attention mechanism is introduced into the grasp detection model to enhance the global and local information interaction ability of point cloud data, thereby improving the grasping ability of the grasp detection model for small target objects. The method proposed in this paper first uses the improved visual language model to predict the plane position information of the target object, then uses the improved grasp detection model to predict all the graspable postures in the scene, and finally uses the plane position information to filter out the graspable postures of the target object. The visual language model and grasp detection model proposed in this paper have achieved excellent performance in various scenarios of public datasets while ensuring a specific generalization ability. In addition, we also conducted real grasp experiments, and the 6-DoF grasp detection method based on visual language guidance proposed in this paper achieved a grasp success rate of 95%.

1. Introduction

Grasp has long been a central focus in robot research, as it is the primary way robots engage in production and interaction. With the advancement of deep learning techniques, particularly the robust feature mapping and learning capabilities of neural networks, more accurate predictions of grasp postures can be achieved, especially in unstructured environments [1,2,3,4]. Many deep-learning-based methods for grasp posture prediction aim to predict the graspable postures of all objects within a multi-object scene. In general, in order to determine the optimal grasp posture, only the one with the highest confidence is selected. This leads to random object grasp with each prediction, which means the robot may not grasp the intended object as specified by the user. Consequently, many researchers have focused on interactive grasp tasks, where the user selects the object to be grasped [5,6,7,8]. The process generally unfolds in two steps in deep-learning-based interactive grasp methods. The first step involves determining the specific location of the target object from the image using text instructions, followed by predicting how to grasp it [9,10]. The second step involves using a planar grasp posture prediction network to predict the feasible grasp postures. However, this approach is limited by the camera’s angle and the scene layout, making it challenging for the model to accurately locate the target object in the first step.

Additionally, the planar grasp method cannot process spatial information, which leads to overlooking potential grasp postures and, consequently, reduced grasp success rates in practical applications. To address these issues, Lu et al. [11] introduced the VL-Grasp model. This novel approach integrates a visual language model with a six degrees of freedom (6-DoF) grasp posture detection model for interactive grasp tasks, yielding favorable results in real grasp experiments. Other researchers have incorporated semantic knowledge as prior information in task-oriented grasp to help robots adapt to various situations [12,13]; however, these methods are limited to specific environments and lack generalization. In response, Tang et al. [14] proposed the GraspGPT model, which leverages text instructions and a large language model interface to recognize and locate multiple objects, achieving a successful grasp.

Although these studies have significantly contributed to interactive grasp tasks, they have the following issues: First, identifying targets through large-scale language models can be time-consuming and may have specific network requirements when deployed locally; second, the existing visual language models used for grasping tasks have limited robustness and poor generalization ability during the training process; and third, existing grasp models often demonstrate inadequate performance when dealing with small objects. To address these challenges, this paper proposes a new 6-DoF grasp method based on visual language guidance based on Lu et al. [11]. The proposed method takes text instructions and RGBD images as input, utilizing a newly designed visual language model to detect the planar position of the target object. The process then uses an improved 6-DoF grasp model to identify all possible graspable postures within the scene. The object position predicted by the visual language model is subsequently used to select the optimal grasp posture. To assess the performance of the proposed method, we compare the recognition and prediction capabilities of the visual language and grasp detection models across different datasets. Additionally, real grasp experiments are conducted to validate the effectiveness of the proposed approach, demonstrating promising results.

In summary, our contributions to this paper are as follows:

(1)

A novel visual language model is introduced, which enhances the multi-head attention mechanism to optimize the training process and improve the model’s recognition and detection capabilities.

(2)

A new 6-DoF grasp model is proposed to address the challenges of grasping small and multi-scale targets. This model incorporates a point cloud feature encoder and multiple sampling mechanisms to enhance its ability to grasp small objects.

(3)

The end-to-end interactive grasp of small target objects is achieved by integrating the visual language model with the 6-DoF grasp and detection model. A real grasp experiment is conducted to validate the effectiveness of the proposed method.

2. Related Works

2.1. Visual Language Model

A visual language model’s task is to determine a target object’s position based on textual instructions. The model’s output typically consists of detection boxes and segmentation masks employed for target detection and image segmentation tasks, respectively. Traditional methods [15,16,17] utilize convolutional neural networks [18] and long short-term memory networks [19] to extract features. More recently, the Transformer mechanism [20] has gained widespread adoption in target detection tasks [21,22] due to its ability to capture global context and support parallel processing. Additionally, the Transformer excels at capturing long-range dependencies in sequences, making it well-suited for natural language processing and cross-modal learning tasks. In visual research, target detection and image segmentation are typically studied separately, but these tasks can share the same backbone network for early-stage feature extraction. Consequently, unified frameworks [23,24] have been proposed to perform specific tasks by leveraging a shared backbone network and distinct heads. For instance, the RefTR model proposed by Li et al. [23] separately extracts features from input text and images through the network. Incorporating the Transformer mechanism can generate separate outputs for target detection and segmentation.

2.2. Grasp Detection Model

Grasp detection models can be divided into planar and 6-DoF grasp according to coverage ranges. In the study of planar grasp, Lenz et al. [25] used a five-dimensional grasp box to represent the grasping posture. They generated the grasp results through a deep convolutional neural network. However, this method is highly dependent on the local features of the grasp scene. Kumra et al. [26] proposed the GR-ConvNet model to generate grasp postures via pixel-by-pixel detection. Morrison et al. [27] proposed a novel grasp label representation that can automatically generate pixel-level grasp labels from the grasp box, which improves grasp accuracy to a certain extent. However, planar grasp detection ignores many possible grasp results due to the lack of spatial information. In a study of 6-DoF grasp, Fang et al. [28] proposed the GraspNet-1Billion large-scale grasp dataset and an end-to-end grasp posture prediction network given a point cloud input. Chen et al. [29] proposed the TransSC network to complete the point cloud shape to address the problem of incomplete sparse point cloud features, and used the PointNetGPD [4] network to evaluate the grasp results. Finally, they cooperated with the MoveIt task builder for motion planning. In order to improve the model’s sampling ability of the entire scene, Ma et al. [30] proposed a new grasp method to solve the feature learning problem when the grasp scene scale is unbalanced.

Although the above-mentioned visual language model and grasp detection model have achieved good results, the existing visual language model training process is unstable, and the encoding ability of images and texts is insufficient. Therefore, when facing scenes where multiple objects are in contact with each other, the recognition and detection results are not precise enough. At the same time, the existing grasp detection model has weak grasp performance for small objects in stacked scenes. In order to solve these problems, this paper proposes a 6-DoF grasp detection method based on visual language guidance. It constructs a new visual language model and grasp detection model to improve the interactive grasp ability of small target scenes.

3. Method

3.1. Overall Framework

The overall framework of the 6-DoF grasping method based on visual language guidance proposed in this paper is illustrated in Figure 1. Visual language-based grasping is achieved through the visual language model and the grasp detection model.

First, an RGB image of the scene and textual instructions describing the object to be grasped are provided as input. The textual instructions, written in English, should specify the object’s type, color, and position as accurately as possible. Next, the visual language model extracts and encodes the input image and text features separately. The corresponding segmentation mask and target detection maps are then generated through the decoding process via the visual language converter. Subsequently, the RGB image and depth map are fed into the grasp detection model to reconstruct the 3D scene point cloud. This point cloud is then used by the improved grasp detection model proposed in this paper to predict the optimal grasping posture. Finally, the target detection map generated by the visual language model is used to identify the target object region and select the best 6-DoF grasping posture, which is then output through collision detection and grasp confidence ranking.

3.2. Improved Visual Language Model

The visual language model is depicted in Figure 2. For the input image and text, features are extracted using the image feature extraction network and the text feature encoder, respectively. The image feature extraction network employs ResNet50 [31], while the text feature encoder utilizes BERT [32]. The visual language encoder integrates the multimodal feature inputs from both language and text using six Transformer encoder layers based on the Transformer architecture [20] for multimodal encoding. Subsequently, the image–text pair is fed into the query encoder, which generates query phrase embeddings. The query decoder performs joint reasoning on these embeddings and decodes the multi-task features, which are then passed to the detection and segmentation heads. These heads produce a set of detection boxes and masks to guide the subsequent grasp configuration process.

After the model extracts image features and text features, respectively, in order to stabilize the training process of the model and improve the performance of the visual language encoder, we designed a new multi-head attention mechanism, the overall structure of which is shown in Figure 3. Figure 3a is a post-normalized multi-head attention mechanism. This method can produce a more substantial regularization effect, which helps improve the model’s performance. Figure 3b is a pre-normalized multi-head attention mechanism. Normalization is performed before the residual is added, resulting in a more significant identity path, which helps to accelerate convergence and more stable gradients. To combine the advantages of the two methods, we use a hybrid normalized multi-head attention mechanism [33] and use Dynamic Tanh (DyT) to replace the normalization layer, as shown in Figure 3c. DyT is shown in Figure 3d, where $\alpha$ is a learnable parameter. This operation aims to simulate the behavior of the normalization layer by learning the appropriate scaling factor through $\alpha$ and compressing the extreme values through the bounded tan function. DyT can improve the final performance of the model without adjusting the training hyperparameters.

3.3. Improved 6-DoF Grasp Detection Model

The overall structure of the grasp detection model proposed in this paper is illustrated in Figure 4. It consists of three stages: the point cloud feature encoding stage, the scene sampling stage, and the grasp configuration regression stage. The scene sampling stage is only used during the model test and is not involved in the model training.

In the point cloud feature encoding stage, the point cloud data is unordered, densely packed, sparsely distributed, and unstructured, which poses challenges during feature extraction. We employ the PointNetSA module from the PointNet++ [34] network to address these challenges for point cloud processing. The PointNetSA module uses farthest-point sampling to select several feature center points from the input point cloud. These center points are then used to group the point clouds, and the PointNet [35] network is applied to perform feature sampling on each group. We introduce a Transformer block after the PointNetSA module. This addition improves the model’s attention to relevant regions of the point cloud. Furthermore, the network’s feature extraction capabilities for multiple targets are strengthened through a network feature fusion structure, which integrates multi-scale features.

To address the model’s poor robustness when grasping small objects due to the uneven label distribution in the dataset, we applied the Scale-Balanced-Grasp [30] uniform scene sampling method. In the scene sampling stage, the objects are first segmented using the DSN segmentation network. This is followed by interpolation and fusion of the point cloud features obtained in the feature encoding stage with the segmented features. The fused features are then used to perform balanced sampling on the original point cloud features.

The DSN segmentation network, shown in Figure 5 [30], takes the original scene point cloud as input and extracts point cloud features through the point cloud segmentation converter. The network structure of the segmentation converter mirrors that of the point cloud encoder, utilizing the PointNetSA module for feature processing. The Transformer block applies attention, but the number of feature fusion layers is reduced to two. After extracting the scene features, the foreground and center direction modules are employed for data refinement and dimensionality reduction. Finally, the point cloud segmentation mask and center offset are obtained through inverse distance weighted interpolation. This interpolation method transfers the features of known points to target points based on distance, with the calculation formulas provided in Equations (1) and (2).

$$f^{(j)}(x)={\displaystyle \frac{{\displaystyle {\sum }_{i=1}^k{w}_i(x)f_i^{(j)}}}{{\displaystyle {\sum }_{i=1}^k{w}_i(x)}}}$$

(1)

$$w_i(x)={\displaystyle \frac{1}{d{(x,x_i)}^p}},j=1,\dots ,C$$

(2)

In Equations (1) and (2), $w_i$ represents the weight of the calculation point, which is inversely proportional to the distance. The closer the distance, the greater the influence. Typically, $f^{(j)}$ is the value of the unknown point, $f_i^{(j)}$ is the value of the known point, $k$ represents the points taken from the known point set for interpolation calculation, and $p$ signifies the influence of distance on the weight.

When sampling point cloud features, fixed-position sampling methods often overlook the features of small targets. We employ a multi-scale cylindrical group network during the feature capture and regression stages to comprehensively capture a wide range of sampling points across the scene. This network samples point cloud features using cylindrical sampling methods at varying depths and scales. The network architecture is illustrated in Figure 6.

The original scene point cloud is input into the network, where four cylinders with different radii and depths are used to sample the point cloud. The cylindrical features from these four different scales are stacked and combined into a new dimensional space. Next, shared convolution is applied, allowing all point clouds to share the same parameters, enabling weight sharing during training. A maximum pooling layer is then used to extract key features. The features obtained from the four cylinders with different radii are concatenated to form a complete cylindrical sampling feature. The calculation formulas are presented in Equations (3) and (4), where $T_{out}$ represents the complete cylindrical sampling feature, $T_i$ denotes the cylindrical feature with varying radii, $n$ represents the four cylinders with different radii, $C1D$ corresponds to one-dimensional convolution for dimensionality reduction, $D_{input}$ is the input original point cloud, $C{y}_i(x)$ denotes the cylindrical point cloud extraction at different depths, $k$ refers to the four cylinders with different depths, $smlp(x)$ represents the shared convolution, and $Max(x)$ indicates the maximum pooling layer.

$$T_{out}=C1D\left({\displaystyle {\sum }_{i=1}^n{T}_i}\right)$$

(3)

$$T=Max\left(Smlp\left({\displaystyle {\sum }_{i=1}^{k}C{y}_i(D_{input})}\right)\right)$$

(4)

To enhance the semantic features following feature extraction, we employ the ApproachNet [28] network to derive approximate vector features from the balanced sampling features described above. These vector features guide the fusion of local features at different scales. To further improve the interaction between local and global features, we introduce a local attention module, the structure of which is shown in Figure 7.

After incorporating local attention into both the cylindrical feature and the approach vector feature, these features are combined and input into the OperationNet [28] network to predict the in-plane rotation, approach distance, grasp width, and grasp confidence. Additionally, the data is fed into the ToleranceNet [28] network to assess the tolerance of the grasp point, specifically the robustness of the grasp operation to positional errors.

3.4. The Proposed Grasp Detection Method

This article proposes a 6-DoF grasp detection method based on visual language guidance. The overall grasp detection process is shown in Figure 1, and the specific implementation steps are as follows:

(1)

Obtain RGB and depth images to be grasped and describe the objects to be grasped in the scene using English text, including the target object’s type, color, and shape.

(2)

The improved visual language model uses RGB images and text as inputs to locate the position of the target object. The multi-head attention mechanism introduced in the model can improve the stability of model training and enhance the model’s feature extraction ability. The improved visual language model can predict segmentation masks and object detection images to guide the subsequent model’s grasp posture selection.

(3)

The improved grasp detection model reconstructs the scene point cloud using RGB and depth maps. After scene segmentation, cylindrical feature sampling, and point cloud feature extraction, it predicts all graspable poses in the scene.

(4)

We use the detection results of the visual language model to screen all graspable poses in the scene and obtain the optimal gripping pose through nonmaximum suppression and collision detection.

4. Implementation Details

4.1. Dataset and Evaluation Metrics

The RoboRefIt [36] dataset is used for model training in the visual language model. This dataset, designed for visual language detection and segmentation, contains RGBD images of 66 objects across 50,758 scenes. It provides 2D detection bounding boxes and segmentation mask labels for each scene. The dataset is divided into a training set, test set A, and test set B. Test set A consists of scenes featuring objects that appear in the training set, while test set B contains scenes with objects not seen in the training set. The intersection-over-union (IoU) ratio is used as the evaluation metric for the detection-bounding boxes, with a threshold of 0.5, and the average IoU ratio is used to assess the segmentation masks.

We also employed the GraspNet-1Billion dataset for the grasp detection model and used the average grasp precision as the evaluation metric for the proposed model. Additionally, to better assess the model’s performance in perceiving and grasping multi-scale objects, we use categories $A{P}_s$, $A{P}_m$, and $A{P}_l$ to evaluate the model’s grasp performance on small, medium, and large objects. The size divisions for grasp are defined as follows: 0–4 cm for small, 4–7 cm for medium, and 7 cm and above for large objects.

4.2. Loss Calculation and Training Details

The training loss function in the visual language model is the sum of the grasp detection box loss and the segmentation mask loss. The loss function for the grasp detection box is presented in Equations (5)–(7), where the weighted sum of L1 loss and generalized IoU loss is used. In this context, $L_{loss1}$ represents L1 loss, $L_{iou}$ represents generalized IoU loss, $b_0$ is the true detection box label, $b$ is the predicted detection box, $C$ is the minimum enclosing rectangular box containing both the true detection box $b_0$ and the predicted detection box $b$, and $N$ is the number of samples.

$$L_d={\lambda }_{iou}{L}_{iou}\left(b_0,b\right)+{\lambda }_{L1}{L}_{loss1}\left(b_0,b\right)$$

(5)

$$L_{iou}={\displaystyle \frac{\left|b_0\cap b\right|}{\left|b_0\cup b\right|}}-{\displaystyle \frac{\left|C-b_0\cup b\right|}{\left|C\right|}}$$

(6)

$$L_{loss1}={\displaystyle \frac{{\displaystyle {\sum }_{i=1}^N\left|b_0-b\right|}}{N}}$$

(7)

The loss function for the segmentation mask is shown in Equations (8)–(12), where the weight sum of focal loss $L_{focal}$ [37] and $L_{dice}$ loss [38] is used. Here, $s_0$ is the true mask label, $s$ is the predicted mask, $p_t$ represents the difference between the predicted and true values, and ${\alpha }_t$ is the weight factor.

$$L_s={\lambda }_{focal}{L}_{focal}\left(s_0,s\right)+{\lambda }_{dice}{L}_{dice}\left(s_0,s\right)$$

(8)

$$L_{focal}=-{\alpha }_t{\left(1-p_t\right)}^{\gamma }\log \left(p_t\right)$$

(9)

$${\alpha }_t=\left\{\begin{array}{l}\alpha ,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}if\hspace{0.17em}\hspace{0.17em}{s}_0=1\\ 1-\alpha ,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}otherwise\end{array}\right.$$

(10)

$$p_t=\left\{\begin{array}{l}s,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}if\hspace{0.17em}\hspace{0.17em}{s}_0=1\\ 1-s,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}otherwise\end{array}\right.$$

(11)

$$L_{dice}=2\left[{\displaystyle \frac{s_0\left({\displaystyle {\sum }_i^N{s}_i^2}+{\displaystyle {\sum }_i^N{s}_{i0}^2}\right)-2s\left({\displaystyle {\sum }_i^N{s}_i{s}_{i0}}\right)}{{\left({\displaystyle {\sum }_i^N{s}_{i0}^2+{\displaystyle {\sum }_i^N{s}_{i0}^2}}\right)}^2}}\right]$$

(12)

We define the total grasp loss in the grasp detection model as shown in Equation (13). $L_{approach}$ represents the loss of the approximation vector, $L_{Rotation}$ is the rotation angle loss, $\alpha$ is a hyperparameter with a value of 0.2, $W$ is the sample weight in the grasp scene, and $L_{Graspable}$ is the grasp robustness loss. In Equation (14), $c_i$ is the binary prediction for whether a point cloud in $i$ is graspable. If point cloud $i$ is graspable, $c_i^{*}$ is 1; otherwise, $c_i^{*}$ is 0. Additionally, $s_{ij}$ is the predicted confidence score for view $j$ in point cloud $i$, $s_{ij}^{~}$ is the true label value for view $j$ in point cloud $i$, $\left|v_{ij}\right.,\left.v_{ij}^{*}\right|$ represents the degree of difference, and $1(x)$ limits the difference between the predicted value and the true label to within 5 degrees. The classification loss $L_{cls}$ is defined using the softmax function, and the regional loss $L_{reg}$ is defined using the smooth L1 function.

$$L_{grasp}=W\left(L_{approach}+\alpha L_{Rotation}\right)+L_{Graspable}$$

(13)

$$\begin{array}{l}{L}_{approach}\left(c_i,s_{ij}\right)={\displaystyle \frac{1}{N_{cls}}}{\displaystyle \sum _i{L}_{cls}\left(c_i,c_i^{\ast }\right)}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+{\lambda }_1{\displaystyle \frac{1}{N_{reg}}}{\displaystyle \sum _i{\displaystyle \sum _j{c}_i^{\ast }1\left(\left|v_{ij},v_{ij}^{\ast }\right|<5^{°}\right)L_{reg}}}\left(s_{ij},s_{ij}^{\ast }\right)\end{array}$$

(14)

In Equation (15), $R_{ij}$ represents the grasping rotation, while $S_{ij}$, $W_{ij}$, and $d$ represent the grasp confidence, grasp width, and grasp approach depth, respectively. For $L_{cls}$, we use the sigmoid cross-entropy loss function in Equation (16), $T_{ij}$ denotes the maximum perturbation the grasp posture can resist.

$$\begin{array}{l}{L}_{Ratation}\left(R_{ij},S_{ij},W_{ij}\right)={\displaystyle \sum _{d=1}^K\left(\frac{1}{N_{cls}}\right.}{\displaystyle \sum _{ij}{L}_{cls}^d}\left(R_{ij},R_{ij}^{\ast }\right)\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+{\lambda }_2{\displaystyle \frac{1}{N_{reg}}}{\displaystyle \sum _{ij}{L}_{reg}^d\left(S_{ij},S_{ij}^{\ast }\right)}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\left.+{\lambda }_3{\displaystyle \frac{1}{N_{reg}}}{\displaystyle \sum _{ij}{L}_{reg}^d\left(W_{ij},W_{ij}^{\ast }\right)}\right)\end{array}$$

(15)

$$L_{Graspable}={\displaystyle \frac{1}{N_{reg}}}{\displaystyle \sum _{d=1}^K{\displaystyle \sum _{ij}{L}_{reg}^d}}\left(T_{ij},T_{ij}^{\ast }\right)$$

(16)

During the visual language model training, we froze the ResNet-50 weights and used pre-trained BERT weights. The model was trained for 90 epochs with a batch size of 16 and an initial learning rate of 0.0001. The input scene consisted of 20,000 point cloud samples for the visual grasp model. The Adam optimizer was used for training, with 50 epochs, a batch size of 6, and an initial learning rate of 0.001. Both models were trained in a PyTorch 1.6 environment with CUDA 12.1, utilizing an AMD EPYC 7302 processor and three NVIDIA GeForce RTX 3090 GPUs.

5. Results

Our experiments are categorized into two types: dataset-based experiments and real-world grasp experiments.

5.1. Experiments Based on Dataset

Table 1. Visual language model performance comparison.

Method	Backbone Network	Task	Test Set A	Test Set B	Params/MB	Time/ms
VL-Grasp [11]	ResNet50	TD	86.92	54.12	127.16	44.37
		OS	85.46	61.49
	ResNet101	TD	85.62	55.97	-	-
		OS	83.89	60.72
our	ResNet50	TD	87.96	55.19	128.36	65.53
		OS	88.23	62.44

shows the recognition prediction performance, model parameter quantity, and inference speed of the visual language model. It compares the performance of the visual language model proposed in this paper with that of VL-Grasp [11].

In test set A, the visual language model proposed in this paper achieved the best results in target detection (TD) and object segmentation (OS) tasks. When using ResNet50 as the backbone network, the visual language model proposed in this paper leads the original model in feature extraction by relying on the data processing capability of the improved multi-head attention encoding layer. Since test set B contains some new scenes that do not appear in the training set, test set B tests the model’s generalization ability more. The visual language model proposed in this paper also achieved good results. Two scenes from test set A and test set B were selected for visualization, with the detection and segmentation results shown in Figure 8. The figure displays the input text and image, along with the detection and segmentation results predicted by the model.

We used the GraspNet-1 Billion dataset to verify the grasp performance of the proposed model and compared it with other mainstream grasp detection methods [39,40,41] that use point clouds as input. We recorded each model’s average grasp values AP, AP_0.4, and AP_0.8 in seen, similar, and novel scenes. The results are shown in

Table 2. Comparison of grasp detection model performance.

Method	Seen			Similar			Novel			Params /MB	Time /ms
	AP	AP0.8	AP0.4	AP	AP0.8	AP0.4	AP	AP0.8	AP0.4
GPD [40]	22.87	28.53	12.84	21.33	27.83	9.64	8.24	8.89	2.67	-	-
PointnetGPD [4]	25.96	33.01	15.37	22.68	29.15	10.76	9.23	9.89	2.74	-	-
GraspNet-1B [28]	27.56	33.43	16.95	26.11	34.18	14.23	10.55	11.25	3.98	1.03	296
Li et al. [41]	36.55	47.22	19.24	28.36	36.11	10.85	14.01	16.56	4.82	-
GSNet [39]	67.12	78.46	60.90	54.81	66.72	46.17	24.31	30.52	14.23	-	-
Ma et al. [30]	63.83	74.25	58.66	58.46	70.05	51.32	24.63	31.05	12.85	1.14	43.29
our	65.13	76.15	60.24	59.87	70.90	52.29	25.10	31.67	13.17	1.58	87.83

. As can be seen from the table, the grasp detection model proposed in this paper achieved the best results in similar and novel scenes. Compared with the model proposed by Ma et al. [30], after adding the network feature fusion structure and local attention mechanism, the model’s overall performance in all scenes improved. In the seen scenes, the average grasp value AP increased by 1.3, the average grasp value AP increased by 1.41, and the average grasp value AP increased by 0.47 in the unfamiliar scenes.

In order to verify the grasp performance of the model proposed in this paper when dealing with objects of different sizes, we divide the grasp posture predicted by the model into small, medium, and large specifications according to the grasp width and calculate the corresponding values in three scenes, respectively. The results are shown in

Table 3. Grasp performance of the grasp detection model on multi-scale objects.

Method	Seen			Similar			Novel
	APS	APM	APL	APS	APM	APL	APS	APM	APL
Baseline	9.44	45.99	54.13	5.15	35.54	47.82	4.91	15.26	19.83
Ma et al.	18.29	52.60	64.34	10.03	42.77	57.09	9.29	18.74	24.36
Ours	19.93	55.39	70.65	10.65	45.84	64.38	10.05	20.53	27.05

. As can be seen from the table, thanks to the accurate sampling of small objects in the scene by the segmentation network, the grasp detection accuracy of small targets by the model proposed by Ma and the model proposed in this paper has been dramatically improved. The model proposed in this paper introduces a structure of fusion of different deep features, which improves the model’s local and global interaction ability. At the same time, adding local attention also improves the model’s attention to important features. Therefore, the accuracy of the model proposed in this paper when grasping objects of various sizes has been improved to a certain extent.

We visualized the results of the proposed model and the comparison model [28,30] when grasping small target objects, and the visualization results are shown in Figure 9. The more the grasp posture is biased towards red, the higher the grasp confidence is, and the more it is biased towards blue, the lower the grasp confidence is.

It can be seen from the figure that when facing small target objects such as strawberries and toy models, the GraspNet model without small target feature sampling assigns very few grasp postures to these small target objects, and the grasp postures are very simple. In addition, the model itself does not extract local features in place, resulting in serious collisions between the final generated grasp postures. After introducing the scale feature collection strategy and segmentation network, Ma’s model and the model proposed in this paper have increased the attention to small target objects and generated a series of grasp postures. Compared with the model proposed by Ma, the proposed model introduces a more flexible local attention mechanism, which makes the grasp postures predicted by the model more uniform and fitting for the object.

5.2. Real Grasp Experiment

We first built a real grasp detection platform, as shown in Figure 10. We used the KR5 R1400 robot and the Dahuan two-finger gripper to grasp objects and used the Realsense-D435i depth camera to obtain the RGBD image of the grasping scene in an “eye-on-hand” manner. We selected six objects, as shown in Figure 10b, for real grasp experiments, including apples, water cups, oranges, Coke cans, bananas, and mice that are common in everyday life. These objects appear in the RoboRefIt dataset and have different shapes and colors. Therefore, the model’s recognition and segmentation capabilities and grasp and detection capabilities for multiple objects can be effectively tested.

We select one object from the six objects in Figure 10b as the grasped object of this group of experiments and cyclically select the other three objects as interference items to add to the grasping scene. We conducted experiments with four objects each time and repeated 20 times as a group of experiments. A total of six groups of experiments are conducted, so 120 grasp experiments are finally completed. The experimental results are shown in

Table 4. Real grasp experiment results.

Scene	1	2	3	4	5	6
Grasp times	20	20	20	20	20	20
Successful grasp	18	19	17	20	20	20
Grasp success rate	114/120 = 95.00%

. Scenes 1–6 represent the six objects to be grasped: apple, cup, orange, cola, banana, and mouse. It can be seen from the table that the grasp detection method proposed in this paper has achieved excellent grasp results in various grasp scenes and different target objects to be grasped, with a grasp success rate of 95.00%.

Some grasp samples of the six objects to be grasped are shown in Figure 11, which shows the text description, RGB image, target detection result, target segmentation result, and grasp result of different objects to be grasped.

Figure 12 shows the visualization results of some failed grasp experiments, which were mainly caused by grasping the wrong object, colliding with other objects during the grasp process, and having an imperfect grasp posture. Therefore, in future research, it is necessary to design the robot’s motion trajectory path and optimize the grasp posture angle to improve the grasp success rate.

6. Conclusions

In order to meet the needs of intelligent grasp, improve the trainability and generalization ability of the visual language model, and improve the recognition and grasp success rate of small target objects, this paper proposes a 6-DoF grasp method based on visual language guidance. We verified the visual language model and grasp detection model on different data sets. The results show that the visual language model proposed in this paper has achieved high recognition detection results in both target detection and segmentation results. Compared with other grasp models, the grasp accuracy is also improved when grasping small target objects. We conducted real grasp experiments, which showed that the grasp success rate of the method proposed in this paper reached 95.00%.

Although the method proposed in this article achieves the grasp of specific objects in multi-object scenes, the model’s grasp detection ability is insufficient when dealing with other interference factors, such as the stacking of multiple objects and lighting. Moreover, during grasp experiments, the robot is prone to colliding with different objects, causing changes in the position of objects in the scene and ultimately affecting the grasp operation. Therefore, corresponding occlusion object segmentation algorithms should be added in subsequent research, and robot path planning algorithms should be designed to improve the overall grasp success rate of the model.