Fast UOIS: Unseen Object Instance Segmentation with Adaptive Clustering for Industrial Robotic Grasping

Abstract

Segmenting unseen object instances in unstructured environments is an important skill for robots to perform grasping-related tasks, where the trade-off between efficiency and accuracy is an urgent challenge to be solved. In this work, we propose a fast unseen object instance segmentation (Fast UOIS) method that utilizes predicted center offsets of objects to compute the positions of local maxima and minima, which are then used for selecting initial seed points required by the mean-shift clustering algorithm. This clustering algorithm that adaptively generates seed points can quickly and accurately obtain instance masks of unseen objects. Accordingly, Fast UOIS first generates pixel-wise predictions of object classes and center offsets from synthetic depth images. Then, these predictions are used by the clustering algorithm to calculate initial seed points and to find possible object instances. Finally, the depth information corresponding to the filtered instance masks is fed into the grasp generation network to generate grasp poses. Benchmark experiments show that our method can be well transferred to the real world and can quickly generate sharp and accurate instance masks. Furthermore, we demonstrate that our method is capable of segmenting instance masks of unseen objects for robotic grasping.

1. Introduction

Intelligent robot technology has always been the focus of research in the field of robotics, which can be applied to different real-world scenarios, such as healthcare, factories, and households [1,2]. In typical grasping tasks, the robot needs to perceive the environment through sensors and obtain the pose of the object to be grasped, as shown in Figure 1. However, designing efficient and accurate grasping algorithms in unstructured environments is an urgent problem to be solved due to the uncertainty of occlusion, object shape and pose. In particular, this requires that the robot can generalize previously learned knowledge to unseen objects. This paper focuses on unseen object instance segmentation to address the aforementioned issue, with the goal of quickly and accurately segmenting any arbitrary object instance in tabletop environments.

Traditional grasp algorithms mainly adopt analytical methods [3]. The methods use models based on geometry, kinematics, and dynamics to calculate feasible grasps. Due to the fact that these methods rely on modeling the physical interaction between robot end-effectors and objects, they often struggle to adapt well to unknown environments [4]. The data-driven approach is based on machine learning, which utilizes neural networks to learn feasible grasps from input data. Jiang et al. [5] proposed a two-step learning algorithm to learn a seven-dimensional oriented rectangle to represent the gripper configuration, including 3D position, 3D orientation, and gripper opening width. Thanks to the development of deep learning, object detection methods based on convolutional neural networks (CNNs) are used in grasp detection. Lenz et al. [6] proposed a two-step cascaded system based on deep learning, which utilizes two neural networks to detect potential grasps in a coarse-to-fine manner. The system becomes very time-consuming due to the exhaustive search methods used to find potential grasp rectangles. Inspired by [7], subsequent work used anchor-based ideas to design two-stage [8,9] and single-stage [10,11] grasp detection models, respectively. In addition to using exhaustive search and anchor-based methods, other works [12,13] also used the grid division method to regress grasp rectangles.

Different from the direct regression-based methods for grasp rectangles, Morrison et al. [14] proposed a pixel-wise grasp generation model. The model can generate antipodal grasps from input images at real-time speed. Due to the lightweight design of the model architecture, there is still significant room for improvement in terms of accuracy. To make a trade-off between the performance and efficiency of the model, a lot of subsequent work [15,16,17,18,19,20] has been conducted. The above methods are mainly designed for single-object scenes, so they will be affected by non-target objects in cluttered scenes and they may generate infeasible grasps. Recent work adopted the multi-view idea by averaging multiple grasps [21] or choosing to generate grasps at the best view [22]. They achieved higher grasp success rates while also increasing computational costs. To further reduce the influence from non-target objects, other methods also used semantic segmentation [23] and instance segmentation [24] to improve the accuracy of grasp detection. For the grasping of unknown objects in cluttered scenes [25,26,27,28], recent work [25,27,28] proposed unseen object instance segmentation (UOIS) and applied it to robotic grasping.

This paper proposes a CNN-based tabletop object instance segmentation for object 4D pose generation, in which we try to solve the trade-off problem between speed and accuracy. Our scheme first generates semantic masks and 3D offsets to object centers from an organized point cloud, and then adopts a mean-shift algorithm to obtain instance masks. In order to obtain better initial seed points, previous work [25,28] used a distance-based algorithm to select them carefully. However, this method is time-consuming and does not yield stable results. To effectively solve this problem, we use an adaptive method to generate initial seed points. First, the positions of local maximum and minimum are calculated based on the predicted 3D offsets. These positions are then used to choose the initial seed points from the center votes. Compared with [25,28], the proposed method of adaptive initialization of seed points can greatly improve the efficiency of the entire segmentation pipeline and improve the overall performance to a certain extent. In particular, our method performs well in segmenting instance masks of tabletop objects, which is very beneficial for grasping tabletop objects. The main contributions of this paper are as follows:

• We propose a fast unseen object instance segmentation method and apply it to robotic grasping. Dual experiments on data sets and robots demonstrate the effectiveness of the proposed method.
• We propose a method for adaptive initialization of seed points to optimize the mean-shift clustering algorithm, which significantly improves the efficiency of the unseen object instance segmentation pipeline. Our method achieves competitive performance compared with benchmark methods.

2. Problem Statement

A typical grasp configuration can be represented by a 5D grasp rectangle [6,8] in the image coordinates, which can be described as

$$\{x,y,\vartheta ,w,h\}$$

(1)

where $(x,y)$ represents the grasp center position, $\vartheta$ represents the grasp angle, w represents the width of the grasp rectangle, and h represents the height of the grasp rectangle.

Object detection-based grasp detectors usually use a 5D grasp representation, which makes it difficult for them to generate pixel-by-pixel grasps due to the time-consuming search of grasp rectangles. Instead of using the above 5D grasp representation, Morrison et al. [14] proposed an improved 5D grasp representation, which can be expressed in the robot frame as

$$g_r=\{{\mathbf{p}}_r,{\theta }_r,w_r,q\}$$

(2)

where ${\mathbf{p}}_r=(x,y,z)$ is the gripper’s center position, ${\theta }_r$ is the gripper’s rotation around the z-axis, $w_r$ is the required gripper width, and q is the grasp quality score. The improved grasp representation can be described in the image space as

$$g_I=({\mathbf{p}}_I,{\theta }_I,w_I,q)$$

(3)

where ${\mathbf{p}}_I=(u,v)$ represents the center point of the grasp in image coordinates, ${\theta }_I$ represents the rotation in the camera’s reference frame, $w_I$ represents the grasp width in image coordinates, and q represents the same scalar as in Equation (2).

q denotes the quality of the grasp at each point in the image. The value is a scalar between 0 and 1, where a value closer to 1 indicate higher grasp quality. ${\theta }_I$ denotes the angle of the grasp at each point. Because the grasp is symmetrical around $\pm \pi /2$ radians, the angle is expressed in $[-\pi /2,\pi /2]$. $w_I$ denotes the required width at each point. The value is the range of [0, $w_{\mathrm{m}ax}$] pixels where $w_{\mathrm{m}ax}$ represents the maximum width of the gripper, which can be converted to a physical measurement using the depth camera parameters and measured depth.

For a grasp obtained in the image space, it can be converted to the robot frame using the following transformations:

$$g_r=T_{RC}\left(T_{CI}\left(g_I\right)\right)$$

(4)

where $T_{CI}$ converts the image space into 3D camera space using the intrinsic parameters of the camera, and $T_{RC}$ converts 3D camera space into the robot frame using the calibration between the robot and the camera.

For a set of grasps in the image space, this can be expressed as

$${\mathbf{G}}_I=\{\mathbf{Q},{\mathbf{\Theta }}_I,{\mathbf{W}}_I\}\in {\mathbb{R}}^{3\times H\times W}$$

(5)

where $\mathbf{Q}$, ${\mathbf{\Theta }}_I$, and ${\mathbf{W}}_I$ represent three images composed of q, ${\theta }_I$, and $w_I$ at each pixel in an image, respectively.

The best grasp in the image space can be calculated as

$$g_I^{*}={\max }_{\mathbf{Q}}{\mathbf{G}}_I$$

(6)

The best grasp $g_r^{*}$ in the robot coordinates can be calculated by substituting Equation (6) into Equation (4). Typical planar grasp methods are susceptible to interference from nearby objects in cluttered object scenes. This may cause the robot to perform an incorrect grasp. We attempt to address the above problem using an unseen object instance segmentation model, which first segments possible instance masks from the input image. Then, the depth information that corresponds to the isolated masks or the closest-to-camera instance mask is extracted and fed into the grasp generator to generate pixel-wise grasps.

3. Method

Figure 2 shows the pipeline for segmenting unseen objects and generating grasp poses. The pipeline contains two main stages. In the first stage, Fast UOIS takes a three-channel organized point cloud $\mathcal{P}$ of XYZ coordinates as input and outputs instance masks of objects. It should be noted that $\mathcal{P}$ is calculated by backprojecting a depth image given camera intrinsics. In the second stage, the grasp generation algorithm takes the depth image and segmented instance masks as input and generates feasible grasps.

3.1. Network Architecture

In the first stage, instance masks of unseen objects are obtained from the input point cloud $\mathcal{P}$. We use an encoder–decoder architecture [25] based on U-Net [29], which performs instance segmentation by fusing features at different levels. Given an input point cloud $\mathcal{P}$ and features ${\mathcal{F}}_{\eta }\left(\mathcal{P}\right)$ extracted from it, where ${\mathcal{F}}_{\eta }$ represents the encoder–decoder network with weights $\eta$. The semantic segmentation masks $\mathcal{S}\in {\mathbb{R}}^{H\times W}$ and the 3D offsets $\mathcal{V}\in {\mathbb{R}}^{H\times W\times 3}$ of the object centers can be calculated, respectively, as

$${\mathcal{H}}_{\varphi }:\mathcal{F}\left(\mathcal{P}\right)\to \mathcal{S}$$

(7)

$${\mathcal{M}}_{\gamma }:\mathcal{F}\left(\mathcal{P}\right)\to \mathcal{V}$$

(8)

where $\mathcal{H}$ and $\mathcal{M}$ both represent a convolutional layer, their weights are $\varphi$ and $\gamma$, respectively, and the semantic segmentation masks includes three classes: background, table, and foreground/object.

Figure 2. The pipeline for unseen object instance segmentation with adaptive clustering and grasp pose generation. CGR consists of a convolutional layer (Conv), a group normalization layer (GN) and a ReLU. ESP represents the efficient spatial pyramid modules [30].

Figure 2. The pipeline for unseen object instance segmentation with adaptive clustering and grasp pose generation. CGR consists of a convolutional layer (Conv), a group normalization layer (GN) and a ReLU. ESP represents the efficient spatial pyramid modules [30].

Subsequently, the proposed adaptive clustering algorithm predicts pixel-wise classes from the input foreground/object masks $\mathcal{O}$, the point cloud $\mathcal{P}$, and the 3D offsets $\mathcal{V}$. The implementation details are described in Section 3.2.

3.2. Adaptive Clustering

To generate pixel-wise instance masks, we perform mean-shift clustering [31] on the predicted object centers $\mathcal{P}+\mathcal{V}$ in 3D space. Since $\mathcal{V}$ are 3D offsets, $\mathcal{P}+\mathcal{V}$ is the predicted object centers for each pixel. For simplicity, $\mathcal{P}+\mathcal{V}$ is called center votes $\mathcal{U}$. The mean-shift algorithm finds regions containing high-density data through a non-parametric method with a kernel density estimation. We use the kernel density estimation with a Gaussian kernel, which can be expressed as

$$\mathcal{Y}\left(\widehat{x}\right)={\displaystyle \frac{1}{N}}\sum _{n=1}^{N}K\left({\displaystyle \frac{\left|\right|\widehat{x}-{\widehat{x}}_n{\left|\right|}^2}{{\sigma }^2}}\right)$$

(9)

where bandwidth $\sigma >0$, Gaussian kernel $K\left(t\right)=\exp (-t^2/2{\sigma }^2)$, $\widehat{x}\subset {\mathbb{R}}^D$ represents the seed points, and ${\left\{{\widehat{x}}_n\right\}}_{n=1}^N\subset {\mathbb{R}}^D$ represents the data points to be clustered, which are the center votes $\mathcal{U}$.

For the seed points $\widehat{x}$, their initial selection affect the performance and efficiency of the final clustering. In order to trade off the performance and efficiency of clustering, we design an algorithm for adaptively generating initial seed points. Considering a single object, the positions at the maximum and minimum Euclidean distances of its center offsets can be used to select initial seed points from the predicted centers. Due to the uncertainty of the number of objects in unstructured scenes, the seed points selected by the above method cannot be assigned to different objects. To further solve this problem, we generate initial seed points within a local window. Given the center votes $\mathcal{U}$, the initial seed points can be calculated as

$$\widehat{x}=\mathcal{H}\left(\mathcal{U},f\left(\left|\right|{\mathcal{V}}_{\mathcal{O}}{\left|\right|}_F\right)+g\left(\left|\right|{\mathcal{V}}_{\mathcal{O}}{\left|\right|}_F\right)\right)$$

(10)

where ${\mathcal{V}}_{\mathcal{O}}\in {\mathbb{R}}^{\left|\mathcal{O}\right|\times 2}$ represents the 2D center offsets of the foreground pixels, $f(·)$ and $g(·)$ denote the positions where the local maximum and minimum values of the given input are calculated, respectively, and $\mathcal{H}(r,c)$ represents taking out the points corresponding to positions of c in r.

Whether a certain position $(u,v)$ is selectable can be determined by checking if its value in $\left|\right|{\mathcal{V}}_{\mathcal{O}}{\left|\right|}_F$ is equal to the local maximum. This process can be calculated as

$$P(u,v)=\left\{\begin{array}{c}\mathrm{True},\mathrm{if}\left|\right|{\mathcal{V}}_{\mathcal{O}}{\left|\right|}_F(u,v)=\mathrm{M}\left(\right||{\mathcal{V}}_{\mathcal{O}}{\left|\right|}_F)(u,v)\hfill \\ \mathrm{False},\mathrm{otherwise}\hfill \end{array}\right.$$

(11)

where $\mathrm{M}(·)$ represents the max-pooling operation, whose kernel size and stride are $k\times k$ and 1, respectively. Equation (11) is used to select the position of the local maximum. When the result is true, it indicates that the position is retained for the initial seed point calculation using Equation (10). The selection of the position for the local minimum can be determined by computing $-\left|\right|{\mathcal{V}}_{\mathcal{O}}{\left|\right|}_F$ using Equation (11).

Figure 3 shows the process of adaptive initialization seed points. First, we calculate the Frobenius norm of the 2D center offsets ${\mathcal{V}}_{\mathcal{O}}$ shown in Figure 3a. Secondly, the max-pooling layer is used to calculate the local maximum and minimum values of $\left|\right|{\mathcal{V}}_{\mathcal{O}}{\left|\right|}_F$ and $-\left|\right|{\mathcal{V}}_{\mathcal{O}}{\left|\right|}_F$, respectively, and the seed points are determined through Equations (10) and (11). The results are shown in Figure 3b and Figure 3c, respectively. Finally, the positions obtained above are combined, and the result is shown in Figure 3d. Compared with [25,28], the proposed initialization seed point algorithm can better allocate the points to different positions and adaptively set the number of initial seed points.

3.3. Loss Functions

Following [25], we use semantic segmentation loss ${\mathcal{L}}_{SS}$, center offset loss ${\mathcal{L}}_{CO}$, clustering loss ${\mathcal{L}}_C$, and separation loss ${\mathcal{L}}_{Sep}$ to train the model for generating semantic masks $\mathcal{S}$ and center offsets $\mathcal{V}$.

Semantic Segmentation Loss: We use weighted cross-entropy, which can be calculated as

$$\begin{array}{c}{\mathcal{L}}_{SS}=\sum _i{w}_i{\mathcal{L}}_{CE}({\mathcal{S}}_i,{\widehat{\mathcal{S}}}_i)\end{array}$$

(12)

where ${\mathcal{S}}_i$ and ${\widehat{\mathcal{S}}}_i$ are the predicted and the ground truth probabilities of pixel i, respectively, and ${\mathcal{L}}_{CE}$ is the cross-entropy. The weight $w_i$ is inversely proportional to the number of pixels with labels equal to ${\widehat{\mathcal{S}}}_i$, normalized to sum to 1.

Center Offset Loss: We use a weighted smoothing L1 loss (Huber loss), which can be calculated as

$$\begin{array}{c}{\mathcal{L}}_{CO}=\sum _i{w}_i{\mathcal{L}}_H({\mathcal{V}}_i,{\widehat{\mathcal{V}}}_i)\end{array}$$

(13)

where ${\mathcal{V}}_i$ and ${\widehat{\mathcal{V}}}_i$ denote the predicted and ground truth center offsets, respectively, the weight $w_i$ is inversely proportional to the number of pixels with labels equal to ${\widehat{\mathcal{S}}}_i$ in Equation (12), and ${\mathcal{L}}_H$ denotes the Huber loss.

Clustering Loss: We unfold the Gaussian mean-shift (GMS) for several iterations and perform it on the points to be clustered. GMS iteratively shifts a set of $D=\left|\mathcal{O}\right|$ 3D seed points $\widehat{X}\in {\mathbb{R}}^{D\times 3}$ to higher-density regions. Let ${\widehat{X}}^{\left(l\right)}$ denote the points at the lth iteration of GMS. ${\widehat{X}}^{\left(0\right)}$ is initialized to the center votes ${\mathcal{U}}_{\mathcal{O}}\in {\mathbb{R}}^{\left|\mathcal{O}\right|\times 3}$ of the foreground pixels. We use the following loss to ${\widehat{X}}^{\left(l\right)}$ and ${\mathcal{U}}_{\mathcal{O}}$, with corresponding object instance labels $Y\in {\mathbb{R}}^{\left|\mathcal{O}\right|}$:

$$\begin{array}{cc}{\mathcal{L}}_C^{\left(l\right)}({\widehat{X}}^{\left(l\right)},\mathcal{Z},Y)& =\sum _{i=1}^D\sum _{j\in \mathcal{O}}{w}_{ij}\mathbb{1}\{Y_i=Y_j\}{d}^2({\widehat{X}}_i^{\left(l\right)},{\mathcal{Z}}_j)+w_{ij}\mathbb{1}\{Y_i\ne Y_j\}{[\delta -d({\widehat{X}}_i^{\left(l\right)},{\mathcal{Z}}_j)]}_{+}^2\end{array}$$

(14)

where $w_{ij}$ is inverse proportional weight with respect to class size, $d(·,·)$ is the Euclidean distance, and ${[·]}_{+}=max(·,0)$, $\mathbb{1}\{·\}$ is the indicator function. For simplicity, ${\mathcal{U}}_{\mathcal{O}}$ is renamed to $\mathcal{Z}$. This loss function influences the kernel density estimation modes to be close to its point, and at least $\delta$ away from points not belonging to the cluster. $\delta$ is set to 0.1.

Applying Equation (14) to all points belonging to objects would result in excessive memory usage, so a stochastic version was used instead in [25,28]. They sampled a set $\mathcal{I}\subset \{1,2,\cdots ,|\mathcal{O}\left|\right\}$ containing $\mathcal{N}$ indices and set ${\widehat{X}}^{\left(0\right)}={\mathcal{Z}}_{\mathcal{I}}$ and ran GMS clustering only on these points. Different from [25,28], we use the method proposed in Section 3.2 to generate initial seed points. Similar to [25,28], when the number of generated initial seed points exceeds $\mathcal{N}$, we periodically sample a set containing $\mathcal{N}$ seed points. $\mathcal{N}$ is set to 50 and 300 during training and testing, respectively. We unroll GMS for L iterations and apply ${\mathcal{L}}_C^{\left(l\right)}$ at each iteration. L is set to 5 and 10 during training and testing, respectively.

Separation Loss: We employ a separation loss, which encourages the center votes to not necessarily be at the center of an object, as long as they are far from the center votes of other objects. To do this, consider the following equation:

$$M_{ij}={\displaystyle \frac{exp(-\tau d(c_j,{\mathcal{U}}_i))}{{\sum }_{j^{\prime }=1}exp(-\tau d(c_{j^{\prime }},{\mathcal{U}}_i)}}$$

(15)

where $c_j$ is the jth ground truth object center, $i\in \mathcal{O}$, and $\tau >0$ is a hyperparameter, which is set to 15. Equation (15) is simply the distance from center vote ${\mathcal{U}}_i$ to all object centers scaled by $\tau$ with a softmax applied. The separation loss uses cross-entropy, which can be expressed as

$${\mathcal{L}}_{Sep}\left(M_{ij}\right)=-\sum _{j=1}\mathbb{1}\{Y_i=j\}log\left(M_{ij}\right)$$

(16)

The total loss is denoted as ${\mathcal{L}}_{Total}={\lambda }_{SS}{\mathcal{L}}_{SS}+{\lambda }_{CO}{\mathcal{L}}_{CO}+{\lambda }_C{\mathcal{L}}_C+{\lambda }_{Sep}{\mathcal{L}}_{Sep}$. The hyperparameter weights ${\lambda }_{SS}$, ${\lambda }_{CO}$, ${\lambda }_C$ and ${\lambda }_{Sep}$ are set to 3, 5, 1, and 1 respectively.

4. Experiment

4.1. Datasets

We trained our model using the tabletop object dataset (TOD) [25]. The TOD dataset is a non-photorealistic synthetic dataset consisting of 40k synthetic scenes of cluttered objects on a tabletop. When evaluating model performance, we used the OCID [32] and OSD [33] datasets constructed under real scenarios. OCID has 2346 RGB-D images with a size of $640\times 480$, including cluttered tabletop and floor objects. To facilitate subsequent evaluation, we named the datasets in OCID that only contain cluttered objects on tabletop or floor as OCID-T and OCID-F, respectively. For the OCID-F dataset, the value of depth zero in each scene is set to 1600 mm to simulate a tabletop and named OCID-FT. OCID uses a semi-automatic approach to annotation, building incrementally by placing an additional object in each scene and using the depth difference between two consecutive images to mark new objects. The OSD has 110 RGB-D images with a size of $640\times 480$. Objects are placed on a tabletop, and their annotations are manually labeled.

4.2. Metrics

To evaluate the performance of instance segmentation, we used precision (P), recall (R), and F-measure (F). Precision, recall, and F-measure are calculated as $P={\sum }_i|b_i\cap g\left(b_i\right)|/{\sum }_i\left|b_i\right|$, $R={\sum }_i|b_i\cap g\left(b_i\right)|/{\sum }_j\left|g_j\right|$ and $F=2PR/(P+R)$, where $b_i$ is the set of pixels belonging to predicted object i on the tabletop, $g\left(b_i\right)$ is the set of pixels of the matched ground truth object of $b_i$, and $g_j$ is the set of pixels for ground truth object j. The above three metrics are called the overlap P/R/F, since the true positives are counted by the pixel overlap of the whole object. In addition, we also used the boundary P/R/F metric to evaluate how sharp the predicted boundary matches against the ground truth boundary. See [25] for more details. We reported all P/R/F measures in the range 0 to 100.

4.3. Details and Setups

The proposed model was implemented using 1.9.1 version of PyTorch. The model was trained using the Adam optimizer for 150k iterations, with a learning rate of $1\times {10}^{-4}$ and a batch size of 8. During the training stage, we applied multiplicative gamma noise to enhance the depth images similar to [25,28], and add Gaussian process noise to the back-projected point clouds. The kernel size of the max-pooling layer for adaptive clustering was set to $13\times 13$. The model was trained using a 2.4 GHz Intel Xeon Silver 4210R CPU and NVIDIA GeForce RTX3090 graphics card. During the testing stage, we removed any cluster of pixels that is smaller than 800. We set the kernel size of the max-pooling layer according to the object size of different datasets. For OCID and OSD, we set the kernel size of the max-pooling layer to $9\times 9$ and $13\times 13$, respectively. It should be noted that when using adaptive clustering, we removed the largest connected component in the post-processing stage, and other post-processing operations are consistent with [25,28]. In robotic grasping, we utilized a Franka Emika robotic arm with a Franka Hand. A RealSense D435i RGB-D camera was used to capture the required RGB-D images.

4.4. Experiments on Datasets

Quantitative Results:

Table 1. Evaluation of our method against SOTA methods on OCID and OSD.

Method	Time (s)	Parm (M)	OCID						OSD
			Overlap			Boundary			Overlap			Boundary
			P	R	F	P	R	F	P	R	F	P	R	F
Mask R-CNN † [34]	-	-	82.7	78.9	79.9	79.4	67.7	71.9	73.8	72.9	72.2	49.6	40.3	43.1
UOIS-Net-3D [25]	0.120	22.3	88.8	89.2	88.8	86.9	80.9	83.5	85.7	82.1	83.0	74.3	67.5	70.0
UCN [26]	0.250 ‡	-	83.1	90.7	86.4	77.7	74.3	75.6	78.7	83.8	81.0	52.6	50.0	50.9
UOAIS-Net [35]	0.074	77.4	89.9	90.9	89.8	86.7	84.1	84.7	84.9	86.4	85.5	68.2	66.2	66.9
MSMFormer [27]	0.278	-	88.4	90.2	88.5	84.7	83.1	83.0	79.5	86.4	82.8	53.5	71.0	60.6
FD-TNN [28]	0.123	27.2	90.0	89.9	89.7	86.4	83.1	84.4	88.0	87.4	87.7	74.3	72.8	73.4
UOIS-Net-3D-AC	0.065	22.3	88.8	88.7	88.5	86.7	80.5	83.1	86.3	82.6	83.7	74.7	68.1	70.6
FD-TNN-AC	0.068	27.2	90.4	91.0	90.5	88.1	83.9	85.6	88.2	87.2	87.7	75.0	72.9	73.7
Ours-OC	0.120	22.3	89.0	89.4	89.0	86.9	81.2	83.7	87.0	84.9	85.9	73.2	69.1	70.8
Ours-AC	0.065	22.3	89.1	89.4	89.0	86.8	81.2	83.6	87.5	85.3	86.2	73.8	69.7	71.4

compares the performance of different methods on OCID and OSD, where the bold numbers represent the best performances. “†” indicates that the results of this experiment are reported in [25]. “‡” indicates that the time is reported by the original paper, while the times required by other methods are obtained through our testing on the OSD dataset. UOIS-Net-3D-AC and FD-TNN-AC represent the experimental results obtained by applying our proposed adaptive clustering algorithm to the training models provided by [25,28], respectively. It should be clarified that when applying UOIS-Net-3D-AC to test the OCID dataset, we did not simulate tabletop scenes. Ours-OC represents applying the initialization seed points method used in [25] to the model trained by our proposed method. Ours-AC represents that the experimental results are obtained after training and testing our proposed adaptive clustering algorithm. UOIS-Net-3D-AC has a slight decrease in performance compared to the original methods, but the computational consumption is roughly reduced by half. Specifically, UOIS-Net-3D-AC achieved a 0.7% improvement in the overlap F-measure and a 0.6% improvement in the boundary F-measure. In addition, Ours-OC and Ours-AC achieved better scores than UOIS-Net-3D in various F-measures on OCID and OSD, which demonstrated the effectiveness of the adaptive clustering algorithm. In particular, FD-TNN-AC with an adaptive clustering approach achieved state-of-the-art performance. Compared with the state-of-the-art FD-TNN, FD-TNN-AC can achieve better performance with nearly half the computational cost. On OCID, FD-TNN-AC achieved relative improvements of 0.8% and 1.2% in overlap F-measure and boundary F-measure compared to FD-TNN, respectively. On OSD, FD-TNN-AC performed better than FD-TNN in overall performance.

Figure 4 shows the stability of different methods on OCID. For FD-TNN and UOIS-Net-3D, we tested the results 10 times each and took their average as the final result. It can be seen from Figure 4 that the results of FD-TNN and UOIS-Net-3D fluctuate to a certain extent due to the selection of the initial seed points of the clustering algorithm, and the maximum absolute error is about 0.2%. FD-TNN-AC and Ours-AC achieved stable results every time and outperformed the corresponding original methods overall.

To evaluate the performance of the proposed method for segmentation instance masks of objects on tabletops, we conducted instance segmentation tests on the OCID-F and OCID-FT datasets respectively, with results shown in

Table 2. Evaluation of our method on OCID-F and OCID-FT.

Method	Dataset	OCID
		Overlap			Boundary
		P	R	F	P	R	F
FD-TNN-AC	OCID-F	88.0	87.1	87.0	82.5	79.7	80.6
FD-TNN-AC	OCID-FT	89.5	90.2	89.5	87.4	82.2	84.4
Ours-AC	OCID-F	88.0	84.5	85.6	82.5	77.8	79.4
Ours-AC	OCID-FT	87.6	89.1	88.1	85.5	79.6	82.0

. The experimental results in Table 2 show that the performance of FD-TNN-AC is better for tabletop objects than for floor objects. Compared with the results obtained on OCID-F, the overlap F-measure and boundary F-measure obtained on OCID-FT are relatively improved by 2.5% and 3.8%, respectively. This performance is also reflected in the results that Ours-AC achieved on the OCID-F and OCID-FT datasets. Overall, although our method was affected by partial data from floor scenes, this issue could be mitigated to some extent by simulating floor scenes as tabletop scenes.

Qualitative Results: Figure 5 shows the results of segmentation instance masks on OCID and OSD by Ours-AC and UOIS-Net-3D. Rows 1 to 3 of Figure 5 show the results on OCID. The other rows show the results on the OSD. The results of Figure 5b,c are from UOIS-Net-3D and Ours-AC, respectively. Our method demonstrates the ability to accurately segment instance masks when objects are close to each other on both OCID and OSD datasets.

Impact of k-Value: In

Table 3. Evaluation of different kernel sizes.

$\mathit{k}\times \mathit{k}$	OCID						OSD
	Overlap			Boundary			Overlap			Boundary
	P	R	F	P	R	F	P	R	F	P	R	F
$9\times 9$	89.06	89.44	89.02	86.85	81.21	83.60	87.53	85.36	86.22	73.88	69.74	71.46
$11\times 11$	89.03	89.42	89.00	86.79	81.20	83.57	87.52	85.37	86.21	73.88	69.79	71.49
$13\times 13$	89.00	89.40	88.98	86.74	81.20	83.55	87.52	85.37	86.21	73.88	69.79	71.49
$15\times 15$	88.93	89.33	88.91	86.63	81.15	83.47	87.47	85.33	86.17	73.67	69.65	71.31
$17\times 17$	88.90	89.30	88.87	86.58	81.12	83.43	87.46	85.33	86.17	73.69	69.67	71.33

, we test the sensitivity of Ours-AC to the settings of kernel size of k over $\left\{9,11,13,15,17\right\}$. The value of k determines the number of initial seed points. When k is too small, the number of generated initial seed points will increase, which is beneficial to improving model performance to a certain extent, i.e., F-measure scores on OCID. However, this also results in increased computational costs. On the other hand, when k is too large, the number of generated seeds will decrease. Although this will reduce computational costs, it is not conducive to improving model performance. In fact, the value of k can be set relatively reasonably based on the size of objects in the image.

4.5. Robotic Grasping

We applied Ours-AC to the Franka Emika robotic arm with a Franka hand and a wrist-mounted RGB-D camera for grasping unseen objects in cluttered scenes (Figure 6). It should be clarified that we remove any clusters smaller than 500 pixels in the post-processing stage to accommodate grasp scenes. The task of the robot was to grasp objects on the table and then place them in a bin. The proposed method was first used to segment object instances from camera-captured images, and then the depth information corresponding to the isolated instance masks was fed into the grasp generation model. When the isolated instance masks were not available, the depth information corresponding to the instance mask closest to the camera was fed into the grasp generation model. The grasp model used GG-CNN [14] trained on the Cornell dataset. The robot setup and grasping process are kept consistent with existing work [14,15,28] for a fair comparison. Before each grasp attempt, the camera was positioned approximately 520 mm above and parallel to the surface of the table to obtain the depth image of the objects to be grasped. To perform a grasp, the robot first moved to a pre-grasping position. Subsequently, the robot moved straight down until the grasp pose was met or a collision was detected via force feedback in the robot. Finally, the gripper was closed, lifted, and moved to the placement area to place the object.

The 10 randomly selected objects in Figure 6a were shaken in a box and emptied into the robot’s workspace for each grasp attempt. The robot attempted multiple grasps, and any objects that were grasped were removed. This process would continue until all objects were grasped, three consecutive object grasps were failures or all objects were outside the observable workspace. The results are shown in

Table 4. Grasp success rate in cluttered scenes.

Method	Gripper	GPU	Time (ms)	Grasp Success Rate (%)
2020/Morrison et al. [14]	Parallel-jaw	RTX3090	12 †	87
2020/Kumra et al. [15]	Parallel-jaw	RTX3090	14 †	93.5
2022/Wu et al. [37]	Parallel-jaw	RTX2080	26	83.3
2022/Wang et al. [38]	Parallel-jaw	RTX3090	142 †	93.5
2022/Liu et al. [9]	Parallel-jaw	RTX2080Ti	47	90.2
2023/Cao et al. [19]	Parallel-jaw	RTX2080Ti	-	85.9
2023/Yu et al. [8]	Parallel-jaw	TITAN RTX	40	91.7
2024/Fu et al. [28]	Parallel-jaw	RTX3090	137	97.0
Ours	Parallel-jaw	RTX3090	77	93.4

, where the bold numbers represent the best performances. The results of other algorithms are reported in their original papers. “†” represents the time required for this method, which was obtained from our testing. We ran this experiment 10 times, and the robot achieved a grasp success rate of 93.4% (99/106) compared with 87%(83/96) in Morrison et al. [14]. Compared to the current state-of-the-art method [28], our grasp success rate is 3.6% lower. However, our grasp method reduces the time consumption by 60 ms. It should be noted that using only depth maps makes our grasp model less dependent on backgrounds to a large extent. In addition, using instance segmentation to weaken the complexity of the grasping environment enabled our method to handle more complex and cluttered environments, thus avoiding the limitation of grasp methods to simple scenarios with multiple objects to some extent [15].

Figure 7 shows some instance segmentation results and snapshots of the grasping states. A demonstration video of robotic experiments can be found here https://reurl.cc/E4znAk (accessed on 23 December 2023). Our method was able to accurately segment instance masks of objects in most cases. However, there were also instances of segmentation failures, such as under-segmentation of objects that were too close to each other. In addition, we created an object set comprising 27 household objects (Figure 6a,b) and 17 adversarial objects (Figure 6c). For each experimental run, we randomly selected 10 adversarial and 10 household objects to mix and emptied them onto the table in front of the robot. We ran this experiment 10 times, and the robot achieved a grasp success rate of 82.4% (183/222) compared with the 68% (134/196) in Morrison et al. [14]. The experimental results demonstrate that our method is robust in real-world grasping experiments and has the potential for related robotic grasping applications.

4.6. Discussion of Failure Cases

In this section, further analysis is conducted on experimental results of segmenting unseen object instances. Compared with benchmark methods, the proposed approach was able to generate more accurate segmentation results to some extent when objects were close together or overlapping. However, there were also several typical cases of segmentation failures. The main failures involved under-segmentation, such as in the results of the third and sixth rows in Figure 5 and the first row in Figure 7. In these examples, the proposed method failed to segment two similar spherical objects or two overlapping books. Another scenario was over-segmentation, as seen in the result of the sixth row in Figure 5. Future work could explore efficient methods using multi-view images or combining multimodal information (e.g., RGB images) to address these challenges. In practical scenarios, research could focus on robot manipulation strategies to further address these issues, e.g., push–grasp synergy methods.

5. Conclusions

This paper proposes a fast segmentation of unseen object instances and demonstrates grasping on a real robot. The proposed method uses an adaptive clustering algorithm by calculating the positions of local maximum and minimum from the predicted center offsets and obtaining the points at the corresponding positions in the center votes as the initial seed points. The experimental results demonstrate that the proposed method can achieve comparable performance to state-of-the-art methods with less computational costs. In particular, we show the superiority of our method for segmenting instance masks of unseen tabletop objects. The proposed method also demonstrates the effectiveness of grasping unknown objects in cluttered scenes.

In the future, we plan to optimize the algorithm for deployment on embedded AI computing devices. Additionally, we will further optimize the model to improve the accuracy and speed of segmenting unseen object instances. Furthermore, 4D pose representations generally have faster inference speeds than 6D pose representations, but this also makes the estimated grasp poses more restricted in space. Therefore, we will develop a 6D pose estimation algorithm in the next phase to enable more diverse grasping in the world space.