Parallel Point Clouds: Hybrid Point Cloud Generation and 3D Model Enhancement via Virtual–Real Integration

Abstract

Three-dimensional information perception from point clouds is of vital importance for improving the ability of machines to understand the world, especially for autonomous driving and unmanned aerial vehicles. Data annotation for point clouds is one of the most challenging and costly tasks. In this paper, we propose a closed-loop and virtual–real interactive point cloud generation and model-upgrading framework called Parallel Point Clouds (PPCs). To our best knowledge, this is the first time that the training model has been changed from an open-loop to a closed-loop mechanism. The feedback from the evaluation results is used to update the training dataset, benefiting from the flexibility of artificial scenes. Under the framework, a point-based LiDAR simulation model is proposed, which greatly simplifies the scanning operation. Besides, a group-based placing method is put forward to integrate hybrid point clouds, via locating candidate positions for virtual objects in real scenes. Taking advantage of the CAD models and mobile LiDAR devices, two hybrid point cloud datasets, i.e., ShapeKITTI and MobilePointClouds, are built for 3D detection tasks. With almost zero labor cost on data annotation for newly added objects, the models (PointPillars) trained with ShapeKITTI and MobilePointClouds achieved 78.6% and 60.0% of the average precision of the model trained with real data on 3D detection, respectively.

1. Introduction

With the rapid development of deep learning, computer vision technology has achieved competitive results on many tasks compared to humans, such as image classification [1,2], object detection [3,4], medical imaging [5,6], and object tracking [7]. However, challenges still exist, especially in 3D perception tasks [8], due to the increase of the data dimension. Precise 3D information understanding is indispensable in applications with high safety requirements such as autonomous driving and unmanned aerial vehicles, and point clouds are one of the commonly used data formats in such scenarios [9,10]. However, the annotation of point clouds generated by active sensors such as Light Detection and Ranging (LiDAR) is time consuming. Taking the LiDAR used in autonomous driving as an example, due to the sparsity and incompleteness of the point clouds generated by LiDAR, annotators need to refer to images or other inputs to precisely label the whole point cloud. This greatly increases the difficulty and time for point cloud annotation.

One of the popular ways to reduce data labeling is using the virtual data from [11]. However, although simulated data are becoming more and more visually realistic, they still suffer from the domain difference from real scenes [12]. To mitigate the gap between virtual and real scenes, we propose a new point cloud generation method that integrates real environmental contexts and virtual or scanned object information. Our method can maintain the layout of real scenes, thus reducing the difference between synthetic data and real data. Besides, the proposed point-based representation and LiDAR simulation greatly simplify the process of collision detection and make it easy to plug into existing data processing pipelines.

Deep learning methods greatly rely on the training dataset. In most scenarios of training deep learning models, researchers focus more on the upgrading of models and algorithms than the updating of the dataset [13]. One of the main reasons is that the updating of the dataset in real scenes is inconvenient and sometimes unaffordable (e.g., expensive LiDAR sensors and high annotation cost) for most researchers. Wang [14] proposed the Artificial societies, Computational experiments, and Parallel execution (ACP) methodology for the management and control of complex systems. Inspired by the ACP method, we integrate the model enhancement and point cloud generation into a closed-loop framework. Our virtual–real interaction data generation method enables us to adjust the point clouds flexibly at a very low cost. Therefore, we can use the evaluation results of the current model to guide the generation of point clouds. By iteratively updating both point clouds and model parameters, the performance of the model can be continuously promoted.

In conclusion, our contributions can be summarized as follows:

• We design a unified framework to integrate the point cloud generation and model enhancement, which changes the training of the model from an open-loop to a closed-loop mechanism. In this manner, both the performance of the models and the quality of point clouds are iteratively improved;
• The complexity of the collision computation between laser beams and meshes on objects can be totally avoided by our modeling of LiDAR sensors. In addition, a group-based placing method is put forward to solve the virtual object placing problem in real scenes;
• We build a hybrid point cloud dataset called ShapeKITTI with real point scenes and virtual objects and evaluate our method on 3D detection tasks. With almost zero annotation cost for newly added objects, we achieved 78.5% of the 3D performance of the model trained with the real KITTI dataset.

On the one hand, the parallel point cloud framework proposes to use software-defined and auto-labeled point clouds to train and evaluate 3D models. This greatly reduce the cost of data annotation. On the other hand, the parallel point cloud framework integrates both point cloud generation and model updating. This speeds up the enhancement of 3D models and can promote the development of intelligent industries such as autonomous driving, intelligent robots, and logistics.

2. Related Works

In this section, we mainly discuss the related works on the ACP method, synthetic datasets, and 3D detection methods. The ACP method inspired us to propose the framework of parallel point clouds. We built two hybrid point cloud datasets and conducted experiments on 3D detection to validate the effectiveness of our approach.

2.1. ACP Methodology

ACP theory was proposed by Wang et al. [14] for the modeling and control of complex systems. It includes three parts, i.e., artificial societies, computational experiments, and parallel execution. Recently, ACP theory has been extended to many areas such as computer vision [15], image generation [16], machine learning [17], autonomous driving [18], intelligent transportation [19], control theory [20], and so on. For computer vision tasks, artificial societies are used to model the real world and generate large-scale visual data with automatic annotation. Computational experiments are used to train and evaluate visual models for different vision tasks. Parallel execution feeds back the evaluation results to the construction of artificial scenes. The ACP method inspired us to rethink the relation of the data and model. Most of the deep learning methods use data to update the parameters of models in an open-loop manner. The evaluation of the model cannot be used to update the data due to the difficulty of changing real data.

2.2. Synthetic Datasets

With the rapid development and wide application of deep learning methods [21], effective and inexpensive generation and labeling of large-scale data are becoming an urgent problem to solve. To avoid the high cost of data generation in the real world, Many researchers turn to the virtual world for the solution. Predinger et al. [22] built a laboratory in a virtual city to analyze driving and travel behaviors. They built the environment of the virtual city using OpenStreetMap and CityEngine. Then, they added cars and transportation facilities in Unity and simulated the actions of cars and the interactions between cars and road networks. Ros et al. [23] constructed virtual traffic scenes and generated the depth and semantic labels with Unity and Computer-Aided Design (CAD) models. To improve the diversity of the dataset, they rendered different weather for the scenes. Gaidon et al. [24] imitated the real KITTI [25] dataset and constructed corresponding virtual KITTI sequences. The data in virtual KITTI had a similar structure and visual effect as real KITTI. Diverse annotations can be automatically generated for multiple vision tasks including object detection, segmentation, tracking, depth estimation, and so on. The ParallelEye [26] dataset was built based on the map information from the real world and helps to improve the performance of detection models [27]. The above methods generate virtual dataset by constructing virtual scenes from scratch. This is flexible, but increases the workload. Johnson-Roberson proposed to take advantage of existing video games such as Grand Theft Auto V (GTA V) to collect virtual image data and annotations. This further reduces the cost of constructing virtual scenes and improves the realness of synthetic data.

Different from the generation of image data, LiDAR point clouds are sparse and contain 3D information, which makes it hard to effectively process them using existing CNNs. Yue et al. [28] simulated LiDAR in GTA V and collected many point cloud data. They used virtual point clouds to train the segmentation model mixed with real data and improved the performance of the segmentation model. Fang et al. [29] combined real background points and virtual foreground points in the same scenes. The mixed data reduced the domain gap between real point clouds and synthetic point clouds. Our work in this paper adopted the idea of using hybrid point clouds to train the 3D models to reduce human annotation cost, but simplified the point cloud generation process.

2.3. Three-Dimensional Object Detection with Point Clouds

According to the data format used in the network of point clouds, most 3D detection models can be roughly categorized into point-based methods, voxel-based methods, and range-view-based methods.

Point-based methods [8] extract features from the original format of point clouds. PointNet is a pioneer work to learn 3D information from points. It uses one-dimensional convolution to encode each point and applies a max-pooling layer to aggregate global features. Such a design achieves the permutation invariance of unordered point clouds and helps to extract spatial and structural features. F-PointNet [30] extracts the local points at the second stage and uses an extended PointNet to predict positive points. VoteNet [31] generates proposals from point clouds using Hough voting. It uses seed points to help the regression of final 3D objectives. In voxel-based methods, sparse point clouds are first transformed into regular 2D or 3D grids [32,33] by voxelization. The new data format has a similar organization as image data; thus, it can be processed with traditional Convolutional Neural Networks (CNNs). VoxelNet [32] divides the point clouds into 3D cells and uses 3D CNNs to extract 3D features. PointPillars [34] ignores the vertical dimension in voxelization and uses the 2D CNN to accelerate the processing speed of feature extraction. Range-view-based methods project point clouds to the front view, which maintains the 3D information and can be processed effectively using CNNs. LaserNet [35] directly regresses the 3D bounding box form the range view inputs and adopts adaptive NMS to generate the final results. RCD [36] proposes a learnable dilation convolution to handle the size variation.

Three-dimensional detection was used as the example of 3D tasks to validate the effectiveness of our parallel point cloud framework.

3. Method

The framework of our method is shown in Figure 1. It comprises three parts, which are shown in different colors. Lines in yellow show the generation of hybrid point clouds, which includes the simulation of the LiDAR sensor and the combination of hybrid point clouds. They are illustrated in Section 3.1 and Section 3.2, respectively. Lines in red show the computational experiments on 3D models such as 3D object detection. Lines in blue illustrate our human-in-the-loop optimization of the data generation and the training process, which are discussed in Section 3.3.

3.1. Point-Based LiDAR Simulation

In this part, we discuss how to simulate the effects of LiDAR scanning for real or virtual objects with our virtual LiDAR. A point-based method was adopted to avoid the tedious computation of the collision between LiDAR beams and meshes of objects. The simulation of the LiDAR sensor is illustrated in Figure 2. By imitating the mechanism of real LiDAR, a virtual LiDAR was built that had similar horizon and vertical resolutions. For objects to be scanned, the original mesh models were first transformed into point clouds. Finally, equivalent scanning grids were proposed to simulate the scanning of the LiDAR sensor and generate the point clouds.

The beams of LiDAR can be encoded by a two-dimensional index $[i,j]$, where i is the id of the beam in the vertical dimension and j is the id of the beam in the horizontal dimension. The coordination system of virtual LiDAR is illustrated in the left of Figure 2. By only considering the range of scanning within the Field of View (FoV) of the camera, we denote the minimum angle in the vertical and horizontal directions as ${\alpha }_0$ and ${\beta }_0$, respectively. The resolution of virtual LiDAR is denoted as $r_v$ and $r_h$ in the vertical and horizontal directions. Then, we have:

$$\begin{array}{c}\hfill \begin{array}{c}{\alpha }_{i,j}={\alpha }_0+i\times r_v,\hfill \\ {\beta }_{i,j}={\beta }_0+j\times r_h,\hfill \end{array}\end{array}$$

(1)

where $\alpha$ and $\beta$ are the angle of the LiDAR beam in the vertical and horizontal directions. To calculate the projection of the beam with the index of $[i,j]$ onto the image plane, we assumed the distance of a point on the beam to the LiDAR sensor is $l_0$. Then, the coordinates of the point $P_{i,j}=[x_{i,j},y_{i,j},z_{i,j},1]$ are:

$$\begin{array}{c}\hfill \begin{array}{c}{x}_{i,j}=l_0\times cos(-{\beta }_{i,j})\times sin\left({\alpha }_{i,j}\right),\hfill \\ y_{i,j}=l_0\times cos(-{\beta }_{i,j})\times cos\left({\alpha }_{i,j}\right),\hfill \\ z_{i,j}=l_0\times sin\left({\beta }_{i,j}\right).\hfill \end{array}\end{array}$$

(2)

Assuming the LiDAR sensor has no relative rotation with respect to the camera, the rotation matrix can be ignored in the calculation of the extrinsic parameters of the camera. Let us denote the translation matrix as $C_w$, which is defined as:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill C_w=\left[\begin{array}{cccc}1& 0& 0& -x_0\\ 0& 1& 0& -y_0\\ 0& 0& 1& -z_0\\ 0& 0& 0& 1\end{array}\right]\hspace{1em},\end{array}\end{array}$$

(3)

where $[x_0,y_0,z_0]$ is the translation between the LiDAR coordinates and the camera coordinates. The intrinsic matrix K can be denoted as:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill K=\left[\begin{array}{cccc}{f}_x& s& p_x& 0\\ 0& f_y& p_y& 0\\ 0& 0& 1& 0\end{array}\right]\hspace{1em},\end{array}\end{array}$$

(4)

where $[f_x,f_y]$ are the focus-related parameters, $[p_x,p_y]$ is the offset, and s is the shear parameter. Then, we can project the 3D point $P_{i,j}$ onto the image plane. The projected point $\widehat{P_{i,j}}=[\widehat{x_{i,j}},\widehat{y_{i,j}},\widehat{z_{i,j}}]$ can be calculated as:

$$\widehat{P_{i,j}}=K{C}_w{P}_{i,j}^T$$

(5)

Then, for each beam, their projected position on the image plane is calculated to generate the equivalent scanning grids. For each point on the grids, the mean distance $d_{i,j}$ to all its neighbors can be calculated. A receptive range $s_{i,j}$ is assigned for point $\widehat{P_{i,j}}$ as follows:

$$s_{i,j}=\gamma \times d_{i,j}/2$$

(6)

where $\gamma$ is used to adjust the receptive range, and it is within the range $[0,1]$. The points from the virtual object will be kept if they are within the range of $s_{i,j}$ from $\widehat{P_{i,j}}$. If there is more than one point assigned to the same grid, we randomly keep one of them. With such a design, we can project the point clouds onto the image plane and compare them with the equivalent grids to see if they can be scanned by the virtual LiDAR. To simulate the reflectance of the LiDAR scan, we used the distance of the point and a random function as the inputs. Then, the reflectance r is:

$$r=max\left(min\right((0.7-\lambda \times l)+rand(0,0.3),1),0),$$

(7)

where $\lambda$ is used to adjust the reflectance strength, l is the distance of the point to the LiDAR sensor, and $rand\left(\right)$ is the random function used to generate a random value within [0,0.3]. The $min$ and $max$ function are used to restrict the value of the reflectance within $[0,1]$.

3.2. Hybrid Point Cloud Integration

To improve the realness of synthetic point clouds and avoid the heavy data annotation work, we separated the background point clouds and foreground point clouds. Real environmental point clouds were used as the background to reduce the domain difference. Virtual point clouds or scanned point clouds of objects of interest were used as the foreground to generate annotated data automatically. The public point cloud dataset KITTI [25] was used as the source of real point clouds. The integration of the hybrid point cloud pipeline is illustrated in Figure 3. To eliminate the influence of real points from foreground objects, all the points within the 3D bounding box were removed according to the labels of KITTI. Point clouds of foreground objects can either be generated from CAD models or scanned objects. For CAD models, we obtained their surface points with our surface point sampling method. To scan the objects in the real world, an iPad pro device with a LiDAR sensor was used. It can conveniently generate the point clouds of the objects of interest. We also proposed an automatic placing algorithm to choose proper places for foreground virtual point clouds. Although, in our scenario, foreground point clouds can be placed in the positions where real foreground objects are removed, such a placing mechanism cannot be generalized to background scenes with no real objects. Then, background and foreground points are combined, and virtual points are further processed by our virtual LiDAR to keep the visible points.

Existing CAD models can be used as the sources to generate the point clouds of the objects of interest. Considering that only outside points of an object can be captured in real scenes, we proposed a surface point sampling method to produce and upsample the points lying on the surface of the CAD models. As shown in Figure 4, firstly, the complete point clouds are generated from the CAD model with uniform sampling. Complete point clouds include both the outside and inside points of the CAD model. Therefore, it is necessary to remove the points lying inside the object. Making use of current 3D graphic software such as Blender, surface points can be extracted easily. However, in this paper, we tried to avoid using external software or applications to develop an online processing mechanism. We used the multiview projection to decide which points were on the surface of the object. Five views (front view, back view, left view, right view, and bird’s eye view) were used to create effects according to real situations. Finally, the points from different views were combined, and duplicate points were eliminated to generate the surface point clouds. The surface point clouds were further upsampled to guarantee that every beam of the virtual LiDAR could generate the point when it collided with the target.

Foreground point cloud generation: Two approaches were adopted to generate the foreground point clouds, i.e., generating from CAD models and generating from scanned real objects.

CAD models can provide a flexible imitation of common objects, but are not realistic enough. Existing real objects that we just removed only contain partial information (visible parts) and thus cannot be used for flexible data generation. Therefore, we proposed to use mobile LiDAR to reconstruct the objects in the real world. An iPad Pro (2020) was used to scan the real objects. We used the app called 3D Scanner APP [37] to scan objects and transform them into point clouds. The 3D models and corresponding point clouds are shown in Figure 5.

Group-based placing algorithm: To make the integration of virtual and real point clouds reasonable, we proposed a group-based placing algorithm to find the proper areas to place the virtual points into the real background points. Firstly, 2D keypoints on the BEV plane are uniformly defined with a resolution of $res$. Then, k neighbors for each keypoint are found according to the x and y coordinates using the constrained K-Nearest Neighbor (KNN) algorithm with a range of r. Finally, we computed the maximum and minimum height for the points within each group. If the height difference $d_h$ is below the threshold ${\tau }_0$, the corresponding keypoint is selected as a candidate position. For the object that is only provided with the normalized coordinates, we randomly generated the size $S_r$ to obtain its absolute coordinates using the constrained normal distribution, which is defined as:

$$S_r=max(min(X_s,S_{min}),S_{max}),X_s\sim N(S_{\mu },S_{\delta }^2),$$

(8)

where $S_{min}$ and $S_{max}$ are the minimum and maximum values of the size, respectively. $X_s$ is a random variable with a normal distribution. The mean value and the variance of the normal distribution are $S_{\mu }$ and $S_{\delta }$, respectively. To find a suitable orientation for the foreground object, we proposed a “random try and region-based validation” method.

We evenly generated $N_a$ anchor angles. For each anchor angle, a random shift, which obeys uniform distribution, is added. The t-th value of the angle is:

$$A_t=t\times \pi /N_a+X_a,X_a\sim U(-\pi /N_a,\pi /N_a),t=0,1,2,\dots ,N_a-1.$$

(9)

Then, we tried each random angle and cropped points in the 3D region formed by the candidate position, a random size, and a random angle with 3D Region of Interest (RoI) pooling. If the height difference of the points in the 3D region is lower than ${\tau }_1$, the current box is labeled as a valid one and the corresponding position, size, and angle information is recorded for placing the foreground object. To avoid the collision between multiple foreground objects, we iteratively removed the keypoints that lied in the current valid box. The group-based placing algorithm is illustrated in Algorithm 1.

3.3. Human-in-the-Loop Optimization

The parallel point cloud framework makes it possible for consistent optimization of both the point cloud generation and model parameters’ update. In this part, we mainly discuss the adjustment of the point cloud generation with the human in the loop.

There are many factors to be considered in the virtual point cloud generation such as the types of virtual objects, the density of points of the object, the size of the object, and so on. Although machine intelligence such as auto-augmentation [38] may have the potential to solve these problems, it demands massive computational capacity and needs carefully designed learning networks. In our situation, taking advantage of the human experience and the flexibility of virtual data was the most direct and effective way. After the training process, several randomly sampled data were used to evaluate the performance of the current model. Then, we visualized the outputs and asked three volunteers to comment on the detection results. Based on their opinions, the parameters in the generation of point clouds, such as the mean number of foreground objects in each frame, the mean size of foreground objects, the receptive range of the virtual LiDAR, the reflectance rate of virtual points, and so on, were optimized accordingly. After several iterations, the quality of the point clouds and the performance of the detectors were improved.

Algorithm 1: Group-based placing algorithm.

4. Experiments

In this section, we discuss the construction of the ShapeKITTI and MobilePointClouds datasets in Section 4.1 and Section 4.2. ShapeKITTI was built by using CAD models as the foreground, and MobilePointClouds was built by using scanned objects as the foreground. In Section 4.3, we evaluate our method on the 3D detection task. The optimization of data generation with the human in the loop is discussed in Section 4.4.

4.1. ShapeKITTI

We proposed a hybrid point cloud dataset, ShapeKITTI, using KITTI and ShapeNet as our sources of real and virtual point clouds. There were 3712 samples from the KITTI dataset and 200 car models from ShapeNet used to generate the training data for 3D detectors. The left 3769 KITTI samples were used to evaluate the performance of 3D detection. When generating the reflectance of virtual point clouds, $\lambda$ was set to 0.01. In our group-based placing algorithm, the resolution was set as 0.16m and the threshold as 0.1. We selected at most 64 points for each keypoint within 0.5 m around the keypoint. The demos of candidate placing generated by the group-based placing method are shown in Figure 6. As we can see, most of the proposed regions are flat, being suitable and reasonable for parking or driving. Imitating the parameters of the real LiDAR (Velodyne HDL-64E) used in the KITTI dataset, we configured our virtual LiDAR as shown in

Table 1. The configurations of the virtual scanning grids.

Attribute	H-Resolution (Channels)	V-Resolution	H-FoV	V-FoV	Detection Range
Value	0.4° (64)	0.2°	[−45°, 45°]	[−24.8°, 2°]	120 m

. The demos of our hybrid point clouds are shown in Figure 7. The proposed group-based placing algorithm distributes most of the added objects evenly into possible regions. Besides, it also effectively avoids the collision between adjacent objects.

4.2. MobilePointClouds

To enrich the source of object-level point clouds, we took advantage of the LiDAR sensor on the latest iPad Pro device and the 3D Scanner APP [37] to obtain more object-level point clouds from the real world. There were 40 real cars scanned to generate their point clouds. There were 3712 samples from the KITTI dataset used to generate the training data with 40 scanned objects for the 3D detectors. The left 3769 KITTI samples were used to evaluate the performance of 3D detection. Other settings were the same as those in the construction of ShapeKITTI. Following the steps described in Section 3, we set up another dataset called MobilePointClouds.

To fairly compare the difference of foreground objects between our hybrid datasets and KITTI dataset, we placed the virtual foreground objects into the locations of the original KITTI objects so that they had the same contexts. The comparisons are shown in Figure 8. Both the hybrid dataset and real KITTI showed the characteristics of linear scanning. While the density of point clouds in the hybrid dataset was higher than that in real KITTI, that may be caused by the higher complexity of the real environment where the emitted laser has a higher probability of loss.

4.3. Three-Dimensional Object Detection

In this part, we evaluate the ShapeKITTI dataset on real-time 3D detectors (PointPillars [34] and SECOND [33]). We trained all four models for 20 epochs with the Adam optimizer. The initial learning rates were set to 0.003 for these one-stage detectors. We evaluated the performance of the model under an IoU of 0.7 on the KITTI validation set. The results are shown in

Table 2. Performance on the KITTI validation set of detectors trained with the real KITTI and ShapeKITTI.

Method	3D AP			BEV AP			AOS AP
	Moderate	Easy	Hard	Moderate	Easy	Hard	Moderate	Easy	Hard
PointPillars-K	77.58	86.68	75.82	87.44	89.79	84.77	89.34	90.63	88.36
PointPillars-SK	60.94	75.54	58.26	75.93	86.49	73.95	74.23	86.90	72.38
PointPillars-MPC	46.28	56.32	45.81	71.86	85.49	71.53	71.08	85.51	70.62
SECOND-K	78.06	87.62	76.52	87.28	89.52	83.89	89.45	90.49	88.41
SECOND-SK	45.04	60.13	42.66	68.40	79.59	64.82	64.11	80.13	42.66
SECOND-MPC	38.83	53.62	36.87	54.33	70.35	55.01	58.52	74.22	55.90

, and we abbreviate the real KITTI dataset as K, the ShapeKITTI dataset as SK, and the MobilePointClouds dataset as MPC. For the widely used detector PointPillars (fast and light-weight), using ShapeKITTI, we could achieve 78.6% and 86.8% of the performance of models trained with the real KITTI dataset for 3D precision and BEV precision, respectively. This is a very promising result for a dataset that requires almost zero human annotations. With only 40 cars in our MobilePointClouds dataset, we achieved 60.0% and 82.2% of the performance of the PointPillars model trained with real KITTI for 3D and BEV precision, respectively, under moderate mode.

From the results in Table 2, we can see that detectors trained with SK achieved better results than MPC. Such superiority may come from the larger number of models in SK compared with MPC. We also found that both SK and MPC failed to achieve superior results compared to the real KITTI dataset. Although we tried to imitate the mechanisms of real LiDAR, several gaps between the real KITTI and our hybrid datasets still exist, such as the locations of foreground objects, the number or diversity of foreground models, and the distribution of point clouds on foreground objects. We analyze the influence of the first two factors in the following sections (Section 5.1 and Section 5.3. ). We think that the influence of the locations of foreground models can be handled to a large extent by our current group-based placing algorithm, and the influence of the number or diversity of foreground models will diminish with more kinds of foreground objects added. The distribution gap of point clouds on foreground objects between real and hybrid datasets was mainly caused by the difference in the scanning processes. This has a strong influence on the feature extraction, and many related approaches and methods have been proposed in transfer learning [39,40] and domain adaptation [41,42]. However, finding solutions to this is beyond the scope of this paper.

4.4. Human-in-the-Loop Optimization

The above experiments were all based on the final version of ShapeKITTI, which has gone through many generations of updates. In this part, we elaborate the process to update our hybrid point cloud dataset with the human in the loop. Taking PointPillars as an example, we evaluated the 3D AP on the KITTI val set for different generations of ShapeKITTI. The results are shown in

Table 3. The process of updating the ShapeKITTI dataset.

Version	Moderate	Easy	Hard	Feedback	Advice	AP
V0	10.32	10.64	10.53	Works for a few types of car	Add more cars	-
V1	16.64	22.14	13.98	Bad estimation for the size	Refine the size of the CAD models	6.14
V2	22.40	32.85	19.52	Bad performance for sparse objects	Generate more sparse objects	5.94
V3	26.85	29.43	23.51	Bad estimation for the height	Adjust the height of the objects	4.45
V4	29.74	38.99	27.10	Bad performance for sparse objects	Generate more sparse objects	2.89
V5	31.04	39.22	30.30	…	…	2.30

. We trained the 3D detectors for each version of the data and obtained feedback and advice from volunteers. For each iteration, we randomly sampled three groups of detection results from the validation data. They were provided to the volunteers so that they could evaluate the performance of the model trained with the current data. The feedback included the number of virtual objects, the size of virtual objects, the type of virtual objects, the density of generated point clouds, and the position of virtual objects. We collected 45 comments for each version of the data. Considering the possible variation of the feedback, we only selected the most common comment. The generation process of the hybrid dataset was then adjusted according to this advice, and the detectors were retrained with the new data.

5. Discussion

In this section, we analyze the design and effectiveness of the components in the proposed parallel point cloud framework. All the models (PointPillars) were trained on the MobilePointClouds dataset and evaluated on the KITTI validation set. To avoid the uncertainty introduced by the random functions in our method, we fixed the random seeds in this part (therefore, some results may be slightly different from the results in Section 4).

5.1. The Effectiveness of the Group-Based Placing Algorithm

In this part, we validated the effectiveness of the proposed Group-based Placing algorithm (GP) by comparing it to the Fully Random Placing algorithm (FRP) and the Ground-Truth-based Placing algorithm (GTP). For FRP, the objects were randomly placed in the range of view of the virtual LiDAR. For GTP, the ground-truth of the KITTI dataset was used to place virtual objects. The results are shown in

Table 4. Comparison of different placing algorithms.

Method	3D AP			BEV AP			AOS AP
	Moderate	Easy	Hard	Moderate	Easy	Hard	Moderate	Easy	Hard
FRP	35.25	40.59	31.93	61.22	72.62	56.29	60.17	75.88	55.37
GP	42.03	52.91	38.72	61.96	76.94	58.48	64.86	83.74	60.35
GTP	45.24	54.36	44.13	70.91	82.54	68.66	72.93	83.88	70.36

. The proposed group-based placing algorithm achieved much better results than the fully random placing algorithm and competitive performance compared with the ground-truth-based placing method. Form the results, we can see that the locations for placing objects had a great influence on the quality of the hybrid dataset. Randomly placing foreground objects might ignore the relations between foreground objects and the background, for example: it may place a car in a wall. Our method can automatically find the proper places for foreground objects, as shown in Figure 7, thanks to the perception ability of the local regions by analyzing the group statistics.

5.2. The Receptive Field of the Virtual LiDAR

In the proposed virtual LiDAR, most of the settings were designed by referring to the real Velodyne HDL-64E LiDAR such as the vertical and horizontal resolution. In this part, we discuss the influence of the value of the receptive field (in pixels) on the performance of 3D object detection. As shown in

Table 5. Comparison of different receptive fields in the proposed virtual LiDAR.

Value (m)	3D AP			BEV AP			AOS AP
	Moderate	Easy	Hard	Moderate	Easy	Hard	Moderate	Easy	Hard
0.5	18.24	22.44	16.11	45.23	60.34	39.70	39.68	49.81	35.51
1.0	40.63	43.41	37.66	57.01	68.05	55.02	62.85	75.67	58.52
1.5	42.03	52.91	38.72	61.96	76.94	58.48	64.86	83.74	60.35
2.0	39.10	46.50	37.05	57.77	70.25	56.58	62.31	71.64	57.07
2.5	38.28	41.04	34.68	54.05	63.11	52.73	60.63	69.11	55.56

, we obtained the best performance with the value of the receptive field set as 1.5 pixels. When we decreased the value of the receptive field, the diversity of the generated point clouds also declined because the captured points of each laser beam were restricted to a smaller range. On the contrary, the diversity would increase with a larger receptive field; however, more noise would be introduced, and the accuracy would be lower. According to the results shown in Table 5, setting the value of the receptive field to 1.5 achieved a good trade-off between diversity and accuracy.

5.3. The Number of Foreground Models

In this part, we compare the models trained using the dataset generated with different numbers of foreground models. It is worth noting that the “number” here refers to how many 3D models we used to generate the hybrid dataset instead of the number of objects in the generated hybrid dataset. One 3D model can generate multiple objects with different settings and background environments. The changes of the average precision for 3D detection, BEV detection, and AOS under moderate mode are illustrated in Figure 9. In general, using more foreground models usually results in better performance. However, some exceptions exist in 3D detection where more models lead to a relatively bad performance sometimes. One possible reason behind this is that the newly added models have distinct 3D structures from the objects in the KITTI validation set; therefore, they may lead to more serious divergence between the real (KITTI) and hybrid (MobilePointClouds) domains. Besides, we also found that even with only one foreground 3D model, we can still achieve an AP of 24.82 for 3D detection, which illustrates the effectiveness of our virtual LiDAR scanning method and automatic group-based placing algorithm.

5.4. The Number of Objects in the Dataset

As shown in Figure 10, we analyzed the influence of the number of foreground objects in the hybrid dataset on the performance of the trained models. The biggest increase was witnessed when the number of objects increased from 2500 to 5000. When we added more objects, the increase slowed down and sometimes could not be guaranteed. With the increase of the number of objects, the main obstacles that constrained the performance may change from the number of objects to the difference between real data and hybrid data.

5.5. The Time Cost of Each Operation in the Pipeline

To analyze the time consumption in the proposed hybrid point cloud generation method, we took the generation of hybrid data containing one foreground object as an example and divided the whole pipeline into six parts, as shown in Figure 11. “Data loading” denotes the loading of foreground and background points, as well as other configuration files. “LiDAR configuration” denotes the generation of equivalent scanning grids as depicted in Figure 2. “Place proposing” and “place selection” compose the group-based placing algorithm, as shown in Algorithm 1. “Virtual scanning” denotes the scanning process of the virtual LiDAR, and “label generation” denotes the generation of labels for hybrid point clouds. The whole process took 0.54 s, and the loading of the data took most of this time. All the data were stored on a hard disk; therefore, the loading process can be accelerated with a solid-state drive. The virtual scanning took 0.203 s, and the group-based placing algorithm took 0.101 s. Both of these processes are fully automatic and avoid manual participation, which can achieve higher efficiency compared with the human–machine interactive graphic software.

6. Conclusions and Future Works

In this paper, we proposed a simple and effective hybrid point cloud generation pipeline with almost zero human annotation. It greatly reduces the cost of constructing a large-scale point cloud dataset. Besides, we proposed a human-in-the-loop optimization method to build a closed-loop data generation and model update framework inspired by the ACP method. The experiments with 3D detectors verified the effectiveness of our approach. Three conclusions can be summarized for this paper. Firstly, the closed-loop framework was proven to be effective in the development of 3D detection models. Secondly, the proposed point-based LiDAR simulation method was validated to be effective in the generation of object-level point clouds. Thirdly, the proposed group-based placing algorithm was validated to be effective in the integration of foreground and background point clouds.

In future, we will further increase the diversity of our hybrid dataset by generating more 3D models, especially the scanned models with mobile devices. With the popularity of mobile devices with LiDAR or a ToF camera, the acquisition of 3D models will become more and more simple. It is possible to build a rich dataset by collecting the models that will be widespread through the Internet, which is promising to promote the development of 3D perception. Besides, improving the quality of hybrid point clouds with deep learning models is also very attractive. There still exists a gap between hybrid point clouds and real ones in the distribution of points. The influence of the environment on the LiDAR sensor in the real world is hard to simulate. With the development of generative adversarial networks, it will be possible for networks to learn to refine the hybrid point clouds and make them good substitutes for real point clouds.