3D Component Segmentation Network and Dataset for Non-Cooperative Spacecraft

: Spacecraft component segmentation is one of the key technologies which enables autonomous navigation and manipulation for non-cooperative spacecraft in OOS (On-Orbit Service). While most of the studies on spacecraft component segmentation are based on 2D image segmentation, this paper proposes spacecraft component segmentation methods based on 3D point clouds. Firstly, we propose a multi-source 3D spacecraft component segmentation dataset, including point clouds from lidar and VisualSFM (Visual Structure From Motion). Then, an improved PointNet++ based 3D component segmentation network named 3DSatNet is proposed with a new geometrical-aware FE (Feature Extraction) layers and a new loss function to tackle the data imbalance problem which means the points number of different components differ greatly, and the density distribution of point cloud is not uniform. Moreover, when the partial prior point clouds of the target spacecraft are known, we propose a 3DSatNet-Reg network by adding a Teaser-based 3D point clouds registration module to 3DSatNet to obtain higher component segmentation accuracy. Experiments carried out on our proposed dataset demonstrate that the proposed 3DSatNet achieves 1.9% higher instance mIoU than PointNet++_SSG, and the highest IoU for antenna in both lidar point clouds and visual point clouds compared with the popular networks. Furthermore, our algorithm has been deployed on an embedded AI computing device Nvidia Jetson TX2 which has the potential to be used on orbit with a processing speed of 0.228 s per point cloud with 20,000 points.


Introduction
The OOS of spacecraft has already become one of the hot study areas in aerospace science over the past decades [1], particularly on fault detection approaches, influence of disturbances, modeling errors, and various uncertainties in the real systems [2].Most existing OOS missions, such as DEOS (Deutsche Orbital Servicing Mission) [3], iBOSS (Intelligent Building Blocks for On-Orbit Satellite Servicing) [4], and SIS (Space Infrastructure Servicing) [5], are based on cooperative spacecraft, which are equipped with cooperative marks and can communicate with the servicing spacecraft [6].However, more general cases in OOS are aimed at non-cooperative spacecraft.For instance, the OOS tasks can be repairing a malfunctioned satellite that has no cooperative mark and communication ability.Without cooperative marks and sometimes even no prior knowledge of the target spacecraft, the OSS task for non-cooperative spacecraft is still an open research area facing many technical challenges [7].
The component-level segmentation of non-cooperative spacecraft is the premise and core technology to achieve the more complex navigation-related tasks of the OOS task.

1.
A multi-source dataset for 3D spacecraft component segmentation is proposed.The dataset includes both visual point clouds and lidar point clouds using 3D reconstruction and 3D scanning.Finally, the dataset covers 32 3D spacecraft models including 3 component categories: body, panel, antenna.

2.
We peoposed a 3D spacecraft component segmentation network named 3DSatNet based on PointNet++ and made some improvements A geometrical-aware feature extraction layer named FE layer using ISS (Intrinsic Shape Signatures) [27] keypoint detection methods is proposed to extract geometric and transformation invariant features.Furthermore, a new weighted cross-entropy loss is proposed to increase the weight of components with fewer points, such as antenna.We fine tune the 3DSatNet by freezing the feature extractor and retraining the last full connected layers.We made some model pruning methods on PointNet++ to reduce the model size and compute power demand.

3.
When the partial prior point clouds of the target spacecraft are known, we proposed a 3DSatNet-Reg network by adding a Teaser-based [28] 3D point cloud registration module behind 3DSatNet to obtain higher component segmentation accuracy.For the purpose of achieving semantic corrections, we assigned the label of the prior point clouds to the closest point on semantic point clouds generated by 3DSatNet.
The remaining sections in this paper are organized as follows.Section 2 detailed presents the 3D multi-source spacecraft segmentation dataset and the architecture of the proposed 3DSatNet Network.Section 3 presents the experimental results.The result is discussed in Section 4. This paper is concluded in Section 5.

Materials and Methods
As shown in Figure 1, this section describes the 3D spacecraft component segmentation materials and methods we proposed.The 3DSatNet is trained on earth using our 3D spacecraft component segmentation dataset.The input of the 3DSatNet network is the 3D point cloud of the target spacecraft which is reconstructed from a sequence of on-orbit visual images or lidar-SLAM.The 3D point cloud is transferred into semantic point clouds via the 3DSatNet.When the partial prior point clouds of the spacecraft are known, 3D point cloud registration module based on Teaser [28] is added to 3DSatNet to obtain higher component segmentation accuracy, which is named as 3DSatNet-Reg.
Section 2.1 is the introduction of the 3D spacecraft component segmentation dataset we proposed to train the 3DSatNet model.Section 2.2 detailedly introduces the proposed 3D component segmentation network named 3DSatNet, and Section 2.3 presents the network 3DSatNet-Reg for accurate segmentation.To obtain a rather completed 3D model, the flight trajectory of the simulation engine is designed in Figure 3a.Besides, to obtain accurate scanning results, it is necessary to rebuild the collision model to the max for each component imported into the spacecraft.The final scanned 3D point cloud is shown in Figure 3c.
Generation of 3D point clouds via VisualSFM: VisualSFM [29] is a GUI application for 3D reconstruction using 2D images.VisualSFM is able to run very fast by exploiting multicore parallelism in feature detection, feature matching, and bundle adjustment.There are two methods to gain image sequences: Blender simulation, or using the camera on the Nvidia Jetson TX2 https://developer.nvidia.com/embedded/jetson-tx2,accessed on 15 March 2022.Generation of 3D point clouds via 3D scanner: The scanner selected in this experiment is EinScan Pro 2X Plus Handheld 3D scanner https://www.einscan.com/handheld-3dscanner/2x-plus/,accessed on 17 March 2022, as shown in Figure 6a below.Figure 6b shows the scene of the 3D scanner scans a 3D printing model of spacecraft.According to the different materials of spacecraft, the scanning mode includes handheld fine scanning and handheld fast scanning.The handheld fine scanning mode is used to scan high reflective objects by pasting control points on the target, while the handheld fast scanning mode which does not need to paste control points is feasible for scanning general materials.As the initial CAD models have different sizes and shapes, varying from around 50 cm to 6 m, which makes it difficult for MLP-based models to deal with.Furthermore, different CAD models have different coordinate systems, for example, AcrimSAT satellite may use the left vertex of the panel as the origin of coordinates while LRO https://nasa3d.arc.nasa.gov/detail/jpl-vtad-lro, accessed on 23 April 2022 may use the centroids.Thus, point cloud pre-processing methods including size normalization and centroid-based centralization is required.
The pre-processing methods of point clouds are based on Equations ( 1) and (2), after this pre-processing, the point clouds are transformed within the range of (0, 1) in x, y, and z-axis.Where P ⊂ R m×3 is raw point clouds, m is point numbers, P is the centralized and normalized point clouds data.
For a point cloud P ⊂ R m×3 containing n points, P i represents the ith points in P, P i represents the ith point in the point cloud after centralization.P morm represents the point cloud after normalization.P min represents the minimum value of the three channels (x, y, z) in P i .Similarly, P max represents the maximum value of the three channels.Through the above two steps, the coordinate range of the spacecraft is normalized to (0, 1), and the coordinate origin is changed to the center of mass of the spacecraft.
In this paper, our annotation system is based on open-source software CloudCompare https://www.cloudcompare.org/,accessed on 15 March 2022, and Figure 7 presents its details.
As shown in Table 2, the 3D spacecraft component segmentation dataset divides spacecraft into three categories: body, panel, antenna.In this dataset, the points distribution between different components in a spacecraft can be largely varied, a solar panel occupies 8000-15,000 points, while an antenna only occupies 300-1500 points in point clouds with 20,000 points.The results of some annotations are shown in Figure 8.As shown in Figure 9, the format of the annotated point clouds is described as follows: the first three columns are x, y, z coordinates, the following three columns are normal vectors, and the last column is semantic labels.
A total of 66 raw point clouds of four types are generated, as is shown in Figure 3 each containing 20,000 points.Then, various methods are adopted to enhance the obtained point clouds, such as randomly adding noise points, translating and rotating point clouds, and adding different degrees of occlusion.After data enhancement, 792 point clouds are obtained.  The point cloud number is the enhanced point cloud number and the summary of the two versions.
To facilitate the testing of various models, the final released dataset consists of two versions: one with 20,000 points per point cloud which provides a detailed representation of the three-dimensional structure of spacecraft, the other with around 2500 points per point clouds which share similar point numbers per shape as the ShapeNet dataset.To be specific, each point cloud with 20,000 points is downsampled to around 2500 points using ISS methods.

3DSatNet
The proposed 3DSatNet is inspired by PointNet++ [26].The pipeline of the proposed 3DSatNet is demonstrated in Figure 10.Firstly, A geometrical-aware feature extraction layer named the FE layer is proposed to extract geometric and transformation invariant features.When extracting features, the point cloud is continuously down-sampled.Then, the FP (Feature Propagation) layer is used for feature aggregation, which includes an up-sampling and skip connection parts.Moreover, a new loss function is proposed to increase the segmentation accuracy of the component with fewer points, such as antenna.The following sections will describe more details of the network.

The Proposed Geometrical-Aware FE Layers
The proposed FE layers contain three parts: ISS (Intrinsic Shape Signatures) point selection, ball query, and PointNet.We first partition the set of points into overlapping local regions by the ISS [27], ball query, and then feed them into PointNet for further feature extractions.
ISS [27] point selection method has better coverage of the structure point set given the same number of centroids.The ISS saliency measure is based on the Eigenvalue Decomposition (EVD) of the scatter matrix S(p).
N (p) means the neighborhood point of the point p, S(p) means the scatter matrix of the points belonging to the neighborhood of p. q represents the points in N (p).N means the total number of N (p).
Given S(p), the eigenvalues λ 1 , λ 2 , λ 3 (λ 1 > λ 2 > λ 3 ) are computed, points whose ratio between two successive eigenvalues is below a threshold are retained: where γ 12 = 0.5 and γ 23 = 0.5.After the detection step, a point will be considered as an ISS keypoint if it has the maximum salience value in a given neighborhood, as is shown in Figure 11.After obtaining the ISS key points, we take them as the center of the sphere and draw a sphere with a certain radius (given that point clouds in the dataset have been normalized, the radius is taken as 0.2 in this paper).Then the points located in the sphere are indexed.Although the number of points varies across groups, the PointNet layer can convert them into feature vectors with fixed-length.
The PointNet network is used as the backbone to extract feature vectors of point clouds.In order to reduce the model size, this paper prunes the PointNet network structure.We convert the MLP to matrix form, it can be found that the hidden feature vector is as follows: (x i , y i , z i ) are coordinates of one point P i in the point cloud P, k represents the output chanel of MLP, in our network k = 64.The learnable parameters of the MLP can act as a many rotation martix.The output of the T-Net is a 3 × 3 rotation matrix.Considering that the point cloud is put into a 3 × 64 MLP, the 64 output channels is stroung enough to make the network's rotation invariant approximately.

The FP Layers
FP layers share a similar structure as PointNet++'s decoder part, which adopts a hierarchical propagation strategy with distance-based interpolation and across-level skip links.Taking the FP3 layer as an example, for the 1 × 1024 feature output by FE3 layer, it was simply repeated 128 times to 128 × 1024.Then the feature was combined with the 128 × 896 features output by the FE2 layer to form a new 128 × (1024 + 896) feature, and then mapped to 128 × 256 by MLP.

Loss Function
As stated in Section 1 that data distribution between different components in a spacecraft can be largely varied.To improve the segmentation accuracy of components with few points(e.g., antenna), we proposed a new weighted cross-entropy loss.
N is a weighted parameter indicating that the point belongs to a certain category, N c represents the number of points belonging to the class c, y c is a one-hot vector.If the segmentation result is consistent with the prior label, the value of y c is set as 1; otherwise, the value will be set as 0. p c represents the probability that the predicted sample belongs to class c.

3DSatNet-Reg
In OOS tasks, several pictures and several frames of lidar scanning results of the target spacecraft can be obtained, and then visual 3D reconstruction technology and lidar 3D reconstruction techniques are used to obtain partial prior point clouds.Aiming at the problems of spatial target component segmentation with prior point clouds, this section proposes a Teaser-based [28] component semantic segmentation network 3DSatNet-Reg.The labels of the prior point clouds labeled on earth are assigned to the closest point of the raw semantic point cloud gained by 3DSatNet.The pipeline of the proposed 3DSatNet-Reg is shown in Figure 13.The point cloud registration problem aims to find the optimal rotation, translation and scale transformation matrices of the labeled prior point clouds A = {a i } N i=1 and the on-orbit semantic point cloud B = {b i } N i=1 .Considering the influence of the noise of the on-orbit semantic point cloud, the problem can be modeled as Equation ( 7): where i models the measurement noise, and o i is a vector of zeros if the pair (a i , b i ) is an inlier.i ≤ β i , where β i is a given bound.
The critical technology to realize partial prior point clouds and overall semantic point cloud registration is to detect the corresponding point pairs on the two point clouds.Traditional feature extraction methods such as FPFH (Fast Point Feature Histogram) and VFH (Viewpoint Feature Histogram) can't perform well, so we use D3Feat [30] to realize local feature detection and description which can be used to find key points and point correspondences with noise.
ICP (Iterative Closest Point) [31] is a commonly used registration algorithm when the initial pose between the two point clouds is not too large.Considering the large poses and scale difference between the prior point clouds and the semantic point cloud, a highly robust Teaser [28] registration algorithm is added to achieve a rough registration and estimation scale of two point clouds before ICP.Then, ICP is used to optimize the registration results.
Teaser [28] is a registration algorithm that can tolerate a large amount of noise.Teaser uses a TLS (Truncated Least Squares) cost that makes the estimation insensitive to a large fraction of spurious correspondences.It assumed the noise is unknown but bounded and proposed a general approach to decouple the estimation of scale, translation, and rotation.
The TRIMS (Translation and Rotation Invariant Measurements) [28] is used to estimate the scale ŝ.
Equation ( 8) can be solved exactly and in polynomial time via adaptive voting.
bounded noise s ij ≤ α ij .Given the scale estimate ŝ produced by the scale estimation and TIMS (Translation Invariant Measurements).The rotation estimate R is defined as Equation ( 9) which can be solved in polynomial time via a tight semi-definite relaxation.bounded noise ij ≤ δ ij .

R = arg min
After obtaining the scale and rotation estimates ŝ and R, t is estimated from (a i , b i ) in Equation (10) which can be solved exactly and in polynomial time via adaptive voting.tj = arg min Finally, the ICP [31] algorithm is used to complete fine registration.Estimate the combination of rotation and translation using a point to point distance metric minimization technique defined in Equation ( 11) which will best align each source point to its match found in the previous step.Then transform the labeled prior point clouds using the obtained scale, translation, and rotation.
(R, t) = argmin After registration, we calculate the closest point between the on-orbit semantic point cloud and the labeled prior point clouds and assign the label to the closest point.

Overall Performance
This section demonstrates the proposed 3DSatNet on both the ShapeNet-Part dataset and the proposed spacecraft component segmentation dataset.Then, we compare it with SOTA (State-of-the-Art) models.

Overall Performance on the ShapeNet-Part Dataset
The ShapeNet-Part dataset consists of 16,881 3D objects, covering 16 shape categories.Most of the point cloud instances are annotated with less than six labels, and there exist 50 parts labels in total.Following the official split policy announced by [32], the dataset is split into training, validation, and testing parts, containing 12,137, 1870, and 2874 shapes respectively.Both the object categories and the object components within the categories are highly imbalanced in the ShapeNet-Part dataset.
In this section, 3DSatNet is compared with the popular networks, including CNNbased, Graph-based, Transformer-based, and MLP-based methods.All of the networks are trained and tested on Nvidia RTX 3090 https://www.nvidia.cn/geforce/graphics-cards/30-series/rtx-3090-3090ti/, accessed on 22 April 2022 with 24 GB memory.We count the GPU occupancy of these networks.In terms of the metric for evaluation, we adopt the instance average mIoU and the class mIoU (mean Intersection-over-Union).The IoU of the shape is calculated by the mean value of IoUs of all components in that category, and instance mIoU is the average of IoUs for all testing shapes.The class mIoU is the average of IoUs for all testing classes.In the table, techniques like PointNet++, SO-Net also exploit normals besides point coordinates as the input features.
mIoU derives from the confusion matrix.Given that there exist τ + 1 classes in spacecraft point clouds (Body: 0, Panel: 1, Antenna: 2), p ij denotes the amount of points of class i inferred to class j.Assuming that p ij is true positive, so p ji and p ii represent false positives and true negatives respectively.Under the circumstances, we can derive the mIoU as Equation ( 12): It can be seen in Table 4 that although KPconv [23] achieves the highest class mIoU and Instance mIoU, the large model size and params numbers make it unsuitable for the OOS tasks.The Transformer-based point transformer occupied the most GPU memory, which can't be deployed on Nvidia TX2 (the fastest, most power-efficient embedded AI computing device that has the potential to be used on-orbit).The proposed 3DSatNet not only achieves the similar instance mIoU and class mIoU as SpiderCNN [33] and DGCNN [34] but also achieves an increase of 0.6% compared to the initial PointNet++_SSG and 0.2% compared to PointNet++_MSG.Particularly, our 3DSatNet achieves the smallest GPU usage (equal to PointNet) because we remove the T-Net part of PointNet and use the geometrical-aware FE layers.The proposed 3D spacecraft component segmentation dataset is insufficient enough compared with the ShapeNet-Part dataset, which even cannot be improved despite we adopt multiple sensors to acquire the 3D point clouds.Training the 3DSatNet directly on our dataset takes the risk of over-fitting and affects the generalization of the model.Thus, we train a fine-tune model based on the spacecraft dataset and initialize the model weights with ones that are pre-trained on the ShapeNet-Part dataset.To be specific, we establish a pre-trained weights on the ShapeNet-Part dataset and then froze layers except for the FC layers.Only the FC layers are trained during the training process.The training process is run on the Nvidia RTX 3090 with 24 GB memory, and all testing process based on the proposed dataset is run on the Nvidia Jetson TX2.Nvidia Jetson TX2 https:// www.nvidia.cn/autonomous-machines/embedded-systems/jetson-tx2/,accessed on 23 April 2022 is a power-efficient embedded 7.5-watt AI computing device.It's built around an NVIDIA Pascal™-family GPU and loaded with 8 GB of memory and 59.7 GB/s of memory bandwidth.
The first experiment demonstrates the influence of different training data on segmentation results through several comparative experiments.As shown in Table 5, considering that the data obtained on-orbit are actual point clouds, the test part only contains actual lidar point clouds and actual visual data.The training process requires large amounts of data and cannot be completed using only actual lidar or actual visual point clouds.Therefore, in the training data, the Lidar only in  Compare experiment A with B, adding visual point clouds to the training part can improve the instance mIoU, experiment C and D can get the same conclusion.So we suggest using all sources of point cloud to train the fine-tuned 3DSatNet network.
Secondly, we try to deploy all the advanced algorithms in Table 4 on TX2.Unfortunately, KPconv [23], Point transformer [36], SpiderCNN [33], and other algorithms can not be deployed due to their high GPU memory usage and large model size.
The segmentation results are listed in Table 4.Although DGCNN achieves the best instance mIoU, the KNN computation and dynamic kernel computation are quite timeexpensive.Our 3DSatNet achieves an IoU of 84.7%, which is superior to the other three MLP-based models.This experiment demonstrates that the ISS point selecting methods and the weighted cross-entropy loss proposed in our algorithm can improve the segmentation accuracy.Furthermore, the pruning of T-Net can make the model size reduced by 13.3% compared with PointNet++_SSG.

Components Segmentation Results Using 3DSatNet and 3DSatNet-Reg
In this section, we conduct an experiment to support our claim that the 3DSatNet-Reg has better performance than 3DSatNet.The experiment is based on a lidar point cloud generated by Arisim using the CAD model of Mars https://nasa3d.arc.nasa.gov/detail/eoss-mgs, accessed on 23 April 2022 with special components and a visual point cloud generated by visualSFM using the CAD model of Aqua https://nasa3d.arc.nasa.gov/detail/jpl-vtad-aqua, accessed on 23 April 2022.The visual point cloud contains a lot of noise.The visualization and precision analysis of the component segmentation results via 3DSatNet and 3DSatNet-Reg are presented in Figure 14 and Table 6.
Note that the 3D component segmentation results generated by other networks are listed in Figure A1.The lidar point cloud contains a special component that does not appear in the dataset, as the red box shown in lidar point clouds in Figure 14.Due to this inexistent component, 3DSatNet fails to generate semantic labels.However, when the prior model of the component is known, its semantic label can be predicted by the 3DSatNet-Reg and achieve an IoU of 89.2%.
Due to the presence of noise in the visual point cloud, the component segmentation results are far from expectations.The 3DSatNet network segments poorly even upon these popular components, such as body, panel.When the prior of the panel is provided, we can get the accurate segmentation result of the panel using registration.
The segmentation precision is shown in Table 6.Quantitative analysis shows that our 3DSatNet-Reg has achieved better results than the 3DSatNet, and can accurately generate the semantic labels of the components which are inexistent in the dataset using the supervision of registration.

Ablation
We design and perform extensive ablation studies on the proposed 3D spacecraft component segmentation dataset to analyze the significance of different components proposed for the 3DSatNet: the Fine Tune methods, the proposed geometrical-aware FE layer, and Our loss function.
We run tests on the 3D spacecraft component segmentation dataset under the same settings, and Table 7 presents the results.Model A means to train the 3DSatNet network on our dataset directly.Model B, C, and D use the weights trained on the ShapeNet-Part dataset as the weights of the trunk network, and only the FC layers and trained on our dataset.Model C uses the geometrical-aware FE layer, and model D uses both the FE layer and the proposed loss function.
As is shown in Table 7, directly training 3DSatNet network on our dataset has the problem of over-fitting.Considering there is only one class in our dataset, so we only compute the instance mIoU.Comparing model D to model B and model C, we observe that the proposed FE layer and the proposed loss function significantly improve the part segmentation's network performance.Compared to model B, model D's instance mIoU improved by 1.5%.These ablation experiments are further described in the following sections.In this experiment, we compare the proposed ISS with FPS and random selection methods.The three point selection methods are used to select 5 to 150 points.The sampling results are then processed by 3DSatNet on the spacecraft dataset.
The component segmentation results using three point selection methods are shown in Figure 15.The points selected by FPS contain no structure information, which leads to the network performing badly on some spacecraft's components with few points.The ISS point selection method achieved the best performance.When the number of selected key points is 20, the performance of the ISS-based point selection method achieved 11% higher accuracy than the FPS.With the increase of sampling number, the performance of the ISS is slightly better than FPS and a random selection, the advantage of the proposed method became less evident.That is because, with a large sampling numbers, randomly selected and FPS can ultimately sample the points along the spacecraft edges.The experiment results demonstrate that if the reconstructed point cloud is sparse, for example, the total number is below 150, it is crucial to use the ISS point selection method for a better component segmentation accuracy.

Comparisons on the Different Loss Functions
This experiment tests the different component segmentation performances using two different losses of point clouds: the proposed loss function and cross-entropy loss.The component segmentation visualization is shown in Figure 16.The accuracy and IoU with are defined the same as [25] are listed in Table 8.The red dots represent the points predicted incorrectly, and the blue dots represent the points predicted correctly.
It can be seen in Figure 16 and Table 8 that while using the proposed loss function, the component segmentation accuracy of body, panel, and antenna are all increased.Particularly, the IoU of the antenna significantly increase by 15.8% while the segmentation IoU of other categories not be affected.

Cross-entropy loss
Weighted cross-entropy loss Cross-entropy loss Weighted cross-entropy loss The proposed loss function Cross-entropy loss

Registration Results
This section verifies the feasibility of the registration scheme proposed in this paper.Experiments are carried out on two types of point clouds: visual point clouds and lidar point clouds.The experimental results are shown in Figure 17.
As shown in Figure 17, the first two columns are the results of D3Feat feature points (red points shown in Figure 17) extracted from the global point cloud and the prior point clouds respectively.The third column demonstrates the initial poses of the prior point clouds and the semantic point clouds gained by 3DSatNet, and their registration results are depicted in the fourth column.Further, the fifth column visualized the accuracy evaluation results of registration, and the sixth column presents the diagram of average precision.A conclusion can be drawn from Figure 17 that the proposed 3DSatNet-Reg can correctly register the two point clouds.

Dataset Limitations
The proposed 3D spacecraft component segmentation dataset is insufficient enough to train large scale networks, such as Kpconv [23], CapsuleNet [38], DGCNN [34], and Point Transformer [36].To solve this problem, firstly, we have adopted multiple sensors to acquire 3D point clouds.Then, we initialize the model weights with ones that are pre-trained on the ShapeNet-Part dataset.Therefore, future sequels of the dataset will include more spacecraft to facilitate the study of deep learning techniques beyond the application of 3D component segmentation about non-cooperative spacecraft.Furthermore, more advanced point cloud filtering methods can be used to improve the quality of the visual point clouds.For example, we can adopt GAN (Generative Adversarial Networks) to complete the point clouds.

Relation to Real Mission Constraints
In real missions, lidar is used to compute distances and relative poses.When the distance is long, the spacecraft contains only a few points, which cannot conduct component segmentation.For practical application, it is necessary to perform a fly-around task before component segmentation, using lidar odometry to reconstruct complete spacecraft point clouds, or using visual SLAM reconstruction to obtain sparse point clouds.In 3DSatNet-Reg, the prior point clouds obtained are not complete, and the density of the prior point clouds and semantic point clouds is also different.Future work will use some point clouds from lidar odometry to verify the registration accuracy.

Further Refinement of Segmentation Accuracy of Components with Fewer Points
To find a 3D component segmentation network suitable for spacecraft, our methods are based on the PointNet++ network and contain many improvements.A new weighted cross-entropy loss is proposed to increase the weight of components with fewer points.The pruning operation is performed on the model to reduce the size of 3DSatNet.The effect of improvement is still insignificant due to the limitation of MLP characterization capability.It is necessary to explore lightweight networks and introduce attention mechanisms to achieve higher-precision 3D spacecraft component segmentation results.Moreover, the accuracy of semantic correction in 3DSatNet-Reg depends on the accuracy of registration, more efforts should be made to increase registration accuracy.

Conclusions
This paper proposes a 3D spacecraft component segmentation dataset with multisource spacecraft point clouds.Then, A 3D spacecraft component segmentation network named 3DSatNet is proposed with a geometrical-aware FE layer to extract geometric and transformation invariant features and a new loss function to increase the segmentation performance of spacecraft components with fewer points.Experiments carried out using the proposed dataset and the ShapeNet prove that our network achieves better segmentation instance mIoU compared to all MLP-Based networks, and achieves the smaller forward time and model size compared to CNN-based, Graph-based networks.Specifically, when the prior point clouds are known, 3DSatNet-Reg is recommended.By registering the labeled prior point clouds with the on-orbit semantic point cloud, 3DSatNet-Reg has the ability to segment special components that are not contained in the dataset.

Figure 1 .
Figure 1.The pipeline of the proposed 3D spacecraft component segmentation system.

Figure 3 .
Figure 3. Airsim simulation system: (a) Airsim simulated trajectory.(b) Airsim simulated trajectory.(c) Airsim simulated point cloud.Blender2.8.2 https://www.blender.org/,accessed on 15 March 2022 is used to gain the simulation image sequence of spacecraft.As shown in Figure 4a, the CAD model of the Aqua satellite is imported into Blender, and a circle with a radius of 10 m at 10 m above the CAD model is drawn.Setting the camera's light center aligned with the model, 600 simulation images with the size of 1024 × 1024 are obtained by shooting around along the circular trajectory.Lately, the simulated vision point clouds are obtained via VisualSFM, and the noise within them is manually removed to get the relatively clean point clouds as shown in Figure 4b.

Figure 4 .Figure 5 .
Figure 4. Blender simulation system: (a) Blender simulation platform.(b) Simulation visual point cloud.The acquisition of actual image sequences uses the camera on the Nvidia Jetson TX2 developer component https://developer.nvidia.com/embedded/jetson-tx2,accessed on 15 March 2022, as shown in Figure 5a.The image acquisition frequency is 30 FPS, The TX2's camera is aligned to the 3D printed model, 600 actual images with the size of 1920 × 1080 are obtained by shooting around along the circular trajectory.The real vision point clouds are obtained using the VisualSFM method.

Figure 10 .
Figure 10.The architecture of the proposed 3DSatNet.

Figure 11 .
Figure 11.The ISS keypoint extraction result.The red points are the selected ISS keypoints.

Figure 12 .
Figure 12.Comparison of PointNet structure and the PointNet network structure used in our FE layer: (a) Partical structure of initial PointNet.(b) Partical structure of PointNet used in our FE layer.

Figure 14 .
Figure 14.Visualization of the component segmentation results via different networks.(Blue: panel, Green: body, Red: Antenna, Grey: special components.In 3DSatNet error and 3DSatNet error Figures: red points represent wrong segmented points, blue points represent points that are segmented correctly).

Figure 15 .
Figure 15.Component segmentation accuracy of three point selection methods.

Figure 16 .
Figure 16.The segmentation error visualization of cross-entropy and the proposed loss function.(red: wrong segmented points; blue: correct segmented points).

Table 1 .
The configuration of the simulation lidar in AirSim.

Table 2 .
Label of components in 3D spacecraft component segmentation dataset.

Table 3 .
Summary of the proposed 3D spacecraft component segmentation dataset.

Table 4 .
Comparisons with the SOTA methods on the ShapeNet-Part dataset and 3D spacecraft component segmentation dataset (%).Overall Performance on 3D Spacecraft Component Segmentation Dataset This section contains two groups of experiments.The first group experiments explore how to train the network, and the second group compares 3DSatNet network with SOTA networks in our dataset.All experiments in this section are tested on embedded computing devices.
Table 5 means that the training part contains only simulated lidar and real lidar point clouds, while Lidar & Visual indicates that the training sample contains four kinds of point clouds.

Table 5 .
Studies on the different sources in 3D component segmentation dataset (%).

Table 6 .
Accuracy evaluations of the component segmentation result via different models (%).

Table 8 .
Accuracy evaluations of cross-entropy loss and the proposed loss function (%).