Noncooperative Spacecraft Pose Estimation Based on Point Cloud and Optical Image Feature Collaboration

Abstract

Pose estimation plays a crucial role in on-orbit servicing technologies. Currently, point cloud registration-based pose estimation methods for noncooperative spacecraft still face the issue of misalignment due to similar point cloud structural features. This paper proposes a pose estimation approach for noncooperative spacecraft based on the point cloud and optical image feature collaboration, inspired by methods such as Oriented FAST and Rotated BRIEF (ORB) and Robust Point Matching (RPM). The method integrates ORB feature descriptors with point cloud feature descriptors, aiming to reduce point cloud mismatches under the guidance of a transformer mechanism, thereby improving pose estimation accuracy. We conducted simulation experiments using the constructed dataset. Comparison with existing methods shows that the proposed approach improves pose estimation accuracy, achieving a rotation error of 0.84° and a translation error of 0.022 m on the validation set. Robustness analysis reveals the method’s stability boundaries within a 30-frame interval. Ablation studies validate the effectiveness of both ORB features and the transformer layer. Finally, we established a ground test platform, and the experimental data results validated the proposed method’s practical value.

1. Introduction

With the continuous advancement of space technology, the number of satellite launches has rapidly increased, while the number of decommissioned and failed spacecraft is also growing. These not only waste valuable orbital resources but also increase the collision risk to high-value satellites. Therefore, various on-orbit service technologies have emerged, such as on-orbit life extension [1,2] and space debris removal [3,4]. Obtaining the pose information of the target is crucial for ensuring the success of the task. However, for uncontrolled noncooperative spacecraft, it is challenging to directly obtain their pose, which presents a significant challenge for the implementation of on-orbit services.

Under the condition of unknown target structural information, pose estimation methods for noncooperative spacecraft can be broadly classified into two categories based on the type of algorithm strategy employed. On one hand, methods based on visual sensors, such as a monocular camera, extract and match features from sequential images, using corresponding keypoints between frames to solve for the essential matrix [5,6] and estimate the pose. However, such methods are prone to accumulating errors over time. On the other hand, methods based on sensors such as a monocular camera [7], stereo camera [8], time-of-flight (TOF) camera [9], and LiDAR [10] reconstruct or directly obtain the target’s point cloud, using inter-frame point cloud registration to estimate pose changes. These methods have been more widely applied due to their robustness. Depending on the data source, pose estimation methods can be broadly classified into image-based and point cloud-based. Simultaneous Localization and Mapping (SLAM) methods based on visual image data are an important research branch. In 2009, researchers implemented 3D reconstruction and pose estimation for noncooperative targets with unknown structural information based on Fast SLAM [11]. In 2015, researchers applied Oriented FAST and Rotated BRIEF (ORB) features to SLAM for pose estimation [12,13]. For point cloud-based pose estimation methods, Iterative Closest Point (ICP) [14], a classic point cloud matching technique, is widely used for pose estimation of noncooperative targets; it directly calculates the Euclidean distance between point clouds to determine the matching relationship. In 2019, researchers used ICP combined with median filtering methods to achieve pose estimation of noncooperative targets [15]. In the same year, researchers employed the ICP algorithm for pose estimation of noncooperative targets based on identified satellite components [16]. Although ICP is highly robust, it is significantly affected by parameters and is prone to getting stuck in the local optimum. Many variants of ICP have been developed to improve its performance [17,18], but most still face the challenge of getting trapped in local minima. Additionally, point cloud geometry-based registration methods represent another important category of approaches [19,20,21]. Common geometric feature descriptors include the Euclidean distance between neighboring points, normal vector angles, etc. Utilizing deep learning networks, methods based on 3D voxel convolution [22] and graph convolution [23] can extract more robust and informative geometric features. Convolutional layers [24] and Transformer structures [25] can be used to replace the Euclidean distance between point clouds for estimating the matching matrix. The classification of pose estimation methods for noncooperative spacecraft is shown in

Table 1. Classification of pose estimation methods for noncooperative spacecraft.

Category	Sub-Category	Sensors	Key Algorithms
Image-based methods	Feature extraction and matching [5,6]	Monocular camera	Essential matrix solving
	SLAM approaches [11,12,13]	Monocular/Stereo camera	Fast SLAM, ORB-SLAM
Point cloud-based methods	ICP and ICP variants [14,15,16,17,18]	Monocular/Stereo/TOF camera, LiDAR	Euclidean distance
	Geometry-based methods [19,20,21]	LiDAR/TOF camera	Euclidean distance, Normal vectors
	Learning-based methods [22,23,24,25]	LiDAR/TOF camera	3D voxel CNN, Graph CNN, Transformer

.

However, the process of point cloud matching using geometric features still inevitably encounters issues of incorrect point correspondence, especially in the presence of similar structures on satellite panels, as shown in Figure 1a. Considering that the ORB algorithm’s feature descriptors incorporate directional information and rotational invariance, feature descriptors from visual images are more likely to achieve stable and accurate matching relationships under small temporal variations, as shown in Figure 1b. Therefore, inspired by methods such as ORB [26] and Robust Point Matching (RPM) [27], this paper proposes a noncooperative spacecraft pose estimation method based on point cloud and optical image feature collaboration. This method integrates feature descriptors from images with deep learning-based point cloud geometric feature descriptors, using a transformer to guide the prediction of point cloud matching correspondences, achieving pose estimation for noncooperative spacecraft. The main contributions of this paper are as follows:

(1)

This paper proposes a point cloud and optical image feature collaborative pose estimation network (POCPE-Net). The proposed network comprehensively considers both image features and point cloud features, providing a novel approach to pose estimation for spacecraft based on point cloud matching.

(2)

Based on the constructed point cloud and image dataset, comparative experiments and ablation studies were conducted between the proposed method and several existing methods. The results show that our method achieves superior accuracy on the validation set, with an MAE (R) of 0.84° and an MAE (t) of 0.022 m.

(3)

The method’s practicality was validated through a ground-based experiment using the measured data. The proposed approach achieves an MAE (R) of 0.97° and an MAE (t) of 0.015 m in real-world scenarios, demonstrating robust performance.

The organization of this paper is as follows: Section 2 introduces the proposed method, Section 3 presents the experimental results validating the effectiveness of the method, and Section 4 concludes the paper.

2. The Proposed Method

This paper proposes the point cloud and optical image feature collaborative pose estimation network (POCPE-Net). The overall network architecture of the proposed method is shown in Figure 2.

The input to the network consists of two frames of point cloud positions $X\in {\mathbb{R}}^{N_1\times 3}$, $Y\in {\mathbb{R}}^{N_2\times 3}$ and images $I_X$, $I_Y$. First, the point cloud feature descriptors ${\tilde{F}}_P^X$, ${\tilde{F}}_P^Y$ and image feature descriptors $F_O^X$, $F_O^Y$ are extracted separately. Using the attention mechanism of the Transformer, the fused feature-driven point cloud matching correspondences are obtained, and the matching matrix $M^i$ is derived through Sinkhorn normalization. Using weighted SVD to solve for the i-th iteration’s pose transformation $\{R^i,t^i\}$, the point cloud $X$ undergoes an SE(3) transformation and enters the (i + 1)-th iteration. After the specified number of iterations, the final pose transformation $\{R,t\}$ is output. This work builds upon several excellent existing methods, specifically reusing RPM-Net’s point cloud feature extractor and traditional ORB feature descriptors due to their superior feature representation and registration capabilities. Our key innovation lies in employing Transformer attention mechanisms to optimize feature fusion and collaboration, leveraging 3D point cloud-to-2D image feature correlations to enhance both the accuracy and robustness of point cloud matching, thereby improving pose estimation precision.

2.1. Feature Extraction

First, point cloud geometric feature descriptor $F_P$ is generated following the method described in [27]. The geometric feature descriptor of a single point $x_C$ is shown in Equations (1) and (2).

$$F_P^{x_C}=[x_C,\{\Delta x_{C,i}\},\{\mathrm{PPF}(x_C,x_i)\}],x_i\in \tilde{N}(x_C)$$

(1)

$$\tilde{N}(x_C)=\{x_1,x_2,\cdots ,x_k\}$$

(2)

where $\Delta x_{C,i}$ represents the vector differences between the point cloud and the positions of its i-th neighbors, $\tilde{N}(x_C)$ represents the set of k neighbors of $x_C$, and $\mathrm{PPF}(x_C,x_i)$ are 4D point pair features (PPFs) [28] defined by Equation (3).

$$\begin{array}{cc}\hfill \mathrm{PPF}(x_C,x_i)=[& \angle (n_C,\Delta x_{C,i}),\angle (n_i,\Delta x_{C,i}),\hfill \\ & \angle (n_C,n_i),{‖\Delta x_{C,i}‖}_2]\hfill \end{array}$$

(3)

where $n_C$ and $n_i$ are the normal vectors of points $x_C$ and $x_i$, respectively, which can be estimated using neighboring points. $\angle$ denotes the angle between vectors.

Using the feature extraction module shown in Figure 3, the point cloud geometric feature descriptors $F_P$ are refined into the point cloud feature descriptors ${\tilde{F}}_P$ with dimensions $N\times C$, $N=N_1,N_2$. Based on the transformation relationships $T_I^X$ and $T_I^Y$ between the point cloud and the image coordinate system, the corresponding pixel points in $I_X$ and $I_Y$ are selected, and the 32-dimensional ORB [26] feature descriptors $F_O$ for each pixel are computed. The ORB feature descriptor combines the FAST keypoint detector and the BRIEF descriptor. The ORB feature descriptor is a 256-bit binary string which is divided into 32 bytes (8 bits per group) to obtain a 32-dimensional image feature descriptor $F_O$. The point cloud feature descriptor $[{\tilde{F}}_P^X,{\tilde{F}}_P^Y]$ is then concatenated with $[F_O^X,F_O^Y]$ to generate fused features for both source and target point clouds with dimensions $N_1\times (C+32)$ and $N_2\times (C+32)$, respectively. Taking the point cloud feature descriptor dimension C = 96 as an example, the two concatenated features form a 128-dimensional hybrid feature vector. The feature histogram of a single point is shown in Figure 4.

2.2. Transformer

The Transformer structure primarily consists of self-attention and cross-attention layers, iterated $N_T$ times. The structure of the attention layer is shown in Figure 5. For self-attention layers, the input features $f_i=f_j$ are the features of the same frame, representing the attention operation of the features on themselves. For cross-attention layers, $f_i$ and $f_j$ represent the features of different frames (the order depends on the direction of the cross-attention). The input vectors of the attention layer are typically referred to as queries, keys, and values. The attention weights are calculated based on the dot product of the query vector Q with the corresponding key vector K for each value V. Information is retrieved from the value vector V according to the attention weights computed from the dot product of Q and each corresponding key vector K, where softmax denotes the normalized exponential function that converts the dot products into probability distributions. Formally, the attention layer is represented by Equation (4). This study utilizes a modified Linear Transformer, as presented in [29], illustrated in Figure 5.

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(Q{K}^{\mathrm{T}})V$$

(4)

2.3. Matching and Weighted SVD

Then, the Matching Matrix $M$ is computed using the Sinkhorn normalization algorithm [30]. The input to Sinkhorn normalization is the initial confidence matrix derived from the weighted dot product of source point cloud features and target point cloud features. The Sinkhorn normalization algorithm iteratively adjusts the initial confidence matrix into a doubly stochastic matrix (where the sums of rows and columns are all 1) while preserving the structural information of the original matrix. This is achieved by minimizing an objective function with an additional entropy term. Based on the Sinkhorn-normalized matrix, virtual correspondence points are computed in Equations (5) and (6).

$$M=\mathrm{Sinkorn}({\displaystyle \frac{1}{\tau }}〈F_{\mathrm{Trans}}^X,F_{\mathrm{Trans}}^Y〉)$$

(5)

$${\widehat{y}}_j={\displaystyle \sum _{k=1}^{N_2}{M}_{jk}{y}_k},(j=0,\cdots N_1,k=0,\cdots N_2)$$

(6)

where $y_k$ is the kth point in the target point cloud and ${\widehat{y}}_j$ is the virtual matching point corresponding to the j-th point in the source point cloud.

Finally, the pose transformation matrix is solved using weighted SVD, as shown in Equation (7).

$$R,t=\underset{R,t}{\mathrm{argmin}}{\displaystyle \sum _{j=1}^{N_2}{\omega }_j{‖R{x}_j+t-{\widehat{y}}_j‖}_2}$$

(7)

where ${\omega }_j$ is the weight of the matching point pair $\{x_j,{\widehat{y}}_j\}$, defined as ${\omega }_j={\displaystyle {\sum }_j^{N_2}{M}_{jk}}$.

The loss function of the network refers to [27] and consists of two parts, $L_T$ and $L_M$, as shown in Equations (8)–(10). Here, $R_{gt}$ represents the rotation matrix label between the source point cloud and the target point cloud, and $R$ denotes the predicted rotation matrix. Similarly, $t_{gt}$ represents the translation vector label between the source point cloud and the target point cloud, while $t$ denotes the predicted translation vector. In this paper, following the empirical findings from Reference [27], $\lambda$ is set to 0.01 to reduce training difficulty of the network.

$$L_T={\displaystyle \frac{1}{N_2}}{\displaystyle \sum _j^{N_2}\left|(R_{gt}{x}_j+t_{gt})-(R{x}_j+t)\right|}$$

(8)

$$L_M=-{\displaystyle \frac{1}{N_1}}{\displaystyle \sum _j^{N_1}{\displaystyle \sum _k^{N_2}{M}_{jk}}}-{\displaystyle \frac{1}{N_1}}{\displaystyle \sum _k^{N_2}{\displaystyle \sum _j^{N_1}{M}_{jk}}}$$

(9)

$$L=L_T+\lambda L_M$$

(10)

3. Experiments

3.1. Dataset and Experimental Setup

Training the proposed method requires a large amount of registered point cloud and image data of spacecraft, which existing public datasets cannot meet. To construct the dataset, we collected 93 satellite CAD models and used the ray tracing method to simulate optical images and depth maps, generating corresponding point clouds from various viewpoints. The model sizes are randomly distributed between 2 m and 8 m. When generating simulated point clouds from depth maps, the depth data were preserved with millimeter-level precision. Some examples of the CAD model optical images and point cloud data are shown in Figure 6. For each model, we generated a sequence of 90 frames of data under fly-around observations, with the orbit shown in Figure 7. To ensure the network’s generalization capability, in each sequence, the target is assigned a random initial rotation within the range of three axes [0, 90°]. Additionally, for each frame, random rotations of [0.75°, 1.25°] and random translations of [0, 0.5 m] are applied relative to the previous frame. Additionally, we randomly added noise, which is sampled from N(0, 0.01 m), and clipped to the range [−0.05, 0.05] on each axis. We selected the current frame and the 5th subsequent frame to form a pair of point clouds. Finally, we selected simulation data from 60 models as the training set and data from 33 models as the validation set, resulting in 6300 training and 2700 validation data points.

Some hyperparameters in the network include the following: the number of point cloud neighbors set to 64, the dimension of the point cloud feature descriptors set to 96, Transformer iterations set to 3, and the pose iteration count set to 3. The point clouds were downsampled to 1024 × 3. The optimizer employed during the training process of the network is “Adam”, with an initial learning rate parameter set to 0.001. The batch size of the training is 8. The criterion for stopping training is reaching 500 epochs or the scheduler’s learning rate falling below 10⁻⁷.

To evaluate the performance, we use the anisotropic mean absolute error (MAE(R), MAE(t)) and the isotropic errors (Error(R), Error(t)) [27] as the quantitative evaluation metrics. Among them, the definition of anisotropic errors is given in Equations (11) and (12), and the definition of isotropic errors is provided in Equations (13) and (14). In Equations (11)–(14), ${\mathrm{Euler}}_{zyx}$ denotes the Euler angles in the Z-Y-X rotation sequence, $\mathrm{tr}(\cdot )$ represents the trace of a matrix, the ground-truth translation vector is labeled as $t_{gt}=(t_{gx},t_{gy},t_{gz})$, and the predicted translation vector is labeled as $t=(t_x,t_y,t_z)$.

$$\mathrm{MAE}\left(R\right)={\displaystyle \frac{1}{3}}{\displaystyle \sum _{i=1}^3\left|{\mathrm{Euler}}_{zyx}\left(R_{gt}\right)-{\mathrm{Euler}}_{zyx}\left(R\right)\right|}$$

(11)

$$\mathrm{MAE}\left(t\right)={\displaystyle \frac{1}{3}}(\left|t_{gx}-t_x\right|+\left|t_{gy}-t_y\right|+\left|t_{gz}-t_z\right|)$$

(12)

$$\mathrm{Error}\left(R\right)=\arccos \left({\displaystyle \frac{\mathrm{tr}\left({R_{gt}}^{-1}R\right)-1}{2}}\right)$$

(13)

$$\mathrm{Error}\left(t\right)=\sqrt{{(t_{gx}-t_x)}^2+{(t_{gy}-t_y)}^2+{(t_{gz}-t_z)}^2}$$

(14)

The training process visualization of the proposed network is illustrated in Figure 8.

3.2. Experimental Results and Analysis

We compared our method with several traditional methods, including ICP [14], FGR [31], SAC-IA [32], and SAC-IA + ICP, as well as several learning-based methods, including PointNetLK [33], IDAM [24], and RPM-Net [27]. Figure 9 shows some visualization results of point cloud registration from our method. The matching results in the figure visually demonstrate the accuracy of the proposed method. The quantitative results in

Table 2. Quantitative results of different methods.

Method	MAE (Anisotropic)		Error (Isotropic)		Time (ms)
	R/°	t/m	R/°	t/m
ICP	16.09	0.227	37.48	0.459	76
FGR	2.47	0.048	4.60	0.092	680
SAC-IA	2.08	0.040	4.04	0.078	257
SAC-IA + ICP	1.59	0.029	3.01	0.058	447
PointNetLK	12.22	0.479	23.06	0.928	66
IDAM	1.41	0.073	2.85	0.146	27
RPM-Net	0.91	0.024	1.83	0.050	106
Ours	0.84	0.022	1.72	0.045	68

demonstrate that the proposed method achieves state-of-the-art performance, with a mean angular error (MAE) of 0.84° in rotation (R) and 0.022 m in translation (t), as well as an isotropic error of 1.72° in rotation (Error (R)) and 0.045 m in translation (Error (t)). Compared to single-modality point cloud-based methods, our multi-modal feature fusion approach significantly improves estimation accuracy. Specifically, traditional geometric methods (e.g., ICP and FGR) are limited by their sensitivity to initial values and reliance on parameter tuning. Among deep learning methods, our approach builds upon RPM-Net’s robust feature extraction framework and further enhances pose estimation by introducing an image feature collaboration mechanism and transformer attention module. The largest performance gap occurs in rotation estimation, where our method reduces RPM-Net’s error by 7.7%, attributed to the transformer’s ability to refine cross-modal feature correlations. Additionally, our method maintains computational efficiency, outperforming most traditional methods in speed while remaining competitive with advanced deep learning approaches.

During training, the frame interval between the source point cloud and the target point cloud was set to five frames. However, in practical applications, the frame interval may vary, with larger intervals indicating greater differences between the source and target point clouds. To investigate the performance of the proposed method under different frame intervals, we conducted comparative experiments with varying frame intervals. The experimental results are shown in Figure 10. The left vertical axis of Figure 10 corresponds to MAE (R), while the right vertical axis corresponds to MAE (t). The results indicate that as the frame interval increases, both rotation and translation estimation results degrade, with rotation estimation deteriorating more rapidly. This may be because image feature descriptors perform well when the rotation range is small, effectively guiding point cloud matching. However, for large rotation angles, the guidance provided by the image features weakens, leading to a decline in point cloud matching accuracy. The above robustness analysis reveals the method’s stability boundaries within a 30-frame interval. For large-scale scenarios, future improvements could incorporate distributed optimization algorithms to enhance the pose estimation method [34,35].

3.3. Ablation Studies

To verify the effectiveness of incorporating ORB features and the Transformer structure, ablation experiments were conducted, with the results shown in

Table 3. Quantitative results of the POCPE-Net with different components.

Method	MAE (Anisotropic)		Error (Isotropic)
	R/°	t/m	R/°	t/m
Baseline	1.13	0.058	2.28	0.115
Baseline + Trans	0.86	0.023	1.77	0.047
POCPE-Net (Baseline + Trans + ORB)	0.84	0.022	1.72	0.045

. The quantitative results show that the Transformer’s attention mechanism significantly improves accuracy, indicating its superior ability to extract the matching relationships of collaborative features compared to conventional methods, such as brute-force feature matching based on Euclidean distance. Moreover, the introduction of ORB feature descriptors enables using image feature matching relationships to partially supervise the point cloud matching results, further enhancing matching accuracy. The results demonstrate that adding ORB features and the Transformer structure has a positive impact on improving point cloud matching accuracy.

3.4. Validation

To validate the practical application value of the proposed method, we constructed a ground-testing environment. The experimental setup is shown in Figure 11.

We fastened the LiDAR and camera equipment to ensure a stable pairing between point clouds and optical images and varied the satellite model’s pose for measurements.

Measurements were conducted by varying the pose of the satellite model. The experimental preprocessing included the following: (1) Acquiring multiple checkerboard calibration images to obtain the camera intrinsic matrix using the calibration method. (2) Determining the extrinsic matrix of the LiDAR–camera system via the Perspective-n-Point (PnP) algorithm to establish ground truth. (3) After background filtering of the raw data (as shown in Figure 12), clean target foreground point cloud image sequences were obtained (with continuous frame processing results presented in Figure 13).

The inter-frame pose transformations were predicted using the proposed method, and the point cloud matching results are shown in Figure 14. Since traditional methods are overly dependent on initial values and exhibit unstable accuracy, we compare our approach only with learning-based methods. The quantitative evaluation results are presented in

Table 4. Quantitative results of pose estimation based on measured data.

Method	MAE (Anisotropic)		Error (Isotropic)
	R/°	t/m	R/°	t/m
PointNetLK	14.79	0.068	36.63	0.145
IDAM(GNN)	3.66	0.104	7.76	0.051
RPM-Net	1.64	0.020	3.39	0.039
POCPE-Net (ours)	0.97	0.015	1.77	0.028

, demonstrating that the proposed method also maintains high accuracy on real measurement data.

4. Conclusions

To address the accuracy degradation of existing pose estimation methods in noncooperative target scenarios, this paper proposes a novel spacecraft pose estimation method based on the synergy of point cloud and optical image features. The method integrates ORB feature descriptors with point cloud feature descriptors, aiming to reduce point cloud mismatches under the guidance of a transformer mechanism, thereby improving pose estimation accuracy. Comparative experiments conducted on a point cloud–optical image dataset demonstrate that our method achieves superior performance, with a mean angular error (MAE) of 0.84° in rotation (R) and 0.022 m in translation (t) on the validation set, outperforming existing point cloud registration methods. Robustness analysis reveals the method’s stability boundaries within a 30-frame interval. Ablation studies validate the effectiveness of both ORB features and the transformer layer. Ground validation platform tests confirm the method’s practical applicability, offering a new solution for enhancing pose estimation accuracy of noncooperative space targets. By fusing optical and point cloud features, this work advances LiDAR–camera joint measurement techniques and supports on-orbit servicing missions. Due to the prerequisite of point cloud and image data pre-registration, future research could further investigate multi-source data alignment and fusion methods, along with comprehensive error analysis.