A Large-Scale Invariant Matching Method Based on DeepSpace-ScaleNet for Small Celestial Body Exploration

Abstract

Small Celestial Body (SCB) image matching is essential for deep space exploration missions. In this paper, a large-scale invariant method is proposed to improve the matching accuracy of SCB images under large-scale variations. Specifically, we designed a novel network named DeepSpace-ScaleNet, which employs an attention mechanism for estimating the scale ratio to overcome the significant variation between two images. Firstly, the Global Attention-DenseASPP (GA-DenseASPP) module is proposed to refine feature extraction in deep space backgrounds. Secondly, the Correlation-Aware Distribution Predictor (CADP) module is built to capture the connections between correlation maps and improve the accuracy of the scale distribution estimation. To the best of our knowledge, this is the first work to explore large-scale SCB image matching using Transformer-based neural networks rather than traditional handcrafted feature descriptors. We also analysed the effects of different scale and illumination changes on SCB image matching in the experiment. To train the network and verify its effectiveness, we created a simulation dataset containing light variations and scale variations named Virtual SCB Dataset. Experimental results show that the DeepSpace-ScaleNet achieves a current state-of-the-art SCB image scale estimation performance. It also shows the best accuracy and robustness in image matching and relative pose estimation.

1. Introduction

With the development of deep space exploration technology, future Small Celestial Body (SCB) probes require highly autonomous detection capability for intelligent exploration in deep space [1,2,3]. The extraction and matching of image features are the basis and prerequisite for autonomous visual navigation, and their accuracy directly affects the accuracy and performance of navigation.

Before landing, the probes need to observe the SCB from orbit, take photographs of the SCB’s surface and extract image features manually or automatically. Position estimation and navigation are achieved by matching these reference image features during the landing phase [4,5]. Images taken during the comet characterisation and close observation phases were applied to the landing phase, as shown in Figure A1 (in Appendix A) of the Rosetta mission [6]. However, deep space differs from ground-based scenes because the altitude changes dramatically from orbiting to landing. There are two difficulties with the image matching of the SCB: (1) The images have enormous scale feature variations. For example, in the Hayabusa2 mission, the SCB surface image taken at 20 km altitude is very different from 500 m altitude [5]. When the SCB is photographed from a great distance, the SCB occupies only a few tens of pixels in the whole image. (2) In the process of SCB exploration, the illumination of the reference and navigation images varies considerably. Variations in image features such as scale and illumination affect the robustness of the matching, further affecting the accuracy of the navigation.

Existing matching methods for SCB images focus on traditional methods such as feature points and interest regions [7], line features [8] and crater matching [9,10,11]. These methods work most of the time, but the accuracy will decrease if the scale and lighting are very different. In many computer vision tasks, some feature detection and matching methods based on deep learning have shown their effectiveness [12,13]. Unfortunately, the above methods are not designed for extreme scale changes, so they perform poorly when the images change massively during deep space exploration [14,15]. For the problem of scale changes, the multi-scale feature extraction method based on image pyramids can extract the feature levels in the neighbourhood to find the correspondence of image pairs [15,16,17]. However, the image pyramid layers are limited. For two images with large-scale variations, it is possible that the extracted multi-scale local features are not correlated, resulting in the correct correspondence not being obtained [15]. Some methods of estimating the scale ratio between two images have been proposed to solve this problem and improve the matching accuracy [14,15,18,19]. The main principle is to adjust the unrelated scale levels to be related by first estimating the scale ratio and then matching the images. However, the largest scale ranges that these methods can address are also unsuitable for challenging SCB scenarios.

To overcome these limitations, this paper proposes the DeepSpace-ScaleNet and integrates it into our proposed large-scale invariant matching method. DeepSpace-ScaleNet is a scale estimation network adapted to the SCB scene to improve the effect of image matching more effectively. The DeepSpace-ScaleNet architecture includes two modules, GA-DenseASPP module and CADP module. The former incorporates channel and spatial information through a global attention mechanism. Meanwhile, the dense connectivity of ASPP is used to extract features for large-scale scenes. The latter uses the Transformer to capture the connections between two correlation maps and improve the accuracy of the scale distribution estimation. With these two modules, we have improved the overall performance of the algorithm. Our proposed approach effectively addresses the critical shortcomings of state-of-the-art methods. As shown in Figure 1, it can significantly improve the performance of local features under large-scale changes in images.

The main contributions of this paper are as follows:

(1)

As far as we know, we are the first to propose a solution to the problem of matching images of SCB with large-scale variations.

(2)

We have creatively used scale estimation networks in SCB image matching and designed DeepSpace-ScaleNet by using the architecture of Transformer. This is unprecedented in previous SCB matching methods.

(3)

We designed the novel GA-DenseASPP module and CADP module in DeepSpace-ScaleNet, which can improve the performance of the algorithm.

(4)

We created the first simulated SCB dataset named Virtual SCB Dataset in the field, which can be applied to image matching, relative localization and more.

The rest of this article is organized as follows. In Section 2, we describe the related work image local feature detection and multi-scale modules, especially in deep space. Section 3 describes the SCB image matching method, including DeepSpace-ScaleNet. Section 4 introduces the Virtual SCB Dataset we created and applied to this work. Section 5 and Section 6 evaluate the experimental simulation and give the conclusions of this paper, respectively.

2. Related Work

Image local features widely used in the deep space scene can provide important reference information for SCB navigation, yet they also face significant challenges. One of the major challenges is matching images with large-scale variations because the surface appearance at lower altitudes differs significantly from those at high altitudes.

Table A1. A summary of some representative technical methods and our methods.

Research Field in Deepspace Exploration	Related Work	The Advantages	The Disadvantages	Our Contributions (Proposing a New Method)
Image Local Feature Extraction And Matching	SIFT [24] + FLANN [27]	A robust performance to scale changes	Not available when scene lighting changes dramatically	More robust to changes in light
	SuperPoint [12] + SuperGlue [13]	A robust performance to changes in light	Inability to adapt to large-scale changes	More robust to large-scale changes
	Pattern matching based on meteorite craters, etc. [9,10,11]	Suitable for SCB Scenarios	Deactivated when there are no craters in the field of view	No requirement for craters, rocks, etc., in the field of view, and a wide range of application
Estimation Of Image Scale Ratio	ScaleNet [14]	Valid for most scale changes	Low accuracy in deep space scenes	Higher accuracy of scale estimation in deep space scenes
Multi-scale Modules	Image feature pyramids [16,17,41,42,43,44]	Ability to overcome scale changes	Limits to computational efficiency	More lightweight
	ASPP [50,51,52,53]	Ability to extract multi-scale features in the network	Limited by deeper features	Better extraction of large-scale scene features

in Appendix B briefly summarises some representative technical approaches and our method.

2.1. Local Feature Extraction and Matching

Feature-based pipelines generally follow feature detection, description and matching [20,21]. Some traditional manual feature detection algorithms such as Harris [22] and FAST [23] are based on corner point detection, which aligns with an intuitive understanding of features. However, these methods cannot be applied when large-scale changes exist. The blob speckle-based SIFT [24] and its modified version SURF [25] introduce the DOG-based scale space theory and solve the problems of target rotation, scaling and lighting changes to some extent. After the feature spot is detected, the area around the feature needs to be described with a feature vector. The SIFT descriptor describes the feature with a 128-dimensional vector, while ORB [26] uses a binary, which is much faster. The most essential feature point matching method is Brute-force (BF) matching, and later the Flann-based matcher was utilized for efficiency [27]. Deep learning-based feature detection methods have also been proposed in recent years. Key. Net extracts manual feature points over multiple-scale layers via CNN to solve the problem of the lack of true annotation of features [28]. LF-Net proposes a new deep feature extraction framework and trains the method by known relative poses and correspondences [29]. SuperPoint is a feature point detection and descriptor extraction method based on self-supervised training [12]. It adopts a self-learning method to avoid extensive manual annotations and has stronger robustness to seasons, lighting, etc. SuperGlue uses graph neural networks to match features [13]. Thanks to its added attention to contextual aggregation mechanisms, it can learn the geometric transformations and conventional a priori knowledge of the 3D world. LOFTR proposes a coarse-to-fine local dense feature matching method that achieves better results in weakly textured regions [30].

Currently, there are fewer methods specifically for SCB image feature detection. The existing methods are generally based on the above standard local feature extraction and matching approaches for navigation and surface reconstruction. Due to the lack of training data, most researchers still use traditional manual features. For example, Takeishi, N. et al. evaluated the performance of several common feature detection and description methods and found that the performance of SIFT is acceptable [7]. Nonetheless, SIFT also fails to match when large-scale changes occur in the image [5]. In addition to interest points, more advanced features such as line features [8,31], shadow features [32] and craters [9,10,11] on the surface of SCB are also used for SCB navigation. The primary methods for SCB image matching are feature point-based and pattern matching [33]. The feature point-based approach provides a straightforward and efficient matching process through the distance between the two descriptors [34,35]. Pattern matching uses the image land markers extracted from the observed images to match the features with the land markers in the reference terrain library [33,36]. The craters are extracted from the images taken during the landing of the probe, and pattern matching is performed with a reference database based on features such as the crater’s edge, diameter, relative distance and angle to obtain information on the absolute position and attitude of the probe [37,38,39,40]. However, many landmark features seen in high orbits cannot be observed again in low orbits because the altitude of the space probe has changed. For example, as the probe descends, the number of craters in view decreases. Therefore, it is necessary to propose a matching method suitable for large-scale changes in SCB images.

2.2. Image Scale Estimation

To address the image matching problem caused by large-scale changes, Zhou et al. proposed a scale-invariant image matching approach SLM-BOF, inspired by the scale space theory, to tackle the extensive scale changes [18]. In 2021, Fu et al. introduced a learning-based network to estimate the scale ratio, which has much higher accuracy than SLM-BOF [15]. By contrast, ScaleNet outputs a scale distribution in logarithmic space rather than a regressed single scale ratio, which helps the network converge to a reliable model [14]. In addition, the image box embedding that approximates the visible surface overlap measures is used to estimate the scale ratio between two images [19]. Unfortunately, the above methods can only estimate scale ratios within restricted limits due to the gaping differences between deep space and ground scenes. However, they cannot satisfy the requirements of SCB missions. To overcome these limitations, we proposed DeepSpace-ScaleNet better to achieve the large-scale invariant matching of SCB images.

2.3. Multi-Scale Modules

How to capture multi-scale context is an important issue in computer vision tasks. In deep space, as shown in Figure 1, the SCB may occupy only a few or even one pixel in the image during the fly-around phase. However, it may occupy half or more image pixels during the landing phase. Therefore, this process requires special modules to capture the multi-scale context.

Image pyramids are the typical method to capture multi-scale contexts by fusing multiple resized input images [16,17,41,42,43,44]. However, the major limitation of such networks is that they consume vast amounts of GPU memory, which is almost impractical for lightweight SCB probes.

Networks with encoder-decoder architecture fuse features of different resolutions produced by different layers [45,46,47,48]. In the encoder, the resolution of the input image is gradually reduced, and the deep feature maps have larger receptive fields. In the decoder, the resolution of the image is gradually recovered, and the shallow feature maps have more object details. PSPNet proposed the pyramid pooling module (PPM) to capture features at several grid scales [49]. The ASPP and its follow-ups expand the receptive field by parallel atrous convolution layers of different dilation rates [50,51,52,53]. Deformable convolution is proposed in [54,55] to focus on pertinent image regions. In [56], they proposed Receptive Field Block (RFB), which combines a multi-branch convolution layer and dilated pooling or convolution layer, imitating the structure of RF in human visual systems. A pyramidally attended feature extraction module (PAFE) was constructed to make multi-scale features more attention-enhanced in [57]. How to design multi-scale modules for deep space imagery is the question to be considered.

3. Methods

As shown in Figure 1, extracting critical points between two images with different scales is challenging, resulting in poor performance of image matching methods. The deep space scene is a typical example, where the number of pixels occupied by the SCB is different between the orbiting phase and the landing phase of the probe. To solve the problem, we proposed a large-scale invariant method for SCB image matching and designed the DeepSpace-ScaleNet for estimating the scale of the SCB images photographed in the two phases mentioned above.

3.1. Large-Scale Invariant Method for SCB Image Matching

The percentage of SCB pixels in the image field varies a lot when the probes are at different orbital altitudes of the SCB. Traditional image pyramid-based multi-scale feature extraction methods consider only a few adjacent scale layers [15]. It leads to the following problems: on the one hand, the feature descriptors generated from the two feature pyramids are too different to correspond; on the other hand, multi-scale feature extraction methods may cause the SCB to be almost drowned out by the background noise in the upper-level features map, which is also a significant challenge in computer vision.

Consequently, our approach suggests predicting the scale ratio before image matching to overcome the above limitations. As in Figure 2, the scale ratio between Images A and B is predicted by DeepSpace-ScaleNet. Both images were resized according to this ratio before subsequent images were matched. For example, the scale ratio for Images A and B is S. When S is greater than 1, it means that the size of the SCB in Image A is S times larger than in Image B. Therefore, we upsample Image B by a ratio of S and match it with Image A. When S is less than 1, the image of Image A is sampled on the image by a ratio of S.

3.2. Scale Distributions

ScaleNet formulates the scale estimation as a probability distribution prediction and experimentally demonstrates the effectiveness of probability distribution predictions relative to regression models [14].

Inspired by it, our DeepSpace-ScaleNet outputs a probability distribution of length $L$. Given two images, the final scale ratio is calculated by the following. Estimate the scale distribution $s_{A\to B}^i$ weighted by the predicted scale probabilities $p_i$ from ImageA to ImageB. The same operation is performed for ImageB to ImageA. Equation (1) computes the scale ratios ${\stackrel{\rightharpoonup }{S}}_{A\to B}$ and ${\stackrel{\rightharpoonup }{S}}_{B\to A}$, and they are inverse. We combine them as ${\widehat{S}}_{A\to B}$ and calculate the final scale factor between the images $S_{A\to B}$.

$${\stackrel{\rightharpoonup }{S}}_{A\to B}={\displaystyle \sum _{i=0}^{L-1}{p}_i}\cdot {\log }_{\sigma }(s_{A\to B}^i), {\stackrel{\rightharpoonup }{S}}_{B\to A}={\displaystyle \sum _{i=0}^{L-1}{p}_i}\cdot {\log }_{\sigma }(s_{B\to A}^i)$$

(1)

$${\widehat{S}}_{A\to B}=\frac{{\stackrel{\rightharpoonup }{S}}_{A\to B}-{\stackrel{\rightharpoonup }{S}}_{B\to A}}{2}$$

(2)

$$S_{A\to B}={\sigma }^{{\widehat{S}}_{A\to B}}$$

(3)

3.3. DeepSpace-ScaleNet

To tackle the problems mentioned in the abstract, we propose DeepSpace Scalenet, which fills the gap of scale estimation for deep space scenes. Compared with several state-of-the-art methods via comprehensive experiments, Deepspace-Scalenet can estimate more accurate scales.

3.3.1. Deepspace-Scalenet Architecture

As shown in Figure 3, DeepSpace-ScaleNet accepts ImageA and ImageB as input. The features are first extracted by a pre-trained model and then passed through the GA-DenseASPP module. Specifically, the global attention mechanism fuses the channel and spatial information, and the multi-scale information is obtained by atrous convolution. After that, the refined features are processed by the method [58] to obtain self-correlation maps, $C_A$, $C_B$ and cross-correlation maps $C_{A-B}$. Finally, the three correlation maps are concatenated in the channel dimension and fed into the CADP to output scale distribution.

As shown in Figure 3, our proposed method uses a similar architecture as ScaleNet [14]. However, it replaces the modules of ScaleNet with more efficient ones, making the network more applicable to multi-scale scenes such as deep space backgrounds. We introduce two Crucial modules: Global Attention-DenseASPP (GA-DenseASPP) and Correlation-aware Distribution Predictor (CADP).

3.3.2. Global Attention-DenseASPP (GA-DenseASPP)

The attention mechanism [59,60,61,62,63] has been widely used in computer vision tasks such as feature extraction and semantic segmentation. The global attention mechanism [59] captures both channel and spatial information and magnifies salient cross-dimension receptive regions. To extract multi-scale information accurately, we proposed GA-DenseASPP.

As shown in Figure 4, we take the feature map $F\in R_{512\times \mathrm{H}\times \mathrm{W}}$ as input, where $512$, $\mathrm{H}$, $\mathrm{W}$ indicates the channels, height and width of $F$, respectively. Then, the GAM [59] processes the feature map and refines it through the channel attention gate and spatial attention gate, respectively, which can better explore the channel and spatial relationship of the feature map. The above-mentioned process is formulated in Equations (4) and (5).

$$F^{\prime }={\mathrm{M}}_{\mathrm{c}}\left(F\right)\otimes F$$

(4)

$$F^{″}={\mathrm{M}}_{\mathrm{s}}\left(F^{\prime }\right)\otimes F^{\prime }$$

(5)

where $F^{\prime }$ and $F^{″}$ are channel-refined and spatial-refined feature maps, respectively; ${\mathrm{M}}_{\mathrm{c}}$ and ${\mathrm{M}}_{\mathrm{s}}$ are attention maps of the channel and spatial module, respectively; $\otimes$ denotes element-wise multiplication.

Finally, the final refined feature map is fed into Dense Atrous Spatial Pyramid Pooling (DenseASPP). Dense Atrous Spatial Pyramid Pooling consists of a cascade of atrous convolution layers that can encode multi-scale information from feature maps through dense connections.

Compared with the ASPP used in ScaleNet, GA-DenseASPP is more efficient for multi-scale scenes such as deep space, and can accurately extract information from features of different scales.

The feature map extracted by the pre-trained model is fused with spatial and channel information by GAM. Then, it is fed into a cascade of atrous convolution layers of DenseASPP.

The attention mechanism is added to DeepSpace-ScaleNet as an effective module. We use Grad-CAM [64] to visualize the feature maps in Figure 5. It takes the gradient of the predicted scale ratios flowing into the attention module to produce feature visualizations, highlighting regions that contribute highly to scale estimation. As shown in Figure 5, the network pays more attention to the visible textures on the SCB surface and selectively ignores the deep space background and the invisible areas of the SCB surface. The attention module enables the network to extract image features that are useful for prediction and helps it output a more accurate scale distribution.

3.3.3. Correlation-aware Distribution Predictor (CADP)

The fully connected layers cannot capture the inner relationship between correlation maps, so this may lead to inaccurate scale estimation. To solve this problem, we proposed CADP module. It consists of a few stacked Transformer downsample blocks, an average pooling layer and a set of fully connected layers. The efficiency of the Transformer has been demonstrated in many computer vision tasks, as its multi-head self-attention layer can integrate information from the entire image, even at the lowest layers [51].

As shown in Figure 6, CADP takes the concatenated correlation map as input, indicating the channels, height and width $C_{cat}$, respectively. Then, $C_{cat}$ is fed to Transformer downsample blocks to integrate the inner relationship in correlation maps and reduce its shape gradually. After that, the downsampled correlation map is forward fed into an average pooling layer, and its height and width are reduced to 1. Finally, it is flattened and processed by a set of fully connected layers to predict scale distribution.

The Transformer downsample block is based on the BoTNet [65]. The number of Transformer downsample blocks is N. The block consists of two 1 × 1 convolution layers, a multi-head self-attention layer and ends with a 3 × 3 convolution layer. The multi-head self-attention layer can capture long-range dependencies across the entire correlation maps, but it requires a lot of computational resources to train. For a trade-off between the accuracy of scale estimation and computational efficiency of multi-head self-attention, we use 1 × 1 convolution layer to reduce the channel of the concatenated correlation map. For the same reason as above, we set the stride of the 3 × 3 convolution layer to 2. Each convolution layer is followed by a batch normalization layer [66] and an activation layer.

Compared with only fully connected layers in ScaleNet, CADP is more correlation-aware. Our experiments demonstrate that it is an effective and efficient module to capture the relationship between correlation maps.

3.3.4. Loss Function

We use the Kullback–Leibler divergence loss function to measure the distance between predicted scale distribution $P_{A\to B}$ and ground-truth distribution $P^{gt}{}_{A\to B}$.

$$L\mathrm{oss}(A,B)=KL(P_{A\to B},P^{gt}{}_{A\to B})$$

(6)

3.3.5. Learning Rate and Optimizer

The learning rate plays an essential role in the model’s training as a necessary parameter of the neural network. The model needs a large learning rate in the first few epochs to perform a sharper and faster gradient descent. As the epoch grows, the model needs a small learning rate to stabilize the convergence process. The decay ratio can control the learning rate to decrease gradually with the growth of the epoch.

Therefore, setting an appropriate learning rate and decay ratio determines the accuracy of the model prediction. We set the initial learning rate to ${10}^{-4}$, and the learning rate decays when it reaches 5, 15, 25 and 35 with a decay ratio of 0.1.

As for the optimizer, we chose Stochastic gradient descent (SGD). Because the SGD is more suitable than Adam for extended experiments, another advantage of SGD is that it reduces the computational burden remarkably and speeds up the iterative process of each epoch.

4. Virtual SCB Dataset

Considering that there are only images from the SCB exploration mission, there is a lack of public datasets available for feature detection and scale estimation. We use Blender to generate the SCB image dataset. Blender is an open-source 3D simulation engine that supports the use of python scripts. The three-dimensional SCB models used in this paper are 67P/Churyumov–Gerasimenko [67] and Ryugu [68]. The specific process is shown in Figure 7. The SCB’s centre of mass is used as the origin of the world coordinate system. Four light sources are placed near the SCB to simulate the camera’s images in different sunshine conditions. Images are taken when the camera is placed at different orbital heights in order to take images at different scales. The camera’s Z-axis always points to the SCB to ensure that the SCB is in the centre of the image. The camera is moved at each height, and 51 images are taken to ensure sufficient overlap between adjacent images. The resolution of the camera is set to 1024 × 1024 pixels.

Figure 8 shows the real 67 P image (left) and our simulated image (right). Compared with the real image, the texture and shadow of the simulated image are very realistic and can reflect the surface characteristics of the real image.

Figure 9a–d shows simulated images of SCB with different illumination from the same viewing angle; (e–h) are SCB shot at different altitudes (4, 10, 20, 40 km). The observation shows that when the illumination and scale change drastically, the surface characteristics of the SCB change significantly.

The altitude range of our generated SCB images is (5150) km. A set of images was generated at 5 km intervals, with 51 SCB images in each set. When we generated image pairs, each image was paired with one to three images from other altitudes with adjacent shooting angles. As shown in

Table 1. Virtual SCB Dataset.

Scale Ratio Range	All	$1 < \mathbf{s}\le 5 \mathbf{or} 1 < \frac{1}{\mathit{s}} \le 5$	$5 < \mathbf{s} \le 10 \mathbf{or} 5 < \frac{1}{\mathit{s}} \le 10$	$10 < \mathbf{s}\le 15 \mathbf{or} 10 < \frac{1}{\mathit{s}} \le 15$	$15 < \mathbf{s}\le 20 \mathbf{or} 15 < \frac{1}{\mathit{s}} \le 20$
Number of Pairs	9145	2276	3349	2265	755
Proportion	100%	30.4%	36.6%	24.8%	8.3%

, we generated 9145 pairs of images and divided the dataset into four scale ratio ranges. During the training and testing of DeepSpace-ScaleNet, 6655 pairs were used for training, 864 pairs for validation and 1626 pairs for testing. To estimate the impact of lighting on matching, we divide the dataset into a FixedLight Dataset and a VariableLight Dataset according to whether the paired lighting conditions change.

5. Experiments and Results

5.1. Implementation Details

DeepSpace-ScaleNet accepts images of arbitrary size as input. In this work, these images are resized to 240 × 240 and then fed into the network for training. Considering the true scale distribution of our dataset in the range [0.05, 20], we set L = 13 bins and σ = 2.

The dilation rate of DenseASPP is set to {1, 6, 12, 18}, and the number N of the Transformer Downsample Module is set to 2.

Our data augmentation includes flip, rotation and colour augmentation. All our experiments are run on RTX 3080 GPU. It uses a pre-trained VGG model as the feature extractor. The batch size is set to 48. The network is trained at about 50 epochs. The whole training process takes about 1 h.

5.2. Image Scale Estimation

In this section, we compare DeepSpace-ScaleNet with other state-of-the-art methods to demonstrate the effectiveness of our method. We use a learning-based model ScaleNet [14], and a physics-based method named BoundingBoxes Algorithm.

In contrast to the relative scale estimation problem in ScaleNet [14], a large proportion of the SCB images are of a prominent target on a black deep-space background. To verify whether detecting only the target bounding box is an effective estimator of the scale factor, we use Rembg [68] (a phenomenal artificial intelligence-based background remover). It removes the deep space background, segments foreground objects and then detects the bounding box to estimate the image scale difference by comparing the change in the scale of the external frame between the two images.

The evaluation metrics include the average L1 discrepancy $E_1$ [15] and the average abs discrepancy $E_2$. The $E_1$ was first proposed in [15] to calculate the difference between truth scale ratios and predicted scale ratios. Since $E_1$ is not sensitive enough for large-scale image pairs of deep space scenes, we propose the $E_2$, which enables the evaluation to focus on image pairs with large-scale differences.

The evaluation metrics mentioned above are formulated in Equations (7) and (8).

The average L1 discrepancy $E_1$:

$$E_1=\frac{1}{N}{\displaystyle \sum _{i=1}^N\left|{\log }_2{s}_i-{\log }_2{\widehat{s}}_i\right|}$$

(7)

The average abs discrepancy $E_2$:

$$E_2=\frac{1}{N}{\displaystyle \sum _{i=1}^N\left|s_i-{\widehat{s}}_i\right|}$$

(8)

where $S$ and $\widehat{S}$ are ground truth scale ratio and predicted scale ratio, respectively. $N$ is the number of image pairs.

Table 2. Quantitative comparison of the average abs discrepancy and average L1 discrepancy.

Model	All	$1 < \mathbf{s} \le 5 \mathbf{or} 1 < \frac{1}{\mathit{s}} \le 5$	$5 < \mathbf{s} \le 10 \mathbf{or} 5 < \frac{1}{\mathit{s}} \le 10$	$10 < \mathbf{s} \le 15 \mathbf{or} 10 < \frac{1}{\mathit{s}} \le 15$	$15 < \mathbf{s} \le 20 \mathbf{or} 15 < \frac{1}{\mathit{s}} \le 20$
BoundingBoxes Algorithm	1.184/0.280	0.036/0.044	0.075/0.150	1.966/0.423	7.556/1.233
ScaleNet	0.793/0.173	0.104/0.116	0.118/0.130	1.519/0.183	3.933/0.517
DeepSpace-ScaleNet	0.203/0.052	0.047/0.053	0.075/0.046	0.410/0.049	0.693/0.085

is a quantitative comparison of the average abs discrepancy and average L1 discrepancy on Virtual SCB Dataset test images. The smaller the error, the more accurate the proportion estimate. It can be seen that DeepSpace-ScaleNet achieves excellent performance under different scales. An order of magnitude improves the estimation accuracy of our proposed model compared with other state-of-the-art methods. Under small scales (1 < s ≤ 5 or 1 < $\frac{1}{s}$ ≤ 5), the accuracy of the BoundingBoxes algorithm is slightly better than our method. The outer outline of the SCB is visible when the scale difference is negligible, making it easier to use BoundingBoxes to estimate the scale. However, the whole SCB or part of the SCB can fill the image field of view when the scale becomes large. As shown in

Table 3. Qualitative results of scale estimation of the above-mentioned state-of-the-art methods.

	Image Pairs	Ground-Truth	BoundingBoxes Algorithm	ScaleNet	DeepSpace-ScaleNet
(a)		1.15	1.14	1.32	1.16
(b)		8	8.29	7.23	8.06
(c)		13.5	13.43	11.49	13.53
(d)		18	13.07	12.75	18.02

c,d, this causes the BoundingBoxes algorithm to be inaccurate. Moreover, ScaleNet is generally better than the BoundingBoxes algorithm, especially at the large-scale level, where it shows a good deal of strength. However, it is still not accurate enough compared to our method. In particular, the accuracy improvement of the proposed method is more significant under large scales (15 < s ≤ 20 or 15 < $\frac{1}{s}$ ≤ 20). The results verify that DeepSpace-ScaleNet is more suitable for large-scale backgrounds such as deep space scenes.

The qualitative results of scale estimation are shown in Table 3. Image pairs were taken from test sets under different scales. As we can see, DeepSpace-ScaleNet is more robust to changes in illumination and changes in viewpoint than other methods, resulting in a more accurate scale distribution.

5.3. Ablation Study

In this section, several ablation experiments were conducted to demonstrate the effectiveness of the above modules. The same parameters were used in all experiments. The evaluation metrics included the average L1 discrepancy [15] $E_1$ and the average abs discrepancy $E_2$.

We used ASPP to replace GA-DenseASPP. The dilation rates of both were set to $\left\{1,6,12,18\right\}$.

We used the fully connected layers to replace CADP.

The effectiveness of our proposed module for scale estimation can be seen in

Table 4. Ablation experiment on different components of DeepSpace-ScaleNet.

	Average Absolute Discrepancy	Average L1 Discrepancy
DeepSpace-ScaleNet w/o GA-DenseASPP	0.236	0.057
DeepSpace-ScaleNet w/o CADP	0.237	0.061
DeepSpace-ScaleNet	0.203	0.052

. Our proposed GA-DenseASPP is more effective for deep space scenes, which can extract multi-scale information accurately. Then, CADP can integrate the inner relationship of the correlation maps, which is more correlation-aware than the fully connected layers.

5.4. Image Matching

In this section, we conducted image matching experiments using the Virtual SCB Dataset and compared the matching results with and without DeepSpace-ScaleNet. The combination method of SIFT [24] (extraction method) + FLANN [27] (matching method) and the combination method of SuperPoint [12] (extraction method) + SuperGlue [13] (matching method) were used to extract image features and match images. Notably, the former combination method is the most representative traditional-based method, and the latter is a state-of-the-art learning-based approach.

In Figure 10, we present our image matching visualization results. When DeepSpace-ScaleNet is not used, SIFT + FLANN and SuperPoint + SuperGlue suffer from an insufficient number of matching points and incorrect matching point correspondence. In contrast, the performance of matching points improves significantly with DeepSpace-ScaleNet. Not only are there more matching pairs, but also the accuracy has increased. The experimental results show that the proposed method can efficiently mitigate the problem under large-scale variation and outperform state-of-the-art approaches in image matching.

To further analyse the contribution of DeepSpace-ScaleNet, several experiments were conducted in the FixedLight Dataset and VariableLight Dataset. We use the Correct Matching Rate to calculate the matching accuracy of each algorithm when the number of feature point matching pairs corresponding to two images is greater than a certain threshold, which is considered a correct matching pair. After matching all the pairs of images, the Correct Matching Rate is used to calculate the matching accuracy of each algorithm [14]. $N_{\left(All Matching Pairs\right)}$ is all matching pairs in the dataset and $N_{\left(Correct Matching Pairs\right)}$ is the number of all correct matching pairs.

$$Correct Matching Rate=\frac{N_{(Correct Matching Pairs)}}{N_{(All Matching Pairs)}}\times 100\%$$

(9)

Figure 11 demonstrates that the performance of both the matching methods improved after combining DeepSpace-ScaleNet. SuperPoint + SuperGlue + DeepSpace-ScaleNet even achieved matching rates of 90.04 and 85.06%, indicating that only a small number of images failed to be matched correctly. ScaleNet also proved to be effective, indicating that the combined scale estimation method is indeed meaningful for image matching. Also, a comparison of results using two different datasets shows that SuperPoint + SuperGlue is somewhat robust to illumination but less invariant to scale. This is also in line with our experience. The experimental results show that state-of-the-art matching results can be achieved when combined with DeepSpace-ScaleNet.

5.5. Relative Pose Estimation

In SCB exploration missions, matching image pairs are obtained primarily for relative pose estimation. In this section, we evaluate the impact of the DeepSpace-ScaleNet on the relative pose estimation of SCB images. As in 5.4, SIFT + FLANN and SuperPoint + SuperGlue were selected as image extraction and matching methods. Specifically, we compared the AUC of the pose error at the thresholds (5, 10, 20°) calculated by SIFT and FLANN and SuperPoint and SuperGlue with and without DeepSpace-ScaleNet in the FixedLight Dataset and VariableLight Dataset.

The AUC of the pose error is the maximum of the angular errors in rotation and translation, which is often used in existing positional error estimates [13,30].

Table 5. Pose estimation in the FixedLight dataset.

	AUC@5	AUC@10	AUC@20
SIFT + FLANN	4.46	7.88	14.05
SIFT + FLANN + ScaleNet	5.32 (19.34%↑)	9.69 (23.02%↑)	18.06 (28.56%↑)
SIFT + FLANN + Ours	5.55 (24.41%↑)	10.50 (33.27%↑)	19.10 (35.96%↑)
SP + SG	3.65	6.73	12.31
SP + SG + ScaleNet	4.97 (36.25%↑)	10.56 (56.90%↑)	21.84 (77.44%↑)
SP + SG + Ours	8.55 (134.25%↑)	16.96 (152.01%↑)	34.74 (182.21%↑)

and

Table 6. Pose estimation in the VariableLight dataset.

	AUC@5	AUC@10	AUC@20
SIFT + FLANN	2.29	4.19	7.59
SIFT + FLANN + ScaleNet	3.03 (32.29%↑)	5.18 (23.68%↑)	9.18 (20.93%↑)
SIFT + FLANN + Ours	3.07 (34.18%↑)	5.63 (34.51%↑)	9.87 (29.99%↑)
SP + SG	3.58	6.43	11.36
SP + SG + ScaleNet	5.23 (46.20%↑)	10.42 (62.05%↑)	20.63 (81.56%↑)
SP + SG + Ours	6.97 (94.79%↑)	13.96 (117.10%↑)	29.42 (158.93%↑)

report the errors of pose estimation in the FixedLight Dataset and the VariableLight Dataset. As we can see, outstanding improvements are achieved when combined with DeepSpace-ScaleNet. At AUC@20, an improvement of 182.21% can be obtained compared to a single-scale approach such as SuperPoint + SuperGlue. Compared to ScaleNet, the method proposed in this paper works better and achieves the best results in all experimental results. Meanwhile, SIFT + FLANN also has a 35.96% improvement. In addition, SIFT outperforms SuperPoint + SuperGlue in both @5@10@20 when the light environment is stable and the opposite in the light-varying dataset.

Figure 12 shows the error of the relative pose estimation for each matching method without combining the scale estimation method and with combining ScaleNet and the proposed method DeepSpace-ScaleNet in this paper, respectively. The dataset was divided into four scale levels for analysis according to the dataset partitioning in Section 4. Figure 12a indicates the error at the FixedLight Dataset. SuperPoint + SuperGlue outperforms SIFT + FLANN when the scale range is below level 1 (1 < s ≤ 5 or 1 < $\frac{1}{s}$ ≤ 5). This is because the former has better generalization performance. However, once the scale variation exceeds level 1 (1 < s ≤ 5 or 1 < $\frac{1}{s}$ ≤ 5), the performance of SuperPoint + SuperGlue drops steeply, as SIFT + FLANN is more robust to large-scale variation. When combined with DeepSpace-ScaleNet, the performance of both is improved. In particular, SuperPoint + SuperGlue shows a significant improvement. This result suggests that DeepSpace-ScaleNet can improve the accuracy of relative pose estimation, especially in large-scale variation. Thus, adding scale estimation can lead to good results for single-scale estimation networks without the need to recreate the data for training. Furthermore, one phenomenon that can be observed is that DeepSpace-ScaleNet achieves better results than ScaleNet. Figure 12b shows the situation on the VariableLight Dataset. Comparing the same coloured line segments in Figure 12 shows that the matching accuracy decreases when the illumination level changes. Specifically, SIFT is less robust to illumination, while SuperPoint + SuperGlue is more robust to illumination changes. Combined with the method proposed in this paper, the performance of both is improved.

6. Conclusions

This paper proposed a large-scale invariant method-based DeepSpace-ScaleNet for SCB image matching at large-scale and light variations. Our proposed network improves image matching accuracy at scale and illumination variations by estimating the scale ratio of the two images. On the one hand, the proposed approach has better feature extraction capability for large-scale deep space scenes while estimating more accurate scale distributions. On the other hand, this mechanism of first estimating the image scale and then resizing and matching the image is more suitable for SCB exploration. The results demonstrate its superiority in that it can correctly boost the performance of all the matching methods when the scale and lighting of the image are substantially changed. The method proposed in this paper could potentially be applied to future deep space exploration missions in China, including the autonomous navigation of SCB probes and 3D reconstruction. We will explore a more lightweight and efficient network of widely used onboard processors in the future.