Cross-View Localization Based on Few-Shot Learning for Mars Rover via MarsCVFP Guidance

Highlights

What are the main findings?

• We innovatively propose a two-stage cross-view localization framework for Mars rovers. The first stage incorporates a few-shot learning model, MarsCVFP, which can implicitly learn and extract cross-view invariant features on the Martian surface without relying on explicitly defined or specific learning targets. In the second stage, a modified template matching algorithm is employed to achieve robust global localization.
• We introduce a multi-scale feature pyramid structure (MSFPS) and a feature interaction module (FIM) to capture discriminative fine-grained features, especially in Martian environments characterized by weak textures and unstructured features. Alongside, we design a multi-resolution contrastive loss ($L_{CF}$) within the MarsCVFP to alleviate the degradation in feature consistency extraction caused by the spatial resolution discrepancies between rover and orbiter imagery.

What is the implication of the main finding?

• We validate our framework on 85 unit-planned sites and 20 panoramic sites traversed by the Zhurong rover. The proposed framework consistently outperforms both traditional approaches and representative learning-based methods across diverse terrains, including dunes, bedrock, craters, and flat plains. It achieves a localization success rate above 82% while maintaining an accuracy of better than 4 pixels, even under coarse prior position uncertainties spanning 40 m × 40 m.

Abstract

High-precision localization of Mars rovers is essential for safe path planning and efficient navigation toward scientific targets. As planetary rovers traverse the surface, their positional uncertainty accumulates, which can be corrected through global localization by registering rover images to orbital maps. To date, image-based solutions are widely adopted; however, substantial manual intervention is often required, which is time-consuming and limits the range of autonomous navigation. To address these challenges, we propose a two-stage localization framework, comprising the Mars cross-view few-shot training paradigm (MarsCVFP), Mars cross-view feature extraction network (MCVN) trained under MarsCVFP, and a robust template matching algorithm. Specifically, the MarsCVFP model can leverage implicit cross-view feature as guidance without relying on a large amount of high-precision location-level supervision and explicitly annotated, specific learning targets in the scene. MCVN can capture discriminative fine-grained features on the weakly textured and unstructured surface of Mars by constructing the multi-scale feature pyramid structure (MSFPS) and the feature interaction module (FIM). We validate our framework on 85 unit-planned sites and 20 panoramic sites, respectively, as traversed by the Zhurong rover. The experimental results demonstrate that our framework consistently outperforms both the traditional approaches and the representative learning-based methods across diverse terrains, including dunes, bedrock, craters, and flat plains, achieving a localization success rate above 82% while maintaining a localization accuracy of better than 4 pixels, even under coarse prior positions uncertainties spanning 40 m × 40 m.

1. Introduction

Mars is a prime candidate for future human colonization [1]. Its exploration not only deepens our understanding of planetary geological structure and climate evolution, but also advances the search for the origins of life and lays a critical scientific and technological foundation for future interplanetary migration [2]. Mars rovers play a pivotal role in Mars exploration, offering the ability to traverse complex terrains and perform detailed in situ scientific investigations on the Martian surface [3,4]. Since the successful landing of Pathfinder and the surface traverses of the Sojourner rover in 1997 [5], a series of Mars rover missions have been implemented, including Spirit [6], Opportunity [7], Curiosity [8], and Perseverance [9]. As China’s first Mars exploration mission, the Zhurong rover commenced its surface journey on 22 May 2021 (UTC+8), after being deployed from the lander. By 18 May 2022 (UTC+8), before entering a dormant state, it had operated for 360 Martian days and traveled a total of 1921.5 m on Martian surface [10]. In contrast to Earth-based robotics, Mars rovers operate without access to global navigation satellite systems (GNSS), which compounded by communication constraints such as high packet loss and long transmission delays, further exacerbates the challenge of obtaining timely and accurate position in harsh and uncertain Martian environment [11]. These challenges highlight the urgent need to achieve high-precision localization, which is essential for ensuring safe traverses and the effective execution of scientific exploration tasks [12].

At present, the localization methods for Mars rovers can be broadly categorized into two types: relative localization and global localization. Relative localization determines the rover’s position and orientation within a local coordinate system, using onboard sensors such as stereo cameras, inertial measurement units (IMUs), and wheel odometers. Representative relative localization approaches include dead reckoning [13], visual-inertial odometry [14], bundle adjustment [15,16], and cross-site visual localization [17,18]. These methods can effectively achieve high-precision localization during the rover’s short-distance traverses. However, as the travel distance increases, such methods inevitably accumulate localization errors, which in turn leads to a significant degradation in the stability and safety of rover operations. To overcome this, global localization techniques are used to determine the rover’s precise locations over long-distance traverses within a global coordinate system (e.g., the Mars body-fixed coordinate system). The commonly used global localization approaches can be broadly categorized into radio-based [19], astronomical [20], and image-matching-based methods [21]. Among these, radio-based and astronomical techniques generally offer low accuracy and suffer from long localization latencies, making them inadequate for high-precision localization and navigation tasks. In contrast, image-matching-based approaches leverage a high-resolution orbital image basemap to achieve more accurate and efficient global localization, which has attracted increasing attention in Mars rovers. During the Mars Exploration Rover (MER) mission, the Spirit and Opportunity rovers performed infrequent orthorectification of 360-degree panoramic images captured by the onboard navigation and terrain cameras (NaTeCams) and matched them to HiRISE images, which provide high-resolution (0.25 m/pixel) orbital views [22]. Curiosity rover’s global localization approach basically follows the processing paradigm developed in the MER mission. Before the Perseverance and Zhurong rovers, global localization was performed manually on the ground by humans for the Mars rovers, which takes significant time, and the need for it limits how far the rover can autonomously navigate without ground-in-the-loop control [23]. The Cartograph and Geographic information Science Laboratory at The Ohio State University developed an image-based bundle adjustment method that connects lander and rover images, enabling landing site mapping and precise rover localization on lander-generated base maps [24]. NASA’s Jet Propulsion Laboratory (JPL) has recently proposed an on-board map-based localization method to provide global position for the Mars Helicopter Ingenuity mission [25]. This work, along with other representative studies on global localization for Mars drones, [26,27,28], demonstrates notable progress in the field. However, compared with the lander or drone-acquired imagery, orbital images lack distinct and rich surface textures and exhibit larger discrepancies in both resolution and perspective relative to the rover NaTeCam images. Consequently, image matching techniques based on rover, drone, or lander data are not well suited for global localization between the rover and the orbiter on Mars. Enhancing the automation and robustness of global localization, particularly during Mars rover traverses across typical terrains, such as dunes, bedrock, craters, and flat plains, remains a critical bottleneck problem.

Inspired by recent advances in autonomous driving [29], we formulate Mars rover global localization as a cross-view localization problem (Figure 1), where the goal is to determine the rover’s position on the orbiter’s map by establishing robust feature correspondences between ground-level rover images and orbital imagery captured from drastically different viewpoints. However, due to the unique characteristics of the Martian environment, the Mars cross-view localization (MCVL) task faces two significant challenges.

Figure 1. Problem formulation of the Mars rover localization. Taking the Tianwen-1 mission as an example, the Mars rover and orbiter acquire rover images and orbital images, respectively. The stereo images captured by the Mars rover’s navigation camera are transformed into orthophotos through orthorectification. Using prior position information, the relevant orbital orthophoto for localization is indexed. By analyzing the rover image and the orbital map, the pixel position of the rover in the orbital image slice is determined. This pixel position is then used to obtain the latitude and longitude coordinates in the Mars body-fixed coordinate system, thereby determining the rover’s global localization.

Firstly, the Martian surface is relatively flat and lacks prominent texture or structural features, which undermines the stability and discriminability of visual cues. Moreover, images acquired by the rover’s navigation camera and those captured by the orbiters differ significantly in appearance and resolution due to their distinct imaging domains, resulting in substantial challenges for establishing reliable cross-view correspondences.

Secondly, existing supervised training methods for planetary surface cross-view localization typically require high-precision geographic coordinates for images captured by Mars rovers, which are extremely difficult to obtain. In addition, the reliance on manually annotated object-level features, such as ridges, rocks or craters, as invariant surface cues significantly limit the scalability and practicality of traditional supervised approaches. Such features are frequently sparse or completely absent over extensive regions of the Martian surface in realistic planetary exploration scenarios.

To address these challenges, we propose a two-stage cross-view global localization framework for Mars rovers, with a focus on the Tianwen-1 mission. The proposed framework consists of the Mars cross-view few-shot training paradigm (MarsCVFP) model, the Mars cross-view feature extraction network (MCVN), and a robust template matching algorithm for global localization. The MarsCVFP leverages implicit Martian surface cues such as dunes, bedrock, craters, and flat terrain without requiring a large amount of high-precision location-level supervision or explicitly annotated specific feature categories as learning targets in the scene, a strategy we refer to as the few-shot training paradigm. To further strengthen cross-view feature learning on the weakly textured and unstructured surface of Mars, we introduce a multi-scale feature pyramid structure (MSFPS) and a feature interaction module (FIM) within MCVN based on Vision Transformer (ViT), enabling adaptive extraction of consistency features across hierarchical scale-space representations. Furthermore, a multi-resolution contrastive loss is designed within MarsCVFP to mitigate the degradation of feature consistency caused by the spatial resolution discrepancies between rover orthophotos and orbital images. Finally, we construct a pyramid structure to perform feature matching across multiple levels of the feature extraction hierarchy, enabling robust global localization.

Our framework is designed to enhance the autonomy of global localization for Mars rovers in real exploration scenarios, which are characterized by weak textures and unstructured surfaces, and to overcome the limitations of existing training pipelines. The core objective is to employ a few-shot learning model to replace conventional supervised frameworks, aiming to extract the distinctive and invariant features between the Mars rover and orbiter imagery, while simultaneously reducing the domain gap. Our main contributions are briefly summarized as follows:

(1)

We innovatively propose a two-stage cross-view localization framework for Mars rovers. The first stage incorporates a few-shot learning model, MarsCVFP, which can implicitly learn and extract cross-view invariant features on the Martian surface without relying on explicitly defined or specific learning targets. In the second stage, a modified template matching algorithm is employed to achieve robust global localization.

(2)

We introduce a multi-scale feature pyramid structure (MSFPS) and a feature interaction module (FIM) to capture discriminative fine-grained features, especially in Martian environments characterized by weak textures and unstructured features. In addition, we design a multi-resolution contrastive loss (${\mathcal{L}}_{CF}$) within the MarsCVFP to alleviate the degradation in feature consistency extraction caused by the spatial resolution discrepancies between the rover and orbiter imagery.

(3)

We validate the effectiveness of the proposed framework through comprehensive comparative experiments and extensive ablation studies. Compared with traditional global localization methods used in the Zhurong and Perseverance rovers, as well as the learning-based cross-view localization models, our approach demonstrates superior robustness across diverse terrains and varying coarse prior positions, while maintaining high-precision localization accuracy.

The rest of the paper is organized as follows: Section 2 provides a concise review of related works. Section 3 details our proposed two-stage cross-view localization framework for the Mars rover, including the Mars cross-view feature extraction network (MCVN), the Mars cross-view few-shot training paradigm (MarsCVFP) model for cross-view consistency feature guidance, and a modified template matching algorithm for aligning the rover orthophotos with the orbiter digital orthophoto maps (DOMs). Section 4 presents the experimental results that demonstrate the effectiveness of the framework and discusses its limitations. Finally, Section 5 concludes the paper and outlines directions for future research.

2. Related Work

In this section, we first briefly review the traditional methods for cross-view localization on planetary surfaces, analyzing their advantages and limitations. Next, we explore learning-based cross-view localization methods on planetary surfaces, highlighting their reliance on strongly supervised annotated datasets and revealing their limitations when applied to real scenarios. Finally, we discuss the development of geo-localization in structured Earth environments, highlighting the differences and unique challenges that arise when extending cross-view localization from Earth to planetary surfaces.

2.1. Traditional Cross-View Localization on Planetary Surfaces

Traditional planetary surface cross-view localization formulates the problem as image registration between rover images and the orbital map. Previous image registration approaches include normalized cross-correlation [30], sum-of-squared differences [31], and mutual information [32]. These approaches generally have a significant outlier rate, preventing adoption in real missions. Particle filters have been proposed to address this but require images across multiple locations [33]. To enhance feature construction for cross-view localization, existing studies typically follow the principles of viewpoint consistency or scale consistency. Di et al. [21] proposed SIFT-based and rock-based methods to achieve global localization using data from the Opportunity and Courage rovers, respectively, which attain a localization accuracy of better than one pixel in the HiRISE image. Verma et al. [23] proposed a sensible technology that uses a modified census transform to achieve sub-meter global localization accuracy in the NASA Mars 2020 mission. Li et al. [34,35] achieved cross-view localization by matching the geometric relationships of landmarks, including ridges, rocks, and craters. Building upon this method, subsequent studies, such as [36,37], have primarily focused on automatically extracting corresponding features from rover and orbiter images, and aligning their spatial distribution patterns, thereby enabling precise rover localization in high-resolution orbiter imagery.

Traditional planetary surface cross-view localization models offer relatively high processing efficiency and localization accuracy. However, their applicability is strictly constrained by specific scene characteristics, which require prominent textures and structural features [23]. Such surface features are often discontinuous or even absent along the traverses of Mars rovers, thereby necessitating human-in-the-loop interventions to align rover orthophotos with orbital base maps. As a result, the traditional models for planetary surfaces still suffer from a severely insufficient level of automation [38].

2.2. Learning-Based Cross-View Localization on Planetary Surfaces

To overcome the limitations of traditional planetary surface cross-view localization models that are strictly constrained by scene characteristics, recent studies have explored learning-based approaches for cross-view localization on planetary surfaces. Wu et al. [39] proposed a planetary localization neural network (PLaNNet) based on a Siamese convolutional neural network to match surface-perspective imagery with orbital images for planetary roving applications. This approach improves the automation and computational efficiency of rover cross-view localization. Chen et al. [40] proposed an image-based metric localization network to extract image features and generate a location heatmap, which serves as the basis for accurate estimation of query locations. However, this approach provides only coarse localization accuracy, making it challenging for practical mission applications. Zhao et al. [41] proposed a novel cross-view localization framework that autonomously registers a single rover NaTeCam image to an orbital image. However, this approach is heavily constrained by its dependence on geometric profile features, making it difficult to achieve cross-view localization on planetary surfaces with sparse texture and a lack of distinct structural features.

Overall, while the existing models have made significant contributions, their training pipelines rely heavily on strongly supervised annotated datasets, specifically the high-precision localization of rover images within the orbital maps, which substantially increases the cost of data labeling. In addition, these models typically require explicitly defined and annotated surface objects, such as rocks or craters. As a result, when such predefined features are absent in the scene, localization often fails, thereby posing critical challenges to the robustness of learning-based cross-view localization models.

2.3. End-to-End Cross-View Localization in Earth Environments

The end-to-end cross-view localization task on Earth is defined as establishing an effective mapping relationship from horizontal views to bird’s-eye views, while simultaneously learning and extracting cross-view consistent features between street-view images and remote sensing satellite imagery. It represents a core issue in the field of autonomous driving on the Earth. Shi et al. [42] proposed a feature aggregation strategy via learning multiple spatial-aware feature aggregation (SAFA) modules to bring corresponding deep features closer in the embedding space. Wang et al. [43] introduced a differentiable spherical transformation based on IPM, simplifying the localization task to an image alignment problem. Moreover, Refs. [44,45] use a CNN-based Siamese network to construct a simplified but effective architecture for cross-view localization on the streets of the Earth. Wang et al. [46] proposed a Top-to-Ground Aggregation (T2GA) module that capitalizes aerial orthographic views to aggregate features down to the ground level, leveraging reliable off-ground information to improve feature alignment. In addition, with the advent of foundation models, recent advancements in geo-localization have incorporated them as the initial backbones of the training process [47]. StreetCLIP [48] introduces a meta-learning framework for the visual feature encoder of the original CLIP [49], aiming to enhance the model’s zero-shot capability in tackling open-domain geo-localization. SeCo [50] incorporates temporal augmentations into the MoCov2 [51] self-supervised learning framework, whereas Dino-MC [52] extends DINO [53]. GAIR [54] efficiently processes and learns geospatial data features by explicitly modeling remote sensing data, street-view imagery, and their associated geolocation information on Earth.

Although the above models demonstrate promising performance in geo-localization tasks on Earth [55], they are not directly applicable to planetary surfaces. Unlike Earth environments, planetary surfaces lack distinct structured features and rich textures, which severely limits the availability of altitude-related cues such as vehicles, trees, and buildings, thereby complicating cross-view localization. Moreover, the scarcity of labeled cross-view data on planetary surfaces further increases the difficulty of this task. To address these challenges, we propose a two-stage cross-view localization framework for Mars rovers, which consists of a few-shot learning cross-view feature extraction model that does not rely on explicitly annotated specific feature categories in the scene and a modified template matching algorithm.

3. Methods

In this section, we first present a brief overview of the cross-view localization framework. Next, in Section 3.1, we provide a detailed discussion of the Mars cross-view few-shot learning training paradigm (MarsCVFP), followed by an in-depth explanation of the Mars cross-view feature extraction network (MCVN) integrated within it in Section 3.2, along with a detailed description of the model’s loss function in Section 3.3. Finally, in Section 3.4, we present a description of our improved template matching algorithm.

Cross-view localization for Mars rovers presents significant challenges due to the domain gap between orbital and ground-level observations, the sparse surface texture, and the absence of distinct structural features on Martian surface. To address these challenges, we propose a two-stage cross-view framework for Mars rovers, as illustrated in Figure 2. The framework includes the MarsCVFP model, which extracts cross-view consistency features using a small number of cross-view annotated image pairs that does not rely on explicitly annotated specific learning targets in the scene and an improved template matching algorithm that aligns rover orthophotos with orbiter imagery to accurately determine the rover’s position. The following sections provide a detailed description of the proposed framework, developed and evaluated for the Zhurong rover in this study.

Figure 2. Overall workflow of the proposed Mars rover cross-view localization framework: (a) M the proposed cross-view consistency feature extraction model for unique Martian surface textures and structures; (b) a robust matching algorithm that aligns rover orthophoto features with orbital imagery features to achieve global localization.

3.1. Mars Cross-View Few-Shot Training Paradigm (MarsCVFP)

For the task of cross-view localization of the Zhurong rover, we assume that a rough prior localization is available, with an average relative localization error of approximately 2% [17], obtained through cross-site visual localization techniques applied to the Zhurong rover. Given a 360-degree panoramic image $G$, comprising 12 to 15 pairs of stereo images captured by the NaTeCam onboard the Zhurong rover, and an $L\times L$ HiRISE orbital image $S$ representing the local area where $G$ was taken, the orbital image $S$ is cropped around the rover’s estimated position based on the relative localization with a crop width equal to the sum of the rover’s positional uncertainty and the width of the stereo orthomosaic. Our cross-view localization objective is to estimate the 2D image coordinates $X\in {[0,L]}^2$ within $S$ that correspond to the ground location of the NaTeCam at the time of capturing $G$.

Training for the cross-view localization task necessitates a large number of high-precision positional annotations in orbital imagery for each orthophoto generated at a rover stop. In supervised frameworks, these ground-truth labels are often generated by applying Gaussian smoothing to these annotated positions [56]. However, the difficulty of accurately establishing precise spatial correspondences between rover-captured and orbiter-acquired images renders conventional supervised learning approaches impractical, as they rely on high-precision location-level supervision and object-specific learning targets. To overcome this challenge, we propose the MarsCVFP model, which obviates the need to explicitly define the object-level learning features for a specific scene, as shown in Figure 3. We eliminate the need for costly process of large-scale and strong annotations. Instead, we only utilize a small subset of high-precision annotated image pairs for self-distillation training, consisting of rover orthophotos generated from NaTeCam imaging sites and the corresponding orbiter DOM. The training primarily relies on the feature information from these few-shot, high-precision cross-view annotated image pairs, which eliminates the need for precise object-level annotations and instead captures broad scene characteristics, including the textures and structures of dunes, rocks, craters, and plains across cross-view image pairs. Compared with conventional supervised paradigms, our approach substantially reduces annotation costs while enabling the model to implicitly capture consistent cross-view features, even in scenarios with sparse surface textures on Mars.

Figure 3. The proposed MarsCVFP model. During the gradient backpropagation process, the student’s parameters are updated using Stochastic Gradient Descent (SGD), while the teacher’s parameters are updated via the Exponential Moving Average (EMA) of the student’s parameters. ${\mathcal{L}}_{raw}$ is the loss function used in DINOv2 model, while ${\mathcal{L}}_{CF}$ is new loss function introduced to address the challenges posed by the unique environment of the Martian surface and cross-view consistency feature extraction. The detailed information about ${\mathcal{L}}_{CF}$ can be found in Section 3.3 and will not be further elaborated here.

In the MarsCVFP model, we establish a teacher network and a student network, both of which share the same model architecture, as specifically described in Section 3.2. We train the student network to match the output of the teacher network, Specifically, we construct different distorted view, or crops, of an image with multi-crop strategy [57]. From a given image, we generate a set of different views $V$, which include global views $F_{global}$ that cover most of the original image’s area and local views $F_{local}$ that cover a smaller portion of the original image. All crops are processed through the student network, while only the global views are passed through the teacher network. We group the input data and incorporate few-shot, high-precision feature information into the training of the MarsCVFP model. More precisely, we pre-construct a set of positive samples $S_p$ by manually registering orbiter images with the rover’s ortho-photos. Subsequently, we randomly select three orbiter images from the remaining orbiter image dataset as negative samples $S_n$. These positive and negative sample sets independently generate global and local view images, which are then processed through few-shot learning. An Exponential Moving Average (EMA) of the model weights is maintained during training to stabilize parameter updates and improve generalization. Under weak-texture Martian conditions, EMA reduces the adverse effects of noisy or ambiguous gradients from homogeneous terrain, facilitating more consistent feature optimization. In the final output feature stage of the teacher network, we follow Equation (1) to maximize the similarity of positive sample pairs while minimizing that of negative pairs. This can drive the student network to capture consistent representations across local-to-global correspondences, thereby enhancing its ability to extract cross-view features between the rover and orbiter images.

$$\underset{{\theta }_t}{\min }\left\{cossim\left(F_G,F_{Sn}\right)\right\}+\underset{{\theta }_t}{\max }\left\{cossim(F_G,F_{Sp})\right\}$$

(1)

where $cossim(·)$ is the computation of cosine similarity. $F_G$ is the feature map output from the Mars rover orthomosaic image, $F_{Sp}$ and $F_{Sn}$ represent the feature maps output from the orbiter’s positive and negative sample images, respectively.

Due to the significant resolution difference between the rover and orbiter images, aligning the same features across scale spaces becomes challenging. We establish a coarse-to-fine correspondence between the rover and orbiter images to guide the model training. Specifically, we extend the architecture by integrating an autoencoder that projects the input feature maps into a lower-dimensional space, thereby learning the potential features representations, and use reshaping to restore the spatial size of the feature map, aligning them with those of the original input images. Subsequently, we apply max-pooling with strides $\mathcal{M}$ of 2, 4, 8, and 16 to$F_R$ in both teacher and student networks, yielding a series of pooled feature maps $F_{\mathcal{M}}$. The multi-resolution feature maps produced by the teacher network are used as pseudo-labels to supervise the outputs of the student network at corresponding resolutions. With the introduction of the coarse-to-fine multi-resolution strategy, the MarsCVFP demonstrates rapid convergence in the initial stages of training, driven primarily by coarse-grained features. As training progresses, the strategy facilitates the learning of pixel-level discriminative feature representations, enabling the model to comprehensively perceive spatial information across multiple scales, which can improve the model’s robustness and applicability to Mars.

3.2. Mars Cross-View Feature Extraction Network (MCVN)

The MarsCVFP few-shot learning framework effectively establishes consistent feature representations of the same region across cross-view target domains without requiring explicit designation of object-level learning targets in the scenes, thereby reducing the reliance on strongly supervised labels required by existing models. However, the Martian surface, characterized by a lack of rich textures and structured features, presents considerable challenges for robust feature extraction under varying environmental conditions. In addition, the original Vision Transformer (ViT) [58] architecture lacks inherent multi-scale spatial perception, which limits its ability to capture fine-grained local edge features such as prominent rocks, rims of impact craters, and terrain boundaries. These limitations adversely affect the performance of downstream tasks, including high-precision image registration and localization. To address these challenges, we propose a Mars cross-view feature extraction network, as shown in Figure 4, which equips the model with local and global information interaction capabilities, along with multi-level scale-aware spatial perception, thereby enabling the effective extraction of more precise, fine-grained features on the Martian surface and enhancing cross-view localization performance.

Figure 4. The overall architecture of the Mars cross-view feature extraction network (MCVN). Our model adopts a two-branch design comprising three core components: (1) a plain ViT-branch, whose encoder layers are evenly divided into $n=4$ blocks to facilitate staged feature interaction; (2) a multi-scale feature pyramid structure (MSFPS), introduced as a CNN-based branch, comprising two linear projection layers and a set of depth-wise separable convolutions with diverse receptive fields, designed to effectively capture multi-scale spatial features; and (3) a feature interaction module (FIM), which integrates the features from two branches across different stages, enabling the effective extraction of more precise, fine-grained features on the Martian surface.

3.2.1. Multi-Scale Feature Pyramid Structure (MSFPS)

Inspired by [59,60], we integrate a pyramid structure into the original ViT model, combining the advantages of CNN in capturing local image features with the strengths of ViT in understanding complex image structures and semantic relationships. The multi-scale feature pyramid structure is composed of two linear projection layers and a set of depth-wise separable convolutions with multi-receptive fields, as shown in Figure 4a. It is designed to model the local spatial contexts of images parallel with the patch embedding layer, so as not to alter the original architecture of ViT. Specifically, we adopt a standard convolutional stem from ResNet [61], consisting of three convolutional layers followed by a max-pooling operation, to process the input image of size $H\times W$. Subsequently, a series of $3\times 3$ convolutions with stride 2 are applied to progressively reduce the size of feature maps and increase the channel dimension. This process produces a set of multi-scale feature maps $\left\{ {\mathcal{F}}_1, {\mathcal{F}}_2, {\mathcal{F}}_3 \right\}$ with spatial resolutions of $1/8$, $1/16$, and $1/32$ of the original image size, respectively. These feature maps are then flattened and concatenated into a unified feature token $\mathcal{F}\in {\mathbb{R}}^{\left(HW/8^2+HW/{16}^2+HW/{32}^2\right)\times D}$. The resulting token sequence is subsequently passed through a linear projection layer to reduce its dimensionality, after which the features are divided into $M$ groups along the channel dimension. Different feature groups are processed using depth-wise convolutional layers with distinct kernel sizes, enabling the decoupling of spatial and channel-wise representations. This design enhances the MSFPS model’s sensitivity to subtle terrain structures while significantly reducing computational cost and effectively capturing diverse spatial contextual information. After processing, the features from all groups are concatenated and passed through a linear projection layer to increase their dimensionality. This process can be formally described as follows:

$${\mathcal{F}}_{CNN}=LP\left(DWConv\right(LP\left(\mathcal{F}\right)\left)\right)$$

(2)

where $LP(·)$ denotes linear projection operation, and $Dwconv(·)$ denotes a set of depth-wise convolutional layers with different kernel sizes.

3.2.2. Feature Interaction Module (FIM)

To address the bias introduced by architectural differences in texture feature extraction on the Martian surface and to enhance both the granularity and completeness of modality representations (e.g., high–low frequency semantics, as well as global–local contextual information), we propose the feature interaction module, which comprises a multi-scale self-attention mechanism and a feed-forward network, as shown in Figure 4b. Specifically, it injects the multi-scale features extracted by the CNN into the ViT. This design mitigates the limitations of the ViT in terms of inadequate intra-patch information interaction and the absence of hierarchical representations. Concurrently, the high-level semantic features extracted by the ViT are propagated back into the CNN-branch, further enhancing its long-range dependency modeling and semantic representation capacity. In order to effectively fuse the ViT feature ${\mathcal{F}}_{vit}^i\in {\mathbb{R}}^{HW/{16}^2\times D}$ and the multi-scale feature $\left\{ {\mathcal{F}}_1, {\mathcal{F}}_2, {\mathcal{F}}_3 \right\}$ obtained through the MSFPS module, we directly add the features ${\mathcal{F}}_{vit}^i$ and ${\mathcal{F}}_2\in {\mathbb{R}}^{HW/{16}^2\times D}$ through element-wise addition, yielding the set ${\mathcal{F}}^{\prime }=\left\{ {\mathcal{F}}_1, {\mathcal{F}}_2^{\prime }, {\mathcal{F}}_3 \right\}$. Subsequently, we use self-attention mechanism to effectively integrate features from both CNN and Transformer branches, reinforcing the robustness of representations to modality-specific differences. This process can be formulated as:

$${\mathcal{F}}_{new}=selfAttention\left(norm\right({\mathcal{F}}^{\prime }\left)\right)$$

(3)

where $norm(\cdot )$ denotes LayerNorm [62], $selfAttention(\cdot )$ is the multi-scale deformable attention [63]. Finally, we use the fused feature ${\mathcal{F}}_{new}\in {\mathbb{R}}^{\left(HW/8^2+HW/{16}^2+HW/{32}^2\right)\times D}$ obtained through the self-attention mechanism to update the input features from both the CNN branch (${\mathcal{F}}_{CNN}$) and the ViT branch (${\mathcal{F}}_{Vit}$). Specifically, for the i-th stage, we align the feature map sizes of ${\mathcal{F}}_{new}^1\in {\mathbb{R}}^{HW/8^2\times D}$ and ${\mathcal{F}}_{new}^3\in {\mathbb{R}}^{HW/{32}^2\times D}$ to match ${\mathcal{F}}_{new}^2\in {\mathbb{R}}^{HW/{16}^2\times D}$ using bilinear interpolation. These three components are then fused via element-wise addition to generate ${\mathcal{F}}_{ViT}^{\prime }\in {\mathbb{R}}^{HW/{16}^2\times D}$, which is added to ${\mathcal{F}}_{ViT}^i$ as the input of the next ViT layer, following the layer normalization and feed-forward neural network stages. Then, we use ${\mathcal{F}}_{new}$ and ${\mathcal{F}}_{CNN}^i$ to obtain the updated feature ${\mathcal{F}}_{CNN}^{i+1}$. This process can be formulated as:

$${\mathcal{F}}_{ViT}^{i+1}= {\mathcal{F}}_{ViT}^i+FFN\left(norm\left({\mathcal{F}}_{ViT}^{\prime }\right)\right)$$

(4)

$${\mathcal{F}}_{CNN}^{i+1}={\mathcal{F}}_{CNN}^i+FFN\left(norm\right({\mathcal{F}}_{new}\left)\right)$$

(5)

where $FFN(\cdot )$ denotes the feed-forward network [64]. Through cross-architecture feature fusion and bidirectional interaction, the model effectively leverages multi-scale and multi-level features, thereby enhancing its capability to represent and generalize cross-view consistent features under challenging Martian surface conditions characterized by sparse textures and a lack of structured features.

3.3. Loss Function

In the MarsCVFP model, we apply ${\mathcal{L}}_{local}$ and ${\mathcal{L}}_{global}$ losses [65,66] between the features derived from the class token of our Mars cross-view feature extraction network, which are extracted, respectively, from the student and teacher networks using local and global crops of the same image. Specifically, we pass the class token through the MarsCVFP head, a multi-layer perceptron (MLP) that outputs a vector of prediction scores. To prevent model collapse, we use Sinkhorn–Knopp centering [57] to the feature vector $F_t$ produced by the teacher MarsCVFP head, yielding a centered vector $F_t^{\prime }$. Both $F_t^{\prime }$ and the student feature vector $F_s$ are then processed using softmax normalization to produce the corresponding probability distribution $p_t$ and $p_s$, respectively. To provide a more definitive learning target for the student, we apply sharpening to the teacher distribution $p_t^{\prime }$. Finally, we treat $p_t^{\prime }$ as a “pseudo-label” and use the cross-entropy loss to measure the divergence between $p_t^{\prime }$ and $p_s$, as follows:

$${\mathcal{L}}_{local}=-\sum _{i=1}^{N_{local}}log\left({\displaystyle \frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(F_{s,i}\cdot \left(F_{t,j}-m_t\right)/\tau \right)}{{\sum }_{k=1}^{N_{local}}\mathrm{e}\mathrm{x}\mathrm{p}\left(F_{s,i}\cdot \left(F_k-m_k\right)/\tau \right)}}\right)$$

(6)

where $N_{local}$ denotes the feature vectors of the local crops obtained by the student network. $m_k$ denotes the mean of the corresponding feature vector $F_k=\left[F_t, F_s\right]$. If $F_k$ is derived from the teacher network, then $m_k=m_t$. If $F_k$ is derived from the student network, then $m_k=0$. $\tau$ is the temperature parameter used to control the sharpness of the softmax distribution. Similarly, we use $N_{global}$ (the global crops obtained by student model) to compute ${\mathcal{L}}_{global}$, which is similar with Equation (6).

In addition, we randomly mask a subset of input global patches provided to the student, but not to the teacher network. The masked tokens from the student are then passed through the student’s iBOT [67] head. In parallel, the teacher’s iBOT head processes the corresponding visible patch tokens. Subsequently, the softmax and centering steps are applied to the feature maps $F_{ti}$ and $F_{si}$, which are obtained from the teacher and student networks, respectively, following the same procedure, as described previously. We use a cross-entropy loss to quantify the similarity and discrepancy between the feature representations generated by student and teacher networks for each masked patch, thereby enabling the student to simultaneously capture both global and local features, as follows:

$${\mathcal{L}}_{iBOT}=-\sum _i{p}_{ti}\mathrm{l}\mathrm{o}\mathrm{g}\left(p_{si}\right)$$

(7)

where $i$ denotes the patch indices for the masked tokens. $p_{si}$ and $p_{ti}$ represent the probability distributions over the randomly masked global patches from student network and the corresponding global patches from teacher network, respectively. For the class tokens obtained from global crops in student network across different images, we apply KoLeo loss [68] to encourage a uniform distribution of the features within each batch, thereby promoting a well-dispersed feature space for the class token embeddings, as follows:

$${\mathcal{L}}_{koleo}=-{\displaystyle \frac{1}{n}}\sum _{i=1}^n\mathrm{l}\mathrm{o}\mathrm{g}(d_n,i)$$

(8)

where $n$ is the number of vectors in the given set $(x_1,\dots ,x_n)$, and $d_n,i={min}_{j\ne i}\Vert x_i-x_j\Vert$ is the minimum distance between $x_i$ and any other point within the batch. We also apply ${\mathcal{l}}_2$ normalization to the features before computing this regularizer.

Furthermore, to effectively capture both large-scale information (e.g., terrain boundaries) and fine-grained details (e.g., craters and rocks) crucial for cross-view localization of the Mars rover while concurrently addressing the inconsistencies in feature representations induced by the spatial resolution disparities between the Mars rover and orbiter image domains, we propose a coarse-to-fine contrastive loss ${\mathcal{L}}_{CF}$ to guide the MarsCVFP model training, as follows:

$${\mathcal{L}}_{CF}=-\sum _{\mathcal{M}}{w}_{\mathcal{M}}\left[log{\displaystyle \frac{exp\left({\mathcal{M}}_p^{+}/\tau \right)}{{\sum }_i^N\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathcal{M}}_i/\tau \right)}}+\lambda L_{KL}^{+}\right]$$

(9)

where $w_{\mathcal{M}}$ are the weights derived from max-pooling with strides of 2, 4, 8, and 16, where each weight is inversely proportional to the pooling stride. ${\mathcal{M}}_i$ denotes the cosine similarity between the feature map $F_G$ of the rover DOM and the feature $F_S$ of the orbiter DOM across different spatial resolutions. ${\mathcal{M}}_p^{+}$ denotes the cosine similarity values computed for the set of positive image pairs in the few-shot learning cross-view labels. The hyperparameter τ is used as a reference to control the sensitivity of the model to difficult candidate regions. $L_{KL}^{+}$ is the Kullback–Leibler divergence between the probability distribution output by the teacher $p_{\mathcal{M}}^t$ and the student $p_{\mathcal{M}}^s$ for the set of positive image pairs, aiming to minimize the discrepancy between their outputs.

Finally, we introduce a weighting coefficient $\sigma$ for the loss term ${\mathcal{L}}_{CF}$ to balance the overall loss function ${\mathcal{L}}_{all}$, as follows:

$${\mathcal{L}}_{all}=L_{raw}+\sigma L_{CF}$$

(10)

where ${\mathcal{L}}_{raw}={\mathcal{L}}_{local}+{\mathcal{L}}_{global}+{\mathcal{L}}_{iBOT}+{\mathcal{L}}_{koleo}$. ${\mathcal{L}}_{raw}$ loss ensures stable self-distillation training, and its joint optimization with and ${\mathcal{L}}_{CF}$ significantly improves the model’s generalization in cross-view feature consistency for Mars rover localization.

3.4. Robust Template Matching and Localization Estimation

After applying the MarsCVFP model, we obtain feature representations from Mars rover orthophotos and the orbiter images that demonstrate high consistency in both scale and viewpoint. In order to achieve robust and reliable localization estimation of the Mars rover, we propose an improved template matching algorithm to match the feature representation of the Mars rover orthomosaic image against that of the orbital image, as shown in Figure 5. Specifically, we construct a multi-layer pyramid for the orbital feature $F_o$ using max-pooling with down-sampling ratios $\mathsf{\tau }$ of 1/2, 1/4, and 1/8, where each layer represents a scaled version of the original feature. Similarly, the Mars rover orthomosaic feature $F_r$ is rescaled to generate template features at the corresponding resolutions. We then apply the census transform, a non-parametric local image transform that is invariant to monotonic intensity variations, to describe each pixel location on both rover DOM feature $F_r^{\tau }$ and orbital feature $F_o^{\tau }$ at each pyramid level, as follows:

$$\begin{array}{cc}\mathrm{D}\left(x, y\right)=\mathrm{c}\left(I\left(x,y\right),I\left(x+i,y+i\right)\right)& i,j\in W\end{array}$$

(11)

$$W=\left\{\left(i,j\right) | i,j\in \left\{-1, 0, 1\right\}, \left(i,j\right)\ne \left(0, 0\right)\right\}$$

(12)

where $\mathrm{c}\left(,\right)$ is a comparison operator between nearby pixel intensities, defined as $\mathrm{c}=1$ if $I\left(x,y\right)>I\left(x+i,y+i\right)$ and $\mathrm{c}=0$ otherwise. $W$ is a $3\times 3$ local window centered on each pixel. Self-comparisons are excluded, resulting in an 8-field descriptor for each pixel location $\left(x, y\right)$ in an image $I$ [69].

Figure 5. An improved template matching algorithm with a multi-scale strategy, in which missing and background data are excluded from the computation of matching costs. * represents the element-wise multiplication. The purple arrows indicate the computation of the sliding window, and the red numbers represent the different descriptor bits in the Census transform.

To address the issue of missing and background data caused by geometric occlusions inherent in rover DOM generated from the rover’s NaTeCam viewpoint, we construct a binary validity mask $Q$, where missing regions are assigned a value of 0 (excluded from matching cost computation) and valid regions are assigned a value of 1. Then, we compute the Hamming distance $H$ between two descriptors by summing the bitwise comparison results over the window $W$. We take one scale space as an example, as shown below:

$$H\left(D_o,D_r\right)={\sum }_{k=1}^{\left|W\right|}H\left(d_o,d_r\right)$$

(13)

where $D_o$ and $D_r$ denote the descriptors at the center pixel within the sliding window $W$, extracted from the orbital image features $F_o^{\tau }$, and the Mars rover features $F_r^{\tau }$, respectively. Similarly, $d_o$ and $d_r$ represent the Census values of the nearby pixels.

Finally, we shift the rover DOM feature map across the orbital feature map and calculate the spatial correlation score at each position using our improved template matching algorithm at each scale level. The overall score $S$ is obtained by aggregating the correlation scores across all scale levels, as follows:

$$S\left(x,y\right)=1/\sum _{\tau }\sum _{i,j}H\left(\underset{orbiter}{\underbrace{D_o\left(x+i, y+j\right)}},\underset{rover}{\underbrace{D_r\left(i, j\right)}}\right)$$

(14)

where $\left(x,y\right)$ denotes the estimated position of the Mars rover on the orbital map. We posit that higher scores at different grid positions indicate a greater likelihood of the query camera being located at those positions. Consequently, the pixel location with the maximum correlation score is recorded as the final localization result.

4. Experiments and Discussion

In this section, we first present the experimental datasets and the implementation details of the network training. Then, we introduce the evaluation metrics and compare our proposed two-stage cross-view localization framework for a Mars rover with several state-of-the-art (SOTA) approaches to demonstrate its effectiveness. In addition, we perform ablation studies to validate the necessity of each component.

4.1. Experimental Datasets and Implementation Details

To facilitate research in Mars rover cross-view localization, we constructed the Mars Cross-View Localization Dataset (MCVLD), which includes rover images and the corresponding orbital map collection. Specifically, we selected over 900 stereo images acquired by the NaTeCam onboard the Zhurong rover, covering 85 unit-planned sites and 20 panoramic sites, with horizontal field-of-view angles of $120°$ and $360°$, respectively. To mitigate the perspective discrepancies between Mars rover and orbital images, we utilized the cartographic techniques and mapping workflows implemented in the Zhurong mission [17], to automatically orthorectify horizontal perspective images acquired at all Zhurong rover sites into orthographic views. The resulting orthophotos have a size of 400 × 400 pixels, with each pixel representing to a ground distance of 0.05 m. To more accurately assess the localization accuracy, we up-sampled the original orbital images from a resolution of 0.25 m/pixel to 0.05 m/pixel. Using expert-guided image registration, we then extracted orbital image patches centered on the rover’s ground-truth location to construct positive sample pairs in the MCVLD dataset. The high-precision reference positions were determined via manual image registration performed by experienced experts. Specifically, 5–10 tie points were visually selected based on distinctive geological landmarks to align the rover orthoimage with the HiRISE map. A translational transformation was estimated by minimizing misalignment across the selected points, and pixel deviations were computed for verification. The tie points with deviations exceeding one pixel were discarded and replaced, and the process was iterated until sub-pixel alignment was achieved. The final registered image and its overlapping region were then exported for subsequent analysis. To generate negative samples, we randomly translated the orbital images using shift ratios of 25%, 50%, and 75%, where each ratio represents the translation distance as a percentage of the actual ground coverage of the orbital image. Subsequently, we applied data augmentation techniques, including image rotation, Ipa Cutout [70], and brightness and contrast adjustments, to expand the MCVLD dataset to 20 k image pairs, comprising approximately 5 k positive and 15 k negative pairs.

As for implementation, we began by initializing the ViT network with pretrained weights from the original ViT-B [71] model and retrained the Mars cross-view feature extraction network using the COCO [72] dataset, which provides strongly supervised semantic annotations, to acquire a pretrained model endowed with initial generalization ability for downstream tasks. Subsequently, we fine-tuned the pretrained model for the Mars rover cross-view localization task within the proposed MarsCVFP framework, utilizing our few-shot, high-precision cross-view dataset that does not rely on explicitly defined scene-specific learning targets. A few-shot learning was employed to further adapt the model to the target domain and enhance cross-view consistency during this stage. The final model was obtained through this two-stage training process implemented in Python 3.8.20 and Pytorch 1.9.0, conducted on an Ubuntu 18.04 system equipped with two Intel Xeon Silver 4210R CPUs and dual NVIDIA GeForce RTX 3090 GPUs. During the training stage, the MarsCVFP was optimized using the AdamW optimizer [73] over 100 epochs, with a learning rate of 0.0001 and a mini-batch size of 4. We set $\sigma =0.5$ to balance the few-shot learning loss ${\mathcal{L}}_{raw}$ in the teacher–student network and the cross-view multi-resolution few-shot learning loss ${\mathcal{L}}_{CF}$. The temperature parameter $\tau$ for ${\mathcal{L}}_{CF}$ was set to 0.1.

4.2. Evaluation Metrics

For each Mars rover orthophoto, we generated three levels of prior bias for the corresponding orbital image patches: 5 m, 10 m, and 20 m, with the associated search areas covering 25 m × 25 m, 30 m × 30 m, and 40 m × 40 m, respectively. The prior bias represents the uncertainty range of the rover’s estimated position. For example, under the 20 m prior bias setting, the rover’s true location may deviate by up to 20 m in both the latitude and longitude directions from its estimated position. By setting different levels of prior bias, we simulated the uncertainty in cross-site localization of the Mars rover, which has an average relative localization error of 2% [17,18]. These settings reflect practical scenarios in which the rover’s prior position estimate may vary in accuracy due to localization drift and allows us to evaluate the robustness and adaptability of different cross-view localization models under varying search ranges and realistic uncertainty levels.

4.2.1. Cross-View Localization Success Rate

The localization success rate refers to the probability that the Mars rover can accurately determine its position within a specified error threshold. It serves as an important metric for evaluating the accuracy and reliability of the localization method [23]. Since the orthophotos generated from the Mars rover images were pre-aligned in orientation, no directional deviation was assumed during localization. To evaluate localization performance, we defined a threshold distance $d$ based on the Euclidean distance and considered the rover localization to be successful if the estimated position lay within this distance from the reference position for a given test case. Given the original resolution of the HiRISE DOM is 0.25 m/pixel, we up-sampled the HiRISE orbital image by a factor of five using bicubic interpolation to match the spatial resolution of the rover orthoimage (0.05 m/pixel) and set $d$ to 0.50 m, 0.75 m, and 1.0 m to assess the success rate of the proposed framework under different levels of localization precision. These thresholds are specified in physical units (meters) to maintain resolution independence. In the actual implementation, these thresholds are converted to pixel units in the up-sampled coordinate system (10, 15, and 20 pixels at 0.05 m/pixel), ensuring consistency with the computed localization errors. The localization success rate is defined as follows:

$$SR={\displaystyle \frac{S_{suc}}{S_{all}}}\times 100\%$$

(15)

where $S_{suc}$ and $S_{all}$ are the number of test sites within the threshold distance d (i.e., the number of successful localization sites) and the total number of test sites, respectively.

4.2.2. Cross-View Localization Error

We used both the mean and median as evaluation metrics for localization error, which is defined as the distance between the predicted position $L_{pre}$ and the reference position $L_{ref}$. The mean localization error reflects the overall accuracy across all test cases, while the median localization error, being less influenced by outliers, provides a more robust reference for localization performance. Following Equation (16), we quantified the localization deviation for each test case as the Euclidean distance in pixel space and converted it to meters. We then aggregated these deviation values across the test dataset to compute the mean and median localization errors.

$$Error=\sqrt{{\left(x_{pre}-x_{ref}\right)}^2+{\left(y_{pre}-y_{ref}\right)}^2}\times 0.05$$

(16)

where $\left(x_{pre},y_{pre}\right)$ and $\left(x_{ref},y_{ref}\right)$ are the pixel coordinates of the predicted and the reference position of the Mars rover on the orbital map, respectively. All evaluations were conducted in the pixel coordinate system of the orbital imagery (0.05 m/pixel). The final localization accuracy was subsequently converted into the physical coordinate space and reported in meters.

4.3. Experimental Results

In this section, we select various mainstream feature backbone networks and conduct extensive experiments (with different architectures and model sizes) on COCO and our MCVLD, to verify the effectiveness of the MCVN. In addition, we conducted comparative experiments with different SOTA cross-view localization methods, covering street-level scenes for autonomous driving and planetary surface scenes in deep space exploration. These results indicate that our two-stage strategy, which integrates our MarsCVFP model with the improved template matching algorithm, significantly outperforms both traditional and end-to-end cross-view localization methods on planetary surfaces.

4.3.1. Comparisons with Different Backbones and Frameworks

We conducted experiments on two datasets: the COCO dataset, which contains semantically labeled images for evaluating general feature consistency using metrics such as Average Precision for Masks (${\mathrm{A}\mathrm{P}}^{\mathrm{m}}$), ${\mathrm{A}\mathrm{P}}_{50}^{\mathrm{m}},$ and ${\mathrm{A}\mathrm{P}}_{75}^{\mathrm{m}}$ (corresponding to IOU thresholds of 0.5 and 0.75), and our MCVLD dataset, which consists of paired rover and orbital images for assessing the robustness of cross-view localization on planetary surfaces. For the MCVLD dataset, we adopted a search area of 25 m × 25 m and evaluated localization performance using success rate ($SR$), with a threshold distance of 1.00 m, as well as localization error ($Error$). This comprehensive evaluation highlights the effectives of our framework in both general visual understanding tasks and the specific cross-view localization scenarios on Mars. Table 1 compares our proposed MCVN with representative backbone networks under the DINOv2 framework, including CNN-based (VGG, ResNet, ConvNeXt), Transformer-based (ViT-S, ViT-B), and hybrid models (Swin-S, ViT-Adapter, ViT-CoMer). While CNNs efficiently extract local features, they lack global modeling capacity. Transformers capture long-range dependencies but with higher computational cost and reduced robustness in weak-texture scenes. In contrast, our MCVN, designed as a hybrid architecture, effectively balances accuracy, robustness, and efficiency, demonstrating clear superiority and providing informed guidance for model selection in Mars rover cross-view localization under challenging conditions. For instance, MCVN-S achieves an improvement of +6.5% in mask ${\mathrm{A}\mathrm{P}}_{50}^{\mathrm{m}}$ and +4.6% in mask ${\mathrm{A}\mathrm{P}}_{75}^{\mathrm{m}}$ compared to plain ViT-S under the 4 × schedule. Meanwhile, on the MCVLD dataset, the MCVN-B outperforms ViT-CoMer-B with an 6.3% higher $SR$@1.00 m and a 20.4% reduction in Mars rover cross-view localization error, while maintaining a comparable number of parameters.

Table 1. Performance comparison of cross-view feature extraction and localization with different backbones.

Method	#Param	COCO		MCVLD
		${\mathbf{A}\mathbf{P}}_{50}^{\mathbf{m}}$	${\mathbf{A}\mathbf{P}}_{75}^{\mathbf{m}}$	$\mathbf{S}\mathbf{R}$ (%)	$\mathit{E}\mathit{r}\mathit{r}\mathit{o}\mathit{r}$ (m)
Comparison across backbone networks
VGG-19 [74]	21.4	8.5	34.6	9.1	3.74
ResNet-101 [75]	44.5	45.9	36.1	13.7	2.95
ConvNeXt-B [76]	89.1	64.7	44.8	42.3	1.63
ViT-S [58]	87.5	62.1	42.2	54.4	1.32
Swin-T [77]	94.2	66.2	45.9	57.2	1.26
ViT-Adapter-S [59]	96.7	65.8	45.8	58.9	1.11
ViT-CoMer-S [60]	98.6	67.2	47.4	61.5	1.09
MCVN-S (Ours)	99.2	68.6	46.8	68.2	0.89
ConvNeXt-L [76]	178.3	65.4	45.1	54.4	1.46
ViT-B [58]	161.8	65.2	44.5	56.7	1.11
Swin-B [77]	155.4	66.5	46.2	56.8	1.14
ViT-Adapter-B [59]	168.2	67.8	46.9	59.1	1.09
ViT-CoMer-B [60]	169.5	72.4	50.8	62.4	1.08
MCVN-B (Ours)	170.8	70.6	51.3	68.7	0.86

We further evaluate the cross-view localization performance of MCVN and several SOTA backbone networks under different training paradigms, such as SCRL, SwAV, and others. The results, summarized in Table 2, demonstrate that our proposed MarsCVFP training paradigm achieves competitive performance in general visual semantic understanding tasks and outperforms existing SOTA models in Mars rover cross-view localization, showcasing its superior adaptability. Specifically, MarsCVFP with ViT-S achieved a 15.4% improvement in $SR$@1.00 m and a 35.6% reduction in cross-view localization error compared to DINOv2 with ViT-S, the best-performing baseline among other training paradigms. Furthermore, when combined with MCVN-S, MarsCVFP achieves a +7.2% increase in $SR$@1.00 m and a 17.7% reduction in localization error compared to MarsCVFP with ViT-CoMer-S, strongly demonstrating the superiority of our proposed model.

Table 2. Performance comparison of cross-view feature extraction and localization with different paradigms.

Method	Arch.	${\mathbf{A}\mathbf{P}}^{\mathbf{m}}$	$\mathit{S}\mathit{R}$ (%)	$\mathit{E}\mathit{r}\mathit{r}\mathit{o}\mathit{r}$ (m)
Comparison across training paradigms
SCRL [78]	ViT-S	38.7	22.7	2.23
SwAV [57]	ViT-S	39.1	34.3	2.26
BYOL [79]	ViT-S	37.4	29.5	1.79
iBOT [67]	ViT-S	43.2	40.1	1.54
DINO [65]	ViT-S	42.8	45.4	1.48
DINOv2 [66]	ViT-S	43.5	54.4	1.32
MarsCVFP (Ours)	ViT-S	44.6	69.8	0.85
MarsCVFP	ViTAE-S	44.2	72.1	0.83
MarsCVFP	ViT-Adapter-S	45.3	84.8	0.69
MarsCVFP	ViT-CoMer-S	46.3	85.3	0.62
MarsCVFP	MCVN-S	45.6	92.5	0.51
MarsCVFP (Ours)	MCVN-B	46.7	94.6	0.46

4.3.2. Visualization and Analysis of MarsCVFP Model

We list four examples of Mars cross-view localization of our framework in typical scenarios to demonstrate the excellent performance of our framework, which is specifically designed to focus on regions of prominent topographic variation that are commonly observable in both orbital and rover images. It accurately captures the contour edges of surface features such as dunes, prominent rocks, and impact craters, and demonstrates strong robustness in flat terrains and semantically weak scenes, as shown in Figure 6a–c. In addition, a representative failure case under shadow variation is presented in Figure 6d, providing valuable insights into the limitations of the MarsCVFP model in handling complex illumination scenes.

Figure 6. Cross-view localization examples under 40 m × 40 m search range in different scenes. In the orbital images, the darkened red region represents the search range. In position estimation images (0.05 m/pixel), the yellow triangle mark is the estimated position, and the green pentagram mark is the ground truth. Our MarsCVFP model performs well in dune, bedrock, crater, and patterned flat terrain scenes in (a–c), and it shows increased localization error under severe illumination changes and complex shadow scenes in (d).

Dune scene. The presence of dunes and slope streaks on the Martian surface presents a significant challenge for MCVL task. Owing to viewpoint differences, these features exhibit distinct textural appearances in rover and orbital images compared to other terrains. Our model incorporates consideration of such dune-dominated scenes during the construction of the MCVLD dataset and can enable effective Mars rover cross-view localization by establishing correspondences between the two views, as illustrated in Figure 6a.

Rock and crater scenes. Exposed bedrock and impact craters represent typical geomorphological features on the Martian surface, characterized by distinct and stable texture patterns, as shown in Figure 6b. Our model accurately captures the contour edges of these features and achieves precise localization, regardless of their completeness or the proportion of the image they occupy. This demonstrates the robustness of our model in identifying and leveraging geologically meaningful structures for accurate Mars rover cross-view localization.

Flat terrain scene. On flat terrains, the absence of prominent landmarks and the presence of repetitive textures from small rocks and gravel lead to weakly distinguishable feature patterns, so MCVL is more difficult. This example demonstrates that our model addresses this challenge by extracting implicit textural distribution patterns and leveraging the spatial geometric relationships among features, thereby successfully locating the Mars rover’s position on the orbital map, as shown in Figure 6c.

Shadow variation scenes. As shown in Figure 6d, the rover and orbital images were captured under different solar elevation angles, resulting in significant differences in texture appearance and shadow positioning for the same terrain region across different time points. These variations lead to inconsistencies in texture appearance and spatial alignment, introducing localization errors. This is an area that requires further investigation. One potential avenue for improvement is to enhance the images through the illumination-consistent processing and the reduction of noise to improve cross-view localization accuracy.

To gain a more intuitive understanding of the representations learned by MarsCVFP and its cross-view localization performance, we visualize the attention map and spatial correlation map in Figure 7. In the dune scene, the attention maps show that our model focuses on transitional zones with pronounced geomorphological changes between dunes and adjacent plains, while largely ignoring flat-terrain objects such as fine-grained dust and rocks. In the rock and crater scene, attention is directed toward prominent geological structures, including exposed bedrock, craters, and their surrounding areas. In the flat terrain scenes, the model concentrates on sand-gravel mixed regions that present strong discriminative feature and distinctive texture patterns. However, in scenes with shadow variations, the model sometimes confuses illumination-induced artifacts with actual rock features, leading to occasional localization errors.

Figure 7. Visualization of attention maps and spatial correlation maps. In the attention maps, warmer colors represent areas of greater importance as perceived by the MarsCVFP model. In the spatial correlation maps, warmer colors indicate higher likelihoods of rover’s location. In the (a–c) scenes, our model exhibits strong unimodality with accurate cross-view localization. In contrast, scene (d) shows localization failure due to ambiguous multi-peak responses and peak deviations.

From the correlation maps, our model generally produces a single prominent peak in most scenes (Figure 7a–c), enabling accurate cross-view localization. In contrast, inconsistent illumination and shadow patterns in Figure 7d. lead to ambiguous multi-peak responses and peak shifts, resulting in localization failure. This also demonstrates a significant challenge that must be overcome to further improve the model’s robustness in the MCVL task. MarsCVFP achieves strong correlation and consistency between rover and orbital images in most scenarios, effectively capturing cross-view correspondences. This performance stems from the model’s extensive pretraining on a large, diverse real scenarios dataset, its generalization training on the dedicated MCVLD dataset, which covers diverse and challenging Martian terrain, and the architectural advantages of the proposed MarsCVFP model.

4.3.3. Comparison with State-of-the-Art Cross-View Localization Methods on Planetary Surfaces

To validate the effectiveness of our cross-view localization framework, we compare our method with four SOTA architectures: (1) a traditional cross-view localization approach, which employs SIFT feature matching between orthorectified rover images and orbital imagery, has achieved global localization with the Opportunity NaTeCam images [21]; (2) a global localization model employing improved census transform and template matching, deployed on the Perseverance rover in the Mars 2020 mission [80]; (3) an end-to-end cross-view localization model based on contour guidance and cross-scale feature transfer, which is designed for planetary surface scenarios [67]; (4) an advanced model from the domain of autonomous driving on Earth, the HC-Net [81] model, which has achieved leading performance in cross-view localization tasks.

We use two evaluation metrics, success rate $SR$ and $Error$ to quantitatively evaluate the cross-view localization performance of our model and four SOTA methods across different search ranges (40 m × 40 m, 30 m × 30 m, and 25 m × 25 m) on Martian terrains scenes traversed by the Zhurong rover, as shown in Table 3. The quantitative results show that our method consistently outperforms the other four state-of-the-art techniques across all search ranges, significantly improving localization success rate while maintaining the global localization error within acceptable engineering requirements.

Table 3. Performance comparison with SOTA cross-view localization models under varying search ranges.

Approach	Search Range	Localization Success Rate (%)			Localization Error (m)
		d@0.5 m	d@0.75 m	d@1.00 m	Mean	Median
SIFT-based [21]	25 m × 25 m	28.29	31.20	42.86	0.95	1.23
	30 m × 30 m	19.15	32.14	34.65	3.21	3.54
	40 m × 40 m	12.76	26.63	31.42	6.56	7.82
Improved census transform & template matching [80]	25 m × 25 m	35.54	45.30	61.39	0.86	0.79
	30 m × 30 m	32.82	45.30	47.85	1.53	1.14
	40 m × 40 m	28.75	31.58	42.91	2.28	1.72
CG&CSFT-Net [67]	25 m × 25 m	26.90	38.42	48.37	0.91	1.15
	30 m × 30 m	21.48	32.59	40.25	2.65	3.32
	40 m × 40 m	19.22	22.64	37.18	3.94	4.76
HC-Net [81]	25 m × 25 m	0.00	3.74	7.62	2.37	2.15
	30 m × 30 m	0.00	1.37	5.19	7.66	6.54
	40 m × 40 m	0.00	0.00	0.00	18.92	17.43
MarsCVFP & improved template matching (ours)	25 m × 25 m	58.67	91.14	94.65	0.46	0.31
	30 m × 30 m	55.28	86.43	90.87	0.58	0.45
	40 m × 40 m	47.14	74.65	82.42	0.84	0.63

Within the 25 m × 25 m search ranges, our method achieves a localization success rate of 58.67% under the 0.50 m threshold, significantly outperforming the census-based method used on the Perseverance rover (35.54%) and the SIFT-based method (28.29%). Other methods fail to achieve accurate localization at this threshold. At the 0.75 m threshold, our two-stage method success rate increases to 91.14%, outperforming the best-performing learning-based method (CG&CSFT-Net, 38.42%) and the top performing traditional method (census-based, 45.30%). At the 1.00 m threshold, our method further improves SR to 94.65%, meeting the accuracy requirements of practical engineering applications, whereas CG&CSFT-Net and census-based methods only reach 48.37% and 61.39%, respectively. Under the 40 m × 40 m search ranges, all methods experience a drop success rate. Nonetheless, our approach maintains a highest success rate of 82.42% at 1.00 m threshold, outperforming the census-based method (42.91%) and CG& CSFT-Net (37.18%). Regarding localization error, our two-stage framework attains a mean error of 0.84 m and a median error of 0.63 m, outperforming both the census-based method (mean: 2.28 m; median: 1.72 m) and the CG&CSFT-Net method (mean: 3.94 m; median: 4.76 m). The other two methods exhibit errors exceeding 5 m. These results demonstrate the superior robustness and localization accuracy of our two-stage method, underscoring its strong potential for real-scenarios in planetary navigation and localization tasks under varying search ranges constraints.

To provide a more intuitive understanding of each method’s performance, we present representative localization examples under different horizontal field of view (HFoV) experimental settings, including panoramic sites ($HFoV=360°$) and unit-planning sites ($HFoV=120°$) from the Zhurong rover mission. As shown in Figure 8b,e, traditional cross-view localization methods based on SIFT and census algorithms demonstrate robust performance in large-scale bedrock and impact crater terrains. However, the SIFT-based method performs worse than the census-based method in dune field scenes (Figure 8a,d), primarily due to its strong dependence on distinctive keypoints and descriptors, which are significantly degraded under repetitive textures and varying illumination. The census-based method, while generally more robust, also struggles in relatively flat terrains with low feature distinctiveness, where reliable localization cannot be achieved (Figure 8c,f). As shown in Figure 8e, the CG&CSFT-Net method exhibits strong localization performance in crater-rich terrains. However, its robustness and generalization are limited in dune, bedrock, and flat terrains, particularly under weak textures and the absence of large-scale structural cues. Meanwhile, the HC-Net method performs poorly across multiple representative Martian scenes due to the architecture constraints. It relies on façade features such as buildings and vegetation to guide feature alignment, which are largely absent on planetary surfaces. This highlights the limitations of Earth-based methods when applied to the unique environments of Mars. Compared to the four aforementioned SOTA methods, our model consistently achieves more robust and accurate cross-view localization across various Martian terrains, including dunes, bedrock, craters, and flat regions, demonstrating strong potential for global localization. However, as illustrated in Figure 8d–f, its performance declines at unit-planning sites due to the reduced horizontal field of view ($HFoV=120°$). Feature studies with large-scale data may be required to further analyze the impact of HFoV on localization accuracy.

Figure 8. Qualitative comparison of different cross-view localization methods under panoramic sites ($HFoV=360°$) and unit-planning sites ($HFoV=120°$) within 40 m × 40 m search ranges. (a,d) represent dune scenes, (b,e) represent bedrock and crater scenes, and (c,f) represent flat terrains scenes. The estimated results, representing the likelihood of the Mars rover’s location, are overlaid on the corresponding orbital images. The GT mark represents the ground-truth position. Our method produces a continuous probability map over the search space, where warmer colors indicate higher probabilities, and achieves high localization accuracy.

4.4. Ablation Study

In this section, we perform comprehensive ablation studies on the proposed models and loss function, incorporating both quantitative and qualitative evaluations. The individual contributions of each component to Mars cross-view localization performance are assessed using two key metrics: localization success rate ($SR$) and localization error ($Error$). All evaluations are performed under the challenging Martian terrain conditions of various sites traversed by the Zhurong rover, thereby demonstrating the superior performance of our framework and validating the effectiveness of each design choice.

4.4.1. Ablation Experiments on the Components

To validate the effectiveness of the key components in our cross-view localization framework, including the multi-scale feature pyramid structure (MSFPS), the feature interaction module (FIM), and the MarsCVFP training paradigm, we performed ablation studies under a unified experimental setting. Specifically, we adopt the DINOv2 architecture with a ViT-B backbone as the baseline model, with the number of feature encoder layers set to four. This model, trained on the MCVLD dataset, is referred to as the “Base” model. Building upon this baseline, we incrementally added the proposed components, culminating in the final framework: the MarsCVFP framework trained in a few-shot learning manner with the MCVN-B backbone. The quantitative results are summarized in Table 4.

Table 4. Results of the ablation study with different modules over a 40 m × 40 m search range.

Approach	Time (s)	Localization Success Rate (%)			Localization Error (m)
		d@0.50 m	d@0.75 m	d@1.00 m	Mean	Median
Base	2.94	12.75	21.58	30.83	3.46	2.59
Base+MSFPS	7.56	22.34	35.17	42.65	2.37	1.84
Base+MSFPS+FIM	16.78	24.52	42.90	49.92	1.68	1.02
Base+MCVN+MarsCVFP (Ours)	28.33	32.98	62.24	71.36	0.95	0.68

We can observe that when MSFPS is introduced to provide multi-scale and multi-receptive-field CNN features to the ViT backbone (features are directly added), the localization success rate ($SR$) increases from 21.58% to 35.17% at the 0.75 m threshold and from 30.83% to 42.65% at the 1.00 m threshold. Moreover, the mean localization error decreases from 3.46 m to 2.37 m, and the median localization error is reduced from 2.59 m to 1.84 m. Replacing the direct addition operation with the proposed FIM leads to further improvements: a 7.27% increase in $SR$ at the 1.00 m threshold and a 29.11% reduction in localization error. Overall, compared to the baseline model, MCVN-B achieves a substantial performance gain, with a total $SR$ improvment of 19.1% at the 1.00 m threshold, a mean error of 1.68 m, and a median error of 1.02 m in localization Error. These results clearly demonstrate that MSFPS and FIM significantly enhance the plain ViT’s ability to extract cross-view consistent features, enabling more effective aggregation of heterogeneous spatial information across the Martian surface.

Furthermore, when the MarsCVFP training paradigm is used in place of the original DINO model, the localization success rate improves consistently across all thresholds, with the highest gain of 21.44% observed at the 1.00 m threshold. Simultaneously, the mean localization error is further reduced by 43.45%, from 1.68 m to 0.95 m. These gains stem from the MarsCVFP paradigm’s ability to implicitly learn robust cross-domain representations between the rover and orbital imagery under self-distillation training, thereby enhancing the alignment of heterogeneous Martian surface views. However, these improvements come at the cost of increased computational time, likely due to the added architectural complexity and enhanced representational capacity of our framework.

The SOTA performance of our model stems from the synergistic integration of the MCVN network and the MarsCVFP training paradigm. Figure 9a–d present the visualizations of cross-view features extraction using different key components. With the introduction of MCVN, terrain edge contours in both the rover orthophoto features and the orbital DOM features are significantly enhanced, and some fine-grained, distinctive features become more apparent, as shown in Figure 9c. However, due to the inherent domain gap, feature representations from corresponding regions may still exhibit spatial misalignment. When incorporating the MarsCVFP training paradigm, these limitations are effectively addressed. As shown in Figure 9d, MarsCVFP substantially improves the feature similarity between orbiter and rover views, thereby enhancing cross-domain correspondence and achieving more consistent and coherent feature alignment across viewpoints.

Figure 9. Qualitative comparison of (a) “Base”, (b) “Base+MSFPS”, (c) “Base+MSFPS+FIM”, and (d) “Base+MCVN+MarsCVFP (Ours)” models. The MCVN network is composed of the ViT, MSFPS. and FIM modules. We selected the DINOv2 training paradigm with ViT-S network as our “Base” model to demonstrate the effectiveness of the modules, MCVN and MarsCVFP. In the orbital feature images, the black box indicates the estimated position, and the center of the yellow box indicates the ground-truth position.

4.4.2. Ablation Experiment on Loss Functions

To evaluate the gain effects of our proposed loss function ${\mathcal{L}}_{CF}$ on the MarsCVFP model, we conducted comparative experiments against the widely used InfoNCE loss and Triplet loss, all built upon the same baseline loss ${\mathcal{L}}_{raw}$. As presented in Table 5, the proposed ${\mathcal{L}}_{CF}$ loss consistently outperforms the alternatives loss functions across various thresholds, achieving the highest localization success rate improvements of 5.58% and 7.87% over the InfoNCE and Triplet loss, respectively, at the 0.50 m threshold. In terms of localization error, the ${\mathcal{L}}_{CF}$ loss yields a mean error of 0.84 m, representing improvements of 5.62% and 9.68% compared to models trained with InfoNCE and Triplet loss added to the original ${\mathcal{L}}_{raw}$ loss. Furthermore, training with the proposed ${\mathcal{L}}_{CF}$ results in faster model conver-gence. Therefore, in terms of both efficiency and localization accuracy, ${\mathcal{L}}_{CF}$ proves to be the optimal choice. The ablation study result on different loss functions demonstrates that the MarsCVFP model trained with $L_{raw}$ + $L_{CF}$ (ours) effectively guides the network to simultaneously attend to both the coarse- and fine-grained information, significantly enhancing its ability to address the resolution discrepancy between the rover DOMs and the orbiter DOMs. This leads to improved feature alignment across domains and results in optimal localization performance.

Table 5. Results of ablation study with different losses over a 40 m × 40 m search range.

Loss Function	Localization Success Rate (%)			$\mathit{E}\mathit{r}\mathit{r}\mathit{o}\mathit{r}$ (m)
	d@0.50 m	d@0.7 5 m	d@1.00 m	Mean
$L_{raw}$	32.98	62.24	71.36	0.95
$L_{raw}$ + $L_{InfoNCE}$	41.56	70.82	78.49	0.89
$L_{raw}$ + $L_{Triplet}$	34.27	65.31	74.50	0.93
$L_{raw}$ + $L_{CF}$ (Ours)	47.14	74.65	82.42	0.84

In addition, we apply max-pooling with strides $s$ of $[s_1=16, s_2=8, \dots ,s_5=1]$ for ${\mathcal{L}}_{CF}$ computation. We examine the cross-view localization performance using either the specific $s_i$ or the complete ${\mathcal{L}}_{CF}$ for loss computation. As shown in Table 6, an increase of$i$ in $s_i$ generally leads to a decrease in localization error, with a significant decrease observed from $s_2$ to $s_3$. However, from $s_4$ to $s_5$, there is no apparent decrease in localization error, perhaps because the extremely fine-grained feature does not adequately guide the network to perceive coarse-grained information. In contrast, integrating multi-scale levels from $s_1$ to $s_5$ in the configuration of ${\mathcal{L}}_{CF}$ more effectively guides the cross-view localization model to aggregate both coarse- and fine-grained feature information of the Martian surface, resulting in the best localization performance.

Table 6. Effect of loss function application at multi-scale levels on cross-view localization error.

Multi-Scale Levels	Cross-View Localization Error (m)
	Mean	Median
$s_1=16$	0.95	0.86
$s_2=8$	0.94	0.84
$s_3=4$	0.86	0.71
$s_4=2$	0.85	0.69
$s_5=1$	0.85	0.68
$w$/all (Ours)	0.84	0.63

5. Conclusions

We have proposed a two-stage cross-view localization method for the Mars rover that integrates the Mars cross-view feature extraction network (MCVN), the Mars cross-view few-shot training paradigm (MarsCVFP) model, and the robust template matching and validated its effectiveness and stability on 85 unit-planned sites and 20 panoramic sites traversed by the Zhurong rover. The experimental results show that our method improves cross-view feature correlation while preserving the distinctiveness of both orbital and rover features. It outperforms both traditional approaches and end-to-end learning-based cross-view localization models across varying search ranges, maintaining high accuracy, even with coarse prior position. The framework effectively addresses the limitations of the traditional SIFT-based method, which has been used to achieve cross-view localization with the Opportunity rover’s NaTeCam data, as well as the census-based method applied to the Perseverance rover, particularly under challenging terrains such as dunes, bedrock, craters, and flat plains. Ablation studies further confirm the individual contributions of MCVN and MarsCVFP. Future work will explore fully end-to-end cross-view localization methods to enable more efficient localization and navigation on Mars.