Noise-Robust Multiscale Structural Feature for Optical and SAR Image Matching

Highlights

This paper proposes a novel noise-robust multiscale structural feature (NMSF) method for robust optical and SAR image matching to effectively suppress speckle noise and mitigate modality discrepancies.

What are the main findings?

• Multiscale structural maps (MSM) are constructed using standard RTV models with a principled exponential scaling scheme, enabling coherent multiscale information integration.
• A multilayer features of oriented gradient (MFOG) descriptor with square grid nested architecture achieves multiscale integration without dimensional inflation.

What are the implications of the main findings?

• The proposed approach overcomes the noise sensitivity of traditional phase congruency methods, offering robust performance under geometric distortions and nonlinear intensity variations.
• This method provides a highly reliable and efficient solution for multimodal remote sensing applications, with strong generalization across diverse scenes and sensors.

Abstract

Optical and synthetic aperture radar (SAR) image matching is fundamental to numerous remote sensing applications. Although phase congruency-based methods have been widely adopted for this task, the structural features they extract remain inherently sensitive to speckle noise, which significantly degrades matching performance. To address this limitation, we propose a novel noise-robust multiscale structural feature (NMSF) method. Unlike existing relative total variation (RTV)-based approaches that modify the RTV model or rely on phase congruency, our framework employs the standard RTV model with a principled exponential scaling scheme to construct multiscale structural maps (MSM), effectively capturing salient structural information across multiple scales while suppressing noise and mitigating modality discrepancies. Building upon the MSM, we develop an MSM-ORB detector with non-maximal suppression to extract reliable and well-distributed keypoints. We further introduce a multilayer features of oriented gradient (MFOG) descriptor with a square grid nested architecture, which fundamentally differs from the log-polar structures used in prior methods and achieves multiscale integration without dimensional inflation. Extensive experiments on publicly available datasets demonstrate that NMSF consistently outperforms state-of-the-art approaches, delivering superior robustness and accuracy under nonlinear intensity variations, geometric distortions, and severe speckle noise.

1. Introduction

Image matching is a popular research topic in the field of remote sensing image processing, and it has many potential applications such as change detection [1] and image fusion [2]. Nowadays, the available data include optical, infrared, light detection and ranging (LiDAR), and synthetic aperture radar (SAR) images. With these multimodal data, many effective matching methods with high performance have been proposed in recent years [3,4,5]. Despite these advances, robust and accurate matching of optical and SAR images remains a challenge. This difficulty stems primarily from two factors. First, the fundamentally different imaging mechanisms of optical and SAR sensors give rise to disparate intensity and textural representations of the same geographical scene. As illustrated in Figure 1, these nonlinear radiometric discrepancies obscure common visual appearances, making the extraction of shared, modality-invariant features inherently problematic. Second, SAR images are contaminated by inherent speckle noise, as shown in Figure 1b, which degrades image contrast, blurs fine structural details, and significantly impairs the performance of conventional matching methods.

Figure 1. Example of optical and SAR image pair with severe speckle noise and significant geometric and nonlinear intensity differences.

Many optical-to-SAR image matching methods have been proposed, but in general, they are either intensity-based or feature-based [6,7]. Intensity-based methods aim to establish correspondences by directly evaluating the statistical or structural similarity within predefined local windows [8,9]. Ye et al. [10] integrated fast Fourier transform (FFT) and a weighted edge density map into a histogram of oriented phase congruency framework, leveraging the illumination-invariant properties of phase congruency (PC) for cross-modal registration. More recently, Gao et al. [11] introduced a structural similarity measurement strategy based on self-similarity adjacent self-convolution features, specifically designed to enhance matching efficiency in multimodal remote sensing image registration. Another notable approach is the channel features of oriented gradients (CFOG) method [12], which constructs a dense, pixel-wise feature description and employs an FFT-accelerated similarity measure to obtain correspondences. While these intensity-based methods have demonstrated satisfactory accuracy for optical and SAR image matching, they frequently suffer from substantial computational complexity and exhibit pronounced sensitivity to geometric deformations and significant nonlinear radiometric variations between modalities [13,14].

Conversely, feature-based methods achieve registration by extracting the shared and discriminative features invariant to modality differences. This process typically involves three sequential stages: feature detection, description, and matching [15]. Among the most widely used approaches are gradient-based methods, which have demonstrated strong performance in many multimodal registration tasks, particularly when intensity variations between modalities are moderate. However, when applied to optical-SAR matching under severe speckle conditions or significant nonlinear intensity differences, conventional gradient features can experience degraded repeatability and descriptor distinctiveness [16]. To address these challenges, several enhanced gradient strategies have been proposed. Yao et al. [17] integrated a co-occurrence filter (CoF) with a high-order gradient computation strategy to capture more consistent gradient features. Gao et al. [18] employed an average squared gradient to generate partial main orientation maps, yielding more robust gradient orientation information. Similarly, Hu et al. [19] utilized multiscale neighborhood weights to calculate the principal gradient direction of the underlying structure. Zheng et al. [20] further improved descriptor robustness by incorporating second-order gradients with steerable filtering. In parallel, gradient-domain optimization methods have been developed to directly model the consistency of gradient fields between images. Li et al. [21] proposed a differential total variation method that formulates registration as minimizing the total variation in the residual image, exploiting the observation that image gradients are more stationary than intensities under severe distortions. In a broader framework, Li et al. [22] further developed a hierarchical sparse representation approach, where optimally registered images are sparsified hierarchically in the gradient domain with separation of sparse errors. These methods have demonstrated promising robustness to spatially varying intensity distortions on medical and remote sensing images. Nevertheless, gradient-domain optimization methods rely on iterative optimization over the entire image domain, which can be computationally intensive and may encounter local minima when handling large geometric deformations such as significant rotation or scale changes. These approaches are therefore most effective when initial alignment is already close and the primary challenge is intensity distortion rather than large geometric transformation, which is a complementary scenario to the feature-based matching problem addressed in this work.

In pursuit of more robust alternatives, researchers have increasingly explored feature representations based on PC and local self-similarity (LSS) [23,24]. PC-based methods exploit the frequency-domain information to identify stable edge points that are impervious to intensity reversal, which are common in multimodal imagery [25]. Zhang et al. [26] designed the histogram of the orientation of weighted phase (HOWP) method for robust multimodal image matching. Fan et al. [27] proposed a multiscale adaptive binning phase congruency feature for optical and SAR image matching, which combined a nonlinear diffusion-based Harris–Laplace detector with a novel PC descriptor. Zhu et al. [28] introduced the R2DF2 matching method, which employs a rotation-invariant maximum index map derived from Log-Gabor filter responses, achieving notable robustness to both intensity and rotation differences. Liang et al. [29] presented an image method for optical and SAR images based on phase congruency and multi-moment features. Despite their efficacy, PC-based methods are not without shortcomings. On one hand, the computation of the PC across multiple scales is computationally intensive. On the other hand, the PC features can be sensitive to strong speckle noise, which may lead to inaccuracies in both feature detection and description [30].

Concurrently, LSS measures have emerged as a powerful tool for capturing the internal geometric layout of a local region. By computing the similarity of a patch to its surrounding neighborhood, these descriptors offer a high degree of resilience against the complex noise and textural variations inherent in SAR scenes [31]. Xiong et al. [32] proposed the adjacent self-similarity (ASS) method, which designs a minimum self-similarity map and an index map that are largely insensitive to intensity differences and noise. Fan et al. [33] introduced pyramid features of orientated self-similarity, integrating PC into the LSS model to achieving more robust encoding of structural information. Hong et al. [34] developed the reinforced local information of local self-similarity method, which augments feature representations by systematically integrating local image information through a specifically designed regional extraction scheme. Although these advancements have yielded improved matching performance, many existing feature-based methods still struggle to maintain robust noise invariance while simultaneously preserving sufficient discriminative power, particularly in challenging scenarios characterized by severe coherent speckle noise [35,36].

In recent years, deep learning-based methods have shown encouraging potential in optical and SAR image matching [37,38]. Ye et al. [39] proposed a hybrid matching method that utilizes attention-enhanced structural features, effectively combining the strengths of handcrafted priors and learned representations to improve matching accuracy. Cao et al. [40] introduced a PC network which performs convolutional neural network (CNN) operation on PC feature maps to enhance structure feature representation. To mitigate the effect of noise interference, Yang et al. [41] designed a lightweight CNN coupled with an SAR image random mask noise-resistant feature reconstruction network, exhibiting enhanced robustness and superior matching performance. Additionally, Zhou et al. [42] proposed a geometry preserving dense registration network to address local nonrigid deformations frequently encountered between multimodal image pairs. More recently, modality transfer-based approaches have surfaced as a distinct and promising research direction [43,44]. These methods aim to reduce representational discrepancies by translating one image modality into the domain of the other prior to matching. Hong et al. [45] utilized semantic segmentation results to guide generative adversarial networks (GAN) for optical-to-SAR modality transfer, thereby enhancing cross-modal structural consistency. Nie et al. [46] proposed PromptMID, a framework that generates modality-invariant descriptors by integrating diffusion models with vision foundation models and achieves cross-modal feature alignment via prompt-driven guidance. Despite the substantial progress achieved by these learning-based methods, their performance is generally relied heavily on the availability of large-scale, high-quality annotated training data, which are often difficult to obtain in practical multimodal remote sensing applications.

Among feature-based approaches, the relative total variation (RTV) model [47] has recently attracted attention for its ability to decompose images into structural and textural components while preserving salient edges. Several RTV-based methods have been proposed for optical-SAR image matching. Fan et al. [16] developed the multi-orientation RTV (MORTV) method, which modifies the RTV model itself by extending the original two-direction formulation to eight directions to capture more directional structural information. Fan et al. [3] also proposed the modality-invariant structural feature representation (MISFR) method, which integrates RTV with PC by computing PC moments on RTV structural maps, yielding enhanced PC moments for feature detection and description. Pang et al. [48] introduced RTV-scale invariant feature transform (RTV-SIFT), which constructs an RTV iteration space, detects keypoints using RTV-Harris, and builds descriptors on enhanced phase congruency edge maps. More recently, RTV-radiation variation insensitive feature transform (RTV-RIFT) [49] applies RTV for texture suppression and then computes PC on the RTV maps for structure extraction, with descriptors built on maximum orientational mask maps derived from Log-Gabor filter responses. A careful examination of these methods reveals that they fall into two categories: those that modify the RTV model itself and those that use RTV as an intermediate preprocessing step while ultimately relying on PC for core feature representation. Consequently, these methods either incur increased computational complexity from model modification or remain susceptible to speckle noise due to their dependence on PC.

To address the aforementioned challenges and achieve robust matching performance between optical and SAR images, this paper proposes a novel noise-robust multiscale structural feature (NMSF) method. The proposed method consists of three main components. First, we construct a multiscale structural map (MSM) by applying the standard RTV model with a principled exponential scaling scheme, which captures complementary structural information across multiple scales while effectively suppressing speckle noise and textural clutter. Second, we develop an MSM-ORB detector that performs feature point extraction directly on the MSM, followed by non-maximal suppression to ensure a well-distributed keypoint set with improved repeatability. Third, we introduce a multilayer features of oriented gradient (MFOG) descriptor with a square grid nested architecture that integrates multiscale structural information through a layer-wise scheme without inflating the final feature dimensionality.

The proposed method differs fundamentally from existing RTV-based approaches in three key aspects. First, unlike MORTV which modifies the RTV model itself, our method employs the standard RTV model without any modification. Our innovation lies not in altering the model, but in how we utilize it through the MSM construction and the subsequent feature extraction and description stages. Second, in contrast to MISFR, RTV-SIFT, and RTV-RIFT, which all rely on PC for feature detection or description, our method adopts a pure RTV framework. Both feature detection and description operate entirely on RTV-derived structural maps, completely eschewing PC and thus avoiding its inherent speckle sensitivity. Third, our MFOG descriptor adopts a square grid nested architecture that achieves multiscale integration without dimensional inflation, fundamentally differing from the log-polar structures used in all existing RTV-based methods. It is important to emphasize that NMSF, like other handcrafted feature-based methods, operates within the feature-based paradigm and is intended to complement existing gradient-based and gradient-domain optimization methods rather than replace them. The main contributions of this paper can be summarized as follows:

• We propose a pure RTV-based matching framework that performs both feature detection and description entirely on structural maps derived from standard RTV decomposition, completely eschewing phase congruency to avoid its inherent speckle sensitivity.
• We introduce an MSM representation constructed using the standard RTV model with a principled exponential scaling scheme, which establishes explicit scale-to-spatial-layer correspondence for coherent multiscale information integration.
• We develop a novel MFOG descriptor with a square grid nested architecture that integrates multiscale information without dimensional inflation, fundamentally differing from the log-polar structures and concatenation-based strategies used in existing methods.

The remainder of this paper is organized as follows. Section 2 provides a detailed description of the proposed NMSF, including MSM representation and MFOG descriptor. Section 3 presents the experimental results and analysis. Finally, Section 4 concludes the paper.

2. Proposed Matching Method

In this section, the proposed NMSF is introduced to address the inherent challenges of optical and SAR image matching. First, we introduce the MSM representation, which leverages the RTV model to effectively decouple salient structures from noise and texture, thereby providing a cleaner and more stable foundation for subsequent feature extraction. Second, building upon the MSM, we adapt the ORB detector to identify repeatable and structurally significant keypoints. Third, we propose a novel MFOG descriptor, which employs a multilayer architecture to integrate complementary structural information across scales without inflating the feature dimensionality. The general flowchart of the proposed NMSF is illustrated in Figure 2. The subsequent subsections provide a detailed exposition of each component.

Figure 2. Flowchart of the proposed NMSF. Red circles denote keypoints, and yellow lines represent the matching correspondences between keypoints.

2.1. MSM Representation

The major difficulty in optical and SAR image matching resides in modality inconsistency arising from disparate imaging physics. Optical sensors capture reflective and emissive properties of the Earth surface, whereas SAR responds to dielectric and geometric roughness characteristics. Consequently, identical ground features often appear with divergent intensity and textural patterns across the two modalities. While PC-based methods [26,27] have gained attention for their invariance to contrast and illumination changes, they exhibit a critical limitation in the context of SAR imagery: sensitivity to coherent speckle noise. The frequency-domain computations inherent to PC models are susceptible to the multiplicative noise prevalent in SAR, which can corrupt the phase information and lead to spurious edge responses, ultimately degrading both feature detection repeatability and descriptor fidelity [30].

To mitigate this, we introduce RTV model as a robust structural feature representation prior. The RTV model, originally conceived for structure-texture decomposition, offers an attractive alternative to existing filtering-based multiscale representations. Unlike local edge-preserving filters such as anisotropic diffusion filter (ADF) [8], CoF [17], and pointwise shape-adaptive texture filter (PSATF) [36], which operate on local neighborhoods and may struggle to suppress spatially correlated speckle noise in SAR imagery, RTV performs a global optimization that explicitly separates structural components from textural and noise components based on the relative total variation measure. The key insight is that the ratio of windowed total variation to windowed inherent variation is significantly smaller in texture regions than in structure regions, enabling RTV to suppress entire texture patterns rather than merely smoothing local intensity variations. Moreover, unlike PC, which operates in the frequency domain and is susceptible to phase corruption under severe speckle noise, RTV performs spatially adaptive regularization in the spatial domain, effectively suppressing speckle noise and textural clutter while preserving the integrity of primary geometric structures. This property is particularly advantageous for SAR imagery, and by applying RTV to both optical and SAR images, we obtain structure maps that exhibit a significantly higher degree of cross-modal consistency, thereby establishing a more reliable common ground for subsequent feature detection and description.

The extraction of a structural map S from an input image I via the RTV model is formulated as the following optimization problem:

$$\arg \underset{S}{\min }{\displaystyle \sum _p\left\{{\left(S_p-I_p\right)}^2+\lambda \cdot \mathrm{RTV}\left(p\right)\right\}}$$

(1)

where p is the image pixel, and the regularizer term RTV(⋅) is defined as:

$$\mathrm{RTV}\left(p\right)={\displaystyle \frac{G_{\sigma }\ast \left|{\partial }_{x}S\right|}{\left|G_{\sigma }\ast {\partial }_{x}S\right|+\epsilon }}+{\displaystyle \frac{G_{\sigma }\ast \left|{\partial }_{y}S\right|}{\left|G_{\sigma }\ast {\partial }_{y}S\right|+\epsilon }}$$

(2)

Here, G_σ is a Gaussian function with standard deviation σ, and ∂_x and ∂_y represent the partial derivative operators in the horizontal and vertical directions, respectively. ∗ denotes the convolution operation, and ε is a small positive number to avoid division by zero. The parameter λ serves as a balancing weight that governs the trade-off between fidelity to the original image and the degree of structural smoothing. A larger λ yields a more abstract and smoothed structural representation, whereas a smaller λ retains finer details. The parameter σ controls the spatial scale of the texture suppression. A large σ directs the model to suppress patterns over a broader neighborhood, thereby extracting coarser structural elements.

It is important to clarify the role of RTV within our framework. RTV is employed as a structural extraction tool that provides the initial structural representations for both modalities. It is not claimed to be a cross-modal correspondence model in itself; rather, it serves as a preprocessing step that reduces modality-specific textural and noise variations. The subsequent components of our framework are specifically designed to establish cross-modal correspondences from these structural representations. In conventional scale-space construction, such as Gaussian pyramids, the multiscale representation is typically achieved by varying only the spatial scale while keeping the filter characteristics fixed. This approach offers limited flexibility in controlling the trade-off between structure preservation and texture suppression across scales. The RTV model, by contrast, provides two independent parameters: λ controls the degree of structural smoothing, and σ controls the spatial scale of texture suppression. This dual-parameter flexibility is a distinctive advantage of RTV-based multiscale representation. Nevertheless, when RTV is applied with a single parameter pair, it performs structural extraction at only one scale. At that particular scale, weaker structural features may fall below the regularization threshold and be suppressed together with speckle noise. To overcome this limitation and fully exploit the dual-parameter flexibility of RTV, we construct an MSM that captures complementary structural information across different spatial scales. In our framework, the RTV model is implemented on both optical and SAR images to produce their respective structural maps:

$$S = \mathrm{RTV}(I,\lambda ,\sigma )$$

(3)

Recognizing that structural information exists across different scales, and that the optimal scale for matching may vary depending on the scene content and sensor resolution, we construct an MSM. Rather than relying on a single parameter setting, we employ the RTV model with a cascade of parameter pairs (λ_l,σ_l) to generate a stack of structural maps, each capturing salient features at a distinct scale level. This multiscale representation is formally defined as:

$$\mathit{MSM} = \{S_1,S_2,\dots ,S_L\}$$

(4)

where S_l denotes the lth structural map in the stack, and L is the total number of scales. The parameters for each scale are determined by an exponential scaling scheme, ensuring a coherent and progressive coarsening of the structural representation:

$${\lambda }_l = {\lambda }_0·2^{(l-1)/3}, {\sigma }_l = {\sigma }_0·2^{(l-1)/3}, l = 1,2,\dots L$$

(5)

where λ₀ and σ₀ denote the initial base parameters, empirically set to λ₀ = 0.01 and σ₀ = 1.2 in this work.

The multiscale design of MSM explicitly addresses the concern that RTV may suppress weak but repeatable structures at any single scale. By employing multiple parameter pairs, the MSM generates structural maps with complementary characteristics: finer-scale maps (smaller λ and σ) retain localized details and weak structures that might be lost at coarser scales, while coarser-scale maps (larger λ and σ) capture more global and robust structural skeletons that are less sensitive to residual noise and fine textural variations. A structure that may be suppressed at one scale due to its weakness or similarity to noise is likely to be preserved at another scale, ensuring that weak but meaningful structural information is retained across the multiscale representation. This multiscale strategy is conceptually analogous to the hierarchical philosophy in sparse representation methods [21,22], where information is represented at multiple levels to capture structures of varying scales and saliencies.

Figure 3 visually validates the efficacy of the proposed MSM representation. The figure presents the structural map extraction results for a representative SAR image under three distinct parameter settings. It is evident from the results that the RTV model successfully preserves the primary geometric contours and salient structural features across all scales while concurrently suppressing the effects of speckle noise and texture. Crucially, the different structural maps exhibit complementary characteristics: finer-scale maps retain localized details, whereas coarser-scale maps capture more global and robust structural skeletons. This inherent complementarity implies that the MSM can substantially enrich the structural information available for optical and SAR images, thereby providing a more noise-robust foundation for the feature detection and description stages.

Figure 3. Illustration of structural map extraction using the RTV model with different parameter settings. (a) original SAR image. (b) λ₁ = 0.01, σ₁ = 1.2. (c) λ₃ = 0.0159, σ₃ = 1.9049. (d) λ₅ = 0.0252, σ₅ = 3.0238.

2.2. MSM-ORB Feature Detection

The reliable detection of repeatable and distinctive keypoints across optical and SAR images is a prerequisite for successful feature-based matching [14]. Conventional detectors, such as the Difference of Gaussian (DoG) or the Harris–Laplace operator, exhibit degraded repeatability when applied to original optical-SAR image pairs [6,20,27]. This degradation stems from the fact that intensity-based corner responses are inherently unstable under nonlinear radiometric differences and are susceptible to texture and coherent speckle noise existed in SAR data. To address this limitation, we propose a robust detection strategy termed MSM-ORB, wherein feature point extraction is performed not on the original image, but rather on the previously constructed MSM. This design is motivated by two principal observations. First, the MSM representation effectively decomposes each modality into a structural component that is largely invariant to sensor-specific intensity mappings and textural variations. By suppressing speckle noise and texture while preserving salient geometric contours, the MSM yields a substantially higher cross-modal similarity compared to the raw intensity images. Consequently, corner-like responses detected on the MSM are more likely to correspond to repeatable structural landmarks rather than spurious noise artifacts. Second, the MSM provides a multiscale characterization of the scene structural hierarchy, enabling the detector to identify keypoints that are stable across a range of spatial resolutions. It is worth noting that these advantages are not shared by all multiscale representations, as the effectiveness of structural decomposition depends critically on the underlying filtering strategy. Our experimental results in Section 3.5.1 confirm that RTV-based MSM provides superior performance for cross-modal keypoint detection compared to alternative multiscale constructions. This combination of enhanced inter-modal structural consistency and multiscale representation directly contributes to an improved feature repeatability rate.

The procedure of the MSM-ORB detector proceeds as follows. First, a multiscale image pyramid is constructed by stacking the first structural map of the MSM together with its progressively downsampled versions (by a factor of two) derived from the coarser structural layers. This pyramid serves as the input for scale-space feature detection. At each pyramid level, the features from accelerated segment test (FAST)-9 algorithm are employed to efficiently identify candidate corner points. To further refine the quality of the detected keypoints and ensure their localization accuracy, the Harris corner measure is subsequently applied to rank and filter the FAST candidates, retaining only those with sufficiently strong corner responses. Finally, non-maximal suppression with a radius of two pixels is applied to eliminate redundant detections and guarantee a spatially well-distributed set of feature points across the image domain.

Figure 4 provides a qualitative and quantitative comparison of feature detection results on a representative optical and SAR image pair, contrasting the traditional multiscale ORB detector (TM-ORB) with the proposed MSM-ORB. For a fair comparison, both detectors employ identical non-maximal suppression parameters and are configured to retain the top 1000 feature points. As visually evident from Figure 4, the feature points localized by MSM-ORB are mainly situated on distinctive and well-defined structural boundaries. In contrast, TM-ORB detector yields numerous points that are scattered across homogeneous or noise-dominated textural areas, which are less likely to find reliable correspondences in the counterpart modality. Quantitatively, the feature repeatability rate is 20.3% for the original ORB detector and improves markedly to 25.7% for the proposed MSM-ORB. This 5.4% increase in repeatability substantiates the effectiveness of performing feature detection on the MSM representation, confirming its capacity to extract more repeatable keypoints across the optical-SAR modality gap.

Figure 4. Comparison results of different feature detectors.Green dots denote keypoints on optical image and red dots denote keypoints on SAR image.

To endow the detected feature points with rotational invariance, we adopt the intensity centroid technique to assign a dominant orientation to each keypoint. Crucially, however, the orientation computation is conducted on the structural map rather than on the original intensity image. Gradient directions estimated from raw SAR imagery are unstable due to the multiplicative nature of speckle noise, which can induce spurious high-frequency variations and severely bias the computed centroid. By contrast, the structural map provides a smoothed and noise-suppressed representation wherein the underlying geometric structures are faithfully preserved. This distinction is particularly significant because the orientation assignment in the standard ORB detector relies on gradient computations, which are inherently sensitive to nonlinear intensity differences and speckle noise. By performing both feature detection and orientation assignment on the RTV structural maps, we ensure that the gradient information used for orientation estimation is derived from structural contours rather than from noise-dominated textural regions, yielding more consistent orientation assignments across modalities. As a result, the intensity centroid computed on the MSM yields a far more robust and reliable estimate of the principal orientation. Furthermore, in recognition of the frequent occurrence of intensity reversal between optical and SAR images, we constrain the assigned orientation to the range of [0°, 180°). This constraint effectively mitigates the ambiguity introduced by contrast inversion, thereby enhancing the consistency of orientation assignment across modalities.

It is worth emphasizing that the MSM-ORB detector does not modify the core mechanisms of the ORB algorithm, including FAST corner detection, Harris response ranking, and non-maximum suppression. Rather, the contribution lies in the strategic adaptation of this efficient detector to operate on the MSM representation, which yields improved repeatability without sacrificing computational efficiency. This adaptation is best understood in the context of the broader NMSF framework, where MSM-ORB works synergistically with the MSM structural representation and the MFOG descriptor to achieve robust cross-modal matching.

2.3. MFOG Descriptor Construction

The construction of a robust and discriminative feature descriptor is important for establishing reliable correspondences between optical and SAR images. In parallel with gradient-based descriptors, which have demonstrated strong performance in many multimodal scenarios, our MFOG descriptor offers an alternative approach by leveraging structural maps generated through RTV decomposition. This design is particularly advantageous when SAR images are contaminated by severe speckle noise, where gradient computations on raw images may be compromised. Many state-of-the-art descriptors, however, operate on a single-scale local neighborhood surrounding each keypoint [18,20,36]. While computationally expedient, such single-scale representations inherently lack the capacity to fully describe the structural information that manifests across multiple spatial resolutions. This limitation is particularly acute in multimodal scenarios, where the optimal scale for structural characterization may vary locally due to differences in sensor resolution and scene content. To mitigate this shortcoming, recent research has gravitated toward multiscale feature representation methods, which aggregate information from multiple concentric or variably sized support regions [15]. Although these approaches have yielded notable improvements in matching accuracy, they generally resort to a straightforward concatenation of feature vectors derived from each scale. This design choice inevitably leads to a substantial inflation of descriptor dimensionality, thereby incurring an increase in computational overhead during the subsequent matching stage. For instance, when using the same three MSM scales, the dimension of concatenation-based variant is three times the dimension of the proposed MFOG.

To address the dual challenge of achieving multiscale representational capacity while maintaining a compact and efficient descriptor, we propose a novel HOG-like descriptor termed MFOG. The MFOG descriptor is distinguished by a unique multilayer architecture that operates directly upon the previously constructed MSM. In contrast to previous multiscale strategies that concatenate independent feature vectors from different scales, the MFOG employs a layer-wise extraction scheme wherein a single, unified feature region is partitioned into nested sub-regions, each of which derives its gradient information from a different scale of the underlying MSM. This architectural innovation enables a single descriptor to intrinsically encode structural information spanning fine, medium, and coarse scales without any expansion of the final feature dimension. This is the fundamental distinction between MFOG and conventional multiscale descriptors: achieving multiscale integration without dimensionality inflation.

As illustrated in Figure 5, the MFOG descriptor is structured as a three-layer architecture comprising the layers L₁, L₂, and L₃. Each layer performs an identical computational procedure (the accumulation of oriented gradient histograms), but operates on a different structural map from the MSM stack. Specifically, the topmost layer L₁ derives its gradient information from the finest-scale structural map S₁, capturing geometric details and localized edge structures. The intermediate layer L₂ utilizes the medium-scale structural map S₂, providing a representation that is more robust to fine-texture variations and residual noise. The bottommost layer L₃ relies on the coarsest-scale structural map S₃, which encodes the most stable and global structural skeletons. In this configuration, a single descriptor constructed from a given keypoint neighborhood integrates appearance information from multiple spatial scales in a spatially organized manner. The fine-scale layer supplies discriminative local details, while the coarser layers provide robustness against modality-specific distortions and speckle noise. Crucially, this multiscale integration is achieved without augmenting the dimensionality of the output feature vector. This design fundamentally differs from existing RTV-based descriptors such as MORTV, which adopts a log-polar spatial structure with three radial bins and ten angular bins. In contrast, our square grid structure provides uniform spatial sampling that naturally aligns with the ORB detector’s square support region, ensuring consistent feature description across the detection and description stages. The detailed computational procedure for constructing the MFOG descriptor is outlined as follows:

Figure 5. Illustration of the proposed MFOG descriptor. The fan-shaped icons indicate oriented gradient histogram accumulation within the corresponding spatial cells.

• For a given feature point p, we first generate the first three layers of the MSM, i.e., S₁, S₂, and S₃, using the multiscale RTV model defined in Equation (2). As established in Section II-A, lower-indexed structural maps retain finer discriminative details, whereas higher-indexed maps convey coarser structural information. For each structural map S_l(l = 1,2,3), a fixed-size square neighborhood of width 15σ_l (where σ_l is the scale parameter associated with S_l) centered at p is selected as the feature support region. Consequently, three feature regions R_l(p)(l = 1,2,3) are obtained, each offering complementary structural perspectives. To ensure rotational invariance, all three feature regions are subsequently rotated in alignment with the dominant orientation assigned to the keypoint.
• Each rotated feature region R_l(p) is uniformly divided into a 6 × 6 grid of non-overlapping spatial cells. According to the proposed multilayer architecture depicted in Figure 5, this 6 × 6 grid is further partitioned into three nested layers. The outer boundary of the grid corresponds to L₁, the intermediate annular region to L₂, and the central block to L₃. This structured partitioning ensures a hierarchical spatial encoding of the local neighborhood.
• For each spatial cell within the three layers, a histogram of oriented gradients is computed. More importantly, the gradient information associated with each layer is derived from a distinct structural map, thereby aligning the scale of structural representation with the spatial extent of the layer. Specifically, for every cell in the topmost layer L₁, an eight-bin orientation histogram spanning [0, 180°) to account for potential intensity reversal is computed directly on the fine-scale structural map S₁. For every cell in the intermediate layer L₂, an identical eight-bin orientation histogram is computed on the medium-scale structural map S₂. For every cell in the bottommost layer L₃, an identical eight-bin orientation histogram is computed on the coarse-scale structural map S₃.

Let V_l denote the concatenated vector of all cell-level orientation histograms derived from different layers. The final MFOG descriptor for feature point p is then constructed by sequentially concatenating these layer-wise vectors:

$$\mathrm{MFOG}(p) = \{V_1,V_2,V_3\}$$

(6)

As illustrated in Figure 5, the MFOG descriptor aggregates a total of 36 grid-level orientation histograms (each layer contributing 6 × 6 = 36 cells divided among the layers as per the nested partitioning). Consequently, the dimensionality of the MFOG descriptor is fixed at 36 × 8 = 288. This compact dimensionality is a direct consequence of the proposed multilayer architecture, which enables the synergistic fusion of multiscale structural information without resorting to the dimensional inflation characteristic of concatenation-based multiscale approaches.

In summary, the proposed MFOG descriptor confers two principal advantages over conventional alternatives. First, by anchoring gradient computations on the MSM rather than raw images, the descriptor inherits enhanced robustness against nonlinear radiometric differences and speckle noise. Second, the novel multilayer architecture facilitates the efficient integration of multiscale structural cues within a compact feature vector, thereby achieving an optimal balance between discriminative power and computational efficiency. The effectiveness of this architectural design is validated in our ablation experiments, where MFOG substantially outperforms both single-scale variants and the concatenation-based multiscale variant, confirming that the nested square grid architecture, not merely the MSM representation, is critical to achieving superior performance. Following descriptor construction, putative correspondences are established using the nearest neighbor distance ratio (NNDR) criterion, and the final set of geometrically consistent matches is recovered via the fast sample consensus (FSC) algorithm [50].

3. Experiments and Analysis

In this section, we conduct a comprehensive experiment to evaluate the performance of the proposed NMSF method for optical-to-SAR image matching. To thoroughly assess its performance, we compare NMSF against eight state-of-the-art methods, including six traditional feature-based approaches and two recent deep learning-based methods. The traditional methods comprise multiscale histogram of local main orientation (MS-HLMO) [18], R2FD2 [28], ASS [32], weighted structure saliency feature (WSSF) [36], locally normalized image feature transform (LNIFT) [51], and second-order tensor orientation feature transformation (SOFT) [52]. The deep learning-based methods include ReDFeat [37], which recouples detection and description through a mutual weighting strategy and employs a Super Detector with learnable non-maximum suppression layers, and MINIMA-LG [38], which fine-tunes the LightGlue matcher within the MINIMA framework to achieve strong cross-modal generalization capability. All competing methods are implemented with their default parameter settings as specified in the original publications to ensure a fair comparison. To quantitatively measure the matching performance, three widely adopted evaluation metrics, including the number of correct matches (NCM), success rate (SR), and root mean square error (RMSE) are utilized. Higher values of NCM and SR indicate superior matching performance, whereas lower RMSE values signify more precise geometric alignment. In accordance with standard evaluation, an image pair is considered a registration failure if the number of correctly matched points falls below 5, in which case the RMSE is assigned a penalty value of 10 pixels.

3.1. Datasets Description

The evaluation is performed on three distinct public datasets to ensure a thorough assessment of matching robustness under varying geometric, radiometric, and noise conditions. These datasets are carefully selected to provide complementary evaluations: Dataset 1 enables controlled assessment of geometric and noise robustness through synthetic transformations, Dataset 2 provides real-world noisy imagery with inherent speckle contamination, and Dataset 3 evaluates the generalization capability across diverse scenes and sensor characteristics.

• Dataset 1: The primary dataset is the publicly available Optical-to-SAR (OS) dataset introduced in [53]. This comprehensive collection comprises numerous precisely registered optical and SAR image pairs acquired over diverse urban landscapes, including cities such as Beijing, Agra, and Omaha. All images possess a spatial resolution of 1 m. To systematically assess the performance of the proposed NMSF under different conditions, we partition the OSdataset into five experimental groups, each consisting of 200 randomly selected image pairs. Group 1 retains the image pairs in their original form without any synthetic rotation or scaling, establishing a baseline for evaluating matching performance under intrinsic radiometric disparities and speckle noise alone. Group 2 subjects each pair to a random rotation angle uniformly sampled from the interval [0,90°), applied either clockwise or counterclockwise, to evaluate the rotational invariance. Group 3 independently rescales each pair using a random scale factor drawn uniformly from the range [0.5,2], simulating substantial resolution discrepancies between optical and SAR sensors. Group 4 simultaneously transforms each pair with both a random rotation angle and a random scale factor, employing the identical parameter ranges specified for Groups 2 and 3, thereby constituting the most challenging scenario that combines both geometric distortions. Group 5 is specifically designed for noise robustness evaluation, where synthetic Gaussian noise is added to the optical images with variance ranging from 0.01 to 0.09, and speckle noise is simulated in the SAR images by adjusting the equivalent number of looks (ENL) from 9 to 1, following the strategy in [27]. Nine noise levels are defined according to the parameter configurations, where level 1 represents the mildest noise condition with Gaussian variance of 0.01 and ENL of 9, and level 9 represents the most severe condition with Gaussian variance of 0.09 and ENL of 1. This group enables controlled assessment of noise robustness under well-defined and progressively increasing noise levels.
• Dataset 2: To further validate the practical effectiveness of the proposed method under real noise conditions, we randomly select 200 image pairs from the GFGE_SOdataset, originally introduced in [41], to construct Dataset 2. The SAR images in this dataset are acquired by the GF-3 satellite, while the corresponding optical images are sourced from Google Earth. Both modalities share a ground sampling distance of 5 m. These image pairs are characterized by substantial inherent speckle noise and cover a diverse range of challenging landscapes. This dataset serves as a critical testbed for evaluating the method’s robustness against real-world speckle noise that is not synthetically generated, thereby complementing the controlled noise experiments conducted on Dataset 1.
• Dataset 3: To evaluate the generalization capability of the proposed method, we incorporate a third dataset aggregated from various sources in the literature [3,5,17,19,26,28]. This collection contains 50 additional optical and SAR image pairs featuring diverse scenes and sensor characteristics. Specifically, the SAR images are primarily acquired by the TerraSAR-X and Gaofen-3 satellites, while the corresponding optical images are primarily sourced from Google Earth. Notably, these image pairs exhibit a range of geometric transformations, including rotation and scale differences, as well as significant speckle noise contamination in the SAR component. This supplementary dataset provides a benchmark for assessing the practical robustness and generalizability of NMSF in complex scenarios.

3.2. Implementation Details

The proposed NMSF method is governed by several key parameters, which are configured as follows to balance performance and computational efficiency. For the construction of the MSM, we follow the parameter guidelines established in prior work on multiscale structural decomposition. The base parameters are set to λ₀ = 0.01 and σ₀ = 1.2. All remaining parameters of the RTV model are retained at their original default values, as specified in [47]. During feature detection on the MSM, the principal parameters are inherited from the standard ORB detector implementation. Specifically, the non-maximal suppression radius is fixed at two pixels to guarantee a well-distributed set of keypoints.

In the MFOG descriptor, the radius of the feature support region for each keypoint is set to 15σ_l, where σ_l corresponds to the scale parameter of the associated structural map. The multilayer architecture is instantiated with three layers, as described in Section II-C. This configuration yields a compact 288-dimensional descriptor that effectively integrates fine-, medium-, and coarse-scale structural information without incurring excessive computational overhead.

3.3. Noise Robustness Test

To comprehensively evaluate the robustness of the proposed NMSF method against noise interference, we conduct a series of assessments on both simulated and real datasets.

• Evaluation on Simulated Dataset: To validate the effectiveness of RTV in suppressing noise while preserving structural information, we conduct a comparative experiment by replacing RTV with six representative filtering methods on Group 5, while keeping all other components of the matching pipeline unchanged. The evaluated filters include Gaussian filter (GF), ADF [8], CoF [17], PSATF [36], local normalization filter (LNF) [51], and gradient domain guided image filter (GDGIF) [54]. We evaluate the performance of these filters under nine noise levels defined in Section 3.1, separately for optical Gaussian noise and SAR speckle noise. Figure 6 illustrates a representative example of the simulated image pairs under different noise conditions, clearly depicting the progressive degradation in image quality. Table 1 reports the average NCM values achieved by each filter under different noise conditions. Each experiment is repeated ten times and the results are reported as mean ± standard deviation. Among the six competing filters, PSATF and GDGIF demonstrate the best overall performance, followed by CoF, LNF, ADF, and GF in descending order. RTV consistently achieves the highest NCM across all noise levels for both Gaussian and speckle noise, demonstrating superior noise suppression capability. Moreover, the standard deviations of RTV are consistently smaller than those of other methods, indicating greater stability and reliability. The performance gap between RTV and the other filters becomes more pronounced as noise severity increases, confirming that RTV is particularly effective in challenging noise scenarios where other filters struggle to preserve structural information. These results confirm that RTV provides an optimal balance between noise suppression and structure preservation, making it a suitable choice for the proposed framework.
• Building upon the above analysis, we further evaluate the overall matching performance of NMSF against eight state-of-the-art methods using the same simulated dataset with nine noise levels, where each level corresponds to a specific combination of Gaussian noise variance applied to optical images and ENL applied to SAR images. Figure 7 reports the average NCM achieved by all competing methods across the nine noise levels. The proposed NMSF method consistently maintains the highest NCM across all noise levels, exhibiting a notably slower performance decline compared to the other methods. While SOFT and ASS demonstrate a certain degree of noise robustness, and deep learning-based methods such as MINIMA-LG and ReDFeat show competitive performance under mild noise conditions, NMSF achieves the highest overall NCM, particularly under severe noise levels. Traditional methods such as LNIFT and MS-HLMO, which rely respectively on local normalization and Gaussian filtering, exhibit marked performance degradation as noise increases. Although these filters can mitigate certain forms of image degradation, they are less effective at suppressing the multiplicative speckle noise inherent to SAR imagery. The deep learning-based methods, while benefiting from large-scale training, still fall behind NMSF in the most challenging noise scenarios, confirming the advantage of our RTV-based structural representation in providing robust and noise-resistant features for optical-SAR matching. For a more intuitive qualitative comparison, Figure 8 displays the feature matching results for a representative image pair subjected to the third noise level. The proposed NMSF method establishes accurate correspondences along salient structural contours. In contrast, the tradition methods, such as MS-HLMO and LNIFT, produce moderate correct matches with outliers. ReDFeat, and MINIMA-LG achieve relatively better performance than several traditional methods, yet remain less accurate than NMSF. These qualitative observations are consistent with the quantitative results in Figure 7, confirming the superior noise robustness of NMSF.
• Evaluation on Real Dataset: To further validate the practical effectiveness of the proposed method under real noise conditions, we conduct experiments on Dataset 2, where SAR images inherently contain severe speckle noise. Table 2 summarizes the quantitative comparison results averaged over the 200 image pairs for nine state-of-the-art methods. The proposed NMSF method achieves the highest average NCM of 91.7, substantially outperforming the second best method WSSF at 78.6 by a margin of 13.1 correct matches. Correspondingly, NMSF attains a leading SR of 95.0%, signifying its enhanced reliability in establishing valid correspondences across diverse scenes. Furthermore, NMSF yields the lowest RMSE of 2.65 pixels, indicating superior geometric alignment precision. It is worth noting that deep learning-based methods, such as MINIMA-LG and ReDFeat, achieve moderate performance with NCM of 51.3 and 43.4, respectively, yet still fall behind NMSF by a considerable margin. This suggests that although learning-based approaches benefit from large-scale training, their generalization to unseen real noise patterns remains limited, whereas our handcrafted RTV-based structural representation offers more consistent and reliable performance in practical scenarios. For a more intuitive visual comparison, Figure 9 presents the feature matching results for five representative image pairs selected from Dataset 2. The proposed NMSF method establishes dense and accurate correspondences along salient structural contours across all five image pairs. In contrast, most competing methods exhibit mismatches, sparse correspondences, or even complete matching failures, particularly in image pairs with severe speckle noise or complex scene structures. WSSF achieves relatively better visual results among the traditional methods, while the deep learning-based methods show competitive performance on some image pairs but fail on others, indicating limited generalization. Collectively, the quantitative metrics reported in Table 2 and the qualitative evidence provided in Figure 9 demonstrate that the noise-robust multiscale structural representation employed by NMSF offers a clear advantage for optical-to-SAR image matching under real noise conditions. The consistent performance across diverse scenes confirms that our RTV-based framework effectively suppresses speckle noise while preserving the structural information essential for reliable feature matching, validating its practical applicability in challenging real-world scenarios.

Figure 6. Simulated image pairs with different noise levels. (a) Original optical images. (b,c) Simulated optical images with variances of 0.03 and 0.08. (d) Original SAR Image. (e,f) Simulated SAR images with three-look and eight-look.

Table 1. Average NCMs of different filtering methods under various noise levels.

Method	Gaussian Noise Variance
	0.01	0.02	0.03	0.04	0.05	0.06	0.07	0.08	0.09
GF	104 ± 2.22	91 ± 2.13	84 ± 2.25	76 ± 2.05	69 ± 2.16	62 ± 2.20	57 ± 2.32	50 ± 2.38	43 ± 2.31
ADF	136 ± 2.13	128 ± 1.96	116 ± 2.16	105 ± 2.12	93 ± 2.10	85 ± 1.99	72 ± 2.06	63 ± 2.16	57 ± 2.13
CoF	190 ± 1.91	177 ± 1.95	163 ± 1.92	151 ± 1.98	139 ± 1.97	125 ± 1.89	114 ± 2.03	107 ± 2.11	101 ± 2.14
LNF	148 ± 2.05	137 ± 2.07	123 ± 1.98	112 ± 2.23	103 ± 2.11	94 ± 2.15	87 ± 2.16	80 ± 2.28	71 ± 2.17
PSATF	209 ± 1.93	192 ± 1.86	185 ± 1.95	173 ± 1.96	161 ± 1.92	147 ± 1.97	135 ± 1.98	120 ± 2.03	112 ± 2.11
GDGIF	193 ± 1.96	175 ± 2.06	163 ± 1.95	148 ± 1.96	135 ± 1.98	123 ± 2.06	115 ± 2.11	103 ± 2.09	99 ± 2.09
RTV	236 ± 1.82	210 ± 1.94	201 ± 1.92	193 ± 1.87	179 ± 1.96	162 ± 1.90	151 ± 1.86	133 ± 1.95	127 ± 1.91
Method	Speckle Noise ENL
	1	2	3	4	5	6	7	8	9
GF	41 ± 2.43	50 ± 2.34	55 ± 2.29	60 ± 2.26	66 ± 2.18	74 ± 2.20	81 ± 2.23	90 ± 2.16	99 ± 2.13
ADF	55 ± 2.36	61 ± 2.30	70 ± 2.31	81 ± 2.27	90 ± 2.25	101 ± 2.18	109 ± 2.12	123 ± 2.14	130 ± 2.15
CoF	92 ± 2.11	99 ± 2.04	106 ± 2.12	117 ± 1.97	126 ± 1.98	138 ± 2.06	146 ± 1.97	161 ± 1.96	177 ± 1.97
LNF	69 ± 2.15	76 ± 2.24	85 ± 2.32	91 ± 2.16	100 ± 2.11	110 ± 2.07	117 ± 2.16	125 ± 2.09	137 ± 2.15
PSATF	106 ± 2.21	113 ± 2.15	132 ± 2.12	143 ± 2.05	158 ± 1.97	170 ± 1.95	176 ± 1.94	188 ± 1.98	202 ± 1.96
GDGIF	90 ± 2.14	97 ± 2.16	105 ± 2.25	114 ± 2.15	121 ± 2.07	133 ± 2.18	146 ± 2.12	158 ± 2.11	178 ± 2.13
RTV	123 ± 1.93	129 ± 1.97	149 ± 1.98	166 ± 1.89	173 ± 1.94	188 ± 1.93	197 ± 1.87	212 ± 1.89	239 ± 1.83

Figure 7. Average NCM under different noise levels for nine comparison methods.

Figure 8. Qualitative comparison results of nine methods for the image pair with the third level of noise interference. (a) MS-HLMO. (b) LNIFT. (c) ASS. (d) R2FD2. (e) WSSF. (f) SOFT. (g) ReDFeat. (h) MINIMA-LG. (i) NMSF. The red circles represent keypoints and yellow lines denote the corresponding matching pairs.

Table 2. Quantitative comparative results on the Dataset 2.

Criterion	Method
	MS-HLMO	LNIFT	ASS	R2FD2	WSSF	SOFT	ReDFeat	MINIMA-LG	NMSF
NCM	38.7	53.4	55.8	48.6	78.6	59.7	43.4	51.3	91.7
SR	43.5	75.5	78.0	52.5	86.0	51.5	37.0	40.5	95.0
RMSE	7.35	5.28	4.17	5.91	3.77	6.13	7.71	7.18	2.65

Figure 9. Qualitative comparison results of nine methods for five typical image pairs in Dataset 2. The red circles denote keypoints and yellow lines represent the feature matching correspondences.

3.4. Matching Performance Analysis

This subsection provides a comprehensive performance evaluation of the proposed NMSF against eight state-of-the-art approaches, namely MS-HLMO, LNIFT, ASS, R2FD2, WSSF, SOFT, ReDFeat, and MINIMA-LG. The assessment is conducted on two distinct datasets to ensure a thorough investigation of matching robustness under varying geometric and radiometric conditions. Both quantitative metrics and qualitative visualizations are presented to validate the comparative analysis.

3.4.1. Quantitative Analysis on Dataset 1

Table 3 reports the quantitative comparison results on Dataset 1, which is partitioned into four experimental groups with progressively increasing geometric complexity.

Table 3. Quantitative comparative results of nine methods on Dataset 1.

OS Dataset	Criterion	Method
		MS-HLMO	LNIFT	ASS	R2FD2	WSSF	SOFT	ReDFeat	MINIMA-LG	NMSF
Group 1	NCM	40.2	71.6	72.8	77.3	91.8	90.7	73.5	74.7	114.3
	SR	81.5	87.0	100	95.0	100	97.0	83.5	86.0	100
	RMSE	5.65	4.64	2.43	3.26	2.76	2.61	3.93	3.76	2.11
Group 2	NCM	21.3	35.3	64.7	45.8	71.5	79.9	31.7	29.5	103.1
	SR	76.0	71.0	89.0	72.5	94.5	92.5	57.5	52.5	96.0
	RMSE	5.88	5.68	3.63	4.82	3.13	3.06	5.97	6.37	2.37
Group 3	NCM	23.8	25.3	67.1	47.2	70.3	75.2	65.6	27.6	100.2
	SR	73.5	51.5	90.5	85.5	90.5	89.0	77.0	51.5	95.5
	RMSE	6.08	6.97	3.36	3.89	3.72	3.35	4.65	6.79	2.41
Group 4	NCM	17.7	22.6	62.9	35.5	67.6	68.9	29.7	23.4	93.5
	SR	61.5	46.0	82.5	68.5	87.0	84.5	51.5	47.0	93.0
	RMSE	6.85	7.15	3.59	5.17	3.86	3.61	6.59	6.87	2.61
Average value	NCM	25.8	38.7	66.9	51.5	75.3	78.7	50.1	38.8	102.8
	SR	73.1	63.9	90.5	80.4	93.0	90.8	67.3	59.3	96.1
	RMSE	6.12	6.11	3.25	4.29	3.37	3.16	5.29	5.95	2.38

• Group 1 consists of image pairs with no scale or rotation transformation. Under these baseline conditions, the proposed NMSF achieves the highest NCM of 114.3 and a SR of 100%, along with WSSF and ASS which also attain 100% SR. Notably, NMSF yields the lowest RMSE of 2.11 pixels, indicating superior localization accuracy even in the absence of geometric distortions. WSSF delivers a strong performance with an NCM of 91.8, demonstrating its effectiveness in handling radiometric differences and speckle noise. In contrast, LNIFT and MS-HLMO exhibit comparatively lower NCM values, as their filtering strategies provide limited robustness against severe speckle degradation. ReDFeat and MINIMA-LG achieve competitive NCM values of 73.5 and 74.7, respectively, benefiting from their learned feature representations on well-aligned images. SOFT also demonstrates strong performance with an NCM of 90.7, confirming the effectiveness of its tensor-based orientation encoding.
• Group 2 introduces significant rotation differences among the image pairs. As shown in Table 3, NMSF consistently outperforms all competing methods across all three evaluation metrics, achieving an NCM of 103.1, an SR of 96.0%, and an RMSE of 2.37 pixels. SOFT achieves the second-best performance with an NCM of 79.9 and an SR of 92.5%, demonstrating its robust rotational invariance through second-order tensor orientation encoding. WSSF and ASS also demonstrate commendable rotational invariance, with SR values of 94.5% and 89.0%, respectively. In contrast, ReDFeat and MINIMA-LG exhibit a dramatic performance drop under rotation transformations, with NCM values falling to 31.7 and 29.5, respectively. This significant decline is attributed to the absence of explicit rotation normalization mechanisms in these learning-based methods, which heavily rely on training data distributions and lack built-in rotational invariance. R2FD2 exhibits moderate performance, whereas LNIFT and MS-HLMO show a marked decline in both NCM and SR, likely due to the sensitivity of their orientation assignment strategy to speckle noise.
• Group 3 evaluates performance under scale variations, with each image pair subjected to a random scaling factor. The quantitative results clearly indicate that NMSF achieves the highest NCM of 100.2 and the lowest RMSE of 2.41 pixels among all tested methods. SOFT secures the second-best performance with an NCM of 75.2 and an SR of 89.0%, maintaining robustness against scale changes through its multi-scale feature extraction strategy. ReDFeat achieves an NCM of 65.6, benefiting from its multi-scale feature learning capability, which provides moderate robustness to scale variations. WSSF and ASS maintain strong SR values of 90.5%, respectively, but their NCM figures lag behind that of NMSF. MINIMA-LG, which lacks explicit scale-handling mechanisms, drops to an NCM of 27.6 with an SR of only 51.5%, confirming its limited capability to accommodate scale variations. Notably, LNIFT fail to register approximately half of the image pairs in this group, resulting in substantially reduced SR values of 51.5%. This limitation stems from the fact that neither method explicitly incorporates scale-invariant mechanisms during feature extraction or description.
• Group 4 represents the most challenging scenario, wherein each image pair undergoes simultaneous rotation and scaling transformations. Under these combined geometric distortions, NMSF sustains the best overall performance with an NCM of 93.5, an SR of 93.0%, and an RMSE of 2.61 pixels. SOFT achieves the second-best performance with an NCM of 68.9 and an SR of 84.5%, demonstrating its robustness to combined geometric distortions through the synergy of tensor-based orientation encoding and multi-scale representation. WSSF and ASS again deliver comparable SR values of 87.0% and 82.5%, yet their NCM values remain distinctly lower than that of NMSF. ReDFeat and MINIMA-LG exhibit significant performance degradation under simultaneous geometric distortions, with NCM values of 29.7 and 23.4, respectively. This confirms that the absence of explicit rotation normalization, combined with limited scale robustness in MINIMA-LG, severely compromises their performance under complex geometric deformations. The performance of R2FD2 and LNIFT deteriorates sharply due to their inability to accommodate severe scale changes, while MS-HLMO records the lowest NCM as its multiscale Harris detector fails to extract a sufficient number of repeatable feature points.

Overall, the average results across all four groups demonstrate the clear superiority of the proposed NMSF method. NMSF attains an average NCM of 102.8, an average SR of 96.1%, and an average RMSE of 2.38 pixels. SOFT achieves an average NCM of 78.7, an SR of 90.8%, and an RMSE of 3.16 pixels, securing the second-best overall performance among the evaluated methods. WSSF follows closely with an average NCM of 75.3 and an SR of 93.0%. ReDFeat and MINIMA-LG, while performing well under no geometric distortion (Group 1), exhibit considerable performance degradation under rotation and combined transformations due to the absence of explicit rotation normalization and limited scale robustness. The remaining methods yield considerably lower average metrics, confirming the overall advantage of NMSF across all geometric deformation scenarios.

3.4.2. Quantitative Analysis on Dataset 3

To further assess the generalization capability of the proposed method, quantitative experiments are conducted on Dataset 3, which comprises 50 optical and SAR image pairs characterized by diverse geometric transformations and significant speckle noise, as described in Section 3.1. The detailed quantitative analysis results are presented in Figure 10, where a registration failure is indicated by an RMSE value of 10 pixels. Across the 50 image pairs, NMSF consistently achieves the highest NCM and the lowest RMSE among all nine evaluated methods. Notably, NMSF successfully registers all 50 image pairs with RMSE values consistently below 3 pixels, and in several instances below 2 pixels, demonstrating exceptional robustness and reliability. WSSF delivers the second-best performance, registering the majority of image pairs with RMSE values generally below 3 pixels. SOFT follows closely, achieving competitive results on most image pairs, though its performance exhibits greater variability compared to NMSF, particularly under large geometric deformations. ReDFeat and MINIMA-LG achieve moderate performance on image pairs with mild transformations but exhibit notable performance degradation on pairs with severe rotations or scale variations, reflecting the limitations of learning-based methods in generalizing to unseen geometric distortions. ASS demonstrates respectable results on a subset of image pairs, consistently outperforming MS-HLMO, LNIFT, and R2FD2. In contrast, MS-HLMO, LNIFT, and R2FD2 exhibit multiple registration failures, especially on image pairs with severe noise or substantial scale and rotation differences, underscoring their limited capacity to handle complex modality variations.

Figure 10. Quantitative comparison results for nine methods on Dataset 2. (a) NCM. (b) RMSE.

Among all competing methods, only NMSF, WSSF, and SOFT achieve 100% success rates. However, NMSF substantially outperforms WSSF and SOFT in terms of average NCM and maintains uniformly lower RMSE values, confirming its superior robustness and precision. The deep learning-based methods, while competitive under mild conditions, fail to consistently generalize across the diverse scenes and transformations present in Dataset 3, highlighting the advantage of our handcrafted RTV-based structural representation in providing stable and reliable matching performance across varying conditions. These quantitative findings on Dataset 3 corroborate the conclusions drawn from Dataset 1 and Dataset 2, affirming that NMSF provides robust and accurate optical-to-SAR image matching under challenging real-world conditions.

3.4.3. Qualitative Analysis

To provide an intuitive visual comparison of matching performance, Figure 11 and Figure 12 present the qualitative results for representative image pairs from Dataset 1 and Dataset 3, respectively. Figure 11 displays matching results for four groups of image pairs with varying geometric transformations, while Figure 12 presents results for representative image pairs from Dataset 3, which feature diverse scenes and varying geometric distortions. A close inspection of these figures reveals that the proposed NMSF method consistently identifies and correctly matches the shared structural patterns between optical and SAR images across a wide range of challenging conditions, including severe speckle noise, nonlinear intensity variations, and substantial geometric distortions such as scale and rotation differences. The correspondences established by NMSF are densely and accurately distributed along salient structural contours across all test pairs.

Figure 11. Qualitative comparison results of nine methods for four groups of image pairs in Dataset 1. The red circles denote keypoints and the yellow lines denote the matching correspondences.

Figure 12. Qualitative comparison results of nine methods for five typical image pairs in Dataset 3.The red circles denote keypoints and yellow lines denote the matching correspondences.

Among the competing methods, SOFT and WSSF produce visually coherent matching results on the majority of test pairs, particularly when geometric deformations are relatively small. SOFT demonstrates robust rotational invariance through its second-order tensor orientation encoding, maintaining reliable matches under pure rotation. However, its performance degrades under simultaneous rotation and scale transformations. WSSF achieves competitive results but exhibits visibly reduced NCMs in cases involving severe noise or combined geometric distortions. ASS shows moderate performance on image pairs with mild transformations but struggles with significant geometric deformations. R2FD2 demonstrates acceptable robustness to pure rotation but displays a clear performance decline and even fails when scale variations are introduced. ReDFeat benefits from its multi-scale feature learning mechanism, achieving moderate robustness to scale variations. However, due to the absence of explicit rotation normalization, its performance drops markedly under rotation transformations, particularly large-angle rotations. MINIMA-LG, which inherits the detection characteristics of SuperPoint without dedicated rotation or scale handling mechanisms, exhibits the most significant performance degradation under geometric distortions among the learning-based methods, frequently producing mismatches or even complete matching failures under severe transformations. LNIFT and MS-HLMO frequently generate misaligned matches, especially on image pairs with severe geometric transformations or high levels of speckle noise, reflecting their limited capacity to handle modality variations. LNIFT, lacking scale-space design, fails entirely under scale transformations, while MS-HLMO’s multiscale Harris detector struggles to extract sufficient repeatable feature points under combined geometric distortions.

The satisfactory matching performance attained by NMSF can be ascribed to two principal factors. First, the conversion of the original optical and SAR images into structural maps via the RTV model effectively suppresses modality-specific radiometric discrepancies and noise interference, while the construction of MSM yields a robust and hierarchically organized representation of scene structures across multiple spatial scales. Second, the synergistic combination of the MSM-ORB detector and the MFOG descriptor plays a critical role. The MSM-ORB detector, equipped with a non-maximal suppression strategy, reliably extracts repeatable and structurally meaningful feature points from the MSM. Concurrently, the MFOG descriptor captures rich multiscale structural information in a layer-wise manner, enhancing both discriminative power and robustness to residual distortions without incurring an increase in feature dimensionality. Together, these design elements endow NMSF with a distinct advantage in achieving accurate and reliable optical-to-SAR image matching under severe modality differences.

3.5. Ablation Study

To analyze the contribution of each core component within the proposed NMSF method, a comprehensive series of ablation experiments is conducted. The evaluation dataset is constructed from three sources to ensure diverse and representative testing conditions. From Dataset 1, we randomly select 50 image pairs from each of the four groups (Groups 1–4), yielding 200 image pairs that cover baseline conditions, rotation, scale, and combined geometric transformations. From Dataset 2, we randomly select 50 image pairs characterized by severe real speckle noise. From Dataset 3, we include all 50 image pairs featuring diverse scenes and sensor characteristics. This selection strategy results in a total of 300 test image pairs, providing a comprehensive benchmark for evaluating the contribution of each component under varying geometric, radiometric, and noise conditions. Three criteria, including NCM, SR, and RMSE, are used to evaluate the results.

3.5.1. Ablation Analysis of MSM Representation

To validate the effectiveness of the MSM, we separately replace it with six widely adopted multiscale representation strategies, including Gaussian scale space (GSS), anisotropic diffusion scale space (ADSS), co-occurrence scale space (COSS), local normalization scale space (LNSS), pointwise shape-adaptive texture scale space (PSATSS), and multi-orientation RTV scale space (MORTVSS). Throughout this comparative experiment, both the MSM-ORB detector and the MFOG descriptor are retained in their original configurations to isolate the effect of the structural representation. Table 4 summarizes the average quantitative results obtained by the seven multiscale representation variants across the combined test data.

Table 4. Average results of different multiscale representation methods.

Criterion	GSS	ADSS	COSS	LNSS	PSATSS	MORTVSS	MSM
NCM	28.6	54.3	63.4	55.6	77.3	97.2	96.8
SR	49.7	67.3	80.0	68.3	88.3	97.7	97.3
RMSE	6.26	5.06	3.89	4.72	3.43	2.61	2.43

Among the six competing multiscale representations, PSATSS achieves better performance with an NCM of 77.3 and an SR of 88.3%, followed by COSS with an NCM of 63.4 and an SR of 80.0%. LNSS and ADSS yield moderate results with NCM values of 55.6 and 54.3, respectively, while GSS produces the lowest performance with an NCM of 28.6 and an SR of 49.7%. MORTVSS, which is constructed using the modified RTV model with eight directional orientations, achieves an NCM of 97.2 and an SR of 97.7%, representing a substantial improvement over all other competing methods. The proposed MSM achieves an NCM of 96.8 and an SR of 97.3%, which are comparable to MORTVSS, while yielding the lowest RMSE of 2.43 pixels among all methods.

It is worth noting that MORTVSS achieves a slightly higher NCM than MSM, which can be attributed to the fact that MORTV extends the standard RTV model from two directions (horizontal and vertical) to eight directions, thereby capturing more comprehensive directional structural information. However, our MSM, constructed using the standard RTV model, achieves highly competitive performance with only a marginal difference in NCM (96.8 vs. 97.2) while outperforming MORTVSS in terms of RMSE (2.43 vs. 2.61 pixels). This demonstrates that the standard RTV model, when properly configured with a principled exponential scaling scheme, is sufficient to construct an effective multiscale structural representation for optical-SAR matching. The comparable performance between MSM and MORTVSS validates our design choice of employing the standard RTV model, which avoids the additional computational complexity associated with the multi-orientation extension while still achieving state-of-the-art matching accuracy.

The proposed MSM substantially surpasses the other five competing methods, including GSS, ADSS, COSS, LNSS, and PSATSS, across all evaluation criteria. In contrast to these methods, the MSM leverages the RTV model to adaptively filter out image noise and suppress textural clutter while preserving salient structural contours. This dual capability effectively mitigates the inherent modality discrepancies between optical and SAR imagery. Furthermore, by providing a hierarchically organized set of structural maps, MSM offers a more reliable and consistent foundation for both feature detection and subsequent description, thereby enhancing keypoint repeatability and descriptor distinctiveness. These results verify the advantage of the MSM representation in optical-to-SAR image matching and confirm that the standard RTV model, despite its simpler formulation compared to MORTV, is a suitable and effective choice for constructing multiscale structural representations.

3.5.2. Ablation Analysis of MFOG Descriptor

To assess the contribution of the proposed MFOG descriptor, we construct three degraded descriptor variants while keeping the MSM representation and MSM-ORB detector unchanged. The variants are defined as follows. MFOG-L derives gradient information exclusively from the coarse-scale structural map, MFOG-S relies solely on the fine-scale structural map, and MFOG-C adopts a conventional concatenation strategy wherein gradient histograms are computed on all three structural maps independently and then concatenated into a single high-dimensional vector. In addition, to evaluate the impact of the layer number on the descriptor configuration, we construct two variants with different nested layers: MFOG-2L (a two-layer nested square grid) and MFOG-4L (a four-layer nested square grid). Table 5 reports the average NCM, SR, and RMSE for the four descriptor configurations.

Table 5. Average results of different descriptors.

Criterion	MFOG-L	MFOG-S	MFOG-C	MFOF-2L	MFOG-4L	MFOG
NCM	31.7	48.1	73.3	69.5	90.4	96.8
SR	63.3	72.7	91.0	86.3	94.7	97.3
RMSE	5.24	4.56	3.18	2.87	2.62	2.43

The proposed MFOG descriptor achieves the highest overall performance, with an average NCM of 96.8, an SR of 97.3%, and an RMSE of 2.43 pixels. MFOG-C, which integrates multiscale information through direct concatenation, yields an NCM of 73.3 and an SR of 91.0%, confirming the value of incorporating multiscale structural information. However, MFOG-C incurs a substantially higher feature dimensionality of 864, which is three times that of the proposed MFOG (288 dimensions), leading to increased computational overhead during descriptor matching and reduced matching efficiency. This comparison clearly demonstrates that the proposed MFOG achieves superior performance with only one-third of the dimensionality of the conventional concatenation strategy. The significant performance gap between MFOG and MFOG-C can be attributed to the inherent architectural difference. MFOG-C simply stacks independently computed histograms from different scales without establishing meaningful cross-scale relationships. In contrast, MFOG employs a square grid nested architecture that partitions a unified feature region into three nested layers, where each layer derives gradient information from a distinct structural scale. This design enables the descriptor to encode multiscale structural information in a spatially organized and coherent manner, achieving more discriminative feature representations without increasing dimensionality.

A comparison between the single-scale variants reveals that MFOG-S (NCM of 48.1, SR of 72.7%) substantially outperforms MFOG-L (NCM of 31.7, SR of 63.3%), suggesting that finer structural details provide more discriminative information for establishing reliable correspondences. Nevertheless, both single-scale variants are markedly inferior to MFOG, highlighting the necessity of multiscale integration. These findings validate that the multilayer architecture of MFOG, built upon the square grid nested spatial structure, effectively encodes rich multiscale information in a compact feature representation, fundamentally differing from conventional log-polar descriptors and concatenation-based multiscale strategies used in existing methods. This design achieves an optimal balance between discriminative power and computational efficiency, representing a key innovation of our approach.

Furthermore, the comparison among different layer configurations validates the rationality of the three-layer setting. As shown in Table 5, MFOG-2L yields an NCM of 69.5 and an SR of 86.3%. The limited representational capacity of two layers is insufficient to capture the full spectrum of structural information required for robust cross-modal matching. Conversely, MFOG-4L achieves a competitive NCM of 90.4 and an SR of 94.7%. However, its feature dimension is nearly twice that of the three-layer MFOG, which significantly increases the computational overhead during descriptor matching. More importantly, a four-layer partition within a fixed-size feature region may result in an overly narrow innermost layer, leading to insufficient spatial sampling and increased sensitivity to local noise, which explains why its performance does not surpass that of the three-layer MFOG. Therefore, the three-layer MFOG achieves an optimal balance between multiscale representational capacity, descriptor distinctiveness, and computational efficiency.

These findings validate that the multilayer architecture of MFOG, built upon the square grid nested spatial structure, effectively encodes rich multiscale information in a compact feature representation, fundamentally differing from conventional log-polar descriptors and concatenation-based multiscale strategies used in existing methods. This design achieves an optimal balance between discriminative power and computational efficiency, representing a key innovation of our approach.

3.5.3. Ablation Analysis of NMSF

To systematically validate the independent contributions of each core component within the proposed NMSF framework, we construct three degraded variants by removing or replacing individual modules while keeping the others unchanged. NMSF-NoMSM removes the MSM representation entirely, using the standard ORB detector and HOG descriptor on raw images. This variant serves as the baseline to evaluate the contribution of the structural representation. NMSF-NoMORB retains the MSM representation and the MFOG descriptor, but replaces the MSM-ORB detector with the standard ORB detector applied directly to raw images. This variant isolates the contribution of the MSM-ORB detector by assessing the impact of removing its adaptation to the MSM. NMSF-NoMFOG retains both the MSM representation and the MSM-ORB detector, but replaces the MFOG descriptor with a single-scale HOG descriptor computed on the MSM. This variant isolates the contribution of the MFOG descriptor. The full NMSF method, which integrates all three components, is denoted as NMSF-All.

Table 6 reports the average NCM, SR, and RMSE for each variant. The full NMSF-All achieves the best performance with an NCM of 96.8, an SR of 97.3%, and an RMSE of 2.43 pixels. Removing MSM (NMSF-NoMSM) causes the most significant performance degradation, with NCM dropping to 18.2 and SR falling to 25.3%, confirming that the structural representation is the foundation of the framework. Removing the MSM-ORB detector (NMSF-NoMORB) reduces NCM to 37.5 and SR to 48.3%, indicating that the detector’s adaptation to MSM provides substantial additional value beyond using the raw image input. Removing MFOG (NMSF-NoMFOG) results in an NCM of 48.1 and an SR of 72.7%, demonstrating that the multilayer descriptor architecture contributes meaningfully to the overall performance. The progressive performance decline confirms that each component contributes incrementally and that the synergistic combination of all three modules yields the best results.

Table 6. Experimental results of ablation studies of NMSF.

Method	NCM	SR	RMSE
NMSF-All	96.8	97.3	2.43
NMSF-NoMSM	18.2	25.3	8.76
NMSF-NoMORB	37.5	48.3	7.59
NMSF-NoMFOG	48.1	72.7	4.56

4. Conclusions

In this paper, we have presented a novel noise-robust multiscale structural feature (NMSF) method for optical and SAR image matching. The proposed method first decomposes the source images into structural maps by employing the standard RTV model, and subsequently constructs an MSM representation that effectively captures salient geometric information across multiple spatial scales. Building upon the MSM, a robust MSM-ORB detector incorporating a non-maximal suppression strategy is developed to extract reliable and repeatable feature points. Furthermore, a novel MFOG descriptor is introduced to encode multiscale structural information for each feature point in a compact, layer-wise manner, thereby enhancing discriminative power without inflating the feature dimensionality. Extensive experimental evaluations on publicly available optical-to-SAR image datasets demonstrate that the proposed NMSF method consistently outperforms eight state-of-the-art approaches. The results confirm that NMSF exhibits superior robustness against nonlinear intensity differences, geometric distortions including scale and rotation variations, and the speckle noise interference, thereby achieving highly encouraging matching performance. In future work, we will focus on enhancing the adaptability of the NMSF framework to handle low-quality remote sensing imagery and explore its applicability to other multimodal registration tasks. We also acknowledge that our method may suffer performance degradation in scenes with insufficient structural information. Future work will explore integration with learning-based techniques or gradient-domain optimization methods to further enhance adaptability and registration precision under challenging conditions.