Deep Hybrid Fusion Network for Inverse Synthetic Aperture Radar Ship Target Recognition Using Multi-Domain High-Resolution Range Profile Data

Abstract

Most existing target recognition methods based on high-resolution range profiles (HRRPs) use data from only one domain. However, the information contained in HRRP data from different domains is not exactly the same. Therefore, in the context of inverse synthetic aperture radar (ISAR), this paper proposes an advanced deep hybrid fusion network to utilize HRRP data from different domains for ship target recognition. First, the proposed network simultaneously processes time-domain HRRP and its corresponding time–frequency (TF) spectrogram through two branches to obtain initial features from the two HRRP domains. Next, a feature alignment module is used to make the fused features more discriminative regarding the target. Finally, a decision fusion module is designed to further improve the model’s prediction performance. We evaluated our approach using both simulated and measured data, encompassing ten different ship target types. Our experimental results on the simulated and measured datasets showed an improvement in recognition accuracy of at least 4.22% and 2.82%, respectively, compared to using single-domain data.

1. Introduction

Due to its long-range capabilities and all-weather operation, radar automatic target recognition (RATR) plays a crucial role in both military and civilian applications [1,2,3,4,5,6]. High-resolution range profile (HRRP) data can be viewed as the projection of scattering point echoes along the radar’s line of sight [7]. These data encapsulate the size and distribution of the target’s scattering points. Their acquisition process is straightforward, and unlike inverse synthetic aperture radar (ISAR) images, the quality of HRRP data is not compromised by focusing algorithms or non-cooperative target motion. As a result, HRRP target recognition has been extensively studied and implemented in RATR systems. The fundamental process of HRRP target recognition is illustrated in Figure 1.

As shown in Figure 1, feature extraction plays a critical role in target recognition and directly impacts the performance of the recognition system [8]. In HRRP target recognition, there has been a significant shift from traditional feature extraction to the adoption of neural networks for this purpose. Traditional feature extraction methods can be categorized into those with physical interpretability [9] and those without [10,11,12,13]. Although these methods have achieved high accuracy on their respective datasets, the extraction of features heavily relies on the researchers’ expertise. Without such expertise, it is challenging to effectively ensure the accuracy and stability of the algorithms.

Recently, many studies have utilized deep neural network methods to analyze HRRP data, as deep learning can enhance system accuracy by eliminating the need for manual feature extraction and providing better feature representation capabilities. Some of these studies focus on time-domain HRRPs. For instance, a deep learning method for multi-static radar target recognition was used to automatically extract features from HRRP data [14]. Considering the time-shift and azimuth sensitivity of HRRP data, a convolutional neural network (CNN) with large convolution kernels and strides was proposed for target recognition [15]. Additionally, a stacked corrective autoencoder (SCAE) was used to extract features from HRRP, with the average profile employed as a correction term [16]. An enhanced variational autoencoder (VAE) was introduced to capture probabilistic latent features [17], while a sparse encoder was used to learn sparse representations of high-dimensional data, yielding better recognition performance than traditional autoencoders [18]. Furthermore, some studies have focused on the temporal information in time-domain HRRPs and have employed sequential models to address this issue [19,20,21,22,23]. Specifically, by considering the temporal correlations within the range cells of input data, a combination of CNN, bidirectional recurrent neural network (BiRNN), and attention mechanisms was used to improve the robustness of target recognition [19]. Additionally, [21] proposed a deep learning model combining CNN with transformers to recognize the spatiotemporal structures embedded in HRRPs.

In contrast to the studies focusing on time-domain HRRPs, other studies have concentrated on the time–frequency (TF) spectrogram of one-dimensional HRRPs. For example, a two-dimensional CNN was devised to extract spectrogram features from HRRP data [24]. A multi-scale CNN was proposed to address the challenging task of feature extraction in space target recognition based on TF spectrograms, yielding improved recognition results [25]. Moreover, an attentional CNN model with multi-resolution spectrograms was proposed for target recognition [26].

Although these recognition methods have potential in improving recognition performance, they only utilize HRRP data from a single domain. In fact, due to different generation mechanisms, HRRPs in different domains contain information about different aspects of the target [8,27]. By integrating multi-domain information, it may be possible to achieve a more comprehensive representation of the target. Recently, research has proposed combining multi-domain HRRP data for target recognition. Specifically, a multi-input convolutional gated recurrent unit (MIConvGRU) fusion model was introduced to fully leverage three domains of HRRP data (i.e., time-domain HRRP, TF spectrogram, and power spectral density (PSD)) for target recognition [6]. Experimental results demonstrated that this method can significantly enhance radar target recognition performance. However, it does not account for how correlation between initial features from different domains might affect the discriminative ability of the fused features. Therefore, we aimed to design a fusion method that fully considers this correlation to improve the system’s recognition performance, while utilizing only two domains of HRRP data. Fusion recognition methods can be classified by information abstraction level into data-level [28], feature-level [29], and decision-level fusion [30]. Data-level fusion is the simplest but provides minimal performance improvement. Decision-level fusion aggregates predictions from multiple classifiers, adding diversity and increasing decision reliability. However, it operates at the highest abstraction level, leading to some loss of important details. Feature-level fusion is generally considered a more flexible and effective fusion approach [31], and it can compensate for the drawbacks of decision-level fusion. To leverage the advantages of both feature-level and decision-level fusion, we propose using a hybrid fusion method for target recognition, aiming to enhance the overall performance of the model.

In this paper, we propose a deep hybrid fusion network to effectively integrate HRRP data from two domains for ship target recognition. First, in the feature extraction stage, two CNNs independently extract features from the time-domain HRRP data and their corresponding TF spectrogram. These extracted features serve as the initial input for subsequent modules. Second, considering the correlation between the two modalities, we introduce the supervised contrastive learning (SupCon) loss to align the initial features from both domains, with the expectation that the fused and aligned features will make the samples more discriminative. Finally, a neural network-based decision fusion module is implemented following the feature alignment module. This hybrid fusion strategy is expected to further improve the model’s recognition performance. The primary contributions of this work can be summarized as follows:

(1) We explored the effectiveness of combining time-domain HRRP data and their TF spectrogram for ship target recognition. We propose a deep hybrid fusion method to obtain a more comprehensive and discriminative representation of the target, thereby achieving satisfying recognition performance.

(2) In the feature-level fusion part, we design a feature alignment module based on SupCon loss to better fuse the features from the two domains. Compared to traditional feature-level fusion methods, the alignment module fully takes into account the correlation between the two modalities and its impact on the discriminability of the fused features, thereby improving recognition performance.

(3) In the decision-level fusion part, a neural network-based decision fusion module is created to enhance recognition accuracy and reliability. The method’s effectiveness was validated with both simulated and measured data.

This paper is organized as follows: Section 2 covers data formats and preprocessing. Section 3 explains the proposed method. Section 4 validates the method with simulated and measured data. Section 5 discusses results and future work. Section 6 concludes the paper.

2. Data Preprocessing

2.1. Time-Domain HRRP

The time-domain HRRP data used in this study consist of multiple HRRPs obtained before ISAR imaging. The reason for this approach is twofold: first, the primary focus of this paper is on using multi-domain HRRPs for fusion target recognition; second, the recognition process does not require azimuth focusing and is not affected by ISAR image quality. We perform some preprocessing steps to address the three sensitivities of HRRP: (1) we first perform envelope alignment on these HRRPs to overcome the time-shift sensitivity of HRRPs [32]. (2) Given the aspect sensitivity of HRRPs [16,33], studies indicate that the average HRRP yields a smoother and more concise signal shape than individual HRRPs, enhancing the depiction of a target’s scattering characteristics in a specific aspect frame [3,34,35]. From a signal processing viewpoint, the average profile consistently represents the target’s physical structure within a frame.This approach effectively reduces speckle effects and alleviates issues caused by noise spikes and amplitude variations. Therefore, we decided to compute the average profile of the aligned HRRP sequence. According to the literature [7], the average profile is defined as follows:

$${\mathbf{x}}^{AP}=\left[{\displaystyle \frac{1}{M}}\sum _{i=1}^M\left|x_{i1}\right|,{\displaystyle \frac{1}{M}}\sum _{i=1}^M\left|x_{i2}\right|,\dots ,{\displaystyle \frac{1}{M}}\sum _{i=1}^M\left|x_{ir}\right|\right]$$

(1)

where ${\left\{{\mathbf{x}}_i\right\}}_{i=1}^M$ represents an envelope-aligned complex-valued HRRP sequence, ${\mathbf{x}}_i=\left[x_{i1},x_{i2},\dots ,x_{ir}\right]$ is the $ith$ sample in the HRRP sequence, and r is the HRRP sample dimension. Since the aligned HRRP sequence remains in a complex form, and what we commonly refer to as a time-domain HRRP is in a real form, we need to perform a modulus operation on each sample in the sequence before obtaining the average profile.

Additionally, we perform amplitude normalization on the obtained average profile to address the amplitude-scale sensitivity. Our normalization calculation method is as follows:

$${\overline{\mathbf{x}}}^{AP}={\displaystyle \frac{{\mathbf{x}}^{AP}}{max\left({\mathbf{x}}^{AP}\right)}}$$

(2)

After the above operations, the profile ${\overline{\mathbf{x}}}^{AP}$ is used as the final representation of the time-domain HRRP for subsequent processing.

2.2. Spectrogram

For the spectrogram, we first use TF analysis to convert ${\overline{\mathbf{x}}}^{AP}$ from the one-dimensional time-domain to the two-dimensional time–frequency domain. Short-Time Fourier Transform (STFT) captures moment-specific signal characteristics by analyzing a segment of the signal within a time window. For a discrete signal $x\left(k\right)$, its STFT can be expressed as follows:

$$STFT\left(m,\omega \right)=\sum _{k=-\infty }^{\infty }x\left(k\right)w(k-m)e^{-j\omega k}$$

(3)

where $x\left(k\right)$ is the signal to be transformed, and $w\left(·\right)$ is the time window function. After obtaining the STFT representation of the signal $x\left(k\right)$, its spectrogram is the squared magnitude of the STFT:

$$spectrogram\left\{x\left(k\right)\right\}\left(m,\omega \right)={\left|STFT\left(m,\omega \right)\right|}^2$$

(4)

As indicated by the definition of the spectrogram, this representation captures the local characteristics of the HRRP within small distance units. Compared to the time-domain HRRP, it better describes variations in the HRRP. To eliminate the effect of scale, we also normalize the obtained spectrogram. The final spectrogram is then used for further processing. Figure 2 illustrates the processing of the time-domain HRRP and its amplitude-normalized spectrogram representation. After preprocessing, the time-domain HRRP and its spectrogram are treated as two distinct modalities for the subsequent fusion process.

3. The Proposed Method

In this section, we introduce the proposed deep hybrid fusion network for multi-domain HRRP target recognition. As illustrated in Figure 3, the proposed method’s framework can be divided into three main modules: feature extraction, feature alignment, and decision fusion. In the feature extraction module, the preprocessed time-domain HRRP and its corresponding spectrogram, as described in Section 2, are fed into CNNs to extract initial features, which then serve as inputs for the subsequent modules. Then, we align the two sets of initial features to enhance their discriminative power when fused. Finally, the two aligned initial features and their integrated representations are input into the following decision fusion module.

3.1. Initial Feature Extraction

In deep neural networks, CNNs are particularly effective in reducing network complexity and enhancing generalization ability due to their characteristics of local connections and weight sharing [36,37,38,39,40,41]. Therefore, the proposed method utilizes CNNs to obtain initial features from both the time-domain HRRP and its spectrogram. It is worth noting that the preprocessed time-domain HRRP represents the target in one dimension, while the spectrogram is a two-dimensional representation. Consequently, a one-dimensional CNN is used for extracting initial features from the time-domain HRRP, while a two-dimensional CNN is designed to extract features from the spectrogram.

The detailed architecture of the two CNNs is depicted in Figure 4. For the spectrogram CNN, “Conc.128 × 5 × 5/BN/ReLU” indicates 128 feature maps with a kernel size of 5 × 5, followed by batch normalization (BN) [42] and a rectified linear unit (ReLU) activation function [43]. “Dropout” signifies the use of dropout regularization [44]. “Max pool 2 × 2” represents max pooling with a pool size of 2 × 2, while “Fully connected 2560 × 1000” denotes a fully connected layer with 2560 input units and 1000 output units. The softmax layer then produces the predicted labels. The CNN for time-domain HRRP data has a similar layout, with the convolutional and pooling layers utilized in one dimension rather than two.

For both networks, the output from the first fully connected layer is considered the initial feature for each modality. In our approach, if the dimensions of the initial features from the two modalities are different, we need to transform them to the same dimension for further processing. The HRRP branch takes the preprocessed HRRP, ${\overline{\mathbf{x}}}^{AP}$, as input to extract the initial feature $\mathbf{h}$. Assuming $f_H$ represents the CNN for HRRP with parameters ${\psi }_H$, the extraction of the initial HRRP feature can be expressed as follows:

$$\mathbf{h}=f_H({\overline{\mathbf{x}}}^{AP};{\psi }_H)$$

(5)

Similarly, let ${\mathbf{x}}^s$ denote the preprocessed spectrogram and $\mathbf{s}$ represent the initial extracted features from the spectrogram. The extraction process for the initial spectrogram features can be formally expressed as follows:

$$\mathbf{s}=f_S({\mathbf{x}}^s;{\psi }_S)$$

(6)

3.2. Feature Alignment

Our task is to recognize ship targets by combining a time-domain HRRP and its spectrogram. Since these two representations are generated through different mechanisms, they encapsulate distinct information about the target. We hypothesize that they possess a certain degree of complementarity, which forms the basis for our combined recognition approach. By leveraging this complementarity, we aim to enhance the model’s predictive performance. In feature-level fusion methods based on deep neural networks, the most common techniques are concatenation and summation [45], as shown in Figure 5. Concatenation involves directly merging the two features, resulting in a feature with a dimension equal to the sum of the dimensions of the original features. Summation, on the other hand, adds the two features element-wise, resulting in a fused feature with the same dimension as the original features.

Although these two methods, along with some existing weighted feature fusion methods, encompass all the information from the features being fused, they might overlook the correlation between the initial features of the two modalities. This could potentially limit the improvement in sample discriminability. In our task, the two modalities are HRRP and its corresponding spectrogram, which represent two different aspects of the target. As a result, the correlation between these two features is expected to be quite low. In this scenario, the feature distributions of the two modalities in the feature space might be disordered. Specifically, the feature distributions of the same category from the two modalities might not be close to each other. Conversely, the initial feature distributions of different categories from the two modalities might be close or even overlap, as shown in the left part of Figure 6. Directly utilizing the aforementioned fusion methods in such cases can result in the fused features containing similar information from different categories, leading to some ambiguity with other categories. Even though the two modalities’ features have a certain degree of complementarity, this ambiguity can restrict the performance improvement from the fusion. To address this issue, if we can minimize the inclusion of similar information from other classes in the fused features, we can reduce the ambiguity to some extent, thereby increasing the discriminability of the samples and improving the model’s predictive performance.

Therefore, we propose a feature alignment method to solve this problem. We aim for feature alignment to make the distributions of the two modalities of the same class as close as possible, which can reduce the disorder and, consequently, the ambiguity in the fused features. In the field of transfer learning, the maximum mean discrepancy (MMD) loss [46,47] is frequently utilized to assess the distance between multi-domain feature distributions and has demonstrated outstanding performance. We aimed to use the MMD loss function during training to constrain the feature distributions of the two modalities to become closer, thereby achieving alignment. However, due to the low correlation, the feature distributions of the two modalities from different classes might be close to or even overlapping each other. Since the MMD loss function is a global unsupervised alignment method, it could lead to the “misalignment” phenomenon illustrated in Figure 6 [4]. This phenomenon, if it occurs, could be detrimental or even harmful to reducing the ambiguity in the fused features.

To tackle this problem, this paper introduces SupCon loss [48] as an innovative solution. Supervised contrastive learning is a specialized form of contrastive learning that uses label constraints to incorporate a large number of positive and negative sample pairs during the learning process. A positive sample pair consists of two samples that share the same class label, while a negative sample pair comprises two samples with different class labels. It has been shown to enhance the performance of a classification model by learning rich representations of the samples. For a given sample embedding ${\mathbf{z}}_i$, we refer to the samples in the batch that belong to the same class as positive samples ${\mathbf{z}}_p$. The set of positive samples is denoted as $P_i$. Given a sample embedding and its positive sample set, the SupCon loss is defined as follows:

$$L_i^{SupCon}({\mathbf{z}}_i,P_i)=-{\displaystyle \frac{1}{|P_i|}}\sum _{{\mathbf{z}}_p\in P_i}log{\displaystyle \frac{exp(\sin ({\mathbf{z}}_i,{\mathbf{z}}_p)/\tau )}{{\sum }_{j\ne i}exp(\sin ({\mathbf{z}}_i,{\mathbf{z}}_j)/\tau )}}$$

(7)

where j represents the index traversing all samples, and $\tau$ is a scaling parameter. The SupCon loss can be intuitively understood as the average loss defined over each positive pair. In our task, the positive pairs include samples of the same modality and class, as well as samples of the same class but different modalities. In fact, we can rewrite Equation (7) in the form of a combination of two terms as follows:

$$\begin{array}{cc}\hfill & L_i^{SupCon}({\mathbf{z}}_i,P_i)=\hfill \\ \hfill & {\displaystyle \frac{1}{|P_i|}}\sum _{{\mathbf{z}}_p\in P_i}\left(\underset{\mathrm{Tightness}}{\underbrace{-({\mathbf{z}}_i^T·{\mathbf{z}}_p)/\tau }}+\underset{\mathrm{Contrast}}{\underbrace{log\sum _{j\ne i}exp\left(({\mathbf{z}}_i^T·{\mathbf{z}}_j)/\tau \right)}}\right)\hfill \end{array}$$

(8)

We reformulate the SupCon loss into a combination of a tightness term and a contrast term, which elucidates its fundamental purpose. The tightness term aims to maximize the similarity among samples within the same class, thereby promoting the alignment of same-class samples, including those from different modalities in our task. Conversely, the contrast term aims to minimize the similarity between samples of different classes, driving them further apart. This process can be depicted as shown in Figure 7.

Compared to the MMD-based alignment method, the SupCon loss offers the following advantages: (1) it uses label constraints for alignment, which can prevent misalignment; (2) the contrast term further reduces the inclusion of information from other classes in the fused features, thereby decreasing ambiguity and enhancing the discriminability of the samples.

In a batch of samples, the overall SupCon loss is the average of the losses of the N samples:

$$L^{SupCon}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{L}_i^{SupCon}$$

(9)

3.3. Decision Fusion

The core idea of decision fusion is to enhance the overall system’s accuracy, stability, and robustness by aggregating the results from multiple classifiers. In complex real-world scenarios, the outcome of a single classifier is often affected by various external factors, leading to significant risks and uncertainties in target recognition results. Decision-level fusion methods analyze and combine the outputs of multiple classifiers, mitigating the impact of any single classifier’s errors and thereby improving target recognition accuracy.

Therefore, this article adopts a neural network-based decision fusion method to integrate the results of multiple classifiers to enhance model performance. The two initial features described in Equations (5) and (6), after aligning the features, are referred to as ${\mathbf{h}}^{\prime }$ and ${\mathbf{s}}^{\prime }$, respectively. The integrated feature obtained by the element-wise addition of the two is

$$\mathbf{m}={\mathbf{h}}^{\prime }\oplus {\mathbf{s}}^{\prime }$$

(10)

For decision fusion, we construct three base classifiers $f_c^m:x_n^m\to y_n,m=1,2,3$. All three base classifiers are composed of fully connected layers, with network parameters denoted as ${\psi }_{cm},m=1,2,3$. Their inputs are ${\mathbf{h}}^{\prime }$, ${\mathbf{s}}^{\prime }$, and $\mathbf{m}$, respectively. Assuming the prediction vectors of the three classifiers are

$${\mathbf{p}}^m=\left(p_1^m,p_2^m,\dots ,p_k^m\right),m=1,2,3$$

(11)

where k represents the total number of classes in the classification task, and $p_k^m$ is the softmax probability of classifier $f_c^m$ for the $kth$ class. After obtaining the three initial decision vectors, we concatenate them and input the combined vector into a meta classifier $f_c^o$ to obtain the final prediction vector ${\mathbf{p}}^o$. This process can be expressed by the following equation:

$${\mathbf{p}}^o=f_c^o\left(\left[{\mathbf{p}}^1,{\mathbf{p}}^2,{\mathbf{p}}^3\right];{\psi }_{co}\right)$$

(12)

where ${\psi }_{co}$ denotes the parameters of the meta-classifier network. The entire decision fusion process is illustrated in Figure 8. It is important to note that the three initial classifiers must have good classification performance to serve as a solid foundation for decision fusion. Therefore, during training, we impose a cross-entropy loss constraint on the three base classifiers to ensure their basic classification performance. This can be expressed with the following equation:

$$L_B=L_{ce1}+L_{ce2}+L_{ce3}$$

(13)

where $L_{ce1},L_{ce2},L_{ce3}$ are the cross-entropy loss functions for the three base classifier branches, respectively. Additionally, to ensure that the final integrated decision vector is effective for our task, we also impose a cross-entropy loss constraint on it, denoted as $L_T$. Therefore, the total loss function of the proposed deep hybrid fusion network mainly consists of three parts. The first part is $L^{SupCon}$, used by the feature alignment module. The second part is $L_B$, which ensures the performance of the three base classifiers. The final part is the task-specific loss function $L_T$. The overall loss function is as shown in the following equation:

$$L_{total}=\alpha L^{SupCon}+\beta L_B+L_T$$

(14)

where $\alpha$ and $\beta$ are adjustable hyperparameters used to balance these three components. To present our proposed method more clearly, we describe the training process. We train the fusion model using Adam optimizer, with a learning rate of 0.001 and a batch size of 16. The overall training procedure is outlined in Algorithm 1.

Algorithm 1  Training Process of the Proposed Method

1. Set the network architecture for the proposed method, including the number of fully connected layers, the number of units in each fully connected layer, the batch size, and other hyperparameters.

2. Initialize the parameters for the two feature extraction networks ${\psi }_H,{\psi }_S$; the base classifier parameters ${\psi }_{cm},m=1,2,3$; and the meta-classifier parameters ${\psi }_{co}$.

3. while not converged do

4. Randomly sample a batch of examples ${\left\{{\mathbf{x}}_i\right\}}_{i=1}^B$ and their corresponding labels ${\left\{y_i\right\}}_{i=1}^B$ from the entire dataset.

5. Following the preprocessing steps outlined in Section 2, obtain the preprocessed time-domain profile ${\overline{{\mathbf{x}}_i}}^{AP}$ and spectrogram ${\mathbf{x}}_i^s$ for each sample.

6. Use ${\overline{{\mathbf{x}}_i}}^{AP}$ and ${\mathbf{x}}_i^s$ as inputs to obtain the initial features of the two modalities, ${\mathbf{h}}_i$ and ${\mathbf{s}}_i$, through Equations (5) and 6, respectively.

7. Using ${\left\{{\mathbf{s}}_i\right\}}_{i=1}^B$, ${\left\{{\mathbf{h}}_i\right\}}_{i=1}^B$, and ${\left\{y_i\right\}}_{i=1}^B$, compute $L^{SupCon}$ through Equations (8) and (9).

8. In the decision fusion section, obtain the prediction vector ${\mathbf{p}}^o$ using Equations (10)–(12).

9. Calculate the total loss $L_{total}$ and update the network parameters.

10. end while

4. Experiments and Results

4.1. Simulated Data

In our research, we exclusively utilized HRRP data obtained prior to ISAR imaging to validate the efficacy of joint multi-domain HRRPs in target recognition. To more intuitively describe the target’s motion and posture, we analyze the simulation data from the perspective of ISAR images. For the simulation data, we designed 3D models of ten types of ships to create the dataset. It is important to note that each ship category includes two pitch angles for top-view images and two azimuth angles for side-view images. For top-view images, each pitch angle features two azimuth movement patterns. An ISAR image is generated at regular angle intervals within each movement pattern, producing 25 images per movement pattern. Consequently, each pitch angle results in 50 images, with a total of 100 images for both pitch angles combined. The side-view images follow a similar pattern, resulting in 100 top-view images and 100 side-view images for each category, considering the two pitch angles and two azimuth angles. The specific imaging parameters are presented in

Table 1. Detailed motion parameters of ship targets.

Top-View			Side-View
Target	T1	T2–T10	Target	T1	T2–T10
Pitch angle ($\theta$)	80°/85°	80°/85°	Azimuth angle ($\phi$)	10°/15°	10°/15°
Initial azimuth angle ($\phi$)	5°	5°	Initial pitch angle ($\theta$)	40°	40°
Azimuth motion 1	0.04°/s	0.27°/s	Pitch motion 1	0.08°/s	0.51°/s
Azimuth angle interval 1	0.02°	0.132°	Pitch angle interval 1	0.04°	0.240°
Azimuth motion 2	0.08°/s	0.54°/s	Pitch motion 2	0.16°/s	1.01°/s
Azimuth angle interval 2	0.04°	0.211°	Pitch angle interval 2	0.08°	0.384°

. In this table, T1 represents target 1, and the geometric relationship between azimuth and pitch angles is illustrated in Figure 9. The radar simulation parameters are detailed in

Table 2. Settings of radar parameters for simulated data.

Parameter	Value
Center frequency	8.075 GHz
Bandwidth	150 MHz
PRF	200 Hz
Observation time	0.32 s

, which lists the radar’s center frequency, bandwidth, pulse repetition frequency (PRF), and accumulation time. The pitch angle $\theta$ is defined as the angle between the positive direction of the z-axis and the target, while the azimuth angle $\phi$ is defined as the angle between the projection on the $xOy$ plane and the positive direction of the x-axis, with the ship’s bow pointing in the positive direction of the x-axis. Ideally, the typical preprocessed HRRP data of the ten target classes are shown in Figure 10, with their corresponding spectrograms depicted in Figure 11.

To evaluate the proposed method’s performance across different signal-to-noise ratios (SNRs), Gaussian noise with SNRs of 10 dB, 5 dB, and 3 dB was added to the original echoes. For each SNR dataset, we used the top-view data corresponding to a pitch angle of 80° and the side-view data corresponding to an azimuth angle of 10° as the training samples, with the rest used as test samples. The recognition accuracy of the proposed method is shown in

Table 3. Recognition accuracy of the proposed method under different SNRs.

SNR	3 dB	5 dB	10 dB
Acc.	91.31%	93.08%	94.23%

. Additionally, we provide the confusion matrices for the three SNR levels, as shown in Figure 12.

We also conducted comparative experiments using different fusion techniques. In these experiments, the initial feature extraction network structure, batch size, learning rate, and other hyperparameters were kept consistent for fairness. The methods compared include two classical feature-level fusion techniques: the first concatenates the initial features of the two modalities before inputting them into the classifier, while the second adds the corresponding elements of the initial features before sending them to the prediction layer. Additionally, we compared two weighted methods that can learn the contribution of each modality: the extended-GRU (Ex-GRU) fusion method [49] and the attention mechanism (AM)-based fusion method [31]; as well as the MMD-based feature alignment method (MMD_align). Another method under comparison is the MIConvGRU, which combines HRRP data from three domains. Furthermore, to verify the effectiveness of the decision fusion module, we also conducted an experiment using only the proposed feature alignment method (Only_align). Comparative experiments were conducted under different SNRs, and the recognition performance of each method is shown in

Table 4. Comparison of different methods with the proposed method on the simulated dataset.

SNR	3 dB	5 dB	10 dB
Concatenation	88.16%	89.90%	90.75%
Addition	88.23%	89.39%	90.93%
Ex-GRU [49]	88.65%	90.05%	90.60%
AM [31]	87.89%	90.21%	91.27%
MMD_align	87.14%	87.94%	89.38%
MIConvGRU [6]	88.18%	90.61%	92.09%
Only_align	90.79%	92.67%	93.35%
Ours	91.31%	93.08%	94.23%

and Figure 13. From Figure 13, it can be seen more intuitively that the proposed method has a better recognition performance compared to previous fusion methods. Notably, our alignment method outperforms the MMD-based feature alignment method in target recognition. Furthermore, compared to the MIConvGRU approach that combines HRRP data from three domains, our method achieves better recognition performance while using only two domains. Additionally, combining feature alignment with decision fusion for target recognition yields a higher recognition compared to using feature alignment alone.

4.2. Measured Data

We additionally confirmed the proposed method’s effectiveness with measured data obtained via an X-band radar. Like the simulated dataset, the real measured multi-domain HRRP data included preprocessed time-domain HRRPs and spectrograms captured before ISAR imaging. Additionally, the measured data covered ten target categories, with data collected over various periods for each category. Figure 14 shows the typical preprocessed time-domain HRRPs for these ten target categories, while Figure 15 displays the corresponding spectrograms. In this experiment, we randomly selected 50 samples from each category as training samples, with the remaining samples used for testing. The number of training and testing samples for each of the ten target categories is detailed in

Table 5. Details of Training and Test samples for the Ten-target measured Dataset.

Target	T1	T2	T3	T4	T5	T6	T7	T8	T9	T10
Training Samples	50	50	50	50	50	50	50	50	50	50
Test Samples	450	131	230	435	299	120	128	49	56	89

.

Similar to the experiments conducted on the simulated dataset, we also performed a series of comparative experiments on the measured dataset. The comparison methods used are the same as those applied to the simulated dataset. The recognition results, shown in

Table 6. Comparison of different methods with the proposed method on the measured dataset.

	T1	T2	T3	T4	T5	T6	T7	T8	T9	T10	Accuracy (%)
Concatenation	99.11	90.08	95.65	99.77	92.64	100	93.58	100	76.79	75.28	95.23
Addition	93.11	93.89	97.39	99.77	96.99	98.33	94.04	93.88	55.36	92.13	94.94
Ex-GRU [49]	92.00	98.47	95.22	95.63	94.98	99.17	95.41	100	85.71	74.16	93.98
AM [31]	97.33	100	95.22	98.39	96.99	95.00	91.74	95.92	85.71	60.67	94.80
MMD_align	95.11	95.42	96.52	94.71	91.30	95.83	93.12	100	76.79	78.65	93.40
MIConvGRU [6]	97.11	99.23	99.13	99.54	95.98	99.17	92.20	87.76	80.36	73.03	95.71
Only_align	98.00	100	95.22	97.47	98.66	97.50	98.17	95.92	82.14	79.78	96.53
Ours	98.89	98.47	96.96	99.77	99.67	100	96.79	100	64.29	89.89	97.49

, demonstrate that our proposed method achieves a higher recognition rate, further validating its effectiveness. Similar to the results on simulated data, our alignment method surpasses the MMD-based feature alignment method in target recognition. Moreover, compared to the MIConvGRU approach that uses HRRP data from three domains, our method delivers better recognition performance with only two domains. Additionally, integrating feature alignment with decision fusion leads to higher recognition rates than using only feature alignment.

4.3. Ablation Study

In this section, to validate the superiority of the proposed fusion method combining time-domain HRRPs and spectrograms over single-modality recognition, we conducted ablation experiments on both simulated and measured data under different SNRs. The feature extraction networks for the two modalities are as shown in Figure 4. The experimental results are presented in

Table 7. Ablation study.

	HRRP	Spectrogram	Proposed Fusion Method	Accuracy (%)
	✓	×	×	86.88
3 dB	×	✓	×	86.57
	✓	✓	✓	91.31
	✓	×	×	88.70
5dB	×	✓	×	88.86	Simulated
	✓	✓	✓	93.08
	✓	×	×	89.79
10 dB	×	✓	×	88.93
	✓	✓	✓	94.23
	✓	×	×	94.67
	×	✓	×	93.92	Measured
	✓	✓	✓	97.49

. From these results, it is evident that when only a single modality is used for target recognition, the recognition rates are relatively low, regardless of whether the data are measured or simulated. In contrast, our proposed fusion method, which integrates both modalities, consistently yields a higher recognition accuracy.

To more intuitively demonstrate the effectiveness of the proposed method compared to single-modality recognition, we used t-distributed stochastic neighbor embedding (t-SNE) [50] to visualize the features extracted by the proposed fusion method and the two single-modality features in the test set of the measured data. The results are shown in Figure 16. Qualitatively, the figure shows that the proposed fusion method yields a better feature distribution, further confirming the effectiveness of the proposed fusion method over single-modality recognition.

5. Discussion

5.1. Comparison

Table 4 and Figure 13 show the comparison results between our proposed method and other fusion methods on the simulated data. Meanwhile, the comparison results on the measured data are shown in Table 6. From these tables, it is evident that our proposed method consistently outperforms other methods. The comparison experiments can be divided into two parts. First, when only using our proposed alignment method, due to our consideration of reducing the ambiguity of fused features, our method already shows certain superiority compared to some previous fusion methods. This can be seen from the first to the sixth rows of Table 4 and Table 6. Second, when we further applied a decision fusion module, the recognition accuracy improved even more. This is because decision fusion can integrate the output results of multiple classifiers to obtain more stable discrimination results, as shown in the sixth and seventh rows of Table 4 and Table 6.

5.2. Impact of Feature Alignment on Reducing Ambiguity

As mentioned in Section 3, our proposed feature alignment fusion method aims primarily to reduce the ambiguity of fused features. In this section, we quantitatively evaluate the impact of our feature alignment method on reducing ambiguity. When the ambiguity of fused features is minimized, their discriminability should be maximized. We measure this discriminability using the ratio of inter-class distance to intra-class distance (r) [8]. A larger ratio indicates higher discriminability, implying lower ambiguity. We conducted a quantitative analysis of the various comparison methods on the test set of the measured data, and the results are shown in Figure 17. The x-axis in the figure represents various fusion methods, while the y-axis indicates the ratio of inter-class distance to intra-class distance for the resulting fused features. From the figure, it is evident that the fused features obtained through our proposed alignment method have the highest discriminability compared to those obtained through other fusion methods. This demonstrates the effectiveness of our feature alignment method in reducing the ambiguity of fused features.

5.3. Future Work

HRRP data often contain non-target areas, and when these regions are contaminated with noise, it can negatively affect recognition accuracy. This issue requires careful attention, as most current methods ignore the impact of redundant information in non-target areas on target recognition. In future research, we plan to focus on tackling this challenge.

6. Conclusions

In this study, we investigated the effectiveness of combining a time-domain HRRP with its spectrogram for ISAR ship target recognition. We introduced a hybrid deep fusion method that integrates these two modalities through feature alignment and decision fusion. At the feature-level fusion stage, the correlation between the two modalities and its effect on the ambiguity of the fused features were considered, and the alignment module was proposed to enhance the discriminability of the fused features, thereby improving recognition performance. Additionally, to leverage the advantages of hybrid fusion, a neural network-based decision fusion method was employed after the feature alignment module to further enhance prediction performance. The experimental validation on both simulated and measured datasets demonstrates the effectiveness of our approach in ship target recognition, highlighting its potential for practical applications.