Multi-Channel Physical Feature Convolution and Tri-Branch Fusion Network for Automatic Modulation Recognition

Abstract

Automatic modulation recognition (AMR) plays a critical role in intelligent wireless communication systems, particularly under conditions with a low signal-to-noise ratio (SNR) and complex channel environments. To address these challenges, this paper proposes a three-branch fusion network that integrates complementary features from the time, frequency, and spatial domains to enhance classification performance. The model consists of three specialized branches: a multi-channel convolutional branch designed to extract discriminative local features from multiple signal representations; a bidirectional long short-term memory (BiLSTM) branch capable of capturing long-range temporal dependencies; and a vision transformer (ViT) branch that processes constellation diagrams to exploit global structural information. To effectively merge these heterogeneous features, a path attention module is introduced to dynamically adjust the contribution of each branch, thereby achieving optimal feature fusion and improved recognition accuracy. Extensive experiments on the two popular benchmarks, RML2016.10a and RML2018.01a, show that the proposed model consistently outperforms baseline approaches. These results confirm the effectiveness and robustness of the proposed approach and highlight its potential for deployment in next-generation intelligent modulation recognition systems operating in realistic wireless communication environments.

1. Introduction

AMR is a crucial process for determining the specific modulation format applied to a given signal, serving a pivotal function in diverse signal processing domains like cognitive communications, spectrum surveillance, and electronic warfare. In cognitive communication frameworks, AMR facilitates the automatic adjustment of modulation parameters by communication devices in response to varying channel conditions, enabling real-time adaptation to environmental changes. Conversely, in non-collaborative settings such as electronic warfare, AMR aids in identifying the modulation scheme of intercepted signals, thereby supporting demodulation, target recognition, and optimal interference management [1].

When faced with radio frequency signals of unknown characteristics, AMR plays a crucial role as an initial step, enabling devices to demodulate both standard and non-standard modulation schemes. Effective AMR techniques optimize the utilization of transmission medium resources, thereby improving the interference resilience of contemporary cognitive radio systems. Systems employing adaptive modulation schemes can utilize AMR to continuously monitor channel conditions, dynamically adjusting the modulation scheme to achieve optimal resource allocation [2,3,4]. The advancement of efficient AMR algorithms is considered a promising and essential approach to effectively address increasingly intricate application scenarios. As a result, the progress in these algorithms has drawn substantial interest from researchers and professionals in the field of communication systems. In conventional practice, AMR methods are typically classified into two major groups: likelihood-driven and feature-driven approaches. Likelihood-based approaches formulate AMC as a composite hypothesis testing task, in which multiple hypotheses are evaluated to represent various modulation schemes. These methods are grounded in the Bayesian framework, which guarantees optimal performance in terms of classification accuracy under the minimum error misclassification criterion [5,6]. Under the likelihood-based (LB) framework, the likelihood of the observed signal is evaluated for each possible modulation hypothesis, and the selection of the modulation type is carried out via a likelihood ratio test [7].

Feature-based (FB) methods focus on obtaining different features in the signal, which are then used for classification [8]. Common LB techniques include the ALRT [9,10], GLRT [11], and HLRT [12]. In the FB approach, recent advancements have incorporated techniques such as phase diagram entropy. For example, the authors of [13] use phase diagram entropy to effectively characterize modulations in underwater acoustic communication, enhancing recognition and classification accuracy. The authors of [14] applied this entropy method to spread spectrum modulation recognition, demonstrating its robustness in the presence of noise and interference. The authors of [15,16] introduced a Wavelet-based Adaptive Modulation Recognition Network (WAN) that synergistically integrates Digital Signal Processing (DSP) with deep learning, significantly improving modulation recognition in low-SNR conditions. The authors of [17] utilized compressive sensing to directly reconstruct signal features for digital modulation recognition, enhancing efficiency while maintaining accuracy. The authors of [18] proposed a modulation classification method that combines compressed sensing with high-order cumulants and cyclic spectrum, achieving high accuracy in cognitive radio signal recognition under low-SNR conditions.

However, traditional methods discussed above have notable limitations. They often lack sufficient discriminative power when dealing with complex models and are heavily reliant on predefined features [19]. To address these limitations, AI methodologies—especially machine learning (ML) and deep learning (DL)—have been widely adopted in recent years across computer vision, speech classification, and information retrieval. Methods grounded in machine learning learn informative representations directly from raw samples to encode task-relevant information. Over diverse application domains—computer vision [20], natural language processing [21], recommender systems [22], object detection [23], and anomaly detection [24,25]—they have demonstrated substantial success.

A variety of deep-learning families—CNNs [26], RNNs [27], GNNs [28], and Transformers [29]—have been introduced in recent years. Taken together, they have shown notable effectiveness in the previously discussed application areas. Then, deep-learning-based AMR techniques have attracted growing interest. Unlike traditional approaches that depend on manual feature extraction, modern deep models learn representations and perform classification simultaneously in a unified end-to-end system, facilitating automatic feature learning and classification within a single model. The specific meanings of the abbreviations used in this article are shown in Table 1.

Table 1. List of abbreviations and acronyms.

Abbreviation	Full Form
ALRT	Average likelihood ratio test
GLRT	Generalized likelihood ratio test
HLRT	Hybrid likelihood ratio test
ANN	Artificial neural network
SVM	Support vector machine
HMM	Hidden Markov model
CNNs	Convolutional neural network
RNNs	Recurrent neural network
GNNs	Graph neural network
LSTM	Long short-term memory network
SPWVD	Smooth pseudo-Wigner–Ville distribution
BJD	Born–Jordan distribution

To strengthen architectural design, neural networks are now often optimized with deep learning methods; notable families are CNNs, RNNs, Transformers, and hybrid variants. For instance, in refs. [30,31], multi-layer CNNs are utilized to extract features from IQ signals for classification and recognition tasks. In [32], complex convolutional techniques are applied to extract multi-dimensional features, improving classification accuracy. The method in [33] relies on a dual-layer LSTM to derive AP features from the received signal and drive the classifier. Moreover, refs. [34,35,36] combine RNNs with CNNs to simultaneously capture temporal and spatial information, facilitating feature complementarity. In [37], a custom convolutional block is designed to capture spatiotemporal correlations within modulated signals through various asymmetric convolutional kernels. Finally, ref. [38] presents a dual-path network, where one path utilizes CNNs for time domain and spectral feature extraction, and the other employs LSTM to capture temporal dependencies.

In addition to optimizing network architectures, signal preprocessing is also a critical step in AMR [39]. A range of preprocessing strategies has been introduced, including the extraction of spectral features through specific processing methods [40], statistical features [41], constellation diagrams [42,43], SPWVD images, and IQ signals. Other approaches focus on extracting amplitude/phase features [38,44], and combining IQ signals with separate I and Q components. For instance, introduced a feature fusion method that integrates manual features, the SPWVD, and the BJD using CNNs, which significantly enhances the discriminative power of the extracted signal features.

Originally proposed by Vaswani et al., the Transformer architecture is based on attention mechanisms and was designed for NLP sequence-modeling tasks. Through concurrent cross-position attention—absent in classic recurrent or convolutional models—the Transformer effectively captures distant dependencies and global contextual information. In [45], a combination of CNN for spatial feature extraction and Transformer for modeling global dependencies was used to improve recognition performance. Similarly, ref. [19] employed CNNs for local feature extraction, followed by a Transformer to capture sequence dependencies, which are then input to a GNN for classification, yielding strong recognition reliability.

To adapt the Transformer framework for image-level categorization, the ViT [46] partitions each image into fixed-dimension patches, which are subsequently represented as tokens and fed into the Transformer as a sequence. By operating on this sequence of patch embeddings, ViT can capture global contextual relationships across the image, effectively overcoming the restricted receptive field inherent to traditional convolution-based approaches.

As evident from the literature, most current network models process signal inputs either as standalone IQ signals or as preprocessed IQ signals. Additionally, model architectures are predominantly implemented in a fusion format, where classification outputs are generated through simple concatenation. Given these limitations, along with the reliance on single-path architectures and fixed feature fusion strategies in existing modulation recognition methods, this paper proposes a new framework designed to improve the performance of modulation recognition. In the designed architecture, a CNN ingests multi-channel inputs containing physical prior features, while a Bi-LSTM layer is employed to capture two-way spatiotemporal correlations and modulation cycle characteristics, and a three-branch fusion pathway using ViT for constellation diagram.

The main contributions and innovations of this work are as follows:

• Multi-channel physical prior feature fusion input: The original complex signal is converted into multiple physically meaningful channels, including IQ (real and imaginary parts), phase, phase difference, amplitude, second-order spectrum fast Fourier transform(FFT), and fourth-order spectrum (higher-order spectrum). Four CNN channels process the IQ signal, together with one original-signal channel, emphasizing physical interpretability rather than treating IQ or spectra in isolation.
• Three-branch joint structure (CNN, BiLSTM, ViT): Each branch processes different features for final comprehensive discrimination. The CNN branch uses the above multi-channel inputs; the BiLSTM branch captures modulation periodicity and long-term temporal dependencies; and the IQ signal is converted into a constellation diagram for modulation-type identification via a Vision Transformer. Fusing signal- and image-based cues yields more accurate results.
• Path attention fusion module: Instead of simple concatenation, this module adapts to varying feature dependencies across modulation types. It enables the model to adjust under diverse SNR conditions while ensuring stable feature quality. The three path outputs are adaptively weighted to generate the final feature vector for classification and identification.

The remainder of this paper is organized as follows. Section 2 introduces the signal and identification tasks. Section 3 introduces the architecture of each branch in the proposed three-branch architecture and the signal processing methods in CNN. In Section 4, the complete three-branch architecture is outlined. The experiments and their discussion are presented in Section 5, while Section 6 contains the conclusions.

2. Signal Model and Task

Modulation is a fundamental operation in wireless communication systems, in which the parameters of a high-frequency carrier are varied according to the information-bearing base band signal. By mapping the information onto the carrier, modulation enables reliable and accurate data transmission over practical communication channels. The choice of modulation format strongly influences spectral efficiency and implementation complexity, as well as the system’s robustness to noise and interference. In a conventional wireless link, the signal observed at the receiver can be modeled as follows:

$$x\left(t\right)=s\left(t\right)\ast h\left(t\right)+n\left(t\right),$$

(1)

where $s\left(t\right)$ is the noise-free complex base-band envelope of the received signal; ∗ denotes the convolution operation; $n\left(t\right)$ is Additive White Gaussian Noise with zero mean and variance ${\sigma }_n^2$; $h\left(t\right)$ is the impulse response. The IQ representation is highly flexible. For mathematical operations, it simplifies computation and transformation processes; for hardware design, it enables compact architectures and efficient implementation, the signal can be expressed using orthogonal decomposition as $x\left(t\right)=I\left(t\right)+\mathrm{j}Q\left(t\right)$, where $I\left(t\right)=\mathrm{Re}\left[x\right(t\left)\right]$ denotes the in-phase component, and $Q\left(t\right)=\mathrm{Im}\left[x\right(t\left)\right]$ represents the quadrature component. To analyze various modulation schemes, the transmitted signal can be represented as $s\left(t\right)$, where the characteristics of the modulated signal are identified under different conditions.

$$s\left(t\right)=\left[A_m\sum _n{a}_{n}g(t-n{T}_s)\right]·\cos \left(2\pi (f_c+f_m)t+{\varphi }_0+{\varphi }_m\right),$$

(2)

In the parameter set, $A_m$ refers to the modulation amplitude and $a_n$ to the symbol sequence. $g\left(t\right)$ denotes the pulse-shaping waveform within one symbol duration. The symbol period is denoted by $T_s$, with $f_c$ and $f_m$ specifying the carrier and modulation frequencies, and ${\varphi }_0$ and ${\varphi }_m$ indicating the initial and modulation phases.

For AMR tasks, the receiver discerns the modulation method of the sent signal by evaluating either the raw received waveform or related statistical measures. This process can be mathematically represented as follows.

$$y=\arg \underset{i\in \{1,2,\dots ,N\}}{\max }\mathrm{Pr}(y=i\mid Q\left(r\left(t\right)\right)),$$

(3)

$Q\left(r\right(t\left)\right)$ represents the feature extraction function for the received signal, $Pr(·)$ represents the posterior probability, and y is the final predicted class result [15]. Accurately identifying the modulation format in modulation recognition relies on extracting discriminative features from the received signal. The effectiveness of such algorithms is usually assessed through their classification accuracy and computational efficiency.

In practice, we do not reconstruct a full physical transmitter to synthesize $s\left(t\right)$ within this work; instead, we directly use the complex base band IQ sequences provided by the public datasets RML2016.10a and RML2018.01a as realizations of the received signal $x\left(t\right)$. During the construction of these datasets, a noise-free complex base band signal $s\left(t\right)$ is first generated under a predefined modulation scheme (e.g., BPSK, QPSK, 8PSK, 16QAM, 64QAM, etc.) using simulated or experimental communication chains, and then passed through channel impairments $h\left(t\right)$ and corrupted by additive white Gaussian noise $n\left(t\right)$ to obtain the received waveform $x\left(t\right)=s\left(t\right)\ast h\left(t\right)+n\left(t\right)$, with each sample annotated by its true modulation label. Here, $s\left(t\right)$ can be regarded as the complex base band envelope corresponding to different modulation schemes; its discrete-time counterpart forms a zero-IF, normalized IQ sequence, and the subsequent multi-branch network discriminates among modulation types by exploiting the resulting differences in their temporal and IQ-plane features.

3. The Proposed Method

We outline the detail of the proposed three-branch modulation classification model in this section, which combines domain knowledge with deep learning-driven feature extraction. Specifically, the model consists of three parallel branches—CNN, BiLSTM, and ViT —each designed to capture unique and complementary features from raw IQ signals.

The CNN branch utilizes multiple handcrafted features derived from the IQ sequence, such as amplitude, phase, differential phase, and higher-order frequency components. These features are processed through multi-path convolutions to extract comprehensive representations from over both temporal and frequency domain representations. The BiLSTM branch processes the original IQ sequence directly, learning long-range temporal dependencies using a bidirectional recurrent structure. Simultaneously, the ViT branch converts the IQ signal into a 2D constellation diagram and employs Transformer-based self-attention mechanisms to capture global spatial relationships.

Each branch generates a high-level feature vector that represents distinct aspects of the input signal. These features are then integrated using a Path Attention Module (PAM), which ensures robust and accurate modulation classification across a variety of wireless communication scenarios.

3.1. IQ Signal Processing for Multi-Channel CNN

Wireless receivers often model incoming signals as complex numbers composed of in-phase (I) and quadrature (Q) elements. Amplitude and phase information—key to distinguishing various modulation formats—are jointly represented by these components. In this study, the IQ signal is modeled as a two-dimensional real-valued matrix ${\mathbf{X}}_{\mathrm{IQ}}\in {\mathbb{R}}^{L\times 2}$, in which L stands for the temporal sequence length, and the columns map to the in-phase and quadrature components, respectively.

To further enhance the model’s discriminative ability, we compute several additional features derived from the original IQ sequence. Specifically, the amplitude is calculated as $A\left(t\right)=\sqrt{I{\left(t\right)}^2+Q{\left(t\right)}^2}$; the phase as $\varphi \left(t\right)=\arctan \left({\displaystyle \frac{Q\left(t\right)}{I\left(t\right)}}\right)$, and the instantaneous frequency, such as the phase difference, is given by $\Delta \varphi \left(t\right)=\varphi \left(t\right)-\varphi (t-1)$. These features are concatenated channel-wise to form a multi-channel time-series matrix ${\mathbf{X}}_{\mathrm{CNN}}\in {\mathbb{R}}^{C\times L}$, where $C\in \{5,6,7\}$ represents the number of derived features used. This enriched representation is then fed as input to the CNN branches.

3.2. Multi-Channel CNN

As can be seen in Figure 1, the proposed model employs a multi-branch CNN architecture, which processes a seven-channel input derived from the original complex IQ signal. These channels include the real and imaginary components, amplitude, phase, differential phase, and the real as well as imaginary components of the FFT spectrum. To capture a wide range of signal characteristics from both time and frequency domains, the model utilizes five parallel branches. The first branch employs three 1D convolutional stages to operate on five time domain features, where each stage includes batch normalization, a ReLU activation function, an SE mechanism, and max pooling. The SE module adaptively reweighs the relevance level of every feature channel to the target task, allowing the model to focus on modulation-relevant components while suppressing irrelevant or noisy features. This branch generates 256 output channels, with a temporal resolution reduced to $L/8$.

Figure 1. Multi-channel CNN branch graph.

The second branch extracts frequency domain features using four convolutional layers applied to the FFT spectrum, producing 512 channels at a resolution of $L/16$. The third and fourth branches handle higher-order spectral features, derived by applying the FFT to the squared and fourth-power IQ signals, respectively. These branches capture nonlinear signal structures and weak harmonics that are often overlooked in standard spectral analysis, which is particularly valuable for distinguishing complex modulation schemes. The fifth branch, in contrast, directly down-samples the original five time domain features without convolution, preserving the raw information.

To expose frequency domain cues to the CNN branch, each IQ segment is transformed by a fast Fourier transform (FFT). The resulting spectral magnitude and phase features are concatenated with time domain channels (I, Q, amplitude, phase, and phase difference) to form a unified multi-channel input. This design allows the convolutions to learn temporal and spectral patterns within a single representation, improving separability for modulation families that appear similar in the time domain but differ in their spectral signatures. In practice, the spectral channels also highlight bandwidth usage, line spectra, and weak harmonics at moderate-to-high SNRs, thereby strengthening the model’s robustness without incurring prohibitive computational cost.

Once the temporal dimensions of all branches are aligned, their outputs are concatenated into a unified 1797-channel feature map. A subsequent 1 × 1 convolution operation decreases the number of channels to 512, followed by global average pooling and flattening to produce the final CNN output, denoted as $F_{\mathrm{CNN}}$. All convolutional layers in the model use the ReLU activation function to introduce nonlinearity. The modular structure of the model allows for easy expansion or modification, such as integrating attention mechanisms or cyclostationary feature extractors, which strengthens the model’s capability to perform across various wireless signal recognition scenarios.

The ReLU function, which was introduced in earlier research, has become the most widely adopted choice for modulation classification tasks. As opposed to traditional activation functions exemplified by sigmoid and tanh, ReLU provides faster training, improved classification accuracy, better generalization, and effectively addresses the vanishing gradient problem. By introducing sparsity into the hidden units, as shown in Equation (4), the ReLU function enhances the efficiency of representation and reinforces the model’s robustness and adaptability when applied to classification tasks.

$$f\left(x\right)=\max (0,x)=\left\{\begin{array}{cc}x,\hfill & \mathrm{if}x\ge 0\hfill \\ 0,\hfill & \mathrm{if}x<0\hfill \end{array}\right..$$

(4)

This architecture leverages both physical signal priors and deep representation learning, allowing for reliable and precise classification across various modulation types. The dimensional representation of the process is summarized in Table 2.

Table 2. Multi-channel hierarchical output process table.

Layer (Name)	Output Shape	Description
Input	[B, 7, L]	7-channel IQ signal input
TimeConv1	[B, 64, L/2]	Conv1d (5 → 64, k = 3, p = 1) + BN + ReLU + SELayer + MaxPool (2) (BRNM)
TimeConv2	[B, 128, L/4]	Conv1d (64 → 128, k = 3, p = 1) + BRNM
TimeConv3	[B, 256, L/8]	Conv1d (128 → 256, k = 3, p = 1) + BRNM
FFTConv1	[B, 64, L/2]	Conv1d (2 → 64, k = 3, p = 1) + BRNM
FFTConv2	[B, 128, L/4]	Conv1d (64 → 128, k = 3, p = 1) + BRNM
FFTConv3	[B, 256, L/8]	Conv1d (128 → 256, k = 3, p = 1) + BRNM
FFTConv4	[B, 512, L/16]	Conv1d (256 → 512, k = 3, p = 1) + BRNM
FFT2Conv1	[B, 64, L/2]	Same as FFTConv1, input is FFT of squared IQ
FFT2Conv2	[B, 128, L/4]	Same as FFTConv2
FFT2Conv3	[B, 256, L/8]	Same as FFTConv3
FFT2Conv4	[B, 512, L/16]	Same as FFTConv4
FFT4Conv1	[B, 64, L/2]	Same as FFTConv1, input is FFT of ${\mathrm{IQ}}^{ 4}$
FFT4Conv2	[B, 128, L/4]	Same as FFTConv2
FFT4Conv3	[B, 256, L/8]	Same as FFTConv3
FFT4Conv4	[B, 512, L/16]	Same as FFTConv4
RawFeatures	[B, 5, L]	Raw input channels: real, imag, phase, dphase, amplitude
Concat	[B, 1797, min_len]	Concatenate all branches, align time dimension
FusionConv	[B, 512, min_len]	Conv1d (1797 → 512, kernel = 1) + BN + ReLU
GlobalAvgPool	[B, 512, 1]	AdaptiveAvgPool1d (1)
Flatten	[B, 512]	Flatten layer

⁴ Input is the Fast Fourier Transform of the IQ signal raised to the fourth power.

3.3. Vision Transformer

In this paper’s design, the second branch, ViT, as illustrated in Figure 2, processes the input through patch embedding and positional encoding applied to the constellation graph derived from the IQ sequence. The [CLS] token, an additional learnable global vector, is prepended to the input token sequence before each sample. This token participates in all computations throughout the process, allowing the transformer to incorporate information from the entire input. As a result, the token helps represent the overall feature output. The subsequent sections describe this process in detail, and the overall architecture is shown Figure 3.

Figure 2. ViT* (where the asterisk denotes the extra learnable [CLS] token) with CLS token.

Figure 3. ViT- based feature extraction framework.

The first four modules in the diagram primarily focus on converting the IQ signals into constellation diagrams. These diagrams are then transformed into sequences, with positional information added, in preparation for input into the transformer.

To capture the structural modulation characteristics, the raw IQ signal sequence is converted into a 2D constellation map. Each IQ pair($I_t$, $Q_t$) is treated as a complex symbol and plotted onto a 2D grid. A 64 × 64 grayscale image is then created using normalized binning and density mapping, providing a visual representation that is suitable for input into the Vision Transformer branch.

The constellation map $\mathbf{X}\in {\mathbb{R}}^{H\times W}$ is divided into non-overlapping patches, where each patch has a spatial size of $p\times p$. Consequently, the map is partitioned into $N={\displaystyle \frac{HW}{p^2}}$ patches. Let $X_i\in {\mathbb{R}}^{p\times p}$ denote the i-th patch, and $\mathrm{Flatten}(·)$ be the vectorization operator that reshapes a patch into a column vector. Then, the patch embeddings are obtained as

$${\mathbf{x}}_i=\mathrm{Flatten}\left({\mathbf{X}}_i\right)\in {\mathbb{R}}^{p^2},i=1,2,\dots ,N,$$

(5)

The 64 × 64 image is sliced into 8 × 8 patches to become a patch sequence. Then, to enable the model to have sequence awareness, position encoding was introduced. The encoded sequences were concatenated, and a learnable token was positioned at the head of the sequence to generate the final global representation, resulting in a sequence length of 65.

Each patch is embedded by a linear mapping as:

$${\mathbf{z}}_i=\mathbf{E}{\mathbf{x}}_i+{\mathbf{p}}_i\in {\mathbb{R}}^d,$$

(6)

where $\mathbf{E}\in {\mathbb{R}}^{d\times P^2}$ is a learnable projection matrices; ${\mathrm{p}}_i\in {\mathbb{R}}^d$ is position code of the ith patch; the construction of input sequences is achieved by adding learnable classification tokens:

$${\mathit{Z}}_o=\left[{\mathit{z}}_{\mathrm{cls}};{\mathit{z}}_1;{\mathit{z}}_2;\dots ;{\mathit{z}}_N\right]\in {\mathbb{R}}^{(N+1)\times d},$$

(7)

It is then input into the ViT and processed by the transformer encoder module. The structure of the ViT follows the same principles as the original transformer. Every transformer layer includes:

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left({\displaystyle \frac{Q{K}^{\top }}{\sqrt{d_k}}}\right),$$

(8)

for every head:

$${\mathrm{head}}_i=\mathrm{Attention}(Q{W}_i^Q,K{W}_i^K,V{W}_i^V),$$

(9)

Through this design, each ${\mathrm{head}}_i$ concentrates on specific information patterns and can model relationships between positions. Once concatenated, these outputs are multiplied by matrix $W^o$ (size $n_h{d}_v\times d$) to form the final matrix.

Multi-Head:

$$\mathrm{MultiHead}(Q,K,V)=\mathrm{Concat}({\mathrm{head}}_1,\dots ,{\mathrm{head}}_n)W^o,$$

(10)

where Q, K, and V denote the query, key, and value matrices; $W_i^Q$, $W_i^K$, and $W_i^V$ are the learnable projection matrices for the i-th attention head; and $W^o$ is the output projection matrix. The dimensionalities d and $d_k$ represent the embedding dimension and the per-head projection dimension, respectively.

Multiple identical layers form the Transformer encoder, where each layer integrates a multi-head self-attention block with a feed-forward neural network. In order to counteract the vanishing gradient phenomenon prevalent in deep architectures, each sub-layer incorporates residual connections alongside layer normalization. The feed-forward sub-layer follows a standard fully connected network structure. As shown in Figure 4, by employing multi-head attention, every token in the sequence can access information from all other tokens, with computed weights quantifying the relevance of their positions. This process effectively captures dependencies across sequence elements. By distributing computations across multiple attention heads, each independently calculating attention weights, the model is enabled by the mechanism to attend to different characteristics of the sequence. The outputs from all heads are then aggregated, producing a rich, multi-perspective representation.

Figure 4. Illustration of the multi-head self-attention mechanism.

MLP (Multi-Layer Perceptron), which functions similarly to a feedforward network (FFN), is made up of two dense layers with a GELU nonlinearity in between. Consisting of two fully connected layers, this part applies the GELU activation after both layers to introduce nonlinearity. This notably strengthens the model’s capability to model intricate transformations of token embeddings. The structure of this component is as follows:

$$\mathrm{MLP}\left(x\right)=\mathrm{GELU}(x{W}_1+b_1)W_2+b_2,$$

(11)

where $x\in {\mathbb{R}}^d$ denotes the input token embedding, $W_1\in {\mathbb{R}}^{d\times d_{\mathrm{ff}}}$ and $W_2\in {\mathbb{R}}^{d_{\mathrm{ff}}\times d}$ are learnable weight matrices, $b_1$ and $b_2$ are bias terms, and $d_{\mathrm{ff}}$ is the intermediate hidden dimension, which is typically set to four times the embedding dimension d (e.g., $512\to 2048$). For deep neural architectures, each sublayer is further equipped with residual connections and layer normalization to facilitate stable optimization and reliable gradient propagation. For deep neural architectures, each sublayer is designed with residual pathways and layer normalization, facilitating stable optimization and reliable gradient transmission. Specifically, the update rules for each encoder block are as follows:

$$Z^{\prime }=Z+\mathrm{MultiHead}\left(\mathrm{LN}\left(Z\right)\right){\mathbf{Z}}_{\mathrm{out}}=Z^{\prime }+\mathrm{FFN}\left(\mathrm{LN}\left(Z^{\prime }\right)\right),$$

(12)

where $LN(·)$ denotes layer normalization. The transformer’s output is a sequence of token embeddings enriched with contextual information. This design not only retains the original signal features but also alleviates issues like vanishing gradients in deeper networks. To obtain a global representation, the token at position 0 in the output sequence is extracted and used as the overall feature vector:

$${\mathbf{F}}_{\mathrm{Trans}}=\mathrm{LN}\left({\mathbf{Z}}_{\mathrm{out}}\left[0\right]\right)\in {\mathbb{R}}^d,$$

(13)

This vector is subsequently used in the Path Attention Module (PAM) to align with the CNN and BiLSTM outputs. The specific dimensions of these are shown in Table 3.

Table 3. ViT-branch-specific input and output process table.

Module Name	Input/Output Dimension	Operation/Functional Description
Constellation Map Generation	IQ signal	Complex mapping, 2D histogram generation, and Gaussian smoothing; converts the IQ sequence into a 2D spatial modulation distribution.
Patch Embedding	Input: constellation image	Divide the image into patches, flatten them, and project via an MLP; embeds each patch into a fixed-length vector representation.
Add [CLS] + Positional Encoding	Embedded patches	Add a learnable [CLS] token and positional encodings; provides a global representation and spatial position awareness for the Transformer encoder.
Transformer Encoder	Embedded sequence	Stacked Transformer encoder layers that model long-range dependencies in the constellation map using self-attention mechanisms.
Extract Token	Transformer output	Take the output corresponding to the [CLS] token; produces an image-level global representation of the modulation signal.
Branch Output	–	Fused with the CNN and LSTM branch outputs; represents global spatial features of modulation patterns.

3.4. BiLSTM

The framework diagram of LSTM is shown in Figure 5. To effectively capture the long-term dependencies inherent in modulated signals, we integrate a BiLSTM network as the third feature extraction branch. Unlike traditional CNNs, which focus on local patterns, RNNs, particularly LSTMs, are well-suited for learning temporal structures within sequential data. The bidirectional architecture of the network enables it to process both past and future contextual information, making it particularly effective for recognizing modulation types with periodic or time-varying characteristics.

Figure 5. Architecture of the BiLSTM.

The input to this branch is the original IQ signal matrix ${\mathbf{X}}_{\mathrm{IQ}}\in {\mathbb{R}}^{L\times 2}$, where $L=128$ denotes the sequence length, and the two channels indicate the in-phase and quadrature components. Next, the sequence is input into a BiLSTM layer configured with $h=256$ hidden units in each direction. The resulting forward and backward outputs are merged, yielding a combined temporal feature vector:

$${\mathbf{F}}_{\mathrm{BiLSTM}}=\left[{\mathit{h}}_{t+1};{\overleftarrow{\mathit{h}}}_{t-1}\right]\in {\mathbb{R}}^{512},$$

(14)

where ${\mathit{h}}_{t+1}$ and ${\overleftarrow{\mathit{h}}}_{t-1}$ denote the final hidden states from the forward and backward directions.

This 512-dimensional feature vector captures the high-level temporal dynamics of the input signal, complementing the local spatial features extracted by the CNN branch and the global structural features obtained through the ViT branch. The BiLSTM features are then passed to the path attention module, where they are adaptively fused with the outputs from the other branches. The bidirectional structure of the LSTM improves the model’s capacity to learn transitions, repetitions, and time-varying frequency domain patterns. This makes BiLSTM particularly effective for identifying modulation schemes with distinct waveform cycles or timing characteristics, such as PSK and QAM, when compared to unidirectional LSTM.

Following the detailed explanation of each branch’s functionality, the overall architecture of the model is presented to complete the system design.

4. Three-Branch Fusion Classification Method

In AMR, variations in modulation type and SNR can lead to considerable differences in signal characterization. Relying on a single feature extraction method may result in limited robustness or insufficient generalization. To resolve this limitation, a multi-branch fusion network is introduced, allowing features to be extracted in collaboration across three distinct routes. This approach not only addresses the issue of robustness but also improves recognition accuracy. For instance, when a modulation type is identified in the multi-channel CNN branch and the same modulation type is later confirmed through the constellation diagram in the transformer, the agreement between the two results increases our confidence in the correctness of the recognition output.

The three feature vectors are then adaptively integrated using a Path Attention Module (PAM), which allows dynamic fusion based on the signal type and quality. With its multi-path design, the AMR system achieves a marked increase in robustness.

The main processing flow of the proposed three-branch fusion network can be summarized as follows:

• IQ signal preprocessing: The raw complex base band IQ samples are normalized and reshaped into the required formats for sequence-based processing and constellation construction.
• Parallel feature extraction: The preprocessed IQ sequence is fed simultaneously into three branches. The CNN branch extracts local multi-scale time–frequency features, the BiLSTM branch models long-range temporal dependencies and modulation periodicity, and the ViT branch takes the corresponding constellation image as input to learn global spatial structures based on the [CLS] token representation.
• Feature aggregation: The high-level feature vectors produced by the three branches are concatenated into a joint feature representation, which collects complementary information from the time domain, sequence dynamics, and constellation space.
• Adaptive path fusion: The concatenated feature is passed through the path attention module (PAM), which generates branch-wise weights and produces a fused feature by assigning different importance to the CNN, BiLSTM, and ViT representations.
• Classification: The fused feature is fed into the final fully connected layers and softmax to output the posterior probabilities of all modulation types, and the class with the highest probability is taken as the predicted modulation label.

To provide a clearer overview of the proposed architecture, Figure 6 illustrates the overall structure of the three-branch fusion network. The IQ signal is simultaneously fed into the CNN, BiLSTM, and ViT branches to obtain three complementary feature representations, which are then fused and passed to the final classifier. The detailed implementation of each branch and the fusion module is further shown in Figure 7.

Figure 6. Overall structure of the proposed three-branch fusion network.

Figure 7. Detailed architecture of the three-branch feature extraction and fusion module.

4.1. Overall Framework Description

As depicted in Figure 7, the proposed AMR framework’s overall architecture is outlined. The specific details have been outlined in the previous section. The input IQ signal sequence, represented as the matrix ${\mathbf{X}}_{\mathrm{IQ}}\in {\mathbb{R}}^{128\times 2}$, is processed in parallel by three distinct feature extraction branches.

The CNN branch first transforms the IQ sequence into a multi-channel temporal representation by incorporating additional physically meaningful features, such as amplitude, phase, and derivative. This results in a signal tensor ${\mathbf{X}}_{\mathrm{CNN}}\in {\mathbb{R}}^{C\times L}$, where C ranges from 5 to 7 channels. The branch then applies a series of 1D convolutional layers along the time axis to extract localized temporal patterns, followed by max-pooling and channel attention mechanisms. The output is a flattened feature vector ${\mathbf{F}}_{\mathrm{CNN}}\in {\mathbb{R}}^{512}$.

The BiLSTM branch treats the original IQ sequence as a temporal input, learning long-range dependencies through bidirectional LSTM layers, resulting in a feature representation ${\mathbf{F}}_{\mathrm{BiLSTM}}\in {\mathbb{R}}^{512}$.

The ViT branch transforms the IQ sequence into a 2D constellation diagram ${\mathbf{X}}_{\mathrm{img}}\in {\mathbb{R}}^{1\times 64\times 64}$, which is then embedded into patch sequences and fed into Transformer encoders. The final token output serves as the global feature ${\mathbf{F}}_{\mathrm{Trans}}\in {\mathbb{R}}^{512}$.

4.2. The Path Attention Module and Classifier

Unlike traditional fixed fusion methods, such as simple series or averaging, the Path Attention Module (PAM) is designed to adaptively assign weights to each branch according to the properties of the input signal. This adaptability enables the model to prioritize features with the highest informational value under varying modulation schemes and SNRs. For instance, frequency-sensitive modulations like FSK benefit more from features extracted by the CNN branch, while modulations with strong temporal structure, such as PSK, are more effectively captured by the BiLSTM branch.

To implement this adaptive mechanism, the outputs from the three branches are first passed through a shared, lightweight attention estimator that computes soft attention weights. These weights, denoted as $\alpha$, $\beta$, and $\gamma$, are then used by the Path Attention Module to dynamically adjust the contribution of each path. The final representation is derived through the attention-weighted fusion of outputs from the three parallel branches.

$${\mathbf{F}}_{\mathrm{Final}}=\alpha ·{\mathbf{F}}_{\mathrm{CNN}}+\beta ·{\mathbf{F}}_{\mathrm{BiLSTM}}+\gamma ·{\mathbf{F}}_{\mathrm{Trans}},\alpha +\beta +\gamma =1,$$

(15)

Firstly, the outputs of the three branches are spliced one by one to obtain ${\mathbf{F}}_{\mathrm{cat}}=\left[{\mathbf{F}}_{\mathrm{CNN}};{\mathbf{F}}_{\mathrm{BiLSTM}};{\mathbf{F}}_{\mathrm{Trans}}\right]\in {\mathbb{R}}^{3\times 512}$, in which we use an MLP and softmax to obtain the fusion weights, $\left[\alpha ,\beta ,\gamma \right]=\mathrm{Softmax}\left(\mathrm{MLP}\left({\mathbf{F}}_{\mathrm{cat}}\right)\right)$; and finally, we obtain the outputs, ${\mathbf{F}}_{\mathrm{Final}}\in {\mathbb{R}}^{512}$, which is used as the input to the classification layer.

Two fully connected layers were used in the classification layer. The first layer, FC1, reduces the dimensionality from 512 to 256, applying ReLU activation for further compression and nonlinear transformation. The output is then passed through the second layer, FC2. In the last step, the output is processed using the softmax activation function.

4.3. Loss Function

The entire network is trained jointly, where the softmax classifier output serves as the basis for computing the cross-entropy loss. Since all three branches contribute to the final feature vector ${\mathbf{F}}_{\mathrm{Final}}$, the loss is back-propagated through the path attention module and each branch. This allows the model to optimize jointly, ensuring that the classification performance guides the feature learning across the entire architecture.

To complete the optimization, We use the standard cross-entropy loss function:

$$\mathcal{L}=-\sum _{i=1}^C{y}_i\log \left({\widehat{y}}_i\right),$$

(16)

where y is the one-hot encoded true label and ${\widehat{y}}_i$ is the softmax output for class i; C is the number of categories. The model is trained to minimize this loss using stochastic gradient descent.

4.4. Data Flow and Representation

The data flows and internal transformations throughout the network can be summarized as follows in Table 4:

Table 4. Overall input–output dimensions of the proposed three-branch framework.

Branch	Input Shape	Output Shape	Key Feature
CNN	$2\times 128$	512	Extracts multi-scale time–frequency features from the IQ signals.
BiLSTM	$128\times 2$	512	Models temporal dependencies and modulation periodicity.
ViT	$1\times 64\times 64$	512	Captures global constellation structures via Transformer attention.
PAM + FC1 + FC2	$3\times 512\to 512$	C (classes)	Fuses multi-branch features adaptively and performs classification.

5. Experiments and Results

5.1. Dataset Description

We evaluate the effectiveness of automatic AMR algorithms on the RML2018.01a dataset [47], which is widely used for comparative studies due to its scale and diversity of channel conditions. The corpus covers multiple impairments—including additive white Gaussian noise (AWGN), carrier frequency offset (CFO), timing offset, and sampling rate offset (SRO) [15]—making performance assessment more challenging as the data volume increases. Unless otherwise specified, signals are provided as complex base band IQ sequences (zero-IF); therefore, the dataset does not define a fixed RF carrier or absolute modulation/symbol frequency, and all models operate on discrete-time, normalized samples. Each example in RML2018.01a contains 1024 samples per I and Q channel, We follow the standard SNR grid of $-20:2:30$ dB and evaluate all 24 modulation types. For complementary validation, we also report results on the smaller RML2016.10a dataset with SNR $-20:2:18$ dB and 11 modulation types; each example there contains 128 IQ samples. In both datasets, the injected noise model is AWGN, while RML2018.01a additionally includes CFO, timing offset, and SRO to emulate practical channel distortions.

5.2. Experimental Details

In the experiment, the RML2016.10a dataset and the RML2018.01a dataset were divided into training and validation sets in a 8:2 ratio. The dataset was divided according to the aforementioned ratio at all SNR levels and signal modulation formats to ensure the balance of the dataset. The experiment process uses an NVIDIA A10 GPU, the environment are pytorch2.3.1 and the language is python3. During training, the initial learning rate was set to 0.001, and the cross-entropy loss function (CrossEntropyLoss) was used as the loss function. Terminate training when the validation accuracy fails to improve over ten epochs. To optimize the learning process, we used the Adam optimizer. To validate the performance of the proposed algorithm, we evaluated it based on classification performance metrics. To confirm the superiority of the proposed method, we compared and evaluated it against several high-performance baseline methods, specifically: MCLDNN [35], IC-AMCNET [48], MCNET [37], CGDNET [36], and ResNet [49]. For consistency in comparison, all methods were conducted under identical conditions.

5.3. Results and Analysis

As shown in Figure 8, the training loss consistently decreases over 30 epochs, indicating effective optimization and stable convergence. The loss drops rapidly during the initial stages and gradually stabilizes at a low value, demonstrating the robustness of the proposed framework without signs of overfitting.

Figure 8. Training loss curve over 30 epochs in RML2016.10a.

Figure 9 presents the confusion matrix of the proposed modulation recognition framework evaluated at an SNR of 10 dB, providing a quantitative depiction of the classification outcomes across eleven modulation categories: QPSK, PAM4, AM-DSB, GFSK, QAM64, AM-SSB, 8PSK, QAM16, WBFM, CPFSK, and BPSK. The diagonal components indicate correctly identified samples, whereas the off-diagonal elements represent instances of misclassification. In general, the framework exhibits remarkably high classification performance under this moderate SNR condition. The results demonstrate the robustness and reliability of the adopted feature extraction and decision-making strategies, even when subjected to low-level noise disturbances.

Figure 9. Confusion matrix at SNR = 10 dB in RML2016.10a.

Despite achieving near-perfect recognition performance for most modulation types, the proposed model consistently exhibits a higher misclassification rate for AM-SSB across all SNR levels. This phenomenon primarily arises from the inherent spectral similarity between AM-SSB and AM-DSB, as well as the potential feature overlap with WBFM.

In the RML2016.10a dataset, the recognition performance generally stabilizes once the SNR exceeds 10 dB. Within the high-SNR range (10–18 dB), the proposed method consistently achieves an accuracy of approximately 95%, whereas MCLDNN fluctuates around 92%, IC-AMCNET fluctuates around 86%, MCNET fluctuates around 83%, CGDNET fluctuates around 80%, and ResNet maintains roughly 85%. By comparison, the proposed method demonstrates superior accuracy, confirming its improvement.

Figure 10 depicts the classification performance of a selected set of PSK and APSK modulation formats across a wide SNR range. In the severely noisy regime (e.g., SNR (≤$-10$ dB)), all modulation types perform poorly, reflecting the expected degradation in signal discriminability under strong noise interference. Nonetheless, some schemes such as 16PSK begin to show noticeable improvements earlier than others, suggesting favorable feature separability under moderate distortions.

Figure 10. Accuracy versus SNR for selected PSK/APSK modulations in RML2018.01a.

As the SNR increases into the transitional region, differences among modulation types become more evident. Lower-order PSK signals (e.g., BPSK, QPSK) achieve rapid gains in accuracy, approaching near-perfect recognition at moderate SNR levels. In contrast, higher-order APSK schemes (such as 64APSK and 128APSK) exhibit slower convergence, indicating their increased sensitivity to noise due to denser symbol constellations.

Beyond 10 dB, the majority of modulations reach stable classification, with most curves flattening near unity. However, subtle discrepancies persist—particularly among high-order APSK formats—where residual misclassifications hint at inherent ambiguity in feature spaces despite favorable SNR conditions.

As presented in Figure 11, the classification performance of several modulation formats—including OOK, FM, GMSK, OQPSK, and four AM variants—is analyzed over a wide SNR range.

Figure 11. Accuracy vs. SNR curves in low-order and analog modulation performance in RML2018.01a.

At extremely low SNR levels (e.g., below $-10\mathrm{dB}$), all schemes exhibit low recognition accuracy, reflecting the challenges of distinguishing signal structures submerged in heavy noise. However, as the SNR improves, accuracy increases rapidly for most formats. Notably, AM-SSB-WC and AM-SSB-SC demonstrate faster convergence to high accuracy, likely benefiting from their relatively distinct spectral and envelope features. OOK also displays a steep performance rise, attributable to its binary nature and amplitude-based distinguishability.

In contrast, AM-DSB-SC shows persistent instability and under-performance even at high SNRs, suggesting a more ambiguous signal representation under the current model. GMSK and OQPSK follow intermediate trends, with smoother curves and stable classification from mid-to-high SNRs.

Figure 12 illustrates the classification accuracy of the proposed model across a range of SNR values for a subset of amplitude-based modulation schemes, including both amplitude shift keying (ASK) and quadrature amplitude modulation (QAM) variants. This focused evaluation provides deeper insight into the model’s ability to discriminate among modulations that share similar amplitude domain characteristics yet differ in constellation complexity.

Figure 12. Accuracy versus SNR for QAM/ASK-style modulation performance in RML2018.01a.

As the figure demonstrates, lower-order ASK and QAM types (e.g., 4ASK, 8ASK, and 16QAM) achieve rapid convergence to high classification accuracy as the SNR surpasses 0 dB, benefiting from relatively distinct constellation structures and reduced symbol ambiguity. In contrast, higher-order QAM schemes (e.g., 128QAM and 256QAM) exhibit more gradual performance improvement with increasing SNR, requiring significantly higher signal quality to achieve comparable accuracy levels. This behavior can be attributed to the reduced Euclidean distance between constellation points and heightened sensitivity to noise in these densely packed modulations.

Notably, despite the intrinsic complexity of high-order QAM formats, the proposed architecture maintains consistent accuracy progression across the entire SNR range, ultimately achieving near-saturation accuracy beyond 10 dB for most schemes. These results suggest that the feature extraction and classification layers are sufficiently expressive and robust to manage the finer granularity required for high-order modulations.

In summary, the comparative analysis across the three modulation groups reveals several consistent trends. First, lower-order modulation schemes (e.g., BPSK, 4ASK, OOK) demonstrate superior robustness to noise and rapidly achieve high recognition accuracy even in low-to-moderate SNR regimes. Second, the classification difficulty increases markedly with modulation order and constellation complexity, as evidenced by the delayed convergence and lower peak accuracies of higher-order formats such as 256QAM, 128APSK, and AM-DSB-SC. Third, modulation types characterized by spectral redundancy or envelope variation (e.g., AM and FM) present non-monotonic or irregular trajectories, indicating a greater challenge in discriminative feature extraction under noisy conditions.

These findings collectively highlight the importance of adaptive and hierarchical feature learning strategies in automatic modulation recognition. The ability to extract both local and global patterns is essential not only for handling noise-robust modulations but also for resolving fine-grained distinctions among visually similar, high-order schemes. Therefore, designing architectures capable of multi-scale representation is key to achieving generalizable performance across diverse modulation families.

Figure 13 presents the row-normalized confusion matrix obtained at a SNR of 10 dB, providing a comprehensive visualization of the model’s classification behavior under moderately high channel quality. The matrix enables detailed assessment of the model’s ability to discriminate among the 24 modulation categories, highlighting both strengths in identification and patterns of confusion.

Figure 13. Confusion matrix under 10 dB SNR in RML2018.01a.

As evident from the strong diagonal dominance, the proposed architecture achieves near-perfect classification accuracy for the majority of modulation types. Particularly, lower-order PSK and ASK classes (e.g., BPSK, QPSK, 4ASK) exhibit almost ideal separability, reflecting their distinct signal structures and robustness to noise at this SNR level. Moreover, analog modulation formats such as AM-DSB-SC and AM-SSB-WC also exhibit slight confusion, which may stem from their signal envelope similarity in the time–frequency domain. Importantly, no major systematic bias or model instability is observed across the matrix, affirming the generalization capacity of the model across both the digital and analog domains.

This confusion matrix not only confirms the model’s high discriminative power at SNR = 10 dB but also provides actionable insights into class-specific vulnerabilities, guiding future refinement in feature extraction or attention calibration. Overall, the results validate the effectiveness of the proposed fusion architecture in achieving fine-grained modulation recognition under realistic wireless conditions.

5.4. Comparison of Methods

To thoroughly evaluate the performance of the proposed model under varying channel conditions, Figure 14 illustrates the classification accuracy of six representative models across a wide range of SNR levels. Due to differences in the environmental configuration and data processing employed in this paper, the results of the comparative methods differ from those provided in the original paper. As expected, the recognition performance of all models improves with increasing SNR. However, it is evident that the proposed method consistently achieves the highest accuracy across most SNR values, particularly under high-SNR scenarios (above 10 dB), where precise modulation classification is essential for reliable communication.

Figure 14. Accuracy achieved by different AMR approaches across a range of SNRs in RML2018.01a.

The embedded zoom-in subplot provides a magnified view of the top-4 performing models—namely, the proposed method, IC-AMCNET, MCNET, and MCLDNN—in the high-SNR region. This local comparison reveals that the proposed method not only achieves the best peak accuracy but also exhibits the most stable and steadily improving performance, reflecting its enhanced generalization ability and effective suppression of residual noise. In contrast, competing methods show performance saturation or fluctuations, suggesting potential limitations in handling signal variations or overlapping features.

Furthermore, the superior performance of the proposed model in low-to-mid SNR regimes (e.g., from $-10$ dB to 10 dB) underscores its strong robustness under adverse channel conditions. This can be attributed to its hybrid architectural design and efficient feature extraction mechanisms, which facilitate better discrimination of modulation patterns even under heavy noise contamination.

Although the accuracy curves in Figure 14 show only a modest improvement over baseline models at medium–high SNR levels, a more comprehensive comparison reveals the advantages of the proposed method. As summarized in Table 5, our tri-branch fusion architecture achieves the highest overall accuracy, as well as the best macro-F1 and Cohen’s Kappa scores among all evaluated methods. These indicators reflect not only improved recognition capability but also a more balanced performance across different modulation types. Furthermore, the proposed approach exhibits enhanced stability in low-to-mid SNR conditions, where competing models often suffer from pronounced fluctuations or class-specific degradation. This combined evidence demonstrates that, despite the seemingly small gap in the raw accuracy curves, the proposed method provides consistently superior and more robust classification performance across the entire SNR range.

Table 5. Performance comparison of different models.

Method	Overall Accuracy	Macro-F1	Kappa	Time (s/Epoch)	Epochs
Proposed	0.590	0.592	0.574	190	65
MCLDNN	0.561	0.562	0.549	162	50
MCNET	0.566	0.565	0.551	81	75
CGDNET	0.505	0.510	0.483	203	45
ResNet	0.535	0.537	0.518	132	80
IC-AMCNET	0.582	0.584	0.566	104	70

Collectively, these results highlight the model’s dual advantage: robustness to noise in challenging conditions and high-fidelity classification under clean signals. This performance advantage makes it well-suited for practical deployment AMR systems, particularly in dynamic and noisy wireless environments such as cognitive radio networks, UAV-based communications, and battlefield spectrum monitoring.

Table 5 provides a comparative analysis of the proposed method against several established baselines in terms of accuracy, macro-F1, Kappa, training time, and convergence epochs. The proposed model consistently demonstrates superior performance, achieving a favorable trade-off between predictive precision and statistical consistency. Compared to alternative methods, such as MCLDNN and MCNET, which show partial strengths in individual metrics, the proposed approach maintains a more balanced profile across all evaluation criteria. However, the approach proposed herein involves a greater number of branches, resulting in relatively longer training times. In addition to achieving competitive or superior overall accuracy, the proposed model exhibits more stable behavior across the SNR range and a more balanced recognition capability over different modulation types, as reflected by the accompanying quantitative metrics. From the perspective of computational complexity, IC-AMCNet requires approximately 8.6 M parameters, while MCLDNN, MCNet, and CGDNet use about 0.40 M, 0.13 M, and 0.65 M parameters, respectively, and ResNet involves an exceptionally large number of parameters. Although the total number of parameters of the proposed tri-branch architecture remains non-negligible, its complexity stays within a reasonable range and can be further reduced to around 4 M by employing different weighting strategies at runtime, without changing the overall performance pattern. Taken together, these observations demonstrate that the proposed approach attains a favorable trade-off between classification performance, robustness, and computational cost.

5.5. Ablation Experiment

To assess the functional contribution of each architectural branch, an ablation experiment was performed by progressively removing key components from the proposed model. As shown in Figure 15, the recognition performance of four model configurations was evaluated across a broad SNR range. The complete framework (“the proposed”), which combines the multi-channel physical feature convolutional module, a BiLSTM branch, and a ViT-based transformer path, consistently achieved the highest accuracy at nearly all SNR levels. In contrast, the “only cnn” variant—relying solely on the convolutional physical features—exhibited notably lower performance, especially in higher SNR regimes, reflecting its inability to capture sequential or global contextual cues. The “cnn+bilstm” and “cnn+vit” models, each incorporating one auxiliary branch, demonstrated moderate improvements due to their ability to extract either temporal dependencies or long-range representations. Nevertheless, both remained inferior to the fully fused architecture.

Figure 15. Ablation study of the proposed architecture in RML2018.01a.

Although Figure 15 shows that the accuracy improvement of the full tri-branch model over the CNN-only baseline is numerically modest at some SNR levels, the additional BiLSTM and ViT branches are introduced to enhance the model in several complementary aspects rather than to create a completely different accuracy curve. The BiLSTM branch focuses on temporal feature modeling, enabling the network to capture symbol-to-symbol dependencies and subtle temporal patterns that are easily smeared by noise, which leads to more stable decisions in low- and mid-SNR regimes. In parallel, the ViT-based constellation branch emphasizes global spatial relationships in the IQ plane, allowing the model to exploit long-range correlations and class-specific constellation structures that are difficult to represent with local CNN kernels alone. As confirmed by the ablation results in Figure 15, removing either of these two branches causes a consistent decrease in performance across the SNR range, while the full tri-branch model achieves a better balance between robustness and representation capacity with only a moderate increase in complexity. This indicates that the proposed architecture improves not only raw accuracy but also the reliability and stability of modulation recognition under varying channel conditions.

The embedded zoom-in window in Figure 15 further magnifies the results for SNR $\ge 10$ dB, revealing that the proposed model retains a consistent performance margin over its reduced counterparts, even when noise effects are minimal. This validates the synergistic effect of integrating all three feature pathways, highlighting that temporal modeling and global attention mechanisms jointly enhance the baseline convolutional encoding. Collectively, these results affirm the necessity of the complete tri-branch fusion strategy for maximizing classification robustness across varying channel conditions.

6. Conclusions

This paper proposes a three-branch fusion classification model comprising three distinct branches: a multi-channel physical feature convolution branch, a bidirectional long-short time series branch, and a ViT branch. This approach extracts multiple features from the time–frequency domain for recognition, utilises BiLSTM for auxiliary discrimination, and employs ViT to analyse signal constellation diagrams for final classification validation, thereby achieving high-performance classification. Experiments conducted on the RML2016.10a and RML2018.01a datasets demonstrate that the proposed method achieves superior recognition performance with enhanced accuracy.

Future research still faces several challenges. First, applying the proposed method to real-world systems and verifying its effectiveness using practical measurement data remain essential, as most existing studies are still based on simulation results. Second, enhancing the robustness of modulation recognition in harsh environments is equally important. In particular, achieving reliable accuracy under low signal-to-noise ratio conditions remains a key issue that requires further investigation.