CAEP: Cross-Modal Adaptive Embedding Prediction for Self-Supervised Modulation Classification

Abstract

Although self-supervised learning methods have shown promising progress in addressing the issue of scarce labeled data in automatic modulation classification, they remain constrained by heavy reliance on extensive negative samples and an inability to effectively capture inter-modal feature correlations. To overcome these limitations, we propose a novel self-supervised automatic modulation classification algorithm based on multi-path embedding prediction, termed CAEP. In CAEP, the raw signal is first dynamically segmented into current and future sub-series. Then, dedicated encoders are utilized to extract embeddings for both sub-series and leverage current information to predict future states, while randomly masking the corresponding time–frequency images transformed from the time-domain signal to predict the obscured spectral components. Furthermore, latent temporal embeddings are deployed to predict information within the time–frequency domain to achieve cross-modal retrieval. Finally, a classification head is connected alongside a temporal modal encoder, which is fine-tuned using a limited set of labeled samples to accomplish modulation classification. Experimental results on two benchmark datasets demonstrate that the proposed method achieves robust performance across varying noise conditions.

1. Introduction

Automatic modulation classification (AMC) is crucial for wireless communications and electromagnetic signal processing. To address severe spectrum scarcity and stringent demands for high-precision processing due to mobile terminal data and the intelligence-driven evolution of communication systems, AMC provides the foundational support for future intelligent communication systems to achieve autonomous adaptation, dynamic perception, and secure controllability by efficiently leveraging electromagnetic spectral features to identify signal modulation types accurately. Consequently, it serves as an indispensable strategic enabler in several practical scenarios, including cognitive radio communications, electronic intelligence (ELINT) reconnaissance, satellite communications, and unmanned aerial vehicle (UAV) telemetry and control [1].

Traditional AMC methodologies can be categorized into likelihood-based [2] and feature-based [3] types, which typically leverage conventional classifiers such as Support Vector Machines (SVMs) [4] and K-Nearest Neighbors (KNN) [5] for modulation identification. However, with the advent of 5G and 6G technologies, the order of modulation schemes has escalated significantly. Furthermore, the increasing complexity of noise in real environments necessitates recognition algorithms that exhibit both high precision and robust anti-interference capabilities [6]. Recently, deep learning-based methods have become a focus in electromagnetic signal recognition, such as Convolutional Neural Networks (CNNs) [7], Recurrent Neural Networks (RNNs) [8], Graph Neural Networks (GNNs) [9], and Transformers [10]. Driven by their superior capacity for automatic feature extraction and their end-to-end learning paradigms, these approaches have gained significant prominence for high recognition efficiency.

Nevertheless, existing deep learning-based AMC algorithms remain constrained by specific bottlenecks. A primary limitation is their reliance on large volumes of high-quality labeled data, which is often impractical to meet given the complexity of noise environments. Self-supervised learning (SSL) leverages vast amounts of unlabeled data to autonomously acquire discriminative representations for diverse signal types, thus significantly reducing reliance on labeled samples, substantially lowering annotation costs. SemiAMC pioneered the successful integration of self-supervised contrastive learning into AMC [11]. Additionally, data augmentation is frequently employed to enhance data diversity, thereby facilitating more effective discrimination of features across distinct modulation types. Xiao et al. introduced “semantic-preserving” data augmentation (DA) techniques and a novel contrastive loss with debiased hardness weights to ensure both signal robustness and semantic invariance [12]. To mitigate the bottleneck of data imbalance, Zhang et al. incorporated diverse DA strategies to bolster data variability, which attain high-precision recognition by computing similarities among various augmented samples [13]. In addition to contrastive learning, generative learning represents another prevalent self-supervised paradigm. Chen et al. proposed a generative SSL algorithm based on reconstructing masked time–frequency diagrams, employing a mutual information maximization strategy to predict spectral information obscured by noise, demonstrating its superior classification capability [14].

However, some DA-based self-supervised methods have the issue of their reliance on large numbers of fine-tuning samples and single-modality data, which incurs significant fine-tuning costs and neglects information across diverse modalities, such as frequency-domain representations, time–frequency spectrograms [15], Gramian Angular Fields (GAFs) [16], Pseudo Wigner–Ville Distributions (PWVDs) [17], and constellation diagrams [18]. Therefore, numerous researchers have extended SSL techniques to the few-labeled fine-tuning and cross-modal scenarios. Cai et al. combined DA with contrastive learning to discern multi-view discrepancies among different sample types and then introduced a Mixture of Experts (MoE) module coupled with a dedicated expert entropy loss to enhance model diversity, thereby achieving efficient recognition under few-shot fine-tuning via an ensemble learning strategy [19]. Liu et al. proposed a novel contrastive learning-based multi-modal fusion algorithm designed to extract critical features from fused multi-modal data [20]. Bai et al. utilized a cross-modal contrastive training method to effectively capture both feature-level and instance-level representations across time-series and constellation diagram modalities [18]. Meanwhile, Liu et al. enhanced pre-training stability by comparing the saliency of time–frequency diagrams and time-domain signals, combined with the Sinkhorn–Knopp loss function [17]. Liu et al. introduced a novel signal modality, termed Phase Density, to effectively characterize the phase distribution and proposed a “Top-K” contrastive learning strategy to filter out low-quality negative samples, showing superior recognition performance across various noise environments, even in one-shot fine-tuning scenarios [21].

Despite the promising achievements of SSL-based methods applied in multi-modal AMC, several limitations still persist: (1) contrastive learning paradigms rely heavily on negative samples, which not only incurs substantial construction costs but also makes it difficult to guarantee the reliability of these negative pairs; (2) generative frameworks often focus exclusively on “explicit data”, thereby neglecting latent feature representations. Joint-Embedding Predictive Architecture (JEPA) [22], as a novel SSL paradigm, addresses these issues by predicting target latent embeddings using current input embeddings and auxiliary variables, eliminating the need for decoders and negative samples [23]. Therefore, this article introduces the concept of JEPA into the AMC domain and proposes a novel cross-modal self-supervised AMC algorithm through Adaptive Embedding Prediction (CAEP). Specifically, the proposed algorithm addresses both intra-modal and inter-modal feature relationships. Intra-modal prediction comprises two primary components: first, time-domain signals are split into current and future subsequences via a dynamic sliding strategy, enabling the prediction of future information based on current observations. Meanwhile, time-domain signals are transformed into time–frequency images via Short-Time Fourier Transform (STFT), where random masking is applied to reconstruct the obscured spectral information, thereby significantly enhancing the perception of intra-modal dependencies. For inter-modal prediction, latent features acquired from the time domain are utilized to predict time–frequency representations, facilitating effective cross-modal alignment. After the prediction phase, the model undergoes fine-tuning with a limited set of labeled samples to accomplish the downstream task of AMC. In summary, the contributions of this work are as follows:

• We propose a novel cross-modal SSL-based AMC method by matching both intra-modal and inter-modal embeddings in the latent space to eliminate the reliance on extensive negative samples and decoder networks, ensuring the robustness of embedding representations but also significantly reducing the computational cost associated with model parameters and contrastive pairs.
• Through multi-path embedding prediction that integrates both inter-modal and intra-modal information, the proposed method effectively captures local temporal correlations and spectrogram modalities. Furthermore, it significantly encourages single-modal encoders to learn synergistic consistency between dual-modal features, thus remarkably enhancing the reliability of representations across different modalities.
• The performance of the proposed method has been experimentally validated on two public datasets, showing that the model demonstrates superior discriminative capability, leading to improved recognition efficiency across various noise levels, even when fine-tuned with a limited number of labeled samples.

The rest of this paper is structured as follows: Section 2 outlines the foundational principles of JEPAs. Section 3 subsequently details the proposed CAEP. Section 4 presents the experimental methodology and a comprehensive analysis of the results. Finally, Section 5 provides the conclusion.

2. Preliminaries

The fundamental principle of JEPAs revolves around acquiring latent representations by leveraging a subset of input data to forecast another subset. Figure 1 provides a concise overview of the JEPA mechanism. Given source data x and target data y sampled from the dataset D, the corresponding encoders are utilized to derive source and target latent embeddings $s_x$ and $s_y$ from x and y, respectively. Subsequently, a predictor network is constructed to bridge $s_x$ into the predicted representation ${\widehat{s}}_y$ according to the target data while incorporating auxiliary variables z (specifically, learnable mask tokens) to enhance predictive ability. Ultimately, a regularization loss, such as Mean Square Error (MSE), is calculated between the predicted ${\widehat{s}}_y$ and the actual target representation $s_y$ to train the encoder networks.

Figure 1. The structure of two kinds of JEPAs.

Structurally, JEPAs can be conceptualized as a synthesis of Joint-Embedding Architectures (JEAs) and Generative Architectures (GAs), although they diverge from both paradigms. Unlike GAs, which aim to regenerate pixel-level data, or JEAs, which strictly align embeddings between two instances, JEPAs seek to integrate the source representation and latent variables to forecast the target representation obtained by encoders, enhancing the perception of the correlation of the source and target data, as depicted in Figure 1. By conducting information prediction in the latent space without the need for positive–negative pairs, JEPAs avoid instability caused by the poor quality of the negative pairs and suppress noise via implicit bias in self-distillation [24], thereby reducing the issue of model collapse, in contrast to JEAs and GAs.

Owing to these distinct merits, JEPAs have gained widespread adoption across diverse domains, including images [22], video processing [25], SAR [26], signal representation learning [27], and multi-modal analysis [28]. Nevertheless, their deployment within the realm of AMC remains comparatively scarce.

Therefore, this paper introduces the basic principle of JEPAs to AMC. In addition to employing the classic JEPA paradigm for masked spectrogram prediction, as illustrated in Figure 1a, the multi-path pre-training algorithm is further introduced that incorporates the prediction of embeddings across distinct temporal segments within the time-domain signal, as depicted in Figure 1b, which is designed to facilitate more comprehensive multi-modal feature perception.

3. Method

3.1. Problem Definition

The objective of AMC is to identify the modulation type from received electromagnetic signals. Fundamentally, this task can be modeled as a mapping problem from the signal space to the modulation category space.

In a single-input single-output communication scenario, the received signal is typically represented in its complex baseband form:

$$s\left(t\right)=s_I\left(t\right)+j{s}_Q\left(t\right)$$

(1)

where $s_I\left(t\right)$ and $s_Q\left(t\right)$ denote the in-phase and quadrature components, respectively. In practical processing, the continuous-time signal is discretized into a sequence of length T:

$$s=[s\left(1\right),s\left(2\right),\dots ,s\left(T\right)]\in {\mathbb{C}}^T$$

(2)

where $\mathbb{C}$ denotes the set of complex numbers. In the automatic modulation classification task, let the set of modulation categories be $Y=y_1,y_2,\dots ,y_K$, where K represents the number of modulation types. The goal of automatic modulation classification is to learn a mapping function $f:{\mathbb{C}}^T\to Y$ that can accurately predict the corresponding modulation category for any input signal s.

Furthermore, given the high cost of acquiring large-scale labeled data in practical applications, this paper adopts a self-supervised learning strategy for feature pre-training, followed by fine-tuning on a small amount of labeled samples. Given an unlabeled dataset ${\mathcal{D}}_u={\left\{s_i\right\}}_{i=1}^N$, the model learns a feature encoder $E(·):{\mathbb{C}}^T\to {\mathbb{R}}^d$ through self-supervised tasks. This encoder aims to extract discriminative embedding representations $z=E\left(s\right)$ from raw signals. Subsequently, the model is fine-tuned on a limited labeled dataset ${\mathcal{D}}_l=\left\{(s_i,y_i)\right\}$, ultimately achieving efficient modulation classification.

3.2. Overview of CAEP

The proposed CAEP framework, illustrated in Figure 2, leverages the JEPA to construct a multi-modal self-supervised pre-training task. In the intra-modal prediction phase (Figure 2A), the framework obtains dual-domain inputs via signal splitting and STFT, subsequently employing the time-predictor and image-predictor to perform sub-series forecasting and masked token recovery within the latent space. In the inter-modal prediction phase (Figure 2B), the alignment of cross-modal embeddings through MSE loss forces the time-encoder and image-encoder to capture semantic consistency across the time and frequency domains. By incorporating a stop-gradient mechanism to prevent representation collapse, the pre-trained time-encoder is then utilized in the fine-tuning phase (Figure 2C) to extract precise signal features, enabling robust classification of modulation patterns, such as BPSK and QAM64, even under low signal-to-noise ratios.

Figure 2. The overall architecture of the proposed CAEP framework.

3.3. Intra-Modal Embedding Prediction

The intra-modal pre-training tasks are designed to exploit temporal dependencies and local correlations within a single representation dimension. To model the dynamic evolution of signals along the time axis, this work employs prediction as the self-supervised objective. The proposed dynamic segmentation method is illustrated in Figure 3.

Figure 3. The dynamic segmentation method. Learning feature representations via dynamic prediction across diverse window sampling.

Given an input signal S of length L, two sub-series of length l are randomly intercepted along the time axis to serve as the current sub-series $s_t$ and the future sub-series $s_{t+\Delta }$:

$$s_t=S[\tau :\tau +l],s_{t+\Delta }=S[\tau +\delta :\tau +\delta +l]$$

(3)

where $\tau$ is the starting point, l represents the length of the observation window, and $\delta$ denotes the time-step stride. We select $\tau =0$ and $l=L/2$ and set the step stride $\delta \sim \mathcal{U}(0,l)$ to preserve dynamic segmentation. Under this configuration, the required signal length is $\delta +l$, which is no more than the length of the input signal L, as $\delta +l\le 2l=L$. Therefore, the original signal length is sufficient for dynamic segmentation in each pre-training epoch, ensuring that the completeness of the acquired current and future sub-series. Subsequently, a time-domain encoder $E_t(·)$ is utilized to map these sub-series into a latent embedding space:

$$z_t=E_t\left(s_t\right),z_{t+\Delta }=E_t\left(s_{t+\Delta }\right)$$

(4)

A time-domain predictor $P_t(·)$ is constructed to infer the embedding representation of the future state, denoted as ${\widehat{z}}_{t+\Delta }=P_t\left(z_t\right)$, based on the current state $z_t$. The optimization objective of this task is to minimize the MSE between the predicted value and the ground-truth embedding:

$${\mathcal{L}}_{temp}={\parallel {\widehat{Z}}_{t+\Delta }-\mathrm{sg}\left(z_{t+\Delta }\right)\parallel }^2$$

(5)

where $sg(·)$ denotes the stop-gradient operation, which prevents gradient propagation to the target encoder. By forcing the predictor to learn solely from source embeddings, it is often utilized as a critical constraint against representation collapse within the JEPA framework [29].

Considering that time–frequency analysis can reveal the non-stationary characteristics of electromagnetic signals, this work obtains the time–frequency representation of the signal, denoted as $X=STFT\left(s\right)$, through the STFT. Subsequently, an image-encoder $E_f(·)$ is employed to process the original complete time–frequency map and the masked time–frequency map $X_{mask}$ (subjected to random local occlusion) to extract the corresponding embedding representations $z_f$ and $z_f^m$, respectively. Here, the masking ratio is set to $\chi$. A predictor $P_f(·)$ is designed to recover the complete features ${\widehat{z}}_f=P_f\left(z_f^m\right)$ from the corrupted semantic information. Accordingly, the mask reconstruction loss is defined as

$${\mathcal{L}}_{mask}={\parallel {\widehat{z}}_f-\mathrm{sg}\left(z_f\right)\parallel }^2$$

(6)

The time-domain signal encoder adopts a 1D-ResNet18 architecture, while the image-encoder utilizes a 2D-ResNet18 structure. Both encoders are followed by a linear projection head. For the predictive modules, both the time-domain and image-predictors consist of a multi-layer perceptron (MLP) with two linear layers. The intra-modal pre-training task integrates the aforementioned two loss components through weighted fusion, achieving an initial aggregation of the signal’s multi-dimensional features:

$${\mathcal{L}}_{intra}={\mathcal{L}}_{temp}+{\lambda }_1{\mathcal{L}}_{mask}$$

(7)

where ${\lambda }_1$ is the loss weighting of the masked prediction.

3.4. Inter-Modal Embedding Prediction

To model the complementary and reciprocal relationships between the time and time–frequency domains, this paper introduces a cross-modal embedding mapping task. This task requires the model to possess the capability of inferring high-order time–frequency distributions from low-order temporal signals. Specifically, a cross-modal predictor $P_{inter}(·)$ is utilized to predict the corresponding time–frequency embedding ${\widehat{z}}_f$ from the time-domain embedding $z_f$, denoted as ${\widehat{z}}_f=P_{inter}\left(z_t\right)$. Subsequently, an alignment constraint loss is imposed:

$${\mathcal{L}}_{inter}={\parallel {\widehat{z}}_f-\mathrm{sg}\left(z_f\right)\parallel }^2$$

(8)

This process compels the encoders across different modalities to reach a semantic-level consensus, thereby significantly enhancing the robustness of the learned representations. By integrating the intra-modal and inter-modal tasks, the final joint optimization objective function is formulated as

$${\mathcal{L}}_{total}={\mathcal{L}}_{intra}+{\lambda }_2{\mathcal{L}}_{inter}$$

(9)

where ${\lambda }_2$ is the weighting parameter of the inter-modal embedding prediction loss.

3.5. Fine-Tuning

Upon completion of the pre-training phase, the trained encoder $E_t(·)$, which possesses robust feature extraction capabilities, is retained and frozen (or fine-tuned). In the downstream AMC task, a linear classification head is appended to the encoder, and the model undergoes supervised training using the cross-entropy loss function:

$${\mathcal{L}}_{cls}=-{\displaystyle \sum _{j=1}^K}{y}_{j}log\left({\widehat{y}}_j\right)$$

(10)

where $y_j$ is the ground-truth label value of the sample for the j-th category. During the fine-tuning stage, the parameters of both the encoder and the classification head are simultaneously updated via the backpropagation algorithm to achieve high-precision identification of modulation signals.

4. Experiment

4.1. Datasets

The performance of CAEP is evaluated on two open-source datasets, which employ similar simulation and noise modeling strategies.

RadioML2016.10b [30]: This dataset comprises two analog modulation types (WBFM and AM-DSB) and eight digital modulation types (CPFSK, BPSK, 8PSK, GFSK, QAM16, QAM64, 4PAM, and QPSK). The signal-to-noise ratio (SNR) ranges from −20 dB to 18 dB with a step of 2 dB. Each signal contains 128 time steps, divided into in-phase (I) and quadrature (Q) components. The dataset is composed of 1,200,000 signal samples in total, containing 6000 samples per category at each SNR level. Figure 4 shows the temporal waveforms and spectrograms of modulation type QAM16 at different SNR levels in RadioML2016.10b datasets.

Figure 4. The temporal waveforms and spectrograms of QAM16 at (a) 0 dB, (b) 8 dB, and (c) 18 dB in the RadioML2016.10b dataset. In the temporal waveform, the blue part designates the I component and the orange is the Q component.

RadioML2016.04c [31]: This dataset encompasses 162,060 signal samples across eleven modulation categories (the ten types found in RadioML2016.10b plus AM-SSB). It maintains the same SNR range and signal format as the 10b version. Notably, the number of samples per modulation category in this dataset is unbalanced, varying from 4120 to 24,940. The temporal I/Q signals and time–frequency spectrograms of modulation class QAM16 at various SNRs are shown in Figure 5.

Figure 5. The temporal waveforms and spectrograms of QAM16 at (a) 0 dB, (b) 8 dB, and (c) 18 dB in the RadioML2016.10c dataset.

4.2. Implementation Details

In the self-supervised pre-training phase, the initial learning rate is set to 0.001 with a batch size of 512. The balancing coefficients for the two loss components, ${\lambda }_1$ and ${\lambda }_2$, are both assigned a value of 0.5, and the masking ratio $\chi$ is set to 0.9. During the fine-tuning stage, the learning rates for the encoder and the classification head are configured at 0.0005 and 0.005, respectively, with a batch size of 64. Moreover, due to the length of each sample in the two datasets being 128, a specific temporal sampling scheme is designed: the starting point of the current sub-series is fixed at 0 with a length of 64; the starting point of the future sub-series is determined by $\delta$. Here, $\delta$ is randomly sampled from a uniform distribution within the interval [0, 64], ensuring that the model can effectively capture dynamic correlations across various time spans. The detailed process and rationality of dynamic segmentation can be followed in Section 3.3.

The datasets are partitioned into pre-training, validation, and test sets according to a ratio of 6:2:2. To verify the effectiveness of the pre-trained features, a small part of labeled data is selected from the pre-training set for model fine-tuning with the validation set and a test set reserved for final modulation recognition. This paper adopts the Overall Accuracy (OA) and the Kappa coefficient as the comprehensive evaluation metrics.

4.3. Comparison with Related Works

To validate the superiority of CAEP, we conduct a performance comparison against a series of state-of-the-art self-supervised and contrastive learning models, including SemiAMC [11], MoCoV3 [32], APFS [33], DINO [34], TcssAMR [35], CC [36], CMAE [37], MCLHN [12], and MM-CL [21].

Among these methods, SemiAMC, MoCoV3, APFS, and DINO represent the effective adaptation of traditional self-supervised frameworks from the computer vision field to modulation classification tasks. Furthermore, methods such as TcssAMR, CMAE, and MCLHN introduce adaptive improvements in data augmentation strategies and loss function designs specifically tailored for AMC scenarios. More advanced approaches, including CC and MM-CL, delve into the integration of multi-modal information, thereby substantially broadening the research horizon of AMC technologies.

Table 1 presents the 1-shot and 5-shot classification results of various methods across two datasets with the SNR ranging from 0 dB to 18 dB. On the RadioML2016.10b dataset which contains 1.2 million samples, CAEP achieves an OA of 0.740, outperforming the second-best method, MM-CL, by approximately 1.7 percentage points. Meanwhile, the proposed method also demonstrates a significant performance leap on the RadioML2016.10c dataset. Such a growth curve under massive data regimes indicates that CAEP possesses higher information capacity. It can effectively ingest the vast knowledge brought by large-scale pre-training and transform data scale into recognition precision, showing superior scalability over traditional convolutional or contrastive learning frameworks when processing high-dimensional electromagnetic data.

Table 1. The performance comparison on different datasets and metrics at 0–18 dB with 1-shot fine-tuning.

Methods	RML2016.10b		RML2016.04c
	OA	Kappa	OA	Kappa
SemiAMC [11]	0.365	0.295	0.530	0.475
MoCoV3 [32]	0.624	0.583	0.704	0.668
APFS [33]	0.416	0.351	0.646	0.603
DINO [34]	0.619	0.576	0.705	0.670
TcssAMR [35]	0.554	0.505	0.604	0.556
CC [36]	0.612	0.569	0.699	0.662
CMAE [37]	0.595	0.550	0.685	0.646
MCLHN [12]	0.634	0.593	0.672	0.632
MM-CL [21]	0.723	0.693	0.802	0.777
CAEP	0.740	0.709	0.807	0.783

Furthermore, CAEP exhibits outstanding performance in terms of classification consistency and stability. On both RadioML2016.10b and RadioML2016.04c, their Kappa coefficients rank first (0.709 and 0.783, respectively), reflecting clearer discriminative boundaries when faced with multi-class confusion. Notably, CAEP’s standard deviation on RML2016.10b is only 0.010, making it the method with the least fluctuation.

Under the 5-shot fine-tuning configuration, the experimental results presented in Table 2 demonstrate that the CAEP framework achieves state-of-the-art performance on the RML2016.10b dataset; its OA of 0.807 and Kappa of 0.786 significantly surpass all baseline methods, including MM-CL and MoCoV3, thereby validating the superior feature modeling capability of the JEPA-based latent-space prediction mechanism for complex modulated signals. Furthermore, CAEP maintains a high-level OA of 0.854 on the RML2016.04c dataset, remaining in the top tier and confirming the framework’s robustness across diverse channel conditions; this underscores the clear generalization advantages of dual-domain embedding prediction over traditional contrastive learning (e.g., MoCoV3 and DINO) and masked reconstruction (e.g., CMAE) approaches.

Table 2. The performance comparison on different datasets and metrics at 0–18 dB with 5-shot fine-tuning.

Methods	RML2016.10b		RML2016.04c
	OA	Kappa	OA	Kappa
SemiAMC [11]	0.500	0.445	0.778	0.750
MoCoV3 [32]	0.718	0.688	0.848	0.828
APFS [33]	0.636	0.595	0.831	0.810
DINO [34]	0.705	0.672	0.850	0.831
TcssAMR [35]	0.701	0.668	0.849	0.830
CC [36]	0.712	0.680	0.850	0.832
CMAE [37]	0.690	0.690	0.852	0.833
MCLHN [12]	0.677	0.641	0.830	0.808
MM-CL [21]	0.790	0.767	0.897	0.884
CAEP	0.807	0.786	0.854	0.835

By observing the confusion matrices at various SNR levels in Figure 6 and Figure 7, it is clearly evident that as the SNR gradually increases from 0 dB to 18 dB, the diagonal elements of the matrices become increasingly prominent, with the color intensity rapidly concentrating along the main diagonal. This indicates that the CAEP framework effectively leverages the physically consistent representations learned during the self-supervised pre-training phase. In the low-to-medium SNR range of 0–4 dB, despite the presence of some noise interference, the model is still capable of accurately capturing modulation types with distinct features, such as BPSK, CPFSK, and GFSK. When the SNR reaches above 10 dB, the confusion matrices tend toward an ideal diagonal matrix, demonstrating the exceptional classification precision of the proposed method under high-SNR conditions.

Figure 6. Confusion matrices of CAEP on the RadioML2016.10c dataset with 1-shot fine-tuning at various SNR levels: (a) 0 dB, (b) 2 dB, (c) 4 dB, (d) 6 dB, (e) 8 dB and (f) 10 dB.

Figure 7. Confusion matrices of CAEP on the RadioML2016.10c dataset with 1-shot fine-tuning at various SNR levels: (a) 12 dB, (b) 14 dB, (c) 16 dB and (d) 18 dB.

4.4. Ablation Study

To delve into the specific contributions of individual components within the CAEP framework to modulation recognition performance, we design four sets of comparative experiments on the RadioML2016.04c dataset. These experiments cover the evolutionary process from uni-modal feature extraction to multi-modal collaborative learning.

• Time-domain prediction only: Only the temporal prediction task ${\mathcal{L}}_{temp}$ is retained to validate the model’s capability in modeling the temporal dynamics of raw waveforms.
• Time–frequency mask prediction only: Only the latent masked representation reconstruction task ${\mathcal{L}}_{mask}$ in the time–frequency domain is retained to evaluate the independent discriminative utility of time–frequency structural features.
• No cross-modal prediction: Both intra-modal prediction tasks are retained, but the inter-modal embedding alignment loss ${\mathcal{L}}_{inter}$ is removed to investigate the incremental value of cross-modal correlation information.
• Multi-channel collaborative prediction: The complete CAEP method proposed in this paper is applied, which integrates dual constraints from both intra-modal and inter-modal tasks to achieve comprehensive representation learning.

Regarding the experimental setup, this paper focuses on evaluating the model’s generalization performance under scenarios with extremely sparse labels. Specifically, we randomly sample 1% and 5% of the labeled data for downstream fine-tuning and record the average OA across the high range 0–18 dB. Furthermore, to investigate the influence of the masking mechanism on semantic extraction, two masking ratios, 0.75 and 0.9, are configured for comparative testing under each few-shot fine-tuning setting. Aside from these variables, all other experimental conditions remain consistent, including the number of pre-training steps and learning rate, and the optimizer configurations remain consistent, to ensure the objectivity and fairness of the comparative results.

From the results of Table 3, the performance of utilizing either time-domain prediction ${\mathcal{L}}_{temp}$ or time–frequency mask prediction ${\mathcal{L}}_{mask}$ individually is relatively limited. However, when combined, all evaluation metrics show a marginal improvement under both 1-shot and 5-shot scenarios. This suggests that the evolutionary information in the time domain and the distributional characteristics in the time–frequency domain possess underlying logical complementarity, providing a more discriminative foundational representation for subsequent cross-modal prediction.

Table 3. The performance comparison on different datasets and OAs at 0–18 dB with 5-shot fine-tuning.

${\mathcal{L}}_{\mathit{temp}}$	${\mathcal{L}}_{\mathit{mask}}$	${\mathcal{L}}_{\mathit{inter}}$	$\mathit{\chi }=0.75$		$\mathit{\chi }=0.9$
			1-Shot	5-Shot	1-Shot	5-Shot
✓			0.679	0.819	0.680	0.821
	✓		0.620	0.835	0.621	0.839
✓	✓		0.688	0.837	0.690	0.840
✓	✓	✓	0.710	0.849	0.807	0.854

Furthermore, increasing the masking ratio has a positive impact on the depth of feature extraction. Notably, at $\chi =0.9$, the performance of the full model in the 1-shot task far exceeds that at $\chi =0.75$. This reflects that under the drive of high-difficulty prediction tasks (i.e., high-ratio occlusion), the JEPA framework is forced to learn more intrinsic electromagnetic semantics rather than simple local features, thereby significantly enhancing the model’s generalization capability and robustness in resource-constrained environments.

Moreover, the inter-modal embedding prediction loss ${\mathcal{L}}_{inter}$ is the key component for the leap in performance. Under the extreme sparsity of a 0.9 masking ratio and only 1% labeled samples, the introduction of this component causes the OA to surge from 0.690 to 0.807. This significant increment objectively proves that the semantic alignment between time-domain temporal dynamics and time–frequency structural features can greatly compensate for the limitations of a single modality in feature expression. Consequently, the model can still construct stable signal manifold patterns even with minimal supervision.

4.5. Sensitivity Analysis

4.5.1. Masking Ratio

To investigate the specific impact of $\chi$ on the pre-training representation capability within the time–frequency domain latent prediction task, this paper conducts a parameter sensitivity analysis on the RML2016.04c dataset. The masking ratios are set to 0.1, 0.25, 0.5, 0.75, and 0.9, aiming to examine the model’s semantic recovery capability under varying degrees of information deficiency. In the experimental evaluation phase, the model is fine-tuned under sparse supervision conditions with 1% and 5% labeled samples. The average OA across the high SNR range is recorded, and the experimental results are illustrated in Figure 8.

Figure 8. Impact of masking ratios $\chi$ on the classification performance under limited labeled data settings on the RML2016.04c dataset.

The results demonstrate the influence of the masking ratio on modulation classification performance under varying sparse labeling conditions. The experimental results reveal a significant non-linear correlation between the masking ratio and downstream task accuracy, where the optimal masking point is closely coupled with the volume of available labeled samples.

In the extremely constrained scenario with only 1% labeled samples, the classification performance exhibits an overall upward trend as the masking ratio increases, reaching a peak of 0.807 at a ratio of 0.9 as illustrated in Figure 8. This indicates that under conditions of extreme information scarcity, using a high-ratio occlusion to retain only a bit of the time–frequency information during the pre-training phase compels the model to bypass shallow local features. Instead, it is forced to learn the structured manifold of the signal from a global perspective. Such a “high-difficulty” prediction task substantially bolsters the generalization capability of the learned representations.

However, as the number of labeled samples increases to 5% labeled samples, the optimal masking ratio shifts from 0.9 to 0.25. This shift objectively reveals the delicate balance between the difficulty of the pre-training task and the volume of downstream supervisory information. When supervisory information is relatively sufficient, the downstream task’s demand for feature precision and detail preservation outweighs the need for global abstraction. A lower masking ratio helps retain more discriminative details, thereby facilitating more refined decision boundaries.

Additionally, the fluctuations in the 5-shot curve across different masking ratios are significantly smaller than those in the 1-shot curve, while consistently maintaining a higher performance level. This validates the positive impact of increasing the volume of annotated data on the model’s stability and performance.

4.5.2. Loss Weighting Analysis

Apart from the masking ratio, it is also crucial to assess the pre-training performance of the two loss weightings ${\lambda }_1$ and ${\lambda }_2$. Table 4 presents the sensitivity analysis of the hyperparameters on the RML2016.04c dataset to evaluate their influence on classification performance in the 1-shot scenario.

Table 4. Sensitivity analysis of hyperparameters ${\lambda }_1$ and ${\lambda }_2$ on the RML2016.04c dataset.

${\mathit{\lambda }}_1$	${\mathit{\lambda }}_2$	OA	Kappa
	0.1	0.644	0.594
0.1	0.5	0.618	0.563
	1.0	0.615	0.563
	0.1	0.796	0.770
0.5	0.5	0.807	0.783
	1.0	0.788	0.761
	0.1	0.560	0.500
1.0	0.5	0.620	0.564
	1.0	0.614	0.560

Table 4 presents the sensitivity analysis of hyperparameters on the RML2016.04c dataset to evaluate their influence on classification performance. The experimental results indicate that the coordination between these two parameters is crucial for optimal representation learning. Specifically, the CAEP framework achieves its peak performance when both ${\lambda }_1$ and ${\lambda }_2$ are set to 0.5, yielding an OA of 0.807 and a Kappa coefficient of 0.783. Deviations from this balanced weighting, such as increasing ${\lambda }_1$ to 1.0 while maintaining ${\lambda }_2$ at 0.1, result in a significant performance drop to an OA of 0.560. This sensitivity suggests that an appropriate trade-off between intra-modal and inter-modal embedding predictions is essential; either excessive or insufficient weighting of these objective functions may disrupt the latent-space forecasting task, thereby hindering the extraction of robust discriminative features under complex channel conditions.

4.6. Experiments of Dynamic Segmentation Effectiveness

In this section, we compare the impact of different splitting strategies—Direct-split and Dynamic-split—on the classification performance using the RML2016.04c dataset under 1-shot and 5-shot settings. Regarding the direct segmentation method, the original signal is split into two sub-series of completely equal length, with the starting point set at zero, i.e., $s_t=s[0:L/2],s_{t+\Delta }=s[L/2:L]$.

The results in Table 5 indicate that the Dynamic-split strategy consistently outperforms the Direct-split approach across both configurations. Consequently, the dynamic strategy proves to be more robust in data-constrained environments, significantly enhancing the model’s feature representation capabilities for few-shot automatic modulation classification.

Table 5. Performance comparison of different splitting strategies on the RML2016.04c dataset.

Splitting Strategy	1-Shot	5-Shot
Direct-split	0.790	0.846
Dynamic-split	0.807	0.854

5. Conclusions

This paper proposes CAEP, a multi-modal self-supervised pre-training framework based on the JEPA, aimed at addressing performance bottlenecks of AMC in scenarios with sparse labeling. By introducing dual paths for embedding prediction, CAEP can deeply extract physically consistent semantic features from complex environments without relying on negative pairs or decoders. In the intra-modal phase, dynamic sub-series segmentation is employed to flexibly split raw samples into current and future segments, capturing temporal correlations by predicting future information from the current time. Meanwhile, predicting masked spectrogram tokens enhances the model’s ability of time–frequency domain representation. Furthermore, the inter-modal embedding feature prediction is also proposed to obtain cross-modal latent alignment effectively.

Experimental results on the RadioML2016 series of benchmark datasets demonstrate that CAEP achieves competitive performance under 1-shot and 5-shot fine-tuning settings. Moreover, the proposed method exhibits superior representational capacity and robustness, particularly in large-scale data scenarios, validating the effectiveness of the JEPA paradigm for SSL-based AMC tasks.

However, CAEP has several limitations such as overlooking adaptability across diverse SNR environments and only considering two modalities. Future research could focus on enhancing the algorithm’s generalization under varying noise conditions and incorporating additional modalities to enhance extreme low-SNR robustness. Moreover, with the rapid advancement of large language models and intelligent agents, their integration into self-supervised AMC techniques is worth exploring.