Open-Set Automatic Modulation Recognition Based on Circular Prototype Learning and Denoising Diffusion Model

Abstract

Automatic modulation recognition (AMR) technology is a critical component of modern communication systems. However, conventional AMR methods based on the closed-set assumption struggle to detect unknown classes that may appear during testing. To address this limitation, this paper proposes an open-set automatic modulation recognition (OSAMR) framework, termed CPLDiff, that integrates circular prototype learning (CPL) with a denoising diffusion model (DDM) to detect unknown classes. The core idea of CPLDiff is to jointly leverage the class-level and instance-level information of the training samples. To achieve this, CPL is used to extract class-level information, while the diffusion model is employed to extract instance-level information. (1) Circular Prototype Learning: Prototype vectors are pre-optimized and fixed, and a bias radius is introduced to expand the feasible encoding space. (2) Denoising Diffusion Model: Noise is added to the sample, and the DDM is used to remove this noise. The probability of a sample belonging to a known class is proportional to the extent of noise removal. (3) Final Integration: The outputs of the CPL and the DDM are combined to perform OSAMR. We conducted comparative experiments and evaluated the proposed method using diverse metrics to ensure a comprehensive assessment of its effectiveness. The experimental results demonstrate that the CPLDiff method significantly improves the detection capability for unknown classes compared to state-of-the-art methods.

1. Introduction

Signal modulation technology plays a critical role in modern communication systems [1]. It converts low-frequency baseband signals into high-frequency radio frequency (RF) signals and uses variations in parameters such as frequency and amplitude to carry information, resulting in modulation schemes such as amplitude modulation (AM), frequency modulation (FM), phase modulation (PM), quadrature amplitude modulation (QAM), frequency shift keying (FSK), and phase shift keying (PSK). Different modulation schemes directly affect the transmission capacity and spectral usage efficiency of the communication systems.

Accurately identifying the modulation type of the received signal is essential for subsequent demodulation and information processing. While the transmitter and receiver agree on the modulation scheme in cooperative scenarios, no prior parameters of the transmitted signals, such as signal power, carrier frequency and phase offsets, timing information, and so on, are provided to the receiver in non-cooperative situations. The blind identification of modulation schemes is a major task for the receivers in such non-cooperative scenarios, which is referred to as the automatic modulation recognition (AMR) technology [1]. The key advantage of AMR lies in its ability to identify the modulation scheme of the received signal without any prior knowledge, increasing its importance in scenarios such as electronic warfare, spectrum monitoring [2], resource allocation [3], and cognitive radio [4].

In recent years, the rapid development of deep learning has provided a more efficient and robust solution for AMR. Deep learning-based methods leverage well-trained artificial neural networks (ANNs) to automatically extract discriminative features, and they first require labeled samples to train the artificial neural network, after which the trained network can be used to recognize unlabeled test samples [5,6]. During these two stages, most deep learning methods assume that the class set of the unlabeled test samples is consistent with that of the labeled training samples. This assumption is known as the closed-set assumption [1]. In terms of sample types, researchers [7] have classified them into the following two categories:

• Known Known Classes (KKCs): Classes for which we know that labeled samples exist during the training phase.
• Unknown Unknown Classes (UUCs): Classes for which no information is available during the training phase.

Based on this definition, the closed-set assumption means that both test samples and training samples belong to the set of KKCs.

AMR methods based on the closed-set assumption typically fail to hold in today’s complex and diverse communication environments, where signals may belong to an entirely new modulation type. This scenario is referred to in academia as the open-set scenario: only KKC samples are available during the training phase, but both KKC and UUC samples may appear during the testing phase. When faced with UUC samples, AMR methods based on the closed-set assumption will incorrectly classify them as one of the KKCs. This overconfidence over UUC samples can be critical in fields such as access control and electronic warfare. Open-set automatic modulation recognition (OSAMR) is an emerging technology for open-set scenarios, and its differences with close-set AMR are shown in

Table 1. Difference between close-set AMR and open-set AMR.

Task	Training	Testing	Goal
Close-set AMR	KKCs	KKCs	classifying KKCs
Open-set AMR	KKCs	KKCs and UUCs	identifying KKCs and rejecting UUCs

. In addition to classifying KKC samples, OSAMR is required to detect UUCs. The ability to reject UUCs makes OSAMR a key technology for applications such as authorization, monitoring, and adversarial scenarios [8,9,10]. Furthermore, OSAMR is an important step toward achieving adaptive modulation recognition systems, of which its goal is to enable autonomous learning and recognition. The overall process of adaptive modulation recognition systems involves certain tasks: novelty discovery [11], labeling, and class incremental learning [12,13]. OSAMR technology plays a crucial role in the novelty discovery task, and its performance directly impacts the subsequent learning steps.

The main constraint of open-set recognition lies in that only KKC samples are available during the training phase [7,14], and algorithms must be developed based solely on these samples to effectively detect UUCs. Under this constraint, some methods [15,16], based on metric learning, aim to achieve compact embeddings of the samples to reduce the overlap probability between KKCs and UUCs. Other methods use generative adversarial networks (GANs) [17] to simulate fake UUC samples, enhancing the recognition model’s ability to understand the open world. Additionally, some methods [18,19] approach the problem from a reconstruction perspective, where the reconstruction error for KKC samples is intuitively smaller than that for UUC samples. Given the inherent limitations of open-set tasks and the shortcomings of existing methods, the main challenges of OSAMR are summarized as follows:

• How to fully utilize the KKC samples during training remains the key challenge of OSAMR, as only these samples are accessible. UUC samples that appear in testing are invisible during the training phase. According to the information bottleneck theory [20], any supervised learning is to extract minimal but sufficient statistics with respect to the objective function. Therefore, methods that adopt metric learning may suffer significant information loss. Meanwhile, methods attempting to simulate UUC samples offer limited and uncertain benefits as the number of UUCs is unknown and could be extremely large. The key to achieving OSAMR lies in fully extracting the information from the training samples.
• How to enhance the detection capability for UUCs while maintaining the recognition accuracy of KKCs. In open-set recognition tasks, improving the UUC detection performance is often achieved at the cost of the recognition accuracy of KKCs. In other words, an improvement in one of performance typically results in a decline in the other. The ideal open-set recognition technology would achieve a higher detection rate for UUCs at the expense of only a slight reduction in the KKC recognition accuracy.

To address the above challenges, we propose CPLDiff, an OSAMR method designed to fully exploit the information carried by the training samples and achieve a higher UUC detection capability at a low cost in KKC recognition accuracy. The core idea of CPLDiff is quantifying two levels of information by extending circular prototype learning with a diffusion model. The former primarily focuses on class-level information by discovering the differences among KKCs and enriching the feature distribution. The latter learns the unique characteristics of individual samples to extract instance-level information. In particular, the diffusion model, by gradually adding and then removing noise, can achieve better reconstruction and generation results. OSAMR is achieved using similarity scores obtained from circular prototype learning and denoising scores derived from the diffusion model. In summary, the main contributions of CPLDiff are as follows:

• We enhance prototype learning by optimizing and fixing each prototype and encouraging samples to surround their corresponding prototype in a circular manner. This approach is termed circular prototype learning (CPL).
• We propose a diffusion model-based OSAMR strategy, where a certain amount of noise is randomly added to the samples. The probability of a sample belonging to the KKCs is proportional to the amount of noise removed by the diffusion model.
• We extend circle prototype learning with the diffusion model to more fully exploit the information in the training samples and jointly utilize both methods for the combined prediction of the samples.

2. Related Works

2.1. Traditional Automatic Modulation Recognition

Traditional automatic modulation recognition (AMR) techniques can be categorized into two categories: likelihood-based (LB) and feature-based (FB) methods [1]. The former compute the likelihood function [21] of the signal and match it to the maximum likelihood value, while the latter extract features from the signal, such as high-order cumulants [22], and utilize machine learning classifiers for recognition [22,23]. Likelihood-based recognition methods are computationally complex and require perfect channel-state information (CSI) [24] at the receiver end. Feature-based methods rely on features manually designed by experts in the signal domain. With the rapid advancement of communication technology and the increasing complexity of communication environments, traditional AMR techniques face significant challenges. Their low robustness makes them increasingly inadequate for diverse modulation schemes and complex communication environments.

2.2. Deep Learning-Based Automatic Modulation Recognition

Deep learning-based AMR methods can be categorized based on the types of artificial neural networks, including convolutional neural network (CNN) models [25,26,27,28,29,30], Long Short-Term Memory (LSTM) models [31], Transformer models [32,33], and hybrid models [34,35,36]. They can also be classified according to the type of inputs, such as raw IQ inputs, transformed inputs (e.g., statistical features, spectrograms, constellation diagrams), and mixed inputs. Additionally, they can be divided based on the learning approaches, including supervised, semi-supervised [37], and self-supervised learning [38]. In the early stages of research, ref. [25] employed VGG networks with raw IQ data as input, achieving promising results for modulation recognition. Liu et al. [27] proposed a new DL-AMR model based on Residual Neural Networks (ResNet) and DenseNet, allowing features learned from multiple layers to be effectively transferred to the classification module. In [26], to better adapt a 2D-CNN to the sequential signals, the S2M operator was introduced to convert time-series signals into 2D square matrices. To incorporate additional expert knowledge of the signal, ref. [39] integrated the instantaneous features of the signal into the CNN input and demonstrated the effectiveness of expert knowledge in improving performance. Rajendran et al. [40] transformed the I-Q signals into amplitude and phase and fed them into an LSTM network, which achieved high recognition accuracy. CNN models typically focus on spatial features, while LSTM models focus on temporal features, and combining both may lead to enhanced recognition performance. MCLDNN [34] combines 1D-CNNs, 2D-CNNs, and LSTM, achieving more efficient feature extraction and faster convergence speed. Ref. [38] constructed a contrastive learning model using two modalities of signals, time-series data and constellation diagrams, and transferred the encoder weights to downstream classification and clustering tasks, thus expanding feasible pathways for AMR technology.

2.3. Open-Set Recognition

Open-set recognition was first introduced by Scheirer et al. in [7], where the task definition was clarified and a simple OSR framework was proposed. Over years of development, OSR methods can generally be divided into two categories: discriminative model-based methods and generative model-based methods [14,41].

Discriminative Models. Discriminative models focus primarily on learning the information that enables each KKC to be differentiated from each other. Early research by Hendrycks et al. [42] proposed a simple OSR baseline that uses the maximum SoftMax probability from a neural network as the sample score. To detect UUC samples, a threshold is determined practically or computationally. Bendale et al. [43] analyzed the limitations of the SoftMax function and thus proposed the OpenMax function as a replacement for SoftMax. OpenMax adjusts the logits using the mean activation vector (MAV) and a Weibull distribution. This approach does not require modifications to the network structure or the training loss function, making it simple and easy to implement, but with higher computational complexity. Another feasible approach is to increase the number of neurons in the network’s output layer, which is taken by Proser [44]. Proser increased the number of output neurons from C to $C+N$, where C represents the number of KKC, and N represents the number of classifier placeholders. Classifier placeholders simulate the center for UUC samples, activating the highest value when the input is from a UUC and the second-highest when it originates from one of the KKCs.

Another promising research area is prototype learning, which belongs to the family of discriminative models. Prototype learning uses prototype points to represent class centers and assumes samples are located closer to their belonging prototype. The first work to combine prototype learning with deep learning was GCPL [16]. It was aimed at enabling neural networks to extract more robust and discriminative representations, which can enhance performance in closed-set recognition, open-set recognition, and incremental learning. More recent works include RPL [45] and its improved version, ARPL [15]. RPL uses a reciprocal point (RP) to represent the otherness of a single KKC, such that training samples are pushed away from their corresponding RP, while samples from other classes (including UUCs) are pulled closer to the RP. ARPL further enhances the performance of RPL by imposing a radius between the RP and its KKC samples.

Generative Models. Generative models achieve open-set recognition through two independent approaches. The first involves generating fake samples to assist discriminative models in understanding the open world, while the second embeds original samples and then performs reconstruction. G-OpenMax [46] enhances OpenMax with an additional GAN and adds an extra activation neuron to represent the UUCs. OSRCI [47] follows a similar approach to G-OpenMax, where the generated fake samples need to resemble a specific KKC class. These fake samples then serve as the boundary between KKCs and UUCs. Similarly, ARPL+CS [15] generates confusing samples to stand for UUCs. Based on the reconstruction error, C2AE [18] was the first to use an autoencoder for OSR. C2AE trains an encoder on a closed-set classification task and fixes its weights. It then decodes and reconstructs the features extracted by the encoder. Since the encoder is trained for a classification task, the feature vectors it produces lack sufficient information to support effective reconstruction. CROSR [19] recognized this limitation of C2AE and addressed it by performing both classification and reconstruction tasks simultaneously.

3. Materials and Methods

3.1. Problem Definition

In this subsection, we provide a concise yet clear explanation of the open-set recognition task. Given a C-KKC classification task and an available training dataset,

$${\mathcal{D}}_{tr}=\{({\mathbf{x}}_1,y_1),({\mathbf{x}}_2,y_2),\dots ({\mathbf{x}}_n,y_n)\},$$

(1)

where $y_i\in \{1,2,\dots ,C\}$ is the label of ${\mathbf{x}}_i$, and n is the size of the training dataset. And with an unlabeled testing dataset

$${\mathcal{D}}_{te}=\{\left({\mathbf{x}}_1\right),\left({\mathbf{x}}_2\right),\dots ,\left({\mathbf{x}}_m\right)\},$$

(2)

where m is the number of testing samples, the goal of OSR is to correctly label each sample in ${\mathcal{D}}_{te}$. The true label of any test sample belongs to the set $\{1,2,\dots ,C,C+1,\dots ,C+U\}$, where U is the number of UUCs. As U is unclear in testing, an OSR algorithm needs to label any sample of the KKCs with $\{1,2,\dots ,C\}$ and any sample of UUCs with $\{C+1\}$, which means a rejection for classification. The performance of an OSR algorithm is determined by both the identifying accuracy for KKCs and the rejection rate for UUCs.

3.2. Overview

We display the proposed circular prototype learning and diffusion model (CPLDiff) OSAMR workflow in Figure 1. The framework is composed of the following three modules:

• Circular prototype learning for the close-set prediction and similarity score.
• Denoising diffusion model for denoising the score.
• Score integration and prediction calibration.

A detailed explanation of the workflow for the proposed CPLDiff framework is as follows. First, CPL optimizes the location of prototype points, ensuring they are evenly distributed in the latent space. Once optimized, the prototype points are fixed, meaning their positions do not update during the training process. CPL then extends the optimal representation $\mathbf{z}$ (obtained through the encoding of the input signal $\mathbf{s}$) from the prototype point to a circular region around it. This extension enriches the information contained in $\mathbf{z}$ and prevents excessive information loss during the encoding process. Based on the distance between $\mathbf{z}$ and each of the circular regions, CPL outputs a distance score $S_1$ and a closed-set prediction $\overline{y}$ (Section 3.3). Recognizing the sample’s belonging class solely based on CPL only utilize the class-level information contained in the sample. To further verify the prediction results output by CPL, a score calculation method based on the denoising diffusion model (DDM) is attached after CPL. We add a certain amount of random noise to the original signal $\mathbf{s}$ and use the diffusion model to remove the noise. The extent of noise removal is quantified as a score $S_2$, which reflects the instance-level information, and the probability that the sample belongs to the KKCs is proportional to this score (Section 3.4). Finally, scores $S_1$ and $S_2$ are weight-summed to obtain the final score, which is then used to correct the closed-set prediction result $\overline{y}$ (Section 3.5).

3.3. Circular Prototype Learning

3.3.1. Data Process and Encoding

One complex-value modulation signal can be denoted as

$$s\left(t\right)=s_i\left(t\right)+\mathbf{j}·s_q\left(t\right),$$

(3)

where $s_i\left(t\right)$ and $s_q\left(t\right)$ represent the in-phase (Real) and quadrature (Image) components at time $t\in \{1,2,3,\dots ,L\}$, and L is the sampled length of the signal. As the signal has two channels, it is generally stored in a computer file system in a matrix manner as

$$\mathbf{s}=\left[\begin{array}{c}{\mathbf{s}}_i\\ {\mathbf{s}}_q\end{array}\right],$$

(4)

where ${\mathbf{s}}_i=[s_i\left(1\right),s_i\left(2\right),\dots ,s_i\left(L\right)]$, and ${\mathbf{s}}_q=[s_q\left(1\right),s_q\left(2\right),\dots ,s_q\left(L\right)]$. To eliminate the effects caused by fluctuations in the output power of the signal transmitter, we normalize the power of the signals according to the following formula:

$$\mathbf{s}={\displaystyle \frac{\mathbf{s}}{std\left(\mathbf{s}\right)}},$$

(5)

where the $std(·)$ function calculates the standard deviation of elements in matrix $\mathbf{s}$. The normalization computational overhead is proportional to the matrix size. In real-world environments, the signals’ amplitude may vary in a wide range; thus, conducting normalization helps avoid gradient explosion and achieve a fast convergence speed. In addition to normalizing the signal power, we enhance the signal diversity by introducing the Givens transform [48] during the preprocessing stage, which is formulated as $s^{\prime }\left(t\right)=(s_i\left(t\right)+\mathbf{j}·s_q\left(t\right))·exp(\mathbf{j}·\varphi )$ or in a matrix manner as

$$\left[\begin{array}{c}{\mathbf{s}}_i^{\prime }\\ {\mathbf{s}}_q^i\end{array}\right]=\left[\begin{array}{cc}cos\varphi & -sin\varphi \\ sin\varphi & cos\varphi \end{array}\right]\left[\begin{array}{c}{\mathbf{s}}_i\\ {\mathbf{s}}_q\end{array}\right],$$

(6)

where $\varphi \in [0,2\pi )$ is the rotation angle. Modulation signals are complex-valued signals, where the real part represents the in-phase channel, and the imaginary part represents the quadrature channel. Applying a Givens transform to a modulation signal effectively rotates the original complex signal clockwise around the origin of the complex plane and generate more signals with varying initial phases but identical modulation types. This online data augmentation, like cropping, scaling in the CV domain, increases the model’s robustness and is not unnecessary for test samples.

To encode a signal into a latent representation, we use a convolutional neural network (CNN) as the encoder, which is formulated as

$$\mathbf{z}=L_2\left(f_{\theta }\left(\mathbf{s}\right)\right),$$

(7)

where $f_{\theta }(·)$ denotes SigRes34 [38], a CNN model, $L_2(·)$ is the L2 normalization function, and $\mathbf{z}\in {\mathbb{R}}^{1\times D}$ is the representation vector of $\mathbf{s}$. L2 normalization is adopted on the raw encoded representation to simplify the distance metric into cosine similarity. Moreover, L2 normalization constrains the feature representation into a hyper-surface, making it possible for the next prototype pre-optimization.

3.3.2. Prototype Pre-Optimization

In this section, we introduce the rationale and methodology behind the pre-optimization (PPO). The classic prototype learning method, GCPL, assigns a prototype vector to each KKC, denoted as $\{{\mathcal{P}}_1,{\mathcal{P}}_2,\dots ,{\mathcal{P}}_C\}$, where the values of these prototype vectors are set as learnable parameters, meaning their locations are continuously updated throughout the training process. The loss function of GCPL consists of a cross-entropy loss and a center loss, both of which focus on minimizing the distance between prototypes and samples. However, they do not consider the relative distances between the prototype vectors themselves. After updates, some prototype vectors may end up being far apart, while others are close to each other. This imbalance results in the open-set risk being focused more on the prototype vectors that are closer together. Therefore, in addition to considering the sample-to-prototype distance, the prototype-to-prototype distance should also be taken into account.

PPO is proposed to address the issue of neglecting the prototype-to-prototype distance. First, PPO assumes that the distribution of prototype vectors in the feature space should be uniform, which helps evenly distribute the open-set risk across all prototype vectors. Second, the distribution of prototype vectors should be discrete, which helps reduce the open-set risk. Therefore, prior to training the encoder on the training set, the distribution of the prototype vectors is optimized. According to the main ideas of PPO, the designed loss function is

$${\mathcal{L}}_p={\displaystyle \frac{1}{C\times C}}\sum _{j=1}^C\sum _{k=1}^C\mathrm{CosineSim}({\mathcal{P}}_j,{\mathcal{P}}_k),$$

(8)

$$\mathrm{CosineSim}({\mathbf{x}}_1,{\mathbf{x}}_2)={\displaystyle \frac{{\mathbf{x}}_1·{\left({\mathbf{x}}_2\right)}^T}{\max (\parallel {\mathbf{x}}_1{\parallel }_2·\parallel {\mathbf{x}}_2{\parallel }_2,ϵ)}}.$$

(9)

The function $\mathrm{CosineSim}(·,·)$ calculates the cosine similarity between the inputs, as defined in Equation (9), and $ϵ$ is introduced to prevent division by zero. When minimizing Equation (8), the average similarity between prototype vectors is minimized, or in other words, the average distance is maximized. Since the prototype vectors are constrained on a convex hyper-sphere through L2 normalization, the minimum value of Equation (8) is attained only when all prototype vectors are uniformly distributed on the hyper-sphere. After PPO, all prototype vectors are fixed and no longer updated during the subsequent encoder training process. While other methods, such as GCPL and SoftMax, jointly train their encoders and classifiers, our separate training strategy does not include more computation overhead than these joint training methods.

3.3.3. Circular Constraints

GCPL, RPL, and ARPL all encourage samples to either approach prototype points or move away from reciprocal points, resulting in a gradual reduction in information entropy during this process. Consider two samples ${\mathbf{s}}_1$ and ${\mathbf{s}}_2$ that belong to the same KKC. There remains a significant difference between these two samples in the original signal space even though they belong to the same category. However, when ${\mathbf{s}}_1$ and ${\mathbf{s}}_2$ are embedded into the latent space, denoted as ${\mathbf{z}}_1$ and ${\mathbf{z}}_2$, they both cluster around the corresponding prototype point. The result is that the difference between ${\mathbf{z}}_1$ and ${\mathbf{z}}_2$ becomes much smaller than that between ${\mathbf{s}}_1$ and ${\mathbf{s}}_2$. Moreover, the closer ${\mathbf{z}}_1$ and ${\mathbf{z}}_2$ are to the prototype vector of their respective class, the smaller their difference.

For closed-set classification tasks, extracting shared features from samples within the same class helps improve recognition performance. However, for open-set recognition, excessive loss of information increases the risk of misclassifying UUCs as KKCs. This is because crucial information for distinguishing UUCs from KKCs may be eliminated during the embedding process.

If following the logit calculation strategy in GCPL, formulated as

$$p(y=k|s;f_{\theta },\mathcal{P})\propto \mathrm{CosineSim}(f_{\theta }\left(\mathbf{s}\right),{\mathcal{P}}_k),$$

(10)

then for any input $\mathbf{s}$, its optimal encoding position in the latent space should coincide with its prototype vector. During the process where $f_{\theta }\left(\mathbf{s}\right)$ moves closer to the prototype point from other regions in the latent space, the richness of the information it carries tends to diminish. The proposed circular constraints are intended to alleviate the downward trend of information richness during the encoding of the input $\mathbf{s}$. From Equation (10), we can infer that the reason for the reduction in information entropy is that there is only one optimal solution for $f_{\theta }\left(\mathbf{s}\right)$. Therefore, expanding the number of optimal solutions for $f_{\theta }\left(\mathbf{s}\right)$ will be beneficial in preserving more information in the feature representation. We modified Equation (10) as follows:

$$p(y=k|s;f_{\theta },\mathcal{P},\mathcal{R})\propto {\displaystyle \frac{1-\left|\right(\mathrm{Cosine}\_\mathrm{sim}(f_{\theta }\left(\mathbf{s}\right),{\mathcal{P}}_k)-\mathcal{R}|}{\tau }},$$

(11)

where $\mathcal{R}\in [0,1]$ is a hyper-parameter. When $\mathcal{R}=1$, Equation (11) degrades to Equation (10). When $\mathcal{R}<1$, the number of optimal solutions for $f_{\theta }\left(\mathbf{s}\right)$ increases from a single point to infinite. Then, the loss function to train $f_{\theta }$ is

$${\mathcal{L}}_c(\mathbf{s};f_{\theta },\mathcal{P},\mathcal{R})=-\log p(y=k|s;f_{\theta },\mathcal{P},\mathcal{R}).$$

(12)

3.4. Denoising Score on Denoising Diffusion Model

The denoising diffusion model (DDM) [49,50] is a deep learning method that generates high-quality data samples by simulating a gradual noise removal process. Its core idea is derived from the diffusion process in statistical physics, particularly the transition from a high-entropy (disordered, high-noise) state to a low-entropy (ordered, low-noise) state. The DDM consists of two main components: the forward diffusion process and the reverse denoising process, as shown in Figure 2. The forward process is a Markov chain that progressively adds Gaussian noise to the data until it reaches a state that is close to a Gaussian distribution. The reverse process, in contrast, is a parameterized Markov chain in which a neural network is trained to determine how to remove noise from the noised data at each step, thereby gradually restoring the original data.

Due to its high quality and diversity in image generation, DDMs (denoising diffusion models) are gradually replacing GANs and becoming a key technology in various applications such as text-to-image generation [51], video generation [52], and speech synthesis [53]. DDMs have demonstrated strong capabilities in data generation, suggesting their ability to effectively capture data distributions, thereby providing an additional pathway for open-set recognition. All open-set recognition research based on DDMs in this paper was conducted using the DDIM framework provided by Mousai [54].

3.4.1. $\mathcal{V}$-Objective Training

In the original denoising diffusion probabilistic model (DDPM) [49], during the reverse denoising process, the diffusion model is tasked with predicting the noise added at time step $t-1$ based on the noised ${\mathbf{s}}_t$ at time step t. This prediction is implemented using an artificial neural network (ANN). As denoising diffusion models evolved, additional prediction objectives have been introduced alongside the noise prediction objective, such as the $\mathbf{v}$-objective and the $\mathbf{s}\left(0\right)$-objective. Essentially, training a DDM is about training an ANN so that it can accurately predict the desired objective. Since the prediction target of the ANN has the same shape as the input, most DDMs employ a U-Net architecture for the ANN. The DDM used for open-set recognition in this study utilizes UnetV0 provided by Mousai. For simplicity and to avoid the region devour issue mentioned in [55], we trained the diffusion model using data from a single KKC, resulting in the generation of C sets of model weights.

In Figure 2, the maximum diffusion step is set to T; then, for any time step t, the noised data ${\mathbf{s}}_t$ can be obtained as

$$\mathbf{s}\left({\sigma }_t\right)=\alpha \left({\sigma }_t\right)·{\mathbf{s}}_0+\beta \left({\sigma }_t\right)·ϵ,$$

(13)

$$\alpha \left({\sigma }_t\right)=sin\left({\displaystyle \frac{\pi }{2}}·{\sigma }_t\right),$$

(14)

$$\beta \left({\sigma }_t\right)=cos\left({\displaystyle \frac{\pi }{2}}·{\sigma }_t\right),$$

(15)

$${\sigma }_t={\displaystyle \frac{t}{T}},$$

(16)

where $ϵ$ is the noise sampled from a standard normal distribution $\mathcal{N}(0,1)$, and ${\sigma }_t$ means the noise level. The v-objective is formulated as

$$\mathbf{v}\left({\sigma }_t\right)={\displaystyle \frac{\partial \mathbf{s}\left({\sigma }_t\right)}{\partial {\sigma }_t}}=\alpha \left({\sigma }_t\right)·ϵ-\beta \left({\sigma }_t\right)·{\mathbf{s}}_0.$$

(17)

The intended meaning of $\mathbf{v}\left({\sigma }_t\right)$ is the noise-added velocity at a noise level of ${\sigma }_t$. With noised data ${\mathbf{s}}_t$ and noise level ${\sigma }_t$ as inputs, the v-objective-based DDM tries to estimate an ANN model

$$\widehat{\mathbf{v}}\left({\sigma }_t\right)=f_u(\mathbf{s}\left({\sigma }_t\right),{\sigma }_t),$$

(18)

where $f_u(·,·)$ denotes UNetV0 in this paper. The parameters in $f_u$ are optimized by minimizing the following function:

$${\mathbb{E}}_{t\sim [0,T]|t\in \mathbb{Z},{\mathbf{s}}_{{\sigma }_t},{\sigma }_t}[\parallel f_u(\mathbf{s}\left({\sigma }_t\right),{\sigma }_t)-\mathbf{v}\left({\sigma }_t\right){\parallel }_2^2].$$

(19)

In the training phase, the value of the maximum time step T is set very high, but the maximum value of ${\sigma }_t$ equals 1 however high T is. It should be noted that for every input $\mathbf{s}$, its corresponding noise level ${\sigma }_t$ is randomly sampled from $U[0,1]$.

3.4.2. DDIM-Based Denoising Score

A well-trained DDM is capable of gradually denoising from pure noise to synthesize data that are consistent with the categories of the training samples. The core idea of using a DDM for OSR is to add noise of a fixed level to the input. For KKC samples, this noise can be effectively removed, whereas for UUC samples, the noise removal is less effective.

The first step in implementing OSR with a DDM involves adding a specific level of noise to the original input $\mathbf{s}\left(0\right)$, as described in Equation (13). The noise level should be neither too small nor too large: a small noise level results in uniformly high noise removal, while a large noise level leads to uniformly low noise removal. In this study, the added noise level was set to 0.25, and the parameter T was set to 100. The value of T during inference can differ from its value during training, as the noise level’s maximum value remains 1 regardless of the value of T. The second step involves denoising the input using the DDIM algorithm with $f_u,\mathbf{s}\left({\sigma }_t\right),{\sigma }_t$ following the following specific steps:

• Input $\mathbf{s}\left({\sigma }_t\right)$ and ${\sigma }_t$ into $f_u$ to produce an estimate of the velocity as

  $$\widehat{\mathbf{v}}\left({\sigma }_t\right)=f_u(\mathbf{s}\left({\sigma }_t\right),{\sigma }_t).$$

  (20)
• Calculate an estimation of $\mathbf{s}$ at time step 0 based on Equations (13) and (17) as

  $$\widehat{\mathbf{s}}\left(0\right)=\alpha \left({\sigma }_t\right)·\mathbf{s}\left({\sigma }_t\right)-\beta \left({\sigma }_t\right)·\widehat{\mathbf{v}}\left({\sigma }_t\right).$$

  (21)
• Calculate a noise estimation of $ϵ$ at noise level ${\sigma }_t$ based on Equations (13) and (17) as

  $$\widehat{ϵ}\left({\sigma }_t\right)=\beta \left({\sigma }_t\right)·\mathbf{s}\left({\sigma }_t\right)+\alpha \left({\sigma }_t\right)·\widehat{\mathbf{v}}\left({\sigma }_t\right).$$

  (22)
• Calculate an estimation for $\mathbf{s}$ at time step $t-1$ as

  $$\widehat{\mathbf{s}}\left({\sigma }_{t-1}\right)=\alpha \left({\sigma }_{t-1}\right)·\widehat{\mathbf{s}}\left(0\right)+\beta \left({\sigma }_{t-1}\right)·\widehat{ϵ}\left({\sigma }_t\right).$$

  (23)

As shown in the above equations, the maximum value of t equals 25 as we set $T=100$ and noise levels $\{{\sigma }_0=0.0,{\sigma }_1=0.01,\dots ,{\sigma }_{25}=0.25\}$ are pre-scheduled.

The above denoising steps should be repeated until $t=0$, which is shown in Figure 2. After the denoising procedure, a quantized score is required for OSR; thus, we designed the denoising score using the original input ${\mathbf{s}}_0$ and its estimation ${\widehat{\mathbf{s}}}_0$, which is formulated as

$$S_{ddm}(\mathbf{s}\left(0\right);f_u)=exp\left(-\parallel \mathbf{s}\left(0\right)-\widehat{\mathbf{s}}{\left(0\right)\parallel }_2^2\right)$$

(24)

3.5. Class-Wise Threshold Co-Calibration

The primary motivation for incorporating the DDM is to utilize instance-level denoising scores to further refine the close prediction results along with CPL scores, which are class-level. The introduction of denoising scores is excepted to bring more criteria.

In this paper, we treat an OSAMR task with C KKCs as C binary classification tasks, with the following steps.

For a given unlabeled testing sample $\mathbf{s}$, calculate its CPL logits and subsequently obtain its predicted label $\widehat{y}$ with the following equation:

$$Logi{t}_{cpl}(\mathbf{s},{\mathcal{P}}_k;f_{\theta })=1-|\mathrm{CosineSim}(f_{\theta }\left(\mathbf{s}\right),{\mathcal{P}}_k)-\mathcal{R}|,$$

(25)

$$\widehat{y}=\arg \underset{k}{\max }{\left\{Logi{t}_{cpl}(\mathbf{s},{\mathcal{P}}_k;f_{\theta })\right\}}_{k=1}^C,$$

(26)

where the value of $Logi{t}_{cpl}(\mathbf{s},{\mathcal{P}}_k;f_{\theta })$ is limited to $[-1,1]$. Now that the predicted label $\widehat{y}$ is available, the denoising score of $\mathbf{s}$ can be obtained through Equation (24). It should be noted that the parameters of $f_u$ are instantiated with those trained using samples of class $\widehat{y}$. The next step is to calculate the integrated score:

$$S(\mathbf{s};\mathcal{P},f_{\theta },f_u,\widehat{y})=\lambda ·Logi{t}_{cpl}(\mathbf{s},{\mathcal{P}}_{\widehat{y}};f_{\theta })+(1-\lambda )·S_{ddm}(\mathbf{s};f_u^{\widehat{y}}),$$

(27)

where $f_u^{\widehat{y}}$ denotes UNetV0 trained with samples of class $\widehat{y}$, and the hyper-parameter $\lambda$ is adopted to balance the two components. The magnitudes of the two scores are consistent, and they can be directly added together. Additionally, a threshold is required for class $\widehat{y}$, denoted as $t{h}_{\widehat{y}}$, with which calibration on the close-set prediction $\widehat{y}$ can be achieved as

$$y=\left\{\begin{array}{cc}\widehat{y}\hfill & S(\mathbf{s};\mathcal{P},f_{\theta },f_u,\widehat{y})\ge t{h}_{\widehat{y}},\hfill \\ C+1\hfill & \mathrm{else}.\hfill \end{array}\right.$$

(28)

A crucial aspect of this process is that the close prediction result is produced using CPL, while the DDM does not possess classification capabilities. Open-set recognition can be accomplished using only CPL; however, it cannot be achieved using only a DDM, as the DDM relies on the close prediction results generated via CPL.

The threshold for each KKC can be obtained manually or statistically. As for determining a threshold for a class by statistical methods, one general approach is to select a threshold based on the scores of training samples that can cover $\delta$ of those samples as known, where $\delta \in [0,1]$. The threshold determination process for class k under the $\delta$ condition is formulated in

$$t{h}_k=\underset{th}{min}\{p[S(\mathbf{s};\mathcal{P},f_{\theta },f_u,k)>th|th\in \mathbb{R}]\ge \delta \}.$$

(29)

Determining a threshold for each KKC does not introduce significant time complexity, as the threshold for a single KKC is determined using only the training samples of that class without involving samples from other classes. In other words, each sample is used only once, which is consistent with determining a shared threshold for all KKCs.

4. Results

4.1. Datasets

To evaluate the effectiveness of the proposed CPLDiff framework in the OSAMR task, we conducted comparative experiments on two publicly available modulation signal datasets: RadioML2016.10a and RadioML2016.04c [56]. A concise description and the OSAMR task settings are provided in

Table 2. Dataset description and OSAMR task settings.

Items	RadioML2016.10a	RadioML2016.04c
Max carrier frequency offset	50 Hz	100 Hz
Max sampling rate offset	500 Hz	1000 Hz
Energy normalization	Yes	No
Number of modulation schemes	11	11
Signal shape	$2\times 128$	$2\times 128$
SNR range (dB)	−20∼18, with an interval of 2	−20∼18, with an interval of 2
Number of signals per SNR	11,000	8103
Number of sinusoids used in frequency selective fading	8	8
Channel environment	Additive Gaussian white noise, selective fading (Rician + Rayleigh), Center Frequency Offset (CFO), Sample Rate Offset (SRO)	Additive Gaussian white noise, selective fading (Rician + Rayleigh), Center Frequency Offset (CFO), Sample Rate Offset (SRO)
KKCs	AM-DSB, AM-SSB, BPSK, GFSK, PAM4, QAM16, QAM64, QPSK, WBFM	AM-DSB, AM-SSB, BPSK, GFSK, PAM4, QAM16, QAM64, QPSK, WBFM
UUCs	8PSK, CPFSK	8PSK, CPFSK
Training size–testing size	7:3 class-wise	7:3 class-wise

.

4.2. Experimental Setup

4.2.1. Environment

All experiments were implemented using PyTorch 1.12.1 and conducted on a computer with an Intel (R) 12600KF CPU with 128 GB of RAM and an NVIDIA RTX 3090 GPU with 24 GB of RAM.

4.2.2. Parameters and Models

In the following experiments, we set $\tau =0.05$ and $\mathcal{R}=0.2$ in Equation (11). The AdamW [57] optimizer was used for updating $f_{\theta }$ and $f_u$, the learning rate was set to 0.001 for both the training of the CPL and DDM, the batch size was 640, and the total numbers of training epochs were 300 for CPL and 400 for DDM. For prototype pre-optimization, the number of training epochs was 20,000. In the denoising score calculation procedure, T was set to 100, and noise level $\sigma$ was set to 0.25. The hyper-parameter $\lambda$ in Equation (27) was set to 0.8. The modulated signal encoder was SigRes34 in [38].

4.2.3. Evaluation Metrics

There are three commonly used evaluation metrics for OSR tasks, including the area under the receiver operating characteristic (AUROC) curve, Open-Set Classification Rate (OSCR), and True Negative Rate (TNR).

• AUROC. The receiver operating characteristic (ROC) curve is used to evaluate the prediction performance of a binary classification model. This is achieved by plotting the model’s True Positive Rate (TPR) against the False Positive Rate (FPR) at various threshold settings, resulting in a curve that describes the classification model’s performance. The AUROC refers to the area under the ROC curve, which is commonly used to quantify the overall performance of a binary classification model. Its value ranges from 0 to 1, with larger values indicating better classification performance.
• OSCR. This metric improves the AUROC by replacing the TPR value with the Correct Classification Rate (CCR) while keeping the same FPR. In this manner, the OSCR takes into accounts the classification performance of the OSR algorithm. The value of the OSCR also ranges from 0 to 1 and is commonly lower than the value of the AUROC for declines in classification accuracy.
• TNR. This metric indicates the detection rate for UUC samples. As the goal of the OSR is to detect UUC samples while remaining high in KKC recognition accuracy, the TNR is generally obtained under the condition that the TPR equals 95%.

4.3. Experimental Results

To evaluate the effectiveness of the proposed CPLDiff framework in the OSAMR task, we simulated open-set modulation recognition scenarios using two publicly available modulation signal datasets, RadioML2016.10a and RadioML2016.04c, with specific settings detailed in Table 2. Unless otherwise specified, all experimental figures and charts in this subsection are based on relatively clean data, specifically data with Signal-to-Noise Ratios (SNRs) of 12, 14, 16, and 18 dB. The OSAMR experimental results of CPLDiff were compared against those of the SoftMax baseline method [42], GCPL [16], ARPL [15], and ARPL+CS [15]. To ensure fairness in comparison, identical data processing methods and feature extractors were applied across all approaches.

4.3.1. OSAMR Performance

Table 3. OSR results on RadioML2016.10a dataset.

Class	SoftMax	GCPL	ARPL	ARPL+CS	CPLDIff
AM-DSB 1	100.0	100.0	100.0	97.87	100.0
AM-SSB	99.74	99.72	99.77	98.53	99.96
BPSK	99.91	99.97	99.99	99.93	99.99
GFSK	99.99	100.0	99.97	99.99	100.0
PAM4	99.52	99.94	99.92	99.40	99.57
QAM16	72.27	99.65	99.35	67.59	99.30
QAM64	99.48	99.92	99.88	99.72	99.56
QPSK	97.94	90.44	89.67	91.84	97.74
WBFM	99.99	100.0	100.0	96.62	99.74
AUROC 2	76.59	92.67	90.94	63.96	98.26
OSCR	71.68	81.86	83.49	59.89	90.85
TNR 3	31.41	75.12	65.04	31.62	96.95

1. The first nine rows indicate the class-wise AUROC (percentage). 2. The AUROC row is not the average of KKCs. Instead, we determine the threshold for each KKC using Equation (29) under the same $\delta$. With $\delta$ ranging from 0 to 1 by 0.0001, 10,000 pairs of (TPR, FPR) and (CCR, FPR) are obtained for the subsequent AUROC, OSCR, and TNR calculations. In this manner, the unbalance in the number of KKCs and UUCs is eliminated. 3. The TNR is obtained under the condition that $\delta =0.95$ for each KKC.

and

Table 4. OSR results on RadioML2016.04c dataset.

Class	SoftMax	GCPL	ARPL	ARPL+CS	CPLDIff
AM-DSB	100.0	100.0	100.0	100.0	100.0
AM-SSB	99.90	99.90	99.94	99.91	99.94
BPSK	99.99	99.93	99.99	99.99	99.99
GFSK	97.83	100.0	99.69	99.97	100.0
PAM4	99.89	99.62	99.97	99.94	99.86
QAM16	93.20	98.70	98.48	98.25	98.39
QAM64	98.66	98.77	98.69	98.11	97.09
QPSK	99.27	98.40	89.09	92.55	99.26
WBFM	100.0	99.99	100.0	99.99	99.97
AUROC	93.39	98.47	88.82	91.21	99.31
OSCR	92.13	96.12	87.10	89.96	97.83
TNR	72.89	94.98	75.60	76.00	98.59

See table notes in Table 3.

present the AUROC, OSCR, and TNR results for the RadioML2016.10a and RadioML2016.04c datasets. The results indicate that the proposed CPLDiff method achieves the highest values across all the quantitative metrics, demonstrating superior OSR performance compared to the other methods. Specifically, as shown in rows 0 to 8 of Table 3 for class-wise AUROC values, the SoftMax baseline method exhibits a relatively low value for Class QAM16, suggesting that the UUCs are prone to confusion with this class. Moreover, the GCPL method also shows a lower AUROC value for Class QPSK compared to other classes, and a similar phenomenon is observed in the ARPL and ARPL+CS methods. Therefore, the AUROC values across different classes reveal that the comparison methods tend to confuse the UUCs with certain KKC(s). In contrast, for the CPLDiff method, the class-wise AUROC values in Table 3 remain consistently high, with the AUROC values for Class QAM16 and Class QPSK exceeding 98. A similar phenomenon can be observed in Table 4.

Figure 3 shows the ROC and OSCR curves obtained from open-set experiments on the 10a and 04c datasets. The ROC curve was generated by gradually increasing the threshold, resulting in a series of (TPR, TNR) pairs, and the OSCR curve is generated in a similar manner. The results in the figures demonstrate that the CPLDiff method achieves the best OSAMR performance. Specifically, for the CPLDiff method, it rapidly increases the TPR and CCR while maintaining a low FPR. In contrast, for the other methods, as the TPR and CCR increase, the FPR also rises at a certain rate, failing to maintain a low FPR at higher TPR levels.

4.3.2. Recognition on KKC and Detection on UUC

The goal of OSAMR is to ensure the recognition performance on KKCs while maximizing the detection capability for UUCs. As described in Section 3.1, the open-set recognition task can be viewed as a $C+1$ classification task, where all UUCs are treated as a single class. To specifically and clearly demonstrate the performance of the proposed CPLDiff algorithm in terms of recognition on KKCs and detection on UUCs, we present the TNR data, as well as the prediction confusion matrix for the $C+1$ classification task.

Table 3 and Table 4 present the detection rates for UUCs using the proposed CPLDiff algorithm on the 10a and 04c datasets, as shown in the TNR row. It is evident from the table that the proposed algorithm achieves the highest TNR values. Specifically, for the 10a dataset, the detection rate for the unknown class is 96.95%, which is over a 20% improvement compared to the second-highest GCPL algorithm at 75.12%.

Figure 4 provides a more detailed view through the confusion matrices for the OSAMR tasks from the CPLDiff, GCPL, and SoftMax algorithms, clearly illustrating the confusions between KKC and KKC, as well as between KKC and UUC. The conditions for generating these confusion matrices are outlined in the table notes of Table 3. In the confusion matrices, the true labels for KKCs and UUCs are provided, where the predicted labels include “Unknown” in addition to the KKC labels. From the displayed matrices, it is clear that there is no significant confusion between KKC and UUC for CPLDiff. The proposed algorithm maintains a high recognition rate for KKC while also achieving a high detection rate for UUC. In contrast, GCPL struggles to distinguish between QAM16 and QAM64, two KKCs, and exhibits considerable confusion between UUCs and the QPSK modulation scheme in the RadioML2016.10a dataset. The SoftMax baseline method performs best in KKC recognition but has the poorest detection rate for UUCs.

4.3.3. OSAMR Results at Different SNRs

Due to the impact of noise on signals in real-world environments, we conducted experiments at different SNR levels. For the 10a and 04c datasets, we divided the SNR (dB) into five levels—{−20, −18, −16, −14}, {−12, −10, −8, −6}, {−4, −2, 0, 2}, {4, 6, 8, 10}, and {12, 14, 16, 18}—and conducted experiments using data at each of these noise levels. The experimental results are shown in Figure 5 and Figure 6.

From the experimental results, three conclusions can be drawn. First, under low-SNR (dB) conditions of [−20, −6], the AUROC performance of all methods is close to 50%, meaning that they cannot distinguish between KKCs and UUCs. In such low-SNR conditions, the energy of the noise far exceeds that of the signal, causing the signal’s meaningful features to be masked by the noise. This results in a high level of randomness in the algorithm’s performance, and small differences may not accurately reflect the superiority of one algorithm over another.

Second, as the SNR increases, the AUROC performance of all algorithms shows an upward trend. Among them, the proposed algorithm exhibits the largest improvement, which also indirectly suggests that noise has a significant impact on the DDM. Since the DDM aims to restore pure original data, the introduction of noise can significantly interfere with the denoising process in the DDM.

Third, at higher SNR levels, the AUROC performance of the proposed method outperforms the other algorithms. In real-world environments, the SNR (dB) typically falls within the range of [10, 30], indicating that the proposed algorithm is more suitable for application in practical scenarios.

4.3.4. Visualization

In this subsection, we present the cumulative distribution function (CDF) of the test sample scores in Figure 7, which helps to better understand the distribution of sample scores. From the cumulative distribution function of the scores, it can be observed that the scores of KKC samples are mostly concentrated in the range [0.95, 1], while the scores of UUC samples are generally lower than those of the KKC samples.

Many methods define the open-set recognition task as a binary classification problem and assign a score to each sample. In this open-set recognition paradigm, all known known classes (KKCs) share the same threshold. The approach of using a single threshold for all KKCs is effective under the condition that the score distributions of all KKCs are consistent. However, as shown in Figure 7, the score distributions are inconsistent due to variations in the data across different categories, making the use of a shared threshold severely flawed. Therefore, it is essential to determine thresholds individually for each KKC. This does not introduce significant time complexity, as the threshold for a single KKC is determined using only the training samples of that class without involving samples from other classes. In other words, each sample is used only once, which is consistent with determining a shared threshold for all KKCs.

4.4. Further Experiments

4.4.1. Effectiveness of DDM

The primary motivation for incorporating a DDM is to utilize instance-level denoising scores to further refine the close prediction results along with CPL scores. Therefore, in this context, we compare the results of performing open-set recognition using only CPL with those achieved using CPLDiff (the combination of CPL and DDM) to validate the effectiveness of introducing the DDM in

Table 5. Improvement validation by introducing DDM to CPL.

Metrics	RML2016.10a		RML2016.04c
	CPL	CPLDiff	CPL	CPLDiff
AUROC	86.17	98.26	89.08	99.31
OSCR	79.48	90.85	87.79	97.83
TNR	67.66	96.93	76.15	98.59

. It is evident that CPL already exhibits a certain level of open-set recognition capability. Upon incorporating the denoising scores generated by the DDM, the open-set recognition metrics for both datasets showed varying degrees of improvement. This indicates that the instance-level information enhanced by the DDM contributes to the effectiveness of open-set recognition.

4.4.2. Few-Shot Performance

We further conducted experiments in few-shot scenarios, as the DDM is capable of generating synthetic samples for data augmentation. For instance, Chen et al. [58] employed a conditional diffusion model to generate synthetic data to assist the model in few-shot cases and proved its effectiveness. Following their work, we evaluated the data augmentation effectiveness of the DDM in both close-set AMR and open-set AMR tasks.

Specifically, we limited the available training data to 5% samples per class and augmented the training dataset with additional 2000 synthetic samples per class generated by the conditional diffusion model. Notably, the conditional diffusion model was consistently trained with the limited 5% samples. We selected one synthetic sample for each class and visualized the data in terms of waveforms and constellation diagrams in Figure 8.

For few-shot close-set AMR tasks, we compared the classification performance of Softmax, GCPL, ARPL, ARPL+CS, CPL and CPL+, where CPL+ means the CPL trained jointly with the original and synthetic samples. The results are shown in

Table 6. Few-shot AMR results with 5% training samples.

Method	RML2016.10a			RML2016.04c
	Precision	Recall	F1 Score	Precision	Recall	F1 Score
SoftMax	81.02	81.31	81.13	93.96	94.23	93.76
GCPL	80.27	80.41	78.85	91.41	91.84	91.21
ARPL	80.95	79.33	77.95	90.44	90.17	90.77
ARPL+CS	75.48	75.59	75.30	92.62	92.64	92.42
CPL	81.98	81.69	81.24	93.36	93.92	93.32
CPL+	82.29	81.89	81.33	93.77	94.12	93.32

, where the precision, recall, and F1 score are commonly used metrics in classification tasks, and recall refers to the accuracy. From the table, we can draw the following conclusions. First, compared to 70% training samples, limited available data introduces a decline in the AMR performance for all methods. Second, SoftMax achieves the best recognition performance among the comparison methods, and CPL is competitive against SoftMax, exhibiting more robustness in few-shot scenarios. Finally, CPL+, which shows improvements from CPL, demonstrated the effectiveness of the data augmentation strategy with the DDM.

For few-shot open-set AMR, we compared the three OSR metric results in

Table 7. Few-shot OSAMR results with 5% training samples.

Method	RML2016.10a			RML2016.04c
	AUROC	OSCR	TNR	AUROC	OSCR	TNR
Softmax	48.17	38.68	11.67	80.53	76.43	44.16
GCPL	69.02	55.94	19.69	87.52	82.73	30.33
ARPL	36.81	29.19	2.000	51.96	47.49	2.534
ARPL+CS	32.58	23.96	0.986	60.90	57.14	8.380
CPLDiff	90.45	73.44	51.96	92.69	87.11	74.65
CPLDiff+	94.69	77.49	66.63	95.85	90.43	80.98

, where CPLDiff+ means training the CPL module with additional synthetic samples. It is obvious that all the methods suffered declines in terms of the three OSR metrics, which indicates that the insufficiency of training data poses great challenges for the methods to learn competitive encoders and classifiers. While the comparison methods suffer significant performance declines, CPLDiff shows more robustness in such few-shot scenarios, and CPLDiff+ further enhances performance from that of CPLDiff. The CPL adopts data-free pre-prototype optimization before training the encoder, which provides CPL with more robust prototype retrievals. Additionally, the DDM is UNet-enabled, which brings more adaptivity to few-shot scenarios.

5. Conclusions

In this paper, the CPLDiff method is proposed to achieve open-set automatic modulation recognition, requiring a detection ability over unknown classes. We analyze the shortcomings of existing OSR methods and demonstrate the main challenge that CPLDiff is to solve. In order to fully exploit and leverage the training samples of known known classes, we combine CPL and a DDM, with the former extracting class-level information and the latter extracting instance-level information.

To demonstrate the OSAMR effectiveness of CPLDiff, we conducted several comparative experiments on the public datasets RadioML2016.10a and RadioML2016.04c. Experimental results show that the CPLDiff algorithm outperforms other state-of-the-art methods with respect to the AUROC, OSCR, and TNR metrics, even when the unknown unknown classes share great similarity with some of the known known classes. The OSAMR results at different SNRs indicate that our method is suitable for real-world environments. We further evaluate the robustness of CPLDiff in resource-constrained settings. However, there exist several limitations in the CPLDiff algorithm. First, due to the co-inference and denoising loop in the DDM, CPLDiff is time-consuming compared to other methods. Second, a large scale of training samples is required to fully leverage the DDM. In future research, the OSR paradigms with a DDM need more exploration except for with the proposed denoising score, such as by developing a lightweight DDM and data augmentation strategy.