Spectrum Sensing for Noncircular Signals Using Augmented Covariance-Matrix-Aware Deep Convolutional Neural Network

Abstract

This work investigates spectrum sensing in cognitive radio networks, where multi-antenna secondary users aim to detect the spectral occupancy of noncircular signals transmitted by primary users. Specifically, we propose a deep-learning-based spectrum sensing approach using an augmented covariance-matrix-aware convolutional neural network (CNN). The core innovation of our approach lies in employing an augmented sample covariance matrix, which integrates both a standard covariance matrix and complementary covariance matrix, thereby fully exploiting the statistical properties of noncircular signals. By feeding augmented sample covariance matrices into the designed CNN architecture, the proposed approach effectively learns discriminative patterns from the underlying data structure, without stringent model constraints. Meanwhile, our approach eliminates the need for restrictive model assumptions and significantly enhances the detection performance by fully exploiting noncircular signal characteristics. Various experimental results demonstrate the significant performance improvement and generalization capability of the proposed approach compared to existing benchmark methods.

1. Introduction

The increasing demand for high-speed wireless communication has led to a growing need for spectrum resources [1]. However, the traditional static spectrum allocation policy grants exclusive access to licensed spectrum bands for primary users (PUs), leaving secondary users (SUs) unable to utilize these resources, even when they are idle. To address this inefficiency, cognitive radio (CR) technology has been introduced [2]. As a paradigm of software-defined radio, CR offers a promising solution to the spectrum scarcity problem by enabling dynamic spectrum access, thereby accommodating the escalating data rate requirements in wireless communications [3,4]. In fact, CR systems enhance spectral efficiency by opportunistically allowing SUs to access an underutilized spectrum without causing harmful interference to PUs (a concept known as spectrum reuse). To enable this, SUs must continuously perform spectrum sensing, which involves reliably detecting the presence or absence of PUs in a given frequency band. As such, spectrum sensing constitutes a fundamental function of cognitive radio systems and has become a central research topic in both academia and industry [5,6,7].

In the past few years, numerous approaches have been proposed to address the critical challenge of spectrum sensing in CR systems. Among these, energy detection (ED) [8] stands as a mature technique to determine spectrum occupancy by comparing the received signal power against a threshold derived from the prior noise power. However, its performance is highly susceptible to noise uncertainty, due to the dynamic and often unknown nature of noise power in practical environments. Subsequently, maximum eigenvalue detection (MED) [9] emerged by leveraging the eigenvalue properties of a sampled covariance matrix, which also critically relies on prior knowledge of noise variance, making it similarly vulnerable to noise uncertainty [10]. To overcome the reliance on prior noise power estimation and improve performance in non-independent and identically distributed noise environments, several advanced techniques have been developed. Notable examples include eigenvalue moment ratio (EMR) [11], Hadamard ratio detection (HDM) [12], and largest absolute value (LAV) detection [13]. While these methods demonstrate improved robustness against noise uncertainty, they frequently exhibit degraded performance in challenging scenarios such as low signal-to-noise ratio conditions and unstable noise distributions. Importantly, all of the aforementioned approaches are fundamentally model-driven, relying on predefined statistical assumptions and manually designed features. Their effectiveness is thus inherently constrained by the validity of these assumptions.

In contrast to model-driven approaches, deep learning [14,15] has emerged as a powerful paradigm for spectrum sensing, leveraging its capability to autonomously extract discriminative features from raw data, without relying on predefined statistical models. CNNs [16] have been widely adopted due to their proficiency in capturing spatial structures within signal representations. Pioneering this direction, the CM-CNN [17] utilizes a sample covariance matrix (CM) as input to learn environment-adaptive test statistics, and Gao et al. [18] directly processed raw I/Q samples through deep neural networks to exploit their inherent modulation structures. Further innovations have integrated temporal dynamics. For example, the authors in [19] processed both current and historical sensing data, to implicitly model primary user activity patterns, while CNN-LSTM (long short-term memory) [20] combines CNN spatial feature extraction with LSTM sequential modeling to exploit time-dependent signal correlations. For enhanced signal characterization, the method in [21] employs time–frequency representations derived from short-time Fourier transform to capture joint time–frequency features. Recent advances have mainly focused on cross-user interactions and advanced architectures: CNN-Transformer [22] models inter-user signal dependencies through self-attention mechanisms, and channel-attention based parallel CNN-LSTM [23] uses attention-guided spatial feature refinement coupled with parallel temporal modeling.

While the aforementioned methods have demonstrated significant progress, they often implicitly exploit the circularity property of signals. Circular signals exhibit statistical properties that are rotationally invariant in the complex plane. However, not all practical digital modulation schemes produce circular signals. Non-circular (NC) signals, characterized by an asymmetric distribution of their constellation points in the complex plane, are frequently encountered in real-world CR systems. Prominent examples include binary phase shift keying (BPSK), unequal quadrature phase shift keying (UQPSK), and pulse amplitude modulation (PAM), etc. For such signals, valuable discriminatory information resides not only in the conventional covariance matrix, but also in the complementary covariance matrix. In consideration of this, researchers have developed enhanced spectrum sensing techniques specifically designed to leverage non-circularity. For instance, the NC-HDM method [24] extends the conventional HDM test by incorporating a complementary covariance matrix, significantly improving detection for NC signals compared to its circular counterpart. Similarly, the NC-LAV (largest absolute value) algorithm [25] integrates non-circular information into the LAV framework, yielding substantial performance gains over the original LAV method [13] in NC signal environments. Recently, the authors in [26] jointly exploited both covariance matrices for spectrum sensing under uncalibrated antennas. Nevertheless, these NC-enhanced techniques remain model-driven, inheriting fundamental limitations regarding adaptability to diverse and unstable noise environments.

Motivated by the strengths of CNNs for matrix-shaped data structures and the rich statistical structure inherent in non-circular (NC) signals, we propose an augmented covariance-matrix-aware convolutional neural network (ACM-CNN) for spectrum sensing in cognitive radio systems. The key innovation of the proposed method lies in the construction of an augmented covariance matrix that explicitly integrates both standard covariance and complementary covariance matrices. Such a joint representation enables comprehensive exploitation of both the second-order correlation and the non-circular features of the received signal. Unlike previous noncircular based methods that remain constrained by analytical mathematical models, the proposed ACM-CNN method leverages a data-driven approach to automatically learn complex and discriminative patterns embedded in the augmented matrix, thereby improving robustness to unstable noise and low-SNR conditions. Extensive simulation results demonstrate that ACM-CNN significantly outperformed existing state-of-the-art model-driven methods, as well as conventional deep-learning-based techniques, especially in scenarios involving non-circular signals and uncertain noise environments.

The remaining sections of this paper are organized as follows: Section 2 introduces the system model and the problem formulation. Section 3 presents the proposed spectrum sensing method using an augmented covariance-matrix-aware deep CNN. Section 4 evaluates the performance of the proposed algorithm via extensive numerical experiments. Finally, Section 5 concludes the paper.

Notation: We use the superscripts “∗”, “$\mathsf{T}$”, and “$\mathsf{H}$” to denote the conjugate, the transpose, and the conjugate transpose, respectively. For a vector $\mathit{x}\in C^n$, we use $x_i$ to represent its i-th element. For a matrix $\mathit{X}\in C^{m\times n}$, $x_{ij}$ denotes the element at the intersection of its i-th row and j-th column. We use $\mathit{x}\sim \mathcal{CN}(\mathit{x},\mathit{\mu };\mathbf{\Sigma })$ to denote that $\mathit{x}$ follows real normal and complex circularly symmetric normal distributions with mean $\mathit{\mu }$ and covariance $\mathbf{\Sigma }$, respectively. In addition, we use $j\triangleq \sqrt{-1}$, $\mathbf{0}$, and $E(·)$ to denote the imaginary unit, the null matrix, and the expectation operation, respectively.

2. Signal Model

Let us consider a multi-antenna cognitive radio (CR) scenario, in which a secondary user (SU) with M receiving antennas aims to detect the radio spectrum hole of $M_0$ primary users (PUs) and seeks to opportunistically use this spectrum when the PUs are idle. Specifically, the SU is attempting to decide if any of the PUs are transmitting based on N available observations. Let $\mathit{x}\left(n\right)={[x_1\left(n\right),x_2\left(n\right),\dots ,x_M\left(n\right)]}^{\mathsf{T}}$ denote the receiving discrete-time sample at time n of SU in the CR system. We assume that each PU in the CR system has one antenna, which transmits a signal or does not transmit a signal randomly over a certain spectrum with incumbent licensees. Therefore, there are just two states of the PU (active or inactive). As such, the spectrum sensing problem for a multi-antenna SU can be formulated as a binary hypothesis testing problem:

$$\begin{array}{cc}\hfill & H_1:\mathit{x}\left(n\right)=\mathit{H}\mathit{s}\left(n\right)+\mathit{u}\left(n\right),\hfill \\ \hfill & H_0:\mathit{x}\left(n\right)=\mathit{u}\left(n\right),\hfill \end{array}$$

(1)

where $H_1$ stands for the signal-presence hypothesis; $H_0$ denotes the signal-absence hypothesis; $\mathit{H}\in {\mathbb{C}}^{M\times M_0}$ denotes the channel matrix between the PU and the SU, which is usually unknown but deterministic during the sensing period (coherence time); $\mathit{s}\left(n\right)={[s_1\left(n\right),s_2\left(n\right),...,s_{M_0}\left(n\right)]}^{\mathsf{T}}$ denotes the signal vector emitted by PU at time n; $\mathit{u}\left(n\right)={[u_1\left(n\right),u_2\left(n\right),...,u_M\left(n\right)]}^{\mathsf{T}}$ denotes the additive noise vector at time n.

We assume that $s_i\left(n\right)(i=1,...,M_0)$ is exactly a non-circular signal, with its variance being ${\sigma }_{s_i}^2=E\left[\left|{s_i}^2\right|\right]$ and $E\left[{s_i}^2\right]={\kappa }_i{e}^{j{\varphi }_i}{\sigma }_{s_i}$ with ${\varphi }_i\in [-\pi ,\pi )$ and ${\kappa }_i\in [0,1]$ being the noncircularity phase and the noncircularity rate, respectively. Moreover, we assume that the noise ${\left\{u_i\left(n\right)\right\}}_{i=1}^M$ is statistically independent of each other, independent of the signals, and follows a circular Gaussian distribution, i.e., $E\left[{u_i}^2\left(n\right)\right]=0$ and $u_i\left(n\right)\sim \mathcal{CN}(0,{\sigma }_{u_i}^2)(i=1,\dots ,M)$, where ${\sigma }_{u_i}^2=E\left[\left|{u_i}^2\left(n\right)\right|\right]$ is the noise variance. Note that ${\sigma }_{u_i}^2$ is not necessarily equal to ${\sigma }_{u_j}^2$ for $i\ne j$ in practice, which corresponds to the case of an uncalibrated multiple antenna receiver [26]. As such, the augmented covariance matrix can be expressed as

$$\begin{array}{cc}\hfill & \underline{\mathit{R}}=E\left[\underline{\mathit{x}}\left(n\right){\underline{\mathit{x}}}^{\mathsf{T}}\left(n\right)\right]=\left[\begin{array}{cc}\mathit{R}& \tilde{\mathit{R}}\\ {\tilde{\mathit{R}}}^{\ast }& {\mathit{R}}^{\ast }\end{array}\right]\hfill \end{array}$$

(2)

where $\underline{\mathit{x}}\left(n\right)={[{\mathit{x}}^{\mathsf{T}}\left(n\right),{\mathit{x}}^{\mathsf{H}}\left(n\right)]}^{\mathsf{T}}$ denotes the augmented signal vector consisting of $\mathit{x}\left(n\right)$ and ${\mathit{x}}^{\ast }\left(n\right)$ and

$$\begin{array}{c}\hfill \mathit{R}=E\left[\mathit{x}\left(n\right){\mathit{x}}^{\mathsf{H}}\left(n\right)\right],\tilde{\mathit{R}}=E\left[\mathit{x}\left(n\right){\mathit{x}}^{\mathsf{T}}\left(n\right)\right].\end{array}$$

(3)

Herein, $\mathit{R}$ and $\tilde{\mathit{R}}$ stand for the conventional covariance matrix and the complementary covariance matrix, respectively. Note that if the signal vector of PU $\mathit{s}\left(n\right)$ is noncircular, then we have $\tilde{\mathit{R}}\ne \mathbf{0}$, otherwise $\tilde{\mathit{R}}=\mathbf{0}$. The commonly-used noncircular signals with $\tilde{\mathit{R}}\ne \mathbf{0}$ can be the BPSK, offset QPSK, UQPSK, and minimum shift keying (GMSK) signal, etc.

Now, the spectrum sensing problem at hand is to decide whether the PU signals ${\left\{s_i\left(n\right)\right\}}_{i=1}^{M_0}$ exist or not from the available N noisy observations ${\left\{\mathit{x}\left(n\right)\right\}}_{n=1}^N$. To fully exploit the full statistical property of noncircular signals and improve the detection performance, it is crucial to construct a test statistic using the augmented covariance matrix $\underline{\mathit{R}}$. Typical model-driven methods include NC-HDM [24] and NC-LAV [25], which, however, cannot be applied to the case with uncertain or unstable noise. To address this, in the following, we use an augmented covariance-matrix-aware deep CNN, i.e., a data-driven approach, to learn a robust and generalized test statistic for enhancing the spectrum sensing performance of noncircular signals.

3. Spectrum Sensing Using Augmented Covariance-Matrix-Aware Deep CNN

In this section, we propose a deep-learning-based spectrum sensing framework based on augmented covariance-matrix-aware convolutional neural networks, which is illustrated in Figure 1. The proposed framework mainly consists of three stages: data preprocessing, offline training, and online detection, as will be elaborated as follows:

3.1. Data Preprocessing

In each sensing period, the SU collects N observations ${\left\{\mathit{x}\left(n\right)\right\}}_{n=1}^N$ to perform a spectrum sensing. Here, we adopt the full second-order statistical sample covariance matrix of ${\left\{\mathit{x}\left(n\right)\right\}}_{n=1}^N$, i.e., the usual sample covariance matrix denoted as $\widehat{\mathit{R}}$, while the complementary sample covariance matrix is denoted as $\widehat{\tilde{\mathit{R}}}$. Both $\widehat{\mathit{R}}$ and $\widehat{\tilde{\mathit{R}}}$ are designed as the input of the CNN. The sample covariance matrix $\widehat{\mathit{R}}$ and complementary sample covariance matrix $\widehat{\tilde{\mathit{R}}}$ from N observations are, respectively, calculated as

$$\begin{array}{cc}\hfill & \widehat{\mathit{R}}={\displaystyle \frac{1}{N}}\sum _{n=0}^{N-1}\mathit{x}\left(n\right){\mathit{x}}^{\mathsf{H}}\left(n\right),\hfill \\ \hfill & \widehat{\tilde{\mathit{R}}}={\displaystyle \frac{1}{N}}\sum _{n=0}^{N-1}\mathit{x}\left(n\right){\mathit{x}}^{\mathsf{T}}\left(n\right).\hfill \end{array}$$

(4)

Then, we obtain an augmented sample covariance matrix

$$\begin{array}{cc}\hfill & \underline{\widehat{\mathit{R}}}=\left[\begin{array}{cc}\widehat{\mathit{R}}& \widehat{\tilde{\mathit{R}}}\\ {\widehat{\tilde{\mathit{R}}}}^{\ast }& {\widehat{\mathit{R}}}^{\ast }\end{array}\right].\hfill \end{array}$$

(5)

The reason for using an augmented sample covariance matrix is that it contains not only the energy information but also the correlation information between antennas and between the usual covariance matrix and complementary covariance matrix. To facilitate subsequent processing by CNNs, which typically operate on real-valued inputs, we decompose the complex-valued augmented covariance matrix $\underline{\widehat{\mathit{R}}}$ into its real and imaginary components. Specifically, we construct the input of CNNs as

$$\begin{array}{c}\hfill \underline{\mathcal{R}}=\left[\Re \left\{\underline{\widehat{\mathit{R}}}\right\}\Vert \Im \left\{\underline{\widehat{\mathit{R}}}\right\}\right]=\left[\begin{array}{cc}\mathcal{R}& \tilde{\mathcal{R}}\\ {\tilde{\mathcal{R}}}^{\ast }& {\mathcal{R}}^{\ast }\end{array}\right]\in {\mathbb{R}}^{2M\times 2M\times 2},\end{array}$$

(6)

where $\Re \{·\}$ and $\Im \{·\}$ denote the real and imaginary components, respectively; and $[·\Vert ·]$ denotes the stacking operation of a matrix to form a three-dimensional tensor. In a word, the two real-valued matrices $\Re \left\{\underline{\widehat{\mathit{R}}}\right\}$ and $\Im \left\{\underline{\widehat{\mathit{R}}}\right\}$ are concatenated along a new dimension to form a three-dimensional augmented covariance tensor $\underline{\mathcal{R}}$. Similarly, $\mathcal{R}$, $\tilde{\mathcal{R}}$, ${\tilde{\mathcal{R}}}^{\ast }$, and ${\mathcal{R}}^{\ast }$ represent the tensor formed by stacking the real and imaginary parts of $\widehat{\mathit{R}}$, $\widehat{\tilde{\mathit{R}}}$, ${\widehat{\tilde{\mathit{R}}}}^{\ast }$, and ${\widehat{\mathit{R}}}^{\ast }$, respectively.

Meanwhile, the binary hypothesis labels are encoded as one-hot vectors to align with the output layer of the CNN classifier:

$$\begin{array}{cc}\hfill H_1& \to z={[1,0]}^{\mathsf{T}}\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill H_0& \to z={[0,1]}^{\mathsf{T}}.\hfill \end{array}$$

(8)

This representation enables the CNN network to output class probabilities directly, where the first element in z corresponds to the probability of $H_1$ and the second to the probability of $H_0$.

The unitization of the augmented sample covariance matrix in Equation (5) has significant advantages in spectrum sensing for noncircular PU signals. Specifically, it can contribute to learning more discriminative patterns and a generalized test statistic. To illustrate this, we consider a noncircular signal case with $N=\mathrm{100,000}$ and $M=8$, where the matrix-shaped heatmaps of the real and imaginary parts of $\underline{\widehat{\mathit{R}}}$ under $H_1$ and $H_0$ are shown in Figure 2 for comparison. Herein, Figure 2a,b represent the real parts of the augmented covariance matrix under the presence and absence of the primary user signal, respectively, while Figure 2c,d show the corresponding imaginary parts. The two block matrices along the main diagonal correspond to the conventional covariance matrices, whereas the off-diagonal blocks represent the complementary covariance matrices (also known as pseudo-covariance). It is seen that the pseudo-covariance matrices (i.e., the off-diagonal blocks) exhibit a clear distinction between the cases with and without the presence of the primary user signal. This indicates that the pseudo-covariance of non-circular signals provides additional discriminative information that can be effectively used to distinguish between signal presence and absence, thereby enabling the learning of a more accurate data-driven test statistic.

3.2. Offline Training for Spectrum Sensing of Noncircular Signals

In the offline training stage, the labeled K training samples for spectrum sensing are collected to construct the training set:

$$\begin{array}{c}\hfill (Y,Z)=\{(y^{\left(1\right)},z^{\left(1\right)}),(y^{\left(2\right)},z^{\left(2\right)}),\dots ,(y^{\left(K\right)},z^{\left(K\right)})\}\end{array}$$

(9)

where $(y^{\left(k\right)},z^{\left(k\right)})$ denotes the $k-\mathrm{th}(k=1,2,\dots ,K)$ training sample of the training set $(Y,Z)$. For a single example $(y,z)$, y represents the input data for the neural network, which is the augmented covariance matrix calculated by Equation (5). We note that $z\in \{{[0,1]}^{\mathsf{T}},{[1,0]}^{\mathsf{T}}\}$ represents the corresponding label of $H_0$ and $H_1$. The output of CNN is formulated as a normalized class score vector:

$$\begin{array}{c}\hfill h_{\mathit{\theta }}\left(y^{\left(k\right)}\right)=\left[\begin{array}{c}{h}_{\mathit{\theta }|H_1}\left(y^{\left(k\right)}\right)\\ h_{\mathit{\theta }|H_0}\left(y^{\left(k\right)}\right)\end{array}\right],\end{array}$$

(10)

where $h_{\mathit{\theta }|H_1}\left(y^{\left(k\right)}\right)+h_{\mathit{\theta }|H_0}\left(y^{\left(k\right)}\right)=1$. We note that $h_{\mathit{\theta }}(·)$ is a mapping from $y^{\left(k\right)}$ to $z^{\left(k\right)}$ with CNN parameters $\mathit{\theta }$. In this case, $h_{\mathit{\theta }|H_i}\left(y^{\left(k\right)}\right)$ represents the class score under $H_i$. Because the output of CNN behaves like a probability distribution, we can define the loss function $\mathcal{L}\left(\mathit{\theta }\right)$ using cross-entropy as

$$\begin{array}{cc}\hfill \mathcal{L}\left(\mathit{\theta }\right)& \triangleq -{\displaystyle \frac{1}{K}}\sum _{k=1}^K\left(z_1^{\left(k\right)}log{h}_{\mathit{\theta }|H_1}\left(y^{\left(k\right)}\right)+(1-z_1^{\left(k\right)})log(1-h_{\mathit{\theta }|H_1}\left(y^{\left(k\right)}\right))\right)\hfill \end{array}$$

(11)

where $z_1^{\left(k\right)}$ denotes the first element of $z^{\left(k\right)}$. A smaller loss function value $\mathcal{L}\left(\mathit{\theta }\right)$ indicates that the output of CNN, i.e., $h_{\mathit{\theta }}\left(y^{\left(k\right)}\right)$, and the label $z^{\left(k\right)}$ tend to be closer. Our objective is to derive the optimal parameter ${\mathit{\theta }}^{\ast }$ that minimizes the loss function $\mathcal{L}\left(\mathit{\theta }\right)$, that is

$$\begin{array}{c}\hfill {\mathit{\theta }}^{\ast }=\underset{\mathit{\theta }}{argmin}\mathcal{L}\left(\mathit{\theta }\right).\end{array}$$

(12)

However, obtaining a closed-form analytical solution for $\mathcal{L}\left(\mathit{\theta }\right)$ is intractable. We therefore employ gradient descent methods to compute a suboptimal solution. Based on Bayesian criterion, we have

$$\begin{array}{cc}\hfill & P\left(y\right|H_1)={\displaystyle \frac{P\left(H_1\right|y)·P\left(y\right)}{P\left(H_1\right)}}={\displaystyle \frac{h_{{\mathit{\theta }}^{\ast }|H_1}\left(y\right)·P\left(y\right)}{P\left(H_1\right)}}\hfill \\ \hfill & P\left(y\right|H_0)={\displaystyle \frac{P\left(H_0\right|y)·P\left(y\right)}{P\left(H_0\right)}}={\displaystyle \frac{h_{{\mathit{\theta }}^{\ast }|H_0}\left(y\right)·P\left(y\right)}{P\left(H_0\right)}},\hfill \end{array}$$

(13)

where $P\left(y\right|H_i)$ denotes the conditional probability given $H_i$, $P\left(y\right)$ is the marginal probability, and $P\left(H_i\right)$ indicates the priori probability of $H_i$ in the training process.

According to the Neyman–Pearson theorem, we can obtain a CNN-based likelihood ratio test:

$$\begin{array}{c}\hfill T_{\mathrm{CNN}}\left(y\right)={\displaystyle \frac{h_{{\mathit{\theta }}^{\ast }|H_1}\left(y\right)}{h_{{\mathit{\theta }}^{\ast }|H_0}\left(y\right)}}·{\displaystyle \frac{P\left(H_0\right)}{P\left(H_1\right)}}={\displaystyle \frac{h_{{\mathit{\theta }}^{\ast }|H_1}\left(y\right)}{h_{{\mathit{\theta }}^{\ast }|H_0}\left(y\right)}}\gtrless \gamma ,\end{array}$$

(14)

where the test statistic $T_{\mathrm{CNN}}\left(y\right)$ denotes the CNN-based likelihood ratio, and the threshold $\gamma$ can be derived from the false alarm constraint. For the convenience of analysis, we set $P\left(H_1\right)=P\left(H_0\right)=0.5$ in the training process.

3.3. Covariance-Matrix-Aware CNN Structure

Note that CNNs exhibit strong capabilities for extracting discriminative features from tensor-shaped data, such as images. Motivated by this property, we leverage CNNs to extract informative features from a tensor-shaped representation $\underline{\mathcal{R}}$ for spectrum sensing of noncircular signals.

We design a CNN-based architecture composed of two convolutional layers, followed by two fully connected layers, as illustrated in Figure 3. Specifically, ${\mathrm{C}}_i(1\le i\le 2)$ and $F_i(1\le i\le 2)$ denote the i-th convolutional and fully connected layers, respectively. Each convolutional layer is followed by a rectified linear unit (ReLU) activation function to introduce non-linearity and enhance the representational capacity of the network. The first convolutional layer ${\mathrm{C}}_1$ is responsible for capturing local spatial features and correlations in $\underline{\mathcal{R}}$, while the second layer ${\mathrm{C}}_2$ further integrates and abstracts higher-level semantic features from the previous representations. The output of the convolutional layers is then reshaped into a one-dimensional feature vector via a flattening operation, which is subsequently passed to the first fully connected layer ${\mathrm{F}}_1$, with ReLU activation for further high-level reasoning. The final fully connected layer ${\mathrm{F}}_2$ generates a two-dimensional output vector, which is passed through a softmax function to obtain the posterior probability of each class.

It is worth noting that, unlike typical CNN designs, we do not adopt pooling layers in the proposed CNN architecture. Although pooling operations such as max-pooling or average-pooling can reduce the spatial resolution and computational cost, they inevitably lead to the loss of fine-grained feature details. In spectrum sensing, the presence or absence of weak signals may rely on subtle variations in local structures, and retaining full spatial information is essential to ensure reliable detection performance. Therefore, our structure maintains the original resolution in the convolutional feature maps to preserve as much discriminative information as possible.

3.4. Online Detection for Spectrum Sensing of Noncircular Signals

For online detection, the multi-antenna SU collects an unlabeled sample $\tilde{y}$ and then sends $\tilde{y}$ to the well-trained CNN model $h_{{\mathit{\theta }}^{\ast }}(·)$ of (14) to obtain the following test statistic:

$$\begin{array}{c}\hfill T_{\mathrm{CNN}}\left(\tilde{y}\right)={\displaystyle \frac{h_{{\mathit{\theta }}^{\ast }|H_1}\left(\tilde{y}\right)}{h_{{\mathit{\theta }}^{\ast }|H_0}\left(\tilde{y}\right)}}\underset{H_0}{\overset{H_1}{\gtrless }}\gamma .\end{array}$$

(15)

Once we have obtained the test statistic $T_{\mathrm{CNN}}\left(\tilde{y}\right)$, we can quickly make a decision by comparing it with a preset threshold $\gamma$. The selection of the threshold $\gamma$ is determined using the desired probability of false alarm $P_f$. The procedure for how to obtain the threshold $\gamma$ is detailed in Algorithm 1. In addition, for clarification, we summarize the above overall spectrum sensing procedure for noncircular signals in Algorithm 2, which is referred to as the augmented covariance matrix CNN (ACM-CNN) algorithm, where $I_{\max }$ is the maximum number of iterations.

Algorithm 1 Selection of threshold $\gamma$ using desired probability of false alarm $P_f$

1: Construct $Y_0=\left\{y^{\left(k\right)}\right|z^{\left(k\right)}=0,1\le k\le K\}$ from the training dataset $(Y,Z)=\{(y^{\left(1\right)},z^{\left(1\right)}),(y^{\left(2\right)},z^{\left(2\right)}),\dots ,(y^{\left(K\right)},z^{\left(K\right)})\}$, where $z^{\left(k\right)}=0$ indicates that the k-th training sample is generated under $H_0$, and $K_0$ denotes the number of samples of $Y_0$ that satisfy $z^{\left(k\right)}=0,\forall k\in \{1,\dots ,K\}$, in Y; 2: for$$ to $K_0$ do 3:  Compute the test statistic $T_{\mathrm{CNN}}\left(y_0^{\left(k_0\right)}\right)={\displaystyle \frac{h_{{\mathit{\theta }}^{\ast }\mid H_1}\left(y_0^{\left(k_0\right)}\right)}{h_{{\mathit{\theta }}^{\ast }\mid H_0}\left(y_0^{\left(k_0\right)}\right)}}$ where $y_0^{\left(k_0\right)}\in Y_0$; 4: end for 5: Sort the $K_0$ test statistics $\left\{T_{\mathrm{CNN}}\left(y_0^{\left(k_0\right)}\right)\right\}$ in ascending order as $T_{\mathrm{CNN}}^{\left(1\right)}\le T_{\mathrm{CNN}}^{\left(2\right)}\le \dots \le T_{\mathrm{CNN}}^{\left(K_0\right)}$; 6: Determine the index corresponding to $P_f$ as per $k^{\ast }=⌊(1-P_f)K_0⌋$ where $⌊·⌋$ denotes rounding to the nearest integer; 7: Set the decision threshold: $\gamma =T_{\mathrm{CNN}}^{\left(k^{\ast }\right)}$.

Algorithm 2 ACM-CNN

% Data Preprocessing 1: Acquire discrete-time observation signals ${\left\{\mathit{x}\left(n\right)\right\}}_{n=1}^N$ by sampling the received signals from M antennas; 2: Compute the augmented covariance tensor $\underline{\mathcal{R}}$ based on ${\left\{\mathit{x}\left(n\right)\right\}}_{n=1}^N$ using Equations (4)–(6); % Offline Training Phase 1: Construct the labeled training dataset $(Y,Z)=\{(y^{\left(1\right)},z^{\left(1\right)}),(y^{\left(2\right)},z^{\left(2\right)}),\dots ,(y^{\left(K\right)},z^{\left(K\right)})\}$; 2: Initialize the CNN model parameters ${\mathit{\theta }}^{\left(0\right)}$ and set iteration counter $i=0$; 3: for$$ to $I_{max}$ do 4:    Compute the loss function $\mathcal{L}\left({\mathit{\theta }}^{\left(i\right)}\right)$ using Equation (11); 5:    Update the model parameters ${\mathit{\theta }}^{\left(i\right)}$ using the Adam optimizer [27]; 6: end for 7: Obtain the final model parameters ${\mathit{\theta }}^{\ast }\leftarrow {\mathit{\theta }}^{\left(i\right)}$; % Online Detection Phase 1: Set the decision threshold $\gamma >0$ according to Algorithm 1; 2: Acquire unlabeled test data $\tilde{y}$; 3: Input $\tilde{y}$ into the trained CNN model to obtain the predicted result $\tilde{z}$; 4: Make spectrum occupancy decision by comparing each $\tilde{z}$ with threshold $\gamma$: decide $H_1$ (occupied) if $\tilde{z}>\gamma$, otherwise decide $H_0$ (idle), as described in Equation (15).

4. Simulation Results

This section presents numerical experiments to corroborate the performance of the proposed ACM-CNN algorithm. We consider a multi-antenna CR system, where a PU randomly chooses to transmit a non-circular signal or not using $M_0=4$ antennas, and a SU equipped with $M=8$ antennas receives the PU signals and attempts to determine whether the PU is active or inactive. The transmitted signals are noncircularity and complex Gaussian distributed, i.e., $s_k\left(n\right)\sim \mathcal{CN}(0,{\sigma }_s^2)$ ($k=1,\dots ,M_0$), and the noncircularity rate is given as $\kappa =\left|{\displaystyle \frac{\mathbb{E}\left[s_k{\left(n\right)}^2\right]}{\mathbb{E}\left[\right|s_k\left(n\right){|}^2]}}\right|$. Except for the last experiment to explore the detection performance of different methods versus noncircularity rate $\kappa$, the rest of the numerical experiments are carried out under the condition of $\kappa =0.99$. The noises for the different receive antennas ${\left\{u_m\left(n\right)\right\}}_{m=1}^M$ are statistically independent and also independent of the PU signals. Specifically, the noises are generalized according to

$$\left[\begin{array}{c}{u}_1\left(n\right)\\ u_2\left(n\right)\\ \dots \\ u_M\left(n\right)\end{array}\right]=\left[\begin{array}{cccc}{\sigma }_{a_1}& & & \\ & {\sigma }_{a_2}& & \\ & & \ddots & \\ & & & {\sigma }_{a_K}\end{array}\right]\left[\begin{array}{c}{u}_1^0\left(n\right)\\ u_2^0\left(n\right)\\ \dots \\ u_M^0\left(n\right)\end{array}\right]\left[\begin{array}{cccc}{\sigma }_{t_1}& & & \\ & {\sigma }_{t_2}& & \\ & & \ddots & \\ & & & {\sigma }_{t_N}\end{array}\right],$$

(16)

where $u_m^0\left(n\right)$ is a standard circularly symmetric complex Gaussian noise, i.e., $u_m^0\left(n\right)\sim \mathcal{CN}(0,1)$, and ${\sigma }_{a_m}^2$ and ${\sigma }_{t_n}^2$ indicate the noise variances across antennas and time, respectively. Furthermore, the logarithmic values of ${\sigma }_{a_m}^2$ and ${\sigma }_{t_n}^2$ are assumed to follow uniform distributions to construct the noise uncertainty. Specifically, they are defined as

$$\begin{array}{c}\hfill 10{log}_{10}{\sigma }_{a_m}^2\sim \mathcal{U}(-{\epsilon }_a,{\epsilon }_a),\\ \hfill 10{log}_{10}{\sigma }_{t_n}^2\sim \mathcal{U}(-{\epsilon }_t,{\epsilon }_t).\end{array}$$

(17)

where $\mathcal{U}(a,b)$ denote the uniform distributions from a to b. Meanwhile, ${\epsilon }_a$ and ${\epsilon }_a$ represent the uncertainty factors of the noise variance for the antennas and time, respectively, which are used to characterize the degree of fluctuation of the noise power in decibels (dB). The signal-to-noise ratio (SNR) is defined as

$$\begin{array}{c}\hfill \mathrm{SNR}\triangleq 10{log}_{10}{\displaystyle \frac{MN{\sigma }_s^2}{{\sum }_{m=1}^M{\sigma }_{a_m}^2{\sum }_{n=1}^N{\sigma }_{t_n}^2}}\end{array}$$

(18)

where N is the number of samples. The channel matrix $\mathit{H}$ between the PU and SU is modeled as a Rician fading channel, that is

$$\begin{array}{c}\hfill \mathit{H}\triangleq \sqrt{{\displaystyle \frac{\alpha }{\alpha +1}}}{\mathit{H}}_{\mathrm{LOS}}+\sqrt{{\displaystyle \frac{1}{\alpha +1}}}{\mathit{H}}_{\mathrm{NLOS}},\end{array}$$

(19)

where $\alpha$ denotes the Rician factor, representing the power ratio between the deterministic line-of-sight (LOS) component ${\mathit{H}}_{\mathrm{LOS}}$ and the scattered non-line-of-sight (NLOS) component ${\mathit{H}}_{\mathrm{NLOS}}$. Each element in ${\mathit{H}}_{\mathrm{LOS}}$ or ${\mathit{H}}_{\mathrm{NLOS}}$ is assumed to follow a standard complex Gaussian distribution, i.e., $\mathcal{CN}(0,1)$. The LOS component is considered quasi-static, while the NLOS component varies randomly over time to reflect realistic multipath fading. In the simulations, $\alpha$ is set to 100.

For comparison, we selected several typical spectrum sensing algorithms as benchmarks, which included the covariance-matrix-aware convolutional neural network (CM-CNN) method [17], noncircular local average variance (NC-LAV) [25], local average variance (LAV) [13], noncircular-based Hadamard (NC-HDM) ratio test [24], Hadamard (HDM) ratio test [12], eigenvalue moment ratio (EMR) [11], and energy detection (ED) [8]. All simulation results were obtained by averaging 60,000 Monte Carlo realizations. There were additional 35,000 training samples for the data-driven methods, including the proposed ACM-CNN and the CM-CNN [17]. The hyperparameters of the augmented covariance matrix based CNN in our simulations are given in

Table 1. Hyperparameters of the augmented covariance-matrix-based CNN.

Input: Augmented covariance matrix array $(16\times 16\times 2)$
Layers	Size
${\mathrm{C}}_1+$ ReLu	$20@(3\times 3)$
${\mathrm{C}}_2+$ ReLu	$50@(3\times 3)$
${\mathrm{F}}_1+$ ReLu	$500\times 3200$
${\mathrm{F}}_2+$ Softmax	$2\times 500$
Output: Score Vector $(2\times 1)$

. The simulations were conducted on a personal computer equipped with an Intel Core i7-13800H CPU (Intel Corporation, Santa Clara, CA, USA) and an NVIDIA RTX 2000 Ada Generation Laptop GPU (NVIDIA Corporation, Santa Clara, CA, USA). It should be noted that the online detection time per sample was approximately 0.0398 seconds by averaging 1000 trials, thereby supporting applicability to real-time detection in practical scenarios.

4.1. ROC Detection Performance Comparison

First, we evaluate the detection performance of the proposed method by plotting receiver operating characteristic (ROC) curves, which illustrate the probability of detection versus the probability of false alarm. Three sets of experiments were conducted: (i) i.i.d. noise; (ii) varying the noise variance in antennas (${\epsilon }_a\ne 0$) while the keeping noise variance fixed in time (${\epsilon }_t=0$); and (iii) varying the noise variance in time (${\epsilon }_t\ne 0$) while keeping the noise variance in antennas fixed (${\epsilon }_a=0$). For each case, we evaluated the detection performance under different numbers of observed samples N. In this case, the SNR was fixed at $-18\mathrm{dB}$.

Figure 4 and Figure 5 show the ROC curves of the different methods under i.i.d. noise with a low sample size ($N=40$) and a high sample size ($N=360$), respectively. It can be observed that our method consistently outperformed all other methods, regardless of the sample size. Specifically, in Figure 4, our ACM-CNN method achieved a detection probability close to 50% under the probability of false alarm $P_f=0.1$, which was more than 15 percentage points higher than both CM-CNN and ED. The remaining methods almost failed under such a low-sample environment. In Figure 5, all methods were able to detect under the high-sample environment; however, our method still achieved the best detection performance. It is worth noting that the ED method benefits from prior knowledge of the averaging noise power. Except for ED, all other methods operate without any prior information, i.e., totally blind detection methods. Among these totally-blind detection methods, it is seen that the data-driven approaches outperformed the model-driven ones. Furthermore, we see that the methods that exploit noncircularity demonstrated superior performance when compared to their counterparts that do not use the noncircularity (e.g., NC-HDM outperformed HDM). In order to verify the effectiveness and convergence of the proposed model during training, we also plot the training loss versus the number of training iterations of the ACM-CNN in Figure 6, where the parameter settings are kept the same as those of Figure 5. It is seen that the loss value gradually decreased as the number of training iterations was increased, and eventually converged to a value close to zero. This demonstrates that the trained model was able to effectively learn discriminative features from the data and converged to a stable training state.

To further understand how the proposed ACM-CNN model extracts discriminative features for spectrum sensing, we visualized the intermediate feature maps of the convolutional layers under hypothesis $H_1$. Specifically, the augmented covariance tensor $\underline{\mathcal{R}}$ with sample size $N=160$ was input into the trained model, and a forward pass was performed. The output feature maps of C1 and C2 are normalized and visualized in Figure 7 and Figure 8, respectively. In the figures, “Ch q” denotes the q-th channel of the output feature map.

From the visualizations, it can be observed that the network effectively captured both the circular and noncircular components embedded in $\underline{\mathcal{R}}$. Specifically, the two block matrices along the main diagonal encode the conventional covariance structure, which reflects circular signal characteristics (e.g., Ch 9 in Figure 7 and Ch 21 in Figure 8), while the off-diagonal blocks represent the complementary covariance, which arises exclusively in noncircular signals (e.g., Ch 10 in Figure 7 and Ch 20 in Figure 8). These results demonstrate that the proposed ACM-CNN is capable of jointly exploiting both circular and noncircular statistical features, thereby enabling the learning of more discriminative and robust data-driven test statistics for spectrum sensing.

Figure 9 and Figure 10 illustrate the ROC curves under the condition of ${\epsilon }_a\ne 0$ for $N=40$ and $N=360$, respectively. It can be observed that several methods failed in the presence of unstable noise across the antennas (e.g., ED and EMR), while only the data-driven methods (ACM-CNN and CM-CNN) and those based on the Hadamard ratio test (HDM and NC-HDM) remained functional in such scenarios. However, Hadamard-based methods only achieved satisfactory performance under large sample size conditions. In contrast, the data-driven methods not only performed better overall, but also maintained effectiveness even in the low-sample environment.

By comparing Figure 4 and Figure 9, as well as Figure 5 and Figure 10, we can see that ACM-CNN, CM-CNN, HDM, and NC-HDM exhibited some performance degradation under unstable noise across antennas. Nevertheless, the decline was within an acceptable range, indicating that these four methods exhibited a certain degree of robustness to unstable noise. Our method still achieved the best detection performance under all tested conditions.

Figure 11 and Figure 12 illustrate the ROC curves under the condition of ${\epsilon }_t\ne 0$ for $N=40$ and $N=360$, respectively. By comparing Figure 4 and Figure 11, as well as Figure 5 and Figure 12, we can observe that all methods experienced some performance degradation under temporally unstable noise, indicating that they all exhibited a certain degree of robustness to such noise. Compared to the unstable noise across antennas, the adverse impact of temporally unstable noise was relatively smaller. In Figure 12, ACM-CNN achieved a detection probability exceeding 96% at a false alarm rate $P_f=0.1$, significantly outperforming all other methods.

4.2. Probability of Detection Versus the Number of Observations and SNR

Figure 13 shows how the detection probability of the respective algorithms changed with the number of samples N when the probability of false alarm was fixed at 0.1. It can be seen that under an i.i.d. noise environment, the detection performance of all algorithms improved as the number of samples N increased. Our ACM-CNN algorithm was the best among all algorithms for all sample numbers N.

Figure 14 shows a curve of the detection probability changing with SNR under an i.i.d. noise environment with fixed $P_f=0.1$. It can be seen that, under all SNR conditions, our proposed ACM-CNN algorithm was optimal.

Figure 13 illustrates the probability of detection versus number of samples N for the various detection algorithms under an i.i.d. noise environment, with the probability of false alarm fixed at 0.1. The results demonstrate that as N increased, all methods achieved a progressively better detection performance, which is expected, since more samples provide richer statistical information for signal detection. Among all the evaluated algorithms, our proposed ACM-CNN significantly outperformed the other methods across the entire range of sample numbers. In particular, ACM-CNN exhibited a steeper performance gain with increasing N, reaching a detection probability of more than 90% at a number of samples about 175. Model-driven approaches like LAV and HDM performed substantially worse, showing limited improvement with an increasing number of samples, which highlights their insufficiency for leveraging high-dimensional data patterns compared to deep-learning-based methods.

Figure 14 shows the probability of detection versus SNR under an i.i.d. noise assumption and fixed false alarm rate ($P_f=0.1$). The results clearly indicate that ACM-CNN consistently achieved the highest detection probability at every SNR level. Especially in the low-SNR regime (e.g., SNR < −20 dB), ACM-CNN exhibited a remarkable performance, whereas the other methods failed to maintain effective performance. As the SNR improved, all algorithms began to benefit, but the performance gap between ACM-CNN and the rest remained evident, underscoring its superior capacity for feature extraction and classification.

4.3. Generalizability

Since the proposed ACM-CNN method is data-driven, the deployed neural network model was trained offline based on the data distribution in the training set. However, during online detection, the noise model, SNR, and other environmental factors may differ from those present in the training phase. Therefore, it is meaningful to investigate the generalizability of the proposed method under conditions mismatched between training and testing. Specifically, we evaluate the generalization performance of ACM-CNN by presenting its ROC curves under three scenarios: mismatched SNRs, different degrees of antenna noise uncertainty, and different degrees of time noise uncertainty between training and testing, respectively.

Figure 15 illustrates the ROC curves of the proposed ACM-CNN method under SNR conditions mismatched between training and detection. Specifically, “ACM-CNN ($p\mathrm{dB}/q\mathrm{dB}$)” denotes a model trained on a dataset with SNR = p dB and evaluated on a test dataset with SNR = q dB. As observed in the figure, the detection performance remained nearly identical across models trained at different SNR levels when evaluated on the same test dataset. This indicates that our method successfully extracted signal-relevant information from the sample covariance matrix and was largely unaffected by variations in SNR. Consequently, ACM-CNN demonstrated strong generalizability and robustness to SNR mismatch between training and detection.

Figure 16 illustrates the ROC curves of the proposed ACM-CNN method with different degrees of antenna noise uncertainty between training and detection. Specifically, “ACM-CNN ($v_a\mathrm{dB}/w_a\mathrm{dB}$)” denotes a model trained on a dataset with ${\epsilon }_a=v_a$ dB and evaluated on a test dataset with ${\epsilon }_a=w_a$ dB. As observed in the figure, the neural network model trained under an antenna noise uncertainty of ${\epsilon }_a=5$ dB performed slightly worse than the model trained with ${\epsilon }_a=0$ dB when tested on a dataset with ideal uniform noise (${\epsilon }_a=0$ dB). However, when evaluated on test sets with nonzero antenna noise uncertainty (${\epsilon }_a\ne 0$ dB), the ${\epsilon }_a=5$ dB-trained model outperformed its counterparts trained under ideal conditions. This phenomenon indicates that, while the presence of unstable noise across antennas during training introduces interference that slightly degrades peak detection performance, it simultaneously enhances the robustness to noise uncertainty. Consequently, the model trained with an antenna noise uncertainty of ${\epsilon }_a\ne 0$ dB exhibited slower performance degradation under mismatched antenna noise conditions, demonstrating superior generalization capability to inter-antenna noise variations.

Figure 17 illustrates the ROC curves of the proposed ACM-CNN method with different degrees of time noise uncertainty between training and detection. Specifically, “ACM-CNN ($v_t\mathrm{dB}/w_t\mathrm{dB}$)” denotes a model trained on a dataset with ${\epsilon }_t=v_t$ dB and evaluated on a test dataset with ${\epsilon }_t=w_t$ dB. Unlike the significant impact of unstable noise across antennas on model training, unstable noise across time had a much smaller effect on the training outcome and evaluation result. This is primarily because, prior to being fed into the neural network, the input data undergo covariance matrix computation, during which temporal variations are effectively averaged out. As a result, the influence of temporal noise uncertainty is substantially mitigated. These findings demonstrate that the proposed ACM-CNN method exhibits strong robustness and generalization capabilities in the presence of temporal unstable noise.

4.4. Probability of Detection Versus Noncircularity

Similarly to existing non-circular signal-based methods, our proposed ACM-CNN leverages both the circular and noncircular components inherently present in such signals. In the special case of circular signals, where the noncircular component is zero, noncircular-signal-based methods naturally degenerate into their circular counterparts. To assess how different levels of noncircularity rate $\kappa$ affect the performance of ACM-CNN, we present the detection probabilities of various algorithms as a function of $\kappa$, as illustrated in Figure 18. It can be observed that the performance of the three algorithms (ACM-CNN, NC-LAV, and NC-HDM) that exploit noncircular information improved significantly as $\kappa$ increased, whereas those that do not utilize noncircular features remained unaffected by changes in $\kappa$. When the noncircularity rate $\kappa$ approached zero, the performance of ACM-CNN closely aligned with that of CM-CNN. This demonstrates that ACM-CNN can gracefully reduce to a circular-signal-based model in the absence of noncircularity, while effectively leveraging additional structure when noncircularity is present, thus validating its adaptability and effectiveness for noncircular signals.

5. Conclusions

In this work, we have discussed the problem of spectrum sensing in multi-antenna cognitive radio networks by emphasizing the importance of non-circular signal characteristics. We proposed an augmented covariance-matrix-aware convolutional neural network (ACM-CNN) that integrates both conventional covariance and complementary covariance matrices into a unified learning framework. This design allows the model to fully exploit the second-order statistical features unique to non-circular signals, which are typically neglected in conventional spectrum sensing methods. Simulation results show that ACM-CNN significantly outperformed both traditional model-driven algorithms and CNN-based approaches in non-circular signal scenarios.