Physical Layer Authentication Exploiting Antenna Mutual Coupling Effects in mmWave Systems

Abstract

Impersonation attacks pose a significant threat to millimeter wave (mmWave) wireless systems due to the unique characteristics (e.g., highly directional beams) of mmWave communications. To this end, this paper proposes a novel physical layer authentication (PLA) scheme that leverages the antenna array-specific mutual coupling (MC) feature to validate the identity of the transmitter. In particular, we first demonstrate the authentication feasibility of the MC feature by modeling its amplitude and phase characteristics using generalized Gaussian and Laplace distributions, respectively. Then, based on the amplitude and phase of the MC feature, we design a kernel-based authentication scheme to further improve device distinguishability in mmWave systems. Moreover, we provide analytical expressions of the false alarm and detection probabilities, enabling a theoretical characterization of the proposed authentication scheme performance. Finally, numerical results are provided to verify the reliability and effectiveness of the proposed authentication scheme under various settings. The results indicate that the proposed scheme can provide performance gain compared to the single-dimensional feature-based schemes under different signal–noise ratio scenarios.

1. Introduction

Due to the bandwidth limitations of conventional wireless technologies, addressing the high data rate demands of fifth-generation (5G) wireless communications requires the adoption of novel solutions. Millimeter-wave (mmWave) communication, with its broad bandwidth range from 30–300 GHz and significantly enhanced data rates, is poised to play a crucial role in 5G network designs [1]. Such a technology can provide several advantages, such as narrow antenna beams [2], a wide transmission spectrum [3], low transmission latency, and the ability to achieve much higher data rates with comparable coverage and thus support the growing network demands of the Internet of Everything (IoE) [4] and vehicle-to-everything (V2X) [5] applications.

Despite the promising capabilities of mmWave communication systems, they are inherently vulnerable to identity-based impersonation attacks, primarily due to their key characteristics such as highly directional beamforming, limited coverage, and the potential for beam misalignment [6]. These vulnerabilities enable adversaries to impersonate legitimate devices or users, thereby gaining unauthorized access, disrupting communication, or engaging in malicious activities [7]. The impact of such attacks is particularly critical in several critical applications like autonomous driving and industrial automation, where the integrity of communication is paramount [8]. Consequently, robust authentication mechanisms are essential to establish secure communications and prevent unauthorized access for the concerned mmWave systems.

Cryptographic-based authentication has been the primary method for securing wireless communication systems [9,10]. It ensures security through key management and complicated mathematical computation. However, the management and distribution of these keys present significant challenges in the resource-constrained and dynamic mmWave systems [11]. Recently, physical layer authentication (PLA) has gained attention as a complementary solution. Due to the low communication overhead and enhanced system compatibility, it offers additional security without relying solely on cryptographic techniques [12,13]. The main principle of PLA is that it exploits unique channel features or hardware characteristics to construct and authenticate the identities of devices or users, which can typically be divided into two types: channel-based approaches [14,15,16,17,18,19] and hardware imperfection-based approaches [20,21,22,23,24].

Channel-based authentication primarily leverages the spatial variability in uncorrelated wireless channels across distinct geographical areas to achieve identification of wireless devices or users. For example, the authors in [14] leverage the sparsity of mmWave channels for user identification. Xiao et al. propose a method to detect Sybil attacks by exploiting the spatial variability in channel frequency responses, making it suitable for both wide-band and narrow-band systems [15]. The authors in [16] introduce an end-to-end physical layer authentication scheme for dual-hop wireless networks, utilizing location-specific features of channel amplitude and delay intervals, while incorporating artificial jamming to counter impersonation and replay attacks. Ferrante et al. focus on channel estimation-based authentication in Rayleigh fading channels, deriving an outer bound for the error region and optimizing attack strategies to minimize authentication errors [17]. Additionally, Wang et al. exploit the sparsity of mmWave channels to detect pilot contamination attacks under both static and dynamic environments [18]. While channel-based authentication is effective in mitigating unauthorized access, it primarily verifies the location or spatial signature of the transmitter rather than the identity of the device itself [19]. This dependence on spatial characteristics creates difficulties in situations where devices exhibit high mobility or experience frequent changes in location.

Hardware impairment-based authentication presents a highly effective and efficient method for bolstering security in mmWave communication systems. By exploiting the unique hardware characteristics of devices, this approach provides strong resistance to spoofing, requires minimal overhead, and adapts seamlessly to dynamic environments, offering enhanced security and making it a preferable choice for safeguarding mmWave networks from various security threats [20]. Zhao et al. propose an authentication scheme based on physical-layer phase noise fingerprints, combining phase noise and RF fingerprints to provide robust, low-computation security in emerging networks [21]. Polak et al. focus on exploiting phase noise to uniquely identify wireless devices [22]. Maeng et al. develop a power allocation strategy for fingerprint-based authentication in mmWave drone networks, optimizing both secrecy and achievable rate to strengthen security [23]. Shi et al. demonstrate that multi-input multi-output (MIMO) technology can improve the accuracy and stability of device identification by using multiple transmitters to enhance feature extraction [24].

It is worth noting that the aforementioned studies signify significant progress in the design of authentication solutions in mmWave systems. However, there are still several aspects for consideration to further motivate the development of PLA in mmWave systems. First, most existing hardware impairment-based PLA schemes mainly utilize hardware imperfections in the radio frequency chain (such as phase noise, carrier frequency offset, and I/Q imbalance), while ignoring hardware features such as mutual coupling (MC) effects that exist in massive MIMO antenna arrays. Recent research in [25] has also shown that MC is an inherent impairment in antenna arrays resulting from manufacturing tolerances and environmental factors. This effect is strongly influenced by the configuration of the antenna array and exhibits greater robustness to environmental changes, thus providing novel insights for the design of the PLA scheme. Second, existing authentication schemes often rely solely on the single-dimensional physical layer features of mmWave communication systems, which limits their performance due to the low distinguishability of these features. Finally, existing works primarily rely on the machine-learning approach, and the statistical performance evaluation of the authentication schemes remains an open issue.

Based on the above observations, we utilize the amplitude and phase of MC feature to design a kernel-based authentication scheme to determine the origin of the current received signals. Moreover, the analytical expressions of the authentication performance metrics, like detection and false alarm probabilities, are also derived to conduct the theoretical performance analysis. The main contributions of this paper are as follows:

• To demonstrate the authentication feasibility of the MC feature, we first utilize the generalized Gaussian and Laplace distributions to precisely characterize the statistical model of amplitude and phase of MC, respectively.
• To further enhance the device distinguishability, we leverage two unique hardware-specific fingerprints in terms of the amplitude and phase of MC to design a kernel-based identity authentication scheme tailored for the mmWave communication systems.
• Based on the knowledge of statistical signal processing and hypothesis testing, we analytically derive expressions for the false alarm and detection probabilities, providing a theoretical characterization of the authentication performance of the proposed authentication scheme.
• Finally, we resort to a series of experiments to verify the reliability, effectiveness of the proposed authentication scheme under various settings. The results also validate the correctness of the proposed theoretical performance metrics and demonstrate the satisfactory detection performance even in the low signal–noise ratio scenario.

While the proposed scheme demonstrates strong potential in exploiting MC features for authentication, it is important to acknowledge several practical considerations that may affect its robustness. For instance, variations in antenna configuration (e.g., element spacing or orientation), operating frequency, or transmit power can alter the electromagnetic environment and, consequently, the MC characteristics. Such changes may lead to mismatches between stored fingerprints and real-time measurements, potentially degrading authentication accuracy. Therefore, additional calibration mechanisms or adaptive re-learning schemes may be required in dynamic or reconfigurable deployment scenarios. These aspects will be considered in future extensions of this work.

Notation: ${(·)}^{*}$, ${(·)}^T$, and ${(·)}^H$ denote conjugate, transpose, and conjugate transpose operators, respectively. ${\mathbb{C}}^{M\times N}$ represents the set of complex-valued $M\times N$ matrices. A circularly symmetric complex Gaussian random vector $\mathbf{x}$ with zero mean and covariance matrix $\mathbf{C}$ can be denoted by $\mathbf{x}\sim \mathcal{N}(\mathbf{0},\mathbf{C})$. $\mathbb{E}\{·\}$ represents the expectation operator. ≜ represents definitions. $I_n(·)$ is the nth order modified Bessel function of the first kind. $\mathsf{\Gamma }(·)$ is the Gamma function.

2. Problem Formulation and System Model

2.1. Problem Formulation

As shown in Figure 1, we consider an mmWave wireless communication system that includes a legitimate base station (Alice), a potential spoofing attacker (Eve), and a single-antenna user equipment (UE). The UE aims to receive messages from the authentic base station, which is equipped with an M-antenna uniform linear array (ULA) over a downlink communication. A quasi-omni beam pattern is formed at the UE side to ensure signal reception. Eve attempts to impersonate the legitimate base station by using an impersonated identity. To protect against such impersonation, the UE utilizes unique hardware-dependent mutual coupling (MC) features to authenticate the identity of the received signal frame.

2.2. Antenna MC Model

In antenna array applications, MC can indeed introduce significant errors that affect the performance of the system. MC refers to the interaction between elements of an antenna array, where the signal received or transmitted by one antenna element affects the signals in other nearby elements. This can distort the signals, degrade the array’s radiation pattern, and impact the accuracy of signal-processing algorithms. MC effects are significantly influenced by factors such as antenna polarization, geometry design, and manufacturing materials. These factors create unique electromagnetic interactions between antenna elements, which can be exploited for various applications, including PHY-layer authentication.

The geometry of an antenna array is a fundamental factor influencing MC parameters. It directly impacts the electromagnetic interactions between elements, determining the magnitude and phase of the coupling coefficients. We first consider coupling in the transmitting mode in a uniform linear array (ULA). Because of space constraint, the antenna dipoles are closely positioned on in a uniform linear array ULA. When one element radiates, a portion of its energy is intercepted by neighboring elements. We take the transmitting mode as an example to illustrate the basic principle of MC effect generation in an antenna array. More specifically, without loss of generality, two dipole $m_1$ and $m_2$ of a ULA are positioned relative to each other as illustrated in Figure 2. When a signal source is connected to dipole $m_1$, the energy produced will propagate towards the antenna designated as ⓪, subsequently radiating into space ① and toward the $m_2$ antenna ②. The energy impinging upon $m_2$ antenna induces currents that tend to reradiate a portion of the energy ③, while facilitating the transmission of the remainder toward $m_2$ generator ④. A portion of the rescattered energy ③ has the potential to be redirected toward antenna $m_1$ ⑤. This process may persist indefinitely. A similar procedure would ensue if antenna dipole $m_2$ was activated and antenna dipole $m_1$ served as the parasitic element. When both antenna dipoles are simultaneously activated, the resulting radiated and rescattered fields from each antenna must be combined vectorially to determine the total field at any given observation point. Consequently, the overall contribution to the far-field pattern of a specific element within the array is influenced not only by the excitation provided by its own generator (termed direct excitation) but also by the cumulative parasitic excitation. This parasitic excitation is contingent upon the couplings from and the excitation levels of the other generators [26]. This coupling effect is generally represented as a modification in the apparent driving impedance of the components, a phenomenon commonly termed mutual impedance variation.

By applying the principles of fundamental electromagnetics and circuit theory [26], the coupling matrix ${\mathbf{C}}_{\mathrm{T}}$ of the transmit antenna array can be modeled in a standard way as

$$\begin{array}{c}\hfill {\mathbf{C}}_{\mathrm{T}}=(Z_A+Z_T){(\mathbf{Z}+Z_T\mathbf{I})}^{-1},\end{array}$$

(1)

where $Z_A$, $Z_T$ and $\mathbf{Z}\in {\mathbb{C}}^{M\times M}$ are the load impedance, antenna impedance, and the mutual impedance matrix, respectively. $\mathbf{Z}$ is the mutual impedance matrix given by

$$\mathbf{Z}=\left[\begin{array}{cccc}{Z}_A+Z_T& z_{1,2}& \dots & z_{1,M}\\ z_{2,1}& Z_A+Z_T& \dots & 0\\ ⋮& ⋮& ⋮& ⋮\\ z_{M,1}& z_{M,2}& \dots & Z_A+Z_T\end{array}\right],$$

(2)

where $z_{m_1,m_2}$ is the mutual impedance between the $m_1$th and $m_2$th antenna elements and is a complex value that is a function of the distance antenna elements. In the absence of MC, the matrix $\mathbf{Z}$ presented in (2) is characterized as diagonal. However, when mutual coupling occurs between the antenna elements, nonzero elements emerge in the off-diagonal positions. Figure 3 demonstrates the plot of the of the normalized impedance matrix elements for a ULA of 20 dipoles spaced $\lambda /2$ apart with terminating impedance $Z_A=Z_T$. We can see that the coupling between adjacent elements remains relatively uniform throughout the array. Furthermore, the strength of the coupling diminishes significantly as one moves further from the diagonal. For a ULA, $\mathbf{Z}$ is a symmetric Toeplitz matrix and thus the coupling matrix ${\mathbf{C}}_{\mathrm{T}}\in {\mathbb{C}}^{M\times M}$ of the antenna array can be expressed as banded symmetric Toeplitz matrices [27,28]. We assume that the number of nonzero mutual coupling coefficients of the transmit antenna array is $M_1$ and then ${\mathbf{C}}_{\mathrm{T}}$ could be rewritten as

$$\begin{array}{c}\hfill {\mathbf{C}}_{\mathrm{T}}=\mathrm{Toeplitz}\{{\mathbf{c}}_{\mathrm{T}},{\mathbf{0}}_{(M-M_1)\times 1}^{\mathrm{T}}\},\end{array}$$

(3)

where ${\mathbf{c}}_{\mathrm{T}}={[c_{\mathrm{T},1},c_{\mathrm{T},2},\dots ,c_{\mathrm{T},M_1}]}^{\mathrm{T}}$ with $c_{\mathrm{T},m}=a_{\mathrm{T},m}^c{e}^{j{\psi }_{\mathrm{T},m}^c}$. $a_{\mathrm{T},m}^c$ and ${\psi }_{\mathrm{T},m}^c$ are the amplitude value and phase value of the mth mutual coupling coefficient, respectively, and $0<\left|c_{\mathrm{T},M_1}\right|<\cdots <c_{\mathrm{T},1}=1$.

To apply mutual coupling features, we need to know statistical model for the factors $c_{\mathrm{T},m}$. According to [29], the amplitude value and phase value of the mutual coupling amplitude coefficient $a_{\mathrm{T},m}^c$ can be modeled as zero-mean generalized Gaussian random variables, that is,

$$\begin{array}{c}\hfill f_{a_{\mathrm{T},m}^c}\left(x\right)={\displaystyle \frac{{\beta }_a}{2{\alpha }_a\mathsf{\Gamma }\left(\frac{1}{{\beta }_a}\right)}}exp\left[-{\left|{\displaystyle \frac{x}{{\alpha }_a}}\right|}^{{\beta }_a}\right],\end{array}$$

(4)

where ${\alpha }_a$ and ${\beta }_a$ denote the scale and the shape parameters, respectively.

It is worth noting that ${\psi }_{\mathrm{T},m}^p$ is the phase value of the MC coefficient $c_{\mathrm{T},m}$ and is based on [30] ${\psi }_{\mathrm{T},m}^c$, which can be modeled as a Laplacian distribution, i.e.,

$$\begin{array}{c}\hfill f_{{\psi }_{\mathrm{T},m}^c}\left(x\right)={\displaystyle \frac{1}{\sqrt{2{\sigma }_{\psi }^2}}}exp\left(-\sqrt{{\displaystyle \frac{2}{{\sigma }_{\psi }^2}}}\left|x\right|\right),\end{array}$$

(5)

where ${\sigma }_{\psi }^2$ denotes the scale parameter. These parameters (i.e., ${\alpha }_a$, ${\beta }_a$ and ${\sigma }_{\psi }^2$) are related to dipole distance and hardware impairments (such as the load impedance, antenna impedance, and the mutual impedance matrix). These parameters have inherent characteristics that can be used to distinguish the identity of the device. So far, we have provided the statistical models for ${\mathbf{C}}_{\mathrm{T}}$ and following a similar manner, the coupling matrix ${\mathbf{C}}_{\mathrm{R}}$ of the receiving antenna array has a similar expression form. We assume that the number of nonzero mutual coupling coefficients of the transmit antenna array is $N_1$ and then ${\mathbf{C}}_{\mathrm{R}}$ could be rewritten as

$$\begin{array}{c}\hfill {\mathbf{C}}_{\mathrm{R}}=\mathrm{Toeplitz}\{{\mathbf{c}}_{\mathrm{R}},{\mathbf{0}}_{(N-N_1)\times 1}^{\mathrm{T}}\}.\end{array}$$

(6)

It is worth noting that the assumption of identical ULA configurations with fixed inter-element spacing may not hold in adversarial settings. A well-equipped attacker may deliberately design antenna arrays to mimic the MC characteristics of a target device, especially after observing transmitted signals over time. In such cases, the uniqueness and stability of MC fingerprints could be compromised. Therefore, to ensure authentication robustness in the presence of adaptive adversaries, the proposed method could be extended by incorporating additional features such as time-varying channel statistics, gain/phase distortion fingerprints, or employing a dynamic challenge-response mechanism. These considerations form part of our planned future work.

2.3. Signal Model

Consider a massive MIMO communication system with a transmitter equipped with an array of M transmit antennas and a receiver with an array of N receive antennas, and a potential adversary with a transmit array of M elements. The potential adversary attempts to impersonate as the legitimate transmitter for inserting harmful information into the concerned communication system. All of them are equipped with ULAs with interelement spacing being half-wavelength. Suppose that M orthogonal signals are transmitted by M antennas at time t, i.e., $\mathbf{s}\left(t\right)={[s_1\left(t\right),\cdots ,s_M\left(t\right)]}^{\mathrm{T}}$, where $s_m\left(t\right)$ is the mth baseband signal and $\mathbb{E}\left\{\right|s_m\left(t\right){|}^2\}=1$ and $\mathbb{E}\left\{\right|s_m\left(t\right)s_n\left(t\right)\left|\right\}=0$ (for $m\ne n$). Let $d_{\mathrm{T}}$ and $d_{\mathrm{R}}$ be the distance between two adjacent antennas in the transmitter and receiver, respectively (referred to as the separation distance). The extended Saleh-Valenzuela (SV) model in [31] is employed to characterize the concerned channel. Let ${\theta }_l$ and ${\varphi }_l$ be the direction of departure (DoD) and direction of arrival (DoA) for the lth propagation path, respectively. Then, The steering vectors of the transmitter and receiver are expressed, respectively, by

$$\begin{array}{c}\hfill {\mathbf{a}}_{\mathrm{T}}\left({\theta }_l\right)=[1,e^{j2\pi {\displaystyle \frac{d_{\mathrm{T}}}{\lambda }}sin{\theta }_l},...,e^{j2\pi {\displaystyle \frac{(M-1)d_{\mathrm{T}}}{\lambda }}sin{\theta }_l}{]}^{\mathrm{T}},\end{array}$$

(7)

$$\begin{array}{c}\hfill {\mathbf{a}}_{\mathrm{R}}\left({\varphi }_l\right)=[1,e^{j2\pi {\displaystyle \frac{d_{\mathrm{R}}}{\lambda }}sin{\varphi }_l},...,e^{j2\pi {\displaystyle \frac{(N-1)d_{\mathrm{R}}}{\lambda }}sin{\varphi }_l}{]}^{\mathrm{T}}.\end{array}$$

(8)

We consider narrow-band quasi-static fading channels and there are L propagation paths between the transmitter and receiver. This study investigates the uplink communication in a single transmission frame that is composed of K signal blocks. It is assumed that the channels remain unchanged in each block. The value of K is an arbitrary integer that is determined by the channel coherence time. If the transmitter UE transmits the $M\times 1$ signal vector $\mathbf{s}\left(t\right)$ at time t for the kth block, the signal received at the BS is given by

$$\begin{array}{cc}\hfill \mathbf{y}(t,k)& =\sum _{l=1}^L{a}_l\left(k\right)\left[{\mathbf{C}}_{\mathrm{R}}{\mathbf{a}}_{\mathrm{R}}\left({\varphi }_l\right)\right]{\left[{\mathbf{C}}_{\mathrm{T}}{\mathbf{a}}_{\mathrm{T}}\left({\theta }_l\right)\right]}^{\mathrm{T}}\mathbf{s}\left(t\right)+\mathbf{n}(t,k)\hfill \\ \hfill & =\left[{\mathbf{C}}_{\mathrm{R}}{\mathbf{A}}_{\mathrm{R}}\left(\varphi \right)\right]\mathrm{diag}\left\{\mathbf{a}\left(k\right)\right\}{\left[{\mathbf{C}}_{\mathrm{T}}{\mathbf{A}}_{\mathrm{T}}\left(\theta \right)\right]}^{\mathrm{T}}\mathbf{s}\left(t\right)+\mathbf{n}(t,k),\hfill \end{array}$$

(9)

where ${\mathbf{A}}_{\mathrm{T}}\left(\theta \right)=[{\mathbf{a}}_{\mathrm{T}}\left({\theta }_l\right),\cdots ,{\mathbf{a}}_{\mathrm{T}}\left({\theta }_L\right)]$ and ${\mathbf{A}}_{\mathrm{R}}\left(\varphi \right)=[{\mathbf{a}}_{\mathrm{R}}\left({\varphi }_l\right),\cdots ,{\mathbf{a}}_{\mathrm{R}}\left({\varphi }_L\right)]$; $\mathbf{n}(t,k)$ is a noise vector with zero-mean white Gaussian distribution with variance ${\sigma }_n^2{\mathbf{I}}_N$; $\mathbf{a}\left(k\right)={[a_1\left(k\right),\cdots ,a_L\left(k\right)]}^{\mathrm{T}}$ with $a_l\left(k\right)$ being the channel gain along the lth path.

The objective of this paper is develop a novel authentication scheme utilizing the MC feature to resist against the identity-based impersonation attack. One can see from (9) that we need to estimate MC in ${\mathbf{C}}_{\mathrm{T}}$ based on the received observation signal vector.

3. Proposed Kernel-Based Authentication Scheme

3.1. Estimate MC

Using the orthogonality property of transmitted signals (a matched filter method) and performing the vectorization operation yields the $MN\times 1$ virtual data vector as

$$\begin{array}{cc}\hfill \mathbf{y}\left(k\right)& =[{\mathbf{C}}_{\mathrm{T}}\otimes {\mathbf{C}}_{\mathrm{R}}][{\mathbf{A}}_{\mathrm{T}}\odot {\mathbf{A}}_{\mathrm{R}}]\mathbf{a}\left(k\right)+\mathbf{n}\left(k\right)\hfill \\ \hfill & =\mathbf{C}\mathbf{A}(\theta ,\varphi )\mathbf{a}\left(k\right)+\mathbf{n}\left(k\right),\hfill \end{array}$$

(10)

where $\mathbf{A}(\theta ,\varphi )=[\mathbf{a}({\theta }_1,{\varphi }_1),\cdots ,\mathbf{a}({\theta }_L,{\varphi }_L)]$ denotes the ideal steering matrix of the virtual array with $\mathbf{a}({\theta }_l,{\varphi }_l)={\mathbf{a}}_{\mathrm{T}}\left({\theta }_l\right)\otimes {\mathbf{a}}_{\mathrm{R}}\left({\varphi }_l\right)$; $\mathbf{n}\left(k\right)\sim \mathbb{CN}(\mathbf{0},{\sigma }_n^2{\mathbf{I}}_{MN})$ denotes the noise term; and the MC matrix $\mathbf{C}\in {\mathbb{C}}^{MN\times MN}$ is expressed by

$$\begin{array}{cc}\hfill \mathbf{C}& ={\mathbf{C}}_{\mathrm{T}}\otimes {\mathbf{C}}_{\mathrm{R}}\hfill \\ \hfill & =\mathrm{Toeplitz}\{{\mathbf{C}}_{\mathrm{R}},c_{\mathrm{T},1}{\mathbf{C}}_{\mathrm{R}},\cdots ,c_{\mathrm{T},M_1}{\mathbf{C}}_{\mathrm{R}},{\mathbf{0}}_{N\times N},\cdots ,{\mathbf{0}}_{N\times N}\}.\hfill \end{array}$$

(11)

Collecting K signal blocks forms the following expression as

$$\begin{array}{c}\hfill \mathbf{Y}=\mathbf{C}\mathbf{A}(\theta ,\varphi )\mathbf{H}+\mathbf{N},\end{array}$$

(12)

where $\mathbf{Y}=\left[\mathbf{y}\right(1),\cdots ,\mathbf{y}(K\left)\right]$, $\mathbf{H}=\left[\mathbf{a}\right(1),\cdots ,\mathbf{a}(K\left)\right]$ and $\mathbf{N}=\left[\mathbf{n}\right(1),\cdots ,\mathbf{n}(K\left)\right]$. It is noticed that the received signal model in (12) is not limited to be within a half wavelength separation distance and hence this can be readily extended for any antenna element spacing. Here, we consider that the transmit antenna array has the identical number of nonzero mutual coupling coefficients of the receive array, i.e., $M_1=N_1$. To determine the angles and mutual coupling without ambiguity, it may be inferred from [32] that the necessary condition of $M_1=N_1$ is $M_1=N_1⩽min\{(L(MN-L)-2L)/2, M/2, N/2\}$.

$\blacksquare$

MC compensation: In this subsection, the goal is to extract MC. It is noticed that we first need to jointly estimate DoD and DoA with unknown MC. To facilitate the joint accurate DoD and DoA estimation, it is necessary to implement MC compensation. In particular, the first and last $M_1$ ($N_1$) transmit (receiving) antenna elements are designated as auxiliary elements. The remaining elements are renumbered from 1 to $M_2=M-2M_1$ and from 1 to $N_2=N-2N_1$, respectively. We denote by $\overline{\mathbf{Y}}$ the $M_2{N}_2\times K$ virtual received data matrix obtained from the $M_2+N_2$ non-auxiliary elements. To mitigate the effect of mutual coupling, the data from the $2(M_1+N_1)$ auxiliary elements is discarded, and $\overline{\mathbf{Y}}$ is directly used for angle estimation. More specifically, define the matrices ${\mathbf{Q}}_{\mathrm{T}}\triangleq [{\mathbf{0}}_{M_2\times M_1},I_{M_2},{\mathbf{0}}_{M_2\times M_1}]$ and ${\mathbf{Q}}_{\mathrm{R}}\triangleq [{\mathbf{0}}_{N_2\times N_1},I_{N_2},{\mathbf{0}}_{N_2\times N_1}]$ and then the received data $\overline{\mathbf{Y}}$ can be obtained by left multiplying ${\mathbf{Q}}_{\mathrm{T}}\otimes {\mathbf{Q}}_{\mathrm{R}}$ on both sides of $\mathbf{Y}$ in (12) as

$$\begin{array}{cc}\hfill \overline{\mathbf{Y}}& =({\mathbf{Q}}_{\mathrm{T}}\otimes {\mathbf{Q}}_{\mathrm{R}})\mathbf{Y}=[{\mathbf{Q}}_{\mathrm{T}}{\mathbf{C}}_{\mathrm{T}}\otimes {\mathbf{Q}}_{\mathrm{R}}{\mathbf{C}}_{\mathrm{R}}]\mathbf{A}(\theta ,\varphi )\mathbf{H}+({\mathbf{Q}}_{\mathrm{T}}\otimes {\mathbf{Q}}_{\mathrm{R}})\mathbf{N}\hfill \\ \hfill & =\overline{\mathbf{C}}\mathbf{A}(\theta ,\varphi )\mathbf{H}+\overline{\mathbf{N}},\hfill \end{array}$$

(13)

where $\overline{\mathbf{C}}={\mathbf{Q}}_{\mathrm{T}}{\mathbf{C}}_{\mathrm{T}}\otimes {\mathbf{Q}}_{\mathrm{R}}{\mathbf{C}}_{\mathrm{T}}\in {\mathbb{C}}^{M_2{N}_2\times MN}$ is an equivalent composite MC matrix, and $\overline{\mathbf{N}}\in {\mathbb{C}}^{M_2{N}_2\times K}$ is a new noise matrix. It is easy to calculate the covariance matrix of $\overline{\mathbf{Y}}$ as

$$\begin{array}{c}\hfill {\mathbf{R}}_{\overline{\mathbf{Y}}}=\mathbb{E}\left\{\overline{\mathbf{Y}}{\overline{\mathbf{Y}}}^{\mathrm{H}}\right\}=\overline{\mathbf{C}}\mathbf{A}(\theta ,\varphi ){\mathbf{R}}_{\mathbf{H}}{\mathbf{A}}^{\mathrm{H}}(\theta ,\varphi ){\overline{\mathbf{C}}}^{\mathrm{H}}+{\sigma }_n^2{\mathrm{I}}_{M_2{N}_2},\end{array}$$

(14)

where ${\mathbf{R}}_{\mathbf{H}}=\mathbb{E}\left\{\mathbf{H}{\mathbf{H}}^{\mathrm{H}}\right\}$. Performing the eigendecomposition on the $M_2{N}_2\times M_2{N}_2$ covariance matrix ${\mathbf{R}}_{\overline{\mathbf{Y}}}$, then we have

$$\begin{array}{c}\hfill {\mathbf{R}}_{\overline{\mathbf{Y}}}={\overline{\mathbf{E}}}_s{\overline{\mathsf{\Lambda }}}_s{\overline{\mathbf{E}}}_s^{\mathbf{H}}+{\overline{\mathbf{E}}}_n{\overline{\mathsf{\Lambda }}}_n{\overline{\mathbf{E}}}_n^{\mathbf{H}},\end{array}$$

(15)

where ${\overline{\mathbf{E}}}_s$ denotes the signal subspace consisting of the eigenvectors corresponding to the largest L eigenvalues and ${\overline{\mathbf{E}}}_n$ denotes the noise subspace containing the rest eigenvectors of ${\mathbf{R}}_{\overline{\mathbf{Y}}}$. It was proved in [32] that the subspace ${\overline{\mathbf{E}}}_s$ consisting of non-auxiliary components, is resistant to MC and can be directly utilized for accurately estimating the angles since the mutual coupling effect is compensated.

$\blacksquare$

Angle estimation applying the complex matrix method: Here a complex matrix method is employed to estimate the DoD and DoA. Observing (13), we find that the subspace spanned by $\overline{\mathbf{C}}\mathbf{A}(\theta ,\varphi )$ is identical to the subspace spanned by the column eigenvectors in matrix ${\overline{\mathbf{E}}}_s$. Then, it should have a unique nonsingular matrix $\mathsf{\Gamma }$ to satisfy

$$\begin{array}{c}\hfill {\overline{\mathbf{E}}}_s=\overline{\mathbf{C}}\mathbf{A}(\theta ,\varphi )\mathbf{Y}=\overline{\mathbf{A}}(\theta ,\varphi )\mathsf{\Delta }\mathbf{Y},\end{array}$$

(16)

where $\overline{\mathbf{A}}(\theta ,\varphi )=[\overline{\mathbf{a}}({\theta }_1,{\varphi }_1),\cdots ,\overline{\mathbf{a}}({\theta }_L,{\varphi }_L)]$ and $\mathsf{\Delta }=\mathrm{diag}\{\mathsf{\Delta }({\theta }_1,{\varphi }_1),\cdots ,\mathsf{\Delta }({\theta }_L,{\varphi }_L)\}$. Choosing different submatrices from ${\overline{\mathbf{E}}}_s$ constructs the shift invariance matrices of the transmit and receive arrays for angle estimation. Define ${\mathsf{\Psi }}_{\mathrm{T}}\triangleq {\mathbf{Y}}^{-1}{\mathsf{\Phi }}_{\mathrm{T}}\mathbf{Y}$ and ${\mathsf{\Psi }}_{\mathrm{R}}\triangleq {\mathbf{Y}}^{-1}{\mathsf{\Phi }}_{\mathrm{R}}\mathbf{Y}$, where ${\mathsf{\Phi }}_{\mathrm{T}}=\mathrm{diag}\{e^{j2\pi {\displaystyle \frac{d_{\mathrm{T}}}{\lambda }}sin{\theta }_1},\cdots ,e^{j2\pi {\displaystyle \frac{d_{\mathrm{T}}}{\lambda }}sin{\theta }_L}\}$ and ${\mathsf{\Phi }}_{\mathrm{R}}=\mathrm{diag}\{e^{j2\pi {\displaystyle \frac{d_{\mathrm{R}}}{\lambda }}sin{\varphi }_1},\cdots ,e^{j2\pi {\displaystyle \frac{d_{\mathrm{R}}}{\lambda }}sin{\varphi }_L}\}$ are the shift invariance matrices of the transmit and receive arrays, respectively. Let ${\mathbf{D}}_0={\mathbf{Y}}^{-1}{\mathbf{I}}_L\mathbf{Y}$. Based on ${\mathsf{\Phi }}_{\mathrm{T}}$ and ${\mathsf{\Phi }}_{\mathrm{R}}$, a complex matrix denoted by ${\mathbf{G}}_i$ ($i=\mathrm{T},\mathrm{R}$) can be expressed as

$$\begin{array}{cc}\hfill {\mathbf{G}}_i& ={({\mathsf{\Psi }}_i+{\mathbf{D}}_0)}^{-1}({\mathsf{\Psi }}_i-{\mathbf{D}}_0)\hfill \\ \hfill & ={\mathbf{Y}}^{-1}{({\mathsf{\Phi }}_i+{\mathbf{I}}_L)}^{-1}({\mathsf{\Phi }}_i-{\mathbf{I}}_L)\mathbf{Y}.\hfill \end{array}$$

(17)

To simplify notation, let ${\omega }_{i,l}=2\pi {\displaystyle \frac{d_i}{\lambda }}sin{\varphi }_l$ and then the eigenvalues of ${\mathbf{G}}_i$ are given by

$$\begin{array}{c}\hfill {({\mathsf{\Phi }}_i+{\mathbf{I}}_L)}^{-1}({\mathsf{\Phi }}_i-{\mathbf{I}}_L)=\mathrm{diag}\left\{jtan{\displaystyle \frac{{\omega }_{i,1}}{2}},\cdots ,jtan{\displaystyle \frac{{\omega }_{i,L}}{2}}\right\}.\end{array}$$

(18)

Combining ${\mathsf{\Psi }}_{\mathrm{T}}$ and ${\mathsf{\Psi }}_{\mathrm{R}}$ constructs a complex pairing matrix ${\mathbf{G}}_{\mathrm{T}}+{\mathbf{G}}_{\mathrm{R}}$, whose eigenvalues can be calculated as

$$\begin{array}{cc}\hfill & {({\mathsf{\Phi }}_{\mathrm{T}}+{\mathbf{I}}_L)}^{-1}({\mathsf{\Phi }}_{\mathrm{T}}-{\mathbf{I}}_L)+j{({\mathsf{\Phi }}_{\mathrm{R}}+{\mathbf{I}}_L)}^{-1}({\mathsf{\Phi }}_{\mathrm{R}}-{\mathbf{I}}_L)\hfill \\ \hfill & =\mathrm{diag}\left\{tan{\displaystyle \frac{{\omega }_{\mathrm{R},1}}{2}}+jtan{\displaystyle \frac{{\omega }_{\mathrm{T},1}}{2}},\cdots ,tan{\displaystyle \frac{{\omega }_{\mathrm{R},L}}{2}}+jtan{\displaystyle \frac{{\omega }_{\mathrm{T},L}}{2}}\right\}.\hfill \end{array}$$

(19)

One can see from (19) that the lth eigenvalue is uniquely associated with the parameters $e^{j{\omega }_{\mathrm{R},l}}$ and $e^{j{\omega }_{\mathrm{T},l}}$, where the real and imaginary parts correspond to the real and imaginary parts of the eigenvalue, respectively. Hence, if we let ${\lambda }_l$ be the eigenvalue estimates of ${\mathbf{G}}_{\mathrm{T}}+{\mathbf{G}}_{\mathrm{R}}$, then the estimates of ${\widehat{\theta }}_l$ and ${\widehat{\varphi }}_l$ for $l=1,\cdots ,L$ are expressed, respectively, as

$$\left\{\begin{array}{cc}{\widehat{\theta }}_l\hfill & =arcsin\{{\displaystyle \frac{\lambda }{d_{\mathrm{T}}\pi }}arctan\left[\mathrm{Im}\left({\lambda }_l\right)\right]\},\hfill \\ {\widehat{\varphi }}_l\hfill & =arcsin\{{\displaystyle \frac{\lambda }{d_{\mathrm{R}}\pi }}arctan\left[\mathrm{Re}\left({\lambda }_l\right)\right]\}.\hfill \end{array}\right.$$

(20)

The DoD and DoA estimation includes two key steps: (1) decoupling operation; (2) complex matrix based on angle estimation.

The decoupling operation is performed by setting the auxiliary elements. Secondly, the angles are estimated using the complex matrix method.

$\blacksquare$

Mutual coupling coefficients estimation: So far we have obtained DoD and DoA estimation and then we can perform MC estimation via the received data of the full virtual array.

To this end, we first calculate the $MN\times MN$ sample covariance matrix of $\mathbf{Y}$ in (12) as ${\mathbf{R}}_{\mathbf{Y}}={\displaystyle \frac{1}{K}}{\sum }_{k=1}^K{\mathbf{y}}_k{\mathbf{y}}_k^{\mathrm{H}}$, and then performing the eigendecomposition on ${\mathbf{R}}_{\mathbf{Y}}$ as

$$\begin{array}{c}\hfill {\mathbf{R}}_{\mathbf{Y}}={\mathbf{E}}_s{\mathsf{\Lambda }}_s{\mathbf{E}}_s^{\mathbf{H}}+{\mathbf{E}}_n{\mathsf{\Lambda }}_n{\mathbf{E}}_n^{\mathbf{H}},\end{array}$$

(21)

where ${\mathbf{E}}_s$ and ${\mathbf{E}}_n$ are the signal subspace and noise subspace, respectively. To facilitate MC estimate, we resort to the following lemma regarding the MC matrix.

Lemma 1.

For a complex symmetric Toeplitz matrix $\mathbf{X}=Toeplitz\left\{\mathbf{x}\right\}\in {\mathbb{C}}^{K\times K}$ and a complex vector $\mathbf{z}$, we have the following relationship

$$\begin{array}{c}\hfill \mathbf{X}\mathbf{z}=\mathbf{D}\mathbf{x},\end{array}$$

(22)

where $\mathbf{D}={\mathbf{D}}_1+{\mathbf{D}}_2$ with the ith ($i=1,\cdots ,M$) row and the jth ($j=0,\cdots ,K-1$) column entries being

$${\left[{\mathbf{D}}_1\right]}_{i,j}=\left\{\begin{array}{c}{x}_{i+j},\hfill \\ 0,\hfill \end{array}\right.\begin{array}{c}i+j⩽K-1\\ \mathrm{otherwise}\end{array}$$

(23)

$${\left[{\mathbf{D}}_2\right]}_{i,j}=\left\{\begin{array}{c}{x}_{i-j},\hfill \\ 0,\hfill \end{array}\right.\begin{array}{c}i⩽j⩽1\\ \mathrm{otherwise}\end{array}$$

(24)

Let ${\delta }_l=\left[{\mathbf{C}}_{\mathrm{T}}{\mathbf{a}}_{\mathrm{T}}\left({\theta }_l\right)\right]\otimes \left[{\mathbf{C}}_{\mathrm{R}}{\mathbf{a}}_{\mathrm{R}}\left({\varphi }_l\right)\right]$ and, based on Lemma 1 and (10), we have

$$\begin{array}{cc}\hfill {\delta }_l& =[{\mathbf{D}}_{\mathrm{T}}\left({\theta }_l\right)\otimes {\mathbf{D}}_{\mathrm{R}}\left({\varphi }_l\right)]\mathbf{c}\hfill \\ \hfill & =\mathbf{D}({\theta }_l,{\varphi }_l)\mathbf{c},\hfill \end{array}$$

(25)

where $\mathbf{c}\triangleq {\mathbf{c}}_{\mathrm{T}}\otimes {\mathbf{c}}_{\mathrm{R}}$. Now, we construct a cost function

$$\begin{array}{c}\hfill J=\sum _{l=1}^L\left|\right|{\mathbf{E}}_n^{\mathrm{H}}{\delta }_l{\left|\right|}^2,\end{array}$$

(26)

and then the estimate of $\mathbf{c}$ can be expressed as

$$\begin{array}{c}\hfill \widehat{\mathbf{c}}=arg\underset{c}{min}\sum _{l=1}^L\left|\right|{\mathbf{E}}_n^{\mathrm{H}}\mathbf{D}({\widehat{\theta }}_l,{\widehat{\varphi }}_l)\mathbf{c}{\left|\right|}^2\iff arg\underset{c}{min}{\mathbf{c}}^{\mathrm{H}}\mathbf{B}\mathbf{c},\end{array}$$

(27)

where $\mathbf{B}={\sum }_{l=1}^L\left[{\mathbf{D}}^{\mathrm{H}}({\widehat{\theta }}_l,{\widehat{\varphi }}_l){\mathbf{E}}_n^{\mathrm{H}}{\mathbf{E}}_n\mathbf{D}({\widehat{\theta }}_l,{\widehat{\varphi }}_l)\right]$ is postive-definite Hermitan matrix. Then, since $\mathbf{c}\left(1\right)=1$, (27) becomes a constrained optimization problem and we can solve this optimization to extract $\widehat{\mathbf{c}}$. According to the structure of $\mathbf{c}\triangleq {\mathbf{c}}_{\mathrm{T}}\otimes {\mathbf{c}}_{\mathrm{R}}$, we have

$$\begin{array}{c}\hfill {\widehat{c}}_{\mathrm{T},m}=\widehat{\mathbf{c}}(m{M}_1+1),m=0,...,M_1.\end{array}$$

(28)

Consider that there are K samples for ${\widehat{\mathbf{c}}}_{\mathrm{T}}$ and construct an estimate matrix ${\widehat{\mathbf{C}}}_{\mathrm{T}}=[{\widehat{\mathbf{c}}}_{\mathrm{T},1},\cdots ,{\widehat{\mathbf{c}}}_{\mathrm{T},K}]$ where ${\widehat{\mathbf{c}}}_{\mathrm{T},k}={[{\widehat{c}}_{\mathrm{T},1k},\cdots ,{\widehat{c}}_{\mathrm{T},M_{1}k}]}^{\mathrm{T}}$. Performing vectorization and reforming the elements yields $K{M}_1$ estimates ${\widehat{c}}_{\mathrm{T},i}={\widehat{a}}_{\mathrm{T},i}{e}^{j{\widehat{\mathsf{\Psi }}}_{\mathrm{T},i}}$ for $i=1,...,K{M}_1$. Due to the existence of the estimation error, the amplitude and phase of MC estimates ${\widehat{c}}_{\mathrm{T},i}$ can be decomposed as

$$\begin{array}{c}\hfill {\widehat{a}}_{\mathrm{T},i}=a_{\mathrm{T},i}^c+a_{\mathrm{T},i}^{ϵ},i=1,...,K{M}_1,\end{array}$$

(29)

$$\begin{array}{c}\hfill {\widehat{\mathsf{\Psi }}}_{\mathrm{T},i}={\mathsf{\Psi }}_{\mathrm{T},i}^c+{\mathsf{\Psi }}_{\mathrm{T},i}^{ϵ},i=1,...,K{M}_1,\end{array}$$

(30)

where $a_{\mathrm{T},i}^{ϵ}$ and ${\mathsf{\Psi }}_{\mathrm{T},i}^{ϵ}$ are the amplitude and phase component of the estimate error, respectively, and we have $a_{\mathrm{T},i}^{ϵ}\sim \mathcal{N}(0,{\sigma }_{a,ϵ}^2)$ and ${\mathsf{\Psi }}_{\mathrm{T},i}^{ϵ}\sim \mathcal{N}(0,{\sigma }_{\mathsf{\Psi },ϵ}^2)$.

Based on (29) and (30), we can calculate the scale parameter ${\alpha }_a$ and the shape parameter ${\beta }_a$ associated with the amplitude value, and the scale parameter ${\sigma }_{\psi }^2$ associated with the phase value, and these parameters govern the mean and variance in the normal distribution for the kernel-based test statistics (which will be introduced in Section 3.2).

MC is not isolated in practice and often coexists with other RF impairments such as mutual impedance, antenna mismatch, and multi-path interference. Although our compensation and subspace projection techniques are designed to mitigate these influences, their separation under hardware constraints remains challenging. Extending the model to jointly estimate and compensate for multiple types of hardware impairments will be explored.

3.2. Validation Decision

Given the knowledge that the adversary Eve has the ability to modify transmit signals or channels to compromise the authentication system, Bob employs a hypothesis test [33] to determine if the message was initiated by Alice. There are two hypotheses regarding Bob as

• $H_0$: the signal frame is from Alice,

  $$\begin{array}{c}\hfill {\widehat{a}}_{\mathrm{T},i}=a_{\mathrm{T},i}^c+a_{\mathrm{T},i}^{ϵ},i=1,...,K{M}_1,\end{array}$$

  (31)

  $$\begin{array}{c}\hfill {\widehat{\mathsf{\Psi }}}_{\mathrm{T},i}={\mathsf{\Psi }}_{\mathrm{T},i}^c+{\mathsf{\Psi }}_{\mathrm{T},i}^{ϵ},i=1,...,K{M}_1,\end{array}$$

  (32)
• $H_1$: the signal frame is not from Alice,

  $$\begin{array}{c}\hfill {\widehat{a}}_{\mathrm{T},i}\ne a_{\mathrm{T},i}^c+a_{\mathrm{T},i}^{ϵ},i=1,...,K{M}_1,\end{array}$$

  (33)

  $$\begin{array}{c}\hfill {\widehat{\mathsf{\Psi }}}_{\mathrm{T},i}\ne {\mathsf{\Psi }}_{\mathrm{T},i}^c+{\mathsf{\Psi }}_{\mathrm{T},i}^{ϵ},i=1,...,K{M}_1,\end{array}$$

  (34)

Since the amplitude and phase of MC associated with the antenna array for the adversary Eve is different from that of Alice, thus the above two hypotheses can further formulated as

$$\begin{array}{cc}\hfill & H_0:{\widehat{a}}_{\mathrm{T},i}=a_{\mathrm{T},i}^c+a_{\mathrm{T},i}^{ϵ},{\widehat{\mathsf{\Psi }}}_{\mathrm{T},i}={\mathsf{\Psi }}_{\mathrm{T},i}^c+{\mathsf{\Psi }}_{\mathrm{T},i}^{ϵ},\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hfill \end{array}$$

(35a)

$$\begin{array}{cc}\hfill & H_1:{\widehat{a}}_{\mathrm{T},i}=a_{\mathrm{T},i}^c+a_{\mathrm{T},i}^{ϵ}+a_{\mathrm{T},i}^e,{\widehat{\mathsf{\Psi }}}_{\mathrm{T},i}={\mathsf{\Psi }}_{\mathrm{T},i}^c+{\mathsf{\Psi }}_{\mathrm{T},i}^{ϵ}+{\mathsf{\Psi }}_{\mathrm{T},i}^e\hfill \end{array}$$

(35b)

where $a_{\mathrm{T},i}^e$ and ${\mathsf{\Psi }}_{\mathrm{T},i}^e$ ($i=1,...,K{M}_1$) are the amplitude and phase component of the nonzero offset (attack) data caused by inherent antenna characteristics of Eve, respectively, and $a_{\mathrm{T},i}^e\sim \mathcal{N}(0,{\sigma }_{a,e}^2)$ and ${\mathsf{\Psi }}_{\mathrm{T},i}^e\sim \mathcal{N}(0,{\sigma }_{\mathsf{\Psi },e}^2)$.

We adopt the kernelized energy detector proposed in [34] to perform the authentication decision, i.e.,

$$\begin{array}{c}\hfill T={\displaystyle \frac{1}{N(N-1)}}\sum _{i=1}^N\sum _{j=1,j\ne i}^N\mathsf{\Omega }(x_i,x_j)\underset{H_0}{\overset{H_1}{\gtrless }}\gamma ,\end{array}$$

(36)

where $\mathsf{\Omega }(x_i,x_j)$ is a nonlinear kernel function and N is the number of variables. The asymptotic distribution of the kernel-based test statistic in (36) can be determined according to the U-statistics theory in [35] (Chapter 5). In particular, when N is sufficiently large, the test statistic in (36) follows a normal distribution in both hypotheses as follows:

$$T\sim \left\{\begin{array}{c}\mathcal{N}({\mu }_0,{\sigma }_0^2),\hfill \\ \mathcal{N}({\mu }_1,{\sigma }_1^2),\hfill \end{array}\right.\begin{array}{c}\mathrm{under}{H}_0\\ \mathrm{under}{H}_1\end{array}$$

(37)

where the mean ${\mu }_{\ell }$ and variance ${\sigma }_{\ell }^2$ for $\ell =0,1$ can be computed, respectively, as

$$\begin{array}{c}\hfill {\mu }_{\ell }={\mathbb{E}}_{X_i,X_j}\left\{\mathsf{\Omega }(x_i,x_j)\right\}={\displaystyle \frac{1}{N(N-1)}}\sum _{i=1}^N\sum _{j=1,j\ne i}^N\mathsf{\Omega }(x_i,x_j),\end{array}$$

(38)

$$\begin{array}{cc}\hfill {\sigma }_{\ell }^2& ={\displaystyle \frac{4}{N}}\left[{\mathbb{E}}_{X_i}\{{\left({\mathbb{E}}_{X_j}\left\{\mathsf{\Omega }(x_i,x_j)\right\}\right)}^2-[{\mathbb{E}}_{X_j}\left\{\mathsf{\Omega }(x_i,x_j)\right\}{]}^2\right]\hfill \\ \hfill & ={\displaystyle \frac{4}{N}}\left\{{\displaystyle \frac{1}{N}}\sum _{i=1}^N{\left[{\displaystyle \frac{1}{N}}\sum _{j=1}^N\mathsf{\Omega }(x_i,x_j)\right]}^2-{\mu }_k^2\right\}.\hfill \end{array}$$

(39)

Notably, gain and phase elements have different distribution, so we can employ the kernel-based test statistic in (36) to develop two different detectors exploiting amplitude and phase variables. Hence, based on (36), we have

$$\begin{array}{c}\hfill T_a={\displaystyle \frac{1}{K{M}_1(K{M}_1-1)}}\sum _{i=1}^{K{M}_1}\sum _{j=1,j\ne i}^{K{M}_1}\mathsf{\Omega }(a_i,a_j)\underset{H_0}{\overset{H_1}{\gtrless }}{\gamma }_a,\end{array}$$

(40)

$$\begin{array}{c}\hfill T_{\mathsf{\Psi }}={\displaystyle \frac{1}{K{M}_1(K{M}_1-1)}}\sum _{i=1}^{K{M}_1}\sum _{j=1,j\ne i}^{K{M}_1}\mathsf{\Omega }({\mathsf{\Psi }}_i,{\mathsf{\Psi }}_j)\underset{H_0}{\overset{H_1}{\gtrless }}{\gamma }_{\mathsf{\Psi }}.\end{array}$$

(41)

Based on the extracted MC and the corresponding statistics, we can perform an authentication decision by employing the kernel-based authentication decision. Now, we are ready to introduce our proposed authentication decision procedure in the wireless communication system from the practical perspective. There are two key stages: (1) the fine sensing stage; (2) the fast validation stage. In the fine sensing stage, amplitude (resp. phase) values for the MC estimates are utilized to calculate the asymptotic distribution of the corresponding kernel-based test statistic under the hypothesis $H_0$, defined as a normal distribution with mean ${\mu }_{a,0}$ (resp. ${\mu }_{\mathsf{\Psi },0}$) and variance ${\sigma }_{a,0}^2$ (resp. ${\sigma }_{\mathsf{\Psi },0}^2$) to determine the amplitude decision threshold ${\gamma }_a$ (the phase decision threshold ${\gamma }_{\mathsf{\Psi }}$). In the fast validation stage, the measured values of the kernel-based test statistic in (40) and (41) for the extracted MC in terms of the amplitude and phase values are compared with ${\gamma }_a$ and ${\gamma }_{\mathsf{\Psi }}$, respectively, to determine the origin of the current signal frame.

The uniqueness of MC-based fingerprints can be limited among devices that share identical antenna designs or manufacturing processes. A sophisticated adversary might replicate similar MC patterns using comparable hardware configurations, posing a spoofing risk. To address this, future enhancements may include combining MC features with other device-specific signatures (e.g., gain/phase distortions) or employing randomized challenge–response protocols to improve resilience against impersonation attacks.

4. Performance Analysis

In this section, we first derive the false alarm probability $P_f$ and detection probability $P_d$ for the proposed authentication scheme by applying the theory in [36], and calculate the two decision thresholds for a constant false alarm probability. Then, the mean value and the variance in the asymptotic distribution of the kernel-based test statistic are derived theoretically for the amplitude variable following the generalized Gaussian distribution, and the phase variable following Laplacian distribution scenarios.

4.1. Representation for $P_f$ and $P_d$

Let $P_{f,a}$ and $P_{f,\mathsf{\Psi }}$ be the false alarm probabilities exploiting the amplitude elements and phase in MC, respectively, and $P_{d,a}$ and $P_{d,\mathsf{\Psi }}$ be the detection probabilities exploiting the amplitude elements and phase in MC, respectively. Based on the result for the asymptotic distribution of the kernel-based test statistic in (37)–(39), $P_{f,x}$ and $P_{d,x}$ for $x=\{a,\mathsf{\Psi }\}$ can be expressed, respectively, by

$$\begin{array}{c}\hfill P_{f,x}=1-Q\left({\displaystyle \frac{{\gamma }_x-{\mu }_{x,0}}{{\sigma }_{x,0}^2}}\right),\end{array}$$

(42)

$$\begin{array}{c}\hfill P_{d,x}=1-Q\left({\displaystyle \frac{{\gamma }_x-{\mu }_{x,1}}{{\sigma }_{x,1}^2}}\right).\end{array}$$

(43)

To find the individual thresholds ${\gamma }_a$ and ${\gamma }_{\mathsf{\Psi }}$ corresponding to the proposed authentication using amplitude decision and phase decisions, the Neyman–Pearson criterion is adopted as in [37], where $P_{f,x}$ in (42) is set to a constant in the range of $[0.05,0.1]$ compatible with the authentication standards in [38,39]. Therefore, the threshold ${\gamma }_x$ is determined as

$$\begin{array}{c}\hfill {\gamma }_x=Q^{-1}(1-P_{f,x}){\sigma }_{x,0}^2+{\mu }_{x,0},x=\{a,\mathsf{\Psi }\}.\end{array}$$

(44)

In this paper, we adopt the AND rule to make a final decision for identity validation. Thus, combining (42) and (43), $P_f$ and $P_d$ are written, respectively, as

$$\begin{array}{cc}\hfill P_f& =P_{f,a}{P}_{f,\mathsf{\Psi }}\hfill \\ \hfill & =\left[1-Q\left({\displaystyle \frac{{\gamma }_a-{\mu }_{a,0}}{{\sigma }_{a,0}^2}}\right)\right]\left[1-Q\left({\displaystyle \frac{{\gamma }_{\mathsf{\Psi }}-{\mu }_{\mathsf{\Psi },0}}{{\sigma }_{\mathsf{\Psi },0}^2}}\right)\right],\hfill \end{array}$$

(45)

$$\begin{array}{cc}\hfill P_d& =P_{d,a}{P}_{d,\mathsf{\Psi }}\hfill \\ \hfill & =\left[1-Q\left({\displaystyle \frac{{\gamma }_a-{\mu }_{a,1}}{{\sigma }_{a,1}^2}}\right)\right]\left[1-Q\left({\displaystyle \frac{{\gamma }_{\mathsf{\Psi }}-{\mu }_{\mathsf{\Psi },1}}{{\sigma }_{\mathsf{\Psi },1}^2}}\right)\right].\hfill \end{array}$$

(46)

4.2. Representation for ${\mu }_{a,\ell }$ and ${\sigma }_{a,\ell }^2$

There are many bounded kernel functions such as Gaussian and Laplacian kernels, as listed in

Table 1. Bounded kernel functions.

Kernel Name	Expression
Gaussian	$\mathsf{\Omega }({\mathbf{x}}_i,{\mathbf{x}}_j)=exp\left(-{\displaystyle \frac{\parallel {\mathbf{x}}_i-{\mathbf{x}}_j{\parallel }^2}{2{\eta }^2}}\right)$
Laplacian	$\mathsf{\Omega }({\mathbf{x}}_i,{\mathbf{x}}_j)=exp\left(-{\displaystyle \frac{\parallel {\mathbf{x}}_i-{\mathbf{x}}_j\parallel }{2{\eta }^2}}\right)$

[40], and in this paper we examine the bounded Gaussian kernel function for performance analysis and simulations of the kernel-based authentication scheme, as this kernel function is adopted in much of the relevant literature [40]. To express $P_f$ in (45) and $P_d$ in (46) for the proposed kernel-based authentication scheme jointly utilizing 2-dimension MC features in terms of amplitude and phase, it needs to derive the theoretical values of ${\mu }_{a,\ell }$, ${\mu }_{\mathsf{\Psi },\ell }$, ${\sigma }_{a,\ell }^2$ and ${\sigma }_{\mathsf{\Psi },\ell }^2$ in (38) and (39) corresponding to the considered amplitude and phase distributions.

We first consider ${\mu }_{a,\ell }$ and ${\sigma }_{a,\ell }^2$. Under $H_0$, the amplitude estimate of MC is viewed as a linear combination of the generalized Gaussian distributed random variable (true amplitude) and Gaussian distributed random variable (estimate error). Applying these considerations in (38), the mean value of the test statistics $T_a$ in (40) under $H_0$ is developed as (48) at the top of the next page, where $\left(a\right)$ is obtained by employing integral of exponential functions in ([41] p. 108) to calculate ${\mathbb{E}}_{U_i,U_j}\{\cdots \}$ and $\left(b\right)$ is achieved by changing variables ($v_i\to \sqrt{2(2{\sigma }_{ϵ}^2+{\eta }^2)}x$ and $v_j\to \sqrt{2(2{\sigma }_{ϵ}^2+{\eta }^2)}y$ ), and $f(x_{i_1},y_{i_2})$ is written as

$$\begin{array}{cc}\hfill f(x_{i_1},y_{i_2})& =exp\left(2x_{i_1}{y}_{i_2}\right)\hfill \\ \hfill & \times exp\left(-{\displaystyle \frac{\sqrt{2(2{\sigma }_{ϵ}^2+{\eta }^2)}\left(|x_{i_1}{|}_a^{\beta }+{\left|y_{i_2}\right|}_a^{\beta }\right)}{{\alpha }_a^{{\beta }_a}}}\right).\hfill \end{array}$$

(47)

$$\begin{array}{cc}\hfill {\mu }_{a,0}& ={\mathbb{E}}_{X_i,X_j}\left\{exp\left(-{\displaystyle \frac{{(x_i-x_j)}^2}{2{\eta }^2}}\right)\right\}{\mathbb{E}}_{U_i,U_j,V_i,V_j}\left\{exp\left(-{\displaystyle \frac{{(u_i-u_j)}^2+{(v_i-v_j)}^2}{2{\eta }^2}}\right)\right\}\hfill \\ \hfill & ={\mathbb{E}}_{V_i,V_j}\left\{exp\left(-{\displaystyle \frac{{(v_i-v_j)}^2}{2{\eta }^2}}\right){\mathbb{E}}_{U_i,U_j}\left\{exp\left(-{\displaystyle \frac{{(u_i-u_j)}^2+2(u_i-u_j)(v_i-v_j)}{2{\eta }^2}}\right)\right\}\right\}\hfill \\ \hfill & \stackrel{\left(a\right)}{=}{\mathbb{E}}_{V_i,V_j}\left\{exp\left(-{\displaystyle \frac{{(v_i-v_j)}^2}{2{\eta }^2}}\right){\displaystyle \frac{\eta exp\left(\frac{{\sigma }_{ϵ}^2{(v_i-v_j)}^2}{2{\sigma }_{ϵ}^2{\eta }^2+{\eta }^4}\right)}{\sqrt{2{\sigma }_{ϵ}^2+{\eta }^2}}}\right\}\hfill \\ \hfill & \stackrel{\left(b\right)}{=}{\mathbb{E}}_{X,Y}\left\{2\eta \sqrt{2{\sigma }_{ϵ}^2+{\eta }^2}exp\left(2xy-x^2-y^2\right)\right\}\hfill \\ \hfill & \approx {\displaystyle \frac{\eta {\beta }_a^2\sqrt{2{\sigma }_{ϵ}^2+{\eta }^2}}{2{\alpha }_a^2{\mathsf{\Gamma }}^2\left(\frac{1}{{\beta }_a}\right)}}\sum _{i_1}^{N_1}\sum _{i_2}^{N_2}{W}_{i_1}{W}_{i_2}f(x_{i_1},y_{i_2}).\hfill \end{array}$$

(48)

Based on ${\mu }_{a,0}$, the variance in the test statistics in (40) under $H_0$ is developed as (50), which is obtained by using variable changes ($v_j\to \sqrt{({\sigma }_{ϵ}^2+{\eta }^2)}y$, and $u_j\to \sqrt{{\displaystyle \frac{2{\sigma }_{ϵ}^2({\sigma }_{ϵ}^2+{\eta }^2)}{3{\sigma }_{ϵ}^2+{\eta }^2}}}z$), and $f(x_{i_1},y_{i_3},z_{i_4})$ is expressed by

$$\begin{array}{cc}\hfill & f(x_{i_1},y_{i_3},z_{i_4})=exp\left(-\sqrt{{\displaystyle \frac{4{\sigma }_{ϵ}^2}{3{\sigma }_{ϵ}^2+{\eta }^2}}}{x}_{i_1}{z}_{i_4}\right)\hfill \\ \hfill & \times exp(-\sqrt{2}{x}_{i_1}{y}_{i_3})exp\left(-{\displaystyle \frac{\left(\right|x_{i_1}{\left|\sqrt{2({\sigma }_{ϵ}^2+{\eta }^2)}\right)}^{{\beta }_a}}{{\alpha }_a^{{\beta }_a}}}\right).\hfill \end{array}$$

(49)

$$\begin{array}{cc}\hfill {\sigma }_{a,0}^2& ={\displaystyle \frac{4}{K{M}_1}}\left[{\displaystyle \frac{{\beta }_a^3{\eta }^2({\sigma }_{ϵ}^2+{\eta }^2)}{4{\alpha }_a^3{\mathsf{\Gamma }}^3\left(\frac{1}{{\beta }_a}\right)\sqrt{\pi (3{\sigma }_{ϵ}^2+{\eta }^2)}}}\sum _{i_1}^{N_1}\sum _{i_2}^{N_2}{W}_{i_1}{W}_{i_2}\sum _{i_3}^{N_3}\sum _{i_4}^{N_4}{W}_{i_3}{W}_{i_4}f(x_{i_1},y_{i_3},z_{i_4})f(x_{i_2},y_{i_3},z_{i_4})\right.\hfill \\ \hfill & \left.\times exp\left(-\sqrt{{\displaystyle \frac{2{\sigma }_{ϵ}^2}{3{\sigma }_{ϵ}^2+{\eta }^2}}}{y}_{i_3}{z}_{i_4}\right)exp\left(-{\displaystyle \frac{\left(\right|y_{i_3}{\left|\sqrt{{\sigma }_{ϵ}^2+{\eta }^2}\right)}^{{\beta }_a}}{{\alpha }_a^{{\beta }_a}}}\right)-{\mu }_{a,0}^2\right].\hfill \end{array}$$

(50)

Similarly, we can derive the mean value ${\mu }_{a,1}$ and the variance ${\sigma }_{a,1}^2$ in the test statistics $T_a$ in (40) under hypothesis $H_1$. Since the offset data $a_{\mathrm{T},i}^e$ and estimate error $a_{\mathrm{T},i}^{ϵ}$ follow Gaussian distribution, we can obtain ${\mu }_{a,1}$ and ${\sigma }_{a,1}^2$ by replacing ${\sigma }_{ϵ}^2$ with ${\sigma }_{ϵ}^2+{\sigma }_e^2$ in (48) and (50).

To obtain closed-form analytical expressions for ${\mu }_{a,\ell }$ and ${\sigma }_{a,\ell }^2$, we consider the amplitude estimate samples, i.e., (${\widehat{a}}_{\mathrm{T},i}$, ${\widehat{a}}_{\mathrm{T},j}$) follow real Gaussian distribution with zero mean and variance ${\sigma }_{\widehat{a}}^2$ where ${\sigma }_{\widehat{a}}^2$ is the summation in the amplitude and the estimate error variances under the hypothesis $H_0$, that is, ${\sigma }_{\widehat{a}}^2={\sigma }_a^2+{\sigma }_{ϵ}^2$, while under the alternative hypothesis $H_1$, ${\sigma }_{\widehat{a}}^2$ is the summation of the amplitude, the estimate error and offset data variances under the hypothesis $H_1$, i.e., ${\sigma }_{\widehat{a}}^2={\sigma }_a^2+{\sigma }_{ϵ}^2+{\sigma }_e^2$. Exploiting the integral of exponential functions in [41] (p. 108), ${\mu }_{a,\ell }$ for $\ell =0,1$ is expressed as

$$\begin{array}{c}\hfill {\mu }_{a,\ell }={\displaystyle \frac{\eta }{\sqrt{2{\sigma }_{\widehat{a}}^2+{\eta }^2}}}.\end{array}$$

(51)

Following a similar procedure conducted in (52), the ${\sigma }_{a,\ell }^2$ defined in (39) for the two hypotheses is written as

$$\begin{array}{c}\hfill {\sigma }_{a,\ell }^2={\displaystyle \frac{4}{K{M}_1}}\left[{\displaystyle \frac{{\eta }^2}{\sqrt{3{\sigma }_{\widehat{a}}^4+4{\sigma }_{\widehat{a}}^2{\eta }^2+{\eta }^4}}}-{\displaystyle \frac{{\eta }^2}{2{\sigma }_{\widehat{a}}^2+{\eta }^2}}\right].\end{array}$$

(52)

4.3. Representation for ${\mu }_{\mathsf{\Psi },\ell }$ and ${\sigma }_{\mathsf{\Psi },\ell }^2$

Now, we derive the mean value ${\mu }_{\mathsf{\Psi },\ell }$ and variance value of ${\sigma }_{\mathsf{\Psi },\ell }^2$ of the test statistics $T_{\mathsf{\Psi }}$ in (41). Under $H_0$, the distribution of the extracted phase samples will be a linear combination of Gaussian error random variable and Laplacian phase random variable in (5). Under this case, the mean value of the test statistics $T_{\mathsf{\Psi }}$ under $H_0$ with the Laplacian phase assumption ${\mu }_{\mathsf{\Psi },0}$, is written by using the Laplacian distribution in ((48), (a)) as (53).

$$\begin{array}{cc}\hfill {\mu }_{\mathsf{\Psi },0}& ={\displaystyle \frac{\eta \sqrt{2}\pi exp\left(\frac{2{\sigma }_{\mathsf{\Psi },ϵ}^2+{\eta }^2}{{\sigma }_{\mathsf{\Psi }}^2}\right)}{{\sigma }_{\mathsf{\Psi }}^2}}\left[\sqrt{{\displaystyle \frac{2{\sigma }_{\mathsf{\Psi },ϵ}^2+{\eta }^2}{\pi }}}exp\left(-{\displaystyle \frac{2{\sigma }_{\mathsf{\Psi },ϵ}^2+{\eta }^2}{{\sigma }_{\mathsf{\Psi }}^2}}\right)+\left({\displaystyle \frac{{\sigma }_{\mathsf{\Psi }}}{2}}-{\displaystyle \frac{2{\sigma }_{\mathsf{\Psi },ϵ}^2+{\eta }^2}{{\sigma }_{\mathsf{\Psi }}}}\right)\mathrm{erfc}\left({\displaystyle \frac{\sqrt{2{\sigma }_{\mathsf{\Psi },ϵ}^2+{\eta }^2}}{{\sigma }_{\mathsf{\Psi }}}}\right)\right].\hfill \end{array}$$

(53)

Based on (39) and following a similar manner to that adopted in (53), the variance in the test statistics $T_{\mathsf{\Psi }}$ in (41) with the Laplacian phase scenario for the hypothesis $H_1$ is obtained as (56), where $f_X\left(x_i\right)$ is the PDF of the linear combination of Gaussian and Laplacian distributions defined as [42]

$$\begin{array}{cc}\hfill f_X\left(x_i\right)& ={\displaystyle \frac{1}{\sqrt{8}{\sigma }_{\mathsf{\Psi }}}}exp\left({\displaystyle \frac{{\sigma }_{\mathsf{\Psi },ϵ}^2}{{\sigma }_{\mathsf{\Psi }}^2}}\right)\left[exp\left(-{\displaystyle \frac{\sqrt{2}{x}_i}{{\sigma }_{\mathsf{\Psi }}}}\right)\right.\hfill \\ \hfill & \times \mathrm{efrc}\left({\displaystyle \frac{{\sigma }_{\mathsf{\Psi },ϵ}}{{\sigma }_{\mathsf{\Psi }}}}-{\displaystyle \frac{x_i}{\sqrt{2{\sigma }_{\mathsf{\Psi },ϵ}}}}\right)\hfill \\ \hfill & \left.+exp\left({\displaystyle \frac{\sqrt{2}{x}_i}{{\sigma }_{\mathsf{\Psi }}}}\right)\mathrm{efrc}\left({\displaystyle \frac{{\sigma }_{\mathsf{\Psi },ϵ}}{{\sigma }_{\mathsf{\Psi }}}}+{\displaystyle \frac{x_i}{\sqrt{2{\sigma }_{\mathsf{\Psi },ϵ}}}}\right)\right],\hfill \end{array}$$

(54)

and $\mathcal{M}\left(x_i\right)$ is expressed, respectively, as

$$\begin{array}{c}\hfill \mathcal{M}\left(x_i\right)=exp\left({\displaystyle \frac{\sqrt{2}{x}_i}{{\sigma }_{\mathsf{\Psi }}}}\right)\mathrm{erfc}\left({\displaystyle \frac{x_i}{\sqrt{2({\sigma }_{\mathsf{\Psi },ϵ}^2+{\eta }^2)}}}+{\displaystyle \frac{\sqrt{({\sigma }_{\mathsf{\Psi },ϵ}^2+{\eta }^2)}}{{\sigma }_{\mathsf{\Psi }}}}\right).\end{array}$$

(55)

$$\begin{array}{cc}\hfill {\sigma }_{\mathsf{\Psi },0}^2& ={\displaystyle \frac{{\eta }^2\pi exp\left(\frac{2({\sigma }_{\mathsf{\Psi },ϵ}^2+{\eta }^2)}{{\sigma }_{\mathsf{\Psi }}^2}\right)}{K{M}_1{\sigma }_{\mathsf{\Psi }}^2}}{\int }_{-\infty }^{\infty }{\left[\mathcal{M}\left(x_i\right)+\mathcal{M}(-x_i)\right]}^2{f}_X\left(x_i\right)\mathrm{d}{x}_i-{\displaystyle \frac{4{\mu }_{\mathsf{\Psi },0}^2}{K{M}_1}}.\hfill \end{array}$$

(56)

Substituting (54) and (55) into (56) and applying the approximation $\mathrm{erfc}\left(x\right)\approx \alpha e^{-x^2}$, the variance in the test statistics $T_{\mathsf{\Psi }}$ in (41) with the Laplacian phase scenario for the hypothesis $H_1$ is determined as in (57), where $D_1$–$D_6$ are given in Appendix A.

$$\begin{array}{cc}\hfill {\sigma }_{\mathsf{\Psi },0}^2& ={\displaystyle \frac{{\eta }^2\pi exp\left(-\frac{(3{\sigma }_{\mathsf{\Psi },ϵ}^2+2{\eta }^2)}{{\sigma }_{\mathsf{\Psi }}^2}\right)}{K{M}_1\sqrt{8}{\sigma }_{\mathsf{\Psi }}^3}}\left[\sqrt{\pi ({\sigma }_{\mathsf{\Psi },ϵ}^2+{\eta }^2)}exp\left(-{\displaystyle \frac{3({\sigma }_{\mathsf{\Psi },ϵ}^2+{\eta }^2)}{2{\sigma }_{\mathsf{\Psi }}^2}}\right)\left(D_6^2+(D_4^2-D_6^2)\mathrm{erf}\left({\displaystyle \frac{3\sqrt{{\sigma }_{\mathsf{\Psi },ϵ}^2+{\eta }^2}}{\sqrt{2}{\sigma }_{\mathsf{\Psi }}}}\right)\right.\right.\hfill \\ \hfill & \left.-D_4^2\mathrm{erf}\left({\displaystyle \frac{3{\sigma }_{\mathsf{\Psi },ϵ}^2+{\eta }^2}{\sqrt{2}{\sigma }_{\mathsf{\Psi }}\sqrt{{\sigma }_{\mathsf{\Psi },ϵ}^2+{\eta }^2}}}\right)\right)+{\sigma }_{\mathsf{\Psi },ϵ}\sqrt{{\displaystyle \frac{\pi ({\sigma }_{\mathsf{\Psi },ϵ}^2+{\eta }^2)}{2(3{\sigma }_{\mathsf{\Psi },ϵ}^2+{\eta }^2)}}}exp\left(-{\displaystyle \frac{3{\sigma }_{\mathsf{\Psi },ϵ}^2+2{\eta }^2}{{\sigma }_{\mathsf{\Psi }}^2}}\right)\hfill \\ \hfill & \times \left(D_5{D}_6^2+(D_3{D}_4^2-D_5{D}_6^2)\mathrm{erf}\left({\displaystyle \frac{\sqrt{3{\sigma }_{\mathsf{\Psi },ϵ}^4+4{\sigma }_{\mathsf{\Psi },ϵ}^2{\eta }^2+{\eta }^4}}{{\sigma }_{\mathsf{\Psi }}^2}}\right)+\left(D_1{D}_2^2-D_3{D}_4^2\right)\mathrm{erf}\left({\sigma }_{\mathsf{\Psi },ϵ}{\displaystyle \frac{\sqrt{3{\sigma }_{\mathsf{\Psi },ϵ}^2+{\eta }^2}}{{\sigma }_{\mathsf{\Psi }}\sqrt{{\sigma }_{\mathsf{\Psi },ϵ}^2+{\eta }^2}}}\right)\right)\hfill \\ \hfill & +D_5{D}_6\sqrt{8\pi {\sigma }_{\mathsf{\Psi },ϵ}^2}\sqrt{{\displaystyle \frac{{\sigma }_{\mathsf{\Psi },ϵ}^2+{\eta }^2}{2{\sigma }_{\mathsf{\Psi },ϵ}^2+{\eta }^2}}}exp\left(-{\displaystyle \frac{3{\sigma }_{\mathsf{\Psi },ϵ}^4+3{\sigma }_{\mathsf{\Psi },ϵ}^2{\eta }^2+{\eta }^4}{{\sigma }_{\mathsf{\Psi }}^2(2{\sigma }_{\mathsf{\Psi },ϵ}^2+{\eta }^2)}}\right)\mathrm{erfc}\left({\displaystyle \frac{3{\sigma }_{\mathsf{\Psi },ϵ}^4+4{\sigma }_{\mathsf{\Psi },ϵ}^2{\eta }^2+{\eta }^4}{{\sigma }_{\mathsf{\Psi },ϵ}{\sigma }_{\mathsf{\Psi }}\sqrt{2{\sigma }_{\mathsf{\Psi },ϵ}^4+3{\sigma }_{\mathsf{\Psi },ϵ}^2{\eta }^2+{\eta }^4}}}\right)\hfill \\ \hfill & +{\displaystyle \frac{8{\sigma }_{\mathsf{\Psi }}}{3\sqrt{2}}}exp\left(-{\displaystyle \frac{6({\sigma }_{\mathsf{\Psi },ϵ}^2+{\eta }^2)}{{\sigma }_{\mathsf{\Psi }}^2}}\right)+D_5\sqrt{8\pi {\sigma }_{\mathsf{\Psi },ϵ}}exp\left({\displaystyle \frac{3{\sigma }_{\mathsf{\Psi },ϵ}^2}{{\sigma }_{\mathsf{\Psi }}^2}}\right)\mathrm{erfc}\left({\displaystyle \frac{2{\sigma }_{\mathsf{\Psi },ϵ}}{{\sigma }_{\mathsf{\Psi }}}}+{\displaystyle \frac{{\sigma }_{\mathsf{\Psi },ϵ}^2+{\eta }^2}{{\sigma }_{\mathsf{\Psi }}{\sigma }_{\mathsf{\Psi },ϵ}}}\right)\hfill \\ \hfill & \left.+8D_6\sqrt{{\displaystyle \frac{\pi }{2}}({\sigma }_{\mathsf{\Psi },ϵ}^2+{\eta }^2)}exp\left({\displaystyle \frac{3({\sigma }_{\mathsf{\Psi },ϵ}^2+{\eta }^2)}{{\sigma }_{\mathsf{\Psi }}^2}}\right)\mathrm{erfc}\left({\displaystyle \frac{3\sqrt{{\sigma }_{\mathsf{\Psi },ϵ}^2+{\eta }^2}}{{\sigma }_{\mathsf{\Psi }}}}\right)\right]-{\displaystyle \frac{4{\mu }_{\mathsf{\Psi },0}^2}{K{M}_1}}.\hfill \end{array}$$

(57)

Similarly, we can derive the mean value ${\mu }_{\mathsf{\Psi },0}$ and the variance ${\sigma }_{\mathsf{\Psi },0}^2$ in the test statistics $T_{\mathsf{\Psi }}$ in (41) under hypothesis $H_1$. Since the offset data ${\mathsf{\Psi }}_{\mathrm{T},i}^e$ and estimate error ${\mathsf{\Psi }}_{\mathrm{T},i}^{ϵ}$ follow Gaussian distribution, the phase is also a sum of Gaussian and Laplacian distributions and thus we can obtain ${\mu }_{\mathsf{\Psi },1}$ and ${\sigma }_{\mathsf{\Psi },1}^2$ by replacing ${\sigma }_{\mathsf{\Psi },ϵ}^2$ with ${\sigma }_{\mathsf{\Psi },ϵ}^2+{\sigma }_{\mathsf{\Psi },e}^2$ in (53) and (57).

5. Numerical Results

5.1. Parameter Settings

Monte Carlo simulations are employed to evaluate the performance of the proposed authentication scheme. Unless specified otherwise, the system parameters are set as follows: $M=10,N=8,K=128,$ and $L=3$ propagation paths, with angles $(\theta =[{25}^{\circ },{45}^{\circ },{65}^{\circ }])$ and $(\varphi =[{20}^{\circ },{50}^{\circ },{80}^{\circ }])$, respectively. The operating frequency is set to 30 GHz, with a bandwidth of 1 GHz. To characterize the differences in fingerprints between the legitimate BS (Alice) and the attacker (Eve), the standard deviations for the MC amplitude are set as ${\sigma }_a=0.6$ for Alice and ${\sigma }_e=0.1$ for Eve. The standard deviations for the MC phase are set as ${\sigma }_{\mathsf{\Psi }}=1.5$ for Alice and ${\sigma }_{\mathsf{\Psi },e}=1.8$ for Eve. To assess the average authentication performance, specifically the detection ($P_d$) and the false alarm probabilities ($P_f$), a total of 10,000 simulations are conducted. Also, the root mean square error is adopted as the metric to evaluate the feature extraction of the MC fingerprint.

5.2. Performance Illustration of Feature Extraction

Figure 4 illustrates the relationship between the root mean square error (RMSE) of MC and the signal-to-noise Ratio (SNR), which is used to assess the extraction accuracy of the MC features under the settings of ($M=10,N=8,K=128,$ and $L=3$). It is clearly demonstrated from Figure 4 that the RMSE for both the real and imaginary parts of the MC features gradually decreases as the SNR improves, indicating an improvement in feature extraction precision. Note that a more accurate feature extraction allows for a clearer distinction between legitimate users and attackers, which is crucial for improving the reliability and accuracy of identity verification.

It should be noted that mutual coupling (MC) characteristics are influenced by antenna geometry, inter-element spacing, and the surrounding electromagnetic environment. In practical deployments, factors such as nearby materials, user mobility, and platform-induced scattering may introduce temporal variations in MC, which could impact the stability of authentication performance. Future work will focus on evaluating the robustness of the proposed scheme under such dynamic conditions through over-the-air (OTA) experiments utilizing programmable antenna arrays and software-defined radio (SDR) platforms.

Figure 5 and Figure 6 present the pseudospectra for DoA and DoD estimation, respectively. In both figures, we observe that each of the three largest peaks corresponds accurately to the true values of the angles $\theta$ and $\varphi$, respectively. Importantly, no false peaks are observed, indicating that the estimation method is highly reliable. Accurate identification of the DoA and DoD is crucial for the successful decoupling of the MC features. By precisely estimating the arrival and departure angles, the MC features can be more effectively isolated and analyzed, leading to better performance in feature extraction and, consequently, in the overall authentication process. This high level of accuracy ensures that the system can distinguish between different signal sources and refine its decision-making process in real-time authentication applications.

5.3. Model Validation

To validate the accuracy of the proposed theoretical framework, in Figure 7, we compare the simulation results with the theoretical models under different SNR levels of $\{0,5,10\}$ dB. It is observed that the simulation results closely match the theoretical curves derived from Equations (45) and (46). This alignment between theory and simulation validates the correctness of the proposed theoretical performance metric models. The close match between the theoretical and simulation results indicates the reliability and accuracy of the derived analytical expressions for the kernelized detector scheme. This further implies that the established theoretical models can be effectively applied for precise performance characterization. Additionally, we observe that with the increase in SNR, higher $P_d$ can be achieved under the same $P_f$ constraint. For instance, given a fixed $P_f=0.1$, $P_d$ under three SNR conditions are 0.73, 0.84, and 0.91, respectively.

5.4. Impacts of System Parameters on Authentication Performance

We first illustrate in Figure 8 the impact of frame length K on the proposed scheme. As shown in Figure 8, it is observed that with the increase in K, the authentication performance improves, demonstrating an upward trend in $P_d$. However, we also note that the performance gain reaches a saturation point as K increases further. For instance, when $P_f=0.1$, $P_d$ under $K=\{32,64,128\}$ are 0.59, 0.66, and 0.67, respectively. This indicates that increasing K improves the detection rate, but the performance gain diminishes as K approaches a certain value. In addition, further increases in frame length yield minimal improvement in detection accuracy. Considering the balance between authentication performance and system efficiency, an appropriate $K=64$ is selected to avoid unnecessary communication overhead.

Then, in Figure 9, we assess the impact of fingerprint similarity between legitimate users and attackers on the detection performance of the proposed authentication scheme under the settings of SNR = {0, 5, 10} dB. The parameter ${\sigma }_{\mathsf{\Psi }}$ varies from 1 to 1.8, with ${\sigma }_{\mathsf{\Psi }}=1.8$ representing the worst-case scenario. As shown in Figure 9, $P_d$ exhibits a decreasing tendency as ${\sigma }_{\mathsf{\Psi }}$ increases. Despite this, even in the worst-case scenario, the proposed scheme maintains a detection probability of 0.68, 0.84, and 0.90 at ${\sigma }_{\mathsf{\Psi }}=1.8$ for SNR of 0 dB, 5 dB, and 10 dB, respectively. These results demonstrate the robustness of the proposed scheme, which can be attributed to the amplitude of the MC feature to maintain reliable identity authentication. This additional security layer ensures that the authentication performance remains acceptable under challenging conditions.

5.5. Performance Comparison

In this subsection, we present a performance comparison among the proposed authentication scheme using only the phase of MC, and the proposed authentication scheme using both the amplitude and phase of MC feature under two different SNR conditions: 0 dB and 5 dB. As shown in Figure 10, the authentication scheme that utilizes both amplitude and phase features of MC performs significantly better than the scheme using only the phase, especially under lower SNR conditions. For example, given a $P_f=0.2$, $P_d$ of the two feature-based scheme is 0.75 while $P_d$ of the phase-based scheme is 0.66 under SNR = 0 dB. For SNR = 5 dB, the $P_d$ values corresponding to the two cases are 0.86 and 0.82, respectively. The results suggest that exploiting both amplitude and phase features significantly enhances the detection accuracy, as it provides a more comprehensive analysis of the received signals, thereby improving the overall authentication decision-making process. This comparison underscores the importance of incorporating multiple features from the MC feature, leading to more reliable and accurate authentication in challenging scenarios with varying SNR conditions.

6. Conclusions

In this paper, we proposed a robust authentication scheme that leverages both the amplitude and phase of the MC feature to improve the accuracy and reliability of identity verification. In particular, we exploited the zero-mean generalized Gaussian and Laplacian distributions to model the statistical feature regarding the amplitude and phase of MC feature. Based on the modeled statistical knowledge, we then constructed a kernel-based test statistic to design an amplitude-phase based identity authentication scheme. Moreover, we also performed a theoretical analysis of the proposed scheme by deriving the analytical expressions of detection and false alarm probabilities. The results demonstrate that: (1) our theoretical models effectively depict authentication performance in terms of false alarm and detection probabilities; (2) incorporating both amplitude and phase features significantly enhances detection accuracy; (3) the proposed scheme remains robust across various SNR levels, ensuring reliable authentication even in a noise-interference environment. We hope the proposed scheme can hold promise as an effective solution to mitigate identity-based impersonation attacks in mmWave communication systems.

While the proposed framework demonstrates promising authentication capabilities using MC features, several real-world considerations must be addressed before deployment. First, the current evaluation is based on simulations and theoretical modeling; future work will involve experimental validation using SDR-based prototypes to assess robustness under realistic hardware and environmental conditions. Second, the susceptibility of MC profiles to hardware degradation such as antenna failures, RF front-end malfunctions, or calibration drift, can result in inconsistent fingerprints and degraded authentication performance. Third, physical changes in the environment (e.g., moving objects, weather fluctuations, or platform vibrations) may alter the electromagnetic coupling between antenna elements. These factors highlight the need for adaptive re-learning schemes, robustness-aware thresholds, and possibly the fusion of MC with other stable hardware or channel-based features to enhance resilience. Exploring these directions will be the focus of our future research.