Bilevel Optimization for ISAC Systems with Proactive Eavesdropping Capabilities

Abstract

Integrated sensing and communication (ISAC) has attracted extensive attention as a key technology to improve spectrum utilization and system performance for future wireless sensor networks. At the same time, active surveillance, as a legitimate means of surveillance, can improve the success rate of surveillance by sending interference signals to suspicious receivers, which is important for crime prevention and public safety. In this paper, we investigate the joint optimization of performance of both ISAC and active surveillance. Specifically, we formulate a bilevel optimization problem where the upper-level objective aims to maximize the probability of successful eavesdropping while the lower-level objective aims to optimize the localization performance of the radar on suspicious transmitters. By employing the Rayleigh quotient, introducing a decoupling strategy, and adding penalty terms, we propose an algorithm to solve the bilevel problem where the lower-level objective is convex. With the help of the proposed algorithm, we obtain the optimal solution of the analog transmit beamforming matrix and the digital beamforming vector. Performance analysis and discussion of key insights, such as the trade-off between eavesdropping success probability and radar localization accuracy, are also provided. Finally, comprehensive simulation results validate the effectiveness of our proposed algorithm in enhancing both the eavesdropping success probability and the accuracy of radar localization.

1. Introduction

The field of integrated sensing and communication (ISAC) marks a major advancement in wireless technology, combining wireless communication and radar sensing capabilities to enhance spectral efficiency and improve spatial–temporal resolution of signals. This integration is particularly crucial with the advent of future 6G technologies, where ISAC plays a pivotal role in high-speed data transmission and precise monitoring, especially in applications such as autonomous driving, wireless sensing and intelligent transportation systems [1]. However, in the realm of wireless communication, the open nature of the wireless medium makes it inherently susceptible to security vulnerabilities. Proactive eavesdropping, a legitimate and strategic form of surveillance, has become an essential capability for government agencies. Unlike passive eavesdropping, which can be limited by geographical factors or channel conditions, proactive eavesdropping involves the eavesdropper actively transmitting interference signals towards the target receiver to enhance the chances of successful interception. This approach holds practical application prospects, such as in vehicles or drones equipped with legal eavesdropping devices [2].

Building upon this, several studies have examined the integration of security measures within ISAC systems, addressing the simultaneous optimization of both communication and sensing functionalities. The authors of ref. [3] explore the use of Reconfigurable Intelligent Surfaces (RISs) to enhance the physical layer security (PLS) of wireless systems with ISAC. The paper discusses how RISs can mitigate threats like eavesdropping by intelligently manipulating the propagation of signals. However, while it emphasizes communication security, the interaction between communication security and radar sensing within ISAC systems still remains unexplored. Additionally, the authors of ref. [4] focus on a secure cell-free ISAC system, where both communication and sensing are subject to eavesdropping. The authors optimize beamforming strategies to ensure the security of both communication and sensing tasks. However, they treat these two objectives separately, making it difficult to efficiently coordinate the resources and balance both goals. In parallel, the authors of ref. [5] investigate a full-duplex ISAC system, where they jointly optimize beamforming and artificial noise to secure communications against eavesdroppers while simultaneously sensing them. However, their defensive framework aims to guarantee secrecy rates, which contrasts with our goal of proactively maximizing the eavesdropping probability. Additionally, the authors of ref. [6] study the integration of communication and sensing in ISAC networks for 6G. While it highlights the challenges of securing both communication and sensing functionalities, it does not provide a comprehensive solution for jointly optimizing both objectives. The authors suggest that while communication security and sensing can complement each other, an optimal framework to manage both simultaneously is still lacking.

Therefore, although these works address various aspects of ISAC security, they generally treat communication and sensing security as independent objectives. This approach may lead to difficulties in effectively balancing the dual goals, especially in mission-critical scenarios where both communication and sensing accuracy are essential. In particular, existing methods fail to efficiently allocate resources between eavesdropping prevention and radar localization, making it challenging to optimize both aspects simultaneously.

To address these limitations, bilevel optimization steps in as a powerful tool for solving multi-objective optimization problems [7]. In recent years, bilevel optimization has been adopted in several communication systems, particularly in the context of ISAC. For example, He et al. [8] utilize the bilevel optimization to optimize communication quality-of-service (QoS) and radar detection simultaneously, with the upper level focusing on communication performance and the lower level handling radar interference. Similarly, in [9], the authors consider an RIS-assisted SWIPT system, where bilevel optimization separates communication and power harvesting into two levels—maximizing information delivery at the upper level and harvested power at the lower level. This approach effectively balances the two objectives, optimizing overall system performance.

Building on these insights, our paper introduces a bilevel optimization approach that effectively coordinates communication security and radar sensing in ISAC systems. Specifically, we propose a system where a full-duplex legitimate monitor, equipped with a large-scale antenna array, uses hybrid beamforming to simultaneously perform proactive eavesdropping and radar sensing. The optimization focuses on maximizing the eavesdropping success probability while ensuring high radar localization accuracy. To achieve this, we first transform the eavesdropping success probability into a deterministic signal-to-interference-plus-noise ratio (SINR) difference and simplify the problem using Rayleigh quotient transformation. We then decouple the optimization problem and introduce penalty terms to handle non-convex components. The solution is derived by optimizing the beamforming vectors for both eavesdropping and radar tasks under certain system constraints. Furthermore, we discuss the simulation results, which demonstrate the effectiveness of our proposed approach in balancing both eavesdropping and radar performances.

We summarize our main contributions of this paper as follows:

•

We present a comprehensive system model for proactive ISAC surveillance, characterizing the distinct signal flows for eavesdropping, radar sensing, and jamming within a unified full-duplex architecture.

•

We formulate a mathematical optimization problem that maximizes the eavesdropping success probability while maintaining radar positioning accuracy. To solve this problem, we introduce a decoupling framework with penalty terms to handle non-convex components, enabling effective bilevel optimization of the problem’s convex objectives.

•

We conduct a detailed analysis of the influence of factors such as noise power and antenna number on both the upper and lower-level objectives. Through simulations, we demonstrate that our bilevel algorithm not only achieves high radar localization accuracy but also maintains a comparable eavesdropping probability to existing benchmark methods.

The rest of this paper is structured as follows. Section 2 presents the system model and formulates the mathematical optimization for integrated surveillance and radar sensing. In Section 3, the optimal algorithm is introduced for dealing with the formulated problem. In Section 4, numerical results are presented to validate the effectiveness of the proposed algorithm. Finally, Section 5 concludes the whole paper.

Notations: In this paper, scalars, vectors, and matrices are denoted by lowercase letters (e.g., x), bold lowercase letters (e.g., $\mathit{x}$), and bold uppercase letters (e.g., $\mathit{X}$), respectively. The operators ${(·)}^{*}$, ${(·)}^T$, and ${(·)}^H$ represent the complex conjugate, transpose, and Hermitian (complex conjugate transpose), respectively. Given a matrix $\mathit{A}$, the notation $∥\mathit{A}∥$ stands for its Frobenius norm. The set ${\mathbb{C}}^{m\times n}$ denotes the space of complex matrices of size $m\times n$, and $\mathit{I}$ stands for the identity matrix. The notation $\mathcal{CN}(\mathit{\mu },\Sigma )$ indicates a circularly symmetric complex Gaussian distribution with mean vector $\mathit{\mu }$ and covariance matrix $\Sigma$. The expectation and probability operators are denoted by $\mathbb{E}\{·\}$ and $\mathcal{P}(·)$, respectively.

2. System Model and Problem Formulation

In this section, we present the system model for wireless legitimate surveillance with radar capabilities and establish the optimization problem.

2.1. System Model

We consider a legitimate surveillance system that includes a suspicious transmitter S, a suspicious receiver D, and a legitimate monitor E as shown in Figure 1 [10]. It is important to note that the term “legitimate monitor” as used in this paper presupposes that the operating entity has been granted the legal authority to conduct such proactive jamming and surveillance activities. An analysis of the specific national statutes and regulatory frameworks that grant this authority—such as 47 U.S.C. § 333 of the Communications Act in the United States [11] or the Regulation of Investigatory Powers Act 2000 in the United Kingdom [12]—is beyond the technical scope of this work. Our focus remains on the signal processing and optimization challenges that arise under the fundamental assumption of a legally sanctioned operation. In this system, the transmitter S and suspicious receiver D are each equipped with a single antenna to transmit and receive the signals. The legitimate monitor E is equipped with N antennas and M radio frequency (RF) chains ($M\ll N$), where N antennas are used for both transmitting and receiving signals. E intends to actively eavesdrop on suspicious transmissions between transmitter S and receiver D. To enhance the eavesdropping performance, E operates in the full-duplex (FD) mode, allowing it to simultaneously eavesdrop on the communication while interfering with the transmission between S and D. This interference is intentionally introduced by E to degrade the communication quality at D, which results in a lower SINR at D and, consequently, improves E’s ability to eavesdrop on the transmission. By reducing D’s SINR, E effectively boosts its own SINR and enhances eavesdropping performance.

Additionally, since the legitimate monitor can make contributions to legitimate monitoring in all kinds of ways, such as tracking suspicious users, detecting potential obstacles, or assisting in communication, we assume that the legitimate monitor operates as a phased-array radar, continuously transmitting detection signals in N directions of 180° during the scanning period. Each of these directions corresponds to one of the N beams generated by E’s radar system. Let ${\mathit{h}}_{se}\in {\mathbb{C}}^{N\times 1},h_{sd}\in {\mathbb{C}}^{1\times 1}$, and ${\mathit{h}}_{ed}\in {\mathbb{C}}^{N\times 1}$ denote the eavesdropping channel, suspicious channel, and interference channel, respectively.

Regarding the transmitting module, we consider two types of signals: the suspicious signal and the interference signal. The suspicious signal is the communication signal transmitted from the suspicious transmitter S to the suspicious receiver D, which E is trying to eavesdrop on, and this signal can be written as ${\tilde{\mathit{U}}}_n^H{\mathit{h}}_{se}\sqrt{p_s}s\left(l\right)$, where $s\left(l\right)$ is the baseband signal transmitted by S, $p_s$ is the transmit power, ${\mathit{h}}_{se}$ is the channel from S to E, and ${\tilde{\mathit{U}}}_n$ is the analog receive beamforming matrix at direction ${\theta }_n$. This is the target signal that E aims to recover during eavesdropping. The interference signal, on the other hand, is intentionally generated by E to degrade D’s reception quality. The legitimate monitor detects N directions consecutively in one scan cycle, probing one direction at a time. One RF chain is used for radar detection in each direction, while the remaining $M-1$ RF chains are used to generate interference to disturb D’s signal reception. Upon detecting the n-th direction, the transmitted signal at angle ${\theta }_n$ can be written as

$$\begin{array}{c}\hfill {\mathit{x}}_n\left(l\right)={\mathit{U}}_n{\mathit{p}}_{n,l}{c}_n\left(l\right),\end{array}$$

(1)

where $c_n\left(l\right)$ refers to the transmitted signal sequence with $\mathbb{E}\left\{\right|c_n\left(l\right){|}^2\}=1$. This signal sequence serves as the common carrier for both the radar detection and interference signals. By sharing the same waveform, E is able to perform both functions simultaneously while efficiently utilizing its limited RF chains. The analog beamforming matrix ${\mathit{U}}_n=[{\mathit{u}}_{n,1},{\mathit{u}}_{n,2},\dots ,{\mathit{u}}_{n,M}]\in {\mathbb{C}}^{N\times M}$ determines the spatial direction of signal transmission, while the digital transmit beamforming vector ${\mathit{p}}_{n,l}={[p_{n,l,1},p_{n,l,2},\dots ,p_{n,l,M}]}^T\in {\mathbb{C}}^{M\times 1}$ adjusts the power and phase of each stream at a specific time index l. The signal index $l=1,2,\dots ,L$ denotes the time slot within a single beamforming direction during a full radar scan cycle. In other words, for every direction ${\theta }_n$, the monitor transmits L symbols before switching to the next. Together, the matrix ${\mathit{U}}_n$ and the potentially time-varying vector ${\mathit{p}}_{n,l}$ form the hybrid beamformer, where ${\mathit{U}}_n$ provides coarse, wideband analog phase shifting and ${\mathit{p}}_{n,l}$ allows for fine-grained digital control.

For simplicity in the subsequent optimization problem, we assume that this digital beamformer remains constant within the transmission block for any given direction n. Thus, we set ${\mathit{p}}_{n,l}={\mathit{p}}_n$ for all $l=1,\dots ,L$, and our goal is to optimize a single, time-invariant digital beamformer ${\mathit{p}}_n$ for each direction.

For the receive module, the received signal includes the signal received by the radar and the signal received by the suspect receiver. The received signal at the legitimate monitor E consists of suspicious signals, radar detection echoes, radar clutter, and noise. The received signal at the legitimate monitor E for the ${\theta }_n$ direction can be expressed as

$$\begin{array}{c}\hfill {\mathit{y}}_e\left(l\right)=\underset{\mathrm{suspicious}\mathrm{signal}}{\underbrace{{\tilde{\mathit{U}}}_n^H{\mathit{h}}_{se}\sqrt{p_s}s\left(l\right)}}+\underset{\mathrm{radar}\mathrm{echo}}{\underbrace{{\tilde{\mathit{U}}}_n^H{\mathit{A}}_n{\mathit{U}}_n{\mathit{p}}_{n,l}{c}_n\left(l\right)}}\\ \hfill +\underset{\mathrm{radar}\mathrm{clutter}}{\underbrace{{\tilde{\mathit{U}}}_n^H\sum _j{\mathit{A}}_j{\mathit{U}}_n{\mathit{p}}_{n,l}{c}_n\left(l\right)}}+\underset{\mathrm{noise}}{\underbrace{n\left(l\right)}},\end{array}$$

(2)

where ${\tilde{\mathit{U}}}_n=[{\tilde{\mathit{u}}}_{n,1},\dots ,{\tilde{\mathit{u}}}_{n,M}]\in {\mathbb{C}}^{N\times M}$ represents the analog receive beamforming matrix, while ${\mathit{U}}_n$ and ${\mathit{p}}_{n,l}$ are the previously defined analog and digital transmit beamformers at E, respectively. Furthermore, $p_s$ is the corresponding transmit power from S, and ${\mathit{A}}_j$ represents the channel matrix for the j-th clutter source. The term $\mathit{n}\left(l\right)\sim \mathcal{CN}(\mathbf{0},{\sigma }^2\mathit{I})$ denotes additive Gaussian white noise (AWGN) with ${\sigma }^2$ representing the noise power. When the number of antennas N is substantial, the radar detection system can achieve high angular resolution, which in turn significantly diminishes the clutter noise to a comparatively minimal level. Additionally, the clutter noise is inherently constrained by the maximum power limit due to the finite transmit power. By applying the central limit theorem, the aggregate effect of the clutter from multiple directions and the noise, denoted by ${\tilde{\mathit{U}}}_n^H{\displaystyle \sum _j}{\mathit{A}}_j{\mathit{U}}_n{\mathit{p}}_{n,l}+\mathit{n}$, can be simply treated as a Gaussian random variable with a variance of ${\tilde{\sigma }}^2$ [13,14].

The received signal at the suspicious receiver D includes suspicious signals, interfering signals from E, and noise. Thus the received signal at D for the ${\theta }_n$ direction can be written as

$$\begin{array}{c}\hfill y_d\left(l\right)=\underset{\mathrm{suspicious}\mathrm{signal}}{\underbrace{h_{sd}\sqrt{p_s}s\left(l\right)}}+\underset{\mathrm{jamming}\mathrm{signal}}{\underbrace{{\mathit{h}}_{ed}^H{\mathit{U}}_n{\mathit{p}}_{n,l}{c}_n\left(l\right)}}+\underset{\mathrm{noise}}{\underbrace{\mathit{n}\left(l\right)}},\end{array}$$

(3)

where $n\sim \mathcal{CN}(0,{\sigma }^2)$ denotes the AWGN, which follows a complex normal distribution with the noise variance being ${\sigma }^2$.

In order to distinguish between suspicious signals and radar echo signals, we have designed vectors ${\tilde{\mathit{w}}}_{s,n}$ and ${\tilde{\mathit{w}}}_{r,n}$ to reside in the null-spaces of ${\tilde{\mathit{U}}}_n^H{\mathit{A}}_{sr,n}{\mathit{U}}_n$ and ${\tilde{\mathit{U}}}_n^H{\mathit{h}}_{se}$, respectively. Thus, we can redefine ${\tilde{\mathit{w}}}_{s,n}\triangleq {\mathit{Z}}_{s,n}{\mathit{w}}_{s,n}$ and ${\tilde{\mathit{w}}}_{r,n}\triangleq {\mathit{Z}}_{r,n}{\mathit{w}}_{r,n}$, where ${\mathit{w}}_{s,n}$ and ${\mathit{w}}_{r,n}\in {\mathbb{C}}^{(M-1)\times 1}$ are the equivalent digital receive beamforming vectors. Additionally, ${\mathit{Z}}_{s,n}$ and ${\mathit{Z}}_{r,n}\in {\mathbb{C}}^{M\times (M-1)}$ are defined as the null spaces of ${\tilde{\mathit{U}}}_n^H{\mathit{A}}_n{\mathit{U}}_n$ and ${\tilde{\mathit{U}}}_n^H{\mathit{h}}_{se}$, respectively. Since the ranks of ${\tilde{\mathit{U}}}_n^H{\mathit{A}}_{sr,n}{\mathit{U}}_n$ and ${\tilde{\mathit{U}}}_n^H{\mathit{h}}_{se}$ are both 1, ${\mathit{Z}}_{s,n}$ and ${\mathit{Z}}_{r,n}$ are always present. This null-space design approach effectively mitigates the primary interference between the radar echo signals and suspicious signals.

With the interference between the radar and communication signals properly mitigated through the null-space projection design, we are now able to quantitatively evaluate the performance of both functionalities. To this end, we introduce multiple SINR expressions that correspond to different receiving roles and tasks within the system. These SINRs reflect the system’s ability to perform eavesdropping, communication disruption, and accurate radar localization simultaneously. Then we can obtain the SINRs of the system, each corresponding to different functionalities.

• ${\mathrm{SINR}}_E$: the SINR at the legitimate monitor E for decoding the suspicious communication signal from S to D. This measures E’s ability to eavesdrop on the signal from S.
• ${\mathrm{SINR}}_D$: the SINR at the suspicious receiver D, which receives the communication from S and interference from E. This reflects the quality of D’s reception, where interference from E degrades D’s signal, improving E’s eavesdropping ability.
• ${\mathrm{SINR}}_{S,\mathrm{radar}}$: the SINR at E when using radar to detect the echo reflected from S. This measures E’s radar performance in detecting S’s position.
• ${\mathrm{SINR}}_{D,\mathrm{radar}}$: the SINR at E when receiving radar echoes from D. This reflects E’s ability to detect and track D’s location, aiding in joint sensing and decision-making.

These four SINRs are written as

$$\begin{array}{cc}\hfill {\mathrm{SINR}}_{\mathrm{E}}& ={\displaystyle \frac{{\displaystyle \sum _{n=1}^N}{∥{\mathit{w}}_{s,n}^H{\mathit{Z}}_{s,n}^H{\tilde{\mathit{U}}}_n^H{\mathit{h}}_{se}∥}^2{p}_s}{{\displaystyle \sum _{n=1}^N}{\tilde{\sigma }}^2{\mathit{w}}_{s,n}^H{\mathit{w}}_{s,n}}},\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill {\mathrm{SINR}}_{\mathrm{D}}& ={\displaystyle \frac{{\displaystyle \sum _{n=1}^N}{∥{\mathit{h}}_{sd,n}∥}^2{p}_s}{{\displaystyle \sum _{n=1}^N}\left({∥{\mathit{h}}_{ed,n}^H{\mathit{U}}_n{\mathit{p}}_n∥}^2\right)+N{\sigma }^2}},\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill {\mathrm{SINR}}_{\mathrm{S},\mathrm{radar}}& ={\displaystyle \frac{{\displaystyle \sum _{n=1}^N}{∥{\mathit{w}}_{s,r,n}^H{\mathit{Z}}_{s,r,n}^H{\tilde{\mathit{U}}}_n^H{\mathit{A}}_{sr,n}{\mathit{U}}_n{\mathit{p}}_n∥}^2}{{\displaystyle \sum _{n=1}^N}{\tilde{\sigma }}^2{\mathit{w}}_{s,r,n}^H{\mathit{w}}_{s,r,n}}},\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill {\mathrm{SINR}}_{\mathrm{D},\mathrm{radar}}& ={\displaystyle \frac{{\displaystyle \sum _{n=1}^N}{∥{\mathit{w}}_{D,r,n}^H{\mathit{Z}}_{D,r,n}^H{\tilde{\mathit{U}}}_n^H{\mathit{A}}_{rd,n}{\mathit{U}}_n{\mathit{p}}_n∥}^2}{{\displaystyle \sum _{n=1}^N}{\tilde{\sigma }}^2{\mathit{w}}_{D,r,n}^H{\mathit{w}}_{D,r,n}}},\hfill \end{array}$$

(7)

where ${\mathit{A}}_{sr}$ denotes the matrix of detection channels between radar and suspicious transmitters, and ${\mathit{A}}_{rd}$ denotes the matrix of detection channel between radar and suspicious receiver. Additionally, ${\mathrm{SIN}\mathrm{R}}_{\mathrm{E}}$ and ${\mathrm{SIN}\mathrm{R}}_{\mathrm{D}}$ are calculated over one scan period.

In our primary model, we assume ideal self-interference (SI) cancellation for the full-duplex monitor E. However, in practical implementations, imperfect analog and digital cancellation would result in residual self-interference (RSI). This RSI manifests as an additional noise-like term at the receiver of E. Consequently, the SINR for the eavesdropping link, ${\mathrm{SIN}\mathrm{R}}_E$, would be modified. Specifically, the power of the RSI, denoted as ${\sigma }_{\mathrm{RSI}}^2$, is added to the noise and clutter term in the denominator.

The structure of our bilevel optimization problem remains intact with this modification, meaning our proposed solution methodology is still applicable. To quantitatively address the level of tolerable RSI, we can directly analyze the condition for successful eavesdropping, ${\mathrm{SINR}}_E>{\mathrm{SINR}}_D$. By incorporating the RSI power, ${\sigma }_{\mathrm{RSI}}^2$, into the denominator of ${\mathrm{SIN}\mathrm{R}}_E$ as defined in (4), this condition becomes     

$${\displaystyle \frac{{\displaystyle \sum _{n=1}^N}{∥{\mathit{w}}_{s,n}^H{\mathit{Z}}_{s,n}^H{\tilde{\mathit{U}}}_n^H{\mathit{h}}_{se}∥}^2{p}_s}{\left({\displaystyle \sum _{n=1}^N}{\tilde{\sigma }}^2{\mathit{w}}_{s,n}^H{\mathit{w}}_{s,n}\right)+{\sigma }_{\mathrm{RSI}}^2}}>{\mathrm{SINR}}_{\mathrm{D}}$$

(8)

Rearranging this inequality allows us to establish a clear upper bound on the tolerable RSI power:

$${\sigma }_{\mathrm{RSI}}^2<{\displaystyle \frac{{\displaystyle \sum _{n=1}^N}{∥{\mathit{w}}_{s,n}^H{\mathit{Z}}_{s,n}^H{\tilde{\mathit{U}}}_n^H{\mathit{h}}_{se}∥}^2{p}_s}{{\mathrm{SINR}}_{\mathrm{D}}}}-{\displaystyle \sum _{n=1}^N}{\tilde{\sigma }}^2{\mathit{w}}_{s,n}^H{\mathit{w}}_{s,n}$$

(9)

This inequality provides a precise, quantitative answer: the tolerable RSI power is not a fixed value but is contingent upon the strength of the eavesdropped signal at E, the target’s link quality (${\mathrm{SINR}}_{\mathrm{D}}$), and the receiver’s original noise floor. Our framework is effective as long as the system’s SI cancellation capability can meet this condition.

2.2. Problem Formulation

The core challenge in this ISAC system is to jointly optimize two coupled objectives: maximizing the probability of successful eavesdropping and ensuring radar localization accuracy. Owing to the inherent hierarchy of these tasks—surveillance being the primary mission, with sensing as a concurrent requirement—we model this problem using a bilevel optimization framework.

In this framework, the upper-level objective aims to maximize the probability of successful eavesdropping, while the lower-level objective is to optimize the localization performance of the radar on suspicious transmitters. This leader–follower structure reflects the practical separation of mission priority and allows each subtask to adapt to the outcome of the other. To provide a clear visual representation of this bilevel interaction, we illustrate the proposed framework in Figure 2.

To evaluate the success of the upper-level objective—legitimate eavesdropping—we adopt the SINR-based decoding criterion established in [15]. Specifically, if ${\mathrm{SINR}}_{\mathrm{E}}<{\mathrm{SINR}}_{\mathrm{D}}$, the legitimate monitor is incapable of error-free information eavesdropping. On the other hand, if ${\mathrm{SINR}}_{\mathrm{E}}>{\mathrm{SINR}}_{\mathrm{D}}$, the legitimate monitor can ensure the reliability of its information decoding. Based on this principle, we define the following indicator function to characterize the occurrence of successful eavesdropping:

$$\begin{array}{c}\hfill Y=\left\{\begin{array}{cc}1,\hfill & {\mathrm{SINR}}_{\mathrm{E}}\ge {\mathrm{SINR}}_{\mathrm{D}},\hfill \\ 0,\hfill & {\mathrm{SINR}}_{\mathrm{E}}<{\mathrm{SINR}}_{\mathrm{D}}.\hfill \end{array}\right.\end{array}$$

(10)

Therefore, the upper-level objective is to maximize the successful eavesdropping probability, denoted as $P_{\mathrm{eav}}=\mathcal{P}({\mathrm{SINR}}_{\mathrm{E}}\ge {\mathrm{SINR}}_{\mathrm{D}})$. While deriving a closed-form expression for $P_{\mathrm{eav}}$ is intractable for our complex system, it is well known that the probability of correct detection (or decoding) is a monotonically increasing function of the received SINR, as detailed in classical detection theory [16,17]. Thus, maximizing the SINR gap, ${\Delta }_{\mathrm{SINR}}={\mathrm{SINR}}_{\mathrm{E}}-{\mathrm{SINR}}_{\mathrm{D}}$, provides a direct and tractable proxy for maximizing the true probability of success. So, the upper-level objective can be written as

$$\underset{{\mathit{U}}_n,{\mathit{p}}_n,{\tilde{\mathit{U}}}_n}{max}{\displaystyle \frac{{\displaystyle \sum _{n=1}^N}{∥{\mathit{w}}_{s,n}^H{\mathit{Z}}_{s,n}^H{\tilde{\mathit{U}}}_n^H{\mathit{h}}_{se}∥}^2{p}_s}{{\displaystyle \sum _{n=1}^N}{\tilde{\sigma }}^2{\mathit{w}}_{s,n}^H{\mathit{w}}_{s,n}}}-{\displaystyle \frac{{\displaystyle \sum _{n=1}^N}{∥{\mathit{h}}_{sd,n}∥}^2{p}_s}{{\displaystyle \sum _{n=1}^N}{∥{\mathit{h}}_{ed,n}^H{\mathit{U}}_n{\mathit{p}}_n∥}^2+N{\sigma }^2}}.$$

(11)

Then, because the lower-level objective is the localization performance of radar at the suspicious transmitter, we can formulate it as the ${\mathrm{SINR}}_{\mathrm{S},\mathrm{radar}}$. Combining the upper-level and the lower-level objectives, and adding some constraints, we can rewrite the bilevel problem as

$$\begin{array}{ccc}\hfill & \underset{{\mathit{U}}_n,{\mathit{p}}_n,{\tilde{\mathit{U}}}_n}{max}& {\displaystyle \frac{{\displaystyle \sum _{n=1}^N}{∥{\mathit{w}}_{s,n}^H{\mathit{Z}}_{s,n}^H{\tilde{\mathit{U}}}_n^H{\mathit{h}}_{se}∥}^2{p}_s}{{\displaystyle \sum _{n=1}^N}{\tilde{\sigma }}^2{\mathit{w}}_{s,n}^H{\mathit{w}}_{s,n}}}-{\displaystyle \frac{{\displaystyle \sum _{n=1}^N}{∥{\mathit{h}}_{sd,n}∥}^2{p}_s}{{\displaystyle \sum _{n=1}^N}{∥{\mathit{h}}_{ed,n}^H{\mathit{U}}_n{\mathit{p}}_n∥}^2+N{\sigma }^2}},\hfill \end{array}$$

(12a)

$$\begin{array}{ccc}\hfill & s.t.& {\mathit{U}}_n,{\mathit{p}}_n,{\tilde{\mathit{U}}}_n\in \underset{{\mathit{U}}_{z,n},{\mathit{p}}_{z,n},{\tilde{\mathit{U}}}_{z,n}}{argmax}{\displaystyle \frac{{\displaystyle \sum _{n=1}^N}{∥{\mathit{w}}_{s,r,n}^H{\mathit{Z}}_{s,r,n}^H{\tilde{\mathit{U}}}_{z,n}^H{\mathit{A}}_{sr,n}{\mathit{U}}_{z,n}{\mathit{p}}_{z,n}∥}^2}{{\displaystyle \sum _{n=1}^N}{\tilde{\sigma }}^2{\mathit{w}}_{s,r,n}^H{\mathit{w}}_{s,r,n}}},\hfill \end{array}$$

(12b)

$$\begin{array}{ccc}\hfill & & {∥{\mathit{U}}_{z,n}{\mathit{p}}_{z,n}∥}^2\le p_{max},\left|{\mathit{U}}_{z,n}(i,j)\right|=1,\hfill \end{array}$$

(12c)

$$\begin{array}{ccc}\hfill & & {\displaystyle \frac{{∥{\mathit{w}}_{D,r,n}^H{\mathit{Z}}_{D,r,n}^H{\tilde{\mathit{U}}}_{z,n}^H{\mathit{A}}_{rd,n}{\mathit{U}}_{z,n}{\mathit{p}}_{z,n}∥}^2}{{\displaystyle \sum _{n=1}^N}{\tilde{\sigma }}^2{\mathit{w}}_{D,r,n}^H{\mathit{w}}_{D,r,n}}}\ge {\gamma }_r,\forall n,\hfill \end{array}$$

(12d)

where the variables ${\mathit{U}}_{z,n},{\mathit{p}}_{z,n},{\tilde{\mathit{U}}}_{z,n}$ represent the same physical quantities as ${\mathit{U}}_n,{\mathit{p}}_n,{\tilde{\mathit{U}}}_n$, respectively. These variables are introduced merely for convenience in the subsequent algorithm exposition, and their inclusion helps facilitate the optimization process. The constraints ${∥{\mathit{U}}_{z,n}{\mathit{p}}_{z,n}∥}^2\le p_{\max }$ reflect the power limits, while the condition ${\mathrm{SINR}}_{\mathrm{D},\mathrm{radar}}\ge {\gamma }_r$ guarantees the effectiveness of radar detection.

The above problem is a bilevel problem. Here, the lower-level problem aims to maximize the ${\mathrm{SIN}\mathrm{R}}_{\mathrm{S},\mathrm{radar}}$ (12b) under the power limitation and other constraints. Within the set of the optimal solution to the lower-level problem, the upper-level problem aims to maximize the ${\mathrm{SINR}}_{\mathrm{E}}-{\mathrm{SINR}}_{\mathrm{D}}$ (12a). Unlike the traditional single-level problem, we need to obtain the set of optimal solutions at the lower level and then bring in the upper level to find the optimal solution at the upper level. However, since bilevel optimization requires a convex lower-level problem, which is not satisfied in our case, we cannot apply it directly. In the following, we transform the original problem into an equivalent form that can be tackled more effectively.

3. Proposed Algorithm

In this section, we first use the Rayleigh quotient to put the problem in its simplest form. Then, we decouple our problems and introduce penalty terms so that we can develop a solution for the lower-level problem with convex underlying goals. Finally, we introduce the bilevel algorithm to solve the problem.

3.1. Simplification and Reformulation of Problem (12)

We now consider the optimization problem in lower-level problem (12b). Let us define

$$\begin{array}{cc}\hfill {\mathit{w}}_s& \triangleq {\left[{\mathit{w}}_{s,r,1}^H,{\mathit{w}}_{s,r,2}^H,\dots ,{\mathit{w}}_{s,r,N}^H\right]}^H\in {\mathbb{C}}^{(M-1)N\times 1},\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill {\mathit{h}}_s& \triangleq \left[\begin{array}{c}{\left({\mathit{Z}}_{s,r,1}^H{\mathit{A}}_{sr,1}^H{\mathit{U}}_{z,1}{\mathit{p}}_{z,1}\right)}^H\\ {\left({\mathit{Z}}_{s,r,2}^H{\mathit{A}}_{sr,2}^H{\mathit{U}}_{z,2}{\mathit{p}}_{z,2}\right)}^H\\ ⋮\\ {\left({\mathit{Z}}_{s,r,N}^H{\mathit{A}}_{sr,N}^H{\mathit{U}}_{z,N}{\mathit{p}}_{z,N}\right)}^H\end{array}\right]\in {\mathbb{C}}^{(M-1)N\times 1}.\hfill \end{array}$$

(14)

Then, we can transform the problem (12b) into the following problem:

$$\begin{array}{c}\hfill \underset{{\mathit{w}}_s}{max}{\displaystyle \frac{{∥{\mathit{w}}_s^H{\mathit{h}}_s∥}^2}{{\tilde{\sigma }}^2{\mathit{w}}_s^H{\mathit{w}}_s}}.\end{array}$$

(15)

To solve this, we first analyze the numerator. Since ${\mathit{w}}_s^H{\mathit{h}}_s$ is a scalar, its squared norm can be expanded as:

$$\begin{array}{cc}{∥{\mathit{w}}_s^H{\mathit{h}}_s∥}^2& =\left({\mathit{w}}_s^H{\mathit{h}}_s\right){\left({\mathit{w}}_s^H{\mathit{h}}_s\right)}^H=\left({\mathit{w}}_s^H{\mathit{h}}_s\right)\left({\mathit{h}}_s^H{\mathit{w}}_s\right)\hfill \\ & ={\mathit{w}}_s^H\left({\mathit{h}}_s{\mathit{h}}_s^H\right){\mathit{w}}_s.\hfill \end{array}$$

(16)

By substituting this back into (15), the optimization problem becomes

$$\underset{{\mathit{w}}_s}{max}{\displaystyle \frac{{\mathit{w}}_s^H\left({\mathit{h}}_s{\mathit{h}}_s^H\right){\mathit{w}}_s}{{\mathit{w}}_s^H{\mathit{w}}_s}}.$$

(17)

This expression is a generalized Rayleigh quotient of the form ${\displaystyle \frac{{\mathit{x}}^{H}A\mathit{x}}{{\mathit{x}}^H\mathit{x}}}$. Here, the matrix $A={\mathit{h}}_s{\mathit{h}}_s^H$ is rank-1 and, importantly, Hermitian, since $A^H={\left({\mathit{h}}_s{\mathit{h}}_s^H\right)}^H={\mathit{h}}_s{\mathit{h}}_s^H=A$. The maximum value of this quotient is the largest eigenvalue of A, and the optimal vector ${\mathit{w}}_s$ is the corresponding principal eigenvector. For a rank-1 matrix like ${\mathit{h}}_s{\mathit{h}}_s^H$, the principal eigenvector is simply ${\mathit{h}}_s$ itself. This observation justifies the formulation and leads directly to Lemma 1.

Lemma 1. 

Based on the Rayleigh quotient [18], a closed-form expression for the optimal ${\mathit{w}}_s$ can be achieved as

$$\begin{array}{c}\hfill {\mathit{w}}_s={\displaystyle \frac{{\mathit{h}}_s}{{\tilde{\sigma }}^2}}.\end{array}$$

(18)

Since problem (15) is a typical Rayleigh quotient maximization problem, we insert Equation (18) into problem (15) and then substitute the original forms of vector ${\mathit{h}}_s$ and vector ${\mathit{w}}_s$ into the newly obtained equation. Then we can formulate the new problem as follows:

$$\begin{array}{c}\hfill \underset{{\mathit{U}}_{z,n},{\mathit{p}}_{z,n},{\tilde{\mathit{U}}}_{z,n}}{max}{\displaystyle \frac{{\displaystyle \sum _{n=1}^N}{∥{\mathit{Z}}_{s,r,n}^H{\tilde{\mathit{U}}}_{z,n}^H{\mathit{A}}_{sr,n}{\mathit{U}}_{z,n}{\mathit{p}}_{z,n}∥}^2}{{\tilde{\sigma }}^2}}.\end{array}$$

(19)

The objective function in (19) is non-convex and involves highly coupled variables, making it difficult to solve directly. To make the problem tractable, we employ a penalty-based decoupling strategy, a standard technique for handling complex constraints in optimization theory [19]. The core idea is to transform a problem with hard equality constraints into an approximately equivalent problem by adding quadratic penalty terms to the objective function, which penalize any deviation from the original constraints.

Following this approach, we introduce several auxiliary variables to decouple the complex expressions. Let us define

$$\begin{array}{cc}\hfill \mathit{X}& \triangleq {\mathit{Z}}_{s,r,n}^H{\tilde{\mathit{U}}}_n^H{\mathit{A}}_{sr,n}{\mathit{U}}_n{\mathit{p}}_n,\mathit{Y}\triangleq {\mathit{U}}_n{\mathit{p}}_n,\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill {\mathit{X}}_z& \triangleq {\mathit{Z}}_{s,r,n}^H{\tilde{\mathit{U}}}_{z,n}^H{\mathit{A}}_{sr,n}{\mathit{U}}_{z,n}{\mathit{p}}_{z,n},{\mathit{Y}}_z\triangleq {\mathit{U}}_{z,n}{\mathit{p}}_{z,n}.\hfill \end{array}$$

(21)

Here, the variables without the subscript ‘z’ are for the upper level, while those with ‘z’ are their counterparts in the lower-level problem, introduced for convenience in the algorithm. By incorporating these definitions and their corresponding penalty terms, we can reformulate the lower-level problem as:

$$\begin{array}{ccc}\hfill & & {\mathit{U}}_n,{\mathit{p}}_n,{\tilde{\mathit{U}}}_n\in \underset{{\mathit{U}}_{z,n},{\mathit{p}}_{z,n},{\tilde{\mathit{U}}}_{z,n},{\mathit{X}}_z,{\mathit{Y}}_z}{argmax}{\displaystyle \frac{{\displaystyle \sum _{n=1}^N}{∥{\mathit{X}}_z∥}^2}{{\tilde{\sigma }}^2}}-\mu {∥{\mathit{Y}}_z-{\mathit{U}}_{z,n}{\mathit{p}}_{z,n}∥}^2\hfill \\ \hfill & & -\mu {∥{\mathit{X}}_z-{\mathit{Z}}_{s,r,n}^H{\tilde{\mathit{U}}}_{z,n}^H{\mathit{A}}_{sr,n}{\mathit{U}}_{z,n}{\mathit{p}}_{z,n}∥}^2,\hfill \\ \hfill & & s.t.{∥{\mathit{U}}_{z,n}{\mathit{p}}_{z,n}∥}^2\le p_{\max },\left|{\mathit{U}}_{z,n}(i,j)\right|=1,\hfill \\ \hfill & & {\displaystyle \frac{{\displaystyle \sum _{n=1}^N}{∥{\mathit{Z}}_{D,r,n}^H{\tilde{\mathit{U}}}_{z,n}^H{\mathit{A}}_{rd,n}{\mathit{Y}}_z∥}^2}{{\tilde{\sigma }}^2}}\ge {\gamma }_r,\hfill \\ \hfill & & {\mathit{X}}_z={\mathit{Z}}_{s,r,n}^H{\tilde{\mathit{U}}}_{z,n}^H{\mathit{A}}_{sr,n}{\mathit{U}}_{z,n}{\mathit{p}}_{z,n},{\mathit{Y}}_z={\mathit{U}}_{z,n}{\mathit{p}}_{z,n}.\hfill \end{array}$$

(22)

Similarly, we can employ (18) in the upper-level problem (12a) and combine the obtained result with Equation (22) where maximizing the objective in the lower-level problem is converted into minimizing the corresponding negative objective. Then we can obtain the formulated bilevel problem as follows:   

$$\begin{array}{ccc}\hfill & \underset{{\mathit{U}}_n,{\mathit{p}}_n,{\tilde{\mathit{U}}}_n}{min}& {\displaystyle \frac{{\displaystyle \sum _{n=1}^N}{∥{\mathit{h}}_{sd,n}∥}^2{p}_s}{{\displaystyle \sum _{n=1}^N}\left({∥{\mathit{h}}_{ed,n}^H\mathit{Y}∥}^2\right)+N{\sigma }^2}}-{\displaystyle \frac{{\displaystyle \sum _{n=1}^N}{∥{\mathit{Z}}_{s,n}^H{\tilde{\mathit{U}}}_n^H{\mathit{h}}_{se,n}∥}^2{p}_s}{{\tilde{\sigma }}^2}},\hfill \end{array}$$

(23a)

$$\begin{array}{ccc}\hfill & s.t.& {\mathit{U}}_n,{\mathit{p}}_n,{\tilde{\mathit{U}}}_n\in \underset{{\mathit{U}}_{z,n},{\mathit{p}}_{z,n},{\tilde{\mathit{U}}}_{z,n},{\mathit{X}}_z,{\mathit{Y}}_z}{argmax}-({\displaystyle \frac{{\displaystyle \sum _{n=1}^N}{∥{\mathit{X}}_z∥}^2}{{\tilde{\sigma }}^2}}-\mu {∥{\mathit{Y}}_z-{\mathit{U}}_{z,n}{\mathit{p}}_{z,n}∥}^2\hfill \\ \hfill & & -\mu {∥{\mathit{X}}_z-{\mathit{Z}}_{s,r,n}^H{\tilde{\mathit{U}}}_{z,n}^H{\mathit{A}}_{sr,n}{\mathit{U}}_{z,n}{\mathit{p}}_{z,n}∥}^2),\hfill \end{array}$$

(23b)

$$\begin{array}{ccc}\hfill & & {∥{\mathit{U}}_{z,n}{\mathit{p}}_{z,n}∥}^2\le p_{max},\left|{\mathit{U}}_{z,n}(i,j)\right|=1,\hfill \end{array}$$

(23c)

$$\begin{array}{ccc}\hfill & & {\displaystyle \frac{{\displaystyle \sum _{n=1}^N}{∥{\mathit{Z}}_{D,r,n}^H{\tilde{\mathit{U}}}_{z,n}^H{\mathit{A}}_{rd,n}{\mathit{Y}}_z∥}^2}{{\tilde{\sigma }}^2}}\ge {\gamma }_r,\forall n,\hfill \end{array}$$

(23d)

$$\begin{array}{ccc}\hfill & & \mathit{X}={\mathit{Z}}_{s,r,n}^H{\tilde{\mathit{U}}}_n^H{\mathit{A}}_{sr,n}{\mathit{U}}_n{\mathit{p}}_n,\mathit{Y}={\mathit{U}}_n{\mathit{p}}_n,\hfill \end{array}$$

(23e)

$$\begin{array}{ccc}\hfill & & {\mathit{X}}_z={\mathit{Z}}_{s,r,n}^H{\tilde{\mathit{U}}}_{z,n}^H{\mathit{A}}_{sr,n}{\mathit{U}}_{z,n}{\mathit{p}}_{z,n},{\mathit{Y}}_z={\mathit{U}}_{z,n}{\mathit{p}}_{z,n}.\hfill \end{array}$$

(23f)

3.2. Proposed Algorithm to the Bilevel Problem

To solve the constrained bilevel optimization problem in our ISAC system, we adopt a first-order penalty-based approach inspired by recent advances in bilevel optimization [20]. Although the lower-level subproblem in our formulation is convex, the upper-level objective involves non-convex constraints, making the bilevel structure difficult to handle directly using standard techniques such as KKT-based reformulations or implicit differentiation.

To address this, we reformulate the bilevel problem by explicitly defining the upper-level objective $f(x,y)$, the lower-level objective $\tilde{f}(z,y)$, and the lower-level constraints $\tilde{g}(x,z)$. These functions encode the physical objectives of our system. $f(x,y)$ reflects the performance metric related to the legitimate monitor, such as signal reception or eavesdropping capability; $\tilde{f}(z,y)$ quantifies the radar sensing quality; $\tilde{g}(x,z)$ captures the power and SINR constraints associated with the radar operations. With these definitions, the original bilevel problem can be expressed in the standard form:

$$\begin{array}{c}f=minf(x,y)\hfill \\ \mathrm{s}.\mathrm{t}. y\in \underset{z}{\arg \min }\left\{\tilde{f}(x,z)\mid \tilde{g}(x,z)\le 0\right\}.\hfill \end{array}$$

(24)

The above structure matches the setting studied in [20], where the lower-level problem is a convex problem but the upper-level problem may be a non-convex one. Under such conditions, a penalty reformulation is proposed to approximate the bilevel problem using a minmax formulation, which can be solved with traditional methods.

In our specific case, since all decision variables appear in either y or z, and no variable is independently assigned to x, we set $\mathit{x}\leftarrow \varnothing$, $\mathit{y}\leftarrow \{{\mathit{U}}_n,{\mathit{p}}_n,{\tilde{\mathit{U}}}_n,\mathit{X},\mathit{Y}\}$ and $\mathit{z}\leftarrow \{{\mathit{U}}_{z,n},{\mathit{p}}_{z,n},{\tilde{\mathit{U}}}_{z,n},{\mathit{X}}_z,{\mathit{Y}}_z\}$. The bilevel problem is thus written as

$$\begin{array}{c}f=minf\left(y\right)\hfill \\ \mathrm{s}.\mathrm{t}.y\in \underset{z}{argmin}\left\{\tilde{f}\left(z\right)\mid \tilde{g}\left(z\right)\le 0\right\}.\hfill \end{array}$$

(25)

To proceed with solving the bilevel problem formulated in Equation (25), we follow the first-order penalty approach introduced in [20]. Specifically, we introduce a penalty function to handle the lower-level constraint by incorporating it into the objective.

$${\tilde{P}}_{\mu }(x,z)=\tilde{f}(x,z)+\mu {\parallel {\left[\tilde{g}(x,z)\right]}_{+}\parallel }^2,$$

(26)

where $\mu >0$ is a penalty parameter and ${[·]}_{+}$ denotes the component-wise maximum with zero. This penalized formulation approximates the lower-level constrained problem by smoothing the feasible region.

Based on this, the bilevel problem can be approximately reformulated as

$$f_{\mu }=\underset{x,y}{min}\left\{f(x,y)\mid y\in arg\underset{z}{min}{\tilde{P}}_{\mu }(x,z)\right\}.$$

(27)

To further eliminate the constraint $y\in \underset{z}{\mathrm{argmin}}{\tilde{P}}_{\mu }\left(z\right)$, we consider the value function of the penalized lower-level problem. Let $V_{\mu }:=\underset{z}{\min }{\tilde{P}}_{\mu }(x,z)$ denote the minimum value of the penalized lower-level objective. Then, the constraint $y\in arg{min}_z{\tilde{P}}_{\mu }(x,z)$ is equivalent to enforcing $\underset{z}{\min }{\tilde{P}}_{\mu }(x,z)=V_{\mu }$. This motivates a value-function-based penalization of the form

$$f\left(y\right)+\rho \left({\tilde{P}}_{\mu }(x,y)-V_{\mu }\right),$$

(28)

where $\rho >0$ is an outer-level penalty parameter. As shown in [20], minimizing this relaxed objective yields a solution that asymptotically satisfies the original bilevel constraints as $\rho \to \infty$. From $\underset{z}{min}{\tilde{P}}_{\mu }(x,z)=V_{\mu }$, we have

$$\underset{x,y}{min}f(x,y)+\rho \left({\tilde{P}}_{\mu }(x,y)-\underset{z}{min}{\tilde{P}}_{\mu }(x,z)\right),$$

(29)

which can be further written as the following minmax problem:

$$\underset{x,y}{min}\underset{z}{max}{P}_{\rho ,\mu }(x,y,z),$$

(30)

where the penalized objective is defined by

$$P_{\rho ,\mu }(x,y,z):=f(x,y)+\rho \left({\tilde{P}}_{\mu }(x,y)-{\tilde{P}}_{\mu }(x,z)\right).$$

(31)

We now turn to the structure of the functions involved in the objective and constraints. It is worth noting that the constant modulus constraint, $\left|{\mathit{U}}_{z,n}(i,j)\right|=1$, is handled differently from other constraints. Instead of introducing a corresponding penalty term, we enforce this constraint directly in our algorithm by parameterizing the elements of the analog beamformer as complex exponentials. This ensures the constraint is always satisfied by construction. The remaining functions and constraints are handled by the penalty framework. Specifically, the upper-level objective $f\left(y\right)$ and the lower-level function $\tilde{f}\left(z\right)$ are split into components as follows:

$$\begin{array}{c}f\left(y\right)=f_1\left(y\right)={\displaystyle \frac{{\displaystyle \sum _{n=1}^N}{∥{\mathit{h}}_{sd,n}∥}^2{p}_s}{{\displaystyle \sum _{n=1}^N}{∥{\mathit{h}}_{ed,n}^H\mathit{Y}∥}^2+N{\sigma }^2}}-{\displaystyle \frac{{\displaystyle \sum _{n=1}^N}{∥{\mathit{Z}}_{s,n}^H{\tilde{\mathit{U}}}_n^H{\mathit{h}}_{se,n}∥}^2{p}_s}{{\tilde{\sigma }}^2}},\\ \tilde{f}\left(z\right)={\tilde{f}}_2\left(z\right)=-{\displaystyle \frac{\sum _{n=1}^N{∥{\mathit{X}}_z∥}^2}{{\tilde{\sigma }}^2}}+\mu {∥{\mathit{Y}}_z-{\mathit{U}}_{z,n}{\mathit{p}}_{z,n}∥}^2+\mu {∥{\mathit{X}}_z-{\mathit{Z}}_{s,r,n}^H{\tilde{\mathit{U}}}_{z,n}^H{\mathit{A}}_{sr,n}{\mathit{U}}_{z,n}{\mathit{p}}_{z,n}∥}^2.\end{array}$$

(32)

The function $f_1\left(y\right)$ represents the upper-level performance metric associated with the legitimate monitor reception quality or eavesdropping capability, while $f_2\left(y\right)$ quantifies the penalty term that comes from the radar sensing quality, which we want to minimize to reduce interference.

Similarly, the constraints are divided as

$$g_1\left(z\right)={∥{\mathit{U}}_{z,n}{\mathit{p}}_{z,n}∥}^2-p_{\max },g_2\left(z\right)={\gamma }_r-{\displaystyle \frac{{\displaystyle \sum _{n=1}^N}{∥{\mathit{Z}}_{D,r,n}^H{\tilde{\mathit{U}}}_{z,n}^H{\mathit{A}}_{rd,n}{\mathit{Y}}_z∥}^2}{{\tilde{\sigma }}^2}}.$$

(33)

The constraints $g_1\left(y\right)$ and $g_2\left(y\right)$ capture the power and SINR constraints for the radar operations. Specifically, $g_1\left(y\right)$ ensures that the power consumption does not exceed the maximum power $p_{\max }$, and $g_2\left(y\right)$ captures the SINR constraints to guarantee radar sensing quality.

The decomposition of the objective and constraints into separate components for the upper and lower levels is a direct consequence of the penalty-based reformulation. This allows us to handle the minimization and maximization of the functions separately, which simplifies the optimization process and ensures that the penalization of constraint violations is smoothly incorporated. This provides a natural way to handle the original bilevel problem as a minmax problem. Therefore, the function in (31) can be decomposed using the additive structure of f and $\tilde{f}$, yielding

$$\begin{array}{cc}\hfill P_{\rho ,\mu }(x,y,z)& =f_1(x,y)+f_2\left(x\right)+\rho \left({\tilde{f}}_1(x,y)-{\tilde{f}}_1(x,z)\right)\hfill \\ \hfill & +\rho \mu \left(\parallel {\left[\tilde{g}(x,y)\right]}_{+}{\parallel }^2-{\parallel {\left[\tilde{g}(x,z)\right]}_{+}\parallel }^2\right)\hfill \\ \hfill & +\rho \left({\tilde{f}}_2\left(y\right)-{\tilde{f}}_2\left(z\right)\right).\hfill \end{array}$$

(34)

Given that $P_{\rho ,\mu }(x,y,z)$ is non-convex in $(x,y)$, concave in z, and built from smooth and proximally well-behaved functions, the corresponding minmax problem can be equivalently reformulated as

$$\underset{x}{min}\underset{y}{max}\left\{h(x,y)+p\left(x\right)-q\left(y\right)\right\},$$

(35)

where the individual terms are defined as

$$\begin{array}{cc}\hfill p\left(x\right)& =\rho {\tilde{f}}_2\left(x\right),q\left(y\right)=\rho {\tilde{f}}_2\left(y\right),\hfill \\ \hfill h(x,y)& =f_1\left(x\right)+\rho \mu \left(\parallel {\left[{\tilde{g}}_1\left(x\right)\right]}_{+}{\parallel }^2+{\parallel {\left[{\tilde{g}}_2\left(x\right)\right]}_{+}\parallel }^2\right)\hfill \\ \hfill & -\rho \mu \left(\parallel {\left[{\tilde{g}}_1\left(y\right)\right]}_{+}{\parallel }^2+{\parallel {\left[{\tilde{g}}_2\left(y\right)\right]}_{+}\parallel }^2\right).\hfill \end{array}$$

(36)

Thus, the original bilevel optimization problem has been successfully reformulated as a structured minmax problem. This reformulation enables the application of the traditional methods for solving bilevel problems. To efficiently solve the resulting minmax problem, we design an iterative algorithm that alternately updates the variables x and y while dynamically adjusting the penalty parameters $\rho$ and $\mu$. It is important to note that this algorithmic framework is robust; as established in our system model, practical imperfections such as RSI do not alter the fundamental problem structure, thus ensuring the applicability of the proposed method. The detailed procedure is outlined as Algorithm 1.

To elaborate, Algorithm 1 is designed to solve the structured minmax problem by alternately updating the variables y and x. At each iteration, the algorithm first updates y by solving the inner maximization problem. Specifically, it minimizes the negative of the objective function, that is $-\left(h\right(x,y)+p(x)-q(y\left)\right)$, to approximate the maximum with respect to y given the current estimate of x. Once the optimal y corresponding to the current x is obtained, the algorithm substitutes it into the objective and proceeds to update x by solving the outer minimization problem. In this way, the algorithm progressively minimizes the outer objective while accounting for the adversarial behavior of y. Meanwhile, the penalty parameters $\rho$ and $\mu$ are dynamically adjusted during the iterations to gradually enforce constraint satisfaction and improve solution stability. This alternating optimization strategy enables the algorithm to approach a saddle point of the original minmax formulation and effectively solve the reformulated bilevel problem.

Convergence and Feasibility Analysis. Regarding the convergence and feasibility of Algorithm 1, it is important to note that for a non-convex problem such as this, guaranteeing convergence to a global optimum is generally intractable. Our objective is instead to find a high-quality stationary point. The feasibility of the solution is ensured by the penalty method itself; as the penalty parameters $\rho$ and $\mu$ are progressively increased during the iterations, the solution is driven towards the feasible set of the original constrained problem [20]. While a formal convergence proof for our specific alternating algorithm is beyond the scope of this paper, this structure is a standard and effective heuristic designed to find a stationary point of the reformulated minmax problem. The practical effectiveness of this convergence is validated in our simulation results.

Computational Complexity Analysis. The overall computational complexity of the proposed Algorithm 1 is dominated by the iterative updates of variables x and y. In each outer iteration, the algorithm solves two main subproblems: a maximization with respect to y and a minimization with respect to x. Assuming a gradient-based method is used to solve each subproblem for a fixed number of inner steps, the cost per iteration is primarily determined by the gradient computation of the functions $h(x,y)$, $p\left(x\right)$, and $q\left(y\right)$. The dimensions of the variables x and y are mainly determined by the number of antennas N and RF chains M. Consequently, the complexity of computing the gradient for each step involves matrix–vector multiplications and scales polynomially with these parameters. Let $T_{\mathrm{outer}}$ be the number of outer iterations and $T_{\mathrm{inner}}$ be the number of inner gradient steps for each subproblem. The total computational complexity can be characterized as $\mathcal{O}(T_{\mathrm{outer}}·T_{\mathrm{inner}}·\mathrm{poly}(N,M))$, where $\mathrm{poly}(N,M)$ represents the cost of a single gradient evaluation. As our simulation results demonstrate, the algorithm is practically efficient, converging within a few hundred outer iterations, which validates the effectiveness of this approach.

4. Simulation Results

In this section, we present the simulation results that demonstrate the superiority of our bilevel algorithm.

Algorithm 1 The proposed method for the minmax problem (35)

Input:$N,M$ ($N\gg M$), $x^0,y^0$, $\rho >0$, $\mu =0.5$, tolerance $={10}^{-20}$, maxIters $=3000$, $t=0$, etc.  for$i=0,1,2,\dots$do      for $j=0,1,2,\dots$ do          $t\leftarrow 0$          while $\parallel x^{t+1}-x^t\parallel +\parallel y^{t+1}-y^t\parallel \ge$ tolerance and $t\le$ maxIters do             Update $y^{t+1}$ by solving:                 ${\displaystyle \underset{y}{min}\left\{-(h(x^t,y)+p\left(x^t\right)-q\left(y\right))\right\}}$             Update $x^{t+1}$ by solving:                 ${\displaystyle \underset{x}{min}\left\{h(x,y^{t+1})+p\left(x\right)-q\left(y^{t+1}\right)\right\}}$             Update $\rho$, $\mu$             $t\leftarrow t+1$          end while      end for  end for

4.1. Simulation Setup

In our simulations, we adopt a hybrid analog–digital architecture where the numbers of antennas and RF chains are set to $N=16$ and $M=4$, respectively. The inter-antenna spacing is set to half the wavelength ($d=0.5\lambda$). The main beam is directed toward an angle of ${\theta }_n=\pi /4$, and other angles ${\theta }_1$ and ${\theta }_2$ are randomly generated around $\pi /6$ and $\pi /4$, respectively, to simulate the beam directions.

The noise power is set as $\sigma =\tilde{\sigma }=3$$\mathrm{W}$. The impact of noise on performance is not considered separately. Additionally, the maximum transmit power is fixed at $P_s=10$ dBm. The gain factor ${\beta }_n$ is set to 1.2. The constraint of the radar’s reception of signals from the suspicious receiver D, denoted as ${\gamma }_r$, is specified in relevant sections.

The locations of the suspicious transmitter S, suspicious receiver D, and the legitimate monitor S are set as follows: S is placed at the origin $(0,0)\mathrm{m}$, D is located at $(0,60)\mathrm{m}$, and S is positioned at $(80,0)\mathrm{m}$.

The wireless channels between $S\text{-}E$, $S\text{-}D$, and $E\text{-}D$ are generated based on a composite fading model that includes both large-scale and small-scale effects. The large-scale fading is modeled as a distance-dependent path loss using the logarithmic model.

$$L\left(d\right)=30+36{log}_{10}\left(d\right)\left(\mathrm{dB}\right),$$

(37)

where d is the Euclidean distance in meters between the transmitter and receiver. The corresponding path loss gain is given by $G\left(d\right)={10}^{-L\left(d\right)/10}$. The small-scale fading follows a Rician distribution, where the Rician K-factors $K_{\mathrm{SE}}$, $K_{\mathrm{SD}}$, and $K_{\mathrm{ED}}$ are set to ${10}^{0.3}$, ${10}^{0.8}$, and ${10}^{0.6}$, corresponding to $3\mathrm{dB}$, $8\mathrm{dB}$, and $6\mathrm{dB}$, respectively. The Rician K-factors represent the power ratio between the line-of-sight (LoS) and non-line-of-sight (NLoS) components. A higher $\eta$ indicates a stronger LoS dominance. The final channel matrices are computed as the product of the large-scale fading gain and the Rician fading component.

To supplement the SINR metric and provide a more direct assessment of radar performance, we also evaluate the probability of detection ($P_D$). Following established methodologies in detection theory [21,22], we calculate the $P_D$ by mapping our simulated ${\mathrm{SIN}\mathrm{R}}_{S,\mathrm{radar}}$ results for a given constant Probability of False Alarm ($P_{FA}$). This allows for a direct translation from the SINR, which our algorithm optimizes, to a more practical operational metric. For this calculation, we set the false alarm rate to $P_{FA}={10}^{-3}$ and the number of pulses to $L=4$.

4.2. Performance Verification

To evaluate the performance of the proposed bilevel algorithm, we compare it with a benchmark scheme as described below.

Conventional Constrained Single-Level Scheme. To highlight the advantages of our bilevel approach, we compare it against a more conventional single-level constrained optimization scheme that aims to maximize the upper-level objective, namely, the active eavesdropping capability, while incorporating the lower-level goal, the radar localization performance, as a constraint. Specifically, this scheme solves the following single-level problem:

$$\begin{array}{cc}\hfill \underset{{\mathit{U}}_n,{\mathit{p}}_n,{\tilde{\mathit{U}}}_n}{max}& {\displaystyle \frac{{\displaystyle \sum _{n=1}^N}{\parallel {\mathit{Z}}_{s,n}^H{\tilde{\mathit{U}}}_n^H{\mathit{h}}_{se,n}\parallel }^2{p}_s}{{\tilde{\sigma }}^2}}-{\displaystyle \frac{{\displaystyle \sum _{n=1}^N}{\parallel {\mathit{h}}_{sd,n}\parallel }^2{p}_s}{{\displaystyle \sum _{n=1}^N}{\parallel {\mathit{h}}_{ed,n}^H\mathit{Y}\parallel }^2+N{\sigma }^2}}\hfill \end{array}$$

(38a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& {\mathrm{SINR}}_{S,\mathrm{radar}}\ge {\gamma }_{r1},\hfill \end{array}$$

(38b)

$$\begin{array}{cc}\hfill & {\mathrm{SINR}}_{D,\mathrm{radar}}\ge {\gamma }_{r2},\hfill \end{array}$$

(38c)

$$\begin{array}{cc}\hfill & \parallel {\mathit{U}}_{z,n}{\mathit{p}}_{z,n}{\parallel }^2\le p_{\max },\left|{\mathit{U}}_{z,n}(i,j)\right|=1,\forall n,i,j.\hfill \end{array}$$

(38d)

Because the radar performance is enforced via the constraint in (38b), the resulting beamforming solution is functionally radar-aware. This provides a fair and meaningful comparison against our bilevel formulation, allowing us to evaluate which optimization philosophy—hierarchical co-optimization versus constrained single-level optimization—achieves a better trade-off between the two objectives.

Figure 3 illustrates the convergence behavior of the bilevel algorithm, showing how the radar’s sum rate changes with the number of iterations. From the figure, it is evident that the algorithm converges quickly, reaching near convergence after approximately 80 to 100 iterations. After this point, the sum rate stabilizes, indicating that the algorithm achieves efficient convergence with relatively few iterations.

We also compare the performance of the bilevel algorithm against single-level algorithm in terms of the noise power parameter as shown in Figure 4. Although the single-level algorithm (represented by the red curve) slightly outperforms the bilevel algorithm in eavesdropping performance (represented by the blue curve), the bilevel algorithm demonstrates superior overall performance by optimizing both the upper-level and lower-level objectives. As we can see, the radar performance of the bilevel algorithm (represented by the yellow curve) surpasses the radar performance of the single-level algorithm (represented by the purple curve). As the noise power increases, the bilevel algorithm maintains a higher radar performance, showing its robustness in handling noise.

Next, the performance with respect to the number of RF chains is illustrated in Figure 5. As the number of RF chains increases, both eavesdropping and radar performance improve for both the bilevel and single-level algorithms. However, the key insight here is the radar performance (represented by the red curve for the bilevel algorithm and the purple curve for the single-level algorithm) clearly demonstrates that the bilevel algorithm maintains a significantly higher advantage. Meanwhile, the dashed curves in Figure 5 depict the corresponding detection probabilities. Consistent with the SINR trends, the bilevel scheme sustains an almost unit-level $P_D$ across the entire RF-chain range, whereas the single-level scheme starts below $0.5$ and only approaches near-perfect detection once $M\ge 8$, further underscoring the bilevel algorithm’s robustness in preserving radar reliability.

Additionally, as the number of RF chains increases, the eavesdropping performance of the bilevel algorithm (represented by the blue curve) gradually surpasses that of the single-level algorithm (represented by the yellow curve). This shows that the bilevel optimization approach better balances radar and eavesdropping objectives, resulting in superior radar performance especially as the number of RF chains increases. This growing advantage can be attributed to the increased spatial degrees of freedom for precoding; with more RF chains, the bilevel algorithm can better leverage spatial multiplexing to craft more sophisticated beamformers that effectively nullify interference while simultaneously satisfying the radar performance constraints.

Additionally, in Figure 6, we analyze the performance with respect to the number of antennas. As the number of antennas increases, it is evident that the eavesdropping performance (represented by the blue curve for the bilevel algorithm and the yellow curve for the single-level algorithm) shows minimal difference between the bilevel and single-level algorithms. Both algorithms exhibit a general improvement as the number of antennas grows, with the bilevel algorithm maintaining slightly better performance in some cases. However, the key distinction between the two approaches lies in radar performance. The bilevel algorithm significantly outperforms the single-level algorithm in radar detection, demonstrating a superior ability to leverage additional antennas for better radar functionality.

Although we observe that, with a higher number of antennas, the bilevel radar performance starts to slightly decrease, this counter-intuitive trend is not an algorithmic flaw but a feature of sophisticated trade-off management. This phenomenon can be attributed to two factors: first, a power dilution effect where the fixed total power is spread across more antennas, and second, a strategic decision by the bilevel optimizer. With more spatial degrees of freedom available, the algorithm intelligently determines that a marginal sacrifice in the secondary radar objective can be traded for a more significant gain in the primary eavesdropping objective. This reduction is still less pronounced compared to the substantial improvements seen in the single-level radar performance. The bilevel optimization method still maintains its edge by achieving a more balanced and optimal use of antennas across both radar and eavesdropping tasks, ensuring that the trade-off does not heavily compromise the overall performance.

Finally, we evaluate the robustness of our proposed algorithm under varying channel conditions by sweeping the Rician K-factor from −5 dB to 15 dB, with the results shown in Figure 7. The K-factor indicates the ratio of the LoS power to the NLoS power; a lower K-factor indicates a more challenging, scattering-rich environment, while a higher K-factor represents a more stable, LoS-dominant channel. As observed, the key advantage of the proposed bilevel algorithm lies in its robust radar performance. The radar SINR (represented by the red curve) remains consistently high across the entire range of the K-factor, demonstrating that our algorithm successfully protects the lower-level objective regardless of the channel quality, which varies from NLoS-dominant (low K) to LoS-dominant (high K). Regarding the eavesdropping performance, both schemes exhibit similar behavior, with performance generally improving as the channel becomes more predictable with a higher K-factor.

Across various system parameters (noise power, RF chains, antenna count) and channel conditions (Rician K-factor), our scheme consistently demonstrates a superior ability to manage the trade-off between radar localization and proactive eavesdropping.

4.3. Discussion on Model Assumptions and Limitations

In this section, we elaborate on the key assumptions of our system model to clarify their suitability and limitations, thereby contextualizing our results.

Gaussian Clutter and Noise Model: The assumption that aggregate clutter and noise follow a Gaussian distribution is a foundational and practical simplification in the radar signal processing literature. This is justified by the central limit theorem, particularly when clutter arises from a large number of independent, randomly distributed scatterers. We acknowledge, however, that this model has limitations. In environments characterized by a few dominant, "spiky" clutter sources (e.g., large buildings), the interference may exhibit heavier tails and could be more accurately described by other statistical models, such as the K-distribution or Weibull distribution.

Rician Fading Channel Model: Our use of the Rician fading model is standard for wireless channels that include a dominant LoS path alongside scattered multipath components. This is highly relevant for ISAC scenarios, where a stable LoS path is often crucial for reliable sensing. The Rician K-factor allows us to parameterize the strength of this LoS component. To explicitly validate our algorithm’s performance across different channel realities, our robustness analysis with respect to the K-factor as shown in Figure 7 demonstrates that our algorithm maintains strong radar performance even in challenging low-K (NLoS-dominant) conditions.

Perfect Channel State Information (CSI): Following the prevailing practice in ISAC research [23], we assume perfect CSI at the transmitter. This idealized setting establishes an unequivocal performance upper bound and allows the intrinsic advantage of our bilevel optimization structure to emerge without the confounding influence of channel-estimation errors. Nevertheless, CSI in real deployments is inevitably imperfect. According to [23], uncertainty in CSI induces a fundamental trade-off between communication throughput and the accuracy of channel-state estimation, thereby motivating robust physical-layer designs. Several mature techniques can be embedded into our framework to mitigate such uncertainty, including worst-case robust optimization—which bounds the estimation error within an ellipsoidal set and, via the S-procedure, yields a tractable semidefinite program [24], chance-constrained formulations that ensure probabilistic performance guarantees when only error statistics are available, and distributionally robust methods that hedge against model mismatch. Integrating these robust counterparts into the following problem while preserving the upper-level resource-allocation strategy constitutes a primary direction of our future work.

5. Conclusions

In this paper, a bilevel optimization algorithm has been developed for ISAC systems, aiming to maximize the active eavesdropping probability while ensuring high radar localization performance. We reformulated the problem by using the Rayleigh quotient, decoupling the objectives and introducing penalty terms. This allowed us to apply a bilevel algorithm to optimize both beamforming matrices. Extensive simulation results demonstrate that our algorithm effectively maintains high radar localization accuracy while achieving nearly the same active eavesdropping probability as existing methods, showcasing its superior performance in ISAC scenarios. Future work will consider extending the proposed framework to multi-user scenarios and enhancing robustness under imperfect channel conditions.