Hierarchical Beamforming Optimization for ISAC-Enabled RSU Systems in Complex Urban Environments

Abstract

Integrated Sensing and Communication (ISAC)-enabled Roadside Units (RSUs) encounter significant performance trade-offs between target sensing and multi-user communication in complex urban environments, where conventional optimization methods are prone to converging to local optima and joint optimization methods often yield sub-optimal results due to conflicting objectives. To address the challenge of trade-off between sensing and communication performance, this paper proposes a hierarchical beamforming optimization solution designed to tackle joint sensing–communication problems in such scenarios. The overall optimization problem is decomposed into a two-level “leader-follower” structure. In the leader layer, we introduce a max–min strategy based on the bisection method to transform the non-convex Signal-to-Interference-plus-Noise Ratio (SINR) optimization problem into a second-order cone constraint problem and solve the communication beamforming vector. In the follower layer, the Signal-to-Clutter-plus-Noise Ratio (SCNR) maximization problem is converted into a Semi-Definite Programming (SDP) problem solved via the CVX toolbox. Additionally, we introduce a “spatiotemporal resource isolation” mechanism to project the sensing beam onto the null space of the communication channel. The hierarchical optimization solution jointly optimizes communication SINR and sensing SCNR, enabling an effective balance between sensing accuracy and communication reliability. Simulation results demonstrate the proposed method’s effectiveness in simultaneously improving sensing accuracy and communication reliability.

1. Introduction

The advancement of 6G technology has brought Integrated Sensing and Communication (ISAC) to the forefront of Vehicle-to-Everything (V2X) research [1,2,3]. ISAC-enabled V2X systems facilitate comprehensive data exchange between vehicles and their environment, including infrastructure, pedestrians, and networks [4,5,6]. To optimize urban road management efficiency, Simultaneous Localization and Mapping (SLAM) technology is increasingly being integrated with ISAC [7]. While traditional SLAM implementations primarily rely on vision and LiDAR sensors, recent breakthroughs have shown that wireless channel propagation characteristics can be used to construct highly accurate 2D environmental maps [8,9]. This synergy establishes a novel co-design paradigm for ISAC-SLAM integration in V2X. Recent research demonstrate that an ISAC base station (BS) can effectively function as a roadside unit (RSU) [2,10,11]. These advanced ISAC-RSUs enhance environmental sensing and establish real-time SLAM capabilities through low-latency V2X links, compensating for onboard sensor limitations. Smart city implementations have validated this approach, with [12] demonstrating that MIMO radar-equipped RSUs can utilize echo signals for channel estimation and adaptive beamforming. This enables ISAC-RSU as the core infrastructure for future 6G vehicular networks. In dense urban environments, buildings and other obstacles can significantly impede signal propagation. Relay communication effectively extends coverage and ensures continuous connectivity between vehicles and infrastructure by mitigating the impact of physical obstructions. In high-density traffic scenarios, relay nodes facilitate multi-path data transmission, enhancing communication reliability and stability through redundancy and improved link diversity [2]. By distributing the communication load across multiple nodes, relay systems increase overall network capacity, which is crucial for supporting the concurrent communication demands of numerous connected vehicles and devices [13]. Ref. [10] has shown that RSUs, as critical components of next-generation urban communication infrastructures, possess significant capabilities in directional relay-assisted transmission. Specifically, these ISAC-RSUs can simultaneously sense urban road conditions and function as relay base stations, transmitting directional communication signals to user equipment (UE) [14]. Furthermore, Ref. [15] introduced a hybrid reconfigurable intelligent surface-assisted downlink ISAC system that offers a new reference approach for integrating RSUs and reconfigurable intelligent surfaces to enhance UE communication rates. Refs. [16,17] propose deep learning (DL)-based predictive beamforming for ISAC in vehicular networks and demonstrate DL’s potential for next-generation vehicular ISAC systems.

However, the simultaneous delivery of high-quality communication and precise detection becomes particularly challenging in complex multi-user, multi-target scenarios. Balancing communication-centric and sensing-centric objectives remains a critical issue in ISAC systems due to competing demands for spectral resources, beamforming design, and signal processing strategies [18]. Furthermore, RSUs face substantial signal interference that adversely affects performance in both communication and sensing domain. In communication, downlink signals from adjacent RSUs generate multiple access interference (MAI) with UEs, while spatiotemporal resource reuse further aggravates multi-user interference (MUI) [19]. In sensing, the overlap of RSU echo signals in the time–frequency domain can degrade the signal-to-noise ratio (SNR). These dual-domain interference problems necessitate advanced beamforming solutions for ISAC-RSU systems. Additionally, urban multi-path effects and building occlusions constrain their detection capabilities, as sensing beams reflect off buildings, creating clutter that interferes with the ISAC-RSU receiver. These challenges require the development of enhanced beamforming designs for ISAC-RSU systems [20,21,22,23].

Contemporary studies demonstrate that beamforming design plays a crucial role in establishing the Pareto frontier, which quantifies the fundamental trade-off between sensing and communication performance in ISAC systems. ISAC beamforming aims to achieve optimal performance trade-offs while suppressing mutual interference between sensing and communication signals [24]. In the context of ISAC beamforming methods, Refs. [16,17] address dynamic channel variations caused by high mobility, leveraging neural networks to forecast beamforming vectors and optimize ISAC performance. However, the internal weights of its network are difficult to map to physical-layer parameters (such as beam pointing angle and null notch depth), limiting their effectiveness in fault diagnosis. In contrast, conventional methods offer closed-form solutions derived from analytical algorithms based on channel state information, thereby facilitating optimization and parameter adjustment. Ref. [25] investigated a weighted minimum mean square error (WMMSE) ISAC solution based on mutual information (MI) to transform a non-convex problem into a convex one, utilizing MI as a metric to assess both sensing and communication performance. Furthermore, cell-free ISAC MIMO systems represent a significant advancement in the simultaneous provision of communication services to users and the detection of targets [26,27,28,29]. Specifically, Ref. [26] proposed a joint beamforming design for cell-free ISAC MIMO systems, where distributed MIMO access points collaboratively facilitate communication for user equipment (UEs) while simultaneously sensing targets. This study employs a max–min fairness formulation and derives the optimal structure of beamforming vectors to suppress both MAI and MUI, thereby improving the communication SINR [27]. Additionally, in low-altitude urban environments, objects exhibiting a significant radar cross-section (RCS), including structures and metallic advertisements, can generate scattered clutter patches that interfere with the receiver’s performance. In order to address the issue of environmental clutter, Ref. [29] proposed a constant-modulus waveform design method for ISAC systems by incorporating cyclic optimization, Dinkelbach’s transform, and the ADMM algorithm. This method effectively addresses the non-convex optimization problem while optimizing both radar detection performance and communication service quality, but it suffers from high computational complexity due to excessive matrix operations and exhibits relatively slow convergence rates.

In response to these issues, we propose a dual-base-station ISAC system with a transmitting RSU and a receiving RSU, suitable for road surveillance and directional relay communication in urban areas. The BS transmits downlink signals to the ISAC-RSU, providing high-quality services. We propose a hierarchical optimization solution for ISAC beamforming, specifically designed for ISAC-RSU systems operating in complex environments. The main contributions of this paper are as follows:

We construct a hierarchical and collaborative ISAC beamforming optimization framework, and it has the potential to overcome the limitations imposed by the direct coupling of sensing and communication objectives in existing single-stage optimization approaches. The leader layer prioritizes ensuring communication performance for multiple UEs by adopting a max–min fairness strategy to optimize the SINR. The communication beamforming vector is derived using the bisection method in conjunction with second-order cone programming, thereby guaranteeing that the communication links satisfy both fairness and reliability requirements across UEs. Under the constraint of fixed communication beamforming parameters from the follower layer, the problem of maximizing the sensing SCNR is formulated as a semi-definite programming (SDP) problem, enabling collaborative design of the sensing beam through convex optimization techniques. Therefore, it achieves a coherent synergy by prioritizing communication performance to precisely enhance sensing performance, in contrast to the performance trade-offs inherent in traditional approaches. Finally, we utilize the sensing and communication (S&C) MI rate as a metric to evaluate whether the beamforming solution can effectively balance sensing and communication performance.

Notations: Bold letters denote vectors and matrices. Italicized letters denote variables. ${[\cdot ]}^{\mathrm{T}}$ and ${[\cdot ]}^{\mathrm{H}}$ denote the transpose operation and the Hermitian operation, respectively. ${ℂ}^{N\times N}$ denotes a $N\times N$ complex matrix. Re[a] denotes the real part of a scalar a. $\mathrm{tr}(A)$ denotes the trace of a matrix A. $\mathbb{E}[\cdot ]$ denotes the expectation operation. $‖a‖$ and ${‖A‖}_{\mathrm{F}}$ denote the l₂-norm of a and the Frobenius norms of A. $\mathcal{CN}(0, {\sigma }^2)$. represents a circularly symmetric complex Gaussian random variable with zero mean and variance ${\sigma }^2$.

2. System Model

In this study, we consider a bistatic ISAC-RSU system comprising a transmit RSU and a receive RSU. Each unit is equipped with a half-wavelength-spaced uniform linear array (ULA), consisting of $N_{\mathrm{t}}$ transmit antennas and $N_r$ receive antennas, respectively. The system configuration is illustrated in Figure 1. The transmit RSU emits S sensing streams toward targets and U communication streams toward UEs in the downlink, enabling simultaneous data transmission and sensing. The targets reflect the transmitted sensing signals, generating echo returns that are captured by the receive RSU. Additionally, the BS transmits a downlink communication signal to the ISAC transmitter, carrying relayed information intended for the UEs. It is assumed that the receive RSU also receives sensing clutter originating from C propagation paths in the surrounding environment.

Figure 1. ISAC-RSU system model using beamforming.

For the sake of simplicity, we consider that the transmit RSU is equipped with digital beamforming capabilities, which enable us to proceed with the establishment of the signal model.

2.1. Signal Model

This sub-section introduces the ISAC transmitted signal model, which is specifically designed to enable target sensing while simultaneously supporting relay communication with UEs. The ISAC transmitter is capable of transmitting U communication streams to cooperative users and S sensing streams to non-cooperative targets. $x_u[l]\in ℂ$ and $x_s[l]\in ℂ$ denote the transmitted communication signal and the transmitted sensing signal for the l-th symbol, respectively. The data symbols of these streams can be accommodated within a fast time snapshot of the ISAC transmitter. The transmitted signal is a linear combination of the communication and sensing signals. Therefore, we exploit the waveform diversity capability of the MIMO ISAC system to transmit distinct signals in specific directions. The transmitted signal $x[l]\in {ℂ}^{N_{\mathrm{t}}}$ is expressed as follows:

$$x[l]={\displaystyle \sum _{u=1}^U{w}_u{x}_u[l]+}{\displaystyle \sum _{s=1}^S{w}_s{x}_s[l]}={\displaystyle \sum _{n=1}^N{w}_n{x}_n[l]},$$

(1)

where $w_u$ and $w_s$ denote transmit beamforming vectors for the u-th user and for the s-th target, respectively, $N=U+S$ denotes the aggregate number of the communication and sensing streams sent by the transmitter, and $w_n\in {ℂ}^{N_{\mathrm{t}}}$ denotes the transmit beamforming vector included $w_u$ and $w_s$.

We assume that there are L symbols in the signal; then, Equation (1) can be expressed as

$$X=\overline{W}\overline{X}\in {ℂ}^{N_{\mathrm{t}}\times L},$$

(2)

$$\overline{W}=[w_1,w_2,\dots ,w_N]\in {ℂ}^{N_{\mathrm{t}}\times N},$$

(3)

$$\overline{X}={[x_1,x_2,\dots ,x_N]}^{\mathrm{T}}\in {ℂ}^{N\times L},$$

(4)

where $\overline{W}$ denotes the transmitted beamforming matrix, and $\overline{X}$ denotes the transmitted symbol sequence matrix; the transmit sequence vector of L symbols for each stream is denoted as

$$x_n={[x_n[1],x_n[2],\dots ,x_n[L]]}^{\mathrm{T}}.$$

(5)

In this analysis, we assume that the messages carried by the sensing and communication signals are statistically independent and that the sensing signals can be generated using pseudo-random coding. Furthermore, we assume that the symbols have unit average energy, $\mathbb{E}[{\left|x_n[l]\right|}^2]=1$.

2.2. Communication Model

We represent $h_u\in {ℂ}^{1\times N_{\mathrm{t}}}$ as the communication stationary channel from the transmitter to the u-th user, excluding non-line-of-sight (N-LOS) channels from consideration. The communication channel matrix $H_c$ can be expressed as

$$H_c={[h_1,h_2,\dots ,h_U]}^{\mathrm{T}}\in {ℂ}^{U\times N_{\mathrm{t}}}.$$

(6)

We assume that the communication channel is a steering matrix from the transmitter to the users, i.e., $H_c=A_u^{\mathrm{T}}$. The steering matrix is given by

$$A_u=[a({\theta }_1),a({\theta }_2),\dots ,a({\theta }_U)],$$

(7)

where $a({\theta }_u)=[1,e^{\mathrm{j}\mathrm{\pi }\sin {\theta }_u},\dots ,e^{\mathrm{j}\mathrm{\pi }(N_{\mathrm{t}}-1)\sin {\theta }_u}]$ denotes the steering vector, and ${\theta }_u$ represents the azimuth angle from the transmitter to the u-th user. Thus, the communication channel matrix $H_c={[a({\theta }_1),a({\theta }_2),\dots ,a({\theta }_U)]}^{\mathrm{T}}$.

For the communication signals received by users, it is assumed that the channel remains constant over the transmission of L consecutive symbols. The signal received by each user consists of five components: the desired communication signal, MUI, sensing interference, BS communication interference, and noise. The received signal at the u-th user can be expressed as

$$\begin{array}{lll}{y}_u[l]& =& h_{u}x[l]+h_u{w}_{\mathrm{g}}{x}_g[l]+n_u[l]\\ & =& h_u{w}_u{x}_u[l]+{\displaystyle \sum _{u\prime \in \mathbf{U}/\left\{u\right\}}{h}_u{w}_{u\prime }{x}_{u\prime }[l]+}{\displaystyle \sum _{s=1}^S{h}_u{w}_s{x}_s[l]}\\ & & +h_u{w}_{\mathrm{g}}{x}_g[l]+n_u[l],\end{array}$$

(8)

where $=\{1,2,\dots ,U\}$, $w_g$ denotes transmit precoder vector of the BS to the transmit RSU, $x_g[l]$ denotes transmit symbol signal of the BS to the transmit RSU, and $n_u[l]~(0,{\sigma }_u^2)$ denotes the receiver noise of u-th user. Notably, since $h_u{w}_{\mathrm{g}}{x}_g[l]$ represents the signal transmitted from the BS to the RSU, it acts as an interference signal for the UE.

Due to the BS’s supplementary role in relaying the communication stream to the transmit RSU, it is equipped with a ULA of $N_{\mathrm{t}}$ transmit antennas spaced at half-wavelength intervals, dedicated to communication. When the BS transmits only the communication stream to the transmit RSU, the beamforming vector $w_g$ can be articulated as follows:

$$w_g=\sqrt{p_g}{a}_g,$$

(9)

where $p_g$ denotes the beamforming power, $a_g=[1,e^{\mathrm{j}\mathrm{\pi }\sin {\theta }_g},\dots ,e^{\mathrm{j}\mathrm{\pi }(N_{\mathrm{t}}-1)\sin {\theta }_g}]\in {ℂ}^{1\times N_{\mathrm{t}}}$ denotes communication channel from BS to transmit RSU, and ${\theta }_g$ represents the azimuth angle from the BS to the transmit RSU.

Upon referring to Equation (8), it can be deduced that the desired communication signal received by the u-th user is denoted as $h_u{w}_u{x}_u[l]$, while the remaining signals are classified as interferences and noise. Consequently, the SINR of the received signal is used as a metric for evaluating communication performance. The SINR for the receiver associated with the u-th user can be expressed as

$$\begin{array}{ll}{\mathrm{SINR}}_u& ={\displaystyle \frac{\mathbb{E}[{\left|h_u{w}_u{x}_u[l]\right|}^2]}{\mathbb{E}[{\left|{\displaystyle \sum _{u\prime \in U/\left\{u\right\}}{h}_u{w}_{u\prime }{x}_{u\prime }[l]}\right|}^2]+\mathbb{E}[{\left|{\displaystyle \sum _{s=1}^S{h}_u{w}_s{x}_s[l]}\right|}^2]+\mathbb{E}[{\left|h_u{w}_g{x}_g[l]\right|}^2]+\mathbb{E}[{\left|n_u[l]\right|}^2]}}\\ & ={\displaystyle \frac{{\left|h_u{w}_u\right|}^2}{{\left|{\displaystyle \sum _{u\prime \in U/\left\{u\right\}}{h}_u{w}_{u\prime }}\right|}^2+{\left|{\displaystyle \sum _{s=1}^S{h}_u{w}_s}\right|}^2+{\left|h_u{w}_g\right|}^2+{\sigma }_u^2}}.\end{array}$$

(10)

2.3. Sensing Model

For the sensing model, we assume the existence of C clutter patches within the environment and take into account the availability of two sensing channels: $G_s$ (transmit RSU–targets–receive RSU) and $G_c$ (transmit RSU–clutter patch–receive RSU), given by

$$G_s={\alpha }_{s}a({\varphi }_s)a^{\mathrm{H}}({\phi }_s),$$

(11)

$$G_c={\alpha }_{c}a({\varphi }_c)a^{\mathrm{H}}({\phi }_c),$$

(12)

where $G_s\in {ℂ}^{N_r\times N_t}$ and $G_c\in {ℂ}^{N_r\times N_t}$ denote the beam-space sensing channel matrices corresponding to the target and clutter, respectively, ${\alpha }_s~\mathcal{CN}(0,{\zeta }_s^2)$, denotes the sensing target channel gain, and ${\alpha }_c~\mathcal{CN}(0,{\zeta }_c^2)$ denotes the sensing clutter channel gain. ${\varphi }_s$ and ${\varphi }_c$ denote the direction of arrival (DOA) of the targets and clutter patches, respectively, as observed at the receive RSU. ${\phi }_s$ and ${\phi }_c$ denote the direction of departure (DOD) from the transmit RSU toward the targets and clutter patches, where $S=\{1,2,\dots ,S\}$.

It is worth noting that we employ the Swerling-I model for the beam-space sensing channel, which assumes slow fluctuations in the RCS. Consequently, the sensing channel remains stable during the transmission of L sensing and communication symbols. The signal received at the receive RSU at time instant l is given by

$$\begin{array}{ll}y[l]& ={\displaystyle \sum _{s=1}^S{G}_{s}x[l]}+{\displaystyle \sum _{c=1}^C{G}_{c}x[l]}+n_s[l]\\ & ={\displaystyle \sum _{s=1}^S{G}_s({\displaystyle \sum _{u=1}^U{w}_u{x}_u[l]+}{\displaystyle \sum _{s=1}^S{w}_s{x}_s[l]})}+{\displaystyle \sum _{c=1}^C{G}_c({\displaystyle \sum _{n=1}^N{w}_n{x}_n[l]})}+n_s[l]\\ & ={\displaystyle \sum _{s=1}^S{G}_s{w}_s{x}_s[l]}+{\displaystyle \sum _{s=1}^S{G}_s({\displaystyle \sum _{u=1}^U{w}_u{x}_u[l]+{\displaystyle \sum _{s\prime \in S//\left\{s\right\}}{w}_{s\prime }{x}_{s\prime }[l]})}}+{\displaystyle \sum _{c=1}^C{G}_c({\displaystyle \sum _{n=1}^N{w}_n{x}_n[l]})}+n_s[l].\end{array}$$

(13)

Assuming the presence of L symbols within the signal, Equation (13) can be reformulated as follows:

$$\begin{array}{ll}Y=& {\displaystyle \sum _{s=1}^S{\alpha }_{s}a({\varphi }_s)a^{\mathrm{H}}({\phi }_s)W_s{X}_s}+{\displaystyle \sum _{s=1}^S{\alpha }_{s}a({\varphi }_s)a^{\mathrm{H}}({\phi }_s)({\displaystyle \sum _{u=1}^U{W}_u{X}_u+{\displaystyle \sum _{s\prime \in S/\left\{s\right\}}{W}_{s\prime }{X}_{s\prime }})}}\\ & +{\displaystyle \sum _{c=1}^C{\alpha }_{c}a({\varphi }_c)a^{\mathrm{H}}({\phi }_c)({\displaystyle \sum _{n=1}^N{W}_n{X}_n})}+N_s,\end{array}$$

(14)

where the desired sensing signals received by the receive RSU are ${\displaystyle {\sum }_{s=1}^S{\alpha }_{s}a({\varphi }_s)a^{\mathrm{H}}({\phi }_s)W_s{X}_s}$, while the other signals are classified as clutter, MUI, and noise. Subsequently, the signal-to-clutter-plus-noise ratio (SCNR) of the received signal is adopted as a metric for performance evaluation. The SCNR of the receiver can be obtained as

$$\mathrm{SCNR}={\displaystyle \frac{\mathbb{E}[{‖{\displaystyle \sum _{s=1}^S{\alpha }_{s}a({\varphi }_s)a^{\mathrm{H}}({\phi }_s)W_s{X}_s}‖}_{\mathrm{F}}^2]}{\mathbb{E}[{‖{\displaystyle \sum _{s=1}^S{\alpha }_{s}a({\varphi }_s)a^{\mathrm{H}}({\phi }_s)({\displaystyle \sum _{u=1}^U{W}_u{X}_u+{\displaystyle \sum _{s\prime \in S/\left\{s\right\}}{W}_{s\prime }{X}_{s\prime }})+}}‖}_{\mathrm{F}}^2]+\mathbb{E}[{‖{\displaystyle \sum _{c=1}^C{\alpha }_{c}a({\varphi }_c)a^{\mathrm{H}}({\phi }_c)({\displaystyle \sum _{n=1}^N{W}_n{X}_n})}‖}_{\mathrm{F}}^2]+\mathbb{E}[{‖N_s‖}_{\mathrm{F}}^2]}}.$$

(15)

The numerator of the SCNR represents the mathematical expectation of the desired sensing signal, while the denominator includes the sum of the mathematical expectations of multi-user interference, clutter from patches, and noise. Notably, the incorporated clutter patch enhances the SCNR closed-form expression, allowing for a better representation of complex urban environments. This study focuses exclusively on optimizing ISAC transmission beamforming and does not address clutter mitigation techniques at the receive RSU.

3. A Hierarchical Optimization Solution for ISAC Beamforming

In this section, our objective is to obtain optimal communication beamforming vectors ${\left\{w_u\right\}}_{u\in U}$ and sensing beamforming vectors ${\left\{w_s\right\}}_{s\in S}$ by jointly optimizing communication SINR and sensing SCNR. We propose a hierarchical optimization solution for ISAC beamforming that enables a seamless transition from communication-centric to sensing-centric design, thereby facilitating progressive convergence toward a globally optimal solution. The complex joint optimization problem is decomposed into multiple interrelated sub-problems. Specifically, communication beamforming is formulated as leader-layer optimization, while sensing beamforming is addressed in follower-layer optimization, allowing for coordinated optimization toward the overall objective. Notably, this hierarchical decomposition optimization is conceptually analogous to bi-level formulations, with its primary innovation lying in structural organization rather than the introduction of a novel optimization principle.

3.1. A Maximize–Minimize Optimization Solution Based on the Bisection Method

For leader-layer optimization, our objective is to maximize the SINR, as higher SINR directly improves communication beamforming performance. Consequently, we adopt a max–min optimal strategy to achieve robust SINR enhancement and determine the associated beamforming variables ${\left\{w_u\right\}}_{u\in U}$. The problem can be formulated as

$$\underset{\left\{w_u\right\}}{\max } \underset{\left\{u\right\}}{\min }{\displaystyle \frac{{\left|h_u{w}_u\right|}^2}{{\left|{\displaystyle \sum _{u\prime \in U/\left\{u\right\}}{h}_u{w}_{u\prime }}\right|}^2+{\left|{\displaystyle \sum _{s=1}^S{h}_u{w}_s}\right|}^2+{\left|h_u{w}_g\right|}^2+{\sigma }_u^2}}$$

(16)

$$\mathrm{s}.\mathrm{t}. {{\displaystyle \sum _{u\in U}‖w_u‖}}^2\le P^{(c)},$$

(17)

where $P^{(c)}$ denotes the communication power and total transmitter power $P$ consists of communication power $P^{(c)}$ and sensing power $P^{(s)}$, given by

$$P=P^{(c)}+P^{(s)}.$$

(18)

Constraint (18) is a Pareto boundary constraint (power allocation trade-off). To determine the optimal solution for Problem (16), we establish a lower bound for the SINR applicable to all users.

$${\mathrm{SINR}}_u={\gamma }_u, u=1,\dots ,U,$$

(19)

$${\gamma }_0=\underset{\left\{u\right\}}{\min }{\gamma }_u,$$

(20)

which implies ${\mathrm{SINR}}_u\ge {\gamma }_0, \forall u\in U$. ${\gamma }_0$ denotes the lowest SINR received by U users. Thus, the constraint is given by

$${\displaystyle \frac{{\left|h_u{w}_u\right|}^2}{{\displaystyle \sum _{u\prime \in \mathbb{U}/\left\{u\right\}}{\left|h_u{w}_{u\prime }\right|}^2}+{\displaystyle \sum _{s=1}^S{\left|h_u{w}_s\right|}^2}+{\left|h_u{w}_g\right|}^2+{\sigma }_u^2}}\ge {\gamma }_0,$$

(21)

where we can re-write the constraint as

$$(1+{\displaystyle \frac{1}{{\gamma }_0}}){\left|h_u{w}_u\right|}^2\ge {\left|h_u{w}_u\right|}^2+{\displaystyle \sum _{u\prime \in \mathbb{U}/\left\{u\right\}}{\left|h_u{w}_{u\prime }\right|}^2}+{\displaystyle \sum _{s=1}^S{\left|h_u{w}_s\right|}^2}+{\left|h_u{w}_g\right|}^2+{\sigma }_u^2.$$

(22)

Due to $N=U+S$, it can be simplified to obtain

$$(1+{\displaystyle \frac{1}{{\gamma }_0}}){\left|h_u{w}_u\right|}^2\ge {\displaystyle \sum _{n=1}^N{\left|h_u{w}_n\right|}^2}+{\left|h_u{w}_g\right|}^2+{\sigma }_u^2.$$

(23)

We take the square root of constraint (23) and reformulate it as a second-order cone constraint. Since the left-hand side of the inequality is a nonlinear function, we apply an arbitrary phase shift to transform the variable into its real part, a manipulation that preserves the squared norm, i.e.,

$$\sqrt{{\left|h_u{w}_u\right|}^2}=\left|h_u{w}_u{e}^{-j\psi }\right|=\mathrm{Re}\{h_u{w}_u\},$$

(24)

where $\psi \in [0,2\pi ]$ denotes the arbitrary angle. Constraint (23) can be expressed as

$$\sqrt{(1+{\displaystyle \frac{1}{{\gamma }_0}})}\mathrm{Re}\{h_u{w}_u\}\ge ‖{[h_u{w}_1,\dots ,h_u{w}_N,h_u{w}_g,{\sigma }_u]}^{\mathrm{T}}‖.$$

(25)

Constraint (25) is formulated as a second-order cone constraint. Since the objective function is quasi-concave, the optimization problem in Problem (16) can be transformed into the following feasibility problem.

$$\underset{\left\{w_u\right\}}{\max }{\displaystyle \frac{{\left|h_u{w}_u\right|}^2}{{\left|{\displaystyle \sum _{u\prime \in U/\left\{u\right\}}{h}_u{w}_{u\prime }}\right|}^2+{\left|{\displaystyle \sum _{s=1}^S{h}_u{w}_s}\right|}^2+{\left|h_u{w}_g\right|}^2+{\sigma }_u^2}}$$

(26)

$$\mathrm{s}.\mathrm{t}. \sqrt{(1+{\displaystyle \frac{1}{{\gamma }_0}})}\mathrm{Re}\{h_u{w}_u\}\ge ‖{[h_u{w}_1,\dots ,h_u{w}_N,h_u{w}_g,{\sigma }_u]}^{\mathrm{T}}‖,$$

(27)

$${{\displaystyle \sum _{u\in U}‖w_u‖}}^2\le P^{(c)}.$$

(28)

In addressing Problem (26), the presence of unknown variables ${\gamma }_0$ and ${\left\{w_s\right\}}_{s\in S}$ makes it impractical to directly solve for ${\left\{w_u\right\}}_{u\in U}$. Recent studies have tackled this issue by introducing a lower bound as a surrogate for ${\gamma }_0$, treating ${\gamma }_0$ as a known parameter when optimizing the other variables [30]. However, the effectiveness of this approach depends critically on the choice of the fixed parameter; significant deviation from the true optimal value can degrade the performance of the remaining variables.

To ensure optimized communication performance for users and achieve a sufficiently high SINR, it is essential to determine ${\left\{w_s\right\}}_{s\in S}$, which in turn enhances the value of ${\gamma }_0$. Therefore, we adopt a communication-centric beamforming strategy to design ${\left\{w_s\right\}}_{s\in S}$ with the objective of maximizing ${\gamma }_0$. The sensing beamforming vectors, constructed via conjugate beamforming, are projected into the null space of the communication channels. This strategy effectively mitigates the interference of sensing transmissions on communication performance, thereby improving the user’s SINR. The variables ${\left\{{\overline{w}}_s\right\}}_{s\in S}$ and ${\left\{w_s^{\mathrm{NS}}\right\}}_{s\in S}$ can be expressed as follows:

$${\overline{w}}_s^{\mathrm{NS}}=(I-H_c^{\mathrm{H}}{(H_c{H}_c^{\mathrm{H}})}^{†}{H}_c)a({\phi }_s)$$

(29)

$$w_s^{\mathrm{NS}}=\sqrt{P^{(s)}}{\displaystyle \frac{{\overline{w}}_s^{\mathrm{NS}}}{‖{\overline{w}}_s^{\mathrm{NS}}‖}}.$$

(30)

After the ${\left\{{\overline{w}}_s\right\}}_{s\in S}$ is replaced by the ${\left\{w_s\right\}}_{s\in S}$, constraint (25) can be reformulated as a second-order cone problem, which can be solved using a convex solver. Specifically, we apply the standard bisection method to determine the maximum SINR value, denoted as ${\gamma }_0$, that satisfies the given constraints. First, we establish ${\gamma }_{\min }$ and ${\gamma }_{\max }$ as the lower and upper bounds of the SINR range, respectively, initialized based on the theoretical limits of feasible SINR values. At each iteration, the algorithm computes the midpoint of the current interval and uses the CVX solver to assess its feasibility. If a feasible solution is obtained, the lower bound is updated to this midpoint; otherwise, the upper bound is reduced to it. This iterative process continues until the gap between the upper and lower bounds falls below or equals $\lambda =0.01$. By progressively narrowing the search interval, the method efficiently converges to an accurate approximation of the optimal SINR. Finally, we can determine the variable ${\left\{w_u\right\}}_{u\in U}$ in accordance with the optimal SINR.

Furthermore, a concise proof of convexity for problem (26) is required to establish a theoretical foundation for the convergence and optimality of the optimization algorithm. First, the power constraint ${{\displaystyle {\sum }_{u\in U}‖w_u‖}}^2\le P^{(c)}$ defines a closed convex set, specifically a 2-norm ball in ${ℂ}^N$. Subsequently, objective function (27) can be reformulated as a constraint ${\mathrm{SINR}}_u\ge \zeta , u=1,\dots U$ featuring a quadratic function of ${\left\{w_u\right\}}_{u\in U}$ on the left-hand side and a constant on the right-hand side. Since $|h_u{w}_u{|}^2=w_u^{\mathrm{H}}{h}_u^{\mathrm{H}}{h}_u{w}_u$ is a semi-positive semi-definite quadratic form, the constraint inequality can be expressed as $w_u^{\mathrm{H}}{A}_k{w}_u\ge b_k$, and its feasible domain is the convex set (the leader-layer set of the semi-positive definite quadratic function). Constrant (25) is transformed by constrant (23), and it also transforms the semi-positive definite quadratic form. The objective function is convex, and the feasible region defined by the constraints constitutes a convex set. Consequently, the leader-layer optimization problem qualifies as a convex optimization problem, and the bisection method is capable of converging to the global optimum under appropriate conditions. Additionally, the optimality of problem (26) should be verified by examining the corresponding dual analysis. Algorithm 1 provides a concise overview of the leader-layer optimization process. For detailed derivations and explanations, please refer to Appendix A.

In summary, this sub-section introduces a maximin optimization method that utilizes the sensing beamforming vector—constructed via conjugate beamforming—and the optimal communication beamforming vector as the final outputs. However, although the solution guarantees optimal communication performance, it does not necessarily achieve optimal sensing performance, as the strategy is inherently communication-centric. In the following section, we retain the optimized ${\left\{w_u\right\}}_{u\in U}$ derived in this section and use it as a fixed component to jointly optimize the sensing-centric beamforming vector ${\left\{w_s\right\}}_{s\in S}$. Furthermore, we derive the convergence of leader-layer optimization in Appendix B.1.

Algorithm 1. Leader-layer optimization
Input: ${\gamma }_{\min }$, ${\gamma }_{\max }$ and $\lambda$
Output: ${\left\{w_u\right\}}_{u\in U}$
1	Initialize:
2	Convert (16) to ${\mathrm{SINR}}_u\ge {\gamma }_0, u=1,\dots U.$
3	Convert (22) to second-order cone constraint (24).
4	Compute ${\left\{w_s^{\mathrm{NS}}\right\}}_{s\in S}$ by (26) and (27).
5	Replace ${\left\{w_s^{\mathrm{NS}}\right\}}_{s\in S}$ with ${\left\{w_s\right\}}_{s\in S}$.
6	Maximize ${\mathrm{SINR}}_u$ with constraint (24) and ${\displaystyle {\sum }_{u\in U}{‖w_u‖}^2\le P^{(c)}}$.
7	Optimization of ${\gamma }_0$.
8	Repeat
9	Establish ${\gamma }_{\min }(t)$ and ${\gamma }_{\max }(t)$ as the lower and upper bounds.
10	Compute the midpoint ${\gamma }_0(t)$ of the current range.
11	Use the CVX solvers to assess its feasibility.
12	If a feasible solution is obtained, ${\gamma }_{\min }(t+1)={\gamma }_0(t)$.
13	Otherwise, ${\gamma }_{\max }(t+1)={\gamma }_0(t)$.
14	Update midpoint ${\gamma }_0(t+1)$
15	Until ${\gamma }_{\max }(t+1)-{\gamma }_{\min }(t+1)<\lambda$ convergence
16	Obtain ${\gamma }_0, {\left\{w_s^{\mathrm{NS}}\right\}}_{s\in S}$
17	Solving for (25) results in ${\left\{w_u\right\}}_{u\in U}$.

3.2. Semi-Definite Programming Solvers Based on the CVX Toolbox

For the follower layer, the sensing performance is mainly measured based on the value of SCNR. Thus, we present a sensing SCNR maximization optimization problem, given by

$$\underset{\left\{w_s\right\}}{\max } \mathrm{SCNR}$$

(31)

$$\mathrm{s}.\mathrm{t}.{{\displaystyle \sum _{s\in S}‖w_s‖}}^2\le P^{(s)}.$$

(32)

where the SCNR is defined in Equation (15).

We reformulated the SCNR optimization problem to an SDP which better optimizes the maximization problem, and ${\alpha }_{s}a({\varphi }_s)a^{\mathrm{H}}({\phi }_s)W_s={\overline{G}}_s, s=1,\dots ,S$. Thus, the numerator of SCNR in Equation (15) can be simplified as

$$\begin{array}{l}\mathbb{E}[{‖{\displaystyle \sum _{s=1}^S{\overline{G}}_s{X}_s}‖}_{\mathrm{F}}^2]={\displaystyle \sum _{s=1}^S\mathbb{E}[{‖{\overline{G}}_s{X}_s‖}_{\mathrm{F}}^2]}\\ ={\displaystyle \sum _{s=1}^S\mathrm{Tr}\left\{\mathbb{E}[X_s{X}_s^{\mathrm{H}}] \mathbb{E}[{\overline{G}}_s^{\mathrm{H}}{\overline{G}}_s]\right\}}\\ ={\displaystyle \sum _{s=1}^S\mathrm{Tr} \mathbb{E}[{\overline{G}}_s^{\mathrm{H}}{\overline{G}}_s]},\end{array}$$

(33)

which is obtained by expansion of the Frobenius norm, interchanging expectation and trace, where $\mathbb{E}[X_s{X}_s^{\mathrm{H}}]=I$. Further, Equation (33) can be simplified as

$$\begin{array}{l}={\displaystyle \sum _{s=1}^S\mathrm{Tr} \mathbb{E}[{\left|{\alpha }_s\right|}^{2}a({\varphi }_s)a^{\mathrm{H}}({\phi }_s)W_s{W}_s^{\mathrm{H}}a({\phi }_s)a^{\mathrm{H}}({\varphi }_s)]}={\displaystyle \sum _{s=1}^S{\zeta }_s^2{‖a^{\mathrm{H}}({\phi }_s)W_s‖}^2}\\ ={\displaystyle \sum _{s=1}^S{\zeta }_s^2}\mathrm{Tr} (A_s{\displaystyle \sum _{s=1}^S{\overline{W}}_s}),\end{array}$$

(34)

where ${\mathit{A}}_s=a({\varphi }_s)a^{\mathrm{H}}({\phi }_s), {\overline{W}}_s=w_s{w}_s^{\mathrm{H}}, \forall s\in S$, and the constraint ${\mathrm{SINR}}_u\ge {\gamma }_0$ are rewritten as

$$(1+{\displaystyle \frac{1}{{\gamma }_0}})\mathrm{Tr} (Q_u{\overline{W}}_u)-\mathrm{Tr} (Q_u{\displaystyle \sum _{n=1}^N{\overline{W}}_n})-\mathrm{Tr} (Q_u{\overline{W}}_a)\ge {\sigma }_u^2$$

(35)

where $Q_u=h_u^{\mathrm{H}}{h}_u$, ${\overline{W}}_u=w_u{w}_u^{\mathrm{H}}$, and ${\overline{W}}_g=w_g{w}_g^{\mathrm{H}}$.

In the following, we not only optimize the SCNR but also incorporate the revised SINR constraint Equation (36) into the SCNR optimization framework.

$$\mathrm{Tr} (Q_u{\displaystyle \sum _{s=1}^S{\overline{W}}_s})\le (1+{\displaystyle \frac{1}{{\gamma }_0}})\mathrm{Tr} (Q_u{\overline{W}}_u)-{\sigma }_u^2-\mathrm{Tr} (Q_u{\displaystyle \sum _{u=1}^U{\overline{W}}_u})-\mathrm{Tr} (Q_u{\overline{W}}_g),$$

(36)

Thus, a reformulated SCNR optimization problem can be expressed as Problem (37).

$$\underset{\left\{w_s\right\}}{\max }{\displaystyle \sum _{s=1}^S{\zeta }_s^2}\mathrm{Tr} (A_s{\displaystyle \sum _{s=1}^S{\overline{W}}_s})$$

(37)

$$\mathrm{s}.\mathrm{t}. \mathrm{Tr} (Q_u{\displaystyle \sum _{s=1}^S{\overline{W}}_s})\le (1+{\displaystyle \frac{1}{{\gamma }_0}})\mathrm{Tr} (Q_u{\overline{W}}_u)-{\sigma }_u^2-\mathrm{Tr} (Q_u{\displaystyle \sum _{u=1}^U{\overline{W}}_u})-\mathrm{Tr} (Q_u{\overline{W}}_g),$$

(38)

$${\displaystyle \sum _{s\in \mathbb{S}}\mathrm{Tr} ({\overline{W}}_s)}\le P^{(s)},$$

(39)

where $Q_u$, ${\overline{W}}_u$, ${\overline{W}}_g$ and ${\gamma }_0$ are known. Problem (37) is an SDP that can be efficiently solved using CVX toolbox and standard convex SDP solvers [26,27]. The covariance matrix ${\overline{W}}_s$ obtained from the SDP solution typically has high rank (i.e., non-rank-1). Finally, we can obtain the sensing beamforming vector $w_s$.

Furthermore, a concise proof of convexity for problem (37) is required. The semi-positive definite constraint implies that $W_s\succcurlyeq 0$ belongs to the cone of positive semi-definite matrices, which forms a convex cone. The power constraint on ${\displaystyle {\sum }_{s\in S}\mathrm{Tr} ({\overline{W}}_s)}\le P^{(s)}$ is linear, and its feasible region thus constitutes a convex set. The objective function is linear, and the SDP problem is convex optimization subclass. Algorithm 2 provides a concise overview of the follower -layer optimization process.

Algorithm 2. Follower-layer optimization
Input: ${\left\{h_u, w_u\right\}}_{u\in U}$, $w_g$ $\mathrm{and} {\gamma }_0$
Output: ${\left\{w_s\right\}}_{s\in S}$
1	Initialize:
2	Convert numerator of SCNR to (30).
3	$\mathrm{Convert} {\mathrm{SINR}}_u\ge {\gamma }_0$ to (32).
4	$\mathrm{Compute} Q_u=h_u^{\mathrm{H}}{h}_u$$, {\overline{W}}_u=w_u{w}_u^{\mathrm{H}}$ $\mathrm{and} {\overline{W}}_g=w_g{w}_g^{\mathrm{H}}$.
5	$\mathrm{Replace}{\left\{w_s^{\mathrm{NS}}\right\}}_{s\in S}$$\mathrm{with}{\left\{w_s\right\}}_{s\in S}$.
6	$\mathrm{Maximize} {\displaystyle {\sum }_{s=1}^S{\zeta }_s^2\mathrm{Tr} (A_s{\displaystyle {\sum }_{s=1}^S{\overline{W}}_s})}$ $\mathrm{with} \mathrm{constraint} (32) \mathrm{and}{\displaystyle {\sum }_{s\in S}{‖w_s‖}^2\le P^{(s)}}$.
7	Convert problem to the SDP form.
8	$\mathrm{Optimization} \mathrm{of}{\left\{w_s\right\}}_{s\in S}$.
9	$\mathrm{Use} \mathrm{the} \mathrm{CVX} \mathrm{solvers} \mathrm{to} \mathrm{assess} \mathrm{its} \mathrm{feasibility} ({\overline{W}}_s$ is non-rank-1).
10	$\mathrm{Solving} \mathrm{for} \left(33\right) \mathrm{results} \mathrm{in}{\left\{w_s\right\}}_{s\in S}$.

This hierarchical optimization solution prioritizes communication performance at the leader-layer level, thereby implementing a communication-centric design. However, while optimizing the communication beamforming vectors, the solution simultaneously accounts for sensing performance. Consequently, the approach not only significantly enhances communication quality but also achieves near-optimal performance for both sensing and communication functionalities. Furthermore, we derive the convergence of follower-layer optimization in Appendix B.2.

To assess whether the hierarchical optimization outcome approaches global optimality, we employ the sensing-and-communication mutual information (S&C MI) rate as a unified metric for evaluating integrated S&C performance. The S&C MI rate is defined as

$$R_{sc}={\omega }_c{R}_c+{\omega }_s{R}_s,$$

(40)

$$R_c=\log \det (I+{\displaystyle \sum _{u\in U}{\mathrm{SINR}}_u}),$$

(41)

$$R_s=\log \det (I+\mathrm{SNR}).$$

(42)

where ${\omega }_c$ denotes the communication power ratio, and ${\omega }_s$ denotes the sensing power ratio, ${\omega }_c+{\omega }_s=1$. The S&C MI combines sensing mutual information (SMI) and communication mutual information (CMI) as a novel metric for assessing the performance of ISAC. The CMI rate represents the maximum achievable channel coding rate, while SMI is an increasing function of SNR, and both metrics exhibit similarities in their physical and mathematical properties. In addition, their mathematical isomorphism is evident in shared concave–convex properties and logarithmic dependencies—enabling direct cross-domain performance comparison while preserving the unique physical interpretations of each subsystem.

In the subsequent analysis, it is necessary to evaluate the trade-off between communication and sensing performance based on the S&C MI rate. We minimize sensing interference on communication by projecting the sensing beam ($w_s$) onto the null space of the communication channel ($h_u$) as formalized in Equation (26), thereby reducing the ${\left|{\displaystyle {\sum }_{s=1}^S{h}_u{w}_s}\right|}^2$ term in the denominator of Equation (10). The underlying principle of this optimization framework is to shrink the inferior region of the Pareto frontier, effectively driving the SINR value closer to their theoretical upper bounds under a given power budget. This results in a more compact and efficient Pareto boundary, which is subsequently validated through simulation. Notably, SINR–SCNR coupling can influence the Pareto frontier, and we provide analytical expressions in Appendix C.

In the following, we analyze the complexity of the algorithm; the computational complexity of ISAC hierarchical optimization communication to sensing (ISAC-OCS) consists of two parts, where the complexity of solving the bisection method is $𝒪{((\log }_2\mathrm{Y})U(U{L}^3{N}_t^3\left)\right)$ and the complexity of solving the SDP problem is $𝒪(L^{3.5}{N}_t^{3.5}(US\left)\right)$. Thus, the complexity of ISAC-OCS is $𝒪{((\log }_2\mathrm{Y})(U^2{L}^3{N}_t^3)+L^{3.5}{N}_t^{3.5}(US\left)\right)$ and the complexity of maximize–minimize optimization (MMO) is $𝒪{((\log }_2\mathrm{Y})(U^2{L}^3{N}_t^3\left)\right)$ in [26]. We compare the complexity of other algorithms; the complexity of Weighted Minimum Mean Square Error ISAC (WMMSE-ISAC) is $𝒪((U+1)N_t^3+U{R}^3+N_r^3)$ in [25] and the complexity of CM waveform design is $𝒪(L^3{N}_R^3+N_D{N}_A(2M+3)L^2{N}_T^2)$ in [29].

4. Simulation Results

In this section, to verify the effectiveness of ISAC-OCS, we evaluate the sensing and communication performance associated with the previously discussed ISAC beamforming solutions. We consider a scenario in which the transmit RSU simultaneously delivers $U=5$ data streams to UEs and transmits $S=1$ sensing signal to the target, while the receive RSU captures $C=3$ clutter patches. Both the transmit RSU and BS are each equipped with $N_{\mathrm{t}}=16$ transmit antennas, and the receive RSU is additionally equipped with $N_r=16$ receive antennas. The parameters for the communication model include a specified communication noise variance ${\sigma }_u^2=1$. For the sensing model, we consider parameters such as sensing noise variance ${\sigma }_s^2=1$ and sensing channel gain variance ${\zeta }_s={\zeta }_c=1$. Additionally, we set the transmit power of the ISAC transmitter at P = 0 dBW, along with a defined Lagrangian parameter $\lambda =0.1$ and noise variance ${\sigma }^2=1$. In our analysis, we randomize the positions of the transmit RSU, receive RSU, UEs, and target, resulting in randomized azimuth angles for these entities. Under this setup, we conduct a comparative evaluation of the MMO beamforming method [26] and the WMMSE-ISAC method [25] in relation to the ISAC-OCS. Notably, ISAC-OCS demonstrates a significant advancement in methodological design compared to MMO and WMMSE-ISAC. MMO focuses solely on communication beamforming design and lacks an integrated perceptual SCNR optimization component. Although WMMSE-ISAC combines perceptual mutual information and communication mutual information through weighted summation into a unified objective for joint optimization, this approach is susceptible to substantial performance degradation in one of the functionalities due to inappropriate weight selection. All simulation experiments presented in this section were conducted in a standardized computational environment to ensure the reliability and reproducibility of the results. For the software framework, MATLAB R2024a was employed as the core platform. In terms of hardware configuration, the experiments were run on a workstation equipped with an Intel® Core™ i9-14900HX processor.

To evaluate the angular resolution of various beamforming techniques, we conduct a comparative analysis of the beam patterns generated by three distinct methods, with the communication power ratio set to 0.5. As illustrated in Figure 2, the proposed multi-beam architecture simultaneously incorporates both communication-directed and sensing-oriented beams, necessitating joint evaluation using two critical metrics, (i) the absolute peak intensity of beam lobes, and (ii) inter-peak isolation, characterized by null depth in concave regions.

Figure 2. Multi-beam pattern comparisons for three methods.

The MMO method adopts a communication-centric beamforming strategy, resulting in a beam pattern with lower peaks in the sensing beams (approximately 2.3 dB lower than ISAC-OCS) and higher peaks in the communication beams (up to 6.2 dB higher than ISAC-OCS) compared to the other approaches. However, its shallow inter-beam null depths fundamentally limit angular discrimination capability due to insufficient suppression of sidelobe interference. In contrast, the ISAC-OCS solution enhances the peak gains of sensing beams (peak gain > 0.9 dB relative to WMMSE-ISAC) at the expense of communication beam attenuation (peak reduction ≈ 1.8 dB relative to WMMSE-ISAC). Notably, it achieves significantly deeper null depths (≥12 dB) between adjacent beams compared to both MMO and WMMSE-ISAC, owing to stronger interference suppression. By analyzing the communication beam peak characteristics of this solution, we conclude—as previously discussed—that it does not follow a conventional communication-centric beamforming strategy. Instead, it enables joint optimization of communication and sensing performance. In summary, although the ISAC-OCS yields relatively lower communication peaks, it exhibits more pronounced concave drops between adjacent lobes, thereby enhancing beamforming angular resolution.

After clutter suppression is applied to the received echo signals, target angle estimation is performed. Figure 3 shows the normalized spatial spectrum obtained via the MUSIC algorithm for three optimized beamforming methods. As a high-resolution subspace method, the MUSIC algorithm theoretically achieves superior resolution compared to conventional beamforming techniques, exceeding the limitations imposed by the Rayleigh criterion. Given that the experiment focuses on angle estimation for a single target, the utilization of four receiving antenna sensors suffices. The findings demonstrate that three optimized methods effectively detect the target angle; however, the beam spectrum peak generated by the ISAC-OCS design is notably the sharpest, signifying enhanced spatial resolution and more effective sidelobe suppression. The collateral valve inhibition achieved by ISAC-OCS exceeds that of MMO by more than 1.6 dB and outperforms WMMSE-ISAC by over 1.2 dB. Furthermore, the angle estimation accuracy associated with ISAC-OCS surpasses that of the other two methods, thereby corroborating its superior sensing performance.

Figure 3. Normalized spectrum of the MUSIC algorithm for three beamforming methods.

The Root Mean Square Error (RMSE) obtained from DOA estimation serves as a crucial metric for evaluating the perceptual performance of beamforming design. The experimental framework employs rigorous Monte Carlo validation to ensure a comprehensive performance assessment. Figure 4 illustrates a comparison of the RMSE in angle estimation across the three optimized beamforming methods’ varying noise power levels. As noise power diminishes to lower values, the RMSE values for all three methods converge, indicating minimal differences in estimation performance under low-noise conditions. Nevertheless, throughout the entire noise power spectrum, the RMSE corresponding to ISAC-OCS remains consistently lower than other beamforming methods, evidencing its ability to sustain high estimation accuracy even in complex noise environments. This result further substantiates the reliability of the ISAC-OCS beamforming design approach with respect to sensing performance.

Figure 4. RMSEs versus the noise power for three beamforming methods.

Beyond angular resolution, the joint performance of communication and sensing is evaluated via SMI and CMI. With the sensing power ratio ranging from 0.01 to 0.99, the SMI rate at SNR = 0 dB and SNR = 5 dB is illustrated in Figure 5. This investigation evaluates the characteristics of SMI across three distinct methodologies under varying conditions of sensing power allocation and signal quality.

Figure 5. The SMI rate versus sensing power ratio for three beamforming methods.

The systematic comparison demonstrates that the ISAC-OCS technique consistently outperforms both WMMSE-ISAC and MMO methods in terms of sensing performance, with this advantage remaining evident across various channel quality conditions. All methods examined exhibit fundamentally similar behavioral patterns, where enhanced signal conditions yield proportional improvements in SMI rates at fixed sensing power configurations. When the sensing power is 0.2, the SMI rate of ISAC-OCS is about 0.65 dB higher than that of MMO and approximately 0.1 dB higher than that of WMMSE-ISAC. The overall findings indicate that although all approaches experience improvements in enhanced signal environments, the ISAC-OCS method emerges as the most robust solution for improving sensing performance.

For communication performance, Figure 6 considers two scenarios in which the transmitting SNR is established at 0 dB and 5 dB, respectively. The figure illustrates the fluctuation of the CMI rate in relation to the communication power ratio, utilizing three distinct beamforming optimization solutions across different SNR levels. For each curve, 200 Monte Carlo experiments are performed. In Figure 6, all methodologies exhibit fundamentally similar behavioral patterns, characterized by an enhancement in CMI rates as the allocation of communication power increases across the entire operational range. Notably, the MMO and ISAC-OCS techniques, which share identical theoretical frameworks for deriving beamforming vectors, demonstrate almost indistinguishable communication performance results (the CMI rate of ISAC-OCS is marginally higher than that of MMO, with a difference of approximately 0.08 dB). Both methodologies consistently outperform the WMMSE-ISAC approach across all assessed SNR conditions, thereby confirming their superior efficacy in maintaining communication quality.

Figure 6. The CMI rate versus communication power ratio for three beamforming methods.

Figure 7 examines the CMI rate of the three optimized beamforming methods as a function of user distance under varying transmission SNRs. The experimental results reveal that, irrespective of transmission SNR fluctuations, the CMI rates for ISAC-OCS and MMO outperform those of WMMSE-ISAC. Although ISAC-OCS and MMO exhibit comparable overall trends, a detailed analysis indicates that ISAC-OCS achieves marginally higher rates (the CMI rate of ISAC-OCS is marginally higher than that of MMO, with a difference of approximately 0.09 dB), highlighting its advantage in communication performance.

Figure 7. The CMI rate versus UE distance for three beamforming methods.

Similarly, Figure 8 evaluates the CMI rate as the number of users varies under different transmission SNR conditions, yielding results consistent with those in Figure 6 and Figure 7: the CMI rate for ISAC-OCS consistently exceeds that of MMO and is substantially superior to WMMSE-ISAC (the CMI rate of ISAC-OCS is marginally higher than that of MMO, with a difference of approximately 0.07 dB). These findings collectively affirm the comprehensive performance benefits of the ISAC-OCS beamforming strategy within integrated communication and sensing systems. The experimental results substantiate that the proposed methodology achieves communication performance comparable to that of MMO while substantially outperforming WMMSE-ISAC. This finding highlights the necessity for further investigation into the comparative sensing performance characteristics of these competing schemes, as the communication-sensing trade-off is a critical design consideration in integrated systems.

Figure 8. The CMI rate versus number of UEs for three beamforming methods.

Furthermore, it is observed that the CMI rate decreases as the number of UEs increases. This is because more UEs means more communication interference. On the one hand, more MUI degrades the performance of communication. On the other hand, the more communication to sensing interference leads to the decrease in SMI rate. In Figure 9, we compare the ISAC-OCS hierarchical optimization method with the deep learning-based method [31], observing the differences in the CMI rate and SMI rate between the two methods.

Figure 9. The performance versus the number of UEs for two methods.

As the number of UEs increases, the rate of all four schemes exhibits a decreasing trend. The SMI (DL-based ISAC) consistently achieves a higher rate than SMI (ISAC-OCS) across all UE numbers. As the number of UEs increases from 2 to 6, the CMI (ISAC-OCS) achieves a higher rate than CMI (DL-based ISAC). The advantage of DL-based ISAC may stem from its advanced deep learning-driven resource allocation and information processing mechanisms, which enable better adaptability to the growing number of UEs. However, the DL-based ISAC method is highly reliant on high-quality training data. Performance degradation occurs significantly when there is a discrepancy between the actual operational scenario and the training dataset. In contrast, ISAC-OCS does not require an offline training phase. Although its SMI rate is lower than that of DL-based approaches, it operates without dependence on specialized computing hardware such as GPUs. Moreover, unlike DL-based ISAC methods, which incur substantial computational costs during model training, ISAC-OCS achieves efficient resource utilization. Consequently, ISAC-OCS demonstrates superior adaptability in complex and dynamic environments, particularly with respect to computational efficiency and practical deployment.

While ISAC-OCS demonstrates communication performance comparable to MMO, it has been specifically optimized for sensing capabilities, leading to enhanced sensing performance relative to the MMO method and superior overall performance compared to WMMSE-ISAC, as supported by Figure 3, Figure 4 and Figure 5. To further explore the joint communication and sensing performance across different methods, we analyze the S&C rate in relation to power allocation and SNR.

In Figure 10, we investigate S&C rate variations as functions of communication power ratios for different schemes under transmit SNR values of 0 dB and 5 dB, respectively. Each performance curve is obtained by averaging over 200 independent Monte Carlo trials to ensure statistical robustness. As anticipated, the S&C rate curves corresponding to higher SNRs consistently surpass those obtained at lower SNRs, indicating the significant influence of signal quality on overall system performance. Among the evaluated methods, ISAC-OCS exhibits marginally higher S&C rates than both WMMSE-ISAC and MMO across the entire range of communication power ratios (when the communication power is approximately 0.5, the S&C MI rate of ISAC-OCS is about 0.41 dB higher than that of MMO and approximately 0.29 dB higher than that of WMMSE-ISAC).

Figure 10. The S&C rate versus communication power allocation for three beamforming methods.

Furthermore, it is observed that the S&C rates achieved at intermediate power allocation settings (${\omega }_c=0.5$ and ${\omega }_c=0.75$) are generally lower than those obtained under extreme optimization biases (${\omega }_c=0.01$ or ${\omega }_c=0.99$). This indicates that the overall performance at these intermediate points fails to match the specialized effectiveness achieved in purely sensing-centric or communication-centric configurations. However, allocating power according to balanced ratios (${\omega }_c=0.5$ and ${\omega }_c=0.75$) is beneficial in scenarios that require the simultaneous fulfillment of both sensing and communication needs, providing a trade-off solution under practical system constraints. In contrast, the adoption of extreme power allocation (${\omega }_c=0.01$ or ${\omega }_c=0.99$) is only appropriate for scenarios dominated by either communication or sensing demands and may result in significant performance degradation if applied indiscriminately in integrated operational environments.

In Figure 11, the communication power ratios are set at 0.5 and 0.25, respectively. The figure illustrates the variation in the S&C MI rate curves as functions of SNR for the different examined methods. Evidently, the S&C MI rates achieved by the ISAC-OCS scheme consistently surpass those attained by the baseline approaches across the entire SNR range (when the noise power is approximately 1, the S&C MI rate of ISAC-OCS is about 0.52 dB higher than that of MMO and approximately 0.28 dB higher than that of WMMSE-ISAC). This enhanced performance highlights the efficacy of ISAC-OCS in simultaneously optimizing sensing and communication objectives.

Figure 11. The S&C rate versus noise power for three beamforming methods.

At the end of the preceding section, we analyze the asymptotic complexity of the three methods. The ISAC-OCS method is distinguished by its combination of the bisection method and convex optimization, which results in a notably higher complexity order compared to MMO. Figure 12 illustrates the runtime curves of the three methods as a function of the number of antennas.

Figure 12. Variation in the hundred run times with the number of antennas.

As illustrated in Figure 12, MMO exhibits the shortest runtime across various specification arrays, while ISAC-OCS exhibits a longer runtime compared to MMO. Notably, WMMSE-ISAC incurs a greater computational time than ISAC-OCS as the number of array elements increases ($N_{\mathrm{t}}=N_r=20$ and $N_{\mathrm{t}}=N_r=24$). Therefore, MMO and WMMSE-ISAC exhibit the lowest and highest levels of complexity compared to the others, aligning with the previous analysis.

In summary, the CMI rate, SMI rate, and S&C MI rate provided by ISAC-OCS are among the fastest compared to the other methods. Although the computational complexity of ISAC-OCS is slightly higher compared to the MMO, it achieves a superior balance between communication and sensing performance while maintaining a high S&C MI rate. Consequently, the moderate increase in computational complexity is justifiable for practical implementation in real-world ISAC-RSU systems.

5. Conclusions

In this paper, we propose a hierarchical optimization solution for ISAC beamforming, specifically tailored for ISAC-RSU systems operating in complex environments. We decompose the complex joint optimization problem into multiple interrelated sub-optimization layers. On the one hand, communication beamforming was treated as leader-layer optimization. The proposed model initially employed a maximize–minimize optimization strategy to achieve SINR optimization and utilized the bisection method to determine the maximum SINR value, ultimately solving the communication beamforming vector. On the other hand, the sensing beamforming functioned as follower-layer optimization. We employ SDP solvers based on the CVX toolbox to solve the sensing SCNR maximization problem, thereby facilitating the collaborative resolution of the global objective. This optimized solution not only enhances communication performance for receiving users but also significantly improves the sensing capabilities of the ISAC-RSU systems. Despite the increased complexity of the method, simulation results indicate that the proposed model exhibits superior sensing and communication performance compared to the others. The results presented above further underscore the efficacy of ISAC-OCS in achieving an optimal balance between communication and sensing, effectively managing the inherent trade-offs characteristic of ISAC systems. In summary, these results demonstrate the potential of ISAC-OCS as a comprehensive solution for next-generation RUS systems that demand flexible management of trade-offs between communication throughput and sensing accuracy. Looking ahead, we intend to investigate a sophisticated coordination mechanism capable of dynamically allocating resources based on real-time demands and environmental conditions, thereby ensuring that neither communication nor sensing performance is unduly compromised. This mechanism is expected to further validate the effectiveness of ISAC-OCS in harmonizing the dual objectives of ISAC systems, offering a promising framework for future ISAC networks where both capabilities must coexist seamlessly. In addition, our next step will involve initiating research on dynamic channels and channel estimation errors. Specifically, we will aim to utilize sensing-aided echo signals to correct channel estimation inaccuracies, thereby improving system robustness. We will incorporate the time-varying nature of clutter in urban environments and will exploit the real-time sensing capability of the ISAC-RSU proposed in this paper to ensure that SCNR optimization remains consistently aligned with the actual surrounding conditions. Subsequently, we will introduce a “Doppler frequency offset estimation module” to address mobile targets or UEs. This module will enable real-time updates of UE channels, allowing for dynamic adjustment of both the communication and the sensing beamforming vector. In our future work, we also plan to explore adaptive environment-aware predictive beamforming using DL-based techniques to enhance the real-time adaptability and overall efficiency of integrated sensing and communication systems. Such advancements will support emerging applications such as autonomous driving, intelligent infrastructure monitoring, and real-time human–machine interaction.

Appendix B.1

We prove that the convergence of leader-layer optimization, which combines the max–min SINR approach with the bisection method, is rigorously established. The detailed proof is presented as follows:

The core logic of the leader layer, which employs the “binary search method for solving max-min SINR,” is to iteratively narrow the feasible region of SINR, defined as [${\gamma }_{\min }$, ${\gamma }_{\max }$], in order to determine the maximum achievable ${\gamma }_0$ that satisfies the given power constraint.

(i)

Monotonicity

In each iteration, if the midpoint ${\gamma }_{\mathrm{mid}}=({\gamma }_{\min }+{\gamma }_{\max })/2$ is feasible (i.e., there exists a vector $w_u$ that satisfies ${\mathrm{SINR}}_u\ge {\gamma }_{\mathrm{mid}}$), then update ${\gamma }_{\min }={\gamma }_{\mathrm{mid}}$; otherwise, update ${\gamma }_{\max }={\gamma }_{\mathrm{mid}}$. The length of the feasible region, given by $\Delta \gamma ={\gamma }_{\max }-{\gamma }_{\min }$, decreases exponentially with respect to the number of iterations k: $\Delta {\gamma }^{(k)}=\Delta {\gamma }^{(0)}/2^k$.

(ii)

Convergence

When the termination condition $\Delta \gamma \le \lambda$ (where $\lambda$ = 0.01 in this paper) is satisfied, ${\gamma }_{\mathrm{mid}}$ converges to the true optimal value ${\gamma }_0$, and the convergence rate is $𝒪(1/2^k)$, indicating exponential convergence. At this point, the corresponding communication beamforming vectors $w_u$ can be obtained by solving a second-order cone program using the CVX toolbox. The solution is unique, as the second-order cone constraints define a strictly convex feasible set and the objective function is quasi-concave, ensuring the existence and uniqueness of the optimal solution.

Appendix B.2

We prove the convergence of follower-layer optimization, and follower layer reformulates the SCNR maximization problem as an SDP, with convergence ensured by the strong duality property inherent in convex optimization.

The objective function of follower-layer optimization can be reformulated as a linear function with respect to ${\overline{W}}_s=w_s{w}_s^{\mathrm{H}}, \forall s\in S$, while the constraint is expressed as a linear matrix inequality (LMI). This formulation results in a strictly convex optimization problem.

According to the strong duality theory in convex optimization, the optimal value of the primal SDP problem is equal to that of its dual problem, and there exists an optimal solution $W_s^{*}$. When solving this problem using the CVX toolbox via an interior-point method, the algorithm converges to the global optimum, with convergence accuracy guaranteed by the default solver settings, such as a duality gap tolerance of no more than ${10}^{-8}$.

In summary, leader-layer optimization converges uniquely to $w_u^{*}$, and with $w_u^{*}$ fixed, follower-layer optimization converges uniquely to $w_s^{*}$. Therefore, the entire hierarchical optimization scheme achieves convergence within a finite number of iterations. Specifically, only one binary search iteration at the upper level (to attain $w_u^{*}$) and one SDP solution at the lower level (to attain $w_s^{*}$) are required to obtain the globally optimal solution, eliminating the need for iterative cycling between levels.