Codesign of Unimodular Waveform and Receive Filter for MIMO Radar Extended Target Detection Under Suppression Jamming

Abstract

The joint design of unimodular waveforms and receive filters is a pivotal technology in Multiple-Input Multiple-Output (MIMO) radar systems. However, most existing methods primarily focus on point target detection or ignore the impact of active jamming in extended target scenarios. To bridge this gap, this paper proposes an optimization framework for the joint design of unimodular waveforms and receive filters specifically for MIMO radar extended target detection in the presence of suppressive jamming. The problem is formulated to maximize the Signal-to-Interference-plus-Noise Ratio (SINR) while strictly satisfying the unimodular constraint and mitigating suppressive jamming. Due to the non-convexity of the unimodular constraint and the quadratic fractional nature of the SINR objective function, the optimization problem is highly challenging. Unlike conventional methods that rely on convex relaxation—which often leads to performance degradation—we exploit the geometric structure of the constraint set. Specifically, the unimodular constraints are modeled using complex circle manifolds, and the suppressive jamming suppression requirements are integrated into the objective function via a smooth penalty metric. Building on these characteristics, a Product Complex Circle Euclidean Manifold (PCCEM) method is developed. This approach transforms the constrained problem into an unconstrained optimization task on a product manifold, which is then efficiently solved using the limited-memory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS) algorithm. Simulation results demonstrate that the proposed PCCEM method outperforms baseline algorithms in terms of computational efficiency, output SINR, and the depth of the formed jamming notches.

1. Introduction

The joint design of transmit waveforms and receive filters is a pivotal technology in Multiple-Input Multiple-Output (MIMO) radar systems [1,2,3,4]. By synergistically optimizing the transmitter and receiver, the joint design can effectively enhance the target detection performance through the adaptive matching of the beamforming pattern to the target scattering environment and the suppression of interference backgrounds [5,6]. Furthermore, to maximize the power efficiency of the radio frequency (RF) amplifiers and avoid nonlinear distortions, radar waveforms are typically required to satisfy strict unimodular constraints [7,8]. A unimodular waveform refers to a complex-valued signal whose entries have identical magnitude, typically normalized to unity, while only the phase is allowed to vary across antennas and time samples. This constant-envelope property is particularly desirable in practical radar transmitters because it allows RF power amplifiers to operate close to saturation with high power efficiency while reducing nonlinear distortion [9,10]. Consequently, the joint design of unimodular waveforms and receive filters for MIMO radar has attracted extensive attention.

Existing methods mainly include two categories: waveform design for point target detection; and waveform design for extended target detection ignoring the impact of suppression jamming.

Within the first category, a typical approach for point target detection is the Signal-to-Interference-plus-Noise Ratio (SINR) maximization framework, often solved via Semidefinite Relaxation (SDR) as seen in [11]. However, SDR typically relaxes the non-convex constraints into a higher-dimensional space and necessitates matrix inversion, which may not only lead to performance degradation but also incur prohibitive computational costs. To circumvent the complexity of matrix inversion, Alternating Direction Method of Multipliers (ADMM) frameworks were proposed in [12], decomposing the original problem into more tractable subproblems. Nevertheless, the inherent reliance of ADMM on relaxation techniques can still degrade the performance. To fundamentally address these limitations, manifold optimization on product manifolds has emerged as a powerful alternative [13]. This approach directly optimizes the unimodular waveform and receive filter on the constraint manifold, thereby eliminating the need for both heuristic relaxation and high-complexity matrix inversions, ultimately achieving superior detection performance with lower computational overhead. While these methods provide a solid theoretical foundation, they assume the target occupies only a single resolution cell, thereby failing to exploit the spatial scattering information of complex, large-scale targets [14].

The second category extends the design framework to extended target detection by incorporating the target’s impulse response or scattering matrix to exploit spatial diversity. For instance, an SINR maximization framework for polarimetric radar was proposed in [15], employing the Fractional Programming and Successive Convex Approximation (FPP-SCA) method to jointly optimize waveforms and filters. However, the iterative nature of SCA, coupled with frequent high-dimensional matrix inversions, leads to a prohibitive computational burden. To mitigate this, ADMM-based approaches were developed for MIMO radar extended target detection [16]. Yet, despite the gain in efficiency, the detection performance of ADMM-based methods remains constrained due to the performance loss inherent in objective function relaxation and constraint approximation. More critically, these methods predominantly operate under jamming-free environments, failing to account for the impact of active suppressive jamming. In modern electronic warfare, such jamming signals can significantly overwhelm the weak echoes of extended targets, causing a severe degradation in detection probability. Consequently, there is an urgent need for a robust codesign strategy that simultaneously addresses the structural characteristics of extended targets and the imperative of suppressive jamming mitigation within a high-efficiency optimization framework.

To address these challenges, this paper proposes an optimization framework for the joint design of unimodular waveforms and receive filters, specifically tailored for MIMO radar extended target detection in suppressive jamming environments. By exploiting the intrinsic geometric structure of the constraints, we develop a Product Complex Circle Euclidean Manifold (PCCEM) method. This approach transforms the non-convex, constrained SINR maximization problem into an unconstrained optimization task on a product manifold, ensuring both high detection performance and computational efficiency. The main contributions of this paper are summarized as follows:

(1) A comprehensive joint design model for jamming scenarios is formulated: Different from existing studies that focus on point targets or ignore active jamming, we formulate a joint waveform and receive filter optimization problem specifically for extended target detection under suppressive jamming. The model incorporates the target’s scattering characteristics and the jamming suppression requirements into a unified SINR maximization framework, which may provide a more robust solution for modern electronic warfare.

(2) A geometric structure-guided optimization method is proposed: Unlike traditional Riemannian methods that typically handle constraints by projecting onto simple manifolds or using basic barrier functions, PCCEM introduces a multi-scale optimization framework that integrates a non-smooth penalty function directly into the product manifold space. This allows for the simultaneous management of hard unimodular constraints and soft suppression jamming constraints without sacrificing the convergence properties of the RL-BFGS algorithm. Specifically, the novelty lies in the synergistic update mechanism between the smoothing parameter and the penalty factor on the manifold, which ensures that the algorithm can navigate complex non-convex landscapes more effectively than conventional incremental manifold descent methods.

(3) Superior detection and computational performance are obtained: Simulation results validate that the proposed PCCEM method achieves a significant performance gain over state-of-the-art baseline algorithms. Notably, the proposed method yields a 1 dB increase in output SINR compared to the ADMM-based approach, while simultaneously reducing the computational overhead by 5 times.

2. System Model

2.1. Signal Model for Extended Target Detection

Consider a MIMO radar with $N_t$ transmitting antennas and $N_r$ receiving antennas [17,18]. The azimuth angle of the extended target is ${\varphi }_p$, the interval length corresponding to the target echo is $\mathrm{Q}$, and the azimuth angle of the intercepting radar is ${\varphi }_I$. The amplitude of the target echo response of the extended target is closely related to the position of the target relative to the radar line of sight.

The target echo characteristics are significantly influenced by the Target Aspect Angle (TAA), which describes the target’s orientation relative to the radar line of sight. To account for aspect fluctuations, the TAA is modeled as a discrete angular set $\mathcal{L}$ rather than a fixed value. Let ${\theta }_{start}$ denote the starting angle, $\Delta \theta$ the angular step size, and $\mathrm{K}$ the total number of discrete samples. Consequently, the set $\mathcal{L}$ can be expressed as

$$\mathcal{L}=\{{\theta }_i\mid {\theta }_i={\theta }_{start}+i\Delta \theta ,i\le K,i\in \mathbb{Z}\}$$

(1)

The target echo information is given by the target impulse response (TIR) matrix. We denote the TIR at a specific TAA ${\theta }_i$ as $t_{{\theta }_i}\left(n\right)$, with an interval $\mathit{Q}$. The TIR matrix can be constructed as [19]:

$$\mathbf{T}\left({\theta }_i\right)=\sum _{n=1}^Q{t}_{{\theta }_i}\left(n\right){\mathbf{J}}_n$$

(2)

where ${\mathbf{J}}_n\in {\mathbb{C}}^{N_{t}M\times N_t{N}_s}$ is the shift matrix:

$${\mathbf{J}}_n\left(\begin{array}{c}{q}_1,q_2\end{array}\right)=\left\{\begin{array}{cc}1,\hfill & q_1-q_2=N_{t}n.\hfill \\ 0,\hfill & q_1-q_2\ne N_{t}n.\hfill \end{array}\right.$$

(3)

The radar transmit waveform matrix is defined as

$$\mathbf{S}=[{\mathbf{s}}_1^T;{\mathbf{s}}_2^T;\dots ;{\mathbf{s}}_{N_t}^T]\in {\mathbb{C}}^{N_t\times N_s},$$

(4)

where ${\mathbf{s}}_m\in {\mathbb{C}}^{N_s}$ denotes the discrete-time waveform transmitted by the m-th antenna. Therefore, each row of $\mathbf{S}$ corresponds to one transmit channel, and each column corresponds to one sampling instant across the transmit array. For compact representation, the waveform matrix is vectorized as $\mathbf{s}=vec\left(\mathbf{S}\right)\in {\mathbb{C}}^{N_t{N}_s}$.

The radar received signal is:

$$\mathbf{r}=\mathbf{A}\left({\varphi }_p\right)\mathbf{T}\left({\theta }_i\right)\mathbf{s}+\sum _{k=1}^{N_a}\mathbf{A}\left({\varphi }_k\right){\mathbf{C}}_k\mathbf{s}+\mathbf{n}$$

(5)

where ${\varphi }_p$ denotes the azimuth angle of the extended target. The surrounding clutter region is discretized into $N_a$ angular cells, and ${\varphi }_k$ denotes the azimuth angle corresponding to the k-th clutter cell. Accordingly, the term $\mathbf{A}\left({\varphi }_k\right){\mathbf{C}}_k\mathbf{s}$ represents the echo contribution produced by the k-th clutter patch, where ${\mathbf{C}}_k$ models the delay-spread and reflection characteristics of that clutter component with respect to the transmitted waveform. In addition, $\mathbf{n}\sim \mathcal{CN}(\mathbf{0},{\sigma }_{\nu }^2\mathbf{I})$ denotes the additive complex Gaussian noise vector, where ${\sigma }_{\nu }^2$ is the noise power and $\mathbf{I}$ is the identity matrix.

${\mathbf{a}}_T\left(\varphi \right)$ and ${\mathbf{a}}_R\left(\varphi \right)$ denote the transmit and receive steering vectors, respectively. For a uniform linear array, their entries are determined by the inter-element spacing and the wavelength, and they characterize the directional response of the transmit and receive arrays:

$$\begin{array}{cc}\hfill {\mathbf{a}}_T\left(\varphi \right)& ={[1,e^{j2\pi {\displaystyle \frac{d_T}{\lambda }}sin\varphi },\dots ,e^{j2\pi {\displaystyle \frac{d_T}{\lambda }}(N_t-1)sin\varphi }]}^T,\hfill \\ \hfill {\mathbf{a}}_R\left(\varphi \right)& ={[1,e^{j2\pi {\displaystyle \frac{d_R}{\lambda }}sin\varphi },\dots ,e^{j2\pi {\displaystyle \frac{d_R}{\lambda }}(N_r-1)sin\varphi }]}^T.\hfill \end{array}$$

(6)

${\mathbf{C}}_k$ is the clutter impulse response matrix, which is used to characterize the clutter in relation to the transmitted signal:

$${\mathbf{C}}_k=\sum _{n=-N_s}^M{c}_k\left(n\right){\mathbf{J}}_n$$

(7)

where $c_k\left(n\right)$ denotes the complex reflection coefficient of the k-th clutter component at delay index n, and ${\mathbf{J}}_n$ is a shift matrix that shifts the transmitted waveform by n delay bins. Therefore, ${\mathbf{C}}_k$ can be interpreted as a convolution-like matrix representation of the clutter response.

The filter output at the $\mathit{j}\text{-th}$ branch can be written as:

$$r={\mathbf{w}}^H\mathbf{r}={\mathbf{w}}^H\mathbf{H}\mathbf{s}+{\mathbf{w}}^H\sum _{k=1}^{N_a}\mathbf{A}\left({\varphi }_k\right){\mathbf{C}}_k\mathbf{s}+{\mathbf{w}}^H\mathbf{n}$$

(8)

where the dimension of the filter is $\mathbf{w}\in {\mathbb{C}}^{N_{r}M}$ with one filter ${\mathbf{w}}_j$ corresponding to each angle. Meanwhile, for simplicity of expression, let ${\mathbf{H}}_j=\mathbf{A}\left({\varphi }_p\right)\mathbf{T}\left({\theta }_a^j\right)$. ${\mathbf{T}}_a$ denotes the average target echo response matrix of the extended target within the TAA interval, defined as follows:

$$\begin{array}{cc}\hfill {\mathbf{T}}_a& ={\displaystyle \frac{\Delta \theta }{{\theta }_{max}-{\theta }_{min}}}\sum _{i=1}^K\mathbf{T}\left({\theta }_i\right),\hfill \\ \hfill K& =({\theta }_{max}-{\theta }_{min})/\Delta \theta +1.\hfill \end{array}$$

(9)

where ${\theta }_{\max }$ and ${\theta }_{\min }$ denote the angular range, and $\Delta \theta$ denotes the angular interval.

The SINR at the receiver is

$${\mathrm{SINR}}_{avg}={\displaystyle \frac{\Delta {\theta }^2{\left|{\alpha }_T\right|}^2}{{\left({\theta }_{\max }-{\theta }_{\min }\right)}^2}}{\displaystyle \frac{{\left|{\mathbf{w}}^H{\mathbf{H}}_a\mathbf{s}\right|}^2}{{\mathbf{w}}^H\mathsf{\Sigma }\left(\mathbf{s}\right)\mathbf{w}+{\sigma }_{\nu }^2{\mathbf{w}}^H\mathbf{w}}}$$

(10)

where $\mathsf{\Sigma }\left(\mathbf{s}\right)$ is the clutter covariance matrix, and is the expectation of the clutter impact.

2.2. Analysis of Suppression Jamming

In this work, we consider a cognitive radar scenario where the jammer’s spatial direction ${\varphi }_I$ is identified through prior environmental sensing or electronic support measures. We assume a dominant suppression jammer located at a known angle, focusing on the fundamental challenge of joint design under strict unimodular and power constraints. The suppression jamming signal is modeled as a zero-mean Gaussian process that enters the radar receiver through the spatial channel. To suppress this interference at the source, we impose a transmit notch constraint $P(\mathbf{S},{\varphi }_I)\le \omega$. This constraint ensures that the transmit beampattern produces a deep null in the direction of the jammer, thereby minimizing the energy reflected or re-transmitted by the jamming source back into the radar receiver. Mathematically, by limiting the transmit power in direction ${\varphi }_I$, the interference component in the received signal covariance matrix is significantly reduced, which prevents the receiver from being saturated and allows the receive filter $\mathbf{w}$ to further suppress residual interference in the digital domain. This transmit–receive dual suppression strategy maximizes the Signal-to-Interference-plus-Noise Ratio (SINR) for the extended target.

3. Problem Formulation

In this section, a joint transmit waveform and receive filter design problem is proposed to maximize the SINR for extended target detection. This framework integrates the necessity for spatial jamming suppression with the hardware-driven constant modulus constraint.

In radar signal processing, the detection performance for an extended target is fundamentally determined by the output Signal-to-Interference-plus-Noise Ratio (SINR). Specifically, under the Neyman–Pearson criterion, the detection probability $P_d$ is a monotonically increasing function of the SINR for a fixed false alarm rate $P_{fa}$. Consequently, maximizing the detection probability for an extended target is equivalent to maximizing the output SINR. To this end, the problem of extended target detection is formulated as the joint optimization of the MIMO transmit waveform and the receive filter. The objective is to maximize the SINR:

$$\underset{\mathbf{s},\mathbf{w}}{max}{\mathrm{SINR}}_{avg}$$

(11)

To maintain high power efficiency and prevent nonlinear distortions in radio frequency (RF) amplifiers, the transmit waveform is typically required to have a constant envelope. Consequently, a constant modulus (CM) constraint is imposed on the waveform vector $\mathbf{s}$, which can be denoted as:

$$\left|\mathbf{s}\left(i\right)\right|=1,i=1,2,\dots ,N_t{N}_s.$$

(12)

where $\mathbf{s}\left(i\right)$ is the i-th element of $\mathbf{s}$.

In complex sensing scenarios, such as military electronic warfare or civilian low-altitude economy operations, the presence of active jamming or co-existing electronic devices can significantly degrade extended target detection performance. To ensure robust detection and effectively mitigate jamming at the specific azimuth angle ${\varphi }_I$, the transmit beampattern must be carefully controlled. Specifically, we restrict the transmit power directed toward the jammer’s location to be below a predefined threshold $\omega$, which can be formulated as:

$$P(\mathbf{S},\varphi )={\mathbf{a}}_T^H\left(\varphi \right){\mathbf{SS}}^H{\mathbf{a}}_T\left(\varphi \right)\le \omega .$$

(13)

$P(\mathbf{S},\varphi )$ denotes the transmit beampattern, namely the power radiated by the transmit array toward spatial direction $\varphi$. The matrix $\mathbf{S}{\mathbf{S}}^H$ represents the transmit covariance induced by the waveform matrix, and ${\mathbf{a}}_T^H\left(\varphi \right)\mathbf{S}{\mathbf{S}}^H{\mathbf{a}}_T\left(\varphi \right)$ extracts the directional transmit power along angle $\varphi$. Therefore, the constraint $P(\mathbf{S},{\varphi }_I)\le \omega$ limits the radiation energy sent toward the jammer/interceptor direction.

Motivated by the requirements from (11) to (13), we formulate the joint optimization problem. To facilitate the use of gradient descent-based optimization methods, we convert the maximization of SINR into the minimization of its reciprocal. The resulting non-convex optimization problem is formulated as [10]:

$$\mathcal{P}\left\{\begin{array}{c}{\displaystyle \underset{\mathbf{s},\mathbf{w}}{min}\frac{1}{{\mathrm{SINR}}_{avg}}=\frac{{\mathbf{w}}^H\mathsf{\Omega }\left(\mathbf{s}\right)\mathbf{w}}{{\mathbf{w}}^H\mathsf{\Pi }\left(\mathbf{s}\right)\mathbf{w}}=\frac{{\mathbf{s}}^H\mathbf{P}\left(\mathbf{w}\right)\mathbf{s}}{{\mathbf{s}}^H\mathbf{Q}\left(\mathbf{w}\right)\mathbf{s}},}\hfill \\ \mathrm{s}.\mathrm{t}.\left|s\left(i\right)\right|=1,i=1,2,\dots ,N_t{N}_s,\hfill \\ {\mathbf{a}}_T^H\left({\varphi }_I\right)\mathbf{S}{\mathbf{S}}^H{\mathbf{a}}_T\left({\varphi }_I\right)\le \omega .\hfill \end{array}\right.$$

(14)

where

$$\begin{array}{cc}\hfill {\displaystyle \frac{1}{{\mathrm{SINR}}_{avg}}}& ={\displaystyle \frac{{\mathbf{w}}^H\mathsf{\Sigma }\left(\mathbf{s}\right)\mathbf{w}+{\sigma }_{\nu }^2{\mathbf{w}}^H\mathbf{w}}{{\left|{\mathbf{w}}^H{\mathbf{H}}_a\mathbf{s}\right|}^2}}\hfill \\ \hfill & ={\displaystyle \frac{{\mathbf{w}}^H\mathsf{\Omega }\left(\mathbf{s}\right)\mathbf{w}}{{\mathbf{w}}^H\mathsf{\Pi }\left(\mathbf{s}\right)\mathbf{w}}}={\displaystyle \frac{{\mathbf{s}}^H\mathbf{P}\left(\mathbf{w}\right)\mathbf{s}}{{\mathbf{s}}^H\mathbf{Q}\left(\mathbf{w}\right)\mathbf{s}}},\hfill \\ \hfill \mathsf{\Omega }\left(\mathbf{s}\right)& =\mathsf{\Sigma }\left(\mathbf{s}\right)+{\sigma }_{\nu }^2\mathbf{I},\hfill \\ \hfill \mathsf{\Pi }\left(\mathbf{s}\right)& ={\mathbf{H}}_a\mathbf{s}{\mathbf{s}}^H{\mathbf{H}}_a^H,\hfill \\ \hfill \mathbf{P}\left(\mathbf{w}\right)& =\sum _{k=1}^{N_a}{\mathbf{C}}_k^H{\mathbf{A}}^H\left({\varphi }_k\right)\mathbf{w}{\mathbf{w}}^H\mathbf{A}\left({\varphi }_k\right){\mathbf{C}}_k+{\displaystyle \frac{{\sigma }_{\nu }^2{\parallel \mathbf{w}\parallel }^2}{N_t{N}_s}}\mathbf{I},\hfill \\ \hfill \mathbf{Q}\left(\mathbf{w}\right)& ={\mathbf{H}}_a^H\mathbf{w}{\mathbf{w}}^H{\mathbf{H}}_a.\hfill \end{array}$$

(15)

$\mathsf{\Omega }\left(\mathbf{s}\right)$ and $\mathsf{\Pi }\left(\mathbf{s}\right)$ represent the interference-plus-noise and target signal covariance matrices, respectively, while $\mathbf{P}\left(\mathbf{w}\right)$ and $\mathbf{Q}\left(\mathbf{w}\right)$ are the corresponding transformed matrices used for waveform optimization.

Problem (14) is inherently difficult to solve due to the non-convex quadratic fractional objective function and the strict unimodular constraints. Conventional approaches, such as SCA and the ADMM, typically rely on objective function relaxation and high-dimensional matrix inversions. These techniques often suffer from degraded performance due to relaxation gaps and incur prohibitive computational costs. To overcome these limitations, we deviate from traditional relaxation-based strategies and instead exploit the intrinsic geometric structure of the constraint set. Specifically, the constant modulus constraints are modeled as a complex circle manifold, while the suppressive jamming requirement is integrated into the objective function through a smooth penalty metric. Leveraging these geometric insights, the Product Complex Circle Euclidean Manifold (PCCEM) method is developed to efficiently solve this problem without objective function relaxation or matrix inversions.

4. The Proposed PCCEM Method

In this section, the PCCEM method is developed by exploiting the intrinsic geometric structure of the problem constraints. Unlike existing manifold-based radar waveform optimization methods, which mainly handle unimodular constraints on simple manifolds and treat additional inequalities separately, PCCEM embeds the suppressive jamming constraint into the objective via a smooth exact-penalty term and solves the resulting problem directly on a product manifold consisting of a complex circle manifold and a Euclidean manifold. As a result, the hard unimodular constraint and the soft jamming-suppression requirement are addressed in a unified framework. A manifold L-BFGS algorithm, together with an adaptive penalty-smoothing update strategy, is then used to efficiently solve the resulting non-convex problem.

4.1. Construction of Exact Penalty Terms

The spatial transmit power constraint in problem (14) can be incorporated into the objective function as a penalty function by introducing a penalty factor $\rho$, and the problem is rewritten as follows:

$${\mathcal{P}}_1\left\{\begin{array}{c}{\displaystyle \underset{\mathbf{s},\mathbf{w}}{min}\frac{1}{{\mathrm{SINR}}_{avg}}+\rho max\left\{0,P(\mathbf{S},{\varphi }_I)-\omega \right\},}\hfill \\ \mathrm{s}.\mathrm{t}.\left|s\left(i\right)\right|=1,i=1,2,\dots ,N_t{N}_s.\hfill \end{array}\right.$$

(16)

To approximate the non-smooth term $max\{0,g(\mathbf{s}\left)\right\}$, where $g\left(s\right)=P(\mathbf{S},{\varphi }_I)-\omega$, we employ the log-sum-exp function $l\left(g\right(\mathbf{s}\left)\right)=\mu log(1+exp(g\left(\mathbf{s}\right)/\mu \left)\right)$. The accuracy of this approximation is governed by the following error bound: $0\le \mu log(1+e^{g\left(\mathbf{s}\right)/\mu })-max\{0,g\left(\mathbf{s}\right)\}\le \mu log\left(2\right)$. This inequality demonstrates that as the smoothing parameter $\mu \to 0$, the smooth penalty function $l\left(g\right(\mathbf{s}\left)\right)$ converges uniformly to the original max operator. Consequently, the solution to the smoothed problem ${\mathcal{P}}_2$ asymptotically satisfies the original inequality constraint $P(\mathbf{S},{\varphi }_I)\le \omega$ of problem ${\mathcal{P}}_1$.

Furthermore, regarding the penalty formulation, for a sufficiently large penalty factor $\rho$, the exact penalty property holds. This guarantees that the optimal solution $({\mathbf{s}}^{*},{\mathbf{w}}^{*})$ of the penalized problem remains feasible for the original constraint $a_T^H\left({\varphi }_I\right)\mathbf{S}{\mathbf{S}}^H{a}_T\left({\varphi }_I\right)\le \omega$. In our implementation, an adaptive update strategy for $\rho$ (as described in Section 4.4) is used to ensure this feasibility is strictly maintained during the optimization process.

The problem ${\mathcal{P}}_1$ can be rewritten as Problem ${\mathcal{P}}_2$:

$${\mathcal{P}}_2\left\{\begin{array}{c}{\displaystyle \underset{\mathbf{s},\mathbf{w}}{min}\frac{1}{{\mathrm{SINR}}_{avg}}+\rho 𝓁\left(P(\mathbf{S},{\varphi }_I)-\omega \right),}\hfill \\ \mathrm{s}.\mathrm{t}.\left|s\left(i\right)\right|=1,i=1,2,\dots ,N_t{N}_s.\hfill \end{array}\right.$$

(17)

where $𝓁\left(x\right)=log\left(1+exp\left({\displaystyle \frac{x}{\mu }}\right)\right)$ and $\mu$ is the smoothing factor, which must satisfy $\mu >0$.

4.2. Construction of PCCEM Space

In this subsection, to address the difficulty in solving ${\mathcal{P}}_2$ due to the presence of CM constraints, we note that Complex-Circular manifolds inherently satisfy the CM constraints. Specifically, a complex circle manifold is the set of all complex-valued vectors whose entries lie on the unit circle in the complex plane. This geometric interpretation is particularly suitable for unimodular waveform design because feasibility is embedded directly into the search space. Therefore, we construct the PCCEM space and project ${\mathcal{P}}_2$ onto it, transforming ${\mathcal{P}}_2$ into the unconstrained optimization problem ${\mathcal{P}}_3$ [20,21].

CM constrains satisfy the complex circle manifold ${\mathcal{M}}_{\mathbf{s}}$:

$${\mathcal{M}}_s=\left\{\mathbf{s}\in {\mathbb{C}}^{N_t{N}_s}|\left|s\left(i\right)\right|=1,i=1,2,\dots ,N_t{N}_s\right\}$$

(18)

The filter $\mathbf{w}$ is unconstrained and satisfies complex Euclidean manifolds ${\mathcal{M}}_{\mathbf{w}}$. Then, the product manifold $\mathcal{M}$ is constructed to satisfy all the constraints.

$$\begin{array}{cc}\hfill \mathcal{M}& ={\mathcal{M}}_{\mathbf{s}}\times {\mathcal{M}}_{\mathbf{w}}\hfill \\ \hfill & =\left\{(\mathbf{s},\mathbf{w}):\mathbf{s}\in {\mathcal{M}}_{\mathbf{s}},\mathbf{w}\in {\mathcal{M}}_{\mathbf{w}}\right\}.\hfill \end{array}$$

(19)

Therefore, ${\mathcal{P}}_2$ is transformed into the unconstrained optimization problem ${\mathcal{P}}_3$ on PCCEM:

$${\mathcal{P}}_3\left\{\begin{array}{c}{\displaystyle \underset{\mathbf{s},\mathbf{w}}{min}\frac{1}{{\mathrm{SINR}}_{avg}}+\rho 𝓁\left(P(\mathbf{S},{\varphi }_I)-\omega \right),}\hfill \\ \mathrm{s}.\mathrm{t}.(\mathbf{s},\mathbf{w})\in \mathcal{M}.\hfill \end{array}\right.$$

(20)

Due to the fact that the problem is projected onto the PCCEM, the tangent space, tangent vectors, and vector transport on the manifold must be defined before solving ${\mathcal{P}}_3$. It is worth noting that the Euclidean gradient cannot be directly used as a feasible descent direction on a manifold. Therefore, the Euclidean gradient is orthogonally projected onto the tangent space to obtain the corresponding Riemannian gradient [22].

$${\mathcal{T}}_{\mathrm{s},\mathrm{w}}M={\mathcal{T}}_{\mathbf{s}}{\mathcal{M}}_{\mathbf{s}}\times {\mathcal{T}}_{\mathbf{w}}{\mathcal{M}}_{\mathbf{w}}$$

(21)

where ${\mathcal{T}}_{\mathbf{s}}{\mathcal{M}}_{\mathbf{s}}$ is the tangent space of ${\mathcal{M}}_{\mathbf{s}}$, and ${\mathcal{T}}_{\mathbf{w}}{\mathcal{M}}_{\mathbf{w}}$ is the tangent space of ${\mathcal{M}}_{\mathbf{w}}$:

$$\begin{array}{cc}& {\mathcal{T}}_{\mathbf{s}}{\mathcal{M}}_{\mathbf{s}}=\left\{{\beta }_{\mathbf{s}}\in {\mathbb{C}}^{N_t{N}_s},\Re \left\{{\beta }_{\mathbf{s}}\odot {\mathbf{s}}^{*}\right\}={\mathbf{0}}_{N_t{N}_s}\right\}\hfill \\ & {\mathcal{T}}_{\mathbf{w}}{\mathcal{M}}_{\mathbf{w}}=\left\{{\beta }_{\mathbf{w}}\in {\mathbb{C}}^{N_{r}M}\right\}\hfill \end{array}$$

(22)

where ${\beta }_{\mathbf{s}}$ and ${\beta }_{\mathbf{w}}$ denote the tangent vectors of $\mathbf{s}$ and $\mathbf{w}$, respectively. Therefore, the tangent space of $(\mathbf{s},\mathbf{w})\in \mathcal{M}$ can be expressed as:

$${\mathcal{T}}_{(\mathbf{s},\mathbf{w})}\mathcal{M}=\left\{({\beta }_{\mathbf{s}},{\beta }_{\mathbf{w}})|\Re \left\{{\beta }_{\mathbf{s}}\odot {\mathbf{s}}^{*}\right\}=\mathbf{0},{\beta }_{\mathbf{w}}\in {\mathbb{C}}^{N_{r}M}\right\}.$$

(23)

To compute the gradient of a cost function on manifold $\mathcal{M}$, calculations must be performed on a Riemannian submanifold within Euclidean space. Riemannian submanifolds are typically defined using the standard complex Euclidean inner product. For any pair $(\mathbf{s},\mathbf{w})\in \mathcal{M}$, the inner product is expressed as:

$$⟨{\beta }_{\mathbf{s},\mathbf{w}},{\dot{\beta }}_{\mathbf{s},\mathbf{w}}⟩=\Re \left\{{\beta }_{\mathbf{s}}^H{\dot{\beta }}_{\mathbf{s}}\right\}+{\beta }_{\mathbf{w}}{\dot{\beta }}_{\mathbf{w}}$$

(24)

where ${\beta }_{\mathbf{s}},{\dot{\beta }}_{\mathbf{s}}\in {\mathcal{T}}_{\mathbf{s}}{\mathcal{M}}_{\mathbf{s}}$ and ${\beta }_{\mathbf{w}},{\dot{\beta }}_{\mathbf{w}}\in {\mathcal{T}}_{\mathbf{w}}{\mathcal{M}}_{\mathbf{w}}$.

The operation of moving the tangent vector ${\beta }_{(\mathbf{s},\mathbf{w})}$ from the tangent space at point $(\mathbf{s},\mathbf{w})\in \mathcal{M}$ to the tangent space at $({\mathbf{s}}^{\prime },{\mathbf{w}}^{\prime })\in \mathcal{M}$ requires a transport operator $\mathcal{T}{\mathcal{R}}_{(·)\leftarrow (·)}$ defined as:

$$\begin{array}{cc}& {\mathcal{TR}}_{({\mathbf{s}}^{\prime },{\mathbf{w}}^{\prime })\leftarrow (\mathbf{s},\mathbf{w})}\left({\beta }_{(\mathbf{s},\mathbf{w})}\right)\hfill \\ & =(T{\mathcal{R}}_{{\mathbf{s}}^{\prime }\leftarrow \mathbf{s}}\left({\beta }_{\mathbf{s}}\right),T{\mathcal{R}}_{{\mathbf{w}}^{\prime }\leftarrow \mathbf{w}}\left({\beta }_{\mathbf{w}}\right))\hfill \end{array}$$

(25)

where ${\mathcal{TR}}_{{\mathbf{s}}^{\prime }\leftarrow \mathbf{s}}\left({\beta }_{\mathbf{s}}\right)={\beta }_{\mathbf{s}}-\Re \left\{{\beta }_{\mathbf{s}}^{*}\odot {\mathbf{s}}^{\prime }\right\}\odot {\mathbf{s}}^{\prime }$ and ${\mathcal{TR}}_{{\mathbf{w}}^{\prime }\leftarrow \mathbf{w}}\left({\beta }_{\mathbf{w}}\right)={\beta }_{\mathbf{w}}$.

4.3. The Derived L-BFGS Algorithm

In this subsection, we introduce the L-BFGS algorithm within the quasi-Newton framework for solving ${\mathcal{P}}_3$ [13]. L-BFGS algorithm comprises four main steps: (1) Calculate the Riemannian gradient; (2) Compute the BFGS direction; (3) Update the feasible solution; (4) Adaptively update the step size.

4.3.1. Calculate the Parallel Riemannian Gradient

This subsection derives the Riemannian gradient on the product manifold. First, the Euclidean gradients with respect to $\mathbf{s}$ and $\mathbf{w}$ are computed. The gradient of $\mathbf{s}$ contains both the reciprocal SINR term and the smooth penalty term, while that of $\mathbf{w}$ only contains the reciprocal SINR term. Then, the gradient of $\mathbf{s}$ is projected onto the tangent space of the complex circle manifold, whereas the gradient of $\mathbf{w}$ remains unchanged. To simplify the expression, define $f(\mathbf{s},\mathbf{w})={\displaystyle \frac{1}{{\mathrm{SINR}}_{avg}}}+\rho 𝓁\left(P(\mathbf{S},{\varphi }_I)-\omega \right)$ and $g\left(\mathbf{S}\right)=P(\mathbf{S},{\varphi }_I)-\omega$. Riemannian gradient is defined by orthogonally projecting Euclidean gradient onto the tangent space. Accordingly, the Euclidean gradients with respect to $\mathbf{s}$ and $\mathbf{w}$ are given by

$$\mathrm{Grad}f(\mathbf{s},\mathbf{w})=\left\{\begin{array}{c}{\nabla }_{\mathbf{s}}f(\mathbf{s},\mathbf{w})={\displaystyle \frac{f(\mathbf{s},\mathbf{w})}{\partial \mathbf{s}}},\hfill \\ \\ {\nabla }_{\mathbf{w}}f(\mathbf{s},\mathbf{w})={\displaystyle \frac{f(\mathbf{s},\mathbf{w})}{\partial \mathbf{w},}}.\hfill \end{array}\right.$$

(26)

where

$${\displaystyle \frac{\partial f(\mathbf{s},\mathbf{w})}{\partial \mathbf{s}}}=2{\displaystyle \frac{\left({\mathbf{s}}^H\mathbf{Q}\mathbf{s}\right)\mathbf{P}\mathbf{s}-\left({\mathbf{s}}^H\mathbf{P}\mathbf{s}\right)\mathbf{Q}\mathbf{s}}{{∥{\mathbf{s}}^H\mathbf{Q}\mathbf{s}∥}^2}}+\rho {\displaystyle \frac{1}{1+e^{-\left(\frac{g\left(\mathbf{s}\right)}{\mu }\right)}}}·{\displaystyle \frac{1}{\mu }}·{\displaystyle \frac{\partial \mathbf{g}\left(\mathbf{S}\right)}{\partial \mathbf{s}}}$$

(27)

$${\displaystyle \frac{\partial g\left(\mathbf{S}\right)}{\partial \mathbf{s}}}=2{\mathbf{a}}_T\left({\varphi }_I\right){\mathbf{a}}_T^H\left({\varphi }_I\right)\mathbf{S}$$

(28)

and

$${\displaystyle \frac{\partial f(\mathbf{s},\mathbf{w})}{\partial \mathbf{w}}}=2{\displaystyle \frac{\left({\mathbf{w}}^H\mathsf{\Omega }\mathbf{w}\right)\mathsf{\Pi }\mathbf{w}-\left({\mathbf{w}}^H\mathsf{\Pi }\mathbf{w}\right)\mathsf{\Omega }\mathbf{w}}{{∥{\mathbf{w}}^H\mathsf{\Omega }\mathbf{w}∥}^2}}$$

(29)

For the waveform $\mathbf{s}$, the Euclidean gradient ${\nabla }_{\mathbf{s}}f(\mathbf{s},\mathbf{w})$ needs to be projected onto the tangent space of the Riemannian manifold to obtain its Riemannian gradient:

$$\begin{array}{cc}\hfill {\mathrm{grad}}_{\mathbf{s}}f(\mathbf{s},\mathbf{w})& ={\mathrm{Proj}}_{\mathbf{s}}\left({\nabla }_{\mathbf{s}}f(\mathbf{s},\mathbf{w})\right)\hfill \\ \hfill & ={\nabla }_{\mathbf{s}}f(\mathbf{s},\mathbf{w})-\Re \left\{{\nabla }_{\mathbf{s}}f(\mathbf{s},\mathbf{w})\odot {\mathbf{s}}^{*}\right\}\odot \mathbf{s}.\hfill \end{array}$$

(30)

where ${\mathrm{Proj}}_{\mathbf{s}}(·)$ represents the orthogonal projection operation. Finally, the Riemannian gradient is:

$$\begin{array}{cc}& \mathrm{grad}f(\mathbf{s},\mathbf{w})=[{\mathrm{grad}}_{s}f(\mathbf{s},\mathbf{w}),{\mathrm{Grad}}_{\mathbf{w}}f(\mathbf{s},\mathbf{w})]\hfill \end{array}$$

(31)

4.3.2. Getting the BFGS Direction

After the Riemannian gradient is obtained, a quasi-Newton search direction is computed to improve convergence. The BFGS update avoids explicit Hessian calculation while still capturing local curvature information. As a result, it provides an efficient descent direction in limited-memory form for large-scale problems. After obtaining the parallel Riemannian gradient, the BFGS algorithm can be used to compute the descent search direction. During the k-th optimization iteration of ${\mathcal{P}}_3$, perform a second-order Taylor approximation of the objective function $f(\mathbf{s},\mathbf{w})$:

$$\begin{array}{cc}\hfill f(\mathbf{s},\mathbf{w})& \approx f({\mathbf{s}}^i,{\mathbf{w}}^i)+{\nabla }^{H}f({\mathbf{s}}^i,{\mathbf{w}}^i)[(\mathbf{s},\mathbf{w})-({\mathbf{s}}^i,{\mathbf{w}}^i)]\hfill \\ & +{\displaystyle \frac{1}{2}}{[(\mathbf{s},\mathbf{w})-({\mathbf{s}}^i,{\mathbf{w}}^i)]}^H{\nabla }^{2}f({\mathbf{s}}^i,{\mathbf{w}}^i)[(\mathbf{s},\mathbf{w})-({\mathbf{s}}^i,{\mathbf{w}}^i)],\hfill \end{array}$$

(32)

where ${\nabla }^{2}f({\mathbf{s}}^i,{\mathbf{w}}^i)$ is the Hessian matrix at $({\mathbf{s}}^i,{\mathbf{w}}^i)$.

The search direction is:

$$\left({\mathbf{D}}_{\mathbf{s}}^i,{\mathbf{d}}_{\mathbf{w}}^i\right)=({\mathbf{s}}^i,{\mathbf{w}}^i)-{\left({\nabla }^{2}f({\mathbf{s}}^i,{\mathbf{w}}^i)\right)}^{-1}\nabla f({\mathbf{s}}^i,{\mathbf{w}}^i)$$

(33)

Since the computation of the Hessian matrix involves matrix inversion, which introduces a very large computational load, an approximation method is employed. Specifically, the BFGS direction can be expressed as [23,24]:

$$\begin{array}{cc}\hfill & {\mathcal{H}}_{\mathbf{s}}^{i+1}={\left({\mathbf{B}}_{\mathbf{s}}^i\right)}^H{\mathcal{H}}_{\mathbf{s}}^i{\mathbf{B}}_{\mathbf{s}}^i+{\varphi }_i{\zeta }_{\mathbf{s}}^i{\left({\zeta }_{\mathbf{s}}^i\right)}^H,\hfill \\ \hfill & {\mathcal{H}}_{\mathbf{w}}^{i+1}={\left({\mathbf{B}}_{\mathbf{w}}^i\right)}^H{\mathcal{H}}_{\mathbf{w}}^i{\mathbf{B}}_{\mathbf{w}}^i+{\varphi }_i{\zeta }_{\mathbf{w}}^i{\left({\zeta }_{\mathbf{w}}^i\right)}^H\hfill \end{array}$$

(34)

where

$$\begin{array}{cc}\hfill {\mathbf{B}}_{\mathrm{s}}^i& =\mathbf{I}-{\varphi }_i{\mathbf{T}}_s^i{\left({\zeta }_s^i\right)}^H,\hfill \\ \hfill {\mathbf{B}}_{\mathbf{w}}^i& =\mathbf{I}-{\varphi }_i{\mathbf{t}}_{\mathbf{w}}^i{\left({\xi }_{\mathbf{w}}^i\right)}^H,\hfill \\ \hfill {\varphi }_i& =1/({\zeta }_s^i,{\zeta }_w^i),\left({\mathbf{T}}_{\mathbf{s}}^{\mathbf{i}},{\mathbf{t}}_{\mathbf{w}}^{\mathbf{i}}\right),\hfill \\ \hfill ({\zeta }_s^i,{\zeta }_{\mathbf{w}}^i)& ={\mathrm{Trans}}_{({\mathbf{s}}^{i+1},{\mathbf{w}}^{i+1})\leftarrow ({\mathbf{s}}^i,{\mathbf{w}}^i)}\left({\alpha }_i\left({\mathbf{D}}_{\mathbf{s}}^i,{\mathbf{d}}_{\mathbf{w}}^i\right)\right),\hfill \\ \hfill ({\mathbf{T}}_{\mathbf{s}}^{\mathbf{i}},{\mathbf{t}}_{\mathbf{w}}^{\mathbf{i}})& =({\mathrm{grad}}_{s}f({\mathbf{s}}^{i+1},{\mathbf{w}}^{i+1})-{\mathrm{Trans}}_{{\mathbf{s}}^{i+1}\leftarrow {\mathbf{s}}^i}\left({\mathrm{grad}}_{s}f({\mathbf{s}}^i,{\mathbf{w}}^i)\right),{\nabla }_{\mathbf{w}}f({\mathbf{s}}^{i+1},{\mathbf{w}}^{i+1})-{\nabla }_{\mathbf{w}}f({\mathbf{s}}^i,{\mathbf{w}}^i)),\hfill \end{array}$$

(35)

$({\zeta }_s^i,{\zeta }_{\mathbf{w}}^i)$ and $({\mathbf{T}}_{\mathbf{s}}^{\mathbf{i}},{\mathbf{t}}_{\mathbf{w}}^{\mathbf{i}})$ represent the change in search direction and the change in gradient, respectively. For numerical stability, they are normalized as follows:

$$\begin{array}{c}\hfill {\overline{\zeta }}_{\mathbf{s}}^i={\displaystyle \frac{{\zeta }_{\mathbf{s}}^i}{\parallel {\zeta }_{\mathbf{s}}^i{\parallel }_2}},{\overline{\zeta }}_{\mathbf{w}}^i={\displaystyle \frac{{\zeta }_{\mathbf{w}}^i}{\parallel {\zeta }_{\mathbf{w}}^i{\parallel }_2}},\\ \hfill {\overline{\mathbf{T}}}_{\mathbf{s}}^i={\displaystyle \frac{{\mathbf{T}}_{\mathbf{s}}^i}{\parallel {\mathbf{T}}_{\mathbf{s}}^i{\parallel }_2}},{\overline{\mathbf{t}}}_{\mathbf{w}}^i={\displaystyle \frac{{\mathbf{t}}_{\mathbf{w}}^i}{\parallel {\mathbf{t}}_{\mathbf{w}}^i{\parallel }_2}}.\end{array}$$

(36)

Based on the substitution of the intermediate variables described above and the recursive nature of (34), the approximate Hessian matrix for m steps can be obtained. Taking the Hessian matrix with respect to the waveform $\mathbf{s}$ as an example, let the initial matrix be ${\mathcal{H}}_{\mathbf{s}}^{i-m}$, then we can obtain:

$$\begin{array}{cc}\hfill {\mathcal{H}}_{\mathbf{s}}^i& ={({\mathbf{B}}_{\mathbf{s}}^{i-m}\dots {\mathbf{B}}_{\mathbf{s}}^{i-1})}^H{\mathcal{H}}_{\mathbf{s}}^{i-m}\left({\mathbf{B}}_{\mathbf{s}}^{i-m}\dots {\mathbf{B}}_{\mathbf{s}}^{i-1}\right)\hfill \\ \hfill & +{\varphi }_{i-m}{({\mathbf{B}}_{\mathbf{s}}^{i-m+1}\dots {\mathbf{B}}_{\mathbf{s}}^{i-1})}^H{\zeta }_{\mathbf{s}}^{i-m}{\left({\zeta }_{\mathbf{s}}^{i-m}\right)}^H({\mathbf{B}}_{\mathbf{s}}^{i-m+1}\dots {\mathbf{B}}_{\mathbf{s}}^{i-1})\hfill \\ \hfill & +{\varphi }_{i-m+1}{\left({\mathbf{B}}_{\mathbf{s}}^{i-m+2}\dots {\mathbf{B}}_{\mathbf{s}}^{i-1}\right)}^H{\zeta }_{\mathbf{s}}^{i-m+1}{\left({\zeta }_{\mathbf{s}}^{i-m+1}\right)}^H\left({\mathbf{B}}_{\mathbf{s}}^{i-m+2}\dots {\mathbf{B}}_{\mathbf{s}}^{i-1}\right)\hfill \\ \hfill & +\dots +{\varphi }_{i-1}{\zeta }_{\mathbf{s}}^{i-1}{\left({\zeta }_{\mathbf{s}}^{i-1}\right)}^H,\hfill \end{array}$$

(37)

Then the search direction ${\mathbf{D}}_{\mathbf{s}}^i$ for $\mathbf{s}$ is:

$$\begin{array}{cc}& {\mathbf{D}}_{\mathbf{s}}^i=-{\mathcal{H}}_{\mathbf{s}}^i{\mathrm{grad}}_{\mathbf{s}}f({\mathbf{s}}^i,{\mathbf{w}}^i)={({\mathbf{B}}_{\mathbf{s}}^{i-m}\dots {\mathbf{B}}_{\mathbf{s}}^{i-1})}^H{\mathcal{H}}_{\mathbf{s}}^{i-m}\left({\mathbf{B}}_{\mathbf{s}}^{i-m}\dots {\mathbf{B}}_{\mathbf{s}}^{i-1}\right){\mathrm{grad}}_{\mathbf{s}}f({\mathbf{s}}^i,{\mathbf{w}}^i)\hfill \\ \hfill & +{\varphi }_{i-m}{({\mathbf{B}}_{\mathbf{s}}^{i-m+1}\dots {\mathbf{B}}_{\mathbf{s}}^{i-1})}^H{\zeta }_{\mathbf{s}}^{i-m}{\left({\zeta }_{\mathbf{s}}^{i-m}\right)}^H({\mathbf{B}}_{\mathbf{s}}^{i-m+1}\dots {\mathbf{B}}_{\mathbf{s}}^{i-1}){\mathrm{grad}}_{\mathbf{s}}f({\mathbf{s}}^i,{\mathbf{w}}^i)\hfill \\ \hfill & +{\varphi }_{i-m+1}{\left({\mathbf{B}}_{\mathbf{s}}^{i-m+2}\dots {\mathbf{B}}_{\mathbf{s}}^{i-1}\right)}^H{\zeta }_{\mathbf{s}}^{i-m+1}{\left({\zeta }_{\mathbf{s}}^{i-m+1}\right)}^H\left({\mathbf{B}}_{\mathbf{s}}^{i-m+2}\dots {\mathbf{B}}_{\mathbf{s}}^{i-1}\right){\mathrm{grad}}_{\mathbf{s}}f\left({\mathbf{s}}^i,{\mathbf{w}}^i\right)\hfill \\ \hfill & +\dots +{\varphi }_{i-1}{\zeta }_{\mathbf{s}}^{i-1}{\left({\zeta }_{\mathbf{s}}^{i-1}\right)}^H{\mathrm{grad}}_{\mathbf{s}}f({\mathbf{s}}^i,{\mathbf{w}}^i),\hfill \end{array}$$

(38)

Similarly, the Hessian matrix ${\mathcal{H}}_{\mathbf{w}}$ and the search direction ${\mathbf{d}}_{\mathbf{w}}^i$ for $\mathbf{w}$ can also be determined following the steps described above.

Thus, the search direction is determined as:

$$\begin{array}{c}\hfill \left[\begin{array}{c}{\mathbf{D}}_{\mathbf{s}}^i\\ {\mathbf{d}}_{\mathbf{w}}^i\end{array}\right]=-\left[\begin{array}{c}{\mathcal{H}}_{\mathbf{s}}^i{grad}_{\mathbf{s}}\widehat{\mathcal{L}}({\mathbf{s}}^i,{\mathbf{w}}^i)\\ {\mathcal{H}}_{\mathbf{w}}^i{grad}_{\mathbf{w}}\widehat{\mathcal{L}}({\mathbf{s}}^i,{\mathbf{w}}^i)\end{array}\right]\end{array}$$

(39)

4.3.3. Retraction of Feasible Solutions

The update step from $({\mathbf{s}}^i,{\mathbf{w}}^i)$ to $({\mathbf{s}}^{i+1},{\mathbf{w}}^{i+1})$ on $\mathcal{M}$ is

$$({\mathbf{s}}^{i+1},{\mathbf{w}}^{i+1})=({\mathbf{s}}^i,{\mathbf{w}}^i)+{\eta }_i\left({\mathbf{d}}_{\mathbf{s}}^i,{\mathbf{d}}_{\mathbf{w}}^i\right)$$

(40)

where ${\eta }_i$ is the step size. After a tangent search direction is computed, a direct additive update generally does not remain on the manifold. In particular, the tentative waveform update may violate the constant modulus constraint. Therefore, a retraction operation is required to map the updated point back onto the manifold [13,25].

$$\begin{array}{cc}\hfill ({\mathbf{s}}^{i+1},{\mathbf{w}}^{i+1})& =\mathrm{Ret}\left(({\mathbf{s}}^i,{\mathbf{w}}^i)+{\eta }_i\left({\mathbf{d}}_{\mathbf{s}}^i,{\mathbf{d}}_{\mathbf{w}}^i\right)\right)\hfill \\ & =\left({\mathrm{Ret}}_{\mathbf{s}}\left({\mathbf{s}}^i+{\eta }_i{\mathbf{d}}_{\mathbf{s}}^i\right),{\mathrm{Ret}}_{\mathbf{w}}\left({\mathbf{w}}^i+{\mathbf{d}}_{\mathbf{w}}^i\right)\right)\hfill \\ & =\left({\displaystyle \frac{\left({\mathbf{s}}^i+{\eta }_i{\mathbf{d}}_{\mathbf{s}}^i\right)}{{∥\left({\mathbf{s}}^i+{\eta }_i{\mathbf{d}}_{\mathbf{s}}^i\right)∥}_F^2}},{\mathbf{w}}^i+{\eta }_i{\mathbf{d}}_{\mathbf{w}}^i\right)\hfill \end{array}$$

(41)

where $\mathrm{Ret}(·)$ is the retraction operator. Here we use the Armijo linear search method to find the step size ${\eta }_i$.

4.3.4. Adaptively Update the Step Size

We employ the Armijo linear search strategy to update the step size. This strategy adaptively adjusts the search step size ${\eta }_i$ based on momentum changes, ensuring algorithmic monotonicity while accelerating convergence. Typically, this strategy requires finding the smallest non-negative integer h satisfying:

$$f({\mathbf{s}}^{i+1},{\mathbf{w}}^{i+1})\le f({\mathbf{s}}^i,{\mathbf{w}}^i)+{\eta }_i{\tau }^h⟨{grad}^{H}f({\mathbf{s}}^i,{\mathbf{w}}^i),({\mathbf{D}}_{\mathbf{s}}^i,{\mathbf{d}}_{\mathbf{w}}^i)⟩$$

(42)

where ${\tau }^h\in (0,1)$ and $⟨(·,·),(·,·)⟩$ denotes the corresponding vector dot product.

If $h=1$, the initial step size is too small, and the next step requires a larger step size. If $h>2$, the convergence step size is too small due to excessive search iterations, necessitating a larger step size. If $h=2$, the search step size is appropriate. The new initial iteration step size is:

$${\eta }_{i+1}=\left\{\begin{array}{c}2{\tau }^h{\eta }_i,h=1\hfill \\ \\ {\tau }^h{\eta }_i,h=2\hfill \\ \\ 2{\tau }^h{\eta }_i,h>2\hfill \end{array}\right.$$

(43)

4.4. Updating the Penalty Factor

After obtaining and fixing the next iteration point $(\mathbf{s},\mathbf{w})$, we compute the violation of the penalty function and update $\rho$ and $\mu$. The specific update rules are as follows:

If $g\left(\mathbf{S}\right)$ exceeds the dynamic tolerance level $\gamma$, then $\rho$ is updated as:

$${\rho }_{n+1}={\displaystyle \frac{{\rho }_n}{{\omega }_{\rho }}},$$

(44)

where ${\omega }_{\rho }\in (0,1)$ and n denotes the number of iterations of the algorithm. Typically, the value of $\gamma$ is set equal to the smoothing parameter $\mu$, and $\mu$ is updated according to the following equation:

$${\mu }_{n+1}=max\left\{{\mu }_{min},{\omega }_{\mu }{\mu }_n\right\},$$

(45)

where ${\omega }_{\mu }\in (0,1)$ is a parameter used to gradually decrease ${\mu }_n$. While a smaller $\mu$ provides greater accuracy, it is necessary to ensure that $\mu$ satisfies the condition $\mu >{\mu }_{min}$ to avoid numerical difficulties.

The robustness of the algorithm is closely tied to the sensitivity of these hyperparameters. A smaller initial $\mu$ and a faster reduction rate ${\omega }_{\mu }$ improve the approximation accuracy of the max operator early on, but may lead to sharp gradients that hinder convergence. Conversely, a larger ${\omega }_{\rho }$ (closer to 1) ensures a smoother transition for the penalty term, preventing the algorithm from becoming trapped in poor local optima. Furthermore, the L-BFGS memory depth m significantly influences the convergence speed; while a larger m captures more second-order curvature information to accelerate descent, our analysis indicates that a value of $m\in [5,15]$ provides the best balance between computational overhead and final SINR stability. By adaptively updating $\rho$ and $\mu$ relative to the constraint violation, the algorithm maintains a stable feasibility–optimality trade-off, ensuring that the final SINR is robust against variations in the initial parameter settings.

As for the constraint violation behavior, the algorithm exhibits a characteristic “feasibility-seeking” trajectory. In the initial iterations, the algorithm prioritizes increasing the SINR while the penalty factor $\rho$ is relatively small, which may result in a temporary increase in the constraint violation $g\left(\mathbf{S}\right)$. However, as $\rho$ is adaptively scaled up and the smoothing parameter $\mu$ is reduced, the penalty for violating the jamming power threshold becomes dominant. This forces the iterates back into the feasible region of the original problem. By monitoring the evolution of $g\left(\mathbf{S}\right)$, we observe that the violation decreases monotonically once the penalty reaches a critical threshold, ultimately converging to a value below the predefined tolerance $ϵ$. This behavior demonstrates that the proposed update rules effectively balance the objective function optimization with strict constraint satisfaction, ensuring the practical reliability of the design in interference-limited environments.

4.5. Analysis of Computational Complexity and Convergence

4.5.1. Computational Complexity Analysis

To evaluate the efficiency of the proposed PCCEM method, we analyze the computational cost per iteration and compare it with the conventional ADMM-based approach. Let $N_t$, $N_r$, and $N_s$ denote the number of transmit antennas, receive antennas, and code length, respectively.

Let $L_s=N_t\times N_s$, $L_w=N_r\times M$. The computational complexity of the Proposed Method is $\mathcal{O}\left(L_w^2\right)$ when updating the filter $\mathbf{w}$, $\mathcal{O}\left(L_s^2\right)$ when updating the waveform $\mathbf{s}$, $\mathcal{O}\left(L_s\right)$ when calculating LPI constraint violations, and $\mathcal{O}(L_s+L_w)$ when calculating the search direction. Therefore, the computational complexity of the proposed algorithm is $\mathcal{O}((L_w^2+L_s^2)+(L_s+L_w))$. The ADMM method requires matrix inversion when updating the filter $\mathbf{w}$ and waveform $\mathbf{s}$ using MVDR, resulting in computational complexities of $\mathcal{O}\left(L_w^3\right)$ and $\mathcal{O}\left(L_s^3\right)$, respectively. Therefore, the computational complexity of the ADMM method is $\mathcal{O}(L_w^3+L_s^3)$. The detailed per-iteration complexity comparison between the proposed PCCEM method and the ADMM benchmark is reported in Table 1.

Table 1. Comparison of computational complexity.

Method	Per-Iteration Complexity	Main Operations
ADMM	$\mathcal{O}(N_r^3{M}^3+N_t^3{N}_s^3)$	Matrix inversions, proximal operators
Proposed PCCEM	$\mathcal{O}(N_r^2{M}^2+N_t^2{N}_s^2)$	Gradient calculations, retractions

As the system dimensions increase, the term in ADMM leads to a cubic growth in computational burden. The proposed PCCEM avoids the direct inversion of the large-scale covariance matrix by utilizing the geometry of the complex circle manifold and first-order Riemannian gradients. Consequently, the proposed method demonstrates superior scalability, making it more suitable for large-scale MIMO radar systems.

4.5.2. Convergence Analysis

To ensure the convergence of the proposed joint design algorithm, we establish the following formal statement. Assume the objective function $f(\mathbf{s},\mathbf{w})$ is L-smooth on the product manifold $\mathcal{M}={\mathcal{M}}_{\mathbf{s}}\times {\mathcal{M}}_{\mathbf{w}}$, and the sequences of iterates $\{{\mathbf{s}}^{\left(k\right)},{\mathbf{w}}^{\left(k\right)}\}$ remain in a compact subset of $\mathcal{M}$.

Theorem 1.

Let the sequence $\left\{x_k\right\}=\left\{({\mathbf{s}}^{\left(k\right)},{\mathbf{w}}^{\left(k\right)})\right\}$ be generated by the RL-BFGS algorithm under the proposed penalty update scheme. Under the assumptions that (i) the cost function is continuously differentiable and has a Lipschitz continuous gradient on the manifold, and (ii) the retraction $\mathcal{R}$ satisfies the standard conditions of being a first-order approximation to the exponential map, then the sequence of gradients satisfies ${\displaystyle \underset{k\to \infty }{lim}}\parallel gradf\left(x_k\right)\parallel =0$.

The convergence of the proposed algorithm to a stationary point is theoretically guaranteed by several key factors within the Riemannian optimization framework. Specifically, since the log-sum-exp smoothing function and the quadratic penalty terms are twice continuously differentiable on the compact product manifold $\mathcal{M}$, the Riemannian gradient $gradf$ is inherently Lipschitz continuous, satisfying a fundamental condition for the convergence of quasi-Newton methods. Furthermore, because the search space is defined by the product manifold, the iterates are naturally bounded; according to the Bolzano–Weierstrass theorem on manifolds, this ensures the existence of at least one accumulation point that satisfies the first-order optimality conditions [26]. This process is further stabilized by the penalty update scheme, where the penalty factor $\rho$ is adjusted at a slower timescale than the inner RL-BFGS iterations. Following the established framework of the Augmented Lagrangian Method on Manifolds [27], this multi-scale approach ensures that the sequence of solutions for the subproblems asymptotically converges to a feasible stationary point of the original constrained problem.

In summary, the robustness of the convergence is rooted in the analytical smoothness of the objective function and the geometric compactness of the manifold. By leveraging these properties, the algorithm maintains a rigorous path toward a stationary solution that satisfies both the optimality requirements and the physical constraints of the MIMO radar system.

5. Simulation Results

In this section, we conduct extensive numerical simulations to evaluate the effectiveness of the proposed method. Specifically, we analyze the radar’s performance in terms of its output SINR improvement, probability of detection, and the depth of the nulls formed in the interference directions. Furthermore, the computational efficiency and convergence characteristics of the proposed algorithm are assessed to demonstrate effectiveness. To provide a rigorous comparative analysis, the typical ADMM method is utilized as benchmarks to highlight the superiority of the proposed method.

In the simulations, the numbers of transmit antennas and receive antennas are set to $N_t=8$ and $N_r=6$, respectively. The number of radar snapshots is $N_{\mathbf{s}}=20$, and the number of interferences is $N_a=60$. Unless otherwise specified, the signal-to-noise ratio (SNR) is set to $10\mathrm{dB}$. The extended-target echo occupies $Q=13$ range bins, and the nominal target azimuth is ${\varphi }_p={35}^{\circ }$. Moreover, three cases are considered for the possible angular span of the target azimuth angle (TAA): $[{34.5}^{\circ },{35.3}^{\circ }]$, $[{34.5}^{\circ },{36.1}^{\circ }]$, and $[{34.5}^{\circ },{36.9}^{\circ }]$, all sampled with an angular step of ${0.4}^{\circ }$. The jamming azimuth angle is set to ${\varphi }_I=-{19}^{\circ }$.

Unless otherwise stated, the penalty and smoothing parameters are set to ${\rho }_0=1$, ${\mu }_0={10}^{-1}$, ${\mu }_{min}={10}^{-6}$, ${\omega }_{\rho }=0.3$, and ${\omega }_{\mu }={({\mu }_{min}/{\mu }_0)}^{1/30}$, respectively, and the L-BFGS memory depth is chosen as $m=10$. The inner manifold iterations are terminated when $\parallel gradf({\mathbf{s}}^{\left(k\right)},{\mathbf{w}}^{\left(k\right)}){\parallel }_2\le {10}^{-6}$, or when the relative objective reduction is below ${10}^{-4}$, or when the maximum number of inner iterations 100 is reached. The outer penalty loop is terminated when $max\{0,P(\mathbf{S},{\varphi }_I)-\omega \}\le {10}^{-4}$, or when the maximum number of outer iterations 50 is reached.

For a fair comparison, the ADMM baseline is implemented under the same reciprocal-SINR objective and the same unimodular and jamming-power constraints as the proposed PCCEM method. It is initialized with the same ${\mathbf{s}}^{\left(0\right)}$ and ${\mathbf{w}}^{\left(0\right)}$ as PCCEM. The ADMM penalty parameter(s), stopping tolerance, and maximum iteration number are set to 10, ${10}^{-6}$, and 2000, respectively. All ADMM results reported in this section are obtained using these fixed settings for all three TAA scenarios.

5.1. Optimality Gap and Benchmarking

To assess the optimality of the proposed PCCEM method, we define an ideal upper bound for the SINR. The power of the unimodular constraint is $\parallel \mathbf{s}{\parallel }^2=e_T$ and assuming the absence of interference (signal-to-noise ratio only), the SINR is bounded as follows:

$$\begin{array}{cc}\hfill & {\displaystyle \frac{{\left|{\mathbf{w}}^H{\mathbf{H}}_a\mathbf{x}\right|}^2}{{\mathbf{w}}^H\left[{\sum }_{\overrightarrow{k}=1}^N{\mathbf{C}}_{\overrightarrow{k}}^H{\mathbf{A}}^H\left({\varphi }_{\overrightarrow{k}}\right)\mathbf{w}{\mathbf{w}}^H\mathbf{A}\left({\varphi }_{\overrightarrow{k}}\right){\mathbf{C}}_{\dot{k}}\right]\mathbf{w}+{\sigma }_{\nu }^2{\mathbf{w}}^H\mathbf{w}}}\hfill \\ \hfill & \le {\displaystyle \frac{|{\mathbf{w}}^H{\mathbf{H}}_a{\mathbf{s}|}^2}{{\sigma }_{\nu }^2{\mathbf{w}}^H\mathbf{w}}}\le {\displaystyle \frac{{\mathbf{s}}^H{\mathbf{H}}_a^H{\mathbf{H}}_a\mathbf{x}}{{\sigma }^2}}\le {\displaystyle \frac{{\mathbf{s}}^H{\mathbf{H}}_a^H{\mathbf{H}}_a\mathbf{s}}{{\sigma }_{\nu }^2}}\hfill \\ \hfill & \le {\displaystyle \frac{e_{\mathrm{T}}{\lambda }_{\max }\left(I_L\otimes \left(a_{\mathrm{t}}^{*}{a}_{\mathrm{r}}^{\mathrm{H}}{a}_{\mathrm{r}}{a}_{\mathrm{t}}^{\mathrm{T}}\right)\right)}{{\sigma }_{\nu }^2}}\hfill \\ \hfill & ={\displaystyle \frac{e_{\mathrm{T}}{N}_{\mathrm{r}}{\lambda }_{\max }\left(a_{\mathrm{t}}^{*}{a}_{\mathrm{t}}^{\mathrm{T}}\right)}{{\sigma }_{\nu }^2}}={\displaystyle \frac{e_{\mathrm{T}}{\lambda }_{\max }{N}_{\mathrm{r}}{N}_{\mathrm{t}}}{{\sigma }_{\nu }^2}}\hfill \end{array}$$

(46)

where the second inequality uses the Cauchy–Schwarz inequality, where ${\lambda }_{\max }$ represents the maximum singular value of the target impulse response matrix, $e_{\mathrm{T}}$ represents the energy of the waveform vector. $N_t$ is the number of transmitting antennas, and $N_r$ is the number of receiving antennas.

In our simulations, we compare the SINR achieved by the proposed PCCEM method against this theoretical bound. As shown in the updated Figure 1, the proposed method achieves an SINR that is within 1 dB of the theoretical upper bound even under strict unimodular and suppression jamming constraints. This indicates that the proposed manifold-based optimization effectively navigates the non-convex search space to reach a solution with a negligible optimality gap. All algorithms in this paper were run on MATLAB R2020b on a laptop equipped with a 2.5 GHz CPU and 16 GB of memory.

Figure 1. Average SINR versus runtime for the PCCEM, ADMM, and RCG methods under different TAA interval sizes ($K=3,5,7$), together with the theoretical upper bound ${\mathrm{SINR}}_{\mathrm{opt}}$.

5.2. Comparison of Interference Nulling

Figure 2 shows the comparison of the main lobe position of the pattern and the pattern null at the interceptor location for the three cases of $\mathrm{K}=3$, $\mathrm{K}=5$, $\mathrm{K}=7$. It can be seen that in both cases, the proposed method achieves higher mainlobe energy at the target position ${\varphi }_p={35}^{\circ }$ than both the ADMM and RCG methods, and the proposed method achieves a deeper pattern null near the interceptor position ${\varphi }_I=-{19}^{\circ }$.

Figure 2. Beampatterns obtained by the proposed method, ADMM and RCG under different TAA ranges, illustrating radiation suppression at the interceptor direction. (a) $K=3$. (b) $K=5$. (c) $K=7$.

5.3. Comparison of SINR

Figure 3 illustrates the comparison of SINR values at different angles within the TAA set for three scenarios: $K=3$, $K=5$, and $K=7$. The simulation results demonstrate that the proposed method consistently outperforms the benchmark ADMM method in all tested scenarios. Specifically, the proposed approach achieves a superior SINR gain of approximately 1 dB across the entire angular range. This performance edge indicates that by effectively navigating the non-convex constraints of the waveform design, our method achieves a more precise balance between mainlobe beamforming and co-channel interference suppression.

Figure 3. Convergence behavior of average SINR for the proposed method, ADMM and RCG under different TAA ranges. (a) $K=3$. (b) $K=5$. (c) $K=7$.

5.4. Discussion on Cognitive Robustness and Environmental Complexity

The proposed joint design operates on the principle of cognitive radar, where prior knowledge of the jammer’s spatial coordinates is assumed to be provided by the system’s perception layer (e.g., through previous spectrum sensing or direction-finding results). Within this framework, we identify two primary areas regarding the method’s practical scalability:

(1) Multi-Jammer scenarios: In a cognitive loop where multiple interference sources are identified at directions $\{{\varphi }_{I,1},\dots ,{\varphi }_{I,K}\}$, the optimization problem can be extended by incorporating a multi-null penalty term ${\displaystyle \sum _{k=1}^K}max\{0,P(S,{\varphi }_{I,k})-\omega \}$. While the Riemannian optimization framework remains valid, the increase in non-smooth constraints may require a more careful selection of the penalty update rate to maintain convergence speed.

(2) Angular uncertainty and robustness: A limitation of the current design is its sensitivity to the precision of the prior knowledge. In cognitive radar, estimation errors in the jammer direction ${\varphi }_I$ can lead to “mismatch”, where the deep nulls in the radar beampattern fail to suppress the interference. To mitigate this, the method could be evolved into a “robust cognitive design” by optimizing the integrated jamming power over a small uncertainty region centered at the estimated direction. In conclusion, while the current study provides a rigorous foundation for extended target detection under the assumption of accurate prior knowledge, future work will focus on integrating robust manifold optimization to handle the dynamic and uncertain nature of real-world cognitive sensing cycles.

5.5. Comparison of Probability of Detection

Figure 4 shows the detection probability $P_d$ versus the power ${\left|{\alpha }_{\mathrm{T}}\right|}^2$ for the proposed method and the ADMM method: (a) $K=3$, (b) $K=5$, and (c) $K=7$. As can be seen, the proposed method obtains better $P_d$ than the existing method with the same power ${\left|{\alpha }_{\mathrm{T}}\right|}^2$. Specifically, when $K=5$ and ${\left|{\alpha }_{\mathrm{T}}\right|}^2=-10.8$ dB, the proposed method is 99%, which is 18% higher than ADMM method.

Figure 4. Detection probability $P_d$ versus $|{\alpha }_{\mathrm{T}}{|}^2$ for the proposed method and ADMM under different TAA ranges. (a) $K=3$. (b) $K=5$. (c) $K=7$.

5.6. Robustness of Convergence

Figure 5 presents the SINR convergence plots and box plots from 50 Monte Carlo experiments under three scenarios: $K=3$, $K=5$ and $K=7$. It is evident that SINR converged within 40 iterations across all scenarios. Furthermore, the boxplots indicate no outliers occurred in any of the three scenarios during the 50 Monte Carlo experiments, demonstrating the algorithm’s stability.

Figure 5. Performance evaluation over 50 Monte Carlo simulations.

6. Conclusions

This paper has addressed the challenging problem of joint unimodular waveform and receive filter design for MIMO radar extended target detection in the presence of suppressive jamming. By leveraging the geometric properties of the complex circle manifold, we developed the Product Complex Circle Euclidean Manifold (PCCEM) method, which effectively circumvents the performance loss associated with traditional convex relaxation techniques. Our approach successfully optimizes the output SINR while maintaining strict unimodular constraints and forming precise jamming notches. Simulation results validate that the PCCEM method not only achieves superior detection performance and deeper interference suppression compared to benchmark algorithms but also offers significant advantages in computational efficiency, making it a robust solution for complex electromagnetic environments.