Conditioning Analysis of Orthogonal Polynomial Models for Receiver Nonlinear Behavioral Model

Abstract

Receiver nonlinear distortion severely impacts modern wireless systems. Traditional power series polynomial models suffer from numerical instability in parameter estimation, especially at high orders or with memory. This paper investigates orthogonal memory polynomial models from the perspectives of memory depth, nonlinear order, input signal distribution, and temporal correlation of the input signal, focusing on effective methods for improving the condition number. Comprehensive analysis reveals that the condition number of the Gram matrix grows rapidly with polynomial order and memory depth for the conventional polynomial, while orthogonal polynomials remain well-conditioned due to their inherent orthogonality and normalization. Notably, orthogonal polynomials maintain robust performance even when the input distribution does not perfectly match the basis weight function. Experiments using OFDM and 3-carrier WCDMA signals confirm that orthogonal polynomials achieve condition numbers orders of magnitude lower than those of power series, along with superior fitting accuracy.

1. Introduction

With the evolution of wireless communication systems toward higher data rates and wider bandwidths, receiver nonlinear distortion has emerged as a major performance-limiting factor. The RF front-end components, including low-noise amplifiers (LNAs), mixers, and analog-to-digital converters (ADCs), exhibit inherent nonlinear characteristics. When combined with modern modulation formats that feature multi carrier structures, large bandwidth, and high peak to average power ratios (PAPR), the resulting nonlinear distortion—such as spectral regrowth and in band intermodulation—severely degrades signal quality [1,2,3].

Recent advances in digital signal processing have enabled effective post compensation of nonlinear distortions in the digital domain. The core of such compensation lies in constructing an accurate behavioral model of the receiver’s nonlinearity, as illustrated in Figure 1. Among various modeling approaches, the polynomial model is widely used due to its simple structure and analytical tractability [4,5,6,7]. Although receiver nonlinearity is often weaker than that of a transmitter power amplifier, it can significantly degrade performance in modern wideband and high-dynamic-range receivers, especially in the presence of strong blockers or when using direct-conversion architectures [8,9]. Conventional polynomial models have been widely adopted for receiver nonlinearity characterization [8,9,10,11], but their parameter estimation often suffers from numerical instability, which can significantly degrade the accuracy of model extraction and compensation performance [12,13,14]. This instability has been well documented in the context of power amplifier digital predistortion [15,16], but also occurs in receiver modeling when the polynomial order or memory depth is high.

Figure 1. Block scheme for nonlinear distortion mitigation by using the band-splitting filter [8] and reference receiver [9].

To address this issue, researchers have explored orthogonal polynomials for nonlinear modeling and digital predistortion (DPD) of power amplifiers (PAs), demonstrating improved numerical stability [11,12,13,14]. Yet, systematic studies on applying orthogonal polynomial models to receiver nonlinear distortion cancellation remain relatively scarce. Moreover, most existing works on orthogonal polynomial modeling focus on transmitter nonlinearity [6,17,18], and the conditioning properties of different orthogonal bases in the receiver context have not been systematically investigated under realistic input distributions, memory depths, and temporal correlations.

This paper fills this gap by analyzing the condition number of the Gram matrix formed by different polynomial bases under various input signal distributions and by comparing their numerical stability and modeling accuracy. Through comprehensive analysis and experiments using realistic communication signals (OFDM and 3-carrier WCDMA), we demonstrate that orthogonal polynomial bases significantly improve numerical stability while maintaining high modeling accuracy. The results provide valuable insights for designing robust nonlinear compensation schemes in high-spectral-efficiency receiver systems. The proposed orthogonal polynomial approach can serve as a pre-processing module to compensate receiver nonlinearity or be embedded into the neural network as a physically informed layer [19,20]. This remains an interesting direction for future research. The applicability and effectiveness of such models in receiver scenarios warrant further investigation.

The remainder of this paper is organized as follows. Section 2 presents the receiver model and defines the condition number metric. Section 3 analyzes the conditioning of different polynomial bases under memoryless and memory scenarios. Section 4 provides experiment results using OFDM and WCDMA signals, followed by discussion. Section 5 concludes the paper.

2. Receiver Behavior Model and Conditioning Analysis

2.1. Receiver Behavioral Model

The nonlinearity of a receiver primarily originates from three analog components in the front-end: the low-noise amplifier (LNA), the mixer, and the analog-to-digital converter (ADC). The LNA exhibits gain compression and AM-PM conversion when the input signal power exceeds its 1 dB compression point, especially in the presence of strong out-of-band interferers. The mixer introduces intermodulation products due to its switching operation and finite port isolation. The ADC contributes quantization nonlinearity and may clip when the input signal exceeds its full-scale range. Although these nonlinearities are typically weaker than those of a high-power transmitter amplifier, they become significant in modern wideband receivers with high dynamic range, such as those used in cognitive radio, full-duplex communication, and 5G New Radio (NR) systems. The combined effect of these components can be captured by a behavioral model that treats the receiver as a black box, as illustrated in Figure 2. This approach has been widely adopted in the literature [8,9,10,11]. Among various behavioral models, the memory polynomial is one of the most popular due to its simplicity and ability to represent weakly nonlinear systems with memory [4,5].

Figure 2. Structure of typical receiver front-end.

The input RF signal can be expressed as a bandpass signal around the carrier frequency:

$$x_{\mathrm{RF}}(n)=\mathrm{Re}\left\{x\left(n\right){\mathrm{e}}^{j{\omega }_{\mathrm{c}}n}\right\},$$

(1)

where $x(n)=A(n)e^{j\phi \left(n\right)}=x_I(n)+j{x}_Q(n)$ is the complex baseband signal, $A\left(n\right)$ and $\phi \left(n\right)$ are the amplitude and phase, respectively, and ${\omega }_c$ is the carrier angular frequency. Similarly, the receiver front-end output can be expressed as

$$y_{\mathrm{IF}}(n)=\mathrm{Re}\{y(n){\mathrm{e}}^{j{\omega }_{\mathrm{IF}}n}\}=\mathrm{Re}\left\{\left[y_{\mathrm{I}}(n)+j{y}_{\mathrm{Q}}(n)\right]e^{j{\omega }_{\mathrm{IF}}n}\right\},$$

(2)

which is around ${\omega }_{IF}$. The nonlinear relationship between the complex baseband input $x\left(n\right)$ and output $y\left(n\right)$ can be represented by a memory polynomial, which is a widely used behavioral model for nonlinear systems with memory [4,5,6,7]:

$$y(n)=\sum _{k=1}^K{a}_k\ast {\varphi }_k\left[x\left(n\right)\right]=\sum _{m=0}^M\sum _{k=1}^K{a}_{k,m}{\varphi }_k\left[x\left(n\right)\right],$$

(3)

where $a_{k,m}$ denotes the impulse response for each polynomial order taking memory effects into account, M denotes the memory length, the symbol * denotes the convolution, and ${\varphi }_k(·)$ denotes the k-th polynomial basis function. The Equation (3) can be rewritten in matrix form as

$$\mathbf{Y}=\mathbf{\Phi }\mathbf{a},$$

(4)

where $\mathbf{Y}={\left[y\left(M\right),y(M+1),\dots ,y\left(N\right)\right]}^{\mathrm{T}}$, $\mathbf{a}={\left[a_{1,0},\dots ,a_{K,0},\dots ,a_{1,M},\dots ,a_{K,M}\right]}^{\mathrm{T}}$, and $\mathbf{\Phi }=\left[{\mathbf{\Phi }}_{1,0},\dots ,{\mathbf{\Phi }}_{K,0},\dots ,{\mathbf{\Phi }}_{1,m},\dots ,{\mathbf{\Phi }}_{K,m},\dots ,{\mathbf{\Phi }}_{1,M},\dots ,{\mathbf{\Phi }}_{K,M}\right]$ with the element ${\mathbf{\Phi }}_{k,m}={\left[{\varphi }_k\left[x(M-m)\right],\dots ,{\varphi }_k\left[x(N-m)\right]\right]}^T$, and ${(·)}^T$ denotes matrix transpose. The corresponding LS estimate for the model coefficients is given by

$$\mathbf{a}={({\mathbf{\Phi }}^{\mathrm{H}}\mathbf{\Phi })}^{-1}{\mathbf{\Phi }}^{\mathrm{H}}\mathbf{Y},$$

(5)

and ${(·)}^{\mathrm{H}}$ denotes the Hermitian transpose. It shows that the major implementation challenge of solving the model coefficients $\mathbf{a}$ is the inversion of the matrix ${\mathbf{\Phi }}^{\mathrm{H}}\mathbf{\Phi }$, which is often ill-conditioned and causes numerical instability [12,13,15].

2.2. Condition Number and Numerical Stability

The numerical stability of the LS solution is measured by the condition number of the Gram matrix $\mathbf{G}={\mathbf{\Phi }}^{\mathrm{H}}\mathbf{\Phi }$, defined as

$$\kappa \left(\mathbf{G}\right)={\displaystyle \frac{\mid {\lambda }_{max}\mid }{\mid {\lambda }_{min}\mid }},$$

(6)

where ${\lambda }_{max}$ and ${\lambda }_{min}$ are the maximum and minimum eigenvalues of $\mathbf{G}$, respectively. A large condition number indicates that the matrix is ill-conditioned, making the solution highly sensitive to perturbations and numerical errors [7]. The Gram matrix has the following block structure:

$$\mathbf{G}=\left(\begin{array}{cccc}{\mathbf{\Phi }}_{1,0}^H{\mathbf{\Phi }}_{1,0}& {\mathbf{\Phi }}_{1,0}^H{\mathbf{\Phi }}_{2,0}& \dots & {\mathbf{\Phi }}_{1,0}^H{\mathbf{\Phi }}_{K,M}\\ {\mathbf{\Phi }}_{2,0}^H{\mathbf{\Phi }}_{1,0}& {\mathbf{\Phi }}_{2,0}^H{\mathbf{\Phi }}_{2,0}& \dots & {\mathbf{\Phi }}_{2,0}^H{\mathbf{\Phi }}_{K,M}\\ ⋮& ⋮& \ddots & ⋮\\ {\mathbf{\Phi }}_{K,M}^H{\mathbf{\Phi }}_{1,0}& {\mathbf{\Phi }}_{K,M}^H{\mathbf{\Phi }}_{2,0}& \dots & {\mathbf{\Phi }}_{K,M}^H{\mathbf{\Phi }}_{K,M}\end{array}\right).$$

(7)

Each element represents the cross correlation between different basis functions at possibly different time delays. The condition number is, therefore, directly influenced by the correlations among the basis functions and the input signal. Significant disparities in the magnitudes of matrix elements often lead to a large condition number, which can make the matrix nearly singular, resulting in ill-conditioning.

When the memory effect is not considered (M = 0), the matrix $\mathbf{\Phi }$ simplifies to $\mathbf{\Phi }=[{\mathbf{\Phi }}_{1,0},\dots ,{\mathbf{\Phi }}_{K,0}]$, then the correlation between basis functions of different orders largely determines the condition number. We first consider the memoryless case ($M=0$) where the input samples are independent and identically distributed (i.i.d.). This allows us to isolate the effect of amplitude distribution on the condition number. The impact of correlation is studied separately in the following subsection.

For the conventional power series model, ${\varphi }_k\left(x\right)=x{\left|x\right|}^{k-1}$. Figure 3 shows the condition number as a function of polynomial order K for different input distributions using the power series basis. The input samples are independent and identically distributed to isolate the effect of amplitude distribution from temporal correlation. Because the power series basis functions are highly correlated, the Gram matrix becomes ill-conditioned, especially when the input signal has a large dynamic range or the model order is high.

Figure 3. Condition number of conventional power series polynomial with different input distributions (memoryless).

Theoretically, when the basis functions are orthogonal under the input signal distribution, i.e., $\mathbb{E}\left[{\varphi }_i{\varphi }_j^{\ast }\right]=0,i\ne j$, the Gram matrix becomes diagonal and the condition number is minimized ($\kappa =1$). However, achieving perfect orthogonality in practice is challenging, especially when memory effects are present and the input signal distribution is unknown a priori. Nevertheless, selecting basis functions that approximately match the input distribution can significantly improve conditioning.

3. Conditioning Analysis of Different Polynomial Basis Functions

In this section, we analyze the influence of nonlinear order and memory depth on the condition number using the conventional power series and three orthogonal polynomial basis functions: Legendre, Hermite, and Laguerre. The power series is selected as the baseline because it is the most widely used model for receiver nonlinearity in both academia and industry [4,5]. The orthogonal bases are chosen because their weight functions correspond to common input signal distributions: Legendre for uniform (e.g., ideal quantization) [21], Hermite for Gaussian (e.g., noise or wideband OFDM) [14], and Laguerre for Rayleigh amplitude (e.g., OFDM signals) [22].

3.1. Basis Function Definition and Normalization

Before analyzing the conditioning of different polynomial bases, we provide the explicit complex baseband forms and normalization strategies for the four basis functions used in this paper. In all cases, the direction term $x/\left|x\right|$ preserves the phase information, and the normalization ensures that the Gram matrix of the basis functions approaches the identity matrix when the input distribution matches the polynomial’s weight function. Power series (conventional):

$${\varphi }_k\left(x\right)=x·{\left|x\right|}^{k-1},k=1,\dots ,K.$$

(8)

Legendre polynomials (normalized for uniform distribution):

$${\varphi }_k\left(x\right)=\sqrt{{\displaystyle \frac{2k+1}{2}}}{\mathrm{P}}_{\mathrm{k}}({\displaystyle \frac{2\mathrm{r}}{{\mathrm{A}}_{\max }-1}})·{\displaystyle \frac{x}{\left|x\right|}},$$

(9)

where $P_k(·)$ is the Legendre polynomial of order k, $A_{max}=max\left(\right|x\left|\right)$ over the training set, and the factor $\sqrt{(2k+1)/2}$ ensures unit norm under the uniform distribution on $[-1,1]$. Hermite polynomials (normalized for Gaussian distribution):

$${\varphi }_k\left(x\right)={\displaystyle \frac{1}{\sqrt{k!}}}{\mathrm{He}}_k\left({\displaystyle \frac{(r-{\mu }_{\left|x\right|})}{{\sigma }_{\left|x\right|}}}\right)·{\displaystyle \frac{x}{\left|x\right|}},$$

(10)

where ${\mathrm{He}}_k(·)$ is the probabilist’s Hermite polynomial, and $1/\sqrt{k!}$ is the normalization factor, and ${\mu }_{\left|x\right|}$ and ${\sigma }_{\left|x\right|}$ are the mean and standard deviation of $\left|x\right|$ over the training set. Laguerre polynomials (normalized for Rayleigh-distributed amplitude):

$${\varphi }_k(x)=L_k({\displaystyle \frac{{\left|x\right|}^2}{p_{\mathrm{avg}}}})·{\displaystyle \frac{x}{\left|x\right|}},$$

(11)

where $p_{\mathrm{avg}}={\mathbb{E}\left[\right|x|}^2]$, $L_k(·)$ is the standard Laguerre polynomial, which already has unit norm under the exponential distribution. For OFDM signals, the amplitude $\left|x\right|$ follows a Rayleigh distribution, and ${\displaystyle \frac{{\left|x\right|}^2}{p_{\mathrm{avg}}}}$ follows an exponential distribution.

3.2. Influence of Polynomial Order and Memory Depth

Figure 4a shows the condition number as a function of K with $M=4$ for different basis functions. The power series condition number grows rapidly (exponentially) with K, especially for exponential and Gaussian inputs where the high-order terms introduce large numerical disparities, making the Gram matrix nearly singular. In contrast, orthogonal polynomial bases, when properly normalized and matched to the input distribution, yield very small condition numbers. Owing to the finite sample size N and truncation effects, the condition numbers for exponential and Gaussian distributions also increase with K, but they remain far superior to the power series.

Figure 4. Condition number of different orthogonal basis polynomials: (a) variance of K with M = 4; (b) variance of M with K = 3.

Figure 4b depicts the condition number as a function of memory depth M (fixed $K=7$). For the power series, each additional delay adds K new columns that are correlated with existing columns, leading to an approximate linear growth in the condition number with M. For orthogonal basis functions, the condition number increases mildly because different delay basis functions are nearly orthogonal for independent and identically distributed inputs and normalization ensures unit column norms, keeping the Gram matrix close to block diagonal.

3.3. Impact of Input Correlation

To investigate the impact of input temporal correlation on the conditioning of different polynomial bases, we generate a first order autoregressive (AR(1)) process with varying correlation:

$$x\left[n\right]=\phi x[n-1]+w\left[n\right],w\left[n\right]\sim \mathcal{N}(0,1),$$

(12)

where $\phi \in (0,1)$ controls the correlation strength. By adjusting $\phi$, we can quantitatively evaluate the influence of the input signal’s memory effect on the cross correlation among the columns of the design matrix $\mathbf{\Phi }$ and, thereby, analyze the numerical stability of different polynomial basis functions in least squares estimation. We use a Gaussian AR(1) input and compare the condition numbers of the power series and Hermite basis function (since Gaussian input matches Hermite polynomials), and the simulation results are shown in Figure 5.

Figure 5. Conditionnumber vs. K and M for AR(1) input with different $\phi$: (a) M = 3; (b) K = 3.

It is worth noting that the AR(1) model generates stationary Gaussian signals, whereas real modulated signals such as OFDM and WCDMA are cyclostationary and have more complex correlation structures determined by their power spectral density. The AR(1) model is used here as a simplified, controllable tool to illustrate the effect of temporal correlation strength on conditioning. For the conventional power series, even moderate correlation leads to a significant increase in the condition number because same order basis functions at different delays become correlated, and higher $\phi$ exacerbates this effect. For the Hermite basis function, the condition number also rises very slowly with $\phi$ and M, and it remains orders of magnitude smaller than that of the power series.

Although different delay basis functions become statistically correlated, the orthogonal basis function benefits from column normalization and orthogonality among different orders at the same delay. These properties keep the singular value distribution of the Gram matrix much more uniform than that of the power series, resulting in a far smaller condition number even under strong temporal correlation. In practice, the receiver’s input signal typically has strong temporal correlation (e.g., modulated signals after pulse shaping). Hence, the power series is highly prone to ill conditioning, while normalized orthogonal polynomials maintain excellent numerical stability, making them a reliable choice for high precision nonlinear compensation.

Regarding computational complexity, the proposed orthogonal polynomial model has real-time overhead comparable to that of the conventional power series. For each input sample, both models require computing M delayed inputs and K basis functions per delay. Using power series, recursive calculation of $x{\left|x\right|}^{k-1}$ costs about $M(2K-1)$ real multiplications. For orthogonal polynomials, the three-term recurrence costs about $2K$ real multiplications per delay, plus one multiplication for the direction term. Thus the difference is within a factor of two. The main advantage lies in the training phase: Orthogonal bases yield a well-conditioned Gram matrix, enabling faster and more stable coefficient estimation compared to the ill-conditioned power series, which may require regularization or higher precision.

4. Experimental Results

Before presenting the experiments, we briefly discuss the practical applicability of the proposed orthogonal polynomial model. The model preserves the memory polynomial structure and only replaces the basis functions; therefore, it remains fully compatible with existing communication protocols such as OFDM and WCDMA. Moreover, the three-term recurrence relations of orthogonal polynomials enable efficient hardware implementation on FPGAs and DSPs, as demonstrated in prior works on digital predistortion [11,14]. The model is suitable for direct-conversion and IF receivers where the nonlinearity can be approximated by a memory polynomial of moderate order and memory depth. Its numerical stability advantage becomes critical for high-PAPR signals or when the receiver operates over a wide dynamic range.

To validate these points., we evaluate the performance using two high-PAPR realistic signals: an OFDM signal (64 subcarriers, 16QAM modulation, cyclic prefix length 16) and a 3 carrier WCDMA signal (each carrier with 4 channel QPSK superposition, root raised cosine shaping). The input–output data used in this paper are obtained from an actual wideband direct-conversion receiver [3]. Wideband modulated signals (OFDM and 3 carrier WCDMA) are generated and applied to the receiver input, and the corresponding output signals are captured through a high-speed digitizer. The measured data are then preprocessed by removing the first few samples to eliminate transient effects and normalized to unit power. From the captured sequences, the first 3000 valid samples are used for training, and the next 2000 samples are used for testing. This split is performed independently for each signal type.

Figure 6 shows the normalized power spectral density (PSD) of the receiver nonlinear output and the model predictions for the OFDM and 3-carrier WCDMA signals (with behavioral models of $K=7$, $M=3$). Table 1 reports the normalized mean square error (NMSE) for OFDM and 3-carrier WCDMA signals with different model polynomial bases. The conventional power series exhibits the largest errors ($-15\mathrm{dB}$). Legendre and Hermite polynomials achieve moderate improvement (about $-30\mathrm{dB})$, while the Laguerre polynomials give the best performance, with NMSE below $-35\mathrm{dB}$ (for OFDM) and $-30\mathrm{dB}$ (for 3-carrier WCDMA) respectively. For OFDM signals, the amplitude follows an approximate Rayleigh distribution; the squared amplitude is exponentially distributed, making the Laguerre polynomial (which corresponds to an exponential distribution) the best match. Consequently, Laguerre achieves the lowest condition number and highest accuracy. The conventional polynomial, due to its ill-conditioned Gram matrix, suffers from severe accuracy degradation.

Figure 6. PSD comparison for OFDM and 3-carrier WCDMA signals ($K=7$, $M=3$) (a) OFDM. (b) 3-carrier WCDMA.

Table 1. NMSE (dB) for different basis functions ($K=7$, $M=3$).

Signal	Conventional	Legendre	Laguerre	Hermite
OFDM	−14.8	−30.5	−35.9	−27.8
3 carrier WCDMA	−17.1	−28.3	−30.9	−29.6

Although the amplitude distribution of 3-carrier WCDMA differs from that of OFDM, the numerical stability advantage of orthogonal polynomials remains evident. The power series still exhibits large errors, while Laguerre and Hermite both maintain low errors. Even when the input distribution does not perfectly match the basis function’s weight, orthogonal polynomials outperform the power series due to superior numerical stability.

In practice, the exact input distribution is often unknown a priori. However, approximate matching can be achieved by the following methods: (1) selecting the basis based on the known signal type (e.g., Laguerre for OFDM, Hermite for Gaussian noise, Legendre for uniformly quantized signals); (2) estimating the amplitude distribution from a short training sequence via histogram and choosing the best-matched basis; (3) using a universal orthogonal basis that provides near-optimal uniform approximation. Even when the distribution does not perfectly match, our experiments with WCDMA (which does not strictly follow any of the three distributions) show that orthogonal bases still significantly outperform the power series, demonstrating robustness to distribution mismatch.

5. Conclusions

This paper systematically analyzed the conditioning properties of orthogonal polynomial bases for receiver nonlinear distortion modeling. We deeply compared conventional polynomial models and orthogonal polynomial models from the perspectives of memory depth, nonlinear order, input signal distribution, and temporal correlation of the input signal, focusing on effective methods for improving the condition number. Comprehensive analysis and extensive experiments using OFDM and 3-carrier WCDMA signals revealed that the power series basis leads to condition numbers that grow exponentially with polynomial order K and memory depth M, causing severe ill-conditioning, whereas orthogonal polynomials maintain condition numbers orders of magnitude lower, thereby ensuring robust least squares estimation. Furthermore, orthogonal bases achieve a lower normalized mean square error (NMSE) than the power series, especially for high order models, due to improved numerical stability; this advantage is most pronounced when the input signal distribution matches the orthogonal polynomial’s weighting function. For high-PAPR signals commonly used in modern communications, employing distribution-matched orthogonal polynomials is essential for reliable nonlinear modeling and compensation, though even when the distribution is not perfectly matched, orthogonal bases still outperform the power series. This paper provides valuable guidance for the design of robust nonlinear compensation schemes in future high-spectral-efficiency receiver systems.