Closed-Form Approximation to the Average Symbol Error Probability for Cross-QAM over κ–μ Fading Channels with Experimental Validation in the Millimeter-Wave Band

Abstract

This work presents a closed-form approximation to the symbol error probability (SEP) for cross-quadrature amplitude modulation (cross-QAM) schemes over $\mathit{\kappa }$–$\mathit{\mu }$ fading channels. The proposed formulation enables accurate performance evaluation while avoiding computationally expensive numerical integration. The analysis covers millimeter-wave (mmWave) frequencies at 55, 60, and 65 GHz, under both line-of-sight (LoS) and non-line-of-sight (nLoS) conditions, and for multiple transmitter–receiver polarization configurations. A key contribution of this work is the experimental validation of the theoretical expression with real channel-measurement data, which confirms the applicability of the $\mathit{\kappa }$–$\mathit{\mu }$ model in realistic mmWave scenarios. Furthermore, we perform a detailed parametric study to quantify the influence of $\mathit{\kappa }$ and $\mathit{\mu }$ on adaptive modulation performance, providing practical insights for 5G and future 6G systems. The proposed framework bridges theoretical analysis and experimental validation, offering a computationally efficient and robust tool for the design and evaluation of advanced modulation schemes in generalized fading environments.

1. Introduction

1.1. Related Work

The advancement of mobile communication technologies has been driven by the growing demand for faster, more reliable, and lower-latency connections, as well as the need to support an increasing number of connected devices. 5G networks emerge as a response to these challenges, offering substantial improvements over previous generations of mobile networks [1]. The growing demand for high data rates in wireless communication systems makes research in this field essential to ensure the quality of services and applications. A significant portion of these studies focuses on mitigating the negative effects of the wireless channel, with fading being the main phenomenon affecting this type of communication [2].

To address the reduction in spectral efficiency, it is common to increase the order of the modulation. Thus, QAM (quadrature amplitude modulation) schemes are often employed in various studies in the literature [3,4]. Currently, non-square constellations are widely used in communication systems because they exhibit lower amplitude fluctuations than square constellations [5,6]. Additionally, the performance of QAM in photonics-aided mmWave systems with multi-antenna diversity has been investigated in [7], reinforcing its practical relevance in next-generation architectures.

The advanced applications envisioned for wireless networks, especially in the mmWave band for 5G and future 6G systems, will inevitably lead to a wide range of propagation conditions. Although traditional fading models such as Rayleigh [8], Hoyt [9], Rice [10], Nakagami-m [11], and Weibull [12] may still be applicable, their ability to adapt to more complex scenarios is limited. However, more flexible and generalized models, such as the $\alpha$–$\mathit{\mu }$ [13], $\mathit{\kappa }$–$\mathit{\mu }$, and $\eta$–$\mathit{\mu }$ [14] models, offer greater potential to represent these challenging conditions. Recent studies also highlight the relevance of generalized models such as $\mathit{\kappa }$–$\mathit{\mu }$ and Beaulieu–Xie for analyzing realistic fading and interference environments [14,15].

These generalized models have proven particularly effective in representing propagation characteristics in real-world applications, including spectrum sensing [16], vehicular-to-vehicular (V2V) scenarios [17], device-to-device (D2D) communication systems [18], body area networks (BAN) [19], underwater ultrasonic links [20], 5G Frequency Range 2 (FR2) deployments [21], narrowband transmission systems [22], and mmWave channels [23,24,25,26]. Recent experimental investigations, such as the one conducted in [27], confirmed the flexibility of the $\alpha$–$\mathit{\mu }$ model in describing composite fading more accurately than classical models, further reinforcing the usefulness of generalized approaches. More recent studies have also validated the suitability of generalized fading models for next-generation scenarios, including $\mathit{\kappa }$–$\mathit{\mu }$ shadowed fading with hardware impairments [28], fading channel models for outdoor mmWave communications [29], and symbol error probability analysis under $\alpha$–$\mathit{\kappa }$–$\mathit{\mu }$ fading [30]. In particular, recent studies have also further highlighted the relevance of the $\mathit{\kappa }$–$\mathit{\mu }$ model in next-generation systems, ranging from the characterization of novel modulation techniques [31], to secure transmission strategies in uplink Sparse Code Multiple Access (SCMA) [32], and even mixed fading products with practical and RIS-enabled applications [33].

The literature provides strong evidence that these generalized models offer enhanced adaptability to various wireless contexts [21,23,24]. In contrast, conventional models often fall short in representing particular propagation conditions, notably in mmWave scenarios [23,24,25,26], where the envelope probability function density (PDF) of the received signal may exhibit bimodal characteristics. These effects, common in severe fading conditions, underscore the necessity of employing generalized models to capture the full range of physical phenomena observed in modern wireless.

Building on this line of research, the present study turns to the generalized $\mathit{\kappa }$–$\mathit{\mu }$ fading model [14], which is widely recognized for its flexibility in characterizing a variety of multipath propagation conditions, including both LoS (line-of-sight) and nLoS (non-line-of-sight) scenarios. This model captures small-scale fading effects by accounting for the presence of multiple dominant components and clustered scatterers, offering enhanced analytical tractability and encompassing several classical models as special cases. A closed-form approximation is derived for the SEP (symbol error probability) of cross-QAM schemes over the $\mathit{\kappa }$–$\mathit{\mu }$ model. The analytical result is validated through performance simulations conducted in MATLAB R2024b. These simulations were carried out at mmWave frequencies of 55, 60, and 65 GHz. The evaluation is based on channel models developed from an extensive measurement campaign reported in [23], which considered both LoS and nLoS conditions in indoor scenarios. The results provide valuable insights into the behavior of QAM schemes in mmWave environments, clarifying their practical applicability and the challenges they may face in future high-frequency communication systems.

We align our discussion with the 3GPP TR 38.901 channel model for $0.5–100\mathrm{GHz}$, which serves as the industry reference for 5G evaluations. Although TR 38.901 does not prescribe the $\mathit{\kappa }$–$\mathit{\mu }$ distribution per se, $\mathit{\kappa }$–$\mathit{\mu }$ generalizes key fading conditions underlying the standard models (including Rician, Rayleigh, and Nakagami-m), and numerous mmWave measurement campaigns in $28–65\mathrm{GHz}$ have reported excellent goodness-of-fit for $\mathit{\kappa }$–$\mathit{\mu }$ and related generalized distributions [21,23,24]. We therefore adopt $\mathit{\kappa }$–$\mathit{\mu }$ as an analytically tractable, generalized, and measurement-supported model consistent with 5G propagation.

1.2. Contributions and Organization

This work presents an adaptation of the analytical framework to the $\mathit{\kappa }$–$\mathit{\mu }$ fading model, which, unlike the commonly considered $\eta$–$\mathit{\mu }$ model in the literature, captures multiple dominant components and clustered scatterers typical of mmWave propagation. This adaptation required reformulations in the expansions of the Bessel function and approximations to the complementary error function, due to the coupling between the parameters $\mathit{\kappa }$ and $\mathit{\mu }$, preserving the convergence and accuracy of the expressions.

Furthermore, we propose a closed-form approximation to the SEP of cross-QAM modulation schemes over $\mathit{\kappa }$–$\mathit{\mu }$ fading channels, particularly for the mmWave band. This expression avoids the need for numerical integration, which provides computational efficiency.

The work also provides experimental validation using measurement data from campaigns at 55, 60, and 65 GHz, covering scenarios with LoS and nLoS conditions, as well as various polarization combinations, which have not been addressed in previous studies.

Finally, we offer a detailed parametric study on the influence of the parameters $\mathit{\kappa }$ and $\mathit{\mu }$ on modulation performance, providing practical insights for the design of robust systems in generalized fading environments.

The remainder of this paper is organized as follows. Section 2 reviews the cross-QAM scheme over $\mathit{\kappa }$–$\mathit{\mu }$ fading. In Section 3, a closed-form approximation to the SEP of cross-QAM over the $\mathit{\kappa }$–$\mathit{\mu }$ fading model is derived, representing a key contribution of this work. Section 4 analyzes the experimental data from [23] to validate the applicability of the model to mmWave environments, including statistical fitting with the $\mathit{\kappa }$–$\mathit{\mu }$ distribution. Section 5 presents a performance evaluation based on MATLAB simulations at 55, 60, and 65 GHz, considering LoS/nLoS scenarios and different polarization configurations. Finally, Section 6 summarizes the main findings and conclusions.

2. Basic Review

2.1. Cross M-QAM Modulation

QAM is widely employed in digital communication systems due to its ability to efficiently transmit information through simultaneous modulation of carrier amplitude and phase. In this scheme, the constellation comprises $M=2^b$ symbols, with b denoting the number of bits carried by each symbol.

For odd values of b satisfying $b\ge 5$, a variation known as the cross M-QAM constellation is often adopted. Unlike conventional square-grid layouts, this structure arranges the symbols in a cross-shaped geometry to approximate a more compact constellation. This approach reduces the average symbol energy while supporting Gray-like labeling strategies. The resulting constellation points are mathematically described as

$$s=\pm \left(2u-1\right)\pm \left(2v-1\right)j,$$

(1)

where the indices u and v follow these rules:

$$\begin{array}{cc}\hfill u& =1,\dots ,3·2^{(b-5)/2},\hfill \\ \hfill v& =\left\{\begin{array}{c}1,\dots ,3·2^{(b-5)/2};\mathrm{if}u⩽2^{(b-3)/2}\hfill \\ 1,\dots ,2^{(b-3)/2};\mathrm{if}u>2^{(b-3)/2}\hfill \end{array}\right.,\hfill \end{array}$$

and j is the imaginary unit.

To normalize the constellation energy, a scale factor k is introduced, given by

$$k=\frac{1}{\sqrt{E_{\mathbf{M}}}},$$

(2)

where $E_{\mathbf{M}}$ denotes the average energy per symbol in the cross-QAM constellation. According to [34], this average energy is expressed as

$$E_{\mathbf{M}}=\frac{31M-32}{48}.$$

(3)

The constellation points are classified based on the number of adjacent symbols they have in the horizontal and vertical directions:

(a)

8 symbols have two neighbors;

(b)

$3\sqrt{2M}-16$ symbols have three neighbors;

(c)

$M-3\sqrt{2M}+8$ symbols have four neighbors.

An illustration of this structure for $M=128$ is shown in Figure 1, where the different categories of points are highlighted.

Assuming an additive white Gaussian noise (AWGN) channel, the SEP for cross-QAM modulation is analytically given by [34]

$$\begin{array}{c}\hfill \begin{array}{c}{P}_e(\mathit{\gamma })=\left(2-\frac{3}{2}\sqrt{\frac{2}{M}}\right)erfc\left(k\sqrt{\mathit{\gamma }}\right)-\left(1+\frac{2}{M}-\frac{3}{2}\sqrt{\frac{2}{M}}\right)\stackrel{2}{erfc}\left(k\sqrt{\mathit{\gamma }}\right),\hfill \end{array}\end{array}$$

(4)

where $\mathit{\gamma }=\frac{E_{\mathbf{M}}}{N_0}={log}_2\left(M\right)\frac{E_b}{N_0}$ represents the instantaneous signal-to-noise ratio (SNR) at the receiver, with $E_b$ being the average bit energy and $N_0$ the one-sided noise power spectral density in [watts/Hz].

This modulation format provides improved spectral efficiency for high data rate transmissions by enabling non-square constellations, which are particularly advantageous when M is not a power of four. Although square constellations are generally easier to implement, cross-QAM offers a compelling trade-off between the average energy per symbol and the constellation compactness [2]. It is therefore a practical choice for adaptive modulation schemes in modern wireless communication standards.

2.2. Fading Channels

In a fading propagation scenario, the instantaneous value of $\mathit{\gamma }$ will vary randomly, and its PDF will depend on the considered type of fading. Thus, the average SEP can be calculated as [35]

$$\overline{P_e}={\mathit{\int }}_0^{\infty }{\mathcal{P}}_e\left(\mathit{\gamma }\right)p\left(\mathit{\gamma }\right)d\mathit{\gamma },$$

(5)

where ${\mathcal{P}}_e\left(\mathit{\gamma }\right)$ is the symbol error probability of a QAM scheme conditioned on a fixed $\mathit{\gamma }$ and $p\left(\mathit{\gamma }\right)$ is the PDF of $\mathit{\gamma }$.

2.3. The $\mathit{\kappa }$–$\mathit{\mu }$ Fading Model

The $\mathit{\kappa }$–$\mathit{\mu }$ distribution is a flexible and generalized small-scale fading model capable of characterizing both LoS and nLoS propagation conditions. It has been extensively employed to describe wireless channels where the received signal is composed of multiple clusters of multipath components, each potentially containing a dominant wave.

According to [14], the PDF of the instantaneous SNR $\mathit{\gamma }$ over $\mathit{\kappa }$–$\mathit{\mu }$ fading is expressed as

$$\begin{array}{c}\hfill \begin{array}{c}p(\mathit{\gamma })=\frac{\mathit{\mu }{(1+\mathit{\kappa })}^{(\mathit{\mu }+1)/2}{\mathit{\gamma }}^{(\mathit{\mu }-1)/2}}{{\overline{\mathit{\gamma }}}^{(\mathit{\mu }+1)/2}{\mathit{\kappa }}^{(\mathit{\mu }-1)/2}{e}^{\mathit{\mu }\mathit{\kappa }}}·exp\left(-{\displaystyle \frac{\mathit{\mu }(1+\mathit{\kappa })\mathit{\gamma }}{\overline{\mathit{\gamma }}}}\right)I_{\mathit{\mu }-1}\left(2\mathit{\mu }\sqrt{{\displaystyle \frac{\mathit{\kappa }(1+\mathit{\kappa })\mathit{\gamma }}{\overline{\mathit{\gamma }}}}}\right),\hfill \end{array}\end{array}$$

(6)

where $\mathit{\mu }>0$ corresponds to the number of multipath clusters (or the clustering level of the scattered components), $\mathit{\kappa }\ge 0$ represents the ratio of the total power of the dominant components to that of the scattered waves, $\overline{\mathit{\gamma }}$ denotes the average received SNR, and $I_{\mathit{\mu }-1}(·)$ is the modified Bessel function of the first kind and order $\mathit{\mu }-1$.

The $\mathit{\kappa }$–$\mathit{\mu }$ model encompasses several classical distributions as special cases [14], including Rice $(\mathit{\kappa }=K,\mathit{\mu }=1)$, Nakagami-m$(\mathit{\kappa }\to 0,\mathit{\mu }=m)$, Rayleigh ($\mathit{\kappa }$$\to 0$, $\mathit{\mu }=1)$, and one-sided Gaussian ($\mathit{\kappa }$→ 0, $\mathit{\mu }={\displaystyle \frac{1}{2}})$. Further analytical treatments of QAM over mixed $\mathit{\kappa }$–$\mathit{\mu }$ and $\eta$–$\mathit{\mu }$ fading channels can be found in [36], highlighting the importance of generalized models in performance evaluation.

To facilitate analytical tractability, particularly in integrals involving the PDF, an infinite series representation of the Bessel function can be employed. A truncated form of the infinite series of the Bessel function, as presented in [37] (p. 375, Eq. 9.6.10), for $z>0$ is given by

$$I_{\mathit{\mu }-1}\left(z\right)\approx {\left(\frac{z}{2}\right)}^{\mathit{\mu }-1}{\displaystyle \sum _{j=0}^n}\frac{{\left(\frac{z^2}{4}\right)}^j}{j!\Gamma \left(\mathit{\mu }+j\right)},$$

(7)

where $\Gamma (·)$ denotes the Gamma function.

Substituting (7) into (6) with $z=2\mathit{\mu }\sqrt{\frac{\mathit{\kappa }(1+\mathit{\kappa })\mathit{\gamma }}{\overline{\mathit{\gamma }}}}$, we obtain the following approximation to the PDF of $\mathit{\gamma }$:

$$\begin{array}{c}\hfill \begin{array}{c}p(\mathit{\gamma })\approx \frac{e^{-\mathit{\mu }\mathit{\kappa }}{\mathit{\mu }}^{\mathit{\mu }}{(1+\mathit{\kappa })}^{\mathit{\mu }}}{{\overline{\mathit{\gamma }}}^{\mathit{\mu }}}{\mathit{\gamma }}^{\mathit{\mu }-1}·exp\left(-{\displaystyle \frac{\mathit{\mu }(1+\mathit{\kappa })\mathit{\gamma }}{\overline{\mathit{\gamma }}}}\right){\displaystyle \sum _{j=0}^n}{\displaystyle \frac{{\mathit{\mu }}^{2j}{\mathit{\kappa }}^j{\left(1+\mathit{\kappa }\right)}^j{\mathit{\gamma }}^j}{j!\Gamma \left(\mathit{\mu }+j\right){\overline{\mathit{\gamma }}}^j}}.\hfill \end{array}\end{array}$$

(8)

3. Closed-Form Approximation to the SEP of Cross-QAM over $\mathbf{\kappa }$–$\mathbf{\mu }$ Model

Applying $P_e(\mathit{\gamma })$ for the cross M-QAM constellation given in (4), and $p\left(\mathit{\gamma }\right)$ for the $\mathit{\kappa }$–$\mathit{\mu }$ fading model presented in (8), to $\overline{P_e}$ given in (5), we obtain

$$\overline{P_e}\left(\overline{\mathit{\gamma }}\right)\approx \frac{e^{-\mathit{\mu }\mathit{\kappa }}{\mathit{\mu }}^{\mathit{\mu }}{(1+\mathit{\kappa })}^{\mathit{\mu }}}{{\overline{\mathit{\gamma }}}^{\mathit{\mu }}}{\displaystyle \sum _{j=0}^n}\left(\frac{{\mathit{\mu }}^{2k}{\mathit{\kappa }}^j{\left(1+\mathit{\kappa }\right)}^j{\mathit{\gamma }}^j}{j!\Gamma \left(\mathit{\mu }+j\right){\overline{\mathit{\gamma }}}^j}\left(AX-BY\right)\right),$$

(9)

where

$$A=2-\frac{3}{2}\sqrt{\frac{2}{M}},$$

(10)

$$B=1+\frac{2}{M}-\frac{3}{2}\sqrt{\frac{2}{M}},$$

(11)

and X and Y are respectively defined as

$$X={\mathit{\int }}_0^{\infty }erfc\left(k\sqrt{\mathit{\gamma }}\right){\mathit{\gamma }}^{j+\mathit{\mu }-1}exp\left(-\frac{\mathit{\mu }(1+\mathit{\kappa })\mathit{\gamma }}{\overline{\mathit{\gamma }}}\right)d\mathit{\gamma },$$

(12)

$$Y={\mathit{\int }}_0^{\infty }{erfc}^2\left(k\sqrt{\mathit{\gamma }}\right){\mathit{\gamma }}^{j+\mathit{\mu }-1}exp\left(-\frac{\mathit{\mu }(1+\mathit{\kappa })\mathit{\gamma }}{\overline{\mathit{\gamma }}}\right)d\mathit{\gamma },$$

(13)

where k is the scaling factor defined in (2).

Initially, the term X is determined by applying the substitution from Equation (14), which involves a change of variables:

$$\mathit{\gamma }=\frac{x^2}{k^2}\Rightarrow d\mathit{\gamma }=\frac{2x}{k^2}dx.$$

(14)

This substitution changes the integration limits from $\mathit{\gamma }=0$ (corresponding to $x=0$) to $\mathit{\gamma }=\infty$ (corresponding to $x=\infty$). Thus, we can then rewrite the term X as

$$X=\frac{2}{k^{2\mathit{\mu }+2j}}{\mathit{\int }}_0^{\infty }erfc\left(x\right)x^{2\mathit{\mu }+2j-1}exp\left(-\frac{\mathit{\mu }(1+\mathit{\kappa })}{k^2\overline{\mathit{\gamma }}}{x}^2\right)dx.$$

(15)

As stated by [38], the complementary error function is approximated to

$$erfc\left(x\right)\approx \frac{exp(-x^2)}{6}+\frac{exp\left(-\frac{4}{3}{x}^2\right)}{2}.$$

(16)

Hence,

$$\begin{array}{c}\hfill \begin{array}{c}{\displaystyle X\approx \frac{1}{3k^{2\mathit{\mu }+2j}}{\mathit{\int }}_0^{\infty }{x}^{2\mathit{\mu }+2j-1}exp\left(-\frac{\mathit{\mu }(1+\mathit{\kappa })+k^2\overline{\mathit{\gamma }}}{k^2\overline{\mathit{\gamma }}}{x}^2\right)dx}\hfill \\ {\displaystyle +\frac{1}{k^{2\mathit{\mu }+2j}}{\mathit{\int }}_0^{\infty }{x}^{2\mathit{\mu }+2j-1}exp\left(-\frac{3\mathit{\mu }(1+\mathit{\kappa })+4k^2\overline{\mathit{\gamma }}}{3k^2\overline{\mathit{\gamma }}}{x}^2\right)dx.}\hfill \end{array}\end{array}$$

(17)

We use the following definite integral from the integral table in [39]:

$${\mathit{\int }}_0^{\infty }{x}^{2m}exp(-\beta x^{2n})dx=\frac{\Gamma \left(v\right)}{2n{\beta }^v},$$

(18)

where $v=\frac{2m+1}{2n}$.

By comparing Equations (17) and (18), we obtain: $n=1$, $v=\frac{\left(2\mathit{\mu }+2j-1\right)+1}{2}=\mathit{\mu }+j$, ${\beta }_1=-\frac{\mathit{\mu }(1+\mathit{\kappa })+k^2\overline{\mathit{\gamma }}}{k^2\overline{\mathit{\gamma }}}$ and ${\beta }_2=-\frac{3\mathit{\mu }(1+\mathit{\kappa })+4k^2\overline{\mathit{\gamma }}}{3k^2\overline{\mathit{\gamma }}}$. Therefore, Equation (15) results in

$$\begin{array}{c}\hfill \begin{array}{c}X\approx {\displaystyle \frac{\Gamma \left(\mathit{\mu }+j\right){\overline{\mathit{\gamma }}}^{\mathit{\mu }+j}}{2}}·\left[\frac{1}{3}{\left(\frac{1}{\mathit{\mu }(1+\mathit{\kappa })+k^2\overline{\mathit{\gamma }}}\right)}^{\mathit{\mu }+j}+{\left(\frac{3}{(3\mathit{\mu }(1+\mathit{\kappa })+4k^2\overline{\mathit{\gamma }}}\right)}^{\mathit{\mu }+j}\right].\hfill \end{array}\end{array}$$

(19)

The term Y can be found considering the same variable substitution presented in (14) and can be rewritten as

$$Y=\frac{2}{k^{2\mathit{\mu }+2j}}{\mathit{\int }}_0^{\infty }{erfc}^2\left(x\right)x^{2\mathit{\mu }+2j-1}exp\left(-\frac{\mathit{\mu }(1+\mathit{\kappa })}{k^2\overline{\mathit{\gamma }}}{x}^2\right)dx.$$

(20)

We expand the approximation of [38] given in (16),

$${erfc}^2\left(x\right)\approx \frac{exp(-2x^2)}{36}+\frac{exp\left(-\frac{7}{3}{x}^2\right)}{6}+\frac{exp\left(-\frac{8}{3}{x}^2\right)}{4},$$

(21)

and we apply the approximation to the Equation (20) to obtain

$$\begin{array}{cc}\hfill Y& \approx \frac{1}{18k^{2\mathit{\mu }+2j}}{\mathit{\int }}_0^{\infty }{x}^{2\mathit{\mu }+2j-1}exp\left(-\frac{\mathit{\mu }(1+\mathit{\kappa })+2k^2\overline{\mathit{\gamma }}}{k^2\overline{\mathit{\gamma }}}{x}^2\right)dx\hfill \\ \hfill & +\frac{1}{3k^{2\mathit{\mu }+2j}}{\mathit{\int }}_0^{\infty }{x}^{2\mathit{\mu }+2j-1}exp\left(-\frac{3\mathit{\mu }(1+\mathit{\kappa })+7k^2\overline{\mathit{\gamma }}}{3k^2\overline{\mathit{\gamma }}}{x}^2\right)dx\hfill \\ \hfill & +\frac{1}{2k^{2\mathit{\mu }+2j}}{\mathit{\int }}_0^{\infty }{x}^{2\mathit{\mu }+2j-1}exp\left(-\frac{3\mathit{\mu }(1+\mathit{\kappa })+8k^2\overline{\mathit{\gamma }}}{3k^2\overline{\mathit{\gamma }}}{x}^2\right)dx.\hfill \end{array}$$

(22)

Using the definite integral in Equation (18) again, Equation (20) results in the following:

$$\begin{array}{c}\hfill \begin{array}{c}Y\approx \frac{\Gamma \left(\mathit{\mu }+j\right){\overline{\mathit{\gamma }}}^{\mathit{\mu }+j}}{2}\left[\frac{1}{18}{\left(\frac{1}{\mathit{\mu }(1+\mathit{\kappa })+2k^2\overline{\mathit{\gamma }}}\right)}^{\mathit{\mu }+j}\right]\hfill \\ +\frac{\Gamma \left(\mathit{\mu }+j\right){\overline{\mathit{\gamma }}}^{\mathit{\mu }+j}}{2}\left[\frac{1}{3}{\left(\frac{3}{3\mathit{\mu }(1+\mathit{\kappa })+7k^2\overline{\mathit{\gamma }}}\right)}^{\mathit{\mu }+j}\right]\hfill \\ +\frac{\Gamma \left(\mathit{\mu }+j\right){\overline{\mathit{\gamma }}}^{\mathit{\mu }+j}}{2}\left[\frac{1}{2}{\left(\frac{3}{3\mathit{\mu }(1+\mathit{\kappa })+8k^2\overline{\mathit{\gamma }}}\right)}^{\mathit{\mu }+j}\right].\hfill \end{array}\end{array}$$

(23)

Applying Equations (23) and (19) to (9), we obtain the closed-form approximation to the average SEP of the cross M-QAM scheme over the $\mathit{\kappa }$–$\mathit{\mu }$ fading channel as follows:

$$\begin{array}{c}\hfill \overline{P_e}\left(\overline{\mathit{\gamma }}\right)\approx \frac{e^{-\mathit{\mu }\mathit{\kappa }}{\mathit{\mu }}^{\mathit{\mu }}{(1+\mathit{\kappa })}^{\mathit{\mu }}}{2}{\displaystyle \sum _{j=0}^n}\{\frac{{\mathit{\kappa }}^j{\mathit{\mu }}^{2j}{(1+\mathit{\kappa })}^j}{j!}\left[\left(2-\frac{3}{2}\sqrt{\frac{2}{M}}\right)\left(\frac{1}{3}{\left(\frac{1}{\mathit{\mu }(1+\mathit{\kappa })+k^2\overline{\mathit{\gamma }}}\right)}^{\mathit{\mu }+j}+{\left(\frac{3}{3\mathit{\mu }(\mathit{\mu }+\mathit{\kappa })+4k^2\overline{\mathit{\gamma }}}\right)}^{\mathit{\mu }+j}\right)\right.\\ \hfill \left.-\left(1+\frac{2}{M}-\frac{3}{2}\sqrt{\frac{2}{M}}\right)\left(\frac{1}{18}{\left(\frac{1}{\mathit{\mu }(1+\mathit{\kappa })+2k^2\overline{\mathit{\gamma }}}\right)}^{\mathit{\mu }+j}+\frac{1}{3}{\left(\frac{3}{3\mathit{\mu }(1+\mathit{\kappa })+7k^2\overline{\mathit{\gamma }}}\right)}^{\mathit{\mu }+j}+\frac{1}{2}{\left(\frac{3}{3\mathit{\mu }(1+\mathit{\kappa })+8k^2\overline{\mathit{\gamma }}}\right)}^{\mathit{\mu }+j}\right)\right]\}.\end{array}$$

(24)

4. Field Measurements and Channel Modeling

In this work, the channel model described in [23] was adopted to evaluate the performance of cross-QAM schemes in the mmWave band. The measurements were carried out at the iTEAM Research Institute, Universitat Politècnica de València, Spain, in a laboratory environment characterized by a high density of reflective and scattering surfaces, including metallic cupboards, benches, computers, and other furniture. This rich multipath environment results from propagation mechanisms such as reflection, diffraction, and scattering. The measurement area covers a volume of approximately 13.5 m × 7 m × 2.6 m, enclosed by reinforced concrete floors and ceilings and walls made of wood and plasterboard.

The measurement campaign, as detailed in [23], aimed to characterize short-term fading in the 55–65 GHz frequency range. The setup included a vector network analyzer (VNA), vertically polarized antennas, and a receiver-side XY positioning system forming a uniform rectangular array (URA). Among the various configurations studied, Scenarios 1, 2, 3, and 4 were selected for analysis in this work due to their representativeness of different propagation conditions.

• Scenario 1: LoS with crossed polarization (HV), distance of 3.29 m between transmitter (Tx) and receiver (Rx). This setting emphasizes diffuse components by attenuating the direct path.
• Scenario 2: LoS with co-polarized antennas (VV) at the same distance of 3.29 m, providing strong antenna coupling and emphasizing the direct path with minimal attenuation.
• Scenario 3: nLoS with co-polarized antennas (VV) at the same distance of 3.29 m, with an absorber obstructing the LoS to emulate common indoor blockage.
• Scenario 4: Favorable LoS with VV polarization and reduced separation of 2.77 m, resulting in a strong dominant component.

These scenarios encompass a range of propagation conditions, from strong direct paths to obstructed environments, providing a comprehensive basis for evaluating signal fading effects in high-frequency communication systems.

Table 1. Fitting results obtained for Scenarios 1, 2, 3, and 4 considering the $\mathit{\kappa }$–$\mathit{\mu }$ distribution, as described in [23].

Scenario	Frequency	$\mathit{\kappa }$	$\mathit{\mu }$	NMSE [dB]
1–LoS–HV–3.29 m	55 GHz	$0.43$	$0.99$	$-17.57$
	60 GHz	$0.84$	$0.85$	$-16.38$
	65 GHz	$0.77$	$0.98$	$-17.35$
2–LoS–VV–3.29 m	55 GHz	$2.92$	$0.96$	$-14.97$
	60 GHz	$24.47$	$0.19$	$-16.73$
	65 GHz	$2.66$	$0.83$	$-14.94$
3–nLoS–VV–3.29 m	55 GHz	$2.04$	$0.81$	$-17.31$
	60 GHz	$1.19$	$1.57$	$-16.19$
	65 GHz	$1.05$	$0.84$	$-16.47$
4–LoS–VV–2.77 m	55 GHz	$24.43$	$0.25$	$-21.16$
	60 GHz	$1.80$	$2.06$	$-19.62$
	65 GHz	$3.53$	$1.32$	$-17.89$

, adapted from [23], summarizes the estimated parameters of the $\mathit{\kappa }$–$\mathit{\mu }$ fading model for Scenarios 1, 2, 3, and 4, at operating frequencies of 55, 60, and 65 GHz. The table includes (i) the $\mathit{\kappa }$ parameter; (ii) the $\mathit{\mu }$ parameter; and (iii) the normalized mean square error (NMSE) (the NMSE is computed as $10{log}_{10}\left(\frac{{\sum }_i{(p_{\mathrm{meas}}\left({\mathit{\gamma }}_i\right)-p_{\mathrm{model}}\left({\mathit{\gamma }}_i\right))}^2}{{\sum }_i{p}_{\mathrm{meas}}^2\left({\mathit{\gamma }}_i\right)}\right)$, where $p_{\mathrm{meas}}$ and $p_{\mathrm{model}}$ denote the measured and fitted PDF, respectively. More negative values indicate a better goodness of fit), which reflects the accuracy of the model fitting to the empirical data in each evaluated scenario. All values presented in Table 1 were directly reproduced from [23], which reported the $\mathit{\kappa }$–$\mathit{\mu }$ model fitting to the measurement data.

The results in Table 1 highlight the distinct small-scale fading characteristics across the four analyzed scenarios, as captured by the estimated parameters $\mathit{\kappa }$ and $\mathit{\mu }$ of the $\mathit{\kappa }$–$\mathit{\mu }$ distribution.

In Scenario 1 (LoS, HV polarization, 3.29 m), the estimated values of $\mathit{\kappa }$ are relatively low (between 0.43 and 0.84), indicating a weak dominance of the LoS component. This is expected due to the cross-polarized configuration, which reduces antenna coupling and attenuates the direct path. The $\mathit{\mu }$ values remain close to 1 across all frequencies, suggesting a mildly dispersive environment with a limited number of significant multipath clusters. The NMSE values, ranging from $-17.57$ dB to $-16.38$ dB, reflect a good fit between the model and the measured data.

In Scenario 2 (LoS, VV polarization, 3.29 m), the values of $\mathit{\kappa }$ vary significantly, from 2.66 to 24.47, with the peak occurring at 60 GHz. This strong direct component results from the co-polarized antenna configuration, which maximizes LoS reception. Notably, the extremely high $\mathit{\kappa }=24.47$ paired with a very low $\mathit{\mu }=0.19$ at 60 GHz points to a channel dominated almost entirely by a deterministic LoS component, with minimal contribution from diffuse multipath. This configuration is physically consistent with highly direct propagation and low scattering. The NMSE values, between $-16.73$ dB and $-14.94$ dB, indicate a slightly more complex fitting process than in Scenario 1, possibly due to the sparsity of the scattered energy in some frequency bands.

Scenario 3 (nLoS, VV polarization, 3.29 m) presents $\mathit{\kappa }$ values between 1.05 and 2.04, consistent with the absence of a direct path but the presence of strong reflected components acting as dominant signal contributors. This is typical in indoor nLoS conditions, where reflections from walls or furniture can mimic the role of an LoS path. The values of $\mathit{\mu }$ range from 0.81 to 1.57, with the highest multipath richness occurring at 60 GHz. The NMSE performance ($-17.31$ dB to $-16.19$ dB) remains comparable to that of the LoS cases, suggesting that the $\mathit{\kappa }$–$\mathit{\mu }$ model adapts well even under obstruction.

Finally, Scenario 4 (LoS, VV polarization, 2.77 m) stands out with the highest average value of $\mathit{\kappa }$ observed in all scenarios, reaching $24.43$ at 55 GHz, alongside an extremely low $\mathit{\mu }=0.25$. This reflects a propagation condition with a strongly dominant LoS component and a highly sparse or unstable diffuse component, typical of short-range, highly directive links. At 60 GHz, $\mathit{\kappa }$ drops sharply to $1.80$ and $\mathit{\mu }$ rises to $2.06$, suggesting an increase in scattering due to greater interaction with surrounding surfaces. However, at 65 GHz, $\mathit{\kappa }$ increases again to $3.53$ while $\mathit{\mu }$ decreases to $1.32$, indicating a slight rebalancing between the dominant and scattered components. Scenario 4 also yields the lowest NMSE values among all scenarios (from $-21.16$ dB to $-17.89$ dB), demonstrating the excellent fit of the $\mathit{\kappa }$–$\mathit{\mu }$ model in this near-optimum LoS propagation environment.

In summary, the estimated parameters $\mathit{\kappa }$ and $\mathit{\mu }$ are physically coherent with the polarization and geometric configurations of each scenario. The $\mathit{\kappa }$–$\mathit{\mu }$ distribution shows strong modeling capability across both highly directive LoS scenarios and more scattered nLoS conditions, as evidenced by the consistently low NMSE values.

It is worth noting that in some LoS scenarios, the estimated values of $\mathit{\mu }$ are less than one. This condition indicates a propagation environment with a very limited number of multipath clusters, often associated with highly directive LoS links in which the received signal is dominated by one or a few strong components and only sparse diffuse scattering. Physically, $\mathit{\mu }<1$ can represent even more severe fading than the Rayleigh case, since Rayleigh assumes $\mathit{\mu }=1$ with fully scattered components of equal power. Such conditions have also been reported in other mmWave experimental works [23,24], confirming that $\mathit{\mu }<1$ is a meaningful and realistic outcome in these practical scenarios.

5. Performance Evaluation

In order to analyze the performance of the cross M-QAM scheme in an indoor mmWave environment, a simulated communication system based on this modulation scheme was developed in MATLAB. The system transmitter generates N random symbols from the M-QAM constellation. The propagation channel between the transmitter (Tx) and the receiver (Rx) is modeled as a slow $\mathit{\kappa }$–$\mathit{\mu }$ fading channel, assuming that its characteristics remain practically constant during the transmission of each symbol. Thus, the channel introduces a multiplicative gain to the envelope of the received symbols, generated in MATLAB using the Inverse Transform Sampling method [40].

A vector of circularly symmetric complex Gaussian noise with zero mean and variance $N_0/2$, determined by the configured signal-to-noise ratio, is added to the symbol vector already affected by the channel. The receiver uses a zero-forcing equalizer at the input, followed by a maximum likelihood detector. The SEP is estimated by computing the ratio of incorrectly detected symbols to the total number of transmitted symbols. The performance curves presented in this section were obtained for values of $E_b/N_0$ ranging from 0 dB to 40 dB, with $N={10}^8$ transmitted symbols for each point.

The developed system was initially simulated for $M=32$, 128, 512, considering an $\mathit{\kappa }$–$\mathit{\mu }$ channel with the values of $\mathit{\kappa }$ and $\mathit{\mu }$ presented in Table 1 for Scenarios 1 (LoS, HV) and 2 (LoS, VV). The estimated $\overline{P_e}$ values (markers) at 55, 60, and 65 GHz are shown in Figure 2, Figure 3 and Figure 4 for Scenarios 1 and 2. The markers represent the results obtained through Monte Carlo simulations, while the solid and dashed lines correspond to the theoretical symbol error rate (SER) curves derived from the closed-form approximation in (24).

Analyzing the results presented in Figure 2, Figure 3 and Figure 4, it can be observed that the theoretical and simulated curves exhibit strong agreement in all simulated cases, validating the proposed approximation (24). This strong agreement has been achieved for relatively low values of n in (24) ($n=15$), due to the rapid convergence of the series whose truncated series is presented in (7).

At 55 GHz (Figure 2), Scenario 2 consistently achieves superior SEP performance across all $E_b/N_0$ values and modulation orders. This improvement results from the strong antenna coupling provided by co-polarization (VV), which maximizes the reception of the LoS component and minimizes signal degradation. In contrast, in Scenario 1, despite the presence of a physical LoS, the cross-polarization (HV) configuration causes significant attenuation of the direct path, leading to worse performance. The estimated channel parameters reinforce this interpretation: Scenario 1 presents $\mathit{\kappa }=0.43$ and $\mathit{\mu }=0.99$, indicating a scenario with limited dominance of the LoS path and moderate scattering. In contrast, Scenario 2 shows $\mathit{\kappa }=2.92$ and $\mathit{\mu }=0.96$, pointing to a strong direct component with a similar level of dispersion.

When the frequency increases to 60 GHz (Figure 3), the performance gap between the two scenarios becomes more evident. Scenario 2 maintains its clear advantage in all conditions analyzed. The case of a high value $\mathit{\kappa }=24.47$ in this scenario indicates a highly dominant direct path, while the low value $\mathit{\mu }=0.19$ reflects a very limited multipath richness. At the same frequency, in Scenario 1, the estimated parameters are $\mathit{\kappa }=0.84$ and $\mathit{\mu }=0.85$, suggesting a more balanced distribution between the direct and scattered components, yet it is insufficient to overcome the polarization losses. As a result, Scenario 2 offers a more favorable propagation environment for the operation of the system.

Lastly, at 65 GHz (Figure 4), Scenario 2 continues to outperform Scenario 1 regardless of the modulation order or $E_b/N_0$ level. The channel parameters for Scenario 2 ($\mathit{\kappa }=2.66$, $\mathit{\mu }=0.83$) still indicate a strong LoS component, maintaining a similar behavior as in lower frequencies. On the other hand, Scenario 1 presents $\mathit{\kappa }=0.77$ and $\mathit{\mu }=0.98$, suggesting a less prominent direct component and a channel with moderate multipath dispersion, resulting in reduced performance.

Figure 5, Figure 6 and Figure 7 illustrate the system SEP performance for the values $M=32$, 128, and 512, under a $\mathit{\kappa }$–$\mathit{\mu }$ fading channel, with the values of $\mathit{\kappa }$ and $\mathit{\mu }$ specified in Table 1 for Scenarios 3 (nLoS, VV, 3.29 m) and 4 (LoS, VV, 2.77 m). The estimated $\overline{P_e}$ values (markers) at 55, 60, and 65 GHz are depicted in Figure 5, Figure 6, and Figure 7, respectively, along with the theoretical curves for Scenario 3 (dashed lines) and Scenario 4 (solid lines). In all the cases analyzed, Scenario 4 consistently outperforms Scenario 3 in all modulation orders and SNR ranges, highlighting it as the most favorable propagation condition evaluated.

In Figure 5, which corresponds to the 55 GHz frequency, the performance superiority of Scenario 4 is particularly evident due to the high $\mathit{\kappa }$ value ($24.43$) and low $\mathit{\mu }$ value ($0.25$), indicating a strong dominant component and limited multipath dispersion. The VV polarization, combined with a shorter Tx-Rx separation, further improves antenna coupling and enhances direct signal reception.

In scenarios characterized by high $\mathit{\kappa }$ and low $\mathit{\mu }$ values, such as those observed in Figure 3 (Scenario 2, 60 GHz) and Figure 5 (Scenario 4, 55 GHz), a divergence between the theoretical and simulated curves emerges at high $E_b/N_0$ values—specifically, above approximately 20 dB ($M=32$), 25 dB ($M=128$), and 30 dB ($M=512$) for Scenario 2 and above 25 dB, 30 dB, and 35 dB, respectively, for Scenario 4. This discrepancy may stem from the approximation of the complementary error function and the truncation of an infinite series to a finite number of terms, which can introduce minor inaccuracies that become more prominent in high-SNR regimes. Nonetheless, the overall agreement between the analytical and the simulated results supports the validity and robustness of the proposed theoretical approximation for the practical SNR range.

Figure 6, corresponding to 60 GHz, still demonstrates the superior performance of Scenario 4 over Scenario 3, although the gap between them becomes less pronounced. In this case, Scenario 4 features more balanced values of $\mathit{\kappa }=1.80$ and $\mathit{\mu }=2.06$, indicating a favorable interaction between a dominant LoS component and multipath diversity. Scenario 3, with $\mathit{\kappa }=1.19$ and $\mathit{\mu }=1.57$, also offers reasonable robustness, but with a weaker direct signal contribution.

Finally, in Figure 7 (65 GHz), Scenario 4 continues to outperform Scenario 3 in terms of SEP across all modulation orders and SNR ranges. With $\mathit{\kappa }=3.53$ and $\mathit{\mu }=1.32$, the channel in Scenario 4 maintains a strong LoS component while preserving moderate multipath richness. In contrast, Scenario 3, characterized by lower values of $\mathit{\kappa }=1.05$ and $\mathit{\mu }=0.84$, exhibits more severe fading conditions, which significantly degrade performance, particularly for higher modulation orders.

Overall, the simulation results confirm the accuracy of the proposed closed-form approximation to various modulation orders, polarization configurations, and frequencies in the mmWave band. A slight divergence between theoretical and simulated curves was observed in scenarios characterized by very high $\mathit{\kappa }$ and low $\mathit{\mu }$, particularly in high-SNR regimes. However, for the SNR ranges typically encountered in practical communication systems, the proposed approximation remains remarkably accurate and provides a reliable tool for performance evaluation and system design in realistic fading environments.

6. Conclusions

This work presented a closed-form approximation to the SEP of cross M-QAM schemes over $\mathit{\kappa }$–$\mathit{\mu }$ fading channels. The proposed formulation enables an accurate and efficient performance evaluation without the need for extensive numerical simulations. The expression was validated through Monte Carlo simulations in realistic environments within the mmWave spectrum (55, 60 and 65 GHz), considering both LoS and nLoS conditions, different Tx–Rx distances, modulation orders, and polarization configurations.

The results demonstrated excellent agreement between the analytical and the simulated curves, even for high-order modulations and high-SNR regimes, confirming the robustness of the proposed model. Although minor discrepancies were observed in scenarios with very high $\mathit{\kappa }$ and low $\mathit{\mu }$ values—particularly at extreme SNR levels—the approximation remained highly accurate within the practical operating range of modern wireless systems.

The analysis further revealed that the polarization configuration and the presence of a LoS component play a critical role in determining the SEP performance. Scenarios with VV polarization and strong LoS consistently achieved superior performance at all evaluated SNR levels. However, under lower SNR conditions, the performance gap between LoS and nLoS scenarios tends to narrow. The observed behavior across different SNR thresholds highlights the joint influence of the parameters $\mathit{\kappa }$ and $\mathit{\mu }$ on system performance.

In summary, the derived expression offers a valuable tool for the design and evaluation of high-frequency communication systems, particularly in emerging 5G and 6G networks. Future research may extend this framework to other generalized fading models and explore adaptive modulation and coding strategies based on analytical SEP expressions.