Enabling Intelligent 6G Communications: A Scalable Deep Learning Framework for MIMO Detection

Abstract

Artificial intelligence (AI) has emerged as a transformative technology in the evolution of massive multiple-input multiple-output (ma-MIMO) systems, positioning them as a cornerstone for sixth-generation (6G) wireless networks. Despite their significant potential, ma-MIMO systems face critical challenges at the receiver end, particularly in signal detection under high-dimensional and noisy environments. To address these limitations, this paper proposes MIMONet, a novel deep learning (DL)-based MIMO detection framework built upon a lightweight and optimized feedforward neural network (FFNN) architecture. MIMONet is specifically designed to achieve a balance between detection performance and complexity by optimizing the neural network architecture for MIMO signal detection tasks. Through extensive simulations across multiple MIMO configurations, the proposed MIMONet detector consistently demonstrates superior bit error rate (BER) performance. It achieves a notably lower error rate compared to conventional benchmark detectors, particularly under moderate to high signal-to-noise ratio (SNR) conditions. In addition to its enhanced detection accuracy, MIMONet maintains significantly reduced computational complexity, highlighting its practical feasibility for advanced wireless communication systems. These results validate the effectiveness of the MIMONet detector in optimizing detection accuracy without imposing excessive processing burdens. Moreover, the architectural flexibility and efficiency of MIMONet lay a solid foundation for future extensions toward large-scale ma-MIMO configurations, paving the way for practical implementations in beyond-5G (B5G) and 6G communication infrastructures.

1. Introduction

Artificial intelligence (AI) aided in multiple-input output (MIMO) in becoming a stepping stone toward the development of sixth-generation (6G) wireless communication networks. With a large number of transmitting and receiving antennas, MIMO technology enables the B5G technology to meet user demands of high data rates [1]. However, with multiple transmitting and receiving antennas, signal detection becomes a very complicated task at the receiver end. To overcome detection problems in MIMO technology, several research studies have been carried out based on both conventional and AI-driven MIMO detection techniques [2]. The conventional MIMO detectors, including minimum mean square error (MMSE) and zero-forcing (ZF), have a lower computational complexity but struggle to maintain performance [1]. To overcome performance issues in MMSE and ZF, the maximum-likelihood detector (MLD) offers the best performance; however, it has an extremely high computational complexity [3]. Although linear detection techniques offer reduced computational complexity compared to optimal detection schemes, they still require matrix inversion operations, which become computationally intensive and impractical in systems with a large number of antennas, a hallmark of massive MIMO (ma-MIMO) deployments [2]. To further enhance convergence speed and detection accuracy, more advanced iterative solvers such as the Gauss–Seidel (GS) [4] and Conjugate Gradient (CG) [5] algorithms have been investigated for MIMO signal detection. These methods exploit the structure of the Gram matrix to achieve more efficient iterative updates and are generally better suited for large-scale systems. Despite these improvements, GS and CG methods still fall short of achieving optimal detection performance, particularly in scenarios involving high-order modulation, strong inter–antenna interference, or ill-conditioned channel matrices [6,7]. Thus, while these linear and iterative detection approaches offer promising avenues for complexity reduction, their suboptimal performance in practical MIMO environments underscores the need for alternative strategies, particularly those leveraging data-driven and AI-assisted techniques that can adapt to the dynamic nature of next-generation wireless systems. Now, to find a balance between complexity and detection performance, researchers are trying to integrate AI, more specifically machine learning (ML) and deep learning (DL), into next-generation wireless communication networks [1]. DL is playing a vital part in developing promising solutions for next-generation wireless network problems, including signal detection [3]. Among all DL applications for wireless communication systems, signal detection is one of the primary topics. A recent study shows that by increasing channel awareness and allowing adaptive feedback mechanisms, the research study shown in [8] demonstrates how AI-driven techniques can improve overall system performance. Over the past decade, numerous research studies have been carried out that use DL techniques to overcome the signal detection issue in MIMO systems [3,9,10,11,12,13,14,15]. Despite demonstrating decent performance, these DL-aided signal detection techniques still have a considerable gap between detection accuracy and computational complexity, which demands further investigation. Based on this motivation, this research study presents a DL-driven signal detection method known as the MIMONet detector for MIMO systems. The newly developed model employs a feed-forward neural network (FFNN) model to improve detection performance and lower computational complexity. The proposed MIMONet is an extension of our previous work [3] that utilized a deep neural network (DNN) model for signal detection in the MIMO system. The primary contribution of this paper is highlighted as follows:

• We developed a DL-driven signal detection method known as the MIMONet detector for signal detection in a MIMO system and explored the study under diverse channel conditions.
• We developed a customized FFNN network architecture through selective feature-based network settings, which helped MIMONet to achieve optimal performance compared to benchmark MIMO detectors.
• MIMONet was analyzed through the bit error rate (BER) and complexity against both traditional MIMO detectors (MMSE, ZF, MLD, GS, CG, OCDBOX, and ADMIN) and AI-based detectors (AIDETECT, AMP-DNN, OAMP-Net, OAMP-Net2, and DetNet), highlighting its relative performance and efficiency.

The remaining part of the paper is structured in the following sections: Section 2 presents the related work based on traditional and AI-based MIMO detection methods. In Section 3, discuss the architecture of the MIMO system model, traditional MIMO detectors, and DL-based detection in MIMO systems. Section 4 presents the problem formulation and data preparation for the proposed MIMO detection method. Section 5 proposes the DL-driven MIMO detector called MIMONet for signal detection in MIMO systems. Section 6 presents a detailed discussion based on the simulation setup and results. Section 7 discusses the computational complexity of the proposed MIMO detector and other benchmark MIMO detection methods. Finally, Section 8 presents the conclusion of the paper and potential future direction.

2. Related Work

The rapid evolution of wireless communication technologies has established MIMO systems as a cornerstone for the realization of 6G networks. By leveraging large antenna arrays, MIMO systems substantially enhance spectral efficiency, data throughput, and connectivity [16]. However, the scaling of MIMO architectures introduces considerable challenges, notably in managing the increased channel complexity and meeting the stringent requirements for real-time signal detection [17]. These challenges limit the exploitation of MIMO’s full potential in emerging 6G environments. Consequently, the literature encompasses extensive research on both conventional and AI-based detection techniques. This section presents a comprehensive review of traditional MIMO detection methods, explores the emerging role of AI and DL approaches, and identifies critical research gaps that underpin the motivation for developing optimized AI-driven architectures such as the proposed MIMONet. Yang and Hanzo [2] provide a seminal survey categorizing MIMO detection algorithms into three principal classes: optimal, linear, and nonlinear methods. Their analysis elucidates the theoretical performance bounds and practical constraints of these categories, particularly in large-scale MIMO contexts. Optimal detection methods, typified by MLD, achieve a minimum BER by exhaustively evaluating all possible transmitted signal vectors. Nevertheless, the computational complexity of MLD grows exponentially with the number of transmit antennas and modulation order, rendering it impractical for real-time applications in systems utilizing high-order Quadrature Amplitude Modulation (QAM) or massive antenna arrays [18,19]. To address these computational challenges, linear detection schemes such as ZF and MMSE detectors have been widely adopted. The ZF detector mitigates inter-stream interference through channel matrix inversion, albeit at the expense of noise amplification in poorly conditioned channels. MMSE detection improves noise resilience by incorporating statistical noise properties into the estimation process. However, both approaches suffer from degraded BER performance under high noise levels or spatially correlated fading scenarios [20,21]. Despite their favorable computational efficiency, these linear methods inherently struggle to suppress residual interference and face scalability limitations in large-scale MIMO deployments [2]. The traditional MIMO detection techniques provide a robust theoretical and practical foundation due to their analytical tractability and efficiency, and their scalability constraints in ma-MIMO systems have considerable interest in AI-driven alternatives. AI-based detection frameworks, particularly those employing DL, have gained traction for their capacity to learn complex channel characteristics and offer adaptive, computationally efficient solutions capable of surpassing conventional performance limits. Among AI-aided methods, Mohammad et al. [22] introduced WeSNet, a weight-scaling neural network architecture designed for complexity-scalable MIMO detection. Building upon the Detection Network (DetNet) paradigm, WeSNet employs monotonically non-increasing profile functions to modulate layer-wise weights adaptively during inference, enabling partial weight computation and reducing computational complexity by approximately 50% without compromising detection accuracy. Extensions such as L-WeSNet and R-WeSNet incorporate trainable weight profiles and log-regularized sparsity constraints, respectively, facilitating the dynamic pruning of redundant network parameters with minimal BER impact. Performance evaluations demonstrated that WeSNet achieved a tenfold reduction in computational complexity relative to ML detection while outperforming DetNet across diverse MIMO configurations. This work exemplifies how deep unfolding and weight modulation techniques can yield efficient detectors tailored for resource-constrained 6G applications. The research study presented in [23] proposed the Maximum a Normalizing Flow Estimate (MANFE) framework, addressing MIMO signal detection under non-Gaussian or unknown noise distributions. Unlike classical detectors predicated on additive white Gaussian noise (AWGN) assumptions, MANFE leverages normalizing flows to probabilistically estimate noise distributions from raw data in an unsupervised manner, enabling near-ML detection performance in challenging noise environments, including impulsive and heavy-tailed scenarios. To mitigate computational demands, the authors developed a low-complexity variant, G-GAMP MANFE, which employs Generalized Approximate Message Passing (G-GAMP) for preliminary estimates, effectively constraining the ML search space and achieving ML-level performance with significantly reduced complexity. A study presented in [24] proposed a DNN-based partial maximum a posteriori detection technique called PMAP-Net, which achieves benchmark performance in a variety of MIMO settings while drastically lowering computational cost. This study shows that adding DNNs to multi-stage detection frameworks can improve computational efficiency and detection reliability. Albreem et al. [25] conducted an extensive survey categorizing DL-based MIMO detectors into fully connected neural networks, convolutional neural networks (CNNs), and unfolded iterative models such as DetNet and OAMP-Net. Their analysis highlights the efficacy of model-driven learning approaches that integrate domain knowledge within neural architectures to achieve near-optimal detection with reduced training complexity. Expanding this perspective, Omondi and Olwal [26] reviewed a broad spectrum of DL techniques, including Graph Neural Networks (GNNs), Reinforcement Learning (RL), and hybrid unfolding models, underscoring the necessity for scalable, generalizable AI models capable of adapting to diverse user loads, modulation schemes, and channel conditions anticipated in 6G MIMO networks. Focusing on practical implementations, a research study proposed AIDETECT, a lightweight AI-based detector optimized for MIMO systems [3]. Trained on realistic channel models, AIDETECT outperforms traditional and baseline AI detectors in symbol error rate. A subsequent comparative study demonstrated AIDETECT’s superiority over advanced models such as DetNet and OAMP-Net2, achieving more than 97% gain at 20 dB signal-to-noise ratio (SNR) while incurring lower training overhead. Nguyen et al. [27] evaluated multiple unfolding-based detectors, emphasizing the benefits of incorporating algorithmic priors into deep network layers. Their results indicate enhanced generalization across modulation schemes and antenna configurations, establishing deep unfolding architectures as highly suitable for adaptive MIMO detection. Furthermore, the noncoherent detection methods are becoming increasingly relevant in massive MIMO systems, mainly due to the challenges and high overhead involved in acquiring accurate channel state information (CSI) when dealing with a large number of antennas and users. Instead of relying on CSI, noncoherent techniques offer an alternative approach to signal detection that can significantly reduce complexity and resource requirements. For example, a research study presented in [28] proposed space-time block coding schemes using PSK constellations that allow for reliable decoding without needing CSI at either the transmitter or receiver, making them especially useful for uplink scenarios where pilot resources are limited [28]. Other techniques, like autocorrelation-based detection, beamspace transformations, and differential feedback, have also shown good results in separating users in low-SNR and far-field conditions [29,30]. Overall, these efforts highlight the growing appeal of noncoherent detection as a practical and scalable solution for future massive MIMO networks, especially where coherent methods may be too resource-intensive or impractical. Collectively, these AI-driven advancements represent a paradigm shift in MIMO detection methodologies. From probabilistic generative models to deep unfolding networks, DL approaches effectively address the limitations of traditional detectors. Nevertheless, challenges persist in balancing detection accuracy, computational complexity, and practical deployment feasibility. These challenges motivate the design of novel architectures like MIMONet, which aim to unify efficient FFNN design with application-aware optimization for 6G ma-MIMO systems.

3. MIMO Detection: A System Model

We consider a MIMO communication system comprising $P_t$ transmit and $Q_u$ receive antennas. The received signal model in the complex domain is expressed as follows:

$$\tilde{\mathbf{r}}=\tilde{\mathbf{G}}\tilde{\mathbf{s}}+\tilde{\mathbf{w}},$$

(1)

where $\tilde{\mathbf{r}}\in {\mathbb{C}}^{Q_u\times 1}$ denotes the complex-valued received signal vector, $\tilde{\mathbf{s}}\in {\mathbb{C}}^{P_t\times 1}$ is the transmitted symbol vector, $\tilde{\mathbf{G}}\in {\mathbb{C}}^{Q_u\times P_t}$ is the Rayleigh fading channel matrix, and $\tilde{\mathbf{w}}\in {\mathbb{C}}^{Q_u\times 1}$ represents the additive white Gaussian noise (AWGN) vector with zero mean and variance ${\sigma }^2$. The noise vector $\tilde{\mathbf{w}}$ is modeled as a complex Gaussian random vector:

$$\tilde{\mathbf{w}}\sim \mathcal{CN}(\mathbf{0},{\sigma }^2{\mathbf{I}}_{Q_u}),$$

(2)

where $\mathcal{CN}(·)$ denotes the complex normal distribution, ${\sigma }^2$ is the noise variance per complex dimension, and ${\mathbf{I}}_{Q_u}$ is the $Q_u\times Q_u$ identity matrix. After transforming to the real-valued domain, the noise vector becomes $\mathbf{w}\in {\mathbb{R}}^{2Q_u\times 1}$ and is distributed as

$$\mathbf{w}\sim \mathcal{N}(\mathbf{0},{\displaystyle \frac{{\sigma }^2}{2}}{\mathbf{I}}_{2Q_u}),$$

(3)

where $\mathcal{N}(·)$ represents the multivariate real Gaussian distribution. The factor ${\displaystyle \frac{1}{2}}$ arises due to the decomposition of each complex dimension into two real dimensions, each having half the variance. We assume perfect channel state information (CSI) at the receiver. Solving the complex-valued MIMO channel model directly is challenging. The real and imaginary components of the transmitted and received symbol vectors are separated to provide an equivalent real-valued channel model without losing generality [31]. The conversion of a complex-valued model to a real-valued one is given as follows:

$$\left[\begin{array}{c}Re\left(\tilde{\mathbf{r}}\right)\\ Im\left(\tilde{\mathbf{r}}\right)\end{array}\right]=\left[\begin{array}{cc}Re\left(\tilde{\mathbf{G}}\right)& -Im\left(\tilde{\mathbf{G}}\right)\\ Im\left(\tilde{\mathbf{G}}\right)& Re\left(\tilde{\mathbf{G}}\right)\end{array}\right]\left[\begin{array}{c}Re\left(\tilde{\mathbf{s}}\right)\\ Im\left(\tilde{\mathbf{s}}\right)\end{array}\right]+\left[\begin{array}{c}Re\left(\tilde{\mathbf{w}}\right)\\ Im\left(\tilde{\mathbf{w}}\right)\end{array}\right].$$

(4)

Defining

$$\begin{array}{cc}\hfill \mathbf{r}& =\left[\begin{array}{c}Re\left(\tilde{\mathbf{r}}\right)\\ Im\left(\tilde{\mathbf{r}}\right)\end{array}\right]\in {\mathbb{R}}^{2Q_u\times 1},\hfill \\ \hfill \mathbf{s}& =\left[\begin{array}{c}Re\left(\tilde{\mathbf{s}}\right)\\ Im\left(\tilde{\mathbf{s}}\right)\end{array}\right]\in {\mathbb{R}}^{2P_t\times 1},\hfill \\ \hfill \mathbf{G}& =\left[\begin{array}{cc}Re\left(\tilde{\mathbf{G}}\right)& -Im\left(\tilde{\mathbf{G}}\right)\\ Im\left(\tilde{\mathbf{G}}\right)& Re\left(\tilde{\mathbf{G}}\right)\end{array}\right]\in {\mathbb{R}}^{2Q_u\times 2P_t},\hfill \end{array}$$

(5)

We obtain the canonical real-valued MIMO system model:

$$\mathbf{r}=\mathbf{G}\mathbf{s}+\mathbf{w},$$

(6)

For clarity, the system can also be expressed in an expanded matrix form:  

$$\left[\begin{array}{c}{r}_1\\ r_2\\ ⋮\\ r_{Q_u}\end{array}\right]=\left[\begin{array}{cccc}{g}_{1,1}& g_{1,2}& \dots & g_{1,P_t}\\ g_{2,1}& g_{2,2}& \dots & g_{2,P_t}\\ ⋮& ⋮& \ddots & ⋮\\ g_{Q_u,1}& g_{Q_u,2}& \dots & g_{Q_u,P_t}\end{array}\right]\left[\begin{array}{c}{s}_1\\ s_2\\ ⋮\\ s_{P_t}\end{array}\right]+\left[\begin{array}{c}{w}_1\\ w_2\\ ⋮\\ w_{Q_u}\end{array}\right]$$

(7)

3.1. Conventional MIMO Detection

Conventional MIMO detection methods have been proposed, which help to understand the MIMO detection system model and lead us toward the further development of more advanced MIMO detection methods.

3.1.1. Maximum Likelihood Detector

The maximum likelihood detector (MLD) minimizes the difference between the received vector and all possible transmitted symbol vectors as follows:

$${\widehat{\mathbf{s}}}_{\mathrm{MLD}}=arg\underset{\mathbf{s}\in {\mathcal{A}}^N}{min}{\parallel \mathbf{r}-\mathbf{G}\mathbf{s}\parallel }^2,$$

(8)

where $\mathcal{A}$ is the modulation constellation, and $N=2P_t$ is the dimension of $\mathbf{s}$ [1,2].

3.1.2. Zero-Forcing Detector

The ZF detector attempts to eliminate inter-stream interference by computing the pseudo-inverse of the channel matrix:

$$\widehat{\mathbf{s}}={\left({\mathbf{G}}^T\mathbf{G}\right)}^{-1}{\mathbf{G}}^T\mathbf{r}.$$

(9)

Each element of the resulting symbol vector is then quantized to the nearest constellation point:

$${\widehat{\mathbf{s}}}_{\mathbf{i}}=arg\underset{b\in \mathcal{A}}{min}|z_i-b|.$$

(10)

However, ZF suffers from noise enhancement, particularly when $\mathbf{G}$ is ill-conditioned [3].

3.1.3. Minimum Mean Square Error Detector

To mitigate noise amplification, MMSE incorporates noise statistics into the detection:

$$\mathbb{E}\left[\mathbf{s}\right|\mathbf{r}]={\left({\mathbf{G}}^T\mathbf{G}+{\displaystyle \frac{{\sigma }^2}{E_m}}\mathbf{I}\right)}^{-1}{\mathbf{G}}^T\mathbf{r},$$

(11)

where $E_m$ is the average symbol energy. Each symbol is then mapped to the nearest constellation point:

$${\widehat{\mathbf{s}}}_{\mathbf{i}}=arg\underset{b\in \mathcal{A}}{min}\left|\mathbb{E}\left[s_i\right|\mathbf{r}]-b\right|.$$

(12)

3.2. Gauss–Seidel Detector

The Gauss–Seidel method solves the linear system iteratively. For matrix $\mathbf{A}={\mathbf{G}}^T\mathbf{G}$, we decompose as follows:

$$\mathbf{A}=\mathbf{D}+\mathbf{L}+\mathbf{U},$$

(13)

where $\mathbf{D}$, $\mathbf{L}$, and $\mathbf{U}$ are the diagonal, strictly lower, and strictly upper triangular parts of $\mathbf{A}$, respectively. The iterative update is

$${\widehat{\mathbf{s}}}^{\left(n\right)}={(\mathbf{D}+\mathbf{L})}^{-1}\left({\widehat{\mathbf{s}}}_{\mathrm{MF}}-\mathbf{U}{\widehat{\mathbf{s}}}^{(n-1)}\right),$$

(14)

where ${\widehat{\mathbf{s}}}_{\mathrm{MF}}={\mathbf{G}}^T\mathbf{r}$ is the matched filter output. Despite improved convergence, GS is inherently sequential and, thus, not easily parallelizable [11,32].

3.3. Conjugate Gradient Detector

The CG method is preferred for large-scale systems due to its rapid convergence. It iteratively computes

$${\widehat{\mathbf{s}}}^{(n+1)}={\widehat{\mathbf{s}}}^{\left(n\right)}+{\alpha }^{\left(n\right)}{\mathbf{p}}^{\left(n\right)},$$

(15)

where ${\mathbf{p}}^{\left(n\right)}$ is the search direction and ${\alpha }^{\left(n\right)}$ is the step size. The directions satisfy the $\mathbf{A}$-conjugacy condition:

$${\left({\mathbf{p}}^{\left(n\right)}\right)}^T\mathbf{A}{\mathbf{p}}^{\left(j\right)}=0\mathrm{for}n\ne j.$$

(16)

CG is computationally efficient but involves matrix–vector operations that may be burdensome in hardware-constrained environments [11,32].

3.4. Optimized Coordinate Descent Detector

The OCD algorithm enhances coordinate descent by integrating preprocessing and channel-aware step sizes. The update for the k-th symbol is

$${\widehat{\mathbf{s}}}_{\mathbf{i}}={\left(\parallel {\mathbf{G}}_i{\parallel }^2+{\mathbf{w}}_0\right)}^{-1}{\mathbf{G}}_i^T\left(\mathbf{r}-\sum _{j\ne i}{\mathbf{G}}_j{\mathbf{s}}_j\right),$$

(17)

where ${\mathbf{G}}_i$ is the i-th column of the channel matrix, and ${\mathbf{w}}_0$ is the noise power. OCD achieves near-optimal performance with low per-iteration complexity, making it suitable for hardware implementations. While offering near-optimal performance with low complexity, OCD’s sequential updates can lead to latency, suggesting a need for parallelization strategies in future high-throughput applications [32].

3.5. DL-Based MIMO Detection

In DL-based MIMO detection, the MIMO detector calculates the estimated transmitted signal $\widehat{\mathbf{s}}$ by feeding the network with the information of the received signal $\mathbf{r}$ and channel $\mathbf{G}$. Now, the estimated transmitted signal $\widehat{\mathbf{s}}$ is measured by the nonlinear mapping function $m(\mathbf{G},\mathbf{r};\theta )$ through the following equation:

$$\widehat{\mathbf{s}}=m(\mathbf{G},\mathbf{r};\theta ),$$

(18)

where $m(·)$ denotes the inference rule that is used to detect the hidden symbols based on $\mathbf{r}$ and $\mathbf{G}$, and $\theta$ includes all parameters related to the DNN. The DNN aims to determine the ideal value of $\theta$ that reduces the associated expenses by developing a cost objective function that quantifies the difference between the actual and estimated symbols. The goal is to reduce the mean squared error between $\widehat{\mathbf{s}}$ and $\mathbf{s}$ as follows:

$$\widehat{\theta }=\underset{\theta }{argmin}f(\widehat{\mathbf{s}},S),$$

(19)

where $f(·)$ represents the cost function [9] and S is the transmitted symbol set. Here we discuss our previously proposed work [3] used as our benchmark models.

3.6. AIDETECT MIMO Detector

Our previously proposed deep learning-enhanced signal detection framework, called AIDETECT, is designed for efficient signal detection in MIMO systems. It offers improved accuracy and robustness compared to conventional and other DL-based methods while maintaining lower complexity. For AIDETECT, the detected signal vector $\widehat{\mathbf{s}}$ is determined by reducing the cost objective function f as

$$\widehat{\mathbf{s}}=arg\underset{\mathbf{s}\in S^{P_t}}{min}f(({\mathbf{r}}_{\mathrm{Re}},{\mathbf{G}}_{\mathrm{Re}}),({\mathbf{r}}_{\mathrm{Im}},{\mathbf{G}}_{\mathrm{Im}}))$$

(20)

where $f(({\mathbf{r}}_{\mathrm{Re}},{\mathbf{G}}_{\mathrm{Re}}),({\mathbf{r}}_{\mathrm{Im}},{\mathbf{G}}_{\mathrm{Im}}))$ is a cost objective function that depends on $\mathbf{r}$ and $\mathbf{G}$ extracted by $\widehat{\mathbf{s}}$. We set the AIDETECT method as our benchmark method along with the MLD, ZF, and MMSE detection methods [3].

4. Problem Formulation and Data Preparation

4.1. Problem Formulation

Modern wireless communication systems increasingly rely on MIMO technologies to meet the demands for higher data rates, spectral efficiency, and robustness. However, signal detection in MIMO systems remain a critical challenge due to their high-dimensional nature, especially under varying channel conditions and interference. Traditional model-based detection algorithms, such as ZF, MMSE, or MLD, often suffer from scalability issues and increased computational complexity, particularly in large-scale MIMO configurations [33]. To overcome these limitations, AI and ML techniques have emerged as powerful alternatives. ML algorithms offer the capability to learn nonlinear relationships, adapt to diverse channel conditions, and generalize well from training data. These methods can enhance signal detection by dynamically adapting to channel variations and noise characteristics, thus supporting the development of ultra-reliable low-latency communications in next-generation networks. Furthermore, AI-driven solutions can optimize channel utilization and improve spectral efficiency through intelligent resource allocation based on real-time traffic patterns. In this work, we aim to design and evaluate a DL-driven signal detection model for MIMO systems by employing an FFNN trained on synthetic data generated for various modulation schemes. The objective is to develop a robust and generalizable detection framework capable of outperforming conventional algorithms in terms of both accuracy and computational efficiency.

4.2. Data Generation and Preprocessing

The data preparation phase plays a crucial role in determining the performance and generalization of any ML-based model. In our simulation framework, we begin by generating a set of transmitted symbols based on a predefined modulation scheme, such as QAM or quadrature phase shift keying (QPSK), with a specified constellation size (e.g., 4-QAM, 16-QAM). To create the training and testing datasets, two distinct sets of random indices are generated to form the transmitted symbol vectors, denoted by ${\mathbf{s}}_{\mathrm{train}}$ and ${\mathbf{s}}_{\mathrm{test}}$, respectively. The sizes of these vectors are defined by the parameters $n_{\mathrm{train}}$ and $n_{\mathrm{test}}$. Each transmitted symbol vector is paired with a corresponding MIMO channel matrix $\mathbf{G}$, which is generated using a Rayleigh fading distribution. The Rayleigh model is suitable for representing environments with multiple non-line-of-sight signal propagation paths and is widely adopted for modeling wireless channels. The elements of $\mathbf{G}$ are normalized such that their magnitudes follow a Rayleigh distribution with a unit scaling parameter, ensuring realistic channel behavior. The received signal vector $\mathbf{r}$ is computed based on the canonical MIMO system equation, where $\mathbf{w}$ is the additive white Gaussian noise vector, and the SNR is varied across a predefined range to evaluate the robustness of the model under different noise conditions. For each SNR point, a large number of samples are generated to ensure sufficient diversity in the training and testing processes. Feature extraction is then performed by separating the real and imaginary parts of the received signal $\mathbf{r}$ and the channel matrix $\mathbf{G}$. This results in a concatenated feature vector containing $\mathrm{Re}\left\{\mathbf{r}\right\}$, $\mathrm{Im}\left\{\mathbf{r}\right\}$, $\mathrm{Re}\left\{\mathbf{G}\right\}$, and $\mathrm{Im}\left\{\mathbf{G}\right\}$, which is used as input to the neural network. The ground truth labels corresponding to the transmitted symbols $\mathbf{s}$ are also stored and later used for supervised learning during the training phase. The complete data pipeline ensures that the model is exposed to a wide variety of transmission scenarios, enabling it to learn robust feature representations and improve its generalization capability. This diverse training data serves as the foundation for developing a neural network-based MIMO detector that can adapt to complex channel conditions and outperform traditional detection algorithms.

5. MIMONet: A DL-Based MIMO Detector

The proposed MIMONet is a DL-driven MIMO detector designed to achieve optimal detection performance with low computational complexity. The MIMONet detector utilizes a state-of-the-art DL-based customized FFNN model structured specifically for MIMO signal detection, which operates effectively across a range of SNR. By leveraging a customized FFNN architecture, the MIMONet detector efficiently performs signal detection, ensuring both high detection accuracy and low resource usage. The functionality for designing and implementing the MIMONet detector can be broken down into two primary phases: network architecture design and training/testing of the detector. Figure 1 presents the systematic architecture of the MIMONet detector. The following sub-section presents a detailed explanation of each phase.

5.1. MIMONet Network Architecture

The MIMONet network architecture consists of several key components: an input layer, three hidden layers, and an output layer. The feature data, which includes both real and imaginary components of the received signal and channel matrix, is passed through the network to output the predicted transmitted symbols. The structure of the MIMONet architecture allows the model to learn intricate relationships between the input features and the output symbols, optimizing the detection process. This process is further broken down based on network layers:

• Input Layer: In the input layer, the received signal $\mathbf{r}$ and channel matrix $\mathbf{G}$ are processed by first separating their real and imaginary parts:

  $$\mathbf{r}=\mathrm{Re}\left(\mathbf{r}\right)+j\mathrm{Im}\left(\mathbf{r}\right),\mathbf{G}=\mathrm{Re}\left(\mathbf{G}\right)+j\mathrm{Im}\left(\mathbf{G}\right)$$

  (21)
  The real and imaginary components of the received signal and channel matrix are concatenated into a feature vector $\mathbf{X}$ that passes through the network’s hidden layers, which process and learn the patterns present in the data.
• Hidden Layers: Each hidden layer of the MIMONet consists of 128 neurons and employs a fully connected layer along with a clipped ReLU activation function to introduce nonlinearity. The output of the first hidden layer ${\mathbf{h}}_1$ is computed as follows:

  $${\mathbf{h}}_1={\varphi }_1\left({\mathsf{\Psi }}_1·\mathbf{X}+{\beta }_1\right)$$

  (22)

  where ${\mathsf{\Psi }}_1$ and ${\beta }_1$ represent the weight matrix and bias vector of the first hidden layer, and ${\varphi }_1$ is the activation function (clipped ReLU). Similarly, the second and third hidden layers produce the outputs ${\mathbf{h}}_2$ and ${\mathbf{h}}_3$ as

  $${\mathbf{h}}_2={\varphi }_2\left({\mathsf{\Psi }}_2·{\mathbf{h}}_1+{\beta }_2\right)$$

  (23)

  $${\mathbf{h}}_3={\varphi }_3\left({\mathsf{\Psi }}_3·{\mathbf{h}}_2+{\beta }_3\right)$$

  (24)

  where ${\mathsf{\Psi }}_2, {\mathsf{\Psi }}_3$ are the weight matrices for the second and third hidden layers, ${\beta }_2, {\beta }_3$ are the corresponding bias vectors, and ${\varphi }_2, {\varphi }_3$ are the activation functions for the second and third hidden layers. These hidden layers allow the network to progressively learn more abstract features of the received signal and channel data.
• Output Layer: The final output layer of the network produces the predicted transmitted symbols $\widehat{\mathbf{s}}$. This output is obtained by applying an argmax function to the output of the third hidden layer:

  $$\widehat{\mathbf{s}}=argmax\left({\mathsf{\Psi }}_4·{\mathbf{h}}_3+{\beta }_4\right)$$

  (25)
  Similarly, ${\mathsf{\Psi }}_4$ and ${\beta }_4$ represent the weight matrix and bias vector for the output layer, respectively. This operation identifies the most likely transmitted symbol, as predicted by the MIMONet detector.

5.2. MIMONet: Training and Testing

The training and testing phases of the MIMONet detector involve generating training and testing data, processing the received signals through the customized FFNN-based MIMONet detection model, and evaluating the performance of the detector using SER.

5.2.1. Data Generation

The first step in training and testing the MIMONet detection model is the generation of training and testing datasets. The transmitted signal $\mathbf{s}$ is drawn randomly from a QAM constellation, and the received signal $\mathbf{r}$ is calculated using the MIMO channel model Equation (1). After collecting the transmitted data at the receiver end, the received signal $\mathbf{r}$ and channel matrix $\mathbf{G}$ are then separated into their real and imaginary components and concatenated into the feature vector $\mathbf{X}$:

$$\mathbf{X}=\left[\mathrm{Re}\left(\mathbf{r}\right),\mathrm{Im}\left(\mathbf{r}\right),\mathrm{Re}\left(\mathbf{G}\right),\mathrm{Im}\left(\mathbf{G}\right)\right]$$

(26)

This feature vector is used as input to the neural network during both training and testing.

5.2.2. Training

During the training process, the network learns to minimize the loss between the predicted symbols $\widehat{\mathbf{s}}$ and the true transmitted symbols s. The loss function used is the mean squared error loss, given by

$$\mathcal{L}={\displaystyle \frac{1}{N_{\mathrm{Train}}}}\sum _{i=1}^{N_{\mathrm{Train}}}{\left({\widehat{\mathbf{s}}}_i-{\mathbf{s}}_i\right)}^2$$

(27)

where $N_{\mathrm{Train}}$ is the number of training samples. The weights and biases of the network are updated using backpropagation and the Adam optimizer. The gradient of the loss function with respect to each parameter is calculated, and the weights are updated using the following update rule:

$${\mathsf{\Psi }}_i={\mathsf{\Psi }}_i-\eta ·{\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathsf{\Psi }}_i}},{\beta }_i={\beta }_i-\eta ·{\displaystyle \frac{\partial \mathcal{L}}{\partial {\beta }_i}}$$

(28)

where $\eta$ is the learning rate.

5.2.3. Testing

During testing, the trained network is used to predict the transmitted symbols $\widehat{\mathbf{s}}$ based on the received signal and channel matrix. The predicted symbols are compared with the true transmitted symbols to compute the SER. The SER is defined as the fraction of incorrectly detected symbols:

$$\mathrm{SER}={\displaystyle \frac{1}{N_{\mathrm{Test}}}}\sum _{i=1}^{N_{\mathrm{Test}}}\mathbb{I}\left({\widehat{\mathbf{s}}}_i\ne {\mathbf{s}}_i\right)$$

(29)

where $N_{\mathrm{Test}}$ is the number of test samples, and $\mathbb{I}$ is the indicator function, which is 1 if the symbol is incorrectly detected and 0 otherwise. In addition to SER, we also evaluate the BER, which measures the proportion of incorrectly detected bits across all transmitted symbols. After decoding the predicted symbols $\widehat{\mathbf{s}}$ into their corresponding bitstreams $\widehat{\mathbf{b}}$, BER is calculated as follows:

$$\mathrm{BER}={\displaystyle \frac{1}{N_{\mathrm{bits}}}}\sum _{i=1}^{N_{\mathrm{bits}}}\mathbb{I}\left({\widehat{\mathbf{b}}}_i\ne {\mathbf{b}}_i\right)$$

(30)

where $N_{\mathrm{bits}}$ is the total number of transmitted bits. The inclusion of both SER and BER allows for a comprehensive evaluation of the detector’s performance at both symbol and bit levels. To extract the final estimated transmitted signal $\widehat{s}$ based on the entire network model from start to end is derived through the following mathematical formulation:

$$\widehat{\mathbf{s}}=argmax\left({\mathsf{\Psi }}_4·{\varphi }_3[{\mathsf{\Psi }}_3·{\varphi }_2({\mathsf{\Psi }}_2·{\varphi }_1({\mathsf{\Psi }}_1·\mathbf{X}+{\beta }_1)+{\beta }_2)+{\beta }_3]+{\beta }_4\right)$$

(31)

where ${\mathsf{\Psi }}_i$ represents the weight matrices at different layers, ${\beta }_i$ represents the bias terms, ${\varphi }_i$ denotes the activation functions applied at each layer, and the argmax function selects the output with the highest probability corresponding to the transmitted signal. The training and testing procedure of the MIMONet MIMO detector is outlined in Algorithm 1, which presents the detailed steps involved in the process:

Algorithm 1: MIMONet: Deep learning-based MIMO detector

6. Simulation Setup and Results Discussion

To replicate the proposed method, a custom simulator based on MATLAB R2022b was developed. This simulator ran on a stand-alone Intel i9-10900K CPU @ 3.70 GHz with 128 GB RAM and included a 32GB GPU for improved computational efficiency.

Table 1. Simulation Parameters.

Parameter	Value
Number of transmitters	4, 8, 16
Number of receivers	4, 8, 32
Modulation	QPSK, QAM
Constellation size	4, 16
Data size	1 Million
SNR Range	0:2:20 dB and 0:5:20 dB
Training Function	Adam
Number of neurons	128
Number of hidden layers	3
Maximum epochs	2000
Mini Batch Size	10
Initial learning rate	0.001
L2-regularization	0.0001

outlines the parameters used in the simulation setup.

6.1. Results Discussion

To evaluate the effectiveness of the proposed DL-driven MIMONet detector, its performance is systematically benchmarked against both traditional MIMO detectors (MMSE, ZF, MLD, GS, CG, OCDBOX, and ADMIN) and several state-of-the-art AI-driven detectors such as AIDETECT, AMP-DNN, OAMP-Net, OAMP-Net2, and DetNet. The assessment is carried out through comprehensive simulation studies across two representative MIMO configurations, 4 × 4 and 8 × 8 systems, each featuring an equal number of transmit and receive antennas. These scenarios are designed to test the robustness, accuracy, and scalability of MIMONet under varying system complexities.

6.1.1. Performance Analysis of MIMONet Detector Against Traditional MIMO Detectors

To measure the performance of the MIMONet detector against traditional MIMO detectors, we conducted simulations for 4 × 4 and 8 × 8 MIMO scenarios. Figure 2 presents the BER performance of various signal detection algorithms evaluated within a 4 × 4 MIMO system using QAM modulation as a function of the average SNR per receive antenna. The DL-based detector, MIMONet, exhibits clear superiority across the full SNR range. At an SNR of 20 dB, MIMONet achieves an exceptionally low BER, outperforming conventional detectors by two to three orders of magnitude. This significant performance margin highlights the capacity of DNNs to learn complex signal features and optimize detection boundaries that are difficult to model analytically. In contrast, the MLD algorithm shows a distinct performance trajectory underperforming at low SNRs but demonstrating rapid improvement beyond 15 dB, eventually ranking as the second-best performer in high-SNR conditions. This behavior is consistent with the theoretical nature of MLD, which becomes increasingly effective as the noise level diminishes. Among conventional linear and iterative detectors, the GS method consistently achieves marginally better BER performance compared to others, while ZF lags slightly at higher SNR values. MMSE and ADMIN follow similar performance patterns, with MMSE offering a slight edge at elevated SNR levels.

Apart from a small-scale 4 × 4 MIMO system, we also considered a mid-range 8 × 8 MIMO system to analyze the performance of the MIMONet detector against the benchmark detectors. Figure 3 presents the BER performance of various detection algorithms within an 8 × 8 MIMO system using QAM modulation. Compared to the 4 × 4 configuration, this higher-dimensional setup introduces increased detection complexity and greater signal interference. Despite these challenges, the DL-based MIMONet consistently demonstrates the lowest BER across all SNR values, showcasing its remarkable ability to generalize and scale in larger antenna systems. The MLD algorithm continues to exhibit strong performance in the high-SNR regime, ranking as the second-best performer. However, its BER curve lies noticeably above that of MIMONet, illustrating the exponential performance trade-off and computational limitations associated with exhaustive search strategies in larger MIMO configurations. In contrast, conventional detection techniques including MMSE, ZF, CG, GS, OCDBOX, and ADMIN exhibit closely grouped, relatively high BER curves. Their limited improvement at increasing SNR levels indicates difficulty in effectively mitigating inter-stream interference in the expanded signal space of the 8 × 8 system.

In addition to the 4 × 4 and 8 × 8 MIMO scenarios, we extended our simulation study to a 16 × 32 ma-MIMO system. Figure 4 presents the BER performance of various detection algorithms within a 16 × 32 ma-MIMO system using QAM modulation. Despite the higher-dimensional setup, the simulation results show that the proposed MIMONet model performs better than the traditional benchmark MIMO detectors at all SNR points.

6.1.2. Performance Analysis of MIMONet Detector Against AI-Based MIMO Detectors

To compare the performance of the MIMONet detector against AI-based MIMO detectors, we conducted simulation experiments based on QPSK and 16-QAM modulation schemes for 8 × 8 MIMO scenarios. Figure 5 presents the BER performance of several DL-based detection algorithms for an 8 × 8 MIMO system using QPSK modulation. The simulation results show that, among all AI-based MIMO detection methods, MIMONet consistently demonstrates superior detection accuracy across the entire SNR range. At a 20 dB SNR, MIMONet achieves a BER nearing ($1\times {10}^{-6}$), significantly outperforming all competing models by several orders of magnitude. AIDETECT emerges as the second-best performer; however, its performance remains substantially inferior to that of MIMONet. The AMP-DNN and OAMP-Net2 algorithms exhibit moderate detection capability, while DetNet and OAMP-Net record the highest BER values, indicating limited robustness in lower-SNR regimes and constrained generalization under QPSK conditions. This analysis underscores MIMONet’s exceptional learning capacity and resilience, even when trained with relatively modest computational resources and low-order modulation schemes. Moreover, Figure 6 extends the performance evaluation to a more complex modulation format—16-QAM—within the same 8 × 8 MIMO system. Despite the increased modulation complexity, MIMONet once again outperforms all other algorithms, achieving a BER below ($3\times {10}^{-5}$) at 20 dB SNR. The detection margin between MIMONet and the next-best algorithm, AIDETECT, becomes even more pronounced compared to the QPSK scenario, highlighting MIMONet’s scalability and its proficiency in learning sophisticated signal patterns. Although AMP-DNN improves in higher SNR regions, it remains unable to match MIMONet’s performance. Meanwhile, DetNet, OAMP-Net, and OAMP-Net2 retain their relative positions as lower-performing detectors, particularly struggling to suppress symbol errors effectively in dense signal constellations. These findings confirm that MIMONet’s advanced neural architecture and extensive training regimen are highly effective in managing the added challenges posed by higher-order modulation in large-scale MIMO systems.

6.1.3. MIMONet Under Diverse Channel Conditions

Our study primarily focused on the Rayleigh fading channel. However, to ensure a more comprehensive evaluation across different channel conditions, we incorporated the Rician fading channel to represent the distinct wireless propagation conditions. The Rician fading channel represents a practical scenario where the line of sight (LOS) components exist, making the evaluation more representative of diverse communication conditions. In this context, the Rician K-factor, which typically ranges from 1 to 10, quantifies the ratio of the LOS power to the power of the scattered non-line-of-sight (NLOS) components and serves as a critical parameter characterizing the degree of fading and the dominance of the LOS path. To validate MIMONet’s performance under different propagation environments, we conducted additional simulations for a 4 × 4 MIMO system with 16-QAM modulation across the Rician fading channel. The following Figure 7 presents a performance analysis of MIMONet against linear detectors MMSE and ZF for a 4 × 4 MIMO system with 16-QAM modulation under two different channels, including a Rayleigh fading channel and the Rician fading channel. The simulation results show that MIMONet, MMSE, and ZF with a Rician fading channel perform better than MIMONet, MMSE, and ZF with a Rayleigh fading channel because of the LOS effect.

6.1.4. Training and Validation of MIMONet for 4 × 4 and 8 × 8 MIMO Systems

We also analyze the training, testing, and validation of the MIMONet detector based on RMSE and loss for 4 × 4 and 8 × 8 MIMO systems. Figure 8 shows the training performance of MIMONet for a 4 × 4 MIMO system using 16-QAM modulation. The root mean square error (RMSE) curve exhibits a rapid decline from above 1.4 to below 0.4 over 2000 epochs, indicating effective learning and stable convergence. The training and validation loss curves follow a similar trend, demonstrating consistent reduction without signs of overfitting. The close alignment between training and validation results confirms that MIMONet generalizes well to unseen data in moderate MIMO configurations. In Figure 9, the training dynamics for an 8 × 8 MIMO system show a higher initial RMSE (above 2.2) due to increased complexity but follow a steady decline to below 0.5. The corresponding loss curve decreases smoothly, indicating stable optimization throughout. Despite the larger system size, the close match between training and validation metrics demonstrates MIMONet’s scalability and robustness, validating its effectiveness in high-dimensional MIMO detection tasks.

7. Computational Complexity

The computational complexity of the ZF, MMSE, and MLD MIMO detectors mainly depends on the number of antennas and constellation size; however, for AI-based MIMO detectors, it majorly depends upon network architecture [3]. In this scenario, due to the customized lightweight FFNN architecture, the MIMONet has much lower computational complexity than benchmark MIMO detectors.

Computational Complexity of MIMONet Detector

The MIMONet model is designed using a customized FFNN architecture consisting of three hidden layers, each with 128 neurons. It employs the Clipped ReLU activation function, which offers a balance between nonlinearity and gradient stability, making it suitable for DL applications in signal detection. The computational complexity of the model is predominantly influenced by the fully connected layers. In general, the complexity of such a network can be represented by the total number of weight multiplications across all layers:

$${\mathcal{C}}_{\mathrm{MIMONet}}=\mathcal{O}\left(\sum _{l=1}^{L^{\prime }}{n}_l·n_{l-1}\right)=\mathcal{O}\left(128·a+{128}^2+128·m\right)$$

(32)

Here, $L^{\prime }$ denotes the number of layers $n_l$, and $n_{l-1}$ represents the number of neurons in the current and preceding layers, respectively; a is the input feature dimension, and m is the output dimension. This formulation captures the cumulative computational load during a single forward pass through the network.

Table 2. The computational complexity of MIMO detectors based on Big O notation.

MIMO Detectors	Computational Complexity
MLD [2]	$\mathcal{O}\left(Q^{\prime P_t}\right)$$Q^{\prime }=modulationorder$
MMSE [12]	$\mathcal{O}\left(P_t^3\right)+\mathcal{O}\left(Q_u{P}_t^2\right)+\mathcal{O}\left(P_t{Q}_u^2\right)$
ZF [12]	$\mathcal{O}\left(P_t^3\right)+\mathcal{O}\left(Q_u{P}_t^2\right)$
CG [34]	$\mathcal{O}\left((T+1)(4P_t^2+20P_t)\right)$$T=iterations$
NS [35]	$\mathcal{O}\left((T-1)(2P_t^3+2P_t^2-2P_t)\right)$
GS [36]	$\mathcal{O}\left(6T{P}_t^2\right)$
OCDBOX [37]	$\mathcal{O}\left(T(2Q_u{P}_t+P_t)\right)$
ADMIN [37]	$\mathcal{O}(P_t^3+Q_u^2+Q_u+T(P_t^2+P_t))$
OAMP-Net [38]	$\mathcal{O}\left(P_t^3\right)$
OAMP-Net2 [39]	$\mathcal{O}\left(L^{\prime }{Q}_u^3\right)$ ($L^{\prime }$ = number of layers)
DetNet2 [10]	$\mathcal{O}\left(L^{\prime }{Q}_u{P}_t^2\right)$
AMP-DNN [40]	$\mathcal{O}\left(L^{\prime }{Q}_u{P}_t\right)$
AIDETECT [11]	$\mathcal{O}({\sum }_{n=3}^N{N}_{n-1}{N}_n$ ($N_n$ = number of neurons in nth layer)
MIMONet	$\mathcal{O}\left({\sum }_{l=1}^{L^{\prime }}{n}_l·n_{l-1}\right)=\mathcal{O}\left(128·a+{128}^2+128·m\right)$

summarizes the computational complexity of different MIMO detectors. In addition to the table comparison, we conducted a complexity comparison of MIMONet against the benchmark MIMO detectors, as shown in Figure 10. This complexity comparison is carried out in terms of the number of FLOPs, which provides a fair evaluation of the computational cost related to each detection approach.

8. Conclusions and Future Work

AI has significantly advanced the capabilities of MIMO systems, positioning them as key enablers for B5G and 6G networks. However, reliable signal detection remains a core challenge due to the increasing complexity of MIMO environments. This work proposes MIMONet, a lightweight DL-based detector leveraging a customized FFNN architecture for efficient MIMO signal detection. The proposed MIMONet demonstrates strong performance across both 4 × 4 and 8 × 8 MIMO configurations, achieving benchmark performance while maintaining low computational complexity compared to conventional model-based and learning-based detectors. These results validate the potential of data-driven architectures for efficient and scalable signal detection in modern wireless systems. To expand MIMONet’s applicability to realistic 6G situations, further research will investigate its scalability to ultra-massive MIMO systems (such as 64 × 64 and 128 × 128), multi-user MIMO, and frequency-selective channels. Additionally, to improve generalization, we will use sophisticated training techniques, including transfer learning and data augmentation, and examine MIMONet’s resilience to non-Gaussian noise models. Future work will also focus on federated learning for distributed MIMO, noncoherent detection, and a thorough examination of hardware-level performance and training complexity, all of which will be backed by COST2100 or the NYU mm-Wave channel models.