A Comparative Analysis of DNN and Conventional Signal Detection Techniques in SISO and MIMO Communication Systems

Abstract

This paper investigates the performance of deep neural network (DNN)-based signal detection in multiple input, multiple output (MIMO), communication systems. MIMO technology plays a critical role in achieving high data rates and improved capacity in modern wireless communication standards like 5G. However, signal detection in MIMO systems presents significant challenges due to channel complexities. This study conducts a comparative analysis of signal detection techniques within both the single input, single output (SISO), and MIMO frameworks. The analysis focuses on the entire transmission chain, encompassing transmitters, channels, and receivers. The effectiveness of three traditional methods—maximum likelihood detection (MLD), minimum mean square error (MMSE), and zero-forcing (ZF)—is meticulously evaluated alongside a novel DNN-based approach. The proposed study presents a novel DNN-based signal detection model. While this model demonstrates superior computational efficiency and symbol error rate (SER) performance compared to more conventional techniques like MLD, MMSE, and ZF in the context of a SISO system, MIMO systems face some challenges in outperforming the conventional techniques specifically in terms of computation times. This complexity of MIMO systems presents challenges that the current DNN design has yet to fully address, indicating the need for further developments in wireless communication technology. The observed performance difference between SISO and MIMO systems underscores the need for further research on the adaptability and limitations of DNN architectures in MIMO contexts. These findings pave the way for future explorations of advanced neural network architectures and algorithms specifically designed for MIMO signal-processing tasks. By overcoming the performance gap observed in this work, such advancements hold significant promise for enhancing the effectiveness of DNN-based signal detection in MIMO communication systems.

1. Introduction

This research investigates data detection techniques for future communication systems beyond 5G, focusing on overcoming challenges associated with increased data rates, spectral efficiency, and reliability. The growing complexity of 5G necessitates the use of more antennas in terms of multiple inputs, multiple outputs (MIMO), which introduces new detection hurdles such as higher computational demands, inter-antenna interference, and the need for accurate channel information. The classical detection methods like maximum likelihood detection (MLD), minimum mean square error (MMSE), and zero-forcing (ZF) exhibit limitations in MIMO systems due to their increased complexity with a growing number of antennas [1,2]. This performance gap between SISO and MIMO systems motivates further exploration of deep neural network (DNN) architectures for MIMO data detection. Therefore, this study proposes a DNN-based MIMO detector to address these limitations and achieve reliable, high-quality wireless networks with large antenna arrays. The primary objective is to develop a MIMO detector leveraging a deep learning model for superior signal detection and error minimization compared to conventional approaches such as MLD, MMSE, and ZF. The proposed DNN detector is hypothesized to effectively handle the complexity of large-scale antenna systems, as in MIMO, and deliver much better performance in simulations, as well as real-life implementations. The proposed DNN-based detector’s efficacy is carefully assessed and compared with the traditional techniques (MLD, MMSE, and ZF). This covers performance measures including computing time, the signal-to-noise ratio (SNR), and the symbol error rate (SER).

The analysis presented in this work is based on the design and implementation of a DNN-based MIMO detector to enhance signal identification and achieve an SER reduction in MIMO systems. This work also explores the trade-off between computational efficiency and detection accuracy, considering the feasibility of deploying DNN-based detectors in beyond-5G systems. The performance of the DNN detector is evaluated and compared with traditional methods (MLD, MMSE, and ZF) to assess its efficacy relative to these established techniques. In addition to the MIMO systems, this work employs DNN to provide a method of signal detection in SISO systems as a base case. This DNN-based model, in contrast to conventional approaches, is engineered to efficiently manage the complexities of MIMO systems, offering improved computational efficiency and SER performance. This work not only assesses the DNN model’s efficacy but also looks into ways to increase system reliability and signal detection accuracy going forward.

To optimize the performance of DNN-based models, suitable training algorithms and techniques, such as stochastic gradient descent, have been implemented. A comprehensive analysis and interpretation of the simulation results have also been presented, highlighting the implications for improved detection accuracy, enhanced system reliability, and potential gains in spectral efficiency.

This investigation aims to provide valuable insights and recommendations for future research directions to further refine the effectiveness and efficiency of not only the DNN-based data detection methods but also other machine learning methods. The comparative analysis presented in this work offers a valuable understanding of the strengths and limitations of DNN-based approaches, paving the way for future advancements in 6G and beyond. The study strives to provide insightful perspectives and contribute to the development of communication systems beyond the 5G era by acknowledging and addressing these limitations.

The key contributions of this study are as follows: (1) the novel DNN-based signal detection model can tackle the particular difficulties facing MIMO systems, such as high computing demands and inter-antenna interference; (2) the proposed DNN-based detector’s efficacy is carefully assessed and compared with traditional techniques (MLD, MMSE, and ZF), covering performance measures including computing time, SNR, and SER; (3) a better understanding of the capabilities and limits of the model is developed before the study is expanded to complex MIMO situations by adding SISO system values as a baseline, which is used to test the DNN’s performance in less complicated circumstances; (4) 3D visualizations provide more profound understandings of the relationship between training functions, network architecture, and SER performance, which helps optimize DNN configurations for particular signal detection applications in MIMO systems; (5) the door to further investigations into the flexibility and constraints of DNN designs in MIMO scenarios is opened by highlighting the advantages and disadvantages of using DNN-based detectors for sophisticated wireless communication systems. The remaining sections of this paper are structured as follows: Section 2 presents the theoretical foundation for SISO and MIMO detection. Section 3 provides a comprehensive review of the relevant literature. Section 4 details the system description and methodology employed in this research. Section 5 presents the results and discussions, followed by the conclusion in Section 6.

2. Theoretical Foundation

Accurate signal detection is a critical necessity within wireless communication systems, as it directly influences the mitigation of symbol errors and the optimization of overall system throughput and spectral efficiency. In the context of SISO and MIMO systems, the transmission chain from a transmitter to a receiver undergoes distinct formulations.

For the SISO scenario, Equation (1) encapsulates the relationship, where ‘y’ represents the received signal, ‘x’ signifies the transmitted signal, ‘H’ denotes the channel matrix, and ‘n’ symbolizes noise.

$$y=Hx+n$$

(1)

Meanwhile, in the case of MIMO systems, Equation (1) extends to encompass multiple antennas and transmissions, represented as Equation (2), where ‘h’ denotes channel coefficients, ‘x’ represents transmitted symbols, and ‘n’ denotes noise.

$$y=\left[\begin{array}{cc}{h}_{11}& h_{12}\\ h_{21}& h_{22}\end{array}\right]\mathit{x}+\mathit{n}$$

(2)

$$y_1=h_{11}{x}_1+h_{21}{x}_2+n$$

(3)

$$y_2=h_{12}{x}_1+h_{22}{x}_2+n$$

(4)

The relations in (2)–(4) are represented in Figure 1.

There are several challenges that hinder signal detection in wireless communication systems. Noise, an inherent element, introduces uncertainty into the received signals. Additionally, interference arises from multiple users sharing the same frequency band, causing signal overlap and distortion. Furthermore, fading effects due to multi-path propagation lead to time-varying channel conditions, further complicating the process of signal detection.

The following sections delve deeper into these techniques, exploring their principles, mathematical foundations, strengths, limitations, and performance evaluation metrics. This theoretical foundation aims to provide insights into the capabilities and limitations of various techniques, paving the way for advanced DNN-based data detection models for massive MIMO systems.

2.1. Signal Detection Techniques

In wireless communication systems, accurate signal detection is crucial because it directly affects how well symbol mistakes are fixed, how much data is sent, and how efficiently the spectrum is used. The transmission chain from the transmitter to the receiver works in different ways in SISO and MIMO systems. In MIMO systems, the extra complexity comes from the fact that they use multiple antennas. The existing signal detection techniques can be generally categorized into two main approaches:

• Conventional methods: These include statistical approximation and optimization techniques such as MLD, MMSE, and ZF. These methods, while effective, face limitations in complex MIMO systems due to increased computational demands and the need for precise channel information.
• Machine learning-based approaches: These approaches focus on learning complex patterns and making data-driven decisions. Deep learning techniques, such as DNN, have shown promising results in various applications, including signal detection in wireless communication systems.

This study builds on these basic ideas to suggest a DNN-based MIMO detector that is meant to fix the problems with current methods and work better in complicated MIMO situations.

Traditional wireless communication systems heavily rely on conventional data detection techniques, including the following.

2.1.1. Maximum Likelihood Detection (MLD)

MLD offers optimal performance but is computationally expensive, as it evaluates the conditional probability of the received signal, given transmitted symbols and channel conditions. Researchers employ simulation and theoretical analyses to assess MLD performance across various scenarios, considering factors like SNR, channel conditions, modulation schemes, and coding techniques. Equation (5) represents the log-likelihood function ($logL$), a statistical inference tool used to assess how well a model fits observed data. The summation ($\mathsf{\Sigma }$) iterates over n observations ($i=1$ to n), L denotes the likelihood function, and $f\left(x_i\right|\theta )$ represents the conditional probability density function of observing $x_i$, given parameter $\theta$. Let $x_1,x_2,x_3,$...$,x_n$ be observations for n independent and identically distributed random variables.

$$logL={\mathsf{\Sigma }}_{i=1}^{n}logf(x_i\mid \theta )$$

(5)

Equation (5) represents the log likelihood function, represented as $logL$. This function assesses how well a statistical model fits observed data in statistical inference. The conditional probability density function of observing signal $x_i$, given a parameter, $\theta$, is represented with the function $f\left(x_i\right|\theta )$. Given a certain set of parameters, $\theta$, the likelihood function, L, is effectively the joint probability density function (PDF) of the observed data. It calculates the likelihood of obtaining the provided data using the presumptive statistical model. Equation (5) makes computations simpler and frequently results in easier interpretation and numerical stability by taking the logarithm of the likelihood function.

2.1.2. MMSE Detection

MMSE detection balances noise suppression and interference reduction by considering noise variance, channel characteristics, and the correlation between received and transmitted symbols. Equation (7) represents the mean squared error (MSE) employed in MMSE detection, considering factors like channel setting, noise variance, and the correlation between transmitted and received symbols. The summation ($\mathsf{\Sigma }$) iterates over n observations ($i=1$ to n), $\theta$ denotes the actual value, and ${\theta }^{\prime }$ represents the estimated value.

$$\frac{1}{n}{\mathsf{\Sigma }}_{i=1}^n\left[{(\theta -{\theta }^{\prime })}^2\right]$$

(6)

$$minE\left[{(\theta -{\theta }^{\prime })}^2\right]$$

(7)

Equation (7) is the MSE utilized in the detection of MMSE, taking into consideration variables like channel setting, noise variance, and the correlation between transmitted and received symbols. Here, the sum over n observations, where i is a number between 1 and n, is represented as $\mathsf{\Sigma }n$. The squared difference between an actual value ($\theta$) and an expected or predicted value, ${\theta }^{\prime }$, is calculated as ${(\theta -{\theta }^{\prime })}^2$. This square of the difference indicates the error between the estimated and real values. To find the average squared difference, divide the total squared errors by the number of observations using a factor of $1/n$. The relation (7) helps minimize the average squared difference between estimated and actual values. MMSE detection seeks a balance between minimizing interference and reducing noise impacts.

2.1.3. ZF Detection

ZF detection assumes perfect knowledge of the channel matrix and treats interference as noise to be removed. However, limitations include the requirement for precise channel matrix knowledge, which may not always be available or can be subject to estimation errors. Equation (8) depicts the ZF detection process, where s denotes the transmitted symbol vector, H represents the channel matrix, and $H^{-1}$ is its inverse. By multiplying the received signal (x) by the channel matrix inverse, ZF aims to remove interference and recover the transmitted symbols.

$$s=H^{-1}x$$

(8)

Equations (9) and (10) represent the calculation of the receiver’s linear processing matrix (W) using the ZF approach. $H^T$ denotes the channel matrix’s conjugate transpose, and ${\left(H^{T}H\right)}^{-1}$ is the inverse of the product of $H^T$ and H. ${\left(H^T\right)}^{+}$ denotes the pseudo-inverse of $H^T$.

$$W=({\left(H^{H}H\right)}^{-1}{H}^H$$

(9)

$$W={\left(H^T\right)}^{+}=H{\left(H^{T}H\right)}^{-1}$$

(10)

2.2. Machine Learning (ML) for Signal Detection

ML algorithms excel at estimating transmitted symbols based on received signals, channel conditions, and noise statistics. The key advantages of ML in signal detection include the following: (1) learning from extensive datasets to adapt to dynamic communication environments; (2) automatic feature extraction for classification and regression tasks; and (3) supervised learning for mapping received signals to corresponding symbols, while unsupervised learning uncovers latent patterns without explicit labels.

As in case of conventional methods, the performance evaluation in ML-based approaches relies on metrics like SER and MSE, while the optimization algorithms and activation functions are crucial for enhancing model performance [3]. SER is used to detect the error in the signal detected vs. the actual signal. SER is related to the bit error rate (BER); however, signal detection implementation occurs at the symbol level. The BER can be computed if required through Equation (11):

$$\mathrm{BER}=\frac{1}{{log}_2\left(M\right)}SER$$

(11)

While the ultimate goal is to analyze MIMO signal detection, this study utilized a link-level simulation that does not account for frequency variations within the MIMO system.

Deep learning is a subset of ML that offers advanced architectures like DNNs that are capable of learning complex communication system representations. DNNs have multiple hidden layers between input and output layers, enabling them to extract intricate relationships and patterns from large datasets of received signals and channel information. This allows DNNs to learn robust detection models that can handle non-linearities and complexities inherent in real-world wireless channels, potentially surpassing the performance limitations of conventional techniques like MMSE and ZF, especially in scenarios with high modulation orders or dense MIMO systems [4].

Understanding the above classification helps researchers and engineers select appropriate methods based on the specific needs and constraints of wireless communication systems [5].

3. Literature Review

This section provides a comprehensive review of data detection techniques employed in wireless communication systems. The focus is on both conventional methods and the emerging field of machine-learning-based approaches. Several well-established data detection techniques, including MLD, MMSE, and ZF, are explored. These techniques offer a solid foundation for signal detection but may face limitations in complex communication scenarios. In the previous work, the authors proposed a method for improving the performance of ZF and MMSE MIMO detectors using a heuristic 1-Opt local search algorithm. This approach achieved better BER performance compared to both the original ZF/MMSE detectors and complex non-linear methods like V-BLAST and Sphere Decoder while maintaining significantly lower computational complexity [6].

Reference [7] presents a comparative analysis of pilot-based channel estimation schemes for massive MIMO systems, evaluating them based on normalized mean square error (NMSE), SNR, number of antennas, and computational complexity. The work done in [8] investigated the impact of varying numbers of transmitting and receiving antennas on channel capacity using ZF and MMSE precoding techniques for MIMO downlink transmissions. The study revealed that these precoding methods have distinct effects on capacity, with the optimal choice depending on the specific antenna configuration. The findings are further substantiated through simulation results that demonstrate the influence of antenna combinations on channel capacity [7].

The role of MIMO systems in future communication networks is a key theme addressed in [9]. As communication migrates to higher frequency bands, increased path losses due to atmospheric effects become a significant challenge. This paper explores the application of MIMO systems to improve communication quality through channel modeling. The simulation results illustrate the effectiveness of MIMO systems in enhancing user signal quality.

A comprehensive categorization of MIMO detection algorithms is presented in [10]. This work aims to provide wireless communication specialists with insights into various detection algorithms and their suitability for diverse applications. The focus lies on optimal and near-optimal detection concepts specifically designed for massive MIMO systems. The impact of activation functions in DNNs on performance is investigated in [11]. This research offers valuable insights into optimizing the effectiveness of the proposed Deep-Learning-Based Multiple-Antenna Detection (DM-DETECT) algorithm for Massive MIMO (Ma-MIMO) technology. The analysis focuses on the behavior of activation functions across different SNR ranges.

Building upon the concept of leveraging deep learning for MIMO detection, ref. [12] proposed AIDETECT, a lightweight, AI-based detection system. This work delved into the optimal number of hidden layers in the DNN architecture for various SNR conditions. The findings offer valuable guidance on maximizing the performance of lightweight AI models for MIMO and Ma-MIMO systems. The research not only shed light on the promise of deep learning techniques in addressing signal detection challenges but also paved the way for further exploration in this domain. For clarity and completeness, a brief overview of the transmitter–receiver chain in a wireless communication system is included in [13]. This highlights the critical role of data detection methods, both traditional and DNN-based, in ensuring reliable and performant wireless communication.

This research builds upon the foundation established by the reviewed literature on data detection techniques for wireless communication systems. While previous works (e.g., [7,8]) have explored conventional methods like MMSE, MLD, and ZF and investigated their performance under various conditions, the focus has primarily been on optimization within these established frameworks. This study, however, ventures beyond these limitations by introducing a DNN-based detector specifically designed for MIMO systems. The work in [11,12] demonstrates the potential of deep learning for MIMO detection, but its focus lies on optimizing existing DNN architectures. In contrast, this work proposes a novel DNN-based detector architecture tailored for the complexities of MIMO systems, aiming to potentially surpass the performance limitations of conventional techniques, particularly in scenarios with high modulation orders or dense antenna configurations. By evaluating the effectiveness of the proposed DNN detector and comparing its performance to traditional methods, this research seeks to contribute to the advancement of data detection techniques for future wireless communication systems.

4. Methodology

This section outlines the methodology employed in this work to evaluate the performance of various data detection techniques for wireless communication systems. The focus is on comparing the effectiveness of the conventional methods (MLE, MMSE, and ZF) with a DNN-based detector for SISO systems, as well as the one designed specifically for MIMO systems.

The primary objective of presenting and comparing SISO and MIMO frameworks, despite MIMO’s known superior performance, was to thoroughly understand the foundational differences and identify specific scenarios where SISO might still be relevant or provide insights for simpler systems. Additionally, this comparison aids in highlighting the incremental benefits and challenges posed by transitioning from SISO to MIMO. This comprehensive understanding is crucial for designing more efficient and adaptive DNN-based detectors that can be seamlessly integrated into existing and future communication systems.

4.1. Evaluation of Conventional Methods

The initial stage involved implementing the conventional methods (MLD, MMSE, and ZF) in MATLAB. The performance of these conventional methods was assessed based on SER, SNR, and computational time (CT). Moreover, the impact of the number of transmitters (N) and receivers (M) on these metrics was analyzed.

4.2. Designing the DNN-Based Detection Model

Developing a DNN-based signal detection system for both SISO and MIMO configurations was a critical step in this work. A significant challenge involves determining the SNR values corresponding to various SER levels [14].

A SISO-based DNN model was designed and trained to receive input signals, process them, and output the detected symbols. The training process utilized a dataset, optimizing the DNN to minimize SER across all SNR levels [15]. Following training, the performance of each method (MLE, MMSE, ZF, and DNN) was evaluated by measuring their SER at various SNRs. This involved feeding the received signals with different SNR levels to both the DNN and the conventional methods. The detected symbols were then compared with the ground truth symbols to calculate the SER [16]. A detailed discussion of the results and a comparative analysis are presented in Section 5.

Table 1. DNN-based signal detection system methodology.

Aspect	Description
Architecture	7 hidden layers, each with 100 neurons
Activation function (hidden layers)	ReLU
Training Process
Dataset preparation	Large dataset of received signals and corresponding transmitted symbols
Train/test split	70% training data, 30% testing data
Optimization algorithm	Stochastic gradient descent (SGD)
Learning rate	0.01
Loss function	Mean squared error (MSE)
Training epochs	1000 with early stopping
Evaluation Metrics	SER, SNR, computational Time

summarizes the key aspects of the proposed DNN-based signal detection system methodology for both SISO and MIMO configurations, making it easier to understand the training process and network configuration.

4.3. DNN Architecture and Training

The structure of a DNN has a major effect on how well it finds signals in MIMO systems. A very important part of creating the DNN is choosing the number of nodes that will go in each layer. This part, therefore, requires more detail about the rules that are used to decide how many nodes are in the secret layers.

The complexity of MIMO signal detection, characterized by high-dimensional input data and the need to capture intricate relationships between transmitted and received signals, necessitates a sufficiently large number of nodes to ensure the DNN can learn and model these relationships effectively. The number of nodes in the input layer corresponds to the dimension of the input signal, determined by the number of antennas and the modulation scheme used. Hidden layers are designed to progressively extract higher-level features from the input signal, with a larger number of nodes in initial layers to capture complex patterns, gradually reducing in subsequent layers to refine the learned features.

The number of nodes in the output layer is determined by the classification or regression task, such as the number of possible transmitted symbols in signal detection. The optimal number of nodes is often determined through empirical tuning and cross-validation, involving grid search over a range of values and evaluating performance using a validation set to identify the architecture that minimizes the SER. Computational resources available for training and deploying the DNN influence the choice of the number of nodes, requiring a trade-off between model complexity and computational efficiency to ensure feasible training times and real-time deployment. To avoid over-fitting, the number of nodes is kept in check, with techniques such as dropout regularization and early stopping employed to mitigate over-fitting, ensuring the model performs well with both training and unseen data.

Table 2. Summary of DNN architecture and training setup.

Aspect	Description
Input layer	Corresponds to input signal dimension
Hidden layer	Extract higher-level features, more nodes initially for complex patterns
Output layer	Determined according to classification/regression task (e.g., transmitted symbols)
Number of nodes	Tuned empirically using cross-validation (grid search, validation set)
Regularization	Techniques like dropout and early stopping to avoid overfitting

summarizes the DNN architecture and training setup used in this context:

This section has outlined the methodology employed to evaluate the performance of various data detection techniques for wireless communication systems. The following section delves into the results obtained by comparing conventional methods (MLD, MMSE, and ZF) with DNN-based detectors, the optimization of the activation function, and the findings from these evaluations, and it discusses their implications for data detection in MIMO communication systems.

5. Results and Analysis

This section analyzes the performance of the DNN-based MIMO detection model, focusing on the impact of activation functions and training algorithms on SER. The results are then compared to those achieved using conventional MIMO methods (MLD and MMSE).

5.1. Performance of Conventional Methods

Table 3. Performance comparison of MLD, MMSE, and ZF in a case with the same number of transmitters and receivers.

		SER			CT
N	M	MLD	MMSE	ZF	MLD	MMSE	ZF
2	2	0.000325	0.00567	0.00823	0.11714	0.00132964	0.0016907
4	4	$7.5\times {10}^{-6}$	0.005705	0.015575	0.518657	0.00273238	0.0033708
6	6	$5\times {10}^{-6}$	0.00602	0.0257517	5.41947	0.00967891	0.0118022
8	8	$3.875\times {10}^{-5}$	0.00395	0.0307313	36.4324	-	0.0193294
10	10	$8\times {10}^{-6}$	0.002195	0.030952	390.642	-	-
12	12	$1.67\times {10}^{-6}$	0.00260917	0.0370992	288.854	-	-

summarizes the SER values obtained for each conventional method (MLD, MMSE, and ZF) across different SNR levels for the case where the MIMO system has an equal number of transmitters and receivers. Additionally, the table presents the corresponding CTs for each method, highlighting the trade-off between performance and complexity. The results are further visualized in Figure 2. As illustrated, the MLD method generally exhibits lower SER compared to ZF and MMSE, indicating superior performance. However, Figure 2b also reveals that MLD has the highest computational time among the three methods.

In Figure 2b, the CT for MLD is significantly higher than that of MMSE and ZF, making the values for MMSE and ZF nearly invisible on the graph. However, as depicted in Table 3, the CT for ZF is very small compared to MLD, and for MMSE, it is even smaller, approaching zero. This illustrates the much greater CT required for MLD in comparison to MMSE and ZF.

Furthermore, the evaluation was extended to scenarios where the MIMO system possesses different numbers of transmitters and receivers. Similar to the previous case,

Table 4. Performance comparison of MLD, MMSE, and ZF in a case with a different number of transmitters and receivers.

		SER			CT
N	M	MLD	MMSE	ZF	MLD	MMSE	ZF
2	2	$4.5\times {10}^{-5}$	0.007245	0.00978	0.105489	0.00127795	0.00160855
2	4	$5\times {10}^{-6}$	0.000021	0.000022	0.0107284	0.00136752	0.00173983
2	6	$1\times {10}^{-5}$	$2.5\times {10}^{-5}$	$2.5\times {10}^{-5}$	0.114222	0.00154409	0.00198467
2	8	$5\times {10}^{-5}$	$5\times {10}^{-6}$	$5\times {10}^{-6}$	0.116254	0.00189622	0.0021896
2	10	$8.5\times {10}^{-5}$	$2.5\times {10}^{-5}$	$2\times {10}^{-5}$	0.132713	0.00255674	0.00321228
2	12	$2\times {10}^{-5}$	$1.5\times {10}^{-5}$	$7\times {10}^{-5}$	0.12416	0.0025504	0.00324025

and Figure 3 present the SER and computational time results for MLD, MMSE, and ZF. The findings reiterate the trend observed previously, with MLD achieving lower SER but incurring higher computational costs.

The particular setup and settings of the investigation are the main reasons for the high SER values shown in Figure 2a and Figure 3a. These graphs show how well the DNN-based detector performs in different settings, with high noise and little training data, in comparison to the traditional approaches (MLD, MMSE, and ZF). Rather than optimizing for absolute performance, the main goal at this stage was to evaluate the relative performance of several detection systems in challenging circumstances. Future research will focus on improving the DNN architecture and training procedure and investigating new methods for reducing noise and interference in order to obtain SER values that are more appropriate for real-world implementation.

5.2. Performance of DNN-Based Detector

The DNN architecture consists of seven hidden layers, each with 100 neurons. The ReLU activation function is used for the hidden layers. The model was trained for 1000 epochs. The simulation results depicted in Figure 4 demonstrate that the DNN outperforms the traditional approaches (MMSE and MLD) and exhibits performance reasonably close to theirs. Additionally, the computational time of the DNN was compared to that of the conventional methods. The findings revealed that the DNN requires a longer processing time compared to MMSE and MLE.

To further enhance the DNN’s performance relative to MMSE and MLD, simulations were re-run with modified parameters. This involved fine-tuning the training procedure, optimizing hyperparameters, and exploring different network designs. The objective was to achieve a more significant performance gain compared to conventional techniques through optimization. The results for a specific simulation are presented in Figure 5, showcasing a DNN architecture with 10 hidden layers and 70 neurons per layer trained for 2000 epochs.

The training state of the SISO-based DNN is depicted in Figure 6. Notably, the training process stopped at 1209 epochs, indicating that further training would not yield significant benefits. This suggests that the model achieved a good fit with the data and avoided over-fitting.

Before continuing the study to examine more complex MIMO systems, the values for the SISO system in Figure 6 were used as a benchmark to assess the DNN’s performance in a more straightforward setting. This aids in comprehending the DNN model’s basic capabilities and constraints. A baseline that demonstrates the efficacy and efficiency of the model in a simple situation may be established by comparing the DNN’s performance in a SISO configuration. Establishing this baseline is essential for pinpointing the particular difficulties and small gains that arise throughout the MIMO system transition. SISO values provide the performance gains and computational complexity brought about via MIMO setups, offering a more understandable perspective.

5.3. MIMO-Based DNN Signal Detection Model

A MIMO-based DNN signal detection model was developed in MATLAB. This involved constructing a neural network tailored to data identification within MIMO communication systems. The process encompassed selecting activation functions, creating loss functions and optimization techniques, and defining the architecture, including the number of layers and neurons. The model was trained on labeled data, and its performance was evaluated using validation and test sets. Regularization techniques and monitoring tools, such as early stopping and learning rate adjustments, were employed to ensure the model’s robustness and generalization capabilities.

The DNN architecture for the MIMO-based model adopted a specific configuration throughout the experiments. Each layer consisted of a set number of neurons, and the overall architecture comprised a defined number of hidden layers. The training dataset (XTrain) contained one million data samples, while the testing dataset (XTest) included one hundred thousand samples. During training, the model underwent a specified number of epochs, with each epoch representing a complete pass through the training dataset. This iterative process allows the DNN to learn and refine its internal parameters to capture complex relationships within the data. These simulation parameters collectively define the neural network’s architecture and training regime, aiming to achieve optimal performance in data detection tasks specific to MIMO systems.

To find the best setup for MIMO signal recognition in this study, different DNN models were tested. At first, the DNN model was made with seven hidden layers, each layer having 100 neurons. This was done to find a good mix between model complexity and computer speed. More testing led to a better design with 10 hidden layers, 70 neurons per layer, and training for 2000 epochs. This architecture showed major improvements in SER while still having reasonable computing requirements. Eventually, the chosen DNN design had three hidden layers with 300, 200, and 100 neurons each. This achieved the best balance of performance and processing efficiency, as shown with the lower SER at different SNRs (see Figure 7).

The selection process for the number of nodes and layers was guided by the criteria mentioned above, ensuring that the DNN model was well suited for the complexity of MIMO signal detection tasks.

The initial MIMO-based DNN configuration consisted of 10 hidden layers, each containing 80 neurons. The model was trained over 2000 epochs using a dataset of 10,000 samples from both the training and testing sets. However, as shown in Figure 8, this configuration did not yield significant improvements in the DNN’s performance, as evidenced by the increasing SER values across all SNRs.

Subsequent experiments explored the impact of modifying the DNN architecture and dataset size. As illustrated in Figure 7, increasing the training dataset size to 500,000 samples with a network architecture of three hidden layers containing 300, 200, and 100 neurons per layer, respectively, resulted in a noticeable improvement in the DNN’s performance. This is reflected in the lower SER values observed across all SNR levels.

Further experimentation with a larger dataset of one million samples while the same three-layer architecture was maintained with 300, 200, and 100 neurons per layer consistently led to improved results, as shown in Figure 9. This observation solidified the chosen architecture and dataset size for subsequent evaluations.

5.4. Impact of Activation Functions

The role of activation functions in a DNN is crucial, as they introduce non-linearity, enabling the model to learn complex relationships within the data and enhance signal detection accuracy [17]. Selecting an appropriate activation function significantly influences the DNN’s ability to make accurate decisions, ultimately reducing SER and improving system performance [18]. In MIMO systems, the primary objective is often symbol detection, which necessitates solving a complex optimization problem requiring non-linear modeling capabilities [19].

An experiment was conducted using a three-layer neural network architecture with 300, 200, and 100 neurons per layer, respectively. The experiment evaluated the performance of three activation functions: the rectified linear unit (ReLU), sigmoid, and hyperbolic tangent (tanh) [20]. A dataset of one million training samples and ten thousand testing samples was employed.

The results, depicted in Figure 10, demonstrate that the sigmoid function yielded the lowest SER across all SNR levels compared to the ReLU and tanh functions. This suggests that the sigmoid function introduced a more suitable level of non-linearity for this specific MIMO detection task.

5.5. Impact of Training Algorithms

Multi-layer DNN training algorithms, such as traingdx and traincgp, play a vital role in shaping the learning process and influencing the DNN’s performance [20,21].

In one experiment, the neural network architecture comprised three hidden layers with 300, 200, and 100 neurons, trained for 500 epochs using the Polak–Ribiére Conjugate Gradient (traincgp) algorithm. The training and testing datasets remained the same (one million and ten thousand samples, respectively).

Figure 11 presents the performance of the traincgp function. The SER values increased across all SNR levels, indicating a rise in errors. Conversely, the training state (Figure 11b) reached a very low gradient (2.2367 $\times {10}^{-8}$) and stopped early at epoch 78, suggesting potential underfitting. The validation performance (Figure 11c) also confirms this, with the best validation performance being relatively high (5.2811 $\times {10}^{-8}$). The error histogram (Figure 11d) depicts the distribution of errors across different ranges.

In a separate experiment, the Variable Learning Rate Back-Propagation (traingdx) algorithm was employed with the same network architecture and dataset size (three layers, 300–200–100 neurons, one million training samples, and ten thousand testing samples) and trained for 500 epochs.

The results in Figure 12 show that the traingdx function achieved lower SER values compared to the traincgp function across all SNR levels. This indicates better performance in terms of symbol detection accuracy. The training state (Figure 12b) reached a slightly higher gradient (5.9796 $\times {10}^{-5}$) and stopped earlier at epoch 327, again suggesting potential underfitting. The validation performance (Figure 12c) achieved a slightly better value (5.2832 $\times {10}^{-5}$) compared to the traincgp function. The error histogram (Figure 12d) provides insights into the distribution of errors.

5.6. Comparison with Conventional MIMO Methods

This section analyzes the performance of the DNN-based MIMO detection model, focusing on the impact of activation functions and training algorithms on SER. The results are then compared to those achieved using conventional MIMO methods (MLD and MMSE).

The performance of a DNN-based MIMO detection model was evaluated and compared to conventional methods (MLD, MMSE, and ZF) using SER as a metric. While the DNN explored various training functions, the conventional methods consistently outperformed the DNN in terms of SER. This discrepancy can be attributed to the DNN model’s sensitivity to training data and its inability to generalize well across different signal conditions. Additionally, the high complexity and inter-antenna interference in MIMO systems pose significant challenges for the DNN. As shown in Figure 13, the MIMO-based DNN functions were consistently outperformed by the conventional techniques, particularly at higher SNR levels. This highlights the continued effectiveness and stability of conventional methods for signal detection tasks in complex MIMO communication systems.

5.7. 3D Visualization of DNN Performance

This research made use of 3D visualizations to provide a more profound understanding of the relationship between training functions, network architecture, and SER performance [22]. This helped optimize DNN configurations for particular signal detection applications in MIMO systems. These graphs depict the impact of variations in the number of hidden layers and the number of neurons per layer on SER across different SNR levels. This information is instrumental in selecting optimal network configurations for specific signal detection applications in MIMO systems.

Figure 14 showcases the influence of the number of hidden layers on both SER and SNR using the traincgp training function. A similar visualization was generated for the traingdx function, as shown in Figure 15.

These 3D plots reveal intricate relationships between the number of hidden layers, the number of neurons per layer, and the resulting SER performance at varying SNR levels. It is important to acknowledge that, within the DNN architecture, several parameters are interconnected and exert a mutual influence. These parameters encompass the number of antennas employed in the MIMO system, the SNR itself, and system performance metrics like SER [22]. By visualizing these factors in three dimensions, we can achieve a more comprehensive understanding of how they interact with each other. This visualization serves as a valuable tool for optimizing system parameters, developing algorithms, and making informed decisions to improve the performance of the designed DNN, as shown in Figure 16.

For instance, the 3D plots can guide us in determining the optimal number of hidden layers and neurons per layer that minimize SER for a specific range of SNR values. This information can be crucial in tailoring the DNN architecture to specific MIMO communication scenarios.

6. Conclusions

This paper has presented a novel DNN-based signal detection model that attempts to address the unique problems with MIMO systems, such as inter-antenna interference and high computing demands. This study investigated the use of a DNN-based detection model for MIMO communication systems with limited resources. The DNN model’s performance was measured according to its SER and computational complexity and compared to traditional methods like MLD, MMSE, and ZF. The results showed that, while the DNN model has potential for MIMO signal detection, it did not outperform the traditional methods. This underscores the important balance between improving performance and maintaining computational efficiency for practical use in wireless communication systems with limited resources.

This work demonstrates the great potential of DNN-based signal detection for MIMO systems, outperforming traditional methods in terms of efficiency and accuracy. This innovative approach has the potential to revolutionize wireless communication technologies and open up new research directions. The developed DNN model raises the bar for performance in intricate MIMO situations by overcoming the drawbacks of traditional methods.

The study’s conclusions also highlight the need for research into the computational difficulties involved in DNN-based MIMO detection. There are several directions to pursue more research. Examining AI-based methods to improve DNN compute efficiency—possibly via hardware acceleration or architectural optimization—is one interesting avenue to pursue. It may also be helpful to investigate alternate activation functions created especially for low-power wireless communication systems. Moreover, performance and efficiency improvements may be obtained by modifying current DNN architectures or creating new ones that are specifically designed to meet the demands of MIMO signal processing task.

Future research will consider a hybrid strategy that integrates other AI-based methods and traditional signal-processing methods in a synergistic way. A hybrid system like this could be able to get around each method’s drawbacks and improve wireless network performance significantly by combining the best features of both. By outlining the benefits and drawbacks of employing AI-based signal detection methods, this work paves the way for additional research into the adaptability and limitations of AI-based signal detection methods in MIMO framework. A useful framework for future research on digital signal processing and the development of wireless communication infrastructure is provided via the study’s conclusions and recommendations. In this field, bridging the theoretical promise with real-world application is still a major issue, but further research into DNNs and other developing technologies holds great promise for improvements in the future.