High-Speed Parallel Neural Network Equalizer Based on Hard Decision

Abstract

This paper proposes a novel three-layer parallel deep neural network (DNN) equalizer that achieves superior performance and resource efficiency compared to conventional decision feedback equalizers (DFEs). Evaluated in a 56 GBd PAM4 transmission system over a 306 mm channel, the proposed equalizer reduces the bit error rate (BER) by an order of magnitude relative to a first-order DFE while matching the performance of a second-order DFE, all with comparable computational complexity. Notably, the parallel architecture accelerates equalization processing without compromising BER performance and fundamentally eliminates the inherent feedback delay element in DFE structures, thereby overcoming a key timing bottleneck of traditional equalizers. Simulation results demonstrate that under similar power constraints, the DNN equalizer achieves higher throughput than its second-order DFE counterpart, highlighting its significant advantages in resource utilization efficiency.

1. Introduction

The rapid development of information technology has driven the emergence of big data, cloud computing, and artificial intelligence, leading to surging demand for large-scale data center construction and higher requirements for high-speed, reliable data transmission capabilities. Against this backdrop, the importance of high-performance channel equalization technology has become increasingly prominent. Although traditional equalizers, such as DFEs [1,2,3] are widely used, they still exhibit significant limitations. DFEs are particularly vulnerable to burst errors, where a single misjudged signal can trigger consecutive error propagation in scenarios with severe inter-symbol interference. Additionally, the structural characteristic of DFEs, which relies on feedback loops combined with delay constraints in the forward path, imposes limitations on transmission rate improvement.

In recent years, researchers have proposed a series of improvement schemes to address DFE limitations. For example, Cheng et al. proposed a low-power 16 Gbps equalizer employing a single adaptive system, which implements DFEs using low-power techniques to enhance energy efficiency. Its data path includes a continuous-time linear equalizer (CTLE) [4,5,6] and a two-tap DFE [7]. Xing et al. proposed a novel adaptive DFE based on the light pattern protection (LPP) algorithm, which ensures consistent convergence direction even under extreme data flow conditions and maintains convergence speed during equalization [8]. Feng’s team developed a high-performance PAM-4 digital equalizer for 64 Gb/s backplane I/O interfaces, featuring a tap-configurable feedforward equalizer (FFE) [9,10,11] and a partially expanded DFE. The DFE processes the last two post-cursors, while the FFE further suppresses noise and crosstalk. The configurable FFE taps enable better adaptation to different channel types [12]. Ahn et al. presented a filtered FFE scheme that shortens the channel impulse response to 2 unit intervals, enabling efficient data recovery with a 1-tap DFE [13]. Despite these improvements, such approaches often introduce higher complexity, computational cost, and potential instability, particularly in time-varying channels.

The application of neural networks in signal equalization represents a paradigm shift in this field. Recent research has begun exploring the potential of neural networks to overcome the inherent limitations of traditional equalizers. For instance, Lilan Wei proposed a deep learning-based equalizer capable of real-time adaptation to channel conditions, significantly improving BER performance [14]. Q Liu et al. investigated a convolutional neural network (CNN)-based blind equalizer for shortwave channels, finding that the convolutional neural network equalizer (CNNE) achieves lower BER than recurrent neural network equalizers (RNNE). Moreover, the characteristics of the classification layer enhance equalization performance at the same Signal-to-Noise Ratio while reducing CNN structural complexity [15]. S Lavania et al. studied neural network-based adaptive channel equalization through simulation analysis of BPSK signal transmission over multiple SUI channels (modeled for fixed wireless applications), demonstrating that RNN equalizers consistently outperform MLP equalizers in BER performance [16]. These studies collectively demonstrate that neural networks offer significant advantages in equalizer construction, particularly in handling complex nonlinear channel distortions, adapting to time-varying conditions, and achieving superior performance with efficient architectural designs.

With advancements in intelligent computing, modern communication systems impose stringent requirements on latency and bit error rate. For instance, Gao et al. proposed a CNN-based time-domain equalizer for visible light communication (VLC) systems, demonstrating significant BER performance improvements [17]. Numerical simulations showed that at 20 dB SNR with 16/32/64-QAM modulation, the equalizer reduces BER from approximately 0.5 to below 1 × 10⁻⁵, well beneath the forward error correction (FEC) limit of 3.8 × 10⁻³.

Neural networks possess tremendous potential for learning from complex channel behavior in a data-driven manner, offering significant advantages. Through training, neural networks can adapt to various channel conditions, providing flexible and robust solutions for equalization challenges. However, the training process requires large amounts of labeled data, and the inference stage demands substantial computational resources, posing challenges for real-time applications.

Despite these challenges, the integration of neural networks in signal equalization offers unique advantages. Their ability to learn from data and adapt to new environments enables them to address various signal distortion problems. Additionally, the parallel processing capability of neural networks can significantly reduce equalization time, making them highly suitable for high-speed data transmission applications.

This paper presents a novel deep parallel neural network equalizer architecture. The equalizer employs a deeply parallel architecture to achieve rapid convergence while incorporating a hard-decision mechanism to maintain hardware-friendly characteristics, significantly reducing implementation costs while improving processing efficiency. Compared to conventional solutions, this design adaptively learns channel characteristics through a data-driven methodology without requiring precise mathematical modeling, effectively compensating for nonlinear distortions in short-distance transmission while maintaining low computational complexity. Simulation results demonstrate that in a 306-mm channel environment, the DNN [18,19,20,21,22,23] equalizer achieves a bit error rate one order of magnitude lower than the first-order DFE (feedback taps is 1) while matching the equalization performance of the second-order DFE (feedback taps is 2), providing an innovative solution that combines high performance and low complexity for data center short-distance interconnects.

Compared with existing neural network equalizers, the proposed architecture offers three key innovations: (1) a parallel input structure that eliminates the feedback timing constraint inherent in DFEs; (2) a hard-decision-oriented output mapping compatible with PAM4 signaling, enabling efficient hardware implementation; and (3) a lightweight three-layer design that achieves second-order DFE performance with significantly reduced resource usage.

2. Deep Neural Network Equalization Technology

The DNN equalizer architecture developed in this study is illustrated in Figure 1a. The network structure is defined by the number of layers and neurons in each layer. We employ a three-layer neural network architecture where the first layer serves as the input layer with N neurons, the second layer is a hidden layer containing n neurons, and the final layer is the output layer with $n_{main}$ neurons. This configuration can be expressed as:

$$N=n_{pre}+n_{main}+n_{post}$$

(1)

where $n_{pre}$ represents the number of pre-cursor neurons, $n_{main}$ denotes the parallelism degree, and $n_{post}$ indicates the number of post-cursor neurons.

Figure 1. (a) Parallelized neural network equalizer architecture. (b) Single-input neural network equalizer architecture.

Figure 1b illustrates a single-input neural network architecture capable only of single-path data transmission, unlike the parallel signal transmission capability shown in Figure 1a. The key advantage of Figure 1a over Figure 1b is that with the same number of input layer neurons, the parallel architecture supports multiple simultaneous transmissions, significantly reducing transmission time.

The adoption of a three-layer DNN architecture is grounded in both theoretical and practical considerations. While a single hidden layer structure theoretically possesses sufficient channel response inversion capability when adequately expanded, such configuration often leads to parameter inefficiency and overfitting risks. Our three-layer design enables more efficient hierarchical feature extraction: the first hidden layer captures fundamental distortion characteristics, while subsequent layers perform nonlinear combination of these features for final decision-making. This architecture achieves superior balance between model capacity, training stability, and generalization performance compared to an excessively wide single-layer network. Simulation results demonstrate that our approach maintains competitive computational efficiency while achieving bit error rate performance comparable to second-order DFEs, providing reliable signal recovery assurance for high-speed communication systems.

The network receives parallel quantized digital signals from the ADC module into the input layer. To enhance detection accuracy, the equalizer simultaneously processes $n_{main}$ input signals as $n_{pre}$ pre-cursor and $n_{post}$ post-cursor signal components. Due to the fully connected nature between layers, intermediate symbols incorporate more pre- and post-cursor information than explicitly defined by $n_{pre}$ and $n_{post}$. The output layer, as the final network layer, contains $n_{main}$ neurons to efficiently process multiple parallel signals leveraging the neural network’s parallelized structure.

The hidden and output layers can be viewed as arrays of neurons. Activations are computed by summing weighted inputs and bias terms, followed by application of a nonlinear function $\sigma \left(x\right)$ to obtain the neuron output y.

$$y=\sigma (\sum _i{w}_i{x}_i+b)=\sigma (W^{T}X+b)=\sigma \left(z\right)$$

(2)

$$z=\sum _i{w}_i{x}_i+b=W^{T}X+b$$

(3)

where $W=w_i$ is the weight vector, $X=x_i$ represents the input vector, b denotes the bias term, and the superscript T indicates matrix or vector transpose. For the jth neuron in the m-layer, its output $a_j^m$ is calculated as:

$$a_j^m=\sigma (\sum _k{w}_{jk}^{m-1}{a}_j^{m-1}+b_j^{m-1})=\sigma \left(z_j^m\right),\left(m=2,3\right)$$

(4)

In vector form, this expression becomes:

$${\mathbf{a}}^m=\sigma ({\mathbf{w}}^{m-1}\ast {\mathbf{a}}^{m-1}+{\mathbf{b}}^{m-1})=\sigma \left({\mathbf{z}}^m\right),(m=2,3)$$

(5)

where $w^{m-1}=w_{jk}^{m-1}$ represents the weight matrix of the mth layer and $b^{m-1}=b_j^{m-1}$ is the corresponding bias vector. The nonlinear function $\sigma \left(x\right)$, known as the activation function, enables neural networks to handle highly complex data representations beyond simple linear mappings. For our three-layer DNN structure, we employ the ReLU function in the hidden layer and the Sigmoid function in the output layer.

The activation functions used in this study are defined as follows.

$${\sigma }_1\left(x\right)=\left\{\begin{array}{cc}x,& x\ge 0\\ 0,& x<0\end{array}\right.$$

(6)

$${\sigma }_2\left(x\right)=1/(1+e^{-x})$$

(7)

The choice of activation functions directly impacts both engineering implementation complexity and system performance. While linear activation functions offer computational simplicity, they cannot effectively handle nonlinear distortions in communication channels. This study adopts an engineering practical nonlinear activation function scheme: the hidden layer employs the ReLU function, whose piecewise linear characteristics ensure nonlinear representation capability while facilitating hardware implementation; the output layer utilizes the Sigmoid function to compress output values to the $(0,1)$ range, directly meeting the requirements of subsequent decision circuits. This scheme requires only minimal logic resources in hardware implementation, maintaining nonlinear processing capability while satisfying the low-latency demands of real-time processing, thereby providing an optimized solution for hardware deployment of equalizers.

After the DNN signal output, each symbol is determined by a decision function. In this study, the PAM4 modulation format is adopted, with symbol levels set to $\{1,-1,3,-3\}$. This level selection is based on the principle of equal minimum Euclidean distance between symbols, where the spacing between adjacent symbols is uniformly 2, ensuring balanced bit error performance under additive white Gaussian noise channels. Since the output layer of the DNN employs a Sigmoid activation function, the output values are constrained to the interval $[0,1]$. To map the continuous outputs to discrete symbols, the decision thresholds are set based on the principle of uniform quantization, as defined in Equation (8):

$$PAM4=\left\{\begin{array}{cc}0,& a<1/6\\ 1,& 1/6\le a<1/2\\ 2,& 1/2\le a<5/6\\ 3,& a\ge 5/6\end{array}\right.$$

(8)

where a represents the output value of the neural network.

These thresholds evenly divide the output interval into four subintervals, each corresponding to one of the four PAM4 symbols. In hardware implementation, the two most significant bits of the DNN output value are directly taken to efficiently map the four decision results of 0, 1, 2, and 3. This mapping corresponds to natural binary coding, eliminating the need for additional multiplication or comparison operations and effectively reducing hardware overhead.

In the BER calculation, natural binary coding is used to map binary bits to PAM4 symbols. Compared with NRZ coding, PAM4 modulation requires only half the baud rate and Nyquist frequency to achieve the same data transmission rate. Based on the three-layer DNN structure, a hard-decision detection scheme is adopted, which significantly reduces computational overhead by simplifying the hardware architecture, thereby providing a solid foundation for hardware implementation.

In this work, the neural network is trained using the Mean Squared Error (MSE) as the loss function to quantify the discrepancy between the network output ${\mathbf{a}}^{\left[3\right]}$ and the expected output ${\mathbf{y}}_{\exp }$, as shown in Equation (9):

$$C={\displaystyle \frac{1}{2}}{∥{\mathbf{a}}^{\left[3\right]}\left(x\right)-{\mathbf{y}}_{\exp }\left(x\right)∥}^2$$

(9)

where ${\parallel ·\parallel }^2$ denotes the squared Euclidean norm. To minimize the loss function, the backpropagation algorithm is employed to compute the gradients of the network parameters, and the stochastic gradient descent (SGD) algorithm is used for parameter updates [24]. The update rules for weights and biases are given in Equations (10) and (11):

$$w^{m-1}=\left(1-\eta ·{\displaystyle \frac{\lambda }{n}}\right)w^{m-1}-\eta ·{\displaystyle \frac{\partial C}{\partial w_{jk}^{m-1}}},(m=2,3)$$

(10)

$$b^{m-1}=b^{m-1}-\eta ·{\displaystyle \frac{\partial C}{\partial b_j^{m-1}}},(m=2,3)$$

(11)

During training, the learning rate $\eta$ is fixed at $0.051$, and the regularization parameter $\lambda$ is set to $0.88$ for L2 regularization to prevent overfitting. The model is trained using mini-batches with a batch size of 1024, i.e., each batch contains 1024 samples, and each sample consists of $n_{main}$ features.

The training adopts an early stopping mechanism to control the number of iterations. After each training epoch, the bit error rate (BER) on the validation set is computed. Training terminates when the validation BER falls below a preset threshold or when the number of epochs reaches a preset upper limit.

3. Experimental Simulation

The experimental platform utilized in this study is MATLAB R2020b. The experimental procedure involves signal synchronization using a pseudo-random bit sequence (PRBS). The complete simulation setup is illustrated in Figure 2.

Figure 2. Setup and structure of the simulation system.

In this study, a channel model is established based on a four-port S-parameter file (306mm.s4p) (Figure 3). The S-parameter data are first read, and frequency-domain analysis is conducted to characterize the channel’s insertion loss ($S_{21}$) and return loss ($S_{11}$). A rational function approximation is then applied to fit the S₂₁ parameter, with the delay factor set to 0.98 to enhance model stability, resulting in a pole-residue frequency-domain analytical model of the channel. Subsequently, an arbitrary input signal is combined with the obtained rational function model, and time-domain response simulation is performed to compute the output waveform after the signal traverses the channel. This approach enables accurate time-domain modeling and signal integrity assessment for high-speed interconnect channels.

Figure 3. Insertion loss and return loss of simulated channel with a 306 mm Megtron7N PCB traces and two connectors.

As shown in the scatter plots and eye diagrams in Figure 4, Figure 5 and Figure 6, high-frequency insertion loss causes severe degradation of transmission signal quality. Specifically, Figure 4 displays the scatter plot of the original PAM4 signal, while Figure 5 and Figure 6 clearly demonstrate the significant signal distortion after channel transmission. To address the challenges of time-varying channel characteristics, this study employs an online adaptive DNN equalizer that utilizes decision-generated labels as training data to achieve real-time fine-tuning of network parameters. This decision-directed adaptive mechanism enables the equalizer to dynamically track channel variations, effectively compensate for signal distortion, and thus maintain optimal equalization performance consistently.

Figure 4. Input Signal Scatter Plot.

Figure 5. Output signal scatter plot.

Figure 6. Output eye diagram of the signal.

In tests with 56 GBd PAM4 signals (Figure 7), the DNN equalizer configured with only one degree of parallelism and one hidden neuron achieves a bit error rate two orders of magnitude lower than first-order DFEs with an equivalent number of taps, while performing on par with second-order DFEs. This indicates that even with a minimal parallel structure and simple hidden layer design, the DNN equalizer can reach the equalization level of second-order DFEs.

Figure 7. The convergence process of 56GBd PAM4 encoding, DNN with first-order DFE and second-order DFE at the same number of taps.

Compared to traditional DFEs, the proposed parallel DNN equalizer not only eliminates the timing constraints associated with feedback loops, theoretically supporting unlimited channel rates, but also achieves higher resource utilization efficiency and faster convergence speed through its parallel architecture, providing an effective solution for high-speed data communication systems.

Based on the simulation results presented above, the proposed parallel DNN equalizer demonstrates significant advantages in high-speed channel transmission. As shown in Figure 8, the equalizer successfully achieves effective equalization of 64-bit data streams, addressing the issue of information insufficiency during equalization through increased parallelism while substantially improving processing speed. Figure 8b indicates that at a parallelism degree of 64, the minimum bit error rate reaches the order of ${10}^{-6}$, comparable to the performance of second-order DFEs. This confirms that even under high parallelism conditions, excellent equalization performance can be maintained through structural optimization.

Figure 8. (a) The convergence process of 56GBd PAM4 encoding and DNN at different parallelisms. (b) Violin plot with low BER.

The comparison of convergence characteristics in Figure 8 and Figure 9 shows that the parallel input equalizer supports simultaneous processing of multiple signals and achieves a significantly faster convergence speed than the single-input structure. The simulation results indicate that the 64-bit parallel configuration converges at 35 ms, whereas the single-input counterpart converges at 40 ms; therefore, the 64-bit parallel configuration converges 5 ms faster than the single-input structure. Comparing the 16-bit parallel structure with the single-input structure, it not only converges faster but also requires only 34 hidden layer neurons, while the single input requires 64 neurons, reducing the number of neurons needed in the hidden layer by 30. These results highlight the dual advantages of the parallel scheme in resource utilization and convergence speed, demonstrating its potential for high-speed hardware implementation under practical constraints such as inference latency and memory bandwidth.

Figure 9. (a) The convergence process of 56GBd PAM4 encoding and DNN under a parallelism of 1. (b) Violin plot with low BER.

4. Conclusions

This paper has presented a high-speed channel equalizer based on a parallel three-layer DNN. Simulation results confirm that the proposed equalizer achieves a bit error rate comparable to a second-order DFE in a 56 GBd PAM4 system, while its parallel architecture effectively overcomes the timing constraints of DFE feedback loops and significantly boosts processing throughput. Furthermore, the design demonstrates superior convergence speed and resource utilization compared to single-input DNN equalizers. These attributes make it a highly promising solution for next-generation high-speed data links.

Despite the promising results, several challenges remain for practical deployment. Future work will prioritize the hardware implementation of the equalizer, focusing on optimizing its computational complexity and power efficiency. We also plan to explore more advanced adaptive algorithms to enhance its robustness in rapidly time-varying channels. Investigating hybrid architectures that fuse neural networks with classical signal processing techniques may offer a path to further reduce computational overhead. Finally, extending the approach to support higher-order modulation formats and multi-channel scenarios will be a critical step toward widespread adoption.