A High-Accuracy Normalization Unit Using Multi-Bit Random Variables

Abstract

Stochastic computing (SC) has the characteristics of low complexity and is expected to solve the bottleneck problem of conventional binary computing. Stochastic normalization units are widely used in stochastic decoding and stochastic signal detection, and have achieved hardware efficiency far exceeding conventional methods. However, they also have problems such as 1-bit representation and low calculation accuracy, calculation overflow, and fluctuation in the sum of normalized probabilities, which lead to prolonged processing latency and degraded hardware efficiency. Thus, this paper proposes a novel stochastic normalization unit using multi-bit random variables. Benefiting from the high representation accuracy of multi-bit random variables, the accuracy of the proposed unit is greatly improved. Meanwhile, the proposed unit completely solves the problems of calculation overflow and fluctuation in the sum of normalized probabilities. Simulation results show that the proposed 3-bit unit achieves a fourfold improvement in convergence speed and 2 times higher hardware efficiency compared to the state-of-the-art stochastic normalization unit. Finally, we verify that the proposed 3-bit unit demonstrates a 75% improvement in hardware efficiency for stochastic sparse-code multiple-access (SCMA) detection.

1. Introduction

As a novel computing paradigm, stochastic computing (SC) [1] has the characteristics of low complexity and is expected to solve the bottleneck problem of conventional binary computing [2]. Stochastic computing is a unary representation and computing system that represents a number with an unweighted stochastic bit stream, where the proportion of 1s in the stream represents the numerical value. The most attractive feature of stochastic computing is that complex mathematical operations in conventional binary computing systems can be implemented with basic logic gates. For linear operations, AND gates and multiplexers (MUXs) can conduct multiplication and multiply–accumulate (MAC) operations, respectively [3]. For nonlinear operations, the JK flip-flop can be used to implement a normalization operation of two inputs [4].

Stochastic normalization units are widely used in stochastic decoding [4,5,6,7] and stochastic signal detection [8,9]. Although there have been many works aiming to improve their convergence speed and accuracy, stochastic normalization units still face a long bit-stream length due to the limitation of the representation ability of stochastic bits [10,11]. Another problem is that non-scaling addition between stochastic bits is difficult to implement, which can lead to calculation overflow and loss of precision, while the latter can lead to fluctuation in the sum of normalized probabilities. The Integral Stochastic Computing (ISC) [12] and amplitude and frequency encoding (AFE) for SC [13] demonstrate a promising way to further improve the representation and calculation accuracy by using multi-bit random variables. ISC and AFE proved that multi-bit random variables can be used in addition, multiplication, and other calculations, which provides the stochastic normalization unit a way to further improve accuracy.

In this study, we improve the accuracy and convergence speed of the stochastic normalization unit by using the idea of multi-bit random variables, and solve the problems of calculation overflow and fluctuation in the sum of normalized probabilities. Simulation results demonstrate that our 3-bit implementation achieves fourfold improvement in convergence speed and doubles the hardware efficiency compared to conventional stochastic normalization units. Furthermore, hardware implementation for stochastic sparse-code multiple-access (SCMA) detection shows a 75% improvement in hardware efficiency, validating the practical benefits of our approach.

The rest of this paper is organized as follows: Section 2 briefly reviews the preliminaries of existing stochastic normalization units and multi-bit random variables. Section 3 describes the proposed stochastic normalization unit and the further improvements. The computing and hardware performance will be presented in Section 4. Finally, the paper concludes in Section 5.

2. Preliminaries

2.1. Existing Stochastic Normalization Units

In stochastic computing (SC), a numerical value x ($0\le x\le 1$) is represented by the proportion of ‘1’s in a stochastic bit stream, with each bit generated by a binary random variable X. Such streams are commonly produced using a comparator and a random number source (RNS), as shown in Figure 1a. A major benefit of SC lies in its ability to perform complex arithmetic operations using simple logic circuits. For example, multiplying two independent stochastic bit streams can be achieved efficiently with a single AND gate, while more intricate multiply–accumulate (MAC) operations can be implemented using a multiplexer (MUX) architecture, as illustrated in Figure 1b and Figure 1c, respectively.

To implement the normalization function in stochastic computing (SC), we first require a division operation. SC dividers are typically implemented as sequential circuits using Markov processes, as illustrated in Figure 2. A classic approach employs a JK flip-flop, which inherently performs normalization on two input streams [14]. Alternatively, counter-based dividers [15] have been proposed to compute binary division results, where multi-bit tracking enables precise division before reconversion to stochastic bit streams.

The normalization unit for M inputs can be constructed using the aforementioned divider combined with linear operations. A straightforward approach extends the JK flip-flop design (Figure 3a), though this cannot directly output binary-normalized probabilities. Alternatively, counter-based normalization (Figure 3b) tracks probabilities explicitly [16]. While both methods employ MUX-based scaled addition to prevent overflow, they sacrifice computational accuracy.

The joint probability tracking (JPT) method [7,9] improves precision through non-scaled addition and tracking forecast memory (TFM). To address overflow, JPT records overflow counts and compensates by skipping updates (e.g., canceling four updates as shown in Figure 4). However, this effectively shortens the bit-stream length, degrading representation and computational accuracy.

In summary, due to the nature of stochastic bits, both scaling and non-scaling addition will result in a loss of precision. Moreover, the reduction in precision will also result in fluctuations in the sum of normalized probabilities. To solve the above problems, a new computing and representation system is required.

2.2. Multi-Bit Random Variables

Conventional SC is represented and calculated using 1-bit random variables. The Integral Stochastic Computing (ISC) [12] and amplitude and frequency encoding (AFE) for SC [13] demonstrate a promising way to further improve the representation and calculation accuracy by using multi-bit random variables, as shown in Figure 5a and Figure 5b, respectively. In multi-bit random variables, for a number x, its multi-bit random variable X should satisfy $E(X)=x$ as well. ISC and AFE proved that multi-bit random variables can be used in addition, multiplication, and other calculations, as shown in Figure 6, which provides the stochastic normalization unit a way to further improve accuracy.

3. Joint Normalization Unit for Multi-Bit Random Variable

3.1. High-Precision Multi-Bit Random Variable Generation

For higher representation accuracy, we propose applying a W-bit binary number to an n-bit random variable generator as shown in Figure 7a. We retain the highest $n-1$ bits, generate the probability bit of the lower bit, and add it to the higher bit to get the n-bit random variable. Figure 7c shows the mean square error (MSE) when different methods are used to represent different values of x. Results show that the MSE of the proposed method is much smaller than that of stochastic bit and other multi-bit random variables, which makes it possible to improve the accuracy of the normalization unit.

3.2. Normalization Unit for Multi-Bit Random Variable

It has been shown that multi-bit random variables can be added and multiplied [12], so they can also be divided according to Markov processes. Figure 8a shows the divider for a multi-bit random variable. Assuming that a steady state has been reached, the result of the next cycle $z^{t+1}$ can be calculated by the current input $X^t,Y^t$, and the current result $z^t$ as

$$\begin{array}{cc}\hfill z^{t+1}& =z^t+\eta (X^t-Y^t·z^t),\hfill \end{array}$$

(1)

where $\eta$ is the step length. Obviously, this is a Markov process. When this Markov process converges, we find the expectation of both sides of the equation at the same time, and we have

$$\begin{array}{cc}\hfill E(z^{t+1})& =E(z^t)+\eta (E(X^t)-E(Y^t)·E(z^t)),\hfill \\ \hfill E(z^{t+1})& =E(z^t)=E(z),\hfill \end{array}$$

(2)

namely,

$$\begin{array}{c}\hfill E(z)={\displaystyle \frac{E(x)}{E(y)}}={\displaystyle \frac{x}{y}}.\end{array}$$

(3)

To implement the normalization unit, replace Y with the sum of the inputs, as shown in Figure 8b. The addition of multi-bit random variables can naturally expand the bit width, so the addition here will not cause calculation overflow. Meanwhile, the result of the addition can be shared by different dividers.

3.3. Constant Sum of Normalized Probabilities

In the above probability normalization unit, the loss of addition calculation accuracy may cause fluctuations in the sum of normalized probabilities. However, this problem does not exist in the proposed structure. Suppose the input multi-bit random variables $\{X_1^t,X_2^t,\cdots ,X_M^t\}$ satisfy $E\left[X_i^t\right]=x_i$; then, the i-th normalized probability updates as

$$\begin{array}{cc}\hfill P_i^{t+1}& =P_i^t+\eta (X_i^t-\sum _{k=1}^M{X}_k^t·P_i^t).\hfill \end{array}$$

(4)

We sum all the tracking probabilities with the assumption that the sum of the probabilities in the previous iteration is 1, namely, ${\sum }_{i=1}^M{P}_i^t=1$; then,

$$\begin{array}{cc}\hfill \sum _{i=1}^M{P}_i^{t+1}& =\sum _{i=1}^M\left[P_i^t+\eta (X_i^t-\sum _{k=1}^M{X}_k^t·P_i^t)\right]\hfill \\ \hfill & =\sum _{i=1}^M{P}_i^t+\eta \sum _{i=1}^M{X}_i^t-\eta \sum _{k=1}^M{X}_k^t·\sum _{i=1}^M{P}_i^t\equiv 1.\hfill \end{array}$$

(5)

Equation (5) shows that as long as we set $P_i^0={\displaystyle \frac{1}{M}}$ to satisfy ${\sum }_{i=1}^M{P}_i^0=1$ during initialization, ${\sum }_{i=1}^M{P}_i^t=1$ can always be maintained in subsequent iterations. Therefore, in order to reduce the complexity and the impact of limited bit width on accuracy, we can cancel the last probability tracking and set the probability $P_M^t=1-{\sum }_{i=1}^{M-1}{P}_i^t$.

To verify the above conclusion, we assume that $M=4$ and the inputs $\{x_1,x_2,x_3,x_4\}=\{0.2,0.4,0.6,0.8\}$, and the normalized probabilities are $\{P_1,P_2,P_3,P_4\}=\{0.1,0.2,0.3,0.4\}$. Suppose $\eta =2^{-4}$; Figure 9 shows that the proposed method can ensure that the sum of the normalized probabilities is 1, while other methods cannot, completely solving the calculation overflow and fluctuation problems.

3.4. Re-Randomize According to Normalized Probabilities

The output of the probability normalization unit is often the tracked probability mass function (PMF). But in stochastic LDPC decoding and SCMA detection, a multi-bit random variable Z according to the tracked PMF should be generated. This is usually achieved through cumulative distribution function (CDF) sampling, which means that some additional adders are required to calculate the CDF, as mentioned in [9].

To reduce hardware overhead, the proposed structure can also track the CDF instead of the PMF, as shown in Figure 10. In the circuit that tracks the CDF, not only the sum of all inputs is needed, but also their cumulative sum one by one, which does not incur any additional hardware overhead. Similarly, only $M-1$ probabilities need to be tracked, and the probabilities should be initialized as $P_i^0={\displaystyle \frac{i}{M}}$.

However, it should be noted that the CDF sampling step is a randomization process. It may introduce large representation noise, significantly reducing the potential gain brought by the multi-bit random variable. To reduce the impact of this noise, we use low-discrepancy (LD) sequences [18,19], such as Sobol sequences, as the random numbers of the CDF sampling.

4. Performance Simulation and Complexity Analysis

4.1. Performance Simulation

To illustrate the computational accuracy of the proposed method, we randomly generate normalized inputs $\{x_1,x_2,x_3,x_4\}$ with $M=4$ and $\eta =2^{-4}$. We compare the performance differences between different methods by calculating the MSE of the probability distribution of Y. For a fair comparison, the MUX method [17] is augmented with TFM modules for probability tracking and generation of Y. As mentioned in Section 3, the process of generating Y introduces additional noise, which limits the improvement in tracking probability accuracy brought about by the increase in bit width. Figure 11 shows the performance of $X_i$ using uniformly distributed random numbers and the Sobol sequence for different bit widths. When using uniformly distributed random numbers, the MSE can hardly be reduced after the bit width exceeds 2 bits. When using the Sobol sequence, the performance can be significantly improved when the stochastic stream length is long.

The comparison of the proposed method and other methods is shown in Figure 12. It can be seen that the performance of the proposed method is significantly higher than that of the state-of-the-art methods, and the convergence speed to achieve the same MSE is more than twice as fast.

4.2. Hardware Implementation

We synthesize all the structures under Semiconductor Manufacturing International Corporation (SMIC) 65 nm CMOS technology with the Synopsys Design Compiler (DC). The throughput–area ratio (TAR) is defined by the ratio of throughput (TP) to the area, as

$$\begin{array}{c}\hfill \mathrm{TAR}={\displaystyle \frac{\mathrm{TP}}{\mathrm{Area}}},\end{array}$$

(6)

which can be used to survey the hardware efficiency. We implement the proposed structure with bit widths of 1, 2, 3, and 4 bits. We compare the hardware implementation results of the proposed structure and other structures when the MSE is roughly equal, as shown in

Table 1. Hardware implementation of the proposed structure for different bit widths.

Structure	Proposed
Technology	65 nm
Bit Width	1	2	3	4
Area (${\mathrm{um}}^2$)	1336.32	1754.64	2156.04	2528.28
Frequency (MHz)	500	500	500	500
MSE	$1.2\times {10}^{-3}$	$1.3\times {10}^{-3}$	$1.3\times {10}^{-3}$	$1.2\times {10}^{-3}$
MSE Convergence Cycle	128	80	64	64
Throughput (MS/s)	3.9	6.3	7.8	7.8
TAR (MS/(s · ${\mathrm{mm}}^2$))	$2.9\times {10}^3$	$3.6\times {10}^3$	$3.6\times {10}^3$	$3.1\times {10}^3$

and

Table 2. Hardwareimplementation comparison with different schemes.

Structure	Proposed	JPT [7]	MUX [17]	MUX-UDC [16]
Technology	65 nm	65 nm	65 nm	65 nm
Bit width	3	1	1	1
Area (${\mathrm{um}}^2$)	2156.04	1133.64	1002.24	743.40
Frequency (MHz)	500	500	500	1000
MSE	$1.3\times {10}^{-3}$	$1.5\times {10}^{-3}$	$1.3\times {10}^{-3}$	$1.3\times {10}^{-3}$
Cycle	64	256	1024	2048
Throughput (MS/s)	7.8	2.0	0.5	0.5
TAR (MS/(s · ${\mathrm{mm}}^2$))	$3.6\times {10}^3$	$1.7\times {10}^3$	$0.49\times {10}^3$	$0.66\times {10}^3$
TAR Ratio	2.10	1.00	0.28	0.38

.

It can be seen that the area of the proposed structure increases linearly with the increase in input bit width. However, increasing the bit width in this MSE interval cannot significantly improve the convergence speed, so the hardware efficiency of the proposed structure reaches its maximum when the input bit width is 3 bits. Although the proposed architecture contains multiplication, one of the multipliers has a small bit width and does not significantly increase the critical path compared to the JPT method. In summary, the proposed scheme achieves 2× the hardware efficiency of the JPT scheme.

4.3. Application Verification

In algorithms represented by message-passing algorithms (MPAs), normalization calculations are widely used. For example, in MPAs, variable nodes (VNs) need to calculate the normalization of the product of messages passed by adjacent factor nodes (FNs) as

$$\begin{array}{c}\hfill P(x=c)={\displaystyle \frac{{\displaystyle \prod _i}{P}_i(x=c)}{{\displaystyle \sum _{a\in \mathcal{S}}}{\displaystyle \prod _i}{P}_i(x=a)}},\end{array}$$

(7)

where $\mathcal{S}$ is the set of all possible symbols. It can be seen that it has a very high complexity, and the complexity of the FNs may be even higher, which is also the main problem of the MPA.

Sparse code multiple access (SCMA) is a promising non-orthogonal multiple-access (NOMA) technology candidate for the next-generation communication system. However, messaging-passing algorithm (MPA)-based SCMA detection can achieve near-maximum-likelihood (ML) performance with extremely high complexity [20]. To address this problem, some other NOMA methods and their detection methods [21,22] have been proposed, which can also achieve performance close to ML detection through linear or macrosymbol detection. Despite their lower complexity, these methods are not directly applicable to SCMA detection. Recently, the stochastic SCMA detector [9] used the idea of stochastic computing, replaced complex FN calculations with MUXs, and used the JPT normalization unit for normalization in VNs. To further improve the hardware efficiency, we use the proposed normalization unit in the VNs. We used a 1/3-code-rate Turbo code from the LTE standard, and the SCMA system has $K=6$ users spread over $N=4$ resource elements (REs) with a modulation order of 4. The bit error ratio (BER) performance simulation results and hardware implementation results are shown in Figure 13 and

Table 3. Hardware implementation comparison for SCMA detector.

Structure	Proposed			JPT [7,9]
Technology	65 nm			65 nm
Bit Width	2	3	4	1
Area (${\mathrm{um}}^2$)	1754.64	2156.04	2528.28	1133.64
Frequency (MHz)	500	500	500	500
BER (SNR = 4dB)	$1.0\times {10}^{-5}$	$2.2\times {10}^{-5}$	$2.2\times {10}^{-5}$	$2.0\times {10}^{-5}$
Decoding Cycle	60	45	40	150
Throughput (MS/s)	8.3	11.1	12.5	3.3
TAR (MS/(s · ${\mathrm{mm}}^2$))	$4.8\times {10}^3$	$5.2\times {10}^3$	$4.9\times {10}^3$	$2.9\times {10}^3$
TAR	1.62	1.75	1.68	1.00

, respectively.

It can be seen that when the bit width of the proposed scheme is 1, 2, 3, and 4 bits, it takes 100, 60, 45, and 40 decoding cycles (DCs) to achieve similar performance. Further increasing the bit-width accuracy will not significantly speed up the convergence, but the area increases linearly. Therefore, when the input bit width is 3 bits, the hardware efficiency reaches its maximum, which is about 1.75 times that of the JPT method.

Similarly, the proposed stochastic normalization unit can also be used in MPA-based MIMO detection. There has been some work using message passing based on conventional stochastic computing [23,24,25]. These studies also involve normalization units, and we believe that the proposed scheme can also be used in MIMO detection to enable more efficient computation. In addition, the Softmax function, which is widely used in the field of artificial neural networks (ANNs) [26], also contains a normalization function, so the proposed unit is also expected to be applied in the field of neural network accelerators.

5. Conclusions

A multi-bit random variable is a promising way to further improve the representation and calculation accuracy of stochastic computing (SC). Based on it, we propose a high-accuracy normalization unit. It completely solves the problem of the sum of normalized probabilities not being equal to 1, caused by scaling and overflow in stochastic bit computing. Compared to other state-of-the-art stochastic normalization units, it converges 4 times faster and achieves 2.5 times higher hardware efficiency. It provides new possible solutions for the efficient implementation of SCMA detection, MIMO detection, and neural network accelerators.