Rethinking the Role of Normalization and Residual Blocks for Spiking Neural Networks

Abstract

Biologically inspired spiking neural networks (SNNs) are widely used to realize ultralow-power energy consumption. However, deep SNNs are not easy to train due to the excessive firing of spiking neurons in the hidden layers. To tackle this problem, we propose a novel but simple normalization technique called postsynaptic potential normalization. This normalization removes the subtraction term from the standard normalization and uses the second raw moment instead of the variance as the division term. The spike firing can be controlled, enabling the training to proceed appropriately, by conducting this simple normalization to the postsynaptic potential. The experimental results show that SNNs with our normalization outperformed other models using other normalizations. Furthermore, through the pre-activation residual blocks, the proposed model can train with more than 100 layers without other special techniques dedicated to SNNs.

1. Introduction

Recently, spiking neural networks (SNNs) [1] have attracted substantial attention due to ultra-low power consumption and high friendliness with hardware such as neuromorphic-chips [2,3] and field-programmable gate array (FPGA) [4]. In addition, SNNs are biologically more plausible than artificial neural networks (ANNs) because their neurons communicate with each other through spatio-temporal binary events (spike trains), similar to biological neural networks (BNNs). However, SNNs are difficult to train since spike trains are non-differentiable.

Several researchers have focused on the surrogate gradient to efficiently train SNNs [5,6,7,8,9]. The surrogate gradient is an approximation of the true gradient and is applied to the backpropagation (BP) algorithm [10]. Recent studies have successfully trained deep SNNs using this method [11]. However, it is still challenging to train deepened models due to the increasing difficulty of controlling spike firing.

To control the spike firing properly, we propose a novel and simple normalization: postsynaptic potential normalization. Contrary to the standard batch/layer normalizations, our normalization removes the subtraction term from the standard normalization and uses the second raw moment instead of the variance as the division term. We can automatically control the firing threshold of the membrane potential and spike firing by conducting this simple normalization to the postsynaptic potential (PSP). The experimental results on neuromorphic-MNIST (N-MNIST) [12] and Fashion-MNIST (F-MNIST) [13] show that SNNs with our normalization outperform other models using other normalizations. We also show that the proposed method can train the SNN, consisting of more than 100 layers without other special techniques dedicated to SNNs.

The contributions of this study are summarized as follows.

• We propose a novel and simple normalization technique based on the firing rate. The experimental results show that the proposed model can simultaneously achieve high classification accuracy and low firing rate;
• We trained deep SNNs based on the pre-activation residual blocks [14]. Consequently, we successfully obtained a model with more than 100 layers without other special techniques dedicated to SNNs.

The remainder of the paper is organized as follows. In Section 2, Section 3 and Section 4, we describe the related works, SNN used in this paper, and our normalization technique. Section 5 presents the experimental results. Finally, Section 6 presents the conclusion and future works.

2. Related Works

2.1. Spiking Neuron

SNN consists of spiking neurons that model the behavior of biological neurons and handle the firing timing of the spikes. Owing to the differences in approximations, several spiking neuron models have been proposed, such as the integrate-fire (IF) [15], leaky-integrate-and-fire (LIF) [16], Izhikevich [17], and Hodgkin–Huxley model [18]. In this study, we adopt the spike response model (SRM) [19] to deal with the refractory period (Section 3).

The refractory period is an essential function of biological neurons to suppress the spike firing. Spike firing occurs when the neuron’s membrane potential exceeds the firing threshold. From a biological perspective, the membrane potential is calculated using PSP, representing the electrical signals converted from the chemical signals. These behaviors are represented within the chemical synapse model shown in Figure 1a [20]. SRM was implemented to approximate this synaptic model better than IF/LIF neurons, which are widely used in SNNs.

Figure 1. Illustration of fundamental components of SNN. (a) Biological information and signal processing between presynaptic and postsynaptic neurons in SNN. (b) A spiking neuron processes and communicates binary spiking events over time. (c) The postsynaptic neuron changes the membrane potential through each layer of the post-synaptic potential (PSP). It generates the output spikes when the membrane potential reaches the neuronal firing threshold [21]. (d) The cumulative distribution function (“Forward pass”) of our surrogate gradient and itself (“Backward pass”) [6].

2.2. Training of Spiking Neural Networks

It is well-known that SNNs are difficult to train due to non-differential spike trains. Researchers are working on this problem, and their solutions can be divided into two approaches: first, the ANN-SNN conversion [22,23,24,25], and second, the usage of the surrogate gradient [5,6,7,8,9]. The ANN-SNN conversion method uses the trained ANN parameters of SNN. The sophisticated and state-of-the-art ANN model can be reused through this method. However, this conversion approach requires many time-steps during inference and increases the power consumption. In contrast, the surrogate gradient is used to directly train SNNs by approximating the gradient of the non-differentiable spiking neurons. The surrogate gradient approach was adopted since the model obtained by surrogate gradient requires far fewer inference time-steps than the ANN-SNN conversion model [26].

2.3. Normalization

One of the techniques that have contributed to the success of ANNs is Batch Normalization (BN) [27]. BN is used to reduce the internal covariate shift, leading to a smooth landscape [28] while corresponding to the homeostatic plasticity mechanism of BNNs [29]. Using a mini-batch, BN computes the sample mean and standard deviation (STD). Meanwhile, several variants have been proposed to compute the sample mean and STD, such as Layer Normalization (LN) [30], Instance Normalization (IN) [31], and Group Normalization (GN) [32]. In particular, LN is effective at stabilizing the hidden state dynamics in recurrent neural networks for time-series processing [30].

Several normalization methods have also been proposed in the field of SNNs, such as threshold-dependent BN (tdBN) [33] and BN through time (BNTT) [34]. Thus, tdBN incorporates the firing threshold into BN, whereas BNTT computes BN at each time step. Furthermore, some studies used BN as is [35]. These studies applied the normalization to the membrane potential. In contrast, our method was applied to PSP, as shown in Figure 2b, to simplify the normalization form (Section 4).

Figure 2. Normalization methods for spiking neural networks (SNNs). Each subplot shows a feature map (a,b) or post-synaptic input (b) tensor, with N as the batch axis, C as the channel axis, (H, W) as the spatial axis, and T as the time axis in the figure. Green voxels (a,b), previous methods, and blue (c), our proposed method, are normalized by the same second central moment and uncentered second moment, respectively.

3. Spiking Neural Networks Based on the Spike Response Model

In this section, we describe the SNN used in this study. Our SNN is constructed using SRM [19]; it uses SLAYER [6] as the surrogate gradient function to train the SRM.

3.1. Spike Response Model

We adopt SRM as a spiking neuron model [19]. SRM model is based on combining the effects of the incoming spike arriving at the spiking neuron. It also has a function to the spike firing when the membrane potential $u\left(t\right)(t=1,2,\cdots ,T)$ reaches the firing threshold. Figure 1b,c indicate the behavior of this model. The equations are given as follows:

$$u_i(t)=\sum _j{w}_{ij}(\epsilon \ast s_j)\left(t\right)+(\nu \ast s_i)\left(t\right),$$

(1)

$$s_i\left(t\right)=f_s(u_i\left(t\right)-\theta ),$$

(2)

where $w_{i,j}$ is the synaptic weight from the presynaptic neuron j to the postsynaptic neuron i. $s_j\left(t\right)$ is the spike train inputted from the presynaptic neuron j, $s_i\left(t\right)$ is the output spike train of the postsynaptic neuron i, ∗ is a temporal convolution operator, and $\theta$ is a threshold used to control the spike generation. $f_s$ is the Heaviside step function, which fires the spike when the membrane potential $u_i\left(t\right)$ exceeds the firing threshold $\theta$ as shown in Figure 3. In addition, $\epsilon (·)$ and $\nu (·)$ are the spike response and refractory kernels formulated using the exponential function as follows:

$$\epsilon (t)=\frac{t}{{\tau }_s}{e}^{1-\frac{t}{{\tau }_s}},$$

(3)

$$\nu (t)=-2\theta e^{-\frac{t}{{\tau }_r}},$$

(4)

where ${\tau }_s$ and ${\tau }_r$ are the time constants of spike response and refractory kernels, respectively. Note that $\epsilon \ast s_j\left(t\right)$ represents the PSP. After firing, the postsynaptic neuron goes into the refractory period and cannot fire until its membrane potential resets to its resting potential.

Figure 3. A dynamic threshold in SRM. The membrane potential (orange solid line) of the postsynaptic neuron is described by the superposition of the PSPs. Each input spike arrival and output spike are denoted by a black arrow and red arrow, respectively. At the moment of spiking, the firing threshold (green dashed line) increases and decreases with time to the initial value of the firing threshold.

The main role of the refractory period suppresses the firing rate at a given spike interval. If the spike interval T is constant, the firing rate without the refractory period is given by $1/T$. On the other hand, if the refractory period r is taken into account, it can be rewritten as $1/(T+r)$. Therefore, the firing rate decreases as the refractory period increases, as shown in Figure 4. In SNNs, the firing rate is proportional to the computational cost. Namely, using the refractory period ensures biological plausibility and reduces computational costs.

Figure 4. Relationship between refractory period $r=(\nu \ast s_i)$ and firing rate for the different spike intervals $T=1,3,5$ [ms].

3.2. Multiple Layers Spike Response Model

By using Equations (1) and (2), the SNNs with multi-layers can be described as follows:

$${\mathit{a}}^{\left(l\right)}(t)=(\epsilon \ast {\mathit{s}}^{\left(l\right)})(t),$$

(5)

$${\mathit{u}}^{(l+1)}(t)={\mathit{W}}^{\left(l\right)}{\mathit{a}}^{\left(l\right)}(t)+(\nu \ast {\mathit{s}}^{(l+1)})(t),$$

(6)

$${\mathit{s}}^{(l+1)}(t)=f_s({\mathit{u}}^{(l+1)}(t)),$$

(7)

where ${\mathit{a}}^{\left(l\right)}\left(t\right)\in {\mathbb{R}}_{\ge 0}^{\mathrm{C}\times \mathrm{W}\times \mathrm{H}}$ and ${\mathit{s}}^{\left(l\right)}\left(t\right)\in {\{0,1\}}^{\mathrm{C}\times \mathrm{W}\times \mathrm{H}}$ are the PSP and input spike tensor of time step t; $\mathrm{C}$ is the number of channels; and $\mathrm{W}$ and $\mathrm{H}$ are the width and height of the input spike tensor, respectively. Since ${\mathit{a}}^{\left(l\right)}\left(t\right)$ does not take a value less than zero, we consider an excitatory neuron. Furthermore, ${\mathit{W}}^{\left(l\right)}\in {\mathbb{R}}^M$ is the weight matrix representing the synaptic strengths between the spiking neurons in l and $l+1$ layers; M is the number of neurons of $l+1$-th layer.

3.3. Deep SNNs by Pre-Activation Blocks

A deep neural network is essential to recognize complex input patterns. In particular, ResNet is widely used in ANNs [14,36], and its use in SNNs is expanding.

The ResNet’s networks are divided into the pre-activation and post-activation residual blocks, as follows (Figure 5):

$$\mathrm{Pre}:{\mathit{h}}^{(k+1)}(t)={\mathit{h}}^{\left(k\right)}(t)+G({\mathit{h}}^{\left(k\right)}(t)),$$

(8)

$$\mathrm{Post}:{\mathit{h}}^{(k+1)}(t)=F({\mathit{h}}^{\left(k\right)}(t)+G({\mathit{h}}^{\left(k\right)}(t))),$$

(9)

where ${\mathit{h}}^{\left(k\right)}$ and ${\mathit{h}}^{(k+1)}$ are the input and output in the $k+1$ block, respectively. G represents the residual function, corresponding to “Conv-Func-Conv” and “Func-Conv-Func-Conv” in Figure 5; F represents the Func layer (“Spike-PSP-Norm”). Note that the refractory period is used in F. In the experimental section, we compare these blocks and show that deep SNNs can be trained using the pre-activated residual blocks. This result shows that identity mapping is an essential tool to train the deep SNNs, similar to ANNs [14].

Figure 5. Network blocks. (a) Post-activation residual block, (b) Pre-activation residual block, and (c) Func layer. Postsynaptic potential (PSP) and Norm in the Func layer represent Equation (5) and normalization. Spike represents $f_s(·+r)$, where r is the refractory period.

3.4. Surrogate-Gradient

We use SLAYER [6] as one of the surrogate gradient algorithms to tarin the SNN with multi-layers. In SLAYER, the derivative of the spike activation function $f_s$ of the $l+1$ layer is approximated as follows (Figure 1d):

$${\mathbf{\rho }}^{(l+1)}(t)=\frac{1}{\alpha }\exp (-\beta |{\mathit{u}}^{(l+1)}(t)-\mathbf{\theta }|),$$

(10)

where $\alpha$ and $\beta$ are hyperparameters to adjust the peak value and sharpness for the surrogate gradient, and $\mathbf{\theta }\in {\mathbb{R}}^M$ is the firing threshold. SLAYER can be used to train SRM as described in [6].

4. Normalization of Postsynaptic Potential

In this section, we explain the derivation of our normalization, which is called postsynaptic-potential normalization, as shown in Figure 6a.

Figure 6. (a) Overview of the forward propagation phase using postsynaptic potential (PSP) normalization on the N-MNIST dataset. (b) Example of feature maps, which are integrated along the time axis, without PSP normalization. (c) As in (b) but for feature maps with it. PSP normalization controls the activity of the network and prevents the over-firing of the neurons.

As the depth of the SNN becomes deeper, it becomes more difficult to control spike firing properly (Figure 6b,c). To tackle this problem, we first introduce the following typical normalization into the PSP.

$${\widehat{\mathit{u}}}^{(l+1)}(t)={\mathit{W}}^{\left(l\right)}{\widehat{\mathit{a}}}^{\left(l\right)}(t)+(\nu \ast {\mathit{s}}^{(l+1)})(t),$$

(11)

$${\widehat{\mathit{a}}}^{\left(l\right)}(t)=\frac{{\mathit{a}}^{\left(l\right)}\left(t\right)-{\mathbb{E}}_X\left[{\mathit{a}}^{\left(l\right)}\right]}{\sqrt{{\mathbb{V}}_X\left[{\mathit{a}}^{\left(l\right)}\right]+\lambda }}\odot \gamma +\xi ,$$

(12)

where $\gamma$ and $\xi$ are trainable parameters, and the ⊙ operator denotes the Hadamard product; each variable of ${\mathbb{E}}_X\left[{\mathit{a}}^{\left(l\right)}\right]$ and ${\mathbb{V}}_X\left[{\mathit{a}}^{\left(l\right)}\right]$ is approximated as follows:

$${\mathbb{E}}_X[a_i^{\left(l\right)}]\approx \frac{1}{X}\sum _{x=1}^X{a}_i^{\left(l\right)}(x),$$

(13)

$${\mathbb{V}}_X[a_i^{\left(l\right)}]\approx \frac{1}{X}\sum _{x=1}^X(a_i^{\left(l\right)}(x)-{\mathbb{E}}_X[a_i^{\left(l\right)}])\odot (a_i^{\left(l\right)}(x,t)-{\mathbb{E}}_X[a_i^{\left(l\right)}]),$$

(14)

where $a_i^{\left(l\right)}\left(x\right)$ represents the x-th variable required to compute these statistics of the i-th variable of ${\mathit{a}}^{\left(l\right)}\in {\mathbb{R}}_{\ge 0}^{\mathrm{C}\times \mathrm{W}\times \mathrm{H}\times \mathrm{N}\times \mathrm{T}}$ (N is the mini-batch size), and X depends on what kind of summation to compute. For example, if we compute these equations as in BN, $X=W\times H\times N\times T$. In addition, if we compute them as in LN, $X=W\times H\times C\times T$. Note that the normalization to PSP means that it inserts before the convolution or fully connected layers. This position differs from the other normalization ones, which use normalization to the membrane potential [33,34,35].

As shown in Equation (12), ${\widehat{\mathit{a}}}^{\left(l\right)}\left(t\right)$ may take minus. Therefore, ${\widehat{\mathit{a}}}^{\left(l\right)}\left(t\right)<0$ is not valid since neurons of SLAYER represent excitatory neurons. This phenomenon clearly arises from the trainable parameter $\xi$ and the shift parameter ${\mathbb{E}}_X\left[{\mathit{a}}^{\left(l\right)}\right]$. Thus, we modify Equation (12) as follows:

$${\widehat{\mathit{a}}}^{\left(l\right)}(t)=\frac{{\mathit{a}}^{\left(l\right)}\left(t\right)}{\sqrt{{\mathbb{V}}_X\left[{\mathit{a}}^{\left(l\right)}\right]+\lambda }}\odot \gamma .$$

(15)

Next, we consider the case when ${\widehat{\mathit{u}}}^{(l+1)}\left(t\right)$ reaches the firing threshold $\mathbf{\theta }$.

$$\mathbf{\theta }={\mathit{W}}^{\left(l\right)}\widehat{\mathit{a}}(\mathit{t})+(\nu \ast {\mathit{s}}^{(l+1)})(t),$$

(16)

$$={\widehat{\mathit{W}}}^{\left(l\right)}\frac{{\mathit{a}}^{\left(l\right)}\left(t\right)}{\sqrt{{\mathbb{V}}_X\left[{\mathit{a}}^{\left(l\right)}\right]+\lambda }}+(\nu \ast {\mathit{s}}^{(l+1)})(t).$$

(17)

Here, we have merged the trainable parameter $\gamma$ and the weight matrix ${\mathit{W}}^{\left(l\right)}$ into ${\widehat{\mathit{W}}}^{\left(l\right)}$. This merging is possible because of the normalization performed before multiplying ${\mathit{W}}^{\left(l\right)}$. Then, we express Equation (17) as follows:

$${\widehat{\mathit{W}}}^{\left(l\right)}{\mathit{a}}^{\left(l\right)}(t)=\sqrt{{\mathbb{V}}_X[{\mathit{a}}^{\left(l\right)}]+\lambda }(\mathbf{\theta }-\nu \ast {\mathit{s}}^{(l+1)}(t)):=\widehat{\mathbf{\theta }}.$$

(18)

Equation (18) shows that the firing threshold varies dynamically as shown in Figure 3, which is consistent with the activity of cortical neurons in the human brain [37,38,39,40]. The refractory period $(\nu \ast {\mathit{s}}^{(l+1)})\left(t\right)$ and $\sqrt{{\mathbb{V}}_X\left[{\mathit{a}}^{\left(l\right)}\right]+\lambda }$ can decrease $\widehat{\mathbf{\theta }}$ and scaling, respectively.

Next, we focus on the scale factor $\sqrt{{\mathbb{V}}_X\left[{\mathit{a}}^{\left(l\right)}\right]+\lambda }$. As shown in Equation (18), the firing threshold $\widehat{\mathbf{\theta }}$ becomes larger as the variance (second central moment) ${\mathbb{V}}_X\left[{\mathit{a}}^{\left(l\right)}\right]$ increases. However, considering the behavior of the membrane potential, $\widehat{\mathbf{\theta }}$ should become larger when the value of PSP (not variance) increases. Thus, we modify the equation as follows.

$${\widehat{\mathit{a}}}^{\left(l\right)}(t)=\frac{{\mathit{a}}^{\left(l\right)}\left(t\right)}{\sqrt{{\mathbb{E}}_X[{\left({\mathit{a}}^{\left(l\right)}\right)}^2]+\lambda }},$$

(19)

where ${\mathbb{E}}_X[{\left({\mathit{a}}^{\left(l\right)}\right)}^2]$ represents the second raw moment consisting of the following variable,

$${\mathbb{E}}_X[{(a_i^{\left(l\right)})}^2]\approx \frac{1}{X}\sum _{x=1}^X{a}_i^{\left(l\right)}\left(x\right)\odot a_i^{\left(l\right)}\left(x\right).$$

(20)

By using this equation, we do not have to compute the mean beforehand, in contrast to using the variance.

In addition to ${\mathbb{E}}_X[{\left({\mathit{a}}^{\left(l\right)}\right)}^2]$, there is a hyperparameter $\lambda$ in the scale factor. $\lambda$ is usually set to a small constant, e.g., $\lambda ={10}^{-3}$ because it plays the role of the numerical stability. Figure 7 shows the relationship between ${\mathbb{E}}_X[{(a_i^{\left(l\right)})}^2]$ and $\widehat{\theta }$ when changing $\theta$ and $\lambda$. As shown in this figure, $\widehat{\theta }$ monotonically decreases as ${\mathbb{E}}_X[{(a_i^{\left(l\right)})}^2]$ decreases. In particular, $\widehat{\theta }$ is close to zero when $\lambda$ is sufficiently small, regardless of the initial threshold $\theta$. $\widehat{\theta }\approx 0$ means that spikes fire at all times even if the membrane potential is significantly small, making it difficult to train a proper model. Thus, we set a relatively large value ($\lambda =0.1$) as the default value.

Figure 7. Relationship between ${\mathbb{E}}_X[{(a_i^{\left(l\right)})}^2]$ and $\widehat{\theta }$.

5. Experiments

In this section, we evaluate two PSP normalizations: BN (the most common normalization) and LN (which is effective in time-series processing, such as SNN). We called them PSP-BN ($X=W\times H\times N\times T$) and PSP-LN ($X=W\times H\times C\times T$).

5.1. Experimental Setup

We evaluated PSP-BN and PSP-LN on the spatio-temporal event and static image datasets. We used N-MNIST [12] and F-MNIST [13]. N/F-MNISTs are widely used datasets containing 60 K training and 10K test samples with 10 classes. Each size is $34\times 34\times$ 30,000 events (N-MNIST), and $28\times 28$ pixels (F-MNIST). We partitioned the 60 K data using 54 K and 6 K as our training and validation data, respectively. We also resized the F-MNIST image from $28\times 28$ to $34\times 34$ to achieve higher accuracy.

We evaluated the performance of several spiking convolutional neural network models, such as 14-convolutional layers on N/F-MNIST. We also used more deep models, such as ResNet-106 on N-MNIST and F-MNIST, respectively.

We used hyperparameters shown in Table 1 in all experiments and implemented by PyTorch. We used the default initialization of PyTorch and showed the best accuracies of all models. All experiments were conducted using a single Tesla V100 GPU. In addition to this computational resource limitation, we randomly sampled 6 K of the training data for both datasets to train in each epoch since SLAYER requires a significant amount of time to train.

Table 1. Hyperparameter setting on N-MNIST and F-MNIST.

Hyperparameter	N-MNIST	F-MNIST
${\tau }_s$	10	10
${\tau }_r$	10	10
$\alpha$	10	10
$\beta$	10	10
$\theta$	10	10
optimizer	AdaBelief	AdaBelief
learning rate	${10}^{-2}$	${10}^{-2}$
weight decay	${10}^{-4}$	${10}^{-4}$
weight scale	10	10
mini-batch size	10	10
time step	300	100
epoch	100	100

5.2. Effectiveness of Postsynaptic Potential Normalization

We first evaluate the effectiveness of our normalizations. Table 2 presents the accuracies of PSP-BN and PSP-LN and other approaches. Note that we set our normalization before the convolution as described in Section 4, which is different from the position proposed in previous studies [33,34,35]. This table illustrates that PSP-BN and PSP-LN achieve high accuracies on both datasets compared to the other approaches.

Table 2. Accuracies of N-MNIST and F-MNIST obtained from different methods. PSP-BN and PSP-LN are our normalization methods, and None is the model without normalization. Here, “c”, “n”, and “o” represent the convolution, normalization, and output neurons, respectively. In addition, each layer and spatial dimension in the network are separated by “-” and “×”.

Method	Dataset	Network Architecture	Acc. (%)
BN [35]	N-MNIST	34×34×2-8c3n-{16c3n}*5-16c3n-{32c3n}*5-10o	85.1
BNTT [34]	N-MNIST	34×34×2-8c3n-{16c3n}*5-16c3n-{32c3n}*5-10o	90.0
tdBN [33]	N-MNIST	34×34×2-8c3n-{16c3n}*5-16c3n-{32c3n}*5-10o	81.8
PSP-BN	N-MNIST	34×34×2-n8c3-{n16c3}*5-n16c3-{n32c3}*5-10o	97.4
PSP-LN	N-MNIST	34×34×2-n8c3-{n16c3}*5-n16c3-{n32c3}*5-10o	98.2
None	N-MNIST	34×34×2-8c3-{16c3}*5-16c3-{32c3}*5-10o	40.6
BN [35]	F-MNIST	34×34-16c3n-{32c3n}*5-32c3n-{64c3n}*5-10o	10
BNTT [34]	F-MNIST	34×34-16c3n-{32c3n}*5-32c3n-{64c3n}*5-10o	10
tdBN [33]	F-MNIST	34×34-16c3n-{32c3n}*5-32c3n-{64c3n}*5-10o	40.5
PSP-BN	F-MNIST	34×34-n16c3-{n32c3}*5-n32c3-{n64c3}*5-10o	88.6
PSP-LN	F-MNIST	34×34-n16c3-{n32c3}*5-n32c3-{n64c3}*5-10o	89.1
None	F-MNIST	34×34-16c3-{32c3}*5-32c3-{64c3}*5-10o	84.1

We also investigate the effect of the proposed method on the firing rate. Figure 8 and Figure 9 show the firing rates of each method. As shown in Figure 8, our normalized models can suppress the firing rate in most layers compared to the unnormalized model. Furthermore, Figure 9 and Table 2 show that our normalized models can simultaneously achieve high classification accuracy and low firing rate compared to other normalizations. These results verify the effectiveness of our normalizations.

Figure 8. Comparison of firing rates. The top is N-MNIST and the bottom is F-MNIST.

Figure 9. Total firing rates of each model.

Then, we also analyze the training and inference times of the proposed method. Figure 10 shows the computational cost of BN, PSP-BN, and PSP-LN. The training time of PSP-BN and PSP-LN is shorter than BN because our normalization method does not require training parameters ($\gamma$ and $\xi$) as Equation (17). On the other hand, the training time of PSP-BN and PSP-LN are almost the same because these differ only in X. In addition, there is no significant difference in inference time for each normalization. These results show that our normalization is suitable for training SNN.

Figure 10. Comparison of the training and inference time by using BN, PSP-BN and PSP-LN. The top is the training time and the bottom is the inference time.

5.3. Performance Evaluation of Deep SNNs by Residual Modules

Finally, we evaluate the performance of SNNs using the residual blocks. Table 3 shows the performance of SNNs using the pre-activation and post-activation residual blocks. As shown in this table, the accuracy is substantially improved using the pre-activation residual blocks. This result shows that the post-activation employed in previous studies without refractory period [5,11,33] is unsuitable for SNNs with a refractory period. Thus, while ensuring the biological plausibility, due to the refractory period, we can obtain deep SNNs beyond 100 layers using our normalizations and pre-activation residual blocks.

Table 3. Performance comparison using post-activation and pre-activation residual blocks. We use ResNet-106 on N-MNIST and F-MNIST datasets, respectively.

Meshod	Dataset	Network Architecture	Acc. (%)
PSP-BN	N-MNIST	Post-activation ResNet-106	10.0
PSP-BN	N-MNIST	Pre-activation ResNet-106	75.4
PSP-LN	N-MNIST	Post-activation ResNet-106	10.0
PSP-LN	N-MNIST	Pre-activation ResNet-106	86.8
PSP-BN	F-MNIST	Post-activation ResNet-106	10.0
PSP-BN	F-MNIST	Pre-activation ResNet-106	81.6
PSP-LN	F-MNIST	Post-activation ResNet-106	10.0
PSP-LN	F-MNIST	Pre-activation ResNet-106	82.1

6. Discussion and Conclusions

In this study, we proposed an appropriate normalization method for SNN. The proposed normalization removes the subtraction term from the standard normalization and uses the second raw moment as the denominator. Our normalized models outperformed other normalized models based on existing normalization such as BN, BNTT, and tdBN by inserting this simple normalization before the convolutional layer. Furthermore, our proposed model with pre-activation residual blocks can train with more than 100 layers without any other special techniques dedicated to SNNs.

Besides the type of normalization, some papers pointed out that tuning hyperparameters ${\tau }_s$ and $\lambda$ is essential for high accuracy [41,42]. Investigating this aspect, we found that PSP-BN is sensitive to $\lambda$, whereas PSP-LN is robust to ${\tau }_s$ and $\lambda$ (Figure 11 and Figure 12). These results imply that the effectiveness of tuning hyperparameters depends on X. We will conduct more detailed analysis in this regard in the future. In addition, we will also analyze the effect on other datasets and networks. Furthermore, we aim to extend postsynaptic normalization based on tdBN to develop robust normalization techniques for the thresholds in spiking neurons.

Figure 11. Accuracy comparison of F-MNIST dataset with respect to changing hyperparameter ${\tau }_s$ ($=\{1,5,10\}$).

Figure 12. Accuracy comparison of F-MNIST with respect to changing $\lambda$ ($=\{{10}^{-10},{10}^{-5},{10}^{-3},{10}^{-1}\}$).