Comparison of Bagging and Sparcity Methods for Connectivity Reduction in Spiking Neural Networks with Memristive Plasticity

Abstract

Developing a spiking neural network architecture that could prospectively be trained on energy-efficient neuromorphic hardware to solve various data analysis tasks requires satisfying the limitations of prospective analog or digital hardware, i.e., local learning and limited numbers of connections, respectively. In this work, we compare two methods of connectivity reduction that are applicable to spiking networks with local plasticity; instead of a large fully-connected network (which is used as the baseline for comparison), we employ either an ensemble of independent small networks or a network with probabilistic sparse connectivity. We evaluate both of these methods with a three-layer spiking neural network, which are applied to handwritten and spoken digit classification tasks using two memristive plasticity models and the classical spike time-dependent plasticity (STDP) rule. Both methods achieve an F1-score of 0.93–0.95 on the handwritten digits recognition task and 0.85–0.93 on the spoken digits recognition task. Applying a combination of both methods made it possible to obtain highly accurate models while reducing the number of connections by more than three times compared to the basic model.

1. Introduction

Neural network-based intelligent systems are widely employed in a wide range of tasks, from natural language processing to computer vision and signal processing. In edge computing, however, the use of deep learning methods still poses a variety of challenges, including latency and power consumption constraints, both during training and inference.

Neuromorphic computing devices, in which the information is encoded and processed in the form of binary events called spikes, offer a prospective solution to these problems. Modern neuroprocessors, e.g., TrueNorth [1], Loihi [2], or Altai https://motivnt.ru/neurochip-altai (accessed 8 February 2024), have been shown to achieve power consumption at the order of milliwatts [3]. Thus, these devices offer a powerful inference interface, and they can be used to deploy spiking neural networks (SNNs), thereby allowing both inference and training directly on the neurochip. This can be extremely useful for edge computing applications by reducing power consumption and latency.

In turn, memristor-based training poses its own unique set of challenges and limitations. The most prominent one arises from the hardware implementation of synapses, where every neuron can have only a limited amount of synapses [4,5,6], thus imposing limitations on the number of weights that a given network may have. In this regard, sparsely connected spiking networks, where the connectivity can be reduced depending on the hardware specifications, are a plausible solution.

In this study, we compared two methods of reducing connectivity in memristive spiking neural networks: a bagging ensemble of spiking neural networks and a probabilistic sparse SNN. Using a three-layer SNN with inhibitory and excitatory synapses, we solved the handwritten and spoken digits classification tasks, as well as compared the outcomes for the proposed connectivity reduction types and three plasticity models. The main contributions of this work are as follows:

–

We design a probabilistic sparse connectivity approach to creating a two-layer spiking neural network (achieved by implementing a bagging ensemble of two-layer SNNs) and then compare these two methods.

–

We propose an efficiency index that facilitates comparisons between different methods of connectivity reduction, and we will look to apply it to the SNNs used in the study.

–

We demonstrate that both connectivity reduction methods achieve competitive results on handwritten and spoken digit classification tasks, and that it can be used with the memristive plasticity models.

–

We show that the model that uses both connectivity reduction techniques simultaneously outperforms both methods in terms of the accuracy-per-connection efficiency metric.

The rest of the study is structured as follows: In Section 2, we provide a brief overview of the existing connectivity reduction methods for SNNs. In Section 3, we describe the datasets we use, the plasticity models, the base spiking neural structure, and the sparsity methods (which we utilized for comparison). In Section 4, we provide the accuracy estimations for the proposed approaches and discuss the obtained results in Section 5. Finally, we detail our conclusions in Section 6.

2. Literature Review

Connectivity reduction concerning spiking and artificial neural networks has been studied in several existing works.

A number of works have proposed to use a probabilistic coefficient to form connections between neurons in a spiking neural network. For example, the work of [7] used a network that consists of three layers of neurons. The first layer is responsible for encoding the input samples (images) into Poisson spike sequences. The second layer consists of 4000 excitatory and 1000 inhibitory leaky integrate and fire neurons. The output layer also consists of LIF neurons, the number of which corresponds to the number of classes in the selected dataset (EMNIST [8], YALE [9], or ORL [10]). The connections within the second layer are formed in accordance with the selected probability of 0.2. In this case, the weights of the synapses change depending on the spatial location of neurons and the dynamics of spike activity. The connections between the encoding layer and the second layer of neurons are excitatory without exhibiting plasticity. Probabilistic linking and changing weights using spatial location can achieve high classification accuracy on the image datasets.

Another approach to connectivity reduction that is present in the literature is based on designing locally connected SNNs, the weights in which are created in a sparse fashion according to a certain rule. In [11], for example, a routing scheme that used a hybrid of short-range direct connectivity and an address event representation network was developed. Without providing any benchmark results, the authors focused on the details of mapping a given SNN to the proposed architecture, and they showed that it yielded up to a 90% reduction in connectivity. The authors of the [12] study proposed a way through which to reduce connectivity in a three-layer network operating on the Winner Takes All principle. The input image, encoded by the first layer using frequency coding, was divided into small fragments that were then sent to individual neurons in the excitation layer. This made it possible to reduce the number of connections with local plasticity by up to 50% while maintaining the accuracy on the MNIST dataset at approximately 90%. The authors of [13] proposed a joined connectivity and weight learning approach inspired by the synapse elimination and synaptogenesis in biological neurons. The gradient in this work was redefined as an additional synaptic parameter, thereby facilitating a better adaptation of the network topology. A multilayer convolutional SNN trained using a surrogate gradient approach and pruned according to the designed method demonstrated an accuracy of about 89 and 92%, with less than 5% connectivity in the whole SNN for the MNIST and CIFAR datasets, respectively. Increasing the proportion of connectivity to 40% improved the quality of solving classification problems to 98.9 and 92.8%. The possibility of a sharp decrease in the number of connections can be caused by a large number of layers and neurons in the original network. Overall, this work demonstrates the fundamental applicability of the approaches used in classical machine learning for spiking neural networks.

In [14], a two-stage pruning method for on-chip SNNs was developed. The pruning was first performed during training based on weight update history and spike timing. After training, it was then via weight thresholding. By training a deep SNN with time-to-first-spike coding using the proposed approach, the authors decreased latency by a factor of 2 and reduced the network connectivity by 92% without accuracy loss. Another example can be found in [15], where the authors used a method of zeroing weights above a given threshold and achieved a 70% reduction in connectivity. In this paper, the network consists of a mixture of formal and spiking convolutional layers, and the resulting sparse hybrid network achieved more than 71% accuracy on the IVS 3cls [16] dataset. In [17], sparsity in a multilayer convolutional spiking network is achieved by limiting the number of connections associated with each neuron, and this is based on calculating the contribution of a neuron to the operation of the entire network. The proposed approach is shown to achieve high accuracy on such classical datasets as DVS-Gesture [18] (98%), MNIST (99%), CIFAR-10 (94%), and N-MNIST [19] (99%), with a 50% reduction in the number of connections. Finally, in [20], a sparse SNN topology was proposed, where the connectivity reduction was performed via a combination of pruning and quantization based on the power law weight-dependent plasticity model. Connectivity reduction was performed based on a threshold value at which the weights become zero. After training, the three-layer, fully connected SNN designed in the study achieved a classification accuracy of 92% on the MNIST dataset.

Thus, currently employed methods of reducing the connectivity in spiking neural networks are mostly encompassed by pruning, quantization, and local connectivity. However, ensemble learning, where multiple smaller networks are used together to form a stronger classifier, can be also viewed as a single sparse network. In this work, we explored this path to connectivity reduction and compared it to a probabilistic, locally connected SNN topology that was proposed in the work of [21], and which was investigated in our previous research with different types of plasticity models [22,23,24].

3. Materials and Methods

3.1. Datasets and Preprocessing

To train and evaluate the proposed methods, we used two benchmark datasets: the scikit-learn Digits (Digits) [25] and Free Spoken Digits Dataset (FSDD) [26]. The first consists of 1797 $8\times 8$ images of handwritten numbers, and the second contains 3000 audio recordings of spoken numbers from 0 to 9 in English. The choice of these datasets over larger and more widely used classification datasets such as MNIST, CIFAR-10, or N-MNIST was motivated by computational requirements. The training process for spiking neural networks when using local plasticity rules requires extensive computational experiments to select the combination of hyperparameters. We automated this process (see Section 4 for details), thereby placing a limit on the time required to train the network. The Digits dataset was quite difficult in comparison to MNIST due to containing less information about the handwritten digits in terms of image size and dataset volume (see the examples in Figure 1).

Both datasets had 10 classes, which could be broken down as 180 samples per class for Digits and 300 samples per class for FSDD. Additionally, the samples in FSDD varied by speaker as follows: 6 speakers in total and 50 recordings for each digit per speaker with different intonations.

The raw data were preprocessed as follows:

(1)

Feature representation: For Digits, their original vector representation in the form of pixel intensities was used without changes; for FSDD, a vector representing an audio sample was obtained by splitting the audio into frames, which was achieved by extracting 30 Mel-frequency cepstral coefficients [27] (MFCCs) and then averaging them across frames.

(2)

Normalization: Depending on the type of plasticity, the input vectors were normalized either by reducing to a zero mean and one standard deviation (standard scaling) or by L2 normalization.

(3)

Gaussian Receptive Fields (GRFs): This step was intended to increase the separability of the data by transforming it into a space of higher dimension. At this stage, the range of each of the normalized feature vectors was divided into M equal intervals. At each interval $j=1,\cdots ,M$, a Gaussian peak was constructed with a center ${\mu }^j$ and standard deviation $\sigma$ (see Equation (1), Figure 2). The value of each component $x_i$ of the input vector was replaced by a set of values $G_j\left(x\right)$, which characterized the proximity of $x_i$ to the center of the j-th receptive field. Thus, the dimension of the input vector increased by the factor of M.

(4)

Spike encoding: To convert the normalized and GRF-processed input vectors into spike sequences, we used a frequency-based approach. With this encoding method, each input neuron (spike generator) emits spikes at a frequency $\nu$ during the entire sample time $t_{\mathrm{e}}$, where $\nu ={\nu }_{max}·k$. Here, ${\nu }_{max}$ is the maximum frequency of spike emission and k is the value of the input vector component. After time $t_{\mathrm{e}}$ has passed, the generators do not emit spikes for $t_{\mathrm{p}}=50$ ms to allow for the neuron potentials to return to their original values.

$$G_j\left(x_i\right)=exp\left(\frac{{(x_i-{\mu }^j)}^2}{{\sigma }^2}\right).$$

(1)

Figure 2. An example of Gaussian receptive fields with the number of fields being equal to 5. The input feature $x_i$ was intersected with overlapping Gaussian fields to produce a vectorized feature representation: $G_j\left(x_i\right),j\in [0,5)\cap \mathbb{N}$.

Figure 2. An example of Gaussian receptive fields with the number of fields being equal to 5. The input feature $x_i$ was intersected with overlapping Gaussian fields to produce a vectorized feature representation: $G_j\left(x_i\right),j\in [0,5)\cap \mathbb{N}$.

To illustrate the relative complexity of both datasets, we performed dimensionality reduction using principal component analysis (PCA) on both datasets after the feature engineering stage. This method allows us to reduce the feature space to two dimensions and visually assess the degree of nonlinear separability of the samples. Its results are shown in Figure 3.

It can be clearly seen that the classification boundaries for the MFCC-encoded FSDD dataset have to be much more complex in order to achieve high accuracy. In other words, in this work, the handwritten digits dataset acts as a weak baseline, and it was used to assess the general capability of the sparsity methods under consideration, while the spoken digits dataset played the role of a more challenging benchmark.

3.2. Synaptic Plasticity Models

In this work, we considered two memristive plasticity models: nanocomposite (NC) [28] and poly-p-xylylene (PPX) [29]. These models were proposed to approximate the real-world dependence of synaptic conductance change $\Delta w$ on the value of the conductance w and on the time difference $\Delta t$ between presynaptic and postsynaptic spikes, and they are defined in Equations (2) and (3).

$$\Delta w(\Delta t)=\left\{\begin{array}{c}{A}^{+}·w·\left[1+tanh\left(-\frac{\Delta t-{\mu }^{+}}{{\tau }^{+}}\right)\right]\mathrm{if}\Delta t>0;\hfill \\ A^{-}·w·\left[1+tanh\left(\frac{\Delta t-{\mu }^{-}}{{\tau }^{-}}\right)\right]\mathrm{if}\Delta t<0.\hfill \end{array}\right.$$

(2)

In Equation (2), $A^{+}=0.074$, $A^{-}=-0.047$, ${\mu }^{+}=26.7$ ms, ${\mu }^{-}=-22.3$ ms, ${\tau }^{+}=9.3$ ms, ${\tau }^{-}=10.8$ ms.

$$\Delta w(\Delta t)=\left\{\begin{array}{c}\frac{|\Delta t|}{\tau }{\alpha }^{+}{e}^{-{\beta }^{+}\left(\frac{w_{max}-w}{w_{max}-w_{min}}\right)}{e}^{-{\gamma }^{+}{\left(\frac{\Delta t}{\tau }\right)}^2}\mathrm{if}\Delta t>0;\hfill \\ \frac{|\Delta t|}{\tau }{\alpha }^{-}{e}^{-{\beta }^{-}\left(\frac{w-w_{min}}{w_{max}-w_{min}}\right)}{e}^{-{\gamma }^{-}{\left(\frac{\Delta t}{\tau }\right)}^2}\mathrm{if}\Delta t<0.\hfill \end{array}\right.$$

(3)

Here, $\tau =10$ ms, ${\alpha }^{+}=0.316$, ${\alpha }^{-}=-0.011$, ${\beta }^{+}=2.213$, ${\beta }^{-}=-5.969$, ${\gamma }^{+}=0.032$, ${\gamma }^{-}=0.146$, $w_{max}=1$, and $w_{min}=0$.

Both memristive plasticity rules are shown in Figure 4. It can be seen that the models differ both in their dependence on the initial weight and in their spread along the $\Delta t$ axis: NC plasticity is localized within the $\left[-25,25\right]$ ms range and is relatively symmetric, while the PPX plasticity covers a much larger $\Delta t$ range and exhibits asymmetric behavior depending on the initial weight and the sign of $\Delta t$. Due to these differences, the training process differed for both rules, thus facilitating a broader study of the capabilities and limitations of the proposed methods.

Additionally, we considered a classical additive spike timing-dependent plasticity (STDP) [30] model to study the impact of sparcity on the memristor-based network in relation to the simpler synapse models.

3.3. Spiking Classification Models

Within the framework of the frequency approach to encoding input data, we considered a hybrid architecture consisting of a three-layer Winner Takes All (WTA) network [21], and this serves as a feature extraction module in combination with a formal classifier.

The WTA network is based on three layers (see Figure 5). The input layer consists of spike generators that convert input vectors into spike sequences according to the algorithm described above. The size of the input layer corresponds to the size of the input vector after preprocessing steps. The generated spike sequences are transmitted to the layer of leaky integrate-and-fire (LIF) neurons with an adaptive threshold (excitatory layer). This layer is connected to the input via trainable weights with one of the previously described plasticities according to the “all-to-all” rule. The number of neurons in the excitatory layer can be optimized depending on the complexity of the problem being solved. In turn, the excitatory layer is connected to the third layer of non-adaptive LIF neurons of the same size, which is called the inhibitory layer. Connections from the excitatory to the inhibitory layer are not trainable and have a fixed positive weight $w_{\mathrm{syn},}\mathrm{exc}>0$. In this case, each neuron in the excitatory layer is connected to a single neuron (partner) in the inhibitory layer. The connections directed from the inhibitory layer to the excitatory layer are called inhibitory connections. These connections are static and have a weight $w_{\mathrm{syn},}\mathrm{inh}<0$. Each neuron in the inhibitory layer is connected to all neurons in the excitatory layer except to its partner. Finally, the generators in the input layer are also connected to inhibitory neurons by randomly distributed static links with a weight $w_{\mathrm{syn},}\mathrm{gen}>0$. In all our experiments, the number of such connections was equal to 10% of the number of connections between the input and excitatory layers.

The spiking neural network was implemented using the NEST simulator [31].

We chose logistic regression (LGR), which was optimized for multi-class problems, and we used the one-versus-all (OVR) scheme as the formal classifier [32].

In this work, we considered two methods for reducing the connectivity in the WTA network: an ensemble of several classifiers trained using the bagging technique and sparse connectivity between layers.

3.3.1. Classification Ensemble

The bagging method was chosen as an ensemble creation technique; several identical classifiers were trained on subsets of input data, after which their predictions were then aggregated by voting. This method has several advantages compared to using a single larger network; in particular, it reduces the total number of connections within the network and increases the classification speed due to parallelization. In addition, it allows you to break unwanted correlations in the training dataset, thus resulting in improved architecture stability.

Connectivity within an ensemble is controlled using the following parameters:

–

$n_{\mathit{estimators}}$: Defines the number of models within the ensemble.

–

max_features: Determines the proportion of input features that are passed to the input of each of the models in the ensemble.

In addition, the bagging architecture allows one to regulate the number of examples on which each network is trained using the max_samples parameter.

The ensemble was implemented using the BaggingClassifier method of the Scikit-Learn [25] library. In all experiments, which were based on the preliminary empirical observations, the parameters $max\_features=1.0$ and $max\_samples=0.7$ were fixed.

3.3.2. Sparse Connectivity

Another way through which to reduce the connectivity of a spike network is to set a rule that allows you to regulate the number of connections and their organization. To this end, we formally placed the neurons of the excitatory and inhibitory layers on two-dimensional square grids of a 1 mm × 1 mm size (the dimension was chosen for the convenience of further presentation), and these were oriented mirror-image relative to each other. The neurons on the grids may be arranged irregularly. The initialization of sparse connections occurred according to the following algorithm:

(1)

The presynaptic neuron projects onto the plane of the postsynaptic neurons.

(2)

The projection of the presynaptic neuron becomes the center of a circular neighborhood, and all postsynaptic neurons within which will be connected to this presynaptic neuron with some probability.

This process is shown schematically in Figure 6.

Thus, connectivity within the network is regulated using two parameters:

–

Probability P of connection formation between pre- and post-synaptic neurons.

–

The radius of the circular neighborhood is R. This parameter is defined only for connections between the inhibitory and excitatory layers since neurons in the input layer do not have a spatial structure.

Both methods, as well as their combination, are expressed in a generalized form in Algorithm 1. This algorithm can be used for both proposed methods, as well as their combination, since the behavior of the network is determined by the parameters $n_{\mathrm{estimators}}$, $P_{\mathrm{gen}-\mathrm{exc}}$, $P_{\mathrm{inh}-\mathrm{exc}}$, and $R_{\mathrm{inh}-\mathrm{exc}}$. In the case of a sparse WTA network, an ensemble consists of one network ($n_{\mathrm{estimators}}=1$); furthermore, in the case of a bagging ensemble of fully-connected WTA networks, randomized sparse connectivity is not applied ($P_{\mathrm{gen}-\mathrm{exc}}=P_{\mathrm{inh}-\mathrm{exc}}=R_{\mathrm{inh}-\mathrm{exc}}=\mathbf{None}$). A combined approach, therefore, requires specifying all four parameters. The generalized decoding procedure using a logistic regression approach is detailed in Algorithm  2.

Algorithm 1 SNN learning process

Input: training data matrix $X^{\mathrm{train}}$ with dimensions $(M,N)$ of the preprocessed input vectors ${\mathbf{x}}_i$ of each sample in the dataset, neuron parameters, plasticity parameters, and synapse parameters Training parameters $\mathrm{epochs}=1$. Sparsity parameters: $n_{\mathrm{estimators}}$, max_features, max_samples, $n_{\mathrm{neurons}}$, $P_{\mathrm{gen}-\mathrm{exc}}$, $P_{\mathrm{inh}-\mathrm{exc}}$, and $R_{\mathrm{inh}-\mathrm{exc}}$ Network parameters: Table 1 Output data An ensemble containing $n_{\mathrm{estimators}}$ spiking neural networks, which are vectors of neuron activity frequencies in the excitatory layer for each example of the training set ${\mathbf{v}}_i$. 1: Generate weights 2: if $P_{\mathrm{gen}-\mathrm{exc}}$ or ($P_{\mathrm{inh}-\mathrm{exc}}$ and $R_{\mathrm{inh}-\mathrm{exc}}$) then 3:    Generate randomized sparse weights based on the provided sparsity parameters. 4: else 5:    Generate fully connected initial weights. 6: end if 7: Neural network initialization: neurons, synapses, and initial weights. 8: Bagging: randomly attribute at most $max\_samples·M$ samples with $max\_features·N$ features to each of the networks in the ensemble. 9: for k in $epochs$ do 10:    for each ${\mathbf{x}}_i$ in $X^{train}$ do 11:      for each $k_{ij}$ in ${\mathbf{x}}_i$ do 12:         Generating spikes sequences ${\mathbf{x}}_{ij}^{\mathrm{seq}}$ with length $t_{\mathrm{e}}$ and frequency $f_{ij}$. 13:      end for 14:      Simulating all SNNs in the ensemble that are bound to ${\mathbf{x}}_i$ during $t_{\mathrm{e}}$ time steps using spike sequences array ${\mathbf{x}}_i^{seq}$. 15:      Simulating SNN without input signal during $t_{\mathrm{p}}$ time steps for membrane potential to relax to initial value. 16:    end for 17: end for 18: Stop updating weights. 19: Collecting and saving frequency vectors of excitatory neuron layer activities ${\mathbf{f}}_i$ during presenting samples of input data. 20: Return An ensemble of SNN models; vectors of neuronal spiking frequencies ${\mathbf{f}}_i$.

Algorithm 2 SNN decoding process

Input: a collection F containing output frequencies of the excitatory layers of SNNs in the ensemble corresponding to each sample in the dataset, where each element ${\mathbf{f}}_i$ contains $K_i$ frequency vectors obtained from the $K_i$ SNNs in the ensemble. Output data: a vector C of predicted class labels for each sample in the dataset. 1: for each ${\mathbf{f}}_i$ in F do 2:    for each ${\mathbf{f}}_{ik}$ in ${\mathbf{f}}_i$, where $k\in [0,K_i]$ do 3:      Applying the logistic regression model to predict the class label ${\mathbf{c}}_{ik}$ for the sample ${\mathbf{v}}_i$. 4:    end for 5:    Apply the most common element voting scheme to the obtained collection of class labels ${\mathbf{c}}_{ik}$: the final predicted class is determined as the most frequently occurring class. 6:    Record the resulting class label to the vector C. 7: end for 8: Return a vector of predicted class labels C.

Table 1. Experimental settings for the WTA network.

		Digits			FSDD
Reduction Type	Parameter	STDP	NC	PPX	STDP	NC	PPX
Base	norm	L2	STD	STD	L2	STD	L2
	$n_{\mathrm{fields}}$	5	5	5	10	10	10
	$n_{\mathrm{neurons}}$	550	550	550	550	550	550
	$n_{\mathrm{estimators}}$	1	1	1	1	1	1
	${\tau }_{\mathrm{m},}\mathrm{exc}$	130	130	130	130	130	130
	${\tau }_{\mathrm{m},}\mathrm{inh}$	30	30	30	30	30	30
	frequency	600	350	450	800	800	800
	$t_{\mathrm{ref},}\mathrm{exc}$	5	4	6	5	4	4
	$t_{\mathrm{ref},}\mathrm{inh}$	3	3	3	3	3	3
	$w_{\mathrm{syn},}\mathrm{exc}$	18	20	20	20	20	20
	$w_{\mathrm{syn},}\mathrm{inh}$	−13	−15	−15	−13	−15	−13
Sparse Conn.	$P_{\mathrm{gen}\_\mathrm{exc}}$	0.4	0.4	0.4	0.4	0.4	0.4
	$P_{\mathrm{inh}\_\mathrm{exc}}$	0.4	0.4	0.4	0.4	0.4	0.4
	$R_{\mathrm{inh}\_\mathrm{exc}}$	0.9	0.9	0.9	0.9	0.9	0.9
Bagging	$n_{\mathrm{neurons}}$	50	50	50	50	50	50
	$n_{\mathrm{estimators}}$	11	11	11	11	11	11
Bagging + Sparse	$n_{\mathrm{neurons}}$	50	50	50	50	50	50
	$n_{\mathrm{estimators}}$	11	11	11	11	11	11
	$P_{\mathrm{gen}\_\mathrm{exc}}$	0.7	0.7	0.7	0.7	0.7	0.7
	$P_{\mathrm{inh}\_\mathrm{exc}}$	0.7	0.7	0.7	0.7	0.7	0.7
	$R_{\mathrm{inh}\_\mathrm{exc}}$	0.9	0.9	0.9	0.9	0.9	0.9

Table 1. Experimental settings for the WTA network.

		Digits			FSDD
Reduction Type	Parameter	STDP	NC	PPX	STDP	NC	PPX
Base	norm	L2	STD	STD	L2	STD	L2
	$n_{\mathrm{fields}}$	5	5	5	10	10	10
	$n_{\mathrm{neurons}}$	550	550	550	550	550	550
	$n_{\mathrm{estimators}}$	1	1	1	1	1	1
	${\tau }_{\mathrm{m},}\mathrm{exc}$	130	130	130	130	130	130
	${\tau }_{\mathrm{m},}\mathrm{inh}$	30	30	30	30	30	30
	frequency	600	350	450	800	800	800
	$t_{\mathrm{ref},}\mathrm{exc}$	5	4	6	5	4	4
	$t_{\mathrm{ref},}\mathrm{inh}$	3	3	3	3	3	3
	$w_{\mathrm{syn},}\mathrm{exc}$	18	20	20	20	20	20
	$w_{\mathrm{syn},}\mathrm{inh}$	−13	−15	−15	−13	−15	−13
Sparse Conn.	$P_{\mathrm{gen}\_\mathrm{exc}}$	0.4	0.4	0.4	0.4	0.4	0.4
	$P_{\mathrm{inh}\_\mathrm{exc}}$	0.4	0.4	0.4	0.4	0.4	0.4
	$R_{\mathrm{inh}\_\mathrm{exc}}$	0.9	0.9	0.9	0.9	0.9	0.9
Bagging	$n_{\mathrm{neurons}}$	50	50	50	50	50	50
	$n_{\mathrm{estimators}}$	11	11	11	11	11	11
Bagging + Sparse	$n_{\mathrm{neurons}}$	50	50	50	50	50	50
	$n_{\mathrm{estimators}}$	11	11	11	11	11	11
	$P_{\mathrm{gen}\_\mathrm{exc}}$	0.7	0.7	0.7	0.7	0.7	0.7
	$P_{\mathrm{inh}\_\mathrm{exc}}$	0.7	0.7	0.7	0.7	0.7	0.7
	$R_{\mathrm{inh}\_\mathrm{exc}}$	0.9	0.9	0.9	0.9	0.9	0.9

The classification process is additionally visualized in Figure 7. Depending on the experimental settings, the number of the models in the ensemble may be equal to one (base WTA and sparse WTA), and the weight initialization may be performed in accordance with the sparse connectivity approach or in a fully connected manner.

4. Experiments and Results

Experiments on the Digits dataset were conducted using hold-out cross-validation; then, 20% of the training examples were used for testing. On FSDD, a fixed testing dataset was used. For all experiments, such parameters as the number of neurons in the networks, the number of receptive fields, and the number of networks in the ensemble were selected for each plasticity and for each dataset by maximizing the training classification accuracy. The selection was performed automatically using the tree-parzen estimator (TPE) algorithm implemented in HyperOpt [33] (an open-source Python package). For all of the three methods under consideration, the parameters across plasticity models and datasets were fixed, thus ensuring a fair comparison. The source code of all the connectivity reduction methods is available in our repository (see Funding information below).

In Table 1, we present the hyperparameters of SNN that were used for each of the considered datasets and plasticity models:

–

norm—the input normalization method: L2 or standard scaling (STD);

–

$n_{\mathrm{fields}}$—the number of Gaussian receptive fields (GRFs);

–

$n_{\mathrm{neurons}}$—the number of excitatory neurons in the network;

–

$n_{\mathrm{estimators}}$—the number of networks in the bagging ensemble (for the ensemble approach specifically, everywhere else it is equal to 1);

–

${\tau }_{\mathrm{m},\mathrm{exc}}$ and ${\tau }_{\mathrm{m},\mathrm{inh}}$—the characteristic time of the membrane potential decay for the excitatory and inhibitory neurons in milliseconds;

–

frequency—the maximal spiking frequency of the Poisson generators;

–

$t_{\mathrm{ref},\mathrm{exc}}$ and $t_{\mathrm{ref},\mathrm{inh}}$—the refractory time for the excitatory and inhibitory neurons in milliseconds;

–

$w_{\mathrm{syn},\mathrm{exc}}$ and $w_{\mathrm{syn},\mathrm{inh}}$—the synaptic weights of the excitatory-to-inhibitory and inhibitory-to-excitatory connections, respectively;

–

$P_{\mathrm{gen}-\mathrm{exc}}$—the probability of forming a connection between an input and an excitatory neuron in the sparse WTA network;

–

$P_{\mathrm{inh}-\mathrm{exc}}$—the probability of forming a connection between an inhibitory and an excitatory neuron in the sparse WTA network;

–

$R_{\mathrm{inh}-\mathrm{exc}}$—the projection radius for inhibitory-to-excitatory connections in the sparse WTA network.

In all experiments, the time for submitting one example to the WTA network $t_{\mathrm{e}}$ was 350 ms, which was followed by a relaxation period $t_{\mathrm{p}}$ of 50 ms, thereby resulting in a processing time of $t_{\mathrm{e}}+t_{\mathrm{p}}=400$ ms per sample, where learning took place over one epoch. As a baseline, we conducted an experiment using the classical WTA network (see Section 3.3).

Additionally, we presented the number of connections within the network with a breakdown by type of the pre- and postsynaptic neurons.

Table 2. Connectivity within the base WTA network.

Connection	Digits	FSDD
Gen-to-Exc	176,000	165,000
Exc-to-Inh	550	550
Inh-to-Exc	301,950	301,950
Gen-to-Inh	17,600	16,500

demonstrates the connectivity within the base WTA network. The connections for the base network and the bagging ensemble were calculated, as shown in Equation (4). In the equations, $N_{\mathrm{inp}}$ is equal to the number of input features, $N_{\mathrm{exc}}$ denotes the number of excitatory neurons, $p_{\mathrm{Gen}-\mathrm{to}-\mathrm{Inh}}=0.1$ is a fraction of the inhibitory neurons connected to the generator layer, $N_{\mathrm{ens}}$ is the number of estimators within the ensemble, and $N_{\mathrm{ens}}=1$ is for the base WTA network. For the sparse WTA network, the connections were counted manually from the weight checkpoints due to their stochastic nature.

$$\begin{array}{c}{N}_{\mathrm{Gen}-\mathrm{to}-\mathrm{Exc}}=N_{\mathrm{inp}}·N_{\mathrm{exc}}·N_{\mathrm{ens}},\\ N_{\mathrm{Exc}-\mathrm{to}-\mathrm{Inh}}=N_{\mathrm{exc}}·N_{\mathrm{ens}},\\ N_{\mathrm{Inh}-\mathrm{to}-\mathrm{Exc}}=N_{\mathrm{exc}}·(N_{\mathrm{exc}}-1)·N_{\mathrm{ens}},\\ N_{\mathrm{Gen}-\mathrm{to}-\mathrm{Inh}}=p_{\mathrm{Gen}-\mathrm{to}-\mathrm{Inh}}·N_{\mathrm{inp}}·N_{\mathrm{exc}}·N_{\mathrm{ens}}.\end{array}$$

(4)

Here, $N_{\mathrm{inp}}$ is equal to the number of input features, $N_{\mathrm{exc}}$ denotes the number of excitatory neurons, $p_{\mathrm{Gen}-\mathrm{to}-\mathrm{Inh}}=0.1$ is a fraction of the inhibitory neurons connected to the generator layer, $N_{\mathrm{ens}}$ is the number of estimators within the ensemble, and $N_{\mathrm{ens}}=1$ is for the base WTA network. For the sparse WTA network, the connections were counted manually from the weight checkpoints due to their stochastic nature.

After applying the considered methods for reducing the number of connections in the WTA network, the connectivity of the resulting models was obtained, as presented in

Table 3. Connectivity within the sparse WTA network.

Reduction Type	Connection	Digits			FSDD
		STDP	NC	PPX	STDP	NC	PPX
Bagging	Gen-to-Exc	176,000	176,000	176,000	165,000	165,000	165,000
	Exc-to-Inh	550	550	550	550	550	550
	Inh-to-Exc	26,950	26,950	26,950	26,950	26,950	26,950
	Gen-to-Inh	17,600	17,600	17,600	16,500	16,500	16,500
Sparse Conn.	Gen-to-Exc	70,400	70,503	70,495	65,929	65,739	66,651
	Exc-to-Inh	550	550	550	550	550	550
	Inh-to-Exc	113,041	112,294	112,282	112,207	112,499	112,522
	Gen-to-Inh	17,600	17,600	17,600	16,500	16,500	16,500
Bagging + Sparse	Gen-to-Exc	123,112	124,300	123,750	116,006	115,258	115,214
	Exc-to-Inh	550	550	550	550	550	550
	Inh-to-Exc	17,732	18,502	17,996	18,161	18,106	17,886
	Gen-to-Inh	17,600	17,600	17,600	16,500	16,500	16,500

.

The experimental results for applying different types of reduction methods are presented in

Table 4. Experimental results for the WTA network. The maximum F1-scores for each experiment are highlighted in bold.

Reduction Type	Digits			FSDD
	STDP	NC	PPX	STDP	NC	PPX
Base	0.85	0.95	0.91	0.93	0.92	0.84
Bagging	0.95	0.94	0.94	0.93	0.89	0.90
Sparse Conn.	0.94	0.93	0.94	0.84	0.85	0.92
Bagging + Sparse	0.87	0.93	0.94	0.91	0.88	0.89

, and they are also expressed using the F1-score metric, which is defined in Equation (5).

$$\mathrm{F}1\_\mathrm{score}=\frac{2·\mathrm{Precision}·\mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}.$$

(5)

The obtained accuracies are consistent with and superior to the results reported in our previous works on spiking neural networks with memristive plasticity and without sparse connectivity:

Digits

0.83–0.86 F1 from the article [23]. The best performance was obtained by a one-layer SNN with 1600 neurons and 2,660,800 connections (vs. the bagging WTA SNN model with 550 neurons and 221,000 connections)

FSDD

0.81–0.93 F1 from the article [34], where the best scores were achieved by WTA SNN with 400 neurons and 243,600 connections. For comparison, the proposed sparse bagging WTA SNN model containing 550 neurons and 151,217 connections achieved the same performance (F1: 0.93).

It also follows from

Table 5. Comparison of results for both of the classification tasks. * indicates the accuracy metric. The maximum performance scores for our experiments and literature sources are highlighted in bold.

Method	Spiking	Performance Scores
Digits
RandomForest [35]	No	0.98 *
Quantum neural network [36]	No	0.93 *
Decision Tree [37]	No	0.86 *
One-layer SNN with frequency encoding [23]	Yes	0.86
Ours (Bagging STDP) (this paper)	Yes	0.95
Ours (Bagging + Sparse PPX) (this paper)	Yes	0.94
FSDD
Two-layer convolution neural network (CNN) (eight classes) [38]	No	0.80 *
Hopf reservoir computer with a CNN readout [39]	No	0.97 *
Liquid state machine [40]	Yes	0.88 *
WTA SNN with frequency encoding [34]	Yes	0.93
Ours (Bagging STDP) (this paper)	Yes	0.93
Ours (Sparse PPX) (this paper)	Yes	0.92

that the resulting SNN models with sparse connectivity and memristive plasticity demonstrated high accuracy compared to the other algorithms, including non-spiking ones.

5. Discussion

To evaluate the effectiveness of the connectivity reduction methods, we introduced the connectivity index $\kappa$, as defined in Equation (6):

$$\kappa =\frac{N_{\mathrm{sparse}}}{N_{\mathrm{base}}}.$$

(6)

Here, $N_{\mathrm{sparse}}$ and $N_{\mathrm{base}}$ are the total number of connections in the sparse network and the equivalent fully connected WTA network, respectively. Based on this definition, the efficiency of the connectivity reduction method can be assessed by calculating the ratio of the classification accuracy to the connectivity index (see Equation (7), where the efficiency is represented by the index $\eta$).

$$\eta =\frac{\mathrm{F}1\_\mathrm{score}}{\kappa }.$$

(7)

The values of the connectivity and efficiency indices for different datasets, plasticities, and network types are presented in

Table 6. Efficiency estimation of the different sparsity types. The best $\eta$ values are marked in bold.

Reduction Type	Plasticity	Dataset	$\mathit{\kappa }$	$\mathit{\eta }$
Bagging	STDP	Digits	0.45	2.13
	NC		0.45	2.11
	PPX		0.45	2.11
	STDP	FSDD	0.43	2.15
	NC		0.43	2.06
	PPX		0.43	2.08
Sparse Conn.	STDP	Digits	0.41	2.31
	NC		0.41	2.30
	PPX		0.41	2.29
	STDP	FSDD	0.40	2.08
	NC		0.40	2.11
	PPX		0.40	2.28
Bagging + Sparse	STDP	Digits	0.32	2.71
	NC		0.33	2.80
	PPX		0.32	2.92
	STDP	FSDD	0.31	2.91
	NC		0.33	2.68
	PPX		0.31	2.86

. The motivation for the proposed indices lies in assessing the accuracy-per-connection ratio, which is a more robust comparison metric for the proposed methods compared to raw accuracy.

From the table above, it follows that, in our experiments, the relative efficiency of spike networks with sparse connectivity in comparison to ensembles of spike networks was found to be slightly higher; on average, across plasticities and datasets, the efficiency of the sparsely connected WTA network was 2.2, while the efficiency of bagging was, on average, equal to 2.1. However, due to the small scale of this difference, we concluded that both methods can be effectively used to reduce connectivity depending on the specifics of the problem and the hardware requirements. If the experimental setting facilitates only the reduction in static connections and supports ensembles, bagging is a preferable option, while sparse connectivity may be used in situations where only a single larger network is feasible.

The combination of these two methods yielded the highest efficiency of 2.8, on average, with an average connectivity index equal to 0.32. Therefore, if combining both approaches is possible for a given problem, the resulting accuracy-per-connection efficiency will be the highest.

6. Conclusions

In this work, we compared two approaches to connectivity reduction in memristive spiking neural networks: the bagging ensemble technique and probabilistic sparse connectivity. Using a three-layer WTA network, we demonstrated that both methods achieved competitive performance on the handwritten digits and spoken digits classification tasks, with a combination of both approaches achieving the highest efficiency. On the Digits dataset, the bagging ensemble yielded an F1-score of 0.95, 0.94, and 0.94 for the STDP, NC, and PPX plasticity rules, respectively, while the sparse WTA network achieved 0.94, 0.93, and 0.94, respectively; furthermore, a combined Bagging + Sparse model, in turn, yielded F1-scores of 0.87, 0.93, and 0.94, respectively. On FSDD, the F1-score values lay within the 0.89–0.93 range for the ensemble of WTA networks, within the 0.84–0.92 interval for the sparse WTA network, and within the 0.88–0.91 range for the combined model. The resulting models were found to be superior in accuracy to well-known spiking neural network solutions, and they also corresponded to the level of other non-spike algorithms. Additionally, by studying the ratio between the proposed connectivity index and the F1-score, we showed that the bagging ensemble and the sparse WTA network achieved an almost equal efficiency, while the combination of both methods yielded a 20% higher average efficiency coefficient value.

Thus, the created combination of methods can be used as a computational technology for creating spike neural network models for implementation on neurochips in inference mode. Also, the developed architectures of spiking neural networks can be used for the subsequent implementation of online learning on neuromorphic chips with memristive connections. In our future research, we plan to expand the scope of the classification problems for image and audio data that can be solved using the proposed methods (e.g., the CIFAR-10, Google Speech Commands dataset, etc.), as well as work on hardware implementations of the designed networks.