Analog Implementation of a Spiking Neuron with Memristive Synapses for Deep Learning Processing

Abstract

Analog neuromorphic prototyping is essential for designing and testing spiking neuron models that use memristive devices as synapses. These prototypes can have various circuit configurations, implying different response behaviors that custom silicon designs lack. The prototype’s behavior results can be optimized for a specific foundry node, which can be used to produce a customized on-chip parallel deep neural network. Spiking neurons mimic how the biological neurons in the brain communicate through electrical potentials. Doing so enables more powerful and efficient functionality than traditional artificial neural networks that run on von Neumann computers or graphic processing unit-based platforms. Therefore, on-chip parallel deep neural network technology can accelerate deep learning processing, aiming to exploit the brain’s unique features of asynchronous and event-driven processing by leveraging the neuromorphic hardware’s inherent parallelism and analog computation capabilities. This paper presents the design and implementation of a leaky integrate-and-fire (LIF) neuron prototype implemented with commercially available components on a PCB board. The simulations conducted in LTSpice agree well with the electrical test measurements. The results demonstrate that this design can be used to interconnect many boards to build layers of physical spiking neurons, with spike-timing-dependent plasticity as the primary learning algorithm, contributing to the realization of experiments in the early stage of adopting analog neuromorphic computing.

1. Introduction

In recent years, generative AI has become the most popular application of artificial intelligence (AI) for the general public. This technology has showcased unprecedented capabilities in text [1], video, and image generation [2]; however, these advancements come with a significant caveat: the power consumption required to operate. The power consumption expended during GPT-3 training is estimated at around 1287 MWh [3]. This energy consumption is deemed unsustainable and will only increase as AI-enabled devices and services increase, consequently scaling to more complex generative models. To address the problem, novel computing architectures for AI are being developed, showing a dramatic increase in power efficiency. The area of computing that deals with novel computing architectures is referred to as neuromorphic computing [4]. Neuromorphic computing draws inspiration from the human brain to replicate its functions and structures on complementary metal oxide semiconductor (CMOS) technology devices, using high neuron connectivity, sparse processing, and distributed computing [5,6,7].

This hardware approach utilizes the analog domain to handle multiple inputs and react asynchronously, achieving this by employing a bioplausible realization of neuron cells at the silicon level, called spiking neurons [8]. These neurons can process information sparsely with minimal power consumption, higher noise tolerance, and combine memory and processing on a single silicon die (edge computing) [9,10,11]. They use only a few milliwatts of power while providing parallel processing capabilities and integrating thousands of neurons. Another missing link in the search for efficient computing for AI is the implementation of artificial synapses. The space between neurons, which enables information transmission between neurons and reinforces the activity between them, is called a synapse [12]. Nowadays, many device alternatives with different materials have been proposed to achieve this end goal, such as phase change memories (PCM) [13], floating-gate transistors [14], and memristors [15].

Memristors are two-terminal electrical devices that exhibit a functional relationship between charge and flux. When a voltage difference, $V\left(t\right)$, is applied across their terminals, it results in the movement of charged dopants between doped and undoped regions, causing a change in the device’s resistance value [16]. These devices can process and store information simultaneously while typically consuming nano-amperes of current; this is useful in computing systems where this process is called multiply–accumulate (MAC) operation and is related to matrix multiplication and storage; these devices can reduce the energy consumption of computing systems while improving the reliability [17] and can be utilized in different circuits to mimic the behavior of biological synapses [18,19,20].

This paper presents an analog hardware implementation of a spiking neuron architecture, employing commercial components and memristive devices acting as artificial synapses. Section 1 provides a brief introduction. Section 2 presents the neuron and synaptic modeling, as well as some concepts regarding the synaptic weight adjustment in neurons and the electrical characteristics of the proposed neuron. Section 3 presents the simulation results, the methodology utilized to characterize the neuron, the results obtained with the physical implementation, and the synaptic weight adjustment. Lastly, Section 4 discusses the results and future work.

2. Materials and Methods

In this section, we will review the modeling of neurons and memristive synapses, the concept of weight adjustment in neurons, and the learning mechanisms. Lastly, we will cover the implementation of spiking neurons and the selection of components.

2.1. Neuron Modeling

Spiking neurons are relevant in computational neuroscience and artificial neural networks. They aim to replicate the behavior of neurons in the brain, where information is transmitted through brief electrical pulses, known as spikes or action potentials. These spikes act as the fundamental units of communication between neurons. In a spiking neuron model, the neuron’s membrane potential changes in response to inputs from other neurons or outside stimuli. Once the membrane potential reaches a specific threshold, the neuron generates an output spike, which is transmitted to other connected neurons. After spiking, the membrane potential resets to a resting state, and the process starts anew.

The Hodgkin and Huxley model [21] accurately describes the behavior of neurons through four differential equations. This has high fidelity with respect to biological neurons’ behavior, but it comes at a high computational cost. Leaky integrate-and-fire [22] is a simpler model compared with the Hodgkin and Huxley model, but it maintains the overall behavior of the neurons. This behavior is modeled by Equation (1), where $v_m\left(t\right)$ is the membrane’s potential, $E_L$ is the membrane’s potential at rest, ${\tau }_m$ is the time constant of charging of the neuron, which is represented by $R_m{C}_m$, where $R_m$ is the membrane’s resistance and $C_m$ is the membrane’s capacitance. $I_{syn}\left(t\right)$ is the excitatory input current of the neuron. This current charges the neuron’s capacitance, modifying the neuron’s voltage, $v_m\left(t\right)$. Once this voltage surpasses a threshold $v_{th}$, an output spike is generated, and $v_m\left(t\right)$ takes the value of $E_L$.

$${\tau }_m\frac{d_{v_m}\left(t\right)}{dt}=E_L-v_m\left(t\right)+R_m{I}_{syn}\left(t\right)$$

(1)

2.2. Memristor Modeling

Memristors are two-terminal electronic components that exhibit a relationship between the charge and the magnetic flux at their terminals [23]. They were theorized by Leon Chua in 1971 [16] and are considered the fourth fundamental passive circuit element alongside resistors, capacitors, and inductors. Electrical modeling of memristors involves mathematically representing their behavior to understand and predict their performance in circuits. The first mathematical models for memristors were developed from a memristor’s equivalent electrical circuit consisting of two resistors connected in series [24]; improvements were proposed with the use of window functions with adjustable parameters [25]; this improved the convergence of the model in corner cases where the regions between the doped and undoped semiconductors reach the limit of the thin dielectric layer.

Vourkas Memristor Model

In this subsection, the macro model for a memristor proposed by [26] will be explained. The schematic diagram of a memristor macromodel is depicted in Figure 1a. Figure 1b represents the typical electrical symbol of a memristor. The cross-section of a memristor is shown in Figure 1c, in which a region of insulating material of TiO₂ separates two metal plates that form the terminals of the device.

Two regions of material with different conductivity are distinguished inside the thin insulating material. The region denoted by $L_0$ represents the total thickness of the insulating material. On the other hand, L is the distance between the border of the two regions of the TiO₂ composite material, with respect to the border with the lower terminal (metallic contact). The upper region of the TiO₂ material develops a low resistance of R value due to the vacancies of oxygen atoms that occur due to the migration of atoms in response to the effect of the intense electric field present in the structure. The upper region is, therefore, considered a doped region, extending a distance with a value of $L_0-L$. On the other hand, the lower region that receives the oxygen atoms behaves as an insulator and develops a very high resistance, with a value denoted by $R_t$. In the region with thickness L, a charge transport process develops through the tunneling mechanism, coupled in series with the upper region ($L_0-L$), with a lower resistive value. The resistance values in both regions are very different, with a notable difference in $R_t\gg R$, so the Vourkas model focuses on determining only the value of $R_t$ and is considered to develop a proportional resistive value to the width of the tunnel barrier L and that the electron conduction would be dominated by the effective width of the tunnel barrier, which varies as a function of time and the magnitude of voltage applied between the terminals of the device. Also, it is considered that the border between the two regions, and, therefore, the value of L, changes depending on the migration of oxygen deficiencies due to the effect of an electrical potential applied between the terminals.

The macromodel behavior proposed by [26] is defined by Equation (2):

$$I\left(t\right)=G(L,t)V_M\left(t\right)$$

(2)

where I is the current that flows in the memristor, G is the conductance, and $V_M$ is the voltage applied between the top and the bottom terminals. Next, the rate of change of L is defined by

$$\dot{L}=f(V_M,t)$$

(3)

In quantum mechanics theory, the tunnel resistance, $R_t$, is determined by Equation (4), whose value is inversely proportional to the product of the transmission coefficient, $T_{0,V_M}$, which is voltage-dependent, and the effective density of states, $N_{eff}$, of the $Ti{O}_2$ material. $T_{0,V_M}$ is a function of $V_M$, which provides sufficient energy for the electrons to pass through the thin dielectric film (of size L) and approaches ${(1+\frac{m{V}_o{a}^2}{2{\hslash }^2})}^{-1})$, where m is the electron mass, ${\hslash }^2$ is the Planck’s constant, $V_o$ is the potential barrier of height, and a is the thickness of the one-dimensional square potential barrier system. On the other hand, the resistance $R_t$ is exponentially proportional to L.

$$R_t\left(V_M\right)=\frac{1}{N_{eff}·T_{0,V_M}}·exp(2k_{V_M}·L)$$

(4)

Equation (4) can be redefined as

$$R_t\left(L_{V_M,t}\right)=\frac{f_0}{L_{V_M,t}}·exp\left(2L_{V_M,t}\right)$$

(5)

Equation (5) defines a new voltage- and time-dependent parameter $L_{V_M,t}$ that replaces the voltage-dependent parameters $T_{0,V_M}$ and $k_{V_M}$ with no significant error implication to avoid dealing with theoretical values of some materials’ properties that commonly differ in experimental practice. Equation (5) calculates the resistance in the memristor, subject to the state variable L. The fitting parameter $f_0$ models specific material properties obtained by device characterization. The heuristic Equation (6) defines the state variable L. Time dependency of $L_{V_M,t}$ is evident after integrating over time Equation (7) through the 1 F capacitor shown in Figure 1. The resultant value passes to Equation (6).

$$L(V_M,t)=L_0·(1-\frac{m}{r(V_M,t)})$$

(6)

Equation (6) delivers the expected response of L as a function of time t and $V_M$. Therefore, $L_0$ represents the largest dimension that L can reach. On the other hand, the time- and voltage-dependent parameters, denoted as $r(V_M,t)$, and the term m (adjustment parameter) are used to determine the boundary of the tunnel barrier width. Equation (6) determines the initial and current position of L, being defined between the limits of the two boundary values. The parameter $r(V_M,t)$ defines both the dynamics of the device and its current state. Its value must be maintained between a valid interval. The on-and-off dynamics of the memristor depend on the internal device structure, mainly on the type of oxide used, which determines the drift speed of the ions that move along $L_0$ due to the effect of the applied electric field. These dynamics are incorporated into the macromodel assuming that the change in L is fast if $V_M$ exceeds a certain positive threshold voltage, denoted as $V_{SET}$ ($V_{SET}$  <  $V_M$ ), or a negative threshold voltage $V_{RESET}$ ($V_M$  <  $V_{RESET}$). Otherwise, if $V_M$ is below both thresholds, e.g., $V_{RESET}$  ≤  $V_M$  ≤   $V_{SET}$, the change in L and, therefore, in the memristance is practically zero. The function proposed in [26] that models the rate of change of L, as a function of the voltage $V_M$ and the time, t, is defined by Equation (7).

$$\dot{r}(V_M,t)=\left\{\begin{array}{ccc}{\alpha }_{RESET}·\frac{V_M+V_{RESET}}{c+|V_M+V_{RESET}|}\hfill & si& V_M\in [-V_0,V_{RESET})\\ \\ b·V_M\hfill & si& V_M\in [V_{RESET},V_{SET}]\\ \\ {\alpha }_{SET}·\frac{V_M-V_{SET}}{c+|V_M-V_{SET}|}\hfill & si& V_M\in (V_{SET},+V_0]\end{array}\right.$$

(7)

The parameters ${\alpha }_{SET}$, ${\alpha }_{RESET}$, b, and c are constants used to adjust the model dynamics. In this case, since the change in memristance is stated to be fast, if the voltage applied to the terminals of the memristor exceeds any of the threshold voltages of the memristor, then $a_x\gg b$ must be satisfied. The constant c will be bounded in the interval 0 < c < 1. In Figure 1a, the schematic diagram of the Vourkas macromodel is presented. The voltage-controlled current source, $G_r$, generates a current controlled by the voltage applied to the memristor terminals, $V_M$, proportional to $\dot{r}(V_M,t)$. This current is integrated through the unity value capacitor, $C_r$, thus providing the value of $r(V_M,t)$. The voltage-controlled current source, $G_{pm}$, evaluates Equation (5), for which $r(V_M,t)$ was previously calculated, through $G_r$ and, consequently, $L(V_M,t)$, through Equation (6). Finally, code block Listing 1 includes the SPICE macromodel presented in [26], which is used in this work in all the electrical simulations.

Listing 1. SPICE code for the Vourkas macromodel.

2.3. Synaptic Weight Adjustment in Neurons

Hebbian learning is the learning mechanism proposed by Donald Hebb, wherein, when a cell actively and repeatedly participates in the firing process of another cell, a change is induced in one or both cells, thus enabling the first cell to fire the second cell more efficiently. This mechanism is deemed the most probable mechanism for long-term learning [27]. The next sections outline the different mechanisms for synaptic weight adjustment observed in living organisms.

2.3.1. Long-Term Potentiation and Long-Term Depression on Memristive Synapses

Long-term potentiation (LTP) and long-term depression (LTD) involve the long-lasting strengthening or weakening of synaptic connections between neurons. LTP occurs when repeated stimulation makes a synapse more efficient at transmitting signals between neurons. It increases the strength of the synaptic connection, which can lead to enhanced neuronal communication and potentially improved learning and memory. LTD is the opposite process in which synaptic transmission is weakened over time due to low-frequency stimulation or lack of stimulation, leading to a decrease in the strength of the synaptic connection [28].

2.3.2. Spike-Timing-Dependent Plasticity

Spike-timing-dependent plasticity (STDP) is a neurobiological phenomenon observed in synaptic connections, where the precise timing of action potentials, the time window $\Delta$T, in the presynaptic and postsynaptic neurons, determines the direction and magnitude of changes in synaptic strength. Specifically, when the presynaptic neuron fires before the postsynaptic neuron, synaptic strength is potentiated, while, if the postsynaptic neuron fires before the presynaptic neuron, synaptic strength is depressed. This precise timing mechanism is crucial for refining neural circuits and encoding information in the brain. In artificial neural networks, STDP is an unsupervised learning rule where synaptic weight is modified based on the time window, $\Delta$T, between presynaptic and postsynaptic impulses [29]. The change in the synaptic weight is modeled by Equation (8), where $A_{+}$ and $A_{-}$ are constant values representing the potentiation and depression of neurons, ${\tau }_{+}$ and ${\tau }_{-}$ are constants influencing the slope of the function, and the time window $\Delta t$ is considered between presynaptic and postsynaptic impulses [30].

Spiking neural networks encode information in spike potentials rather than in a continuous variable like traditional deep neural networks (DNNs). In traditional DNNs, the training of weights is completed with gradient descent [31]; this optimization algorithm is effective and reliable, and the activation functions of the neurons must be continuous and differentiable to apply the chain rule [32]. The spiking neurons have continuous functions but are not differentiable at all points; under these circumstances, gradient descent cannot be utilized to adjust the synaptic weight; the STDP can modify the synaptic weight based on local information, the pre- and postsynaptic neurons [33].

$$STDP(\Delta t)=\Delta w=\left\{\begin{array}{cc}{A}_{+}exp\left(\frac{-\Delta t}{{\tau }_{+}}\right),& \Delta t>0\\ -A_{-}exp\left(\frac{-\Delta t}{{\tau }_{-}}\right),& \Delta t<0\end{array}\right.$$

(8)

Equations (4) and (5), presented in Section 2.2, model the change in tunnel resistance as a function of the time and the applied voltage to a memristor device. This model requires process-dependent parameters for a specific memristor device. Therefore, a change in the memristance, herein the tunnel resistance, corresponds to a change in the synaptic weight through its equivalent memristor conductance. The synaptic weight, modeled in Equation (8), on the other side, computes the change in the synaptic weight considering the time separation between the action potentials of two interconnected spiking neurons under certain predefined constant values. Consequently, using a SPICE memristor macromodel, presented in code block Listing 1, allows us to closely simulate the STDP learning rule, expressed in Equation (8). The STDP learning rule can be simulated in SPICE by simply plugging in a memristor macromodel, like the one introduced in Section 2.2, between two spiking neurons’ circuit cells, as shown in Figure 2. Circuit cells have to be able to feed back their output to their input whenever a spike is produced. Doing so potentiates the synaptic strength (memristor conductance) if the presynaptic neuron fires before the postsynaptic neuron. It depreciates the synaptic strength if the presynaptic neuron fires after the postsynaptic neuron. The key point in a CMOS circuit synaptic implementation is to maintain the connection of the memristor terminals to the output potentials present in the pre- and postsynaptic neurons to produce changes in the memristance as a function of the neuron spiking activity. The next subsection will explain the proposed circuit implementation to produce STDP learning.

2.4. Analog Leaky Integrate-and-Fire Functional Blocks and Control Signals

Numerous architectures for analog leaky integrate-and-fire (LIF) neurons have been documented in the literature. The main component of such architectures is an integrator, responsible for receiving input spikes, integrating them over time, and producing an output spike once a certain voltage threshold $v_{th}$ is attained. Various components, such as BJT transistors, silicon-controlled rectifiers [34], and floating-gate transistors [14], can be used as an integrator and trigger stage. In this work, we adopt the design proposed in [35] as a starting point. This design proposes a reconfigurable architecture where the integrator can function as a buffer, thus providing the ability to deliver a higher output current to feed other neurons in a subsequent layer and to feed back the membrane potential to the right-side memristors’ terminals connected to the input of the antecedent layer; see Figure 2. Self-directed channel (SDC) memristors are utilized to implement the synaptic weights between neurons; these devices are currently available and enable the implementation of proof of concept for this spiking neural network as an analog system implementation prototype.

The block diagram of the LIF neuron proposed in [35] is introduced in Figure 3.

This design employs an operational amplifier (opamp) that is initially configured as an integrator; once the integrated voltage reaches a threshold $v_{th}$, the voltage comparator triggers the generation of four control signals (${\varphi }_1,{\varphi }_2,{\varphi }_{int}$, and ${\varphi }_{fire}$). Two of these signals, ${\varphi }_{int}$ and ${\varphi }_{fire}$, modify the state of the voltage-controlled switches that, in turn, reconfigure the opamp to act as a buffer. This buffer propagates a generated spiking signal, which is suited to drive the memristors in the synapses.

The design of the LIF neuron presented in Figure 3 and the memristive synapse it drives are quite important because these elements can be used as the starting building blocks for a much wider array of spiking neurons and synapses, essentially to implement power-efficient architectures for deep learning neural networks (DNNs) in the analog domain, capable of being trained in situ with the STDP learning rule. An outline for such a DNN abstraction is presented in Figure 4, where multiple layers of neurons are interconnected as fully or sparsely connected by memristive synapses.

Phase Control for Reconfiguration of the LIF Neuron

Figure 3 shows the main components of the analog spiking neuron circuit. These are an opamp to implement an integrator circuit, a comparator to compare the membrane potential, $V_{mem}$, with a predefined voltage threshold, $V_{th}$, a spiking generator circuit block, and a control signals generator block (CSGB). A capacitor and a resistor are also part of the proposal, reproducing the membrane capacitance and the leaky resistance present in the LIF neuron. Using switches implemented with transmission gates (TGs), the LIF circuit changes its interconnections to behave like a leaky integrator during the integration phase and as a voltage buffer when the firing mechanism is activated when the membrane potential reaches $V_{th}$.

The state of the different TGs is controlled through the use of the CSGB. The CSGB presented in [35] is only behaviorally described by the time diagram presented in Figure 5. The control signals ${\varphi }_1$ and ${\varphi }_2$ are reminiscent of the behavior of two mono-stable timers in series, in which the end of the first mono-stable timer ${\varphi }_1$ activates the second timer ${\varphi }_2$. The ${\varphi }_{fire}$ signal represents the elapsed time between both timers, and the ${\varphi }_{int}$ is the opposite of the ${\varphi }_{fire}$ signal.

The proposed electrical circuit for the generation of the control signals ${\varphi }_1$, ${\varphi }_2$, ${\varphi }_{int}$, and ${\varphi }_{fire}$ is introduced in Figure 6; this circuit comprises two 555 timers configured as mono-stable timers. The first timer is triggered by the output signal of the LT1715 CMOS comparator, shown in Figure 3; this signal is reshaped using two CMOS inverters. The output of the first timer (${\varphi }_1$) is then fetched through a CMOS inverter and a resistor–capacitor (RC) circuit, which in turn eliminates all constant signals and only reacts with any change in voltage, functioning as an impulse function. This signal is reshaped and fetched to the second timer, producing the signal ${\varphi }_2$ at the output of this second timer. Signal ${\varphi }_{fire}$ can be viewed as the logic XOR operation of the signals ${\varphi }_1$ and ${\varphi }_2$. The implementation of a logic XOR gate requires only 4 CMOS transistors. By inverting the ${\varphi }_{fire}$ signal, we can obtain the ${\varphi }_{int}$ signal.

2.5. Electrical Characteristics of the Block Components of the LIF Neuron

A physical implementation of a LIF neuron requires the utilization of different commercial components. These components must be chosen under the specifications outlined in [35] and the availability of simulation models for LTSpice; the process is elaborated on in the subsequent sections.

2.5.1. Integration/Buffer and Comparator Module

In

Table 1. Comparison table of the main electrical characteristics for the integration and buffer opamp.

	Value Presented in [35]	LTC6268 [36]	ADA4860 [37]
Rise speed [$\frac{V}{\mu s}$]	784	400	695
Fall speed [$\frac{V}{\mu s}$]	500	260	560
DC Gain [dB]	72	-	-
Unity gain at frequency [MHz]	272	350	230

, the characteristics required in the integrator/buffer module considered in [35] are presented alongside the characteristics of the selected components. Considering the characteristics of the commercially available devices, the search is narrowed down to two components, the ADA4680 and the ALTC6268. Although the ADA4680 has greater rise and fall speed, it does so with a lower output voltage of 2 volts peak to peak, in comparison with the slower LTC6268 that can output a voltage between 0.5 and 4.5 volts; given that the voltage required to drive the memristors is above 2 volts, the ADA4680 would not be able to drive these devices. Taking this into consideration, the LTC6268 is selected.

The comparator has a single design parameter, a fast transient response that can detect pulses with a time pulse width of 500 ns. Any comparator with a propagation delay (which is the time it takes to propagate an input signal to the output) lower than 500 ns would be suitable for this application; the LT1714 comparator with a propagation delay of 4.4 ns is selected due to low cost and ease of manufacturing.

2.5.2. Selection of Transmission Gates Modules

As discussed in Section 2.4, the opamp circuit that works as an integrator or buffer is reconfigured using TG devices. A TG behaves as a voltage-controlled switch and modifies the operation of the opamp-based circuit. An ideal CMOS switch exhibits zero resistance when turned on and infinite resistance when turned off, but these ideal specifications are not realistic in practice.

Table 2. Comparison table of the main electrical characteristics for transmission gates.

	ADG619 [39]	ADG733 [40]	ADG788 [41]
On-resistance [$\mathsf{\Omega }$]	6.5	4.5	4.5
On-resistance flatness [$\mathsf{\Omega }$]	0.7	0.5	0.5
Dual supply range [V]	$\pm 5$	$\pm 2.5$	$\pm 2.5$
Propagation delay to on condiciont ($t_{on}$) [ns]	40@3.3 V	21@1.5 V	21@1.5 V

shows the characteristics of the proposed TG. The ADG619 has a wider supply range while in dual mode, a low propagation delay with a higher voltage on the logic input, a low on-resistance (the total resistance of the switch while closed) [38], and an overall lower on-resistance flatness (which is the variation in the on-resistance during the entire voltage excursion).

2.5.3. Spiking Generator Block Design

The spiking generator block produces the neuron’s spike waveform, which will be fed the positive differential input of the opamp circuit when it operates as a buffer and drives the memristors. It supplies a reference voltage, $V_{ref}$ during the leaky integration phase. The waveform is shaped by four control signals, ${\varphi }_1,{\varphi }_2,{\varphi }_{int}$, and ${\varphi }_{fire}$, shown in Figure 5 in a time diagram. Signal ${\varphi }_1$ controls the period of the active spike; this time is utilized in the synaptic weight adjustment. Signal ${\varphi }_2$ controls the absolute refractory period of the neuron, which is the period of time when the neuron ignores all incoming input impulses; this phenomenon naturally occurs on biological neurons and avoids an over-stimulation of neurons. Signal ${\varphi }_{fire}$ represents the time at which the neuron is firing an impulse, and ${\varphi }_{int}$ the time at which the neuron is integrating input impulses.

Figure 7 shows the proposed electrical circuit implementation for the spiking generation module. This module uses five TGs to generate the desired signal output shape. The RC circuit incorporated in this proposal determines the relaxation time of the neuron. The values of 500 K$\mathsf{\Omega }$ resistor and 500 $pF$ capacitor are selected for two main reasons:

• The commercial memristors by Knowm exhibit the greatest change in memristance at a low frequency (1 Hz) and the smallest change at a high frequency (1 KHz) [42].
• A total relaxation time of 1.25 ms is within the absolute refractory period of mammal neurons, where neurons cannot generate new output spikes [43], contributing to sparse computing with low power.

3. Results

This section introduces the electrical simulations and measurements for the implemented LIF neuron prototype. The tests were conducted to obtain the tuning curve as a neuron figure of merit and the comparison between the theoretical STDP learning process of living organisms modeled by Equation (7) and the STDP produced with two LIF neuron circuit cells linked with the Vourkas memristor macromodel in code block Listing 1.

3.1. Electrical Simulation of the LIF Neuron

The electrical simulation of the neuron was conducted in the simulator LTSpice, with a constant input current of 4 $\mathsf{\mu }$A. With this input stimulus, the LIF neuron generated the output spikes shown in Figure 8. The subsequent step for the characterization of the LIF neuron is the acquisition of the tuning curve. The tuning curve is the response of any sensory neuron to external stimuli and the behavior it exhibits [44].

Figure 9 illustrates the methodology utilized for the neuron characterization. The testbench is configured to create a series of simulations; in each simulation, a current source that acts as the neuron’s input changes its value. The current source starts with a value of 0 $\mathsf{\mu }$A and reaches a stop value of 16 $\mathsf{\mu }$A; the current source increases by 0.1 $\mathsf{\mu }$A, and, in each simulation, the generated signals at the neuron’s output are stored in a RAW file. Subsequently, this RAW file is parsed to Python, where the data are extracted, analyzed, and used to calculate the frequency of the output spikes. This information is then utilized to generate a tuning curve, which can be seen in red in Figure 10. The neuron’s behavior can be approximated by Equation (9), which is calculated with a least squares regression algorithm described in Algorithm 1.

$$f\left(x\right)=126.5\times ln\left(\frac{x}{0.09247}\right)$$

(9)

Algorithm 1: Least squares regression algorithm for logarithm fitting.

1: Define the curve fit for a logarithmic set as $y=aln\left(\frac{x}{b}\right)$ 2: while$\sigma$ < 0.9 do 3:     for each (x,y) point in the set of length N do 4:         Evaluate with $r_i$ = ln($x_i$), and calculate the following sum for the values of x 5:         $S_r$ = ${\sum }_{i=1}^N{r}_i$ 6:         $S_{rr}$ = ${\sum }_{i=1}^N{r}_i^2$ 7:         Evaluate with $Z_i$ = $Y_i$, and calculate the following sum for the values of y 8:         $S_z$ = ${\sum }_{i=1}^N{Z}_i$ 9:         $S_{rz}$ = ${\sum }_{i=1}^N{r}_i{Z}_i$ 10:        Calculate D, defined as $D=S_r^2-N{S}_{rr}$ 11:        calculate the value of a, defined as $(S_r{S}_z-N{S}_{rz})/D$ 12:        calculate the value of b, defined as $(S_r{S}_{rz}-S_z{S}_{rz})/D$ 13:        calculate the error, defined as $e_i=z_i-Z_i$, $z_i=a{r}_i+b$ 14:        calculate the sum of the square error, defined as S=${\sum }_{i=1}^N{e}_i^2$ 15:        calculate the variance, defined as ${\sigma }^2=\frac{S}{N-2}$ 16:     end for 17: end while

3.2. Hardware Implementation of the LIF Neuron

Figure A1, Figure A2, Figure A3 and Figure A4 contain the schematics used to design a printed circuit board (PCB) for the LIF neuron. The PCB design shows the commercial components used for the neuron prototype, testing points, and trimmers that enable tuning the neuron parameters, such as relaxation time, control signal timing, and $V_{th}$, among others. Figure 11 shows a 3D rendering of the PCB.

The assembled LIF neuron prototype shown in Figure 11 was tested with a constant neuron’s input current of 4 $\mathsf{\mu }$A. The neuron response is introduced in Figure 12. The experimental testbench for the LIF neuron prototype is shown in Figure 13.

Methodology for the Characterization of the Physical LIF Neuron

The methodology described in Figure 9 for characterizing the LIF neuron relies on the simulator being able to modify the input current source of the neuron and store the values generated at the output of the neuron. To achieve the same result in the implementation of the neuron, the test equipment needs to modify the input current, read the output spikes from the neuron, and store the acquired data. Figure 14 outlines the methodology utilized to achieve this result. A voltage-controlled current source (VCSS) is connected in series with a shunt amplifier; this amplifier monitors the current supplied by the VCSS and modifies the value if necessary; the output current will be the input of the spiking neuron. The output spikes are monitored with an oscilloscope, which is controlled by a PC with the use of a virtual instrument [45]; this communication protocol enables the interaction with the oscilloscope and retrieves the data automatically; the VCSS is controlled by a microcontroller, which also monitors the current supplied to the spiking neuron. The data points gathered by the oscilloscope are stored on the PC, which will be used to calculate the frequency of the output spikes.

Figure 15 shows the electrical response of the LIF neuron prototype. The data points obtained from the physical implementation closely follow the behavior of the LIF neuron simulation, with a shift in the frequency response; the data can be approximated with Equation (10), which is also calculated with a least squares regression and in accordance with Equation (9), utilized for the simulated data points with different gain values.

$$f\left(x\right)=140.5\times ln\left(\frac{x}{0.09247}\right)$$

(10)

3.3. Simulation of the Synaptic Weight Adjustment of a Memristive Synapse

The testbench used to simulate the synaptic weight adjustment with the aid of the memristor’s macromodels in code block Listing 1 is presented in the following sections.

3.3.1. Long-Term Potentiation on Memristive Synapse

Figure 16 shows the potentiation response of a memristive synapse placed between two LIF neurons. As expected, the presynaptic neuron $X_1$ fires first due to the constant current input applied $I_1$; see Figure 16c. It is observed that, after several spikes produced by the $X_1$ LIF neuron, the postsynaptic LIF neuron $X_2$ starts spiking at a slower rate. Therefore, the convolution between the presynaptic and postsynaptic spikes surpasses the positive voltage memristor threshold, potentiating the synapse as shown in Figure 16b. The voltage across the memristor is shown in Figure 16a.

3.3.2. Spike-Timing-Dependent Plasticity (STDP) Process on a Memristive Synapse

Figure 17c shows the testbench used to induce a periodic cycle of potentiation and depression on a memristive synapse. Based on the STDP process, the memristor reaches the highest (positive and negative) value of resistance change when the time windows between the presynaptic and postsynaptic spikes reach a minimum value; see Figure 17b. If the time window reaches zero or a value close to zero, the change in the synaptic weight decreases exponentially; this is due to a physical constraint in memristive devices. In this testbench, a memristor is placed at the output of two neurons with different firing frequencies; the difference in the output spike frequency results in a cycle of potentiation and depression of the synapses, as shown in green in Figure 17a.

Figure 17b demonstrates the synaptic weight adjustment in an analog component (memristor) between two neurons; this cycle of potentiation and depression is the same process that would take place in a DNN, which modifies the value of the synaptic weight between the neurons using the gradient descent rule [33].

Figure 18b shows the obtained synaptic weight adjustment curve as a function of $\Delta t$. The obtained curve with the proposed LIF neuron circuit is in agreement with the typical curve present in living organisms [46]; see Figure 18a. The synaptic weight adjustment increases as the $\Delta t$ decreases. When this $\Delta t$ value reaches zero, the weight decreases exponentially as it implies the generation of very sharp spikes, exceeding any positive or negative memristor’s voltage threshold. However, the internal memristor dynamics are unable to produce a significant change in the memristance due to the oxygen drift velocity mechanism in, e.g., TiO₂ dielectric films being much slower than the brevity of time of the electric field present between the terminals of the memristor. In this test, the maximum weight adjustment is around 750 $\mathsf{\mu }$S.

4. Conclusions

This article presents the design and implementation of a leaky integrate-and-fire neuron prototype with commercial components for neuromorphic computing. The LIF neuron circuit simulations and the spiking production’s electrical measurements showed a small deviation in the tuning curve behavior. This is due mainly to the tolerance values in each electronic component soldered on the PCB. Despite these variations, the neuron LIF realization on the PCB retains the overall behavior obtained in the simulation. The implementation of this prototype enables the testing of various configurations for a spiking neuron. It enables the modification of the firing rate, the leakage current of the LIF neuron, and changes in the shape of the output spike; these optimizations are aimed to maximize the rate of change in the memristor. The physical measurements enable us to understand the core elements and design a version of this neuron tailored for a VLSI process node that can be integrated into a custom design.

The rate of change of the memristance in the simulation for the artificial synapse element motivates us to explore the interconnection of the neuron LIF cells in the PCB to form a spiking neural network for neuromorphic computing experimentation.

The implementation has a total power consumption of around 350 mWs, from which 190 mWs correspond to the three linear voltage regulators utilized for the reference signals of the spiking generator block; a single neuron is capable of driving 4500 neurons with memristive synapses, considering that each neuron would consume a maximum of 20 $\mathsf{\mu }$A.

The method presented for characterizing the spiking neuron can also be used to measure and create an accurate memristive synapse model. This synapse model can be implemented in Python to develop deep-learning models that can be integrated into circuits. Using spiking neuron circuit cells and memristive synapses enables sparse system operation.

Using purpose-specific chips to implement LIF neurons enables AI accelerators to process deep learning with high energy efficiency and low power consumption. These devices can serve as co-processors for real-time pattern recognition tasks, including monitoring for wake words (similar to Alexa devices), triggering requests, and enabling services for IoT devices. This is particularly useful for the remotely battery-powered devices commonly used in IoT applications.

Future Work

In our future work, our goal is to take the tuning curve of the physical neuron, combine it with the equation that describes the behavior of the LIF neuron, and program it in Python to create simulations that mimic the physical prototype. We could then instantiate it multiple times to develop a deep learning architecture simulation framework. Accomplishing this will enable us to experiment by exploring information encoding and decoding schemes, solving pattern recognition problems, and controlling robotics.