Memristive System Based Image Processing Technology: A Review and Perspective

Abstract

As the acquisition, transmission, storage and conversion of images become more efficient, image data are increasing explosively. At the same time, the limitations of conventional computational processing systems based on the Von Neumann architecture continue to emerge, and thus, improving the efficiency of image processing has become a key issue that has bothered scholars working on images for a long time. Memristors with non-volatile, synapse-like, as well as integrated storage-and-computation properties can be used to build intelligent processing systems that are closer to the structure and function of biological brains. They are also of great significance when constructing new intelligent image processing systems with non-Von Neumann architecture and for achieving the integrated storage and computation of image data. Based on this, this paper analyses the mathematical models of memristors and discusses their applications in conventional image processing based on memristive systems as well as image processing based on memristive neural networks, to investigate the potential of memristive systems in image processing. In addition, recent advances and implications of memristive system-based image processing are presented comprehensively, and its development opportunities and challenges in different major areas are explored as well. By establishing a complete spectrum of image processing technologies based on memristive systems, this review attempts to provide a reference for future studies in the field, and it is hoped that scholars can promote its development through interdisciplinary academic exchanges and cooperation.

1. Introduction

With the advent of the Internet of Things, cloud computing, and the big data era, there has been explosive growth in the scale of information. However, the physical separation among perception, computation, and storage in the conventional computing architecture requires frequent data shuttling among the units, thereby causing significant system consumption and speed loss and making it difficult to meet the requirements of information analysis and processing [1,2,3]. Therefore, developing new electronic components for intelligent processing systems that are closer to the structure and function of biological brains has become a hot research topic in the fields of modern electronic circuits and image processing [4,5,6].

Image processing technology, which aims to automatically acquire high-level and abstract information from images, after which it simulates how human eyes work with such information, has become increasingly useful in human life and social production. Exploring the basic structure of human brains, simulating their working mechanisms, and establishing neural network models that integrate perception, storage, and computation as a whole have gradually become research hotspots in the fields of image processing and cognitive computing [7]. The current mainstream neural network models can simulate the reasoning and learning functions of human brains to a certain extent, and they have shown some potential in image processing [8]. However, they are confined to certain types and structures with limited processing capabilities. Additionally, the existing ones lack the process of information perception, transmission, and storage prior to the processing stage. Furthermore, the hardware for neural networks is essential to truly realize the conversion from theoretical studies of brain cognition to new technologies of brain-computer intelligence. Nevertheless, most of the current research focuses on the theoretical analysis of the network structures and algorithms, and the research on implementation schemes for neural network hardware is still in its infancy [9,10,11]. Influenced by factors, such as device size, energy consumption, and integrability, conventional implementation schemes for image processing cannot well trade-off the relationship between speed, accuracy, and system consumption [12,13,14]. We schematically compare the traditional image processing systems and memristor-based image processing system as shown in Figure 1. From the perspective of the device, leakage currents become a problem when the channel length and the gate dielectric thickness of a transistor get closer to the scaling limit [3]. With respect to the architecture, the data transfer between processors and memory units significantly reduces both speed and energy efficiency (referred to as the ‘von Neumann bottleneck’). Furthermore, the performance mismatch between the memory and processing units leads to great latency (also called the ‘memory wall’).

The successful preparation of memristors provides a fresh perspective on the hardware implementation of artificial neural networks. It was proposed by Leon Chua, a scientist at the University of California, Berkeley [15] and discovered by Hewlett-Packard (HP) laboratories in 2008 as the fourth fundamental electronic component after the resistor, capacitor, and inductor [16]. Experiments have shown that the memristor has properties, such as non-volatility, variable resistance, nanoscale size, threshold characteristics, low power consumption, and synapse-like structure [17,18,19]. In particular, by taking full advantage of being synapse-like, the memristor can be used as an “electronic synapse” or an “artificial synapse” in the hardware design of neural networks [20]. Further, after choosing a proper memristor model to simulate the weight of the neural network, a more integrated architecture for hardware implementation can be constructed and applied to different image processing tasks [21,22]. Compared with conventional artificial neural networks, the memristive ones incorporate powerful perception ability, massive storage capacity, and intelligent processing mechanisms to enable deeper analysis and exploration, which are expected to solve slow training speed and insufficient online processing capability in image processing [23,24,25].

By collating relevant research on memristive system-based image processing technology (including relevant mathematical models and their applications), this paper comprehensively elaborates on the fusion mechanism of memristive systems and image processing from three aspects, namely the mathematical models of memristive systems, the conventional image processing based on memristive systems, and the image processing based on memristive neural networks. Furthermore, the study summarizes the main directions, progresses, and problems in this field, analyses its development law, and strives to establish a complete spectrum for the reference of researchers in various fields.

2. Mathematical Models

The memristor is a two-terminal non-linear passive circuit element in nanometre and with memory characteristics, whose resistance is variable and controlled by the intensity, polarity, and duration of power supply. By applying an external voltage to the memristor, the conductive properties of its internal functional layer can be changed from a high resistance state (HRS) to a low resistance state (LRS). In particular, three types of theory, i.e., ionic migration, quantum tunnelling, and charge trapping/de-trapping, dominate the study of memristors’ internal physical mechanisms and dynamic characteristics, and they explain most of the observed memristive phenomena [26].

(1)

Ionic migration: This memristor type usually has the active metal (e.g., Ag) as the top electrode and the inert metal (e.g., Pt) as the bottom electrode. By applying a positive voltage to the top electrode, the active metal will be electrolyzed into metal cations. They will move toward the bottom electrode under the external electric field and then return to metal atoms, the accumulation of which form a metal filament conductive channel for the memristor to transit from the HRS to the LRS. Conversely, by applying a positive voltage to the bottom electrode, the formed conductive channel will gradually break, and the memristor will switch from the LRS to the HRS.

(2)

Quantum tunnelling: The internal functional layer of this type of memristor is mainly a metal oxide (e.g., TiO_x). The Schottky barrier between the metal electrode and the functional layer is adjusted by applying an external voltage to switch the resistive state of the memristor. It disappears when the memristor is in the LRS, whereas it reappears when the memristor is in the HRS.

(3)

Charge trapping/de-trapping: For a memristor whose functional layer is the metal oxide film, there exists an empty state in the film. When a positive voltage is applied to the top electrode, the empty state traps the injected electrons and stores them, and when the empty state is filled, a conductive channel is formed, after which the memristor switches from the HRS to the LRS. By contrast, when a positive voltage is applied to the bottom electrode, the electrons in the empty state are released, the formed conductive channel is broken, and the memristor changes from the LRS to the HRS.

During the memristor fabrication process, a small parameter variation may lead to huge differences between devices, and even significantly affect circuit performance. Meanwhile the unstable performance between memristor cells and the cells themselves makes the integration of the device challenging.

As a result, most applied research on memristors always using their mathematical models. As the fourth circuit element, the memristor represents the interrelationship between the magnetic flux φ and the charge q, as shown in Figure 2.

The memristors can be divided into two categories, i.e., being charge-controlled and being flux-controlled. For the charge-controlled ones, their flux φ is a single-valued function of the charge q, which is expressed as follows:

$$\phi =f(q)$$

(1)

Taking the time t derivative of both sides of Equation (1) gives us the following:

$$\frac{\mathrm{d}\phi }{\mathrm{d}t}=\frac{\mathrm{d}\phi (q)}{\mathrm{d}q}\cdot \frac{\mathrm{d}q}{\mathrm{d}t}$$

(2)

Further, based on the voltage v = dφ/dt and the current i = dq/dt, the relation between volt and ampere for the memristors can be obtained as follows:

$$v=M(q)\cdot i$$

(3)

where the function M(q), which represents the memristance, satisfies the following mathematical expression:

$$M(q)\equiv \frac{\mathrm{d}\phi (q)}{\mathrm{d}q}$$

(4)

For the flux-controlled memristors, their charge q is a single-valued function of the flux φ, which is expressed as follows:

$$q=f(\phi )$$

(5)

Taking the time t derivative of both sides of Equation (5) gives us the following:

$$\frac{\mathrm{d}q}{\mathrm{d}t}=\frac{\mathrm{d}q(\phi )}{\mathrm{d}\phi }\cdot \frac{\mathrm{d}\phi }{\mathrm{d}t}$$

(6)

Based on the voltage v = dφ/dt and the current i = dq/dt, the relation between voltage v and current i for the two sides of the memristors can be derived as follows:

$$i=G(\phi )\cdot v$$

(7)

where the function G(q), which represents the memristance, satisfies the following mathematical expression:

$$G(\phi )\equiv \frac{\mathrm{d}q(\phi )}{\mathrm{d}\phi }$$

(8)

In 2008, a simple linear memristor model based on the ionic migration theory was proposed by D. Strukov’s research team [16], and its structure is shown in Figure 3.

Let us assume that the total thickness of the titanium dioxide functional layer is D, and the one of the doped layer is W. R_on denotes the minimal resistance of the memristor, while R_off represents the maximum. The resistance M(t) of the HP memristor is expressed as:

$$M(t)=R_{\mathrm{L}}\cdot x(t)+R_{\mathrm{H}}\cdot [1-x(t)]$$

(9)

$$\frac{\mathrm{d}x}{\mathrm{d}t}=ki(t),k=\frac{{\mu }_v{R}_{\mathrm{L}}}{D^2}$$

(10)

where x represents the internal state variable of the memristor, μ_v represents the average ionic mobility, i represents the current passing through the memristor, and the constant k is the ratio of the rate of change to the current.

On this basis, a nonlinear memristive model with window functions was constructed in the literature [27] to better describe the boundary effect and nonlinear drift of memristors. In a study conducted by [28] a Simmons tunnelling barrier model was proposed based on the quantum tunnelling theory. It accurately presented the properties of memristive devices, but its mathematical model was more complex, and showed no direct explicit relationship between voltage and current, thereby being unconducive to the subsequent research and applications. Additionally, in 2013, S. Kvatinsky’s research team at the Technion-Israel Institute of Technology [29] put forward a more simplified mathematical version, which was named the ThrEshold Adaptive Memristor (TEAM) model. Two years later, the team [30] further designed the corresponding Voltage ThrEshold Adaptive Memristor (VTEAM) model, with a simple structure as well as certain generality to simulate the threshold characteristics of voltage-controlled memristive devices. In 2017, Fang Liang’s team from the National University of Defense Technology, China [31] brought forward a general TiO_x memristive model by combining the nonlinear drift, ionic migration and negative differential resistance (NDR) effect of memristors. In addition, using traditional analogue circuit components, some researchers [32,33,34] realized the memristive circuit simulation based on Chua’s theory as a way to simulate the basic memristive characteristics. In this paper, the abovementioned mathematical models, which are summarized in

Table 1. Mathematical models of memristors.

Model Type	Current-Voltage Relationship	Dynamic Equation of State Variable
HP Memristive Model [16]	$v(t)=\left(R_L\frac{x(t)}{D}+R_H\left(1-\frac{x(t)}{D}\right)\right)i(t)$	$\frac{\mathrm{d}x}{\mathrm{d}t}=ki(t),k=\frac{{\mu }_v{R}_{\mathrm{L}}}{D^2}$
Nonlinear Memristive Model [27]	$i(t)=w^n(t)\beta \sinh (\alpha v(t))+x[\exp (\gamma v(t))-1]$	$\frac{dw(t)}{dt}=a{v}^m(t)f(w)$
Simmons Memristive Model [28]	$\begin{array}{l}i(t)=\tilde{A}\left(x,v_g\right){\varphi }_1\left(v_g,x\right)\exp \left(-B\left(v_g,x\right)\cdot {\varphi }_1{\left(v_g,x\right)}^{1/2}\right)-\hfill \\ \tilde{A}\left(x,v_g\right)\left({\varphi }_1\left(v_z,x\right)+e\left|v_z\right|\right)\times \exp \left(-B\left(v_g,x\right)\cdot {\left({\varphi }_1\left(v_g,x\right)+e{v}_{\epsilon }\right)}^{1/2}\right)\hfill \\ v_g=v-i(t)R_s\hfill \end{array}$	$\frac{\mathrm{d}x(t)}{\mathrm{d}x}=\left\{\begin{array}{l}{C}_{eft}\sinh \left(\frac{i}{i_{off}}\right)\exp \left[-\exp \left(\frac{x-a_{aff}}{w_c}-\frac{\left|i\right|}{b}\right)-\frac{x}{w_c}\right], i>0\hfill \\ C_{on}\sinh \left(\frac{i}{i_{on}}\right)\exp \left[-\exp \left(-\frac{x-a_{on}}{w_c}-\frac{\left|i\right|}{b}\right)-\frac{x}{w_c}\right], i<0\hfill \end{array}\right.$
TEAM Memristive Model [29]	$\begin{array}{l}v(t)=\left[R_L+\frac{R_H-R_L}{x_{off}-x_{on}}\left(x-x_{on}\right)\right]\cdot i(t)\hfill \\ v(t)=R_L\exp \left(\frac{\lambda }{x_{ofi}-x_{on}}\left(x-x_{on}\right)\right)\cdot i(t)\hfill \end{array}$	$\frac{\mathrm{d}x(t)}{\mathrm{d}t}=\left\{\begin{array}{ll}{k}_{off}{\left(\frac{i(t)}{i_{off}}-1\right)}^{{\alpha }_{df}}\cdot f_{off}(x)\hfill & 0<i_{off}<i\hfill \\ 0\hfill & i_{an}<i<i_{of}\hfill \\ k_{on}{\left(\frac{i(t)}{i_{on}}-1\right)}^{{\alpha }_{on}}\cdot f_{on}(x)\hfill & i<i_{on}<0\hfill \end{array}\right.$
VTEAM Memristive Model [30]	$\begin{array}{c}i(t)={\left[R_{\mathrm{L}}+\frac{R_{\mathrm{H}}-R_{\mathrm{L}}}{x_H-x_L}\cdot \left(x-x_L\right)\right]}^{-1}\cdot v(t)\\ i(t)=\frac{e^{-\frac{\lambda }{x_H-x_L}\cdot \left(x-x_L\right)}}{R_L}\cdot v(t)\end{array}$	$\frac{\mathrm{d}x}{\mathrm{d}t}=\left\{\begin{array}{cc}{k}_{\mathrm{off} }{\left(\frac{v(t)}{v_{\mathrm{th} 1}}-1\right)}^{{\alpha }_{\mathrm{of} }}& f_{\mathrm{off} }(x),0<v_{\mathrm{th}1}\le v\\ 0,& v_{\mathrm{th}2}<v<v_{\mathrm{th}1}\\ k_{\mathrm{on} }{\left(\frac{v(t)}{v_{\mathrm{th}2}}-1\right)}^{{\alpha }_{\mathrm{on} }}{f}_{\mathrm{on} }(x),& v\le v_{\mathrm{th}2}<0\end{array}\right.$
General TiOx Memristive Model [31]	$i\left(t\right)=\left\{\begin{array}{c}{x}^{n_1}{k}_{{\mathrm{on}}_1}\sinh \frac{v}{v_{{\mathrm{on}}_1}}+\left(1-x^{n_1}\right)k_{\mathrm{r}}\left({\mathrm{e}}^{v/v_{\mathrm{r}}}-1\right),v\left(t\right)\ge 0\\ x^{n_2}{k}_{{\mathrm{on}}_2}\sinh \frac{v}{v_{{\mathrm{on}}_2}}+\left(1-x^{n_2}\right)k_{\mathrm{off}}\sinh \frac{v}{v_{\mathrm{off}}},v\left(0\right)<0\end{array}\right.$	$\frac{\mathrm{d}x}{\mathrm{d}t}=\left\{\begin{array}{l}{\alpha }_1\sinh {\beta }_{1}v-\gamma x,v>0\hfill \\ {\alpha }_2\sinh {\beta }_{2}v-\gamma x,v<0\hfill \end{array}\right.$

and compared in

Table 2. Comparative information of different memristive models.

	HP Memristive Model [16]	Nonlinear Memristive Model [27]	Simmons Memristive Model [28]	TEAM Memristive Model [29]	VTEAM Memristive Model [30]	General TiOx Memristive Model [31]
Physical Support	Yes	No	Yes	No	No	No
Physical Mechanism	Ionic Migration	Ionic Migration	Quantum Tunneling	No	No	Ionic Migration
Model Complexity	Simple	Simple	Complex	Moderate	Moderate	Moderate
Applied Range	Wider	Wider	Narrower	Wider	Wider	Wider

, manifest the fundamental features of memristors to some extent. However, their correlation with the physical realization of memristors is not strong enough, and it cannot fully characterize the electrochemical properties of memristive devices.

3. Traditional Image Processing Based on Memristive Systems

The memristor can perform logic calculations directly on the device, making it possible to achieve a true integration of storage and computing. Therefore, it brings new opportunities for the development of traditional image processing technologies.

3.1. Image Storage Based on Memristive Systems

Image processing is a type of memory access-intensive application, which places high demands on memory, requiring both enough capacities to store large-scale image data and fast access speed to ensure processing performance. Currently, non-volatile memories contain the flash memory (NAND), resistive random-access memory (RRAM), phase-change memory (PCM), spin-transfer torque magnetic random-access memory (STT-RAM), and ferroelectric random-access memory (FeRAM). This paper compares the characteristics of various types of new volatile and non-volatile memory devices in terms of capacity, size, read/write performance, lifetime, power consumption, and current technical bottlenecks, etc., with the specific information summarized in

Table 3. Comparative information of different memory devices.

Parameter	DRAM	NAND	STT-RAM	RRAM	FeRAM	PCM
Capacity	~16 Gb	~1 Tb	~64 Mb	~1 TB	~64 MB	~8 Gb
Technology level	~20 nm	~16 nm	~32 nm	~11 nm	~65 nm	~5 nm
Feature Size	6–10 F2	4–11 F2	16–60 F2	4–14 F2	15–34 F2	4–8 F2
Read Operation Time	<10 ns	10–50 us	2–20 ns	10–50 ns	20–80 ns	10–100 ns
Write Operation Time	<10 ns	0.1–1 ms	5–35 ns	10–50 ns	10–5 ns	20–120 ns
Lifetime	>1015	104–106	1012–1015	108–1010	1012–1014	108–1012
Data Retention	Refresh	10 Years	>10 Years	10 Years	10 Years	>10 Years
Write Power	0.1 ~0.1 nJ/b	0.1–1 nJ/b	1.6–5 nJ/b	~0.1 nJ/b	<1 nJ/b	<1 nJ/b
Idle Power	High	Low	Low	Low	Low	Low
Non-volatile	Volatile	Non-volatile	Non-volatile	Non-volatile	Non-volatile	Non-volatile
Destructive Read	Destructive	Non-destructive	Non-destructive	Non-destructive	Destructive	Non-destructive
Major Technical Bottlenecks	Memory refresh, volatility, limited memory process	Limited lifetime performance, low storage density	Small capacity, high write power consumption, poor stability	Unclear material storage mechanism	Small capacity, destructive read, low storage density	Small capacity, narrow range of material operable temperature

. It is found that memristor-based RRAM has a series of outstanding advantages, such as small size, non-volatility, low power consumption, high density, fast erasure, and compatibility with CMOS processes, making it one of the most promising memory devices.

In 2011, Hu et al. proposed a memristor crossbar array that could be applied to image processing (see Figure 4a) [35]. Together with the peripheral control circuit, the random storage of binary, grey scale and colour images could be successfully realized. When storing binary images, the image information was mapped into pulse sequences of varying amplitudes using a voltage converter as the input of the memristor crossbar array, as shown in Figure 4b. As for the grey scale and colour images, the image information was mapped into pulse sequences of varying widths using the voltage converter, which were then used as the input to the memristor crossbar array. It is worth noting that voltage pulses of different widths were obtained by controlling the timing of write operation, which finally enabled the storage of images, as illustrated in Figure 4c,d.

In the literature [35], a memristor-based resistive random access memory (MRRAM) was mentioned. Through improvement, it stored binary and multi-valued input information with different memristances. The effectiveness of storing ASCII characters and images was verified through simulation experiments, and a new scheme for storing grey scale images was discussed as well. In Tan et al.’s study [36], ITO/CeO_2−x/AlOy/Al structured memristors were prepared to realize the perception and non-volatile storage of different multispectral images. Wang constructed a storage circuit based on the 2T2M structured memristive synapse to achieve the storage and recovery of binary images [37]. Compared with conventional storage technology, this circuit effectively reduced the storage space and improved the storage efficiency. In 2017, the research team of Prof. Duan [38] at Southwest University, China, successfully prepared a memristor with silver chalcogenide as the functional layer and constructed a memristive synapse with spike rate- and timing-dependent plasticity by analysing its electrochemical properties. Based on this, an improved memristor crossbar array was designed to realize the storage of grey scale images. One year later, Chen et al. designed a vision system on the basis of combining the optical sensor with the memristor, in which the former was used to detect UV light and convert it into voltage pulses of corresponding intensity, and the latter was adopted to store the converted voltage signal, which realized the perception and storage of UV images [32]. In 2020, Wang Xiaoping and her team members from Huazhong University of Science and Technology raised a memristor-CMOS hybrid storage circuit, where the memristor was utilized to store the bit information of images, while CMOS was applied to conduct control, isolation, and logic operations [39]. A series of simulations confirmed that this memory circuit could achieve improved performance in UHF image storage applications. In summary, most of the studies on memristor-based image storage use memristive synapses for crossbar arrays to keep image information, which reduces the storage density to a certain extent. However, the stability of image memory devices is affected by the issue of current leakage in crossbar arrays. Therefore, avoiding or reducing the leakage is one of the problems of memristor-based image storage technology that must be addressed urgently.

3.2. Image Compression Based on Memristive Systems

With the rapid development of sensor technology, the sizes of image data are also expanding rapidly. Meanwhile, higher requirements are put forward on the clarity and transmission rate of images. Applying memristors to image compression can effectively reduce their storage space and improve their transmission speed at the same time. Therefore, the corresponding circuit implementation scheme has been widely studied by scholars in the related fields.

Li et al. constructed a 128 × 64 memristor crossbar array based on the prepared Ta/HfO₂/Pd memristor, and its circuit structure is presented in Figure 5 [40].

Taking advantage of the high parallelism, non-volatility, low power consumption and small size of memristors, this circuit took a single memristor to store image information at the 6-bit precision, which further achieved functions, such as image compression, convolution and filtering. Additionally, a memristor-based image compression framework is presented in Figure 6 [41], considering the loss of two-dimensional discrete wavelet transform. The framework consisted of three memristor crossbar arrays, where the computational one was used to conduct the data multiplication and addition operations, the intermediate array stored the coefficients of row-column transformation, and the final one was used to keep the compressed data of the original image. The image compression could be achieved by taking the generated pulses through a multilayer voltage sensor as input, mapping the image pixels into memristive conductance through the computational array, and then storing them in the other two crossbar arrays. The research conducted by Berco et al. in 2020 proposed a programmable photoelectronic memristor gate circuit, which could perform state switching between optical and electrical signals, to realize in-situ image compression [42]. A research team from Dalian University of Technology [43] designed the simplest fractional-order chaotic memory circuit that identified pseudo-random sequences in image compression through phase diagrams, Lyapunov exponential spectra, and bifurcation diagrams, which achieved image compression for the second time and reduced the storage costs significantly.

3.3. Image Reconstruction Based on Memristive Systems

High-resolution image information is a prerequisite for the subsequent image processing and analysis. Therefore, effectively and quickly achieving high-resolution image reconstruction has become an urgent problem to be solved in this field. The image reconstruction algorithm based on memristive systems has certain advantages in terms of reconstruction quality and algorithm operation efficiency.

In 2017, Patrick et al. constructed a hardware-implemented sparse coding system using a 32 × 32 memristor crossbar array, as shown in Figure 7. The system input image information as sparsely coded pulses into the array and performed high-resolution reconstruction of the input through online learning. The experimental results demonstrated that the system has the advantage of low power consumption and high speed when performing data-intensive tasks (e.g., real-time video-based reconstruction) [44].

Additionally, a study designed a metal-oxide-based memristive synaptic circuit that enabled “negative (−)”, “zero (0)”, and “positive (+)” synaptic weights [45]. Based on this, the corresponding neuronal circuit was built to realize the on-chip cyclic learning algorithm, and the super-resolution reconstruction of a single frame was completed, as depicted in Figure 8.

A memristor-based compressive sampling encoder that could be integrated with an image sensor to achieve super-resolution reconstruction was put forward by Wang et al. [39]. A series of simulations demonstrated the superior performance of the encoder, with low power consumption and low hardware overhead. In addition, Dong et al. [46] designed a multi-channel pulse coupled neural network based on the nanoscale memristor, which effectively solved the problem of parameter estimation in neural networks by simulating the dynamic changes of connection coefficients. The model was further applied to the task of the super-resolution reconstruction of multi-frame images, and its correctness and effectiveness were experimentally demonstrated.

3.4. Others

With the ease of 3D stacking, the memristive system can efficiently complete matrix multiplication and realize the integration of storage and computation. By adjusting the variable parameters and connection methods of the system, and by adding peripheral control circuits, different nonlinear mapping functions are obtained to realize other image processing techniques (e.g., image interpolation, edge detection, image filtering, and image encryption).

Based on the mathematical model of the spintronic memristive device, Dong et al. [47] analysed its electrical characteristics and resistance variation through mathematical derivation and circuit simulation. Additionally, a memristor crossbar array was made by integrating functions, such as image storage and interpolation (as shown in Figure 9).

A study conducted by Yang et al. [38] showed an improved memristive cell neural network as well as an adaptive thresholding algorithm based on spatial distribution, and they achieved the edge extraction of colour images. The paper [48] discussed a memristive mask circuit based on the computation-in-memory (CIM) architecture, which is shown in Figure 10. The core of this mask circuit was a multi-bit analogue adder based on the memristor crossbar array, which selects the memristive cells to be accessed through the row-column switches. Each of the cells stored 8 bits of data according to the change of memristance, which were defined as pixel values in image processing. By controlling the multi-bit adder, integrator, and input module, the circuit could update the memristance with little dependence on the higher-level computing unit. Additionally, operations, such as image denoising, edge detection, and feature extraction were achieved by constructing different mask operators. The research [49] on the memristor-based 2D convolutional circuit implemented the image colour transformation and compression, while the reference [50] to the structure of the human retina achieved functions, such as image smoothing and edge detection.

A memristor-CMOS-based general logic circuit was studied by Yang et al. [51], and furthermore, a new memristor-based full adder circuit and a binary image encryption circuit were designed. Moreover, two available encryption methods were proposed to improve the reliability of encryption results. For Wang et al. [52], they studied a new memristive chaotic circuit to implement image encryption. Through a series of computer simulations, it was proven that the image encryption algorithm based on the new circuit has higher security and better decoding capability compared to the conventional one.

3.5. Summary

Currently, memristive system-based conventional image processing is in a rapid development stage, and some progress has been made in the same field. However, there still exist many problems that must be solved:

(1)

The instability and variability of memristive devices have an impact on the accuracy of image processing. Therefore, it will be a significant study to explore the internal physical mechanism of memristive devices and to study their electrochemical properties under the influence of different external factors, to build a mathematical model that can accurately describe their behaviour.

(2)

Conventional image processing circuits do not consider the possible faults of memristive circuits in practice. Nevertheless, research on fault diagnosis can effectively help reduce the circuit overhead as well as improve algorithm operation efficiency and image processing accuracy while increasing the robustness and anti-interference capability of the circuit.

(3)

On the one hand, the design of the peripheral circuits in some image processing applications is too complex, which increases the power consumption of the system operation. On the other hand, the one with a simple structure and high compatibility can result in enhanced efficiency for complex conventional image processing tasks.

4. Image Processing Based on Memristive Neural Networks

The successful preparation of memristors brings new ideas for simulating the cognitive functions of artificial synapses. By applying memristive synapses to the hardware implementation of neural networks, a new type of neural network with high integration can be built. It possesses powerful image processing capabilities and plays an important role in fields with high computational complexity, such as image recognition, classification, and segmentation.

4.1. Image Recognition Based on Memristive Neural Networks

In the literature [53], an impulsive neural network based on memristors was constructed in which the memristive synapses used STDP rules to update the weights, and the memristive neurons adopted the “winner-take-all” strategy to complete the task of handwriting recognition. It was found that its recognition accuracy could reach 83%. A study conducted by Yakopcic et al. [54] presented a memristor-based convolutional neural network to perform convolutional operations using memristor crossbar arrays, and the accuracy of its handwritten digit recognition reached 94%. Furthermore, a transformation method for neural network models was brought forward [40], as shown in Figure 11.

Specifically, the method of sparsity was taken to divide the original neural network into appropriately sized sub-networks. The limited hardware accuracy was solved by quantizing the input data and somewhat improved to approximately 99.8% from the software side. In the study [55], a 1M structured memristive synapse was introduced to the memristor-based multilayer neural network, and an adaptive backpropagation algorithm was applied to train the neural networks, thereby achieving character recognition. Kang Jinfeng’s team at Peking University [56] reported a memristor-based binary neural network. It was trained online, its weight update was achieved using the 2T2R structure of the memristive synapse, and its correctness and effectiveness were verified on the MNIST dataset with a recognition accuracy of 97.4%. In addition, Hu et al. [22] used 2 phase-change memories to construct artificial synapses, based on which a 3-layer perceptron network was built, and they proved its correctness on the MNIST dataset with a recognition accuracy of 82.2%. For Wang et al. [57], they constructed a memristor-based convolutional neural network, which was significantly improved in terms of array area and energy efficiency compared with previous ones for the handwriting recognition task. In 2020, a research team from Tsinghua University [58] designed a memristor-based convolutional neural network (see Figure 12). Meanwhile, a hybrid training method was suggested to enhance the robustness of the network, and the handwriting recognition task realized an accuracy of over 96%.

In addition, memristor-based neural networks have been applied to other image recognition tasks. For instance, Professor Wu Huaqiang and his team members from Tsinghua University constructed a multilayer perceptron neural network based on 1T1R memristive synapses [59], as shown in Figure 13. The network achieved grey scale face image recognition from the Yale Face Database through online learning, and the recognition rate could reach 88.08% for 9000 test images with noise added. Other researchers [60] investigated a hierarchical temporal memory (HTM) network based on memristors, which applied sparse distributed representations to obtain spatial information of input signals, after which they used parallel learning to adjust the network weights and finally verified the correctness and effectiveness of the network through face recognition tasks. As for the memristor-based probabilistic neural network [61], it carried out product multiplication using memristor crossbar arrays as well as normalization operations on weights to reduce the complexity of the circuit. The network was validated on the Iris Flower dataset with a recognition accuracy of 98%. Furthermore, the multilayer perceptron neural network studied by Yu et al. [62] showed increased adaptive capability by introducing nonlinear features in the learning process and superior performance on general datasets, such as MNIST, Iris, and Car Evaluation.

4.2. Image Classification Based on Memristive Neural Networks

In 2013, Alibart et al. [63] successfully prepared a TiO_2−x-based memristor, after which they developed a single-layer perceptron (SLP) neural network based on the TiO_2−x memristor crossbar array to achieve image classification. Its circuit structure is displayed in Figure 14.

Another (SLP) neural network was made based on 2M memristive synapses [10], and its circuit structure is presented in Figure 15. The network, which was trained using delta rules, achieved the classification of 3 × 3-pixel black-and-white images. Professor Strukov’s team at the University of California, Santa Barbara [64] prepared a 20 × 20 memristor crossbar array, as depicted in Figure 16. The array adopted TiO_2−x and Al₂O₃ as the functional and stacked layers, respectively, after which it was interconnected with traditional CMOS peripheral circuits, thereby constructing an SLP neural network to achieve the image classification with an accuracy of more than 97%.

Wang et al. [65] prepared a three-dimensional structured memristor and applied it to image classification, which improved the operational efficiency of the algorithm and opened a new path for the in-depth integration of computer vision and novel nanodevices. Additionally, a memory computing framework based on memristors was proposed by Zhang et al. in 2021 [66], which used a greedy search algorithm to improve the robustness and anti-interference capability of the system, and its accuracy reached 92.3% on the classification tasks involving the CIFAR-10 dataset.

4.3. Image Segmentation Based on Memristive Neural Networks

As early as 2014, Myonglae et al. [67] proposed a memristor-based visual recognition system, where the system used a programmable gate array to convert image signals into pulse signals and performed weight updates based on STDP learning rules. As a result, the foreground and background segmentation of figure images from “0” to “9” were achieved. One year later, Chiu and his team members [68] constructed a differential 2R crossbar array, which applied RRAM as a cache to reduce system energy consumption, and they verified its correctness and effectiveness using image segmentation tasks. In the literature [69], a fully convolutional neural network based on memristors was introduced. It utilized voltage selectors and memristor arrays to construct its max-pooling layers as well as a sliding window approach to enhance operation efficiency. Moreover, the weight updates of memristor arrays were implemented through the ex-situ training method, and the effectiveness of the proposed network was finally verified through image segmentation. The study [70] designed a memristor-based cell neural network based on the fractional-order calculus theory, as illustrated in Figure 17.

In the process of image edge extraction, it took the fractional-order control method to increase the high-frequency information and retain more low-texture information. The simulation results proved that the edge images extracted by this network had more complete and clear contour information and richer texture detail information. Another example is the prepared memristor with NbO_x as its functional layer [71]. An artificial sensory neuron was constructed, then in combination with an InGaZnO₄ optical sensor (see Figure 18), which encoded optical information into impulses, image segmentation in complex backgrounds was achieved by such a pulse-coupled neural network. It is believed that this study has paved the way for the integration of neuromorphology and bioelectronics. In 2021, Chen et al. [72] proposed an efficient memristor-based fully convolutional neural network, which adopted a convolutional kernel-first (CKF) algorithm to achieve effective parameter pruning, thereby significantly reducing circuit power consumption and demonstrating high accuracy and adaptiveness for medical image segmentation tasks.

4.4. Others

Tsai et al. [73] reported a long short-term memory network, which mapped and programmed the network weights into the phase-change memory devices, as demonstrated in Figure 19. Compared to other methods, this network realized the software-equivalent text prediction as well as a larger improvement in the accuracy of weight mapping and text prediction.

Another long short-term memory network was built on a 128 × 64 1T1R memristor array [74], as shown in Figure 20. Through utilizing the memristor arrays to store synaptic weights for different time steps, the network performed the prediction task of the number of global airline passengers and the recognition task of human gait, and it verified the feasibility of the memristor-based long short-term memory neural network in performing tasks, such as linear regression and pattern recognition.

Moreover, Farkhani et al. [75] designed a neuromorphic computing system based on spintronic memristors, where the read circuit was replaced with a proposed real-time sensing circuit, and the input signals were turned into the switching of magnetic moments, thereby substantially reducing circuit energy consumption, providing system operational efficiency, and achieving the real-time tracking of targets. As for the study [76], the chaotic trajectories of memristive circuits were included, which combined the homotopy analysis method (HAM) and multi-objective optimization (MO) to tackle the high computational complexity and low computational efficiency of traditional analysis methods.

In this paper, the architectural characteristics of several image processing algorithms based on memristive neural networks are comprehensively summarized, including their input coding patterns, weight representations and the data types of interlayer communication. The specific comparative information is summarized in

Table 4. Comparative information of memristive neural network-based image processing.

Reference	Architectural Characteristics of Image Processing Algorithms Based on Memristive Neural Networks
	Input Coding	Weight Representation	Neural Network Communication
[67]	Amplitude Encoding/Time Encoding Analogue Signal	Differential Amplifier	Multi-precision
[69]	Amplitude Encoding Analogue Signal	Multi-precision	MSB
[38]	Amplitude Encoding Analogue Signal	Differential Amplifier	Multi-precision
[40]	Amplitude Encoding Analogue Signal	Differential Amplifier	Multi-precision
[62]	Amplitude Encoding Analogue Signal	Peripheral Circuit Processing	MSB

.

The above key research questions will provide references for building the next generation of novel memristive neural networks with integrated perception-storage-computation architectures.

4.5. Summary

With the expanded research in nanomaterials science and image processing technology, image processing based on memristive neural networks has become one of the hot issues in the study on neural network hardware implementation schemes. Currently, there are the following problems that must be solved timely.

(1)

The existing memristive synaptic circuits can only simulate the basic functions and behavioural characteristics of biological synapses, and they lack enough theoretical support from computational neuroscience. Therefore, it is crucial to design a fully functional and simple structured memristive synaptic circuit, which can address the problems of insufficient portray, unclear mechanism, and single plasticity of the conventional ones.

(2)

There is the accumulation of computational errors in memristor-based neural network circuits, which is mainly owing to the discrete nature of memristors, and it is difficult to avoid at the device level. Therefore, designing a newly structured memristor crossbar array that offsets the accumulated errors can provide a new perspective for the hardware implementation of neural networks.

(3)

The current research on image processing based on memristive neural networks is still stuck in the simulation of existing artificial neural networks. Therefore, the next research hotspot will involve taking both the advantages of neurocomputing science and image processing studies, exploring brain-inspired neural network training algorithms, and building memristive neural networks with brain-like memory.

5. Prospects

As the fourth passive circuit component, the memristor has certain memory properties, with its resistance value changing dynamically with the flowing charge and its high similarity with the synapse in the human brain. Using memristors to construct artificial synapses for neuromorphic computation is of great significance to the new intelligent information processing systems and integrated image storage and computation. Memristive system-based image processing technology is an interdisciplinary field of research, covering materials, devices, circuits, architectures, algorithms and integration technologies. We list major challenges and potential solutions for memristive system-based image processing technology, as shown in

Table 5. Key challenges and possible strategies of memristive system-based image processing technology on the device, hardware, and algorithm levels.

		Key Challenges	Possible Strategies
Device level	Materials	Fabricate standard-process and compatible new materials and interconnect materials with high conductance	Use alternative organic materials, 2D, and functional materials, and develop new processes for new materials
	Models	Less computational complexity and high physics fidelity for large-scale system simulation	Build mathematical models of memristors, combined physical and empirical behavior of devices
Hardware level	Peripheral circuits	Efficient read/write scheme for digital/analog mode	Use analog circuits, field programmable gate array (FPGA), and look-up-table (LUT) connected to the chips and approximate circuits
	Synaptic circuits	The operating mechanism is still obscure, the cognition function modeling is not good, and the fault diagnosis system is still in progress.	Develop the novel memristive synapse circuit will possess biological synaptic features
Algorithm level	Operations	Develop a general computing system for data mapping, dot product, and STDP	Experimentally build applications with a memristive crossbar
	Training and testing accuracies	Develop practical network topology and learning algorithm	Develop hybrid algorithms, and brain-inspired systems consist of both ANNs and SNNs

.

(1)

At the device level, the device stability is critical to the computing accuracy, as the drift of conductance states with time or environmental changes will result in undesired synaptic weight changes. On the one hand, more reliable and eco-friendly memory devices and memristive arrays are required. On the other hand, the construction of scalable and highly stable memristive mathematical models, following the physical mechanisms of memristor devices and the special properties of memristors, is one of the future directions to further promote image processing research based on memristive systems.

(2)

At the hardware level, in the short term, memristors will be specially utilized to accelerate the construction of artificial neural networks. Compared with conventional computer processors, their analogue signals are processed in a massively parallel manner, which increases the computational speed and fault tolerance simultaneously and significantly reduces the system power consumption. This parallel computing and low power consumption feature is well suited for image processing tasks with large data volumes and high computational complexity. In the long term, artificial synapses built on memristors will be one of the new approaches for facilitating the hardware implementation of brain-like neural networks. Nevertheless, the current memristive synaptic circuits can merely simulate the basic functions and behavioural characteristics of biological synapses, and they receive insufficient theoretical support from computational neuroscience. Therefore, the design of the memristive synaptic circuits with multiple biological synaptic properties can provide a new idea and platform for exploring a general memristive system-based image processing architecture to address the problems of insufficient portray, unclear mechanism, single plasticity, etc. Meanwhile, peripheral circuits control the read/write process in the memristor-based image processing systems. memristor-based image processing systems are expected to further improve the performance of online learning and reduce the complexity of peripheral programming circuits in the future.

(3)

At the algorithm level, the learning algorithms of memristor based image processing systems are still under development. The conventional computing system has the problems of high cost and difficult training when simulating impulsive neural networks, whereas the unique dynamic memory and reconfigurable characteristics of memristors can realize not only the diverse biological synaptic plasticity for artificial synapses but also the natural compatibility of artificial neural networks and impulsive neural networks. The image processing algorithm based on memristive systems can learn from deep learning and computational neuroscience to solve the problems of slow training speed and the insufficient online processing capability of conventional artificial neural networks in image processing applications. With better understanding of neuronal communications and functionalities, general learning algorithms should be designed to promote hardware development as well.

6. Conclusions

Memristors have been widely studied in image possessing for their synapse-like properties, low power consumption, high efficiency, integrability, etc. Two of their major applications are memristive system-based traditional image processing, including image compression, reconstruction, and edge extraction, and memristive neutral network-based image processing, including image recognition, classification, and segmentation. In neural networks, memristors are mainly adopted as synaptic devices to realize the hardware mapping of synaptic weights under pulse stimulation and to store the synaptic weights in real time for in-situ computation. The parallel computing capability of the memristor array improves the operational efficiency of the neural network and reduces the energy consumption of the system. Additionally, it is believed that the image processing technology based on memristive systems has very promising prospects in terms of its computational speed, computational energy efficiency, and processing accuracy, etc. Therefore, to develop a new type of energy-efficient memristor-based image processing system, collaborative innovations are needed in areas, such as mathematical modelling, architecture, and algorithm implementation.