Optical Convolutional Neural Networks: Methodology and Advances (Invited)

Abstract

As a leading branch of deep learning, the convolutional neural network (CNN) is inspired by the natural visual perceptron mechanism of living things, showing great application in image recognition, language processing, and other fields. Photonics technology provides a new route for intelligent signal processing with the dramatic potential of its ultralarge bandwidth and ultralow power consumption, which automatically completes the computing process after the signal propagates through the processor with an analog computing architecture. In this paper, we focus on the key enabling technology of optical CNN, including reviewing the recent advances in the research hotspots, overviewing the current challenges and limitations that need to be further overcome, and discussing its potential application.

1. Introduction

Convolutional neural networks (CNNs), as an important category of deep neural networks, are inspired by the natural visual perceptron mechanism of living things [1]. Since the first modern sense framework of CNNs, known as LeNet-1 [2], emerged in 1989, numerous representative CNN frameworks have been developed, including LeNet-5 (1998) [3], AlexNet (2012) [4], ZFNet (2014) [5], VGGNet (2015) [6], GoogLeNet (2015) [7], and ResNet (2016) [8]. Meanwhile, abundant progress has been made to deepen CNNs’ complexity and reduce the number of parameters [9,10,11]. Owing to the continuous optimization of network frames, CNNs have been widely used in image recognition [2,3,4,5,6,7,8,9,11,12,13], speech recognition [14,15,16], gaming [17,18], medicine [19,20], autonomous driving [21,22], and other fields.

The explosive increase in Internet data year by year has called for more intelligent and effective data processing [23]. As is well-known, there is a positive correlation between the accuracy of a CNN and the number of parameters [24]. Therefore, it has more stringent requirements on the computing hardware due to the demands of massive data processing and high-precision processing. For electrical hardware processors, performance improvements have followed Moore’s Law over the past few decades [25,26]. As the chip manufacturing process has gradually approached its physical limitations in recent years, the growth rate of single-chip computing power has gradually slowed [27,28], and semiconductor technology has entered the post-Moore era. Additionally, using the Von Neumann computing paradigm in the traditional computing hardware, such as CPU, GPU, FPGA, ASIC, etc., it is an indisputable fact that the discrete architecture of processor and memory makes it inevitable to trade-off between bandwidth and power consumption [29,30,31,32]. Hence, it is an obvious sharp conflict with the ever-increasing demand for high-performance processing and the slowing growth of computing power [28].

Optical devices, as an alternative, have been regarded as a competitive candidate in the “more than Moore” era [33] with the superiority of ultralarge bandwidth and ultralow power consumption. Compared with electrical vector-matrix multiplication (VMM), it is able to achieve better performance using optical devices, with the computing speed increasing by three orders of magnitude and the power consumption decreasing by two orders of magnitude [34]. In recent years, optical computing solutions, because of their intrinsic high computing speed [35], high computational density [36,37], and low power consumption [38,39], have been massively demonstrated by means of both discrete systems and integrated chips. Meanwhile, numerous review works have put more emphasis on optical matrix multiplication [40,41], special-technology-based optical computing [42,43,44,45,46,47,48,49], and optical neural networks [50,51,52,53,54,55,56]. CNNs, as one of the main branches, require more than 80% of full calculations to execute the convolution operation [57]. Accelerating the convolution process in the optical domain provides a subversive way to improve the computing speed and decrease the power consumption, which has not been systematically reviewed. Here, we focus on the key enabling technology of optical CNNs, review the recent advances in optical CNNs, discuss the realization mechanism of optical convolution, and make a prospect and an approach to further development toward the next generation of artificial neural networks.

2. Development of Optical Convolution Neural Network

Generally, a CNN is composed of convolutional layers, pooling layers, fully connected layers, and nonlinear activations. With the convolutional layer, convolution operations are conducted to extract the features of input images. The typical CNN architecture of LeNet-5 is shown in Figure 1a. The convolutional layer with multiple kernels concurrently performs the convolutional operations to extract various feature images. The pooling layer following the convolutional layer is used to subsample and compress features, reduce the amount of calculation, and alleviate overfitting. Immediately afterwards, the fully connected layers are able to realize the full connection of parameters and generate the final classification results. Additionally, various nonlinear activation functions are employed following the convolutional layers and the fully connected layers, aiming to lead the nonlinear properties to the network.

Figure 1b shows the principle of one convolution operation with an $m\times n$ kernel. One convolution operation can be divided into two processes: (1) multiplication between elements in the kernel matrix and data matrix and (2) addition of all multiplication results, which can be expressed as follows:

$$y={\displaystyle \sum _{i=1}^m{\displaystyle \sum _{j=1}^n{w}_{ij}{x}_{ij}}},$$

(1)

where $w_{ij}$ is the element of the convolution kernel, $x_{ij}$ is the element of the input data, $m$ is the row count, and $n$ is the column count in the convolutional kernel. From Equation (1), one convolution operation is converted into vector-vector multiplication (VVM), and parallel convolution of multiple kernels is represented as a VMM [58]. At the same time, the optical matrix operation has been widely investigated [59], which makes it convenient to accelerate the convolution operation process with optical methods.

The optical architecture enabling the convolution operation has been blooming. For most of the reported optical CNNs, the convolution operations of CNNs are accelerated in the optical domain, and the rest remain in the electrical domain, which absorbs both the respective advantages for the ultra-bandwidth and low loss of light and the high precision recognition of electricity. Optical CNNs based on the implementation principle are generally divided into four categories: diffraction-based optical CNNs, interference-based optical CNNs, wavelength division multiplexing-based (WDM-based) optical CNNs, and tunable optical attenuation-based optical CNNs.

2.1. Optical CNN Based on Optical Diffraction

Optical computing with free-space optical diffraction has been demonstrated over decades [60,61]. The basic scheme is the Fourier transform property of the lens [62]. When coherent light passes through the lens, a two-dimensional (2D) Fourier transform is performed without power consumption, and the focal plane behind the lens (which is also called the Fourier plane) presents the result of the Fourier transform. Since the convolution operation in the spatial domain is able to transform into a multiplication operation in the Fourier domain following the convolution theorem [63], the convolution operation is implemented with a “4F system” by adding a specially designed mask to the Fourier plane of the first lens.

As shown in Figure 2, the noted “4F system” is composed of two lenses placed coaxially with a distance of 2F, and the distance between the input plane and output plane is 4F. The Fourier transform of the input image $g(x,y)$ is accomplished in the focal plane of the first lens

$$G(u,v)=\mathcal{F}\{g(x,y)\},$$

(2)

where $\mathcal{F}$ represents the Fourier transform and $u,v$ are the coordinates in the frequency domain. In the Fourier plane, the specially designed mask $M(u,v)$ introduces additional amplitude and phase modulation,

$$G\prime (u,v)=G(u,v)M(u,v).$$

(3)

Then, the inverse Fourier transform is realized with the second lens, which can be expressed as

$$\begin{array}{cc}\hfill g_o(x\prime ,y\prime )& ={\mathcal{F}}^{-1}\{G\prime (u,v)\}\hfill \\ & ={\mathcal{F}}^{-1}\{G(u,v)M(u,v)\}\hfill \\ & =g(x,y)\ast m(x,y),\hfill \end{array}$$

(4)

where ${\mathcal{F}}^{-1}$ is the inversive Fourier transform, $m(x,y)$ is the inverse Fourier transform of $M(u,v)$, and $\ast$ represents the convolution operation. From Equation (4), the convolution operation of the input image is simultaneously achieved when passing through the “4F system”, and then the feature maps are recorded by the camera sensor and converted into digital electrical signals to conduct nonlinear activation, pooling, and final recognition in the fully connected layer.

The CNN on optical diffraction was proposed and experimentally demonstrated in 2018 [64], and the schematic of the hybrid optoelectronic CNN as well as the experimental setup is shown in Figure 3a. The optical convolution operation was accomplished using the diffractive optical element (DOE) as the phase mask placed in the Fourier plane of the “4F system”. The optical feature image was converted into a digital electrical signal with a camera sensor, and the processes in terms of nonlinearity activation and full connection were still addressed in the electrical domain. The parallel convolution of multiple kernels was realized via the optimization design of the phase mask. The experimental accuracy reached 44% on the grayscale CIFAR-10 dataset, which improved by approximately 50% relative to the single digital fully connected layer (accuracy of 30%). Apart from the diffractive optical element, the optical metasurface as the phase mask offered another approach using multiple physical lays or wavelength-sensitive material to conduct multichannel free-space convolution [65,66,67,68]. An amplitude-only weighting optical CNN in the Fourier plane was proposed in 2020 using a digital micromirror device (DMD) as a reconfigurable amplitude modulation device [69]. A typical schematic representation of a “4F system” on digital micromirror devices is shown in Figure 3b. Benefitting from the large-scale and high-speed properties of digital micromirror devices, classification tasks on two megapixel matrices at 10 kHz rates were performed. The classification accuracies were up to 98% on the MNIST database and 54% on the CIFAR-10 database. After that, multiple kernel parallel convolution operations based on the optical diffraction orders (Figure 3c) were realized by the same research group [70]. In 2023, using the two-dimensional Dammann grating as the key element for generating multiple displaced images, massive parallelism convolution acceleration was experimentally demonstrated with a computing accuracy of 8 bit [71]. A spatial light modulator (SLM) was used for data input, and the distance between the data input plane and Dammann grating corresponded to the kernel stride. The MNIST database was classified with an accuracy of 97.3%. Figure 3d shows the lensless architecture of the optical CNN [72]. In this scheme, the amplitude mask was placed close to the sensor. Different pixels of the input image were transmitted through different portions of the amplitude mask, underwent different attenuations, and finally focused on the camera sensor. The image sensor performed a weighted sum of the beams, where the weights were determined by the amplitude mask. The accuracy was up to 97.21% on the MNIST database.

The on-chip Fourier transform scheme was also investigated to realize the optical CNN [73,74,75,76]. The Fourier transform property of integrated star couplers was simulated to realize the optical CNN in 2020 [74]. In 2022, the optical integrated diffractive neural network (IDNN) chip in Figure 3e was fabricated on the silicon-on-insulator (SOI) platform [73]. Two on-chip cascaded diffractive cells were used to conduct the operation of the Fourier transform and inverse Fourier transform, where 10 Mach–Zehnder interferometers (MZIs) were used to load the input data and 10 MZIs were used for kernel adjustment. A convolution operation was performed on the proposed integrated diffractive neural network chip, and the recognition of 1D sequences and 2D images was carried out experimentally. In addition to optical CNN, direct classifications of Iris flower, MNIST dataset, and Fashion-MNIST dataset were conducted using the integrated diffractive neural network chip with experimental accuracies of 98.0%, 91.4%, and 80.4%, respectively. In addition to the above realization, other interesting schemes have also been reported, such as in situ training [77], position robustness [78], incoherent light-based [79], and so on [80,81], which contributed to the further development of optical CNNs.

2.2. Optical CNN Based on Optical Interference

Benefitting from the development of photonic integration technology, matrix multiplication has been extensively focused on light interference [82,83,84]. Since the convolution operation can be mathematically converted to matrix multiplication, it has become a very popular scheme to realize optical CNNs by exploiting the interference properties of light [85,86,87,88,89,90,91]. As the basic unit to build a coherent unitary matrix for discrete convolution operation, the structure diagram of the MZI is shown in Figure 4a, where the splitting ratio of two optical couplers is fixed at 50:50, the internal phase shifter provides an additional phase $\theta (0\le \theta \le \pi )$ to control the splitting ratio of the MZI, and the external phase shifter adds a relative phase $\phi (0\le \phi \le 2\pi )$ to two output arms. Ignoring the optical propagation loss, the transfer matrix $T(\theta ,\phi )$ for a single MZI unit is written as

$$\begin{array}{cc}\hfill T(\theta ,\phi )& =P_{\phi }C{P}_{\theta }C\hfill \\ & =\frac{1}{2}\left[\begin{array}{cc}{e}^{i\phi }& 0\\ 0& 1\end{array}\right]\left[\begin{array}{cc}1& i\\ i& 1\end{array}\right]\left[\begin{array}{cc}{e}^{i\theta }& 0\\ 0& 1\end{array}\right]\left[\begin{array}{cc}1& i\\ i& 1\end{array}\right]\hfill \\ & =\frac{1}{2}\left[\begin{array}{cc}{e}^{i\phi }(e^{i\theta }-1)& i{e}^{i\phi }(e^{i\theta }+1)\\ i(e^{i\theta }+1)& 1-e^{i\theta }\end{array}\right]\hfill \\ & =\left[\begin{array}{cc}{t}_{11}& t_{12}\\ t_{21}& t_{22}\end{array}\right],\hfill \end{array}$$

(5)

where $P_{\phi }$ is the transfer matrix of the external phase shifter, $C$ is the splitting ratio of the optical coupler, $P_{\theta }$ represents the internal phase shifter, and $t$ is a simplified representation of the elements in the matrix.

An arbitrary unitary matrix was formed by constructing an MZI-based special mesh network. Two mainstream unitary matrix mesh networks (one was rectangular architecture [82] and the other was triangular architecture [83]) are shown in Figure 4b,c. In both architectures, $N(N-1)/2$ MZIs were needed to make up an N-dimensional unitary matrix mesh. The transfer matrix of a single MZI unit in the mesh network connecting channels $m$ and $n(m=n-1)$ can be written as

$$T_{m,n}^{(k)}({\theta }^{(k)},{\phi }^{(k)})=\left[\begin{array}{cccccccc}1& & & & & & & \\ & 1& & & & & & \\ & & \ddots & & & & & \\ & & & t_{11}& t_{12}& & & \\ & & & t_{21}& t_{22}& & & \\ & & & & & \ddots & & \\ & & & & & & 1& \\ & & & & & & & 1\end{array}\right],$$

(6)

where $k$ is the serial number of the MZI. The transfer matrix of the entire mesh $T$ can be decomposed into

$$T={\displaystyle \prod _{(m,n)\in S}{T}_{m,n}^{(k)}({\theta }^{(k)},{\phi }^{(k)})},$$

(7)

where $S$ defines the specific ordered sequence. For the triangular unitary MZI mesh shown in Figure 4c, according to Equation (6), the transfer matrix of four rows of MZIs can be written as

$$\begin{array}{c}\hfill T_{2,1}^{k=1,3,6,10}=\left[\begin{array}{ccccc}{t}_{11}\hfill & t_{12}\hfill & \hfill & & \\ t_{21}\hfill & t_{22}\hfill & \hfill & & \\ & \hfill & 1\hfill & \hfill & \\ & \hfill & & 1\hfill & \\ & \hfill & & \hfill & 1\hfill \end{array}\right],T_{3,2}^{k=2,5,9}=\left[\begin{array}{ccccc}1\hfill & \hfill & & \hfill & \\ & t_{11}\hfill & t_{12}\hfill & \hfill & \\ & t_{21}\hfill & t_{22}\hfill & \hfill & \\ & \hfill & & 1\hfill & \\ & \hfill & & \hfill & 1\hfill \end{array}\right],\hfill \\ \hfill T_{4,3}^{k=4,8}=\left[\begin{array}{ccccc}1\hfill & \hfill & & \hfill & \\ & 1\hfill & \hfill & & \\ & \hfill & t_{11}\hfill & t_{12}\hfill & \\ & \hfill & t_{21}\hfill & t_{22}\hfill & \\ & \hfill & & \hfill & 1\hfill \end{array}\right],T_{5,4}^{k=7}=\left[\begin{array}{ccccc}1\hfill & \hfill & & \hfill & \\ & 1\hfill & \hfill & & \\ & 1\hfill & \hfill & \\ & \hfill & t_{11}\hfill & t_{12}\hfill \\ & \hfill & & t_{21}\hfill & t_{22}\hfill \end{array}\right].\hfill \end{array}$$

(8)

Sequentially, according to Equation (7), the entire transfer matrix of the unitary MZI mesh can be inferred as

$$T=T_{2,1}^{10}{T}_{3,2}^9{T}_{4,3}^8{T}_{5,4}^7{T}_{2,1}^6{T}_{3,2}^5{T}_{4,3}^4{T}_{2,1}^3{T}_{3,2}^2{T}_{2,1}^1.$$

(9)

Since the MZI mesh can represent any unitary matrix, it is worth using the MZI mesh to represent any real-valued matrix through singular value decomposition (SVD) [92]. Any real-valued matrix $M$ with dimensions of $p\times q$ can be decomposed by SVD as

$$M=U\mathsf{\Sigma }{V}^{†},$$

(10)

where $U$ is a $p\times p$ unitary matrix, $\mathsf{\Sigma }$ is a $p\times q$ rectangular diagonal matrix with non-negative real numbers on the diagonal, and $V^{†}$ is the complex conjugate of the $q\times q$ unitary matrix $V$. From Equation (7), unitary matrices $U$ and $V^{†}$ can be implemented with the MZI mesh. The rectangular diagonal matrix $\mathsf{\Sigma }$ can be implemented with optical attenuators or optical amplifiers. Therefore, any real-valued matrix can be realized with the dimension-matched MZI mesh. Since any real-valued matrix $M$ is realized, parallel convolution of $p$ real-valued kernels with $q$ elements in each kernel can be performed with the MZI mesh.

The most representative optical interference-based CNN was proposed in 2018 [85]. The rectangular architecture MZI mesh in Figure 5a was used as the kernel matrices, and the optical delay lines introduced different time delays to implement multiple convolutional layers. The classification accuracy of the handwritten digits was simulated to 97% in the MNIST database. Then, 4 × 4 MZI-based linear optical processors were experimentally demonstrated in multiple material platforms, such as silicon-on-insulator (SOI) and silicon nitride (SiN) [86]. The triangular architecture MZI mesh has also been proposed to realize optical CNNs [87]. The fabricated SOI chip of the triangular architecture MZI mesh network is shown in Figure 5b [89] to realize three 3 × 3 kernels. Recognition accuracies of 86.9% for the MNIST database and 79.3% for Fashion-MNIST were experimentally demonstrated.

Apart from the MZI-based unitary matrix realizing the convolution operation, the optical coherent dot-product chip (OCDC) realized by the interference principle was also an important implementation of optical CNN [88]. The schematic of the chip is shown in Figure 5c, where the light wave is first divided into several branches and then sent into the dot-product chip with one as the reference and the rest to perform the dot-product operation. In each computing branch of Figure 5c, double modulators are cascaded to accomplish the dot-product. The reference branch was utilized to introduce an amplitude bias to perform real-valued computing. By reusing the chip, the AUTOMAP neural network, including matrix multiplication and convolution computing, was experimentally demonstrated and achieved image reconstruction quality comparable to that of a 32 bit digital computer.

In addition to the interference of coherent light with the MZI, the combination of WDM technology and interference was another important realization of optical CNN [37,93,94]. Figure 5d shows the application of WDM technology in a simplified butterfly-style MZI mesh [93]. In this scheme, incoherent light was utilized to avoid additional hardware costs for phase control between input channels and the influence of phase fluctuations of optical signals in off-chip fibers when using coherent light. In their experimental demonstration, a handwritten digit recognition accuracy of 94.16% was realized. As shown in Figure 5e, a compact optical convolution processing unit (OCPU) was realized by combining a multimode interference mechanism and WDM technology [37]. In this work, the process of multiplication was accomplished with multimode interference, and the process of addition was realized with square-law detection of multiple wavelengths with photodetectors. With two 4 × 4 multimode interference regions and four phase shifters, three correlated 2 × 2 real-valued kernels were realized with a convolution operation precision of 5 bit. Kernels were reconfigured by adjusting phase shifters, the compute density of 12.74 TMACs/s/mm² was demonstrated, and a classification accuracy of 92.17% was realized on handwritten digits in the MNIST database. The greatest advantage of the proposed OCPU is that the number of reconfigurable units scales linearly with the scale of the kernel size, which translates into solid potential for large-scale integration.

2.3. Optical CNN Based on Wavelength Division Multiplexing

A WDM-based CNN offers a new solution by comprehensive use of the multidimensional physical information of light waves in terms of the wavelength, amplitude, spatial, temporal, polarization, and so on. By leading into the wavelength dimension in Figure 6, VVM operation is divided into the multiplication process and the addition process, where the multiplication process is accomplished using electrooptic modulations, and the addition process is achieved via wavelength multiplexing and photoelectrical conversion at the photoelectric detector. WDM-based VVMs are familiarly split into two categories: one is the wavelength-independent weight-loaded architecture (shown in Figure 6a), and the other is the wavelength-sensitive weight-loaded architecture (Figure 6b), in which the division criteria is whether wavelength-sensitive variable optical amplitude modulators (VOAMs) are used.

First, for the wavelength-independent weight-loaded VVM shown in Figure 6a, $n$ wavelength components loaded with input data vector $X$ were parallelly sent into the computing hardware, where each light wave carried one element of the data vector $X$. Then, wavelength components underwent amplitude modulation to load the weight vector $W$, which was the multiplication process of elements in two vectors. As a main optical device in the WDM-based CNN, the variable optical amplitude modulators can be implemented using MZIs [95], microring resonators (MRRs) [96,97,98,99,100,101,102,103], phase change materials (PCMs) [39,104], and waveshapers [35,105,106,107,108,109]. The addition process was conducted using square-law photoelectric detection and the summation of the optical intensity of wavelength components at the photodetector. To obtain the summation signal of different wavelength components, the beat of optical sidebands between adjacent wavelength components should be effectively avoided. Therefore, in the WDM-based CNN system, the wavelength interval of wavelengths fed to the photodetector as well as the bandwidth for both the modulator and photodetector need to satisfy the following condition:

$$f_w-2f_m>f_p,$$

(11)

where $f_w$ is the wavelength interval of adjacent wavelength components, $f_m$ is the modulation bandwidth, and $f_p$ is the bandwidth of the photodetectors.

Figure 6b shows a wavelength-sensitive weight-loaded VVM architecture where the process of multiplication was performed with wavelength-sensitive variable optical amplitude modulators [97,98,99,100,101,102]. Wavelength components loaded with data vector $X$ were transmitted through one physical channel by wavelength-division multiplexing. Then, $n$ wavelength-sensitive variable optical amplitude modulators serially loaded $n$ elements of the weight vector $W$ and completed the process of multiplication. Finally, the weighted light waves were fed to a photodetector to complete the addition function.

The classical MRR-based real-valued VMM architecture on the parallel data inputting is shown in Figure 6c, where the MRR weight banks were used as the wavelength-sensitive variable optical amplitude modulator to implement the multiplication process. The wavelength components after wavelength multiplexing were loaded with data $X$ and then divided into $k$ channels using an optical coupler, where each channel contained $n$ wavelength components. In each channel, $n$ add-drop MRRs were used to couple the light waves with $n$ wavelengths to the drop port of the MRR. The proportion of light that each microring couples to the drop port $h_{ij}$ depends on the weight $w_{ij}$. Then, the transfer matrix for the drop ports and through ports of MRRs can be written as

$$H_{{}^{drop}}=\left[\begin{array}{c}\begin{array}{cccc}{h}_{11}& h_{12}& \cdots & h_{1n}\end{array}\\ ⋮\\ \begin{array}{cccc}{h}_{k1}& h_{k2}& \cdots & h_{kn}\end{array}\end{array}\right],$$

(12)

$$H_{{}^{through}}=\left[\begin{array}{c}\begin{array}{cccc}1-h_{11}& 1-h_{12}& \cdots & 1-h_{1n}\end{array}\\ ⋮\\ \begin{array}{cccc}1-h_{k1}& 1-h_{k2}& \cdots & 1-h_{kn}\end{array}\end{array}\right].$$

(13)

$H_{{}^{drop}}$ at the drop port and $H_{{}^{through}}$ at the through port were sent into the balanced photodetector (BPD) to make both the photoelectric conversion and difference. The transfer matrix $H$ of the MRR weight banks coupled with BPDs can be written as

$$\begin{array}{cc}\hfill H& =H_{{}^{through}}-H_{{}^{drop}}\hfill \\ & =\left[\begin{array}{c}\begin{array}{cccc}1-2h_{11}& 1-2h_{12}& \cdots & 1-2h_{1n}\end{array}\\ ⋮\\ \begin{array}{cccc}1-2h_{k1}& 1-2h_{k2}& \cdots & 1-2h_{kn}\end{array}\end{array}\right].\hfill \end{array}$$

(14)

As seen from Equation (14), real-valued VMM can be performed by adjusting the coupling coefficient of each MRR.

Additionally, WDM-based CNNs with serial data input approaches have also been reported [35,96,101,102,103,105,106,107,108,109,110]. As shown in Figure 6d, the weight vector $W$ was still weighted on the amplitudes of the wavelength components, and the 1D data $X$ were serially modulated to the amplitude of the wavelength components. Owing to the delay between different wavelengths via the delay unit (single mode fiber, dispersion compensating fiber, optical tunable delay lines, etc.), the weighted optical signal generated a time delay among the adjacent wavelength components. Following this method, the input data $X$ were serially loaded into the light waves with a single intensity modulator, and different elements of input data $X$ were fed to the photodetector at the same time with precise delay control. Owing to using a single modulator to load the data in the CNN system, it released the demand for full clock synchronization.

In 2018, Reference [99] proposed a WDM-based photonics convolutional neural network accelerator (PCNNA). Here, the MRR weight banks were the wavelength-sensitive variable optical amplitude modulators to adjust the kernels. The input data were modulated into intensities of different wavelengths. The weights were loaded while passing through the MRR weight banks, and wavelength-based weight components were summed at the photodetectors to generate the convolutional results. In Figure 7a, the analog–digital hybrid CNN computing architecture (e.g., digital electronics and analog photonics (DEAP)), as the most popular architecture, was proposed in 2019 [97]. Two MRR arrays were occupied to load the data matrix and the weight matrix. The simulation recognition accuracy on 500 images from the MNIST test dataset reached 97.6%. After that, optical tunable delay lines were introduced to serially load the input data [102,103,111] and were experimentally demonstrated in 2022 [101]. As shown in Figure 7b, different from other schemes, one convolution operation only needs a single wavelength component, and the WDM technology enables the multiple wavelength channel to form an on-chip photonic tensor flow processor (PTFP). Using the multi-dimensional multiplexing of light waves, including wavelength, space, and time, high-order tensors can be directly processed in the optical domain. The integrated photonics tensor core on phase change materials is demonstrated in Figure 7c [39]. In this scheme, phase change materials work as wavelength-independent variable optical amplitude modulators to enable efficient calculation without power to maintain the state. A chip-based optical frequency comb was introduced as the light source to perform high parallelism optical convolution processing with a speed of 2 tera-MAC operations per second. The experimental recognition accuracy on 10,000 images in the MNIST test dataset was up to 95.3%. A phase-change metasurface mode converter (PMMC) was also fabricated to form an optical CNN with 6-bit weight precision [104]. The waveguide of two spatial modes was controlled by the phase change material-based phase-gradient metasurface, and 2 × 2 real-valued kernels were experimentally demonstrated.

Otherwise, a parallel photonics convolution accelerator with ten 3 × 3 kernels in Figure 7d is illustrated using a soliton optical frequency comb as the light source to operate with a speed of 11 TOPS [35]. Herein, the 2D image data are flattened into a 1D vector and modulated to multiple wavelengths to achieve electro-optical conversion in the single intensity modulator with the serial data input approach. The weight matrix is represented by the amplitude of optical combs and adjusted with a waveshaper. The proposed photonic convolution accelerator cooperates with a fully connected optical layer to classify handwritten digits with an experimental accuracy of 88% on 50 images. With the homologous function (Figure 7e), laser arrays are used to replace the optical frequency comb to improve the robustness of the computing system and obtain an experimental recognition accuracy of 96.01% [106]. The on-demand real-time reconfiguration between the optical perceptron and optical convolution operator is also experimentally demonstrated. As illustrated in Figure 7f, the binary optical convolution operation [110] is also realized by controlling the dual-drive Mach–Zehnder modulators to work at two quadrature points, where the input data and the weights simultaneously drive two arms of the modulator and an optical fiber is occupied for dispersive delay. In 2023, a compact MRR-based photonic processing unit was reported (Figure 7g) with all the essential photonic components integrated, and a photonic-core compute density of over 1 TOPS/mm² was achieved [36]. Using the through port of the MRR as the monitor port, a weight precision of 9 bit was realized. Negative elements in kernels were accomplished via subtraction between two channels, and a handwritten digit recognition accuracy of 96.6% was experimentally illustrated, which is comparable with a digital computer.

2.4. Optical CNN Based on Tunable Optical Attenuation of Coherent Light

The last category of optical CNNs is generally based on the tunable attenuation of coherent light. In this architecture, the data matrix and the weight matrix are represented by the intensities of light waves, and two cascaded intensity modulations are used to load the input data and weights to accomplish the dot-product operation in the optical domain. In addition, optical-electrical conversion is achieved on each optical path, and the sum operation is conducted in the electrical domain.

As shown in Figure 8a [112], two acousto-optical modulator (AOM) arrays were utilized, with one for data loading and the other for kernel weighing. A single laser was employed as the coherent light and divided into several equal power beams. Then, the light waves passed through two acousto-optical modulator arrays and converted the optical power into electrical voltage at the PD array. Subsequently, a switching array decided whether the voltages were accumulated positively or negatively to realize real-valued computing. The proof-of-concept experiment was verified with a recognition accuracy of 98.9% on the MNIST dataset and 91.5% on the Fashion-MNIST dataset. P-doped–intrinsic–N-doped (PIN) current-controlled attenuators have also been reported to adjust the optical power for the weight matrix (Figure 8b) [113]. The PIN junction works in the forward bias state, and the optical power is tuned by changing the injected carriers. The input data are also represented by the optical power, and the linear dot-product process is realized with tunable optical attenuation. The addition operation is conducted in the electrical domain on the PD, and nonlinear activation is accomplished with an on-chip microring modulator. An integrated end-to-end photonic deep neural network (PDNN) with one convolutional layer and two fully connected layers was experimentally demonstrated. In this architecture, through the design of the physical connection mode of the photoelectric link, convolution and full connection were realized by the combination of photoelectricity, and nonlinearity was realized by using a microring modulator. The classification results can be directly generated at the output port of the integrated chip. The printed letters were demonstrated with a two-class classification accuracy of 93.8% and a four-class classification accuracy of 89.8%. In addition, serial dot-product coupled with integrated balanced homodyne detection was also proposed [114]. The VVM was completed in each optical channel and the parallel multiple channels enabled the VMM operation. In addition, other elements, such as MRR and MZI, have also been reported as tunable optical attenuators to realize optical CNNs [88,101].

3. Discussion and Conclusions

As an emerging research direction, optical CNNs have garnered significant attention since their inception.

Table 1. Comparison of optical CNN schemes.

Type	Parallelism	Computing Speed	Integration Density	Reconfigurability
Diffraction	high	high	low	low
Interference	low	medium	medium	high
WDM	high	high	high	high

presents a comparison of four optical CNN schemes, highlighting their parallelism, computing speed, integration density, and reconfigurability.

Diffraction-based optical CNNs exhibit advantages in terms of parallelism, computing speed, and scalability due to their utilization of spatial light. The presence of spatial light allows for a large number of neurons in each layer, facilitating the expansion of multiple channels and kernels in the spatial domain, thus enabling high parallelism. The abundance of neurons and high parallelism contributes to achieving high computing speeds. However, diffraction-based optical CNNs also suffer from notable disadvantages. The discrete components used make the system bulky, and attempts at integration result in performance degradation. Moreover, kernel-loading devices such as DOE and metamaterials are nearly impossible to reconfigure, while SLM and DMD devices have low data rates (typically ~kHz).

An interference-based optical CNN excels in reconfigurability. This scheme often utilizes MZI for kernel matrix loading, enabling rapid refresh rates in the range of tens of GHz. Despite the advantages of high-speed reconstruction offered by MZIs, their relatively large volume limits the integration density of interference-based schemes. Furthermore, the use of coherent light restricts the transmission of only one light at a time in the optical waveguide, thereby constraining parallelism and computing speed.

WDM-based optical CNN represents a promising and extensively researched solution. This scheme fully exploits the optical wavelength dimension, leading to high parallelism. The use of MRR as a wavelength-sensitive optical attenuator, with its radius as small as several micrometers and modulation rate reaching tens of GHz, further enhances computing speed, integration density, and reconfigurability.

The optical CNN based on tunable optical attenuation depends on the specific characteristics of the optical attenuator used. Due to the absence of a unified conclusion, it is not listed in the table. Additionally, apart from the four types of optical CNNs discussed above, there are other noteworthy optical CNN solutions, such as photon frequency synthesis [115] and photodetectors with adjustable responsivity [116], which warrant further research.

Presently, although optical CNNs exhibit advantages in terms of bandwidth, latency, and computational speed compared with electrical architectures, optical CNNs face challenges in surpassing the limitations of small realizable matrix sizes, a limited range of realizable functions, and low computing precision. Consequently, extensive efforts are required for optical CNNs to gain widespread usage.

First, on-chip large-scale integration needs to be broken through. The reported on-chip integrated optical accelerated computing architectures only realize the integration of tens of thousands devices at present, which is far less than their electrical counterparts. In optical computing, the power required for calculations such as electro-optical conversion, photoelectric conversion, and analog-to-digital conversion remains basically unchanged. By integrating more photonic devices, the common power required for computing will be averaged, thereby improving the energy efficiency of optical computing and giving full play to the advantages of optical computing.

Second, more functions should be realized with optical methods. Most optical CNN solutions primarily focus on the optical implementation of convolutions. Although there are related studies on the optical implementation of nonlinear activation and full connection [35,38,91,113,117,118,119,120], these studies remain relatively limited and warrant further exploration. In particular, limited by the small matrix size, it is still a challenge to achieve large-scale full connections using optical methods. Implementing more functions using optical methods will be conducive to expanding the application field of optical computing and promoting the practical application of optical computing.

Finally, the development of an in situ trainable arbitrary reconfigurable computing architecture is essential. At present, most optical CNN implementations adopt the offline training method, and the weight matrix is pretrained in the neural network simulation model. As a result, a deviation between the simulation model and the experimental system inevitably appears. In situ training, which updates the weight directly and performs the computation at the original place, offers a new form to accelerate the reconfiguration performance of the neural network and improve its precision. Several recently proposed in situ training schemes, such as physics-aware training [121], adaptive training [122], and other relevant methods [123,124], have been successfully introduced to optical computing. These approaches can be effectively incorporated into the optical CNN framework. By incorporating in situ training, it is more convenient to reconstruct the network structure (such as changing the size and number of convolution kernels, changing the number of convolution layers, etc.), and the influence of errors can be considered during the training process. In this way, the refactoring of more different, more complex classification tasks on the same hardware will become a reality, rather than the simple tasks that are fixed today (such as classifying handwritten digits).

In summary, various optical methods have been used to conduct convolution operations with the benefit of high computing speed, low power consumption, and robust parallelism. Here, the review primarily introduces the recent advancements in optical CNNs, classifying them into four categories and summarizing their respective strengths and weaknesses. The challenges faced by optical CNNs are also summarized, and the future prospects of this technology are discussed. Next, the untapped potential of optical CNNs warrants further exploration, as they hold promise in shaping next-generation artificial neural network platforms.