Deep Learning and Adjoint Method Accelerated Inverse Design in Photonics: A Review

Abstract

For photonic applications, the inverse design method plays a critical role in the optimized design of photonic devices. According to its two ingredients, inverse design in photonics can be improved from two aspects: to find solutions to Maxwell’s equations more efficiently and to employ a more suitable optimization scheme. Various optimization algorithms have been employed to handle the optimization: the adjoint method (AM) has become the one of the most widely utilized ones because of its low computational cost. With the rapid development of deep learning (DL) in recent years, inverse design has also benefited from DL algorithms, leading to a new pattern of photon inverse design. Unlike the AM, DL can be an efficient solver of Maxwell’s equations, as well as a nice optimizer, or even both, in inverse design. In this review, we discuss the development of the AM and DL algorithms in inverse design, and the advancements, advantages, and disadvantages of the AM and DL algorithms in photon inverse design.

1. Introduction

With inverse design methods, we can optimize the parameters of the integrated device, obtain a specific structure of the device, or both. Among many well-established inverse design methods, two methods of traditional inverse design originating from the research group of the University of Utah and the research group of Stanford University [1,2] have received much attention due to their high performance. In general, inverse design always combines a mathematical model with an optimization algorithm: the former describes the underlying physics, and the latter discovers the suitable shape or parameters, according to the user-predefined target performance of the device.

According to its two ingredients, inverse design can be improved from two aspects: to find solutions to Maxwell’s equations more efficiently and to employ a more suitable optimization scheme. These two types of techniques can be utilized together, and inverse design has already benefited from the rapid development of computational electromagnetic algorithms for various applications. The most popular solvers for Maxwell’s equations include the finite element method (FEM), the finite-difference time-domain (FDTD) method [3,4,5,6], the method of moments (MoM) [7,8,9], and hybrid methods [10,11].

Due to traditional computational electromagnetic frameworks becoming mature and the fast development of the various optimization algorithms, many improvements have been developed in terms of the optimization process. Based on what type of optimization scheme is employed, inverse design methods can be classified into gradient-based algorithms and non-gradient ones [12].

Non-gradient algorithms are also known as gradient-free methods. The representative methods belonging to the gradient-free scheme include evolutionary algorithms [13,14] and search algorithms (including nonlinear search methods) [15]. Evolutionary algorithms can be further divided into methods based on genetic algorithms (GA) [13,16], methods based on particle swarm optimization (PSO) [14,17], and hybrid methods tailored to specific problems [18]. GA, a typical heuristic approach, is widely used for its effectiveness, simplicity, and intuitiveness [19]. Heuristic optimization relies on a somewhat limited parameterization of the solution space and subsequent random testing of a large number of parameter sets. PSO is an optimization algorithm based on swarm intelligence, which has a strong global search ability. Except when employed alone, it is always hybridized with other search algorithms to construct a more effective optimization algorithm: for example, by mixing PSO with GA, faster and more accurate global optimization can be achieved. Due to the high computational cost of solving Maxwell’s equations, these methods tend to only work for targets with a few unknowns or with relatively simple geometries, as they require testing of a large number of solutions to find a satisfactory optimization result [20]. Many applications demand the sensitivity of some parameters along with acceptable optimization results: in essence, the sensitivity is the gradients of the optimization objective function for the given optimization parameters. Although the sensitivity can be calculated by sing the method of finite difference (FD) approximation, the associated computational cost increases in proportion to the product of the problem size and the number of design variables, which quickly makes the computation prohibitive.

Gradient-based optimization is probably the most widely used technique in photonic inverse design: in this scheme, the gradients of the objective function, relative to all design parameters of the device, are calculated in each iteration; the design parameters are then changed in the gradient direction to improve the performance of the device. The AM [21,22] and the level set method [23,24,25] are two typical methods belonging to the gradient-based optimization scheme. The level set algorithm is a numerical method based on partial differential equations, which is used to describe the contour changes of an object over a specified variable. At a high level, the major benefit offered by the level set is that it provides a systematic way to organize design possibilities. The AM is one of the most widely used algorithms in gradient-based photonic inverse design. Because the AM can calculate the derivatives over all the design parameters and because it modifies the parameters in proportion to the figure of merit (FOM) gradient using only forward and adjoint simulations at each iteration, regardless of the number of design parameters [26,27], it has been successfully applied to various photonic systems [27,28]. This is one of the major reasons why the AM has been widely adopted for the topology or shape optimization of photonics devices [28], where the number of design parameters can be very large to describe complex free-form geometries. The traditional AM was recently extended to nonlinear device modeling in the frequency domain [29]. Another attractive virtue of the AM is that it can significantly reduce the computational cost of a sensitivity calculation [30,31].

Although inverse design methods, both gradient-based and non-gradient-based, have achieved significant breakthroughs in developing new optical functions, they can encounter the difficulty of a low efficiency because they inevitably call an electromagnetic solver during the optimization: consequently, their computational costs become very high, as the design parameters to be optimized increase. In addition, gradient-based methods can easily fall into local optimal solutions, resulting in slow convergence. As a result, the inverse design approach is hampered, and other algorithms need to be developed to address these problems.

To solve the inherent difficulties of traditional inverse design methods, deep learning (DL) and its variants have emerged as an alternative solution in recent years [32]. DL, as a subset of machine learning (ML), has the potential to handle tricky high-degree-of-freedom designs [33]. With the fast development of DL and its applications in various fields, many DL networks, such as fully connected networks, convolutional networks, generative networks, and recurrent neural networks, have been utilized in photonic inverse design, and they have achieved remarkable results [34]: for example, predictive and generative models based on data-driven methods have been developed for the analysis and design of photonic crystals (PhC) [35]; in [36], a network to design free-form, all-dielectric metasurface devices was proposed by combining the conditional generative adversarial network (CGAN) with the Wasserstein generative adversarial network (WGAN). However, there are also many limitations when using DL in photon inverse designs, such as training dataset problems and inverse design non-unique problems. In addition, different designs may be produced with almost identical spectra, which may prevent correct optimization from convergence.

The combination of DL with the AM can take advantage of both techniques and thus alleviate their inherent limitations. For example, such a hybrid method was developed in [37], where photon neural networks were efficiently trained in situ and the AM was utilized to derive the photon simulation of the backpropagation algorithm. In [27], a hybrid inverse design framework was proposed by combining adjoint optimization and automatic ML with explainable AI, where ML and AI were designed to discover geometric features that cause local minima. In [38], a hybrid design approach for electromagnetic devices was developed by directly incorporating adjoint variable computation into the generative neural network, which showed the positive effects arising from the combination of DL and AM.

The rest of this review will be organized as follows. In Section 2, the AM for inverse design is discussed. In Section 3, the DL-equipped inverse design is presented. In Section 4, the hybrid AM and DL designs are reviewed. Finally, some concluding remarks are given in Section 5.

2. AM for Inverse Design

2.1. AM

The AM can effectively calculate gradients and plays an important role in inverse design for photonic applications. A very simple example of using the AM can be found in [20], which visually illustrated the mathematical process in the context of electromagnetism. Although the AM has been mainly applied to linear photonic devices [39,40,41], it has indeed been extended to model nonlinear devices in the frequency domain [29]. In particular, the method was used to design compact photonic switches in a Kerr nonlinear material, where low-power and high-power pulses were routed in different directions [29].

A schematic outlining the process of applying the AM to both linear and nonlinear cases is shown in Figure 1.

2.1.1. Linear AM

Maxwell’s equations in terms of a linear system for an optical device naturally make itself compatible with the AM [39,40,41]. The associated compart linear system can be written as follows:

$$\mathbf{A}·\mathit{e}=\mathit{b}$$

(1)

Equation (1) is a compact form, where $\mathbf{A}$ is the system matrix, $\mathit{e}$ represents the electric field, and $\mathit{b}$ is the excitation vector. Depending on the discretization method, matrix $\mathbf{A}$ can be sparse or dense. In particular, it is sparse when the linear system is generated using the FDTD [42] or the FEM [43] while it is dense when generated using the method of moments (MoM) [44].

Without loss of generality, a general real-valued objective function $\mathcal{L}=\mathcal{L}\left(\mathit{e}\right(\mathbf{\Phi }\left)\right)$ is considered, where $\mathit{e}$ is the solution vector of Equation (1) and $\mathbf{\Phi }$ is the vector denoting the design parameters associated with the system. In general, $\mathbf{\Phi }$ can be any of the design parameters, such as the parameters for the underlying passive structure, as well as those for the active modulators such as the modulation strength or the modulation phases. For example, $\mathbf{\Phi }$ can be the system object’s dielectric constant or geometric parameters. The goal of the AM is to compute the sensitivity $\frac{\partial \mathcal{L}\left(\mathit{e}\right(\mathbf{\Phi }\left)\right)}{\partial \mathbf{\Phi }}$. Suppose that $\phi$ is one element in $\mathbf{\Phi }$; what follows shows how to compute the sensitivity by computing $\partial \mathcal{L}\left(\mathit{e}\right(\mathbf{\Phi }\left)\right)/\partial \phi$,

$$\frac{\partial \mathcal{L}\left(\mathit{e}\right(\mathbf{\Phi }\left)\right)}{\partial \phi }=\frac{\partial \mathcal{L}}{\partial \mathit{e}}·\frac{\partial \mathit{e}}{\partial \phi }$$

(2)

In general, $\mathcal{L}\left(\mathit{e}\right(\phi \left)\right)$ is an explicit function of $\mathit{e}$. The term $\frac{\partial \mathcal{L}}{\partial \mathit{e}}$ can be obtained analytically with negligible computational cost. The other term $\frac{\partial \mathit{e}}{\partial \phi }$ can be computed by taking derivatives on both sides of Equation (1) with respect to $\phi$:

$$\frac{\partial \mathbf{A}}{\partial \phi }·\mathit{e}+\mathbf{A}·\frac{\partial \mathit{e}}{\partial \phi }=0$$

(3)

According to Equation (3), we have

$$\frac{\partial \mathit{e}}{\partial \phi }=-{\mathbf{A}}^{-1}·\frac{\partial \mathbf{A}}{\partial \phi }·\mathit{e}$$

(4)

By substituting Equation (4) into Equation (2), we obtain

$$\frac{\partial \mathcal{L}\left(\mathit{e}\right(\phi \left)\right)}{\partial \phi }=-\left(\frac{\partial \mathcal{L}}{\partial \mathit{e}}·{\mathbf{A}}^{-1}\right)·\frac{\partial \mathbf{A}}{\partial \phi }·\mathit{e}=-{\mathit{e}}_{adj}^T·\frac{\partial \mathbf{A}}{\partial \phi }·\mathit{e}$$

(5)

where T represents transposition and ${\mathit{e}}_{adj}$ can be written as

$${\mathbf{A}}^T·{\mathit{e}}_{adj}=-{\left(\frac{\partial \mathcal{L}}{\partial \mathit{e}}\right)}^T$$

(6)

If $\mathbf{A}$ is an explicit function in terms of $\phi$, $\frac{\partial \mathbf{A}}{\partial \phi }$ can also be analytically calculated with a negligible cost. As a result, to evaluate the sensitivity using Equation (5), the solution of the physical field $\mathit{e}$ and its adjoint counterpart ${\mathit{e}}_{adj}$ would like to dominate the computational cost of the whole optimization process. After these two fields are obtained, the sensitivity of the photonic device concerning any number of parameters can be evaluated with a few additional computational costs.

2.1.2. Nonlinear AM

Although the AM was originally developed to handle linear systems, it has been extended to nonlinear systems [29]. In [29], a set of parameters $\mathbf{\Phi }$ were optimized that optimize a real-valued objective function $\mathcal{L}=\mathcal{L}(\mathit{e},{\mathit{e}}^{\ast },\mathbf{\Phi })$, where $\mathcal{L}$ is, most generally, a nonlinear function of its arguments and ${\mathit{e}}^{\ast }$ is the conjugation of $\mathit{e}$. Here, in nonlinear problems, $\mathit{e}$ is the solution to a nonlinear equation:

$$\mathbf{f}(\mathit{e},{\mathit{e}}^{\ast },\mathbf{\Phi })=0$$

(7)

For example, Equation (7) may represent steady-state Maxwell’s equations with an intensity-dependent permittivity distribution, where $\mathit{e}$ is the electric field distribution. For problems studied here, the natural choice is to take $\mathit{e}$ and ${\mathit{e}}^{\ast }$ as independent parameters, which is necessary for differentiation, as opposed to separately treating the real and imaginary parts of $\mathit{e}$ [29,45]. The solution to Equation (7) may be found with any nonlinear equation solver, such as the Newton–Raphson method [46].

The goal of the optimization is to optimize the objective function concerning the design variables $\mathbf{\Phi }$. With that aim in mind, it is necessary to compute the sensitivity of $\mathcal{L}$ to each element of $\mathbf{\Phi }$. Similar to Equation (2), the derivative of the objective function concerning a single parameter $\phi$ can be derived as

$$\frac{d\mathcal{L}}{d\phi }=\frac{\partial \mathcal{L}}{\partial \phi }+\frac{\partial \mathcal{L}}{\partial \mathit{e}}·\frac{d\mathit{e}}{d\phi }+\frac{\partial \mathcal{L}}{\partial {\mathit{e}}^{\ast }}·\frac{d{\mathit{e}}^{\ast }}{d\phi }$$

(8)

Equation (8) can be rewritten into the matrix form as

$$\frac{d\mathcal{L}}{d\phi }=\frac{\partial \mathcal{L}}{\partial \phi }+\left[\begin{array}{cc}\frac{\partial \mathcal{L}}{\partial \mathit{e}}& \frac{\partial \mathcal{L}}{\partial {\mathit{e}}^{\ast }}\end{array}\right]·\left[\begin{array}{c}\frac{d\mathit{e}}{d\phi }\\ \frac{d{\mathit{e}}^{\ast }}{d\phi }\end{array}\right]$$

(9)

The terms $\frac{d\mathit{e}}{d\phi }$ and $\frac{d{\mathit{e}}^{\ast }}{d\phi }$ can be obtained by differentiating Equation (7) as follows:

$$\frac{d\mathbf{f}}{d\phi }=\frac{\partial \mathbf{f}}{\partial \phi }+\frac{\partial \mathbf{f}}{\partial \mathit{e}}·\frac{d\mathit{e}}{d\phi }+\frac{\partial \mathbf{f}}{\partial {\mathit{e}}^{\ast }}·\frac{d{\mathit{e}}^{\ast }}{d\phi }=0$$

(10)

Combining Equation (10) with its complex conjugate yields

$$\left[\begin{array}{cc}\frac{\partial \mathbf{f}}{\partial \mathit{e}}& \frac{\partial \mathbf{f}}{\partial {\mathit{e}}^{\ast }}\\ \frac{\partial {\mathbf{f}}^{\ast }}{\partial \mathit{e}}& \frac{\partial {\mathbf{f}}^{\ast }}{\partial {\mathit{e}}^{\ast }}\end{array}\right]·\left[\begin{array}{c}\frac{d\mathit{e}}{d\phi }\\ \frac{d{\mathit{e}}^{\ast }}{d\phi }\end{array}\right]=-\left[\begin{array}{c}\frac{\partial \mathbf{f}}{\partial \phi }\\ \frac{\partial {\mathbf{f}}^{\ast }}{\partial \phi }\end{array}\right]$$

(11)

As a result, we can rewrite Equation (9) as

$$\frac{d\mathcal{L}}{d\phi }=\frac{\partial \mathcal{L}}{\partial \phi }-\left[\begin{array}{cc}\frac{\partial \mathcal{L}}{\partial \mathit{e}}& \frac{\partial \mathcal{L}}{\partial {\mathit{e}}^{\ast }}\end{array}\right]·{\left[\begin{array}{cc}\frac{\partial \mathbf{f}}{\partial \mathit{e}}& \frac{\partial \mathbf{f}}{\partial {\mathit{e}}^{\ast }}\\ \frac{\partial {\mathbf{f}}^{\ast }}{\partial \mathit{e}}& \frac{\partial {\mathbf{f}}^{\ast }}{\partial {\mathit{e}}^{\ast }}\end{array}\right]}^{-1}·\left[\begin{array}{c}\frac{\partial \mathbf{f}}{\partial \phi }\\ \frac{\partial {\mathbf{f}}^{\ast }}{\partial \phi }\end{array}\right]$$

(12)

In analogy with the linear case, we obtain the gradient $\frac{d\mathcal{L}}{d\phi }$ by solving an additional linear system. To this end, a complex-valued adjoint field ${\mathit{e}}_{adj}$ is defined as the solution to

$${(\partial \mathbf{f}/\partial \mathit{e})}^T·{\mathit{e}}_{adj}+{(\partial {\mathbf{f}}^{\ast }/\partial \mathit{e})}^T·{\mathit{e}}_{adj}^{\ast }=-{(\partial \mathcal{L}/\partial \mathit{e})}^T$$

(13)

where T represents transposition. The variable ${\mathit{e}}_{adj}$ and its conjugate can be computed by solving the following system:

$${\left[\begin{array}{cc}\frac{\partial \mathbf{f}}{\partial \mathit{e}}& \frac{\partial \mathbf{f}}{\partial {\mathit{e}}^{\ast }}\\ \frac{\partial {\mathbf{f}}^{\ast }}{\partial \mathit{e}}& \frac{\partial {\mathbf{f}}^{\ast }}{\partial {\mathit{e}}^{\ast }}\end{array}\right]}^T·\left[\begin{array}{c}{\mathit{e}}_{adj}\\ {\mathit{e}}_{adj}^{\ast }\end{array}\right]=-\left[\begin{array}{c}\frac{\partial \mathcal{L}}{\partial {\mathit{e}}^T}\\ \frac{\partial \mathcal{L}}{\partial {\mathit{e}}^{\ast T}}\end{array}\right]$$

(14)

As can be seen, the adjoint problem, which is required to determine the derivative of the objective function, is linear, even though the physical problem is nonlinear, as defined by Equation (7). Finally, the gradient of the objective function can be written as

$$\frac{d\mathcal{L}}{d\phi }=\frac{\partial \mathcal{L}}{\partial \phi }+2\Re ({\mathit{e}}_{adj}^T·\partial \mathbf{f}/\partial \phi )$$

(15)

where $\Re (·)$ denotes taking the real part. In deriving Equation (14), we have used the fact that both $\mathcal{L}(·)$ and $\phi$ are real [29]. In the case of multiple parameters $\mathbf{\Phi }$, we just need to replace $\partial \mathbf{f}/\partial \phi$ with the matrix $\partial \mathbf{f}/\partial \mathbf{\Phi }$. The cost of computing the gradients $\partial \mathbf{f}/\partial \mathbf{\Phi }$ increases slowly with the number of elements in $\mathbf{\Phi }$ since ${\mathit{e}}_{adj}$ only needs to be solved once regardless of the number of parameters. This virtue makes large-scale, gradient-based optimization possible.

2.2. Application of AM in Photonic Inverse Design

The AM can be integrated into gradient-based topology optimization, which aims to distribute materials across a given design domain to reach a predefined performance goal [47]. Although topology optimization was originally developed to solve mechanical design problems, it is now widely employed in the fields of photonic crystals, waveguides, resonators, filters, and plasma excitations in system design. In topology optimization, the material density of each element or mesh point is a design variable by parameterizing the geometry as elements like pixels. Consequently, the optimization is always featured by a large number of design parameters and complete design freedom [48]. Such flexibility during the design is often viewed as one of the great advantages of topology optimization. In [47], the design domain was taken to be larger than in the original work to show that the topology optimization can effectively utilize all available space. In [49], mechanical design problems with more than a billion design variables proved that topology optimization in any practical sense was able to provide unlimited design freedom.

In fact, the AM perfectly matches the topology optimization scheme because it can offer sensitivity information at a low computational cost, which makes a large number of design parameters possible or more efficient [50,51]. Due to this nice match, the AM has been used for topology optimization in the context of mechanical engineering and photonics engineering for decades [51,52,53]. In [50], the T-junction in the photonic crystal waveguide was designed. A low-loss and broadband two-mode (de-)multiplexer, optimized via topology optimization with the AM and experimentally verified, has been developed for (de-)multiplexing the fundamental and first-order transverse-electric modes in a silicon photonic wire [54]. In [55], topology optimization is used with the AM for the design of optical sub-wavelength gratings. Completely unexpected metasurface designs for challenging multi-frequency, multi-angle problems, including designs for fully coupled multi-layer structures with arbitrary per-layer patterns, can be computationally discovered via topology optimization with the AM [56]. In [57], a design method based on topology optimization is applied to optical waveguide devices.

Compared with other design methods, the AM can obtain the gradient of the objective function relative to all design parameters through two full-field simulations [58]. This optimization process is suitable for structures that simultaneously achieve various design goals and have a large parameter space [59]. As a result, except for topological optimization, the AM has also been widely applied to other types of photonic inverse design tasks, more or less, along with different acceleration techniques. In [28], large metasurfaces were optimized, where the AM is used to calculate the gradient of electric field strength relative to all design parameters. In the optimization process, multipole expansion theory (MET) was employed to accelerate the computation, which offered orders-of-magnitude-higher efficiency than the FDTD-based optimization method did. In [60], a dual-wavelength metastructure was optimized by the AM. It was shown that the proposed design method was more efficient than the traditional one used in metasurface design. In [59], a tunable mode converter filled with liquid crystals was designed using an inverse design framework based on the AM. In this work, the AM with multiple constraints was developed along with a multi-objective function, which can be applied to any multi-state optics. The original AM cannot be directly combined with the mode extension method because the associated gradients cannot be as clearly defined as that in the finite difference method, finite element time domain, or finite element method. To overcome this difficulty, automatic differentiation techniques were employed to generalize the adjoint variable method to arbitrary computational graphs [61].

It should be noted that, depending on the type of analysis, the additional computational cost per constraint sensitivity may range from a few percent for cases where extra right-hand sides are generated for an already factorized system, to almost 100% for the transient problem cases.

As discussed in Section 2.3, the AM is a local optimization method. In [62], to alleviate this drawback, the AM was integrated with global optimization, i.e., particle swarms, where a 6-times-improved angle and 6-times-improved efficiency were reached compared with current state-of-the-art devices.

Inherently, the conventional AM can be hardly applied to the inverse design of digital devices because one cannot calculate the gradient of a digital pattern [63]. In [63], an efficient inverse design of “digital” subwavelength nanophotonic devices was realized by employing the AM, where a single-mode 3 dB power divider and a dual-mode demultiplexer were designed. Compared with the direct-binary search (DBS) brute-force method, the AM can improve the design efficiency by nearly five times. And, the performance optimization can reach approximately the same level. The digital AM is a hybrid of topology optimization and the brute-force method, which improves the inverse design efficiency of high-performance digital subwavelength nanophotonic devices.

Although the AM was originally developed in terms of linear problems, it is nowadays indeed extended to nonlinear cases. For example, the AM equipped with topology optimization was employed to design one-dimensional nonlinear nanophotonic structures [47], where the third-order instantaneous Kerr material nonlinearity was considered. The strong nonlinearity in topology optimization-based computational design problems makes it difficult to solve the system directly [64]. Therefore, the iterative approach is widely adopted. In the iterative approach, the descent direction can be defined based on the adjoint derivative, where the state and adjoint variables are obtained by solving the partial differential equations and corresponding adjoint equations, respectively.

2.3. Limitations of AM in Photon Inverse Design

Although the AM has been successfully applied to a variety of photonic systems for optimization design, it is still limited by some inherent drawbacks. As it is known, the AM is inherently a local optimization method since it heavily relies on gradient-based information [65]. Thus, it shares the same drawbacks as the gradient-based methods. In particular, the AM, more generally speaking, gradient-based optimization algorithms, is susceptible to becoming stuck in local minimum valleys or saddle points because the design space for electromagnetic structures is predominantly nonconvex [31,66,67]. A remedy is to run the optimization process multiple times, typically by using random starting points. With the remedy, a single optimization target can always be derived [68]. Of course, a nice starting point can be selected if a region of high-performance devices is known in advance to avoid multiple runs of the optimization process.

The AM is feasible for very large problems because the computation of the sensitivities for an arbitrary number of design variables constitutes only a solution process of one extra linear system. However, the storage of the gradient information for the whole optimization process may become a main challenge. A remedy is to obtain the gradient information on the fly instead of storing it [51]. Similar to other traditional inverse design methods, adjoint topology optimization still has inherent drawbacks such as time-consuming numerical calculations or simulation processes [69].

The explicit computation of gradients in the AM is sometimes challenging for certain photonics problems [70]. The applications of the AM can be affected by a set of constraints that must be satisfied for specific engineering applications. For the associated optimizations, spatial filters, thresholding steps, or additional merit function terms are often necessary. For instance, the adjoint-based shape optimizations are mostly constrained to specific geometries such as spherical or ellipsoidal Mie scatterers [71]. They rely on approximating transmission coefficient gradients with polynomial proxy functions [72], approximating finite differences [73], or using methods like the level set method [20] or surface integrals over scatterer boundaries [60]. These operations can disrupt gradient computation [26,65,74]. To make it worse, for general photonic devices, which have complex light–matter interactions, their performance metrics and design parameters are often not well described by analytical forms or proxy functions. Also, in the case of rigorous coupled-wave analysis (RCWA) [75], the transmission and reflection properties of a structure depend on a global scattering matrix with elements that are functions of the eigenmodes of each layer. While it is possible to derive adjoint fields in RCWA, changing variables or scatterer geometries requires additional derivation steps for different cases, leading to extra work when switching parameterizations [70].

If the constraint is taken to be ‘hard’ and so must be satisfied at all stages of the optimization procedure, we need to know both the value of the constraint function and its linear sensitivity to the design variables. The latter requires an extra adjoint calculation. The more hard constraints exist, the more extra adjoint calculations are required. This type of constraint, therefore, undermines the computational cost benefits of the AM. If the number of hard constraints becomes as large as the number of design variables, the benefit can entirely be lost.

In [31], limitations of the discrete AM were discussed in the context of computational fluid dynamics. Suppose the objective function has a least-squares form as

$$\mathcal{L}\left(\mathit{e}(\mathbf{\Phi })\right)=\frac{1}{2}\sum _n{\left({\mathit{e}}_n(\mathbf{\Phi })-{\mathit{E}}_n\right)}^2$$

(16)

The gradient can be computed by

$$\frac{\partial \mathcal{L}}{\partial \phi }=\sum _n\frac{\partial \mathcal{L}}{\partial \mathit{e}}·\frac{\partial \mathit{e}}{\partial \phi }({\mathit{e}}_n(\mathbf{\Phi })-{\mathit{E}}_n)$$

(17)

The second-order derivative can be written as

$$\frac{{\partial }^2\mathcal{L}}{\partial {\phi }_i\partial {\phi }_j}\approx \sum _n\left(\frac{\partial \mathcal{L}}{\partial \mathit{e}}\frac{\partial \mathit{e}}{\partial {\phi }_i}\right)\left(\frac{\partial \mathcal{L}}{\partial \mathit{e}}\frac{\partial \mathit{e}}{\partial {\phi }_j}\right)$$

(18)

where it is assumed that ${\mathit{e}}_n(\mathbf{\Phi })-{\mathit{E}}_n$ is small. The direct linear approach described using Equations (16)–(18) gives the approximate Hessian matrix, leading to very rapid convergence for the optimization iteration. In contrast, the AM provides no information on the Hessian, so optimization methods such as BFGS [76], which build up an approximation to the Hessian, take more steps to converge than the direct linear approach for least-squares applications. Of course, for the case with a large number of design variables, the AM may still be more efficient, since the cost of each step is significantly higher when the sensitivities are evaluated directly.

3. DL for Photonic Inverse Design

Although the inverse design methods including those combined with the AM have been widely employed in photonic research, the process can still be time and computationally intensive [77] because, in each iterative step during the optimism, Maxwell’s equations should be repeatedly solved in terms of a new set of parameters. Since fabrication techniques allow for more complex three-dimensional designs, photonic designs can be more complex and the ranges of the search parameters become larger. The associated optimization process becomes increasingly resource-intensive [78,79,80].

Inspired by the fast development of DL, people have combined the DL techniques with inverse design [33,66,81,82,83]. At present, DL has been developed rapidly in the field of photonic device inverse design, which can be more efficient than traditional iterative optimization methods.

The interaction of nonlinear optics and DL domains has revealed great potential in recent years [84], both for the understanding of physical systems, e.g., to speed up the study of nonlinear pulse propagation [85] or to improve nonlinear effect compensation [86] in fibers and for the development of photonic-based hardware to accelerate DL calculations relying on third [87] or second [88]-order nonlinear effects.

Figure 2 shows some DL networks applied to photonic inverse design. DL networks are featured by their capability in capturing and modeling highly nonlinear data relationships. A typical DL architecture is a multilayer stack of simple modules, all (or most) of which are subject to learning, and many of which compute non-linear input–output mappings [89]. With multiple non-linear layers, a DL network can effectively predict nonlinear optical (NLO) phenomena such as a photonic device (NLO crystal) or a photonic technique (two-photon excitation microscopy). Therefore, DL can mimic nonlinear physics-based relationships, e.g., those between photonic-system geometries and their electromagnetic responses; they provide a fresh perspective on the forward and inverse problems.

When applied to the inverse design, DL can be used as an efficient solver to provide a fast solution to Maxwell’s equations. To this end, DL models are trained with nonlinear activation functions and backpropagation to intelligently learn nonlinear relationships between input parameters and output electric fields. The well-trained neural models, including DL ones, always offer the possibility of finding solutions outside the boundaries of the training data. That is, the models can transfer knowledge, an approach of the Maxwell’s equations known as “transfer learning” [94], which is generally not available in the traditional electromagnetic solvers.

The DL models can also be a nice optimizer for the inverse design, especially for high-dimensional systems. As discussed previously, the AM is inherently a local optimization scheme that suffers from the lack of global information. According to the successful applications of DL in various fields, DL can be more robust than the AM in recovering global information. Additionally, in the absence of prior knowledge of the parameter landscape, many problems become difficult to be optimized as the number of parameters increases. However, DL can overcome this difficulty to some extent due to its nice generalizability after the neural networks are well trained.

3.1. DL Networks Applied to Photonic Inverse Design

Deep neural networks are often built on top of fully connected networks (FCNs) and convolutional neural networks (CNNs). FCNs can be viewed as the most primitive type of neural networks, which are composed of multiple layers of neurons, each being connected to all other ones on the adjacent layer. The fully connected nature offers FCNs the ability to simulate complex transformations. The computation between two adjacent layers can be mathematically described by a matrix–vector product or its tensor counterpart. Suppose that the number of neurons is N for the two adjacent layers, and the computational complexity of the associated layers reaches $O\left(N^2\right)$. The high complexity prevents FCN in practice from having a large number of layers and neurons. CNNs are an improved alternative to FCNs where the time-consuming matrix–vector operations (or its tensor counterpart) are replaced with the convolution operations. The invariance of the translation of the input tensor in CNNs enhances the ability of the networks to capture features from image/audio data with strong spatial/temporal correlation. When it comes to sequential data, recurrent neural networks (RNNs) [95] become the most commonly used models, which can be unambiguously generalized to sequentially connected neurons.

3.1.1. Fully Connected Networks (FCNs)

In [92], the plasmonic waveguide-coupled with cavities structure (PWCCS) was optimized by employing a FCN, as shown in Figure 3. The effectiveness of the FCN in optimizing the PWCCS was verified via the selection of Fano resonance derived from PWCCS and plasmon-induced transparency effects. In this work, the genetic algorithm was employed to design network structures and to guide the selection of hyperparameters for the FCN. Such an approach not only enables the high-precision inverse design of PWCCS but also optimizes some key performance indicators of the transmission spectrum. In [96], the inverse design of the edge state of the curved wave topology was realized via FCNs. In this work, two neural networks were constructed for the forward and backward predictions of bandgap width and geometric parameters. The dataset used to train a feed-forward artificial neural network was generated using the plane wave expansion method (PWE) with a sweeping of the arrangement radii.

3.1.2. Convolutional Neural Networks (CNNs)

As it is known, CNNs can capture the local correlation of spatial information in the input data. This property is highly desired in photonics applications as the devices are always represented by high-dimensional spatial data. Consequently, CNNs and their improved variations have attracted many researchers who applied them to the solution of inverse design problems. For example, it has been proved that CNNs had the potential to solve the problem of the inverse design of thin film metamaterials, probing the parameter space of a given material and thickness library globally, which can be difficult and expensive for traditional methods [97]. The full generalization ability of neural networks, especially CNNs, to generate systems with the desired spectral response to accommodate various input design parameters has been revealed.

In [98], the CNN was employed to be an inverse design tool to achieve high numerical accuracy in plasma element surfaces, which proved that the CNN was an excellent tool for the design. More specifically, the CNNs are capable of identifying peaks and troughs of the spectrum at the expense of low computational cost. Key geometric parameters can reach an accuracy as high as ±8 nm. In this work, the comparison of CNNs and FCNs shows that CNNs had higher generalization capabilities. Additionally, it has been demonstrated that batch normalization can improve the performance of CNNs.

3.1.3. Recurrent Neural Networks (RNNs)

RNNs tackle problems associated with sequential data such as sentences and audio signals. The network receives sequential data one at a time and incrementally generates new data series. It was demonstrated in [99] that, for photonic design, RNNs were suitable to model optical signals or spectra in the time domain with a specific line shape originating from various modes of resonance. In [100], RNNs were implemented to analyze optical signals and to equalize noise in high-speed fiber transmission. In [77], RNNs were utilized to find the correlation within 2D cross-sectional images of plasmonic structures, where the results showed that the network was able to predict the absorption spectra from the given input structural images. It was revealed in [101] that the performance of RNNs can be enhanced by adopting advanced varieties of RNNs, such as Long Short-Term Memory (LSTM) [102] and gated recurrent unit (GRU). In [103], traditional neural networks, RNNs with LSTM and an RNN with GRU were used to construct the inverse design of MPF. As indicated by the results in [103], according to the training results, it was found that GRU-based RNNs were the most effective in predicting frequency response with the highest accuracy. The study in [104] demonstrated that, in combination with CNNs, RNNs were also utilized to enhance the approximation of the optical responses of nanostructures that are illustrated in images. As revealed in [99], network systems hybridizing CNNs and RNNs present a promising method for modeling and designing photonic devices with unconventional spatiotemporal properties of light.

3.1.4. Deep Neural Networks (DNNs)

A DNN is a network where the number of hidden layers is much greater than 1. The DNNs can be divided into discriminant neural networks and generative ones. In this subsection, we focus on the discriminant DNNs and delay the discussion on the generative ones in the following Section 3.1.5.

As pointed out in [105], neural networks can be used in two different ways. The first method takes the configuration of the device to be designed including the structural parameter (such as the geometrical shape of a nanostructure) as the input and the predicted electromagnetic response of the device (such as transmission spectra or differential scattering cross-section) as the output. These neural networks can be used to replace the computationally expensive electromagnetic simulations in the optimization loop, greatly reducing the design time. In [105], the DNNs belonging to this method were denoted by forward-modeling networks because they compute electromagnetic response from the device. Essentially, a forward-modeling DNN is an electromagnetic solver. The second type of neural network, named after inverse-design networks, takes the electromagnetic response as the input and directly outputs the configuration of the device. The inverse-design networks act as an inverse operator that converts the electromagnetic response to the configuration of the device. However, one significant difficulty in training inverse-design DNNs arises from a fundamental property of the inverse scattering problem: the same electromagnetic response can be produced using many different designs. Different from the previous remedy to this difficulty where the training dataset was divided into distinct groups, where there was a unique design within each group corresponding to each response, the approach of cascading an inverse-design network with forward modeling was proposed in [105]. The obtained network was named the tandem DNN. Numerical experiments in [105] indicated that the tandem DNNs can be trained by datasets containing nonunique electromagnetic scattering instances.

In [93], the so-called iterative DNN was proposed, where trained weights of the forward modeling network [105] were fixed and gradient-descent methods based on backpropagation were employed. The iterative DNN and tandem DNNs were compared in terms of the transmission spectrum design of bow nanoantennas, where the results demonstrated that the two types of DNN architectures reached comparable performance.

In [106], three models were investigated for designing nanophotonic power splitters with multiple splitting ratios. The first model employed the DNN as a forward-modeling network to predict the spectral response (SPEC) given hole vectors (HV). The second one utilized the DNN as an inverse-design network to construct HV given a target SPEC. The third one was based on a stochastic generative model which implicitly integrated forward and inverse-design networks. In addition, a bidirectional network consisting of the forward and inverse-design networks was proposed [107]. The forward-modeling network was trained subsequently to the inverse-design network, which was used to predict the geometry from the transmission spectrum.

In [108], DNN models were used as both the electromagnetic solvers and the optimizer. By building the DNN with an FDTD solver, the proposed method can be used to efficiently design a silicon photonic grating coupler, one of the fundamental silicon photonic devices with a wavelength-sensitive optical response.

To facilitate multi-tasks inverse design, a topology optimization method based on the DNN in the low-dimensional Fourier domain was proposed in [109]. The DNN took target optical responses as inputs and predicted low-frequency Fourier components, which were then utilized to reconstruct device geometries. By removing high-frequency components for reduced design degrees of freedom (DoFs), the minimal features were controlled and the training was sped up. In [110], a forward-modeling DNN was paired with evolutionary algorithms, where the DNN was used only for preselection and initialization. The method utilizes a global evolutionary search within the forward model space and leverages the huge parallelism of modern GPUs for fast inversion.

A well-organized tutorial was given in [111], where the process of deep inverse learning applied to AEM problems was given in a step-by-step approach. In [111], common pitfalls of training and evaluation of DL models were discussed and a case study of the inverse design of a GaSb thermophotovoltaic cell was given.

3.1.5. Deep Generative Models

Generative models generally seek to learn the statistical distribution of data samples rather than the mapping from input to output data. In this respect, they can be viewed as the complement techniques of the conventional optimization and inverse design [35]. In generative models, the joint distribution of the input and output is employed to optimize a certain objective in a probabilistically generative manner instead of determining the conditional distribution and thus decision boundaries.

Generative adversarial networks (GANs) are the typical deep generative models built using generators and discriminators, which have been used for the design and optimization of dielectric and metallic metasurfaces due to their ability to generate massive nanostructures efficiently [112,113].

In [114], a conditional deep CGAN was employed as an inverse-design network, which was trained on colored images encoded with a range of material and structural parameters. It was demonstrated that, in response to target absorption spectra, the CGAN can identify an effective metasurface in terms of its class, material properties, and overall shape. In [115], a meta-heuristic optimization framework, along with a resistant autoencoder (AE), was designed and demonstrated to greatly improve the optimization search efficiency for meta-device configurations featuring complex topologies.

In [36], a network combining the CGAN with the WGAN was proposed to design free-form, all-dielectric metasurface devices. This work indicated that the proposed approach stabilized the training process and gave the network the ability to handle synthetic metasurface design problems. It also demonstrated in this work that GAN-based approaches were the preferred solution to multifunctional inverse design problems.

In [12], research was conducted on inverse models that utilize unsupervised generative neural networks to exploit the benefits of not requiring training data samples. However, time-consuming electromagnetic solvers must be integrated into the generative neural networks to guarantee compliance with the laws of physics. After training, unsupervised generative neural networks can also generate new geometric parameters based on the optical response of the target without the need for any electromagnetic solver [12].

3.2. Limitations of DL in Photon Inverse Design

Although DL is attractive in inverse design photonics, there are still limitations with DL algorithms. These limitations include the DoFs, the local minimum, the difficulty in preparing the DL training dataset, the nonunique solutions, and difficulties associated with generalization.

3.2.1. Degrees of Freedom

In topological optimization, the number of DoFs can be very large, which always makes the DL very hard to train. The relationship between geometric parameters and optical performance is very similar to the mathematical non-deterministic polynomial difficulty (NP difficulty) problem [116] and very hard to define using explicit functions. As shown in Figure 4 from [12], the designer extracts the geometric parameter x from a device structure, such as an empirical structure, the QR code one, or an irregular one.

What ML method can be employed in inverse design is related to the DoFs of the photonic structure. For the case with a relatively small number DoFs, many DL networks can be employed in the inverse design. However, the DoFs continue to grow to thousands or more, and the huge dimension of the optimization space prohibits methods that require large amounts of data or a lot of simulation iterations. In this case, the generative models, such as GANs and VAEs, can be leveraged to reduce the dimensionality of the design structure and optical response [33,117].

Additionally, the DoFs of photonic devices could potentially impact the NLO phenomenon. For example, due to the varying interlayer interactions associated with different twist angles, the twisting DoFs have been widely applied to engineer the bands of van der Waals layered structures. In that work, the twist-angle-dependent second-harmonic generation (SHG) from twisted bilayer graphene samples was demonstrated along with their correlation with the evolving hybrid band structure [118]. Nonlinear optics will probably be increasingly important since people can reach a better understanding of the underlying physics although the design involves many DOFs, i.e., many photonic DOFs, such as spatial modes, frequencies, and polarizations, and many design parameters, such as the space- and time-dependent distributions of refractive index and loss/gain in photonic structures [119]. Furthermore, reconfigurable control of many modal DOFs can be achieved with spatial light modulators or digital micromirror devices, which are now available with resolutions (i.e., number of pixels) close to ${10}^7$ [119]. In general, the number of design DOFs routinely available to optical scientists and engineers now exceeds the technology from past decades by 3 to 5 orders of magnitude [119].

3.2.2. Training Dataset

DL often requires large and high-quality training datasets, which are generally unavailable in practice. The large amount of data required for DL could not be readily available and is usually created using simulation methods such as RCWA, FEM, and FDTD, which are time- and computation-expensive [33,90,120]. In [35], a dataset of 20,000 two-dimensional photonic crystal unit cells and their associated band structures was built, enabling the training of supervised learning models. Using these datasets, a high-accuracy CNN for band structure prediction was demonstrated, with orders-of-magnitude speedup compared with conventional theory-driven solvers. In [107], a DNN was trained using a dataset composed of 15,000 randomly generated device layouts containing eight geometrical parameters. In [121], to train a forward network model for prediction of electromagnetic response, a dataset of 90,000 samples was generated using $S^4$ software (version 2) [122], where the lattice constant was set between 50 nm and 150 nm and the maximum thickness of each dielectric was limited to 150 nm.

An effective way is to co-construct large datasets of various optical designs with the efforts of the optical community [33,123] by avoiding the repeated generation of simulation data. To use generative models instead of real datasets is also a nice alternative [32]. Another solution is to develop networks suitable for small training datasets. In [27], an approach based on explainable artificial intelligence was developed for the inverse design, demonstrating that their method works equally well on smaller datasets. In [124], an efficient tandem neural network with a backpropagation optimization strategy was developed to design one-dimensional photonic crystals with specific bandgaps using a small dataset.

In [125], methods to generate datasets with high quality were discussed to overcome the problem arising from the lack of training data. The optimization algorithm in [126] to generate training data may perform less efficiently as it is time-consuming for the case of many free parameters. The concept of iterative training data generation has emerged as the remedy, which allows the neural network to learn from previous mistakes and to significantly improve its performance on specific design tasks. However, iterative procedures can be computationally expensive since data generation is slow and the network may need to be re-trained several times on increasing amounts of training samples [115]. The solution is to reduce the amount of re-training by accelerating convergence by assessing the quality and uncertainty of the ANN output from multiple predictions. In addition, transfer learning has also been applied in nano-optics problems to enhance the performance of the ANN when the available data are limited [127].

3.2.3. Nonuniqueness Problem

Since different designs may produce almost identical spectra, this may prevent ML algorithms from converging correctly. Many researchers have developed various solutions to this problem. In [77,121], a method adding forward modeling networks to the inverse design DNN architecture was employed, which can be viewed as one of the most common ways to overcome the nonunique problem. In [128], an approach modeling the design parameters as multimodal probability distributions rather than discrete values was proposed, which can also overcome the nonunique problems. In [129], a physics-based preprocessing step to solve the size mismatch problem and to improve spectral generalizability was proposed. Numerical experiments in this work indicated that the proposed approach can provide accurate prediction capabilities outside the training spectrum. In [124], an algorithm called adaptive batch normalization was proposed, which was a simple and effective method that accelerates the convergence speed of many neural networks.

3.2.4. Local Minimum

Similar to the AM, DL also to some extent relies on gradient information to carry out the optimization. Thus, DL inverse design can also suffer from local minimum problems. In [130], an approach to overcome the local minimum problem was proposed by constructing a combined loss function that also fits the Fourier representation of the target function. Physically, the Fourier representation contains information about the target function’s oscillatory behavior, which corresponds directly to specific parameters in the input space for many resonant photonic components. In [33], the generative models were incorporated into traditional optimization algorithms to help avoid local minimum problems. Because generative models are essentially transforming raw data into another representation, the local minimum problem in the original parameter space may be eliminated, which can alleviate the local minimum problem if optimized in a sparse representation. In [131], deep generative neural networks based on global optimization networks (GLOnets) were configured to perform categorical global optimization of photonic devices. This work showed that, compared with traditional algorithms, GLOnets can find the global optimal several orders of magnitude faster.

3.2.5. Generalization Ability

The lack of generalization can also limit DL in inverse design [132]. DL is like a black box, making it challenging to understand and interpret its results and reliability, especially when dealing with imperfect datasets or data generated by adversarial methods. Due to the challenges of validating and replicating DL results, a set of community-wide recommendations in biology for DL reporting and validation (data, optimization, model, and evaluation) was recently published. The given recommendations help to handle questions that need to be addressed when reporting DL results and are also largely applicable to the fields of optics and photonics [91].

3.2.6. Problem of Rerunning

When the optimization goal is modified, a new inverse optimization process needs to be rerun, maybe from scratch. Since a single run of traditional inverse design optimization is always time- and computation-expensive as it may require hundreds of rounds of simulations, rerunning will consume a large amount of computing resources [77]. Moreover, when the optimization goal is modified, it is necessary to recreate the dataset if DL is employed, which is large in amount and not readily available. In the AM, the change of optimization goal requires additional derivation steps for different cases, leading to extra work when switching parameterizations [70].

4. Hybridization of AM and DL For Inverse Design

The use of DL can overcome some limitations of the AM, and vice versa. Many recent efforts have been made to hybrid DL with the AM [132]. It was demonstrated that adjoint optimizations are capable of achieving state-of-the-art performance and are orders of magnitude more computationally efficient than alternative optimization methods, including DL [133,134,135]. As pointed out in Section 2.3, the AM is local in nature and therefore limited by corresponding limitations. The hybridization of DL with the AM can overcome this problem to some extent [27]. As shown in Figure 5 from [27], an inverse design framework was proposed by combining adjoint optimization, automatic ML, and explainable artificial intelligence.

Redefining adjoint-based optimization as training that generates neural networks is often applied to physical systems that can take advantage of gradients to improve performance where training is performed by calculating the forward and adjoint electromagnetic simulations of outputted devices [38]. Recently, GANs and adversarial autoencoders (AAEs) have been coupled with adjoint topology optimization techniques for optimizing the diffractive dielectric gratings [38,115,136]. It has been previously shown that integrating the AAE network with a conventional adjoint topology optimization formalism can result in a speedup of approximately 4900 times for thermal emitter optimization as opposed to utilizing the conventional topology optimization method [120]. In [136], robustness against geometric erosion and dilation was achieved for the devices by conducting 30 adjoint-based topology optimization iterations on the 50 most effective GAN-generated devices. As demonstrated in [63], GANs and VAEs can achieve complex topologies by using image-based representations, and utilizing the AM can be an effective way of enhancing their performance. Additionally, generative models trained on physically informed loss or using the AM during training can also be improved by subsequent optimization-based refinement steps [137]. In [38], a network based on a conditional generative neural network and the AM was developed, which is capable of producing ensembles of highly efficient topology-optimized metasurfaces operating across a range of parameters.

In [37], a method for performing backpropagation in an ANN based on a photonic circuit was proposed, where the AM was used to derive the photonic of the backpropagation algorithm. In [138], a framework named DeepAdjoint was proposed. As a general, open-source, multi-objective “all-in-one” global photonics inverse design application framework, the method in ML optimization pipelines is made simpler and improved by hybridizing pre-trained deep generative networks with the AM. The framework employs GANs as an inverse-design network, which can predict device class, material properties (such as refractive index and Drude plasma frequency), and the nanoscale geometric structuring (such as planar topology and layer thickness) of metal–insulator–metal metasurfaces simultaneously.

In [139], the so-called neural-adjoint (NA) was developed. NA trains a neural network to approximate the input–output relationship and then, starting from different random locations, uses gradient descent towards locally optimal values with the AM. In [140], the NA inverse design method was studied for the inverse design of all-dielectric metasurfaces. The NA method is an effective method for predicting the high-dimensional total dielectric metasurface geometry required to generate the desired infrared absorption spectrum, even without any specialized knowledge in the field.

In [141], a recurrent neural AM based on NA was proposed. The results indicate that the proposed method is efficient in designing optical multilayer films for specific spectrum filters. In general, the hybridization of the AM and DL has been proven beneficial in handling photonic nonlinear phenomena. For example, the NLO phenomenon was more properly handled by techniques like Conditional GLOnets [38], which employed physics-driven gradients to iteratively enhance the nonlinear mapping between inputs and device layout, akin to the optimization process based on the AM.

5. Conclusions

The AM is now widely employed in the fields of photonic crystals, waveguides, resonators, filters, and plasma excitations in system design. Compared with other design methods, the AM can obtain the gradient of the objective function relative to all design DoFs through two full-field simulations, which makes it suitable for structures that simultaneously achieve various design goals and have a large parameter space. Except for topological optimization, the AM has been applied to many other types of inverse design tasks of photons, maybe with different acceleration techniques. It is also extended to solve nonlinear optimization problems. DL technology plays an important role in optical reverse design, which can significantly reduce simulation time; improve efficiency and accuracy; and is of great significance for the design, manufacturing, and application of optical devices. Different from the AM, DL can be an efficient solver of Maxwell’s equations as well as a nice optimizer. This makes DL techniques more widely employable in inverse design tasks. However, both the AM and DL have their own drawbacks. The AM is inherently a local optimization method since it heavily relies on gradient-based information. The optimization can suffer from becoming stuck in local minimum valleys or saddle points. The memory usage of the AM can also prevent it from very large problems. At the same time, DL may encounter the problems of local minima, difficulty in obtaining training datasets, the existence of non-unique solutions, and troubles in generalization. Fortunately, some of the difficulties can be solved or alleviated by combining DL with the AM.