Diffractive Neural Network Enabled Spectral Object Detection

Highlights

What are the main findings?

• We proposed an innovative DNN-SOD diffractive neural network architecture that leverages spectral characteristics and field-of-view segmentation to enable direct spectral feature reconstruction and target detection for infrared targets.
• The architecture achieved 84.27% on an infrared target dataset, demonstrating its feasibility for large-scale remote sensing tasks.

What is the implication of the main finding?

• This study presents a new paradigm of applying optical computing to spectral remote sensing target detection, overcoming the limitations of traditional optical computing methods that fail to fully exploit spectral properties of targets and handle large-scale data effectively.
• It provides a novel pathway for integrated sensing-computing information processing in future sky-based remote sensing, highlighting the potential of optical computing inference in real-world applications.

Abstract

This article introduces a diffractive neural network-enabled spectral object detection approach (DNN-SOD) to efficiently process massive sky-based multidimensional light field data. DNN-SOD combines the novel exploitation of target spectral features with the intrinsic parallelism of optical computing to process multidimensional information efficiently. DNN-SOD detects targets by segmenting the spectral data cube and processing it with the DNN. The DNN maps spectral intensity to the designated area of the detector, then reconstructs spectral curves, and differentiates targets by comparing them with reference spectral signatures. Classification results from individual sub-spectral data cubes are compiled in sequence, enabling accurate target detection. Simulation results indicate that the architecture achieved an accuracy of 91.56% on the MNIST multi-spectral dataset and 84.27% on the infrared target multi-spectral dataset, validating its feasibility for target detection. This architecture represents an innovative outcome at the intersection of remote sensing and optical computing, significantly advancing the dissemination and practical adoption of optical computing in the field.

1. Introduction

With the rapid advancement of sky-based remote sensing, multidimensional light field acquisition technologies, such as spectral [1,2,3] and polarization [4,5] imaging, have seen widespread application. However, exponential growth in data volume outpaces processing capabilities, creating a bottleneck for time-sensitive tasks such as infrared target detection [6] and extreme disaster prediction [7,8]. An approach to addressing the contradiction between the growth of data volume and the need for high-speed information processing is to introduce neuromorphic computing architectures. Among them, optical neural networks, as a branch of neuromorphic computing, have gained widespread attention and rapid development in recent years. It leverages the propagation of multidimensional light fields to perform computational tasks, offering advantages such as parallelism, high throughput, and low power consumption [9,10,11]. First proposed by Lin et al. in 2018, the diffractive neural network (DNN) manipulates the light field through multiple layers of stacked optical planes, achieving the mapping between the input and output light fields [12].

Using the wavelength-dependent and polarization-dependent response of optical elements in DNN, a multidimensional multiplexed DNN can be constructed. In 2023, D. Mengu et al. proposed a diffractive optical network-based multi-spectral imaging system that achieves multi-spectral snapshot imaging [13]. In 2024, Z Wang et al. proposed an opto-intelligence spectrometer using DNN that can achieve spectral reconstruction at the speed of light [14]. The aforementioned DNNs leverage the independent characteristics of different dimensions and channels of the light field. However, architectures based on these DNNs do not integrate and utilize the information contained in each channel of the multidimensional optical field for tasks such as inference and detection.

In this paper, we propose a DNN-enabled spectral object detection method that leverages the spectral characteristics of targets for detection through DNN-based reconstruction. The motivation for this architecture is leveraging optical computing to efficiently process massive multidimensional light-field data from sky-based remote sensing payloads, resolving the contradiction between rapidly evolving data acquiring capabilities and limited onboard data processing capacity.

DNN-SOD takes multi-spectral data of the scene as input and maps the different spectral channels of the input light field to corresponding regions of the complementary metal-oxide-semiconductor (CMOS) detector. The energy distribution curve of incident light at different wavelengths is obtained by calculating the intensity of the corresponding regions of the CMOS detector. Compared to traditional multi-wavelength DNNs, DNN-SOD considers the fact that different materials exhibit unique spectral characteristics. It utilizes DNN to reconstruct the spectral feature curves from the spectral data cube, and the detection of objects is achieved by comparing the reconstructed spectral curves with the existing spectrum database. The simulation results show that DNN-SOD can effectively utilize multi-spectral data to perform an object detection task.

The proposed architecture has the potential to bring about a breakthrough in spectral remote sensing, particularly in applications such as infrared target detection, environmental monitoring, and mineral exploration.

2. Materials and Methods

2.1. Architecture of DNN-SOD

Compared with traditional models deployed on GPUs or CPUs, DNN constructs neural networks in the optical domain using optical elements to perform data processing. As illustrated in Figure 1, traditional electronic algorithms (electronic neural networks) perform inference and training on GPUs or CPUs to achieve the mapping from data input to information output. In contrast, the DNN converts data from the digital domain to the optical domain through optoelectronic modulation devices and constructs a neural network in hardware via optical diffraction. This enables the processing of multidimensional data in the optical domain, performing computation through mathematical operations abstracted from light field propagation, and realizing the mapping from data input to information output.

DNN-SOD is a DNN-based architecture for spectral information processing aimed at object detection. DNN-SOD is based on the fact that different targets exhibit unique spectral signatures, theoretically enabling the detection of targets with different spectral features through the analysis of their spectral data cubes. As shown in Figure 2, it is a conceptual diagram of the DNN-SOD system. For remote sensing applications such as ship detection and unmanned aerial vehicle (UAV) detection, the target occupies a very small proportion of the high spatial resolution image. For a spectral data cube with high spatial resolution, we divide it into several subspectral data cubes and send them to the DNN in sequence. Different wavelength channels propagate forward through the DNN in parallel. Sub-regions corresponding to the number of wavelength channels are set on the detector plane, where the intensity distributions of different spectral bands in the input spectral data cube are mapped by the DNN to the corresponding sub-regions on the detector plane. By measuring the intensity distributions in each region, the spectral distribution curve of the input subspectral data cube can be obtained. Target classification is achieved by comparing the computed spectral curve with the ground-truth spectral characteristic curves of different target types. Target detection in high spatial resolution images is achieved by arranging the output classification results of each spectral data cube slice in the original sequence.

In this paper, we place the DNN-SOD module after photo-electroconversion to overcome the data volume bottleneck in sky-based remote sensing of multidimensional light fields. Future research could explore the integration of the DNN-SOD module with the image system of the satellite payload. In this case, DNN-SOD operates in spatially incoherent illumination, and segmentation of the spectral data cube in the optical domain can be achieved using micro-lens fiber integral field units (IFU) [15,16]. Recent research has already combined optical fibers with diffractive neural networks, which we believe is a highly promising research direction [17].

2.2. Forward Propagation Model of DNN-SOD

A fully connected neural network relies on numerical computation, utilizing matrix multiplication and non-linear activation functions. The output of each layer is obtained by multiplying the input vector by a weight matrix, adding a bias, and applying an activation function. In contrast, a diffractive neural network (DNN) computes based on optical diffraction propagation, where information passes through multiple diffractive layers, typically consisting of phase modulation surfaces. Each layer transforms the wavefront according to the diffraction principles, following the laws of wave optics. DNN-SOD utilizes multiple diffractive layers to map the spectral data cube of the scene in the input plane to the energy distribution of different wavelengths in the output plane. The forward propagation model of DNN-SOD is illustrated in Figure 3a, the optical component modulates the complex amplitude of the incident wave for a given wavelength $\lambda$, the output of the ith neuron in the lth plane can be expressed as

$$n_p^l(x,y,z,\lambda )=w_i^l(x,y,z,\lambda )·t_i^l(x_i,y_i,z_i,\lambda )·m_i^l(x,y,z,\lambda ),$$

(1)

where $m_i^l(x,y,z,\lambda )$ is the input complex amplitude of the neuron and equals to the linear superposition of the diffracted complex amplitudes from all neurons in the $l-1$th plane.

$$m_i^l(x,y,z,\lambda )=\sum _k{n}_{k,i}^{l-1}.$$

(2)

$t^l(x,y,z,\lambda )$ is complex amplitude transmission of lth diffractive element at wavelength ${\lambda }_m$.

$$t^l(x,y,z,\lambda )=a^l(x,y,z,\lambda )exp\left(j{\varphi }^l(x,y,z,\lambda )\right).$$

(3)

$w_i^l(x,y,z,{\lambda }_m)$ represents the propagation of the light wave between two adjacent planes, which can be described using the Rayleigh–Sommerfeld equation,

$$w_i^l(x,y,z,\lambda )={\displaystyle \frac{z-z_i}{r^2}}\left({\displaystyle \frac{1}{2\pi r}}+{\displaystyle \frac{1}{j\lambda }}\right)exp\left({\displaystyle \frac{j2\pi r}{\lambda }}\right),$$

(4)

where $r=\sqrt{{\left(x-x_i\right)}^2+{\left(y-y_i\right)}^2+{\left(z-z_i\right)}^2}$ and $j=\sqrt{-1}$. A detector at the output plane measures the intensity of the resulting optical field,

$$I_i=\sum _{k=1}^K\sigma \left({\lambda }_k\right)·I_i\left({\lambda }_k\right)=\sum _{i=1}^K\sigma \left({\lambda }_k\right)·{\left|m_i^{M+1}\right|}^2,$$

(5)

where K refers to the number of channels of the spectral datacube and $\sigma \left({\lambda }_k\right)$ is the spectral response function of the CMOS detector at wavelength ${\lambda }_k$.

As shown in Figure 3b, we choose multilevel diffractive optical elements (DOE) as the diffractive element of the DNN, which are phase-only elements that offer the advantage of minimal energy loss. Therefore, $a^l(x,y,z,\lambda )$ is 1, and complex amplitude transmission is $t^l(x,y,z,\lambda )=exp\left(j{\varphi }^l(x,y,z,\lambda )\right)$. The proposed architecture is also compatible with other types of diffractive elements. At a given wavelength, the phase introduced by the DOE can be expressed as

$$\varphi (x_i,y_i,\lambda )={\displaystyle \frac{2\pi }{\lambda }}\left(n_{\lambda }-1\right)h(x_i,y_i),$$

(6)

where $n_{\lambda }$ is the refractive index corresponding to wavelength $\lambda$, and $h(x_i,y_i)$ is the height profile of the DOE at position $(x_i,y_i)$. The height profile of DOE can be mathematically expressed as

$$h(x_i,y_i)=h_b-h_l(x_i,y_i),$$

(7)

where $h_b$ is the thickness of the substrate, and $h_l(x,y)$ is the DOE etching map at position $(x,y)$. To determine the etching depth at each position, we set the phase difference range corresponding to the central wavelength with a range of 0 to $2\pi$, and define the maximum etching depth as the etching depth corresponding to a phase difference of $2\pi$ at the central wavelength. The maximum etching depth can be expressed as

$$h_{lmax}={\displaystyle \frac{{\lambda }_c}{n_{{\lambda }_c-1}}},$$

(8)

where ${\lambda }_c$ is the central wavelength, and $n_{{\lambda }_c}$ is the refractive index of the medium corresponding to the central wavelength. An etching weight matrix $W(x,y)$ is defined here as a trainable parameter, with values constrained between 0 and 1, and the etching map of the DOE can be expressed as

$$h_l(x_i,y_i)=h_{l_{max}}W(x_i,y_i).$$

(9)

In the above forward propagation model, the light in each wavelength channel is spatially coherent, meaning there is a stable phase relationship between any two points on a given plane. The proposed architecture can also be extended to spatially incoherent light. Typical approaches for modeling incoherent light include the random phase superposition method, mode decomposition, and the convolution-based method. The random phase superposition method approximates incoherence for a single input example by averaging the output intensity patterns from numerous coherent input fields with random phase distributions [18,19]. The mode decomposition method decomposes an incoherent scene into multiple coherent point sources, and the propagation result of the optical field is equal to the linear superposition of the propagation of each point source. The core idea of the convolution-based method is that, under the assumption of a linear and shift-invariant system, the propagation result of the optical field can be regarded as the convolution of the scene with the system’s point spread function (PSF). As shown in the

Table 1. Characteristics and computational complexity of incoherent light models.

Incoherent Light Modeling Method	Computational Complexity	Characteristics
Random phase superposition	$O(M·N^{2}logN)$	High accuracy with rapidly growing cost
Mode decomposition	$O(K·N^{2}logN)$	Suitable for sparse scenes; parallelizable
Convolution-based method	$O(N^{2}logN)$	Fastest approach, limited by linearity and shift invariance

Notes: $O(·)$ denotes computational complexity; M is the number of random phase realizations; K is the number of point sources in mode decomposition; $N\times N$ is the input field size.

, the three modeling methods are compared in terms of computational complexity and applicable conditions.

Spatially incoherent light modeling can enhance the application prospects of the proposed framework in passive scenarios such as remote sensing and security [20]. However, compared to coherent light illumination, the computational burden under incoherent light conditions will increase [21], but it can be alleviated through parallel computing and other methods.

2.3. Dataset Generation and Training of DNN-SOD

DNN-SOD requires multi-spectral data of the scene as training samples. In this paper, we validate the principles and feasibility of DNN-SOD by constructing a multi-spectral dataset based on the MNIST (Modified National Institute of Standards and Technology) handwritten number dataset. The process of generating datasets is shown in Figure 4. First, the data in the MNIST dataset is resized and binarized. For a particular class of handwritten digits, we assume that all non-zero pixels have the same spectral feature. The spectral characteristics are randomly generated with values between 0 and 1 for each wavelength. The binarized image is multiplied by the corresponding value in the spectral curve for each wavelength, yielding the intensity distribution for that wavelength. A three-dimensional spectral data cube is obtained by stacking the intensity distributions for each wavelength along the spectral dimension. A total of 60,000 spectral data cubes were generated, each with dimensions of $M\times N\times C$, where $M\times N$ represents the spatial dimensions and C is the number of channels in the spectral data cube. The dataset was divided into training, validation, and test sets in a ratio of 8:1:1.

Similarly to electronic neural networks, the weights of DNN-SOD also require training. The difference is that in DNN-SOD, the weights refer to optical parameters such as the phase distribution of DOE. The forward propagation model of DNN-SOD is described by the multidimensional optical field propagation, whereas the backpropagation process employs a gradient descent algorithm similar to that used in electronic neural networks. As shown in Figure 5, multi-spectral data is fed into the forward propagation model, and the output results are compared with the ground truth to compute the loss. The gradient descent algorithm is used for backpropagation to optimize the weight parameters. These parameters will guide the subsequent fabrication of the DOE.

We define the ground-truth light intensity $I_{\mathrm{gt}}$ on the detector plane for each class of handwritten numbers. The normalized spectral intensity of handwritten digit “5” is shown in the Figure 6a. As shown in Figure 6b, we divide the detector surface into nine sub-regions corresponding to nine wavelength channels. In the generated dataset, since each non-zero pixel of a specific handwritten digit has the same spectral curve in the spatial domain, the ground truth light intensity for each detector sub-region on the surface should be proportional to the intensity at the corresponding wavelength in the spectral feature curve. Therefore, the true light intensity value for a sub-region is set to the intensity value corresponding to the wavelength in the spectral curve, while the light intensity in the remaining areas outside the sub-region is set to 0.

To ensure both the distinguishability of the detector’s output and the accuracy of the computed spectral curve, we define two types of mean squared error (MSE) loss, which can be expressed as

$$L_{\mathrm{intensity}}=\mathrm{MSE}(I_{\mathrm{measurement}},I_{\mathrm{gt}}),$$

(10)

$$L_{\mathrm{curve}}=\mathrm{MSE}(f_{\mathrm{measurement}},f_{\mathrm{gt}}),$$

(11)

where $I_{\mathrm{measurement}}$, $I_{\mathrm{gt}}$, $f_{\mathrm{measurement}}$, and $f_{\mathrm{gt}}$ represent the detector’s measured intensity and the ground truth intensity, as well as the calculated and true values of the spectral curve, respectively. During the backpropagation, the Adam optimizer is used to update the etching weight matrix of the multi-layer diffractive plane in DNN-SOD with a linear combination of two loss functions,

$$L=\alpha ·L_{\mathrm{intensity}}+\beta ·L_{\mathrm{curve}},$$

(12)

where $\alpha$ and $\beta$ are the weighting factors.

3. Results

3.1. Preliminary Validation on the Multi-Spectrum MNIST Dataset

In this article, we validate the proposed architecture in the near-infrared band, selecting a central wavelength of 1550 nm, with a spectral resolution of 20 nm, and defining a total of 9 spectral channels. The spectral bands in the multi-spectral data cube are 1470 nm, 1490 nm, 1510 nm, 1530 nm, 1550 nm, 1570 nm, 1590 nm, 1610 nm, and 1630 nm, respectively. The corresponding refractive indices of the medium are 1.4450, 1.4447, 1.4445, 1.4443, 1.4440, 1.4438, 1.4435, 1.4433, and 1.4431, respectively. The architecture proposed in this article is also applicable to other electromagnetic spectral bands and can be flexibly adjusted in terms of spectral resolution and the number of spectral channels according to the specific application requirements. For generating the multi-spectral dataset, the images in the MNIST dataset are first padded from 28 × 28 to 400 × 400, and the multi-spectral dataset is constructed using the method described in the previous section. As shown in Figure 4, the spectral features for each handwritten digit class are generated. The DNN in this article consists of three layers of diffractive elements, with each neuron having a size of 8 μm × 8 μm and each layer containing $400\times 400$ neurons. According to the half-cone diffraction angle formula, achieving full connectivity in the DNN requires that the distance between adjacent layers meet certain conditions. Nine sub-regions are defined in the detector plane, with each sub-region corresponding to a specific wavelength.

We employ Adam to train the diffractive element parameters, with an initial learning rate of 0.01, which decays to 80% of its previous value every 10 epochs. Based on empirical experience and a review of the relevant literature, we set the two constants $\alpha$ and $\beta$ at 100 and 1, respectively. The model was implemented using Python version 3.8.0 and the PyTorch framework (version 2.1). We use a desktop computer equipped with an NVIDIA GeForce RTX 3080 GPU and an Intel(R) Core(TM) i9-12900K CPU to train the model.

The test results are shown in Figure 7. Figure 7a presents the spectral data of selected handwritten digit inputs. The intensity distribution on the detector is measured after the handwritten digits’ spectral data cubes pass through the DNN-SOD architecture. As shown in Figure 7b, different spectral components are directed to designated regions of the detector through the dispersion control of DNN-SOD. By calculating the total intensity in each sub-region, the spectral characteristic distribution curve shown in Figure 7c can be obtained. By comparing the computed spectral characteristic curves with the ground truth spectral characteristic curves of different handwritten digits, a similarity characteristic vector is obtained, as illustrated in Figure 7d. This feature vector is then processed through a Softmax operation to determine the classification result of the handwritten digit. The test set contains 5000 spectral data cubes, with 4578 correctly identified, resulting in an accuracy of 91.56%.

The DOEs are manufactured on a 1 mm thick quartz substrate using photolithography. During the training process, the etching weights are treated as trainable variables, and the etching depth at each pixel of the DOE is obtained by multiplying the etching weights by the maximum etching depth. As shown in Figure 8, the etching weights of the trained three-layer DOE are presented, where the maximum etching depth is 3.39 μm and the minimum etching depth is 0. The diffraction efficiency is a key performance metric of DOEs. In this work, phase-only DOEs are employed, which theoretically can achieve a diffraction efficiency of 1. However, in practical nanofabrication, the phase needs to be quantized, which reduces the diffraction efficiency. For an Q-bit quantized phase DOE, the diffraction efficiency as a function of the quantization level can be expressed as

$$\eta ={\left({\displaystyle \frac{sin(\pi /2^Q)}{\pi /2^Q}}\right)}^2,$$

(13)

where Q is the number of phase quantization bits. On the basis of the above expression, the diffraction efficiencies of DOEs with different bit quantizations are listed in the

Table 2. Diffraction efficiency of DOEs with different phase quantization bits.

Bit Number	Quantization Levels	Diffraction Efficiency $\mathit{\eta }$
1	2	40.5%
2	4	81.0%
3	8	95.0%
4	16	98.7%

. In practical fabrication, considering both the diffraction efficiency and the fabrication complexity and cost, a 3-bit DOE is typically chosen.

3.2. Validation on Dataset with Infrared Targets

After validating the theoretical feasibility of DNN-SOD using multi-spectral data based on handwritten digits, we further verified the effectiveness of the DNN-SOD architecture on data with infrared targets. We evaluated the effectiveness of DNN-SOD on two distinct infrared datasets. The first dataset is constructed from existing infrared data [22], while the second is a multi-spectral infrared target dataset acquired using a near-infrared camera equipped with bandpass filters. These two datasets provide complementary validation scenarios that allow us to assess the robustness of the proposed method under various infrared imaging conditions. We hereafter refer to these two datasets as Dataset I and Dataset II for clarity.

For Dataset I, we constructed the data based on existing infrared imagery. The targets and backgrounds were first separated, and two spectral curves were defined to represent the baseline spectral characteristics of the targets and background, respectively. Before assigning the spectral curves to each spatial pixel, we added a random vector of the same dimension, with values ranging from 0 to 0.08, in order to better simulate a multi-spectral data cube. For Dataset II, we employed an InGaAs infrared camera in combination with bandpass filters to capture real multi-spectral infrared target data. As illustrated in Figure 9a,c, we visualize the spectral curves of randomly selected pixels from targets and backgrounds in the two datasets.

As shown in Figure 9b,d, for both Dataset I and Dataset II, we divide the spectral data cube of the scene along the spatial dimension into multiple sub-spectral data cubes. The training, testing, and validation sets are divided according to a ratio of 6:2:2. The spatial dimension of the hyperspectral datacube is 300 × 300, and after slicing, the size of each sub-spectrum datacube is 25 × 25. As shown in

Table 3. Key parameters of the Dataset I and Dataset II.

Dataset	Number of Spectrum Channel	Center Wavelength(nm)	Number of Sub Spectrum Cubes
Dataset I	9	800, 900, 1000, 1100, 1200, 1300, 1400, 1500, 1600	31,104
Dataset II	5	1000, 1100, 1200, 1300, 1400	20,736

, the number of channels, central wavelengths, data dimensions, and other relevant information for the two datasets are summarized. We train the parameters of DNN-SOD separately using the two datasets.

We set sub-regions on the detector plane corresponding to the number of channels of the dataset., each representing a different wavelength channel. The ground truth intensity for each sub-region and spectral distribution curves for each sub-spectral cube are calculated on the basis of the ratio of the sum of the grayscale values of each channel. Each subspectral data cube undergoes spectral analysis according to the previously described process.

During the training on Dataset I and Dataset II, we trained the models for 30 epochs with batch sizes of 16 and 8, respectively, and the corresponding convergence curves are shown in Figure 10a. The key hyperparameters of DNN-SOD are the number of DOE neurons, the number of DOE layers, and the inter-layer spacing. In our experiments, we made the underlying assumption that the performance variations in DNN-SOD are primarily caused by the parameter under investigation. Therefore, we adopted a controlled experimental design in which only one variable was changed at a time while all other factors were kept fixed. We use accuracy and false alarm rate as evaluation metrics to investigate the impact of these parameters on DNN-SOD, and the results are shown in Figure 10b–d. For both Dataset 1 and Dataset 2, increasing the number of neurons and layers in DNN-SOD initially improves the performance metrics, which then stabilize. This is because, as the number of neurons and layers increases, the performance of DNN-SOD approaches its theoretical limit, which is consistent with its linear mathematical model. As the interlayer spacing increases, the performance of DNN-SOD exhibits an initial improvement followed by a decrease. The gradual improvement can be attributed to the fact that, as the inter-layer spacing increases, the layers tend to form a fully connected structure, enabling more effective backward propagation of features. However, when the spacing continues to increase, the incidence angles at which neurons receive light from the previous layer become smaller, which in turn degrades the overall system performance. Considering fabrication cost and inter-layer alignment accuracy, the DNN-SOD for infrared target detection in this study consists of three DOE layers, with each layer containing 500 × 500 neurons, and inter-layer distance is 0.1 m.

The spectral reconstruction curves processed by DNN-SOD are shown in the Figure 11. Figure 11a presents the reconstruction curves of sub-spectrum data cubes (i) and (ii) from Dataset I, while Figure 11b shows the reconstruction curves of sub-spectrum data cubes (iii) and (iv) from Dataset II. It can be observed from the figure that the energy distributions of the spectral reconstruction curves, both with and without targets, exhibit distinctive characteristics. This validates the feasibility of DNN-SOD for target detection based on spectral features.

Under the conditions of a three-layer DOE, with each layer containing $500\times 500$ neurons and an inter-layer spacing of 0.1 m, the confusion matrices corresponding to Dataset I and Dataset II are shown in Figure 12a. In this case, the accuracies obtained for Dataset I and Dataset II are 84.27% and 79.79%, respectively. The false alarm rates are 15.25% and 20.43% for Dataset I and Dataset II, respectively. Finally, the results of all subspectral data cubes are arranged according to their original positions in the original spectral data cube, thereby achieving target detection. As shown in Figure 12b, the subspectral data cube (i) in Figure 9 is classified as a target, and the subspectral data cube (ii) is classified as a background. As shown in Figure 12c, the visualized detection results for both Dataset I and Dataset II are derived from the outcome corresponding to Figure 12b. It is worth noting that some backgrounds have still been misclassified as targets. In future work, the detection accuracy can be further improved and the false alarm rate reduced by integrating a back-end electrical neural network.

The above results verify the feasibility of using the DNN-SOD architecture to detect targets such as drones. The combination of spectral features and optical computing inference is expected to become an effective solution for target detection in the future. As shown in

Table 4. Comparison of DNN-SOD with existing DNN processors.

DNN Processor	Number of Channel	Task	Dataset	Accuracy (%)	Reference
DNN-SOD	Multi	Detection	UAV Dataset	84.27	Ours
DNN	Single	Classification	MNIST	91.75	[12]
PCONN	Single	Classification	ISDD	94.69	[20]
ICONN	Single	Classification	ISDD	93.25	[23]

, we selected three baseline methods for comparison with DNN-SOD. Among them, DNN is the pioneering work in this field, which first proposed using diffractive optics for information processing [12]. Both PCONN [20] and ICONN [23] try to employ DNN for target detection through classification-based discrimination, which is similar in approach to the methodology in this article. We did not include comparisons with conventional electronic neural networks, as our focus is on exploring the potential of optical neural networks for spectral object detection. Due to fundamental differences in processing paradigms and the early stage of optical neural network development, direct comparisons with electronic networks are beyond the scope of this study. Although DNN-SOD shows lower accuracy compared with these three methods, it distinguishes itself by leveraging multi-band information for computational inference and target detection, in contrast to traditional DNNs that operate under single-band conditions. The relatively lower accuracy of DNN-SOD can be attributed to two main reasons. First, DNN-SOD is designed to handle more complex task scenarios, where the similarity between the background and the target often leads to a higher false alarm rate. Second, when segmenting the spectral data cube, DNN-SOD may split the target into several parts. These two factors place higher demands on the overall architecture. In future work, we plan to enhance the detection performance of DNN-SOD by introducing optical non-linearity and incorporating lightweight electronic neural networks in the backend.

4. Discussion

4.1. Research Implications

In this paper, we propose the DNN-SOD architecture for target detection and validate its feasibility on multi-spectral datasets. The simulation results demonstrate that DNN-SOD has the capability to process multi-spectral data and identify the scene using spectral features. Target detection within the field of view is achieved by sequentially classifying the subspectral data cubes.

To the best of our knowledge, this is the first study to combine optical computing inference with remote sensing spectral data processing, marking a significant advancement in both the fields of optical computing and remote sensing technology. In the optical community, compared to conventional DNN models, which typically focus on spatial information alone, the DNN-SOD utilizes the full spectrum of available data, ensuring that subtle spectral differences are fully exploited to improve detection performance. In the remote sensing community, the introduction of optical computing inference provides a new solution for real-time on-orbit processing of massive remote sensing data. This capability is particularly important for sky-based payloads, where large volumes of data are continuously generated, and timely decision-making is critical. By incorporating optical computing into the target detection pipeline, the DNN-SOD architecture addresses the challenge of processing and analyzing data in real time, making it a valuable tool for future remote sensing missions.

In addition, the proposed architecture offers great flexibility for spectral extension, allowing the selection of wavelength, number of spectral bands, and spectral resolution according to practical requirements. Moreover, current nanofabrication technologies are fully capable of supporting the fabrication of the DOE in DNN-SOD. Since DOEs have already been widely applied to improve image resolution and other fields, their spatial adaptability does not face insurmountable bottlenecks. Therefore, the proposed approach has significant engineering value for practical aerospace remote sensing payloads.

4.2. Limitation and Future Perspectives

Although this study has successfully developed a DNN-based architecture for spectrum object detection and achieved accuracy up to 84.27%, there are several limitations that should be addressed in future research.

Firstly, the two infrared target datasets used in this work have nine and five spectral channels, respectively, which is somewhat limited for fully validating the scalability of DNN-SOD to a larger number of spectral channels. In future work, we plan to create a more comprehensive multi-spectral dataset This dataset will cover a wider variety of environmental conditions, including different geographic locations, seasons, and weather patterns, as well as a broader range of target types. By incorporating more diverse scene data, we aim to enhance the model’s robustness and adaptability, enabling it to handle a wider array of real-world conditions.

Secondly, the functionality of the diffraction neural network in this paper is still quite basic compared to complex electrical neural networks [24,25]. It faces challenges in handling more complex situations, such as when a target is divided into multiple segments during the segmentation process, which may lead to an increased false alarm rate. The main reason for this limitation is the relatively shallow depth and small scale of the current optical neural network. In upcoming research, metasurfaces can be used to construct diffractive neural networks. Metasurfaces offer the potential to dynamically manipulate light fields, further improving the inference performance of the optical neural network. [26]. Moreover, DNN-SOD can also be integrated with an in-sensor back-end [27,28,29] and near-sensor computing [30] to introduce nonlinearity into the architecture. This integrated approach will be capable of tackling complex sensing tasks in a wide range of applications, including infrared target detection, environmental monitoring, and mineral exploration. Moreover, optical computing-based multi-spectral information processing methods can also be integrated with state-of-the-art hyperspectral algorithms to perform a range of tasks, such as super-resolution imaging [31,32,33] and image fusion [34].

In future work, we aim to integrate the DNN-SOD prior to the photonic-to-electronic conversion process of the detector, using IFU for field-of-view segmentation. By leveraging the multidimensional optical fields in natural scenes for computational inference and integrating back-end in-sensor computing and near-sensor computing modules, we aim to create a novel end-to-end optical field multidimensional perception architecture to perform computation and inference tasks tailored to specific application scenarios.

5. Conclusions

In this paper, we propose a DNN-based target detection architecture, where target detection is achieved by reconstructing the spectral characteristics of the target in the optical domain. The performance of DNN-SOD is validated using two infrared multi-spectral datasets containing UAV targets. Experimental results demonstrate that DNN possesses the potential to process spectral data and accomplish target detection tasks, laying the foundation for the development of novel sky-based remote sensing and information processing architectures in the future.