Macroscopic Fourier Ptychographic Imaging Based on Deep Learning

Abstract

Fourier Ptychography (FP) is a powerful computational imaging technique that enables high-resolution, wide-field imaging by synthesizing apertures and leveraging coherent diffraction. However, the application of FP in long-distance imaging has been limited due to challenges such as noise and optical aberrations. This study introduces deep learning methods following macroscopic FP to further enhance image quality. Specifically, we employ super-resolution convolutional neural networks and very deep super-resolution, incorporating residual learning and residual neural network architectures to optimize network performance. These techniques significantly improve the resolution and clarity of FP images. Experiments with real-world film samples demonstrate the effectiveness of the proposed methods in practical applications. This research highlights the potential of deep learning to advance computational imaging techniques like FP, paving the way for improved long-distance imaging capabilities.

1. Introduction

Optical imaging is the primary method by which individuals acquire information. Traditional optical imaging systems face limitations due to optical diffraction, detector pixel sizes, lens aberrations, and disordered imaging media. These factors significantly impact spatial resolution and spatial bandwidth products in imaging and can even render direct imaging impossible. Fourier ptychographic imaging (FP) is a super-resolution imaging technique applicable to both macroscopic and microscopic scales. By synthesizing apertures and employing coherent diffraction imaging, it is possible to achieve high-resolution and wide-field imaging simultaneously. Zheng et al. provide a comprehensive review of various Fourier ptychographic microscopy (FPM) configurations and their applications in imaging different samples, such as biological specimen slices and nanoscale integrated circuits, which have led to numerous significant advancements [1,2,3,4,5].

Compared to FPM, the imaging of distance objects using FP is relatively limited. In 2014, Zheng et al. reported the first instance of 3D refocusing and macroscopic imaging at a distance of up to 0.7 m using aperture-scanning Fourier ptychography [6]. Holloway et al. enhanced the spatial resolution of long-distance images by a factor of ten or more by employing camera arrays coupled with coherent illumination [7]. They also demonstrated the first working prototype of FP in a reflection imaging geometry capable of imaging optically rough objects [8]. Zhang et al. introduced an FP method utilizing sparse representation to effectively suppress noise and enhance the performance of reconstructed high-resolution images [9]. Yang et al. examined the impact of laser coherence in various modes and the surface properties of different materials for diffused targets [10]. In FP setups, measurements are often affected by various degradations, including Gaussian noise, Poisson noise, speckle noise, and pupil location errors, which can significantly impair reconstruction quality. To effectively tackle these degradations, Bian et al. proposed a novel FP reconstruction method based on a gradient descent optimization framework, while Cui et al. introduced a pose correction scheme utilizing camera calibration and homography transform approaches [11,12]. Yang et al. proposed a partition reconstruction algorithm, and they were able to resolve the 40 µm line width of the resolution object and obtain a spatial resolution gain of 4× with a working distance of 2 m [13]. Single shot [14], the farthest imaging distance and largest synthetic aperture [15], and moving-target FP imaging [16] have also been demonstrated.

With the continuous advancement of computer performance, deep learning technology has gradually gained prominence and achieved significant success in practical applications. Researchers have applied deep learning methods to FPM. Yu et al. proposed a multi-convolutional network feature fusion neural network framework to realize FPM image reconstruction [17]. Jiang et al. modeled the FPM forward imaging process using a convolutional neural network (CNN) to recover complex objects [18]. They utilized TensorFlow to establish the network and conduct the optimization process, analyzing the algorithm’s performance against standard FPM. Recently, Wu et al. introduced a reduced-angle ptychotomographic method that employs deep learning for 3D nanoscale imaging [19]. However, the integration of FP with deep learning to improve the super-resolution imaging of long-distance targets has yet to be reported.

In this paper, we introduce a deep learning method to further improve the imaging quality of FP for distant objects. The training data are derived from Technical Committee 12 of the International Association of Pattern Recognition (IAPR TC-12), and a super-resolution convolutional neural network is employed to process the macroscopic results of FP [20,21]. In the experiment, the target samples were not limited to the USAF resolution chart; instead, film samples that are more relevant to real-life scenarios were utilized. A no-reference evaluation standard was implemented to analyze the image quality due to the pixel expansion associated with FP.

Our results demonstrate that the imaging quality of macroscopic targets can be improved by integrating deep learning networks following FP reconstruction. This paper provides valuable insights for other programs utilizing computational imaging methods to reconstruct macroscopic objects. By employing deep learning networks to post-process the outcomes of existing computational imaging techniques, it is expected to further improve their reconstruction quality.

2. Theory of Fourier Ptychographic Imaging

2.1. Imaging Model of FP

The schematic diagram of macroscopic FP is shown in Figure 1. The incoming plane wave is transformed into a converging spherical wave as the illumination light field $\mu (x,y)$ by the lens. Let the transmission function of the thin target object be denoted as $O(x,y)$, and the light field that passes through the target is represented as $\Psi \left(x,y\right)=\mu (x,y)O(x,y)$. When the focus of the converging spherical wave is adjusted to the aperture plane (defined as $(x^{\prime },y^{\prime })$) of the camera, the light field from the object converges and satisfies the conditions of Fraunhofer diffraction. Its Fourier transform is presented on this plane, as depicted below:

$${\Psi }^{\prime }\left(x^{\prime },y^{\prime }\right)={\displaystyle \frac{\exp \left(jkz\right)\exp \left(\frac{jk\left(x^{\prime 2}+y^{\prime 2}\right)}{2z}\right)}{j\lambda z}}{F}_{{\displaystyle \frac{1}{\lambda z}}}\left[\Psi \left(x,y\right)\right]$$

(1)

where $k=2\pi /\lambda$ is the wavenumber, and $F_{1/\lambda z}$ denotes a two-dimensional Fourier transform scaled by $1/\lambda z$. Let the aperture function of the lens be $A(x^{\prime }-c_{x^{\prime }},y^{\prime }-c_{y^{\prime }})$, where $(c_{x^{\prime }},c_{y^{\prime }})$ is the center position of the camera aperture. The aperture function is:

$$A\left(\gamma ,\nu \right)=\left\{\begin{array}{c}1, {\gamma }^2+{\nu }^2\le d^2\\ 0, otherwise\end{array}\right.$$

(2)

where $d$ represents the aperture diameter. Given the constraints of the camera aperture, the image intensity captured by the camera is:

$$I\left(\mathrm{x},\mathrm{y},c_{x^{\prime }},c_{y^{\prime }}\right)\propto {\left|F\left[{\Psi }^{\prime }\left(x^{\prime },y^{\prime }\right)\right]A\left(x^{\prime }-c_{x^{\prime }},y^{\prime }-c_{y^{\prime }}\right)\right|}^2$$

(3)

When a single image is captured, the resolution of the camera is determined by the Rayleigh criterion, expressed as $\Delta x=0.61\lambda /NA$. ($NA=(d/2)/f$ is the numerical aperture and $f$ denoting the focal length).

2.2. Imaging Process of FP

Macro-FP acquires a series of low-resolution images using aperture scanning and restores the synthesized high-resolution image through phase iteration algorithms. Please refer to the Supplementary Materials Section S1 for a depiction of the FP algorithm process. The flowchart illustrating the reconstruction process of macroscopic FP is shown in Figure 2.

The camera captures image intensity by scanning the aperture both horizontally and vertically in the Fourier domain, resulting in a series of intensity values denoted as $I_i$, where $i$ ranges from $1$ to $N$, corresponding to the $i\mathrm{t}\mathrm{h}$ scan when the aperture center is positioned at $(c_{x^{\prime }},c_{y^{\prime }})$. The mean value $\overline{I}$ is calculated through linear superposition and averaging of the $I_i$ values. The Fourier transform of $I_i$ yields the initial information in the frequency domain, represented as $H\left(K\right)$. The spatial spectrum intercepted by the aperture is given by $H\left(K\right)\ast A(x^{\prime }-c_{x^{\prime }},y^{\prime }-c_{y^{\prime }})$. The light field captured by the camera, denoted as ${\varphi }_i^k$, is the inverse Fourier transform of the intercepted spatial spectrum. After processing the image information to update the complex amplitude ${\varphi }_i^k$ in the spatial domain, the corresponding region of $H\left(K\right)$ is updated following the Fourier transform of ${\varphi }_i^{k+1}$. Finally, based on the iteration parameters, the updated frequency domain information, $\overline{H\left(k\right)}$, undergoes an inverse Fourier transform to produce a high-resolution image.

2.3. Simulation Results

In the simulation experiment, a resolution target image with dimensions of 512 × 512 pixels is utilized, featuring line pairs ranging from 20 pixels to 1 pixel in width. The wavelength of the coherent illumination source is $\lambda =632.8 \mathrm{n}\mathrm{m}$, the focal length of the focusing lens is 80 mm, the aperture size is 10 mm, and the diffraction distance is 1.5 m. Under these conditions, the resolution limit of the imaging system with a single aperture corresponds precisely to line pairs with a width of 9 pixels in the resolution target image, as illustrated in Figure 3. During FP processing, a 7 × 7 sampling matrix is established in the frequency domain with an overlap rate of 61%. The synthesized aperture size is 3.34 times larger, which corresponds to an improvement in resolution by 3.34 times, and line pairs with a width of 3 pixels in the resolution target image can be well resolved. Following FP processing, the pixel numbers of the image are expanded, and the recovered image is enlarged from 512 × 512 pixels to 652 × 652 pixels.

3. Deep Learning Network Architectures

In this paper, we apply four types of deep learning network architectures to further enhance the image quality of FP. In 2015, Dong et al. proposed a deep learning method called the SRCNN for single image super-resolution (SR) [22]. The SRCNN architecture is quite simple, consisting of only three convolutional layers. This method directly learns an end-to-end mapping between low-resolution and high-resolution images. Bicubic interpolation to subsample and upsample the high-resolution image is used to obtain the processed low-resolution image (pre-processing). The pre-processed low-resolution images, along with their corresponding original high-resolution images, are then used to construct training pairs. The process of constructing these training pairs and the SRCNN architecture is illustrated in Figure 4.

Jiwon et al. presented a highly accurate single-image super-resolution (SR) method utilizing a very deep super-resolution (VDSR) inspired by VGG-Net, which is commonly used for ImageNet classification [23,24]. A significant improvement in accuracy was achieved by increasing the depth of the VDSR network, resulting in a final model that employed 20 weight layers. An effective yet straightforward training procedure was implemented for VDSR to learn residuals, utilizing extremely high learning rates—104 times higher than those used in the SRCNN. The residual learning method takes the difference between the pre-processed low-resolution image. The original high-resolution image is the residual image and combines with the low-resolution image as the training pair. The process of constructing training pairs and the VDSR architecture is illustrated in Figure 5.

While maintaining the original network architecture, the SRCNN-residual network incorporates a residual learning strategy into the SRCNN framework to mitigate the learning gradient. The structure of the SRCNN-residual network is illustrated in Figure 6.

Deeper neural networks are more challenging to train. He et al. presented a residual learning framework to ease the training of networks that are substantially deeper than the previous networks [25]. They explicitly reformulate the layers as learning residual functions concerning the layer inputs, instead of learning unreferenced functions. The residual learning framework is shown in Figure 7.

The problem of obtaining $H\left(x\right)$ can be reframed as solving the residual mapping function of the network, as $F\left(x\right)=H\left(x\right)-x$. Assuming that the current depth of the network achieves the lowest error rate, we just need to set $F\left(x\right)=0$ to ensure that the state of the next layer still remains optimal as the network depth continues to increase. The residual learning framework is integrated into the VDSR network to mitigate issues of gradient explosion and gradient vanishing, thereby further optimizing the network architecture, as illustrated in Figure 8.

4. Imaging Quality Evaluation Criteria

In the FP simulation, the pixels of the image expanded from a single low-resolution image of 512 × 512 pixels to 652 × 652 pixels. This digital resolution expansion is based on the aperture overlap ratio and the size of the grid. In actual FP experiments, a large exposure dynamic range is employed during Fourier domain sampling to ensure the quality of the restoration. All sampled images in the frequency domain are linearly superimposed and averaged to obtain the initial estimate of the image. Consequently, there is a significant difference in the number of pixels between the restored image and the single low-resolution image at the center frequency. This discrepancy results in the absence of a standard reference image, making conventional image quality evaluation methods, such as peak signal-to-noise ratio (PSNR) and structural similarity index measurement (SSIM), unsuitable for quantitatively assessing the restored results. Instead, a no-reference image quality evaluation method is utilized. We employ three no-reference evaluation criteria—standard deviation, average gradient, and image entropy—to assess image quality. Please refer to the Supplementary Materials Section S2 for a description of the no-reference evaluation criteria.

5. Macroscopic FP Experiment Based on Deep Learning

In previous macroscopic FP experiments, resolution targets were the most commonly used objects. However, these targets differ significantly from everyday objects, as their grayscale distribution varies dramatically across space. Usually, researchers have directly employed deep learning frameworks instead of the FP algorithm to obtain high-resolution images [26]. To the best of our knowledge, the integration of deep learning networks with macroscopic FP to further enhance image quality has not yet been reported.

To facilitate a more effective comparison of the performance of four deep learning frameworks, the same network training hyperparameters are set. The scaling factor is set to 4, the input size is 41 × 41, the number of iterations is 100, the initial learning rate is 0.1, and the decay factor is multiplied by 0.1 every 10 iterations. The deep learning network architecture is trained using 800 images from the publicly available IAPR TC-12 image dataset.

In the simulation experiment, the FP high-resolution image was generated with an overlap rate of 61%, a frequency domain sampling grid of 17 × 17, and 70 iterations. The FP image was subsequently post-processed using four different network structures, as illustrated in Figure 9. In this figure, “Center input” is the center image from 17 × 17 sampling low-resolution images, and “Recov” is the recovered FP image.

Based on the simulation results, all four network architectures demonstrated improvements in the three evaluation criteria for the FP image. The VDSR-residual network structure exhibited the best performance, increasing the mean gradient by as much as two times.

After successfully verifying the performance of the deep learning network architectures in the simulations, a macroscopic FP is utilized. To account for the diversity of optical samples, film samples are employed as the target objects, as they are more relevant to real-life scenarios. The experimental optical setup is illustrated in Figure 10.

The LP He–Ne laser operating at 632.8 nm (Thorlabs, Newton, NJ, USA) is spatially filtered and expanded to produce a clean Gaussian beam. A polarizer is positioned before the spatial light filter to regulate the light intensity. A focusing lens with a focal length of 200 mm converges the Gaussian beam to illuminate the target object, and the spatial spectrum of the object’s domain is formed at the aperture of an imaging system. This imaging system comprises a CCD camera (acA2040-90μm, with a pixel size of 3.75 μm, Basler, Ahrensburg, Germany), an 80 mm fixed-focus lens, and an adjustable aperture (DaHeng, BeiJing, China), all mounted on a precision translation platform (Xianghe Nate, NT101TA75M, LangFang, China). The diameter of the adjustable aperture is set to 4 mm. The overlap rate of the scanning aperture is 87.5%, and a frequency domain sampling grid of 7 × 7 is employed to capture low-resolution images. After 200 iterations, a high-resolution FP synthetic aperture image is generated.

Using the USAF resolution chart (R1DS1N, Thorlabs, Newton, NJ, USA) as the target, positioned 650 mm away from the camera, the resolution of the center observed image (Group 2, Element 3 in Figure 11b) is limited to 99.21 μm. By acquiring a 7 × 7 grid of images with 87.5% overlap between neighboring images, an effective synthetic aperture ratio (SAR) of 1.75 is achieved. As illustrated in Figure 11c, the spatial resolution of the FP reconstruction has improved to 62.75 μm (Group 3, Element 1), representing a 1.58× enhancement in resolution.

When the deep learning framework further processes the reconstructed image, as shown in Figure 12, it cannot illustrate the advantages of deep learning networks because the training data comprise images of people, animals, plants, and landscapes.

From the experimental results, the improvement in image resolution is not significant, and some degradation can even be observed. This may be due to the fact that most of the images used as training data in the IAPR TC-12 public dataset are undistorted images of humans and landscapes, which differ greatly from the resolution chart. We utilize film samples as macroscopic objects, as they are more representative of everyday scenes than the resolution chart. The recovered results are further optimized for image quality through a deep learning network framework. The output results are analyzed locally, and the experimental findings are presented in Figure 13.

Due to coherent illumination, diffraction rings appear in the central input image, which captures the central spatial frequency through the sub-aperture. To eliminate this phenomenon and reduce image noise for improved restoration results, a high dynamic range exposure time is employed during image acquisition. The image data are processed using a Fourier ptychographic algorithm, which effectively eliminates this issue in the Recov image. Compared to the central input image, the Recov result exhibits a smoother appearance. When four deep learning network architectures are applied to further process the Recov image, the image quality is enhanced even more. The experimental data indicate that the deep learning framework optimizes image quality to a certain extent, as evidenced by various no-reference evaluation metrics. This optimization results in an increase in image standard deviation and information content, particularly in the average gradient, which contributes to a sharper image.

The experimental results indicate that the VDSR network, utilizing a ResNet learning framework, significantly enhances image entropy and standard deviation, particularly in improving the average gradient. By employing a residual learning strategy, the learning gradient of the SRCNN network structure is effectively reduced without altering its simple architecture, resulting in superior performance in image quality enhancement compared to the traditional SRCNN structure. The four deep learning network architectures further optimize the quality of high-resolution images obtained through Fourier transform synthetic aperture techniques. Following the secondary optimization of the deep learning framework, the imaging quality of the experimental samples shows slight improvements in both image entropy and standard deviation, along with excellent performance in enhancing the average gradient of the images. Additionally, four deep learning frameworks are applied to process the FPM images (refer to the Supplementary Materials Section S3), leading to further improvements in image quality.

6. Conclusions

In conclusion, FP and image super-resolution are techniques that can be applied at both macroscopic and microscopic scales, utilizing physical and computational methods, respectively. However, the integration of FP with deep learning to enhance the super-resolution imaging of long-distance targets has not been previously reported. We conducted both simulations and experiments on macroscopic FP. Four deep learning network frameworks were introduced to further enhance the imaging quality of FP for distant objects. The target samples were not limited to USAF resolution charts; instead, we utilized film samples that are more relevant to real-life scenarios. Our results demonstrate that the imaging quality of macroscopic targets can be significantly improved by combining deep learning networks with FP reconstruction. This paper provides valuable insights for other projects that employ computational imaging methods to reconstruct macroscopic objects. By applying deep learning networks to the post-processing of existing computational imaging results, we anticipate further improvements in reconstruction quality.