Fourier Ptychographic Microscopy with Optical Aberration Correction and Phase Unwrapping Based on Semi-Supervised Learning

Abstract

Fourier ptychographic microscopy (FPM) has recently emerged as an important non-invasive imaging technique which is capable of simultaneously achieving high resolution, wide field of view, and quantitative phase imaging. However, FPM still faces challenges in the image reconstruction due to factors such as noise, optical aberration, and phase wrapping. In this work, we propose a semi-supervised Fourier ptychographic transformer network (SFPT) for improved image reconstruction, which employs a two-stage training approach to enhance the image quality. First, self-supervised learning guided by low-resolution amplitudes and Zernike modes is utilized to recover pupil function. Second, a supervised learning framework with augmented training datasets is applied to further refine reconstruction quality. Moreover, the unwrapped phase is recovered by adjusting the phase distribution range in the augmented training datasets. The effectiveness of the proposed method is validated by using both the simulation and experimental data. This deep-learning-based method has potential applications for imaging thicker biology samples.

1. Introduction

Fourier ptychographic microscopy (FPM) [1] is an innovative computational imaging technique designed to address the challenge of achieving both a large field of view and high resolution, with which traditional microscopy techniques struggle to balance [2]. FPM achieves high space/bandwidth product and quantitative phase imaging without the precise mechanical scanning devices or the interferometric systems [3], offering a cost-effective solution with promising application potential [4]. However, conventional FPM methods suffer from image reconstruction issues such as low computational efficiency, susceptibility to noise, aberration effects and phase wrapping problem [5]. To address these challenges, various improvements have been proposed [6,7,8,9], including Wirtinger flow optimization [10] and the simulated annealing algorithm [11]. With the advancement of deep learning, its application in FPM has also seen widespread adoption [12,13,14,15,16].

Currently, the mainstream applications of deep learning in FPM can be categorized into two types. The first type leverages large training datasets to directly learn the mapping from low-resolution amplitude to high-resolution amplitude and phase [17]. Examples include convolutional neural network-based models like Ptychnet [18], along with subsequent improvements introduced by Zhang et al. [19], which incorporates multi-scale architectures, residual connections, attention mechanisms, and various other enhancements. F. Bardozzo et al. applied generative adversarial networks (GANs) to achieve high-resolution, artifact-free reconstructions with robust cross-explainability and clinical validation [20]. T. Feng et al. proposed a linear-space-variant (LSV) model to achieve uniform full-field imaging while effectively eliminating vignetting-induced artifacts without segmenting the images [21]. K. Wu et al. introduced a blind deep-learning-based preprocessing method (BDFP) that enhances noise robustness by modeling noise distribution for data augmentation [22]. These data-driven approaches achieve image reconstruction by learning complex mapping relationships, and most of them are supervised learning. Once trained, the rapid inference is allowed within the same experimental setup.

The second type integrates the physical imaging model as prior knowledge, forming a physics-guided neural network architecture [23]. For instance, Jiang et al. [24] and Zhang et al. [25] simulated the forward imaging process of FPM using a neural network, with high-resolution amplitude and phase images as trainable parameters. By using optimization algorithms like gradient descent, the network is trained by its low-resolution amplitude output data distribution, which matches the experimentally collected data distribution. Zhang et al. [26] incorporated a single iteration of the FPM forward algorithm as a data preprocessing step, using the neural network for subsequent optimization to reduce network complexity [27]. Wu et al. proposed the denoising diffusion probabilistic models for Fourier ptychography method [28], which combines diffusion models with Wirtinger gradient descent techniques, offering a new approach to FPM reconstruction. These physically guided methods typically do not require manual GT labels and can be considered self-supervised learning, offering the advantage of low-cost reconstruction; however, their parameter optimization process was often tailored to single sample, limiting the advantage of deep learning that can optimize results using large datasets.

When encountering phase wrapping problem that arise with samples exhibiting significant phase distribution differences, FPM typically employs physical methods or phase unwrapping algorithms to address the problem, such as introducing solutions with varying refractive indices into the sample. And traditional unwrapping algorithms tend to exhibit substantial reconstruction errors when dealing with complex phase structures [29,30]. The advancement of deep learning offers a novel approach to phase unwrapping in FPM [31].

Based on our previous work on a Fourier ptychographic transformer (FPT) [32], this paper further introduces a semi-supervised Fourier ptychographic microscopy method, inspired by two mainstream approaches in the application of deep learning to FPM, termed SFPT. Semi-supervised learning bridges the gap between supervise learning and self-supervise learning by leveraging both labeled and unlabeled data [33]. In this framework, unlabeled data, integrated with the forward imaging principles of FPM, provide additional learning signals such as pupil aberration to the labeled data, thereby improving model performance. We utilize self-supervised learning to reconstruct pupil aberration parameters of the Zernike model from real datasets. Considering the detrimental effects of information loss and the inherent noise during the forward downsampling process, relying only on the captured low-resolution amplitude images as a self-supervised constraint is insufficient for directly learning accurate high-resolution amplitude and phase. Therefore, we introduce a simulated dataset based on the already reconstructed pupil function for data augmentation to rectify the network reconstruction outputs via supervised learning, thereby further enhancing imaging quality. Additionally, we adopted a data-driven approach that allows the network to directly learn the unwrapped phase mapping beyond the 2$\pi$ range. The proposed method combines the strengths of physics-guided neural network with end-to-end deep learning, effectively reconstructing pupil aberration as well as high-resolution amplitude and phase information.

2. Principle of FPM

The FPM algorithm can be divided into a forward downsampling algorithm and an inverse reconstruction algorithm. The forward downsampling algorithm downscales high-resolution amplitude and phase maps into low-resolution amplitude images corresponding to specific illumination conditions, typically used to simulate the low-resolution intensity images captured by a CCD camera in an FPM system, thus generating simulation datasets. This algorithm employs two high-resolution images to simulate the amplitude $A(x,y)$ and phase $P(x,y)$, thereby modeling the optical response $O(x,y)$ of the sample under FPM system illumination:

$$O\left(x,y\right)=A\left(x,y\right)e^{j·P\left(x,y\right)}=o\left(x,y\right)e^{j\left(k_x·x+k_y·y\right)},$$

(1)

where $o(x,y)$ represents the sample transmission function, and $k_x,k_y$ denotes the plane wave vectors along the x-axis and y-axis. The wave vector $k_i$ corresponding to the i-th LED can be expressed as follows:

$$k_i=(k_{xi},k_{yi})=\left({\displaystyle \frac{sin{\theta }_{xi}}{\lambda }},{\displaystyle \frac{sin{\theta }_{yi}}{\lambda }}\right),$$

(2)

where $({\theta }_{xi},{\theta }_{yi})$ represents the illumination angle of the i-th LED, and $\lambda$ denotes the wavelength. When the i-th LED illuminates the sample with the wave vector $k_i$, it effectively shifts the sample spectrum $O_i\left(k\right)$ to be centered around $k_{xi},k_{yi}$, expressed as

$$O_i(k_x-k_{xi},k_y-k_{yi})=F\left\{o(x,y)e^{j(k_{xi}x+k_{yi}y)}\right\},$$

(3)

where $F\left\{·\right\}$ denotes the Fourier transform. As light passes through the objective lens, the field of view undergoes low-pass filtering via the pupil function. Here, we define the initial pupil function $P_0\left(k_x,k_y\right)$ as the contrast transfer function of an aberration-free system’s objective lens:

$$P_0\left(k_x,k_y\right)=CTF\left(k_x,k_y\right)=\left\{\begin{array}{c}1,{k_x}^2+{k_y}^2\le {(NA•{\displaystyle \frac{2\pi }{\lambda }})}^2\hfill \\ 0,{k_x}^2+{k_y}^2>{(NA•{\displaystyle \frac{2\pi }{\lambda }})}^2\hfill \end{array}\right.,$$

(4)

where $NA$ represents the numerical aperture of the objective lens. Thus, the intensity image $I_i(x,y)$ recorded by the CCD corresponding to the i-th LED is expressed as

$$I_i(x,y)={\left|g_i(x,y)\right|}^2={\left|F^{-1}\left\{{\mathsf{\Phi }}_i\left(k_x,k_y\right)\right\}\right|}^2={\left|F^{-1}\left\{P_i\left(k_x,k_y\right)O_i\left(k_x-k_{xi},k_y-k_{yi}\right)\right\}\right|}^2.$$

(5)

The inverse reconstruction algorithm in FPM achieves phase recovery and aperture synthesis super-resolution by iteratively updating the sample transmission function $O_i\left(x,y\right)$ between the spatial and frequency domains, simultaneously shifting in the frequency domain and introducing amplitude information in the spatial domain. This process can be expressed as

$${{\mathsf{\Phi }}_i}^{\prime }\left(k_x,k_y\right)=F\left\{\sqrt{I_{ci}(x,y)}{\displaystyle \frac{g_i(x,y)}{\left|g_i(x,y)\right|}}\right\},$$

(6)

$$O_{i+1}\left(k_x-k_{xi},k_y-k_{yi}\right)=O_i\left(k_x-k_{xi},k_y-k_{yi}\right)+\alpha {\displaystyle \frac{P_i^{\ast }\left(k_x,k_y\right)}{{\left|P_i\left(k_x,k_y\right)\right|}_{max}^2}}\left[{{\mathsf{\Phi }}_i}^{\prime }\left(k_x,k_y\right)-{\mathsf{\Phi }}_i\left(k_x,k_y\right)\right],$$

(7)

$$P_{i+1}\left(k_x,k_y\right)=P_i\left(k_x,k_y\right)+\beta {\displaystyle \frac{O_i^{\ast }\left(k_x-k_{xi},k_y-k_{yi}\right)}{{\left|O_i\left(k_x-k_{xi},k_y-k_{yi}\right)\right|}_{max}^2}}\left[{{\mathsf{\Phi }}_i}^{\prime }\left(k_x,k_y\right)-{\mathsf{\Phi }}_i\left(k_x,k_y\right)\right],$$

(8)

where $O_{i+1}\left(k_x-k_{xi},k_y-k_{yi}\right)$ and $P_{i+1}\left(k_x,k_y\right)$ are optimized based on the gradient descent method [34], and $\alpha$ and $\beta$ are update step sizes that need to be predefined. Repeating Equations (5)–(8), the complex field corresponding to the spectra of all i illuminations is updated sequentially, completing one iteration cycle. The cycle is repeated until the phase converges, finalizing the reconstruction.

3. Semi-Supervised Fourier Ptychographic Transformer Network

To address the optical aberration issues encountered by the neural network in practical optical imaging systems, we employ a semi-supervised learning framework. This framework is structured into two stages—self-supervised training and supervised training—as illustrated in Figure 1. For self-supervised learning, we employed a Zernike model with learnable weights to reconstruct the pupil aberration. The first fifteen Zernike terms were weighted by their corresponding learnable coefficients, summed to construct the pupil aberration, and incorporated into the downsampling step of the FPM forward process. With this learnable pupil function, the generated low-resolution amplitude sequences were compared with the input to calculate the loss and update both the network parameters and the aberration weights. After a predefined number of training epochs, the training transitioned to the second stage (the supervised training) during which the pupil aberration weights were fixed, and simulated datasets were created using the pupil function for data augmentation. The remaining training epochs are completed using supervised learning to further enhance the neural network performance and improve the overall imaging quality.

4. Fourier Ptychographic Transformer Backbone Network

We utilized a Fourier ptychographic transformer (FPT) as the backbone network [32], providing an efficient end-to-end training paradigm with rapid inference capabilities. As shown at the top panel in Figure 2, the network utilizes all low-resolution amplitude images as the input to ensure the complete information required for reconstruction. The architecture is primarily composed of convolutional neural networks (CNNs) and a self-attention mechanism [35]. The initial layers of the network employ convolutional operations for feature extraction, where the translation invariance and local receptive field properties of convolution are well suited for capturing shallow texture features of images. The deeper layers employs self-attention mechanism for extracting complex, high-dimensional features [36]. The self-attention mechanism effectively captures long-range dependencies, enabling super-resolution and phase recovery.

The input of the network consists of 225 channels’ low-resolution amplitude images, denoted as $X_i\in {\mathbb{R}}^{C\times H\times W}$, where H and W represent the height and width of the amplitude image, while C denotes the channel dimension which is 225. Initially, the shallow texture feature extraction is performed through a convolutional layer to obtain $X_0$. Subsequently, deep feature extraction is carried out using 8 transformer blocks, each comprising an image to patch module, a pixel self-attention module, a channel self-attention module, a patch to image module, and a multi-layer perceptron (MLP) module.

In each transformer block, $X_n$ is first converted into a window patch sequence $X_n^t\in {\mathbb{R}}^{C\times P\times P\times {\displaystyle \frac{H}{P}}\times {\displaystyle \frac{W}{P}}}$ via the image to patch module, which is inspired by the shifted windows in the Swin transformer [37], where H represents the window patch size. This module alternates pixel assignments into patches, ensuring that the deep network captures long-range spatial dependencies. Then, feature extraction is conducted along two dimensions: the pixel self-attention module calculates the self-attention relationship $f_p\left(X_n^{tp}\right)$ among pixels within each spatial patch, focusing on capturing spatial-level detail features, expressed as follows:

$$f_p\left(X_n^{tp}\right)=softmax\left(L_Q\left(X_n^{tp}\right)\ast {\displaystyle \frac{L_K{\left(X_n^{tp}\right)}^T}{\sqrt{P\times P}}}\right)L_V\left(X_n^{tp}\right),X_n^{tp}\in {\mathbb{R}}^{C\times \left({\displaystyle \frac{H}{P}}\times {\displaystyle \frac{W}{P}}\right)\times \left(P\times P\right)},$$

(9)

where $X_n^{tp}$ represents the reshape of $X_n^t$, and $L_Q,L_K,L_V$ represent linear transformations that preserve dimensional consistency. The channel self-attention module computes the self-attention relationships $f_c\left(X_n^{tc}\right)$ among channel features, thereby enhancing the network capacity to promote cross-channel interactions, expressed as follows:

$$f_c\left(X_n^{tc}\right)=softmax\left(L_Q\left(X_n^{tc}\right)\ast {\displaystyle \frac{L_K{\left(X_n^{tc}\right)}^T}{\sqrt{C}}}\right)L_V\left(X_n^{tc}\right),X_n^{tc}\in {\mathbb{R}}^{\left({\displaystyle \frac{H}{P}}\times {\displaystyle \frac{W}{P}}\right)\times \left(P\times P\right)\times C},$$

(10)

where $X_n^{tc}$ represents the reshape of $X_n^t$. The results from both attention modules are then summed and restored to image distribution through the patch to image module, yielding $X_n^s$, which is subsequently fed into the MLP for nonlinear feature extraction, producing the block output $X_{n+1}$. The upsample module achieves $\times 2$ super-resolution enhancement through pixel transformation. Finally, a convolutional layer performs channel fusion, outputting high-resolution amplitude and phase images.

Loss Function

The loss function is composed of three parts: L1 loss, SSIM [38] loss between the output amplitude and phase of the network and the ground truth, as well as the amplitude loss between the reconstructed low-resolution amplitudes and the input amplitudes.

$$\begin{array}{c}loss=w_1·L_{L1}(X_A,Y_A)+w_1·L_{L1}(X_P,Y_P)+w_2·L_{SSIM}(X_A,Y_A)+\\ w_2·L_{SSIM}(X_P,Y_P)+w_3·L_{amp}(A_1,A_2)\end{array}$$

(11)

$$L_{L1}(x,y)={\displaystyle \frac{1}{n}}\sum _{i=1}^n\left|y_i-f\left(x_i\right)\right|,$$

(12)

$$L_{amp}(A_1,A_2)={\displaystyle \frac{1}{n}}\sum _{i=1}^n{\displaystyle \frac{{\displaystyle \sum _{i=1}^m}\left|x_i-D_{fpm}\left(f\left(x_i\right)\right)\right|}{{\displaystyle \sum _{i=1}^m}{x}_i}},$$

(13)

$$L_{SSIM}(x,y)=1-{\displaystyle \frac{(2{\mu }_x{\mu }_y+c_1)({\sigma }_{xy}+c_2)}{({\mu }_x^2+{\mu }_y^2+c_1)({\sigma }_x^2+{\sigma }_y^2+c_2)}},$$

(14)

where n denotes the number of pixels in a amplitude, m denotes the number of LEDs, $D_{fpm}$ denotes the forward downsampling process of FPM, ${\mu }_x{\mu }_y$ denotes the mean of $x,y$, ${\sigma }_x,{\sigma }_y$ represents the standard deviation of $x,y$, ${\sigma }_{xy}$ signifies the covariance of $x,y$, $c_1,c_2$ are constant, and $w_1,w_2,w_3$ are hyper-parameters. During Stage 1 of self-supervised learning, $w_1=1$, while $w_2$ and $w_3=0$; during Stage 2 of supervised learning, $w_1=0$, and both $w_2$ and $w_3=1$.

5. Experiments

5.1. Training Details

The network was trained on an NVIDIA GTX 4090 for a total of 200 epochs. The batch size used for training was 4, with an initial learning rate of $1\times {10}^{-4}$, which was reduced by 25% every 30 epochs. The optimizer employed was Adaptive Moment Estimation (Adam). The entire training process took approximately 2 h.

5.2. Experimental System and Model Parameters

The FPM experimental system we use includes the following parameters: the objective lens NA is 0.16, optical magnification is 4×, the LED array consists of 15 × 15 LEDs spaced 2 mm apart, the illumination wavelength is 0.625 $\mathsf{\mu }$m, the pixel size of the CCD camera is 4.4 $\mathsf{\mu }$m, and the vertical distance between the sample plane and the LED array is 60 mm. The model’s hyperparameters are configured as follows: it employs 8 transformer blocks, an embedding dimension of 512, and a window patch size of 16. The overall computational complexity of the model is 667 GFLOPs, with approximately 31.8 M parameters. The inference time is approximately 100 ms.

5.3. Reconstruction Results

The augmented training dataset used the Div2k [39] dataset, which is rich in high-frequency details as the ground truth. Corresponding low-resolution amplitude sequences were generated using the FPM forward process, with all system parameters consistent with the real experimental setup.

5.3.1. Simulated Dataset Reconstruction

To validate the efficacy of the proposed method, we conducted simulations comparing the two-stage reconstruction performance of the network under varying intensities of noise and optical aberrations. Gaussian-distributed noise with a mean of zero and variances of 0.10, 0.05, and 0.01 was introduced. The comparative results are illustrated in Figure 3. As shown in Figure 3, it is evident that even in the presence of noise effects of different intensities, the first stage of the network effectively reconstructs the pupil function with aberrations, while the supervised learning in the second stage significantly enhances the reconstruction outcomes in terms of SSIM and PSNR.

Further comparative analysis of simulated dataset reconstructions using various methods is presented in Figure 4, where we conducted comparative experiments between the traditional FPM algorithm, the deep learning-based FuNN [26], and SFPT. As shown in Figure 4, our method attains superior performance in PSNR and SSIM metrics for amplitude and phase reconstruction, exhibiting noticeably reduced visual artifacts.

5.3.2. Real Experiment Dataset Reconstruction

To further substantiate the network efficacy in a real experimental environment, we utilized a real experimental system to collect the red blood cell dataset and compared results of different reconstruction methods, as illustrated in Figure 5. As shown in Figure 5, our method produces clearer visual outcomes on real data, while the aberration reconstruction is also comparable to that of traditional FPM pupil aberration reconstruction.

We further collected a variety of samples such as fibers, human vein cells and plexus. Moreover, we compared their reconstruction results in the two stages of the SFPT, as illustrated in Figure 6. As shown in Figure 6, the supervised learning in the second stage effectively enhances the visual fidelity of the reconstructed amplitude and phase. The clearer phase reconstruction accurately reflects the refractive index distribution of the cells, thereby indirectly revealing cellular metabolic activity, pathological states, and physiological processes such as proliferation or apoptosis.

5.4. Phase Unwrapping

To enable the network to learn the mapping from low-resolution amplitude images to high-resolution unwrapped phase image, we set the ground truth phase range for the Stage 2 to −2$\pi$ to 2$\pi$ and generate an augmented dataset to train the network. The reconstruction capability of SFPM for thicker samples with phase ranges exceeding 2$\pi$ was validated using a simulated dataset. The reconstruction results are presented in Figure 7. As shown in Figure 7, SFPM demonstrates a certain degree of reconstruction capability for phase data exceeding 2$\pi$.

To further validate the model’s performance on in a real experimental environment, we captured images of bullfrog erythrocytes, approximately 16–25 $\mathsf{\mu }$m thick [40], and the phase range was calculated to be approximately 2$\pi$ to 4$\pi$ in water. We compared the reconstructed phase of traditional FPM with the reconstructed phase obtained by SFPT trained on different phase range datasets; the reconstruction results are shown in Figure 8:

Figure 8c shows the inference results of the network trained with an augmented dataset generated using phase values distributed from −$\pi$ to $\pi$, while Figure 8d shows the inference results of the network trained with an augmented dataset generated using phase values distributed from −2$\pi$ to 2$\pi$. Comparing the traditional FPM algorithm and unwrapping algorithms, it is evident that the dataset trained with the −$\pi$ to $\pi$ phase distribution encounters phase saturation problems when reconstructing phases beyond the 2$\pi$ period.

In contrast, the reconstruction results in Figure 8d exhibit distinct cell nuclei that resemble the unwrapped phase output of the FPM algorithm, indicating that adjusting the distribution of training data enables the network to accurately reconstruct the authentic phase. This method allows the network to precisely restore the unwrapped phase without incurring additional costs.

5.5. Ablation Experiment

To validate the necessity of each key module in SFPT, we performed ablation experiments, with the results presented in

Table 1. Results of the SFPT key module ablation experiment.

Supervised Learning	Zernike Modes	Pixel and Channel Self-Attention	Amplitude PSNR/SSIM	Phase PSNR/SSIM
×	×	×	17.97/0.2317	14.94/0.3168
×	×	√	18.021/0.2326	15.74/0.3175
×	√	×	27.16/0.8540	18.51/0.9014
√	×	×	18.29/0.2261	15.00/0.2493
√	√	×	26.37/0.8595	21.07/0.9080
√	√	√	26.53/0.8591	21.67/0.9176

. As shown in Table 1, supervised learning, Zernike modes, and the pixel and channel self-attention mechanism in the transformer network all contribute to varying degrees of improvement in the reconstruction results.

6. Conclusions

In this paper, we introduce a semi-supervised Fourier ptychographic microscopy reconstruction network based on the FPT, termed SFPT. SFPT harnesses the sequence modeling capabilities of the FP-transformer to effectively reconstruct high-resolution amplitude and phase, integrates the FPM forward algorithm with Zernike modes for aberration correction and utilizes augmented datasets to learn the unwrapping phase.

Through comparative reconstruction results on simulated and real data, we demonstrate that our method can efficaciously perform a super-resolution reconstruction of amplitude and phase even in the presence of aberrations and noise interference. By adjusting the distribution domain of a network’s training dataset, we achieved phase unwrapping within a specified range and conducted comparative validation on real data, underscoring the effectiveness of the proposed method. However, the proposed method still has several limitations. For instance, SFPT demands substantial computational resources, with the computational cost of the transformer escalating rapidly as the image size increases. For larger images, it requires partitioning the image for reconstruction and subsequent stitching. Additionally, the method necessitates at least 12GB of GPU memory for model training. Although the network demonstrates a certain degree of robustness, it needs to be fine-tuned when dealing with significant variations in aberrations, which incurs additional time costs. In future works, we will explore more lightweight reconstruction algorithms and effective strategies for real-time dynamic FPM reconstructions, which represents the mainstream trajectory for advancing the practical applicability of FPM.