High Bandwidth-Utilization Digital Holographic Reconstruction Using an Untrained Neural Network

Abstract

Slightly off-axis digital holographic microscopy (DHM) is the extension of digital holography imaging technology toward high-throughput modern optical imaging technology. However, it is difficult for the method based on the conventional linear Fourier domain filtering to solve the imaging artifacts caused by the spectral aliasing problem. In this article, we propose a novel high-accuracy, artifacts-free, single-frame, digital holographic phase demodulation scheme for low-carrier-frequency holograms, which incorporates the physical model into a conventional deep neural network (DNN) without training beforehand based on a massive dataset. Although the conventional end-to-end deep learning (DL) method can achieve high-accuracy phase recovery directly from a single-frame hologram, the massive datasets and ground truth collection can be prohibitively laborious and time-consuming. Our method recognizes such a low-carrier frequency fringe demodulation process as a nonlinear optimization problem, which can reconstruct the artifact-free phase details gradually from a single-frame hologram. The phase resolution target and simulation experiment results quantitatively demonstrate that the proposed method possesses better artifact suppression and high-resolution imaging capabilities than the physical methods. In addition, the live-cell experiment also indicates the practicality of the technique in biological research.

1. Introduction

Quantitative phase imaging (QPI) [1,2,3] emerges as a powerful, label-free approach to studying transparent cell structure, which makes a significant contribution to biological research and medical diagnosis. It uses the refractive index as the endogenous contrast agent to generate subcellular-specific quantitative maps of analyzed live bio-structure [4,5]. Digital Holographic Microscopy (DHM) [6,7,8,9,10] records complete wavefront information by the principle of interferometry and can complete real-time, highly robust QPI with a single-frame hologram. DHM has emerged as a valuable means in the biomedical field, such as for living cell analysis [11], drug release monitoring in vitro [12], and the optical metrology of nanostructures [13].

According to whether there is a titled angle between the object light and the reference light, the holography is categorized into in-line and off-axis digital holography (DH). In-line DH can achieve high-bandwidth phase reconstruction, but the reconstruction result of the object is far from the ground truth due to the influence of twin images. So, it requires multiple holographic imaging at different distances from the object plane along with iterative algorithms [14,15], which degrades the temporal resolution. The off-axis configuration [16,17] introduces an additional coherent reference beam that is tilted and superimposed with the object beam, which separates the autocorrelation and cross-correlation terms in the spatial frequency domain (SFD). However, this configuration requires a sufficiently high carrier frequency hologram at the expense of the imaging system’s space bandwidth product (SBP). Therefore, the slightly off-axis DH configuration [18,19], as a high-SBP single-frame demodulation solution, is proposed. It separates the conjugated spectrum while leaving the autocorrelation term partially overlapped with information-carrying, cross-correlation terms. This novel DH configuration makes full use of the sensor’s spatial bandwidth, which can significantly improve the detector’s spatial bandwidth utilization (SBU) [20].

Notably, in the slightly off-axis DH system, the inevitable overlapping of spectrum produces phase artifacts, which significantly degrades the imaging quality and impairs the system’s practicality. To address this problem and achieve high-resolution, artifact-free imaging, many solutions have been proposed. Based on the theory of “cepstrum” and homomorphic filtering, the nonlinear filtering approach [21] is proposed to achieve zero-order suppression. It takes the logarithm of the spatial spectrum as the filtering space and employs the additive property to separate the $\pm 1$-order terms and the zero-order term. However, this method needs to guarantee that one of the diffraction terms must be confined to one quadrant of the SFD. Furthermore, the Kramers–Kronig (KK) relation [22,23] is exploited for phase demodulation, which introduces a complex analytic function and utilizes the half-space bandwidth of the sensor to achieve high-SBP imaging. However, it requires separation of the cross-correlation terms of the interferogram in the extended SFD and needs to guarantee that the intensity of the reference wave is much stronger than the object wave. Of note, an exquisite low-carrier frequency fringe demodulation approach has been presented, namely, variational Hilbert quantitative phase imaging (VHQPI) [24]. It adaptively alleviates the overlapped-spectrum problem and improves the system’s reliability, demonstrating the excellent practicality in biological applications. Recently, a high-throughput, artifact-free, slightly off-axis holographic reconstruction method based on Fourier ptychographic microscopy (FPM) [25] has been proposed. In this method, the complex object field can be iteratively reconstructed from the recorded hologram by solving a nonlinear optimization problem.

With the development of artificial intelligence (AI), deep-learning-based (DL-based) imaging schemes provide a new paradigm for QPI [26,27,28,29]. Driven by a large datasets, deep neural networks (DNN) can perform complete wavefront reconstruction of specimens efficiently [30]. However, obtaining datasets and corresponding ground truth, especially when the scheme is applied to bio-samples, can be pretty time-consuming and cumbersome. Deep Image Prior (DIP) [31] applies an untrained network to the solution of several inverse problems without a massive training dataset and ground truth, which can fit a randomly initialized DNN to a single corrupted image. Inspired by the DIP, an untrained network model named “PhysenNet” [32] is proposed, which incorporates a complete physical model into the conventional DNN to achieve QPI from a single-frame intensity map. Recently, an unsupervised blind phase retrieval method called Deep Digital in-line holography (Deep-DIH) [33] has been proposed. Unlike the traditional DL-based phase retrieval model, it can achieve highly robust, high-resolution phase retrieval from a single-frame, in-line hologram iteratively without a large number of datasets.

Inspired by the successful application of the interplay between DNN and the physical model, in this work, we propose a high-SBU, artifacts-free, low-carrier-frequency fringe demodulation approach for the slightly off-axis system, which completes phase recovery iteratively using an untrained neural network. Unlike the massive-data-driven DL training model, our method utilizes a single-frame hologram as the input of the model. After the primary demodulation based on the FT method, the complex amplitude will be updated by an unsupervised network iteratively to reconstruct the final phase results. In this process, an auto-encoder network [33,34] with an “hourglass” architecture is used to constantly learn the delicate phase details of objects instead of the artifacts. Furthermore, the physical model will be migrated to the cost function of the network, which can help the DL model to correct the network in the parameter space effectively. In short, our method can improve the DH system’s SBU to 53.9% under the single-shot, multiplex-free situation while achieving high-resolution, artifacts-free QPI. The results of the phase resolution target and the simulation experiment can quantitatively demonstrate the artifact suppression and high-resolution imaging capability of our method, while the results of the live-cell experiment indicate the practicality of the method in biological research.

2. Principle

In DH wavefront recording, a Mach–Zehnder interference light path is designed, as shown in Figure 1a. The sample is illuminated by a laser at 532 nm, and the complex object optical field ($O(x,y)=A_O(x,y)exp\left(i{\phi }_O(x,y)\right)$) passing through the objective lens is encoded into intensity modulations by interfering with the reference wave ($R(x,y)=A_R(x,y)exp\left(i{\phi }_R(x,y)\right)$). The intensity distribution of the recorded hologram can be expressed as:

$$f(O,R)={|O(x,y)+R(x,y)|}^2={\left|O\right|}^2+{\left|R\right|}^2+O^{*}R+R^{*}O$$

(1)

where $A_O(x,y)$ and $A_R(x,y)$ denote object wave amplitude and reference wave amplitude, respectively. Moreover, ${\phi }_O(x,y)$ and ${\phi }_R(x,y)$ are the phase of the object wave and reference wave, and “*” denotes the complex conjugation. The hologram is captured by a CCD detector (The Imaging Source DMK 23U274, 1600 × 1200, 4.4 $\mathsf{\mu }$m), as shown in Figure 1b. In this work, we turn the DH system into a slightly off-axis state while causing the zero-order term to partially overlap with the ±1-order terms, as shown in Figure 1c. The +1-order term ($R^{*}O$) can be extracted by the spectral filtering window. The recovered wrapped phase can then be recovered by inverse Fourier transform. However, the phase aberration is inevitably caused by the objectives in the lens system. This problem can be addressed by employing the phase aberration compensation technology based on principal component analysis (PCA) [35,36]. Then, the phase unwrapping method based on reliability [37] analysis is used to obtain the final phase result.

Of note, in the DH system, the hologram can demonstrate various spatial carrier frequencies due to different modulation methods, which will influence the SBU of the sensor. According to the relationship between object bandwidth and sensor’s bandwidth, the SBU can be defined by the ratio of the area of cross-correlation terms to camera bandwidth ($1/\Delta \times 1/\Delta$, where $\Delta$ is the pixel size) in the SFD. Figure 1d–g discuss the SBU under the four different spectrum states. Figure 1d shows that under the quasi-off-axis situation with a horizontally modulated reference beam, the bandwidth of the object is $B_o=0.25B_c$ (where $B_c$ is the the bandwidth of camera). Assuming the spatial resolution of the DH system is limited by the spatial sampling of the CCD, and the area of the distribution of the spectrum can be calculated as $\pi {(B_o/2)}^2$ in the diffraction-limited microscopy system. By definition, the SBU of the DH system can be written as:

$$SBU=2\pi {\Delta }^2{(B_o/2)}^2$$

(2)

Therefore, if the SFD of the hologram has M × N pixels, the area of the cross-correlation terms is $2\pi (M/8)(N/8)$. The SBU is $2\pi (MN/64)/\left(MN\right)=9.82$%. Additionally, under the situations shown in Figure 1e–g, the bandwidths of the object are $B_o=0.5B_c$, $B_o=2B_c/(2+3\sqrt{2})$, and $B_o=2B_c/(2+\sqrt{2})$, respectively. Moreover, the corresponding SBU can be calculated as 39.27%, 16.12%, and 53.90%, respectively. It can obviously demonstrate that the slightly off-axis scheme has a higher SBU compared to the quasi-off-axis scheme.

3. Method

When the supervised network is used for image restoration, the cost function is defined to penalize the error between the network’s output and the ground truth. Unlike this network, DIP [31] (an unsupervised blind image restoration method) can fit a randomly initialized convolutional neural network (CNN) to a single corrupted image. This method does not need a massive training dataset and ground truth and can naturally learn the uncorrupted and realistic parts using an untrained network in an iteratively convergent manner. Here, inspired by the DIP, we designed a novel learning scheme using the physical model in the training process’s cost function for low-carrier frequency fringe demodulation in the slightly off-axis DH system. In the process of fringe demodulation, we recognize the artifacts-free complex amplitude reconstruction of the hologram as a nonlinear optimization problem.

Essentially, the proposed method migrates the physical model to the unsupervised training process, in which the final artifacts-free recovered results can be regarded as the solution to the nonlinear inverse problem. After the network generates the main body of the object, it continues to generate the delicate phase details of the object instead of the phase artifacts.

As depicted in Figure 2, the complex amplitude can be reconstructed from the captured hologram (shown in Figure 2a) based on the FT method [6], and then the real and imaginary parts of the reconstructed complex analytic signal are fed into the network as a two-channel input. A deep convolutional auto-encoding network with randomly initialized weights is trained to obtain the real and imaginary parts of the output. The reconstructed output (i.e., the reconstructed hologram, as shown in Figure 2g) can be expressed as $f(O,R)$, where O is the object wave and R is the reference wave. The cost function can be defined using the reconstructed hologram and the captured hologram, which will converge to a minimum by updating the reconstructed complex amplitude as follows:

$$w={argmin\parallel H-f(O,R)\parallel }_2^2$$

(3)

where $\omega$ is the weight of the network, and H is the captured hologram. When the cost function based on the gradient descent method is minimized, the network will search for the optimal results by tuning the model parameters within the parameter space. Specifically, the network first generates the primary instance, which is the rough shape estimate of the reconstructed object. Furthermore, the network gradually recovers the artifact-free phase details of the object from coarse to fine.

In our method, we use an auto-encoding network with an “hourglass” architecture [33,34], as shown in Figure 2d, which encodes a high-dimensional network input into lower-dimensional space and reconstructs a high-dimensional output object from the latent variables. The network architecture uses batch normalization [38] and ReLU activation [39] to accelerate the model convergence. Figure 2d shows the hyper-parameters, the tensor dimensions, and the channels of each layer of the network. Notably, in this network, the wavelet transform and its inverse transform are used to realize down-sampling and up-sampling, which replace the pooling, stridden convolution, or interpolation. According to [40], using a wavelet transform could impose sparsity in the reconstruction process and result in more precise reconstruction with lower distortion, which is more suitable for artifacts-free phase recovery in the slightly off-axis system.

4. Simulation

In this section, the simulation experiment is designed to demonstrate the validity and performance of the proposed method. To discuss the properties of the method quantitatively, we design the comparison experiment with two single-frame fringe demodulation algorithms based on the physical model: the conventional FT method and VHQPI [24]. We implement our method using the PyTorch Framework [41] in a GPU workstation with an NVIDIA GeForce RTX2080 Ti graphics card. A fixed learning rate of 0.005 for simulation experiment is adopted for the Adam optimizer [42]. A simulated hologram of size 160 × 160 obtained excellent training results after 5000 iterations.

Specifically, we select two images from MATLAB as intensity and a phase of the complex amplitude of the object wave as $O(x,y)=A_O(x,y)exp\left[i{\phi }_O(x,y)\right]$. Likewise, in the reference light arm, the plane wave field $R(x,y)=A_R(x,y)exp\left[i{\phi }_R(x,y)\right]$ is propagated. However, it is difficult to guarantee that the reference light intensity is a constant due to the system’s instability and environmental disturbances, so we simulate the reference wave intensity using a two-dimensional standard Gaussian distribution as $A_R\left(x,y\right)=\frac{1}{2\pi }exp(-(x^2+y^2)/2)$. According to Equation (1), we can naturally obtain the simulated hologram (as shown in Figure 3e). It is also worth noting that we add a pupil function constraint to the iterative update of the object light in the method. This is because, unlike the lens-free in-line DH system, the slightly off-axis DH system is the imaging system with an objective lens. Adding pupil function constraints during the reconstruction process better complies with the physical model’s iterative structure. Moreover, the learning process of the object wave’s complex amplitude is often accompanied by high-frequency artifacts, and such a constraint can cleverly realize iterative correction of the artifacts.

Figure 3 demonstrates the performance of our method compared with the conventional FT method and the VHQPI. Figure 3b,c show the phase recovery results through the FT method and the VHQPI, respectively, whose enlarged views are shown in Figure 3(b1,c1). In the FT-based phase reconstruction, we implement SFD filtering using the window calculated by the simulated numerical aperture (NA). The imaging result is severely disturbed by artifacts due to the overlapping-spectrum problem. Because of the restrictions of the limited physical model, VHQPI performs a limited capability of zero-order spectrum suppression, and the reconstructed phase results still have certain phase artifacts. In contrast, the result of our method can suppress phase artifacts efficiently and achieve high-accuracy phase imaging, as shown in Figure 3(d,d1). Figure 3a is the ground truth gained by phase recovery after theoretical background removal, and its magnified view is shown in Figure 3(a1). Intuitively, we can easily find that our results are more similar to the ground truth than the other two methods. To discuss the performance of methods quantitatively, the Root Mean Square Error (RMSE) is calculated to measure the average of the squares of the pixel-wise errors between the ground truth image and the reconstructed image. Furthermore, the difference distribution between the recovered results and the ground truth are shown in Figure 3(b2,c2,d2). We can conclude that our method has higher reconstruction accuracy and better artifact-suppression ability.

In the real DH system, the captured hologram is always affected by noise due to environmental factors and imperfect detectors, etc. We investigate the phase reconstruction ability and artifacts-suppression ability of our method in the presence of additive white Gaussian noise. In this experiment, we add different noise levels (Sigma = 0.1 and 0.2) to the simulated hologram as input to the network. Figure 4 shows the reconstructed results, demonstrating that our method has excellent artifacts-suppression ability, even with the effect of noise.

5. Experiment

To verify the performance of the proposed method in a real-world system, we designed the DH experiment for the standard phase resolution target and COS-7 (ATCC CRL-1651, CV-1 in Origin Simian-7) cells. Here, we use the Digital holographic smart computational light microscope (DH-SCLM) [3] developed by SCILab and turn it to a slightly off-axis state for hologram acquisition. In the DH-SCLM, the object wave transmitting the objective lens (UPLanSAPO ×10/0.4NA, Olympus, Japan) interferes with the reference light and is recorded by the camera (The Imaging Source DMK 23U274, 1600 × 1200, 4.4 $\mathsf{\mu }$m). The central wavelength of the illumination is 532 nm. In our method, according to the cost function in Equation (3), a reference light intensity map needs to be captured separately to obtain the reconstructed hologram, which is a no-need-repeat system calibration process.

Figure 5 shows the reconstructed results of the standard phase resolution target (QPTTM, Benchmark Technologies Corporation, United States, Ri = 1.52). When the zero-order and ±1-order are overlapped with each other in the SFD (shown in Figure 5i), the phase results recovered by the conventional FT method are severely impaired by imaging artifacts due to the autocorrelation term, as shown in Figure 5b,f. In VHQPI, the frequency component extraction (uVID) [43] can remove the zero-order term to a certain extent. However, influenced by the physical limitations (e.g., iterative instability and parameter uncertainty), the reconstructed phase results still suffer from imaging artifacts, as shown in Figure 5c,g. The proposed method migrates the physical model to the training process’s cost function to recover the artifacts-free phase results iteratively, as shown in Figure 5d,h. The results of the three methods show that our method and VHQPI can clearly see the fourth element in the ninth group of the phase resolution target, while the FT method can only see the second element in the ninth group. However, the VHQPI’s result has much coarser background noise due to the imaging artifacts, and that of our method is much smoother and cleaner. These demonstrate that our method can improve the imaging resolution while having better artifacts-suppression capability during low-carrier frequency fringe demodulation. Notably, the captured hologram and its spectrum based on the slightly off-axis DH system are shown in Figure 5a,e,i, and their SBU can reach 53.9% after calculation. Therefore, we can naturally conclude that the proposed method can achieve high-SBU, artifacts-free QPI under the single-frame, multiplex-free acquisition system [20].

To verify the practicality of the proposed method in biological research, we conducted an experiment on COS-7 cells under a slightly off-axis DH system. Figure 6 shows the results of phase reconstruction based on the FT method and our method. The captured hologram and its spectrum under the ×20/0.45NA objective lens are shown in Figure 6a,d. After calculation, the SBU of the system’s sensor can reach 31.7% in Figure 6d. The yellow filter window in Figure 6d is calculated from the numerical aperture of the objective used in the system, which is applied to spectrum filtering for the FT method. Figure 6b shows the phase recovery result based on the FT method. We select three regions of interest (ROI, Area1, Area2, and Area3) on the specimens, and their detailed views are shown in Figure 6(e1–e3). It can be revealed that the fringe-like error caused by overlapping spectrum dramatically degrades the phase recovery quality. Indeed, reducing the size of the FT filter window may be a way to alleviate artifacts, but this will cause phase imaging blur since the object’s high-frequency information cannot be enclosed in the limited filtering window. In contrast, Figure 6c shows smoother and cleaner phase results reconstructed by the proposed method, and the magnified views are shown in Figure 6(f1–f3). In our method, as the iterative process continues to converge, the auto-encoding network effectively recovers the imaging details of the object instead of artifacts. The live-cell experiment results vividly demonstrate that our method has excellent artifact-suppression ability in imaging and high practicality in biological research.

6. Conclusions

We have demonstrated a low-carrier-frequency fringe demodulation method for high-SBU DH imaging. Exploiting phase imaging with an untrained neural network, this approach can increase the SBU of the system’s sensor from 9.82% of the quasi-off-axis DH to 53.9% under the slightly off-axis DH system. Specifically, we recognize the artifacts-free phase recovery under the slightly off-axis DH as a nonlinear optimization problem. Then, we skillfully migrate the physical model to the cost function of the DL training process to correct the parameter of the neural network. This process will converge to an optimal state iteratively and will gradually recover the delicate phase details of the object from coarse to fine.

Different from previous methods [44,45,46] for zero-order suppression in the slightly off-axis system, the proposed method only requires a single hologram. Based on the principle of “cepstrum” and homomorphic filtering, the nonlinear filtering [21] and the KK method [22,23] are proposed to suppress the zero-order term in the SFD. However, these methods inevitably require intensity restrictions on the object and reference beams to satisfy the condition that the object-reference ratio is less than 1 [25]. Moreover, the VHQPI method is proposed to achieve zero-order term suppression in the SFD by adaptively and automatically extracting the background component of the image using uVID. However, such an iterative process of extracting frequency components has certain physical limitations that make it difficult to obtain an exact background component.

Although DL provides a good imaging paradigm for QPI, the conventional DL models tend to rely on a large number of the dataset. However, it may be prohibitively laborious and time-consuming for a real-world DH system to collect datasets and generate the corresponding ground truth. A novel scheme was proposed using an untrained neural network in [31,32,33], which fits a complete physical model into a conventional DNN without training beforehand based on a large dataset. Inspired by this, our method is proposed to achieve high-SBU, artifact-free QPI from a single-frame hologram. In addition, we consider that the method can be combined with other techniques in the future, such as optical diffraction tomography [47,48,49] and common-path off-axis interferometers [50,51], etc. Additionally, we envision that the adopted idea of a physics-enhanced DL model in this work can also be enlightening for the diverse computational imaging systems in the future.