Pixel Resolution Imaging in Parallel Phase-Shifting Digital Holography

Abstract

Parallel phase-shifting digital holography (PPSDH) employing a polarization image sensor can suppress zero-order and twin-image noise through a single exposure, achieve instantaneous measurement of complex-valued dynamic objects, and have broad applications in the areas of biomedicine, etc. To improve the imaging resolution of PPSDH, we propose an oversampled super-pixel image reconstruction method, which can be expressed as the implementation of nearest-neighbor interpolation to replace blank pixels in sparse sub-phase-shift holograms. We found experimentally that the maximum spatial lateral resolution of the reconstructed image based on the existing super-pixel method, B-spline, bicubic, bilinear, and the proposed nearest-neighbor interpolation was 12.4 µm, 11.4 µm, 9.8 µm, 8.8 µm, and 7.8 µm, respectively. The main reason for not reaching the ideal value of 6.9 µm was the inherent residual zero-order and twin-image noise, which needs to be removed in the future.

1. Introduction

Digital holography (DH) enables two-dimensional quantitative complex amplitude imaging or simultaneous amplitude and phase imaging, which is a very powerful method for metrology, analysis, and detection, with broad applications in the areas of biomedicine, instrument manufacturing, aquatic science, extraterrestrial life detection, 3D imaging and displays, security and defense, etc. [1,2,3,4]. Zero-order and twin-image noise are serious obstacles to achieving high-quality imaging in in-line DH, and the phase-shifting (PS) technique is often required for noise suppression. Among the existing noise removal techniques, the traditional temporal PS technique can theoretically provide the best quality of reconstructed images [5,6]. However, due to the nonlinear response and response time of practical PS devices such as piezoelectric transducers, temporal phase-shifting digital holography (PSDH) has the following drawbacks: (1) it is extremely sensitive to environmental vibration and turbulence; (2) the relative phase shifts between interferograms cannot be precisely controlled [7,8]; and (3) real-time measurement of fast-moving targets is not possible [9,10].

To exceed the limitations of PSDH, Y. Awatsuji et al. proposed parallel phase-shifting digital holography (PPSDH) in 2004, using an optical-path-length-shifting array device with a pixelated periodic thickness distribution, or a micropolarizer array device with a pixelated periodic polarization direction distribution, to capture multiple digital holograms with different phase shifts in a single exposure [9,11,12,13,14,15]. The spatial PS technique based on the pixelated polarization mask has the advantages of being essentially simple to align and compose, achromatic over a wide range, etc. [13,16], but also has the drawback that there was no available commercial cost-effective polarization camera until the launch of the polarization-dedicated image sensor IMX250MZR by Sony for snapshot polarization imaging in 2018 [17,18,19]. The IMX250MZR sensor, with an image resolution of 2448 × 2048 and a pixel pitch of 3.45 µm, not only promotes the development of snapshot polarization imaging, but also solves the core device problem of PPSDH, and enables PPSDH to be more widely studied [20,21,22,23,24,25,26,27].

Similar to polarization imaging, “hologram demosaicing” is also a crucial imaging pipeline of PPSDH [28,29,30]. There are currently two classical demosaicing methods in PPSDH: the first is the super-pixel method (SPM), which discards the blank pixels in the sparse sub-phase-shift holograms [16,31,32]; and the second is the hologram interpolation method (HIM), which uses adjacent pixel values instead of blank pixels [9]. Among them, the SPM, which regards a set of four neighboring pixels corresponding to the four phase shifts as a “unit cell”, is simple to implement, but can only calculate the complex amplitude value of the optical field at one point, resulting in the maximum lateral spatial resolution and image resolution of the reconstructed image of PPSDH both being reduced to one-half of that of PSDH in the horizontal and vertical directions, respectively. Although the HIM is more complex than the SPM, it can avoid the loss of image resolution [23]. Xia, P et al. compared the root-mean-square error of the reconstructed image by the HIM based on bilinear, bicubic, and B-spline through simulation in 2013 [30], and after that there has been no specific analysis of the differences between the SPM and the HIM in PPSDH.

Lateral spatial resolution is one of the main parameters of PPSDH. To solve the problem of limited lateral resolution caused by the decrease in the sampling frequency of the traditional SPM in practical applications, we propose an oversampled super-pixel method (OSPM), referring to the sensor-shifting implementation of pixel super-resolution (PSR) for on-chip lensless digital holographic microscopy (DHM) [33,34]. The proposed OSPM can not only achieve pixel resolution imaging in PPSDH, but also can be represented in the form of an HIM based on nearest-neighbor interpolation for comparing the differences between the SPM and HIM. In Section 2, the principles and methods of the proposed technique are explained. The experimental verification of the proposed technique and discussion of the results are shown in Section 3, which is followed by the conclusions in Section 4.

2. Principles and Methods

2.1. Hologram Recording

Figure 1 shows the optical schematic of the PPSDH system based on the Mach–Zehnder interference structure. A polarized He–Ne laser source (Thorlabs HNL020LB) with a wavelength of 632.8 nm, whose output amplitude is continuously adjusted by a polarizer, acts as the source beam for the proposed PPSDH system. The beam is spatially filtered and collimated to generate a uniform beam with a plane wavefront. The ratio of the p-polarized and s-polarized components of the source’s linearly polarized light is continuously adjusted by a half-wave plate (HWP). The reflected beam (s-polarized) from a polarizing beam splitter (PBS) illuminates the transparent object, and the illumination light is modulated by the complex amplitude of the object to form the object beam $u\left(x,y\right)$, while the transmitted beam (p-polarized) acts as the plane wavefront reference beam. The object diffracted beam and the reference beam are combined by a non-polarizing beam splitter (BS), and their polarization states become right-handed circular polarization and left-handed circular polarization, respectively, through a quarter-wave plate (QWP) with the fast axis oriented at −π⁄4. The polarization direction of the object diffracted beam $o\left(x,y\right)$ (right-handed) and the reference beam $r\left(x,y\right)$ (left-handed) at the plane of the monochromatic polarization camera (FLIR BFS-U3-51S5P-C) has the same rotational angular velocity, but in opposite directions. Thus, relative phase shifts of $\delta$ = 0, π/2, π, 3π/2 are formed between the object beam and the reference beam through a micropolarizer filter array with the optical axis direction of 0, π/4, π/2, 3π/4. Due to the circular arrangement of a set of 2 × 2 adjacent pixels with different polarization directions, the four sub-phase-shift holograms $I_0\left(x,y;0\right)$, $I_0\left(x,y;\pi /2\right)$, $I_0\left(x,y;\pi \right)$, $I_0\left(x,y;3\pi /2\right)$ in a spatially multiplexed hologram $I_0\left(x,y;\delta \right)$ (SMH) recorded by the polarization camera are also cross-distributed, as in a checkerboard [7,26,27,35,36].

2.2. Hologram Reconstruction

Figure 2 illustrates the SPM calculation procedure of image reconstruction for PPSDH [16,31,32]. Pixels with the same phase shifts are extracted to obtain four sparse sub-phase-shift holograms: $I_0\left(x,y;0\right)$, $I_0\left(x,y;\pi /2\right)$, $I_0\left(x,y;\pi \right)$, and $I_0\left(x,y;3\pi /2\right)$. The SPM obtains four aligned phase-shift holograms—$I\left(x,y;0\right)$, $I\left(x,y;\pi /2\right)$, $I\left(x,y;\pi \right)$, and $I\left(x,y;3\pi /2\right)$—by downsampling the sparse sub-holograms and discarding the blank pixels. The PS technique provides complex amplitude distribution of the object diffracted beam $o\left(x,y\right)$ at the plane of the image sensor, and the diffraction calculation of $o\left(x,y\right)$ is performed by the angular spectrum method (ASM) to obtain the reconstructed image of the object beam $u\left(x,y\right)$, using the following equations [26,37]:

$$o\left(x,y\right)=\left[I\left(x,y;0\right)-I\left(x,y;\pi \right)\right]+j\left[I\left(x,y;\pi /2\right)-I\left(x,y;3\pi /2\right)\right]$$

(1)

$$u\left(x,y\right)={ℱ}^{-1}\left\{ℱ\left[o\left(x,y\right)\right]exp\left[-jkz\sqrt{1-{\left(\lambda f_x\right)}^2-{\left(\lambda f_y\right)}^2}\right]\right\}$$

(2)

where $ℱ$ and ${ℱ}^{-1}$ denote the Fourier transform and inverse Fourier transform, respectively; k = 2π/λ is the wave number; λ is the wavelength of the illumination light source; z is the numerical diffraction propagation distance, which is equal to the recorded distance from the object to the sensor; and $\left(f_x,f_y\right)$represents the two-dimensional spatial frequency components.

The object diffracted field $o\left(x,y\right)$ calculated by the SPM can be equivalently regarded as the result of direct sampling of $o\left(x,y\right)$ by an imaginary light-field image sensor with a pixel pitch of $2∆$ and an image resolution of $N_x\times N_y/4$, where $N_x,N_y$ and $∆$represent the image resolution and the pixel pitch of the actual sensor, respectively. Considering the horizontal direction, a frequency sampling bandwidth of $B_{SPx}=1/2∆$ corresponding to the $2∆$ super-pixel (SP) sampling interval of the light-field sensor filters out high-frequency information beyond $B_{SPx}$ in the diffraction field, with a bandwidth of $B_x=N_x∆/\lambda z$: (1) when $B_{SPx}>B_x$, the imaging lateral resolution mainly depends on $B_x$; (2) when $B_{SPx}<B_x$, the lateral resolution mainly depends on $B_{SPx}$, and is not more than twice the sampling interval of the light-field sensor, which equals $4∆$ in the SPM. Therefore, the lateral resolution can be improved by reducing the sampling interval or increasing the sampling bandwidth of the imaginary light-field sensor.

To solve the above problem of the SPM, we propose an OSPM for PPSDH, referring to PSR for on-chip DHM [33,34]; Figure 3 depicts the proposed technique. As shown in Figure 3a, the OSPM calculates four object light diffraction field images by overlapping each independent SP, which is equivalent to four images captured after moving the imaginary light-field sensor in the horizontal and vertical directions by half an SP offset. After interpolating and merging four low-resolution images, a light-field image with an SP interval of $∆$ and an image resolution of $N_x\times N_y$ can be obtained. Figure 3b shows that the implementation of the OSPM for enhancing the sampling frequency of the light field can be simplified to perform nearest-neighbor interpolation on the extracted sparse phase-shift holograms $I_0\left(x,y;\delta \right)$ in the form of HIM, obtain four interpolated aligned phase-shift holograms $I\left(x,y;\delta \right)$, and calculate the object light field $u\left(x,y\right)$ according to Equations (1) and (2). Compared with the traditional numerical calculation implementation of interpolation [23,30], a simpler implementation can be represented by the convolution of an image and an interpolation kernel using the following equation [26]:

$$I\left(x,y;\delta \right)=I_0\left(x,y;\delta \right)\ast f\left(x,y\right)$$

(3)

where ∗ denotes the convolution operation, and $f\left(x,y\right)$ represents the interpolation kernel.

Table 1. The classical convolutional interpolation kernels [31].

Method	$\mathrm{Kernel}\mathrm{Function}f\left(x,y\right)=h{h}^T$	Size
0 elements	$h={\left( 1 \right)}^T$	$1\times 1$
Nearest-neighbor	$h={\left( 1 1 \right)}^T$	$2\times 2$
Bilinear	$h={\left( \frac{1}{2} 1 \frac{1}{2} \right)}^T$	$3\times 3$
Bicubic	$h={\left(-\frac{1}{16} 0 \frac{9}{16} 1 \frac{9}{16} 0-\frac{1}{16} \right)}^T$	$7\times 7$
B-spline	$h={\left(-\frac{3}{152} 0-\frac{9}{76} 0 \frac{91}{152} 1 \frac{91}{152} 0-\frac{9}{76} 0-\frac{3}{152}\right)}^T$	$11\times 11$

shows the classical convolutional interpolation kernels, including nearest-neighbor interpolation [31]. To reduce the calculation time, [26] proposed a fast image reconstruction technique based on the convolution theorem.

The proposed OSPM increases the frequency sampling bandwidth of the object light diffraction field from $B_{SPx}=1/2∆$ to $B_{OSPMx}=1/∆$, which is equivalent to increasing the imaging lateral resolution in the case of $B_{SPx}<B_x<B_{OSPMx}$. In addition, the maximum lateral resolution can also be increased from $4∆$ to $2∆$, which is twice the pixel pitch $∆$ of the actual image sensor. To further increase the sampling frequency, additional physical operations such as a source or sensor shifting are required [34]. However, due to the limited bandwidth $B_x$ of the diffraction field caused by the minimum recording distance, which is equal to the thickness of devices such as the BS in Figure 1, further PSR imaging in PPSDH has a smaller effect on the resolution improvement than the digital holographic microscopy [34,38].

Since the proposed OSPM can be regarded as the HIM based on nearest-neighbor interpolation, the image resolution and lateral spatial resolution of the reconstructed image of the HIM are not worse than those of the SPM. In addition, the nearest-neighbor interpolation—which may be ignored in PPSDH due to poor reconstruction accuracy—has a smaller kernel size and, thus, can retain more high-frequency information of the target, which perhaps corresponds to a higher lateral resolution.

3. Results and Discussion

The proposed OSPM was verified experimentally based on the PPSDH system shown in Figure 1. A negative test target USAF1951 (Thorlabs R1DS1N) was used as a recording object. The central 2048 × 2048 pixels of the FLIR polarization camera based on the IMX250MZR sensor were used for hologram recording. Figure 4a–e show the reconstructed enlarged amplitude images obtained using the conventional SPM, and Figure 4f–j show those obtained using the proposed OSPM at different recording distances z.

Among them, the reconstructed image resolutions by the SPM and OSPM are 1024 × 1024 and 2048 × 2048, respectively, while the lateral spatial resolution of the reconstructed images increases with decreasing z. The critical distances z beyond where the lateral resolution of the SPM and OSPM is limited by the sampling interval are 77 mm and 38.5 mm in the case of $B_x=B_{SPx}$ and $B_x=B_{OSPMx}$, respectively. The maximum lateral spatial resolution of the reconstructed images at recording distance z = 29.059 mm corresponds to group 6 element 3 (80.6-line pairs/mm, 12.4 µm) and group 7 element 1 (128.0-line pairs/mm, 7.8 µm) of the target. Thus, the proposed OSPM can simultaneously improve the image resolution and the maximum lateral spatial resolution of the reconstructed image.

Figure 5 shows the magnified amplitude images and Fourier spectral images of the normalized reconstructed images of the hologram with z = 29.059 mm, obtained using the HIM for comparing the effects of the interpolation algorithms. From the bottom to the top of Table 1, the sizes of the interpolation kernels are reduced from 11 × 11 to 1 × 1, and the attenuation effects of the corresponding low-pass filters on high-frequency information of the diffracted field $o\left(x,y\right)$ are gradually weakened. Thus, the lateral resolutions of the reconstructed $u\left(x,y\right)$ images based on B-spline, bicubic, bilinear, and nearest-neighbor interpolation are gradually improved, corresponding to group 6 element 4 (11.4 µm), element 5 (9.8 µm), and element 6 (8.8 µm), and group 7 element 1 (7.8 µm) of the target, respectively. However, the reconstructed image with zero interpolation is seriously degraded by the residual zero-order and twin-image noise. Thus, the proposed OSPM implemented by nearest-neighbor interpolation has the highest spatial lateral resolution of 7.8 µm.

The maximum lateral resolution of the proposed OSPM is more than the theoretical resolution of 2$∆$ = 6.9 µm, which may be an inherent error introduced by super-pixels. Figure 6 compares the reconstructed images of PSDH and PPSDH by numerical simulation. The simulation target was the processed reconstructed image of the R1DS1N with z = 29.059 mm, and the simulated pixel pitch was 3.45 µm, corresponding to group 7 element 2 (6.9 µm). The simulation results were in good agreement with the experimental results shown in Figure 5. Group 7 element 2 can be resolved in PSDH reconstructed images, but not in PPSDH, which is influenced by the residual zero-order and twin images in PPSDH.

The analysis of the above experimental results is mainly aimed at the target reconstructed image with z = 29.059 mm, which can highlight the differences between the proposed OSPM and SPM, between interpolation methods, and between PSDH and PPSDH. Our experimental results prove that the proposed OSPM can effectively improve the maximum imaging lateral resolution for PPSDH compared with the traditional SPM. In the OSPM, the nearest-neighbor interpolation with the smallest kernel size is used, which can retain more of the high-frequency information of the object while suppressing the residual zero-order and twin-image noise. Compared with classical interpolation methods, such as bilinear interpolation, the lateral resolution of the reconstructed image of OSPM can reach up to 7.8 µm.

The failure to achieve the ideal resolution of 6.9 µm that can be obtained by PSDH may be an inherent problem of PPSDH, caused by the residual zero-order and twin-image noise in Figure 5f. When the phase in the complex-valued test target that is different from the intensity modulation of USAF1951 changes drastically, the high-frequency information contained in the +1 order and −1 order images of the target is very strong. Since the frequency center of the −1 order image is located at the vertex of the spectrum rectangle, its strong high-frequency information is reflected in the low-frequency region, resulting in degradation of the reconstructed +1 order image. Therefore, the residual noise removal of PPSDH is one of the main problems to be solved in the future [39], and considering new sub-phase-shift hologram interpolation methods [17,40] or the interdependence between the four sub-phase-shift holograms [28,29] will be helpful to solve the above residual noise problem in PPSDH.

In addition, as shown in Figure 4, when the recording distance z is gradually reduced, the bandwidth $B_x$ of the diffracted light field at the detector plane is also gradually reduced, and the requirement for the sampling bandwidth of the sensor is also reduced accordingly. When $B_x$ ≤ $B_{SPx}$ ($z$ $\ge$ 77 mm), the OSPM and SPM will have similar spatial resolution, and an interpolation method with a larger kernel size can be used to reduce the root-mean-square error of the reconstructed image [30].

4. Conclusions

In this study, we propose an oversampled super-pixel image reconstruction method that can approximately achieve pixel resolution imaging, and the main reason for not reaching the ideal value is the inherent residual zero-order and twin-image noise of PPSDH. Compared with the existing image reconstruction methods in PPSDH, the proposed technique can retain more high-frequency information of +1 order images of the target, while suppressing the low-frequency components of the residual noise. However, the strong high-frequency components of the target may still lead to serious degradation of the reconstructed image quality; thus, the removal method of residual noise needs further research in the future.