A New Method to Retrieve the Three-Dimensional Refractive Index and Specimen Size Using the Transport Intensity Equation, Taking Diffraction into Account

Abstract

Refractive index retrieval is possible using the transport intensity equation (TIE), which presents advantages over interferometric techniques. The TIE method is valid only for paraxial ray assumptions. However, diffraction can nullify these TIE model assumptions. Therefore, the refractive index is problematic for reconstruction in three-dimensions (3D) using a set of defocused images, as diffraction effects become prominent. We propose a method to recover the 3D refractive index by combining TIE and deconvolution. A brightfield (BF) microscope was then constructed to apply the proposed technique. A microsphere was used as a sample with well-known properties. The deconvolution of the BF-images of the sample using the microscope’s 3D point spread function led to significantly reduced diffraction effects. TIE was then applied for each set of three images. Applying TIE without taking into account diffraction failed to reconstruct the 3D refractive index. Taking diffraction into account, the refractive index of the sample was clearly recovered, and the sectioning effect of the microsphere was highlighted, leading to a determination of its size. This work is of great significance in improving the 3D reconstruction of the refractive index using the TIE method.

1. Introduction

Spectroscopic analysis of microscopic specimens sometimes requires knowledge of their refractive indices. The refractive index is an important biophysical parameter, which is used in many studies. In mineralogy and crystallography [1], the refractive index is taken into account in the characterization of precious stones. In medicine, it can be used as a marker of disease. Zhuo et al. [2] showed that it is possible to diagnose tissue cancers via the refractive index distribution, which contains information about the molecular scale organization of tissue. The refractive index maps of red blood cells are used as an indicator of the morphological alterations of host cells infected by Plasmodium falciparum [3].

Standard devices for the refractive index measurement lead to mean values of this important parameter. Progress is being made in the development of new devices and methods that enable the determination of the refractive index in three-dimensions (3D). These methods should help to raise the accuracy of specimen analysis. The most well-established methods for recovering the quantitative phase and the optical parameters are the interferometric method, such as digital holography. Using this method, Florian et al. [4] performed a 3D reconstruction of refractive index of pollen grain, with a precision of 0.01. The interferometric method needs coherent illumination. Therefore, it induces speckle noise in the measurements, which prevent high quality results [5]. Phase or refractive index retrieval is possible by using the Transport Intensity Equation (TIE) which is a non-interferometric method [6]. TIE has been increasingly investigated over recent years due to its unique advantages over interferometric techniques. It uses partially coherent illumination; it is computationally simple with the need of (at least) three brightfield (BF) defocused images, and it does not require a complicated optical system. However, the TIE method is valid only for paraxial ray approximations [7], and diffraction can nullify these TIE model assumptions. This phenomenon drastically affects the quality of the 3D reconstruction of the refractive index [8] as the diffraction effects become prominent.

In this work, we introduce a method to clearly recover the 3D refractive index of a sample with well-known properties by using TIE and taking the diffraction into account.

2. Materials and Methods

2.1. Experimental Set-Up

In Figure 1, we can see the set-up of the experimental device. This consisted of a BF horizontal microscope with a 16 bit monochrome Complementary Metal-Oxide-Semiconductor (CMOS) camera with a pixel size of 5.5 µm × 5.5 µm, and an effective image chip size of 11.264 mm × 5.984 mm, presenting 2048 × 1088 pixels. The microscope enabled automated sequential acquisition of the BF images. The images were recorded by defocusing the sample slightly in the z-direction (optical axis) through a stepper motor MTS25-Z8. The motor was monitored in three directions (X, Y, Z) by three servocontrols. Therefore, we were able to scan the region of interest in the sample. The servocontrols were connected to the PC, which enables the control of the defocus distance Δz and the image acquisition protocol by using a MatLab operation code.

2.2. Sample Preparation

In this work, the sample was made up of 1 µm polystyrene microspheres beads (refractive index $n=1.586$ at wavelength $\mathsf{\lambda } = 640 \mathrm{nm}$ [9]) at the temperature $T=17$ °C. The solution of microspheres was diluted with distilled water ($n=1.33$ at the temperature $T=17$ °C). A small drop of the diluted solution was placed and spread on a slide made of the highest purity corrosion-resistant glass with dimensions 75 mm × 25 mm × 1 mm. After drying the sample for 30 minutes, it was used for microscopic examination under partially coherent illumination at a wavelength $\lambda = 640 \mathrm{nm}$.

2.3. Measurement Procedure

We performed the measurements in transmission mode. Three sets of images were recorded in the same conditions:

-

image T_d, the contribution of the background and the dark current in the detector; it was achieved with nothing in the microscope and with the source illumination turned off;

-

image T_b, the bright reference; it was achieved with an empty slide in the microscope;

-

image T_s, the raw image of the sample, it was achieved with the sample in the microscope.

Each set of images was recorded for each plane of the sample by moving it along the optical axis. The corrected image T of the sample is defined by Equation (1) (flat-field correction), as is commonly described by Tadrous [10], Brydegaard et al. [11], and Agnero et al. [12]:

$$\mathit{T}=\frac{{\mathit{T}}_{\mathit{s}}-{\mathit{T}}_{\mathit{d}}}{{\mathit{T}}_{\mathit{b}}-{\mathit{T}}_{\mathit{d}}}$$

(1)

This operation cancels out the different emissive yields of the light emitting diode (LED) source, electronic gains, and exposures for each illumination, and also the variation in the illumination intensity over the field of view [11].

In practice, one major disadvantage of TIE method is the sequential capture of images. It requires the precise motion of the sample, which can be shifted from a plane to another due to a little vibration in the measurement environment. This induces artefacts in some of the recorded images [13]. Therefore, care was taken to avoid any movement that was able to produce vibration as the image acquisition was running.

2.4. Analysis Methods

Optical systems generally are not free of aberrations or imperfections that affect the quality of the recorded images. Consequently, the image of a point object is an extended distribution of intensity called point spread function (PSF). During data analysis, we firstly sought to remove from the corrected images the distortions of observation due to the optical acquisition system and the contributions from out-of-focus planes for the sample. In this approach, we introduced the model of an appropriate PSF required for a deconvolution, which should remove any artefact. At the second time, we used the TIE method to extract the refractive index and the size of the specimen.

Deconvolution requires a realistic model of the phenomena that induce distortions in the image formation process. The term “deconvolution microscopy” generally assumes fluorescence microscopy [14,15], and PSF deconvolution imaging theory has traditionally been linked to fluorescence. A main reason behind this association is that for the case of self-luminous objects, one needs only to consider signal intensity, leading to a unique PSF and making deconvolution a linear process. In general, this is not the case in BF microscopy, where two point spread functions are needed to describe image formation [16]. In fluorescence microscopy, the 3D image $\mathit{i}\left(x, y, z\right)$ of an object corresponds to the convolution of the object $\mathit{o}\left(x, y, z\right)$ with the PSF(x, y, z) [17]:

$$\mathit{i}=\mathit{o}\otimes PSF+\mathit{B}$$

(2)

where B models noise, and the symbol ⊗ denotes the convolution operation. BF microscopy is presented as an alternative to fluorescence in deconvolution image processing, because of the possibility for observing unstained objects. However, its corresponding PSF consists of two separate components, one for phase ($PS{F}_p$) and one for absorption ($PS{F}_a$). Therefore, the BF-3D image $\mathit{i}\left(x, y, z\right)$ is generally described as the sum of the convolutions of the real (P) and imaginary (A) parts of the object scattering potential with the corresponding PSFs [18]:

$$\mathit{i}=\mathit{P}\otimes PS{F}_p+\mathit{A}\otimes PS{F}_a+\mathit{B}$$

(3)

This excludes the application of linear deconvolution processing to remove distortion. However, in the presence of the pure phase object as it is the case in this work, BF imaging essentially reduces to Equation (2), making it possible to perform linear deconvolution [16]. Therefore, knowing the PSF leads to the determination of the object through deconvolution. The PSF can be:

-

computed from the optical properties of the microscope system,

-

estimated from the measurements of the microspheres.

The experimental PSF takes into account all of the aberrations introduced by the whole image acquisition system [19]. It is determined by recording images of a specimen (assumed to be a point object) in different planes of defocus above and below the focal plane. A point object does not exist in practical applications; the object necessarily has a particular size. Therefore, for the measured data to correspond to the microscope’s PSF, it is very important that the object size is smaller than the diffraction limit. If this is not the case, the measured data will instead correspond to the convolution of the PSF with the object shape, according to Equation (2) [20]. Unfortunately, using a very small object introduces the problem of signal detection, due to a very low signal-to-noise ratio. This requires detectors of high sensitivity. The signal to noise ratio is so low that often a bead is used whose size is not smaller than diffraction limit, but of the order of the microscope’s resolution [21]. This leads to an overestimated PSF. The experimental PSF is noisy and needs to be devoid of noise, which is a limitation in a deconvolution algorithm. The method of denoising sometimes distorts the real PSF of the microscope. Lai et al. [22] used a denoising method based on singular value decomposition to get a denoised PSF, but it produced small visible artificial ripples to the denoised PSF [19].

The computed PSF, in particular the model of Gibson and Lanni [23], also takes into account the conditions in which the experiment is carried out, and it has an advantage of being estimated for all of the object space. It is free of noise, flexible, and suitable to any specific condition of the experimentation, unlike experimental PSF, which is not modifiable because it is measured to represent a specific condition in which the images of biological sample must be recorded. The BF-PSF model was reported previously by Hernandez and Gutierrez [16]. However, they do not assume a real point object in their model. The model of Gibson and Lanni [23] is more an accurate model for PSF generation that is suitable for the brightfield setting [10], and this was discussed and publicly made available as an ImageJ plug-in by Besson et al. [24]. According to the model, the image of a point object located at $X_p=\left(x_p, y_p, z_p\right)$ is defined by Equation (4) [23]:

$$PSF\left(X,X_p\right)={\left|A{\int }_0^1{e}^{i w\left(\rho , X, X_p\right)} J_0\left(\frac{2\pi }{\lambda } r NA \rho \right)\rho d\rho \right|}^2$$

(4)

where $X=\left(x, y, z\right)$ is a point on the image plane, $NA$ is the numeric aperture of the objective, $A$ is an amplitude constant, $\rho$ is the radius of the microscope’s limiting aperture in the microscope’s back focal plane, $J_0$ denotes the Bessel function of the first kind of order zero, and $w\left(\rho , X, X_p\right)$ is the phase which enables aberrations to be taken into account in the model, $r=\sqrt{{\left(x-x_p\right)}^2+{\left(y-y_p\right)}^2}$. The phase $w\left(\rho , X, X_p\right)$ is defined by Equation (5), as there are two layers (immersion medium of refractive index $n_c$ and the specimen layer of refractive index $n_m$) [25]:

$$\begin{array}{cc}\hfill w\left(\rho , X, X_p\right)& =k[z_p\sqrt{n_m{}^2-N{A}^2{\rho }^2}-WD\sqrt{n_{c^{\ast }}{}^2-N{A}^2{\rho }^2}\hfill \\ & +\left(z_p-z_n+n_c\left(\frac{WD}{n_{c^{\ast }}}-\frac{z_p}{n_m}\right)\right)\sqrt{n_c{}^2-N{A}^2{\rho }^2}]\hfill \end{array}$$

(5)

The parameters $n_{c^{\ast }}$ and $WD$ are the design refractive index of the immersion medium and working distance of the objective, respectively, and $k$ is the wave number. The model of Gibson and Lanni [23] allowed for the restoration of our BF images.

Defining the wave field traversing the specimen by $\underset{\_}{E_z}\left(x,y\right)=\sqrt{I_z\left(x,y\right)}{e}^{-j{\varphi }_z\left(x,y\right)}$, where $I_z\left(x,y\right)$ is the intensity, ${\varphi }_z\left(x,y\right)$ the phase, $z$ denotes the position along the optical axis, and $\left(x,y\right)$ are the two-dimensional coordinates in the plane orthogonal to the optical axis. Therefore, the denotation $I_z\left(x,y\right)$ indicates the intensity $I\left(x,y,z\right)$. The TIE is obtained by substituting $\underset{\_}{E_z}\left(x,y\right)$ into the paraxial wave equation and taking the imaginary part of the resulting expression; the TIE is defined by [6]:

$$\frac{\partial I_z\left(x,y\right)}{\partial z}=-\frac{\lambda }{2\pi }{\nabla }_{\perp }·\left(I_z\left(x,y\right){\nabla }_{\perp }{\varphi }_z\left(x,y\right)\right)$$

(6)

where $\mathsf{\lambda }$ is the wavelength of the illumination source and ${\nabla }_{\perp }$ denotes the gradient operator in the lateral dimensions $\left(x,y\right)$. Due to its non-interferometric nature, the illumination can be quasi-monochromatic and partially-coherent [26]. The numerical solution for the TIE in the case of a pure phase object (transparent object), is defined by Equation (7) [7], resulting from the standard equation of Poisson [27,28]:

$${\varphi }_z\left(x,y\right)=\mathit{T}{\mathit{F}}^{-1}\left[\frac{\mathit{T}\mathit{F}\left[-\frac{2\pi }{\lambda } \frac{1}{I_z\left(x,y\right)} \frac{\partial I_z\left(x,y\right)}{\partial z}\right]}{4{\pi }^2\left(u^2+{\nu }^2\right)}\right]$$

(7)

where $\mathit{T}{\mathit{F}}^{-1}$ and $\mathit{T}\mathit{F}$ denote the inverse Fourier Transform and Fourier Transform respectively, and $\left(u,\nu \right)$ denotes the spatial frequency variables corresponding to the coordinates $(x,y$). Equation (7) is valid only for paraxial ray assumptions. The phase of the wave field is related to the variation of the image intensity along the optical axis through Equation (7). In practice, the intensity derivative $\partial I_z\left(x,y\right)/\partial z$ in the observation plane $z$ cannot be measured directly; we adopted the following approximation (Equation (8)) using two defocused images, one above the observation plane ($I\left(x,y,z+\Delta z\right)$) and one below ($I\left(x,y,z-\Delta z\right)$):

$$\frac{\partial I_z\left(x,y\right)}{\partial z}\approx \frac{I\left(x,y,z+\Delta z\right)-I\left(x,y,z-\Delta z\right)}{2·\Delta z}$$

(8)

where $\Delta z$ is the defocus distance. Phase distortions induced in the wave field as the wave propagates from point $r_{z_0}\left(x,y\right)$ to $r_z\left(x,y\right)$ can be related to the refractive index of the specimen through solutions to the eikonal equation [8,29] for the phase:

$${\varphi }_z\left(x,y\right)-{\varphi }_{z_0}\left(x,y\right)=\frac{2\pi }{\lambda }{{\displaystyle \int }}_{z_0}^z\Delta n_{z^{\prime }}(x, y)d{z}^{\prime }$$

(9)

$\Delta n_z\left(x, y\right)$ is the refractive index difference between the specimen and its surrounding medium (specimen layer). Taking the specimen to be located directly above the origin $r_{z_0}\left(x,y\right)$, hence ${\varphi }_{z_0}\left(x,y\right)=0$, and for points inside the specimen, it is found that [8]:

$${\varphi }_z\left(x,y\right)=\frac{2\pi }{\lambda }{\int }_0^z\Delta n_{z^{\prime }}(x, y)d{z}^{\prime }$$

(10)

Therefore, if ${\varphi }_z\left(x,y\right)$ is known in multiples planes (with small value of defocus distance $\Delta z$) inside the specimen, its 3D refractive index distribution can be recovered via Equation (11) [30] through sequential method (using finite differences) applied to pairs of consecutive planes (one pair after one sequentially) with the same value of defocus distance:

$$n_z\left(x,y\right)=n_m+\frac{\lambda }{2\pi } \frac{\partial {\varphi }_z\left(x,y\right)}{\partial z}$$

(11)

where $n_m$ is the refractive index of the specimen layer. To retrieve the 2D phase ${\varphi }_z\left(x,y\right)$ corresponding to the observation plane $z$, we used Equation (7) and three recorded images of the sample were needed, one in the observation plane, and the both of the others below and above the observation plane with a defocus distance $\Delta z$. This procedure was repeated to obtain ${\varphi }_z\left(x,y\right)$ in multiples planes by changing the observation plane with the same value of $\Delta z$ until the whole specimen was scanned along the optical axis. The defocus distance is an important factor in the quality of the reconstruction. When $\Delta z$ was large, the hypothesis of linearity that enables the above approximation (Equation (8)) of the intensity derivative $\partial I_z\left(x,y\right)/\partial z$ was not valid, and this nonlinearity affected the accuracy of the reconstruction. For small values of $\Delta z$ where the linearity was preserved, there was a reduction of signal to noise ratio due to the term $I\left(x,y,z+\Delta z\right)-I\left(x,y,z-\Delta z\right)$, which becomes smaller. This required a compromise between the defocusing, accuracy, and signal to noise ratio. Therefore, when we are in the presence of multiple plane images and we would like to retrieve the 2D phase $\varphi \left(x,y\right)$ corresponding to the focal plane, we cannot simply use pairs of plane images to recover the phase from average of them [13]. In this work, we used the method proposed by Zhong et al. [31], which performs fitting (through Gaussian process regression) on each pixel’s intensity in the frequency domain along the optical axis, taking into account all the images in different planes. This method allowed an estimation of the derivative $\partial I_z\left(x,y\right)/\partial z$, which led us to retrieve the phase in the focal plane. Consequently, the specimen’s size $t\left(x,y\right)$ was obtained by using Equation (12) [32]:

$$t\left(x,y\right)=\frac{\lambda \varphi \left(x,y\right)}{2\pi \left(n_f-n_m\right)}$$

(12)

where $n_f$ is the refractive index of the specimen in the focal plane.

3. Results and Discussion

During the computation of the PSF for our experimental device, two parameters were not directly accessible: the depth $z_p$ of the point object in the specimen layer, and the refractive index $n_c$ of the immersion medium (air) during the experimentation [25]. Our sample of microspheres of diameter 1 µm was allowed to dry during experimentation. This enabled us to assume that the point object was on the interface of air and specimen layer, so $z_p$ = 0. The measured pressure and temperature in the measurement environment ($P=992 \mathrm{hPa}$, $T=17$ °C) enabled the determination of the refractive index, $n_c$, of the immersion medium (air) during experimentation; $n_c=1.00024$ by using Equation (13) [33,34]:

$$n_c=1+\left(6.4328·{10}^{-5}+\frac{2.94981·{10}^{-2}}{146-{\lambda }^{-2}}+\frac{2.554·{10}^{-4}}{41-{\lambda }^{-2}}\right)\frac{15·P}{T}$$

(13)

where $P$, $T$, and $\lambda$ are in bar, in degrees Celsius and in $\mathsf{\mu }\mathrm{m}$ respectively. Figure 2 presents the PSF of our device. The images of the sample in different planes were affected by the contribution of out-of-focus planes (Figure 3a–c). Restoration of these images was achieved by using the Richardson–Lucy algorithm [35,36] implemented in MatLab, taking into account the 3D-PSF. The number of iterations used was 25. This led to a reduction not only in the effects of diffraction but also in the noise and pixel intensity, due to the contribution of the adjacent planes (Figure 3d–f).

Applying the TIE method sequentially on the corrected images led to an extracted refractive index of the specimen (Figure 4a–g). We can observe in Figure 4a–g that the outlines of the microsphere became smaller as we moved away from the focal plane ($z=0 \mathsf{\mu }\mathrm{m}$). This was due to effect of sectioning. In a plane of the microsphere, the spatial distribution of its refractive index was not homogeneous. The homogeneity of refractive index distribution was drastically damaged as we moved away from the focal plane. This was due to diffraction effects [37] that became prominent. In this case, the paraxial ray assumptions, which were required to apply TIE, were nullified [8]. This induced artefacts in the refractive index reconstruction (Figure 4a–c). TIE method drastically failed to recover the 3D refractive index distribution of the sample for an optical system with a high numeric aperture, as was the case for our device ($N{A}_{ob}=0.75$) [37].

A reconstruction of the refractive index after the deconvolution of the corrected images by the use of the 3D-PSF is presented in Figure 4h–n. The sectioning effect was more greatly appreciated through the spatial distribution of refractive index. The circular outlines of the microsphere were well highlighted. The artefacts that were observed in the planes $z=-0.4 \mathsf{\mu }\mathrm{m}$, $z=-0.3 \mathsf{\mu }\mathrm{m}$, $z=-0.2 \mathsf{\mu }\mathrm{m}$ were removed. In each plane of the microsphere, the homogeneity of refractive index distribution was greatly improved and tended to cover all of the circular outline of the microsphere, mostly in the focal plane.

Table 1. Mean values of the refractive index of the microsphere at the wavelength 640 nm for each plane Z (μm).

	Z = −0.4	Z = −0.3	Z = −0.2	Z = −0.1	Z = 0	Z = 0.1	Z = 0.2
Refractive index before deconvolution	1.444	1.460	1.526	1.512	1.556	1.558	1.515
Refractive index after deconvolution	1.584	1.585	1.587	1.586	1.587	1.585	1.586

presented a comparison between the mean values of the microsphere’s refractive index in the presence of the diffraction, and in the absence of diffraction. A mean value of the refractive index for the microsphere was an average over all the points inside the microsphere that corresponded to a plane. Before the deconvolution (presence of diffraction), the mean value of the refractive index seemed to be a function of the z-position. We mainly observed a decrease of refractive index values while we moved away from the focal plane. This was due to the diffraction dependence along the optical axis, which becomes more significant as we moved away from the focal plane. From $z=-0.1 \mathsf{\mu }\mathrm{m}$ to $z=-0.2 \mathsf{\mu }\mathrm{m}$, an increase occurred. This could explain an effect of the vibration which occurred during the measurement. This induced the sample to be shifted from a plane to another during the measurement. This was a drawback for the TIE method, which required the precise motion of the sample. After the deconvolution (taking into account 3D-PSF), the mean value of the refractive index was quasi-constant. The value in the focal plane where diffraction effects are greatly reduced was also in agreement with the refractive index of $1.586$, which is the reference value [9] at $640 \mathrm{nm}$. In Figure 5, we can see the size’s map $t\left(x,y\right)$ of the specimen. It was obtained by using Equation (12), taking $n_f=1.587$ as the mean value of microsphere’s refractive index in the focal plane and $n_m=1.33$ as the refractive index of the distilled water used in the sample preparation. The microsphere’s diameter as deduced from the map was $d=0.943 \mathsf{\mu }\mathrm{m}$. This value represents the intensity of the pixel in the middle of the microsphere (Figure 5). The graph for the mean values of the refractive index as a function of z-position also led to the recovery of the diameter of the microsphere. The diameter was defined by the full width at half maximum (FWHM) indicated in the graph by the double arrow (Figure 6). The diameter, which was deduced from the graph, is $1.1 \mathsf{\mu }\mathrm{m}$. These two diameter values were close to $1 \mathsf{\mu }\mathrm{m}$ (the reference diameter of the microsphere used for the microscopic examination in this work), and they were in relative variation, between $6\%$ and $10\%$.

4. Conclusions

In this work, we introduced a method to reconstruct the 3D refractive index distribution by combining TIE and deconvolution. A BF microscope using a camera was constructed, and the proposed technique was applied for a technical sample with well-known properties. The microscope’s 3D-PSF, which takes into account optical aberrations, was computed. Its influence on the 3D refractive index recovery, using the TIE method, was demonstrated. Experimental results show that taking the diffraction into account while performing TIE on BF-images can significantly improve the accuracy of the 3D refractive index reconstruction. It was demonstrated that the method is able to infer the size of the sample accurately. Therefore, this approach was used in our laboratory to extract the optical parameters. The proposed technique can be applied, not only to homogeneous specimens, but also to specimens presenting heterogeneous compositions in their structures, such as biological cells. However, they must be pure phase objects, where it is possible to perform linear deconvolution according to Equation (2); otherwise the proposed method cannot be used.