A Novel Demodulation Algorithm Based on the Spatial-Domain Carrier Frequency Fringes Method

Abstract

Structured illumination microscopy (SIM) has attracted much attention from researchers due to its high accuracy, high efficiency, and strong adaptability. In SIM, demodulation is a key point to recovering three-dimensional topography, which directly affects the accuracy and validity of measurement. The traditional demodulation methods are the phase-shift method and Fourier transform method. The phase-shift method has a high demodulation accuracy, but its time consumption is too long. The Fourier transform method has high efficiency, but its demodulation accuracy is lower due to the loss of high frequency information during the process of filtering. However, in actual measurement, due to the gamma effect of the projector and charge-coupled device (CCD), the phase-shift interval is not strictly equal to the default value, which causes phase-shift error. Therefore, the restored topography contains carrier frequency fringes, which affects the accuracy of the measurement and limits the wide application of SIM. In this paper, a novel demodulation algorithm based on spatial-domain carrier frequency shift is proposed to solve the problem. Through recombining multiple full-period phase-shift images, the error spectrum and the signal spectrum are separated from each other in the frequency domain, so as to eliminate the effect of carrier frequency fringes. Simulations and experiments are carried out to verify the feasibility of the proposed method.

1. Introduction

In recent decades, micro–nano three-dimensional surface topography detection has attracted much attention from related researchers due to the surface topography of micro–nano-structure devices directly determining the physical and chemical properties of the devices [1,2,3].

Optical measurement methods have become the mainstream micro–nano three-dimensional surface topography detection method due to its non-contact, high-precision in recent years, which include laser confocal scanning microscopy, white light interference microscopy, and structured illumination microscopy, etc. [4,5,6,7]. In particular, SIM has been widely applied in material sciences, biomedicine, and micro-electronics manufacturing due to its high precision, high efficiency, and strong adaptability. The basic theory of SIM is to project a sinusoidal fringe pattern on the surface of the sample, and the height information of the surface will be modulated by the fringe. Simultaneously obtaining the surface modulation in the images captured by CCD while the sample is scanned through the focus of the objective lens. The core step of SIM is to demodulate the modulation distribution from the raw images, then combine the scanning step interval and peak extraction algorithm, so the three-dimensional surface topography of the sample can be recovered [8,9,10,11].

At present, the phase-shift method and Fourier transform method are most usually utilized to demodulate the modulation distribution. The phase-shift method calculates the modulation distribution through a set of full-period phase-shift fringe patterns, which has high demodulation accuracy, but the demodulation efficiency is low due to the need for capturing multiple images. The Fourier transform method achieves the demodulation by extracting +1 order signal spectrum in the frequency domain, which has high demodulation efficiency due to only one fringe image being needed to complete the process. Meanwhile, some high frequency information is lost during filtering, which causes the demodulation accuracy to decrease. In actual measurement, due to the gamma effect of the projector and CCD, the captured fringes are not sinusoidal, which causes an error spectrum to appear around the signal spectrum. Therefore, the restored topography contains carrier frequency fringes, which affects the accuracy of the measurement. Furthermore, the phase-shift method and Fourier transform both cannot eliminate the effect of carrier frequency fringes, which limits the application of SIM [12,13,14,15].

In the last several decades, the traditional methods to solve the gamma effect mainly include the phase-compensation method, the gamma calculation method, the gamma pre-correction coding method, and the lens-defocusing method. The phase-compensation method can obtain the phase distribution before distortion by setting up the phase error lookup table and compensating for the phase caused by the distortion. The calculation amount of this method is large, and many groups of pre-measurement are needed to determine the error lookup table in the actual application process. Both the gamma calculation method and the gamma pre-correction coding method solve the gamma value of the whole system through the model to complete the compensation. Compared with the gamma calculation method, the model of the gamma pre-correction coding method is more complex and closer to the real situation, but its calculation process will take more time. Similarly, both methods also require extensive measurements up front to build a high-precision gamma model of the system. The lens-defocusing method is to make a CCD camera out of focus; the slight defocus will play the role of superimposing a low-pass filter, which can eliminate the influence of higher harmonics to a certain extent, but this method will also lose the high frequency information in the topography, which is not suitable for a high-precision phase-reconstruction field. At present, most of the mainstream methods to eliminate the gamma effect are the form of compensation after calculating the gamma value of the whole system through the model in advance, and only when a system parameter is solidified can it be calculated; the method is not universal. Moreover, in the process of solving the model, multiple images need to be collected for calculation, and the data processing time is long, and the efficiency is low. And this is also added to the introduction [16,17,18,19,20,21,22,23,24,25,26].

To resolve the problem, a novel demodulation algorithm based on spatial-domain carrier frequency shift is proposed in this paper, whose schematic is shown in Figure 1a. Figure 1b is the schematic of the SIM measurement system. On the basis of the traditional N-step phase-shift method, rearranging the N images of each scanning position to generate a spatial-domain carrier frequency fringe (SCFF) image, as illustrated in Figure 2. Due to the introduction of a phase-shift frequency 1/N in the SCFF image, the error spectrum and the +1 order signal spectrum are separated from each other during shifting. Subsequently, a filter window with a suitable size is used to extract the signal spectrum, which can achieve demodulation without the effect of the error spectrum. Compared with the traditional phase-shift method, the proposed method can eliminate the effect of phase-shift error on measurement, which increases the measurement accuracy and validity. And compared to the Fourier transform method, because the error spectrum moves away from the signal spectrum, the filter window can take a larger size. The larger the filter window, the more high-frequency information obtained, which means better measurement accuracy. Simulations and experiments are both carried out to verify the feasibility of the proposed method in eliminating the gamma effect compared with the traditional demodulation method.

2. Materials and Methods

As shown in Figure 1b, the SIM measurement system is mainly composed of a digital micromirror device (DMD) which is used to project the fringe pattern, a microscope objective lens, and a CCD. Firstly, assume that a sinusoidal pattern is projected on the sample, the reflected light intensity distribution can be noted as follows:

$$I\left(x,y\right)=I_0+C\left(x,y\right)\mathit{cos}\left(2\pi f_{0}x+{\phi }_0\right)$$

(1)

where $I_0$ is the background intensity, $C\left(x,y\right)$ denotes the modulation of the projected fringe pattern, $f_0$ is the spatial frequency of the projected fringe pattern, and ${\phi }_0$ is the constant which presents the initial phase of the fringe pattern.

When considering the influence of the gamma effect on the measurement system, there is an index γ between the light intensity actually received by the CCD and the theoretical value, which can be expressed as follows:

$$I_{real}\left(x,y\right)={I\left(x,y\right)}^{\gamma }$$

(2)

where γ is the gamma value of the measurement system, usually, $1\le \gamma \le 3$.

In order to verify the ability of our proposed method in eliminating phase-shift error caused by gamma effect, assume that $\gamma =3$ in the simulation, as shown in Figure 3a.

We applied Fourier transform in Figure 3a; the corresponding spectral distribution is shown in Figure 3b.

It can be seen from Figure 3b, due to the gamma effect, that there are not only the back intensity spectrum and the signal spectrum in the spectrum, but also many high order harmonics, which are close to the signal spectrum and will affect the subsequent demodulation process.

In the process of demodulation, we applied the traditional phase-shift method to demodulate Figure 3a; the corresponding modulation distribution is shown in Figure 4a. It can be seen from Figure 4a that the modulation distribution obtained by the traditional phase-shift method contains carrier frequency fringes, which will appear in the subsequent recovery morphology and affect the measurement accuracy. And when it utilizes the Fourier transform method to demodulate, the size of the filter window should be set to a suitable size due to the high-order harmonics being very close to the signal spectrum. If the filter window is large, the influence of high-order harmonics and the signal spectrum are filtered out together, and the influence of high-order harmonics on the demodulation result cannot be excluded, as shown in Figure 4b; if the filter window is small, some useful high-frequency information will be lost, as shown in Figure 4c. Therefore, these two traditional demodulation methods cannot exclude the influence of carrier frequency fringes on the measurement result under the condition of considering the gamma effect, which will cause inaccurate measurement results.

In the proposed method, according to the rearrangement method illustrated in Figure 2, the images generated by the N-step phase-shift method are recombined into a new SCFF image. The SCFF image constructed by column is expanded by N times in the x direction, then its expression can be derived from Equations (1) and (2), which can be expressed as follows:

$$I_{SCFF}\left(x^{\prime },y\right)={\left\{I_0\left(INT\left({\displaystyle \frac{x^{\prime }}{N}}\right),y\right)+C\left(INT\left({\displaystyle \frac{x^{\prime }}{N}}\right),y\right)cos\left[2\pi f_t{x}^{\prime }+2\pi f_{0}INT\left({\displaystyle \frac{x^{\prime }}{N}}\right)+{\phi }_0\left(INT\left({\displaystyle \frac{x^{\prime }}{N}}\right),y\right)\right]\right\}}^{\gamma }$$

(3)

Convert it into a combination of continuous and periodic functions, which can be denoted as follows:

$$I_{SCFF}\left(x^{\prime },y\right)={\left\{I_0\left(INT\left({\displaystyle \frac{x^{\prime }}{N}}\right),y\right)+C\left(INT\left({\displaystyle \frac{x^{\prime }}{N}}\right),y\right)cos\left[2\pi \left(f_t+f_s\right)x^{\prime }+{\phi }_0\left(INT\left({\displaystyle \frac{x^{\prime }}{N}}\right),y\right)-2\pi f_{s}MOD\left({\displaystyle \frac{x^{\prime }}{N}}\right)\right]\right\}}^{\gamma }$$

(4)

where $f_s=f_0/N$ is the expression form of the spatial frequency in the SCFF image after the spatial frequency of a single fringe image is expanded by N times, $f_t=1/N$ represents the normalized spatial frequency of SCFF image, and $MOD$ is the operation of taking the remainder.

The SCFF image of Figure 3a is shown in Figure 5a, and Figure 5b is the frequency spectrum distribution of Figure 5b.

It can be seen from Figure 5b that the frequency spectrum of the SCFF image is shifted, which makes the high-order harmonics and signal spectrum be separated from each other. During subsequent processing, a larger filter window can be applied to extract the signal spectrum, which can retain more high-frequency information while eliminating the influence of high-order harmonics and improving the demodulation accuracy. The modulation distribution of Figure 5a is shown in Figure 6a.

The modulation distribution of the certain axial scanning position can be obtained by extracting one column every N columns in Figure 6a, as shown in Figure 6b.

According to the proposed demodulation method, the modulation distribution of each axial position can be obtained, and the axial modulation response curve of each pixel of the measured surface is obtained.

Finally, it is combined with the peak extracting algorithm to find the real axial position corresponding to the maximum modulation and utilizes $h=z_{real}\times ∆z$ to achieve the 3D topography recovery.

3. Results

3.1. Simulation

In order to verify the feasibility of the proposed method, a simulation has been carried out based in MATLAB. The simulated sample is a stage with two discontinuous steps, whose heights from the bottom to top are 70 nm and 120 nm, respectively. In the simulation, a four-step phase fringe pattern is projected on the sample and images are captured simultaneously.

The captured image by CCD at the scanning position of 120 nm is shown in Figure 7a, while the frequency spectrum is shown in Figure 7b. It can be seen from Figure 7b that the signal spectrum is close to the high-order harmonics; it is difficult to obtain as much high-frequency information as possible when avoiding extracting high-order harmonics.

In order to illustrate the advantages of the SCFF method, we compared the traditional demodulated method, the applied traditional phase-shift method, the Fourier transform method, and the SCFF method by demodulating the simulation data. Figure 8a shows the recovered topography by traditional phase-shift method; the corresponding cross-line in X = 251 pixel is shown in Figure 8b. Figure 8c illustrates the recovered topography by traditional Fourier transform method when the filter window is 10 × 10; the corresponding cross-line in X = 251 pixel is shown in Figure 8d. Figure 8e illustrates the recovered topography by traditional Fourier transform method when the filter window is 20 × 20; the corresponding cross-line in X = 251 pixel is shown in Figure 8f. It can be seen that the recovered topography by the traditional phase-shift method contains carrier frequency fringes. The Fourier transform method with the 10 × 10 filter window eliminates the carrier frequency fringes, but it loses a lot of high-frequency information at the same time, so that the recovered step is not steep. The Fourier transform method with the 20 × 20 filter window restores the topography and has a steeper step, but the carrier fringes could not be eliminated.

The simulated SCFF image is shown in Figure 9a, and the corresponding spectrum distribution is shown in Figure 9b. It can be seen from Figure 9b that the higher harmonics are far apart from the signal spectrum in the frequency domain; therefore, a large filter window can be applied in extracting the signal spectrum without worrying about the influence of introducing high-order harmonics, and more high-frequency information can be retained while ensuring the demodulation accuracy.

We applied the Fourier transform method with different filter windows to demodulate, for example, 10 × 10, 20 × 20, and 30 × 30; the demodulation results are shown in Figure 10. It can be seen from Figure 8 and Figure 10 that the SCFF image can utilize a larger filter window without carrier frequency fringes, which means the proposed method can retain more high-frequency information on the basis of eliminating the carrier frequency fringes and improving the demodulation accuracy.

3.2. Experiments

In order to further verify the reliability and stability of the proposed method in actual measurement, experiments have been carried out. The experimental system is demonstrated in Figure 11; it is composed of a white light source whose central wavelength and band width are 580 nm and 160 nm, respectively, a DMD with a size of 1920 × 1080 pixels and a pixel interval of 7.56 μm, two tube lenses, a splitter mirror, a microscopic objective (Olympus, Tokyo, Japan, 100×, NA = 0.9), a PZT scanning stage which has a resolution of 0.5 nm, and a CCD containing 2048 × 1536 pixels and a pixel size of 3.45 μm × 3.45 μm. In the experiment, a set of eight-step phase-shift fringes whose periods are 8 pixels is utilized to project the fringe pattern, the interval scanning length is 50 nm, and the total scanning steps are 80.

In this experiment, a step surface is used as the sample whose absolute height is 138 nm, as shown in Figure 12.

At the scanning position Z = 2000 nm, eight full-period phase-shift fringe images are utilized as the demodulation object; we applied the phase-shift method, the Fourier transform method, and the proposed SCFF method to demodulate them.

In addition, in order to quantitatively analyze the improvement of the demodulation system’s accuracy by the proposed method compared with the Fourier transform method and the phase-shift method, two evaluation parameters are provided to evaluate the demodulation system results. Firstly, in the relatively flat area of the tested samples, root-mean-square-error (RMSE) was used for evaluation: $RMSE=\sqrt{{\displaystyle \frac{\sum _{i=1}^n{\left(X_i-X_{mean}\right)}^2}{n}}}$. Secondly, SLOPE was used for evaluation in the region where the tested sample changes sharply: $SLOPE={\displaystyle \frac{Y_2-Y_1}{X_2-X_1}}$. The larger the RMSE, the greater the gamma effect on the demodulation results. The larger the SLOPE, the more high-frequency information contained in the demodulation result

The demodulation results of the phase-shift method are shown in Figure 13. It can be seen that the demodulation result contains carrier frequency fringes, and the amplitude of these carrier frequency fringes varies greatly, which affects the subsequent topography recovery. Calculated using the data in Figure 13, RMSE = 0.0671@(101~300 pixels) and SLOPE = 0.0332@(250 ± 5 pixels). The assessment is carried out within a certain pixel range to exclude edge ringing effects. Therefore, the phase-shift method cannot eliminate the influence of carrier frequency fringes.

The demodulation results of the Fourier transform method with a 15 × 15 filter window are shown in Figure 14. There are no obvious carrier frequency fringes in Figure 14b. However, it is not steep at the step, as shown in Figure 14c. Calculated using the data in Figure 14, RMSE = 0.0105@(101~300 pixels) and SLOPE = 0.0109@(250 ± 5 pixels).

The demodulation results of the Fourier transform method with an 18 × 18 filter window are shown in Figure 15. It can be obviously seen that carrier fringes are in the demodulation results. Calculated using the data in Figure 15, RMSE = 0.1876@(101~300 pixels) and SLOPE = 0.0277@(250 ± 5 pixels).

Figure 14 and Figure 15 prove that the Fourier transform method can eliminate the carrier frequency fringes in the demodulation result with a suitably sized filter window, but it is at the cost of high-frequency information, which will cause inaccurate demodulation results.

The demodulation results of the proposed SCFF method with an 18 × 18 filter window are shown in Figure 16. There are no obvious carrier frequency fringes in the demodulation results, and the step is steep. Calculated using the data in Figure 16, RMSE = 0.0132@(101~300 pixels) and SLOPE = 0.0256@(250 ± 5 pixels).

The demodulation results of the proposed SCFF method with a 20 × 20 filter window are shown in Figure 16. Compared with the demodulation results of the proposed SCFF method with an 18 × 18 filter window, the carrier frequency fringes are more obvious. Calculated using the data in Figure 17, RMSE = 0.014@(101~300 pixels) and SLOPE = 0.0274@(250 ± 5 pixels).

In order to facilitate the comparison of results, RMSE and SLOPE evaluation results of several demodulation methods are listed in

Table 1. RMSE and SLOPE evaluation results of several demodulation methods.

Methods	RMSE	SLOPE
Phase-shifting	0.0671	0.0332
Fourier transform filter window 15 × 15	0.0105	0.0109
Fourier transform filter window 18 × 18	0.1876	0.0277
Proposed method filter window 18 × 18	0.0132	0.0256
Proposed method filter window 20 × 20	0.014	0.0274

.

By comparing the demodulation results of several methods in Table 1, it can be seen that the phase-shift method cannot eliminate carrier frequency fringes and the Fourier transform method and the proposed SCFF method both have the ability to eliminate carrier frequency fringes; however, this is by sacrificing high-frequency information. Moreover, with the same size of filter window, the ability of eliminating carrier frequency fringes of the proposed SCFF method is better than the Fourier transform method.

4. Discussion

In this paper, a novel demodulation algorithm based on the spatial-domain carrier frequency fringes method is proposed. The simulation and experiment are carried out to verify that the proposed method can eliminate the carrier frequency fringes in the recovered topography, which means the measurement accuracy of the system is improved.

In summary, this paper solves a commonly existing but not noticed problem in structured illumination microscopy. By building the SCFF image with a set of full-period phase-shift fringe images, we can ensure that high-order harmonics and carrier frequency are separated in the frequency domain. Therefore, a large size filter window can be applied to extract the signal spectrum. And the high-order harmonics are mainly caused by the gamma effect of the measurement system camera and projector, which will lead to the introduction of carrier frequency fringe in the topography recovery results. The proposed method effectively eliminates the influence of carrier frequency fringe on the basis of preserving as much high-frequency information as possible, thus improving the measurement accuracy and reliability.