Deep Learning in the Phase Extraction of Electronic Speckle Pattern Interferometry

Abstract

Electronic speckle pattern interferometry (ESPI) is widely used in fields such as materials science, biomedical research, surface morphology analysis, and optical component inspection because of its high measurement accuracy, broad frequency range, and ease of measurement. Phase extraction is a critical stage in ESPI. However, conventional phase extraction methods exhibit problems such as low accuracy, slow processing speed, and poor generalization. With the continuous development of deep learning in image processing, the application of deep learning in phase extraction from electronic speckle interferometry images has become a critical topic of research. This paper reviews the principles and characteristics of ESPI and comprehensively analyzes the phase extraction processes for fringe patterns and wrapped phase maps. The application, advantages, and limitations of deep learning techniques in filtering, fringe skeleton line extraction, and phase unwrapping algorithms are discussed based on the representation of measurement results. Finally, this paper provides a perspective on future trends, such as the construction of physical models for electronic speckle interferometry, improvement and optimization of deep learning models, and quantitative evaluation of phase extraction quality, in this field.

1. Introduction

Electronic speckle pattern interferometry (ESPI) is a nondestructive optical measurement technique known for its simplicity, high speed, large field of view, excellent anti-interference performance, and suitability for dynamic measurements [1,2,3,4]. ESPI has numerous research applications in fields such as industry and engineering, including strain [5,6], displacement [7,8], vibration measurements [9,10], and material defect detection [11,12]. Phase information can be extracted by analyzing the interference fringe image, and the measured physical quantities, such as displacement and deformation, are closely related to the phase. As a result, one challenge in ESPI is the accurate extraction of phase information from interference fringe images for the analysis of the measured physical phenomena. Therefore, the difficulty, accuracy, generalization, and extraction speed of phase extraction methods are crucial for this technology.

Typically, image sensors and human eyes perceive and record the intensity distribution of light field through the photoelectric conversion effect. Because the frequency of a light wave being much larger than the sampling frequency, it is impossible to directly perceive and record the phase information of a light field. Fortunately, the phase delay changes the intensity distribution as the light field propagates. Consequently, the corresponding phase distribution can be deduced from the recorded light field intensity distribution, and the physical quantity to be measured can then be calculated through the phase transformation [13]. This process is known as phase extraction.

Current methods for phase extraction in ESPI can be categorized into two types, namely classical phase extraction methods, which are represented by techniques such as phase shifting [14] and carrier methods [15], and emerging data-driven methods based on deep learning. Phase shifting methods consider the known phase differences between multiple frames of fringe patterns, whereas carrier methods consider the frequency spectrum distribution characteristics of fringe patterns to extract phase information. Classical phase extraction methods have been widely used in industrial engineering, providing rich phase information and enabling economical and efficient noncontact full-field measurements. However, these methods involve complex and cumbersome steps requiring manual parameter tuning. Furthermore, dealing with noise, background interference, and phase discontinuities remains challenging. These challenges necessitate researchers to investigate novel methods for addressing these problems.

Deep learning network models, with convolutional neural networks (CNN) as a representative, have demonstrated the capability to extract complex features and patterns from images and achieved satisfying results in computer vision, data image processing, and related fields [16]. Some scholars have introduced deep learning into ESPI phase extraction and achieved notable results [17].

Because of the differences in measurement methods, the measurement results of ESPI technology typically manifest as ESPI fringe patterns or ESPI-wrapped phase maps. Deep learning methods are increasingly being used in both these results. This paper provides an overview of the fundamental principles and image characteristics of ESPI, emphasizing the application of existing deep learning methods in these two distinct measurement results. In addition, this paper compares the performance of various deep learning methods and highlights their advantages in addressing the challenges faced by conventional methods. Furthermore, the limitations of existing methods are detailed, and a prospective analysis of the development trends, covering aspects such as the construction of physical models for ESPI, improvement and optimization of deep learning models, and quantitative evaluation of phase extraction quality, is presented. This paper provided valuable insights for the advancement of phase extraction in ESPI.

The main contributions of this paper are as follows:

(1)

The application of the existing deep learning methods for phase extraction in the field of ESPI is comprehensively analyzed and summarized. By comparing the performance of different methods and the potential advantages of deep learning compared with traditional methods, this paper provides readers with a comprehensive understanding and evaluation of this topic.

(2)

The development prospects of deep learning methods for phase extraction in the field of ESPI are explored. This paper discusses the possible challenges of current deep learning methods in ESPI phase extraction, and it proposes directions for future improvement and development to provide a reference for researchers.

(3)

This paper provides knowledge and guidance for scholars who want to use deep learning methods to measure ESPI strain and displacement in the future.

2. Production of ESPI Images

Speckle was initially regarded as a common optical phenomenon. However, in holography, speckles considerably affect the quality of holographic images Therefore, in holography, speckles are considered noise, and reduction of speckle interference is a critical topic of research. In 1966, Ennos [18] discovered that randomly distributed speckles can be used as information carriers, including information on light intensity and phase. Since then, people have conducted considerable research on speckle measurement technology. In ESPI, a charge-coupled device is used to record speckle images, which are processed on a computer. Speckle is a noncontact full-field optical measurement method. This technology overcomes the limitations of previous optical measurement technology applications and is a speckle interferometric measurement method implemented by electronics and digital methods. This method has high accuracy, rapid response, and full-field noncontact.

Currently, the ESPI technology measurement system consists of several key components, including lasers, CCD cameras, computers, image acquisition cards, and some optical elements. Figure 1 displays a typical ESPI measurement optical path, where M represents a reflective prism, and BS₁ and BS₂ are spectral prisms. The laser beam is categorized into a measurement beam and a reference beam after passing through spectral prism BS₁, and the measurement beam is irradiated onto the measured object. After passing through the beam expander, the measurement beam interferes with the reference beam after passing through reflective prism BS₂, and the CCD camera records interference fringes, transfers the image information to the computer for processing, and displays the test results in real time.

The recorded speckle information before and after object deformation is expressed as follows:

$$I_1=I_0+I_r+2\sqrt{I_0{I}_r}\mathrm{c}\mathrm{o}\mathrm{s}\theta$$

(1)

$$I_2=I_0+I_r+2\sqrt{I_0{I}_r}\mathrm{c}\mathrm{o}\mathrm{s}\left(\theta +\phi \right)$$

(2)

where $I_1$ and $I_2$ represent the intensity of the speckle field before and after deformation, respectively; $I_0$ and $I_r$ represent the intensity of the two coherent beams, respectively; $\theta$ is the phase difference in the speckle field on the object plane; $\phi$ is the change inphase difference caused by object deformation. The relationship between $\phi$ and deformation displacement $W$ is as follows:

$$\phi =\frac{2\pi }{\lambda }W$$

(3)

By subtracting the light intensity of the speckle pattern before and after the deformation of the measured object, as follows:

$$∆I=I_1-I_2=4\sqrt{I_0{I}_r}\mathrm{s}\mathrm{i}\mathrm{n}\frac{2\theta +\phi }{2}\mathrm{s}\mathrm{i}\mathrm{n}\frac{\phi }{2}$$

(4)

With the change in the value of $\phi$, the light intensity difference $∆I$ also changes, which allows us to observe fringe information that is not visible in a certain light field intensity, resulting in alternating bright and dark fringe images.

The measurement results of ESPI are usually the ESPI fringe pattern derived from the formula and the ESPI-wrapped phase map obtained by installing a phase shifter on the reference optical path and using a stepwise phase shift. As displayed in Figure 2, both contain considerable noise, and the ESPI fringe pattern presents a black and white, alternating light and dark fringe structure, with the density and direction of the fringes reflecting the phase difference changes on the object surface. The ESPI-wrapped phase map limits the phase information to the range $\left[-\pi ,\pi \right]$ and displays it in grayscale or pseudo-color. The physical quantity to be measured is directly related to the phase distribution information of both. Therefore, studying advanced electronic speckle interferometry phase extraction methods is critical for quickly and accurately analyzing the measured physical quantity from the ESPI fringe pattern or wrapped phase map and has become a critical topic of research in current ESPI.

3. Application of Deep Learning in the Phase Extraction of ESPI Fringe Patterns

The fringe centerline method is a classic phase extraction method for ESPI fringe patterns. Compared with other phase extraction methods, such as Fourier transform and wavelet transform, the fringe centerline method is the most direct phase extraction method based on a single fringe pattern. In this method, first, the fringe pattern is filtered for noise removal, the skeleton line of the ESPI fringe pattern is extracted, the series of skeleton lines is identified, and the phase values at the location of the skeleton lines are obtained. Finally, a specific algorithm was used to interpolate the identified skeleton lines to obtain full-field phase information. Figure 3 displays the steps of the method. The following sections explain the application of deep learning in this method.

3.1. Application of Deep Learning in ESPI Fringe Pattern Filtering

The ESPI fringe pattern obtained from actual measurement contains considerable speckle noise. Therefore, prior to extracting the phase, the first step is to perform filtering processing. According to the various densities of fringes, the ESPI fringe pattern can be categorized into low-density fringe pattern, high-density fringe pattern, variable-density fringe pattern, and discontinuous fringe pattern. Conventional fringe pattern filtering methods include the sine and cosine mean method, the window Fourier transform method, and the two-dimensional wavelet transform method [20]. These methods are classified according to various objects. In the actual processing process, the following problems are persistent:

(1)

Slow processing speed, rendering handling multiple frames simultaneously impossible.

(2)

Suboptimal denoising effect on complex images, such as high- and variable-density ESPI fringe patterns.

(3)

Poor generalization, necessitating repeated fine-tuning for various objects, and lack of a general model.

(4)

Substantial computational parameters.

To address the problems of slow speed and the inability to process multiple frames simultaneously in conventional filtering methods, in 2019, Hao et al. from Tianjin University were inspired by the FFD-Net model [21] and introduced it into ESPI fringe pattern denoising [22]. This method can process 50 images simultaneously and requires only 8 s for processing, but the speed of the method is slow.

In 2020, Lin et al. [23] from Shandong University addressed the challenge of poor denoising performance of conventional methods for ESPI fringe patterns with high variation in density. They introduced the FPD–CNN network, which considerably outperformed the conventional windowed Fourier transform (WFT) method and coherence enhancing diffusion (CED) method, achieving the peak-signal-to-noise ratio (PSNR), the mean of structural similarity index (SSIM), and the mean absolute error (MAS) metrics of 27.88 (dB), 0.972, and 7.642, respectively, at a processing speed of just 0.57 s (Figure 4). The FPD–CNN method provides superior denoising performance for fringe patterns with varying densities. In 2022, Wang et al. [24] from Shenyang Aerospace University incorporated the squeeze and excitation (SE) attention mechanism into the DnCNN denoising model, introducing the AD-CNN network for ESPI fringe pattern denoising. This method achieved a PSNR of 20.1532 (dB) on a self-made dataset, surpassing the performance of the classic denoising model DnCNN and the WFF method, which achieved PSNR values of 17.0934 and 17.0068 dB, respectively.

To improve the generalization ability of the model, in 2022, Xu et al. from Tianjin University [25] constructed an MDD-Net network model using a skip connection method to filter various types of ESPI fringe patterns. The team applied this method to the measurement of the out-of-plane displacement of thermal changes in PCB boards, and its measurement accuracy can reach the nanometer level. This method outperforms existing SOOPDE, WFF, and BL-Hilbert-L2 methods in terms of detail protection, speckle reduction, and shape preservation under various types of density fringes. MDD-Net outperforms the FFD-Net proposed in reference [22] in discontinuous density fringe filtering denoising, the former having an SNR of 40.6 and SSIM of 0.991, which are superior to the latter’s SNR of 38.387 and SSIM of 0.965, indicating its superior generalization ability. Figure 5 displays the specific filtering effect.

This research achieved considerable success in terms of generalization and accuracy, but increased the model parameter size and reduced the processing speed. To solve this problem, in 2022, Gurrola-Ramos et al. from the Mathematics Research Center, Guanajuato, Mexico, proposed a lightweight residual dense network based on the U-Net structure (LRDUNet) [26]. This network reduced the number of parameters using grouped convolution and density convolution modules. The number of parameters in this method was only 4.21 M, and the processing speed reached 0.00665 s. This method has excellent denoising performance, with a PSNR of 40.1427 dB. Figure 6 displays the specific structure of this network.

Table 1. Comparison of denoising performance indicators for each model [21,23,24,25,26].

Performance Indicators	FFD-Net [21]	AD-CNN [24]	FPD-CNN [23]	MDD-Net [25]	LRDUNet [26]
PSNR (dB)	35.2269	25.0397	27.88	/	40.1427
SSIM	0.9932	0.9487	0.972	0.991	0.9980
MAE	0.0117	/	0.0135	/	0.0067
Parameters (M)	0.475	/	1.86	1.445	4.21
Running time (s)	0.024	0.0469	0.57	0.041	0.00665

Note: 1. The above data are the experimental results under their respective data sets. 2. The calculation formula for evaluation indicators is as follows:

compares the denoising performance indicators for each model, where the “Parameters” values of the FFD-Net and MDD-Net model were converted from the literature [22,26], respectively. The conversion formula is 1 M = 1024 K.

$$MAE=\frac{1}{m}{\sum }_{i=1}^m\left|y_i-y_i^{’}\right|$$

(5)

$$MSE=\frac{1}{m}{\sum }_{i=1}^m (y_i-y_i^{’}{)}^2$$

(6)

$$PSNR=10lo{g}_{10}\frac{({MAX}_I{)}^2}{MSE}=20lo{g}_{10}\frac{{MAX}_I}{\sqrt{MSE}}$$

(7)

$$SSIM=\frac{\left(2{\mu }_x{\mu }_y+c_1\right)\left({\sigma }_{xy}+c_2\right)}{\left({\mu }_x^2+{\mu }_y^2+c_1\right)\left({\sigma }_x^2+{\sigma }_y^2+c_2\right)}$$

(8)

$$RMSE=\sqrt{\frac{1}{m}{\sum }_{i=1}^m (y_i-y_i^{’}{)}^2}$$

(9)

In the above formula, $m$ is the number of samples in the dataset, and $y_i$ and $y_i^{\text{'}}$ prime represent the true value and predicted value of the ith sample, respectively. ${MAX}_I$ is the maximum value of the image point color; if each sample point is represented by 8 bits, then it is 255. μ_x and μ_y represent the mean values of images $X$ and $Y$, respectively. ${\sigma }_x$ and ${\sigma }_y$ represent the variances of images $X$ and $Y$, respectively. ${\mathsf{\sigma }}_{XY}$ represents the covariance of images $X$ and $Y$. $c_1$ and $c_2$ are constants.

The existing deep learning ESPI fringe pattern denoising methods require large models with poor generalization to different types of fringe patterns. Therefore, future development requires adaptability, high accuracy, and small model parameters.

3.2. Application of Deep Learning in the Fringe Skeleton Extraction Method

After filtering, the fringe skeleton lines should be extracted. The quality of skeleton line extraction directly affects the accuracy of phase extraction. The conventional fringe skeleton line is based on image binarization technology, which performs morphological operations and connectivity analysis on binarized images to extract fringe skeleton lines. Early representative skeleton extraction methods included a binarization threshold refinement algorithm [27], stripe peak tracking method [28], and binary graph method based on two-dimensional derivative symbols [19]. Subsequent studies have considered the characteristics of stripe direction and proposed new stripe skeleton line extraction methods, such as the coupled gradient vector flow field method [29], direction-coupled gradient vector flow field extraction method [30], anisotropic gradient vector flow field extraction method [31], and gradient vector field of variational image segmentation solution [32]. The conventional fringe skeleton line extraction method considers the image features and directionality of fringes, but has the following problems:

(1)

Low accuracy, poor generalization, slow extraction speed, and inability to process multiple frames simultaneously.

(2)

Poor extraction of damaged fringe patterns.

Deep learning has a higher tolerance to structural changes and requires less prior knowledge of the input image structure than conventional methods. Therefore, this method is used for extracting fringe skeleton lines. According to the processing methods, this method can be divided into image segmentation tasks and image generation tasks.

For (1), in 2019, Li et al. at Tianjin University proposed using the U-Net network for ESPI fringe skeleton line extraction [33]. This method treats skeleton line extraction as an image segmentation task from the background, successfully achieving skeleton line extraction. The method achieved an accuracy of 0.91 and a processing speed of 0.5 FPS and can extract different types of fringe skeleton lines with excellent generalization. However, this method performs poorly on fringe patterns with low contrast and considerable speckle noise. In 2022, the team improved on previous work by introducing the Channel-wise Cross Fusion Transformer into the U-Net network [34], which solved the problem of low accuracy. This method achieved an accuracy of 0.9878 and simultaneously performed phase extraction on the extracted fringe skeleton lines. Compared with conventional methods, the phase obtained by this method is smoother, and the performance regarding evaluation metrics is better, with an SSIM of 0.908 and root mean square error (RMSE) of 0.003. However, this method did not mention the average time for fringe extraction. Figure 7 displays a comparison of phase extraction results of various methods in the literature [34].

For problem (2), Liu et al. from Tianjin University proposed the M-Net network in 2020, which is an image segmentation task [35]. This method was used to analyze the experimental results obtained from the dynamic measurement of the thermal deformation of ceramic plates under strong light. Compared with conventional gradient vector field (VID-GVF) based on variational image decomposition [32] and U-Net [33] networks, M-Net achieved a superior extraction effect on the skeleton broken or missing stripes and can automatically fill in the damaged and broken parts. In terms of evaluation indicators, the accuracy, recall rate, and AUC score of the M-Net network were 0.959, 0.680, and 0.968, respectively, which were higher than those of VID-GVF (0.911, 0.531, and 0.735) and U-Net (0.952, 0.582, and 0.949). Figure 8 displays the specific experimental results, with red boxes representing damaged fringes.

At Hebei University of Technology, Wang et al. [36] considered skeleton line extraction as an image generation task. In the literature [36], Wang et al. used Pix2pix cGan [37] as a skeleton line extraction network, which can extract stripe skeleton lines and solve problem (1). This method is fast, reaching 11.7 FPS and an accuracy of 0.9704.

Table 2. Comparison of the performance of five methods for extracting skeleton lines [32,33,34,35,36].

Methods	Acc	Sensitivity	Specificity	F1	CC	Running Time(s)
D-GVF [32]	/	/	/	/	0.9775	/
U-net [33]	0.9133	0.6884	0.8621	0.6254	0.9408	0.5
CTrans U-net [34]	0.9878	0.8890	0.9887	0.8881	0.9905	/
M-Net [35]	0.9571	0.7412	0.9411	0.7411	0.9521	/
Pix2pix cGan [36]	0.9704	/	/	0.8174	0.8633	11.7

Note: The above data are the experimental results under their respective data sets. The calculation formula for evaluation indicators is as follows:

exhibits the comparison of the performance of five methods for extracting skeleton lines.

$$\mathrm{A}\mathrm{c}\mathrm{c}=\frac{TP+TN}{TP+TN+FP+FN}$$

(10)

$$Sensitivity=\frac{TP}{\left(TP+FN\right)}$$

(11)

$$Specificity=\frac{TN}{\left(FP+TN\right)}$$

(12)

$$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}=\frac{TP}{TP+FN}$$

(13)

$$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}=\frac{TP}{TP+FP}$$

(14)

$$F_1=2\times \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}$$

(15)

True Positive (TP): the number of samples that are actually positive and correctly predicted as positive.

True Negative (TN): the number of samples that are actually negative and correctly predicted as negative.

False Positive (FP): the number of samples that are actually negative but incorrectly predicted as positive.

False Negative (FN): the number of samples that are actually positive but incorrectly predicted as negative.

Existing deep learning methods have fewer applications in fringe skeleton line extraction, partly because the physical model for filtering denoising is more explicit, and more mature CNN models have been devised in other image processing fields, which can be directly imported into ESPI denoising. Most deep learning-based fringe skeleton line extraction methods add structural components in the encoder and decoder paths to adapt to ESPI fringe processing. Compared with the following methods, the previous method still requires specific algorithms for skeleton line extraction after filtering denoising, which increases the difficulty of fringe skeleton line extraction. Therefore, directly extracting fringe skeleton lines from the original fringe image without preprocessing could be the focus of research in the future.

3.3. Application of Deep Learning in the Interpolation Algorithm

The extracted fringe skeleton lines are discrete sampling points. To obtain the phase information of the entire field, an interpolation algorithm is required to predict the unsampled phase distribution of the entire field. Classic interpolation algorithms include bilinear interpolation [38], nearest neighbor interpolation [39], C spline interpolation [40], and interpolation methods based on the heat conduction theory [31]. Although these methods are simple and fast, they exhibit the following problems:

(1)

The interpolation result is not sufficiently accurate, and the resulting phase is not sufficiently smooth.

(2)

The quality requirements for the skeleton line are high.

(3)

Interpolation speed is slow.

Currently, the phase extraction interpolation algorithm for ESPI fringe patterns still relies on neural networks or simple machine learning methods, as the previous method displays in image format, whereas the subsequent one is related to mathematical methods.

In 2007, Tang et al. at Tianjin University first proposed using a BP neural network for fringe skeleton line interpolation [41]. Figure 9 displays the results, which reveal that the phase information obtained by the network is smoother. However, this method requires specialized partial differential equation methods for preprocessing and lacks specific evaluation metrics to quantify the interpolation results. Determining the true phase accuracy obtained from the interpolated image alone is impossible.

In 2011, Wang et al. from North China University of Technology proposed the radial basis function (RBF) neural network interpolation method, which solves the problem of high-quality requirements for skeleton lines and can obtain excellent results even for severely fractured skeleton lines [42]. However, similar to [41], specific quantitative metrics for evaluating the phase information obtained after interpolation were not provided. In addition, this method did not fully leverage the experimental data information from ESPI fringe patterns.

In 2020, Chen from Tianjin University proposed using hierarchical recurrent neural networks for interpolation processing, which solved the problem of slow processing speed [19]. In the thermal deformation experiment of steel plates under high-intensity laser irradiation, this method could perform interpolation operations on the fringe patterns obtained from dynamic measurements in only 4 s, which considerably lowered the 25 s required in the conventional method. Compared with the BP neural network method, the processing time of both methods was identical, but the phase information of the previous method was smoother. Figure 10 displays the specific interpolation results.

Currently, interpolation algorithms have not been improved considerably, and the existing literature lacks quantitative evaluation metrics for phase information after interpolation. In this paper, because the evaluation of phase information only based on the smoothness of the image was inadequate, other evaluation metrics were required for quantification, For example, comparing the phase information after interpolation of classic methods with that based on neural network interpolation, calculating SSIM or RMSE between the two can make the performance comparison between methods more specific.

4. Application of Deep Learning in Phase Extraction of the ESPI-Wrapped Phase Image

Figure 11 displays the phase extraction method based on wrapped phase maps. The wrapped phase map is obtained using phase shifting, is then filtered and denoised, and finally unwrapped to obtain the final full-field phase information. The following sections describe the application of deep learning in the phase extraction of ESPI-wrapped phase maps.

4.1. Application of Deep Learning in the Phase Shift Method

According to the different methods of changing the phase difference, the classical phase shift method can be divided into the time phase shift and spatial phase shift method. The time phase shift method changes the phase shift amount on the time series to measure the deformation of the object and obtain deformation information. The spatial phase shift method obtains phase information by applying various phase shifts at one time. According to the different phase shift amounts, the three-step phase shift method [43], four-step phase shift method [44], five-step phase shift method [45], and so on have emerged successively. Their main difference is in the number of interferometric images used and the measurement accuracy. This article is focused on the most commonly used four-step phase shift algorithm. The four-step phase shift method collects four speckle patterns before and after object deformation, with a phase shift of 90° for each time in the speckle pattern collection. The intensity distribution of the four speckle interference patterns collected by the CCD can be expressed as follows:

$$\begin{gathered} I_1\left(x,y\right)=A\left(x,y\right)+B\left(x,y\right)cos\left[\phi \left(x,y\right)\right] \\ {I}_2(x,y)=A\left(x,y\right)+B\left(x,y\right)cos\left[\phi \left(x,y\right)+\pi /2\right] \\ {I}_3(x,y)=A\left(x,y\right)+B\left(x,y\right)cos\left[\phi \left(x,y\right)+\pi \right] \\ {I}_4\left(x,y\right)=A\left(x,y\right)+B\left(x,y\right)cos\left[\phi \left(x,y\right)+3\pi /2\right] \end{gathered}$$

(16)

Solving the above equations simultaneously, we can obtain the following phase of the optical field:

$$\phi (x)=arctan\frac{I_4\left(X,Y\right)-I_2\left(X,Y\right)}{I_1\left(X,Y\right)-I_3\left(X,Y\right)}$$

(17)

Improved four-step phase shift methods such as 4 + 2 [46] and 4 + 1 time phase shift techniques [47] have since emerged.

From a mathematical perspective, the classical phase shifting method involves adding a known phase shift under various conditions to increase the number of equations, rendering the number of equations greater than or equal to the unknowns, and subsequently solving for the phase value. When using conventional phase shifting methods for phase extraction, selecting an appropriate method based on the differences in processing objects and acquisition speed is necessary. In practice, the following problems are observed:

(1)

Conventional phase shifting requires the use of multiple images for phase extraction, whereas phase extraction based on a single fringe pattern exhibits lower accuracy. Balancing the two to achieve higher phase accuracy using fewer images remains challenging.

(2)

The process of time phase shifting requires the acquisition of multiple images, which requires considerable time and limits dynamic measurement.

In 2019, at the Nanjing University of Science and Technology, Feng et al. first demonstrated that deep neural networks can be trained to perform fringe analysis [48]. This study proposed a two-stage convolutional neural network. In the first network, the background intensity information in the fringe pattern was removed, and the output of the second network was the numerator and denominator of the arctangent function, as shown in Equation (17). Finally, the phase was calculated based on the output of network using the arctangent function. In this method, the phase was extracted using a single fringe pattern, and conventional phase shifting methods were combined to solve problem (1). The average absolute error of phase extraction using this method, Fourier transform (FT), and WFT was 0.087, 20, and 0.19 rad, respectively. Thus, the phase extraction accuracy improved considerably. However, this method requires the calculation of numerous accurate intermediate data from experimental data before training.

To solve problem (2), Zhang et al., at Shenzhen University, proposed a training strategy for phase shifting networks (PSNs) in 2021 [49]. The input of the method is a single frame of built-in interferogram, whereas the output comprises multiple phase shifting interferograms. This simulates the process of acquiring multiple images in conventional phase shifting. The speed of this method is only 65 ms, which is faster than the 160 ms of classic four-step phase shifting and slightly lower than the 30 ms of the single-frame interferogram phase extraction method. However, in terms of phase extraction RMSE, this method is only 0.11 rad, which is considerably lower than the 0.51 rad of the single-frame interferogram phase extraction method and close to the 0.09 rad of the classic four-step phase shifting. Figure 12 displays the specific method of reference [49].

4.2. Application of Deep Learning in ESPI-Wrapped Phase Image Filtering

The result obtained using the phase shifting method is an ESPI-wrapped phase map. The phase of the wrapped phase map is typically in the range $\left[-\pi ,\pi \right]$ and contains considerable noise. Therefore, filtering should be performed before applying the unwrapping algorithm to obtain the full-field phase. Compared with ESPI fringe filtering, the filtering of ESPI-wrapped phase maps mainly deals with noise and false phase jumps. Classic filtering methods include the module $2\pi$ filtering method and the window Fourier filtering method based on convex optimization [19], but the following problems exist:

When severe noise is present in the ESPI-wrapped phase map, it leads to numerous false phase jumps. Conventional methods encounter challenges in handling false phase jumps during the denoising process, resulting in inaccurate unwrapping of denoised results.

(1)

Conventional filtering methods are time-consuming and require considerable parameter calculation.

(2)

Conventional methods have poor denoising capabilities for high-density and unevenly distributed phase maps.

To solve problem (1), Yan et al. from Shanghai University and Nanyang Technological University in Singapore used CNN to denoise the numerator and denominator of the noisy arctan function and subsequently obtain a clear wrapped phase map using the denoised numerator and denominator [50]. This method can remove noise levels with an SNR of −4 dB at a speed of only 8.4 s. Furthermore, using this method to denoise the wrapped phase map for phase extraction, the RMSE increased with the increase in the phase frequency but was less than 0.055 rad. Figure 13 displays the specific method.

To solve problem (2), in 2021, Tianjin University’s Li et al. introduced the DBDNet denoising network from other image denoising fields into the field of ESPI-wrapped phase map denoising [51]. The input of this network is the noisy wrapped phase map and the denoised phase map is the output, which theoretically reduces network parameters. Compared with conventional denoising methods OCPDE, WFLPF, and deep learning method FFD-Net, this method had a PSNR of 20.5648, which is significantly higher than the values of the previous three methods, 14.0865, 12.7554, and 17.3057, respectively. However, the method does not mention specific network calculation parameters. In subsequent work, the team improved and proposed a new network DBDNet2 in 2022 to solve problem (3) [52]. In ablation experiments, the denoising effect quantitative indexes (PSNR and SSIM) of this method were 19.1034 and 0.9078, respectively, which were higher than DBDNet’s 18.8819 and 0.8984, respectively.

Table 3. Comparison of ESPI package phase map filtering methods based on deep learning [50,51,52].

Research Studies	Method	Proposed Solution	Advantages	Disadvantages
Yao et al. [50]	Improved ResNet	Solves the problem of false phase jumps.	Excellent noise reduction performance; solves the problem of false phase jumps.	The model calculation parameters are too large.
Li et al. [51]	DBDNet	Solves the problem of phase denoising for large model calculation parameters and high-density and high-speckle noise.	Small number of model parameters; strong ability to recognize details.	Poor ability to obtain more comprehensive phase information.
Li et al. [52]	DBDNet2	Solves the problem of poor denoising ability for high-density noise uneven phase maps.	Excellent denoising performance and strong feature extraction ability.	May not be versatile.

summaries the ESPI package phase map-filtering methods based on deep learning.

Conventional ESPI image filtering methods and deep learning-based filtering methods have achieved considerable success, with their own advantages and disadvantages.

Table 4. Conventional and deep learning-based filtering methods.

Method	Advantages	Disadvantages
Conventional filtering method	The physical model has strong interpretability; it does not require a large amount of training data; and some algorithms are relatively simple.	Rely on accurate physical models; sensitive to parameter adjustments; unable to handle multiple objects simultaneously.
Filtering method based on deep learning	Strong adaptability; no parameter tuning is required after network training; batch processing is possible.	The demand for a large amount of labeled data; poor interpretability.

is a comparison of both methods.

Thus, conventional ESPI image filtering methods exhibit considerable advantages in physical model interpretability and algorithm simplicity, whereas deep learning methods have advantages in adaptability and handling complex relationships. Therefore, for datasets in various application scenarios, appropriate methods should be selected according to the needs.

4.3. Application of Deep Learning in the Phase Unwrapping Algorithm

The phase distribution obtained after the phase shifting process is typically encapsulated in an arctangent function, as shown in Formula (17). This phase distribution is wrapped, that is, the phase value only changes within the range $\left[-\pi ,\pi \right]$. This is a discontinuous phase distribution that does not match the true phase distribution. To obtain continuous and meaningful phase information, a phase unwrapping operation is required. Phase unwrapping is the process of unfolding the wrapped speckle phase into a continuous phase distribution to eliminate the periodic accumulation of phases, rendering the phase continuous and meaningful throughout the domain.

Currently, phase unwrapping methods based on deep learning can be classified into two categories, namely image segmentation-based methods and one-step direct unwrapping methods. “Segmentation” denotes determining the addition of two unknown integral multiples at each pixel in the wrapped phase to recover the true phase. One-step direct unwrapping methods learn the feature mapping between the wrapped phase map and the unwrapped image through convolutional networks.

4.3.1. Unwrapping Algorithm Based on Image Segmentation

In some cases, phase unwrapping is considered an image segmentation task. In 2018, G.E. Spoorthi et al. [53] of the Indian Institute of Technology proposed the deep learning phase unwrapping framework PhaseNet. Compared with the unwrap function in MATLAB and the quality-guided phase unwrap method (QGPU), PhaseNet has a phase MSE of 2 and a time cost of 0.18 s at an SNR of 0 dB, which is considerably better than the values obtained by the unwrap function (11 and 24 s for phase MSE and time cost, respectively) and QGPU (17 and 0.05 s, respectively). However, this method requires post-processing steps and can only unwrap phases within the range −36 to +36 radians. In 2020, the team proposed PhaseNet2.0, which increased the unwrapping range to 100 radians and eliminated the necessity for post-processing in PhaseNet [54]. Compared to QGPU, MATLAB unwrap function, least square phase unwrapping (LSPU), and calibrated least square phase unwrapping (CLSPU) methods, PhaseNet2.0 unwraps phases more smoothly on the image, with root mean square of 2.5%, which is significantly better than the values of 178.2%, 85.6%, 24.3%, and 26.3%, respectively, obtained by the former methods. Figure 14 displays the specific unwrapping effect.

Similarly, considering phase unwrapping as an image segmentation task, in 2019, Zhang et al. from Hangzhou Dianzi University used an extended DeepLabV3+ network as the backbone to solve the problem of conventional methods being prone to failure in high-noise datasets [55]. Using a Gaussian white noise dataset for phase unwrapping, the RMSE of this method is 1.2325, demonstrating considerably superior unwrapping performance compared with the conventional path-following method with an RMSE of 19.7591. In 2019, Zhang et al. from Shenyang Institute of Automation of the Chinese Academy of Sciences and the School of Optical Sciences of the University of Arizona in the United States combined CNN denoising network and CNN segmentation network to improve performance [56]. The RMSE of this method is only 0.0027, and the processing speed of the GPU is 0.22 s, which is significantly better than 6.5202 and 19.53 s of the branch cutting algorithm, and 6.2683 and 330.44 s of the QGPU method, respectively.

4.3.2. One-Step Direct Unwrapping Algorithm

Another group of scholars considered the one-step unwrapping method. In 2019, at the Northwestern Polytechnical University, Wang et al. introduced Residual Block into the U-Net network and proposed a one-step unwrapping method deep learning phase unwrapping (DLPU) [57], as displayed in Figure 15. The SSIM of the phase-unwrapped image and the reference image reached 0.991. In terms of robustness, compared with the least squares (LS) method and the quality guidance (QG) method, when the true phase height increased from 5 to 100, the SSIM index of DLPU decreased from 0.99 to 0.80, which was significantly better than those LS and QG methods (down to 0.2). The maximum error pixel ratio of DLPU is 0.2, which is 3.55 times lower than the ratios of the LS and QG methods (0.71). Subsequently, at the Beijing University of Technology in 2020, Qin et al. were inspired by VGG, U-Net, and ResNet, and proposed the VUR-Net method [58]. Xu et al. from Tianjin University [59] considered reducing the computational burden when improving the accuracy of phase unwrapping and proposed the PU-M-Net method. This approach effectively reduced speckles during the phase unwrapping process. On a high-noise dataset, PU-M-Net achieved an SSIM of 0.810 and MSE of 1.673, outperforming the LS method with 6.870 and 0.562, as well as the QGPU method with 2.585 and 0.771, respectively.

Table 5. Summary of unwrapping methods based on deep learning [53,54,55,56,57,58,59].

Researchers	Get Phases Directly	SSIM	MSE	RMSE	Advantages	Disadvantages
G.E.Spoorthi et al. [53]	No	/	2	/	High accuracy	Requires post-processing
G.E.Spoorthi et al. [54]	No	/	/	/	High accuracy	Lack of comparison with similar methods
Zhang et al. [55]	No	/	1.5608	1.2325	High accuracy	There is a significant error in high-speckle conditions
Zhang et al. [56]	No	/	/	0.0027	It can identify phase discontinuity	Requires pre- and post-processing
Wang et al. [57]	Yes	0.991	3.150	/	Directly obtain phase and high accuracy	Limited dynamic range, and the effect at phase jump points still needs to be improved
Qin et al. [58]	Yes	0.816	1.737	/	Excellent performance	Significantly increases training costs
Xu et al. [59]	Yes	0.810	1.673	/	High computational efficiency	Lack of generalization analysis

Note: The above data are the experimental results under their respective data sets.

compares all the deep learning-based unwrapping methods mentioned above.

Based on the analysis, phase unwrapping methods based on image segmentation usually use simple networks and require preprocessing or post-processing to obtain reliable results. However, one-step direct unwrapping methods use more complex networks and can directly obtain reliable results.

Compared with conventional unwrapping algorithms, deep learning-based unwrapping methods exhibit a strong generalization ability, fast processing speed, and accurate phase information. However, these methods still require a large-scale dataset. Therefore, improving the accuracy and feature extraction efficiency of the network as well as reducing the requirements of the dataset are problems that need to be addressed in the current research. Addressing dataset requirements while achieving one-step direct unwrapping of the phase with high accuracy becomes a prominent topic in this field.

5. Summary and Outlook

Compared with conventional methods of phase extraction from electronic speckle interferometric images, deep learning methods have demonstrated advancements in terms of accuracy, processing speed, and generalization. Research in this field is still in its nascent stages, and several promising avenues warrant in-depth exploration:

(1)

Integration of the physical model into the deep learning model.

The generation mechanism of ESPI images is well understood, whereas deep learning models are considered “black boxes” with challenging internal mechanisms. Integrating prior knowledge with the mature generation process of ESPI images into deep learning models for different application scenarios can become a novel research direction for developing new data-driven deep learning models.

(2)

Improvement and optimization of deep learning models.

Most existing deep learning models focus on the classical models such as CNN and their enhancements. With the continuous development of deep learning technology, novel models are expected to emerge. In the next phase, investigating the application of various models to phase extraction in electronic speckle interferometric images is crucial. Moreover, existing deep learning models are limited to specific steps in the phase extraction process and lack an end-to-end complete model. Therefore, introducing new deep learning models and constructing end-to-end phase extraction models will become a prominent research direction in the future.

(3)

Quantitative evaluation of phase extraction quality.

Existing phase extraction methods based on deep learning models are typically applied to specific steps, such as filtering and fringe skeleton extraction. Evaluation metrics for the algorithm are confined to these intermediate steps, without focusing on the quality of phase extraction. This renders the quantitative evaluation and comparative analysis of phase extraction quality challenging. Therefore, addressing the urgent need for a quantitative evaluation of phase extraction quality is essential.

(4)

Limited practical engineering applications.

At present, the ESPI phase extraction method based on deep learning is in its early stage of development, and thus is mainly only applied in the laboratory. There are only a few notable applications of these algorithms outside the laboratory, and there are currently very few practical applications in the field of industrial engineering. The application of deep learning methods to ESPI phase extraction has great potential in industrial engineering and other fields. This application may result in more efficient and accurate phase extraction methods and provide better support for industrial applications in related fields.

(5)

Expanding the dataset.

Currently, limited publicly available ESPI image datasets are available in the industry, hindering the validation of the practical generalization ability of deep learning models and impeding meaningful horizontal comparisons between research outcomes. In the next step, studies should focus on generating ESPI image datasets through numerical simulation and constructing experimental systems in conjunction with practical application scenarios. A comprehensive analysis of factors influencing image quality and further dataset expansion is essential.