Assessment of Fringe Pattern Decomposition with a Cross-Correlation Index for Phase Retrieval in Fringe Projection 3D Measurements

Phase retrieval from single frame projection fringe patterns, a fundamental and challenging problem in fringe projection measurement, attracts wide attention and various new methods have emerged to address this challenge. Many phase retrieval methods are based on the decomposition of fringe patterns into a background part and a fringe part, and then the phase is obtained from the decomposed fringe part. However, the decomposition results are subject to the selection of model parameters, which is usually performed manually by trial and error due to the lack of decomposition assessment rules under a no ground truth data situation. In this paper, we propose a cross-correlation index to assess the decomposition and phase retrieval results without the need of ground truth data. The feasibility of the proposed metric is verified by simulated and real fringe patterns with the well-known Fourier transform method and recently proposed Shearlet transform method. This work contributes to the automatic phase retrieval and three-dimensional (3D) measurement with less human intervention, and can be potentially employed in other fields such as phase retrieval in digital holography.


Introduction
Fringe projection optical three-dimensional (3D) shape measurement methods are becoming more and more popular in recent years due to their ability to provide high-resolution, high-speed, whole-field 3D reconstruction of objects in a non-contact manner. They have been extensively investigated and widely used in numerous fields such as industrial and scientific, biomedical, kinematics, biometric identification, and cultural heritage and preservation applications [1][2][3][4][5]. Phase retrieval is a key step in fringe projection measurement, and is of fundamental importance to the successful application of the method [1,6]. Phase retrieval can be achieved from multiple frame fringe patterns with well-known phase shift algorithms or from single frame fringe patterns with the well-known Fourier transform method [7,8]. In the measurement of objects in fast motion or in a temporally unstable environment, it is difficult or costly (e.g., using a high speed camera) to take several fringe projection patterns in an extremely short period of time. Therefore, phase retrieval based on a single frame fringe projection pattern is highly desirable in these cases.
By now, phase retrieval from a single frame fringe projection pattern has been extensively studied. Interested readers may refer to [1] for a comprehensive review of phase retrieval in fringe projection profilometry. The Fourier transform method and window Fourier transform (WFT) method are two widely used and well-known single frame phase retrieval techniques [8,9]. In addition, the wavelet transform method and the more recently developed empirical mode decomposition (EMD), variational image decomposition (VID) and Shearlet transform-based methods [10][11][12][13] have been proposed for phase retrieval from a single projection fringe pattern. Although the principles of these methods are different, many of them rely on background elimination, which can be formulated as a fringe pattern decomposition problem.
Parameter selection in single frame projection fringe pattern phase retrieval is important but has received less attention. For instance, the Fourier transform-based fringe decomposition and corresponding phase retrieval performance are related to the filtering window size [8]. Shearlet transform decomposition and corresponding phase retrieval results are related to the decomposition layer [13]. An inappropriate parameter value may degrade the decomposition results. Therefore, it is important to choose an appropriate value of model parameter to produce desirable results. Usually, the optimal parameter selection is conducted manually by trial and error with lots of experiments due to the lack of decomposition assessment rules.
In this paper, we propose a cross-correlation criteria to assess the fringes and background decomposition for automatically selecting the optimal parameter of Fourier transform and Shearlet transform-based fringe pattern decomposition methods. The proposed cross-correlation index is calculated by using the decomposed background part and fringe part, and thus it does not require the ground truth data. The contribution of the paper is twofold: first, the cross-correlation index to assess the decomposition results of fringe pattern is proposed and verified to be simple but feasible. Second, the proposed cross-correlation metric is suitable for the Fourier transform and Shearlet transform parameters selection and maybe extended to other phase retrieval methods such as WFT and EMD. The organization of this paper is as follows: in Section 2, a brief introduction of Fourier transform and Shearlet transform method with corresponding parameter descriptions is presented. After that, the cross-correlation metric of the decomposed background and fringe is proposed. In Section 3, the proposed cross-correlation metric is verified by simulated and experimental data and results discussion are given. Section 4 concludes the paper.

Fourier Transform Method for Fringe Projection
In fringe projection, a fringe pattern captured by a CCD can be expressed as: where a(x, y) is the background, b(x, y) and φ(x, y) are the modulation intensity and the optical phase, and f 0 is carrier frequency. Equation (1) can be rewritten in complex form where c(x, y) = b(x,y) 2 exp(j(φ(x, y))), and c * (x, y) denotes the complex conjugate. The Fourier transform of fringe pattern I(x, y) with frequency f 0 is comprised by spectrum components separated from each other: whereÎ v x , v y is the Fourier transform of fringe pattern I(x, y) with frequency coordinate v x , v y , A v x , v y is the Fourier transform of a(x, y), C v x , v y is the Fourier transform of c(x, y) and , v y denotes the Fourier transform of fringe part. This allows selectively filtering with a window to obtain the background spectrum component Then by applying the inverse Fourier transform, one can obtain the background part and fringe part. The fringe part is: where F −1 denotes inverse Fourier transform. The decomposition results are depended on the Fourier transform parameter (filtering window size), which should be carefully selected.

Shearlet Transform Method for Fringe Projection
In the Shearlet transform method, firstly, a forward Shearlet transform is performed on I(x,y) to obtain the transform coefficients: where SH denotes forward Shearlet transform operator on image I(x,y); secondly, the transform coefficients undergo a hard thresholding before an inverse Shearlet transform is carried out [13]. Thirdly, an inverse Shearlet transform will operate on the remaining resultant Shearlet coefficients to obtain the fringe part: where i is the decomposition scale, and SH −1 is inverse Shearlet transform. In this study, we investigate the cross-correlation of the background part and fringe part. Therefore, we set the thresholding with threshold value of zero and set the Shearlet coefficients of the first scale of SH I to be zero. By these, the background of a projection fringe pattern can be removed, and the fringe part can be obtained. The Shearlet transform parameter (decomposition layer i) needs to be carefully selected.
With the obtained fringe v(x, y) in Equation (4) or Equation (6), the phase distribution with carrier is calculated as follows: where H denotes the Hilbert transform, Re{ } and Im{ } respectively denote the real and imaginary parts, and ϕ c (x, y) is the carrier which should be removed to produce a pure phase.

The Proposed Cross-Correlation Index for Decomposition Assessment
As mentioned above, the decomposition result depends on the parameter selection and the result assessment requires an assessment index. It can be supposed that fringe pattern f = u + v is decomposed into background u and fringe v. Background varies slowly, while fringe exhibits texture feature. Here we assume that the background part and fringe part are highly uncorrelated or lowly correlated and that they have small cross-correlation distance metric. Given the background part u and fringe part v, the cross-correlation distance metric (CrossUV) is: where u and v are the mean values of u and v respectively, M and N are the sizes of u. The error of the finally unwrapped phase is: where ψ(i, j) and ψ 0 (i, j) are the retrieved and theoretical phase values, respectively. The Structural Similarity (SSIM) metric is calculated on decomposed fringe x and referenced fringe y, which is: where µ x , µ y , σ x , σ y and σ xy are the local means, standard deviations, and cross-covariance for images x and y, and c 1 and c 2 are regularization constants [14].

Experimental Results and Discussions
Next, we use the cross-correlation metric to test the relation of background and fringe decomposed from simulated and real fringe projection patterns. In this study, the well-known Fourier transform method and recently proposed Shearlet transform method are employed to decompose the fringe projection pattern. The adopted fringe patterns are respectively shown in Figures 1-3. Figure 1a,b are simulated fringe pattern with different carry frequencies. The simulated projection fringe patterns of a sphere shape with abrupt changes (Figure 1) with the sizes of 512 × 512 pixels are generated by: with phase: where Re{ } denotes the real part. The spatial frequency of the fringe pattern is set to f 0 = 1/8 for Figure 1a and f 0 = 1/16 for Figure 1b, and the background illumination a(x,y) is 0.5 × φ(x, y) which makes the background outside of object region different to the background inside of test object, the modulation intensity b(x, y) is 1. Figure 1c,d show the noisy fringe patterns corresponding to Figure 1a,b with Gaussian random noise with variance of 0.2 added [12]. In addition, Figure 1   Similarly, Figure 2a,b are simulated fringe pattern with smooth changes with the sizes of 512 × 512 pixels. They are generated by: I(x, y) = a(x, y) + b(x, y) cos(2 × peaks(x, y) + 2π f 0 x) (13) where peaks(x, y) is the peaks function in Matlab (Mathworks, Natick, MA, USA), a(x, y) = 5 × δ(peaks(x, y))/δx, b(x, y) = 0.5. The carrier frequency f 0 for Figure 2a,b are respectively 1/8 and 1/16. Figure 2c,d show the noisy fringe patterns corresponding to Figure 2a,b with Gaussian random noise with variance of 0.2 added [11][12][13]. In addition, Figure 2 Figure 2c,d show the noisy fringe patterns corresponding to Figure 2a,b with Gaussian random noise with variance of 0.2 added [11][12][13]. In addition, Figure 2 Figure 4 shows the diagram of phase retrieval of Fourier transform and Shearlet transform. With the derived fringe, the wrapped phase is obtained by Hilbert transform and arc tangentatan operator on the decomposed fringe part. Further, the unwrapped phases were obtained by quality guided phase unwrapping algorithm [15]. To obtain the pure unwrapped phases without the carrier term, the carrier was removed from the unwrapped phases by Zernike fitting method [16]. To sum up, the decomposed fringes, and unwrapped phase are obtained, from which the assessment indexes of error and SSIM are calculated to give overall assessment of decomposition results. To use the true data in  With the derived fringe, the wrapped phase is obtained by Hilbert transform and arc tangentatan operator on the decomposed fringe part. Further, the unwrapped phases were obtained by quality guided phase unwrapping algorithm [15]. To obtain the pure unwrapped phases without the carrier term, the carrier was removed from the unwrapped phases by Zernike fitting method [16]. To sum up, Sensors 2018, 18, 3578 9 of 25 the decomposed fringes, and unwrapped phase are obtained, from which the assessment indexes of error and SSIM are calculated to give overall assessment of decomposition results. To use the true data in the assessment of experimental fringe pattern, the fringes part and unwrapped phase by four steps phase shift method are considered as the true data [12].
Sensors 2018, 18, x 9 of 25 the assessment of experimental fringe pattern, the fringes part and unwrapped phase by four steps phase shift method are considered as the true data [12].  the assessment of experimental fringe pattern, the fringes part and unwrapped phase by four steps phase shift method are considered as the true data [12].    (a-1) (a-2) (a-3) (e-1) (e-2) (e-3)             Figure 9 shows the assessment index of CrossUV, SE and SSIMV for Figure 3 by the Fourier transform method. Figure 10 shows the decomposed background and fringe for Figure 3 by the Shearlet transform method with decomposition layers of 3 and 4. Figure 11 shows the retrieved phase and phase error for Figure 3b by the Fourier transform under optimal CrossUV, SE and SSIMV and Shearlet transform method under different decomposition scales. Table 1 shows optimal CrossUV, SE, and SSIMV computed from simulated and experimental fringe patterns by Fourier transform method. The positions of optimal CrossUV, SE, and SSIMV are shown in the plots of Figures 5,7 and 9 in the case of the minimal CrossUV, SE and maximal SSIMV. Table 2 shows the CrossUV, SE and SSIMV computed from the decomposed simulated and experimental fringe patterns by Shearlet transform method with decomposition layer 3 and 4. Specially, the CrossUV, and SSIMV in Table 2 are from the decomposed fringe parts in Figures 8 and  10, and the SE in Table 2 from the retrieved phase in Figure 11.
Further, it can be drawn that the parameter with minimal cross-correlation is generally consist that with the minimal SE and SSIMV. The optimal decomposed results also show this accordance. For instance, the optimal SE and CrossUV in Table 1 for Figure 1b are both under the 8th parameter. Also, the optimal SSIMV exists at the 8th parameter value. With these, we can conclude that the quality of decomposition results can be assessed by cross-correlation of decomposed fringe and background, i.e., smaller cross-correlation metric corresponds to better decomposition results and phase retrieval results.
For the Shearlet transform, as shown in Figures 8, 10 and 11 as well as Table 2, the decomposition results vary with different decomposition layers. Taking the results for Figure 3b for example, the decomposed background with a decomposition layer of 3 still contains a lot of fringes as shown in Figure 10(c-1). The results lead to a larger cross-correlation metric of 7.16 × 10 −1 as illustrated in Table 2, compared to 4.10 × 10 −3 which is obtained from the decomposition with decomposition layer of 4. Overall, decomposition results with decomposition scale 3, in terms of CrossUV, SE and SSIM, is better than that from decomposition results with decomposition scale 4 for the fringe patterns with a large frequency, which are fringe patterns with a small frequency. On the contrary, the decomposition result with decomposition scale 4, in terms of CrossUV, SE and SSIM, is better than that with decomposition scale 3 for a small frequency, which are fringe patterns with small frequency. It is also seen that the minimal CrossUV, SE, and maximal SSIM are at the same decomposition scale, which demonstrates that the index of cross-correlation is able to assess the decomposition results.    It is also noted that while 2D image quality assessment has been an active research topic, 3D image quality assessment is more difficult and lacks new quality metrics. On one hand, we only use the SE to assess unwrapped phases (3D data). On the other hand, phase unwrapping is a difficult problem leading because the unwrapped phase is sensitive to the decomposed fringe. As shown in Table 1, the optimal parameter position with respect to cross-correlation, to some extent, deviates from that with SE of unwrapped phase and SSIM of decomposed fringe. This deviation might be related to the accuracy of unwrapped phases, or the accuracy of retrieved wrapped phase. The future work is to reduce the deviation by considering these two issues. However, cross-correlation of decomposed background and fringe generally indicates quality of the decomposition results and phase result quality and can be used as an assessment index for decomposition. It is also noted that while 2D image quality assessment has been an active research topic, 3D image quality assessment is more difficult and lacks new quality metrics. On one hand, we only use the SE to assess unwrapped phases (3D data). On the other hand, phase unwrapping is a difficult problem leading because the unwrapped phase is sensitive to the decomposed fringe. As shown in Table 1, the optimal parameter position with respect to cross-correlation, to some extent, deviates from that with SE of unwrapped phase and SSIM of decomposed fringe. This deviation might be related to the accuracy of unwrapped phases, or the accuracy of retrieved wrapped phase. The future work is to reduce the deviation by considering these two issues. However, cross-correlation of decomposed background and fringe generally indicates quality of the decomposition results and phase result quality and can be used as an assessment index for decomposition.

Conclusions
In this work, we have conducted a performance assessment of the decomposition results of fringe patterns under different parameter values and proposed a cross-correlation metric index to assess the decomposition results. The results from both the Fourier transform and Shearlet transform

Conclusions
In this work, we have conducted a performance assessment of the decomposition results of fringe patterns under different parameter values and proposed a cross-correlation metric index to assess the decomposition results. The results from both the Fourier transform and Shearlet transform methods demonstrate that an optimal (minimal) cross-correlation index exists under a set of parameter values. The optimal decomposition for fringe pattern in terms of cross-correlation is verified by the decomposed fringe with SSIM index, and is also verified by the unwrapped phase with SE index. Our proposed cross-correlation metric for the assessment of the decomposition results without the need of ground truth data is simple, and yet feasible in the application of automatic parameter selection in phase retrieval methods of Fourier transform and Shearlet transform, and may be extended to WFT and EMD, etc. Future work will focus on the improvement of the proposed method in the application of Fourier transform method with respect to the assessment accuracy and speed. This work should prove beneficial for the automatic 3D fringe projection measurement with less human intervention, and could be extended to other fields such as phase retrieval in digital holography.

Conflicts of Interest:
The authors declare no conflict of interest. Figure A1 shows the decomposed results for Figure 1a by the Fourier transform method with different parameter values. Subfigures (a-s) are the backgrounds decomposed by the Fourier transform method with 1st to 19th filtering window sizes of [2:1:20], respectively. Figure A2 shows the decomposition results for Figure 1b  methods demonstrate that an optimal (minimal) cross-correlation index exists under a set of parameter values. The optimal decomposition for fringe pattern in terms of cross-correlation is verified by the decomposed fringe with SSIM index, and is also verified by the unwrapped phase with SE index. Our proposed cross-correlation metric for the assessment of the decomposition results without the need of ground truth data is simple, and yet feasible in the application of automatic parameter selection in phase retrieval methods of Fourier transform and Shearlet transform, and may be extended to WFT and EMD, etc. Future work will focus on the improvement of the proposed method in the application of Fourier transform method with respect to the assessment accuracy and speed. This work should prove beneficial for the automatic 3D fringe projection measurement with less human intervention, and could be extended to other fields such as phase retrieval in digital holography.

Conflicts of Interest:
The authors declare no conflict of interest. Figure A1 shows the decomposed results for Figure 1a by the Fourier transform method with different parameter values. Subfigures (a-s) are the backgrounds decomposed by the Fourier transform method with 1st to 19th filtering window sizes of [2:1:20], respectively. Figure A2 shows the decomposition results for Figure 1b by the Fourier transform method with different parameters. Subfigures (a-n) are the backgrounds decomposed by the Fourier transform method with 1st to 14th filtering window sizes of [4:2:30], respectively. Figure A3 shows the decomposition results for Figure 3a by the Fourier transform method with different parameter values. Subfigures (a-q) are the decomposed backgrounds by the Fourier transform method with 1st to 17th filtering window sizes of [4:1:20], respectively. Figure A4 shows the decomposition results for Figure 3b