DEFECT DEPTH ESTIMATION IN INFRARED THERMOGRAPHY WITH DEEP LEARNING

Infrared thermography has already been proven to be a significant method in non-destructive evaluation since it gives information with immediacy, rapidity, and low cost. However, the thorniest issue for the wider application of IRT is quantification. In this work, we proposed a specific depth quantifying technique by employing the Gated Recurrent Units (GRUs) in composite material samples via pulsed thermography (PT). Finite Element Method (FEM) modeling provides the economic examination of the response pulsed thermography. In this work, Carbon Fiber Reinforced Polymer (CFRP) specimens embedded with flat bottom holes are stimulated by a FEM modeling (COMSOL) with precisely controlled depth and geometrics of the defects. The GRU model automatically quantified the depth of defects presented in the stimulated CFRP material. The proposed method evaluated the accuracy and performance of synthetic CFRP data from FEM for defect depth predictions.


INTRODUCTION
Non-destructive evaluation (NDE) has emerged as an important method for the evaluation of the properties of components or systems without damaging their structure. Several state of the arts methodologies such as Pulsed Phase Thermography (PPT) [1], Principal Component Thermography (PCT) [2], Differential of Absolute Contrast5 (DAC) [3], Thermographic Signal Reconstruction (TSR) [4], as well as Candid Covariance Free Incremental Principal Component Thermography [5], have been implemented to process thermographic sequences and improve the defect visibility. These techniques can be beneficial for qualitative analysis of composite materials. However, to introduce a method for conducting quantitative study (defect depth estimation) with deep learning is a novel topic to be explored.
Quantitative analysis is playing an important role in the modern industrial field of non-destructive testing (NDT). Defect characterization is one of the current topics of interest in quantitative data analysis of active thermography. In this topic, thermographic information involves extracting quantitative subsurface properties from defects such as defect depth, lateral size, thermal resistivity, from an experimental Thermal Non-destructive Testing (TNDT) dataset utilized to characterize defects. Several approaches have already been proposed to analyze the depth in the region of defects in pulsed thermography. In general, these approaches evaluate the defects depth by using the maximum thermal contrast C_max, the instant of maximum thermal contrast Tc_max or artificial neural networks to analyze the depth of defects based on mathematical equations. The Peak Temperature Contrast Method [6] estimated the depth based on the characteristic time from peak contrast. The peak contrast corresponding to the maximum contrast has a proportional correlation with the square of the defect depth. Daribi et al. [7] proposed neural networks for defect characterization of defect depth. The results demonstrated that the networks should be trained by representative and nonredundant data in order to obtain a high degree of classification accuracy.
In this work, there is an attempt to detect the depth of defects in Carbon Fiber Reinforced Polymer (CFRP) via Gated Recurrent Units (GRUs) [8]. GRU is an updated recurrent neural network (RNN) particularly designed for time series prediction. GRU can be considered as a variation of Long Short-Term Memory (LSTM) [9]. Compared with the LSTM and RNN temporal model, the GRU has adapted a few learning parameters which could save computational expenses for training and obtained an excellent performance. According to our knowledge, this is the first time that the thermal temporal characteristic model (GRU; RNN; LSTM etc.) is used to qualify the depth of defects.
We modeled a 3D version of CFRP specimen stimulation from COMSOL. Then, it was further tested on the systemic data by the GRU model to validate its accuracy. The remainder of this paper is structured as follows: Section 2 provides the pulsed thermography theory and conception indication, as well as the detailed characteristics of FEM simulations. Section 3 proposed a GRU model-based defects depth estimation strategy and introduces the GRU deep learning model architectures. Section 4 provides the experimental results analysis. Section 5 concludes this paper.

PULSE THERMOGRAPHY
In pulsed thermography (PT), a high-power exponential heating impulse is applied to the samples, and a thermal response is measured during a period of time. Due to the heat conduction, a surface region which has an internal defect underneath the surface perturbs the thermal waves propagation on the surface of specimens in comparison to the sound (non-defective) region. We can then see the changes of the temperature variation, since the internal defects possess different thermo-physical (conductivity, density and heat capacity) that have an impact on heat flow. These thermal differences can be observed as surface features and recorded with an infrared camera as indicated in Figure 1 [10].
Temporal evolutions can be observed from the defective regions and subsurface sound regions. A thermal contrast is acquired as a feature vector which is obtained distinctly via the thermal value from the defective region subtracted from the corresponding value from the surrounding sound region [11] as indicated in Equation (1), where Td(t) is the temperature value on the pixel point of the defect area. The temperature value on the reference point of the sound area is T_s(t). Then the ∆T(t) is the absolute thermal contrast extracted from the defect and sound region. The thermal contrast is an excellent technique to distinguish the temperature difference to learn the depth of the defects, as shown in Figure 1.

FINITE ELEMENT MODELLING WITH TRANSIENT HEAT TRANSFER
Finite Element Modelling (FEM: COMSOL, etc.) [12] has become an important and economical platform to evaluate the thermal response of pulsed thermography, which builds up models for the platform of components that allow us to flexibly examine all specific physical aspects of thermal data such as geometries and properties of materials. Simulated thermograms matched well with experimental data.
In this work, COMSOL will be utilized as a 3D based simulation to build up models for the synthetic data to provide for depth estimation of artificial defects. A Carbon Fiber Reinforced Polymer (CFRP) structure-based specimen containing artificial defects of different shapes (flat-bottom holes) is modeled as shown in Figure 2. A high thermal pulse is projected on the surface of specimens. Due to the existence of the temperature gradients in the sample, a thermal front propagates from the high temperature region on the surface to the region underneath. A delamination or discontinuities create a lower thermal diffusion rate to the heat flow and then reflect the unnormal thermal patterns on the surface. The COMSOL software is utilized as the heat transfer simulation model for obtaining the temperature evaluations of the surface of each sample, as indicated below [13]: where in Equations (2)-(4), the density is ( ), and the absolute temperature is T(K).
Time variable is set as t(s). A rectangular coordinate system (x, y, z) as in Equation (3) in anisotropic media can lead to numerous possible solutions. w, , , ! 6 ) 5 7 are the main conductivity rates and conductivity rates which respectively for three coordinates (x, y, z) in 3D thermal modeling. The convection heat transfer is given by # $%&' In Equation (4), a boundary condition that regarding to the thermal heat transfer from radiation and convection between the all the specimen surfaces and the ambient temperature has been indicated, where ()* is an initial environment temperature which consider all the external environment as the same temperature;, $ 6 ) 8 / 9 is the Stefan-Boltzmann physical constant which links the temperature with energy; ε represents surface emissivity. Table 1 briefly illustrates the physical properties of CFRP specimens and pulsed thermography parameters involved in the COMSOL simulation in this experiment. All properties and parameters in Table 1 are cited from the literature [14]. Table 2 and Table 3 provide the description of defect characteristics. Each row in every CFRP samples has the same depth of defect, which included 22 different constant values from 0.5 to 2.2 to be set from the first row in training sample 1 to the last row in the training sample 6. We extracted 5 vectors from each defect region for GRU model training with depth estimation. The testing group has 4 CFRP samples (30:30). In order to differ from training samples, the testing group consists of samples A, B, C, D. The defects in the test samples range from [0.5, 2.2] but have a size differs from that of the training samples. The CFRP geometrics of the structure of training sample 1 is stimulated based on Figure 2.

TEMPERATURE AND THERMAL CONTRAST CURVES
In transient thermography, previous researchers [6] concluded that the time of maximum contrast ∆T that corresponds to the maximum contrast ∆T in the temperature contrast curves has an approximately proportional relationship with the square of the depth of the defect (A B ). Simultaneously, the proportionality coefficient of this relationship rested with the lateral size of the depth: the smaller the defects, the lower maximum contrast ∆T and earlier peak time. As indicated in Figure 3, an interesting case is highlighted. We observed the temperature evaluation of three defects of the same size (18 :18) but different depth such as (0.5 mm,1.0 mm,1.5 mm). Although the three defects have the same shape, it can be demonstrated that the shallower the defect, the higher the peak temperature value which can be obtained.
In Figure 4, a corresponding synthetic thermogram frame for training sample 2 in t=15s generated from COMSOL is indicated at (a); Nine artificial flat bottom (b) flat-bottom-holes were embedded with different depths in the shapes of either circles or squares. In Figure 5, the data distribution of the thermal contrast curves is illustrated. Five contrast vectors have been extracted above each defect region on the surface from the different points in the defects (upper left; upper right; center; lower right; lower left; lower right) to reduce the inaccurate influences caused by the small temperature variation. Notice that, all the training sequences to be processed in this work are extracted from the first 1409 frames of a whole 2000 frames thermal sequence in total (including the information of peak thermal contrast CDE and corresponding CDE.) As a result, the last 581 frames of thermal curves were not extracted in this work which would show a dramatic decrease of the thermal contrasts in the graphs. This partial extraction in COMSOL saves the computational expense for the training in GRU to some extent.

PROPOSED STRATEGY FOR DEFECTS DEPTH ESTIMATION
In this section, a defect depth estimation strategy has been proposed to detect and identify the depth of each defect in thermal images as indicated in Figure 6. This design of detection system originated from the GRU neural network. The infrared thermal module and simulations are provided by COMSOL (FEM simulation) to examine the depth of defects in pulsed thermography (PT). First, the synthetic thermal sequences are acquired from FEM (COMSOL) based on the heat transfer modeling in FEM. Then, several thermal contrast feature vectors are extracted from each defect region to feed into the input of GRU. The output of GRU consists of a unique node which estimated the depth during the training. The mean absolute error is chosen as the loss function with the GRU model in the equation as below Equation (5). In the end, the predicted depth output from the GRU is based on the feature extraction of thermal contrast vectors.
The GRU takes each vector at a time point in the input. This learning model was trained for 2500 epochs of each process. The training loss converged to the optimistic value and then flattened. In addition, the number of training curves (batch size) is set m. I, I J are the ground truth and estimated depth respectively.

GATED RECUR1RENT UNITS MODEL WITH DEPTH ESTIMATOR
Due to the time continuity of the thermal sequence, each frame collected from the experiment is related to the recent historical frame; therefore, the time series memory deep learning model can be applied to the thermography data based on this feature. The GRU model is originally from RNN which is a time series model can handle the continuous information like thermal sequences [15]. During the cooling period of the thermal data, the temperature evaluation curves over time are acquired from the given infrared frames. The Learned GRU model is able to distinguish whether the pixel from a defective region or a sound region due to the training period. Therefore, the multiple units of GRU could be applied to extract the features of the temperature evaluations from the samples based on physical properties.
The GRU neural network in Figure 7 are structured as follows [16]. In equations (6)-(9), Q , R represent the update, reset gate units respectively. S (!) , S (T) , S are the weight factors for each gate unit. E is the current input. ℎ UV is the input from the previous time step in the hidden state. ℎ , is the input of current memory from the hidden state. W (X) , W (Y) , W % are the weight factors for previous time information ℎ UV . Z(. ) represents the sigmoid function. ° is denoted as the Hadamard product. [17] ℎ , =tanh _S(` ) + Q °W % (ℎ UV )a (8) As shown in Figure 8, in this work, the original thermal sequences were reshaped into vectors. The particular thermal contrast vectors are directly fed into the input of the GRU network structure. Each thermal contrast vector in the defect region is decoded with the depth value of corresponding defect at the output of GRU based on the thermal properties from the training sequences. In order to select the points for the simulated thermal sequences to extract temperature curves vectors, 5 different locations inside each defect surface in these defective areas were selected. Since the temperature of the defect region is not even, these selected points accounted for small temperature variations and change above each defect surface region. Each thermal contrast vector is vectorized with the same length in the time of the thermal sequence. Therefore, the input values of GRU are fed into the particular thermal contrast vectors. The output from the decode section (to be connected with the fully connected layer) are set with the corresponding defect depth of each vector. The estimated depth values in the defect region output from GRU are based on the thermal properties from the training sequences.

INFERENCE AND TRAINING
In this research, the training processing on GPU (NVIDIA GeForce GTX 1080Ti) took about 30 min. The operating system was set as: Ubuntu 14. 04. CPU: i7-5930k. Memory: 64 GB. Adam was introduced as optimizer. The whole training process was also conducted using the Adam optimization. Some main hyper parameters and training parameters are set as below: weight decay 0.0001, the learning rate 0.001 and learning momentum 0.9. In this work, the temperature variety and contrast reflect on each vector. Therefore, the GRU time step was set to 1429 (1429 frames as a time step as input).

DATA PROCESSING
To reducing the overfitting issue, the cross-validation verification method is proposed in this study. First, we collected 270 thermal contrast curves from the simulation data (5 thermal contrast curves extracted from each defect averagely). Then, we shuffled all of the data and spilt 80% of the data to train with GRU model, while 20% was utilized to validate the performance of the GRU.
All obtained training data (the thermal contrast evaluation curves) is normalized and truncated as a fixed length of duration. Simultaneously, the input data is normalized by subtracting from the mean value of the thermal curves ( b c ) and dividing by the standard deviation of the thermal contrast ( , c ) using Eq. (10).   Table 4 The results of depth estimation of defects located in the designated specimen * MAE means mean square error. An average over sample for absolute different between actual and predicted observation.

Results analysis -Mean Absolute Error (MAE)
In statistics, the mean absolute error (MAE) [18] is one of the metrics to evaluate how close the forecasts are to the eventual outcomes. In the machine learning field, it can indirectly reflect the accuracy and performance of the machine learning model (GRU). In this work, we adapted the MAE to assess the performance of the GRU for depth estimation with infrared thermography. As we can see from Figure 9 (a), we trained the GRU to estimate the data from testing before the data normalization. The obtained training loss is 0.055. The MAE converged to 0.0165. The error between the predictive value and actual value is within the range of N−0.17CC, 0.17CC]. After the standard deviation normalization for all the distributed data in Figure 9 (b), the predictive value trends to approach the actual depth for the defect. The MAE error shrinks to the range of N−0.11CC, 0.11CC] and the training loss converged to 0. 0295. This shows an acceptable performance with an improved estimate of the depth by the GRU model with standard deviation normalization.
In table 4, the estimated output 1 and the MAE 1 are obtained from raw data without normalization. The estimated output 2 and the MAE 2 result from raw data with normalization. Based on Table 4, the calculated accuracy in the GRU model for the depth estimation reached 90 % before data normalization (standard deviation). After normalization, the results provide an accuracy greater than 95 %. This performance demonstrated that the GRU enable a high performance for accurate depth estimation. This estimation is attributed to the ideal environment without experimental issues (noise; defective pixels). As shown in the Figure 10 below, the thermal data distribution from training (before; after) normalization has been indicated. In the data distribution(a), each group of color data curves (yellow; red; green) represents a different specific depth from defects. The thermal data is normalized by Eq. (10) in Figure 10 (b). The distinguishable features of difference between the depths is recognized by the following principle: shallow defects have greater maximum thermal contrasts that occur earlier than deep defects. These results outperformed the previous works obtained from [7] for depth estimation in automated infrared thermography with regular neural networks.
(a) (b) Figure 9. The mean absolute error and the training loss with GRU before (a) and after (b) standard deviation normalization (a) (b) Figure 10. The data distribution before (a) and after (b) standard deviation normalization from all the selected locations for training

CONCLUSION
This work elaborated the complicated and non-linear issues of evaluating defect depths in composite materials via infrared thermography with a GRU learning model. The methodology proposed here employed a GRU model combined with pulsed thermography to analyze the depth of defects. The simulated samples provide an economical platform for GRU training and depth estimation. Quantitative analysis of defect depth (subsurface features) has been evaluated by a GRU based statistical method through developed neural network modeling and cross validation experimental verification. It has been proven that the GRU modeling can produce an advanced depth detection. For future work, the experimental data have to be evaluated for the robustness of the GRU model. Further, other types of deep learning models and modified versions of the GRU model have to be applied to increase the depth estimation ability in this topic.